Hi, i got this from my maintenance manual.
" to prevent the outboard from coming up or trailing out, when shifted into reverse or throttling back rapidly, oil in The trim system must be lockedin a static position. This is accomplished with the up and down pilot check valves. Thus, not allowing oil in the system moving in either direction"
Without knowing the motor more, i would suspect a leaking check valve due to debris in the system, i would recomend either dissasembly or replacement of the trim system. If you have a manual it should lay out dissasembly and removal, aswell as some troubleshooting flow charts.
Also, if your not familiar with what a check valve is, its basically a ball bearing with a spring which only allows flow in 1 direction, if debris gets in the system it can cause the ball to not seat properly and then it will allow fluid to flow in both directions giving you symptoms as you described....
When your motor is trimmed up, does it slowly creep down? When its trimmed down can you lift it up manually, if you can do either of those its definetly a check valve problem.
Good luck!