Yes, I believe that your understanding is correct, as confirmed in a discussion on Telegram Developers Channel pasted below--Todd Fleming is a contributor to EOSIO codebase:
16th of August
User: is there any way for an user to remove RAM allocated that they own ? let’s say that I own a malicious contract that once an action hit, check ram of user, and fullfill it with [useless] data, and I dont provide a remove method
Todd Fleming: The contract is the only thing that can free the ram
User: and you cannot do anything if contract consume so much ram of user account ?
Todd Fleming: Right. Be careful what contracts you send actions to.
A temporary solution where you use an intermediate to interact with other contracts has been proposed here.