Yes, I believe that your understanding is correct, as confirmed in a discussion on Telegram Developers Channel pasted below--Todd Fleming is a contributor to EOSIO codebase:
16th of August
User: is there any way for an user to remove RAM allocated that they own ? let’s say that I own a malicious contract that once an action hit,
check ram of user, and fullfill it with [useless] data, and I dont
provide a remove method
Todd Fleming: The contract is the only thing that can free the ram
User: and you cannot do anything if contract consume so much ram of user
Todd Fleming: Right. Be careful what contracts you send actions to.
Update 27th August: This mechanism enables what is now considered an exploit.
A contract can intercept a
transfer action when it's being sent tokens and use up the unused RAM of the sender. In other words, if you send tokens to an account that has a contract, it is possible that the account will use your unused RAM, if any. Details here, and a temporary solution where you use an intermediate to interact with other contracts has been proposed here. This is particularly dangerous for exchanges that manage withdrawals from an account with unused RAM.
Update 9th September: The exploit below has been fixed in EOSIO. See @PhillipHamnett-EOS42's answer.