Yes! If a user has add/remove (term) access to any given asset, they may link terms to those assets via the asset itself. However, they cannot link the asset to a term from within the glossary.
The reason for this is that the user has explicit access to add/remove terms, but not to update the glossary.