The spell text doesn’t define what happens when the spell wears off, so it would be up to the individual DM to decide if that’s the case. being charmed by charm person simply means, as others have pointed out, that the charmer is regarded as a trusted friend and ally and their words and actions are perceived in the most favorable way possible. I think any conjecture about how people respond to being charmed beyond what’s written is something that should be decided on the campaign level, as either interpretation could work.