I think that context is really important. I've mostly seen this term used by neurodivergent people when expressing frustration with not being understood by the general populace. Also, these conversations were usually in spaces created by and for neurodivergent people, so the use of normie to indicate everyone else makes sense to me. In that context, it always comes across as kind of self-deprecating to me, an acknowledgement that the person speaking isn't considered normal because of their condition.
Based on the context you've described, I'm not surprised you don't like the term. If that was were I first encountered it I wouldn't like it either.
And if recording isn't allowed, bring a notebook and pen. I frequently take notes at the doctor's because I don’t trust myself to remember all the important bits later.