Exempt from wearing masks

I am exempt from wearing a mask and when i go out i wear a badge, but i feel very self conscious about it because people stare at me , and some of my friends have told me when they have been out shopping they have been pulled up by other customers for not wearing masks.  In all shops and hair salons they have signs on doors saying no entry without masks, but I think they should also say except people who are exempt ,  Has anyone else had any issues with this.