I'm probably old enough now that I'm outside of any group of people who would approach me to act like I'm a freak, and floating 265-270 I'm not. But every now and then, leaving the gym, someone may rant on about how they used to be in shape or I may get asked how I do it - usually by people who will never, ever do it (if you're over 30 and you have to ask how do you make it to the gym....).
But, every time this has happened I feel that people are entirely over-reacting. If you vaguely go to the gym, someone at some point will be amazed and make a comment about it. It gets harder when you have kids and/or a time consuming job, but if you really love something, you will do what you can to find a way even if it means circuit training.