Explaining Myself

I was thinking last night, people will ask me if we have kids, which is understandable, but then when I tell them we don't have any, they will often ask why.
Usually, they will ask in a way like they're leading. "You just neverrrrr......"

Fortunately, it's our choice, but suppose we have some medical reason that prevented us from having kids? How incredibly rude is that?

Mind your own business, people!