Could be that if it doesn't fit just right, it's uncomfortable. If he says it's a little too big why not suggest getting it resized? I k n ow quite a few married men that don't wear their wedding rings for many different reasons. Mostly they are all Machinist and work in grease, oil, engines and all sorts of things that could ruin it. Some guys also aren't into jewelry.
If it makes you that upset, why don't you just talk to him? When my hubby and I got married, he said he'd probably never wear his ring because he works with boats and tools and he didn't want to lose it or get it messed up. He's never taken it off since I put it on when we got married. It's all scraped and scratched with dings and dents. But it means alot to him to wear. But even if he didn't wear it, he's still mine. LOL.
Just tell your hubby how you feel and see what he says. Maybe he has a good reason he's not wearing it. ya never know til you ask.