Is something wrong in my marriage?

Okay so me and my husband have been together for three years we've been married though for 3 months. Now we used to have sex about 4 times a week, then it slowly went down to 2-3 and now it's been about a week and a half since we have had sex. We have a daughter together and I am a stay at home mom but I also take care of my mother that lives with us because she has MS. Which my husband works nights which he loves and he loves my mother and helps me when ever he can he remodeled her bathroom so she could get in there with her wheelchair. So I really don't think that has anything to do with it stress wise. But my husband does have back problems which he goes to the doctor for and I understand that to. But it makes me feel like he doesn't find me attractive. Now I forgot to mention that we are in our early 20s. I might be being selfish but it really makes me feel horrible about myself. When we do have sex its great for both of us and he let's me know. And yes I know for a fact he isn't cheating. Because if he isn't at work he is here with the family, and he never comes home late or goes to work early so that isn't possible. But I just want some ideas of what could be going on. I've talked to him about it and he gets irritated with me and says I don't know why you would think I'm not attracted to you. Oh and not to mention when he looks at porn ( doesn't bother me) he "you know" and just tonight I asked him what he looked at and he wouldn't tell me he didn't want me thinking he was weird? Please I need some advise! And don't be RUDE!