though I have someone, I feel more alone

I love my boyfriend, I really do, but him being white makes me feel alone. Not the fact that he doesn’t understand me or my culture, because he understands, but he doesn’t understand how the looks and the whispers bother me. I don’t understand why they don’t bother him? 

I asked him, “Do you think it would be easier if I was white?” and he said yes. I don’t know why I would think he would answer Does it matter? or No because I still have to face issues if you are white. But I did and it hurt me to hear yes and I didn’t want to continue the conversation.