Is my boyfriend slightly bi and should I be concerned?

What does it matter? You've been together for 3 1/2 years and you know he's not going to run off with another guy or girl. Even if he does have a bisexual nature, it doesn't matter because he is committed to you. If your sex life hasn't changed, if his personality hasn't changed, or any important factors haven't changed, it shouldn't matter. He may be like some women, who have no sexual desires toward a woman, but they do think she is pretty... it could be something similar to that.