Should we have sex? is it too soon?

Ive been unofficially dating my man for 5weeks now. I dont know if we should have sex or not. he has always said he is happy to wait, but recently he thinks we are ready. i know we both care for each other and our relationship is good, but if we do i also know it will have bad consequences for me since im the female and will be viewed as easy and a whore. I know i sohuldnt care what other people think if we both feel comfortable enough to do it, but i also dont want to ruin my social life, i know my close friends wont mind, but thing are good at the moment, and i dont want to ruin it just for having sex. However, i can tell he really wants to, and although he says its not a problem, forget about it, i know it does annoy him slightly and he does really want to. I dont know what to do for the best.