His drug use is causing conflict in our relationship or am I being too harsh?

Well, first and foremost you need to educate yourself on drugs, or at the very least the ones your boyfriend is taking and says he is not addicted to. Also, talk to recovered addicts or a detox counsellor to affirm this for you. I am not going to go into here, it would just take far to long, and with a great deal of controversy from those still practising as well. Secondly, there is no such thing as a responsible daily drug user, or addict/alcoholic. You will always come second to the drugs, always, whether you choose to believe this or not. If you stay with him long enough you will most definitely see this for yourself and after much time has ped, there will also be not much left of your self esteem, and dignity, and it is very exhausting living like this. Thirdly, we lie, alot! Lastly, but most importantly, your denial I'm afraid has allowed you to accept much less for yourself and for your life, than you would ever otherwise do, sorry to say. Bottom line! Either get out while you still can because it will get harder and harder to, and you will become someone you never though you would. Or go to al-anon which can show you ways to cope and try to remain yourself while you remain living with an addict, but this really will become futile towards the end. Take care, and educate yourself for your sake, please.