My boyfriend takes drugs sometimes and I feel threatened by it?

I've been dating a guy for a few months and recently became his girlfriend. He's a year younger than me (I'm 19), he's sweet, funny, interesting, and I really like him. But he takes drugs sometimes when he's out with friends. He'll smoke weed with his boys about once a week and sometimes on big nights out once every 2 months or so he'll take ecstacy. I feel threatened by it, half because I'm scared for his safety, and half because I'm scared he'll be all messed up on something and get with some girl. I can't really pinpoint it it just bothers me somehow. I've dabbled myself but not on the scale he does, I don't want to be controlling and ask him to stop, so how can I get over this?