There are many things that should be prohibited if we did it all because people could hurt themselves on them. The government where I live, for instance, have banned the use of certain types of fireworks. Why? Because some idiots have been combining them with alcohol-intoxication and been injured. So much for the idea that people are wise enough to take care of themselves...
IMHO Cannabis is potentially dangerous to you, and you shouldn't be "fooling around" with it. However, that I don't want it, does not validate prohibiting others from doing what they want with their own bodies. And as long as the ones using these substances aren't bugging people around them, I see no reason why I should bother!