The only time I worry about it is when my stash is getting low.
You can do whatever you want in moderation and if it doesn't affect your work or life then I don't really see a problem in it. I drink about a 12 pack of beer a night and smoke bud but it doesn't effect my work or life so I don't look at is as a problem.
