I really don't know much about it beyond the anecdotes I've heard from friends and others, but aren't there cases of psychotic behavior (going berserk) and also suicide? BTW, I wouldn't use the term "natural" as an indicator of how benign it is -- the following are also natural: arsenic, anthrax, mercury, water hemlock, and castor beans.
I do it all the time, though I avoid a few that have poisonous lookalikes. I picked some oyster mushrooms on the way home from the grocery store today in the middle of town. Even a child can safely identify a number of great edible mushrooms.
