America is the land of the free, home of the brave and we are generally really proud of our freedom. Over the last decade or so, many of our rights have been limited, partially due to the attacks on September 11th, and partially as people continue to become more private as social networking and sharing grows.
One of the most important things you can have on you and know, is The Photographer's Right, printable PDF you can download put together by Attorney Bert P. Krages II. It's a comprehensive, foldable article that he suggest you keep on you or in your camera bag at all times. The entire PDF has been transcribed into this Flickr discussion, with examples of photos people have taken.
Andrew Kantor has a nice rant on the misinformation of your photographic rights, some of it is rather comical. Fredrik Silverglimth also covers your rights as a photographer a little more in depth, and provides some sample model release forms for download.
What all of these articles cover is the same basic subject matter though. Your rights as a photographer in America are pretty loose, and there are really only a few legal reasons for you to not take photographs. Most of the exceptions make pretty common sense, like military installations or other items that could be a breach of National Security, inside locker rooms or changing rooms in stores as they could be a breach of privacy or on private property.
There have been a few cases reported recently of photographers taking pictures from a public street or sidewalk of buildings and getting harassed by building security. Note, most buildings aren't public property but photographing them is not a crime from a public street.
Likewise, if you are in a public place and you are photographing other people in a public place, strangers, celebrities, etc., it is not illegal. They can ask you to stop, but legally, there is nothing that can be done. Crime scenes, accidents and natural disasters are also not illegal to take photos of.
As a photographer, there should be some discretion made on your part though. It's not illegal to hang out at the playground and photograph kids, but it could be somewhat suspicious to others. Know your rights, keep up to date on them and know that avoiding confrontation is most likely always the safest bet. Having a camera doesn't give you a license to do whatever you want, but it also doesn't mean others can dictate what you can't photograph.