It has changed in Tokyo.
The first time I went in 2003, people were smoking everywhere, but they had just banned smoking on the train platforms.
When I was there this last summer, you couldn't even smoke on the street anymore. They had big adds about not walking around and smoking.
yeah they have little smoking areas everywhere....and the signs on the ashtrays are hilarious

edit: found them right here