I have more confidence exploring way out there with one--grew up hiking and riding on dirt roads and paths, but once I'm out of home territory, I know how easy it is to be lost. And then my dumb ass might put an unecessary burden on the local search-n-rescue guys and gals.
I like clamping my RAM mount onto the bars, powering up my cheap waterproof Garmin 76CX, and heading out at random. If I find a cool road I want to take, I just ignore my directions and take it. Later, I have a track of the ride, and can show it to others more easily.
In strange cities, I can almost always find the gas station and the gay bar, no problem.
One other cool thing--often other folks (GPSExchange and ADVrider, etc.) are generous with their GPS tracks, and you can download 'em and put 'em on your own GPS and head off for shenanigans.
Some nice guy on ADVrider made me a track of logging roads on Vancouver Island. These were either labeled wrong on the not-great maps I had, or I just wouldn't have known how to pick the right way through the maze of roads and forests (while avoiding most of the active logging areas for safety's sake). I got to spend days riding alone through towering forests, beside the ocean, next to waterfalls in the middle of nowhere, thanks to the guy's kindness and my GPS. I even managed to get a bit lost a couple of times.
Just don't forget to learn to read and follow and carry paper maps as well, and learn to ignore the GPS when fun beckons. And always carry spare batteries. Someday I'll hardwire mine to the bike battery properly.