It would be a rather spectacular coincidence to be born or employed in the optimal spot. After your last job tied to a specific office, factory, or field, you should probably go somewhere, but where?
Outside of the continental US, I am a fan of both Dominica and Puerto Rico. Within the US, I advocate creeping across state borders to nab comparatively better tax statuses, such as moving from Vermont to New Hampshire or moving from New York to Connecticut.
Connecticut's advantage over New York could widen as NYC attempts to raise taxes again and again. My home town of New Canaan has real estate opportunities from time to time. We bounce back and forth between Connecticut and Maine depending on the season.
If I had a clean slate, then Wyoming would probably be at the top of my list for an ideal place to retire. But if you want to drill down to a specific town, then where? I love small town America. With the possible exception of New Zealand, it is where I am most comfortable.
My first screen for an ideal retirement spot is average wealth due to positive externalities on crime, healthcare, and services (I used the Bloomberg Index of Affluent Micro Areas). Then I removed towns with state income taxes. The winners? In third place, Claremont-Lebanon NH. The median home value is about $200,000 and citizens don't have a state sales tax or income tax (but have a 5% tax on dividends and interest). New Hampshire's crime rate is the second lowest in the US, just behind Vermont; its murder rate is less than 2% of St. Louis or Baltimore.
In the number two spot, another New Hampshire town: Laconia, NH between Lake Winnipesaukee and Winnisquam Lake. Average homes cost $220,000 (more on the waterfront). With nearby New York, Connecticut, and Massachusetts among the most taxed states in the country, New Hampshire offers a competitive alternative along with low crime, a clean environment, and natural beauty.
Ski towns including Steamboat Springs and Breckenridge dot the Bloomberg Index's richest micro areas. My number one favorite small town for retirement is one of them: Jackson, Wyoming. Wyoming retirees pay no income tax (and they are next door to Montana which charges no sales tax). Its crime rate is among the three lowest in the US. It is one the most beautiful states (Thrillist ranks it at #8) from awe-inspiring mountains and glorious ranches (my favorite one on the market today is Marton Ranch in Alcova). Average homes in Jackson cost $412,000. For my money, it is the best bet retirement in small town America.
If you ever tire of skiing in Wyoming, check out Chamonix. Shareholders ski free and has been a great pair with a Peak Resorts (NASDAQ:SKIS) short on top of paying some of its dividends in lift tickets - so far the long side is up over 10% and the short side is down over 20% with several years of world class skiing tossed in for free.