When a drug company starts off a new project, a lot of things go into the decision. Most of them are scientific decisions, but a big one that isn't is the projected market size. It's a business, and if you keep developing things that don't earn out their costs (and plenty more), you won't be part of the business for long.
These market numbers aren't the most reliable in the world; Pfizer (NYSE:PFE), for example, appears to have been surprised by how well Viagra did, and Bayer (BAY) and Lilly (NYSE:LLY) were likewise surprised that their follow-ups didn't repeat. For a more recent example, try Pfizer's Exubera. Its potential as a big winner was already much eroded by the time it finally made it to market, but surely it's selling even below the worst projections.
But underserved markets give you something you can depend on. A safe, effective anti-obesity drug would clearly reap billions, not that I'm expecting to see one. An effective HDL-raising therapy would do the same in the cardiovascular market (but hold on tight if you're trying to develop one of those, too). And CNS is full of opportunities, like Alzheimer's. Mind you, those opportunities are there because people keep trying and failing to do much for the diseases, but there's definitely a fortune waiting for the first thing that does.
As you can see, the risk-reward curve is pretty similar to what you see in finance. If you want the big returns, you have to take the big risks. "Big risk" is a relative term around here, though, since even the plainest of vanilla rip-off me-toos can implode on you, taking all its costs with it. But in general, it's the same no-free-lunch graph as everywhere else in the world.
There are some exceptions, but the problem (as always) is that it's usually impossible to see them coming. Lipitor is the first example that comes to mind ; Warner-Lambert just about killed it because it was going to be the umpteenth statin, and they didn't think its market share would justify the development costs. (I should have mentioned that one back in the first paragraph, when I was talking about shaky market projections!) It was only after the drug got well into the clinic that its potential began to show itself, just as Exubera was far along before its deficiencies became clear.
On a macro level, one of the big problems is the disconnect between underserved markets and underserved populations. Tropical diseases like malaria are an instant example. An effective antimalarial would be taken by huge numbers of people, but many of them still couldn't begin to afford the cheapest pharmaceuticals in the world, which is a real dilemma. Of course, there's also the possibility that the sudden introduction of such a drug might help precipitate a Malthusian crisis in countries with traditionally high death rates, but better to deal with that than have the current situation, I'd say.
There are several methods that have been tried to bring things in line. The Orphan Drug Act is an example from inside the US (making diseases with smaller numbers of patients more financially attractive), and there's perennial talk of something similar for tropical diseases through prizes and other incentives. A different world would do things still differently, but we don't, to the best of my ability to see, live in one.