From my extensive research I can tell you exclusively that the following three things do not exist:
- A software development methodology that will always successfully deliver
From my experience it seems that the first two have become reasonably well accepted as myths, even if occasionally they end up being used as metaphors for things that might exist in the real world.
However it seems that the third one we struggle with. So let's make this quite clear.
If you, or someone you know, has a stupid idea, no amount of Waterfall or Agile or whatever comes next method will make that idea less stupid. The only difference is that if you run a Waterfall approach you won't find out it's a stupid idea until you've spent a lot of money. If you are running in a truly Agile way you are likely to find it is stupid earlier, and cheaper.
Except, of course, people have been increasingly sold the idea that Agile is a magic method that can turn lead into gold and turn stupid ideas into good ones. While it is true that in the world of The Lean Startup some companies have been able to pivot from stupid to good, what is often forgotten is that we suffer from a cognitive glitch called survivorship bias that means we focus on successes and ignore the things that didn't work. It is survivorship bias in part that keeps people playing the lottery. That and terrible education about the basics of statistics.
Agile does not make undeliverable things deliverable. It does not reduce your rate of project failure. If anything, if you think about it, if you are taking a truly Agile approach you should see an increase in projects that are stopped, but they will be stopped before you have spent as much money as you would have done in Waterfall approaches. Unless you are using Agile approaches for projects where Waterfall approaches work better (known problems with known solutions, for the most part) in which case you are in for a world of pain.
Nobody likes the idea of failure. Despite exaltations for us to Fail Fast and Embrace Failure, it's such a culturally loaded term, and such a sliding scale from "acceptable" to "you're fired!" that I'm increasingly concluding that becoming friendly with failure is a fool's errand. But until we can get our heads around the idea that not all ideas are good ideas, and even if everyone thinks this idea is a good idea you won't know until you try, modern approaches to software development and delivery will continue to disappoint because they aren't mermaid unicorns.