The Scrum methodology is often introduced because developers want it. On paper, it empowers them—if applied correctly—and developers need decision-making empowerment because development is a creative endeavor.

Of all the different types of people I've known, hackers and painters are among the most alike. What hackers and painters have in common is that they're both makers. Along with composers, architects, and writers, what hackers and painters are trying to do is make good things.
—Paul Graham

On the other hand, there is a shortage of highly skilled developers and companies think claiming to "do agile development" is a low cost option to make themselves more attractive to developers.

So, Scrum is being widely adopted and quickly becoming the de-facto standard methodology for software development.

A Forrester study from 2009 observed that 35% of organizations used agile. Actuation Consulting’s 2013 research shows that 73.68% have adopted agile to develop products. In comparison, only 12.5% of respondents use waterfall.

Unfortunately, the sad truth is that around 70% of agile adoptions fail. This happens because Scrum, and agile, are much more complex to adopt than they appear to be.

Ken Schwaber [...] suggests only 30% [of scrum-adopting companies] will become “excellent development organizations.” [...] Change experts like Harvard Professor John Kotter regularly say 70% of major change efforts fail.

High with the fumes of Dunning-Kruger, companies initially think adoption is easy enough. However, as Scrum is introduced, it becomes clear that the whole company has to embrace the agile principles, including sales and management, for the adoption to succeed in the long run, and many companies find they get more than the bargained for and go in denial. They start to blame Scrum for their own faults, and abandon it or start some weird chimera sometimes called "ScrumBut".

Instead, the correct way forward is to change the company management style and become a pluralistic organization, but it's a big step for most CEOs and management, mostly in terms of ego and self image, as well as fear of letting go and change resilience.

Stop thinking of the management team at the top of the organization. Start thinking of the software developers, the designers, the product managers, and the front line sales people as the top of the organization. The “management team” isn’t the “decision making” team. It’s a support function. You may want to call them administration instead of management, which will keep them from getting too big for their britches.
—Joel Spolsky

There appear to be no shortcuts: until high-tech companies become pluralistic workplaces, they will not be good places to work for highly skilled, highly creative professionals such as software developers.

TL;DR:

1. top-down management generally sucks for software development because building software is a creative act which requires diffuse responsibility to be done effectively;
2. scrum (or, better, agile principles) are good tools to diffuse responsibility without creating anarchy;
3. CEOs don't want to give away control and become a support figure to the company, which makes it impossible for their companies to become pluralistic, and so they keep on being ineffective.

Agile and scrum are great tools, but the context is much larger than a methodology.

Do you think it would be cool to work in a pluralistic company with Joel as a CEO? Come work at Stack Exchange, we are hiring full-stack developers anywhere in the world!

discuss this post on Hacker News