Should visionaries be more pessimistic?: optimistic vs pessimistic project management
I recently finished reading How Big Things Get Done: The Surprising Factors That Determine the Fate of Every Project. One of the book's authors, Bent Flyvbjerg, has built a database detailing the outcomes of 16,000 projects. It reveals that only 8.5 percent of large-scale projects are completed on time and on budget. An even smaller 0.5 percent met cost, schedule, and benefits targets. The authors explain ways to increase successful project delivery, but the book leaves the impression that fewer ambitious projects should be undertaken. It makes you more skeptical about visionary leaders who propose ambitious projects to solve massive problems.
In many ways, I consider myself an optimist. I adopt that stance because I believe it drives companies' and humanity's progress. So, in a way, I felt this book challenged my view that individuals and society benefit from people who take on ambitious projects with optimism. It prompted me to think about my own role as a project leader.
In my career, I have led three major projects, though not mega-projects by the standards of the book (costing over a billion dollars). Nonetheless, they were substantial projects spanning years involving dozens of engineers, designers, and salespeople. Honestly reflecting, I would not describe any of those projects as unqualified successes yet. Their delivery took much longer than planned, and they have not yet met my ambitious goals.
What should I conclude from this? Should I become more pessimistic and pursue fewer projects?
There are conflicting philosophies on how optimistic or pessimistic leaders should be when proposing new projects. Broadly speaking, arguments exist for pessimistic, optimistic, or agnostic approaches. Over the years, I’ve grown my understanding of those approaches and integrated them into my own philosophy for leading large-scale projects. This article synthesizes what I have learned over the last 10 years of building products with varying degrees of success.
Arguments for being pessimistic
My day job is as a product manager at Culture Amp. In a nutshell, that mainly involves inventing new products or product features that will benefit customers and our business and then figuring out how to build them and bring them to market. The most important part of my job is determining what is worth building. To know what is worth building, you need to know approximately how much it will cost to build. To do that, product managers work with engineers to break down project proposals and estimate how long they will take to complete. For anyone who has worked as a product manager for more than 5 minutes, you quickly realize that these estimates are always dramatic underestimates. Whenever you are given a timeframe estimate, you mentally double or triple it. And then when you finally complete the project, you realize it took twice as long as what you thought was a conservative estimate. This is referred to as Hofstadter's Law.
Hofstadter's Law states that projects always take longer than you expect, even when you take into account Hofstadter's Law. The general phenomenon of being overly optimistic when planning was first investigated by the famous psychologists Daniel Kahneman and Amos Tversky. Through their research in the 1970s, they identified something we now know as the planning fallacy.
The planning fallacy is a phenomenon in which predictions about how much time will be needed to complete a task display an optimism bias. Despite knowing that past tasks have typically taken longer than planned, people still think they can complete future tasks more quickly than they actually can. Bent Flyvbjerg’s mega-project data set offers extreme evidence for the ubiquity of this effect, with only 8.5 percent of large-scale projects being completed on time. The antidote to the planning fallacy is something known as reference class forecasting.
Reference class forecasting is based on the idea that rather than trying to estimate the delivery time based on the specifics of a project, you look at how long similar projects took. So you find the most similar project to the one you are working on and then use that project's delivery time as your estimate. But then, of course, you still have to remember that you are probably still underestimating the time required (based on Hofstadter’s Law). The significant upside of this approach is that your estimates are a lot more accurate. That means you end up with slightly less egg on your face after missing project deadlines.
However, there is a critical downside to the pessimistic approach. Project estimates and deadlines act as an extremely important signal to teams. Engineers don’t build in a vacuum; they know how long a project is supposed to take, and that influences how they work. That phenomenon pushes people towards more optimistic approaches.
Arguments for being optimistic
Optimistic approaches to estimation take seriously the signaling function of estimates and deadlines. Engineers and designers make very different decisions based on how long they have to complete a project. When they are given a lot of time, they opt for ‘gold-plated’ solutions. They make sure things are done perfectly and don’t look for ways the project can be completed faster. This is known as Parkinson's Law.
Parkinson's Law states that work expands to fill the time available for completion. So if you give a team a month to complete something that should only take a week, then (psychologically speaking) the task will increase in complexity and become more daunting so as to fill that time. Without an imminent deadline people always allow work to stretch out, often resulting in inefficiency and time-wasting.
Knowing the powerful influence of a project deadline, leaders can use deadlines to positively influence how teams approach complex projects. Several years ago, I led the build and launch of a software product designed to help organizations create data-driven personal development plans. Partway through the project, we made an internal announcement that we would launch the product in 4 months. That announcement massively changed how we approached the work. As the deadline approached, the team started to make decisions about how we could cut scope to get the product out. Ultimately, we launched the product without one of the key features (the reporting dashboard). When selling the product, that forced us to demo that part of the product using a wireframe mockup of what we were planning to build.
For everyone involved, this was a far more stressful way of working. However, it was extremely beneficial. It helped us seriously start learning what we needed to prioritize for customers to buy the product. It also gave the team an extremely clear purpose—all the bullshit, busy work that comes with working in a large bureaucracy faded away. Engineers started declining non-critical meetings, and all attention focused on the critical bottlenecks. This strategic optimism is deployed by many visionaries and project leaders.
Stephen Wolfram, the founder CEO of Wolfram Research, which creates complex computing systems, has shared how he uses deadlines to influence how his organization works. He commented, “We know every project will take longer than we plan—so we could make better estimates. However, when we do that, the project takes even longer.” He believes having a certain level of false optimism at the start of a project can be beneficial.
Dee Hock, the founder and CEO of the Visa credit card, fully internalized Parkinson’s Law and planned around it. He led Visa to complete the miraculous feat of building a telecom network and computerized authorization system that served as the bedrock of modern payments in just 9 months. Dee maintained that “if you give computer people more time, they will just consume it.” So he always insisted on shorter projects with uncompromising deadlines.
Elon Musk is the most famous advocate of this approach. He is known for setting highly ambitious and aggressive timelines for his companies. Musk sets challenging deadlines to motivate his teams to work faster and more efficiently, aiming to accelerate innovation and achieve significant milestones in a shorter period than might otherwise be considered realistic. Musk's methodology yields impressive results, driving rapid development and breakthroughs in multiple industries. However, this approach has also led to a pattern of missing the initially ambitious deadlines. Musk's philosophy appears to be that setting "impossible" deadlines can lead to progress that would not have been achieved with more conservative timelines, even if the original deadlines are not met.
To the extent that this optimistic approach is effective, I think it works because of four reasons:
It keeps the finish line in sight—that maximizes motivation and focuses attention towards only critical tasks.
It unlocks new thinking—applying constraints unlocks a new part of your brain, you consider options you would have otherwise ignored.
It forces people to be decisive—it avoids what I call “research and decision-making theater,” where people complete time-consuming research to justify what is ultimately a gut reaction about what they think is best.
It forces scope cutting—for most products, 80% of the value comes from 20% of the functionality; when the deadline approaches, you cut the low-impact, nice-to-have additions.
As with everything in life, it isn’t all upside. In my experience there are six significant downsides to taking the optimistic approach:
Lower quality work - In some instances, the ‘quick fix’ compromises the core value proposition of your product. For example, if the sign-in page is broken and users can’t access your app, it doesn’t matter how long it took you to complete; customers will not get any value from it.
Slowing down future work - The accumulation of technical debt often slows down future work, so the total cost of the project is underestimated.
Team burnout - The intensity of working towards a deadline can’t be sustained forever. Eventually, people burn out and quit.
Unethical or illegal behavior - The pressure to achieve an excessively ambitious goal can induce people to take unethical actions they might not otherwise.
You have to be a bad guy - The signaling effect of a deadline is only realized when it is enforced. When you push back the deadline and make people feel okay about it, you undermine what a deadline means. Elon Musk is famous for saying some version of “Get it done, or I accept your resignation.” Most people don’t have the heart to be that ruthless.
You miss deadlines - With more aggressive deadlines, you’re more likely to miss them, disappointing colleagues and customers.
These trade-offs have led some people to develop middle-ground approaches. They take a more agnostic stance.
Arguments for being agnostic
Agnostic approaches form around the idea that you can’t know everything from the outset. So don’t kid yourself; build the project plan as you go. This general class of project management approaches are referred to as agile or lean startup methodologies.
Agile methodologies break down the product development cycle into smaller segments, allowing teams to develop, test, and release parts of the product incrementally. This ensures that the product evolves in response to user needs and market demands, thereby reducing the risk of delivering a product that does not meet customer expectations. This is an extremely beneficial approach to managing a project, but nonetheless, company leaders, customers, and sales will want to know when they can expect some slice of that value. That need pushes you towards giving rough estimates to help other people plan. You need a strategy for responding in that scenario.
Rachel Morely, the former Chief Product Officer at Culture Amp, always told me never to commit to a public timeline until the team was more than 50% of the way through the work. The idea behind this approach is that it is impossible to know from the outside what is involved in a complex project, so you can minimize the downsides of missing a deadline by simply not stating one. There was a lot of wisdom in this advice, but one of the major drawbacks was that we always mis-estimated when we were 50% of the way through the project.
Dennis Hotson, a tech lead I worked with many years ago, captured the essence of this challenge with a cheeky phrase he would always tell me. As we approached the end of a project, he would always say, “We’re 90% complete, with 90% still to go”. Whenever he said it, the entire team would knowingly laugh. We laughed because, in our hearts, we knew it was true. As you approach the end of a major project, you always uncover a million tiny things that need to be built to complete the project. As I now know, this is such a common scenario that software engineers refer to as the ninety-ninety rule.
The major drawback of this more agnostic project management style is that it doesn’t help you decide what projects to start. Without estimating how long something will take, you can’t know if it is a project worth taking on. Given this limitation for a period of time, Culture Amp adopted the Shape Up methodology.
The Shape Up methodology is an approach to product development introduced by Basecamp. It offers an alternative to traditional Agile frameworks like Scrum and Kanban, aiming to address some of the challenges and shortcomings observed in these methods. Shape Up emphasizes having a "fixed appetite" for a project but a "variable scope."
Fixed Appetite refers to the predetermined amount of time allocated to a project or a feature. Unlike traditional methods that estimate how long tasks will take and adjust deadlines accordingly, Shape Up sets a fixed time box for how much time the team is willing to spend on a project. This could be two weeks, six weeks, or any other duration, but once set, it's not extended. The appetite is fixed based on the strategic importance of the project and available resources, not on estimations of the work's complexity.
Variable Scope means that the scope of what will be delivered can vary because the delivery time frame is fixed. Instead of squeezing all planned features into the allocated time (and potentially compromising quality or overworking the team), Shape Up suggests adjusting the scope of work to fit the appetite. This means being willing to cut features, reduce complexity, or find simpler solutions to ensure the project is completed within the time box. The core idea is to deliver the most valuable and viable version of the project within the fixed time frame rather than extending time to meet an overly ambitious scope.
There were a lot of benefits to the Shape Up approach. Ultimately, at Culture Amp, it proved too challenging to shift the entire organization over to this style of work. My intuition is that it works well for small businesses but as a company grows and requires more complex coordination between engineering, marketing, and sales, it begins to break. Nonetheless, after 12 months of working within this framework, I internalized the core principles. Now, rather than passively accepting project estimates from engineers, I more firmly state our appetite for solving a given problem and then ask them to consider how they might solve it within that time frame.
Combining pessimism, optimism, and agnosticism
Over the years, I have learned to amalgamate these three approaches into a single system. Once I am committed to building a new product or feature, we use the pessimistic approach with the engineering team to get an initial estimate. We use reference class forecasting to get what we think will be an accurate estimate, adding some margin for error to it, which becomes our outward-facing estimate. Marketing teams can then start planning around the expectation of the product being ready to sell by a particular time frame.
However, we then use one of the optimistic approaches within the team. Within the engineering and design team, we commit to a series of milestones that will have us complete the project long before the communicated completion date. As best we can, we delete the marketing date from our minds and remain focused on our aggressive interim milestones. Ideally, we are working with a small group of beta customers to whom we can commit to iteratively shipping each product milestone.
To make this tenable, we use an agnostic strategy. While each milestone has a fixed appetite (i.e., we can’t move the date), the scope is variable. Teams can adjust their approach to match the time frame. The team can cut non-critical scope and modify the product's design to achieve each milestone.
This approach balances the advantages and disadvantages of each approach – it keeps teams motivated, ensures you get the product in customers' hands quickly, and allows marketing teams to plan. Since settling on this approach, every project I touch now runs smoothly, always going to plan... said no one ever. In reality, my team has moved from everything being a total shit show to things being a tolerable shit show. But at the end of the day, a tolerable shit show can take you a long way. Culture Amp has continued to ship extremely valuable products and grow at an impressive rate over the last 5 years I have been there! Strategically using pessimism, optimism, and agnosticism has helped us get there.
