I’ve seen more approaches to software project estimation than I can recall. Story points. T-shirt sizes. Bottom-up estimation. Top-down estimation. Parametric estimation. Three-point estimation… These approaches all have one thing in common. They’re disguises for a whole lot of guessing.
What other choice is there? Tell business stakeholders, “It’ll be ready when it’s ready”? Probably not. So we’re stuck trying to come up with reasonable timeframes. But trying to forecast how long a given project is likely to take is a form of fortunetelling. You assemble some experienced minds, gather their opinions, and hope for the best.
That hope isn’t often rewarded. A study by Oxford University shows that of 5,392 large scale tech projects surveyed, the average overrun was 92 percent. In other words, projects took nearly twice as long as planned. The problem isn’t that planners are reckless or stupid. The problem is that software building involves more complexities, variables and contingencies than any mind can possibly account for.
Mercifully, this is a problem almost tailor-made for AI.
The AI we’re talking about here isn’t the generative kind—it’s the other AI, the one that ingests rafts of data to identify patterns and make suggestions or predictions. Let’s consider, in the simplest terms, how human and machine intelligence might collaborate to create a forecast.
Humans provide two things: an approximate scope of work, and the people assigned to deliver it. Knowing what’s to be done and who will do it, it becomes the job of AI to predict the forecast time required.
To do this, an essential ingredient remains. We need a reliable record of how long past work has taken to complete—our historical actuals. Normally, this would be the end of the conversation. Few organizations have anything approaching a reliable record of how long past work took to complete. But happily, Socratic identifies this automatically. This includes having a detailed snapshot of every task at each point in its lifecycle, the work phases it passed through, and how long it spent in each.
With scope, people, and historical actuals in hand, we have our starting point. Let’s look at ways AI may be brought to bear.
An initial version of AI-driven forecasting might behave as follows.
The model considers:
If the math of this sounds too straightforward to require more than a spreadsheet, remember that each of these variables is in a constant state of change.
The number of tasks identified can and will change over the course of the project, but especially early on as scope comes into focus.
Assignees will change too—including the fact that different people may work the same task at different points. (For example, a designer passing the task to a developer, once design work is complete.)
Throughput rates are living things as well. Teams speed or slow their rates of delivery based on personnel changes, technology changes, and a host of other variables not visible to the naked eye.
Safe to say that even this “initial” approach to intelligent forecasting is light years beyond how most companies forecast today. But there are limitations. With this approach, we see:
Luckily, machine learning is up to these challenges…
For the uninitiated, Monte Carlo is a statistical technique used to model complex systems across a wide range of fields—traffic control, stock predictions, and nuclear reactors, to name a few. The idea is to use the law of large numbers to predict how a complex system might behave.
Monte Carlo runs a myriad scenarios using our best empirical data and knowledge of the "rules of physics" for the system at hand. By gathering results from enough randomized scenarios, we can get an idea not only of what is most likely to occur, but also visualize the range of possible outcomes.
What does this look like in the realm of software project forecasting? The basic steps and inputs are as follows:
(1) Set up N tasks with random (but representative) durations.
(2) Set up a pool of M people to complete the tasks.
(3) Assign the N tasks to the M people.
(4) Determine the completion dates for each task.
(5) Construct a burndown chart from these dates.
(6) Repeat steps 1-5 Q times.
Humans provide N; the machine provides the rest. With Socratic, this includes randomly-sampled task durations based on your organization’s historical actuals, which is needed for the first step. We’ve set the number of model runs (Q) to 1,000. (The more you do, the more statistical significance you gain, but it also starts to absorb some serious compute cycles. In our own analysis, a thousand runs strikes a nice balance between system responsiveness and forecast reliability.)
Simply put, this model reveals all of the possible durations if the same body of work was delivered by the same people, a thousand times. This range of outcomes is reflected by a date range rather than a single date.
With Monte Carlo simulation, we’re able to absorb complexities the previous, more deterministic algorithm can’t. These include nuances like:
Plainly, this approach represents a big leap from the previous model. As opposed to a deterministic algorithm, this operates as a sort of semi-supervised one. Here the algorithm combines our domain knowledge of how work gets done with observations of real outcomes.
Neural nets are powerful AI tools used to recognize complex patterns in “unstructured” input data. Think of MP3 files, a digitized time series of recorded sound waves. There are clearly patterns in the data—the human ear can easily differentiate between hard rock and blues, after all. But it’s very difficult to instruct a machine in what precisely makes the difference. Neural nets get around this problem by deducing thousands, if not millions, of interacting rules to arrive at a conclusion. None of those rules need to be intuitive or grounded in anything “physical” — it's simply whatever statistical relationships maximize the accuracy of the network.
A main limitation of neural nets is they need very large datasets to go on. In forecasting terms, this means we need lots and lots of completed projects we can point to, and mark as having been timely, early, or late, so the model can begin to discover the patterns behind each. While we build up this volume of training data, Monte Carlo is the way.
Granted, there’s another disadvantage to this kind of advanced forecasting: the model can’t tell you how it arrived at its conclusion. This is inherent to most state-of-the-art machine learning techniques. As a general rule of thumb, the more complex the model, the more effects it can account for. But also the less explainable its conclusions will be. Its forecasts are likely to be richer; why or how is a mystery.
But isn’t this a tradeoff most project owners would gladly accept?