Tracking surprise in financial forecasts 😱

As we approach the start of a new fiscal year, I’m thinking through performance metrics and targets for my capital finance team.

A classic metric in finance is comparing forecasts to actuals: take a look at what you expected, relative to what actually happened, and keep that within some tolerance – like 10%. Nothing wrong with this. We want small variances! But, this is a lagging indicator. Whatever caused the variance happened out on a construction site many months ago and is only now showing up in the financials. We can use this to get better, perhaps, at forecasting. It doesn’t retroactively fix the problem on site.

I’ve been thinking through potential metrics to get ahead of issues, measure our internal project communications, and adherence to governance. Basically a metric for how surprised we are each month by financial changes.

First, some background, without turning this into an AACE paper. An important role for my team is the assembly of an Estimate at Completion (EAC), which is basically what we think the project will actually cost by the end of delivery. It includes incurred costs, approved change orders, commercial claims, and trends. Adding these together yields the EAC and there’s all sorts of discipline around them. In addition, the project has a risk tracker that quantifies the likelihood and financial impact of a whole host of things that could happen, but haven’t, yet. The EAC is expected to fluctuate each month as risks materialize, trends and claims are adjusted, and new scope gets approved.

So, my experimental metric is tracking the proportion of a monthly change in the EAC that can be attributed to an item in the risk tracker from the previous month. In other words, our estimate changes because a risk we were tracking materialized. This is much better from a controls perspective than unexpected things happening each month that change the EAC.

I don’t want to get too hung up on “attributable”. I’m sure there will be changes that can only be partially attributed to a specific risk, as well as changes that could be attributed to many risks simultaneously. I’m good with at least a predominance of attribution to something in the risk register. This does, though, require that we have some good version control on the register. So that we don’t just change what it says to suddenly be attributable to what happened. This will be a new step of archiving the risk register each month, since we ordinarily want it to be continuously updated.

Of course, no one expects that all changes would be anticipated. This affects the target to be set. Given we’d be new at this, something like 80% of changes to the EAC this month being attributable to a risk from last month seems like a reasonable starting point. Aiming for 100% right away could drive the wrong behaviour.

That’s my proposal. We’ll pilot it on a few contracts and adjust, as necessary. If the metric is trending well, that gives our forecasts credibility and our executives comfort that they’re well informed. If it is trending poorly, we’ll know there’s work to do, likely through more careful risk reviews.