The Wrong Bullet Holes

The Wrong Bullet Holes?

Abraham Wald became famous for his work on protecting fighter planes during the Second World War: you may have heard the story. ‘Planes were returning from hostile engagements riddled with bullet holes. The instinctive response was obvious: put more armour around the areas where the damage was greatest. 

But Wald saw the problem differently.  The planes being examined were the ones that had survived.  The bullet holes showed where a plane could take damage and still make it home. The areas without bullet holes were potentially the areas where a hit was fatal. 

The lesson is often used to illustrate survivorship bias. But I think there’s another lesson for change programmes, and particularly assurance and control.  We can spend far too much time looking at the wrong bullet holes. 

Programme teams are usually very good at measuring what is easy to measure.  Spend matters.  Progress matters.  Milestones matter.  Risks and issues matter. Moreover, in my career I have observed what I term ‘sector bias’: the skills that underpin career success in a sector permeate assurance and control.  For example,  

in Financial Services, where products can be calculated with the accuracy of movements of the stars, delivery to budget often predominates.  In Retail, where day-to-day agility drives the business, cutting the best supplier deal and haggling hard is given more prominence than sound planning (Retailers are, in my experience, relatively reluctant planners of change).  In engineering environments delivery to tight spec is coveted, but I have more than once opined that the micrometer precision that goes into the production of a fan blade, or the milling of a propeller shaft, cannot be carried into the delivery of complex change.  

But many of these are lagging indicators. By the time they tell you something is wrong, the underlying problem may already be well established.  I remember one programme where the Programme Director reported, with some satisfaction, that the programme was 10% under budget. It sounded like good news, but what wasn’t mentioned was that we were also around 20% behind schedule.  I’m not a massive fan of over-engineering earned-value as a measure, but this example does illuminate the need to track both columns in the ‘double-entry’ book.  The programme was spending less because it wasn’t doing as much work as it should have been doing. The number wasn’t wrong. The interpretation was. 

So where should we look when assuring delivery of a programme? 

• Watch the key milestones, particularly those that constrain everything that follows. Don’t just look at whether something has slipped; look at whether the rate of slippage is increasing. 

• Watch for scope being quietly dropped. A programme can appear to be recovering while simply delivering less. 

• Watch partner performance. Your programme may be reporting green while a critical supplier, implementation partner or dependency is quietly becoming the constraint. 

• Look carefully at the commercial model. Does it encourage the behaviours you need? Are there consequences for poor performance? Is there an incentive for genuine over-performance? 

• Build in exit points. Look at the overall programme design: not every programme should be committed to travelling all the way to the destination simply because it has started the journey. 

• And don’t ignore mood and morale in the delivery team, and the wider business. A successful programme can occasionally drain life from ‘business as usual’. People often know something is wrong before the reporting does. If experienced people become unusually quiet, optimism disappears from the room, meetings become increasingly defensive, or teams start explaining rather than solving, pay attention. 

I’d also make two other observations based on our experience. A CIO invited us in because although his organisation’s programme was reporting green, he sensed all was not well.  After a quick review our counsel was that the programme was far from green; it was red and had been red for some weeks. The crucial point here is that it is important to ensure your reporting and assurance have fidelity. 

Second, on another programme we assured our scope was to check that it had mobilised well: it turned out the programme had left the metaphorical port some weeks previously very much sailing on a wave of misplaced optimism.  Thereafter, it became a very hard programme to deliver to plan. Ensuring early assurance is robust and completed before the programme is well advanced is therefore strongly recommended. If this isn’t the case, at least ensure the organisation is cognisant of the potentially expensive risk it is going to live with. 

Conclusion
The challenge, of course, as discussed in my last article in this series, is that you can’t protect against everything. Just as you can’t armour every part of a fighter plane without making it too heavy to fly, you can’t build an assurance regime around every conceivable risk without slowing the programme down and consuming the very resources you need to deliver it. 

The answer therefore isn’t to measure more.  It’s to focus on protecting the things that matter most and which present the real risk. So, ask yourself: are you protecting the issues that typically become apparent but ultimately don’t damage your delivery significantly — or the ones that could actually bring your programme down but with which you may be less familiar? And are you doing robust pre-flight checks to control the controllable risks?  

These are two guiding principles that underpin our assurance and control service to customers when working with them to design an approach that strikes a good balance between risk appetite, cost, and our significant learned experience of why programmes succeed or fail.