Projecting Success are honoured to have another article published in the Association for Project Management Project magazine. We continue to push the boundaries of knowledge in a quest to improve project delivery productivity.
I’ve spent the last 12 months on a quest to try and discover how government manages the experience acquired from project delivery. Terry Williams (2008) collated many of the arguments that help to illustrate the challenges with learning lessons from project delivery, summarising that ‘it is not clear that current techniques actually achieve their purpose and there were a range of issues identified that need addressing’. Has anything changed?
I have submitted a series of FOI requests in an attempt to gain access to this experience and share it beyond organisational stovepipes for the benefit of society. I appreciate that this is a costly process at a time when government can least afford it, but these costs are inconsequential when compared to the costs of not leveraging experience. The benefits of sharing these lessons is not limited to the departments who identify them, it extends across organisational boundaries, into the supply chain and across industry. If government can shave 5% off project delivery costs as a consequence of these lessons then maybe many of the cuts to social care could be reversed. It is a cause worth fighting for.
So what have I discovered so far? Although there are pockets of good practice, there are only a handful of organisations who commit tangible effort to leveraging their experience. But even then, the product of their effort is difficult to measure; the return on investment is difficult to quantify which results in initiatives losing momentum over time, particularly when compared against competing priorities. The process of managing ‘lessons learned’ remains a significant challenge for most departments.
Four departments replied regarding their lessons learned processes. For some departments the process is mandated and policed but for others it isn’t enforced. For one department it is left to the discretion of individual projects:
[XYZ] does not have any policies or processes on lessons learned. Lesson learned methodology is standard Programme and Project Management practice which programmes and projects deploy as they see fit.
For government, the ultimate lessons learned review is a public enquiry. The enquiry into the Edinburgh Tram is reported to have cost £7.2m and increasing at £100k per week. If there is a justification to “establish why the project incurred delays, cost considerably more than originally budgeted for and delivered significantly less than was projected”, then is there an obligation to conduct similar reviews on other projects? The next level down is a review by the NAO, but as I found with the SouthEast Flexible Ticketing Programme, although the highlight report is published, the results of the underpinning forensic examination of the programme tends to be limited to a small circle of people. The third level down is driven by departmental policy, which is variable. If public money is being spent and is there an obligation on the public sector to leverage this experience for public benefit? Does the same argument apply to private sector projects, particularly where shareholder capital is at risk?
Identification and access:
The over-riding principle of lessons learned is that the lessons are discoverable, in a form for others to do something with. I’d like to share 3 responses to FOI requests to illustrate the magnitude of the challenge
Department 1: “our estimation that locating, retrieving and extracting the lessons data from centrally held information on projects/programmes (over 50 in number) will take over 100 hours of work”.
Department 2: “….would require [XYZ] to go through 7 years of SharePoint records and then identify the necessary reports which contained the lessons learned information. There are also other Change and Transformational programmes that have been undertaken by other teams across [XYZ] who will have a similar paper and electronic records,”
Department 3: I asked for the lessons learned on a major IT project and its 2 successors; a project reputed to be in one of the government’s top ten project failures. It took the department nearly 4 months to provide an answer citing the need to conduct a lengthy public interest test. The organisation finally reported “[XYZ] staff have already spent approximately 124 hours on the handling of your request”. It was eventually rejected on cost grounds (the FOI Act allows rejection for responses requiring >24 hours of effort) and no information was made available; it felt like a scene out of ‘Yes Minister’. Yet this is a project that regularly featured in the national press for cost and schedule overrun.
I acknowledge that some departments seek to reduce the burden of FOI requests by highlighting the work involved, but how can public bodies leverage this information when it can take up to 4 weeks to extract it ? Wouldn’t the public rightly expect this information to be available within a matter of minutes and also shared beyond organisational silos?
Conversely, a £50m failed project had produced a 96 page report that forensically examined the lessons learned. The initial FOI request to access this report was refused, but access was granted following appeal. It is an excellent report that has value far beyond the boundaries of the project or the department. This report would otherwise have remained buried within the project or distribution constrained within a very small circle of people. How many more examples exist?
I have also written to the Infrastructure Projects Authority to gain access to assurance reports, but this was rejected outright. Whilst I empathise with their need to have a safe space to discuss project issues, should these reports be embargoed in perpetuity? They not only provide insights into how experience could be leveraged, they also provide valuable insights for the supply chain on risks to delivery, which for some could be catastrophic. I agree that it’s a delicate subject but I’m not convinced that the answer is to lock everything away. The Information Commissioner will rule on that one.
Where lessons have been recorded they vary significantly in quality. In one instance of a failed £100m project a lesson on stakeholders was documented as ‘stakeholders’. Another project described a lesson as ‘benefits clearer’. A department that had a project with a documented cost increase of £2.6bn provided a set of lessons amounting to a page of A4. Very few consider the use case and how the experience will be leveraged, which often means that it can degrade into a box ticking process. This results in anodyne statements of the obvious that are difficult to exploit; something that Terry Williams commented on in 2008. There are some notable exceptions, but not many.
Many of the lessons are introspective, viewed through the perspective of the person writing the document. The bulk of the investment is delivered through the supply chain, yet the majority of lessons are aimed at procurement officials rather than delivery organisations. I acknowledge that if suppliers share lessons then they may lose their competitive advantage, but with the appropriate controls and incentives middle ground can surely be found. If suppliers wish to engage in the delivery of public contracts then they have an obligation to ensure that they are contributing to improvements in delivery productivity, and be rewarded appropriately.
There is a significant variation in how the experience is leveraged. I haven’t yet been able to identify an organisation that is able to demonstrate the return on investment in such a capability, which ultimately influences the extent to which it endures. One of the best organisations I identified have recently reduced their central team by 75% because of cost reductions and parallel priorities. Another organisation contractually mandated their suppliers to conduct a lessons learned review at the end of a project. They ended up with cabinets and Sharepoint folders full of documentation that they struggled to leverage. The way that technical experience is leveraged often differs to P3M experience, yet processes suggest a one size fits all solution.
I appreciate that some practitioners favour a community of interest approach to sharing knowledge, believing that lessons learned databases do not work. Although I agree that this is an essential component of the solution, the answer resides in creating the evidence to invest in improvement, encourage others to avoid what is eminently avoidable and leverage good practice and previous investment.
Government invests a huge amount of money in programmes and projects. The £455bn cost of the major projects portfolio represents the tip of a very large and expensive iceberg of publically funded investments. Although I appreciate that with continual cutbacks there is a pressure to focus on delivering projects rather than the luxury of project reflection. If this is the consensus then we should stop the lessons learned processes because they aren’t delivering the value we expect from them. But public resistance to this would be significant. Ultimately, government have a duty to ensure that lessons are being captured from this investment and used to improve the productivity of future delivery. The same could be said for the private sector who have a duty to their shareholders.
I have collated over 15,000 lessons and read hundreds of research papers on the subject. I am increasingly convinced that the answer lies in the data, building on the excellent work by NASA and others. Providing tailored evidence, at the point of need, helping to nudge people towards leveraging the pool of project delivery experience. It is the data that enables senior level decisions to be taken on how to prioritise investments on improvements. The skill is in configuring the data to reflect the use case and having a large enough pool of data to provide relevant insights. Some of the tools emerging data science tools help us to tackle this challenge, but they only form part of an end-end solution. By leveraging project delivery experience we have an opportunity to transform our profession. It’s a challenge we must grasp with both hands. Can we afford to do otherwise?