GxP Lifeline
How to Measure Risk Management Outcomes
In his book Decision Making: Risk Management, Systems Thinking and Situation Awareness, Dr. Alan McLucas introduces the concept of the Risk Management Paradox:
“The task of managing risks effectively is confounded by a classical paradox. That is, if risks are being effectively managed as a matter of routine, there will be very few surprises. Nobody becomes aware of just how effective careful risk-management actions have proven to be. Nobody slaps the manager on the back and congratulates them for a job exceedingly well done. In stark contrast, however, if risks are managed poorly, the whole world lines up to say so.”
This paradox provides two critical insights. The first, and most obvious, is that being a Risk Manager in an organization is a thankless task – one that rarely draws praise, yet they are the first to be put under scrutiny when outcomes are not as planned. The second insight is that organizations are not adept at measuring the outcomes of risk management and the value it is adding to the organization.
The task of measuring the benefits risk management brings to an organization is a challenging one. To overcome this challenge, the measurement of risk management performance needs to consider a wide range of factors.
Risk Management Measurement can be divided into three distinct categories:
- Conformance. This measures whether the organization is conforming with its own risk management policy directives.
- Maturity. This measures the maturity of the risk management program within the organization against industry best practice.
- Value Add. This measures the extent to which risk management is contributing to the achievement of the organization’s objectives and outcomes.
Conformance
Like all programs within an organization the risk management program should be subject to conformance auditing. This auditing is aimed at ensuring that the fundamental requirements detailed in the organization Risk Management Policy are being adhered to.
For some organizations, the measurement of conformance to the risk management policy is the only measurement that occurs. Deriving conclusions as to the performance of the risk management program based solely on conformance to the policy is, however, fundamentally flawed.
It is conceivable that an organization has 100% conformance against all the risk management policy requirements and yet their risk management is not contributing to the achievement of effective outcomes. This is what I refer to as “doing risk management” rather than managing risk.
Maturity Assessment
One of the first steps involved in establishing a risk management framework for any organization is to evaluate existing management processes and systems. The most effective means of understanding the current status of the risk management processes within an organization is through the conduct of a risk maturity assessment.
The following is the output from the assessment conducted by Paladin Risk Management Services.
The levels of maturity are shown in the matrix below:
| LEVEL 1 | LEVEL 2 | LEVEL 3 | LEVEL 4 | LEVEL 5 |
|---|---|---|---|---|
| Awareness | Understanding | Initial Application | Embedded | Mature |
| There is a general understanding within the organisation of the benifits of Risk Management to the oranisation, however, at this stage, no active measures have been taken that would sonstitute the implementation of a Risk Management Framework. | A Risk Management Framework has been designed and implementation has commenced or has been programmed to commence in the near future. There may be some Risk Management being done within the oranisation, however, this is on an ad-hoc basis and is rellant on individuals within the organisation, as opposed to leadership from senior management. | A Risk Management Framework has been implemented in all key functional areas within the oranisation; however, there are areas within the organisation that have yet to incorporate sound Risk Management practices into their processes. | A Risk Management Framework has been implemented in all key functional areas within the oranisation, however, not all of the functional areas can be regarded as 'best practice' in relation to their Risk Management but steps are being taken to continually improve. | A Risk Management Framework has been implemented in all key functional areas within the oranisation, and all of the functional areas can be regarded as 'best practice' in relation to their Risk Management. |
Organizations should strive to improve their risk maturity over time, understanding, however, that to truly embed an effective risk management framework into an organization will take some time.
Value Add
While measuring compliance and the maturity of the risk management program are critical, what is not being captured by the majority of organizations is the contribution risk management is making to the achievement of the organization’s objectives.
The irony is that metrics that are currently being measured by organizations to indicate performance can provide an insight into the contribution risk management is making.
If an organization continues to improve its risk maturity over time, then it follows that the performance against these metrics will also improve. Whilst it is by no means a linear relationship, improved risk maturity will result in improved performance.
The following series of diagrams give an indicator of what this may look like in successive maturity assessments (noting the improvement in the KPIs):
| Performance Measure | Performance |
|---|---|
| No Safety Incidents (annual) | 20 |
| Staff Turnover | 27% |
| Customer Satisfaction | 73% |
| Profit after Tax | 4.50% |
| No of reportable Compliance Incidents | 8 |
| Fines for compliance breaches | $850k |
| Average time to fill vacancies | 10 weeks |
| Performance Measure | Performance |
|---|---|
| No Safety Incidents (annual) | 12 |
| Staff Turnover | 19% |
| Customer Satisfaction | 84% |
| Profit after Tax | 6.50% |
| No of reportable Compliance Incidents | 4 |
| Fines for compliance breaches | $250k |
| Average time to fill vacancies | 6 Weeks |
| Performance Measure | Performance |
|---|---|
| No Safety Incidents (annual) | 6 |
| Staff Turnover | 14% |
| Customer Satisfaction | 92% |
| Profit after Tax | 9.77% |
| No of reportable Compliance Incidents | 1 |
| Fines for compliance breaches | $50k |
| Average time to fill vacancies | 4 Weeks |
What these diagrams demonstrate in practical terms is that every time the organization benchmarks its risk maturity, it also needs to benchmark its performance measures.
It needs to be recognized, however, that this is not an exact science, and as such a direct relationship cannot be proven, but it does provide an excellent indication of a correlation between improved risk management and improved performance.
When it comes to measuring the outcomes of risk management there is no exact science; a correlation is the best you can achieve.
.jpg?sfvrsn=944d6fcc_10)
Rod Farrar is the Director of Paladin Risk Management Services, an Australian-based risk management business that provides risk management training and consultancy services to government and industry. Paladin’s flagship courses, the Diploma of Risk Management and Business Continuity and the Advanced Diploma of Governance, Risk and Compliance, have been attended by over 300 participants from all locations across Australia as well as Indonesia, New Zealand, PNG and Solomon Islands. Contact him at rod@paladinrisk.com.au.
