What If Everything Is Equally Important
The Product Risk Analysis is biased toward functional risks. Let's consider a broader picture of the solution under test.
One of my recent projects is for a large financial institution that is moving from one data center infrastructure to another. The requirement can be summarized as "Same as the previous infrastructure, just from a different place." There is no change from the user's perspective - the applications are functionally the same. The changes are in the underlying infrastructure and in repositioning the integrations. I assume the key business impacts are improved operational costs, a general cloud journey, and the need for a controlled environment.
Challenges of Product Risk Analysis
One catch with this institution is that there are external industry compliance requirements in the mix. The common testing approach would be to have a Product Risk Assessment (PRA). When the organization does that, though, everything has an extreme impact. There is no requirement that the customer would willingly prioritize less due to the regulations. And we would have to test everything extensively. We could - but time is money. The requirements and regulations are business critical, for sure, but a nuance is missing from the PRA.
This is yet another example of how software testing models fall short when we deal with non-software testing activities like compliance. I was recently at a meeting where we could not convey the testing needed using existing old books and schoolings of testing, as the technologies and ways of working have outrun the vocabulary. Whatever words you use, use them. More words tend to help [Maaret Pyhäjärvi].
The PRA is biased toward focusing on product risks—in the functional requirements of the thing being built. However, there are at least two other Ps to consider risks for Process and Project. These Ps from decades-old testing frameworks address risks in delivering the thing (Project) and risks in doing it the right way (Process). When dealing with regulated organizations, the Process is equally important as the Product. If an auditor cannot see that the right Process was followed, the company can be out of business, no matter what the product quality. Compliance is often the "license to operate," even for public organizations.
Risk Modelling Research
When you read the research from Daniel Kahneman & Amos Tversky, the classic model of "Degree of Risk = Probability x Impact" is flawed. Dr. Warren Black writes:
Another relevant piece of research to consider is "How Big Things Gets Done" (Flyvbjerg & Gardner). The data collected by Flyvbjerg & Gardner shows that only 48% are on budget, only 9% are on budget and time, and only 0.5% are on both budget time and benefits. If you estimate your project to be average, it will, at best, be on a budget but not time and not on outcomes.
What to do
If a PRA is legally required, though, we are in a pickle. While it seems like best practice, it has flaws and biases that need to be worked into the PRA. Alternatively, work using another risk modeling approach. I think risk modeling would be similar to strategizing—the art of leveraging resources toward a future state.
I would probably reframe the situation in the map below. The users need an application (understood). Applications need integrations, and while they are known, they are being changed—sometimes hand-held changes. Integrations need external compliance (somewhat internalized). Applications and compliance both require stable infrastructure.
The map is probably half wrong, but with a discussion of user visibility and the maturity of the elements, we get a visualization of the situation and nature of the change. The mapping exercise tells us that not everything is "same-same."While there is a key technical change deep down, the most changed element is the external integrations. And that is where we should prioritize testing and learning. Not only to reduce risk but to evolve the solution as a whole.
