"Analysis of competing hypotheses" to the rescue in incident response cases
Background on Analysis of Competing Hypotheses (ACH)
Analysis of competing hypotheses also known under the name ACH is a methodology used amongst others to avoid cognitive bias, or as a minimum be aware of these and not fall into the “normal” pitfalls. A cognitive bias is a systematic error in thinking that occurs when people are processing and interpreting information in the world around them and affects the decisions and judgments that they make.
The ACH was developed and described by Richard J. Heuer Jr. an CIA intelligence analyst back in the 1970’s however the model and thought process is still really useful.
The ACH can be found in chapter 8 in “Phycology of intelligence Analysis” available on the CIA website.
"Structured analysis uses structured techniques to mitigate the adverse impact on our analysis of known cognitive limitations and pitfalls. The most distinctive characteristic is that structured techniques externalize and decompose our thinking in a manner that enables it to be reviewed and critiqued piece by piece, or step by step, by other knowledgeable analysts."
The 8 steps of Analysis of Competing Hypotheses (ACH)
Identify the possible hypotheses to be considered. Use a group of analysts with different perspectives to brainstorm the possibilities.
Make a list of significant evidence and arguments for and against each hypothesis.
Prepare a matrix with hypotheses across the top and evidence down the side. Analyze the “Diagnosticity” of the evidence and arguments— that is, identify which items are most helpful in judging the relative likelihood of the hypotheses.
Refine the matrix. Reconsider the hypotheses and delete evidence and arguments that have no diagnostic value.
Draw tentative conclusions about the relative likelihood of each hypothesis. Proceed by trying to disprove the hypotheses rather than prove them.
Analyze how sensitive your conclusion is to a few critical items of evidence. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation.
Report conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.
Identify milestones for future observation that may indicate events are taking a different course than expected.
Source: Phycology of intelligence Analysis - Step-by-Step Outline of Analysis of Competing Hypotheses
ACH in the Cyber domain
The ACH model is known especially in the Cyber domain of Threat Intelligence when trying to avoid bias of the information during the analyze phase in the CTI life-cycle.
The idea is to come to the most plausible hypothesis by mapping how likely the evidence would support the hypothesis. Instead of trying to fit evidence to a single hypothesis at a time, it is better to do the comparison all at once, as this could help in removing selection bias. This method can also help the analyst to find some hypothesis, that might not normally be considered.
The model can however also be applied with good output during an incident response or investigation engagement of a potential risk,.
Story line
As an example we use a report from “THE DFIR REPORT” BazarCall to Conti Ransomware via Trickbot and Cobalt Strike as we here know a large amount of the incident
The threat actors used BazarCall to install Trickbot in the environment which downloaded and executed a Cobalt Strike Beacon. From there the threat actor discovered the internal network before moving laterally to a domain controller for additional discovery. A couple days later, the threat actors came back and executed Conti ransomware across the domain.
The initial access were performed through a phishing mail with a weaponized excel spreadsheet attachment.
From hereon the adversary gained persistence and adding additional capabilities through the usage of Cobalt Strike
The full timeline is well described in the blog post.
What type of hypotheses could be used during an incident or investigation
These are just some examples where the ACH has helped in the past, it will focus around what hypothesis that could be created.
Question: Who is the adversary in investigation “31337 attack”
H1 - Targeted attack.
H2 - Commodity attack.
H3 - Insider.
H4 - Strategic partner.
H5 - There were no attack, but a operational issue.
Evidence
In order for the ACH to be completed it is now a matter of identifying evidence that will be counted as either positive(supporting) or negative towards the hypothesis.
In some cases a large amount of evidence has to be added to get a good confidence score between the hypothesis, in other cases as the below a small amount is needed in order to have good indication.
It is also an important note that each piece of evidence has to be mapped towards the hypothesis, there-by attempting to remove the bias from the analyst, that are seeing a larger part of the incident.
As an example evidence derived from the blog post could be:
Evidence 1: “The initial access identified was performed by using a phishing email with an attachment
Evidence 2: “The adversary needed to gain knowledge around the infrastructure they has compromised, utilizing both Living-of-the-land tools and additional tools like ADFind and Bloodhound”
Evidence 3: “The adversary utilized are using “Cobalt Strike” a commercial remote access tool”
Evidence 4: “The adversary used both procdump and NTDSUTIL to gain access to credentials both locally on systems as well on the Windows AD Server“
Evidence 5: “The adversary deployed Conti Ransomware on the infrastructure”
ACH Scoring - Evidence vs Hypothesis
The below tables is an example of a simplified attempt to perform the ACH comparing the Evidence with the Hypothesis.
The ACH scoring shows that the most plausible hypothesis would be in the following order from most (Targeted adversary) to least (Operational issue).
Targeted attack
Strategic partner
Commodity attack
Insider
Operational issue
As mentioned earlier the scoring of the different hypothesis during an incident can also be used when deciding on the countermeasures and time importance.