When working in a SOC (Security Operations Center) it is often required to perform metrics or KPI’s on the detection capabilities for reporting to upper management or customers , and the flaw that some are doing, is the attempt to do “Time to detect” and “Time to respond” as these are known from the IT Incident Management, where it is often referred to as MTTD (Mean time to detect). The metrics/KPI’s is often to be to ensure improvements and uphold SLA’s
These metrics/KPI’s has the flaw when it comes to Security Incident Management from a SOC’s perspective that the time to detect, is only a number that does not in any way identify where in the attack chain the adversary was found, and the time is in many cases just a number as these are easy and understandable from C-level perspective, but what if this could be improved ?
In this blog we will try to describe another way to get valuable KPI/Metric that can be used both inside the SOC and, used as a reporting state to internal as well as external parties.
MITRE's ATT&CK Framework (https://attack.mitre.org/) has become a standard that is adopted by both security vendors as well as for SOC's. It is a knowledge base of adversary tactics and techniques based on real-world observations, and constantly evolving.
Using MITRE ATT&CK in a SOC should be something everyone is doing as it is an expandable framework as well as a good communication way to hand over information from the SOC to IT-operation on how to mitigate or detect the techniques.
Description of measurable vectors
Use the Tactics element of the MITRE ATT&CK as a variable of no later than this stage, when an adversary should be detected.
Use the same tactics element as a performance indicator on where you as a SOC want to be on detection adversaries no later than at a tactics level. This is goal or SLA, on how the SOC wants to improve.
In your risk management or in the SOC there is often a list on what adversary types you see as a threat that your building detection capabilities towards.
Time to detect and respond
MITRE ATT&CK Enterprise Tactics
The MITRE ATT&CK Enterprise Tactics is divided into 12 stages that an adversary often have to go through, these are described from left to right where the “first” stage on the left is Initial Access(TA0001) and the final” stage on the right is the Impact(TA0040)
The full list of Enterprise Tactics can be found at MITRE’s website
MITRE’s ATT&CK Enterprise Tactics is often being described as being part of the last five stages in the Lockheed Martin’s Cyber Kill Chain, where the first two steps in the Cyber Kill Chain is covered in the MITRE PRE-ATT&CK Tactics
One of the good things about using the MITRE ATT&CK in metrics is that using the Tactics is technology agnostic, meaning that MITRE have currently mapping across the following areas
Adversaries
The Adversary element is trying to identify what we should be looking for, this is not at the level where we define activity groups, but more the generic overall terms.
Examples of Adversaries that can be used, this list can be modified both expanding and decreasing over time, meaning it is a dynamical vector.:
Nation State Sponsored or Crime Groups
For this to be possible, the SOC would need knowledge on what threat groups is active towards who you are protecting, with this, a catalog of Tactics, Techniques and Procedures (TTPs) can be created and maintained.
Commodity attacks
Commodity attacks is defined by attackers that are not going for the organisation specific but as a wide-spread attack such as phishing, CEO Fraud, BEC. This category can go after individual private people or companies.
Insider
An insider can be in the category such as disgruntled employees, ex-employees that prior to leaving steals information
Strategic partners
Organisations often out-source larger parts of their infrastructure or critical assets, allowing them access into the organisations infrastructure and or high privileges. Based on this it is often required to monitor these strategic partners to protect against the trust that have been given them.
Time
The MTTD(Mean Time To Detect) and MTTR(Mean Time To Respond) should be a combination of your tools and the team.
MTTD is often when the tools (e.g. SIEM) on average, detects an event that is potential malicious and MTTR is from when the alert has occurred when is it being evaluated by a person. When doing such a sort of metric you also have to include time, and time is of course a vector but remember that this can differentiate as it depends on when the log event turn up in the system.
As an example, systems that are logging in real-time to the SIEM has a faster time based detect ratio, then where log files are on a scheduled time being pulled into the SIEM. So, the measurements has to include from when the logs becomes “visible” for the SIEM.
Based on this the metric being developed and reported on has to take this vector into consideration when setting the MTTD.
The MTTR is often by none SOC people mis-interpenetrated as the respond stage is equal to remediation, these two things is not directly related, as the analysis part and decision part can take time, and especially with more targeted attacks it is often important to ensure that as much knowledge on the incident is uncovered before remediation. With this in mind you will get the best consistent measurements.
Building the Metrics
In order for a metric to be valuable it need to be easy to understand, easy to record, consistent and provide value to both the SOC and to the “outside” receiving party. We have often observed metric that is for C-level only and the numbers will not provide value other than a number that is often inconsistent and thereby providing values that will give false safety.
Another important thing a metric need to include from experience is that it has to be able to be tied with a story that is easily to understand for none technical people.
Blank - Sample reporting metric
Description of the metrics moving parts
Often it is seen in metrics that an incident is an incident, but the fact is that depending on the Adversary defined there is a big difference in the incident types between a Nation state sponsored and an insider adversary. The detection methods used and at what stages these are best detected varies.
Detect no later
The “Detect no later” marked with orange is the state where the SOC has to have detected the adversary at this tactics.
This means that for the “Nation State Sponsored” -adversary has to be detected at any of these tactics
Initial Access
Execution
Persistence
Privilege Escalation
Defense Evasion
Credential Access
Discovery
Lateral Movement
Future - Detect no later
The “Future - Detect no later” marked with green. This is the future wanted state, so the goal that the SOC is working towards in order to get optimized visibility and detection capabilities.
Using the same adversary as previous, this would mean that the SOC should able to detect at any of these tactics
Initial Access
Execution
Persistence
Adding the numbers to the metric
During the incident there is not a lot of time spending on ensuring that these metrics are filled out, but the tactics seen as the initial identification of the incident should be recorded/tagged, so when there are more quite time of the incident the data collection can be transferred into the metric, this can either be a manual or automated task.
Now the though behind this is that any incident started should make an addition to the “Tactics” field where the initial alert triggered, that resulted in the incident started.
And in the “Adversary” will often start as unknown, and during the incident analysis the number should be moved into one of the other specific adversaries if it get known.
Example on how this could be recorded using TheHive (http://thehive-project.org/)
With a incident framework like TheHive the data can be extracted through API and using in the template mentioned above.
An example of a monthly reporting based on this could look like this.
This can of course not go alone, and especially numbers hitting further to the right of the expected “Detect no later” will need explanation and how to improve this in the future.
And this is where we are tying the metrics together with the full scope of the MITRE ATT&CK framework, as the analyst can go through the incident and identify blind spots that resulted in going above the “SLA”.
Hope that you can use this and modify or evolve, this blog post should primarily be seen as a inspiration on building KPI’s, SLA’s and Metrics that are easy for a SOC to use and one that make sense as it needs to be consistent.