3 Critical Steps You May Be Missing When Measuring Security Risk in your Organization

Why is risk such an important concept in information security? Risk provides a common language that enables information security teams to communicate with partners outside of information security, such as engineering and business teams. Risk management drives cost/benefit decisions between security, engineering, and business needs. Without a mature security risk management capability, an information security team will struggle to be successful, even if they have solid capabilities in other areas. 

One requirement of building a mature security risk management capability is the ability to measure risk. Unfortunately, measuring risk is challenging. There are many perspectives on how to do it, but there are no broadly accepted methods and holistic guidance. NIST SP 800-55 Rev. 1, “Performance Measurement Guide for Information Security,” provides a starting point, but focuses more on measuring performance than risk.

I don’t claim to have a holistic solution or process either; however, my experience has taught me three critical steps that can provide a high return on investment to enhance your risk measurement capability.  

Step 1: Customize vulnerability severity scores to your organization

Security teams may focus solely on using static third-party vulnerability severity scores and forgo considering the impact of the vulnerability in their unique environment. Doing this is understandable as organizations often have many vulnerabilities, and it is easier to use an existing score. However, if teams only use third-party vulnerability severity scores, they may end up improperly prioritizing security activities and putting their organization at risk. 

For example, most security teams use CVE (Common Vulnerabilities and Exposures) records as a source of publicly known vulnerabilities. These CVEs have vulnerability severity scores generated by the Common Vulnerability Scoring System (CVSS). CVSS scores for CVEs often come with severity ratings provided by software vendors or subject matter experts based on the general nature of the vulnerability. Some security teams will only use this score to prioritize their vulnerabilities, but this approach omits the evolving nature of vulnerability severity and the impact specific to an organization’s environment.

While some CVSS users may only consider the base metrics for risk scoring vulnerabilities, the metrics in all three groups are required to provide a more comprehensive risk score for your organization. These challenges and risk scoring components are analogous in many other methodologies beyond CVSS.

Recommendations

At Salesforce, the Information Security Team uses multiple techniques to move from simply applying a severity score to more thoroughly evaluating risk, including:

• Incorporating temporal metrics into vulnerability risk scoring through threat intelligence platforms and vulnerability management tools
• Evaluating environmental characteristics of vulnerable assets such as availability requirements, level of data confidentiality, and customer-facing exposure  
• Starting simple and dynamically tuning the risk scoring approach over time as more data on risk factors become available

Step 2: Use key risk indicators to monitor your risk

Key Risk Indicators, or KRIs, can be defined as measures of risk with targets or thresholds set by the organization. Organization leaders often define the targets and thresholds to represent the level at which a risk becomes unacceptable. KRIs are commonly used from the top levels of organizational leadership to risk management security leaders to proactively monitor risk.

Sounds good so far, but how do you create a key risk indicator? 

First, the measure has to be related to risk. Risk measures identify what could occur to raise the likelihood or impact of an adverse security event. For example, risk likelihood can increase if researchers discover a new high-risk vulnerability for software in the system. Risk impact can increase if a system starts storing more sensitive data. 

Second, the measure has to have an associated threshold. Organizations typically define a risk indicator threshold based on risk appetite and tolerance. Risk appetite, as defined by the FAIR Institute is a “target level of loss exposure that the organization views as acceptable, given business objectives and resources,” and Risk Tolerance is “the degree of variance from the organization’s risk appetite that the organization is willing to tolerate.” For example, an organization may determine that it cannot tolerate any ‘critical’ severity vulnerabilities in its high-risk environments. Therefore, the threshold for the KRI “Number of ‘critical’ vulnerabilities in high-risk environments” is 0.  

Third, what makes an indicator ‘key’? A team can produce many risk indicators, but ‘key’ risk indicators are a subset of risk indicators chosen for their importance to leaders across the organization. For example, some KRIs may relate to engineering and IT and others may be related to the product and business. KRIs are usually shared and monitored beyond just the Information Security team.

Recommendations

There are many approaches to developing Key Risk Indicators. Some methods Salesforce uses include:

• Align risk indicators to organization frameworks, e.g., security controls, security maturity, top risks or threats used in presentations to organization leadership
• Start by piloting a small number of KRIs and expand based on feedback and value to the audience
• Involve multiple stakeholders in key risk indicator development, including representatives from the target audience, technical subject matter experts, and engineering and business stakeholders 

Step 3: Consider ‘known unknowns’

The third step, “Consider ‘known unknowns,’” is the most straightforward of the three steps and  may be the most important for some organizations. Most organizations have known unknowns in their environment. These include legacy applications that haven’t gone through a security review in several years, security debt that has become more risky due to evolving threats, special projects with assets outside the normal security assessment scope, or code that the security team doesn’t have enough time to review.

These known unknowns can fly under the radar for years and become implicitly accepted in an organization. Unfortunately, they can also pose the highest risks. The best way to start in the right direction is to make sure your unknowns are at least known and break out of the cycle of implicit acceptance. 

Recommendations

Salesforce uses several different approaches to manage known unknowns. Some that may be useful to other organizations include: 

• Consider creating a dedicated project or projects to identify known unknowns and ensure they are part of the regular security assessment scope
• Integrate known unknowns into your risk measurement framework and prioritize remediations along with the ‘known knowns’
• Update policies, standards, and guidance to address sources of known unknowns and prevent them in the future 

Measuring risk is not always straightforward, but Security is a marathon, not a sprint. So, what’s next? Check out the recommendations for each step (or create your own!) and identify what you can do today to better measure risk moving forward. If you find something that works, consider sharing it. We can all use the help in this marathon!