Metrics Tracker

Contents:

• Overview
• Who the metrics are designed for and who can view them
• Accessing the Metrics Tracker
• Using the Metrics Tracker
• Benchmark groups and how are they calculated
• The Upskilling Engagement metric
• The Threat Response metric
• The OWASP Completion metric
• The Coding Language Skills metric
• The MITRE ATT&CK Coverage metric
• How to drill down to see data at the team level
• Changes to historic metric scores

Overview

The Metrics Tracker helps you track how your organization is performing across cyber resilience metrics, providing the ability to benchmark against other industries.

The following metrics are available:

• Upskilling Engagement
• Threat Response
• OWASP Completion
• Coding Language Skills
• MITRE ATT&CK coverage

Who the metrics are designed for and who can view them

These metrics are designed for a senior level audience who can use this information to:

• monitor performance and review progress to date across the organization.
• demonstrate impact and return on investment of Immersive usage.
• understand change over time and comparison to industry benchmarks.
• identify gaps and areas of weakness, next steps, and priority focus areas at the strategic level, drilling down into teams and individuals where needed. 

Although all managers and organization admins can view the organization-wide overarching metrics, the extent to which you can see more detailed data related to each overarching metric will depend on your role.

• Organization admins can view all breakdown data and any recommended actions.
• Team managers can only view breakdown information specific to the teams and users that they manage underneath the relevant metrics.
• Crisis simulation managers and workforce managers can only view the overarching metrics, but will not be able to view detailed data related to each metric.

Accessing the Metrics Tracker

To access the Metrics Tracker:

• From the main navigation menu, click Reports > Metrics Tracker. 

Using the Metrics Tracker

To use the Metrics Tracker:

1. From the Benchmark against drop-down list, select one of the following options:

   • All Industries
   • Financial Services
   • Technology
   • Consulting
   • Manufacturing
   • Best in Class

   See Benchmark groups and how are they calculated for more information. 

2. From the lefthand panel, click the desired report:

   • Upskilling Engagement (listed under Security Teams): Tracks platform usage (excluding cyber ranges) by cyber professionals with an Immersive Labs license. See The Upskilling Engagement metric for more information.
   • Threat Response (listed under Security Teams): Tracks engagement with cyber threat intelligence labs published in the last six months. See The Threat Response metric for more information.
   • OWASP Completion (listed under Developers): Tracks engagement with the OWASP Top 10 lab collection by each software developer. See The OWASP Completion metric for more information.
   • Coding Language Skills (listed under Developers): Tracks developers' progress in their primary programming languages. See The Coding Language Skills metric for more information.
   • MITRE ATT&CK Coverage (listed under Security Teams): Tracks coverage of MITRE ATT&CK techniques and subtechniques by cyber professionals with an Immersive Labs or Cyber Range Exercising license. See The MITRE ATT&CK coverage metric for more information.

   Note: If you don't have any data yet, a message appears, explaining that there is currently no data available to view. 

3. Review your results:
   • For each metric, you can download a CSV file with your results.
   • You can filter metrics by month and year.

Benchmark groups and how are they calculated

For sector-specific benchmark groups, scores are based on the 75th percentile for that group to ensure that customers are able to benchmark against high performing groups for that sector and to avoid the data being skewed by organizations with low scores. The All Industries benchmark group score is also based on the 75th percentile across all organizations. The Best in Class benchmark group is based on the 99th percentile of all organizations.

The Upskilling Engagement metric

This metric focuses on engagement with lab content by Immersive Labs licensed users. The metric measures relative regularity and completion of lab content.

There are two types of designations in this report: one for completion within the last 6 months (regularity of completion) and one for completion of at least 10 labs (amount completed).

For completion within the last 6 months:

• A high engagement designation is presented if >=80% of users have completed lab content within the last 6 months.
• A moderate engagement designation is presented if >40% but <80% of users have completed lab content within the last 6 months.
• A low engagement designation is considered if <=40% of users have completed lab content within the last 6 months.

For completion of at least 10 labs:

• A high completion designation is presented if >=75% of users have completed at least 10 labs.
• A moderate completion designation is presented if >25% but <75% of users have completed at least 10 labs.
• A low completion designation is presented if <=25% of users have completed at least 10 labs.

The Threat Response metric

This metric focuses on a combination of speed and degree of completion of threat labs by users with an Immersive Labs license.

The metric combines how quickly users completed a threat lab after it was published and how many users completed it, based on all threat labs published within the last 180 days.

There are two types of designations that are called out within this report: one for average response time for completion of threat labs and one for how many labs have been completed by at least 5 users.

For average response time:

• A slow response time designation is presented if average completion is >= 30 days.
• A medium response time designation is presented if average completion is > 7days but < 30 days.
• A fast response time designation is presented if average completion is <= 7days.

For completion of labs by at least 5 users:

• A high completion designation is presented if 5 or more users have completed >= 75% of threat labs within a week of publication
• A moderate completion designation is presented if 5 or more users have completed > 25% and < 75% of threat labs within a week of publication
• A low completion designation is presented if 5 or more users have completed <= 25% of threat labs within a week of publication.

Note: The threat lab category commenced in 2020, so no data is available prior to this date. This metric also includes users who may no longer be licensed to your organization due to the focus on speed of response at a particular point in time. However, labs that have only been completed by a single user within your organization are identified within the Recommended Focus Areas section of the report (currently accessible to organization admins only due to the potential inclusion of individuals from any team).

Scores in this metric may be impacted if labs are removed from the threat lab category at a later date, although this will be minimized wherever possible.

The OWASP Completion metric

This metric focuses on users identified as software developers within the platform based on license type or self-attestation of role. Specifically, if they have self-identified as a software developer in the platform and have an Immersive Labs or Immersive Application Security license, or if they have not self-identified but have an Immersive Application Security license, they will be included.

The metric measures the percentage completion of the OWASP Top 10 lab collection by each software developer.

Note: Completion of the Demonstrate lab in this collection counts as 100% completion. If the Demonstrate lab is not completed, then this is the percentage of labs in the collection that have been completed. This metric includes developers that have not completed any labs within this collection and is an average of all software developers scores.

• A high completion designation is presented if >=75% of software developers have completed the whole collection.
• A moderate completion designation is presented if >25% but <75% of software developers have completed the whole collection.
• A low completion designation is presented if <=25% of software developers have completed the whole collection.

Note: This metric only provides scoring after November 2021, when the OWASP Top 10 collection was published on the platform. Self-attestation of role was also introduced in 2022, so you may see an uptick or change in score at this time.

Scores in this metric beyond user completion of content may be impacted by changes to the number of labs within the current OWASP Top 10 collection (for example, if a new lab is published to this collection or an existing lab is removed/archived). This will change current scores, but will not change historical scores.

The Coding Language Skills metric

This metric focuses on users identified as software developers within the platform based on license type or self-attestation of role and an inference of primary coding language based on the labs that have been completed. If they have self-identified as a software developer in the platform and have an Immersive Labs or Immersive Application Security license, or if they have not self-identified but have an Immersive Application Security license, they will be included.

The metric measures the percentage of labs that each software developer has completed in their primary language capped at 10 labs, where 10 labs is a score of 100%. Completion of 10 labs is considered applicable across the majority of language-specific content that Immersive currently provides. This metric includes developers that have done no language labs and is an average of all software developers scores.

• A high completion designation is presented when >=75% of software developers have completed some language specific content.
• A moderate completion designation is presented when >25% but <75% of software developers have completed some language specific content.
• A low completion designation is presented when <=25% of software developers have completed some language specific content.

Note: Self-attestation of role was introduced in 2022, so you may see an uptick or change in score at this time.

Scores in this metric beyond user completion of content may be impacted by changes to the language tags of published labs (for example, removal/archiving of currently tagged labs or removal of or change to language tags on currently published labs). This will be minimized wherever possible, but may change both current and historical scores.

The MITRE ATT&CK Coverage metric

A technique score is determined for each technique in the MITRE ATT&CK framework. This is made up of two factors:

• Breadth of coverage: The proportion of subtechniques covered within that technique – whether content related to each subtechnique has been completed
• Depth of coverage: The number of users that have covered each subtechnique –  how many users have completed content related to each subtechnique (capped at a total of 10 users for the highest possible score)

This metric represents the average technique score, considering only those with mapped platform content.

There are two call-outs for this metric:

The number of techniques that have every subtechnique covered by your organization:

• Good >=75% of techniques
• Moderate >25% but <75%
• Low <=25%

The number of techniques that have more than five people covering each of its subtechniques:

• Good >=70% of techniques
• Moderate >20% but <70%
• Low <=20%

Note: We made changes to our data models in 2022, so the MITRE metric only shows data from January 2022 onwards.

As with all metrics in the Metrics Tracker, it includes data from both assigned and self-directed learning.

Only content that’s mapped to the MITRE ATT&CK Framework is included in this metric.

Only users who have an Immersive Labs license and are licensed during the time period chosen are included in the metric. (For example, if a user is licensed in January and February but is no longer licensed in March, then they won’t be included in the metric value from this time, which may impact your technique coverage scores.)

How the MITRE ATT&CK metric differs from the current MITRE ATT&CK framework heatmap

The existing MITRE ATT&CK framework heatmap only includes completion of relevant lab content.

If you have a license for Cyber Range Exercising, the MITRE ATT&CK metric includes completion of both relevant lab content and relevant Immersive-created Cyber Range Exercises. If you don’t have Cyber Range Exercising, this metric only includes completion of relevant lab content.

There are also some differences in the underlying calculations used. The current heatmap illustrates your organization's relative coverage of different techniques within the framework, highlighting areas where your coverage is comparatively higher or lower.

Conversely, this new metric offers a standardized approach to coverage. It defines a target level of coverage applicable to all organizations, and then assesses how an individual organization's coverage aligns with this benchmark.

How to drill down to see data at the team level

You can drill down within the breakdown section to view both team and individual specific information for users and teams that you manage; however, you can currently only view the metric itself at the organization level.

Changes to historic metric scores

While historic metric scores may be impacted by changes to an organization's licenses or users (for example, if user data is removed from the platform following a deletion request), this is not expected to occur frequently. Metric specific changes that may occur are listed under the detail of each metric.