The Capability By Area of our reporting suite can be found under Reports in the main navigation menu. This guide includes the following sections:
- What are Capability by Area reports?
- Capability by Area: data points
- Capability by Area: available filters
About the Capability by Area reports
Capability by Area, tailored for technical and administrative managers seeking to enhance their team's capabilities and reinforce organizational resilience. This in-platform report offers a strategic overview of skillset gaps, enabling managers to pinpoint areas for skill development, strategic training allocation, and succession planning to bolster team resilience effectively.
Addressing the critical need for managers to understand the skillsets within their teams, the Capability by Area report empowers technical leads to assess their team's abilities comprehensively, identify potential vulnerabilities, and strategize skill-focused training initiatives. With a renewed focus on showcasing key insights on team skill progression and highlighting linchpin teams and members, this feature facilitates informed decision-making in talent recruitment, succession planning, and resource allocation to fortify team capabilities.
By providing visibility into potential vulnerabilities and outlining the effort required to address skill gaps, this revamped report aligns with our mission to equip managers with the tools needed to maintain operational resilience and sustain team effectiveness. Accessible to organizational and team admins, Crisis Simulation managers, and Team Simulation managers, this feature serves as a vital resource for enhancing organizational resilience, scenario testing, and preparing for unforeseen challenges effectively. Stay tuned for the enhanced Capability by Area report, facilitating informed decision-making and empowering managers to navigate skill gaps and bolster team readiness within your organization.
Capability by Area: data points
The Capability by Area section includes the following Dashboards:
|
Note: Dashboard items available will be dependent on your license. E.g., The Application Security Dashboard will only be relevant if you have access to our Application Security content. |
Who is included in calculations?
Security Teams Dashboard
This includes users that have access to the following lab content: cyber fundamentals and offensive and defensive cyber. These labs cover a breadth of cybersecurity content, from introductory content on cybersecurity fundamentals, to advanced defensive content for threat hunters, security analysts, and incident responders, as well as offensive content for ethical hackers and penetration testers.
Application Security Dashboard
Calculations include users that have access to our application security content (AppSec), which enables users to explore vulnerabilities in real code.
Cloud Security Dashboard
This includes users that have access to our cloud security content, which equips individuals with the knowledge, skills and judgment to secure their organizations in the cloud.
Crisis Responders Dashboard
This dashboard includes any user that has participated in a Crisis Sim exercise.
You can find a breakdown of what's included in each section below, as well as how you can turn the data into actionable insights for your organization.
|
Note: You’ll only be able to see data and dashboards relating to your permissions on our platform. As an Organization Manager, you’ll have access to data relating to your entire
|
Security Teams
This dashboard highlights how your Security Teams are performing in terms of their cyber capabilities by showcasing capability gaps, coverage across content, and resource dependencies. The data points available are based on individual lab and collection engagement, as well as Team Sim engagement.
What data points does it include and what do they mean?
- Capability gaps
The scatter plot shows breadth and depth of lab coverage. You can see the percentage of lab coverage (x-axis) against the average number of learners completing labs (y-axis).
The plot is interactive; hovering over the points will indicate the content area, as well as coverage and completion percentages. Selecting a point will open up a pop-up window with the information for you to export to a CSV file.
We define coverage as when a lab is completed by a user.
You should therefore aim to have all points in the top quadrant to ensure good coverage of capabilities across the entire organization.
Having points on the bottom right quadrant indicates a high coverage of topic areas (i.e., all lab categories have been completed by at least one user); however, it also shows that the number of individuals having that knowledge is low and so there could be single points of failure and higher exposure to risk.
- Coverage breakdown
The table complements the above scatter plot by providing you with coverage details by content category. Use the dropdown to filter by a specific category (Defensive or Offensive Cyber) or by the MITRE ATT&CK Framework, which will then populate the table with subcategories of that category (e.g., Reverse Engineering for Offensive, Defensive Fundamentals for Defensive Cyber), or by MITRE framework tactics.
- Lab recommendations
Recommendations for your organization based on user activity on our platform. You can use this to add labs to your custom collections and enhance their content.
Our recommendations tool looks at lab content that users in your organization that have already completed and recommends adjacent content. It also looks at content that not enough users in your organization have engaged with.
- Cyber capability score
This includes the number of Cyber Threat Intelligence labs not completed, as well as a breakdown of how your score is calculated.
Our Cyber Threat Intelligence lab category includes labs that focus on new vulnerabilities and emerging cyber threats and so this metric is important as it demonstrates that your team's capabilities are up-to-date.
- Critical resource dependencies
A list of individuals that have uniquely completed the highest number of labs in the offensive, defensive, and cyber threat intelligence content categories.
- Assigned lab abandonments
A list of labs that have been abandoned, including completion data and average time taken and accuracy. This equips managers with information on where there may be skill gaps that need to be addressed. Only labs that have been assigned to the learner are included in calculations here.
A lab abandonment is an attempt that was started more than seven days ago but was not completed.
Labs included in the calculations are those that started between the selected date to 90 days before the selected date.
Application Security and Cloud Security Dashboards
These dashboards highlight how the specific business functions are performing in terms of their cyber capabilities by providing quality scores of content coverage, recommendations of content to complete, insight into resource reliance and risk, and mapping against the MITRE ATT&CK framework. The data points available on these dashboards are based on individual lab and collection engagement.
What data points do they include and what do they mean?
We'll specify in brackets next to the data point if this is only applicable on one of the dashboards. If we haven't specified this, the data point is available on both:
- OWASP coverage (Application Security Dashboard)
The percentage of users that have earned the OWASP Top Ten badge.
- MITRE ATT&CK (Cloud Security Dashboard)
Percentage coverage of content against the Framework. Labs relevant to cloud security that have been completed at least once are included here.
- Threat Intelligence Labs: completions (Application Security Dashboard)
This includes the average number of users completing CTI labs relevant to application security, as well as the actual number of completions. Calculations include labs completed from 90 days before the selected date.
- Coverage by App Sec language (Application Security Dashboard)
Percentage coverage by programming languages, allowing managers to make strategic upskilling decisions.
- Coverage by subcategory
Coverage of labs relevant to each area by their content subcategory. Eg., for App Sec this includes secure engineering, secure testing, secure fundamentals, secure operations, secure tooling, and secure coding.
- Lab abandonments
A list of labs that have been abandoned, including completion data, as well as average time taken and accuracy, equipping you with information on where there may be skill gaps that need to be addressed.
Only labs that have been assigned to the learner are included here. An abandonment is classified as a lab attempt that was started more than seven days ago but not completed.
- Lab recommendations
Recommendations for your organization based on user activity on our platform. You can use this to add labs to your custom collections and enhance their content.
Our recommendations tool looks at lab content users in your organization have already completed and recommends adjacent content. It also looks at content that not enough users in your organization have engaged with.
- Capability gaps
This includes critical resource dependencies (people) and areas with resource gaps (labs).
These insights can be used to identify areas of the business where individuals need to be upskilled to minimize risk.
Crisis Responders Dashboard
This area captures your organization's capabilities based on how your employees have exercised their skills through our Cyber Crisis Simulator (Crisis Sim product); how have your Executive teams responded to real world crisis scenarios and how does this relate to their capabilities and your organization's management of risk?
What data points does it include and what do they mean?
- Exercises activity
This table lists Crisis Sim exercises assigned and includes number of participants assigned, number of completions/engagement (for presentation exercises), when the exercise was last assigned, as well as average confidence and decision scores. This includes exercises that are currently running (haven't reached their deadline or haven't reached 100% completion), as well as exercises that are closed. Managers are able to select data points to drill into the specific data, as well as download the table to CSV.
- Low capability themes
A list of high-risk themes across crisis scenarios. This is based on the lowest average decision score.
- Low score injects
Equips you with a list of decision points your team struggled with.
- Lower risk teams
A list of teams with the highest average overall scores.
- Higher risk teams
A list of teams with the lowest average overall scores. You can use this list as users to consider upskilling and then further exercising.
- Average score v. average confidence
Capability by user:
A visual representation of a user’s average score across exercises, mapped against their average confidence in their decision, allowing you to view at a glance individuals that require support (particularly those that have high confidence levels against a low score).
Capability by inject:
Injects are decision points within Crisis Sim exercises. The graph provides a visual representation of the average score across exercises, mapped against the average confidence of particular injects, showing you the decision-points users have struggled with and where upskilling is required.
How are these calculated?
Decision score is based on the option rankings measure and averaged:
● Weak 25%
● Okay 50%
● Good 75%
● Great 100%
Confidence score is based on the confidence measure and averaged:
● Not at all confident 0%
● Not very confident 25%
● Somewhat confident 50%
● Confident 75%
● Very confident 100%
Capability by Area: available filters
We have included filters on the left hand-side to help you manipulate the data and extract the information you need.
Filters available on Security Teams, Application Security, and Cloud Security Dashboards:
- Team name: As an Organization Manager, you can use this filter to drill down into specific teams and observe trends across teams.
- License type: Use this filter if you'd like to breakdown results by product. For example, you might want to view how the users that have access to Crisis Sim have engaged with security content.
- Report as of: This date filter allows you to look back in time to evaluate improvements in your organization's or team's capabilities. The date populated by default will be today's date and the data included in the dashboards will be from the selected date (or default date) to 90 days prior.
- Has current license (Y/N): Use this filter to narrow down results to users that currently have a platform license.
- Email (Application Security and Cloud Security only): use this to search for a specific learner and see how they're performing in simulations of cyber attacks, allowing you to understand their capabilities and where they can improve. This could help fuel PDRs and upskilling individuals.
- AppSec language (Application Security Dashboard only): Use this filter to select the languages relevant to your organization or team.
- Cloud technology (Cloud Security Dashboard only): Use this filter to select the cloud technologies relevant to your organization or team.
Crisis Responders filters:
- Email: use this to search for a specific learner and see how they're performing in Crisis Sim exercises.
- Team name: use this filter to filter by team's, as set up on our platform. Using this filter won't take exercises ran in presentation mode into account.
- Scenario name: a scenario is a story in Crisis Sim that acts as a template for Crisis Sim exercises. Use this filter to drill the data down to that particular template. There may be scenarios around specific capabilities/ content themes that you want to drill down into.
- Exercise name: exercises are instances of scenarios that have been assigned to users. Drill down into how users have performed in a particular exercise with this filter.
- Selected date range: the filter allows you to select a specific time frame of your choosing; you'll be displayed with data (crisis sim engagement and capability metrics) relevant to the dates specified.
|
Note: Our dashboards are interactive. Select what you're interested in on each report to drill into the specific data points. This will open a pop-up window with the particular data set. You'll then be able to download the data set to a CSV. For example, hovering over the dots in the chart below will tell you which learner is represented: Selecting a specific dot will bring up a list of injects across exercises the learner has engaged with and their decision and confidence scores for each.
|