Answers to common questions about running Crisis Simulation exercises, interpreting After Action Reports, understanding scoring terminology, and how performance calculations work.
Troubleshooting FAQs
Why can't I find my custom scenario in the catalog when scheduling an exercise?
The new scenario won't be listed when scheduling an exercise if it was not published when you created it. In this case, the scenario is in a draft state and won't appear in the scenario catalog.
To publish the scenario, select the 'Create Scenario' button.
Below the two 'Create' options, you'll find a list of your 'Draft Scenarios'.
- Search for the scenario and select 'Edit'.
- Select the 'Review' button on the top right, and then 'Publish'.
You'll now be able to schedule an exercise with this scenario.
Why are performance indicators not shown?
Performance indicators will only appear if feedback is enabled for your scenario. To see these indicators, check your scenario settings and make sure the feedback option is turned on. This allows participants to receive valuable insights into their performance during and after the exercise.
How can I find specific text within a scenario?
To locate specific text, you need to open each inject (scenario step) individually and use the search function within your browser or editor. Press Ctrl+F (or Cmd+F on Mac) to bring up the search bar, then type the text you're looking for. This method helps you quickly navigate to relevant information within each inject.
What are interludes, and how should I use them in a scenario?
Interludes are special sections within a scenario designed to stimulate discussion, encourage participants to reflect, or provide additional context and information. Unlike regular injects, interludes don't require participants to make decisions or submit votes – they're purely informational or thought-provoking. Use interludes to break up the flow, introduce new concepts, or prompt group discussion without advancing the scenario timeline.
Why can't I see or run a scenario I just created?
If you've recently created a scenario but can't find it in the scenario catalog, it's likely still in draft mode. To make it available for use, you must 'Publish' your draft scenario. Once published, it will appear in the scenario catalog, where you can select it and choose the appropriate exercise type to run your new scenario.
Where can I find the answers to justification questions after an exercise?
After completing a scenario or exercise, you can review all participant responses – including answers to justification questions – on the 'Results' page. This section covers how participants responded and the reasoning behind their decisions, which can be valuable for debriefs and lessons learned.
How can I test a scenario before running it with participants?
You can preview your scenario in 'Test' mode, which allows you to experience the scenario flow without consuming a credit. This is useful for checking content, timing, and logic. Please note that reporting features are disabled in test mode, so results and analytics won't be generated during your preview.
Why does my After Action Report (AAR) appear empty or incomplete?
An After Action Report (AAR) may appear empty or lack detail if certain scenario settings weren't enabled. For a complete AAR, make sure both Ranking and Confidence options are enabled when running your scenario. These settings allow the system to collect and display detailed participant data in the final report.
Why can't I see any scenarios in the catalog?
If you're unable to view scenarios in the catalog, it may be due to your access rights. Only users with Crisis Sim Admin or Manager permissions can see the full catalog of scenarios. If you have standard user access, you'll only see content that has been specifically assigned to you. Contact your administrator if you need broader access.
How can I see or read responses from specific participants?
To view individual participant responses, you must enable the "Show Participant Name" option when setting up your scenario. This setting prompts users to enter their names when joining the exercise, allowing you to track and review their answers. Please note that this feature is available only in "Presentation" mode.
How can I collect lessons learned or feedback from participants?
You can gather feedback and lessons learned by enabling justification questions within your scenario. Additionally, you can prompt users for feedback or next steps at the final inject. This approach encourages participants to reflect on their experience and share insights that can be used to improve future exercises.
What are some suggestions for customizing a scenario effectively?
When creating multiple injects that cover similar topics or share the same title, it's helpful to add differentiators such as "ransomware 1/2/3." This practice makes it easier to track the logic flow and understand how each inject fits into the overall scenario. Clear labeling also aids in scenario editing and review.
What should I do if I receive the error: "Participant can't join a crisis sim exercise, and the WebSocket connection status is red" in Presentation Mode?
This error typically indicates a problem with WebSocket connectivity, which is essential for real-time updates in Crisis Simulation Presentation Mode. As a quick workaround, try connecting using mobile data or a different network. For a permanent solution, work with your IT team to confirm your browsers and network meet the system requirements, especially regarding WebSockets. For more details, refer to the WebSockets Diagnostics article in this guide.
Why aren't the participant counter or votes updating in Crisis Simulation Presentation Mode?
If the participant counter or voting results aren't updating in real time, it's likely due to WebSocket connection issues. Please refer to the previous FAQ on WebSocket diagnostics for troubleshooting steps and solutions.
After Action Report FAQs
What is an AAR?
At the end of a crisis sim exercise, an After Action Report (AAR) will be available to Crisis Sim Managers. This will inform them on:
- Overall Performance
- Inject Breakdown (Metrics relating to decision points)
- Participant Breakdown
- Next Steps (through related scenarios and lab collections that can be assigned)
How is a new AAR different from previous reports (View Results)?
AAR is in addition to and complements the 'View Results' Report. AAR equips you with actionable insights and metrics on performance, whereas the 'View Results' report outlines the specific decisions made by participants and allows you to export these to CSV. Use 'View Results' to examine specific responses; use the AAR to determine next steps.
AAR also includes support for reporting by teams in single-player mode. Managers can assign exercises that include team reporting options by region, department, or enterprise when an exercise is created or edited.
Who can view AARs?
Anyone designated as Crisis Sim Manager can access and view AARs. This role is typically allocated to those responsible for crisis management in your organization (e.g., Incident Response Manager).
Are AARs available for all exercise types (e.g., Presentation, Single-Player, and Drills)?
Yes, as long as either response confidence or ranking is enabled in the scenario settings (this is the default case for all Immersive Labs scenarios). Support for team reporting is only available in single-player mode.
The report will become available 30 minutes after the first completion of the exercise.
Why are AARs valuable?
The report provides the insights crisis managers need to effectively plan next steps for teams and individuals to improve their crisis response readiness.
What are some typical use cases for reporting by teams?
- Assign team exercises that scale across the organization.
- Compare teams' performance against one another to prioritize exercising strategies.
- Assess organizational cyber readiness and risks by specific teams.
How do I view an AAR?
At the end of an exercise, Crisis Sim Managers can follow these steps:
- Navigate to 'Exercise' in the main navigation menu and select 'Immersive Crisis Simulations.'
- Search for the specific exercise in the 'Managed by Me' or 'All Exercises' area: use the search bar and/or filters on the left-hand side to bring up the exercise you want to view results for.
- Select the exercise and then the 'View After Action Report' button.
Note: The AAR button will be accessible 30 minutes after the first completion.
Are AARs available retroactively (i.e., for exercises ended before its release)?
Yes. All exercises (including exercises that ran in the past) will have AARs available. If you don't have an AAR appearing, jump to the next FAQ.
Why didn't I get an AAR for an exercise?
The report will become available 30 minutes after the first completion. Contact our Support Team if a report isn't generated, and we'll investigate.
What data points can I see in the report?
What is visible depends on what you have chosen to measure in your exercise.
All data points will be available if your exercise is set up to measure response confidence and ranking of inject options. We measure both by default in all Immersive Labs scenarios.
If you create your own custom scenarios and want to obtain After Action Reports, make sure you've enabled these options when creating your content. Response confidence can be found in the 'Analysis' area of the content creator, which is just below 'Ranking.'
If your exercise isn't measuring response confidence, the data points related to this, such as decision confidence by inject, won't be available.
Similarly, if you haven't enabled ranking inject options, the data points related to this, such as Decision Score by Inject, won't be in your report.
Note: Data points relating to reporting by teams are only available in a single-player mode.
Why doesn't my AAR for an exercise include a High Performers and Low Performers section? This feature is available in AARs for other exercises I've run.
If you did your exercise in Presentation mode, the AAR won't include these two sections in the Participant Breakdown area of the report. There will only be data for one participant in this area because, in Presentation exercises, only one participant selects answers: the facilitator.
Can I export/download an AAR?
Future releases will allow you to export the report to PDF format.
Terminology FAQs
What is a good overall score?
This is located in the Overall Performance section.
Based on our experience with IL clients today:
>= 75% – Excellent
>= 50% – Good
>= 25% – Fair
>= 0% – Needs improvement
The scoring guidelines apply to Immersive Labs scenarios. Consider this when interpreting the score if you've created your own scenarios or heavily customized option rankings within Immersive Labs scenarios.
What are the related scenarios and collections in 'Next Steps' based on?
These are chosen from the 'attack vector' of the exercise's scenario.
What is an inject?
We call every decision point in scenarios an inject; scenarios are made up of injects. Injects (decision points) comprise inject options that rank weak, okay, good, or great.
What is a playthrough?
A playthrough is a completion of the exercise.
Calculation FAQs
How is the Overall Score calculated?
The score on the exercise is an average of participants' decisions (selection of inject options), with an adjusted element to account for confidence levels.
For example, selecting a weak option with a high level of confidence is a weaker decision than choosing a weak option with a low confidence level. This is because a high level of confidence could lead to the participant swaying others in a crisis or not checking with team members before acting, which could translate to higher risk for the organization.
We use the Dunning-Kruger curve to adjust for overconfidence bias in the score. If the exercise doesn't measure participants' response confidence, the overall score will be the Decision Score (see next FAQ).
How is the Decision Score calculated and ranked?
It is an average of an individual's scores based on their selected inject option. Inject options are ranked:
- Great – 100%
- Good – 75%
- Okay – 50%
- Weak – 25%
For example, in an exercise with five injects, where a participant selected the Great option for three injects and the Okay option for two, the Decision Score would be calculated as: (100+100+100+50+50)/5 = 80.
How is Decision Confidence calculated?
This is found in the Participant Breakdown section. It is the average of the response confidence scores.
- Very confident – 100%
- Confident – 75%
- Somewhat confident – 50%
- Not very confident – 25%
- Not at all confident – 0%
How are the Strongest and Weakest Injects determined?
We consider the strongest injects to be the highest-scoring answers as voted by the participants. The weakest injects are the lowest-scoring answers as voted on by the participants.
How are High and Low Performers determined?
High performers are individual participants (Drill exercises) or teams (Single Player exercises) with an overall score equal to or higher than 50%.
Low performers are those individuals or teams that have scored lower than 50% in the exercise.