This article explains how to get the most out of customized Workforce Exercising content by maintaining learning quality and making sure your data is as robust as possible.
A benefit of Workforce Exercising is that you can customize the content of scenarios to suit your needs, tailoring them to your specific organizational context. How you customize the content will depend on the outcomes you want to achieve.
- Education and engagement: If you're purely focused on engagement and education, some aspects – such as learning quality – will be more important than others.
- Data: If you also want to use the response data captured during Workforce Exercising to inform risk management or demonstrate the impact of interventions, you need to ensure you customize the scenarios with data in mind.
This guide focuses on two outcomes:
- Maximizing learning
- Maintaining data quality
Maximizing learning
Creating an educational experience that engages people, is relevant to them, and provides in-the-moment feedback is important to achieving high-quality learning. The language you use, the feedback you provide, and the way you use narrative and story will all influence how people experience your content and what they take from it.
Here are some things to consider when customizing scenarios.
1. The language that you use
At Immersive, we want to make it as easy as possible for everyone to engage with our content and understand the points being made. We write in a conversational style, avoiding unnecessarily technical language and jargon. We also avoid idioms, as they can be difficult for non-native speakers to understand.
Tools such as Grammarly can help you simplify the language you use and catch any typos or errors.
Key Takeaway: Has jargon, overly-technical language, and localized idioms been avoided?
2. In-the-moment learning
Most scenarios provide immediate feedback on people's responses. Although you can turn this off, we don't advise it if education is your primary goal, as it supports in-the-moment learning. You can customize feedback to reflect your organizational context and policies better.
When customizing feedback, we recommend that you clearly state the best action or choice in the circumstances and explain why, and why any particular choice made should be avoided if possible.
Remember, you're providing feedback on the chosen action, not the person.
Key Takeaway: Have you provided sufficient feedback that helps people understand what to improve and why?
3. Telling an engaging story
The role of narrative and storytelling in learning is well known. Scenarios give you a great opportunity to engage people in a story, and the ability to customize this means you can make it as relevant to their role and context as possible. Incorporating rich media assets in your story – such as images, audio files, and video files – all contribute to a more engaging experience. If you create your own images, remember to add any required alt text.
The ability to branch within the content lets you change the outcomes and parts the person encounters based on their choices. If you make everyone's journey through the story different, be aware that this can affect your data, as it will be less comparable across individuals.
Even if you don't use branching, if you customize any response options, feedback, or part content, make sure there's still a narrative flow between them and that the story doesn't become illogical or disjointed. You can often maintain narrative flow by using careful wording at the beginning of each part to link to the varied response options.
Key Takeaway:
- Does each part and options link correctly?
- Are there enough relevant assets? And are these accessible?
Maintaining data quality
Measuring behavior is a tricky thing. Within Workforce Exercising scenarios, we don't directly measure how an individual behaves. To do that, you'd have to take an objective measurement using your internal systems, such as checking whether an individual has plugged a USB device into their laptop when they shouldn't. This can be difficult to do in practice, and doesn't tell you much about why they made that choice.
So we do the next best thing: use scenario-based judgments that ask people what they are most likely to do in a particular situation. This lets them practice their responses in a safe environment, free from judgment, and that's what's needed if people are to be honest and open.
When customizing content, there are several things to consider if you want to make the data from your scenarios as robust as possible.
4. Number and type of decision points
We keep our scenarios as short as possible to maintain engagement. We also need to ensure everyone has enough decisions to provide multiple data points per person. This increases the reliability of the data you collect by reducing the impact of lucky guesses.
Our scenarios include at least five decision points. This lets them still be completed within 10 minutes and provides a minimum number of data points to reference across related areas. We recommend using at least 5 decision points to maintain the reliability of the data.
Type of decision point
We also recommend that the decisions individuals encounter in a scenario cover related risk areas, for example, all parts may relate to behaviors within the social engineering topic and subsequent reporting. This helps make sure you have reliable data for each risk area.
Some scenarios cover multiple risk areas, where each decision part relates to a completely different risk area. This is due to the nature of the scenario and is a good way to get a high-level overview across different areas as efficiently as possible – but be aware that the data from these exercises for each risk area is likely to be less reliable as a result.
Key Takeaway:
- Does each user encounter a minimum of five injects?
- Do these cover related or similar risk areas?
5. Language used
When designing questions and response options, think about whether you might unintentionally lead people to respond in a certain way. You may be using adjectives within the scenario that suggest a particular action, process, or activity is negative or positive – for example, "a suspicious email" rather than "an email."
By referring to things in a particular way, you can "prime" an individual to respond in a certain way. You can read more about priming here. When a potential phishing email is received, the question "What do you do next?" is very different from "What do you do to make sure the email is legitimate?" or "Do you report the email?" – the latter two are more likely to signal to people that the email is in some way suspicious and will influence their decision.
This can increase the likelihood that people choose a response based on what they think you want them to choose, or lead them to notice or respond differently than they normally would. Try not to give any clues in the wording of either the question or the response options.
Response level feedback
All of our narrative scenarios include response-level feedback. This allows supporting in-the-moment learning while also collecting data. Our Security Hygiene Compass is a completely data-focused tool, so it doesn't provide feedback on each response – this reduces the chance that feedback will influence future choices.
It's up to you whether to use response-level feedback, and if so, whether you'd like to customize the provided feedback. If you do, think carefully about the language used and the information provided to minimize any potential impact on remaining responses across the scenario.
Key Takeaway
- Have all adjectives, loaded words, or priming language that can give clues to users been removed?
- Is your response feedback likely to influence your data?
6. Focusing on behavior
We focus on the actions people would choose to take rather than what they think or know about a situation. Knowing the right response doesn't necessarily mean that someone would act securely when put in that situation. This may be due to competing priorities, external pressures, or poorly implemented or difficult-to-follow processes.
You want people to respond honestly to identify potential problem areas. For Workforce Exercising scenarios to work well, encourage people to be as honest as possible and use scenario reporting to foster an open culture of honesty, reflection, and learning – rather than focusing on penalizing those who haven't responded how you'd like.
To make sure we focus on behavior in our scenarios, our instructions reflect "what would you do?" – the behavior you would take – rather than "what should you do?", which is a more knowledge-focused question.
Actions and options
When writing response options, it's easy to overcomplicate things. For effective measurement, keep things simple. Make sure each response option represents a single action, rather than multiple parts. For example, "report the email to X" or "ask what a colleague thinks" – rather than "ask what a colleague thinks, then report the email to X." Otherwise, it's hard to know the first and primary choice people make and what action they prioritize.
Risk areas
Each Workforce Exercising scenario is tagged to a risk area. These reflect overarching topic areas that the decisions – and associated behaviors – in the scenario relate to. For example, identifying that an email may not be genuine is a behavior that falls within the wider social engineering area.
Extensively customizing a scenario may mean the provided risk area is no longer appropriate, and a different one should be chosen. Always check that the risk area listed still reflects the behaviors and actions covered in the scenario. There may be overlap between risk areas, but choose the one that you feel reflects it most strongly.
Key Takeaway:
- Do options ask "what would you do?" rather than "what should you do?"
- Is there only one action per option?
- Have the appropriate risk area tags been used?
7. Maximizing variance in the data
To use data effectively to understand potential risk areas, you need sufficient variance in the data you collect. We achieve this through the design of response options. For example, if everyone scored 100% across every scenario part and risk area, that wouldn't tell you much. It might suggest that you're performing well in every area – but that's unlikely to be the case. What's more likely is that the way you're measuring something is flawed. Maybe the measure is too easy? Maybe you're measuring the wrong thing?
You need variation in your data to get useful, actionable insights. This lets you identify where individuals, teams, or the organization as a whole may be stronger or weaker – and then use that information to target resources or upskill where it's needed most.
To achieve variation, we recommend using at least 4 response options that span the full range of potential rankings. Ideally, you don't want one clearly correct response option alongside three weak ones. More options increase complexity and give you room to use "distractor" choices – options that aren't harmful, but also don't do anything to help the situation.
Key Takeaway:
- Does each decision have at least four options?
-
Do the rankings cover a wide range of weightings?