Hey there 👋 I’m Lavinia, and each Sunday I write about modern ways of running L&D. If you want to learn more: Offbeat Fellowship | Offbeat Sparks | Offbeat Fest. Here are my favorite L&D people to follow, a list of L&D books you might want to check out, and a list of L&D conferences all around the world.
Most L&D professionals have heard of the Kirkpatrick model. Its official definition is “The Kirkpatrick Model gives organizations a framework/reference to bucket data together to identify the impact a program or initiative has on an overall organization.” What it’s most known for is its four levels: reaction, learning, behavior, and results.
There is A LOT of discussion about whether this model is actually useful in practice or not. I’m not here to settle it, but I do feel the need to share some of the critique this model has received over the years:
1/ The model looks at the individual in isolation. Whether people learn, change their behavior, or improve business results is not a reflection of the individual alone. If the best training is available, but nothing else changes in the organization or even in the market, training alone won’t create any miracles.
2/ The causal link chain doesn’t hold. As the model is presented, you’d expect that if people are happy with a training, this means they learned more than those who aren’t happy (reaction), and those who acquired more knowledge change their behavior more than those who acquired less knowledge (learning), and those who’ve changed their behaviors produced better results than those who haven’t (behavior). Researchers are critical of these correlations, going as far as to say they found inconsistent or weak associations between different levels.
3/ In practice, it gets used as a sequential checklist, so most organizations never even reach the levels where the causal question matters. Most L&D teams that I spoke with treat Kirkpatrick sequentially. Let’s first gather data about reaction, then about learning, and so on and so forth. Except that the effort put into the first two levels seems directly proportional to the effort put into the last two levels, which are the most important ones.
Now, you can say whatever you want about the model, but you can’t say it’s not damn popular. So if you’re well aware of its drawbacks, but you still choose to use it, I wanted to point out three ways you can examine the data you’ll correlate with the different Kirkpatrick levels.
A. As separate sets of data
This happens a lot when Kirkpatrick isn’t being used. Data is collected mostly about reaction and learning levels, while behavior and business results are usually ignored because they’re harder to come by. Early on, when an L&D team might come across the model, it still happens that each level is treated as independent diagnostic information, valuable on its own terms, without being asked to predict the next level or to prove an effect against a baseline.
So if you simply have a bunch of data on each level but haven’t made any attempt to check whether patterns exist, your next step could be bringing everything together.
B. Sequentially
This way of looking at your data points is related to critique number two that I explored earlier and answers some important questions: does a person’s or cohort’s score at one level predict their score at the next? Is reaction really the first domino in a chain that ends in results?
Let’s say you run a sales training program and you have data at every level. If you put the data together and learn that those who offered high NPS scores didn’t bring in better business results than those who scored lower, it means that although the training was perceived as useful, the follow-up results didn’t come. This exercise is interesting because it can give you proof to open up a conversation about what else needs to happen in the organization to support the downstream effect of a training (or any other learning initiative).
C. Comparison with a control group
Looking at the data sequentially means comparing participants of a training program. This final way of looking at data forces you to compare your participants with another group that wasn’t part of your training.
Of course, for this control group, you won’t have reaction data. And while you could have learning and behavior data by gathering it separately, that’s not mandatory. The most important data you need is results. This way, you can compare the results of a cohort of participants who went through your sales training with the results of a cohort who didn’t.
Even if comparing this data tells you that your participants produced better results, I wouldn’t go as far as to say that your learning initiative is what produced those results. I would make sure to ask what else is different for the two groups. The market? The clients? The management? The background and experience? Isolating the effectiveness of learning isn’t easy and I’d be cautious in reaching and sharing conclusions I haven’t studied in depth.
To be clear, this isn’t a piece recommending the Kirkpatrick model or bashing it. In all honesty, the same train of thought can be applied to other measurement processes as well.
What you have to avoid is:
1/ Only looking at level 1 (reaction) and level 2 (learning)
2/ Looking at different data in isolation
3/ Making sure you look outside what the training or the individual produces, and into the entire system
We’re currently running a research study on L&D measurement, and we’re looking forward to hearing your thoughts. 💭
Quick Sponsor Break
This week’s newsletter is sponsored by our friends at LearnUpon. 89% of L&D teams use AI, but only 12% lead with intentional strategy. Based on insights from 1,320 learning pros, LearnUpon’s latest report reveals how top teams cut through administrative noise, elevate the human touch, and step up as strategic Learning Architects. Discover the research-backed blueprint to drive real business impact.
3 L&D Resources Worth Exploring
Learning by Doing: Hyper Island’s Experiential Methodology. What would your programs look like if they started with the problem instead of the theory? Hyper Island builds all its learning around what it calls the Learning Spiral: do something, reflect on it, draw conclusions, apply what you learned, and go round again. Learners work on live client briefs rather than case studies, and theory only comes in once they need it to make sense of what happened. Worth borrowing next time you’re tempted to open a program with slides.
How to Engineer a Speak-Up Culture. Psychological safety has been having a moment for a few years now. Part of creating it in an organization involves encouraging people to speak up. But is that enough? This guide argues it’s not. If people are encouraged to speak up, but the response when they point out mistakes is punishment, they’ll learn not to do so the next time. So creating a speak-up culture is as much about encouraging people to share their thoughts as it is about encouraging leaders to respond positively to the behavior. Very useful take!
L&D as the engine behind change management. A few weeks ago, GoodHabitz asked me to offer my take on why change is managed so poorly in organizations, and whether L&D can do anything about it. I’ll leave you with a quote, and a nudge to read the entire interview: “In the absence of change management departments within organizations, L&D can jump in and play the part. Our facilitation skills and bird's-eye view can become assets in supporting leaders in creating change management plans that go beyond upskilling, as well as create spaces where everyone else can gather to figure out what those plans mean for their day-to-day jobs.”
Meet L&D peers in your city
Live events all around the world, made to facilitate networking and exchange ideas that inform, inspire, and innovate the future of learning. Wanna bring Offbeat Sparks to your city? Click here.
The AI Find: AI and the 5Di
I started September by hosting a fireside chat with Maven on supporting AI adoption as an L&D team. This session was part of a larger series called The AI-Native L&D Leader, which ran until September 10th. I finally had some time this week to look over recordings from other sessions, and I’ll share with you some of the things I’ve learned over the coming weeks.
The first session I want to highlight is The Good, The Bad and the Ugly: Ways to use AI in Learning by Nick Shackleton-Jones. The full recording is available on demand, but I wanted to share one specific thing he shared.
If you’re not familiar with 5Di, it’s a six-stage human-centred learning design process that goes through:
– Define. Define the business problem and the outcome in terms of results, not learning objectives.
– Discover. Talk to the audience to surface the concerns they care about and the tasks that actually underpin performance.
– Design. Build a solution that can be either resources (things that give immediate answers to a specific task), experiences (things that stretch capability), or a combination of both.
– Develop. Build it, incrementally rather than as a finished course.
– Deploy. Put it where people are, at the point of need.
– Iterate. Continuous feedback and adjustment based on user testing.
What Nick answered in his Maven talk was where and how AI could or shouldn’t be used in the 5Di process.
– Define
NO: Use AI to define the outcomes (because you’re not building relationships & it defaults to education)
YES: Use AI to spot patterns in the activity data
– Discover
NO: Use AI to suggest the problems & motivations people have
YES: Use AI to help with the thematic analysis & discovery report
– Design
NO: Use AI to come up with the design (no buy-in & old educational ideas)
YES: Use AI as a contributor suggesting resources and experiences
– Develop
NO: Using AI to create modules (they are generic content & nobody will do them)
YES: Use AI to help create scenarios, simulations and resources/buddies
– Deploy
NO: Replacing in-person interaction (the value is the experience not content)
YES: Use AI to personalize, adapt, and track (e.g., via a buddy/micro-challenges)
– Iterate
YES: Correlate business metrics & L&D activity
YES: Suggest areas for improvement & spot outdated content
While the AI space is still extremely confusing and constantly changing, what I’ve been seeing in the past few months is a bit more maturity and critical thinking regarding where AI should and shouldn’t be used, and I couldn’t be happier. Might things change in the future? Sure! But it doesn’t mean we should simply not spend time thinking about what part of our work can be handed to AI and what part should stay human.
Surprise of the week
Each week, Offbeat brings you a new random surprise. Sometimes it might be big － a challenge you can join to win something nice. And sometimes it might be small, like a video or a nice Instagram account. Curious? Hit open!
