How to Pilot an Online Course: eLearning Testing Before Launch

To test an online course before launch, run it past four kinds of check, in order. First a functional check: every link, video, quiz, certificate and email, on the devices your learners use. Then an expert review of the content. Then think-aloud sessions, where three to five people from your audience work through one module while you watch. Finally a pilot cohort: a small group who take the whole course, start to finish, before anyone else.

Most guides to eLearning testing stop at the first round. They’re good on broken links and browser quirks and say almost nothing about whether the course teaches. This guide covers all four rounds, with what to look for in each, how many people you need, and how to turn what you find into fixes.

Four questions, four kinds of test

Course testing gets muddled because one word covers several jobs. Separate them by the question each one answers:

The last three are what instructional designers call formative evaluation (testing a course while you can still change it, as opposed to judging it once it’s finished). The classic sequence, from Walter Dick and Lou Carey’s model of instructional design, moves from one learner at a time to a small group to a field test, with the materials closer to finished at each stage. As one study summarising it puts it, “the one-to-one evaluation occurs in the earliest phase of product development”, while the small-group evaluation comes “when the materials are almost in the final form” (Chinien, 1990). That order is the point: the earlier you watch a learner, the cheaper the fix.

To make each round concrete, we’ll follow one example: an HR consultant building a short self-paced course that teaches new managers to run a useful one-to-one meeting with each member of their team. The final task is a plan for a real one-to-one, plus written responses to three scenarios. It’s an invented example, but typical of the courses independent creators build.

Step 1: Decide what “ready” means

Before you test anything, write down what would make you launch, and what would make you stop. Without that, every round produces a pile of opinions and no decision.

A useful set of launch criteria covers the four questions:

• It works: no broken links, videos or quizzes on the devices on your list; completion and certificates trigger correctly.
• It’s right: the subject expert has signed off every module.
• Learners can follow it: no step where most think-aloud testers got stuck.
• It teaches: a set share of pilot learners complete the final task to the standard in your learning objectives.

For the HR consultant, the last criterion might read: “At least two-thirds of pilot learners submit a one-to-one plan that meets the rubric, without help from me.” The exact share is a judgement call. What matters is that it’s written down before the results arrive.

Step 2: Run the functional check

Treat the course as software, because it is. The functional check is the round most guides cover well, so here it is as a short list rather than a long one:

• Every path through the course, as a learner would take it: enrol through the real checkout or invitation with a test account, not only the course builder’s preview.
• Every device and browser on your list. At minimum a phone, a laptop and whatever your learners use at work. Play every video on mobile data as well as wifi.
• Every quiz, answered right and wrong. Check that scoring, feedback, retake rules and pass marks do what you intended.
• Every automated message: welcome email, reminders, completion and certificate.
• The awkward behaviour: skipping ahead, going back, leaving halfway and returning a week later, finishing on a different device.
• Accessibility basics: captions on every video, transcripts, readable contrast, and the whole course usable with a keyboard alone. The current web accessibility standard, WCAG 2.2, was published as a W3C Recommendation on 5 October 2023 and added criteria such as minimum target sizes and alternatives to dragging (W3C, 2023), both of which matter for drag-and-drop activities on a phone.

Then hand the same list to someone who didn’t build the course. Builders click where they know the button is; a fresh tester finds the button isn’t where they expected.

Step 3: Get an expert review

A subject expert checks two things: that nothing is wrong, and that nothing important is missing or out of date. If you are the expert, find another one. Your own blind spots are exactly what a reviewer is for.

Give the reviewer the objectives and the final task as well as the content, and ask specific questions: Would someone who passes this assessment be able to do the job? Is anything here something a beginner doesn’t need yet? What would a practitioner do differently?

For the HR consultant, a reviewer who manages a team might point out that the scenarios all involve a cooperative employee, and that new managers most need practice with the one-to-one that goes badly. That’s a design fix, and it’s far cheaper at this stage than after recording.

If you can, also ask someone who designs courses to look at the structure: whether each objective has practice and an assessment, and whether the practice comes before the test. Our online course design checklist covers what they’d look for.

Step 4: Watch three to five learners, thinking aloud

This is the round that finds the most, and the one most creators skip. Build one complete module, including its practice and quiz, then sit with one learner at a time while they work through it.

The method is the think-aloud test from usability research: “you ask test participants to use the system while continuously thinking out loud” (Nielsen, 2012). You hear where they’re unsure, what they think an instruction means, and why they skipped a video, at the moment it happens, rather than reconstructed afterwards in a survey.

How many people? Usability research suggests far fewer than most people assume. Jakob Nielsen’s analysis, based on a model he developed with Thomas Landauer, estimated that a first round of five users finds about 85% of the usability problems in a design, and recommended spending a budget of 15 users on three rounds of five, fixing problems between each (Nielsen, 2000). Two cautions apply. That figure is about usability, finding where people get stuck, not about measuring learning. And it assumes testers are alike:

So if your course serves two quite different groups, such as first-time managers and experienced managers new to the company, test a few of each rather than five of one.

Running a session

1. Recruit from your real audience, not friends or colleagues who know the subject. For the HR consultant, that means people who have recently become managers.
2. Explain the purpose: you’re testing the course, not them, and confusion is useful information. Ask permission before recording the screen or audio, and say how you’ll use it.
3. Give a realistic starting point: “You’ve just enrolled. Work through module one as you would at home.” Then stay quiet. Nielsen’s advice is blunt, “Shut up and let the users do the talking”, because interruptions “can very easily change user behavior”.
4. Prompt only to keep them talking (“What are you thinking now?”), not to help. If someone is truly stuck, note it, help them past it, and treat it as the most important finding of the session.
5. Debrief for five minutes: what was hardest, what they’d skip, what they expected that wasn’t there.

Note every stall, misreading, skipped item and wrong quiz answer, with the time it happened. After three sessions, the same few problems usually keep appearing. Fix those, then test the revised module with two or three new people.

At Avondale University, we are in the process of updating traditional 6 credit units with new video lectures, case studies and immersive learning. The benefit of re-working existing units is that you both have a foundation of material to work from, and student feedback from the years (or in some cases decades) a course has been taught. However, we’re in the process of putting in place a feedback mechanism in the development of new courses to make sure that we’re not just trying to anticipate what course takers struggle with, but have actual feedback and insight to base our conclusions on as we work toward building better learning experiences online.

Step 5: Run a pilot cohort through the whole course

The pilot answers the question the earlier rounds can’t: can learners get from enrolment to the final task, at their own pace, without you beside them? Run it once the whole course is built, with a group of five to fifteen people from your audience and a firm end date.

Collect three kinds of evidence:

• Can they do the task? Final-task results, marked against the same rubric you’ll use after launch. This is the main result. For the HR consultant: how many plans meet the standard, and which scenarios trip people up?
• Where do they slow down or stop? Lessons where people drop out, and quiz questions most of the group gets wrong, point to specific fixes. Our guide to online course completion rates covers what to look for in the data, and formative assessment explains how low-stakes checks along the way show you where learners are before the final task does.
• What do they say? A short survey at the end with specific questions, and ideally a follow-up a few weeks later asking whether they’ve used what they learned.

Why satisfaction isn’t enough

It’s tempting to treat a happy pilot group as a pass. The research suggests caution. In a randomised crossover experiment with introductory physics students at Harvard, the same students were taught some topics through active learning and others through lectures by experienced, highly rated instructors, with identical handouts. Active instruction raised test scores by 0.46 standard deviations, but lowered students’ feeling of learning by 0.56 standard deviations (Deslauriers et al., 2019).

The authors traced part of the gap to effort: the harder work of active learning felt, to students, like learning less. That was a university classroom, not a self-paced course, but the lesson carries over. A pilot module full of practice may get cooler feedback than a smooth video, and still teach more. Read the survey alongside the final-task results, and when the two disagree, trust the results. If you want learners to accept effortful practice, tell them why it’s there.

Step 6: Sort the findings and fix in the right place

A pilot produces more findings than you can act on before launch. Sort them by what they cost learners:

Then trace each fix to where it belongs, as the ADDIE model suggests: a confusing quiz question is a development fix; testers who already knew everything point back to analysis. For quiz items that most testers miss, check the question before the lesson: our guide to writing multiple-choice questions covers the usual causes.

Retest anything you changed substantially. A rewritten scenario needs a new tester, not a note that says “fixed”.

Common mistakes

1. Testing only what’s easy to test. Links and videos are quick to check. Whether learners can do the task takes a pilot, and it’s the result that matters.
2. Recruiting the wrong testers. Colleagues and friends know too much, or are too kind. Test with people who match the learners you’ll sell to.
3. Helping during sessions. Every time you explain, you remove the finding you were looking for.
4. Piloting too late to change anything. If the whole course is recorded before anyone sees it, the pilot can only find problems you can’t afford to fix. Test the first module first.
5. No launch criteria. Without a written standard, a pilot becomes a debate about how good is good enough.

What to do next

Write your launch criteria today: one line each for “it works”, “it’s right”, “learners can follow it” and “it teaches”. Then book three think-aloud sessions on your first module with people from your audience, before you build the second.