How StoryAudit is validated
For a tool whose whole job is telling you what's actually inside a course, the accuracy of that reading is the entire product. Here's how it's tested, and what it won't guess.
Tested against real courses, not sample data
Every release is checked against real Articulate Storyline course files bundled with the app itself: real slide XML, real trigger logic, real media and caption files, pulled from actual Storyline courses, not synthetic data built to match a spec. A release that fails these checks doesn't go out.
931 automated checks against 3 real Storyline courses, on every release.
Current as of this release. See the changelog for what's new since.
Three real Storyline courses are bundled in the app's own assets/ folder: demo-course.story (51 slides), demo-course-v2.story (51 slides, a later save of the same course used for the before/after comparisons above), and demo-course-team-b.story (51 slides, the sample used by Batch Check's demo sweep). A fourth real course is kept aside for one additional check, described below.
When a mistake is found, the fix becomes permanent
Testing isn't done once and left alone. When a real course exposes a case StoryAudit handles incorrectly (a trigger structure it misses, a quiz type it doesn't recognize, an edge case in how Storyline encodes something), that case gets fixed and then locked in as a permanent, automatic check, so it can't quietly break again without being caught before the next release goes out.
That's not yet true of every fix that's out in the world; some are validated by hand against real courses rather than locked in as an automatic check (see the second list below). Closing that gap for what's already live, rather than only for what breaks next, is ongoing work, not a claim we're making about the whole codebase today.
Storyline version support
StoryAudit reads the .story project package format used across Storyline 2, Storyline 3, and Storyline 360. Storyline 3 and Storyline 360 share that structure directly (Articulate kept them on the same underlying build lineage until Storyline 3's final update in 2023), and Storyline 2 uses the same package format one generation earlier. The regression suite above runs primarily on Storyline 360-era courses, since that's where day-to-day development happens. Backward compatibility with Storyline 2 and Storyline 3 has been separately confirmed by running real client course files from each (not synthetic files built to match the spec) through the same parser, with correct results. If a file uses a genuinely different package structure StoryAudit can't read, it says so immediately on drop rather than guessing.
What gets checked
This is split by what's actually true of each: what's checked automatically on every release, versus what's been validated by hand against real courses but doesn't yet have a dedicated automatic check locking it in place.
Checked automatically, every release:
- Accessibility data agrees with itself: alt text and closed-caption coverage is checked against the actual resolvable asset a slide uses (not just a toggle attribute), and Health Check and Compare are checked to report the identical caption verdict for the identical file. That's the specific way this drifted in an earlier round, and the reason it's now locked down as a permanent check instead of a claim.
- Media asset identity: files are matched between course versions by their actual byte content (an MD5 check against Storyline's own declared checksums), not by filename. This is checked against every MD5-declared image, audio, or video asset that resolves through the app's own asset-matching logic (a small number of entries, mostly unused caption-track references, aren't expected to resolve and are correctly skipped).
- A course containing a "draw from bank" quiz slide is recognized correctly by both the parsing and reporting code paths, appears in the outline, and compares as unchanged against itself. This uses a fourth real course kept specifically for this check.
- Severity counts stay internally consistent: the headline's critical/advisory totals, the same numbers broken down by category, Batch's per-course counts, the run ledger, and policy-rule pass/fail all reconcile to the same numbers, including after a finding is waived.
- Comparing a course to itself, including through a save/reload cycle and simulating an older saved snapshot, never fabricates a change.
- Blank and duplicate slide/scene titles, and the configurable alt-text-length and body-font-size policy rules, are checked against real course data (both a pass and a fail case for each policy rule). Batch Check's cross-course settings-consistency comparison is checked against a real 3-course batch, including the differences-only table it renders.
- Quiz answer keys: all 10 of demo-course.story's real quiz questions (8 Multiple Choice, 2 True/False, the only two question types this particular course uses) are checked against a hand-verified answer key.
- Trigger diff and completion path: comparing demo-course.story to demo-course-v2.story is checked to correctly report the added trigger that jumps to a non-existent slide, by slide title, and the shortest-path walk on demo-course.story is checked to reach the course's completion slide in the real, once-measured number of slides.
- Course settings diff: comparing demo-course.story to demo-course-v2.story is checked to correctly report the one player setting that changed between the two files, by name and before/after value.
Validated by hand against real courses, not yet locked into an automatic check:
- Question types beyond Multiple Choice and True/False (Multiple Response, Matching Drag-and-Drop, Freeform Drag-and-Drop) aren't exercised by the bundled demo course, so their answer-key extraction is validated by hand rather than by an automatic check.
- Course structure beyond the bank-slide case and the one settings change above: slide masters and layouts, the remaining player and project-level settings, and variable-condition branching logic.
- Dead air detection (slides where audio or video ends well before the timeline does) is checked against synthetic timing cases and was verified by hand against a real course; not yet locked to a bundled test file.
What StoryAudit won't guess
Just as important as what gets checked is what StoryAudit refuses to quietly estimate:
A quiz question type StoryAudit doesn't fully support gets an explicit on-screen notice to verify it by hand in Storyline, rather than a silently wrong or missing answer key entry.
Image content search is disclosed as byte-for-byte matching only. A resized or re-compressed copy of an image won't register as the same file, and the app says so rather than implying broader visual matching it doesn't do.
Course duration is labeled an estimate, because it is one: Storyline doesn't record actual seat time, so any duration figure is a labeled approximation, never presented as measured fact.
The spelling check matches against a curated list of commonly-misspelled words plus repeated-word slips, because it runs entirely on your machine with no dictionary service. It will catch the common cases and miss the rest, and the report says so on the finding itself. A clean spelling result is not proof a course has no typos.
Style guide compliance only ever checks against a profile you define yourself (allowed fonts, sizes, colors). With no profile set up, the check reports that nothing was checked, in Health Check and in Batch Check's policy rule alike, rather than a pass. And readability is a labeled estimate built from sentence and word length (Flesch), not a judgment of whether the writing is any good.
StoryAudit is built to be honest about which answers it's actually sure of.
The one place StoryAudit writes text
Everything above is about reading a course file and reporting on it. There's one feature that works differently. On the Translation Check tab, after checking an XLIFF translation export for untranslated segments and variable risk, StoryAudit can optionally generate a first pass translation for what's left, using the translation model built into Chrome or Edge itself, on device.
It never touches a course file. The tab only accepts a standalone XLIFF translation export, the kind Storyline produces for translators, and no course needs to be loaded for the check or the draft to work.
Every segment it drafts is marked in the file itself, not just in StoryAudit's own display. XLIFF 1.2 output carries state="needs-review-translation" on each drafted line; XLIFF 2.0 output carries an equivalent <note appliesTo="target" category="mt-draft">, since 2.0 doesn't have a segment-level attribute this tool can set without rewriting a tag that was already in your file. Either way, any translation tool that opens the file afterward will show the segment as an unreviewed machine draft.
And a segment it can't translate safely, one with a variable placeholder it isn't confident preserving, for instance, is left in the source language rather than guessed at.
Questions about a specific check, or want more detail for a technical review? Email support@storyaudit.app. For data handling and infrastructure, see the security overview.
See this for yourself
Explore a real course in the live app, no signup and no upload step. Then start a free trial to run it on your own files.