Behind the Scenes: Playtesting

We test because we don’t know.

Have you ever wondered how to run a playtest? What it takes to be a good playtester, and what feedback is useful? Lead Tester Roman Penna explains the playtest process for Horizons and shares his insights from years of testing.

Note: We recommend reading this post on Patreon using a computer or tablet. Some of the images—screenshots from our actual bugsheets!—may be difficult to read in your email or on mobile.

Hello, Horizons subscribers! My name is Roman and I am one of the lead testers for Horizons—and I’m here to give you a behind-the-scenes look at playtesting! 

The main reasons we playtest are to discover any problems with the design and to make sure the article delivers on the fantasy it’s aiming to fulfill. This not only includes testing the mechanics and game design, but also making sure the core ideas are conveyed in a way that’s easily understood. 

Horizons does two rounds of testing for each article, which gives the team more time to iterate and discover problems, and also to try things that may be a bit weirder and wackier. If testing shows that a part of the design doesn’t work in the first round, then it’s possible to course-correct and test the revised design in the next round.

Setting Up for a Test

As the editor-in-chief, Hannah sends the testers an article along with some of the design brief and what, if anything, needs particular focus in testing. For example, a couple of new stat blocks may need to be tested in combat, while an article on a rules-light new environment might only need a read-through from the perspective of experienced GMs. 

The first thing I do is read the article through once. On this first readthrough I’m mostly getting a vibe for the article and not looking for anything in particular. I’ll get a sense for what level I want to playtest at, and whether anything strikes me to watch out for when I’m running the game. If I raise any feedback during this phase, it’s usually related to the maths of an adversary or whether I feel something is unclear. Like this example from ”Look Up to Cuculan!”:

In this case, the other lead tester (Anna Guimarães, aka strange) corroborated my experience, and the content was revised before the next playtest phase to clarify that Cuculan is a town within the Cloudlands of Nim.

Running a Test

After my readthrough, I’ll find people to test the content (usually 4 to 6 players). Then I spend time preparing the way I normally would for a TTRPG session, with another quick readthrough of the article. If we’re testing a new class, subclass, or spells, I’ll give the person playing that character the information a couple of days before so they can prepare. 

When creating characters, I ask the players to use content they are familiar with and constrain the magic items (if any) that are available to them. I do this so we can concentrate on the design being tested without other distractions or delays during testing. I then log the party composition for the team to look at when they’re reviewing feedback, so they can rule out any weirdness that may have arisen from a particular party comp.

I’ll let the testers know what we’re looking for and any known bugs that are already being addressed. I ask testers to only raise problems and give feedback, not solutions. We’re only testing the content in one or two scenarios, and don't have the full context of the design goals that the rest of the team does. Once they have our playtest report, the designers can choose what needs to change and in what way. Keeping this in mind is one of the most important parts of being a tester. 

During the session, I’ll usually wrap up each combat early if I feel like I have all the data I need, and continuing a combat wouldn’t reveal anything new; so we can test as much content as possible in a single session. At the end, I ask for broad feelings on the content and then go systematically through the article, letting my players speak to each section. I encourage and facilitate discussion, seeking to tease out explanations behind the players’ reactions to the content. Once the players are finished giving feedback, I ask them how they'd like to be credited in Horizons when it comes out.

Entering Bugs

Once I’ve collected all of the testers’ feedback, I take a day away from my notes before coming back to write up bugs. The Horizons team uses a shared Google spreadsheet to write and to track when bugs are actioned.

I’ll read through each section of my notes and separate them into distinct bugs that I’ll then write up. Here are some examples from “Look up to Cuculan!” in Horizons 1.

And here are some examples from “Crimson Strings and Stolen Prayers” in Horizons 2, with feedback on the stat blocks for Emrys and Irene Hawthorne. 

As you can see, my feedback in the bugsheet is a lot more coherent (and actionable) than my notes, which only serve to remind me what issues were raised. Once feedback from all the testers is in, Hannah goes into the bugsheet and reviews it, asking clarifying questions if needed, then gives the designer requests and suggestions for revisions.

In the second example with the Hawthornes, the designer (Sadie Lowry) ended up reducing both Emrys and Irene’s AC and leaning into Emrys being an Abjuration wizard, but also increased their Challenge Rating and the expected level for player characters to fight them at—basically, Sadie and Hannah felt it was worth increasing the level range of the storyline in order to let the Hawthornes live up to their full potential as epic villains.

Testers don’t only give feedback on problems, but also on what worked well or was well received by the players! Positive feedback is always good, and helpful to the designers when deciding what things to keep or change. Here's some positive feedback for "Look Up to Cuculan!":

And here's some for "Crimson Strings and Stolen Prayers":

Ready to Publish

Once my testing is complete, I usually won’t see the article again until layout—other people who haven’t read the article yet will test it in the second round with a fresh perspective. Testers also assist with layout review to make sure all the PDF bookmarks and hyperlinks are working and help catch any lingering typos. During this process, I also get to see all of the completed art, which makes layout review even more enjoyable for me. After that, I wait with batedbreath for release day, while testing the next issue.

I hope you enjoyed a brief dive into the world of playtesting! You can find me lurking around on Bluesky and the Wildmage Press Discord if you have any further questions about testing. ’Til then,

In spe et fide permanemus,

Roman Penna, Lead Tester