Primer updates and year in review

Hey all, jumping on the year review train a bit early.

Contents

• Updates to tier rewards in 2023
• Reflections on the year
• Update on tools work
• Videos in progress

Updates to tier rewards in 2023

The tiers will remain the same, but there are some updates to rewards:

• More updates for Insiders – I haven’t been systematic about giving updates to Insiders. And especially now that billing is monthly, you all deserve more. I’m committing to publishing an update for the Insider ($5) tier and higher at the beginning of each month in 2023, starting in January. These will be similar to this post, but likely shorter, since they will only cover a month.
• “Advisors” will change to “Public thanks” – The surveys don’t get a lot of engagement, so they don’t appear to be valuable to patrons, and they also don’t provide enough information to be useful. So starting in 2023, surveys will be included in public posts about video publishes in hopes of getting more feedback. The tier will retain the benefit of having your username included in the video description.

I’m also open to other ideas for tier benefits you might value.

Reflections on the year

It was an interesting year for Primer.

The Coin Flip Cheaters game

Primer published its first interactive, Coin Flip Cheaters, which you can currently find on the homepage of primerlearning.org, or on the google play store. This was mostly an open-ended experiment to get a sense for publishing interactives.

On the production side, it unsurprisingly took a lot of extra work to go from simulated coin flips that worked for a video to an interactive game. Part of it was coding. For videos, the bar for code performance is very low. If a scene takes an entire day to run and render, it’s annoying, but it ultimately matters very little. But for a game, things need to work in real time, the assets need to be compact, and you need to worry about edge cases and cheating. For those of you who tried the game right when it came out, you may have noticed that the leaderboard contained some very suspicious scores. It turned out that I neglected to make sure people couldn’t just change the score value in the browser to send whatever number they wanted. Thankfully, detailed play info was sent to the server along with the scores, so we were able to check whether scores matched what actually happened in the game.

On the game design side, it was an interesting exercise to turn a statistics task into a game. I figured a “game” that told you to calculate p-values wouldn’t actually be that fun, so I ended up structuring it around survival, inspired by the “roguelike” genre of video games. Even optimal strategies will eventually lose, and the challenge is to make the best choices you can to give yourself more time to get lucky. Playing the game optimally turned out to be much more complicated than calculating p-values, etc., since you also have to consider the cost vs information value of an additional flip. It’s also more of a Bayesian game than a Frequentist game. If you want to know more, there was intensive discussion in the coin-flip-cheaters room in the discord.

On the engagement front, some quick, context-free stats: As of my checking of the datastore, the game has been played 145,761 times, by 50,230 unique user IDs. Most of that activity was in the first month, but in the past month there were 3935 plays, so some people are still playing with it.

As for my overall reflection, I didn’t have much basis for setting expectations ahead of time, but I feel subjectively pleased with how it turned out. There was a section of highly engaged people who delved into statistics or even learned how to create bots to play the game. I didn’t intend for people to make bots, but far be it from me to shut down someone’s motivation to work on a technical project. There was also a fair number of people with a less robust stats/coding background who joined the discussion and presumably learned from the experience. I also learned a lot from the experience. If I learned, other people learned, and now there’s a baseline experience to build from, that’s worth celebrating in my book. 🥂

I have some ideas for future games. No promises here, as none are scheduled to be made for now, but just to give you an idea of what I’m excited about:

• One is a daily multiplayer game theory game. Each day, there would be a decision to make, and the results will depend in part on what other players choose that day. It seems like a really interesting way to reinforce lessons from the hawk/dove video, it could be extended to include game theory concepts and game variations covered in future videos, and it could produce an interesting data set on how real humans play these games, at least in a particular context.
• A game that lets you build and manipulate simulated economies. This one would be quite a distance down the road. For the econ series, we’re building a tool that will let us set up economies with different parameters and explore the consequences (more on that below), and it seems natural enough that it could be packaged into a user-facing product, eventually. EVENTUALLY. The coin flip game is pretty simple but was still quite a lot of work, and that gave me a dose of humility. But this is an exciting thought to squint at on the horizon.

I’ve said “we” or “us” a few times now. Who is we? Well…

Primer is now a small team, not just me

A little over a month ago, I welcomed Matias Quezada as the first full-time member of Primer aside from me. He’s a software engineer. It usually takes a few months to find your stride in a new role, but he’s already been extremely productive, taking over the majority of the tools work and allowing me to focus most of my time on writing scripts and sketching out simulations and visualizations.

There are also a few other people who have been helping out on a part-time or contract basis, and I want to give them some recognition as well.

• John Sullivan – A friend and former coworker from Khan Academy. They set up the backend for the coin flip game, and have been contributing their deep knowledge of all things coding to various other bits of Primer work. John also makes shmeppy.com, an online map tool for tabletop rpgs.
• Angelica Pascua – Angelica is the 3D artist behind the blob hats and other models used in recent videos. I love her work so much and am also very glad to be past the days of blundering through blender on my own or hunting for suitable public domain models. You can see her portfolio at angelicapascua.art.
• Mathieu Keith – Mathieu has been making music for Primer videos since “Simulating the Evolution of Aggression” in 2019. I’m beyond happy to have beautiful, unique, thematically linked music for every video. Mathieu is a freelancer, and you can reach him for business inquiries at mathieu.keith@gmail.com.

Grant funding

Early in the summer, I got a grant from the Effective Altruism (EA) Infrastructure Fund to accelerate growth. This is what allowed me to hire Matias. This is the first grant Primer has gotten, or I have gotten. I want to share a few thoughts on grants in general, and this grant in particular.

Grants are better than sponsorships, since you aren’t being paid to say something regardless of whether you think it’s true or useful. But there is still a risk that you’ll change what you do so you can please the grant makers, earning the next round of funding. Ideally, grants are made when there is already alignment of purpose and trust in the recipient’s ability to execute. I think that’s true so far, but I don’t want to sweep risks under the rug.

On the EA grant, I’m pretty aligned with Effective Altruism in general. There are some risks to the association, though. There are some ideas that are weird by most people’s standards, and it’s also tangled up in the FTX situation. But the way I see it, there are two core ideas to EA. The first is that we should take seriously the fact that we have quite a lot of material wealth in the rich world, and we can make a big difference for a lot of people at relatively little cost to ourselves. The second is that we can use analytical tools to improve the positive impact we make with our finite resources. I endorse both of those wholeheartedly. The rest is details. And some of the ideas associated with EA also happen to be intellectually interesting and would make good Primer videos. I was already motivated to make videos that are attractive to EA funders, so as of now, I don’t feel conflicted about this specific grant.

Despite the specific comfort, I do feel a general discomfort about growing too dependent one source of funds. I want to make sure Primer is able to stay independent. As part of this, I plan to apply for some other grants. YouTube ads, merchandise, and licensing all play a role as well, and it’s possible we’ll include some paid sponsorships in the future (though I’d like to avoid them if we can).

The last part is all of you. Voluntary support from small donors is the revenue source with the best possible incentives. We’re making these videos for you, so you should be who Primer’s finances care the most about. As always, thanks to all of you for supporting!

Tools work

The main difficulty in making Primer videos is that there isn’t a single tool out there that’s great for both simulations and scripted animation. Unity is great for animated simulations. After all, it’s a game engine, and a simulation is basically a game with no players. But while creating cutscenes is part of video game development, many studios use other tools for their cutscenes, so creating a timeline-based UI for orchestrating a scene hasn’t been a priority for Unity. Over the past few years, I constructed a kind of Rube Goldberg machine with a mostly code-based interface for lining simulations and other animations up with voiceover. This contraption worked, but was had to run all at once, making iteration difficult. And would sometimes fail quietly, leading to work having to be redone, and adding a lot of time to the creation process. After checking out other game engines, John and I decided to stick with Unity and build custom tools to enable a smoother workflow for making Primer videos. And now that Matias has joined, much of his work is focused on this.

Here’s a small sample animation using a timeline in Unity. It probably doesn’t seem like much, but the ability to preview code-generated animations by scrubbing, and the ability to change timing and other parameters on the fly, will save so much time in finishing videos.

Videos in progress

With people in place working on the tools, I’m finally spending the majority of my time researching, writing, and sketching visualizations.

There are two videos currently being produced. They are both in the stage where the scope, outline, and a draft script are in place, and prototype animations/sims are being constructed. After that, there will be some iteration loops where the script and animations inform each other, leading up to final polish and publish.

These are the videos:

• Understanding The Binomial Distribution – Likely the next video to be published. This is an interlude between the frequentist hypothesis test video (How To Catch A Cheater With Math) and the promised Bayes video. I decided to prioritize the binomial video largely because the script seemed easy to write, so there would be something concrete to work on with Matias right away. But I was pleased to discover that there are going to be some interesting animations leading up to understanding the binomial coefficient, also called the “choose” operation.
• Simulating a Barter Economy – I’ve been feeling overdue for an economics video for a while. This one will essentially combine the sims from the first two econ videos into one. Blobs will be able to harvest mangoes and logs each day, and at the end of the day they’ll have the opportunity to trade. It should be a solid intro to trade concepts you’d find in an econ 101 textbook. What I’m most excited about, though, is using this simulation as a stepping stone to simulating more complex economies and economic concepts. The plan for the following econ video is to attempt to create conditions in which the blobs will begin using gold as a medium of exchange (without it being hard-coded).

And there are a few more videos in the earlier writing stages, including some of the long-promised evolution videos.

Thanks for sticking with me all the way through this!

Justin