Revamping the mnemonic medium around reader control

When you read non-fiction, you’re in the driver’s seat. You can skip to the last page and read only the conclusion. You can riffle through the pages, reading only the headings; or you can spend a week reading ten pages with extreme care. You don’t need to focus on just one text: you can compare one book’s ideas to another sitting by its side. Great non-fiction authors exert careful control over their prose, but once a book arrives in your hands, it becomes a tool in your service. You’ll use each book according to your own needs and interests.

This reader-centricity distinguishes non-fiction texts from other informational mass mediums, like videos and lectures. Those forms make it much more difficult for viewers to “drive” the experience, for instance to focus especially on the parts they find most interesting. In fact, abdication of control is part of the appeal. It’s fun to put yourself in the hands of a master explicator like 3blue1brown. I watch his videos when I don’t want to be in the driver’s seat; I want to sit back and enjoy seeing the topic through Grant’s eyes.

When the goal is enablement, though, reader centricity offers some important advantages. Authors model what their readers might already know and what they might be interested in, and then structure their texts accordingly. In most contexts, a one-size-fits-all (or even -fits-many) solution is impossible, but that’s okay: readers can collaborate with the author to mold the experience to their interests. In this way, texts enable readers in a wider range of contexts than author-centric mediums can reach. Perhaps more importantly, texts support readers in remaining relentlessly focused on their own sense of what’s meaningful.

All this said, I can now articulate a key problem for the mnemonic medium: it glues authors to the driver’s seat. Its key insight is combining spaced repetition memory prompts with narrative prose. Those author-provided memory prompts make it easy for people to remember what they read, but at the cost of sharply shifting control back to authors. Reading a mnemonic text is very unlike reading a normal text. The current interactions demand not only that you read a text in full, but that you repeatedly study—and commit to memory—whatever the author thinks is important, in whatever form the author chooses. The memory system isn’t “yours”; it’s on loan from the author, kept under glass. As Gary Wolf has pointed out to me, it’s an authoritarian medium.

Where today’s mnemonic medium succeeds and fails

Quantum Country succeeds despite these limitations because it’s a primer in a well-established technical field. Because it’s a primer, it can safely assume that most readers have little prior experience. They may be especially willing (and it may be especially appropriate) to defer to the author. Because Quantum Country is an introduction to a well-established field, there’s a set of topics it’s expected to cover. Its table of contents is partially a matter of authorial choice, but much is a reflection of general consensus. Readers who want to understand the field won’t commonly feel the need to pick and choose from these foundational concepts. And because quantum computing is a technical topic, the content of the prompts is less contingent on an author’s choice of metaphor or phrasing. It’s closer to capturing some standardized representation of physical law. So readers are more likely to be happy internalizing prompts as written, rather than wanting to rephrase them in terms which better match the way they think about the topic.

The mnemonic medium’s current design works much less well in contexts where these assumptions don’t hold. For instance, in my own mnemonic essay on How to write good prompts, many readers have substantial experience with prompt-writing, while others have never written one before. Readers’ areas of interest and levels of commitment vary widely. The essay’s topic is not well-established or standardized: I’m inventing my own abstractions, which resonate well with some readers’ experiences and poorly with others’. And because it’s a non-technical topic, the prompts are much more contingent on my authorial choices of metaphor, naming, phrasing, etc. Based on reader feedback, other authors’ similar mnemonic essays have experienced similar problems.

This isn’t just a problem with relatively informal topics, or with less-committed readers. One valuable application of the mnemonic medium is to augment important academic papers. Summer intern Ozzie Kirkby has been exploring topics in decentralized technologies, so as an experiment, I adapted the Interplanetary Filesystem (IPFS) paper into the mnemonic medium, and he logged his experience reading it. Many of my prompts worked as written, but others focused on aspects he didn’t personally care to explore. Ozzie found himself wanting to internalize much more detail in some sections, so he wrote additional prompts in his log (pretending he could add them to Orbit that way). A couple prompts felt too complex given his prior knowledge, so he split them into smaller prompts. I believe that issues like these would occur with most paper-reading experiences, since contexts and motivations vary so widely, and since papers are read much more tactically than essays.

Edit and delete buttons aren’t enough

Practically speaking, the reader feedback I get sounds like simple requests for control: can you add a button that lets me delete prompts? Can you let me edit the author’s text? I’ve been hesitant to implement these simple features because I think the mnemonic medium’s fundamental model needs re-shaping. A traditional text is a tool to be used as the reader sees fit. The reader’s in the driver’s seat. But edit and delete buttons aren’t enough to remove mnemonic medium authors from the driver’s seat. Even with those buttons, the reader would just be along for the ride, perhaps making adjustments along the way.

Part of the problem here is the medium’s positive starting assumption that the reader is expected to collect all the author’s prompts (except those they veto). This creates a school-like learning aesthetic, a sense that the author is assigning you to get X, Y, and Z out of the text. Some readers will respond to this with frustration and abandonment; others will respond by dutifully but passively studying things they don’t really care about, in practical misalignment with Orbit’s values (“helps you deepen your relationship with whatever you care about most … not for things you think you ‘should’ be engaged with … not ‘educational’ in tone”). This mismatch is really what keeps me from advancing the mnemonic medium (as it exists) only in Quantum Country-like “primer” contexts, contexts in which many readers don’t mind abdicating control: I want to promote a more active, less dutiful stance towards learning.

Another part of the problem is that an editing affordance, absent a more complete authoring experience for readers, would still strongly privilege the author’s writing. Such a medium would permit readers to adjust the author’s wording to better match their understanding, but wouldn’t permit readers to capture an interesting connection they noticed to some idea outside the text. Readers couldn’t capture a detail they found interesting, but for which the author didn’t provide a prompt. Readers couldn’t capture an idea the text inspired the next day. More philosophically, this asymmetry promotes the idea that prompts are something you consume, not something you create. Imagine if you were forbidden to use any writing materials of your own while carefully studying a book: you’re only permitted to write in the page’s gutter margins. Your thoughts would shrink accordingly.

Of course, a simple solution here is to add an authoring interface to Orbit. One possibility I’m excited about is that authors’ prompt-writing practices will scaffold readers’ own prompt-writing abilities. But this won’t happen if readers can’t fluidly write prompts when inspired to do so. People today can add their own prompts in Anki or SuperMemo while they read a mnemonic essay. But my instinct is that it’s harmful to create a strong separation between engaging with author-provided prompts, and writing prompts of your own—for instance by requiring readers to visit a separate app or page to write their own prompts while reading. If you’re looking at prompts, you should be able to write prompts. Author-provided prompts should feel like material in your hands, malleable to your needs and interests, not a different “kind” of thing from prompts you write for yourself. I want the fluidity of plaintext: copy and paste between documents, edit and combine coarsely or finely, bulk-manipulate, generate, grep, pipe, tweet, etc.

Memory fantasies

Let’s step back and examine the medium’s original goals. To indulge in science fiction fantasies for a moment, it’d be great to jack a plug into our neck, flip a switch, and deeply understand any topic. That’s not possible (for now)—but how close can we get?

Imagine that whenever you read—or thought—something interesting or important, you’d simply remember that idea. Learning wouldn’t be quite as easy as plug-and-play, but you could read a book (once) and remember every meaningful detail; you could tinker with those ideas on your workbench and remember everything you noticed as you put them into practice; you could discuss those ideas with a collaborator and remember all the implications that arose. To fend off dystopian objections, imagine that you can also effortlessly correct any memories which turn out to be false or unhelpful. All this is also impossible, but we can at least come somewhat closer.

Spaced repetition memory systems let us remember a specific detail at the cost of 20-60 seconds over the course of the first year, and < 10 seconds per year thereafter. Not quite effortless, but also not terribly onerous—if you have a good prompt already written. Writing good prompts requires much more effort than reviewing does, and it requires a skill few people have yet developed. This suggests a slightly more achievable version of our fantasy: a genie automatically creates spaced repetition prompts for everything interesting you read, think, or hear; you remember all those details at the cost of, say, ten minutes’ daily review.

Our genie would need two skills: it must read your mind (i.e. to notice what you find interesting, in what context, and understood in what way); and it must write effective spaced repetition prompts (i.e. which cue retrieval of the details which constitute understanding). As a more plausible approximation, perhaps you can imagine hiring someone to follow you around all day, to sit in on your meetings, to read what you’re reading, to listen as you think aloud—and to write memory prompts for everything which seems important. You could call them your “chief of memory,” a nod to your chief of staff. This would be awfully expensive (and intrusive), of course; and because this assistant can’t read your mind, their work would be imperfect. But it’s interesting to consider as a non-sci-fi model you could actually implement if you had the means. Can we approximate this model more affordably?

Books are surprisingly analogous to at least part of this situation. If you’re wealthy, you can hire a personal tutor to teach you about a topic. This is inconvenient in some respects, and of course, it’s quite expensive. Happily, thanks to the printing press and the internet, you have the option to read a book about the topic instead. In some respects the book will offer a better experience than you’d have with a tutor: when you buy a book, you can (indirectly, partially) hire the greatest domain expert in the world to teach you. The book’s prose will (hopefully) represent careful editing and sculpted narrative, rather than improvised explanation. And of course you can read much more quickly than you can listen. There’s no equivalent conversational analogue to “flipping through the pages” or “scanning the headings.”

We can apply a similar logic to our “chief of memory,” at least for the portion of your day spent reading. The idea here is that many of the details you’d find important in those texts would overlap with details other readers would find important. And so perhaps the work that your “chief of memory” would need to do as you read this text would overlap substantially with the work that others’ “chiefs of memory” would need to do. If there’s enough overlap, it might become a high-leverage opportunity for a domain expert—perhaps the author, perhaps a different expert—to write prompts covering these overlaps. Indeed, that domain expert may be able to write higher-quality prompts than a general-purpose “chief of memory” could for that specific text.

Now we arrive at an idea resembling the mnemonic medium—and still not the mnemonic medium. In our thought experiment, the idea is that you effortlessly remember everything you find interesting or meaningful. The author-provided prompts are just material for that process. More concretely, imagine that as you read, you talk aloud to your “chief of memory” by raising an eyebrow or gesturing at material you find important. If the author’s already done the work to write prompts about that bit, and the prompts match what your assistant thinks you found interesting, they can take a shortcut and use the author’s prompt. Otherwise, they have to take the slow path of new writing a prompt by hand. Perhaps in some cases you speak aloud: “this detail actually relates to the problem I’ve been having with my research project; it’s a reason to consider doing X instead of Y.” Unlike the mnemonic medium, this workflow is driven by the reader’s interest, rather than the author’s specifications. The author-provided prompts are a shortcut, not a list of expectations.

Okay, now imagine you have no “chief of memory.” I recognize that we’ve come awfully far from “I know Kung Fu” at this point, but if we restrict ourselves to just the portion of your day spent reading, how close can we get in software using author-provided prompts? Ozzie and I have been exploring some approaches along these lines. Our prototypes are quite nascent, but perhaps you can imagine highlighting details you find interesting while you read; if the author has provided relevant prompts, you’ll collect those; otherwise, we might provide a bulk-editing interface for shaping annotations into prompts, including those you might create from nothing.

Trading away effortlessness

There are many problems with this approach! One central tension is that in almost all cases, the correct amount of authority to assign to the author is not zero; the correct amount of control and responsibility to assign to the reader is not 100%. How should we negotiate the spectrum of control between authors and readers?

The author’s prompts are not in fact just a shortcut, as I’d described earlier. They also carry meaning. Prompts signal what the author finds important. They communicate a norm around what it means (or at least what the author believes it means) to understand a topic. They give authors the opportunity to communicate the prose’s ideas in a different way, or even to create conversation between the prompts and the prose—but that’s a topic for another essay. They cue attention and participation; when they’re working well, they create a feeling of support and safety.

And so maybe we should still present the author’s prompts, perhaps as annotations, but ask readers to “opt in” to each prompt they’d like to collect. The trouble here is that interaction is a cost center in interface design. If 80% of the time, 80% of the users want to collect 80% of the author’s prompts, a naive opt-in mechanic would require readers to perform a huge number of interactions to indicate the common case. Imagine reading through Quantum Country’s first chapter and clicking on 112 prompts in the margins to collect them all. You can perhaps improve the situation by batching those interactions in end-of-session “review areas,” akin to Orbit’s current review areas—but I don’t think that’s enough.

If we’re not careful, we won’t just require users to perform excessive interactions: we’ll also distract them and create decision fatigue. Imagine that as you’re reading a text, you’re constantly evaluating prompts that appear in the margin alongside the text: do I want to collect this prompt? Or do I want to write my own? Your eyes dart back and forth between the text and the sidebar; your attention is drawn away from the text. The frenzied nature of this design can be improved by bulk operations at the end of sections, but it’s still hard to imagine asking users to explicitly decide whether they’d like to keep each of the 112 prompts in Quantum Country’s first chapter. It’s already quite an imposition to ask them to try to remember the answers.

Another difficult problem is reconciliation. Imagine that you find a detail particularly interesting, so you highlight it. You see that the author’s provided a prompt about that passage—great. But now you need to decide: is the author’s prompt about the sense I found interesting? You need to form some picture of the prompts you would write, if you were to write prompts, then read the authors’ prompts and perform a sort of diff. Worse: if you don’t quite trust the author as a prompt writer, you also need to evaluate the quality of their prompts. My experiments with reading others’ mnemonic texts suggest that both these activities are quite taxing.

I think a successful approach here is likely to be more incremental, annealing the prompt set through a number of stages. Perhaps you mark passages you find interesting with some very coarse interaction. If a passage inspires some specific prompts you know the author won’t cover—for instance because they’re about a connection to your present project—you can write those inline as you read. Intermittently, at the end of sections, you review the author-provided prompts from the passages you’ve marked, both to reinforce your memory and to offer a lightweight opportunity to discard or edit those which obviously don’t work. Perhaps you notice at this point that the author didn’t provide prompts about some detail you found important, so you write some on the spot. Then, over the following weeks in review sessions, you refine the prompts from this text, modifying and trashing those which don’t inspire, removing inadvertent duplicates, filling in details with new prompts, adding connections you hadn’t noticed, and so on. But critically, the text’s prompts feel like yours; they’re co-mingled with your own prompts about your own ideas. Taylor Rogalski suggested a metaphor I like here: someone sent you their Google Doc, then you clicked File > Make a Copy so you could scribble all over it with impunity.

A postscript on machine learning and language models

I imagine that many of my readers have been chanting it this whole time. What about language models?! Why insist that authors (or those who adapt their texts to the mnemonic medium) do all this work? What about the long tail of texts which will never be adapted?

I know. I’m interested in these questions too. I’ve experimented with several approaches along these lines, and my impression so far is that some partial automation here is possible, for certain kinds of prompts, but that a decent solution will require a great deal of work and a great deal of insight. I don’t believe the simple approaches floating around are likely to represent viable paths. But I do think this work is worth doing; if you’re interested in and capable of taking it on as a research project, and you’d like me to supervise (and possibly help fund) or advise your work, please reach out.

My instinct is that we’ll be best off approaching this as an augmentation rather than an automation problem, at least in the near future. I have some specific workflow ideas along these lines, but they’ll have to wait for another essay.

————————————

I’d like to thank Ozzie Kirkby for prototyping ideas around this problem with me this summer; Nick Barr, Ty Jung, and Taylor Rogalski for extended discussions and whiteboarding; and Gary Wolf for valuable correspondence on this topic.

And I’d like to thank all of you, Patreon sponsors, for your kind support. It’s quite remarkable to have the opportunity to pursue open-ended exploration like this. Your contributions make it possible, and I’m grateful.