A malleable reading environment

Inspired by last month's letter, I recently wrote this grant proposal for a new project to begin late this year. I decided not to submit it after all, at least in this form, but I thought you all might it stimulating, despite its brevity. (The grant had a very tight length limit!)

---

If you watch over the shoulder of an experienced programmer, you’ll see an environment exquisitely tuned for an expert doing serious work. Dense overlays surface rich information for understanding the system. Sharp tools allow the programmer to rapidly manipulate their material. This environment can fluidly shift with the needs of the work. And if you watch a different engineer, you’ll see different interfaces, reflecting different needs.

Now, watch a scientist reading papers or books on their computer. You’ve moved from a dynamic airplane cockpit to Fischer Price. Digital reading environments are impoverished, both in comparison to their physical equivalents, and in terms of what computers make possible. These systems give us a dictionary, search, and perhaps some clunky annotation tools. Serious readers deserve serious reading tools.

People have made unusual digital reading tools before. One system makes it easy to follow citations. Another adds LLM chat. Yet another lets readers manipulate clippings on a spatial canvas. But these projects are siloed, incompatible. Each must reimplement a complete environment before it can add its special tool. Readers can’t use one system’s citation tool and another’s annotation tool. The work can’t accumulate.

And so I propose a malleable reading substrate—one which can be flexibly adapted and extended like a programmer’s tools, or as writers have done with Emacs for decades. Unlike those systems, which require tedious programming to extend, this substrate will be designed with LLMs in mind, so that idiosyncratic workflows can be improvised on the fly.

This project has two interlocking workstreams: creating the first malleable digital reading environments, and using that environment as a lab to invent and develop novel interfaces for scholarly reading. The latter lends itself to parallelism, to a small team of designers exploring in a focused studio environment.

A small sample of the reading interactions I’d like to explore:

• Programmable lenses and highlighters. e.g. Surface contradicting evidence from your library in the margins. Shade citations which do not support the claim being made. Highlight a key term in purple to add its definition to your spaced repetition system.

• Multi-device workflows. e.g. Do a quick pass over several papers using a stylus and tablet. Then your desktop simultaneously displays all the passages you marked, so you can draw connections across documents.

• Social connections. e.g. Discussions about this paper from your team’s chat system appear in the paper’s margins, and vice versa.

• Multiscale representations. e.g. The scroll bar displays visual information about where you’ve spent time, the density of your markup, etc. Seeing the structure of the whole text while viewing a part.

• Deep integration with writing tools. Reading, writing, and thinking are inseparable. Your reading tool should be bidirectionally connected to your writing tool.

---

Thanks to Ozzie Kirkby for helpful comments on an earlier draft.