Preparatory notes for a manifesto

A personal note: my mother has a rapidly advancing cancer, and I've been quite distracted these past few weeks coordinating her care. I ask your patience with slow(er than usual) responses and inconsistent access to audio recording gear.

I've been spending a lot of time recently trying to articulate the perspective at the center of my research. These disparate projects all feel like part of one larger vision to me, but I haven't done a great job of explaining that. It's not just a matter of communicating to others: I haven't articulated it well for myself, which can make it hard to steer.

So this month I'd like to share some preparatory notes towards some clearer articulation of what I'm doing here. They're incomplete and wrong in many ways, but in many places they do represent a real increment in my thinking. And one thing I've learned about research is that often all I can ask for is to improve my understanding, one increment at a time.

The awe at root

My central obsession is with humanity’s ability to transcend itself: to invent outward forms which become part of us and then redefine our inward capacities. We are the same, biologically, as our pre-literate, pre-iconic ancestors, but our minds are utterly alien.

What’s even more directly inspiring for me is that single individuals and small groups can create such forms. The invention of written language feels abstract and diffuse. I’m tremendously inspired as a designer by the likes of Playfair, Mendeleev, the Lumière brothers, Sutherland, etc. I want to lay bricks in that cathedral. I want to continue the arc of human transformation begun by written language and cave paintings.

The frustration at root

Language, paper, ink, graphics, video—astounding, yes. And yet: not good enough. I feel a profound frustration at the inadequacies of our mediums for knowledge work. I feel the excise and waste. My central complaint is that the mediums are inert. I have to drive everything: orchestrating my attention, understanding, internalizing, practicing, sensemaking, drawing connections, creating anew. The words are dead on the page; the environment is largely unaware of the practical mechanics of cognition, scholarship, and creation.

So I strain to understand complex material, remember little of it, and deeply internalize even less. I lose track of papers and videos in a bottomless “to-read” pile, just as I lose track of notes on my own ideas. Working on a topic, I’ll read and mark up dozens of articles, but those marks end up lost in the margins; there’s enormous excise in seeing the whole, making sense of it, connecting it to ongoing conversations and notes over time.

I want materials and workspaces which actively support knowledge work, designed to reify our deepest understandings of human cognition, scholarship, and creative work. If a paper and pen can make me an alien to prehistoric humans, I want to make a surface which is to paper as paper is to no paper. When I strain at the page, I want it to be because of the essential complexity of the ideas, not because I’m needlessly taxing my working memory or metacognition.

“Books don’t work”, broadly construed

It’s not just “books don’t work” in the sense that I forget much of what I read. I also fail to absorb key points without even realizing it. My understanding too often remains very shallow. I strain to visualize and manipulate complex dynamic systems described with inert words and symbols. I fail to spot connections to other things I’m reading. I fail to connect my reactions to ongoing conversations with friends and colleagues. I make notes in the margins, and they usually stay there lost in the folds, useless. I have a hundred papers lost in open tabs, but I end up reading a paper linked in a recent tweet instead of the ones I’d endorse as most relevant/meaningful. I have mental and practical routines which can help with most of these problems, but they’re costly and unreliable, so I use them inconsistently.

It’s not just “books don’t work” for the purpose of learning. What I worship is the feeling of understanding something complex and beautiful, then making it one’s own, making something new of it. My interest is in creative engagement with ideas, in enablement, in doing. And so the problem is not with the isolated inertness of individual books but with my whole creative and scholarly workspace: the connections between the materials, the index cards strewn on the table, my own writing in progress, the computational systems I create from my writing.

It’s not just “books don’t work” in the sense of literal books. When I say “reading and writing”, I include static figures, videos, audio lectures, whiteboard drawings, and so on. The real project is cognitively convivial communications environments which proactively support the consumption, synthesis, and production of creative thought.

A sense of possibility

The point isn’t digital reading or digital knowledge work for its own sake. Like many of the personal computing pioneers, I see computers as a means to an end. Sensors to collect context and expressive input. Computation which can decide what to do with that context and input in real-time. Displays which can present dynamic information that proactively supports the knowledge work. Networks that ground the information in social and collaborative contexts.

I’ve seen what happens when we take those raw ingredients seriously. We get environments like modern digital audio workstations and CAD tools, which have utterly transformed the nature of creative work in those fields. My sense of possibility here comes from a parallel frustration: that almost no one competent has even tried to reproduce those successes for knowledge/scholarly work.

Many academics have tried, of course. But I also have a subtler belief—that success here involves inventive interface design work. Academics don’t really do that. I believe what’s needed is the imaginative design energy which created the iPhone and Facebook Paper, but directed towards enablement. I worked on Apple’s most inventive contribution to reading interfaces: the 3D “page curl” interaction when turning a page. It’s sad. Almost no one I met there read or cared about reading. But I did see designers creating Final Cut Pro and Logic, with tremendous imagination and passion. That’s what’s needed here. That’s what gives me such a strong sense of possibility.

The vision of the Primer

My quest is against the dead page. We can think of that as two targets: the words themselves—the content—and the page or desk they sit upon—the workspace. I’ve mostly talked about the latter: extra tools and layers which augment memory/learning, surface connections, orchestrate attention. That stuff sits on top of and between the content.

But I’m also deeply frustrated with the inert content itself. That’s one of my deep connections to the vision of the Primer. It’s a vision of situated communication, the “multimedia” we deserve. Words woven seamlessly into environments of use. Directly manipulatable responsive representations.

I don’t want to watch Grant Sanderson explain quaternions with a beautiful 3D visualization I can’t actually use myself. I don’t want to watch a video about EQing audio track in Logic, constantly pausing and switching windows to compare the knobs in the picture to my own running copy. I don’t want to read a paper about a novel interface design, zoom into the tiny dead figures in the PDF, then try to line that explanation up with the live demo in my browser. I don’t want to look at a statistic in a ML paper and rummage through endless Python scripts to find the details of the evaluation which produced it.

I want universal deixis. I want to break down barriers between discussion about a dynamic representation and the thing itself. It’s “don’t separate by mode of production”, not just in terms of producing a static explanation, per Tufte, but in terms of the full creative process.

Part of this vision is that it needs to be much easier to create and even improvise dynamic representations. But that others (including Bret) are working on that; it feels less centrally like my own personal quest.

Differences with pioneers

Relative to Bush in “As We May Think”. The memex is mostly a database. It’s primarily about instant access. Its main write-oriented verb is to create a coarse connection between documents. I think this is a good metaphor for a lot of what computing has done for the core elements of knowledge work and scholarly work: unlimited storage, instant access, a search feature. The machine can present whatever information you want. It’s up to you to decide what it should present, and to make something of it.

Relative to JCRL in “Man-computer symbiosis”. For him, computers are for calculating and processing information. We outsource costly computation to them but generate and explore the hypotheses ourselves. I get the image of a very elaborate desk calculator when I read him. I’m interested in a different kind of symbiosis. I want a communications media environment which provides active cognitive support, which is aware of how thinking, learning, sensemaking, and creation happen—both for humans in general, and for me in particular—and which presents affordances, adaptations, and information layers accordingly. It’s not a desk calculator; it’s more like magic paper which transforms your engagement with what’s inscribed upon it.

Relative to Alan Kay. His vision is of a new universal literacy—that we should all be able to read and write dynamic systems. I’m interested in that project, but my focus is less on fundamentally computational content (like simulations) and more on using computation to make communications media more active participants in understanding and creation. His instinct was to focus on children, so that the new literacy could shape their whole understanding of the world. My instinct is to focus on adult experts, because that’s where I see the most compelling levers. Also because trying to impact children means navigating corrosive condescension/paternalism (in oneself) and enormous entrenched sociocultural institutions.

The role of AI

My sense of possibility in this space was plenty strong before the current wave of AI. Most people interested in AI for knowledge work are focused on generating output, on automating parts of the creative process. The user plays a manager driving agents which do the substantive work. I’m happy to automate the mundane, but I locate the creative center within the user. I want to support the complex processes which happen inside the user’s mind—in understanding, sensemaking, ideating, expressing. This is more a model of symbiosis and extended cognition than outsourcing.

That said, I want my cognitive workspace to have awareness and (to some degree) agency—to provide active support as I understand, synthesize, and create. I want that support to be richly grounded in my context, in everything I’m reading and writing, in my own aims and activities. I think AI is likely to be an important instrumental ingredient in that.

For example, suppose I’m writing notes about a new idea. A forgotten paper buried deep in my “to-read” list contains some relevant background. It’s suddenly quite relevant to my interests in the moment, upgraded from “hm, maybe I should check this out someday.” I want my environment to notice that connection and to unobtrusively surface it to me. Most “reading lists” present the interface of an endless abstract to-do list. I want a “reading list” that deepens my orientation to what I care about in the moment, presenting entries not as a static database query but responsively to my context.

Another example: suppose I’ve written some flashcards about details I found particularly interesting. I want to engage in retrieval practice on the substance of those cards, not their form. But after a while, I pattern match to their form. AI can vary the surface language to avoid that—and, perhaps, can helpfully vary and deepen the substance over time as well.

Lots more relevant examples: multi-level summarization of texts, re-surfacing old notes which might have relevance to new work, marginalia which point out the parts of texts which might be most relevant to my work, automatically synthesized dynamic representations of static descriptions, etc etc…

The project, overall

Hard to summarize. Transcending paper, ink, written language, static figures and videos. An environment tailored for human cognition, which actively augments reading–writing–thinking. (Like a full-body exoskeleton for the mind?) “Paper” that works on your behalf to support understanding and creation.

---

Thanks to Ozzie Kirkby, Geoffrey Litt, Michael Nielsen, and Sara LaHue for helpful conversations as I've pulled on these threads.