Augmenting scholarship: a proto-proposal

Thank you all for your kind notes these past months as I've supported my mother through her final stages of cancer. She passed away a few weeks ago, and I'm glad that I was able to be with her to make her final days as meaningful as possible.

I'm still very much recovering. I expect I will be for some time. So I felt deeply disoriented when I was notified a few days ago that I've been awarded a large grant I proposed earlier this summer. It would make it possible for me to hire a full-time team of several collaborators for a year, and to build out a studio space for our work. For this month's letter, I'd like to share the main section of proposal with you all.

I confess I'm feeling quite ambivalent about this award. The funding is timeboxed for a year starting from October, and ideally I'd have the team already recruited by that start date. I am—to put it mildly—not feeling like a creative leader at the moment, and certainly not a high-capacity recruiter. I may try to defer the award. Also, I've been really enjoying my collaboration with Taylor on programmable attention, and this project would take me in a different direction. (We have another report on that project coming up.) We'll see. More as I have it…

The awe at root

A few thousand years ago, our ancestors didn’t have written language. We’re almost the same as those humans, biologically, but what goes on in our minds is alien, incomprehensible. Growing up in a world with paper and pen has changed us. We can follow a chain of thought beyond our working memory. We can think abstractly, analytically. We can build on ideas of people we’ve never met, and record our own for posterity. Natural selection didn’t give us those powers. Other people did. We inherit the astounding power to shape tools which then shape us, the power to transcend our own mental capacities.

Written language was the work of whole cultures. But what excites me most as a designer is that small groups and even individuals can invent transformative tools for thought. I think of Playfair and statistical graphics in the eighteenth century, or Mendeleev and the periodic table of elements in the nineteenth. Tools like these are discovered as much as invented: their power lies in distilling deep domain insights (like periodic law) into external forms we can use to think and communicate.

In the twentieth century, the computer enabled a new class of these forms—dynamic media which behave and respond. When the modern architect drafts a site plan, his medium comes alive to actively support his creative thought. He sees in real time how his choices affect sunlight, material strain, and building cost. When the modern music producer mixes a track, spectrum visualizers let him quickly spot and fix muddy segments; automation curves let him express and manipulate the sound consistently. We can see similar stories in climate modeling, genomics, and other quantitative domains. These new media absorb part of the cognitive load, so that we can explore and refine ideas more closely to the speed of thought. In so many arts and sciences, dynamic media have once again created alien minds.

The inadequacies of scholarly media

Meanwhile, the basic media of scholarship have changed remarkably little. By that, I mean reading, writing, sketching, presenting—the communications media we think with and through. These media are astounding triumphs of human civilization. And yet: they’re profoundly inadequate. We often fail to absorb key ideas from what we read, without noticing. We forget details we found fascinating last week. We find our working memory constantly overloaded. We write notes in margins, then struggle to see the big picture, to make sense of the thread connecting various texts and our own ideas. We drown in an undifferentiated backlog of browser tabs and citations.

We nominally use computers for all these tasks, but we’re not really using them. We’re mostly just working with pictures of paper on screens. The miracles of dynamism are mostly confined to simple retrieval: search, databases, rapid access. In other domains, we have dynamic media which dance with us, which provide active support for our difficult creative work. Meanwhile, in scholarship, the paper lies inert, and we shoulder the full cognitive load. Echoing Engelbart: bricks are still strapped to our pencils.

Where is our AutoCAD for scholarship? Where is our Logic for scholarship?

You could say: maybe scholarly media are special. Maybe there’s something about reading or writing or presenting which prevents the transformative tools for thought we’ve seen in other domains. But in fact, modern science has uncovered a great deal about human cognition, sensemaking, and insight. Surely we can incorporate those powerful ideas into new media.

For example, cognitive psychologists understand the dynamics of memory formation quite well. There’s something like a recipe one can use to reliably and efficiently remember anything—the trouble is that books don’t naturally perform it. With collaborator Michael Nielsen, I invented a new kind of book which incorporates that insight, so that the act of reading naturally performs that recipe. We published a quantum computing textbook in that new “mnemonic medium” and demonstrated that with a small time overhead, our readers reliably remember hundreds of detailed points from the text, long after reading. Meanwhile, without support, most readers forgot many key details. So, clearly, it’s possible to radically augment reading. Some of my other recent experiments have demonstrated promising directions in sensemaking, writing, attention, and other central practices of scholarship.

Today’s software industry is enormous. Why hasn’t it pursued this problem more seriously? We could quibble about whether the market for architects and musicians is larger than the market for scholarship. But I favor another explanation. Powerful new tools for thought require both imaginative design insight and deep domain insight. I first learned design at Apple during the early days of the iPad. I watched designers with long-held passions for music and film pour creative ideas into Logic and Final Cut, tools that have transformed their respective industries. Meanwhile, I helped build Apple’s most innovative contribution to reading interfaces: the 3D “page curl” effect that follows your finger when you turn a page. I don’t think it’s an accident that Apple designers’ ideas about reading interfaces were limited to window dressing. None of my colleagues read seriously. That’s not specific to Apple; it’s been my experience with industry software designers in general. Scholarship is just not part of the culture. Meanwhile, academics in cognitive psychology and computer-supported collaborative work have domain insight, but bold design work is rare in their culture.

The project at hand

And so, I propose a program of augmented scholarship. We’ll invent tools and media which will transform the intellectual work scholars do through communications media—reading, writing, sketching, presenting. If paper and pen made us aliens to prehistoric humans, I want to make a surface for thought which is to paper as paper is to no paper. I want to make future scholars unrecognizably more capable.

My primary approach is to repeat what has worked in so many other domains transformed by dynamic media, but which has rarely been attempted for scholarship: to combine deep insights about the nature of human cognition and scholarly work with skillful and imaginative design work. While we catch scholarship up with the last revolution in media, we’ll also participate in the next one. Modern language and vision models offer dizzying opportunities to make scholarly workspaces contextually aware, not only of what a user is reading presently, but of how that connects to everything else the user exploring. Now, I’m not interested in automating the creative process. I locate the creative center within the scholar. But just as a pen can become an extension of the hand and mind, I believe AI can become a supportive extension of sensemaking and discovery.

I’ve spent the last few years building prototypes towards this vision of augmented scholarship, mostly alone or in dyads, and with minimal resources. Now that my work is accumulating, and frontier models have become more capable, it’s time to take a larger swing. I’d like to assemble a team of talented designers and technologists in a studio environment for an intense year of invention and experimentation. We’ll draw on my past few years of foundational prototypes and develop new ideas together. Because this project’s success or failure will depend on the richness of our domain and design insights, we’ll kick off the year with a collaborative workshop, inviting some of the most inspiring inventors in the world—people one could never ordinarily recruit for a full-time role. Embedded within [the institute], we’ll create a living laboratory, our prototypes serving [the institute's] scholars, and our observation of their use serving our creative process. We’ll publish our design work continuously, and we’ll end the year by delivering open-source demonstrations of new approaches to augmented scholarship, validated by internal and external usage observation.