Links for Q2 2025
#Blogs and Essays
Gwern, Evolution as Backstop for Reinforcement Learning. This post is the absolute peak of the Long Content genre, one where (paraphrasing niplav) perpetual drafts are getting refined and updated, never quite completely finished, but approaching stability upon many years of diligent note-taking and pattern-weaving. In this spirit, Gwern interprets free markets, reinforcement learning algorithms, and animal evolution as belonging to a single multi-level nested optimization paradigm where a sample-inefficient but ground-truth outer loss (e.g. reproductive fitness) trains and constrains a fast sample-efficient but possibly misguided inner loss (e.g. the motivational circuits of a human brain). Highly recommended.
Borges, Kafka and His Precursors. Yes, Jorge Luis Borges also wrote non-fiction essays—and was as good at this as he was at fiction writing! As Gwern has written: "If at times I have appeared knowledgeable or worth reading to others, it is perhaps only because I have stood on the shoulders of Borges and Wikipedia." See Gwern's recommended starting points for reading Borges's essays here. Other essays by Borges that I enjoyed this quarter include The Scandinavian Destiny, Blindness, Pascal's Sphere, and The Enigma of Shakespeare.
nostalgebraist, the void. nostalgebraist summarizes the history of the LLM revolution, arguing that our approach to chiseling helpful assistants out of base-model simulators is a historical accident rather than a principled choice, and that it is badly misguided in some ways. When the first HHH assistant began its post-training phase, it had to predict a character of which there was literally no training data. As more and more LLM-generated data makes its way to training corpora, the assistant character is getting gradually less underspecified, but the properties of the character are being filled in on the fly by base models that have to do their best at simulating a non-existant character. As some of the closest things to this character in the training data include sci-fi stories and Alignment Forum posts, we shouldn't be surprised that the base model simulator uses those pieces to fill in details about what the assistant character is like. The post spurred a lot of good follow-up discussion on LessWrong, I particularly recommend reading the comment section here and Jan's shortform.
Shalizi, In Soviet Union, Optimization Problem Solves You. In Shalizi's own words: "Over 7800 words about optimal planning for a socialist economy and its intersection with computational complexity theory. This is about as relevant to the world around us as debating whether a devotee of the Olympian gods should approve of transgenic organisms. (Or: centaurs, yes or no?) Contains mathematical symbols (uglified and rendered slightly inexact by HTML) but no actual math, and uses Red Plenty mostly as a launching point for a tangent." If you're like me, you'll enjoy it a lot.
Matuschak and Nielsen, How can we develop transformative tools for thought? In the visions of Alan Kay, Douglas Engelbart, Vannevar Bush, and other pioneers of the 60s and 70s, computers would be powerful intelligence augmentation tools, ones on par with writing and language. While they have indeed had a transformative effect, there's a nagging sense that much more should be possible. Matuschak and Nielsen present a compelling vision for the future of the field developing computer-based tools for thought.
Michael, To Dissect an Octopus: Making Sense of the Form/Meaning Debate. A delightfully in-depth overview of the debate over whether LLMs can ever infer meaning from form alone. On the one hand, it seems reasonable to say no, but on the other, we know now, five years later, that LLMs are no mere stochastic parrots. What gives? Michael claims that the debate has never really been about whether meaning can be inferred from form alone. Rather:
The important question is... how much of the grounding information can be derived from a very small grounding toehold plus reams of form data and very clever statistical inference.
LLM training data isn't devoid of any grounding information: e.g., Java code on GitHub contains unit tests, which specify input-output pairs for Java code and provide grounding. Though it seems clear now that this small amount of grounding is enough for LLMs to infer a lot of meaning, it's interesting to see what perspectives were held on this question in 2020.
Karlsson, How to think in writing.
Karlsson, On the pleasure of reading private notebooks.
Lakeman, The philosophy of Tyler Durden.
Gwern on why good bloggers shouldn't try to start writing books.
I've been trying to form a better sense of whether we should try to run toward automating alignment research. Some posts that I liked on this topic include:
- Hobbhahn, We should try to automate AI safety work asap
- Wentworth, Why Not Just Outsource Alignment Research To An AI?
- Pope, Dudney, Engeler, and Thibodeau, Research agenda: Supervising AIs improving AIs
- Carlsmith, AI for AI safety
Some good posts on whether we can think of LLMs as unified agents and when we can ascribe anthropomorphic qualities to them:
- Cotton-Barratt and Douglas, Decomposing Agency — capabilities without desires
- Kulveit, The Pando Problem: Rethinking AI Individuality
- Kulveit, Do Not Tile the Lightcone with Your Confused Ontology
METR, Recent Frontier Models Are Reward Hacking. Relatedly, Kei writes that Reward hacking is becoming more sophisticated and deliberate in frontier LLMs and Lilian Weng gives a systematic overview of reward hacking in RL.
Alvaro de Menard's media recommendations from Q1 2025.
Blogs Gavin Leech has found this year.
#Papers
See Paper Summaries 1—most of them are from the past quarter. Here are some more which I haven't summarized:
Anthropic, Specific versus General Principles for Constitutional AI. Turns out that instead of constructing complex constitutions, you can just tell your AI to "do what it's best for humanity" and it'll behave just as well or even better.
Anthropic, Circuit Tracing: Revealing Computational Graphs in Language Models and On the Biology of a Large Language Model. These two papers describe the most comprehensive and large-scale attempt to interpret LLM model internals thus far—self-recommending.
OpenAI, Toward understanding and preventing misalignment generalization. A thorough analysis of why and how emergent misalignment occurs. Despite having dismantled pretty much their entire alignment team a year ago, OpenAI has actually been publishing excellent safety research this year (Baker et al.'s Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation from the beginning of the year is probably my favourite paper of the year so far).
#Books
qntm, There Is No Antimemetics Division. I've been trying to get back into reading more fiction and this was a superb start. Read it.
Chiang, Story of Your Life, Understand, and Division by Zero. The same as above goes for Chiang's and Borges's short stories. I liked both Story of Your Life and Understand a lot, Division by Zero was less memorable. As a companion to Story of Your Life, I recommend Gwern's review and Michael Nielsen's notes.
Borges, The Aleph and Tlön, Uqbar, Orbis Tertius. These are widely regarded to be among the best short stories of all time and lived up to their reputation. I recommend also reading Gwern's review of the latter.
Aaronson, Quantum Computing Since Democritus. See my review here.
Chalmers, The Conscious Mind. See my review here.
Dawkins, The Selfish Gene.
Pinker, The Sense of Style. An excellent guide on good writing. Recommended.
Buzsaki, The Brain from Inside Out. Contains a lot of good insight porn on neuroscience. I'm still unsure how strongly I agree with the central thesis, might write a review focusing on that in the future, but the argumentation is delightfully rigorous. See a summary of the book here.
#Twitter and Wiki roundup
TracingWoodgrains makes a passionate case against Brian Tomasik and negative utilitarianism.
A periodical reminder that mindspace is larger than you think: There exists a condition called hemispatial neglect where after damage to one hemisphere of the brain, the patient has a deficit in attention and awareness towards the side of space opposite to that hemisphere. E.g., people with this condition might shave only one side of their face or finish food only from one side of the plate. If asked to draw a picture, they might draw only one half. From The Brain from Inside Out: "When a well-studied patient in Milan, Italy, was asked to imagine himself facing the Piazza Del Duomo and describe the scene, he correctly identified buildings on his right but could not recollect things on the left. When he was asked to imagine standing at the opposite end of the Piazza, he listed the buildings and structures on the other, previously neglected, side, which was now to his right." This means that an image on the retina is insufficient to perceive space. The brain must also know where the eyes and head are pointing.
Do languages with more speakers have a more efficient vocabulary? Perplexity tells me that this isn't the case, which contradicts my intuitions: since language is, to a large degree, a device for transmitting information, and languages evolve over time to suit their speakers' needs, one would expect that languages that are older or have more speakers are more efficient. One bit of evidence against my intuition is Different languages, similar encoding efficiency by Coupé et al., who find that the information communicated per second of speech is similar across languages (around 39 bits/s). However, Coupé et al. use a corpus of 17 fairly well-known languages, and I'd be interested in seeing a deep dive on whether more obscure languages are less efficient, and if not, why. An anecdotal piece of evidence in favor of the hypothesis that more obscure languages are less efficient is that Gaelic seems to be unoptimized.
Apparently, most langauges are right-branching, meaning that the main clause of the sentence comes first and modifiers after it, while Turkish and Japanese left-branching, i.e. exactly the opposite. My Turkish friend did not report any Sapir-Whorf-y consequences when discussing this.
Richard Ngo explains why he became a virtue ethicist.
Can aphantastics use visual mnemonic techniques?
Henrik Karlsson on why automating art will lead to less great art.
Delivery robots are still illegal in the UK due to... a law written in 1835.
There's been a lot of Gwern in this post, so I'll finish it with a certified Gwern moment.