Link Archive - Jan 2025

A list of links I found worth saving in January 2025. This is the first of what I hope will be a monthly series.

This post gives an overview of where AI stands as of December 2024. The author takes a rational and conservative approach, using first principles. For example, while not buying into the hype from Sam Altman and other leading AI labs, they also believe that model scaling is far from over and that there’s no real evidence to suggest otherwise. I liked the point that Altman and Sutskever have incentives to shape the narrative, so their statements might not be the whole truth. The post touches on inference scaling too. Balanced piece.

I think Human-Computer Interaction (HCI) is important. A computer is fundamentally a tool, and it is most useful when the interaction removes bottlenecks and inconveniences. Human-AI interaction isn’t really an established field yet, but I believe it will be. The author argues that instead of interacting with current LLMs as if they were people, we should treat them as tools. They suggest an LLM could dynamically generate an interface, similar to a computer. What would that even look like? Conversing with AI can be slow and isolated, just back-and-forth in a box. Human-computer interaction isn’t like that. What if we fundamentally changed how we interact with AI? Thoughtful post.

I first encountered this blog post at the end of 2022, around the arrival of ChatGPT, and I recently rediscovered it. The author tried to simulate a Unix terminal environment using ChatGPT, and surprisingly, they succeeded. This might not seem unusual now, but it really surprised me when I first started using ChatGPT. The author is clever, and they take it a step further by going “all meta,” which I won’t spoil. This showed me that the LLM could simulate a system, in this case a Unix system. It gave me the sense that this thing was significant and could change a lot. I still believe that, and I owe some of that early shock to this blog post.

I initially knew François Chollet through his Twitter activity as an AI skeptic, or at least somewhat of an AI skeptic, and his work on ARC prizes. So it was exciting to see him interviewed by Dwarkesh Patel. I really enjoyed it because Dwarkesh grilled François on his views on LLMs. I think Dwarkesh’s approach was a bit unfair because, while François made some valid points, Dwarkesh seemed intent on pushing him to say what he wanted, but he ultimately failed to do so. While listening, I felt torn between them. I agree with Dwarkesh’s overall viewpoint but align with François’s specifics on intelligence and the core of reasoning. It was a conflicting but invigorating discussion, and I think I should re-watch it.

After the Dwarkesh podcast, I turned to Lex Fridman’s podcast to hear more from François Chollet. And, in classic Lex Fridman fashion, he delivered. This first one is an earlier interview from 2019, when François was much younger. In current public settings, François appears more glum and calm, but in this interview, I could see a sparkle and a smile in his eyes and on his face, which was refreshing. The podcast goes into the early days of Keras and TensorFlow, which I really enjoyed.

In this podcast, both François and Lex are a bit older. I watched this because I was simultaneously reading François’s paper on the measure of intelligence. He’s soft-spoken but has clear views on intelligence. While I don’t agree with everything he says, since I think LLMs and other AI systems are capable of more generalization than François suggests, his skepticism is not unfounded. His logic is sound, but I think he underestimates the intuitive power of these machine learning systems.

This is a useful method for visualizing and interpreting models. The paper tries to reverse-engineer what features each neuron or layer responds to in an image. The basic idea is that, through iteration, we generate an image that a particular neuron fires most strongly for. This yields interesting images of what the neuron has learned, and I find it very compelling.

This week was all about the buzz around DeepSeek’s release of R1. Ben Thompson distilled what mattered. Ultimately, the main beneficiaries are customers and the general public. I think the broader point is that open-sourcing is both morally and practically right, since AI models are becoming commoditized and costs are racing to the bottom.

This is an interview with the DeepSeek CEO, and based on the interview, he is very ambitious and focused on innovation, which I admire. Previously, China was primarily focused on keeping up with the US, but he feels they need a culture of innovation. He argues for moving from following to leading in some areas. I’m glad he is bullish on open-sourcing and that his company remains committed to it.

This is an interesting take, drawing an analogy between Facebook’s unconventional approach to advertising and the previous advertising agency model. Facebook directly connected consumers and advertisers via algorithms. Meanwhile, traditional advertising companies acted as middlemen between ad sellers and big brands. The author connects this to OpenAI reasoning models and basically says that those agents will not replace individuals inside traditional companies. Instead, AI-driven organizations may start without those roles from the outset.

It’s pretty interesting to see Amodei’s reaction to this, and I think he’s right. If you’re on the American side, the most logical thing to do is to clamp down on AI exports, specifically chip exports. But the Biden administration didn’t constrain chips on the bandwidth side, which is why DeepSeek had a workaround using maximum bandwidth. If they clamp down on both bandwidth and compute, I’m still not sure that would be the fundamental solution to this arms race.

That’s it for January. I plan to keep saving these as I go.