From teaching machines to teaching people: an interview with Michael Littman

Welcome back to AI Pioneers, our series of in-depth conversations with leading figures in artificial intelligence. In this interview, we speak with Michael Littman about the evolution of reinforcement learning, the role of creativity and culture in how humans and machines learn, and the challenges generative AI poses for education. We also explore his approach to science communication, his use of music to convey computer science concepts, and his unexpected connection to the first multi-media email attachment.

You’ve been in the field of reinforcement learning for decades. How did you get into it, and which questions have you found most interesting across your career?

I was drawn to reinforcement learning when I was a college student in the mid to late ’80s. At that time, people were just starting to talk about neural networks, and the machine learning techniques being used were supervised, which is very focused on the input and output. I didn’t find that so interesting – I wanted to know how to make them behave.

So, I created my own reinforcement-learning system to address the game of noughts and crosses, or tic-tac-toe. It was a small enough game that I could run it on my little computer at home, but it made me address some interesting questions. For example, what does it mean to get feedback? And how do you use feedback at the end of the game to know what you should have done at the beginning of the game?

I started to explore this topic without knowing that it was reinforcement learning. It wasn’t until 1988, when I got out of college, that I discovered the link. I was working in a research lab with a researcher named David Ackley, who had done work on similar kinds of models. He used to be a student of Geoff Hinton, so he was very familiar with neural networks and very interested in behavior. I explained to him what I was playing with and why I found it so interesting, and he said “oh, that’s called reinforcement learning, Rich Sutton just published a paper on this – let’s get him to come and give a talk.”

In this way, I got to hear about Q-learning probably before any of the papers on Q-learning were published in around 1989.

How has reinforcement learning evolved since then?

That’s actually a really interesting question, because what we now call deep reinforcement learning is in many ways a departure from what we have been doing for many years. Now the approach is to throw a lot of compute at the problem and see how well things work, but the resulting systems do solve some really interesting behavioral problems.

For example, we now don’t just have programs that can play noughts and crosses, but also Go and video games and all sorts of really tough challenges. The field has moved a lot in the last couple of years as people have combined deep neural networks with training those networks on decision-making problems.

That said, it still doesn’t feel solved to me. The things that drew me in early on are still really hard. The fact that we’ve thrown neural nets at them doesn’t make them better understood. It just makes them more solved in a practical sense.

The amount of data that you have to give these systems until they can make
reasonable decisions is exhausting – it requires that the system has seen almost all possibilities. We don’t see them making the kind of creative leaps that animals can. Like Rich Sutton, I was very inspired by the relationship between computer and animal decision-making. I think that we’re still pretty far from what animals are able to do to just navigate the world. There’s still a lot of progress to be made.

What do you think needs to happen for AI systems to be able to kind of make those creative leaps that you mentioned earlier? Do you think we can do that within current paradigms?

It’s a good question, and I’ve talked to a lot of people about this. Some folks feel like the right answer is just more data. They say that we just need to train these systems over and over again, giving them lots of exposure to things, and then they’ll make good decisions. There’s some evidence of that, but the scale of that data and the complexity of the decisions that it supports just doesn’t work. You can’t get to the full space of the kinds of survival decisions that need to be made just with lots of examples, right? There has to be something else.

Typically, systems are subject to the outcomes of their decisions. If the answer to a problem is seven, it’s not enough to just announce the number seven – it has to believe that the answer is seven. It has to be the case that if that was the right thing, something positive happens. If that was the wrong thing, something negative happens. And the system is able to adjust itself within that feedback loop. That’s reinforcement learning.

Right now, I think that the methods we have scale so poorly with the amount of data that we can’t really give them opportunities to see the consequences of their actions. We can do it in data-limited spaces like math, where we can use AI to prove a theorem, and then we can verify that the proof is correct. Then we can train the system with real experience, and that has been quite effective.

If you’re thinking about the real world, such as a robot that’s able to navigate in someone’s house, then we’re very far behind. The things that people are trying to do at the moment involve scaling up the amount of data. We have to make it so that we have a whole warehouse full of robots that are all doing things so that they can learn from that to actually make better decisions. But this direction just doesn’t scale. It’s too slow, and there are too many special cases that have to be experienced. At the end of the day, this is not a winning strategy.

I think we need to do something a lot more like what happens with real critters – with people and animals. There are folks who acknowledge that. Animals and human beings get to where they are through some combination of an evolutionary process that gets you to birth, and then a more learning-oriented process that happens after birth.

The evolutionary process up until birth introduces some really important structures and concepts that we don’t have time to build ourselves. We can’t evolve AI systems this way – it requires millions of years. So we’re going to build that part in and then we’re going to let learning work on top of that. But knowing the right stuff to build in is really hard, and so I think people are stumped at this part.

Then, there are folks who are developing world models, which is reinforcement learning-adjacent – it’s definitely addressing some of the same motivations. In reinforcement learning, we’ve had notions of world models since the beginning, such as the idea that the learner has the ability to predict what will happen when it takes certain actions. The question becomes, what is the structure of that world model? And do we have enough data to constrain it in a way that’s going to be useful for making good decisions? I think the jury’s still out on that.

What kind of research questions are you finding most interesting at the moment?

I’ve been focusing on some things that came out of my book, Code to Joy. In particular, I’ve moved away a bit from this notion of just building an agent or a program and then letting it figure out how to solve problems itself. I think human beings maybe think that’s what it’s like to be a human being. But, the fact of the matter is that we’re born into very rich cultures, of ideas and of artifacts.

There are things all around us that are all trying to point us in a direction of how best to survive, interact, and make good decisions – we’re being given very detailed signals from our conspecifics, or from other members of our culture. When we convey tasks to each other, we tend to do it in a way that involves lots of different kinds of signals.

In the book, I explored the notion of explicit instruction, or programs. You can tell somebody what to do by saying, do this and then this and then this, and then if this happens, do this, but if this other thing happens, do that. You can spell it all out. That’s a pretty effective way of making machines do things and it’s the basis of all the computers that we have today.

But there are other models as well. For example, in supervised learning you give example instructions. Instead of saying, this is what you do in this circumstance, you say, here’s a circumstance. Here’s the right thing to do in that circumstance. Here’s a different circumstance. Here’s the right thing to do in that circumstance. We’ll stack up thousands or millions of examples like that and let the computer figure out the rest. We do this with people too, but we tend to use smaller sets of examples, because what person could process a million examples? Still, that notion of example instructions is another way that we convey tasks to each other and to computers.

Another method is reinforcement learning, which uses explicit incentives. The idea is that we’re going to very explicitly explain what the task is and what the reward will be for completing it. I’m not going to tell you what to do in any given circumstance, but if you see you’re not accomplishing the task, you should probably try something else to get your reward. That’s the essence of reinforcement learning, and we do this with people as well.

The fourth quadrant of this set of ideas is example incentives, which are not used as often. It’s the idea that I’ll show you what I do and you should figure out from that what I’m trying to accomplish. You can only see my behavior, but you’re going to try to figure out the description of the end state from that.

What I find super interesting is that when we convey tasks to people, we tend to mix and match all these different mechanisms together. For example, we’ll say, “here’s what you should do”, and then you explain it with an example, or give an explanation that you can apply to more general circumstances. Or, I can tell you what you should be doing. We use explicit instructions, example instructions, and explicit incentives, all the time, and all mixed together.

In computer science, we’re just doing one at a time: it’s either programming or supervised learning or reinforcement learning, depending on which you think is the best for the particular task. Individually, they all have their limitations. It’s only when you start to mix and match that you start to be able to buttress the shortcomings of the different models and you end up with something that is actually more robust.

That’s the research that I’m engaged in now. From this starting point, how do we build a system that can combine these different feedback modalities? Is it some kind of Bayesian reasoner? How do we make that scale? Is there anything that’s happening in these large language models that we can leverage to try to do this kind of multimodal specification of tasks and behaviors? There are a lot of questions to investigate.

I like your point that culture is a form of implicit learning. I suppose that that’s what we’re doing with LLMs – we’re giving them all of the data on what the culture is, and getting them to learn from that.

I’d like to say that I think people are really smart, but we’re not that smart – we’re not NP-complete smart. What we are really good at is taking the pieces that are around us and assembling them into something that’s a little bit better than what we have before. If you iterate that, you can get to some very advanced stuff. That’s human creativity at work.

But usually we test our AI systems without that. We just say, you’re in a world, good luck. I think that is provably intractable. The only way that we manage to survive is that we leverage each other.

We have strong survival incentives as well.

Yes, and the need to survive. Lately I’ve been working with one of my PhD students on this, and I’m referring to it as “skin in the game”. He is doing these experiments where he asks large language models to make consequential decisions, but there’s no consequences for the models. They have no skin in the game – they make these decisions without a survival instinct. The fact that we’re asking them to make decisions without consequence to themselves means that we should really be very skeptical of the recommendations they give us.

You published Code to Joy just as ChatGPT was coming out. Did you anticipate the impact of AI coding on the way people would learn to code?

I have a whole section in the book where I’m marveling at GPT-2. It’s incredible that it can talk about colors and yet has no eyes, or in the way that it can complete sentences in novel ways. I did make a note that people were starting to play around LLMs to generate code, and that this could be very useful. Still, that was written before ChatGPT, and before Claude Code. It is very different now.

The theme of the book outlives the rise of LLMs – people should learn to program because it’s empowering. As a person who’s been programming for many years, it’s very freeing to be able to tell the machine to complete a task for me. I was trying to make the case that while coding is annoying, the syntax is weird, and it makes you have to be very careful and logical, the benefits are there and the concepts are not as hard as you think. They are built on top of things that everybody does every day. There’s nothing really that foreign about it – it’s just an extreme version of things that you already know.

Now, it’s different. We live in a world where you can just talk to the computer and it will write the code for you, then you can give it feedback and it can adjust its behavior based on your feedback. I think it’s still hard to express a thing to a computer. You have to know what it is that you want, and you have to notice when you’re not getting what you want – that’s still really important. But the barrier to be able to start playing with this technology has dropped. To me, that is really interesting and exciting.

There’s a lot of conversation about vibe coding and whether AI is having a positive or negative impact on coding. Do you see opportunities in it?

I think it does a lot of the steps of what I convey in the book, which makes it so much easier for a person to specify their task. For example, they have all these computer files and they want to organize them in a particular way. They might not know how to say that in code, but they can vibe code it. So I think it’s incredibly powerful and useful, and I’ve been using it for all sorts of things myself. In some cases, it’s enabled me to start projects that I’ve been wanting to do for decades. I can just ask Claude Code and see what happens.

However, in some cases it’s a total disaster. I really wanted a program that would help me learn how to sight read – to be able to sing a song just by looking at music notation. I assembled a set of songs that I know already, then the computer could write them out in notes and I could see if I could recognize the whole song from the pattern. I vibe coded that with Claude Code and some of the examples worked great. Others, not so much.

One song I didn’t recognize, and the computer insisted it was Baa Baa Black Sheep and even played the song for me. I said that definitely isn’t Baa Baa Black Sheep, and the computer apologised for making it up! Why make it up, rather than using the real song?

So, it’s not the case that all problems are now solved, but it is the case that more problems are solved. We now have a really interesting opportunity, and I very much encourage people to play with it. It can give you the ability to create things that you maybe always wanted but didn’t have a way to reify, to make it real in the world.

What draws you to combine computer science, singing, science communication and all of that jazz?

I grew up watching this little video series called Schoolhouse Rock. They showed these adorable, catchy little songs, which were animated and explained different concepts. That is how I learned a bunch of things about parts of speech, such as the difference between an adverb and an adjective. That was really hard for me as a kid, but the songs just made it super clear. They addressed elements of American history. I learned the preamble to the US Constitution because they put it to music. I thought, I know the song, but I don’t know the Constitution – but wait, it’s the same thing.

So I saw the tremendous power of teaching through songs – they’re such an engaging mechanism for getting ideas across. I don’t know how to write songs, but I do know how to write words. I will typically take existing melodies from pop songs, and rewrite the words so that they’re about something that I want to try to teach or reinforce. For me, that’s mostly computer science and AI – I have a whole series of songs.

Has there been anything about communicating your work to non-specialists that’s changed the way you think about your research?

Yes, definitely. The direction that I took in the book was a result of a course that I was teaching – an introduction to computer science for non-computer scientists. I always marveled at the fact that different fields have different approaches to welcoming people into them. Most universities will have an introduction to psychology class where you can learn about the whole field of psychology, which can be used as an introduction to people who are going to study it. But if you just take that one course and then never do psychology again, you at least have this conceptual framework and a notion of what it is that psychologists do. I love that psychology, biology, and economics all do that.

In computer science, our attitude is that we don’t want to talk to you until you know how to program. Our introductory classes will just be about learning how to program because otherwise we can’t have any kind of meaningful conversation with students about our field. After that, there are all sorts of interesting things that we get into. We teach people about computation and what it means and how it relates to physics and to mathematics and to cognitive science. There’s all sorts of things that you can cover once students get through that initial barrier.

I think that’s a terrible model from a public relations perspective. Now people have to learn this very tough and arguably arcane act of writing computer code before we can even get to a point of telling them what it is that we do as computer scientists, why we think it’s important, why we think we deserve societal support and what we’re offering back to society. We only get one shot to express that, and we’re wasting our opportunity.

So I wanted to create an introductory class where we’re going to tell you everything that happens in computer science, and do a little programming because it’s hard to understand it without that. But the aim is not to leave this class as a programmer. The aim is to leave this class as a human being who now has an understanding of what computing is. My audience for those classes were people who were not computer scientists, nor planning to be. They were just interested in learning. That really gave me a flavor of what people bring to the conversation when they don’t come from the field.

How has the rise of AI influenced education and the way that you teach?

In my current role at the university I am 1/4 time professor and 3/4 time university administrator overseeing AI. AI has infiltrated itself into all of our classes in a really deep and empowering, but also incredibly disruptive, way. The way that we have been teaching doesn’t work when students have access to generative AI. It’s as if every student has the answers to all the homework questions and exam questions, and we’re just asking them to not look at them. But of course, that’s not really realistic. They feel like they have to look at it and they do, and then they don’t learn anything.

This is forcing everybody at the university to really take a step back and ask – what is education? It wasn’t homework, it wasn’t lectures – it was something else. What is it and how do we continue to have that? This is a challenging problem to solve, but the ways that people are responding to it are really interesting.
One angle is that perhaps we need to pay more attention to the process of people working through problems, and less about the answer at the end. With generative AI, students can get to the answer without going through the path of understanding. We have to find another way of leading people down that path to help their minds develop in a direction that is fruitful.

Earlier we were talking about the fact that our culture gives us all the opportunities to take the next step forward, and this is why people seem so smart. We don’t have to reinvent society from scratch every generation. The challenge of how we get that information into people’s heads is now in question. So yes, I think AI is having a profound effect on education, which is mostly disruptive. But having people re-evaluate what we are doing well and what we are not doing well in education is valuable regardless.

People are also leveraging AI to do some really creative things. Rather than a lecturer delivering to a common denominator, an AI system could potentially deliver information that’s tailored to each individual person. We’re already seeing some really good uses of that out in the world.

It’s just that it’s so early now that we don’t have guardrails yet – we don’t have the mechanisms in place that ensure good outcomes for when people are using AI to learn. There is all kinds of evidence that it does weird things to people’s heads. We don’t know what the 3rd order and 4th order effects of this are, or how to protect people against the negative ones yet. It’s a process and we’re trying to work this out, as a society.

I don’t like when it’s framed as professors versus students and that there’s antagonism. In all the conversations that I’ve had with students, I can tell that they’ve got skin in the game. This matters to them. They know that it’s not just like, “great, I can cheat now”, and they realize that that’s not going to give them any kind of competitive edge in life. They need to learn, to install those concepts into their minds, and they don’t know how to do that. So I don’t think that this is a case where professors figure it out and then impose it onto the students. I think this is something that we all have to work on together.

It sounds like a very difficult problem.

Yes, it’s a very hard place to be. I’ve spoken to professors who’ve been in the field for a really, really long time and who know what they’re doing. They have incredible instincts. When I ask a lot of them how they’re going to deal with generative AI, they say they’re going to retire now – even though we’ve never needed them more.

But the fact of the matter is that it’s an incredible amount of work for everybody to rebuild the structure of education from the ground up. Some people don’t have the energy for that anymore. I say that to acknowledge the fact that the people who are grappling with this are doing very important and difficult work.

And the way we approach it matters quite a lot. We don’t want to lose our skills because we rely on AI. AI offers tremendous opportunities, but I feel it removes a lot of our incentives – it allows us to be lazy. You can learn a lot better with AI, but you have to decide to use positive effort.

Have you found any solutions so far?

I am starting to see some really good ideas. Recently, we had a university level administrator retreat. I was asked to find someone to speak about AI, so I decided to throw everybody a curveball and brought in an art professor from down the hill. We have one of the world’s best art schools right next to Brown.

This art professor is addressing the question of how to educate artists. The way that he thinks about this is just fascinating because art school was already doing a lot of things like this. Art school is not about making the art and then hand in the art. Art school is about the creative process, and about the techniques and so forth that you bring to bear to eventually get to an artifact. The artifact is what you’re shooting at, but the educational process is focused on how you get there and not the end point. I think at the end of the day, artists are going to make what they want to make. The art school doesn’t care what they make, but they care a lot about the process by which they make it.

The professor that I invited to speak has been applying these ideas in the context of AI. AI can help you make the thing, but it’s not going to help you with the process. And in art school, they never cared that much about the thing anyway, only the process. I thought that there’s a lesson in that for us all.

People in the humanities create artifacts, like an essay or a position piece. People in computing create programs. People in the arts create an image or a sculpture. We’re all involved in that kind of creative process in one way or another. We should be figuring out how to teach people at that process level and not at the end effect level. I think when we engage people in that process, even if the AI is part of the process, forcing them to reflect on what’s happening and what all the steps are is really important.

For example, two other faculty members and I taught an experimental class last spring. We taught an introduction to computer science where they all had access to Claude Code. We told them which programs to write, but then they had to organize Claude to create those programs, and they had to be responsible for the end result.

It wasn’t as though the students could say, well this is what Claude did, so I suppose I’m done now. They had to ask – did this actually work? How did it work? Would this be extensible? Would this be secure? We took them through hypothetical evaluations of what they were doing. We were basically using art school techniques to teach programming. I think the students learned a really valuable lesson about what their role is in the creative process. We’re creating software engineers that understand what their role is in the process of making the software be real.

The problem is that it’s hard to scale this. This was a 20-person class, which is a similar class size to art school classes. In computer science, we teach classes to 300-400 students. I don’t know how that’s going to work.

I wonder if cross-disciplinary skills will become more important as AI lowers the barrier to entry for individual disciplines.

One thing that we’re discovering is that, in a typical startup, there are typically three people that work really closely together. There’s a product manager who has the vision of the entire artifact, a designer who is making it so, and an engineer who makes the innards work. Those three people have to work together really well to create good stuff, but the boundaries between those roles are evaporating. With the help of an agentic AI system, you can now have a single person that is the product manager, the designer, and the engineer.

We don’t know how to teach people to take on that role, but I think that’s the direction that we have to go in. Again, I’m doing an experimental class this coming spring where that’s explicitly what we’re going to do. We’re going to bring in people who aren’t necessarily computer programmers at all. It’s going to be some people who have a background in design, but not the technical side; some that are in the technical side, but not design; and some people who can think about product management. We will aim to help them so they can individually do all those roles in an integrated way with a singular vision.

Things are changing very fast! And you’ve witnessed many other technological changes throughout your career. You were involved in sending the first email attachment ever sent, which I found very amusing. How did that happen, and what’s it been like to watch these technologies emerge over time?

That’s a good question. Regarding the first email attachment, that happened because I was working at a company called Bellcore, which was the research and development arm for the phone companies in the US for a window of time. Because the role was to support all these different phone companies, a lot of people in the lab did work on standards, or how to make things interoperable across systems. And if different manufacturers make phones, they need to understand what that protocol is for one phone to make another phone ring.

At that time, the internet was really coming into its own. The email standard already existed, but what if we wanted to send richer emails, like emails that include images or sound or animation – what would that look like? That was a big thing then. We had a representative on an international committee to lay out what that standard should be. It was called MIME, which stands for Multipurpose Internet Mail Extensions. You want to have multimedia which is more than just ASCII text or letters. You want to be able to send images and sound and so forth, or even just formatted text. We needed a way for my mail program to send a message to your mail program, and for your mail program to make sense of it, even if it was developed by somebody else.

At Bellcore, I sang with Nathaniel Bornstein at a chorus called the Telephone Chords. I didn’t actually work with him technically, but he was helping to lead the committee and wanted to give an example of an email attachment he could send as a demonstration once they had finalized the format. He rewrote the words of a famous barbershop quartet song and put together a formatted text and an image of all of us singing, along with the audio of us singing his song. He combined them all together in a single email message and then sent it to the other members of the committee. That was the first time that there was content like that emailed from one place to the other.

I like the thought that went into the first email attachment, and that it was made memorable by singing.

I like to think Nathaniel was inspired by things like the first telegraph message, which was, “what hath God wrought?” And the first message transmitted from the moon, “one small step for man, one giant leap for mankind”. I think he was thinking that this is one for the ages, but he made it silly because I don’t know – it’s fun to be silly. Over the years, I’ve collected a list of these kinds of communication firsts. There’s a link there to the web page that has the photo and audio of the Telephone Chords that made up that first attachment.

Are you looking forward to the future to see what may come and what?

The future seems to be so close now, right? What is going to happen two years from now? Oh my gosh, I have no idea.

What am I looking forward to? I would like us to have a little bit more of a handle on what we’re doing and why we’re doing it. I like the idea that people could be more empowered by the existence of generative AI. I am unsettled by the idea that not only education but science and maybe even social structures are being undermined and rethought from the ground up. I’m hopeful that maybe we’ll come up with good things to replace the stuff that we’re discarding, but I worry.

I’m a fairly optimistic person, but I’m cautiously optimistic. There are some folks who believe that this is the beginning of utopia and that it is going to change everything – we’re going to have infinite resources, infinite knowledge, infinite flexibility. But I don’t know. People are still people. People need to process the world that they’re in, to absorb it in some way. It’s happening so fast that it’s really hard for people to absorb. It’d be great if we could just take a beat. Like the whole world goes on vacation and we just figure this out. But we have to do it while it’s moving, and that does concern me.

What do I look forward to in AI? I do think that there are questions that motivated the field early on that aren’t answered. Even though we have systems that can sort of do it, we don’t have an answer to the question that we asked. The question wasn’t just, can we have such a thing? We knew that we could to the extent that we believe that people are a kind of computer program, and not some kind of magical, mystical puppeteer from another dimension. If we’re basically programs, then we can write programs that do the things that people do.

That wasn’t the question. The question was how do you build intelligence? What is that really like? And are there things about it that we could improve and use to help people be more compassionate or more insightful or more flexible? I don’t think the existence of these chatbots has really answered these questions, but they’ve given us new ways to ask the question and new ways to pursue answering the question. And I’m excited to have people do that.

Here’s an analogy that I really like. When computer programs got really good at chess and backgammon, and ultimately when they got really good at Go, some people thought that we wouldn’t be interested in learning to play them anymore. But other people took it as an opportunity for us to study these games that we care about a lot at a level that was just unavailable to us before. And the result was that people play those games differently now because of what we learned from watching the computers play those games. They’re still fun for humans. It’s just that we have more insight into the structure of those games.

The AI programs that we have now, these chatbot programs, are clearly doing something internally that we’ve very long wanted to know how to design. Now we’ve got a program that can actually do it. We could study that and it could help us answer some of these questions – how do you make a computer program that can flexibly weigh evidence from different sources and come to a well-reasoned conclusion?

It might be in there. It might be in those programs. We just have to dive in. And so there’s an opportunity for us to just become much better at the things that we were trying to solve in the past, because we now have this new capability. So that’s something that I feel really excited about. We might finally figure out what we mean when we say something’s intelligent.

About Michael Littman

Michael L. Littman is University Professor of Computer Science and Associate Provost for Artificial Intelligence at Brown University. His research, which focuses on machine learning and decision-making under uncertainty, has been recognized with three best-paper awards and three influential paper awards. He received an outstanding educator award for his contributions to AI teaching and outreach. Littman is a Fellow of the Association for the Advancement of Artificial Intelligence, the American Academy of Arts and Sciences, and the Association for Computing Machinery. He recently served as Division Director for Information and Intelligent Systems at the National Science Foundation. His book, “Code to Joy: Why Everyone Should Learn a Little Programming”, was published in October 2023 by MIT Press. He chaired the panel that wrote the 2021 report for The One Hundred Year Study on Artificial Intelligence (AI100). He also co-chaired and co-authored the National AI R&D Strategic Plan (2023 update).

Ella Scallan is Assistant Editor for AIhub

Ella Scallan is Assistant Editor for AIhub