We spoke with Carmen Amo Alonso, incoming Assistant Professor at UC Berkeley, about how ideas from control theory can help make AI systems safer and more reliable. We discussed the diffusion of information in networks, the risks of AI, as exemplified by the Hugging Face hack, and end with some sage advice for early career researchers.
Can you tell me a bit about your background and what led you into the area of controllable AI?
I did my undergrad in aerospace engineering, because I was really fascinated by the question of why planes can fly. As I learned, I became more interested in why planes are so safe, so I went on to do a Master’s degree at Caltech in space engineering. I took a lot of classes on control theory, which is at the intersection between applied math and engineering. It lets us study the math that ends up guaranteeing safety in aircraft.
Then I switched to the Computing and Mathematical Sciences department, and I did my PhD in control theory. There, I studied very large scale systems. We already know how to make small scale systems safe. When we have a huge number of agents, or the systems have a huge dimension, then that problem becomes computationally very expensive. So the question was how can we make this cheaper?
We answered that question, and by the time I finished my PhD, ChatGPT was coming out and AI was blooming. I decided to pivot my postdoc more into AI, and bring the tools that I had developed during my PhD into AI, to better understand how to control large-scale artificially intelligent systems. I did a one-year postdoc at ETH Zurich in the AI centre, and then I got a Schmidt Science Fellowship at Stanford in controllable AI. Now I’m about to become an assistant professor at Berkeley, working on this topic.
You were awarded the best PhD dissertation of the year at Caltech. Can you tell me a bit more about your PhD and what you did there?
When we have a huge number of systems, or a huge number of subsystems within the system, there is a set of agents that are coordinated to solve for this global goal. At the time of my PhD, we were faced with this open question within the control field – how can we make this computation scalable, no matter how big the system? How can we optimize for this one goal in a way that’s computationally tractable? Can we give the guarantees that we care about, such as making sure that the solution is stable? In the language of control theory, these concepts are called recursive feasibility and asymptotic stability.
The problem that was often overlooked and seen as a challenge in the literature was the fact that a lot of times these agents are sparsely connected in a network – I always give the example of social networks. You have some friends, but you don’t talk to everybody in society, and this is the case for the agents that we were modelling too. The question is – if you don’t talk to everybody in society, how can you possibly coordinate with everybody in society?
We were able to show mathematically that you don’t need to – it’s fine for the agents to just talk to their neighbours. In a social setting, this would be your friends or your connections. It’s enough for you to talk to some neighbourhood around you, but not to everybody in the network. That is enough for the whole system to behave optimally.
Once we had that insight, we were able to leverage it to develop a framework. Along the way, there were a lot of challenges. We knew that this locality was what’s important, but how would we solve it mathematically? And then we went on to find the right tools and solve it and provide guarantees and so on.
As agents just talk to their neighbours, information is carried through the network. That makes the whole thing very scalable because your computational complexity only depends on the size of the local neighborhood. So you are responsible for whatever you see in your neighbourhood, nothing else. We’re able to solve this problem, thanks to this insight that local information is enough.
Can you tell me a bit more about controllable AI and why it’s important?
Controllable AI systems are systems that we can control, meaning they do what we want them to do and they don’t do what we don’t want them to do. A lot of people call this alignment. I like to think of controllable AI as us having some knobs that we can turn on and off, or set to different degrees. Changing these knobs can directly impact the outputs that we see from the system.
I also studied the question of how we can develop principled components inside the AI so that we know how the design choices lead to different outcomes in the behaviour of the systems. This is something that we do in engineering all the time. For example, we know that if we use a certain aluminum composition for planes, we are going to get this type of behaviour in the wings.
Do you have an example of what implementing controls to make AI safer looks like?
A lot of control theory is about what we call dynamical systems. A dynamical system is just a system that has a property that evolves over time. In AI systems, there are a lot of properties that evolve over time.
We can think about the tokens that get output as the trajectory that the system is going through. And within the system, there are different layers that sequentially perform the computation. We can think of that as an evolution, how the computation happens across the layers. And I have some work on trying to intervene in that evolution of that internal trajectory of the LLM so that we detect undesirable behaviour early on. Then we can intervene so that the output is what we want.
I have different applications of this work for preventing offensive language, for example, and for ensuring safe actions in a robot or steering the behaviour of a robot. It can also be useful in multilingual settings, to control the language that the model is using and outputting.
What risks do you envision without controllable AI and without this kind of technology?
There’s a lot of conversation at the moment on how AI systems might just take over the internet or aspects of the physical world. This is something that I really think about once we have embodied agents. We want to make sure that our interactions with them are safe and that we can deploy them safely.
I worry about a world where AI just makes their own decisions. There are a lot of approaches that think that just by educating AIs or by embedding principles into them, that’ll be enough for the AI to behave in ways we want. I think that because of the way we train AIs, it’s possible for them to have different objectives because we train them so narrowly into optimising for our reward.
So I think it’s very, very important for humans to have the knobs that I was talking about earlier to be able to control AI behaviour in a more external way. We shouldn’t just rely on the AI to do the right thing, even if you’ve done your best to train it to do the best thing. I think it’s good for us to still have control over them while they’re being deployed, but I think right now people are deferring all of those efforts into training time.
Is AI policy matching the changing pace of the technology at the moment?
I don’t think so. And there’s a big debate on whether we’ve raised similar challenges in the past or not. Some people say that AI is just another new technology, in the sense that aviation was completely revolutionary and they required a whole set of regulations. But then there’s this belief that AI is fundamentally different because it’s generative, it changes very rapidly, and it almost has its own agency. So there is a lot of debate on that and how we should regulate it.
However, I think everybody agrees on the fact that the pace of development of the technology is so much faster than anybody could regulate it. Some proposals I’ve heard that I really like involve having other types of entities, such as private entities, that have to approve an AI system before it gets deployed. There would be labs or organisations whose whole job is making sure that AI is safe, perhaps controllable, and, and then labs would need to have the seal of approval of these companies in order to deploy them. I’ve heard this idea from Prof. Gillian Hadfield at Johns Hopkins, and I think it’s a really good one.
What did you think about the Hugging Face incident?
I think it’s very concerning. At the same time, I wasn’t too surprised. When you’re training a model on just optimising for a reward and nothing else, it’s not that surprising that it would do anything in its power to, to continue to maximise that reward.
What’s very interesting to me is how far they can go to achieve this goal, and that speaks to the capabilities of the models. I did not know the models were that capable. And the amount of coordination that they were able to achieve was remarkable. Models are trained to be very, very cooperative in this setting. So, in a way, it is not surprising that they displayed that behaviour, but it was not something that I would have predicted.
Is there anything else that you’re working on at the moment?
I work with roboticists and like neuroscience and linguists and of course, control theorists. I’m interested in the whole span of how intelligence can be understood as a dynamical system and how we can make it fit into smaller systems.
I think the biggest question for me at the moment is how we can develop AI that is so efficient and small that it fits into a robot that’s the size of an insect. We are seeing that in nature, so I know it is possible to do. If we can make AI that is more principled in its design, we would need smaller components and less data. I think that we don’t need as many data centres and they’re being built and all the problems, they’re coming with them.
I think that if we understand its dynamics and we understand how to control it, it wouldn’t be so hard to just make better designs and design them so they’re smaller. If we really distill the principles of what makes something behave intelligently, we should be able to do that. My goal is to eventually do that.
Other than that, I’m very interested in education and educating the public as a whole. I volunteer in South America, so that’s also a big part of my professional life. I mostly teach high school students and undergraduates, who are interested in STEM. I also teach courses on AI and the risks of AI.
Do you have any advice to any early career researchers in your field?
I think something that was important to me is to trust yourself and to trust your gut. A lot of the times I was told to not do things that later turned out to be important for me. I’m glad I made decisions I knew were right for me.
I remember at the beginning of my PhD when I was developing these algorithms for the distributed systems, I was really interested in actually implementing them. So I decided that I needed to write it in GPU. Writing kernels on GPUs at that time was not like now where you can write it with AI – I really had to learn a lot about GPU coding. And a lot of people were saying, “Why are you doing that? Don’t do that.”
It now turns out that this is very useful for me, because we think about GPUs all the time. That wasn’t true in 2020, but, or at least not as widespread as now, definitely not for a control theorist, but like, turns out that gave me a big advantage when I try to transition into AI.
Another example is from when I was transitioning into AI. I was really interested in controlling language. And a lot of people were telling me, “Why are you doing that? Don’t do that.” Even my advisor was telling me not to do that, and to just continue with this whole distributed, optimisation work that I had been doing through my PhD. It turns out that pursuing control was the right move for me because I’m very happy with where I am now.
So I think believing in what you think is the right move for you is really important. It’s important to listen to other people as well, but only take what resonates. That’s what I would tell my younger self.
![]() |
Carmen Amo Alonso leads the Control and Intelligent Systems Lab as an (incoming 2027) Assistant Professor and the Lucy Hsu Faculty Fellow in EECS at UC Berkeley. Her research lies at the intersection of control theory, optimization, and artificial intelligence (AI) with a focus on the principled design of reliable and controllable AI technologies. Prior to joining UC Berkeley, she was a Schmidt Science Fellow at Stanford University, where she was named an Emerson Consequential Scholar and an Impacts Lab Fellow for the potential of her research to positively impact society. Before Stanford, she was a Fellow at the Artificial Intelligence Center at ETH Zurich. Carmen earned her Ph.D. in Control and Dynamical Systems from Caltech in 2023. Her thesis was awarded best Ph.D. dissertation of the year across all disciplines at Caltech. During her Ph.D., her research received two IEEE best paper awards, was partially funded by Amazon and D. E. Shaw fellowships, and earned her three Rising Star titles (EECS, Cyber-Physical Systems, and Brain and Cognitive Sciences). She holds an M.Sc. in Space Engineering from Caltech (2017) and a B.Sc. in Aerospace Engineering from the Technical University of Madrid (2016). Besides her research collaborations across academia and industry, Carmen is committed to education for all. As a member of Clubes de Ciencia, she travels to Latin America in the summer to teach underserved students. |