Rebuilding the brain with neuromorphic computing: an interview with Oliver Rhodes

The SpiNNaker supercomputer. Image credits: University of Manchester.

What can we learn from the brain about building better computers? Oliver Rhodes discusses how neuromorphic computing draws on the brain’s architecture to develop new approaches to more efficient information processing.

Can you give me an overview of your background and the research that you’re involved with?

Neuromorphic computing is quite a broad subject, which essentially looks to biology as inspiration to develop next-generation computing systems. We work off this principle: we know the brain is this really amazing computer, and it’s able to do things that a lot of modern computing systems aren’t able to do, and it also does them in an incredibly energy efficient way.

We’d like to try to replicate that with some of the systems we build. In the context of the current climate around AI, while we’ve seen that it’s made really big steps forward and is able to do really impressive things, there are certain things that it still can’t do that the brain is able to do. So we look to the brain for inspiration to try and solve some of those next generation challenges.

Neuromorphic computing covers all aspects of this: from looking at algorithms that might be solving a particular problem, to the systems and subsystems that would be running those algorithms, right down to a chip or devices level. We often use neural networks, which are also employed in artificial intelligence models, such as in deep learning and transformer models. We then go further to try to replicate the spiking neurons that are present in the brain to harness the additional efficiencies this can bring.

We are also looking at new ways we could match the performance of the brain – this could mean doing things extremely quickly. The brain can do things with low latency and it’s able to operate on a really low energy budget. It’s very, very efficient. It’s also able to integrate huge amounts of information simultaneously from lots of different modalities – vision/audio/touch/smell etc. You can do all those things combined on maybe three meals a day, and that gives you enough energy. Being able to harness those things is the area of neuromorphic computing.

Within the research community, neuromorphic systems are often employed from two perspectives: first, to do artificial intelligence type tasks, and secondly, to model neural systems, such as trying to simulate parts of the brain to help us to better understand it. This last part speaks to how the relationship between neuroscience and computing isn’t just one-way. While we’re using neuroscience to build better AI, we also want AI and neuromorphic computing to help us understand intelligence itself.

The International Centre for Neuromorphic Systems (ICNS) is a great place to explore these problems, bringing together researchers from across the neuromorphic stack, including chip designers, neuroscientists, and sensors and algorithms experts. The ICNS also builds on the neuromorphic computing heritage at Manchester, which was the site of the design and development of the SpiNNaker neuromorphic supercomputer.

So you’re trying to emulate the brain structure in the physical chip, as well as in the way that the algorithms are structured.

Yes, in short – I’ll explain more. First, a lot still isn’t understood about the brain, which means it’s not trivial to use it as a blueprint to build a system from. But there are things we do know and can use. The neural system, which you can think of as the algorithm that our brain is running, has evolved over millions and millions of years together with the wires and the cells and the actual hardware, where the hardware is our brain. So those two things have evolved together to be synergistic.

We work on a couple of principles. One core principle, from sensing right through to computing, is to replicate the spiking processing in neurons. This begins with the brain, so we capture the spikes in neural networks. Generally, when a neuron receives a spike, that’s a trigger for it to do something. If it doesn’t receive any spikes, it has nothing to respond to, so it doesn’t need to do anything. In our systems, we make use of that principle by putting processors to sleep for periods of no incoming spikes. If spikes are coming in, then we call that an event which would drive some type of activity, and we call this event-driven processing.

Another core principle is how we handle memory. Your standard computer has some RAM chips. All your data is stored in there, and when the processor wants to do some calculations, the data is shuffled into the processor from RAM, the calculations are done, and then the data is put back. But the brain doesn’t have a RAM area where it can load stuff from, compute things, and then put it back. All its storage is distributed. So all the different connections between neurons are tuned to allow the network to store different bits of information – that’s how we memorize things. It’s also how we have memories at different timescales. For example, you have a working memory of this conversation here, but you can also remember when you were a kid. All of these memories aren’t in some hard-drive – they’re stored in a distributed way. That’s a very key feature of these systems.

So, we talk about doing in-memory computing because we keep the information close to where it’s used. This is very important because it means you don’t have to move data around, which in turn saves a lot of energy. We build systems to try to harness this energy saving feature, but it places constraints on the types of algorithms that you can implement. Part of the research we do is looking at how to design the algorithms that would work within those hardware constraints.

Tell me a bit more about SpiNNaker.

SpiNNaker is a one million core supercomputer designed and built in the University of Manchester. The project was conceived and led by Steve Furber, who is an emeritus professor here. This was a 20-year effort to build a system that could model large-scale networks of spiking neurons on low power. The individual processing elements – the cores – are very small, very slow, low-power processes, the kind you might have in a mobile phone from the ‘90s. The system gets its capability by combining 1 million of them.

With these spiking neural networks, it’s not the individual neurons that take up a lot of the processing. It’s the communication between the neurons that is really challenging computationally. SpiNNaker was set up with this very neural-inspired routing architecture, which allows the transmission of these small packets of information that represent the spikes to any of the other 1 million processors in the system. This was very novel at its time, and it’s one of the core systems in the timeline of neuromorphic computing. This year SpiNNaker was recognized for the most prestigious award in the community, the Mahowald-Mead Prize for Neuromorphic Computing.

Recently, there’s been some further developments of it – there is a second generation version out now, SpiNNaker2. There are also industrial systems that are loosely inspired by it, or at least similar to it. For example, Intel’s Loihi system also follows a lot of the same principles. So SpiNNaker has made quite a significant contribution to the neuromorphic landscape, and we’re still building on this as a community.

How has the rapid progress in AI affected the field of neuromorphic computing?

Trying to build algorithms on top of neuromorphic systems is difficult because we don’t truly know enough yet about how the brain codes information. When I’m thinking of, for example, a banana, we don’t know exactly how that’s being represented in my brain. And so we can’t necessarily use all of those principles that the neural systems would be using. This has been where progress in AI has been helpful. We’ve made quite big advances in recent years, borrowing lessons from the machine learning community in how to train networks of spiking neurons in a rigorous way, using the same principles as we’ve seen in artificial neural networks.

I think one of the things that’s really helped AI technology become accessible and led to its massive boom is the way that you can code it very simply. There are high-level programming languages which make it very easy to build neural network models, such as the PyTorch package in Python. The people who produce PyTorch have very nicely abstracted away a lot of the low level tasks that you need to make it run really well, particularly on a GPU system. Researchers found that AI models are really well suited to the same operations that you would need to do to render your display, and so these graphical processing units, GPUs, are very popular for running these workloads. But your average machine learning programmer doesn’t have to go into the details of how to map their model onto the individual processing units of the GPU – this is taken care of for them. So there’s this really nice abstracted stack.

One of the challenges to disrupting that, which is what neuromorphic computing wants to do, is that there isn’t that mature stack there. We can build new chips, and theoretically they can run these types of workloads super efficiently. Then at a high level, we can write the models that we think could run on them. However, actually compiling it down onto the new systems is another big research area, which still isn’t mature. If you take that algorithm and run it one way on your chip, it might perform really badly. If you then tried to redistribute it on the same system, it could run much more efficiently or much faster. That kind of learning step is still going on in the neuromorphic community.

Do all of the well-known machine learning techniques work on neuromorphic systems?

It’s difficult to make an apples to apples comparison on these things. I wouldn’t say you could compare this to something like ChatGPT because the level of resources available to train those systems is different to what we have available in academia. There has been ongoing research into ways of doing different types of learning: whether that be reinforcement learning, whether that be online learning, which is something we look at a lot and I’ll explain now.

Your traditional AI model is going to be trained offline in a data centre somewhere, and it’s going to churn through huge stacks of data to extract lots of patterns before it would be deployed in practice. But we know that humans don’t do this – brains are able to carry out “online learning”. As they’re sensing things, they’re able to extract meaning from them. You don’t have to go and look at 10,000 pictures of a dog for your brain to learn what a dog is. You do it by interacting with the environment. And that’s something we actively looked at because it means we can design systems that you can’t keep shipping data back and forth to – for example, systems on a remote satellite.

This is a very active area of research and different algorithms have been proposed. However, their tendency is to capture what the traditional AI models would do, but in an online sense. So they’re still trying to replicate what the error backpropagation algorithm does, because it’s been shown to get results.

I think we are still some way from being able to say we could do reinforcement learning in the same way that a human or animal brain would be able to do, because we lack the data. There’s this inherent challenge of trying to understand how the brain is representing things or concepts with these spikes. That’s still an open question, but there’s a lot of potential there. Neuromorphic systems are able to show an advantage in terms of energy or latency, even when we’re simply replicating some of the existing algorithms from the machine learning community. So improving the algorithms these systems run should only extend that advantage.

This sounds like a very interdisciplinary field. Does progress in neuromorphic computing partly depend on progress in neuroscience?

Yes, so this is a good question, and I’ll start with my personal route into it. I used to work as a mechanical engineer doing computational modeling. Back in 2017, I found a way into the research group in Manchester working on SpiNNaker, by applying models to neurons instead of physical structures or aircraft wings.

One thing I was lucky with at the time was the Human Brain Project. This was a European initiative with over half a billion euros of funding over ten years, including from the EU and match funding from the project partners. Within the project, there were a lot of neuroscientists, computer architects, and even roboticists who were looking at embodying these kinds of models to learn more about them. It created the ideal proving ground for multidisciplinary research. For me that was a really good environment, and this kind of interdisciplinary exchange is something we still try to embrace. As a neuromorphic community, we’re very varied in our backgrounds – there tends to be a lot of physicists or neuroscientists, and also AI specialists. You need to have a broad skill set, and you need to be multidisciplinary minded to appreciate the different perspectives.

Back when I joined the field, people weren’t trying to do machine learning with spiking neural networks as much because there were challenges around implementing algorithms. At that time, there was a much more even balance between people trying to do neuroscience with these systems, and people trying to do AI with them. Around 2018 or 2019, researchers found a way to do error backpropagation and training. Suddenly, and then obviously with the popularity of AI, that’s taken the community massively towards the AI space and away from the neuroscience side of things. That is obviously now influencing the goals of new researchers in the area, and applications of neuromorphic technology.

However, there are still big challenges in the computational neuroscience field, in the sense that it’s difficult for researchers there to know if their models are working as they should, because it is still difficult to record neural activity at any kind of scale. In some sense, I think that’s why the community has gone more towards the deep learning route – it’s just more accessible at the moment. However, I suspect what will happen is that we might reach a ceiling with this and we might need to look back to neuroscience to make the next steps. Hopefully as neuroscience continues to evolve, it’ll give us new insights into how we could refine these systems.

By simulating parts of the brain, can neuromorphic computing help to progress neuroscience research?

One of the key goals of the SpiNNaker system was to help accelerate neuroscience. Within the Human Brain Project, one of our key milestones was that we were the first platform to be able to run a benchmark cortical model in real time, which was faster than anyone else had been able to run it. The model was built using neural parameters and connectivity measured from biology, and its output captured observed cortical dynamics. The model’s connectivity had proved a computational challenge for conventional HPC systems, as the large scale and sparse connectivity wasn’t a good fit for their underlying architectures. However with SpiNNaker, which was designed to support these kinds of neural workloads, we were able to run the model faster and more efficiently, even using this research system built on relatively old technology. I should note that those records have now been beaten because modern computers are continually evolving, but the improvements brought about by neuromorphic technology still stand.

Figure 1 – Cortical microcircuit model executed on the SpiNNaker system in real time: a) from left to right, model connectivity, overall structure, output spike recordings, and observed firing rates; b) SpiNNaker system, comprised of individual chips containing neural processing cores (top), and these chips combined into boards (bottom) which help the system scale to 1 million cores.

This new architectural paradigm we created showed the potential of conducting neuroscience simulations faster than ever before, using neuromorphic systems. This is important because it shows the potential benefit of these simulations. If we could get to the point where we could understand the mechanisms underpinning something like Alzheimer’s disease, then we could capture this in a model and tailor it to a patient at risk of the condition. If we could then run this simulation model faster than real time, we could look at how things might degenerate for that person over say ten years. And if it takes you ten years to run that simulation, it’s not a lot of help. But if you can run that in six months or even in hours, then you have this potential tool for helping understand and potentially treat these kinds of conditions.

This is an exciting area, but again, it’s in its infancy. We’re not quite at the point where your local doctor will be able to run a model like this, but we’d like to see things get to that point. We could also look at how a patient might respond to different treatments, or perhaps help to design new treatments – for example, things like deep brain stimulation for Parkinson’s disease. These are invasive procedures, so anything you can do beforehand to help understand how an individual might respond to that would be very useful. So this is an open research area and it is hard. At the moment, the majority of the funding is very AI focused, but I do think it’s important to keep these other aspects of neuromorphic systems alive so that we’re able to progress them when the neuroscience is ready, because they also have potential to make a huge societal impact.

What potential do you think neuromorphic computing will have – do you think it will eventually become mainstream, or have an explosion in progress like AI has seen?

I think so. But as I’ve mentioned, there’s this lack of maturity at the moment. This partly comes from this multidisciplinary nature where to make an advance, you need to bring together knowledge from all sorts of domains to solve the problem. Doing all of those things simultaneously is quite challenging. There aren’t millions of people doing this worldwide, and there isn’t this core knowledge that’s been pulled together. It feels like we go slowly relative to the AI community, which is going at an incredible pace due to the industry pull.

One area where we see there’s already commercially available technology is in sensing. Some of the most mature neuromorphic products are dynamic vision sensors, which are like cameras. When a normal camera takes a video, it takes a grid of pixels and stitches them together one by one to make an image frame, and these frames are then played consecutively to make a video. We know the human vision system doesn’t work like this – it just detects changes. If there is one ‘pixel’ in the visual field that has changed, such as going from dark to light, then only that change would be detected.

Researchers have built sensors that work in this way, so you don’t need to build an array to detect visual information. Instead you just look at where changes are occurring. The overall sensor can be much faster because it’s not trying to capture all this data in every step. And because the pixels don’t need to be related anymore, then it can have a higher dynamic range – so you can have a very bright area in one region and a very dark area in another; there doesn’t have to be some overall compromise to match that. For example, ICNS researchers have recordings from these cameras of a rocket taking off. If you view this with a normal camera, everything gets saturated by the rocket fuel igniting. With one of these cameras, you can see the rocket plume, but also still see things in the sky.

These neuromorphic vision sensors are commercially available and easy to set up. What we often see is people using them to record ‘neuromorphic data’, but then processing that data using conventional AI algorithms, such as artificial neural networks, because this works well and is more accessible than using neuromorphic chips. Personally, I’d like to see steps forward in the neuromorphic algorithms and processors for computing with this data, but at present these systems are at a lower technology readiness level. Therefore making advances in these areas is a priority for academic researchers.

This is still a hard problem, partly because the AI community is very competitive, and partly because we’re still trying to understand these systems. I think where we have the biggest potential is at the edge, so looking at a sensor and a processor that could go on your mobile phone. These days it’s going to take advantage of this improved sensing capability, but also the power efficiency that comes from the neuromorphic chip. That’s where we see the advantage of using a neuromorphic system rather than a small GPU, because you’re able to process things more efficiently.

And again, that comes back to the brain inspired principle. So we have our brain on the top of our head, near our eyes. which are one of our key sensors. But there’s a lot of processing that happens within the eye itself, and then as the signals pass from the eye into the brain. This kind of near sensor computing also has great potential, because you need to be able to process very fast data that’s coming in, and filter out only the important things that need to go to some higher level of processing.

I think this is where neuromorphic computing is trying to tackle some of the next generation challenges for the whole field of information processing systems. I think AI research has been enjoying collecting more data, building bigger and bigger models, and still being able to scale up GPU systems to handle the workloads. But we are going to reach a point where that’s no longer possible, and then being more efficient with how we sense things from our environments and what we pass on to the rest of the system, is going to start to become more important.

An example where we’ve been exploring these concepts is the Nimble AI project. This is an EU funded project exploring energy-efficient neuromorphic sensor-processing systems. We’re building neuromorphic vision sensors and enabling them with near-sensor computing engines.

We’re also trying to go even further with how we take inspiration from biology. We don’t have a uniform sensing array – instead we have a foveated sensor, which can sense in multiple resolutions. Our eyes have a dense set of receptors in the centre of our retina, but then towards the edges, these are much sparser – this is where our peripheral vision is. By concentrating resources in the centre of the eye, and then moving this around by moving the eye, it creates an efficient system for sensing the environment.

Figure 2 – Example of efficient ‘neuromorphic sensing’ from the NimbleAI project: left, shows detection of the car moving in a static background, with the dynamic vision sensor only outputing information capturing changes in the visual field. Right, foveation is used to sense the regions of interest in high resolution, with lower resolution used for the periphery.

With Nimble AI, we’re building a system that replicates that, by building a low-resolution sensor, with a high resolution sensing region which can be steered around the field of view to regions of interest. In Manchester, we’ve looked at how to steer this high-resolution region within the visual field based on the live incoming visual data, with all these technologies I’ve been talking about. So we explored algorithms to be able to take the incoming visual data and to predict which was the most interesting part for us to focus on. We’ve also built a small form factor hardware accelerator that can run that algorithm really efficiently, a neuromorphic engine implementing the algorithm. From there, we can steer this region of interest with very low latency and very low power, so the sensor can adapt in real time to what it’s seeing.

One of the applications of this work is smart glasses, which are becoming increasingly popular, allowing users to view content and interact with a computer using glasses and their eye movements. With these smart glasses, a design goal is often to track where someone is looking. That can be useful for psychological experiments, medical diagnosis, and interacting with a system – for example, in virtual or augmented reality. Being able to detect whether someone is looking at a particular area or at the real world around them is valuable. You also don’t want the glasses to be bulky, so being able to do this in a small form factor with low energy consumption is a great application with significant potential.

What future work are you planning in this area?

We work a lot on algorithms at the moment. Over the last decade, there have been quite a lot of chips developed, including very low-power edge systems, but also large-scale data centre-style systems. We’ve made good progress in that area. The next breakthrough the community needs is probably on the algorithmic side, to realise the benefits of these hardware systems. We need to step back and look at how these approaches fit with traditional machine learning, and how we can better exploit what neural systems can do. That includes doing things online, taking on new information rapidly instead of learning from large datasets, and combining different modalities.

Your brain does this all the time – for example, right now you’re listening to me, but you’re also watching my mouth move. Your brain uses all of those signals to check what you’re hearing. There’s potential to build better performing systems by doing the same thing with AI. The challenge is that as soon as you add another modality to a system, the power and computational requirements typically increase. We want to keep these low, so eventually these systems could be implemented on a mobile device. That’s one of the areas we work on.

Across the wider group in Manchester, we also work on next-generation sensing. I mentioned event cameras, but other researchers in the group have deployed those cameras on the International Space Station. They’re using different types of data to understand where future applications of the technology could be.

Oliver Rhodes is Senior Lecturer in Bio-Inspired Computing within the International Centre for Neuromorphic Systems at the University of Manchester, UK. His research focuses on development and application of neuromorphic computing, taking inspiration from the brain to develop next-generation information processing algorithms and hardware. This work explores the cross-over of machine learning, computer architecture, and computational neuroscience, with the goals of furthering our understanding of the brain and developing energy efficient computing.

Ella Scallan is Assistant Editor for AIhub

Ella Scallan is Assistant Editor for AIhub