Introduction
At the start of this series, in Part 1: The artificial neuron, I introduced the concept of the simple artificial neuron. Here’s a quick reminder of what that model looked like::
Simple though it is, this basic model is capable of some impressive things when combined to form a neural network. Even the simplest of these networks can learn to control robotic arms and copy a person’s musical, film or literature preferences. Recent developments have even seen this basic artificial neuron produce sophisticated text and images when used in Large Language Models and Generative AI systems. However, compared to even the simplest of biological neurons, it’s a minnow.
To explain this I need to reintroduce you to the biological neuron and explain some of its hidden complexities. Then I’ll show you how I’ve evolved the basic artificial neuron into something more biologically plausible, whilst keeping it relatively light in terms of processing requirements.
Biological plausibility
Bio what?
“Biological plausibility” is a term you’ll come across a lot in the world of artificial intelligence. It’s a fancy way of saying “similar to what actually happens in a real brain”. The reason it’s important is because one of the aims of AI is to reproduce intelligence in a machine. Not just the appearance of intelligence, but actual intelligence.
The hypothesis is that we’re much more likely to achieve that if we use mechanisms that occur in real brains. When our artificial neurons do things based on what we see in real neurons, we call this biological plausibility.
In Part 3 – Teaching the machine, I described two different ways of training neural networks. The first of these, back-propagation, has its roots in mathematics, and is not considered to be biologically plausible. This is because it relies on an error signal being calculated at the final outputs of the network. This error signal then needs to be fed back through all the preceding layers of neurons. There is no such mechanism in the brain to support this approach, and so it can’t possibly be how the brain learns.
The second, Hebbian learning, arises as a direct result of observations of real neurons. It happens locally within a single neuron and relies only on the relationship between the neuron’s inputs and its output. Within the biological neuron, the output does ripple back to the synapses, in the form of a back-propagated action potential (BAP). This BAP can be measured and changes to synaptic strengths can be observed when this happens. Hebbian learning can therefore be considered to be biologically plausible.
Dendritic trees
However, there’s more to biological plausibility than just the learning approach. The simple artificial neuron assumes all inputs connect directly to the centre of the neuron (known as the soma). Here, they are added up as a single set and an output is produced. In a biological neuron, synapses cluster together on tree-like branches known as dendrites. Each segment of these dendrites then sends a combined output from its cluster of synapses onward to the soma. This biological insight can be confusing, but it also helps us with a problem that arises in artificial neural networks.
When you experiment with artificial neurons, you quickly discover there’s a limit to how many inputs you can give a single neuron before it stops adding any real value. That number, in my experience, is roughly 10, but you can stretch it to around 50 with a bit of tuning. That sounds like a decent number at first glance, until you compare it with a real neuron. In biological systems, the number of inputs is in the thousands, with typical numbers being between 2000 and 6000.
That’s quite a difference, but it starts to make sense when you look more closely at what’s going on. All those synapses providing inputs to a neuron connect to it via structures called dendrites. These dendrites spread out like microscopic trees and the synapses connect to their branches. The dendrites are made up of many segments, and each of these segments contains a small group of synapses. (Wikipedia provides a good introduction to the structure of a biological neuron if you’re interested).
It turns out that a typical neuron has around 100 of these dendrite segments and each of these segments accommodates roughly 10 to 50 synapses. This level of connectivity for one segment correlates nicely with the number of inputs an artificial neuron can deal with effectively. Even more interesting is that these segments show “firing” behaviours similar to the core of the neuron itself (the soma).
Long story short
This is starting to get complicated, so I’m going to skip some details and jump to a conclusion. I’ve created simulations of biologically plausible neuron models that incorporate features such as dendrite and soma spiking coupled with calcium influx synaptic plasticity. These are concepts better covered by research papers, so I won’t explain them here. If you’re interested in knowing more, trust me when I say, the rabbit hole is very very deep. (I’ve been there.)
Suffice to say that these simulations demonstrate that each dendrite segment in a complex neuron behaves in a similar way to the simple artificial neuron model. The neuron itself also behaves as a simple artificial neuron, with the outputs from the dendrite segments acting as its inputs. Furthermore, the overall output of the neuron propagates back to the dendrites and participates in the local learning process. This allows the dendrite segments to compete with each other to form an auto-associative memory (see Part 4 – The machine that teaches itself).
The end result is that just one neuron can learn, without external reward, to break a set of input patterns into smaller features involving a subset of the inputs. With external reward, it can then learn to respond to specific features and ignore others. The more patterns you present to this neuron, the more features it identifies, and the fewer dendrite segments are used to represent each feature. This “sparse encoding” matches the behaviour seen by researchers when they observe real neurons in action.
The result of all this complexity is that just one such neuron is the equivalent of around 100 simple artificial neurons, working together as an auto associative network with a bit of Hebbian learning added on for good measure. Isn’t nature wonderful!
Simplifying the complexity
Although my early simulations of biological neurons worked, they were very processor intensive. Spiking behaviour required millisecond time steps and integration, and calcium influx required multiple calculations. In the interests of efficiency, I took the results from my simulations and encapsulated them in a (relatively) simplified model, as shown below:
This enriched model takes the simple artificial neuron and adapts it slightly to emulate the behaviour of a single dendrite segment. The input weights to each segment represent the strengths of the connecting synapses. The core of the neuron, the soma, is represented by another simple artificial neuron. The dendrite segments then feed into this single simple artificial neuron through an additional set of “internal” weights. Experiments have shown that dendrite branches adjust their strength in a similar manner to synapses so there is biological precedent for this feature of the model.
There is one slight difference in the artificial neurons representing the dendrite segments. The threshold function is no longer symmetrical. Instead of a sigmoid, this function is linear for the first half of the input range and then jumps to fully on for the second half of the input range. Hence for X < 0.5, output Z = X, but for X > 0.5, output Z = 1.0. This emulates the slightly different summing and firing behaviour seen in dendrites compared to the soma.
Competitive learning
To support this new model, we also need a slightly modified learning approach. Learning for the soma is still Hebbian in nature (see Part 3 – Teaching the machine), but learning for the dendrite segments includes an extra term.
Soma learning: Change in weight = reward x Y x Z x learning rateDendrite learning: Change in weight = X x Y x (Y - Z) x learning rateThe effect of this extra term, Y – Z, is to ensure that when the neuron is firing strongly, the weights for the strongest firing dendrite segments are reinforced while all other weights are reduced in strength. This competitive learning results in behaviour very similar to that demonstrated by the auto-associative network described in Part 4 – The Machine that Teaches Itself.
So, in conclusion, this simplified model achieves the same outcome as my more complex “spiking” model, but at a far lower computational cost.
The gift that keeps on giving
So, now we have an artificial neuron that is just a little bit closer to its biological equivalent. Next, we need to tackle the bigger picture; the brain itself. In previous posts I’ve described feed-forward and recurrent artificial neural networks, but I haven’t touched on how those match up to the organisation of biological brains.
That’s where I intend to go in the next but one post, by looking at the layers of the brain and what each of those layers might be doing for us. I say “might” because, despite all the research, there is no clear consensus about how the brain achieves its incredible feats, let alone how reasoning and awareness arise from that squashy organ.
But before that, I intend to come back to a practical use of simple feed-forward networks that is when trying to understand the latest large language models. The thing I’m going to try to explain helps us turn language into numbers suitable for training machines, and also gives a hint about what might be going on in the first input layer of the brain. It’s called an “embeddings model” and to explain it, I’m going to describe how I used it to get a neural network to “understand” the universal language of music.