![\"\"](\"https:\/\/jonayre.uk\/blog\/wp-content\/uploads\/2024\/10\/Understanding-neural-networks-Part-5_-Enhancing-the-artificial-neuron-1-1.png\")
Simple though it is, this basic model is capable of some impressive things when combined to form a neural network. Even the simplest of these networks can learn to control robotic arms and copy a person\u2019s musical, film or literature preferences. Recent developments have even seen this basic artificial neuron produce sophisticated text and images when used in Large Language Models and Generative AI systems. However, compared to even the simplest of biological neurons, it\u2019s a minnow.<\/p>\n\n\n\n
To explain this I need to reintroduce you to the biological neuron and explain some of its hidden complexities. Then I\u2019ll show you how I\u2019ve evolved the basic artificial neuron into something more biologically plausible, whilst keeping it relatively light in terms of processing requirements.<\/p>\n\n\n\n
Biological plausibility<\/h2>\n\n\n\n
Bio what? <\/p>\n\n\n\n
\u201cBiological plausibility\u201d is a term you\u2019ll come across a lot in the world of artificial intelligence. It\u2019s a fancy way of saying \u201csimilar to what actually happens in a real brain\u201d. The reason it\u2019s important is because one of the aims of AI is to reproduce intelligence in a machine. Not just the appearance of intelligence, but actual intelligence.<\/p>\n\n\n\n
The hypothesis is that we\u2019re much more likely to achieve that if we use mechanisms that occur in real brains. When our artificial neurons do things based on what we see in real neurons, we call this biological plausibility. <\/p>\n\n\n\n
In Part 3 – Teaching the machine<\/a>, I described two different ways of training neural networks. The first of these, back-propagation, has its roots in mathematics, and is not considered to be biologically plausible. This is because it relies on an error signal being calculated at the final outputs of the network. This error signal then needs to be fed back through all the preceding layers of neurons. There is no such mechanism in the brain to support this approach, and so it can\u2019t possibly be how the brain learns. <\/p>\n\n\n\n The second, Hebbian learning, arises as a direct result of observations of real neurons. It happens locally within a single neuron and relies only on the relationship between the neuron\u2019s inputs and its output. Within the biological neuron, the output does ripple back to the synapses, in the form of a back-propagated action potential (BAP). This BAP can be measured and changes to synaptic strengths can be observed when this happens. Hebbian learning can therefore be considered to be biologically plausible.<\/p>\n\n\n\n However, there\u2019s more to biological plausibility than just the learning approach. The simple artificial neuron assumes all inputs connect directly to the centre of the neuron (known as the soma). Here, they are added up as a single set and an output is produced. In a biological neuron, synapses cluster together on tree-like branches known as dendrites. Each segment of these dendrites then sends a combined output from its cluster of synapses onward to the soma. This biological insight can be confusing, but it also helps us with a problem that arises in artificial neural networks. <\/p>\n\n\n\n When you experiment with artificial neurons, you quickly discover there\u2019s a limit to how many inputs you can give a single neuron before it stops adding any real value. That number, in my experience, is roughly 10, but you can stretch it to around 50 with a bit of tuning. That sounds like a decent number at first glance, until you compare it with a real neuron. In biological systems, the number of inputs is in the thousands, with typical numbers being between 2000 and 6000. <\/p>\n\n\n\n That\u2019s quite a difference, but it starts to make sense when you look more closely at what\u2019s going on. All those synapses providing inputs to a neuron connect to it via structures called dendrites. These dendrites spread out like microscopic trees and the synapses connect to their branches. The dendrites are made up of many segments, and each of these segments contains a small group of synapses. (Wikipedia provides a good introduction to the structure of a biological neuron<\/a> if you\u2019re interested).<\/p>\n\n\n\n It turns out that a typical neuron has around 100 of these dendrite segments and each of these segments accommodates roughly 10 to 50 synapses. This level of connectivity for one segment correlates nicely with the number of inputs an artificial neuron can deal with effectively. Even more interesting is that these segments show \u201cfiring\u201d behaviours similar to the core of the neuron itself (the soma).<\/p>\n\n\n\n This is starting to get complicated, so I\u2019m going to skip some details and jump to a conclusion. I\u2019ve created simulations of biologically plausible neuron models that incorporate features such as dendrite and soma spiking coupled with calcium influx synaptic plasticity. These are concepts better covered by research papers, so I won\u2019t explain them here. If you\u2019re interested in knowing more, trust me when I say, the rabbit hole is very very deep. (I\u2019ve been there.)<\/p>\n\n\n\n Suffice to say that these simulations demonstrate that each dendrite segment in a complex neuron behaves in a similar way to the simple artificial neuron model. The neuron itself also behaves as a simple artificial neuron, with the outputs from the dendrite segments acting as its inputs. Furthermore, the overall output of the neuron propagates back to the dendrites and participates in the local learning process. This allows the dendrite segments to compete with each other to form an auto-associative memory (see Part 4 – The machine that teaches itself<\/a>).<\/p>\n\n\n\n The end result is that just one neuron can learn, without external reward, to break a set of input patterns into smaller features involving a subset of the inputs. With external reward, it can then learn to respond to specific features and ignore others. The more patterns you present to this neuron, the more features it identifies, and the fewer dendrite segments are used to represent each feature. This \u201csparse encoding\u201d matches the behaviour seen by researchers when they observe real neurons in action.<\/p>\n\n\n\n The result of all this complexity is that just one such neuron is the equivalent of around 100 simple artificial neurons, working together as an auto associative network with a bit of Hebbian learning added on for good measure. Isn\u2019t nature wonderful!<\/p>\n\n\n\n Although my early simulations of biological neurons worked, they were very processor intensive. Spiking behaviour required millisecond time steps and integration, and calcium influx required multiple calculations. In the interests of efficiency, I took the results from my simulations and encapsulated them in a (relatively) simplified model, as shown below:<\/p>\n\n\n\n This enriched model takes the simple artificial neuron and adapts it slightly to emulate the behaviour of a single dendrite segment. The input weights to each segment represent the strengths of the connecting synapses. The core of the neuron, the soma, is represented by another simple artificial neuron. The dendrite segments then feed into this single simple artificial neuron through an additional set of \u201cinternal\u201d weights. Experiments have shown that dendrite branches adjust their strength in a similar manner to synapses so there is biological precedent for this feature of the model.<\/p>\n\n\n\n There is one slight difference in the artificial neurons representing the dendrite segments. The threshold function is no longer symmetrical. Instead of a sigmoid, this function is linear for the first half of the input range and then jumps to fully on for the second half of the input range. Hence for X < 0.5, output Z = X, but for X > 0.5, output Z = 1.0. This emulates the slightly different summing and firing behaviour seen in dendrites compared to the soma.<\/p>\n\n\n\n To support this new model, we also need a slightly modified learning approach. Learning for the soma is still Hebbian in nature (see Part 3 – Teaching the machine<\/a>), but learning for the dendrite segments includes an extra term.<\/p>\n\n\n\n The effect of this extra term, Y – Z, is to ensure that when the neuron is firing strongly, the weights for the strongest firing dendrite segments are reinforced while all other weights are reduced in strength. This competitive learning results in behaviour very similar to that demonstrated by the auto-associative network described in Part 4 – The Machine that Teaches Itself<\/a>. <\/p>\n\n\n\n So, in conclusion, this simplified model achieves the same outcome as my more complex \u201cspiking\u201d model, but at a far lower computational cost.<\/p>\n\n\n\n So, now we have an artificial neuron that is just a little bit closer to its biological equivalent. Next, we need to tackle the bigger picture; the brain itself. In previous posts I\u2019ve described feed-forward and recurrent artificial neural networks, but I haven\u2019t touched on how those match up to the organisation of biological brains.<\/p>\n\n\n\n That\u2019s where I intend to go in the next but one post, by looking at the layers of the brain and what each of those layers might be doing for us. I say \u201cmight\u201d because, despite all the research, there is no clear consensus about how the brain achieves its incredible feats, let alone how reasoning and awareness arise from that squashy organ.<\/p>\n\n\n\n But before that, I intend to come back to a practical use of simple feed-forward networks that is when trying to understand the latest large language models. The thing I’m going to try to explain helps us turn language into numbers suitable for training machines, and also gives a hint about what might be going on in the first input layer of the brain. It’s called an “embeddings model” and to explain it, I’m going to describe how I used it to get a neural network to “understand” the universal language of music.<\/p>\n","protected":false},"excerpt":{"rendered":" The artificial neuron used in most neural networks is just a speck in comparison to even the simplest of biological neurons. In Part 5, I introduce a more sophisticated model of my own making. It\u2019s time to step it up a bit and get more brain-like.<\/p>\n","protected":false},"author":1,"featured_media":333,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-332","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-business"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\tDendritic trees<\/h2>\n\n\n\n
Long story short<\/h2>\n\n\n\n
Simplifying the complexity<\/h2>\n\n\n\n
![\"\"](\"https:\/\/jonayre.uk\/blog\/wp-content\/uploads\/2024\/12\/Understanding-neural-networks-Part-5_-Enhancing-the-artificial-neuron-3.png\")
Competitive learning<\/h2>\n\n\n\n
Soma learning: Change in weight = reward x Y x Z x learning rate<\/code><\/pre>\n\n\n\nDendrite learning: Change in weight = X x Y x (Y - Z) x learning rate<\/code><\/pre>\n\n\n\nThe gift that keeps on giving<\/h2>\n\n\n\n