A Digital Brain
Other smart people are having similar ideas that I am and I want to ride on their coat tails.
Not two days did this article sit in my drafts folder before I found this paper on Arxiv:
From the above paper’s introduction:
Biological nervous systems are created in a fundamentally different way than current artificial neural networks. Despite its impressive results in a variety of different domains, deep learning often requires considerable engineering effort to design high-performing neural architectures. By contrast, biological nervous systems are grown through a dynamic self organizing process.
This is something that’s been on my mind lately.
After my research spike on using LLM’s to try and teach, I had a conversation with a friend about what we were both working on. I noticed in his Github he had published some work researching RAGs. We eventually got to talking about ideas we had in the field of AI and neural-nets in general, and I shared with him some thinking around training neural nets in a more biological inspired way.
Ok, so to be honest, the paper I linked above is different in that they refer to biological influences on a running net. My thoughts were specifically in training, but we both shared the ideas of growing the network in a way that mimics biological neuron growth more closely.
For the sake of posterity with no defense of this idea at all, I was thinking that if you changed the graph of your neurons during training and back-propagation, perhaps as part of the gradient descent (does that even make sense?), then you might see a different kind of performance. Maybe with a lower computation and horrendously high VRAM requirement?
Honestly I didn’t even want to write this post right now. I wanted to pause and do some research and build a prototype. I wanted to make sure I knew what I was talking about before I started talking.
I did start building a prototype, and have been iterating on it for a week and a half now.
I also started researching. Luckily there’s a steady influx of new information and huge audience doing similar things to myself on Hacker News. Lo and behold a few days ago, the above article was shared and I discovered it’s existence. I also discovered the existence of other research on my exact idea, linked as references in the above paper.
Do I know what I’m talking about yet? I still don’t think so. The fact that others are thinking similar things (others that are way smarter than I am) though is a validation. So.
Yolo.
Full send on this article.
This is where my prototype is right at this second:
I’m slowly building out my original ideas in a visible and observable way. Soon I’ll get to the business end of the neural net and have to implement proper back propagation and some kind of search (probably gradient descent since I know that word).
At that point, I’ll go back to researching and reading the papers I’ve slowly been hoarding. I’ll transform all the matrix math and statistics stuff into visualizable structures and start working these ideas in.
Since I try to make most of my posts contain some kind of nugget of wisdom, I’ll close with this: if you have an idea about something outside your domain of expertise, or are asking questions nobody else seems to be asking, don’t immediately dismiss yourself. I got into the habit of dismissing these thoughts as ignorant fantasy. Surely the smart people who are devoting their lives to this would have thought of this, if it was sane. As I’m discovering, maybe not! Maybe your idea or question has merit.
Then go do some science.