I have an announcement: i discovered that what i re-invented was already thought in a similar way and published by A.Graves of Google, with the name of ACT - Adaptive Computation Time.
Adaptive Computation Time allows a network (usually recurrent) to decide when “contiune looping” and when to stop return an output.
It allows networks to make much complex operations by manteining the same architecture with less parameters, because it re-uses the computational power of the same neural circuits many times.
The key difference between my network and the ACT RNN is that i apply it to reservoirs while in the paper they explore the usage on standard RNNs. In related work they try to apply the same mechanism to transformers with Universal Transformers, that in my opinion hold much potential.
The findings in this paper and in related research confirm many of my suppositions about this:
-ACT based networks are Turing Complete (there exists a set of parameters that allows to represent any Turing Computable function, so any function we need as far as we know), and this is essential for reaching AGI.
-They have higher performance with the same number of parameters, expecially on mathematical and logical tasks, as i thought.
What i didn’t knew is that ACT can be made fully differentiable and trained with GD.
I’ve also red much about Universal Transformer, it is an interesting concept that unifies ACT with Transformers, and you obtain an architecture that is turing complete by design. (similar to applying my idea to Transformers)
Furthermore, i’ve red something about NeuralGPU that is really INSANE. Near 100% performance in many aritmethic and logic tasks.
But I see this are really theoretical concepts, far from being large-scale implemented.
That’s why I like ResNet, you have GitHub repo’s from Microsoft already out there, and others on GitHub have already coded this up and you can run with it and create your own models from scratch and play around with model right now, without a ton of upfront development.
Potential for Success:
-The project shows potential for improvement and refinement through various modifications and enhancements.
-The idea of implementing controls in different ways, such as connecting them linearly to every node of the reservoir, suggests adaptability and the ability to fine-tune the network’s behavior.
-The introduction of a partial memory refresher system and the ability to control individual nodes separately demonstrate a commitment to addressing potential issues and improving performance.
-Variable neuroplasticity, if successfully implemented, could lead to more efficient training and optimization of the network’s parameters.
Potential for Failure:
-Despite the proposed improvements, the project is still in a conceptual stage and has not undergone empirical testing or validation.
-Implementing complex control mechanisms and variable neuroplasticity introduces additional challenges in terms of training and optimization, which may not yield the desired results.
-The use of probabilistic controls, while potentially beneficial, also introduces an element of uncertainty that may affect the network’s performance.
…Good luck
Yes exactly. I don’t know when i’ll have time to continue the research, and in the mean time i had even better ideas. Before starting i think i’d better read much literature about.
I have the feeling that AI in this era is still not able to think for itself very much. Your project may be the solution to this problem. I wish you could do it.
In this discussion focusing on a new type of neural network called Self Control Reservoir Network (SCRN), bocchesegiacomo01 begins by describing SCRN as inherently distinct and potentially revolutionary in the realm of Language Models and Deep Learning, with an architecture inspired by the human brain and the ability to perform complex operations. They link to their published paper: Self Control Reservoir Network: enhancing intelligence for Neural Networks.
bocchesegiacomo01 later expands on their initial concept in response to bruce.dambrosio’s suggestion about using iterative models like GPT-4. He clarifies that his model operates on a more fundamental level, capable of executing any logical and mathematical operation, applicable to various fields.
jwatte raises concerns about the tractability of the training process, noting past failed attempts due to insufficient gradients and backpropagation issues. However, curt.kennedy proposes the ZORB training approach as an intriguing possibility, which uses the Moore-Penrose pseudo inverse and does not require gradient calculations. bocchesegiacomo01 appreciates this suggestion and plans on exploring its applicability in non-feed-forward networks.
N2U provides detailed feedback on the proposed paper, recognizing the work’s potential but suggesting a more comprehensive and comparative approach when targeting areas of originality. In addition, they emphasize the need for empirical evidence to support the theoretical claims made in the paper. They also stress on the importance of transparent academic practice by acknowledging previous related works. Moreover, they recommend pulling down the paper until it undergoes a thorough revision with guidance from a mentor within the scientific community to prevent misinterpretation and potential academic repercussions.
In another reply, curt.kennedy shares enthusiasm for the concepts of Liquid State Machines and how AI technologies are shifting from being GPU/CPU bound to evolving within FPGA and high-speed arenas. N2U encourages aspiring researchers, suggesting they incorporate external machine learning experience into their existing fields of study,
In closing, bruce.dambrosio brings an interesting article to the discussion: a prototype of ‘real’ liquid reservoir computer that predicts events better than some digital computers.
Summarized with AI on Dec 2 2023
AI used: gpt-4-32k