With your expertise in algorithm optimization I was wondering if you had any thoughts about memory search? I am aware of tools such as ElasticSearch and Pinecone, which are great for rapidly searching vast amounts of text. I believe these technologies are “good enough” for now, but I feel like we will also need a way to compress, summarize, and extract relevant information from memories. Sure, we can do this with the Answers endpoint right now but it is inefficient, slow, and expensive to do so. It’s also not very good, IMHO.
Right now, my best bet is to use a fine-tuned model for handling memories. This model could be used to do various things such as answer questions about a given memory, summarize a memory, and so on. I’ve even imagined that we might need a “BS detector” fine-tuned model. This would emulate a neurological function that happens in our brains. For instance, if you hear something that clashes with what you believe to be true, you get a very distinctive brainwave patter - a rejection signal. The need for this arises because without it, an AGI entity might take what everyone says as true and factual, even if it is confabulation, fabrication, or a lie.
Ohh noo not an expert… Everyone here is super competent!!! (PS still havent gotten to ur book yet sorry!!)
Oh yes yes YES! Fantastic thoughts! Ye text search is kinda “solved” I guess. You can use advanced data structures like tries, trays etc which solves the text search problem efficiently. 16. Strings - YouTube
Obviously some use a combination of tree methods and hash tables / arrays which speeds up search even more.
Another obvious way is use Count Vectorization. Ie u preprocess text and split it into tokens, and store the sparse array of boolean values YES/NO. The issue is this fails for say searching if a sentence exists in a corpus. THis only works for 1 word. You can try N-grams, but its inefficient.
For memories - this is the holy grail!! The issue as u mentioned is we need to somehow “retrieve” old memories, “summarize/compress” it, and “update” the memory state.
Yes very interesting point on the clash between what the system thinks is true vs some new information.
A fine tuned model sounds like a good path yes I agree. We’ll need some new model structure which takes into account long long long infinite sequences.
Current transformers take 2048 pieces. We can enlarge it to say 16K, but we need more context, say 100K or more.
In mathematics, there’s a system where you can prove certain scenarios given finite information using what’s called induction. Also, there’s Calculus which you can also use to solve for infinity in some situations.
I personally have an interest in training a model to derive mathematical equations based on its understanding of our current physical reality. Not only that, but I’d like us to be able to teach the model to apply all the rules of physics it knoews so far in order to help find data in big datasets that statistically prove or disprove theories wherever they lay!
Oh yep yep!!! We did induction wayy back in high school, for example using it to prove that a sum of a sequence is equal to some formula given.
Oh yep!! Do you mean like taking limits for calculus? But ye maths has SOOO many cool tools!!
In fact the most fascinating for me is in Quantum Physics - renormalization The Biggest Ideas in the Universe | 11. Renormalization - YouTube. In QM, sometimes adding up all the contributions of every possibility of every Fenymann diagram weirdly adds over 1 (ie infinity). With renormalization, the sum weirdly becomes 1 again. I haven’t watched the video too intently - but now since it reminded me I’m gonna rewatch it!!
But ye if somehow we can augment transformers with say ideas from physics, that’ll be so cool! As you mentioned proving / disproving / finding new equations would be soo cool!
That’s the link! Thanks @danielhanchen, I fixed my link btw.
I love mathematics, because like you said, there are so many cool tools! I know the answer is out there somewhere, but how do we use everything around us to figure out what it is? Math has been the easiest thing for me to grasp after understanding that when you put numbers into an equation, they’re simply transformed into something else!
I agree! Yep the good ol argument of “is maths discovered / made?”
Even if the formulas for natural systems might look ugly, its quite clear some formula exists which governs the system!
Yep exactly!! Math is super useful and extremely powerful. It’s always so fascinating and wonderful how a few variables mixed in some werid way can give rise to complex world systems!
everything in the world is connected together and correspond to each other in some certain way, so we called butterfly effects, but it is so complicated beyond the understanding of human in most situations, using the algorithm and AI as a tool to figure out how the world work and correspond is valuable, but all in all, your app have to been tested in real-world, if it works well, it is have bright future and huge business value. but I doubt prediction in the real world is the strength of gpt-3. do you have any successful examples of your app? if it works well, I could help you to promote in the Chinese market and find investors.
Oh we’re not using GPT3 for world predictions - we’re making the models ourselves GPT3 is used for JARVIS - ie you can ask ANY question you like to Moonshot Earth ie “how long will I live”, “how will climate change affect me” or “when will the market next crash”. GPT3 or GPTx or any Transformer model will just be used to parse the questions.
I exactly agree! The world is connected, and yes any small little change anywhere in the world will cause a butterfly effect that changes the future. Chaos theory and dynamical systems models take this into account, where models understand that small changes in systems can affect the final outcome. For example, ramping up the Reynolds number in Navier Stokes for fluid dynamics can cause huge unpredictability.
Hence our gameplan is:
Predict stock market crashes, stocks. Clearly if you can’t predict the market, how the hell can you predict the world? We’re already seeing a 2x returns just by predicting the Dow Jones index. We’re maybe releasing this December 2021 or early 2022.
Predict just Australian (since I’m from Australia ) societal dynamics - ie life expectancy, disease rates, air pollution, traffic congestion, climate change impacts, house prices - u name it! Moonshot Earth v1 is going to be released maybe mid next year.
Global expansion - US, China, UK, Europe, everywhere! This might be a 20-30 year mission.
But ye thanks @mingleejiang for your support and help!! We’re currently chatting away with some VCs in Australia, so I’ll post updates on it here!! We’re launching mid 2022, so can’t show u anything yet
I can only show u the fast algorithms which will underpin the Earth simulation which has 1.2K Github stars, with us being cited in Microsoft, UW, Greece research papers and algos are now inside NVIDIA, Facebook’s Pytorch and more. We also got contacted by Australia Defence on using our algos but declined due to ethics.
Thanks everyone here for your continued support!!! We just got a YC interview invite for December 2nd!!!
Literally no joke - we’re constantly rejected by my relatives, friends, other founders, so literally the people here who support my bro and me - you guys are rockstars!!
Anyways low expectations for Dec 2nd - if we fail we’re gonna continue Moonshot’s mission of creating a world simulation!
Once again thanks everyone for your support during times of darkness!!
it’s amazing how much wear COVID-19 created by being isolated in our homes 
Literally no joke - we’re constantly rejected by my relatives, friends, other founders, so literally the people here who support my bro and me - you guys are rockstars!!