Seeing how Sydney was initially managed really surprised me. Was it a failure of communication? Ignorance? The things that the bot was spitting out was very alarming (Some of it R-rated). Their solution of essentially limiting the conversation length was also strange. It wasn’t like some serious prompt injection was required, some people were falling into this pit by accident. Seemed like a complete backstep from cGPT.
After seeing the AGI post I felt a bit relieved as well. However these same issues are clearly going to occur again. Value alignment is critical and based on what happened with Sydney, it seems that even massive companies with their own departments invested in this technology aren’t being proactive.
I am hopeful, but not optimistic.