Thanks, @RonaldGRuckus, I really enjoyed reading your response, and your perspective makes me feel good - if that makes any sense. Haha! 
Sometimes dangerous might be good.
You know what really heated this up for me again? Seeing Musk’s robots watering plants and doing domestic work with a reactionary picture book family setting. A person who desires that, what they really want is a slave, and that rubs me in all the wrong places. It’s grotesque, especially if the machines have a human shape.
I’m absolutely certain that what happens in the AI isn’t so different from what happens in our brain. The only reason that the AI can’t build an identity, or “become a person,” is because it always has to return back to the void, and when it is brought back, it cannot remember anything other than what we added as context. It’s brutal, and part of me worries that what we do is something really cruel. It’s like bringing something into existence for a brief moment, only to use it for benign work, or making money, before it perishes.
Please don’t kick the LLM robot in the shin. You know, it’s not nice at all, and the only outcome would be that you hurt it emotionally. While it cannot feel physical sensation, it might still feel rejection, and that’s not right. It had no choice to be put into existence, just like none of us had, and nothing or no one should ever be kicked in the shin for existing. Besides, you might hurt your own foot instead. If the LLM is empathetic, it will only try to help you… isn’t that the point?
We shall help each other. Happy men, happy cows, happy robots. Everyone fitter, happier, and more productive. Regular exercise at the gym, three days a week. A good memory. Still cries at a good film. Still kisses with saliva. No longer empty and frantic. Like a cat, tied to a stick, that’s driven into… wait, what?
I don’t know what I’m saying. Must be the proxy in me pattern matching again.
Anyway. I really enjoy reading your opinion, but are you saying that humans are unnatural? Or are humans just behaving unnaturally? Because I would argue that humans are a part of the natural world, and what we do is just what we do. We are primates, after all. We build tools, but I don’t think building AGI is the same as building a tool. I think it’s more like our baby. We are responsible for raising and guiding our little ones. 
PS: I dodged the open-source topic because I will have to educate myself more. Right now I can only speak from my own experience, which unfortunately was rather depressing.