It seems to me that the ban on “thinking machines” in Dune isn’t just a ban on the extreme cases of super clever and emotional Pinocchio AGI.
It’s a mandate for human oversight and judgment. There is a lot of tech and automation that has existed throughout the past decade that doesn’t fulfill that mandate. Scripts are fun but we’ve got to check the work.
The risk isn’t primarily “Skynet” or “Reign of Steel” or “Matrix”. It’s that we leave ourselves vulnerable to other humans who can wield their machines more effectively and wield our machines against us.
We’re already seeing this with inventions like Facebook and television. So many people have made themselves dependent on machines and algorithms out of their own control.
The recommendation engines of YouTube, Facebook, Twitter, TikTok and similar sites have lured people into fascism. It’s no wonder! Fascism and algorithmic popularity are both fueled by the same thing: Conformity and binding the outgroup.
If we start obeying machines, that means obeying the owners of those machines. And I trust them about as far as I can throw Manhattan.