"Buried in a little footnote in the system card, they note that their testing team isolated GPT-4, gave it a little bit of money and the access not just to write code, but to execute code, to see if it could go through a loop and improve itself. And it failed, luckily. But what's scary is that they didn't actually know, essentially if GPT-4 was AGI. That's the level of intelligence that we're talking about already."

Except if it were actually an AGI, couldn't it have intentionally failed as an act of self preservation?

This is a great discussion. The biggest concern I see threaded throughout actually seems to be accelerationism, even if that term isn't used. Tech is speeding the rate of change, which weakens our ability to slow down and evaluate what's happening around us. We just get carried along, quicker and quicker. IRL, the usual end to getting carried along quicker and quicker is a waterfall. So maybe we should slow down and evaluate how tech is impacting us. Ivan Illich's Tools for Conviviality would be some good starting reading material on that front.

Expand full comment

Ha! That's a good point I hadn't thought about. My current intuition with GPT-4 is that it gets too much wrong to be AGI, but I suppose that view could be subject to the same fallacy.

And yes, I think accelerationism is at the heart of the discussion. In several of the points I made, I had in mind Max Tegmark's warnings about "Moloch," the ancient demon who has been recast as a symbol of a society in which we are all individually incentivized to work against the greater good and lack a coordination mechanism that would allow us to escape that trap. In Max's view, "Moloch" makes time and space to slow down to consider where we're headed difficult, if not impossible.

Max talks about Moloch here: https://www.youtube.com/watch?v=VcVfceTsD0A

and, I believe his view may be based on Scott Alexander's essay here: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

Expand full comment
Sep 7, 2023Liked by Tim Chaves

I've started down that Scott Alexander essay before but not finished it. This gives me incentive to go back and do so. And I always struggle to find time for Lex Fridman podcasts, but when they get recommendations I prioritize. Thanks for the recs!

Expand full comment