Decoding Academia 18: AI Edition (Audio)
Added 2023-04-21 23:38:04 +0000 UTC*WARNING*
This is a severely waffley episode. Matt and I talk about the paper but we end up getting into the weeds in some fairly obscure and likely rather idiosyncratic debates.
This might sound like what we normally do but I am providing this warning for a reason. This is a loooong episode with a looooong discussion. Matt is very interested in AI and I have various opinions some of which are well thought out and others... less so.
If you are interested in the topic this might be your jam and this is useful preparatory work for the upcoming AI episode but otherwise you might not find this discussion particularly entertaining.
For those interested in the paper, it is a rather technical paper called 'Attention is all you need' (Vaswani et al., 2017) that Matt recommended because it is one of the breakthrough papers that helped to create LLMs. That said I think the paper will be a bit of a struggle for anyone not up to speed with the technical details of AI research. It was hard for me to follow at least. Nonetheless, it is attached to the video post anyone who is curious. The discussion ends up spiralling away from the paper in any case.
Audio version should be available above.
Comments
Alex Kiefer also makes awesome chip tune music that I bet Chris would love: https://exilefaker.bandcamp.com/album/ockhams-chainsaw-2
Alexis
2023-06-15 13:47:34 +0000 UTCCheers Alexis... but philosophers... come on!
Christopher Kavanagh
2023-06-15 03:20:23 +0000 UTCThis was great — well worth signing up to the patreon for! Now, the bad news: sorry, Chris and Matt, I think you may need to call in… the philosophers of mind 💀💀💀 Not to say Chris and Matt’s perspectives are not fascinating; I just think so many of the deep issues here are unavoidably philosophical. As a mere surfer of the discourse, may I recommend this https://podcasts.apple.com/gb/podcast/space-time-mind/id847581558?i=1000570039158 And make a plea for either Alex Kiefer or the great Pete Mandik as a guest on DTG. I look forward to continuing my patronage; you guys are awesome.
Alexis
2023-06-14 15:07:32 +0000 UTCI emailed Monash's library cc decodingthegurus@gmail.com to see if they could figure something out.
Kirsten Greed
2023-04-27 04:10:40 +0000 UTCThe video link that is... Although, I wonder if the whole Decoding Academia series could be accessible via institutional logins.
Kirsten Greed
2023-04-27 03:38:35 +0000 UTCHmmm... we won't object to that!
Christopher Kavanagh
2023-04-27 02:50:27 +0000 UTCI wish this link was public so I could post it into my online university machine learning course chat!
Kirsten Greed
2023-04-26 22:05:59 +0000 UTCIt seems, Chris, like you're invested in ChatGPT not "understanding" its output and simply following an algorithm. I think Matt is taking more of an agnostic approach of what it means to understand something. ChatGPT's token embeddings alone (see the Wolfram article for a description) represent a very nuanced understanding of what words mean and how they relate to each other in different ways. It seems like you're thinking of ChatGPT as the English speaker in the Chinese room following the algorithm to respond to the Chinese text, whereas I would think of ChatGPT as the person-in-the-room + the-chinese-repsonse-book. That system as a whole understands Chinese in some way.
brianshmrian
2023-04-26 15:27:46 +0000 UTCWe aim to please.
Christopher Kavanagh
2023-04-26 09:56:23 +0000 UTCHey! That was my point too! I don't care if my toaster understands toast... I just want it to make it.
Christopher Kavanagh
2023-04-26 09:56:16 +0000 UTCYou want it, you got it!
Christopher Kavanagh
2023-04-26 09:54:24 +0000 UTCHow did we do?
Christopher Kavanagh
2023-04-26 09:54:16 +0000 UTC"How many letters are there in the alphabet?" Maybe I got this :-)
Kirsten Greed
2023-04-26 05:09:29 +0000 UTCYou might well be right, but honestly, approximately 90% of data scientists should get triggerings now and again
Barrett
2023-04-25 11:08:13 +0000 UTCThanks for picking this paper. I've tried to glance at it a few times ever since ChatGPT came out last year. While I'm a computer programmer, I've never worked on neural networks or really any AI so it doesn't make much sense to me. The best accessible explanation on the LLM transformer architecture that I've seen is from Stephen Wolfram: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ In terms of whether it really understands anything, I'm with Matt. Why does it matter? If it produces useful output then that's all that really matters. Well... unless it's somehow conscious and suffering like the rest of us.
brianshmrian
2023-04-24 23:50:24 +0000 UTCYes, I get the point, but it remains a somewhat misleading approach to explaining yourself, I think. It also doesn't necessarily support your argument. As Matt said, you can also describe the brain in an oversimplified manner, yet brains are conscious... I'm just suggesting that there might be more effective ways to convey your ideas without triggering 90% of data scientists :)
Martin Unland Elorrieta
2023-04-24 06:12:56 +0000 UTCI imagine the point is there's nothing spooky going on, the computer doesn't 'understand' the image represented by the pixels
Barrett
2023-04-23 19:54:03 +0000 UTCI understand Chris's intention, but saying that the only thing GPT is doing is calculating probabilities for the next word is just like saying "the only thing a computer is doing is calculating the intensity of each pixel on the display screen"...
Martin Unland Elorrieta
2023-04-23 13:44:41 +0000 UTCThanks! I’m going to feed this back to Matt.
Christopher Kavanagh
2023-04-23 06:48:10 +0000 UTCI rather enjoyed that one, I'm not sure I now know what the transformer is (although I've heard about it being the key piece of architecture several times over the past month without figuring it out so not matt's fault). I appreciated the debate on the broader topic though because it remained grounded and focused on some of the important, but nuanced details. Which seems rare in the current public AI discourse, but maybe that's just the contrast between the wonky academic approach versus the typical riding the waves of dopamine of a novel topic into fantastical guruesque riffs on skynet (or recently, patriotAI). I found the discussion of agency and functional pragmatism really useful. I'm not convinced they aren't, but I wonder if Matt's AI experiments are as interesting as he seems to think (take that in the most charitable way possible though). The what is a photon/how is a photon or similarly the simpson's sign joke "Sneed's Feed & Seed (formerly Chuck's)" are examples, as I imagine the topology of both, even if it isn't the exact same nouns, are repeated a lot in the training material. e.g., "Nobody Asks "How is X?" has its own Know your meme page. Sneed's has several Simpson's fan/trivia pages explaining it and I have to imagine the Sneed tokens are pretty rare. I guess my point is i'd expect it to pick up grammatical constructs like the "everyone asks what nobody asks how is X" and notice that they occur with a wide range of X tokens, and it probably doesn't represent a whole lot of 'emergence'. That said, and I think similar points were made toward the end of the episode, if it already has captured enough natural language that it can explain the joke and there are only so many joke constructions, then maybe it does functionally "understand the joke" for all intents and purposes? thanks and looking forward to the AI episode!
Barrett
2023-04-23 02:53:10 +0000 UTCBring on the full waffle!
Kyle Wilson
2023-04-22 23:49:13 +0000 UTCSounds perfect accompaniment to a morning of domestic chores
matthew parsons
2023-04-22 07:34:02 +0000 UTC
