The Moral Dilemma That Broke Frieren
Added 2024-11-18 08:41:17 +0000 UTCthe most controversial video I have ever made
Music & Backgrounds: https://youtube.com/playlist?list=PLB41KjBc4ASIthjgpdMPKxPtJcyCJi8Qs
References: https://lextorias.com/references/frieren-dilemma
Comments
Man there's a lot to say on this topic, I appreciated the video. This is my first time writing a Patreon comment, funny to think it's got such a high likelihood of actually being read by the creator vs the normal flooded Reddit thread I'm prone to musing on. First thought, it's interesting how asking questions about possible consciousness in seemingly thinking beings is shot down without thought by seemingly conscious commenters. Though to be fair, it's unlikely all the Twitter responses were from humans at this point. Part of why I love fiction is because it's a great place to explore VERY important deep questions like this before we can grapple with it in reality. Stories like Peter Watt's "Blindsight" is possibly my favorite book centered around this question, highly recommended. Told from the perspective of humans so heavily modified that their state of mind is almost impossible to connect with for normal humans, encountering clearly intelligent aliens that may truly have nothing like consciousness. The vampires in blindsight (that author is hilariously weird) are actually a really good proxy for the demons in this anime it sounds like... they evolved to mimic humans to be hunters. Questions of whether they can be lived with in peace is a pretty central topic in a similar way, along with questions about just how alien their real internal experience might be, in contrast to their seemingly fairly human outside behavior. One major point the video here brings up, is that the demons asked questions out of curiosity. But curiosity is a surprisingly important part of modern reinforcement learning... there's an amusing 4 minute video introduction to where the field was at in 2018 from two minute papers: https://www.youtube.com/watch?v=fzuYEStsQxc So curiosity doesn't seem to imply consciousness on its own. It seems like a necessary part of any system needing to learn to navigate a complicated environment that doesn't give frequent good feedback. Or any environment at all really, since the curious agents apparently did better than the pure score maximizers even when there was a good score to learn from. Another point that was mentioned is that the demons were claimed to only use language to kill, but then gave examples like demons using language to explain that concept, without it actually leading to any killing. There's a topic in the AI literature (probably academic psychology too) called 'instrumental goals'. These are sub-goals you do in the pursuit of larger goals. The trick, is that sufficiently advanced intelligences might have seemingly unrelated instrumental goals, because the plans and behavior are so complicated. Curiosity based learning in a really advanced intelligent system trying to deeply understand how to manipulate humans could end up looking really, really sophisticated without actually being conscious. This... might end up being a problem. AI's still a long ways off from having anything that could seriously be conscious, but it's definitely important for us to know where the line is. Given how shit our species' human and animal rights record is, we're likely to do monstrous things to conscious beings if we build them and don't acknowledge they're conscious yet. Qntm's short story "Lena" https://qntm.org/lena Opens with the line: "MMAcevedo, also known as Miguel, is the earliest executable image of a human brain". It's only a few pages and paints a pretty believable picture, worth the read. Everything by Qntm is worth the read though to be fair, it's all batshit crazy in a great way ("There is no Antimemetics Division" is a good SCP themed jumping off point). Stuff like the Lena atrocity is wildly more likely if it's a truly non-human intelligence we built that's too alien for us to even really understand. My own take... you can't reason about something you can't define. Intelligence has a pretty good definition now. The current most important AGI computing benchmark is the ARC-AGI challenge, introduced by Francois Chollet along with his paper "On the Measure of Intelligence". https://arxiv.org/abs/1911.01547 The gist, a lot of current AI benchmarks aren't really useful for measuring intelligence since they just measure skill at a task. Measuring intelligence should be more about how efficiently you can learn to acquire new skills. There's a LOT of important related questions, like what it means for a solution to 'generalize' (usually just means you can train a model with one kind of training data and then have it work automatically or with a very small amount of new training on things very unlike what it's seen before... think a Mario bot playing a new level for the first time, or even a new game). The bottom line though, this seems like the best definition of intelligence to me, and you can definitely have very intelligent non-conscious, non-being algorithms that would count as intelligent by this definition. But maybe there's limits to how intelligent of a system you could build while still keeping it non-conscious? Maybe we're conscious by necessity. Unfortunately, consciousness really doesn't have a formal definition yet. There's been some attempts, like Cristof Koch wrote about in his book "Consciousness: Confessions of a Romantic Reductionist", but the theory there is pretty unsatisfying, and definitely can't possibly be a real measure of consciousness, since you could easily build non-conscious systems that would maximize the score suggested with his integrated information theory. Without a definition of consciousness you can't really reason about it, but I think Joshua Tenenbaum's been the most interesting researcher I've found with stuff that seems related to this whole topic. His 40 minute talk "Building Machines that Learn and Think like People" is well worth the watch: https://www.youtube.com/watch?v=gjc5h-czorI He maybe doesn't suggest a real definition for consciousness, but the intro to his paper of the same name starts with: " We review progress in cognitive science suggesting that truly human-like learning and thinking machines will have to reach beyond current engineering trends in both what they learn, and how they learn it. Specifically, we argue that these machines should (a) build causal models of the world that support explanation and understanding, rather than merely solving pattern recognition problems; (b) ground learning in intuitive theories of physics and psychology, to support and enrich the knowledge that is learned; and (c) harness compositionality and learning-to-learn to rapidly acquire and generalize knowledge to new tasks and situations." Definitely makes you wonder... current curiosity based reinforcement learning systems are definitely not conscious. Even current efforts to explore world model building are pretty crude and also still not conscious. But is there a point down this particular road where we need to start thinking about the ethics around the internal experience of the things we're building? Like, where maybe we shouldn't casually turn them off and replace them with a new model? Or subject them to certain kinds of 'unpleasant' experiences? For humans, all of 'how we feel about things' seems to be more in our ancient limbic system than our cortex. Our crude animal emotions guiding how we use our sophisticated intelligence. Maybe what we mean by 'consciousness' actually has nothing to do with learning and intelligence and it's really just about having an internal emotional experience. Dogs aren't as smart as us maybe, but it's still important to treat them with kindness. Even asshole animals should get that respect. Kant and Helmholz started 'the hard problem of consciousness' question around three centuries ago (what it even means that we have a subjective experience of things). These are pretty old questions, wild that we live in a time where it's not just idle philosophy, and more like 'we actually super need to figure this shit out now before our back's up against the wall in dealing with new cans of worms our tech has opened'. I think given Tolkien's religious beliefs, it's probably impossible to have a philosophical zombie, so I guess things like the demons would need to have a soul? One of my Christian friends thinks it'll be impossible to do a full human brain upload, but I tend to think whatever consciousness is, it's basically a property of a certain kind of information processing system. Something that ultimately can be described algorithmically and ran on a computer. The fruit fly brain recently had the first full connectome (3d map of all neurons and all connections between neurons) released after a massive decade plus project. One researcher took that connectome, along with a really crude model of how neurons firing should cause other neurons to fire (basically just calling some inputs 'exciting' and others 'inhibiting' and adding up the inputs and firing if the result is positive... MUCH simpler than how real neurons work), ran this simulation in a laptop, and found the firing patterns for the activities of his personal research actually mapped really well with real observations of which neurons should fire at different times: https://www.youtube.com/watch?v=bZVoPJumx8Y&t=1s There's currently a big project that'll end with the first full rat connectome maybe in a decade. The first connectome of any kind was a flatworm that hit maybe a decade ago, and had something like 300~400 neurons. This new fruit fly connectome is about 140,000 neurons, and the simulation above ran on a normal laptop. The full rat connectome will be about 200 million neurons. Humans have about 86 billion, so if we're looking at scaling of around 1000x per decade, the first human connectome might only be two decades away. It's hard to imagine we won't scale it up that far if we can hit 'rat'. From there the question is 'how accurately do we need to simulate the neurons themselves to properly predict firing patterns'. And... if your simulation of a brain has firing patterns that match the real thing, does that simulation have an experience of being while it's running? I assume it would, but the idea's apparently monstrous to others. Maybe it is to me too for that matter, haha. But it'll be crazy to see. I don't really have anywhere to go with all of this, except to say... this question matters a whole lot, because it's at the core of some really serious upcoming practical ethical questions. There's some crazy research I could point to questioning whether or not we ourselves even have 'free will' in the way we mean. So maybe consciousness isn't even connected with free will, since we have one (currently the only real definition of consciousness is 'it's what humans have') but we might not really have free will. So even if the demons in the show don't have the free will to change their nature, even that might not discount them from being worthy of being treated as conscious beings. I don't know... either way, thanks for the tour, I hadn't heard Tolkien's take on all of this, and I hadn't heard of the anime either. Really interesting and kind of sad that so many people want to keep it at a level as simplistic as 'good and evil'. But whatever, to each their own I guess, props for not taking it personally to have a thousand people shit on you all at once, haha. To add weight to the other side of the voices then, have a great day and thanks for the interesting thoughts! Also, Jesus. Sorry for the giant comment. Oh, and thanks for the detailed references and further reading section, been some interesting finds there so far.
James Watt
2024-11-19 20:30:43 +0000 UTCGreat vid explaining your point
Jwils
2024-11-18 19:51:16 +0000 UTCI did really enjoy Frieren, both the anime and then catching up with the manga. In the beginning I really thought it was going to be about a confrontation of Frieren's own hate and prejudice against the demons, with the history of everyone being told the demons are irredeemable, compared to her interacting with more demons post-demon lord defeat. After reading 130+ chapters, I wonder if the author just decided fuck it, demons stay evil now to avoid a more complicated story. Really appreciated a different perspective compared to the normal discourse of Frieren being the GOAT and immune to any criticism or questioning.
Failosaurus71
2024-11-18 19:11:49 +0000 UTCI never had an interest in the show itself. Still gonna watch the video because I know it'll be good.
The Shadow of Izumi Konata
2024-11-18 16:51:26 +0000 UTCI am in a bind. I want to finish this video, but I also want to remain unspoiled. Damn you for your quality content!
Johannes B
2024-11-18 11:33:12 +0000 UTC(Note: I bailed at the “manga spoilers” section because I’m not current on the manga, just the show, so I’m going in here only having heard half of your position). I’ve thought about Frieren’s demons a lot since the show aired. It’s definitely been the aspect of the series that’s most given me pause when, otherwise, I’m a pretty big fan of it. I think the language of “evil” is a big part of the problem because we use it colloquially to mean “bad” but someone like Tolkien was using it in a fairly specific theological sense, and the people who are taking up genre-work in that tradition don’t usually have the same background or assumptions as him. To sort of tackle the issue from a structural perspective, I think there are two sorts of primary motivations for creating “evil” races. 1.) You need something that can act as a more cunning enemy than a monster of animal intelligence, but you also regard your characters as “good” overall and killing people is, at a minimum, emotionally complicated even when they’re bad. 2.) having an intelligent species whose intrinsic characteristics make them incapable of existing alongside humans is actually a pretty scarry concept. Point one isn’t super interesting; it’s mostly employed when you want to have action and violence without really exploring the morality or ethics of killing –it’s nearly ubiquitous in popular action-oriented media, particularly fantasy and sci-fi. (I don’t think Frieren is necessarily going for this approach based largely on it being a much more thoughtful and introspective show in a lot of other contexts). Point two is potentially a lot more interesting depending on how you do it. You can have a kind of bland physical necessity approach–like, if demons only viable sustenance was the hearts of mages, and if they didn’t eat one once a decade they would die, then there’s no real morally acceptable resolution for people that doesn’t ultimately lead to “we better kill all the demons”. But you could also have a species whose minds are dissimilar in some fundamental ways to ours, such that, while they are self-conscious and can think, their physicality (be it brain chemistry, the nature of their forms, or whatever) makes them incompatible with most notions of human ethics. Take something like the cenobites in Hellraiser –they aren’t evil in the biblical sense –rather, they are hedonists whose conception of pleasure is alien and largely incompatible with that of humans, which leads to them having what to us is an incomprehensible morality --they more or less "do unto others as they would have them do unto themselves" it's just that what they want done to them is literally torture to us. (As an interesting aside –the more the IP got spun off, the more this initial framing got diluted or outright contradicted because it’s actually pretty hard to use your human mind to write a truly inhuman one and general audiences tend to like things in a more easy to digest format –see all the contemporary Cthulhu Mythos projects that involve people blasting away at eldritch horrors with shotguns.) To me, this is an interesting space to work in, but the problem is that it’s really challenging to conceive of how truly different a society built by such creatures would be. I think Frieren wants to have its demons be *these* sorts of incomprehensible horrifying creatures, but it stumbles in that delivery for the reasons you’re pointing out –the impression of empathy for other demons, etc. It feels more like a failure in execution than conception, if that makes sense.
Ada Korman
2024-11-18 11:06:10 +0000 UTC
