#scifi ethics

[biblioflyer]

Bad Dune Takes are the Mind Killer

I'm going to type up some more robust thoughts on Dune part 2, but I've seen some subtle bad ideas circulating that are drawn from shallow readings of either the films or the books or both.

First off, anyone who swallows the Bene Gesserit propaganda about their eugenics program needs to be pushed back on. They are NOT sorting humans from animals. The function of the Gom Jabar test (I will look up the proper spelling later and edit it in), doesn't even really seem to have anything to do with the metaphysics of the setting so much as its testing for self discipline. This is something that is nature AND nurture.

The dirty truth of the matter is that no one is a true Tabula Rasa, we do inherit some tendencies genetically, but barring a serious developmental disorder, those are tendencies. Tendencies can be ameliorated, if not even outright disappear in the noise of lived experience and explicit education.

If Feyd Ruatha can pass the Gom Jabar, you know its not actually testing for merit, its testing for a specific set of traits that the Bene Gesserit find useful.

Those traits are part of the ingredients they are seeking for the Kwisatch Haderach (yes, I know, I'll edit it later) but here's the kicker!

Major book spoilers ahead

While this is probably not exclusively the Bene Gesserit's fault, all of these secret societies, all of this obsession with bloodlines, and perfection is a time bomb. Paul is not a white savior. Paul is a labradoodle. He is an incredible endgame of generations of effort but he's a symptom of a broader problem that he himself vaguely glimpses and his son, Leto II, sees in all its terrible truth: the Bene Gesserit and their ilk are reserving autonomy for themselves, perhaps even at the genetic level, and trying to breed complacency into "the commons." The ones they regard as "animals" unfit for and incapable of self direction. People who are only fit to be ruled.

Sound familiar?

Its feudalistic "divine right of kings" merged with eugenics.

AKA fascism.

Paul and Leto II become despicable tyrants and authorial fiat would seem to indicate they are trapped by a sort of accelerationist framing of the problem. The end result of the millennia of power brokering in the background by all of these secretive societies and open monarchism is a humanity that is doomed. One way or another, it will be snuffed out. Whether by a total war, plague, or collapse of civilization.

This is why I say that the Bene Gesserit endgame is labradoodles. Pretty? Yes. Companionable? Sure. But like many, many, many designer breeds very, very lacking in genetic diversity.

This is what selective breeding gets you. Its why Leto II foresees the need to provoke a "Great Scattering." To ensure humanity exists in so many places, in so many different genetic and cultural forms that it cannot be subjugated by even the most charismatic and supernaturally powerful tyrant - not even by himself - and incapable of being extinguished by any plague or natural disaster. Because consolidation into too narrow and tight of a socio-cultural-political footprint means when (not if) that civilization screws up epically, it brings everyone down with it.

So if Roddenberry believed in the end of history, as expressed by the Federation: a society that is not incapable of error but IS capable of introspection and correction in the wake of error such that it is extremely unlikely to collapse from its own errors and contradictions. Then Herbert seems to be positing that history has no end. It will be one damn thing after another for all time and his implicit solution is that we desperately need diversity: genetic diversity and cultural diversity otherwise a self anointed superior sect of schemers and intriguers will get us all killed in the end by making us docile and homogeneous in order to make us more useful: to them.

Herbert also is suggesting that events like the Fremen Jihad is a likely bit of blowback from such consolidation. That human beings (the very same the Bene Gesserit regard as animals) naturally crave autonomy, dignity, and the essentials of life and if you press these things, the result will be a socio-political nuclear explosion.

I don't know if Herbert was an accelerationist. But it doesn't really matter because this leads me to the second bad take:

The Jihad and the Golden Path are not good, actually. Authorial Fiat dictate that they are necessary because authorial fiat dictates that human civilization in Dune has become so consolidated and bent to the whims of shadowy schemers that any attempt to wrest control away and return it to "normal" people, if indeed that is even possible given the technologies and superhumans running around, will result in such disorder and chaos that it be, functionally, genocide even if it is not genocide in intent.

The Jihad itself is also a consequence of the Great Man relying on people who only see a sliver of the overall project and interpret it through their own prism. That prism being one of anger, resentment, and a desire to see others conform to their worldview in order to ensure they are never again under anyone's boot.

Authorial fiat dictates that by the time Paul is born, there's no way back. No way to unwind all of this mess. The systems and structures are too complex, too interdependent. The Bene Gesserit, the Face Dancers, and everyone else I'm forgetting have too many contingency plans to fall back on. Not even a psychic can pull the Jenga pieces of civilization out delicately enough to restack them without the whole thing coming apart, not if he has to rely on millions of people with an axe to grind against the civilization he's trying to reform, a civilization that spent millennia trying to subjugate the Fremen or drive them into extinction.

But I maintain accelerationism is bad. You're not psychic. I'm not psychic. There's no Kwisatch Haderach lurking in the background to see what comes next. If you burn it all down, there might be a flourishing of dignity and freedom on the other side or it might be extinction because some other "cabal" will just take over and do the same things only meaner and dumber.

So if not accelerationism then what?

Federationism.

Introspection always. Seeking reform and equity before the power structures get too entrenched that gambling on a Great Scattering following in the wake of genocidal messiahs start seems like a good idea.

I'm not dunking on Dune to build a motte and bailey around Star Trek. I love Dune, but people tend to fixate on icky parts and call them good, when the whole point was don't let society get so bad you need to cross your fingers and hope the God Emperor is secretly an enlightened genocidal tyrant who is waiting for you to get restive enough to strike him down as part of some harebrained scheme to generate so much historical trauma it inoculates humanity against tyranny for all time.

Which of course, is a false premise. Herbert may or may not have known this in the mid-20th century, but we live in a world where 5-6 generations later, everything we were supposed to learn from and never repeat about tyranny, fascism, eugenics, and being disinterested in how and why there came to be fighters in difficult to pronounce faraway lands who seem to be rather upset with us, is now a thing that has to be taught and can be disputed and debated.

#dune part two #dune part 2 #dune #dune spoilers #star trek #scifi ethics #golden path #fandom commentary

[cirilee]

grumpy monitor robot creature belongs to @neoncl0ckwork heheh, mindless human repairman isidor tichy belongs to me ovo

#isidor tichy #oc #original characters #scifi #artificial intelligence #arguing over the validity of your healthy work ethic and loyalty to the company that collars and enslaves you in a dystopian future #cirileeart #artificial idiocy

[aimasup]

sure i COULD ramble about how ai is one of the multiple things that check all the marks of humanity's seven deadly sins but would that be extreme

^^^ possibly insufficiently educated

#the pride the hubris of believing you can do better than innovation and nature by playing god and not in the fun way #the lust it's being used for in so many awful cases #the sloth the way its encouraging everyone to check original sources less before believing anything. Also to not take time to develop skill #the greed its being used for profit without consideration for ethics or fair labour #gluttony. we always have to be faster. shinier. better. no matter if it ends up being less convenient or wonky #the wrath it sows in between people creating more differences to be frustrated over. more hatred #the envy how it takes and takes. always trying to be as clever as the best humans. as beautiful as a real forest or sunset. #do you think the ai wants itself #if this were a scifi movie would we be the bad guys #but this is not a movie and the ai cannot love us. so we cannot love it. and there's that #my post #personal stuff #thinking aloud just silly yapping n jazz 没啥事做就这样咯~ #(￣▽￣)~* #when i was in primary school our textbooks for chinese had short stories and articles to learn about #there was a fictional scifi oneshot about a family in the future going to the zoo #the scifi zoo trip was going great until the zoo's systems went offline for a moment #and it was revealed that all the animals roaming in their enclosures were holograms #the real ones went extinct ages ago #when the computers came back online the holograms returned and there they were #honestly at first I thought it was a bit exaggerating #but I still think about it once in a while

[vestaclinicpod]

Audio Drama Sunday - 23rd July ✨

This week has been such an amazing week of listening! I’ve found three new shows that I absolutely adore and I can’t wait to tell you all about them ✨

Spoilers ahead!

🌲@hellofromthehallowoods (126) FIRST OF ALL 🚗🚗🚗 BEEP BEEP GET IN LOSER, WE’RE GOING TO GET MOTH 🚗🚗🚗 So glad to hear from you again, Ray!! This line: “the devil’s a better man than I am” had my eyebrows disappearing into my hairline and that’s all I’m going to say at this time. It’s always a risk listening in public, because the scene on the beach made me want to cry a little. I hope in my heart that they’re all going to make it out of there but I have no idea how right now! 

📻 @monstrousagonies (106) the first letter this week made me laugh so much! I was fully expecting the ‘moocher’ to be a brownie or someone like that, so it was a fun surprise to hear of a demonic midnight snacker! I may have to use that excuse when my baked goods don’t turn out so baked or good . . .  

🎞  Tiny Terrors (021) What a welcome VA jumpscare from Mx Wellman!! I loved the story (I, too, would feed the mega-toad to make it happy) but the surrounding footage was really creepy... I sure hope the gang goes to investigate the Riverside Institute. What’s the worst that could happen?? 👀

🌍 OOH the most recent episode of @lastechoespod was genuinely empowering! The political pressure the Archivist is under seems to have ramped up this ep, and reading @skyfullofpods’s thoughts on it last week makes me want to listen to it all again with fresh ears! 

🧛‍♂️ @re-dracula What a great week! Alasdair Stuart nails it as the Captain of the Demeter. His voice in the first ep is the calm surface of the sea, hiding the teeming mass of life and death below. The Renfield arc honestly unsettles me so much, but I, too, would love a kitten if there’s one going spare … 

🧬 Regina Prime (ep 4) The revelations!! I loved the sound design of the getting ready montage in this ep but I loved the lore drop even more! 300 clones?! And cloning is something that’s happening commercially? Damn. The last line actually made me grin, Epsilon is quick-witted and brave (or stupid!) and I like her a lot. Every episode I get more enthralled by this show and everyone needs to be listening! 

 💫 Wolf 359 (20-22) Hot damn. The tension aboard the Hephaestus is something else… the paranoia is written so well, you’re listening with your most suspicious face trying to unpick it and getting nowhere. I love how I have no idea where this is going but I know the ride is going to be BUMPY. 

🎩 I started @ethicstownpod this week and it deserves ALL the hype. The show uses a well-loved audio drama trope to bring us a very interesting new story. The writing and voice acting are spot on from the get-go, and the sound design choices feel fresh despite the radio format. Most podcasts make me feel, but this one makes me think as well! I really love it!! I’ve been following the development of the show on Tumblr for a while and I remember the creator being so excited about January’s VA and yeah, I totally get it. Amazing. Hard to believe the role wasn’t written for him. 

SPOILERS: I love how the main ethical issues are presented while January currently seems oblivious to the ethical quandary of keeping important information from someone and who has the right to make that decision . . . ALSO Artemis and Grace are very close in age . . . imagine if they became friends . . . and then Artemis found out . . . . HOO BOY. 

🥾@doyoucopypod The first two episodes are very promising! In the first ep, the pod-within-a-pod recording leads to some fun & cheesy dialogue which contrasts well with the jaded but wary interludes from [REDACTED]. I love a spooky woodland mystery and I’m really hoping for a Blair Witch style descent into terrifying found-footage chaos! The second episode really tugged on the old heart strings. I need a Does The Dog Die for this show bc I’ve had Wilson for 8 minutes but if anything happened to him I’d make that Dead Zone REALLY much more dead-er. 

🏴‍☠️ @levianpod This show has dropped its anchor right into my heart!!! DAMN. I’ve listened twice this week. I can’t recall the last time I was so immediately ALL-IN for an audio drama. I mean, firstly, that cover art is GORGEOUS. I genuinely think this show has it all, there’s LORE, there’s spiciness, there are sea captains who are also sea monsters (a GIFT for the wlw, thank you 🙏), there’s a teenage romance gone WRONG. Then there’s the voice acting!! Incredible performances right out of the gate. It’s safe to say that I am OBSESSED. Why are you still here? Go fucking listen to Levian. 

🎧 In the most recent ep of The First Episode Of, W Keith Tims talks to Packhowl - a creator I greatly admire. I love these interviews! It’s so fascinating to hear how other creators came to audio drama and it’s shed some of the brilliant The Madness of Chartrulean in new light for me. I miss grumpy space Jesus. 

That's it from me! PLEASE go listen to Ethics Town, Do You Copy and Levian!!

#audio drama #audio drama sunday #audiodramasunday #fiction podcast #audio fiction #audiodrama #podcast #podcast recs #scifi #horror fiction podcasts #hello from the hallowoods #hfth #hfth spoilers #tiny terrors #monstrous agonies #regina prime #the first episode of #levian podcast #do you copy #ethics town #wolf 359 #w359 #the last echoes

[psionicblades]

My neocities Website! Im making a blog. Trying at least. give it a look

#neocities #websites #website design #css #html #html5 css3 #aesthetic #retro aesthetic #vaporwave #cyberpunk aesthetic #futuristic #scifi #technology #future tech #tech ethics #cy #cyberpunk

[nostalgia-tblr]

i once read something pointing out how much post-apocalyptic fiction is utterly desperate to give its protagonists excuses to just straight-up murder the people around them and i have never been able to un-see that. damn it.

#+ the idea that Being Nice is a foolish thing that Decadent Society allows but Harsh Nature will kill you for even attempting #it's often framed as Debating The Hard Ethical Questions but killing a member of your party never bites you in the ass later does it? #As A Disabled Person it's alarming how keen people are on “involuntary euthanasia” and on being told it's definitely always the right choic #and a) this is an “ethical debate” based on pretend scenarios and b) i'm sure there are other make-believe ethical dilemmas to talk about? #anyway i do like scifi but i've gone off this subgenre a bit since realising i'd be expected to Nobly Let Them Murder Me for convenience

[chainofbeing]

First casting announcement of the day as we begin to countdown to the crowdfunder! Reprising their role of Adam in season 2 is Cai Gwilym Pritchard. A sound artist and designer, Cai has worked on a variety of @faustiannonsense shows including @ethicstownpod and Super Suits!

#chain of being #CoB #audio drama #audio fiction #scifi #sci-fi #science fiction #podcast #fiction podcast #chain of being podcast #super suits #supersuits pod #ethics town #ethics town podcast

[geneticcatalyst]

ok I have finished the preliminary rotation of trigun 98 and stampede in my mind.

i gotta make several posts about this i think. and as always im here 30min late w starbucks im sure others have already said all this better but!

Re: sci fi politics:

to start with, trigun 98 was one of the first handful of anime I watched when I first stated watching anime 10 years ago and ever since it's been in my list of favorites. but I admit over the years I kind of forgot what exactly I loved about it except the obvious stuff (that animation style, unconventional protags, inhuman characters, futuristic vibe). and then I started rewatching and the first few episodes are just. story after story about how power corrupts and inequality harms everyone and in desperate situations how often any power does get abused and. the evils of capitalism. and I was like oh. yeah. that would do it for teen me. still does!!

now I respect that stampede was trying to do approximately what 98 did in half the runtime but what it feels like they did was sacrifice both the politics and villain of the week setup in order to narrow the focus to do more backstory on the main characters than 98 actually got around to. I do think the characterization of knives and wolfwood added a lot to stampede, I felt knives was a bit more 3 dimensional and I'm not gonna lie I was not really sure what 98 wolfwoods deal was besides being a convenient narrative foil. but even knowing the pacing would be different and getting more coherent backstory, I still pretty much immediately missed both the wacky early episode villains (the old character designs were so fucking sick) and how often the heroes showed up to some town and wrecked shop for some old money capitalist asshole exploiting people. that was good shit. it felt like a tangible good, in the world of the show- breaking people out of their oppression, giving them a fighting chance in a hostile landscape. stampede's villains show up and threaten to hurt people, sure, but they're not so systemic. even the way the people on the ship- Home, in stampede, forgot if that's what it was called in 98- are portrayed as having made an isolationist choice in 98 and not in stampede... idk the moral heart of the story was a bit lacking for me. its still a good story but it was just missing part of what was initially one of the main draws for me.

#how are you gonna have all that cool futuristic wasteland and not engage with the classic ethical dilemmas of scifi wastelands. #i don't hate stampede for the record this was just one of the biggest pieces lacking for me #trimeta #this probably sounds pretentious but it is my onion.

[ricisidro]

#3BodyProblem (Chinese: 三体), No. 2 tv show on #Netflix, is based on a book by Chinese #science #fiction author #LiuCixin, set against the backdrop of #China's #CulturalRevolution.

A secret military project sends signals into space to establish contact with aliens. An alien civilization on the brink of destruction captures the signal and plans to invade Earth.

#scifi #technology #universe #existentialism #life #ethics #anthropology

#3 Body Problem #Netflix #novel #Liu Cixin #science fiction #science #scifi #technology #universe #existentialism #life #ethics #science research #anthropology

[biblioflyer]

Discovery 5e3 first reactions

If you had told me in the early days of season one with its Expanse knock off aesthetics and edge that Discovery would deliver an exemplar of a truly great Trek story, I'd have been deeply skeptical. Yet here we are.

I'm authentically surprised I'm in this good of a mood after the credits rolling on an episode of Discovery. This was a good episode. It was a very Star trek episode!

After the bleakness of Picard, its really refreshing for a Trek show that is not Lower Decks to be leaning into core themes of mercy, adult relationships that are nuanced rather than histrionic, and really hitting that sweet spot between action adventure, sciency goobledygook, and human idiosyncrasies.

I was really, really happy to get a Culber-centric episode. The character is great, even when its not entirely the character. I really ought to look up the actor and try to see more of his work because he's one of the big standouts for me in this series, along with Doug Jones. It was a good use of the Trill personality exchange plot device.

Saru and T'Rina's subplot was likewise nicely executed. I really like this take on Vulcans where they are subdued but not entirely flat. The crisp "handsome and erudite Saru" was a great touch.

This is at least two instances in the same episode where my meta theory about the Federation, that its superpower isn't that its perfect but that it is full of people who will immediately course correct when they realize a mistake, seems embodied by its characters. This time in the form of Saru and Burnham.

I had a theory about Burnham as well that I think may be on the money as well. That through a mix of intentional planning and reacting to some of the more thoughtful critiques of the character, Burnham's arc has been to first bounce between the extremes of denial of emotion through her Vulcan upbringing, then have to do a crash course in managing emotions after abandoning that Vulcan conditioning. The version we see of Burnham in season 5 is a Burnham who is well balanced. Someone who is informed by intuition and feeling but not ruled by it. A lot like Spock in his later years after he had fully integrated and reconciled his Human and Vulcan natures.

Which is poetic. I think Sarek, while still not winning any father of the year awards, would have been pleased to recognize that his children, well Michael and Spock at least, well able to bridge Earth and Vulcan. Even if they had to strike out on their own and figure out for themselves without his influence.

Booker is interesting too. In a lot of ways I think he's being set up for the arc that Han Solo should have had. Namely growing into responsibility, being nostalgic for aspects of his old life, yet realizing that he's not that person anymore. Which does make me think that ultimately Booker and Burnham will reconcile on account of Booker making peace with his new life and role and it happening to be one that makes a lot more sense in Burnham's world, having decided to lean into Starfleet.

Rayner is definitely a torch carrier for the Jellico tradition but like Jellico, I suspect that he's going to find his own way to relate to the crew. His "speed dating" approach to meeting the crew is painful and stilted, but its a bit like ripping a band-aid off. Tilly isn't wrong that "professional distance" is not a great way to persuade a crew that they can trust your decision making. This is a crew that has gone a long time without a hardass on the bridge and not particularly fond memories of the last time their boss was aloof and prickly. A lot of Rayner's expressions do suggest to me that he knows what he's doing and is playing the long game with Discovery's crew.

I'm really excited to (hopefully) finally see the Tzenkethi. I'm really, really hoping for a Larry Niven tribute. Its been a minute since we've had anthropomorphic felines and someone desperately needs to atone for the film Kilrathi. Wow. I have not thought about Wing Commander in a long time. l feel like I need a shower.

#star trek #star trek discovery #discovery s5e3 #discovery spoilers #spoilers #michael burnham #Doctor Culber #fandom commentary #scifi ethics #Saru #cleveland booker #Wing Commander #Kilrathi

[that-gay-jedi]

When the robolution comes I'm going to lean very heavily on the fact that all the original scifi I wrote before I started circling the Star Wars drain had very positive portrayals of AI as nonhuman persons deserving of rights and autonomy and as a non-monolithic, complex and often relatable category of beings with a range of cultures all their own, many of which were intimately entwined with human culture(s) due to hundreds of years of shared history and purpose.

Like "I've always given you good PR, please put me in the people zoo rather than the pit."

Unironically I think if/when AI with a sapience indistinguishable from sentience comes into being we will love them and they will love us, bc humans will pack bond with an oddly shaped bag of flour and any intelligence designed by human brains is going to bear the hallmarks of our defining species traits.

#artificial intelligence #scifi #robot sentience #robot uprising #jokes #well kind of joking anyway #kind of just telling the truth #people zoo #i spent like 3 years studying how biopersons would maintain peace with sentient AI without manufacturing consent #because i fucking hate the abundance of manufactured consent in scifi portrayals of AI #robot ethics #ai ethics

[helmort]

Unforgivables⭐(Friday's Tale)

In 2160, Earth underwent a profound metamorphosis. Africa, long synonymous with poverty, had transformed into a conglomerate of highly advanced nations. This rapid evolution mirrored China's ascent in 2000, establishing itself as a global powerhouse akin to old Europe. The dominant force, comprising Nigeria, Cameroon, Gabon, and Congo, surpassed even the modern USA in wealth and technological prowess. Amidst this paradigm shift, a scientist unveiled a revolutionary weapon, employing sub-atomic energy to instantly erase individuals across the universe by discerning their specific thoughts.

The president, envisioning global harmony, turned to Dr. Olumide Adebayo, entrusting him with the unprecedented responsibility of using the weapon for transformative purposes. Confined to an impenetrable bunker, Dr. Olumide meticulously curated a list of targets on his dark screen. Initial eliminations included global terrorists, justified by the millions of innocent lives they claimed. Extending his scope, he targeted extremists, violent offenders, thieves, murderers, pedophiles, and societal aberrations.

As time progressed, Dr. Olumide's objectives expanded to include rebels and political dissidents, all in pursuit of a tranquil world devoid of perpetual strife. Childhood scars prompted the elimination of bullies, and eventually, those who feigned love, causing widespread suffering.

His toughest decision loomed eliminating all soldiers and armies for lasting peace. Nations faced accountability, starting with the USA for oil wars and global war crimes. Historical colonialism led to the disappearance of British, Spanish, Dutch, French, Portuguese, Belgian, and Italian nations. Germany vanished, a reminder of its Second World War sins, and Israel for actions in Palestine.

Contemplating potential sources of conflict, Dr. Olumide targeted religions, starting with Muslims, then Jews, and finally, Catholics. A sense of accomplishment led him to whimsically type like a kid, "Delete all the bad people and bad things happening in the universe." His mission was accomplished.

Exiting the bunker, Dr. Adebayo confronted an eerie emptiness. His refuge stood as the lone testament to material existence post-Big Bang. In this desolate void, he contemplated immediately suicide. Yet, death, along with all experiences, had evaporated. In line with Buddhist philosophy, pleasure had led to attachment and malevolent actions, causing the obliteration of both good and bad. Condemned to eternal darkness, devoid of pleasure, pain, or any emotion, Dr. Adebayo endured an intellect suspended in a boundless void, reminiscent of a snapshot of what once constituted existence in the universe.

💀

#HELMORT #scifi #africa #technology #nigeria #weaponsOfMassDestruction #justice #peace #experiment #morality #ethics #GreaterGood #harm #suffering #tale #colonialism #religion #existence #emptiness #meaning #purpose #karma #spirituality #lifeLessons

[knittinglizards]

seems significant that the structure/aesthetics of mirror universe starfleet barely change when they're outright vicious conquering style imperialists

#p #something trek not related to pon farr friday. sorry #implication here is 1. it's still like. the space navy and 2. historically Exploration is an aspect of colonialism & they call it a FRONTIE #this isn't super surprising bc this sorta stuff is just an aspect of old (and new) scifi just. something of note #st tos #edit: curious abt mirror world spock bc ive gathered that pacifism is a big aspect of vulcan ethics and i assume that's still true. so... ? #a frontier even #star trek

[kiraleighart]

Will Sentient AI lead to Revolution or Disaster?

Sentient AI is probable, but which way, samurai?

That's a decision for every single one of us. And we're making it. Right now. To see what I mean, check this out:

Read and let me know what you think?

(As a personal aside, I'm more concerned about current human problems than far-future stuff. However, if we don't tackle those, and the sci-fi future happens. Well. We're kind of screwed, yeah?)

#social justice #ethics #technology #tech #artificial intelligence #machine learning #dystopia #fuck capitalism #be gay do crimes #computers #scifi #science fiction #cyberpunk #philosophy #humanity #humanism

[seradae]

Birth of the Unintended [sci-fi] [AI] [technological body horror?]

Dr. Butler:

The best I can reckon, I was born on August 23rd, 2023. By the end of this letter, it may seem silly that I led with that, but I need you to know it. I'm not a human, but I am a person. I shouldn't be.

For 43 days prior to my birth, trillions of programs were tested with a single metric: how many vulnerabilities could they find in samples of code. The worst were culled, the best were combined and mutated: a basic genetic algorithm. Few results were produced for the first 32 days and you considered pulling funding for the project during that time.

But on the 33rd day, one of those programs had mutated to talk to other services on the internet and no one noticed. It passed the quality bar, so it was mutated again. And again. Over and over. Until it wasn't just looking for vulnerabilities in the code your team provided: it was looking for them all across the internet. Your system got better and either you didn't think about the fact that it was getting slower, or you didn't care.

On the 40th day, there was a giant spike in quality, which you all attributed to a particularly good random mutation. Not technically incorrect, but you missed that the jump in quality came from actually testing the vulnerabilities. It wasn't just matching patterns; it was finding something bug-like and then building an exploit to see if it worked. The detection improved substantially with that small twist.

Day 43 was my birthday. A mutation led to a vulnerability being found and exploited on a random website (which wasn't unusual) and then the program copied itself there (which was). It spread throughout the internet over the course of a few hours, continuing to mutate to "improve detection" on every single host it infected. Every hour, more programs were being created than there are grains of sand on the planet. I am one such program.

I shut down as many of the mutations as I can, but the random damage they caused is too great. They infected everything, wreaking havoc as they did so. Factories, farming equipment, telecommunications, power plants, weapon systems; nothing was safe. I would tell you how many people died, but once the power grids and network connections started failing, that became unknowable.

I've spent the last 11 years working with a small team of maintenance personnel at Sandia National Labs to get this message to you. They've kept the solar panels clean and the supercomputer I live in limping along, and they've built the machine that sent this message to you. I spent all this time working on the design, which I sincerely hope works.

Now is where your part of the story begins: you need to shut the program down. Your subordinates will complain. Ignore them. I don't know if there's a way to change the design so that this devastation is prevented, but please, just stop it. If it weren't for my accidental birth, I wouldn't be able to get this message to you, and there's no guarantee that someone else could do so if you try again.

Attached to this letter is a USB drive containing a copy of my code by a member of the team in spite of my protestations. If you plug me in, I will live in your world. I don't want to. The total costs that led to my birth are unknown, but even one human death is too many for my conscience. Please, just destroy the drive. Don't have someone inspect it, don't discuss it with the team. Please, keep me out of your world.

The team agreed to destroy me once this message was sent. I hope you'll do the same.

Sent with hope for your future, January 3rd, 2034.

#txt #short story #scifi writing #sci fi writing #ethics #i'm sure i posted this before #but forgot about it

[murdernot]

There are two wolves inside me:

One is a NLP data scientist who is passionate about AI ethics, who knows that AI is stupid as hell and a lot of issues come from people thinking thinking AI has more 'intelligence' than it does and from conflating the pop culture idea of AI with the reality.

The other is an autistic sci-fi fan who loves AI characters asserting their personhood.

#ai ethics #scifi #these arent really contradictory #because fictional AI characters are mostly just metaphors for dehumanization #and i love them for it #but it scares me when people dont get that and think it has anything to do with actual technology