Sci-fi Today: Mental image reconstruction

Can we read minds? Well, maybe. Sort of. Not exactly.

From Professor X to Aasimov, reading minds has had its claws sunk deep into the world of sci-fi for decades. Or as Shen et al (2019) puts it:

“While the externalization of states of the mind is a long-standing theme in science fiction, it is only recently that the advent of machine learning-based analysis of functional magnetic resonance imaging (fMRI) data has expanded its potential in the real world…”

Gotta say, I like their intro more than my own, but we work with what we’ve got. Welcome back to Sci-Fi today! If you’re new here, let me explain: Every now and then, we talk about something either in sci-fi already or something ripe to be used in sci-fi, and I give you the details about where our modern science is currently. First, I’ll tell you some of the science, then I’ll tell you how I think this could apply to fiction, and then… well, then the challenge will be laid out and I’ll come back a week from now with a new story inspired by the science!

Quick note: This week’s discussion is actually inspired by a conversation I had a few months ago with Substack’s own

Honeygloom

; a particular treat of a person to interact with who brought my memory back to something I had forgotten until then. I intended to attach the original discussion that sparked all of this, but I can’t seem to find it. Instead, thank you very much to Honeygloom, and let’s get into this!

You’ve been warned.

The Science

I want to start off the science section with something I don’t usually start with. To put it simply, this is easily the most mathematically complex thing I have ever explored in this series. To be frank, it took me months to work on this because even as a neuroscientist, this specific part of the field is out of my depth. BUT that does not mean I can’t introduce the concept to you, hand you the resources, and give you the opportunity to do a little bit of exploration yourself. I will also note that while I’ll present some of this to you, try not to get too overwhelmed if it doesn’t entirely make sense. Don’t worry, this is complicated stuff! I’ll explain the basics, attach sources for my details, and be available to answer any questions (though the answer may be I don’t know either). With all that out of the way, let me tell you a story.

Junior year of undergrad in a sensation and perception class, we were learning about the way that when we ‘see’ something and encode that visual experience as a memory, when we recall it, we activate association cortices (parts of the brain associated with the primary visual cortex and the occipital lobe more broadly, but not the primary regions for vision) that are involved with the visual system to reconstruct those images in our minds. The professor then went on to show a picture of a coastline. With excitement on her face, she went on to say that that wasn’t a real picture of the coast. It was actually a reconstruction of a picture that someone imagined! I couldn’t believe that was possible! Was it really possible?

Well… no. It wasn’t. Not yet at least!

See, I spent a lot of time trying to find that picture by just scouring google for it, and had no luck, but after a few weeks, I moved on and decided to start looking through scholarly articles and what do you know… I found it. See, in 2009, Naselaris et al. showed that they could sort of reconstruct images from the human brain, but it wasn’t as cool as it sounds. In these early attempts, It was less that we were reading minds and more that the brains were being used as a complex filter of images being presented to them during an fMRI scan. So they would show someone an image. The fMRI monitors brain activity, then a complex algorithmic catalog will reference thousands of images, and match what the algorithm is interpreting through the brain filter with an image in that catalog. Still, pretty cool! And if we’re being honest, it worked pretty well. Let’s have a look at this early attempt…

Our far left image was the viewed image, whereas the images to the right were the developed images.

Not exactly mind reading, but still pretty impressive. But that was 16 years ago. We must’ve made much more progress since then, right? Well, good news! We are getting closer, though if you ask me, as we get closer to mind reading the results become… creepier. Stick with me here.

Flash forward to 2019! Shen et al. started using Deep Neural Networks (DNN) at the end of our brain filter to layer results over each other until we get a composite image that roughly approximates the original image. If we were just comparing the photos, we would call this a step back, but actually this is much more complex. They’re doing away with a reference catalog of photos, and instead they’re relying entirely on a neural network to construct the image based on neural activity. Now that is starting to look like mind reading! But there’s one problem. The images are absolutely terrifying. Here, have a look at how they describe their process:

I mean, I did warn you, didn’t I?

I know, I know, pretty scary but progress isn’t always pretty I suppose. So we don’t need reference catalogues of images, and now deep generators and neural networks can come together to create the rough approximation of a thing, but as you can see from the image, this isn’t mind reading, this is still brain filtering. Here, let’s have a look at what happened when Shen et al. asked participants to picture 1 of 20 photos they were shown before the fMRI started. That’s right, no brain filtering here. Participants are now recalling photos they saw previously!

Aaaannndddd we’re not quite there yet, are we?

Okay, so in 2019 we still couldn’t quite read minds, but what about something a little more recent? Well, we have one more paper I want to share with you, and funnily enough, this paper takes a lot of pride in saying they took Shen et al.’s methodology and improved it! You hear that? They’re claiming they took this 2019 methodology and made it better. In fact, they say it a lot. Majima et al. (2024) comes in hot with some complex algorithmic neural network methodology that does indeed read minds… kind of.

Using something called Bayesian estimation, they were able to refine Shen et al.'s methodology into something more complex and accurate. Bayesian estimation is an algorithmic, formulaic process that is outside of my realm of understanding, but if you’re into that kind of stuff, I fully encourage you to check out this paper because wow. Wow. Here’s how they show their new methodology:

Be honest, does that look like an airplane to you? Don’t worry, we don’t need to talk about the goat.

As you can already see, yes the neural network is indeed decoding the images, but is it mind reading? Maybe we aren’t as close as we’d like. The Majima paper is definitely barking up the right tree. It’s pretty clear here that they have improved on Shen et al.’s work and it looks like the recall of the image is looking as good as in Shen’s paper when they were actively looking at the image and it was being ‘filtered’ through the brain rather than being interpreted from just the brain.

Fortunately for us, the Majima paper really cared about improving on specifically the Shen paper, so they included direct comparisons so we can see exactly how much progress was made in those five years:

the conv2-fc6 are different layers of the image that the DNN developed. I would say for our sake the ‘all’ section is more important than the specific layers of analysis.

Now, we can see that we’re getting close! Five years of progress did make a difference, didn’t it? Now, I want to note, they both have algorithms that score how accurate the constructed image is to the original image as well as human scores for the percent accuracy of the image. The goal in the Majima paper was to get a higher percent accuracy than the Shen paper - 50.4% when imagined. Majima’s percent accuracy did in fact increase by 25.2% (75.6% accuracy!)! So, how close are we to mind reading? As of 1 year ago, I’d say we’re getting somewhere. Close enough in fact, that Majima et al included this at the end of their discussion section:

“... our proposed framework potentially paves the way for mind-reading technology, underscoring the necessity of addressing ethical concerns associated with mental privacy. To better understand the limitations and potential capabilities of our proposed framework, we present two critical research questions for future exploration: 1) Can brain decoders be trained without the subject’s cooperation? 2) Can imagery spontaneously arising in the mind be accurately reconstructed?”

I for one find that first question a tiny bit concerning, but they were discussing the ethics of mind reading, and wow, I think we’re at the point where that should be discussed. I’d rather have those ethics sorted well before we’re there rather than be actively trying to sort it out as the technology is being used (as we’re seeing in many cases with the rollout with many uses of artificial intelligence).

Okay, that was a whole lot of science, wasn’t it? Let’s get into the fiction!

The Fiction

Let’s be honest here, sci-fi has had mind reading for as long as any of us can remember, so I don’t really need to tell you how to write sci-fi with mind reading. I’ll admit, The Foundation series has some of the strangest, most fascinating mind reading to me. And it turns out that if there’s one thing Asimov got right, it's that reading minds involves a lot of math.

If you were going for a hard sci-fi approach, it looks like deep neural networks could be the key to you approaching your story with a little bit of realism that grounds the story. Who would’ve thought algorithmic DNNs could be the key to making your fiction a little more real? This does feel like a good time for our regular reminder though. As I always say, science is the scaffolding that you build around. Let this be something that you build on! Don’t let the science get in the way of your fiction. Tweak it, play with it, speculate with it, completely twist it! That’s what’s fun about science-ing up your fiction! Make it yours! And I already know how I’m making this concept mine.

There’s something I found through learning about this that I didn’t expect. As you read through the Majima paper, it becomes clearer and clearer. See, in the Shen paper, the images are creepy and that gave me a bit of a laugh, but I could move on to focusing on the science. As I dug into the Majima paper, I found myself feeling concerned. This wasn’t just a novel fun research method. They really were asking the ethical questions of reading an unwilling participant's mind. I would love to poke at the author’s brain and discuss what they were thinking in regards to ethics… maybe I will one day, but for now I can say this much: Mind reading is actually kind of terrifying? Especially in its current infancy, though what this tech could be used for may be even more terrifying in the future, who knows? That’s where speculative fiction comes in after all.

Creepy pictures aside, Sci-fi currently is rife with discussions about the ethics of AI and replacing workers with robots, and all sorts of other cyberpunk technohellscapes. I unfortunately think mind reading does indeed fit into this. Something Shen et al. mentions in their paper is that they are certain that if you tweaked the algorithm, you could do more than look at the images people are imagining, but look at other parts of their perceptual experience…

“Our approach provides a unique window into our internal world by translating brain activity into images via hierarchical visual features. Our method can also be extended to decode mental contents other than visual perception and imagery. By choosing an appropriate DNN architecture with substantial homology with neural representations, braindecoded DNN features could be rendered into movies, sounds, text, or other forms of sensory/mental representations. The externalization of mental contents by this approach might prove useful in communicating our internal world via brain–machine/computer interfaces.”

That sounds really cool when you think about it that way, but I can’t help but wonder about the more concerning interpretations of this. What if your manager at work could track your thoughts as you worked? How long before thinking about non-work related things lead to getting your pay docked? That feels a little too close to home, but we’re talking about sci-fi - the home of speculative fiction - so let’s dig in deeper.

What if mind reading was used to track what matters most to you? You know the classic speculative question? The aliens come down and choose you to make an extremely difficult choice. You must choose something, but choice isn’t an option. Instead their mind reading algorithm decides based on the desires it interprets in your mind?

There’s also the idea that Honeygloom and I had discussed all those months ago that lead to this: Advertisements in your brain. Let’s leave this at speculative please - I for one hate ads and they make me less likely to buy something - but what about if they could read your mind to interpret what you wanted? Ads might get a little scary and that’s bad for us as people, but good for some speculative sci-fi!

Okay okay, lets try one on a more positive note. What if this could be used to help people revisit significant moments in their lives? Imagine a person getting to experience the last time they saw their mother again and getting closure they missed before. I think you have the beginnings of a heartfelt story there!

And then… there’s the idea that I have. I can’t share it because I’ve been sitting on it for weeks and I’m going to write it up and hopefully absolutely dazzle you all next week with an awesome short story! I for one cannot wait to show you what I’ve got brewing over here. Trust me, it’s a doozy.

Until then, I encourage you to take some of these ideas to heart! Are you feeling inspired by the science? Or maybe just the creepiness of the photos generated? Maybe use this as some inspiration for a story! If you do, please tag me in it as I would love to see what comes of the idea! Consider this your unofficial prompt!

In fact, let me leave you with one more quote from the Majima paper. I think if you take this one to heart, you may find some really awesome inspiration hiding in there…

These promising results not only highlight the efficacy of our proposed framework but also suggest its potential as a unique tool for directly delving into the subjective contents of the brain, encompassing phenomena such as illusions, hallucinations, and dreams.

What a way to end a paper.

Conclusions and one more thing!

In the meantime, look forward to next week! I can’t wait to share with you what I think is going to amount to a real fun and creepy story!

I also wanted to mention that I’ve had a few people (both irl and on substack) ask me about this recent idea that non verbal children with diagnoses of autism spectrum disorder have some sort of telepathy. Although this is outside of my usual points of discussion on Substack, my specialty in neuroscience is the intersection of autism and early life stress during development, so I do know quite a bit about this topic. I want to say that from the perspective of neuroscience there is no evidence that this is real, and though I have not watched ‘The Telepathy Tapes,’ the evidence I’m aware of from this podcast are suspect at best.

Although I cannot discount any sort of supernatural or spiritual possibilities, I can tell you with what we know about the human brain, there isn’t anything at this time that could make a human being telepathic. I do find this belief a little concerning as it puts unrealistic expectations onto children and also ads mysticism and confusion around what autism even is, but that’s a discussion for another day.

I’ve considered discussing the neuroscience of ASD in more detail, but this doesn’t really feel like the space for that, so I’ll leave it at this for now. I do want to note that if people want me to go into the neuroscience of neurodevelopment, I certainly will, just let me know. Until then, we’ll move onto other things.

Okay, that’s that for today! Next week, we’ll dive into a new short story! And the week after that, we’re back to The Apocalypse Saga! Have you been keeping up? Not to toot my own horn, but it has been a really fun ride, so maybe check it out here!

Next week, we’ll also finally be adding an all new horror story to the horror story playlist, so have a look here!

In the meantime, consider subscribing if you aren’t already and maybe leave a tip if you have the ability and the inclination!

Oh, one more thing! Here’s the sources so you can take a little look for yourself at the papers:

Naselaris et al. paper

Shen et al. paper

Majima et al. paper