I have a recurring dream. No, sadly it’s not that interesting. I’m on a plane and as I look out of the window, I suddenly notice that we are at ground level and stopping at traffic lights. I realise I am in a dream, as 747s do not generally obey traffic lights and zebra crossings on transatlantic flights.

Then, given the dormant nerd in me, I start to actively marvel at the reality of the dream, how real it all seems. I even start to do little logical tests — as my dream is supposedly being created by my brain, should it be able to create unexpected conclusions?

So it all got a little confusing on my last overnight flight when I started watching the Leonardo di Caprio film, Inception.

For any of you who have not seen it, it’s about a man who can go into people’s dreams and discover things from inside them. It all gets a bit complicated because in the dream, people can have dreams and so on.

This dream recursion gives the film its central dilemma: how do you know if you are still in a dream or back in reality? Of course, just as I was pondering this complexity I fell asleep. Or did I?

The problem with all of this is, of course, that the very reality you are measuring in your dream is unreliable. What you hold of your dream when you wake up is reliant on your memory. So you remember thinking in your dream that it looked so real and took two hours, but is that memory real, or just a set of memory by-products post-created. A remembered two-hour experience created in a second?

In fact, it may be that the dreaming process is about memory. For our memory to function, just like a computer, we need two things: firstly, efficient storage and therefore conceptual data compression, and secondly, the ability to access that information in a meaningful way.

Compression is very important as we need to deal in what matters. Although it may be very impressive to hear of people who can reproduce perfect drawings of every detail of a large building from memory, it is actually a handicap.

We do not need to memorise the position of every brick joint but rather the general impression of the bricks. Not only do we save large amounts of storage by replacing thousands of brick joints’ positions with conceptual measures, but also ‘small, older-looking, Georgian red bricks’ is actually a more useful description.

Some believe that during sleep, we unpack the day’s memories and reprocess them, boiling them down to a conceptual reflux. This process of unpacking and examining ideas, they say, manifests itself as dreams. I find this quite compelling as there is no doubt that I have clearer recollections of things like faces of people I’ve briefly met after a night’s sleep.

Related:

The second part of this process is analogous to the concept of a content addressable memory (CAM). In computing, the normal idea of memory is to put in an address and the memory will give the content that’s stored at that location. In a CAM, the idea is to put in a piece of content and the memory finds where it occurs in the memory.

A CAM is a very important part of being able to process an idea and move along a chain of concepts. This process is similar to the idea of creating an index rather like that found in a search engine, only much cleverer. Once the analysis is done the content becomes far more accessible.

Once we start down this slippery slope of which of our perceptions are real, we can get in real trouble. The feeling of being lost in a world where we can only measure things with a ruler — but where that ruler could be changing length all the time — is taken to its extreme discomfort in the idea of Boltzmann brains.

The idea here is that the universe is big and complex, and there is a principle of physics that says that something that is more ordered is less likely to come into existence.

So, to cut to the chase, your brain is much more likely to have popped into existence instantaneously floating in space with a set of false memories which happen to be encoded in your synapses, than a whole normal universe with lots of these other complex brains, dogs, chewing gum, paper clips, Intel processors, cathedrals and things.

So you only exist for this instant but with a set of false memories of a non-existent universe.

The problem is that the physics says that the floating about thing is far more likely. Now at this point you’re inclined to think you can get the full story on Wikipedia and have a metaphysical crisis, or get some magic mushrooms because this is all too weird.

Or if you really don’t need this kind of stuff in your life, go and do the most prosaic thing you can like tidying up your hard drive to prove the world is really straightforward and a bit boring.

However, the key point is that, in reality, you have no way of knowing that the illusion of your reality isn’t that of a Boltzmann brain floating in nothing, or perhaps this is a dream – or perhaps you are just running out of disk space.

Mike Lynch is the founder and CEO of UK software company Autonomy