TLDR: I’ve recently wondered about whether the Universe is truly random, and I thought I’d write down a few thoughts on the subject. As a heads up, this post is more about sharing a personal journey I’m on than teaching a skill or tool (unlike many other blogposts). Feel free to chat with me about these ideas on Twitter as I’m still working through them myself and I’d love to hear your perspective.
I typically tweet out new blogposts when they’re complete @iamtrask. Thank you for your time and attention, and I hope you enjoy the post.
The Oxford English Dictionary defines randomness as “a lack of pattern or predictability in events”. I like this definition as it reveals that “randomness” is more about the predictive ability of the observer than about an event itself. Consider the following consequence of this definition:
Consequence: Two people can accurately describe the same event as having different degrees of randomness.
Consider when two meteorologists are trying to predict the probability that it will rain today. One person (we will call them the “Ignorant Meteorologist”) is only allowed a record of how often it has rained in the region between the years 1900 and 2000. The second person (the “Smart Meteorologist”) is also allowed this information, but this second individual is also allowed to know today’s date. These two individuals would consider the likelihood of rain to be very different. The Ignorant Meteorologist would simply say, “it rains 40% of the time in the region, thus there is a 40% chance of rain today.”. What else can he/she say? Given the information provided, the degree of randomness is very high. However, the “Smart Meteorologist” is more informed. He/she might be able to say “it is the dry season, thus there is only a 5% chance of rain today”.
If we asked each of these meteorologists to continue to make predictions over time. They would each make predictions with different degrees of randomness (one higher and one lower) in accordance with the availability of information. However, the event is no more or less random in and of itself. It’s only more or less random in relation to an invidual and other available predictive information.
Perhaps this makes you feel that randomness is no longer real, that it is only in the eye of the beholder. However, I believe that the context of Machine Learning provides a much more precise definition of randomness. In this context, one can think of randomness as a measure of compatability between 3 datasets: “input”, “target”, and “model”.
Input: is data that is readily known. It is what we’re going to try to use to predict something else. All potential “causes” are contained within the input.
Target: is the data that we wish to predict. This is the thing that we say is either random or not random.
Model: this “dataset” is simply a set of instructions for transforming the input into the target. When a human is making a prediction, this is the human’s intelligence. In a computer, it’s a large series of 1s and 0s which operate gates and represent a particular transformation.
Thus, we can view randomness as merely a degree of compatibility between these three datasets. If there is a high degree of compatibility (input -> model -> target is very reliable), then there is a low degree of randomness.
Now that we’ve identified what I believe to be the most practical and commonly used definition of randomness, I’d like to introduce the bigger version of randomness, which we’ll call Randomness (with a capital R). This Randomness refers to whether, given infinite knowledge and intelligence, a future event could be predicted before it happens (a “model” dataset could exist for that event). This Randomness also implies whether or not something is caused by another thing. If it is NOT caused but simply IS of its own accord, it is (capital “R”) Random.
The simple answer is that, from our perspective, it has a degree of randomness that is rapidly decreasing. We are consistently able to predict future events with greater accuracy. Furthermore, predicting outcomes given our interaction allows us a certain degree of control over the future (because we can choose interactions which we predict will lead to the desired outcome). This plays out in the advancement of every sector: agriculture, healthcare, finance (bigtime), politics, etc. However, because we cannot predict the Universe entirely, it is somewhat random. (lowercase “r”)
Whether the Universe is uppercase “R” Random is a different question entirely. We can, however, make some progress on this question:
Claim 1: Causality exists
In short, we can predict things with far better than random accuracy. Thus, some things tend to cause other things. We might even be able to say that some things absolutely cause other things unless affected by some unpredictable randomness, but we don’t even need this bold of a claim. Simply stated, much of the Universe is probably causal because we can predict events with better than random accuracy. To claim otherwise would be extremely unlikely and would imply that the entirety of human innovation and intelligence throughout history leading to prosperity and survival was simply a coincidence. It’s possible, but very unlikely. We’ll go ahead and accept the notion that causality exists in the Universe.
Claim 2: The Universe is not entirely Random.
For all things in the Universe that are not Random but are instead caused, randomness (lowercase) in their behavior is a result of either random or Random objects exerting upon it. Thus, asking whether the Universe is Random is about asking whether or not there exists a Random object within it. It is not about asking whether every object is inherently random, because cause and effect can transfer the unpredictable behavior of a Random object across the Universe via things that are merely random.
Claim 3: Because we can observe cause and effect relationships, the Universe is, at most, a mixture of Random and causal objects, and at least, exclusively made of causal objects.
This brings us to the root of our question. When we repeatedly ask “and what caused that?” over and over again, where do we end up? Well, there are 4 possible states of the Universe (finite/finite space/time)
-
Finite Time + Finite Space If time is finite, there was a beginning. If there was a beginning, there was a TON of Randomness which began the Universe. Thus, Randomness at least has existed within the Universe (although whether it still exists is less certain).
-
Finite Time + Infinite Space (see above)
-
Infinite Time + Finite Space Laws of entropy determine that if this was our Universe, you wouldn’t be reading this blogpost as all the energy in the Universe would have disipated to equilibrium an infinite number of years ago. I suppose there are counter arguments to this, but I don’t personally find them particularly strong. We have a rather large amount of empirical evidence that energy tends to dissipate (perhaps more empirical evidence for this than for any other claim in the Universe?).
-
Infinite Time + Infinite Space This state is interesting because there’s theoretically an infinite amount of energy (inside the infinite space) alongside an infinite amount of time for it to disipate. Thus, while I don’t have solid footing to say that this Universe does not exist, I think we can make a reasonble case for (capital R) Randomness in this Universe. Specifically, because the state of the Universe at any given time “t” is, itself, infinite, there are an infinite number of potential causes for an event. Thus, every event is Random because there are an infinite number of potential causes for any event. It may be asymtotically predicatable given proximity to some causal events playing a more dominant role, but in the limit every event is Random.
Conclusion: Randomness with a capital R either exists or has existed before in the Universe because all 3 plausible configurations of the Universe necessitate events that have no cause, and an event with no cause cannot be predicted and is therefore Random.
To the extent that there are varying degrees of randomness, there are varying degrees of ignorance. Someone might say, “I know for sure that the temperature today will be between -100 degrees and 200 degrees”. This statement can be said with a high degree of certainty, even though the same person will have almost no ability to tell exact temperature down to the 10th decimal point. Thus, degrees of randomness correspond to degrees of ignorance about a problem. In this case, there is a degree of randomness within the temperature on any given day, but there is also some structure, allowing wide ranges to be predicted with great accuracy.
Additionally, it follows that two different individuals can perceive the same pattern as being random or not. Perhaps this is the best case for randomness simply measuring the ignorance of an individual. Two individuals of varying degrees of intelligence can look at the same pattern and accurately describe it as more or less random. A laymen might say “it rains 40% of the days of the year, thus today’s chance of rain is 40%”. However, a meterologist would find less randomness in the patterns of rain within a region than a non-meterologist. A meterologist might claim, “well, it is the spring, and in the spring it rains on 60% of days.” Thus, the same pattern can be perceived as being more or less random by two individuals. One person has identifed an additional piece of structure that the other has not realized (the season of year). Neither person is incorrect. Both accurately describe the degree of randomness in their world with respect to the rain. Given two individuals with varying amounts of information/intelligence, the same phenomenon can have a different degree of randomness. –> <!– Point 1: Randomness does not exist, only lack of information and/or intelligence.
Despite how much like the definition above, I’d like to refine it a bit with a different perspective. “pattern or predictability” is really a state of relationship between three datasets. Something is predictable if one dataset can be transformed into the other using a method described in the third. In the case of Machine Learning, the two datasets are our “input” and “target” datasets, and the third dataset is our “model” (or a set of instructions for their transformation). Thus, randomness is a measurement of compatability between these three datasets. If the “input” dataset is lacking or the “model” dataset is inaccurate, there is a high degree of randomness. If, however, the input dataset can be perfectly transformed into the target dataset using the model, then randomness is low”.
This fits with the more common description of “randomness” where we simply consider the “model” to be a human. Either the human has the ability to take input data and transform it into a target or not.
Point 2: Randomness is only a measure of ignorance.
Consider perhaps the most extreme example of this: random number generation. In Computer Science, it is common practice to have fully determinate random number generation. What does this mean? It means that we can generate the exact same random numbers multiple times. To the outside eye reading the numbers, no identifiable pattern could be found. However, if one knows the key and the method for generation, one can produce the sequence exactly. Consider this example:
This begs the question. Are these numbers random or not? If I simply showed you a sequence of numbers:
you could confidently say that they are quite random. Even more so, if I asked you to predict what the next number would be, you would have almost no ability to accurately predict it (because the pattern is completely random to you). However, if you have the code that can generate these numbers, you can predict the next number exactly with 100% confidence. So, are these numbers random or not? Well, it depends on whether or not you have the code to generate them. Your measure of randomness is based on your level of knowledge. If you are ignorant, these numbers are ~perfectly random. If you are not, they are perfectly NOT random.
Is the Universe Random?
The simple answer is that, from our perspective, it has a degree of randomness that is rapidly decreasing. We are consistently able to predict future events with greater accuracy. Furthermore, predicting outcomes given our interaction allows us a certain degree of control over the future. This plays out in the advancement of every sector: agriculture, healthcare, finance (bigtime), politics,… and maybe even religion.
So that answer is pretty easy when asked from this perspective, but the next question is a bit harder. Is the Universe truly Random? We’ll use Random with a capital “R” to denote a randomness that is impossible to predict. In other words, given absolute knowledge of the entire Universe, something that is Random is something that truly cannot be predicted given the state of every/any other object in the Universe. Furthermore, it is also something that cannot be “caused”. Something that is Random is simply a standalone “effect”.
To answer this question, let’s start with what we know. There are some events that we can predict with better than random accuracy. Thus, we know there is some notion of causality in the Universe. The real mystery isn’t what we can predict, it’s the origin of what we cannot predict. Do we make mistakes in our predictions based on a lack of knowledge/intelligence or is there a truely Random object somewhere in the Universe that continually “shakes things up” (think “butterfly effect”) adding noise to the causality.
Now, let’s assume that science and technology progresses enough to where we learn every cause/effect relationship out there. Where does this put us? In my mind, this breaks down into 4 Universes. One with/without one or more sources of Randomness, and one that is either finite or infinite. Let’s take these four scenarios one by one.
In my opinion, we can eliminate one of these Universes as a contradiction. A Universe that is truly infinite but not Random is impossible. Why? An infinite Universe means that there is an infinite number of potential causes. An infinite number of causes with non-zero probability is Random. No amount of study or knowledge will ever quantify the state of all relevant causes. The probability of an event can only be asymtotically approached. Thus, a Universe must either be Random or finite.
Searching for Randomness in the Universe
Well, one place to start is simply where you are. What is causing the thing in front of you? Several things? What caused those? Many more things? What caused those? Before long, it should become quiet evident that there is simply too many things for you to ever fully understand everything. Even just holding it in your brain is impossible. Collecting the data is impossible…
This is actually an interesting place to stop and smell the roses. Remember, randomness is not about reality. It’s about your level of knowledge. Furthermore, we just quickly realized that your level of knowledge (in isolation) is certainly too small to ever account for the randomness of the Universe. Thus, no matter how hard you look (on your own), you will undoubtedly never be able to chase this causality train to the beginning. However, mankind has invented a tool for this: the historical record.
The historical record is an amazing thing. It allows me to search for causality my entire life, write down the causes I find, and pass it on to the next individual. Thus, the next person can pick up where I left off. Note, the scientific record is an important subset of this record, but truly we pass on useful information about reality and the Universe through far more than just scientific journals.
So, what does it mean for you to take the ideas of someone before you? Well, the whole point of the system is that you don’t have to go investigate these things yourself. Instead, you choose to have a degree of belief that they are accurate findings. Furthermore, this reduces the randomness you see in the world substantially. In fact, I’d bet that 99% of the non-randomness (structure) in your world is a result of someone else teaching you what they have learned themselves. Reading, writing, arithmetic, culture, occupation, science, art… all these things are passed down from generation to generation. Furthermore, these are how you interpret reality. Even our earlier analogy of meterology. No modern meterologist re-discovers everything he/she uses to predict the rain. Instead, they reduce the randomness in their predictions by believing that others have performed accurate investigations themselves. Now, what is randomness? It’s a degree of intelligence about a pattern. To simplify, belief reduces randomness.
Now, you might slightly reject this statement (frankly, as I do). After all, we have empirical evidence on the accuracy by which our own beliefs may predict the future. If we take on random beliefs, we’re going to find out pretty quickly. If I suddenly believe that english letters are actually math numbers and vice versa, I’m going to have a really hard time predicting reality based on what I read. So, there’s this notion of testing beliefs based on how well they are able to help us predict our world. We are more likely to accept a belief if it is able to help us predict the future events.
Damn… the human condition… what an amazing limitation.