The problem of free will is as old as philosophy itself. It has been debated for thousands of years by scholars all over the world, and, to this day, with no consensus. In a nutshell, the problem can be formulated as follows: if the universe as a whole follows some kind of algorithm, how can any of its parts be free to choose their own future evolution?
That the universe is comprehensible and describable by mathematical algorithms is in itself a little miracle. It is also a testament to human ingenuity that we were able to come up with these algorithms and use them to predict the behavior of the universe in many simple circumstances. This may in fact have been the single most important discovery ever made by humans: that the universe exhibits patterns, and that even very complicated patterns oftentimes arise from very simple laws.
The nature of these simple laws is quite subtle. In 1814 Laplace, the great French scholar, wrote an essay in which he envisioned an intellect so vast that for him nothing would be uncertain and the future, just like the past, would be present before its eyes. In modern terms we would think of such an intellect, or demon, as a modern computer with near infinite computing power. If we had complete knowledge of the laws of physics, we could use such a computer to predict the future just as well as we can see into the past. The effectiveness of Laplace's demon rests on the assumption that the fundamental laws are deterministic, meaning that the state of the universe at any given time completely determines the state of the universe at any future time.
Clearly, in a deterministic universe there is no place for free will. At least the free will that matters. But in order to have a meaningful discussion on anything, we better define the thing that we want to discuss. So, what is a reasonable definition of free will? A quick look at Wikipedia tells us that free will is the ability to choose between different possible courses of action unimpeded. This looks like a reasonable enough definition of free will to me, so let's stick with that. Now, let's think about the definition carefully. What this is really telling us is that free will is the ability for an agent to choose two different things in the same exact situation, meaning that if I have to choose between chocolate or vanilla ice cream, and I choose chocolate, and someone rewinds time and makes me choose again, I may choose vanilla this time. But this is simply incompatible with a deterministic universe as defined above, because the future is completely determined by the past: if the initial state of the universe is exactly the same (in the ice cream example the initial state is the same, because we are rewinding time), there simply cannot be two different outcomes. I will always choose chocolate, no matter how many time I rewind time. Future and past are one, always present before the eyes of God. Or Laplace's demon, whichever you prefer.
However, things are never so simple. Please meet physics's scourge, quantum mechanics. Quantum mechanics is weird, as you may have heard. Particles really behave like waves, unless you observe them. Then they inexplicably and suddenly turn into particles again, in a process that is called the collapse (of the wavefunction, to be introduced shortly). So, quantum mechanics is weird. But why does that matter for our discussion? Because quantum mechanics is fundamentally non-deterministic. Mathematically, particles in quantum mechanics are described by an object called the wavefunction, which roughly gives the probability (for the pedantic, it's a probability amplitude really) of finding the particle at a particular location in space if you decide to observe it. The equation describing the evolution of the wavefunction, the celebrated Schrodinger equation, is completely deterministic, just like any of the equations describing classical waves in water or air for example. The non-deterministic quality of quantum mechanics manifests itself with the measurement process. The wavefunction evolves undisturbed according to the deterministic Schrodinger equation, until a measurement is performed, upon which the wave turns into a particle and the wavefunction randomly collapses according to the probability distribution that it encodes. One could say that quantum mechanics suffers from multiple personality disorder: it is a deterministic theory when nobody is watching, but it suddenly turns probabilistic if a measurement is performed.
We now believe that the world at the most fundamental level is quantum. This strange mix of determinism and randomness in quantum mechanics then implies that our universe is fundamentally non-deterministic, at least if measurements are performed on a regular basis. Now, I don't want to enter into the separate and very delicate issue of defining what constitutes a measurement in quantum mechanics, because it would lead us astray. Suffice it to say that quantum mechanics introduces an element of unpredictability in our theories about the universe. What are the consequences for free will? It is a common misconception that quantum mechanics, and its non-determinism, can accommodate free will. In fact, pure randomness is at odds with free will as much as pure determinism. If your "choices" are the result of external influences outside your control, and if I can predict them with 100% accuracy every time, your will is not free, as anyone would agree. In the same way, if I cannot predict your choices just because they are completely random, no one would call that free will either. Consider the thought experiment of me "choosing" between chocolate or vanilla ice cream and then rewinding time to see if there is any difference. In a quantum world, if I first choose chocolate and then rewind time, at the second round I may choose vanilla, not because my will is free and I choose to, but simply because a quantum process in my brain makes a neuron fire this time that didn't fire before.
Take a look again at Wikipedia's definition of free will: the ability to choose between different possible courses of action unimpeded. In a deterministic universe, different courses of action are a priori impossible as all events are completely determined by previously existing causes. In a purely random world, different outcomes are possible (just think of the many-world interpretation of quantum mechanics, where the different outcomes are physically realized as different parallel universes), but the ability to choose is nullified as the outcomes are randomly selected. Quantum mechanics exhibits a peculiar mix of determinism and randomness, but nothing else, and is therefore as incompatible with free will as classical mechanics is. Moreover, the unpredictability in quantum mechanics is not due to chaos or a complete lack of patterns. Quantum mechanics is still an (extremely elegant) mathematical theory, and even if Laplace's demon cannot exactly predict what you are going to have for breakfast, it can for sure predict all the possible outcomes and the associated probabilities with infinite accuracy. So again, no room for free will.
In a quantum world, that is a world in which determinism and randomness are the only two causal relationships between events, every action we ever took was influenced by a series of external factors going back to the day we were born, or even to the beginning of life on earth if one wants to take into account innate biological qualities encoded in genes. Note that it is crucial to agree on the definition of free will. If by free will one means the freedom to do what one wills, then free will trivially exists, because we are constantly doing what we will. But that is not what the notion of free will entails, not if we want to have a meaningful discussion on the subject at least. Free will should really be understood as the freedom to choose what one wills, and then act accordingly, or just the freedom to be liberated by any will, which is the same as saying the freedom to be liberated by the laws of physics (as we understand them, more on that later), i.e. a full-blown miracle. The reality is that our actions are determined by our wills, which are in turn determined by countless other factors outside our control. As Schopenhauer famously said: Man is free to do what he wills, but he cannot will what he wills.
Neuropsychological findings corroborate this view. One of the pioneering experiment in the field was conducted by Benjamin Libet, in which he asked each subject to choose a random moment to flick their wrist while he measured the associated activity in their brain. Astoundingly, Libet found that the unconscious brain activity leading up to the conscious decision by the subject to flick their wrist began approximately half a second before the subject consciously felt that they had decided to move. The conscious awareness, in a sense, was a “story” that the higher cognitive parts of the brain told to account for the action.
Now, before closing this overlong blog post, a couple of caveats. First, Laplace's demon is assumed to have an infinite intellect, or infinite computing power, and is therefore able to predict every outcome (even if probabilistically) with infinite precision. This is fine in theory, but it will never be realized in practice. A normal computer has finite accuracy and makes round-off errors which accumulate and may affect predictions, especially if one combines quantum's uncertainty with chaos theory. A brain is a very complex system, and most likely chaotic in the technical sense, meaning that slight changes in initial conditions produce exponentially big deviations in the predictions. Therefore, if the precision in recording the initial conditions is less than perfect, a real-life computer will never be able to accurately predict anything beyond the Lyapunov time, the characteristic time it takes for a dynamical system to becomes chaotic. So, even though you don't really have free will and your actions could be predicted in principle by a powerful enough computer, what if such a computer doesn't exist and will never exist? What if the size of a computer that can predict human behavior exceeds the radius of the observable universe? What if building such a computer violates some fundamental law of physics? Would you say that you have free will in that case? That is up to debate, but I think there is an argument to be made in this case in favor of free will. Replace the metaphysical question of "are our actions really free?" with the very concrete one "can we build a computer inside our universe that is in principle able to predict any choice a human can make?". There you have an operational definition of free will that escapes the realm of speculation and philosophy and enters the realm of science.
The second caveat has to do with the fact that we don't have a complete picture of the physical world yet. The theory of quantum gravity has so far escaped us, and there are many puzzling features of our universe that our theories cannot explain. Therefore, we must be humble and admit our ignorance, recognizing that there may be a "third way" between determinism and randomness. Maybe the unified theory of physics contains elements that are intrinsically non-algorithmic, thus allowing for the existence of some form of free will. All we can say now is that the laws of physics as we understand them are incompatible with the strong notion of free will I delineated above. We must not forget however that our picture of the natural world is incomplete. Incidentally, the concept of a soul may also be considered a third way. In this case each living being is endowed with an eternal and incorruptible essence which defies the laws of physics and is therefore neither deterministic nor random. This essence would comprise all mental abilities, including the freedom to choose and to will the wills. More realistically, one could imagine the strong emergence of non-deterministic, non-random laws in certain physical systems (like brains) that are in principle non reducible to the microscopic laws, which do not admit free will. These new qualitatively different laws may then allow for free will even though the microscopic laws that generate them do not. A bit contrived, but possible, I believe.
Finally, remember the operational definition of free will? Are we capable of predicting human behavior now? Not exactly, but we have seen progress in that direction lately. At present we don't have computers powerful enough to predict every single action a given person will take, not to mention the detailed knowledge of the inner workings of the brain that would require. What we can do however, and what we are seeing, with the rise of artificial intelligence and biotechnology, is the development of algorithms that are increasingly more capable of hacking humans, that is algorithms that understand certain behavioral mechanisms better than the average person and are then able to exploit that knowledge to manipulate or enhance individuals. In the words of Israeli philosopher and historian Yuval Noah Harari, we are not just in a technological crisis. We are in a philosophical crisis. The philosophical framework that served us so well in the past centuries, based on human agency and individual free will, is being challenged not only by abstract considerations like the ones in this post, but more concretely by recent technologies that emerged from the combination of biology and new learning algorithms, fueled by the immense amount of data that certain companies have collected on a large portion of the population. Believing in free will may in fact expose us to these algorithms and make us easy preys to hacking techniques that may go from selling us products through adds to manipulate the electorate to vote for a particular candidate.
And what about the moral sphere? What will be the consequences for our justice systems, which has free will as its universal and persistent foundation? Without free will, we would have to rethink ethics and law completely, as people would be guilty but not really responsible for their actions, because they couldn't have chosen otherwise, or if they could, it wasn't up to them but just a random occurrence outside of their control. People would still need to be incarcerated for murder of course, but our whole perspective on the punishment may change completely, as these people were not really responsible for where they were born, to which parents, or for their genetic endowment. Maybe that will help us understand that prison should really be used to reform instead of punishing inmates. Hopefully that will make us more humane towards our less fortunate brothers, and more sympathetic to the common human condition.