There's a huge elephant in the room of the physics community that few people are willing to recognise. Since its inception more than 400 years ago, physics has been widely regarded as the queen of Science, and with good reasons. In the early days of the scientific endeavour, the widespread belief was that while other less rigorous disciplines, like chemistry and biology, carried on classifying compounds and species, physics sought to understand Nature at the deepest level, by organising empirical knowledge in mathematically precise theories. For a long time, the main difference between physics and other natural sciences was that physics, instead of merely describing phenomena on a case-by-case basis, tried to come up with unified descriptions which accounted for a myriad of different and seemingly unrelated phenomena.
Physics made extensive use of organising principles that were extremely simple to state, but had powerful, broad and non-trivial implications. What physicists discovered was that Nature exhibited mathematical patterns, therefore even very different phenomena could often be described by the same set of equations. The textbook example is provided by Newton's momentous insight that the motion of the heavenly bodies is the same as the motion of an apple falling from a tree, two phenomena that seem wildly dissimilar at first sight, but are both described by the universal Law of Gravity, as it turned out. In Newton's age, suggesting that the Laws governing the motion of the heavens were the same as the earthly laws affecting the daily life of peasants was a bold move, and probably a tiny bit blasphemous.
Someone even went as far as proclaiming that all science is either physics or stamp collecting (Rutherford eventually won the Nobel prize in Chemistry some years later, sweet irony). There was more than a bit of arrogance in Rutherford's cutting comment, but back then he could have been excused on the basis that, while not completely accurate, his remark was not that far from the truth either. Things eventually changed, chemistry and biology evolved from their stamp collecting infancy to proper sciences, fully on a par with their more senior and mathematically inclined cousin. Nowadays, chemistry and biology also have their own organising principles with huge explanatory power, just think of Darwinian evolution by natural selection, a scientific theory that encompasses the biological evolution of life (and ideas, see the concept of meme) in the universe.
The fiercest defenders of physics's superiority over all other sciences would still argue that, while scientific disciplines in their own right, chemistry and biology are still ultimately applied physics, since the basic laws governing any chemical or biological process are just the fundamental laws of physics, albeit interacting in extremely complicated ways. Molecules are groups of atoms held together by the electromagnetic force and behaving according to the principles of quantum mechanics, while cells are groups of very complicated macromolecules that ultimately follow the same rules, just at a different scale. Starting from the basic laws of physics, one can then reconstruct the behaviour of larger systems like molecules or cells, at least in principle. This is the reductionist approach that seeks to explain any physical system in terms of its basic constituents, and the philosophical attitude that has dominated physics until today.
All of this sounds reasonable enough, but there is a problem. Complex systems usually exhibit emergent properties. Emergence has become a buzzword in the science community and is often misused as a plausible-sounding substitute for magic. Here, I will use the definition of emergence that seems the most useful to me: a property in a system is said to be emergent if the constituents of the system do not have that property on their own. A typical example is water and its peculiar features, like surface tension and cohesion. These are emergent properties because it wouldn't make sense to attribute them to the individual hydrogen or oxygen atoms: the properties emerge when many such atoms interact with each other in non-trivial ways. Cohesion for example is due to the collective action of hydrogen bonds between water molecules, resulting in electrostatic attraction. One can then see emergence as something that arises from the interactions between the constituents, and is therefore inherently irreducible to the individual components. Sometimes people also distinguish between weak and strong emergence, with the former being the kind of emergence that is amenable to computer simulation, while the latter is not, and thus represents a radical form of emergence in which new basic laws arise at some given length scale that are not only irreducible but even non-derivable from the fundamental ones. We do not currently know of any example of strong emergence in the universe (someone argues that consciousness may be the only known example of this), therefore I will focus on the weak form from now on.
What is the significance of emergence for reductionism? While it may be possible in principle to explain any given phenomenon in terms of physical laws, this is unrealistic and many times impossible in practice. When describing the dynamics of fluids, for example, we don't bother writing down the equations of motion for all the molecules of the fluid. Calling this impractical would be using an euphemism: a gram of fluid typically contains a number of Avogadro (~10²³) of molecules, therefore a fundamental description of the system is out of the realm of possibility. No computer is powerful enough to simulate the individual components of a macroscopic system (at least not a classical one, but what about quantum computers? There's a different story for another time). For that reason, the only feasible approach is to describe the system in terms of its new emergent, coarse-grained properties. This is why we don't talk about electrons and quarks when studying a population of ants, or why we don't talk about genes and hormones when studying world history. While the physical laws of electrons and quarks are what ultimately give rise to ants, and similarly the biological laws of genes and hormones are what dictate the evolution and interactions of human populations, these systems are best understood in terms of their emergent macroscopic properties, like ants' morphology and behaviour or ideologies and states in the case of history. This is what ultimately makes sense of our arbitrary decision to separate science in different disciplines: starting from physics, which provides the most fundamental description of Nature, the other disciplines deal with features that emerge at each new critical scale, like molecules for chemistry, living organisms for biology and ideas for history.
Now that we have clarified the role of physics and its relation to other sciences, we are ready to address the main point of this article. Physics has accumulated over the centuries an impressive amount of knowledge about the natural world. We went from not understanding the motion of the planets or the nature of friction to being able to essentially track the evolution of the entire universe from an instant after the Big Bang until today. This is a monumental achievement of the human race and should be celebrated as such. This, however, doesn't mean that there are no blind spots: there are still many things that we don't understand, both at the fundamental level and at the macroscopic level. There is a difference between the two that it's worth to emphasise. At the macroscopic level, for example, we don't understand consciousness or turbulence, but very few scientists would argue that the reason is because we are ignorant about some new fundamental law that plays a role at those scales. Most agree that the fundamental laws at play in all macroscopic phenomena (consciousness and turbulence for sure) are indeed known, and they usually are a cocktail of electrodynamics, quantum mechanics and gravity, nothing else. What makes these problems complicated to solve is not our ignorance about some new principle of nature, but rather the exceedingly intricate way these basic laws interact with each other, producing emergent behaviours. The complexity of the brain is due to the mathematical pattern of its neurons and synapses, not its biological substrate.
At the fundamental level, on the other hand, there is one thing that still eludes us, which has become the Holy Grail of physics: the theory of quantum gravity. This is a theory that would unify quantum mechanics with Einstein's theory of gravity and give us a way to understand, among other things, the beginning of the universe, and what the hell goes on inside a black hole. It would also probably help us fix the worst prediction in the history of physics, and one of the major problems plaguing theoretical physics right now, i.e. the cosmological constant problem. The problem in simple terms is this: there is a constant of nature, which corresponds to the energy density of empty space, which our models predict having a value tens (sometimes hundreds) of orders of magnitude above the measured one. Mind you, not tens or hundreds of times larger, tens or hundreds orders of magnitude larger. This has rightly been called the vacuum catastrophe and many people believe and hope that the final theory of quantum gravity will give an answer to this (embarrassing) riddle that has eluded theoretical physicists for decades. For the pedantic, one could still retrieve the experimental value of the cosmological constant by invoking a cancelling effect between its bare value and the quantum contributions that arise in perturbation theory. This is not seen as an elegant solution, or a solution at all really, because it would require miraculous cancellations between two unrelated values. In the technical jargon, the cosmological constant problem is a naturalness problem: the value it takes in our universe is not impossible, just extraordinarily unnatural.
Some physicists, however, disagree, and they think that the problem has already been solved. This rebel faction, that has attracted more and more followers with time, argues that we do have a quantum theory of gravity capable of addressing the problem, albeit in an unconventional way: String Theory. String theory was developed in the 70s and is a theoretical framework in which elementary particles are replaced by extended one-dimensional strings. Different particles then, like electrons and quarks, just correspond to different vibrational patterns of the same fundamental string. This simple idea allows one to merge quantum mechanics and gravity at high energies without getting nonsensical results. String theory is then a theory of quantum gravity (and the only reasonable attempt that we have at that) but not everyone believes that it is the theory of quantum gravity. The reason is that string theory has an enormous number of vacuum states (quantum state with the lowest possible energy), typically estimated to be at least 10^500, and these might be sufficiently diverse to accommodate almost any phenomena that might be observed at low energies. This has led people to sarcastically call string theory a theory of anything, rather than a theory of everything.
Let me explain. String theory in its purest form is actually an extremely elegant and predictive theory. It even predicts the particle content of the universe as well as its number of space-time dimensions D! According to (super) string theory, D=10; any other number would render the theory mathematically inconsistent. Now the immediate problem is that we in our universe observe D=4, so it seems like the theory is just wrong and ready for the dustbin. This however would be premature, as there are many ways to make sense of an extradimensional theory. The most common one is to imagine that the extra dimensions are curled up to a very small size, and therefore hidden to us. Unfortunately, the curling up (compactification) of the extra dimensions comes at a cost: if we want the theory to be phenomenologically viable, we have to give up its uniqueness. Alas, the price to pay in string theory is enormous, and the reason why so many people are skeptical about it as the final theory of quantum gravity, as well as its proposed solution of the cosmological constant problem.
What is this price? It can be shown that string theory, while unique and highly predictive at high energies, becomes exactly the opposite at low energies. The low energy structure of string theory is actually that of a multiverse, as first noted by one of the fathers of string theory, Leonard Susskind. String theory contains a vast number of solutions, each physically realised in a different pocket universe with its own dimension, particle content, and its own set of values for the fundamental constants of nature. Susskind called this collection of low-energy vacua the landscape of string theory, in analogy with the notion of fitness landscape in evolutionary biology. The analogy is particularly fitting, as one could imagine the particular geometrical structure of the extra dimensions as providing the "DNA" for the observable 4D universe. Pocket universes in the landscape could be dramatically different from our own, but it would be a mistake to believe that in the string theory multiverse "anything goes". While the very laws of physics may be different between the individual universes, they must be consistent with the mathematical structure of the Mother Theory, namely string theory. In other words, string theory sets the stage, and within that stage allows for great variations of its parameters, including the value of the vacuum energy density, which we now turn to.
The landscape explains the cosmological constant problem in a simple but tricky way. The argument is that the actual value of the cosmological constant is not directly predicted by the theory but depends on which universe we happen to live in. The multiverse then allows for basically any value of the cosmological constant compatible with the fundamental laws of string theory. Why do we live in a universe that has such an unnatural value for the vacuum energy? Because this value lies in the very narrow range that allows complex life to evolve and thrive. This is the basic philosophical consideration at the heart of the anthropic principle, which in its basic form is just the observations that the universe must be compatible with the conscious and sapient life that observes it. Now you can probably see why this is so controversial. Many physicists think this is an outrageous unscientific idea, but I don't agree. The anthropic principle is often made out to be more esoteric than it actually is, yet its basic tenet is nothing else than a selection effect: we observe unnatural values for the fundamental constants of nature because we live in an unnatural region of the multiverse/landscape, and we live in an unnatural region simply because the rest of it is uninhabitable.
If you think this is appalling, let me ask you this question. Can you tell me why the distance between the Earth and the Sun is exactly 149.6 million km? Natural philosophers of the past gave a lot of significance to this number and they tried to derive it from a fundamental theory. They were convinced that an all-encompassing theory of Nature should be able to explain the miraculous value of the Earth-Sun distance. God for sure intended for his favorite planet to be habitable and wrote the laws of nature accordingly. As you can already guess, it turned out that the particular distance between our planet and our sun had no special significance and could not be derived from any basic theory or principle. Instead, it was an environmental (as opposed to fundamental) constant. The universe is huge, it contains a very large number of solar systems, with countless stars and planets orbiting around them at various distances. Where do we live? In an ordinary planet at exactly the right distance from our ordinary star to be in the habitable zone, the range of orbits around a star within which a planetary surface can support liquid water and ultimately complex life.
The reasoning behind the anthropic explanation of the cosmological constant problem is exactly the same, with one (albeit non-trivial) difference. The landscape of possibilities in the case of solar systems is accessible to us, we just need to point our telescopes to the night sky. On the other hand, the laws of physics strongly prohibit an observer to escape his own universe and freely wander in the multiverse to observe other worlds, and perform experiments on them. We are stuck on our little ball in this case. The multiverse is in principle inexplorable, no matter how clever we are, or how advanced our technology is. This impossibility has very concrete effects, like the cosmic variance, the statistical uncertainty inherent in observations of the universe at extreme distances. This is due to the fact that we can only observe one realisation of all the possible observable universes, therefore we cannot make reliable statistical conclusions on the universe as a whole.
Another clue in favour of the landscape is that the particle content of our universe is not particularly elegant or beautiful. The standard model of particle physics, for example, contains three sets of matter particles. Each set is called a generation, or family. The existence of the other two families besides the one making up all visible matter is an unsolved mystery. We don't know why the universe may need these extra families, nor why there are exactly three in total. Many particles seem redundant, without no clear purpose. This is naturally explained in a multiverse scenario, in which the high energy theory is symmetrical and aesthetically appealing, but each of the low energy solutions (vacua) breaks the symmetry and appears largely arbitrary and "ugly". Incidentally, this is also what we observe in the animal kingdom, where natural selection acts on random genetic mutations, sometimes producing inefficient and seemingly unintelligent designs. In this analogy, the random mutations in the DNA corresponds to jumping between different low energy vacua of string theory, i.e. different universes in the multiverse.
The landscape idea of string theory is still very controversial, but it is slowly converting even the skeptics. The reason is simple: there is no viable alternative to the hard problems plaguing theoretical physics. If fully accepted, this may be the most monumental paradigm shift in the history of physics since Galileo and Newton. If 10 dimensional super string theory is indeed the whole story, and we do live in an a priori unknowable multiverse, the end of fundamental physics as we know it may be close. Apart from a few "real" problems that are still left to solve, like the black hole information paradox or the quantum origin of the universe, most of the unsolved problems in physics as listed on Wikipedia are naturalness problems, readily solved in a multiverse scenario. Is there a fundamental theory that can explain the values of all dimensionless physical constants? No, they are environmental parameters, they change from universe to universe. Why did the universe have such low entropy in the past, allowing time to have a definite direction? Again, one could argue that time asymmetry is needed for the emergence of life, so our universe just happened to have low enough entropy in the beginning to be able to remain far outside of thermal equilibrium for a long time. Why are there only 3 dimensions of space? Anthropic arguments again suggest that all but (3+1)-dimensional spacetimes might correspond to dead worlds, devoid of observers, as explained in this paper by Tegmark.
Whatever you may think about the multiverse hypothesis, it is undeniable that this would be the end of physics as we know it. With most of the problems solved by anthropic considerations, there wouldn't be much left to explain for fundamental physics, and the dreams of a final theory showing how our particular universe came into existence might be misguided. It would also be impossible to explore other kinds of physics outside of computer simulations: no matter how much our technology develops, we will never be able to escape our own observable universe and take a peek at the landscape. Without anything else to discover, physics will then ironically regress to a stamp collecting phase, merely cataloging what happens in our own neighborhood, oblivious to the wonders of the multiverse. Maybe we have reached our human limit in understanding physics, and the future belongs to other less developed sciences like biology, where much more is left to discover. The way the brain works is completely mysterious to us, and consciousness is pretty much uncharted territory. These are also problems that, if solved, will have a definite and lasting impact on society, especially when combined with the rise of artificial intelligence.
Or maybe we just need to face the reality that we are biological systems ourselves and not some kind of demigods who fell from the heavens. Our capacity to understand the world could be limited and there may be problems that will forever lie outside our abilities. It's not at all clear that we will ever be able to understand the origin of the universe, or consciousness for that matter. But many people before me had declared physics dead, and they were all proven wrong in spectacular fashion. I hope I'll join them soon.