Can there be a theory of everything?
Dear John
5th December 2000
In your recent book What Remains To Be Discovered you expressed some pessimism about the progress being made at the frontiers of science. I want to explain, as a scientist at one frontier, why I am so optimistic about our progress-especially on the fundamental issue of the unification of quantum theory (understanding the ultimate nature of atoms) with general relativity and cosmology (understanding the nature of space and time) to make a combined quantum theory of gravity.
The problem of quantum gravity is how to complete the revolution initiated by Max Planck's overthrow of Newtonian physics exactly 100 years ago. Quantum theory, general relativity, modern cosmology and elementary particle physics are all steps in the construction of the new theory. The objective is to explain how the rules (or laws) that regulate the smallest entities in the world, particles of matter and radiation, also account for the large-scale behaviour of the universe and the strange phenomena within it, black holes for one thing. To put it another way, the issue is how to make the mathematics that describes things that come in bits ("quanta") compatible with general relativity's treatment of space-time as a continuum.
Some people call this the problem of achieving a "theory of everything." I prefer to call it, a bit more modestly, a theory of all physical phenomena we currently know about. Why does it matter? Because it addresses our basic idea of what the universe is, and what space and time are. A revolution in our understanding of what the universe is must have profound consequences for our understanding of who we are and what meaning we can give to our existence.
Moreover, there are clear connections between how we understand the organisation of the universe and how we understand the organisation of our societies. People who lived in strict hierarchical societies imagined a universe organised on strict hierarchical lines, as in Ptolemy and Aristotle. The classic liberal conception of society is not unlike the notion of a world of atoms with properties defined by their relation to an absolute background of space and time. There is also some connection between the struggle of modern legal theorists to define what law is in the absence of an absolute notion of rights and scientists' attempts to understand how the universe could exist without an eternal and absolute space and time.
So how do we solve the quantum gravity problem, the merger of quantum theory with general relativity? The problem is hard, but not insurmountable. One aspect of science that is generally not stressed in popularisations is strategy. Science works because scientists-like artists, politicians and mountain climbers-have learned how to choose strategies to attack seemingly impossible problems. More than that, science works because it is a democratic community in which each participant can choose their own strategy. This makes it likely that if there is a strategy that works someone will find it. What keeps us going is that no amount of prior success guarantees that the strategy that worked on the last problem will work on the next.
This being said, here is one possible strategy for quantum gravity. First, study the problem in different experimental domains. Use what you find to invent partial theories that each answer a set of questions appropriate to one domain. Then, when progress has been made in several domains, put all the partial theories on the table and see if you can invent a story that weaves them together. This was the way the Copernican revolution proceeded, with Galileo and Kepler studying different domains of what became Newtonian physics, and it is also the way that quantum theory itself was invented.
In the case of quantum gravity, recent work has led to progress in four domains. Each is defined by the answer to a cluster of questions.
1) What are space and space-time made out of? What are the smallest units of space and time? What is the simplest thing that can happen?
2) What kinds of disturbances can travel through quantum space-time? To put it another way, what happens when you "jiggle" it?
3) Does combining the quantum with space and time lead to the prediction of any new and surprising phenomena?
4) Does quantum theory make sense applied to the whole universe?
Each question has been taken up by a community of people who have developed a theory that answers it. Most importantly, each theory has led to predictions which can be used to check whether the theory is right. In the case of the first question, some experiments that test for the presence of the smallest possible units of space and time have been done.
I will say more about these theories and the experiments that test them later. For now I will give only their names. In the order of the questions above, the theories are: loop quantum gravity, string theory, black hole thermodynamics and relational quantum theory. They are each the product of a lively group of people who have surprised themselves with how far they have got, even on an apparently impossible problem.
Lee
Dear Lee
7th December 2000
Of course, there is more to science than the nature of space and time and the still-absorbing question of the origin of the universe. Indeed, there are some who say that these are arcane pursuits, and place the discovery of the helical structure of DNA higher up the list of 20th-century achievements than quantum theory and Einstein's general theory of relativity. But I agree with you that the latter are not trivial pursuits: understanding the world we live in is surely the purpose of science.
My pessimism (as you call it) about the outlook in this field is not irrevocable. When the difficulties have been resolved, I shall celebrate too. But I don't think the uncertainties of the past 20 years will be easily dealt with.
Five years ago (when I began my book), my purpose was to draw attention to what seemed to be an important contradiction in theoretical science. Einstein's theory represents gravitational forces throughout the universe as a gravitational field, an all-pervasive entity recognisable primarily by the gravitational pull on material objects everywhere. For many years, people have tried and failed to bring this theory within the framework of quantum mechanics-to make space-time come in "bits." In 1933, L? Rosenfeld made a valiant first attempt, but there was no hope of success until the 1950s, by which time three people (the late Richard Feynman among them) had independently found ways of making electro-magnetic radiation (radio, light and X-rays) compatible with quantum principles. Why not then do the same for Einstein's gravitational field?
People did the maths for the gravitational field and ended up with nonsense answers. To be fair, much the same happens with the radiation field: if you calculate something like the mass of the electron, the result proves to be infinitely large. But the practitioners have found ways of subtracting infinity from the calculated result that gives something according with reality. When they work their mumbo-jumbo, they have predictions that agree with reality to eight places of decimals. The trouble with the project to make gravitation compatible with quantum theory is that nobody has found a similar procedure for making the gravitational infinities go away.
Reflective scientists in the field say one should not worry: "the infinities will disappear when we have a proper theory." But that capacity to live with such loose ends is symptomatic of fundamental physics in the past few decades. If the universe began with a Big Bang, what was the cause? Uniquely for physical phenomena, there is no persuasive answer (the creationists say the answer is obvious). Why is the density of the universe nearly, but not quite, enough to prevent it contracting again? How does it come about that the particles of matter have just the properties they seem to have?
Most scientists in any field are not reflective. People hang on to the conceptual baggage with which they are familiar. Why worry about gaps in basic understanding when your own problem, say that of designing a better computer chip, cannot be affected by a fundamental recasting of the conceptual framework that determines how the universe behaves?
You say that everything has now changed and that there is a comprehensive framework of understanding. I shall believe it when I know more. Your own background is in what is called string theory, which takes the view that the irreducible particles of matter are not point-like bits (as we are taught at school), but have some kind of structure that occupies no perceptible space or time. Although there are several different kinds of particles (electrons, protons and so on), there is just one kind of object called a "string." The diversity of the real world arises because the strings vibrate with defined frequencies, and because these frequencies correspond to the energy, or the mass, of the various particles of the real world.
String theory is central to your account of the new theoretical framework. But as an observer of the scene for many years, I have noted the repeated waxing and waning of enthusiasm for strings. Over the past 25 years, there have been several big conferences to announce the imminence of a breakthrough. Then disappointment sets in, the graduate students get depressed-and we all wait for the next conference. None the less, your four questions offer a way forward. I believe the most important is the third. Are there phenomena unique to the new framework that can be observed, so that the theory can be tested? If so, what are they?
A final point about your sociology of discovery: I do not believe that Ptolemy built an unrealistic cosmology because Alexandrian society was organised hierarchically. The ancients were usually wrong not because they were stupid or socially constrained, but because they had poor data to work with and lacked the benefit of the ideas left behind by earlier generations. Science is a factual business; experiment is a tyrannical king. I hope that you and your colleagues pass its test.
John
Dear John
8th December 2000
I agree with much of what you say. Many people who claim to be working on fundamental problems in physics are neither reflective nor terribly interested in conceptual problems, such as the nature of space or time or causality. This is the main reason why progress has been so slow. To make real progress requires a combination of reflection and intelligence and, as Paul Feyerabend told us, science thrives when many different views are cultivated. One thing I will miss about working in England, after my two years here, is that there is more interest in reflective thought and a bit less in academic politics, in which people attempt to resolve scientific questions by denying positions to those who pursue approaches different from their own.
It is odd that you call me a string theorist, as many string theorists see me as a leader of the opposing school, and indeed most of my work in the last 15 years has been in loop quantum gravity, which is seen as an opposing proposal to string theory. My own view, which underlies my optimism, is that string theory and loop quantum gravity are partial theories, which each solve part of the problem. But many string theorists and some people working on loop quantum gravity disagree with me.
But science is about nature, not sociology, personality or politics. So a few years ago, I decided to try to take an unbiased look at what had actually been shown by each of the approaches to quantum gravity, in order to see if the results might fit together to give us the framework for one theory. I came to the conclusion that useful and potentially true results had been arrived at from several directions. These included not only string theory and loop quantum gravity, but twistor theory, noncommutative geometry, and contextual or topos approaches to quantum cosmology. This is what led me to the view I presented in my first letter.
Beyond this, there is a reason why string theory has failed to deliver on its promises. Several of those promises were based on expectations about nature that turned out to be wrong. One wrong idea was that space-time could be assumed to have a fixed background. In fact, string theory must accommodate the main lesson of general relativity, which is that there is no fixed background because space and time describe a network of relationships. This integration was accomplished in loop quantum gravity. Just in the last year we have seen how the result of that insight can be fitted together with string theory to make a theory that works.
The second wrong idea about string theory was that the theory that unified the description of all the interactions would be unique, in the sense that there could only be one mathematically consistent unified theory. This idea allowed people to be complacent about the fact that there is no experimental support for string theory. If unification implies uniqueness, they reasoned, it was more important to get the structure of the unified theory right, because then experiment would have to agree with the only true theory. But the evidence has gone in the other direction: the more particles and forces that are unified in a theory the less unique it appears to be. Grand unified theories have more free, adjustable parameters than the theories of the forces they unify, and any realistic supersymmetric theory has still more parameters. String theory continues this trend, as there appear to be an infinite number of equally consistent string theories, and they exist in all dimensions from zero up to ten.
It has been conjectured that all the string theories are approximate descriptions of different physical phases of some basic "stuff." This stuff would be described by another theory, which physicists have called M theory-an appropriate name for a theory that has not been invented. The phases would be like the different phases of water, gas, liquid and ice, but there would have to be an infinite number of them. Moreover, all of them can be realised in nature, so that the question of why we live in a three-dimensional world, with a particular set of elementary particles and forces, would become an historical question. It would be settled not by principles, but by a better understanding of the history of our particular universe. This would have to include an understanding of what happened before the Big Bang to bring our universe, with its observed features, into existence. I might add that my own cosmological natural selection idea-the claim that universes have "offspring" created in black holes-was invented to address this question. Whether that theory turns out to be true, or is disproved by observations, I believe the general lesson will remain: whether there exists a final, unified theory or not, the properties of the elementary particles we observe are in part determined by events in the history of our universe, and are not just the consequences of general principles.
As you say, in the end experiment will decide. Let me say a bit about the experiments that I believe will test theories of quantum gravity. Viewed on a certain very small scale, space does indeed appear to be made of discrete bits, just as matter is made of atoms. For example, loop quantum gravity predicts that there is a smallest possible volume to a region of space, and all the approaches predict that there is a maximum amount of information that can pass through a surface of a fixed area. The fundamental units are very tiny-the unit of length is about 20 orders of magnitude smaller than an atomic nucleus. The problem is then to do an experiment which can detect the predicted atomic structure of space and time.
An analogy may help here. Long before atoms had been detected directly, Einstein showed that their existence was necessary to explain the behaviour of ordinary matter. By measuring the surface tension of a liquid, or by watching the dancing of a grain of pollen in a liquid, he could determine the size of an atom. The question is now whether we can do something similar to detect the atomic structure of space. And the answer seems to be that we can. A young Italian physicist named Giovanni Amelino-Camelia and his friends have invented several experiments which are already probing the structure of space at the quantum gravity scale. The point is that the predicted quantum structure of space has a small effect on the propagation of very energetic particles. This effect is compounded when particles travel the vast distances from their sources to us. In observing very energetic cosmic rays or the spectrum of light that reaches us from very distant gamma ray bursts, you are probing the structure of space on extremely small scales. As these experiments are refined, it is possible that we will be able to rule out or confirm some predictions of the different quantum theories of gravity.
I agree with you that string theory and other approaches were oversold, and the result was, as you say, alternating waves of euphoria and cynicism. I suspect it is the same in other sciences. And part of the problem is the organisation of science itself. Technically proficient specialists without ideas of their own have easy careers so long as they work within the dominant approach, while young scientists of great promise who think for themselves have trouble finding positions. I sometimes doubt that the great individuals who have pursued their own ambitious approaches to quantum gravity, such as Roger Penrose, Bryce DeWitt and David Finkelstein, would find it easy to find academic posts in the present climate. I am not sure what to do about this except to work to free us from the rigidly hierarchical and bureaucratic practices that persist in our universities and funding agencies.
Lee
Dear Lee
9th December 2000
I apologise for mistaking you for a string theorist. I can see that it's a little like mistaking a psychiatrist for a psychologist (or vice versa). But your mood has changed between the first and second letters. In the first, you stressed the successes of quantum gravity; in the second you emphasise the difficulties. So I want to return to what you called my pessimism about fundamental physics.
We overlook the huge investment of intellectual capital that goes into the development of novel theories in fundamental physics, and how long they take. It took 25 years from Planck's tentative proposal that heat and light radiation can be transferred only in finite amounts ("quanta") to the point at which there was a usable theory for the properties of ordinary materials in the real world-the differences between metals, semi-conductors and insulators, for example. That 25 years engaged some of the world's best scientists, from Albert Einstein to Linus Pauling.
The result, in the 1930s, was a great ferment of excitement, comparable with that sweeping through the biological sciences now that the structure of DNA is understood. People busied themselves applying the new theory to all kinds of problems in the real world: What keeps the sun shining? How does a photographer's light meter function? The heroes of that time collected at least a score of Nobel prizes. Yet the revolution was not complete and, even now, is still unfinished. That is why you are occupied with quantum gravity.
My hunch is that reaching the goal of quantum gravity will be more arduous than any previous prize in the century-long adventure of quantum theory. One almost trivial difficulty is that, for the first quarter of a century, people were able to use existing mathematics; now they have to make it up as they go along. More serious is the difficulty of conceptualising what each novel element of a developing theoretical enterprise really signifies; years can pass before people hear themselves saying, "So that's what it means!" And then there's the problem of experimental tests; it will be luck indeed if natural phenomena-you mention gamma-ray bursts-provide telling proof one way or the other. Half a century from now, it may seem much easier. Only the long view makes sense here.
My own view is that the search for a unified theory will never end, people will always be seeking a more detailed and exacting explanation of what makes our universe (and all the other universes) tick. The intervals between these episodes will not shorten as the centuries pass; the timescale is determined not only by the increasing technical difficulties but by the time required for conceptualisation. I do not believe that there will ever be a theory of everything.
You are right to emphasise that science must be properly organised to accommodate these ambitious undertakings. We must find ways of encouraging diversity. I suppose that most agencies which make research grants would say that is what they do now. But the same people sit in judgement on all grant applications, so sameness is hard to avoid.
I have a proposal. Return responsibility for spending research funds, mainly taxpayers' money, to the institutions that do the work. Univer-sities and institutes are best placed to tell who is good and productive. The obvious complaint that the research enterprise would then be unaccountable could be countered by periodic assessments to tell what use had been made of past subventions. Let us hope that some government tries it sometime. But the timescale for realising this ambition is probably similar to that required to solve the riddle of quantum gravity.
Yours
John
Dear John
10th December 2000
Yes, I agree that quantum gravity will be based on new mathematics and new concepts. I am optimistic precisely because a new mathematics has been invented in the last decades that seems well suited for describing quantum space-times. This new mathematics tells us that the world can be described as a system of relationships which evolves in time. In such a world, there are no "bits" or "things" with absolute properties; instead all properties stem from relationships between things. There is also a new principle, called the holographic principle, which states that physics is not about some absolute description of the world, "as it really is," rather it is about the information that flows from one part of the universe to another. There is much still to do and, as you say, experiment is paramount. But a new generation of theorists have found ways to test theories of quantum gravity, something we thought was impossible.
On the organisational issues, I would suggest, first, giving control of the funds directly to the people who actually do the work, regardless of age or status, and second, investing in ambition and vision over "more of the same" and accept the risk that comes with that. If we do this, the opening decades of this century will see great progress, not only on quantum gravity, but on the other hard problems such as cancer, the brain, the origin of life and the origin of the universe.
Lee
Dear Lee
11th December 2000
Where do we stand? I agree with you that there will indeed be a theory of quantum gravity sometime. I believe that will indeed make it possible (with extra work) to understand where the universe came from and why the particles of matter are what they are. The same development will tell us about black holes but, more importantly, will also open people's eyes to a whole slew of problems we never knew existed. It will be a momentous development.
Where we differ is about timing. I think it will be a long job, you are more optimistic-but each of us, I notice, has been pretty reticent about the date of the year in which we will open the champagne. Not that the date matters to science at large.
John
