Quantum Supremacy and the Ethics of Irreproducibility

When Google's Sycamore processor reportedly solved, in 200 seconds, a problem that the team estimated would take a classical supercomputer ten thousand years, the headlines wrote themselves. The technical paper, published in Nature in October 2019, was more measured.[^1] The estimate of classical runtime was contested almost immediately — IBM, who had access to the supercomputer in question, argued the same problem could be solved in two and a half days with a smarter algorithm, and produced a careful note arguing precisely that.[^2] The supremacy result was, briefly, news, then a controversy, then a slow return to a different, harder kind of news: a result that no other laboratory could independently verify, because no other laboratory had a 53-qubit superconducting processor calibrated in the same way.

A year later, in December 2020, a Chinese team led by Pan Jian-Wei announced a related claim using photons rather than superconducting qubits, with a problem and apparatus engineered to be even less classically tractable.[^3] The same epistemic pattern obtained. The result was verifiable only by other photonic teams, of which there were essentially none with the same hardware, and the runtime advantage rested on simulations of what classical computers could not in principle do. Whatever one thinks about supremacy as a concept, an honest reader of the literature must acknowledge that we have entered a regime in which the central empirical claim of a research programme cannot be checked by anyone other than the people who made the claim.

I want to set aside, for this essay, the technical question of whether the Sycamore and Pan claims were correct in their own terms. I am interested in something prior: in what kind of knowledge a result of this form can be, and in what we owe one another as a community of inquiry when verification by other hands has, for the first time in modern science, become impossible.

Reproducibility as a Constitutional Norm

That science is a fundamentally communal activity, and that the communal character of its knowledge is what distinguishes it from private revelation, is older than the modern scientific institutions in which we now operate. Karl Popper's classic statement of the demarcation problem — that a scientific claim is one that admits, in principle, of refutation — presupposes that refutation could be performed by someone other than the original claimant.[^4] Without that presupposition the demarcation collapses; a claim that only one party could ever falsify is functionally equivalent to one that no party could ever falsify, and the line between science and authority dissolves.

Ian Hacking pushed the point further in Representing and Intervening: scientific knowledge is constituted not by the act of observation but by the act of intervention, by the repeated production of phenomena under controlled conditions.[^5] We trust the existence of electrons not because we have seen them but because we have made them do work. The trust is collective because the doing is collective: somewhere in the apparatus of a particle accelerator there is a person who has, by independent labour, gotten the thing to work in their own hands.

Quantum supremacy results, as currently produced, do not fit this picture. The Sycamore chip exists in exactly one place, calibrated in exactly one way, accessible to exactly one team. Its claim to have outperformed classical computers rests on a counterfactual — what the supercomputer would have done — that no one can directly check. The Chinese photonic results are in a similar position, with the additional complication that the apparatus is built around components whose performance characteristics are themselves only available from their manufacturers' specifications.

It would be too quick to say that this makes the results unscientific. Astronomy has long included claims about events — the death of particular stars, the trajectory of particular galaxies — that can never be independently verified, because they happened in places we cannot return to. Palaeontology rests on inferences from fossils that are, in many cases, unique specimens. We have made our peace with these epistemic situations by developing institutional procedures — peer review of methodology, public deposit of data, collegial trust among practitioners — that compensate for the absence of strict reproducibility.

The honest question about quantum supremacy is whether those institutional procedures are adequate to the load they are now being asked to bear.

What we are watching is not the failure of reproducibility but its slow institutional thinning — and the corresponding need for forms of public accountability we have not yet had to invent.

The Moral Costs of Asymmetric Verification

The trouble with claims that only their authors can verify is not merely epistemic but moral. Consider three positions that the rest of the research community might take with respect to a Sycamore-style result.

The first is trust: the result is accepted on the strength of the team's reputation and the integrity of the apparatus, with the understanding that further work will eventually permit verification. This is, in practice, what has happened, and it has not been unreasonable. The Google team made their data available; their methods were detailed; their conclusions were stated with appropriate caveats; and the subsequent classical-algorithm work that partially deflated their claim was conducted in public. Trust, in this case, has been a working solution.

The second position is skepticism: the result is held in suspension until independent verification becomes possible, on the ground that science without reproducibility is not science. This position has the virtue of consistency but the vice of immobility. If we declined to accept any quantum result that could not be replicated by a second laboratory, we would have very few quantum results, and a research field that could not produce knowledge would not be a research field for long.

The third position, which I find most interesting and least often articulated, is vigilance. On this view we accept the result provisionally, as one accepts the testimony of a careful witness in a court that has not yet ruled, while developing new institutional procedures to compensate for the unavailability of strict reproducibility. Vigilance would, among other things, require: public pre-registration of the experimental design; deposit of raw data in archives controlled by parties other than the experimenters; development of cross-verification protocols between non-identical apparatus (a result on superconducting qubits should be redone, at least in part, on photons, and conversely); and a willingness on the part of the field to develop and acknowledge a distinction between reproduced and coherent results — a coherent result being one that is consistent with the theory and with adjacent measurements, even though no direct replication is available.

None of these procedures exists in mature form. All of them are within reach.

Why It Matters Beyond the Laboratory

A reader might reasonably ask why this question concerns anyone outside the small community of practitioners. The answer is that the precedent we set in the supremacy era will be the precedent that governs claims of considerably greater consequence in the decade ahead.

Quantum computers, if and when they reach useful scale, will be deployed against problems whose solutions are highly consequential — the simulation of catalysts and proteins, the breaking of public-key cryptography, the optimisation of large logistic systems. Many of those solutions will, by the nature of the computation, be unverifiable by classical means. The community will have to develop a vocabulary for trusting answers that cannot be independently checked, and the moral seriousness of that vocabulary will determine whether the answers can be safely acted upon.

The same problem already exists, in less dramatic form, in machine learning. A large language model produces outputs that no individual reader can verify against the corpus on which the model was trained; the training procedure is too expensive to repeat, the corpus too vast to read, the model's behaviour too sensitive to randomness in the optimisation. When a clinical-decision-support system trained on such a model recommends a course of treatment, the recommendation is, in the epistemic sense, a Sycamore result writ small: a claim whose verification by independent hands is structurally impossible.

In medicine we have at least the safety net of clinical trials, in which a treatment recommended by any source whatever must be checked against patient outcomes before it is deployed. The corresponding institution does not yet exist for quantum computing or for large-model recommendations more generally. Building it is one of the moral projects of the next decade.

A Last Note on the Word Supremacy

The word itself is unfortunate. The original technical usage — quantum computational supremacy, coined by John Preskill in 2012 — was, I think, an attempt to be precise about a narrow technical milestone. Its political reception has not honoured that precision. What was meant as a benchmark has been turned into a banner, and the banner has done damage that careful reading of the original papers does not warrant.

Lorraine Daston and Peter Galison's history of scientific objectivity is instructive here: scientific epistemic virtues are made, not given, and the vocabulary we choose for new kinds of knowing shapes the moral seriousness of the knowing.[^6] The quantum-computing community would do its public reputation a service, and itself a deeper service, by retiring the word and choosing one that better describes what these experiments have actually done. Computational advantage is a serviceable substitute. First glimpse would be more honest. Either, I think, would help us think more clearly about what we now collectively know — and what we still owe one another to find out.

[^1]: Arute et al. (2019). [^2]: Pednault et al. (2019). [^3]: Pan et al. (2020). [^4]: Popper (1959), esp. ch. 4. [^5]: Hacking (1983), ch. 16. [^6]: Daston & Galison (2007), esp. the discussion of mechanical objectivity.