When Machine Truth Is Unverifiable: Epistemic Responsibility in the Age of Opaque Models

It is now possible to consult, in the ordinary course of a working day, a software system that will produce in seconds a draft of any kind of expert advice: a legal memorandum, a medical differential, a policy brief, a translation, a summary of a hundred-page contract. The advice will be presented in confident prose, in the appropriate register, by a system that has read a quantity of text no human will ever match. It will sometimes be wrong. Most importantly — and most strangely — when it is wrong it will be wrong in ways that the system itself cannot detect, and that no other system can readily verify against. The system has no internal record of its sources; it has no way to mark which of its claims it is confident in; it does not know what it does not know, in the modest, useful sense in which an experienced doctor or lawyer often does.

This is a new epistemic situation, but it is also an old one. We have always taken the word of experts whose internals we could not inspect. The doctor's diagnosis, the lawyer's reading of the statute, the historian's interpretation of an archive — none of these comes with the chain of inference exposed for our examination. What they come with is something subtler: an institutional context in which, at some point in the chain, a person can be asked to explain. The medical journal, the bar examination, the peer-reviewed monograph, the cited bibliography — all of these are devices that hold the expert's testimony accountable to a community of other experts, and that community, in turn, is accountable to the rest of us by means of credentials, reputations, and the slow accretion of trust.

What is novel about large opaque models is not that we cannot inspect their reasoning — we have never been able to inspect a doctor's reasoning either — but that the chain of institutional accountability breaks somewhere in the middle. There is no one to ask. The model is operated by a company whose internals are proprietary, trained on data the company did not produce, on procedures only partially documented in public papers, and deployed through interfaces whose terms of service explicitly disclaim the kind of liability that bar associations and medical boards exist to enforce. The opacity is not at the level of the model; it is at the level of the social arrangement around the model.

The Older Problem of Testimony

The philosophical literature on testimony, which one might expect to be a niche concern, turns out to be exactly the resource we need. C. A. J. Coady's book Testimony (1992) and John Hardwig's earlier paper "Epistemic Dependence" (1985) both argue that an enormous fraction of what we know — arguably most of it — we know not because we have verified it ourselves but because we have been told it by someone we have reason to trust.[^1] We know that the Earth orbits the Sun, that aspirin reduces inflammation, that the Treaty of Westphalia was signed in 1648, primarily because we read it somewhere and the source seemed credible. Knowledge, on this view, is a fundamentally social achievement; it is something a community possesses and that individuals access by being members in good standing of that community.

The condition Coady identifies as essential to the working of testimony is that the testifier be the kind of agent for whom the question "How do you know?" has a coherent answer. The doctor knows because she trained for ten years, examined the patient, took a history, ordered tests, and consulted the literature. The historian knows because she sat in the archive and read the letters. The chain of how does not need to be fully laid out in front of us, but it must, in principle, exist. The trust is in the chain, not in the person.

When a model speaks, we have no chain to trust. We have a confident statement, presented in the cadence of expertise, from an apparatus whose internal grounds for the statement are not available even to its designers.

This is where the trouble lies. Hardwig points out that even within science, the individual investigator depends, for what she knows, on testimony from other investigators — the chemist takes the physicist's word for the spectrum; the medical researcher takes the chemist's word for the assay. The dependence is not a defect; it is the form of the knowledge. What sustains the dependence is that, at every link in the chain, the testifying party can be asked to explain, and the explanation can in principle be checked by a third party who has the relevant training. The chain has, at every link, a person.

When a large model produces a confident answer to a question, there is no such person anywhere in the chain.

Three Inadequate Responses

It is tempting to respond to this situation in one of three ways, each of which I think is mistaken.

The first response is prohibition: large models should not be used for consequential decisions, on the ground that their testimony is illegitimate. This is consistent but, I think, untenable. The systems are useful enough that they will be used, regardless of philosophical objection, and a posture of refusal cedes the institutional design problem to the only people willing to take it up — namely, the companies whose interest in the design is not the public's.

The second response is credentialing the model: we should test models on standardised problems, certify them above some threshold, and then permit them to operate within their certified domain. This is the response that the major frontier-model laboratories appear to favour, and it is being implemented in several jurisdictions. The trouble is that the standardised tests are themselves opaque (the test items are usually hidden to prevent training contamination, which means that the certification cannot be independently verified) and that performance on a benchmark is, as a great deal of recent work has shown, an unreliable predictor of performance in the real situations the benchmark was meant to approximate.

The third response is interpretability: we should make the models' internal reasoning available for inspection, so that the chain of testimony can be restored. This is the response one hears most often from inside the technical community. The work is interesting and worth doing, but it is, in my view, oversold. The current state of interpretability research can identify some of the features that some kinds of models attend to in some kinds of decisions. It cannot, except in toy cases, produce an audit trail of the kind that the expert testimony tradition requires. We are not close to being able to produce one, and the obstacles are not merely technical; some of them may be in principle.[^2]

A Fourth Response: Reconstituting the Institutional Chain

The response I want to argue for is institutional. If the chain of accountability is broken inside the model, it must be reconstituted around the model.

What would that look like? Several things, none of them futuristic. First, the deployment of model-mediated advice in any consequential domain should be conditioned on the existence of a human professional who is responsible for the advice as given, in the same sense that a doctor is responsible for the prescription she signs. This is, in some jurisdictions, already the legal position; in many it is not, and the disclaimers on the user agreements try to push it elsewhere.

Second, the model's training data, training procedure, and deployment protocols should be auditable by a body other than the company that owns the model, in the way that pharmaceutical trials are auditable by regulators other than the manufacturer. Frank Pasquale's Black Box Society argued this case nearly a decade ago for scoring systems generally; the argument applies a fortiori to large generative models, whose outputs are richer and consequential surface area larger.[^3]

Third — and this is the one I find both most important and most often overlooked — the public should be told, plainly and at the point of use, when the advice they are receiving is model-generated, and they should be told what kind of mistakes that advice is prone to. The current convention, in which model outputs are presented in the same register and visual treatment as a human professional's notes, is an erosion of informed consent. It is also, I think, an erosion of professional dignity, since it allows real expertise to be confused with confident pattern-matching.

The Reader's Position

For the rest of us, who are not in a position to design institutions for the certification of frontier models, I think the appropriate disposition is one of disciplined skepticism. Model-generated text should be read with the same kind of critical attention one would give to an unsigned editorial in an unfamiliar publication: useful, possibly correct, possibly informative, but not yet entitled to one's belief. The temptation, encouraged by the smoothness of the prose, is to extend the trust we have built up for human writers to the new register. We should not extend that trust, because the conditions under which we developed the trust — the existence somewhere in the chain of a person to ask — do not obtain.

This is not a counsel of paranoia. It is the older counsel of literacy. To read carefully is to keep some distance from the text, to ask after its sources, and to be willing to put it down when the sources do not bear examination. We have always known how to do this. We have, for some decades, been encouraged not to bother, on the ground that the algorithms could be trusted to do the work of skepticism for us. They cannot. The skepticism is ours to keep.

[^1]: Coady (1992), esp. ch. 4; Hardwig (1985). [^2]: Burrell (2016) distinguishes three sources of opacity: intentional corporate secrecy, technical illiteracy, and a third, harder kind that arises from the mismatch between high-dimensional optimisation and human reasoning. [^3]: Pasquale (2015), esp. ch. 6.