Steve Watson

Wrong Directions for AGI

One of my criteria for attributing understanding to a system is that the internal states of the system must be capable of deriving or corresponding to a partial model of that which they understand (to the degree that it is understood anyway.) LLMs are simply incapable of passing such a test since they don’t contain a model of the world at all, but only a model of the language corpus on which they are trained.

Against this it might be said that, in a certain sense, you could derive a model of the world from an LLM simply by asking it to describe the relevant world. Since the language produced is in some correspondence with the world – since someone receiving that description could use it to construct such a model – it follows that the internal states of the LLM can derive a correspondence with the world and my criterion of understanding would be satisfied.

In such a case, however, the LLM product has a correspondence only in so far as it is mediated by a natural language speaker (of the LLM’s base language) and therefore its appearance of ‘understanding’ can only be a projection/reflection of the understanding of that speaker. The point can be made clear to our intuitions using two examples that I’ve used in the same context previously (in conversation anyway.)

• In the first place, consider a LLM created by exposure to the entire corpus of Chinese language text, but managed by only non-Chinese speakers. It would be impossible under any circumstances to derive a model of the world from the productions of such a LLM. This indicates that any supposedly derivable model is dependent upon the semantic capacities of the natural language producers and not of the LLM. or of the consumers.

• In the second place, if there is lingering doubt about this – or if it is thought that the consumers of the LLM products could learn the language over time, given enough exposure to the LLM products (as people sometimes erroneously argue concerning the output of Searle’s Chinese Room, by entirely misunderstanding the claim of the CR experiment) – consider the case of a LLM created by exposure to the corpus of an unknown language such as Linear A. In such a case we know that no matter what the output of the LLM, there is no-one anywhere who could construct a world model from it. Moreover, there is no possibility that the LLM could be used to translate from the Linear A language to, say, English, from which a model could be derived.