{"id":750,"date":"2026-05-24T11:41:21","date_gmt":"2026-05-24T11:41:21","guid":{"rendered":"https:\/\/stevewatson.info\/blog\/?p=750"},"modified":"2026-05-24T11:59:11","modified_gmt":"2026-05-24T11:59:11","slug":"wrong-directions-for-agi","status":"publish","type":"post","link":"https:\/\/stevewatson.info\/blog\/2026\/05\/24\/wrong-directions-for-agi\/","title":{"rendered":"Wrong Directions for AGI"},"content":{"rendered":"<p><span style=\"color: #000000;\">One of my criteria for attributing understanding to a system is that the internal states of the system must be capable of deriving or corresponding to a partial model of that which they understand (to the degree that it is understood anyway.) LLMs are simply incapable of passing such a test since they don\u2019t contain a model of the world at all, but only a model of the language corpus on which they are trained.<\/span><\/p>\n<p><span style=\"color: #000000;\">Against this it might be said that, in a certain sense, you <em>could<\/em> derive a model of the world from an LLM simply by asking it to describe the relevant world. Since the language produced is in some correspondence with the world \u2013 since someone receiving that description could use it to construct such a model \u2013 it follows that the internal states of the LLM <em>can<\/em> derive a correspondence with the world and my criterion of understanding would be satisfied. <\/span><\/p>\n<p><span style=\"color: #000000;\">In such a case, however, the LLM product has a correspondence <em>only in so far as it is mediated by a natural language speaker<\/em> (of the LLM\u2019s base language) and therefore its appearance of \u2018understanding\u2019 can only be a projection\/reflection of the understanding of that speaker. The point can be made clear to our intuitions using two examples that I\u2019ve used in the same context previously (in conversation anyway.)<\/span><\/p>\n<ul>\n<li><span style=\"color: #000000;\"><span style=\"color: #000000;\">In the first place, consider a LLM created by exposure to the entire corpus of Chinese language text, but managed by only non-Chinese speakers. It would be impossible under any circumstances to derive a model of the world from the productions of such a LLM. This indicates that any supposedly derivable model is dependent upon the semantic capacities of the natural language producers and not of the LLM. or of the consumers.\n<p><\/span><\/span><\/li>\n<li><span style=\"color: #000000;\">In the second place, if there is lingering doubt about this \u2013 or if it is thought that the consumers of the LLM products could learn the language over time, given enough exposure to the LLM products (as people sometimes erroneously argue concerning the output of Searle\u2019s Chinese Room, by entirely misunderstanding the claim of the CR experiment) \u2013 consider the case of a LLM created by exposure to the corpus of an unknown language such as Linear A. In such a case we know that no matter what the output of the LLM, there is no-one anywhere who could construct a world model from it. Moreover, there is no possibility that the LLM could be used to translate from the Linear A language to, say, English, from which a model could be derived.<\/span><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>One of my criteria for attributing understanding to a system is that the internal states of the system must be capable of deriving or corresponding to a partial model of that which they understand (to the degree that it is understood anyway.) LLMs are simply incapable of passing such a test since they don\u2019t contain [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-750","post","type-post","status-publish","format-standard","hentry","category-cogsci"],"_links":{"self":[{"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/posts\/750","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/comments?post=750"}],"version-history":[{"count":5,"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/posts\/750\/revisions"}],"predecessor-version":[{"id":756,"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/posts\/750\/revisions\/756"}],"wp:attachment":[{"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/media?parent=750"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/categories?post=750"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stevewatson.info\/blog\/wp-json\/wp\/v2\/tags?post=750"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}