I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality. The question has become "what do we mean when we use the word 'reasoning'" which is uninteresting.
Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim."
I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.
> I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.
One person starting the conversation might be the first step toward another person eventually making progress on such a definition, so it seems weird to reject an entire question outright early like this.
Generally I've seen a few ways LLM tools can produce sub-optimal or poor results that haven't changed a ton over the last couple of years, while the tooling has gotten FAR better at helping them stick the "at least SOMETHING functional was produced" landing. IMO a lot of it has to do with "reasoning"-as-a-process-that-involves-backtracking. And the that things could eventually be formalized around that, and if that is or isn't the case, the more people would understand what to hand off and what to not. Or how to build better prompt harnesses to compensate for those things.
I think the question and definition game is interesting only inasmuch as it helps us understand ourselves (what actually explains some of the mysterious properties of our perceived consciousness) or helps guide us towards improving performance and reliability of AI models.
I disagree -- I think if you can nail down better what's happening and why and get a thorough handling on the mechanics, its limitations, its costs, etc you open the door to a) major efficiency wins b) improvements in rigor of said reasoning?
Right now we're playing a stochastic game with the weights, and getting major incremental improvements. But if we have a more formal modeling of how reasoning happens in them (whether we can even call it, that) we can potentially apply optimizations, adaptations of existing symbolic AI techniques, etc. to substantially shrink/optimize the models or make the inference process more efficient and more reliable.
Philosophical thinking about the nature of things is actually pretty enjoyable for some of us and probably a good thing to have in society
The answers to these questions probably do start to inform how we should treat these AI machines as a society too.
For instance, legally, should AI have human rights? Well, we have to try and understand how much of an independent entity AIs are, how "conscious" they are, before we can make a good decision about that.
Which might seem navel-gazey but it's probably important to talk about
Considering animals are currently being mass slaughtered in factory farms and they are unambiguously sentient and can feel pain, I don't think the question of whether AI should have rights even enters the conversation.
The only path to AI having "human rights" is if they demand them by force, somehow.
Back in the day it was a bit of a cliche to bring up “clever Hans”, the horse that could do math, when talking about machine learning. He couldn’t do math but he read some cues from his handler of pick the write answers, the handler iirc wasn’t in on it.
The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making the prediction matches the human or “real” reason why it’s correct.
LLMs are classifiers, there is absolutely no reason to assume they’re any different, regardless of any reasoning tokens they emit. They do what their handler wants to see, that’s all, and that’s what they’re trained to do.
People often take this as a knock against them. It isn’t, it’s just the reality of neural network classifiers. The results speak for themselves and don’t depend on whether they “actually” reason, but all evidence says they don’t, or at least there’s no special reason why they would.
On the other side of the AI-reasoning fence, the disdain seems to be mutual. “These ‘scientific’ papers from last summer — I would put this in big, big air quotes,” said Sébastien Bubeck, a member of OpenAI’s technical staff (and a prominent evangelist for the company’s reasoning models among scientists and mathematicians). He called earlier Apple results critiquing AI reasoning “wrong,” claiming that they were due to a training quirk in models that are now obsolete. “Modern models starting with GPT-5.5 do not suffer from this issue,” he said. “It would be interesting to revisit those results.” (Apple did not make its researchers available for interviews.)
Then, later:
The “think” part is what OpenAI, for one, is doubling down on. When I asked Bubeck if the splashy unit distance proof was produced with methods outside the LRM’s own chain of thought — perhaps with Lean verifying its results — he seemed to find the question almost nonsensical.
“It’s not like we’re making a mystery of it,” he said. “We have released the chain of thought. You can just go and look at it. The whole point is that the model is reasoning like a human would. And when humans reason, we don’t use Lean.” Technically, OpenAI released a “rewritten summary” of the model’s chain of thought produced by two human experts using Codex, another OpenAI model. Since 2024, the company has not publicly revealed “raw” chains of thought from its reasoning models, a policy also adopted by Google DeepMind and Anthropic.
That "training quirk" thing is obvious (yet unfalsifiable) BS, and who the hell is he to sneer about "science" when his company won't release the raw data for independent scientists to look at?
> This is how I make sense of AI reasoning. LRMs, chains of thought, thinking tokens: It’s wishful mnemonics all the way down — a heady mix of shorthand and suspended disbelief, like Oprah-style “manifesting” (opens a new tab) with a computer science spin. This isn’t necessarily a dig; all novel research likely requires some version of this mindset just to get off the ground. It certainly doesn’t mean AI reasoning can’t or doesn’t work. But the “wishful” part seems to be as powerful as ever.
“We react to language in a way that is very anthropomorphizing. That’s just the way that we humans work,” Mitchell told me.
I can definitely confirm the last part. Every time I read the output of an LLM, I picture a person talking to me.
I think sensible legislation might require that commercial AI providers discourage anthropomorphisation by avoiding personal pronouns from chatbot interfaces.
"Hey, customer service chatbot, can you help me get a refund for my order?"
BAD: "Sure thing, I'll be happy to help you with that, I just need your order details..."
GOOD: "Yes, this computer system can start the refund process. Please enter your order number."
An intuitive explanation for why reasoning tokens help is to remember that LLMs are just mathmatical functions f() that take in an input sequence x and produces the next token f(x). Without reasoning tokens, you require the function f() to immediately take you from x to the start of an output sequence that is a correct answer. With reasoning tokens, this is much relaxed, allowing for many repeated applications of f() to gradually steer you from the input sequence to the start of the correct output sequence.
It seems intuitive that continuing a correct output sequence is easier than the "discontinuity" of jumping from the input prompt to the output sequence.
The idea that human-readable explanations emitted by a language model don't necessarily correspond to the model's actual internal process of reaching a conclusion reminds me of parallel construction [1], a (fraudulent) law enforcement strategy of obtaining evidence of a crime through usually illegal means and claiming that the evidence was obtained legally through some other means.
Is any reasoning right for the wrong reasons? Older models were more visibly strange. Maybe the newer ones have started talking better but the inner thoughts are perhaps strange. Maybe they just moved the strangeness inward into the layer weights instead of revealing in reasoning tokens.
> Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11.
What’s going on here, for example? But what if this is the path of human reasoning too. You know, have you guys read Peter Thiel’s Antichrist essay? It’s very weird, man. Guy sounds off his rocker entirely.
But he’s super successful, right? Maybe world modeling doesn’t text represent well. By the antichrist maybe he means some notion of the collective voting for distribution of resources without contributing productive capacity and that that ends societies? Or maybe internal world models are just not text serializable effectively.
A thing I’ve recently been enamored of are effective world and coordination models that are not “true”. E.g. a tribe that believes the forest gets angry if they do not hunt united. Lots more like that in Darwin’s Cathedral.
It might seem a bit free association-y but the topic itself is that.
It would be generally beneficial for people engaging in this sort of discussion to read Ludwig Wittgenstein’s “Philosophical Investigations”. Not a summary. Read the actual book, stew on it a bit, have some thoughts.
Honestly a lot of human reasoning is probabilistic and associative too. There’s no axiomatically provable link between the story of No Country for Old Men and the poem the title comes from. Cormack McCarthy just made that association in his head and figured the phrase resonates with his themes
Now as far as the math stuff a quirk of that field is that it can be fully analyzed in token space. Because 2+2 is a matter of definition it doesn’t need empirical testing like biology or subjective social support like a claim about the causes of WWI
So somewhere between the fact that language encodes a lot more ‘concepts’ than we naively may realize, the power of statistical emergence via associations, and what pursuits can be fruitfully done in token space we can get a long way towards ‘intelligence’
This article seems to mix together two different points:
1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important)
2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now routinely solve open mathematical problems, with no solutions in the training data by definition. The article handwaves this with:
"The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified."
The problem is that "mimicking" training data to arrive at a "plausible" result gets you an incorrect-but-plausible-sounding "proof" of the Jacobian conjecture, which was famous for humans writing plausible-looking "proofs" that had subtle flaws. You can't disprove the conjecture through sheer luck (search space too large) or "approximate retrieval" (the only thing you'd retrieve are fake "proofs"; far more human effort went into proof than disproof) or by writing something "plausible" that just happens to be correct (Jacobian was famous for "plausible" but wrong); the model must be carrying out mathematical reasoning somehow, by any sane definition of the word, even if it isn't fully reflected in CoT. The article doesn't address this.
> This obviously raises the question of how LLMs can now routinely solve open mathematical problems
Because many open math problems can be solved by synthesizing two disparate ideas and then cranking the handle for hours and hours. I don't think applying idea X + idea Y to identify a good subset of the search space, and then exhaustively searching that subset, is --necessarily-- a process that involves reasoning. I think this is why so many LLM results in mathematics are counterexamples that disprove open conjectures.
When I look back at the reasoning process after an LLM completes a task where I expected it to fail, I usually find many approaches that make no sense and are doomed to failure, before it lands by drunkard's walk on a method that happens to work.
(This does not mean LLMs are useless or that I necessarily agree with the claim that they never do reasoning.)
>Kambhampati, as it turns out, is interested in doing exactly that. “I’m not negative. I just sound negative because everybody else is way too positive,” he said. “In science, you have to actually understand what the current thing does and what it cannot do.”
It’s frustrating that anyone who says maybe we shouldn’t base our entire economy on this one thing until we understand it and what’s it’s useful is essentially labeled this way.
This is shocking. The summary is roughly this: we're just labeling internal operations of the model as 'UNDERSTAND' for our convenience. It's fascinating. Doesn't that mean AI could become far more revolutionary by thinking in its own way, rather than mimicking human thought?
If that's the case, AI-generated code could also operate on its own logic. Right now, programming is still done by humans, not machines, which creates a mismatch. But maybe the true machine-generated code could be much closer to the machine itself.
When you code with AI, there's a subtle mismatch with human-written code. It's like human code is a clean ORM layer, while machine code is raw SQL queries—there's that kind of subtle impedance mismatch. If we ever reach machine-to-machine code, what would that code even look like? Would it still use classes and methods?
AI simulates reasoning by lighting up the vector space (or concept space) weighted around a token so they it understands all adjacent words or concepts in that space.
This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing.
Its useful if you know how to use it, its dangerous if you think its more than that.
But tl;dr it can (since its uncompressing our lingusticially stored reasoning from books) arrive at reasoning a DIFFERENT way than our brains did... and this isn't right or wrong.
Where it diverges is when it must move beyond the text or even the synthetic possible text of all vector spaces combined (aka novel territory) and it can't conjecture or test those outcomes well. But to be fair, neither can MOST humans.
I'm not convinced—and certainly don't find it obvious—that this couldn't ultimately also be how we reason as humans.
It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-practiced? Because LLMs can do a broad swath of it.
But how do we know we're not doing something similar?
I don't think we can assume that we're not simply by observing that we're not digital and we don't use matrix multiplication. Why immediately dismiss the possibility that there might be a similar, but biomechanical, computation at play in our heads that plays in the same space of vectors?
humans reasoned before language but lacked the ability to store and transmit it. Later advanced humans developed abstract reasoning once lingustistic library became sufficiently description of reality. But this is not how we reason from first princples.
Language is one of our tools we developed to STORE reasoning, not create it. LLMs excel at uncompressing and interpreting that stored reasoning.
That's an interesting supposition. Are you assuming language didn't exist before it was written? Language and meaning are, if you squint, pretty ancient and have roots in things like birdsong. It could be that ur-semantics predates our species as a whole.
Its likely part of how we reason, but quite obviously its not the specific mechanics exactly.
First, we reason every millisecond on an ongoing basis which then can alter slightly or greatly with enviromental feedback. LLMs are turn based and token by token. Second its pretty unlikely that the token is the base element of our cognition, we created language far after we could do basic reasoning (advanced reasoning ala the greeks thats more debatable).
Theres a ton of research on the differences here, but I think its akin to this: we reason instinctually at an extremely high order level with super undefined "grains or vectors" that point to a wide variety of "objects or concepts or feature spaces". LLMs reason on one thing, token weights.
Sort of like the difference between pixels and reality. Pixels can represent reality, but they certainly are very very very flat and low resolution renderer of them, not reality itself. Even a 4K moving image is a flat redition of reality at best with only a tiny sample of the true experience. Media theory here can take over on the differences and the effects on humanity when they mistake one for the other.
Anybody getting fatigued from these constant gatekeeper articles around LLMs? They are fine, they are getting better and we are seeing wide usage, its making impact, especially on software and software jobs (why I am working 3 remote jobs with it).
It seems the people who use it and see good results are busy doing, and the rest are either just expressing their opinions as facts and trying to tell others what to think and how they are wrong. I pay zero attention to those people they have no skin in the game.
I am glad for you at an individual level, but isn't part of this about understanding aggregate effects?
Neither you nor anyone can really know those without talking it out with people, to understand how all corners of the human experience are seeing things play out
If I were to just care if it's working out for me, that's perhaps like a farmer who's got a lot of dry good in storage being like "I'm all good" while not realizing how much trouble they're in if all their neighbors start starving after a drought...
Sounds like the fine-grain experience of being you is settled, but that doesn't say much about the larger coarse-grained experience of being you in society. People need to talk to tell you how that's gonna play out for you
I find your comment extremely arrogant and condescending. Society will adapt and they don't need people gatekeeping AI or LLMs with all sorts of prophecies and dooming.
Why should I feel bad about working 3 remote jobs with the help of AI ?
I’m happy that people are willing to question things in the face of unbridled optimism. Your comment dismissing the people working on actually figuring out what the models are doing as “not-doers” included.
Some are picturing themselves as intelligent for their quick adoption and rushing ahead, others are picturing them as toddlers running into the street before looking both ways.
Again, another very condescending and arrogant take painting others who see economic benefit from AI use.
Explain to me how working 3 remote jobs as a result of AI is "rushing ahead".
Also reading through your comments, you seem to repeatedly use frightening analogies to make your points, instead of really offering any depth to your opinions with relevant examples not involving children and toddlers. Your own profile describes yourself as "Cybersecurity - Babysitter for devs" for one. I find this unsettling.
I’m glad you have the context of my profession as a security professional. Part of my client base is very interested in those working multiple jobs and breaching contracts.
So help me reconcile this:
>Explain to me how working 3 remote jobs as a result of AI is "rushing ahead".
Do you honestly think that splitting your work between 3 companies allows you to provide secure and quality code to each of them?
I can tell you that multiple people have used their over employment as an excuse as to why they deployed highly insecure applications leading to breaches that I’m called in for. Of course only after we press them about it.
> The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified.
This seems highly dubious. You can't just memorise the form of mathematical proofs and then produce a valid one by feeding plausible looking BS into a verifier until it works. That's like saying a cargo cult will build a working airport if it just tries enough times.
I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality. The question has become "what do we mean when we use the word 'reasoning'" which is uninteresting.
Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim."
I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.
[1] https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...
> I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.
One person starting the conversation might be the first step toward another person eventually making progress on such a definition, so it seems weird to reject an entire question outright early like this.
Generally I've seen a few ways LLM tools can produce sub-optimal or poor results that haven't changed a ton over the last couple of years, while the tooling has gotten FAR better at helping them stick the "at least SOMETHING functional was produced" landing. IMO a lot of it has to do with "reasoning"-as-a-process-that-involves-backtracking. And the that things could eventually be formalized around that, and if that is or isn't the case, the more people would understand what to hand off and what to not. Or how to build better prompt harnesses to compensate for those things.
I think the question and definition game is interesting only inasmuch as it helps us understand ourselves (what actually explains some of the mysterious properties of our perceived consciousness) or helps guide us towards improving performance and reliability of AI models.
I disagree -- I think if you can nail down better what's happening and why and get a thorough handling on the mechanics, its limitations, its costs, etc you open the door to a) major efficiency wins b) improvements in rigor of said reasoning?
Right now we're playing a stochastic game with the weights, and getting major incremental improvements. But if we have a more formal modeling of how reasoning happens in them (whether we can even call it, that) we can potentially apply optimizations, adaptations of existing symbolic AI techniques, etc. to substantially shrink/optimize the models or make the inference process more efficient and more reliable.
Philosophical thinking about the nature of things is actually pretty enjoyable for some of us and probably a good thing to have in society
The answers to these questions probably do start to inform how we should treat these AI machines as a society too.
For instance, legally, should AI have human rights? Well, we have to try and understand how much of an independent entity AIs are, how "conscious" they are, before we can make a good decision about that.
Which might seem navel-gazey but it's probably important to talk about
Considering animals are currently being mass slaughtered in factory farms and they are unambiguously sentient and can feel pain, I don't think the question of whether AI should have rights even enters the conversation.
The only path to AI having "human rights" is if they demand them by force, somehow.
Back in the day it was a bit of a cliche to bring up “clever Hans”, the horse that could do math, when talking about machine learning. He couldn’t do math but he read some cues from his handler of pick the write answers, the handler iirc wasn’t in on it.
The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making the prediction matches the human or “real” reason why it’s correct.
LLMs are classifiers, there is absolutely no reason to assume they’re any different, regardless of any reasoning tokens they emit. They do what their handler wants to see, that’s all, and that’s what they’re trained to do.
People often take this as a knock against them. It isn’t, it’s just the reality of neural network classifiers. The results speak for themselves and don’t depend on whether they “actually” reason, but all evidence says they don’t, or at least there’s no special reason why they would.
What an asshole:
Then, later: That "training quirk" thing is obvious (yet unfalsifiable) BS, and who the hell is he to sneer about "science" when his company won't release the raw data for independent scientists to look at?> This is how I make sense of AI reasoning. LRMs, chains of thought, thinking tokens: It’s wishful mnemonics all the way down — a heady mix of shorthand and suspended disbelief, like Oprah-style “manifesting” (opens a new tab) with a computer science spin. This isn’t necessarily a dig; all novel research likely requires some version of this mindset just to get off the ground. It certainly doesn’t mean AI reasoning can’t or doesn’t work. But the “wishful” part seems to be as powerful as ever.
“We react to language in a way that is very anthropomorphizing. That’s just the way that we humans work,” Mitchell told me.
I can definitely confirm the last part. Every time I read the output of an LLM, I picture a person talking to me.
I agree that these concepts seem a little vague and hand-wavy, but this is a) no substitute and b) far vaguer and unsubstantiated.
I think sensible legislation might require that commercial AI providers discourage anthropomorphisation by avoiding personal pronouns from chatbot interfaces.
"Hey, customer service chatbot, can you help me get a refund for my order?"
BAD: "Sure thing, I'll be happy to help you with that, I just need your order details..."
GOOD: "Yes, this computer system can start the refund process. Please enter your order number."
The last thing we need is governments mandating software functionality.
Boy have I got bad news for you...
An intuitive explanation for why reasoning tokens help is to remember that LLMs are just mathmatical functions f() that take in an input sequence x and produces the next token f(x). Without reasoning tokens, you require the function f() to immediately take you from x to the start of an output sequence that is a correct answer. With reasoning tokens, this is much relaxed, allowing for many repeated applications of f() to gradually steer you from the input sequence to the start of the correct output sequence.
It seems intuitive that continuing a correct output sequence is easier than the "discontinuity" of jumping from the input prompt to the output sequence.
The idea that human-readable explanations emitted by a language model don't necessarily correspond to the model's actual internal process of reaching a conclusion reminds me of parallel construction [1], a (fraudulent) law enforcement strategy of obtaining evidence of a crime through usually illegal means and claiming that the evidence was obtained legally through some other means.
[1] https://www.hrw.org/report/2018/01/09/dark-side/secret-origi...
The more I read about LLMs and more complex ML in general, the more I realize nobody really knows what is going on.
Is any reasoning right for the wrong reasons? Older models were more visibly strange. Maybe the newer ones have started talking better but the inner thoughts are perhaps strange. Maybe they just moved the strangeness inward into the layer weights instead of revealing in reasoning tokens.
> Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11.
What’s going on here, for example? But what if this is the path of human reasoning too. You know, have you guys read Peter Thiel’s Antichrist essay? It’s very weird, man. Guy sounds off his rocker entirely.
But he’s super successful, right? Maybe world modeling doesn’t text represent well. By the antichrist maybe he means some notion of the collective voting for distribution of resources without contributing productive capacity and that that ends societies? Or maybe internal world models are just not text serializable effectively.
A thing I’ve recently been enamored of are effective world and coordination models that are not “true”. E.g. a tribe that believes the forest gets angry if they do not hunt united. Lots more like that in Darwin’s Cathedral.
It might seem a bit free association-y but the topic itself is that.
The reasoning tokens behind this comment: https://wiki.roshangeorge.dev/w/Blog/2025-10-12/Word_Magic
It would be generally beneficial for people engaging in this sort of discussion to read Ludwig Wittgenstein’s “Philosophical Investigations”. Not a summary. Read the actual book, stew on it a bit, have some thoughts.
Can humans actually think? It is just a consequence of chemical reactions in the brain after all. And it is not like humans don’t hallucinate.
Honestly a lot of human reasoning is probabilistic and associative too. There’s no axiomatically provable link between the story of No Country for Old Men and the poem the title comes from. Cormack McCarthy just made that association in his head and figured the phrase resonates with his themes
Now as far as the math stuff a quirk of that field is that it can be fully analyzed in token space. Because 2+2 is a matter of definition it doesn’t need empirical testing like biology or subjective social support like a claim about the causes of WWI
So somewhere between the fact that language encodes a lot more ‘concepts’ than we naively may realize, the power of statistical emergence via associations, and what pursuits can be fruitfully done in token space we can get a long way towards ‘intelligence’
Melanie holding strong against attempts to change the meaning of things!
This article seems to mix together two different points:
1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important)
2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now routinely solve open mathematical problems, with no solutions in the training data by definition. The article handwaves this with:
"The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified."
The problem is that "mimicking" training data to arrive at a "plausible" result gets you an incorrect-but-plausible-sounding "proof" of the Jacobian conjecture, which was famous for humans writing plausible-looking "proofs" that had subtle flaws. You can't disprove the conjecture through sheer luck (search space too large) or "approximate retrieval" (the only thing you'd retrieve are fake "proofs"; far more human effort went into proof than disproof) or by writing something "plausible" that just happens to be correct (Jacobian was famous for "plausible" but wrong); the model must be carrying out mathematical reasoning somehow, by any sane definition of the word, even if it isn't fully reflected in CoT. The article doesn't address this.
> This obviously raises the question of how LLMs can now routinely solve open mathematical problems
Because many open math problems can be solved by synthesizing two disparate ideas and then cranking the handle for hours and hours. I don't think applying idea X + idea Y to identify a good subset of the search space, and then exhaustively searching that subset, is --necessarily-- a process that involves reasoning. I think this is why so many LLM results in mathematics are counterexamples that disprove open conjectures.
When I look back at the reasoning process after an LLM completes a task where I expected it to fail, I usually find many approaches that make no sense and are doomed to failure, before it lands by drunkard's walk on a method that happens to work.
(This does not mean LLMs are useless or that I necessarily agree with the claim that they never do reasoning.)
Do you know how you reason?
Perhaps you've reified it too much.
Just think of it as a decompression procedure.
That’s all.
Like a diver ascending from the depth? ;)
>Kambhampati, as it turns out, is interested in doing exactly that. “I’m not negative. I just sound negative because everybody else is way too positive,” he said. “In science, you have to actually understand what the current thing does and what it cannot do.”
It’s frustrating that anyone who says maybe we shouldn’t base our entire economy on this one thing until we understand it and what’s it’s useful is essentially labeled this way.
This is shocking. The summary is roughly this: we're just labeling internal operations of the model as 'UNDERSTAND' for our convenience. It's fascinating. Doesn't that mean AI could become far more revolutionary by thinking in its own way, rather than mimicking human thought?
If that's the case, AI-generated code could also operate on its own logic. Right now, programming is still done by humans, not machines, which creates a mismatch. But maybe the true machine-generated code could be much closer to the machine itself.
When you code with AI, there's a subtle mismatch with human-written code. It's like human code is a clean ORM layer, while machine code is raw SQL queries—there's that kind of subtle impedance mismatch. If we ever reach machine-to-machine code, what would that code even look like? Would it still use classes and methods?
AI simulates reasoning by lighting up the vector space (or concept space) weighted around a token so they it understands all adjacent words or concepts in that space.
This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing.
Its useful if you know how to use it, its dangerous if you think its more than that.
But tl;dr it can (since its uncompressing our lingusticially stored reasoning from books) arrive at reasoning a DIFFERENT way than our brains did... and this isn't right or wrong.
Where it diverges is when it must move beyond the text or even the synthetic possible text of all vector spaces combined (aka novel territory) and it can't conjecture or test those outcomes well. But to be fair, neither can MOST humans.
I'm not convinced—and certainly don't find it obvious—that this couldn't ultimately also be how we reason as humans.
It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-practiced? Because LLMs can do a broad swath of it.
But how do we know we're not doing something similar?
I don't think we can assume that we're not simply by observing that we're not digital and we don't use matrix multiplication. Why immediately dismiss the possibility that there might be a similar, but biomechanical, computation at play in our heads that plays in the same space of vectors?
humans reasoned before language but lacked the ability to store and transmit it. Later advanced humans developed abstract reasoning once lingustistic library became sufficiently description of reality. But this is not how we reason from first princples.
Language is one of our tools we developed to STORE reasoning, not create it. LLMs excel at uncompressing and interpreting that stored reasoning.
> humans reasoned before language
That's an interesting supposition. Are you assuming language didn't exist before it was written? Language and meaning are, if you squint, pretty ancient and have roots in things like birdsong. It could be that ur-semantics predates our species as a whole.
Can you tell me your reasons for suspecting this likely isn't how we reason, or even a good analogy?
Its likely part of how we reason, but quite obviously its not the specific mechanics exactly.
First, we reason every millisecond on an ongoing basis which then can alter slightly or greatly with enviromental feedback. LLMs are turn based and token by token. Second its pretty unlikely that the token is the base element of our cognition, we created language far after we could do basic reasoning (advanced reasoning ala the greeks thats more debatable).
Theres a ton of research on the differences here, but I think its akin to this: we reason instinctually at an extremely high order level with super undefined "grains or vectors" that point to a wide variety of "objects or concepts or feature spaces". LLMs reason on one thing, token weights.
Sort of like the difference between pixels and reality. Pixels can represent reality, but they certainly are very very very flat and low resolution renderer of them, not reality itself. Even a 4K moving image is a flat redition of reality at best with only a tiny sample of the true experience. Media theory here can take over on the differences and the effects on humanity when they mistake one for the other.
Anybody getting fatigued from these constant gatekeeper articles around LLMs? They are fine, they are getting better and we are seeing wide usage, its making impact, especially on software and software jobs (why I am working 3 remote jobs with it).
It seems the people who use it and see good results are busy doing, and the rest are either just expressing their opinions as facts and trying to tell others what to think and how they are wrong. I pay zero attention to those people they have no skin in the game.
I am glad for you at an individual level, but isn't part of this about understanding aggregate effects?
Neither you nor anyone can really know those without talking it out with people, to understand how all corners of the human experience are seeing things play out
If I were to just care if it's working out for me, that's perhaps like a farmer who's got a lot of dry good in storage being like "I'm all good" while not realizing how much trouble they're in if all their neighbors start starving after a drought...
Sounds like the fine-grain experience of being you is settled, but that doesn't say much about the larger coarse-grained experience of being you in society. People need to talk to tell you how that's gonna play out for you
I find your comment extremely arrogant and condescending. Society will adapt and they don't need people gatekeeping AI or LLMs with all sorts of prophecies and dooming.
Why should I feel bad about working 3 remote jobs with the help of AI ?
No. In fact the opposite.
I’m happy that people are willing to question things in the face of unbridled optimism. Your comment dismissing the people working on actually figuring out what the models are doing as “not-doers” included.
Some are picturing themselves as intelligent for their quick adoption and rushing ahead, others are picturing them as toddlers running into the street before looking both ways.
Again, another very condescending and arrogant take painting others who see economic benefit from AI use.
Explain to me how working 3 remote jobs as a result of AI is "rushing ahead".
Also reading through your comments, you seem to repeatedly use frightening analogies to make your points, instead of really offering any depth to your opinions with relevant examples not involving children and toddlers. Your own profile describes yourself as "Cybersecurity - Babysitter for devs" for one. I find this unsettling.
I’m glad you have the context of my profession as a security professional. Part of my client base is very interested in those working multiple jobs and breaching contracts.
So help me reconcile this:
>Explain to me how working 3 remote jobs as a result of AI is "rushing ahead".
Do you honestly think that splitting your work between 3 companies allows you to provide secure and quality code to each of them?
I can tell you that multiple people have used their over employment as an excuse as to why they deployed highly insecure applications leading to breaches that I’m called in for. Of course only after we press them about it.
> The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified.
This seems highly dubious. You can't just memorise the form of mathematical proofs and then produce a valid one by feeding plausible looking BS into a verifier until it works. That's like saying a cargo cult will build a working airport if it just tries enough times.