OC below by @HaraldvonBlauzahn@feddit.org
What called my attention is that assessments of AI are becoming polarized and somewhat a matter of belief.
Some people firmly believe LLMs are helpful. But programming is a logical task and LLMs can’t think - only generate statistically plausible patterns.
The author of the article explains that this creates the same psychological hazards like astrology or tarot cards, psychological traps that have been exploited by psychics for centuries - and even very intelligent people can fall prey to these.
Finally what should cause alarm is that on top that LLMs can’t think, but people behave as if they do, there is no objective scientifically sound examination whether AI models can create any working software faster. Given that there are multi-billion dollar investments, and there was more than enough time to carry through controlled experiments, this should raise loud alarm bells.
If you have to use AI - maybe your work insists on it - always demand it cite its sources, hope they are relevant, and go read those instead.
Reponding to another comment in opensource@lemmy.ml:
Writing code is itself a process of scientific exploration; you think about what will happen, and then you test it, from different angles, to confirm or falsify your assumptions.
What you confuse here is doing something that can benefit from applying logical thinking with doing science. For exanple, mathematical arithmetic is part of math and math is science. But summing numbers is not necessarily doing science. And if you roll, say, octal dice to see if the result happens to match an addition task, it is certainly not doing science, and no, the dice still can’t think logically and certainly don’t do math even if the result sometimes happens to be correct.
For the dynamic vs static typing debate, see the article by Dan Luu:
https://danluu.com/empirical-pl/
But this is not the central point of the above blog post. The central point of it is that, by the very nature of LLMs to produce statistically plausible output, self-experimenting with them subjects one to very strong psychological biases because of the Barnum effect and therefore it is, first, not even possible to assess their usefulness for programming by self-experimentation(!) , and second, it is even harmful because these effects lead to self-reinforcing and harmful beliefs.
And the quibbling about what “thinking” means is just showing that the arguments pro-AI has degraded into a debate about belief - the argument has become “but it seems to be thinking to me” even if it is technically not possible and also not in reality observed that LLMs apply logical rules, cannot derive logical facts, can not explain output by reasoning , are not aware about what they ‘know’ and don’t ‘know’, or can not optimize decisions for multiple complex and sometimes contradictory objectives (which is absolutely critical to any sane software architecture).
What would be needed here are objective controlled experiments whether developers equipped with LLMs can produce working and maintainable code any faster than ones not using them.
And the very likely result is that the code which they produce using LLMs is never better than the code they write themselves.
What’s the difference between copying a function from stack overflow and copying a function from a llm that has copied it from SO?
LLM are sort of a search engine with advanced language substitution features nothing more nothing less.
That is actually missing an important issue, hallucinations.
Copying from SO means you are copying from a human who might be stupid or lie but rarely spews out plausible sounding hot garbage (not never though) and because of other users voting and reputation etc etc, you actually do endup with a decently reliable source.
With an LLM you could get something made up based on nothing related to the real world. The LLM might find your question to be outside of it’s knowledge but instead of realizing it it would just make up what it thinks sounds convincing.
It would be like if you asked me how that animal that is half horse and half donkey is called and instead of saying “shit i’m blanking” I would say “Oh, that is called a Drog” and I couldn’t even tell you that I just made up that word because I will now be convinced that this is factual. Btw it’s “mule”
So there is a real difference until we solve hallucinations, which right now doesn’t seem solvable but at best reduced to insignificance (maybe)
That’s why you meed to know the cavieats of the tool you are using.
LLM hallucinate. People willing to use them need to know, where is more prone to hallucinate. Which is where the data about the topic you are requesting is more fuzzy. If you ask for the capital of France is highly unlikely you will get an hallucination, if you as for the color of the hair of the second spouse of the fourth president of the third French republic, you probably will get an hallucination.
And you need to know what are you using it for. If it’s for roleplay, or any not critical matters you may not care about hallucinations. If you use them for important things you need to know that the output needs to be human reviewed before using it. For some things it may be worth the human review as it would be faster that writing from zero, for other instances it may not be worth it and then a LLM should not be used for that task.
As an example I just was writing some lsp library for an API and I tried the LLM to generate it from the source documentation. I had my doubts as the source documentation is quite bigger that my context size, I tried anyway but I quickly saw that hallucinations were all over the place and hard to fix, so I desisted and I’ve been doing it myself entirely. But before that I did ask the LLM how to even start writing such a thing as it is the first time I’ve done this, and the answer was quite on point, probably saving me several hours searching online trying to find out how to do it.
It’s all about knowing the tool you are using, same as anything in this world.
LLM are poor snapshots of a search engine with no way to fix any erroneous data. If you search something on Stack you get the page with several people providing snippets and debating the best approach. The LLM does not give you this. Furthermore if the author goes back and fixes an error in their code the search will find it whereas the LLM will give you the buggy code with no way to reasonably update it
LLM have major issues and even bigger limitations. Pretending they are some panacea is going to disappoint.
LLM also does not bully you for asking. Nor it says “duplicated question” for non duplicated questions… There’s a reason people prefer LLM to SO nowadays.
It’s not panacea. But it’s not the doom world destroying useless machine that some people like to tell it is.
It’s a useful tool for some task if you know how to use it. Everyone who actively use it is because we have find put that it works for us better than other tools for that task, of not we would not use it.
Giving my own personal experience, I tend to ask first to an LLM rather that what I used to do digging in old SO answers because I get the answer quicker and a lot of the times just better. It’s not perfect by any stretch of the imagination, but it serves me a purpose.
For instance last week I needed a PowerShell command to open an app compatibility menu from the command line. I asked and got this as a response:
(New-Object -ComObject Shell.Application).Namespace((Split-Path “C:\Ruta\A\TuPrograma.exe”)).ParseName((Split-Path “C:\Ruta\A\TuPrograma.exe” -Leaf)).InvokeVerb(“P&roperties”)
Worked at first try, exactly as I wanted.
You are free to try a search engine with the query “PowerShell command to open an app compatibility menu from the command line” and check for yourself how little help the firsts results get you.
It’s a tool, as many others. The magic lies in knowing when and how to use it. For other things I may not use it, but after a couple of years using it I’m developing a good sense of which questions does it handle well and which questions is better not even to try.
It takes an enormous amount of energy and processing power to create these shitty snapshots so in many ways it is doom considering it will dramatically increase our energy usage.
I get it, you are an AI supporter but you fail to critically analyze it or even understand it. What tool would you use that you can’t correct errors to or even determine how it works. You are really operating on faith here that the black box your getting an answer from is giving you the correct answer.
Perhaps a code snippet works, but after this is where it all falls apart. What if the snippet does not work or causes a problem. The LLM has nothing to offer you here.
Not really.
I self host my own LLM. Energy consumption for queries is lower than gaming according to my own measures. And the models are not made so frequently (I use models made last year still). And once the model is done is infinitely reusable by anyone.
I get that you are starting by the axiom “AI is bad” and then create the arguments needed to support that axiom. Instead of going the other way around with an open mind.
I told you my own personal experience with it. Take it as you want. For me, my situation will be the same. I would keep using same as I use any other tool that works for me, and will stop using it when there’s something better same as I’ve done countless times. I’m not easy to peer pressure into any particular stance, so I can form my own opinions based on what I test for myself. I really think a lot of arguments against AI boil down to some sort of political stance. AI hurt a series of small artists which had a very big voice in some spaces, and thus an anti-AI political movement was created. My own copyleft morals made me undisturbed by this original complains about generative AI, and the rest of arguments have been very unconvincing, straight up fake, logical fallacies, or just didn’t really check out with the reality I was able to test by myself.
For instance I saw other post today saying how 3 watts hour per query was an absolute energy waste for a household. When that’s absolutely nothing compared to the 30.000 watts hour a typical household spend each day, even with quite and amount of queries. Sincerely I spent last few months with one of these devices to measure energy consumption attached to my computer and AI energy usage was really underwhelming compared with what people told me it was going to be. AAA gaming is consistently more energy hungry.
I know you are willfully being ignorant here as AI data centers are projected to use more electricity than the entire nation of Japan by 2030.
Your own hosted LLM is not the problem nor the issue we are even discussing and quite frankly a little insulting you bring it up
I am not anymore anti-AI than any tool that you can’t determine is accurate nor correct if there is an issue with it. LLM have a long way to go before they are even a fraction of what they claim to be.
Another problem is they do not cite where they get their answers from. Without the ability to audit the answers you are given you won’t know how accurate they are.
I have listed several legitimate gripes about LLM. I find your fanboism misplaced and I think you are just playing devil’s advocate at this point. AI is a hype train and I am sick of it already.
I will just copy my other response about datacenters energy usage, ignore the parts not related to our conversation:
Google is not related with chatgpt. Chatgpt parent company is openAI which is a competitor with google.
A more rational explanation is that technology and digital services on general have been growing and are on the rise. Both because more and more complex services are being offered, and more importantly more people are requesting those services. Whole continents that used not to be cover by digital services are now covered. Generative AI is just a very small part of all that.
The best approach to reduce CO2 emissions is to ask for a reduction in human population. From my point of view is the only rational approach, as with a growing population there’s only two solutions, pollute until we die, or reduce quality of life until life is not worth living. Reducing population allows for fewer people to live better loves without destroying the planet.
It also arises the question on why am I responsible if a big tech company decided to make an llm query of every search or overuse the technology, when I am talking about a completely different usage of that technology, that doesn’t even reach a 20-30 queries a day which would have a power usage of less than a few hundreds wh at most, which os negligible in the scheme of global warming and my total energetic footprint.
How it’s being a fanboy saying that “It works for me in some particular cases and not others, it’s a tool that can be used”.
Please, read again this conversation and do a second guess on who is a radical extremist here.
In the case we were talking, writing code, I am the auditor of the answers. I do not ““vibe code”” I read the code that’s proposed, understand it, and if it’s code that I would have written I copy it, if not I change it. “Vibe coding” is an example of bad usage of the tool that would lead to problems. All code not written by yourself and copied from other source should be reviewed. Once it pass my review is as good as my own code. If it fail it would fail the same as any other code witten by me, as it’s something that I was clearly unable to see.
For instance a couple of months ago I wrote a small API service that worked fine at first and suddenly stopped working a few weeks in production. It was a stupid mistake I made, and I needed no LLM to do that mistake. The service was so simple that I didn’t really even used LLM there. But I made a mistake regardless. I could have use AI and get the same bad function that caused the issue. And the blame would still be mine for not seeing the problem.
Once again is a tool. If some jackass decide to vibe code an app and it’s a shit app, is a bad use of the tool. But some other people can de proper reviews and analysis of the generated code and assume full responsibility of any failures of that code.
I fear this is a problem that may never be solved. I mean that people of any intelligence fall for the mind’s biases.
There’s just too little to be gained feelings-wise. Yeah, you make better decisions, but you’re also sacrificing “going with the flow”, acting like our nature wants us to act. Going against your own nature is hard and sometimes painful.
Making wrong decisions is objectively worse, leading to worse outcomes, but if it doesn’t feel worse (because you’re not attributing the effects of the wrong decisions to the right cause, i.e. acting irrationally), then why should a person do it. If you follow the mind’s bias towards attributing your problems away from irrationality, it’s basically a self-fulfilling prophecy.
Great article.
Another similar article that’s really good:
LLMs can’t think - only generate statistically plausible patterns
Ah still rolling out the old “stochastic parrot” nonsense I see.
Anyway on to the actual article… I was hoping it wouldn’t make these basic mistakes:
[Typescript] looks more like an “enterprise” programming language for large institutions, but we honestly don’t have any evidence that it’s genuinely more suitable for those circumstances than the regular JavaScript.
Yes we do. Frankly if you’ve used it it’s so obviously better than regular JavaScript you probably don’t need more evidence (it’s like looking for “evidence” that film stars are more attractive than average people). But anyway we do have great papers like this one.
Anyway that’s slightly beside the point. I think the article is right that smart people are not invulnerable to manipulation or falling for “obviously” stupid ideas. I know plenty of very smart religious people for example.
However I think using this to dismiss LLMs is dumb, in the same way that his dismissal of Typescript is. LLMs aren’t homeopathy or religion.
I have used LLMs to get some work done and… guess what, it did the work! Do I trust it to do everything? Obviously not. But sometimes I don’t need perfect code. For example recently I asked it to create an example SystemVerilog file for me utilising as many syntax features as possible (testing an auto-formatter). It did a pretty good job. Saved some time. What psychological hazard have I fallen for exactly?
Overall, B-. Interesting ideas but flawed logic.
Ah still rolling out the old “stochastic parrot” nonsense I see.
It is a bunch of stochastic parrots. It just happens frequently that the words they are parroting were orginally written by a bunch of intelligent people which were knowledgeable in their fields.
Note this does makes the parrots intelligent - in the same way that a book written by Einstein to explain special relativity has any own intelligence. Einstein was intelligent, his words transport his intelligent ideas, but the book conveying them to other people (as, the printed pages with cardboard cover) is as dumb as a stone. You would not ask a piece of cardboard so solve a math problem, would you?
Your comment doesn’t account for the fact that LLMs can generalise. Often not very well but they can produce outputs for inputs not seen in their training sets. Otherwise what would be the point?
You would not ask a piece of cardboard so solve a math problem, would you?
Uhhh you know LLMs can solve quite complex maths problems? Including novel ones.
LLMs can’t think - only generate statistically plausible patterns
Ah still rolling out the old “stochastic parrot” nonsense I see.
Ah still rolling out the old “computers think” pseudo-science.
I have used LLMs to get some work done and… guess what, it did the work!
Ah yes the old pointless vague anecdote.
What psychological hazard have I fallen for exactly?
Promoting pseudo-science.
Overall D. Neither interesting nor new nor useful.
Ah yes the old pointless vague anecdote.
If your argument is “LLMs can’t do useful work”, and then I say “no, I’ve used them to do useful work many times” how is that a pointless vague anecdote? It’s a direct proof that you’re wrong.
Promoting pseudo-science.
Sorry what? This is bizarre.
Amen
And to add that smart people fall for dumb biases, we just need to look at the object oriented mania of the 2000s to late 2010s to see us shoehorn in one paradigm into everything without critically considering whether it made sense over other models.
Can an LLM do everything I need yet? No.
But is a stochastic parrot good enough to help me complete a function and help me restructure code? Yes definitely.
Claude is good enough for so much of the low value code I write that is actually a useful tool. I have to review the code but it’s useable.
I use AI search to lookup functions that I don’t need detailed docs for, or to help me debug arcane library specific errors (just had one earlier today where in polars the list and array types are very much not interchangeable and the explode method was failing).
I still read the docs on things that are critical, and I write the critical paths and dictate structure and understand the problem im solving well.
It’s really amazing the number of people trying to argue that LLMs are useless, while simultaneously so many people are using them successfully. Makes me wonder if they’ve even tried them.
What called my attention is that assessments of AI are becoming polarized and somewhat a matter of belief.
Proceed to write a belief as a statement in the following paragraph
If you think LLMs doesnt think (I won’t argue that they arent extremely dumb), please define what is thinking, before continuing, and if your definition of thinking doesn’t apply to humans, we won’t be able to agree.
The burden of proof is on those who say that LLMs do think.
I asked for your definition, I cannot prove something if we do not agree on a definition first.
You also missread what I said, I did not said AI were thinking.
The burden of proof is on the one who made an affirmation.
I’m not the one who made an affirmation which field experts doesn’t know the answer.
But depending of your definition of thinking, some can be answered.I don’t think y’all are disagreeing but maybe this sentence is somewhat confusing:
If you think LLMs doesnt think (I won’t argue that they arent extremely dumb), please define what is thinking,
Maybe the “doesnt” shouldn’t be there.
No it is here because that’s what they claim.
Nobody yet know how it work, we don’t know how LLMs process information.
Anyone who claim it really think, or it isn’t thinking, is believing, this is not something the current ML field know.Well, the neural network is given a prefix (series of tokens) and a token, and it spits out how likely is it that the token follows the prefix. Text is generated by calculating this probability for all known tokens, then picking one random, weighted based on the calculated probabilities.
And the brain is made out of neurons that sends electric signals between them and operate muscles.
That doesnt explain how the brain think.It allows us to conclude that an LLM doesn’t “think” about what it is saying. Based on the mechanics, the LLM doesn’t even know it’s a participant in the conversation.
I don’t think the current common implementation of AI systems are “thinking” and I’ll base my argument on Oxford’s definitions of words. Thinking is defined as “the process of using one’s mind to consider or reason about something”. I’ll ignore the word “mind” and focus on the word “reason”. I don’t think what AIs are doing counts as reasoning as defined by Oxford. Let’s go to that definition: “the power of the mind to think, understand, and form judgments by a process of logic”. I take issue with the assertion that they form judgments. For completeness, but I don’t think it’s definition is particularly relevant here, a judgment is: “the ability to make considered decisions or come to sensible conclusions”.
I think when you ask an LLM how many 'r’s there are in Strawberry and questions along this line you can see they can’t form judgments. These basic but obscure questions are where you see that the ability to form judgements isn’t there. I would also add that if you “form judgments” you probably don’t need to be reminded you formed a judgment immediately after forming one. Like if I ask an LLM a question, and it provides an answer, I can convince it that it was wrong whether or not I’m making junk up or not. I can tell it it made a mistake and it will blindly change it’s answer whether it made a mistake or not. That also doesn’t feel like it’s able to reason or make judgments.
This is where all the hype falls flat for me. It feels like sometimes it looks like a concrete wall, but occasionally that concrete wall is made of wet paper. You can see how impressive the tool is and how paper thin it is at the same time. It’s cool, it’s useful, it’s fake, and that’s ok. Just be aware of what the tool is.
I think when you ask an LLM how many 'r’s there are in Strawberry and questions along this line you can see they can’t form judgments.
Like a LLMs you are making the wrong affirmation based lacking knowledge.
Current LLMs input, and output tokens, they dont ever see the individual letters, they see tokens, for straberry, they see 3 tokens:
They dont have any information on what characters are in this tokens. So they come up with something. If you learned a language only by speaking, you’ll be unable to write it down correctly (except purely phonetical systems), instead you’ll come up with what you think the word should be written.
I would also add that if you “form judgments” you probably don’t need to be reminded you formed a judgment immediately after forming one.
You come up with the judgment before you are aware of it: https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-reveal-our-choices-before-were-even-aware-of-them--st
can tell it it made a mistake and it will blindly change it’s answer whether it made a mistake or not. That also doesn’t feel like it’s able to reason or make judgments.
That’s also how the brain can works, it come up with a plausible explanation after having the result.
See the experience which are spoken about here: https://www.youtube.com/watch?v=wfYbgdo8e-8I showed the same behavior in humans of some behavior you observed in LLMs, does this means that by your definition, humans doesnt think ?
If the LLM could reason, shouldn’t it be able to say “my token training prevents me from understanding the question as asked. I don’t know how many 'r’s there are in Strawberry, and I don’t have a means of finding that answer”? Or at least something similar right? If I asked you what some word in a language you didn’t know, you should be able to say “I don’t know that word or language”. You may be able to give me all sorts of reasons why you don’t know it, and that’s all fine. But you would be aware that you don’t know and would be able to say “I don’t know”.
If I understand you correctly, you’re saying the LLM gets it wrong because it doesn’t know or understand that words are built from letters because all it knows are tokens. I’m saying that’s fine, but it should be able to reason that it doesn’t know the answer, and say that. I assert that it doesn’t know that it doesn’t know what letters are, because it is incapable of coming to that judgement about its own knowledge and limitations.
Being able to say what you know and what you don’t know are critical to being able to solve logic problems. Knowing which information is missing and can be derived from known things, and which cannot be derived is key to problem solving based on reason. I still assert that LLMs cannot reason.
I’m saying that’s fine, but it should be able to reason that it doesn’t know the answer, and say that.
That is of course a big problem. They try to guess too much stuff, but it’s also why it kinda works. Symbolics AI have the opposite problem, they are rarely useful, because they can’t guess stuff, they are rooted in hard logic, and cannot come up with a reasonable guess.
Now humans also try to guess stuff and sometimes get it wrong, it’s required in order to produce results from our thinking and not be stuck in a state where we don’t have enough data to do anything, like a symbolic AI.Now, this is becoming a spectrum, humans are somewhere in the middle of LLMs and symbolics AI.
LLMs are not completely unable to say what they know and doesnt know, they are just extremely bad at it from our POV.The probleme with “does it think” is that it doesn’t give any quantity or quality.
Is the argument that LLMs are thinking because they make guesses when they don’t know things combined with no provided quantity or quality to describe thinking?
If so, I would suggest that the word “guessing” is doing a lot of heavy lifting here. The real question would be “is statistics guessing”? I would say guessing and statistics are not the same thing, and Oxford would agree. An LLM just grabs tokens based on training data on what word or token most likely comes next, it will just be using what the statistically most likely next token or word is. I don’t think grabbing the highest likely next token counts as guessing. That feels very algorithmic and statistical to me. It is also possible I’m missing the argument still.
Is the argument that LLMs are thinking because they make guesses
No, it’s that you can’t root the argument that they don’t think over the fact they make stuff up, because humans too. You could root it in the amount of things it guess wrong, but it’s extremely hard to measure.
Again, I’m not claiming that they think, but that we don’t know until one or the other is proven.
Right now, thinking one, or the other is true, is belief.I think you can make a strong argument that they don’t think rooted in words should mean something and that statistics and thinking don’t mean the same thing. To me, that feels like a fairly valid argument.
Since LLMs runs on CPUs with a lot of memory, do you agree that my calculator is thinking?
This argument makes no more sense than trying to say that a plant is thinking because brains are made of cells and so are plants.
You think computation is thinking ?
I asked for your definition of thinking.
The OP talked about belief, then made a statement using a word that is not precisely defined.
If you think computation is thinking then by your definition the LLM is thinking.
But that’s your definition of thinking.
’ Please succinctly answer a question of philosophy that has plagued mankind for thousands of years. can’t? <crosses arms with a superior smirk> I win’
Claiming LLMs can’t think with the current informations available, and calling that not a belief, is claiming to have a response to this philosophy question.
The only sensible answer is saying you don’t know, or being aware and communicating that your statement is a belief.
