The topic is hot nowadays, and I have some ideas about it. It's easier to think about them when they're written somewhere, so here they are.
I also added some LLM-generated infographics to each section. Because, you see, they both illustrate and meta-illustrate my ramblings.
Do LLMs solve new problems, or only those they've seen in datasets?
I think of problems and their novelty using the following model. Say you have a set of solved intellectual problems-proving a theorem, writing a high-load backend, drawing a webtoon, or conducting empirical research. Each of the problems was solved using some methods. A set of known methods forms the second level of our pyramid. Then, methods have some ideas behind them, and some methods share their underpinning ideas. That's how we get the third level. Proceeding upward through the pyramid in this manner, we arrive at more and more abstract concepts.
No doubt, on the lower level, there are many problems not solved: many backends not written yet, many pictures not drawn. Doing this work is solving new problems, in a sense. After all, before LLMs it wasn't possible to do so automatically with considerable amount of generalization. In software engineering we had project templates, or research protocols for empirical studies, but a human was necessary. Now it's not so. But compared to what humans could do, there's only improvement in speed, accuracy and scalability.
One may ask: are those novelties real? We already have blueprints for them on the second level of our pyramid. For example, a course on software engineering that teaches you how to design systems, or techniques used to draw various kinds of paintings, artistic styles and schools.
To me, there are no signs of discoveries on the second level made by LLMs yet. I see only level-one problem solving, boosted and scaled, no doubt, but should we call it a novelty? Of course, there are signs of new approaches to the old work, like focusing more on requirements and specifications in software development, but those improvements happen with heavy involvement of humans, not autonomously.
How does a new method emerge? My understanding is this. When a human uses a method from the second level to produce something on the first level, they possibly (not always) learn something about the method itself. Its boundaries, upsides and downsides, this stuff. If someone used a second-level method ten times, from the accumulated experience, they possibly extract some useful aspect of it which wasn't there before. If they do the hard work of reflecting on that and sharing with others, then the level-two method grows a bit, enriching level two as a whole. When a genius (whatever that means) stumbles upon a problem, maybe s/he even reaches level 3 or above.
How exactly does this addition happen? My first guess is that humans have a very long context. An LLM runs, solves its task, and then disappears into the void. All the context is lost. A human solves the task and stores the experience somewhere. Humans live many decades, so this experience, distilled, repeated, distilled again, sometimes (not always, but there are 9 billion of us) molds into something new.
If it's the only difference between us and LLMs, then we'll see second-level discoveries from LLMs soon (but possibly won't understand them).
One important question is whether this experience can be accumulated purely in form of words or not. Humans definitely accumulate their experience beyond the speech level (or maybe we should say "below"). Feelings, emotions, body sensations, some guttural ideas that are hard to express. I'd even say (and some psychologists like Wilhelm Reich would confirm) that it's our main level. In contrast, modern AI agents store their context either as text or other "final stage" artifacts. They don't edit their network weights right away after each interaction with the world, but (so long as we stick to artificial neural network metaphor for human brain) humans do exactly that.
Maybe there are other differences, more qualitative than quantitative ones, like human brains utilizing quantum physics effects or something.
When to use LLMs' help and when to avoid it
Talking about helpers and tools, as with any other tool, there are cases when they're helpful, cases when they're not particularily helpful, and cases when they're outright detrimental.
My take on LLMs for any complex intellectual work, such as programming, is this. You may delegate work to LLMs, but you may also delegate understanding. Delegating work while retaining understanding is generally not risky. Delegating understanding, on the contrary, is very risky.
There are cases when you already understand something. If you're a skilled backend developer, and you use an LLM to write code for you but still review its results, you delegate only work. You get pure benefits with virtually no costs or risks.
Then, there are cases when you don't understand something and don't want to understand it. Maybe the thing is so ubiquitous and so well-documented, and the outcomes of the work are so standardized, that you don't care and decide to trust the LLM and the state of the art it absorbed from training. As an example, you're again a backend developer, but this time you need a front-end for your hobby project. You don't care much about the front-end architecture or the performance, and you definitely don't plan to become a front-end guy. You also assume that millions of front-end apps have already been written by now, and thousands of books and guides on the subject are available. Then you delegate both work and understanding, accepting the risks of being unable to scale the results or modify them as you want—or having an accidental JS crypto miner working on your users' machines.
Finally, there are cases when understanding is an indispensable part of your goal. As a maximal example, imagine that you're a student striving to finish a computer science course. You need to make your way through programming assignments. Should you delegate this work to LLMs? Your only goal is understanding, and the lab work is how you gain this understanding. By delegating it, you undermine the whole endeavor.
There are nuances between those three corner cases. For instance, when starting a weekend project, what interests you more: getting a grip on new tech, or building a usable contraption doing some job for you? Maybe the first time you go with a contraption that just does the job, and postpone the understanding until you decide to improve the tool you've got.
There's one more dimension to this, namely, the long-term effects of each choice. A back-end dev from the second example will never learn a bit of JavaScript, though successfully accomplishing their short-term goals. This means that the market loses a full-stack dev where it could get one "in the good old days". Forget about the market, but for the dev it means less technical breadth, less ability to communicate with their peers, less agility if the technology trends change, and less experience to draw on something "on the level two" (see the previous section). On the other hand, they saved a lot of time and money for themselves and their employer. What outweighs what in each case — that is the question to answer.
Another rule I formulated for myself: is the result disposable? If I want to get some work from an LLM, delegating to it both the work and the understanding — how valuable is this work and for how long will it remain valuable? If it'll be used several times before being thrown in the trash, then of course I'll use an LLM. If I want a thing to last and be reused, then understanding is important to me and I shouldn't delegate that understanding.
In the software realm, applications mostly fall into the disposables category, but frameworks and libraries belong to the latter, lasting one.
In the art domain, marketing art falls into the disposables, along with word illustrations for language learning cards — but visual style experiments or, say, paintings that go beyond visual art and express something of universal value, are designed to last (although only time will tell whether they truly will).
In writing, all the reports, requirements, and documentation fall into disposables (to the extent that, in the case of software, one can generate bespoke documentation for oneself from the source code, which calls into question the need for documentation at all), but poems, novels, and scientific monographs should last (or at least compete with other poems, novels, and monographs for the right to last).
Why is disposability a criterion? Because if I plan to keep a thing for long, I would inevitably want to improve it. I'll add features to a framework, write new chapters to a book. I wouldn't be able to do that without understanding.
