Recently I asked an agent for a feature where the screen will get dimmer after some time, and eventually after a day the screen will turn off and then turn back on if I tap, which should be pretty easy in esphome.

But: some past agent decided what it wanted wasn’t in esphome, so it made its own custom module in c++ for the display, spelunking in pdf datasheets; then the weeds were so deep the new agent decided to add on to that, and to read the power controller and display controller spec sheets and somehow implement my dimming feature by writing raw values to some nondescript registers.

The contraption worked until it didn’t and once every so often my esp32 would just completely hang with black screen and no connnectivity and needed a power cycle, presumably because whatever low level register writing the agent implemented eventually breaks something. I told the agent to get rid of all custom stuff and wherever possible just use default esphome yaml, now it’s been working fine. I don’t really have a point with the anecdote, just that I’m also getting tired of such scenarios like the OP.

This... quite describes what the current AI feels like to me. So I really don't see a problem. If AI does too much for you, you should, uh, just prompt it to do less.

Everyone talks about how their AI escapes sandbox and virtualization or delete their entire repos or something. Perhaps they're using a skynet I'm not aware of. From my experience when I added "discuss; don't edit any file yet" at the end of the prompt I've never seen the LLM go against my will and edit the files anyway.

I think the solution is toward more formalization of the behavior we expect of various systems. E.g., produce a Lean proof that a program never behaves in a certain way, and if you can’t, iterate on the program until it satisfies such a proof. I’ve only tinkered around with this a little bit, but it seems promising on toy problems. Scaling up to tech-company level infra is going to take quite some time however I think.

Creating products with AI is like creating a tightly packed ball of wool. Then, to try to understand what happened you have to somehow pick a strand and pull it out to have a look. Very difficult with tightly packed wool.

I've run some experiments in the past month that get me closer to being able to work with this opaque bundle.

It's the main thing I think about every day, how to solve this problem.

When carpenters moved from mostly hand tools to mostly power tools, it didn't take away much of the need to understand the techniques of woodworking, it just made it all faster.

Reading a nonfiction book is very different than reading a book summary, even if (as is often the case), the ideas in the book could have been condensed way down. The point is that actually reading the whole book means you spend far more time engaging with the ideas, and so they become more deeply embedded in your neural network. In things like sports, this is just obvious, in that no one expects someone to become a good soccer player by just reading the rulebook, and obviously more practice is the main way you get better.

The developers of chatbot technology have or are about to hit an inflection point where the tool is essentially done, then they're going to add too much sugar trying to make it some sort of god tool where it becomes overly complicated, overly convoluted, with dodgy results and too much magic.

My LLM is an assistant that helps my designs and decision process, implements the final work expertly, and I'm always in the driver's seat. And the development speed is astronomical.

You aren't going to get that. Why would they even offer you that? You're going to be the meatbag peripheral to an AI.

This is why I'm still primarily using tab complete. Maybe will spin up an agent here and there to add tests or for repetitive work where I can say "follow the approach I just did for this one case across all remaining cases"

Right now we’re still in the benchmaxxing phase, I feel like most AI models of combination of models and harness are not really a “product” yet in the sense of having a particular and opinionated UX engineered to be helpful at a specific task, there much more still in a race to be the smartest.

There’s a lot of room for diversification of the actual character of this interaction, for example in the direction you mention. I think once that starts we’ll get into the real “useful” phase, right now we’re really just showing off and exploring how much it can do, even if it’s incidentally really useful in some cases.

Personally I agree with the parent post, I hate it when AI tries to lead me “would you like me to ...” I’d much prefer something awaiting my instructions than wasting tokens tryitn or steer me to doing something inevitably dumb.

I jumped in to perform the same thing again, and it spat out a PNG. And then when I asked it generated a little python script to generate the SVG. The Result was garbage. It took me a while to get the result I had gotten consistently last time, and the results included it hallucinating a few svgs in the process.

I do wonder if we have already passed the best possible models and are over specialising them at this point.

A few model iterations ago, I found its output when explaining what it wanted to do much easier to parse. Nowadays I just feel stupid because it keeps making up names for things and I have no idea what it means.

Earlier today it said something about "the two-terminal solution" and I was just asking it to make a simple Python script to test an HTTP endpoint. I just don't understand what it's talking about any more.

Has anyone else seen this?

I agree with this, but I'd add the caveat you have to do your own digging and researching (as the article says) to identify these tasks well. If you ask it for ideas, it gets painful.

YMMV (and I am very open to getting better at this).

Not talking about prompting Claude "please write me a philosophical dialogue". It has no skin in the game. If it writes 20 paragraphs of slop and is factually wrong and costs other people labor, is there any correction mechanism?

We can correct it explicitly via training I guess, but it's debatable to what extent accountability applies to AI.

We like to pretend that we have our nice epistemology, separated into a priori and a posteroi and all that, but at the end of the day, accountability is what underpins social relations and keeps things going (even if you're libertarian, you have to accept this - even if you don't agree some state should centralize it, it's still just true in terms of basic human social relations).

Of course, another possibility is utter domination - I am so much better than you (for some definition of better) that none of your puny responses matter. OK. Then don't claim both that, and that you're weird for disliking AI and that AI has no problems when integrating with society - too many people have these basic cognitive dissonances.

With enough experience you can just glance at a diff and see if it "smells". If it does, you drill down and see if you were right. If not, you trust the process and approve.

(And I'm talking about basic run of the mill code here, if you're doing something that kills people, don't go by feels)

I don't think that's the case and it's becoming more obvious every day. From people who cannot distinguish truth or fiction on online platforms, good code from bad, to insecure software that causes supply chain attacks, to investment vehicles or fake medicine peddled that does not work, the idiotic MBA manager who does not understand the workers they manage and so on. That is why we're living on BS metrics and crappy reviews, because to the people who manage organizations their own workplace is a Blackbox. The critique that "AI" systems are semantically blind token processors is correct, it's just not bothering anyone because it already applied to most modern companies.

When people become passive consumers rather than broadly competent users how something works cannot be evaluated so by definition all they have to go off is appearance, which is why we live in the age of influencers and marketing, because that is now the only factor that actually matters.

This will mess you up if you ever need to store a significant amount of files on the file system directly, or even deal with resource usage for batch processes, or have to dig into non-trivial transaction handling when dealing with DBs without having learnt enough about them, or even the whole domain of SQL and query optimization that ORMs will not protect you from and so many other things.

I don't think you need to know everything, but you definitely need to learn.

> But surely at some point you need to stop, right?

I guess, but typically due to your own lack of curiosity or mental exhaustion, or even limited time to spend on something, rather than the new knowledge and skills not being good to have. At least knowing when AI is heading in the wrong direction for any given domain will save your bacon.

With AI generated systems, no one knows what's going on. Not even the AI. When an issue pops up (which has a very good chance to happen because of the complexity of integrations), debugging it can be like reverse engineering obfuscated assembly code.

This is very interesting. Which book series is this?

> I read David Nicholas Williams's short blog post titled It's not empowering to hand off the details, in which he argues that AI cannot replace expertise because doing good work requires deep familiarity with the details.

> I couldn't agree more!

> He articulates very well the puzzling phenomenon I see every day with "AI-pilled" colleagues (including many engineers) who trust the model with far more than they should. They delegate the thinking and designing along with the execution, lose touch with the codebase, miss out on critical learnings, and then complain when the slop they produced doesn't solve the original problem.

> But I think the argument is incomplete without a counterexample. There is one phase where handing off work to AI is genuinely empowering, and that is worth defining and formalizing.

If you had the data basis (user reports, trends, ...) you could let the AI decide upon that what to build.

Outsourcing critical thinking is seductive and a slippery slope. LLM lures juniors into feeling productive, and molds them into ineffectual middle managers to be readily replaced when there's a bad quarter.

AI is the new "excel spreadsheet with ridiculous formulas on top of VB script on top of pivot tables" for the non-programmers. Yes, the AI might not do it efficiently or even always give the right answers, but the spreadsheets didn't either. For non-programmers, the computer has always been a black box, data goes in, data comes out. Usually it's right, sometimes it's wrong and the users mostly had no idea why or how. But now they don't need the middleware app. They don't need to take their data and split it up into very specially structured files, and put just the right pieces of information on just the right lines and in the right spaces. They don't need to remember to do "(foo and (bar or baz))" to get the right answer, as opposed to "(foo and bar or baz)" and get the wrong one. AI for non-programmers is the closest we've ever come to "natural language programming" for people to use. It doesn't matter that it gets things wrong sometimes, or does it inefficiently or makes something unmaintainable unless you use the AI again. What matters to them is they can say "take all this data here and swizzle the frobs" in their native language instead of learning a specific programming syntax and system. And they can point it at some arbitrary data in whatever format they happen to have the data in, and they have a halfway decent chance of getting something that does what they want.

It's not going to be pretty, it's not going to be perfect and it's probably going to break in subtle and unexpected ways. But as long as most of the time its mostly good enough, people will still use it enthusiastically. We're effectively looking at "worse is better" for the modern age and for people that aren't and never were going to be programmers.

This frees me up to spend more time at the design level, solving harder or more novel problems.

It's also what we see in the wild. There are so many things going on, so many news about "AI". Are things getting better in any meaningful way? Why are we getting so much from "AI" yet things keep getting worse? The only real, objective and verifiable gain has been the stock price of a handful of companies, most of them heavily into infrastructure and manufacturing hardware.

Seems like a pretty clear pattern. There's so much output, but everything is worse. And the only argument we hear for why this is actually a good thing is just how much more output there is. Hmmm

But useless ceos generate bad output just like useless direct coders.

It's like everyone is adopting the once only-for-the-rich mindset where artisan is actually a derogatory term. Where Micheal Angelo is the same as the cow stall poop shoveler, because they actually do something directly themselves.

But an LLM has to interpret your words and make judgment calls about the logical details, which is a very different thing. An LLM sufficiently intelligent to make acceptable judgment calls in all situations (acceptable enough to ship to production without human review of those details) would likely be indistinguishable from AGI.