There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways.
Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the extra linefeed was steering the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm...)
No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.
tsukikage 6 hours ago [-]
Meanwhile, entirely too frequently I see unit tests that have atol / rtol values which permit errors of one part in 200 or even higher, where people bother with unit tests at all. At some point someone in the pytorch ecosystem used rtol=5e-03 in a unit test and crowds of people started passing that around unquestioned.
With attention matrix sizes being what they are, that's high enough that you can literally zero out a row or two and still have the test suite pass. Guess how I know!
Check your numbers, folks.
shostack 8 hours ago [-]
Debugging any LLM output when you also have done substantial harness engineering is a total pita and I wish there were better tools for it to isolate issues.
I spent ages tracking down start appears to be an issue with the current Deepseek v4 flash 0731 version that would cause it to output giant walls of gibberish in Hermes with reasoning turned on.
catlifeonmars 8 hours ago [-]
This is fascinating. I’m struggling to understand how that was causing such a large difference in the output. Is the “autoparser” vulnerable to injections somehow? How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?
stillpointlab 5 hours ago [-]
My take, as someone who just read through the github issues conversation.
The model was outputting reasoning traces that were supposed to lead to tool calls. So the model might do something like:
<think>
I should use a tool
</think>
... should make the tool call here
But a \n was slipping through from the last line of the reasoning trace so the parser was generating:
<think>
I should use a tool
</think>
And that extra new line before the closing </think> would occasionally trigger the model to question itself with an "Actually ... " digression. In long running conversations this would end up looping because the "Actually ..." part would reason it should call a tool, then a new trailing \n would trigger an "Actually ..." and then it ends up in a loop.
tarruda 2 hours ago [-]
> I’m struggling to understand how that was causing such a large difference in the output.
It is incremental, the more a pattern appears in the context, the more likely it was to continue appearing in future turns.
So the model was likely trained to end a reasoning trace with a single linefeed and a `</think>`. It was also likely trained that two consecutive linefeeds sometimes produce an "Actually..." sequence.
So you can think of it as:
- the first time the reasoning trace was parsed, the two trailing linefeeds followed by </think> were added to the context by the template.
- next time the model was finishing a reasoning block, it added an extra linefeed instead of just closing directly with </think>. This slightly increases the chance that the next token will begin an "actually" sequence instead of closing.
- If it caused an actually, that was added to the context, further increasing the chance of a self correction at the end of the thinking block. The more self correction paragraphs are added, the higher the chance that following turns will have more.
- Eventually it can result in a state where it enters that loop forever (or at least for a very long time).
> Is the “autoparser” vulnerable to injections somehow?
The autoparser was (and still is) incorrectly parsing a trailing linefeed as part of a reasoning block. The were two ways to fix this, both of which must be implemented for the fix to be complete IMO:
- fix the autoparser definition to ensure remove surrounding whitespace is not returned as part of the text blocks
- trim leading/trailing whitespace in the encoding phase, so it fixes bugs or even "injections" where the client deliberately adds the whitespace to trigger problems.
For this specific issue, the maintainer later fixed by trimming the extra linefeed before passing to the template.
> How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?
That is model specific. Ultimately, a token stream is being produced and parsed by the inference engine, and each model uses different tokens/formats. The goal of the autoparser engine was to simplify the creation of parsers for new models by inferring the delimiter tokens from the chat template.
The llama.cpp API server returns pre-parsed data, so clients don't need to do any parsing to know what is a thinking block, a text block or a tool call.
big-chungus4 16 hours ago [-]
I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch
raffraffraff 14 hours ago [-]
Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two.
However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).
If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.
petu 13 hours ago [-]
There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune.
> Qwen3.8-4B is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture.
formerly_proven 12 hours ago [-]
That's not distillation, that's model homeopathy.
picofarad 8 hours ago [-]
Too strong for homeopathy but your point is taken. 0.3% vs 0.000005%
formerly_proven 7 hours ago [-]
4B / 2.4T is 1/600 or 0.16%, almost a 3X dilution. A bit on the weak side for homeopathy, but still readily available for many ingredients :)
embedding-shape 11 hours ago [-]
> Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason
This is almost every ML model, if the task isn't part directly or indirectly of the datasets they use for training it, then the model is gonna be pretty trash at it. What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great.
So, how do you avoid having to spend hours on figuring out if the model is just dumb, or don't know the task? Your own private benchmarks! Figure out a way, ideally without using another LLM, to score how good a model is at doing your specific task. Come up with 3-5 examples for this benchmark yourself, ask a SOTA LLM to fill out 45 more, review everything VERY closely, then use this whenever you want to figure out if $new_model actually is an improvement over what you use today, and once you have a bunch of different tasks, you'll see that all these HUGE improvements tend to be specifically for the benchmarks they mention in the press release, as many of your own benchmarks won't show that big of a difference in reality.
Yes there is a higher upfront cost, but if you're building longer-term projects that rely on LLM models, particularly local ones that seem very benchmaxxed a lot of the times, you need a quick and reproducible way of scoring them somehow, where you can just add more models to compare, and you need to keep these benchmarks to yourself.
catlifeonmars 8 hours ago [-]
> What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great.
Hehe, this kind of sounds like the opposite of generalization. As in it’s just specialization at scale.
meroes 7 hours ago [-]
It is specialization at scale.
It’s paying hundreds of thousands of RLHF’ers from every subject and through some dystopian income stream.
It’s decent money don’t get me wrong, but you aren’t paid at if a task isn’t completed in time for example.
porridgeraisin 10 hours ago [-]
This is the way. It's also very important to automate as much of this verification as possible into the harness, rather than sit there and prod it in the chat.
Of course some things are not auto verifiable, and you'll have to give human judgement and input there, but you'll save a lot more time if you spend 1 week painstakingly writing checks for as many little things as possible and integrating them into the harness.
6wf9j 12 hours ago [-]
I found Gemma 4 models quite good at translating/explaining song lyrics. I actually prefer using 26B-A4B for this task instead of GPT
BoredomIsFun 7 hours ago [-]
Llama 3.2 3B was surprisingly good for general-purpose text manipulation tasks. I'd argue it might be better than many modern tiny models for that.
wccrawford 12 hours ago [-]
That's funny, because I just went through the opposite.
I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions.
I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out. I spent hours on it.
Then I got fed up and asked Claude.
Immediately, it told me that the winget version of llama.cpp is for Vulkan, and I needed a different one and pointed at it. It doubled my speed.
Then it figured out what I was doing wrong with settings.json (wrong spot, global settings can't go in the per-model file, etc etc) and fixed all that, and got it working.
Then it tuned it somewhat.
Then I showed it the official settings pages for both models, and it undid the tuning and all the damage I had done with my tinkering, and got everything working.
In 30 minutes.
It was absolutely amazing.
Every time I see people recommending Qwen locally with llama.cpp, they just say "download it" and act like anyone that can't get it running is an idiot. But if there's a "using this settings.json" tutorial somewhere, I didn't find it, and neither did Google over a week of searching.
But Claude got it done for me.
Now, I admit, I haven't played with it much. Just before all this, I ran out of Claude on the $20 plan and bumped up to $100, and It's been so amazing that it's really hard to work on the local. Especially since it feels like Qwen3.8 35b a3b is probably around the corner, and why mess with 3.6 when 3.8 will probably release soon?
ohyes 7 hours ago [-]
Am I the only one who just downloads directly from LM Studio and just runs the server there? It’s trivial.
stymaar 16 hours ago [-]
> It took him two hours of passing errors to Claude for the endpoint to start working
What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with.
I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).
raffraffraff 15 hours ago [-]
I'm not saying this is the case in the particular example, but these days a lot of people aren't bothering to read a 1 page README, and are instead letting claude "just fucking do it already". I see it at work.
Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.
funlang 15 hours ago [-]
Guilty. Used AI for 2 years, now I feel dumb. Going back to basics this break. No Claude, no ChatGPT.
microtonal 14 hours ago [-]
Add Just [...] And you have also fully adopted the writing patterns.
lelanthran 14 hours ago [-]
Seems like he's self-aware, though - he's taking steps to move away from brain-atrophy.
bretpiatt 9 hours ago [-]
My ability to detect sarcasm is not good. From looking at funlang's profile and other comments the profile looks like a LLM generated bot.
Forums with full no verification pseudonyms seem like they have a real challenge ahead. How long until we need humanhackernews.com with public pseudonyms and a private trusted verification?
enraged_camel 14 hours ago [-]
To me laziness has nothing to do with it. I'd rather focus on my limited time on things that actually matter, e.g. the design of an important product feature or the root cause of a bug and the proper fix. I delegate everything else to Claude. It's really nice to be able to say to Fable "see if you can get this running locally" and then come back 10-30 mins later and read the results.
I've discovered a lot of neat tooling this way that I otherwise would not have bothered tryin to set up because that can take up a lot of time. And even when it's fast, you're suffering from context-switching penalties. You framed it as "read the latest github issues and pull the README" but anyone who has worked in tech for an appreciable amount of time knows that that stuff can turn into a deep rabbit hole.
catlifeonmars 8 hours ago [-]
FWIW laziness is a strength, not a weakness in software engineering :)
LeBit 13 hours ago [-]
In the README there is usually a section where it explains which llama.cpp parameters to configure (sampling, penalties, context size, etc) to get better results.
fxtentacle 15 hours ago [-]
My experience with Claude is that it suffers badly from “not invented here” syndrome. So probably it rebuilt something like llama from scratch and then 2 hours suddenly seems reasonable (if you don’t question the approach). And that’s the thing, someone with no experience isn’t going to question it.
WhyNotHugo 15 hours ago [-]
Not necessarily rebuild llama from scratch, build attempting to build it without cmake and manually invoking all the build commands would be quite in character.
catlifeonmars 8 hours ago [-]
I wonder if this is an artifact of RL, where the training heavily emphasizes codegen. It may be that the model is just better at generating code than reusing libraries, so it prefers the lowest cost approach.
I also wonder if this manifests much less in contexts where the libraries/frameworks are a large part of the training set. It may be that the model doesn’t generalize well so it’s always better to use knowledge in its training set vs attempting to understand how to use a new, potentially never before seen (from the model perspective) api
Schiendelman 15 hours ago [-]
Your mileage may vary. I tried this a couple months ago and spent a full day on it just not working before giving up. Anything I sent, it wouldn't run.
jonplackett 1 days ago [-]
I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
tharkun__ 20 hours ago [-]
It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ...
All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period.
Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance.
Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box!
It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.
aktenlage 16 hours ago [-]
Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.
tharkun__ 6 hours ago [-]
If I use the 12B Unified (dense) model I mentioned without MTP, then I get 37t/s, input ~700t/s.
It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`, which is not ideal. I'm actually right now contending with 35k context but using q8_0 KV quantization. More like 35t/s coz with those settings I can't use MTP.
But I'm not ready to go back to 4t/s. It's not interactive enough for me. That said, I had tried to use the Gemma E4B for example to have it build itself that websearch/fetch skill. It utterly failed, as did previous qwens.
I don't see a Gemma 4 26B A3B GGUF for download, but there is a gemma-4-26B-A4B-it-MXFP4_MOE.gguf that should fit into my overall RAM and then use lots of CPU like the Qwen 3.8. I guess I'll give it a try just to see the difference in speed though I don't expect anything "usable" out of that tbh.
tharkun__ 3 hours ago [-]
Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun.
That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding.
I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking.
EDIT: Oh look, I can now use larger batch sizes again and that gets prompt_eval reads back to 700+
Regarding how good it is: Probably, maybe, better than the 12B? It immediately noticed an error the 12B made, where it hadn't followed a skill's format description. But it's also annoyingly not following instructions the other one was and instead just gives me its opinion lol! And it couldn't figure out how to read a simple pi SKILL.md and run an included script. It complained that it wasn't registered as a tool. But like all other model changes, will get used to it and adjust I suppose :)
b112 15 hours ago [-]
I wish qwen3.8 had a MoE variant, but the skinny is it won't be coming.
big_babol 10 hours ago [-]
I'm having the same constraint, 8GB VRAM, and was trying the past months to get a local model running to drive my Home Assistant setup via voice, so many tool calls, little room for errors. I've tried multiple models, different temperature and top p values, different context windows... turns out that a quantized gemma4 is the best pick so far but qwen3.5:9b isn't far off. Surprisingly newer models are not necessarily performing better.
kQq9oHeAz6wLLS 9 hours ago [-]
I've had great success using Gemma4 even on old hardware (4GB VRAM). I haven't found a Qwen model that can match it on very low end hardware.
velcrovan 19 hours ago [-]
That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.
s1gsegv 17 hours ago [-]
They made a kind of strange decision with Qwen3.8 27B, the template defaults the reasoning_effort to xhigh. I found if you set it to medium it doesn’t just sit there churning forever.
velcrovan 6 hours ago [-]
I had heard of this and actually did set the reasoning to medium ahead of time…
apothegm 7 hours ago [-]
Is there an easy way for a n00b with LMStudio to switch it to medium? Asking for a friend… XD
ekianjo 16 hours ago [-]
xhigh gives better results
dofm 13 hours ago [-]
Not necessarily.
I have seen xhigh go down several rabbit holes, dwell on edge cases and write worse code as a result; it literally distracted itself into writing a complex chain of functions ignoring my prompt, when on “low” reasoning it gets it right on a prompt that requires a few lines of code in the right places.
Simon Willison’s blog has another example (SVG of a circle).
It’s a bit like how giving LLMs access to web search tools can cause them to go down a blind alley based on their first “reasoning” output that then leaves them unable to solve a puzzle correctly that they can fully solve on their own.
freehorse 11 hours ago [-]
With qwen 27b, setting the right reasoning effort for the specific task is important. With xhigh it has a chance at hard problems that bigger models may even fail. But for many everyday tasks, I have found that no reasoning and a system prompt instructing it to be brief is good enough. Note that even with thinking disabled, it may still get into long "chain of thought" reasoning state (out of thinking blocks) if the task is hard and you do not give further instructions, esp with access to tools etc.
chmod775 13 hours ago [-]
Not if it fills up its entire context with "But wait..."
dofm 4 hours ago [-]
xhigh tends not to do that. Uses caveman-ish language. But the reasoning trace does tend to obsess about stuff that it should just ask you about.
cellularmitosis 17 hours ago [-]
We don’t know what quantization level was used for the weights or the kv cache for you or for parent poster, so this is probably an apples to oranges comparison.
spacebacon 17 hours ago [-]
[dead]
noman-land 14 hours ago [-]
I've recently learned and then observed that oMLX serves local models much, much faster than LM Studio.
jonplackett 4 hours ago [-]
I’ve been using the mlx version with orb studio an opencode
jonplackett 13 hours ago [-]
Set its thinking lower. This is a known issue. It still thinks A LOT with lower reasoning levels
phront 10 hours ago [-]
Maybe giving pi more output by setting higher value to maxTokens will resolve his issue
w-ll 18 hours ago [-]
its all still somewhat of a dice roll
16 hours ago [-]
NamlchakKhandro 18 hours ago [-]
[flagged]
riddlemethat 20 hours ago [-]
I got the qwen 3.8 abliterated model running on my MacBook Pro M5 48GB and it's pretty nice having a local model that can do a lot of experimentation without rails.
rahimnathwani 17 hours ago [-]
orcarouter or obliteratus?
prettyblocks 1 days ago [-]
My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?
jonplackett 1 days ago [-]
It’s hot and also LOUD and runs the battery down quick.
But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in.
It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever?
I have an m5 max 64gb and I think anything slower would be quite painful.
coldtea 21 hours ago [-]
>It’s hot and also LOUD and runs the battery down quick.
Who'd run this on battery?
Or do you mean kills the battery faster even when used while charging?
LeifCarrotson 20 hours ago [-]
There was a lovely window of a few years when processors were fast enough and low-power enough that real development work could trivially happen on a Macbook Air in a lounge.
That window is waning as more and more memory and graphics processing power is being used locally.
I suppose the future is most likely going to involve farming out AI requests to your desktop machine, your company's compute farm, or a cloud endpoint, but if you're expecting to run an AI offline on your lap with a battery...your lap will get warm.
LeBit 13 hours ago [-]
Running your dev environment on your laptop is so 2025.
b112 15 hours ago [-]
I envision a future when the potential energy in a laptop battery, could take out 17 city blocks if it exploes.
dmitrygr 18 hours ago [-]
No idea what you are talking about. My battery lasts longer than ever while running vim and make and GCC. It’s amazing. Not sure why your windows are closing.
pmontra 18 hours ago [-]
Because the local LLM, which you are not running, is running for much longer than gcc and is eating the battery. Different choices, different outcomes.
alibarber 15 hours ago [-]
Bur the implication here in the parent comment is that for ‘real dev work’ to be done you must be running an LLM, and therefore you can’t do it on a MacBook anymore.
Turns out you can still accomplish stuff with a text editor and compiler.
pmontra 9 hours ago [-]
Of course we can code without a LLM, but if we want to, as the sibling comment says, don't run a local LLM on a laptop. I'm offloading the LLM to the LLM vendor, like most of us IMO. The CPU is still busier then it used to be, because of the local harness.
adastra22 17 hours ago [-]
So don’t run a local LLM?
adrianN 20 hours ago [-]
Running the laptop hot ages the battery even when you’re plugged in.
chmod775 13 hours ago [-]
Merely existing noticeably ages most battery chemistries.
You can take half an hour out of every week inconveniencing yourself to protect your battery, or you can spend an hour once a year just putting in a new one (or having it done for you in-store if you're using an Apple device) and save yourself the nagging worry.
jonplackett 13 hours ago [-]
I was trying to do some work on the train with no signal.
coldtea 9 hours ago [-]
Couldn't you just enjoy the ride?
seanmcdirmid 1 days ago [-]
I have an m3 max 64GB and it runs ok, I was playing around with it last night. Its reasoning is better, but the token/s still makes an MoE model (35B A3B) preferable so I switched back.
Try using Goose instead of Claude's harness? Goose doesn't load as much context in at the start, so it might be more usable. I've definitely been able to get it past 128K, although I typically don't go beyond 70-80k. MoE might also be better at this.
mrob 14 hours ago [-]
If you're running it while idle and don't need the quickest results, reducing the clock speed improves energy efficiency (and in your case avoids overheating the battery). There will be some optimal speed that maximizes computations per joule that depends on the specific load and can only be found by measurement.
On Linux, you can cap CPU frequencies with "cpupower". Does MacOS have any equivalent?
jonplackett 13 hours ago [-]
The Mac unfortunately just has two performance modes ‘all out power and melting’ or ‘cold and really really slow’
dyauspitr 19 hours ago [-]
I remember doing this with crypto mining way back in the day. That machine will die within a year.
akg_67 22 hours ago [-]
Macs fan control set to full blast and lifting the Mac above table for air circulation across bottom.
I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.
Footprint0521 22 hours ago [-]
Just curious, what is your ram and how many tokens per second do you get with that?
akg_67 21 hours ago [-]
RAM 32GB, tokens are less than 20, may be about 12-15. Main issue is context window, I have set to 64k but practically beyond 32k, too much swap and slow down.
pram 21 hours ago [-]
I am getting like ~25 tps with M5 Max 64GB. Unsloth Desktop and Qwen 3.8 27B Q6 KXL
Forgeties79 21 hours ago [-]
Yikes isn’t 95 well above the danger zone for most components or have things changed? I remember crypto mining in the early 2010’s and making sure to stay under 85C on my GPU.
WD-42 18 hours ago [-]
Stuff runs so hot these days. I think one of the parameters pushed to keep Moores law from completely flattening was heat. My desktop regularly hits 95 just compiling code.
pantulis 12 hours ago [-]
I guess thermal throttling will inevitably kick in.
walrus01 21 hours ago [-]
If you're going to try to use a mac laptop for long term inference you'll probably want to buy one of those $30 laptop fan cooling pads (sold for absurd sized 17 inch x86-64 gaming laptops) on Amazon. Otherwise you'll roast the battery over time and significantly hurt its cycle life and capacity. If you can get significant airflow under the laptop that'll help a lot.
lukan 1 days ago [-]
I don't have the hardware but a often mentioned advice is to put your mac into energy saving mode - it still will work, a bit slower, but stays cool.
PalmPilotProMax 10 hours ago [-]
Imagine spending all that money on Apple hardware only to throttle it to a fraction of its performance lol
Steve Jobs would be proud. People really are holding their Apple hardware wrong.
nicce 12 hours ago [-]
Having a server in the basement helps a lot :-D Then tailscale from everywhere.
1 days ago [-]
ls612 23 hours ago [-]
I tested Gemma 4 a couple months ago and yes it ramps the fans to max. Definitely a thing that should only be done on wall power.
downrightmike 1 days ago [-]
Mineral oil bath?
datadrivenangel 22 hours ago [-]
just decent air cooling and you'll be okay. it will get up to 75/80C though for my M5 MBP.
1 days ago [-]
rurban 18 hours ago [-]
We were trying running a local gpt-oss 80GB model on a H100, and honestly I was surprised how dumb it was.
anon373839 18 hours ago [-]
GPT-OSS 20B didn’t really merit the fanfare even when it was released; it’s definitely not competitive now. Even the 120B version has been well eclipsed by smaller LLMs at this point. The last version of Qwen 27B/35B was better, and now the new one is even better than that!
vikramkr 8 hours ago [-]
Was there a more recent refresh or is this the model from a year ago? The frontier models were barely functional and almost useless a year ago (gpt oss was pre opus 4.5!) - I would be very surprised if the original drop is anything more than totally obsolete/irrelevant at this point
rurban 7 hours ago [-]
Yes, the old entirely stupid old gpt-oss. But Sonnet and GPT were very useful then already, qwen also.
nick_ 18 hours ago [-]
gpt-oss is about a year older than qwen 3.8 27b
hosteur 13 hours ago [-]
How much RAM? And what do you use it for if I might ask?
alexchantavy 1 days ago [-]
How many tok/s are you getting? What gen mbp?
mattdw 1 days ago [-]
My M4 Pro 48GB gets about 13tok/s, in both 3.6 and 3.8 27b Qwens. Qwen A3B and Gemma get closer to 100tok/s from memory but the results are pretty poor for coding tasks.
Edited to add: for agentic workflow I’m running omlx which tells me it has about a 90% cache hit rate (tradeoff is some disk and mem space) - that noticeably changes the felt speed.
MrScruff 1 days ago [-]
I get around 20 tok/s, 4 bit quant, MTP, 4 bit KV cache quantisation. On an M4 Pro 48Gb.
dominotw 1 days ago [-]
i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two.
mistersquid 21 hours ago [-]
Seems threads about local LLMs on Apple hardware feature comments listing M3/4/5 at 48GB 64GB and not 128GB.
That is, users with M-series hardware that have less-than-max RAM share results whereas users with max RAM do not.
Speculating (not extrapolating), maybe users with machine that have max RAM are less interested in running local LLMs and are less averse to paying services for compute?
Personally, I’d love to see what output max RAM M-series Apple hardware in these threads.
asats 20 hours ago [-]
Qwen3.8-27B runs at 59.5 tok/s on my M4 Max, 40-core GPU, 128 GB
I use it occasionally for classification and other tasks but I wouldn't trust those smaller models with the real work and for larger data processing it's too slow, e.g. a dataset I wanted to classify would've taken 56 days on my laptop vs just paying the cheap Luna prices to openai and getting it done in a few hours.
try-working 20 hours ago [-]
Not sure I would trust Luna with that. Deepseek Pro Max and Code Mode I would be more inclined to trust.
digikata 13 hours ago [-]
I've been using Qwen3.6-37B-A3B on an M1 Max w/ llama.cpp and for my practical uses I prefer it to qwen3.8. When 3.8 does answer its slower and, qualitatively, marginally better than qwen3.6, but 3.8 often ends up in unresolved thought loops and runs slower. The Moe 3.6 on my setup is much faster, 500t/s peaks, 30t/s typical, vs 3.8 150 peak, 4-9 t/s typical.
While I've spend a little time tuning, I'm assuming there will be deeper tuning for 3.8 that might close the gap.
FireCrack 22 hours ago [-]
I feel like it's 50/50 between people doing that, and people that have spent a lot of time tuning a system they are pointing at focused and well specified problems.
petcat 24 hours ago [-]
Yeah, that's my experience. It's a big "wow" factor to get a non-trivial LLM running on my Mac, but it's actually not that useful. Like trying to use Photoshop at 8 FPS.
LeBit 13 hours ago [-]
It’s not because you didn’t find use cases for local LLMs that there are none.
I use local LLMs on my Mac Mini M4 Pro with 48G to review text messages tone, act as a text correction tool, act as a code review tool, to do code agent work, generate code snippets, etc
Gemma 4 26B A4B gives me steady 20 tps.
coldtea 21 hours ago [-]
Regarding this analogy, fps don't matter as much for Photoshop, since it's not an immediate mode GUI. 8 fps would be quite ok for comfortably getting feedback on live image filters and such.
a11r 22 hours ago [-]
M3 Pro 36GB. I am getting 17 tps with MTPLX.
StarlaAtNight 1 days ago [-]
how quick does it respond? what are specs of your laptop?
Gareth321 1 days ago [-]
I tried it on my M1 MacBook Pro. It's slow but surprisingly smart as a general purpose LLM. Maybe GPT-5.3 level. I gave it a bunch of tools and it can search the internet, make product recommendations, document, code, etc.
alexpotato 1 days ago [-]
Had the same reaction so had Grok create a script to:
- find a free GPU droplet on digital ocean
- fire it up
- pull in a snapshot of the model + extra files/packages etc
- set up a ssh tunnel so that the localhost:8000 routes to the above
Then I just configured OpenCode to use the above and was off to the races.
Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)
embedding-shape 1 days ago [-]
> Works out to be about ~$2/hr all said
What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.
alexpotato 9 hours ago [-]
It's actually a bit less than $2 but I rounded up just to make the math easier.
And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.
embedding-shape 9 hours ago [-]
> It's actually a bit less than $2 but I rounded up just to make the math easier.
That's cool, what actual GPU though? I'm still curious :P
alexpotato 2 hours ago [-]
gpu-6000adax1-48gb RTX 6000 Ada GPU Droplet - 1X
b112 1 days ago [-]
Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model.
I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure.
(persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)
alexpotato 9 hours ago [-]
> the only difference is I setup a persistent drive,
Going to try this out vs the snapshot!
I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.
chorlton2080 1 days ago [-]
Does it need to respond fast? For important applications, I'm sure we'd all be fine waiting 20 minutes for a high quality, usable answer. Or is it the need for interative refinements that make speed relevant?
jonplackett 1 days ago [-]
It requires patience but it’s more like waiting 5 mins for it to do tasks. You need to be much more involved though and do things slower than Claude where you can trust it to do a lot of tasks at once. It doesn’t have the context for that
dominotw 1 days ago [-]
if you are so sure about what the final shape of your output is then its prbly not a common use of ai
bmitc 9 hours ago [-]
Which exact model are you running? With only 48GB of RAM, by the time I got a model small enough, it was pretty bad in performance both in speed and reasoning.
applicative 1 days ago [-]
Did you read even the title?
system2 1 days ago [-]
Reread what he said maybe?
a11r 22 hours ago [-]
Even a 4-bit quant of Qwen3.8 27b is indistinguishable from Gemini 3.7 flash in our internal tests. With an RTX5090 card and ninfer, you can get ~800 TPS token generation (c=8) and ~140 Tokens per second single stream.
walrus01 22 hours ago [-]
Much of this is why I stick to the rule of:
a) Don't quantize your KV cache
b) Don't run quantizations of the LLM that are worse than the best available Q8 (the largest possible file size unsloth GGUF for a given model like qwen 3.8 27B as an example). I would rather things go slowly but I have confidence that it's doing things more accurately.
utopiah 16 hours ago [-]
Comments are mostly showing off M5s and 5090s without addressing the article.
ThouYS 11 hours ago [-]
It is good that someone is having such a deep look. This is not exclusive to LLMs in the least. Every non-trivial program depends on hundreds of little details being correct.
That is also why I recommend including health checks in such programs. Some function that checks that all assumptions hold. That could be an endpoint, an automated test, a periodic diagnostic job, etc...
InvertedRhodium 24 hours ago [-]
I’m running Qwen3.8 aggressive uncensored Q4_K_P on a 4090 in a loop against the 2026 CrackMe CTF challenges.
Using oh-my-pi in a prebuilt environment that I let Qwen build too.
Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.
treesknees 18 hours ago [-]
This is similar to my use case as well. I’ll use a cloud model to identify security issues in a codebase and then use a report/spec output as input to the local model to build tests or verify the issue is real.
CamperBob2 23 hours ago [-]
How's it performing on the challenges?
InvertedRhodium 23 hours ago [-]
I only kicked this off last night before bed, so I've just got up to see the result of the first task.
So the puzzle apparently admits multiple accepted inputs. The agent found a valid password by reverse-engineering the program, but did not recover the canonical secret from the answer key.
huseyinkeles 22 hours ago [-]
I don't know why but your post was marked as [dead] for some reason. Just vouched for it.
InvertedRhodium 22 hours ago [-]
Thanks! Might have been the multiple attempts at getting it to format nicely in a short amount of time.
mlvljr 23 hours ago [-]
[dead]
nullpoint420 21 hours ago [-]
At least I'd be in control of model quality vs. when Anthropic decides to randomly drop the quality of their offering
runeks 11 hours ago [-]
Somewhat off topic, but I've started wondering if we can actually make local LLMs feel smarter than the frontier closed models, by post-training it for your specific use case.
Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that specific code base while using the tickets to teach the model about past bugs and features. Not sure exactly how, but it could involve doing reinforcement learning solving a past bug on that historic version of the code base, and rewarding the model if it comes up with the correct solution (as defined by the linked PR which fixed the bug).
So essentially post-training your local open-weight model using reinforcement Learning with Verifiable Rewards (RLVR) on your software products history of bug reports and their ultimate solution. And the same for new features.
Zylokloto 10 hours ago [-]
You could do this for sure but its quite a lot of effort and a normal software stack is quite well known due to the massive amount of github projects and other opensource code.
You can also get a lot out of a harness in this case or your Agents.md or Claude.md file by just enhancing the context.
You might even question yourself if you are doing something wrong if a modern LLM really struggles with your code.
We do the finetuning only on small semantic data were it helps a lot.
I'm still wondering when we see smaller models (faster and cheaper) for more specific stacks like spring boot + java + angular + english only or so. Interstingly enough, i assumed LLMs are really good in any language but it seems that non english languages do reduce the ooutput quality of an LLM. At least last years GTC there was a talk about it.
There are companies though which ahve this exact problem with programming languages you normally don't see. ABAP for example is a very well known SAP language.
runeks 6 hours ago [-]
> We do the finetuning only on small semantic data were it helps a lot.
This sounds interesting. Would you care to expand a bit on how you do this? What is this semantic data?
> There are companies though which ahve this exact problem with programming languages you normally don't see.
This applies to my company. We have a substantial code base in our own dialect of APL. You think post-training would help substantially here?
Zylokloto 3 hours ago [-]
So we have free text paragraphs which describe machines. Our model detects if the machine in question is of the main category we are looking for, then our finetuned model extracts from the free form text semantic information about the machine.
Like color, features, horsepower etc.
This would normally take quite a long time to do manually but we already had the semantic version of these texts because the company was doing this for a while.
We now use gemma or Qwen (we regularly re-finetune the newest models to just see if they get better and they actually do) and then use these finetuned models to save us a lot of time.
----
If I had a coding language which isn't available much online and coding LLMs are bad on it, I would definitly try to finetune this but its defintily a lot more work than just doing what I explained above.
Depending on what your usage of this APL Dialect is, it might be easier to fine tune it to migrate from your APL to something a lot more common.
If this is not an option at all:
You need to start creating data for the finetuning. You need a few hundred up to a few thousand of them in different formats like Q & A pairs. Documentation, syntax, a lot of diverse small examples.
The intersting thing about this data generation: You can either leverage, to a certain degree , what you already have, or you start collecting them through your team/work collegues or you do them by hand.
You can put the text into RAG and experiment with context engineering until a LLM is 'good enough' in it to be able to help you generating examples for and with you.
Like you give an LLM all the relevant context for it, then you let it generate pairs:
{
"instruction": "Write an expression to find the maximum along the rows of a 2D array.",
"input": "Array matrix: A",
"dialect_notes": "Custom dialect uses ⌆ (max-reduce) and ⌥ for axis specification instead of /[1].",
"output": "⌆ ⌥2 A"
}
(I have no clue about APL this is just a random example I asked an LLM to generate).
You might have luck and finding communities with the same issue you face.
nonethewiser 10 hours ago [-]
I wonder the same thing. Code is pretty open ended though so I wonder if it’s not the best example.
On the one hand it would definitely be useful for something like classifying support requests into priority. But would it be worth it to just train your own model? I guess one advantage is you could give it some well informed guidelines without training something on lots of data.
There is probably a better example between discrete labeling and code though.
rovr138 10 hours ago [-]
Instead, I'd build tooling for the model to be able to query the tickets, pr, commits, diff, etc
This is something I can reuse better.
runeks 6 hours ago [-]
I'd like to have a model that just "knows" our code base and past issues, instead of having to query them for the same reason I don't want the model to have to query a dictionary to speak proper English.
I could be mistaken here, but I'm curious to see if it would improve both output speed and quality. Currently, every new prompt I start it basically spends 10-15 minutes "getting familiar with the code" which is both annoying to wait for and wasteful money-wise.
heywoods 15 hours ago [-]
So to what extent does this apply to cloud hosted LLM’s? Are there benchmarks that score models across cloud providers? My experience using LLM’s during day time vs evening sessions has felt “night and day” and I’ve chalked that mostly up to it must be my imagination or just the general indeterministic nature of LLM’s. Sessions resumed after a day away also feel “dumb” sometimes so I can see an aggressive kv cache eviction policy playing a role if it’s reasonable to extrapolate what the article is saying about local inference.
Are things like kv cache eviction policies and memory budgets shipped with recommended configurations based on the hardware and software serving the inference requests? or are they configured dynamically by the cloud provider hosting the model to manage multi-tenant load?
catlifeonmars 1 days ago [-]
> I will make you read the really long unpleasant version with math.
This is the version I want to read :)
I assume it is unpleasant in spite of the math, not because of it?
a1o 24 hours ago [-]
I thought it was a link too because of the line under the with math but it isn’t. :/
JacobJack 1 days ago [-]
> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts.
Genuine question : is there something fundamentally wrong with Ollama ?
I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).
I thought the main advantage of VLLM was better concurrency management (better batching).
But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.
embedding-shape 1 days ago [-]
People who use Ollama generally (not everyone obviously) don't always clearly understand what quantization they use when running models, so people end up saying "I tried running Qwen 3.8 27b locally and it was dumb" while Ollama would default to a Q4 version of the model, which has very different results from the BF16 weights, doesn't really speak to the model itself because it's been so quantized in that case.
Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.
For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.
kangalioo 1 days ago [-]
From what I've heard, Ollama has a bad reputation because it's a thin wrapper around llama.cpp without attributing it properly, thereby stealing recognition from the maintainers doing most of the work
b112 1 days ago [-]
It seems, and that seems is entirely my unvalidated impression, that Ollama lags in features, as they're integrating after the fact those changes. But (seriously) an LLM told me that, when some aspects of MoE models were better supported with the latest llama.
And it did in that case make a significant difference.
smcleod 24 hours ago [-]
It's very far behind llama.cpp, vLLM and SGLang in features yes. In part because of that but also due to some poor default settings it generally performs a lot worse as well.
zepearl 24 hours ago [-]
I know nothing about "vLLM" -> anything better compared to "llama.cpp"?
I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.
Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).
I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.
cube00 1 days ago [-]
> is there something fundamentally wrong with Ollama ?
1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB VRAM: https://docs.ollama.com/context-length
2) Registry mistrust due to Deepseek R1 naming. What model do you download/start with `ollama run deepseek-r1`? Not Deepseek R1, but this "for research purposes" thinking finetune of Llama 3 released alongside R1 paper: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama... . For actual R1 you have to pull `deepseek-r1:671b`
3) Can't reuse existing ggufs you have. If you point it to gguf, it would create a copy of it in ollama folder.
4) Doesn't accept engine parameters from CLI args. Only env variables or modelfile. But some things are only in env variables (e.g. KV cache quantization). And even then not even close to what's available in llama.cpp.
5) Often meaningfully slower than llama.cpp
6) VC-funded wrapper for llama.cpp. For long time had questionable attribution to it. Now looks good.
> (and also because VLLM is not super Windows friendly)
llama.cpp is Windows friendly and fairly easy. Not GUI app with installer, but easy. Zip with binaries are on github, run
will pull the model you expect from hugging face (or -m for manually downloaded .gguf) and start web server with API and HTML chat app.
Foobar8568 10 hours ago [-]
7) can't use reranker with it, asked by people for one year or two...
8) ...
JacobJack 2 hours ago [-]
Thank you for your responses. They've been very helpful.
I'm going to test the llama.cpp binaries directly. That seems to me to be the closest to what Ollama is trying to achieve, after all.
VCFundedGenYer 24 hours ago [-]
In my limited testing of ollama (in a Docker container because I don't want Meta touching bare metal disk), I noticed that first of all the model is abandoned. Last updated in 2023. Secondly, it basically can't do anything. Anything I asked of it resulted in a lot of "I can't do that" so it's just stupid.
Macha 23 hours ago [-]
Are you confusing llama (the model from meta that popularised a lot of open AI stuff but is quite dated now) and ollama (the set of tools for running models, originally llama but no longer limited to it, not maintained by meta)
rasitakyol 14 hours ago [-]
[flagged]
mkhalil 16 hours ago [-]
"Why LLMs ARE dumber than they appear" is much closer to the reality I live in.
throwdbaaway 21 hours ago [-]
> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ...
If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.
> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)
But this failure mode can still happen.
Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.
So, perhaps the lesson here is "don't use vllm at home"?
I have been using open source LLMs locally for the past 5 months like Qwen, Llama, DeepSeek and others, and I have also noticed that current local models feel significantly dumber than closed source commercial models like ChatGPT, Claude, and Gemini. One main reason I think is the amount, variety, and quality of original authentic data on which they are being trained on, and also the training method plays a significant role in the performance difference between local open source models and closed source commercial models.
RevEng 2 hours ago [-]
They are also an order of magnitude difference in number of parameters and that matters a lot.
fenestella 21 hours ago [-]
The section on system prompts and context window management is spot on; most people don't realize how much the default quantization in popular runners degrades logic compared to full FP16. I'd be curious to see if the author has benchmarked the impact of KV cache compression on longer context reasoning, as that usually seems to be where my local Llama 3 setup starts to fall apart.
Abishek_Muthian 12 hours ago [-]
After 3 years of running local models I think the model which fits unquantized (BF16) in the VRAM is the best model for general purpose tasks; fine-tuned SLMs or utilities based on non language models for solving a specific problem (e.g. TTS,STT,RMBG etc.) have been the best use of local AI for me.
Local models for coding, is just not worth the effort IMO; unless of course you have the hardware to fit it unquantized in your VRAM.
Roark66 11 hours ago [-]
The problem is benchmarking. Not everyone has a 500k token workstream of the model they are setting up for the first time to run it against 10 different config and compare differences.
And if you download benchmarks from the net they are likely poisoned by models being trained on them.
IronWolve 23 hours ago [-]
sglang, 150+ tok/s on a 5090 in ubuntu 26.04 via wsl.
gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090, dspark, medium reasoning, 96k context.
Using opencode and it built a old fashioned arcade vertical shooter with no issues.
Images are ok'ish, just had grok create updated images, and it came out great.
OsamaMustafaa 12 hours ago [-]
I believe whoever lays down the best structure around LLM will take the lead. Proven in Anthropic vs OpenAI.
freepiai 5 hours ago [-]
I think it's maybe because we:
a) Load it up over time with skills and mcp servers and other junk
b) We start to ask it ridiculously complicated tasks because we've normalized the power so we scale our expectations.
arcanemachiner 16 hours ago [-]
Jeez, I thought I could get away with q8_0 KV cache. Guess not.
djoldman 10 hours ago [-]
"I can't wait to run this new sota model locally. I'll just use the quantized version that is certain to be better than [other model I'm running]."
This is fast becoming one of my top old-man-yells-at-clouds pet peeves.
Reported performance metrics are ONLY good for the exact model weights.
Quantizing a model, or changing it in any way, requires new evaluation to know how well it performs.
Quantizing a good model doesn't mean the quantized version is good.
RevEng 2 hours ago [-]
Especially when there are many ways to do the quantization. You can get quite different results from different methods.
mrgaro 18 hours ago [-]
Any DGX Spark users in this thread? What's your favourite model to run on it?
pama 9 hours ago [-]
Without doubt, dsv4-flash-0731. Original weights; needs two connected DGX.
mrgaro 5 hours ago [-]
Thanks, I however have just one DGX.
pama 2 hours ago [-]
No favorite in single spark usage really. Glimmer is interesting because it scales well under load. Qwen3.6 MoE was fast enough on low concurrency. Qwen3.8 works but slower than the DS flash on the two sparks.
happybox2016 15 hours ago [-]
Rate limiting on free LLM APIs is usually where the pain lies. I've seen 5 concurrent reqs hit 20K/day limit in under 2 hours. Does anyone know a free API that still allows some reasonable concurrent requests?
serbuvlad 15 hours ago [-]
Asking for free inference is like asking for free gold in today's economy. :)
dowonseo 8 hours ago [-]
Honestly expected yet another post dunking on local LLMs with some comparisons, got setup advice and benchmarking methodology instead.
redbear2026 14 hours ago [-]
And here i am with a unsloth UD Q4_K_XL quant. Its a good model still.
woadwarrior01 17 hours ago [-]
RTN quantization of weights
10 hours ago [-]
giuscri 11 hours ago [-]
what a beautiful non-slop article!!! (i’m not ironic)
shevy-java 15 hours ago [-]
No. They are dumb.
anotherCodder 1 days ago [-]
most of the time when a local model feels dumb its not the quant, its the chat template. a lot of gguf mints just drop the template from the metadata and the runtime silently falls back to chatml. model still talks fine so nobody notices, it just gets noticeably dumber. got burned by this myself serving qwen, now i grep the gguf for the template tokens before i blame anything else. second place is sampling, people run whatever defaults their ui ships instead of what the vendor recommends and then compare that to benchmark numbers that were run greedy or with the official settings
washadjeffmad 22 hours ago [-]
I've been comparing against TextGen and llama.cpp while I port to LocalAI and have been surprised by what's happening over the API, even with the defaults and jinja. It's been a fair reminder not to eschew familiarizing myself with the repos.
dannyw 21 hours ago [-]
Nothing beats the classic of figuring out something yourself with your brain, but I also like dictating to LLMs a stream of consciousness with what I'm interested in (while forcing it to NOT give any answers or opinions), and getting back file names it suggests I look at and explore.
Modern frontier LLMs can still be used as rubber ducks, and it's a great.
13639366668 9 hours ago [-]
[flagged]
evidaxis 4 hours ago [-]
[flagged]
CodeWithLeo 10 hours ago [-]
[flagged]
futurist_hp 12 hours ago [-]
[flagged]
koba3 23 hours ago [-]
[dead]
luciana1u 19 hours ago [-]
[flagged]
goglidesdev 20 hours ago [-]
[flagged]
deadcatfound 19 hours ago [-]
[dead]
paulyy_y 22 hours ago [-]
[flagged]
jasonjmcghee 19 hours ago [-]
It's like tongue-in-cheek intentional slop though.
The written text is good lol
dannyw 21 hours ago [-]
You've missed a really great human-authored piece then.
nineteen999 21 hours ago [-]
[flagged]
PrinceNaroliya 18 hours ago [-]
[flagged]
NamlchakKhandro 18 hours ago [-]
word salad? again in english?
luciana1u 11 hours ago [-]
The punchline nobody wants to hear: your local model isn't dumber, it's just finally talking to you the way you actually sound.
achierius 11 hours ago [-]
AI account?
Rendered at 22:19:56 GMT+0000 (Coordinated Universal Time) with Vercel.
Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the extra linefeed was steering the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm...)
No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.
With attention matrix sizes being what they are, that's high enough that you can literally zero out a row or two and still have the test suite pass. Guess how I know!
Check your numbers, folks.
I spent ages tracking down start appears to be an issue with the current Deepseek v4 flash 0731 version that would cause it to output giant walls of gibberish in Hermes with reasoning turned on.
The model was outputting reasoning traces that were supposed to lead to tool calls. So the model might do something like:
But a \n was slipping through from the last line of the reasoning trace so the parser was generating: And that extra new line before the closing </think> would occasionally trigger the model to question itself with an "Actually ... " digression. In long running conversations this would end up looping because the "Actually ..." part would reason it should call a tool, then a new trailing \n would trigger an "Actually ..." and then it ends up in a loop.It is incremental, the more a pattern appears in the context, the more likely it was to continue appearing in future turns.
So the model was likely trained to end a reasoning trace with a single linefeed and a `</think>`. It was also likely trained that two consecutive linefeeds sometimes produce an "Actually..." sequence.
So you can think of it as:
- the first time the reasoning trace was parsed, the two trailing linefeeds followed by </think> were added to the context by the template.
- next time the model was finishing a reasoning block, it added an extra linefeed instead of just closing directly with </think>. This slightly increases the chance that the next token will begin an "actually" sequence instead of closing.
- If it caused an actually, that was added to the context, further increasing the chance of a self correction at the end of the thinking block. The more self correction paragraphs are added, the higher the chance that following turns will have more.
- Eventually it can result in a state where it enters that loop forever (or at least for a very long time).
> Is the “autoparser” vulnerable to injections somehow?
The autoparser was (and still is) incorrectly parsing a trailing linefeed as part of a reasoning block. The were two ways to fix this, both of which must be implemented for the fix to be complete IMO:
- fix the autoparser definition to ensure remove surrounding whitespace is not returned as part of the text blocks
- trim leading/trailing whitespace in the encoding phase, so it fixes bugs or even "injections" where the client deliberately adds the whitespace to trigger problems.
For this specific issue, the maintainer later fixed by trimming the extra linefeed before passing to the template.
> How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?
That is model specific. Ultimately, a token stream is being produced and parsed by the inference engine, and each model uses different tokens/formats. The goal of the autoparser engine was to simplify the creation of parsers for new models by inferring the delimiter tokens from the chat template.
The llama.cpp API server returns pre-parsed data, so clients don't need to do any parsing to know what is a thinking block, a text block or a tool call.
However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).
If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.
Also if you have less than 24GB VRAM, then ollama defaults to 4K context. If that "Qwen 3.8" uses thinking, it might be running out of context and forgetting what it was even answering mid-generation. If that's the case, then try increasing context size: https://docs.ollama.com/context-length , but also: https://sleepingrobots.com/dreams/stop-using-ollama/
https://huggingface.co/Qwen/Qwen3-4B/blob/main/README.md
SO, if that isn't official it explains the results I got. They were dreadful.
Probably this:
https://huggingface.co/empero-ai/Qwen3.8-4B-Distill
> Qwen3.8-4B is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture.
This is almost every ML model, if the task isn't part directly or indirectly of the datasets they use for training it, then the model is gonna be pretty trash at it. What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great.
So, how do you avoid having to spend hours on figuring out if the model is just dumb, or don't know the task? Your own private benchmarks! Figure out a way, ideally without using another LLM, to score how good a model is at doing your specific task. Come up with 3-5 examples for this benchmark yourself, ask a SOTA LLM to fill out 45 more, review everything VERY closely, then use this whenever you want to figure out if $new_model actually is an improvement over what you use today, and once you have a bunch of different tasks, you'll see that all these HUGE improvements tend to be specifically for the benchmarks they mention in the press release, as many of your own benchmarks won't show that big of a difference in reality.
Yes there is a higher upfront cost, but if you're building longer-term projects that rely on LLM models, particularly local ones that seem very benchmaxxed a lot of the times, you need a quick and reproducible way of scoring them somehow, where you can just add more models to compare, and you need to keep these benchmarks to yourself.
Hehe, this kind of sounds like the opposite of generalization. As in it’s just specialization at scale.
It’s paying hundreds of thousands of RLHF’ers from every subject and through some dystopian income stream.
It’s decent money don’t get me wrong, but you aren’t paid at if a task isn’t completed in time for example.
Of course some things are not auto verifiable, and you'll have to give human judgement and input there, but you'll save a lot more time if you spend 1 week painstakingly writing checks for as many little things as possible and integrating them into the harness.
I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions.
I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out. I spent hours on it.
Then I got fed up and asked Claude.
Immediately, it told me that the winget version of llama.cpp is for Vulkan, and I needed a different one and pointed at it. It doubled my speed.
Then it figured out what I was doing wrong with settings.json (wrong spot, global settings can't go in the per-model file, etc etc) and fixed all that, and got it working.
Then it tuned it somewhat.
Then I showed it the official settings pages for both models, and it undid the tuning and all the damage I had done with my tinkering, and got everything working.
In 30 minutes.
It was absolutely amazing.
Every time I see people recommending Qwen locally with llama.cpp, they just say "download it" and act like anyone that can't get it running is an idiot. But if there's a "using this settings.json" tutorial somewhere, I didn't find it, and neither did Google over a week of searching.
But Claude got it done for me.
Now, I admit, I haven't played with it much. Just before all this, I ran out of Claude on the $20 plan and bumped up to $100, and It's been so amazing that it's really hard to work on the local. Especially since it feels like Qwen3.8 35b a3b is probably around the corner, and why mess with 3.6 when 3.8 will probably release soon?
What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with.
I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).
Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.
Forums with full no verification pseudonyms seem like they have a real challenge ahead. How long until we need humanhackernews.com with public pseudonyms and a private trusted verification?
I've discovered a lot of neat tooling this way that I otherwise would not have bothered tryin to set up because that can take up a lot of time. And even when it's fast, you're suffering from context-switching penalties. You framed it as "read the latest github issues and pull the README" but anyone who has worked in tech for an appreciable amount of time knows that that stuff can turn into a deep rabbit hole.
I also wonder if this manifests much less in contexts where the libraries/frameworks are a large part of the training set. It may be that the model doesn’t generalize well so it’s always better to use knowledge in its training set vs attempting to understand how to use a new, potentially never before seen (from the model perspective) api
All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period.
Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance.
Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box!
It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.
It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`, which is not ideal. I'm actually right now contending with 35k context but using q8_0 KV quantization. More like 35t/s coz with those settings I can't use MTP.
But I'm not ready to go back to 4t/s. It's not interactive enough for me. That said, I had tried to use the Gemma E4B for example to have it build itself that websearch/fetch skill. It utterly failed, as did previous qwens.
I don't see a Gemma 4 26B A3B GGUF for download, but there is a gemma-4-26B-A4B-it-MXFP4_MOE.gguf that should fit into my overall RAM and then use lots of CPU like the Qwen 3.8. I guess I'll give it a try just to see the difference in speed though I don't expect anything "usable" out of that tbh.
That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding.
I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking.
Regarding how good it is: Probably, maybe, better than the 12B? It immediately noticed an error the 12B made, where it hadn't followed a skill's format description. But it's also annoyingly not following instructions the other one was and instead just gives me its opinion lol! And it couldn't figure out how to read a simple pi SKILL.md and run an included script. It complained that it wasn't registered as a tool. But like all other model changes, will get used to it and adjust I suppose :)I have seen xhigh go down several rabbit holes, dwell on edge cases and write worse code as a result; it literally distracted itself into writing a complex chain of functions ignoring my prompt, when on “low” reasoning it gets it right on a prompt that requires a few lines of code in the right places.
Simon Willison’s blog has another example (SVG of a circle).
It’s a bit like how giving LLMs access to web search tools can cause them to go down a blind alley based on their first “reasoning” output that then leaves them unable to solve a puzzle correctly that they can fully solve on their own.
But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in.
It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever?
I have an m5 max 64gb and I think anything slower would be quite painful.
Who'd run this on battery?
Or do you mean kills the battery faster even when used while charging?
That window is waning as more and more memory and graphics processing power is being used locally.
I suppose the future is most likely going to involve farming out AI requests to your desktop machine, your company's compute farm, or a cloud endpoint, but if you're expecting to run an AI offline on your lap with a battery...your lap will get warm.
Turns out you can still accomplish stuff with a text editor and compiler.
You can take half an hour out of every week inconveniencing yourself to protect your battery, or you can spend an hour once a year just putting in a new one (or having it done for you in-store if you're using an Apple device) and save yourself the nagging worry.
Try using Goose instead of Claude's harness? Goose doesn't load as much context in at the start, so it might be more usable. I've definitely been able to get it past 128K, although I typically don't go beyond 70-80k. MoE might also be better at this.
On Linux, you can cap CPU frequencies with "cpupower". Does MacOS have any equivalent?
I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.
Steve Jobs would be proud. People really are holding their Apple hardware wrong.
Edited to add: for agentic workflow I’m running omlx which tells me it has about a 90% cache hit rate (tradeoff is some disk and mem space) - that noticeably changes the felt speed.
That is, users with M-series hardware that have less-than-max RAM share results whereas users with max RAM do not.
Speculating (not extrapolating), maybe users with machine that have max RAM are less interested in running local LLMs and are less averse to paying services for compute?
Personally, I’d love to see what output max RAM M-series Apple hardware in these threads.
I use it occasionally for classification and other tasks but I wouldn't trust those smaller models with the real work and for larger data processing it's too slow, e.g. a dataset I wanted to classify would've taken 56 days on my laptop vs just paying the cheap Luna prices to openai and getting it done in a few hours.
While I've spend a little time tuning, I'm assuming there will be deeper tuning for 3.8 that might close the gap.
I use local LLMs on my Mac Mini M4 Pro with 48G to review text messages tone, act as a text correction tool, act as a code review tool, to do code agent work, generate code snippets, etc
Gemma 4 26B A4B gives me steady 20 tps.
- find a free GPU droplet on digital ocean
- fire it up
- pull in a snapshot of the model + extra files/packages etc
- set up a ssh tunnel so that the localhost:8000 routes to the above
Then I just configured OpenCode to use the above and was off to the races.
Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)
What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.
And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.
That's cool, what actual GPU though? I'm still curious :P
I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure.
(persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)
Going to try this out vs the snapshot!
I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.
a) Don't quantize your KV cache
b) Don't run quantizations of the LLM that are worse than the best available Q8 (the largest possible file size unsloth GGUF for a given model like qwen 3.8 27B as an example). I would rather things go slowly but I have confidence that it's doing things more accurately.
That is also why I recommend including health checks in such programs. Some function that checks that all assumptions hold. That could be an endpoint, an automated test, a periodic diagnostic job, etc...
Using oh-my-pi in a prebuilt environment that I let Qwen build too.
Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.
Challenge: Wallpaper
https://github.com/crackmesone/ctf-2026-challenges-public/tr...
I'm using Kimi K3 as the evaluator because, again, Codex and co. wouldn't even evaluate the output. Kimi's verdict:The agent reverse-engineered the 912-byte ELF, including the alphabet check, nibble state machine, move gate, and goal state.
It eventually produced:
I independently verified the underlying input against the actual binary: which returns: and exits 0.The wrinkle is that the official answer key is:
So the puzzle apparently admits multiple accepted inputs. The agent found a valid password by reverse-engineering the program, but did not recover the canonical secret from the answer key.Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that specific code base while using the tickets to teach the model about past bugs and features. Not sure exactly how, but it could involve doing reinforcement learning solving a past bug on that historic version of the code base, and rewarding the model if it comes up with the correct solution (as defined by the linked PR which fixed the bug).
So essentially post-training your local open-weight model using reinforcement Learning with Verifiable Rewards (RLVR) on your software products history of bug reports and their ultimate solution. And the same for new features.
You can also get a lot out of a harness in this case or your Agents.md or Claude.md file by just enhancing the context.
You might even question yourself if you are doing something wrong if a modern LLM really struggles with your code.
We do the finetuning only on small semantic data were it helps a lot.
I'm still wondering when we see smaller models (faster and cheaper) for more specific stacks like spring boot + java + angular + english only or so. Interstingly enough, i assumed LLMs are really good in any language but it seems that non english languages do reduce the ooutput quality of an LLM. At least last years GTC there was a talk about it.
There are companies though which ahve this exact problem with programming languages you normally don't see. ABAP for example is a very well known SAP language.
This sounds interesting. Would you care to expand a bit on how you do this? What is this semantic data?
> There are companies though which ahve this exact problem with programming languages you normally don't see.
This applies to my company. We have a substantial code base in our own dialect of APL. You think post-training would help substantially here?
Like color, features, horsepower etc.
This would normally take quite a long time to do manually but we already had the semantic version of these texts because the company was doing this for a while.
We now use gemma or Qwen (we regularly re-finetune the newest models to just see if they get better and they actually do) and then use these finetuned models to save us a lot of time.
----
If I had a coding language which isn't available much online and coding LLMs are bad on it, I would definitly try to finetune this but its defintily a lot more work than just doing what I explained above.
Depending on what your usage of this APL Dialect is, it might be easier to fine tune it to migrate from your APL to something a lot more common.
If this is not an option at all:
You need to start creating data for the finetuning. You need a few hundred up to a few thousand of them in different formats like Q & A pairs. Documentation, syntax, a lot of diverse small examples.
The intersting thing about this data generation: You can either leverage, to a certain degree , what you already have, or you start collecting them through your team/work collegues or you do them by hand.
You can put the text into RAG and experiment with context engineering until a LLM is 'good enough' in it to be able to help you generating examples for and with you.
Like you give an LLM all the relevant context for it, then you let it generate pairs:
{ "instruction": "Write an expression to find the maximum along the rows of a 2D array.", "input": "Array matrix: A", "dialect_notes": "Custom dialect uses ⌆ (max-reduce) and ⌥ for axis specification instead of /[1].", "output": "⌆ ⌥2 A" }
(I have no clue about APL this is just a random example I asked an LLM to generate).
You might have luck and finding communities with the same issue you face.
On the one hand it would definitely be useful for something like classifying support requests into priority. But would it be worth it to just train your own model? I guess one advantage is you could give it some well informed guidelines without training something on lots of data.
There is probably a better example between discrete labeling and code though.
This is something I can reuse better.
I could be mistaken here, but I'm curious to see if it would improve both output speed and quality. Currently, every new prompt I start it basically spends 10-15 minutes "getting familiar with the code" which is both annoying to wait for and wasteful money-wise.
Are things like kv cache eviction policies and memory budgets shipped with recommended configurations based on the hardware and software serving the inference requests? or are they configured dynamically by the cloud provider hosting the model to manage multi-tenant load?
This is the version I want to read :)
I assume it is unpleasant in spite of the math, not because of it?
Genuine question : is there something fundamentally wrong with Ollama ?
I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).
I thought the main advantage of VLLM was better concurrency management (better batching).
But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.
Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.
For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.
And it did in that case make a significant difference.
I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.
Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).
I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.
Yes https://news.ycombinator.com/item?id=47788385
1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB VRAM: https://docs.ollama.com/context-length
2) Registry mistrust due to Deepseek R1 naming. What model do you download/start with `ollama run deepseek-r1`? Not Deepseek R1, but this "for research purposes" thinking finetune of Llama 3 released alongside R1 paper: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama... . For actual R1 you have to pull `deepseek-r1:671b`
3) Can't reuse existing ggufs you have. If you point it to gguf, it would create a copy of it in ollama folder.
4) Doesn't accept engine parameters from CLI args. Only env variables or modelfile. But some things are only in env variables (e.g. KV cache quantization). And even then not even close to what's available in llama.cpp.
5) Often meaningfully slower than llama.cpp
6) VC-funded wrapper for llama.cpp. For long time had questionable attribution to it. Now looks good.
> (and also because VLLM is not super Windows friendly)
llama.cpp is Windows friendly and fairly easy. Not GUI app with installer, but easy. Zip with binaries are on github, run
will pull the model you expect from hugging face (or -m for manually downloaded .gguf) and start web server with API and HTML chat app.If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.
> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)
But this failure mode can still happen.
Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.
So, perhaps the lesson here is "don't use vllm at home"?
Local models for coding, is just not worth the effort IMO; unless of course you have the hardware to fit it unquantized in your VRAM.
And if you download benchmarks from the net they are likely poisoned by models being trained on them.
Using opencode and it built a old fashioned arcade vertical shooter with no issues.
Images are ok'ish, just had grok create updated images, and it came out great.
a) Load it up over time with skills and mcp servers and other junk b) We start to ask it ridiculously complicated tasks because we've normalized the power so we scale our expectations.
This is fast becoming one of my top old-man-yells-at-clouds pet peeves.
Reported performance metrics are ONLY good for the exact model weights.
Quantizing a model, or changing it in any way, requires new evaluation to know how well it performs.
Quantizing a good model doesn't mean the quantized version is good.
Modern frontier LLMs can still be used as rubber ducks, and it's a great.
The written text is good lol