NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
▲Inkling-2-Mini in 5.4 GB Memory (github.com)
Aurornis 1 days ago [-]
If you're new to quantized models: Don't get too excited about claims of fitting large models into tiny RAM footprints from random GitHub repos. There is always a tradeoff, and it's too easy to benchmaxx KL-divergence numbers and even single benchmark scores. Just because you can get something to run on a device doesn't mean it's going to be useful. The headline claim of this repo (12X better) doesn't even apply to Inkling-2-Mini

This repo's README is a mess of different histories as the author bounces from one popular model to the next. It refers to vague things like "the 1.2GB file" that probably made sense in an early vibecoding pass, but are now lost after the repo has been pushed around to different popular models.

The truth is that random GitHub repos with messy READMEs almost never hold a real secret that the mainstream quantizations have missed. I would recommend sticking to mainstream quantizations like the official GGML release, Unsloth's quants, or the more well known community quantizers like bartowski or AesSedai. They typically follow an optimized size-quality tradeoff so you can find the one that fits your needs.

When you see a repo like this where everything is so obviously vibecoded that the README is a rambling mix of projects and claims, you should probably go looking for a more authoritative source. The local LLM communities get spammed with 10 of these types of projects per day but I have yet to see a real breakthrough come from any of the vibecoded ones. The real work is coming from people who study this and have compute to test at scale or work on implementing advanced techniques which require inference engine support, like trellis quants.

ricardobeat 3 hours ago [-]
They also mention a custom quant of Qwen 3.8 27B at 7.2GB.

Ternary-Bonsai-2-27B-PTQ1_0.gguf, which is 7.2GB, is already runnable locally and scores much higher (84% vs their 78% MMLU for example). I wonder if their agents just ended up "rediscovering" that quant since the size matches exactly.

> random GitHub repos [...] almost never hold a real secret that the mainstream quantizations have missed

As an anecdote, In late October I tried joining https://openkernelchallenge.dev, having successfully built an inference engine before using AI workflows, but that reality became quickly obvious. As I expected, it was a struggle to break the top 100 as every idea I had, with my limited knowledge of ML and GPUs, has apparently already been tried and developed much further.

There is value in unlocking new software purely by combining things that already exist in a novel way, but for cutting-edge research, human ingenuity still wins - for now.

eyegor 1 hours ago [-]
FYI the bonsai models are trash at actual coding work once you get beyond 64k context or so. The public benchmark scores are almost worthless when it gets stuck in loops and constantly forgets what it's working on. Inkling 2 mini is much more capable at coding work under a harness, but it is incredibly slow.
mark_l_watson 6 hours ago [-]
You give really good advice. I am retired so I have time to play with many model + harness + inference engine combinations, but for people working and with things to do I would also recommend finding popular well-vetted setups and spend more time working at the application layer.
dominotw 4 hours ago [-]
> am retired so I have time to play with many model + harness + inference engine combinations,

Damn. Now that ski season is starting i see lots of young retirees that are there by 'magic of capitalism' ( or so they say) .

Its hitting me hard this year from my impeding doom and still young kids at home.

nojs 23 hours ago [-]
> it's too easy to benchmaxx KL-divergence numbers

Could you elaborate on this? Intuitively how can the distribution over logits be similar and the results different?

Aurornis 18 hours ago [-]
KL divergence is computed over samples from a text corpus. WikiText-2 is commonly used, which is a sampling of Wikipedia articles.

Quantization processes are somewhat complex and are more involved than uniformly truncating bits. Look up imatrix quants for a sampling of some of the techniques used. The short story is that the quantization is performed in a way that prioritizes weights which matter to a certain corpus. The corpus used as the input to that process therefore biases where the quantization losses go.

It's common to use a corpus like WikiText-2 from Wikipedia articles, but you can imagine a random sampling of wikipedia articles may not be favorable to, for example, programming work. A lot of quantizers keep their corpus as a secret sauce, so you may not even be able to see what they're prioritizing or even how some of them measure KL divergence.

You can get amazing KL divergence numbers if you basically train the quantization on some text corpus and then measure KL divergence on it.

This is also why you can't compare KL divergence numbers from different sources unless you've verified that they're computed in the same way.

There are a lot of quants out there that claim amazing KL divergence numbers but are really bad for programming work, for example. If you want to do continuations of Wikipedia articles they might be great.

liuliu 22 hours ago [-]
Validation dataset. Nobody shares or talks about their validation dataset v.s. their calibration data when doing this kind of quantization / publishing KLD numbers.
electroglyph 21 hours ago [-]
yup, this. calibrating on wikitext and then validating on it is a shady move
nojs 20 hours ago [-]
For an open weight source model the data available for this validation is effectively infinite and trivial to obtain - so this should only happen if the creator is being deliberately misleading.
22 hours ago [-]
cowlevel 1 days ago [-]
Yep. Tried GLM-5.3 1 or 2bit (don't remember), it should be something like 80% as good according to the divergence chart, in reality it's dogshit.
charcircuit 16 hours ago [-]
80% good for a single token. So after thousands of tokens you can diverge a lot.
rosstex 15 hours ago [-]
this comment was a lightbulb moment
adhamsalama 10 hours ago [-]
Same. Never thought of it that way before.
Espressosaurus 23 hours ago [-]
Yeah, q8_0 is where I prefer to live, q4_0 if absolutely necessary. And quantizing the KV cache is for suckers.
enoch2090 19 hours ago [-]
Just played with MAYA with q2 quantization of glm-5.3-flash last night, the model really had a rough time doing a tool call, while q3 sort of works
AdamConwayIE 8 hours ago [-]
Yeah, I use the 4bpw EXL3 quantization of glm-5.3-flash across 2x DGX Sparks. I've run a number of benchmarks comparing it to the official API and it's on par, but I've seen Unsloth's benchmarks for it. I wonder how much EXL3 makes a difference.
alfiedotwtf 14 hours ago [-]
GLM 5.3 Flash doesn’t seem to hold quality as you go down in quants - check out the graphs of Unsloth’s guide - their Q4_K_M vs Q4_XS is miles apart. Compare that to their MiniMax 3.1 Flash where even Q3_XS holds up against the Q4s
teaearlgraycold 22 hours ago [-]
I haven’t had issues with 8 bit KV.
jychang 12 hours ago [-]
You're not looking hard enough then. Modern optimized attention isn't global GQA anymore, you're losing a lot of entropy by throwing out half the bits.

For comparison, 2025 era Qwen3 32b would use 256 KiB/token so 32 GiB of KV cache at 128K context.

Modern Qwen 3.8 27b with GDN layers would use 64 KiB/token so 8 GiB of KV cache at 128K context.

Quantizing the KV cache to 8 bit on top of that is not as lossless as it was 1-2 years ago.

It gets really wild depending on the model. Deepseek 4.1 is at 890 bytes/token. BYTES, not kilobytes. You lose a LOT of information per token if you try to quantize that.

Espressosaurus 5 hours ago [-]
Yeah, my informal testing shows a measurable drop in correctness and ability to even complete the task (not just KLD!) on a sample set I’ve been using for speed optimization using even q8 vs f16. It gets more pronounced and leads to looping behavior in some models as I drop the quantization. I haven’t bothered testing below q4_0. But the loss is real with Qwen 3.8 FN or 27b.
naasking 23 hours ago [-]
I've had few issues with mxfp4 weights and k6_v4 KV cache.
whoanoway 1 days ago [-]
[dead]
Havoc 18 hours ago [-]
Yeah noticed that with the Bonsai models.

Was a respectable effort but a 27GB model cut down to a couple gigs tends to perform about as well as models in that size class not the "27B" it says on the label

Esp on world knowledge.

mrbonner 1 days ago [-]
+1: one of the analysis table has a column named “Text Size” in GBs which I have no idea what it means. I assume it is a carried over artifact lost in long context session with coding agents.
dvaplima 22 hours ago [-]
Thanks for the deep explanation, I really got excited for a moment. Looking at that consumption of ram, what is the best model available?
zacksiri 7 hours ago [-]
I tried inkling-2-mini with high thinking. It's a good model, though it does have tendency to over think but if you instruct it right it gets the job done.

I tested with some tasks i usually assign to the qwen 3.8 27b, for example setup a grafana dashboard for a service i just setup. It did it beautifully.

I'll do some more testing to see how it performs. This thing is doing 150 tok / s on my hardware which is 2x A4500 with NVLink.

I had to do some custom kernel to make it work namely INT8 kv cache. The setup i used is documented here:

https://github.com/zacksiri/llm-lab

an0malous 1 days ago [-]
Our future hinges on the capability gap between frontier lab models and open source local models
singpolyma3 1 days ago [-]
What gap?
rs_rs_rs_rs_rs 1 days ago [-]
The gap is so big you can't see it.
claude-ai 1 days ago [-]
Yeah, this model is ~comparable to Opus 4.6. That was, not too long ago, a frontier model.

I didn't find any non-sloppy factual report, but this channel has a good video on it: https://www.youtube.com/watch?v=OdMpNiByntM

dofm 22 hours ago [-]
I find this claim a bit difficult to believe.

I like it. It is very good indeed for a small MoE, it writes nicely and it is quite unusual in "personality", I suppose, but I find it difficult to believe it is that good, just based on my own interactive experiments.

I don't really trust benchmarks at this point, or videos with "insane" in the title for that matter. But I do not yet believe it is better than Muse Glimmer in a practical sense as a general coding assistant, and from that I would be a little surprised if it really really outperforms Qwen 3.8 27B in the way their claims suggest.

Maybe it is more obvious in long-context work, but I am not sure that most coding agent tasks are like long-context one-shot Godot game prompts where success is a bit more open-ended, where solutions can be more broadly creative.

Most of the time we have specific constraints and a set of particular requirements that we need to reach step-by-step. I am not convinced it is going to be as good there.

I also think the interactive stuff matters, and this model is not as attentive or knowledgeable as Muse Glimmer (or in my less-detailed experience, Qwen 3.8 27B, though it is quite quick by comparison)

TML themselves say: "Inkling 2 Mini is not the strongest small model for multi-turn chat or for instructions with many constraints".

They also say it relies on tool-calling to fill a knowledge gap (which my own minimal tests bear out)

It is enormously promising and it appears to be quite creative; perhaps looping gives it that. But I am wary of seeing what I want to see, when a ~30B dense model is, on average, going to plod through and do better on many tasks because of better basic knowledge.

(Edited multiple times for clarity)

wolttam 21 hours ago [-]
A lot of how “good” a model feels comes down to post-trained behaviour, rather than knowledge capacity. A small model that responds coherently to facts fed to it (e.g. via tools) may punch way above a heavier model that gets by more on its sheer knowledge base.

I can totally believe that this model may even exceed Opus 4.6 in some areas (except knowledge), simply because 4.6 came along early enough in the RLVR pandemonium that it hadn’t yet had certain behaviours instilled into it that help smaller models succeed (such as relentlessly checking and verifying everything it does)

dofm 21 hours ago [-]
Yeah. Post-training is clearly a lot of why I kind of like Muse Glimmer, even if Qwen 3.8 27B might be more capable. Muse Glimmer is a good colleague.

I am just re-running with Inkling 2 Mini a sequence of prompts I used with Muse Glimmer to write a content exporter in PHP (extract pages from a CMS, write into hierarchical folders, convert the HTML to markdown, add YAML front-matter, process any images so they are also extracted and renamed, etc.).

It is generally doing fine. Similar approaches and good tool calling.

(In medium reasoning mode using the hybrid Q8 model rather than the Unsloth Q4)

At some point I will dig in and get it to write something more serious for me, test out its Go understanding, see how it gets along in Pi.

And, yeah — it is doing all of this without any system prompt. Like Muse Glimmer, it does not need telling it is a senior engineer, or any advice about when to call tools, or really any of that.

But I have seen some behaviours that put it behind Muse Glimmer: it really doesn't take correction as well (dug its heels in before finally reaching for a search tool, has actually misunderstood a correction, confused itself by thinking I said something it said, etc.).

I guess as you can see I am trying not to read too much into all this because local LLMs definitely have their own often disappointing hype cycle. But it's undeniable that this is huge progress from this lab in a really short time.

thatsabadlook 10 hours ago [-]
Having used a lot of models along the frontier. Yes even Chinese models. I think a lot of people's perception of good is closer to "this slot machine I paying out today!". I'm not kidding.

I suggest to pretty much everyone to try different providers and models and do their own objective assessments. An awful lot of the differences between the models now is marketing.

theresanagent 1 hours ago [-]
[flagged]
wolvoleo 1 days ago [-]
Exactly. It seems to not really be a permanent gap but more a time gap. Locally you're about a year or so behind SOTA. Is that terrible? Not really, a year ago cloud model users were raving about what they had.

There's some things like knowledge breadth that really need a huge model but those things can be compensated for with web lookups (which is preferable for source validation and currency anyway)

kakugawa 1 days ago [-]
Yeah, a good balance would be a local reasoning model paired w/ a knowledge pack + web-based searches. You want your obscure coding language? Provide a pre-vetted knowledge pack locally that it knows how to use. Same with any other domain-specific knowledge.
Tade0 1 days ago [-]
For that, Qwen 3.8 Flash Next is split into two shards, one with the model, the other with the n-gram table, which functionally holds its world knowledge.

Personally I recommend ISTA-DASLab's version, pruned to half the number of experts to specialise in coding:

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

I didn't think my somewhat dated RX7700S would be supported in Strata, but it is and I'm getting 120tps+ prefill and 20tps+ generation with it, which is already practical for my purposes.

Datagenerator 8 hours ago [-]
This model is very capable and has been archived for preservation [1]

[1] https://archive.org/details/ISTA-DASLab_Qwen3.8-Flash-Next-G...

wolvoleo 23 hours ago [-]
Oh yeah coding isn't my main thing though.

I tend to just use Claude for that because I can get a lot of it for the $20 a month, and because there's nothing privacy-sensitive in coding jobs anyway.

wolvoleo 1 days ago [-]
Yes especially now that local context isn't so costly anymore. With llama3 it would kill my vram just to have a few thousand tokens. With recent models it's hardly an impact anymore. Not sure how they manage that but it's great. It still takes more processing but I also run an older card that is not very good at prefill.
jeffyaw 1 days ago [-]
That’s what I do with typed, there’s a knowledgebase that loads with an embedding model for code related lookups.
InsideOutSanta 1 days ago [-]
> Locally you're about a year or so behind SOTA

If that holds, I'll be able to do 99% of my work with local models around March 2027.

wolvoleo 23 hours ago [-]
Well the thing is, people want the best of the best.

You can probably already do that now if you try. But it won't be Opus 5.5 obviously.

InsideOutSanta 22 hours ago [-]
> Well the thing is, people want the best of the best.

People often say that, but I don't believe it's actually true. There's a point of diminishing returns where, for the same task, you see no real difference between running it on Opus 5.5 and running it on a comparatively shitty model like Longcat 2.5. The number of tasks this applies to is growing almost every week.

In the last roughly five months, I went from using 100% US SOTA models to using 5% US SOTA models and doing the remaining 95% of the work with much cheaper Chinese models.

13415 22 hours ago [-]
That is very much what I'm hoping for.
furyofantares 1 days ago [-]
Would local models catch up if frontier models paused new training for a year?

I'm not sure for. It might just be you can get these nice local models because of distillation from a much more powerful model.

swiftcoder 1 days ago [-]
Adversarial distillation hasn’t been a major factor in the past year. If the SOTA stops moving, all the other labs catch up to them in 6 months to a year, and distillation into smaller models continues to close the gap between small/large models during that time
thereader12 1 hours ago [-]
I think we should avoid spreading that term. We shouldn't let it become legitimized
furyofantares 23 hours ago [-]
Do you know why the other labs are 6 months behind?
swiftcoder 13 hours ago [-]
The big western labs have a decently large first-mover advantage, and are flush with cash/hardware that the Chinese labs have less access to.

But maybe the biggest issue right now is that the big western labs have to stay at the cutting edge (at least up until they IPO), no matter the cost. If the Chinese labs caught up to them before the IPO, their valuation would be liable to go up in smoke.

singpolyma3 23 hours ago [-]
Marketing
sumedh 22 hours ago [-]
You can have your own benchmarks which will show you the real picture.
spwa4 22 hours ago [-]
Every model is now distillation in a way. The US models are now hiring people do, well do their job and make traces of that available to them. This is a desperation move since:

1) this takes away the advantage they AI labs had ("distilling humans" by stealing data, essentially) (or to put it bluntly: distilling humans by paying for data is a lot more expensive)

2) Because this is LLM reacting to humans who are reacting to LLMs this is no longer good data (data should be independent from training according to the math, or you will start to see, imho, what people are complaining about with ChatGPT 6.2 models: this data is no longer resulting in improvement across the board. It's getting better at some things at the cost of getting worse at others. This might even be a fundamental problem, rather than just one the labs are causing for themselves)

3) Is this really the game you want to play against China? Getting more training data through letting people generate it? I don't think US and EU combined can win that game. And ...

https://www.youtube.com/watch?v=JV4BwRxDZi0

smashandgrab 23 hours ago [-]
What’s really neat is you don’t have to worry about random degradations in model performance when SOTAs decide they want the next model to look even better by comparison.
dofm 21 hours ago [-]
And it doesn't ever get "sunsetted".

I really think this whole aspect is under-appreciated.

wolvoleo 23 hours ago [-]
Yes or when they throttle or cut you off because their server usage is too high.
thatsabadlook 21 hours ago [-]
I would say more like 6 months. There are models that fit between this and frontier that are open weight that have myself and many others not using frontier lab subscriptions at all anymore. If it was good enough for people to think the world was ending 6 months ago, it's good enough for me now.
dlisboa 23 hours ago [-]
Yeah, not terrible at all. Let’s say in a year we get an Opus 5.5 level local model. That would be a game changer.
wolvoleo 23 hours ago [-]
Is even Opus 4.6 standard so bad?

I was using Opus 4.8 until a couple months ago because the early Opus 5 models were really terrible at text.

dlisboa 22 hours ago [-]
I don’t think so, it just depends a bit on what you want. I was using Sonnet 5 with good results with a lot of supervision, switching to Opus 5.5 has allowed a significant change to fully agentic workflows. It gets almost everything right almost in a single shot. Some decisions ar just surprisingly good.
pwython 18 hours ago [-]
I remember being blown away by Opus 4.5 while I was mostly using Sonnet 4.5 happily. Opus was only for special occasions.
consumer451 22 hours ago [-]
If there is no gap, then why aren't Cursor and all other coding harness defaulting to the open weight models?
agentmaxxing 1 days ago [-]
[flagged]
zulux 1 days ago [-]
Weirdly, I use lab models to make the local models useful. Nothing like getting Claude to spin up the system that will put it out of work.
wolvoleo 1 days ago [-]
Yes me too lol. Most of my Claude work is about fine-tuning my local setup. Especially now as I'm rearranging it all for strata.

My goal is not to be 100% local but to have all my personal data on local models only. Claude I just use to build stuff so it doesn't contain much data about me.

ivewonyoung 24 hours ago [-]
What tools are you converging on for local use?
wolvoleo 23 hours ago [-]
llama.cpp, litellm, openwebui, whisper (via homeassistant) and strata mainly.

Looking to replace whisper with something better though. Especially at the end of a recording it tends to hallucinate weirdly.

leobg 9 hours ago [-]
Have you tried nemotron?
idle_zealot 24 hours ago [-]
> put it out of work

What a cynical way to look at things. I prefer to think of it as a mentoring relationship, sharing and growing knowledge, like a parent would a child or a teacher a student. This relationship is core to our culture and is how we promote progress while retaining capability. In this case the generational efficiency gain takes the form of going from giving Anthropic $200 a month to paying $5 more on my utility bill. Now that's progress ;)

dotancohen 1 days ago [-]
To be fair, the models putting us out of work were trained on our own output as well.
syntaxing 1 days ago [-]
I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel.
Aurornis 1 days ago [-]
It's less than 24 hours old. The inference engines haven't been optimized for it yet.
__rito__ 16 hours ago [-]
Ollama claims to support it, but haven’t played with it yet.
meatmanek 15 hours ago [-]
On my M1 Max MBP when I was testing yesterday, Ollama had the same speed as the Unsloth llama.cpp build, around 18 tokens per second during token generation. Based on 5.4B active parameters at 4 or 8 bits/parameter and 400GB/s memory bandwidth, I'd expect 74 to 148 tokens/second.

The 18 tokens per second number smells suspiciously like the model is being processed as if it were a dense model somehow: 400 GB/s / 22 GB/token ~= 18 tokens/second (the model is ~22GB on disk at Q8).

monster_truck 15 hours ago [-]
Support and actually optimized performance are often weeks apart.
flatline 1 days ago [-]
I’m getting ~6.7 tok/s out of it on an a laptop RTX 4060 with 8GB VRAM — over twice the speed I can get out of qwen, and arguably fast enough to turn it loose on real work. For a minimal hardware setup, this is easily the best speed for performance I’ve found in a model. The nemotron models are the next best, they are much faster, much more verbose, and much less accurate.
merbanan 22 hours ago [-]
40+ TG and 1000+ PP on 8GB VRAM RTX2060 Super on an IQ3_XS model. The support and speed should be there in a couple of days.

I have not tested the sanity of the model though. But I think a carefully quantized version of the model will give good results but most likely that is a couple of weeks away.

krautsourced 1 days ago [-]
"arguably" as in I'd argue against it...
dotancohen 23 hours ago [-]
What are you running it under? llama.cpp directly?
flatline 23 hours ago [-]
Compiled llama.cpp using the NVIDIA toolkit. Running with `-ub 16` or lower was the main speedup. I’d expect further optimizations over the next few weeks.
dotancohen 23 hours ago [-]
Thanks. If you do manage to squeeze some more out of it I'd love to hear. I'll probably start toying around with it Wednesday or Thursday.
flatline 15 hours ago [-]
[flagged]
6_7 1 days ago [-]
I'm so sorry you're getting how many tok/s?
flatline 1 days ago [-]
Last gen hardware before local AI was even a consideration. Things have actually gotten measurably better.
zargon 22 hours ago [-]
Check the grandparent's username. (And comment history.)
ac29 20 hours ago [-]
I get 30-45 t/s decode on the full weights on strix halo with optimized kernels and speculative decoding. Far faster than Qwen27B.

(numbers at low depth, it obviously falls with depth)

vineyardmike 1 days ago [-]
According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster.
syntaxing 1 days ago [-]
This has been observed by the broad community for llama cpp.
nicce 1 days ago [-]
llama.cpp is one of the slowest these days
saghm 1 days ago [-]
What alternatives do people use instead? I hadn't heard of any others (except I guess ollama, which I thought no one used anymore)
montroser 1 days ago [-]
Depends on your hardware, but this build of [ninfer][1] runs inkling-2-mini for me at 77 tok/s versus llama.cpp at 30 tok/s. So far it has been pretty good. Solved some things that qwen3.8-27b could not.

[1]: https://github.com/iamwavecut/ninfer-all

saghm 1 days ago [-]
Ah, I have an AMD card, so doesn't look like that's viable for me
dcan 22 hours ago [-]
You’ll want Radiance with an AMD card: https://codeberg.org/StillDeadcode/radiance

(Though, it hasn’t been updated for Inkling-2-Mini yet, there is an open merge request)

saghm 18 hours ago [-]
I'd definitely try it for Qwen if I could but my card is a generation too old (6900 XT, so RDNA2). It's honestly kind of wild that a five year old card can even be useable at all for this sort of thing but I've gotten a quantized Qwen 2.8 27B fitting in the 16 GB with okayish results (40 tokens per second with 64K context isn't completely terrible given that it's only the cost of electricity and not playing games at the same time). I'm still hoping that Qwen 4 might be the model to finally make it good enough that it would be comparable to what I can get from the extremely small free daily limit of opencode zen, but we'll see when the weights come out.

As an aside, for the gaming side of things, this card will potentially get even more usable next year since AMD has said they're going to add support for FSR4 to that generation, which is also pretty crazy.

Rzor 12 hours ago [-]
>As an aside, for the gaming side of things, this card will potentially get even more usable next year since AMD has said they're going to add support for FSR4 to that generation, which is also pretty crazy.

Merry Christmas then, because it arrived earlier than expected:

>Last Updated: October 20th, 2026.

>Highlights

>New Features

>AMD FSR Upscaling 4.1 support for:

>AMD Radeon™ RX 6000 Series Graphics products

https://www.amd.com/en/resources/support-articles/release-no...

msdz 1 days ago [-]
Ollama is just worse (well, depends on your use cases) pre-configured llama.cpp, I believe. I barely know anything about local hosting, but you might wanna look into vLLM (cross-architecture) or MLX (for Apple Silicon).

Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4.

ac29 24 hours ago [-]
Hardware/model specific inference engines tend to be fastest. Fortunately Opus/Sol/Neon are all pretty good at that, so I've just been making my own engines when I find something I want to play around with.
Tade0 1 days ago [-]
Strata, but it doesn't support Inkling's architecture yet:

https://github.com/niko1221/strata

akazantsev 9 hours ago [-]
And now, it does. 70 tok/s on the 7900 XTX. It seems to be the fastest engine for AMD GPUs.
Tade0 7 hours ago [-]
Wow, that was fast.

Strata recognises my 7700S as a 7600XT, but nevertheless it's easily 50% faster than any llama.cpp configuration I could come up with.

That being said I'm getting just 8.4tok/s, so I guess I'll stick to Qwen, which gets twice of that.

staplehorse6553 13 hours ago [-]
[dead]
rsolva 13 hours ago [-]
I got it to ~60 t/s decode for prose, up tp 100 t/s for code and 1800 t/s prefill after lettin Claude loose for a while, so it has already surpassed Qwen 3.8 27b on my hardware, AMD 7900XTX.

Have not pushed the latest changes yet, doing some benching, but here is my repo: https://forge.dmz.skyfritt.net/ruben/llama-inkling-2-mini

veemjeem 24 hours ago [-]
It takes a while for 3rd party AMD engines to catch up, but 3rd party engines like capy-flash & basalt already run inkling2 at more than 100tps (5k prefill on 5070ti, 16k prefill on 5090). Maybe radiance (R9700 3rd party engine) will add inkling2 support soon?
woadwarrior01 1 days ago [-]
The NV in NVFP4 stands for NVIDIA. :)
22 hours ago [-]
plasticchris 1 days ago [-]
It’s just not optimized in the same way. A little tweaking on my vllm fork and it runs faster on my 7900xtxes.
rsolva 13 hours ago [-]
Have you got vLLM running with ROCm on these cards?
plasticchris 10 hours ago [-]
Yeah, it is a bit messy and RDNA 3 centric since it’s just tinkering in my spare time but it works really well.
gscott 15 hours ago [-]
Apparently the model is chatty good for a 5090 that can handle it.
hacker_88 15 hours ago [-]
Try sensanova 9b looped
baby 1 days ago [-]
Can you autoresearch optimize it?
anon373839 22 hours ago [-]
[flagged]
minraws 16 hours ago [-]
I had a rough time with llama.cpp so fixed the bugs in it for performance some notes you need a minimum of 12gb of vram for credible performance with decent context length above 100k...

The best I could do with established techniques and staying at nvfp4 levels of quant, is around 40-45 toks/s on 5070 Ti with 800 toks/s prefill around 128k kv cache... with the kernels matching output byte for byte and basic manual verification of the patched code.

3060 sits around 20-25 toks/s with prefill around 350-400 toks/s i think maybe it can be improved but my 3060 system was ddr4 so not the greatest.

Either way at below 6gb you cant run beyond q4 quants at above 10-20 tok/s with pretty small context size, afaik that gets awfully close to the roofline depending on expert cache hit rate... unless you quant the logits into the ground.

christkv 1 hours ago [-]
I have been using the 8bit version over the weekend on my framework (with a shitty llm studio runtime for now) and I'm pretty impressed. It did get stuck in a loop at one point (could be studio llm) but it managed to keep on task for the most part and managed to deliver some stories.
manlymuppet 1 days ago [-]
My Quantitive!
1 days ago [-]
testmasterflex 3 hours ago [-]
Lol
XCSme 22 hours ago [-]
So, if we run 4 or those on a 3090, can we get better answers through boosting?
Dwedit 1 days ago [-]
How does this method compare to the use of an importance matrix?
aktenlage 1 days ago [-]
Probably better but more effort
IronWolve 1 days ago [-]
Only mx version?
Prospernwo 1 days ago [-]
MX/Apple Silicon only for the released Inkling files, yes. The ht2/ht3 codes are hashed block-vector trellis codes meant to fit 4 KB of SRAM, decoded by custom Metal kernels, and the drafter is MLX-only too - none of that runs in llama.cpp today.

The GGUF route is in the repo (gguf_compiler.py emits plans over llama.cpp's own tensor types and the docs say the result runs in llama.cpp as-is), so porting these plans is clearly intended, but right now a 4090 gets you the Qwen files to read about rather than run.

Pragmata 1 days ago [-]
> Don't post generated text or AI-edited text. HN is for conversation between humans.

Please [review the commenting guidelines](https://news.ycombinator.com/newsguidelines.html)

kroaton 1 days ago [-]
50 comments, 234 points and literally nobody has tried this model. This either smells like astroturfing or the quality of HN has gone to shit completely.
AdamTReineke 18 hours ago [-]
I've been running it half the day on a 3090 working through a new project and it's a workhorse. I feel like it would hold its own against Luna.

Edit: At medium context (80k+) in my Pi harness, it seems to get confused about line endings and the edit tool. It will call sed a lot and wants to write Python scripts to edit files vs using the write tool.

Edit again: My AGENTS.md had an instruction that was making it obsess over line endings.

BonerWiener 20 hours ago [-]
Can't speak on behalf of HN, only my own opinion after trying it.

For the size of the model, it has some impressive abilities. It performs really great (rtx 3090) at coding tasks, but i have not tried for longer, more complex tasks with my current harness.

I also had some refusal issues - but i guess it will be fixed by abliteration from the community.

Check out r/LocalLlama - recent and hot topics are filled with this model and opinions

Havoc 17 hours ago [-]
Especially with the canonical link to the thinkingmachine site being ranked #41.

Still...I'm glad that they're getting hype and hn frontpage attention at all. The prior inkling release didn't really seem to catch on while /r/localllama seems much more stoked about this one

Tade0 12 hours ago [-]
The naming was a tad misleading. They've named it "Small", but it's half a trillion parameters.
XCSme 12 hours ago [-]
I tested the full version, not this quant, and it's NOT better than Qwen 3.8 27B in my tests. I did run it via OpenRouter though, I am running the tests now locally too.

https://aibenchy.com/compare/thinkingmachines-inkling-high/r...

seanmcdirmid 12 hours ago [-]
Im downloading right now and will try tomorrow. Supposedly better than Qwen 3.5 35b A3b although not as fast, so I’m curious although I doubt it will be better than the 2.75 quantized version of Qwen Flash Next that is my best local model right now (definitely uses less memory though).
zeroq 2 hours ago [-]
I wish all youtube videos praising quantized models would be presented with both audio and video quantized to 4 bits.
woadwarrior01 1 days ago [-]
From the model card:

> The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.

What's the point in quantizing a QAT model? Just for social media karma?

gavmor 1 days ago [-]
Just to reiterate the question: QAT approximates quantization-errors in full-size fine-tuning, so that the model itself is insensitive to (or accommodating of, I'm not sure) them when quantized to the corresponding format.

Quantization to a non-corresponding format promises none of those accommodations.

svnt 23 hours ago [-]
I think they say they go back to BF16 and then requant/finetune smaller (and to a different format).

Afaik these are necessary steps to do the work and then get to the basic end state they want (model that will run on a Mac with 8GB or 16GB free ram). Going down to smaller models gives at least the option to run them on way more devices (in this case the majority of apple silicon MacBooks), even if performance is not as strong.

asaddhamani 1 days ago [-]
QAT models are trained to quantize well? I don’t get the concern here.
woadwarrior01 1 days ago [-]
Please read their blog post. This isn't a run of the mill QAT post trained model (like Gemma 4 QAT). It was natively trained in NVFP4, through and through.

> Most of Inkling 2 Mini’s weights were trained and are deployed natively in NVFP4, with the last four layers in MXFP8.

https://thinkingmachines.ai/news/inkling-2-mini/

Prospernwo 1 days ago [-]
[flagged]
cheapcheerful 1 days ago [-]
"Look at my quant 2"

/igotthatreference.jpg

nico 1 days ago [-]
Yeah, I enjoyed the reference, though I didn't really understand it until I saw the picture on https://lookatmyquant.pages.dev
tomaskafka 1 days ago [-]
Pikamander2 11 hours ago [-]
Look at my quant!

Yeah, my quant is amazing!

Notice the diff?

Yeah that's right, he is Asian!

hugmynutus 1 days ago [-]
*TL;DR* Promising initial results but shoddy research. OP's quant did not get 1st or 2nd prize in that math competition.

----

Gave it a quick browse to verify what my local clanker told me, it seemed to check out

I believe it is Dynamically allocating weights based on fisher information of conversation traces & For some K-quant look up table stuff it can store in an importance matrix to get better compression.

The results are complete nonsense and really don't back up any of the README claims. For Inkling-2-Mini; at 5.4 GB its KL is worse than Unsloth's UD-IQ1_S: 0.0895 vs 0.0808 on English/chat/code, and 0.287 vs 0.124 across 13 languages (e.g.: horrible). It then achieved better results by post-training the qaunt'd model on sample inputs (AFAIT).

According to anthropomorphic-clud their results for GSM8K & HumanEval are within the noise regime and vastly over stated.

collabspherestu 1 days ago [-]
[flagged]
swiftcoder 13 hours ago [-]
[flagged]
amauriya 14 hours ago [-]
[flagged]
peter_d_sherman 1 days ago [-]
[dead]
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 21:22:05 GMT+0000 (Coordinated Universal Time) with Vercel.