That's an interesting approach, but... what's the performance impact? It seems the check was done only against really trivial tasks. Why not run some known coding benchmark against the original and pruned one?
Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.
22 hours ago [-]
AmSach 22 hours ago [-]
[flagged]
22 hours ago [-]
aetherspawn 1 days ago [-]
Does it improve tps for Mac?
A 6x improvement from 50tps to 300tps for example would be pretty amazing. There’s apparently a threshold where virtually all problems can be brute forced by having a lot of fast tokens rather than a larger model.
hxii 1 days ago [-]
Can the same process be theoretically repeated with LFM2.5-24B-A2B ?
22 hours ago [-]
tamnv 1 days ago [-]
I really like topics about optimization like this one. Great job, OP!
peter_d_sherman 1 days ago [-]
Looks absolutely stellar!
Another step forward for local AI / local LLM inferencing on (very!) resource-constrained consumer GPU hardware!
orchardmark 1 days ago [-]
[flagged]
Rendered at 17:12:43 GMT+0000 (Coordinated Universal Time) with Vercel.
Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.
A 6x improvement from 50tps to 300tps for example would be pretty amazing. There’s apparently a threshold where virtually all problems can be brute forced by having a lot of fast tokens rather than a larger model.
Another step forward for local AI / local LLM inferencing on (very!) resource-constrained consumer GPU hardware!