Its cool, but it goes against the whole optimization that NLP has gone through with BPE and redundancy/repetition in large data in general.
Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.
Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.
nivter 1 hours ago [-]
> you will need a very domain-specific use for this to be useful imo
The main premise of this idea is how general and widely applicable it is. They also show examples of how it works on images.
A common tokenizing paradigm would be a huge step towards consolidating how different modalities are preprocessed - something I believe is quite useful for intelligence to scale well across modalities.
vatsachak 9 hours ago [-]
Oh man, that haskell quicksort tokenization has got me feeling great
aburayhan 2 hours ago [-]
[flagged]
Rendered at 09:52:29 GMT+0000 (Coordinated Universal Time) with Vercel.
Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.
Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.
The main premise of this idea is how general and widely applicable it is. They also show examples of how it works on images.
A common tokenizing paradigm would be a huge step towards consolidating how different modalities are preprocessed - something I believe is quite useful for intelligence to scale well across modalities.