I laugh everytime I have to explain that "Apache Arrow format is more efficient than JSON. Yes, the format is called 'Apache Arrow.'"
data_ders 11 hours ago [-]
if I could tell myself in 2015 who had just found the feather library and was using it to power my unhinged topic modeling for power point slides work, and explained what feather would become (arrow) and the impact it would have on the date ecosystem. I would have looked at 2026 me like he was a crazy person.
Yet today I feel it was 2016 dataders who is the crazy one lol
ayhanfuat 11 hours ago [-]
Indeed. feather was a library to exchange data between R and pandas dataframes. People tend to bash pandas but its creator (Wes McKinney) has changed the data ecosystem for the better with the learnings coming from pandas.
jtbaker 10 hours ago [-]
I know pandas has a lot of technical warts and shortcomings, but I'm grateful for how much it empowered me early in my data/software career, and the API still feels more ergonomic to me due to the years of usage - plus GeoPandas layering on top of it.
Really, prefer DuckDB SQL these days for anything that needs to perform well, and feel like SQL is easier to grok than python code most of the time.
sirfz 2 hours ago [-]
chdb's new DataStore API looks really neat (drop in pandas replacement) and exactly how I envisioned a faster pandas could be without sacrificing its ergonomics
0xcafefood 11 hours ago [-]
Do people bash pandas? If so, it reminds me of Bjarne's quip that the two types of programming languages are the ones people complain about and the ones nobody uses.
polars people do - although I wouldn't call polars something that nobody uses.
ayhanfuat 11 hours ago [-]
I also use polars in new projects. I think Wes McKinney also uses it. If I remember correctly I saw him commenting on some polars memory related issues on GitHub. But a good chunk of polars' success can be attributed to Arrow which McKinney co-created. All the gripes people have with pandas, he had them too and built something powerful to overcome those.
mistrial9 10 hours ago [-]
I saw Wes speak in the early days of Pandas, in Berkeley. He solved problems that others just worked around for decades. His solutions are quirky but the work was very solid. His career advanced a lot IMHO for substantial reasons.. Wes personally marched through swamps and reached the other side.. others complain and do what they always have done.. I personally agree with the criticisms of the syntax, but Pandas is real and it was not easy to build it.
aynyc 8 hours ago [-]
What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?
tosh 8 hours ago [-]
parquet is optimized for storage and compresses well (=> smaller files)
feather is optimized for fast reading
aynyc 7 hours ago [-]
Given the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.
yencabulator 6 hours ago [-]
You can, still, gain a lot of performance by doing less I/O.
outside1234 7 hours ago [-]
What people have done in the face of cheaper storage is store more data.
farsa 3 hours ago [-]
Storage getting cheaper did not really reach the cloud providers and for self-hosting it has recently gotten even more expensive due to AI bs.
I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.
yencabulator 6 hours ago [-]
Generally Parquet files are combined in an LSM style, compacting smaller files into larger ones. Parquet isn't really meant for the "journal" of level-0 append-one-record style storage, it's meant for the levels that follow.
aynyc 5 hours ago [-]
So feather for journaling and parquet for long term processing?
yencabulator 4 hours ago [-]
You basically can't do row by row appends to any columnar format stored in a single file. You could kludge around it by allocating arenas inside the file but that's still a huge write amplification, instead of writing a row in a single block you'd have to write a block per column.
amluto 2 hours ago [-]
You can do row by row appends to a Feather (Arrow IPC — the naming is confusing). It works fine. The main problem is that the per-append overhead is kind of silly — it costs over 300 bytes (IIRC) per append.
I wish there was an industry standard format, schema-compatible with Parquet, that was actually optimized for this use case.
yencabulator 2 hours ago [-]
Creating a new record batch for a single row is also a huge kludge leading to lot of write amplification. At that point, you're better off storing rows than pretending it's columnar.
I actually wrote a row storage format reusing Arrow data types (not Feather), just laying them out row-wise not columnar. Validity bits of the different columns collected into a shared per-row bitmap, fixed offsets within a record allow extracting any field in a zerocopy fashion. I store those in RocksDB, for now.
> Creating a new record batch for a single row is also a huge kludge leading to lot of write amplification.
Sure, except insofar as I didn’t want to pretend to be columnar. There just doesn’t seem to be something out there that met my (experimental) needs better. I wanted to stream out rows, event sourcing style, and snarf them up in batches in a separate process into Parquet. Using Feather like it’s a row store can do this.
> kantodb
Neat project. I would seriously consider using that in a project of mine, especially now that LLMs can help out with the exceedingly tedious parts. (The current stack is regrettable, but a prompt like “keep exactly the same queries but change the API from X to Y” is well within current capabilities.)
gregw2 2 hours ago [-]
Agreed.
There is room still for an open source HTAP storage format to be designed and built. :-)
dionian 5 hours ago [-]
Have you considered something like iceberg tables?
aynyc 1 hours ago [-]
Yes, but parquet hates small files.
pm90 10 hours ago [-]
Its nice to see useful, impactful interchange formats getting the attention and resources they need, and ecosystems converging around them. Optimizing serialization/deserialization might seem like a "trivial" task at first, but when moving petabytes of data they quickly become the bottlenecks. With common interchange formats, the benefits of these optimizations are shared across stacks. Love to see it.
sudoshred 4 hours ago [-]
Intuitively appreciating that these "boring fundamentals" are the default bottlenecks is a aign of senior+ swe capability.
aerzen 8 hours ago [-]
I like arrow for its type system. It's efficient, complete and does not have "infinite precision decimals". Considering Postgres's decimal encoding, using i256 as the backing type is so much saner approach.
mempko 9 hours ago [-]
We use Apache Arrow at my company and it's fantastic. The performance is so good. We have terabytes of time-series financial data and use arrow to store it and process it.
kccqzy 8 hours ago [-]
We use Apache Arrow at my company too. It is part of a migration from an old in-house format. When it works it’s good. But there are just way too many bugs in Arrow. For example: a basic arrow computation on strings segfaults because the result does not fit in Arrow’s string type, only the large string type. Instead of casting it or asking the user to cast it, it just segfaults. Another example: a different basic operation causes an exception complaining about negative buffer sizes when using variable-length binary type.
thinkharderdev 7 hours ago [-]
This will obviously depend on which implementation you use. Using the rust arrow-rs crate you at least get panics when you overflow max buffer sizes. But one of my enduring annoyances with arrow is that they use signed integer types for buffer offsets and the like. I understand why it has to be that way since it's intended to be cross-language and not all languages have unsigned integer types. But it does lead to lots of very weird bugs when you are working in a native language and casting back and forth from signed to unsigned types. I spent a very frustrating day tracking down this one in particular https://github.com/apache/datafusion/issues/15967
dionian 8 hours ago [-]
stumbled upon it recently while optimizing parquet writes. It worked flawlessly and 10-20x'd my throughput
actionfromafar 11 hours ago [-]
I had to look up what Arrow actually does, and I might have to run some performance comparisons vs sqlite.
It's very neat for some types of data to have columns contiguous in memory.
skeeter2020 11 hours ago [-]
>> some performance comparisons vs sqlite.
That's not really the purpose; it's really a language-independent format so that you don't need to change it for say, a dataframe or R. It's columnar because for analytics (where you do lots of aggregations and filtering) this is way more performant; the data is intentionally stored so the target columns are continuous. You probably already know, but the analytics equivalent of SQLite is DuckDB. Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is.
mandeepj 9 hours ago [-]
> Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is.
Not sure if I misunderstood, what are the chances those different consumers / tools / operations are running in your memory space?
daddykotex 9 hours ago [-]
Not an expert, so I could be wrong, but my understanding is that you could just copy those bytes directly from the wire to your memory and treat those as the Arrow payload you're expecting it to be.
You still have to transfer the data, but you remove the need for a transformation before writing to the wire, and a transformation when reading from the wire.
cestith 8 hours ago [-]
If you are in control of two processes on a single machine instance, you could share the memory between a writer and a read-only consumer.
The key phrase though would seem to be “memory representation”m and not “same memory”. You can spit the in-memory representation out to an Arrow file or an Arrow stream, take it in, and it’s in the same memory layout in the other program. That’s kind of the point of Arrow. It’s a standard memory layout available across applications and even across languages, which can be really convenient.
shadow28 6 hours ago [-]
Arrow supports zero-copy data sharing - checkout the Arrow IPC format and Arrow Flight.
actionfromafar 10 hours ago [-]
Thanks! This is all probably me using the familiar sqlite hammer where I really shouldn't.
nu11ptr 11 hours ago [-]
If I recall, Arrow is more or less a standardized representation in memory of columnar data. It tends to not be used directly I believe, but as the foundation for higher level libraries (like Polars, etc.). That said, I'm not an expert here so might not have full info.
tormeh 11 hours ago [-]
You can absolutely use it directly, but it is painful. The USP of Arrow ist that you can pass bits of memory between Polars, Datafusion, DuckDB, etc. without copying. It's Parquet but for memory.
skeeter2020 11 hours ago [-]
This is true, and as a result IME the problem space is much smaller than Parquet, but it can be really powerful. The reality is most of us don't work in environments where Arrow is needed.
tosh 10 hours ago [-]
Take a look at parquet.
You can also store arrow on disk but it is mainly used as in-memory representation.
data_ders 11 hours ago [-]
yeah not necessarily compute (though it has a kernel)!
it's actually many things IPC protocol wire protocol, database connectivity spec etc etc.
in reality it's about an in-memory tabular (columnar) representation that enables zero copy operations b/w languages and engines.
and, imho, it all really comes down to standard data types for columns!
8 hours ago [-]
HoldOnAMinute 5 hours ago [-]
I read that entire page and I could not tell you what Apache Arrow is, or what it does.
gchamonlive 4 hours ago [-]
The post celebrates Apache Arrow's 10 years anniversary, so it's assuming you already know what it is and what it does, which I think is fair. If you don't you can always refer to the docs.
depr 5 hours ago [-]
All you had to do was click the logo to go to the homepage
Rendered at 02:50:18 GMT+0000 (Coordinated Universal Time) with Vercel.
Yet today I feel it was 2016 dataders who is the crazy one lol
Really, prefer DuckDB SQL these days for anything that needs to perform well, and feel like SQL is easier to grok than python code most of the time.
feather is optimized for fast reading
I wish there was an industry standard format, schema-compatible with Parquet, that was actually optimized for this use case.
I actually wrote a row storage format reusing Arrow data types (not Feather), just laying them out row-wise not columnar. Validity bits of the different columns collected into a shared per-row bitmap, fixed offsets within a record allow extracting any field in a zerocopy fashion. I store those in RocksDB, for now.
https://git.kantodb.com/kantodb/kantodb/src/branch/main/crat...
https://git.kantodb.com/kantodb/kantodb/src/branch/main/crat...
https://git.kantodb.com/kantodb/kantodb/src/branch/main/crat...
Sure, except insofar as I didn’t want to pretend to be columnar. There just doesn’t seem to be something out there that met my (experimental) needs better. I wanted to stream out rows, event sourcing style, and snarf them up in batches in a separate process into Parquet. Using Feather like it’s a row store can do this.
> kantodb
Neat project. I would seriously consider using that in a project of mine, especially now that LLMs can help out with the exceedingly tedious parts. (The current stack is regrettable, but a prompt like “keep exactly the same queries but change the API from X to Y” is well within current capabilities.)
There is room still for an open source HTAP storage format to be designed and built. :-)
It's very neat for some types of data to have columns contiguous in memory.
That's not really the purpose; it's really a language-independent format so that you don't need to change it for say, a dataframe or R. It's columnar because for analytics (where you do lots of aggregations and filtering) this is way more performant; the data is intentionally stored so the target columns are continuous. You probably already know, but the analytics equivalent of SQLite is DuckDB. Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is.
Not sure if I misunderstood, what are the chances those different consumers / tools / operations are running in your memory space?
You still have to transfer the data, but you remove the need for a transformation before writing to the wire, and a transformation when reading from the wire.
The key phrase though would seem to be “memory representation”m and not “same memory”. You can spit the in-memory representation out to an Arrow file or an Arrow stream, take it in, and it’s in the same memory layout in the other program. That’s kind of the point of Arrow. It’s a standard memory layout available across applications and even across languages, which can be really convenient.
You can also store arrow on disk but it is mainly used as in-memory representation.
it's actually many things IPC protocol wire protocol, database connectivity spec etc etc.
in reality it's about an in-memory tabular (columnar) representation that enables zero copy operations b/w languages and engines.
and, imho, it all really comes down to standard data types for columns!