any measured commit latency overhead from the vector index on a streaming job, once it's built? post benchmarkss build time + query latency but not the write path stuff.
cloud8bits 2 hours ago [-]
We had reported index build time, but not continuous maintenance. Good feedback, we will include it a future blog.
To answer this now:
a) Assuming the vectors are generated, its one additional set of sharded writes to object storage. updates/deletes are logged, then read side merged, during index lookup.
b) This is same as how secondary indexing works, and it adds about 10-15% more to writer pipeline. Most expensive part still remains producing columnar files for the actual data.
Hope that helps.
rjshmhndr 4 hours ago [-]
vector search with < 100msecs on a lake is unheard of - pretty impressive
jauntywundrkind 8 hours ago [-]
Pretty good write up, talking to the problem, surveying what and how other players do things?
Went looking to see if there was going to be anything open (mostly it's product pitch), found:
> Iceberg support builds on the Hudi-backed indexing work in Apache XTable (Incubating). We developed the design in the open with engineers from Uber and Walmart to support their planet-scale tables. The open RFC documents the storage layout, query path and consistency model.
> The design is settled and ships in Lakegres today; the Hudi implementation is being upstreamed and XTable is in release-candidate testing ahead of OSS releases that other engines can integrate.
The problem doesn’t really seem to be solved though. Most attempts at decentralized index storage will be limited by what this project calls nprobes. The authors propose a cheap search first approach but it’s unclear how it scales.
cloud8bits 8 hours ago [-]
[flagged]
vinish1811 7 hours ago [-]
[dead]
cloud8bits 8 hours ago [-]
[dead]
jyasveer 9 hours ago [-]
great
cloud8bits 9 hours ago [-]
[flagged]
tokYoComPleX 9 hours ago [-]
This is a solidly built feature with great performance eclipsing other competing solutions on the market.
CBLT 8 hours ago [-]
Since you created an account just to compliment this product, it would probably be best to explain your relationship to the product in question. Even a simple "not affiliated, just a happy user" would be great.
Rendered at 02:33:27 GMT+0000 (Coordinated Universal Time) with Vercel.
To answer this now:
a) Assuming the vectors are generated, its one additional set of sharded writes to object storage. updates/deletes are logged, then read side merged, during index lookup.
b) This is same as how secondary indexing works, and it adds about 10-15% more to writer pipeline. Most expensive part still remains producing columnar files for the actual data.
Hope that helps.
Went looking to see if there was going to be anything open (mostly it's product pitch), found:
> Iceberg support builds on the Hudi-backed indexing work in Apache XTable (Incubating). We developed the design in the open with engineers from Uber and Walmart to support their planet-scale tables. The open RFC documents the storage layout, query path and consistency model.
> The design is settled and ships in Lakegres today; the Hudi implementation is being upstreamed and XTable is in release-candidate testing ahead of OSS releases that other engines can integrate.
With two links inline, https://github.com/apache/incubator-xtable/pull/935 https://github.com/apache/hudi/blob/master/rfc/rfc-109/rfc-1...