Amazing. I remember being really disappointed in mirai last year when learning that it couldn’t provide shared access to an in-memory arrow table. Multicore processing in R is great when it works but too often it’s just a matter of time before unwanted and unnecessary data copies completely destroy performance.
shikokuchuo 3 days ago [-]
Author here: I wrote mizu (R) and pymizu (Python), packages that offer lock-free async and parallel computing over shared memory, built on the same C core, libmizu. The link above is for the Python package.
A task round trip drops to under one microsecond, from around 100 µs over sockets with Python's ProcessPoolExecutor or mirai, my other R framework. These are numbers from my laptop (an M4 Pro), and benchmark scripts are in the repos for you to run.
Workers claim tasks off lock-free queues, there is no task dispatcher, and system calls are avoided on the hot path.
`PoolExecutor` is a drop-in `concurrent.futures.Executor`, tasks are awaitable in asyncio, NumPy arrays and polars, pyarrow and pandas frames cross as zero-copy views. As mizu and pymizu share the same wire format, data frames and Arrow types also cross zero-copy between worker pools of different languages.
Of course, this is single-machine only as it uses shared memory. The API is still experimental. Usability feedback and bug reports are most welcome at the repos. I'd love to hear about places you'd like to see this used.
I have no use for this but I want to commend you on defending your claims and taking objective numeric measurements. The lack of these lately on performance improvement based projects has been disturbing
shikokuchuo 54 minutes ago [-]
That does sound... disturbing. I also run benchmarks regularly on both Linux and Mac to ensure not too much divergence between the two.
rmah 4 hours ago [-]
Impressive work! This looks like it could be a game changer for some of our tasks... I don't mean to be one of those pushy people but... does it, perhaps support Polars?
shikokuchuo 1 hours ago [-]
You're in luck! In fact anything that speaks the Arrow C data interface (polars, pyarrow, duckdb).
Rendered at 17:25:40 GMT+0000 (Coordinated Universal Time) with Vercel.
A task round trip drops to under one microsecond, from around 100 µs over sockets with Python's ProcessPoolExecutor or mirai, my other R framework. These are numbers from my laptop (an M4 Pro), and benchmark scripts are in the repos for you to run.
Workers claim tasks off lock-free queues, there is no task dispatcher, and system calls are avoided on the hot path.
`PoolExecutor` is a drop-in `concurrent.futures.Executor`, tasks are awaitable in asyncio, NumPy arrays and polars, pyarrow and pandas frames cross as zero-copy views. As mizu and pymizu share the same wire format, data frames and Arrow types also cross zero-copy between worker pools of different languages.
Of course, this is single-machine only as it uses shared memory. The API is still experimental. Usability feedback and bug reports are most welcome at the repos. I'd love to hear about places you'd like to see this used.
Write-up with full benchmarks and the R/Python code examples: https://opensource.posit.co/blog/2026-11-05_mizu-0-1-0/