High-demand open-source features with detailed plans already in place — the work is scoped, the need is proven, only funding is missing. Sponsoring through QuantStack means your investment lands upstream and benefits the entire ecosystem. Back a project and get it shipped.
Conversion of Jupyter notebooks to PDF currently relies on nbconvert in the backend, which in turns uses a headless browser for producing the PDF. We propose to…
Jupyter Ecosystem
Raster processing tools in JupyterGIS
JupyterGIS currently offers a set of vector processing and conversion tools. These capabilities are powered by a GDAL WebAssembly (WASM) build running in the br…
Jupyter Ecosystem
Bringing processing tools to the JupyterGIS Python API
JupyterGIS currently supports several vector processing and conversion tools, currently available only through the JupyterGIS user interface. We plan to extend …
Package Management
Package requests for emscripten-forge
Emscripten-forge is a conda package distribution specifically designed for WebAssembly. While the number of available emscripten-forge packages is growing quick…
Scientific Computing
SVE2 support in xsimd
xsimd is a C++ scientific library that abstracts low-level high performances computing primitives across different hardwares. We will add support for the latest…
Scientific Computing
Float16 support in xsimd
xsimd is a C++ scientific library that abstracts low-level high performances computing primitives across different hardwares. We will add vectorized support for…
Scientific Computing
Implementing Kazushige Goto Algorithms for Matrix Operations in xtensor
This project aims to integrate Kazushige Goto’s highly optimized matrix multiplication algorithms into the xtensor framework, leveraging the xsimd library for S…
Apache Arrow and Parquet
Complete BinaryView / StringView support in Arrow C++
BinaryView is a more recent and more efficient alternative to Arrow's standard Binary type. It allows for inlined storage of short strings and fast prefix compa…
Apache Arrow and Parquet
Complete Decimal32 / Decimal64 support in Arrow C++
Decimal32 and Decimal64 are more compact and computationally more efficient data types than the standard Decimal128.
Apache Arrow and Parquet
Complete Float16 support in Arrow C++
Float16 is a more compact data type than Float32 and Float64, and sees growing usage in applications where its limited precision is sufficient.
Apache Arrow and Parquet
Complete Run-End-Encoded support in Arrow C++
Like dictionary encoding, run-end-encoding allows representing some kinds of data more efficiently.
Apache Arrow and Parquet
Parquet reader optimizations
Converting Parquet optional values to nullable Arrow data is often a performance bottleneck. We will optimize that step for the most common cases.
Can't find a project?
If you have a project in mind that you think would be relevant to our expertise, contact us to discuss it.
Apache Parquet is an open source, column-oriented data file format designed for
efficient data storage and retrieval. Together with Apache Arrow for in-memory data,
it has become the de facto standard for efficient columnar analytics.
While Parquet and Arrow are most often used together, they have incompatible physical
representations of data with optional values: data where some values can be
missing or "null". While Arrow uses a validity bitmap for each schema field and nesting level,
Parquet condenses that information in a more sophisticated structure called definition
levels (borrowing ideas from Google's Dremel project).
Converting between those two representations is non-trivial and often turns out
a performance bottleneck when reading a Parquet file as in-memory Arrow data.
Even columns that practically do not contain any nulls can still suffer from it if
the data is declared nullable (optional) at the schema level.
We propose to optimize the conversion of null values from Parquet in Arrow C++
for flat (non-nested) data:
decoding Parquet definition levels directly into an Arrow validity bitmap, rather than using an
intermediate representation as 16-bit integers;
avoiding decoding definition levels entirely when a data page's statistics shows
it cannot contain any nulls (or, conversely, when it cannot contain any non-null values).
As a subsequent task, these optimizations may be extended so as to apply to schemas
with moderate amounts of nesting.
This work will benefit applications using Arrow C++ or any of its language
bindings (such as PyArrow, R-Arrow...).
Depending on the typology of Parquet data, this could make Parquet reading 2x
faster, even more in some cases. If you are unsure whether your workload could
benefit, we can discuss this based on sample Parquet files you provide us.
Are you interested in this project? Either entirely or partially, contact us for more information on how to help us fund it