API Reference

This documents the public C ABI. Every function maps Vulkan concepts to columnar data operations. Opaque handles keep driver internals out of the public surface.

Core Types

enum VcDtype

Column data type enumeration. Corresponds to GPU-side uint / float types in GLSL shaders.

enumerator VC_DTYPE_U8 = 0

enumerator VC_DTYPE_U16 = 1

enumerator VC_DTYPE_U32 = 2

enumerator VC_DTYPE_U64 = 3

enumerator VC_DTYPE_I8 = 4

enumerator VC_DTYPE_I16 = 5

enumerator VC_DTYPE_I32 = 6

enumerator VC_DTYPE_I64 = 7

enumerator VC_DTYPE_F32 = 8

enumerator VC_DTYPE_F64 = 9

enumerator VC_DTYPE_STRING = 10

enumerator VC_DTYPE_LIST = 11

enumerator VC_DTYPE_STRUCT = 12

type VcContext: Opaque handle to a GPU context. Internally owns a VkInstance, VkPhysicalDevice, VkDevice, one compute VkQueue, a VkCommandPool, and a VmaAllocator.

type VcColumn: Opaque handle to a typed column buffer on the GPU. Backed by a VkBuffer allocated through VMA with VK_BUFFER_USAGE_STORAGE_BUFFER_BIT so compute shaders can read and write it directly.

Context Lifecycle

VcContext *vc_create_context(void)

Status: fully implemented.

Creates a Vulkan instance (VK_API_VERSION_1_3), selects a physical device (preferring AMD vendorID == 0x1002, falling back to any compute-queue GPU), creates a logical device with one compute queue, a VMA allocator, and a command pool with VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT.

In debug builds (_DEBUG defined), attempts to enable VK_LAYER_KHRONOS_validation and VK_EXT_DEBUG_UTILS_EXTENSION_NAME. Gracefully falls back if the layer is not installed.

Returns:: A new context, or NULL on failure.

void vc_destroy_context(VcContext *ctx)

Status: fully implemented.

Tears down the command pool, VMA allocator, logical device, and instance in reverse creation order. Passing NULL is a no-op.

Column Management

VcColumn *vc_create_column(VcContext *ctx, VcDtype dtype, uint64_t length, const void *host_data)

Status: fully implemented.

Allocates a VkBuffer through VMA with VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT (shader access + transfer source/target for host staging). Uses VMA_MEMORY_USAGE_AUTO so the driver picks device-local or host-visible memory as appropriate.

If host_data is provided, creates a transient staging buffer (TRANSFER_SRC, HOST_ACCESS_SEQUENTIAL_WRITE | CREATE_MAPPED), memcpy’s the data into it, issues vkCmdCopyBuffer from staging to the device-local buffer, inserts a TRANSFER_WRITE → SHADER_READ pipeline barrier, submits, and waits.

Edge case: zero-length columns allocate a minimal 4-byte buffer to keep Vulkan happy. Large no-data allocations (>64 MiB) also use the minimal allocation — the logical length is stored but the backing buffer is placeholder-sized.

Parameters:

ctx – GPU context.
dtype – Element data type.
length – Number of elements.
host_data – Host data to upload, or NULL to leave uninitialized.

Returns:

A new column, or NULL on failure.

void vc_destroy_column(VcContext *ctx, VcColumn *col)

Status: fully implemented.

Calls vmaDestroyBuffer and frees the column struct. Passing NULL for either argument is a no-op.

int vc_read_column(VcContext *ctx, VcColumn *col, void *host_buffer, uint64_t host_buffer_size)

Status: fully implemented.

Reads GPU column data back to the host through a staging buffer:

Allocates a temporary staging buffer (TRANSFER_DST, HOST_ACCESS_RANDOM | CREATE_MAPPED).
Inserts a SHADER_WRITE → TRANSFER_READ barrier on the column buffer (ensures any prior compute writes are visible to the transfer engine).
Issues vkCmdCopyBuffer from the column buffer to the staging buffer.
Inserts a TRANSFER_WRITE → HOST_READ barrier on the staging buffer.
Submits, waits, memcpy’s from staging mapped memory to host_buffer.

Zero-length columns return success immediately.

Parameters:

ctx – GPU context.
col – Column to read.
host_buffer – Destination buffer on the host.
host_buffer_size – Size of the destination buffer in bytes.

Returns:

0 on success, -1 on error (e.g., buffer too small).

uint64_t vc_column_length(VcColumn *col): Status: fully implemented. Returns the number of elements. Returns 0 if col is NULL.

VcDtype vc_column_dtype(VcColumn *col): Status: fully implemented. Returns the element data type. Returns VC_DTYPE_U8 if col is NULL.

Null Support (Arrow Validity Bitmaps)

The library supports nullable columns through Arrow-compatible validity bitmaps. A validity bitmap is a packed array of uint32_t words, one bit per element, LSB-first per word. A set bit means the element is valid; a cleared bit means null. The bitmap length is ceil(N/32) words for a column with N elements.

VcColumn maps 1:1 to an Arrow PrimitiveArray: validity_buffer is Arrow buffers[0] and buffer is Arrow buffers[1]. This enables zero-copy wrappers in Polars, DataFusion, or PySpark bindings.

VcColumn *vc_create_column_nullable(VcContext *ctx, VcDtype dtype, uint64_t length, const void *host_data, const uint32_t *validity, uint64_t null_count)

Status: fully implemented.

Creates a nullable column. Behaves identically to vc_create_column() for the data buffer, then additionally allocates and uploads the validity bitmap if validity is non-NULL and null_count > 0.

Vulkan primitives: allocates a STORAGE_BUFFER | TRANSFER_DST buffer for the validity bitmap through VMA, copies the host-side bitmap via a transient staging buffer and vkCmdCopyBuffer, then inserts a TRANSFER_WRITE -> SHADER_READ barrier.

Edge case: if validity is NULL or null_count == 0, the column is created without a validity buffer (i.e., all elements are considered valid).

Parameters:

ctx – GPU context.
dtype – Element data type.
length – Number of elements.
host_data – Host data to upload, or NULL.
validity – Arrow-format validity bitmap (packed uint32), or NULL.
null_count – Number of null elements.

Returns:

A new nullable column, or NULL on failure.

int vc_column_has_validity(VcColumn *col)

Status: fully implemented.

Returns 1 if the column has a validity buffer, 0 otherwise. Returns 0 if col is NULL.

uint64_t vc_column_null_count(VcColumn *col)

Status: fully implemented.

Returns the null count set at column creation time. Note: operations that remove elements (vc_filter) do not recompute the null count; the value is preserved from the input column. A future compute-shader pass can recompute it from the output validity buffer.

Returns 0 if col is NULL.

int vc_read_column_validity(VcContext *ctx, VcColumn *col, void *host_buffer, uint64_t host_buffer_size, uint32_t *validity_out)

Status: fully implemented.

Reads both column data and validity bitmap back to the host. First calls vc_read_column() for the data, then performs a vkCmdCopyBuffer from the validity buffer to a transient staging buffer and memcpy’s the bitmap to validity_out.

If the column has no validity buffer, validity_out is left untouched. The caller must provide a validity_out buffer of at least ceil(col->length / 32) uint32_t words.

Parameters:

ctx – GPU context.
col – Column to read.
host_buffer – Destination buffer for data (passed to vc_read_column).
host_buffer_size – Size of the data destination buffer.
validity_out – Destination buffer for validity bitmap.

Returns:

0 on success, -1 on error.

Arrow C Data Interface

The library implements the Apache Arrow C Data Interface for zero-copy interop with Polars, DataFusion, Spark, DuckDB, Velox, and any other framework that speaks Arrow. VcColumn maps 1:1 to an Arrow PrimitiveArray (buffers[0] = validity, buffers[1] = data).

VcColumn *vc_import_arrow(VcContext *ctx, VcArrowArray *array, VcArrowSchema *schema)

Status: fully implemented.

Import an Arrow array into a VcColumn. The data and validity buffers are copied from host to GPU via staging buffers and vkCmdCopyBuffer. The Arrow structs must remain valid until the call returns.

The Arrow C Data Interface stores buffer pointers inline, immediately after the VcArrowArray struct (not in private_data). Use reinterpret_cast<const void**>(&aa + 1) to access them.

Constraints: array->n_buffers >= 2, array->offset == 0 (sliced arrays not supported). Supports primitive numeric types (C/S/I/L/c/s/i/l/f/g), strings (u/U), lists (+l), and structs (+s).

VcArrowArray aa;
VcArrowSchema as;
// ... framework populates aa and as ...
VcColumn* col = vc_import_arrow(ctx, &aa, &as);
// ... run GPU operations on col ...
vc_destroy_column(ctx, col);

Parameters:

ctx – GPU context.
array – Arrow array to import.
schema – Arrow schema describing the data type.

Returns:

A new VcColumn on GPU, or NULL on failure.

int vc_export_arrow(VcContext *ctx, VcColumn *col, VcArrowArray *out_array, VcArrowSchema *out_schema)

Status: fully implemented.

Export a VcColumn to Arrow-compatible host buffers. Reads GPU data and validity (if present) back to host via staging buffers. Fills out_array and out_schema with buffer pointers and type metadata.

The caller must call out_array->release(out_array) and out_schema->release(out_schema) to free the exported host buffers when done. The structs themselves are not freed — the caller owns them (typically stack-allocated).

VcArrowArray aa;
VcArrowSchema as;
vc_export_arrow(ctx, col, &aa, &as);
// buffers are inline at (const void**)(&aa + 1)
// ... consume the Arrow array ...
aa.release(&aa);
as.release(&as);

Parameters:

ctx – GPU context.
col – Column to export.
out_array – [out] Populated Arrow array struct.
out_schema – [out] Populated Arrow schema struct.

Returns:

0 on success, -1 on error.

Integration Example: Polars in Python

import ctypes
import polars as pl
from pyarrow.cffi import ffi as arrow_ffi

lib = ctypes.CDLL("libvulkan_columnar.so")
ctx = lib.vc_create_context()

# Export Polars Series as Arrow C Data Interface structs.
series = pl.Series("vals", [1, 2, None, 4, 5], dtype=pl.UInt32)
arrow_array = series.to_arrow()
c_array_ptr, c_schema_ptr = arrow_ffi.export_array_and_schema(
    arrow_array, arrow_array.type)

# Import to GPU.
col = lib.vc_import_arrow(ctx, c_array_ptr, c_schema_ptr)

# Run GPU operations.
mask_col = lib.vc_create_column(ctx, 0, ...)  # U8 mask
filtered = lib.vc_filter(ctx, col, mask_col, byref(out_len))

# Export back.
result_array = ffi.new("VcArrowArray*")
result_schema = ffi.new("VcArrowSchema*")
lib.vc_export_arrow(ctx, filtered, result_array, result_schema)

# Import into Polars.
result_series = pl.Series.from_arrow(
    arrow_ffi.import_array_and_schema(result_array, result_schema))

lib.vc_destroy_context(ctx)

Operations

VcColumn *vc_filter(VcContext *ctx, VcColumn *data_col, VcColumn *mask_col, uint64_t *out_length)

Status: fully implemented.

Stream compaction (data[mask != 0]) via a 3-pass algorithm:

Pass 1 — GPU: bitmap + intra-workgroup Blelloch scan (filter_pass1.comp). 256 threads per workgroup. Each thread evaluates its mask element (0 or 1), then the workgroup runs a Blelloch exclusive prefix sum in 256 elements of shared memory (upsweep + downsweep). Output: per-element exclusive prefix (prefixes buffer) and per-workgroup total (totals buffer). Pipeline barrier ensures totals are host-readable after dispatch.

Pass 2 — CPU: workgroup-total exclusive scan. Maps the totals buffer, reads per-workgroup sums, computes an exclusive prefix sum serially, writes the per-workgroup base offsets to the offsets buffer. This is O(num_workgroups) — typically a few thousand iterations even for billion-element columns. The last running total is the output column length.

Pass 3 — GPU: scatter (filter_pass3.comp). Each thread reads its mask value and element index. If mask is non-zero, it computes scatter_pos = prefixes[gid] + offsets[wg] and writes data[gid] to output[scatter_pos].

Vulkan pipeline construction: one VkPipelineLayout with a 7-binding descriptor set layout (data, mask, prefixes, offsets, output, input validity, output validity) and a single push constant (element count + has_validity flag). Pass 1 uses bindings 0-3; pass 3 uses all 7 bindings. Both passes share the same layout.

Validity propagation: when the input column is nullable, pass3 scatters validity bits to the output using atomicOr. Each surviving element writes its validity bit to the corresponding output position — no contention since each output position is unique. The output validity buffer is zeroed via vkCmdFillBuffer before scatter.

Output: a new VcColumn whose buffer is the direct scatter output (device-local, STORAGE_BUFFER | TRANSFER_SRC). No host round-trip needed before chaining further GPU operations.

Edge cases

Zero-length input: returns NULL with *out_length = 0.
Mask type must be VC_DTYPE_U8.
Data and mask must have equal length.

Parameters:

ctx – GPU context.
data_col – Input data column.
mask_col – U8 mask column (non-zero = keep).
out_length – [out] number of surviving elements.

Returns:

A new filtered column, or NULL on failure.

VcColumn *vc_cast(VcContext *ctx, VcColumn *col, VcDtype target_dtype)

Status: fully implemented.

Element-wise type conversion between numeric data types. Supports all 10 VcDtype values as both source and target. Each element is processed independently — an element-wise parallel map.

Shader: type_cast.comp — a single GLSL compute shader that packs all type-pair logic with dispatch-time branching via push constants (src_dtype, dst_dtype). All data is read and written as raw uint arrays with custom pack/unpack helpers for sub-32-bit types. Intermediate conversions pass through float64 to preserve precision and avoid double rounding for integer-to-integer casts.

Vulkan pipeline construction: one VkPipelineLayout with a 2-binding descriptor set layout (source storage buffer at binding 0, destination at binding 1) and a single push constant block (element_count, src_dtype, dst_dtype).

Output: a new VcColumn with target_dtype, allocated via VMA with host access for readback. Data stays in VRAM for chaining further GPU operations.

Edge cases

Zero-length input: returns an empty column of the target dtype.
NaN input when casting to integer: clamped to 0.
Negative float to unsigned: clamped to 0.
Out-of-range float to integer: saturated to the target type’s min/max.
Identity cast (same source and target dtype): produces a copy with identical values.
Null context or column: returns NULL.

Parameters:

ctx – GPU context.
col – Input column.
target_dtype – Desired output data type.

Returns:

A new cast column, or NULL on failure.

int vc_reduce(VcContext *ctx, VcColumn *col, VcReduceOp op, uint32_t flags, VcScalar *out)

Status: fully implemented.

Reduce an entire column to a single scalar via a two-level parallel reduction: sum, min, max, mean, count, first, or last.

Shader: reduce.comp — shared-memory binary tree reduction. 256 threads per workgroup. Each thread loops over stride elements, accumulating into registers (value + auxiliary word). The tree reduces both words to 1 per WG, writing 4 u32 (val_lo, val_hi, aux_lo, aux_hi) to the intermediate buffer. A CPU-side pass merges per-WG results. Only the final 8-byte scalar crosses PCIe.

Vulkan pipeline construction: one VkPipelineLayout with a 3-binding descriptor set layout (input buffer, validity bitmap, output buffer) and a single push constant block (element count, op selector, dtype, skip-nulls flag, stride). The intermediate buffer is host-mapped via VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT for direct CPU read without staging copy.

Return types by operation

Edge cases

Zero-length input: returns identity value (0 for sum/count, 0.0 for mean, +inf or max integer for min, 0 for first/last).
VC_REDUCE_SKIP_NULLS: skips invalid elements. All-null input returns the identity value.
Single-workgroup columns: intermediate buffer has 1 entry; CPU pass reads it directly.
Multi-workgroup columns: CPU pass merges per-WG intermediate values in O(num_workgroups). The intermediate buffer is host-mapped to avoid a staging buffer and extra sync point.
FIRST/LAST: track the global element index alongside the value. Sentinels ensure unvisited threads don’t corrupt the reduction tree.

Parameters:

ctx – GPU context.
col – Input column.
op – VC_REDUCE_SUM, VC_REDUCE_MIN, VC_REDUCE_MAX, VC_REDUCE_MEAN, VC_REDUCE_COUNT, VC_REDUCE_FIRST, or VC_REDUCE_LAST.
flags – 0 or VC_REDUCE_SKIP_NULLS.
out – [out] Scalar result (dtype + union).

Returns:

0 on success, -1 on error.

Chained Execution

Operations are designed to chain without host round-trips. vc_filter returns a VcColumn* whose buffer is device-local with STORAGE_BUFFER | TRANSFER_SRC usage — immediately consumable by vc_cast or any future operation. The column handle carries metadata (length, dtype) that downstream operations read without GPU-to-host synchronization.

Chaining avoids the PCIe tax on intermediate results. Benchmark results on an RX 9070 (Debug build):

Scenario	64 KB	4 MB	64 MB
GPU chained (VRAM)	0.87 ms	3.35 ms	88.9 ms
GPU separate ops	—	5.70 ms	154 ms
CPU reference	1.38 ms	22.0 ms	331 ms

Scatter (filter) output feeds directly into an element-wise map (cast) without ever touching host RAM. The GPU beats CPU by 1.6–6.6x on chained operations.

If a vc_filter returns zero surviving elements, the returned column has length 0 and can still be passed to downstream operations (they will produce zero-length output). The caller is responsible for freeing every VcColumn* returned by an operation.

Utilities

uint32_t vc_dtype_size(VcDtype dtype)

Status: fully implemented.

Returns the size in bytes of the given data type (U8 = 1, U32 = 4, F64 = 8, etc.). Does not touch Vulkan — a pure lookup table.

VcColumn *vc_sort(VcContext *ctx, VcColumn *col)

Status: fully implemented.

Sorts a column in ascending order using LSD radix sort.

Vulkan primitives:

Histogram pass: dispatches num_wgs workgroups (256 threads each). Each WG computes a private 257-bin shared-memory histogram (0-255 digit bins + bin 256 for nulls), then writes the per-WG histogram to a host-visible STORAGE_BUFFER. The host reads the per-WG histogram and computes per-WG exclusive-scan offsets for each digit.
Reorder pass: dispatches num_wgs workgroups. Each WG processes its slice of the column (wg * 256 * stride elements), computes per-digit local ranks via an O(256) shared-memory scan, and writes each element to per_wg_base_offset[wg][digit] + local_rank in the output buffer.
Ping-pong buffers: two STORAGE_BUFFER allocations swapped each digit pass. No host round-trip during the sort loop.
Sub-u32 types (U8, I8, U16, I16) are widened to U32/I32 via a pre-processing compute pass before the radix sort, then repacked to the original type at the end.

Algorithm: LSD (least significant digit) radix sort. 4 passes for 32-bit dtypes (U8–F32), 8 passes for 64-bit (U64–F64). Each pass sorts by one byte. Signed types XOR-flip MSB at extraction; floats convert to total order via sign-bit complement.

Null handling: nulls are counted in bin 256. During reorder, null elements are placed after all valid elements. The output null count is the sum of ‘bin 256’ counts across all WGs.

Edge cases:

Zero-length input: returns an empty column of the same dtype.
Null context or column: returns NULL.
Sub-u32 types: transparently widened and repacked.
Already-sorted / reverse-sorted / duplicates: handled correctly.
Single-element / prime-length columns: handled correctly.
Nullable columns with mixed valid/null elements: nulls placed at end, valid elements sorted ascending.

Parameters:

ctx – GPU context.
col – Input column.

Returns:

A sorted column (same length and dtype), or NULL on error.

Gather (Indirect Indexing)

VcColumn *vc_gather(VcContext *ctx, VcColumn *col, VcColumn *indices)

Status: fully implemented.

Extract rows from a column by index. Each output element i is col[indices[i]]. This is the fundamental primitive for materializing join results and performing arbitrary row reordering.

Shader: gather.comp — single-pass parallel indirect copy. 256 threads per workgroup. Each thread reads indices[gid] and copies in_data[indices[gid]] to out_data[gid]. 64-bit columns use words=2 mode for paired uvec2 load/store. Zero arithmetic, zero divergence — purely memory-bandwidth-bound.

Vulkan primitives: one VkPipelineLayout with a 5-binding descriptor set layout (source data, source validity, indices, output data, output validity) and push constants (element count, words, source length, validity flags). Output validity buffer is pre-filled with 1s via vkCmdFillBuffer; null bits are cleared with atomicAnd to avoid data races.

Null handling: if the source element at a given index is null, the output element is null. Out-of-bounds indices produce null output elements (value zeroed, validity bit cleared).

Edge cases:

Indices must be VC_DTYPE_U32 (non-U32 returns NULL).
Zero-length indices: returns an empty column of the source dtype.
Empty source column: all output elements are null (all indices OOB).
Duplicate indices: the same source row appears multiple times in output (fully supported).

Parameters:

ctx – GPU context.
col – Source data column (any dtype).
indices – U32 index column.

Returns:

A new column of the same dtype as col, or NULL on failure.

type VcGroupByResult

Result of a vc_groupby call. Contains two columns: keys (unique group keys, same dtype as the original key column) and aggregates (per-group aggregate values, VC_DTYPE_F64).

VcColumn *keys

VcColumn *aggregates

VcGroupByResult *vc_groupby(VcContext *ctx, VcColumn *key_col, VcColumn *val_col, VcReduceOp op, uint32_t flags)

Status: fully implemented (all dtypes).

Groups rows by key and computes a per-group aggregate (sum, min, max, count, or mean). Keys must be sorted before reduction — vc_groupby internally invokes vc_sort on a packed (key, value) pair column. For MEAN, two GPU passes run internally (SUM + COUNT) and the result is divided on CPU — zero additional shader changes.

Vulkan primitives:

Pack-and-sort: Keys and values are packed into uint64_t pairs with a sort-key transform (I32 XOR MSB, F32 sign-bit flip) to ensure correct sort order. vc_sort (LSD radix sort, 4 shader passes per element on the GPU) produces a sorted uint64_t column, which is unpacked into interleaved [k0, v0, k1, v1, ...] in a device-local STORAGE_BUFFER.
Segment detection (groupby_segments.comp): Compares consecutive key words via atomicOr on a host-visible flag bitmask. One workgroup, 256 threads.
Per-WG reduction (groupby_reduce.comp): Each workgroup processes a 256-element slice. A Hillis-Steele inclusive prefix sum on segment flags assigns a 0-based segment ID to each lane. A tree reduction (8 strides, shared memory) accumulates values within each segment. Each segment leader writes a 4-word (key_lo, key_hi, agg_lo, agg_hi) to a host-visible STORAGE_BUFFER.
Merge pass (CPU): Adjacent partial results with the same key (from segments that cross workgroup boundaries) are merged. Keys in the output are in ascending order.
Hash-based path (VC_GROUPBY_HASH flag): Bypasses sort and segment detection when cardinality is below 256. The group_hash.comp shader builds a per-WG 256-entry hash table in shared memory (shared uint), with thread 0 processing elements sequentially to avoid LDS data races on RDNA. Each WG’s hash buckets are serialized to global memory, then CPU-sorted and merged across WGs.

Algorithm: Four-pass pipeline: pack + GPU sort -> segment boundary detection -> per-WG segment-local reduction -> CPU merge of boundary partials. Aggregates are always stored as F64 for sum, or as the promoted F64 equivalent of the value for min/max.

Edge cases:

Zero-length input: returns empty keys and aggregates columns.
64-bit key types: fully supported.
Single-element input: correct (one group, aggregate = value).
Cross-WG boundary segments: partial results from adjacent workgroups are CPU-merged.
Hash path (VC_GROUPBY_HASH): returns correct results for cardinality <= 256 per WG. Table-full condition returns a partial result (first 256 distinct keys only). Not suitable for high-cardinality data.
Null context, key column, or value column: returns NULL.
Key/value length mismatch or dtype mismatch: returns NULL.

Parameters:

ctx – GPU context.
key_col – Key column (U8-U32, I8-I32, F32).
val_col – Value column (same dtype as key).
op – VC_REDUCE_SUM, VC_REDUCE_MIN, VC_REDUCE_MAX, or VC_REDUCE_COUNT.
flags – 0, VC_REDUCE_SKIP_NULLS, or VC_GROUPBY_HASH.

Returns:

A VcGroupByResult with sorted unique keys and per-group aggregates, or NULL on error. Free with vc_groupby_result_destroy.

void vc_groupby_result_destroy(VcContext *ctx, VcGroupByResult *result): Frees a VcGroupByResult and its contained columns.

type VcJoinResult

Result of a join call. Contains two U32 columns: left_indices (row indices into the left key column) and right_indices (row indices into the right key column). For left joins, unmatched rows have 0xFFFFFFFF as the right index. For outer joins, unmatched right rows have 0xFFFFFFFF as the left index.

VcColumn *left_indices

VcColumn *right_indices

VcJoinResult *vc_inner_join(VcContext *ctx, VcColumn *left_key, VcColumn *right_key)

Status: fully implemented (all numeric key types, partitioned right tables).

Inner join on a single key column. Returns matching index pairs. Supports 32-bit (U32, I32, F32, U8, U16, I8, I16) and 64-bit (U64, I64, F64) key types. Right tables are automatically partitioned into shared-memory-sized chunks when exceeding ~2K keys (32-bit) or ~1K keys (64-bit).

Vulkan primitives:

Shared-memory hash table (join_hash.comp): Open addressing with MurmurHash3 32-bit/64-bit finalizer. Right keys inserted cooperatively into shared arrays. Left keys probed with linear search. Matches written via Hillis-Steele prefix sum scatter.
Per-workgroup dispatch: Each WG builds the right table in shared memory and probes its slice of the left table. Partitioning uses multiple sequential dispatches when the right table exceeds shared memory.
Match flags buffer (binding 4): atomicOr in shader tracks which left rows matched, used by left/outer join for null-filling unmatched rows.

Algorithm: Shared-memory hash join (build + probe in one kernel). 64-bit variant hashes both halves of the key with a mixed MurmurHash3 variant. Sort-key transforms applied for signed/float types to ensure hash determinism.

Edge cases:

Zero-length left or right: returns empty columns.
All numeric key types supported (U/I/F, 8-64 bit).
Duplicate keys: right-side first occurrence wins; left duplicates all match.
Null context or columns: returns NULL.
Dtype mismatch: returns NULL.

Parameters:

ctx – GPU context.
left_key – Left (probe-side) key column.
right_key – Right (build-side) key column (same dtype).

Returns:

A VcJoinResult with matching index pairs, or NULL on error. Free with vc_join_result_destroy.

VcJoinResult *vc_left_join(VcContext *ctx, VcColumn *left_key, VcColumn *right_key)

Status: fully implemented.

Left join. Returns all left rows; unmatched rows get 0xFFFFFFFF as the right index. Same key type support and partitioning as vc_inner_join.

VcJoinResult *vc_outer_join(VcContext *ctx, VcColumn *left_key, VcColumn *right_key)

Status: fully implemented.

Outer join. Returns all left rows (unmatched: 0xFFFFFFFF right index) followed by unmatched right rows (0xFFFFFFFF left index). Same key type support and partitioning as vc_inner_join.

void vc_join_result_destroy(VcContext *ctx, VcJoinResult *result): Frees a VcJoinResult and its contained columns.

Element-wise Arithmetic

enum VcArithOp

Arithmetic operation selector.

enumerator VC_ARITH_ADD = 0

enumerator VC_ARITH_SUB = 1

enumerator VC_ARITH_MUL = 2

enumerator VC_ARITH_DIV = 3

VcColumn *vc_arith(VcContext *ctx, VcColumn *lhs, VcColumn *rhs, VcArithOp op)

Status: implemented (>= 32-bit types only; U8/U16/I8/I16 require prior cast).

Element-wise arithmetic between two columns of the same dtype and length. Operates on 64-bit unsigned pairs in the shader — signed and float types are cast to unsigned before dispatch. Null propagation: if either operand is null, the result is null.

Shader: elementwise.comp — unified shader handling all arithmetic and comparison operations via a push-constant op selector. 256 threads per workgroup, each processes one element. 64-bit values use uvec2 pairs (lo, hi) with carry-chain arithmetic.

Vulkan primitives: one VkPipelineLayout with a 6-binding descriptor set layout (lhs data, rhs data, lhs validity, rhs validity, output data, output validity) and push constants (element count, op, words, scalar value, validity flags). Output buffers are VMA-allocated with VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE.

Edge cases:

Null context or columns: returns NULL.
Dtype or length mismatch: returns NULL.
Sub-32-bit types: returns NULL (cast to U32/I32 first).
Division by zero: result set to null.
Zero-length columns: returns empty column of the same dtype.

Parameters:

ctx – GPU context.
lhs – Left operand column.
rhs – Right operand column (same dtype and length).
op – Arithmetic operation.

Returns:

A new column with the result, or NULL on failure.

VcColumn *vc_arith_scalar(VcContext *ctx, VcColumn *lhs, VcScalar rhs, VcArithOp op, bool rhs_on_left)

Status: implemented (>= 32-bit types only; scalar always on right).

Element-wise arithmetic between a column and a scalar. The scalar value is passed via push constants (rhs_lo, rhs_hi) — no additional GPU buffer allocation. Null propagation: null elements in the column produce null results.

Limitation: rhs_on_left is accepted but not yet implemented — the scalar is always applied on the right side. This matters for non-commutative operations: scalar - col and scalar / col are not currently supported.

Parameters:

ctx – GPU context.
lhs – Left (column) operand.
rhs – Scalar value and dtype.
op – Arithmetic operation.
rhs_on_left – If true, scalar is on the left of the operator (not yet implemented).

Returns:

A new column, or NULL on failure.

Element-wise Comparison

enum VcCompareOp

Comparison operation selector.

enumerator VC_CMP_EQ = 0

enumerator VC_CMP_NEQ = 1

enumerator VC_CMP_LT = 2

enumerator VC_CMP_GT = 3

enumerator VC_CMP_LTE = 4

enumerator VC_CMP_GTE = 5

VcColumn *vc_compare(VcContext *ctx, VcColumn *lhs, VcColumn *rhs, VcCompareOp op, VcDtype mask_dtype)

Status: fully implemented.

Element-wise comparison between two columns of the same dtype and length. Returns a mask column where each element is 1 if the comparison is true, 0 if false. The mask column dtype must be VC_DTYPE_U32 or VC_DTYPE_U8 — U32 is the primary mask format used by vc_filter.

Shader: same elementwise.comp as arithmetic (op code 10–15). 64-bit comparisons (LT, GT, LTE, GTE) compare hi word first, then lo word on equality.

Edge cases:

Null context or columns: returns NULL.
Dtype or length mismatch: returns NULL.
Invalid mask_dtype: returns NULL.
Null propagation: null elements produce null results.

Parameters:

ctx – GPU context.
lhs – Left operand column.
rhs – Right operand column (same dtype and length).
op – Comparison operation.
mask_dtype – VC_DTYPE_U32 or VC_DTYPE_U8.

Returns:

A new mask column (indicated dtype), or NULL on failure.

VcColumn *vc_compare_scalar(VcContext *ctx, VcColumn *lhs, VcScalar rhs, VcCompareOp op, VcDtype mask_dtype, bool rhs_on_left)

Status: fully implemented (scalar always on right).

Element-wise comparison between a column and a scalar. Same semantics as vc_compare but the scalar is passed via push constants.

Parameters:

ctx – GPU context.
lhs – Left (column) operand.
rhs – Scalar value and dtype.
op – Comparison operation.
mask_dtype – VC_DTYPE_U32 or VC_DTYPE_U8.
rhs_on_left – If true, scalar is on the left (not yet implemented).

Returns:

A new mask column, or NULL on failure.

Null Operations

VcColumn *vc_is_null(VcContext *ctx, VcColumn *col)

Status: fully implemented.

Returns a U32 mask column where each element is 1 if the corresponding input element is null, 0 otherwise. For columns without a validity buffer, returns all zeros.

Parameters:

ctx – GPU context.
col – Input column.

Returns:

A U32 mask column, or NULL on failure.

VcColumn *vc_is_not_null(VcContext *ctx, VcColumn *col)

Status: fully implemented (via vc_is_null + host-side inversion).

Returns a U32 mask column where each element is 1 if the corresponding input element is valid, 0 if null.

Parameters:

ctx – GPU context.
col – Input column.

Returns:

A U32 mask column, or NULL on failure.

VcColumn *vc_fill_null(VcContext *ctx, VcColumn *col, VcScalar fill_value)

Status: fully implemented.

Returns a new column where null values are replaced by fill_value and all other values are copied through unchanged.

Parameters:

ctx – GPU context.
col – Input column (may be nullable).
fill_value – Scalar replacement for nulls.

Returns:

A new non-nullable column, or NULL on failure.

VcColumn *vc_drop_nulls(VcContext *ctx, VcColumn *col, uint64_t *out_length)

Status: fully implemented.

Removes all null elements from a column. Returns a new non-nullable column containing only valid elements in their original relative order. If the input has no validity bitmap or zero nulls, returns a copy.

Implementation: reads column data and validity bitmap to host, filters valid elements on CPU, creates a new column. No shader dispatch.

Parameters:

ctx – GPU context.
col – Input column (may be nullable).
out_length – [out] Number of elements in the result, or NULL.

Returns:

A new non-nullable column, or NULL on failure.

Unique / Distinct

VcColumn *vc_unique(VcContext *ctx, VcColumn *col)

Status: fully implemented.

Remove duplicate values from a column. Output is in sorted ascending order. NaN values are each treated as unique (NaN != NaN).

Algorithm: GPU sort via vc_sort (LSD radix sort, 4-8 passes on GPU), then host-side O(n) dedup on sorted data. The sort – the expensive part – stays on GPU. Host dedup is a single linear scan that runs in microseconds even for million-element columns.

Vulkan primitives: delegates to vc_sort. No custom shader is needed for this operation – the sorting shader handles all GPU work.

Edge cases

Zero-length input: returns an empty column of the same dtype.
Single-element input: returns a copy of the element.
All duplicates: returns a single-element column.
NaN handling: each NaN value is treated as a separate unique entry (consistent with IEEE 754 NaN != NaN).
Nullable columns: nulls sort to the end; all nulls collapse to a single null entry (only one null in the output).
Null context or column: returns NULL.

Parameters:

ctx – GPU context.
col – Input column.

Returns:

A new column with duplicates removed, or NULL on failure.

Boolean Mask Operations

VcColumn *vc_bool_and(VcContext *ctx, VcColumn *lhs, VcColumn *rhs)

Status: fully implemented.

Element-wise logical AND on two U32 or U8 mask columns. Non-zero values are treated as true, zero as false. Returns a U32 mask column (1 = true).

U8 inputs are host-side widened to U32 before dispatch.

Parameters:

ctx – GPU context.
lhs – Left mask column (U32 or U8).
rhs – Right mask column (same length).

Returns:

A U32 mask column, or NULL on failure.

VcColumn *vc_bool_or(VcContext *ctx, VcColumn *lhs, VcColumn *rhs)

Status: fully implemented.

Element-wise logical OR on two U32 or U8 mask columns.

Returns:: A U32 mask column, or NULL on failure.

VcColumn *vc_bool_not(VcContext *ctx, VcColumn *col)

Status: fully implemented.

Element-wise logical NOT on a single U32 or U8 mask column. Returns a U32 mask column where 0 becomes 1 and non-zero becomes 0.

Returns:: A U32 mask column, or NULL on failure.

List Column Operations

VcColumn *vc_create_list_column(VcContext *ctx, uint64_t num_rows, const uint32_t *offsets, VcColumn *child_col, const uint32_t *validity, uint64_t null_count)

Status: fully implemented. v0.2

Creates a list column from a child column and per-row offsets. Ownership of child_col transfers to the list column. Offsets use Arrow List layout: offsets[i+1] - offsets[i] elements for row i.

Returns:: A VcColumn* of dtype VC_DTYPE_LIST, or NULL on failure.

int vc_read_list_column(VcContext *ctx, VcColumn *col, uint32_t *host_offsets, VcColumn **out_child)

Status: fully implemented. v0.2

Reads a list column back to the host. host_offsets receives offsets, out_child receives a newly allocated child column.

VcColumn *vc_list_length(VcContext *ctx, VcColumn *col)

Status: fully implemented. v0.2

Per-row list length via GPU shader: offsets[i+1] - offsets[i]. Returns a VC_DTYPE_U32 column.

VcColumn *vc_list_get(VcContext *ctx, VcColumn *col, int64_t index)

Status: fully implemented. v0.2

Extract element at index from each list row. Positive indices count from start, negative from end. Bounds-clamped. Returns column matching the list’s child dtype.

VcColumn *vc_list_explode(VcContext *ctx, VcColumn *col)

Status: fully implemented. v0.2

Flatten a list column into a flat column. Each child element becomes one output row. Null list rows produce null elements.

VcColumn *vc_list_contains(VcContext *ctx, VcColumn *col, const void *value)

Status: fully implemented. v0.2

Check if a scalar value exists in each list row. Returns VC_DTYPE_U32 mask column (1 = contains, 0 = not).

VcColumn *vc_list_sort(VcContext *ctx, VcColumn *col)

Status: fully implemented. v0.2

Sort elements within each list row. Host-side per-subrange std::sort. Returns new list column with sorted subranges, same offsets structure.

VcColumn *vc_list_unique(VcContext *ctx, VcColumn *col)

Status: fully implemented. v0.2

Remove duplicate values within each list row. Sorts subranges then deduplicates adjacent elements. Offsets may shrink.

Struct Column Operations

VcColumn *vc_create_struct_column(VcContext *ctx, uint64_t num_rows, VcColumn *const *fields, uint32_t field_count)

Status: fully implemented. v0.2

Creates a struct column from an array of child columns. All children must have the same length. Ownership transfers — children must not be destroyed separately.

VcColumn *vc_struct_field(VcContext *ctx, VcColumn *col, uint32_t field_index)

Status: fully implemented. v0.2

Returns a non-owning pointer to the field at field_index. The returned column must not be freed.

uint32_t vc_struct_field_count(VcColumn *col)

Status: fully implemented. v0.2

Returns the number of fields in a struct column.

VcColumn *vc_struct_rename_fields(VcContext *ctx, VcColumn *col, const char *const *names, uint32_t count)

Status: fully implemented. v0.2

Rename struct fields. Returns a new struct column sharing the same child data with updated field names.

VcColumn *vc_struct_with_fields(VcContext *ctx, VcColumn *col, VcColumn *const *new_fields, const char *const *names, uint32_t count)

Status: fully implemented. v0.2

Replace or add fields to a struct. Returns a new struct column. Unchanged children are shared; replaced or added children transfer ownership.