API Reference
=============

This documents the public C ABI. Every function maps Vulkan concepts to columnar
data operations.  Opaque handles keep driver internals out of the public surface.

----

Core Types
----------

.. c:enum:: VcDtype

   Column data type enumeration.  Corresponds to GPU-side ``uint`` / ``float``
   types in GLSL shaders.

   .. c:enumerator:: VC_DTYPE_U8  = 0
   .. c:enumerator:: VC_DTYPE_U16 = 1
   .. c:enumerator:: VC_DTYPE_U32 = 2
   .. c:enumerator:: VC_DTYPE_U64 = 3
   .. c:enumerator:: VC_DTYPE_I8  = 4
   .. c:enumerator:: VC_DTYPE_I16 = 5
   .. c:enumerator:: VC_DTYPE_I32 = 6
   .. c:enumerator:: VC_DTYPE_I64 = 7
   .. c:enumerator:: VC_DTYPE_F32 = 8
   .. c:enumerator:: VC_DTYPE_F64 = 9
   .. c:enumerator:: VC_DTYPE_STRING = 10
   .. c:enumerator:: VC_DTYPE_LIST = 11
   .. c:enumerator:: VC_DTYPE_STRUCT = 12

.. c:type:: VcContext

   Opaque handle to a GPU context.  Internally owns a :c:type:`VkInstance`,
   :c:type:`VkPhysicalDevice`, :c:type:`VkDevice`, one compute
   :c:type:`VkQueue`, a :c:type:`VkCommandPool`, and a :c:type:`VmaAllocator`.

.. c:type:: VcColumn

   Opaque handle to a typed column buffer on the GPU.  Backed by a
   :c:type:`VkBuffer` allocated through VMA with
   ``VK_BUFFER_USAGE_STORAGE_BUFFER_BIT`` so compute shaders can read and write
   it directly.

----

Context Lifecycle
-----------------

.. c:function:: VcContext* vc_create_context(void)

   **Status:** fully implemented.

   Creates a Vulkan instance (``VK_API_VERSION_1_3``), selects a physical
   device (preferring AMD ``vendorID == 0x1002``, falling back to any
   compute-queue GPU), creates a logical device with one compute queue, a VMA
   allocator, and a command pool with ``VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT``.

   In debug builds (``_DEBUG`` defined), attempts to enable
   ``VK_LAYER_KHRONOS_validation`` and ``VK_EXT_DEBUG_UTILS_EXTENSION_NAME``.
   Gracefully falls back if the layer is not installed.

   :returns: A new context, or ``NULL`` on failure.

.. c:function:: void vc_destroy_context(VcContext* ctx)

   **Status:** fully implemented.

   Tears down the command pool, VMA allocator, logical device, and instance
   in reverse creation order.  Passing ``NULL`` is a no-op.

----

Column Management
-----------------

.. c:function:: VcColumn* vc_create_column(VcContext* ctx, VcDtype dtype, uint64_t length, const void* host_data)

   **Status:** fully implemented.

   Allocates a :c:type:`VkBuffer` through VMA with
   ``VK_BUFFER_USAGE_STORAGE_BUFFER_BIT  | VK_BUFFER_USAGE_TRANSFER_SRC_BIT |
   VK_BUFFER_USAGE_TRANSFER_DST_BIT`` (shader access + transfer source/target
   for host staging).  Uses ``VMA_MEMORY_USAGE_AUTO`` so the driver picks
   device-local or host-visible memory as appropriate.

   If ``host_data`` is provided, creates a transient staging buffer
   (``TRANSFER_SRC``, ``HOST_ACCESS_SEQUENTIAL_WRITE | CREATE_MAPPED``),
   memcpy's the data into it, issues ``vkCmdCopyBuffer`` from staging to the
   device-local buffer, inserts a ``TRANSFER_WRITE → SHADER_READ`` pipeline
   barrier, submits, and waits.

   Edge case: zero-length columns allocate a minimal 4-byte buffer to keep
   Vulkan happy.  Large no-data allocations (>64 MiB) also use the minimal
   allocation — the logical length is stored but the backing buffer is
   placeholder-sized.

   :param ctx: GPU context.
   :param dtype: Element data type.
   :param length: Number of elements.
   :param host_data: Host data to upload, or ``NULL`` to leave uninitialized.

   :returns: A new column, or ``NULL`` on failure.

.. c:function:: void vc_destroy_column(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented.

   Calls ``vmaDestroyBuffer`` and frees the column struct.  Passing ``NULL``
   for either argument is a no-op.

.. c:function:: int vc_read_column(VcContext* ctx, VcColumn* col, void* host_buffer, uint64_t host_buffer_size)

   **Status:** fully implemented.

   Reads GPU column data back to the host through a staging buffer:

   1. Allocates a temporary staging buffer (``TRANSFER_DST``,
      ``HOST_ACCESS_RANDOM | CREATE_MAPPED``).
   2. Inserts a ``SHADER_WRITE → TRANSFER_READ`` barrier on the column buffer
      (ensures any prior compute writes are visible to the transfer engine).
   3. Issues ``vkCmdCopyBuffer`` from the column buffer to the staging buffer.
   4. Inserts a ``TRANSFER_WRITE → HOST_READ`` barrier on the staging buffer.
   5. Submits, waits, memcpy's from staging mapped memory to ``host_buffer``.

   Zero-length columns return success immediately.

   :param ctx: GPU context.
   :param col: Column to read.
   :param host_buffer: Destination buffer on the host.
   :param host_buffer_size: Size of the destination buffer in bytes.

   :returns: ``0`` on success, ``-1`` on error (e.g., buffer too small).

.. c:function:: uint64_t vc_column_length(VcColumn* col)

   **Status:** fully implemented.  Returns the number of elements.  Returns ``0``
   if ``col`` is ``NULL``.

.. c:function:: VcDtype vc_column_dtype(VcColumn* col)

   **Status:** fully implemented.  Returns the element data type.  Returns
   ``VC_DTYPE_U8`` if ``col`` is ``NULL``.

----

Null Support (Arrow Validity Bitmaps)
--------------------------------------

The library supports nullable columns through Arrow-compatible validity
bitmaps.  A validity bitmap is a packed array of ``uint32_t`` words, one
bit per element, LSB-first per word.  A set bit means the element is
valid; a cleared bit means null.  The bitmap length is ``ceil(N/32)``
words for a column with ``N`` elements.

``VcColumn`` maps 1:1 to an Arrow ``PrimitiveArray``: ``validity_buffer``
is Arrow buffers[0] and ``buffer`` is Arrow buffers[1].  This enables
zero-copy wrappers in Polars, DataFusion, or PySpark bindings.

.. c:function:: VcColumn* vc_create_column_nullable(VcContext* ctx, VcDtype dtype, uint64_t length, const void* host_data, const uint32_t* validity, uint64_t null_count)

   **Status:** fully implemented.

   Creates a nullable column.  Behaves identically to
   :c:func:`vc_create_column` for the data buffer, then additionally
   allocates and uploads the validity bitmap if ``validity`` is non-NULL
   and ``null_count > 0``.

   **Vulkan primitives:** allocates a ``STORAGE_BUFFER | TRANSFER_DST``
   buffer for the validity bitmap through VMA, copies the host-side
   bitmap via a transient staging buffer and ``vkCmdCopyBuffer``, then
   inserts a ``TRANSFER_WRITE -> SHADER_READ`` barrier.

   Edge case: if ``validity`` is ``NULL`` or ``null_count == 0``, the
   column is created without a validity buffer (i.e., all elements are
   considered valid).

   :param ctx: GPU context.
   :param dtype: Element data type.
   :param length: Number of elements.
   :param host_data: Host data to upload, or ``NULL``.
   :param validity: Arrow-format validity bitmap (packed uint32), or ``NULL``.
   :param null_count: Number of null elements.

   :returns: A new nullable column, or ``NULL`` on failure.

.. c:function:: int vc_column_has_validity(VcColumn* col)

   **Status:** fully implemented.

   Returns ``1`` if the column has a validity buffer, ``0`` otherwise.
   Returns ``0`` if ``col`` is ``NULL``.

.. c:function:: uint64_t vc_column_null_count(VcColumn* col)

   **Status:** fully implemented.

   Returns the null count set at column creation time.  Note: operations
   that remove elements (``vc_filter``) do not recompute the null count;
   the value is preserved from the input column.  A future compute-shader
   pass can recompute it from the output validity buffer.

   Returns ``0`` if ``col`` is ``NULL``.

.. c:function:: int vc_read_column_validity(VcContext* ctx, VcColumn* col, void* host_buffer, uint64_t host_buffer_size, uint32_t* validity_out)

   **Status:** fully implemented.

   Reads both column data and validity bitmap back to the host.  First
   calls :c:func:`vc_read_column` for the data, then performs a
   ``vkCmdCopyBuffer`` from the validity buffer to a transient staging
   buffer and memcpy's the bitmap to ``validity_out``.

   If the column has no validity buffer, ``validity_out`` is left
   untouched.  The caller must provide a ``validity_out`` buffer of at
   least ``ceil(col->length / 32)`` ``uint32_t`` words.

   :param ctx: GPU context.
   :param col: Column to read.
   :param host_buffer: Destination buffer for data (passed to ``vc_read_column``).
   :param host_buffer_size: Size of the data destination buffer.
   :param validity_out: Destination buffer for validity bitmap.

   :returns: ``0`` on success, ``-1`` on error.

----

Arrow C Data Interface
-----------------------

The library implements the `Apache Arrow C Data Interface
<https://arrow.apache.org/docs/format/CDataInterface.html>`_ for
zero-copy interop with Polars, DataFusion, Spark, DuckDB, Velox, and
any other framework that speaks Arrow.  ``VcColumn`` maps 1:1 to an
Arrow ``PrimitiveArray`` (buffers[0] = validity, buffers[1] = data).

.. c:function:: VcColumn* vc_import_arrow(VcContext* ctx, VcArrowArray* array, VcArrowSchema* schema)

   **Status:** fully implemented.

   Import an Arrow array into a ``VcColumn``.  The data and validity
   buffers are copied from host to GPU via staging buffers and
   ``vkCmdCopyBuffer``.  The Arrow structs must remain valid until
   the call returns.

   The Arrow C Data Interface stores buffer pointers inline, immediately
   after the ``VcArrowArray`` struct (not in ``private_data``).  Use
   ``reinterpret_cast<const void**>(&aa + 1)`` to access them.

   Constraints: ``array->n_buffers >= 2``, ``array->offset == 0``
   (sliced arrays not supported).  Supports primitive numeric types
   (``C/S/I/L/c/s/i/l/f/g``), strings (``u``/``U``), lists (``+l``),
   and structs (``+s``).

   .. code-block:: c

      VcArrowArray aa;
      VcArrowSchema as;
      // ... framework populates aa and as ...
      VcColumn* col = vc_import_arrow(ctx, &aa, &as);
      // ... run GPU operations on col ...
      vc_destroy_column(ctx, col);

   :param ctx: GPU context.
   :param array: Arrow array to import.
   :param schema: Arrow schema describing the data type.

   :returns: A new ``VcColumn`` on GPU, or ``NULL`` on failure.

.. c:function:: int vc_export_arrow(VcContext* ctx, VcColumn* col, VcArrowArray* out_array, VcArrowSchema* out_schema)

   **Status:** fully implemented.

   Export a ``VcColumn`` to Arrow-compatible host buffers.  Reads GPU
   data and validity (if present) back to host via staging buffers.
   Fills ``out_array`` and ``out_schema`` with buffer pointers and
   type metadata.

   The caller must call ``out_array->release(out_array)`` and
   ``out_schema->release(out_schema)`` to free the exported host
   buffers when done.  The structs themselves are not freed — the
   caller owns them (typically stack-allocated).

   .. code-block:: c

      VcArrowArray aa;
      VcArrowSchema as;
      vc_export_arrow(ctx, col, &aa, &as);
      // buffers are inline at (const void**)(&aa + 1)
      // ... consume the Arrow array ...
      aa.release(&aa);
      as.release(&as);

   :param ctx: GPU context.
   :param col: Column to export.
   :param out_array: [out] Populated Arrow array struct.
   :param out_schema: [out] Populated Arrow schema struct.

   :returns: ``0`` on success, ``-1`` on error.

Integration Example: Polars in Python
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: python

   import ctypes
   import polars as pl
   from pyarrow.cffi import ffi as arrow_ffi

   lib = ctypes.CDLL("libvulkan_columnar.so")
   ctx = lib.vc_create_context()

   # Export Polars Series as Arrow C Data Interface structs.
   series = pl.Series("vals", [1, 2, None, 4, 5], dtype=pl.UInt32)
   arrow_array = series.to_arrow()
   c_array_ptr, c_schema_ptr = arrow_ffi.export_array_and_schema(
       arrow_array, arrow_array.type)

   # Import to GPU.
   col = lib.vc_import_arrow(ctx, c_array_ptr, c_schema_ptr)

   # Run GPU operations.
   mask_col = lib.vc_create_column(ctx, 0, ...)  # U8 mask
   filtered = lib.vc_filter(ctx, col, mask_col, byref(out_len))

   # Export back.
   result_array = ffi.new("VcArrowArray*")
   result_schema = ffi.new("VcArrowSchema*")
   lib.vc_export_arrow(ctx, filtered, result_array, result_schema)

   # Import into Polars.
   result_series = pl.Series.from_arrow(
       arrow_ffi.import_array_and_schema(result_array, result_schema))

   lib.vc_destroy_context(ctx)

----

Operations
----------

.. c:function:: VcColumn* vc_filter(VcContext* ctx, VcColumn* data_col, VcColumn* mask_col, uint64_t* out_length)

   **Status:** fully implemented.

   Stream compaction (``data[mask != 0]``) via a 3-pass algorithm:

   **Pass 1 — GPU: bitmap + intra-workgroup Blelloch scan** (``filter_pass1.comp``).
   256 threads per workgroup.  Each thread evaluates its mask element (``0`` or
   ``1``), then the workgroup runs a Blelloch exclusive prefix sum in 256
   elements of shared memory (upsweep + downsweep).  Output: per-element
   exclusive prefix (``prefixes`` buffer) and per-workgroup total (``totals``
   buffer).  Pipeline barrier ensures totals are host-readable after dispatch.

   **Pass 2 — CPU: workgroup-total exclusive scan.**
   Maps the ``totals`` buffer, reads per-workgroup sums, computes an exclusive
   prefix sum serially, writes the per-workgroup base offsets to the
   ``offsets`` buffer.  This is ``O(num_workgroups)`` — typically a few
   thousand iterations even for billion-element columns.  The last running
   total is the output column length.

   **Pass 3 — GPU: scatter** (``filter_pass3.comp``).
   Each thread reads its mask value and element index.  If mask is non-zero, it
   computes ``scatter_pos = prefixes[gid] + offsets[wg]`` and writes
   ``data[gid]`` to ``output[scatter_pos]``.

   **Vulkan pipeline construction:** one ``VkPipelineLayout`` with a
   7-binding descriptor set layout (data, mask, prefixes, offsets, output,
   input validity, output validity) and a single push constant (element
   count + has_validity flag).  Pass 1 uses bindings 0-3; pass 3 uses all
   7 bindings.  Both passes share the same layout.

   **Validity propagation:** when the input column is nullable, pass3
   scatters validity bits to the output using ``atomicOr``.  Each
   surviving element writes its validity bit to the corresponding output
   position — no contention since each output position is unique.  The
   output validity buffer is zeroed via ``vkCmdFillBuffer`` before
   scatter.

   **Output:** a new ``VcColumn`` whose buffer is the direct scatter output
   (device-local, ``STORAGE_BUFFER | TRANSFER_SRC``).  No host round-trip
   needed before chaining further GPU operations.

   .. rubric:: Edge cases

   - Zero-length input: returns ``NULL`` with ``*out_length = 0``.
   - Mask type must be ``VC_DTYPE_U8``.
   - Data and mask must have equal length.

   :param ctx: GPU context.
   :param data_col: Input data column.
   :param mask_col: ``U8`` mask column (non-zero = keep).
   :param out_length: [out] number of surviving elements.

   :returns: A new filtered column, or ``NULL`` on failure.

.. c:function:: VcColumn* vc_cast(VcContext* ctx, VcColumn* col, VcDtype target_dtype)

   **Status:** fully implemented.

   Element-wise type conversion between numeric data types.  Supports all
   10 ``VcDtype`` values as both source and target.  Each element is
   processed independently — an element-wise parallel map.

   **Shader:** ``type_cast.comp`` — a single GLSL compute shader that packs
   all type-pair logic with dispatch-time branching via push constants
   (``src_dtype``, ``dst_dtype``).  All data is read and written as raw
   ``uint`` arrays with custom pack/unpack helpers for sub-32-bit types.
   Intermediate conversions pass through ``float64`` to preserve precision
   and avoid double rounding for integer-to-integer casts.

   **Vulkan pipeline construction:** one ``VkPipelineLayout`` with a
   2-binding descriptor set layout (source storage buffer at binding 0,
   destination at binding 1) and a single push constant block
   (``element_count``, ``src_dtype``, ``dst_dtype``).

   **Output:** a new ``VcColumn`` with ``target_dtype``, allocated via VMA
   with host access for readback.  Data stays in VRAM for chaining further
   GPU operations.

   .. rubric:: Edge cases

   - Zero-length input: returns an empty column of the target dtype.
   - NaN input when casting to integer: clamped to 0.
   - Negative float to unsigned: clamped to 0.
   - Out-of-range float to integer: saturated to the target type's min/max.
   - Identity cast (same source and target dtype): produces a copy with
     identical values.
   - Null context or column: returns ``NULL``.

   :param ctx: GPU context.
   :param col: Input column.
   :param target_dtype: Desired output data type.

   :returns: A new cast column, or ``NULL`` on failure.

.. c:function:: int vc_reduce(VcContext* ctx, VcColumn* col, VcReduceOp op, uint32_t flags, VcScalar* out)

   **Status:** fully implemented.

   Reduce an entire column to a single scalar via a two-level parallel
   reduction: sum, min, max, mean, count, first, or last.

   **Shader:** ``reduce.comp`` — shared-memory binary tree reduction.
   256 threads per workgroup.  Each thread loops over ``stride`` elements,
   accumulating into registers (value + auxiliary word).  The tree reduces
   both words to 1 per WG, writing 4 u32 (val_lo, val_hi, aux_lo, aux_hi)
   to the intermediate buffer.  A CPU-side pass merges per-WG results.
   Only the final 8-byte scalar crosses PCIe.

   **Vulkan pipeline construction:** one ``VkPipelineLayout`` with a
   3-binding descriptor set layout (input buffer, validity bitmap,
   output buffer) and a single push constant block (element count, op
   selector, dtype, skip-nulls flag, stride).  The intermediate buffer
   is host-mapped via ``VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT``
   for direct CPU read without staging copy.

   .. rubric:: Return types by operation

   +-------------+---------------------------------------------+
   | Op          | ``VcScalar.dtype``                          |
   +=============+=============================================+
   | ``SUM``     | ``U64`` (unsigned), ``I64`` (signed),       |
   |             | ``F64`` (float). Overflow documented.       |
   +-------------+---------------------------------------------+
   | ``MIN``     | Same as input column dtype.                  |
   +-------------+---------------------------------------------+
   | ``MAX``     | Same as input column dtype.                  |
   +-------------+---------------------------------------------+
   | ``MEAN``    | Always ``F64``, regardless of input dtype.   |
   +-------------+---------------------------------------------+
   | ``COUNT``   | ``U64``.                                     |
   +-------------+---------------------------------------------+
   | ``FIRST``   | Same as input column dtype.                  |
   +-------------+---------------------------------------------+
   | ``LAST``    | Same as input column dtype.                  |
   +-------------+---------------------------------------------+

   .. rubric:: Edge cases

   - Zero-length input: returns identity value (0 for sum/count, 0.0 for
     mean, ``+inf`` or max integer for min, 0 for first/last).
   - ``VC_REDUCE_SKIP_NULLS``: skips invalid elements.  All-null input
     returns the identity value.
   - Single-workgroup columns: intermediate buffer has 1 entry;
     CPU pass reads it directly.
   - Multi-workgroup columns: CPU pass merges per-WG intermediate values
     in O(num_workgroups).  The intermediate buffer is host-mapped to
     avoid a staging buffer and extra sync point.
   - FIRST/LAST: track the global element index alongside the value.
     Sentinels ensure unvisited threads don't corrupt the reduction tree.

   :param ctx: GPU context.
   :param col: Input column.
   :param op: ``VC_REDUCE_SUM``, ``VC_REDUCE_MIN``, ``VC_REDUCE_MAX``,
              ``VC_REDUCE_MEAN``, ``VC_REDUCE_COUNT``, ``VC_REDUCE_FIRST``,
              or ``VC_REDUCE_LAST``.
   :param flags: ``0`` or ``VC_REDUCE_SKIP_NULLS``.
   :param out: [out] Scalar result (dtype + union).

   :returns: ``0`` on success, ``-1`` on error.

----

Chained Execution
-----------------

Operations are designed to chain without host round-trips.  ``vc_filter``
returns a ``VcColumn*`` whose buffer is device-local with
``STORAGE_BUFFER | TRANSFER_SRC`` usage — immediately consumable by
``vc_cast`` or any future operation.  The column handle carries metadata
(length, dtype) that downstream operations read without GPU-to-host
synchronization.

Chaining avoids the PCIe tax on intermediate results.  Benchmark results
on an RX 9070 (Debug build):

+---------------------+----------+-----------+-----------+
| Scenario            | 64 KB    | 4 MB      | 64 MB     |
+=====================+==========+===========+===========+
| GPU chained (VRAM)  | 0.87 ms  | 3.35 ms   | 88.9 ms   |
+---------------------+----------+-----------+-----------+
| GPU separate ops    | —        | 5.70 ms   | 154 ms    |
+---------------------+----------+-----------+-----------+
| CPU reference       | 1.38 ms  | 22.0 ms   | 331 ms    |
+---------------------+----------+-----------+-----------+

Scatter (filter) output feeds directly into an element-wise map (cast)
without ever touching host RAM.  The GPU beats CPU by 1.6--6.6x on
chained operations.

If a ``vc_filter`` returns zero surviving elements, the returned column
has length 0 and can still be passed to downstream operations (they will
produce zero-length output).  The caller is responsible for freeing every
``VcColumn*`` returned by an operation.

----

Utilities
---------

.. c:function:: uint32_t vc_dtype_size(VcDtype dtype)

   **Status:** fully implemented.

   Returns the size in bytes of the given data type (``U8`` = 1, ``U32`` = 4,
   ``F64`` = 8, etc.).  Does not touch Vulkan — a pure lookup table.


.. c:function:: VcColumn* vc_sort(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented.

   Sorts a column in ascending order using LSD radix sort.

   **Vulkan primitives:**

   - **Histogram pass:** dispatches ``num_wgs`` workgroups (256 threads each).
     Each WG computes a private 257-bin shared-memory histogram (0-255 digit
     bins + bin 256 for nulls), then writes the per-WG histogram to a
     host-visible ``STORAGE_BUFFER``.  The host reads the per-WG histogram
     and computes per-WG exclusive-scan offsets for each digit.
   - **Reorder pass:** dispatches ``num_wgs`` workgroups.  Each WG processes
     its slice of the column (``wg * 256 * stride`` elements), computes
     per-digit local ranks via an O(256) shared-memory scan, and writes each
     element to ``per_wg_base_offset[wg][digit] + local_rank`` in the output
     buffer.
   - **Ping-pong buffers:** two ``STORAGE_BUFFER`` allocations swapped each
     digit pass.  No host round-trip during the sort loop.
   - **Sub-u32 types** (U8, I8, U16, I16) are widened to U32/I32 via a
     pre-processing compute pass before the radix sort, then repacked to the
     original type at the end.

   **Algorithm:** LSD (least significant digit) radix sort.  4 passes for
   32-bit dtypes (U8--F32), 8 passes for 64-bit (U64--F64).  Each pass sorts
   by one byte.  Signed types XOR-flip MSB at extraction; floats convert to
   total order via sign-bit complement.

   **Null handling:** nulls are counted in bin 256.  During reorder, null
   elements are placed after all valid elements.  The output null count is
   the sum of 'bin 256' counts across all WGs.

   **Edge cases:**

   - Zero-length input: returns an empty column of the same dtype.
   - Null context or column: returns ``NULL``.
   - Sub-u32 types: transparently widened and repacked.
   - Already-sorted / reverse-sorted / duplicates: handled correctly.
   - Single-element / prime-length columns: handled correctly.
   - Nullable columns with mixed valid/null elements: nulls placed at end,
     valid elements sorted ascending.

   :param ctx: GPU context.
   :param col: Input column.

   :returns: A sorted column (same length and dtype), or ``NULL`` on error.

----

Gather (Indirect Indexing)
--------------------------

.. c:function:: VcColumn* vc_gather(VcContext* ctx, VcColumn* col, VcColumn* indices)

   **Status:** fully implemented.

   Extract rows from a column by index.  Each output element i is
   ``col[indices[i]]``.  This is the fundamental primitive for
   materializing join results and performing arbitrary row reordering.

   **Shader:** ``gather.comp`` — single-pass parallel indirect copy.
   256 threads per workgroup.  Each thread reads ``indices[gid]`` and
   copies ``in_data[indices[gid]]`` to ``out_data[gid]``.  64-bit columns
   use ``words=2`` mode for paired ``uvec2`` load/store.  Zero arithmetic,
   zero divergence — purely memory-bandwidth-bound.

   **Vulkan primitives:** one ``VkPipelineLayout`` with a 5-binding
   descriptor set layout (source data, source validity, indices, output
   data, output validity) and push constants (element count, words, source
   length, validity flags).  Output validity buffer is pre-filled with 1s
   via ``vkCmdFillBuffer``; null bits are cleared with ``atomicAnd`` to
   avoid data races.

   **Null handling:** if the source element at a given index is null,
   the output element is null.  Out-of-bounds indices produce null output
   elements (value zeroed, validity bit cleared).

   **Edge cases:**

   - Indices must be ``VC_DTYPE_U32`` (non-U32 returns ``NULL``).
   - Zero-length indices: returns an empty column of the source dtype.
   - Empty source column: all output elements are null (all indices OOB).
   - Duplicate indices: the same source row appears multiple times in
     output (fully supported).

   :param ctx: GPU context.
   :param col: Source data column (any dtype).
   :param indices: U32 index column.

   :returns: A new column of the same dtype as ``col``, or ``NULL`` on failure.


.. c:type:: VcGroupByResult

   Result of a ``vc_groupby`` call.  Contains two columns: ``keys``
   (unique group keys, same dtype as the original key column) and
   ``aggregates`` (per-group aggregate values, ``VC_DTYPE_F64``).

   .. c:member:: VcColumn* keys
   .. c:member:: VcColumn* aggregates


.. c:function:: VcGroupByResult* vc_groupby(VcContext* ctx, VcColumn* key_col, VcColumn* val_col, VcReduceOp op, uint32_t flags)

   **Status:** fully implemented (all dtypes).

   Groups rows by key and computes a per-group aggregate (sum, min, max,
   count, or mean).  Keys must be sorted before reduction — ``vc_groupby``
   internally invokes ``vc_sort`` on a packed ``(key, value)`` pair column.
   For ``MEAN``, two GPU passes run internally (``SUM`` + ``COUNT``) and
   the result is divided on CPU — zero additional shader changes.

   **Vulkan primitives:**

   - **Pack-and-sort:** Keys and values are packed into ``uint64_t`` pairs
     with a sort-key transform (I32 XOR MSB, F32 sign-bit flip) to ensure
     correct sort order.  ``vc_sort`` (LSD radix sort, 4 shader passes per
     element on the GPU) produces a sorted ``uint64_t`` column, which is
     unpacked into interleaved ``[k0, v0, k1, v1, ...]`` in a device-local
     ``STORAGE_BUFFER``.
   - **Segment detection** (``groupby_segments.comp``): Compares consecutive
     key words via ``atomicOr`` on a host-visible flag bitmask.
     One workgroup, 256 threads.
   - **Per-WG reduction** (``groupby_reduce.comp``): Each workgroup processes
     a 256-element slice.  A Hillis-Steele inclusive prefix sum on segment
     flags assigns a 0-based segment ID to each lane.  A tree reduction
     (8 strides, ``shared`` memory) accumulates values within each segment.
     Each segment leader writes a 4-word ``(key_lo, key_hi, agg_lo, agg_hi)``
     to a host-visible ``STORAGE_BUFFER``.
   - **Merge pass (CPU):** Adjacent partial results with the same key (from
     segments that cross workgroup boundaries) are merged.  Keys in the
     output are in ascending order.
   - **Hash-based path** (``VC_GROUPBY_HASH`` flag): Bypasses sort and segment
     detection when cardinality is below 256.  The ``group_hash.comp`` shader
     builds a per-WG 256-entry hash table in shared memory (``shared uint``),
     with thread 0 processing elements sequentially to avoid LDS data races
     on RDNA.  Each WG's hash buckets are serialized to global memory, then
     CPU-sorted and merged across WGs.

   **Algorithm:** Four-pass pipeline: pack + GPU sort -> segment boundary
   detection -> per-WG segment-local reduction -> CPU merge of boundary
   partials.  Aggregates are always stored as ``F64`` for sum, or as the
   promoted ``F64`` equivalent of the value for min/max.

   **Edge cases:**

   - Zero-length input: returns empty keys and aggregates columns.
   - 64-bit key types: fully supported.
   - Single-element input: correct (one group, aggregate = value).
   - Cross-WG boundary segments: partial results from adjacent workgroups
     are CPU-merged.
   - Hash path (``VC_GROUPBY_HASH``): returns correct results for
     cardinality <= 256 per WG.  Table-full condition returns a partial
     result (first 256 distinct keys only).  Not suitable for
     high-cardinality data.
   - Null context, key column, or value column: returns ``NULL``.
   - Key/value length mismatch or dtype mismatch: returns ``NULL``.

   :param ctx: GPU context.
   :param key_col: Key column (U8-U32, I8-I32, F32).
   :param val_col: Value column (same dtype as key).
   :param op: ``VC_REDUCE_SUM``, ``VC_REDUCE_MIN``, ``VC_REDUCE_MAX``,
             or ``VC_REDUCE_COUNT``.
   :param flags: ``0``, ``VC_REDUCE_SKIP_NULLS``, or ``VC_GROUPBY_HASH``.

   :returns: A ``VcGroupByResult`` with sorted unique keys and per-group
             aggregates, or ``NULL`` on error.  Free with
             ``vc_groupby_result_destroy``.


.. c:function:: void vc_groupby_result_destroy(VcContext* ctx, VcGroupByResult* result)

   Frees a ``VcGroupByResult`` and its contained columns.


.. c:type:: VcJoinResult

   Result of a join call.  Contains two ``U32`` columns:
   ``left_indices`` (row indices into the left key column) and
   ``right_indices`` (row indices into the right key column).
   For left joins, unmatched rows have ``0xFFFFFFFF`` as the right index.
   For outer joins, unmatched right rows have ``0xFFFFFFFF`` as the left
   index.

   .. c:member:: VcColumn* left_indices
   .. c:member:: VcColumn* right_indices


.. c:function:: VcJoinResult* vc_inner_join(VcContext* ctx, VcColumn* left_key, VcColumn* right_key)

   **Status:** fully implemented (all numeric key types, partitioned right tables).

   Inner join on a single key column.  Returns matching index pairs.
   Supports 32-bit (U32, I32, F32, U8, U16, I8, I16) and 64-bit
   (U64, I64, F64) key types.  Right tables are automatically partitioned
   into shared-memory-sized chunks when exceeding ~2K keys (32-bit) or
   ~1K keys (64-bit).

   **Vulkan primitives:**

   - **Shared-memory hash table** (``join_hash.comp``): Open addressing
     with MurmurHash3 32-bit/64-bit finalizer.  Right keys inserted
     cooperatively into ``shared`` arrays.  Left keys probed with linear
     search.  Matches written via Hillis-Steele prefix sum scatter.
   - **Per-workgroup dispatch**: Each WG builds the right table in shared
     memory and probes its slice of the left table.  Partitioning uses
     multiple sequential dispatches when the right table exceeds shared
     memory.
   - **Match flags buffer** (binding 4): ``atomicOr`` in shader tracks
     which left rows matched, used by left/outer join for null-filling
     unmatched rows.

   **Algorithm:** Shared-memory hash join (build + probe in one kernel).
   64-bit variant hashes both halves of the key with a mixed MurmurHash3
   variant.  Sort-key transforms applied for signed/float types to ensure
   hash determinism.

   **Edge cases:**

   - Zero-length left or right: returns empty columns.
   - All numeric key types supported (U/I/F, 8-64 bit).
   - Duplicate keys: right-side first occurrence wins; left duplicates all
     match.
   - Null context or columns: returns ``NULL``.
   - Dtype mismatch: returns ``NULL``.

   :param ctx: GPU context.
   :param left_key: Left (probe-side) key column.
   :param right_key: Right (build-side) key column (same dtype).

   :returns: A ``VcJoinResult`` with matching index pairs, or ``NULL`` on
             error.  Free with ``vc_join_result_destroy``.


.. c:function:: VcJoinResult* vc_left_join(VcContext* ctx, VcColumn* left_key, VcColumn* right_key)

   **Status:** fully implemented.

   Left join.  Returns all left rows; unmatched rows get ``0xFFFFFFFF`` as
   the right index.  Same key type support and partitioning as
   ``vc_inner_join``.


.. c:function:: VcJoinResult* vc_outer_join(VcContext* ctx, VcColumn* left_key, VcColumn* right_key)

   **Status:** fully implemented.

   Outer join.  Returns all left rows (unmatched: ``0xFFFFFFFF`` right index)
   followed by unmatched right rows (``0xFFFFFFFF`` left index).
   Same key type support and partitioning as ``vc_inner_join``.


.. c:function:: void vc_join_result_destroy(VcContext* ctx, VcJoinResult* result)

   Frees a ``VcJoinResult`` and its contained columns.

----

Element-wise Arithmetic
-----------------------

.. c:enum:: VcArithOp

   Arithmetic operation selector.

   .. c:enumerator:: VC_ARITH_ADD = 0
   .. c:enumerator:: VC_ARITH_SUB = 1
   .. c:enumerator:: VC_ARITH_MUL = 2
   .. c:enumerator:: VC_ARITH_DIV = 3

.. c:function:: VcColumn* vc_arith(VcContext* ctx, VcColumn* lhs, VcColumn* rhs, VcArithOp op)

   **Status:** implemented (>= 32-bit types only; U8/U16/I8/I16 require prior cast).

   Element-wise arithmetic between two columns of the same dtype and length.
   Operates on 64-bit unsigned pairs in the shader — signed and float types
   are cast to unsigned before dispatch.  Null propagation: if either operand
   is null, the result is null.

   **Shader:** ``elementwise.comp`` — unified shader handling all arithmetic
   and comparison operations via a push-constant op selector.  256 threads
   per workgroup, each processes one element.  64-bit values use ``uvec2``
   pairs (lo, hi) with carry-chain arithmetic.

   **Vulkan primitives:** one ``VkPipelineLayout`` with a 6-binding descriptor
   set layout (lhs data, rhs data, lhs validity, rhs validity, output data,
   output validity) and push constants (element count, op, words, scalar
   value, validity flags).  Output buffers are VMA-allocated with
   ``VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE``.

   **Edge cases:**

   - Null context or columns: returns ``NULL``.
   - Dtype or length mismatch: returns ``NULL``.
   - Sub-32-bit types: returns ``NULL`` (cast to U32/I32 first).
   - Division by zero: result set to null.
   - Zero-length columns: returns empty column of the same dtype.

   :param ctx: GPU context.
   :param lhs: Left operand column.
   :param rhs: Right operand column (same dtype and length).
   :param op: Arithmetic operation.

   :returns: A new column with the result, or ``NULL`` on failure.

.. c:function:: VcColumn* vc_arith_scalar(VcContext* ctx, VcColumn* lhs, VcScalar rhs, VcArithOp op, bool rhs_on_left)

   **Status:** implemented (>= 32-bit types only; scalar always on right).

   Element-wise arithmetic between a column and a scalar.  The scalar value
   is passed via push constants (``rhs_lo``, ``rhs_hi``) — no additional
   GPU buffer allocation.  Null propagation: null elements in the column
   produce null results.

   **Limitation:** ``rhs_on_left`` is accepted but not yet implemented —
   the scalar is always applied on the right side.  This matters for
   non-commutative operations: ``scalar - col`` and ``scalar / col`` are
   not currently supported.

   :param ctx: GPU context.
   :param lhs: Left (column) operand.
   :param rhs: Scalar value and dtype.
   :param op: Arithmetic operation.
   :param rhs_on_left: If true, scalar is on the left of the operator
                       (not yet implemented).

   :returns: A new column, or ``NULL`` on failure.

----

Element-wise Comparison
-----------------------

.. c:enum:: VcCompareOp

   Comparison operation selector.

   .. c:enumerator:: VC_CMP_EQ  = 0
   .. c:enumerator:: VC_CMP_NEQ = 1
   .. c:enumerator:: VC_CMP_LT  = 2
   .. c:enumerator:: VC_CMP_GT  = 3
   .. c:enumerator:: VC_CMP_LTE = 4
   .. c:enumerator:: VC_CMP_GTE = 5

.. c:function:: VcColumn* vc_compare(VcContext* ctx, VcColumn* lhs, VcColumn* rhs, VcCompareOp op, VcDtype mask_dtype)

   **Status:** fully implemented.

   Element-wise comparison between two columns of the same dtype and length.
   Returns a mask column where each element is 1 if the comparison is true,
   0 if false.  The mask column dtype must be ``VC_DTYPE_U32`` or
   ``VC_DTYPE_U8`` — U32 is the primary mask format used by ``vc_filter``.

   **Shader:** same ``elementwise.comp`` as arithmetic (op code 10--15).
   64-bit comparisons (LT, GT, LTE, GTE) compare hi word first, then lo
   word on equality.

   **Edge cases:**

   - Null context or columns: returns ``NULL``.
   - Dtype or length mismatch: returns ``NULL``.
   - Invalid ``mask_dtype``: returns ``NULL``.
   - Null propagation: null elements produce null results.

   :param ctx: GPU context.
   :param lhs: Left operand column.
   :param rhs: Right operand column (same dtype and length).
   :param op: Comparison operation.
   :param mask_dtype: ``VC_DTYPE_U32`` or ``VC_DTYPE_U8``.

   :returns: A new mask column (indicated dtype), or ``NULL`` on failure.

.. c:function:: VcColumn* vc_compare_scalar(VcContext* ctx, VcColumn* lhs, VcScalar rhs, VcCompareOp op, VcDtype mask_dtype, bool rhs_on_left)

   **Status:** fully implemented (scalar always on right).

   Element-wise comparison between a column and a scalar.  Same semantics
   as ``vc_compare`` but the scalar is passed via push constants.

   :param ctx: GPU context.
   :param lhs: Left (column) operand.
   :param rhs: Scalar value and dtype.
   :param op: Comparison operation.
   :param mask_dtype: ``VC_DTYPE_U32`` or ``VC_DTYPE_U8``.
   :param rhs_on_left: If true, scalar is on the left (not yet implemented).

   :returns: A new mask column, or ``NULL`` on failure.

----

Null Operations
---------------

.. c:function:: VcColumn* vc_is_null(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented.

   Returns a ``U32`` mask column where each element is 1 if the
   corresponding input element is null, 0 otherwise.  For columns without
   a validity buffer, returns all zeros.

   :param ctx: GPU context.
   :param col: Input column.

   :returns: A ``U32`` mask column, or ``NULL`` on failure.

.. c:function:: VcColumn* vc_is_not_null(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented (via ``vc_is_null`` + host-side inversion).

   Returns a ``U32`` mask column where each element is 1 if the
   corresponding input element is valid, 0 if null.

   :param ctx: GPU context.
   :param col: Input column.

   :returns: A ``U32`` mask column, or ``NULL`` on failure.

.. c:function:: VcColumn* vc_fill_null(VcContext* ctx, VcColumn* col, VcScalar fill_value)

   **Status:** fully implemented.

   Returns a new column where null values are replaced by ``fill_value``
   and all other values are copied through unchanged.

   :param ctx: GPU context.
   :param col: Input column (may be nullable).
   :param fill_value: Scalar replacement for nulls.

   :returns: A new non-nullable column, or ``NULL`` on failure.

.. c:function:: VcColumn* vc_drop_nulls(VcContext* ctx, VcColumn* col, uint64_t* out_length)

   **Status:** fully implemented.

   Removes all null elements from a column.  Returns a new non-nullable
   column containing only valid elements in their original relative order.
   If the input has no validity bitmap or zero nulls, returns a copy.

   Implementation: reads column data and validity bitmap to host, filters
   valid elements on CPU, creates a new column.  No shader dispatch.

   :param ctx: GPU context.
   :param col: Input column (may be nullable).
   :param out_length: [out] Number of elements in the result, or NULL.

   :returns: A new non-nullable column, or ``NULL`` on failure.

----

Unique / Distinct
-----------------

.. c:function:: VcColumn* vc_unique(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented.

   Remove duplicate values from a column.  Output is in sorted ascending
   order.  NaN values are each treated as unique (NaN != NaN).

   **Algorithm:** GPU sort via ``vc_sort`` (LSD radix sort, 4-8 passes on
   GPU), then host-side O(n) dedup on sorted data.  The sort -- the
   expensive part -- stays on GPU.  Host dedup is a single linear scan that
   runs in microseconds even for million-element columns.

   **Vulkan primitives:** delegates to ``vc_sort``.  No custom shader is
   needed for this operation -- the sorting shader handles all GPU work.

   .. rubric:: Edge cases

   - Zero-length input: returns an empty column of the same dtype.
   - Single-element input: returns a copy of the element.
   - All duplicates: returns a single-element column.
   - NaN handling: each NaN value is treated as a separate unique entry
     (consistent with IEEE 754 NaN != NaN).
   - Nullable columns: nulls sort to the end; all nulls collapse to a
     single null entry (only one null in the output).
   - Null context or column: returns ``NULL``.

   :param ctx: GPU context.
   :param col: Input column.

   :returns: A new column with duplicates removed, or ``NULL`` on failure.

----

Boolean Mask Operations
-----------------------

.. c:function:: VcColumn* vc_bool_and(VcContext* ctx, VcColumn* lhs, VcColumn* rhs)

   **Status:** fully implemented.

   Element-wise logical AND on two U32 or U8 mask columns.  Non-zero values
   are treated as true, zero as false.  Returns a U32 mask column (1 = true).

   U8 inputs are host-side widened to U32 before dispatch.

   :param ctx: GPU context.
   :param lhs: Left mask column (U32 or U8).
   :param rhs: Right mask column (same length).

   :returns: A U32 mask column, or ``NULL`` on failure.

.. c:function:: VcColumn* vc_bool_or(VcContext* ctx, VcColumn* lhs, VcColumn* rhs)

   **Status:** fully implemented.

   Element-wise logical OR on two U32 or U8 mask columns.

   :returns: A U32 mask column, or ``NULL`` on failure.

.. c:function:: VcColumn* vc_bool_not(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented.

   Element-wise logical NOT on a single U32 or U8 mask column.
   Returns a U32 mask column where 0 becomes 1 and non-zero becomes 0.

   :returns: A U32 mask column, or ``NULL`` on failure.

----

List Column Operations
----------------------

.. c:function:: VcColumn* vc_create_list_column(VcContext* ctx, uint64_t num_rows, const uint32_t* offsets, VcColumn* child_col, const uint32_t* validity, uint64_t null_count)

   **Status:** fully implemented.  **v0.2**

   Creates a list column from a child column and per-row offsets.  Ownership
   of ``child_col`` transfers to the list column.  Offsets use Arrow List
   layout: ``offsets[i+1] - offsets[i]`` elements for row ``i``.

   :returns: A ``VcColumn*`` of dtype ``VC_DTYPE_LIST``, or ``NULL`` on failure.

.. c:function:: int vc_read_list_column(VcContext* ctx, VcColumn* col, uint32_t* host_offsets, VcColumn** out_child)

   **Status:** fully implemented.  **v0.2**

   Reads a list column back to the host.  ``host_offsets`` receives offsets,
   ``out_child`` receives a newly allocated child column.

.. c:function:: VcColumn* vc_list_length(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented.  **v0.2**

   Per-row list length via GPU shader: ``offsets[i+1] - offsets[i]``.
   Returns a ``VC_DTYPE_U32`` column.

.. c:function:: VcColumn* vc_list_get(VcContext* ctx, VcColumn* col, int64_t index)

   **Status:** fully implemented.  **v0.2**

   Extract element at ``index`` from each list row.  Positive indices count
   from start, negative from end.  Bounds-clamped.  Returns column matching
   the list's child dtype.

.. c:function:: VcColumn* vc_list_explode(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented.  **v0.2**

   Flatten a list column into a flat column.  Each child element becomes one
   output row.  Null list rows produce null elements.

.. c:function:: VcColumn* vc_list_contains(VcContext* ctx, VcColumn* col, const void* value)

   **Status:** fully implemented.  **v0.2**

   Check if a scalar value exists in each list row.  Returns ``VC_DTYPE_U32``
   mask column (1 = contains, 0 = not).

.. c:function:: VcColumn* vc_list_sort(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented.  **v0.2**

   Sort elements within each list row.  Host-side per-subrange ``std::sort``.
   Returns new list column with sorted subranges, same offsets structure.

.. c:function:: VcColumn* vc_list_unique(VcContext* ctx, VcColumn* col)

   **Status:** fully implemented.  **v0.2**

   Remove duplicate values within each list row.  Sorts subranges then
   deduplicates adjacent elements.  Offsets may shrink.

----

Struct Column Operations
------------------------

.. c:function:: VcColumn* vc_create_struct_column(VcContext* ctx, uint64_t num_rows, VcColumn* const* fields, uint32_t field_count)

   **Status:** fully implemented.  **v0.2**

   Creates a struct column from an array of child columns.  All children must
   have the same length.  Ownership transfers — children must not be destroyed
   separately.

.. c:function:: VcColumn* vc_struct_field(VcContext* ctx, VcColumn* col, uint32_t field_index)

   **Status:** fully implemented.  **v0.2**

   Returns a non-owning pointer to the field at ``field_index``.  The returned
   column must not be freed.

.. c:function:: uint32_t vc_struct_field_count(VcColumn* col)

   **Status:** fully implemented.  **v0.2**

   Returns the number of fields in a struct column.

.. c:function:: VcColumn* vc_struct_rename_fields(VcContext* ctx, VcColumn* col, const char* const* names, uint32_t count)

   **Status:** fully implemented.  **v0.2**

   Rename struct fields.  Returns a new struct column sharing the same child
   data with updated field names.

.. c:function:: VcColumn* vc_struct_with_fields(VcContext* ctx, VcColumn* col, VcColumn* const* new_fields, const char* const* names, uint32_t count)

   **Status:** fully implemented.  **v0.2**

   Replace or add fields to a struct.  Returns a new struct column.
   Unchanged children are shared; replaced or added children transfer ownership.