14 posts tagged with "performance"

Spice v2.4.0-rc.1 is now available! 🔥

Spice v2.4.0-rc.1 is the first release candidate for v2.4.0. It adds performance improvements, S3 event-driven ingestion, SQL results-cache warmup, and adaptive HTTP rate controls. The release also upgrades to DataFusion v55, Ballista v55, Arrow v59, Vortex v0.86, Iceberg v0.11, and Turso v0.81.

Highlights in v2.4.0-rc.1 include:

• Performance & Query Engine: DataFusion v55, Vortex v0.86.1, Iceberg v0.11.0, and Ballista v55 update query planning, storage, and distributed execution.
• S3 Event-Driven Ingestion — ingest newly created objects through SQS notifications.
• Cache Warmup and Shared Fetches — warm the SQL results cache after the first refresh and coalesce concurrent source fetches.
• Adaptive HTTP Rate Controls — bound request waits and coordinate request limits across instances.
• ORC Support — query ORC files through listing connectors.
• Hugging Face Datasets: Query and accelerate files from the Hugging Face Hub.

What's New in v2.4.0-rc.1​

Performance & Query Engine​

This release upgrades Apache DataFusion to the v55.2.0 dependency line and Apache Arrow to v59.3.0. It also upgrades Vortex to v0.86.1, Apache Iceberg to the v0.11.0 fork, and Apache Ballista to v55.

Apache DataFusion v55​

The DataFusion v55 release adds the following improvements:

• Sort pushdown and TopK pruning: Parquet scans reevaluate each unread row group as the threshold for ORDER BY ... LIMIT tightens. They skip groups that cannot contribute to the result. TopK pruning also supports multiple sort columns.
• Join planning: The optimizer converts eligible inner joins to semi joins and removes redundant sides of outer joins. It also orders filter predicates by estimated cost.
• Aggregation and expressions: Multi-column GROUP BY uses column-oriented storage for all supported key types, such as fixed-size binary UUIDs. More string functions preserve dictionary encoding, and IN lists use specialized paths for small integer types.
• Parquet reads: Scans skip nested fields that the declared schema does not contain. They also skip page-index reads when a file has no page index.
• Spill handling: Sorts bound the number of streams in a merge. If memory is insufficient, they spill the largest stream again in smaller batches.
• SQL diagnostics and functions: EXPLAIN accepts PostgreSQL-style options and FORMAT pgjson. New array functions cover element-wise addition, subtraction, scaling, sums, and averages.

Spice carries these changes through its query plans and preserves statistics across plan wrappers. Cayenne keeps Vortex scans below 10 MiB unsplit to avoid repeated footer reads. See #14612.

Vortex v0.79.0 to v0.86.1​

Spice v2.3.2 used the Vortex v0.79.0 fork. This upgrade covers the full upstream range from v0.79.0 through v0.86.1, not only the v0.86 changes:

• Types and arithmetic: Vortex adds native Map arrays, Arrow map conversion, and operations and compression for maps. It also adds union arrays and decimal addition, subtraction, multiplication, and division.
• Row selection: Piecewise-sequence indices represent contiguous selections without an expanded index for every row. Specialized paths handle chunked arrays, fixed-size lists, variable-length lists, and binary values.
• Scans and expressions: Layout scans gain a physical plan and expression optimization. Filters can pass through scalar functions with multiple arguments. Filters on wide lists restrict child elements to the selected range, and constant masks can resolve from metadata.
• Compression: Binary arrays support FSST compression with variable-length offsets. OnPair becomes a stable encoding for reads and gains a storage-backed dictionary. Vortex can convert run-end arrays of lists and decimals.
• File metadata: Files can store custom metadata. Readers cache decoded type descriptions, and file-format editions define the supported encodings and types.
• Memory and execution: Builders append nested values in batches, preserve list views, and propagate buffer allocators. Expression rewrites retain unchanged nodes, and row functions support batch execution.
• Correctness and validation: NULL handling changes cover dictionary predicates, BETWEEN bounds, and empty arrays. Readers add validation for footer offsets, compression metadata, and array indices. Other corrections cover nested scalar hashes, decimal operations, and variable-length binary selections above 4 GiB.

See the complete Vortex v0.79.0 to v0.86.1 changelog for every upstream change. Changes to standalone Vortex bindings and GPU execution do not imply new Spice features.

Apache Iceberg v0.11.0​

Spice updates its Iceberg reader and catalog integrations from v0.10.1 to the v0.11.0 fork for DataFusion 55 and Arrow 59. The upstream v0.11 changes extend reads and catalog compatibility:

• Iceberg v3 reads: The reader applies deletion vectors from Puffin files and carries row identifiers and sequence numbers through scans.
• Metadata-only scans: Metadata-only projections do not require data-column reads. Manifest reads reuse partition types, and positional-delete processing buffers runs instead of allocating a key per row.
• REST catalogs: Clients negotiate server-advertised endpoints and use session-scoped OAuth2 authentication.

The upgrade also aligns Iceberg storage with OpenDAL v0.58. See #14771.

Apache Ballista v55​

Spice.ai Enterprise feature. See the Enterprise documentation.

The Ballista v55 upgrade adds virtual-core resource accounting for distributed tasks. A protocol-version handshake detects incompatible schedulers and executors. Cancellation identifies tasks by their task IDs, and task-state records use an append-only model.

Schedulers and executors must run the same version. Upgrade all cluster components together.

S3 Event-Driven Ingestion​

S3 listing datasets can use refresh_mode: changes with S3 event notifications delivered through SQS:

datasets:
  - from: s3://my-bucket/events/
    name: events
    params:
      file_format: parquet
      s3_region: us-east-1
      s3_auth: iam_role
      s3_changes_queue_url: ${ secrets:events_queue_url }
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      refresh_mode: changes

New object notifications append the object's rows. A periodic listing backfill covers missed or expired notifications. Each dataset needs its own queue and permissions to read and delete SQS messages, alongside its S3 read/list permissions.

This is object ingestion: removal notifications are ignored by default. Set s3_on_object_removed: rebuild to rebuild the entire prefix when an object is removed. An overwrite of an already applied object key is not ingested again; use new object keys for incoming data. See #14121.

SQL Results-Cache Warmup and Shared Fetches​

The SQL results cache can persist query plan shapes and replay them after a dataset's first full or append refresh:

runtime:
  caching:
    sql_results:
      enabled: true
      warmup: on_first_refresh

For example, run these queries against an accelerated orders dataset with warmup enabled:

SELECT id, status FROM orders WHERE id = 1;
SELECT id, status FROM orders WHERE id = 2;

Spice records one query shape because only the equality-filter value differs. After a restart and the dataset's first full or append refresh, warmup reruns that shape with distinct id values from the refreshed dataset, filling the cache before the dataset becomes ready. Keep the local .spice/data directory across restarts, or configure runtime.state.location, to retain recorded query shapes.

Warmup replays up to ten distinct recorded query shapes and tries up to 1,024 distinct filter-value combinations per shape, stopping when the cache is full. A dataset stays not ready until warmup completes. Later refreshes do not repeat warmup. The feature requires the default plan-based cache key; cache_key_type: sql is incompatible. See #14178.

Concurrent cache-miss fetches for the same request now share a source fetch. SQL results caching also includes changes for tables updated during query execution and for stale results served while revalidation runs. HTTP dataset caching supports RFC 5861 stale-if-error handling. See #14142, #14710, #14708, and #14134.

SQL, search, and embedding caches now use Spice's sharded cache backend. Existing engine: moka and engine: pingora values are accepted for configuration compatibility but no longer select a backend.

Adaptive HTTP Rate Controls​

HTTP rate controls now adapt admission to upstream failures while staying within configured request limits. rate_control_acquire_timeout bounds how long a request waits for capacity and defaults to the connector's client timeout. rate_control_failure_threshold and rate_control_window control the response to upstream failures.

In Spice.ai Enterprise, instances that share a runtime.state.location coordinate per-second and per-minute limits through shared state; OSS instances keep these limits in memory independently. Concurrency limits remain local to each instance. Components sharing an upstream origin must use matching rate-control settings. See HTTP rate-control documentation and #14143.

ORC Files and Object Metadata Queries​

Listing connectors support file_format: orc. Object-store listing also uses predicates on metadata columns, including _last_modified, to narrow eligible objects. Queries that select only partition or metadata columns can use those values without reading file contents. See #14075, #14265, #14303, and #14116.

Hugging Face Datasets​

The new Hugging Face data connector queries and accelerates datasets from the Hugging Face Hub. It supports Parquet, CSV, TSV, JSON, and ORC files. Public datasets need no credentials:

datasets:
  - from: hf://datasets/stanfordnlp/imdb/plain_text/
    name: imdb
    acceleration:
      enabled: true

The location format is hf://datasets/<owner>/<dataset>[@<revision>][/<path>]. A path can select a file, a folder, or a glob. A revision can name a branch, a tag, or a commit. Use @~parquet to select the Hub's automatic Parquet conversion.

Set hf_token for private or gated datasets. Set hf_endpoint for a Hub mirror or proxy. Each scan reads one commit. Refreshes follow the selected branch, but the dataset keeps its registered schema until reload. See #14877.

Automatic Primary-Key Handling​

Cayenne keeps one row per primary_key without requiring an on_conflict policy. When a dataset sets time_column, the row with the newest time wins; without it, the last arrival wins. This applies to full and append refreshes as well as writes. Existing explicit conflict policies remain accepted during the deprecation period.

datasets:
  - from: s3://my-bucket/orders/
    name: orders
    time_column: updated_at
    params:
      file_format: parquet
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      primary_key: id

For a read-write dataset whose writes should stay in its acceleration, set acceleration.write_mode: acceleration. Its source need not support writes. This mode cannot be combined with a dataset that refreshes by changes.

Cayenne secondary indexes also support dynamic join filters, and their write handling covers inserts, updates, deletes, and refreshes. See #14726, #14282, and #14593.

PostgreSQL and MySQL replication, and MongoDB change streams, rejected Cayenne datasets that omitted on_conflict. Their validation still required an explicit upsert policy. These sources now accept Cayenne datasets with primary_key alone. See #14880.

For file-mode Cayenne datasets, append refreshes failed with a configuration that combined primary_key, time_column, and retention_sql. The refresh selected a version-resolution path that did not support retention. The refresh now resolves each key's newest version before Cayenne applies retention. See #14878.

Cayenne maintained aggregates now share one compact index of per-key contributions across views. Rebuilds capture concurrent writes and apply them after the scan. The aggregate budget uses 10% of a bounded query pool without the former 512 MiB cap. An unbounded pool retains the 512 MiB budget. See #14762.

Read Datasets from Published Snapshots​

Spice.ai Enterprise feature. See the Enterprise documentation.

A dataset can now read published acceleration snapshots directly, without configuring the original source connector:

datasets:
  - from: s3://my-bucket/spice/snapshots/orders/
    name: orders
    params:
      file_format: snapshot
      s3_region: us-east-1

Spice reads the snapshot metadata to select the engine, restores the published data, and checks for newer snapshots. The dataset is read-only. Snapshot-mode readers can also use S3 notifications delivered through SQS, with periodic checks retained for missed notifications. Each reader process needs its own queue.

This release includes changes to snapshot retries, slow-connection bootstrap, publication metadata, and coordination between snapshot archiving and Cayenne maintenance. Cayenne datasets with a datalake tier cannot create acceleration snapshots. See snapshot documentation, #14529, and #14335.

Connector and Protocol Updates​

• Connector status: ADBC, Databricks Spark Connect and SQL Warehouse, FlightSQL, Glue, HTTP/HTTPS, Iceberg, Localpod, and MongoDB are now Stable data connectors.
• MCP: support for specification version 2026-07-28, alongside the earlier protocol era. See #14043.
• GitHub: nested GraphQL pagination and rate-limit pacing updates; the default concurrency limit is now four. See #14179 and #14431.
• GitHub nested pages: Scans could return incomplete reviews or comments because pagination accepted a short page as complete. The connector now rejects incomplete connections and repeated cursors. It retries a failed nested page without another fetch of the outer page. Datasets with the same token also share the REST quota. See #14862.
• Iceberg REST: Clients could read an empty dataset because the catalog synthesized metadata without snapshots. The catalog now returns the source metadata for Iceberg datasets that Spice reads unchanged. Other datasets return 400 BadRequestException. Clients need their own storage credentials. See #14588 and Breaking Changes.

Other Fixes​

The release includes fixes in the following areas; the linked PRs provide details of the changes:

• Cayenne queries and writes: NULL-aware NOT IN, maintained aggregates, dynamic filters, partition-filter forwarding, memory-mode DML and retention, and primary-key handling across CDC checkpoints. See #14429, #14761, #14370, #14047, and #14344.
• Startup and reloads: retry datasets with unavailable sources, serve existing accelerations during source outages, and invalidate cached plans and results on catalog replacement or dataset unload. See #14623, #14624, #13914, and #14365.
• Federation: local evaluation of casts and functions whose source semantics differ, plus filter pushdown changes for DynamoDB, Cosmos DB, and MongoDB. See #14484, #14601, and #14419.
• HTTP and GraphQL: response-status handling during refresh, retry-budget handling, non-JSON gateway responses, and URL redaction in HTTP errors. See #13538, #14313, #14781, and #14490.
• Search: deletion of obsolete Elasticsearch chunks, non-finite embedding handling, and source-scan coordination during full-text refresh. See #13960, #13902, and #14663.
• Models and tools: tool-call-only assistant turns, required tool choices, streaming tool-use completion, model-load diagnostics, and propagation of the API-key principal into MCP tool calls. See #14232, #14460, #14548, and #14828.
• CDC shutdown and reconnects: source-position recording before accelerations close, and MySQL shared-stream reconnect handling. See #14702 and #14751.
• SQL weekdays: date_part('dow') aligns with EXTRACT(dow), with Sunday represented as zero. See #14796.
• Cayenne schema statistics: Decimal bounds could retain an old scale after schema evolution because maintenance published statistics from the previous schema. Cayenne now rejects statistics from an obsolete schema and keeps row counts conservative. See #14856.
• Vector search: vector_search planning failed after the DataFusion 55 upgrade because a second optimization pass tried to reorder a join with a dynamic filter. The planner now preserves that join's input order. See #14857.

Default accelerator: Datasets and views that enable acceleration without engine now use Cayenne. Explicit engine settings keep their behavior. Storage still defaults to memory. Set mode: file for persistent acceleration. See Breaking Changes for migration guidance.

Dependency Updates​

Contributors​

• @bjchambers
• @ewgenius
• @Jeadie
• @krinart
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov

Breaking Changes​

Cayenne is the default accelerator on supported platforms. A dataset or view that omits acceleration.engine switches from Arrow to Cayenne. To retain Arrow, set engine: arrow explicitly before upgrading. Windows keeps Arrow as its default. Persistent datasets should continue to name their engine and use mode: file.

on_conflict is deprecated and scheduled for removal in v3.0. Cayenne automatically keeps one row per primary key, choosing the newest time_column value when configured, or the last arrival otherwise. Existing explicit policies remain supported during the deprecation period. Review those policies before removing them, especially drop or policies that reject conflicting rows.

on_conflict no longer routes writes to the acceleration. For read-write datasets whose writes should stay in the acceleration, use:

acceleration:
  enabled: true
  engine: cayenne
  write_mode: acceleration

This setting cannot be used with refresh_mode: changes, including a connector's default change-stream mode. The default write_through and write_back modes require a writable source.

Cache engine selection is retired. engine: moka and engine: pingora remain accepted but are ignored. Remove the field and use caching_policy to select eviction behavior.

GitHub connector default concurrency is four. Review explicit concurrency settings if your deployment relied on the previous default.

HTTP rate-control waits are bounded by default. Requests waiting for rate-control capacity now time out after the connector's client timeout. Set rate_control_acquire_timeout to a suitable duration, or 0 to retain the previous unbounded wait behavior.

Cayenne acceleration snapshots are unavailable for datalake-tier datasets. Review snapshot settings on datasets using cayenne_datalake_location; this release disables snapshotting that configuration.

Iceberg REST no longer synthesizes metadata for unsupported datasets. GET /v1/namespaces/{namespace}/tables/{table} returns 400 BadRequestException for accelerated datasets, views, and other datasets that Spice does not read unchanged from Iceberg. If a client used this endpoint for schema discovery, use SQL DESCRIBE or information_schema.columns instead. Query these datasets through /v1/sql or Arrow Flight SQL. For eligible Iceberg datasets, clients read the source metadata and need their own storage access. See the Get a table API.

Cookbook Updates​

The Spice Cookbook provides recipes to help you get started with Spice.

Upgrading​

To upgrade to v2.4.0-rc.1 once the release artifacts are available, use one of the following methods:

CLI:

spice upgrade v2.4.0-rc.1

Docker:

Pull the spiceai/spiceai:2.4.0-rc.1 image:

docker pull spiceai/spiceai:2.4.0-rc.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.4.0-rc.1

AWS Marketplace:

Spice is available in the AWS Marketplace. Marketplace availability follows its published versions.

What's Changed​

Changelog​

• fix(runtime): discard cached logical plans when a hot reload replaces a catalog (fixes #13910) by @claudespice in #13914
• fix(acceleration): let a schema repair correct a checkpoint without resetting the freshness clock (fixes #13817) by @claudespice in #13894
• fix(search): filter a chunked Elasticsearch delete on a field that can match the key (fixes #13714) by @claudespice in #13926
• fix(search): classify a partially non-finite embedding as unindexable on every backend (fixes #13872) by @claudespice in #13902
• fix: stabilize GitHub tests and bound GraphQL registration (fixes #13762) by @lukekim in #13939
• fix(postgres): decode versioned JSONB binary replication values by @phillipleblanc in #13962
• docs: require a reviewed Enhancement before any user-facing surface changes by @lukekim in #13970
• ci: upgrade spiceio setup action to v0.9.0 by @lukekim in #13971
• fix(postgres): preserve microseconds in timestamp writeback by @phillipleblanc in #13963
• feat(hash-index): verify the bloom filter's block index with Verus by @lukekim in #13777
• build(deps-dev): bump js-yaml by @dependabot in #13989
• docs: release notes for v2.3.0 by @bjchambers in #13999
• fix(ci): drop the dangling substrait-compliance submodule pointer by @bjchambers in #14002
• fix(cayenne): release the keyset bytes an abandoned PK checkout accounted (fixes #13668) by @grokspice in #13925
• fix(caching): keep a declared key from disabling eviction and stranding stale rows (fixes #13976) by @bjchambers in #13992
• Add Substrait compliance harness (IBM TPC-H Mode A + FlightSQL Mode B stub) by @lukekim in #13879
• ci: skip DynamoDB TPC-H benches in OSS testoperator dispatch by @phillipleblanc in #14016
• docs: update security support and roadmap after v2.3.0 by @phillipleblanc in #14024
• chore: post v2.3.0 release housekeeping by @bjchambers in #13969
• test(adbc): guard BigQuery corpus offline and in release gate by @phillipleblanc in #14017
• fix(duckdb): deny the regexp built-ins DuckDB cannot answer faithfully (fixes #13809) by @claudespice in #13871
• fix: Update tpch benchmark snapshots for federated/adbc[bigquery].yaml by @app/github-actions in #13984
• fix(search): drop the chunks a shortened row no longer produces from a chunked index (refs #13717) by @claudespice in #13960
• Reduce Cayenne allocations during primary-key validation and filtering by @lukekim in #14009
• test(forks): guard seven fork patches that had no repo-side test by @krinart in #13996
• fix(deps): bump arrow-rs to correctly-rounded Decimal→Float cast (closes #13978) by @Jeadie in #14012
• perf(vortex): defer projection setup on filtered scans until the filter resolves by @bjchambers in #14035
• endgame: include spiceai/skills versioned release by @lukekim in #14031
• fix(caching): partition doomed entries at the survivor cutoff so eviction converges (closes #13994) by @Jeadie in #14021
• fix(deps): bump arrow-rs fork pin for Decimal->Float rounding fix by @Jeadie in #14049
• Fix subqueries with use_source acceleration by @phillipleblanc in #14022
• fix(cayenne): make DELETE, UPDATE and INSERT work on a mode: memory acceleration (fixes #12008) by @bjchambers in #14047
• fix: clarify OpenDAL S3 retry warnings by @lukekim in #14040
• test(s3): run the parquet-overwrite fixtures on RustFS by @bjchambers in #14067
• feat(cayenne): materialize multi-reference CTEs on the query path by @lukekim in #13918
• perf(vortex): answer a constant IN list by probing a set, and falsify it by interval by @bjchambers in #14061
• perf(vortex): skip a scan split whose zones cannot satisfy the filter by @peasee in #14064
• fix(arrow): report an exact row count from the indexed point-lookup scan by @krinart in #13972
• fix(cayenne): apply sort_columns with refresh_mode: full by @peasee in #14063
• fix(smb): pad an empty CREATE buffer so Samba lists the share root (fixes #13293) by @grokspice in #14050
• perf(cache): key the logical-plan cache on SQL text, not parameter values by @bjchambers in #14069
• fix: harden HuggingFace E2E chat against slow Metal generation by @lukekim in #14072
• fix(cayenne): move accelerator filesystem I/O off Tokio workers by @lukekim in #14073
• test(chbench): enable CTE materialization and IVM on mysql/postgres adaptive HTAP by @lukekim in #14070
• ci: run Substrait Mode A TPC-H on pull requests and the merge queue by @lukekim in #14071
• feat(mcp): support MCP specification 2026-07-28 (dual-era) by @lukekim in #14043
• fix(ci): call a linker that died of a signal an infrastructure failure, not a check failure (fixes #13614) by @grokspice in #14044
• fix: Provide temporary directory in docker images by @Jeadie in #14089
• fix: restore OSS installer, CLI and test workflow coverage by @phillipleblanc in #14025
• Delete v2.2.0.md by @Jeadie in #14095
• docs(release): add v2.3.1 release notes by @phillipleblanc in #14094
• fix(test): allow DELETE in the CORS allow-methods assertion by @claudespice in #14098
• fix(ci): stop install-protoc unzipping into a shared ~/.local by @lukekim in #14097
• feat(connectors): add ORC listing format via in-repo FileFormat by @lukekim in #14075
• fix(cayenne): run snapshot bootstrap check before opening the metastore by @Jeadie in #14093
• feat(cache): verify the results-cache namespace prefix with Verus by @lukekim in #14074
• feat(cloud-connect): add a GetDatasets command that answers the /v1/datasets document (refs #13369) by @grokspice in #14051
• Suppress Cayenne startup logs when no Cayenne dataset is configured by @Jeadie in #14042
• fix(cache): re-bind parameter values when revalidating a stale result (fixes #14099) by @bjchambers in #14100
• fix(cluster): support distributed HTTP scans by @phillipleblanc in #14108
• fix(bigquery): keep ILIKE evaluation local by @phillipleblanc in #14110
• docs: update security support for v2.3.1 by @phillipleblanc in #14117
• feat(cayenne): reuse ScanView until write, lag only for read-only CDC by @lukekim in #14055
• fix(deps): remediate open Dependabot alerts by @phillipleblanc in #14111
• fix(testoperator): validate results in every scale factor 1 TPC-H, TPC-DS and ClickBench benchmark by @lukekim in #14119
• fix(cayenne): reject ambiguous metastore paths by @phillipleblanc in #14130
• perf: serve results-cache hits where the request arrives and cut per-hit overhead by @lukekim in #14103
• test(runtime): record query previews in the management export test by @lukekim in #14155
• ci: upgrade spiceio setup action to v0.11.0 by @lukekim in #14152
• feat(caching): Make caching_stale_if_error RFC-5861 compliant (with stale-if-error header) by @Jeadie in #14134
• perf(runtime-table): defer cache-eviction key extraction to entries a delete actually names by @Jeadie in #14138
• fix(runtime): report the acceleration.ready_state deprecation once per component (fixes #13749) by @claudespice in #14006
• fix(runtime): write the inferred Arrow sort order under the prefixed key its validation accepts (fixes #14023) by @claudespice in #14032
• fix(connectors): Fix JSON/Orca files using metadata columns by @Jeadie in #14115
• feat(cayenne): build secondary indexes from indexes in file and memory mode by @phillipleblanc in #14149
• fix(cayenne): round-trip decimal, binary, and time stats and drop them on scale change by @lukekim in #14139
• feat(cayenne): cluster warm and datalake tiers, and write full refreshes as key-range files by @lukekim in #14124
• fix(cayenne): compile the cold-tier pruning test and backtick a doc literal by @lukekim in #14175
• fix(cayenne): make the crates own targets lint and compile by @phillipleblanc in #14200
• test(forks): guard five more fork patches, and drop a row that is not fork state by @krinart in #14015
• perf(cache): promote encoded SQL results to raw after the second decode by @lukekim in #14199
• fix(turso): build a dictionary column directly so a dictionary over a list, map or boolean value reads back (fixes #13033) by @grokspice in #14181
• fix(ci): probe the macOS toolchain before reaching for brew in the release builds by @grokspice in #14203
• fix(ci): skip Metal kernel precompilation in the macOS release build by @grokspice in #14204
• fix(github): paginate nested GraphQL connections and pace to GitHub's rate limits by @lukekim in #14179
• fix(vortex): stop an IN list holding a NULL from panicking the scan by @krinart in #14163
• bench(cayenne): use std::hint::black_box in the clustering bench by @lukekim in #14129
• fix(runtime-table): stop rebuilding SessionContext on every cache fetch by @Jeadie in #14141
• test(chbench): cluster order_line, oorder and customer on the adaptive HTAP arms by @lukekim in #14192
• fix(runtime): count a first load as still loading in the Dataset load summary (fixes #13974) by @claudespice in #14020
• fix(duckdb): push regexp_count down again at a rendering that counts as the kernel does (fixes #13870) by @claudespice in #14153
• build: lint and test the sign-off under the same profile as the merge queue by @lukekim in #14180
• ci: require the Verus proofs in the merge queue as one check by @lukekim in #14189
• ci: run the longest macOS jobs on their own runner pool by @lukekim in #14229
• fix(cayenne): build the DELETE sink inside the execution-time write lock (fixes #13828) by @claudespice in #14218
• build(deps): bump the github-actions-dependencies group across 1 directory with 7 updates by @dependabot in #14231
• fix(cluster): decide what a Flight message carries by its IPC header, not its body length (refs #13737) by @claudespice in #14212
• ci: run Mode A TPC-H on merge queue and trunk/release push only by @lukekim in #14247
• chore(deps): bump spiceai/duckdb-rs to 76655d2f by @lukekim in #14246
• ci: stop exporting empty AWS and DuckLake endpoints to the schema test by @phillipleblanc in #14194
• fix(runtime): reload a localpod dataset when the dataset it reads through is reloaded (fixes #3288) by @claudespice in #14208
• build(deps): bump the aws-sdk group with 3 updates by @dependabot in #14255
• fix(ci): resolve Homebrew prefix when brew is the spice flock wrapper by @lukekim in #14210
• chore(deps): raise the datafusion-table-providers pin to include the NUMERIC result-column fix by @phillipleblanc in #14254
• ci: align the DuckLake bootstrap with the embedded DuckDB, wait for Databricks startup, and stop dispatching legs that cannot pass by @phillipleblanc in #14250
• fix(runtime): install the Spice function deny-list on the PostgreSQL catalog connector (refs #13664) by @claudespice in #14225
• perf(cayenne): share inline-cache view entries by Arc instead of cloning them per scan by @krinart in #14191
• build(deps): bump aws-actions/configure-aws-credentials by @dependabot in #14256
• ci: stop dispatching the indexed turso TPC-H SF1 tests by @phillipleblanc in #14252
• fix(catalog): keep the tables registered under an existing schema by @phillipleblanc in #14193
• feat(cache): Spice sharded cache as the sole LruCache engine by @lukekim in #14206
• ci: lint GitHub Actions definitions with actionlint, and fix the 73 findings it surfaced by @grokspice in #14223
• perf(cache): tighten the Raw SQL results-cache serve path by @lukekim in #14205
• ci: stop triggering the CUDA build on pull requests by @lukekim in #14267
• ci: run CodeQL on pull requests and the merge queue by @lukekim in #14269
• Release 2.3.2 release notes by @krinart in #14271
• chore: make AGENTS.md the canonical agent instructions by @lukekim in #14237
• ci: run CodeQL Analyze on spiceai-dev-runners by @lukekim in #14281
• feat: TypeSafe Jev System One evaluation provider by @lukekim in #14215
• fix(cayenne): tag file statistics bounds as the column's Arrow type (fixes #14280) by @phillipleblanc in #14283
• ci: install spiceio when the runner has no gh (refs #14233) by @lukekim in #14288
• test(forks): 11 repo guards by @krinart in #14261
• docs: add Spice.ai in Action manuscript and companion labs by @lukekim in #13965
• test(duckdb): add an integration test for the index CTE materialization by @sgrebnov in #13885
• fix(cayenne): coalesce inline writes into one batch by @sgrebnov in #14279
• release: Update SECURITY.md and endgame template after 2.3.2 by @peasee in #14293
• fix(cayenne): count each Arrow allocation once in the inline-cache gauge by @krinart in #14272
• feat(s3): SQS event-driven changes for refresh_mode: changes by @lukekim in #14121
• feat(caching): single-flight coalesce concurrent cache-miss fetches by @Jeadie in #14142
• fix(caching): make caching_stale_if_error detect transient HTTP failures on real schemas by @krinart in #14161
• Use Cayenne secondary indexes for dynamic join filters by @phillipleblanc in #14282
• fix(ci): compare DynamoDB sets without relying on element order by @bjchambers in #14328
• docs(release): Remove QA analytics step from endgame by @peasee in #14329
• ci: run CodeQL on the merge queue and trunk, not on pull requests by @lukekim in #14290
• docs(cayenne): reposition the reference, add query serving, re-audit against trunk by @lukekim in #14338
• docs(endgame): update versioned docs release steps by @ewgenius in #14277
• test(forks): guard the ballista per-task file-scan restriction by @krinart in #14292
• fix(cli): surface the full error chain for spice chat connection failures by @krinart in #14289
• fix(ci): degrade the incomplete-sign-off handler when the runner has no gh (refs #14234) by @claudespice in #14304
• fix(runtime-table): serialize a direct write against acceleration snapshot creation (fixes #13548) by @claudespice in #14310
• fix(duckdb): screen regexp_like and regexp_replace as regexp_count is screened (fixes #14148) by @claudespice in #14321
• fix(http): honor retry budget without an extra origin request by @phillipleblanc in #14313
• Fix SchemaCastScanExec's schema conversion in fn partition_statistics by @Jeadie in #14258
• fix(cluster): recognise a Flight keepalive by its empty envelope, not by what its header declares (fixes #13737) by @claudespice in #14327
• perf(http): defer zero-TTL acceleration lookup until origin failure by @phillipleblanc in #14302
• fix(cayenne): keep a key visible when it is re-inserted over a stale-insert tombstone by @sgrebnov in #14312
• perf(cayenne): read only key and filter columns in filtered key deletes by @sgrebnov in #14374
• fix(cayenne): let small protected-snapshot merges run during a long large-tier merge by @sgrebnov in #14296
• perf(cayenne): serve primary-key lookups on a freshly loaded table in ~1 ms by @lukekim in #14314
• fix(cayenne): skip min/max statistics for nested columns (fixes #14368) by @sgrebnov in #14392
• test(caching): cover SchemaCastScanExec statistics projection by name, retype, and SQL filter by @Jeadie in #14146
• fix(turso): keep a quantified comparison out of Turso SQL (fixes #14041) by @grokspice in #14393
• fix(udfs): declare the local_embed dev-dependency the embed tests need (fixes #13092) by @claudespice in #14377
• perf(cayenne): batch metastore manifest rewrites, upsert in place, and run every write on one writer connection by @lukekim in #14369
• fix(runtime-table): ignore zero-row batches in stale fallback by @phillipleblanc in #14331
• Prune object-store file listing by _last_modified predicates by @Jeadie in #14265
• Reading only partition or metadata columns needlessly scans all file contents by @Jeadie in #14116
• fix(duckdb): keep a concat over a binary operand out of the federated plan (fixes #13915) by @claudespice in #14333
• fix(ci): keep the sign-off attribution inside GitHub's 140-character status cap (fixes #14076) by @grokspice in #14390
• fix(ci): expose a present-but-unlinked cc tool on macOS runners instead of routing it through brew install (fixes #13479) by @grokspice in #14387
• ci: re-measure integration.yml's job bounds after the archive consolidation (fixes #13429) by @grokspice in #14388
• fix(ci): start DuckLake's local MinIO from an image that is still published by @grokspice in #14399
• test(runtime-table): make the metric-scraping refresh tests pass under cargo test by @claudespice in #14381
• fix(federation): keep a correlated subquery predicate above a join of two sources (refs #8220) by @claudespice in #14372
• fix(caching): snapshot staleness before the origin fetch for stale_if_error by @Jeadie in #14263
• perf(snapshots): skip unchanged snapshot metadata with a conditional GET by @sgrebnov in #14409
• fix(search): prune the rest of a key group from an Elasticsearch chunked index (refs #13717) by @claudespice in #14320
• fix(bench): derive MySQL's empty-field NULL handling from the column type (refs #13152) by @claudespice in #14345
• fix(graphql): debit a LIMIT by the rows a page returned, not the declared page size (fixes #14308) by @claudespice in #14353
• fix(ci): fit retention_oom's retry budget inside its workflow step, and guard the coupling (fixes #13512) by @grokspice in #14389
• docs: say plainly what Spice is, refresh the README for v2.3, and promote connector statuses by @lukekim in #14410
• fix(ci): pin, checksum and retry the oha download in the E2E graceful-shutdown jobs by @grokspice in #14418
• fix(snapshots): resolve snapshot entries relative to the metadata location (#14425) by @sgrebnov in #14426
• Lower the GitHub connector default concurrency limit to 4 by @lukekim in #14431
• build(deps): bump nvidia/cuda in the docker-dependencies group by @dependabot in #14439
• build(deps): bump the aws-sdk group with 3 updates by @dependabot in #14440
• build(deps): bump the github-actions-dependencies group across 1 directory with 5 updates by @dependabot in #14441
• test(cayenne): bound the refused-build guard by builds, not by the host's speed by @grokspice in #14424
• fix(cache): don't report an invalidation cancelled by runtime shutdown as a failure by @grokspice in #14417
• ci: run remote sign-off on the spiceai-macos pool by @lukekim in #14449
• ci: run CodeQL Analyze on spiceai-macos by @lukekim in #14442
• fix(runtime): keep the built-in date_part so both weekday spellings agree (fixes #13920) by @claudespice in #14154
• fix(cayenne): include the in-memory CDC tier when an overwrite or a retention pass covers the whole table by @lukekim in #14428
• fix: keep NOT IN null-aware through the join reorder and the Cayenne sort-merge rewrite by @lukekim in #14429
• fix(cayenne): discard a compaction whose snapshot an overwrite replaced mid-pass by @lukekim in #14432
• fix(cayenne): stop maintained views, Vortex IN lists and dynamic-filter sharing from returning wrong rows by @lukekim in #14427
• fix(cayenne): hide spilled rows on CDC upsert fallback by @bjchambers in #14416
• fix(ci): skip the integration, ADBC, chDB and E2E gate jobs on pull requests instead of passing them (fixes #13841) by @grokspice in #14454
• fix(cayenne): judge a filtered key delete's captured sources by the index captured with them (refs #13913) by @claudespice in #14455
• fix(ci): clear the macos-15 image's openssl@1.1 symlink before installing MySQL by @lukekim in #14489
• ci: produce CodeQL SARIF in the Analyze job on spiceai-macos by @lukekim in #14500
• fix(cayenne): correctness fixes for the goal-driven adaptive controller, with a closed-loop simulation harness by @lukekim in #14443
• fix(cdc): keep the newest source commit timestamp on a coalesced change batch by @lukekim in #14463
• fix(cayenne): draw a sequence for a current-snapshot append so per-key OCC can order it (fixes #13685) by @claudespice in #14360
• fix(http): name a configured model's load failure instead of reporting it not found (fixes #13303) by @claudespice in #14395
• fix(duckdb): keep inferred source indexes off change-stream accelerations so upserts commit under concurrent reads (refs #13929) by @claudespice in #14396
• fix(duckdb): keep a text cast over a binary operand out of the federated plan (fixes #14355) by @claudespice in #14448
• fix(llms): honor tool_choice required and allowed_tools on mistral.rs-hosted models instead of panicking (fixes #14230) by @claudespice in #14460
• test(runtime): run the load-error counter test in its own process (fixes #13085) by @claudespice in #14462
• fix(runtime): stop a replaced dataset configuration's load from registering over the new one (fixes #1458) by @claudespice in #14367
• fix(install): stop asking for sudo on a first install into a fresh HOME (fixes #14445) by @claudespice in #14495
• perf(cayenne): serve primary-key point lookups in half the time by @lukekim in #14433
• fix(deps): bump DataFusion for upstream fixes to wrong results from filter pushdown, simplification and planning by @lukekim in #14430
• fix(runtime): size every internal DataFusion session from the CPU budget by @bjchambers in #14412
• fix(runtime): serve nested, zoned and half-float columns from the Iceberg catalog API (fixes #4815) by @claudespice in #14480
• fix(acceleration): keep cached results when a snapshot refresh finds no newer snapshot by @sgrebnov in #14497
• fix(runtime): sleep cron tests to the next boundary, not one that already fired (fixes #13759) by @claudespice in #14474
• fix(ci): derive every lint-rust guard make runs, whatever its separator or recipe layout (fixes #13783) by @claudespice in #14475
• fix(cayenne): stop the small-file compaction of a position-mode PK table from deadlocking on its own write lock (fixes #14420) by @claudespice in #14481
• fix(runtime-table): report refresh bytes for the rows each batch holds by @Jeadie in #14469
• fix(spark,databricks): keep Spice-only functions out of SQL sent to Spark Connect and Databricks SQL Warehouse (refs #13664) by @claudespice in #14498
• fix(vortex): size a cached footer by what it retains, not its serialized bytes (fixes #12917) by @claudespice in #14502
• fix(cayenne,telemetry): Register accelerated sink dataset immediately from existing acceleration by @peasee in #13955
• Update spicepod.yml by @Jeadie in #14533
• fix(cayenne): keep a rewrite's count inexact when it retains a late protected snapshot (fixes #14383) by @claudespice in #14385
• fix: Update tpch benchmark snapshots for accelerated/on_zero_results/file[parquet]-cayenne[file]-on_zero_results.yaml by @app/github-actions in #14408
• build(rust): upgrade toolchain to 1.98.1 by @lukekim in #14560
• fix(postgres): release a shared slot's hold on a table its publication cannot drop (fixes #13032) by @claudespice in #14527
• fix(cayenne): size the build side before sort-merging a non-Cayenne outer join by @krinart in #14520
• Update DF Upgrade template by @krinart in #14526
• fix(ci): wait for the refresh to invalidate the results cache, not a fixed 3s by @grokspice in #14598
• fix(deps): move the DataFusion pin past the four unparser fixes, and guard each of them (fixes #13022) by @grokspice in #14570
• fix(dynamodb, cosmosdb, mongodb): push filters down only where the source evaluates them as SQL does by @lukekim in #14419
• feat(snapshots): serve a dataset from published snapshots with from: s3://… and file_format: snapshot by @lukekim in #14529
• perf(cayenne): keep the per-shard PK index across checkpoint flushes and back off futile bakes; fix q17 and two wrong-results races (fixes #14235) by @lukekim in #14555
• test(chbench): lower the MySQL adaptive pods' CDC linger to 500 ms by @lukekim in #14615
• fix(graphql): show the parse-failure bytes in JSON decode previews by @Jeadie in #14530
• Defer source-first cache fallback planning by @phillipleblanc in #14556
• perf(cayenne): order whole-table rewrites per scan partition, then merge by @bjchambers in #14488
• fix(ci): tolerate the parquet-rename race's third DuckDB error in the E2E log scan by @grokspice in #14541
• fix(graphql): retry an inferred credential refusal instead of failing the refresh by @Jeadie in #14539
• fix(cayenne): serve a widened table's files from their persisted statistics (refs #13829) by @claudespice in #14316
• fix(acceleration): checkpoint DuckDB's write-ahead log before a snapshot copies its file (fixes #13912) by @claudespice in #14325
• fix(ci): let E2E cleanup run when the job failed before creating its working directory by @grokspice in #14553
• Replace unmaintained backoff with a workspace crate by @phillipleblanc in #14621
• test: read Expect stand-ins with the installed shell by @phillipleblanc in #14634
• fix(models): apply a tool_choice that forces a call to one round of the tool-use loop (fixes #14459) by @claudespice in #14635
• fix(runtime): don't end a tool-use stream on a stale tool_calls finish (fixes #13309) by @claudespice in #14548
• fix(smb): name listed locations under the share so directory datasets load (fixes #14060) by @claudespice in #14549
• fix(models): name a configured model's load failure in ai() and streaming nsql (fixes #14394) by @claudespice in #14632
• test(chbench): remove the mysql-cayenne[file]-adaptive-split HTAP arm by @sgrebnov in #14620
• fix(cayenne): record sharded CDC keys in the table-wide PK index by @lukekim in #14603
• fix(sqlite): keep TRY_CAST and every cast SQLite answers differently out of the federated plan (fixes #14398) by @claudespice in #14496
• ci: move remaining Ubuntu 22.04 references to 24.04 by @lukekim in #14677
• fix(snapshots): retry a failed snapshot attempt with backoff by @sgrebnov in #14676
• fix(snapshots): large snapshots no longer fail bootstrap on slow connection by @sgrebnov in #14571
• fix(refresh): start a refresh's source scan only when the sink reads it, so the full-text index keeps its rows (fixes #14619) by @claudespice in #14663
• test(cayenne): compare suite answers cell by cell against SQLite and chDB by @lukekim in #14473
• Evaluate any chat model via /v1/evaluate by @lukekim in #14568
• ci: quarantine the management API integration schedule until its dev OAuth client is reactivated (refs #12376) by @grokspice in #14684
• ci: stop scheduled Testoperator Ballista benchmarks by @phillipleblanc in #14682
• fix(cayenne): stop snapshotting datasets with a datalake tier, whose restored copies deleted each other's files by @lukekim in #14583
• fix(cayenne): count each Arrow allocation once for the mem-tier limit and checkpoint write sizing by @krinart in #14675
• fix(ci): classify a sign-off that reached its step budget as signalled, so an expiry publishes no verdict (fixes #13843) by @grokspice in #14685
• fix(runtime): retry a dataset whose source is unreachable at startup (fixes #14609) by @bjchambers in #14623
• fix(cayenne): archive only the snapshot directories a dataset snapshot references (fixes #14605) by @sgrebnov in #14627
• ci: route trunk-push macOS builds to the standard pool by @phillipleblanc in #14645
• ci: install strip for the retention OOM regression test by @phillipleblanc in #14701
• fix(smb): serve every share on a host from the one store registered for it (fixes #14550) by @grokspice in #14689
• fix(runtime): discard cached plans and results when a dataset is unloaded (refs #14251) by @claudespice in #14365
• feat(snapshots): reload snapshot-mode datasets from S3 event notifications (SQS) by @lukekim in #14335
• fix(cayenne): report runs per size tier when protected-snapshot compaction declines (fixes #13622) by @claudespice in #14476
• fix(cayenne): re-run a post-write compaction pass a concurrent append asked for (fixes #13906) by @claudespice in #14479
• fix(http): keep the request URL out of HTTP connector errors (fixes #13534) by @claudespice in #14490
• fix(runtime): load every localpod dataset that reads from one parent at startup (fixes #13087) by @claudespice in #14532
• fix(build): share one cargo metadata helper across the lint guards, so a broken cargo is never a violation (fixes #13121) by @claudespice in #14534
• fix(duckdb): push regexp_replace down again for a one-digit group reference, and refresh the Postgres ClickBench q35 plan (refs #13966) by @claudespice in #14552
• fix(duckdb): keep a cast into binary out of the federated plan (refs #14397) by @claudespice in #14633
• fix(runtime): keep finished async query jobs finished, and delete expired ones by @lukekim in #14585
• fix(federation): keep DataFusion's cast built-ins out of every federated plan (fixes #14444) by @claudespice in #14484
• test(cayenne): cover a swapped join filter through the sort-merge rewrite end to end (refs #14235) by @claudespice in #14551
• fix(runtime): decide every Spark/built-in function collision by name, and refuse an undecided one (fixes #14361) by @grokspice in #14400
• fix(ci): run every bin target's unit tests in the sign-off gate, and give spice connect its own --cloud-region refusal (fixes #13426) by @grokspice in #14406
• test(data_components): compile the federation unparser guards in every scoped test run (fixes #13625) by @grokspice in #14447
• fix(runtime-component): skip an inferred Cayenne index on a floating-point column instead of failing the load (fixes #14590) by @grokspice in #14599
• fix(cache): serve stale cached results during frequent table updates (stale_while_revalidate_ttl) by @sgrebnov in #14708
• Update Turso to 0.8.1 and retry write conflicts a metastore statement raises by @lukekim in #14680
• fix(snapshots): stop a snapshot dataset's load when a reload replaces it by @lukekim in #14673
• fix(cayenne): hand non-partition filters to every partition scan (fixes #12959) by @claudespice in #14370
• fix(postgres-accel): resolve secret references in the sidecar connection parameters (fixes #13296) by @claudespice in #14544
• fix(cayenne): drain the in-memory CDC tier before a full rewrite scans it (fixes #14450) by @claudespice in #14486
• feat(caching): warm SQL results cache on first refresh from persisted plan shapes by @lukekim in #14178
• fix(ci): retry only the container startup in the zero-retry MySQL CDC tests by @grokspice in #14727
• feat(key-index): immutable secondary index runs over compound Arrow keys by @bjchambers in #14592
• fix(federation): keep a fractional-to-integer cast out of the plans pushed to DuckDB, PostgreSQL, MySQL and BigQuery (fixes #14482) by @grokspice in #14601
• fix(testoperator): give accelerated bench configs a 300s ready_wait and name unready datasets on timeout (fixes #13973) by @claudespice in #14728
• fix(federation): keep arrow_typeof and its plan-introspection siblings local on every backend (fixes #14334) by @grokspice in #14695
• test(bench): refresh the tpch_q16 explain snapshots for the null-aware NOT IN plan (fixes #13977) by @claudespice in #14731
• ci: build trunk-push macOS legs on spiceai-macos-large again, keeping one-at-a-time coalescing by @grokspice in #14735
• ci(verus): derive the verified crates from cargo metadata; pin vstd once by @bjchambers in #14709
• docs: raise the test and evidence bar: differential first, exact assertions, performance always measured by @lukekim in #14723
• test(vortex): wait for a dropped segment cache to be freed before asserting it is gone (fixes #13295) by @claudespice in #14470
• ci(integration): report each integration part's own result in its required check by @grokspice in #14743
• fix(cayenne): back off futile bakes under a violated query goal too by @lukekim in #14736
• feat(object_store_occ): add transactional WAL and MVCC snapshots by @lukekim in #14732
• fix(snapshots): bootstrap readers from late snapshot publications by @phillipleblanc in #14468
• fix(cayenne): apply retention_sql to a mode: memory acceleration (fixes #14045) by @claudespice in #14344
• fix(duckdb, sqlite): keep the first copy of a key a write repeats under on_conflict: drop (fixes #14629) by @claudespice in #14748
• fix(cache): release a removed dataset's memory when the plan cache discards its plans (fixes #14251) by @claudespice in #14760
• build(deps): bump the aws-sdk group with 2 updates by @dependabot in #14764
• fix(llms): explain an Anthropic model's refusal of a sampling control instead of passing the bare 400 through (fixes #13564) by @grokspice in #14700
• build(deps-dev): bump dompurify by @dependabot in #14613
• fix(snapshots): allow cayenne_file_path and cayenne_metadata_dir on file_format: snapshot datasets (fixes #14696) by @sgrebnov in #14698
• test(snapshots): make snapshot integration tests robust under parallel CI runs by @sgrebnov in #14765
• feat(cayenne): keep the secondary index current across every write by @bjchambers in #14593
• fix: Update tpch benchmark snapshots for accelerated/file[parquet]-duckdb[memory].yaml by @app/github-actions in #14674
• Isolate Docker integration fixtures across concurrent processes by @phillipleblanc in #14639
• ci: run Substrait Mode A TPC-H at SF 1 on spiceai-macos in every merge-queue entry by @lukekim in #14522
• ci: stop repeating scheduled benchmarks: one source per accelerator, hosted sources weekly, a release commit once by @lukekim in #14681
• build(deps): bump nvidia/cuda by @dependabot in #14763
• fix(http): stop a non-2xx response body replacing an accelerated table's rows (fixes #13515) by @claudespice in #13538
• fix(chat-api): answer a tool-call-only assistant turn instead of panicking (fixes #13207) by @grokspice in #14232
• fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492) by @claudespice in #14670
• fix(spiceai, duckdb, sharepoint): name the Spicepod key a missing-parameter error asks for (refs #14446) by @claudespice in #14769
• Add table-bound ChangeSink ownership and backends by @phillipleblanc in #14704
• fix(ci): serialize rustup installs on shared macOS runners by @lukekim in #14717
• fix(cayenne): keep the per-shard PK index when the table-wide index is discarded by @lukekim in #14604
• Upgrade to DataFusion 55.1 and Arrow 59.3 by @krinart in #14612
• Update openapi.json by @app/github-actions in #14742
• build(deps): bump rustls from 0.23.40 to 0.23.45 by @dependabot in #14773
• Upgrade to iceberg-rust to 0.11.0 and DF 55.2 by @sgrebnov in #14771
• fix(cayenne): write an append into an empty unkeyed table as one load by @phillipleblanc in #14784
• fix(mysql_replication): don't re-send a live member's delivered commits after a reconnect by @lukekim in #14751
• ci(codeql): don't fail the SARIF upload when the merge queue already deleted its ref by @grokspice in #14793
• fix(acceleration): accept time_format iso8601 when the time_column is already a timestamp by @lukekim in #14777
• fix(cache): store SQL results that go stale mid-query by @lukekim in #14710
• fix(cayenne): serve filtered and global maintained aggregates by @lukekim in #14761
• perf(cayenne): build the checkpoint's tombstone union after releasing the capture locks by @lukekim in #14759
• ci: keep no artifacts or caches from PR and merge-queue checks, and run gating macOS jobs on spiceai-macos-large by @lukekim in #14804
• fix(cayenne): keep maintenance from deleting files a snapshot is archiving by @sgrebnov in #14789
• fix(snapshots): never overwrite shared snapshot metadata, and publish more than once to a file:// location by @lukekim in #14582
• perf(cdc): build deferred change rows in the reader while the apply runs by @lukekim in #14746
• fix: Update tpch benchmark snapshots for accelerated/on_zero_results/file[parquet]-cayenne[file]-on_zero_results.yaml by @app/github-actions in #14786
• test(testoperator): add a cold-start time-to-ready regression test by @phillipleblanc in #14785
• test(testoperator): make append tests more robust by @sgrebnov in #14817
• fix(runtime,data_components): wait for change-data-capture sources to record their positions before shutdown closes the accelerations (fixes #14523) by @grokspice in #14702
• fix(graphql): detect non-JSON responses and treat gateway errors as retryable by @lukekim in #14781
• fix(runtime): require keys for s3_auth: key, honor gs:// state location params, and leave newer rate-control state alone by @lukekim in #14584
• build(deps): bump hickory-resolver from 0.26.1 to 0.26.2 by @dependabot in #14774
• fix(listing): skip zero-byte objects (S3 folder markers) in format-selected listings by @sgrebnov in #14822
• Avoid unnecessary repartitioning in indexed dynamic-filter joins by @bjchambers in #14799
• feat(cayenne): keep one row per primary key automatically and deprecate on_conflict by @bjchambers in #14726
• fix(object-store): stop reading a response body at its first error by @phillipleblanc in #14831
• fix(ci): use byte-order collation in the benchmark Postgres container (refs #14815) by @sgrebnov in #14836
• fix(listing): keep a partition predicate as a residual filter on the _location fast path by @grokspice in #14790
• fix(ci): search only the unixodbc keg, not all of Homebrew's lib, in the macOS release builds by @grokspice in #14846
• fix(federation): keep date, timestamp and interval values local on SQLite reached through ADBC or ODBC (fixes #14753) by @claudespice in #14840
• chore: pin the fork at spiceai/datafusion#249 and guard the DataFusion fixes backported from 54 by @krinart in #14800
• Revert "fix(sqlite, duckdb): keep a decimal AVG, and on SQLite a decimal SUM, out of the federated plan (fixes #14492) (#14670)" by @krinart in #14825
• perf(runtime): skip EnsureRequirements while planning a point lookup by @lukekim in #14807
• fix(testoperator): report CH-benCH queries with no rows on either side as vacuous, not as matches by @lukekim in #14810
• feat(acceleration): make Cayenne the default acceleration engine by @phillipleblanc in #14837
• fix(cayenne): keep a join's LIMIT when the oversized-join rewrite makes it a sort-merge join by @lukekim in #14805
• test(bigquery): exclude subtrees an empty join build side never runs from the corpus job count (fixes #14848) by @sgrebnov in #14849
• fix(cayenne): fail contradictory write settings once, and order NULL times below every time by @bjchambers in #14847
• fix(runtime): serve an existing acceleration while its source is unavailable (fixes #14610) by @bjchambers in #14624
• Prune object-store file listing by metadata columns (#14264) by @Jeadie in #14303
• fix(vortex): fold partition values into the file-pruning predicate; test null-equal joins under mode:file by @lukekim in #14797
• feat(rate-control): adaptive, bounded and cluster-coordinated HTTP rate controls by @Jeadie in #14143
• fix(sql): align date_part('dow') with EXTRACT(dow) Sunday=0 by @lukekim in #14796
• fix(runtime): carry API-key principal into MCP tools/call by @lukekim in #14828
• perf(cayenne): hold maintained-aggregate retraction state in a compact shared index by @lukekim in #14762
• fix(deps): keep a hash join's order once it has a dynamic filter, so vector_search plans again by @Jeadie in #14857
• fix(cayenne): fence schema statistics and control maintenance tests by @bjchambers in #14856
• fix(github): retry nested GraphQL pages in place and fail incomplete nested connections by @sgrebnov in #14862
• fix(cayenne): load an append dataset that has retention_sql and orders versions by time by @phillipleblanc in #14878
• fix(cdc): Cayenne replication needs only a primary key, not on_conflict by @phillipleblanc in #14880
• feat(connector-huggingface): Hugging Face datasets data connector (hf://datasets/...) by @lukekim in #14877
• fix(iceberg): Iceberg REST clients read the real table or get an error, never an empty one by @lukekim in #14588

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.3.2...v2.4.0-rc.1

Spice v2.3.2 is now available! ⚡

Spice v2.3.2 makes point lookups and repeated queries faster with Spice Cayenne. SQL results-cache improvements apply to every cached query, regardless of its data source or accelerator. Across two benchmark rounds, the time for a small cache hit fell by 50-52% over HTTP and 44-48% over Flight SQL. For small cache hits, CPU time per server request fell by 58-61% over HTTP and 49-51% over Flight SQL.

Highlights in v2.3.2 include:

• Indexed Point Lookups — Cayenne now honors indexes, so a lookup reads the matching rows instead of scanning
• Cayenne Data Layout — cayenne_cluster_by groups related rows together across storage tiers
• Faster Queries After a Full Refresh — refreshed tables are written so that filtered queries read fewer files
• Faster Cached Responses — these improvements apply to every query in the SQL results cache, regardless of its data source or accelerator. For small results, cache-hit time fell by 44-52%
• Faster Repeat Queries — Cayenne reuses its prepared view of a table until the data changes
• Localpod Fix — a localpod dataset no longer serves data its source has replaced
• Catalog Fix — tables no longer go missing when datasets load at the same time
• Safer Cayenne Configuration — an ambiguous metastore configuration is reported instead of appearing as lost data
• Distributed HTTP Queries — async distributed queries can read HTTP datasets

What's New in v2.3.2​

Faster Point Lookups with Cayenne Indexes​

Every other accelerator already accepted indexes, and Cayenne logged that it ignored them. Cayenne now builds an index for each entry, in both mode: file and mode: memory:

acceleration:
  engine: cayenne
  mode: file # or memory
  indexes:
    '(TenantId, ServiceId)': enabled

A query that pins every column of an index to a value, such as WHERE TenantId = 7 AND ServiceId = 'a', now reads the matching rows directly. Without an index, Cayenne can only skip files whose minimum and maximum values rule the key out, which rarely helps when related rows are spread across the table — a lookup on a 5.4M-row test dataset had to open about half its files for nearly every key.

Indexes only reduce what a query reads, so results are identical either way. A query that uses a range, an IN list, an OR, or a cast on the indexed column reads the table as before. Index definitions are not stored with the table, so adding or removing one takes effect the next time the dataset loads. Floating-point columns cannot be indexed and are reported at load time.

EXPLAIN shows whether a query used an index, and the cayenne_lookup_index_probe_total metric counts lookups by outcome.

Control How Cayenne Lays Out Data​

cayenne_cluster_by stores rows with similar values near each other, so a filtered query reads fewer files. It applies to every storage tier, replacing cayenne_datalake_clustering_columns, which affected only the coldest tier.

acceleration:
  engine: cayenne
  params:
    cayenne_cluster_by: 'tenant_id, event_time'

CREATE TABLE ... CLUSTER BY (column, ...) is also supported, and several cases it previously rejected — a single column in parentheses, and names whose capitalization differs from the column definition — now work. A column name that does not exist is reported when the dataset loads rather than failing later.

Faster Queries After a Full Refresh​

A full refresh previously spread each key across every file it wrote, so a filtered query had to open all of them even when it wanted a single row. A refreshed table is now written so that each file holds a distinct range of the data, and queries filtering on that range read only the files that can match.

This needs no configuration and applies to any dataset whose accelerated table is replaced by a refresh. Datasets that already set cayenne_sort_columns or cayenne_cluster_by keep their existing layout, and the very first load is unchanged.

Faster Cached Query Responses​

Before this release, the runtime planned a query before it checked the SQL results cache. It also repeated other work that a cache hit does not need. The runtime now checks the cache first and returns cached answers directly. These improvements apply to every cached query, regardless of its data source or accelerator.

Across rounds of benchmarking queries that returned a seven-row GROUP BY result, cache-hit time over HTTP fell by 50-52%. Over Flight SQL, cache-hit time fell by 44-48%. In one HTTP round, cache-hit time fell from 63.8 µs to 30.8 µs. In one round over Flight SQL, it fell from 55.3 µs to 30.9 µs.

With task history enabled, the server used 125 µs of CPU time for a small HTTP hit before the change. It used 49 µs after the change. For a small result over Flight SQL, the server used 330 µs before the change. It used 168 µs after the change. These changes reduced CPU time by 61% and 49%, respectively.

For a wide result over HTTP, the server used 241 µs of CPU time before the change. It used 116 µs after the change. This is a 52% reduction. Wide results did not reduce the runtime's cache-hit time in those runs. Some measurements were slower.

Raw cache entries now share each stored batch with streaming JSON HTTP and Flight SQL responses. Buffered HTTP formats — including CSV, plain text, vnd.* envelopes, and JSON with union columns — still clone batches while encoding.

A stream benchmark with eight 20-column batches fell from 1.8722 µs to 226.23 ns. This is an 8.3x speedup. One 200-column batch fell from 2.0190 µs to 131.94 ns. This is a 15x speedup. These figures measure the construction and full consumption of the stream. They do not measure query latency from start to finish.

For compressed entries, the first read keeps the entry compressed. The second read promotes it to raw data when the raw size fits the cache. Later reads avoid decompression. In a cache benchmark, a later read took 17.1-17.8 µs across three payload sizes. A decode took 39.1-154.8 µs. Cached answers, expiry times, and cache settings are unchanged.

Faster Repeat Queries on Cayenne​

Cayenne prepares a view of a table's current contents before it can answer a query. It now reuses that preparation until the data actually changes, instead of rebuilding it, so repeated queries against a table that is not being written return faster and stop querying the metastore entirely once warm.

Datasets fed by continuous change data capture — a cdc: or debezium: source, or refresh_mode: changes — reuse the preparation for up to one second so a burst of incoming changes can share it. Datasets that accept writes always see their own writes immediately. No configuration is required.

Better Pruning for Decimal, Binary, and Time Columns​

Cayenne did not record minimum and maximum values for decimal, binary, time, and 16-bit float columns, so queries filtering on them could not skip files and had to read more data than necessary. Those columns now carry the same statistics as every other type, and clustering on a decimal column works as intended.

After a change that widens a decimal column's scale, Cayenne discards the affected statistics and rebuilds them, so queries read a little more until that completes. Results are unaffected.

Localpod Datasets Stay in Sync with Their Source​

A localpod dataset reads through another dataset in the same Spicepod. When the source dataset was reloaded — because its configuration changed — the localpod dataset kept reading the replaced copy, so it answered with data the source no longer had, and both copies kept refreshing.

A localpod dataset now reloads whenever the dataset it reads through does, including through several levels of chaining, and its cached results are cleared at the same time. This also works when the source is named with its full path, such as localpod:spice.public.parent. Fixes #3288.

Catalog Tables No Longer Go Missing​

When two datasets in the same schema loaded at the same time, one could be silently discarded, and every later query against it failed with Table not found. Both datasets reported that they had loaded successfully, and which one went missing varied between restarts. Datasets that share a schema now always both register.

Cayenne Detects an Ambiguous Metastore at Startup​

Cayenne datasets that each set a different cayenne_file_path, without a shared cayenne_metadata_dir, could open the wrong metadata directory after a restart. Cayenne then started up empty even though the data was still on disk, which looks like a total loss of accelerated data.

This configuration is now rejected at startup, naming each dataset and path involved and linking to the documentation. If a Spicepod uses several Cayenne data paths, set the same cayenne_metadata_dir on each dataset before upgrading.

Distributed Queries over HTTP Datasets​

An async distributed query submitted to /v1/queries failed if it read an unaccelerated HTTP dataset. These queries now run, with credentials, headers, and pagination behaving as they do for a non-distributed query. Fixes #14104.

Other Fixes​

• Partitioned datasets: a full refresh of a partitioned dataset only replaced the partitions the new data reached, so rows deleted at the source stayed queryable, and a refresh that returned no rows changed nothing. Every partition is now replaced.
• Iceberg write-through: a write to a partitioned dataset could deadlock against a refresh running at the same time, leaving both waiting.
• BigQuery: a case-insensitive LIKE could fail the query or return the wrong rows, because BigQuery has no ILIKE. Spice now evaluates it locally. Ordinary LIKE is unchanged.
• PostgreSQL: timestamps written back to PostgreSQL lost everything below the second. Microseconds are now preserved.

Dependency Updates​

No crate versions changed in this release. DataFusion remains at v54.1.0, Arrow remains at v58.3.0, and Vortex remains at v0.79.0.

Spice updates two fork revisions: datafusion-table-providers for the BigQuery and PostgreSQL fixes above, and duckdb-rs so the bundled DuckDB builds against the macOS 27 SDK.

Contributors​

• @lukekim
• @phillipleblanc

Breaking Changes​

cayenne_datalake_clustering_columns is replaced by cayenne_cluster_by. Rename the parameter before upgrading. The new one groups data on every storage tier, not only the coldest:

acceleration:
  engine: cayenne
  params:
    cayenne_cluster_by: 'tenant_id, event_time'

A dataset that sets both cayenne_sort_columns and a cluster key is now rejected when it loads. Remove cayenne_sort_columns to keep the cluster key.

Cayenne datasets using several data paths must share a metastore. If your Cayenne datasets set different cayenne_file_path values, set the same cayenne_metadata_dir on each one before upgrading. Spice now refuses to start on this configuration instead of risking an empty-looking acceleration:

acceleration:
  engine: cayenne
  params:
    cayenne_file_path: /mnt/a/cayenne
    cayenne_metadata_dir: /mnt/shared/cayenne-metadata

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.3.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.3.2 image:

docker pull spiceai/spiceai:2.3.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.3.2

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• fix(postgres): preserve microseconds in timestamp writeback by @phillipleblanc in #13963
• feat(cayenne): reuse ScanView until write, lag only for read-only CDC by @lukekim in #14055
• perf: serve results-cache hits where the request arrives and cut per-hit overhead by @lukekim in #14103
• fix(cluster): support distributed HTTP scans by @phillipleblanc in #14108
• fix(bigquery): keep ILIKE evaluation local by @phillipleblanc in #14110
• feat(cayenne): cluster warm and datalake tiers, and write full refreshes as key-range files by @lukekim in #14124
• fix(cayenne): reject ambiguous metastore paths by @phillipleblanc in #14130
• fix(cayenne): round-trip decimal, binary, and time stats and drop them on scale change by @lukekim in #14139
• feat(cayenne): build secondary indexes from indexes in file and memory mode by @phillipleblanc in #14149
• fix(catalog): keep the tables registered under an existing schema by @phillipleblanc in #14193
• ci: stop exporting empty AWS and DuckLake endpoints to the schema test by @phillipleblanc in #14194
• perf(cache): promote encoded SQL results to raw after the second decode by @lukekim in #14199
• perf(cache): tighten the Raw SQL results-cache serve path by @lukekim in #14205
• fix(runtime): reload a localpod dataset when the dataset it reads through is reloaded (fixes #3288) by @claudespice in #14208

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.3.1...v2.3.2

Announcing the release of Spice v1.10.0! ⚡

Spice v1.10.0 introduces a new Caching Acceleration Mode with stale-while-revalidate (SWR) semantics for disk-persisted, low-latency queries with background refresh. This release also adds the TinyLFU eviction policy for the SQL results cache, a preview of the DynamoDB Streams connector for real-time CDC, S3 location predicate pruning for faster partitioned queries, improved distributed query execution, and multiple security hardening improvements.

What's New in v1.10.0​

Caching Acceleration Mode​

Low-Latency Queries with Background Refresh: This release introduces a new caching acceleration mode that implements the stale-while-revalidate (SWR) pattern. Queries return cached results immediately while data refreshes asynchronously in the background, eliminating query latency spikes during refresh cycles. Cached data persists to disk using DuckDB, SQLite, or Cayenne file modes.

Key Features:

• Stale-While-Revalidate (SWR): Returns cached data immediately while refreshing in the background, reducing query latency
• Disk Persistence: Cached results persist across restarts using DuckDB, SQLite, or Cayenne file modes
• Configurable Refresh: Control refresh intervals with refresh_check_interval to balance freshness and source load

Recommendation: Use retention configuration with caching acceleration to ensure stale data is cleaned up over time.

Example spicepod.yaml configuration:

datasets:
  - from: http://localhost:7400
    name: cached_data
    time_column: fetched_at
    acceleration:
      enabled: true
      engine: duckdb
      mode: file # Persist cache to disk
      refresh_mode: caching
      refresh_check_interval: 10m
      retention_check_enabled: true
      retention_period: 24h
      retention_check_interval: 1h

For more details, refer to the Data Acceleration Documentation.

TinyLFU Cache Eviction Policy​

Higher Cache Hit Rates for SQL Results Cache: A new TinyLFU cache eviction policy is now available for the SQL results cache. TinyLFU is a probabilistic cache admission policy that maintains higher hit rates than LRU while keeping memory usage predictable, making it ideal for workloads with varying query frequency patterns.

Example spicepod.yaml configuration:

runtime:
  caching:
    sql_results:
      enabled: true
      eviction_policy: tiny_lfu # default: lru

For more details, refer to the Caching Documentation and the Moka TinyLFU Documentation for details of the algorithm.

DynamoDB Streams Data Connector (Preview)​

Real-Time Change Data Capture for DynamoDB: The DynamoDB connector now integrates with DynamoDB Streams for real-time change data capture (CDC). This enables continuous synchronization of DynamoDB table changes into Spice for real-time query, search, and LLM-inference.

Key Features:

• Real-Time CDC: Automatically captures inserts, updates, and deletes from DynamoDB tables as they occur
• Table Bootstrapping: Performs an initial full table scan before streaming changes, ensuring complete data consistency
• Acceleration Integration: Works with refresh_mode: changes to incrementally update accelerated datasets

Note: DynamoDB Streams must be enabled on your DynamoDB table. This feature is in preview.

Example spicepod.yaml configuration:

datasets:
  - from: dynamodb:my_table
    name: orders_stream
    acceleration:
      enabled: true
      refresh_mode: changes # Enable Streams capture

For more details, refer to the DynamoDB Connector Documentation.

OpenTelemetry Metrics Exporter​

Spice can now push metrics to an OpenTelemetry collector, enabling integration with platforms such as Jaeger, New Relic, Honeycomb, and other OpenTelemetry-compatible backends.

Key Features:

• Protocol Support: Supports the gRPC (default port 4317) protocol
• Configurable Push Interval: Control how frequently metrics are pushed to the collector

Example spicepod.yaml configuration for gRPC:

runtime:
  telemetry:
    enabled: true
    otel_exporter:
      endpoint: 'localhost:4317'
      push_interval: '30s'

For more details, refer to the Observability & Monitoring Documentation.

S3 Connector Improvements​

S3 Location Predicate Pruning: The S3 data connector now supports location-based predicate pruning, dramatically reducing data scanned by pushing down location filter predicates to S3 listing operations. For partitioned datasets (e.g., year=2025/month=12/), Spice now skips listing irrelevant partitions entirely, significantly reducing query latency and S3 API costs.

AWS S3 Tables Write Support: Full read/write capability for AWS S3 Tables, enabling direct integration with AWS's managed table format for S3. Use standard SQL INSERT INTO to write data.

For more details, refer to the S3 Data Connector Documentation and Glue Data Connector Documentation.

Faster Distributed Query Execution​

Distributed query planning and execution have been significantly improved:

• Fixed executor registration in cluster mode for more reliable distributed deployments
• Improved hostname resolution for Flight server binding, enabling better executor discovery
• Distributed accelerator registration: Data accelerators now properly register in distributed mode
• Optimized query planning: DistributeFileScanOptimizer improvements for faster planning with large datasets

For more details, refer to the Distributed Query Documentation.

Search Improvements​

Search capabilities have been improved with several performance and reliability enhancements:

• Fixed FTS query blocking: Full-text search queries no longer block unnecessarily, improving query responsiveness
• Optimized vector index operations: Eliminated unnecessary list_vectors calls for better performance
• Improved limit pushdown: IndexerExec now properly handles limit pushdown for more efficient searches

For more details, refer to the Search Documentation.

Security Hardening​

Multiple security improvements have been implemented:

• SQL Identifier Quoting: Hardened SQL identifier quoting across all database connectors (PostgreSQL, MySQL, DuckDB, etc.) to prevent SQL injection attacks through table or column names
• Token Redaction: Sensitive authentication tokens are now fully redacted in debug and error output, preventing accidental credential exposure in logs
• Path Traversal Prevention: Fixed tar extraction operations to prevent directory traversal vulnerabilities when processing archived files
• Input Sanitization: Added strict validation for top_n_sample order_by clause parsing to prevent injection attacks
• Glue Credential Handling: Prevented automatic loading of AWS credentials from environment in Glue connector, ensuring explicit credential configuration

Developer Experience Improvements​

• Health probe metrics: Added health probe latency metrics for better observability
• CLI improvements: Fixed .clear history command in the REPL to fully clear persisted history

Contributors​

• @lukekim
• @sgrebnov
• @peasee
• @krinart
• @Jeadie
• @mach-kernel
• @phillipleblanc
• @ewgenius

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates.

The Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.10.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.10.0 image:

docker pull spiceai/spiceai:1.10.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

• Test-operator: Add tpcds_q8 to the default row-count validation skip list by @sgrebnov in #8185
• fix: Remove unwrap_used from test by @peasee in #8212
• Run glue_iceberg_integration_test_catalog as part of main integration tests by @sgrebnov in #8222
• Add TPCH sf100 testoperator spicepods with dispatch by @Jeadie in #8192
• Build with CPU native flags by @lukekim in #8224
• fix: Apply assertion clippy in CI/Makefile only by @peasee in #8229
• feat: Support running queries only in testoperator by @peasee in #8211
• DuckDB query planning: aggregate pushdown by @mach-kernel in #8174
• install.sh improvements by @lukekim in #8252
• Fix .clear history by @lukekim in #8254
• Improve the output of dataset loading by @lukekim in #8256
• Refactor view validation by @lukekim in #8258
• Upgrade AWS crates by @lukekim in #8259
• fix: Pushdown dynamic filters to partition scans by @peasee in #8240
• Harden SQL identifier quoting in connectors by @phillipleblanc in #8276
• Cayenne sort_columns on insert by @lukekim in #8091
• Redact token debug output by @phillipleblanc in #8280
• fix: Cayenne configuration options by @lukekim in #8281
• Prevent path traversal in untar by @phillipleblanc in #8284
• Fix cluster mode executor registration by @mach-kernel in #8292
• Unignore s3_vectors_kafka_stream test by @Jeadie in #8289
• Post-release house keeping by @krinart in #8293
• Improve generate_changelog script by @krinart in #8273
• Acceleration mode caching by @lukekim in #8237
• Sanitization and security checks by @lukekim in #7854
• Add health probe latency metric by @phillipleblanc in #8300
• Add distributed registration for data accelerators by @phillipleblanc in #8299
• Pass IndexedTableProvider down in 'changes_stream' and 'append_stream' by @Jeadie in #8295
• Add dynamodb-streams crate by @krinart in #8283
• Distributed query: resolve executor hostname when determining Flight server binding by @mach-kernel in #8304
• Return computed embeddings from index for partitioned S3Vectors by @Jeadie in #8306
• [DDB Streams] Skeleton for DynamoDB Streams by @krinart in #8296
• DistributeFileScanOptimizer: Improve planning performance by @mach-kernel in #8305
• feat: Add an ExactLeftAccumulator implementation by @peasee in #8302
• deps: Upgrade Vortex to 0.56 by @peasee in #8311
• DynamoDB table bootstrapping + streaming by @krinart in #8312
• Avoid calling S3Vector list_vectors (or equivalent) when indexing into VectorIndexs by @Jeadie in #8282
• Add on_conflict testing support to append benchmark by @sgrebnov in #8314
• docker: Add valid home directory to fix duckdb extension loading issue by @phillipleblanc in #8318
• Add GH Workflow to run Append benchmark test by @sgrebnov in #8321
• Exclude MySQL SF100 from test-operator dispatch by @sgrebnov in #8320
• feat: Update clippy lints by @peasee in #8317
• Add S3 location predicate pruning to listing connector by @phillipleblanc in #8319
• Review feedback for caching mode accelerator by @phillipleblanc in #8326
• Also include Dockerfile home changes for release build by @phillipleblanc in #8327
• Change communication channel from Discord to Slack by @Jeadie in #8330
• Replace Discord link with Slack link in README by @Jeadie in #8331
• fix(glue): Prevent OpenDAL from automatic loading of AWS credentials from environment by @sgrebnov in #8337
• Block on index read for FTS queries by @Jeadie in #8339
• Fix search query provider by @Jeadie in #8343
• Support for writing into AWS S3 Tables by @sgrebnov in #8344
• Acceleration file_create mode by @lukekim in #8347
• Don't block on lock in FTS query path by @Jeadie in #8348
• feat: Add an optimizer rule to replace join accumulator for Cayenne by @peasee in #8316
• S3 Vectors limit updates by @lukekim in #8352
• Sanitize top_n_sample order_by parsing by @phillipleblanc in #8356
• Add distributed registration for data connectors by @phillipleblanc in #8354
• Improve IndexerExec to properly handle limit pushdown by @sgrebnov in #8366
• Fix Cayenne partition_by metadata flaky integration test by @phillipleblanc in #8367
• Rework caching accelerator to use the stale-while-revalidate pattern. by @phillipleblanc in #8365
• Add TinyLFU caching policy by @lukekim in #8370
• Make Arrow acceleration on_conflict verification more robust by @sgrebnov in #8375
• Add additional test for verify_on_conflict_matches_primary_key (Arrow acceleration) by @sgrebnov in #8376
• Add v1.10.0-rc1 release notes by @mach-kernel in #8373
• docs: Remove DuckDB agg pushdown from release notes by @peasee in #8383
• Testoperator dispatch: add Append support and test configurations by @sgrebnov in #8360
• fix: Increase TPCDS DuckDB connection pool size by @peasee in #8386
• fix: Update benchmark snapshots by @app/github-actions in #8385
• fix: Update benchmark snapshots by @app/github-actions in #8389
• Fix Windows build by @phillipleblanc in #8391
• DuckDB aggregate pushdown: fix partitioning and schema rewrite bugs by @mach-kernel in #8397
• Delta table: Store current snapshot ref with table instance by @mach-kernel in #8358
• GetAppDefinition: Check if executor is part of cluster by @mach-kernel in #8396
• 1.10.0-rc1 housekeeping by @mach-kernel in #8394
• Change debug log to warning for vector engine config by @Jeadie in #8378
• Clarify /v1/nsql datasets sampling hint by @phillipleblanc in #8395
• Use bmi1 target feature for x86_64 by @phillipleblanc in #8401
• benchmarks: Default to update snapshots when run on a non-release branch by @phillipleblanc in #8402
• Update threat model for v1.9.2 by @phillipleblanc in #8400
• Fix iceberg tables metadata - assign ids to all fields, including nested by @ewgenius in #8351
• Fix databricks_spark_connect_m2m_integration_test_catalog snapshot by @ewgenius in #8403
• Move all GitHub Actions workflows to use exact commit sha by @phillipleblanc in #8409
• Upgrade datafusion-tableproviders (df v50) by @lukekim in #8261
• Batching for CDC by @krinart in #8359
• fix: make DuckDB attachments logic more robust by @sgrebnov in #8411
• Persistent checkpoints for DynamoDB Streams by @krinart in #8345
• Distributed query: Support AsyncFuncExec and Spice UDFs in Ballista by @mach-kernel in #8414
• Pin GitHub Actions to fix Testoperator build action by @sgrebnov in #8416
• Watermarks support for DynamoDB by @krinart in #8417
• Fix typo in .vscode/launch.json by @sgrebnov in #8415
• DuckSqlExec: Update equivalence properties when rewriting schema by @phillipleblanc in #8420
• Validate that the commit for datafusion-table-providers exists on the spiceai branch by @phillipleblanc in #8421
• Append Tests: add support for retention testing by @sgrebnov in #8419
• New crate google-genai by @Jeadie in #8390
• federation: Improve error message and add debug logging for cast failures by @phillipleblanc in #8422
• DynamoDB Streams Error Handling by @krinart in #8418
• Append tests: add support for with_retention_data to dispatch by @sgrebnov in #8430
• Append tests: add support for test metrics reporting by @sgrebnov in #8432
• test-operator: fix metrics reporting by @sgrebnov in #8435
• Follow-up improvements and bug fixes by @krinart in #8433
• Periodic snapshots for append/changes streams by @krinart in #8407
• Add support for caching_stale_if_error to caching accelerator; fix multiple upstream requests during SWR; fix Arrow accelerator by @phillipleblanc in #8425
• Disable dataset health monitor for dynamic HTTP connector by @phillipleblanc in #8441
• Metrics + snapshots_trigger_threshold for DynamoDB Streams by @krinart in #8437
• Clear the in-flight revalidations cache after a revalidation has completed. by @phillipleblanc in #8443
• dont build 'spicepod-validator' on 'make install' by @Jeadie in #8426
• fix: Update benchmark snapshots by @app/github-actions in #8436
• fix: Disable Cayenne HashJoin rewriter optimizer by @peasee in #8439
• Testoperator: add duckdb-partitioned query override by @sgrebnov in #8446
• Add a check to validate that results cache SWR and caching accelerator SWR are not both set. by @phillipleblanc in #8445
• OTel exporter for push metrics by @lukekim in #8442
• fix: Update benchmark snapshots by @app/github-actions in #8448
• Add snapshot creation logging by @krinart in #8469
• Fix PeriodicReader panic by @krinart in #8471
• fix: Pin CUDA build actions to commits by @peasee in #8477
• DuckDB agg pushdown: gate behind accelerator parameter by @mach-kernel in #8474
• Rename aggregate_pushdown_optimization -> optimizer_duckdb_aggregate_pushdown by @ewgenius in #8485
• Handle throttling exception for DynamoDB streams by @phillipleblanc in #8492

Announcing the release of Spice v1.10.0-rc.1! ⚡

v1.10.0-rc1 is a release candidate for early testing of v1.10 features including an all new caching acceleration mode, tiny_lfu caching policy, a new DynamoDB Streams connector (Preview), improvements to the DynamoDB connector, faster distributed query execution, S3 connector improvements, and security hardening for v1.10.0-stable.

What's New in v1.10.0-rc1​

Caching Acceleration Mode with SWR and TinyLFU​

This release introduces a new caching acceleration mode that implements the stale-while-revalidate (SWR) pattern using Data Accelerators such as DuckDB or Cayenne, enabling queries to return file-persisted cached results immediately while asynchronously refreshing data in the background. Combined with the new TinyLFU cache eviction policy, Spice can now maintain higher cache hit rates while keeping memory usage predictable.

Key Features:

• Stale-While-Revalidate (SWR): Returns cached data immediately while refreshing in the background
• Data Accelerator Support: Cached accelerators can persist data to disk using DuckDB, SQLite, or Cayenne file modes.
• TinyLFU Cache Policy: Probabilistic cache admission policy that maintains high hit rates with minimal overhead
• Predictable Memory Usage: Configurable memory limits with automatic eviction of less frequently used entries

Example Spicepod.yml configuration:

runtime:
  caching:
    sql_results:
      enabled: true
      eviction_policy: tiny_lfu # default lru

datasets:
  - from: s3://my-bucket/data.parquet
    name: cached_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file # Persist cache to disk
      refresh_mode: caching
      refresh_check_interval: 10m

For more details, refer to the Data Acceleration Documentation and Caching Documentation.

DynamoDB Streams Data Connector in Preview​

DynamoDB Connector now integrates with DynamoDB Streams which enables real-time streaming with support for both table bootstrapping and continuous change data capture (CDC). This connector automatically detects changes in DynamoDB tables and streams them into Spice for real-time query, search, and LLM-inference.

Key Features:

• Real-Time CDC: Automatically captures inserts, updates, and deletes from DynamoDB tables
• Table Bootstrapping: Initial full table load before streaming changes

Example Spicepod.yml configuration:

datasets:
  - from: dynamodb:my_table
    name: orders_stream
    acceleration:
      enabled: true
      refresh_mode: changes

For more details, refer to the DynamoDB Connector Documentation.

Cayenne Accelerator Enhancements​

The Cayenne data accelerator now supports:

• Sort Columns Configuration: Optimize inserts by pre-sorting data on specified columns for improved query performance

Example Spicepod.yml configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: sorted_data
    acceleration:
      enabled: true
      engine: cayenne
      mode: file_create
      params:
        sort_columns: timestamp,region

For more details, refer to the Cayenne Documentation.

S3 Connector Improvements​

S3 Location Predicate Pruning: The S3 data connector now supports location-based predicate pruning, dramatically reducing data scanned by pushing down predicates to S3 listing operations. This optimization is especially effective for partitioned datasets stored in S3.

AWS S3 Tables Write Support: Full read/write capability for AWS S3 Tables, enabling fast integration with AWS's table format for S3.

For more details, refer to the S3 Tables Data Connector Documentation and Glue Data Connection Documentation.

Faster Distributed Query Execution​

Distributed query planning and execution have been significantly improved:

• Fixed executor registration in cluster mode for more reliable distributed deployments
• Improved hostname resolution for Flight server binding, enabling better executor discovery
• Distributed accelerator registration: Data accelerators now properly register in distributed mode
• Optimized query planning: DistributeFileScanOptimizer improvements for faster planning with large datasets

For more details, refer to the Distributed Query Documentation.

Search Improvements​

Search capabilities have been improved with several performance and reliability enhancements:

• Fixed FTS query blocking: Full-text search queries no longer block unnecessarily, improving query responsiveness
• Optimized vector index operations: Eliminated unnecessary list_vectors calls for better performance
• Improved limit pushdown: IndexerExec now properly handles limit pushdown for more efficient searches

For more details, refer to the Search Documentation.

Security Hardening​

Multiple security improvements have been implemented:

• SQL identifier quoting: Hardened SQL identifier quoting across all connectors to prevent injection attacks
• Token redaction: Sensitive tokens are now fully redacted in debug output to prevent credential leakage
• Path traversal prevention: Fixed tar extraction to prevent path traversal vulnerabilities
• Input sanitization: Added validation for top_n_sample order_by parsing
• Improved credential handling: Improved credential management in Glue connector

Developer Experience Improvements​

• Health probe metrics: Added health probe latency metrics for better observability
• CLI improvements: Fixed .clear history command in the REPL to fully clear persisted history

Contributors​

• @lukekim
• @sgrebnov
• @peasee
• @krinart
• @Jeadie
• @mach-kernel
• @phillipleblanc
• @ewgenius

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates. The Spice Cookbook still offers 82+ recipes to help you prototype quickly.

Upgrading​

To try v1.10.0-rc1, use one of the following methods:

CLI:

spice upgrade --version 1.10.0-rc1

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.10.0-rc1 image:

docker pull spiceai/spiceai:1.10.0-rc1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 1.10.0-rc1

AWS Marketplace:

🎉 Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

• Test-operator: Add tpcds_q8 to the default row-count validation skip list by @sgrebnov in #8185
• fix: Remove unwrap_used from test by @peasee in #8212
• Run glue_iceberg_integration_test_catalog as part of main integration tests by @sgrebnov in #8222
• Add TPCH sf100 testoperator spicepods with dispatch by @Jeadie in #8192
• Build with CPU native flags by @lukekim in #8224
• Make copilot check for empty copyright header by @krinart in #8245
• fix: Apply assertion clippy in CI/Makefile only by @peasee in #8229
• feat: Support running queries only in testoperator by @peasee in #8211
• DuckDB query planning: aggregate pushdown by @mach-kernel in #8174
• install.sh improvements by @lukekim in #8252
• Fix .clear history by @lukekim in #8254
• fix: Pushdown dynamic filters to partition scans by @peasee in #8240
• Harden SQL identifier quoting in connectors by @phillipleblanc in #8276
• Cayenne sort_columns on insert by @lukekim in #8091
• Redact token debug output by @phillipleblanc in #8280
• fix: Cayenne configuration options by @lukekim in #8281
• Prevent path traversal in untar by @phillipleblanc in #8284
• Fix cluster mode executor registration by @mach-kernel in #8292
• Unignore s3_vectors_kafka_stream test by @Jeadie in #8289
• Post-release house keeping by @krinart in #8293
• Improve generate_changelog script by @krinart in #8273
• Acceleration mode caching by @lukekim in #8237
• Sanitization and security checks by @lukekim in #7854
• Add health probe latency metric by @phillipleblanc in #8300
• Add distributed registration for data accelerators by @phillipleblanc in #8299
• Pass IndexedTableProvider down in 'changes_stream' and 'append_stream' by @Jeadie in #8295
• Add dynamodb-streams crate by @krinart in #8283
• Distributed query: resolve executor hostname when determining Flight server binding by @mach-kernel in #8304
• Return computed embeddings from index for partitioned S3Vectors by @Jeadie in #8306
• [DDB Streams] Skeleton for DynamoDB Streams by @krinart in #8296
• DistributeFileScanOptimizer: Improve planning performance by @mach-kernel in #8305
• feat: Add an ExactLeftAccumulator implementation by @peasee in #8302
• deps: Upgrade Vortex to 0.56 by @peasee in #8311
• DynamoDB table bootstrapping + streaming by @krinart in #8312
• Avoid calling S3Vector list_vectors (or equivalent) when indexing into VectorIndexs by @Jeadie in #8282
• Add on_conflict testing support to append benchmark by @sgrebnov in #8314
• docker: Add valid home directory to fix duckdb extension loading issue by @phillipleblanc in #8318
• Add GH Workflow to run Append benchmark test by @sgrebnov in #8321
• Exclude MySQL SF100 from test-operator dispatch by @sgrebnov in #8320
• feat: Update clippy lints by @peasee in #8317
• Add S3 location predicate pruning to listing connector by @phillipleblanc
• Review feedback for caching mode accelerator by @phillipleblanc in #8326
• Also include Dockerfile home changes for release build by @phillipleblanc in #8327
• Change communication channel from Discord to Slack by @Jeadie in #8330
• Replace Discord link with Slack link in README by @Jeadie in #8331
• fix(glue): Prevent OpenDAL from automatic loading of AWS credentials from environment by @sgrebnov in #8337
• Block on index read for FTS queries by @Jeadie in #8339
• Fix search query provider by @Jeadie in #8343
• Support for writing into AWS S3 Tables by @sgrebnov in #8344
• Acceleration file_create mode by @lukekim in #8347
• Don't block on lock in FTS query path by @Jeadie in #8348
• feat: Add an optimizer rule to replace join accumulator for Cayenne by @peasee in #8316
• S3 Vectors limit updates by @lukekim in #8352
• Sanitize top_n_sample order_by parsing by @phillipleblanc in #8356
• Update version to v1.10.0-rc.1 by @ewgenius in #8362
• Improve IndexerExec to properly handle limit pushdown by @sgrebnov in #8366
• Fix Cayenne partition_by metadata flaky integration test by @phillipleblanc in #8367
• Rework caching accelerator to use the stale-while-revalidate pattern. by @phillipleblanc in #8365
• Add TinyLFU caching policy by @lukekim in #8370

Announcing the release of Spice v1.9.1!🔥

v1.9.1 introduces Amazon Bedrock Nova 2 Multimodal embeddings support with high-dimensional vectors up to 3,072 dimensions and purpose-optimized embeddings for semantic search and retrieval operations, DynamoDB timestamp filter pushdown for more efficient append-mode acceleration with configurable time formatting, HTTP Data Connector health probe configuration for improved endpoint validation reliability, and Spice .NET SDK v0.2 with expanded .NET version support and updated gRPC libraries. This release focuses on bug fixes, stability, and performance improvements.

Amazon Bedrock Nova 2 Multimodal embeddings​

Spice now supports the Amazon Nova 2 Multimodal embeddings models via the Bedrock models provider, enabling high-quality text embeddings for semantic search and vector similarity operations. The Nova embeddings model offers configurable dimensions and advanced features like truncation modes and embedding purpose optimization.

Key Features:

• High-Dimensional Embeddings: Support for up to 3,072 dimensions for rich semantic representations
• Configurable Truncation: Control how input text is truncated when exceeding token limits (START, END, or NONE)
• Purpose Optimization: Optimize embeddings for specific use cases (GENERIC_INDEX, GENERIC_RETRIEVAL, or CLASSIFICATION)
• Multimodal Model: Leverages Amazon's Nova 2 multimodal architecture for consistent embeddings across different content types

Example spicepod.yml configuration:

embeddings:
  - from: bedrock:amazon.nova-2-multimodal-embeddings-v1:0
    name: nova_embeddings
    params:
      dimensions: '3072' # Required: Output dimensions
      truncation_mode: START # Optional: START, END, or NONE (default: NONE)
      embedding_purpose: GENERIC_RETRIEVAL # Optional. GENERIC_INDEX is default

For more details on the embedding parameters and configuration options, refer to the Amazon Nova Embeddings Documentation and the Spice Embeddings Documentation.

DynamoDB Timestamp Filter Pushdown​

The DynamoDB Data Connector now supports timestamp filter pushdown, enabling more efficient append-mode acceleration refreshes by pushing timestamp filters directly to DynamoDB queries. Since DynamoDB stores timestamps as strings rather than native datetime types, this feature includes configurable timestamp formatting to ensure correct parsing and filtering.

Key Features:

• Filters on timestamp columns are now pushed down to DynamoDB, reducing data transfer and improving query performance
• Support for Go-style datetime formatting patterns to handle various timestamp string formats
• Uses ISO 8601 format by default when no custom format is specified

Example spicepod.yml configuration:

datasets:
  - from: dynamodb:sales
    name: sales
    time_column: created_at
    time_format: timestamptz
    params:
      time_format: 2006-01-02T15:04:05.000Z07:00
    acceleration:
      enabled: true
      engine: duckdb
      refresh_mode: append

For more details, refer to the DynamoDB Data Connector Documentation.

HTTP Data Connector Health Probe Configuration​

The HTTP Data Connector now supports configurable health probe paths for endpoint validation. Instead of using a random non-existent path, the system can now validate endpoints using a user-specified path, improving flexibility and reliability for health checks.

Example spicepod.yml configuration:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    params:
      file_format: json
      health_probe: /health-check

For more details, refer to the HTTP Data Connector Documentation.

Spice .NET SDK v0.2​

The Spice .NET SDK has been upgraded with expanded .NET version support, custom User-Agent configuration, and updated gRPC libraries: spice-dotnet v0.2.0. The SDK is available on NuGet.

Key Features:

• Expanded .NET Support: Now supports .NET Standard 2.0, .NET Core 8.0, 9.0, and 10.0.
• Custom User-Agent: Configure custom User-Agent headers for client identification and telemetry.
• Updated gRPC Libraries: Upgraded gRPC dependencies and netstandard for improved performance and reliability

Upgrade Example:

dotnet add package SpiceAI --version 0.2.0

For more details, refer to the .NET SDK Documentation.

Additional Improvements & Bug Fixes​

• Reliability: Fixed view loading to respect topological order, preventing dependency resolution errors.
• Reliability: Migrated from deprecated trust_dns_resolver to hickory_resolver for improved DNS resolution reliability.
• Security: Fixed arbitrary file access vulnerability during archive extraction ("Zip Slip") to prevent potential security exploits.
• Distributed Query: Fixed object store initialization across scheduler/executor gap, improving reliability for distributed query execution.
• Distributed Query: Optimized query routing by preventing runtime.* schema queries from being sent to the scheduler, improving performance for metadata queries.
• Performance: Added Blake3 and xxHash support with xxh3_64 as the default caching hashing algorithm for improved cache and query performance.
• Performance: Optimized default Zstd compression level to 6 for better balance between compression ratio and speed.
• UX: Improved dataset loading output with clearer progress indicators and status messages.

Contributors​

• @Jeadie
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @krinart
• @mach-kernel

Breaking Changes​

No breaking changes.

Cookbook Updates​

No major cookbook updates.

The Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.9.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.9.1 image:

docker pull spiceai/spiceai:1.9.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Changelog​

• fix integration tests: order by the query to make snapshots deterministic by @phillipleblanc in #8198
• Add health probe override by @lukekim in #8236
• Use Moka optionally_get_with for SWR single-in-flight semantics by @lukekim in #8231
• fix: Arbitrary file access during archive extraction ("Zip Slip") by @phillipleblanc in #8242
• Migrate trust_dns_resolver to hickory_resolver by @phillipleblanc in #8243
• fix: Deny assert macros in non-test code by @peasee in #8223
• Distributed query: Object store initialization across scheduler/executor gap, misc bugfixes & improvements by @mach-kernel in #8009
• Add Blake3, enable xxHash, set xxh3_64 as default, add bench by @lukekim in #8157
• Make cache zstd default compression level 6 by @lukekim in #8234
• Use seed for xxh3 by @lukekim in #8232
• DynamoDB Timestamp Filter Pushdown by @krinart in #8235
• Add ready_wait for mongo-arrow benchmarks by @krinart in #8246
• Add support for amazon.nova-2-multimodal-embeddings-v1:0 by @Jeadie in #8225
• Improve the output of dataset loading by @lukekim in #8256
• Load views in topological order by @lukekim in #8255
• Distributed query: Do not send runtime.* schema queries to scheduler by @mach-kernel in #8271
• Remove input length check for Nova model. by @Jeadie in #8270

Announcing the release of Spice v1.9.0-stable! 🌶

v1.9.0-stable introduces Spice Cayenne, a new high-performance data accelerator built on the Vortex columnar format that delivers better than DuckDB performance without single-file scaling limitations, and a preview of Multi-Node Distributed Query based on Apache Ballista. v1.9.0 also upgrades to DataFusion v50, DuckDB v1.4.2, and Delta-Kernel v0.16 for even higher query performance, expands search capabilities with full-text search on views and multi-column embeddings, and delivers many additional features and improvements.

What's New in v1.9.0​

Cayenne Data Accelerator (Beta)​

Introducing Cayenne: SQL as an Acceleration Format: A new high-performance Data Accelerator that simplifies multi-file data acceleration by using an embedded database (SQLite) for metadata while storing data in the Vortex columnar format, a Linux Foundation project. Cayenne delivers query and ingestion performance better than DuckDB's file-based acceleration without DuckDB's memory overhead and the scaling challenges of single DuckDB files.

Cayenne uses SQLite to manage acceleration metadata (schemas, snapshots, statistics, file tracking) through simple SQL transactions, while storing data in Vortex's compressed columnar format. This architecture provides:

Key Features:

• SQLite + Vortex Architecture: All metadata is stored in SQLite tables with standard SQL transactions, while data lives in Vortex's compressed, chunked columnar format designed for zero-copy access and efficient scanning.
• Simplified Operations: No complex file hierarchies, no JSON/Avro metadata files, no separate catalog servers—just SQL tables and Vortex data files. The entire metadata schema is intentionally simple for maximum reliability.
• Fast Metadata Access: Single SQL query retrieves all metadata needed for query planning—no multiple round trips to storage, no S3 throttling, no reconstruction of metadata state from scattered files.
• Efficient Small Changes: Dramatically reduces small file proliferation. Snapshots are just rows in SQLite tables, not new files on disk. Supports millions of snapshots without performance degradation.
• High Concurrency: Changes consist of two steps: stage Vortex files (if any), then run a single SQL transaction. Much faster conflict resolution and support for many more concurrent updates than file-based formats.
• Advanced Data Lifecycle: Full ACID transactions, delete support, and retention SQL execution on refresh commit.

Example Spicepod.yml configuration:

datasets:
  - from: s3:my_table
    name: accelerated_data_30d
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      refresh_mode: append
      retention_sql: DELETE FROM accelerated_data WHERE created_at < NOW() - INTERVAL '30 days'

Note, the Cayenne Data Accelerator is in Beta with limitations.

For more details, refer to the Cayenne Documentation, the Vortex project, and the DuckLake announcement that partly inspired this design.

Multi-Node Distributed Query (Preview)​

Apache Ballista Integration: Spice now supports distributed query execution based on Apache Ballista, enabling distributed queries across multiple executor nodes for improved performance on large datasets. This feature is in preview in v1.9.0.

Architecture:

A distributed Spice cluster consists of:

• Scheduler: Responsible for distributed query planning and work queue management for the executor fleet
• Executors: One or more nodes responsible for running physical query plans

Getting Started:

Start a scheduler instance using an existing Spicepod. The scheduler is the only spiced instance that needs to be configured:

# Start scheduler (note the flight bind address override if you want it reachable outside localhost)
spiced --cluster-mode scheduler --flight 0.0.0.0:50051

Start one or more executors configured with the scheduler's flight URI:

# Start executor (automatically selects a free port if 50051 is taken)
spiced --cluster-mode executor --scheduler-url spiced://localhost:50051

Query Execution:

Queries run through the scheduler will now show a distributed_plan in EXPLAIN output, demonstrating how the query is distributed across executor nodes:

EXPLAIN SELECT count(id) FROM my_dataset;

Current Limitations:

• Accelerated datasets are currently not supported. This feature is designed for querying partitioned data lake formats (Parquet, Delta Lake, Iceberg, etc.)
• The feature is in preview and may have stability or performance limitations
• Specific acceleration support is planned for future releases

For more details, refer to the Distributed Query Documentation.

DataFusion v50 Upgrade​

Spice.ai is built on the Apache DataFusion query engine. The v50 release brings significant performance improvements and enhanced reliability:

Performance Improvements 🚀:

• Dynamic Filter Pushdown: Enhanced dynamic filter pushdown for custom ExecutionPlans, ensuring filters propagate correctly through all physical operators for improved query performance.

• Partition Pruning: Expanded partition pruning support ensures that unnecessary partitions are skipped when filters are not used, reducing data scanning overhead and improving query execution times.

Apache Spark Compatible Functions: Added support for Spark-compatible functions including array, bit_get/bit_count, bitmap_count, crc32/sha1, date_add/date_sub, if, last_day, like/ilike, luhn_check, mod/pmod, next_day, parse_url, rint, and width_bucket.

Bug Fixes & Reliability: Resolved issues with partition name validation and empty execution plans when vector index lists are empty. Fixed timestamp support for partition expressions, enabling better partitioning for time-series data.

See the Apache DataFusion 50.0.3 Release for more details.

DuckDB v1.4.2 Upgrade and Accelerator Improvements​

DuckDB v1.4.2: DuckDB has been upgraded to v1.4.2, which includes several performance optimizations.

Composite ART Index Support: DuckDB in Spice now supports composite (multi-column) Adaptive Radix Tree (ART) indexes for accelerated table scans. When queries filter on multiple columns fully covered by a composite index, the optimizer automatically uses index scans instead of full table scans, delivering significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      indexes:
        '(region, product_id)': enabled

Performance example with composite index on 7.5M rows:

SELECT * FROM sales WHERE region = 'US' AND product_id = 12345;

-- Without index: 0.282s
-- With composite index (region, product_id): 0.037s
-- Performance improvement: 7.6x faster with composite index

DuckDB Intermediate Materialization: Queries with indexes now use intermediate materialization (WITH ... AS MATERIALIZED) to leverage faster index scans. Currently supported for non-federated queries (query_federation: disabled) against a single table with indexes only. When predicates cover more columns than the index, the optimizer rewrites queries to first materialize index-filtered results, then apply remaining predicates. This optimization can deliver significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://sales_data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      params:
        query_federation: disabled # Required currently for intermediate materialization
      indexes:
        '(region, product_id)': enabled

Performance example:

-- Query with indexed columns (region, product_id) plus additional filter (amount)
SELECT * FROM sales
WHERE region = 'US' AND product_id = 12345 AND amount > 1000;

-- Optimized execution time: 0.031s (with intermediate materialization)
-- Standard execution time: 0.108s (without optimization)
-- Performance improvement: ~3.5x faster

The optimizer automatically rewrites the query to:

WITH _intermediate_materialize AS MATERIALIZED (
  SELECT * FROM sales WHERE region = 'US' AND product_id = 12345
)
SELECT * FROM _intermediate_materialize WHERE amount > 1000;

Parquet Buffering for Partitioned Writes: DuckDB partitioned writes in table mode now support Parquet buffering, reducing memory usage and improving write performance for large datasets.

Retention SQL on Refresh Commit: DuckDB accelerations now support running retention SQL on refresh commit, enabling automatic data cleanup and lifecycle management during refresh operations.

UTC Timezone for DuckDB: DuckDB now uses UTC as the default timezone, ensuring consistent behavior for time-based queries across different environments.

Example Spicepod.yml configuration:

datasets:
  - from: s3://my_bucket/large_table/
    name: partitioned_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      retention:
        sql: DELETE FROM partitioned_data WHERE event_time < NOW() - INTERVAL '7 days'

For more details, refer to the DuckDB Data Accelerator Documentation.

HTTP Data Connector​

• Querying endpoints as tables: The HTTP/HTTPS Data Connectors now supports querying HTTP endpoints directly as tables in SQL queries with dynamic filters. This feature transforms REST APIs into queryable data sources, making it easy to integrate external service data.

• Query HTTP endpoint that returns structured data (JSON, CSV, etc.) as if it were a database table

• Configurable retry logic, timeouts, and POST request support for more complex API interactions

Example Spicepod.yml configuration:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    params:
      file_format: json
      max_retries: 3
      client_timeout: 10s
      allowed_request_paths: /search/people
      request_query_filters: enabled
      request_body_filters: enabled

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael'
LIMIT 10;

If a request_body is supplied it will be posted to the endpoint:

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael' and request_body = '{"name": "michael"}'
LIMIT 10;

HTTP endpoints can be accelerated using refresh_sql:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    params:
      file_format: json
      allowed_request_paths: /search/people
      request_query_filters: enabled
      request_body_filters: enabled
    acceleration:
      enabled: true
      refresh_mode: full
      refresh_sql: |
        SELECT request_path, request_query, content 
        FROM tvmaze
        WHERE request_path = '/search/people'
          AND request_query IN ('q=michael', 'q=luke')

For more details, refer to the HTTP Data Connector Documentation.

DynamoDB Data Connector Improvements​

Improved Query Performance: The DynamoDB Data Connector now includes improved filter handling for edge cases, parallel scan support for faster data ingestion, and better error handling for misconfigured queries. These improvements enable more reliable and performant access to DynamoDB data.

Example Spicepod.yml configuration:

datasets:
  - from: dynamodb:my_table
    name: ddb_data
    params:
      scan_segments: 10 # Default `auto` which calculates optimal segments based on number of rows

For more details, refer to the DynamoDB Data Connector Documentation.

S3 Data Connector Improvements​

S3 Versioning Support: Spice now supports S3 Versioning for all connectors using object-store (S3, Delta Lake, etc.), ensuring range reads over versioned files are atomically correct. When S3 versioning is enabled, Spice automatically tracks version IDs during file discovery and uses them for all subsequent range reads, preventing inconsistencies from concurrent file modifications.

Current limitations:

• Multi-file connections (e.g., partitioned datasets) do not yet support version tracking across all files
• Version tracking is automatic when S3 versioning is enabled on the bucket

S3 Single-File Refresh Skipping: Spice now optimizes S3 single-file dataset refreshes by caching file metadata (ETag, Version ID, size, timestamp) and skipping unnecessary data fetches when the underlying file hasn't changed. This optimization dramatically reduces bandwidth usage and improves refresh performance for scenarios where data doesn't change frequently. The feature is enabled by default for accelerated S3 single-file datasets and includes metrics tracking for skipped refreshes.

Example configuration:

datasets:
  - from: s3://my-bucket/data.parquet
    name: s3_data
    acceleration:
      enabled: true
      engine: duckdb
      refresh_check_interval: 10s

When the file's metadata hasn't changed between refresh checks, Spice will skip the data fetch entirely, logging:

Skipping refresh for dataset 's3_data': file metadata unchanged

For more details, refer to the S3 Data Connector Documentation.

Search & Embeddings Enhancements​

Full-Text Search on Views: Full-text search indexes are now supported on views, enabling advanced search scenarios over pre-aggregated or transformed data. This extends the power of Spice's search capabilities beyond base datasets.

Multi-Column Embeddings on Views: Views now support embedding columns, enabling vector search and semantic retrieval on view data. This is useful for search over aggregated or joined datasets.

Vector Engines on Views: Vector search engines are now available for views, enabling similarity search over complex queries and transformations.

Example Spicepod.yml configuration:

views:
  - name: aggregated_reviews
    sql: SELECT review_id, review_text FROM reviews WHERE rating > 4
    embeddings:
      - column: review_text
        model: openai:text-embedding-3-small

For more details, refer to the Search Documentation and Embeddings Documentation.

Dedicated Query Thread Pool (Now Enabled by Default)​

Dedicated Query Thread Pool: Query execution and accelerated refreshes now run on their own dedicated thread pool, separate from the HTTP server. This prevents heavy query workloads from slowing down API responses, keeping health checks fast and avoiding unnecessary Kubernetes pod restarts under load.

This feature was opt-in in previous releases and is now enabled by default. To disable it and revert to the previous behavior, add the following spicepod.yaml configuration:

runtime:
  params:
    dedicated_thread_pool: none

For more details, refer to the Runtime Configuration Documentation.

Query Performance Optimizations​

Stale-While-Revalidate Cache Control: Query results now support "stale-while-revalidate" cache control, allowing stale cached data to be served immediately while asynchronously refreshing the cache entry in the background. This improves response times for frequently-accessed queries while maintaining data freshness. Requires cache key type to be set to "sql (raw)" for proper operation.

Optimized Prepared Statements: Prepared statement handling has been optimized for better performance with parameterized queries, reducing planning overhead and improving execution time for repeated queries.

Large RecordBatch Chunking: Large Arrow RecordBatch objects are now automatically chunked to control memory usage during query execution, preventing memory exhaustion for queries returning large result sets.

Query Result Caching: Compressed Encoding, Stale-While-Revalidate Cache Control​

Zstd Compression Encoding: Query result caching now supports optional Zstandard (zstd) compression encoding to reduce memory usage for cached query results. This is particularly beneficial for large result sets, reducing cache memory footprint while maintaining fast decompression times. Encoding can be configured via the encoding parameter with options none (default) or zstd.

Example configuration:

runtime:
  caching:
    sql_results:
      enabled: true
      max_size: 128MiB
      item_ttl: 1m
      encoding: zstd # Enable zstd compression

HTTP Cache-Control Support: The query result cache now supports the stale-while-revalidate Cache-Control directive, enabling faster response times by serving stale cached results immediately while asynchronously refreshing the cache in the background. This feature is particularly useful for applications that can tolerate slightly stale data in exchange for improved performance.

Example configuration:

runtime:
  caching:
    sql_results:
      enabled: true
      max_size: 128MiB
      item_ttl: 1m
      stale_while_revalidate_ttl: 1m # serve stale items for up to 1 minute after `item_ttl` expires

How it works:

When a cache entry is stale but within the stale-while-revalidate window, Spice will:

1. Immediately return the stale cached result to the client
2. Asynchronously re-execute the query in the background to refresh the cache
3. Future requests will use the refreshed data

Configuration:

Use the Cache-Control HTTP header with the stale-while-revalidate directive:

Cache-Control: max-age=300, stale-while-revalidate=60

This configuration caches results for 5 minutes (300 seconds), and allows serving stale results for an additional 60 seconds while refreshing in the background.

Requirements:

• Must use plan or raw SQL cache keys (set cache_key_type to sql or plan in results_caching configuration)
• Background revalidation re-executes queries through the normal query path
• Timestamp tracking automatically determines cache entry age for staleness checks

Example configuration via HTTP header:

GET /v1/sql
Cache-Control: max-age=600, stale-while-revalidate=120
X-Cache-Key-Type: sql

This feature improves application responsiveness while ensuring data freshness through background updates.

For more details, refer to the Results Caching Documentation.

Security & Reliability Improvements​

Enhanced HTTP Client Security: HTTP client usage across the runtime has been hardened with improved TLS validation, certificate pinning for critical endpoints, and better error handling for network failures.

ODBC Connector Improvements: Removed unwrap calls from the ODBC connector, improving error handling and reliability. Fixed secret handling and Kubernetes secret integration.

CLI Permissions Hardening: Tightened file permissions for the CLI and install script, ensuring secure defaults for configuration files and credentials.

Oracle Instant Client Pinning: Oracle Instant Client downloads are now pinned to specific SHAs, ensuring reproducible builds and preventing supply chain attacks.

AWS Authentication Improvements​

Improved Credential Retry Logic: AWS SDK credential initialization has been significantly improved with more robust retry logic and better error handling. The system now automatically retries transient credential resolution failures using Fibonacci backoff, allowing Spice to tolerate extended AWS outages (up to ~48 hours) without manual intervention.

Key features:

• Automatic retry with backoff: Implements Fibonacci backoff for transient credential failures (network issues, temporary AWS service disruptions)
• Better error handling: Distinguishes between retryable errors (connector errors) and non-retryable errors (misconfiguration)
• Unauthenticated access support: Properly supports unauthenticated access to public S3 buckets without requiring credentials
• Improved error messages: Provides detailed logging with attempt numbers, retry intervals, and error context for better troubleshooting

The improvements ensure more reliable AWS service integration, particularly in environments with intermittent network connectivity or during AWS service degradations.

Observability & Tracing​

DataFusion Log Emission: The Spice runtime now emits DataFusion internal logs, providing deeper visibility into query planning and execution for debugging and performance analysis.

AI Completions Tracing: Fixed tracing so that ai_completions operations are correctly parented under sql_query traces, improving observability for AI-powered queries.

Git Data Connector (Alpha)​

Version-Controlled Data Access: The new Git Data Connector (Alpha) enables querying datasets stored in Git repositories. This connector is ideal for use cases involving configuration files, documentation, or any data tracked in version control.

Example Spicepod.yml configuration:

datasets:
  - from: git:https://github.com/myorg/myrepo
    name: git_metrics
    params:
      file_format: csv

For more details, refer to the Git Data Connector Documentation.

Spice Java SDK 0.4.0​

The Spice Java SDK has been upgraded with support for configurable Arrow memory limit: spice-java v0.4.0

SpiceClient client = SpiceClient.builder()
    .withArrowMemoryLimitMB(1024) // 1GB limit
    .build();

For more details, refer to the Java SDK Documentation.

CLI Improvements​

Install Specific Versions: The spice install command now supports installing specific versions of the Spice runtime and CLI. This enables easy version management, downgrading, or installation of specific releases for testing or compatibility requirements.

Usage:

# Install a specific version
spice install v1.8.3

# Install a specific version with AI flavor
spice install v1.8.3 ai

# Install latest version (existing behavior)
spice install
spice install ai

Note: Homebrew installations require manual version management via brew install spiceai/spiceai/spice@<version>.

Persistent Query History: The Spice CLI REPL (SQL, search, and chat interfaces) now persists command history to ~/.spice/query_history.txt, making your query history available across sessions. The history file is automatically created if it doesn't exist, with graceful fallback if the home directory cannot be determined.

New REPL Commands:

• .clear - Clear the screen using ANSI escape codes for a clean workspace
• .clear history - Clear and persist the query history, removing all stored commands

Tab Completion: Tab completion now includes suggestions based on your command history, making it faster to re-run or modify previous queries.

Example usage:

sql> SELECT * FROM my_table;
sql> .clear          # Clears the screen
sql> .clear history  # Clears command history
sql> # Use arrow keys or tab to access previous commands

For more details, refer to the CLI Documentation.

Additional Improvements & Bug Fixes​

• Reliability: Fixed refresh worker panics with recovery handling to prevent runtime crashes during acceleration refreshes.
• Reliability: Improved error messages for missing or invalid spicepod.yaml files, providing actionable feedback for misconfiguration.
• Reliability: Fixed DuckDB metadata pointer loading issues for snapshots.
• Performance: Ensured ListingTable partitions are pruned correctly when filters are not used.
• Reliability: Fixed vector dimension determination for partitioned indexes.
• Search: Fixed casing issues in Reciprocal Rank Fusion (RRF) for hybrid search queries.
• Search: Fixed search field handling as metadata for chunked search indexes.
• Validation: Added timestamp support for partition expressions.
• Validation: Fixed regexp_match function for DuckDB datasets.
• Validation: Fixed partition name validation for improved reliability.

Contributors​

• @Jeadie
• @lukekim
• @peasee
• @phillipleblanc
• @sgrebnov
• @kczimm
• @krinart
• @mach-kernel

Breaking Changes​

No breaking changes.

Cookbook Updates​

New HTTP Data Connector Recipe: New recipe demonstrating how to query REST APIs and HTTP(s) endpoints. See HTTP Connector Recipe for details.

The Spice Cookbook includes 82 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.9.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.9.0 image:

docker pull spiceai/spiceai:1.9.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed​

Dependencies​

• DataFusion: Upgraded to v50
• Apache Arrow: Upgraded to v56
• DuckDB: Upgraded to v1.4.2
• Delta Kernel: Upgraded to v0.16.0

Changelog​

• Fix for search field as metadata for chunked search indexes by @Jeadie in #7429
• Bump object_store from 0.12.3 to 0.12.4 by @app/dependabot in #7433
• Properly respect disabling snapshots by @phillipleblanc in #7431
• Revert "Properly respect disabling snapshots" by @sgrebnov in #7439
• Revert "Disable snapshots by default" by @sgrebnov in #7438
• Add preview warning for write access mode by @sgrebnov in #7440
• fix: regexp_match for DuckDB datasets by @kczimm in #7443
• Add feature is currently in preview warning for snapshots by @sgrebnov in #7442
• [Logger] Also emit Datafusion logs by @mach-kernel in #7441
• add missing snapshot by @kczimm in #7446
• Fix tracing so that ai_completions are parented under sql_query by @lukekim in #7415
• Enable snapshot acceleration by default by @phillipleblanc in #7451
• Disable acceleration refresh metrics by @krinart in #7450
• Add v1.8 release notes by @phillipleblanc in #7430
• fix: partition name validation by @kczimm in #7452
• Fix lint error due to ignore without reasons by @krinart in #7454
• Add models and CUDA support to spiced install script by @lukekim in #7457
• Post-release 1.8 updates by @phillipleblanc in #7455
• Remove println in datafusion by @phillipleblanc in #7461
• Update end_game.md to notify once release is done by @sgrebnov in #7460
• Remove italics from snapshot logging by @phillipleblanc in #7463
• Update openapi.json by @app/github-actions in #7466
• Fix generate spicepod schema by @phillipleblanc in #7464
• Fix generate acknowledements by @phillipleblanc in #7465
• Update spicepod.schema.json by @app/github-actions in #7469
• fix: Ensure ListingTable partitions are pruned when filters are not used by @peasee in #7471
• Create runtime-secrets crate by @phillipleblanc in #7474
• Create runtime-parameters crate by @phillipleblanc in #7475
• Don't download the snapshot if the acceleration is present by @phillipleblanc in #7477
• Bump hyper-util from 0.1.16 to 0.1.17 by @app/dependabot in #7434
• Add missing remote CLI feature by @lukekim in #7478
• Add 1.8.0 release analytics by @sgrebnov in #7481
• CLI multi-line input support for spice sql by @lukekim in #7479
• fix: duckdb partitioning cannot reload config by @kczimm in #7482
• fix: Make search cache test use a slower uncached search by @peasee in #7473
• Add support for S3 dataset params by @phillipleblanc in #7476
• Add DuckDB TPC-H memory limit variations by @lukekim in #7484
• Add better snapshot validation for incorrectly configured spicepods by @phillipleblanc in #7487
• Move blocking/sync I/O to spawn blocking by @lukekim in #7462
• Add DuckDB file accelerator 2G and 4G dispatches by @lukekim in #7491
• Validate spicepod file exists before running tests by @lukekim in #7492
• Make snapshot reading/writing more robust with Iceberg-like metadata.json by @phillipleblanc in #7486
• Re-use build-testoperator and ensure it's cached by @lukekim in #7494
• Fix duckdb test operator limit casing by @lukekim in #7498
• fix: Update benchmark snapshots by @app/github-actions in #7499
• Create runtime-request-context crate by @Jeadie in #7459
• Add integration tests for Acceleration DB snapshotting by @phillipleblanc in #7489
• Two minor fixes for AI udf tests by @krinart in #7503
• Add model response timeout for ai udf tests by @krinart in #7504
• Show error if FTS is misconfigured for datasets/views by @krinart in #7458
• Add test for chunked search index with search field as metadata by @Jeadie in #7513
• Add sccache for build test operator by @lukekim in #7515
• Enhancement: Add spill_compression to runtime config by @krinart in #7505
• Improve GitHub Data Connector by @lukekim in #7510
• Add RequestTimeoutException to S3 client by @Jeadie in #7514
• Add sha=<> to snapshot logging by @phillipleblanc in #7521
• Add Type to GitHub Data Connector issues and fix double aliasing for project author by @lukekim in #7519
• DuckDB acceleration: fix memory leak in duckdb_arrow_scan by @sgrebnov in #7524
• Fix partition_by accelerations when a projection is applied on empty partition sets by @phillipleblanc in #7526
• Nullable fields for index columns by @Jeadie in #7523
• Fix missing winver dependency for Windows by @krinart in #7538
• Update mongo config for benchmarks by @krinart in #7546
• Add acceleration snapshots cookbook to template by @phillipleblanc in #7527
• Bump github/codeql-action from 3 to 4 by @app/dependabot in #7535
• Bump golang.org/x/sys from 0.36.0 to 0.37.0 by @app/dependabot in #7529
• Make spice chat play nice with Unix pipes by @Jeadie in #7525
• Configurable DuckDB duckdb_index_scan_percentage & duckdb_index_scan_max_count by @lukekim in #7551
• [cherry-pick] Release notes for release 1.8.1 by @krinart in #7556
• Fix 1.8.1 release notes by @krinart in #7558
• FTS index with nonfilterable metadata; search field not metadata by default. by @Jeadie in #7548
• Properly set auth headers in github_release.py by @krinart in #7560
• project_schema when using EmptyExec by @kczimm in #7543
• Add 1.8.1 release analytics by @kczimm in #7561
• Bump golang.org/x/mod from 0.28.0 to 0.29.0 by @app/dependabot in #7530
• Hive-style partitioning for DuckDB file mode by @kczimm in #7563
• Vortex Data Accelerator (Dev grade) by @lukekim in #7566
• Only load eval scorers when eval defined by @Jeadie in #7549
• Bump octocrab from 0.45.0 to 0.47.0 by @app/dependabot in #7531
• Bump regex from 1.11.3 to 1.12.1 by @app/dependabot in #7532
• Fix custom file path for Vortex Data Accelerator by @phillipleblanc in #7570
• Add List type support to Vortex Data Accelerator by @lukekim in #7569
• Bump parking_lot from 0.12.4 to 0.12.5 by @app/dependabot in #7534
• Bump tokio-postgres from 0.7.14 to 0.7.15 by @app/dependabot in #7533
• Remove duplicate line from 1.8.1 release notes by @krinart in #7580
• Upgrade Go from v1.24.2 to v1.25.3 by @lukekim in #7582
• Fix race condition in S3 Vectors index and bucket creation by @kczimm in #7577
• Add runtime-async crate with managed Tokio runtime by @phillipleblanc in #7575
• Optimize GitHub Actions workflows by @lukekim in #7584
• Use 'location' as primary key for document tables by @Jeadie in #7567
• Extend query-related metrics by @krinart in #7571
• Enabling acceleration refresh metrics by using runtime.metrics config by @krinart in #7583
• S3Vector service metrics in client by @Jeadie in #7502
• Fix score order for one test case by @Jeadie in #7595
• ObjectMeta filter pushdown for ObjectStoreTextTable by @Jeadie in #7572
• Return TableProvider from CandidateGeneration::search. by @Jeadie in #7559
• EmptyHashJoinExecPhysicalOptimization, and use in VectorScanTableProvider by @Jeadie in #7587
• Update official Docker builds to use release binaries by @phillipleblanc in #7597
• New Generate Changelog workflow by @krinart in #7562
• BytesProcessedExec to allow optimizer to do limit pushdown by @mach-kernel in #7539
• GitHub Data Connector add Projects, improve rate-limiting and error handling by @lukekim in #7547
• Add copilot-instructions to help improve Copilot reviews by @lukekim in #7606
• Add support for DuckDB table-based partitioning by @sgrebnov in #7581
• fix: Use nextest for integration models tests by @peasee in #7617
• Fix license issue in table-providers by @phillipleblanc in #7620
• Remove Build Docker Image from PR checks by @phillipleblanc in #7621
• Combine PR Lint + Build checks by @phillipleblanc in #7623
• Remove Cache Rust builds step by @phillipleblanc in #7625
• Rename duckdb_partition_mode to partition_mode param by @sgrebnov in #7622
• DuckDB table partitioning: delete partitions that no longer exist after full refresh by @sgrebnov in #7614
• Build integration test binaries in single job by @phillipleblanc in #7624
• Make DuckDB table partition data write threshold configurable by @sgrebnov in #7626
• Handle table relations in HTTP v1/search by @Jeadie in #7615
• Fix E2E Test sporadic failures on the macOS runners by @phillipleblanc in #7627
• Emit query_active_count metric by @krinart in #7589
• fix: Disable go cache in actions by @peasee in #7631
• fix: Don't nullify DuckDB release callbacks for schemas by @peasee in #7628
• Fix integration tests by reverting the use of batch inserts w/ prepared statements by @phillipleblanc in #7630
• Split integration tests into 3 partitions by @phillipleblanc in #7635
• Initial Pepper data accelerator by @lukekim in #7592
• Only build the E2E Test CI binaries once by @phillipleblanc in #7633
• Update BytesProcessedExec snapshots by @mach-kernel in #7637
• Properly set RequestContext for stream execution in Flight by @krinart in #7591
• Add task for creating release branch in docs by @kczimm in #7642
• Add missing mongodb params by @krinart in #7647
• fix: Update benchmark snapshots by @app/github-actions in #7649
• Release notes for v1.8.2 by @Jeadie in #7645
• docs: Update error handling in copilot instructions by @peasee in #7652
• Pepper accelerator INSERT OVERWRITE support by @lukekim in #7643
• fix: Update benchmark snapshots by @app/github-actions in #7650
• Run Datafusion queries on a separate Tokio runtime by @phillipleblanc in #7586
• Add explicit steps for docs DRI in end game by @kczimm in #7658
• Add Release 1.8.2 QA Analytics by @krinart in #7661
• Pepper full / append refresh support by @lukekim in #7662
• Add 'client_timeout' for s3 vector by @Jeadie in #7501
• Improvements to Endgame template by @krinart in #7660
• Fix OSS docker release trigger when release marked as latest by @phillipleblanc in #7668
• Use '#[serde(deny_unknown_fields)]' for base spicepod components by @Jeadie in #7669
• DataFusion upgrade template to include Ballista by @mach-kernel in #7679
• S3 Vector index spilling by @kczimm in #7613
• Refresh request context bindings / fix trunk integration tests by @mach-kernel in #7680
• Fix integration tests for refresh append by @lukekim in #7681
• Distributed query support by @mach-kernel in #7585
• Run acceleration refreshes on separate Tokio runtime by @phillipleblanc in #7671
• Support DESCRIBE in clustered mode by @kczimm in #7686
• use hyphen instead of period for index name spill separator by @kczimm in #7697
• Add streaming option to /nsql endpoint by @kczimm in #7695
• Task History min_sql_duration filter support by @lukekim in #7698
• Spawn object_store IO tasks on the original Tokio runtime by @phillipleblanc in #7689
• Update openapi.json by @app/github-actions in #7700
• Adjust DataFusion runtime worker threads by @phillipleblanc in #7704
• Gate dedicated SQL engine CPU runtime behind opt-in param dedicated_thread_pool by @phillipleblanc in #7705
• Add TPCH S3 refresh spicepod by @phillipleblanc in #7706
• DuckDB: include ANALYZE after write to update query optimizer statistics by @sgrebnov in #7714
• Display execution time in Spice REPL for no results by @sgrebnov in #7713
• Task History capture and store SQL query plans by @lukekim in #7701
• Pepper TPC-H SF-1 benchmark by @lukekim in #7717
• fix: Update benchmark snapshots by @app/github-actions in #7720
• Optimize prepared statements (parameterized queries) by @lukekim in #7703
• Pepper accelerator tests (Clickbench, TPC-H SF-5, SF-100) by @lukekim in #7721
• Add support for DuckDB connection_pool_size param by @sgrebnov in #7716
• Add health probing to testoperator runs by @phillipleblanc in #7709
• Simplify AcceleratedTable by @Jeadie in #7724
• Add some basic indexing tests for 'FullTextDatabaseIndex' by @Jeadie in #7688
• DuckDB: on_refresh_recompute_statistics param + ANALYZE for table-based partitioning by @sgrebnov in #7719
• Fix Windows builds by excluding pepper/vortex by @phillipleblanc in #7729
• Enable separate CPU runtime thread pool for DataFusion by default by @phillipleblanc in #7732
• 'runtime-datafusion' crate for runtime related DataFusion components by @Jeadie in #7666
• Pepper expanded append refresh support by @lukekim in #7670
• Pepper basic partitioning by @lukekim in #7731
• Stable pepper benchmark snapshots by @phillipleblanc in #7739
• Delta Lake Connector: Support AWS_SESSION_TOKEN parameter by @mach-kernel in #7752
• Pepper use SQLite WAL by @lukekim in #7757
• v1.8.3 release notes by @mach-kernel in #7745
• Increase testoperator health check threshold to 50ms by @phillipleblanc in #7767
• Data Accelerator Graceful Shutdown by @lukekim in #7756
• Remove Windows CUDA builds by @phillipleblanc in #7768
• fix: Update benchmark snapshots by @app/github-actions in #7771
• Bump actions/download-artifact from 5 to 6 by @app/dependabot in #7746
• Bump serde from 1.0.226 to 1.0.228 by @app/dependabot in #7743
• Fix casing for keywords and additional columns by @Jeadie in #7770
• Bump actions/upload-artifact from 4 to 5 by @app/dependabot in #7750
• Bump criterion from 0.5.1 to 0.7.0 by @app/dependabot in #7740
• Bump rustls-native-certs from 0.8.1 to 0.8.2 by @app/dependabot in #7744
• Git Data Connector (Alpha) by @lukekim in #7772
• Pepper accelerator delete support by @lukekim in #7616
• Update Helm chart instructions for Helm in end_game.md by @sgrebnov in #7776
• Turso data accelerator by @lukekim in #7472
• Apply retention SQL filter to refresh fetch by @phillipleblanc in #7778
• Add Parquet buffering option for DuckDB partitioned writes (tables mode) by @sgrebnov in #7735
• fix: EmptyExec when list indexes is empty by @kczimm in #7784
• 1.8.3 post-release housekeeping by @mach-kernel in #7783
• feat: Upgrade to Datafusion v50 by @peasee in #7777
• fix: Replace vortex datafusion with public crate by @peasee in #7791
• Full-text search on views by @Jeadie in #7733
• Revert "Apply retention SQL filter to refresh fetch (#7778)" by @phillipleblanc in #7796
• fix: Add ingest duration and acceleration size metrics to testoperator by @peasee in #7792
• Set local timezone to UTC for DuckDB by @phillipleblanc in #7797
• add Timestamp support for partition expressions by @kczimm in #7803
• Fix trunk lint by @krinart in #7804
• Add missing mongodb params by @krinart in #7807
• Embedding columns on view components by @Jeadie in #7795
• Add Turso as a Pepper Catalog metastore by @lukekim in #7793
• Run retention_sql on refresh commit for DuckDB by @lukekim in #7785
• docs: Update datafusion upgrade checklist by @peasee in #7812
• Vector engines on views by @Jeadie in #7808
• Handle refresh worker panics and add recovery test by @phillipleblanc in #7815
• chunk large record batches to control memory usage by @kczimm in #7802
• fix: cannot determine vector dimension for partitioned indexes by @kczimm in #7818
• Upgrade to Turso v0.3 by @lukekim in #7821
• fix: Ensure custom *Exec ExecutionPlans push down dynamic filters by @peasee in #7811
• handle casing in RRF by @Jeadie in #7825
• Enable 'turso' for pepper acceleration by default by @sgrebnov in #7826
• Improved DynamoDB Data Connector by @krinart in #7715
• Initial support for llama.cpp as LLM inference backend by @lukekim in #7794
• Pepper: Implement retention SQL on refresh commit by @phillipleblanc in #7814
• Fix Dockerfiles for arm64 by @lukekim in #7834
• [DynamoDB] Handle filter edge-cases by @krinart in #7830
• [DynamoDB] Support parallelization for Scan request by @krinart in #7829
• Don't feature gate Pepper by @lukekim in #7832
• Fix llama.cpp static link by @lukekim in #7835
• fix: docker nightly builds by @kczimm in #7837
• Use GitHub-hosted macOS runner only for tag releases by @lukekim in #7836
• Fix Bug: DuckDB INTERNAL Error: Failed to load metadata pointer by @sgrebnov in #7839
• Fix docker arm64 build to use aegis in pure-rust mode by @lukekim in #7840
• Revert "Use GitHub-hosted macOS runner only for tag releases" by @lukekim in #7843
• Rename Pepper to Cayenne by @lukekim in #7844
• Tighten CLI permissions and install script by @lukekim in #7845
• Set mvcc for Cayenne Turso metastore by @lukekim in #7850
• Optimize Prepared Statements by @lukekim in #7859
• Remove unwrap from ODBC connector, fix secrets, and kuberenetes secre… by @lukekim in #7846
• Improve and secure HTTP client usage by @lukekim in #7847
• Pin Oracle Instant Client download to a SHA by @lukekim in #7851
• Improve experience for missing or invalid Spicepod.yaml by @lukekim in #7849
• chore: Fix PR linting by @peasee in #7865
• Revert FlightIPC issues by @Jeadie in #7870
• Bump Jimver/cuda-toolkit from 0.2.28 to 0.2.29 by @app/dependabot in #7878
• Optimize macOS and Windows builds by @lukekim in #7863
• Improve error message by adding 'cayenne' to the list of valid accelerator engines by @sgrebnov in #7882
• fix: Kafka message delivery failed by @kczimm in #7883
• fix: allow parameter index without dollar signs by @kczimm in #7887
• docs: Update component criteria by @peasee in #7891
• Temporary disable supports_limit_pushdown for SchemaCastScanExec by @sgrebnov in #7893
• fix: Make integration run with no relevant changes, disable makefile targets on PR by @peasee in #7896
• Add Cayenne benchmark and concurrency tests and remove indexes for Turso MVCC by @lukekim in #7879
• Remove '.embeddings[].metadata' by @Jeadie in #7897
• Revert llama.cpp engine by @lukekim in #7898
• Make Cayenne snapshotting more robust by @sgrebnov in #7899
• Release notes v1.9.0-rc1 by @Jeadie in #7902
• Fix dataset_acceleration_last_refresh_time_ms unit to milliseconds in description by @ewgenius in #7901
• Fix lint error in record_explain_plan functionality by @sgrebnov in #7906
• Cleanup old snapshots after full refresh by @lukekim in #7908
• Cayenne deletion vector caching support by @lukekim in #7903
• Split filters into partition filters (for pruning) and data filters by @lukekim in #7889
• fix: Update benchmark snapshots by @app/github-actions in #7911
• fix: Update benchmark snapshots by @app/github-actions in #7912
• fix: Update benchmark snapshots by @app/github-actions in #7913
• Update spicepod.schema.json by @app/github-actions in #7916
• fix: Update benchmark snapshots by @app/github-actions in #7917
• Add Cayenne & Turso accelerators to E2E CI test matrix by @lukekim in #7922
• Make preview warnings consistent by @lukekim in #7921
• Filter and write optimizations by @lukekim in #7918
• fix: Set sccache region explicitly by @peasee in #7928
• fix: Enable integration test merge group checks by @peasee in #7927
• Update testoperator release branch from 1.8 to 1.9 by @peasee in #7926
• Update DuckDB to 1.4.1 with composite ART scans by @mach-kernel in #7884
• Don't build Windows on trunk pushes by @lukekim in #7931
• fix: Use correct minio secret in build binary push by @peasee in #7934
• Update test-framework workers to use dedicated Flight client by @sgrebnov in #7938
• Fix financebench, configure s3vectors for appropriate snapshotting by @Jeadie in #7935
• Don't try to initialize accelerator if it is disabled by @lukekim in #7932
• Add spark UDFs to Spice by @Jeadie in #7936
• Fix extra async_trait in cayenne metadata catalog by @phillipleblanc in #7942
• deps: Upgrade to Rust 1.90 by @peasee in #7941
• Add cayenne accelerator to README.md by @ewgenius in #7905
• fix: Update benchmark snapshots by @app/github-actions in #7948
• Run integration tests with AWS_EC2_METADATA_DISABLED flag by @sgrebnov in #7952
• Only retry credentials on ConnectorError by @kczimm in #7944
• fix: Improve join reordering by ensuring JoinSelection is applied by @peasee in #7828
• fix: Remove unused deps, consolidate workspace deps by @peasee in #7953
• bump async-openai commit by @kczimm in #7929
• deps: Use vortex fork by @peasee in #7954
• Enable tracing in delta lake integration tests by @sgrebnov in #7951
• Update datasets in S3 vectors test case by @Jeadie in #7956
• Add spiced metrics scraping to test operator by @lukekim in #7937
• Memoize S3 vectors ListIndex API call with configurable TTL by @kczimm in #7910
• Cayenne performance optimizations by @lukekim in #7907
• Setup HotFix issue template by @ewgenius in #7957
• Fix AWS SDK credential cache retry handling by @phillipleblanc in #7943
• Infer RRF join_key from TableProvider::constraints and implement SearchQueryProvider::constraints. by @Jeadie in #7959
• [Optimizer]: DuckDB intermediate materialization (non-federated) by @mach-kernel in #7964
• 1.7.3 post-release housekeeping by @ewgenius in #7962
• Fix digest_many UDF for ColumnarValue::Array. by @Jeadie in #7960
• Fix spiced metrics reporting as part of benchmark tests by @sgrebnov in #7967
• Avoid pushing down Spice specific UDFs to accelerators during federation by @Jeadie in #7909
• CLI file persisted history with .clear and .clear history commands by @lukekim in #7970
• ResultsCache Cache-Control stale-while-revalidate by @lukekim in #7963
• Use GetVectors API instead of returnData by @kczimm in #7083
• Make DuckDB intermediate materialization logic more robust by @sgrebnov in #7971
• [Cayenne] Configurable target Vortex file size by @lukekim in #7972
• fix: Update benchmark snapshots by @app/github-actions in #7974
• Bump github.com/klauspost/compress from 1.17.11 to 1.18.1 by @app/dependabot in #7872
• fix: Update benchmark snapshots by @app/github-actions in #7978
• fix: Update benchmark snapshots by @app/github-actions in #7982
• Run Integration tests on spiceai-dev-runners by @sgrebnov in #7985
• [CLI] Fix cursor issue due to flush by @lukekim in #7981
• fix: Support S3 versioning, Vortex dynamic filter pushdown by @peasee in #7984
• Make cluster a default feature by @lukekim in #7994
• Optimize DuckDB Intermediate Index Materialization for No-Index Case by @sgrebnov in #7998
• HTTP connector with dynamic filter support by @lukekim in #7969
• Revert federation 'can_execute_plan' by @Jeadie in #7999
• Fix stale caching by @lukekim in #7995
• Fix count(*) for http connector by @krinart in #8001
• [CLI] Install specific version by @lukekim in #8006
• Fix stale with revalidate request/response by @lukekim in #8005
• Fallback RequestContext for cluster queries by @Jeadie in #8007
• Use use_rustls_tls for Spice Cloud /connect by @lukekim in #8008
• Use delta-kernel-rs 0.16x + Parquet reader with object meta API changes by @mach-kernel in #8011
• fix: Update datafusion & arrow-rs with S3 versioning fix by @lukekim in #8012
• Add 1.9.0-rc.2 release notes by @sgrebnov in #7993
• Update Datafusion version by @sgrebnov in #8014
• [Acceleration] DuckDB tables mode partitioner + CTE rewrite optimizer by @mach-kernel in #8013
• Update spicepod.schema.json by @app/github-actions in #8015
• Update acknowledgements by @app/github-actions in #8016
• Upgrade shutdown signal Ordering by @krinart in #8017
• Set max-age: 0 during stale by @lukekim in #8018
• Add E2E test release for Helm by @lukekim in #8023
• Bump github.com/olekukonko/tablewriter from 0.0.5 to 1.1.1 by @app/dependabot in #7989
• Bump schemars from 0.9.0 to 1.0.4 by @app/dependabot in #7877
• Update generate_changelog script by @krinart in #8028
• Update QA analytics Release 1.9.0-rc.2 by @krinart in #8027
• [CLI] Improve auto-complete by @lukekim in #8022
• Improve verify helm workflow by @lukekim in #8024
• Bump azure_core from 0.28.0 to 0.30.0 by @app/dependabot in #7986
• Test operator load test row count validation by @lukekim in #8036
• fix: Revert HTTP response offloading by @peasee in #8041
• Disable advanced filters pruning for partitioned tables by @sgrebnov in #8037
• fix: Ensure Vortex UncompressedSizeInBytes is calculated by @peasee in #8044
• Add 1.9.0-rc.3 release notes by @sgrebnov in #8048
• fix: Update test snapshots by @app/github-actions in #8046
• add benchmark spicepods by @Jeadie in #8047
• DynamoDB TPC-H SF1 Benchmarks by @krinart in #8043
• Bump github.com/AzureAD/microsoft-authentication-library-for-go from 1.5.0 to 1.6.0 by @app/dependabot in #7988
• Bump golang.org/x/sys from 0.37.0 to 0.38.0 by @app/dependabot in #7987
• v1.9.0-rc.2 README updates by @lukekim in #8035
• Bump suppaftp from 5.4.0 to 6.3.0 by @app/dependabot in #7875
• Bump ctor from 0.5.0 to 0.6.0 by @app/dependabot in #7873
• WW README Update by @wyattwenzel in #8058
• Reenable dynamic federation support by @Jeadie in #8026
• fix: Prevent SortExec from ordering below SchemaCastScanExec by @peasee in #8061
• Skip logging and return OK() on error during shutdown by @krinart in #8057
• Partition pruning with complex expressions by @lukekim in #8040
• Update openapi.json by @app/github-actions in #8064
• Make DynamoDB snapshots consistent by @krinart in #8069
• Add check for error log by @krinart in #8070
• Fix tracing of 's3_vector_query_and_get' by @Jeadie in #8065
• DuckDB v1.4.2 by @mach-kernel in #8073
• Fix failing OpenAI test by @krinart in #8076
• Enable 'test_recency_scoring' by @Jeadie in #8068
• Test operator: avoid duplicate Flight requests when using --http-clients by @sgrebnov in #8071
• Update load tests to use truth percentile values by @sgrebnov in #8079
• Update DynamoDB to RC by @krinart in #8060
• CachedQueryVector to avoid recomputing embedding vector for spilling/partitioned vector indexes. by @Jeadie in #8059
• Fix DuckDB on_commit sink race by @lukekim in #8081
• Add partitioned duckdb by @lukekim in #8083
• [CLI] Security and santization by @lukekim in #8082
• fix: Update benchmark snapshots by @app/github-actions in #8084
• Fix partition_by expression by @lukekim in #8087
• Data Components security fixes and sanitization by @lukekim in #8086
• Runtime security and sanitization by @lukekim in #8088
• Add spicepod-validator tool and fix spicepods by @lukekim in #8089
• Skip data fetches for S3 single file refreshes by @lukekim in #8072
• MCP security and sanitization by @lukekim in #8090
• Update spicepod.schema.json by @app/github-actions in #8099
• Update acknowledgements by @app/github-actions in #8098
• Add install-dev target back to Makefile by @Jeadie in #8100
• fix 'testoperator run search' by @Jeadie in #8101
• Update datafusion-table-providers - fix nullability inferences for MySQL and PostgreSQL, and fix full text search for PostgreSQL by @ewgenius in #8092
• Remove duplicate install-with-models by @phillipleblanc in #8107
• Improve Cayenne partitioning by @lukekim in #8097
• Testoperator dispatch: respect verify_results dispatch configuration by @sgrebnov in #8106
• Include 'match' column only if chunk offsets found in seach query 'LogicalPlan' by @Jeadie in #8102
• Fix validation path by @lukekim in #8109
• Fix dispatch paths by @lukekim in #8110
• Fix dispatch spicepod paths by @lukekim in #8112
• fix: Update benchmark snapshots by @app/github-actions in #8113
• fix: Update benchmark snapshots by @app/github-actions in #8114
• fix: Update benchmark snapshots by @app/github-actions in #8116
• Update test Spicepods by @lukekim in #8131
• Add validation to reference schema by @lukekim in #8111
• Include root error when failing to find latest timestamp in accelerated table by @sgrebnov in #8132
• fix: HTTP Connector validation, query and body by @lukekim in #8115
• Update nsql model list by @lukekim in #8141
• Update DynamoDB Benchmarks by @krinart in #8135
• Fix Dremio E2E test by @sgrebnov in #8139
• fix: Update MongoDB benchmark snapshots by @app/github-actions in #8143
• fix: Update DynamoDB benchmark snapshots by @app/github-actions in #8142
• fix: Update benchmark snapshots by @app/github-actions in #8145
• fix: Update iceberg[catalog] benchmark snapshots by @app/github-actions in #8144
• Improve HTTP Connector UX by @lukekim in #8146
• QueryOverrides for DynamoDB benchmarks by @krinart in #8151
• test-framework: add row count validation skipping with TPC-DS defaults by @sgrebnov in #8149
• fix: Update benchmark snapshots by @app/github-actions in #8148
• fix: Update benchmark snapshots by @app/github-actions in #8154
• fix: Update benchmark snapshots by @app/github-actions in #8155
• fix: Update test snapshots by @app/github-actions in #8160
• Suppress delta_kernel::listed_log_files warnings by @phillipleblanc in #8158
• Update table providers to fix warning by @phillipleblanc in #8156
• Suppress MCP limit log by @phillipleblanc in #8159
• Remove incorrect tool name validation by @Jeadie in #8161
• Disable results validation for federated/glue[csv].yaml by @phillipleblanc in #8163
• fix: Update benchmark snapshots by @app/github-actions in #8164
• Fix dynamodb overrides by @phillipleblanc in #8165
• Fix dynamo db overrides again by @phillipleblanc in #8166
• few more dynamodb overrides by @phillipleblanc in #8167
• Add stub release notes for v1.9.0-rc.4 by @phillipleblanc in #8168
• Add v1.9.0-rc.4 release notes by @lukekim in #8169
• fix: Cayenne concurrent table creation by @lukekim in #8176
• fix: Avoid pruning bucket partitions for != and NOT IN operators by @sgrebnov in #8177

Announcing the release of Spice v1.9.0-rc.4! 🌶

This release candidate brings DuckDB v1.4.2, Cayenne partitioning improvements, and comprehensive security hardening across the CLI, data connectors, runtime, and MCP. v1.9.0-rc.4 also includes MySQL and PostgreSQL connector improvements with fixed nullability inferences and full-text search support, DynamoDB consistency improvements, HTTP connector validation and UX enhancements, and numerous reliability and performance optimizations. Significant improvements were also made to test and automation infrastructure to ensure high quality releases.

v1.9.0 introduces Spice Cayenne, a new high-performance data accelerator built on the Vortex columnar format that delivers better than DuckDB performance without single-file scaling limitations, and a preview of Multi-Node Distributed Query based on Apache Ballista. v1.9.0 also upgrades to DataFusion v50 for even higher query performance, expands search capabilities with full-text search on views and multi-column embeddings, and delivers many additional features and improvements.

What's New in v1.9.0​

Cayenne Data Accelerator (Beta)​

Introducing Cayenne: SQL as an Acceleration Format: A new high-performance Data Accelerator that simplifies multi-file data acceleration by using an embedded database (SQLite) for metadata while storing data in the Vortex columnar format, a Linux Foundation project. Cayenne delivers query and ingestion performance better than DuckDB's file-based acceleration without DuckDB's memory overhead and the scaling challenges of single DuckDB files.

Cayenne uses SQLite to manage acceleration metadata (schemas, snapshots, statistics, file tracking) through simple SQL transactions, while storing data in Vortex's compressed columnar format. This architecture provides:

Key Features:

• SQLite + Vortex Architecture: All metadata is stored in SQLite tables with standard SQL transactions, while data lives in Vortex's compressed, chunked columnar format designed for zero-copy access and efficient scanning.
• Simplified Operations: No complex file hierarchies, no JSON/Avro metadata files, no separate catalog servers—just SQL tables and Vortex data files. The entire metadata schema is intentionally simple for maximum reliability.
• Fast Metadata Access: Single SQL query retrieves all metadata needed for query planning—no multiple round trips to storage, no S3 throttling, no reconstruction of metadata state from scattered files.
• Efficient Small Changes: Dramatically reduces small file proliferation. Snapshots are just rows in SQLite tables, not new files on disk. Supports millions of snapshots without performance degradation.
• High Concurrency: Changes consist of two steps: stage Vortex files (if any), then run a single SQL transaction. Much faster conflict resolution and support for many more concurrent updates than file-based formats.
• Advanced Data Lifecycle: Full ACID transactions, delete support, and retention SQL execution on refresh commit.

Example Spicepod.yml configuration:

datasets:
  - from: s3:my_table
    name: accelerated_data_30d
    acceleration:
      enabled: true
      engine: cayenne
      mode: file
      refresh_mode: append
      retention_sql: DELETE FROM accelerated_data WHERE created_at < NOW() - INTERVAL '30 days'

Note, the Cayenne Data Accelerator is in Beta with limitations.

For more details, refer to the Cayenne Documentation, the Vortex project, and the DuckLake announcement that partly inspired this design.

Multi-Node Distributed Query (Preview)​

Apache Ballista Integration: Spice now supports distributed query execution based on Apache Ballista, enabling distributed queries across multiple executor nodes for improved performance on large datasets. This feature is in preview in v1.9.0-rc.3.

Architecture:

A distributed Spice cluster consists of:

• Scheduler: Responsible for distributed query planning and work queue management for the executor fleet
• Executors: One or more nodes responsible for running physical query plans

Getting Started:

Start a scheduler instance using an existing Spicepod. The scheduler is the only spiced instance that needs to be configured:

# Start scheduler (note the flight bind address override if you want it reachable outside localhost)
spiced --cluster-mode scheduler --flight 0.0.0.0:50051

Start one or more executors configured with the scheduler's flight URI:

# Start executor (automatically selects a free port if 50051 is taken)
spiced --cluster-mode executor --scheduler-url spiced://localhost:50051

Query Execution:

Queries run through the scheduler will now show a distributed_plan in EXPLAIN output, demonstrating how the query is distributed across executor nodes:

EXPLAIN SELECT count(id) FROM my_dataset;

Current Limitations:

• Accelerated datasets are currently not supported. This feature is designed for querying partitioned data lake formats (Parquet, Delta Lake, Iceberg, etc.)
• The feature is in preview and may have stability or performance limitations
• Specific acceleration support is planned for future releases

DataFusion v50 Upgrade​

Spice.ai is built on the Apache DataFusion query engine. The v50 release brings significant performance improvements and enhanced reliability:

Performance Improvements 🚀:

• Dynamic Filter Pushdown: Enhanced dynamic filter pushdown for custom ExecutionPlans, ensuring filters propagate correctly through all physical operators for improved query performance.

• Partition Pruning: Expanded partition pruning support ensures that unnecessary partitions are skipped when filters are not used, reducing data scanning overhead and improving query execution times.

Apache Spark Compatible Functions: Added support for Spark-compatible functions including array, bit_get/bit_count, bitmap_count, crc32/sha1, date_add/date_sub, if, last_day, like/ilike, luhn_check, mod/pmod, next_day, parse_url, rint, and width_bucket.

Bug Fixes & Reliability: Resolved issues with partition name validation and empty execution plans when vector index lists are empty. Fixed timestamp support for partition expressions, enabling better partitioning for time-series data.

See the Apache DataFusion 50.0.3 Release for more details.

DuckDB v1.4.2 Upgrade and Accelerator Improvements​

DuckDB v1.4.2: DuckDB has been upgraded to v1.4.2, which includes several performance optimizations.

Composite ART Index Support: DuckDB in Spice now supports composite (multi-column) Adaptive Radix Tree (ART) indexes for accelerated table scans. When queries filter on multiple columns fully covered by a composite index, the optimizer automatically uses index scans instead of full table scans, delivering significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      indexes:
        '(region, product_id)': enabled

Performance example with composite index on 7.5M rows:

SELECT * FROM sales WHERE region = 'US' AND product_id = 12345;

-- Without index: 0.282s
-- With composite index (region, product_id): 0.037s
-- Performance improvement: 7.6x faster with composite index

DuckDB Intermediate Materialization: Queries with indexes now use intermediate materialization (WITH ... AS MATERIALIZED) to leverage faster index scans. Currently supported for non-federated queries (query_federation: disabled) against a single table with indexes only. When predicates cover more columns than the index, the optimizer rewrites queries to first materialize index-filtered results, then apply remaining predicates. This optimization can deliver significant performance improvements for selective queries.

Example configuration:

datasets:
  - from: file://sales_data.parquet
    name: sales
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      params:
        query_federation: disabled # Required currently for intermediate materialization
      indexes:
        '(region, product_id)': enabled

Performance example:

-- Query with indexed columns (region, product_id) plus additional filter (amount)
SELECT * FROM sales
WHERE region = 'US' AND product_id = 12345 AND amount > 1000;

-- Optimized execution time: 0.031s (with intermediate materialization)
-- Standard execution time: 0.108s (without optimization)
-- Performance improvement: ~3.5x faster

The optimizer automatically rewrites the query to:

WITH _intermediate_materialize AS MATERIALIZED (
  SELECT * FROM sales WHERE region = 'US' AND product_id = 12345
)
SELECT * FROM _intermediate_materialize WHERE amount > 1000;

Parquet Buffering for Partitioned Writes: DuckDB partitioned writes in table mode now support Parquet buffering, reducing memory usage and improving write performance for large datasets.

Retention SQL on Refresh Commit: DuckDB accelerations now support running retention SQL on refresh commit, enabling automatic data cleanup and lifecycle management during refresh operations.

UTC Timezone for DuckDB: DuckDB now uses UTC as the default timezone, ensuring consistent behavior for time-based queries across different environments.

Example Spicepod.yml configuration:

datasets:
  - from: s3://my_bucket/large_table/
    name: partitioned_data
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      retention:
        sql: DELETE FROM partitioned_data WHERE event_time < NOW() - INTERVAL '7 days'

HTTP Data Connector​

• Querying endpoints as tables: The HTTP/HTTPS Data Connectors now supports querying HTTP endpoints directly as tables in SQL queries with dynamic filters. This feature transforms REST APIs into queryable data sources, making it easy to integrate external service data.

• Query HTTP endpoint that returns structured data (JSON, CSV, etc.) as if it were a database table

• Configurable retry logic, timeouts, and POST request support for more complex API interactions

Example Spicepod.yml configuration:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    params:
      file_format: json
      max_retries: 3
      client_timeout: 10s

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael'
LIMIT 10;

If a request_body is supplied it will be posted to the endpoint:

Example SQL query:

SELECT request_path, request_query, content
FROM tvmaze
WHERE request_path = '/search/people' and request_query = 'q=michael' and request_body = '{"name": "michael"}'
LIMIT 10;

HTTP endpoints can be accelerated using refresh_sql:

datasets:
  - from: https://api.tvmaze.com
    name: tvmaze
    acceleration:
      enabled: true
      refresh_mode: full
      refresh_sql: |
        SELECT request_path, request_query, content 
        FROM tvmaze
        WHERE request_path = '/search/people'
          AND request_query IN ('q=michael', 'q=luke')

DynamoDB Data Connector Improvements​

Improved Query Performance: The DynamoDB Data Connector now includes improved filter handling for edge cases, parallel scan support for faster data ingestion, and better error handling for misconfigured queries. These improvements enable more reliable and performant access to DynamoDB data.

Example Spicepod.yml configuration:

datasets:
  - from: dynamodb:my_table
    name: ddb_data
    params:
      scan_segments: 10 # Default `auto` which calculates optimal segments based on number of rows

S3 Versioning Support​