explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR
  • Why Arrow after CSV and TSV streaming
  • URL patterns and streaming semantics
  • Column typing rules that bite in production
  • Reading streams in Python
  • UI and release cadence caveats
  • How this fits explainx.ai's tool coverage
  • Security and publishing notes
  • Try it in fifteen minutes
  • Hook mechanics: register_output_renderer()
  • Integration sketches beyond pyarrow
  • Comparison to CSV for million-row tables
  • Maintainer and CI notes
  • When this release is not news for you
  • Observability and abuse monitoring
  • Roadmap signals from Willison's release cadence
  • Ethics of publishing queryable public data
  • Troubleshooting common failures
  • Why Willison shipped Arrow before Parquet stream
  • Ecosystem placement next to Lance and DuckDB
  • Related reading
← Back to blog

explainx / blog

datasette-arrows 0.1a0: Streaming Apache Arrow Exports in Datasette

Datasette, Apache Arrow, Open Source, SQLite, Data Tools

Part of AI Tools and Apps

Simon Willison shipped datasette-arrows 0.1a0 on Nov 18, 2026: streaming .arrows exports on Datasette 1.0a43+ with ?_stream=1 and pyarrow consumers.

Nov 19, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
datasette-arrows 0.1a0: Streaming Apache Arrow Exports in Datasette

datasette-arrows 0.1a0 shipped on November 18, 2026 as a reference plugin for Datasette's new streaming export hook. It adds Apache Arrow IPC (.arrows) downloads from tables, views, and SQL queries without loading an entire SQLite file into RAM. Sources: Simon Willison's release note, PyPI, and the datasette/datasette-arrows README.

Willison wrote that he "never releases a new plugin feature without a plugin that uses it." Here, Claude Fable 5.2 suggested Arrow streaming and implemented most of the package in one shot—a useful data point next to Datasette 1.0a43's streaming CSV work we covered earlier in November.

Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

TL;DR

table · 2 cols
PieceDetail
Version0.1a0 on PyPI, 2026-11-18
Installdatasette install datasette-arrows
URLs/db/table.arrows, /db/-/query.arrows?sql=...
Stream all rows?_stream=1 or Export dialog checkbox
MIMEapplication/vnd.apache.arrow.stream
Python depsdatasette>=1.0a43, pyarrow
LicenseApache-2.0

Why Arrow after CSV and TSV streaming

Datasette export dialog showing the new Apache Arrow streaming optionDatasette export dialog showing the new Apache Arrow streaming option Image: Simon Willison

Datasette already streamed CSV and TSV because those formats chunk cleanly. Arrow IPC adds typed columns and zero-copy paths into pandas, Polars, and PyArrow workflows—closer to what LanceDB multimodal pipelines want than string parsing.

For publishers exposing public-interest SQLite—government logs, leaks, research drops—Arrow means a data journalist can read_all() in Python while a million-row table still streams from the server.

URL patterns and streaming semantics

Adapter pipes meeting through a junction evoking typed columnar export streams beside CSVAdapter pipes meeting through a junction evoking typed columnar export streams beside CSV

From the README:

  • Paginated default: /database/table.arrows returns one page like HTML.
  • Full stream: /database/table.arrows?_stream=1 emits one record batch per page until complete.
  • SQL queries: /database/-/query.arrows?sql=select+*+from+table
  • Download header: ?_dl=1 forces file download.
  • Governance: allow_stream and max_stream_mb Datasette settings apply—same guardrails as CSV streaming.

Filters, sort orders, and ?_col= projections apply to exports, so Arrow is not a bypass around Datasette's SQL safety.

Column typing rules that bite in production

SQL query funnel into a typed table, evoking SQLite columns mapped to Arrow typesSQL query funnel into a typed table, evoking SQLite columns mapped to Arrow types

SQLite allows mixed types; Arrow requires one type per column. The plugin infers from declared schema plus first page values:

table · 2 cols
SQLite values seenArrow type
Integersint64
Floats or int/float mixfloat64
Text or text/number mixstring
Blobs or mixed with blobsbinary
All null, no declared typestring

If a later page introduces an incompatible value—text in an INTEGER column—the export drops the connection so clients never treat a truncated stream as complete. Fix with explicit SQL casts, e.g. select cast(value as text) as value from ....

That behavior is stricter than CSV, which happily emits mangled strings. For ML exporters, strictness prevents silent label corruption.

Reading streams in Python

The README documents the minimal consumer:

python
import urllib.request
import pyarrow as pa

with urllib.request.urlopen(
    "http://localhost:8001/data/dogs.arrows?_stream=1"
) as response:
    table = pa.ipc.open_stream(response).read_all()

From there, zero-copy handoff to Polars, ** DuckDB**, or training pipelines is standard. Compare to Willison's Pyodide Datasette experiment themes: SQLite in the browser for demos, Arrow on the wire for analytics scale.

UI and release cadence caveats

Willison's screenshot uses Datasette main, where streaming formats will soon sit beside CSV/TSV under "All rows in this table" in the export dialog. 0.1a0 shipped before that UI grouping landed in a numbered alpha, so early adopters may need main or the next alpha for the polished dialog.

The plugin requires streaming output renderers—present on main, not necessarily on the latest tagged release you run in production. Pin versions explicitly in Docker images.

How this fits explainx.ai's tool coverage

datasette-arrows is small but exemplary: it shows how plugin hooks should ship with reference implementations. The same week brought Vercel Board for agent coordination and CodeRabbit Verify for runtime QA—different layers, same lesson: agents need contracts, not just models.

For local AI, Arrow exports pair with AirLLM-style constrained inference when you are featurizing public tables on modest GPUs—download features, not giant CSV text.

Security and publishing notes

Streaming exports can exfiltrate large tables if allow_stream is on for sensitive databases. Treat .arrows endpoints like CSV download permissions. Row-level auth plugins still apply; Arrow does not magically redact columns.

If you expose Datasette on the public internet, keep stored-query advisory fixes patched—streaming does not fix SQL injection in saved queries.

Try it in fifteen minutes

  1. Install Datasette main or >=1.0a43 with streaming renderers enabled.
  2. datasette install datasette-arrows
  3. datasette serve data.db
  4. Open a table, confirm Arrow in Export data.
  5. curl -o out.arrows 'http://localhost:8001/db/table.arrows?_stream=1'
  6. Load with pyarrow and compare row count to select count(*).

Open an issue on datasette/datasette-arrows if type coercion fails on real-world messy columns—that feedback shapes 1.0.

Hook mechanics: register_output_renderer()

Datasette 1.0a43 extended plugins so authors can register streaming output renderers alongside built-in CSV and TSV. The core idea: fetch a page of rows, serialize a chunk, repeat—never materialize the whole table in the Datasette process. datasette-arrows is the reference implementation for binary columnar output instead of delimited text.

Plugin authors should copy its patterns:

  • Honor ?_stream=1 and dialog Include all rows.
  • Respect allow_stream and max_stream_mb caps.
  • Fail hard on type conflicts rather than emit silent corruption.

If you maintain custom SQLite packs for newsrooms or NGOs, shipping Arrow lowers the bar for partner universities that already standardize on PyArrow.

Integration sketches beyond pyarrow

Polars can read Arrow streams with pl.read_ipc_stream. DuckDB can attach Arrow streams for SQL transforms before Parquet archival. Training stacks that expect HF datasets can still ingest Arrow, convert once to sharded Parquet, and push to LanceDB if you later need vector search.

For agents, a coding agent that knows Arrow schemas can validate column dtypes before fine-tune—fewer silent NaN bugs than CSV guesswork.

Comparison to CSV for million-row tables

CSV wins on universal tool support and human grep. Arrow wins when:

  • Columns are numeric-heavy (counts, lat/lon, embeddings serialized as lists).
  • You need stable dtypes across pages.
  • You pipeline into Python ML without parse overhead.

Benchmark on your table: time table.csv?_stream=1 versus table.arrows?_stream=1 into Polars on the same laptop. Arrow should dominate when columns > 20 or rows > 1M.

Maintainer and CI notes

The GitHub repo ships uv run pytest. Depend on datasette>=1.0a43 and pyarrow. Because streaming renderers lagged tagged releases when 0.1a0 shipped, pin Docker images to datasette:main only if you accept moving targets—otherwise wait for the next alpha Willison promised for export dialog grouping.

Watch Datasette plugins directory for composition: combine datasette-arrows with label plugins for foreign keys in Arrow exports via ?_labels=on.

When this release is not news for you

If you do not run Datasette, skip this plugin. If you only publish 500-row tables, CSV is fine. If you are on Datasette versions before 1.0a43, upgrade stored-query patches before experimenting.

Observability and abuse monitoring

Operators should log .arrows?_stream=1 hits the same way they log large CSV exports. A sudden spike may mean scraping, not legitimate research. Combine max_stream_mb with rate limits at the reverse proxy when tables contain PII.

For multitenant Datasette Cloud (if you run one), partition plugins per deployment so Arrow cannot become a covert exfil channel from restricted databases.

Roadmap signals from Willison's release cadence

Willison's post notes the export dialog UI will group streaming formats in the next alpha after 0.1a0. That means early adopters may field support tickets from users who do not see Include all rows beside Arrow yet. Document ?_stream=1 deep links in your data handbook until the UI catches up.

Future plugins may add Parquet stream, GeoArrow, or embedding-specific layouts using the same hook—datasette-arrows is the template, not the final catalog.

Ethics of publishing queryable public data

Arrow makes re-identification attacks faster when tables include quasi-identifiers. If you publish census microdata or leaked corp tables, review k-anonymity basics even when the format is modern. Streaming does not remove duty of care.

Journalists pairing Datasette with AI Overviews traffic loss stories should remember: machines read Arrow faster than humans read HTML, changing scraper economics overnight.

Troubleshooting common failures

Empty stream after 200 OK: Check whether you forgot ?_stream=1 on a multi-million-row table; the first response may be only one page.

Connection reset mid-download: Often type coercion failure on a later page. Re-run export with a SQL query that casts messy columns to text.

Plugin not listed: Confirm uv run datasette plugins shows datasette-arrows and that you are on a Datasette build with streaming renderers enabled.

pyarrow ImportError: Install pyarrow in the client environment reading the stream, not only on the server.

Foreign keys missing labels: Enable ?_labels=on on the URL or verify the labels plugin chain you expect is active.

Why Willison shipped Arrow before Parquet stream

Arrow IPC is row-batch oriented for incremental HTTP, matching Datasette's page loop. Parquet wants column stripes and is heavier to stream without custom footers. If Parquet streaming arrives later, expect different memory tradeoffs—Arrow remains the low-latency wire format for Python analytics.

Ecosystem placement next to Lance and DuckDB

Teams exporting SQLite snapshots to Lance for vector search can treat datasette-arrows as step one (fast extract) before offline convert. Teams on DuckDB can ATTACH Arrow streams for JOIN against CSV dimensions without double disk use.

None of that is automatic—you still write glue—but you avoid CSV parse tax on wide tables.

Related reading

  • Datasette 1.0a43 streaming exports and advisory
  • LanceDB Realtime Engine for streaming AI tables
  • Rust Anything incremental rewrite MCP
  • Higher SWE-bench scores versus merge rate
  • Google AI Studio custom domain publish panel
Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime

People in this article

  • Simon Willison →Independent open source developer and creator of Datasette
Explore people in AI →
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 16, 2026

Datasette 1.0a43: Private Stored-Query Leak Fixed, Streaming Exports Ship

Simon Willison shipped Datasette 1.0a43 on November 15, 2026 with a GitHub security advisory for private stored queries, plus streaming renderers, row JSON redesign, and companion plugins for maps, apps, and agents.

Jun 21, 2026

Turso: The SQLite-Compatible Database Rewritten in Rust — MVCC, Async I/O, Vector Search, and an MCP Server

Turso rewrites SQLite in Rust — keeping full SQL and file format compatibility while adding MVCC concurrent writes, async io_uring I/O, CDC, vector search, full-text search, and an MCP server mode. 20,000+ stars, 253 contributors, in beta. Here is the full breakdown and why it matters for the agentic era.

Nov 19, 2026

sand3d: Python 3D Modeling Built for LLMs, Not Blender Scripts

Sandflow released sand3d, a Python layer over Blender bpy that expresses washers, brackets, and robot-ready assemblies in short named programs. GitHub listed more than 2,150 stars on November 19, 2026, with Hugging Face datasets and docs sized for LLM prompts.