v1.1 · MIT · DuckDB-powered

Query Parquet files
from the terminal with SQL

ParQL brings pandas-like operations and full SQL to the command line — local files, S3, GCS, Azure, and HTTP. Powered by DuckDB, no scripts required.

Get started View on GitHub
25+ Commands
1.1 Version
MIT License
Features

Everything a Parquet CLI should do

Purpose-built for data engineers who live in the terminal.

DuckDB fast

Columnar, vectorized execution. Streams huge files, only reads the columns you touch.

Full SQL

Windows, CTEs, joins, PIVOT, percentiles — everything DuckDB supports, straight from your shell.

Cloud-native

Read directly from S3, GCS, Azure Blob/ADLS, HDFS, and HTTPS. Predicate pushdown included.

ASCII plots

Histograms, bar charts, scatter plots — instantly, right in the terminal.

Safe by design

Identifier validation and parameterized queries throughout — no SQL injection surface.

Data quality

assert, profile, nulls, outliers — build QA into any pipeline.

Interactive shell

REPL mode for exploration. Load tables, run queries, cache results.

Multi-format output

Table, CSV, TSV, JSON, NDJSON, Markdown, Parquet. Pipe anywhere.

Quick start

Up and running in 30 seconds

# Install from PyPI
pip install parql

# Peek at a file
parql head data/sales.parquet -n 10

# Run SQL over one or many files
parql sql "SELECT country, SUM(revenue) AS total
           FROM t GROUP BY country ORDER BY 2 DESC" \
  -p t=data/sales.parquet

# Profile a dataset
parql profile data/sales.parquet

# Query directly from S3
parql agg s3://my-bucket/events/*.parquet \
  -g event_type -a "count():n"

Every command supports --format json, --format csv, and more — pipe results straight into jq, xsv, or your favourite tool.

Examples

Real workflows, one line each

# Preview + schema at a glance
parql head data/sales.parquet -n 5
parql schema data/sales.parquet
parql sample data/sales.parquet --fraction 0.01

# Distinct values, sorted
parql distinct data/sales.parquet -c country
# Group-by aggregation with a HAVING clause
parql agg data/sales.parquet \
  -g country \
  -a "sum(revenue):total,count():orders" \
  -h "total > 10000" -o "total DESC"

# Window function: running total per user
parql window data/events.parquet \
  --partition user_id --order ts \
  --expr "sum(amount) as running_total"

# Correlation between numeric columns
parql corr data/sales.parquet -c "quantity,price,revenue"
# Assertions — fail the pipeline on bad data
parql assert data/sales.parquet \
  --rule "row_count > 1000" \
  --rule "no_nulls(user_id)" \
  --rule "unique(order_id)"

# Full profile with outlier detection
parql profile data/sales.parquet --include-all

# Nulls per column
parql nulls data/sales.parquet
# Filter and write to CSV
parql write data/sales.parquet out.csv \
  --format csv -w "country='US'"

# Join two datasets
parql join data/users.parquet data/orders.parquet \
  --on user_id --how inner

# String ops with safe parameterized patterns
parql str data/users.parquet \
  -c email --operation extract \
  --pattern "@(.+)$"
Why ParQL

Built for data engineers who live in the terminal

ParQL wraps DuckDB's engine with commands that match how you actually work with Parquet — no scripts, no notebooks, just direct answers.

One-liners, not scripts

Aggregations, joins, window functions, and pivots — each is a single command with sensible flags.

Quality checks built in

assert, profile, nulls, outliers ship as first-class commands, not add-ons.

Local and cloud, same command

Point ParQL at a local file, a glob pattern, or an s3:// / gs:// / abfs:// URL. It just works.

Safe on untrusted data

Identifier validation and parameterized queries throughout — no injection surface, no pickle-based cache.

Cloud & remote

Query data anywhere

Local files, entire cloud buckets, or a URL — ParQL treats them all the same.

# AWS S3
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
parql head s3://bucket/path/data.parquet

# Google Cloud Storage
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json
parql agg gs://bucket/data/*.parquet -g region -a "count():n"

# Public GCS dataset (anonymous)
parql head "gs://anonymous@voltrondata-labs-datasets/diamonds/cut=Good/part-0.parquet"

# Azure ADLS Gen2 / Blob
parql head abfs://container@account.dfs.core.windows.net/path/data.parquet
parql head wasbs://container@account.blob.core.windows.net/path/data.parquet

# HDFS
export HDFS_NAMENODE=namenode
parql head hdfs://namenode/tmp/data.parquet

# Plain HTTPS
parql head https://example.com/data.parquet

# Glob patterns work everywhere
parql agg "data/year=*/month=*/*.parquet" -g year,month
FAQ

Frequently asked questions

How do I query a Parquet file without loading it into pandas?

Use parql sql "SELECT ... FROM t" -p t=file.parquet. ParQL uses DuckDB to stream results without materializing the full file in memory.

Can ParQL read Parquet files directly from S3?

Yes. Set your AWS credentials as environment variables and pass an s3://bucket/key.parquet URL to any command. Glob patterns like s3://bucket/year=*/*.parquet are also supported.

How do I get a data quality report from the command line?

parql profile file.parquet gives you per-column type, null count, distinct count, and numeric/string statistics in one command.

Does ParQL support window functions and pivots?

Yes — parql window and parql pivot expose DuckDB's window functions and PIVOT syntax with simple flags.

Is ParQL safe to run on untrusted files?

All column names are validated as SQL identifiers and every user-supplied pattern is passed as a parameter, so there's no SQL injection surface. The cache uses Parquet (never pickle), so no code execution risk.