.svg)
e6data’s Architectural Bets: our Head of Engineering’s conversation w/Pete at Zero Prime Podcast
Discussing e6data’s Architectural Bets on the Zero Prime Podcast
%20(1).svg)
Our founding engineer and Head of Engineering, Sudarshan, recently went on the Zero Prime podcast and unpacked the internals of our compute engine.
“We don’t treat object stores like cold storage. And we don’t think your planner should be the bottleneck in a high-QPS workload.”
Sudarshan, Founding Engineer, e6data
It’s a story of breaking away from the driver-executor model, rethinking scheduling for the object-store era, and why atomic, per-component scaling actually matters.
The Real Problem (2025 edition)
Everyone says “compute and storage are decoupled.” Not really.
- You scale the cluster because some ad-hoc queries spike.
- That 10% of your workload defines your baseline cluster size.
- Your scheduler doesn’t react in real-time, so you over-provision just in case.
- You get 10% more queries, and you’re forced to double your warehouse size.
Today’s data infra ≠ Today’s compute requirements.
Our Architectural Decisions
We are building e6data by imagining a new playbook. No central coordinator. No one mega-driver. No lock-in to a single table format. Here’s the breakdown:
.svg)
Core Shifts We Made So Far:
1. Disaggregation of internals
- Separate the planner, metadata ops, and workers.
- Each scales independently, not as a monolith.
2. Dynamic, mid-query scaling
- Queries can scale up/down during execution.
- No pre-provisioning for worst-case. Just-in-time compute.
3. Push-based vectorized execution
- We’re similar to DuckDB/Photon but go deeper on compute orchestration.
- Useful when dealing with 1k+ concurrent user-facing queries.
4. No opinionated stack
- Bring your own catalog, governance layer, and format.
- Plug in; don’t port over.
Why It Matters
- Cost: We run 1000 QPS workloads at ~60% lower TCO than other engines.
- Latency: p95 under 2s, even with mixed workloads.
- No Lock-In: Use Iceberg today. Switch to Delta tomorrow. Doesn’t matter to us.
- Infra Reuse: Already on Kubernetes? Cool. We sit inside that.
Where We’re Headed
- Real-time ingest → queryable in <15s from object storage
- Vector + SQL → cosine similarity inside SQL filters
- AI-native enhancements → smart partitioning, query rewriting, and auto-guardrails
Frequently asked questions (FAQs)
We are universally interoperable and open-source friendly. We can integrate across any object store, table format, data catalog, governance tools, BI tools, and other data applications.
We use a usage-based pricing model based on vCPU consumption. Your billing is determined by the number of vCPUs used, ensuring you only pay for the compute power you actually consume.
We support all types of file formats, like Parquet, ORC, JSON, CSV, AVRO, and others.
e6data promises a 5 to 10 times faster querying speed across any concurrency at over 50% lower total cost of ownership across the workloads as compared to any compute engine in the market.
We support serverless and in-VPC deployment models.
We can integrate with your existing governance tool, and also have an in-house offering for data governance, access control, and security.
.svg){kind=logo}
.svg)
.svg)
.svg)
.png)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)