Engineering Notes & Benchmarks
Understand how dataprep.dev is engineered for large datasets. Instead of running unreliable browser benchmarks, we document our testing methodology, architectural decisions, and realistic performance expectations.
How We Process 5GB Files Locally
Our entire stack runs in your browser. We leverage Web Workers to prevent UI freezing and WebAssembly (Wasm) for near-native compute speeds. The diagram below illustrates the data pipeline from file ingestion to clean export.
Step 1 · Input
Raw CSV / Excel
File API · Drag & Drop · No upload
Step 2 · Parse
PapaParse (Streaming)
Chunked reading · Back-pressure aware · Zero memory spikes
Step 3 · Background Thread
Web Workers
Off main thread · UI stays responsive · Parallel execution
Step 4 · Compute Engine
DuckDB-Wasm (In-Memory SQL)
Columnar execution · SIMD vectorization · Apache Arrow zero-copy
Step 5 · Export
Clean Output Export
CSV · Excel · JSON · Direct download to device
Representative Workloads
These are the engineering-grade test cases we use internally to validate performance, stability, and memory behavior. They reflect the types of real-world datasets our users encounter.
The 1GB Shopify Merge
The 5GB Local VLOOKUP
Architectural Comparison
How dataprep.dev compares to traditional approaches for handling large-scale data cleaning tasks.
| Metric | Traditional SaaS | Local Python Script | dataprep.dev |
|---|---|---|---|
| Data Upload Required | Yes | No | No |
| Execution Environment | Cloud Servers | Local OS | Browser Sandbox |
| Setup Time | Account Creation | Pip Install / Env | Zero / Instant |
| Privacy Risk | High | Low | Zero |
Why We Don't Offer Online Benchmarks
Browser performance depends heavily on your available RAM, CPU architecture, browser engine (V8 vs WebKit), and even other open tabs. An online benchmark often measures your device rather than our software. We focus on predictable local execution rather than fake progress bars. Instead, we publish our testing methodology so you can reproduce results on your own hardware.
Why is DuckDB-Wasm faster than hand-written JavaScript?
DuckDB-Wasm compiles a full C++ columnar database engine to WebAssembly. Unlike row-by-row JavaScript iteration, DuckDB processes data in columnar batches using SIMD (Single Instruction, Multiple Data) vectorization — the same technique used by high-performance analytical databases like ClickHouse and Apache Arrow. This means filtering, grouping, and joining millions of rows can approach the throughput of native desktop applications, all within the browser sandbox.
How do Web Workers prevent the browser from freezing?
JavaScript in browsers runs on a single main thread that also handles UI rendering and user interactions. When you process a large dataset synchronously, the main thread is blocked and the page becomes unresponsive. Web Workers run in separate background threads with their own event loop. dataprep.dev dispatches all heavy computation — parsing, SQL execution, transformations — to dedicated workers, keeping the UI thread free to render progress updates and remain interactive at all times.
What are the actual memory limits in the browser?
Modern browsers typically allow a single tab to allocate between 2–4 GB of memory (varies by browser, OS, and device). For datasets that exceed this threshold, dataprep.dev uses streaming strategies — reading, processing, and discarding data in chunks rather than loading entire files into memory at once. This allows us to process files significantly larger than available RAM, at the cost of multiple passes over the data. We recommend 8+ GB of system RAM for the best experience with multi-gigabyte workloads.
Performance varies by hardware, browser version, and dataset characteristics. Examples on this page are representative engineering workloads and do not constitute guaranteed benchmarks for any specific environment.
Try it yourself