[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"glossary-data-pipeline::en":3,"gloss-cluster-data-pipeline::en":20,"gloss-next-data-pipeline::en":9},{"slug":4,"category":5,"name":6,"definition":7,"meta_desc":8,"faq":9,"schema_markup":9,"related":10},"data-pipeline","data-infra","Data Pipeline","A data pipeline is a set of automated, usually sequential, processing steps that move data from one or more source systems (databases, APIs, event streams, file uploads) to a destination (a data warehouse, a vector store, an analytics dashboard, another application) while applying transformations along the way — cleaning malformed records, reshaping formats, enriching rows with additional data, or deduplicating entries. Why it matters for AI\u002FSaaS builders: almost every AI feature depends on a pipeline behind the scenes, even when the product surface looks simple. A \"chat with your data\" feature needs a pipeline that pulls documents from wherever they live, chunks and embeds them, and keeps the vector store in sync as source documents change. A usage-based billing feature needs a pipeline aggregating raw event data into daily\u002Fmonthly rollups. Building these as one-off scripts works for a demo; building them as observable, retryable, idempotent pipelines is what separates a fragile side project from a product customers trust with their data. How it works: pipelines are commonly described as either batch (processing accumulated data on a schedule — nightly, hourly) or streaming (processing events continuously as they arrive, often via a message queue like Kafka or Redis Streams). Most production pipelines are orchestrated by a scheduler\u002FDAG tool (Airflow, Dagster, Prefect, or simpler cron-based job runners for smaller teams) that tracks dependencies between steps, retries failures, and alerts on errors rather than failing silently. A well-designed pipeline is idempotent — re-running it on the same input produces the same output rather than duplicating data — which matters enormously when a step fails partway through and needs to be safely retried. Observability (logging row counts in and out of each stage, tracking pipeline run duration, alerting on anomalies) is what turns a pipeline from a black box into something a team can actually debug at 2am. Worked example: a B2B SaaS syncs customer CRM data nightly to power an AI-generated account-health summary. The pipeline: (1) extract — pull new\u002Fupdated records from Salesforce's API since the last successful run, using a stored `last_synced_at` watermark; (2) transform — normalize field names, drop PII fields not needed downstream, flag records missing required fields for a dead-letter queue instead of silently dropping them; (3) load — upsert into the internal Postgres warehouse table and re-embed any changed account-note fields into the vector store. Each stage logs its row counts to a monitoring dashboard, and a Slack alert fires if extracted-row-count drops more than 50% versus the 7-day average — catching a broken Salesforce API token before it silently stops syncing data for a week.","A data pipeline is an automated sequence of steps that moves and transforms data from source systems to a destination for use downstream.",null,[11,14,17],{"slug":12,"name":13},"batch-processing","Batch Processing",{"slug":15,"name":16},"data-warehouse","Data Warehouse",{"slug":18,"name":19},"etl","ETL",[21,25,28,31,32,36,39,42,45,48,52,55],{"slug":22,"category":5,"name":23,"updated_at":24},"acid","ACID","2026-08-24T02:46:37+00:00",{"slug":26,"category":5,"name":27,"updated_at":24},"ann-search","ANN Search",{"slug":29,"category":5,"name":30,"updated_at":24},"backpressure","Backpressure",{"slug":12,"category":5,"name":13,"updated_at":24},{"slug":33,"category":5,"name":34,"updated_at":35},"bm25","BM25","2026-08-24T02:46:38+00:00",{"slug":37,"category":5,"name":38,"updated_at":24},"cache","Cache",{"slug":40,"category":5,"name":41,"updated_at":24},"cap-theorem","CAP Theorem",{"slug":43,"category":5,"name":44,"updated_at":24},"change-data-capture","Change Data Capture (CDC)",{"slug":46,"category":5,"name":47,"updated_at":24},"chroma","Chroma",{"slug":49,"category":5,"name":50,"updated_at":51},"chunk-overlap","Chunk Overlap","2026-08-24T03:30:02+00:00",{"slug":53,"category":5,"name":54,"updated_at":24},"columnar-storage","Columnar Storage",{"slug":56,"category":5,"name":57,"updated_at":24},"connection-pooling","Connection Pooling"]