Introduction¶
This glossary provides clear, concise definitions of key terms and concepts used throughout this book. It serves as a quick reference to help you understand the language of data engineering. All entries are drawn from the canonical shared/glossary.yml source file, which also provides Russian and Uzbek translations for multilingual editions.
A¶
ACID (Atomicity, Consistency, Isolation, Durability): A set of properties that guarantee reliable transaction processing in database systems. Atomicity ensures that transactions are all-or-nothing. Consistency ensures that a transaction brings the database from one valid state to another. Isolation ensures that concurrent transactions do not interfere with each other. Durability ensures that once a transaction is committed, it will remain so, even in the event of a system failure.
Airbyte: An open-source data integration platform that replicates data from various sources to data warehouses, lakes, and databases.
Airflow (Apache Airflow): An open-source platform for programmatically authoring, scheduling, and monitoring workflows. It uses Directed Acyclic Graphs (DAGs) to define data pipelines.
Amazon Redshift: A fully managed, petabyte-scale data warehouse service in the cloud provided by AWS.
Apache Beam: A unified programming model for both batch and streaming data processing, supporting multiple execution engines (runners) such as Spark and Dataflow.
Apache Cassandra: A highly scalable, distributed NoSQL database designed to handle large amounts of data across many commodity servers with no single point of failure.
Apache Flink: An open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications.
Apache Hadoop: An open-source framework for distributed storage and processing of large datasets using the MapReduce programming model.
Apache Hive: A data warehouse software built on top of Hadoop that provides SQL-like query capabilities for data stored in HDFS or other compatible file systems.
Apache Hudi: An open-source data management framework that provides record-level insert, update, and delete capabilities on data lakes, enabling incremental data processing.
Apache Iceberg: An open table format for huge analytic datasets, designed to improve on the performance and reliability of traditional data lake formats.
Apache Kafka: A distributed event streaming platform capable of handling trillions of events a day, widely used for building real-time data pipelines and streaming applications.
Apache NiFi: A software project designed to automate the flow of data between software systems with a visual interface for building data flows.
Apache Spark: A unified analytics engine for large-scale data processing, providing high-level APIs in Java, Scala, Python, and R, and an optimized engine that supports general execution graphs.
API (Application Programming Interface): A set of rules and protocols that allows one software application to interact with another. Data engineers frequently use APIs to extract data from external systems.
Avro: A row-based data serialization system developed within the Apache Hadoop project, which relies on schemas defined in JSON. It is widely used in streaming pipelines like Kafka.
AWS Glue: A fully managed extract, transform, and load (ETL) service by Amazon that makes it easy to prepare and load data for analytics.
AWS Kinesis: A platform for streaming data on AWS, offering services to load and analyze streaming data in real time.
AWS Lambda: A serverless compute service that lets you run code without provisioning or managing servers, triggered by events.
AWS S3 (Simple Storage Service): An object storage service from Amazon that offers industry-leading scalability, data availability, security, and performance.
Azure Data Factory: A fully managed, serverless data integration service by Microsoft for building ETL and ELT processes.
Azure Databricks: A fast, easy, and collaborative Apache Spark-based analytics platform on Microsoft Azure.
Azure Synapse Analytics: An enterprise analytics service by Microsoft that accelerates time to insight across data warehouses and big data systems.
B¶
Backpressure: A mechanism in stream processing systems where a downstream component signals to an upstream component to slow down data transmission because it cannot keep up with the current rate.
Batch Processing: A method of processing data where a large volume of data is collected over a period of time and then processed all at once, in contrast to stream processing.
Big Data: A term used to describe datasets that are so large or complex that traditional data processing applications are inadequate. Big Data is typically characterized by the ‘3 Vs’: Volume, Velocity, and Variety.
BigQuery: A fully managed, serverless data warehouse by Google that enables scalable analysis over petabytes of data.
C¶
CAP Theorem: A fundamental theorem in distributed systems stating that it is impossible for a distributed data store to simultaneously provide more than two of the following three guarantees: Consistency, Availability, and Partition Tolerance.
CDC (Change Data Capture): A technique used to identify and capture changes made to data in a database and deliver those changes to a downstream process or system in real-time or near real-time.
ClickHouse: An open-source column-oriented database management system that allows generating analytical data reports in real-time.
Columnar Storage: A method of storing data where values from the same column are stored together, highly efficient for analytical queries. Parquet and ORC are examples of columnar file formats.
Compaction: The process of combining many small files into fewer large files to improve query performance in a data lake or lakehouse.
Confluent: A data streaming platform based on Apache Kafka, offering additional enterprise features for building event-driven applications.
Consistency: In the context of databases, the requirement that any transaction will bring the database from one valid state to another, maintaining all defined rules and constraints.
D¶
DAG (Directed Acyclic Graph): A graph structure where nodes are connected by directed edges with no cycles. In data engineering, DAGs are commonly used to represent workflows in tools like Apache Airflow.
Data Catalog: A centralized repository that provides metadata about an organization’s data assets, including information about data sources, schemas, lineage, and ownership.
Data Contract: A formal agreement between data producers and consumers that defines the structure, semantics, and quality expectations of a dataset.
Data Drift: Unexpected and undocumented changes to data structure, semantics, and infrastructure that is a result of modern, fast-paced data development and breaks data pipelines and ML models.
Data Governance: The overall management of the availability, usability, integrity, and security of data used in an organization.
Data Lake: A centralized repository that allows storing all structured and unstructured data at any scale, typically using object storage without enforcing a schema on write.
Data Lineage: The ability to track the flow of data from its origin through all transformations and movements to its final destination, crucial for debugging, auditing, and understanding data dependencies.
Data Mart: A subset of a data warehouse focused on a specific business line or team, designed to provide quick access to relevant data for a particular group of users.
Data Mesh: A decentralized data architecture that treats data as a product and assigns ownership to domain teams, enabling scalable and autonomous data management.
Data Observability: The ability to fully understand the health of data in a system, including freshness, distribution, volume, schema, and lineage.
Data Pipeline: A series of data processing steps where the output of one step is the input to the next, used to move and transform data from source systems to target systems.
Data Product: An independent, reusable data asset built to deliver value to specific consumers. In a data mesh architecture, domain teams treat data as a product with clear ownership, quality guarantees, and interfaces.
Data Quality: The degree to which data is accurate, complete, consistent, timely, and fit for its intended use.
Data Skew: A condition in distributed computing where data is unevenly distributed across partitions, causing some nodes to process significantly more data than others, leading to performance bottlenecks.
Data Vault: A detail-oriented, historical tracking, and uniquely linked set of normalized tables that support one or more functional areas of business.
Data Warehouse: A central repository of integrated data from one or more disparate sources, storing current and historical data optimized for analytical queries.
dbt (data build tool): A transformation workflow that lets teams deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation.
Delta Lake: An open-source storage layer that brings ACID transactions to Apache Spark and big data workloads.
Denormalization: The process of intentionally introducing redundancy into a database schema to improve read performance, common in OLAP systems.
Dimensional Modeling: A data modeling technique used in data warehousing that structures data into fact tables and dimension tables.
Distributed System: A system in which components located on networked computers communicate and coordinate their actions by passing messages. Many modern data engineering tools such as Spark and Kafka are distributed systems.
Docker: A platform that uses OS-level virtualization to deliver software in packages called containers, enabling consistent environments across development and production.
Druid (Apache Druid): A high-performance real-time analytics database designed for fast slice-and-dice analytics on event data.
E¶
Elasticsearch: A distributed, RESTful search and analytics engine capable of addressing a growing number of use cases including log analytics and full-text search.
ELT (Extract, Load, Transform): A data integration process where data is first extracted from source systems, then loaded into a target system, and finally transformed within that target system.
Embedding: A vector representation of data used for semantic search and AI systems, capturing the meaning of text, images, or other data in a high-dimensional numerical space.
ETL (Extract, Transform, Load): A traditional data integration process where data is extracted from source systems, transformed in a staging area, and then loaded into a target system.
Event Sourcing: A pattern where state changes are logged as a sequence of events, storing the entire history of changes to allow reconstructing past states.
Event-Driven Architecture: A software architecture paradigm promoting the production, detection, consumption of, and reaction to events. Often implemented using message brokers like Apache Kafka.
Exactly-Once Semantics: A messaging guarantee in distributed systems ensuring that a message is processed exactly one time, neither lost nor duplicated, even in the event of failures.
F¶
Fact Table: In dimensional modeling, a fact table contains the quantitative data (measures) for analysis, along with foreign keys to dimension tables.
Feature Engineering: The process of using domain knowledge to extract features (characteristics, properties, attributes) from raw data. These features can be used to improve the performance of machine learning algorithms.
Feature Store: A shared repository of offline and online features for machine learning, enabling consistent feature computation and reuse across models.
Fivetran: A cloud-based automated data movement platform that extracts, loads, and transforms data from various sources into a central data warehouse.
G¶
Google BigQuery: A fully managed, serverless data warehouse that enables scalable analysis over petabytes of data using SQL.
Google Cloud Dataflow: A fully managed service for executing Apache Beam pipelines within the Google Cloud Platform ecosystem.
Google Cloud Pub/Sub: An asynchronous and scalable messaging service by Google that decouples services producing messages from services processing those messages.
Google Cloud Storage (GCS): A RESTful online file storage web service for storing and accessing data on Google Cloud Platform infrastructure.
Grafana: A multi-platform open-source analytics and interactive visualization web application used for monitoring and observability.
H¶
HDFS (Hadoop Distributed File System): A distributed file system designed to run on commodity hardware, providing high-throughput access to application data.
Hybrid Transactional/Analytical Processing (HTAP): A database architecture that supports both transactional (OLTP) and analytical (OLAP) workloads in a single system.
I¶
Idempotency: A property of an operation where applying it multiple times has the same effect as applying it once, crucial for ensuring that re-running a failed pipeline job does not produce duplicate or incorrect data.
InfluxDB: An open-source time series database (TSDB) optimized for storing and querying time-stamped data such as metrics and events.
J¶
JSON (JavaScript Object Notation): A lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate.
K¶
Kafka Connect: A framework for connecting Apache Kafka with external systems such as databases, key-value stores, search indexes, and file systems.
Kafka Streams: A client library for building applications and microservices where the input and output data are stored in Kafka clusters.
Kubernetes: An open-source container-orchestration system for automating computer application deployment, scaling, and management.
L¶
Lakehouse: A modern data architecture that combines the best features of data lakes and data warehouses, providing low-cost flexible storage with ACID transactions and schema enforcement.
Lambda Architecture: A data processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream processing methods, consisting of batch, speed, and serving layers.
Latency: The time delay between the initiation of a data request and the receipt of the response. Low latency is critical in real-time data processing systems.
M¶
MapReduce: A programming model for processing large datasets with a parallel, distributed algorithm on a cluster, consisting of Map (filter and sort) and Reduce (summary) functions.
Medallion Architecture: A data design pattern used to logically organize data in a lakehouse, with the goal of incrementally and progressively improving the structure and quality of data as it flows through layers (from Bronze to Silver to Gold).
Metadata: Data about data. In data engineering, metadata includes information such as table schemas, column names, data types, data lineage, and data quality metrics.
Milvus: An open-source vector database built for scalable similarity search and AI applications, designed to handle embedding vectors generated by machine learning models.
MinIO: A high-performance, S3-compatible object storage system designed for cloud-native workloads.
MLOps (Machine Learning Operations): A set of practices that aims to deploy and maintain machine learning models in production reliably and efficiently.
MongoDB: A source-available cross-platform document-oriented database program that stores data in flexible, JSON-like documents.
MySQL: An open-source relational database management system (RDBMS) widely used for web applications and data storage.
N¶
Neo4j: A graph database management system that stores data as nodes, relationships, and properties, optimized for traversing connected data.
Normalization: The process of organizing data in a database to reduce redundancy and improve data integrity by dividing large tables into smaller ones and defining relationships between them.
NoSQL: A category of database management systems that do not use the traditional relational (SQL) model, designed for specific use cases such as document storage, wide-column storage, key-value storage, and graph storage.
O¶
Object Storage: A storage architecture that manages data as objects rather than as file hierarchies or blocks. Amazon S3, Google Cloud Storage, and MinIO are examples of object storage systems.
OLAP (Online Analytical Processing): A category of software tools that provides analysis of data stored in a database, optimized for complex queries and aggregations, typically used for business intelligence and reporting.
OLTP (Online Transaction Processing): A category of data processing that focuses on transaction-oriented tasks, optimized for fast, concurrent read and write operations, typically used for operational applications.
ORC (Optimized Row Columnar): A columnar storage file format optimized for Hadoop workloads, providing efficient compression and encoding schemes.
Orchestration: Scheduling, dependencies, retries, monitoring, and operational control of data workflows to ensure tasks execute in the correct order.
P¶
Parquet: A columnar storage file format widely used in the Hadoop ecosystem, designed to be efficient for both storage and processing, supporting complex nested data structures.
Partition: A division of a database table or dataset into smaller, more manageable pieces, improving query performance by allowing the database to scan only the relevant partitions.
PII (Personally Identifiable Information): Any data that could potentially identify a specific individual. Handling PII requires special security measures, masking, and compliance with regulations like GDPR or CCPA.
PostgreSQL: A free and open-source relational database management system emphasizing extensibility and SQL compliance, widely used in production data engineering environments.
Predicate Pushdown: An optimization technique where filtering conditions (predicates) are pushed down to the storage layer, reducing the amount of data read from disk and transferred over the network.
Presto / Trino: Open-source distributed SQL query engines for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.
Pub/Sub (Publish-Subscribe): A messaging pattern where senders (publishers) send messages to a topic without knowledge of the receivers (subscribers), and subscribers receive messages from topics they are interested in.
R¶
RAG (Retrieval-Augmented Generation): A technique in natural language processing that combines information retrieval with large language models to generate more accurate and contextually relevant responses.
Redis: An open-source, in-memory data structure store used as a database, cache, and message broker, known for its high performance and support for various data structures.
Replication: The process of copying and maintaining database objects in multiple databases that make up a distributed database system, used for high availability and disaster recovery.
Reverse ETL: The process of moving data from a central data warehouse back into operational systems (like CRMs, marketing platforms, or support tools) to make analytics actionable.
S¶
SCD (Slowly Changing Dimension): A dimension in a data warehouse that changes slowly over time. There are several types (Type 1, Type 2, Type 3) that define how historical changes are tracked.
Schema: The structure or blueprint of a database, defining how data is organized, including tables, columns, data types, and relationships.
Schema Evolution: The ability to change the schema of a dataset over time (e.g., adding, removing, or renaming columns) without breaking existing readers or pipelines.
Schema Registry: A centralized repository for managing and validating schemas for streaming data (like Kafka topics), ensuring data compatibility between producers and consumers.
Schema-on-Read: A data management approach where the schema is applied when the data is read rather than when it is written, common in data lakes.
Schema-on-Write: A traditional data management approach where the schema is defined and enforced when data is written to the database, common in relational databases.
Sharding: A database architecture pattern where a large database is partitioned into smaller pieces called shards, each hosted on a separate server, enabling horizontal scaling.
SLA (Service Level Agreement): A formal agreement between a service provider and a client specifying the measurable metrics of the service, such as uptime, availability, and response time.
Snowflake: A cloud computing-based data cloud company offering a cloud-based data storage and analytics service with separate compute and storage scaling.
SQL (Structured Query Language): The standard language for managing and querying relational databases.
Star Schema: A type of dimensional model where a central fact table is surrounded by dimension tables, forming a star-like structure. It is the simplest and most common dimensional model.
Streaming: The continuous processing of data in real-time or near real-time as it is generated, as opposed to batch processing.
T¶
Throughput: The amount of data that can be processed or transferred in a given amount of time. In data engineering, high throughput is often a key performance metric.
Time Travel: The ability to query a table at a specific point in time in the past. Supported by modern table formats like Delta Lake, Iceberg, and Hudi.
Transformation: The process of converting data from one format or structure to another. This is the ‘T’ in ETL and ELT.
U¶
Upsert: A database operation that will update an existing row if a specified value already exists in a table, and insert a new row if the specified value does not already exist.
V¶
Vector Database: A specialized database designed to store and query high-dimensional vector embeddings, which are numerical representations of data used in machine learning and AI applications.
Velocity: One of the ‘3 Vs’ of Big Data, referring to the speed at which data is generated and must be processed.
Volume: One of the ‘3 Vs’ of Big Data, referring to the sheer amount of data that is generated and stored.
W¶
Watermark: A concept in stream processing used to handle late-arriving data. It represents a threshold of event time, indicating that the system expects no more events with a timestamp older than the watermark.
Window Function: A SQL function that performs a calculation across a set of table rows that are somehow related to the current row, unlike aggregate functions which group rows into a single output row.
Workflow Orchestration: The automated coordination and management of data workflows, ensuring that tasks are executed in the correct order and handling dependencies, retries, and failures.
This glossary covers the most important terms you will encounter in your data engineering journey. Refer back to it whenever you need a quick refresher on a concept.