Data Engineering in Action

Building Scalable Pipelines with Open Source Tools and Cloud Platforms¶

About the Author¶

Dr. Kushnazarov Farruh is a leading authority in generative AI, data engineering, and machine learning, possessing a robust background that harmonizes theory and practice. He currently serves as a Senior GenAI/Solution Architect at Alibaba Cloud Intelligence, where he pioneers generative AI solutions and steers comprehensive strategies aligned with organizational goals. Previously, Dr. Farruh served as a Data Scientist at iTutorGroup-Ping An and Midea Group, spearheading groundbreaking research on optimizing communication networks and smart technologies. He has also served as a professor, teaching mathematical modeling, computer science, and data engineering subjects. His illustrious career spans numerous AI-linked projects and scholarly endeavors, resulting in a distinguished portfolio comprising over 20 technical publications.

Preface¶

Welcome to Data Engineering in Action.

If you are reading this, you likely already know that building data systems is difficult. You do not need another lecture explaining that “data is the new oil” or that data engineering is an in-demand career. You need to know how to build systems that do not break at 3:00 AM. You need to know how to orchestrate a pipeline, how to model a lakehouse, and how to feed clean data into a machine learning model without causing a production incident.

Many data engineering books treat the field as a catalog of tools. They teach you the syntax of Apache Spark or the configuration files of Apache Kafka, but they leave you guessing how to wire them together. Other books focus entirely on distributed systems theory, leaving you without a single line of runnable code.

This book is different. It is a practical field guide designed to bridge the gap between theory and production. We follow a single, evolving storyline—the growth of a fictional e-commerce company called TuranMart—to provide the business context for every architectural decision we make.

More importantly, this book treats Artificial Intelligence as a data engineering problem. The rise of Generative AI has proven that models are only as good as the data pipelines that feed them. In this book, you will not just learn how to move tables; you will learn how to build document ingestion pipelines, feature stores, and vector databases to support production RAG applications.

Every chapter in this book follows a strict rhythm: we introduce a business problem, explain the conceptual foundation, show the production design pattern, and then walk you through a guided, runnable lab using Docker. By the end of this book, you will have a portfolio of working data architecture artifacts running on your own machine.

Who This Book Is For¶

This book is designed for:

• Students studying computer science, data science, or information systems who want to build a career in data engineering

• Software engineers who want to transition into data engineering or expand their skill set

• Data analysts who want to move beyond SQL and learn how to build production data pipelines

• Data scientists who want to understand the infrastructure that powers their models

• Anyone who is curious about how modern data systems are built and wants to learn by doing

What You Will Learn¶

By the end of this book, you will be able to:

• Understand the role of a data engineer and how it fits into the broader data team

• Design and implement data models for both OLTP and OLAP systems

• Build batch and streaming data pipelines using Apache Spark and Apache Flink

• Orchestrate complex workflows with Apache Airflow and modern alternatives

• Implement data governance, quality, and security best practices

• Build data platforms on cloud infrastructure (with a focus on Alibaba Cloud)

• Create data pipelines for AI and machine learning, including RAG applications

• Implement feature stores and model serving infrastructure

• Work with vector databases and embeddings

• Apply all these skills to real-world business problems through detailed case studies

How This Book Is Organized¶

The book is organized into six parts:

Part 1: Foundations of Data Engineering introduces you to the field, covering the key roles, the data landscape, and core concepts like data modeling and storage paradigms.

Part 2: Data Storage Solutions provides deep dives into the major categories of data storage systems, from relational and NoSQL databases to data lakes and lakehouses.

Part 3: Data Processing and Orchestration covers the tools and techniques for processing data at scale and managing complex workflows.

Part 4: Data Governance, Security, and Cloud Platforms explores the critical topics of data quality, security, and compliance, and shows you how to build data platforms on Alibaba Cloud.

Part 5: Data Engineering for AI and ML is where we dive into the exciting world of data engineering for artificial intelligence, covering RAG, ML pipelines, feature stores, and vector databases.

Part 6: Business Applications and Case Studies brings it all together with detailed, end-to-end case studies from different industries.

Prerequisites¶

To get the most out of this book, you should have:

• Basic programming knowledge (preferably in Python)

• Basic understanding of SQL and relational databases

• Familiarity with the command line

• A willingness to learn and experiment

If you are not comfortable with Python or SQL, don’t worry. Appendix B provides a quick primer on both.

Setting Up Your Environment¶

All the code examples in this book can be run on your local machine using Docker and open-source tools. Appendix A provides detailed instructions for setting up your development environment. Additionally, a complete GitHub repository with all the code examples, sample data, and hands-on exercises is available at:

https://github.com/k-farruh/data-engineering-in-action-book

A Note on Cloud Platforms¶

While this book covers open-source tools that can run anywhere, it also provides extensive coverage of Alibaba Cloud, one of the world’s leading cloud providers. If you don’t have access to Alibaba Cloud, don’t worry—the concepts and architectures discussed are applicable to any cloud platform, and you can adapt the examples to AWS, Azure, or Google Cloud.

Let’s Get Started¶

Data engineering is a challenging but incredibly rewarding field. The problems are complex, the tools are powerful, and the impact is immense. Every data-driven application, every machine learning model, and every business intelligence dashboard is built on the foundation of robust data pipelines created by data engineers.

I hope this book will be a valuable companion on your journey to becoming a skilled data engineer. Let’s dive in!

Dr. Kushnazarov Farruh
University Tashkent University of Information Technologies named after Muhammad al-Khwarizmi
2026 May 1st

• The students who have taken my data engineering courses over the years and provided invaluable feedback

• The open-source communities behind Apache Spark, Apache Flink, Apache Airflow, and countless other projects

• The Alibaba Cloud team for their support and documentation

• My colleagues and fellow educators in the data community

• My family for their patience and support during the writing process

Thank you all.

Table of Contents¶

Part 1: Foundations of Data Engineering¶

Chapter 1: Introduction to Data Engineering

• The Role of a Data Engineer

• Data Engineering vs. Data Science vs. Data Analytics

• The Data Landscape

• The Importance of Open-Source

• Data Engineering Architecture

Chapter 2: Data Modeling and Storage Paradigms

• Data Types and Structures

• Data Formats

• Data Modeling Fundamentals

• Data Storage Paradigms

• Data Quality and Data Lineage

Chapter 3: The Open-Source Ecosystem

• Why Open-Source Matters

• Navigating the Landscape

• Understanding Open-Source Licenses

• Choosing and Evaluating Projects

• How to Contribute

Part 2: Data Storage Solutions¶

Chapter 4: Relational Databases

• Introduction to Relational Databases

• PostgreSQL: The World’s Most Advanced Open Source Database

• MySQL: The World’s Most Popular Open Source Database

• PostgreSQL vs. MySQL

• Hands-On Exercise

Chapter 5: NoSQL Databases

• The NoSQL Movement

• Document Databases: MongoDB

• Wide-Column Stores: Cassandra

• Key-Value Stores: Redis

• Choosing the Right NoSQL Database

Chapter 6: Object Storage and Data Lakes

• The Rise of Object Storage

• MinIO: Open-Source Object Storage

• Building a Data Lake

• The Medallion Architecture

• Data Lake Best Practices

Chapter 7: Data Warehousing and Lakehouse Architectures

• Traditional Data Warehousing

• The Lakehouse Paradigm

• Delta Lake

• Apache Iceberg

• Apache Hudi

• Choosing Your Lakehouse

Part 3: Data Processing and Orchestration¶

Chapter 8: Data Processing Frameworks: Spark and Flink

• The Evolution of Big Data Processing

• Apache Spark: The Unified Analytics Engine

• Apache Flink: The True Stream Processor

• Practical Development

Chapter 9: Streaming Data with Kafka and Flink

• From Batch Thinking to Streaming Thinking

• Kafka as the Event Backbone

• Flink for Stateful Stream Processing

• Designing TuranMart’s Real-Time Clickstream Pipeline

• Operating Streaming Systems in Production

Chapter 10: Transformations, Testing, and Analytics Engineering

• From Raw Data to Trusted Models

• The Layered Modeling Pattern

• Transformations as Software

• Testing Data Like Software

• Documentation, Lineage, and Discoverability

• Semantic Modeling and Metrics

• A dbt-Style Analytics Engineering Workflow

• Analytics Engineering in the Broader Platform

Chapter 11: Data Orchestration and Workflow Management

• The Need for Orchestration

• Apache Airflow: The Open-Source Standard

• Modern Alternatives: Prefect and Dagster

• Data Orchestration on Alibaba Cloud

Part 4: Analytics, Governance, Security, and Cloud Platforms¶

Chapter 12: Data Observability and Pipeline Reliability

• From Monitoring to Observability

• Reliability Metrics: SLIs, SLOs, SLAs, and Error Budgets

• Instrumenting Pipelines with Metrics, Logs, and Traces

• Data Quality Checks and Data Contracts

• Lineage, Blast Radius, and Change Management

• Alerting, SLO Burn Rate, and Incident Response

• Reliability Architecture for TuranMart

• Common Pitfalls

Chapter 13: Data Governance and Security

• The Pillars of Data Governance

• Implementing Data Quality at Scale

• Data Security Best Practices

• Compliance and Regulations

• Data Catalogs and Discovery

Chapter 14: Data Engineering on Alibaba Cloud

• An Overview of the Alibaba Cloud Data Platform

• A Reference Architecture

• A Deeper Dive into Key Services

• Hybrid and Multi-Cloud Strategies

Chapter 15: Cost, Performance, and Scalability Engineering

• Key concepts: cost, performance, and scale as one system

• Storage and compute trade-offs

• Production design pattern: the cost-performance engineering loop

• Partitioning, file size, and the small-file problem

• Caching, statistics, and query planning

• Scalability patterns for data platforms

Chapter 16: Solution Selection Framework

• The Technology Evaluation Process

• Build vs. Buy vs. Open-Source

• Choosing a Database

• Choosing a Processing Framework

• Choosing a Cloud Provider

• Common Pitfalls to Avoid

Part 5: Data Engineering for AI and ML¶

Chapter 17: Data Engineering for RAG Applications

• Understanding RAG

• Building the Data Pipeline for RAG

• Vector Storage and Retrieval

• Production RAG Systems

• RAG on Alibaba Cloud

Chapter 18: ML Pipeline Engineering

• The ML Lifecycle

• Building Training Data Pipelines

• Model Training and Experiment Tracking

• Model Deployment

• Monitoring ML Systems in Production

• ML Pipelines on Alibaba Cloud

Chapter 19: Feature Stores and Model Serving

• The Feature Store

• Feast: The Leading Open-Source Feature Store

• Model Serving

• Feature Stores and Model Serving on Alibaba Cloud

Chapter 20: Vector Databases and Embeddings

• Embeddings: The Lingua Franca of AI

• The Challenge of Vector Search

• The Vector Database Landscape

• Managing Embedding Pipelines in Production

Part 6: Business Applications and Case Studies¶

Chapter 21: Case Study: Building a Real-time Customer 360 Platform

• Business Goals and Requirements

• Data Sources

• Architecture and Technology Choices

• Implementation Details and Challenges

Chapter 22: Case Study: Fraud Detection in Financial Services

• Business Goals and Requirements

• Challenges of Fraud Detection: Imbalance, Drift, and Delayed Labels

• Real-Time Fraud Detection Reference Architecture

• Feature Stores, Decisioning, Feedback Loops, and Governance

• Guided Lab: Running the Shared SecurePay Fraud-Detection Project

Appendices¶

Appendix A: Setting Up Your Development Environment

Appendix B: A SQL and Python Primer

Appendix C: The Data Engineering Career Path

Appendix D: Further Reading and Resources

Appendix E: Glossary of Terms

How to Use This Book¶

This book is designed to be read sequentially, as each chapter builds on the concepts introduced in previous chapters. However, if you are already familiar with certain topics, you can skip ahead to the chapters that interest you most.

Each chapter follows a similar structure:

1. Introduction: An overview of what you will learn

2. Concepts: Detailed explanations of the key concepts

3. Hands-On Examples: Practical code examples you can run yourself

4. Best Practices: Production-ready advice from real-world experience

5. Chapter Summary: A recap of the key takeaways

6. Exercises: Practice problems to reinforce your learning

Make sure to work through the hands-on exercises and examples. Data engineering is a practical skill that is best learned by doing.

Let’s begin your journey into the world of data engineering!