Data Engineering

Day 1 — Data Fundamentals

Data Formats

• JSON: parsing, serialization, streaming with ijson
• CSV: advanced handling, encoding issues, large file strategies
• Parquet: columnar storage, reading and writing with pyarrow
• Avro: schema evolution, binary serialization

Data Validation

• Pydantic: models, validators, serialization, settings management
• Schema validation strategies for incoming data
• Error handling and data quality reporting

Database Access

• SQLAlchemy Core: engine, connections, raw SQL, transactions
• SQLAlchemy ORM: models, sessions, relationships, queries
• Async database drivers: asyncpg, aiosqlite
• Introduction to ETL and ELT patterns

Day 2 — Pipeline Design

Orchestration

• Apache Airflow or Prefect: DAGs, tasks, scheduling, retries
• Pipeline design patterns: idempotency, checkpointing, backfilling
• Error handling and alerting in pipelines

Data Transformation

• pandas: DataFrames, Series, groupby, merge, pivot
• Advanced transformations: window functions, custom aggregations
• Memory optimization: chunking, dtypes, categorical data

Data Quality and Testing

• Testing data pipelines: fixtures, mock data, assertions
• Data quality checks: completeness, consistency, freshness
• Logging and monitoring pipeline execution

Day 3 — Scaling Up

Distributed Processing

• Distributed processing concepts: partitioning, shuffling, aggregation
• Introduction to Dask for parallel DataFrames
• When to use pandas vs Dask vs Polars

Cloud Integration

• Cloud storage: reading and writing to S3 and GCS
• Cloud-native pipeline services overview
• Infrastructure as Code basics for data pipelines

Real-Time Data

• Streaming concepts: producers, consumers, topics
• Working with message brokers from Python
• Best practices and project architecture for data projects

Practical Exercises

• Building a complete data pipeline from source to destination
• Implementing data validation and quality checks
• Deploying and monitoring a pipeline