Problem Statement

8.1: Quiz

9.1: Transitioning to Apache Spark & Databricks

Dhaval Patel

Data Entrepreneur | YouTuber | Ex - Bloomberg, Nvidia

I have 17 years of experience in Programming and Data Science working for big tech companies like NVIDIA and Bloomberg. I also run a famous YouTube channel called Codebasics where I pursue my passion for teaching. Codebasics is one of the top channels on YouTube when it comes to data science, machine learning, data structures, etc. I firmly believe that “Anyone can code” and I use analogies, simple explanations, and step-by-step storytelling to explain difficult concepts in such a way that even a high school student can understand them easily.

Hemanand Vadivel

Ex- Data Analytics Manager, 8+ Years in Europe, Microsoft Certified, Certified Supply Chain Professional

I’m a mechanical engineer who transitioned to a full time Data & Analytics manager in the UK & Germany by teaching myself Power BI, excel & anything else that was required to solve the problem. I have worked with complex data across Supply Chain, Sales, Marketing, Revenue Management, Finance and HR functions over the last 8 years to deliver effective solutions. To me, Analytics is an extra sensory organ that anyone can develop to become a superhuman at work. I have conducted several workshops and internal training to help non-technical people to become superhuman at work and now, really excited for this opportunity to share all that knowledge with you in this course.

Problem Statement

Prev Next

Problem Statement

In this lecture, we will explore the problem statement.

Contents

1.1: OLAP vs OLTP

1.2: Data Lake, Data Warehouse, Data Lakehouse

2.1: ETL vs ELT

2.2: Medallion Architecture

2.3: Row-based File Formats - CSV, JSON

2.4: Columnar File Formats – Parquet

2.5: Quiz

3.1: Data Normalization & Denormalization

3.2: What is Data Modeling

3.3: Star vs Snowflake Schema, Fact & Dimension Tables

3.4: What is Dimension Modelling?

3.5: Slowly Changing Dimensions (Type 1, 2)

4.1: Problem Statement

4.2: Data Engineering Architecture

5.1: AWS Account Setup

5.2: OLTP Data Understanding

5.3: AWS S3 Overview and Bucket Setup

5.4: Data Ingestion in S3 Data Lake

6.1: Log Files ETL: Local Data Transformation

6.2: AWS Lambda Overview

6.3: Log Files ETL: AWS Lambda Manual Trigger

6.4: Log Files ETL: AWS Lambda Automated Trigger

6.5: CSV Files ETL: AWS Glue

6.6: CSV Files ETL: Automation using S3 Events

6.7: Glue vs Lambda

6.8: Setting up Budget Alerts

7.1: AWS Athena Overview

7.2: Athena Setup

7.3: Ad-Hoc SQL Queries Using Athena

7.4: Amazon Redshift Overview

7.5: Load S3 Data into Redshift

7.6: Build Power BI Dashboard with Redshift Data

7.7: Incremental Data Load in the Data Warehouse

7.8: Exercise

8.1: Quiz

9.1: Transitioning to Apache Spark & Databricks

Problem Statement

Problem Statement

Contents

1: Data Engineering Foundations: Data Storage

1.1: OLAP vs OLTP

1.2: Data Lake, Data Warehouse, Data Lakehouse

2: Data Engineering Foundations: Data Transformation & File Formats

2.1: ETL vs ELT

2.2: Medallion Architecture

2.3: Row-based File Formats - CSV, JSON

2.4: Columnar File Formats – Parquet

2.5: Quiz

3: Data Engineering Foundations: Data Warehousing

3.1: Data Normalization & Denormalization

3.2: What is Data Modeling

3.3: Star vs Snowflake Schema, Fact & Dimension Tables

3.4: What is Dimension Modelling?

3.5: Slowly Changing Dimensions (Type 1, 2)

4: Project: Build Your First ETL Pipeline Using AWS (Introduction)

4.1: Problem Statement

4.2: Data Engineering Architecture

5: Project: Build Your First ETL Pipeline Using AWS (Data Extraction & Storage)

5.1: AWS Account Setup

5.2: OLTP Data Understanding

5.3: AWS S3 Overview and Bucket Setup

5.4: Data Ingestion in S3 Data Lake

6: Project: Build Your First ETL Pipeline Using AWS (Data Transformation)

6.1: Log Files ETL: Local Data Transformation

6.2: AWS Lambda Overview

6.3: Log Files ETL: AWS Lambda Manual Trigger

6.4: Log Files ETL: AWS Lambda Automated Trigger

6.5: CSV Files ETL: AWS Glue

6.6: CSV Files ETL: Automation using S3 Events

6.7: Glue vs Lambda

6.8: Setting up Budget Alerts

7: Project: Build Your First ETL Pipeline Using AWS (Data Warehousing & Analytics)

7.1: AWS Athena Overview

7.2: Athena Setup

7.3: Ad-Hoc SQL Queries Using Athena

7.4: Amazon Redshift Overview

7.5: Load S3 Data into Redshift

7.6: Build Power BI Dashboard with Redshift Data

7.7: Incremental Data Load in the Data Warehouse

7.8: Exercise

8: Final Quiz

8.1: Quiz

9: What’s Next?

9.1: Transitioning to Apache Spark & Databricks

Problem Statement

Problem Statement