CSV Files ETL: Automation using S3 Events

8.1: Quiz

9.1: Transitioning to Apache Spark & Databricks

Dhaval Patel

Data Entrepreneur | YouTuber | Ex - Bloomberg, Nvidia

I have 17 years of experience in Programming and Data Science working for big tech companies like NVIDIA and Bloomberg. I also run a famous YouTube channel called Codebasics where I pursue my passion for teaching. Codebasics is one of the top channels on YouTube when it comes to data science, machine learning, data structures, etc. I firmly believe that “Anyone can code” and I use analogies, simple explanations, and step-by-step storytelling to explain difficult concepts in such a way that even a high school student can understand them easily.

Hemanand Vadivel

Ex- Data Analytics Manager, 8+ Years in Europe, Microsoft Certified, Certified Supply Chain Professional

I’m a mechanical engineer who transitioned to a full time Data & Analytics manager in the UK & Germany by teaching myself Power BI, excel & anything else that was required to solve the problem. I have worked with complex data across Supply Chain, Sales, Marketing, Revenue Management, Finance and HR functions over the last 8 years to deliver effective solutions. To me, Analytics is an extra sensory organ that anyone can develop to become a superhuman at work. I have conducted several workshops and internal training to help non-technical people to become superhuman at work and now, really excited for this opportunity to share all that knowledge with you in this course.

CSV Files ETL: Automation using S3 Events

Lecture Content Locked

Enroll in Bootcamp to Unlock

Contents

1.1: OLAP vs OLTP

1.2: Data Lake, Data Warehouse, Data Lakehouse

2.1: ETL vs ELT

2.2: Medallion Architecture

2.3: Row-based File Formats - CSV, JSON

2.4: Columnar File Formats – Parquet

2.5: Quiz

3.1: Data Normalization & Denormalization

3.2: What is Data Modeling

3.3: Star vs Snowflake Schema, Fact & Dimension Tables

3.4: What is Dimension Modelling?

3.5: Slowly Changing Dimensions (Type 1, 2)

4.1: Problem Statement

4.2: Data Engineering Architecture

5.1: AWS Account Setup

5.2: OLTP Data Understanding

5.3: AWS S3 Overview and Bucket Setup

5.4: Data Ingestion in S3 Data Lake

6.1: Log Files ETL: Local Data Transformation

6.2: AWS Lambda Overview

6.3: Log Files ETL: AWS Lambda Manual Trigger

6.4: Log Files ETL: AWS Lambda Automated Trigger

6.5: CSV Files ETL: AWS Glue

6.6: CSV Files ETL: Automation using S3 Events

6.7: Glue vs Lambda

6.8: Setting up Budget Alerts

7.1: AWS Athena Overview

7.2: Athena Setup

7.3: Ad-Hoc SQL Queries Using Athena

7.4: Amazon Redshift Overview

7.5: Load S3 Data into Redshift

7.6: Build Power BI Dashboard with Redshift Data

7.7: Incremental Data Load in the Data Warehouse

7.8: Exercise

8.1: Quiz

9.1: Transitioning to Apache Spark & Databricks

CSV Files ETL: Automation using S3 Events

Lecture Content Locked

Contents

1: Data Engineering Foundations: Data Storage

1.1: OLAP vs OLTP

1.2: Data Lake, Data Warehouse, Data Lakehouse

2: Data Engineering Foundations: Data Transformation & File Formats

2.1: ETL vs ELT

2.2: Medallion Architecture

2.3: Row-based File Formats - CSV, JSON

2.4: Columnar File Formats – Parquet

2.5: Quiz

3: Data Engineering Foundations: Data Warehousing

3.1: Data Normalization & Denormalization

3.2: What is Data Modeling

3.3: Star vs Snowflake Schema, Fact & Dimension Tables

3.4: What is Dimension Modelling?

3.5: Slowly Changing Dimensions (Type 1, 2)

4: Project: Build Your First ETL Pipeline Using AWS (Introduction)

4.1: Problem Statement

4.2: Data Engineering Architecture

5: Project: Build Your First ETL Pipeline Using AWS (Data Extraction & Storage)

5.1: AWS Account Setup

5.2: OLTP Data Understanding

5.3: AWS S3 Overview and Bucket Setup

5.4: Data Ingestion in S3 Data Lake

6: Project: Build Your First ETL Pipeline Using AWS (Data Transformation)

6.1: Log Files ETL: Local Data Transformation

6.2: AWS Lambda Overview

6.3: Log Files ETL: AWS Lambda Manual Trigger

6.4: Log Files ETL: AWS Lambda Automated Trigger

6.5: CSV Files ETL: AWS Glue

6.6: CSV Files ETL: Automation using S3 Events

6.7: Glue vs Lambda

6.8: Setting up Budget Alerts

7: Project: Build Your First ETL Pipeline Using AWS (Data Warehousing & Analytics)

7.1: AWS Athena Overview

7.2: Athena Setup

7.3: Ad-Hoc SQL Queries Using Athena

7.4: Amazon Redshift Overview

7.5: Load S3 Data into Redshift

7.6: Build Power BI Dashboard with Redshift Data

7.7: Incremental Data Load in the Data Warehouse

7.8: Exercise

8: Final Quiz

8.1: Quiz

9: What’s Next?

9.1: Transitioning to Apache Spark & Databricks

CSV Files ETL: Automation using S3 Events

Lecture Content Locked