Data Ingestion in S3 Data Lake

Dhaval Patel

Data Entrepreneur | YouTuber | Ex - Bloomberg, Nvidia

I have 17 years of experience in Programming and Data Science working for big tech companies like NVIDIA and Bloomberg. I also run a famous YouTube channel called Codebasics where I pursue my passion for teaching. Codebasics is one of the top channels on YouTube when it comes to data science, machine learning, data structures, etc. I firmly believe that “Anyone can code” and I use analogies, simple explanations, and step-by-step storytelling to explain difficult concepts in such a way that even a high school student can understand them easily.

Hemanand Vadivel

Ex- Data Analytics Manager, 8+ Years in Europe, Microsoft Certified, Certified Supply Chain Professional

I’m a mechanical engineer who transitioned to a full time Data & Analytics manager in the UK & Germany by teaching myself Power BI, excel & anything else that was required to solve the problem. I have worked with complex data across Supply Chain, Sales, Marketing, Revenue Management, Finance and HR functions over the last 8 years to deliver effective solutions. To me, Analytics is an extra sensory organ that anyone can develop to become a superhuman at work. I have conducted several workshops and internal training to help non-technical people to become superhuman at work and now, really excited for this opportunity to share all that knowledge with you in this course.

Data Ingestion in S3 Data Lake

Lecture Content Locked

Get Notified for Next Cohort →

Contents

1: Welcome to Data Engineering for Data Analysts

1.1: How will this Course Help You as a Data Analyst?

1.2: What Exactly You’ll Learn?

1.3: Will this Course Help me as a Fresher?

1.4: Pre-requisites

2: Welcome to the Data Engineering Foundations Experience

2.1: Peter decides To Get Into Data Engineering

3: Enhance Your Learning Experience

3.1: How to Get Maximum Out of this Course

3.2: Join Discord for doubt clearing support & networking

3.3: Setting Video Resolution and Video Speed Globally

4: Data Engineering Foundation 1

4.1: OLAP vs OLTP

4.2: Role of Data Engineer, Data Analyst, AI Engineer

4.3: Data Lake, Data Warehouse, Data Lakehouse

4.4: ETL vs ELT

4.5: Quiz

5: Data Engineering Foundations: Data Storage

5.1: OLAP vs OLTP

5.2: Data Lake, Data Warehouse, Data Lakehouse

6: Data Engineering Foundations: Data Transformation & File Formats

6.1: ETL vs ELT

6.2: Medallion Architecture

6.3: Row-based File Formats - CSV, JSON

6.4: Columnar File Formats – Parquet

6.5: Quiz

7: Data Engineering Foundation 2

7.1: Medallion Architecture

7.2: Row-based File Formats - CSV, JSON

7.3: Columnar File Formats - Parquet

7.4: Quiz

8: Data Engineering Foundations: Data Warehousing

8.1: Data Normalization & Denormalization

8.2: What is Data Modeling

8.3: Star vs Snowflake Schema, Fact & Dimension Tables

8.4: What is Dimension Modelling?

8.5: Slowly Changing Dimensions (Type 1, 2)

9: Project: Build Your First ETL Pipeline Using AWS (Introduction)

9.1: Problem Statement

9.2: Data Engineering Architecture

10: Project: Build Your First ETL Pipeline Using AWS (Data Extraction & Storage)

10.1: AWS Account Setup

10.2: OLTP Data Understanding

10.3: AWS S3 Overview and Bucket Setup

10.4: Data Ingestion in S3 Data Lake

11: Project: Build Your First ETL Pipeline Using AWS (Data Transformation)

11.1: Log Files ETL: Local Data Transformation

11.2: AWS Lambda Overview

11.3: Log Files ETL: AWS Lambda Manual Trigger

11.4: Log Files ETL: AWS Lambda Automated Trigger

11.5: CSV Files ETL: AWS Glue

11.6: CSV Files ETL: Automation using S3 Events

11.7: Glue vs Lambda

11.8: Setting up Budget Alerts

12: Project: Build Your First ETL Pipeline Using AWS (Data Warehousing & Analytics)

12.1: AWS Athena Overview

12.2: Athena Setup

12.3: Ad-Hoc SQL Queries Using Athena

12.4: Amazon Redshift Overview

12.5: Load S3 Data into Redshift

12.6: Build Power BI Dashboard with Redshift Data

12.7: Incremental Data Load in the Data Warehouse

12.8: I need a favour

12.9: Exercise

12.10: Important Note

13: Cloud Tools: An Analyst’s Overview

13.1: How to use this chapter?

13.2: AWS vs Azure: An Analyst’s Overview

13.3: Snowflake: An Analyst’s Overview

13.4: Databricks: An Analyst’s Overview

13.5: Apache Airflow: An Analyst’s Overview

13.6: DBT: An Analyst’s Overview

13.7: Quiz

13.8: Exercise

14: Data Engineering Best Practices

14.1: Practice 1: Data Masking for PII Protection

14.2: Practice 2: Versioning Data and Pipelines

14.3: Practice 3: Maintaining Proper Documentation

14.4: Practice 4: Effective Data Modelling