Log Files ETL: AWS Lambda Automated Trigger

Dhaval Patel

Data Entrepreneur | YouTuber | Ex - Bloomberg, Nvidia

I have 17 years of experience in Programming and Data Science working for big tech companies like NVIDIA and Bloomberg. I also run a famous YouTube channel called Codebasics where I pursue my passion for teaching. Codebasics is one of the top channels on YouTube when it comes to data science, machine learning, data structures, etc. I firmly believe that “Anyone can code” and I use analogies, simple explanations, and step-by-step storytelling to explain difficult concepts in such a way that even a high school student can understand them easily.

Hemanand Vadivel

Ex- Data Analytics Manager, 8+ Years in Europe, Microsoft Certified, Certified Supply Chain Professional

I’m a mechanical engineer who transitioned to a full time Data & Analytics manager in the UK & Germany by teaching myself Power BI, excel & anything else that was required to solve the problem. I have worked with complex data across Supply Chain, Sales, Marketing, Revenue Management, Finance and HR functions over the last 8 years to deliver effective solutions. To me, Analytics is an extra sensory organ that anyone can develop to become a superhuman at work. I have conducted several workshops and internal training to help non-technical people to become superhuman at work and now, really excited for this opportunity to share all that knowledge with you in this course.

Log Files ETL: AWS Lambda Automated Trigger

Lecture Content Locked

Get Notified for Next Cohort →

Contents

1: Welcome to Data Engineering for Data Analysts

1.1: How will this Course Help You as a Data Analyst?

1.2: What Exactly You’ll Learn?

1.3: Will this Course Help me as a Fresher?

1.4: Pre-requisites

2: Welcome to the Data Engineering Foundations Experience

2.1: Peter decides To Get Into Data Engineering

3: Enhance Your Learning Experience

3.1: How to Get Maximum Out of this Course

3.2: Join Discord for doubt clearing support & networking

3.3: Setting Video Resolution and Video Speed Globally

4: Data Engineering Foundation 1

4.1: OLAP vs OLTP

4.2: Role of Data Engineer, Data Analyst, AI Engineer

4.3: Data Lake, Data Warehouse, Data Lakehouse

4.4: ETL vs ELT

4.5: Quiz

5: Data Engineering Foundations: Data Storage

5.1: OLAP vs OLTP

5.2: Data Lake, Data Warehouse, Data Lakehouse

6: Data Engineering Foundations: Data Transformation & File Formats

6.1: ETL vs ELT

6.2: Medallion Architecture

6.3: Row-based File Formats - CSV, JSON

6.4: Columnar File Formats – Parquet

6.5: Quiz

7: Data Engineering Foundation 2

7.1: Medallion Architecture

7.2: Row-based File Formats - CSV, JSON

7.3: Columnar File Formats - Parquet

7.4: Quiz

8: Data Engineering Foundations: Data Warehousing

8.1: Data Normalization & Denormalization

8.2: What is Data Modeling

8.3: Star vs Snowflake Schema, Fact & Dimension Tables

8.4: What is Dimension Modelling?

8.5: Slowly Changing Dimensions (Type 1, 2)

9: Project: Build Your First ETL Pipeline Using AWS (Introduction)

9.1: Problem Statement

9.2: Data Engineering Architecture

10: Project: Build Your First ETL Pipeline Using AWS (Data Extraction & Storage)

10.1: AWS Account Setup

10.2: OLTP Data Understanding

10.3: AWS S3 Overview and Bucket Setup

10.4: Data Ingestion in S3 Data Lake

11: Project: Build Your First ETL Pipeline Using AWS (Data Transformation)

11.1: Log Files ETL: Local Data Transformation

11.2: AWS Lambda Overview

11.3: Log Files ETL: AWS Lambda Manual Trigger

11.4: Log Files ETL: AWS Lambda Automated Trigger

11.5: CSV Files ETL: AWS Glue

11.6: CSV Files ETL: Automation using S3 Events

11.7: Glue vs Lambda

11.8: Setting up Budget Alerts

12: Project: Build Your First ETL Pipeline Using AWS (Data Warehousing & Analytics)

12.1: AWS Athena Overview

12.2: Athena Setup

12.3: Ad-Hoc SQL Queries Using Athena

12.4: Amazon Redshift Overview

12.5: Load S3 Data into Redshift

12.6: Build Power BI Dashboard with Redshift Data

12.7: Incremental Data Load in the Data Warehouse

12.8: I need a favour

12.9: Exercise

12.10: Important Note

13: Cloud Tools: An Analyst’s Overview

13.1: How to use this chapter?

13.2: AWS vs Azure: An Analyst’s Overview

13.3: Snowflake: An Analyst’s Overview

13.4: Databricks: An Analyst’s Overview

13.5: Apache Airflow: An Analyst’s Overview

13.6: DBT: An Analyst’s Overview

13.7: Quiz

13.8: Exercise

14: Data Engineering Best Practices

14.1: Practice 1: Data Masking for PII Protection

14.2: Practice 2: Versioning Data and Pipelines

14.3: Practice 3: Maintaining Proper Documentation

14.4: Practice 4: Effective Data Modelling