Log Files ETL: AWS Lambda Manual Trigger

12.1: Final Quiz

13.1: End Note

Dhaval Patel

Data Entrepreneur | YouTuber | Ex - Bloomberg, Nvidia

I have 17 years of experience in Programming and Data Science working for big tech companies like NVIDIA and Bloomberg. I also run a famous YouTube channel called Codebasics where I pursue my passion for teaching. Codebasics is one of the top channels on YouTube when it comes to data science, machine learning, data structures, etc. I firmly believe that “Anyone can code” and I use analogies, simple explanations, and step-by-step storytelling to explain difficult concepts in such a way that even a high school student can understand them easily.

Hemanand Vadivel

Ex- Data Analytics Manager, 8+ Years in Europe, Microsoft Certified, Certified Supply Chain Professional

I’m a mechanical engineer who transitioned to a full time Data & Analytics manager in the UK & Germany by teaching myself Power BI, excel & anything else that was required to solve the problem. I have worked with complex data across Supply Chain, Sales, Marketing, Revenue Management, Finance and HR functions over the last 8 years to deliver effective solutions. To me, Analytics is an extra sensory organ that anyone can develop to become a superhuman at work. I have conducted several workshops and internal training to help non-technical people to become superhuman at work and now, really excited for this opportunity to share all that knowledge with you in this course.

Contents

1: Welcome to Data Engineering for Data Analysts

1.1: How will this Course Help You as a Data Analyst?

1.2: What Exactly You’ll Learn?

1.3: Will this Course Help me as a Fresher?

1.4: Pre-requisites

2: Enhance Your Learning Experience

2.1: How to Get Maximum Out of this Course

2.2: Join Discord for doubt clearing support & networking

2.3: Setting Video Resolution and Video Speed Globally

3: Data Engineering Foundation 1

3.1: OLAP vs OLTP

3.2: Role of Data Engineer, Data Analyst, AI Engineer

3.3: Data Lake, Data Warehouse, Data Lakehouse

3.4: ETL vs ELT

3.5: Quiz

4: Data Engineering Foundation 2

4.1: Medallion Architecture

4.2: Row-based File Formats - CSV, JSON

4.3: Columnar File Formats - Parquet

4.4: Quiz

5: Project: Build Your First ETL Pipeline Using AWS

5.1: Problem Statement

5.2: Data Engineering Architecture

6: Build Your First ETL Pipeline Using AWS (Data Extraction & Storage)

6.1: AWS Account Setup

6.2: OLTP Data Understanding

6.3: AWS S3 Overview and Bucket Setup

6.4: Data Ingestion in S3 Data Lake

7: Build Your First ETL Pipeline Using AWS (Data Transformation)

7.1: Log Files ETL: Local Data Transformation

7.2: AWS Lambda Overview

7.3: Log Files ETL: AWS Lambda Manual Trigger

7.4: Log Files ETL: AWS Lambda Automated Trigger

7.5: CSV Files ETL: AWS Glue

7.6: CSV Files ETL: Automation using S3 Events

7.7: Glue vs Lambda

7.8: Setting up Budget Alerts

8: Project: Build Your First ETL Pipeline Using AWS (Data Warehousing & Analytics)

8.1: AWS Athena Overview

8.2: Athena Setup

8.3: Ad-Hoc SQL Queries Using Athena

8.4: Amazon Redshift Overview

8.5: Load S3 Data into Redshift

8.6: Build Power BI Dashboard with Redshift Data

8.7: Incremental Data Load in the Data Warehouse

8.8: I need a favour

8.9: Exercise

8.10: Important Note

9: Cloud Tools: An Analyst’s Overview

9.1: How to use this chapter?

9.2: AWS vs Azure: An Analyst’s Overview

9.3: Snowflake: An Analyst’s Overview

9.4: Databricks: An Analyst’s Overview

9.5: Apache Airflow: An Analyst’s Overview

9.6: DBT: An Analyst’s Overview

9.7: Quiz

9.8: Exercise

10: Data Engineering Best Practices

10.1: Practice 1: Data Masking for PII Protection

10.2: Practice 2: Versioning Data and Pipelines

10.3: Practice 3: Maintaining Proper Documentation

10.4: Practice 4: Effective Data Modelling

10.5: Practice 5: Creating Clear Pipeline Diagrams

10.6: Quiz

10.7: Exercise

11: Data Engineering related Interview Questions for Data Analysts

11.1: How to use this chapter

11.2: Interview Questions

12: Final Course Completion Test

12.1: Final Quiz

13: Next Steps

13.1: End Note

Log Files ETL: AWS Lambda Manual Trigger

This lecture is a part of Data Analytics Bootcamp 2.0. Upgrade now to access enhanced modules.

This lecture is a part of Data Analytics Bootcamp 2.0.
Upgrade now to access enhanced modules.