Data Engineering Architecture

12.1: Final Quiz

13.1: End Note

Dhaval Patel

Data Entrepreneur | YouTuber | Ex - Bloomberg, Nvidia

I have 17 years of experience in Programming and Data Science working for big tech companies like NVIDIA and Bloomberg. I also run a famous YouTube channel called Codebasics where I pursue my passion for teaching. Codebasics is one of the top channels on YouTube when it comes to data science, machine learning, data structures, etc. I firmly believe that “Anyone can code” and I use analogies, simple explanations, and step-by-step storytelling to explain difficult concepts in such a way that even a high school student can understand them easily.

Hemanand Vadivel

Ex- Data Analytics Manager, 8+ Years in Europe, Microsoft Certified, Certified Supply Chain Professional

I’m a mechanical engineer who transitioned to a full time Data & Analytics manager in the UK & Germany by teaching myself Power BI, excel & anything else that was required to solve the problem. I have worked with complex data across Supply Chain, Sales, Marketing, Revenue Management, Finance and HR functions over the last 8 years to deliver effective solutions. To me, Analytics is an extra sensory organ that anyone can develop to become a superhuman at work. I have conducted several workshops and internal training to help non-technical people to become superhuman at work and now, really excited for this opportunity to share all that knowledge with you in this course.

Data Engineering Architecture

Prev Next

Lecture content Locked

Enroll in Course to Unlock

Contents

1: Welcome to Data Engineering for Data Analysts

1.1: How will this Course Help You as a Data Analyst?

1.2: What Exactly You’ll Learn?

1.3: Will this Course Help me as a Fresher?

1.4: Pre-requisites

2: Enhance Your Learning Experience

2.1: How to Get Maximum Out of this Course

2.2: Join Discord for doubt clearing support & networking

2.3: Setting Video Resolution and Video Speed Globally

3: Data Engineering Foundation 1

3.1: OLAP vs OLTP

3.2: Role of Data Engineer, Data Analyst, AI Engineer

3.3: Data Lake, Data Warehouse, Data Lakehouse

3.4: ETL vs ELT

3.5: Quiz

4: Data Engineering Foundation 2

4.1: Medallion Architecture

4.2: Row-based File Formats - CSV, JSON

4.3: Columnar File Formats - Parquet

4.4: Quiz

5: Project: Build Your First ETL Pipeline Using AWS

5.1: Problem Statement

5.2: Data Engineering Architecture

6: Build Your First ETL Pipeline Using AWS (Data Extraction & Storage)

6.1: AWS Account Setup

6.2: OLTP Data Understanding

6.3: AWS S3 Overview and Bucket Setup

6.4: Data Ingestion in S3 Data Lake

7: Build Your First ETL Pipeline Using AWS (Data Transformation)

7.1: Log Files ETL: Local Data Transformation

7.2: AWS Lambda Overview

7.3: Log Files ETL: AWS Lambda Manual Trigger

7.4: Log Files ETL: AWS Lambda Automated Trigger

7.5: CSV Files ETL: AWS Glue

7.6: CSV Files ETL: Automation using S3 Events

7.7: Glue vs Lambda

7.8: Setting up Budget Alerts

8: Project: Build Your First ETL Pipeline Using AWS (Data Warehousing & Analytics)

8.1: AWS Athena Overview

8.2: Athena Setup

8.3: Ad-Hoc SQL Queries Using Athena

8.4: Amazon Redshift Overview

8.5: Load S3 Data into Redshift

8.6: Build Power BI Dashboard with Redshift Data

8.7: Incremental Data Load in the Data Warehouse

8.8: I need a favour

8.9: Exercise

8.10: Important Note

9: Cloud Tools: An Analyst’s Overview

9.1: How to use this chapter?

9.2: AWS vs Azure: An Analyst’s Overview

9.3: Snowflake: An Analyst’s Overview

9.4: Databricks: An Analyst’s Overview

9.5: Apache Airflow: An Analyst’s Overview

9.6: DBT: An Analyst’s Overview

9.7: Quiz

9.8: Exercise

10: Data Engineering Best Practices

10.1: Practice 1: Data Masking for PII Protection

10.2: Practice 2: Versioning Data and Pipelines

10.3: Practice 3: Maintaining Proper Documentation

10.4: Practice 4: Effective Data Modelling

10.5: Practice 5: Creating Clear Pipeline Diagrams

10.6: Quiz

10.7: Exercise

11: Data Engineering related Interview Questions for Data Analysts

11.1: How to use this chapter

11.2: Interview Questions

12: Final Course Completion Test

12.1: Final Quiz

13: Next Steps

13.1: End Note

Data Engineering Architecture

Lecture content Locked