Data Engineering Bootcamp 1.0: Job Assistance, Live Problem Solving & Virtual Internship

2.1: Setting Video Resolution and Video Speed Globally

2.2: Join Discord

3.1: Install MySQL: Windows

3.2: Install MySQL: Linux, Mac

3.3: Import Movies Dataset in MySQL

3.4: Retrieve Data Using Text Query (SELECT, WHERE, DISTINCT, LIKE)

3.5: Exercise - Retrieve Data Using Text Query (SELECT, WHERE, DISTINCT, LIKE)

3.6: Retrieve Data Using Numeric Query (BETWEEN, IN, ORDER BY, LIMIT, OFFSET)

3.7: Exercise - Retrieve Data Using Numeric Query (BETWEEN, IN, ORDER BY, LIMIT, OFFSET)

3.8: Summary Analytics (MIN, MAX, AVG, GROUP BY)

3.9: Exercise - Summary Analytics (MIN, MAX, AVG, GROUP BY)

3.10: HAVING Clause

3.11: Calculated Columns (IF, CASE, YEAR, CURYEAR)

3.12: Exercise - Calculated Columns (IF, CASE, YEAR, CURYEAR)

3.13: The Data God’s Blessing

3.14: Quiz

3.15: Chapter Summary

4.1: Why do We Need Multiple Tables?

4.2: SQL Joins (INNER, LEFT, RIGHT, FULL)

4.3: Exercise - SQL Joins (INNER, LEFT, RIGHT, FULL)

4.4: Cross Join

4.5: Analytics on Tables

4.6: Join More Than Two Tables

4.7: Exercise - Join More Than Two Tables

4.8: Quiz

4.9: Chapter Summary

5.1: Subqueries

5.2: ANY, ALL Operators

5.3: Co-Related Subquery

5.4: Exercise: Subqueries

5.5: Common Table Expression (CTE)

5.6: CTE Benefits & Other Applications

5.7: Exercise: CTE

5.8: Peter Pandey Flexes his SQL Skills

5.9: Quiz

5.10: Chapter Summary

6.1: Database Normalization and Data Integrity

6.2: Entity Relationship Diagram (ERD)

6.3: Mentor Talk: Art of Googling

6.4: Data Types: Numeric (INT, DECIMAL, FLOAT, DOUBLE)

6.5: Data Types: String (VARCHAR, CHAR, ENUM)

6.6: Data Types: Date, Time (DATETIME, DATE, TIME, YEAR, TIMESTAMP)

6.7: Data Types: JSON, Spatial (JSON, GEOMETRY)

6.8: Luck Favors the LinkedIn Post

6.9: Primary key

6.10: Foreign Key

6.11: Create a Database From an Entity Relationship Diagram - ERD

6.12: Import Data From a CSV File Into a Database

6.13: Insert Statement

6.14: Update and Delete

6.15: I Need a Favour

6.16: Expect the Unexpected: The Intermission Scene

6.17: Quiz

6.18: Chapter Summary

7.1: Window Functions: OVER Clause

7.2: Window Functions: ROW_NUMBER, RANK, DENSE_RANK

7.3: Quiz

8.1: Final Quiz

9.1: Step Into the Practice Room (SQL)

1.1: Join the discord channel

2.1: SQL Practice Questions in Practice Arena

1.1: Python for Data Engineering

1.2: Important Note [ Mandatory ]

2.1: Join Discord for doubt clearing support & networking

2.2: Setting Video Resolution and Video Speed Globally

3.1: Python Installation - Windows

3.2: Python Installation - Linux

3.3: Python Installation - Mac

3.4: How to Download Code and Get Help If I Have a Question?

4.1: Variables

4.2: Numbers

4.3: Strings

4.4: Strings - Exercise

4.5: Quiz: Variables, Numbers and Strings

4.6: Peter shares his Coding Fear with Tony

4.7: Exercise: Variables, Numbers and Strings

4.8: Chapter Summary

5.1: Lists

5.2: Install Pycharm

5.3: If Condition

5.4: For Loop

5.5: Quiz: Lists, If Condition and For Loop

5.6: Peter Found Something Interesting to Practice

5.7: Exercise: Lists, If Condition and For Loop

5.8: Chapter Summary

6.1: Functions

6.2: Dictionary and Tuples

6.3: Modules and Pip

6.4: File Handling

6.5: Quiz: Functions, Dictionaries, Tuples and File Handling

6.6: Peter’s Request to Tony

6.7: Exercise: Functions, Dictionaries, Tuples and File Handling

6.8: Two Deadly Viruses Infecting Learners

6.9: Chapter Summary

7.1: Classes and Objects

7.2: Exception Handling

7.3: __main__ Function

7.4: Quiz: Classes, Exception Handling

7.5: Using Python in a Real Time Problem

7.6: Exercise: Classes and Exception Handling

7.7: Chapter Summary

8.1: Pandas Introduction and Installation

8.2: Dataframe Basics

8.3: Read, Write Excel and CSV Files

8.4: Handle NA values - Part 1

8.5: Handle NA values - Part 2

8.6: Group by

8.7: Concat and Merge

8.8: Data Visualization Using Matplotlib and Seaborn

8.9: Quiz: EDA using Python

9.1: Working with JSON

9.2: What is API?

9.3: Calling APIs With requests Package

9.4: Building APIs With FastAPI

9.5: Quiz: APIs

9.6: Peter fetches data using APIs

9.7: Exercise: APIs

9.8: Chapter Summary

10.1: Logging

10.2: Automated Testing with Pytest

10.3: MySQL Setup: Windows

10.4: MySQL Setup: Linux, Mac

10.5: Working with MySQL in Python

10.6: Data Validation with Pydantic

10.7: Quiz: Logging, Pytest, Pydantic and Databases

10.8: Exercise: Pytest, Pydantic

10.9: Chapter Summary

11.1: Final Quiz

12.1: Step Into the Practice Room (Python)

2.1: Scenario Question

2.2: Have you solved the problem?

2.3: Quality Check -1

2.4: Quality Check - 2

3.1: Scenario Question

3.2: Have you solved the problem?

3.3: Quality Check - 1

3.4: Quality Check - 2

4.1: Scenario Question

4.2: Have you solved the problem?

4.3: Quality Check -1

4.4: Quality Check - 2

5.1: Scenario Question

5.2: Have you solved the problem?

5.3: Quality Check -1

5.4: Quality Check - 2

6.1: Scenario Question

6.2: Have you solved the problem?

6.3: Quality Check - 1

6.4: Quality Check - 2

7.1: Scenario Question

7.2: Have you solved the problem?

7.3: Quality Check -1

7.4: Quality Check -2

8.1: Scenario Question

8.2: Have you solved the problem?

8.3: Quality Check -1

8.4: Quality Check - 2

9.1: Scenario Question

9.2: Have you solved the problem?

9.3: Quality Check -1

9.4: Quality Check - 2

10.1: Coming Soon!

2.1: Introduction to online credibility / open-source contribution

3.1: Set Up Your LinkedIn Profile

3.2: LinkedIn Banners

4.1: Add your Python Project to GitHub

4.2: Share Your Python Skill to the World

4.3: Exercise: Add your SQL project to GitHub and Share your SQL skill to the world

5.1: Knowledge Check

5.2: Transitioning to Data Engineering Foundations

5.3: Checkpoint

1.1: OLAP vs OLTP

1.2: Data Lake, Data Warehouse, Data Lakehouse

2.1: ETL vs ELT

2.2: Medallion Architecture

2.3: Row-based File Formats - CSV, JSON

2.4: Columnar File Formats – Parquet

2.5: Quiz

3.1: Data Normalization & Denormalization

3.2: What is Data Modeling

3.3: Star vs Snowflake Schema, Fact & Dimension Tables

3.4: What is Dimension Modelling?

3.5: Slowly Changing Dimensions (Type 1, 2)

4.1: Problem Statement

4.2: Data Engineering Architecture

5.1: AWS Account Setup

5.2: OLTP Data Understanding

5.3: AWS S3 Overview and Bucket Setup

5.4: Data Ingestion in S3 Data Lake

6.1: Log Files ETL: Local Data Transformation

6.2: AWS Lambda Overview

6.3: Log Files ETL: AWS Lambda Manual Trigger

6.4: Log Files ETL: AWS Lambda Automated Trigger

6.5: CSV Files ETL: AWS Glue

6.6: CSV Files ETL: Automation using S3 Events

6.7: Glue vs Lambda

6.8: Setting up Budget Alerts

7.1: AWS Athena Overview

7.2: Athena Setup

7.3: Ad-Hoc SQL Queries Using Athena

7.4: Amazon Redshift Overview

7.5: Load S3 Data into Redshift

7.6: Build Power BI Dashboard with Redshift Data

7.7: Incremental Data Load in the Data Warehouse

7.8: Exercise

8.1: Quiz

9.1: Transitioning to Apache Spark & Databricks

2.1: Basic Operations

2.2: Reading Data from CSV

2.3: Handle Missing Values

2.4: SQL in pyspark

2.5: Joins

3.1: Reading Spark Plans with explain()

3.2: Spark Architecture

3.3: Transformations vs Actions and Lazy Evaluation

3.4: Narrows vs Wide Transformations

3.5: Partitions & Parallelism: Repartition

3.6: Partitions & Parallelism: Coalesce

3.7: Catalyst Optimizer, Predicate Pushdown, Column Pruning

3.8: Joins at Scale: Broadcast, Shuffle Hash and Hints

3.9: Data Skew and Mitigation Techniques

3.10: Adaptive Query Execution (AQE)

3.11: RDD, Dataframes and Datasets

3.12: Managed vs External Tables

3.13: Unity Catalog: Governance & Access Control

3.14: Understanding ACID

3.15: Time Travel Explained

3.16: What is Delta Lake and Delta Format

4.1: Final Quiz

2.1: Create a Free Azure Account

2.2: Databricks Setup in Azure

2.3: Upload Data in ADLS

2.4: Setup Catalog and Azure Connectors

2.5: Define Raw Schema

3.1: Ingest Dimension Data into Bronze Layer

3.2: Bronze to Silver: Data Cleansing and Transformation

3.3: Silver to Gold: Building BI Ready Tables

4.1: Batch vs Stream Processing

4.2: Autoloader and Structured Streaming

4.3: Ingest Fact Data into Bronze

4.4: Bronze to Silver: Data Cleansing and Transformation

4.5: Silver to Gold: Building BI Ready Tables

4.6: Build Daily Summary Table

5.1: Dimensional Pipeline

5.2: Fact Pipeline

5.3: Daily Refresh Job

6.1: Governance through Unity Catalog

6.2: Build BI Dashboard in Power BI

7.1: Exercise Walkthrough

2.1: Spark Setup Guide

3.1: Scenario Question

3.2: Have you completed the problem?

3.3: Q1: Quality Check - 1

3.4: Q1: Quality Check -2

4.1: Scenario Question

4.2: Have you completed the problem?

4.3: Q2: Quality Check - 1

4.4: Q2: Quality Check -2

5.1: Scenario Question

5.2: Have you completed the problem?

5.3: Q3: Quality Check - 1

5.4: Q3: Quality Check - 2

6.1: Coming Soon!

2.1: What is Smart Job Assistance Portal, and why is it important?

3.1: What is ATS and Why it is Important?

3.2: Build an ATS Resume using Resume Builder Checklist

3.3: Improve the ATS Score of your Resume

3.4: Sample Resumes

4.1: Getting Started: Creating a Portfolio Website

4.2: Add Basic Info: Intro Video

4.3: Add Basic Info: Profile Picture

4.4: Add Basic Info: About Yourself

4.5: Add Resume and Skills

4.6: Add Projects & Certifications

4.7: Generate Portfolio and Share it With Potential Recruiters

5.1: Update Headline, Experience, etc, Sections on LinkedIn

5.2: How to Engage Meaningfully on LinkedIn

5.3: How to approach people on LinkedIn

5.4: List of Influencers to Follow

6.1: Practical Resources to Build Domain Knowledge

7.1: Transitioning to Snowflake and Airflow

2.1: What is Snowflake?

2.2: Snowflake Free Trial and Registration

2.3: Your First SQL Queries with Tasty Bytes

2.4: Key Features and Advantages

2.5: Snowflake vs Traditional Datawarehouse

2.6: Snowflake vs Databricks

3.1: Snowflake Architecture

3.2: Virtual Warehouses

3.3: Databases, Schemas, Tables, Views

3.4: Roles, Users, Access Control

4.1: Integrations

4.2: Stages (Internal and External)

4.3: File Pattern and Format

4.4: COPY Command

4.5: Hands on Practice: Storage Integration

4.6: Hands on Practice: Load Data from S3

4.7: Snowpipe

5.1: Time Travel

5.2: Zero Copy Cloning

6.1: Quiz

2.1: Create Your First DAG

2.2: DAG, Task, Operator

3.1: Hook, Provider, Task Group

4.1: Final Quiz

2.1: Massive/Polygon Setup

2.2: Extract Historical Data using Polygon

2.3: Backfill Raw Historical Data in Snowflake

2.4: Transform and Load into Core

2.5: Populate Dim Tables

2.6: Populate Fact Tables

3.1: Daily File Download through Airflow

3.2: Setup Connection with S3

3.3: Upload Data to S3 via DAG

3.4: Create S3 Stage in Snowflake

3.5: Connect Airflow with Snowflake

3.6: Ingest Data in Snowflake RAW

3.7: Transform from RAW to CORE

3.8: Ingest Invalid Records in Reject Table

4.1: Connect Airflow with Slack

4.2: Add Slack Integration in DAG

4.3: What is SA Layer?

4.4: Build Views for SA Layer

4.5: Power BI Dashboard

2.1: Step Into the Practice Room (Snowflake & Airflow)

2.1: Taking You One Step Closer To Your Job

3.1: Introduction to the Job Application Playbook

2.1: Interview Question Bank: Technical

3.1: Introduction to the Interview Preparation Playbook

4.1: Fill This Form When You Land a Job

5.1: Transition to Virtual Internship (Trust the Process)

2.1: Welcome Note

2.2: Your Onboarding Letter

2.3: Welcome Note From Your Manager

2.4: Let's dive right into Week 1!

2.5: Your First Task

2.6: Getting Help From Your Seniors / Fellow Team Members

2.7: Incoming Task Email 1

2.8: Have you completed this task?

2.9: Quality Check 0

2.10: Quality Check 1

2.11: Quality Check 2

2.12: Quality Check 3

2.13: Congratulations you have completed the first task of your internship.

2.14: You Need to understand Scrum in Jira

2.15: Incoming Task Email 2

2.16: Have you completed the assigned task?

2.17: Scrum & Jira Knowledge Check

2.18: Congratulations you have completed week 1 Successfully.

3.1: Let's dive right into Week 2!

3.2: Metadata Logging

3.3: Incoming Task Email 1

3.4: Have you completed the assigned task?

3.5: Quality Check 1

3.6: Quality Check 2

3.7: Congratulations on finishing this task!

3.8: Incoming Task Email 2

3.9: Have you completed this task?

3.10: Quality Check 1

3.11: Quality Check 2

3.12: Quality Check 3

3.13: Quality Check 4

3.14: Congratulations on successfully completing the task!

3.15: Critical Presentation Deck

3.16: Incoming Task Email 3

3.17: Have you completed this task?

3.18: Congratulations you have completed week 2 successfully

4.1: Let's dive right into the Final 2 Weeks !

4.2: Can You Handle the FinSight Corp Project?

4.3: Incoming Task Email: Capstone Project

4.4: Please don't share datasets

4.5: Have you completed the task?

4.6: Submit Your Notebook

4.7: Implement data quality & validation framework

4.8: Incoming Task Email

4.9: Have you completed the assigned task?

4.10: Quality Check

4.11: Incoming Task Email: Handling Schema Drift & Column Chaos

4.12: Have you handled the schema drift in your solution?

4.13: Quality Check

4.14: Incoming Task Email: Resilience, Quarantine & Final Reporting

4.15: Have you completed this task?

4.16: End Note

4.17: Get Your Letter Of Completion

5.1: Fill This Form When You Land a Job

6.1: Final Note: End of the Bootcamp and All the best

2.1: Backlog Grooming Meeting: Gross Sales Report

2.2: User-Defined SQL Functions

2.3: Exercise: User-Defined SQL Functions

2.4: Gross Sales Report: Monthly Product Transactions

2.5: Gross Sales Report: Total Sales Amount

2.6: Exercise: Yearly Sales Report

2.7: Stored Procedures: Monthly Gross Sales Report

2.8: Stored Procedure: Market Badge

2.9: Benefits of Stored Procedures

2.10: Quiz

3.1: Problem Statement and Pre-Invoice Discount Report

3.2: Performance Improvement # 1

3.3: Performance Improvement # 2

3.4: Database Views: Introduction

3.5: Database Views: Post Invoice Discount, Net Sales

3.6: Exercise: Database Views

3.7: Top Markets and Customers

3.8: Exercise: Top Products

3.9: The Two Most Important Skills for the Data Analyst

3.10: Quiz

4.1: Supply Chain Basics : Simplified

4.2: Problem Statement

4.3: Create a Helper Table

4.4: Database Triggers

4.5: Database Events

4.6: Temporary Tables & Forecast Accuracy Report

4.7: Exercise: CTE, Temporary Tables

4.8: Subquery vs CTE vs Views vs Temporary Table

4.9: User Accounts and Privileges

4.10: Database Indexes: Overview

4.11: Database Indexes: Composite Index

4.12: Database Indexes: Index Types

4.13: Quiz

2.1: Problem Statement & Tech Architecture

2.2: Database CRUD Operations

2.3: Automated Tests Setup for CRUD

2.4: Expense Management: Backend (FastAPI)

2.5: Expense Management: Logging

2.6: Streamlit Introduction

2.7: Expense Management: Frontend (Streamlit)

2.8: Analytics: Backend (FastAPI)

2.9: Analytics: Frontend (Streamlit)

2.10: README and Requirements.txt

2.11: Exercise

3.1: Technical Architecture of the Project

3.2: Installation of Necessary Libraries

3.3: Extract text from a pdf document

3.4: Thresholding in OpenCV

3.5: Regular Expressions or Regex

3.6: Regex Exercise

3.7: Python class for prescription

3.8: Code Refactoring

3.9: Unit Tests using PyTest

4.1: Write patient details class yourself

4.2: Extract name from a document

4.3: Extract remaining fields from a document

4.4: Write patient details class

4.5: Integrate doc parser classes into an extractor

5.1: FastAPI Basics

5.2: Write FastAPI server for our project

5.3: Integration With Frontend And Deployment