Spark Architecture | Codebasics

Go to Home

0% Completed

- Login
- Register

back

- Login
- Register

0% Completed

Contents

1.1: Big Data and Distributed Computing
1.2: Apache Spark and Hadoop
1.3: What is Databricks?
1.4: Databricks Setup

2.1: Basic Operations
2.2: Reading Data from CSV
2.3: Handle Missing Values
2.4: SQL in pyspark
2.5: Joins

3.1: Reading Spark Plans with explain()
3.2: Spark Architecture
3.3: Transformations vs Actions and Lazy Evaluation
3.4: Narrows vs Wide Transformations
3.5: Partitions & Parallelism: Repartition
3.6: Partitions & Parallelism: Coalesce
3.7: Lokis advice to Peter on using AI tools
3.8: Catalyst Optimizer, Predicate Pushdown, Column Pruning
3.9: Joins at Scale: Broadcast, Shuffle Hash and Hints
3.10: Data Skew and Mitigation Techniques
3.11: Adaptive Query Execution (AQE)
3.12: RDD, Dataframes and Datasets
3.13: Managed vs External Tables
3.14: Unity Catalog: Governance & Access Control
3.15: Understanding ACID
3.16: Time Travel Explained
3.17: What is Delta Lake and Delta Format

4.1: Final Quiz

Instructor

Dhaval Patel

Dhaval Patel

Data Entrepreneur | YouTuber | Ex - Bloomberg, Nvidia

I have 17 years of experience in Programming and Data Science working for big tech companies like NVIDIA and Bloomberg. I also run a famous YouTube channel called Codebasics where I pursue my passion for teaching. Codebasics is one of the top channels on YouTube when it comes to data science, machine learning, data structures, etc. I firmly believe that “Anyone can code” and I use analogies, simple explanations, and step-by-step storytelling to explain difficult concepts in such a way that even a high school student can understand them easily.

Spark Architecture

Prev

Lecture Content Locked

Get Notified for Next Cohort →

Login

Log in with Google

Log in with Linkedin

E-Mail Address

Enter your Password

Forgot Password?

Don't have an account? Register Now!

By logging in, you agree to
our Terms and Conditions and Privacy Policy.

Register

Register with Google

Register with Linkedin

Full Name *

Full Name is required.

Enter Your Mobile Number

No results found

Afghanistan +93 Åland Islands +358 Albania +355 Algeria +213 American Samoa +1 Andorra +376 Angola +244 Anguilla +1 Antarctica +672 Antigua and Barbuda +1 Argentina +54 Armenia +374 Aruba +297 Australia +61 Austria +43 Azerbaijan +994 Bahamas +1 Bahrain +973 Bangladesh +880 Barbados +1 Belarus +375 Belgium +32 Belize +501 Benin +229 Bermuda +1 Bhutan +975 Bolivia, Plurinational State of +591 Bonaire, Sint Eustatius and Saba +599 Bosnia and Herzegovina +387 Botswana +267 Bouvet Island +47 Brazil +55 British Indian Ocean Territory +246 Brunei Darussalam +673 Bulgaria +359 Burkina Faso +226 Burundi +257 Cambodia +855 Cameroon +237 Canada +1 Cape Verde +238 Cayman Islands +1 Central African Republic +236 Chad +235 Chile +56 China +86 Christmas Island +61 Cocos (Keeling) Islands +61 Colombia +57 Comoros +269 Congo +242 Congo, the Democratic Republic of the +243 Cook Islands +682 Costa Rica +506 Côte d'Ivoire +225 Croatia +385 Cuba +53 Curaçao +599 Cyprus +357 Czech Republic +420 Denmark +45 Djibouti +253 Dominica +1 Dominican Republic +1 Ecuador +593 Egypt +20 El Salvador +503 Equatorial Guinea +240 Eritrea +291 Estonia +372 Ethiopia +251 Falkland Islands (Malvinas) +500 Faroe Islands +298 Fiji +679 Finland +358 France +33 French Guiana +594 French Polynesia +689 French Southern Territories +33 Gabon +241 Gambia +220 Georgia +995 Germany +49 Ghana +233 Gibraltar +350 Greece +30 Greenland +299 Grenada +1 Guadeloupe +590 Guam +1 Guatemala +502 Guernsey +44 Guinea +224 Guinea-Bissau +245 Guyana +592 Haiti +509 Heard Island and McDonald Islands +61 Holy See (Vatican City State) +39 Honduras +504 Hong Kong +852 Hungary +36 Iceland +354 India +91 Indonesia +62 Iran, Islamic Republic of +98 Iraq +964 Ireland +353 Isle of Man +44 Israel +972 Italy +39 Jamaica +1 Japan +81 Jersey +44 Jordan +962 Kazakhstan +7 Kenya +254 Kiribati +686 Korea, Democratic People's Republic of +850 Korea, Republic of +82 Kuwait +965 Kyrgyzstan +996 Lao People's Democratic Republic +856 Latvia +371 Lebanon +961 Lesotho +266 Liberia +231 Libya +218 Liechtenstein +423 Lithuania +370 Luxembourg +352 Macao +853 Macedonia, the former Yugoslav Republic of +389 Madagascar +261 Malawi +265 Malaysia +60 Maldives +960 Mali +223 Malta +356 Marshall Islands +692 Martinique +596 Mauritania +222 Mauritius +230 Mayotte +262 Mexico +52 Micronesia, Federated States of +691 Moldova, Republic of +373 Monaco +377 Mongolia +976 Montenegro +382 Montserrat +1 Morocco +212 Mozambique +258 Myanmar +95 Namibia +264 Nauru +674 Nepal +977 Netherlands +31 New Caledonia +687 New Zealand +64 Nicaragua +505 Niger +227 Nigeria +234 Niue +683 Norfolk Island +672 Northern Mariana Islands +1 Norway +47 Oman +968 Pakistan +92 Palau +680 Palestinian Territory, Occupied +970 Panama +507 Papua New Guinea +675 Paraguay +595 Peru +51 Philippines +63 Pitcairn +649 Poland +48 Portugal +351 Puerto Rico +1 Qatar +974 Réunion +262 Romania +40 Russian Federation +7 Rwanda +250 Saint Barthélemy +590 Saint Helena, Ascension and Tristan da Cunha +290 Saint Kitts and Nevis +1 Saint Lucia +1 Saint Martin (French part) +590 Saint Pierre and Miquelon +508 Saint Vincent and the Grenadines +1 Samoa +685 San Marino +378 Sao Tome and Principe +239 Saudi Arabia +966 Senegal +221 Serbia +381 Seychelles +248 Sierra Leone +232 Singapore +65 Sint Maarten (Dutch part) +721 Slovakia +421 Slovenia +386 Solomon Islands +677 Somalia +252 South Africa +27 South Georgia and the South Sandwich Islands +44 South Sudan +211 Spain +34 Sri Lanka +94 Sudan +249 Suriname +597 Svalbard and Jan Mayen +47 Swaziland +268 Sweden +46 Switzerland +41 Syrian Arab Republic +963 Taiwan, Province of China +886 Tajikistan +992 Tanzania, United Republic of +255 Thailand +66 Timor-Leste +670 Togo +228 Tokelau +690 Tonga +676 Trinidad and Tobago +1 Tunisia +216 Turkey +90 Turkmenistan +993 Turks and Caicos Islands +1 Tuvalu +688 Uganda +256 Ukraine +380 United Arab Emirates +971 United Kingdom +44 United States +1 United States Minor Outlying Islands +1 Uruguay +598 Uzbekistan +998 Vanuatu +678 Venezuela, Bolivarian Republic of +58 Viet Nam +84 Virgin Islands, British +1 Virgin Islands, U.S. +1 Wallis and Futuna +681 Western Sahara +212 Yemen +967 Zambia +260 Zimbabwe +263

Enter Your Email Address *

Valid email is required.

Enter Password *

Password must be at least 6 characters.

Retype Password *

By continuing, you are agreeing to Codebasics' Terms and Conditions and Privacy Policy.

You must agree to the Terms and Conditions & Privacy Policy.

Have an account? Login here

OTP Verification

Please enter the OTP sent to your email ID.

Enter OTP

The code entered is invalid.

Did not receive the OTP? Please resend