- big data, Hadoop, Spark
- New
- Rating: 4.7 out of 54.7 (5 ratings)
- 461 students
- 1hr 48min of on-demand video
- Created by Pan Dhoni
English
What you’ll learn
- Spark Basics: Students will learn the fundamentals of Spark, equipping them to become proficient Data Engineers.
- Developers will be able to write optimized code, enhancing their engineering capabilities.
- This course is designed to support individuals aiming to pursue careers in Data Engineering, Analysis, and Data Science.
- Additionally, it will prepare students to excel in job interviews related to these fields.
Requirements
- Basic computer knowledge, one programming language
Description
Course Title: Big Data and Apache Spark Essentials
Course Overview:
Dive deep into the world of Big Data with our intensive course designed to equip you with the fundamental knowledge and practical skills needed to harness the power of big data technologies. This course covers the core components of Big Data processing using Hadoop and Spark, offering insights into their architectures, functionalities, and optimization techniques. With a blend of theoretical knowledge and hands-on exercises, learners will emerge ready to tackle big data challenges in real-world scenarios.
Course Content:
Section 1: Understanding Hadoop and HDFS
Lecture 1: About Big Data: Introduction to big data, its characteristics, and why it matters.
Lecture 2: About Hadoop: Overview of Hadoop, its ecosystem, and components.
Lecture 3: HDFS Read: Understanding how HDFS supports read operations, including its process and efficiency.
Lecture 4: HDFS Write: Exploring the write functionality within HDFS and its importance for data storage.
Lecture 5: HDFS Data Block: Insights into how data is stored in blocks within HDFS and the benefits of this approach.
Lecture 6: HDFS Data Replication: Delving into the replication process within HDFS for data safety and availability.
Lecture 7: HDFS High Availability: Strategies for ensuring high availability in HDFS and mitigating the risk of data loss.
Lecture 8: HDFS Rack Awareness: Understanding rack awareness and its role in improving data reliability and access speed.
Section 2: Spark Architecture and Benefits
Lecture 9: Spark Architecture: Detailed exploration of Spark’s architecture and how it enables fast, in memory data processing.
Lecture 10: Spark Advantages: Discussing the key benefits of using Spark over other big data technologies.
Lecture 11: Spark Limitations: A realistic look at Spark’s limitations and how to navigate them.
Lecture 12: SparkSession & SparkContext: Introduction to SparkSession and SparkContext as the foundational elements of working with Spark.
Lecture 13: Spark Unified Solution: Overview of Spark as a unified solution for big data processing, including batch and stream processing.
Section 3: Spark RDDs, Lineage, and DAG
Lecture 14: Spark RDDs: Deep dive into Resilient Distributed Datasets (RDDs), the fundamental data structure of Spark.
Lecture 15: Lineage: Understanding the lineage.
Lecture 16: Spark DAG: Exploration of Directed Acyclic Graph (DAG) and its role in optimizing Spark jobs.
Section 4: Spark Optimization
Lecture 17: SQL Optimization: Techniques for optimizing SQL queries in Spark for improved performance.
Lecture 18: Adaptive Query Plan: Understanding adaptive query planning for optimizing Spark execution plans dynamically.
Target Audience:
This course is ideal for data professionals, software engineers, and IT professionals who wish to gain a solid understanding of big data technologies, especially Hadoop and Spark. Prior knowledge of programming and basic understanding of databases will be beneficial.
Learning Outcomes:
By the end of this course, participants will be able to:
Understand the key concepts and components of Hadoop and Spark.
Perform data processing tasks using HDFS.
Leverage Spark for efficient big data analysis and processing.
Optimize data queries and processes using Spark’s advanced features.
Implement Spark solutions for real-world data challenges.
Enroll in this course to navigate the vast landscape of big data technologies and to acquire the skills necessary to become a proficient big data practitioner.
This course description is structured to provide a clear pathway through the complexities of big data technologies, emphasizing both theoretical background and practical skills acquisition.
Who this course is for:
- Beginners with basic programming knowledge will benefit.
- This course is ideal for data professionals, software engineers, and IT professionals who wish to gain a solid understanding of big data technologies, especially Hadoop and Spark. Prior knowledge of programming and basic understanding of databases will be beneficial.
Show less
Course content
4 sections • 18 lectures • 1h 48m total lengthCollapse all sections
Introduction8 lectures • 38min
- About Big Data05:55
- About Hadoop05:04
- HDFS Read01:45
- HDFS Write06:00
- HDFS Data Block04:00
- HDFS- Data Replication01:59
- HDFS- High Availability07:52
- HDFS- Rack Awareness05:30
Spark Architecture5 lectures • 34min
- Spark Architecture12:43
- Spark Advantages04:42
- Spark Limitations04:29
- SparkSession & SparkContext09:16
- Spark Unified Solution03:05
Spark RDD’s, Lineage and DAG3 lectures • 18min
- Spark RDD’s08:42
- Lineage06:30
- Spark DAG03:17
Spark Optimization2 lectures • 17min
- SQL Optimization09:48
- Adaptive Query Plan07:36
