What you’ll learn
Learn designing an end-to-end Real-time Streaming pipeline for Big Data using latest technologies.
Understand the different components in Big Data streaming pipeline.
Use Kafka as the connecting tool between ETL components in the real-time streaming pipeline.
Use Apache Flink, Spark Streaming and Kafka Streams to perform different transformations and aggregations.
Use Druid and Pinot as OLAP technologies in the streaming pipeline.
Use Superset to visualize the real-time incoming data stream to explore and visualize the transformed data.
Hands-on Practicals helping you build all the components and forming a complete end-to-end pipeline.
Learn multiple technologies used in Real-time Streaming pipelines, and you can use the one that better suits your use-case.
An exposure to Big Data world will help you better appreciate Real-time Streaming pipelines, but is completely optional.
Basic knowledge of Java and Scala will be helpful, but not mandatory.
Getting real-time insights from huge volumes of data is very important for a majority of companies today.
Big data Real-time streaming is used by some of the biggest companies in the world like e-commerce companies, Video streaming companies, Banks, Ride-hailing companies, etc.
Knowing about the concepts of realtime streaming and the various realtime streaming technologies will be a great addition to your skillset and will enable you to build some of the most cutting-edge solutions that exist today.
We have created this Hands-On Course so that you get a good understanding about how realtime streaming systems can be built
This course will ensure that you get a hands-on experience with Apache Kafka, Apache Flink, Spark Streaming, Kafka Streams, Apache Pinot, Apache Druid, and Apache Superset.
This course covers the following topics
- An Introduction to Kafka with hands-on Kafka setup
- Understanding basic transformations and aggregations which can be done in a real time system
- Learn how transformations and aggregations can be done using Apache Flink with hands-on coding exercises
- Learn how transformations and aggregations can be done using Spark streaming with hands-on coding exercises
- Learn how Kafka streams can be used to perform transformations and aggregations with hands-on coding exercises
- Ingest data into Apache Pinot which is an OLAP technology
- Ingest data into Apache Druid which is also an OLAP technology
- Using Apache Superset to create some insightful dashboards
If you are interested in learning how all these technologies can be connected together to build an end to end real-time streaming system, then this course is for you.
Who this course is for:
- Students who want to learn building real-time streaming pipelines from SCRATCH to its Live Project Implementation.
- Students who want to learn latest technologies that are used in Big Data Engineering.
- Developers who want to learn different well-known tools to build streaming pipelines.
- Students who want to pursue and grow career in Data Engineering.
I am software architect by profession. I have worked primarily in building e-commerce technologies and also designed systems in Big Data Analytics domain. I am an active technical interviewer, and have taken 500+ coding and software design interviews. I love to share technical knowledge and help budding engineers with their career growth.
Senior Software Engineer, Part Time Tech Blogger
I am a Software Engineer, a curious techie, and a part-time tech blogger.
I have a good amount of experience in dealing with highly scalable distributed applications in production.
The applications I work on, deal with millions of records coming in every minute and also respond to millions of HTTP requests every minute.
I Love to teach and maintain a personal blog where I write articles on a variety of technical topics.
As an Instructor in Udemy I want to ensure that I am able to teach the concepts that I know in an easy and understandable manner.