Ch 18: Big Data Processing | Exam Fusion Prep

What is Big Data?

Big Data is a massive and complex data set that is difficult to process using traditional data-management tools such as Excel or RDBMS.
The source emphasizes that such data is generated very rapidly and comes in many different forms.

English Summary

Big Data is a vast and complex data set that is difficult to process with traditional data-management tools such as Excel or an RDBMS. It is generated very rapidly and occurs in many forms.

Source Examples

Photos, videos and comments uploaded to Facebook every day
Purchases made on online shopping sites
Every transaction made through an ATM

What is Big Data Processing?

Big Data Processing means processing huge data so that important information can be extracted from it and used in decision making.
Importance: It helps organizations extract valuable insights from data and make informed decisions.

Huge Data→Processing→Useful Information→Decision Making

Key Characteristics of Big Data

Source-derived visual of the five characteristics listed in the PDF.

Characteristic	Source Meaning
Volume	The enormous amount of data generated every day
Velocity	The speed of data — how quickly data is generated and processed
Variety	Different types of data — structured, semi-structured and unstructured
Veracity	The quality and reliability of data
Value	The ability to extract useful information from data

Main Processing Steps

Data Collection: Data is collected from sources such as social media, websites, IoT devices and machines.
Data Storage: Data is stored using technologies such as HDFS and Amazon S3.
Data Processing: Large data is divided into smaller parts and processed using MapReduce, Spark, Flink etc.
Data Analysis: Processed data is visualized and decisions are made on that basis.

Original source visual embedded from PDF page 198: Steps of Big Data Processing.

Popular Tools

Tool	Use as stated in the source table
Hadoop	Distributed storage (HDFS) and processing, traditionally using batch MapReduce; streaming requires suitable additional components.
Spark	Fast and in-memory processing.
Flink	Streaming data processing.
Kafka	Distributed event-streaming platform for publishing, storing and consuming event streams.
Hive	SQL-style querying and data warehousing over distributed storage.
Pig	Scripting language on Hadoop.

Revision Note

Kafka is for event streaming; Hive provides SQL-style analytics. They are different tools and their functions must not be swapped.

Types of Big Data Processing

Type	Description in Source
Batch Processing	Collect data and process it together at once; example: Hadoop.
Real-Time Processing	Process data as soon as it arrives; examples: Apache Storm, Spark Streaming.
Stream Processing	Process continuously arriving data; example: Kafka + Spark.

Original source visual embedded from PDF page 198: Batch Processing vs Stream Processing.

Big Data Types Covered in the Source

1. Structured Data

Organized data stored in tables or databases; easy to process and access.

Excel sheets
SQL databases
Bank transaction records
Students' roll-number list

Tools: MySQL, Oracle, MS SQL Server.

2. Semi-Structured Data

Partly structured data without a fixed schema, though some portions remain organized.

XML files
JSON data
Email
JSON event records; post text/media can be unstructured

Tools: Apache Hive, NoSQL database (MongoDB).

Source Scope

Unstructured data has no predefined tabular model, e.g. free text, images, audio and video. Email headers are structured while the body can be unstructured.

Major Big Data Processing Frameworks

Framework	Features / Components	Uses / Advantages
Hadoop	HDFS: distributed system for storing data. MapReduce: distributed programming model for data processing.	Batch processing, ETL jobs, big-data analysis.
Apache Spark	In-memory processing, fast data processing.	Real-time analytics, machine learning, data mining. Source says it is faster than Hadoop MapReduce and easy to use.
Apache Flink	Distributed stream processing, high performance, low latency.	Event-driven applications and data pipelines.

Big Data Storage

Data Lake

Centralized storage system.
Stores raw data in its original format.
Advantages: flexibility, scalability, low cost.
Examples in source: Amazon S3, HDFS.

NoSQL Database

Non-relational database models, including document, key-value, wide-column and graph. They can hold structured as well as semi-structured data.
Examples: MongoDB, Cassandra, HBase.
Advantages: high scalability, flexibility.

Data Transformation & ETL

Source-derived ETL/ELT visual.

Extract: Collect data from various sources.
Transform: Clean, transform and structure the data.
Load: Load data into a data warehouse or database.
ELT (Extract, Load, Transform): Raw data is loaded first, then processed.

Real-Time Data Processing Frameworks

Apache Kafka: Distributed streaming platform for real-time data pipelines and stream-processing applications.
Apache Storm: Real-time computation system that processes unbounded data streams.
Apache Flink: Stream and batch processing engine providing high throughput and low-latency processing.

Challenges in Big Data Processing

Challenge	Source Explanation
Data Volume	Difficulty in processing and managing massive datasets.
Latency	Ensuring minimum latency for real-time processing.
Data Quality	Ensuring accuracy, consistency and reliability of data.
Scalability	Scaling the system to accommodate increasing data size.
Complexity	Managing diverse data types and processing frameworks.

Future of Big Data Processing

AI & Machine Learning

Automating data analysis and developing models for better decision-making.

Edge Computing

Bringing data processing closer to the data source to reduce latency.

Quantum Computing

A potential future technology for accelerating data processing.

Conclusion from Source

Big Data Processing is important because efficient techniques help manage and analyze large datasets, enabling organizations to make informed, data-driven decisions.

Quick Recall

One-Line Revision

Big Data: Massive + complex + rapidly generated + many forms.
5Vs: Volume, Velocity, Variety, Veracity, Value.
Main Steps: Collection → Storage → Processing → Analysis.
Storage examples: HDFS, Amazon S3.
Processing technologies: MapReduce, Spark, Flink.
Processing types: Batch, Real-Time, Stream.
Structured: tables/databases; tools include MySQL, Oracle, MS SQL Server.
Semi-Structured: XML, JSON and structured message metadata; free text/media are unstructured.
Hadoop: HDFS + MapReduce.
Spark: in-memory, fast processing.
Flink: distributed stream processing, high performance, low latency.
Data Lake: stores raw data in original format.
ETL: Extract → Transform → Load.
ELT: Extract → Load → Transform.
Real-time frameworks: Kafka, Storm, Flink.
Challenges: Volume, Latency, Data Quality, Scalability, Complexity.
Future: AI/ML, Edge Computing, Quantum Computing.

Exam Traps

Volume ≠ Velocity: Volume = amount; Velocity = speed.
ETL ≠ ELT: ETL transforms before loading; source describes ELT as raw data loaded first, then processed.
Batch ≠ Real-Time: Batch handles collected data together; Real-Time processes data as it arrives.