4.11.1 - Big Data, distribution and functional processing

Volume, velocity and variety

Big Data is a relative term for data that exceed the practical capabilities of the usual storage and processing arrangements. A dataset need not cross one universal number of gigabytes to count as “big”. Its size, arrival rate and structure matter in relation to the system trying to use it.

Smartphones can stream locations and events; websites continually receive mouse clicks; video surveillance and networked sensors produce ongoing streams. A transport operator might need to combine structured vehicle coordinates, passengers' text reports and station video. Each source creates a different processing problem.

Do not confuse velocity with volume. A small but fast-arriving stream may demand a millisecond-to-second response. A very large historical archive may be analysed overnight. Both can exceed a system's usual capabilities for different reasons.

Unstructured data and useful patterns

Structured data have a defined organisation, such as a table whose records share named columns. Unstructured data do not arrive in that fixed tabular form: a video or a free-text complaint contains information, but not already as one row of agreed attribute values. Unstructured does not mean meaningless, random or without any internal organisation.

A conventional relational design requires records to fit a defined row-and-column schema. It is well suited to stable records with clear attributes and relationships. For a large collection of raw images and free text, useful features must first be extracted or another representation chosen. Forcing every possible detail into a wide fixed table can create missing values, awkward structures and difficult queries.

This lack of consistent structure often makes analysis harder than sheer storage size. The transport operator must recognise a delay report in natural language and identify a crowd in a video, rather than simply total an existing numeric column.

Machine learning uses patterns learned from data to support tasks such as classification, prediction or feature extraction. A trained model might classify reports by incident type or identify objects in images so that useful summaries can be analysed. Its outputs depend on the training data and model: misleading or unrepresentative data can produce unreliable results. Machine learning helps extract patterns; it does not automatically make every conclusion accurate.

The specification highlights the limits of ordinary relational systems for large, varied, unstructured datasets and scaling across machines. This is a choice of suitable tools, not a claim that no relational system can ever be distributed or store any multimedia reference. A structured booking table may remain relational while its associated large media collection uses another storage/analysis arrangement.

Partitioning work across machines

Distributed processing divides work among multiple networked machines. When a dataset will not fit on one server, both storage and the processing of those data must be arranged across more than one machine. Adding workers helps only when the work can be divided and the results can be combined correctly.

For example, an operator wants the total number of clicks on each page in a large archive. Split the input records into partitions. Workers count page clicks in their own partitions, then matching page totals are combined. A page appearing in two partitions must contribute both counts. The final result is not just whichever worker finishes last.

This organisation brings costs: data or partial results travel across the network, workers may run at different speeds, and a failed worker's work may need to be retried. Dependencies between workers can limit parallelism. A calculation that constantly reads and changes one shared global counter requires coordination and can become a bottleneck.

Reliable processing must also avoid counting a retried partition twice. The system can identify completed partitions and accept each partition's result once. Dividing a job does not remove the need for correct bookkeeping.

Functional features support correct distribution

Functional programming treats computation through functions and their results. The features used in a functional style reduce some sources of interference between workers, making distributed programs easier to reason about. They do not by themselves supply a network scheduler or solve every distributed-system failure.

An immutable data structure is not changed after it is created. A transformation produces a new result rather than modifying a shared input in place. Workers reading the same immutable input cannot overwrite one another's version of that input. Its stable contents also make a repeated calculation easier to check.

Statelessness means that a function does not depend on mutable state left over from earlier calls. A pure function's result depends on its explicit inputs and has no side effects that change hidden shared data. Giving a worker the same input again therefore produces the same result. For independent transformations, moving the call to another machine or changing the order of calls does not alter its meaning. A function reading a changing global counter would not have this property merely because it has a function name.

A higher-order function takes a function as an argument, returns a function, or both. A general operation can apply a supplied transformation to many data items, rather than requiring a different control structure for every transformation. For example, a mapping operation applies the same classification function to each report; a combining operation merges partition summaries using a supplied combination function.

Independent mapped calls can run on different workers because one call does not need another call's changing state. Immutability removes unwanted shared mutation; stateless/pure transformations make results predictable from explicit inputs; higher-order operations express reusable processing patterns that a framework can distribute.

Some combination operations still require care. Counts can be grouped and added in different orders without changing the mathematical total, but joining text or taking successive differences may depend on order. Correct distribution requires a suitable division and combination rule as well as functional features. Learning the native functional representations and implementation belongs in the route's functional-programming lessons; here the assessed demand is to explain why these features help.