2.2.2 - Data mining and heuristics

Mine relationships rather than merely collect records

Data mining analyses large quantities of data to discover useful patterns, relationships, trends or anomalies that may not be obvious. Recording a purchase is data collection. Comparing many purchases to discover that particular items are often bought together is mining the collected data.

A repair service records machine age, fault category, prior repairs and whether another failure occurs within a month. It could search for combinations associated with repeated failures, then use the findings to prioritise preventive inspections. A sudden change in failure rate might be an anomaly worth investigation. Finding a pattern supports a decision; it does not establish a causal mechanism.

Use a small sample to see the reasoning, while remembering that real mining operates across much larger data. Of 20 inspected machines, 8 are older than five years and 5 of these fail again. Of the 12 younger machines, 3 fail again. The repeated-failure rates are 5/8 = 62.5% and 3/12 = 25%. The older group has a higher rate in this sample. It might also contain machines used for longer daily shifts, so age alone has not been proved to cause failure.

A prediction is uncertain. Even in the older group, 3 of 8 do not fail again. Treat “older machines always fail” as a false conclusion. A model may estimate risk, not guarantee an individual outcome.

Interrogate data and test the discovered pattern

A mining process must make data comparable before searching for a relationship. Resolve inconsistent units and categories, identify duplicate records and decide how to treat missing values. An absent age is not an age of zero. Removing every incomplete record may exclude one group disproportionately, so the cleaning decision can itself introduce bias.

Interrogating the data means asking and computing focused questions: select relevant records, group them, count events, compare rates and inspect exceptions. For the machine sample, raw failure counts 5 and 3 would be misleading without denominators 8 and 12. Counting five failures among 800 older machines would tell a very different story.

For a recommendation service, one possible process is to group transactions by customer, count pairs of items that occur together, then compare each pair's occurrence with the items' individual frequencies. Frequently bought items naturally appear in many pairs; frequency alone may not establish a special relationship. Searching many candidate patterns also makes accidental matches more likely.

Check a discovered rule on data not used to choose it. Separate later or held-out records can reveal that a pattern fitted chance details in the original set. Representative data matters: a pattern from one factory may not transfer to machines with different loads elsewhere. Useful complexity questions include how many records and candidate combinations must be examined, how much storage is needed, and whether a change in the data requires updating the model.

Data mining can inform inspection priorities without replacing every human decision. The service can compare the costs of missed failures and unnecessary inspections, investigate causal explanations and monitor later performance. A correlation, a small sample and an unrepresentative dataset give different reasons for caution; name the actual reason in the case.

Use an estimate to guide a decision

A heuristic is a practical rule or estimate used to guide a decision or search. It can reduce the choices considered or rank promising choices quickly. Its usefulness depends on how well it matches the problem; a general heuristic does not guarantee an optimal solution.

A courier on a rectangular street grid might choose the next stop with the smallest Manhattan distance from the current position. For (x,y) to (a,b), the estimate is abs(x-a) + abs(y-b): horizontal plus vertical distance. From (2,1) to (5,3) this is 3+2 = 5 grid units. This fits movement along grid streets, but ignores blocked roads, traffic and waiting time.

The choice is locally plausible: a nearby stop may require little immediate travel. It is different from evaluating every complete ordering of stops. In a shortest-path search such as A*, a heuristic estimate of remaining cost can be combined with known cost so far, commonly written $f = g + h$, to rank states for exploration. Full A* route construction is a separate algorithm; the estimate's role is to direct the search toward promising states. Specific guarantees need additional conditions, not the word “heuristic”.

A rule can be transparent and fast yet poor in a particular setting. Straight-line distance may underestimate travel around a river; an estimate that ignores a closing-time deadline may send the courier to an unusable stop. Choose the rule because of the represented constraints, and test where those assumptions fail.