Barbie Dolls and Candies

Welcome to the world of statistical association rules,
What insight might you gain here?
Well, let’s set you up first with an anecdote here,
Wal-Mart once found customers who buy Barbie dolls,
Also had a 60% chance of buying one of three types of candy bars.

From the computational depths of data mining,
Come the sensational techniques of association rules,
Promising to hunt down every record in your large set of data,
And hand you down “If X then Y” relationships, if they ever existed in your data,
Where X and Y can be single items or conjunctions of items standing in an association,
For example, if (Car=Porsche and Gender=Male and Age<20) then (Risk=High and Insurance=High).

Let’s turn to Market Basket Analysis, a typical example of association rule mining,
Large numbers of transaction records, data collected using bar-code scanners in supermarkets, go home to reside in a market basket database,
Each record lists all items bought by a customer on a single purchase transaction,
Managers get interested to know if certain groups of items are consistently purchased together,
Association rule mining is the hero of the day giving them that valuable insight,
Insights which assist them in adjusting store layouts, cross-selling, promotions, catalog design and identifying customer segments based on those buying patterns.

So, association rules provide information in the form of "if-then" probabilistic statements,
The “if” part is the antecedent and the “then” part, of course, is the antecedent,
The antecedent and consequent are sets of items that are disjoint (do not have any items in common),
Probabilistic being its very attire, every association rule has three numbers that express the degree of uncertainty about the rule,
The first number is called the Support for the rule; the support being simply the number of transactions that include all items in the antecedent and consequent parts of the rule,
The second number is known as the confidence of the rule; Confidence being the ratio of the number of transactions that include all items in the consequent as well as the antecedent (namely, the support) to the number of transactions that include all items in the antecedent,
And the third number in the family is called Lift; Lift is the ratio of confidence to expected confidence, this new term expected confidence introduced here is nothing but number of transactions of antecedent divided by total number of transactions.

Let’s take the aid of an example, shall we?
A supermarket database has 100,000 point-of-sale transactions, out of which 2,000 include both items A and B and 800 of these include item C, total sales of item C are 5,000,
The association rule "If A and B are purchased then C is purchased on same trip" has a support of 0.8 % (=800/100,000) and a confidence of 40% (=800/2,000),
Expected Confidence is 5 % (=5,000/1,00,000) and Lift is Confidence/Expected Confidence = 40%/5% = 8,
Here’s one way to help with interpretation of these numbers,
Support is the probability that a randomly selected transaction from the database will contain all items in the antecedent; the higher the value the better is association,
Confidence is the conditional probability that a randomly selected transaction will include all the items in the consequent given that the transaction includes all the items in the antecedent; again, higher the value the stronger is association,
And Lift tells us how much better a rule is at predicting the result than just assuming the result in the first place; values higher than 1 assert that the rule is better at predicting the result than guessing.

Back in 1998, when asked what does Wal-Mart do with information like that? (Barbie and candies)
'I don't have a clue,' quipped famously Wal-Mart's chief of merchandising, Lee Scott,
While Wal-Mart may have deftly evaded disclosing on a strategic initiative,
The next time you buy something on Amazon.Com and see yet another recommendation -
"Customers who ordered product A also ordered product B or C", you will know its association rules working under the hood.