Initially there was a lot of confusion about the difference between the fields of Inferential Statistics and Data Mining and researchers around the world immediately started working on producing tons of literature to address this issue. Though there is no single universally accepted definition for either of them, many researchers have defined these fields through their individual application perspectives. From what I’ve gathered and known about these fields, this is how I draw a line between them.

Difference between Data mining & Inferential Statistics:

• Data Mining is a field where you discover hidden patterns from already existing large data sets. These hidden patterns discovered are later used for analysis and decision making scenarios in the area of concern. The process of using the discovered hidden patterns is also called as ‘Knowledge Discovery’. Inferential Statistics is the field where you prove or refute a pre-conceived hypothesis (or a null hypothesis) by performing classical statistical methods on a sample of a given population size.
• Data mining starts at an already existing database (usually large datasets) and Inferential Statistics generates its own database using sampling methods on data set.
• Data mining methods employed like classification, clustering, etc., scan the entire dataset in search of hidden patterns while classical statistical methods are run over only a small section of the dataset (the sample).

The above point also infers that Data mining methods are more computer intensive as they have to run through large data sets and hence should be used only when really needed.

So when do you use data mining over inferential statistics?  Well the answer is simple. If you don’t know what you are looking but want to make the best sense of the data you have then use data mining. If you know what you are looking for and want to back it up with proof by checking the data you have you should use inferential statistics.

Click Forensics, a leading solutions provider for improving traffic quality for online ad campaigns, reported that though there is an overall dip in click fraud rate in Q2 2009 over Q1, click fraud traffic from sophisticated sources and scripted programs rose again in Q2 2009. This is evident from the rise of Publisher Collusion Fraud where publishers are employing sophisticated IP rotation methods or botnets  in oder to inflate their commissions.

There is a huge incentive for the Ad Networks and Search Engines to look at this as a serious concern because any hint of discomfort advertiser will lead the advertiser and/or publisher to defect from the system which will eventually lead to revenue losses.  Infact, Click Forensics President, Tom Cuthbert states that the increased diligence from the Ad Networks and Search Engines is the main reason for the click fraud rates to steadily decrease over the past few quarters. Google for example has set up a resource center to raise awareness about this issue. All the docs, presos and videos at this link are excellent reads on the topic.

So, from a technology standpoint, Search Engines and Ad Networks are doing their bit by developing tools and techniques in order to mitigate the effects of click fraud. What is lacking though are studies that look at user, publisher and advertiser behavior towards click fraud. Do they know what click fraud is? How do they validate erratic behavior on their campaigns? Is there a certain level of awareness to help them make informed decisions ?etc., There is enough proof to tell you that often advertisers complain that a seemingly erratic behavior in their campaigns is click fraud while the real reasons for such a behavior might be something completely different. Conducting research hence is important for the Search Engine  and Ad Networks to develop a strategy around increasing awareness around this issue that actually is effective.