Top 10 Algorithms in Data Mining

In the tradition of year-end lists, the Knowledge and Information Systems journal has published a list of the top 10 algorithms in data mining, as compiled based on a survey given at the IEEE International Conference on Data Mining. Here they are (along with referenced to relevant wikipedia articles):

[1] C4.5 et al.
[2] k-means
[3] Support Vector Machines
[4] Apriori
[5] EM
[6] PageRank
[7] AdaBoost
[8] kNN
[9] Naive Bayes
[10] CART

Often times, these sorts of lists can seem rather contrived, but I pleased to see that Wu and colleagues have published a well-written--and thoughtfully referenced--survey of the Machine Learning and Data Mining literature, highlighted by pseudocode and reference equations. A great read!

On Statistical Modeling: The Two Cultures

Yesterday I linked to a pdf copy of Leo Breiman's Statistical Modeling: The Two Cultures, an article I've recently read. For those not familiar, Breiman was, in my view, a central figure in the early days of modern Machine Learning (i.e., 1980+). His work on Classification and Regression Trees, Bagging, and Random Forests were all foundational to the field of Statistical Machine Learning. I think this fact partially contributed to how much I enjoyed reading the aforementioned article--Breiman writes as someone with a deep understanding of his material, the sort that can only be obtained by living its history. He casts the two camps of statistical analysis in terms of a "black box", where the data of interest are generated by the process

Anaylsis in the data modeling community, he argues, begins "...with assuming a stochastic data model for the inside of the black box" (199):

The algorithmic modeling community, however, "...considers the inside of the box complex and unknown. Their approach is to find a function...--an algorithm that operates on x to predict the responses of y" (199):

The overarching message of Breiman's article is an important one that is still relevant today, eight years later. An excerpt summarizes it quite well:

My biostatistician friends tell me, "Doctors can interpret logistic regression." There is no way they can interpret a black box containing fifty trees hooked together. In a choice between accuracy and interpretability, they'll go for interpretability. Framing the question as the choice between accuracy and interpretability is an incorrect interpretation of what the goal of a statistical analysis is. The point of a model is to get useful information about the relation between the response and predictor variables (209-210).

He continues, by pointing out that a model doesn't necessarily have to be simple in order for it to provide useful insight. This point struck me, as, on the surface it seems to defy a central principle of model selection (e.g., MDL, AIC, BIC). Granted, Breiman is referring to a more general type of model selection--data modeling v. algorithmic modeling--but the contrast is still interesting. More importantly, Breiman was exactly right in pointing out the strange statistical reasoning the pervades many biomedical fields--do we want answers to our questions, or do we want to the layperson to understand every step involved in arriving at the answers?