Given a dataset, how do you choose a machine learning algorithm

Question

I have a dataset (observations of variables over a period of x years for 1000 samples) Different samples have different dates of observations. Observations are recorded using dates

Data looks like this:

SampleID Date Var
S1    Date1 Var1 
S1    Date1 Var20 
S1    Date2 Var32 
S1    Date2 Var11 
S1    Date2 Var1001
S2    Date1 Var3411
...

Given such a dataset, I am looking for an ideal learning algorithm that which can be used to predict next Var for a new sample. What techniques do you use to select a machine learning algorithm for a given dataset ? Is there any literature/protocols that I can follow ?

EDIT: I am looking for opinion on whether there is any best practice in selecting machine learning frameworks for a given data type. For example, IMHO if someone need to develop a multi-classifier SVM is a good choice. For binary classifier: both SVM and RF could be considered, similar way is there any review/best practice/literature on adapting machine learning frameworks for a given dataset ?

Thanks.

Answer 1

Here's a quick answer sketch to the general question asked in the title "Given a dataset, how do you choose a machine learning algorithm?"

Input: Are the inputs vectors or general objects? In the latter case, k-NN or kernel-based approaches can be used.
Output: Is the output a real number (regression), a binary variable (binary classification), a categorical variable (multiclass classification), a set of categorical variables (multi-label prediction), a sequence (tagging) or a general object (structured prediction)? For the latter two, CRFs, structured SVMs and the structured Perceptron can be used.
Dimensionality and Sparsity: For high-dimensional sparse data, Pegasos or the passive-aggressive Perceptron are state-of-the-art.
Linear separability: For highly non-linear datasets, ANNs, Decision Trees or kernel SVMs are probably the way to go.
Scalability (how many training instances do you have): stochastic and randomized algorithms.
Prediction speed (how many predictions per second do you need to do): linear models penalized with a sparsity-inducing norm!
Model size: same as above.
Memory requirement (e.g. in mobile phones): online algorithms.

Answer 2

1

It will depend on what your data looks like. Have you tried plotting it? Can you assume independence in the data?

If you have some correlation of a variable to a sample, you might try something very simple like linear regression. If you are looking for the probability of VarX occurring given a random sample chosen, you might want to look into something like naive bayes.

answered Sep 26 '11 at 13:35

nop
106●2●3●10

I agree, it all depends on my data. I looked at distributions too, but here am interested in finding an ideal ML framework that can be used to train a model using 3/4 of my 1000 samples and utilize for predicting next var for a new sample.

(Sep 26 '11 at 15:06) Khader Shameer

TO use Naive Bayes, he would need the Var label to be dependent on some other kind of features, rather than just dates (it would be like classifying spam based on the date an email was sent)

(Sep 27 '11 at 01:05) Leon Palafox

Thanks, but deriving features is a bottle neck here. So I am thinking about something like association rule mining or recommendation engine.

(Sep 27 '11 at 14:33) Khader Shameer

@Leon: Yes, if it's just a list of dates then I agree. If you are looking for patterns in when things arrive (e.g. bimonthly) then it might be useful.

@Khader: If you are going to try to do any analysis involving correlations than you should be able to eyeball the expected correlation of the data by the covariance matrices (i.e. the shape of the ellipse you draw around the data points) as well as some idea about the seperability of the data.

(Sep 28 '11 at 11:04) nop

Answer 3

If you are primarily interested in the co-occurence of the icd-9 codes in the same sample over long periods of time, then you can really adapt many of the techniques for document modelling and IR. Basically you can treat the samples as documents & the codes as words. You can take the bag of words view, ignoring the temporal structure with methods LSA, LDA & other spectral methods, various kinds of clustering or you can try and capture the temporal structure with things like hidden markov models. Pretty much anything that can be applied to documents should be applicable here. However, I suspect you will have fewer observations per user than a typical document has words, in which case you will need to make similar adjustments to your methods that people make when analysing short texts like tweets, web queries & various kinds of online comments.

In a similar way methods from collaborative filtering (recommendation systems) can be adapted by viewing samples as users and the codes as items. Actually these methods tend to be similar to the text modelling methods since the abstract structure of the data is the same.

[EDIT: Another thing I forgot to mention is that you could train a Bayes net or a DBN over your training samples this would allow you to take a test sample and activate the codes from it this would in turn activate other codes by association]

As to the more general question of choosing an appropriate learning method for your data, you really need more information than regression vs binary classification vs multiclass. Some methods like random forests are competitive across a very wide range of problems, but to make intelligent choices about more specialised methods like boosting, kernels & svms, GLMs & GAMs, regularization techniques, you really want to look at things like:

size of training data
number, sparsity & general distribution shape of your features
feature & label noise
correlated & irrelevant features
potential nonlinearities & feature interactions

Answer 4

1

unless you have some knowledge of your problem, i doubt you'll be able to do better than an empirical selection. an interesting way to try to approach this would be to "meta optimize"- look at many problems of all kinds, derive some descriptive covariates, build many models on each data set, and then build a "meta" model predicting the performance of those models on the data. then, when a new problem appears, try to use this meta model to guess what model to use.

answered Sep 27 '11 at 08:39

downer
488●7●16●20

ICD-9 code itself is knowledge. What I wanted to achieve is adapt this data using a suitable machine learning framework so that I can use the dataset to define next ICD-9 code for a new sample. Thanks for the suggestion on Meta optimize approach. Do you have any references / literature to share ?

(Sep 27 '11 at 14:34) Khader Shameer

Given a dataset, how do you choose a machine learning algorithm

Subscription:

Related questions