Regularization vs. crossvalidation for avoiding overfitting

The 2 methods are generally used together since they have different purposes:

• cross-validation makes it possible to measure how a parameterized learning algorithm is able to generalize to data unseen at training time

• regularization is a parameter for a learning algorithm that trades off two types of potential generalization error: bias vs variance. Highly biased models cannot fit the training data as well (fewer degrees of freedom) but on the other hand are less likely to over-fit the training data by learning simpler models hence potentially causing less variance.

It is possible to combine the 2 together by doing a cross validated grid search for the optimal value of the regularization parameter (e.g. C in SVM).

Edit: arguably if the algorithm is able to scale to large datasets (linear training time) and that this data is available cheaply (generally not true for supervised learning) then regularization is less important as the redundancy in the training set will generally act as a natural regularizer that will prevent over-fitting. It is still interesting to do cross validation (maybe online cross validation to make it scalable) so as to measure the remaining amount of overfitting.

There is no rule against using both of them together.

Cross validation is usually a good method to find the best value for your regularization parameter.

You can find the optimum set of parameters using your training cost, then use the cross validation cost and test the parameters. (The training cost includes the regularization, while the CV does not).

Then you can test different values of the regularization parameters and use trial and error to find the best ones.