Theoretical and Practical Questions about Regularization

I've been reading up on regularization and wanted to verify that I haven't misunderstood. Any problems with this summary?

• Regularization is used to prevent overfitting by penalizing model complexity. At a high level, regularization can be thought to "smooth" out the model. Overfitting can be caused by large weights on individual parameters, too many parameters in the model or both.

• L2 regularization penalizes parameters that have large weights. This penalty reduces weights of features across the model, making it "smoother" and more generalizable.

• L1 regularization zeros out individual feature parameters, creating sparseness in the model. Put another way, it helps eliminate irrelevant features from the model. L1 may be useful if your dataset has a large number of (irrelevant) features compared to examples ?

• You can use both L1 and L2 together (the "elastic net") to gain benefits from both hyperparameters. This "will often give you most of the performance of L2 while completely zeroing of the overly noisy features." (source)

• You can make a model using L1 on your data to prune out irrelevant features, then update the model using L2 to smooth out the remaining features.

Does all that sound ok so far? Missing anything big, or misconstrue some concepts?

If that all checks out, I'm curious what are some practical tips for L1/L2 regularization. What are some practical values? Which direction is "strong regularization" - e.g. is 0.001 or 10 more regularization?

What are some tell-tale signs that you are overfitting your data (and need to add regularization)? When your training converges...but the performance metric on your testing set is poor?