Connectivity between subsequent feature maps in CNNs

Short Version:

In a (Deep) Convolutional Neural Network, how to determine which feature maps of one layer should be connected to which feature maps of the previous layer?

Long Version:

On deeplearning.net, CNNs are explained. When it comes to the connectivity between subsequent layers, the text says:

"Notice how the receptive field spans all four input feature maps."

If I am not mistaken, this means that each feature map in the layer n is connected to each feature map in the previous layer n-1.

Furthermore, in the paper Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks an experiment regarding the Public Street View House Numbers dataset is described. The authors state:

"The number of units at each spatial location in each layer is [48, 64, 128, 160] for the first four layers and 192 for all other locally connected layers."

Additionally, the authors say they used convolution kernels of the size 5x5.

Do I understand correctly that even in such a large scale CNN each feature map is connected to all previous feature maps? For example the last convolutional layer in this case: it consists of 192 units (-> 192 feature maps). Each of these feature maps is connected to 192 previous feature maps, covering a 5x5 area. This results in 5 * 5 * 192 + 1 = 4801 parameters for each unit (including the bias). Thus, the whole layer consists of 4801 * 192 = 921792 weight parameters. Computationally, this means at each spatial location a 4801 x 192 matrix multiplication has to be performed. Is this correct?

I can't imagine how this is feasible for large input dimensions like in high definition imagery (in this Street View example the inputs are just tiny 64x64 patches!)

If not, what is a good way to initialize the connections between feature maps? (so the number of parameters gets reduced - I am not speaking of techniques like sparsity targets here which drag some of the parameters towards zero).