Role of transfer functions in neural network

While the following aren't absolutely necessary, they are at least things people often take into consideration:

• Differentiability; if the function isn't differentiable, there are some good/numerically stable algorithms for doing numerical approximations (cf Neural Smithing, by Robert Marks and Russel Reed)
• Extreme behavior; eg, tanh and the sigmoid -- both solutions to the differential equation y' = y(1-y) -- approach a finite value at extreme values. Furthermore, they do not change rapidly. On the other hand, the hinge loss function has a rapid transition from 0 to 1, so numerical methods are necessary when using it (although, this ties in with differentiability above).
• The loss function; it isn't a requirement, but often the loss function at the output layer is chosen because it "matches" the transfer function. In short, the loss and transfer function together make the arithmetic simpler.
• The types of values you want to represent; your output layer could be made up of binary, softmax, linear (often called Gaussian), tanh, or other more exotic flavors (eg, rectified linear units). If the target result is a real value, linear may be what you need. If it's an integer, RLUs can work well. If the task is categorization, binary and softmax can do the task though they vary based on constraints that may exist. Softmax units are often used to assign membership to a single class amongst a group of classes and are essentially a combination of multiple binary units. However, unlike multiple independent binary units, softmax units have an additional constraint: the probabilities over each class must sum to 1.
• Internal dynamics; Tanh units can induce interesting dynamics if used as internal units, though I'm unsure how they differ as output units from logistic units. Some recent papers from Hinton's group demonstrated that rectified linear units offer a very rich representation of latent variables, so work very well in place of binary units.
• Computability; how expensive is it? Is it numerically stable? Can an approximation be used? What kind of bias does the approximation introduce (if any)?
• Symmetries; eg, should it be symmetric about an axis? Anti-symmetric?
• [Strictly?] monotonic?
• Does it take extra parameters? For example, if the units are related to some probability distribution or another that requires 2 parameters (eg, the normal distribution requires an average & standard deviation) then you may need to train sets of parameters in separate stages (akin to linear/quadratic programming problems)

As an example of the computability issue, Rectified Linear Units are essentially linear units, where the [theoretical] transfer function could be defined as:

sum( sigmoid(x - i + 0.5), i = 0 ... N)

That's a rather expensive function to compute. However, in the limit as N goes to infinity, the transfer function approaches

log(1 + e^(x)) ~= max(0,x) (for sufficiently large x)

There's other considerations to make here (max(0,x) is not differentiable, how do you compute a gradient for this? Do you need to add a corrective term to account for the approximation? etc), but I think I've probably given you enough to chew on.