There has been a lot of renewed interest lately in neural networks (NNs) due to their popularity as a model for deep learning architectures (there are non-NN based deep learning approaches based on sum-products networks and support vector machines with deep kernels, among others). Perhaps due to their loose analogy with biological brains, the behavior of neural networks has acquired an almost mystical status. This is compounded by the fact that theoretical analysis of multilayer perceptrons (one of the most common architectures) remains very limited, although the situation is gradually improving. To gain an intuitive understanding of what a learning algorithm does, I usually like to think about its representational power, as this provides insight into what can, if not necessarily what does, happen inside the algorithm to solve a given problem. I will do this here for the case of multilayer perceptrons. By the end of this informal discussion I hope to provide an intuitive picture of the surprisingly simple representations that NNs encode.
It is tempting to assume that with the appropriate choice of weights for the edges connecting the second and third layers of the NN discussed in this post, it would be possible to create classifiers that output over any composite region defined by unions and intersections of the 7 regions shown below.
… For although in a certain sense and for light-minded persons non-existent things can be more easily and irresponsibly represented in words than existing things, for the serious and conscientious historian it is just the reverse. Nothing is harder, yet nothing is more necessary, than to speak of certain things whose existence is neither demonstrable nor probable. The very fact that serious and conscientious men treat them as existing things brings them a step closer to existence and to the possibility of being born.
This post is ¬ meant with all seriousness. Every comparison should ¬ be taken literally, and no traces of humor should be seen when none, in fact, exist. (¬)