Otdfbt

Surprisingly this wasn't asked before - at least I didn't find anything besides some vaguely related questions.

So, what is a recurrent neural network, and what are their advantages over regular NNs?

Recurrent neural networks (RNNs) are a class of artificial neural network
architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information.

Difference with traditional Neural networks using pictures from this book:

And, an RNN:

Notice the difference -- feedforward neural networks' connections
do not form cycles. If we relax this condition, and allow cyclical
connections as well, we obtain recurrent neural networks (RNNs). You can see that in the hidden layer of the architecture.

While the difference between a multilayer perceptron and an RNN may seem
trivial, the implications for sequence learning are far-reaching. An MLP can only
map from input to output vectors, whereas an RNN can in principle map from
the entire history of previous inputs to each output. Indeed, the equivalent
result to the universal approximation theory for MLPs is that an RNN with a
sufficient number of hidden units can approximate any measurable sequence-to-sequence
mapping to arbitrary accuracy.

Important takeaway:

The recurrent connections allow a 'memory' of previous inputs to persist in the
network's internal state, and thereby influence the network output.

Talking in terms of advantages is not appropriate as they both are state-of-the-art and are particularly good at certain tasks. A broad category of tasks that RNN excel at is:

Sequence Labelling

The goal of sequence labelling is to assign sequences of labels, drawn from a fixed alphabet, to sequences of input data.

Ex: Transcribe a sequence of acoustic features with spoken words (speech recognition), or a sequence of video frames with hand gestures (gesture recognition).

Some of the sub-tasks in sequence labelling are:

Sequence Classification

Label sequences are constrained to be of length one. This is referred to as sequence classification, since each input sequence is assigned to a single class. Examples of sequence classification task include the identification of a single spoken work and the recognition of an individual
handwritten letter.

Segment Classification

Segment classification refers to those tasks where the target sequences consist
of multiple labels, but the locations of the labels -- that is, the positions of the input segments to which the labels apply -- are known in advance.

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks.

An RNN can be trained using back-propagation through time (BBTT), such that these backward or self-connections "memorise" previously seen inputs. Hence, these connections are mainly used to track temporal relations between elements of a sequence of inputs, which makes RNNs well suited to sequence prediction and similar tasks.

There are several RNN models: for example, RNNs with LSTM or GRU units. LSTM (or GRU) is an RNN whose single units perform a more complex transformation than a unit in a "plain RNN", which performs a linear transformation of the input followed by the application of a non-linear function (e.g. ReLU) to this linear transformation. In theory, "plain RNN" are as powerful as RNNs with LSTM units. In practice, they suffer from the "vanishing and exploding gradients" problem. Hence, in practice, LSTMs (or similar sophisticated recurrent units) are used.