Day 3 - RNNs and Sequences: From LSTM to Transformer

If images have spatial structure, text, time series, speech, and logs have sequential structure. For such data where previous input affects next output, we use Recurrent Neural Networks (RNN) and their advanced forms, and today's mainstream Transformer. MLS-C01 asks about RNN/LSTM/GRU differences, seq2seq architecture, attention and Transformer concepts, and algorithm selection in time series and NLP.

The Basic Idea of RNN

RNN processes sequences one time step at a time, accumulating past information in hidden state.

h_t = f(W_x * x_t + W_h * h_{t-1} + b)
Output y_t is computed from h_t