Day 3 - RNNs and Sequence Models: From LSTM to Transformer

📌 Key Takeaways

• An RNN processes a sequence by accumulating the past in its hidden state, but it is weak at long-term dependencies because of vanishing and exploding gradients.
• LSTM (gates + cell state) / GRU (fewer gates, lighter and faster) mitigate that weakness.
• In seq2seq (encoder-decoder), the information bottleneck of the fixed context vector was the problem, and attention resolved it.
• The Transformer processes sequences with self-attention alone, with no recurrence, so it is strong at parallel training and long-range dependencies (BERT/GPT/T5).
• AWS mapping: time series = DeepAR/Forecast, text = BlazingText/Comprehend, generation = JumpStart/Bedrock.

Data Where Order Creates Meaning

If images have spatial structure, then text, time series, speech, and logs have sequential structure. For this kind of data, where earlier inputs affect later outputs, we use recurrent neural networks (RNNs) and their successors, and today's mainstream, the Transformer.

MLS-C01 asks about the differences among RNN/LSTM/GRU, the seq2seq structure, the concepts of attention and the Transformer, and algorithm choices for time series and NLP.

The Basic Idea of RNNs

An RNN processes a sequence one time step at a time and accumulates past information in its hidden state.

h_t = f(W_x * x_t + W_h * h_{t-1} + b)
The output y_t is computed from h_t

• h_{t-1}: the "memory" up to the previous step is passed on to the next computation.
• The weights are shared across all time steps (weight sharing along the time axis).
• It naturally handles variable-length sequences.

 x_1        x_2        x_3        x_4
  │          │          │          │
  ▼          ▼          ▼          ▼
[RNN] ──h1─▶[RNN] ──h2─▶[RNN] ──h3─▶[RNN] ──▶ h4
  │          │          │          │
  ▼          ▼          ▼          ▼
 y_1        y_2        y_3        y_4
  └─ Every step shares the same weights W ─┘

Unrolled along the time axis, an RNN is equivalent to a very deep neural network. So during backpropagation (BPTT — backpropagation through time), gradients accumulate through multiplication and vanishing/exploding gradients become severe. As a result, a simple RNN suffers from the long-term dependency problem of being unable to remember information from the distant past. This limitation is the motivation for LSTM and GRU.

Sequence Task Types

The structure differs depending on which side, input or output, is the sequence.

⚠️ Trap: If you see a clue that the input and output lengths differ, the answer is an encoder-decoder (seq2seq), not a per-step output structure. Translation and summarization are the typical cases.

LSTM and GRU

These are RNN variants that mitigate the long-term dependency problem with gate mechanisms.

• Cell state: the LSTM's "highway" that carries information over long distances. The gates control what to erase (forget), what to add (input), and what to emit (output).
• GRU reduces the gates so computation is lighter. It is a good LSTM alternative when data and resources are moderate.

Selection guide: use LSTM when long-term dependencies are decisive and resources are ample, and GRU when it must be lighter.

⚠️ Trap: The remedies for gradient vanishing and exploding differ. Vanishing is solved with structure (gates, residuals, ReLU). Exploding is handled with gradient clipping and a smaller learning rate. If the loss slowly stalls, suspect vanishing; if it suddenly jumps to NaN, suspect exploding.

[LSTM cell]
  previous cell state c_{t-1} ──────┬──▶ c_t   (long-range information path)
                                     │
   input x_t ─┬─▶ forget gate → what to erase
   previous h_{t-1}├─▶ input gate → what to write anew
             └─▶ output gate → what to emit ──▶ h_t

RNN · LSTM · GRU · Transformer Comparison

In a question saying "I added GPUs but RNN training did not get faster," the cause is not the hardware but the sequential processing structure itself. If you want parallelism, you must move to the Transformer family.

seq2seq (Encoder-Decoder)

A structure that maps an input sequence → an output sequence (which may have a different length), as in machine translation.

Input sequence → [Encoder RNN] → Context vector → [Decoder RNN] → Output sequence
"I am happy"                                                (Korean: "I am happy")

• The encoder compresses the whole input into a fixed-length context vector.
• The decoder generates tokens one at a time from that vector.
• Limitation: compressing a long input into a single vector loses information (a bottleneck).

Attention

It appeared to resolve the seq2seq bottleneck. The decoder learns which part of the input to focus on at each output step.

When the decoder generates the Korean word for "happy"
→ it assigns a high attention weight to "happy" in the input

        Input:  I     am    happy
               │     │      │
               ▼     ▼      ▼
   [Encoder hidden states h1, h2, h3]
               │     │      │
        weight 0.1   0.2    0.7      ← similarity to the decoder's current state (query)
               └──┬──┴──────┘
                  ▼
          weighted sum = context for this step
                  ▼
             Decoder → the Korean word for "happy"

• Instead of a fixed context vector, it dynamically references the entire input.
• It reduces information loss on long sequences and gives interpretability of input-output relationships.

The core of attention is the "query, key, value" mechanism. Weights are computed from the similarity between the decoder's current state (query) and each encoder position (key), and the values are then summed using those weights. Applying this idea, without any RNN, to every pair of positions inside a single sequence is self-attention, which is the foundation of the Transformer.

Transformer

"Attention Is All You Need" (2017) dropped the RNN and processed sequences with self-attention alone. It is today's standard for NLP and generative AI.

• Parallel processing: it does not process time steps sequentially like an RNN but handles the whole sequence at once, so GPU training is much faster.
• Long-range dependencies: distant tokens are connected directly through attention.
• Positional encoding: order information is injected separately (attention itself does not know position).
• Representative models: BERT (encoder, understanding), GPT (decoder, generation), T5 (encoder-decoder).

It is at the center of the pretraining-fine-tuning paradigm (large-scale pretraining + downstream fine-tuning), and AWS makes it easy to access through SageMaker JumpStart and Amazon Bedrock.

AWS Mapping for Time Series and NLP

How to Turn Text into Numbers

To feed text into a sequence model, you first have to convert it into a representation.

• SageMaker BlazingText supports both Word2Vec (embeddings) and text classification.
• For tasks where word order matters (translation, named entity recognition), the Bag-of-Words family is unsuitable.

Training Your Own vs Managed AI Services

In time-series forecasting, DeepAR learns many related time series together, letting series with sparse data borrow patterns from other series (a global model). When there are many similar series (demand per store or per SKU), it is stronger than fitting ARIMA separately to each one. "Forecast demand for many products at once" is a DeepAR/Forecast signal.

Reading Time-Series Regression Metrics

If you turn sparse demand data into a "sold/not sold" binary classification and evaluate it with accuracy, most values are 0, so predicting nothing at all still produces a high number. Evaluate regression tasks with RMSE and MAE, and imbalanced classification with recall and PR-AUC.

Question Clue → Answer Mapping

"It's the latest technology, so always Transformer" is a wrong path. If the question has clues such as "managed and fast" or "no ML staff," the answer is not a model structure but a managed AI service.

Symptom → Cause → Remedy

The next post covers how these structures are actually trained — loss, optimizers, learning rate, regularization, and transfer learning.

📖 Glossary

• Sequence data: data where the order itself carries meaning. Text, time series, speech, and logs qualify.
• Hidden state: the RNN's memory vector that carries information up to the previous step and passes it to the next.
• BPTT: applying backpropagation to an RNN unrolled along the time axis. The reason vanishing and exploding gradients get severe.
• Long-term dependency: the property that information from the distant past must be reflected in the current output. A point where simple RNNs are weak.
• Cell state / gates: the path the LSTM uses to carry information over long distances / switches learned to control what to erase, add, and emit.
• seq2seq (encoder-decoder): a structure that compresses an input sequence into a vector and then generates an output sequence of a different length.
• Attention: a mechanism that references the relevant parts of the input by weighting them at each output step. It resolves the bottleneck.
• Self-attention: attention applied to every pair of positions inside a single sequence. The foundation of the Transformer.
• Positional encoding: a device that separately adds the order information attention does not know to the input.
• DeepAR: a SageMaker built-in time-series forecasting algorithm that learns many related time series together.