Why are the context and target tensors of the same size (max_length)

[krishnan-duraisamy]

In Chapter 2, section 2.6 includes samples for creating input-target pairs. It is clearly mentioned that the target is always a single token (next word output). However, in all worked examples, why then does the target tensor need to be same shape and size as the context tensor?

[rasbt]

Hi there,

that's a good question. I assume you are referring to figure 2.13?

Like you said, there is still one target per sequence, that's correct. But it would be wasteful to store them as one target at a time. E.g., for the sequence

[290] ----> 4920
[290, 4920] ----> 2241
[290, 4920, 2241] ----> 287
[290, 4920, 2241, 287] ----> 257

you could store and feed them as 4 separate training samples to the LLM as shown above. In this method, the model is trained on one target token at a time.

Instead, you can do it in one step, where the inputs are

[290, 4920, 2241, 287] 

and the targets are stored in a single tensor (they are the same targets IDs as before though)

     [4920, 2241, 287, 257]