A quest for very long context: Part 1
Experiments on extending transformer context length, including training observations, tradeoffs, and lessons from long-context tuning.
Pythia
1 essay on this theme.
1 essay on this theme.
Experiments on extending transformer context length, including training observations, tradeoffs, and lessons from long-context tuning.