Skip to content

EasyContext

Train long context models on ordinary hardware

Visit official site
Category AI models

What EasyContext is

The project is described as an open source effort that combines existing techniques for very long context on ordinary hardware.

What you can do with it

  • Combine sequence parallelism with zero3 offload
  • Use Flash attention and activation checkpointing
  • Reach context lengths of 700K and 1M tokens

Model notes

  • Llama-2-7B was trained on 8 A100 GPUs at 700K tokens
  • Llama-2-13B was trained on 16 A100 GPUs at 1M tokens
  • The material notes a base for uses such as video generation

Pros & cons

✓ What we like

  • Only existing techniques are combined
  • 700K and 1M token lengths are reported

! What to watch out for

  • Results depend on the listed GPU setups
  • The project states it proposes no new innovation

FAQ

Which techniques are combined?

Sequence parallelism, zero3 offload, Flash attention and activation checkpointing.

What lengths were reached?

700K tokens on 8 A100 GPUs and 1M tokens on 16 A100 GPUs.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review