Skip to content

LongLLaMA

A LLaMA model for very long contexts

Visit official site
Category AI models

What LongLLaMA is

The model is described as a long text language model based on OpenLLaMA and fine-tuned with FoT.

What you can do with it

  • Process long documents
  • Use 256k tokens or more
  • Replace LLaMA weights in existing code

Model notes

  • A 3B base model is provided
  • Inference code is hosted on Hugging Face
  • Evaluation results compare it with OpenLLaMA

Pros & cons

✓ What we like

  • Long context inference code is on Hugging Face
  • Evaluation results versus OpenLLaMA are provided

! What to watch out for

  • The 3B base model is not instruction tuned
  • Short context weight use is stated up to 2048 tokens

FAQ

How long a context is stated?

The material states 256k tokens or more.

What method is used?

The material states Focused Transformer, or FoT.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review