Skip to content

MoBA

Block-sparse attention for long-context language models

Visit official site

What MoBA is

The mechanism is described as a Mixture of Block Attention design that splits long context into blocks and lets each query token attend to the most relevant ones.

What you can do with it

  • Process long sequences with block-sparse attention
  • Switch between full and sparse attention modes
  • Run efficient computation on a 1M context

Getting started

  • Create a Python environment and install dependencies
  • Set the attn parameter to moba in the code
  • Run the example script and the unit tests

Pros & cons

✓ What we like

  • Seamless switching between full and sparse attention
  • A PyTorch implementation is provided

! What to watch out for

  • The described setup needs a Python environment
  • Switching to it is done through an attn parameter

FAQ

What does the gate do?

The material describes a parameter-free top-k gate that picks relevant blocks.

How is it enabled?

The steps name an attn parameter set to moba.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review