Skip to content

RWKV-6 Mixture of Experts

A sparse MoE RWKV-6 model

Visit official site

What RWKV-6 Mixture of Experts is

The material describes an experimental RWKV-6 model that uses sparse mixture of experts and scores roughly on par with Finch 14B on common benchmarks.

What you can do with it

  • Run natural language processing tasks on the model
  • Test the model on limited hardware resources
  • Integrate it to improve parameter efficiency

Model notes

  • Hash routing distributes tokens evenly across experts
  • Shared and new experts give a dual-width FFN
  • New experts start with a high learning rate

Pros & cons

✓ What we like

  • Sparse activation saves training and inference time
  • Token-shift can be applied in new experts

! What to watch out for

  • The architecture is described as costing more VRAM
  • The model is labelled experimental

FAQ

Which model is named?

Flock of Finches 37B-A11B v0.1 is named in the material.

What does MoE change?

Only a portion of parameters is active for any given token.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review