Skip to content

Skywork-MoE

A 14.6B-parameter mixture-of-experts model

Visit official site
Category AI models

What Skywork-MoE is

The model is described as a mixture-of-experts model initialised from the dense Skywork-13B checkpoint, with two techniques for expert diversity and layer-wise loss adjustment.

What you can do with it

  • 14.6B parameters with 16 experts
  • Gating logit normalization for expert diversity
  • Compatibility with Hugging Face, ModelScope and Wisemodel

Application scenarios

  • Evaluate on C-Eval, MMLU and CMMLU benchmarks
  • Run a HuggingFace inference example
  • Try a vLLM-based deployment example

Pros & cons

✓ What we like

  • A fast vLLM deployment method is listed
  • fp8 precision is listed for running the base model

! What to watch out for

  • The named baseline is the dense Skywork-13B checkpoint
  • The material lists Grok-1, DBRX, Mistral 8*22 and Deepseek-V2 as comparisons

FAQ

How large is the model?

The material states 14.6 billion total parameters, 16 experts and 2.2 billion active parameters.

What hardware is named?

The material lists 8xA100/A800 or higher GPUs and fp8 on 8*4090.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review