Skip to content

4M

A framework for multimodal and multi-task model training

Visit official site

What 4M is

The framework trains multimodal and multi-task models on a unified Transformer encoder-decoder, converting modalities into discrete token sequences for conditional generation.

What you can do with it

  • Produce depth and normal maps from an RGB image
  • Reconstruct missing RGB content from partial input
  • Retrieve images that match a text description

Getting started

  • Fetch code and pretrained models from the GitHub repository
  • Install the dependencies described in the docs
  • Load a model and prepare text or image inputs

Pros & cons

✓ What we like

  • Self-consistent prediction across modalities
  • Condition weights steer the generated output

! What to watch out for

  • Setup requires installing dependencies and an environment
  • Pretrained models must be downloaded and loaded

FAQ

What is the core design?

Modalities become discrete token sequences on one Transformer encoder-decoder.

Where is the code?

The material points to a GitHub repository with code and pretrained models.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review