Skip to content

RL4VLM

Fine-tune a vision-language model as a decision agent

Visit official site

What RL4VLM is

The project is described as an open-source effort to fine-tune large vision-language models with reinforcement learning into decision agents, based on LLaVA and the PPO algorithm.

What you can do with it

  • Use the modified LLaVA model
  • Work in the GymCards environment
  • Run RL from prepared SFT checkpoints

Getting started

  • Get the code from the GitHub page
  • Prepare the required SFT checkpoints
  • Set up the conda environment for GymCards or ALFWorld

Pros & cons

✓ What we like

  • Detailed training steps and template scripts are supplied
  • Two conda environments fit different package needs

! What to watch out for

  • Setup asks for prepared SFT checkpoints
  • Two separate conda environments are described

FAQ

Which algorithm is named?

The material states PPO is used for reinforcement learning fine-tuning.

Which environments are included?

The feature list names GymCards and ALFWorld.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review