PPO
Clipped policy optimization
Reinforcement learning that feels like Rust. Compose agents from swappable parts, catch wiring mistakes when you compile, and train at native speed with Burn on CPU, CUDA or Metal.
End-to-end Breakout PPO throughput.
1.6× Stable-Baselines3, 1.4× CleanRL.
40,960 transitions with matched preprocessing and training
workloads.
CUDA · Throughput only, not final reward or sample efficiency. Emulator versions differ. See the benches README for methodology and fairness controls.
Select an algorithm, and for PPO or A2C a network topology, to see an assembly example.
// CartPole · separate models · PPO
use burn::tensor::Device;
use modurl::prelude::*;
use modurl_gym::classic_control::cartpole::CartPoleV1;
let device = Device::flex().autodiff();
let env = CartPoleV1::builder().seed(7).build()?;
let env = VectorizedGymWrapper::from(env);
let mut env = DeviceMultiGymWrapper::new(env, Device::flex(), device.clone());
let observation_space = env.observation_space();
let action_space = env.action_space();
// actor_model, critic and optimizers: setup omitted; see the guide.
let policy = ProbabilisticPolicyModel::new(
actor_model,
CategoricalDistribution,
);
let networks = SeparatePPONetwork::builder()
.actor_network(policy)
.critic_network(critic)
.actor_optimizer(actor_optimizer)
.critic_optimizer(critic_optimizer)
.actor_learning_rate(3e-4)
.critic_learning_rate(3e-4)
.build()?;
let mut agent = PPOAgent::builder()
.observation_space(observation_space)
.action_space(action_space)
.network_info(networks)
.batch_size(2_048)
.mini_batch_size(64)
.clip_range(ConstantSchedule::new(0.2))
.training_horizon(100_000)
.build()?;
agent.learn(&mut env, 100_000)?;
Implementations for on-policy and off-policy training.
Train PPO on CartPole.
git clone https://github.com/ModuRL/ModuRL.git
cd ModuRL
cargo run --release -p examples --example ppo_cartpole
Requires a Rust toolchain with edition 2024 support. Read the getting started guide