# 4. Training with mlagents-learn
Part of [[ML-Agents and Inference Engine]]. Prev: [[3. Platformer Agent Observations Actions Rewards]]. Next: [[5. Running Trained Agents In-Game]].
## 4.1 Trainer config
Create `Training/config/CoinRunner.yaml` **outside** `Assets/` (a sibling of the Unity project or a separate repo). The top-level key under `behaviors` must equal the **Behavior Name** from part 3.
```yaml
behaviors:
CoinRunner:
trainer_type: ppo
hyperparameters:
batch_size: 128 # experiences per gradient step; small for discrete, vector-only tasks
buffer_size: 2048 # experiences collected before each update (multiple of batch_size)
learning_rate: 3.0e-4
beta: 5.0e-3 # entropy bonus: higher = more exploration
epsilon: 0.2 # PPO clip range
lambd: 0.95 # GAE lambda
num_epoch: 3
learning_rate_schedule: linear
network_settings:
normalize: false # our observations are already normalised by hand
hidden_units: 128
num_layers: 2
reward_signals:
extrinsic:
gamma: 0.99 # how far ahead it cares; 0.99 ~ 100 decisions
strength: 1.0
keep_checkpoints: 5
checkpoint_interval: 100000 # also exports an .onnx per checkpoint (used as difficulty levels in part 5)
max_steps: 1000000
time_horizon: 64
summary_freq: 10000
```
These values are the shape of the RollerBall config in the 4.2 tutorial, scaled up for a harder task. They are a starting point; the full list of keys and defaults is on the 4.2 "Training Configuration File" page (verify key names there if the trainer warns about an unknown key).
## 4.2 Train against the Editor
```shell
conda activate mlagents
cd Training
mlagents-learn config/CoinRunner.yaml --run-id=CoinRunner_01
```
When the terminal says it is waiting, press **Play** in Unity. During training ML-Agents speeds up time (the default engine time scale for training is high, so the Game view will look frantic; that's normal).
Good habits:
- **One run-id per experiment**, named for what changed (`CoinRunner_02_noShaping`). `results/<run-id>/` holds the config, checkpoints, TensorBoard logs and the final `CoinRunner.onnx`.
- `--force` overwrites an existing run-id; `--resume` continues one (keep the same config).
- `--initialize-from=CoinRunner_01` starts a new run from an old run's weights (useful after you change rewards slightly, as long as observation and action sizes are the same).
## 4.3 Train against a build (faster)
The Editor is single-environment and has overhead. For real runs:
1. **File > Build Profiles**, Windows, add only the training scene. A Development Build is not needed.
2. Build to `Training/Builds/CoinRunner/CoinRunner.exe`.
3. Run:
```shell
mlagents-learn config/CoinRunner.yaml --run-id=CoinRunner_03 --env=Builds/CoinRunner/CoinRunner.exe --num-envs=4 --no-graphics
```
`--num-envs` launches that many copies of the build; each copy also has the replicated areas from part 3.7, so 4 envs x 9 areas = 36 agents feeding one trainer. `--no-graphics` skips rendering (fine for vector and ray observations; **not** for camera sensors). Run `mlagents-learn --help` for the full option list on your installed version.
Automate the build with a menu item or CI job from [[Build Automation and CI]] so students don't forget to rebuild after changing the agent.
## 4.4 Watch it learn (TensorBoard)
```shell
tensorboard --logdir results --port 6006
```
Open `http://localhost:6006`. The charts to watch:
| Chart | Healthy | Trouble |
|---|---|---|
| Environment / Cumulative Reward | rises from around -1 toward +0.8..+1 | flat near -1: the agent never finds the coin; add shaping or make the first episodes easier |
| Environment / Episode Length | falls as it gets faster | rising: agent is stalling to avoid a penalty |
| Losses / Policy Loss, Value Loss | noisy but bounded | exploding: lower learning_rate |
| Policy / Entropy | slowly decreasing | crashes to near zero early: raise beta |
(Chart names are as I remember them from ML-Agents' TensorBoard output; verify in your run.) If TensorBoard is missing from the env, `pip install tensorboard` inside it.
## 4.5 Make the task learnable: curriculum by hand
If reward stays flat, make early episodes easier and harder later. ML-Agents supports **environment parameters** with curriculum lessons in the YAML, read in C# via `Academy.Instance.EnvironmentParameters.GetWithDefault("name", default)`. For the coin-runner:
```yaml
environment_parameters:
coin_on_ledge_chance:
curriculum:
- name: FlatOnly
completion_criteria:
measure: reward
behavior: CoinRunner
signal_smoothing: true
min_lesson_length: 100
threshold: 0.7
value: 0.0
- name: SomeLedges
completion_criteria:
measure: reward
behavior: CoinRunner
signal_smoothing: true
min_lesson_length: 100
threshold: 0.7
value: 0.3
- name: HalfLedges
value: 0.5
```
And in `CoinRunnerAgent.OnEpisodeBegin`, before placing the coin:
```csharp
coinOnLedgeChance = Academy.Instance.EnvironmentParameters.GetWithDefault("coin_on_ledge_chance", coinOnLedgeChance);
```
This is literally Rogers' level-design advice (Level 9 in [[Level Up! (Review)]]): teach the mechanic in a safe space, then combine it with danger. The agent needs the same ramp a player does. (Verify the curriculum YAML keys against the 4.2 training-configuration page; they have been stable since 2.x but are not re-checked here.)
## 4.6 Bring the model into Unity
1. Stop training (Ctrl+C) or let it reach `max_steps`. The trainer exports `results/<run-id>/CoinRunner.onnx`.
2. Copy it to `Assets/ML/Models/CoinRunner_Final.onnx`. Sentis imports `.onnx` files as a **ModelAsset**.
3. Also copy two intermediate checkpoints (e.g. the ones at 200k and 500k steps from `results/<run-id>/CoinRunner/`) as `CoinRunner_Easy.onnx` and `CoinRunner_Medium.onnx`. These are the difficulty "brains" for part 5.
## Sources
- ML-Agents 4.2 manual (Training ML-Agents, Training Configuration File, Using TensorBoard, Executable): https://docs.unity3d.com/Packages/
[email protected]/manual/index.html
- RollerBall tutorial config: https://docs.unity3d.com/Packages/
[email protected]/manual/Learning-Environment-Create-New.html
- Install and PyTorch notes: https://docs.unity3d.com/Packages/
[email protected]/manual/Installation.html