# 3. Platformer Agent Observations Actions Rewards
Part of [[ML-Agents and Inference Engine]]. Prev: [[2. Installing ML-Agents and Python]]. Next: [[4. Training with mlagents-learn]].
The agent: a capsule in a walled 16 x 16 m arena with a low **ledge** and a **hazard pit** (a trigger tagged `Hazard`, like the volumes in [[Hazards, Traps and Key-Door Puzzles]]). A **coin** spawns at a random spot, sometimes on the ledge. The agent must learn to reach it, jumping when needed, without falling in the pit. This is small enough to train in well under an hour on a laptop (expect to tune), and it is a platformer problem, so it ties straight to [[Character Controllers]].
The structure follows the 4.2 manual's "Making a New Learning Environment" (RollerBall) tutorial, extended with jumping, a hazard and ray sensors.
> [!warning] Not compiled
> Untested code. Uses ML-Agents 4.x APIs from the 4.2 docs and Unity 6 `Rigidbody.linearVelocity`.
## 3.1 Scene and prefab
```
TrainingArea (empty, prefab root) <- everything inside is LOCAL to this
Floor (16x1x16 cube, layer Ground)
Walls x4 (tag Wall)
Ledge (4x1x4 cube at y=0.5..1.0 above floor, layer Ground)
Pit (3x0.2x3 trigger, tag Hazard, red material)
Coin (sphere, trigger collider, tag Coin)
Agent (capsule, Rigidbody, CoinRunnerAgent, Behavior Parameters,
Decision Requester, Ray Perception Sensor 3D)
```
1. Create tags `Coin`, `Hazard`, `Wall`; a layer `Ground`.
2. Agent Rigidbody: Mass 1, Linear Damping 1 (Unity 6 rename of Drag), **Freeze Rotation X, Y, Z** so the capsule stays upright.
3. Everything uses **local positions** relative to `TrainingArea` so you can duplicate the area many times (3.7).
## 3.2 Behavior Parameters
| Field | Value | Why |
|---|---|---|
| Behavior Name | `CoinRunner` | must match the YAML key in part 4 |
| Vector Observation > Space Size | **7** | see `CollectObservations` |
| Stacked Vectors | 1 | velocity is already observed, so no stacking needed |
| Actions > Continuous | 0 | |
| Actions > Discrete Branches | **2**: Branch 0 size **5**, Branch 1 size **2** | move (none/fwd/back/left/right), jump (no/yes) |
| Behavior Type | Default | train when the trainer is connected, else heuristic |
| Model | empty (for now) | part 5 |
**Decision Requester**: Decision Period **5**, Take Actions Between Decisions **on**. At the default 50 Hz fixed step that is 10 decisions per second, which is about human reaction speed and keeps episodes short in decision terms.
**Agent > Max Step**: **1500** (30 s of physics steps at 50 Hz). The episode auto-ends there; ML-Agents marks that as an interruption, not a failure.
## 3.3 Ray Perception Sensor 3D
Add **Ray Perception Sensor 3D** to the agent:
- Detectable Tags: `Coin`, `Hazard`, `Wall`
- Rays Per Direction: 4 (9 rays), Max Ray Degrees: 90, Ray Length: 12
- Sphere Cast Radius: 0.3; Start/End Vertical Offset: 0 / -0.5 (angles the rays slightly down so they see the pit)
The sensor adds its own observations; you do **not** count them in Space Size. It is the trained counterpart of the raycasts you hand-wrote in [[Organic Enemy Sensing]].
## 3.4 The agent script
```csharp
using UnityEngine;
using UnityEngine.InputSystem;
using Unity.MLAgents;
using Unity.MLAgents.Actuators;
using Unity.MLAgents.Sensors;
/// <summary>
/// Learns to reach a coin in a small arena, jumping onto a ledge when needed and avoiding a pit.
/// Observations (7): direction to coin (3), local velocity (3), grounded (1). Plus ray sensor.
/// Actions: branch 0 = move (0 none, 1 fwd, 2 back, 3 left, 4 right), branch 1 = jump (0/1).
/// </summary>
[RequireComponent(typeof(Rigidbody))]
public sealed class CoinRunnerAgent : Agent
{
[Header("Scene refs (all inside the same TrainingArea)")]
[SerializeField] Transform area; // TrainingArea root
[SerializeField] Transform coin;
[SerializeField] Transform ledge;
[Header("Movement")]
[SerializeField] float moveAcceleration = 25f;
[SerializeField] float maxSpeed = 6f;
[SerializeField] float jumpVelocity = 5.5f;
[SerializeField] LayerMask groundMask;
[SerializeField] float groundProbe = 1.1f; // capsule half-height + skin
[Header("Arena")]
[SerializeField] float halfSize = 7f; // keep spawns off the walls
[SerializeField, Range(0f, 1f)] float coinOnLedgeChance = 0.3f;
[Header("Rewards")]
[SerializeField] float coinReward = 1f;
[SerializeField] float hazardPenalty = -1f;
[SerializeField] float progressScale = 0.02f; // shaping; keep small (see 3.5)
Rigidbody _rb;
bool _grounded;
float _prevDistance;
public override void Initialize()
{
_rb = GetComponent<Rigidbody>();
}
public override void OnEpisodeBegin()
{
// Reset agent
_rb.linearVelocity = Vector3.zero;
_rb.angularVelocity = Vector3.zero;
transform.localPosition = RandomFloorPoint(1f);
transform.localRotation = Quaternion.identity;
// Place coin: sometimes on the ledge, which forces a jump
if (Random.value < coinOnLedgeChance && ledge != null)
coin.localPosition = ledge.localPosition + Vector3.up * 1.2f;
else
coin.localPosition = RandomFloorPoint(0.75f);
_prevDistance = DistanceToCoin();
}
Vector3 RandomFloorPoint(float y)
{
// Simple rejection sampling to avoid spawning in the pit or on the ledge footprint
for (int i = 0; i < 20; i++)
{
var p = new Vector3(Random.Range(-halfSize, halfSize), y, Random.Range(-halfSize, halfSize));
if (!Physics.CheckSphere(area.TransformPoint(p), 0.6f, ~groundMask, QueryTriggerInteraction.Collide))
return p;
}
return new Vector3(0f, y, -halfSize + 1f);
}
float DistanceToCoin() => Vector3.Distance(transform.position, coin.position);
public override void CollectObservations(VectorSensor sensor)
{
// Express everything in the arena's frame and normalise to roughly [-1, 1].
Vector3 toCoin = area.InverseTransformDirection(coin.position - transform.position);
sensor.AddObservation(toCoin / (halfSize * 2f)); // 3
Vector3 localVel = area.InverseTransformDirection(_rb.linearVelocity);
sensor.AddObservation(localVel / maxSpeed); // 3
sensor.AddObservation(_grounded ? 1f : 0f); // 1
}
public override void WriteDiscreteActionMask(IDiscreteActionMask actionMask)
{
// Can't jump in the air: mask branch 1, option 1. Masking shrinks what the agent must learn.
if (!_grounded) actionMask.SetActionEnabled(1, 1, false);
}
public override void OnActionReceived(ActionBuffers actions)
{
int move = actions.DiscreteActions[0];
int jump = actions.DiscreteActions[1];
Vector3 dir = move switch
{
1 => Vector3.forward,
2 => Vector3.back,
3 => Vector3.left,
4 => Vector3.right,
_ => Vector3.zero
};
// Accelerate in the arena frame, clamp horizontal speed (like MovementMetrics in Character Controllers)
_rb.AddForce(area.TransformDirection(dir) * moveAcceleration, ForceMode.Acceleration);
Vector3 v = _rb.linearVelocity;
Vector3 flat = new Vector3(v.x, 0f, v.z);
if (flat.magnitude > maxSpeed)
{
flat = flat.normalized * maxSpeed;
_rb.linearVelocity = new Vector3(flat.x, v.y, flat.z);
}
if (jump == 1 && _grounded)
_rb.linearVelocity = new Vector3(_rb.linearVelocity.x, jumpVelocity, _rb.linearVelocity.z);
// --- Rewards ---
// 1. Small time penalty: finish quickly. Scaled so a full episode sums to -1 at most.
if (MaxStep > 0) AddReward(-1f / MaxStep);
// 2. Progress shaping: reward getting closer, penalise getting farther. Potential-based, so it
// can't be farmed by oscillating (net zero over a loop).
float d = DistanceToCoin();
AddReward((_prevDistance - d) * progressScale);
_prevDistance = d;
// 3. Fell off the world (should be impossible with walls, but cheap insurance)
if (transform.localPosition.y < -2f)
{
SetReward(hazardPenalty);
EndEpisode();
}
}
void FixedUpdate()
{
_grounded = Physics.Raycast(transform.position, Vector3.down, groundProbe, groundMask,
QueryTriggerInteraction.Ignore);
}
void OnTriggerEnter(Collider other)
{
if (other.CompareTag("Coin"))
{
AddReward(coinReward);
EndEpisode();
}
else if (other.CompareTag("Hazard"))
{
AddReward(hazardPenalty);
EndEpisode();
}
}
public override void Heuristic(in ActionBuffers actionsOut)
{
// Keyboard control for testing the environment before training (Behavior Type = Heuristic Only).
var d = actionsOut.DiscreteActions;
var kb = Keyboard.current;
d[0] = 0; d[1] = 0;
if (kb == null) return;
if (kb.wKey.isPressed) d[0] = 1;
else if (kb.sKey.isPressed) d[0] = 2;
else if (kb.aKey.isPressed) d[0] = 3;
else if (kb.dKey.isPressed) d[0] = 4;
if (kb.spaceKey.isPressed) d[1] = 1;
}
}
```
### Why each piece is there
- **`Initialize`** instead of `Awake`: ML-Agents calls it once when the agent is enabled and registered.
- **Local frame and normalisation**: networks learn faster when inputs are around -1..1 and do not depend on where the area sits in the world. That is what lets 3.7 replicate areas.
- **Masking the jump while airborne** removes a useless action; the 4.2 docs describe action masking as a way to prevent invalid actions.
- **`EndEpisode` on success or failure**; `MaxStep` handles timeouts (counted as interrupted, so the trainer bootstraps the value rather than treating it as a terminal state).
- **`Heuristic`** with the Input System: always play the environment yourself first. If *you* can't reach the coin with these actions, the agent can't either.
## 3.5 Reward design (the important part)
| Signal | Value | Purpose | Hack risk |
|---|---|---|---|
| Coin | +1, end | the actual goal | none |
| Hazard | -1, end | teach avoidance | if the time penalty is too big, the agent learns to *suicide into the pit* to stop the bleeding. Keep total time penalty <= the hazard penalty. |
| Time | -1/MaxStep per step | urgency | see above |
| Progress | (prevDist - dist) x 0.02 | dense early signal | non-potential shaping (e.g. "+0.01 per step while moving") gets farmed by running in circles. Distance deltas telescope to (start - end) so loops net zero. |
Rule of thumb: start with **sparse reward only** (coin and hazard); add shaping only if learning stalls. Every shaping term is a design opinion the agent will take literally. This is Rogers' scoring advice in another form: players (and agents) optimise what you reward, not what you meant.
## 3.6 Test it manually
1. Behavior Type: **Heuristic Only**. Press Play, drive with WASD and Space.
2. Watch the **Agent** inspector: Cumulative Reward should go up approaching the coin, +1 on pickup, and reset each episode.
3. Fall in the pit: episode resets with about -1.
4. Set Behavior Type back to **Default** before training.
## 3.7 Many areas at once
Experience per minute is the training bottleneck. Two options:
- **Duplicate** the `TrainingArea` prefab (e.g. a 3 x 3 grid, 25 m apart). All agents share the behavior name, so they all feed one trainer.
- **Training Area Replicator** component (ML-Agents, added in 2.x per the changelog: "replicate training areas dynamically during runtime"): put it on an object in the scene, assign the area prefab, set the count and spacing, and it instantiates copies at runtime. Check the component's inspector for current field names.
Do not exceed what your CPU can simulate at the chosen time scale; watch the trainer's steps per second (part 4).
## Sources
- https://docs.unity3d.com/Packages/
[email protected]/manual/Learning-Environment-Create-New.html
- https://docs.unity3d.com/Packages/
[email protected]/api/Unity.MLAgents.Agent.html
- https://docs.unity3d.com/Packages/
[email protected]/api/Unity.MLAgents.Policies.BehaviorParameters.html
- https://docs.unity3d.com/Packages/
[email protected]/manual/index.html (Agents, Sensors, Actions and Rewards sections)
- `Rigidbody.linearVelocity`: https://docs.unity3d.com/6000.5/Documentation/ScriptReference/Rigidbody-linearVelocity.html