# 5. Chart-Driven MIDI Out to the Moogs
Part of [[MIDI Rhythm Game]]. Prev: [[4. Note Highway Input and Judgement]]. Next: [[6. Latency Calibration and Worked Example]].
In synth mode there is no backing clip. The **chart is the band**, and the band is your Mother-32 and Subharmonicon. This part reuses `MidiOutService` from [[Eurorack and Unity]] (section 3): one port, every send serialised behind a lock, 0-based channels, helpers `Send2`, `Send3`, `SendRealtime`.
## 5.1 Why not send notes in `Update`?
`Update` runs once per frame. At 60 fps a note scheduled "for this frame" can go out anywhere in a 16.7 ms span, and on a frame hitch much later. Musicians hear 10 ms of swing. The Eurorack note already solved this for clock with `MidiClockThread` (a `Stopwatch` thread). Here we generalise it: a **time-ordered event queue** ([[Game Programming Patterns (Review)]], Event Queue) drained by a thread, with times on the **dspTime** line so MIDI and the conductor agree.
The thread cannot call `AudioSettings.dspTime` safely (Unity APIs are main-thread only), so the main thread publishes an **anchor** each frame: "smoothed dsp time D was observed at stopwatch time S". The thread computes `dspNow = D + (stopwatchNow - S)`.
> [!warning] One clock source
> If you use the scheduler's clock (5.4), **disable `MidiClockThread`** from the Eurorack note. Two clock streams into the Subharmonicon doubles or wobbles its tempo.
## 5.2 `MidiEventScheduler.cs`
```csharp
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using UnityEngine;
/// <summary>
/// Sends MIDI bytes at precise dspTime-aligned moments from a background thread.
/// Main thread: Schedule(...) and publish the dsp anchor each frame.
/// Worker thread: pops due events and calls MidiOutService (which serialises sends behind a lock).
/// NOT compiled in Unity.
/// </summary>
[DefaultExecutionOrder(-90)] // right after the conductor
public sealed class MidiEventScheduler : MonoBehaviour
{
struct MidiEvent
{
public double dsp; // when to send, on the dspTime line
public byte status, d1, d2;
public byte length; // 1 (realtime), 2 or 3 bytes
public long seq; // tie-breaker so same-time events keep insertion order
}
[SerializeField] RhythmConductor conductor;
[Tooltip("Send this much earlier than the target time to cancel MIDI transport + synth response. Measure it.")]
[SerializeField] double sendEarlySec = 0.0;
readonly List<MidiEvent> _queue = new(); // kept sorted by (dsp, seq)
readonly object _queueLock = new();
readonly object _anchorLock = new();
double _anchorDsp, _anchorSw;
long _seq;
Thread _thread;
volatile bool _running;
Stopwatch _sw;
MidiOutService _out;
public int Pending { get { lock (_queueLock) return _queue.Count; } }
void OnEnable()
{
_out = MidiOutService.Instance;
if (_out == null) { UnityEngine.Debug.LogWarning("[MidiScheduler] no MidiOutService in the scene"); enabled = false; return; }
_sw = Stopwatch.StartNew();
PublishAnchor();
_running = true;
_thread = new Thread(Run) { IsBackground = true, Name = "MidiEventScheduler", Priority = System.Threading.ThreadPriority.AboveNormal };
_thread.Start();
}
void OnDisable()
{
_running = false;
_thread?.Join(200);
_thread = null;
lock (_queueLock) _queue.Clear();
}
void Update() => PublishAnchor();
void PublishAnchor()
{
double dsp = (conductor != null && conductor.IsRunning) ? conductor.SmoothedDsp : AudioSettings.dspTime;
double sw = _sw.Elapsed.TotalSeconds;
lock (_anchorLock) { _anchorDsp = dsp; _anchorSw = sw; }
}
double DspNow()
{
double d, s;
lock (_anchorLock) { d = _anchorDsp; s = _anchorSw; }
return d + (_sw.Elapsed.TotalSeconds - s);
}
public void Schedule3(double dsp, byte status, int d1, int d2) => Enqueue(dsp, status, (byte)d1, (byte)d2, 3);
public void Schedule2(double dsp, byte status, int d1) => Enqueue(dsp, status, (byte)d1, 0, 2);
public void ScheduleRealtime(double dsp, byte status) => Enqueue(dsp, status, 0, 0, 1);
void Enqueue(double dsp, byte status, byte d1, byte d2, byte len)
{
var e = new MidiEvent { dsp = dsp, status = status, d1 = d1, d2 = d2, length = len };
lock (_queueLock)
{
e.seq = _seq++;
// Binary search for the insertion point (events usually arrive nearly in order).
int lo = 0, hi = _queue.Count;
while (lo < hi)
{
int mid = (lo + hi) >> 1;
var m = _queue[mid];
if (m.dsp < e.dsp || (m.dsp == e.dsp && m.seq < e.seq)) lo = mid + 1; else hi = mid;
}
_queue.Insert(lo, e);
}
}
/// <summary>Drops everything not yet sent. Call before stopping, then send note-offs.</summary>
public void ClearPending() { lock (_queueLock) _queue.Clear(); }
void Run()
{
while (_running)
{
double now = DspNow() + sendEarlySec;
bool sent = false;
double nextDue = double.MaxValue;
MidiEvent e = default;
lock (_queueLock)
{
if (_queue.Count > 0)
{
if (_queue[0].dsp <= now) { e = _queue[0]; _queue.RemoveAt(0); sent = true; }
else nextDue = _queue[0].dsp;
}
}
if (sent)
{
switch (e.length)
{
case 1: _out.SendRealtime(e.status); break;
case 2: _out.Send2(e.status, e.d1); break;
default: _out.Send3(e.status, e.d1, e.d2); break;
}
continue; // drain anything else that is due without sleeping
}
double wait = nextDue - now;
if (wait > 0.003) Thread.Sleep(1); // Windows sleep granularity is coarse; stop sleeping near the target
else Thread.SpinWait(100);
}
}
}
```
### Notes on the design
- **`List.Insert` with binary search** is O(n) per insert, but the queue only holds the next ~half second of events (hundreds at most). A heap is the textbook answer; a sorted list is easier to read and debug.
- **`sendEarlySec`** is the MIDI-side latency knob. A 3-byte MIDI message at 31,250 baud takes about 1 ms on the DIN wire, plus USB scheduling and the synth's own response. Measure the total in part 6 instead of guessing.
- **`continue` after a send** drains bursts (a chord, or clock + note on the same tick) without sleeping between them.
- **Same-time ordering** uses `seq`, so a Note Off and a Note On for the same pitch at the same time go out in the order you scheduled them (schedule the Off first).
## 5.3 `ChartMidiSequencer.cs`
Schedules the chart's band tracks a little ahead of time, and the clock for the Subharmonicon.
```csharp
using System.Collections.Generic;
using UnityEngine;
/// <summary>
/// Synth mode: walks the chart's M32/SUB/guide tracks and schedules MIDI ahead of the conductor.
/// Channels are human (1-16) like Mother32Controller / SubharmoniconController. NOT compiled.
/// </summary>
public sealed class ChartMidiSequencer : MonoBehaviour
{
[SerializeField] RhythmConductor conductor;
[SerializeField] MidiEventScheduler scheduler;
[Header("Channels (match the synths; see Eurorack and Unity 1.3.3)")]
[SerializeField, Range(1, 16)] int mother32Channel = 1;
[SerializeField, Range(1, 16)] int subharmoniconChannel = 2;
[Header("Behaviour")]
[SerializeField] bool sendSubharmoniconClock = true;
[SerializeField] bool playGuideOnMother32 = false; // Easy mode: the player's part as a quiet guide
[SerializeField, Range(1, 127)] int guideVelocity = 40;
[SerializeField] double lookAheadSec = 0.5;
[SerializeField] double minNoteSec = 0.02; // gate time floor so short notes still sound
[Tooltip("Start clock this many beats before beat 0 so the Subharmonicon locks tempo first.")]
[SerializeField] int clockPreRollBeats = 4;
struct Pending { public double onSec, offSec; public int ch0, note, vel; }
NoteChart _chart;
readonly List<Pending> _events = new();
readonly HashSet<(int ch0, int note)> _sounding = new();
int _nextEvent;
long _nextClockTick; // 24 per beat, can be negative during pre-roll
double _scheduledUntilSec;
public void Load(NoteChart chart)
{
_chart = chart;
_events.Clear();
AddTracks(ChartTrackRole.BandMother32, mother32Channel - 1, -1);
AddTracks(ChartTrackRole.BandSubharmonicon, subharmoniconChannel - 1, -1);
if (playGuideOnMother32) AddTracks(ChartTrackRole.Playable, mother32Channel - 1, guideVelocity);
_events.Sort((a, b) => a.onSec.CompareTo(b.onSec));
_nextEvent = 0;
_nextClockTick = -clockPreRollBeats * 24L;
_scheduledUntilSec = double.NegativeInfinity;
}
void AddTracks(ChartTrackRole role, int ch0, int velocityOverride)
{
foreach (var t in _chart.TracksWithRole(role))
foreach (var n in t.notes)
{
double on = _chart.BeatToSeconds(n.beat);
double off = System.Math.Max(on + minNoteSec, _chart.BeatToSeconds(n.beat + n.lengthBeats));
// SUB notes are sent as written: the Subharmonicon reads them as offsets from C4 (MIDI 60),
// so author the SUB track around C4 in the DAW (Eurorack and Unity 4.2).
_events.Add(new Pending
{
onSec = on, offSec = off, ch0 = ch0,
note = Mathf.Clamp(n.midiNote, 0, 127),
vel = velocityOverride > 0 ? velocityOverride : Mathf.Clamp(n.velocity, 1, 127)
});
}
}
void Update()
{
if (_chart == null || conductor == null || !conductor.IsRunning || conductor.IsPaused) return;
double horizon = conductor.SongTime + lookAheadSec;
if (horizon <= _scheduledUntilSec) return;
// Notes: schedule on and off together; the scheduler sorts them.
while (_nextEvent < _events.Count && _events[_nextEvent].onSec <= horizon)
{
var e = _events[_nextEvent++];
byte on = (byte)(0x90 | e.ch0), off = (byte)(0x80 | e.ch0);
scheduler.Schedule3(conductor.DspAtSongTime(e.onSec), on, e.note, e.vel);
scheduler.Schedule3(conductor.DspAtSongTime(e.offSec), off, e.note, 0);
_sounding.Add((e.ch0, e.note));
}
// Clock: 24 pulses per quarter note on the chart's beat grid (tempo-map aware).
if (sendSubharmoniconClock)
{
while (true)
{
double beat = _nextClockTick / 24.0;
double sec = beat >= 0 ? _chart.BeatToSeconds(beat) : beat * 60.0 / _chart.InitialBpm;
if (sec > horizon) break;
scheduler.ScheduleRealtime(conductor.DspAtSongTime(sec), 0xF8);
_nextClockTick++;
}
}
_scheduledUntilSec = horizon;
}
/// <summary>Stop cleanly: drop the queue, then release every note we might have left on.</summary>
public void StopAndSilence()
{
scheduler.ClearPending();
var svc = MidiOutService.Instance;
if (svc == null) return;
foreach (var (ch0, note) in _sounding) svc.NoteOff(ch0, note);
_sounding.Clear();
}
void OnDisable() => StopAndSilence();
}
```
### Walkthrough
1. **`Load`** flattens the band tracks into a list of (on, off) pairs in *seconds* (tempo map applied once).
2. **`Update`** schedules everything whose start falls inside `lookAheadSec`. Half a second is long enough to survive a frame hitch and short enough that a pause or stop is responsive.
3. **Clock** pulses are computed from beats, so a tempo change in the chart changes the Subharmonicon's tempo exactly on the beat. The pre-roll sends a bar of clock before beat 0 so the Subharmonicon's tempo is settled when the song starts. (Its sequencer still needs PLAY pressed on the panel; the Eurorack note explains why Start/Stop messages are not relied on.)
4. **`StopAndSilence`** clears the queue first (so nothing new goes out), then sends Note Off for every note we started. Do this on pause, on scene unload and on quit, or the Mother-32 drones forever.
> [!note] `MidiOutService.NoteOff(ch, note)`
> `StopAndSilence` uses the Eurorack note's public `NoteOff(int ch, int note)` with a **0-based** channel, matching how `MidiOutService` numbers channels internally. If you changed that service, adjust here.
## 5.4 Scene setup (synth mode)
1. `Synth Rig` GameObject from [[Eurorack and Unity]]: `MidiOutService` (port name set), `Mother32Controller`, `SubharmoniconController`. **Disable `MidiClockThread`.**
2. `Rhythm` GameObject: `RhythmConductor` (no `music` clip), `MidiEventScheduler`, `ChartMidiSequencer`, `RhythmJudge`, `NoteHighway`, `RhythmInput`.
3. A small bootstrap (part 6) loads the chart into judge, highway and sequencer, then calls `conductor.StartSong(leadIn)`.
4. Press **PLAY** on the Subharmonicon before starting; it will follow Unity's clock.
5. Monitor both synths through your mixer or interface's direct monitoring. Do **not** monitor them through Unity's `AudioInputTap` for gameplay: that adds the input and output buffer latency twice.
## 5.5 Exercises
1. Add a `ControlChange` track role and send the Mother-32's assignable CCs (1/2/4/7 per [[Eurorack and Unity]]) from chart data to open the filter during the chorus.
2. When the player hits a long combo, transpose the SUB part up a fifth by adding 7 to its notes for the next bar. That is music as reward (Level 16).
3. Replace the hardware with the `OnAudioFilterRead` synth from [[FMOD vs Built-in Mixer and Procedural Synth]] for laptop-only classes: schedule notes into it by dsp time instead of MIDI.
## Sources
- MIDI 1.0 timing clock (0xF8, 24 per quarter note) and channel messages: MIDI Association, https://midi.org/midi-1-0-core-specifications
- `System.Diagnostics.Stopwatch`: https://learn.microsoft.com/dotnet/api/system.diagnostics.stopwatch
- `AudioSettings.dspTime` (main-thread clock used for the anchor): https://docs.unity3d.com/6000.5/Documentation/ScriptReference/AudioSettings-dspTime.html