What an RL reward model hears in a 4-bar loop

When you train a reward model to score music, it learns something. The uncomfortable question is what. A scalar comes out — higher is “better” — but better along which axis? Before you optimize a generator against a reward model, it’s worth knowing what the model actually hears.

The setup

Take a reward model trained on human preference over short audio — say 4-bar loops. Then probe it: hold a loop fixed and perturb one thing at a time. Shift the key. Swap the drum timbre. Quantize the groove tighter, then looser. Add a fifth. Watch how the score moves.

What falls out

A few things tend to show up. The model is often more sensitive to production and timbre than to harmony — it rewards a loop that sounds expensive even when the notes are dull. It tends to like rhythmic regularity up to a point and then penalize it as stiffness, which is roughly what humans do. And it has blind spots: transpose a loop and the score barely moves, which tells you the model has learned something close to key-invariance, for better or worse.

Why probe at all

Because optimization is a magnifying glass. Whatever the reward model quietly over-weights, the generator will learn to exploit — and you’ll end up with output that scores beautifully and sounds wrong. The probe is cheap insurance. It turns “the reward model says 0.92” into “the reward model says 0.92 because the loop is well-produced and rhythmically steady,” which is a claim you can actually argue with.