Generative retrieval for cover songs, and what fell out of it

I spent leave on a paper. Generative retrieval for cover-song identification using audio-derived Semantic IDs. Submitted to ICECCME 2026 with Dr. Sarah Ita Levitan.

Pipeline: MERT embeddings, then a learned RQ-VAE that turns each track into a short sequence of discrete codes, then a T5 that generates the target’s codes from the query’s codes.

Three things I didn’t expect.

EnCodec direct codes collapse on low-bitrate audio. Codebook utilization drops under 6%. Within-clique token overlap falls to random. That killed the obvious baseline and motivated the learned RQ-VAE over MERT.

Compositional structure shows up before T5 ever trains. Prefix-overlap on Discogs-VI: random IDs at 1.06 times, EnCodec at 1.8, MERT-RQ-VAE at 4.3 on the first code and 11 on the first two. The audio model already knows two covers share something. T5 amplifies a real signal, not noise.

Generative retrieval beats the bi-encoder. T5 MERT-RQ-VAE wins 5 of 6 metrics on the Discogs-VI test set. MRR 0.271 vs 0.214. R@1 0.237 vs 0.152.

Two bugs cost three pod runs. The HuggingFace mean-init on resize_token_embeddings made every SemID token identical and collapsed the encoder; fix is per-token re-init. Clique-level splits left test items un-indexed, which generative retrieval can’t recover from; fix is pair-level splits.