Most any-to-any voice conversion systems spend their complexity budget on one problem: learning to disentangle what was said from who said it, usually with a VAE, vector quantization plus a mutual-information penalty, or a text bottleneck. kNN-VC (arXiv:2305.18975, Interspeech 2023) skips the learning entirely. The claim is almost provocative in how simple it is: a frozen self-supervised speech encoder already linearizes phonetic content well enough that you can convert a voice with a plain nearest-neighbor lookup, no disentanglement model required. I wanted to know if that claim actually reproduces, not just reads well. Code’s on GitHub.

The method, in one pass

Three stages: encoder, converter, vocoder.

Encoder — a frozen, pretrained WavLM-Large, feature-extracted specifically from layer 6. Not the later layers (22 or 24) that are better for phone recognition — layer 6 is the empirical sweet spot for retaining pitch, prosody, and speaker identity alongside phonetic content. One 1024-dim vector per 20ms of audio.

Converter — for every frame of the source speaker’s speech, find its k=4 nearest neighbors (cosine distance) among a pooled, shuffled set of WavLM frames from the target speaker’s reference recordings, and replace the source frame with their unweighted mean. No training. No learned parameters. This step is a lookup table, not a model.

Vocoder — HiFi-GAN, but with a twist worth pausing on: it’s trained on prematched features — the training set itself gets run through the same kNN-averaging the converter does at inference, so the vocoder learns to reconstruct from data that already looks like noisy, nearest-neighbor-averaged input instead of clean WavLM features. This closes a train/inference mismatch that would otherwise exist, and it’s the idea from this paper I expect to reuse elsewhere: train a downstream model on the actual statistical shape of what it’ll see at inference, not on the clean upstream signal.

Speaker identity survives because the substituted frame is drawn literally from the target speaker’s own recordings — a concatenative-style guarantee, not a learned one. Content survives because the substitution is nearest-neighbor on a feature space that clusters by phone regardless of speaker.

Reproducing Table 1

Tier 1 was inference-only sanity: pretrained WavLM + HiFi-GAN, no training, listen to the output. Tier 2 is the real test — reproduce the paper’s objective metrics (WER, CER, EER) against Table 1, using only pretrained checkpoints, no training.

Full run: 200 source utterances (5 per speaker × 40 speakers) from LibriSpeech test-clean, each converted to all 39 other speakers — 7,800 conversions, scored for intelligibility (Whisper-base WER/CER) and speaker similarity (a pretrained speaker-embedding model, EER).

Metric	Ours	Paper	Delta
WER (converted)	7.81	7.36	+0.45
CER (converted)	3.54	2.96	+0.58
WER (topline, real)	6.08	5.96	+0.12
CER (topline, real)	2.67	2.38	+0.29
EER (%, higher = better)	35.99	37.15	−1.16

Within about half a point on WER/CER and 1.2 points on EER, using nothing but off-the-shelf pretrained checkpoints — close enough to call the headline claim reproduced.

Why the numbers don’t match exactly, and why that’s fine

I’d rather be specific about the gap than wave at it:

Different Whisper checkpoint. The paper used 2023-era Whisper-base defaults; this reproduction pulls the current openai-whisper package. The topline row (real, unconverted speech) already shows ~0.1–0.3 points of drift before kNN-VC even enters the picture — which is useful, because it calibrates how much of the converted-row gap is ASR-version noise versus an actual conversion-quality difference.

Different eval sample. The paper doesn’t publish which 200 test-clean utterances or random seed it used. This reproduction picks a deterministic sample (sorted by speaker then utterance ID) so its own runs are reproducible, but it isn’t utterance-for-utterance identical to the paper’s.

Different speaker-verification model for EER. The paper’s EER protocol traces to an x-vector architecture whose exact trained checkpoint was never published alongside the paper. I used RF5/simple-speaker-embedding, a GE2E model released by kNN-VC’s own first author — the closest available proxy, not confirmed identical to whatever produced Table 1. Worth naming the alternative I tried first: a generic SpeechBrain VoxCeleb x-vector gave EER ~46%, a much larger gap — swapping to the domain-matched model closed most of it.

One bug worth confessing. An earlier version of the eval script had loudness normalization disabled — copied from a Tier 1 notebook cell that had a different reason for disabling it — which alone inflated EER by about 10 points before I caught it and matched the library’s actual default (tgt_loudness_db=-16).

None of that changes the shape of the result. WER/CER land within half a point of the paper and EER within about a point, using only pretrained checkpoints and zero training — enough to call the core claim reproduced without pretending it’s a digit-for-digit replication.