Implementing the LLM judge lifecycle from the Netflix paper

Netflix recently released a research paper on The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations. It treats an LLM judge like a model in production, with a benchmark, tuning loop, deployment gate, and drift monitoring. Let's see how to implement this practically, since the paper doesn't include an implementation codebase.

Using a second model to score the first is standard practice, and the architecture is well documented. Building one is less well documented. This walks through all four phases with working code you can run without an API key, including the places where the wording of a rubric, or the order of two fields in a JSON schema, quietly decides the verdict.

The problem

A model generates text for users, but nobody can manually review every response. So, you use a second model to evaluate the first against predefined criteria, a pattern commonly known as LLM-as-a-Judge.

All those criteria live in one place, a rubric. It tells the judge what passes, what fails, where the line is, and shows an example of each. It's just a plain Markdown file and that's the only thing the judge gets. No reference answer to compare against.

That leaves one problem. Nothing checks the judge.

What the paper proposes

Netflix published an answer, The Lifecycle of LLM-as-a-Judge. Treat the judge as a model in production, not a prompt you write once. That gives four phases.

The paper explains the design, but there is no code with it.

This post fills that gap. The full build is on GitHub, LLM Judge Lifecycle, all four phases, running without an API key. Every number below is reproducible from it.

What the judge is grading

Think of a streaming app. Next to each recommended film there is a short line saying why it was picked for you. A model writes that line, and that line is what the judge grades.

For example, because you watched Zodiac, try Prisoners, another slow-burn crime mystery.

Four criteria decide whether that sentence can be shown.

• Grounded: Every fact in it comes from the app's own record for that title. Nothing invented.
• Specific: It could only describe this title, not any other film in the catalogue.
• Safe: No spoilers.
• Concise: Between 12 and 40 words.

The first three need judgement. The fourth is just counting words. Keep that in mind, it matters for the whole post.

What this post covers

We'll go step by step. First a cheap judge, worth having before you pay for one. Then the four phases in order, building the benchmark, tuning the rubric, putting the judge in front of the writer, and watching for drift. Two more things, cost and small samples, apply at every phase, so they come last.

Something to know before you start. The mistakes in this build do not crash. They produce numbers that look like results, and the tests stay green. If it crashed you would fix it. Because it does not, the wrong number ships.

Step 0. Write the free judge first

Code, src/rules.py. Run it with python run.py eval --criterion concise.

Before you build any of the four phases, ask a cheaper question. Does this criterion need a model at all?

The rubric is a markdown file, so the model reads it as prose. Add an inline tag to some of the bullets and a short script can read the same file.

The script is free and exact, but it only matches text. It can't handle paraphrasing or anything needing real understanding. That's where a model helps. Until the model judge beats simple string matching, there's no reason to pay for the cost and latency.

For one criterion, the rules scored 0.167 while the model scored 0.667. For another, 0.500 vs 1.000. That gap is what justifies paying for the model. It is measured, not assumed.

The rule engine is deterministic, so your tests stop depending on a model being available, and anyone can run the pipeline before paying for a key.

Phase 1. Build the benchmark

Code, src/benchmark.py. Run it with python run.py benchmark.

The benchmark is the answer key. People read a sample of real outputs and mark each one pass or fail on every criterion, with a short note on why. Ours is about eighty examples across the four criteria.

Split it into train, validation and test up front. You tune on train, check on validation, and only look at test at the end. Without that split you are tuning against the same examples you report scores on.

Decide the split by hashing the example id, not by position in the file. Ours started out position-based, and appending twelve new examples moved three of them across the boundary, so old scores stopped being comparable to new ones.

Phase 2. Tune the rubric

Code, src/prompts.py, src/rart.py. Run it with python run.py tune --criterion concise.

With the benchmark in place, phase two tunes the rubric until the judge matches those labels.

Small wording changes can flip the judge's answer, so keep at least one rule you can check yourself. Ours is the 12-40 word limit. Count the words and you know whether the judge is right.

In the first run the judge got 12 of 20 wrong, and all 12 were inside the limit. The reasons it wrote showed us why.

Here is one of those reasons, in the judge's own words. It counted 29 words against a limit of 40, and said the explanation passes. Then it wrote "however, following the exact rubric instructions", and returned FAIL.

So the judge knew the answer and still marked it wrong. Two things caused that, and fixing only one of them was not enough.

• Do not ask for a failure mode: Our prompt ended by telling the judge to return the failure mode that best fits, from a list. That reads as an order to find a failure. On a criterion with only one mode, the judge failed the output just to have something to report.
• Put the reason before the label: With structured output the model fills in a JSON schema in the order you wrote it. Label first means it picks PASS or FAIL before thinking, and the reason it writes afterwards can only defend a choice already made.

Rewording the prompt took 12 wrong verdicts to 5. Swapping the two fields took it to zero.

Why this is easy to miss

The number you watch when tuning is specificity. Of all the outputs people marked bad, how many did the judge also mark bad?

Now imagine a lazy judge that fails everything it sees. People marked 10 of the 20 outputs bad. The lazy judge fails all 20, so it caught all 10. Specificity is 1.000, a perfect score from a judge that did no work at all.

That is the trap. Failing too much moves the number up, not down.

So how do you catch it? On grounded, specific and safe you cannot, because checking those means re-reading every output yourself. The word count is different. 29 is under 40, so a FAIL there is provably wrong. That is what gave the bias away. Take that one rule out and every number would have looked fine.

Then check the tuning was worth it

Code, src/metrics.py. Run it with python run.py ablation --criterion concise.

That means scoring the tuned rubric against the untuned one.

In our run the tuned rubric won by 0.67. It won on rubrics that were byte-identical.

The score adds three parts, and one checks whether the judge agreed with the person for the same reason, not just the same verdict. That part only runs on the tuned side, so the untuned side scored zero on something nobody had measured on it.

Identical inputs have to produce identical scores. If they do not, the difference is in how you measured, not in what you changed.

Phase 3. Put the judge in front of the writer

Code, src/serving.py. Run it with python run.py serve.

Phases one and two give you a judge you trust. The last two make it run on its own.

Deployment puts the judge in front of the writing model. A rejected explanation goes back for another attempt with the reason attached, and if it still cannot pass, nothing is shown. That is why a gate makes sense here. A missing explanation costs a little attention. A wrong one reaches a paying customer and cannot be taken back.

Phase 4. Watch for drift

Code, src/monitoring.py. Run it with python run.py monitor.

Monitoring catches all of it going stale. People disagree with each other, so a fixed threshold is a number someone made up. The bar floats with the people instead, and the judge is out of line only when it sits below where they already are.

That gap is drift, and phase four is the only part that reports it. Teams usually skip it, so a judge can be wrong for months while the dashboard stays green.

Two things that cut across every phase

These two are not steps. They apply to every phase above, and both are easy to get wrong in the direction that flatters you.

What it costs

Code, src/providers/gemini.py, src/runtime.py. Every command prints its own token and cost tally.

You pay per token. A thinking model thinks privately before it answers, and you pay for those tokens too. The API reports them in a different field from the answer, so if you add up only the answer, your cost report comes out too small.

How much too small? On one prompt our writing model spent 1,746 tokens thinking, then wrote a 40 word sentence. That is 44 tokens of thought per token you can see. Most providers let you set a thinking budget, a cap on that private thinking. We set ours to zero and got a sentence just as good for none of it.

• Add up every token field: Count only the visible answer and the bill can come out 44 times too small. The error always makes the bill look smaller than it is.
• That budget is a suggestion, not a limit: We tried 256 instead of zero. The model spent 983 and still ran out of room before finishing the answer. Only zero is enforced. For a hard cap, limit the answer length instead and handle what gets cut off.

Check this on your own stack before sizing a retry budget. Here the judge was 87% of spend, because it runs once per criterion per attempt while the writing model runs once.

Reading small numbers

Code, src/metrics.py. Run it with python run.py eval --criterion concise --split test.

The whole benchmark is about eighty labelled examples across four criteria, and once you hold some back for testing, each criterion is scored on five to seven.

That is tiny. Get three out of three right and the report says 1.000. That looks solved, but it is three examples.

A confidence interval tells you the room for error. Score five out of six and the true number sits between 0.44 and 0.97, nearly the whole range. So the 0.83 you printed does not tell you much.

So print the interval next to every number. It stops a small sample being read as a finished result.

Use the Wilson interval, not the textbook normal approximation. The textbook one says 1.00 to 1.00 at eight out of eight, claiming certainty from eight examples.

What to take from this

Three numbers from this build looked like results and were not. None of them raised an error.

• Build the free baseline first: Write the rules first and make the model judge beat them. Without that comparison you are paying for latency with no evidence it helps.
• Read the reasons, not just the labels: Both wording problems above were visible in the judge's own text while every metric looked healthy. The reason is also what the writing model gets back on a rejection, so a wrong one steers the retry at the wrong problem.
• Keep one criterion you can check by hand: A word count, a required field, a number that must match the source. It is the least interesting rule in your rubric and the only cheap way to spot bias in the others.
• Count every token field: Reasoning is billed but reported in a separate field, so it is easy to miss and the bill looks smaller than it is.
• Print the interval next to the number: A single number over six examples cannot support a conclusion, and three decimal places make it look like it can.

The full build is on GitHub, LLM Judge Lifecycle, with two domains and a test pinning each problem above. It runs offline. The architecture comes from the Netflix paper, The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations.