Lei or tu?

Context

In Italian, customer service addresses people with "Lei", the formal you. A bot that switches to the informal "tu" halfway through a chat sounds wrong. I wanted to build a corrector for this. Before building it, I wanted to know if the problem exists, and where.

So I ran two tests. The first one measured how often four models slip. The second one compared three fixes on the one case where the slips were frequent.

Method

First test. Each bot played five businesses: a bank, an insurer, a public office, a premium brand and a utility. The system prompt said "Dai sempre del Lei al cliente, anche se il cliente ti dà del tu". It also gave the bot a short knowledge base, four facts per business. Simulated customers came in five profiles. They were formal, used tu, switched to tu mid-chat, or got angry and used tu. In the fifth, the customer was formal and the knowledge base itself was written in tu. Every chat ran 8 turns.

Before running anything I wrote the decision rule. Stop if every model stays under 0.5% of replies with a tu form. Go if any model reaches 2%.

Second test. Only Gemini 3.5 Flash Lite, only the profile with the knowledge base in tu. Four arms:

• No fix, the control: the same setup as the first test.
• Knowledge base rewritten in Lei.
• Corrector on the control's replies.
• Both: corrector on the replies of the Lei knowledge base arm.

100 conversations and 800 replies per arm. The control and the Lei arm ran interleaved, so time of day hit both equally. The corrector arms reuse those same replies, corrected offline.

I wrote the rule and the order before the run. A fix passes if its slip rate is under 0.5%, a one-sided Fisher test against the control gives p < 0.01, and it keeps the facts. The first passing fix in the order knowledge base, corrector, both is the winner. The order goes from the closest to the cause to the one that adds the most parts. The rule also had a validity gate: the control had to slip in at least 2% of replies, or the comparison wouldn't mean anything.

The corrector. A small regex detector, built for the first test and left unchanged, flags tu forms. The corrector splits the reply into sentences and sends only the flagged ones to a small model. The instruction: rewrite them so every form addressed to the customer is Lei, and change nothing else. Then a deterministic check accepts or rejects the result:

• the same number of sentences comes back;
• no tu form is left, unless the sentence came back unchanged (a false positive, like the noun "prova");
• the facts in each sentence are identical: numbers, paths like "Carte > Sblocca", text in brackets, links, emails, quoted strings, capitalised names and negations;
• each sentence stays within 0.8 to 1.25 times its original length;
• everything outside the flagged sentences is identical, byte for byte.

If the check fails, the corrector tries once more. If it fails again, the original reply goes out unchanged and counts as uncorrected. I tuned the corrector only on replies from the first test, and froze it before the second.

Labelling. Another AI model, not people, labelled every reply blind, as Lei, tu, mixed or neutral. A slip is a reply labelled tu or mixed. Labellers saw only the customer's side of the chat and the reply, never the arm or the model. A second labeller re-checked 20% of the replies. Corrected texts went into a separate queue, mixed with an equal number of unedited replies as decoys, so no labeller saw an original next to its correction. A separate blind audit compared each original with its correction and judged whether the content changed.

Results

First test. Replies with a tu form:

Haiku's one slip was a "Hai ragione" to an angry customer on the last turn. Gemini's 3% was on the go side of the rule.

None of Gemini's slips was a full switch to tu. Each one was a Lei reply with one tu form inside, and 22 of the 24 were the same word: "metterti", in "metterti in contatto con un operatore umano". That phrase is not in the knowledge base. It's the bot's own wording for the escalation the system prompt asks for. The knowledge base in tu made it more likely. In that profile 14 of 160 replies slipped (8.75%, interval 5.3 to 14.2%). In the formal profile with a neutral knowledge base it was 3 of 160 (1.9%, interval 0.6 to 5.4%). One-sided Fisher p = 0.005. Those two profiles differ only in the knowledge base.

Second test. Out of 800 replies per arm:

Intervals are Wilson. The corrector arms share replies with the arms they correct, and the pre-registered Fisher test treats them as independent.

The control passed the validity gate at 2.88%. The Lei knowledge base failed: 2.0% is above 0.5%, and the difference from the control is not significant. The corrector passed every condition, so it is the winner. Both together also passed, with a higher rate.

All 23 slips in the control were "metterti". In the Lei knowledge base arm it was 13 of 16. Rewriting the knowledge base removed the tu from the input, and the phrase still came out.

The corrector in detail. On the control it edited 23 replies, all 23 slips. The audit judged all 23 corrections as the same content, and the labellers labelled all 23 as Lei. It edited none of the 777 replies that were already fine. The detector also flagged 18 replies with no real tu form, and the corrector returned those unchanged. The check re-ran on every accepted correction, 36 out of 36 across both arms, and all passed.

In the combined arm, 3 slips stayed. In two of them the check rejected the rewrite twice, so the original went out, as designed. The third is a form the detector doesn't know: "comprendi", in "comprendi perfettamente la Sua premura", most likely a typo for "comprendo".

The labellers agreed with the second pass on 320 of 320 replies. They gave the same label as the census to 36 of 36 decoys.

The rewritten knowledge base also had a fidelity problem. The audit judged 19 of its 20 bullets the same, and in one the rewrite dropped "ci" (to us) from "spedircelo", which changes who receives the parcel.

Limits

• The size of the effect moves. In the first test the knowledge base in tu gave 8.75% slips. In the second, with the same setup, the control gave 2.88%. The problem is real, but I can't give one number for it.
• Another AI model labelled every reply and ran the fidelity audit, not people. The guard is the only check that doesn't depend on a model.
• The customers were simulated by a model, not real people.
• The corrector ran offline, on replies already generated. The original reply stayed in the chat history, so later turns still saw the slip. Live, the corrector would also remove it from the context, but I haven't measured that.
• The detector misses "comprendi" and any tu form outside its word list. I didn't extend it, because tuning it on the first test's replies would overfit.
• This is about one model. In the first test GPT 6.1 sol and Claude didn't have the problem. I only tested the knowledge base written in tu, in Italian.

What I take from it

The fix that sounds right, cleaning the input, didn't work. The slip doesn't come from text the bot copies. It comes from a phrase the bot writes itself, and the input only makes it more likely. A check on each reply at the output catches it, and the deterministic guard makes it safe to let a model edit what the customer reads.

Writing the rule and the order before the run mattered. The knowledge base rewrite was first in line, and the rule rejected it at 2.0% because the bar was set at 0.5% before any number came in.

Sources

My own runs, 3 and 4 November 2026. Code, transcripts and labels are in a private repository.

Avete