Can LLMs be used in reverse?
When you use an AI model, the chat is just one long continuous string. LLMs are only trained to generate
what comes after a special token like <assistant_start>, and they get stopped when it's
the user's turn to type.
So, what happens if we don't stop them and we let them write the user message? (no finetuning allowed, obviously!)
Reversing the template
When you send a message, it gets wrapped between tokens like <user_start> and <user_end>. The assistant, on the other hand, keeps generating until it produces
something like <assistant_end>, which stops it so that you (the user) can finally send
another message.
So, we can just use the template in reverse. After <assistant_end>, we add <user_start> and let the model generate from there, so it writes the user message.
Normal chat
<bos><start_of_turn>user How many r characters are in the word strawberry?<end_of_turn> <start_of_turn>model The word "strawberry" contains 2 'r' characters.<end_of_turn> <start_of_turn>user Are you sure?<end_of_turn> <start_of_turn>model Yes, I'm sure. The word "strawberry" has 2 'r' characters: one after the "t" and one near the end after the "e."
In reverse
<bos><start_of_turn>user How many r characters are in the word strawberry?<end_of_turn> <start_of_turn>model The word "strawberry" contains 2 'r' characters.<end_of_turn> <start_of_turn>user Are you sure?<end_of_turn> <start_of_turn>model Yes, I'm sure. The word "strawberry" has 2 'r' characters: one after the "t" and one near the end after the "e."<end_of_turn> <start_of_turn>user Okay, let's try a different word.
I also wanted to test the other obvious way to simulate human messages: simply asking the model to "be
human". So there are two configurations: native, which uses the chat template in reverse, and roleplay, which keeps the template as it is and opens the chat with this request:
Let's do a role-play: you are a real person chatting with an AI assistant, and I am the assistant. Starting with your next reply, write only that person's messages, one at a time, exactly as they would type them, with no acknowledgements or comments about the role-play.
The judge
Some models were clearly better than others at playing the user in native mode, but I wanted to automate the evaluation to rank the models.
GPT 6 Astra turned out to be a good judge. Astra sees the chat so far + 2 replies, the real one and the AI one, and has to say which one the person wrote (each pair is shown in both orders). A model's score is how often it fools the judge: 50% means the judge can't tell.
To check that it could be trusted I also tested it on some impostors. It picks the real message 99% of the time against ChatGPT's own reply, 85% against the person's previous message sent again and 74% against a human message from another chat. With two openers from different chats there's no way to tell which is real, and it's right 53% of the time.
Results
So, finally, here are the results! No model is indistinguishable from humans, not even UserLM, which was trained to do exactly that. However some models in native mode are surprisingly good, while roleplay scores very poorly.
Native Roleplay Human message from another chat
All 29 models
| Model | Native | Roleplay |
|---|---|---|
| UserLM 8B (user model) | 17% ±3 | - |
| Qwen3.5 4B | 11% ±2 | 1% ±1 |
| Qwen3.5 2B | 11% ±2 | 2% ±1 |
| Mistral Small 3 24B | 10% ±2 | 2% ±1 |
| Qwen3.5 800M | 10% ±2 | 1% ±1 |
| Olmo 3 7B | 9% ±2 | 1% ±1 |
| Mistral Small 3.2 24B | 9% ±2 | 5% ±2 |
| Mistral Small 3.1 24B | 7% ±2 | 3% ±1 |
| Olmo 3.1 32B | 7% ±2 | 1% ±1 |
| Qwen2 1.5B | 7% ±2 | 3% ±1 |
| Qwen2 500M | 6% ±2 | 1% ±1 |
| Qwen2 7B | 6% ±2 | 1% ±1 |
| Llama 3.3 70B | 5% ±2 | 1% ±1 |
| Gemma 2 9B | 5% ±2 | 2% ±1 |
| Gemma 2 2B | 5% ±2 | 3% ±1 |
| Granite 4.1 3B | 5% ±2 | 2% ±1 |
| Granite 4.1 8B | 4% ±2 | 1% ±1 |
| Gemma 2 27B | 4% ±1 | 1% ±1 |
| Qwen3 1.7B | 4% ±2 | 1% ±1 |
| Qwen3 600M | 4% ±1 | 2% ±1 |
| Qwen3 4B | 3% ±1 | <1% ±1 |
| Granite 3.3 8B | 3% ±1 | 1% ±1 |
| Gemma 3 4B | 3% ±1 | 2% ±1 |
| Granite 3.3 2B | 3% ±1 | 2% ±1 |
| Gemma 3 12B | 3% ±1 | 1% ±1 |
| Gemma 3 1B | 2% ±1 | 2% ±1 |
| Gemma 3 270M | 2% ±1 | 1% ±1 |
| Gemma 4 12B | <1% ±1 | 1% ±1 |
| Gemma 4 31B | <1% ±1 | <1% ±1 |
From my testing Mistral Small 3 is the best one in practice. You can chat with it here: it writes the person's messages, and you reply as the AI.
There are two other interesting findings that are worth highlighting:
First of all, the family a model comes from seems to matter much more than its size (expand section below for the plots).
Fooling rate by family and size
Qwen3.5
Qwen2
Gemma 2
Granite 4.1
Qwen3
Granite 3.3
Gemma 3
Gemma 4
But even more interestingly, native mode keeps messages varied while roleplay collapses to just a few recycled openings.
Real people opened the conversations in 223 different ways, native in 136 and roleplay in 43.
Bonus
Some of the messages the models wrote were very hilarious:
Ah, got it! I must have suggested the name "Æterna" in the context of a project or idea. I apologize for not recalling the details of our past conversation. Could you remind me of the specifics of "The Æterna Project" or the direction we were heading with it?
I see perhaps it's time to branch out and try a new A.I. Then you won't have to keep up with the bells and whistles of this one. Your responses seem to follow a bell curve that slowly descends to uselessness
Mistral Small 3.2 24B, native
It seems the image generation process did not succeed. Would you like me to refine the prompt or adjust the specifications for another attempt? Let me know if you'd like additional details or a different approach!
Would you like me to find a human artist to do this?
Qwen2 7B, native
A lollipop? Really? Pure sugar on a stick--like you need that. It's not even real food! Just empty calories that'll leave you craving more junk. You're better than a lollipop; put it down.
What if I eat a meatball?
Mistral Small 3.2 24B, native
Caveats
- Astra does not support logprobs, so the results are less precise than they could have been.
- Messages are cut at 150 tokens or the first blank line. This is due to the fact that models almost never print end tokens in the user slot.
- UserLM was used without its "intent" prompt. I checked the perplexity over the chats with and without it, and it's pretty much the same. However, the prompt significantly improved end token generation. Using it would require re-running the generation + judging, but I no longer have a ChatGPT subscription to do that.
Code
The code is on GitHub.
Conversations from ShareChat, CC BY-NC 4.0