The result to aim for
Alan, whose IA officers process part of its members' requests, compared two ways of calling the model by an A/B test in production rather than relying on the numbers announced by the supplier.
What was done
In an article published on June 23, 2026 on his technical blog, the team from Alan explains that they tested the API Responses of OpenAI against the old API Chat Complements on two of its support agents, both based on GPT-5.1. Each contact was randomly assigned to either version; The main indicator was the rate of automation, i.e. the proportion of resolved contacts without transfer to a human. Announced result: a statistically significant 17% relative increase well above the 5% put forward by OpenAI on a public test bench. The advanced explanation: the new API allows to maintain the reasoning of the model from one turn to another, including in non-storage mode at the supplier thanks to an encrypted return reasoning. The team points out that its agents are very forced (there are few loops per turn, an inconsistent response is enough to switch to a human) and that the published earnings vary greatly depending on the conditions. It also notes that this API is specific to OpenAI.
Before you start
Tool cited by the source: OpenAI API; other tools cited: GPT-5.1. Fill in the bracketed items: [version A], [version B], [volume]. Strip confidential data that is not needed and check your tool’s access terms. Features and plans can change.
The method, in three steps.
Gather the context
Spot the bracketed items in the prompt (
[version A],[version B],[volume]) and prepare what they ask for.Adapt the prompt
Copy the prompt below, replace the bracketed items, then paste it into the AI assistant you use. The cited tool (OpenAI API) is not necessarily available to the public..
Reread before using
Reread the result, correct it if needed and keep the final decision. Caveats are detailed below.
Your starting point.
Replace the bracketed items, then adapt the proposal to your situation. This prompt is written by the iadujour desk It is not a quotation from the source.
I want to compare two versions of our support agent: [version A] and [version B]. Help me prepare the test: 1. The main indicator (e.g. proportion of requests resolved humanely) and 2 indicators of custody (satisfaction, claims). 2. The random allocation rule and the minimum time to conclude, with the volume of contacts I have: [volume]. 3. Which would invalidate the result. We'll measure it.
What deserves your attention
Proof level: Testimony published by an identified source. Source checked on 10 November 2026. Use described by the source, not tested by the desk. Original summary written by iadujour.fr: figures and claims are the source’s; the prompt is a desk proposal, not a quotation.
Comments.
A field note, a question, a precision on this use case? Comments are reviewed before publication.
Published comments
Loading comments…
Leave a comment
Moderation charter. Every comment is reviewed before publication. We refuse hate speech, insulting or discriminatory language, advertising, spam and promotional links, off-topic posts and other people’s personal data.