Shadow Mode: Test an AI Agent Before It Talks to Customers

The hardest question about an AI support agent is not “can it answer?” It is “will it answer our customers the way we would?” A demo cannot tell you that. Your own conversations can.

Shadow mode is how you find out without risk. The AI works on real conversations, but everything it writes stays inside your help desk as a private note. Customers never see it. This guide explains how shadow mode works in Cliedesk, what to check while it runs, and how to move from shadow to live.

What shadow mode is

In shadow mode, the AI agent reads each new conversation and works out its answer from your help articles and the customer’s orders and profile. Then, instead of sending the answer, it saves it as a private note: an AI draft.

Your team keeps answering customers as usual. Over a few days or weeks you build up pairs: what the AI would have said, and what your team actually said.

That comparison is the most honest test you can run, because it uses the questions your customers really ask, in the words they really use, including the messy ones nobody thought to put in a test script.

The AI features, including the AI agent and shadow mode, are part of Cliedesk Pro.

Why test on real conversations

A launch checklist usually includes “try some questions”. That is a good start, but it has limits:

• You test the questions you already expect. Customers ask the ones you do not.
• You write tidy questions. Customers write half a sentence, paste a screenshot, or mention “the blue mug from last week”.
• You test the knowledge you remember to check. Shadow mode finds the help articles that are out of date.

Shadow mode covers those gaps, and nobody has to explain an awkward AI reply to a customer.

Start in the test lab

Shadow mode watches live traffic. The AI test lab is for the questions you choose. Use both. Open Settings → AI → Test lab:

• Preview: write what a customer says, or give a conversation ID to replay it. You see the answer, the actions the AI would take and a link to its trace. Actions such as refunds are dry runs: nothing changes.
• Test sets: add questions, paste a CSV with question and expected columns, or take questions from resolved conversations. Run the set in the background.
• Grades: each answer gets Good, Acceptable or Poor, with a reason, a cause and a fix to try.
• Compare models: run up to 3 models side by side and compare answers, grades, speed and cost.
• Regression sets run again when knowledge, policies or models change. If answers that passed now fail, AI changes are not published automatically until an admin confirms the run.

Tests never count in your reports, and they are labelled “test” in AI cost.

How shadow mode works in Cliedesk

It is the default

On a new site, the AI agent is in shadow mode on every channel. The guided setup in Settings → AI → AI agent walks you through knowledge, behaviour, actions, testing and deploying, but it never takes the agent live. It stays in shadow mode until you change it. See AI agent setup.

Nothing is sent, and actions only run as tests

In shadow mode, the AI writes drafts and nothing reaches customers. Playbooks follow the same rule: in shadow mode nothing is sent and actions only run as tests.

You see how good the drafts were

Settings → AI → Shadow and rollout shows how often agents sent the AI’s draft unchanged. On Settings → AI → AI agent deploy, each channel row shows how good the shadow drafts were and a recommended next mode.

Live mode keeps its own safety rules

Shadow mode is not the only safety net. When the agent is live, a weak answer is never sent. Protected topics such as self-harm, threats and high-risk legal, medical or financial advice always go to a person. Exclusion rules keep the AI out of conversations you choose, such as VIPs or partners. Shadow mode is your chance to check that these rules and your own escalation rules catch the conversations you expect.

What to check while the agent runs in shadow

Make a habit of reviewing drafts for a few minutes each day. Look for patterns, not single mistakes.

Accuracy

Is the answer correct? Does it match your policy, your prices and your return window? If it is wrong, open the trace: it shows the sources and policies behind the answer, which usually points straight at the cause.

Grounding

Did the AI find the right article? A wrong answer often means a missing or outdated article, not a bad model. Fix the article and the next draft improves.

Tone and length

Does it sound like your brand? In Settings → AI → AI agent you set the tone, answer length per channel, greeting, sign-off and words to avoid.

Handoffs

Did it hand over the conversations your team would want to handle? Check the handoff reasons. If something should always go to a person, write an escalation rule in plain words, for example “the customer mentions a chargeback or a lawyer”. See AI handoff and review.

Actions

Did it prepare the right action for the right order, with the right amount? In shadow mode actions only run as tests, so this is the safest time to check.

Turn mistakes into fixes

Shadow mode is most useful when every weak draft becomes a fix.

In a conversation, open the AI answers section of the side panel. Rate an answer Good or Bad with a reason, or click Improve and write the correct answer. Choose what it becomes: a draft snippet, an article update, a policy or a playbook fix. Nothing changes until it is approved, and Re-test with this fix replays the conversation so you can see the new answer.

For policy changes, Simulate on past conversations replays up to 20 recent AI conversations with your draft policy. Nothing is sent and nothing is saved. More in AI policies.

From shadow to live, one step at a time

You do not have to jump from silent to fully live. Each channel has its own deploy mode:

1. Shadow: private drafts, nothing sent.
2. Suggest: the draft also goes into the assigned agent’s reply box. The agent sends, edits or deletes it. This is a good middle step: customers only see AI text that a person approved.
3. Live with a share of conversations: in Settings → AI → Shadow and rollout, a Staged rollout lets the AI answer a share of conversations, for example 10 %. The report compares the AI group with the others.
4. Live: the AI answers customers on that channel.

You can also limit the AI to after business hours or to business hours only. Start with the channel where questions are simplest, often live chat for order status, and leave the rest in shadow. The deploy modes guide explains each option.

What about money actions?

Shadow mode also unlocks a specific setting. Refunds, credits, coupons and reships ask a person to approve every amount. An auto-approve amount can only be set after the AI profile has run in shadow mode. Even then, the customer confirms refunds, cancellations and order changes first.

A simple launch plan

• Week 1: set up knowledge and policies, run a test set in the test lab and fix what fails.
• Weeks 2 and 3: leave every channel in shadow. Review drafts daily and turn weak answers into fixes.
• Next: move your simplest channel to Suggest, then to a small share of live conversations.
• After that: go live on that channel and repeat for the next one.

The exact timing depends on your volume. The rule of thumb is simple: go live when your team is sending the drafts unchanged, not before. To learn more about the agent itself, see the AI agent page.