Test changes to the Customer Agent

Custom instructions and knowledge sources apply to every thread the Customer Agent handles in a workspace, so a change you save in production reaches customers on their next message. Three approaches let you check a change before that happens, from a single question in a private chat to a repeatable set of evals in a second workspace.

Choose an approach

The approaches go from lightest to heaviest:

Playground: ask the Customer Agent one question at a time in a private chat. Use it to spot check an answer after you change something.
Test mode: the Customer Agent runs on your real threads and adds private timeline entries showing what it would have replied, without sending anything. Use it to see how your current setup behaves on real traffic before you assign the Customer Agent in live mode.
Staging workspace: run the same set of test questions against a candidate version of your instructions and knowledge in a second workspace, through the API. Use it when the Customer Agent already replies to customers and you want repeatable evals of a change before it goes live.

The playground and test mode both use the instructions and knowledge saved in the workspace. To evaluate a new version without changing what customers get today, use a staging workspace.

Run evals in a staging workspace

A staging workspace is a second Plain workspace that holds the version of your instructions and knowledge you want to test. Your eval script starts a chat thread for each test question, assigns it to the Customer Agent in live mode, and reads the Customer Agent’s reply and agent status. Chat keeps the setup small: the Customer Agent replies on the thread, so you don’t need to set up email in the staging workspace.

Set up the staging workspace

Create a second Plain workspace and add the knowledge sources and custom instructions you want to test. The workspace needs live chat and the headless portal on its plan, and credits, because the Customer Agent uses credits in live mode. Enterprise customers can ask Plain to set this up on a staging workspace.

Create a machine user and API key

In the staging workspace, create a machine user and an API key. The steps below need these permissions:

customer:create and customer:edit
thread:create, thread:read, and thread:assign
chat:create and timeline:read

If a call fails for a missing permission, the error names it.

Set the candidate instructions

Set the instructions you want to test with updateSetting, as shown in set instructions with the API. Setting them from your eval script keeps each run tied to the exact text it tested.

Find the Customer Agent's machine user

The Customer Agent is the machine user of type AI_AGENT that isn’t a connected agent. Look up its ID once and reuse it for every assignment.

Start a chat thread for each test case

For each test question, create a customer with upsertCustomer, create a thread for that customer with channel: CHAT, then send the question as the customer with sendCustomerChat.

Assign the thread to the Customer Agent in live mode

Call assignThread with the Customer Agent’s machine user ID and agentMode: LIVE.

Read the result

Poll the thread until agentStatus is HANDLED or HANDED_OFF, or the Customer Agent is no longer assigned, then read the reply from the timeline.

Apply the change to production

When the evals pass, copy the same custom instructions and knowledge sources to your production workspace.

Find the Customer Agent’s machine user ID

This query returns the Customer Agent’s machine user by filtering out connected agents:

query {
  machineUsers(filters: { type: AI_AGENT, isCustomAgent: false }) {
    edges { node { id publicName } }
  }
}

Start and assign a test thread

These four mutations create a test customer, start a chat thread, send the test question as the customer, and assign the thread to the Customer Agent in live mode:

mutation upsertEvalCustomer($customer: UpsertCustomerInput!) {
  upsertCustomer(input: $customer) {
    customer { id }
    error { message }
  }
}

mutation createEvalThread($thread: CreateThreadInput!) {
  createThread(input: $thread) {
    thread { id }
    error { message }
  }
}

mutation sendEvalQuestion($chat: SendCustomerChatInput!) {
  sendCustomerChat(input: $chat) {
    chat { id }
    error { message }
  }
}

mutation assignToCustomerAgent($assign: AssignThreadInput!) {
  assignThread(input: $assign) {
    thread { id }
    error { message }
  }
}

Each mutation takes one input. Send the question before you assign the thread, so the Customer Agent has a customer message to answer:

{
  "customer": {
    "identifier": { "emailAddress": "eval-1@example.com" },
    "onCreate": {
      "fullName": "Eval customer 1",
      "email": { "email": "eval-1@example.com", "isVerified": true }
    },
    "onUpdate": {}
  },
  "thread": {
    "customerIdentifier": { "customerId": "c_01H14DFQ4PDYBH398J1E99TWSS" },
    "title": "Eval: reset password",
    "channel": "CHAT"
  },
  "chat": {
    "customerId": "c_01H14DFQ4PDYBH398J1E99TWSS",
    "threadId": "th_01H8H46YPB2S4MAJM382FG9423",
    "text": "How do I reset my password?"
  },
  "assign": {
    "threadId": "th_01H8H46YPB2S4MAJM382FG9423",
    "machineUserId": "mu_01H8H46YPB2S4MAJM382FG9423",
    "agentMode": "LIVE"
  }
}

Read the Customer Agent’s result

This query returns the thread’s agent status, its assignee, and the chat messages on its timeline:

query ($threadId: ID!) {
  thread(threadId: $threadId) {
    agentStatus
    agentStatusDetail {
      ... on AgentStatusDetailHandedOff { reason }
    }
    assignedTo { ... on MachineUser { id } }
    timelineEntries(filters: { entryTypes: [CHAT] }, first: 10) {
      edges { node {
        actor { __typename }
        entry { ... on ChatEntry { text } }
      } }
    }
  }
}

agentStatus is empty until the Customer Agent picks up the thread, then IN_PROGRESS while it works. The run has finished when one of these is true:

agentStatus is HANDLED: the Customer Agent replied. Its reply is the ChatEntry with a MachineUserActor
agentStatus is HANDED_OFF: the Customer Agent passed the thread to your team. agentStatusDetail.reason holds a code such as KNOWLEDGE_GAP, TALK_TO_HUMAN_REQUESTED, or NO_RESPONSE_NEEDED, and any hand-off message is a ChatEntry with a MachineUserActor
The Customer Agent is no longer assigned and agentStatus hasn’t changed: the Customer Agent stopped without replying, for example after an error. It leaves a note on the thread saying why

Set a timeout on polling: the Customer Agent retries some failures before it gives up. Compare the reply or the hand-off reason against what you expected for that test case.

Caveats when running evals

Three things decide whether an eval gets a reply:

Run the script with a machine user’s API key: sendCustomerChat only accepts a machine user, so you can’t send test questions through the Plain MCP server, which acts as the signed in user.
Send the question as a chat: a thread started with createThread components gets its reply by email. Until you set up email in the staging workspace, the Customer Agent can’t send that reply, so the thread gets no reply and agentStatus stays IN_PROGRESS.
Keep credits in the staging workspace: when the workspace runs out of credits, the Customer Agent hands off with the reason OUT_OF_CREDITS instead of answering.

Agents

Channels

Platform

Integrations

Workflows

Broadcasts

Help center

Security

Plans & Billing

Test changes to the Customer Agent

Choose an approach

Run evals in a staging workspace

Find the Customer Agent’s machine user ID

Start and assign a test thread

Read the Customer Agent’s result

Caveats when running evals

​Choose an approach

​Run evals in a staging workspace

​Find the Customer Agent’s machine user ID

​Start and assign a test thread

​Read the Customer Agent’s result

​Caveats when running evals

Choose an approach

Run evals in a staging workspace

Find the Customer Agent’s machine user ID

Start and assign a test thread

Read the Customer Agent’s result

Caveats when running evals