> ## Documentation Index
> Fetch the complete documentation index at: https://www.plain.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Test changes to the Customer Agent

> Choose between the playground, test mode, and a staging workspace to check a change to the Customer Agent before customers see it.

[Custom instructions](https://www.plain.com/docs/product/agents/ari/custom-instructions.md) and [knowledge sources](https://www.plain.com/docs/product/agents/knowledge-sources.md) apply to every thread the Customer Agent handles in a workspace, so a change you save in production reaches customers on their next message. Three approaches let you check a change before that happens, from a single question in a private chat to a repeatable set of evals in a second workspace.

## Choose an approach

The approaches go from lightest to heaviest:

* **[Playground](https://www.plain.com/docs/product/agents/ari/playground.md)**: ask the Customer Agent one question at a time in a private chat. Use it to spot check an answer after you change something.
* **[Test mode](https://www.plain.com/docs/product/agents/ari/test-mode.md)**: the Customer Agent runs on your real threads and adds private timeline entries showing what it would have replied, without sending anything. Use it to see how your current setup behaves on real traffic before you assign the Customer Agent in live mode.
* **Staging workspace**: run the same set of test questions against a candidate version of your instructions and knowledge in a second workspace, through the API. Use it when the Customer Agent already replies to customers and you want repeatable evals of a change before it goes live.

The playground and test mode both use the instructions and knowledge saved in the workspace. To evaluate a new version without changing what customers get today, use a staging workspace.

## Run evals in a staging workspace

A staging workspace is a second Plain workspace that holds the version of your instructions and knowledge you want to test. Your eval script starts a chat thread for each test question, assigns it to the Customer Agent in live mode, and reads the Customer Agent's reply and agent status. Chat keeps the setup small: the Customer Agent replies on the thread, so you don't need to set up email in the staging workspace.

**1. Set up the staging workspace**

Create a second Plain workspace and add the knowledge sources and custom instructions you want to test. The workspace needs [live chat](https://www.plain.com/docs/product/channels/chat.md) and the [headless portal](https://www.plain.com/docs/headless-portal.md) on its plan, and [credits](https://www.plain.com/docs/product/billing/credits.md), because the Customer Agent uses credits in live mode. Enterprise customers can ask Plain to set this up on a staging workspace.

**2. Create a machine user and API key**

In the staging workspace, [create a machine user and an API key](https://www.plain.com/docs/graphql/authentication.md). The steps below need these permissions:

* `customer:create` and `customer:edit`
* `thread:create`, `thread:read`, and `thread:assign`
* `chat:create` and `timeline:read`

If a call fails for a missing permission, the error names it.

**3. Set the candidate instructions**

Set the instructions you want to test with `updateSetting`, as shown in [set instructions with the API](https://www.plain.com/docs/product/agents/ari/custom-instructions.md#set-instructions-with-the-api). Setting them from your eval script keeps each run tied to the exact text it tested.

**4. Find the Customer Agent's machine user**

The Customer Agent is the machine user of type `AI_AGENT` that isn't a [connected agent](https://www.plain.com/docs/product/agents/bring-your-own-agent.md). Look up its ID once and reuse it for every assignment.

**5. Start a chat thread for each test case**

For each test question, create a customer with [`upsertCustomer`](https://www.plain.com/docs/graphql/customers/upsert), [create a thread](https://www.plain.com/docs/graphql/threads/create.md) for that customer with `channel: CHAT`, then send the question as the customer with `sendCustomerChat`.

**6. Assign the thread to the Customer Agent in live mode**

Call [`assignThread`](https://www.plain.com/docs/graphql/threads/assignment) with the Customer Agent's machine user ID and `agentMode: LIVE`.

**7. Read the result**

Poll the thread until `agentStatus` is `HANDLED` or `HANDED_OFF`, or the Customer Agent is no longer assigned, then read the reply from the timeline.

**8. Apply the change to production**

When the evals pass, copy the same custom instructions and knowledge sources to your production workspace.

### Find the Customer Agent's machine user ID

This query returns the Customer Agent's machine user by filtering out connected agents:

```graphql theme={null}
query {
  machineUsers(filters: { type: AI_AGENT, isCustomAgent: false }) {
    edges { node { id publicName } }
  }
}
```

### Start and assign a test thread

These four mutations create a test customer, start a chat thread, send the test question as the customer, and assign the thread to the Customer Agent in live mode:

```graphql theme={null}
mutation upsertEvalCustomer($customer: UpsertCustomerInput!) {
  upsertCustomer(input: $customer) {
    customer { id }
    error { message }
  }
}

mutation createEvalThread($thread: CreateThreadInput!) {
  createThread(input: $thread) {
    thread { id }
    error { message }
  }
}

mutation sendEvalQuestion($chat: SendCustomerChatInput!) {
  sendCustomerChat(input: $chat) {
    chat { id }
    error { message }
  }
}

mutation assignToCustomerAgent($assign: AssignThreadInput!) {
  assignThread(input: $assign) {
    thread { id }
    error { message }
  }
}
```

Each mutation takes one input. Send the question before you assign the thread, so the Customer Agent has a customer message to answer:

```json theme={null}
{
  "customer": {
    "identifier": { "emailAddress": "eval-1@example.com" },
    "onCreate": {
      "fullName": "Eval customer 1",
      "email": { "email": "eval-1@example.com", "isVerified": true }
    },
    "onUpdate": {}
  },
  "thread": {
    "customerIdentifier": { "customerId": "c_01H14DFQ4PDYBH398J1E99TWSS" },
    "title": "Eval: reset password",
    "channel": "CHAT"
  },
  "chat": {
    "customerId": "c_01H14DFQ4PDYBH398J1E99TWSS",
    "threadId": "th_01H8H46YPB2S4MAJM382FG9423",
    "text": "How do I reset my password?"
  },
  "assign": {
    "threadId": "th_01H8H46YPB2S4MAJM382FG9423",
    "machineUserId": "mu_01H8H46YPB2S4MAJM382FG9423",
    "agentMode": "LIVE"
  }
}
```

### Read the Customer Agent's result

This query returns the thread's agent status, its assignee, and the chat messages on its timeline:

```graphql theme={null}
query ($threadId: ID!) {
  thread(threadId: $threadId) {
    agentStatus
    agentStatusDetail {
      ... on AgentStatusDetailHandedOff { reason }
    }
    assignedTo { ... on MachineUser { id } }
    timelineEntries(filters: { entryTypes: [CHAT] }, first: 10) {
      edges { node {
        actor { __typename }
        entry { ... on ChatEntry { text } }
      } }
    }
  }
}
```

`agentStatus` is empty until the Customer Agent picks up the thread, then `IN_PROGRESS` while it works. The run has finished when one of these is true:

* **`agentStatus` is `HANDLED`**: the Customer Agent replied. Its reply is the `ChatEntry` with a `MachineUserActor`
* **`agentStatus` is `HANDED_OFF`**: the Customer Agent passed the thread to your team. `agentStatusDetail.reason` holds a code such as `KNOWLEDGE_GAP`, `TALK_TO_HUMAN_REQUESTED`, or `NO_RESPONSE_NEEDED`, and any hand-off message is a `ChatEntry` with a `MachineUserActor`
* **The Customer Agent is no longer assigned and `agentStatus` hasn't changed**: the Customer Agent stopped without replying, for example after an error. It leaves a note on the thread saying why

Set a timeout on polling: the Customer Agent retries some failures before it gives up. Compare the reply or the hand-off reason against what you expected for that test case.

## Caveats when running evals

Three things decide whether an eval gets a reply:

* **Run the script with a machine user's API key**: `sendCustomerChat` only accepts a machine user, so you can't send test questions through the [Plain MCP server](https://www.plain.com/docs/integrations/mcp-server.md), which acts as the signed in user.
* **Send the question as a chat**: a thread started with `createThread` components gets its reply by email. Until you [set up email](https://www.plain.com/docs/product/channels/email.md) in the staging workspace, the Customer Agent can't send that reply, so the thread gets no reply and `agentStatus` stays `IN_PROGRESS`.
* **Keep credits in the staging workspace**: when the workspace runs out of credits, the Customer Agent hands off with the reason `OUT_OF_CREDITS` instead of answering.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.