When To Use Replay Testing
Replay testing is best for:- validating an agent against real customer phrasing
- checking multi-turn context handling
- testing fixes against a specific historical ticket
- reviewing how a changed agent would respond to an old conversation
- turning real examples into candidates for batch tests
Supported Sources
Replay testing can fetch conversations from connected sources that provide conversation history:
Duckie only shows sources that are connected for your organization.
Run A Replay Test
1
Open Replay Chats
Go to Test → Replay Chats.
2
Select an agent
Choose the active agent you want to test.
3
Select a connection
Choose the connected source to fetch conversations from.
4
Fetch conversations
Click Fetch Conversations. Duckie loads replayable conversations and splits them into turns.
5
Pick a conversation and turn
Select a conversation, then move through its turns with Prev Message and Next Message.
6
Generate Duckie's response
Click Generate Duckie Response to run the selected agent from that point in the conversation.
7
Compare the result
Compare the historical Expected response with the generated Duckie response.
Fetch A Specific Ticket
For Slack, Zendesk, HubSpot, and Intercom, you can fetch one conversation directly by ID.1
Select a supported connection
Choose Slack, Zendesk, HubSpot, or Intercom.
2
Enter the ticket ID
Paste the ticket, thread, or conversation ID into the Ticket ID field.
3
Fetch the ticket
Click Fetch Ticket. Duckie adds the replayable conversation if the source returns one.
How Turns Work
Duckie splits each conversation into turns:
Conversations without any historical agent response are filtered out because there is no expected response to compare against.
When you replay later turns, Duckie uses the prior customer messages and prior agent responses as context. If you already generated Duckie responses for earlier turns, those generated responses become the prior assistant context for later turns.
Use Context Settings
Replay tests use the context panel to simulate source metadata for the run. Use the context panel to:- confirm the source Duckie should treat the conversation as coming from
- add source-specific fields when needed
- review internal-note and write-action settings for the run
Test Mode Safety
Replay tests run in testing mode, the same safety layer that protects playground and batch test runs from unintended external side effects. In testing mode:- write app tools, custom tools, and MCP tools are skipped
- responder output is converted to internal notes where the source supports internal notes
- Slack and Discord responder delivery is skipped because they do not have an internal note target
Review Replay History
The Replay Chats History panel shows replay runs created from the Replay Chats page. From history, you can:- load a previous replay run
- open the run details drawer
- inspect steps, tool calls, and generated messages
- delete replay runs you no longer need
Turn Replays Into Regression Tests
Replay testing is useful for finding examples worth preserving. When a replay exposes behavior you want to keep checking, add the scenario to a batch test.1
Find a useful replay
Replay a real conversation and identify the turn you want to preserve.
2
Open Batch Test
Go to Test → Batch Test and open or create a batch.
3
Add the scenario
Use Manage tickets to add a custom ticket with the relevant customer messages and expected response.
4
Run the batch
Run the batch whenever you need repeatable validation.
Next Steps
Playground
Test new scenarios interactively
Batch Testing
Create repeatable regression suites