What It Does
The feedback function:- Reviews runs with low confidence, tool failures, reviewer changes, or poor outcomes
- Groups recurring failures into categories
- Identifies missing knowledge, unclear guardrails, or weak workflow branches
- Proposes updates to agent instructions, workflows, tools, categories, or guidelines
- Reports improvement items to owners
- Measures whether changes improve future runs
Recommended Components
| Need | Duckie component |
|---|---|
| Review run history | Analytics runs |
| Group recurring issues | Categories and attributes |
| Inspect and propose configuration changes | Duckie Assistant feedback agent |
| Track improvements | Issue tracker or ticketing tool |
| Monitor recurring failures | Alerts and daily reports |
Feedback Loop
What to Track
Track categories and attributes that explain both business outcomes and system behavior:| Tracking item | Example values |
|---|---|
| Reason category | Fraud, duplicate, service not received, product not as described, credit not processed. |
| Automation path | Accepted, countered, needs account info, human review, manual handling. |
| Escalation reason | Low confidence, missing evidence, deadline risk, policy question, tool failure. |
| Reviewer decision | Approved, changes requested, rejected, escalated. |
| Outcome | Won, lost, accepted, withdrawn, closed, pending. |
| Failure type | Missing data, wrong classification, weak evidence, message quality, tool error. |
Review Cadence
Use three loops:| Loop | Cadence | Owner |
|---|---|---|
| Daily operations report | Business days | Duckie Assistant reporting agent and operations lead |
| Exception review | As cases happen | Slack escalation agent and reviewer |
| Improvement review | Weekly or biweekly | Duckie Assistant feedback agent and system owner |
Improvement Actions
Common improvements include:- Add a branch to the dispute handling workflow
- Split a broad reason category into more specific categories
- Add a missing evidence checklist
- Tighten an agent instruction
- Add a guardrail for a risky response pattern
- Add a custom tool for repeated manual lookup work
- Update the daily report to surface a recurring issue
The Duckie Assistant feedback agent should propose changes, but sensitive workflow, guardrail, or tool updates should follow your normal review process before going live.
Related Docs
Analytics
Monitor performance and trends.
Runs
Inspect the details behind each case.
Alerts
Notify the team when important patterns appear.
Replay Testing
Test changes against historical cases.