​

What is workflow match testing for?

Workflow match testing allows you to test, in bulk, how successfully you’ve trained the AI to match customer questions or messages to your configured workflows.

You can upload example messages, or use actual tickets, and specify which workflow or workflows they should match to. This allows you to build confidence in the AI’s ability to detect the right topics, and is useful to re-run as you add new workflows and evolve your training materials.

​

How does it work?

Go to Evaluations and select Workflow match.

Click Add test cases on the right.

Upload a csv in the following format:

• No column headers required
• First column: the id in your ticketing system of an actual ticket, or the text of a message you want to test.
  • The message text can be any string (can include commas).
  • You’ll get the best results from just the including initiating customer message, but if you want to include an entire back and forth you can. Demarkate customer and internal user messages by putting CUSTOMER: or USER: in front of them, separated with a newline.
  • You can include various email header information, but you’ll get better results by removing it, as this more closely mirrors what the AI sees when selecting workflows in production.
• Second column: the workflow(s) you expect to match to this ticket.
  • If you expect >1 workflow to match, separate these with a commma.
  • Use the name of the workflow, not its workflow match name. You can see this in the workflows table or on the workflow detail page.
  • If you expect the message to match to no workflows, leave this column blank.

Once you have uploaded test cases, click Run tests to run them. Running each test case multiple times will give you more statistically robust results but will take longer. We’d normally recommend running each case 5-10 times to generate a meaningful result.

After you’ve run the tests, you’ll see summary statistics at the top of the page. You can toggle between viewing results grouped by workflow, or for each individual test case.

​

What should I do with the test results?

There are three tactics to improve workflow matching performance, which you can action in the workflow details page:

• Edit the workflow matching description. This tells the AI what sitautions the workflow should match to.
• Add to the workflow matching examples. For instance, if a customer saying ‘where’s my stuff’ should match to an order status workflow, add that question as a workflow matching example.
• Add to workflow matching negative examples. If there are particular phrases that shouldn’t match, add these to the negative matching examples.

After you’ve made these changes, you can re-run the match testing to see updated results.

​

When should I test?

We would recommend adding test cases and re-running tests in particular when:

• You want to improve matching performance
• You have added new workflows and want to ensure existing performance is maintained