DEV Community

Cover image for RASA - Testing unhappy paths
Petr Janik
Petr Janik

Posted on • Edited on

RASA - Testing unhappy paths

#rasa #chatbot #nlp #ai

In the previous chapter, we have written two unhappy paths for our newsletter_form. Now let's test that those paths work as we want them to.

# tests/test_stories.yml
# ... previous contents ...
  - story: Chitchat during email
    steps:
      - user: |
          I want to subscribe
        intent: subscribe
      - action: newsletter_form
      - active_loop: newsletter_form
      - user: |
          How's the weather?
        intent: chitchat
      - action: utter_ask_continue
      - user: |
          Yes, I do
        intent: affirm
      - action: newsletter_form
      - active_loop: null
      - action: utter_subscribed

  - story: Chitchat during frequency
    steps:
      - user: |
          I want to subscribe
        intent: subscribe
      - action: newsletter_form
      - active_loop: newsletter_form
      - user: |
          my email is [example@email.com](email)
        intent: inform_email
      - action: newsletter_form
      - user: |
          How's the weather?
        intent: chitchat
      - action: utter_ask_continue
      - user: |
          Yes, I do
        intent: affirm
      - action: newsletter_form
      - active_loop: null
      - action: utter_subscribed
Enter fullscreen mode Exit fullscreen mode

If you run the tests, they will all pass.

But if we add another test, testing both interruption during email and interruption during frequency, we will see it fail.

Failed test

We can see why it failed in results/failed_test_stories.yml.

# results/failed_test_stories.yml
version: "2.0"
stories:
- story: Newsletter form two interruptions (/var/folders/ng/z4dw_m9d1bz6m08vw7nlwwwh0000gq/T/tmpqlha244q/7bc3772cad9348a589e105ce91f29c74_test_stories.yml)
  steps:
  - intent: subscribe
  - action: newsletter_form
  - active_loop: newsletter_form
  - intent: chitchat
  - action: utter_ask_continue
  - intent: affirm
  - action: newsletter_form
  - slot_was_set:
    - email: example@email.com
  - intent: chitchat
  - action: utter_ask_continue
  - intent: affirm
  - action: newsletter_form  # predicted: action_two_stage_fallback
  - active_loop: null
  - action: utter_subscribed
Enter fullscreen mode Exit fullscreen mode

To fix this, start an interactive chat – rasa interactive.

Throughout the chat, I said what the correct actions shuld be. At the end of the chat, I chose to save this chat to data/stories.yml.
Let's inspect the story that has been created for us.

# data/stories.yml
# ... previous content ...
  - story: interactive_story_1
    steps:
      - intent: subscribe
      - action: newsletter_form
      - active_loop: newsletter_form
      - slot_was_set:
          - requested_slot: email
      - intent: chitchat
      - action: utter_ask_continue
      - intent: affirm
      - action: newsletter_form
      - slot_was_set:
          - requested_slot: email
      - slot_was_set:
          - email: example@email.com
      - slot_was_set:
          - email: example@email.com
      - slot_was_set:
          - requested_slot: frequency
      - intent: chitchat
      - action: utter_ask_continue
      - intent: affirm
      - action: newsletter_form
      - slot_was_set:
          - requested_slot: frequency
      - slot_was_set:
          - frequency: twice a week
      - slot_was_set:
          - requested_slot: null
      - active_loop: null
      - action: utter_subscribed
Enter fullscreen mode Exit fullscreen mode

Now let's look at new kind of steps that we have not been working with before.

This step is saying, that email is currently being requested:

      - slot_was_set:
          - requested_slot: email
Enter fullscreen mode Exit fullscreen mode

This step is saying, that email has been filled with value example@email.com. We do not need this step for our test to pass.

      - slot_was_set:
          - email: example@email.com
Enter fullscreen mode Exit fullscreen mode

This step is saying, that frequency has been filled with value twice a week. We do not need this step for our test to pass either.

      - slot_was_set:
          - frequency: twice a week
Enter fullscreen mode Exit fullscreen mode

This step means, that no more slots are being requested.

      - slot_was_set:
          - requested_slot: null
Enter fullscreen mode Exit fullscreen mode

You can learn more about requested_slot in my next article or in the documentation.

After droping the unnecessary steps, we get a bit simpler story.

# data/stories.yml
# ... previous content ...
  - story: interactive_story_1
    steps:
      - intent: subscribe
      - action: newsletter_form
      - active_loop: newsletter_form
      - slot_was_set:
          - requested_slot: email
      - intent: chitchat
      - action: utter_ask_continue
      - intent: affirm
      - action: newsletter_form
      - slot_was_set:
          - requested_slot: email
      - slot_was_set:
          - requested_slot: frequency
      - intent: chitchat
      - action: utter_ask_continue
      - intent: affirm
      - action: newsletter_form
      - slot_was_set:
          - requested_slot: frequency
      - active_loop: null
      - action: utter_subscribed
Enter fullscreen mode Exit fullscreen mode

I also reverted changes that have been made to domain.yml and data/nlu.yml and kept only the generated story.

If we retrain the model and run tests again, they will all pass now.

Passing tests

You can learn more about testing unhappy paths in the documentation.

In the next chapter, we will look at how to communicate with our chatbot through REST API.

Repository for this tutorial:

GitHub logo petr7555 / rasa-dev-tutorial

You can checkout the state of the repository at the end of this tutorial by running:

git clone --branch 06-testing-unhappy-paths git@github.com:petr7555/rasa-dev-tutorial.git
Enter fullscreen mode Exit fullscreen mode

Top comments (2)

Collapse
 
cobrainersadmin profile image
Co-brainers-admin •

Hey Petr,

thanks for sharing your rasa testing know how, it is really appreciated.

I am also around to set up regression testing for our rasa implementations. I have really to admit that we are really struggling to get reliable results. I addressed this also in the rasa forum, but no feedback at all from the community.

My main issue: I get failed stories reported without an # marker.

-> How to interprete this ?

Here is what I wrote into the forum:

forum.rasa.com/t/end-to-end-story-...

Best

Andre

Collapse
 
petr7555 profile image
Petr Janik •

Hi Andre,
I'd suggest including how your tests look and what the results of the tests are in the forum. Maybe someone will be able to help then.

Read next

danyaleyman profile image

Using AI for Real-Time Customer Sentiment Tracking

Daniel -

ashinno profile image

Best LLM for Coding and Developers in 2025

Ash Inno -

sunderkumar profile image

Speech to Text using Assembly AI

Sunder Kumar -

gopinath_v_c93227e4855f0d profile image

Assemble AI Challenge

Gopinath V -