In the previous chapter, we have written two unhappy paths for our newsletter_form. Now let's test that those paths work as we want them to.
# tests/test_stories.yml
# ... previous contents ...
- story: Chitchat during email
steps:
- user: |
I want to subscribe
intent: subscribe
- action: newsletter_form
- active_loop: newsletter_form
- user: |
How's the weather?
intent: chitchat
- action: utter_ask_continue
- user: |
Yes, I do
intent: affirm
- action: newsletter_form
- active_loop: null
- action: utter_subscribed
- story: Chitchat during frequency
steps:
- user: |
I want to subscribe
intent: subscribe
- action: newsletter_form
- active_loop: newsletter_form
- user: |
my email is [example@email.com](email)
intent: inform_email
- action: newsletter_form
- user: |
How's the weather?
intent: chitchat
- action: utter_ask_continue
- user: |
Yes, I do
intent: affirm
- action: newsletter_form
- active_loop: null
- action: utter_subscribed
If you run the tests, they will all pass.
But if we add another test, testing both interruption during email and interruption during frequency, we will see it fail.
We can see why it failed in results/failed_test_stories.yml.
# results/failed_test_stories.yml
version: "2.0"
stories:
- story: Newsletter form two interruptions (/var/folders/ng/z4dw_m9d1bz6m08vw7nlwwwh0000gq/T/tmpqlha244q/7bc3772cad9348a589e105ce91f29c74_test_stories.yml)
steps:
- intent: subscribe
- action: newsletter_form
- active_loop: newsletter_form
- intent: chitchat
- action: utter_ask_continue
- intent: affirm
- action: newsletter_form
- slot_was_set:
- email: example@email.com
- intent: chitchat
- action: utter_ask_continue
- intent: affirm
- action: newsletter_form # predicted: action_two_stage_fallback
- active_loop: null
- action: utter_subscribed
To fix this, start an interactive chat – rasa interactive.
Throughout the chat, I said what the correct actions shuld be. At the end of the chat, I chose to save this chat to data/stories.yml.
Let's inspect the story that has been created for us.
# data/stories.yml
# ... previous content ...
- story: interactive_story_1
steps:
- intent: subscribe
- action: newsletter_form
- active_loop: newsletter_form
- slot_was_set:
- requested_slot: email
- intent: chitchat
- action: utter_ask_continue
- intent: affirm
- action: newsletter_form
- slot_was_set:
- requested_slot: email
- slot_was_set:
- email: example@email.com
- slot_was_set:
- email: example@email.com
- slot_was_set:
- requested_slot: frequency
- intent: chitchat
- action: utter_ask_continue
- intent: affirm
- action: newsletter_form
- slot_was_set:
- requested_slot: frequency
- slot_was_set:
- frequency: twice a week
- slot_was_set:
- requested_slot: null
- active_loop: null
- action: utter_subscribed
Now let's look at new kind of steps that we have not been working with before.
This step is saying, that email is currently being requested:
- slot_was_set:
- requested_slot: email
This step is saying, that email has been filled with value example@email.com. We do not need this step for our test to pass.
- slot_was_set:
- email: example@email.com
This step is saying, that frequency has been filled with value twice a week. We do not need this step for our test to pass either.
- slot_was_set:
- frequency: twice a week
This step means, that no more slots are being requested.
- slot_was_set:
- requested_slot: null
You can learn more about requested_slot in my next article or in the documentation.
After droping the unnecessary steps, we get a bit simpler story.
# data/stories.yml
# ... previous content ...
- story: interactive_story_1
steps:
- intent: subscribe
- action: newsletter_form
- active_loop: newsletter_form
- slot_was_set:
- requested_slot: email
- intent: chitchat
- action: utter_ask_continue
- intent: affirm
- action: newsletter_form
- slot_was_set:
- requested_slot: email
- slot_was_set:
- requested_slot: frequency
- intent: chitchat
- action: utter_ask_continue
- intent: affirm
- action: newsletter_form
- slot_was_set:
- requested_slot: frequency
- active_loop: null
- action: utter_subscribed
I also reverted changes that have been made to domain.yml and data/nlu.yml and kept only the generated story.
If we retrain the model and run tests again, they will all pass now.
You can learn more about testing unhappy paths in the documentation.
In the next chapter, we will look at how to communicate with our chatbot through REST API.
Repository for this tutorial:
You can checkout the state of the repository at the end of this tutorial by running:
git clone --branch 06-testing-unhappy-paths git@github.com:petr7555/rasa-dev-tutorial.git
Top comments (2)
Hey Petr,
thanks for sharing your rasa testing know how, it is really appreciated.
I am also around to set up regression testing for our rasa implementations. I have really to admit that we are really struggling to get reliable results. I addressed this also in the rasa forum, but no feedback at all from the community.
My main issue: I get failed stories reported without an # marker.
-> How to interprete this ?
Here is what I wrote into the forum:
forum.rasa.com/t/end-to-end-story-...
Best
Andre
Hi Andre,
I'd suggest including how your tests look and what the results of the tests are in the forum. Maybe someone will be able to help then.