Writing intent tests for skills

The mycroft skills repo uses an automated testing framework to test the skills. This will test a submitted skill together with the deafault skills and should help identifying conflicts.

To help the skill team review the submitted skills you can create intent tests. An intent test is a test which takes an utterance as input and checks for an expected output.

For example the utterance "What's the capital of France" should generate the response "Paris".

This helps the skill team that the skill is working in the way intended by the skill author.

The intent test file

The above test case can be written in an intent.json test file as

{
  "utterance": "what's the capital of france",
  "expected_response": "paris"
}

"utterance" is the utterance that should be tested and "expected_response" is what mycroft should reply.

This file should be placed in the skills test/intent folder and the filename should end with .intent.json to be picked up by the test framework. test/intent/capitalOfFrance.intent.json may be a good name for the test above.

The tools

in the mycroft-core repository a couple of test scripts are available in the test/integrationtests/skills/ folder. To run these, make sure the virtualenvironement for mycroft is activated. This is done by running source .venv/bin/activate from the mycroft-core folder.

The test scripts are:

discover_test.py - A pytest script to run all tests in a skills folder. pytest discover_test.py /opt/mycroft/skills/ will find all skill tests in the /opt/mycroft/skills folder.

single_test.py - Run tests for a single skill. python single_test.py /opt/mycroft/skills/mycroft-datetime.mycroftai will run all tests for The datetime skill.

skill_developers_testrunner.py - script that can be copied to a skill folder and run any tests for that skill.

cd /opt/mycroft/your_new_skill
python skill_developers_testrunner.py

An example skill

Below is a simple skill to play my favourite c64 remixes from https://remix.kwed.org.

from mycroft import MycroftSkill, intent_handler
from mycroft.skills.audioservice import AudioService
from mycroft.audio import wait_while_speaking
from adapt.intent import IntentBuilder

track_dict = {
    'bomb jack': 'http://remix.kwed.org/files/RKOfiles/Chronblom%20-%20Bomb%20Jack%20subtune%206%20(violin%20version).mp3',
    'druid': 'http://remix.kwed.org/files/RKOfiles/Revel%20Craft%20-%20Druid.mp3',
    'crazy comets':  'http://remix.kwed.org/files/RKOfiles/Makke%20-%20Crazy%20Comets%20(Komet%20Non-Stop).mp3',
    'boulder dash': 'http://remix.kwed.org/files/RKOfiles/Mahoney%20-%20BoulderDash%20(Commodore%2069%20mix).mp3',
    'garfield': 'http://remix.kwed.org/files/RKOfiles/Reyn%20Ouwehand%20-%20Garfield.mp3'
}

class C64RemixSkill(MycroftSkill):
    """ Skill for playing Åke's favorite C64 Remixes. """
    def initialize(self):
        """ Connect to the mycroft audio service. """
        self.audio_service = AudioService(self.emitter)

    @intent_handler(IntentBuilder('').require('Play').require('Best') \
                    .require('Commodore'))
    def handle_best(self, message):
        self.speak_dialog('PlayBest')
        wait_while_speaking()
        self.audio_service.play(track_dict['garfield'])

    @intent_handler(IntentBuilder('').require('Play').require('All') \
                    .require('Commodore'))
    def handle_all(self, message):
        # Make a list of all track urls
        tracks = list(track_dict.values())
        self.speak_dialog('PlayAll')
        wait_while_speaking()
        self.audio_service.play(tracks)


def create_skill():
    return C64RemixSkill()

The entire source with dialog files and vocab files can be found here.

It basically has two commands, "play the best Commodore remix" and "play the top Commodore remixes". The first plays a single track and the second enqueues all five songs and starts playing.

We can write the following two test cases:

test/intent/play.best.intent.json:

{
  "utterance": "play the best commodore remix",
  "expected_dialog": "PlayBest"
}

and

``test/intent/play.top.intent.json`:

{
  "utterance": "play the top five commodore remixes",
  "expected_dialog": "PlayAll"
}

Pretty similar to the first basic example but the "expected_dialog" entry is new. This is similar to the "expected_response" but instead of taking a sentence it takes a dialog file. The test will check that the spoken sentence by mycroft is found in the specified dialog file. for PlayAll this will match against the dialog/en-us/PlayAll.dialog file. It contains

Playing the top commodore 64 remixes
Playing my favorite c64 remixes

expected_dialog also allows for parametrized dialogs like in the datetime skill where one of the dialogs contain It is {{time}}. In this case the script will accept a spoken reply of "It is " followed by anything.

Ok now we have our basic tests, let's run them:

 ~/projects/python/mycroft-core$ source .venv/bin/bash
(.venv) ~/projects/python/mycroft-core$ cd test/integrationtests/skills
(.venv) ~/projects/python/mycroft-core/test/integrationtests/skills$ python single_test.py /opt/mycroft/skills/best-c64-remixes

This produces a lot of output ending in

----------------------------------------------------------------------
Ran 2 tests in 4.829s

OK

If a test should fail the output would be something along the lines of

----------------------------------------------------------------------
Ran 2 tests in 4.568s

FAILED (failures=1)

With the tests passing for our skill we can run it together with all skills to make sure our skill doesn't interfere. To do this run

(.venv) ~/projects/python/mycroft-core/test/integrationtests/skills$ pytest discover_test.py /opt/mycroft/skills

and pytest will scan for tests and then start running the them:

============================= test session starts ==============================
platform linux -- Python 3.5.2, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /, inifile:
plugins: cov-2.5.1
collected 84 items                                                             

discover_tests.py ...................................................... [ 65%]
...........................                                              [100%]
=================== 82 passed in 352.01 seconds ====================

Mocking your tests

For cases where the skill needs to access an online resource or require logins or something it might be troublesome getting the skill