Guide

Writing new tests

Name of the test

Each test comes in its own directory, containing all its test steps. The name of the test is defined by the name of this directory.

The name of the test should be short, but expressive. The format for naming a test is currently <test ID>_<short description>.

The <test ID> is the serial number of the test as defined in the Test Plan document. The <short description> is exactly that, a short description of what happens in the test.

Name of the test steps

Each test step is a unique YAML file within the test's directory. The name of the step is defined by its file name.

The test steps must be named XX-<name>.yaml. This is a kuttl convention and cannot be overriden. XX is a number (prefixed with 0, so step 1 must be 01), and <name> is a free form value for the test step.

There are two reserved words you cannot use for <name>:

• assert contains positive assertions (i.e. resources that must exist) and
• errors contains negative assertions (i.e. resources that must not exist)

Refer to the kuttl documentation for more information.

Documentation

Documentation is important, even for tests. You can should provide inline documentation in your YAML files (using comments) and a README.md in your test case's directory. The README.md should provide some context for the test case, e.g. what it tries to assert for under which circumstances. This will help others in troubleshooting failing tests.

Recipes

kuttl unfortunately neither encourages or supports re-use of your test steps and assertions yet.

Generally, you should try to use assert and errors declaration whenever possible and viable. For some cases, you may need to use custom scripts to get the results you are looking for.

Scripts general

Scripts can be executed in a kuttl.dev/TestStep resources from a usual test step declaration.

Your script probably will retrieve some information, and asserts it state. If the assertion fails, the script should exit with a code > 0, and also print some information why it failed, e.g.

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
    # Get some piece of information...
    if test "$result" != "expected"; then
      echo "Expectation failed, should 'expected', is '$result'"
      exit 1
    fi

Also, you may want to use set -e and set -o pipefail at the top of your script to catch unexpected errors as test case failures, e.g.

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
    set -e
    set -o pipefail
    # rest of your script

Getting values of a resource's environment variables

YAML declarations used in assert or errors files unfortunately don't handle arrays very well yet. You will always have to specify the complete expectation, i.e. the complete array.

If you are just interested in a certain variable, and don't care about the rest, you can use a script similar to the following using jq. E.g. to get the value of a variable named FOO for the argo-rollouts deployment in the test's namespace:

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
    val=$(kubectl get -n $NAMESPACE deployments argo-rollouts -o json \
      | jq -r '.spec.templates.spec.containers[0].env[]|select(.name=="FOO").value')
    if test "$val" != "bar"; then
      echo "Expectation failed for for env FOO in argo-rollouts: should 'bar', is '$val'"
      exit 1
    fi