Advanced usage ============== Custom Runnable Identifier -------------------------- In some cases, you might have a wrapper as an entry point for the tests, so Avocado will use only the wrapper as test id. For instance, imagine a Makefile with some targets ('foo', 'bar') and each target is one test. Having a single test suite with a test calling `foo`, it will make Avocado print something like this:: JOB ID : b6e5bdf2c891382bbde7f24e906a168af351154a JOB LOG : ~/avocado/job-results/job-2021-09-24T17.39-b6e5bdf/job.log (1/1) make: STARTED (1/1) make: PASS (2.72 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : ~/avocado/job-results/job-2021-09-24T17.39-b6e5bdf/results.html JOB TIME : 5.49 s This is happening because Avocado is using the 'uri' as identifier with in the current Runnables. You can change that by setting a custom format with the option `runner.identifier_format` in you `avocado.conf` file. For instance: ``` [runner] identifier_format = "{uri}-{args[0]}" ``` With the above adjustment, running the same suite it will produce something like this:: JOB ID : 577b70b079e9a6f325ff3e73fd9b93f80ee7f221 JOB LOG : /home/local/avocado/job-results/job-2021-11-23T13.12-577b70b/job.log (1/1) "/usr/bin/make-foo": STARTED (1/1) "/usr/bin/make-foo": PASS (0.01 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : ~/avocado/job-results/job-2021-11-23T13.12-577b70b/results.html JOB TIME : 0.97 s For the `identifier_format` you can use any f-string that it will use `{uri}`, `{args}` or `{kwargs}`. By default it will use `{uri}`. When using args, since it is a list, you can use in two different ways: "{args}" for the entire list, or "{args[n]}" for a specific element inside this list. The same is valid when using "{kwargs}". With kwargs, since it is a dictionary, you have to specify a key as index and then the values are used. For instance if you have a kwargs value named 'DEBUG', a valid usage could be: "{kwargs[DEBUG]}" and this will print the current value to this variable (i.e: True or False). .. note:: Please, keep in mind this is an experimental feature, and for now you have to use it in combination with :ref:`documentation`. .. note:: Also, be aware this feature it is meant to set custom Runnable identifiers strings only. Test Runner Selection --------------------- To effectively run a job with tests, Avocado makes use of a well described and pluggable interface. This means that users can choose (and developers can write) their own runners. Runner choices can be seen by running ``avocado plugins``:: ... Plugins that run test suites on a job (suite.runner): nrunner nrunner based implementation of job compliant runner And to select a different test runner (if another one exists):: avocado run --suite-runner=other_runner_plugin ... Running tests with an external runner ------------------------------------- It's pretty standard to have organically grown test suites in most software projects, and these usually include a custom-built, specific test runner who knows how to find and run their tests. Still, running those tests inside Avocado may be a good idea for various reasons, including having results in different human and machine-readable formats and collecting system information alongside those tests (the Avocado's Sysinfo functionality), and more. Avocado makes that possible using its "external runner" feature. The most basic way of using it is:: $ avocado-external-runner external_runner foo bar baz In this example, Avocado will report individual test results for tests foo, bar, and baz. The actual results will be based on the return code of individual executions of /path/to/external_runner foo, /path/to/external_runner bar and finally /path/to/external_runner baz. As another way to explain how this feature works, think of the "external runner" as an interpreter. The individual tests as anything that this interpreter recognizes and can execute. A UNIX shell, say /bin/sh could be considered an external runner, and files with shellcode could be viewed as tests:: $ echo "exit 1" > /tmp/fail $ echo "exit 0" > /tmp/pass $ avocado-external-runner /bin/sh /tmp/pass /tmp/fail JOB ID : 874cab7e2639f1e2244246c69a5e0d3e1afefee0 JOB LOG : ~/avocado/job-results/job-2022-01-19T15.33-874cab7/job.log (external-runner-2/2) /bin/sh-/tmp/fail: STARTED (external-runner-1/2) /bin/sh-/tmp/pass: STARTED (external-runner-2/2) /bin/sh-/tmp/fail: FAIL (0.01 s) (external-runner-1/2) /bin/sh-/tmp/pass: PASS (0.01 s) RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : ~/avocado/job-results/job-2022-01-19T15.33-874cab7/results.html JOB TIME : 1.10 s .. note:: This example is pretty obvious and could be achieved by giving /tmp/pass and /tmp/fail shell “shebangs” (#!/bin/sh), making them executable (chmod +x /tmp/pass /tmp/fail), and running them as “SIMPLE” tests. But now consider the following example:: $ avocado-external-runner curl redhat.com "google.com -v" JOB ID : fa68dd49a4c00e5a3c2e0fe45c6b3b0ed1b6495e JOB LOG : ~/avocado/job-results/job-2022-01-19T15.37-fa68dd4/job.log (external-runner-2/2) /bin/curl-google.com: STARTED (external-runner-1/2) /bin/curl-redhat.com: STARTED (external-runner-2/2) /bin/curl-google.com: PASS (0.28 s) (external-runner-1/2) /bin/curl-redhat.com: PASS (5.39 s) RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : ~/avocado/job-results/job-2022-01-19T15.37-fa68dd4/results.html JOB TIME : 6.38 s This effectively makes /bin/curl an “external test runner”, responsible for trying to fetch those URLs, and reporting PASS or FAIL for each of them.