linux-toradex.git/tools/testing/kunit/kunit_parser.py, branch v5.11-rc3

kunit: kunit_tool: Correctly parse diagnostic messages

2020-12-01T22:31:04+00:00

Currently, kunit_tool expects all diagnostic lines in test results to
contain ": " somewhere, as both the subtest header and the crash report
do. Fix this to accept any line starting with (minus indent) "# " as
being a valid diagnostic line.

This matches what the TAP spec[1] and the draft KTAP spec[2] are
expecting.

[1]: http://testanything.org/tap-specification.html
[2]: https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/T/

Signed-off-by: David Gow 
Acked-by: Marco Elver 
Reviewed-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: tool: fix extra trailing \n in raw + parsed test output

2020-11-10T20:44:35+00:00

For simplcity, strip all trailing whitespace from parsed output.
I imagine no one is printing out meaningful trailing whitespace via
KUNIT_FAIL() or similar, and that if they are, they really shouldn't.

`isolate_kunit_output()` yielded liens with trailing \n, which results
in artifacty output like this:

$ ./tools/testing/kunit/kunit.py run
[16:16:46] [FAILED] example_simple_test
[16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29

[16:16:46]     Expected 1 + 1 == 3, but

[16:16:46]         1 + 1 == 2

[16:16:46]         3 == 3

[16:16:46]     not ok 1 - example_simple_test

[16:16:46]

After this change:
[16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46]     Expected 1 + 1 == 3, but
[16:16:46]         1 + 1 == 2
[16:16:46]         3 == 3
[16:16:46]     not ok 1 - example_simple_test
[16:16:46]

We should *not* be expecting lines to end with \n in kunit_tool_test.py
for this reason.

Do the same for `raw_output()` as well which suffers from the same
issue.

This is a followup to [1], but rebased onto kunit-fixes to pick up the
other raw_output() fix and fixes for kunit_tool_test.py.

[1] https://lore.kernel.org/linux-kselftest/20201020233219.4146059-1-dlatypov@google.com/

Signed-off-by: Daniel Latypov 
Reviewed-by: David Gow 
Tested-by: David Gow 
Signed-off-by: Shuah Khan

kunit: tool: fix pre-existing python type annotation errors

2020-11-10T20:11:28+00:00

The code uses annotations, but they aren't accurate.
Note that type checking in python is a separate process, running
`kunit.py run` will not check and complain about invalid types at
runtime.

Fix pre-existing issues found by running a type checker
$ mypy *.py

All but one of these were returning `None` without denoting this
properly (via `Optional[Type]`).

Signed-off-by: Daniel Latypov 
Reviewed-by: David Gow 
Signed-off-by: Shuah Khan

kunit: Don't fail test suites if one of them is empty

2020-10-26T19:21:15+00:00

Empty test suite is okay test suite.

Don't fail the rest of the test suites if one of them is empty.

Fixes: 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests")
Signed-off-by: Andy Shevchenko 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: Fix kunit.py --raw_output option

2020-10-26T19:19:22+00:00

Due to the raw_output() function on kunit_parser.py actually being a
generator, it only runs if something reads the lines it returns. Since
we no-longer do that (parsing doesn't actually happen if raw_output is
enabled), it was not printing anything.

Fixes: 45ba7a893ad8 ("kunit: kunit_tool: Separate out config/build/exec/parse")
Signed-off-by: David Gow 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: test: add test plan to KUnit TAP format

2020-10-09T20:37:49+00:00

TAP 14 allows an optional test plan to be emitted before the start of
the start of testing[1]; this is valuable because it makes it possible
for a test harness to detect whether the number of tests run matches the
number of tests expected to be run, ensuring that no tests silently
failed.

Link[1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
Signed-off-by: Brendan Higgins 
Reviewed-by: Stephen Boyd 
Signed-off-by: Shuah Khan

kunit: show error if kunit results are not present

2020-06-26T20:29:10+00:00

Currently, if the kernel is configured incorrectly or if it crashes before any
kunit tests are run, kunit finishes without error, reporting
that 0 test cases were run.

To fix this, an error is shown when the tap header is not found, which
indicates that kunit was not able to run at all.

Signed-off-by: Uriel Guajardo 
Reviewed-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: subtests should be indented 4 spaces according to TAP

2020-03-26T20:08:41+00:00

Introduce KUNIT_SUBTEST_INDENT macro which corresponds to 4-space
indentation and KUNIT_SUBSUBTEST_INDENT macro which corresponds to
8-space indentation in line with TAP spec (e.g. see "Subtests"
section of https://node-tap.org/tap-protocol/).

Use these macros in place of one or two tabs in strings to clarify
why we are indenting.

Suggested-by: Frank Rowand 
Signed-off-by: Alan Maguire 
Reviewed-by: Frank Rowand 
Signed-off-by: Shuah Khan

kunit: Run all KUnit tests through allyesconfig

2020-03-20T22:17:39+00:00

Implemented the functionality to run all KUnit tests through kunit_tool
by specifying an --alltests flag, which builds UML with allyesconfig
enabled, and consequently runs every KUnit test. A new function was
added to kunit_kernel: make_allyesconfig.
Firstly, if --alltests is specified, kunit.py triggers build_um_kernel
which call make_allyesconfig. This function calls the make command,
disables the broken configs that would otherwise prevent UML from
building, then starts the kernel with all possible configurations
enabled. All stdout and stderr is sent to test.log and read from there
then fed through kunit_parser to parse the tests to the user. Also added
a signal_handler in case kunit is interrupted while running.
Tested: Run under different conditions such as testing with
--raw_output, testing program interrupt then immediately running kunit
again without --alltests and making sure to clean the console.

Signed-off-by: Heidi Fahim 
Reviewed-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: kunit_parser: make parser more robust

2020-03-20T22:15:57+00:00

Previously, kunit_parser did not properly handle kunit TAP output that
- had any prefixes (generated from different configs e.g.
CONFIG_PRINTK_TIME)
- had unrelated kernel output mixed in the middle of
it, which has shown up when testing with allyesconfig
To remove prefixes, the parser looks for the first line that includes
TAP output, "TAP version 14".  It then determines the length of the
string before this sequence, and strips that number of characters off
the beginning of the following lines until the last KUnit output line is
reached.
These fixes have been tested with additional tests in the
KUnitParseTest and their associated logs have also been added.

Signed-off-by: Heidi Fahim 
Reviewed-by: Brendan Higgins 
Signed-off-by: Shuah Khan