linux-toradex.git/tools/testing/kunit/kunit_parser.py, branch v5.12-rc8

kunit: tool: fix minor typing issue with None status

2021-01-16T00:51:47+00:00

The code to handle aggregating statuses didn't check that the status
actually got set to some non-None value.
Default the value to SUCCESS instead of adding a bunch of `is None`
checks.

This sorta follows the precedent in commit 3fc48259d525 ("kunit: Don't
fail test suites if one of them is empty").

Also slightly simplify the code and add type annotations.

Signed-off-by: Daniel Latypov 
Reviewed-by: David Gow 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: tool: surface and address more typing issues

2021-01-16T00:49:34+00:00

The authors of this tool were more familiar with a different
type-checker, https://github.com/google/pytype.

That's open source, but mypy seems more prevalent (and runs faster).
And unlike pytype, mypy doesn't try to infer types so it doesn't check
unanotated functions.

So annotate ~all functions in kunit tool to increase type-checking
coverage.
Note: per https://www.python.org/dev/peps/pep-0484/, `__init__()` should
be annotated as `-> None`.

Doing so makes mypy discover a number of new violations.
Exclude main() since we reuse `request` for the different types of
requests, which mypy isn't happy about.

This commit fixes all but one error, where `TestSuite.status` might be
None.

Signed-off-by: Daniel Latypov 
Reviewed-by: David Gow 
Tested-by: Brendan Higgins 
Acked-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: tool: Fix spelling of "diagnostic" in kunit_parser

2021-01-16T00:46:02+00:00

Various helper functions were misspelling "diagnostic" in their names.
It finally got annoying, so fix it.

Signed-off-by: David Gow 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: kunit_tool: Correctly parse diagnostic messages

2020-12-01T22:31:04+00:00

Currently, kunit_tool expects all diagnostic lines in test results to
contain ": " somewhere, as both the subtest header and the crash report
do. Fix this to accept any line starting with (minus indent) "# " as
being a valid diagnostic line.

This matches what the TAP spec[1] and the draft KTAP spec[2] are
expecting.

[1]: http://testanything.org/tap-specification.html
[2]: https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/T/

Signed-off-by: David Gow 
Acked-by: Marco Elver 
Reviewed-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: tool: fix extra trailing \n in raw + parsed test output

2020-11-10T20:44:35+00:00

For simplcity, strip all trailing whitespace from parsed output.
I imagine no one is printing out meaningful trailing whitespace via
KUNIT_FAIL() or similar, and that if they are, they really shouldn't.

`isolate_kunit_output()` yielded liens with trailing \n, which results
in artifacty output like this:

$ ./tools/testing/kunit/kunit.py run
[16:16:46] [FAILED] example_simple_test
[16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29

[16:16:46]     Expected 1 + 1 == 3, but

[16:16:46]         1 + 1 == 2

[16:16:46]         3 == 3

[16:16:46]     not ok 1 - example_simple_test

[16:16:46]

After this change:
[16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46]     Expected 1 + 1 == 3, but
[16:16:46]         1 + 1 == 2
[16:16:46]         3 == 3
[16:16:46]     not ok 1 - example_simple_test
[16:16:46]

We should *not* be expecting lines to end with \n in kunit_tool_test.py
for this reason.

Do the same for `raw_output()` as well which suffers from the same
issue.

This is a followup to [1], but rebased onto kunit-fixes to pick up the
other raw_output() fix and fixes for kunit_tool_test.py.

[1] https://lore.kernel.org/linux-kselftest/20201020233219.4146059-1-dlatypov@google.com/

Signed-off-by: Daniel Latypov 
Reviewed-by: David Gow 
Tested-by: David Gow 
Signed-off-by: Shuah Khan

kunit: tool: fix pre-existing python type annotation errors

2020-11-10T20:11:28+00:00

The code uses annotations, but they aren't accurate.
Note that type checking in python is a separate process, running
`kunit.py run` will not check and complain about invalid types at
runtime.

Fix pre-existing issues found by running a type checker
$ mypy *.py

All but one of these were returning `None` without denoting this
properly (via `Optional[Type]`).

Signed-off-by: Daniel Latypov 
Reviewed-by: David Gow 
Signed-off-by: Shuah Khan

kunit: Don't fail test suites if one of them is empty

2020-10-26T19:21:15+00:00

Empty test suite is okay test suite.

Don't fail the rest of the test suites if one of them is empty.

Fixes: 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests")
Signed-off-by: Andy Shevchenko 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: Fix kunit.py --raw_output option

2020-10-26T19:19:22+00:00

Due to the raw_output() function on kunit_parser.py actually being a
generator, it only runs if something reads the lines it returns. Since
we no-longer do that (parsing doesn't actually happen if raw_output is
enabled), it was not printing anything.

Fixes: 45ba7a893ad8 ("kunit: kunit_tool: Separate out config/build/exec/parse")
Signed-off-by: David Gow 
Reviewed-by: Brendan Higgins 
Tested-by: Brendan Higgins 
Signed-off-by: Shuah Khan

kunit: test: add test plan to KUnit TAP format

2020-10-09T20:37:49+00:00

TAP 14 allows an optional test plan to be emitted before the start of
the start of testing[1]; this is valuable because it makes it possible
for a test harness to detect whether the number of tests run matches the
number of tests expected to be run, ensuring that no tests silently
failed.

Link[1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
Signed-off-by: Brendan Higgins 
Reviewed-by: Stephen Boyd 
Signed-off-by: Shuah Khan

kunit: show error if kunit results are not present

2020-06-26T20:29:10+00:00

Currently, if the kernel is configured incorrectly or if it crashes before any
kunit tests are run, kunit finishes without error, reporting
that 0 test cases were run.

To fix this, an error is shown when the tap header is not found, which
indicates that kunit was not able to run at all.

Signed-off-by: Uriel Guajardo 
Reviewed-by: Brendan Higgins 
Signed-off-by: Shuah Khan