diff options
Diffstat (limited to 'Documentation/dev-tools')
| -rw-r--r-- | Documentation/dev-tools/checkpatch.rst | 53 | ||||
| -rw-r--r-- | Documentation/dev-tools/clang-format.rst | 2 | ||||
| -rw-r--r-- | Documentation/dev-tools/coccinelle.rst | 21 | ||||
| -rw-r--r-- | Documentation/dev-tools/container.rst | 227 | ||||
| -rw-r--r-- | Documentation/dev-tools/context-analysis.rst | 169 | ||||
| -rw-r--r-- | Documentation/dev-tools/index.rst | 10 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/run_manual.rst | 6 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/run_wrapper.rst | 11 | ||||
| -rw-r--r-- | Documentation/dev-tools/sparse.rst | 19 |
9 files changed, 474 insertions, 44 deletions
diff --git a/Documentation/dev-tools/checkpatch.rst b/Documentation/dev-tools/checkpatch.rst index d5c47e560324..dccede68698c 100644 --- a/Documentation/dev-tools/checkpatch.rst +++ b/Documentation/dev-tools/checkpatch.rst @@ -461,16 +461,9 @@ Comments line comments is:: /* - * This is the preferred style - * for multi line comments. - */ - - The networking comment style is a bit different, with the first line - not empty like the former:: - - /* This is the preferred comment style - * for files in net/ and drivers/net/ - */ + * This is the preferred style + * for multi line comments. + */ See: https://www.kernel.org/doc/html/latest/process/coding-style.html#commenting @@ -608,6 +601,11 @@ Commit message See: https://www.kernel.org/doc/html/latest/process/submitting-patches.html#describe-your-changes + **BAD_COMMIT_SEPARATOR** + The commit separator is a single line with 3 dashes. + The regex match is '^---$' + Lines that start with 3 dashes and have more content on the same line + may confuse tools that apply patches. Comparison style ---------------- @@ -760,7 +758,7 @@ Macros, Attributes and Symbols sizeof(foo)/sizeof(foo[0]) for finding number of elements in an array. - The macro is defined in include/linux/kernel.h:: + The macro is defined in include/linux/array_size.h:: #define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0])) @@ -1009,6 +1007,29 @@ Functions and Variables return bar; + **UNINITIALIZED_PTR_WITH_FREE** + Pointers with __free attribute should be declared at the place of use + and initialized (see include/linux/cleanup.h). In this case + declarations at the top of the function rule can be relaxed. Not doing + so may lead to undefined behavior as the memory assigned (garbage, + in case not initialized) to the pointer is freed automatically when + the pointer goes out of scope. + + Also see: https://lore.kernel.org/lkml/58fd478f408a34b578ee8d949c5c4b4da4d4f41d.camel@HansenPartnership.com/ + + Example:: + + type var __free(free_func); + ... // var not used, but, in future someone might add a return here + var = malloc(var_size); + ... + + should be initialized as:: + + ... + type var __free(free_func) = malloc(var_size); + ... + Permissions ----------- @@ -1245,6 +1266,16 @@ Others The patch file does not appear to be in unified-diff format. Please regenerate the patch file before sending it to the maintainer. + **PLACEHOLDER_USE** + Detects unhandled placeholder text left in cover letters or commit headers/logs. + Common placeholders include lines like:: + + *** SUBJECT HERE *** + *** BLURB HERE *** + + These typically come from autogenerated templates. Replace them with a proper + subject and description before sending. + **PRINTF_0XDECIMAL** Prefixing 0x with decimal output is defective and should be corrected. diff --git a/Documentation/dev-tools/clang-format.rst b/Documentation/dev-tools/clang-format.rst index 1d089a847c1b..6c8a0df5a00c 100644 --- a/Documentation/dev-tools/clang-format.rst +++ b/Documentation/dev-tools/clang-format.rst @@ -88,7 +88,7 @@ Reformatting blocks of code By using an integration with your text editor, you can reformat arbitrary blocks (selections) of code with a single keystroke. This is specially -useful when moving code around, for complex code that is deeply intended, +useful when moving code around, for complex code that is deeply indented, for multi-line macros (and aligning their backslashes), etc. Remember that you can always tweak the changes afterwards in those cases diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 6e70a1e9a3c0..c714780d458a 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -127,6 +127,18 @@ To enable verbose messages set the V= variable, for example:: make coccicheck MODE=report V=1 +By default, coccicheck will print debug logs to stdout and redirect stderr to +/dev/null. This can make coccicheck output difficult to read and understand. +Debug and error messages can instead be written to a debug file instead by +setting the ``DEBUG_FILE`` variable:: + + make coccicheck MODE=report DEBUG_FILE="cocci.log" + +Coccinelle cannot overwrite a debug file. Instead of repeatedly deleting a log +file, you could include the datetime in the debug file name:: + + make coccicheck MODE=report DEBUG_FILE="cocci-$(date -Iseconds).log" + Coccinelle parallelization -------------------------- @@ -208,11 +220,10 @@ include options matching the options used when we compile the kernel. You can learn what these options are by using V=1; you could then manually run Coccinelle with debug options added. -Alternatively you can debug running Coccinelle against SmPL patches -by asking for stderr to be redirected to stderr. By default stderr -is redirected to /dev/null; if you'd like to capture stderr you -can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For -instance:: +An easier approach to debug running Coccinelle against SmPL patches is to ask +coccicheck to redirect stderr to a debug file. As mentioned in the examples, by +default stderr is redirected to /dev/null; if you'd like to capture stderr you +can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For instance:: rm -f cocci.err make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err diff --git a/Documentation/dev-tools/container.rst b/Documentation/dev-tools/container.rst new file mode 100644 index 000000000000..452415b64662 --- /dev/null +++ b/Documentation/dev-tools/container.rst @@ -0,0 +1,227 @@ +.. SPDX-License-Identifier: GPL-2.0-only +.. Copyright (C) 2025 Guillaume Tucker + +==================== +Containerized Builds +==================== + +The ``container`` tool can be used to run any command in the kernel source tree +from within a container. Doing so facilitates reproducing builds across +various platforms, for example when a test bot has reported an issue which +requires a specific version of a compiler or an external test suite. While +this can already be done by users who are familiar with containers, having a +dedicated tool in the kernel tree lowers the barrier to entry by solving common +problems once and for all (e.g. user id management). It also makes it easier +to share an exact command line leading to a particular result. The main use +case is likely to be kernel builds but virtually anything can be run: KUnit, +checkpatch etc. provided a suitable image is available. + + +Options +======= + +Command line syntax:: + + scripts/container -i IMAGE [OPTION]... CMD... + +Available options: + +``-e, --env-file ENV_FILE`` + + Path to an environment file to load in the container. + +``-g, --gid GID`` + + Group id to use inside the container. + +``-i, --image IMAGE`` + + Container image name (required). + +``-r, --runtime RUNTIME`` + + Container runtime name. Supported runtimes: ``docker``, ``podman``. + + If not specified, the first one found on the system will be used + i.e. Podman if present, otherwise Docker. + +``-s, --shell`` + + Run the container in an interactive shell. + +``-u, --uid UID`` + + User id to use inside the container. + + If the ``-g`` option is not specified, the user id will also be used for + the group id. + +``-v, --verbose`` + + Enable verbose output. + +``-h, --help`` + + Show the help message and exit. + + +Usage +===== + +It's entirely up to the user to choose which image to use and the ``CMD`` +arguments are passed directly as an arbitrary command line to run in the +container. The tool will take care of mounting the source tree as the current +working directory and adjust the user and group id as needed. + +The container image which would typically include a compiler toolchain is +provided by the user and selected via the ``-i`` option. The container runtime +can be selected with the ``-r`` option, which can be either ``docker`` or +``podman``. If none is specified, the first one found on the system will be +used while giving priority to Podman. Support for other runtimes may be added +later depending on their popularity among users. + +By default, commands are run non-interactively. The user can abort a running +container with SIGINT (Ctrl-C). To run commands interactively with a TTY, the +``--shell`` or ``-s`` option can be used. Signals will then be received by the +shell directly rather than the parent ``container`` process. To exit an +interactive shell, use Ctrl-D or ``exit``. + +.. note:: + + The only host requirement aside from a container runtime is Python 3.10 or + later. + +.. note:: + + Out-of-tree builds are not fully supported yet. The ``O=`` option can + however already be used with a relative path inside the source tree to keep + separate build outputs. A workaround to build outside the tree is to use + ``mount --bind``, see the examples section further down. + + +Environment Variables +===================== + +Environment variables are not propagated to the container so they have to be +either defined in the image itself or via the ``-e`` option using an +environment file. In some cases it makes more sense to have them defined in +the Containerfile used to create the image. For example, a Clang-only compiler +toolchain image may have ``LLVM=1`` defined. + +The local environment file is more useful for user-specific variables added +during development. It is passed as-is to the container runtime so its format +may vary. Typically, it will look like the output of ``env``. For example:: + + INSTALL_MOD_STRIP=1 + SOME_RANDOM_TEXT=One upon a time + +Please also note that ``make`` options can still be passed on the command line, +so while this can't be done since the first argument needs to be the +executable:: + + scripts/container -i docker.io/tuxmake/korg-clang LLVM=1 make # won't work + +this will work:: + + scripts/container -i docker.io/tuxmake/korg-clang make LLVM=1 + + +User IDs +======== + +This is an area where the behaviour will vary slightly depending on the +container runtime. The goal is to run commands as the user invoking the tool. +With Podman, a namespace is created to map the current user id to a different +one in the container (1000 by default). With Docker, while this is also +possible with recent versions it requires a special feature to be enabled in +the daemon so it's not used here for simplicity. Instead, the container is run +with the current user id directly. In both cases, this will provide the same +file permissions for the kernel source tree mounted as a volume. The only +difference is that when using Docker without a namespace, the user id may not +be the same as the default one set in the image. + +Say, we're using an image which sets up a default user with id 1000 and the +current user calling the ``container`` tool has id 1234. The kernel source +tree was checked out by this same user so the files belong to user 1234. With +Podman, the container will be running as user id 1000 with a mapping to id 1234 +so that the files from the mounted volume appear to belong to id 1000 inside +the container. With Docker and no namespace, the container will be running +with user id 1234 which can access the files in the volume but not in the user +1000 home directory. This shouldn't be an issue when running commands only in +the kernel tree but it is worth highlighting here as it might matter for +special corner cases. + +.. note:: + + Podman's `Docker compatibility + <https://podman-desktop.io/docs/migrating-from-docker/managing-docker-compatibility>`__ + mode to run ``docker`` commands on top of a Podman backend is more complex + and not fully supported yet. As such, Podman will take priority if both + runtimes are available on the system. + + +Examples +======== + +The TuxMake project provides a variety of prebuilt container images available +on `Docker Hub <https://hub.docker.com/u/tuxmake>`__. Here's the shortest +example to build a kernel using a TuxMake Clang image:: + + scripts/container -i docker.io/tuxmake/korg-clang -- make LLVM=1 defconfig + scripts/container -i docker.io/tuxmake/korg-clang -- make LLVM=1 -j$(nproc) + +.. note:: + + When running a command with options within the container, it should be + separated with a double dash ``--`` to not confuse them with the + ``container`` tool options. Plain commands with no options don't strictly + require the double dashes e.g.:: + + scripts/container -i docker.io/tuxmake/korg-clang make mrproper + +To run ``checkpatch.pl`` in a ``patches`` directory with a generic Perl image:: + + scripts/container -i perl:slim-trixie scripts/checkpatch.pl patches/* + +As an alternative to the TuxMake images, the examples below refer to +``kernel.org`` images which are based on the `kernel.org compiler toolchains +<https://mirrors.edge.kernel.org/pub/tools/>`__. These aren't (yet) officially +available in any public registry but users can build their own locally instead +using this `experimental repository +<https://gitlab.com/gtucker/korg-containers>`__ by running ``make +PREFIX=kernel.org/``. + +To build just ``bzImage`` using Clang:: + + scripts/container -i kernel.org/clang -- make bzImage -j$(nproc) + +Same with GCC 15 as a particular version tag:: + + scripts/container -i kernel.org/gcc:15 -- make bzImage -j$(nproc) + +For an out-of-tree build, a trick is to bind-mount the destination directory to +a relative path inside the source tree:: + + mkdir -p $HOME/tmp/my-kernel-build + mkdir -p build + sudo mount --bind $HOME/tmp/my-kernel-build build + scripts/container -i kernel.org/gcc -- make mrproper + scripts/container -i kernel.org/gcc -- make O=build defconfig + scripts/container -i kernel.org/gcc -- make O=build -j$(nproc) + +To run KUnit in an interactive shell and get the full output:: + + scripts/container -s -i kernel.org/gcc:kunit -- \ + tools/testing/kunit/kunit.py \ + run \ + --arch=x86_64 \ + --cross_compile=x86_64-linux- + +To just start an interactive shell:: + + scripts/container -si kernel.org/gcc bash + +To build the HTML documentation, which requires the ``kdocs`` image built with +``make PREFIX=kernel.org/ extra`` as it's not a compiler toolchain:: + + scripts/container -i kernel.org/kdocs make htmldocs diff --git a/Documentation/dev-tools/context-analysis.rst b/Documentation/dev-tools/context-analysis.rst new file mode 100644 index 000000000000..54d9ee28de98 --- /dev/null +++ b/Documentation/dev-tools/context-analysis.rst @@ -0,0 +1,169 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. Copyright (C) 2025, Google LLC. + +.. _context-analysis: + +Compiler-Based Context Analysis +=============================== + +Context Analysis is a language extension, which enables statically checking +that required contexts are active (or inactive) by acquiring and releasing +user-definable "context locks". An obvious application is lock-safety checking +for the kernel's various synchronization primitives (each of which represents a +"context lock"), and checking that locking rules are not violated. + +The Clang compiler currently supports the full set of context analysis +features. To enable for Clang, configure the kernel with:: + + CONFIG_WARN_CONTEXT_ANALYSIS=y + +The feature requires Clang 22 or later. + +The analysis is *opt-in by default*, and requires declaring which modules and +subsystems should be analyzed in the respective `Makefile`:: + + CONTEXT_ANALYSIS_mymodule.o := y + +Or for all translation units in the directory:: + + CONTEXT_ANALYSIS := y + +It is possible to enable the analysis tree-wide, however, which will result in +numerous false positive warnings currently and is *not* generally recommended:: + + CONFIG_WARN_CONTEXT_ANALYSIS_ALL=y + +Programming Model +----------------- + +The below describes the programming model around using context lock types. + +.. note:: + Enabling context analysis can be seen as enabling a dialect of Linux C with + a Context System. Some valid patterns involving complex control-flow are + constrained (such as conditional acquisition and later conditional release + in the same function). + +Context analysis is a way to specify permissibility of operations to depend on +context locks being held (or not held). Typically we are interested in +protecting data and code in a critical section by requiring a specific context +to be active, for example by holding a specific lock. The analysis ensures that +callers cannot perform an operation without the required context being active. + +Context locks are associated with named structs, along with functions that +operate on struct instances to acquire and release the associated context lock. + +Context locks can be held either exclusively or shared. This mechanism allows +assigning more precise privileges when a context is active, typically to +distinguish where a thread may only read (shared) or also write (exclusive) to +data guarded within a context. + +The set of contexts that are actually active in a given thread at a given point +in program execution is a run-time concept. The static analysis works by +calculating an approximation of that set, called the context environment. The +context environment is calculated for every program point, and describes the +set of contexts that are statically known to be active, or inactive, at that +particular point. This environment is a conservative approximation of the full +set of contexts that will actually be active in a thread at run-time. + +More details are also documented `here +<https://clang.llvm.org/docs/ThreadSafetyAnalysis.html>`_. + +.. note:: + Clang's analysis explicitly does not infer context locks acquired or + released by inline functions. It requires explicit annotations to (a) assert + that it's not a bug if a context lock is released or acquired, and (b) to + retain consistency between inline and non-inline function declarations. + +Supported Kernel Primitives +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Currently the following synchronization primitives are supported: +`raw_spinlock_t`, `spinlock_t`, `rwlock_t`, `mutex`, `seqlock_t`, +`bit_spinlock`, RCU, SRCU (`srcu_struct`), `rw_semaphore`, `local_lock_t`, +`ww_mutex`. + +To initialize variables guarded by a context lock with an initialization +function (``type_init(&lock)``), prefer using ``guard(type_init)(&lock)`` or +``scoped_guard(type_init, &lock) { ... }`` to initialize such guarded members +or globals in the enclosing scope. This initializes the context lock and treats +the context as active within the initialization scope (initialization implies +exclusive access to the underlying object). + +For example:: + + struct my_data { + spinlock_t lock; + int counter __guarded_by(&lock); + }; + + void init_my_data(struct my_data *d) + { + ... + guard(spinlock_init)(&d->lock); + d->counter = 0; + ... + } + +Alternatively, initializing guarded variables can be done with context analysis +disabled, preferably in the smallest possible scope (due to lack of any other +checking): either with a ``context_unsafe(var = init)`` expression, or by +marking small initialization functions with the ``__context_unsafe(init)`` +attribute. + +Lockdep assertions, such as `lockdep_assert_held()`, inform the compiler's +context analysis that the associated synchronization primitive is held after +the assertion. This avoids false positives in complex control-flow scenarios +and encourages the use of Lockdep where static analysis is limited. For +example, this is useful when a function doesn't *always* require a lock, making +`__must_hold()` inappropriate. + +Keywords +~~~~~~~~ + +.. kernel-doc:: include/linux/compiler-context-analysis.h + :identifiers: context_lock_struct + token_context_lock token_context_lock_instance + __guarded_by __pt_guarded_by + __must_hold + __must_not_hold + __acquires + __cond_acquires + __releases + __must_hold_shared + __acquires_shared + __cond_acquires_shared + __releases_shared + __acquire + __release + __acquire_shared + __release_shared + __acquire_ret + __acquire_shared_ret + context_unsafe + __context_unsafe + disable_context_analysis enable_context_analysis + +.. note:: + The function attribute `__no_context_analysis` is reserved for internal + implementation of context lock types, and should be avoided in normal code. + +Background +---------- + +Clang originally called the feature `Thread Safety Analysis +<https://clang.llvm.org/docs/ThreadSafetyAnalysis.html>`_, with some keywords +and documentation still using the thread-safety-analysis-only terminology. This +was later changed and the feature became more flexible, gaining the ability to +define custom "capabilities". Its foundations can be found in `Capability +Systems <https://www.cs.cornell.edu/talc/papers/capabilities.pdf>`_, used to +specify the permissibility of operations to depend on some "capability" being +held (or not held). + +Because the feature is not just able to express capabilities related to +synchronization primitives, and "capability" is already overloaded in the +kernel, the naming chosen for the kernel departs from Clang's initial "Thread +Safety" and "capability" nomenclature; we refer to the feature as "Context +Analysis" to avoid confusion. The internal implementation still makes +references to Clang's terminology in a few places, such as `-Wthread-safety` +being the warning option that also still appears in diagnostic messages. diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst index 4b8425e348ab..59cbb77b33ff 100644 --- a/Documentation/dev-tools/index.rst +++ b/Documentation/dev-tools/index.rst @@ -21,6 +21,7 @@ Documentation/process/debugging/index.rst checkpatch clang-format coccinelle + context-analysis sparse kcov gcov @@ -38,11 +39,4 @@ Documentation/process/debugging/index.rst gpio-sloppy-logic-analyzer autofdo propeller - - -.. only:: subproject and html - - Indices - ======= - - * :ref:`genindex` + container diff --git a/Documentation/dev-tools/kunit/run_manual.rst b/Documentation/dev-tools/kunit/run_manual.rst index 699d92885075..98e8d5b28808 100644 --- a/Documentation/dev-tools/kunit/run_manual.rst +++ b/Documentation/dev-tools/kunit/run_manual.rst @@ -35,6 +35,12 @@ or be built into the kernel. a good way of quickly testing everything applicable to the current config. + KUnit can be enabled or disabled at boot time, and this behavior is + controlled by the kunit.enable kernel parameter. + By default, kunit.enable is set to 1 because KUNIT_DEFAULT_ENABLED is + enabled by default. To ensure that tests are executed as expected, + verify that kunit.enable=1 at boot time. + Once we have built our kernel (and/or modules), it is simple to run the tests. If the tests are built-in, they will run automatically on the kernel boot. The results will be written to the kernel log (``dmesg``) diff --git a/Documentation/dev-tools/kunit/run_wrapper.rst b/Documentation/dev-tools/kunit/run_wrapper.rst index 6697c71ee8ca..770bb09a475a 100644 --- a/Documentation/dev-tools/kunit/run_wrapper.rst +++ b/Documentation/dev-tools/kunit/run_wrapper.rst @@ -335,3 +335,14 @@ command line arguments: - ``--list_tests_attr``: If set, lists all tests that will be run and all of their attributes. + +- ``--list_suites``: If set, lists all suites that will be run. + +Command-line completion +============================== + +The kunit_tool comes with a bash completion script: + +.. code-block:: bash + + source tools/testing/kunit/kunit-completion.sh diff --git a/Documentation/dev-tools/sparse.rst b/Documentation/dev-tools/sparse.rst index dc791c8d84d1..37b20170835d 100644 --- a/Documentation/dev-tools/sparse.rst +++ b/Documentation/dev-tools/sparse.rst @@ -53,25 +53,6 @@ sure that bitwise types don't get mixed up (little-endian vs big-endian vs cpu-endian vs whatever), and there the constant "0" really _is_ special. -Using sparse for lock checking ------------------------------- - -The following macros are undefined for gcc and defined during a sparse -run to use the "context" tracking feature of sparse, applied to -locking. These annotations tell sparse when a lock is held, with -regard to the annotated function's entry and exit. - -__must_hold - The specified lock is held on function entry and exit. - -__acquires - The specified lock is held on function exit, but not entry. - -__releases - The specified lock is held on function entry, but not exit. - -If the function enters and exits without the lock held, acquiring and -releasing the lock inside the function in a balanced way, no -annotation is needed. The three annotations above are for cases where -sparse would otherwise report a context imbalance. - Getting sparse -------------- |