linux-toradex.git/scripts/jobserver-exec, branch master

kbuild: compress the kernel with pigz if available

2026-09-25T23:16:36+00:00

The gzip step of a kernel build is very lengthy, especially for larger
builds such as allmodconfig.

gzip itself cannot be run in parallel, however an alternative tool exists
that can, providing the same feature set as gzip itself - pigz - which
works as a drop-in replacement.

On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
1.6s, and with pigz it takes 0.09s, so the performance increase is
significant.

Therefore introduce the KPGZIP build variable which specifies the
compressor to use for the kernel image and default it to pigz if available,
or KGZIP otherwise (which itself defaults to gzip).

Modules, packaging and the embedded kernel configuration (CONFIG_IKCONFIG)
continue to use KGZIP which defaults to a single-threaded gzip instance.

Module compression is run in parallel by make, so pigz offers no benefit
there, packaging sits outside the build and the embedded configuration is
too small to matter.

pigz is executed using scripts/jobserver-exec, which reserves the job slots
make has free - necessary, as otherwise pigz would just grab unbounded
CPUs.

The kernel image compression step is always performed at the end of the
build so should typically have all job slots available.

pigz always provides byte-for-byte identical output across runs regardless
of thread count, but pigz and gzip are not guaranteed to generate the same
binary output, so reproducible builds must use the same tools consistently.

Also update Documentation/kbuild/kbuild.rst to describe KPGZIP and
Documentation/process/changes.rst to list the optional pigz dependency.

While there, add entries for the other compressor programs.

Finally, update scripts/ver_linux to include the pigz version there.

Every x86 build ends with the compression of vmlinux.bin, 36MB for
defconfig and over 200MB for allmodconfig, no-op builds are unchanged.

Whole build, 128-thread Threadripper 9980X, best of N runs:

                                         before   after     delta
                                         -------------------------------
  x86 defconfig, touch mm/vma.c, gcc        7.7s     6.0s     -1.7s (-22%)
  x86 defconfig, touch mm/vma.c, clang      6.9s     5.4s     -1.5s (-21%)
  x86 defconfig, clean, gcc                28.2s    26.3s     -1.8s (-6%)
  x86 defconfig, clean, clang              28.7s    27.2s     -1.5s (-5%)
  x86 allmodconfig, touch mm/vma.c, gcc    23.7s    15.4s     -8.3s (-35%)
  x86 allmodconfig, touch mm/vma.c, clang  22.5s    15.6s     -7.0s (-31%)
  x86 allmodconfig, clean, gcc            294.1s   278.6s     -15.5s (-5%)
  x86 allmodconfig, clean, clang          283.9s   266.8s     -17.1s (-6%)

Suggested-by: Kees Cook 
Link: https://zlib.net/pigz/
Assisted-by: LLM
Reviewed-by: Manuel Ebner 
Reviewed-by: Kees Cook 
Signed-off-by: Lorenzo Stoakes (ARM) 
Link: https://patch.msgid.link/20260923-build-speedup-v4-22-73128809a4a4@kernel.org
Signed-off-by: Nathan Chancellor

scripts/jobserver-exec: propagate child exit status

2026-06-10T18:37:05+00:00

main() called JobserverExec().run() and discarded its return value,
then the script exited with the implicit status 0. As a result, any
Makefile that wired a build step through jobserver-exec saw the step
silently succeed even when the wrapped command had failed.

Two in-tree callers were affected:

  Documentation/devicetree/bindings/Makefile
    cmd_chk_style runs a python checker via jobserver-exec and uses
    "&& touch $@ || true" so failures leave the stamp file untouched
    and the next make rerun reports them again. The swallowed exit
    code made the stamp file get created even on failure, caching the
    failed run and hiding the reported issues until the inputs change.

  scripts/Makefile.vmlinux_o
    cmd_gen_initcalls_lds runs scripts/generate_initcall_order.pl via
    jobserver-exec; a perl failure was masked by the wrapper.

Return the subprocess exit code from main() and pass it to sys.exit()
so the wrapped command's status reaches make.

Signed-off-by: Daniel Golle 
Reviewed-by: Krzysztof Kozlowski 
Link: https://patch.msgid.link/660368ca16e2d3845577a9fd157d2f37f0e09e85.1779908995.git.daniel@makrotopia.org
Signed-off-by: Rob Herring (Arm)

docs: Move the python libraries to tools/lib/python

2025-11-18T16:22:40+00:00

"scripts/lib" was always a bit of an awkward place for Python modules.  We
already have tools/lib; create a tools/lib/python, move the libraries
there, and update the users accordingly.

While at it, move the contents of tools/docs/lib.  Rather than make another
directory, just put these documentation-oriented modules under "kdoc".

Signed-off-by: Jonathan Corbet 
Message-ID: <20251110220430.726665-2-corbet@lwn.net>

scripts/jobserver-exec: add a help message

2025-09-18T17:17:18+00:00

Currently, calling it without an argument shows an ugly error
message. Instead, print a message using pythondoc as description.

Signed-off-by: Mauro Carvalho Chehab 
Message-ID: <64b0339eac54ac0f2b3de3667a7f4f5becb1c6ae.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet

scripts/jobserver-exec: move its class to the lib directory

2025-09-18T17:17:18+00:00

To make it easier to be re-used, move the JobserverExec class
to the library directory.

Signed-off-by: Mauro Carvalho Chehab 
Message-ID: <6be7b161b6c005a9807162ebfd239af6a4e6fa47.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet

scripts/jobserver-exec: move the code to a class

2025-09-18T17:17:18+00:00

Convert the code inside jobserver-exec to a class and
properly document it.

Using a class allows reusing the jobserver logic on other
scripts.

While the main code remains unchanged, being compatible with
Python 2.6 and 3.0+, its coding style now follows a more
modern standard, having tabs replaced by a 4-spaces
indent, passing autopep8, black and pylint.

The code allows using a pythonic way to enter/exit a python
code, e.g. it now supports:

	with JobserverExec() as jobserver:
	    jobserver.run(sys.argv[1:])

With the new code, the __exit__() function should ensure
that the jobserver slot will be closed at the end, even if
something bad happens somewhere.

Signed-off-by: Mauro Carvalho Chehab 
Message-ID: <4749921b75d4e0bd85a25d4d94aa2c940fad084e.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet

scripts: support GNU make 4.4 in jobserver-exec

2023-01-16T11:15:20+00:00

Starting with GNU make 4.4, --jobserver-auth newly uses named
pipe (fifo) instead of part of opened file descriptors:
https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html

Support also the new format.

Signed-off-by: Martin Liska 
Signed-off-by: Masahiro Yamada

scripts/jobserver-exec: parse the last --jobserver-auth= option

2022-11-21T01:18:39+00:00

In the GNU Make manual, the section "Sharing Job Slots with GNU make"
says:

    Be aware that the MAKEFLAGS variable may contain multiple instances
    of the --jobserver-auth= option. Only the last instance is relevant.

Take the last element of the array, not the first.

Link: https://www.gnu.org/software/make/manual/html_node/Job-Slots.html
Signed-off-by: Masahiro Yamada 
Reviewed-by: Nicolas Schier

scripts/jobserver-exec: Fix a typo ("envirnoment")

2021-05-17T03:10:03+00:00

Signed-off-by: Jonathan Neuschäfer 
Signed-off-by: Masahiro Yamada

kbuild: remove PYTHON variable

2021-02-01T01:37:19+00:00

Python retired in 2020, and some distributions do not provide the
'python' command any more.

As in commit 51839e29cb59 ("scripts: switch explicitly to Python 3"),
we need to use more specific 'python3' to invoke scripts even if they
are written in a way compatible with both Python 2 and 3.

This commit removes the variable 'PYTHON', and switches the existing
users to 'PYTHON3'.

BTW, PEP 394 (https://www.python.org/dev/peps/pep-0394/) is a helpful
material.

Signed-off-by: Masahiro Yamada