diff options
Diffstat (limited to 'Documentation/gpu/nova/core')
| -rw-r--r-- | Documentation/gpu/nova/core/fsp.rst | 142 | ||||
| -rw-r--r-- | Documentation/gpu/nova/core/tlv.rst | 184 | ||||
| -rw-r--r-- | Documentation/gpu/nova/core/todo.rst | 76 | ||||
| -rw-r--r-- | Documentation/gpu/nova/core/vbios.rst | 65 |
4 files changed, 388 insertions, 79 deletions
diff --git a/Documentation/gpu/nova/core/fsp.rst b/Documentation/gpu/nova/core/fsp.rst new file mode 100644 index 000000000000..52d618d22bb8 --- /dev/null +++ b/Documentation/gpu/nova/core/fsp.rst @@ -0,0 +1,142 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================== +FSP (Foundation Security Processor) and Secure Boot +=================================================== +This document describes the role of the FSP in the GPU boot sequence on +Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot +flow. It also provides a brief overview of the PRC (Product Reconfiguration +Control) protocol used to query device configuration through FSP. As with +other documents in this directory, the information is subject to change and +is intended to help developers understand the corresponding kernel code. + +What is FSP? +============ +The Foundation Security Processor (FSP) is the GPU's Internal Root of Trust +(IROT). It is a dedicated security processor that boots from immutable ROM +(Boot ROM) inside the GPU and is responsible for establishing the Chain of +Trust before any other firmware is allowed to run. + +FSP runs independently of the host CPU and starts executing as soon as the +GPU is powered on. By the time the nova-core driver is loaded, FSP has +already completed its own secure boot and is ready to accept commands from +the driver. + +Simplified boot flow (Hopper/Blackwell) +======================================= +Starting with Hopper, the boot flow is significantly simplified compared to +earlier GPU generations like Ampere. + +On an **Ampere** GPU, the boot verification chain involves multiple Falcon +engines and multiple ucode stages (see falcon.rst for details):: + + Hardware BROM (SEC2) + -> HS Booter (SEC2) + -> LS GSP-RM (GSP) + +The driver must extract ucode from VBIOS, manage SEC2 and GSP, and +orchestrate the Booter to load GSP-RM. This involves FWSEC-FRTS, devinit, +and the Booter stages. + +On **Hopper/Blackwell** GPUs, FSP replaces this multi-stage process with a +single message-driven interface:: + + FSP (hardware root of trust, boots from ROM) + -> FMC (Falcon Microcontroller, verified by FSP) + -> GSP-RM (verified and loaded by FMC) + +The driver only needs to: + +1. Wait for FSP to complete its own secure boot (polling a scratch register). +2. Send a Chain of Trust (COT) message to FSP with the FMC firmware location, + cryptographic signatures, and GSP boot parameters. +3. FSP authenticates the FMC firmware and boots it, FMC in turn loads GSP-RM. + +There is no SEC2 involvement, no Booter ucode, and no FWSEC-FRTS stage. The +entire secure boot is driven by a single FSP message exchange. + +Chain of Trust (COT) protocol +============================= +The Chain of Trust establishes a cryptographically enforced boot sequence, +ensuring the GPU reaches a known, trusted state. + +The driver communicates with FSP using a message queue (Falcon MSGQ +interface). Each message consists of an MCTP (Management Component Transport +Protocol) transport header and an NVDM (NVIDIA Vendor Defined Message) header, +followed by a protocol-specific payload. + +For Chain of Trust, the payload includes: + +- The system memory address of the FMC firmware image. +- Cryptographic material: a SHA-384 hash, RSA-3K public key, and RSA-3K + signature extracted from the FMC ELF firmware. +- FRTS (Firmware Runtime Services) region information (vidmem offset and size). +- The system memory address of the GSP boot arguments structure. + +FSP verifies the signature against the provided public key and hash, and if +verification succeeds, boots the FMC. The FMC then authenticates and launches +GSP-RM. + +The message flow is:: + + nova-core FSP + | | + | 1. Poll scratch register | + | (wait for FSP boot complete) | + | | + | 2. COT message ------------> | + | (FMC addr, signatures, | + | boot params) | + | | + | |--- Verify FMC signature + | |--- Boot FMC + | |--- FMC loads GSP-RM + | | + | 3. COT response <------------ | + | (success/error) | + | | + +FSP message format +================== +All FSP messages share a common header format consisting of two 32-bit words: + +**MCTP header** (Management Component Transport Protocol): + +- Bit 31: SOM (Start of Message) +- Bit 30: EOM (End of Message) +- Bits 29:28: Packet sequence number +- Bits 23:16: Source Endpoint ID + +**NVDM header** (NVIDIA Vendor Defined Message): + +- Bits 6:0: MCTP message type (0x7e = vendor-defined PCI) +- Bits 23:8: PCI vendor ID (0x10de = NVIDIA) +- Bits 31:24: NVDM type (0x14 = COT, 0x13 = PRC, 0x15 = FSP response) + +PRC (Product Reconfiguration Control) protocol +=============================================== +PRC is an API system exposed through FSP's Management Partition that allows +querying and modifying device configuration without firmware updates. + +Configuration parameters are called "knobs". Each knob has a unique object +ID and controls a specific device behavior. Examples include vGPU mode, ECC +enable, confidential computing mode, and NVLINK configuration. + +Each knob has two values: + +- **Active**: the currently effective value for this boot cycle. +- **Persistent**: the value stored in InfoROM, applied on subsequent boots. + +The nova-core driver uses PRC to read the vGPU mode knob (object ID 0x29) +during early boot, before firmware loading, to determine whether the GPU +should operate in vGPU mode. + +The PRC message format follows the same MCTP/NVDM header structure as COT, +with NVDM type 0x13. The payload contains: + +- A sub-command (e.g., 0x0c for read). +- Flags indicating which value to read (bit 0 = persistent, bit 1 = active). +- The knob object ID. + +The response includes the common FSP response header (with error status) +followed by the knob's 16-bit state value. diff --git a/Documentation/gpu/nova/core/tlv.rst b/Documentation/gpu/nova/core/tlv.rst new file mode 100644 index 000000000000..3ce508e9545a --- /dev/null +++ b/Documentation/gpu/nova/core/tlv.rst @@ -0,0 +1,184 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +================================== +TLV Tags in Nova Firmware Images +================================== + +Nova firmware images use a Type-Length-Value (TLV) format to encapsulate +firmware components and metadata. The TLV file begins with a 4-byte "magic" +header that contains the string "NVFW". Following the header is a sequence of +TLV blocks. + +Each block consists of a 4-byte tag of ASCII characters, a 4-byte length +encoded as a little-endian unsigned integer, and a sequence of bytes, the size +of which is equal to the length rounded up to the next multiple of 4. + +The driver code that reads the TLV and uses its contents is called the parser. +It is the responsibility of the parser to handle missing or malformed tags, +lengths, and values in the TLV. + +:: + + +------+------+------+------+ + | 'N' | 'V' | 'F' | 'W' | Magic header + +------+------+------+------+ + | Tag (4 bytes, ASCII) | TLV block 0 + +---------------------------+ + | Length (4 bytes, LE) | + +---------------------------+ + | | + | Value (length bytes, | + | padded to 4-byte align) | + | | + +---------------------------+ + | Tag (4 bytes, ASCII) | TLV block 1 + +---------------------------+ + | Length (4 bytes, LE) | + +---------------------------+ + | | + | Value (length bytes, | + | padded to 4-byte align) | + | | + +---------------------------+ + | ... | More TLV blocks + +---------------------------+ + +Tags and Length +=============== +TLV tags are always four-character words, with all letters being upper case. +Duplicate tags are not allowed. + +A TLV file may contain additional tags not described in this document. + +Values +====== +Values are one of four types. The type is not encoded in the format; rather, +the parser expects a given tag to have a value of a given type. + +1) Integers, encoded in 32-bit or 64-bit little-endian format. +2) Strings, encoded as-is and required to be only printable ASCII characters + and without a null terminator. +3) An array of bytes, for binary data. +4) Boolean, encoded as single byte, with a value of 0 for False or 1 for True. + +Common Tags +=========== +These tags are shared across firmware types and carry the same meaning +wherever they appear. Unlike the firmware-specific tags below, a common tag +is reserved: its meaning is fixed and may never be redefined for a particular +firmware type. + +``VERS`` (string) + Human-readable firmware version string. Present in all TLV files. + +A TLV image must contain either a single ``BLOB`` tag (firmware embedded +inline) or a ``SIZE``/``FILE`` pair (firmware stored in a separate file). + +``BLOB`` (bytes) + If the firmware microcode binary is stored in the TLV, this tag contains + the actual firmware image bytes. + +``FILE`` (string) + If the firmware binary is stored as a separate file, this tag contains the + name of that file, which is required to be in the same directory as the TLV, + so no paths are allowed in the filename. This tag is always paired with + ``SIZE``, so as to allow the driver to pre-allocate the buffer before + loading the file. + +``SIZE`` (u32) + Total size in bytes of the firmware image to be loaded from the companion + file named by ``FILE``. This tag is mandatory if ``FILE`` exists, so the + size of the firmware image must be known when the TLV is created. If the + firmware image is updated and its size changes, then the TLV must be + updated with it. + +GSP Firmware Tags +================= +``SIGN`` (bytes) + Cryptographic signature for the GSP firmware. + +``BLID`` (string) + The build ID, extracted from the ".note.gnu.build-id" section. + +Booter Firmware Tags +==================== +``DAOF`` (u32) - ``os_data_offset`` + OS data section offset within the firmware image (absolute byte offset). + Maps to the DMEM load source. + +``DASZ`` (u32) - ``os_data_size`` + OS data section size in bytes. + +``CDOF`` (u32) - ``os_code_offset`` + OS code section offset within the firmware image (absolute byte offset). + Maps to the non-secure IMEM load source. + +``CDSZ`` (u32) - ``os_code_size`` + OS code section size in bytes. + +``PLOC`` (u32) - ``patch_loc`` + Signature patch location -- byte offset within the firmware image where the + selected signature should be written. + +``FUSE`` (u32) - ``fuse_version`` + Fuse version of the firmware, used with the hardware fuse register to + select the correct signature index. + +``ENID`` (u32) - ``engine_id`` + Engine ID mask identifying the falcon engine this firmware targets. + +``UCID`` (u32) - ``ucode_id`` + Microcode ID used together with the engine ID to query hardware signature + fuse registers. + +``A0CO`` (u32) - ``app0_code_offset`` + App0 code offset -- start of the secure code region within the firmware + image. Used as the IMEM secure section source. + +``A0CS`` (u32) - ``app0_code_size`` + App0 code size in bytes. + +``NSIG`` (u32) - ``num_sigs`` + Number of signatures included in the ``SIGN`` tag. + +``SIGN`` (bytes) + Concatenated array of firmware signatures. The size of each signature is + the total length of the ``SIGN`` value divided by ``NSIG``. The correct + signature is selected using the fuse-version-derived index. + +Generic Bootloader Tags +======================= +``CDSZ`` (u32) - ``code_size`` + Size in bytes of the bootloader code to copy from the ``BLOB`` tag and + PIO-load into falcon IMEM. + +``STRT`` (u32) - ``start_tag`` + Start tag identifying the IMEM block where execution begins. The falcon + boot address is derived as ``start_tag << 8``. + +GSP Bootloader Tags +=================== +``CDOF`` (u32) - ``code_offset`` + Offset within the firmware image at which the code section starts. + +``DAOF`` (u32) - ``data_offset`` + Offset within the firmware image at which the data section starts. + +``MFOF`` (u32) - ``manifest_offset`` + Offset within the firmware image at which the manifest starts. + +``APPV`` (u32) - ``app_version`` + Application version of the firmware. + +FMC Firmware Tags +================= +``HASH`` (bytes) + SHA-384 hash of the FMC firmware, exactly 48 bytes long. + +``PKEY`` (bytes) + Public key used to verify the FMC firmware. At most 384 bytes (RSA-3072), + but may be shorter. + +``SIGN`` (bytes) + Signature of the FMC firmware. At most 384 bytes (RSA-3072), but may + be shorter. diff --git a/Documentation/gpu/nova/core/todo.rst b/Documentation/gpu/nova/core/todo.rst index d1964eb645e2..d5130b2b08fb 100644 --- a/Documentation/gpu/nova/core/todo.rst +++ b/Documentation/gpu/nova/core/todo.rst @@ -51,82 +51,6 @@ There also have been considerations of ToPrimitive [2]. | Link: https://lore.kernel.org/all/cover.1750689857.git.y.j3ms.n@gmail.com/ [1] | Link: https://rust-for-linux.zulipchat.com/#narrow/channel/288089-General/topic/Implement.20.60FromPrimitive.60.20trait.20.2B.20derive.20macro.20for.20nova-core/with/541971854 [2] -Generic register abstraction [REGA] ------------------------------------ - -Work out how register constants and structures can be automatically generated -through generalized macros. - -Example: - -.. code-block:: rust - - register!(BOOT0, 0x0, u32, pci::Bar<SIZE>, Fields [ - MINOR_REVISION(3:0, RO), - MAJOR_REVISION(7:4, RO), - REVISION(7:0, RO), // Virtual register combining major and minor rev. - ]) - -This could expand to something like: - -.. code-block:: rust - - const BOOT0_OFFSET: usize = 0x00000000; - const BOOT0_MINOR_REVISION_SHIFT: u8 = 0; - const BOOT0_MINOR_REVISION_MASK: u32 = 0x0000000f; - const BOOT0_MAJOR_REVISION_SHIFT: u8 = 4; - const BOOT0_MAJOR_REVISION_MASK: u32 = 0x000000f0; - const BOOT0_REVISION_SHIFT: u8 = BOOT0_MINOR_REVISION_SHIFT; - const BOOT0_REVISION_MASK: u32 = BOOT0_MINOR_REVISION_MASK | BOOT0_MAJOR_REVISION_MASK; - - struct Boot0(u32); - - impl Boot0 { - #[inline] - fn read(bar: &RevocableGuard<'_, pci::Bar<SIZE>>) -> Self { - Self(bar.readl(BOOT0_OFFSET)) - } - - #[inline] - fn minor_revision(&self) -> u32 { - (self.0 & BOOT0_MINOR_REVISION_MASK) >> BOOT0_MINOR_REVISION_SHIFT - } - - #[inline] - fn major_revision(&self) -> u32 { - (self.0 & BOOT0_MAJOR_REVISION_MASK) >> BOOT0_MAJOR_REVISION_SHIFT - } - - #[inline] - fn revision(&self) -> u32 { - (self.0 & BOOT0_REVISION_MASK) >> BOOT0_REVISION_SHIFT - } - } - -Usage: - -.. code-block:: rust - - let bar = bar.try_access().ok_or(ENXIO)?; - - let boot0 = Boot0::read(&bar); - pr_info!("Revision: {}\n", boot0.revision()); - -A work-in-progress implementation currently resides in -`drivers/gpu/nova-core/regs/macros.rs` and is used in nova-core. It would be -nice to improve it (possibly using proc macros) and move it to the `kernel` -crate so it can be used by other components as well. - -Features desired before this happens: - -* Make I/O optional I/O (for field values that are not registers), -* Support other sizes than `u32`, -* Allow visibility control for registers and individual fields, -* Use Rust slice syntax to express fields ranges. - -| Complexity: Advanced -| Contact: Alexandre Courbot - Numerical operations [NUMM] --------------------------- diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst index efd40087480c..9d3379ccfb30 100644 --- a/Documentation/gpu/nova/core/vbios.rst +++ b/Documentation/gpu/nova/core/vbios.rst @@ -46,12 +46,71 @@ region is only accessible to heavy-secure ucode. are of type 0xE0 and can be identified as such. This could be subject to change in future generations. +IFR Header +---------- +On Kepler and later GPUs, the ROM begins with an Init-from-ROM (IFR) header +rather than a standard PCI ROM signature (0xAA55). The driver must parse the +IFR header to find where the PCI ROM images actually start. + +Init-from-ROM (IFR) is a special GPU feature used for power management +on some Nvidia GPUs. It references data in the VBIOS for its operation, +but for drivers the important piece is a header that precedes the +VBIOS PCI Expansion ROM. + +Most such GPUs do not need to parse the IFR header in order to find the +VBIOS, but the Nvidia GA100 is the exception. GA100 lacks a display engine, +so the PRAMIN method (which reads the VBIOS from VRAM via display hardware) +is unavailable, forcing the driver to read the ROM directly via PROM. +On other similar GPUs, either PRAMIN succeeds before PROM is tried, or the +IFR hardware has already applied the ROM offset so that PROM reads +transparently skip the IFR header. + +The driver should first check for the standard 0xAA55 signature at offset 0. +If found, there is no IFR header and the PCI ROM images start at +offset 0. If not found, check for the IFR signature and parse the header to +determine the PCI ROM image offset. + +Fixed Header Format +~~~~~~~~~~~~~~~~~~~ + +The IFR header begins with four 32-bit words at fixed offsets:: + + Offset Name Fields + ------ ------- ------ + 0x00 FIXED0 bits 31:0 - Signature (must be 0x4947564E, ASCII "NVGI") + 0x04 FIXED1 bit 31 - Reserved + bits 30:16 - FIXED_DATA_SIZE Fixed data size (offset to extended section) + bits 15:8 - VERSIONSW Software version + bits 7:0 - Reserved + 0x08 FIXED2 bit 31 - Reserved + bits 30:20 - Reserved (zero) + bits 19:0 - TOTAL_DATA_SIZE Total data size + +Finding the PCI ROM Image Offset +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The method to find this offset depends on `VERSIONSW`. + +- **Version 1 and 2**: Read `FIXED_DATA_SIZE` from `FIXED1` to get the extended + section offset. The PCI ROM image is the 32-bit value at `FIXED_DATA_SIZE + 4`. + +- **Version 3**: Read `TOTAL_DATA_SIZE` from `FIXED2`. The 32-bit value at that + offset is a flash status offset. Add 4096 to get the ROM directory offset, + `ROM_DIRECTORY_OFFSET`. The ROM directory must have signature 0x44524652 + (ASCII "RFRD"). The PCI ROM image offset is the 32-bit value at + `ROM_DIRECTORY_OFFSET + 8`. + +The PCI ROM image offset must be 4-byte aligned. All offsets are relative to the +start of ROM (BAR0 + 0x300000). + VBIOS ROM Layout ---------------- -The VBIOS layout is roughly a series of concatenated images laid out as follows:: +The VBIOS (PCI Expansion ROM) is a series of concatenated images laid out as +follows. On GPUs with an IFR header, this layout begins at the image offset +determined by parsing the IFR header. On older GPUs, it begins at offset 0:: +----------------------------------------------------------------------------+ - | VBIOS (Starting at ROM_OFFSET: 0x300000) | + | VBIOS (Starting at ROM_OFFSET: 0x300000 + IFR image offset) | +----------------------------------------------------------------------------+ | +-----------------------------------------------+ | | | PciAt Image (Type 0x00) | | @@ -173,7 +232,7 @@ Falcon data in the VBIOS which contains the PMU lookup table. This lookup table used to find the required Falcon ucode based on an application ID. The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_) -tokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which indicates the +tokens for a token with the Falcon data token id (0x70) which indicates the offset of the same from the start of the VBIOS image. Unfortunately, the offset does not account for the EFI image located between the PciAt and FwSec images. The `vbios.rs` code compensates for this with appropriate arithmetic. |