- Factor pcie_valid_speed() and pci_bus_speed2lnkctl2() out of bwctrl so
  they can be shared by the DWC core (Hans Zhang)

- Flush MSI writes from endpoint before unmapping the iATU, as we already
  do for MSI-X writes (Niklas Cassel)

- Unmap MSI iATU window before mapping MSI-X window, to avoid a subsequent
  MSI write using a disabled aperture and losing the interrupt (Niklas
  Cassel)

- Change endpoint .pre_init() and .init() callbacks to return errors and
  handle them (Marek Vasut)

* pci/controller/dwc:
  PCI: dwc: Handle return value from endpoint .pre_init callback
  PCI: dwc: Handle return value from endpoint .init callback
  PCI: dwc: ep: Fix unmap potentially unmapping the wrong iATU
  PCI: dwc: ep: Flush cached MSI write before unmapping the iATU
  PCI: dwc: Use common speed conversion function
  PCI: Move pci_bus_speed2lnkctl2() to public header
  PCI: Add public pcie_valid_speed() for shared validation

* pci/controller/root-port-reset:
  misc: pci_endpoint_test: Add AER error handlers
  PCI: dw-rockchip: Implement .reset_root_port() and use for link down
  PCI: qcom: Implement .reset_root_port() and use for link down
  PCI: host-common: Add link down handling for Root Ports
  PCI/ERR: Add support for resetting the Root Ports in a platform-specific way
  PCI: dwc: ep: Clear MSI iATU mapping in dw_pcie_ep_cleanup()

- Allow probing even without child services so it can do power management
  (Brian Norris)

* pci/portdrv:
  PCI/portdrv: Allow probing even without child services

- Allow DPC on all Downstream Ports, not just Root Ports, when OS controls
  AER (Darshit Shah)

* pci/dpc:
  PCI/DPC: Allow DPC on all Downstream Ports when OS controls AER

- Avoid L0s for Realtek RTS525A, where it causes an AER interrupt storm
  (Max Lee)

- Program the same ASPM Control values for every function of multi-function
  devices, as recommended by the PCIe spec (Krishna Chaitanya Chundru)

- Avoid ASPM L0s, L1, and L1 PM Substates based on 'aspm-no-l0s',
  'aspm-no-l1' [1], and 'aspm-no-l1ss' DT properties (Krishna Chaitanya
  Chundru)

* pci/aspm:
  PCI/ASPM: Mask ASPM states based on Devicetree properties
  PCI/ASPM: Disable/restore ASPM on every function for multi-function devices
  PCI/ASPM: Use pcie_capability_clear_and_set_word() for ASPM disable/restore
  PCI/ASPM: Avoid L0s for Realtek RTS525A

Per PCIe r7.0 sec 6.2.4.3, certain Non-Fatal Errors may be signaled using
ERR_COR instead of ERR_NONFATAL.  These "Advisory Non-Fatal Errors" are
listed in sec 6.2.7 and explained in detail in sec 6.2.3.2.4.

Advisory Non-Fatal Errors set bits in the Uncorrectable Error Status
Register as well as one bit in the Correctable Error Status Register
(Advisory Non-Fatal Error Status, bit 13).  The latter is masked by
default, hence these errors are currently not signaled at all (except on
non-compliant products which choose to unmask the bit).

Unmask Advisory Non-Fatal Errors on device enumeration.

Some Non-Fatal Errors are always Advisory, others may be Advisory at the
discretion of the detecting agent.  If multiple errors occur, the agent
may qualify a portion as non-Advisory and signal ERR_NONFATAL in addition
to ERR_COR.  In this case, there's no way to determine which Non-Fatal
Error was Advisory.  Assume none is to ensure that the Uncorrectable Error
code path is taken to recover from the errors.

Introduce aer_compute_anfe_status() to compute Advisory Non-Fatal Error
bits from AER registers, based on this policy.  Use it for Firmware First
error handling in pci_print_aer(), which receives an AER register dump
from the platform (UEFI r2.11 sec N.2.7).

Introduce aer_get_anfe_status() to read AER registers from a device and
feed them to aer_compute_anfe_status().  Use it for native error handling
in aer_get_device_error_info(), which gathers registers from the device
and caches the computed Advisory Non-Fatal Error bits in a new anfe_status
field in struct aer_err_info.

Regardless whether error handling is native or Firmware First, the AER
driver needs to increment error counters, signal a trace event and log
each error.  When Advisory Non-Fatal Errors occur, these steps must be
performed for Correctable Errors and for Uncorrectable Errors.  Achieve
this through a recursive invocation of aer_print_error() (for native error
handling) and pci_print_aer() (for Firmware First error handling).  The
recursive invocation reports the (Advisory) Uncorrectable Errors after
reporting the Correctable Errors.

Note that the First Error Pointer and TLP Prefix Log is only meaningful
for Uncorrectable Errors, but when Advisory Non-Fatal Errors occur,
aer_get_device_error_info() has to populate the first_error and
tlp_header_valid fields in struct aer_err_info for a Correctable Error.
Avoid incorrectly logging those fields for Correctable Errors by amending
__aer_print_error() and aer_print_error() with conditionals.

Sample log output for an Advisory Unsupported Request Error:

  pcieport 0001:00:00.4: AER: Multiple Correctable Error messages received, first one from 0001:0e:00.0
  idxd 0001:0e:00.0: PCIe Bus Error: severity=Correctable
  idxd 0001:0e:00.0:   device [8086:1216] error status/mask=00002000/00000000
  idxd 0001:0e:00.0:   [13] NonFatalErr       |             |
  idxd 0001:0e:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal)
  idxd 0001:0e:00.0:   device [8086:1216] error status/mask=00100000/00000000
  idxd 0001:0e:00.0:   [20] UnsupReq          | Receiver    | Transaction Layer (First)
  idxd 0001:0e:00.0: AER:   TLP Header (Flit): 0x01000104 0x00000000 0x0000080e 0x0f800001

This commit takes inspiration (but differs significantly) from an earlier
submission by Zhenzhong Duan, which in turn was based on a submission by
Qingshun Wang:

  https://lore.kernel.org/r/20240620025857.206647-1-zhenzhong.duan@intel.com/

Prior attempts at supporting Advisory Non-Fatal Errors were submitted by
Yicong Yang and Dio Sun:

  https://lore.kernel.org/r/1614689994-10925-1-git-send-email-yangyicong@hisilicon.com/
  https://lore.kernel.org/r/BJXPR01MB0614C01A9523786117B1F1CBCEC8A@BJXPR01MB0614.CHNPR01.prod.partner.outlook.cn/

Signed-off-by: Lukas Wunner <lukas@wunner.de>
[bhelgaas: fold in https://lore.kernel.org/all/amdnMg_J6T3Sys45@wunner.de,
https://lore.kernel.org/all/120da0565eac0157ffd913423c7cfa66e985ff59.1786800931.git.lukas@wunner.de]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Link: https://lore.kernel.org/r/20240620025857.206647-1-zhenzhong.duan@intel.com/
Link: https://lore.kernel.org/r/1614689994-10925-1-git-send-email-yangyicong@hisilicon.com/
Link: https://lore.kernel.org/r/BJXPR01MB0614C01A9523786117B1F1CBCEC8A@BJXPR01MB0614.CHNPR01.prod.partner.outlook.cn/
Link: https://patch.msgid.link/1b62915ffe06ee5b08e846531c42392e5f244337.1784905909.git.lukas@wunner.de

Some platforms require selectively disabling specific ASPM states on a
given PCIe link to avoid link instability or functional failures caused by
board-level connectivity constraints such as PCB routing, connectors,
slots, or external cabling.

Devicetree supports disabling ASPM L0s, L1, and L1 PM Substates via the
'aspm-no-l0s', 'aspm-no-l1' [1], and 'aspm-no-l1ss' [2] properties.
However, the ASPM driver does not currently honor these properties when
initializing the default link state.

When firmware enables L1 PM Substates before the kernel takes over,
masking aspm_support alone is insufficient to disable them in hardware.
pcie_config_aspm_link() guards L1SS configuration behind a check on
aspm_capable, which is derived from aspm_support. Once aspm_support is
masked, pcie_config_aspm_l1ss() is never called, leaving
firmware-enabled L1SS substates active in hardware.

Fix this by introducing pcie_link_has_aspm_override() to check for DT
override properties on either endpoint of the link. In
pcie_aspm_override_default_link_state(), use it to:

  - Mask aspm_support, aspm_default, and aspm_enabled for any disabled
    state, so software's view of the link stays in sync with what is
    actually programmed in hardware. Leaving aspm_enabled stale would make
    pcie_aspm_enabled() and the aspm sysfs attributes report a state as
    active even after it has been masked, and could cause
    pcie_config_aspm_link()'s "already in requested state" check to skip
    reprogramming hardware to match.

  - Explicitly call pcie_config_aspm_l1ss(link, 0) before masking
    aspm_support when firmware has L1SS active and DT requests disabling L1
    or L1SS, since pcie_config_aspm_link() will no longer do so once
    aspm_capable is derived from the masked aspm_support.

Move the aspm_default initialization and
pcie_aspm_override_default_link_state() call in pcie_aspm_cap_init() to
before the "Restore L0s/L1" block. pcie_aspm_cap_init() disables L1 in
hardware prior to aspm_l1ss_init() and re-enables it only in the restore
block. Calling pcie_config_aspm_l1ss() while L1 is already disabled
satisfies its precondition ("Caller must disable L1 first"), whereas the
previous placement after the restore violated it.

Since the restore block writes back the parent_lnkctl/child_lnkctl snapshot
taken from hardware before the DT override ran, mask the L0s and L1 enable
bits out of that snapshot for any state the override has just disabled in
aspm_support. Otherwise the restore step would unconditionally reprogram
the link back to firmware's original L0s/L1 configuration, defeating the
Devicetree override it is meant to enforce.

Move pcie_config_aspm_l1ss() earlier in the file so it can be called
from pcie_aspm_override_default_link_state().

Link [1]: https://github.com/devicetree-org/dt-schema/pull/188
Link [2]: https://github.com/devicetree-org/dt-schema/pull/190

Signed-off-by: Krishna Chaitanya Chundru <krishna.chundru@oss.qualcomm.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Link: https://patch.msgid.link/20260727-aspm-v6-3-2ebb3ee7ef71@oss.qualcomm.com

pcie_aspm_cap_init() disables ASPM L0s/L1 before touching L1SS config, then
restores the pre-existing state afterward. Both steps only ever touched
link->downstream, i.e. function 0 of the downstream component, leaving
sibling functions (>0) on a multi-function device untouched.

This means the "disable" step does not actually disable ASPM link-wide on a
multi-function device: a sibling function can still have L1 enabled even
after this step runs. PCIe r7.0, sec 7.5.3.7, recommends programming the
same ASPM Control value for all functions of a multi-function device, and
pcie_config_aspm_link() already loops over every function on the bus for
exactly this reason.

Loop over every function on linkbus->devices for both the disable and
restore steps, keeping the existing sec 7.5.3.7 ordering (disable
downstream functions before upstream, restore upstream before downstream
functions). The masked pcie_capability_clear_and_set_word() accessor from
the previous commit makes this safe: it only ever touches the ASPM Control
bits, so function-specific bits elsewhere in LNKCTL (e.g. Read Completion
Boundary, CLKREQ Enable) on sibling functions are left untouched.

Fixes: 7447990137bf ("PCI/ASPM: Disable L1 before disabling L1 PM Substates")
Closes: https://lore.kernel.org/all/20260721143945.86E7D1F000E9@smtp.kernel.org/
Signed-off-by: Krishna Chaitanya Chundru <krishna.chundru@oss.qualcomm.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Link: https://patch.msgid.link/20260727-aspm-v6-2-2ebb3ee7ef71@oss.qualcomm.com

pcie_aspm_cap_init() disables ASPM L0s/L1 on both ends of the Link
before touching L1SS config, then later restores the LNKCTL state
that was in effect beforehand. Both steps use raw
pcie_capability_write_word() calls: the disable step computes the
new value by hand from a snapshot taken earlier in the function, and
the restore step writes that same snapshot straight back.

Switch both steps to pcie_capability_clear_and_set_word(), masked to
PCI_EXP_LNKCTL_ASPMC, matching the accessor pcie_config_aspm_dev()
already uses elsewhere in this file for the exact same register. This
does a live read-modify-write of just the ASPM Control bits instead of
relying on a stale snapshot for the rest of the word, and is
consistent with how the rest of the file already touches this
register. No functional change.

Fixes: 7447990137bf ("PCI/ASPM: Disable L1 before disabling L1 PM Substates")
Closes: https://lore.kernel.org/all/20260721143945.86E7D1F000E9@smtp.kernel.org/
Signed-off-by: Krishna Chaitanya Chundru <krishna.chundru@oss.qualcomm.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Link: https://patch.msgid.link/20260727-aspm-v6-1-2ebb3ee7ef71@oss.qualcomm.com

The PCIe link, when down, needs to be recovered to bring it back. But on
some platforms, that cannot be done in a generic way as link recovery
procedure is platform specific. Add a new pci_host_handle_link_down() that
could be called by the host bridge drivers for a specific Root Port when
the link goes down.

pci_host_handle_link_down() accepts a 'pci_dev' corresponding to the Root
Port that observed the link down event. If CONFIG_PCIEAER is enabled, it
calls pcie_do_recovery() with 'pci_channel_io_frozen' as the state.  This
will result in the execution of the AER Fatal error handling code.  Since
the link down recovery is pretty much the same as AER Fatal error handling,
reuse pcie_do_recovery() here.

The AER .error_detected() callback will be triggered for all of the
downstream devices, but not for the Root Port itself as there is nothing to
do for the Root Ports in the callbacks. Finally, pci_host_reset_root_port()
will be called for the Root Port, which will reset the Root Port using the
.reset_root_port() callback to recover the link. Once that's done, resume
message will be broadcasted to the bridge and the downstream devices,
indicating successful link recovery.

But if CONFIG_PCIEAER is not enabled in the kernel, only
pci_host_reset_root_port() will be called, which will in turn call
pci_bus_error_reset() to just reset the Root Port as there is no way we
could inform the drivers about link recovery.

Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@oss.qualcomm.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Tested-by: Brian Norris <briannorris@chromium.org>
Tested-by: Krishna Chaitanya Chundru <krishna.chundru@oss.qualcomm.com>
Tested-by: Richard Zhu <hongxing.zhu@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260729-pci-port-reset-v9-3-53570b92064d@oss.qualcomm.com