linux-toradex.git/drivers/pci/iov.c, branch v4.15-rc2

PCI: Restore ARI Capable Hierarchy before setting numVFs

2017-10-11T00:15:29+00:00

In the restore path, we previously read PCI_SRIOV_VF_OFFSET and
PCI_SRIOV_VF_STRIDE before restoring PCI_SRIOV_CTRL_ARI:

  pci_restore_state
    pci_restore_iov_state
      sriov_restore_state
        pci_iov_set_numvfs
          pci_read_config_word(... PCI_SRIOV_VF_OFFSET, &iov->offset)
          pci_read_config_word(... PCI_SRIOV_VF_STRIDE, &iov->stride)
        pci_write_config_word(... PCI_SRIOV_CTRL, iov->ctrl)

But per SR-IOV r1.1, sec 3.3.3.5, the device can use PCI_SRIOV_CTRL_ARI to
determine PCI_SRIOV_VF_OFFSET and PCI_SRIOV_VF_STRIDE.  Therefore, this
path, which is used for suspend/resume and AER recovery, can corrupt
iov->offset and iov->stride.

Since the iov state is associated with the device, not the driver, if we
reload the driver, it will use the the corrupted data, which may cause
crashes like this:

  kernel BUG at drivers/pci/iov.c:157!
  RIP: 0010:pci_iov_add_virtfn+0x2eb/0x350
  Call Trace:
   pci_enable_sriov+0x353/0x440
   ixgbe_pci_sriov_configure+0xd5/0x1f0 [ixgbe]
   sriov_numvfs_store+0xf7/0x170
   dev_attr_store+0x18/0x30
   sysfs_kf_write+0x37/0x40
   kernfs_fop_write+0x120/0x1b0
   vfs_write+0xb5/0x1a0
   SyS_write+0x55/0xc0

Restore PCI_SRIOV_CTRL_ARI before calling pci_iov_set_numvfs(), then
restore the rest of PCI_SRIOV_CTRL (which may set PCI_SRIOV_CTRL_VFE)
afterwards.

Signed-off-by: Tony Nguyen 
[bhelgaas: changelog, add comment, also clear ARI if necessary]
Signed-off-by: Bjorn Helgaas 
Acked-by: Alexander Duyck 
CC: Emil Tantilov

PCI: Create SR-IOV virtfn/physfn links before attaching driver

2017-10-11T00:13:38+00:00

When creating virtual functions, create the "virtfn%u" and "physfn" links
in sysfs *before* attaching the driver instead of after.  When we attach
the driver to the new virtual network interface first, there is a race when
the driver attaches to the new sends out an "add" udev event, and the
network interface naming software (biosdevname or systemd, for example)
tries to look at these links.

Signed-off-by: Stuart Hayes 
Signed-off-by: Bjorn Helgaas

PCI: Cache the VF device ID in the SR-IOV structure

2017-10-05T20:54:58+00:00

Cache the VF device ID in the SR-IOV structure and use it instead of
reading it over and over from the PF config space capability.

Signed-off-by: Filippo Sironi 
[bhelgaas: rename to "vf_device" to match pci_dev->device]
Signed-off-by: Bjorn Helgaas

PCI: Remove reset argument from pci_iov_{add,remove}_virtfn()

2017-10-05T20:54:56+00:00

The "reset" argument passed to pci_iov_add_virtfn() and
pci_iov_remove_virtfn() is always zero since 46cb7b1bd86f ("PCI: Remove
unused SR-IOV VF Migration support")

Remove the argument together with the associated code.

Signed-off-by: Jan H. Schönherr 
Signed-off-by: Bjorn Helgaas 
Acked-by: Russell Currey

PCI: Disable VF decoding before pcibios_sriov_disable() updates resources

2017-08-29T22:24:02+00:00

A struct resource represents the address space consumed by a device.  We
should not modify that resource while the device is actively using the
address space.  For VFs, pci_iov_update_resource() enforces this by
printing a warning and doing nothing if the VFE (VF Enable) and MSE (VF
Memory Space Enable) bits are set.

Previously, both sriov_enable() and sriov_disable() called the
pcibios_sriov_disable() arch hook, which may update the struct resource,
while VFE and MSE were enabled.  This effectively dropped the resource
update pcibios_sriov_disable() intended to do.

Disable VF memory decoding before calling pcibios_sriov_disable().

Reported-by: Carol L Soto 
Tested-by: Carol L Soto 
Signed-off-by: Gavin Shan 
Signed-off-by: Alexey Kardashevskiy 
[bhelgaas: changelog]
Signed-off-by: Bjorn Helgaas 
Cc: shan.gavin@gmail.com
Cc: Benjamin Herrenschmidt 
Cc: Paul Mackerras

PCI: Protect pci_driver->sriov_configure() usage with device_lock()

2017-06-14T22:41:19+00:00

Every method in struct device_driver or structures derived from it like
struct pci_driver MUST provide exclusion vs the driver's ->remove() method,
usually by using device_lock().

Protect use of pci_driver->sriov_configure() by holding the device lock
while calling it.

The PCI core sets the pci_dev->driver pointer in local_pci_probe() before
calling ->probe() and only clears it after ->remove().  This means driver's
->sriov_configure() callback will happily race with probe() and remove(),
most likely leading to BUGs, since drivers don't expect this.

Remove the iov lock completely, since we remove the last user.

[bhelgaas: changelog, thanks to Christoph for locking rule]
Link: http://lkml.kernel.org/r/20170522225023.14010-1-jakub.kicinski@netronome.com
Signed-off-by: Jakub Kicinski 
Signed-off-by: Bjorn Helgaas 
Reviewed-by: Christoph Hellwig

PCI: Add sysfs sriov_drivers_autoprobe to control VF driver binding

2017-04-20T13:53:51+00:00

Sometimes it is not desirable to bind SR-IOV VFs to drivers.  This can save
host side resource usage by VF instances that will be assigned to VMs.

Add a new PCI sysfs interface "sriov_drivers_autoprobe" to control that
from the PF.  To modify it, echo 0/n/N (disable probe) or 1/y/Y (enable
probe) to:

  /sys/bus/pci/devices//sriov_drivers_autoprobe

Note that this must be done before enabling VFs.  The change will not take
effect if VFs are already enabled.  Simply, one can disable VFs by setting
sriov_numvfs to 0, choose whether to probe or not, and then re-enable the
VFs by restoring sriov_numvfs.

[bhelgaas: changelog, ABI doc]
Signed-off-by: Bodong Wang 
Signed-off-by: Eli Cohen 
Signed-off-by: Bjorn Helgaas 
Reviewed-by: Gavin Shan 
Reviewed-by: Alex Williamson

PCI: Lock each enable/disable num_vfs operation in sysfs

2017-02-03T19:42:38+00:00

Enabling/disabling SRIOV via sysfs by echo-ing multiple values
simultaneously:

  # echo 63 > /sys/class/net/ethX/device/sriov_numvfs&
  # echo 63 > /sys/class/net/ethX/device/sriov_numvfs

  # sleep 5

  # echo 0 > /sys/class/net/ethX/device/sriov_numvfs&
  # echo 0 > /sys/class/net/ethX/device/sriov_numvfs

results in the following bug:

  kernel BUG at drivers/pci/iov.c:495!
  invalid opcode: 0000 [#1] SMP
  CPU: 1 PID: 8050 Comm: bash Tainted: G   W   4.9.0-rc7-net-next #2092
  RIP: 0010:[]
	    [] pci_iov_release+0x57/0x60

  Call Trace:
   [] pci_release_dev+0x26/0x70
   [] device_release+0x3e/0xb0
   [] kobject_cleanup+0x67/0x180
   [] kobject_put+0x2d/0x60
   [] put_device+0x17/0x20
   [] pci_dev_put+0x1a/0x20
   [] pci_get_dev_by_id+0x5b/0x90
   [] pci_get_subsys+0x35/0x40
   [] pci_get_device+0x18/0x20
   [] pci_get_domain_bus_and_slot+0x2b/0x60
   [] pci_iov_remove_virtfn+0x57/0x180
   [] pci_disable_sriov+0x65/0x140
   [] ixgbe_disable_sriov+0xc7/0x1d0 [ixgbe]
   [] ixgbe_pci_sriov_configure+0x3d/0x170 [ixgbe]
   [] sriov_numvfs_store+0xdc/0x130
  ...
  RIP  [] pci_iov_release+0x57/0x60

Use the existing mutex lock to protect each enable/disable operation.

Signed-off-by: Emil Tantilov 
Signed-off-by: Bjorn Helgaas 
Reviewed-by: Gavin Shan 
CC: Alexander Duyck

PCI: Remove pci_resource_bar() and pci_iov_resource_bar()

2016-11-30T00:05:09+00:00

pci_std_update_resource() only deals with standard BARs, so we don't have
to worry about the complications of VF BARs in an SR-IOV capability.

Compute the BAR address inline and remove pci_resource_bar().  That makes
pci_iov_resource_bar() unused, so remove that as well.

Signed-off-by: Bjorn Helgaas 
Reviewed-by: Gavin Shan

PCI: Don't update VF BARs while VF memory space is enabled

2016-11-30T00:05:09+00:00

If we update a VF BAR while it's enabled, there are two potential problems:

  1) Any driver that's using the VF has a cached BAR value that is stale
     after the update, and

  2) We can't update 64-bit BARs atomically, so the intermediate state
     (new lower dword with old upper dword) may conflict with another
     device, and an access by a driver unrelated to the VF may cause a bus
     error.

Warn about attempts to update VF BARs while they are enabled.  This is a
programming error, so use dev_WARN() to get a backtrace.

Signed-off-by: Bjorn Helgaas 
Reviewed-by: Gavin Shan