kfd_dev_mapping caches the address_space of the first /dev/kfd opener
so that the GPU reset path can call unmap_mapping_range() to zap all
userspace mappings of doorbell and MMIO ranges.  This design has two
bugs that both manifest under SRIOV with multiple containers:

1. Use-after-free / rwsem deadlock.  The cached pointer refers to an
   inode owned by the first opener's container.  When that container
   exits and its inode is released, kfd_dev_mapping becomes a dangling
   pointer.  A subsequent GPU reset dereferences it inside
   unmap_mapping_range(), which takes i_mmap_rwsem on the freed inode,
   causing a hard hang observable as an uninterruptible rwsem wait.

2. Multi-container gap.  Only the first opener's address_space is cached;
   VMAs created by later openers live in a different address_space and
   are never reached by unmap_mapping_range().  After a GPU reset those
   stale mappings keep doorbell and MMIO pages accessible to guest
   userspace with no GPU behind them, risking PCIe transaction timeouts
   and NMI panics.

Fix both bugs with the same approach used by DRM core (drm_drv.c):
create a private pseudo-filesystem at module init time and allocate one
anonymous inode from it.  In kfd_open() redirect every opener's
file->f_mapping to that inode's address_space.  The inode is
module-owned, lives exactly as long as the amdgpu module, and collects
VMAs from all openers in one address_space.  A single
unmap_mapping_range() call in the reset path then correctly reaches
every container's mappings with no dangling pointer risk.

The hang manifests as an NMI backtrace on the GPU reset workqueue stuck
spinning in rwsem_down_read_slowpath() with a corrupted i_mmap_rwsem:

  Workqueue: amdgpu-reset-dev xgpu_ai_mailbox_flr_work [amdgpu]
  Call Trace:
   <TASK>
   kvm_wait+0x1f/0x40
   __pv_queued_spin_lock_slowpath+0x31d/0x3a0
   _raw_spin_lock_irq+0x51/0x80
   rwsem_down_read_slowpath+0xb3/0x550
   down_read+0x48/0xd0
   unmap_mapping_range+0x71/0x140
   kfd_dev_unmap_mapping_range+0x5b/0x140 [amdgpu]
   amdgpu_amdkfd_clear_kfd_mapping+0xd8/0x190 [amdgpu]
   amdgpu_device_gpu_recover+0x232/0x450 [amdgpu]
   xgpu_ai_mailbox_flr_work+0xb5/0xc0 [amdgpu]
   process_one_work+0x18e/0x3e0
   worker_thread+0x2e3/0x420
   kthread+0x10a/0x230

Fixes: 70cadefcc616 ("drm/amdgpu: unmap all user mappings of framebuffer and doorbell before mode1 reset")
Signed-off-by: Asad Kamal <asad.kamal@amd.com>
Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 1128b4a52de1572e87431de837fd9850cb99542c)
Cc: stable@vger.kernel.org

The low 6 bits of cp_hqd_eop_control store the base-2 logarithm
of the EOP ring size. This was calculated as

order_base_2(q->eop_ring_buffer_size / 4) - 1

But order_base_2 can in theory return 0, so this could underflow
(although in practice the ring buffer size cannot be less than 4096).

Change this to

order_base_2(q->eop_ring_buffer_size / 8)

using properties of logarithms.

Also add to the above comment to make the mathematics more clear.

Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit f0f43fcf8b2b3a924cad9444340921c96ed5f634)
Cc: stable@vger.kernel.org

The low 6 bits of cp_hqd_eop_control store the base-2 logarithm
of the EOP ring size. This was calculated as

ffs(q->eop_ring_buffer_size / sizeof(unsigned int)) - 1 - 1

But ffs can in theory return 1 or 0, so this could underflow
(although in practice the ring buffer size cannot be less than 4096).

Change this to

ffs(q->eop_ring_buffer_size / sizeof(unsigned int) / 4)

using properties of logarithms.

Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 4f18c56630383c14bfc6b2d65f88f2f895d2121a)
Cc: stable@vger.kernel.org

kfd_mqd_manager_v12.c (GFX 12.0) and kfd_mqd_manager_v12_1.c (GFX 12.1)
do not implement restore_mqd callbacks, leaving the function pointers
NULL and causing CRIU restore to return -EOPNOTSUPP on GFX12.

Implement restore_mqd for both compute and SDMA queues in
kfd_mqd_manager_v12.c and kfd_mqd_manager_v12_1.c, modeled after the
GFX 11 implementation with the following improvements:
- update cp_mqd_base_addr_lo/hi to the newly allocated MQD address,
  fixing a pre-existing gap shared with v11 where the in-MQD copy
  still pointed at the old checkpoint-time address after restore
- memset the full allocation before memcpy for compute queues to avoid
  stale data in the GTT sub-allocator tail; SDMA MQDs use sizeof(*m)
  since they are packed at mqd_size stride in a shared BO

checkpoint_mqd registration is deferred to a follow-up patch that also
implements get_checkpoint_info, so that checkpoint and restore are
enabled together as a complete and testable unit.

Note: GFX12.1 restore handles XCC0 only. Multi-XCC CRIU restore is
currently unreachable due to a separate validation issue in
kfd_criu_restore_queue(). A pr_warn_once() is emitted if a multi-XCC
device is encountered.

Signed-off-by: Vladimir Marioukhine <Vladimir.Marioukhine@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit b1f9601237d050f5df478464cf51bf1fff29a256)
Cc: stable@vger.kernel.org

Pull kmalloc_obj conversions from Kees Cook:
 "Another run of the Coccinelle script for converting kmalloc()
  family of allocations to kmalloc_obj() via the existing rules
  in scripts/coccinelle/api/kmalloc_objs.cocci"

* tag 'kmalloc_obj-v7.3-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux:
  treewide: refresh kmalloc_obj() conversions
  drm/amd/display: Fix harmless type mismatch in allocation

This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci

This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.

Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.

Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>

Reading /sys/kernel/debug/kfd/mqds while a process holds an active KFD
queue triggers a NULL pointer dereference because the for loop that
calls mqd_mgr->debugfs_show_mqd() is incorrectly placed outside the
if (pqn->q) block that initializes mqd_mgr.

The queue list can contain entries where pqn->q is NULL (kernel queues
where only pqn->kq is valid). In the original code:

  if (pqn->q) {
      ...
      mqd_mgr = q->device->dqm->mqd_mgrs[mqd_type];
      size = mqd_mgr->mqd_stride(...);
  }

  for (xcc = 0; xcc < num_xccs; xcc++) {  // WRONG: outside if block
      mqd = q->mqd + size * xcc;
      r = mqd_mgr->debugfs_show_mqd(m, mqd);
  }

When iterating over a queue node where pqn->q is NULL:
1. The if (pqn->q) block is skipped
2. mqd_mgr remains uninitialized (NULL from declaration)
3. The for loop executes anyway
4. mqd_mgr->debugfs_show_mqd(m, mqd) dereferences NULL

The crash manifests as:

  BUG: kernel NULL pointer dereference, address: 0000000000000000
  #PF: supervisor instruction fetch in kernel mode
  RIP: 0010:0x0
  Call Trace:
   pqm_debugfs_mqds+0x10c/0x1d0 [amdgpu]
   kfd_debugfs_mqds_by_process+0x9b/0x110 [amdgpu]
   seq_read_iter+0x132/0x4b0
   ...

Fix by moving the for loop inside the if (pqn->q) block, so mqd_mgr
and related variables are only used when properly initialized.

Closes: https://gitlab.freedesktop.org/drm/amd/-/work_items/5689
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Link: https://patch.msgid.link/20260831130051.2031435-1-mario.limonciello@amd.com
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 8bfe29d5c798940f797aa24135d2734c3ffce9de)
Cc: stable@vger.kernel.org

MES (Micro Engine Scheduler) does not perform heavy-weight TLB
invalidation after unmapping queues, unlike HWS which does this
automatically. This causes a race condition where in-flight DMA
descriptors can access memory that has been unmapped, leading to page
faults and GPU queue hangs during SVM page migration.

The issue manifests as KFDSVMRangeTest.MultiThreadMigrationTest
failures on gfx1151 (Strix Point) with XNACK mode 1 enabled - the GPU
compute queue hangs with packets submitted but never consumed.

Add kfd_flush_tlb() calls after MES queue removal in two locations:
- evict_process_queues_cpsch(): after all queues removed during eviction
- suspend_queues(): after debug/criu queue suspension (with mem_fence barrier)

This ensures all in-flight memory accesses from unmapped queues are
flushed before memory is freed or migrated.

Signed-off-by: Priya Hosur <Priya.Hosur@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit f5c4f88e0f9c45a8fb9dfac0c1df726c95e41b77)
Cc: stable@vger.kernel.org

Both create_queue_cpsch() and create_queue_nocpsch() unconditionally
call mqd_mgr->restore_mqd() when a CRIU restore is in progress
(qd != NULL), with no NULL guard. On any system where restore_mqd is
not implemented for the given queue type, a user holding
CAP_CHECKPOINT_RESTORE can trigger a kernel NULL pointer dereference
and panic the machine by issuing KFD_IOC_CRIU_OP_RESTORE with a
crafted queue restore object. Note that checkpoint_mqd is likewise
unimplemented on GFX12, so no legitimate CRIU image can reach this
path — only a hand-crafted restore payload.

Add a NULL guard for restore_mqd immediately after mqd_mgr is
resolved, unwinding via the existing error labels and returning
-EOPNOTSUPP if the callback is not implemented. This mirrors the
existing checkpoint_mqd guard in checkpoint_mqd().

Fixes: 48f0bdf4e38e ("drm/amdkfd: Added MQD manager files for GFX12.")
Cc: stable@vger.kernel.org
Signed-off-by: Vladimir Marioukhine <Vladimir.Marioukhine@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

When migration vm range is hole at cpu side(MIGRATE_PFN_MIGRATE set +
MIGRATE_PFN_VALID unset) driver still allocates device pages. There is no
dma map of src pages and migration. j is 0 and svm_migrate_copy_memory_gart()
will return an uninitialized r. That can trigger out_free_vram_pages to drop
all VRAM just set up.

Initialize r and only call the last svm_migrate_copy_memory_gart if j > 0.

Current code postponed the last page to the final copy. This patch flushes on
the last page when reach to the end of current drm_buddy_block; avoids another
svm_migrate_copy_memory_gart.

Cc: stable@vger.kernel.org
Signed-off-by: Xiaogang Chen <xiaogang.chen@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>