summaryrefslogtreecommitdiff
path: root/rust
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@linux-foundation.org>2026-04-15 08:45:00 -0700
committerLinus Torvalds <torvalds@linux-foundation.org>2026-04-15 08:45:00 -0700
commit4a57e0913e8c7fff407e97909f4ae48caa84d612 (patch)
tree96c5d9056a7f1dcaaca5f00749a298a60967b01b /rust
parentafac4c66d1aa6396ce44d94fe895d7b61e085fd4 (diff)
parent83e8d8bbffa8161e94f3aeee4dd09a35062a78c8 (diff)
Merge tag 'drm-next-2026-04-15' of https://gitlab.freedesktop.org/drm/kernel
Pull drm updates from Dave Airlie: "Highlights: - new DRM RAS infrastructure using netlink - amdgpu: enable DC on CIK APUs, and more IP enablement, and more user queue work - xe: purgeable BO support, and new hw enablement - dma-buf : add revocable operations Full summary: mm: - two-pass MMU interval notifiers - add gpu active/reclaim per-node stat counters math: - provide __KERNEL_DIV_ROUND_CLOSEST() in UAPI - implement DIV_ROUND_CLOSEST() with __KERNEL_DIV_ROUND_CLOSEST() rust: - shared tag with driver-core: register macro and io infra - core: rework DMA coherent API - core: add interop::list to interop with C linked lists - core: add more num::Bounded operations - core: enable generic_arg_infer and add EMSGSIZE - workqueue: add ARef<T> support for work and delayed work - add GPU buddy allocator abstraction - add DRM shmem GEM helper abstraction - allow drm:::Device to dispatch work and delayed work items to driver private data - add dma_resv_lock helper and raw accessors core: - introduce DRM RAS infrastructure over netlink - add connector panel_type property - fourcc: add ARM interleaved 64k modifier - colorop: add destroy helper - suballoc: split into alloc and init helpers - mode: provide DRM_ARGB_GET*() macros for reading color components edid: - provide drm_output_color_Format dma-buf: - provide revoke mechanism for shared buffers - rename move_notify to invalidate_mappings - always enable move_notify - protect dma_fence_ops with RCU and improve locking - clean pages with helpers atomic: - allocate drm_private_state via callback - helper: use system_percpu_wq buddy: - make buddy allocator available to gpu level - add kernel-doc for buddy allocator - improve aligned allocation ttm: - fix fence signalling - improve tests and docs - improve handling of gfp_retry_mayfail - use per-node stat counters to track memory allocations - port pool to use list_lru - drop NUMA specific pools - make pool shrinker numa aware - track allocated pages per numa node coreboot: - cleanup coreboot framebuffer support sched: - fix race condition in drm_sched_fini pagemap: - enable THP support - pass pagemap_addr by reference gem-shmem: - Track page accessed/dirty status across mmap/vmap gpusvm: - reenable device to device migration - fix unbalanced unclock bridge: - anx7625: Support USB-C plus DT bindings - connector: Fix EDID detection - dw-hdmi-qp: Support Vendor-Specfic and SDP Infoframes; improve others - fsl-ldb: Fix visual artifacts plus related DT property 'enable-termination-resistor' - imx8qxp-pixel-link: Improve bridge reference handling - lt9611: Support Port-B-only input plus DT bindings - tda998x: Support DRM_BRIDGE_ATTACH_NO_CONNECTOR; Clean up - Support TH1520 HDMI plus DT bindings - waveshare-dsi: Fix register and attach; Support 1..4 DSI lanes plus DT bindings - anx7625: Fix USB Type-C handling - cdns-mhdp8546-core: Handle HDCP state in bridge atomic_check - Support Lontium LT8713SX DP MST bridge plus DT bindings - analogix_dp: Use DP helpers for link training panel: - panel-jdi-lt070me05000: Use mipi-dsi multi functions - panel-edp: Support Add AUO B116XAT04.1 (HW: 1A); Support CMN N116BCL-EAK (C2); Support FriendlyELEC plus DT changes - panel-edp: Fix timings for BOE NV140WUM-N64 - ilitek-ili9882t: Allow GPIO calls to sleep - jadard: Support TAIGUAN XTI05101-01A - lxd: Support LXD M9189A plus DT bindings - mantix: Fix pixel clock; Clean up - motorola: Support Motorola Atrix 4G and Droid X2 plus DT bindings - novatek: Support Novatek/Tianma NT37700F plus DT bindings - simple: Support EDT ET057023UDBA plus DT bindings; Support Powertip PH800480T032-ZHC19 plus DT bindings; Support Waveshare 13.3" - novatek-nt36672a: Use mipi_dsi_*_multi() functions - panel-edp: Support BOE NV153WUM-N42, CMN N153JCA-ELK, CSW MNF307QS3-2 - support Himax HX83121A plus DT bindings - support JuTouch JT070TM041 plus DT bindings - support Samsung S6E8FC0 plus DT bindings - himax-hx83102c: support Samsung S6E8FC0 plus DT bindings; support backlight - ili9806e: support Rocktech RK050HR345-CT106A plus DT bindings - simple: support Tianma TM050RDH03 plus DT bindings amdgpu: - enable DC by default on CIK APUs - userq fence ioctl param size fixes - set panel_type to OLED for eDP - refactor DC i2c code - FAMS2 update - rework ttm handling to allow multiple engines - DC DCE 6.x cleanup - DC support for NUTMEG/TRAVIS DP bridge - DCN 4.2 support - GC12 idle power fix for compute - use struct drm_edid in non-DC code - enable NV12/P010 support on primary planes - support newer IP discovery tables - VCN/JPEG 5.0.2 support - GC/MES 12.1 updates - USERQ fixes - add DC idle state manager - eDP DSC seamless boot amdkfd: - GC 12.1 updates - non 4K page fixes xe: - basic Xe3p_LPG and NVL-P enabling patches - allow VM_BIND decompress support - add purgeable buffer object support - add xe_vm_get_property_ioctl - restrict multi-lrc to VCS/VECS engines - allow disabling VM overcommit in fault mode - dGPU memory optimizations - Workaround cleanups and simplification - Allow VFs VRAM quote changes using sysfs - convert GT stats to per-cpu counters - pagefault refactors - enable multi-queue on xe3p_xpc - disable DCC on PTL - make MMIO communication more robust - disable D3Cold for BMG on specific platforms - vfio: improve FLR sync for Xe VFIO i915/display: - C10/C20/LT PHY PLL divider verification - use trans push mechanism to generate PSR frame change on LNL+ - refactor DP DSC slice config - VGA decode refactoring - refactor DPT, gen2-4 overlay, masked field register macro helpers - refactor stolen memory allocation decisions - prepare for UHBR DP tunnels - refactor LT PHY PLL to use DPLL framework - implement register polling/waiting in display code - add shared stepping header between i915 and display i915: - fix potential overflow of shmem scatterlist length nouveau: - provide Z cull info to userspace - initial GA100 support - shutdown on PCI device shutdown nova-core: - harden GSP command queue - add support for large RPCs - simplify GSP sequencer and message handling - refactor falcon firmware handling - convert to new register macro - conver to new DMA coherent API - use checked arithmetic - add debugfs support for gsp-rm log buffers - fix aux device registration for multi-GPU msm: - CI: - Uprev mesa - Restore CI jobs for Qualcomm APQ8016 and APQ8096 devices - Core: - Switched to of_get_available_child_by_name() - DPU: - Fixes for DSC panels - Fixed brownout because of the frequency / OPP mismatch - Quad pipe preparation (not enabled yet) - Switched to virtual planes by default - Dropped VBIF_NRT support - Added support for Eliza platform - Reworked alpha handling - Switched to correct CWB definitions on Eliza - Dropped dummy INTF_0 on MSM8953 - Corrected INTFs related to DP-MST - DP: - Removed debug prints looking into PHY internals - DSI: - Fixes for DSC panels - RGB101010 support - Support for SC8280XP - Moved PHY bindings from display/ to phy/ - GPU: - Preemption support for x2-85 and a840 - IFPC support for a840 - SKU detection support for x2-85 and a840 - Expose AQE support (VK ray-pipeline) - Avoid locking in VM_BIND fence signaling path - Fix to avoid reclaim in GPU snapshot path - Disallow foreign mapping of _NO_SHARE BOs - HDMI: - Fixed infoframes programming - MDP5: - Dropped support for MSM8974v1 - Dropped now unused code for MSM8974 v1 and SDM660 / MSM8998 panthor: - add tracepoints for power and IRQs - fix fence handling - extend timestamp query with flags - support various sources for timestamp queries tyr: - fix names and model/versions rockchip: - vop2: use drm logging function - rk3576 displayport support - support CRTC background color atmel-hlcdc: - support sana5d65 LCD controller tilcdc: - use DT bindings schema - use managed DRM interfaces - support DRM_BRIDGE_ATTACH_NO_CONNECTOR verisilicon: - support DC8200 + DT bindings virtgpu: - support PRIME import with 3D enabled komeda: - fix integer overflow in AFBC checks mcde: - improve bridge handling gma500: - use drm client buffer for fbdev framebuffer amdxdna: - add sensors ioctls - provide NPU power estimate - support column utilization sensor - allow forcing DMA through IOMMU IOVA - support per-BO mem usage queries - refactor GEM implementation ivpu: - update boot API to v3.29.4 - limit per-user number of doorbells/contexts - perform engine reset on TDR error loongson: - replace custom code with drm_gem_ttm_dumb_map_offset() imx: - support planes behind the primary plane - fix bus-format selection vkms: - support CRTC background color v3d: - improve handling of struct v3d_stats komeda: - support Arm China Linlon D6 plus DT bindings imagination: - improve power-off sequence - support context-reset notification from firmware mediatek: - mtk_dsi: enable hs clock during pre-enable - Remove all conflicting aperture devices during probe - Add support for mt8167 display blocks" * tag 'drm-next-2026-04-15' of https://gitlab.freedesktop.org/drm/kernel: (1735 commits) drm/ttm/tests: Remove checks from ttm_pool_free_no_dma_alloc drm/ttm/tests: fix lru_count ASSERT drm/vram: remove DRM_VRAM_MM_FILE_OPERATIONS from docs drm/fb-helper: Fix a locking bug in an error path dma-fence: correct kernel-doc function parameter @flags ttm/pool: track allocated_pages per numa node. ttm/pool: make pool shrinker NUMA aware (v2) ttm/pool: drop numa specific pools ttm/pool: port to list_lru. (v2) drm/ttm: use gpu mm stats to track gpu memory allocations. (v4) mm: add gpu active/reclaim per-node stat counters (v2) gpu: nova-core: fix missing colon in SEC2 boot debug message gpu: nova-core: vbios: use from_le_bytes() for PCI ROM header parsing gpu: nova-core: bitfield: fix broken Default implementation gpu: nova-core: falcon: pad firmware DMA object size to required block alignment gpu: nova-core: gsp: fix undefined behavior in command queue code drm/shmem_helper: Make sure PMD entries get the writeable upgrade accel/ivpu: Trigger recovery on TDR with OS scheduling drm/msm: Use of_get_available_child_by_name() dt-bindings: display/msm: move DSI PHY bindings to phy/ subdir ...
Diffstat (limited to 'rust')
-rw-r--r--rust/bindings/bindings_helper.h14
-rw-r--r--rust/helpers/device.c5
-rw-r--r--rust/helpers/dma-resv.c14
-rw-r--r--rust/helpers/drm.c56
-rw-r--r--rust/helpers/gpu.c17
-rw-r--r--rust/helpers/helpers.c3
-rw-r--r--rust/helpers/list.c17
-rw-r--r--rust/kernel/device.rs15
-rw-r--r--rust/kernel/dma.rs899
-rw-r--r--rust/kernel/drm/device.rs87
-rw-r--r--rust/kernel/drm/driver.rs10
-rw-r--r--rust/kernel/drm/file.rs8
-rw-r--r--rust/kernel/drm/gem/mod.rs104
-rw-r--r--rust/kernel/drm/gem/shmem.rs228
-rw-r--r--rust/kernel/error.rs1
-rw-r--r--rust/kernel/gpu.rs6
-rw-r--r--rust/kernel/gpu/buddy.rs614
-rw-r--r--rust/kernel/interop.rs9
-rw-r--r--rust/kernel/interop/list.rs339
-rw-r--r--rust/kernel/lib.rs3
-rw-r--r--rust/kernel/uaccess.rs91
-rw-r--r--rust/kernel/workqueue.rs104
22 files changed, 2374 insertions, 270 deletions
diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h
index faf3ee634ced..446dbeaf0866 100644
--- a/rust/bindings/bindings_helper.h
+++ b/rust/bindings/bindings_helper.h
@@ -29,10 +29,12 @@
#include <linux/hrtimer_types.h>
#include <linux/acpi.h>
+#include <linux/gpu_buddy.h>
#include <drm/drm_device.h>
#include <drm/drm_drv.h>
#include <drm/drm_file.h>
#include <drm/drm_gem.h>
+#include <drm/drm_gem_shmem_helper.h>
#include <drm/drm_ioctl.h>
#include <kunit/test.h>
#include <linux/auxiliary_bus.h>
@@ -51,6 +53,7 @@
#include <linux/device/faux.h>
#include <linux/dma-direction.h>
#include <linux/dma-mapping.h>
+#include <linux/dma-resv.h>
#include <linux/errname.h>
#include <linux/ethtool.h>
#include <linux/fdtable.h>
@@ -61,6 +64,7 @@
#include <linux/interrupt.h>
#include <linux/io-pgtable.h>
#include <linux/ioport.h>
+#include <linux/iosys-map.h>
#include <linux/jiffies.h>
#include <linux/jump_label.h>
#include <linux/mdio.h>
@@ -146,6 +150,16 @@ const vm_flags_t RUST_CONST_HELPER_VM_MIXEDMAP = VM_MIXEDMAP;
const vm_flags_t RUST_CONST_HELPER_VM_HUGEPAGE = VM_HUGEPAGE;
const vm_flags_t RUST_CONST_HELPER_VM_NOHUGEPAGE = VM_NOHUGEPAGE;
+#if IS_ENABLED(CONFIG_GPU_BUDDY)
+const unsigned long RUST_CONST_HELPER_GPU_BUDDY_RANGE_ALLOCATION = GPU_BUDDY_RANGE_ALLOCATION;
+const unsigned long RUST_CONST_HELPER_GPU_BUDDY_TOPDOWN_ALLOCATION = GPU_BUDDY_TOPDOWN_ALLOCATION;
+const unsigned long RUST_CONST_HELPER_GPU_BUDDY_CONTIGUOUS_ALLOCATION =
+ GPU_BUDDY_CONTIGUOUS_ALLOCATION;
+const unsigned long RUST_CONST_HELPER_GPU_BUDDY_CLEAR_ALLOCATION = GPU_BUDDY_CLEAR_ALLOCATION;
+const unsigned long RUST_CONST_HELPER_GPU_BUDDY_CLEARED = GPU_BUDDY_CLEARED;
+const unsigned long RUST_CONST_HELPER_GPU_BUDDY_TRIM_DISABLE = GPU_BUDDY_TRIM_DISABLE;
+#endif
+
#if IS_ENABLED(CONFIG_ANDROID_BINDER_IPC_RUST)
#include "../../drivers/android/binder/rust_binder.h"
#include "../../drivers/android/binder/rust_binder_events.h"
diff --git a/rust/helpers/device.c b/rust/helpers/device.c
index a8ab931a9bd1..3be4ee590784 100644
--- a/rust/helpers/device.c
+++ b/rust/helpers/device.c
@@ -25,3 +25,8 @@ __rust_helper void rust_helper_dev_set_drvdata(struct device *dev, void *data)
{
dev_set_drvdata(dev, data);
}
+
+__rust_helper const char *rust_helper_dev_name(const struct device *dev)
+{
+ return dev_name(dev);
+}
diff --git a/rust/helpers/dma-resv.c b/rust/helpers/dma-resv.c
new file mode 100644
index 000000000000..71914d8241e2
--- /dev/null
+++ b/rust/helpers/dma-resv.c
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/dma-resv.h>
+
+__rust_helper
+int rust_helper_dma_resv_lock(struct dma_resv *obj, struct ww_acquire_ctx *ctx)
+{
+ return dma_resv_lock(obj, ctx);
+}
+
+__rust_helper void rust_helper_dma_resv_unlock(struct dma_resv *obj)
+{
+ dma_resv_unlock(obj);
+}
diff --git a/rust/helpers/drm.c b/rust/helpers/drm.c
index fe226f7b53ef..65f3f22b0e1d 100644
--- a/rust/helpers/drm.c
+++ b/rust/helpers/drm.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0
#include <drm/drm_gem.h>
+#include <drm/drm_gem_shmem_helper.h>
#include <drm/drm_vma_manager.h>
#ifdef CONFIG_DRM
@@ -21,4 +22,57 @@ rust_helper_drm_vma_node_offset_addr(struct drm_vma_offset_node *node)
return drm_vma_node_offset_addr(node);
}
-#endif
+#ifdef CONFIG_DRM_GEM_SHMEM_HELPER
+__rust_helper void
+rust_helper_drm_gem_shmem_object_free(struct drm_gem_object *obj)
+{
+ return drm_gem_shmem_object_free(obj);
+}
+
+__rust_helper void
+rust_helper_drm_gem_shmem_object_print_info(struct drm_printer *p, unsigned int indent,
+ const struct drm_gem_object *obj)
+{
+ drm_gem_shmem_object_print_info(p, indent, obj);
+}
+
+__rust_helper int
+rust_helper_drm_gem_shmem_object_pin(struct drm_gem_object *obj)
+{
+ return drm_gem_shmem_object_pin(obj);
+}
+
+__rust_helper void
+rust_helper_drm_gem_shmem_object_unpin(struct drm_gem_object *obj)
+{
+ drm_gem_shmem_object_unpin(obj);
+}
+
+__rust_helper struct sg_table *
+rust_helper_drm_gem_shmem_object_get_sg_table(struct drm_gem_object *obj)
+{
+ return drm_gem_shmem_object_get_sg_table(obj);
+}
+
+__rust_helper int
+rust_helper_drm_gem_shmem_object_vmap(struct drm_gem_object *obj,
+ struct iosys_map *map)
+{
+ return drm_gem_shmem_object_vmap(obj, map);
+}
+
+__rust_helper void
+rust_helper_drm_gem_shmem_object_vunmap(struct drm_gem_object *obj,
+ struct iosys_map *map)
+{
+ drm_gem_shmem_object_vunmap(obj, map);
+}
+
+__rust_helper int
+rust_helper_drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma)
+{
+ return drm_gem_shmem_object_mmap(obj, vma);
+}
+
+#endif /* CONFIG_DRM_GEM_SHMEM_HELPER */
+#endif /* CONFIG_DRM */
diff --git a/rust/helpers/gpu.c b/rust/helpers/gpu.c
new file mode 100644
index 000000000000..a25448d54d72
--- /dev/null
+++ b/rust/helpers/gpu.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/gpu_buddy.h>
+
+#ifdef CONFIG_GPU_BUDDY
+
+__rust_helper u64 rust_helper_gpu_buddy_block_offset(const struct gpu_buddy_block *block)
+{
+ return gpu_buddy_block_offset(block);
+}
+
+__rust_helper unsigned int rust_helper_gpu_buddy_block_order(struct gpu_buddy_block *block)
+{
+ return gpu_buddy_block_order(block);
+}
+
+#endif /* CONFIG_GPU_BUDDY */
diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c
index e05c6e7e4abb..625921e27dfb 100644
--- a/rust/helpers/helpers.c
+++ b/rust/helpers/helpers.c
@@ -57,13 +57,16 @@
#include "cred.c"
#include "device.c"
#include "dma.c"
+#include "dma-resv.c"
#include "drm.c"
#include "err.c"
#include "irq.c"
#include "fs.c"
+#include "gpu.c"
#include "io.c"
#include "jump_label.c"
#include "kunit.c"
+#include "list.c"
#include "maple_tree.c"
#include "mm.c"
#include "mutex.c"
diff --git a/rust/helpers/list.c b/rust/helpers/list.c
new file mode 100644
index 000000000000..18095a5593c5
--- /dev/null
+++ b/rust/helpers/list.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Helpers for C circular doubly linked list implementation.
+ */
+
+#include <linux/list.h>
+
+__rust_helper void rust_helper_INIT_LIST_HEAD(struct list_head *list)
+{
+ INIT_LIST_HEAD(list);
+}
+
+__rust_helper void rust_helper_list_add_tail(struct list_head *new, struct list_head *head)
+{
+ list_add_tail(new, head);
+}
diff --git a/rust/kernel/device.rs b/rust/kernel/device.rs
index 94e0548e7687..6d5396a43ebe 100644
--- a/rust/kernel/device.rs
+++ b/rust/kernel/device.rs
@@ -489,6 +489,17 @@ impl<Ctx: DeviceContext> Device<Ctx> {
// defined as a `#[repr(transparent)]` wrapper around `fwnode_handle`.
Some(unsafe { &*fwnode_handle.cast() })
}
+
+ /// Returns the name of the device.
+ ///
+ /// This is the kobject name of the device, or its initial name if the kobject is not yet
+ /// available.
+ #[inline]
+ pub fn name(&self) -> &CStr {
+ // SAFETY: By its type invariant `self.as_raw()` is a valid pointer to a `struct device`.
+ // The returned string is valid for the lifetime of the device.
+ unsafe { CStr::from_char_ptr(bindings::dev_name(self.as_raw())) }
+ }
}
// SAFETY: `Device` is a transparent wrapper of a type that doesn't depend on `Device`'s generic
@@ -575,7 +586,7 @@ pub struct CoreInternal;
/// The bound context indicates that for the entire duration of the lifetime of a [`Device<Bound>`]
/// reference, the [`Device`] is guaranteed to be bound to a driver.
///
-/// Some APIs, such as [`dma::CoherentAllocation`] or [`Devres`] rely on the [`Device`] to be bound,
+/// Some APIs, such as [`dma::Coherent`] or [`Devres`] rely on the [`Device`] to be bound,
/// which can be proven with the [`Bound`] device context.
///
/// Any abstraction that can guarantee a scope where the corresponding bus device is bound, should
@@ -584,7 +595,7 @@ pub struct CoreInternal;
///
/// [`Devres`]: kernel::devres::Devres
/// [`Devres::access`]: kernel::devres::Devres::access
-/// [`dma::CoherentAllocation`]: kernel::dma::CoherentAllocation
+/// [`dma::Coherent`]: kernel::dma::Coherent
pub struct Bound;
mod private {
diff --git a/rust/kernel/dma.rs b/rust/kernel/dma.rs
index a396f8435739..4995ee5dc689 100644
--- a/rust/kernel/dma.rs
+++ b/rust/kernel/dma.rs
@@ -5,14 +5,31 @@
//! C header: [`include/linux/dma-mapping.h`](srctree/include/linux/dma-mapping.h)
use crate::{
- bindings, build_assert, device,
- device::{Bound, Core},
- error::{to_result, Result},
+ bindings,
+ debugfs,
+ device::{
+ self,
+ Bound,
+ Core, //
+ },
+ error::to_result,
+ fs::file,
prelude::*,
+ ptr::KnownSize,
sync::aref::ARef,
- transmute::{AsBytes, FromBytes},
+ transmute::{
+ AsBytes,
+ FromBytes, //
+ }, //
+ uaccess::UserSliceWriter,
+};
+use core::{
+ ops::{
+ Deref,
+ DerefMut, //
+ },
+ ptr::NonNull, //
};
-use core::ptr::NonNull;
/// DMA address type.
///
@@ -39,7 +56,7 @@ pub trait Device: AsRef<device::Device<Core>> {
/// # Safety
///
/// This method must not be called concurrently with any DMA allocation or mapping primitives,
- /// such as [`CoherentAllocation::alloc_attrs`].
+ /// such as [`Coherent::zeroed`].
unsafe fn dma_set_mask(&self, mask: DmaMask) -> Result {
// SAFETY:
// - By the type invariant of `device::Device`, `self.as_ref().as_raw()` is valid.
@@ -56,7 +73,7 @@ pub trait Device: AsRef<device::Device<Core>> {
/// # Safety
///
/// This method must not be called concurrently with any DMA allocation or mapping primitives,
- /// such as [`CoherentAllocation::alloc_attrs`].
+ /// such as [`Coherent::zeroed`].
unsafe fn dma_set_coherent_mask(&self, mask: DmaMask) -> Result {
// SAFETY:
// - By the type invariant of `device::Device`, `self.as_ref().as_raw()` is valid.
@@ -75,7 +92,7 @@ pub trait Device: AsRef<device::Device<Core>> {
/// # Safety
///
/// This method must not be called concurrently with any DMA allocation or mapping primitives,
- /// such as [`CoherentAllocation::alloc_attrs`].
+ /// such as [`Coherent::zeroed`].
unsafe fn dma_set_mask_and_coherent(&self, mask: DmaMask) -> Result {
// SAFETY:
// - By the type invariant of `device::Device`, `self.as_ref().as_raw()` is valid.
@@ -94,7 +111,7 @@ pub trait Device: AsRef<device::Device<Core>> {
/// # Safety
///
/// This method must not be called concurrently with any DMA allocation or mapping primitives,
- /// such as [`CoherentAllocation::alloc_attrs`].
+ /// such as [`Coherent::zeroed`].
unsafe fn dma_set_max_seg_size(&self, size: u32) {
// SAFETY:
// - By the type invariant of `device::Device`, `self.as_ref().as_raw()` is valid.
@@ -194,12 +211,12 @@ impl DmaMask {
///
/// ```
/// # use kernel::device::{Bound, Device};
-/// use kernel::dma::{attrs::*, CoherentAllocation};
+/// use kernel::dma::{attrs::*, Coherent};
///
/// # fn test(dev: &Device<Bound>) -> Result {
/// let attribs = DMA_ATTR_FORCE_CONTIGUOUS | DMA_ATTR_NO_WARN;
-/// let c: CoherentAllocation<u64> =
-/// CoherentAllocation::alloc_attrs(dev, 4, GFP_KERNEL, attribs)?;
+/// let c: Coherent<[u64]> =
+/// Coherent::zeroed_slice_with_attrs(dev, 4, GFP_KERNEL, attribs)?;
/// # Ok::<(), Error>(()) }
/// ```
#[derive(Clone, Copy, PartialEq)]
@@ -250,9 +267,6 @@ pub mod attrs {
/// Specifies that writes to the mapping may be buffered to improve performance.
pub const DMA_ATTR_WRITE_COMBINE: Attrs = Attrs(bindings::DMA_ATTR_WRITE_COMBINE);
- /// Lets the platform to avoid creating a kernel virtual mapping for the allocated buffer.
- pub const DMA_ATTR_NO_KERNEL_MAPPING: Attrs = Attrs(bindings::DMA_ATTR_NO_KERNEL_MAPPING);
-
/// Allows platform code to skip synchronization of the CPU cache for the given buffer assuming
/// that it has been already transferred to 'device' domain.
pub const DMA_ATTR_SKIP_CPU_SYNC: Attrs = Attrs(bindings::DMA_ATTR_SKIP_CPU_SYNC);
@@ -344,182 +358,275 @@ impl From<DataDirection> for bindings::dma_data_direction {
}
}
-/// An abstraction of the `dma_alloc_coherent` API.
+/// CPU-owned DMA allocation that can be converted into a device-shared [`Coherent`] object.
///
-/// This is an abstraction around the `dma_alloc_coherent` API which is used to allocate and map
-/// large coherent DMA regions.
+/// Unlike [`Coherent`], a [`CoherentBox`] is guaranteed to be fully owned by the CPU -- its DMA
+/// address is not exposed and it cannot be accessed by a device. This means it can safely be used
+/// like a normal boxed allocation (e.g. direct reads, writes, and mutable slices are all safe).
///
-/// A [`CoherentAllocation`] instance contains a pointer to the allocated region (in the
-/// processor's virtual address space) and the device address which can be given to the device
-/// as the DMA address base of the region. The region is released once [`CoherentAllocation`]
-/// is dropped.
+/// A typical use is to allocate a [`CoherentBox`], populate it with normal CPU access, and then
+/// convert it into a [`Coherent`] object to share it with the device.
///
-/// # Invariants
+/// # Examples
///
-/// - For the lifetime of an instance of [`CoherentAllocation`], the `cpu_addr` is a valid pointer
-/// to an allocated region of coherent memory and `dma_handle` is the DMA address base of the
-/// region.
-/// - The size in bytes of the allocation is equal to `size_of::<T> * count`.
-/// - `size_of::<T> * count` fits into a `usize`.
-// TODO
-//
-// DMA allocations potentially carry device resources (e.g.IOMMU mappings), hence for soundness
-// reasons DMA allocation would need to be embedded in a `Devres` container, in order to ensure
-// that device resources can never survive device unbind.
-//
-// However, it is neither desirable nor necessary to protect the allocated memory of the DMA
-// allocation from surviving device unbind; it would require RCU read side critical sections to
-// access the memory, which may require subsequent unnecessary copies.
-//
-// Hence, find a way to revoke the device resources of a `CoherentAllocation`, but not the
-// entire `CoherentAllocation` including the allocated memory itself.
-pub struct CoherentAllocation<T: AsBytes + FromBytes> {
- dev: ARef<device::Device>,
- dma_handle: DmaAddress,
- count: usize,
- cpu_addr: NonNull<T>,
- dma_attrs: Attrs,
-}
+/// `CoherentBox<T>`:
+///
+/// ```
+/// # use kernel::device::{
+/// # Bound,
+/// # Device,
+/// # };
+/// use kernel::dma::{attrs::*,
+/// Coherent,
+/// CoherentBox,
+/// };
+///
+/// # fn test(dev: &Device<Bound>) -> Result {
+/// let mut dmem: CoherentBox<u64> = CoherentBox::zeroed(dev, GFP_KERNEL)?;
+/// *dmem = 42;
+/// let dmem: Coherent<u64> = dmem.into();
+/// # Ok::<(), Error>(()) }
+/// ```
+///
+/// `CoherentBox<[T]>`:
+///
+///
+/// ```
+/// # use kernel::device::{
+/// # Bound,
+/// # Device,
+/// # };
+/// use kernel::dma::{attrs::*,
+/// Coherent,
+/// CoherentBox,
+/// };
+///
+/// # fn test(dev: &Device<Bound>) -> Result {
+/// let mut dmem: CoherentBox<[u64]> = CoherentBox::zeroed_slice(dev, 4, GFP_KERNEL)?;
+/// dmem.fill(42);
+/// let dmem: Coherent<[u64]> = dmem.into();
+/// # Ok::<(), Error>(()) }
+/// ```
+pub struct CoherentBox<T: KnownSize + ?Sized>(Coherent<T>);
+
+impl<T: AsBytes + FromBytes> CoherentBox<[T]> {
+ /// [`CoherentBox`] variant of [`Coherent::zeroed_slice_with_attrs`].
+ #[inline]
+ pub fn zeroed_slice_with_attrs(
+ dev: &device::Device<Bound>,
+ count: usize,
+ gfp_flags: kernel::alloc::Flags,
+ dma_attrs: Attrs,
+ ) -> Result<Self> {
+ Coherent::zeroed_slice_with_attrs(dev, count, gfp_flags, dma_attrs).map(Self)
+ }
+
+ /// Same as [CoherentBox::zeroed_slice_with_attrs], but with `dma::Attrs(0)`.
+ #[inline]
+ pub fn zeroed_slice(
+ dev: &device::Device<Bound>,
+ count: usize,
+ gfp_flags: kernel::alloc::Flags,
+ ) -> Result<Self> {
+ Self::zeroed_slice_with_attrs(dev, count, gfp_flags, Attrs(0))
+ }
-impl<T: AsBytes + FromBytes> CoherentAllocation<T> {
- /// Allocates a region of `size_of::<T> * count` of coherent memory.
+ /// Initializes the element at `i` using the given initializer.
+ ///
+ /// Returns `EINVAL` if `i` is out of bounds.
+ pub fn init_at<E>(&mut self, i: usize, init: impl Init<T, E>) -> Result
+ where
+ Error: From<E>,
+ {
+ if i >= self.0.len() {
+ return Err(EINVAL);
+ }
+
+ let ptr = &raw mut self[i];
+
+ // SAFETY:
+ // - `ptr` is valid, properly aligned, and within this allocation.
+ // - `T: AsBytes + FromBytes` guarantees all bit patterns are valid, so partial writes on
+ // error cannot leave the element in an invalid state.
+ // - The DMA address has not been exposed yet, so there is no concurrent device access.
+ unsafe { init.__init(ptr)? };
+
+ Ok(())
+ }
+
+ /// Allocates a region of coherent memory of the same size as `data` and initializes it with a
+ /// copy of its contents.
+ ///
+ /// This is the [`CoherentBox`] variant of [`Coherent::from_slice_with_attrs`].
///
/// # Examples
///
/// ```
+ /// use core::ops::Deref;
+ ///
/// # use kernel::device::{Bound, Device};
- /// use kernel::dma::{attrs::*, CoherentAllocation};
+ /// use kernel::dma::{
+ /// attrs::*,
+ /// CoherentBox
+ /// };
///
/// # fn test(dev: &Device<Bound>) -> Result {
- /// let c: CoherentAllocation<u64> =
- /// CoherentAllocation::alloc_attrs(dev, 4, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
+ /// let data = [0u8, 1u8, 2u8, 3u8];
+ /// let c: CoherentBox<[u8]> =
+ /// CoherentBox::from_slice_with_attrs(dev, &data, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
+ ///
+ /// assert_eq!(c.deref(), &data);
/// # Ok::<(), Error>(()) }
/// ```
- pub fn alloc_attrs(
+ pub fn from_slice_with_attrs(
dev: &device::Device<Bound>,
- count: usize,
+ data: &[T],
gfp_flags: kernel::alloc::Flags,
dma_attrs: Attrs,
- ) -> Result<CoherentAllocation<T>> {
- build_assert!(
- core::mem::size_of::<T>() > 0,
- "It doesn't make sense for the allocated type to be a ZST"
- );
-
- let size = count
- .checked_mul(core::mem::size_of::<T>())
- .ok_or(EOVERFLOW)?;
- let mut dma_handle = 0;
- // SAFETY: Device pointer is guaranteed as valid by the type invariant on `Device`.
- let addr = unsafe {
- bindings::dma_alloc_attrs(
- dev.as_raw(),
- size,
- &mut dma_handle,
- gfp_flags.as_raw(),
- dma_attrs.as_raw(),
- )
- };
- let addr = NonNull::new(addr).ok_or(ENOMEM)?;
- // INVARIANT:
- // - We just successfully allocated a coherent region which is accessible for
- // `count` elements, hence the cpu address is valid. We also hold a refcounted reference
- // to the device.
- // - The allocated `size` is equal to `size_of::<T> * count`.
- // - The allocated `size` fits into a `usize`.
- Ok(Self {
- dev: dev.into(),
- dma_handle,
- count,
- cpu_addr: addr.cast(),
+ ) -> Result<Self>
+ where
+ T: Copy,
+ {
+ let mut slice = Self(Coherent::<T>::alloc_slice_with_attrs(
+ dev,
+ data.len(),
+ gfp_flags,
dma_attrs,
- })
+ )?);
+
+ // PANIC: `slice` was created with length `data.len()`.
+ slice.copy_from_slice(data);
+
+ Ok(slice)
}
- /// Performs the same functionality as [`CoherentAllocation::alloc_attrs`], except the
+ /// Performs the same functionality as [`CoherentBox::from_slice_with_attrs`], except the
/// `dma_attrs` is 0 by default.
- pub fn alloc_coherent(
+ #[inline]
+ pub fn from_slice(
dev: &device::Device<Bound>,
- count: usize,
+ data: &[T],
gfp_flags: kernel::alloc::Flags,
- ) -> Result<CoherentAllocation<T>> {
- CoherentAllocation::alloc_attrs(dev, count, gfp_flags, Attrs(0))
+ ) -> Result<Self>
+ where
+ T: Copy,
+ {
+ Self::from_slice_with_attrs(dev, data, gfp_flags, Attrs(0))
}
+}
- /// Returns the number of elements `T` in this allocation.
- ///
- /// Note that this is not the size of the allocation in bytes, which is provided by
- /// [`Self::size`].
- pub fn count(&self) -> usize {
- self.count
+impl<T: AsBytes + FromBytes> CoherentBox<T> {
+ /// Same as [`CoherentBox::zeroed_slice_with_attrs`], but for a single element.
+ #[inline]
+ pub fn zeroed_with_attrs(
+ dev: &device::Device<Bound>,
+ gfp_flags: kernel::alloc::Flags,
+ dma_attrs: Attrs,
+ ) -> Result<Self> {
+ Coherent::zeroed_with_attrs(dev, gfp_flags, dma_attrs).map(Self)
}
- /// Returns the size in bytes of this allocation.
- pub fn size(&self) -> usize {
- // INVARIANT: The type invariant of `Self` guarantees that `size_of::<T> * count` fits into
- // a `usize`.
- self.count * core::mem::size_of::<T>()
+ /// Same as [`CoherentBox::zeroed_slice`], but for a single element.
+ #[inline]
+ pub fn zeroed(dev: &device::Device<Bound>, gfp_flags: kernel::alloc::Flags) -> Result<Self> {
+ Self::zeroed_with_attrs(dev, gfp_flags, Attrs(0))
}
+}
+
+impl<T: KnownSize + ?Sized> Deref for CoherentBox<T> {
+ type Target = T;
- /// Returns the raw pointer to the allocated region in the CPU's virtual address space.
#[inline]
- pub fn as_ptr(&self) -> *const [T] {
- core::ptr::slice_from_raw_parts(self.cpu_addr.as_ptr(), self.count)
+ fn deref(&self) -> &Self::Target {
+ // SAFETY:
+ // - We have not exposed the DMA address yet, so there can't be any concurrent access by a
+ // device.
+ // - We have exclusive access to `self.0`.
+ unsafe { self.0.as_ref() }
}
+}
- /// Returns the raw pointer to the allocated region in the CPU's virtual address space as
- /// a mutable pointer.
+impl<T: AsBytes + FromBytes + KnownSize + ?Sized> DerefMut for CoherentBox<T> {
#[inline]
- pub fn as_mut_ptr(&self) -> *mut [T] {
- core::ptr::slice_from_raw_parts_mut(self.cpu_addr.as_ptr(), self.count)
+ fn deref_mut(&mut self) -> &mut Self::Target {
+ // SAFETY:
+ // - We have not exposed the DMA address yet, so there can't be any concurrent access by a
+ // device.
+ // - We have exclusive access to `self.0`.
+ unsafe { self.0.as_mut() }
+ }
+}
+
+impl<T: AsBytes + FromBytes + KnownSize + ?Sized> From<CoherentBox<T>> for Coherent<T> {
+ #[inline]
+ fn from(value: CoherentBox<T>) -> Self {
+ value.0
+ }
+}
+
+/// An abstraction of the `dma_alloc_coherent` API.
+///
+/// This is an abstraction around the `dma_alloc_coherent` API which is used to allocate and map
+/// large coherent DMA regions.
+///
+/// A [`Coherent`] instance contains a pointer to the allocated region (in the
+/// processor's virtual address space) and the device address which can be given to the device
+/// as the DMA address base of the region. The region is released once [`Coherent`]
+/// is dropped.
+///
+/// # Invariants
+///
+/// - For the lifetime of an instance of [`Coherent`], the `cpu_addr` is a valid pointer
+/// to an allocated region of coherent memory and `dma_handle` is the DMA address base of the
+/// region.
+/// - The size in bytes of the allocation is equal to size information via pointer.
+// TODO
+//
+// DMA allocations potentially carry device resources (e.g.IOMMU mappings), hence for soundness
+// reasons DMA allocation would need to be embedded in a `Devres` container, in order to ensure
+// that device resources can never survive device unbind.
+//
+// However, it is neither desirable nor necessary to protect the allocated memory of the DMA
+// allocation from surviving device unbind; it would require RCU read side critical sections to
+// access the memory, which may require subsequent unnecessary copies.
+//
+// Hence, find a way to revoke the device resources of a `Coherent`, but not the
+// entire `Coherent` including the allocated memory itself.
+pub struct Coherent<T: KnownSize + ?Sized> {
+ dev: ARef<device::Device>,
+ dma_handle: DmaAddress,
+ cpu_addr: NonNull<T>,
+ dma_attrs: Attrs,
+}
+
+impl<T: KnownSize + ?Sized> Coherent<T> {
+ /// Returns the size in bytes of this allocation.
+ #[inline]
+ pub fn size(&self) -> usize {
+ T::size(self.cpu_addr.as_ptr())
}
- /// Returns the base address to the allocated region in the CPU's virtual address space.
- pub fn start_ptr(&self) -> *const T {
+ /// Returns the raw pointer to the allocated region in the CPU's virtual address space.
+ #[inline]
+ pub fn as_ptr(&self) -> *const T {
self.cpu_addr.as_ptr()
}
- /// Returns the base address to the allocated region in the CPU's virtual address space as
+ /// Returns the raw pointer to the allocated region in the CPU's virtual address space as
/// a mutable pointer.
- pub fn start_ptr_mut(&mut self) -> *mut T {
+ #[inline]
+ pub fn as_mut_ptr(&self) -> *mut T {
self.cpu_addr.as_ptr()
}
/// Returns a DMA handle which may be given to the device as the DMA address base of
/// the region.
+ #[inline]
pub fn dma_handle(&self) -> DmaAddress {
self.dma_handle
}
- /// Returns a DMA handle starting at `offset` (in units of `T`) which may be given to the
- /// device as the DMA address base of the region.
- ///
- /// Returns `EINVAL` if `offset` is not within the bounds of the allocation.
- pub fn dma_handle_with_offset(&self, offset: usize) -> Result<DmaAddress> {
- if offset >= self.count {
- Err(EINVAL)
- } else {
- // INVARIANT: The type invariant of `Self` guarantees that `size_of::<T> * count` fits
- // into a `usize`, and `offset` is inferior to `count`.
- Ok(self.dma_handle + (offset * core::mem::size_of::<T>()) as DmaAddress)
- }
- }
-
- /// Common helper to validate a range applied from the allocated region in the CPU's virtual
- /// address space.
- fn validate_range(&self, offset: usize, count: usize) -> Result {
- if offset.checked_add(count).ok_or(EOVERFLOW)? > self.count {
- return Err(EINVAL);
- }
- Ok(())
- }
-
- /// Returns the data from the region starting from `offset` as a slice.
- /// `offset` and `count` are in units of `T`, not the number of bytes.
- ///
- /// For ringbuffer type of r/w access or use-cases where the pointer to the live data is needed,
- /// [`CoherentAllocation::start_ptr`] or [`CoherentAllocation::start_ptr_mut`] could be used
- /// instead.
+ /// Returns a reference to the data in the region.
///
/// # Safety
///
@@ -527,19 +634,13 @@ impl<T: AsBytes + FromBytes> CoherentAllocation<T> {
/// slice is live.
/// * Callers must ensure that this call does not race with a write to the same region while
/// the returned slice is live.
- pub unsafe fn as_slice(&self, offset: usize, count: usize) -> Result<&[T]> {
- self.validate_range(offset, count)?;
- // SAFETY:
- // - The pointer is valid due to type invariant on `CoherentAllocation`,
- // we've just checked that the range and index is within bounds. The immutability of the
- // data is also guaranteed by the safety requirements of the function.
- // - `offset + count` can't overflow since it is smaller than `self.count` and we've checked
- // that `self.count` won't overflow early in the constructor.
- Ok(unsafe { core::slice::from_raw_parts(self.start_ptr().add(offset), count) })
+ #[inline]
+ pub unsafe fn as_ref(&self) -> &T {
+ // SAFETY: per safety requirement.
+ unsafe { &*self.as_ptr() }
}
- /// Performs the same functionality as [`CoherentAllocation::as_slice`], except that a mutable
- /// slice is returned.
+ /// Returns a mutable reference to the data in the region.
///
/// # Safety
///
@@ -547,51 +648,11 @@ impl<T: AsBytes + FromBytes> CoherentAllocation<T> {
/// slice is live.
/// * Callers must ensure that this call does not race with a read or write to the same region
/// while the returned slice is live.
- pub unsafe fn as_slice_mut(&mut self, offset: usize, count: usize) -> Result<&mut [T]> {
- self.validate_range(offset, count)?;
- // SAFETY:
- // - The pointer is valid due to type invariant on `CoherentAllocation`,
- // we've just checked that the range and index is within bounds. The immutability of the
- // data is also guaranteed by the safety requirements of the function.
- // - `offset + count` can't overflow since it is smaller than `self.count` and we've checked
- // that `self.count` won't overflow early in the constructor.
- Ok(unsafe { core::slice::from_raw_parts_mut(self.start_ptr_mut().add(offset), count) })
- }
-
- /// Writes data to the region starting from `offset`. `offset` is in units of `T`, not the
- /// number of bytes.
- ///
- /// # Safety
- ///
- /// * Callers must ensure that this call does not race with a read or write to the same region
- /// that overlaps with this write.
- ///
- /// # Examples
- ///
- /// ```
- /// # fn test(alloc: &mut kernel::dma::CoherentAllocation<u8>) -> Result {
- /// let somedata: [u8; 4] = [0xf; 4];
- /// let buf: &[u8] = &somedata;
- /// // SAFETY: There is no concurrent HW operation on the device and no other R/W access to the
- /// // region.
- /// unsafe { alloc.write(buf, 0)?; }
- /// # Ok::<(), Error>(()) }
- /// ```
- pub unsafe fn write(&mut self, src: &[T], offset: usize) -> Result {
- self.validate_range(offset, src.len())?;
- // SAFETY:
- // - The pointer is valid due to type invariant on `CoherentAllocation`
- // and we've just checked that the range and index is within bounds.
- // - `offset + count` can't overflow since it is smaller than `self.count` and we've checked
- // that `self.count` won't overflow early in the constructor.
- unsafe {
- core::ptr::copy_nonoverlapping(
- src.as_ptr(),
- self.start_ptr_mut().add(offset),
- src.len(),
- )
- };
- Ok(())
+ #[expect(clippy::mut_from_ref, reason = "unsafe to use API")]
+ #[inline]
+ pub unsafe fn as_mut(&self) -> &mut T {
+ // SAFETY: per safety requirement.
+ unsafe { &mut *self.as_mut_ptr() }
}
/// Reads the value of `field` and ensures that its type is [`FromBytes`].
@@ -641,18 +702,276 @@ impl<T: AsBytes + FromBytes> CoherentAllocation<T> {
}
}
+impl<T: AsBytes + FromBytes> Coherent<T> {
+ /// Allocates a region of `T` of coherent memory.
+ fn alloc_with_attrs(
+ dev: &device::Device<Bound>,
+ gfp_flags: kernel::alloc::Flags,
+ dma_attrs: Attrs,
+ ) -> Result<Self> {
+ const {
+ assert!(
+ core::mem::size_of::<T>() > 0,
+ "It doesn't make sense for the allocated type to be a ZST"
+ );
+ }
+
+ let mut dma_handle = 0;
+ // SAFETY: Device pointer is guaranteed as valid by the type invariant on `Device`.
+ let addr = unsafe {
+ bindings::dma_alloc_attrs(
+ dev.as_raw(),
+ core::mem::size_of::<T>(),
+ &mut dma_handle,
+ gfp_flags.as_raw(),
+ dma_attrs.as_raw(),
+ )
+ };
+ let cpu_addr = NonNull::new(addr.cast()).ok_or(ENOMEM)?;
+ // INVARIANT:
+ // - We just successfully allocated a coherent region which is adequately sized for `T`,
+ // hence the cpu address is valid.
+ // - We also hold a refcounted reference to the device.
+ Ok(Self {
+ dev: dev.into(),
+ dma_handle,
+ cpu_addr,
+ dma_attrs,
+ })
+ }
+
+ /// Allocates a region of type `T` of coherent memory.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # use kernel::device::{
+ /// # Bound,
+ /// # Device,
+ /// # };
+ /// use kernel::dma::{
+ /// attrs::*,
+ /// Coherent,
+ /// };
+ ///
+ /// # fn test(dev: &Device<Bound>) -> Result {
+ /// let c: Coherent<[u64; 4]> =
+ /// Coherent::zeroed_with_attrs(dev, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
+ /// # Ok::<(), Error>(()) }
+ /// ```
+ #[inline]
+ pub fn zeroed_with_attrs(
+ dev: &device::Device<Bound>,
+ gfp_flags: kernel::alloc::Flags,
+ dma_attrs: Attrs,
+ ) -> Result<Self> {
+ Self::alloc_with_attrs(dev, gfp_flags | __GFP_ZERO, dma_attrs)
+ }
+
+ /// Performs the same functionality as [`Coherent::zeroed_with_attrs`], except the
+ /// `dma_attrs` is 0 by default.
+ #[inline]
+ pub fn zeroed(dev: &device::Device<Bound>, gfp_flags: kernel::alloc::Flags) -> Result<Self> {
+ Self::zeroed_with_attrs(dev, gfp_flags, Attrs(0))
+ }
+
+ /// Same as [`Coherent::zeroed_with_attrs`], but instead of a zero-initialization the memory is
+ /// initialized with `init`.
+ pub fn init_with_attrs<E>(
+ dev: &device::Device<Bound>,
+ gfp_flags: kernel::alloc::Flags,
+ dma_attrs: Attrs,
+ init: impl Init<T, E>,
+ ) -> Result<Self>
+ where
+ Error: From<E>,
+ {
+ let dmem = Self::alloc_with_attrs(dev, gfp_flags, dma_attrs)?;
+ let ptr = dmem.as_mut_ptr();
+
+ // SAFETY:
+ // - `ptr` is valid, properly aligned, and points to exclusively owned memory.
+ // - If `__init` fails, `self` is dropped, which safely frees the underlying `Coherent`'s
+ // DMA memory. `T: AsBytes + FromBytes` ensures there are no complex `Drop` requirements
+ // we are bypassing.
+ unsafe { init.__init(ptr)? };
+
+ Ok(dmem)
+ }
+
+ /// Same as [`Coherent::zeroed`], but instead of a zero-initialization the memory is initialized
+ /// with `init`.
+ #[inline]
+ pub fn init<E>(
+ dev: &device::Device<Bound>,
+ gfp_flags: kernel::alloc::Flags,
+ init: impl Init<T, E>,
+ ) -> Result<Self>
+ where
+ Error: From<E>,
+ {
+ Self::init_with_attrs(dev, gfp_flags, Attrs(0), init)
+ }
+
+ /// Allocates a region of `[T; len]` of coherent memory.
+ fn alloc_slice_with_attrs(
+ dev: &device::Device<Bound>,
+ len: usize,
+ gfp_flags: kernel::alloc::Flags,
+ dma_attrs: Attrs,
+ ) -> Result<Coherent<[T]>> {
+ const {
+ assert!(
+ core::mem::size_of::<T>() > 0,
+ "It doesn't make sense for the allocated type to be a ZST"
+ );
+ }
+
+ // `dma_alloc_attrs` cannot handle zero-length allocation, bail early.
+ if len == 0 {
+ Err(EINVAL)?;
+ }
+
+ let size = core::mem::size_of::<T>().checked_mul(len).ok_or(ENOMEM)?;
+ let mut dma_handle = 0;
+ // SAFETY: Device pointer is guaranteed as valid by the type invariant on `Device`.
+ let addr = unsafe {
+ bindings::dma_alloc_attrs(
+ dev.as_raw(),
+ size,
+ &mut dma_handle,
+ gfp_flags.as_raw(),
+ dma_attrs.as_raw(),
+ )
+ };
+ let cpu_addr = NonNull::slice_from_raw_parts(NonNull::new(addr.cast()).ok_or(ENOMEM)?, len);
+ // INVARIANT:
+ // - We just successfully allocated a coherent region which is adequately sized for
+ // `[T; len]`, hence the cpu address is valid.
+ // - We also hold a refcounted reference to the device.
+ Ok(Coherent {
+ dev: dev.into(),
+ dma_handle,
+ cpu_addr,
+ dma_attrs,
+ })
+ }
+
+ /// Allocates a zeroed region of type `T` of coherent memory.
+ ///
+ /// Unlike `Coherent::<[T; N]>::zeroed_with_attrs`, `Coherent::<T>::zeroed_slices` support
+ /// a runtime length.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # use kernel::device::{
+ /// # Bound,
+ /// # Device,
+ /// # };
+ /// use kernel::dma::{
+ /// attrs::*,
+ /// Coherent,
+ /// };
+ ///
+ /// # fn test(dev: &Device<Bound>) -> Result {
+ /// let c: Coherent<[u64]> =
+ /// Coherent::zeroed_slice_with_attrs(dev, 4, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
+ /// # Ok::<(), Error>(()) }
+ /// ```
+ #[inline]
+ pub fn zeroed_slice_with_attrs(
+ dev: &device::Device<Bound>,
+ len: usize,
+ gfp_flags: kernel::alloc::Flags,
+ dma_attrs: Attrs,
+ ) -> Result<Coherent<[T]>> {
+ Coherent::alloc_slice_with_attrs(dev, len, gfp_flags | __GFP_ZERO, dma_attrs)
+ }
+
+ /// Performs the same functionality as [`Coherent::zeroed_slice_with_attrs`], except the
+ /// `dma_attrs` is 0 by default.
+ #[inline]
+ pub fn zeroed_slice(
+ dev: &device::Device<Bound>,
+ len: usize,
+ gfp_flags: kernel::alloc::Flags,
+ ) -> Result<Coherent<[T]>> {
+ Self::zeroed_slice_with_attrs(dev, len, gfp_flags, Attrs(0))
+ }
+
+ /// Allocates a region of coherent memory of the same size as `data` and initializes it with a
+ /// copy of its contents.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # use kernel::device::{Bound, Device};
+ /// use kernel::dma::{
+ /// attrs::*,
+ /// Coherent
+ /// };
+ ///
+ /// # fn test(dev: &Device<Bound>) -> Result {
+ /// let data = [0u8, 1u8, 2u8, 3u8];
+ /// // `c` has the same content as `data`.
+ /// let c: Coherent<[u8]> =
+ /// Coherent::from_slice_with_attrs(dev, &data, GFP_KERNEL, DMA_ATTR_NO_WARN)?;
+ ///
+ /// # Ok::<(), Error>(()) }
+ /// ```
+ #[inline]
+ pub fn from_slice_with_attrs(
+ dev: &device::Device<Bound>,
+ data: &[T],
+ gfp_flags: kernel::alloc::Flags,
+ dma_attrs: Attrs,
+ ) -> Result<Coherent<[T]>>
+ where
+ T: Copy,
+ {
+ CoherentBox::from_slice_with_attrs(dev, data, gfp_flags, dma_attrs).map(Into::into)
+ }
+
+ /// Performs the same functionality as [`Coherent::from_slice_with_attrs`], except the
+ /// `dma_attrs` is 0 by default.
+ #[inline]
+ pub fn from_slice(
+ dev: &device::Device<Bound>,
+ data: &[T],
+ gfp_flags: kernel::alloc::Flags,
+ ) -> Result<Coherent<[T]>>
+ where
+ T: Copy,
+ {
+ Self::from_slice_with_attrs(dev, data, gfp_flags, Attrs(0))
+ }
+}
+
+impl<T> Coherent<[T]> {
+ /// Returns the number of elements `T` in this allocation.
+ ///
+ /// Note that this is not the size of the allocation in bytes, which is provided by
+ /// [`Self::size`].
+ #[inline]
+ #[expect(clippy::len_without_is_empty, reason = "Coherent slice is never empty")]
+ pub fn len(&self) -> usize {
+ self.cpu_addr.len()
+ }
+}
+
/// Note that the device configured to do DMA must be halted before this object is dropped.
-impl<T: AsBytes + FromBytes> Drop for CoherentAllocation<T> {
+impl<T: KnownSize + ?Sized> Drop for Coherent<T> {
fn drop(&mut self) {
- let size = self.count * core::mem::size_of::<T>();
+ let size = T::size(self.cpu_addr.as_ptr());
// SAFETY: Device pointer is guaranteed as valid by the type invariant on `Device`.
// The cpu address, and the dma handle are valid due to the type invariants on
- // `CoherentAllocation`.
+ // `Coherent`.
unsafe {
bindings::dma_free_attrs(
self.dev.as_raw(),
size,
- self.start_ptr_mut().cast(),
+ self.cpu_addr.as_ptr().cast(),
self.dma_handle,
self.dma_attrs.as_raw(),
)
@@ -660,20 +979,170 @@ impl<T: AsBytes + FromBytes> Drop for CoherentAllocation<T> {
}
}
-// SAFETY: It is safe to send a `CoherentAllocation` to another thread if `T`
+// SAFETY: It is safe to send a `Coherent` to another thread if `T`
// can be sent to another thread.
-unsafe impl<T: AsBytes + FromBytes + Send> Send for CoherentAllocation<T> {}
+unsafe impl<T: KnownSize + Send + ?Sized> Send for Coherent<T> {}
+
+// SAFETY: Sharing `&Coherent` across threads is safe if `T` is `Sync`, because all
+// methods that access the buffer contents (`field_read`, `field_write`, `as_slice`,
+// `as_slice_mut`) are `unsafe`, and callers are responsible for ensuring no data races occur.
+// The safe methods only return metadata or raw pointers whose use requires `unsafe`.
+unsafe impl<T: KnownSize + ?Sized + AsBytes + FromBytes + Sync> Sync for Coherent<T> {}
+
+impl<T: KnownSize + AsBytes + ?Sized> debugfs::BinaryWriter for Coherent<T> {
+ fn write_to_slice(
+ &self,
+ writer: &mut UserSliceWriter,
+ offset: &mut file::Offset,
+ ) -> Result<usize> {
+ if offset.is_negative() {
+ return Err(EINVAL);
+ }
+
+ // If the offset is too large for a usize (e.g. on 32-bit platforms),
+ // then consider that as past EOF and just return 0 bytes.
+ let Ok(offset_val) = usize::try_from(*offset) else {
+ return Ok(0);
+ };
+
+ let count = self.size().saturating_sub(offset_val).min(writer.len());
+
+ writer.write_dma(self, offset_val, count)?;
+
+ *offset += count as i64;
+ Ok(count)
+ }
+}
+
+/// An opaque DMA allocation without a kernel virtual mapping.
+///
+/// Unlike [`Coherent`], a `CoherentHandle` does not provide CPU access to the allocated memory.
+/// The allocation is always performed with `DMA_ATTR_NO_KERNEL_MAPPING`, meaning no kernel
+/// virtual mapping is created for the buffer. The value returned by the C API as the CPU
+/// address is an opaque handle used only to free the allocation.
+///
+/// This is useful for buffers that are only ever accessed by hardware.
+///
+/// # Invariants
+///
+/// - `cpu_handle` holds the opaque handle returned by `dma_alloc_attrs` with
+/// `DMA_ATTR_NO_KERNEL_MAPPING` set, and is only valid for passing back to `dma_free_attrs`.
+/// - `dma_handle` is the corresponding bus address for device DMA.
+/// - `size` is the allocation size in bytes as passed to `dma_alloc_attrs`.
+/// - `dma_attrs` contains the attributes used for the allocation, always including
+/// `DMA_ATTR_NO_KERNEL_MAPPING`.
+pub struct CoherentHandle {
+ dev: ARef<device::Device>,
+ dma_handle: DmaAddress,
+ cpu_handle: NonNull<c_void>,
+ size: usize,
+ dma_attrs: Attrs,
+}
+
+impl CoherentHandle {
+ /// Allocates `size` bytes of coherent DMA memory without creating a kernel virtual mapping.
+ ///
+ /// Additional DMA attributes may be passed via `dma_attrs`; `DMA_ATTR_NO_KERNEL_MAPPING` is
+ /// always set implicitly.
+ ///
+ /// Returns `EINVAL` if `size` is zero, `ENOMEM` if the allocation fails.
+ pub fn alloc_with_attrs(
+ dev: &device::Device<Bound>,
+ size: usize,
+ gfp_flags: kernel::alloc::Flags,
+ dma_attrs: Attrs,
+ ) -> Result<Self> {
+ if size == 0 {
+ return Err(EINVAL);
+ }
+
+ let dma_attrs = dma_attrs | Attrs(bindings::DMA_ATTR_NO_KERNEL_MAPPING);
+ let mut dma_handle = 0;
+ // SAFETY: `dev.as_raw()` is valid by the type invariant on `device::Device`.
+ let cpu_handle = unsafe {
+ bindings::dma_alloc_attrs(
+ dev.as_raw(),
+ size,
+ &mut dma_handle,
+ gfp_flags.as_raw(),
+ dma_attrs.as_raw(),
+ )
+ };
+
+ let cpu_handle = NonNull::new(cpu_handle).ok_or(ENOMEM)?;
+
+ // INVARIANT: `cpu_handle` is the opaque handle from a successful `dma_alloc_attrs` call
+ // with `DMA_ATTR_NO_KERNEL_MAPPING`, `dma_handle` is the corresponding DMA address,
+ // and we hold a refcounted reference to the device.
+ Ok(Self {
+ dev: dev.into(),
+ dma_handle,
+ cpu_handle,
+ size,
+ dma_attrs,
+ })
+ }
+
+ /// Allocates `size` bytes of coherent DMA memory without creating a kernel virtual mapping.
+ #[inline]
+ pub fn alloc(
+ dev: &device::Device<Bound>,
+ size: usize,
+ gfp_flags: kernel::alloc::Flags,
+ ) -> Result<Self> {
+ Self::alloc_with_attrs(dev, size, gfp_flags, Attrs(0))
+ }
+
+ /// Returns the DMA handle for this allocation.
+ ///
+ /// This address can be programmed into device hardware for DMA access.
+ #[inline]
+ pub fn dma_handle(&self) -> DmaAddress {
+ self.dma_handle
+ }
+
+ /// Returns the size in bytes of this allocation.
+ #[inline]
+ pub fn size(&self) -> usize {
+ self.size
+ }
+}
+
+impl Drop for CoherentHandle {
+ fn drop(&mut self) {
+ // SAFETY: All values are valid by the type invariants on `CoherentHandle`.
+ // `cpu_handle` is the opaque handle from `dma_alloc_attrs` and is passed back unchanged.
+ unsafe {
+ bindings::dma_free_attrs(
+ self.dev.as_raw(),
+ self.size,
+ self.cpu_handle.as_ptr(),
+ self.dma_handle,
+ self.dma_attrs.as_raw(),
+ )
+ }
+ }
+}
+
+// SAFETY: `CoherentHandle` only holds a device reference, a DMA handle, an opaque CPU handle,
+// and a size. None of these are tied to a specific thread.
+unsafe impl Send for CoherentHandle {}
+
+// SAFETY: `CoherentHandle` provides no CPU access to the underlying allocation. The only
+// operations on `&CoherentHandle` are reading the DMA handle and size, both of which are
+// plain `Copy` values.
+unsafe impl Sync for CoherentHandle {}
/// Reads a field of an item from an allocated region of structs.
///
/// The syntax is of the form `kernel::dma_read!(dma, proj)` where `dma` is an expression evaluating
-/// to a [`CoherentAllocation`] and `proj` is a [projection specification](kernel::ptr::project!).
+/// to a [`Coherent`] and `proj` is a [projection specification](kernel::ptr::project!).
///
/// # Examples
///
/// ```
/// use kernel::device::Device;
-/// use kernel::dma::{attrs::*, CoherentAllocation};
+/// use kernel::dma::{attrs::*, Coherent};
///
/// struct MyStruct { field: u32, }
///
@@ -682,7 +1151,7 @@ unsafe impl<T: AsBytes + FromBytes + Send> Send for CoherentAllocation<T> {}
/// // SAFETY: Instances of `MyStruct` have no uninitialized portions.
/// unsafe impl kernel::transmute::AsBytes for MyStruct{};
///
-/// # fn test(alloc: &kernel::dma::CoherentAllocation<MyStruct>) -> Result {
+/// # fn test(alloc: &kernel::dma::Coherent<[MyStruct]>) -> Result {
/// let whole = kernel::dma_read!(alloc, [2]?);
/// let field = kernel::dma_read!(alloc, [1]?.field);
/// # Ok::<(), Error>(()) }
@@ -692,17 +1161,17 @@ macro_rules! dma_read {
($dma:expr, $($proj:tt)*) => {{
let dma = &$dma;
let ptr = $crate::ptr::project!(
- $crate::dma::CoherentAllocation::as_ptr(dma), $($proj)*
+ $crate::dma::Coherent::as_ptr(dma), $($proj)*
);
// SAFETY: The pointer created by the projection is within the DMA region.
- unsafe { $crate::dma::CoherentAllocation::field_read(dma, ptr) }
+ unsafe { $crate::dma::Coherent::field_read(dma, ptr) }
}};
}
/// Writes to a field of an item from an allocated region of structs.
///
/// The syntax is of the form `kernel::dma_write!(dma, proj, val)` where `dma` is an expression
-/// evaluating to a [`CoherentAllocation`], `proj` is a
+/// evaluating to a [`Coherent`], `proj` is a
/// [projection specification](kernel::ptr::project!), and `val` is the value to be written to the
/// projected location.
///
@@ -710,7 +1179,7 @@ macro_rules! dma_read {
///
/// ```
/// use kernel::device::Device;
-/// use kernel::dma::{attrs::*, CoherentAllocation};
+/// use kernel::dma::{attrs::*, Coherent};
///
/// struct MyStruct { member: u32, }
///
@@ -719,7 +1188,7 @@ macro_rules! dma_read {
/// // SAFETY: Instances of `MyStruct` have no uninitialized portions.
/// unsafe impl kernel::transmute::AsBytes for MyStruct{};
///
-/// # fn test(alloc: &kernel::dma::CoherentAllocation<MyStruct>) -> Result {
+/// # fn test(alloc: &kernel::dma::Coherent<[MyStruct]>) -> Result {
/// kernel::dma_write!(alloc, [2]?.member, 0xf);
/// kernel::dma_write!(alloc, [1]?, MyStruct { member: 0xf });
/// # Ok::<(), Error>(()) }
@@ -729,11 +1198,11 @@ macro_rules! dma_write {
(@parse [$dma:expr] [$($proj:tt)*] [, $val:expr]) => {{
let dma = &$dma;
let ptr = $crate::ptr::project!(
- mut $crate::dma::CoherentAllocation::as_mut_ptr(dma), $($proj)*
+ mut $crate::dma::Coherent::as_mut_ptr(dma), $($proj)*
);
let val = $val;
// SAFETY: The pointer created by the projection is within the DMA region.
- unsafe { $crate::dma::CoherentAllocation::field_write(dma, ptr, val) }
+ unsafe { $crate::dma::Coherent::field_write(dma, ptr, val) }
}};
(@parse [$dma:expr] [$($proj:tt)*] [.$field:tt $($rest:tt)*]) => {
$crate::dma_write!(@parse [$dma] [$($proj)* .$field] [$($rest)*])
diff --git a/rust/kernel/drm/device.rs b/rust/kernel/drm/device.rs
index 3ce8f62a0056..adbafe8db54d 100644
--- a/rust/kernel/drm/device.rs
+++ b/rust/kernel/drm/device.rs
@@ -6,15 +6,34 @@
use crate::{
alloc::allocator::Kmalloc,
- bindings, device, drm,
- drm::driver::AllocImpl,
+ bindings, device,
+ drm::{
+ self,
+ driver::AllocImpl, //
+ },
error::from_err_ptr,
- error::Result,
prelude::*,
- sync::aref::{ARef, AlwaysRefCounted},
+ sync::aref::{
+ ARef,
+ AlwaysRefCounted, //
+ },
types::Opaque,
+ workqueue::{
+ HasDelayedWork,
+ HasWork,
+ Work,
+ WorkItem, //
+ },
+};
+use core::{
+ alloc::Layout,
+ mem,
+ ops::Deref,
+ ptr::{
+ self,
+ NonNull, //
+ },
};
-use core::{alloc::Layout, mem, ops::Deref, ptr, ptr::NonNull};
#[cfg(CONFIG_DRM_LEGACY)]
macro_rules! drm_legacy_fields {
@@ -227,3 +246,61 @@ unsafe impl<T: drm::Driver> Send for Device<T> {}
// SAFETY: A `drm::Device` can be shared among threads because all immutable methods are protected
// by the synchronization in `struct drm_device`.
unsafe impl<T: drm::Driver> Sync for Device<T> {}
+
+impl<T, const ID: u64> WorkItem<ID> for Device<T>
+where
+ T: drm::Driver,
+ T::Data: WorkItem<ID, Pointer = ARef<Device<T>>>,
+ T::Data: HasWork<Device<T>, ID>,
+{
+ type Pointer = ARef<Device<T>>;
+
+ fn run(ptr: ARef<Device<T>>) {
+ T::Data::run(ptr);
+ }
+}
+
+// SAFETY:
+//
+// - `raw_get_work` and `work_container_of` return valid pointers by relying on
+// `T::Data::raw_get_work` and `container_of`. In particular, `T::Data` is
+// stored inline in `drm::Device`, so the `container_of` call is valid.
+//
+// - The two methods are true inverses of each other: given `ptr: *mut
+// Device<T>`, `raw_get_work` will return a `*mut Work<Device<T>, ID>` through
+// `T::Data::raw_get_work` and given a `ptr: *mut Work<Device<T>, ID>`,
+// `work_container_of` will return a `*mut Device<T>` through `container_of`.
+unsafe impl<T, const ID: u64> HasWork<Device<T>, ID> for Device<T>
+where
+ T: drm::Driver,
+ T::Data: HasWork<Device<T>, ID>,
+{
+ unsafe fn raw_get_work(ptr: *mut Self) -> *mut Work<Device<T>, ID> {
+ // SAFETY: The caller promises that `ptr` points to a valid `Device<T>`.
+ let data_ptr = unsafe { &raw mut (*ptr).data };
+
+ // SAFETY: `data_ptr` is a valid pointer to `T::Data`.
+ unsafe { T::Data::raw_get_work(data_ptr) }
+ }
+
+ unsafe fn work_container_of(ptr: *mut Work<Device<T>, ID>) -> *mut Self {
+ // SAFETY: The caller promises that `ptr` points at a `Work` field in
+ // `T::Data`.
+ let data_ptr = unsafe { T::Data::work_container_of(ptr) };
+
+ // SAFETY: `T::Data` is stored as the `data` field in `Device<T>`.
+ unsafe { crate::container_of!(data_ptr, Self, data) }
+ }
+}
+
+// SAFETY: Our `HasWork<T, ID>` implementation returns a `work_struct` that is
+// stored in the `work` field of a `delayed_work` with the same access rules as
+// the `work_struct` owing to the bound on `T::Data: HasDelayedWork<Device<T>,
+// ID>`, which requires that `T::Data::raw_get_work` return a `work_struct` that
+// is inside a `delayed_work`.
+unsafe impl<T, const ID: u64> HasDelayedWork<Device<T>, ID> for Device<T>
+where
+ T: drm::Driver,
+ T::Data: HasDelayedWork<Device<T>, ID>,
+{
+}
diff --git a/rust/kernel/drm/driver.rs b/rust/kernel/drm/driver.rs
index e09f977b5b51..5233bdebc9fc 100644
--- a/rust/kernel/drm/driver.rs
+++ b/rust/kernel/drm/driver.rs
@@ -5,12 +5,14 @@
//! C header: [`include/drm/drm_drv.h`](srctree/include/drm/drm_drv.h)
use crate::{
- bindings, device, devres, drm,
- error::{to_result, Result},
+ bindings,
+ device,
+ devres,
+ drm,
+ error::to_result,
prelude::*,
- sync::aref::ARef,
+ sync::aref::ARef, //
};
-use macros::vtable;
/// Driver use the GEM memory manager. This should be set for all modern drivers.
pub(crate) const FEAT_GEM: u32 = bindings::drm_driver_feature_DRIVER_GEM;
diff --git a/rust/kernel/drm/file.rs b/rust/kernel/drm/file.rs
index 8c46f8d51951..10160601ce5a 100644
--- a/rust/kernel/drm/file.rs
+++ b/rust/kernel/drm/file.rs
@@ -4,9 +4,13 @@
//!
//! C header: [`include/drm/drm_file.h`](srctree/include/drm/drm_file.h)
-use crate::{bindings, drm, error::Result, prelude::*, types::Opaque};
+use crate::{
+ bindings,
+ drm,
+ prelude::*,
+ types::Opaque, //
+};
use core::marker::PhantomData;
-use core::pin::Pin;
/// Trait that must be implemented by DRM drivers to represent a DRM File (a client instance).
pub trait DriverFile {
diff --git a/rust/kernel/drm/gem/mod.rs b/rust/kernel/drm/gem/mod.rs
index d49a9ba02635..75acda7ba500 100644
--- a/rust/kernel/drm/gem/mod.rs
+++ b/rust/kernel/drm/gem/mod.rs
@@ -5,15 +5,66 @@
//! C header: [`include/drm/drm_gem.h`](srctree/include/drm/drm_gem.h)
use crate::{
- alloc::flags::*,
- bindings, drm,
- drm::driver::{AllocImpl, AllocOps},
- error::{to_result, Result},
+ bindings,
+ drm::{
+ self,
+ driver::{
+ AllocImpl,
+ AllocOps, //
+ },
+ },
+ error::to_result,
prelude::*,
- sync::aref::{ARef, AlwaysRefCounted},
+ sync::aref::{
+ ARef,
+ AlwaysRefCounted, //
+ },
types::Opaque,
};
-use core::{ops::Deref, ptr::NonNull};
+use core::{
+ ops::Deref,
+ ptr::NonNull, //
+};
+
+#[cfg(CONFIG_RUST_DRM_GEM_SHMEM_HELPER)]
+pub mod shmem;
+
+/// A macro for implementing [`AlwaysRefCounted`] for any GEM object type.
+///
+/// Since all GEM objects use the same refcounting scheme.
+#[macro_export]
+macro_rules! impl_aref_for_gem_obj {
+ (
+ impl $( <$( $tparam_id:ident ),+> )? for $type:ty
+ $(
+ where
+ $( $bind_param:path : $bind_trait:path ),+
+ )?
+ ) => {
+ // SAFETY: All GEM objects are refcounted.
+ unsafe impl $( <$( $tparam_id ),+> )? $crate::sync::aref::AlwaysRefCounted for $type
+ where
+ Self: IntoGEMObject,
+ $( $( $bind_param : $bind_trait ),+ )?
+ {
+ fn inc_ref(&self) {
+ // SAFETY: The existence of a shared reference guarantees that the refcount is
+ // non-zero.
+ unsafe { bindings::drm_gem_object_get(self.as_raw()) };
+ }
+
+ unsafe fn dec_ref(obj: core::ptr::NonNull<Self>) {
+ // SAFETY: `obj` is a valid pointer to an `Object<T>`.
+ let obj = unsafe { obj.as_ref() }.as_raw();
+
+ // SAFETY: The safety requirements guarantee that the refcount is non-zero.
+ unsafe { bindings::drm_gem_object_put(obj) };
+ }
+ }
+ };
+}
+#[cfg_attr(not(CONFIG_RUST_DRM_GEM_SHMEM_HELPER), allow(unused))]
+pub(crate) use impl_aref_for_gem_obj;
/// A type alias for retrieving a [`Driver`]s [`DriverFile`] implementation from its
/// [`DriverObject`] implementation.
@@ -27,8 +78,15 @@ pub trait DriverObject: Sync + Send + Sized {
/// Parent `Driver` for this object.
type Driver: drm::Driver;
+ /// The data type to use for passing arguments to [`DriverObject::new`].
+ type Args;
+
/// Create a new driver data object for a GEM object of a given size.
- fn new(dev: &drm::Device<Self::Driver>, size: usize) -> impl PinInit<Self, Error>;
+ fn new(
+ dev: &drm::Device<Self::Driver>,
+ size: usize,
+ args: Self::Args,
+ ) -> impl PinInit<Self, Error>;
/// Open a new handle to an existing object, associated with a File.
fn open(_obj: &<Self::Driver as drm::Driver>::Object, _file: &DriverFile<Self>) -> Result {
@@ -162,6 +220,18 @@ pub trait BaseObject: IntoGEMObject {
impl<T: IntoGEMObject> BaseObject for T {}
+/// Crate-private base operations shared by all GEM object classes.
+#[cfg_attr(not(CONFIG_RUST_DRM_GEM_SHMEM_HELPER), expect(unused))]
+pub(crate) trait BaseObjectPrivate: IntoGEMObject {
+ /// Return a pointer to this object's dma_resv.
+ fn raw_dma_resv(&self) -> *mut bindings::dma_resv {
+ // SAFETY: `self.as_raw()` always returns a valid pointer to the base DRM GEM object.
+ unsafe { (*self.as_raw()).resv }
+ }
+}
+
+impl<T: IntoGEMObject> BaseObjectPrivate for T {}
+
/// A base GEM object.
///
/// # Invariants
@@ -195,11 +265,11 @@ impl<T: DriverObject> Object<T> {
};
/// Create a new GEM object.
- pub fn new(dev: &drm::Device<T::Driver>, size: usize) -> Result<ARef<Self>> {
+ pub fn new(dev: &drm::Device<T::Driver>, size: usize, args: T::Args) -> Result<ARef<Self>> {
let obj: Pin<KBox<Self>> = KBox::pin_init(
try_pin_init!(Self {
obj: Opaque::new(bindings::drm_gem_object::default()),
- data <- T::new(dev, size),
+ data <- T::new(dev, size, args),
}),
GFP_KERNEL,
)?;
@@ -252,21 +322,7 @@ impl<T: DriverObject> Object<T> {
}
}
-// SAFETY: Instances of `Object<T>` are always reference-counted.
-unsafe impl<T: DriverObject> crate::sync::aref::AlwaysRefCounted for Object<T> {
- fn inc_ref(&self) {
- // SAFETY: The existence of a shared reference guarantees that the refcount is non-zero.
- unsafe { bindings::drm_gem_object_get(self.as_raw()) };
- }
-
- unsafe fn dec_ref(obj: NonNull<Self>) {
- // SAFETY: `obj` is a valid pointer to an `Object<T>`.
- let obj = unsafe { obj.as_ref() };
-
- // SAFETY: The safety requirements guarantee that the refcount is non-zero.
- unsafe { bindings::drm_gem_object_put(obj.as_raw()) }
- }
-}
+impl_aref_for_gem_obj!(impl<T> for Object<T> where T: DriverObject);
impl<T: DriverObject> super::private::Sealed for Object<T> {}
diff --git a/rust/kernel/drm/gem/shmem.rs b/rust/kernel/drm/gem/shmem.rs
new file mode 100644
index 000000000000..d025fb035195
--- /dev/null
+++ b/rust/kernel/drm/gem/shmem.rs
@@ -0,0 +1,228 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! DRM GEM shmem helper objects
+//!
+//! C header: [`include/linux/drm/drm_gem_shmem_helper.h`](srctree/include/drm/drm_gem_shmem_helper.h)
+
+// TODO:
+// - There are a number of spots here that manually acquire/release the DMA reservation lock using
+// dma_resv_(un)lock(). In the future we should add support for ww mutex, expose a method to
+// acquire a reference to the WwMutex, and then use that directly instead of the C functions here.
+
+use crate::{
+ container_of,
+ drm::{
+ device,
+ driver,
+ gem,
+ private::Sealed, //
+ },
+ error::to_result,
+ prelude::*,
+ types::{
+ ARef,
+ Opaque, //
+ }, //
+};
+use core::{
+ ops::{
+ Deref,
+ DerefMut, //
+ },
+ ptr::NonNull,
+};
+use gem::{
+ BaseObjectPrivate,
+ DriverObject,
+ IntoGEMObject, //
+};
+
+/// A struct for controlling the creation of shmem-backed GEM objects.
+///
+/// This is used with [`Object::new()`] to control various properties that can only be set when
+/// initially creating a shmem-backed GEM object.
+#[derive(Default)]
+pub struct ObjectConfig<'a, T: DriverObject> {
+ /// Whether to set the write-combine map flag.
+ pub map_wc: bool,
+
+ /// Reuse the DMA reservation from another GEM object.
+ ///
+ /// The newly created [`Object`] will hold an owned refcount to `parent_resv_obj` if specified.
+ pub parent_resv_obj: Option<&'a Object<T>>,
+}
+
+/// A shmem-backed GEM object.
+///
+/// # Invariants
+///
+/// `obj` contains a valid initialized `struct drm_gem_shmem_object` for the lifetime of this
+/// object.
+#[repr(C)]
+#[pin_data]
+pub struct Object<T: DriverObject> {
+ #[pin]
+ obj: Opaque<bindings::drm_gem_shmem_object>,
+ /// Parent object that owns this object's DMA reservation object.
+ parent_resv_obj: Option<ARef<Object<T>>>,
+ #[pin]
+ inner: T,
+}
+
+super::impl_aref_for_gem_obj!(impl<T> for Object<T> where T: DriverObject);
+
+// SAFETY: All GEM objects are thread-safe.
+unsafe impl<T: DriverObject> Send for Object<T> {}
+
+// SAFETY: All GEM objects are thread-safe.
+unsafe impl<T: DriverObject> Sync for Object<T> {}
+
+impl<T: DriverObject> Object<T> {
+ /// `drm_gem_object_funcs` vtable suitable for GEM shmem objects.
+ const VTABLE: bindings::drm_gem_object_funcs = bindings::drm_gem_object_funcs {
+ free: Some(Self::free_callback),
+ open: Some(super::open_callback::<T>),
+ close: Some(super::close_callback::<T>),
+ print_info: Some(bindings::drm_gem_shmem_object_print_info),
+ export: None,
+ pin: Some(bindings::drm_gem_shmem_object_pin),
+ unpin: Some(bindings::drm_gem_shmem_object_unpin),
+ get_sg_table: Some(bindings::drm_gem_shmem_object_get_sg_table),
+ vmap: Some(bindings::drm_gem_shmem_object_vmap),
+ vunmap: Some(bindings::drm_gem_shmem_object_vunmap),
+ mmap: Some(bindings::drm_gem_shmem_object_mmap),
+ status: None,
+ rss: None,
+ #[allow(unused_unsafe, reason = "Safe since Rust 1.82.0")]
+ // SAFETY: `drm_gem_shmem_vm_ops` is a valid, static const on the C side.
+ vm_ops: unsafe { &raw const bindings::drm_gem_shmem_vm_ops },
+ evict: None,
+ };
+
+ /// Return a raw pointer to the embedded drm_gem_shmem_object.
+ fn as_raw_shmem(&self) -> *mut bindings::drm_gem_shmem_object {
+ self.obj.get()
+ }
+
+ /// Create a new shmem-backed DRM object of the given size.
+ ///
+ /// Additional config options can be specified using `config`.
+ pub fn new(
+ dev: &device::Device<T::Driver>,
+ size: usize,
+ config: ObjectConfig<'_, T>,
+ args: T::Args,
+ ) -> Result<ARef<Self>> {
+ let new: Pin<KBox<Self>> = KBox::try_pin_init(
+ try_pin_init!(Self {
+ obj <- Opaque::init_zeroed(),
+ parent_resv_obj: config.parent_resv_obj.map(|p| p.into()),
+ inner <- T::new(dev, size, args),
+ }),
+ GFP_KERNEL,
+ )?;
+
+ // SAFETY: `obj.as_raw()` is guaranteed to be valid by the initialization above.
+ unsafe { (*new.as_raw()).funcs = &Self::VTABLE };
+
+ // SAFETY: The arguments are all valid via the type invariants.
+ to_result(unsafe { bindings::drm_gem_shmem_init(dev.as_raw(), new.as_raw_shmem(), size) })?;
+
+ // SAFETY: We never move out of `self`.
+ let new = KBox::into_raw(unsafe { Pin::into_inner_unchecked(new) });
+
+ // SAFETY: We're taking over the owned refcount from `drm_gem_shmem_init`.
+ let obj = unsafe { ARef::from_raw(NonNull::new_unchecked(new)) };
+
+ // Start filling out values from `config`
+ if let Some(parent_resv) = config.parent_resv_obj {
+ // SAFETY: We have yet to expose the new gem object outside of this function, so it is
+ // safe to modify this field.
+ unsafe { (*obj.obj.get()).base.resv = parent_resv.raw_dma_resv() };
+ }
+
+ // SAFETY: We have yet to expose this object outside of this function, so we're guaranteed
+ // to have exclusive access - thus making this safe to hold a mutable reference to.
+ let shmem = unsafe { &mut *obj.as_raw_shmem() };
+ shmem.set_map_wc(config.map_wc);
+
+ Ok(obj)
+ }
+
+ /// Returns the `Device` that owns this GEM object.
+ pub fn dev(&self) -> &device::Device<T::Driver> {
+ // SAFETY: `dev` will have been initialized in `Self::new()` by `drm_gem_shmem_init()`.
+ unsafe { device::Device::from_raw((*self.as_raw()).dev) }
+ }
+
+ extern "C" fn free_callback(obj: *mut bindings::drm_gem_object) {
+ // SAFETY:
+ // - DRM always passes a valid gem object here
+ // - We used drm_gem_shmem_create() in our create_gem_object callback, so we know that
+ // `obj` is contained within a drm_gem_shmem_object
+ let this = unsafe { container_of!(obj, bindings::drm_gem_shmem_object, base) };
+
+ // SAFETY:
+ // - We're in free_callback - so this function is safe to call.
+ // - We won't be using the gem resources on `this` after this call.
+ unsafe { bindings::drm_gem_shmem_release(this) };
+
+ // SAFETY:
+ // - We verified above that `obj` is valid, which makes `this` valid
+ // - This function is set in AllocOps, so we know that `this` is contained within a
+ // `Object<T>`
+ let this = unsafe { container_of!(Opaque::cast_from(this), Self, obj) }.cast_mut();
+
+ // SAFETY: We're recovering the Kbox<> we created in gem_create_object()
+ let _ = unsafe { KBox::from_raw(this) };
+ }
+}
+
+impl<T: DriverObject> Deref for Object<T> {
+ type Target = T;
+
+ fn deref(&self) -> &Self::Target {
+ &self.inner
+ }
+}
+
+impl<T: DriverObject> DerefMut for Object<T> {
+ fn deref_mut(&mut self) -> &mut Self::Target {
+ &mut self.inner
+ }
+}
+
+impl<T: DriverObject> Sealed for Object<T> {}
+
+impl<T: DriverObject> gem::IntoGEMObject for Object<T> {
+ fn as_raw(&self) -> *mut bindings::drm_gem_object {
+ // SAFETY:
+ // - Our immutable reference is proof that this is safe to dereference.
+ // - `obj` is always a valid drm_gem_shmem_object via our type invariants.
+ unsafe { &raw mut (*self.obj.get()).base }
+ }
+
+ unsafe fn from_raw<'a>(obj: *mut bindings::drm_gem_object) -> &'a Object<T> {
+ // SAFETY: The safety contract of from_gem_obj() guarantees that `obj` is contained within
+ // `Self`
+ unsafe {
+ let obj = Opaque::cast_from(container_of!(obj, bindings::drm_gem_shmem_object, base));
+
+ &*container_of!(obj, Object<T>, obj)
+ }
+ }
+}
+
+impl<T: DriverObject> driver::AllocImpl for Object<T> {
+ type Driver = T::Driver;
+
+ const ALLOC_OPS: driver::AllocOps = driver::AllocOps {
+ gem_create_object: None,
+ prime_handle_to_fd: None,
+ prime_fd_to_handle: None,
+ gem_prime_import: None,
+ gem_prime_import_sg_table: Some(bindings::drm_gem_shmem_prime_import_sg_table),
+ dumb_create: Some(bindings::drm_gem_shmem_dumb_create),
+ dumb_map_offset: None,
+ };
+}
diff --git a/rust/kernel/error.rs b/rust/kernel/error.rs
index decceb6ae855..05cf869ac090 100644
--- a/rust/kernel/error.rs
+++ b/rust/kernel/error.rs
@@ -67,6 +67,7 @@ pub mod code {
declare_err!(EDOM, "Math argument out of domain of func.");
declare_err!(ERANGE, "Math result not representable.");
declare_err!(EOVERFLOW, "Value too large for defined data type.");
+ declare_err!(EMSGSIZE, "Message too long.");
declare_err!(ETIMEDOUT, "Connection timed out.");
declare_err!(ERESTARTSYS, "Restart the system call.");
declare_err!(ERESTARTNOINTR, "System call was interrupted by a signal and will be restarted.");
diff --git a/rust/kernel/gpu.rs b/rust/kernel/gpu.rs
new file mode 100644
index 000000000000..1dc5d0c8c09d
--- /dev/null
+++ b/rust/kernel/gpu.rs
@@ -0,0 +1,6 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! GPU subsystem abstractions.
+
+#[cfg(CONFIG_GPU_BUDDY = "y")]
+pub mod buddy;
diff --git a/rust/kernel/gpu/buddy.rs b/rust/kernel/gpu/buddy.rs
new file mode 100644
index 000000000000..d502ada6ebbd
--- /dev/null
+++ b/rust/kernel/gpu/buddy.rs
@@ -0,0 +1,614 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! GPU buddy allocator bindings.
+//!
+//! C header: [`include/linux/gpu_buddy.h`](srctree/include/linux/gpu_buddy.h)
+//!
+//! This module provides Rust abstractions over the Linux kernel's GPU buddy
+//! allocator, which implements a binary buddy memory allocator.
+//!
+//! The buddy allocator manages a contiguous address space and allocates blocks
+//! in power-of-two sizes, useful for GPU physical memory management.
+//!
+//! # Examples
+//!
+//! Create a buddy allocator and perform a basic range allocation:
+//!
+//! ```
+//! use kernel::{
+//! gpu::buddy::{
+//! GpuBuddy,
+//! GpuBuddyAllocFlags,
+//! GpuBuddyAllocMode,
+//! GpuBuddyParams, //
+//! },
+//! prelude::*,
+//! ptr::Alignment,
+//! sizes::*, //
+//! };
+//!
+//! // Create a 1GB buddy allocator with 4KB minimum chunk size.
+//! let buddy = GpuBuddy::new(GpuBuddyParams {
+//! base_offset: 0,
+//! size: SZ_1G as u64,
+//! chunk_size: Alignment::new::<SZ_4K>(),
+//! })?;
+//!
+//! assert_eq!(buddy.size(), SZ_1G as u64);
+//! assert_eq!(buddy.chunk_size(), Alignment::new::<SZ_4K>());
+//! let initial_free = buddy.avail();
+//!
+//! // Allocate 16MB. Block lands at the top of the address range.
+//! let allocated = KBox::pin_init(
+//! buddy.alloc_blocks(
+//! GpuBuddyAllocMode::Simple,
+//! SZ_16M as u64,
+//! Alignment::new::<SZ_16M>(),
+//! GpuBuddyAllocFlags::default(),
+//! ),
+//! GFP_KERNEL,
+//! )?;
+//! assert_eq!(buddy.avail(), initial_free - SZ_16M as u64);
+//!
+//! let block = allocated.iter().next().expect("expected one block");
+//! assert_eq!(block.offset(), (SZ_1G - SZ_16M) as u64);
+//! assert_eq!(block.order(), 12); // 2^12 pages = 16MB
+//! assert_eq!(block.size(), SZ_16M as u64);
+//! assert_eq!(allocated.iter().count(), 1);
+//!
+//! // Dropping the allocation returns the range to the buddy allocator.
+//! drop(allocated);
+//! assert_eq!(buddy.avail(), initial_free);
+//! # Ok::<(), Error>(())
+//! ```
+//!
+//! Top-down allocation allocates from the highest addresses:
+//!
+//! ```
+//! # use kernel::{
+//! # gpu::buddy::{GpuBuddy, GpuBuddyAllocMode, GpuBuddyAllocFlags, GpuBuddyParams},
+//! # prelude::*,
+//! # ptr::Alignment,
+//! # sizes::*, //
+//! # };
+//! # let buddy = GpuBuddy::new(GpuBuddyParams {
+//! # base_offset: 0,
+//! # size: SZ_1G as u64,
+//! # chunk_size: Alignment::new::<SZ_4K>(),
+//! # })?;
+//! # let initial_free = buddy.avail();
+//! let topdown = KBox::pin_init(
+//! buddy.alloc_blocks(
+//! GpuBuddyAllocMode::TopDown,
+//! SZ_16M as u64,
+//! Alignment::new::<SZ_16M>(),
+//! GpuBuddyAllocFlags::default(),
+//! ),
+//! GFP_KERNEL,
+//! )?;
+//! assert_eq!(buddy.avail(), initial_free - SZ_16M as u64);
+//!
+//! let block = topdown.iter().next().expect("expected one block");
+//! assert_eq!(block.offset(), (SZ_1G - SZ_16M) as u64);
+//! assert_eq!(block.order(), 12);
+//! assert_eq!(block.size(), SZ_16M as u64);
+//!
+//! // Dropping the allocation returns the range to the buddy allocator.
+//! drop(topdown);
+//! assert_eq!(buddy.avail(), initial_free);
+//! # Ok::<(), Error>(())
+//! ```
+//!
+//! Non-contiguous allocation can fill fragmented memory by returning multiple
+//! blocks:
+//!
+//! ```
+//! # use kernel::{
+//! # gpu::buddy::{
+//! # GpuBuddy, GpuBuddyAllocFlags, GpuBuddyAllocMode, GpuBuddyParams,
+//! # },
+//! # prelude::*,
+//! # ptr::Alignment,
+//! # sizes::*, //
+//! # };
+//! # let buddy = GpuBuddy::new(GpuBuddyParams {
+//! # base_offset: 0,
+//! # size: SZ_1G as u64,
+//! # chunk_size: Alignment::new::<SZ_4K>(),
+//! # })?;
+//! # let initial_free = buddy.avail();
+//! // Create fragmentation by allocating 4MB blocks at [0,4M) and [8M,12M).
+//! let frag1 = KBox::pin_init(
+//! buddy.alloc_blocks(
+//! GpuBuddyAllocMode::Range(0..SZ_4M as u64),
+//! SZ_4M as u64,
+//! Alignment::new::<SZ_4M>(),
+//! GpuBuddyAllocFlags::default(),
+//! ),
+//! GFP_KERNEL,
+//! )?;
+//! assert_eq!(buddy.avail(), initial_free - SZ_4M as u64);
+//!
+//! let frag2 = KBox::pin_init(
+//! buddy.alloc_blocks(
+//! GpuBuddyAllocMode::Range(SZ_8M as u64..(SZ_8M + SZ_4M) as u64),
+//! SZ_4M as u64,
+//! Alignment::new::<SZ_4M>(),
+//! GpuBuddyAllocFlags::default(),
+//! ),
+//! GFP_KERNEL,
+//! )?;
+//! assert_eq!(buddy.avail(), initial_free - SZ_8M as u64);
+//!
+//! // Allocate 8MB, this returns 2 blocks from the holes.
+//! let fragmented = KBox::pin_init(
+//! buddy.alloc_blocks(
+//! GpuBuddyAllocMode::Range(0..SZ_16M as u64),
+//! SZ_8M as u64,
+//! Alignment::new::<SZ_4M>(),
+//! GpuBuddyAllocFlags::default(),
+//! ),
+//! GFP_KERNEL,
+//! )?;
+//! assert_eq!(buddy.avail(), initial_free - SZ_16M as u64);
+//!
+//! let (mut count, mut total) = (0u32, 0u64);
+//! for block in fragmented.iter() {
+//! assert_eq!(block.size(), SZ_4M as u64);
+//! total += block.size();
+//! count += 1;
+//! }
+//! assert_eq!(total, SZ_8M as u64);
+//! assert_eq!(count, 2);
+//! # Ok::<(), Error>(())
+//! ```
+//!
+//! Contiguous allocation fails when only fragmented space is available:
+//!
+//! ```
+//! # use kernel::{
+//! # gpu::buddy::{
+//! # GpuBuddy, GpuBuddyAllocFlag, GpuBuddyAllocFlags, GpuBuddyAllocMode, GpuBuddyParams,
+//! # },
+//! # prelude::*,
+//! # ptr::Alignment,
+//! # sizes::*, //
+//! # };
+//! // Create a small 16MB buddy allocator with fragmented memory.
+//! let small = GpuBuddy::new(GpuBuddyParams {
+//! base_offset: 0,
+//! size: SZ_16M as u64,
+//! chunk_size: Alignment::new::<SZ_4K>(),
+//! })?;
+//!
+//! let _hole1 = KBox::pin_init(
+//! small.alloc_blocks(
+//! GpuBuddyAllocMode::Range(0..SZ_4M as u64),
+//! SZ_4M as u64,
+//! Alignment::new::<SZ_4M>(),
+//! GpuBuddyAllocFlags::default(),
+//! ),
+//! GFP_KERNEL,
+//! )?;
+//!
+//! let _hole2 = KBox::pin_init(
+//! small.alloc_blocks(
+//! GpuBuddyAllocMode::Range(SZ_8M as u64..(SZ_8M + SZ_4M) as u64),
+//! SZ_4M as u64,
+//! Alignment::new::<SZ_4M>(),
+//! GpuBuddyAllocFlags::default(),
+//! ),
+//! GFP_KERNEL,
+//! )?;
+//!
+//! // 8MB contiguous should fail, only two non-contiguous 4MB holes exist.
+//! let result = KBox::pin_init(
+//! small.alloc_blocks(
+//! GpuBuddyAllocMode::Simple,
+//! SZ_8M as u64,
+//! Alignment::new::<SZ_4M>(),
+//! GpuBuddyAllocFlag::Contiguous,
+//! ),
+//! GFP_KERNEL,
+//! );
+//! assert!(result.is_err());
+//! # Ok::<(), Error>(())
+//! ```
+
+use core::ops::Range;
+
+use crate::{
+ bindings,
+ clist_create,
+ error::to_result,
+ interop::list::CListHead,
+ new_mutex,
+ prelude::*,
+ ptr::Alignment,
+ sync::{
+ lock::mutex::MutexGuard,
+ Arc,
+ Mutex, //
+ },
+ types::Opaque, //
+};
+
+/// Allocation mode for the GPU buddy allocator.
+///
+/// The mode determines the primary allocation strategy. Modes are mutually
+/// exclusive: an allocation is either simple, range-constrained, or top-down.
+///
+/// Orthogonal modifier flags (e.g., contiguous, clear) are specified separately
+/// via [`GpuBuddyAllocFlags`].
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum GpuBuddyAllocMode {
+ /// Simple allocation without constraints.
+ Simple,
+ /// Range-based allocation within the given address range.
+ Range(Range<u64>),
+ /// Allocate from top of address space downward.
+ TopDown,
+}
+
+impl GpuBuddyAllocMode {
+ /// Returns the C flags corresponding to the allocation mode.
+ fn as_flags(&self) -> usize {
+ match self {
+ Self::Simple => 0,
+ Self::Range(_) => bindings::GPU_BUDDY_RANGE_ALLOCATION,
+ Self::TopDown => bindings::GPU_BUDDY_TOPDOWN_ALLOCATION,
+ }
+ }
+
+ /// Extracts the range start/end, defaulting to `(0, 0)` for non-range modes.
+ fn range(&self) -> (u64, u64) {
+ match self {
+ Self::Range(range) => (range.start, range.end),
+ _ => (0, 0),
+ }
+ }
+}
+
+crate::impl_flags!(
+ /// Modifier flags for GPU buddy allocation.
+ ///
+ /// These flags can be combined with any [`GpuBuddyAllocMode`] to control
+ /// additional allocation behavior.
+ #[derive(Clone, Copy, Default, PartialEq, Eq)]
+ pub struct GpuBuddyAllocFlags(usize);
+
+ /// Individual modifier flag for GPU buddy allocation.
+ #[derive(Clone, Copy, PartialEq, Eq)]
+ pub enum GpuBuddyAllocFlag {
+ /// Allocate physically contiguous blocks.
+ Contiguous = bindings::GPU_BUDDY_CONTIGUOUS_ALLOCATION,
+
+ /// Request allocation from cleared (zeroed) memory.
+ Clear = bindings::GPU_BUDDY_CLEAR_ALLOCATION,
+
+ /// Disable trimming of partially used blocks.
+ TrimDisable = bindings::GPU_BUDDY_TRIM_DISABLE,
+ }
+);
+
+/// Parameters for creating a GPU buddy allocator.
+pub struct GpuBuddyParams {
+ /// Base offset (in bytes) where the managed memory region starts.
+ /// Allocations will be offset by this value.
+ pub base_offset: u64,
+ /// Total size (in bytes) of the address space managed by the allocator.
+ pub size: u64,
+ /// Minimum allocation unit / chunk size; must be >= 4KB.
+ pub chunk_size: Alignment,
+}
+
+/// Inner structure holding the actual buddy allocator.
+///
+/// # Synchronization
+///
+/// The C `gpu_buddy` API requires synchronization (see `include/linux/gpu_buddy.h`).
+/// Internal locking ensures all allocator and free operations are properly
+/// synchronized, preventing races between concurrent allocations and the
+/// freeing that occurs when [`AllocatedBlocks`] is dropped.
+///
+/// # Invariants
+///
+/// The inner [`Opaque`] contains an initialized buddy allocator.
+#[pin_data(PinnedDrop)]
+struct GpuBuddyInner {
+ #[pin]
+ inner: Opaque<bindings::gpu_buddy>,
+
+ // TODO: Replace `Mutex<()>` with `Mutex<Opaque<..>>` once `Mutex::new()`
+ // accepts `impl PinInit<T>`.
+ #[pin]
+ lock: Mutex<()>,
+ /// Cached creation parameters (do not change after init).
+ params: GpuBuddyParams,
+}
+
+impl GpuBuddyInner {
+ /// Create a pin-initializer for the buddy allocator.
+ fn new(params: GpuBuddyParams) -> impl PinInit<Self, Error> {
+ let size = params.size;
+ let chunk_size = params.chunk_size;
+
+ // INVARIANT: `gpu_buddy_init` returns 0 on success, at which point the
+ // `gpu_buddy` structure is initialized and ready for use with all
+ // `gpu_buddy_*` APIs. `try_pin_init!` only completes if all fields succeed,
+ // so the invariant holds when construction finishes.
+ try_pin_init!(Self {
+ inner <- Opaque::try_ffi_init(|ptr| {
+ // SAFETY: `ptr` points to valid uninitialized memory from the pin-init
+ // infrastructure. `gpu_buddy_init` will initialize the structure.
+ to_result(unsafe {
+ bindings::gpu_buddy_init(ptr, size, chunk_size.as_usize() as u64)
+ })
+ }),
+ lock <- new_mutex!(()),
+ params,
+ })
+ }
+
+ /// Lock the mutex and return a guard for accessing the allocator.
+ fn lock(&self) -> GpuBuddyGuard<'_> {
+ GpuBuddyGuard {
+ inner: self,
+ _guard: self.lock.lock(),
+ }
+ }
+}
+
+#[pinned_drop]
+impl PinnedDrop for GpuBuddyInner {
+ fn drop(self: Pin<&mut Self>) {
+ let guard = self.lock();
+
+ // SAFETY: Per the type invariant, `inner` contains an initialized
+ // allocator. `guard` provides exclusive access.
+ unsafe { bindings::gpu_buddy_fini(guard.as_raw()) };
+ }
+}
+
+// SAFETY: `GpuBuddyInner` can be sent between threads.
+unsafe impl Send for GpuBuddyInner {}
+
+// SAFETY: `GpuBuddyInner` is `Sync` because `GpuBuddyInner::lock`
+// serializes all access to the C allocator, preventing data races.
+unsafe impl Sync for GpuBuddyInner {}
+
+/// Guard that proves the lock is held, enabling access to the allocator.
+///
+/// The `_guard` holds the lock for the duration of this guard's lifetime.
+struct GpuBuddyGuard<'a> {
+ inner: &'a GpuBuddyInner,
+ _guard: MutexGuard<'a, ()>,
+}
+
+impl GpuBuddyGuard<'_> {
+ /// Get a raw pointer to the underlying C `gpu_buddy` structure.
+ fn as_raw(&self) -> *mut bindings::gpu_buddy {
+ self.inner.inner.get()
+ }
+}
+
+/// GPU buddy allocator instance.
+///
+/// This structure wraps the C `gpu_buddy` allocator using reference counting.
+/// The allocator is automatically cleaned up when all references are dropped.
+///
+/// Refer to the module-level documentation for usage examples.
+pub struct GpuBuddy(Arc<GpuBuddyInner>);
+
+impl GpuBuddy {
+ /// Create a new buddy allocator.
+ ///
+ /// The allocator manages a contiguous address space of the given size, with the
+ /// specified minimum allocation unit (chunk_size must be at least 4KB).
+ pub fn new(params: GpuBuddyParams) -> Result<Self> {
+ Arc::pin_init(GpuBuddyInner::new(params), GFP_KERNEL).map(Self)
+ }
+
+ /// Get the base offset for allocations.
+ pub fn base_offset(&self) -> u64 {
+ self.0.params.base_offset
+ }
+
+ /// Get the chunk size (minimum allocation unit).
+ pub fn chunk_size(&self) -> Alignment {
+ self.0.params.chunk_size
+ }
+
+ /// Get the total managed size.
+ pub fn size(&self) -> u64 {
+ self.0.params.size
+ }
+
+ /// Get the available (free) memory in bytes.
+ pub fn avail(&self) -> u64 {
+ let guard = self.0.lock();
+
+ // SAFETY: Per the type invariant, `inner` contains an initialized allocator.
+ // `guard` provides exclusive access.
+ unsafe { (*guard.as_raw()).avail }
+ }
+
+ /// Allocate blocks from the buddy allocator.
+ ///
+ /// Returns a pin-initializer for [`AllocatedBlocks`].
+ pub fn alloc_blocks(
+ &self,
+ mode: GpuBuddyAllocMode,
+ size: u64,
+ min_block_size: Alignment,
+ flags: impl Into<GpuBuddyAllocFlags>,
+ ) -> impl PinInit<AllocatedBlocks, Error> {
+ let buddy_arc = Arc::clone(&self.0);
+ let (start, end) = mode.range();
+ let mode_flags = mode.as_flags();
+ let modifier_flags = flags.into();
+
+ // Create pin-initializer that initializes list and allocates blocks.
+ try_pin_init!(AllocatedBlocks {
+ buddy: buddy_arc,
+ list <- CListHead::new(),
+ _: {
+ // Reject zero-sized or inverted ranges.
+ if let GpuBuddyAllocMode::Range(range) = &mode {
+ if range.is_empty() {
+ Err::<(), Error>(EINVAL)?;
+ }
+ }
+
+ // Lock while allocating to serialize with concurrent frees.
+ let guard = buddy.lock();
+
+ // SAFETY: Per the type invariant, `inner` contains an initialized
+ // allocator. `guard` provides exclusive access.
+ to_result(unsafe {
+ bindings::gpu_buddy_alloc_blocks(
+ guard.as_raw(),
+ start,
+ end,
+ size,
+ min_block_size.as_usize() as u64,
+ list.as_raw(),
+ mode_flags | usize::from(modifier_flags),
+ )
+ })?
+ }
+ })
+ }
+}
+
+/// Allocated blocks from the buddy allocator with automatic cleanup.
+///
+/// This structure owns a list of allocated blocks and ensures they are
+/// automatically freed when dropped. Use `iter()` to iterate over all
+/// allocated blocks.
+///
+/// # Invariants
+///
+/// - `list` is an initialized, valid list head containing allocated blocks.
+#[pin_data(PinnedDrop)]
+pub struct AllocatedBlocks {
+ #[pin]
+ list: CListHead,
+ buddy: Arc<GpuBuddyInner>,
+}
+
+impl AllocatedBlocks {
+ /// Check if the block list is empty.
+ pub fn is_empty(&self) -> bool {
+ // An empty list head points to itself.
+ !self.list.is_linked()
+ }
+
+ /// Iterate over allocated blocks.
+ ///
+ /// Returns an iterator yielding [`AllocatedBlock`] values. Each [`AllocatedBlock`]
+ /// borrows `self` and is only valid for the duration of that borrow.
+ pub fn iter(&self) -> impl Iterator<Item = AllocatedBlock<'_>> + '_ {
+ let head = self.list.as_raw();
+ // SAFETY: Per the type invariant, `list` is an initialized sentinel `list_head`
+ // and is not concurrently modified (we hold a `&self` borrow). The list contains
+ // `gpu_buddy_block` items linked via `__bindgen_anon_1.link`. `Block` is
+ // `#[repr(transparent)]` over `gpu_buddy_block`.
+ let clist = unsafe {
+ clist_create!(
+ head,
+ Block,
+ bindings::gpu_buddy_block,
+ __bindgen_anon_1.link
+ )
+ };
+
+ clist
+ .iter()
+ .map(|this| AllocatedBlock { this, blocks: self })
+ }
+}
+
+#[pinned_drop]
+impl PinnedDrop for AllocatedBlocks {
+ fn drop(self: Pin<&mut Self>) {
+ let guard = self.buddy.lock();
+
+ // SAFETY:
+ // - list is valid per the type's invariants.
+ // - guard provides exclusive access to the allocator.
+ unsafe {
+ bindings::gpu_buddy_free_list(guard.as_raw(), self.list.as_raw(), 0);
+ }
+ }
+}
+
+/// A GPU buddy block.
+///
+/// Transparent wrapper over C `gpu_buddy_block` structure. This type is returned
+/// as references during iteration over [`AllocatedBlocks`].
+///
+/// # Invariants
+///
+/// The inner [`Opaque`] contains a valid, allocated `gpu_buddy_block`.
+#[repr(transparent)]
+struct Block(Opaque<bindings::gpu_buddy_block>);
+
+impl Block {
+ /// Get a raw pointer to the underlying C block.
+ fn as_raw(&self) -> *mut bindings::gpu_buddy_block {
+ self.0.get()
+ }
+
+ /// Get the block's raw offset in the buddy address space (without base offset).
+ fn offset(&self) -> u64 {
+ // SAFETY: `self.as_raw()` is valid per the type's invariants.
+ unsafe { bindings::gpu_buddy_block_offset(self.as_raw()) }
+ }
+
+ /// Get the block order.
+ fn order(&self) -> u32 {
+ // SAFETY: `self.as_raw()` is valid per the type's invariants.
+ unsafe { bindings::gpu_buddy_block_order(self.as_raw()) }
+ }
+}
+
+// SAFETY: `Block` is a wrapper around `gpu_buddy_block` which can be
+// sent across threads safely.
+unsafe impl Send for Block {}
+
+// SAFETY: `Block` is only accessed through shared references after
+// allocation, and thus safe to access concurrently across threads.
+unsafe impl Sync for Block {}
+
+/// A buddy block paired with its owning [`AllocatedBlocks`] context.
+///
+/// Unlike a raw block, which only knows its offset within the buddy address
+/// space, an [`AllocatedBlock`] also has access to the allocator's `base_offset`
+/// and `chunk_size`, enabling it to compute absolute offsets and byte sizes.
+///
+/// Returned by [`AllocatedBlocks::iter()`].
+pub struct AllocatedBlock<'a> {
+ this: &'a Block,
+ blocks: &'a AllocatedBlocks,
+}
+
+impl AllocatedBlock<'_> {
+ /// Get the block's offset in the address space.
+ ///
+ /// Returns the absolute offset including the allocator's base offset.
+ /// This is the actual address to use for accessing the allocated memory.
+ pub fn offset(&self) -> u64 {
+ self.blocks.buddy.params.base_offset + self.this.offset()
+ }
+
+ /// Get the block order (size = chunk_size << order).
+ pub fn order(&self) -> u32 {
+ self.this.order()
+ }
+
+ /// Get the block's size in bytes.
+ pub fn size(&self) -> u64 {
+ (self.blocks.buddy.params.chunk_size.as_usize() as u64) << self.this.order()
+ }
+}
diff --git a/rust/kernel/interop.rs b/rust/kernel/interop.rs
new file mode 100644
index 000000000000..3b371d782a59
--- /dev/null
+++ b/rust/kernel/interop.rs
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Infrastructure for interfacing Rust code with C kernel subsystems.
+//!
+//! This module is intended for low-level, unsafe Rust infrastructure code
+//! that interoperates between Rust and C. It is *not* for use directly in
+//! Rust drivers.
+
+pub mod list;
diff --git a/rust/kernel/interop/list.rs b/rust/kernel/interop/list.rs
new file mode 100644
index 000000000000..54265ea036bb
--- /dev/null
+++ b/rust/kernel/interop/list.rs
@@ -0,0 +1,339 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Rust interface for C doubly circular intrusive linked lists.
+//!
+//! This module provides Rust abstractions for iterating over C `list_head`-based
+//! linked lists. It should only be used for cases where C and Rust code share
+//! direct access to the same linked list through a C interop interface.
+//!
+//! Note: This *must not* be used by Rust components that just need a linked list
+//! primitive. Use [`kernel::list::List`] instead.
+//!
+//! # Examples
+//!
+//! ```
+//! use kernel::{
+//! bindings,
+//! interop::list::clist_create,
+//! types::Opaque,
+//! };
+//! # // Create test list with values (0, 10, 20) - normally done by C code but it is
+//! # // emulated here for doctests using the C bindings.
+//! # use core::mem::MaybeUninit;
+//! #
+//! # /// C struct with embedded `list_head` (typically will be allocated by C code).
+//! # #[repr(C)]
+//! # pub struct SampleItemC {
+//! # pub value: i32,
+//! # pub link: bindings::list_head,
+//! # }
+//! #
+//! # let mut head = MaybeUninit::<bindings::list_head>::uninit();
+//! #
+//! # let head = head.as_mut_ptr();
+//! # // SAFETY: `head` and all the items are test objects allocated in this scope.
+//! # unsafe { bindings::INIT_LIST_HEAD(head) };
+//! #
+//! # let mut items = [
+//! # MaybeUninit::<SampleItemC>::uninit(),
+//! # MaybeUninit::<SampleItemC>::uninit(),
+//! # MaybeUninit::<SampleItemC>::uninit(),
+//! # ];
+//! #
+//! # for (i, item) in items.iter_mut().enumerate() {
+//! # let ptr = item.as_mut_ptr();
+//! # // SAFETY: `ptr` points to a valid `MaybeUninit<SampleItemC>`.
+//! # unsafe { (*ptr).value = i as i32 * 10 };
+//! # // SAFETY: `&raw mut` creates a pointer valid for `INIT_LIST_HEAD`.
+//! # unsafe { bindings::INIT_LIST_HEAD(&raw mut (*ptr).link) };
+//! # // SAFETY: `link` was just initialized and `head` is a valid list head.
+//! # unsafe { bindings::list_add_tail(&mut (*ptr).link, head) };
+//! # }
+//!
+//! /// Rust wrapper for the C struct.
+//! ///
+//! /// The list item struct in this example is defined in C code as:
+//! ///
+//! /// ```c
+//! /// struct SampleItemC {
+//! /// int value;
+//! /// struct list_head link;
+//! /// };
+//! /// ```
+//! #[repr(transparent)]
+//! pub struct Item(Opaque<SampleItemC>);
+//!
+//! impl Item {
+//! pub fn value(&self) -> i32 {
+//! // SAFETY: `Item` has the same layout as `SampleItemC`.
+//! unsafe { (*self.0.get()).value }
+//! }
+//! }
+//!
+//! // Create typed [`CList`] from sentinel head.
+//! // SAFETY: `head` is valid and initialized, items are `SampleItemC` with
+//! // embedded `link` field, and `Item` is `#[repr(transparent)]` over `SampleItemC`.
+//! let list = unsafe { clist_create!(head, Item, SampleItemC, link) };
+//!
+//! // Iterate directly over typed items.
+//! let mut found_0 = false;
+//! let mut found_10 = false;
+//! let mut found_20 = false;
+//!
+//! for item in list.iter() {
+//! let val = item.value();
+//! if val == 0 { found_0 = true; }
+//! if val == 10 { found_10 = true; }
+//! if val == 20 { found_20 = true; }
+//! }
+//!
+//! assert!(found_0 && found_10 && found_20);
+//! ```
+
+use core::{
+ iter::FusedIterator,
+ marker::PhantomData, //
+};
+
+use crate::{
+ bindings,
+ types::Opaque, //
+};
+
+use pin_init::{
+ pin_data,
+ pin_init,
+ PinInit, //
+};
+
+/// FFI wrapper for a C `list_head` object used in intrusive linked lists.
+///
+/// # Invariants
+///
+/// - The underlying `list_head` is initialized with valid non-`NULL` `next`/`prev` pointers.
+#[pin_data]
+#[repr(transparent)]
+pub struct CListHead {
+ #[pin]
+ inner: Opaque<bindings::list_head>,
+}
+
+impl CListHead {
+ /// Create a `&CListHead` reference from a raw `list_head` pointer.
+ ///
+ /// # Safety
+ ///
+ /// - `ptr` must be a valid pointer to an initialized `list_head` (e.g. via
+ /// `INIT_LIST_HEAD()`), with valid non-`NULL` `next`/`prev` pointers.
+ /// - `ptr` must remain valid for the lifetime `'a`.
+ /// - The list and all linked `list_head` nodes must not be modified from
+ /// anywhere for the lifetime `'a`, unless done so via any [`CListHead`] APIs.
+ #[inline]
+ pub unsafe fn from_raw<'a>(ptr: *mut bindings::list_head) -> &'a Self {
+ // SAFETY:
+ // - `CListHead` has the same layout as `list_head`.
+ // - `ptr` is valid and unmodified for `'a` per caller guarantees.
+ unsafe { &*ptr.cast() }
+ }
+
+ /// Get the raw `list_head` pointer.
+ #[inline]
+ pub fn as_raw(&self) -> *mut bindings::list_head {
+ self.inner.get()
+ }
+
+ /// Get the next [`CListHead`] in the list.
+ #[inline]
+ pub fn next(&self) -> &Self {
+ let raw = self.as_raw();
+ // SAFETY:
+ // - `self.as_raw()` is valid and initialized per type invariants.
+ // - The `next` pointer is valid and non-`NULL` per type invariants
+ // (initialized via `INIT_LIST_HEAD()` or equivalent).
+ unsafe { Self::from_raw((*raw).next) }
+ }
+
+ /// Check if this node is linked in a list (not isolated).
+ #[inline]
+ pub fn is_linked(&self) -> bool {
+ let raw = self.as_raw();
+ // SAFETY: `self.as_raw()` is valid per type invariants.
+ unsafe { (*raw).next != raw && (*raw).prev != raw }
+ }
+
+ /// Returns a pin-initializer for the list head.
+ pub fn new() -> impl PinInit<Self> {
+ pin_init!(Self {
+ // SAFETY: `INIT_LIST_HEAD` initializes `slot` to a valid empty list.
+ inner <- Opaque::ffi_init(|slot| unsafe { bindings::INIT_LIST_HEAD(slot) }),
+ })
+ }
+}
+
+// SAFETY: `list_head` contains no thread-bound state; it only holds
+// `next`/`prev` pointers.
+unsafe impl Send for CListHead {}
+
+// SAFETY: `CListHead` can be shared among threads as modifications are
+// not allowed at the moment.
+unsafe impl Sync for CListHead {}
+
+impl PartialEq for CListHead {
+ #[inline]
+ fn eq(&self, other: &Self) -> bool {
+ core::ptr::eq(self, other)
+ }
+}
+
+impl Eq for CListHead {}
+
+/// Low-level iterator over `list_head` nodes.
+///
+/// An iterator used to iterate over a C intrusive linked list (`list_head`). The caller has to
+/// perform conversion of returned [`CListHead`] to an item (using [`container_of`] or similar).
+///
+/// # Invariants
+///
+/// `current` and `sentinel` are valid references into an initialized linked list.
+struct CListHeadIter<'a> {
+ /// Current position in the list.
+ current: &'a CListHead,
+ /// The sentinel head (used to detect end of iteration).
+ sentinel: &'a CListHead,
+}
+
+impl<'a> Iterator for CListHeadIter<'a> {
+ type Item = &'a CListHead;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
+ // Check if we've reached the sentinel (end of list).
+ if self.current == self.sentinel {
+ return None;
+ }
+
+ let item = self.current;
+ self.current = item.next();
+ Some(item)
+ }
+}
+
+impl<'a> FusedIterator for CListHeadIter<'a> {}
+
+/// A typed C linked list with a sentinel head intended for FFI use-cases where
+/// a C subsystem manages a linked list that Rust code needs to read. Generally
+/// required only for special cases.
+///
+/// A sentinel head [`CListHead`] represents the entire linked list and can be used
+/// for iteration over items of type `T`; it is not associated with a specific item.
+///
+/// The const generic `OFFSET` specifies the byte offset of the `list_head` field within
+/// the struct that `T` wraps.
+///
+/// # Invariants
+///
+/// - The sentinel [`CListHead`] has valid non-`NULL` `next`/`prev` pointers.
+/// - `OFFSET` is the byte offset of the `list_head` field within the struct that `T` wraps.
+/// - All the list's `list_head` nodes have valid non-`NULL` `next`/`prev` pointers.
+#[repr(transparent)]
+pub struct CList<T, const OFFSET: usize>(CListHead, PhantomData<T>);
+
+impl<T, const OFFSET: usize> CList<T, OFFSET> {
+ /// Create a typed [`CList`] reference from a raw sentinel `list_head` pointer.
+ ///
+ /// # Safety
+ ///
+ /// - `ptr` must be a valid pointer to an initialized sentinel `list_head` (e.g. via
+ /// `INIT_LIST_HEAD()`), with valid non-`NULL` `next`/`prev` pointers.
+ /// - `ptr` must remain valid for the lifetime `'a`.
+ /// - The list and all linked nodes must not be concurrently modified for the lifetime `'a`.
+ /// - The list must contain items where the `list_head` field is at byte offset `OFFSET`.
+ /// - `T` must be `#[repr(transparent)]` over the C struct.
+ #[inline]
+ pub unsafe fn from_raw<'a>(ptr: *mut bindings::list_head) -> &'a Self {
+ // SAFETY:
+ // - `CList` has the same layout as `CListHead` due to `#[repr(transparent)]`.
+ // - Caller guarantees `ptr` is a valid, sentinel `list_head` object.
+ unsafe { &*ptr.cast() }
+ }
+
+ /// Check if the list is empty.
+ #[inline]
+ pub fn is_empty(&self) -> bool {
+ !self.0.is_linked()
+ }
+
+ /// Create an iterator over typed items.
+ #[inline]
+ pub fn iter(&self) -> CListIter<'_, T, OFFSET> {
+ let head = &self.0;
+ CListIter {
+ head_iter: CListHeadIter {
+ current: head.next(),
+ sentinel: head,
+ },
+ _phantom: PhantomData,
+ }
+ }
+}
+
+/// High-level iterator over typed list items.
+pub struct CListIter<'a, T, const OFFSET: usize> {
+ head_iter: CListHeadIter<'a>,
+ _phantom: PhantomData<&'a T>,
+}
+
+impl<'a, T, const OFFSET: usize> Iterator for CListIter<'a, T, OFFSET> {
+ type Item = &'a T;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
+ let head = self.head_iter.next()?;
+
+ // Convert to item using `OFFSET`.
+ //
+ // SAFETY: The pointer calculation is valid because `OFFSET` is derived
+ // from `offset_of!` per type invariants.
+ Some(unsafe { &*head.as_raw().byte_sub(OFFSET).cast::<T>() })
+ }
+}
+
+impl<'a, T, const OFFSET: usize> FusedIterator for CListIter<'a, T, OFFSET> {}
+
+/// Create a C doubly-circular linked list interface [`CList`] from a raw `list_head` pointer.
+///
+/// This macro creates a `CList<T, OFFSET>` that can iterate over items of type `$rust_type`
+/// linked via the `$field` field in the underlying C struct `$c_type`.
+///
+/// # Arguments
+///
+/// - `$head`: Raw pointer to the sentinel `list_head` object (`*mut bindings::list_head`).
+/// - `$rust_type`: Each item's Rust wrapper type.
+/// - `$c_type`: Each item's C struct type that contains the embedded `list_head`.
+/// - `$field`: The name of the `list_head` field within the C struct.
+///
+/// # Safety
+///
+/// The caller must ensure:
+///
+/// - `$head` is a valid, initialized sentinel `list_head` (e.g. via `INIT_LIST_HEAD()`)
+/// pointing to a list that is not concurrently modified for the lifetime of the [`CList`].
+/// - The list contains items of type `$c_type` linked via an embedded `$field`.
+/// - `$rust_type` is `#[repr(transparent)]` over `$c_type` or has compatible layout.
+///
+/// # Examples
+///
+/// Refer to the examples in the [`crate::interop::list`] module documentation.
+#[macro_export]
+macro_rules! clist_create {
+ ($head:expr, $rust_type:ty, $c_type:ty, $($field:tt).+) => {{
+ // Compile-time check that field path is a `list_head`.
+ let _: fn(*const $c_type) -> *const $crate::bindings::list_head =
+ |p| &raw const (*p).$($field).+;
+
+ // Calculate offset and create `CList`.
+ const OFFSET: usize = ::core::mem::offset_of!($c_type, $($field).+);
+ $crate::interop::list::CList::<$rust_type, OFFSET>::from_raw($head)
+ }};
+}
+pub use clist_create;
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index 5d22892447bd..b72b2fbe046d 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -72,12 +72,15 @@ pub mod faux;
pub mod firmware;
pub mod fmt;
pub mod fs;
+#[cfg(CONFIG_GPU_BUDDY = "y")]
+pub mod gpu;
#[cfg(CONFIG_I2C = "y")]
pub mod i2c;
pub mod id_pool;
#[doc(hidden)]
pub mod impl_flags;
pub mod init;
+pub mod interop;
pub mod io;
pub mod ioctl;
pub mod iommu;
diff --git a/rust/kernel/uaccess.rs b/rust/kernel/uaccess.rs
index f989539a31b4..6c9c1cce3c63 100644
--- a/rust/kernel/uaccess.rs
+++ b/rust/kernel/uaccess.rs
@@ -7,10 +7,12 @@
use crate::{
alloc::{Allocator, Flags},
bindings,
+ dma::Coherent,
error::Result,
ffi::{c_char, c_void},
fs::file,
prelude::*,
+ ptr::KnownSize,
transmute::{AsBytes, FromBytes},
};
use core::mem::{size_of, MaybeUninit};
@@ -459,20 +461,19 @@ impl UserSliceWriter {
self.length == 0
}
- /// Writes raw data to this user pointer from a kernel buffer.
+ /// Low-level write from a raw pointer.
///
- /// Fails with [`EFAULT`] if the write happens on a bad address, or if the write goes out of
- /// bounds of this [`UserSliceWriter`]. This call may modify the associated userspace slice even
- /// if it returns an error.
- pub fn write_slice(&mut self, data: &[u8]) -> Result {
- let len = data.len();
- let data_ptr = data.as_ptr().cast::<c_void>();
+ /// # Safety
+ ///
+ /// The caller must ensure that `from` is valid for reads of `len` bytes.
+ unsafe fn write_raw(&mut self, from: *const u8, len: usize) -> Result {
if len > self.length {
return Err(EFAULT);
}
- // SAFETY: `data_ptr` points into an immutable slice of length `len`, so we may read
- // that many bytes from it.
- let res = unsafe { bindings::copy_to_user(self.ptr.as_mut_ptr(), data_ptr, len) };
+
+ // SAFETY: Caller guarantees `from` is valid for `len` bytes (see this function's
+ // safety contract).
+ let res = unsafe { bindings::copy_to_user(self.ptr.as_mut_ptr(), from.cast(), len) };
if res != 0 {
return Err(EFAULT);
}
@@ -481,6 +482,76 @@ impl UserSliceWriter {
Ok(())
}
+ /// Writes raw data to this user pointer from a kernel buffer.
+ ///
+ /// Fails with [`EFAULT`] if the write happens on a bad address, or if the write goes out of
+ /// bounds of this [`UserSliceWriter`]. This call may modify the associated userspace slice even
+ /// if it returns an error.
+ pub fn write_slice(&mut self, data: &[u8]) -> Result {
+ // SAFETY: `data` is a valid slice, so `data.as_ptr()` is valid for
+ // reading `data.len()` bytes.
+ unsafe { self.write_raw(data.as_ptr(), data.len()) }
+ }
+
+ /// Writes raw data to this user pointer from a DMA coherent allocation.
+ ///
+ /// Copies `count` bytes from `alloc` starting from `offset` into this userspace slice.
+ ///
+ /// # Errors
+ ///
+ /// - [`EOVERFLOW`]: `offset + count` overflows.
+ /// - [`ERANGE`]: `offset + count` exceeds the size of `alloc`, or `count` exceeds the
+ /// size of the user-space buffer.
+ /// - [`EFAULT`]: the write hits a bad address or goes out of bounds of this
+ /// [`UserSliceWriter`].
+ ///
+ /// This call may modify the associated userspace slice even if it returns an error.
+ ///
+ /// Note: The memory may be concurrently modified by hardware (e.g., DMA). In such cases,
+ /// the copied data may be inconsistent, but this does not cause undefined behavior.
+ ///
+ /// # Example
+ ///
+ /// Copy the first 256 bytes of a DMA coherent allocation into a userspace buffer:
+ ///
+ /// ```no_run
+ /// use kernel::uaccess::UserSliceWriter;
+ /// use kernel::dma::Coherent;
+ ///
+ /// fn copy_dma_to_user(
+ /// mut writer: UserSliceWriter,
+ /// alloc: &Coherent<[u8]>,
+ /// ) -> Result {
+ /// writer.write_dma(alloc, 0, 256)
+ /// }
+ /// ```
+ pub fn write_dma<T: KnownSize + AsBytes + ?Sized>(
+ &mut self,
+ alloc: &Coherent<T>,
+ offset: usize,
+ count: usize,
+ ) -> Result {
+ let len = alloc.size();
+ if offset.checked_add(count).ok_or(EOVERFLOW)? > len {
+ return Err(ERANGE);
+ }
+
+ if count > self.len() {
+ return Err(ERANGE);
+ }
+
+ // SAFETY: `as_ptr()` returns a valid pointer to a memory region of `count()` bytes, as
+ // guaranteed by the `Coherent` invariants. The check above ensures `offset + count <= len`.
+ let src_ptr = unsafe { alloc.as_ptr().cast::<u8>().add(offset) };
+
+ // Note: Use `write_raw` instead of `write_slice` because the allocation is coherent
+ // memory that hardware may modify (e.g., DMA); we cannot form a `&[u8]` slice over
+ // such volatile memory.
+ //
+ // SAFETY: `src_ptr` points into the allocation and is valid for `count` bytes (see above).
+ unsafe { self.write_raw(src_ptr, count) }
+ }
+
/// Writes raw data to this user pointer from a kernel buffer partially.
///
/// This is the same as [`Self::write_slice`] but considers the given `offset` into `data` and
diff --git a/rust/kernel/workqueue.rs b/rust/kernel/workqueue.rs
index 706e833e9702..7e253b6f299c 100644
--- a/rust/kernel/workqueue.rs
+++ b/rust/kernel/workqueue.rs
@@ -189,12 +189,18 @@ use crate::{
alloc::{AllocError, Flags},
container_of,
prelude::*,
- sync::Arc,
- sync::LockClassKey,
+ sync::{
+ aref::{
+ ARef,
+ AlwaysRefCounted, //
+ },
+ Arc,
+ LockClassKey, //
+ },
time::Jiffies,
types::Opaque,
};
-use core::marker::PhantomData;
+use core::{marker::PhantomData, ptr::NonNull};
/// Creates a [`Work`] initialiser with the given name and a newly-created lock class.
#[macro_export]
@@ -425,10 +431,11 @@ pub unsafe trait RawDelayedWorkItem<const ID: u64>: RawWorkItem<ID> {}
/// Defines the method that should be called directly when a work item is executed.
///
-/// This trait is implemented by `Pin<KBox<T>>` and [`Arc<T>`], and is mainly intended to be
-/// implemented for smart pointer types. For your own structs, you would implement [`WorkItem`]
-/// instead. The [`run`] method on this trait will usually just perform the appropriate
-/// `container_of` translation and then call into the [`run`][WorkItem::run] method from the
+/// This trait is implemented by `Pin<KBox<T>>`, [`Arc<T>`] and [`ARef<T>`], and
+/// is mainly intended to be implemented for smart pointer types. For your own
+/// structs, you would implement [`WorkItem`] instead. The [`run`] method on
+/// this trait will usually just perform the appropriate `container_of`
+/// translation and then call into the [`run`][WorkItem::run] method from the
/// [`WorkItem`] trait.
///
/// This trait is used when the `work_struct` field is defined using the [`Work`] helper.
@@ -934,6 +941,89 @@ where
{
}
+// SAFETY: Like the `Arc<T>` implementation, the `__enqueue` implementation for
+// `ARef<T>` obtains a `work_struct` from the `Work` field using
+// `T::raw_get_work`, so the same safety reasoning applies:
+//
+// - `__enqueue` gets the `work_struct` from the `Work` field, using `T::raw_get_work`.
+// - The only safe way to create a `Work` object is through `Work::new`.
+// - `Work::new` makes sure that `T::Pointer::run` is passed to `init_work_with_key`.
+// - Finally `Work` and `RawWorkItem` guarantee that the correct `Work` field
+// will be used because of the ID const generic bound. This makes sure that `T::raw_get_work`
+// uses the correct offset for the `Work` field, and `Work::new` picks the correct
+// implementation of `WorkItemPointer` for `ARef<T>`.
+unsafe impl<T, const ID: u64> WorkItemPointer<ID> for ARef<T>
+where
+ T: AlwaysRefCounted,
+ T: WorkItem<ID, Pointer = Self>,
+ T: HasWork<T, ID>,
+{
+ unsafe extern "C" fn run(ptr: *mut bindings::work_struct) {
+ // The `__enqueue` method always uses a `work_struct` stored in a `Work<T, ID>`.
+ let ptr = ptr.cast::<Work<T, ID>>();
+
+ // SAFETY: This computes the pointer that `__enqueue` got from
+ // `ARef::into_raw`.
+ let ptr = unsafe { T::work_container_of(ptr) };
+
+ // SAFETY: The safety contract of `work_container_of` ensures that it
+ // returns a valid non-null pointer.
+ let ptr = unsafe { NonNull::new_unchecked(ptr) };
+
+ // SAFETY: This pointer comes from `ARef::into_raw` and we've been given
+ // back ownership.
+ let aref = unsafe { ARef::from_raw(ptr) };
+
+ T::run(aref)
+ }
+}
+
+// SAFETY: The `work_struct` raw pointer is guaranteed to be valid for the duration of the call to
+// the closure because we get it from an `ARef`, which means that the ref count will be at least 1,
+// and we don't drop the `ARef` ourselves. If `queue_work_on` returns true, it is further guaranteed
+// to be valid until a call to the function pointer in `work_struct` because we leak the memory it
+// points to, and only reclaim it if the closure returns false, or in `WorkItemPointer::run`, which
+// is what the function pointer in the `work_struct` must be pointing to, according to the safety
+// requirements of `WorkItemPointer`.
+unsafe impl<T, const ID: u64> RawWorkItem<ID> for ARef<T>
+where
+ T: AlwaysRefCounted,
+ T: WorkItem<ID, Pointer = Self>,
+ T: HasWork<T, ID>,
+{
+ type EnqueueOutput = Result<(), Self>;
+
+ unsafe fn __enqueue<F>(self, queue_work_on: F) -> Self::EnqueueOutput
+ where
+ F: FnOnce(*mut bindings::work_struct) -> bool,
+ {
+ let ptr = ARef::into_raw(self);
+
+ // SAFETY: Pointers from ARef::into_raw are valid and non-null.
+ let work_ptr = unsafe { T::raw_get_work(ptr.as_ptr()) };
+ // SAFETY: `raw_get_work` returns a pointer to a valid value.
+ let work_ptr = unsafe { Work::raw_get(work_ptr) };
+
+ if queue_work_on(work_ptr) {
+ Ok(())
+ } else {
+ // SAFETY: The work queue has not taken ownership of the pointer.
+ Err(unsafe { ARef::from_raw(ptr) })
+ }
+ }
+}
+
+// SAFETY: By the safety requirements of `HasDelayedWork`, the `work_struct` returned by methods in
+// `HasWork` provides a `work_struct` that is the `work` field of a `delayed_work`, and the rest of
+// the `delayed_work` has the same access rules as its `work` field.
+unsafe impl<T, const ID: u64> RawDelayedWorkItem<ID> for ARef<T>
+where
+ T: WorkItem<ID, Pointer = Self>,
+ T: HasDelayedWork<T, ID>,
+ T: AlwaysRefCounted,
+{
+}
+
/// Returns the system work queue (`system_wq`).
///
/// It is the one used by `schedule[_delayed]_work[_on]()`. Multi-CPU multi-threaded. There are