diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2026-10-21 07:19:30 -1000 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2026-10-21 07:19:30 -1000 |
| commit | 22a78a25438d246e4e951960523bee6a5155897d (patch) | |
| tree | a7f0fa921b89128fe082215e995ff8b091f8a07d | |
| parent | a804c5742bff224d4cb4d25dd383554d598be32b (diff) | |
| parent | 14b482c98dfe9683b12338561a6e9bd44f7337a9 (diff) | |
Merge __{get,put}_user() removal branch
This is long over-due: the __get_user() and __put_user() functions are
entirely historical, and are never the rigth thing to use any more. And
when mis-used (read: every case) they are not only slower than the right
thing, but also insecure because they don't check the user address
range.
Most of the usage sites have been converted long ago, but various
left-overs remained. So this started out as a single patch that I wrote
as a "show people how to convert __{get,put}_user() to modern code".
In particular, Thomas last year implemented the very nice new scoped
user access region helpers (see commit e497310b4ffb: "uaccess: Provide
scoped user access regions") that makes the modern interfaces
particularly easy to use and really removes any excuse for the old
horrors.
That small example patch then grew to this, partly using Claude as a
glorified coccinelle script to apply the pattern to other places.
There were a few different reasons for using __put_user() and
__get_user() rather than the alternatives, but the original historical
one was performance: when doing multiple accesses off the same base
pointer, doing a single range check followed by multiple non-checked
accesses used to create smaller and faster code.
The keyword here is "used to". It hasn't been true for years, because
the range check isn't actually the real cost these days. The real cost
of user accesses on modern hardware is turning the user accesses on and
off in the first place (on x86, using the STAC/CLAC instructions).
Those sequences get turned into 'scoped_user_read_access() { }' blocks,
see for example that first patch in the series: "signal: use scoped user
access for old-style sigaction".
Another, historical reason for using __get_user() and friends is that
you actually want to do a kernel pointer access, but the pointer might
not be entirely reliable (typically during fault handling), so you want
to have exception handling for the kernel access.
The modern way to do that kind of non-faulting kernel access is using
"get_kernel_nofault()", but some code predates that helper existing at
all, and used __get_user() as a way to just get a memory access with
exception handling. For that pattern, see for example "alpha: use
get_kernel_nofault() in dik_show_code()"
A third bad reason for using __get_user() and friends was that the
proper get_user() had a "might_fault()" in it, and code that ran in
atomic regions but hadn't disabled page faults would result in a
warning. That kind of code needs to explicitly disable page faults,
because "atomic region" is not well defined at run-time and can be
invisible (eg a preempt-off region in a non-preemptible kernel).
See "vdpa_sim: disable page faults around vringh_need_notify_iotlb()"
for an example of that pattern, where the local_bh_disable() now also
grows a pagefault_disable(), which makes the normal get_user() no longer
warn about the user access in non-process context, because it can see
that the potential page fault will not cause problems.
And finally, the really nasty reason to use __get_user() and friends:
pointers that can be either kernel pointers or user pointers. We've
gotten rid of almost all such cases, but a couple remained. See for
example "x86/sev: Use explicit user-or-kernel accessors for #VC memory
operands".
NOTE! This series removes all __get/put_user() calls from generic code
but only does the final removal of the whole interface on x86. Other
architectures have not been entirely converted, for a variety of reasons:
- arm64 makes __{get,put}_user() be exactly the same as
{get,put}_user() so it's harmless and didn't have the dangers of the
legacy use cases.
- any architecture that uses <asm-generic/uaccess.h> will still use
__{get,put}_user as the underlying helper function name
- several architectures had some legacy uses that weren't entirely
trivial and I didn't want to extend this patch series to things that
I don't know and that don't really matter.
but I did do a couple of the obvious "follows the exact same pattern"
just because it could be done mindlessly and automatically.
Let's see if I broke anything, but we should have done this years ago.
* uaccess-cleanups: (48 commits)
drm/i915/gem: ignore errors on presumed_offset write-back
powerpc/align: use get_user_instr() for a user-mode instruction
powerpc/signal64: use scoped user access in setup_trampoline()
powerpc/signal64: use get_user() for the single context reads
powerpc/signal32: use get_user() for the single context reads
KVM: PPC: Book3S HV: use get_user()/put_user() for the HPT fd
powerpc/powernv: use get_user()/put_user() in the LPC debugfs files
s390: use get_user()/put_user() instead of the unchecked variants
ARM: OABI: use scoped user access in epoll_put_uevent()
ARM: signal: stop using __put_user() for the signal frame
ARM: signal: check the FDPIC function descriptor address
alpha: split the prefetch check in do_page_fault() by mode
openrisc: use get_kernel_nofault() in show_registers()
alpha: use get_kernel_nofault() in dik_show_code()
nios2: use scoped user access for the signal frame
nios2: check the signal frame address before writing to it
hexagon: use scoped user access for the signal frame
alpha: use scoped user access when setting up signal frames
alpha: use scoped user access in the sigreturn path
alpha: use scoped user access in osf_sigaction()
...
50 files changed, 698 insertions, 810 deletions
diff --git a/arch/alpha/kernel/signal.c b/arch/alpha/kernel/signal.c index 0c55a7554d63..441d8dd61055 100644 --- a/arch/alpha/kernel/signal.c +++ b/arch/alpha/kernel/signal.c @@ -64,11 +64,11 @@ SYSCALL_DEFINE3(osf_sigaction, int, sig, if (act) { old_sigset_t mask; - if (!access_ok(act, sizeof(*act)) || - __get_user(new_ka.sa.sa_handler, &act->sa_handler) || - __get_user(new_ka.sa.sa_flags, &act->sa_flags) || - __get_user(mask, &act->sa_mask)) - return -EFAULT; + scoped_user_read_access(act, Efault) { + unsafe_get_user(new_ka.sa.sa_handler, &act->sa_handler, Efault); + unsafe_get_user(new_ka.sa.sa_flags, &act->sa_flags, Efault); + unsafe_get_user(mask, &act->sa_mask, Efault); + } siginitset(&new_ka.sa.sa_mask, mask); new_ka.ka_restorer = NULL; } @@ -76,14 +76,17 @@ SYSCALL_DEFINE3(osf_sigaction, int, sig, ret = do_sigaction(sig, act ? &new_ka : NULL, oact ? &old_ka : NULL); if (!ret && oact) { - if (!access_ok(oact, sizeof(*oact)) || - __put_user(old_ka.sa.sa_handler, &oact->sa_handler) || - __put_user(old_ka.sa.sa_flags, &oact->sa_flags) || - __put_user(old_ka.sa.sa_mask.sig[0], &oact->sa_mask)) - return -EFAULT; + scoped_user_write_access(oact, Efault) { + unsafe_put_user(old_ka.sa.sa_handler, &oact->sa_handler, Efault); + unsafe_put_user(old_ka.sa.sa_flags, &oact->sa_flags, Efault); + unsafe_put_user(old_ka.sa.sa_mask.sig[0], &oact->sa_mask, Efault); + } } return ret; + +Efault: + return -EFAULT; } SYSCALL_DEFINE5(rt_sigaction, int, sig, const struct sigaction __user *, act, @@ -149,51 +152,54 @@ restore_sigcontext(struct sigcontext __user *sc, struct pt_regs *regs) { unsigned long usp; struct switch_stack *sw = (struct switch_stack *)regs - 1; - long err = __get_user(regs->pc, &sc->sc_pc); current->restart_block.fn = do_no_restart_syscall; current_thread_info()->status |= TS_SAVED_FP | TS_RESTORE_FP; sw->r26 = (unsigned long) ret_from_sys_call; - err |= __get_user(regs->r0, sc->sc_regs+0); - err |= __get_user(regs->r1, sc->sc_regs+1); - err |= __get_user(regs->r2, sc->sc_regs+2); - err |= __get_user(regs->r3, sc->sc_regs+3); - err |= __get_user(regs->r4, sc->sc_regs+4); - err |= __get_user(regs->r5, sc->sc_regs+5); - err |= __get_user(regs->r6, sc->sc_regs+6); - err |= __get_user(regs->r7, sc->sc_regs+7); - err |= __get_user(regs->r8, sc->sc_regs+8); - err |= __get_user(sw->r9, sc->sc_regs+9); - err |= __get_user(sw->r10, sc->sc_regs+10); - err |= __get_user(sw->r11, sc->sc_regs+11); - err |= __get_user(sw->r12, sc->sc_regs+12); - err |= __get_user(sw->r13, sc->sc_regs+13); - err |= __get_user(sw->r14, sc->sc_regs+14); - err |= __get_user(sw->r15, sc->sc_regs+15); - err |= __get_user(regs->r16, sc->sc_regs+16); - err |= __get_user(regs->r17, sc->sc_regs+17); - err |= __get_user(regs->r18, sc->sc_regs+18); - err |= __get_user(regs->r19, sc->sc_regs+19); - err |= __get_user(regs->r20, sc->sc_regs+20); - err |= __get_user(regs->r21, sc->sc_regs+21); - err |= __get_user(regs->r22, sc->sc_regs+22); - err |= __get_user(regs->r23, sc->sc_regs+23); - err |= __get_user(regs->r24, sc->sc_regs+24); - err |= __get_user(regs->r25, sc->sc_regs+25); - err |= __get_user(regs->r26, sc->sc_regs+26); - err |= __get_user(regs->r27, sc->sc_regs+27); - err |= __get_user(regs->r28, sc->sc_regs+28); - err |= __get_user(regs->gp, sc->sc_regs+29); - err |= __get_user(usp, sc->sc_regs+30); + scoped_user_read_access(sc, Efault) { + unsafe_get_user(regs->pc, &sc->sc_pc, Efault); + unsafe_get_user(regs->r0, sc->sc_regs+0, Efault); + unsafe_get_user(regs->r1, sc->sc_regs+1, Efault); + unsafe_get_user(regs->r2, sc->sc_regs+2, Efault); + unsafe_get_user(regs->r3, sc->sc_regs+3, Efault); + unsafe_get_user(regs->r4, sc->sc_regs+4, Efault); + unsafe_get_user(regs->r5, sc->sc_regs+5, Efault); + unsafe_get_user(regs->r6, sc->sc_regs+6, Efault); + unsafe_get_user(regs->r7, sc->sc_regs+7, Efault); + unsafe_get_user(regs->r8, sc->sc_regs+8, Efault); + unsafe_get_user(sw->r9, sc->sc_regs+9, Efault); + unsafe_get_user(sw->r10, sc->sc_regs+10, Efault); + unsafe_get_user(sw->r11, sc->sc_regs+11, Efault); + unsafe_get_user(sw->r12, sc->sc_regs+12, Efault); + unsafe_get_user(sw->r13, sc->sc_regs+13, Efault); + unsafe_get_user(sw->r14, sc->sc_regs+14, Efault); + unsafe_get_user(sw->r15, sc->sc_regs+15, Efault); + unsafe_get_user(regs->r16, sc->sc_regs+16, Efault); + unsafe_get_user(regs->r17, sc->sc_regs+17, Efault); + unsafe_get_user(regs->r18, sc->sc_regs+18, Efault); + unsafe_get_user(regs->r19, sc->sc_regs+19, Efault); + unsafe_get_user(regs->r20, sc->sc_regs+20, Efault); + unsafe_get_user(regs->r21, sc->sc_regs+21, Efault); + unsafe_get_user(regs->r22, sc->sc_regs+22, Efault); + unsafe_get_user(regs->r23, sc->sc_regs+23, Efault); + unsafe_get_user(regs->r24, sc->sc_regs+24, Efault); + unsafe_get_user(regs->r25, sc->sc_regs+25, Efault); + unsafe_get_user(regs->r26, sc->sc_regs+26, Efault); + unsafe_get_user(regs->r27, sc->sc_regs+27, Efault); + unsafe_get_user(regs->r28, sc->sc_regs+28, Efault); + unsafe_get_user(regs->gp, sc->sc_regs+29, Efault); + unsafe_get_user(usp, sc->sc_regs+30, Efault); + unsafe_copy_from_user(current_thread_info()->fp, + sc->sc_fpregs, 31 * 8, Efault); + unsafe_get_user(current_thread_info()->fp[31], &sc->sc_fpcr, Efault); + } wrusp(usp); + return 0; - err |= __copy_from_user(current_thread_info()->fp, - sc->sc_fpregs, 31 * 8); - err |= __get_user(current_thread_info()->fp[31], &sc->sc_fpcr); - - return err; +Efault: + return -EFAULT; } /* Note that this syscall is also used by setcontext(3) to install @@ -206,11 +212,8 @@ do_sigreturn(struct sigcontext __user *sc) struct pt_regs *regs = current_pt_regs(); sigset_t set; - /* Verify that it's a good sigcontext before using it */ - if (!access_ok(sc, sizeof(*sc))) - goto give_sigsegv; - if (__get_user(set.sig[0], &sc->sc_mask)) - goto give_sigsegv; + scoped_user_read_access(sc, give_sigsegv) + unsafe_get_user(set.sig[0], &sc->sc_mask, give_sigsegv); set_current_blocked(&set); @@ -272,62 +275,66 @@ setup_sigcontext(struct sigcontext __user *sc, struct pt_regs *regs, unsigned long mask, unsigned long sp) { struct switch_stack *sw = (struct switch_stack *)regs - 1; - long err = 0; - - err |= __put_user(on_sig_stack((unsigned long)sc), &sc->sc_onstack); - err |= __put_user(mask, &sc->sc_mask); - err |= __put_user(regs->pc, &sc->sc_pc); - err |= __put_user(8, &sc->sc_ps); - - err |= __put_user(regs->r0 , sc->sc_regs+0); - err |= __put_user(regs->r1 , sc->sc_regs+1); - err |= __put_user(regs->r2 , sc->sc_regs+2); - err |= __put_user(regs->r3 , sc->sc_regs+3); - err |= __put_user(regs->r4 , sc->sc_regs+4); - err |= __put_user(regs->r5 , sc->sc_regs+5); - err |= __put_user(regs->r6 , sc->sc_regs+6); - err |= __put_user(regs->r7 , sc->sc_regs+7); - err |= __put_user(regs->r8 , sc->sc_regs+8); - err |= __put_user(sw->r9 , sc->sc_regs+9); - err |= __put_user(sw->r10 , sc->sc_regs+10); - err |= __put_user(sw->r11 , sc->sc_regs+11); - err |= __put_user(sw->r12 , sc->sc_regs+12); - err |= __put_user(sw->r13 , sc->sc_regs+13); - err |= __put_user(sw->r14 , sc->sc_regs+14); - err |= __put_user(sw->r15 , sc->sc_regs+15); - err |= __put_user(regs->r16, sc->sc_regs+16); - err |= __put_user(regs->r17, sc->sc_regs+17); - err |= __put_user(regs->r18, sc->sc_regs+18); - err |= __put_user(regs->r19, sc->sc_regs+19); - err |= __put_user(regs->r20, sc->sc_regs+20); - err |= __put_user(regs->r21, sc->sc_regs+21); - err |= __put_user(regs->r22, sc->sc_regs+22); - err |= __put_user(regs->r23, sc->sc_regs+23); - err |= __put_user(regs->r24, sc->sc_regs+24); - err |= __put_user(regs->r25, sc->sc_regs+25); - err |= __put_user(regs->r26, sc->sc_regs+26); - err |= __put_user(regs->r27, sc->sc_regs+27); - err |= __put_user(regs->r28, sc->sc_regs+28); - err |= __put_user(regs->gp , sc->sc_regs+29); - err |= __put_user(sp, sc->sc_regs+30); - err |= __put_user(0, sc->sc_regs+31); - - err |= __copy_to_user(sc->sc_fpregs, - current_thread_info()->fp, 31 * 8); - err |= __put_user(0, sc->sc_fpregs+31); - err |= __put_user(current_thread_info()->fp[31], &sc->sc_fpcr); - - err |= __put_user(regs->trap_a0, &sc->sc_traparg_a0); - err |= __put_user(regs->trap_a1, &sc->sc_traparg_a1); - err |= __put_user(regs->trap_a2, &sc->sc_traparg_a2); - - return err; + long onstack = on_sig_stack((unsigned long)sc); + + scoped_user_write_access(sc, Efault) { + unsafe_put_user(onstack, &sc->sc_onstack, Efault); + unsafe_put_user(mask, &sc->sc_mask, Efault); + unsafe_put_user(regs->pc, &sc->sc_pc, Efault); + unsafe_put_user(8, &sc->sc_ps, Efault); + + unsafe_put_user(regs->r0, sc->sc_regs+0, Efault); + unsafe_put_user(regs->r1, sc->sc_regs+1, Efault); + unsafe_put_user(regs->r2, sc->sc_regs+2, Efault); + unsafe_put_user(regs->r3, sc->sc_regs+3, Efault); + unsafe_put_user(regs->r4, sc->sc_regs+4, Efault); + unsafe_put_user(regs->r5, sc->sc_regs+5, Efault); + unsafe_put_user(regs->r6, sc->sc_regs+6, Efault); + unsafe_put_user(regs->r7, sc->sc_regs+7, Efault); + unsafe_put_user(regs->r8, sc->sc_regs+8, Efault); + unsafe_put_user(sw->r9, sc->sc_regs+9, Efault); + unsafe_put_user(sw->r10, sc->sc_regs+10, Efault); + unsafe_put_user(sw->r11, sc->sc_regs+11, Efault); + unsafe_put_user(sw->r12, sc->sc_regs+12, Efault); + unsafe_put_user(sw->r13, sc->sc_regs+13, Efault); + unsafe_put_user(sw->r14, sc->sc_regs+14, Efault); + unsafe_put_user(sw->r15, sc->sc_regs+15, Efault); + unsafe_put_user(regs->r16, sc->sc_regs+16, Efault); + unsafe_put_user(regs->r17, sc->sc_regs+17, Efault); + unsafe_put_user(regs->r18, sc->sc_regs+18, Efault); + unsafe_put_user(regs->r19, sc->sc_regs+19, Efault); + unsafe_put_user(regs->r20, sc->sc_regs+20, Efault); + unsafe_put_user(regs->r21, sc->sc_regs+21, Efault); + unsafe_put_user(regs->r22, sc->sc_regs+22, Efault); + unsafe_put_user(regs->r23, sc->sc_regs+23, Efault); + unsafe_put_user(regs->r24, sc->sc_regs+24, Efault); + unsafe_put_user(regs->r25, sc->sc_regs+25, Efault); + unsafe_put_user(regs->r26, sc->sc_regs+26, Efault); + unsafe_put_user(regs->r27, sc->sc_regs+27, Efault); + unsafe_put_user(regs->r28, sc->sc_regs+28, Efault); + unsafe_put_user(regs->gp, sc->sc_regs+29, Efault); + unsafe_put_user(sp, sc->sc_regs+30, Efault); + unsafe_put_user(0, sc->sc_regs+31, Efault); + + unsafe_copy_to_user(sc->sc_fpregs, + current_thread_info()->fp, 31 * 8, Efault); + unsafe_put_user(0, sc->sc_fpregs+31, Efault); + unsafe_put_user(current_thread_info()->fp[31], &sc->sc_fpcr, Efault); + + unsafe_put_user(regs->trap_a0, &sc->sc_traparg_a0, Efault); + unsafe_put_user(regs->trap_a1, &sc->sc_traparg_a1, Efault); + unsafe_put_user(regs->trap_a2, &sc->sc_traparg_a2, Efault); + } + return 0; + +Efault: + return -EFAULT; } static int setup_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) { - unsigned long oldsp, r26, err = 0; + unsigned long oldsp, r26; struct sigframe __user *frame; oldsp = rdusp(); @@ -335,25 +342,22 @@ setup_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) if (!access_ok(frame, sizeof(*frame))) return -EFAULT; - err |= setup_sigcontext(&frame->sc, regs, set->sig[0], oldsp); - if (err) + if (setup_sigcontext(&frame->sc, regs, set->sig[0], oldsp)) return -EFAULT; /* Set up to return from userspace. If provided, use a stub already in userspace. */ r26 = (unsigned long) ksig->ka.ka_restorer; if (!r26) { - err |= __put_user(INSN_MOV_R30_R16, frame->retcode+0); - err |= __put_user(INSN_LDI_R0+__NR_sigreturn, frame->retcode+1); - err |= __put_user(INSN_CALLSYS, frame->retcode+2); + scoped_user_write_access(frame, Efault) { + unsafe_put_user(INSN_MOV_R30_R16, frame->retcode+0, Efault); + unsafe_put_user(INSN_LDI_R0+__NR_sigreturn, frame->retcode+1, Efault); + unsafe_put_user(INSN_CALLSYS, frame->retcode+2, Efault); + } imb(); r26 = (unsigned long) frame->retcode; } - /* Check that everything was written properly. */ - if (err) - return err; - /* "Return" to the handler */ regs->r26 = r26; regs->r27 = regs->pc = (unsigned long) ksig->ka.sa.sa_handler; @@ -367,6 +371,9 @@ setup_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) current->comm, current->pid, frame, regs->pc, regs->r26); #endif return 0; + +Efault: + return -EFAULT; } static int @@ -383,9 +390,11 @@ setup_rt_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) err |= copy_siginfo_to_user(&frame->info, &ksig->info); /* Create the ucontext. */ - err |= __put_user(0, &frame->uc.uc_flags); - err |= __put_user(0, &frame->uc.uc_link); - err |= __put_user(set->sig[0], &frame->uc.uc_osf_sigmask); + scoped_user_write_access(frame, Efault) { + unsafe_put_user(0, &frame->uc.uc_flags, Efault); + unsafe_put_user(0, &frame->uc.uc_link, Efault); + unsafe_put_user(set->sig[0], &frame->uc.uc_osf_sigmask, Efault); + } err |= __save_altstack(&frame->uc.uc_stack, oldsp); err |= setup_sigcontext(&frame->uc.uc_mcontext, regs, set->sig[0], oldsp); @@ -397,17 +406,16 @@ setup_rt_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) already in userspace. */ r26 = (unsigned long) ksig->ka.ka_restorer; if (!r26) { - err |= __put_user(INSN_MOV_R30_R16, frame->retcode+0); - err |= __put_user(INSN_LDI_R0+__NR_rt_sigreturn, - frame->retcode+1); - err |= __put_user(INSN_CALLSYS, frame->retcode+2); + scoped_user_write_access(frame, Efault) { + unsafe_put_user(INSN_MOV_R30_R16, frame->retcode+0, Efault); + unsafe_put_user(INSN_LDI_R0+__NR_rt_sigreturn, + frame->retcode+1, Efault); + unsafe_put_user(INSN_CALLSYS, frame->retcode+2, Efault); + } imb(); r26 = (unsigned long) frame->retcode; } - if (err) - return -EFAULT; - /* "Return" to the handler */ regs->r26 = r26; regs->r27 = regs->pc = (unsigned long) ksig->ka.sa.sa_handler; @@ -422,6 +430,9 @@ setup_rt_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) #endif return 0; + +Efault: + return -EFAULT; } diff --git a/arch/alpha/kernel/traps.c b/arch/alpha/kernel/traps.c index 5882c65b7ef1..db6277b3bed0 100644 --- a/arch/alpha/kernel/traps.c +++ b/arch/alpha/kernel/traps.c @@ -81,7 +81,7 @@ dik_show_code(unsigned int *pc) printk("Code:"); for (i = -6; i < 2; i++) { unsigned int insn; - if (__get_user(insn, (unsigned int __user *)pc + i)) + if (get_kernel_nofault(insn, pc + i)) break; printk("%c%08x%c", i ? ' ' : '<', insn, i ? ' ' : '>'); } diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c index 63a24afa7a15..3dfeb854f196 100644 --- a/arch/alpha/mm/fault.c +++ b/arch/alpha/mm/fault.c @@ -108,6 +108,28 @@ __load_new_mm_context(struct mm_struct *next_mm) (((unsigned long *)regs)[(r) <= 8 ? (r) : (r) <= 15 ? (r)-17 : \ (r) <= 18 ? (r)+11 : (r)-10]) +/* + * As of EV6, a load into $31/$f31 is a prefetch, and never faults + * (or is suppressed by the PALcode). Support that for older CPUs + * by ignoring such an instruction. + */ +static bool is_prefetch(struct pt_regs *regs) +{ + unsigned int insn; + int err; + + if (user_mode(regs)) + err = get_user(insn, (unsigned int __user *)regs->pc); + else + err = get_kernel_nofault(insn, (unsigned int *)regs->pc); + if (err) + return false; + + /* ldq ldl ldt lds ldg ldf ldwu ldbu */ + return (insn >> 21 & 0x1f) == 0x1f && + (1ul << (insn >> 26) & 0x30f00001400ul); +} + asmlinkage void do_page_fault(unsigned long address, unsigned long mmcsr, long cause, struct pt_regs *regs) @@ -120,18 +142,9 @@ do_page_fault(unsigned long address, unsigned long mmcsr, vm_fault_t fault; unsigned int flags = FAULT_FLAG_DEFAULT; - /* As of EV6, a load into $31/$f31 is a prefetch, and never faults - (or is suppressed by the PALcode). Support that for older CPUs - by ignoring such an instruction. */ - if (cause == 0) { - unsigned int insn; - __get_user(insn, (unsigned int __user *)regs->pc); - if ((insn >> 21 & 0x1f) == 0x1f && - /* ldq ldl ldt lds ldg ldf ldwu ldbu */ - (1ul << (insn >> 26) & 0x30f00001400ul)) { - regs->pc += 4; - return; - } + if (cause == 0 && is_prefetch(regs)) { + regs->pc += 4; + return; } /* If we're in an interrupt context, or have no user context, diff --git a/arch/arm/kernel/signal.c b/arch/arm/kernel/signal.c index 7be9188d83d9..763e1930a1ea 100644 --- a/arch/arm/kernel/signal.c +++ b/arch/arm/kernel/signal.c @@ -292,7 +292,7 @@ setup_sigframe(struct sigframe __user *sf, struct pt_regs *regs, sigset_t *set) if (err == 0) err |= preserve_vfp_context(&aux->vfp); #endif - err |= __put_user(0, &aux->end_magic); + err |= put_user(0, &aux->end_magic); return err; } @@ -332,8 +332,8 @@ setup_return(struct pt_regs *regs, struct ksignal *ksig, if (fdpic) { unsigned long __user *fdpic_func_desc = (unsigned long __user *)handler; - if (__get_user(handler, &fdpic_func_desc[0]) || - __get_user(handler_fdpic_GOT, &fdpic_func_desc[1])) + if (get_user(handler, &fdpic_func_desc[0]) || + get_user(handler_fdpic_GOT, &fdpic_func_desc[1])) return 1; } @@ -387,11 +387,12 @@ setup_return(struct pt_regs *regs, struct ksignal *ksig, idx = 6 + thumb * 3; if (ksig->ka.sa.sa_flags & SA_SIGINFO) idx += 5; - if (__put_user(sigreturn_codes[idx], rc ) || - __put_user(sigreturn_codes[idx+1], rc+1) || - __put_user(sigreturn_codes[idx+2], rc+2) || - __put_user(retcode, rc+3)) - return 1; + scoped_user_write_access_size(rc, 4 * sizeof(*rc), efault) { + unsafe_put_user(sigreturn_codes[idx], rc, efault); + unsafe_put_user(sigreturn_codes[idx+1], rc+1, efault); + unsafe_put_user(sigreturn_codes[idx+2], rc+2, efault); + unsafe_put_user(retcode, rc+3, efault); + } goto rc_finish; } } else { @@ -403,9 +404,10 @@ setup_return(struct pt_regs *regs, struct ksignal *ksig, * Put the sigreturn code on the stack no matter which return * mechanism we use in order to remain ABI compliant */ - if (__put_user(sigreturn_codes[idx], rc) || - __put_user(sigreturn_codes[idx+1], rc+1)) - return 1; + scoped_user_write_access_size(rc, 2 * sizeof(*rc), efault) { + unsafe_put_user(sigreturn_codes[idx], rc, efault); + unsafe_put_user(sigreturn_codes[idx+1], rc+1, efault); + } rc_finish: #ifdef CONFIG_MMU @@ -442,6 +444,9 @@ rc_finish: regs->ARM_cpsr = cpsr; return 0; + +efault: + return 1; } static int @@ -456,7 +461,7 @@ setup_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) /* * Set uc.uc_flags to a value which sc.trap_no would never have. */ - err = __put_user(0x5ac3c35a, &frame->uc.uc_flags); + err = put_user(0x5ac3c35a, &frame->uc.uc_flags); err |= setup_sigframe(frame, regs, set); if (err == 0) @@ -476,8 +481,10 @@ setup_rt_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) err |= copy_siginfo_to_user(&frame->info, &ksig->info); - err |= __put_user(0, &frame->sig.uc.uc_flags); - err |= __put_user(NULL, &frame->sig.uc.uc_link); + scoped_user_write_access(frame, efault) { + unsafe_put_user(0, &frame->sig.uc.uc_flags, efault); + unsafe_put_user(NULL, &frame->sig.uc.uc_link, efault); + } err |= __save_altstack(&frame->sig.uc.uc_stack, regs->ARM_sp); err |= setup_sigframe(&frame->sig, regs, set); @@ -495,6 +502,9 @@ setup_rt_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) } return err; + +efault: + return -EFAULT; } /* diff --git a/arch/arm/kernel/sys_oabi-compat.c b/arch/arm/kernel/sys_oabi-compat.c index 3d7f7fcf2d87..2917b927162a 100644 --- a/arch/arm/kernel/sys_oabi-compat.c +++ b/arch/arm/kernel/sys_oabi-compat.c @@ -310,18 +310,21 @@ epoll_put_uevent(__poll_t revents, __u64 data, if (in_oabi_syscall()) { struct oabi_epoll_event __user *oevent = (void __user *)uevent; - if (__put_user(revents, &oevent->events) || - __put_user(data, &oevent->data)) - return NULL; - + scoped_user_write_access(oevent, efault) { + unsafe_put_user(revents, &oevent->events, efault); + unsafe_put_user(data, &oevent->data, efault); + } return (void __user *)(oevent+1); } - if (__put_user(revents, &uevent->events) || - __put_user(data, &uevent->data)) - return NULL; - + scoped_user_write_access(uevent, efault) { + unsafe_put_user(revents, &uevent->events, efault); + unsafe_put_user(data, &uevent->data, efault); + } return uevent+1; + +efault: + return NULL; } struct oabi_sembuf { diff --git a/arch/hexagon/kernel/signal.c b/arch/hexagon/kernel/signal.c index d301f4621553..76f54b9d707e 100644 --- a/arch/hexagon/kernel/signal.c +++ b/arch/hexagon/kernel/signal.c @@ -34,59 +34,64 @@ static void __user *get_sigframe(struct ksignal *ksig, struct pt_regs *regs, static int setup_sigcontext(struct pt_regs *regs, struct sigcontext __user *sc) { - unsigned long tmp; - int err = 0; - - err |= copy_to_user(&sc->sc_regs.r0, ®s->r00, - 32*sizeof(unsigned long)); - - err |= __put_user(regs->sa0, &sc->sc_regs.sa0); - err |= __put_user(regs->lc0, &sc->sc_regs.lc0); - err |= __put_user(regs->sa1, &sc->sc_regs.sa1); - err |= __put_user(regs->lc1, &sc->sc_regs.lc1); - err |= __put_user(regs->m0, &sc->sc_regs.m0); - err |= __put_user(regs->m1, &sc->sc_regs.m1); - err |= __put_user(regs->usr, &sc->sc_regs.usr); - err |= __put_user(regs->preds, &sc->sc_regs.p3_0); - err |= __put_user(regs->gp, &sc->sc_regs.gp); - err |= __put_user(regs->ugp, &sc->sc_regs.ugp); + scoped_user_write_access(sc, Efault) { + unsafe_copy_to_user(&sc->sc_regs.r0, ®s->r00, + 32*sizeof(unsigned long), Efault); + + unsafe_put_user(regs->sa0, &sc->sc_regs.sa0, Efault); + unsafe_put_user(regs->lc0, &sc->sc_regs.lc0, Efault); + unsafe_put_user(regs->sa1, &sc->sc_regs.sa1, Efault); + unsafe_put_user(regs->lc1, &sc->sc_regs.lc1, Efault); + unsafe_put_user(regs->m0, &sc->sc_regs.m0, Efault); + unsafe_put_user(regs->m1, &sc->sc_regs.m1, Efault); + unsafe_put_user(regs->usr, &sc->sc_regs.usr, Efault); + unsafe_put_user(regs->preds, &sc->sc_regs.p3_0, Efault); + unsafe_put_user(regs->gp, &sc->sc_regs.gp, Efault); + unsafe_put_user(regs->ugp, &sc->sc_regs.ugp, Efault); #if CONFIG_HEXAGON_ARCH_VERSION >= 4 - err |= __put_user(regs->cs0, &sc->sc_regs.cs0); - err |= __put_user(regs->cs1, &sc->sc_regs.cs1); + unsafe_put_user(regs->cs0, &sc->sc_regs.cs0, Efault); + unsafe_put_user(regs->cs1, &sc->sc_regs.cs1, Efault); #endif - tmp = pt_elr(regs); err |= __put_user(tmp, &sc->sc_regs.pc); - tmp = pt_cause(regs); err |= __put_user(tmp, &sc->sc_regs.cause); - tmp = pt_badva(regs); err |= __put_user(tmp, &sc->sc_regs.badva); + unsafe_put_user(pt_elr(regs), &sc->sc_regs.pc, Efault); + unsafe_put_user(pt_cause(regs), &sc->sc_regs.cause, Efault); + unsafe_put_user(pt_badva(regs), &sc->sc_regs.badva, Efault); + } + return 0; - return err; +Efault: + return -EFAULT; } static int restore_sigcontext(struct pt_regs *regs, struct sigcontext __user *sc) { unsigned long tmp; - int err = 0; - err |= copy_from_user(®s->r00, &sc->sc_regs.r0, - 32 * sizeof(unsigned long)); - - err |= __get_user(regs->sa0, &sc->sc_regs.sa0); - err |= __get_user(regs->lc0, &sc->sc_regs.lc0); - err |= __get_user(regs->sa1, &sc->sc_regs.sa1); - err |= __get_user(regs->lc1, &sc->sc_regs.lc1); - err |= __get_user(regs->m0, &sc->sc_regs.m0); - err |= __get_user(regs->m1, &sc->sc_regs.m1); - err |= __get_user(regs->usr, &sc->sc_regs.usr); - err |= __get_user(regs->preds, &sc->sc_regs.p3_0); - err |= __get_user(regs->gp, &sc->sc_regs.gp); - err |= __get_user(regs->ugp, &sc->sc_regs.ugp); + scoped_user_read_access(sc, Efault) { + unsafe_copy_from_user(®s->r00, &sc->sc_regs.r0, + 32 * sizeof(unsigned long), Efault); + + unsafe_get_user(regs->sa0, &sc->sc_regs.sa0, Efault); + unsafe_get_user(regs->lc0, &sc->sc_regs.lc0, Efault); + unsafe_get_user(regs->sa1, &sc->sc_regs.sa1, Efault); + unsafe_get_user(regs->lc1, &sc->sc_regs.lc1, Efault); + unsafe_get_user(regs->m0, &sc->sc_regs.m0, Efault); + unsafe_get_user(regs->m1, &sc->sc_regs.m1, Efault); + unsafe_get_user(regs->usr, &sc->sc_regs.usr, Efault); + unsafe_get_user(regs->preds, &sc->sc_regs.p3_0, Efault); + unsafe_get_user(regs->gp, &sc->sc_regs.gp, Efault); + unsafe_get_user(regs->ugp, &sc->sc_regs.ugp, Efault); #if CONFIG_HEXAGON_ARCH_VERSION >= 4 - err |= __get_user(regs->cs0, &sc->sc_regs.cs0); - err |= __get_user(regs->cs1, &sc->sc_regs.cs1); + unsafe_get_user(regs->cs0, &sc->sc_regs.cs0, Efault); + unsafe_get_user(regs->cs1, &sc->sc_regs.cs1, Efault); #endif - err |= __get_user(tmp, &sc->sc_regs.pc); pt_set_elr(regs, tmp); + unsafe_get_user(tmp, &sc->sc_regs.pc, Efault); + } + pt_set_elr(regs, tmp); + return 0; - return err; +Efault: + return -EFAULT; } /* @@ -111,8 +116,10 @@ static int setup_rt_frame(struct ksignal *ksig, sigset_t *set, * however, the libgcc signal frame unwinding code checks for * the presence of these two numeric magic values. */ - err |= __put_user(0x7800d166, &frame->tramp[0]); - err |= __put_user(0x5400c004, &frame->tramp[1]); + scoped_user_write_access(frame, Efault) { + unsafe_put_user(0x7800d166, &frame->tramp[0], Efault); + unsafe_put_user(0x5400c004, &frame->tramp[1], Efault); + } err |= setup_sigcontext(regs, &frame->uc.uc_mcontext); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); err |= __save_altstack(&frame->uc.uc_stack, user_stack_pointer(regs)); @@ -128,6 +135,9 @@ static int setup_rt_frame(struct ksignal *ksig, sigset_t *set, pt_set_elr(regs, (unsigned long)ksig->ka.sa.sa_handler); return 0; + +Efault: + return -EFAULT; } /* diff --git a/arch/nios2/kernel/signal.c b/arch/nios2/kernel/signal.c index a5b93a30c6eb..3ef95d6d8d5c 100644 --- a/arch/nios2/kernel/signal.c +++ b/arch/nios2/kernel/signal.c @@ -39,60 +39,60 @@ static inline int rt_restore_ucontext(struct pt_regs *regs, struct ucontext __user *uc, int *pr2) { int temp; - unsigned long __user *gregs = uc->uc_mcontext.gregs; - int err; /* Always make any pending restarted system calls return -EINTR */ current->restart_block.fn = do_no_restart_syscall; - err = __get_user(temp, &uc->uc_mcontext.version); - if (temp != MCONTEXT_VERSION) - goto badframe; - /* restore passed registers */ - err |= __get_user(regs->r1, &gregs[0]); - err |= __get_user(regs->r2, &gregs[1]); - err |= __get_user(regs->r3, &gregs[2]); - err |= __get_user(regs->r4, &gregs[3]); - err |= __get_user(regs->r5, &gregs[4]); - err |= __get_user(regs->r6, &gregs[5]); - err |= __get_user(regs->r7, &gregs[6]); - err |= __get_user(regs->r8, &gregs[7]); - err |= __get_user(regs->r9, &gregs[8]); - err |= __get_user(regs->r10, &gregs[9]); - err |= __get_user(regs->r11, &gregs[10]); - err |= __get_user(regs->r12, &gregs[11]); - err |= __get_user(regs->r13, &gregs[12]); - err |= __get_user(regs->r14, &gregs[13]); - err |= __get_user(regs->r15, &gregs[14]); - err |= __get_user(sw->r16, &gregs[15]); - err |= __get_user(sw->r17, &gregs[16]); - err |= __get_user(sw->r18, &gregs[17]); - err |= __get_user(sw->r19, &gregs[18]); - err |= __get_user(sw->r20, &gregs[19]); - err |= __get_user(sw->r21, &gregs[20]); - err |= __get_user(sw->r22, &gregs[21]); - err |= __get_user(sw->r23, &gregs[22]); - /* gregs[23] is handled below */ - err |= __get_user(sw->fp, &gregs[24]); /* Verify, should this be - settable */ - err |= __get_user(sw->gp, &gregs[25]); /* Verify, should this be - settable */ - - err |= __get_user(temp, &gregs[26]); /* Not really necessary no user - settable bits */ - err |= __get_user(regs->ea, &gregs[27]); - - err |= __get_user(regs->ra, &gregs[23]); - err |= __get_user(regs->sp, &gregs[28]); + scoped_user_read_access(uc, badframe) { + unsigned long __user *gregs = uc->uc_mcontext.gregs; + + unsafe_get_user(temp, &uc->uc_mcontext.version, badframe); + if (temp != MCONTEXT_VERSION) + goto badframe; + /* restore passed registers */ + unsafe_get_user(regs->r1, &gregs[0], badframe); + unsafe_get_user(regs->r2, &gregs[1], badframe); + unsafe_get_user(regs->r3, &gregs[2], badframe); + unsafe_get_user(regs->r4, &gregs[3], badframe); + unsafe_get_user(regs->r5, &gregs[4], badframe); + unsafe_get_user(regs->r6, &gregs[5], badframe); + unsafe_get_user(regs->r7, &gregs[6], badframe); + unsafe_get_user(regs->r8, &gregs[7], badframe); + unsafe_get_user(regs->r9, &gregs[8], badframe); + unsafe_get_user(regs->r10, &gregs[9], badframe); + unsafe_get_user(regs->r11, &gregs[10], badframe); + unsafe_get_user(regs->r12, &gregs[11], badframe); + unsafe_get_user(regs->r13, &gregs[12], badframe); + unsafe_get_user(regs->r14, &gregs[13], badframe); + unsafe_get_user(regs->r15, &gregs[14], badframe); + unsafe_get_user(sw->r16, &gregs[15], badframe); + unsafe_get_user(sw->r17, &gregs[16], badframe); + unsafe_get_user(sw->r18, &gregs[17], badframe); + unsafe_get_user(sw->r19, &gregs[18], badframe); + unsafe_get_user(sw->r20, &gregs[19], badframe); + unsafe_get_user(sw->r21, &gregs[20], badframe); + unsafe_get_user(sw->r22, &gregs[21], badframe); + unsafe_get_user(sw->r23, &gregs[22], badframe); + /* gregs[23] is handled below */ + /* Verify, should these be settable */ + unsafe_get_user(sw->fp, &gregs[24], badframe); + unsafe_get_user(sw->gp, &gregs[25], badframe); + + /* Not really necessary no user settable bits */ + unsafe_get_user(temp, &gregs[26], badframe); + unsafe_get_user(regs->ea, &gregs[27], badframe); + + unsafe_get_user(regs->ra, &gregs[23], badframe); + unsafe_get_user(regs->sp, &gregs[28], badframe); + } regs->orig_r2 = -1; /* disable syscall checks */ - err |= restore_altstack(&uc->uc_stack); - if (err) + if (restore_altstack(&uc->uc_stack)) goto badframe; *pr2 = regs->r2; - return err; + return 0; badframe: return 1; @@ -128,39 +128,44 @@ badframe: static inline int rt_setup_ucontext(struct ucontext __user *uc, struct pt_regs *regs) { struct switch_stack *sw = (struct switch_stack *)regs - 1; - unsigned long __user *gregs = uc->uc_mcontext.gregs; - int err = 0; - err |= __put_user(MCONTEXT_VERSION, &uc->uc_mcontext.version); - err |= __put_user(regs->r1, &gregs[0]); - err |= __put_user(regs->r2, &gregs[1]); - err |= __put_user(regs->r3, &gregs[2]); - err |= __put_user(regs->r4, &gregs[3]); - err |= __put_user(regs->r5, &gregs[4]); - err |= __put_user(regs->r6, &gregs[5]); - err |= __put_user(regs->r7, &gregs[6]); - err |= __put_user(regs->r8, &gregs[7]); - err |= __put_user(regs->r9, &gregs[8]); - err |= __put_user(regs->r10, &gregs[9]); - err |= __put_user(regs->r11, &gregs[10]); - err |= __put_user(regs->r12, &gregs[11]); - err |= __put_user(regs->r13, &gregs[12]); - err |= __put_user(regs->r14, &gregs[13]); - err |= __put_user(regs->r15, &gregs[14]); - err |= __put_user(sw->r16, &gregs[15]); - err |= __put_user(sw->r17, &gregs[16]); - err |= __put_user(sw->r18, &gregs[17]); - err |= __put_user(sw->r19, &gregs[18]); - err |= __put_user(sw->r20, &gregs[19]); - err |= __put_user(sw->r21, &gregs[20]); - err |= __put_user(sw->r22, &gregs[21]); - err |= __put_user(sw->r23, &gregs[22]); - err |= __put_user(regs->ra, &gregs[23]); - err |= __put_user(sw->fp, &gregs[24]); - err |= __put_user(sw->gp, &gregs[25]); - err |= __put_user(regs->ea, &gregs[27]); - err |= __put_user(regs->sp, &gregs[28]); - return err; + scoped_user_write_access(uc, Efault) { + unsigned long __user *gregs = uc->uc_mcontext.gregs; + + unsafe_put_user(MCONTEXT_VERSION, &uc->uc_mcontext.version, Efault); + unsafe_put_user(regs->r1, &gregs[0], Efault); + unsafe_put_user(regs->r2, &gregs[1], Efault); + unsafe_put_user(regs->r3, &gregs[2], Efault); + unsafe_put_user(regs->r4, &gregs[3], Efault); + unsafe_put_user(regs->r5, &gregs[4], Efault); + unsafe_put_user(regs->r6, &gregs[5], Efault); + unsafe_put_user(regs->r7, &gregs[6], Efault); + unsafe_put_user(regs->r8, &gregs[7], Efault); + unsafe_put_user(regs->r9, &gregs[8], Efault); + unsafe_put_user(regs->r10, &gregs[9], Efault); + unsafe_put_user(regs->r11, &gregs[10], Efault); + unsafe_put_user(regs->r12, &gregs[11], Efault); + unsafe_put_user(regs->r13, &gregs[12], Efault); + unsafe_put_user(regs->r14, &gregs[13], Efault); + unsafe_put_user(regs->r15, &gregs[14], Efault); + unsafe_put_user(sw->r16, &gregs[15], Efault); + unsafe_put_user(sw->r17, &gregs[16], Efault); + unsafe_put_user(sw->r18, &gregs[17], Efault); + unsafe_put_user(sw->r19, &gregs[18], Efault); + unsafe_put_user(sw->r20, &gregs[19], Efault); + unsafe_put_user(sw->r21, &gregs[20], Efault); + unsafe_put_user(sw->r22, &gregs[21], Efault); + unsafe_put_user(sw->r23, &gregs[22], Efault); + unsafe_put_user(regs->ra, &gregs[23], Efault); + unsafe_put_user(sw->fp, &gregs[24], Efault); + unsafe_put_user(sw->gp, &gregs[25], Efault); + unsafe_put_user(regs->ea, &gregs[27], Efault); + unsafe_put_user(regs->sp, &gregs[28], Efault); + } + return 0; + +Efault: + return -EFAULT; } static inline void __user *get_sigframe(struct ksignal *ksig, @@ -186,13 +191,17 @@ static int setup_rt_frame(struct ksignal *ksig, sigset_t *set, int err = 0; frame = get_sigframe(ksig, regs, sizeof(*frame)); + if (!access_ok(frame, sizeof(*frame))) + goto give_sigsegv; if (ksig->ka.sa.sa_flags & SA_SIGINFO) err |= copy_siginfo_to_user(&frame->info, &ksig->info); /* Create the ucontext. */ - err |= __put_user(0, &frame->uc.uc_flags); - err |= __put_user(0, &frame->uc.uc_link); + scoped_user_write_access(frame, give_sigsegv) { + unsafe_put_user(0, &frame->uc.uc_flags, give_sigsegv); + unsafe_put_user(0, &frame->uc.uc_link, give_sigsegv); + } err |= __save_altstack(&frame->uc.uc_stack, regs->sp); err |= rt_setup_ucontext(&frame->uc, regs); err |= copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); diff --git a/arch/openrisc/kernel/traps.c b/arch/openrisc/kernel/traps.c index c195be9cc9fc..2ad5e141981d 100644 --- a/arch/openrisc/kernel/traps.c +++ b/arch/openrisc/kernel/traps.c @@ -124,7 +124,7 @@ void show_registers(struct pt_regs *regs) for (i = -8; i < 24; i += 1) { unsigned long word; - if (__get_user(word, &((unsigned long *)esp)[i])) { + if (get_kernel_nofault(word, &((unsigned long *)esp)[i])) { bad_stack: pr_info(" Bad Stack value."); break; @@ -140,7 +140,7 @@ bad_stack: for (i = -6; i < 6; i += 1) { unsigned long word; - if (__get_user(word, &((unsigned long *)regs->pc)[i])) { + if (get_kernel_nofault(word, &((unsigned long *)regs->pc)[i])) { bad: pr_info(" Bad PC value."); break; diff --git a/arch/powerpc/kernel/align.c b/arch/powerpc/kernel/align.c index 61409431138f..c66642cb211c 100644 --- a/arch/powerpc/kernel/align.c +++ b/arch/powerpc/kernel/align.c @@ -298,7 +298,7 @@ int fix_alignment(struct pt_regs *regs) if (is_kernel_addr(regs->nip)) r = copy_inst_from_kernel_nofault(&instr, (void *)regs->nip); else - r = __get_user_instr(instr, (void __user *)regs->nip); + r = get_user_instr(instr, (void __user *)regs->nip); if (unlikely(r)) return -EFAULT; diff --git a/arch/powerpc/kernel/signal.h b/arch/powerpc/kernel/signal.h index 58ecea1cdc27..9e09b55177a6 100644 --- a/arch/powerpc/kernel/signal.h +++ b/arch/powerpc/kernel/signal.h @@ -16,11 +16,11 @@ extern int handle_signal32(struct ksignal *ksig, sigset_t *oldset, extern int handle_rt_signal32(struct ksignal *ksig, sigset_t *oldset, struct task_struct *tsk); -static inline int __get_user_sigset(sigset_t *dst, const sigset_t __user *src) +static inline int get_user_sigset(sigset_t *dst, const sigset_t __user *src) { BUILD_BUG_ON(sizeof(sigset_t) != sizeof(u64)); - return __get_user(dst->sig[0], (u64 __user *)&src->sig[0]); + return get_user(dst->sig[0], (u64 __user *)&src->sig[0]); } #define unsafe_get_user_sigset(dst, src, label) do { \ sigset_t *__dst = dst; \ diff --git a/arch/powerpc/kernel/signal_32.c b/arch/powerpc/kernel/signal_32.c index 7a718ed32b27..1c9fa00dc80d 100644 --- a/arch/powerpc/kernel/signal_32.c +++ b/arch/powerpc/kernel/signal_32.c @@ -968,7 +968,7 @@ static int do_setcontext_tm(struct ucontext __user *ucp, user_read_access_end(); - if (__get_user(tm_cmcp, &tm_ucp->uc_regs)) + if (get_user(tm_cmcp, &tm_ucp->uc_regs)) return -EFAULT; mcp = (struct mcontext __user *)(u64)cmcp; tm_mcp = (struct mcontext __user *)(u64)tm_cmcp; @@ -1009,10 +1009,10 @@ SYSCALL_DEFINE3(swapcontext, struct ucontext __user *, old_ctx, * access_ok since we are dealing with compat * pointers. */ - if (__get_user(cmcp, &new_ctx->uc_regs)) + if (get_user(cmcp, &new_ctx->uc_regs)) return -EFAULT; mcp = (struct mcontext __user *)(u64)cmcp; - if (__get_user(new_msr, &mcp->mc_gregs[PT_MSR])) + if (get_user(new_msr, &mcp->mc_gregs[PT_MSR])) return -EFAULT; } /* @@ -1124,19 +1124,19 @@ SYSCALL_DEFINE0(rt_sigreturn) if (MSR_TM_SUSPENDED(mfmsr())) tm_reclaim_current(0); - if (__get_user(tmp, &rt_sf->uc.uc_link)) + if (get_user(tmp, &rt_sf->uc.uc_link)) goto bad; uc_transact = (struct ucontext __user *)(uintptr_t)tmp; if (uc_transact) { u32 cmcp; struct mcontext __user *mcp; - if (__get_user(cmcp, &uc_transact->uc_regs)) + if (get_user(cmcp, &uc_transact->uc_regs)) return -EFAULT; mcp = (struct mcontext __user *)(u64)cmcp; /* The top 32 bits of the MSR are stashed in the transactional * ucontext. */ - if (__get_user(msr_hi, &mcp->mc_gregs[PT_MSR])) + if (get_user(msr_hi, &mcp->mc_gregs[PT_MSR])) goto bad; if (MSR_TM_ACTIVE(msr_hi<<32)) { @@ -1330,7 +1330,7 @@ SYSCALL_DEFINE0(sigreturn) mcp = (struct mcontext __user *)&sf->mctx; #ifdef CONFIG_PPC_TRANSACTIONAL_MEM tm_mcp = (struct mcontext __user *)&sf->mctx_transact; - if (__get_user(msr_hi, &tm_mcp->mc_gregs[PT_MSR])) + if (get_user(msr_hi, &tm_mcp->mc_gregs[PT_MSR])) goto badframe; #endif if (MSR_TM_ACTIVE(msr_hi<<32)) { diff --git a/arch/powerpc/kernel/signal_64.c b/arch/powerpc/kernel/signal_64.c index 86bb5bb4c143..b2003a10b369 100644 --- a/arch/powerpc/kernel/signal_64.c +++ b/arch/powerpc/kernel/signal_64.c @@ -624,24 +624,27 @@ static long restore_tm_sigcontexts(struct task_struct *tsk, struct sigcontext __ static long setup_trampoline(unsigned int syscall, unsigned int __user *tramp) { int i; - long err = 0; - /* Call the handler and pop the dummy stackframe*/ - err |= __put_user(PPC_RAW_BCTRL(), &tramp[0]); - err |= __put_user(PPC_RAW_ADDI(_R1, _R1, __SIGNAL_FRAMESIZE), &tramp[1]); + scoped_user_write_access_size(tramp, TRAMP_SIZE * sizeof(*tramp), efault) { + /* Call the handler and pop the dummy stackframe*/ + unsafe_put_user(PPC_RAW_BCTRL(), &tramp[0], efault); + unsafe_put_user(PPC_RAW_ADDI(_R1, _R1, __SIGNAL_FRAMESIZE), &tramp[1], efault); - err |= __put_user(PPC_RAW_LI(_R0, syscall), &tramp[2]); - err |= __put_user(PPC_RAW_SC(), &tramp[3]); + unsafe_put_user(PPC_RAW_LI(_R0, syscall), &tramp[2], efault); + unsafe_put_user(PPC_RAW_SC(), &tramp[3], efault); - /* Minimal traceback info */ - for (i=TRAMP_TRACEBACK; i < TRAMP_SIZE ;i++) - err |= __put_user(0, &tramp[i]); + /* Minimal traceback info */ + for (i=TRAMP_TRACEBACK; i < TRAMP_SIZE ;i++) + unsafe_put_user(0, &tramp[i], efault); + } - if (!err) - flush_icache_range((unsigned long) &tramp[0], + flush_icache_range((unsigned long) &tramp[0], (unsigned long) &tramp[TRAMP_SIZE]); - return err; + return 0; + +efault: + return -EFAULT; } /* @@ -711,7 +714,7 @@ SYSCALL_DEFINE3(swapcontext, struct ucontext __user *, old_ctx, * We kill the task with a SIGSEGV in this situation. */ - if (__get_user_sigset(&set, &new_ctx->uc_sigmask)) { + if (get_user_sigset(&set, &new_ctx->uc_sigmask)) { force_exit_sig(SIGSEGV); return -EFAULT; } @@ -754,7 +757,7 @@ SYSCALL_DEFINE0(rt_sigreturn) if (!access_ok(uc, sizeof(*uc))) goto badframe; - if (__get_user_sigset(&set, &uc->uc_sigmask)) + if (get_user_sigset(&set, &uc->uc_sigmask)) goto badframe; set_current_blocked(&set); @@ -797,7 +800,7 @@ SYSCALL_DEFINE0(rt_sigreturn) */ regs_set_return_msr(regs, regs->msr & ~MSR_TS_MASK); - if (__get_user(msr, &uc->uc_mcontext.gp_regs[PT_MSR])) + if (get_user(msr, &uc->uc_mcontext.gp_regs[PT_MSR])) goto badframe; } @@ -809,7 +812,7 @@ SYSCALL_DEFINE0(rt_sigreturn) if (!cpu_has_feature(CPU_FTR_TM)) goto badframe; - if (__get_user(uc_transact, &uc->uc_link)) + if (get_user(uc_transact, &uc->uc_link)) goto badframe; if (restore_tm_sigcontexts(current, &uc->uc_mcontext, &uc_transact->uc_mcontext)) diff --git a/arch/powerpc/kvm/book3s_64_mmu_hv.c b/arch/powerpc/kvm/book3s_64_mmu_hv.c index 2ccb3d138f46..00b8a3f855f4 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_hv.c +++ b/arch/powerpc/kvm/book3s_64_mmu_hv.c @@ -1749,8 +1749,8 @@ static ssize_t kvm_htab_read(struct file *file, char __user *buf, record_hpte(flags, hptp, hpte, revp, 1, first_pass)) { /* valid entry, write it out */ ++hdr.n_valid; - if (__put_user(hpte[0], lbuf) || - __put_user(hpte[1], lbuf + 1)) + if (put_user(hpte[0], lbuf) || + put_user(hpte[1], lbuf + 1)) return -EFAULT; nb += HPTE_SIZE; lbuf += 2; @@ -1853,8 +1853,8 @@ static ssize_t kvm_htab_write(struct file *file, const char __user *buf, __be64 hpte_r; err = -EFAULT; - if (__get_user(hpte_v, lbuf) || - __get_user(hpte_r, lbuf + 1)) + if (get_user(hpte_v, lbuf) || + get_user(hpte_r, lbuf + 1)) goto out; v = be64_to_cpu(hpte_v); r = be64_to_cpu(hpte_r); diff --git a/arch/powerpc/platforms/powernv/opal-lpc.c b/arch/powerpc/platforms/powernv/opal-lpc.c index 2bb326b58ff8..2e6cc2c89c02 100644 --- a/arch/powerpc/platforms/powernv/opal-lpc.c +++ b/arch/powerpc/platforms/powernv/opal-lpc.c @@ -247,19 +247,19 @@ static ssize_t lpc_debug_read(struct file *filp, char __user *ubuf, */ switch(len) { case 4: - rc = __put_user((u32)data, (u32 __user *)ubuf); + rc = put_user((u32)data, (u32 __user *)ubuf); break; case 2: #ifdef __LITTLE_ENDIAN__ data >>= 16; #endif - rc = __put_user((u16)data, (u16 __user *)ubuf); + rc = put_user((u16)data, (u16 __user *)ubuf); break; default: #ifdef __LITTLE_ENDIAN__ data >>= 24; #endif - rc = __put_user((u8)data, (u8 __user *)ubuf); + rc = put_user((u8)data, (u8 __user *)ubuf); break; } if (rc) @@ -317,15 +317,15 @@ static ssize_t lpc_debug_write(struct file *filp, const char __user *ubuf, */ switch(len) { case 4: - rc = __get_user(data, (u32 __user *)ubuf); + rc = get_user(data, (u32 __user *)ubuf); data = cpu_to_be32(data); break; case 2: - rc = __get_user(data, (u16 __user *)ubuf); + rc = get_user(data, (u16 __user *)ubuf); data = cpu_to_be16(data); break; default: - rc = __get_user(data, (u8 __user *)ubuf); + rc = get_user(data, (u8 __user *)ubuf); break; } if (rc) diff --git a/arch/s390/kernel/ptrace.c b/arch/s390/kernel/ptrace.c index d8d5e843859c..5202f9a2d939 100644 --- a/arch/s390/kernel/ptrace.c +++ b/arch/s390/kernel/ptrace.c @@ -539,7 +539,7 @@ static int s390_regs_set(struct task_struct *target, const unsigned long __user *u = ubuf; while (count > 0 && !rc) { unsigned long word; - rc = __get_user(word, u++); + rc = get_user(word, u++); if (rc) break; rc = __poke_user(target, pos, word); diff --git a/arch/s390/kernel/signal.c b/arch/s390/kernel/signal.c index deeef053758c..70094d533866 100644 --- a/arch/s390/kernel/signal.c +++ b/arch/s390/kernel/signal.c @@ -281,7 +281,7 @@ static int setup_frame(int sig, struct k_sigaction *ka, sigset_t *set, struct pt if (frame == (void __user *)-1UL) return -EFAULT; /* Set up backchain. */ - if (__put_user(regs->gprs[15], (addr_t __user *)frame)) + if (put_user(regs->gprs[15], (addr_t __user *)frame)) return -EFAULT; /* Create struct sigcontext on the signal stack. */ memcpy(&sc.oldmask, &set->sig, _SIGMASK_COPY_SIZE); @@ -294,7 +294,7 @@ static int setup_frame(int sig, struct k_sigaction *ka, sigset_t *set, struct pt if (save_sigregs(regs, &frame->sregs)) return -EFAULT; /* Place signal number on stack to allow backtrace from handler. */ - if (__put_user(regs->gprs[2], (int __user *)&frame->signo)) + if (put_user(regs->gprs[2], (int __user *)&frame->signo)) return -EFAULT; /* Create _sigregs_ext on the signal stack. */ if (save_sigregs_ext(regs, &frame->sregs_ext)) @@ -342,7 +342,7 @@ static int setup_rt_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *r if (frame == (void __user *)-1UL) return -EFAULT; /* Set up backchain. */ - if (__put_user(regs->gprs[15], (addr_t __user *)frame)) + if (put_user(regs->gprs[15], (addr_t __user *)frame)) return -EFAULT; /* * Set up to return from userspace. If provided, use a stub @@ -358,8 +358,8 @@ static int setup_rt_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *r /* Store registers needed to create the signal frame. */ store_sigregs(); /* Create ucontext on the signal stack. */ - if (__put_user(uc_flags, &frame->uc.uc_flags) || - __put_user(NULL, &frame->uc.uc_link) || + if (put_user(uc_flags, &frame->uc.uc_flags) || + put_user(NULL, &frame->uc.uc_link) || __save_altstack(&frame->uc.uc_stack, regs->gprs[15]) || save_sigregs(regs, &frame->uc.uc_mcontext) || __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)) || diff --git a/arch/s390/kernel/stacktrace.c b/arch/s390/kernel/stacktrace.c index 18520d333058..191f8a93371e 100644 --- a/arch/s390/kernel/stacktrace.c +++ b/arch/s390/kernel/stacktrace.c @@ -113,7 +113,7 @@ void arch_stack_walk_user_common(stack_trace_consume_fn consume_entry, void *coo sf = (void __user *)user_stack_pointer(regs); pagefault_disable(); while (1) { - if (__get_user(sp, &sf->back_chain)) + if (get_user(sp, &sf->back_chain)) break; /* * VDSO entry code has a non-standard stack frame layout. @@ -121,15 +121,15 @@ void arch_stack_walk_user_common(stack_trace_consume_fn consume_entry, void *coo */ if (!sp && ip_within_vdso(ip)) { sf_vdso = (void __user *)sf; - if (__get_user(ip, &sf_vdso->return_address)) + if (get_user(ip, &sf_vdso->return_address)) break; sp = (unsigned long)sf + STACK_FRAME_VDSO_OVERHEAD; sf = (void __user *)sp; - if (__get_user(sp, &sf->back_chain)) + if (get_user(sp, &sf->back_chain)) break; } else { sf = (void __user *)sp; - if (__get_user(ip, &sf->gprs[8])) + if (get_user(ip, &sf->gprs[8])) break; } /* Validate SP and RA (ABI requires SP to be 8 byte aligned). */ diff --git a/arch/x86/coco/sev/vc-handle.c b/arch/x86/coco/sev/vc-handle.c index 96b62b49b2b5..f25d302efceb 100644 --- a/arch/x86/coco/sev/vc-handle.c +++ b/arch/x86/coco/sev/vc-handle.c @@ -194,75 +194,113 @@ static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt) return __vc_decode_kern_insn(ctxt); } -static enum es_result vc_write_mem(struct es_em_ctxt *ctxt, - char *dst, char *buf, size_t size) +/* + * The memory operand of an emulated string instruction can be a user or a + * kernel address. MOVS to and from MMIO is only emulated for the kernel, + * but the kernel may be copying to or from user memory at the time, and + * INS/OUTS are emulated for both. So these are "user or kernel" accesses + * with fault handling: a single plain MOV of the given size, with an + * exception table entry to catch a fault. User access is enabled by hand + * so that a user address can be reached, without the range check of + * user_access_begin(), which would reject a kernel address. + * + * It must be a single MOV and not a string instruction, because the address + * can itself be MMIO. That takes a nested #VC, and a nested MOVS would nest + * forever. + * + * Page faults are disabled explicitly. A #VC from kernel mode runs as an NMI + * and is atomic anyway, but one from user mode only has interrupts disabled, + * which the page fault handler does not treat as atomic. A fault here is + * not handled: it is reported back as a #PF for the instruction that took the + * #VC, and gets handled there. + * + * The caller has already checked that size is 1, 2, 4 or 8. + */ +static bool vc_get_mem(void *buf, const void *addr, size_t size) { - unsigned long error_code = X86_PF_PROT | X86_PF_WRITE; + union { u8 d1; u16 d2; u32 d4; u64 d8; } val; - /* - * This function uses __put_user() independent of whether kernel or user - * memory is accessed. This works fine because __put_user() does no - * sanity checks of the pointer being accessed. All that it does is - * to report when the access failed. - * - * Also, this function runs in atomic context, so __put_user() is not - * allowed to sleep. The page-fault handler detects that it is running - * in atomic context and will not try to take mmap_sem and handle the - * fault, so additional pagefault_enable()/disable() calls are not - * needed. - * - * The access can't be done via copy_to_user() here because - * vc_write_mem() must not use string instructions to access unsafe - * memory. The reason is that MOVS is emulated by the #VC handler by - * splitting the move up into a read and a write and taking a nested #VC - * exception on whatever of them is the MMIO access. Using string - * instructions here would cause infinite nesting. - */ + guard(pagefault)(); + __uaccess_begin_nospec(); switch (size) { - case 1: { - u8 d1; - u8 __user *target = (u8 __user *)dst; - - memcpy(&d1, buf, 1); - if (__put_user(d1, target)) - goto fault; + case 1: + unsafe_get_user(val.d1, (const u8 __user __force *)addr, efault); break; - } - case 2: { - u16 d2; - u16 __user *target = (u16 __user *)dst; - - memcpy(&d2, buf, 2); - if (__put_user(d2, target)) - goto fault; + case 2: + unsafe_get_user(val.d2, (const u16 __user __force *)addr, efault); + break; + case 4: + unsafe_get_user(val.d4, (const u32 __user __force *)addr, efault); + break; + case 8: + unsafe_get_user(val.d8, (const u64 __user __force *)addr, efault); break; } - case 4: { - u32 d4; - u32 __user *target = (u32 __user *)dst; + __uaccess_end(); - memcpy(&d4, buf, 4); - if (__put_user(d4, target)) - goto fault; + memcpy(buf, &val, size); + return true; + +efault: + __uaccess_end(); + return false; +} + +static bool vc_put_mem(void *addr, const void *buf, size_t size) +{ + union { u8 d1; u16 d2; u32 d4; u64 d8; } val; + + memcpy(&val, buf, size); + + guard(pagefault)(); + __uaccess_begin(); + switch (size) { + case 1: + unsafe_put_user(val.d1, (u8 __user __force *)addr, efault); + break; + case 2: + unsafe_put_user(val.d2, (u16 __user __force *)addr, efault); + break; + case 4: + unsafe_put_user(val.d4, (u32 __user __force *)addr, efault); + break; + case 8: + unsafe_put_user(val.d8, (u64 __user __force *)addr, efault); break; } - case 8: { - u64 d8; - u64 __user *target = (u64 __user *)dst; + __uaccess_end(); + return true; - memcpy(&d8, buf, 8); - if (__put_user(d8, target)) - goto fault; - break; +efault: + __uaccess_end(); + return false; +} + +static bool vc_mem_size_ok(size_t size) +{ + switch (size) { + case 1: + case 2: + case 4: + case 8: + return true; } - default: + return false; +} + +static enum es_result vc_write_mem(struct es_em_ctxt *ctxt, + char *dst, char *buf, size_t size) +{ + unsigned long error_code = X86_PF_PROT | X86_PF_WRITE; + + if (!vc_mem_size_ok(size)) { WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size); return ES_UNSUPPORTED; } - return ES_OK; + if (vc_put_mem(dst, buf, size)) + return ES_OK; -fault: if (user_mode(ctxt->regs)) error_code |= X86_PF_USER; @@ -278,69 +316,14 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt, { unsigned long error_code = X86_PF_PROT; - /* - * This function uses __get_user() independent of whether kernel or user - * memory is accessed. This works fine because __get_user() does no - * sanity checks of the pointer being accessed. All that it does is - * to report when the access failed. - * - * Also, this function runs in atomic context, so __get_user() is not - * allowed to sleep. The page-fault handler detects that it is running - * in atomic context and will not try to take mmap_sem and handle the - * fault, so additional pagefault_enable()/disable() calls are not - * needed. - * - * The access can't be done via copy_from_user() here because - * vc_read_mem() must not use string instructions to access unsafe - * memory. The reason is that MOVS is emulated by the #VC handler by - * splitting the move up into a read and a write and taking a nested #VC - * exception on whatever of them is the MMIO access. Using string - * instructions here would cause infinite nesting. - */ - switch (size) { - case 1: { - u8 d1; - u8 __user *s = (u8 __user *)src; - - if (__get_user(d1, s)) - goto fault; - memcpy(buf, &d1, 1); - break; - } - case 2: { - u16 d2; - u16 __user *s = (u16 __user *)src; - - if (__get_user(d2, s)) - goto fault; - memcpy(buf, &d2, 2); - break; - } - case 4: { - u32 d4; - u32 __user *s = (u32 __user *)src; - - if (__get_user(d4, s)) - goto fault; - memcpy(buf, &d4, 4); - break; - } - case 8: { - u64 d8; - u64 __user *s = (u64 __user *)src; - if (__get_user(d8, s)) - goto fault; - memcpy(buf, &d8, 8); - break; - } - default: + if (!vc_mem_size_ok(size)) { WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size); return ES_UNSUPPORTED; } - return ES_OK; + if (vc_get_mem(buf, src, size)) + return ES_OK; -fault: if (user_mode(ctxt->regs)) error_code |= X86_PF_USER; diff --git a/arch/x86/entry/syscall_32.c b/arch/x86/entry/syscall_32.c index 91123a90710c..a59509bb460a 100644 --- a/arch/x86/entry/syscall_32.c +++ b/arch/x86/entry/syscall_32.c @@ -265,17 +265,8 @@ static noinstr bool __do_fast_syscall_32(struct pt_regs *regs) instrumentation_begin(); local_irq_enable(); /* Fetch EBP from where the vDSO stashed it. */ - if (IS_ENABLED(CONFIG_X86_64)) { - /* - * Micro-optimization: the pointer we're following is - * explicitly 32 bits, so it can't be out of range. - */ - res = __get_user(*(u32 *)®s->bp, - (u32 __user __force *)(unsigned long)(u32)regs->sp); - } else { - res = get_user(*(u32 *)®s->bp, + res = get_user(*(u32 *)®s->bp, (u32 __user __force *)(unsigned long)(u32)regs->sp); - } if (res) { /* User code screwed up. */ diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index 4df792b139e0..4e344fb1ae3e 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -3354,12 +3354,6 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re } } -static inline int -valid_user_frame(const void __user *fp, unsigned long size) -{ - return __access_ok(fp, size); -} - static unsigned long get_segment_base(unsigned int segment) { struct desc_struct *desc; @@ -3423,17 +3417,15 @@ perf_callchain_user32(struct pt_regs *regs, struct perf_callchain_entry_ctx *ent perf_callchain_store(entry, ret_addr); while (entry->nr < entry->max_stack) { - if (!valid_user_frame(fp, sizeof(frame))) - break; - - if (__get_user(frame.next_frame, &fp->next_frame)) - break; - if (__get_user(frame.return_address, &fp->return_address)) - break; + scoped_user_read_access(fp, out) { + unsafe_get_user(frame.next_frame, &fp->next_frame, out); + unsafe_get_user(frame.return_address, &fp->return_address, out); + } perf_callchain_store(entry, cs_base + frame.return_address); fp = compat_ptr(ss_base + frame.next_frame); } +out: pagefault_enable(); return 1; } @@ -3488,17 +3480,15 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs perf_callchain_store(entry, ret_addr); while (entry->nr < entry->max_stack) { - if (!valid_user_frame(fp, sizeof(frame))) - break; - - if (__get_user(frame.next_frame, &fp->next_frame)) - break; - if (__get_user(frame.return_address, &fp->return_address)) - break; + scoped_user_read_access(fp, out) { + unsafe_get_user(frame.next_frame, &fp->next_frame, out); + unsafe_get_user(frame.return_address, &fp->return_address, out); + } perf_callchain_store(entry, frame.return_address); fp = (void __user *)frame.next_frame; } +out: pagefault_enable(); } diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 3a0dd3c2b233..f4700ad6b5ec 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -28,10 +28,6 @@ extern int __get_user_1(void); extern int __get_user_2(void); extern int __get_user_4(void); extern int __get_user_8(void); -extern int __get_user_nocheck_1(void); -extern int __get_user_nocheck_2(void); -extern int __get_user_nocheck_4(void); -extern int __get_user_nocheck_8(void); extern int __get_user_bad(void); #define __uaccess_begin() stac() @@ -56,10 +52,9 @@ extern int __get_user_bad(void); __builtin_choose_expr(sizeof(x)<=sizeof(type),(unsigned type)0,not) /* - * This is used for both get_user() and __get_user() to expand to - * the proper special function call that has odd calling conventions - * due to returning both a value and an error, and that depends on - * the size of the pointer passed in. + * This is used by get_user() to expand to the proper special function + * call that has odd calling conventions due to returning both a value + * and an error, and that depends on the size of the pointer passed in. * * Careful: we have to cast the result to the type of the pointer * for sign reasons. @@ -107,29 +102,6 @@ extern int __get_user_bad(void); */ #define get_user(x,ptr) ({ might_fault(); do_get_user_call(get_user,x,ptr); }) -/** - * __get_user - Get a simple variable from user space, with less checking. - * @x: Variable to store result. - * @ptr: Source address, in user space. - * - * Context: User context only. This function may sleep if pagefaults are - * enabled. - * - * This macro copies a single simple variable from user space to kernel - * space. It supports simple types like char and int, but not larger - * data types like structures or arrays. - * - * @ptr must have pointer-to-simple-variable type, and the result of - * dereferencing @ptr must be assignable to @x without a cast. - * - * Caller must check the pointer with access_ok() before calling this - * function. - * - * Return: zero on success, or -EFAULT on error. - * On error, the variable @x is set to zero. - */ -#define __get_user(x,ptr) do_get_user_call(get_user_nocheck,x,ptr) - #ifdef CONFIG_X86_32 #define __put_user_goto_u64(x, addr, label) \ @@ -156,10 +128,6 @@ extern void __put_user_1(void); extern void __put_user_2(void); extern void __put_user_4(void); extern void __put_user_8(void); -extern void __put_user_nocheck_1(void); -extern void __put_user_nocheck_2(void); -extern void __put_user_nocheck_4(void); -extern void __put_user_nocheck_8(void); /* * ptr must be evaluated and assigned to the temporary __ptr_pu before @@ -207,28 +175,6 @@ extern void __put_user_nocheck_8(void); */ #define put_user(x, ptr) ({ might_fault(); do_put_user_call(put_user,x,ptr); }) -/** - * __put_user - Write a simple value into user space, with less checking. - * @x: Value to copy to user space. - * @ptr: Destination address, in user space. - * - * Context: User context only. This function may sleep if pagefaults are - * enabled. - * - * This macro copies a single simple value from kernel space to user - * space. It supports simple types like char and int, but not larger - * data types like structures or arrays. - * - * @ptr must have pointer-to-simple-variable type, and @x must be assignable - * to the result of dereferencing @ptr. - * - * Caller must check the pointer with access_ok() before calling this - * function. - * - * Return: zero on success, or -EFAULT on error. - */ -#define __put_user(x, ptr) do_put_user_call(put_user_nocheck,x,ptr) - #define __put_user_size(x, ptr, size, label) \ do { \ __typeof__(*(ptr)) __x = (x); /* eval x once */ \ diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 1f721ac84283..164255df3d2e 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -49,7 +49,7 @@ static inline bool check_xstate_in_sigframe(struct fxregs_state __user *buf_fx, * fpstate layout with out copying the extended state information * in the memory layout. */ - if (__get_user(magic2, (__u32 __user *)(buf + fx_sw->xstate_size))) + if (get_user(magic2, (__u32 __user *)(buf + fx_sw->xstate_size))) return false; if (unlikely(magic2 != FP_XSTATE_MAGIC2)) goto err_setfx; @@ -106,14 +106,14 @@ static inline bool save_fsave_header(struct task_struct *tsk, void __user *buf) convert_from_fxsr(&env, tsk); if (__copy_to_user(buf, &env, sizeof(env)) || - __put_user(xsave->i387.swd, &fp->status) || - __put_user(X86_FXSR_MAGIC, &fp->magic)) + put_user(xsave->i387.swd, &fp->status) || + put_user(X86_FXSR_MAGIC, &fp->magic)) return false; } else { struct fregs_state __user *fp = buf; u32 swd; - if (__get_user(swd, &fp->swd) || __put_user(swd, &fp->status)) + if (get_user(swd, &fp->swd) || put_user(swd, &fp->status)) return false; } @@ -153,8 +153,8 @@ static inline bool save_xstate_epilog(void __user *buf, int ia32_frame, if (!use_xsave()) return !err; - err |= __put_user(FP_XSTATE_MAGIC2, - (__u32 __user *)(buf + fpstate->user_size)); + err |= put_user(FP_XSTATE_MAGIC2, + (__u32 __user *)(buf + fpstate->user_size)); /* * For legacy compatible, we always set FP/SSE bits in the bit diff --git a/arch/x86/kernel/fpu/xstate.h b/arch/x86/kernel/fpu/xstate.h index 38a2862f09d3..1ea762e93086 100644 --- a/arch/x86/kernel/fpu/xstate.h +++ b/arch/x86/kernel/fpu/xstate.h @@ -76,9 +76,9 @@ static inline int set_xfeature_in_sigframe(struct xregs_state __user *xbuf, u64 int err; /* Read the xfeatures value already saved in the user buffer */ - err = __get_user(xfeatures, &xbuf->header.xfeatures); + err = get_user(xfeatures, &xbuf->header.xfeatures); xfeatures |= mask; - err |= __put_user(xfeatures, &xbuf->header.xfeatures); + err |= put_user(xfeatures, &xbuf->header.xfeatures); return err; } @@ -99,7 +99,7 @@ static inline int update_pkru_in_sigframe(struct xregs_state __user *buf, u32 pk return err; /* Update PKRU value in the userspace xsave buffer. */ - return __put_user(pkru, (unsigned int __user *)get_xsave_addr_user(buf, XFEATURE_PKRU)); + return put_user(pkru, (unsigned int __user *)get_xsave_addr_user(buf, XFEATURE_PKRU)); } /* XSAVE/XRSTOR wrapper functions */ diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c index 3dcadc13f09a..953d577ebadd 100644 --- a/arch/x86/kernel/ptrace.c +++ b/arch/x86/kernel/ptrace.c @@ -457,7 +457,7 @@ static int genregs_set(struct task_struct *target, const unsigned long __user *u = ubuf; while (count >= sizeof(*u) && !ret) { unsigned long word; - ret = __get_user(word, u++); + ret = get_user(word, u++); if (ret) break; ret = putreg(target, pos, word); @@ -1050,7 +1050,7 @@ static int genregs32_set(struct task_struct *target, const compat_ulong_t __user *u = ubuf; while (count >= sizeof(*u) && !ret) { compat_ulong_t word; - ret = __get_user(word, u++); + ret = get_user(word, u++); if (ret) break; ret = putreg32(target, pos, word); diff --git a/arch/x86/kernel/signal_32.c b/arch/x86/kernel/signal_32.c index 8e87ab586437..0e97ef61fe30 100644 --- a/arch/x86/kernel/signal_32.c +++ b/arch/x86/kernel/signal_32.c @@ -154,11 +154,10 @@ SYSCALL32_DEFINE0(sigreturn) prevent_single_step_upon_eretu(regs); - if (!access_ok(frame, sizeof(*frame))) - goto badframe; - if (__get_user(set.sig[0], &frame->sc.oldmask) - || __get_user(((__u32 *)&set)[1], &frame->extramask[0])) - goto badframe; + scoped_user_read_access(frame, badframe) { + unsafe_get_user(set.sig[0], &frame->sc.oldmask, badframe); + unsafe_get_user(((__u32 *)&set)[1], &frame->extramask[0], badframe); + } set_current_blocked(&set); @@ -181,10 +180,8 @@ SYSCALL32_DEFINE0(rt_sigreturn) frame = (struct rt_sigframe_ia32 __user *)(regs->sp - 4); - if (!access_ok(frame, sizeof(*frame))) - goto badframe; - if (__get_user(*(__u64 *)&set, (__u64 __user *)&frame->uc.uc_sigmask)) - goto badframe; + scoped_user_read_access(frame, badframe) + unsafe_get_user(*(__u64 *)&set, (__u64 __user *)&frame->uc.uc_sigmask, badframe); set_current_blocked(&set); diff --git a/arch/x86/kernel/signal_64.c b/arch/x86/kernel/signal_64.c index d483b585c6c6..62cc62d8e092 100644 --- a/arch/x86/kernel/signal_64.c +++ b/arch/x86/kernel/signal_64.c @@ -253,12 +253,10 @@ SYSCALL_DEFINE0(rt_sigreturn) prevent_single_step_upon_eretu(regs); frame = (struct rt_sigframe __user *)(regs->sp - sizeof(long)); - if (!access_ok(frame, sizeof(*frame))) - goto badframe; - if (__get_user(*(__u64 *)&set, (__u64 __user *)&frame->uc.uc_sigmask)) - goto badframe; - if (__get_user(uc_flags, &frame->uc.uc_flags)) - goto badframe; + scoped_user_read_access(frame, badframe) { + unsafe_get_user(*(__u64 *)&set, (__u64 __user *)&frame->uc.uc_sigmask, badframe); + unsafe_get_user(uc_flags, &frame->uc.uc_flags, badframe); + } set_current_blocked(&set); @@ -372,12 +370,10 @@ COMPAT_SYSCALL_DEFINE0(x32_rt_sigreturn) frame = (struct rt_sigframe_x32 __user *)(regs->sp - 8); - if (!access_ok(frame, sizeof(*frame))) - goto badframe; - if (__get_user(set.sig[0], (__u64 __user *)&frame->uc.uc_sigmask)) - goto badframe; - if (__get_user(uc_flags, &frame->uc.uc_flags)) - goto badframe; + scoped_user_read_access(frame, badframe) { + unsafe_get_user(set.sig[0], (__u64 __user *)&frame->uc.uc_sigmask, badframe); + unsafe_get_user(uc_flags, &frame->uc.uc_flags, badframe); + } set_current_blocked(&set); diff --git a/arch/x86/kernel/stacktrace.c b/arch/x86/kernel/stacktrace.c index ee117fcf46ed..9412ac4ddf46 100644 --- a/arch/x86/kernel/stacktrace.c +++ b/arch/x86/kernel/stacktrace.c @@ -88,16 +88,15 @@ static int copy_stack_frame(const struct stack_frame_user __user *fp, struct stack_frame_user *frame) { - int ret; + int ret = 0; - if (!__access_ok(fp, sizeof(*frame))) - return 0; - - ret = 1; pagefault_disable(); - if (__get_user(frame->next_fp, &fp->next_fp) || - __get_user(frame->ret_addr, &fp->ret_addr)) - ret = 0; + scoped_user_read_access(fp, out) { + unsafe_get_user(frame->next_fp, &fp->next_fp, out); + unsafe_get_user(frame->ret_addr, &fp->ret_addr, out); + } + ret = 1; +out: pagefault_enable(); return ret; diff --git a/arch/x86/lib/getuser.S b/arch/x86/lib/getuser.S index 9d5654b8a72a..7c79eca08b0a 100644 --- a/arch/x86/lib/getuser.S +++ b/arch/x86/lib/getuser.S @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ /* - * __get_user functions. + * The out-of-line helpers for get_user(). * * (C) Copyright 1998 Linus Torvalds * (C) Copyright 2005 Andi Kleen @@ -37,8 +37,6 @@ #include <asm/smap.h> #include <asm/runtime-const.h> -#define ASM_BARRIER_NOSPEC ALTERNATIVE "", "lfence", X86_FEATURE_LFENCE_RDTSC - .macro check_range size:req .if IS_ENABLED(CONFIG_X86_64) RUNTIME_CONST_PTR USER_PTR_MAX, rdx @@ -111,58 +109,6 @@ SYM_FUNC_START(__get_user_8) SYM_FUNC_END(__get_user_8) EXPORT_SYMBOL(__get_user_8) -/* .. and the same for __get_user, just without the range checks */ -SYM_FUNC_START(__get_user_nocheck_1) - ANNOTATE_NOENDBR - ASM_STAC - ASM_BARRIER_NOSPEC - UACCESS movzbl (%_ASM_AX),%edx - xor %eax,%eax - ASM_CLAC - RET -SYM_FUNC_END(__get_user_nocheck_1) -EXPORT_SYMBOL(__get_user_nocheck_1) - -SYM_FUNC_START(__get_user_nocheck_2) - ANNOTATE_NOENDBR - ASM_STAC - ASM_BARRIER_NOSPEC - UACCESS movzwl (%_ASM_AX),%edx - xor %eax,%eax - ASM_CLAC - RET -SYM_FUNC_END(__get_user_nocheck_2) -EXPORT_SYMBOL(__get_user_nocheck_2) - -SYM_FUNC_START(__get_user_nocheck_4) - ANNOTATE_NOENDBR - ASM_STAC - ASM_BARRIER_NOSPEC - UACCESS movl (%_ASM_AX),%edx - xor %eax,%eax - ASM_CLAC - RET -SYM_FUNC_END(__get_user_nocheck_4) -EXPORT_SYMBOL(__get_user_nocheck_4) - -SYM_FUNC_START(__get_user_nocheck_8) - ANNOTATE_NOENDBR - ASM_STAC - ASM_BARRIER_NOSPEC -#ifdef CONFIG_X86_64 - UACCESS movq (%_ASM_AX),%rdx -#else - xor %ecx,%ecx - UACCESS movl (%_ASM_AX),%edx - UACCESS movl 4(%_ASM_AX),%ecx -#endif - xor %eax,%eax - ASM_CLAC - RET -SYM_FUNC_END(__get_user_nocheck_8) -EXPORT_SYMBOL(__get_user_nocheck_8) - - SYM_CODE_START_LOCAL(__get_user_handle_exception) ASM_CLAC .Lbad_get_user: diff --git a/arch/x86/lib/putuser.S b/arch/x86/lib/putuser.S index 46d9e9b98a61..9ad19f907485 100644 --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ /* - * __put_user functions. + * The out-of-line helpers for put_user(). * * (C) Copyright 2005 Linus Torvalds * (C) Copyright 2005 Andi Kleen @@ -56,65 +56,35 @@ SYM_FUNC_START(__put_user_1) SYM_FUNC_END(__put_user_1) EXPORT_SYMBOL(__put_user_1) -SYM_FUNC_START(__put_user_nocheck_1) - ANNOTATE_NOENDBR - ASM_STAC -2: movb %al,(%_ASM_CX) - xor %ecx,%ecx - ASM_CLAC - RET -SYM_FUNC_END(__put_user_nocheck_1) -EXPORT_SYMBOL(__put_user_nocheck_1) - SYM_FUNC_START(__put_user_2) ANNOTATE_NOENDBR check_range size=2 ASM_STAC -3: movw %ax,(%_ASM_CX) +2: movw %ax,(%_ASM_CX) xor %ecx,%ecx ASM_CLAC RET SYM_FUNC_END(__put_user_2) EXPORT_SYMBOL(__put_user_2) -SYM_FUNC_START(__put_user_nocheck_2) - ANNOTATE_NOENDBR - ASM_STAC -4: movw %ax,(%_ASM_CX) - xor %ecx,%ecx - ASM_CLAC - RET -SYM_FUNC_END(__put_user_nocheck_2) -EXPORT_SYMBOL(__put_user_nocheck_2) - SYM_FUNC_START(__put_user_4) ANNOTATE_NOENDBR check_range size=4 ASM_STAC -5: movl %eax,(%_ASM_CX) +3: movl %eax,(%_ASM_CX) xor %ecx,%ecx ASM_CLAC RET SYM_FUNC_END(__put_user_4) EXPORT_SYMBOL(__put_user_4) -SYM_FUNC_START(__put_user_nocheck_4) - ANNOTATE_NOENDBR - ASM_STAC -6: movl %eax,(%_ASM_CX) - xor %ecx,%ecx - ASM_CLAC - RET -SYM_FUNC_END(__put_user_nocheck_4) -EXPORT_SYMBOL(__put_user_nocheck_4) - SYM_FUNC_START(__put_user_8) ANNOTATE_NOENDBR check_range size=8 ASM_STAC -7: mov %_ASM_AX,(%_ASM_CX) +4: mov %_ASM_AX,(%_ASM_CX) #ifdef CONFIG_X86_32 -8: movl %edx,4(%_ASM_CX) +5: movl %edx,4(%_ASM_CX) #endif xor %ecx,%ecx ASM_CLAC @@ -122,19 +92,6 @@ SYM_FUNC_START(__put_user_8) SYM_FUNC_END(__put_user_8) EXPORT_SYMBOL(__put_user_8) -SYM_FUNC_START(__put_user_nocheck_8) - ANNOTATE_NOENDBR - ASM_STAC -9: mov %_ASM_AX,(%_ASM_CX) -#ifdef CONFIG_X86_32 -10: movl %edx,4(%_ASM_CX) -#endif - xor %ecx,%ecx - ASM_CLAC - RET -SYM_FUNC_END(__put_user_nocheck_8) -EXPORT_SYMBOL(__put_user_nocheck_8) - SYM_CODE_START_LOCAL(__put_user_handle_exception) ASM_CLAC .Lbad_put_user: @@ -146,11 +103,6 @@ SYM_CODE_END(__put_user_handle_exception) _ASM_EXTABLE_UA(2b, __put_user_handle_exception) _ASM_EXTABLE_UA(3b, __put_user_handle_exception) _ASM_EXTABLE_UA(4b, __put_user_handle_exception) - _ASM_EXTABLE_UA(5b, __put_user_handle_exception) - _ASM_EXTABLE_UA(6b, __put_user_handle_exception) - _ASM_EXTABLE_UA(7b, __put_user_handle_exception) - _ASM_EXTABLE_UA(9b, __put_user_handle_exception) #ifdef CONFIG_X86_32 - _ASM_EXTABLE_UA(8b, __put_user_handle_exception) - _ASM_EXTABLE_UA(10b, __put_user_handle_exception) + _ASM_EXTABLE_UA(5b, __put_user_handle_exception) #endif diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 2c64b388f616..c5117a7730d6 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -475,9 +475,9 @@ static void set_aliased_prot(void *v, pgprot_t prot) * have the same nominal mm, but if we're a kernel thread, lazy * mm dropping could change our pgd. * - * Out of an abundance of caution, this uses __get_user() to fault - * in the target address just in case there's some obscure case - * in which the target address isn't readable. + * Out of an abundance of caution, this uses copy_from_kernel_nofault() + * to fault in the target address just in case there's some obscure + * case in which the target address isn't readable. */ preempt_disable(); diff --git a/drivers/char/mem.c b/drivers/char/mem.c index 5b93c92c2cf1..c4c78182b56e 100644 --- a/drivers/char/mem.c +++ b/drivers/char/mem.c @@ -372,7 +372,7 @@ static ssize_t read_port(struct file *file, char __user *buf, if (!access_ok(buf, count)) return -EFAULT; while (count-- > 0 && i < 65536) { - if (__put_user(inb(i), tmp) < 0) + if (put_user(inb(i), tmp) < 0) return -EFAULT; i++; tmp++; @@ -392,7 +392,7 @@ static ssize_t write_port(struct file *file, const char __user *buf, while (count-- > 0 && i < 65536) { char c; - if (__get_user(c, tmp)) { + if (get_user(c, tmp)) { if (tmp > buf) break; return -EFAULT; diff --git a/drivers/char/nwflash.c b/drivers/char/nwflash.c index 9f52f0306ef7..c3df2b032e29 100644 --- a/drivers/char/nwflash.c +++ b/drivers/char/nwflash.c @@ -422,7 +422,7 @@ static int write_block(unsigned long p, const char __user *buf, int count) for (offset = 0; offset < count; offset++, pWritePtr++) { uAddress = (unsigned int) pWritePtr; uAddress &= 0xFFFFFFFC; - if (__get_user(c2, buf + offset)) + if (get_user(c2, buf + offset)) return -EFAULT; WriteRetry: @@ -533,7 +533,7 @@ static int write_block(unsigned long p, const char __user *buf, int count) for (offset = 0; offset < count; offset++) { char c, c1; - if (__get_user(c, buf)) + if (get_user(c, buf)) return -EFAULT; buf++; if ((c1 = *pWritePtr++) != c) { diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c index d5aadced7da0..1b7f5bf13e50 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c @@ -1522,8 +1522,9 @@ static int eb_relocate_vma(struct i915_execbuffer *eb, struct eb_vma *ev) /* * We must check that the entire relocation array is safe - * to read. However, if the array is not writable the user loses - * the updated relocation values. + * to read. Whether it is also writable only shows when we + * write back the presumed offsets below, and if that fails we + * fall back to the slow path. */ if (unlikely(!access_ok(urelocs, remain * sizeof(*urelocs)))) return -EFAULT; @@ -1553,6 +1554,7 @@ static int eb_relocate_vma(struct i915_execbuffer *eb, struct eb_vma *ev) remain -= count; do { u64 offset = eb_relocate_entry(eb, ev, r); + int err; if (likely(offset == 0)) continue; @@ -1562,28 +1564,41 @@ static int eb_relocate_vma(struct i915_execbuffer *eb, struct eb_vma *ev) goto out; } /* - * Note that reporting an error now - * leaves everything in an inconsistent - * state as we have *already* changed - * the relocation value inside the - * object. As we have not changed the - * reloc.presumed_offset or will not - * change the execobject.offset, on the - * call we may not rewrite the value - * inside the object, leaving it - * dangling and causing a GPU hang. Unless - * userspace dynamically rebuilds the - * relocations on each execbuf rather than - * presume a static tree. + * Write back the presumed offset. We hold the + * objects' dma_resv locks here, so this must not + * take a page fault: that would take mmap_lock + * under dma_resv, the opposite of the order our + * own fault handler uses. * - * We did previously check if the relocations - * were writable (access_ok), an error now - * would be a strange race with mprotect, - * having already demonstrated that we - * can read from this userspace address. + * If the write fails (the page is read-only, + * copy-on-write, or gone since we read it), we + * must not just carry on. We have already changed + * the relocation value inside the object, and a + * stale presumed_offset could later match again + * and make us skip a relocation that is needed, + * hanging the GPU. So fall back to the slow path, + * as for a failed read: it retries after + * prefaulting and, failing that, marks every + * presumed_offset invalid with faults allowed and + * no locks held. */ offset = gen8_canonical_addr(offset & ~UPDATE); - __put_user(offset, &urelocs[r - stack].presumed_offset); + pagefault_disable(); + err = put_user(offset, &urelocs[r - stack].presumed_offset); + pagefault_enable(); + + /* + * Error handling is disabled on purpose. No, it makes + * no sense, and the code will loop over the relocations + * without writing them back and just waste CPU time, + * but that's apparently what the mprotect test-cases + * want. See commit 97a37c919f62 ("drm/i915/gem: Ignore + * readonly failures when updating relocs") + */ + if (unlikely(err) && 0) { + remain = -EFAULT; + goto out; + } } while (r++, --count); urelocs += ARRAY_SIZE(stack); } while (remain); @@ -1635,11 +1650,11 @@ static int check_relocations(const struct drm_i915_gem_exec_object2 *entry) end = addr + size; for (; addr < end; addr += PAGE_SIZE) { - int err = __get_user(c, addr); + int err = get_user(c, addr); if (err) return err; } - return __get_user(c, end - 1); + return get_user(c, end - 1); } static int eb_copy_relocations(const struct i915_execbuffer *eb) @@ -2837,7 +2852,7 @@ add_timeline_fence_array(struct i915_execbuffer *eb, if (user_fence.flags & __I915_EXEC_FENCE_UNKNOWN_FLAGS) return -EINVAL; - if (__get_user(point, user_values++)) + if (get_user(point, user_values++)) return -EFAULT; syncobj = drm_syncobj_find(eb->file, user_fence.handle); diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c index d01acfb7d93d..e6748856a0a8 100644 --- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c +++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c @@ -1465,7 +1465,7 @@ static int __igt_mmap_access(struct drm_i915_private *i915, return addr; ptr = (unsigned long __user *)addr; - err = __put_user(A, ptr); + err = put_user(A, ptr); if (err) { pr_err("%s(%s): failed to write into user mmap\n", obj->mm.region->name, repr_mmap_type(type)); @@ -1490,7 +1490,7 @@ static int __igt_mmap_access(struct drm_i915_private *i915, intel_gt_flush_ggtt_writes(to_gt(i915)); - err = __get_user(y, ptr); + err = get_user(y, ptr); if (err) { pr_err("%s(%s): failed to read from user mmap\n", obj->mm.region->name, repr_mmap_type(type)); @@ -1738,12 +1738,12 @@ static int prefault_range(u64 start, u64 len) end = addr + len; for (; addr < end; addr += PAGE_SIZE) { - err = __get_user(c, addr); + err = get_user(c, addr); if (err) return err; } - return __get_user(c, end - 1); + return get_user(c, end - 1); } static int __igt_mmap_revoke(struct drm_i915_private *i915, diff --git a/drivers/macintosh/ans-lcd.c b/drivers/macintosh/ans-lcd.c index fa904b24a600..673ff672af8a 100644 --- a/drivers/macintosh/ans-lcd.c +++ b/drivers/macintosh/ans-lcd.c @@ -71,7 +71,7 @@ anslcd_write( struct file * file, const char __user * buf, for ( i = *ppos; count > 0; ++i, ++p, --count ) { char c; - __get_user(c, p); + get_user(c, p); anslcd_write_byte_data( c ); } mutex_unlock(&anslcd_mutex); @@ -102,10 +102,10 @@ anslcd_ioctl(struct file *file, unsigned int cmd, unsigned long arg) break; case ANSLCD_SENDCTRL: temp = (char __user *) arg; - __get_user(ch, temp); + get_user(ch, temp); for (; ch; temp++) { /* FIXME: This is ugly, but should work, as a \0 byte is not a valid command code */ anslcd_write_byte_ctrl ( ch ); - __get_user(ch, temp); + get_user(ch, temp); } break; case ANSLCD_SETSHORTDELAY: diff --git a/drivers/mtd/ubi/cdev.c b/drivers/mtd/ubi/cdev.c index 54acd8394d67..8dd005ef376b 100644 --- a/drivers/mtd/ubi/cdev.c +++ b/drivers/mtd/ubi/cdev.c @@ -861,7 +861,7 @@ static int ubi_get_ec_info(struct ubi_device *ubi, struct ubi_ecinfo_req __user int ec; if (ubi_io_is_bad(ubi, peb)) { - if (__put_user(UBI_UNKNOWN, ureq->erase_counters+read_cnt)) + if (put_user(UBI_UNKNOWN, ureq->erase_counters+read_cnt)) return -EFAULT; continue; @@ -877,7 +877,7 @@ static int ubi_get_ec_info(struct ubi_device *ubi, struct ubi_ecinfo_req __user spin_unlock(&ubi->wl_lock); - if (__put_user(ec, ureq->erase_counters+read_cnt)) + if (put_user(ec, ureq->erase_counters+read_cnt)) return -EFAULT; } diff --git a/drivers/pci/proc.c b/drivers/pci/proc.c index f5fd860a0540..3d7a9ed5fc22 100644 --- a/drivers/pci/proc.c +++ b/drivers/pci/proc.c @@ -67,7 +67,7 @@ static ssize_t proc_bus_pci_read(struct file *file, char __user *buf, if ((pos & 1) && cnt) { unsigned char val; pci_user_read_config_byte(dev, pos, &val); - __put_user(val, buf); + put_user(val, buf); buf++; pos++; cnt--; @@ -76,7 +76,7 @@ static ssize_t proc_bus_pci_read(struct file *file, char __user *buf, if ((pos & 3) && cnt > 2) { unsigned short val; pci_user_read_config_word(dev, pos, &val); - __put_user(cpu_to_le16(val), (__le16 __user *) buf); + put_user(cpu_to_le16(val), (__le16 __user *) buf); buf += 2; pos += 2; cnt -= 2; @@ -85,7 +85,7 @@ static ssize_t proc_bus_pci_read(struct file *file, char __user *buf, while (cnt >= 4) { unsigned int val; pci_user_read_config_dword(dev, pos, &val); - __put_user(cpu_to_le32(val), (__le32 __user *) buf); + put_user(cpu_to_le32(val), (__le32 __user *) buf); buf += 4; pos += 4; cnt -= 4; @@ -95,7 +95,7 @@ static ssize_t proc_bus_pci_read(struct file *file, char __user *buf, if (cnt >= 2) { unsigned short val; pci_user_read_config_word(dev, pos, &val); - __put_user(cpu_to_le16(val), (__le16 __user *) buf); + put_user(cpu_to_le16(val), (__le16 __user *) buf); buf += 2; pos += 2; cnt -= 2; @@ -104,7 +104,7 @@ static ssize_t proc_bus_pci_read(struct file *file, char __user *buf, if (cnt) { unsigned char val; pci_user_read_config_byte(dev, pos, &val); - __put_user(val, buf); + put_user(val, buf); pos++; } @@ -151,7 +151,7 @@ static ssize_t proc_bus_pci_write(struct file *file, const char __user *buf, if ((pos & 1) && cnt) { unsigned char val; - __get_user(val, buf); + get_user(val, buf); pci_user_write_config_byte(dev, pos, val); buf++; pos++; @@ -160,7 +160,7 @@ static ssize_t proc_bus_pci_write(struct file *file, const char __user *buf, if ((pos & 3) && cnt > 2) { __le16 val; - __get_user(val, (__le16 __user *) buf); + get_user(val, (__le16 __user *) buf); pci_user_write_config_word(dev, pos, le16_to_cpu(val)); buf += 2; pos += 2; @@ -169,7 +169,7 @@ static ssize_t proc_bus_pci_write(struct file *file, const char __user *buf, while (cnt >= 4) { __le32 val; - __get_user(val, (__le32 __user *) buf); + get_user(val, (__le32 __user *) buf); pci_user_write_config_dword(dev, pos, le32_to_cpu(val)); buf += 4; pos += 4; @@ -178,7 +178,7 @@ static ssize_t proc_bus_pci_write(struct file *file, const char __user *buf, if (cnt >= 2) { __le16 val; - __get_user(val, (__le16 __user *) buf); + get_user(val, (__le16 __user *) buf); pci_user_write_config_word(dev, pos, le16_to_cpu(val)); buf += 2; pos += 2; @@ -187,7 +187,7 @@ static ssize_t proc_bus_pci_write(struct file *file, const char __user *buf, if (cnt) { unsigned char val; - __get_user(val, buf); + get_user(val, buf); pci_user_write_config_byte(dev, pos, val); pos++; } diff --git a/drivers/pnp/isapnp/proc.c b/drivers/pnp/isapnp/proc.c index 55ae72a2818b..c1f6e6bd0eb2 100644 --- a/drivers/pnp/isapnp/proc.c +++ b/drivers/pnp/isapnp/proc.c @@ -41,7 +41,7 @@ static ssize_t isapnp_proc_bus_read(struct file *file, char __user * buf, for (; pos < 256 && cnt > 0; pos++, buf++, cnt--) { unsigned char val; val = isapnp_read_byte(pos); - __put_user(val, buf); + put_user(val, buf); } isapnp_cfg_end(); diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c index 76dd5b0828d7..4d3e36fb59d5 100644 --- a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +++ b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c @@ -11,6 +11,7 @@ #include <linux/module.h> #include <linux/device.h> #include <linux/kernel.h> +#include <linux/uaccess.h> #include <linux/blkdev.h> #include <linux/vringh.h> #include <linux/vdpa.h> @@ -343,11 +344,17 @@ static void vdpasim_blk_work(struct vdpasim *vdpasim) continue; while (vdpasim_blk_handle_req(vdpasim, vq)) { + bool notify; + /* Make sure used is visible before rasing the interrupt. */ smp_wmb(); local_bh_disable(); - if (vringh_need_notify_iotlb(&vq->vring) > 0) + /* The ring may be in user memory, and we can't take faults here */ + pagefault_disable(); + notify = vringh_need_notify_iotlb(&vq->vring) > 0; + pagefault_enable(); + if (notify) vringh_notify(&vq->vring); local_bh_enable(); diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c index a6514b5ccd86..543a58abcdb8 100644 --- a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c +++ b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c @@ -11,6 +11,7 @@ #include <linux/module.h> #include <linux/device.h> #include <linux/kernel.h> +#include <linux/uaccess.h> #include <linux/etherdevice.h> #include <linux/vringh.h> #include <linux/vdpa.h> @@ -68,6 +69,8 @@ static struct vdpasim_net *sim_to_net(struct vdpasim *vdpasim) static void vdpasim_net_complete(struct vdpasim_virtqueue *vq, size_t len) { + bool notify; + /* Make sure data is wrote before advancing index */ smp_wmb(); @@ -77,7 +80,11 @@ static void vdpasim_net_complete(struct vdpasim_virtqueue *vq, size_t len) smp_wmb(); local_bh_disable(); - if (vringh_need_notify_iotlb(&vq->vring) > 0) + /* The ring may be in user memory, and we can't take faults here */ + pagefault_disable(); + notify = vringh_need_notify_iotlb(&vq->vring) > 0; + pagefault_enable(); + if (notify) vringh_notify(&vq->vring); local_bh_enable(); } diff --git a/drivers/vhost/vringh.c b/drivers/vhost/vringh.c index 9066f9f12ba1..74836eb60c7a 100644 --- a/drivers/vhost/vringh.c +++ b/drivers/vhost/vringh.c @@ -1213,7 +1213,7 @@ static inline int getu16_iotlb(const struct vringh *vrh, return ret; if (vrh->use_va) { - ret = __get_user(tmp, (__virtio16 __user *)ivec.iov.iovec[0].iov_base); + ret = get_user(tmp, (__virtio16 __user *)ivec.iov.iovec[0].iov_base); if (ret) return ret; } else { @@ -1251,7 +1251,7 @@ static inline int putu16_iotlb(const struct vringh *vrh, tmp = cpu_to_vringh16(vrh, val); if (vrh->use_va) { - ret = __put_user(tmp, (__virtio16 __user *)ivec.iov.iovec[0].iov_base); + ret = put_user(tmp, (__virtio16 __user *)ivec.iov.iovec[0].iov_base); if (ret) return ret; } else { diff --git a/drivers/xen/gntdev.c b/drivers/xen/gntdev.c index b71f5bae25b6..49957e3ab49d 100644 --- a/drivers/xen/gntdev.c +++ b/drivers/xen/gntdev.c @@ -849,13 +849,13 @@ static int gntdev_copy(struct gntdev_copy_batch *batch) if (status == GNTST_okay) continue; - if (__get_user(old_status, batch->status[i])) + if (get_user(old_status, batch->status[i])) return -EFAULT; if (old_status != GNTST_okay) continue; - if (__put_user(status, batch->status[i])) + if (put_user(status, batch->status[i])) return -EFAULT; } diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c index c67ec6d282cf..5e56fbc93447 100644 --- a/drivers/xen/privcmd.c +++ b/drivers/xen/privcmd.c @@ -411,12 +411,12 @@ static int mmap_return_error(int err, struct mmap_batch_state *st) gfn |= (err == -ENOENT) ? PRIVCMD_MMAPBATCH_PAGED_ERROR : PRIVCMD_MMAPBATCH_MFN_ERROR; - return __put_user(gfn, st->user_gfn++); + return put_user(gfn, st->user_gfn++); } else st->user_gfn++; } else { /* st->version == 2 */ if (err) - return __put_user(err, st->user_err++); + return put_user(err, st->user_err++); else st->user_err++; } @@ -777,7 +777,7 @@ static long privcmd_ioctl_mmap_resource(struct file *file, rc = HYPERVISOR_memory_op(XENMEM_acquire_resource, &xdata); if (rc) return rc; - return __put_user(xdata.nr_frames, &udata->num); + return put_user(xdata.nr_frames, &udata->num); } mmap_write_lock(mm); diff --git a/kernel/compat.c b/kernel/compat.c index fb50f29d9b36..c754793c7d65 100644 --- a/kernel/compat.c +++ b/kernel/compat.c @@ -180,14 +180,18 @@ int get_compat_sigevent(struct sigevent *event, const struct compat_sigevent __user *u_event) { memset(event, 0, sizeof(*event)); - return (!access_ok(u_event, sizeof(*u_event)) || - __get_user(event->sigev_value.sival_int, - &u_event->sigev_value.sival_int) || - __get_user(event->sigev_signo, &u_event->sigev_signo) || - __get_user(event->sigev_notify, &u_event->sigev_notify) || - __get_user(event->sigev_notify_thread_id, - &u_event->sigev_notify_thread_id)) - ? -EFAULT : 0; + scoped_user_read_access(u_event, Efault) { + unsafe_get_user(event->sigev_value.sival_int, + &u_event->sigev_value.sival_int, Efault); + unsafe_get_user(event->sigev_signo, &u_event->sigev_signo, Efault); + unsafe_get_user(event->sigev_notify, &u_event->sigev_notify, Efault); + unsafe_get_user(event->sigev_notify_thread_id, + &u_event->sigev_notify_thread_id, Efault); + } + return 0; + +Efault: + return -EFAULT; } long compat_get_bitmap(unsigned long *mask, const compat_ulong_t __user *umask, diff --git a/kernel/entry/syscall_user_dispatch.c b/kernel/entry/syscall_user_dispatch.c index 59c861866941..7d1dd7f1bf38 100644 --- a/kernel/entry/syscall_user_dispatch.c +++ b/kernel/entry/syscall_user_dispatch.c @@ -45,11 +45,7 @@ bool syscall_user_dispatch(struct pt_regs *regs) return false; if (likely(sd->selector)) { - /* - * access_ok() is performed once, at prctl time, when - * the selector is loaded by userspace. - */ - if (unlikely(__get_user(state, sd->selector))) { + if (unlikely(get_user(state, untagged_addr(sd->selector)))) { force_exit_sig(SIGSEGV); return true; } diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 7709ea882477..3192806eebfe 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -2404,7 +2404,7 @@ static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr) return -EINVAL; pagefault_disable(); - result = __get_user(opcode, (uprobe_opcode_t __user *)vaddr); + result = get_user(opcode, (uprobe_opcode_t __user *)vaddr); pagefault_enable(); if (likely(result == 0)) diff --git a/kernel/ptrace.c b/kernel/ptrace.c index 4e9822a87aab..3f2fe68ccb14 100644 --- a/kernel/ptrace.c +++ b/kernel/ptrace.c @@ -1365,16 +1365,13 @@ int ptrace_request(struct task_struct *child, long request, struct iovec kiov; struct iovec __user *uiov = datavp; - if (!access_ok(uiov, sizeof(*uiov))) - return -EFAULT; - - if (__get_user(kiov.iov_base, &uiov->iov_base) || - __get_user(kiov.iov_len, &uiov->iov_len)) + if (get_user(kiov.iov_base, &uiov->iov_base) || + get_user(kiov.iov_len, &uiov->iov_len)) return -EFAULT; ret = ptrace_regset(child, request, addr, &kiov); if (!ret) - ret = __put_user(kiov.iov_len, &uiov->iov_len); + ret = put_user(kiov.iov_len, &uiov->iov_len); break; } @@ -1531,11 +1528,8 @@ int compat_ptrace_request(struct task_struct *child, compat_long_t request, compat_uptr_t ptr; compat_size_t len; - if (!access_ok(uiov, sizeof(*uiov))) - return -EFAULT; - - if (__get_user(ptr, &uiov->iov_base) || - __get_user(len, &uiov->iov_len)) + if (get_user(ptr, &uiov->iov_base) || + get_user(len, &uiov->iov_len)) return -EFAULT; kiov.iov_base = compat_ptr(ptr); @@ -1543,7 +1537,7 @@ int compat_ptrace_request(struct task_struct *child, compat_long_t request, ret = ptrace_regset(child, request, addr, &kiov); if (!ret) - ret = __put_user(kiov.iov_len, &uiov->iov_len); + ret = put_user(kiov.iov_len, &uiov->iov_len); break; } #endif diff --git a/kernel/signal.c b/kernel/signal.c index e433de93b430..cb6ec7054eee 100644 --- a/kernel/signal.c +++ b/kernel/signal.c @@ -4553,11 +4553,12 @@ int restore_altstack(const stack_t __user *uss) int __save_altstack(stack_t __user *uss, unsigned long sp) { - struct task_struct *t = current; - int err = __put_user((void __user *)t->sas_ss_sp, &uss->ss_sp) | - __put_user(t->sas_ss_flags, &uss->ss_flags) | - __put_user(t->sas_ss_size, &uss->ss_size); - return err; + scoped_user_write_access(uss, Efault) + unsafe_save_altstack(uss, sp, Efault); + return 0; + +Efault: + return -EFAULT; } #ifdef CONFIG_COMPAT @@ -4606,13 +4607,12 @@ int compat_restore_altstack(const compat_stack_t __user *uss) int __compat_save_altstack(compat_stack_t __user *uss, unsigned long sp) { - int err; - struct task_struct *t = current; - err = __put_user(ptr_to_compat((void __user *)t->sas_ss_sp), - &uss->ss_sp) | - __put_user(t->sas_ss_flags, &uss->ss_flags) | - __put_user(t->sas_ss_size, &uss->ss_size); - return err; + scoped_user_write_access(uss, Efault) + unsafe_compat_save_altstack(uss, sp, Efault); + return 0; + +Efault: + return -EFAULT; } #endif @@ -4790,12 +4790,12 @@ SYSCALL_DEFINE3(sigaction, int, sig, if (act) { old_sigset_t mask; - if (!access_ok(act, sizeof(*act)) || - __get_user(new_ka.sa.sa_handler, &act->sa_handler) || - __get_user(new_ka.sa.sa_restorer, &act->sa_restorer) || - __get_user(new_ka.sa.sa_flags, &act->sa_flags) || - __get_user(mask, &act->sa_mask)) - return -EFAULT; + scoped_user_read_access(act, Efault) { + unsafe_get_user(new_ka.sa.sa_handler, &act->sa_handler, Efault); + unsafe_get_user(new_ka.sa.sa_restorer, &act->sa_restorer, Efault); + unsafe_get_user(new_ka.sa.sa_flags, &act->sa_flags, Efault); + unsafe_get_user(mask, &act->sa_mask, Efault); + } #ifdef __ARCH_HAS_KA_RESTORER new_ka.ka_restorer = NULL; #endif @@ -4805,15 +4805,18 @@ SYSCALL_DEFINE3(sigaction, int, sig, ret = do_sigaction(sig, act ? &new_ka : NULL, oact ? &old_ka : NULL); if (!ret && oact) { - if (!access_ok(oact, sizeof(*oact)) || - __put_user(old_ka.sa.sa_handler, &oact->sa_handler) || - __put_user(old_ka.sa.sa_restorer, &oact->sa_restorer) || - __put_user(old_ka.sa.sa_flags, &oact->sa_flags) || - __put_user(old_ka.sa.sa_mask.sig[0], &oact->sa_mask)) - return -EFAULT; + scoped_user_write_access(oact, Efault) { + unsafe_put_user(old_ka.sa.sa_handler, &oact->sa_handler, Efault); + unsafe_put_user(old_ka.sa.sa_restorer, &oact->sa_restorer, Efault); + unsafe_put_user(old_ka.sa.sa_flags, &oact->sa_flags, Efault); + unsafe_put_user(old_ka.sa.sa_mask.sig[0], &oact->sa_mask, Efault); + } } return ret; + +Efault: + return -EFAULT; } #endif #ifdef CONFIG_COMPAT_OLD_SIGACTION @@ -4827,12 +4830,12 @@ COMPAT_SYSCALL_DEFINE3(sigaction, int, sig, compat_uptr_t handler, restorer; if (act) { - if (!access_ok(act, sizeof(*act)) || - __get_user(handler, &act->sa_handler) || - __get_user(restorer, &act->sa_restorer) || - __get_user(new_ka.sa.sa_flags, &act->sa_flags) || - __get_user(mask, &act->sa_mask)) - return -EFAULT; + scoped_user_read_access(act, Efault) { + unsafe_get_user(handler, &act->sa_handler, Efault); + unsafe_get_user(restorer, &act->sa_restorer, Efault); + unsafe_get_user(new_ka.sa.sa_flags, &act->sa_flags, Efault); + unsafe_get_user(mask, &act->sa_mask, Efault); + } #ifdef __ARCH_HAS_KA_RESTORER new_ka.ka_restorer = NULL; @@ -4845,16 +4848,19 @@ COMPAT_SYSCALL_DEFINE3(sigaction, int, sig, ret = do_sigaction(sig, act ? &new_ka : NULL, oact ? &old_ka : NULL); if (!ret && oact) { - if (!access_ok(oact, sizeof(*oact)) || - __put_user(ptr_to_compat(old_ka.sa.sa_handler), - &oact->sa_handler) || - __put_user(ptr_to_compat(old_ka.sa.sa_restorer), - &oact->sa_restorer) || - __put_user(old_ka.sa.sa_flags, &oact->sa_flags) || - __put_user(old_ka.sa.sa_mask.sig[0], &oact->sa_mask)) - return -EFAULT; + scoped_user_write_access(oact, Efault) { + unsafe_put_user(ptr_to_compat(old_ka.sa.sa_handler), + &oact->sa_handler, Efault); + unsafe_put_user(ptr_to_compat(old_ka.sa.sa_restorer), + &oact->sa_restorer, Efault); + unsafe_put_user(old_ka.sa.sa_flags, &oact->sa_flags, Efault); + unsafe_put_user(old_ka.sa.sa_mask.sig[0], &oact->sa_mask, Efault); + } } return ret; + +Efault: + return -EFAULT; } #endif diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 29260951aa87..88c3b5b513ab 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2388,7 +2388,7 @@ static int copy_user_syms(struct user_syms *us, unsigned long __user *usyms, u32 goto error; for (p = buf, i = 0; i < cnt; i++) { - if (__get_user(usymbol, usyms + i)) { + if (get_user(usymbol, usyms + i)) { err = -EFAULT; goto error; } @@ -3295,8 +3295,8 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr /* * All uoffsets/uref_ctr_offsets/ucookies arrays have the same value - * size, we need to check their address range is safe for __get_user - * calls. + * size. Check their address ranges up front, so that a bad pointer + * fails before anything is looked up or allocated. */ size = sizeof(*uoffsets) * cnt; if (!access_ok(uoffsets, size) || !access_ok(uref_ctr_offsets, size) || @@ -3331,7 +3331,7 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr goto error_free; for (i = 0; i < cnt; i++) { - if (__get_user(uprobes[i].offset, uoffsets + i)) { + if (get_user(uprobes[i].offset, uoffsets + i)) { err = -EFAULT; goto error_free; } @@ -3339,11 +3339,11 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr err = -EINVAL; goto error_free; } - if (uref_ctr_offsets && __get_user(uprobes[i].ref_ctr_offset, uref_ctr_offsets + i)) { + if (uref_ctr_offsets && get_user(uprobes[i].ref_ctr_offset, uref_ctr_offsets + i)) { err = -EFAULT; goto error_free; } - if (ucookies && __get_user(uprobes[i].cookie, ucookies + i)) { + if (ucookies && get_user(uprobes[i].cookie, ucookies + i)) { err = -EFAULT; goto error_free; } diff --git a/net/socket.c b/net/socket.c index c0ab6a3ad8ed..8246cdf18437 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2850,7 +2850,7 @@ int __sys_sendmmsg(int fd, struct mmsghdr __user *mmsg, unsigned int vlen, &msg_sys, flags, &used_address, MSG_EOR); if (err < 0) break; - err = __put_user(err, &compat_entry->msg_len); + err = put_user(err, &compat_entry->msg_len); ++compat_entry; } else { err = ___sys_sendmsg(sock, @@ -2943,16 +2943,16 @@ static int ____sys_recvmsg(struct socket *sock, struct msghdr *msg_sys, if (err < 0) goto out; } - err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), - COMPAT_FLAGS(msg)); + err = put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), + COMPAT_FLAGS(msg)); if (err) goto out; if (MSG_CMSG_COMPAT & flags) - err = __put_user((unsigned long)msg_sys->msg_control - cmsg_ptr, - &msg_compat->msg_controllen); + err = put_user((unsigned long)msg_sys->msg_control - cmsg_ptr, + &msg_compat->msg_controllen); else - err = __put_user((unsigned long)msg_sys->msg_control - cmsg_ptr, - &msg->msg_controllen); + err = put_user((unsigned long)msg_sys->msg_control - cmsg_ptr, + &msg->msg_controllen); if (err) goto out; err = len; @@ -3064,7 +3064,7 @@ static int do_recvmmsg(int fd, struct mmsghdr __user *mmsg, datagrams); if (err < 0) break; - err = __put_user(err, &compat_entry->msg_len); + err = put_user(err, &compat_entry->msg_len); ++compat_entry; } else { err = ___sys_recvmsg(sock, |