linux-toradex.git/arch/x86/kernel/irq_64.c, branch v6.5

x86/percpu: Move irq_stack variables next to current_task

2022-10-17T14:41:05+00:00

Further extend struct pcpu_hot with the hard and soft irq stack
pointers.

Signed-off-by: Thomas Gleixner 
Signed-off-by: Peter Zijlstra (Intel) 
Link: https://lore.kernel.org/r/20220915111145.599170752@infradead.org

x86/softirq/64: Inline do_softirq_own_stack()

2021-02-10T22:34:17+00:00

There is no reason to have this as a seperate function for a single caller.

Signed-off-by: Thomas Gleixner 
Reviewed-by: Kees Cook 
Link: https://lore.kernel.org/r/20210210002513.382806685@linutronix.de

softirq: Move do_softirq_own_stack() to generic asm header

2021-02-10T22:34:16+00:00

To avoid include recursion hell move the do_softirq_own_stack() related
content into a generic asm header and include it from all places in arch/
which need the prototype.

This allows architectures to provide an inline implementation of
do_softirq_own_stack() without introducing a lot of #ifdeffery all over the
place.

Signed-off-by: Thomas Gleixner 
Reviewed-by: Kees Cook 
Link: https://lore.kernel.org/r/20210210002513.289960691@linutronix.de

x86: Select CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK

2021-02-10T22:34:16+00:00

Now that all invocations of irq_exit_rcu() happen on the irq stack, turn on
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK which causes the core code to invoke
__do_softirq() directly without going through do_softirq_own_stack().

That means do_softirq_own_stack() is only invoked from task context which
means it can't be on the irq stack. Remove the conditional from
run_softirq_on_irqstack_cond() and rename the function accordingly.

Signed-off-by: Thomas Gleixner 
Reviewed-by: Kees Cook 
Link: https://lore.kernel.org/r/20210210002513.068033456@linutronix.de

x86/softirq: Remove indirection in do_softirq_own_stack()

2021-02-10T22:34:15+00:00

Use the new inline stack switching and remove the old ASM indirect call
implementation.

Signed-off-by: Thomas Gleixner 
Reviewed-by: Kees Cook 
Link: https://lore.kernel.org/r/20210210002512.972714001@linutronix.de

x86/irq/64: Adjust the per CPU irq stack pointer by 8

2021-02-10T22:34:14+00:00

The per CPU hardirq_stack_ptr contains the pointer to the irq stack in the
form that it is ready to be assigned to [ER]SP so that the first push ends
up on the top entry of the stack.

But the stack switching on 64 bit has the following rules:

    1) Store the current stack pointer (RSP) in the top most stack entry
       to allow the unwinder to link back to the previous stack

    2) Set RSP to the top most stack entry

    3) Invoke functions on the irq stack

    4) Pop RSP from the top most stack entry (stored in #1) so it's back
       to the original stack.

That requires all stack switching code to decrement the stored pointer by 8
in order to be able to store the current RSP and then set RSP to that
location. That's a pointless exercise.

Do the -8 adjustment right when storing the pointer and make the data type
a void pointer to avoid confusion vs. the struct irq_stack data type which
is on 64bit only used to declare the backing store. Move the definition
next to the inuse flag so they likely end up in the same cache
line. Sticking them into a struct to enforce it is a seperate change.

Signed-off-by: Thomas Gleixner 
Reviewed-by: Kees Cook 
Link: https://lore.kernel.org/r/20210210002512.354260928@linutronix.de

x86/irq: Make run_on_irqstack_cond() typesafe

2020-09-22T20:13:34+00:00

Sami reported that run_on_irqstack_cond() requires the caller to cast
functions to mismatching types, which trips indirect call Control-Flow
Integrity (CFI) in Clang.

Instead of disabling CFI on that function, provide proper helpers for
the three call variants. The actual ASM code stays the same as that is
out of reach.

 [ bp: Fix __run_on_irqstack() prototype to match. ]

Fixes: 931b94145981 ("x86/entry: Provide helpers for executing on the irqstack")
Reported-by: Nathan Chancellor 
Reported-by: Sami Tolvanen 
Signed-off-by: Thomas Gleixner 
Signed-off-by: Borislav Petkov 
Tested-by: Sami Tolvanen 
Cc: 
Link: https://github.com/ClangBuiltLinux/linux/issues/1052
Link: https://lkml.kernel.org/r/87pn6eb5tv.fsf@nanos.tec.linutronix.de

x86/entry/64: Move do_softirq_own_stack() to C

2020-06-11T13:15:07+00:00

The first step to get rid of the ENTER/LEAVE_IRQ_STACK ASM macro maze.  Use
the new C code helpers to move do_softirq_own_stack() out of ASM code.

Signed-off-by: Thomas Gleixner 
Signed-off-by: Ingo Molnar 
Acked-by: Andy Lutomirski 
Link: https://lore.kernel.org/r/20200521202117.870911120@linutronix.de

x86: fix vmap arguments in map_irq_stack

2020-06-02T17:59:10+00:00

vmap does not take a gfp_t, the flags argument is for VM_* flags.

Signed-off-by: Christoph Hellwig 
Signed-off-by: Andrew Morton 
Cc: Christian Borntraeger 
Cc: Christophe Leroy 
Cc: Daniel Vetter 
Cc: David Airlie 
Cc: Gao Xiang 
Cc: Greg Kroah-Hartman 
Cc: Haiyang Zhang 
Cc: Johannes Weiner 
Cc: "K. Y. Srinivasan" 
Cc: Laura Abbott 
Cc: Mark Rutland 
Cc: Michael Kelley 
Cc: Minchan Kim 
Cc: Nitin Gupta 
Cc: Peter Zijlstra (Intel) 
Cc: Robin Murphy 
Cc: Sakari Ailus 
Cc: Stephen Hemminger 
Cc: Sumit Semwal 
Cc: Wei Liu 
Cc: Benjamin Herrenschmidt 
Cc: Catalin Marinas 
Cc: Heiko Carstens 
Cc: Paul Mackerras 
Cc: Vasily Gorbik 
Cc: Will Deacon 
Link: http://lkml.kernel.org/r/20200414131348.444715-3-hch@lst.de
Signed-off-by: Linus Torvalds

x86/irq: Move IS_ERR_OR_NULL() check into common do_IRQ() code

2019-08-19T21:19:06+00:00

Both the 64bit and the 32bit handle_irq() implementation check the irq
descriptor pointer with IS_ERR_OR_NULL() and return failure. That can be
done simpler in the common do_IRQ() code.

This reduces the 64bit handle_irq() function to a wrapper around
generic_handle_irq_desc(). Invoke it directly from do_IRQ() to spare the
extra function call.

[ tglx: Got rid of the #ifdef and massaged changelog ]

Signed-off-by: Heiner Kallweit 
Signed-off-by: Thomas Gleixner 
Link: https://lkml.kernel.org/r/2ec758c7-9aaa-73ab-f083-cc44c86aa741@gmail.com