linux-toradex.git/kernel/time/tick-sched.c, branch v4.4.85

tick/nohz: Set the correct expiry when switching to nohz/lowres mode

2016-03-03T23:07:26+00:00

commit 1ca8ec532fc2d986f1f4a319857bb18e0c9739b4 upstream.

commit 0ff53d096422 sets the next tick interrupt to the last jiffies update,
i.e. in the past, because the forward operation is invoked before the set
operation. There is no resulting damage (yet), but we get an extra pointless
tick interrupt.

Revert the order so we get the next tick interrupt in the future.

Fixes: commit 0ff53d096422 "tick: sched: Force tick interrupt and get rid of softirq magic"
Signed-off-by: Wanpeng Li 
Cc: Peter Zijlstra 
Cc: Frederic Weisbecker 
Link: http://lkml.kernel.org/r/1453893967-3458-1-git-send-email-wanpeng.li@hotmail.com
Signed-off-by: Thomas Gleixner 
Signed-off-by: Greg Kroah-Hartman

nohz: Assert existing housekeepers when nohz full enabled

2015-09-02T08:33:22+00:00

The code ensures that when nohz full is running, at least the
boot CPU serves as a housekeeper and it can't be later offlined.

Let's assert this assumption to make sure that we have CPUs to
handle unbound jobs like workqueues and timers while nohz full
CPUs run undisturbed.

Also improve the comments on housekeeper offlining prevention.

Signed-off-by: Frederic Weisbecker 
Acked-by: Thomas Gleixner 
Cc: Chris Metcalf 
Cc: Christoph Lameter 
Cc: Linus Torvalds 
Cc: Paul E. McKenney 
Cc: Peter Zijlstra 
Cc: Preeti U Murthy 
Cc: Vatika Harlalka 
Link: http://lkml.kernel.org/r/1441119060-2230-3-git-send-email-fweisbec@gmail.com
Signed-off-by: Ingo Molnar

nohz: Remove useless argument on tick_nohz_task_switch()

2015-07-29T13:45:01+00:00

Leftover from early code.

Cc: Christoph Lameter 
Cc: Ingo Molnar 
Cc: Peter Zijlstra 
Cc: Preeti U Murthy 
Cc: Rik van Riel 
Cc: Thomas Gleixner 
Cc: Viresh Kumar 
Signed-off-by: Frederic Weisbecker

nohz: Move tick_nohz_restart_sched_tick() above its users

2015-07-29T13:45:01+00:00

Fix the function declaration/definition dance.

Reviewed-by: Rik van Riel 
Cc: Christoph Lameter 
Cc: Ingo Molnar 
Cc: Peter Zijlstra 
Cc: Preeti U Murthy 
Cc: Rik van Riel 
Cc: Thomas Gleixner 
Cc: Viresh Kumar 
Signed-off-by: Frederic Weisbecker

nohz: Restart nohz full tick from irq exit

2015-07-29T13:45:00+00:00

Restart the tick when necessary from the irq exit path. It makes nohz
full more flexible, simplify the related IPIs and doesn't bring
significant overhead on irq exit.

In a longer term view, it will allow us to piggyback the nohz kick
on the scheduler IPI in the future instead of sending a dedicated IPI
that often doubles the scheduler IPI on task wakeup. This will require
more changes though including careful review of resched_curr() callers
to include nohz full needs.

Reviewed-by: Rik van Riel 
Cc: Christoph Lameter 
Cc: Ingo Molnar 
Cc: Peter Zijlstra 
Cc: Preeti U Murthy 
Cc: Rik van Riel 
Cc: Thomas Gleixner 
Cc: Viresh Kumar 
Signed-off-by: Frederic Weisbecker

nohz: Remove idle task special case

2015-07-29T13:44:59+00:00

On nohz full early days, idle dynticks and full dynticks weren't well
integrated and we couldn't risk full dynticks calls on idle without
risking messing up tick idle statistics. This is why we prevented such
thing to happen.

Nowadays full dynticks and idle dynticks are better integrated and
interact without known issue.

So lets remove that.

Reviewed-by: Rik van Riel 
Cc: Christoph Lameter 
Cc: Ingo Molnar 
Cc: Peter Zijlstra 
Cc: Preeti U Murthy 
Cc: Rik van Riel 
Cc: Thomas Gleixner 
Cc: Viresh Kumar 
Signed-off-by: Frederic Weisbecker

timer: Minimize nohz off overhead

2015-06-19T13:18:28+00:00

If nohz is disabled on the kernel command line the [hr]timer code
still calls wake_up_nohz_cpu() and tick_nohz_full_cpu(), a pretty
pointless exercise. Cache nohz_active in [hr]timer per cpu bases and
avoid the overhead.

Before:
  48.10%  hog       [.] main
  15.25%  [kernel]  [k] _raw_spin_lock_irqsave
   9.76%  [kernel]  [k] _raw_spin_unlock_irqrestore
   6.50%  [kernel]  [k] mod_timer
   6.44%  [kernel]  [k] lock_timer_base.isra.38
   3.87%  [kernel]  [k] detach_if_pending
   3.80%  [kernel]  [k] del_timer
   2.67%  [kernel]  [k] internal_add_timer
   1.33%  [kernel]  [k] __internal_add_timer
   0.73%  [kernel]  [k] timerfn
   0.54%  [kernel]  [k] wake_up_nohz_cpu

After:
  48.73%  hog       [.] main
  15.36%  [kernel]  [k] _raw_spin_lock_irqsave
   9.77%  [kernel]  [k] _raw_spin_unlock_irqrestore
   6.61%  [kernel]  [k] lock_timer_base.isra.38
   6.42%  [kernel]  [k] mod_timer
   3.90%  [kernel]  [k] detach_if_pending
   3.76%  [kernel]  [k] del_timer
   2.41%  [kernel]  [k] internal_add_timer
   1.39%  [kernel]  [k] __internal_add_timer
   0.76%  [kernel]  [k] timerfn

We probably should have a cached value for nohz full in the per cpu
bases as well to avoid the cpumask check. The base cache line is hot
already, the cpumask not necessarily.

Signed-off-by: Thomas Gleixner 
Cc: Peter Zijlstra 
Cc: Paul McKenney 
Cc: Frederic Weisbecker 
Cc: Eric Dumazet 
Cc: Viresh Kumar 
Cc: John Stultz 
Cc: Joonwoo Park 
Cc: Wenbo Wang 
Link: http://lkml.kernel.org/r/20150526224512.207378134@linutronix.de
Signed-off-by: Thomas Gleixner

timer: Reduce timer migration overhead if disabled

2015-06-19T13:18:28+00:00

Eric reported that the timer_migration sysctl is not really nice
performance wise as it needs to check at every timer insertion whether
the feature is enabled or not. Further the check does not live in the
timer code, so we have an extra function call which checks an extra
cache line to figure out that it is disabled.

We can do better and store that information in the per cpu (hr)timer
bases. I pondered to use a static key, but that's a nightmare to
update from the nohz code and the timer base cache line is hot anyway
when we select a timer base.

The old logic enabled the timer migration unconditionally if
CONFIG_NO_HZ was set even if nohz was disabled on the kernel command
line.

With this modification, we start off with migration disabled. The user
visible sysctl is still set to enabled. If the kernel switches to NOHZ
migration is enabled, if the user did not disable it via the sysctl
prior to the switch. If nohz=off is on the kernel command line,
migration stays disabled no matter what.

Before:
  47.76%  hog       [.] main
  14.84%  [kernel]  [k] _raw_spin_lock_irqsave
   9.55%  [kernel]  [k] _raw_spin_unlock_irqrestore
   6.71%  [kernel]  [k] mod_timer
   6.24%  [kernel]  [k] lock_timer_base.isra.38
   3.76%  [kernel]  [k] detach_if_pending
   3.71%  [kernel]  [k] del_timer
   2.50%  [kernel]  [k] internal_add_timer
   1.51%  [kernel]  [k] get_nohz_timer_target
   1.28%  [kernel]  [k] __internal_add_timer
   0.78%  [kernel]  [k] timerfn
   0.48%  [kernel]  [k] wake_up_nohz_cpu

After:
  48.10%  hog       [.] main
  15.25%  [kernel]  [k] _raw_spin_lock_irqsave
   9.76%  [kernel]  [k] _raw_spin_unlock_irqrestore
   6.50%  [kernel]  [k] mod_timer
   6.44%  [kernel]  [k] lock_timer_base.isra.38
   3.87%  [kernel]  [k] detach_if_pending
   3.80%  [kernel]  [k] del_timer
   2.67%  [kernel]  [k] internal_add_timer
   1.33%  [kernel]  [k] __internal_add_timer
   0.73%  [kernel]  [k] timerfn
   0.54%  [kernel]  [k] wake_up_nohz_cpu


Reported-by: Eric Dumazet 
Signed-off-by: Thomas Gleixner 
Cc: Peter Zijlstra 
Cc: Paul McKenney 
Cc: Frederic Weisbecker 
Cc: Viresh Kumar 
Cc: John Stultz 
Cc: Joonwoo Park 
Cc: Wenbo Wang 
Link: http://lkml.kernel.org/r/20150526224512.127050787@linutronix.de
Signed-off-by: Thomas Gleixner

nohz: Fix !HIGH_RES_TIMERS hang

2015-05-07T14:15:50+00:00

Simon Horman reported this crash on a system with
high-res timers disabled but nohz enabled:

  > ------------[ cut here ]------------
  > kernel BUG at kernel/irq_work.c:135!

    BUG_ON(!irqs_disabled());

So something enabled interrupts in the periodic tick handling machinery,
and that code path indeed has a local_irq_disable()/enable pair in
tick_nohz_switch_to_nohz() which causes havoc. Fix it.

This patch also fixes a +nohz -hrtimers hang reported by Ingo Molnar.

Reported-by: Simon Horman 
Reported-by: Ingo Molnar 
Tested-by: Simon Horman 
Signed-off-by: Thomas Gleixner 
Cc: Borislav Petkov 
Cc: H. Peter Anvin 
Cc: LAK 
Cc: Magnus Damm 
Cc: Peter Zijlstra 
Link: http://lkml.kernel.org/r/alpine.DEB.2.11.1505071425520.4225@nanos
Signed-off-by: Ingo Molnar

tick: Nohz: Rework next timer evaluation

2015-04-22T15:06:50+00:00

The evaluation of the next timer in the nohz code is based on jiffies
while all the tick internals are nano seconds based. We have also to
convert hrtimer nanoseconds to jiffies in the !highres case. That's
just wrong and introduces interesting corner cases.

Turn it around and convert the next timer wheel timer expiry and the
rcu event to clock monotonic and base all calculations on
nanoseconds. That identifies the case where no timer is pending
clearly with an absolute expiry value of KTIME_MAX.

Makes the code more readable and gets rid of the jiffies magic in the
nohz code.

Signed-off-by: Thomas Gleixner 
Reviewed-by: Paul E. McKenney 
Acked-by: Peter Zijlstra 
Cc: Preeti U Murthy 
Cc: Viresh Kumar 
Cc: Marcelo Tosatti 
Cc: Frederic Weisbecker 
Cc: Josh Triplett 
Cc: Lai Jiangshan 
Cc: John Stultz 
Cc: Marcelo Tosatti 
Link: http://lkml.kernel.org/r/20150414203502.184198593@linutronix.de
Signed-off-by: Thomas Gleixner