This patch fixes 3 issues:
a) it removes the dependency on jiffies, because jiffies are incremented
by a single CPU, and the tick is not synchronized between CPUs. Therefore
relying on it to calculate a window to clip whacky TSC values doesn't work
as it can drift around.
So instead use [GTOD, GTOD+TICK_NSEC) as the window.
b) __update_sched_clock() did (roughly speaking):
delta = sched_clock() - scd->tick_raw;
clock += delta;
Which gives exponential growth, instead of linear.
c) allows the sched_clock_cpu() value to warp the u64 without breaking.
the results are more reliable sched_clock() deltas:
before after sched_clock
cpu_clock: 15750 51312 51488
cpu_clock: 59719 51052 50947
cpu_clock: 15879 51249 51061
cpu_clock: 1 50933 51198
cpu_clock: 1 50931 51039
cpu_clock: 1 51093 50981
cpu_clock: 1 51043 51040
cpu_clock: 1 50959 50938
cpu_clock: 1 50981 51011
cpu_clock: 1 51364 51212
cpu_clock: 1 51219 51273
cpu_clock: 1 51389 51048
cpu_clock: 1 51285 51611
cpu_clock: 1 50964 51137
cpu_clock: 1 50973 50968
cpu_clock: 1 50967 50972
cpu_clock: 1 58910 58485
cpu_clock: 1 51082 51025
cpu_clock: 1 50957 50958
cpu_clock: 1 50958 50957
cpu_clock: 1006128 51128 50971
cpu_clock: 1 51107 51155
cpu_clock: 1 51371 51081
cpu_clock: 1 51104 51365
cpu_clock: 1 51363 51309
cpu_clock: 1 51107 51160
cpu_clock: 1 51139 51100
cpu_clock: 1 51216 51136
cpu_clock: 1 51207 51215
cpu_clock: 1 51087 51263
cpu_clock: 1 51249 51177
cpu_clock: 1 51519 51412
cpu_clock: 1 51416 51255
cpu_clock: 1 51591 51594
cpu_clock: 1 50966 51374
cpu_clock: 1 50966 50966
cpu_clock: 1 51291 50948
cpu_clock: 1 50973 50867
cpu_clock: 1 50970 50970
cpu_clock: 998306 50970 50971
cpu_clock: 1 50971 50970
cpu_clock: 1 50970 50970
cpu_clock: 1 50971 50971
cpu_clock: 1 50970 50970
cpu_clock: 1 51351 50970
cpu_clock: 1 50970 51352
cpu_clock: 1 50971 50970
cpu_clock: 1 50970 50970
cpu_clock: 1 51321 50971
cpu_clock: 1 50974 51324
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Some arch's can't handle sched_clock() being called too early - delay this until sched_clock_init() has been called. Reported-by: Bill Gatliff <bgat@billgatliff.com> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Tested-by: Nishanth Aravamudan <nacc@us.ibm.com> CC: Russell King - ARM Linux <linux@arm.linux.org.uk> Signed-off-by: Ingo Molnar <mingo@elte.hu>
When taking the time of a remote CPU, use the opportunity to couple (sync) the clocks to each other. (in a monotonic way) Signed-off-by: Ingo Molnar <mingo@elte.hu> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Acked-by: Mike Galbraith <efault@gmx.de>
- return the current clock instead of letting callers fetch it from scd->clock Signed-off-by: Ingo Molnar <mingo@elte.hu> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Acked-by: Mike Galbraith <efault@gmx.de>
eliminate prev_raw and use tick_raw instead. It's enough to base the current time on the scheduler tick timestamp alone - the monotonicity and maximum checks will prevent any damage. Signed-off-by: Ingo Molnar <mingo@elte.hu> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Acked-by: Mike Galbraith <efault@gmx.de>
- simplify the remote clock rebasing Signed-off-by: Ingo Molnar <mingo@elte.hu> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Acked-by: Mike Galbraith <efault@gmx.de>
Found an interactivity problem on a quad core test-system - simple CPU loops would occasionally delay the system un an unacceptable way. After much debugging with Peter Zijlstra it turned out that the problem is caused by the string of sched_clock() changes - they caused the CPU clock to jump backwards a bit - which confuses the scheduler arithmetics. (which is unsigned for performance reasons) So revert: # c300ba2: sched_clock: and multiplier for TSC to gtod drift # c0c8773: sched_clock: only update deltas with local reads. # af52a90: sched_clock: stop maximum check on NO HZ # f7cce27: sched_clock: widen the max and min time This solves the interactivity problems. Signed-off-by: Ingo Molnar <mingo@elte.hu> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Acked-by: Mike Galbraith <efault@gmx.de>
Move sched_clock() up to stop warning: weak declaration of `sched_clock' after first use results in unspecified behavior (if -fno-unit-at-a-time). Signed-off-by: Hugh Dickins <hugh@veritas.com> Cc: Mike Travis <travis@sgi.com> Cc: Ben Herrenschmidt <benh@kernel.crashing.org> Cc: Linuxppc-dev@ozlabs.org Signed-off-by: Ingo Molnar <mingo@elte.hu>
The sched_clock code currently tries to keep all CPU clocks of all CPUS somewhat in sync. At every clock tick it records the gtod clock and uses that and jiffies and the TSC to calculate a CPU clock that tries to stay in sync with all the other CPUs. ftrace depends heavily on this timer and it detects when this timer "jumps". One problem is that the TSC and the gtod also drift. When the TSC is 0.1% faster or slower than the gtod it is very noticeable in ftrace. To help compensate for this, I've added a multiplier that tries to keep the CPU clock updating at the same rate as the gtod. I've tried various ways to get it to be in sync and this ended up being the most reliable. At every scheduler tick we calculate the new multiplier: multi = delta_gtod / delta_TSC This means we perform a 64 bit divide at the tick (once a HZ). A shift is used to handle the accuracy. Other methods that failed due to dynamic HZ are: (not used) multi += (gtod - tsc) / delta_gtod (not used) multi += (gtod - (last_tsc + delta_tsc)) / delta_gtod as well as other variants. This code still allows for a slight drift between TSC and gtod, but it keeps the damage down to a minimum. Signed-off-by: Steven Rostedt <srostedt@redhat.com> Cc: Steven Rostedt <srostedt@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: john stultz <johnstul@us.ibm.com> Signed-off-by: Ingo Molnar <mingo@elte.hu>