linux-toradex.git/drivers/dma/fsldma.h, branch v6.4-rc6 Linux kernel for Apalis and Colibri modules fsldma: fix very broken 32-bit ppc ioread64 functionality 2020-08-29T20:50:56+00:00 Linus Torvalds torvalds@linux-foundation.org 2020-08-29T20:50:56+00:00 0a4c56c80f90797e9b9f8426c6aae4c0cf1c9785 Commit ef91bb196b0d ("kernel.h: Silence sparse warning in lower_32_bits") caused new warnings to show in the fsldma driver, but that commit was not to blame: it only exposed some very incorrect code that tried to take the low 32 bits of an address. That made no sense for multiple reasons, the most notable one being that that code was intentionally limited to only 32-bit ppc builds, so "only low 32 bits of an address" was completely nonsensical. There were no high bits to mask off to begin with. But even more importantly fropm a correctness standpoint, turning the address into an integer then caused the subsequent address arithmetic to be completely wrong too, and the "+1" actually incremented the address by one, rather than by four. Which again was incorrect, since the code was reading two 32-bit values and trying to make a 64-bit end result of it all. Surprisingly, the iowrite64() did not suffer from the same odd and incorrect model. This code has never worked, but it's questionable whether anybody cared: of the two users that actually read the 64-bit value (by way of some C preprocessor hackery and eventually the 'get_cdar()' inline function), one of them explicitly ignored the value, and the other one might just happen to work despite the incorrect value being read. This patch at least makes it not fail the build any more, and makes the logic superficially sane. Whether it makes any difference to the code _working_ or not shall remain a mystery. Compile-tested-by: Guenter Roeck <linux@roeck-us.net> Reviewed-by: Guenter Roeck <linux@roeck-us.net> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> 
Commit ef91bb196b0d ("kernel.h: Silence sparse warning in
lower_32_bits") caused new warnings to show in the fsldma driver, but
that commit was not to blame: it only exposed some very incorrect code
that tried to take the low 32 bits of an address.

That made no sense for multiple reasons, the most notable one being that
that code was intentionally limited to only 32-bit ppc builds, so "only
low 32 bits of an address" was completely nonsensical.  There were no
high bits to mask off to begin with.

But even more importantly fropm a correctness standpoint, turning the
address into an integer then caused the subsequent address arithmetic to
be completely wrong too, and the "+1" actually incremented the address
by one, rather than by four.

Which again was incorrect, since the code was reading two 32-bit values
and trying to make a 64-bit end result of it all.  Surprisingly, the
iowrite64() did not suffer from the same odd and incorrect model.

This code has never worked, but it's questionable whether anybody cared:
of the two users that actually read the 64-bit value (by way of some C
preprocessor hackery and eventually the 'get_cdar()' inline function),
one of them explicitly ignored the value, and the other one might just
happen to work despite the incorrect value being read.

This patch at least makes it not fail the build any more, and makes the
logic superficially sane.  Whether it makes any difference to the code
_working_ or not shall remain a mystery.

Compile-tested-by: Guenter Roeck <linux@roeck-us.net>
Reviewed-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 40 2019-05-24T15:27:12+00:00 Thomas Gleixner tglx@linutronix.de 2019-05-20T17:08:05+00:00 ea2305f6a84e87b9398e043297f7982c462abf9f Based on 1 normalized pattern(s): this is free software you can redistribute it and or modify it under the terms of the gnu general public license as published by the free software foundation either version 2 of the license or at your option any later version extracted by the scancode license scanner the SPDX license identifier GPL-2.0-or-later has been chosen to replace the boilerplate/reference in 14 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org> Reviewed-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc) Reviewed-by: Allison Randal <allison@lohutok.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190520170857.915677517@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
Based on 1 normalized pattern(s):

  this is free software you can redistribute it and or modify it under
  the terms of the gnu general public license as published by the free
  software foundation either version 2 of the license or at your
  option any later version

extracted by the scancode license scanner the SPDX license identifier

  GPL-2.0-or-later

has been chosen to replace the boilerplate/reference in 14 file(s).

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)
Reviewed-by: Allison Randal <allison@lohutok.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190520170857.915677517@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
dmaengine: fsldma: Add 64-bit I/O accessors for powerpc64 2019-02-04T07:26:54+00:00 Scott Wood oss@buserror.net 2018-12-22T04:34:45+00:00 6175f6a7ebc77a3c83b425fdb4fc186c771193bf Otherwise 64-bit PPC builds fail with undefined references to these accessors. Cc: Peng Ma <peng.ma@nxp.com> Cc: Wen He <wen.he_1@nxp.com> Fixes: 68997fff94afa (" dmaengine: fsldma: Adding macro FSL_DMA_IN/OUT implement for ARM platform") Signed-off-by: Scott Wood <oss@buserror.net> Signed-off-by: Vinod Koul <vkoul@kernel.org> 
Otherwise 64-bit PPC builds fail with undefined references
to these accessors.

Cc: Peng Ma <peng.ma@nxp.com>
Cc: Wen He <wen.he_1@nxp.com>
Fixes: 68997fff94afa (" dmaengine: fsldma: Adding macro FSL_DMA_IN/OUT implement for ARM platform")
Signed-off-by: Scott Wood <oss@buserror.net>
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dmaengine: fsldma: Adding macro FSL_DMA_IN/OUT implement for ARM platform 2019-01-07T04:20:16+00:00 Peng Ma peng.ma@nxp.com 2018-10-30T02:35:59+00:00 a1ff82a9c165ba26f0e2f4771d0490c513056428 This patch add the macro FSL_DMA_IN/OUT implement for ARM platform. Signed-off-by: Wen He <wen.he_1@nxp.com> Signed-off-by: Peng Ma <peng.ma@nxp.com> Signed-off-by: Vinod Koul <vkoul@kernel.org> 
This patch add the macro FSL_DMA_IN/OUT implement for ARM platform.

Signed-off-by: Wen He <wen.he_1@nxp.com>
Signed-off-by: Peng Ma <peng.ma@nxp.com>
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dmaengine: fsldma: Replace DMA_IN/OUT by FSL_DMA_IN/OUT 2019-01-07T04:20:16+00:00 Wen He wen.he_1@nxp.com 2018-10-30T02:35:58+00:00 a7359e762fdd6eea432fefd9d38851540192e7e3 This patch implement a standard macro call functions is used to NXP dma drivers. Signed-off-by: Wen He <wen.he_1@nxp.com> Signed-off-by: Peng Ma <peng.ma@nxp.com> Signed-off-by: Vinod Koul <vkoul@kernel.org> 
This patch implement a standard macro call functions is
used to NXP dma drivers.

Signed-off-by: Wen He <wen.he_1@nxp.com>
Signed-off-by: Peng Ma <peng.ma@nxp.com>
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dmaengine: fsldma: set BWC, DAHTS and SAHTS values correctly 2017-06-22T13:01:35+00:00 Thomas Breitung thomas.breitung@izt-labs.de 2017-06-19T14:40:04+00:00 ccc077292733f3143b444255fa5ec49a8ff2763b The bits of BWC, DAHTS and SAHTS in the DMA mode register must be cleared before a new value can be or-ed in. Signed-off-by: Thomas Breitung <thomas.breitung@izt-labs.de> Signed-off-by: Wolfgang Ocker <weo@reccoware.de> Signed-off-by: Vinod Koul <vinod.koul@intel.com> 
The bits of BWC, DAHTS and SAHTS in the DMA mode register must be cleared
before a new value can be or-ed in.

Signed-off-by: Thomas Breitung <thomas.breitung@izt-labs.de>
Signed-off-by: Wolfgang Ocker <weo@reccoware.de>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>
dmaengine: fsldma: declare slave capabilities for the generic code 2015-01-13T18:28:08+00:00 Kevin Hao haokexin@gmail.com 2015-01-08T10:38:16+00:00 75dc1775ec83db305a68d153a7ac5eb4e8b634a0 Since commit ecc19d17868b ("dmaengine: Add a warning for drivers not using the generic slave caps retrieval"), the dma drivers are required to fill the caps infos in order to support generic slaves caps retrieval. Otherwise we will get a warning like this: WARNING: at drivers/dma/dmaengine.c:830 Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 3.19.0-rc2-next-20150106-dirty #271 task: c0000001f70a0000 ti: c0000001f7044000 task.ti: c0000001f7044000 NIP: c00000000032b238 LR: c00000000032b234 CTR: c00000000001d258 REGS: c0000001f7047330 TRAP: 0700 Tainted: G W (3.19.0-rc2-next-20150106-dirty) MSR: 0000000080029000 <CE,EE,ME> CR: 24adbe22 XER: 20000000 SOFTE: 1 GPR00: c00000000032b234 c0000001f70475b0 c0000000009b4848 0000000000000040 GPR04: 0000000000000001 0000000000000001 0000000000000000 000000000000000f GPR08: 0000000000000000 c000000000902988 c000000000902988 00000000000052c8 GPR12: 0000000024adbe22 c00000000fff4000 c000000000002038 0000000000000000 GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR24: 0000000000000000 0000000000000000 c000000000972dc8 c0000000007e6fd0 GPR28: c0000001f76d1d30 c0000001f76d1c10 c0000001f76d1c00 0000000000000000 NIP [c00000000032b238] .dma_async_device_register+0x3f8/0x5b8 LR [c00000000032b234] .dma_async_device_register+0x3f4/0x5b8 Call Trace: [c0000001f70475b0] [c00000000032b234] .dma_async_device_register+0x3f4/0x5b8 (unreliable) [c0000001f70476a0] [c00000000032ca78] .fsldma_of_probe+0x298/0x438 [c0000001f7047750] [c00000000037080c] .platform_drv_probe+0x50/0x9c [c0000001f70477d0] [c00000000036e74c] .really_probe+0xa4/0x29c [c0000001f7047870] [c00000000036eae4] .__driver_attach+0x100/0x104 [c0000001f7047900] [c00000000036c1f0] .bus_for_each_dev+0x84/0xe4 [c0000001f70479a0] [c00000000036e164] .driver_attach+0x24/0x38 [c0000001f7047a10] [c00000000036dcc8] .bus_add_driver+0x1c8/0x2ac [c0000001f7047ab0] [c00000000036f14c] .driver_register+0x8c/0x158 [c0000001f7047b30] [c0000000003707a8] .__platform_driver_register+0x6c/0x80 [c0000001f7047ba0] [c000000000898a3c] .fsldma_init+0x2c/0x40 [c0000001f7047c10] [c000000000001818] .do_one_initcall+0xb8/0x234 [c0000001f7047d00] [c000000000878e2c] .kernel_init_freeable+0x188/0x268 [c0000001f7047db0] [c000000000002054] .kernel_init+0x1c/0xfc8 [c0000001f7047e30] [c000000000000884] .ret_from_kernel_thread+0x58/0xd4 Instruction dump: 7fb9f840 3bffffe0 409effac 7f54d378 48000060 813d0050 2f890000 40befdd0 3c62ffe3 38632450 482f0aa9 60000000 <0fe00000> 4bfffdb8 7f03c378 482ed465 Signed-off-by: Kevin Hao <haokexin@gmail.com> Signed-off-by: Vinod Koul <vinod.koul@intel.com> 
Since commit ecc19d17868b ("dmaengine: Add a warning for drivers not
using the generic slave caps retrieval"), the dma drivers are required
to fill the caps infos in order to support generic slaves caps
retrieval. Otherwise we will get a warning like this:
  WARNING: at drivers/dma/dmaengine.c:830
  Modules linked in:
  CPU: 0 PID: 1 Comm: swapper/0 Tainted: G        W       3.19.0-rc2-next-20150106-dirty #271
  task: c0000001f70a0000 ti: c0000001f7044000 task.ti: c0000001f7044000
  NIP: c00000000032b238 LR: c00000000032b234 CTR: c00000000001d258
  REGS: c0000001f7047330 TRAP: 0700   Tainted: G        W        (3.19.0-rc2-next-20150106-dirty)
  MSR: 0000000080029000 <CE,EE,ME>  CR: 24adbe22  XER: 20000000
  SOFTE: 1
  GPR00: c00000000032b234 c0000001f70475b0 c0000000009b4848 0000000000000040
  GPR04: 0000000000000001 0000000000000001 0000000000000000 000000000000000f
  GPR08: 0000000000000000 c000000000902988 c000000000902988 00000000000052c8
  GPR12: 0000000024adbe22 c00000000fff4000 c000000000002038 0000000000000000
  GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  GPR24: 0000000000000000 0000000000000000 c000000000972dc8 c0000000007e6fd0
  GPR28: c0000001f76d1d30 c0000001f76d1c10 c0000001f76d1c00 0000000000000000
  NIP [c00000000032b238] .dma_async_device_register+0x3f8/0x5b8
  LR [c00000000032b234] .dma_async_device_register+0x3f4/0x5b8
  Call Trace:
  [c0000001f70475b0] [c00000000032b234] .dma_async_device_register+0x3f4/0x5b8 (unreliable)
  [c0000001f70476a0] [c00000000032ca78] .fsldma_of_probe+0x298/0x438
  [c0000001f7047750] [c00000000037080c] .platform_drv_probe+0x50/0x9c
  [c0000001f70477d0] [c00000000036e74c] .really_probe+0xa4/0x29c
  [c0000001f7047870] [c00000000036eae4] .__driver_attach+0x100/0x104
  [c0000001f7047900] [c00000000036c1f0] .bus_for_each_dev+0x84/0xe4
  [c0000001f70479a0] [c00000000036e164] .driver_attach+0x24/0x38
  [c0000001f7047a10] [c00000000036dcc8] .bus_add_driver+0x1c8/0x2ac
  [c0000001f7047ab0] [c00000000036f14c] .driver_register+0x8c/0x158
  [c0000001f7047b30] [c0000000003707a8] .__platform_driver_register+0x6c/0x80
  [c0000001f7047ba0] [c000000000898a3c] .fsldma_init+0x2c/0x40
  [c0000001f7047c10] [c000000000001818] .do_one_initcall+0xb8/0x234
  [c0000001f7047d00] [c000000000878e2c] .kernel_init_freeable+0x188/0x268
  [c0000001f7047db0] [c000000000002054] .kernel_init+0x1c/0xfc8
  [c0000001f7047e30] [c000000000000884] .ret_from_kernel_thread+0x58/0xd4
  Instruction dump:
  7fb9f840 3bffffe0 409effac 7f54d378 48000060 813d0050 2f890000 40befdd0
  3c62ffe3 38632450 482f0aa9 60000000 <0fe00000> 4bfffdb8 7f03c378 482ed465

Signed-off-by: Kevin Hao <haokexin@gmail.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>
dmaengine: Freescale: change descriptor release process for supporting async_tx 2014-07-14T16:02:18+00:00 Hongbo Zhang hongbo.zhang@freescale.com 2014-05-21T08:03:03+00:00 43452fadd614b62b84e950838cb7d2419f3aafb1 Fix the potential risk when enable config NET_DMA and ASYNC_TX. Async_tx is lack of support in current release process of dma descriptor, all descriptors will be released whatever is acked or no-acked by async_tx, so there is a potential race condition when dma engine is uesd by others clients (e.g. when enable NET_DMA to offload TCP). In our case, a race condition which is raised when use both of talitos and dmaengine to offload xor is because napi scheduler will sync all pending requests in dma channels, it affects the process of raid operations due to ack_tx is not checked in fsl dma. The no-acked descriptor is freed which is submitted just now, as a dependent tx, this freed descriptor trigger BUG_ON(async_tx_test_ack(depend_tx)) in async_tx_submit(). TASK = ee1a94a0[1390] 'md0_raid5' THREAD: ecf40000 CPU: 0 GPR00: 00000001 ecf41ca0 ee44/921a94a0 0000003f 00000001 c00593e4 00000000 00000001 GPR08: 00000000 a7a7a7a7 00000001 045/920000002 42028042 100a38d4 ed576d98 00000000 GPR16: ed5a11b0 00000000 2b162000 00000200 046/920000000 2d555000 ed3015e8 c15a7aa0 GPR24: 00000000 c155fc40 00000000 ecb63220 ecf41d28 e47/92f640bb0 ef640c30 ecf41ca0 NIP [c02b048c] async_tx_submit+0x6c/0x2b4 LR [c02b068c] async_tx_submit+0x26c/0x2b4 Call Trace: [ecf41ca0] [c02b068c] async_tx_submit+0x26c/0x2b448/92 (unreliable) [ecf41cd0] [c02b0a4c] async_memcpy+0x240/0x25c [ecf41d20] [c0421064] async_copy_data+0xa0/0x17c [ecf41d70] [c0421cf4] __raid_run_ops+0x874/0xe10 [ecf41df0] [c0426ee4] handle_stripe+0x820/0x25e8 [ecf41e90] [c0429080] raid5d+0x3d4/0x5b4 [ecf41f40] [c04329b8] md_thread+0x138/0x16c [ecf41f90] [c008277c] kthread+0x8c/0x90 [ecf41ff0] [c0011630] kernel_thread+0x4c/0x68 Another modification in this patch is the change of completed descriptors, there is a potential risk which caused by exception interrupt, all descriptors in ld_running list are seemed completed when an interrupt raised, it works fine under normal condition, but if there is an exception occured, it cannot work as our excepted. Hardware should not be depend on s/w list, the right way is to read current descriptor address register to find the last completed descriptor. If an interrupt is raised by an error, all descriptors in ld_running should not be seemed finished, or these unfinished descriptors in ld_running will be released wrongly. A simple way to reproduce: Enable dmatest first, then insert some bad descriptors which can trigger Programming Error interrupts before the good descriptors. Last, the good descriptors will be freed before they are processsed because of the exception intrerrupt. Note: the bad descriptors are only for simulating an exception interrupt. This case can illustrate the potential risk in current fsl-dma very well. Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com> Signed-off-by: Qiang Liu <qiang.liu@freescale.com> Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu> Signed-off-by: Vinod Koul <vinod.koul@intel.com> 
Fix the potential risk when enable config NET_DMA and ASYNC_TX. Async_tx is
lack of support in current release process of dma descriptor, all descriptors
will be released whatever is acked or no-acked by async_tx, so there is a
potential race condition when dma engine is uesd by others clients (e.g. when
enable NET_DMA to offload TCP).

In our case, a race condition which is raised when use both of talitos and
dmaengine to offload xor is because napi scheduler will sync all pending
requests in dma channels, it affects the process of raid operations due to
ack_tx is not checked in fsl dma. The no-acked descriptor is freed which is
submitted just now, as a dependent tx, this freed descriptor trigger
BUG_ON(async_tx_test_ack(depend_tx)) in async_tx_submit().

TASK = ee1a94a0[1390] 'md0_raid5' THREAD: ecf40000 CPU: 0
GPR00: 00000001 ecf41ca0 ee44/921a94a0 0000003f 00000001 c00593e4 00000000 00000001
GPR08: 00000000 a7a7a7a7 00000001 045/920000002 42028042 100a38d4 ed576d98 00000000
GPR16: ed5a11b0 00000000 2b162000 00000200 046/920000000 2d555000 ed3015e8 c15a7aa0
GPR24: 00000000 c155fc40 00000000 ecb63220 ecf41d28 e47/92f640bb0 ef640c30 ecf41ca0
NIP [c02b048c] async_tx_submit+0x6c/0x2b4
LR [c02b068c] async_tx_submit+0x26c/0x2b4
Call Trace:
[ecf41ca0] [c02b068c] async_tx_submit+0x26c/0x2b448/92 (unreliable)
[ecf41cd0] [c02b0a4c] async_memcpy+0x240/0x25c
[ecf41d20] [c0421064] async_copy_data+0xa0/0x17c
[ecf41d70] [c0421cf4] __raid_run_ops+0x874/0xe10
[ecf41df0] [c0426ee4] handle_stripe+0x820/0x25e8
[ecf41e90] [c0429080] raid5d+0x3d4/0x5b4
[ecf41f40] [c04329b8] md_thread+0x138/0x16c
[ecf41f90] [c008277c] kthread+0x8c/0x90
[ecf41ff0] [c0011630] kernel_thread+0x4c/0x68

Another modification in this patch is the change of completed descriptors,
there is a potential risk which caused by exception interrupt, all descriptors
in ld_running list are seemed completed when an interrupt raised, it works fine
under normal condition, but if there is an exception occured, it cannot work as
our excepted. Hardware should not be depend on s/w list, the right way is to
read current descriptor address register to find the last completed descriptor.
If an interrupt is raised by an error, all descriptors in ld_running should not
be seemed finished, or these unfinished descriptors in ld_running will be
released wrongly.

A simple way to reproduce:
Enable dmatest first, then insert some bad descriptors which can trigger
Programming Error interrupts before the good descriptors. Last, the good
descriptors will be freed before they are processsed because of the exception
intrerrupt.

Note: the bad descriptors are only for simulating an exception interrupt.  This
case can illustrate the potential risk in current fsl-dma very well.

Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com>
Signed-off-by: Qiang Liu <qiang.liu@freescale.com>
Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>
dmaengine: Freescale: add suspend resume functions for DMA driver 2014-07-14T16:02:18+00:00 Hongbo Zhang hongbo.zhang@freescale.com 2014-05-21T08:03:02+00:00 14c6a3333c8e885604fc98768d8b9a32e08110ac This patch adds suspend and resume functions for Freescale DMA driver. Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com> Signed-off-by: Vinod Koul <vinod.koul@intel.com> 
This patch adds suspend and resume functions for Freescale DMA driver.

Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>
DMA: Freescale: change BWC from 256 bytes to 1024 bytes 2014-01-20T07:43:22+00:00 Hongbo Zhang hongbo.zhang@freescale.com 2014-01-16T06:10:53+00:00 0ca583a239a854fd403bf8b659cdff8c603372c9 Freescale DMA has a feature of BandWidth Control (ab. BWC), which is currently 256 bytes and should be changed to 1024 bytes for best DMA throughput. Changing BWC from 256 to 1024 will improve DMA performance much, in cases whatever one channel is running or multi channels are running simultanously, large or small buffers are copied. And this change doesn't impact memory access performance remarkably, lmbench tests show that for some cases the memory performance are decreased very slightly, while the others are even better. Tested on T4240. Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com> Signed-off-by: Vinod Koul <vinod.koul@intel.com> 
Freescale DMA has a feature of BandWidth Control (ab. BWC), which is currently
256 bytes and should be changed to 1024 bytes for best DMA throughput.
Changing BWC from 256 to 1024 will improve DMA performance much, in cases
whatever one channel is running or multi channels are running simultanously,
large or small buffers are copied.  And this change doesn't impact memory
access performance remarkably, lmbench tests show that for some cases the
memory performance are decreased very slightly, while the others are even
better.
Tested on T4240.

Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>