Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux

Merge tag 'dmaengine-4.10-rc1' of git://git.infradead.org/users/vkoul/slave-dma

Pull dmaengine updates from Vinod Koul:
"Fairly routine update this time around with all changes specific to
drivers:

- New driver for STMicroelectronics FDMA
- Memory-to-memory transfers on dw dmac
- Support for slave maps on pl08x devices
- Bunch of driver fixes to use dma_pool_zalloc
- Bunch of compile and warning fixes spread across drivers"

[ The ST FDMA driver already came in earlier through the remoteproc tree ]

* tag 'dmaengine-4.10-rc1' of git://git.infradead.org/users/vkoul/slave-dma: (68 commits)
dmaengine: sirf-dma: remove unused ‘sdesc’
dmaengine: pl330: remove unused ‘regs’
dmaengine: s3c24xx: remove unused ‘cdata’
dmaengine: stm32-dma: remove unused ‘src_addr’
dmaengine: stm32-dma: remove unused ‘dst_addr’
dmaengine: stm32-dma: remove unused ‘sfcr’
dmaengine: pch_dma: remove unused ‘cookie’
dmaengine: mic_x100_dma: remove unused ‘data’
dmaengine: img-mdc: remove unused ‘prev_phys’
dmaengine: usb-dmac: remove unused ‘uchan’
dmaengine: ioat: remove unused ‘res’
dmaengine: ioat: remove unused ‘ioat_dma’
dmaengine: ioat: remove unused ‘is_raid_device’
dmaengine: pl330: do not generate unaligned access
dmaengine: k3dma: move to dma_pool_zalloc
dmaengine: at_hdmac: move to dma_pool_zalloc
dmaengine: at_xdmac: don't restore unsaved status
dmaengine: ioat: set error code on failures
dmaengine: ioat: set error code on failures
dmaengine: DW DMAC: add multi-block property to device tree
...

+868 -349
+8
Documentation/devicetree/bindings/dma/nbpfaxi.txt
··· 23 23 #define NBPF_SLAVE_RQ_LEVEL 4 24 24 25 25 Optional properties: 26 + - max-burst-mem-read: limit burst size for memory reads 27 + (DMA_MEM_TO_MEM/DMA_MEM_TO_DEV) to this value, specified in bytes, rather 28 + than using the maximum burst size allowed by the hardware's buffer size. 29 + - max-burst-mem-write: limit burst size for memory writes 30 + (DMA_DEV_TO_MEM/DMA_MEM_TO_MEM) to this value, specified in bytes, rather 31 + than using the maximum burst size allowed by the hardware's buffer size. 32 + If both max-burst-mem-read and max-burst-mem-write are set, DMA_MEM_TO_MEM 33 + will use the lower value. 26 34 27 35 You can use dma-channels and dma-requests as described in dma.txt, although they 28 36 won't be used, this information is derived from the compatibility string.
+9 -3
Documentation/devicetree/bindings/dma/qcom_hidma_mgmt.txt
··· 5 5 environments. 6 6 7 7 Each HIDMA HW instance consists of multiple DMA channels. These channels 8 - share the same bandwidth. The bandwidth utilization can be parititioned 8 + share the same bandwidth. The bandwidth utilization can be partitioned 9 9 among channels based on the priority and weight assignments. 10 10 11 11 There are only two priority levels and 15 weigh assignments possible. 12 12 13 13 Other parameters here determine how much of the system bus this HIDMA 14 - instance can use like maximum read/write request and and number of bytes to 14 + instance can use like maximum read/write request and number of bytes to 15 15 read/write in a single burst. 16 16 17 17 Main node required properties: ··· 47 47 the channel nodes appear on their own, not under a management node. 48 48 49 49 Required properties: 50 - - compatible: must contain "qcom,hidma-1.0" 50 + - compatible: must contain "qcom,hidma-1.0" for initial HW or "qcom,hidma-1.1" 51 + for MSI capable HW. 51 52 - reg: Addresses for the transfer and event channel 52 53 - interrupts: Should contain the event interrupt 53 54 - desc-count: Number of asynchronous requests this channel can handle 54 55 - iommus: required a iommu node 56 + 57 + Optional properties for MSI: 58 + - msi-parent : See the generic MSI binding described in 59 + devicetree/bindings/interrupt-controller/msi.txt for a description of the 60 + msi-parent property. 55 61 56 62 Example: 57 63
+1
Documentation/devicetree/bindings/dma/renesas,rcar-dmac.txt
··· 24 24 - "renesas,dmac-r8a7793" (R-Car M2-N) 25 25 - "renesas,dmac-r8a7794" (R-Car E2) 26 26 - "renesas,dmac-r8a7795" (R-Car H3) 27 + - "renesas,dmac-r8a7796" (R-Car M3-W) 27 28 28 29 - reg: base address and length of the registers block for the DMAC 29 30
+2
Documentation/devicetree/bindings/dma/snps-dma.txt
··· 27 27 that services interrupts for this device 28 28 - is_private: The device channels should be marked as private and not for by the 29 29 general purpose DMA channel allocator. False if not passed. 30 + - multi-block: Multi block transfers supported by hardware. Array property with 31 + one cell per channel. 0: not supported, 1 (default): supported. 30 32 31 33 Example: 32 34
+8 -8
Documentation/dmaengine/client.txt
··· 37 37 2. Set slave and controller specific parameters 38 38 39 39 Next step is always to pass some specific information to the DMA 40 - driver. Most of the generic information which a slave DMA can use 41 - is in struct dma_slave_config. This allows the clients to specify 40 + driver. Most of the generic information which a slave DMA can use 41 + is in struct dma_slave_config. This allows the clients to specify 42 42 DMA direction, DMA addresses, bus widths, DMA burst lengths etc 43 43 for the peripheral. 44 44 ··· 52 52 struct dma_slave_config *config) 53 53 54 54 Please see the dma_slave_config structure definition in dmaengine.h 55 - for a detailed explanation of the struct members. Please note 55 + for a detailed explanation of the struct members. Please note 56 56 that the 'direction' member will be going away as it duplicates the 57 57 direction given in the prepare call. 58 58 ··· 101 101 desc = dmaengine_prep_slave_sg(chan, sgl, nr_sg, direction, flags); 102 102 103 103 Once a descriptor has been obtained, the callback information can be 104 - added and the descriptor must then be submitted. Some DMA engine 104 + added and the descriptor must then be submitted. Some DMA engine 105 105 drivers may hold a spinlock between a successful preparation and 106 106 submission so it is important that these two operations are closely 107 107 paired. ··· 138 138 activity via other DMA engine calls not covered in this document. 139 139 140 140 dmaengine_submit() will not start the DMA operation, it merely adds 141 - it to the pending queue. For this, see step 5, dma_async_issue_pending. 141 + it to the pending queue. For this, see step 5, dma_async_issue_pending. 142 142 143 143 5. Issue pending DMA requests and wait for callback notification 144 144 ··· 184 184 185 185 3. int dmaengine_resume(struct dma_chan *chan) 186 186 187 - Resume a previously paused DMA channel. It is invalid to resume a 187 + Resume a previously paused DMA channel. It is invalid to resume a 188 188 channel which is not currently paused. 189 189 190 190 4. enum dma_status dma_async_is_tx_complete(struct dma_chan *chan, 191 191 dma_cookie_t cookie, dma_cookie_t *last, dma_cookie_t *used) 192 192 193 - This can be used to check the status of the channel. Please see 193 + This can be used to check the status of the channel. Please see 194 194 the documentation in include/linux/dmaengine.h for a more complete 195 195 description of this API. 196 196 ··· 200 200 201 201 Note: 202 202 Not all DMA engine drivers can return reliable information for 203 - a running DMA channel. It is recommended that DMA engine users 203 + a running DMA channel. It is recommended that DMA engine users 204 204 pause or stop (via dmaengine_terminate_all()) the channel before 205 205 using this API. 206 206
+5 -5
Documentation/dmaengine/dmatest.txt
··· 34 34 % ls -1 /sys/class/dma/ 35 35 36 36 Once started a message like "dmatest: Started 1 threads using dma0chan0" is 37 - emitted. After that only test failure messages are reported until the test 37 + emitted. After that only test failure messages are reported until the test 38 38 stops. 39 39 40 40 Note that running a new test will not stop any in progress test. ··· 43 43 % cat /sys/module/dmatest/parameters/run 44 44 45 45 To wait for test completion userpace can poll 'run' until it is false, or use 46 - the wait parameter. Specifying 'wait=1' when loading the module causes module 46 + the wait parameter. Specifying 'wait=1' when loading the module causes module 47 47 initialization to pause until a test run has completed, while reading 48 48 /sys/module/dmatest/parameters/wait waits for any running test to complete 49 - before returning. For example, the following scripts wait for 42 tests 50 - to complete before exiting. Note that if 'iterations' is set to 'infinite' then 49 + before returning. For example, the following scripts wait for 42 tests 50 + to complete before exiting. Note that if 'iterations' is set to 'infinite' then 51 51 waiting is disabled. 52 52 53 53 Example: ··· 81 81 82 82 The message format is unified across the different types of errors. A number in 83 83 the parens represents additional information, e.g. error code, error counter, 84 - or status. A test thread also emits a summary line at completion listing the 84 + or status. A test thread also emits a summary line at completion listing the 85 85 number of tests executed, number that failed, and a result code. 86 86 87 87 Example:
+1 -1
Documentation/dmaengine/provider.txt
··· 384 384 - The descriptor should be prepared for reuse by invoking 385 385 dmaengine_desc_set_reuse() which will set DMA_CTRL_REUSE. 386 386 - dmaengine_desc_set_reuse() will succeed only when channel support 387 - reusable descriptor as exhibited by capablities 387 + reusable descriptor as exhibited by capabilities 388 388 - As a consequence, if a device driver wants to skip the dma_map_sg() and 389 389 dma_unmap_sg() in between 2 transfers, because the DMA'd data wasn't used, 390 390 it can resubmit the transfer right after its completion.
+1 -1
Documentation/dmaengine/pxa_dma.txt
··· 29 29 30 30 d) Bandwidth guarantee 31 31 The PXA architecture has 4 levels of DMAs priorities : high, normal, low. 32 - The high prorities get twice as much bandwidth as the normal, which get twice 32 + The high priorities get twice as much bandwidth as the normal, which get twice 33 33 as much as the low priorities. 34 34 A driver should be able to request a priority, especially the real-time 35 35 ones such as pxa_camera with (big) throughputs.
+1
arch/arc/boot/dts/abilis_tb10x.dtsi
··· 129 129 data-width = <4>; 130 130 clocks = <&ahb_clk>; 131 131 clock-names = "hclk"; 132 + multi-block = <1 1 1 1 1 1>; 132 133 }; 133 134 134 135 i2c0: i2c@FF120000 {
+2
arch/arm/boot/dts/spear13xx.dtsi
··· 118 118 block_size = <0xfff>; 119 119 dma-masters = <2>; 120 120 data-width = <8 8>; 121 + multi-block = <1 1 1 1 1 1 1 1>; 121 122 }; 122 123 123 124 dma@eb000000 { ··· 135 134 chan_priority = <1>; 136 135 block_size = <0xfff>; 137 136 data-width = <8 8>; 137 + multi-block = <1 1 1 1 1 1 1 1>; 138 138 }; 139 139 140 140 fsmc: flash@b0000000 {
+32
arch/arm/mach-s3c64xx/pl080.c
··· 117 117 } 118 118 }; 119 119 120 + static const struct dma_slave_map s3c64xx_dma0_slave_map[] = { 121 + { "s3c6400-uart.0", "tx", &s3c64xx_dma0_info[0] }, 122 + { "s3c6400-uart.0", "rx", &s3c64xx_dma0_info[1] }, 123 + { "s3c6400-uart.1", "tx", &s3c64xx_dma0_info[2] }, 124 + { "s3c6400-uart.1", "rx", &s3c64xx_dma0_info[3] }, 125 + { "s3c6400-uart.2", "tx", &s3c64xx_dma0_info[4] }, 126 + { "s3c6400-uart.2", "rx", &s3c64xx_dma0_info[5] }, 127 + { "s3c6400-uart.3", "tx", &s3c64xx_dma0_info[6] }, 128 + { "s3c6400-uart.3", "rx", &s3c64xx_dma0_info[7] }, 129 + { "samsung-pcm.0", "tx", &s3c64xx_dma0_info[8] }, 130 + { "samsung-pcm.0", "rx", &s3c64xx_dma0_info[9] }, 131 + { "samsung-i2s.0", "tx", &s3c64xx_dma0_info[10] }, 132 + { "samsung-i2s.0", "rx", &s3c64xx_dma0_info[11] }, 133 + { "s3c6410-spi.0", "tx", &s3c64xx_dma0_info[12] }, 134 + { "s3c6410-spi.0", "rx", &s3c64xx_dma0_info[13] }, 135 + { "samsung-i2s.2", "tx", &s3c64xx_dma0_info[14] }, 136 + { "samsung-i2s.2", "rx", &s3c64xx_dma0_info[15] }, 137 + }; 138 + 120 139 struct pl08x_platform_data s3c64xx_dma0_plat_data = { 121 140 .memcpy_channel = { 122 141 .bus_id = "memcpy", ··· 153 134 .put_xfer_signal = pl08x_put_xfer_signal, 154 135 .slave_channels = s3c64xx_dma0_info, 155 136 .num_slave_channels = ARRAY_SIZE(s3c64xx_dma0_info), 137 + .slave_map = s3c64xx_dma0_slave_map, 138 + .slave_map_len = ARRAY_SIZE(s3c64xx_dma0_slave_map), 156 139 }; 157 140 158 141 static AMBA_AHB_DEVICE(s3c64xx_dma0, "dma-pl080s.0", 0, ··· 228 207 }, 229 208 }; 230 209 210 + static const struct dma_slave_map s3c64xx_dma1_slave_map[] = { 211 + { "samsung-pcm.1", "tx", &s3c64xx_dma1_info[0] }, 212 + { "samsung-pcm.1", "rx", &s3c64xx_dma1_info[1] }, 213 + { "samsung-i2s.1", "tx", &s3c64xx_dma1_info[2] }, 214 + { "samsung-i2s.1", "rx", &s3c64xx_dma1_info[3] }, 215 + { "s3c6410-spi.1", "tx", &s3c64xx_dma1_info[4] }, 216 + { "s3c6410-spi.1", "rx", &s3c64xx_dma1_info[5] }, 217 + }; 218 + 231 219 struct pl08x_platform_data s3c64xx_dma1_plat_data = { 232 220 .memcpy_channel = { 233 221 .bus_id = "memcpy", ··· 254 224 .put_xfer_signal = pl08x_put_xfer_signal, 255 225 .slave_channels = s3c64xx_dma1_info, 256 226 .num_slave_channels = ARRAY_SIZE(s3c64xx_dma1_info), 227 + .slave_map = s3c64xx_dma1_slave_map, 228 + .slave_map_len = ARRAY_SIZE(s3c64xx_dma1_slave_map), 257 229 }; 258 230 259 231 static AMBA_AHB_DEVICE(s3c64xx_dma1, "dma-pl080s.1", 0,
-24
arch/arm/plat-samsung/devs.c
··· 1124 1124 pd.num_cs = num_cs; 1125 1125 pd.src_clk_nr = src_clk_nr; 1126 1126 pd.cfg_gpio = (cfg_gpio) ? cfg_gpio : s3c64xx_spi0_cfg_gpio; 1127 - pd.dma_tx = (void *)DMACH_SPI0_TX; 1128 - pd.dma_rx = (void *)DMACH_SPI0_RX; 1129 - #if defined(CONFIG_PL330_DMA) 1130 - pd.filter = pl330_filter; 1131 - #elif defined(CONFIG_S3C64XX_PL080) 1132 - pd.filter = pl08x_filter_id; 1133 - #elif defined(CONFIG_S3C24XX_DMAC) 1134 - pd.filter = s3c24xx_dma_filter; 1135 - #endif 1136 1127 1137 1128 s3c_set_platdata(&pd, sizeof(pd), &s3c64xx_device_spi0); 1138 1129 } ··· 1160 1169 pd.num_cs = num_cs; 1161 1170 pd.src_clk_nr = src_clk_nr; 1162 1171 pd.cfg_gpio = (cfg_gpio) ? cfg_gpio : s3c64xx_spi1_cfg_gpio; 1163 - pd.dma_tx = (void *)DMACH_SPI1_TX; 1164 - pd.dma_rx = (void *)DMACH_SPI1_RX; 1165 - #if defined(CONFIG_PL330_DMA) 1166 - pd.filter = pl330_filter; 1167 - #elif defined(CONFIG_S3C64XX_PL080) 1168 - pd.filter = pl08x_filter_id; 1169 - #endif 1170 - 1171 1172 1172 1173 s3c_set_platdata(&pd, sizeof(pd), &s3c64xx_device_spi1); 1173 1174 } ··· 1196 1213 pd.num_cs = num_cs; 1197 1214 pd.src_clk_nr = src_clk_nr; 1198 1215 pd.cfg_gpio = (cfg_gpio) ? cfg_gpio : s3c64xx_spi2_cfg_gpio; 1199 - pd.dma_tx = (void *)DMACH_SPI2_TX; 1200 - pd.dma_rx = (void *)DMACH_SPI2_RX; 1201 - #if defined(CONFIG_PL330_DMA) 1202 - pd.filter = pl330_filter; 1203 - #elif defined(CONFIG_S3C64XX_PL080) 1204 - pd.filter = pl08x_filter_id; 1205 - #endif 1206 1216 1207 1217 s3c_set_platdata(&pd, sizeof(pd), &s3c64xx_device_spi2); 1208 1218 }
+1 -1
drivers/dma/Kconfig
··· 494 494 or vice versa. It does not support memory to memory data transfer. 495 495 496 496 config TEGRA210_ADMA 497 - bool "NVIDIA Tegra210 ADMA support" 497 + tristate "NVIDIA Tegra210 ADMA support" 498 498 depends on (ARCH_TEGRA_210_SOC || COMPILE_TEST) && PM_CLK 499 499 select DMA_ENGINE 500 500 select DMA_VIRTUAL_CHANNELS
+11
drivers/dma/amba-pl08x.c
··· 1793 1793 } 1794 1794 EXPORT_SYMBOL_GPL(pl08x_filter_id); 1795 1795 1796 + static bool pl08x_filter_fn(struct dma_chan *chan, void *chan_id) 1797 + { 1798 + struct pl08x_dma_chan *plchan = to_pl08x_chan(chan); 1799 + 1800 + return plchan->cd == chan_id; 1801 + } 1802 + 1796 1803 /* 1797 1804 * Just check that the device is there and active 1798 1805 * TODO: turn this bit on/off depending on the number of physical channels ··· 2314 2307 ret = -EINVAL; 2315 2308 goto out_no_platdata; 2316 2309 } 2310 + } else { 2311 + pl08x->slave.filter.map = pl08x->pd->slave_map; 2312 + pl08x->slave.filter.mapcnt = pl08x->pd->slave_map_len; 2313 + pl08x->slave.filter.fn = pl08x_filter_fn; 2317 2314 } 2318 2315 2319 2316 /* By default, AHB1 only. If dualmaster, from platform */
+1 -2
drivers/dma/at_hdmac.c
··· 111 111 struct at_dma *atdma = to_at_dma(chan->device); 112 112 dma_addr_t phys; 113 113 114 - desc = dma_pool_alloc(atdma->dma_desc_pool, gfp_flags, &phys); 114 + desc = dma_pool_zalloc(atdma->dma_desc_pool, gfp_flags, &phys); 115 115 if (desc) { 116 - memset(desc, 0, sizeof(struct at_desc)); 117 116 INIT_LIST_HEAD(&desc->tx_list); 118 117 dma_async_tx_descriptor_init(&desc->txd, chan); 119 118 /* txd.flags will be overwritten in prep functions */
+1 -4
drivers/dma/at_xdmac.c
··· 221 221 int irq; 222 222 struct clk *clk; 223 223 u32 save_gim; 224 - u32 save_gs; 225 224 struct dma_pool *at_xdmac_desc_pool; 226 225 struct at_xdmac_chan chan[0]; 227 226 }; ··· 443 444 struct at_xdmac *atxdmac = to_at_xdmac(chan->device); 444 445 dma_addr_t phys; 445 446 446 - desc = dma_pool_alloc(atxdmac->at_xdmac_desc_pool, gfp_flags, &phys); 447 + desc = dma_pool_zalloc(atxdmac->at_xdmac_desc_pool, gfp_flags, &phys); 447 448 if (desc) { 448 - memset(desc, 0, sizeof(*desc)); 449 449 INIT_LIST_HEAD(&desc->descs_list); 450 450 dma_async_tx_descriptor_init(&desc->tx_dma_desc, chan); 451 451 desc->tx_dma_desc.tx_submit = at_xdmac_tx_submit; ··· 1894 1896 } 1895 1897 1896 1898 at_xdmac_write(atxdmac, AT_XDMAC_GIE, atxdmac->save_gim); 1897 - at_xdmac_write(atxdmac, AT_XDMAC_GE, atxdmac->save_gs); 1898 1899 list_for_each_entry_safe(chan, _chan, &atxdmac->dma.channels, device_node) { 1899 1900 atchan = to_at_xdmac_chan(chan); 1900 1901 at_xdmac_chan_write(atchan, AT_XDMAC_CC, atchan->save_cc);
+50 -24
drivers/dma/dmatest.c
··· 164 164 struct task_struct *task; 165 165 struct dma_chan *chan; 166 166 u8 **srcs; 167 + u8 **usrcs; 167 168 u8 **dsts; 169 + u8 **udsts; 168 170 enum dma_transaction_type type; 169 171 bool done; 170 172 }; ··· 433 431 ktime_t comparetime = ktime_set(0, 0); 434 432 s64 runtime = 0; 435 433 unsigned long long total_len = 0; 434 + u8 align = 0; 436 435 437 436 set_freezable(); 438 437 ··· 444 441 params = &info->params; 445 442 chan = thread->chan; 446 443 dev = chan->device; 447 - if (thread->type == DMA_MEMCPY) 444 + if (thread->type == DMA_MEMCPY) { 445 + align = dev->copy_align; 448 446 src_cnt = dst_cnt = 1; 449 - else if (thread->type == DMA_SG) 447 + } else if (thread->type == DMA_SG) { 448 + align = dev->copy_align; 450 449 src_cnt = dst_cnt = sg_buffers; 451 - else if (thread->type == DMA_XOR) { 450 + } else if (thread->type == DMA_XOR) { 452 451 /* force odd to ensure dst = src */ 453 452 src_cnt = min_odd(params->xor_sources | 1, dev->max_xor); 454 453 dst_cnt = 1; 454 + align = dev->xor_align; 455 455 } else if (thread->type == DMA_PQ) { 456 456 /* force odd to ensure dst = src */ 457 457 src_cnt = min_odd(params->pq_sources | 1, dma_maxpq(dev, 0)); 458 458 dst_cnt = 2; 459 + align = dev->pq_align; 459 460 460 - pq_coefs = kmalloc(params->pq_sources+1, GFP_KERNEL); 461 + pq_coefs = kmalloc(params->pq_sources + 1, GFP_KERNEL); 461 462 if (!pq_coefs) 462 463 goto err_thread_type; 463 464 ··· 470 463 } else 471 464 goto err_thread_type; 472 465 473 - thread->srcs = kcalloc(src_cnt+1, sizeof(u8 *), GFP_KERNEL); 466 + thread->srcs = kcalloc(src_cnt + 1, sizeof(u8 *), GFP_KERNEL); 474 467 if (!thread->srcs) 475 468 goto err_srcs; 469 + 470 + thread->usrcs = kcalloc(src_cnt + 1, sizeof(u8 *), GFP_KERNEL); 471 + if (!thread->usrcs) 472 + goto err_usrcs; 473 + 476 474 for (i = 0; i < src_cnt; i++) { 477 - thread->srcs[i] = kmalloc(params->buf_size, GFP_KERNEL); 478 - if (!thread->srcs[i]) 475 + thread->usrcs[i] = kmalloc(params->buf_size + align, 476 + GFP_KERNEL); 477 + if (!thread->usrcs[i]) 479 478 goto err_srcbuf; 479 + 480 + /* align srcs to alignment restriction */ 481 + if (align) 482 + thread->srcs[i] = PTR_ALIGN(thread->usrcs[i], align); 483 + else 484 + thread->srcs[i] = thread->usrcs[i]; 480 485 } 481 486 thread->srcs[i] = NULL; 482 487 483 - thread->dsts = kcalloc(dst_cnt+1, sizeof(u8 *), GFP_KERNEL); 488 + thread->dsts = kcalloc(dst_cnt + 1, sizeof(u8 *), GFP_KERNEL); 484 489 if (!thread->dsts) 485 490 goto err_dsts; 491 + 492 + thread->udsts = kcalloc(dst_cnt + 1, sizeof(u8 *), GFP_KERNEL); 493 + if (!thread->udsts) 494 + goto err_udsts; 495 + 486 496 for (i = 0; i < dst_cnt; i++) { 487 - thread->dsts[i] = kmalloc(params->buf_size, GFP_KERNEL); 488 - if (!thread->dsts[i]) 497 + thread->udsts[i] = kmalloc(params->buf_size + align, 498 + GFP_KERNEL); 499 + if (!thread->udsts[i]) 489 500 goto err_dstbuf; 501 + 502 + /* align dsts to alignment restriction */ 503 + if (align) 504 + thread->dsts[i] = PTR_ALIGN(thread->udsts[i], align); 505 + else 506 + thread->dsts[i] = thread->udsts[i]; 490 507 } 491 508 thread->dsts[i] = NULL; 492 509 ··· 529 498 dma_addr_t srcs[src_cnt]; 530 499 dma_addr_t *dsts; 531 500 unsigned int src_off, dst_off, len; 532 - u8 align = 0; 533 501 struct scatterlist tx_sg[src_cnt]; 534 502 struct scatterlist rx_sg[src_cnt]; 535 503 536 504 total_tests++; 537 - 538 - /* honor alignment restrictions */ 539 - if (thread->type == DMA_MEMCPY || thread->type == DMA_SG) 540 - align = dev->copy_align; 541 - else if (thread->type == DMA_XOR) 542 - align = dev->xor_align; 543 - else if (thread->type == DMA_PQ) 544 - align = dev->pq_align; 545 505 546 506 if (1 << align > params->buf_size) { 547 507 pr_err("%u-byte buffer too small for %d-byte alignment\n", ··· 571 549 filltime = ktime_add(filltime, diff); 572 550 } 573 551 574 - um = dmaengine_get_unmap_data(dev->dev, src_cnt+dst_cnt, 552 + um = dmaengine_get_unmap_data(dev->dev, src_cnt + dst_cnt, 575 553 GFP_KERNEL); 576 554 if (!um) { 577 555 failed_tests++; ··· 751 729 752 730 ret = 0; 753 731 err_dstbuf: 754 - for (i = 0; thread->dsts[i]; i++) 755 - kfree(thread->dsts[i]); 732 + for (i = 0; thread->udsts[i]; i++) 733 + kfree(thread->udsts[i]); 734 + kfree(thread->udsts); 735 + err_udsts: 756 736 kfree(thread->dsts); 757 737 err_dsts: 758 738 err_srcbuf: 759 - for (i = 0; thread->srcs[i]; i++) 760 - kfree(thread->srcs[i]); 739 + for (i = 0; thread->usrcs[i]; i++) 740 + kfree(thread->usrcs[i]); 741 + kfree(thread->usrcs); 742 + err_usrcs: 761 743 kfree(thread->srcs); 762 744 err_srcs: 763 745 kfree(pq_coefs);
+1 -1
drivers/dma/dw/core.c
··· 1569 1569 (dwc_params >> DWC_PARAMS_MBLK_EN & 0x1) == 0; 1570 1570 } else { 1571 1571 dwc->block_size = pdata->block_size; 1572 - dwc->nollp = pdata->is_nollp; 1572 + dwc->nollp = !pdata->multi_block[i]; 1573 1573 } 1574 1574 } 1575 1575
+17 -1
drivers/dma/dw/platform.c
··· 102 102 { 103 103 struct device_node *np = pdev->dev.of_node; 104 104 struct dw_dma_platform_data *pdata; 105 - u32 tmp, arr[DW_DMA_MAX_NR_MASTERS]; 105 + u32 tmp, arr[DW_DMA_MAX_NR_MASTERS], mb[DW_DMA_MAX_NR_CHANNELS]; 106 106 u32 nr_masters; 107 107 u32 nr_channels; 108 108 ··· 118 118 119 119 if (of_property_read_u32(np, "dma-channels", &nr_channels)) 120 120 return NULL; 121 + if (nr_channels > DW_DMA_MAX_NR_CHANNELS) 122 + return NULL; 121 123 122 124 pdata = devm_kzalloc(&pdev->dev, sizeof(*pdata), GFP_KERNEL); 123 125 if (!pdata) ··· 130 128 131 129 if (of_property_read_bool(np, "is_private")) 132 130 pdata->is_private = true; 131 + 132 + /* 133 + * All known devices, which use DT for configuration, support 134 + * memory-to-memory transfers. So enable it by default. 135 + */ 136 + pdata->is_memcpy = true; 133 137 134 138 if (!of_property_read_u32(np, "chan_allocation_order", &tmp)) 135 139 pdata->chan_allocation_order = (unsigned char)tmp; ··· 152 144 } else if (!of_property_read_u32_array(np, "data_width", arr, nr_masters)) { 153 145 for (tmp = 0; tmp < nr_masters; tmp++) 154 146 pdata->data_width[tmp] = BIT(arr[tmp] & 0x07); 147 + } 148 + 149 + if (!of_property_read_u32_array(np, "multi-block", mb, nr_channels)) { 150 + for (tmp = 0; tmp < nr_channels; tmp++) 151 + pdata->multi_block[tmp] = mb[tmp]; 152 + } else { 153 + for (tmp = 0; tmp < nr_channels; tmp++) 154 + pdata->multi_block[tmp] = 1; 155 155 } 156 156 157 157 return pdata;
+2 -1
drivers/dma/dw/regs.h
··· 12 12 #include <linux/interrupt.h> 13 13 #include <linux/dmaengine.h> 14 14 15 - #define DW_DMA_MAX_NR_CHANNELS 8 15 + #include "internal.h" 16 + 16 17 #define DW_DMA_MAX_NR_REQUESTS 16 17 18 18 19 /* flow controller */
+3
drivers/dma/edma.c
··· 2451 2451 int i; 2452 2452 s8 (*queue_priority_mapping)[2]; 2453 2453 2454 + /* re initialize dummy slot to dummy param set */ 2455 + edma_write_slot(ecc, ecc->dummy_slot, &dummy_paramset); 2456 + 2454 2457 queue_priority_mapping = ecc->info->queue_priority_mapping; 2455 2458 2456 2459 /* Event queue priority mapping */
+1
drivers/dma/fsl_raid.c
··· 881 881 { .compatible = "fsl,raideng-v1.0", }, 882 882 {} 883 883 }; 884 + MODULE_DEVICE_TABLE(of, fsl_re_ids); 884 885 885 886 static struct platform_driver fsl_re_driver = { 886 887 .driver = {
+5 -3
drivers/dma/hsu/pci.c
··· 77 77 if (!chip) 78 78 return -ENOMEM; 79 79 80 + ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES); 81 + if (ret < 0) 82 + return ret; 83 + 80 84 chip->dev = &pdev->dev; 81 85 chip->regs = pcim_iomap_table(pdev)[0]; 82 86 chip->length = pci_resource_len(pdev, 0); 83 87 chip->offset = HSU_PCI_CHAN_OFFSET; 84 - chip->irq = pdev->irq; 85 - 86 - pci_enable_msi(pdev); 88 + chip->irq = pci_irq_vector(pdev, 0); 87 89 88 90 ret = hsu_dma_probe(chip); 89 91 if (ret)
+3 -6
drivers/dma/img-mdc-dma.c
··· 292 292 struct mdc_dma *mdma = mchan->mdma; 293 293 struct mdc_tx_desc *mdesc; 294 294 struct mdc_hw_list_desc *curr, *prev = NULL; 295 - dma_addr_t curr_phys, prev_phys; 295 + dma_addr_t curr_phys; 296 296 297 297 if (!len) 298 298 return NULL; ··· 324 324 xfer_size); 325 325 326 326 prev = curr; 327 - prev_phys = curr_phys; 328 327 329 328 mdesc->list_len++; 330 329 src += xfer_size; ··· 374 375 struct mdc_dma *mdma = mchan->mdma; 375 376 struct mdc_tx_desc *mdesc; 376 377 struct mdc_hw_list_desc *curr, *prev = NULL; 377 - dma_addr_t curr_phys, prev_phys; 378 + dma_addr_t curr_phys; 378 379 379 380 if (!buf_len && !period_len) 380 381 return NULL; ··· 429 430 } 430 431 431 432 prev = curr; 432 - prev_phys = curr_phys; 433 433 434 434 mdesc->list_len++; 435 435 buf_addr += xfer_size; ··· 456 458 struct mdc_tx_desc *mdesc; 457 459 struct scatterlist *sg; 458 460 struct mdc_hw_list_desc *curr, *prev = NULL; 459 - dma_addr_t curr_phys, prev_phys; 461 + dma_addr_t curr_phys; 460 462 unsigned int i; 461 463 462 464 if (!sgl) ··· 507 509 } 508 510 509 511 prev = curr; 510 - prev_phys = curr_phys; 511 512 512 513 mdesc->list_len++; 513 514 mdesc->list_xfer_size += xfer_size;
+9 -4
drivers/dma/imx-sdma.c
··· 298 298 * @event_id1 for channels that use 2 events 299 299 * @word_size peripheral access size 300 300 * @buf_tail ID of the buffer that was processed 301 + * @buf_ptail ID of the previous buffer that was processed 301 302 * @num_bd max NUM_BD. number of descriptors currently handling 302 303 */ 303 304 struct sdma_channel { ··· 310 309 unsigned int event_id1; 311 310 enum dma_slave_buswidth word_size; 312 311 unsigned int buf_tail; 312 + unsigned int buf_ptail; 313 313 unsigned int num_bd; 314 314 unsigned int period_len; 315 315 struct sdma_buffer_descriptor *bd; ··· 702 700 sdmac->chn_real_count = bd->mode.count; 703 701 bd->mode.status |= BD_DONE; 704 702 bd->mode.count = sdmac->period_len; 703 + sdmac->buf_ptail = sdmac->buf_tail; 704 + sdmac->buf_tail = (sdmac->buf_tail + 1) % sdmac->num_bd; 705 705 706 706 /* 707 707 * The callback is called from the interrupt context in order ··· 713 709 */ 714 710 715 711 dmaengine_desc_get_callback_invoke(&sdmac->desc, NULL); 716 - 717 - sdmac->buf_tail++; 718 - sdmac->buf_tail %= sdmac->num_bd; 719 712 720 713 if (error) 721 714 sdmac->status = old_status; ··· 1187 1186 sdmac->flags = 0; 1188 1187 1189 1188 sdmac->buf_tail = 0; 1189 + sdmac->buf_ptail = 0; 1190 + sdmac->chn_real_count = 0; 1190 1191 1191 1192 dev_dbg(sdma->dev, "setting up %d entries for channel %d.\n", 1192 1193 sg_len, channel); ··· 1291 1288 sdmac->status = DMA_IN_PROGRESS; 1292 1289 1293 1290 sdmac->buf_tail = 0; 1291 + sdmac->buf_ptail = 0; 1292 + sdmac->chn_real_count = 0; 1294 1293 sdmac->period_len = period_len; 1295 1294 1296 1295 sdmac->flags |= IMX_DMA_SG_LOOP; ··· 1390 1385 u32 residue; 1391 1386 1392 1387 if (sdmac->flags & IMX_DMA_SG_LOOP) 1393 - residue = (sdmac->num_bd - sdmac->buf_tail) * 1388 + residue = (sdmac->num_bd - sdmac->buf_ptail) * 1394 1389 sdmac->period_len - sdmac->chn_real_count; 1395 1390 else 1396 1391 residue = sdmac->chn_count - sdmac->chn_real_count;
+4 -13
drivers/dma/ioat/dma.c
··· 39 39 #include "../dmaengine.h" 40 40 41 41 static char *chanerr_str[] = { 42 + "DMA Transfer Source Address Error", 42 43 "DMA Transfer Destination Address Error", 43 44 "Next Descriptor Address Error", 44 45 "Descriptor Error", ··· 67 66 "Result Guard Tag verification Error", 68 67 "Result Application Tag verification Error", 69 68 "Result Reference Tag verification Error", 70 - NULL 71 69 }; 72 70 73 71 static void ioat_eh(struct ioatdma_chan *ioat_chan); ··· 75 75 { 76 76 int i; 77 77 78 - for (i = 0; i < 32; i++) { 78 + for (i = 0; i < ARRAY_SIZE(chanerr_str); i++) { 79 79 if ((chanerr >> i) & 1) { 80 - if (chanerr_str[i]) { 81 - dev_err(to_dev(ioat_chan), "Err(%d): %s\n", 82 - i, chanerr_str[i]); 83 - } else 84 - break; 80 + dev_err(to_dev(ioat_chan), "Err(%d): %s\n", 81 + i, chanerr_str[i]); 85 82 } 86 83 } 87 84 } ··· 338 341 { 339 342 struct ioat_dma_descriptor *hw; 340 343 struct ioat_ring_ent *desc; 341 - struct ioatdma_device *ioat_dma; 342 344 struct ioatdma_chan *ioat_chan = to_ioat_chan(chan); 343 345 int chunk; 344 346 dma_addr_t phys; 345 347 u8 *pos; 346 348 off_t offs; 347 - 348 - ioat_dma = to_ioatdma_device(chan->device); 349 349 350 350 chunk = idx / IOAT_DESCS_PER_2M; 351 351 idx &= (IOAT_DESCS_PER_2M - 1); ··· 608 614 609 615 tx = &desc->txd; 610 616 if (tx->cookie) { 611 - struct dmaengine_result res; 612 - 613 617 dma_cookie_complete(tx); 614 618 dma_descriptor_unmap(tx); 615 - res.result = DMA_TRANS_NOERROR; 616 619 dmaengine_desc_get_callback_invoke(tx, NULL); 617 620 tx->callback = NULL; 618 621 tx->callback_result = NULL;
+14 -7
drivers/dma/ioat/init.c
··· 340 340 dma_src = dma_map_single(dev, src, IOAT_TEST_SIZE, DMA_TO_DEVICE); 341 341 if (dma_mapping_error(dev, dma_src)) { 342 342 dev_err(dev, "mapping src buffer failed\n"); 343 + err = -ENOMEM; 343 344 goto free_resources; 344 345 } 345 346 dma_dest = dma_map_single(dev, dest, IOAT_TEST_SIZE, DMA_FROM_DEVICE); 346 347 if (dma_mapping_error(dev, dma_dest)) { 347 348 dev_err(dev, "mapping dest buffer failed\n"); 349 + err = -ENOMEM; 348 350 goto unmap_src; 349 351 } 350 352 flags = DMA_PREP_INTERRUPT; ··· 829 827 op = IOAT_OP_XOR; 830 828 831 829 dest_dma = dma_map_page(dev, dest, 0, PAGE_SIZE, DMA_FROM_DEVICE); 832 - if (dma_mapping_error(dev, dest_dma)) 830 + if (dma_mapping_error(dev, dest_dma)) { 831 + err = -ENOMEM; 833 832 goto free_resources; 833 + } 834 834 835 835 for (i = 0; i < IOAT_NUM_SRC_TEST; i++) 836 836 dma_srcs[i] = DMA_ERROR_CODE; 837 837 for (i = 0; i < IOAT_NUM_SRC_TEST; i++) { 838 838 dma_srcs[i] = dma_map_page(dev, xor_srcs[i], 0, PAGE_SIZE, 839 839 DMA_TO_DEVICE); 840 - if (dma_mapping_error(dev, dma_srcs[i])) 840 + if (dma_mapping_error(dev, dma_srcs[i])) { 841 + err = -ENOMEM; 841 842 goto dma_unmap; 843 + } 842 844 } 843 845 tx = dma->device_prep_dma_xor(dma_chan, dest_dma, dma_srcs, 844 846 IOAT_NUM_SRC_TEST, PAGE_SIZE, ··· 910 904 for (i = 0; i < IOAT_NUM_SRC_TEST + 1; i++) { 911 905 dma_srcs[i] = dma_map_page(dev, xor_val_srcs[i], 0, PAGE_SIZE, 912 906 DMA_TO_DEVICE); 913 - if (dma_mapping_error(dev, dma_srcs[i])) 907 + if (dma_mapping_error(dev, dma_srcs[i])) { 908 + err = -ENOMEM; 914 909 goto dma_unmap; 910 + } 915 911 } 916 912 tx = dma->device_prep_dma_xor_val(dma_chan, dma_srcs, 917 913 IOAT_NUM_SRC_TEST + 1, PAGE_SIZE, ··· 965 957 for (i = 0; i < IOAT_NUM_SRC_TEST + 1; i++) { 966 958 dma_srcs[i] = dma_map_page(dev, xor_val_srcs[i], 0, PAGE_SIZE, 967 959 DMA_TO_DEVICE); 968 - if (dma_mapping_error(dev, dma_srcs[i])) 960 + if (dma_mapping_error(dev, dma_srcs[i])) { 961 + err = -ENOMEM; 969 962 goto dma_unmap; 963 + } 970 964 } 971 965 tx = dma->device_prep_dma_xor_val(dma_chan, dma_srcs, 972 966 IOAT_NUM_SRC_TEST + 1, PAGE_SIZE, ··· 1081 1071 struct dma_device *dma; 1082 1072 struct dma_chan *c; 1083 1073 struct ioatdma_chan *ioat_chan; 1084 - bool is_raid_device = false; 1085 1074 int err; 1086 1075 u16 val16; 1087 1076 ··· 1104 1095 ioat_dma->cap &= ~(IOAT_CAP_XOR|IOAT_CAP_PQ); 1105 1096 1106 1097 if (ioat_dma->cap & IOAT_CAP_XOR) { 1107 - is_raid_device = true; 1108 1098 dma->max_xor = 8; 1109 1099 1110 1100 dma_cap_set(DMA_XOR, dma->cap_mask); ··· 1114 1106 } 1115 1107 1116 1108 if (ioat_dma->cap & IOAT_CAP_PQ) { 1117 - is_raid_device = true; 1118 1109 1119 1110 dma->device_prep_dma_pq = ioat_prep_pq; 1120 1111 dma->device_prep_dma_pq_val = ioat_prep_pq_val;
+1 -2
drivers/dma/k3dma.c
··· 458 458 if (!ds) 459 459 return NULL; 460 460 461 - ds->desc_hw = dma_pool_alloc(d->pool, GFP_NOWAIT, &ds->desc_hw_lli); 461 + ds->desc_hw = dma_pool_zalloc(d->pool, GFP_NOWAIT, &ds->desc_hw_lli); 462 462 if (!ds->desc_hw) { 463 463 dev_dbg(chan->device->dev, "vch %p: dma alloc fail\n", &c->vc); 464 464 kfree(ds); 465 465 return NULL; 466 466 } 467 - memset(ds->desc_hw, 0, sizeof(struct k3_desc_hw) * num); 468 467 ds->desc_num = num; 469 468 return ds; 470 469 }
-2
drivers/dma/mic_x100_dma.c
··· 554 554 int ret; 555 555 556 556 for (i = first_chan; i < first_chan + MIC_DMA_NUM_CHAN; i++) { 557 - unsigned long data; 558 557 ch = &mic_dma_dev->mic_ch[i]; 559 - data = (unsigned long)ch; 560 558 ch->ch_num = i; 561 559 ch->owner = owner; 562 560 spin_lock_init(&ch->cleanup_lock);
+180 -10
drivers/dma/mv_xor.c
··· 68 68 hw_desc->byte_count = byte_count; 69 69 } 70 70 71 + /* Populate the descriptor */ 72 + static void mv_xor_config_sg_ll_desc(struct mv_xor_desc_slot *desc, 73 + dma_addr_t dma_src, dma_addr_t dma_dst, 74 + u32 len, struct mv_xor_desc_slot *prev) 75 + { 76 + struct mv_xor_desc *hw_desc = desc->hw_desc; 77 + 78 + hw_desc->status = XOR_DESC_DMA_OWNED; 79 + hw_desc->phy_next_desc = 0; 80 + /* Configure for XOR with only one src address -> MEMCPY */ 81 + hw_desc->desc_command = XOR_DESC_OPERATION_XOR | (0x1 << 0); 82 + hw_desc->phy_dest_addr = dma_dst; 83 + hw_desc->phy_src_addr[0] = dma_src; 84 + hw_desc->byte_count = len; 85 + 86 + if (prev) { 87 + struct mv_xor_desc *hw_prev = prev->hw_desc; 88 + 89 + hw_prev->phy_next_desc = desc->async_tx.phys; 90 + } 91 + } 92 + 93 + static void mv_xor_desc_config_eod(struct mv_xor_desc_slot *desc) 94 + { 95 + struct mv_xor_desc *hw_desc = desc->hw_desc; 96 + 97 + /* Enable end-of-descriptor interrupt */ 98 + hw_desc->desc_command |= XOR_DESC_EOD_INT_EN; 99 + } 100 + 71 101 static void mv_desc_set_mode(struct mv_xor_desc_slot *desc) 72 102 { 73 103 struct mv_xor_desc *hw_desc = desc->hw_desc; ··· 258 228 list_for_each_entry_safe(iter, _iter, &mv_chan->completed_slots, 259 229 node) { 260 230 261 - if (async_tx_test_ack(&iter->async_tx)) 231 + if (async_tx_test_ack(&iter->async_tx)) { 262 232 list_move_tail(&iter->node, &mv_chan->free_slots); 233 + if (!list_empty(&iter->sg_tx_list)) { 234 + list_splice_tail_init(&iter->sg_tx_list, 235 + &mv_chan->free_slots); 236 + } 237 + } 263 238 } 264 239 return 0; 265 240 } ··· 279 244 /* the client is allowed to attach dependent operations 280 245 * until 'ack' is set 281 246 */ 282 - if (!async_tx_test_ack(&desc->async_tx)) 247 + if (!async_tx_test_ack(&desc->async_tx)) { 283 248 /* move this slot to the completed_slots */ 284 249 list_move_tail(&desc->node, &mv_chan->completed_slots); 285 - else 250 + if (!list_empty(&desc->sg_tx_list)) { 251 + list_splice_tail_init(&desc->sg_tx_list, 252 + &mv_chan->completed_slots); 253 + } 254 + } else { 286 255 list_move_tail(&desc->node, &mv_chan->free_slots); 256 + if (!list_empty(&desc->sg_tx_list)) { 257 + list_splice_tail_init(&desc->sg_tx_list, 258 + &mv_chan->free_slots); 259 + } 260 + } 287 261 288 262 return 0; 289 263 } ··· 494 450 dma_async_tx_descriptor_init(&slot->async_tx, chan); 495 451 slot->async_tx.tx_submit = mv_xor_tx_submit; 496 452 INIT_LIST_HEAD(&slot->node); 453 + INIT_LIST_HEAD(&slot->sg_tx_list); 497 454 dma_desc = mv_chan->dma_desc_pool; 498 455 slot->async_tx.phys = dma_desc + idx * MV_XOR_SLOT_SIZE; 499 456 slot->idx = idx++; ··· 660 615 * XOR operation with a single dummy source address. 661 616 */ 662 617 return mv_xor_prep_dma_xor(chan, dest, &src, 1, len, flags); 618 + } 619 + 620 + /** 621 + * mv_xor_prep_dma_sg - prepare descriptors for a memory sg transaction 622 + * @chan: DMA channel 623 + * @dst_sg: Destination scatter list 624 + * @dst_sg_len: Number of entries in destination scatter list 625 + * @src_sg: Source scatter list 626 + * @src_sg_len: Number of entries in source scatter list 627 + * @flags: transfer ack flags 628 + * 629 + * Return: Async transaction descriptor on success and NULL on failure 630 + */ 631 + static struct dma_async_tx_descriptor * 632 + mv_xor_prep_dma_sg(struct dma_chan *chan, struct scatterlist *dst_sg, 633 + unsigned int dst_sg_len, struct scatterlist *src_sg, 634 + unsigned int src_sg_len, unsigned long flags) 635 + { 636 + struct mv_xor_chan *mv_chan = to_mv_xor_chan(chan); 637 + struct mv_xor_desc_slot *new; 638 + struct mv_xor_desc_slot *first = NULL; 639 + struct mv_xor_desc_slot *prev = NULL; 640 + size_t len, dst_avail, src_avail; 641 + dma_addr_t dma_dst, dma_src; 642 + int desc_cnt = 0; 643 + int ret; 644 + 645 + dev_dbg(mv_chan_to_devp(mv_chan), 646 + "%s dst_sg_len: %d src_sg_len: %d flags: %ld\n", 647 + __func__, dst_sg_len, src_sg_len, flags); 648 + 649 + dst_avail = sg_dma_len(dst_sg); 650 + src_avail = sg_dma_len(src_sg); 651 + 652 + /* Run until we are out of scatterlist entries */ 653 + while (true) { 654 + /* Allocate and populate the descriptor */ 655 + desc_cnt++; 656 + new = mv_chan_alloc_slot(mv_chan); 657 + if (!new) { 658 + dev_err(mv_chan_to_devp(mv_chan), 659 + "Out of descriptors (desc_cnt=%d)!\n", 660 + desc_cnt); 661 + goto err; 662 + } 663 + 664 + len = min_t(size_t, src_avail, dst_avail); 665 + len = min_t(size_t, len, MV_XOR_MAX_BYTE_COUNT); 666 + if (len == 0) 667 + goto fetch; 668 + 669 + if (len < MV_XOR_MIN_BYTE_COUNT) { 670 + dev_err(mv_chan_to_devp(mv_chan), 671 + "Transfer size of %zu too small!\n", len); 672 + goto err; 673 + } 674 + 675 + dma_dst = sg_dma_address(dst_sg) + sg_dma_len(dst_sg) - 676 + dst_avail; 677 + dma_src = sg_dma_address(src_sg) + sg_dma_len(src_sg) - 678 + src_avail; 679 + 680 + /* Check if a new window needs to get added for 'dst' */ 681 + ret = mv_xor_add_io_win(mv_chan, dma_dst); 682 + if (ret) 683 + goto err; 684 + 685 + /* Check if a new window needs to get added for 'src' */ 686 + ret = mv_xor_add_io_win(mv_chan, dma_src); 687 + if (ret) 688 + goto err; 689 + 690 + /* Populate the descriptor */ 691 + mv_xor_config_sg_ll_desc(new, dma_src, dma_dst, len, prev); 692 + prev = new; 693 + dst_avail -= len; 694 + src_avail -= len; 695 + 696 + if (!first) 697 + first = new; 698 + else 699 + list_move_tail(&new->node, &first->sg_tx_list); 700 + 701 + fetch: 702 + /* Fetch the next dst scatterlist entry */ 703 + if (dst_avail == 0) { 704 + if (dst_sg_len == 0) 705 + break; 706 + 707 + /* Fetch the next entry: if there are no more: done */ 708 + dst_sg = sg_next(dst_sg); 709 + if (dst_sg == NULL) 710 + break; 711 + 712 + dst_sg_len--; 713 + dst_avail = sg_dma_len(dst_sg); 714 + } 715 + 716 + /* Fetch the next src scatterlist entry */ 717 + if (src_avail == 0) { 718 + if (src_sg_len == 0) 719 + break; 720 + 721 + /* Fetch the next entry: if there are no more: done */ 722 + src_sg = sg_next(src_sg); 723 + if (src_sg == NULL) 724 + break; 725 + 726 + src_sg_len--; 727 + src_avail = sg_dma_len(src_sg); 728 + } 729 + } 730 + 731 + /* Set the EOD flag in the last descriptor */ 732 + mv_xor_desc_config_eod(new); 733 + first->async_tx.flags = flags; 734 + 735 + return &first->async_tx; 736 + 737 + err: 738 + /* Cleanup: Move all descriptors back into the free list */ 739 + spin_lock_bh(&mv_chan->lock); 740 + mv_desc_clean_slot(first, mv_chan); 741 + spin_unlock_bh(&mv_chan->lock); 742 + 743 + return NULL; 663 744 } 664 745 665 746 static void mv_xor_free_chan_resources(struct dma_chan *chan) ··· 1254 1083 dma_dev->device_prep_dma_interrupt = mv_xor_prep_dma_interrupt; 1255 1084 if (dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask)) 1256 1085 dma_dev->device_prep_dma_memcpy = mv_xor_prep_dma_memcpy; 1086 + if (dma_has_cap(DMA_SG, dma_dev->cap_mask)) 1087 + dma_dev->device_prep_dma_sg = mv_xor_prep_dma_sg; 1257 1088 if (dma_has_cap(DMA_XOR, dma_dev->cap_mask)) { 1258 1089 dma_dev->max_xor = 8; 1259 1090 dma_dev->device_prep_dma_xor = mv_xor_prep_dma_xor; ··· 1305 1132 goto err_free_irq; 1306 1133 } 1307 1134 1308 - dev_info(&pdev->dev, "Marvell XOR (%s): ( %s%s%s)\n", 1135 + dev_info(&pdev->dev, "Marvell XOR (%s): ( %s%s%s%s)\n", 1309 1136 mv_chan->op_in_desc ? "Descriptor Mode" : "Registers Mode", 1310 1137 dma_has_cap(DMA_XOR, dma_dev->cap_mask) ? "xor " : "", 1311 1138 dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask) ? "cpy " : "", 1139 + dma_has_cap(DMA_SG, dma_dev->cap_mask) ? "sg " : "", 1312 1140 dma_has_cap(DMA_INTERRUPT, dma_dev->cap_mask) ? "intr " : ""); 1313 1141 1314 1142 dma_async_device_register(dma_dev); ··· 1552 1378 1553 1379 dma_cap_zero(cap_mask); 1554 1380 dma_cap_set(DMA_MEMCPY, cap_mask); 1381 + dma_cap_set(DMA_SG, cap_mask); 1555 1382 dma_cap_set(DMA_XOR, cap_mask); 1556 1383 dma_cap_set(DMA_INTERRUPT, cap_mask); 1557 1384 ··· 1630 1455 }, 1631 1456 }; 1632 1457 1633 - 1634 - static int __init mv_xor_init(void) 1635 - { 1636 - return platform_driver_register(&mv_xor_driver); 1637 - } 1638 - device_initcall(mv_xor_init); 1458 + builtin_platform_driver(mv_xor_driver); 1639 1459 1640 1460 /* 1641 1461 MODULE_AUTHOR("Saeed Bishara <saeed@marvell.com>");
+1
drivers/dma/mv_xor.h
··· 148 148 */ 149 149 struct mv_xor_desc_slot { 150 150 struct list_head node; 151 + struct list_head sg_tx_list; 151 152 enum dma_transaction_type type; 152 153 void *hw_desc; 153 154 u16 idx;
+34 -4
drivers/dma/nbpfaxi.c
··· 225 225 struct nbpf_device { 226 226 struct dma_device dma_dev; 227 227 void __iomem *base; 228 + u32 max_burst_mem_read; 229 + u32 max_burst_mem_write; 228 230 struct clk *clk; 229 231 const struct nbpf_config *config; 230 232 unsigned int eirq; ··· 427 425 nbpf_chan_write(chan, NBPF_CHAN_CFG, NBPF_CHAN_CFG_DMS | chan->dmarq_cfg); 428 426 } 429 427 430 - static u32 nbpf_xfer_ds(struct nbpf_device *nbpf, size_t size) 428 + static u32 nbpf_xfer_ds(struct nbpf_device *nbpf, size_t size, 429 + enum dma_transfer_direction direction) 431 430 { 431 + int max_burst = nbpf->config->buffer_size * 8; 432 + 433 + if (nbpf->max_burst_mem_read || nbpf->max_burst_mem_write) { 434 + switch (direction) { 435 + case DMA_MEM_TO_MEM: 436 + max_burst = min_not_zero(nbpf->max_burst_mem_read, 437 + nbpf->max_burst_mem_write); 438 + break; 439 + case DMA_MEM_TO_DEV: 440 + if (nbpf->max_burst_mem_read) 441 + max_burst = nbpf->max_burst_mem_read; 442 + break; 443 + case DMA_DEV_TO_MEM: 444 + if (nbpf->max_burst_mem_write) 445 + max_burst = nbpf->max_burst_mem_write; 446 + break; 447 + case DMA_DEV_TO_DEV: 448 + default: 449 + break; 450 + } 451 + } 452 + 432 453 /* Maximum supported bursts depend on the buffer size */ 433 - return min_t(int, __ffs(size), ilog2(nbpf->config->buffer_size * 8)); 454 + return min_t(int, __ffs(size), ilog2(max_burst)); 434 455 } 435 456 436 457 static size_t nbpf_xfer_size(struct nbpf_device *nbpf, ··· 483 458 size = burst; 484 459 } 485 460 486 - return nbpf_xfer_ds(nbpf, size); 461 + return nbpf_xfer_ds(nbpf, size, DMA_TRANS_NONE); 487 462 } 488 463 489 464 /* ··· 532 507 * transfers we enable the SBE bit and terminate the transfer in our 533 508 * .device_pause handler. 534 509 */ 535 - mem_xfer = nbpf_xfer_ds(chan->nbpf, size); 510 + mem_xfer = nbpf_xfer_ds(chan->nbpf, size, direction); 536 511 537 512 switch (direction) { 538 513 case DMA_DEV_TO_MEM: ··· 1337 1312 nbpf->clk = devm_clk_get(dev, NULL); 1338 1313 if (IS_ERR(nbpf->clk)) 1339 1314 return PTR_ERR(nbpf->clk); 1315 + 1316 + of_property_read_u32(np, "max-burst-mem-read", 1317 + &nbpf->max_burst_mem_read); 1318 + of_property_read_u32(np, "max-burst-mem-write", 1319 + &nbpf->max_burst_mem_write); 1340 1320 1341 1321 nbpf->config = cfg; 1342 1322
+150 -37
drivers/dma/omap-dma.c
··· 166 166 CSDP_DST_BURST_16 = 1 << 14, 167 167 CSDP_DST_BURST_32 = 2 << 14, 168 168 CSDP_DST_BURST_64 = 3 << 14, 169 + CSDP_WRITE_NON_POSTED = 0 << 16, 170 + CSDP_WRITE_POSTED = 1 << 16, 171 + CSDP_WRITE_LAST_NON_POSTED = 2 << 16, 169 172 170 173 CICR_TOUT_IE = BIT(0), /* OMAP1 only */ 171 174 CICR_DROP_IE = BIT(1), ··· 425 422 c->running = true; 426 423 } 427 424 428 - static void omap_dma_stop(struct omap_chan *c) 425 + static void omap_dma_drain_chan(struct omap_chan *c) 426 + { 427 + int i; 428 + u32 val; 429 + 430 + /* Wait for sDMA FIFO to drain */ 431 + for (i = 0; ; i++) { 432 + val = omap_dma_chan_read(c, CCR); 433 + if (!(val & (CCR_RD_ACTIVE | CCR_WR_ACTIVE))) 434 + break; 435 + 436 + if (i > 100) 437 + break; 438 + 439 + udelay(5); 440 + } 441 + 442 + if (val & (CCR_RD_ACTIVE | CCR_WR_ACTIVE)) 443 + dev_err(c->vc.chan.device->dev, 444 + "DMA drain did not complete on lch %d\n", 445 + c->dma_ch); 446 + } 447 + 448 + static int omap_dma_stop(struct omap_chan *c) 429 449 { 430 450 struct omap_dmadev *od = to_omap_dma_dev(c->vc.chan.device); 431 451 uint32_t val; ··· 461 435 val = omap_dma_chan_read(c, CCR); 462 436 if (od->plat->errata & DMA_ERRATA_i541 && val & CCR_TRIGGER_SRC) { 463 437 uint32_t sysconfig; 464 - unsigned i; 465 438 466 439 sysconfig = omap_dma_glbl_read(od, OCP_SYSCONFIG); 467 440 val = sysconfig & ~DMA_SYSCONFIG_MIDLEMODE_MASK; ··· 471 446 val &= ~CCR_ENABLE; 472 447 omap_dma_chan_write(c, CCR, val); 473 448 474 - /* Wait for sDMA FIFO to drain */ 475 - for (i = 0; ; i++) { 476 - val = omap_dma_chan_read(c, CCR); 477 - if (!(val & (CCR_RD_ACTIVE | CCR_WR_ACTIVE))) 478 - break; 479 - 480 - if (i > 100) 481 - break; 482 - 483 - udelay(5); 484 - } 485 - 486 - if (val & (CCR_RD_ACTIVE | CCR_WR_ACTIVE)) 487 - dev_err(c->vc.chan.device->dev, 488 - "DMA drain did not complete on lch %d\n", 489 - c->dma_ch); 449 + if (!(c->ccr & CCR_BUFFERING_DISABLE)) 450 + omap_dma_drain_chan(c); 490 451 491 452 omap_dma_glbl_write(od, OCP_SYSCONFIG, sysconfig); 492 453 } else { 454 + if (!(val & CCR_ENABLE)) 455 + return -EINVAL; 456 + 493 457 val &= ~CCR_ENABLE; 494 458 omap_dma_chan_write(c, CCR, val); 459 + 460 + if (!(c->ccr & CCR_BUFFERING_DISABLE)) 461 + omap_dma_drain_chan(c); 495 462 } 496 463 497 464 mb(); ··· 498 481 499 482 omap_dma_chan_write(c, CLNK_CTRL, val); 500 483 } 501 - 502 484 c->running = false; 485 + return 0; 503 486 } 504 487 505 488 static void omap_dma_start_sg(struct omap_chan *c, struct omap_desc *d) ··· 853 836 } else { 854 837 txstate->residue = 0; 855 838 } 839 + if (ret == DMA_IN_PROGRESS && c->paused) 840 + ret = DMA_PAUSED; 856 841 spin_unlock_irqrestore(&c->vc.lock, flags); 857 842 858 843 return ret; ··· 884 865 unsigned i, es, en, frame_bytes; 885 866 bool ll_failed = false; 886 867 u32 burst; 868 + u32 port_window, port_window_bytes; 887 869 888 870 if (dir == DMA_DEV_TO_MEM) { 889 871 dev_addr = c->cfg.src_addr; 890 872 dev_width = c->cfg.src_addr_width; 891 873 burst = c->cfg.src_maxburst; 874 + port_window = c->cfg.src_port_window_size; 892 875 } else if (dir == DMA_MEM_TO_DEV) { 893 876 dev_addr = c->cfg.dst_addr; 894 877 dev_width = c->cfg.dst_addr_width; 895 878 burst = c->cfg.dst_maxburst; 879 + port_window = c->cfg.dst_port_window_size; 896 880 } else { 897 881 dev_err(chan->device->dev, "%s: bad direction?\n", __func__); 898 882 return NULL; ··· 916 894 return NULL; 917 895 } 918 896 897 + /* When the port_window is used, one frame must cover the window */ 898 + if (port_window) { 899 + burst = port_window; 900 + port_window_bytes = port_window * es_bytes[es]; 901 + } 902 + 919 903 /* Now allocate and setup the descriptor. */ 920 904 d = kzalloc(sizeof(*d) + sglen * sizeof(d->sg[0]), GFP_ATOMIC); 921 905 if (!d) ··· 933 905 934 906 d->ccr = c->ccr | CCR_SYNC_FRAME; 935 907 if (dir == DMA_DEV_TO_MEM) { 936 - d->ccr |= CCR_DST_AMODE_POSTINC | CCR_SRC_AMODE_CONSTANT; 937 908 d->csdp = CSDP_DST_BURST_64 | CSDP_DST_PACKED; 909 + 910 + d->ccr |= CCR_DST_AMODE_POSTINC; 911 + if (port_window) { 912 + d->ccr |= CCR_SRC_AMODE_DBLIDX; 913 + d->ei = 1; 914 + /* 915 + * One frame covers the port_window and by configure 916 + * the source frame index to be -1 * (port_window - 1) 917 + * we instruct the sDMA that after a frame is processed 918 + * it should move back to the start of the window. 919 + */ 920 + d->fi = -(port_window_bytes - 1); 921 + 922 + if (port_window_bytes >= 64) 923 + d->csdp = CSDP_SRC_BURST_64 | CSDP_SRC_PACKED; 924 + else if (port_window_bytes >= 32) 925 + d->csdp = CSDP_SRC_BURST_32 | CSDP_SRC_PACKED; 926 + else if (port_window_bytes >= 16) 927 + d->csdp = CSDP_SRC_BURST_16 | CSDP_SRC_PACKED; 928 + } else { 929 + d->ccr |= CCR_SRC_AMODE_CONSTANT; 930 + } 938 931 } else { 939 - d->ccr |= CCR_DST_AMODE_CONSTANT | CCR_SRC_AMODE_POSTINC; 940 932 d->csdp = CSDP_SRC_BURST_64 | CSDP_SRC_PACKED; 933 + 934 + d->ccr |= CCR_SRC_AMODE_POSTINC; 935 + if (port_window) { 936 + d->ccr |= CCR_DST_AMODE_DBLIDX; 937 + 938 + if (port_window_bytes >= 64) 939 + d->csdp = CSDP_DST_BURST_64 | CSDP_DST_PACKED; 940 + else if (port_window_bytes >= 32) 941 + d->csdp = CSDP_DST_BURST_32 | CSDP_DST_PACKED; 942 + else if (port_window_bytes >= 16) 943 + d->csdp = CSDP_DST_BURST_16 | CSDP_DST_PACKED; 944 + } else { 945 + d->ccr |= CCR_DST_AMODE_CONSTANT; 946 + } 941 947 } 942 948 943 949 d->cicr = CICR_DROP_IE | CICR_BLOCK_IE; ··· 989 927 d->ccr |= CCR_TRIGGER_SRC; 990 928 991 929 d->cicr |= CICR_MISALIGNED_ERR_IE | CICR_TRANS_ERR_IE; 930 + 931 + if (port_window) 932 + d->csdp |= CSDP_WRITE_LAST_NON_POSTED; 992 933 } 993 934 if (od->plat->errata & DMA_ERRATA_PARALLEL_CHANNELS) 994 935 d->clnk_ctrl = c->dma_ch; ··· 1017 952 osg->addr = sg_dma_address(sgent); 1018 953 osg->en = en; 1019 954 osg->fn = sg_dma_len(sgent) / frame_bytes; 955 + if (port_window && dir == DMA_MEM_TO_DEV) { 956 + osg->ei = 1; 957 + /* 958 + * One frame covers the port_window and by configure 959 + * the source frame index to be -1 * (port_window - 1) 960 + * we instruct the sDMA that after a frame is processed 961 + * it should move back to the start of the window. 962 + */ 963 + osg->fi = -(port_window_bytes - 1); 964 + } 1020 965 1021 966 if (d->using_ll) { 1022 967 osg->t2_desc = dma_pool_alloc(od->desc_pool, GFP_ATOMIC, ··· 1322 1247 omap_dma_stop(c); 1323 1248 } 1324 1249 1325 - if (c->cyclic) { 1326 - c->cyclic = false; 1327 - c->paused = false; 1328 - } 1250 + c->cyclic = false; 1251 + c->paused = false; 1329 1252 1330 1253 vchan_get_all_descriptors(&c->vc, &head); 1331 1254 spin_unlock_irqrestore(&c->vc.lock, flags); ··· 1342 1269 static int omap_dma_pause(struct dma_chan *chan) 1343 1270 { 1344 1271 struct omap_chan *c = to_omap_dma_chan(chan); 1272 + struct omap_dmadev *od = to_omap_dma_dev(chan->device); 1273 + unsigned long flags; 1274 + int ret = -EINVAL; 1275 + bool can_pause = false; 1345 1276 1346 - /* Pause/Resume only allowed with cyclic mode */ 1347 - if (!c->cyclic) 1348 - return -EINVAL; 1277 + spin_lock_irqsave(&od->irq_lock, flags); 1349 1278 1350 - if (!c->paused) { 1351 - omap_dma_stop(c); 1352 - c->paused = true; 1279 + if (!c->desc) 1280 + goto out; 1281 + 1282 + if (c->cyclic) 1283 + can_pause = true; 1284 + 1285 + /* 1286 + * We do not allow DMA_MEM_TO_DEV transfers to be paused. 1287 + * From the AM572x TRM, 16.1.4.18 Disabling a Channel During Transfer: 1288 + * "When a channel is disabled during a transfer, the channel undergoes 1289 + * an abort, unless it is hardware-source-synchronized …". 1290 + * A source-synchronised channel is one where the fetching of data is 1291 + * under control of the device. In other words, a device-to-memory 1292 + * transfer. So, a destination-synchronised channel (which would be a 1293 + * memory-to-device transfer) undergoes an abort if the the CCR_ENABLE 1294 + * bit is cleared. 1295 + * From 16.1.4.20.4.6.2 Abort: "If an abort trigger occurs, the channel 1296 + * aborts immediately after completion of current read/write 1297 + * transactions and then the FIFO is cleaned up." The term "cleaned up" 1298 + * is not defined. TI recommends to check that RD_ACTIVE and WR_ACTIVE 1299 + * are both clear _before_ disabling the channel, otherwise data loss 1300 + * will occur. 1301 + * The problem is that if the channel is active, then device activity 1302 + * can result in DMA activity starting between reading those as both 1303 + * clear and the write to DMA_CCR to clear the enable bit hitting the 1304 + * hardware. If the DMA hardware can't drain the data in its FIFO to the 1305 + * destination, then data loss "might" occur (say if we write to an UART 1306 + * and the UART is not accepting any further data). 1307 + */ 1308 + else if (c->desc->dir == DMA_DEV_TO_MEM) 1309 + can_pause = true; 1310 + 1311 + if (can_pause && !c->paused) { 1312 + ret = omap_dma_stop(c); 1313 + if (!ret) 1314 + c->paused = true; 1353 1315 } 1316 + out: 1317 + spin_unlock_irqrestore(&od->irq_lock, flags); 1354 1318 1355 - return 0; 1319 + return ret; 1356 1320 } 1357 1321 1358 1322 static int omap_dma_resume(struct dma_chan *chan) 1359 1323 { 1360 1324 struct omap_chan *c = to_omap_dma_chan(chan); 1325 + struct omap_dmadev *od = to_omap_dma_dev(chan->device); 1326 + unsigned long flags; 1327 + int ret = -EINVAL; 1361 1328 1362 - /* Pause/Resume only allowed with cyclic mode */ 1363 - if (!c->cyclic) 1364 - return -EINVAL; 1329 + spin_lock_irqsave(&od->irq_lock, flags); 1365 1330 1366 - if (c->paused) { 1331 + if (c->paused && c->desc) { 1367 1332 mb(); 1368 1333 1369 1334 /* Restore channel link register */ ··· 1409 1298 1410 1299 omap_dma_start(c, c->desc); 1411 1300 c->paused = false; 1301 + ret = 0; 1412 1302 } 1303 + spin_unlock_irqrestore(&od->irq_lock, flags); 1413 1304 1414 - return 0; 1305 + return ret; 1415 1306 } 1416 1307 1417 1308 static int omap_dma_chan_init(struct omap_dmadev *od)
+1 -4
drivers/dma/pch_dma.c
··· 417 417 { 418 418 struct pch_dma_desc *desc = to_pd_desc(txd); 419 419 struct pch_dma_chan *pd_chan = to_pd_chan(txd->chan); 420 - dma_cookie_t cookie; 421 420 422 421 spin_lock(&pd_chan->lock); 423 - cookie = dma_cookie_assign(txd); 424 422 425 423 if (list_empty(&pd_chan->active_list)) { 426 424 list_add_tail(&desc->desc_node, &pd_chan->active_list); ··· 437 439 struct pch_dma *pd = to_pd(chan->device); 438 440 dma_addr_t addr; 439 441 440 - desc = pci_pool_alloc(pd->pool, flags, &addr); 442 + desc = pci_pool_zalloc(pd->pool, flags, &addr); 441 443 if (desc) { 442 - memset(desc, 0, sizeof(struct pch_dma_desc)); 443 444 INIT_LIST_HEAD(&desc->tx_list); 444 445 dma_async_tx_descriptor_init(&desc->txd, chan); 445 446 desc->txd.tx_submit = pd_tx_submit;
+15 -8
drivers/dma/pl330.c
··· 570 570 571 571 buf[0] = CMD_DMAADDH; 572 572 buf[0] |= (da << 1); 573 - *((__le16 *)&buf[1]) = cpu_to_le16(val); 573 + buf[1] = val; 574 + buf[2] = val >> 8; 574 575 575 576 PL330_DBGCMD_DUMP(SZ_DMAADDH, "\tDMAADDH %s %u\n", 576 577 da == 1 ? "DA" : "SA", val); ··· 725 724 726 725 buf[0] = CMD_DMAMOV; 727 726 buf[1] = dst; 728 - *((__le32 *)&buf[2]) = cpu_to_le32(val); 727 + buf[2] = val; 728 + buf[3] = val >> 8; 729 + buf[4] = val >> 16; 730 + buf[5] = val >> 24; 729 731 730 732 PL330_DBGCMD_DUMP(SZ_DMAMOV, "\tDMAMOV %s 0x%x\n", 731 733 dst == SAR ? "SAR" : (dst == DAR ? "DAR" : "CCR"), val); ··· 903 899 904 900 buf[0] = CMD_DMAGO; 905 901 buf[0] |= (ns << 1); 906 - 907 902 buf[1] = chan & 0x7; 908 - 909 - *((__le32 *)&buf[2]) = cpu_to_le32(addr); 903 + buf[2] = addr; 904 + buf[3] = addr >> 8; 905 + buf[4] = addr >> 16; 906 + buf[5] = addr >> 24; 910 907 911 908 return SZ_DMAGO; 912 909 } ··· 1888 1883 1889 1884 static int pl330_add(struct pl330_dmac *pl330) 1890 1885 { 1891 - void __iomem *regs; 1892 1886 int i, ret; 1893 - 1894 - regs = pl330->base; 1895 1887 1896 1888 /* Check if we can handle this DMAC */ 1897 1889 if ((pl330->pcfg.periph_id & 0xfffff) != PERIPH_ID_VAL) { ··· 2265 2263 } 2266 2264 pm_runtime_mark_last_busy(pch->dmac->ddma.dev); 2267 2265 pm_runtime_put_autosuspend(pl330->ddma.dev); 2266 + 2267 + /* If DMAMOV hasn't finished yet, SAR/DAR can be zero */ 2268 + if (!val) 2269 + return 0; 2270 + 2268 2271 return val - addr; 2269 2272 } 2270 2273
-28
drivers/dma/pxa_dma.c
··· 413 413 static inline void pxad_cleanup_debugfs(struct pxad_device *pdev) {} 414 414 #endif 415 415 416 - /* 417 - * In the transition phase where legacy pxa handling is done at the same time as 418 - * mmp_dma, the DMA physical channel split between the 2 DMA providers is done 419 - * through legacy_reserved. Legacy code reserves DMA channels by settings 420 - * corresponding bits in legacy_reserved. 421 - */ 422 - static u32 legacy_reserved; 423 - static u32 legacy_unavailable; 424 - 425 416 static struct pxad_phy *lookup_phy(struct pxad_chan *pchan) 426 417 { 427 418 int prio, i; ··· 433 442 for (i = 0; i < pdev->nr_chans; i++) { 434 443 if (prio != (i & 0xf) >> 2) 435 444 continue; 436 - if ((i < 32) && (legacy_reserved & BIT(i))) 437 - continue; 438 445 phy = &pdev->phys[i]; 439 446 if (!phy->vchan) { 440 447 phy->vchan = pchan; 441 448 found = phy; 442 - if (i < 32) 443 - legacy_unavailable |= BIT(i); 444 449 goto out_unlock; 445 450 } 446 451 } ··· 456 469 struct pxad_device *pdev = to_pxad_dev(chan->vc.chan.device); 457 470 unsigned long flags; 458 471 u32 reg; 459 - int i; 460 472 461 473 dev_dbg(&chan->vc.chan.dev->device, 462 474 "%s(): freeing\n", __func__); ··· 469 483 } 470 484 471 485 spin_lock_irqsave(&pdev->phy_lock, flags); 472 - for (i = 0; i < 32; i++) 473 - if (chan->phy == &pdev->phys[i]) 474 - legacy_unavailable &= ~BIT(i); 475 486 chan->phy->vchan = NULL; 476 487 chan->phy = NULL; 477 488 spin_unlock_irqrestore(&pdev->phy_lock, flags); ··· 722 739 i = __ffs(dint); 723 740 dint &= (dint - 1); 724 741 phy = &pdev->phys[i]; 725 - if ((i < 32) && (legacy_reserved & BIT(i))) 726 - continue; 727 742 if (pxad_chan_handler(irq, phy) == IRQ_HANDLED) 728 743 ret = IRQ_HANDLED; 729 744 } ··· 1502 1521 return true; 1503 1522 } 1504 1523 EXPORT_SYMBOL_GPL(pxad_filter_fn); 1505 - 1506 - int pxad_toggle_reserved_channel(int legacy_channel) 1507 - { 1508 - if (legacy_unavailable & (BIT(legacy_channel))) 1509 - return -EBUSY; 1510 - legacy_reserved ^= BIT(legacy_channel); 1511 - return 0; 1512 - } 1513 - EXPORT_SYMBOL_GPL(pxad_toggle_reserved_channel); 1514 1524 1515 1525 module_platform_driver(pxad_driver); 1516 1526
+161 -12
drivers/dma/qcom/hidma.c
··· 56 56 #include <linux/irq.h> 57 57 #include <linux/atomic.h> 58 58 #include <linux/pm_runtime.h> 59 + #include <linux/msi.h> 59 60 60 61 #include "../dmaengine.h" 61 62 #include "hidma.h" ··· 71 70 #define HIDMA_ERR_INFO_SW 0xFF 72 71 #define HIDMA_ERR_CODE_UNEXPECTED_TERMINATE 0x0 73 72 #define HIDMA_NR_DEFAULT_DESC 10 73 + #define HIDMA_MSI_INTS 11 74 74 75 75 static inline struct hidma_dev *to_hidma_dev(struct dma_device *dmadev) 76 76 { ··· 555 553 return hidma_ll_inthandler(chirq, lldev); 556 554 } 557 555 556 + #ifdef CONFIG_GENERIC_MSI_IRQ_DOMAIN 557 + static irqreturn_t hidma_chirq_handler_msi(int chirq, void *arg) 558 + { 559 + struct hidma_lldev **lldevp = arg; 560 + struct hidma_dev *dmadev = to_hidma_dev_from_lldev(lldevp); 561 + 562 + return hidma_ll_inthandler_msi(chirq, *lldevp, 563 + 1 << (chirq - dmadev->msi_virqbase)); 564 + } 565 + #endif 566 + 558 567 static ssize_t hidma_show_values(struct device *dev, 559 568 struct device_attribute *attr, char *buf) 560 569 { ··· 580 567 return strlen(buf); 581 568 } 582 569 583 - static int hidma_create_sysfs_entry(struct hidma_dev *dev, char *name, 584 - int mode) 570 + static inline void hidma_sysfs_uninit(struct hidma_dev *dev) 571 + { 572 + device_remove_file(dev->ddev.dev, dev->chid_attrs); 573 + } 574 + 575 + static struct device_attribute* 576 + hidma_create_sysfs_entry(struct hidma_dev *dev, char *name, int mode) 585 577 { 586 578 struct device_attribute *attrs; 587 579 char *name_copy; ··· 594 576 attrs = devm_kmalloc(dev->ddev.dev, sizeof(struct device_attribute), 595 577 GFP_KERNEL); 596 578 if (!attrs) 597 - return -ENOMEM; 579 + return NULL; 598 580 599 581 name_copy = devm_kstrdup(dev->ddev.dev, name, GFP_KERNEL); 600 582 if (!name_copy) 601 - return -ENOMEM; 583 + return NULL; 602 584 603 585 attrs->attr.name = name_copy; 604 586 attrs->attr.mode = mode; 605 587 attrs->show = hidma_show_values; 606 588 sysfs_attr_init(&attrs->attr); 607 589 608 - return device_create_file(dev->ddev.dev, attrs); 590 + return attrs; 591 + } 592 + 593 + static int hidma_sysfs_init(struct hidma_dev *dev) 594 + { 595 + dev->chid_attrs = hidma_create_sysfs_entry(dev, "chid", S_IRUGO); 596 + if (!dev->chid_attrs) 597 + return -ENOMEM; 598 + 599 + return device_create_file(dev->ddev.dev, dev->chid_attrs); 600 + } 601 + 602 + #ifdef CONFIG_GENERIC_MSI_IRQ_DOMAIN 603 + static void hidma_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg) 604 + { 605 + struct device *dev = msi_desc_to_dev(desc); 606 + struct hidma_dev *dmadev = dev_get_drvdata(dev); 607 + 608 + if (!desc->platform.msi_index) { 609 + writel(msg->address_lo, dmadev->dev_evca + 0x118); 610 + writel(msg->address_hi, dmadev->dev_evca + 0x11C); 611 + writel(msg->data, dmadev->dev_evca + 0x120); 612 + } 613 + } 614 + #endif 615 + 616 + static void hidma_free_msis(struct hidma_dev *dmadev) 617 + { 618 + #ifdef CONFIG_GENERIC_MSI_IRQ_DOMAIN 619 + struct device *dev = dmadev->ddev.dev; 620 + struct msi_desc *desc; 621 + 622 + /* free allocated MSI interrupts above */ 623 + for_each_msi_entry(desc, dev) 624 + devm_free_irq(dev, desc->irq, &dmadev->lldev); 625 + 626 + platform_msi_domain_free_irqs(dev); 627 + #endif 628 + } 629 + 630 + static int hidma_request_msi(struct hidma_dev *dmadev, 631 + struct platform_device *pdev) 632 + { 633 + #ifdef CONFIG_GENERIC_MSI_IRQ_DOMAIN 634 + int rc; 635 + struct msi_desc *desc; 636 + struct msi_desc *failed_desc = NULL; 637 + 638 + rc = platform_msi_domain_alloc_irqs(&pdev->dev, HIDMA_MSI_INTS, 639 + hidma_write_msi_msg); 640 + if (rc) 641 + return rc; 642 + 643 + for_each_msi_entry(desc, &pdev->dev) { 644 + if (!desc->platform.msi_index) 645 + dmadev->msi_virqbase = desc->irq; 646 + 647 + rc = devm_request_irq(&pdev->dev, desc->irq, 648 + hidma_chirq_handler_msi, 649 + 0, "qcom-hidma-msi", 650 + &dmadev->lldev); 651 + if (rc) { 652 + failed_desc = desc; 653 + break; 654 + } 655 + } 656 + 657 + if (rc) { 658 + /* free allocated MSI interrupts above */ 659 + for_each_msi_entry(desc, &pdev->dev) { 660 + if (desc == failed_desc) 661 + break; 662 + devm_free_irq(&pdev->dev, desc->irq, 663 + &dmadev->lldev); 664 + } 665 + } else { 666 + /* Add callback to free MSIs on teardown */ 667 + hidma_ll_setup_irq(dmadev->lldev, true); 668 + 669 + } 670 + if (rc) 671 + dev_warn(&pdev->dev, 672 + "failed to request MSI irq, falling back to wired IRQ\n"); 673 + return rc; 674 + #else 675 + return -EINVAL; 676 + #endif 677 + } 678 + 679 + static bool hidma_msi_capable(struct device *dev) 680 + { 681 + struct acpi_device *adev = ACPI_COMPANION(dev); 682 + const char *of_compat; 683 + int ret = -EINVAL; 684 + 685 + if (!adev || acpi_disabled) { 686 + ret = device_property_read_string(dev, "compatible", 687 + &of_compat); 688 + if (ret) 689 + return false; 690 + 691 + ret = strcmp(of_compat, "qcom,hidma-1.1"); 692 + } else { 693 + #ifdef CONFIG_ACPI 694 + ret = strcmp(acpi_device_hid(adev), "QCOM8062"); 695 + #endif 696 + } 697 + return ret == 0; 609 698 } 610 699 611 700 static int hidma_probe(struct platform_device *pdev) ··· 724 599 void __iomem *evca; 725 600 void __iomem *trca; 726 601 int rc; 602 + bool msi; 727 603 728 604 pm_runtime_set_autosuspend_delay(&pdev->dev, HIDMA_AUTOSUSPEND_TIMEOUT); 729 605 pm_runtime_use_autosuspend(&pdev->dev); ··· 786 660 dmadev->ddev.device_terminate_all = hidma_terminate_all; 787 661 dmadev->ddev.copy_align = 8; 788 662 663 + /* 664 + * Determine the MSI capability of the platform. Old HW doesn't 665 + * support MSI. 666 + */ 667 + msi = hidma_msi_capable(&pdev->dev); 668 + 789 669 device_property_read_u32(&pdev->dev, "desc-count", 790 670 &dmadev->nr_descriptors); 791 671 ··· 820 688 goto dmafree; 821 689 } 822 690 823 - rc = devm_request_irq(&pdev->dev, chirq, hidma_chirq_handler, 0, 824 - "qcom-hidma", dmadev->lldev); 825 - if (rc) 826 - goto uninit; 691 + platform_set_drvdata(pdev, dmadev); 692 + if (msi) 693 + rc = hidma_request_msi(dmadev, pdev); 694 + 695 + if (!msi || rc) { 696 + hidma_ll_setup_irq(dmadev->lldev, false); 697 + rc = devm_request_irq(&pdev->dev, chirq, hidma_chirq_handler, 698 + 0, "qcom-hidma", dmadev->lldev); 699 + if (rc) 700 + goto uninit; 701 + } 827 702 828 703 INIT_LIST_HEAD(&dmadev->ddev.channels); 829 704 rc = hidma_chan_init(dmadev, 0); ··· 844 705 dmadev->irq = chirq; 845 706 tasklet_init(&dmadev->task, hidma_issue_task, (unsigned long)dmadev); 846 707 hidma_debug_init(dmadev); 847 - hidma_create_sysfs_entry(dmadev, "chid", S_IRUGO); 708 + hidma_sysfs_init(dmadev); 848 709 dev_info(&pdev->dev, "HI-DMA engine driver registration complete\n"); 849 - platform_set_drvdata(pdev, dmadev); 850 710 pm_runtime_mark_last_busy(dmadev->ddev.dev); 851 711 pm_runtime_put_autosuspend(dmadev->ddev.dev); 852 712 return 0; 853 713 854 714 uninit: 715 + if (msi) 716 + hidma_free_msis(dmadev); 717 + 855 718 hidma_debug_uninit(dmadev); 856 719 hidma_ll_uninit(dmadev->lldev); 857 720 dmafree: ··· 871 730 872 731 pm_runtime_get_sync(dmadev->ddev.dev); 873 732 dma_async_device_unregister(&dmadev->ddev); 874 - devm_free_irq(dmadev->ddev.dev, dmadev->irq, dmadev->lldev); 733 + if (!dmadev->lldev->msi_support) 734 + devm_free_irq(dmadev->ddev.dev, dmadev->irq, dmadev->lldev); 735 + else 736 + hidma_free_msis(dmadev); 737 + 875 738 tasklet_kill(&dmadev->task); 739 + hidma_sysfs_uninit(dmadev); 876 740 hidma_debug_uninit(dmadev); 877 741 hidma_ll_uninit(dmadev->lldev); 878 742 hidma_free(dmadev); ··· 892 746 #if IS_ENABLED(CONFIG_ACPI) 893 747 static const struct acpi_device_id hidma_acpi_ids[] = { 894 748 {"QCOM8061"}, 749 + {"QCOM8062"}, 895 750 {}, 896 751 }; 752 + MODULE_DEVICE_TABLE(acpi, hidma_acpi_ids); 897 753 #endif 898 754 899 755 static const struct of_device_id hidma_match[] = { 900 756 {.compatible = "qcom,hidma-1.0",}, 757 + {.compatible = "qcom,hidma-1.1",}, 901 758 {}, 902 759 }; 903 760 MODULE_DEVICE_TABLE(of, hidma_match);
+8 -1
drivers/dma/qcom/hidma.h
··· 46 46 }; 47 47 48 48 struct hidma_lldev { 49 + bool msi_support; /* flag indicating MSI support */ 49 50 bool initialized; /* initialized flag */ 50 51 u8 trch_state; /* trch_state of the device */ 51 52 u8 evch_state; /* evch_state of the device */ ··· 59 58 void __iomem *evca; /* Event Channel address */ 60 59 struct hidma_tre 61 60 **pending_tre_list; /* Pointers to pending TREs */ 62 - s32 pending_tre_count; /* Number of TREs pending */ 61 + atomic_t pending_tre_count; /* Number of TREs pending */ 63 62 64 63 void *tre_ring; /* TRE ring */ 65 64 dma_addr_t tre_dma; /* TRE ring to be shared with HW */ ··· 115 114 int irq; 116 115 int chidx; 117 116 u32 nr_descriptors; 117 + int msi_virqbase; 118 118 119 119 struct hidma_lldev *lldev; 120 120 void __iomem *dev_trca; ··· 129 127 130 128 struct dentry *debugfs; 131 129 struct dentry *stats; 130 + 131 + /* sysfs entry for the channel id */ 132 + struct device_attribute *chid_attrs; 132 133 133 134 /* Task delivering issue_pending */ 134 135 struct tasklet_struct task; ··· 150 145 int hidma_ll_enable(struct hidma_lldev *llhndl); 151 146 void hidma_ll_set_transfer_params(struct hidma_lldev *llhndl, u32 tre_ch, 152 147 dma_addr_t src, dma_addr_t dest, u32 len, u32 flags); 148 + void hidma_ll_setup_irq(struct hidma_lldev *lldev, bool msi); 153 149 int hidma_ll_setup(struct hidma_lldev *lldev); 154 150 struct hidma_lldev *hidma_ll_init(struct device *dev, u32 max_channels, 155 151 void __iomem *trca, void __iomem *evca, 156 152 u8 chidx); 157 153 int hidma_ll_uninit(struct hidma_lldev *llhndl); 158 154 irqreturn_t hidma_ll_inthandler(int irq, void *arg); 155 + irqreturn_t hidma_ll_inthandler_msi(int irq, void *arg, int cause); 159 156 void hidma_cleanup_pending_tre(struct hidma_lldev *llhndl, u8 err_info, 160 157 u8 err_code); 161 158 int hidma_debug_init(struct hidma_dev *dmadev);
+2 -2
drivers/dma/qcom/hidma_dbg.c
··· 74 74 seq_printf(s, "tre_ring_handle=%pap\n", &lldev->tre_dma); 75 75 seq_printf(s, "tre_ring_size = 0x%x\n", lldev->tre_ring_size); 76 76 seq_printf(s, "tre_processed_off = 0x%x\n", lldev->tre_processed_off); 77 - seq_printf(s, "pending_tre_count=%d\n", lldev->pending_tre_count); 77 + seq_printf(s, "pending_tre_count=%d\n", 78 + atomic_read(&lldev->pending_tre_count)); 78 79 seq_printf(s, "evca=%p\n", lldev->evca); 79 80 seq_printf(s, "evre_ring=%p\n", lldev->evre_ring); 80 81 seq_printf(s, "evre_ring_handle=%pap\n", &lldev->evre_dma); ··· 165 164 void hidma_debug_uninit(struct hidma_dev *dmadev) 166 165 { 167 166 debugfs_remove_recursive(dmadev->debugfs); 168 - debugfs_remove_recursive(dmadev->stats); 169 167 } 170 168 171 169 int hidma_debug_init(struct hidma_dev *dmadev)
+90 -86
drivers/dma/qcom/hidma_ll.c
··· 198 198 } 199 199 } 200 200 201 - static int hidma_post_completed(struct hidma_lldev *lldev, int tre_iterator, 202 - u8 err_info, u8 err_code) 201 + static int hidma_post_completed(struct hidma_lldev *lldev, u8 err_info, 202 + u8 err_code) 203 203 { 204 204 struct hidma_tre *tre; 205 205 unsigned long flags; 206 + u32 tre_iterator; 206 207 207 208 spin_lock_irqsave(&lldev->lock, flags); 209 + 210 + tre_iterator = lldev->tre_processed_off; 208 211 tre = lldev->pending_tre_list[tre_iterator / HIDMA_TRE_SIZE]; 209 212 if (!tre) { 210 213 spin_unlock_irqrestore(&lldev->lock, flags); ··· 221 218 * Keep track of pending TREs that SW is expecting to receive 222 219 * from HW. We got one now. Decrement our counter. 223 220 */ 224 - lldev->pending_tre_count--; 225 - if (lldev->pending_tre_count < 0) { 221 + if (atomic_dec_return(&lldev->pending_tre_count) < 0) { 226 222 dev_warn(lldev->dev, "tre count mismatch on completion"); 227 - lldev->pending_tre_count = 0; 223 + atomic_set(&lldev->pending_tre_count, 0); 228 224 } 229 225 226 + HIDMA_INCREMENT_ITERATOR(tre_iterator, HIDMA_TRE_SIZE, 227 + lldev->tre_ring_size); 228 + lldev->tre_processed_off = tre_iterator; 230 229 spin_unlock_irqrestore(&lldev->lock, flags); 231 230 232 231 tre->err_info = err_info; ··· 250 245 static int hidma_handle_tre_completion(struct hidma_lldev *lldev) 251 246 { 252 247 u32 evre_ring_size = lldev->evre_ring_size; 253 - u32 tre_ring_size = lldev->tre_ring_size; 254 248 u32 err_info, err_code, evre_write_off; 255 - u32 tre_iterator, evre_iterator; 249 + u32 evre_iterator; 256 250 u32 num_completed = 0; 257 251 258 252 evre_write_off = readl_relaxed(lldev->evca + HIDMA_EVCA_WRITE_PTR_REG); 259 - tre_iterator = lldev->tre_processed_off; 260 253 evre_iterator = lldev->evre_processed_off; 261 254 262 255 if ((evre_write_off > evre_ring_size) || ··· 277 274 err_code = 278 275 (cfg >> HIDMA_EVRE_CODE_BIT_POS) & HIDMA_EVRE_CODE_MASK; 279 276 280 - if (hidma_post_completed(lldev, tre_iterator, err_info, 281 - err_code)) 277 + if (hidma_post_completed(lldev, err_info, err_code)) 282 278 break; 283 279 284 - HIDMA_INCREMENT_ITERATOR(tre_iterator, HIDMA_TRE_SIZE, 285 - tre_ring_size); 286 280 HIDMA_INCREMENT_ITERATOR(evre_iterator, HIDMA_EVRE_SIZE, 287 281 evre_ring_size); 288 282 ··· 291 291 evre_write_off = 292 292 readl_relaxed(lldev->evca + HIDMA_EVCA_WRITE_PTR_REG); 293 293 num_completed++; 294 + 295 + /* 296 + * An error interrupt might have arrived while we are processing 297 + * the completed interrupt. 298 + */ 299 + if (!hidma_ll_isenabled(lldev)) 300 + break; 294 301 } 295 302 296 303 if (num_completed) { 297 304 u32 evre_read_off = (lldev->evre_processed_off + 298 305 HIDMA_EVRE_SIZE * num_completed); 299 - u32 tre_read_off = (lldev->tre_processed_off + 300 - HIDMA_TRE_SIZE * num_completed); 301 - 302 306 evre_read_off = evre_read_off % evre_ring_size; 303 - tre_read_off = tre_read_off % tre_ring_size; 304 - 305 307 writel(evre_read_off, lldev->evca + HIDMA_EVCA_DOORBELL_REG); 306 308 307 309 /* record the last processed tre offset */ 308 - lldev->tre_processed_off = tre_read_off; 309 310 lldev->evre_processed_off = evre_read_off; 310 311 } 311 312 ··· 316 315 void hidma_cleanup_pending_tre(struct hidma_lldev *lldev, u8 err_info, 317 316 u8 err_code) 318 317 { 319 - u32 tre_iterator; 320 - u32 tre_ring_size = lldev->tre_ring_size; 321 - int num_completed = 0; 322 - u32 tre_read_off; 323 - 324 - tre_iterator = lldev->tre_processed_off; 325 - while (lldev->pending_tre_count) { 326 - if (hidma_post_completed(lldev, tre_iterator, err_info, 327 - err_code)) 318 + while (atomic_read(&lldev->pending_tre_count)) { 319 + if (hidma_post_completed(lldev, err_info, err_code)) 328 320 break; 329 - HIDMA_INCREMENT_ITERATOR(tre_iterator, HIDMA_TRE_SIZE, 330 - tre_ring_size); 331 - num_completed++; 332 321 } 333 - tre_read_off = (lldev->tre_processed_off + 334 - HIDMA_TRE_SIZE * num_completed); 335 - 336 - tre_read_off = tre_read_off % tre_ring_size; 337 - 338 - /* record the last processed tre offset */ 339 - lldev->tre_processed_off = tre_read_off; 340 322 } 341 323 342 324 static int hidma_ll_reset(struct hidma_lldev *lldev) ··· 396 412 * requests traditionally to the destination, this concept does not apply 397 413 * here for this HW. 398 414 */ 399 - irqreturn_t hidma_ll_inthandler(int chirq, void *arg) 415 + static void hidma_ll_int_handler_internal(struct hidma_lldev *lldev, int cause) 400 416 { 401 - struct hidma_lldev *lldev = arg; 402 - u32 status; 403 - u32 enable; 404 - u32 cause; 417 + if (cause & HIDMA_ERR_INT_MASK) { 418 + dev_err(lldev->dev, "error 0x%x, disabling...\n", 419 + cause); 420 + 421 + /* Clear out pending interrupts */ 422 + writel(cause, lldev->evca + HIDMA_EVCA_IRQ_CLR_REG); 423 + 424 + /* No further submissions. */ 425 + hidma_ll_disable(lldev); 426 + 427 + /* Driver completes the txn and intimates the client.*/ 428 + hidma_cleanup_pending_tre(lldev, 0xFF, 429 + HIDMA_EVRE_STATUS_ERROR); 430 + 431 + return; 432 + } 405 433 406 434 /* 407 435 * Fine tuned for this HW... ··· 422 426 * read and write accessors are used for performance reasons due to 423 427 * interrupt delivery guarantees. Do not copy this code blindly and 424 428 * expect that to work. 429 + * 430 + * Try to consume as many EVREs as possible. 425 431 */ 432 + hidma_handle_tre_completion(lldev); 433 + 434 + /* We consumed TREs or there are pending TREs or EVREs. */ 435 + writel_relaxed(cause, lldev->evca + HIDMA_EVCA_IRQ_CLR_REG); 436 + } 437 + 438 + irqreturn_t hidma_ll_inthandler(int chirq, void *arg) 439 + { 440 + struct hidma_lldev *lldev = arg; 441 + u32 status; 442 + u32 enable; 443 + u32 cause; 444 + 426 445 status = readl_relaxed(lldev->evca + HIDMA_EVCA_IRQ_STAT_REG); 427 446 enable = readl_relaxed(lldev->evca + HIDMA_EVCA_IRQ_EN_REG); 428 447 cause = status & enable; 429 448 430 449 while (cause) { 431 - if (cause & HIDMA_ERR_INT_MASK) { 432 - dev_err(lldev->dev, "error 0x%x, disabling...\n", 433 - cause); 434 - 435 - /* Clear out pending interrupts */ 436 - writel(cause, lldev->evca + HIDMA_EVCA_IRQ_CLR_REG); 437 - 438 - /* No further submissions. */ 439 - hidma_ll_disable(lldev); 440 - 441 - /* Driver completes the txn and intimates the client.*/ 442 - hidma_cleanup_pending_tre(lldev, 0xFF, 443 - HIDMA_EVRE_STATUS_ERROR); 444 - goto out; 445 - } 446 - 447 - /* 448 - * Try to consume as many EVREs as possible. 449 - */ 450 - hidma_handle_tre_completion(lldev); 451 - 452 - /* We consumed TREs or there are pending TREs or EVREs. */ 453 - writel_relaxed(cause, lldev->evca + HIDMA_EVCA_IRQ_CLR_REG); 450 + hidma_ll_int_handler_internal(lldev, cause); 454 451 455 452 /* 456 453 * Another interrupt might have arrived while we are ··· 454 465 cause = status & enable; 455 466 } 456 467 457 - out: 468 + return IRQ_HANDLED; 469 + } 470 + 471 + irqreturn_t hidma_ll_inthandler_msi(int chirq, void *arg, int cause) 472 + { 473 + struct hidma_lldev *lldev = arg; 474 + 475 + hidma_ll_int_handler_internal(lldev, cause); 458 476 return IRQ_HANDLED; 459 477 } 460 478 ··· 544 548 tre->err_code = 0; 545 549 tre->err_info = 0; 546 550 tre->queued = 1; 547 - lldev->pending_tre_count++; 551 + atomic_inc(&lldev->pending_tre_count); 548 552 lldev->tre_write_offset = (lldev->tre_write_offset + HIDMA_TRE_SIZE) 549 553 % lldev->tre_ring_size; 550 554 spin_unlock_irqrestore(&lldev->lock, flags); ··· 560 564 u32 val; 561 565 int ret; 562 566 563 - val = readl(lldev->evca + HIDMA_EVCA_CTRLSTS_REG); 564 - lldev->evch_state = HIDMA_CH_STATE(val); 565 - val = readl(lldev->trca + HIDMA_TRCA_CTRLSTS_REG); 566 - lldev->trch_state = HIDMA_CH_STATE(val); 567 - 568 - /* already suspended by this OS */ 569 - if ((lldev->trch_state == HIDMA_CH_SUSPENDED) || 570 - (lldev->evch_state == HIDMA_CH_SUSPENDED)) 571 - return 0; 572 - 573 - /* already stopped by the manager */ 574 - if ((lldev->trch_state == HIDMA_CH_STOPPED) || 575 - (lldev->evch_state == HIDMA_CH_STOPPED)) 567 + /* The channel needs to be in working state */ 568 + if (!hidma_ll_isenabled(lldev)) 576 569 return 0; 577 570 578 571 val = readl(lldev->trca + HIDMA_TRCA_CTRLSTS_REG); ··· 639 654 u32 val; 640 655 u32 nr_tres = lldev->nr_tres; 641 656 642 - lldev->pending_tre_count = 0; 657 + atomic_set(&lldev->pending_tre_count, 0); 643 658 lldev->tre_processed_off = 0; 644 659 lldev->evre_processed_off = 0; 645 660 lldev->tre_write_offset = 0; ··· 676 691 writel(HIDMA_EVRE_SIZE * nr_tres, 677 692 lldev->evca + HIDMA_EVCA_RING_LEN_REG); 678 693 679 - /* support IRQ only for now */ 694 + /* configure interrupts */ 695 + hidma_ll_setup_irq(lldev, lldev->msi_support); 696 + 697 + rc = hidma_ll_enable(lldev); 698 + if (rc) 699 + return rc; 700 + 701 + return rc; 702 + } 703 + 704 + void hidma_ll_setup_irq(struct hidma_lldev *lldev, bool msi) 705 + { 706 + u32 val; 707 + 708 + lldev->msi_support = msi; 709 + 710 + /* disable interrupts again after reset */ 711 + writel(0, lldev->evca + HIDMA_EVCA_IRQ_CLR_REG); 712 + writel(0, lldev->evca + HIDMA_EVCA_IRQ_EN_REG); 713 + 714 + /* support IRQ by default */ 680 715 val = readl(lldev->evca + HIDMA_EVCA_INTCTRL_REG); 681 716 val &= ~0xF; 682 - val |= 0x1; 717 + if (!lldev->msi_support) 718 + val = val | 0x1; 683 719 writel(val, lldev->evca + HIDMA_EVCA_INTCTRL_REG); 684 720 685 721 /* clear all pending interrupts and enable them */ 686 722 writel(ENABLE_IRQS, lldev->evca + HIDMA_EVCA_IRQ_CLR_REG); 687 723 writel(ENABLE_IRQS, lldev->evca + HIDMA_EVCA_IRQ_EN_REG); 688 - 689 - return hidma_ll_enable(lldev); 690 724 } 691 725 692 726 struct hidma_lldev *hidma_ll_init(struct device *dev, u32 nr_tres, ··· 820 816 tasklet_kill(&lldev->task); 821 817 memset(lldev->trepool, 0, required_bytes); 822 818 lldev->trepool = NULL; 823 - lldev->pending_tre_count = 0; 819 + atomic_set(&lldev->pending_tre_count, 0); 824 820 lldev->tre_write_offset = 0; 825 821 826 822 rc = hidma_ll_reset(lldev);
+9 -2
drivers/dma/qcom/hidma_mgmt.c
··· 282 282 {"QCOM8060"}, 283 283 {}, 284 284 }; 285 + MODULE_DEVICE_TABLE(acpi, hidma_mgmt_acpi_ids); 285 286 #endif 286 287 287 288 static const struct of_device_id hidma_mgmt_match[] = { ··· 376 375 ret = PTR_ERR(new_pdev); 377 376 goto out; 378 377 } 378 + of_node_get(child); 379 + new_pdev->dev.of_node = child; 379 380 of_dma_configure(&new_pdev->dev, child); 380 - 381 + /* 382 + * It is assumed that calling of_msi_configure is safe on 383 + * platforms with or without MSI support. 384 + */ 385 + of_msi_configure(&new_pdev->dev, child); 386 + of_node_put(child); 381 387 kfree(res); 382 388 res = NULL; 383 389 } ··· 403 395 for_each_matching_node(child, hidma_mgmt_match) { 404 396 /* device tree based firmware here */ 405 397 hidma_mgmt_of_populate_channels(child); 406 - of_node_put(child); 407 398 } 408 399 #endif 409 400 platform_driver_register(&hidma_mgmt_driver);
-5
drivers/dma/s3c24xx-dma.c
··· 289 289 struct s3c24xx_dma_phy *s3c24xx_dma_get_phy(struct s3c24xx_dma_chan *s3cchan) 290 290 { 291 291 struct s3c24xx_dma_engine *s3cdma = s3cchan->host; 292 - const struct s3c24xx_dma_platdata *pdata = s3cdma->pdata; 293 - struct s3c24xx_dma_channel *cdata; 294 292 struct s3c24xx_dma_phy *phy = NULL; 295 293 unsigned long flags; 296 294 int i; 297 295 int ret; 298 - 299 - if (s3cchan->slave) 300 - cdata = &pdata->channels[s3cchan->id]; 301 296 302 297 for (i = 0; i < s3cdma->pdata->num_phy_channels; i++) { 303 298 phy = &s3cdma->phy_chans[i];
-3
drivers/dma/sh/usb-dmac.c
··· 652 652 static struct dma_chan *usb_dmac_of_xlate(struct of_phandle_args *dma_spec, 653 653 struct of_dma *ofdma) 654 654 { 655 - struct usb_dmac_chan *uchan; 656 655 struct dma_chan *chan; 657 656 dma_cap_mask_t mask; 658 657 ··· 665 666 chan = dma_request_channel(mask, usb_dmac_chan_filter, dma_spec); 666 667 if (!chan) 667 668 return NULL; 668 - 669 - uchan = to_usb_dmac_chan(chan); 670 669 671 670 return chan; 672 671 }
-4
drivers/dma/sirf-dma.c
··· 1011 1011 { 1012 1012 struct sirfsoc_dma *sdma = dev_get_drvdata(dev); 1013 1013 struct sirfsoc_dma_regs *save = &sdma->regs_save; 1014 - struct sirfsoc_dma_desc *sdesc; 1015 1014 struct sirfsoc_dma_chan *schan; 1016 1015 int ch; 1017 1016 int ret; ··· 1043 1044 schan = &sdma->channels[ch]; 1044 1045 if (list_empty(&schan->active)) 1045 1046 continue; 1046 - sdesc = list_first_entry(&schan->active, 1047 - struct sirfsoc_dma_desc, 1048 - node); 1049 1047 save->ctrl[ch] = readl_relaxed(sdma->base + 1050 1048 ch * 0x10 + SIRFSOC_DMA_CH_CTRL); 1051 1049 }
+1 -5
drivers/dma/stm32-dma.c
··· 527 527 { 528 528 struct stm32_dma_chan *chan = devid; 529 529 struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan); 530 - u32 status, scr, sfcr; 530 + u32 status, scr; 531 531 532 532 spin_lock(&chan->vchan.lock); 533 533 534 534 status = stm32_dma_irq_status(chan); 535 535 scr = stm32_dma_read(dmadev, STM32_DMA_SCR(chan->id)); 536 - sfcr = stm32_dma_read(dmadev, STM32_DMA_SFCR(chan->id)); 537 536 538 537 if ((status & STM32_DMA_TCI) && (scr & STM32_DMA_SCR_TCIE)) { 539 538 stm32_dma_irq_clear(chan, STM32_DMA_TCI); ··· 573 574 int src_bus_width, dst_bus_width; 574 575 int src_burst_size, dst_burst_size; 575 576 u32 src_maxburst, dst_maxburst; 576 - dma_addr_t src_addr, dst_addr; 577 577 u32 dma_scr = 0; 578 578 579 579 src_addr_width = chan->dma_sconfig.src_addr_width; 580 580 dst_addr_width = chan->dma_sconfig.dst_addr_width; 581 581 src_maxburst = chan->dma_sconfig.src_maxburst; 582 582 dst_maxburst = chan->dma_sconfig.dst_maxburst; 583 - src_addr = chan->dma_sconfig.src_addr; 584 - dst_addr = chan->dma_sconfig.dst_addr; 585 583 586 584 switch (direction) { 587 585 case DMA_MEM_TO_DEV:
+1 -2
drivers/dma/zx296702_dma.c
··· 435 435 if (!ds) 436 436 return NULL; 437 437 438 - ds->desc_hw = dma_pool_alloc(d->pool, GFP_NOWAIT, &ds->desc_hw_lli); 438 + ds->desc_hw = dma_pool_zalloc(d->pool, GFP_NOWAIT, &ds->desc_hw_lli); 439 439 if (!ds->desc_hw) { 440 440 dev_dbg(chan->device->dev, "vch %p: dma alloc fail\n", &c->vc); 441 441 kfree(ds); 442 442 return NULL; 443 443 } 444 - memset(ds->desc_hw, 0, sizeof(struct zx_desc_hw) * num); 445 444 ds->desc_num = num; 446 445 return ds; 447 446 }
+1
drivers/of/irq.c
··· 697 697 dev_set_msi_domain(dev, 698 698 of_msi_get_domain(dev, np, DOMAIN_BUS_PLATFORM_MSI)); 699 699 } 700 + EXPORT_SYMBOL_GPL(of_msi_configure);
+4 -17
drivers/spi/spi-s3c64xx.c
··· 341 341 static int s3c64xx_spi_prepare_transfer(struct spi_master *spi) 342 342 { 343 343 struct s3c64xx_spi_driver_data *sdd = spi_master_get_devdata(spi); 344 - dma_filter_fn filter = sdd->cntrlr_info->filter; 345 344 struct device *dev = &sdd->pdev->dev; 346 - dma_cap_mask_t mask; 347 345 348 346 if (is_polling(sdd)) 349 347 return 0; 350 348 351 - dma_cap_zero(mask); 352 - dma_cap_set(DMA_SLAVE, mask); 353 - 354 349 /* Acquire DMA channels */ 355 - sdd->rx_dma.ch = dma_request_slave_channel_compat(mask, filter, 356 - sdd->cntrlr_info->dma_rx, dev, "rx"); 350 + sdd->rx_dma.ch = dma_request_slave_channel(dev, "rx"); 357 351 if (!sdd->rx_dma.ch) { 358 352 dev_err(dev, "Failed to get RX DMA channel\n"); 359 353 return -EBUSY; 360 354 } 361 355 spi->dma_rx = sdd->rx_dma.ch; 362 356 363 - sdd->tx_dma.ch = dma_request_slave_channel_compat(mask, filter, 364 - sdd->cntrlr_info->dma_tx, dev, "tx"); 357 + sdd->tx_dma.ch = dma_request_slave_channel(dev, "tx"); 365 358 if (!sdd->tx_dma.ch) { 366 359 dev_err(dev, "Failed to get TX DMA channel\n"); 367 360 dma_release_channel(sdd->rx_dma.ch); ··· 1084 1091 1085 1092 sdd->cur_bpw = 8; 1086 1093 1087 - if (!sdd->pdev->dev.of_node && (!sci->dma_tx || !sci->dma_rx)) { 1088 - dev_warn(&pdev->dev, "Unable to get SPI tx/rx DMA data. Switching to poll mode\n"); 1089 - sdd->port_conf->quirks = S3C64XX_SPI_QUIRK_POLL; 1090 - } 1091 - 1092 1094 sdd->tx_dma.direction = DMA_MEM_TO_DEV; 1093 1095 sdd->rx_dma.direction = DMA_DEV_TO_MEM; 1094 1096 ··· 1193 1205 1194 1206 dev_dbg(&pdev->dev, "Samsung SoC SPI Driver loaded for Bus SPI-%d with %d Slaves attached\n", 1195 1207 sdd->port_id, master->num_chipselect); 1196 - dev_dbg(&pdev->dev, "\tIOmem=[%pR]\tFIFO %dbytes\tDMA=[Rx-%p, Tx-%p]\n", 1197 - mem_res, (FIFO_LVL_MASK(sdd) >> 1) + 1, 1198 - sci->dma_rx, sci->dma_tx); 1208 + dev_dbg(&pdev->dev, "\tIOmem=[%pR]\tFIFO %dbytes\n", 1209 + mem_res, (FIFO_LVL_MASK(sdd) >> 1) + 1); 1199 1210 1200 1211 pm_runtime_mark_last_busy(&pdev->dev); 1201 1212 pm_runtime_put_autosuspend(&pdev->dev);
+1 -1
drivers/tty/serial/8250/8250_lpss.c
··· 157 157 static const struct dw_dma_platform_data qrk_serial_dma_pdata = { 158 158 .nr_channels = 2, 159 159 .is_private = true, 160 - .is_nollp = true, 161 160 .chan_allocation_order = CHAN_ALLOCATION_ASCENDING, 162 161 .chan_priority = CHAN_PRIORITY_ASCENDING, 163 162 .block_size = 4095, 164 163 .nr_masters = 1, 165 164 .data_width = {4}, 165 + .multi_block = {0}, 166 166 }; 167 167 168 168 static void qrk_serial_setup_dma(struct lpss8250 *lpss, struct uart_port *port)
+4
include/linux/amba/pl08x.h
··· 84 84 * running any DMA transfer and multiplexing can be recycled 85 85 * @lli_buses: buses which LLIs can be fetched from: PL08X_AHB1 | PL08X_AHB2 86 86 * @mem_buses: buses which memory can be accessed from: PL08X_AHB1 | PL08X_AHB2 87 + * @slave_map: DMA slave matching table 88 + * @slave_map_len: number of elements in @slave_map 87 89 */ 88 90 struct pl08x_platform_data { 89 91 struct pl08x_channel_data *slave_channels; ··· 95 93 void (*put_xfer_signal)(const struct pl08x_channel_data *, int); 96 94 u8 lli_buses; 97 95 u8 mem_buses; 96 + const struct dma_slave_map *slave_map; 97 + int slave_map_len; 98 98 }; 99 99 100 100 #ifdef CONFIG_AMBA_PL08X
+8
include/linux/dmaengine.h
··· 336 336 * may or may not be applicable on memory sources. 337 337 * @dst_maxburst: same as src_maxburst but for destination target 338 338 * mutatis mutandis. 339 + * @src_port_window_size: The length of the register area in words the data need 340 + * to be accessed on the device side. It is only used for devices which is using 341 + * an area instead of a single register to receive the data. Typically the DMA 342 + * loops in this area in order to transfer the data. 343 + * @dst_port_window_size: same as src_port_window_size but for the destination 344 + * port. 339 345 * @device_fc: Flow Controller Settings. Only valid for slave channels. Fill 340 346 * with 'true' if peripheral should be flow controller. Direction will be 341 347 * selected at Runtime. ··· 369 363 enum dma_slave_buswidth dst_addr_width; 370 364 u32 src_maxburst; 371 365 u32 dst_maxburst; 366 + u32 src_port_window_size; 367 + u32 dst_port_window_size; 372 368 bool device_fc; 373 369 unsigned int slave_id; 374 370 };
+3 -2
include/linux/platform_data/dma-dw.h
··· 14 14 #include <linux/device.h> 15 15 16 16 #define DW_DMA_MAX_NR_MASTERS 4 17 + #define DW_DMA_MAX_NR_CHANNELS 8 17 18 18 19 /** 19 20 * struct dw_dma_slave - Controller-specific information about a slave ··· 41 40 * @is_private: The device channels should be marked as private and not for 42 41 * by the general purpose DMA channel allocator. 43 42 * @is_memcpy: The device channels do support memory-to-memory transfers. 44 - * @is_nollp: The device channels does not support multi block transfers. 45 43 * @chan_allocation_order: Allocate channels starting from 0 or 7 46 44 * @chan_priority: Set channel priority increasing from 0 to 7 or 7 to 0. 47 45 * @block_size: Maximum block size supported by the controller 48 46 * @nr_masters: Number of AHB masters supported by the controller 49 47 * @data_width: Maximum data width supported by hardware per AHB master 50 48 * (in bytes, power of 2) 49 + * @multi_block: Multi block transfers supported by hardware per channel. 51 50 */ 52 51 struct dw_dma_platform_data { 53 52 unsigned int nr_channels; 54 53 bool is_private; 55 54 bool is_memcpy; 56 - bool is_nollp; 57 55 #define CHAN_ALLOCATION_ASCENDING 0 /* zero to seven */ 58 56 #define CHAN_ALLOCATION_DESCENDING 1 /* seven to zero */ 59 57 unsigned char chan_allocation_order; ··· 62 62 unsigned int block_size; 63 63 unsigned char nr_masters; 64 64 unsigned char data_width[DW_DMA_MAX_NR_MASTERS]; 65 + unsigned char multi_block[DW_DMA_MAX_NR_CHANNELS]; 65 66 }; 66 67 67 68 #endif /* _PLATFORM_DATA_DMA_DW_H */
-3
include/linux/platform_data/spi-s3c64xx.h
··· 40 40 int num_cs; 41 41 bool no_cs; 42 42 int (*cfg_gpio)(void); 43 - dma_filter_fn filter; 44 - void *dma_tx; 45 - void *dma_rx; 46 43 }; 47 44 48 45 /**