tjh.dev
Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge branch 'pci/ntb'
- Account for 64-bit BARs in pci_epc_get_first_free_bar() (Kishon Vijay
Abraham I)
- Add pci_epc_get_next_free_bar() helper (Kishon Vijay Abraham I)
- Return error codes on failure of endpoint BAR interfaces (Kishon Vijay
Abraham I)
- Remove unused pci_epf_match_device() (Kishon Vijay Abraham I)
- Add support for secondary endpoint controller to prepare for NTB endpoint
functionality (Kishon Vijay Abraham I)
- Add configfs support for secondary endpoint controller (Kishon Vijay
Abraham I)
- Add MSI address mapping ops for NTB doorbell support (Kishon Vijay
Abraham I)
- Add ops for endpoint function-specific attributes (Kishon Vijay Abraham
I)
- Allow configfs subdirectory for endpoint function configuration (Kishon
Vijay Abraham I)
- Implement cadence MSI address mapping ops (Kishon Vijay Abraham I)
- Configure cadence LM_EP_FUNC_CFG based on epc->function_num_map (Kishon
Vijay Abraham I)
- Add endpoint-side driver to provide NTB functionality (Kishon Vijay
Abraham I)
- Add host-side driver for generic EPF NTB functionality (Kishon Vijay
Abraham I)
- Document NTB endpoint functionality (Kishon Vijay Abraham I)
* pci/ntb:
Documentation: PCI: Add PCI endpoint NTB function user guide
Documentation: PCI: Add configfs binding documentation for pci-ntb endpoint function
NTB: Add support for EPF PCI Non-Transparent Bridge
PCI: Add TI J721E device to PCI IDs
PCI: endpoint: Add EP function driver to provide NTB functionality
PCI: cadence: Configure LM_EP_FUNC_CFG based on epc->function_num_map
PCI: cadence: Implement ->msi_map_irq() ops
PCI: endpoint: Allow user to create sub-directory of 'EPF Device' directory
PCI: endpoint: Add pci_epf_ops to expose function-specific attrs
PCI: endpoint: Add pci_epc_ops to map MSI IRQ
PCI: endpoint: Add support in configfs to associate two EPCs with EPF
PCI: endpoint: Add support to associate secondary EPC with EPF
PCI: endpoint: Remove unused pci_epf_match_device()
PCI: endpoint: Make *_free_bar() to return error codes on failure
PCI: endpoint: Add helper API to get the 'next' unreserved BAR
PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit BAR
Documentation: PCI: Add specification for the PCI NTB function device
+3944
-73
+38
Documentation/PCI/endpoint/function/binding/pci-ntb.rst
+38
Documentation/PCI/endpoint/function/binding/pci-ntb.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
==========================
4
+
PCI NTB Endpoint Function
5
+
==========================
6
+
7
+
1) Create a subdirectory to pci_epf_ntb directory in configfs.
8
+
9
+
Standard EPF Configurable Fields:
10
+
11
+
================ ===========================================================
12
+
vendorid should be 0x104c
13
+
deviceid should be 0xb00d for TI's J721E SoC
14
+
revid don't care
15
+
progif_code don't care
16
+
subclass_code should be 0x00
17
+
baseclass_code should be 0x5
18
+
cache_line_size don't care
19
+
subsys_vendor_id don't care
20
+
subsys_id don't care
21
+
interrupt_pin don't care
22
+
msi_interrupts don't care
23
+
msix_interrupts don't care
24
+
================ ===========================================================
25
+
26
+
2) Create a subdirectory to directory created in 1
27
+
28
+
NTB EPF specific configurable fields:
29
+
30
+
================ ===========================================================
31
+
db_count Number of doorbells; default = 4
32
+
mw1 size of memory window1
33
+
mw2 size of memory window2
34
+
mw3 size of memory window3
35
+
mw4 size of memory window4
36
+
num_mws Number of memory windows; max = 4
37
+
spad_count Number of scratchpad registers; default = 64
38
+
================ ===========================================================
+3
Documentation/PCI/endpoint/index.rst
+3
Documentation/PCI/endpoint/index.rst
+10
Documentation/PCI/endpoint/pci-endpoint-cfs.rst
+10
Documentation/PCI/endpoint/pci-endpoint-cfs.rst
···
68
68
... subsys_vendor_id
69
69
... subsys_id
70
70
... interrupt_pin
71
+
... primary/
72
+
... <Symlink EPC Device1>/
73
+
... secondary/
74
+
... <Symlink EPC Device2>/
75
+
76
+
If an EPF device has to be associated with 2 EPCs (like in the case of
77
+
Non-transparent bridge), symlink of endpoint controller connected to primary
78
+
interface should be added in 'primary' directory and symlink of endpoint
79
+
controller connected to secondary interface should be added in 'secondary'
80
+
directory.
71
81
72
82
EPC Device
73
83
==========
+348
Documentation/PCI/endpoint/pci-ntb-function.rst
+348
Documentation/PCI/endpoint/pci-ntb-function.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
=================
4
+
PCI NTB Function
5
+
=================
6
+
7
+
:Author: Kishon Vijay Abraham I <kishon@ti.com>
8
+
9
+
PCI Non-Transparent Bridges (NTB) allow two host systems to communicate
10
+
with each other by exposing each host as a device to the other host.
11
+
NTBs typically support the ability to generate interrupts on the remote
12
+
machine, expose memory ranges as BARs, and perform DMA. They also support
13
+
scratchpads, which are areas of memory within the NTB that are accessible
14
+
from both machines.
15
+
16
+
PCI NTB Function allows two different systems (or hosts) to communicate
17
+
with each other by configuring the endpoint instances in such a way that
18
+
transactions from one system are routed to the other system.
19
+
20
+
In the below diagram, PCI NTB function configures the SoC with multiple
21
+
PCI Endpoint (EP) instances in such a way that transactions from one EP
22
+
controller are routed to the other EP controller. Once PCI NTB function
23
+
configures the SoC with multiple EP instances, HOST1 and HOST2 can
24
+
communicate with each other using SoC as a bridge.
25
+
26
+
.. code-block:: text
27
+
28
+
+-------------+ +-------------+
29
+
| | | |
30
+
| HOST1 | | HOST2 |
31
+
| | | |
32
+
+------^------+ +------^------+
33
+
| |
34
+
| |
35
+
+---------|-------------------------------------------------|---------+
36
+
| +------v------+ +------v------+ |
37
+
| | | | | |
38
+
| | EP | | EP | |
39
+
| | CONTROLLER1 | | CONTROLLER2 | |
40
+
| | <-----------------------------------> | |
41
+
| | | | | |
42
+
| | | | | |
43
+
| | | SoC With Multiple EP Instances | | |
44
+
| | | (Configured using NTB Function) | | |
45
+
| +-------------+ +-------------+ |
46
+
+---------------------------------------------------------------------+
47
+
48
+
Constructs used for Implementing NTB
49
+
====================================
50
+
51
+
1) Config Region
52
+
2) Self Scratchpad Registers
53
+
3) Peer Scratchpad Registers
54
+
4) Doorbell (DB) Registers
55
+
5) Memory Window (MW)
56
+
57
+
58
+
Config Region:
59
+
--------------
60
+
61
+
Config Region is a construct that is specific to NTB implemented using NTB
62
+
Endpoint Function Driver. The host and endpoint side NTB function driver will
63
+
exchange information with each other using this region. Config Region has
64
+
Control/Status Registers for configuring the Endpoint Controller. Host can
65
+
write into this region for configuring the outbound Address Translation Unit
66
+
(ATU) and to indicate the link status. Endpoint can indicate the status of
67
+
commands issued by host in this region. Endpoint can also indicate the
68
+
scratchpad offset and number of memory windows to the host using this region.
69
+
70
+
The format of Config Region is given below. All the fields here are 32 bits.
71
+
72
+
.. code-block:: text
73
+
74
+
+------------------------+
75
+
| COMMAND |
76
+
+------------------------+
77
+
| ARGUMENT |
78
+
+------------------------+
79
+
| STATUS |
80
+
+------------------------+
81
+
| TOPOLOGY |
82
+
+------------------------+
83
+
| ADDRESS (LOWER 32) |
84
+
+------------------------+
85
+
| ADDRESS (UPPER 32) |
86
+
+------------------------+
87
+
| SIZE |
88
+
+------------------------+
89
+
| NO OF MEMORY WINDOW |
90
+
+------------------------+
91
+
| MEMORY WINDOW1 OFFSET |
92
+
+------------------------+
93
+
| SPAD OFFSET |
94
+
+------------------------+
95
+
| SPAD COUNT |
96
+
+------------------------+
97
+
| DB ENTRY SIZE |
98
+
+------------------------+
99
+
| DB DATA |
100
+
+------------------------+
101
+
| : |
102
+
+------------------------+
103
+
| : |
104
+
+------------------------+
105
+
| DB DATA |
106
+
+------------------------+
107
+
108
+
109
+
COMMAND:
110
+
111
+
NTB function supports three commands:
112
+
113
+
CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
114
+
invoking this command, the host should allocate and initialize
115
+
MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the
116
+
Endpoint). The endpoint on receiving this command will configure
117
+
the outbound ATU such that transactions to Doorbell BAR will be routed
118
+
to the MSI/MSI-X address programmed by the host. The ARGUMENT
119
+
register should be populated with number of DBs to configure (in the
120
+
lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
121
+
122
+
CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The
123
+
host invokes this command after allocating a buffer that can be
124
+
accessed by remote host. The allocated address should be programmed
125
+
in the ADDRESS register (64 bit), the size should be programmed in
126
+
the SIZE register and the memory window index should be programmed
127
+
in the ARGUMENT register. The endpoint on receiving this command
128
+
will configure the outbound ATU such that transactions to MW BAR
129
+
are routed to the address provided by the host.
130
+
131
+
CMD_LINK_UP (0x3): Command to indicate an NTB application is
132
+
bound to the EP device on the host side. Once the endpoint
133
+
receives this command from both the hosts, the endpoint will
134
+
raise a LINK_UP event to both the hosts to indicate the host
135
+
NTB applications can start communicating with each other.
136
+
137
+
ARGUMENT:
138
+
139
+
The value of this register is based on the commands issued in
140
+
command register. See COMMAND section for more information.
141
+
142
+
TOPOLOGY:
143
+
144
+
Set to NTB_TOPO_B2B_USD for Primary interface
145
+
Set to NTB_TOPO_B2B_DSD for Secondary interface
146
+
147
+
ADDRESS/SIZE:
148
+
149
+
Address and Size to be used while configuring the memory window.
150
+
See "CMD_CONFIGURE_MW" for more info.
151
+
152
+
MEMORY WINDOW1 OFFSET:
153
+
154
+
Memory Window 1 and Doorbell registers are packed together in the
155
+
same BAR. The initial portion of the region will have doorbell
156
+
registers and the latter portion of the region is for memory window 1.
157
+
This register will specify the offset of the memory window 1.
158
+
159
+
NO OF MEMORY WINDOW:
160
+
161
+
Specifies the number of memory windows supported by the NTB device.
162
+
163
+
SPAD OFFSET:
164
+
165
+
Self scratchpad region and config region are packed together in the
166
+
same BAR. The initial portion of the region will have config region
167
+
and the latter portion of the region is for self scratchpad. This
168
+
register will specify the offset of the self scratchpad registers.
169
+
170
+
SPAD COUNT:
171
+
172
+
Specifies the number of scratchpad registers supported by the NTB
173
+
device.
174
+
175
+
DB ENTRY SIZE:
176
+
177
+
Used to determine the offset within the DB BAR that should be written
178
+
in order to raise doorbell. EPF NTB can use either MSI or MSI-X to
179
+
ring doorbell (MSI-X support will be added later). MSI uses same
180
+
address for all the interrupts and MSI-X can provide different
181
+
addresses for different interrupts. The MSI/MSI-X address is provided
182
+
by the host and the address it gives is based on the MSI/MSI-X
183
+
implementation supported by the host. For instance, ARM platform
184
+
using GIC ITS will have the same MSI-X address for all the interrupts.
185
+
In order to support all the combinations and use the same mechanism
186
+
for both MSI and MSI-X, EPF NTB allocates a separate region in the
187
+
Outbound Address Space for each of the interrupts. This region will
188
+
be mapped to the MSI/MSI-X address provided by the host. If a host
189
+
provides the same address for all the interrupts, all the regions
190
+
will be translated to the same address. If a host provides different
191
+
addresses, the regions will be translated to different addresses. This
192
+
will ensure there is no difference while raising the doorbell.
193
+
194
+
DB DATA:
195
+
196
+
EPF NTB supports 32 interrupts, so there are 32 DB DATA registers.
197
+
This holds the MSI/MSI-X data that has to be written to MSI address
198
+
for raising doorbell interrupt. This will be populated by EPF NTB
199
+
while invoking CMD_CONFIGURE_DOORBELL.
200
+
201
+
Scratchpad Registers:
202
+
---------------------
203
+
204
+
Each host has its own register space allocated in the memory of NTB endpoint
205
+
controller. They are both readable and writable from both sides of the bridge.
206
+
They are used by applications built over NTB and can be used to pass control
207
+
and status information between both sides of a device.
208
+
209
+
Scratchpad registers has 2 parts
210
+
1) Self Scratchpad: Host's own register space
211
+
2) Peer Scratchpad: Remote host's register space.
212
+
213
+
Doorbell Registers:
214
+
-------------------
215
+
216
+
Doorbell Registers are used by the hosts to interrupt each other.
217
+
218
+
Memory Window:
219
+
--------------
220
+
221
+
Actual transfer of data between the two hosts will happen using the
222
+
memory window.
223
+
224
+
Modeling Constructs:
225
+
====================
226
+
227
+
There are 5 or more distinct regions (config, self scratchpad, peer
228
+
scratchpad, doorbell, one or more memory windows) to be modeled to achieve
229
+
NTB functionality. At least one memory window is required while more than
230
+
one is permitted. All these regions should be mapped to BARs for hosts to
231
+
access these regions.
232
+
233
+
If one 32-bit BAR is allocated for each of these regions, the scheme would
234
+
look like this:
235
+
236
+
====== ===============
237
+
BAR NO CONSTRUCTS USED
238
+
====== ===============
239
+
BAR0 Config Region
240
+
BAR1 Self Scratchpad
241
+
BAR2 Peer Scratchpad
242
+
BAR3 Doorbell
243
+
BAR4 Memory Window 1
244
+
BAR5 Memory Window 2
245
+
====== ===============
246
+
247
+
However if we allocate a separate BAR for each of the regions, there would not
248
+
be enough BARs for all the regions in a platform that supports only 64-bit
249
+
BARs.
250
+
251
+
In order to be supported by most of the platforms, the regions should be
252
+
packed and mapped to BARs in a way that provides NTB functionality and
253
+
also makes sure the host doesn't access any region that it is not supposed
254
+
to.
255
+
256
+
The following scheme is used in EPF NTB Function:
257
+
258
+
====== ===============================
259
+
BAR NO CONSTRUCTS USED
260
+
====== ===============================
261
+
BAR0 Config Region + Self Scratchpad
262
+
BAR1 Peer Scratchpad
263
+
BAR2 Doorbell + Memory Window 1
264
+
BAR3 Memory Window 2
265
+
BAR4 Memory Window 3
266
+
BAR5 Memory Window 4
267
+
====== ===============================
268
+
269
+
With this scheme, for the basic NTB functionality 3 BARs should be sufficient.
270
+
271
+
Modeling Config/Scratchpad Region:
272
+
----------------------------------
273
+
274
+
.. code-block:: text
275
+
276
+
+-----------------+------->+------------------+ +-----------------+
277
+
| BAR0 | | CONFIG REGION | | BAR0 |
278
+
+-----------------+----+ +------------------+<-------+-----------------+
279
+
| BAR1 | | |SCRATCHPAD REGION | | BAR1 |
280
+
+-----------------+ +-->+------------------+<-------+-----------------+
281
+
| BAR2 | Local Memory | BAR2 |
282
+
+-----------------+ +-----------------+
283
+
| BAR3 | | BAR3 |
284
+
+-----------------+ +-----------------+
285
+
| BAR4 | | BAR4 |
286
+
+-----------------+ +-----------------+
287
+
| BAR5 | | BAR5 |
288
+
+-----------------+ +-----------------+
289
+
EP CONTROLLER 1 EP CONTROLLER 2
290
+
291
+
Above diagram shows Config region + Scratchpad region for HOST1 (connected to
292
+
EP controller 1) allocated in local memory. The HOST1 can access the config
293
+
region and scratchpad region (self scratchpad) using BAR0 of EP controller 1.
294
+
The peer host (HOST2 connected to EP controller 2) can also access this
295
+
scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This
296
+
diagram shows the case where Config region and Scratchpad regions are allocated
297
+
for HOST1, however the same is applicable for HOST2.
298
+
299
+
Modeling Doorbell/Memory Window 1:
300
+
----------------------------------
301
+
302
+
.. code-block:: text
303
+
304
+
+-----------------+ +----->+----------------+-----------+-----------------+
305
+
| BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 |
306
+
+-----------------+ | +----------------+ +-----------------+
307
+
| BAR1 | | | Doorbell 2 +---------+ | |
308
+
+-----------------+----+ +----------------+ | | |
309
+
| BAR2 | | Doorbell 3 +-------+ | +-----------------+
310
+
+-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 |
311
+
| BAR3 | | | Doorbell 4 +-----+ | +-----------------+
312
+
+-----------------+ | |----------------+ | | | |
313
+
| BAR4 | | | | | | +-----------------+
314
+
+-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3||
315
+
| BAR5 | | | | | | +-----------------+
316
+
+-----------------+ +----->-----------------+ | | | |
317
+
EP CONTROLLER 1 | | | | +-----------------+
318
+
| | | +---->+ MSI-X ADDRESS 4 |
319
+
+----------------+ | +-----------------+
320
+
EP CONTROLLER 2 | | |
321
+
(OB SPACE) | | |
322
+
+-------> MW1 |
323
+
| |
324
+
| |
325
+
+-----------------+
326
+
| |
327
+
| |
328
+
| |
329
+
| |
330
+
| |
331
+
+-----------------+
332
+
PCI Address Space
333
+
(Managed by HOST2)
334
+
335
+
Above diagram shows how the doorbell and memory window 1 is mapped so that
336
+
HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access
337
+
buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and
338
+
memory window 1 regions are allocated in EP controller 2 outbound (OB) address
339
+
space. Allocating and configuring BARs for doorbell and memory window1
340
+
is done during the initialization phase of NTB endpoint function driver.
341
+
Mapping from EP controller 2 OB space to PCI address space is done when HOST2
342
+
sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL.
343
+
344
+
Modeling Optional Memory Windows:
345
+
---------------------------------
346
+
347
+
This is modeled the same was as MW1 but each of the additional memory windows
348
+
is mapped to separate BARs.
+161
Documentation/PCI/endpoint/pci-ntb-howto.rst
+161
Documentation/PCI/endpoint/pci-ntb-howto.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
===================================================================
4
+
PCI Non-Transparent Bridge (NTB) Endpoint Function (EPF) User Guide
5
+
===================================================================
6
+
7
+
:Author: Kishon Vijay Abraham I <kishon@ti.com>
8
+
9
+
This document is a guide to help users use pci-epf-ntb function driver
10
+
and ntb_hw_epf host driver for NTB functionality. The list of steps to
11
+
be followed in the host side and EP side is given below. For the hardware
12
+
configuration and internals of NTB using configurable endpoints see
13
+
Documentation/PCI/endpoint/pci-ntb-function.rst
14
+
15
+
Endpoint Device
16
+
===============
17
+
18
+
Endpoint Controller Devices
19
+
---------------------------
20
+
21
+
For implementing NTB functionality at least two endpoint controller devices
22
+
are required.
23
+
24
+
To find the list of endpoint controller devices in the system::
25
+
26
+
# ls /sys/class/pci_epc/
27
+
2900000.pcie-ep 2910000.pcie-ep
28
+
29
+
If PCI_ENDPOINT_CONFIGFS is enabled::
30
+
31
+
# ls /sys/kernel/config/pci_ep/controllers
32
+
2900000.pcie-ep 2910000.pcie-ep
33
+
34
+
35
+
Endpoint Function Drivers
36
+
-------------------------
37
+
38
+
To find the list of endpoint function drivers in the system::
39
+
40
+
# ls /sys/bus/pci-epf/drivers
41
+
pci_epf_ntb pci_epf_ntb
42
+
43
+
If PCI_ENDPOINT_CONFIGFS is enabled::
44
+
45
+
# ls /sys/kernel/config/pci_ep/functions
46
+
pci_epf_ntb pci_epf_ntb
47
+
48
+
49
+
Creating pci-epf-ntb Device
50
+
----------------------------
51
+
52
+
PCI endpoint function device can be created using the configfs. To create
53
+
pci-epf-ntb device, the following commands can be used::
54
+
55
+
# mount -t configfs none /sys/kernel/config
56
+
# cd /sys/kernel/config/pci_ep/
57
+
# mkdir functions/pci_epf_ntb/func1
58
+
59
+
The "mkdir func1" above creates the pci-epf-ntb function device that will
60
+
be probed by pci_epf_ntb driver.
61
+
62
+
The PCI endpoint framework populates the directory with the following
63
+
configurable fields::
64
+
65
+
# ls functions/pci_epf_ntb/func1
66
+
baseclass_code deviceid msi_interrupts pci-epf-ntb.0
67
+
progif_code secondary subsys_id vendorid
68
+
cache_line_size interrupt_pin msix_interrupts primary
69
+
revid subclass_code subsys_vendor_id
70
+
71
+
The PCI endpoint function driver populates these entries with default values
72
+
when the device is bound to the driver. The pci-epf-ntb driver populates
73
+
vendorid with 0xffff and interrupt_pin with 0x0001::
74
+
75
+
# cat functions/pci_epf_ntb/func1/vendorid
76
+
0xffff
77
+
# cat functions/pci_epf_ntb/func1/interrupt_pin
78
+
0x0001
79
+
80
+
81
+
Configuring pci-epf-ntb Device
82
+
-------------------------------
83
+
84
+
The user can configure the pci-epf-ntb device using its configfs entry. In order
85
+
to change the vendorid and the deviceid, the following
86
+
commands can be used::
87
+
88
+
# echo 0x104c > functions/pci_epf_ntb/func1/vendorid
89
+
# echo 0xb00d > functions/pci_epf_ntb/func1/deviceid
90
+
91
+
In order to configure NTB specific attributes, a new sub-directory to func1
92
+
should be created::
93
+
94
+
# mkdir functions/pci_epf_ntb/func1/pci_epf_ntb.0/
95
+
96
+
The NTB function driver will populate this directory with various attributes
97
+
that can be configured by the user::
98
+
99
+
# ls functions/pci_epf_ntb/func1/pci_epf_ntb.0/
100
+
db_count mw1 mw2 mw3 mw4 num_mws
101
+
spad_count
102
+
103
+
A sample configuration for NTB function is given below::
104
+
105
+
# echo 4 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/db_count
106
+
# echo 128 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/spad_count
107
+
# echo 2 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/num_mws
108
+
# echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw1
109
+
# echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw2
110
+
111
+
Binding pci-epf-ntb Device to EP Controller
112
+
--------------------------------------------
113
+
114
+
NTB function device should be attached to two PCI endpoint controllers
115
+
connected to the two hosts. Use the 'primary' and 'secondary' entries
116
+
inside NTB function device to attach one PCI endpoint controller to
117
+
primary interface and the other PCI endpoint controller to the secondary
118
+
interface::
119
+
120
+
# ln -s controllers/2900000.pcie-ep/ functions/pci-epf-ntb/func1/primary
121
+
# ln -s controllers/2910000.pcie-ep/ functions/pci-epf-ntb/func1/secondary
122
+
123
+
Once the above step is completed, both the PCI endpoint controllers are ready to
124
+
establish a link with the host.
125
+
126
+
127
+
Start the Link
128
+
--------------
129
+
130
+
In order for the endpoint device to establish a link with the host, the _start_
131
+
field should be populated with '1'. For NTB, both the PCI endpoint controllers
132
+
should establish link with the host::
133
+
134
+
# echo 1 > controllers/2900000.pcie-ep/start
135
+
# echo 1 > controllers/2910000.pcie-ep/start
136
+
137
+
138
+
RootComplex Device
139
+
==================
140
+
141
+
lspci Output
142
+
------------
143
+
144
+
Note that the devices listed here correspond to the values populated in
145
+
"Creating pci-epf-ntb Device" section above::
146
+
147
+
# lspci
148
+
0000:00:00.0 PCI bridge: Texas Instruments Device b00d
149
+
0000:01:00.0 RAM memory: Texas Instruments Device b00d
150
+
151
+
152
+
Using ntb_hw_epf Device
153
+
-----------------------
154
+
155
+
The host side software follows the standard NTB software architecture in Linux.
156
+
All the existing client side NTB utilities like NTB Transport Client and NTB
157
+
Netdev, NTB Ping Pong Test Client and NTB Tool Test Client can be used with NTB
158
+
function device.
159
+
160
+
For more information on NTB see
161
+
:doc:`Non-Transparent Bridge <../../driver-api/ntb>`
-1
drivers/misc/pci_endpoint_test.c
-1
drivers/misc/pci_endpoint_test.c
+1
drivers/ntb/hw/Kconfig
+1
drivers/ntb/hw/Kconfig
+1
drivers/ntb/hw/Makefile
+1
drivers/ntb/hw/Makefile
+6
drivers/ntb/hw/epf/Kconfig
+6
drivers/ntb/hw/epf/Kconfig
+1
drivers/ntb/hw/epf/Makefile
+1
drivers/ntb/hw/epf/Makefile
···
1
+
obj-$(CONFIG_NTB_EPF) += ntb_hw_epf.o
+753
drivers/ntb/hw/epf/ntb_hw_epf.c
+753
drivers/ntb/hw/epf/ntb_hw_epf.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/**
3
+
* Host side endpoint driver to implement Non-Transparent Bridge functionality
4
+
*
5
+
* Copyright (C) 2020 Texas Instruments
6
+
* Author: Kishon Vijay Abraham I <kishon@ti.com>
7
+
*/
8
+
9
+
#include <linux/delay.h>
10
+
#include <linux/module.h>
11
+
#include <linux/pci.h>
12
+
#include <linux/slab.h>
13
+
#include <linux/ntb.h>
14
+
15
+
#define NTB_EPF_COMMAND 0x0
16
+
#define CMD_CONFIGURE_DOORBELL 1
17
+
#define CMD_TEARDOWN_DOORBELL 2
18
+
#define CMD_CONFIGURE_MW 3
19
+
#define CMD_TEARDOWN_MW 4
20
+
#define CMD_LINK_UP 5
21
+
#define CMD_LINK_DOWN 6
22
+
23
+
#define NTB_EPF_ARGUMENT 0x4
24
+
#define MSIX_ENABLE BIT(16)
25
+
26
+
#define NTB_EPF_CMD_STATUS 0x8
27
+
#define COMMAND_STATUS_OK 1
28
+
#define COMMAND_STATUS_ERROR 2
29
+
30
+
#define NTB_EPF_LINK_STATUS 0x0A
31
+
#define LINK_STATUS_UP BIT(0)
32
+
33
+
#define NTB_EPF_TOPOLOGY 0x0C
34
+
#define NTB_EPF_LOWER_ADDR 0x10
35
+
#define NTB_EPF_UPPER_ADDR 0x14
36
+
#define NTB_EPF_LOWER_SIZE 0x18
37
+
#define NTB_EPF_UPPER_SIZE 0x1C
38
+
#define NTB_EPF_MW_COUNT 0x20
39
+
#define NTB_EPF_MW1_OFFSET 0x24
40
+
#define NTB_EPF_SPAD_OFFSET 0x28
41
+
#define NTB_EPF_SPAD_COUNT 0x2C
42
+
#define NTB_EPF_DB_ENTRY_SIZE 0x30
43
+
#define NTB_EPF_DB_DATA(n) (0x34 + (n) * 4)
44
+
#define NTB_EPF_DB_OFFSET(n) (0xB4 + (n) * 4)
45
+
46
+
#define NTB_EPF_MIN_DB_COUNT 3
47
+
#define NTB_EPF_MAX_DB_COUNT 31
48
+
#define NTB_EPF_MW_OFFSET 2
49
+
50
+
#define NTB_EPF_COMMAND_TIMEOUT 1000 /* 1 Sec */
51
+
52
+
enum pci_barno {
53
+
BAR_0,
54
+
BAR_1,
55
+
BAR_2,
56
+
BAR_3,
57
+
BAR_4,
58
+
BAR_5,
59
+
};
60
+
61
+
struct ntb_epf_dev {
62
+
struct ntb_dev ntb;
63
+
struct device *dev;
64
+
/* Mutex to protect providing commands to NTB EPF */
65
+
struct mutex cmd_lock;
66
+
67
+
enum pci_barno ctrl_reg_bar;
68
+
enum pci_barno peer_spad_reg_bar;
69
+
enum pci_barno db_reg_bar;
70
+
71
+
unsigned int mw_count;
72
+
unsigned int spad_count;
73
+
unsigned int db_count;
74
+
75
+
void __iomem *ctrl_reg;
76
+
void __iomem *db_reg;
77
+
void __iomem *peer_spad_reg;
78
+
79
+
unsigned int self_spad;
80
+
unsigned int peer_spad;
81
+
82
+
int db_val;
83
+
u64 db_valid_mask;
84
+
};
85
+
86
+
#define ntb_ndev(__ntb) container_of(__ntb, struct ntb_epf_dev, ntb)
87
+
88
+
struct ntb_epf_data {
89
+
/* BAR that contains both control region and self spad region */
90
+
enum pci_barno ctrl_reg_bar;
91
+
/* BAR that contains peer spad region */
92
+
enum pci_barno peer_spad_reg_bar;
93
+
/* BAR that contains Doorbell region and Memory window '1' */
94
+
enum pci_barno db_reg_bar;
95
+
};
96
+
97
+
static int ntb_epf_send_command(struct ntb_epf_dev *ndev, u32 command,
98
+
u32 argument)
99
+
{
100
+
ktime_t timeout;
101
+
bool timedout;
102
+
int ret = 0;
103
+
u32 status;
104
+
105
+
mutex_lock(&ndev->cmd_lock);
106
+
writel(argument, ndev->ctrl_reg + NTB_EPF_ARGUMENT);
107
+
writel(command, ndev->ctrl_reg + NTB_EPF_COMMAND);
108
+
109
+
timeout = ktime_add_ms(ktime_get(), NTB_EPF_COMMAND_TIMEOUT);
110
+
while (1) {
111
+
timedout = ktime_after(ktime_get(), timeout);
112
+
status = readw(ndev->ctrl_reg + NTB_EPF_CMD_STATUS);
113
+
114
+
if (status == COMMAND_STATUS_ERROR) {
115
+
ret = -EINVAL;
116
+
break;
117
+
}
118
+
119
+
if (status == COMMAND_STATUS_OK)
120
+
break;
121
+
122
+
if (WARN_ON(timedout)) {
123
+
ret = -ETIMEDOUT;
124
+
break;
125
+
}
126
+
127
+
usleep_range(5, 10);
128
+
}
129
+
130
+
writew(0, ndev->ctrl_reg + NTB_EPF_CMD_STATUS);
131
+
mutex_unlock(&ndev->cmd_lock);
132
+
133
+
return ret;
134
+
}
135
+
136
+
static int ntb_epf_mw_to_bar(struct ntb_epf_dev *ndev, int idx)
137
+
{
138
+
struct device *dev = ndev->dev;
139
+
140
+
if (idx < 0 || idx > ndev->mw_count) {
141
+
dev_err(dev, "Unsupported Memory Window index %d\n", idx);
142
+
return -EINVAL;
143
+
}
144
+
145
+
return idx + 2;
146
+
}
147
+
148
+
static int ntb_epf_mw_count(struct ntb_dev *ntb, int pidx)
149
+
{
150
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
151
+
struct device *dev = ndev->dev;
152
+
153
+
if (pidx != NTB_DEF_PEER_IDX) {
154
+
dev_err(dev, "Unsupported Peer ID %d\n", pidx);
155
+
return -EINVAL;
156
+
}
157
+
158
+
return ndev->mw_count;
159
+
}
160
+
161
+
static int ntb_epf_mw_get_align(struct ntb_dev *ntb, int pidx, int idx,
162
+
resource_size_t *addr_align,
163
+
resource_size_t *size_align,
164
+
resource_size_t *size_max)
165
+
{
166
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
167
+
struct device *dev = ndev->dev;
168
+
int bar;
169
+
170
+
if (pidx != NTB_DEF_PEER_IDX) {
171
+
dev_err(dev, "Unsupported Peer ID %d\n", pidx);
172
+
return -EINVAL;
173
+
}
174
+
175
+
bar = ntb_epf_mw_to_bar(ndev, idx);
176
+
if (bar < 0)
177
+
return bar;
178
+
179
+
if (addr_align)
180
+
*addr_align = SZ_4K;
181
+
182
+
if (size_align)
183
+
*size_align = 1;
184
+
185
+
if (size_max)
186
+
*size_max = pci_resource_len(ndev->ntb.pdev, bar);
187
+
188
+
return 0;
189
+
}
190
+
191
+
static u64 ntb_epf_link_is_up(struct ntb_dev *ntb,
192
+
enum ntb_speed *speed,
193
+
enum ntb_width *width)
194
+
{
195
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
196
+
u32 status;
197
+
198
+
status = readw(ndev->ctrl_reg + NTB_EPF_LINK_STATUS);
199
+
200
+
return status & LINK_STATUS_UP;
201
+
}
202
+
203
+
static u32 ntb_epf_spad_read(struct ntb_dev *ntb, int idx)
204
+
{
205
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
206
+
struct device *dev = ndev->dev;
207
+
u32 offset;
208
+
209
+
if (idx < 0 || idx >= ndev->spad_count) {
210
+
dev_err(dev, "READ: Invalid ScratchPad Index %d\n", idx);
211
+
return 0;
212
+
}
213
+
214
+
offset = readl(ndev->ctrl_reg + NTB_EPF_SPAD_OFFSET);
215
+
offset += (idx << 2);
216
+
217
+
return readl(ndev->ctrl_reg + offset);
218
+
}
219
+
220
+
static int ntb_epf_spad_write(struct ntb_dev *ntb,
221
+
int idx, u32 val)
222
+
{
223
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
224
+
struct device *dev = ndev->dev;
225
+
u32 offset;
226
+
227
+
if (idx < 0 || idx >= ndev->spad_count) {
228
+
dev_err(dev, "WRITE: Invalid ScratchPad Index %d\n", idx);
229
+
return -EINVAL;
230
+
}
231
+
232
+
offset = readl(ndev->ctrl_reg + NTB_EPF_SPAD_OFFSET);
233
+
offset += (idx << 2);
234
+
writel(val, ndev->ctrl_reg + offset);
235
+
236
+
return 0;
237
+
}
238
+
239
+
static u32 ntb_epf_peer_spad_read(struct ntb_dev *ntb, int pidx, int idx)
240
+
{
241
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
242
+
struct device *dev = ndev->dev;
243
+
u32 offset;
244
+
245
+
if (pidx != NTB_DEF_PEER_IDX) {
246
+
dev_err(dev, "Unsupported Peer ID %d\n", pidx);
247
+
return -EINVAL;
248
+
}
249
+
250
+
if (idx < 0 || idx >= ndev->spad_count) {
251
+
dev_err(dev, "WRITE: Invalid Peer ScratchPad Index %d\n", idx);
252
+
return -EINVAL;
253
+
}
254
+
255
+
offset = (idx << 2);
256
+
return readl(ndev->peer_spad_reg + offset);
257
+
}
258
+
259
+
static int ntb_epf_peer_spad_write(struct ntb_dev *ntb, int pidx,
260
+
int idx, u32 val)
261
+
{
262
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
263
+
struct device *dev = ndev->dev;
264
+
u32 offset;
265
+
266
+
if (pidx != NTB_DEF_PEER_IDX) {
267
+
dev_err(dev, "Unsupported Peer ID %d\n", pidx);
268
+
return -EINVAL;
269
+
}
270
+
271
+
if (idx < 0 || idx >= ndev->spad_count) {
272
+
dev_err(dev, "WRITE: Invalid Peer ScratchPad Index %d\n", idx);
273
+
return -EINVAL;
274
+
}
275
+
276
+
offset = (idx << 2);
277
+
writel(val, ndev->peer_spad_reg + offset);
278
+
279
+
return 0;
280
+
}
281
+
282
+
static int ntb_epf_link_enable(struct ntb_dev *ntb,
283
+
enum ntb_speed max_speed,
284
+
enum ntb_width max_width)
285
+
{
286
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
287
+
struct device *dev = ndev->dev;
288
+
int ret;
289
+
290
+
ret = ntb_epf_send_command(ndev, CMD_LINK_UP, 0);
291
+
if (ret) {
292
+
dev_err(dev, "Fail to enable link\n");
293
+
return ret;
294
+
}
295
+
296
+
return 0;
297
+
}
298
+
299
+
static int ntb_epf_link_disable(struct ntb_dev *ntb)
300
+
{
301
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
302
+
struct device *dev = ndev->dev;
303
+
int ret;
304
+
305
+
ret = ntb_epf_send_command(ndev, CMD_LINK_DOWN, 0);
306
+
if (ret) {
307
+
dev_err(dev, "Fail to disable link\n");
308
+
return ret;
309
+
}
310
+
311
+
return 0;
312
+
}
313
+
314
+
static irqreturn_t ntb_epf_vec_isr(int irq, void *dev)
315
+
{
316
+
struct ntb_epf_dev *ndev = dev;
317
+
int irq_no;
318
+
319
+
irq_no = irq - pci_irq_vector(ndev->ntb.pdev, 0);
320
+
ndev->db_val = irq_no + 1;
321
+
322
+
if (irq_no == 0)
323
+
ntb_link_event(&ndev->ntb);
324
+
else
325
+
ntb_db_event(&ndev->ntb, irq_no);
326
+
327
+
return IRQ_HANDLED;
328
+
}
329
+
330
+
static int ntb_epf_init_isr(struct ntb_epf_dev *ndev, int msi_min, int msi_max)
331
+
{
332
+
struct pci_dev *pdev = ndev->ntb.pdev;
333
+
struct device *dev = ndev->dev;
334
+
u32 argument = MSIX_ENABLE;
335
+
int irq;
336
+
int ret;
337
+
int i;
338
+
339
+
irq = pci_alloc_irq_vectors(pdev, msi_min, msi_max, PCI_IRQ_MSIX);
340
+
if (irq < 0) {
341
+
dev_dbg(dev, "Failed to get MSIX interrupts\n");
342
+
irq = pci_alloc_irq_vectors(pdev, msi_min, msi_max,
343
+
PCI_IRQ_MSI);
344
+
if (irq < 0) {
345
+
dev_err(dev, "Failed to get MSI interrupts\n");
346
+
return irq;
347
+
}
348
+
argument &= ~MSIX_ENABLE;
349
+
}
350
+
351
+
for (i = 0; i < irq; i++) {
352
+
ret = request_irq(pci_irq_vector(pdev, i), ntb_epf_vec_isr,
353
+
0, "ntb_epf", ndev);
354
+
if (ret) {
355
+
dev_err(dev, "Failed to request irq\n");
356
+
goto err_request_irq;
357
+
}
358
+
}
359
+
360
+
ndev->db_count = irq - 1;
361
+
362
+
ret = ntb_epf_send_command(ndev, CMD_CONFIGURE_DOORBELL,
363
+
argument | irq);
364
+
if (ret) {
365
+
dev_err(dev, "Failed to configure doorbell\n");
366
+
goto err_configure_db;
367
+
}
368
+
369
+
return 0;
370
+
371
+
err_configure_db:
372
+
for (i = 0; i < ndev->db_count + 1; i++)
373
+
free_irq(pci_irq_vector(pdev, i), ndev);
374
+
375
+
err_request_irq:
376
+
pci_free_irq_vectors(pdev);
377
+
378
+
return ret;
379
+
}
380
+
381
+
static int ntb_epf_peer_mw_count(struct ntb_dev *ntb)
382
+
{
383
+
return ntb_ndev(ntb)->mw_count;
384
+
}
385
+
386
+
static int ntb_epf_spad_count(struct ntb_dev *ntb)
387
+
{
388
+
return ntb_ndev(ntb)->spad_count;
389
+
}
390
+
391
+
static u64 ntb_epf_db_valid_mask(struct ntb_dev *ntb)
392
+
{
393
+
return ntb_ndev(ntb)->db_valid_mask;
394
+
}
395
+
396
+
static int ntb_epf_db_set_mask(struct ntb_dev *ntb, u64 db_bits)
397
+
{
398
+
return 0;
399
+
}
400
+
401
+
static int ntb_epf_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx,
402
+
dma_addr_t addr, resource_size_t size)
403
+
{
404
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
405
+
struct device *dev = ndev->dev;
406
+
resource_size_t mw_size;
407
+
int bar;
408
+
409
+
if (pidx != NTB_DEF_PEER_IDX) {
410
+
dev_err(dev, "Unsupported Peer ID %d\n", pidx);
411
+
return -EINVAL;
412
+
}
413
+
414
+
bar = idx + NTB_EPF_MW_OFFSET;
415
+
416
+
mw_size = pci_resource_len(ntb->pdev, bar);
417
+
418
+
if (size > mw_size) {
419
+
dev_err(dev, "Size:%pa is greater than the MW size %pa\n",
420
+
&size, &mw_size);
421
+
return -EINVAL;
422
+
}
423
+
424
+
writel(lower_32_bits(addr), ndev->ctrl_reg + NTB_EPF_LOWER_ADDR);
425
+
writel(upper_32_bits(addr), ndev->ctrl_reg + NTB_EPF_UPPER_ADDR);
426
+
writel(lower_32_bits(size), ndev->ctrl_reg + NTB_EPF_LOWER_SIZE);
427
+
writel(upper_32_bits(size), ndev->ctrl_reg + NTB_EPF_UPPER_SIZE);
428
+
ntb_epf_send_command(ndev, CMD_CONFIGURE_MW, idx);
429
+
430
+
return 0;
431
+
}
432
+
433
+
static int ntb_epf_mw_clear_trans(struct ntb_dev *ntb, int pidx, int idx)
434
+
{
435
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
436
+
struct device *dev = ndev->dev;
437
+
int ret = 0;
438
+
439
+
ntb_epf_send_command(ndev, CMD_TEARDOWN_MW, idx);
440
+
if (ret)
441
+
dev_err(dev, "Failed to teardown memory window\n");
442
+
443
+
return ret;
444
+
}
445
+
446
+
static int ntb_epf_peer_mw_get_addr(struct ntb_dev *ntb, int idx,
447
+
phys_addr_t *base, resource_size_t *size)
448
+
{
449
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
450
+
u32 offset = 0;
451
+
int bar;
452
+
453
+
if (idx == 0)
454
+
offset = readl(ndev->ctrl_reg + NTB_EPF_MW1_OFFSET);
455
+
456
+
bar = idx + NTB_EPF_MW_OFFSET;
457
+
458
+
if (base)
459
+
*base = pci_resource_start(ndev->ntb.pdev, bar) + offset;
460
+
461
+
if (size)
462
+
*size = pci_resource_len(ndev->ntb.pdev, bar) - offset;
463
+
464
+
return 0;
465
+
}
466
+
467
+
static int ntb_epf_peer_db_set(struct ntb_dev *ntb, u64 db_bits)
468
+
{
469
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
470
+
u32 interrupt_num = ffs(db_bits) + 1;
471
+
struct device *dev = ndev->dev;
472
+
u32 db_entry_size;
473
+
u32 db_offset;
474
+
u32 db_data;
475
+
476
+
if (interrupt_num > ndev->db_count) {
477
+
dev_err(dev, "DB interrupt %d greater than Max Supported %d\n",
478
+
interrupt_num, ndev->db_count);
479
+
return -EINVAL;
480
+
}
481
+
482
+
db_entry_size = readl(ndev->ctrl_reg + NTB_EPF_DB_ENTRY_SIZE);
483
+
484
+
db_data = readl(ndev->ctrl_reg + NTB_EPF_DB_DATA(interrupt_num));
485
+
db_offset = readl(ndev->ctrl_reg + NTB_EPF_DB_OFFSET(interrupt_num));
486
+
writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) +
487
+
db_offset);
488
+
489
+
return 0;
490
+
}
491
+
492
+
static u64 ntb_epf_db_read(struct ntb_dev *ntb)
493
+
{
494
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
495
+
496
+
return ndev->db_val;
497
+
}
498
+
499
+
static int ntb_epf_db_clear_mask(struct ntb_dev *ntb, u64 db_bits)
500
+
{
501
+
return 0;
502
+
}
503
+
504
+
static int ntb_epf_db_clear(struct ntb_dev *ntb, u64 db_bits)
505
+
{
506
+
struct ntb_epf_dev *ndev = ntb_ndev(ntb);
507
+
508
+
ndev->db_val = 0;
509
+
510
+
return 0;
511
+
}
512
+
513
+
static const struct ntb_dev_ops ntb_epf_ops = {
514
+
.mw_count = ntb_epf_mw_count,
515
+
.spad_count = ntb_epf_spad_count,
516
+
.peer_mw_count = ntb_epf_peer_mw_count,
517
+
.db_valid_mask = ntb_epf_db_valid_mask,
518
+
.db_set_mask = ntb_epf_db_set_mask,
519
+
.mw_set_trans = ntb_epf_mw_set_trans,
520
+
.mw_clear_trans = ntb_epf_mw_clear_trans,
521
+
.peer_mw_get_addr = ntb_epf_peer_mw_get_addr,
522
+
.link_enable = ntb_epf_link_enable,
523
+
.spad_read = ntb_epf_spad_read,
524
+
.spad_write = ntb_epf_spad_write,
525
+
.peer_spad_read = ntb_epf_peer_spad_read,
526
+
.peer_spad_write = ntb_epf_peer_spad_write,
527
+
.peer_db_set = ntb_epf_peer_db_set,
528
+
.db_read = ntb_epf_db_read,
529
+
.mw_get_align = ntb_epf_mw_get_align,
530
+
.link_is_up = ntb_epf_link_is_up,
531
+
.db_clear_mask = ntb_epf_db_clear_mask,
532
+
.db_clear = ntb_epf_db_clear,
533
+
.link_disable = ntb_epf_link_disable,
534
+
};
535
+
536
+
static inline void ntb_epf_init_struct(struct ntb_epf_dev *ndev,
537
+
struct pci_dev *pdev)
538
+
{
539
+
ndev->ntb.pdev = pdev;
540
+
ndev->ntb.topo = NTB_TOPO_NONE;
541
+
ndev->ntb.ops = &ntb_epf_ops;
542
+
}
543
+
544
+
static int ntb_epf_init_dev(struct ntb_epf_dev *ndev)
545
+
{
546
+
struct device *dev = ndev->dev;
547
+
int ret;
548
+
549
+
/* One Link interrupt and rest doorbell interrupt */
550
+
ret = ntb_epf_init_isr(ndev, NTB_EPF_MIN_DB_COUNT + 1,
551
+
NTB_EPF_MAX_DB_COUNT + 1);
552
+
if (ret) {
553
+
dev_err(dev, "Failed to init ISR\n");
554
+
return ret;
555
+
}
556
+
557
+
ndev->db_valid_mask = BIT_ULL(ndev->db_count) - 1;
558
+
ndev->mw_count = readl(ndev->ctrl_reg + NTB_EPF_MW_COUNT);
559
+
ndev->spad_count = readl(ndev->ctrl_reg + NTB_EPF_SPAD_COUNT);
560
+
561
+
return 0;
562
+
}
563
+
564
+
static int ntb_epf_init_pci(struct ntb_epf_dev *ndev,
565
+
struct pci_dev *pdev)
566
+
{
567
+
struct device *dev = ndev->dev;
568
+
int ret;
569
+
570
+
pci_set_drvdata(pdev, ndev);
571
+
572
+
ret = pci_enable_device(pdev);
573
+
if (ret) {
574
+
dev_err(dev, "Cannot enable PCI device\n");
575
+
goto err_pci_enable;
576
+
}
577
+
578
+
ret = pci_request_regions(pdev, "ntb");
579
+
if (ret) {
580
+
dev_err(dev, "Cannot obtain PCI resources\n");
581
+
goto err_pci_regions;
582
+
}
583
+
584
+
pci_set_master(pdev);
585
+
586
+
ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64));
587
+
if (ret) {
588
+
ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
589
+
if (ret) {
590
+
dev_err(dev, "Cannot set DMA mask\n");
591
+
goto err_dma_mask;
592
+
}
593
+
dev_warn(&pdev->dev, "Cannot DMA highmem\n");
594
+
}
595
+
596
+
ndev->ctrl_reg = pci_iomap(pdev, ndev->ctrl_reg_bar, 0);
597
+
if (!ndev->ctrl_reg) {
598
+
ret = -EIO;
599
+
goto err_dma_mask;
600
+
}
601
+
602
+
ndev->peer_spad_reg = pci_iomap(pdev, ndev->peer_spad_reg_bar, 0);
603
+
if (!ndev->peer_spad_reg) {
604
+
ret = -EIO;
605
+
goto err_dma_mask;
606
+
}
607
+
608
+
ndev->db_reg = pci_iomap(pdev, ndev->db_reg_bar, 0);
609
+
if (!ndev->db_reg) {
610
+
ret = -EIO;
611
+
goto err_dma_mask;
612
+
}
613
+
614
+
return 0;
615
+
616
+
err_dma_mask:
617
+
pci_clear_master(pdev);
618
+
619
+
err_pci_regions:
620
+
pci_disable_device(pdev);
621
+
622
+
err_pci_enable:
623
+
pci_set_drvdata(pdev, NULL);
624
+
625
+
return ret;
626
+
}
627
+
628
+
static void ntb_epf_deinit_pci(struct ntb_epf_dev *ndev)
629
+
{
630
+
struct pci_dev *pdev = ndev->ntb.pdev;
631
+
632
+
pci_iounmap(pdev, ndev->ctrl_reg);
633
+
pci_iounmap(pdev, ndev->peer_spad_reg);
634
+
pci_iounmap(pdev, ndev->db_reg);
635
+
636
+
pci_clear_master(pdev);
637
+
pci_release_regions(pdev);
638
+
pci_disable_device(pdev);
639
+
pci_set_drvdata(pdev, NULL);
640
+
}
641
+
642
+
static void ntb_epf_cleanup_isr(struct ntb_epf_dev *ndev)
643
+
{
644
+
struct pci_dev *pdev = ndev->ntb.pdev;
645
+
int i;
646
+
647
+
ntb_epf_send_command(ndev, CMD_TEARDOWN_DOORBELL, ndev->db_count + 1);
648
+
649
+
for (i = 0; i < ndev->db_count + 1; i++)
650
+
free_irq(pci_irq_vector(pdev, i), ndev);
651
+
pci_free_irq_vectors(pdev);
652
+
}
653
+
654
+
static int ntb_epf_pci_probe(struct pci_dev *pdev,
655
+
const struct pci_device_id *id)
656
+
{
657
+
enum pci_barno peer_spad_reg_bar = BAR_1;
658
+
enum pci_barno ctrl_reg_bar = BAR_0;
659
+
enum pci_barno db_reg_bar = BAR_2;
660
+
struct device *dev = &pdev->dev;
661
+
struct ntb_epf_data *data;
662
+
struct ntb_epf_dev *ndev;
663
+
int ret;
664
+
665
+
if (pci_is_bridge(pdev))
666
+
return -ENODEV;
667
+
668
+
ndev = devm_kzalloc(dev, sizeof(*ndev), GFP_KERNEL);
669
+
if (!ndev)
670
+
return -ENOMEM;
671
+
672
+
data = (struct ntb_epf_data *)id->driver_data;
673
+
if (data) {
674
+
if (data->peer_spad_reg_bar)
675
+
peer_spad_reg_bar = data->peer_spad_reg_bar;
676
+
if (data->ctrl_reg_bar)
677
+
ctrl_reg_bar = data->ctrl_reg_bar;
678
+
if (data->db_reg_bar)
679
+
db_reg_bar = data->db_reg_bar;
680
+
}
681
+
682
+
ndev->peer_spad_reg_bar = peer_spad_reg_bar;
683
+
ndev->ctrl_reg_bar = ctrl_reg_bar;
684
+
ndev->db_reg_bar = db_reg_bar;
685
+
ndev->dev = dev;
686
+
687
+
ntb_epf_init_struct(ndev, pdev);
688
+
mutex_init(&ndev->cmd_lock);
689
+
690
+
ret = ntb_epf_init_pci(ndev, pdev);
691
+
if (ret) {
692
+
dev_err(dev, "Failed to init PCI\n");
693
+
return ret;
694
+
}
695
+
696
+
ret = ntb_epf_init_dev(ndev);
697
+
if (ret) {
698
+
dev_err(dev, "Failed to init device\n");
699
+
goto err_init_dev;
700
+
}
701
+
702
+
ret = ntb_register_device(&ndev->ntb);
703
+
if (ret) {
704
+
dev_err(dev, "Failed to register NTB device\n");
705
+
goto err_register_dev;
706
+
}
707
+
708
+
return 0;
709
+
710
+
err_register_dev:
711
+
ntb_epf_cleanup_isr(ndev);
712
+
713
+
err_init_dev:
714
+
ntb_epf_deinit_pci(ndev);
715
+
716
+
return ret;
717
+
}
718
+
719
+
static void ntb_epf_pci_remove(struct pci_dev *pdev)
720
+
{
721
+
struct ntb_epf_dev *ndev = pci_get_drvdata(pdev);
722
+
723
+
ntb_unregister_device(&ndev->ntb);
724
+
ntb_epf_cleanup_isr(ndev);
725
+
ntb_epf_deinit_pci(ndev);
726
+
}
727
+
728
+
static const struct ntb_epf_data j721e_data = {
729
+
.ctrl_reg_bar = BAR_0,
730
+
.peer_spad_reg_bar = BAR_1,
731
+
.db_reg_bar = BAR_2,
732
+
};
733
+
734
+
static const struct pci_device_id ntb_epf_pci_tbl[] = {
735
+
{
736
+
PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_J721E),
737
+
.class = PCI_CLASS_MEMORY_RAM << 8, .class_mask = 0xffff00,
738
+
.driver_data = (kernel_ulong_t)&j721e_data,
739
+
},
740
+
{ },
741
+
};
742
+
743
+
static struct pci_driver ntb_epf_pci_driver = {
744
+
.name = KBUILD_MODNAME,
745
+
.id_table = ntb_epf_pci_tbl,
746
+
.probe = ntb_epf_pci_probe,
747
+
.remove = ntb_epf_pci_remove,
748
+
};
749
+
module_pci_driver(ntb_epf_pci_driver);
750
+
751
+
MODULE_DESCRIPTION("PCI ENDPOINT NTB HOST DRIVER");
752
+
MODULE_AUTHOR("Kishon Vijay Abraham I <kishon@ti.com>");
753
+
MODULE_LICENSE("GPL v2");
+54
-6
drivers/pci/controller/cadence/pcie-cadence-ep.c
+54
-6
drivers/pci/controller/cadence/pcie-cadence-ep.c
···
382
382
return 0;
383
383
}
384
384
385
+
static int cdns_pcie_ep_map_msi_irq(struct pci_epc *epc, u8 fn,
386
+
phys_addr_t addr, u8 interrupt_num,
387
+
u32 entry_size, u32 *msi_data,
388
+
u32 *msi_addr_offset)
389
+
{
390
+
struct cdns_pcie_ep *ep = epc_get_drvdata(epc);
391
+
u32 cap = CDNS_PCIE_EP_FUNC_MSI_CAP_OFFSET;
392
+
struct cdns_pcie *pcie = &ep->pcie;
393
+
u64 pci_addr, pci_addr_mask = 0xff;
394
+
u16 flags, mme, data, data_mask;
395
+
u8 msi_count;
396
+
int ret;
397
+
int i;
398
+
399
+
/* Check whether the MSI feature has been enabled by the PCI host. */
400
+
flags = cdns_pcie_ep_fn_readw(pcie, fn, cap + PCI_MSI_FLAGS);
401
+
if (!(flags & PCI_MSI_FLAGS_ENABLE))
402
+
return -EINVAL;
403
+
404
+
/* Get the number of enabled MSIs */
405
+
mme = (flags & PCI_MSI_FLAGS_QSIZE) >> 4;
406
+
msi_count = 1 << mme;
407
+
if (!interrupt_num || interrupt_num > msi_count)
408
+
return -EINVAL;
409
+
410
+
/* Compute the data value to be written. */
411
+
data_mask = msi_count - 1;
412
+
data = cdns_pcie_ep_fn_readw(pcie, fn, cap + PCI_MSI_DATA_64);
413
+
data = data & ~data_mask;
414
+
415
+
/* Get the PCI address where to write the data into. */
416
+
pci_addr = cdns_pcie_ep_fn_readl(pcie, fn, cap + PCI_MSI_ADDRESS_HI);
417
+
pci_addr <<= 32;
418
+
pci_addr |= cdns_pcie_ep_fn_readl(pcie, fn, cap + PCI_MSI_ADDRESS_LO);
419
+
pci_addr &= GENMASK_ULL(63, 2);
420
+
421
+
for (i = 0; i < interrupt_num; i++) {
422
+
ret = cdns_pcie_ep_map_addr(epc, fn, addr,
423
+
pci_addr & ~pci_addr_mask,
424
+
entry_size);
425
+
if (ret)
426
+
return ret;
427
+
addr = addr + entry_size;
428
+
}
429
+
430
+
*msi_data = data;
431
+
*msi_addr_offset = pci_addr & pci_addr_mask;
432
+
433
+
return 0;
434
+
}
435
+
385
436
static int cdns_pcie_ep_send_msix_irq(struct cdns_pcie_ep *ep, u8 fn,
386
437
u16 interrupt_num)
387
438
{
···
506
455
struct cdns_pcie_ep *ep = epc_get_drvdata(epc);
507
456
struct cdns_pcie *pcie = &ep->pcie;
508
457
struct device *dev = pcie->dev;
509
-
struct pci_epf *epf;
510
-
u32 cfg;
511
458
int ret;
512
459
513
460
/*
514
461
* BIT(0) is hardwired to 1, hence function 0 is always enabled
515
462
* and can't be disabled anyway.
516
463
*/
517
-
cfg = BIT(0);
518
-
list_for_each_entry(epf, &epc->pci_epf, list)
519
-
cfg |= BIT(epf->func_no);
520
-
cdns_pcie_writel(pcie, CDNS_PCIE_LM_EP_FUNC_CFG, cfg);
464
+
cdns_pcie_writel(pcie, CDNS_PCIE_LM_EP_FUNC_CFG, epc->function_num_map);
521
465
522
466
ret = cdns_pcie_start_link(pcie);
523
467
if (ret) {
···
527
481
.linkup_notifier = false,
528
482
.msi_capable = true,
529
483
.msix_capable = true,
484
+
.align = 256,
530
485
};
531
486
532
487
static const struct pci_epc_features*
···
547
500
.set_msix = cdns_pcie_ep_set_msix,
548
501
.get_msix = cdns_pcie_ep_get_msix,
549
502
.raise_irq = cdns_pcie_ep_raise_irq,
503
+
.map_msi_irq = cdns_pcie_ep_map_msi_irq,
550
504
.start = cdns_pcie_ep_start,
551
505
.get_features = cdns_pcie_ep_get_features,
552
506
};
+13
drivers/pci/endpoint/functions/Kconfig
+13
drivers/pci/endpoint/functions/Kconfig
···
12
12
for PCI Endpoint.
13
13
14
14
If in doubt, say "N" to disable Endpoint test driver.
15
+
16
+
config PCI_EPF_NTB
17
+
tristate "PCI Endpoint NTB driver"
18
+
depends on PCI_ENDPOINT
19
+
select CONFIGFS_FS
20
+
help
21
+
Select this configuration option to enable the Non-Transparent
22
+
Bridge (NTB) driver for PCI Endpoint. NTB driver implements NTB
23
+
controller functionality using multiple PCIe endpoint instances.
24
+
It can support NTB endpoint function devices created using
25
+
device tree.
26
+
27
+
If in doubt, say "N" to disable Endpoint NTB driver.
+1
drivers/pci/endpoint/functions/Makefile
+1
drivers/pci/endpoint/functions/Makefile
+2128
drivers/pci/endpoint/functions/pci-epf-ntb.c
+2128
drivers/pci/endpoint/functions/pci-epf-ntb.c
···
1
+
// SPDX-License-Identifier: GPL-2.0
2
+
/**
3
+
* Endpoint Function Driver to implement Non-Transparent Bridge functionality
4
+
*
5
+
* Copyright (C) 2020 Texas Instruments
6
+
* Author: Kishon Vijay Abraham I <kishon@ti.com>
7
+
*/
8
+
9
+
/*
10
+
* The PCI NTB function driver configures the SoC with multiple PCIe Endpoint
11
+
* (EP) controller instances (see diagram below) in such a way that
12
+
* transactions from one EP controller are routed to the other EP controller.
13
+
* Once PCI NTB function driver configures the SoC with multiple EP instances,
14
+
* HOST1 and HOST2 can communicate with each other using SoC as a bridge.
15
+
*
16
+
* +-------------+ +-------------+
17
+
* | | | |
18
+
* | HOST1 | | HOST2 |
19
+
* | | | |
20
+
* +------^------+ +------^------+
21
+
* | |
22
+
* | |
23
+
* +---------|-------------------------------------------------|---------+
24
+
* | +------v------+ +------v------+ |
25
+
* | | | | | |
26
+
* | | EP | | EP | |
27
+
* | | CONTROLLER1 | | CONTROLLER2 | |
28
+
* | | <-----------------------------------> | |
29
+
* | | | | | |
30
+
* | | | | | |
31
+
* | | | SoC With Multiple EP Instances | | |
32
+
* | | | (Configured using NTB Function) | | |
33
+
* | +-------------+ +-------------+ |
34
+
* +---------------------------------------------------------------------+
35
+
*/
36
+
37
+
#include <linux/delay.h>
38
+
#include <linux/io.h>
39
+
#include <linux/module.h>
40
+
#include <linux/slab.h>
41
+
42
+
#include <linux/pci-epc.h>
43
+
#include <linux/pci-epf.h>
44
+
45
+
static struct workqueue_struct *kpcintb_workqueue;
46
+
47
+
#define COMMAND_CONFIGURE_DOORBELL 1
48
+
#define COMMAND_TEARDOWN_DOORBELL 2
49
+
#define COMMAND_CONFIGURE_MW 3
50
+
#define COMMAND_TEARDOWN_MW 4
51
+
#define COMMAND_LINK_UP 5
52
+
#define COMMAND_LINK_DOWN 6
53
+
54
+
#define COMMAND_STATUS_OK 1
55
+
#define COMMAND_STATUS_ERROR 2
56
+
57
+
#define LINK_STATUS_UP BIT(0)
58
+
59
+
#define SPAD_COUNT 64
60
+
#define DB_COUNT 4
61
+
#define NTB_MW_OFFSET 2
62
+
#define DB_COUNT_MASK GENMASK(15, 0)
63
+
#define MSIX_ENABLE BIT(16)
64
+
#define MAX_DB_COUNT 32
65
+
#define MAX_MW 4
66
+
67
+
enum epf_ntb_bar {
68
+
BAR_CONFIG,
69
+
BAR_PEER_SPAD,
70
+
BAR_DB_MW1,
71
+
BAR_MW2,
72
+
BAR_MW3,
73
+
BAR_MW4,
74
+
};
75
+
76
+
struct epf_ntb {
77
+
u32 num_mws;
78
+
u32 db_count;
79
+
u32 spad_count;
80
+
struct pci_epf *epf;
81
+
u64 mws_size[MAX_MW];
82
+
struct config_group group;
83
+
struct epf_ntb_epc *epc[2];
84
+
};
85
+
86
+
#define to_epf_ntb(epf_group) container_of((epf_group), struct epf_ntb, group)
87
+
88
+
struct epf_ntb_epc {
89
+
u8 func_no;
90
+
bool linkup;
91
+
bool is_msix;
92
+
int msix_bar;
93
+
u32 spad_size;
94
+
struct pci_epc *epc;
95
+
struct epf_ntb *epf_ntb;
96
+
void __iomem *mw_addr[6];
97
+
size_t msix_table_offset;
98
+
struct epf_ntb_ctrl *reg;
99
+
struct pci_epf_bar *epf_bar;
100
+
enum pci_barno epf_ntb_bar[6];
101
+
struct delayed_work cmd_handler;
102
+
enum pci_epc_interface_type type;
103
+
const struct pci_epc_features *epc_features;
104
+
};
105
+
106
+
struct epf_ntb_ctrl {
107
+
u32 command;
108
+
u32 argument;
109
+
u16 command_status;
110
+
u16 link_status;
111
+
u32 topology;
112
+
u64 addr;
113
+
u64 size;
114
+
u32 num_mws;
115
+
u32 mw1_offset;
116
+
u32 spad_offset;
117
+
u32 spad_count;
118
+
u32 db_entry_size;
119
+
u32 db_data[MAX_DB_COUNT];
120
+
u32 db_offset[MAX_DB_COUNT];
121
+
} __packed;
122
+
123
+
static struct pci_epf_header epf_ntb_header = {
124
+
.vendorid = PCI_ANY_ID,
125
+
.deviceid = PCI_ANY_ID,
126
+
.baseclass_code = PCI_BASE_CLASS_MEMORY,
127
+
.interrupt_pin = PCI_INTERRUPT_INTA,
128
+
};
129
+
130
+
/**
131
+
* epf_ntb_link_up() - Raise link_up interrupt to both the hosts
132
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
133
+
* @link_up: true or false indicating Link is UP or Down
134
+
*
135
+
* Once NTB function in HOST1 and the NTB function in HOST2 invoke
136
+
* ntb_link_enable(), this NTB function driver will trigger a link event to
137
+
* the NTB client in both the hosts.
138
+
*/
139
+
static int epf_ntb_link_up(struct epf_ntb *ntb, bool link_up)
140
+
{
141
+
enum pci_epc_interface_type type;
142
+
enum pci_epc_irq_type irq_type;
143
+
struct epf_ntb_epc *ntb_epc;
144
+
struct epf_ntb_ctrl *ctrl;
145
+
struct pci_epc *epc;
146
+
bool is_msix;
147
+
u8 func_no;
148
+
int ret;
149
+
150
+
for (type = PRIMARY_INTERFACE; type <= SECONDARY_INTERFACE; type++) {
151
+
ntb_epc = ntb->epc[type];
152
+
epc = ntb_epc->epc;
153
+
func_no = ntb_epc->func_no;
154
+
is_msix = ntb_epc->is_msix;
155
+
ctrl = ntb_epc->reg;
156
+
if (link_up)
157
+
ctrl->link_status |= LINK_STATUS_UP;
158
+
else
159
+
ctrl->link_status &= ~LINK_STATUS_UP;
160
+
irq_type = is_msix ? PCI_EPC_IRQ_MSIX : PCI_EPC_IRQ_MSI;
161
+
ret = pci_epc_raise_irq(epc, func_no, irq_type, 1);
162
+
if (ret) {
163
+
dev_err(&epc->dev,
164
+
"%s intf: Failed to raise Link Up IRQ\n",
165
+
pci_epc_interface_string(type));
166
+
return ret;
167
+
}
168
+
}
169
+
170
+
return 0;
171
+
}
172
+
173
+
/**
174
+
* epf_ntb_configure_mw() - Configure the Outbound Address Space for one host
175
+
* to access the memory window of other host
176
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
177
+
* @type: PRIMARY interface or SECONDARY interface
178
+
* @mw: Index of the memory window (either 0, 1, 2 or 3)
179
+
*
180
+
* +-----------------+ +---->+----------------+-----------+-----------------+
181
+
* | BAR0 | | | Doorbell 1 +-----------> MSI|X ADDRESS 1 |
182
+
* +-----------------+ | +----------------+ +-----------------+
183
+
* | BAR1 | | | Doorbell 2 +---------+ | |
184
+
* +-----------------+----+ +----------------+ | | |
185
+
* | BAR2 | | Doorbell 3 +-------+ | +-----------------+
186
+
* +-----------------+----+ +----------------+ | +-> MSI|X ADDRESS 2 |
187
+
* | BAR3 | | | Doorbell 4 +-----+ | +-----------------+
188
+
* +-----------------+ | |----------------+ | | | |
189
+
* | BAR4 | | | | | | +-----------------+
190
+
* +-----------------+ | | MW1 +---+ | +-->+ MSI|X ADDRESS 3||
191
+
* | BAR5 | | | | | | +-----------------+
192
+
* +-----------------+ +---->-----------------+ | | | |
193
+
* EP CONTROLLER 1 | | | | +-----------------+
194
+
* | | | +---->+ MSI|X ADDRESS 4 |
195
+
* +----------------+ | +-----------------+
196
+
* (A) EP CONTROLLER 2 | | |
197
+
* (OB SPACE) | | |
198
+
* +-------> MW1 |
199
+
* | |
200
+
* | |
201
+
* (B) +-----------------+
202
+
* | |
203
+
* | |
204
+
* | |
205
+
* | |
206
+
* | |
207
+
* +-----------------+
208
+
* PCI Address Space
209
+
* (Managed by HOST2)
210
+
*
211
+
* This function performs stage (B) in the above diagram (see MW1) i.e., map OB
212
+
* address space of memory window to PCI address space.
213
+
*
214
+
* This operation requires 3 parameters
215
+
* 1) Address in the outbound address space
216
+
* 2) Address in the PCI Address space
217
+
* 3) Size of the address region to be mapped
218
+
*
219
+
* The address in the outbound address space (for MW1, MW2, MW3 and MW4) is
220
+
* stored in epf_bar corresponding to BAR_DB_MW1 for MW1 and BAR_MW2, BAR_MW3
221
+
* BAR_MW4 for rest of the BARs of epf_ntb_epc that is connected to HOST1. This
222
+
* is populated in epf_ntb_alloc_peer_mem() in this driver.
223
+
*
224
+
* The address and size of the PCI address region that has to be mapped would
225
+
* be provided by HOST2 in ctrl->addr and ctrl->size of epf_ntb_epc that is
226
+
* connected to HOST2.
227
+
*
228
+
* Please note Memory window1 (MW1) and Doorbell registers together will be
229
+
* mapped to a single BAR (BAR2) above for 32-bit BARs. The exact BAR that's
230
+
* used for Memory window (MW) can be obtained from epf_ntb_bar[BAR_DB_MW1],
231
+
* epf_ntb_bar[BAR_MW2], epf_ntb_bar[BAR_MW2], epf_ntb_bar[BAR_MW2].
232
+
*/
233
+
static int epf_ntb_configure_mw(struct epf_ntb *ntb,
234
+
enum pci_epc_interface_type type, u32 mw)
235
+
{
236
+
struct epf_ntb_epc *peer_ntb_epc, *ntb_epc;
237
+
struct pci_epf_bar *peer_epf_bar;
238
+
enum pci_barno peer_barno;
239
+
struct epf_ntb_ctrl *ctrl;
240
+
phys_addr_t phys_addr;
241
+
struct pci_epc *epc;
242
+
u64 addr, size;
243
+
int ret = 0;
244
+
u8 func_no;
245
+
246
+
ntb_epc = ntb->epc[type];
247
+
epc = ntb_epc->epc;
248
+
249
+
peer_ntb_epc = ntb->epc[!type];
250
+
peer_barno = peer_ntb_epc->epf_ntb_bar[mw + NTB_MW_OFFSET];
251
+
peer_epf_bar = &peer_ntb_epc->epf_bar[peer_barno];
252
+
253
+
phys_addr = peer_epf_bar->phys_addr;
254
+
ctrl = ntb_epc->reg;
255
+
addr = ctrl->addr;
256
+
size = ctrl->size;
257
+
if (mw + NTB_MW_OFFSET == BAR_DB_MW1)
258
+
phys_addr += ctrl->mw1_offset;
259
+
260
+
if (size > ntb->mws_size[mw]) {
261
+
dev_err(&epc->dev,
262
+
"%s intf: MW: %d Req Sz:%llxx > Supported Sz:%llx\n",
263
+
pci_epc_interface_string(type), mw, size,
264
+
ntb->mws_size[mw]);
265
+
ret = -EINVAL;
266
+
goto err_invalid_size;
267
+
}
268
+
269
+
func_no = ntb_epc->func_no;
270
+
271
+
ret = pci_epc_map_addr(epc, func_no, phys_addr, addr, size);
272
+
if (ret)
273
+
dev_err(&epc->dev,
274
+
"%s intf: Failed to map memory window %d address\n",
275
+
pci_epc_interface_string(type), mw);
276
+
277
+
err_invalid_size:
278
+
279
+
return ret;
280
+
}
281
+
282
+
/**
283
+
* epf_ntb_teardown_mw() - Teardown the configured OB ATU
284
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
285
+
* @type: PRIMARY interface or SECONDARY interface
286
+
* @mw: Index of the memory window (either 0, 1, 2 or 3)
287
+
*
288
+
* Teardown the configured OB ATU configured in epf_ntb_configure_mw() using
289
+
* pci_epc_unmap_addr()
290
+
*/
291
+
static void epf_ntb_teardown_mw(struct epf_ntb *ntb,
292
+
enum pci_epc_interface_type type, u32 mw)
293
+
{
294
+
struct epf_ntb_epc *peer_ntb_epc, *ntb_epc;
295
+
struct pci_epf_bar *peer_epf_bar;
296
+
enum pci_barno peer_barno;
297
+
struct epf_ntb_ctrl *ctrl;
298
+
phys_addr_t phys_addr;
299
+
struct pci_epc *epc;
300
+
u8 func_no;
301
+
302
+
ntb_epc = ntb->epc[type];
303
+
epc = ntb_epc->epc;
304
+
305
+
peer_ntb_epc = ntb->epc[!type];
306
+
peer_barno = peer_ntb_epc->epf_ntb_bar[mw + NTB_MW_OFFSET];
307
+
peer_epf_bar = &peer_ntb_epc->epf_bar[peer_barno];
308
+
309
+
phys_addr = peer_epf_bar->phys_addr;
310
+
ctrl = ntb_epc->reg;
311
+
if (mw + NTB_MW_OFFSET == BAR_DB_MW1)
312
+
phys_addr += ctrl->mw1_offset;
313
+
func_no = ntb_epc->func_no;
314
+
315
+
pci_epc_unmap_addr(epc, func_no, phys_addr);
316
+
}
317
+
318
+
/**
319
+
* epf_ntb_configure_msi() - Map OB address space to MSI address
320
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
321
+
* @type: PRIMARY interface or SECONDARY interface
322
+
* @db_count: Number of doorbell interrupts to map
323
+
*
324
+
*+-----------------+ +----->+----------------+-----------+-----------------+
325
+
*| BAR0 | | | Doorbell 1 +---+-------> MSI ADDRESS |
326
+
*+-----------------+ | +----------------+ | +-----------------+
327
+
*| BAR1 | | | Doorbell 2 +---+ | |
328
+
*+-----------------+----+ +----------------+ | | |
329
+
*| BAR2 | | Doorbell 3 +---+ | |
330
+
*+-----------------+----+ +----------------+ | | |
331
+
*| BAR3 | | | Doorbell 4 +---+ | |
332
+
*+-----------------+ | |----------------+ | |
333
+
*| BAR4 | | | | | |
334
+
*+-----------------+ | | MW1 | | |
335
+
*| BAR5 | | | | | |
336
+
*+-----------------+ +----->-----------------+ | |
337
+
* EP CONTROLLER 1 | | | |
338
+
* | | | |
339
+
* +----------------+ +-----------------+
340
+
* (A) EP CONTROLLER 2 | |
341
+
* (OB SPACE) | |
342
+
* | MW1 |
343
+
* | |
344
+
* | |
345
+
* (B) +-----------------+
346
+
* | |
347
+
* | |
348
+
* | |
349
+
* | |
350
+
* | |
351
+
* +-----------------+
352
+
* PCI Address Space
353
+
* (Managed by HOST2)
354
+
*
355
+
*
356
+
* This function performs stage (B) in the above diagram (see Doorbell 1,
357
+
* Doorbell 2, Doorbell 3, Doorbell 4) i.e map OB address space corresponding to
358
+
* doorbell to MSI address in PCI address space.
359
+
*
360
+
* This operation requires 3 parameters
361
+
* 1) Address reserved for doorbell in the outbound address space
362
+
* 2) MSI-X address in the PCIe Address space
363
+
* 3) Number of MSI-X interrupts that has to be configured
364
+
*
365
+
* The address in the outbound address space (for the Doorbell) is stored in
366
+
* epf_bar corresponding to BAR_DB_MW1 of epf_ntb_epc that is connected to
367
+
* HOST1. This is populated in epf_ntb_alloc_peer_mem() in this driver along
368
+
* with address for MW1.
369
+
*
370
+
* pci_epc_map_msi_irq() takes the MSI address from MSI capability register
371
+
* and maps the OB address (obtained in epf_ntb_alloc_peer_mem()) to the MSI
372
+
* address.
373
+
*
374
+
* epf_ntb_configure_msi() also stores the MSI data to raise each interrupt
375
+
* in db_data of the peer's control region. This helps the peer to raise
376
+
* doorbell of the other host by writing db_data to the BAR corresponding to
377
+
* BAR_DB_MW1.
378
+
*/
379
+
static int epf_ntb_configure_msi(struct epf_ntb *ntb,
380
+
enum pci_epc_interface_type type, u16 db_count)
381
+
{
382
+
struct epf_ntb_epc *peer_ntb_epc, *ntb_epc;
383
+
u32 db_entry_size, db_data, db_offset;
384
+
struct pci_epf_bar *peer_epf_bar;
385
+
struct epf_ntb_ctrl *peer_ctrl;
386
+
enum pci_barno peer_barno;
387
+
phys_addr_t phys_addr;
388
+
struct pci_epc *epc;
389
+
u8 func_no;
390
+
int ret, i;
391
+
392
+
ntb_epc = ntb->epc[type];
393
+
epc = ntb_epc->epc;
394
+
395
+
peer_ntb_epc = ntb->epc[!type];
396
+
peer_barno = peer_ntb_epc->epf_ntb_bar[BAR_DB_MW1];
397
+
peer_epf_bar = &peer_ntb_epc->epf_bar[peer_barno];
398
+
peer_ctrl = peer_ntb_epc->reg;
399
+
db_entry_size = peer_ctrl->db_entry_size;
400
+
401
+
phys_addr = peer_epf_bar->phys_addr;
402
+
func_no = ntb_epc->func_no;
403
+
404
+
ret = pci_epc_map_msi_irq(epc, func_no, phys_addr, db_count,
405
+
db_entry_size, &db_data, &db_offset);
406
+
if (ret) {
407
+
dev_err(&epc->dev, "%s intf: Failed to map MSI IRQ\n",
408
+
pci_epc_interface_string(type));
409
+
return ret;
410
+
}
411
+
412
+
for (i = 0; i < db_count; i++) {
413
+
peer_ctrl->db_data[i] = db_data | i;
414
+
peer_ctrl->db_offset[i] = db_offset;
415
+
}
416
+
417
+
return 0;
418
+
}
419
+
420
+
/**
421
+
* epf_ntb_configure_msix() - Map OB address space to MSI-X address
422
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
423
+
* @type: PRIMARY interface or SECONDARY interface
424
+
* @db_count: Number of doorbell interrupts to map
425
+
*
426
+
*+-----------------+ +----->+----------------+-----------+-----------------+
427
+
*| BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 |
428
+
*+-----------------+ | +----------------+ +-----------------+
429
+
*| BAR1 | | | Doorbell 2 +---------+ | |
430
+
*+-----------------+----+ +----------------+ | | |
431
+
*| BAR2 | | Doorbell 3 +-------+ | +-----------------+
432
+
*+-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 |
433
+
*| BAR3 | | | Doorbell 4 +-----+ | +-----------------+
434
+
*+-----------------+ | |----------------+ | | | |
435
+
*| BAR4 | | | | | | +-----------------+
436
+
*+-----------------+ | | MW1 + | +-->+ MSI-X ADDRESS 3||
437
+
*| BAR5 | | | | | +-----------------+
438
+
*+-----------------+ +----->-----------------+ | | |
439
+
* EP CONTROLLER 1 | | | +-----------------+
440
+
* | | +---->+ MSI-X ADDRESS 4 |
441
+
* +----------------+ +-----------------+
442
+
* (A) EP CONTROLLER 2 | |
443
+
* (OB SPACE) | |
444
+
* | MW1 |
445
+
* | |
446
+
* | |
447
+
* (B) +-----------------+
448
+
* | |
449
+
* | |
450
+
* | |
451
+
* | |
452
+
* | |
453
+
* +-----------------+
454
+
* PCI Address Space
455
+
* (Managed by HOST2)
456
+
*
457
+
* This function performs stage (B) in the above diagram (see Doorbell 1,
458
+
* Doorbell 2, Doorbell 3, Doorbell 4) i.e map OB address space corresponding to
459
+
* doorbell to MSI-X address in PCI address space.
460
+
*
461
+
* This operation requires 3 parameters
462
+
* 1) Address reserved for doorbell in the outbound address space
463
+
* 2) MSI-X address in the PCIe Address space
464
+
* 3) Number of MSI-X interrupts that has to be configured
465
+
*
466
+
* The address in the outbound address space (for the Doorbell) is stored in
467
+
* epf_bar corresponding to BAR_DB_MW1 of epf_ntb_epc that is connected to
468
+
* HOST1. This is populated in epf_ntb_alloc_peer_mem() in this driver along
469
+
* with address for MW1.
470
+
*
471
+
* The MSI-X address is in the MSI-X table of EP CONTROLLER 2 and
472
+
* the count of doorbell is in ctrl->argument of epf_ntb_epc that is connected
473
+
* to HOST2. MSI-X table is stored memory mapped to ntb_epc->msix_bar and the
474
+
* offset is in ntb_epc->msix_table_offset. From this epf_ntb_configure_msix()
475
+
* gets the MSI-X address and data.
476
+
*
477
+
* epf_ntb_configure_msix() also stores the MSI-X data to raise each interrupt
478
+
* in db_data of the peer's control region. This helps the peer to raise
479
+
* doorbell of the other host by writing db_data to the BAR corresponding to
480
+
* BAR_DB_MW1.
481
+
*/
482
+
static int epf_ntb_configure_msix(struct epf_ntb *ntb,
483
+
enum pci_epc_interface_type type,
484
+
u16 db_count)
485
+
{
486
+
const struct pci_epc_features *epc_features;
487
+
struct epf_ntb_epc *peer_ntb_epc, *ntb_epc;
488
+
struct pci_epf_bar *peer_epf_bar, *epf_bar;
489
+
struct pci_epf_msix_tbl *msix_tbl;
490
+
struct epf_ntb_ctrl *peer_ctrl;
491
+
u32 db_entry_size, msg_data;
492
+
enum pci_barno peer_barno;
493
+
phys_addr_t phys_addr;
494
+
struct pci_epc *epc;
495
+
size_t align;
496
+
u64 msg_addr;
497
+
u8 func_no;
498
+
int ret, i;
499
+
500
+
ntb_epc = ntb->epc[type];
501
+
epc = ntb_epc->epc;
502
+
503
+
epf_bar = &ntb_epc->epf_bar[ntb_epc->msix_bar];
504
+
msix_tbl = epf_bar->addr + ntb_epc->msix_table_offset;
505
+
506
+
peer_ntb_epc = ntb->epc[!type];
507
+
peer_barno = peer_ntb_epc->epf_ntb_bar[BAR_DB_MW1];
508
+
peer_epf_bar = &peer_ntb_epc->epf_bar[peer_barno];
509
+
phys_addr = peer_epf_bar->phys_addr;
510
+
peer_ctrl = peer_ntb_epc->reg;
511
+
epc_features = ntb_epc->epc_features;
512
+
align = epc_features->align;
513
+
514
+
func_no = ntb_epc->func_no;
515
+
db_entry_size = peer_ctrl->db_entry_size;
516
+
517
+
for (i = 0; i < db_count; i++) {
518
+
msg_addr = ALIGN_DOWN(msix_tbl[i].msg_addr, align);
519
+
msg_data = msix_tbl[i].msg_data;
520
+
ret = pci_epc_map_addr(epc, func_no, phys_addr, msg_addr,
521
+
db_entry_size);
522
+
if (ret) {
523
+
dev_err(&epc->dev,
524
+
"%s intf: Failed to configure MSI-X IRQ\n",
525
+
pci_epc_interface_string(type));
526
+
return ret;
527
+
}
528
+
phys_addr = phys_addr + db_entry_size;
529
+
peer_ctrl->db_data[i] = msg_data;
530
+
peer_ctrl->db_offset[i] = msix_tbl[i].msg_addr & (align - 1);
531
+
}
532
+
ntb_epc->is_msix = true;
533
+
534
+
return 0;
535
+
}
536
+
537
+
/**
538
+
* epf_ntb_configure_db() - Configure the Outbound Address Space for one host
539
+
* to ring the doorbell of other host
540
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
541
+
* @type: PRIMARY interface or SECONDARY interface
542
+
* @db_count: Count of the number of doorbells that has to be configured
543
+
* @msix: Indicates whether MSI-X or MSI should be used
544
+
*
545
+
* Invokes epf_ntb_configure_msix() or epf_ntb_configure_msi() required for
546
+
* one HOST to ring the doorbell of other HOST.
547
+
*/
548
+
static int epf_ntb_configure_db(struct epf_ntb *ntb,
549
+
enum pci_epc_interface_type type,
550
+
u16 db_count, bool msix)
551
+
{
552
+
struct epf_ntb_epc *ntb_epc;
553
+
struct pci_epc *epc;
554
+
int ret;
555
+
556
+
if (db_count > MAX_DB_COUNT)
557
+
return -EINVAL;
558
+
559
+
ntb_epc = ntb->epc[type];
560
+
epc = ntb_epc->epc;
561
+
562
+
if (msix)
563
+
ret = epf_ntb_configure_msix(ntb, type, db_count);
564
+
else
565
+
ret = epf_ntb_configure_msi(ntb, type, db_count);
566
+
567
+
if (ret)
568
+
dev_err(&epc->dev, "%s intf: Failed to configure DB\n",
569
+
pci_epc_interface_string(type));
570
+
571
+
return ret;
572
+
}
573
+
574
+
/**
575
+
* epf_ntb_teardown_db() - Unmap address in OB address space to MSI/MSI-X
576
+
* address
577
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
578
+
* @type: PRIMARY interface or SECONDARY interface
579
+
*
580
+
* Invoke pci_epc_unmap_addr() to unmap OB address to MSI/MSI-X address.
581
+
*/
582
+
static void
583
+
epf_ntb_teardown_db(struct epf_ntb *ntb, enum pci_epc_interface_type type)
584
+
{
585
+
struct epf_ntb_epc *peer_ntb_epc, *ntb_epc;
586
+
struct pci_epf_bar *peer_epf_bar;
587
+
enum pci_barno peer_barno;
588
+
phys_addr_t phys_addr;
589
+
struct pci_epc *epc;
590
+
u8 func_no;
591
+
592
+
ntb_epc = ntb->epc[type];
593
+
epc = ntb_epc->epc;
594
+
595
+
peer_ntb_epc = ntb->epc[!type];
596
+
peer_barno = peer_ntb_epc->epf_ntb_bar[BAR_DB_MW1];
597
+
peer_epf_bar = &peer_ntb_epc->epf_bar[peer_barno];
598
+
phys_addr = peer_epf_bar->phys_addr;
599
+
func_no = ntb_epc->func_no;
600
+
601
+
pci_epc_unmap_addr(epc, func_no, phys_addr);
602
+
}
603
+
604
+
/**
605
+
* epf_ntb_cmd_handler() - Handle commands provided by the NTB Host
606
+
* @work: work_struct for the two epf_ntb_epc (PRIMARY and SECONDARY)
607
+
*
608
+
* Workqueue function that gets invoked for the two epf_ntb_epc
609
+
* periodically (once every 5ms) to see if it has received any commands
610
+
* from NTB host. The host can send commands to configure doorbell or
611
+
* configure memory window or to update link status.
612
+
*/
613
+
static void epf_ntb_cmd_handler(struct work_struct *work)
614
+
{
615
+
enum pci_epc_interface_type type;
616
+
struct epf_ntb_epc *ntb_epc;
617
+
struct epf_ntb_ctrl *ctrl;
618
+
u32 command, argument;
619
+
struct epf_ntb *ntb;
620
+
struct device *dev;
621
+
u16 db_count;
622
+
bool is_msix;
623
+
int ret;
624
+
625
+
ntb_epc = container_of(work, struct epf_ntb_epc, cmd_handler.work);
626
+
ctrl = ntb_epc->reg;
627
+
command = ctrl->command;
628
+
if (!command)
629
+
goto reset_handler;
630
+
argument = ctrl->argument;
631
+
632
+
ctrl->command = 0;
633
+
ctrl->argument = 0;
634
+
635
+
ctrl = ntb_epc->reg;
636
+
type = ntb_epc->type;
637
+
ntb = ntb_epc->epf_ntb;
638
+
dev = &ntb->epf->dev;
639
+
640
+
switch (command) {
641
+
case COMMAND_CONFIGURE_DOORBELL:
642
+
db_count = argument & DB_COUNT_MASK;
643
+
is_msix = argument & MSIX_ENABLE;
644
+
ret = epf_ntb_configure_db(ntb, type, db_count, is_msix);
645
+
if (ret < 0)
646
+
ctrl->command_status = COMMAND_STATUS_ERROR;
647
+
else
648
+
ctrl->command_status = COMMAND_STATUS_OK;
649
+
break;
650
+
case COMMAND_TEARDOWN_DOORBELL:
651
+
epf_ntb_teardown_db(ntb, type);
652
+
ctrl->command_status = COMMAND_STATUS_OK;
653
+
break;
654
+
case COMMAND_CONFIGURE_MW:
655
+
ret = epf_ntb_configure_mw(ntb, type, argument);
656
+
if (ret < 0)
657
+
ctrl->command_status = COMMAND_STATUS_ERROR;
658
+
else
659
+
ctrl->command_status = COMMAND_STATUS_OK;
660
+
break;
661
+
case COMMAND_TEARDOWN_MW:
662
+
epf_ntb_teardown_mw(ntb, type, argument);
663
+
ctrl->command_status = COMMAND_STATUS_OK;
664
+
break;
665
+
case COMMAND_LINK_UP:
666
+
ntb_epc->linkup = true;
667
+
if (ntb->epc[PRIMARY_INTERFACE]->linkup &&
668
+
ntb->epc[SECONDARY_INTERFACE]->linkup) {
669
+
ret = epf_ntb_link_up(ntb, true);
670
+
if (ret < 0)
671
+
ctrl->command_status = COMMAND_STATUS_ERROR;
672
+
else
673
+
ctrl->command_status = COMMAND_STATUS_OK;
674
+
goto reset_handler;
675
+
}
676
+
ctrl->command_status = COMMAND_STATUS_OK;
677
+
break;
678
+
case COMMAND_LINK_DOWN:
679
+
ntb_epc->linkup = false;
680
+
ret = epf_ntb_link_up(ntb, false);
681
+
if (ret < 0)
682
+
ctrl->command_status = COMMAND_STATUS_ERROR;
683
+
else
684
+
ctrl->command_status = COMMAND_STATUS_OK;
685
+
break;
686
+
default:
687
+
dev_err(dev, "%s intf UNKNOWN command: %d\n",
688
+
pci_epc_interface_string(type), command);
689
+
break;
690
+
}
691
+
692
+
reset_handler:
693
+
queue_delayed_work(kpcintb_workqueue, &ntb_epc->cmd_handler,
694
+
msecs_to_jiffies(5));
695
+
}
696
+
697
+
/**
698
+
* epf_ntb_peer_spad_bar_clear() - Clear Peer Scratchpad BAR
699
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
700
+
*
701
+
*+-----------------+------->+------------------+ +-----------------+
702
+
*| BAR0 | | CONFIG REGION | | BAR0 |
703
+
*+-----------------+----+ +------------------+<-------+-----------------+
704
+
*| BAR1 | | |SCRATCHPAD REGION | | BAR1 |
705
+
*+-----------------+ +-->+------------------+<-------+-----------------+
706
+
*| BAR2 | Local Memory | BAR2 |
707
+
*+-----------------+ +-----------------+
708
+
*| BAR3 | | BAR3 |
709
+
*+-----------------+ +-----------------+
710
+
*| BAR4 | | BAR4 |
711
+
*+-----------------+ +-----------------+
712
+
*| BAR5 | | BAR5 |
713
+
*+-----------------+ +-----------------+
714
+
* EP CONTROLLER 1 EP CONTROLLER 2
715
+
*
716
+
* Clear BAR1 of EP CONTROLLER 2 which contains the HOST2's peer scratchpad
717
+
* region. While BAR1 is the default peer scratchpad BAR, an NTB could have
718
+
* other BARs for peer scratchpad (because of 64-bit BARs or reserved BARs).
719
+
* This function can get the exact BAR used for peer scratchpad from
720
+
* epf_ntb_bar[BAR_PEER_SPAD].
721
+
*
722
+
* Since HOST2's peer scratchpad is also HOST1's self scratchpad, this function
723
+
* gets the address of peer scratchpad from
724
+
* peer_ntb_epc->epf_ntb_bar[BAR_CONFIG].
725
+
*/
726
+
static void epf_ntb_peer_spad_bar_clear(struct epf_ntb_epc *ntb_epc)
727
+
{
728
+
struct pci_epf_bar *epf_bar;
729
+
enum pci_barno barno;
730
+
struct pci_epc *epc;
731
+
u8 func_no;
732
+
733
+
epc = ntb_epc->epc;
734
+
func_no = ntb_epc->func_no;
735
+
barno = ntb_epc->epf_ntb_bar[BAR_PEER_SPAD];
736
+
epf_bar = &ntb_epc->epf_bar[barno];
737
+
pci_epc_clear_bar(epc, func_no, epf_bar);
738
+
}
739
+
740
+
/**
741
+
* epf_ntb_peer_spad_bar_set() - Set peer scratchpad BAR
742
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
743
+
*
744
+
*+-----------------+------->+------------------+ +-----------------+
745
+
*| BAR0 | | CONFIG REGION | | BAR0 |
746
+
*+-----------------+----+ +------------------+<-------+-----------------+
747
+
*| BAR1 | | |SCRATCHPAD REGION | | BAR1 |
748
+
*+-----------------+ +-->+------------------+<-------+-----------------+
749
+
*| BAR2 | Local Memory | BAR2 |
750
+
*+-----------------+ +-----------------+
751
+
*| BAR3 | | BAR3 |
752
+
*+-----------------+ +-----------------+
753
+
*| BAR4 | | BAR4 |
754
+
*+-----------------+ +-----------------+
755
+
*| BAR5 | | BAR5 |
756
+
*+-----------------+ +-----------------+
757
+
* EP CONTROLLER 1 EP CONTROLLER 2
758
+
*
759
+
* Set BAR1 of EP CONTROLLER 2 which contains the HOST2's peer scratchpad
760
+
* region. While BAR1 is the default peer scratchpad BAR, an NTB could have
761
+
* other BARs for peer scratchpad (because of 64-bit BARs or reserved BARs).
762
+
* This function can get the exact BAR used for peer scratchpad from
763
+
* epf_ntb_bar[BAR_PEER_SPAD].
764
+
*
765
+
* Since HOST2's peer scratchpad is also HOST1's self scratchpad, this function
766
+
* gets the address of peer scratchpad from
767
+
* peer_ntb_epc->epf_ntb_bar[BAR_CONFIG].
768
+
*/
769
+
static int epf_ntb_peer_spad_bar_set(struct epf_ntb *ntb,
770
+
enum pci_epc_interface_type type)
771
+
{
772
+
struct epf_ntb_epc *peer_ntb_epc, *ntb_epc;
773
+
struct pci_epf_bar *peer_epf_bar, *epf_bar;
774
+
enum pci_barno peer_barno, barno;
775
+
u32 peer_spad_offset;
776
+
struct pci_epc *epc;
777
+
struct device *dev;
778
+
u8 func_no;
779
+
int ret;
780
+
781
+
dev = &ntb->epf->dev;
782
+
783
+
peer_ntb_epc = ntb->epc[!type];
784
+
peer_barno = peer_ntb_epc->epf_ntb_bar[BAR_CONFIG];
785
+
peer_epf_bar = &peer_ntb_epc->epf_bar[peer_barno];
786
+
787
+
ntb_epc = ntb->epc[type];
788
+
barno = ntb_epc->epf_ntb_bar[BAR_PEER_SPAD];
789
+
epf_bar = &ntb_epc->epf_bar[barno];
790
+
func_no = ntb_epc->func_no;
791
+
epc = ntb_epc->epc;
792
+
793
+
peer_spad_offset = peer_ntb_epc->reg->spad_offset;
794
+
epf_bar->phys_addr = peer_epf_bar->phys_addr + peer_spad_offset;
795
+
epf_bar->size = peer_ntb_epc->spad_size;
796
+
epf_bar->barno = barno;
797
+
epf_bar->flags = PCI_BASE_ADDRESS_MEM_TYPE_32;
798
+
799
+
ret = pci_epc_set_bar(epc, func_no, epf_bar);
800
+
if (ret) {
801
+
dev_err(dev, "%s intf: peer SPAD BAR set failed\n",
802
+
pci_epc_interface_string(type));
803
+
return ret;
804
+
}
805
+
806
+
return 0;
807
+
}
808
+
809
+
/**
810
+
* epf_ntb_config_sspad_bar_clear() - Clear Config + Self scratchpad BAR
811
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
812
+
*
813
+
* +-----------------+------->+------------------+ +-----------------+
814
+
* | BAR0 | | CONFIG REGION | | BAR0 |
815
+
* +-----------------+----+ +------------------+<-------+-----------------+
816
+
* | BAR1 | | |SCRATCHPAD REGION | | BAR1 |
817
+
* +-----------------+ +-->+------------------+<-------+-----------------+
818
+
* | BAR2 | Local Memory | BAR2 |
819
+
* +-----------------+ +-----------------+
820
+
* | BAR3 | | BAR3 |
821
+
* +-----------------+ +-----------------+
822
+
* | BAR4 | | BAR4 |
823
+
* +-----------------+ +-----------------+
824
+
* | BAR5 | | BAR5 |
825
+
* +-----------------+ +-----------------+
826
+
* EP CONTROLLER 1 EP CONTROLLER 2
827
+
*
828
+
* Clear BAR0 of EP CONTROLLER 1 which contains the HOST1's config and
829
+
* self scratchpad region (removes inbound ATU configuration). While BAR0 is
830
+
* the default self scratchpad BAR, an NTB could have other BARs for self
831
+
* scratchpad (because of reserved BARs). This function can get the exact BAR
832
+
* used for self scratchpad from epf_ntb_bar[BAR_CONFIG].
833
+
*
834
+
* Please note the self scratchpad region and config region is combined to
835
+
* a single region and mapped using the same BAR. Also note HOST2's peer
836
+
* scratchpad is HOST1's self scratchpad.
837
+
*/
838
+
static void epf_ntb_config_sspad_bar_clear(struct epf_ntb_epc *ntb_epc)
839
+
{
840
+
struct pci_epf_bar *epf_bar;
841
+
enum pci_barno barno;
842
+
struct pci_epc *epc;
843
+
u8 func_no;
844
+
845
+
epc = ntb_epc->epc;
846
+
func_no = ntb_epc->func_no;
847
+
barno = ntb_epc->epf_ntb_bar[BAR_CONFIG];
848
+
epf_bar = &ntb_epc->epf_bar[barno];
849
+
pci_epc_clear_bar(epc, func_no, epf_bar);
850
+
}
851
+
852
+
/**
853
+
* epf_ntb_config_sspad_bar_set() - Set Config + Self scratchpad BAR
854
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
855
+
*
856
+
* +-----------------+------->+------------------+ +-----------------+
857
+
* | BAR0 | | CONFIG REGION | | BAR0 |
858
+
* +-----------------+----+ +------------------+<-------+-----------------+
859
+
* | BAR1 | | |SCRATCHPAD REGION | | BAR1 |
860
+
* +-----------------+ +-->+------------------+<-------+-----------------+
861
+
* | BAR2 | Local Memory | BAR2 |
862
+
* +-----------------+ +-----------------+
863
+
* | BAR3 | | BAR3 |
864
+
* +-----------------+ +-----------------+
865
+
* | BAR4 | | BAR4 |
866
+
* +-----------------+ +-----------------+
867
+
* | BAR5 | | BAR5 |
868
+
* +-----------------+ +-----------------+
869
+
* EP CONTROLLER 1 EP CONTROLLER 2
870
+
*
871
+
* Map BAR0 of EP CONTROLLER 1 which contains the HOST1's config and
872
+
* self scratchpad region. While BAR0 is the default self scratchpad BAR, an
873
+
* NTB could have other BARs for self scratchpad (because of reserved BARs).
874
+
* This function can get the exact BAR used for self scratchpad from
875
+
* epf_ntb_bar[BAR_CONFIG].
876
+
*
877
+
* Please note the self scratchpad region and config region is combined to
878
+
* a single region and mapped using the same BAR. Also note HOST2's peer
879
+
* scratchpad is HOST1's self scratchpad.
880
+
*/
881
+
static int epf_ntb_config_sspad_bar_set(struct epf_ntb_epc *ntb_epc)
882
+
{
883
+
struct pci_epf_bar *epf_bar;
884
+
enum pci_barno barno;
885
+
struct epf_ntb *ntb;
886
+
struct pci_epc *epc;
887
+
struct device *dev;
888
+
u8 func_no;
889
+
int ret;
890
+
891
+
ntb = ntb_epc->epf_ntb;
892
+
dev = &ntb->epf->dev;
893
+
894
+
epc = ntb_epc->epc;
895
+
func_no = ntb_epc->func_no;
896
+
barno = ntb_epc->epf_ntb_bar[BAR_CONFIG];
897
+
epf_bar = &ntb_epc->epf_bar[barno];
898
+
899
+
ret = pci_epc_set_bar(epc, func_no, epf_bar);
900
+
if (ret) {
901
+
dev_err(dev, "%s inft: Config/Status/SPAD BAR set failed\n",
902
+
pci_epc_interface_string(ntb_epc->type));
903
+
return ret;
904
+
}
905
+
906
+
return 0;
907
+
}
908
+
909
+
/**
910
+
* epf_ntb_config_spad_bar_free() - Free the physical memory associated with
911
+
* config + scratchpad region
912
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
913
+
*
914
+
* +-----------------+------->+------------------+ +-----------------+
915
+
* | BAR0 | | CONFIG REGION | | BAR0 |
916
+
* +-----------------+----+ +------------------+<-------+-----------------+
917
+
* | BAR1 | | |SCRATCHPAD REGION | | BAR1 |
918
+
* +-----------------+ +-->+------------------+<-------+-----------------+
919
+
* | BAR2 | Local Memory | BAR2 |
920
+
* +-----------------+ +-----------------+
921
+
* | BAR3 | | BAR3 |
922
+
* +-----------------+ +-----------------+
923
+
* | BAR4 | | BAR4 |
924
+
* +-----------------+ +-----------------+
925
+
* | BAR5 | | BAR5 |
926
+
* +-----------------+ +-----------------+
927
+
* EP CONTROLLER 1 EP CONTROLLER 2
928
+
*
929
+
* Free the Local Memory mentioned in the above diagram. After invoking this
930
+
* function, any of config + self scratchpad region of HOST1 or peer scratchpad
931
+
* region of HOST2 should not be accessed.
932
+
*/
933
+
static void epf_ntb_config_spad_bar_free(struct epf_ntb *ntb)
934
+
{
935
+
enum pci_epc_interface_type type;
936
+
struct epf_ntb_epc *ntb_epc;
937
+
enum pci_barno barno;
938
+
struct pci_epf *epf;
939
+
940
+
epf = ntb->epf;
941
+
for (type = PRIMARY_INTERFACE; type <= SECONDARY_INTERFACE; type++) {
942
+
ntb_epc = ntb->epc[type];
943
+
barno = ntb_epc->epf_ntb_bar[BAR_CONFIG];
944
+
if (ntb_epc->reg)
945
+
pci_epf_free_space(epf, ntb_epc->reg, barno, type);
946
+
}
947
+
}
948
+
949
+
/**
950
+
* epf_ntb_config_spad_bar_alloc() - Allocate memory for config + scratchpad
951
+
* region
952
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
953
+
* @type: PRIMARY interface or SECONDARY interface
954
+
*
955
+
* +-----------------+------->+------------------+ +-----------------+
956
+
* | BAR0 | | CONFIG REGION | | BAR0 |
957
+
* +-----------------+----+ +------------------+<-------+-----------------+
958
+
* | BAR1 | | |SCRATCHPAD REGION | | BAR1 |
959
+
* +-----------------+ +-->+------------------+<-------+-----------------+
960
+
* | BAR2 | Local Memory | BAR2 |
961
+
* +-----------------+ +-----------------+
962
+
* | BAR3 | | BAR3 |
963
+
* +-----------------+ +-----------------+
964
+
* | BAR4 | | BAR4 |
965
+
* +-----------------+ +-----------------+
966
+
* | BAR5 | | BAR5 |
967
+
* +-----------------+ +-----------------+
968
+
* EP CONTROLLER 1 EP CONTROLLER 2
969
+
*
970
+
* Allocate the Local Memory mentioned in the above diagram. The size of
971
+
* CONFIG REGION is sizeof(struct epf_ntb_ctrl) and size of SCRATCHPAD REGION
972
+
* is obtained from "spad-count" configfs entry.
973
+
*
974
+
* The size of both config region and scratchpad region has to be aligned,
975
+
* since the scratchpad region will also be mapped as PEER SCRATCHPAD of
976
+
* other host using a separate BAR.
977
+
*/
978
+
static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb,
979
+
enum pci_epc_interface_type type)
980
+
{
981
+
const struct pci_epc_features *peer_epc_features, *epc_features;
982
+
struct epf_ntb_epc *peer_ntb_epc, *ntb_epc;
983
+
size_t msix_table_size, pba_size, align;
984
+
enum pci_barno peer_barno, barno;
985
+
struct epf_ntb_ctrl *ctrl;
986
+
u32 spad_size, ctrl_size;
987
+
u64 size, peer_size;
988
+
struct pci_epf *epf;
989
+
struct device *dev;
990
+
bool msix_capable;
991
+
u32 spad_count;
992
+
void *base;
993
+
994
+
epf = ntb->epf;
995
+
dev = &epf->dev;
996
+
ntb_epc = ntb->epc[type];
997
+
998
+
epc_features = ntb_epc->epc_features;
999
+
barno = ntb_epc->epf_ntb_bar[BAR_CONFIG];
1000
+
size = epc_features->bar_fixed_size[barno];
1001
+
align = epc_features->align;
1002
+
1003
+
peer_ntb_epc = ntb->epc[!type];
1004
+
peer_epc_features = peer_ntb_epc->epc_features;
1005
+
peer_barno = ntb_epc->epf_ntb_bar[BAR_PEER_SPAD];
1006
+
peer_size = peer_epc_features->bar_fixed_size[peer_barno];
1007
+
1008
+
/* Check if epc_features is populated incorrectly */
1009
+
if ((!IS_ALIGNED(size, align)))
1010
+
return -EINVAL;
1011
+
1012
+
spad_count = ntb->spad_count;
1013
+
1014
+
ctrl_size = sizeof(struct epf_ntb_ctrl);
1015
+
spad_size = spad_count * 4;
1016
+
1017
+
msix_capable = epc_features->msix_capable;
1018
+
if (msix_capable) {
1019
+
msix_table_size = PCI_MSIX_ENTRY_SIZE * ntb->db_count;
1020
+
ctrl_size = ALIGN(ctrl_size, 8);
1021
+
ntb_epc->msix_table_offset = ctrl_size;
1022
+
ntb_epc->msix_bar = barno;
1023
+
/* Align to QWORD or 8 Bytes */
1024
+
pba_size = ALIGN(DIV_ROUND_UP(ntb->db_count, 8), 8);
1025
+
ctrl_size = ctrl_size + msix_table_size + pba_size;
1026
+
}
1027
+
1028
+
if (!align) {
1029
+
ctrl_size = roundup_pow_of_two(ctrl_size);
1030
+
spad_size = roundup_pow_of_two(spad_size);
1031
+
} else {
1032
+
ctrl_size = ALIGN(ctrl_size, align);
1033
+
spad_size = ALIGN(spad_size, align);
1034
+
}
1035
+
1036
+
if (peer_size) {
1037
+
if (peer_size < spad_size)
1038
+
spad_count = peer_size / 4;
1039
+
spad_size = peer_size;
1040
+
}
1041
+
1042
+
/*
1043
+
* In order to make sure SPAD offset is aligned to its size,
1044
+
* expand control region size to the size of SPAD if SPAD size
1045
+
* is greater than control region size.
1046
+
*/
1047
+
if (spad_size > ctrl_size)
1048
+
ctrl_size = spad_size;
1049
+
1050
+
if (!size)
1051
+
size = ctrl_size + spad_size;
1052
+
else if (size < ctrl_size + spad_size)
1053
+
return -EINVAL;
1054
+
1055
+
base = pci_epf_alloc_space(epf, size, barno, align, type);
1056
+
if (!base) {
1057
+
dev_err(dev, "%s intf: Config/Status/SPAD alloc region fail\n",
1058
+
pci_epc_interface_string(type));
1059
+
return -ENOMEM;
1060
+
}
1061
+
1062
+
ntb_epc->reg = base;
1063
+
1064
+
ctrl = ntb_epc->reg;
1065
+
ctrl->spad_offset = ctrl_size;
1066
+
ctrl->spad_count = spad_count;
1067
+
ctrl->num_mws = ntb->num_mws;
1068
+
ctrl->db_entry_size = align ? align : 4;
1069
+
ntb_epc->spad_size = spad_size;
1070
+
1071
+
return 0;
1072
+
}
1073
+
1074
+
/**
1075
+
* epf_ntb_config_spad_bar_alloc_interface() - Allocate memory for config +
1076
+
* scratchpad region for each of PRIMARY and SECONDARY interface
1077
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1078
+
*
1079
+
* Wrapper for epf_ntb_config_spad_bar_alloc() which allocates memory for
1080
+
* config + scratchpad region for a specific interface
1081
+
*/
1082
+
static int epf_ntb_config_spad_bar_alloc_interface(struct epf_ntb *ntb)
1083
+
{
1084
+
enum pci_epc_interface_type type;
1085
+
struct device *dev;
1086
+
int ret;
1087
+
1088
+
dev = &ntb->epf->dev;
1089
+
1090
+
for (type = PRIMARY_INTERFACE; type <= SECONDARY_INTERFACE; type++) {
1091
+
ret = epf_ntb_config_spad_bar_alloc(ntb, type);
1092
+
if (ret) {
1093
+
dev_err(dev, "%s intf: Config/SPAD BAR alloc failed\n",
1094
+
pci_epc_interface_string(type));
1095
+
return ret;
1096
+
}
1097
+
}
1098
+
1099
+
return 0;
1100
+
}
1101
+
1102
+
/**
1103
+
* epf_ntb_free_peer_mem() - Free memory allocated in peers outbound address
1104
+
* space
1105
+
* @ntb_epc: EPC associated with one of the HOST which holds peers outbound
1106
+
* address regions
1107
+
*
1108
+
* +-----------------+ +---->+----------------+-----------+-----------------+
1109
+
* | BAR0 | | | Doorbell 1 +-----------> MSI|X ADDRESS 1 |
1110
+
* +-----------------+ | +----------------+ +-----------------+
1111
+
* | BAR1 | | | Doorbell 2 +---------+ | |
1112
+
* +-----------------+----+ +----------------+ | | |
1113
+
* | BAR2 | | Doorbell 3 +-------+ | +-----------------+
1114
+
* +-----------------+----+ +----------------+ | +-> MSI|X ADDRESS 2 |
1115
+
* | BAR3 | | | Doorbell 4 +-----+ | +-----------------+
1116
+
* +-----------------+ | |----------------+ | | | |
1117
+
* | BAR4 | | | | | | +-----------------+
1118
+
* +-----------------+ | | MW1 +---+ | +-->+ MSI|X ADDRESS 3||
1119
+
* | BAR5 | | | | | | +-----------------+
1120
+
* +-----------------+ +---->-----------------+ | | | |
1121
+
* EP CONTROLLER 1 | | | | +-----------------+
1122
+
* | | | +---->+ MSI|X ADDRESS 4 |
1123
+
* +----------------+ | +-----------------+
1124
+
* (A) EP CONTROLLER 2 | | |
1125
+
* (OB SPACE) | | |
1126
+
* +-------> MW1 |
1127
+
* | |
1128
+
* | |
1129
+
* (B) +-----------------+
1130
+
* | |
1131
+
* | |
1132
+
* | |
1133
+
* | |
1134
+
* | |
1135
+
* +-----------------+
1136
+
* PCI Address Space
1137
+
* (Managed by HOST2)
1138
+
*
1139
+
* Free memory allocated in EP CONTROLLER 2 (OB SPACE) in the above diagram.
1140
+
* It'll free Doorbell 1, Doorbell 2, Doorbell 3, Doorbell 4, MW1 (and MW2, MW3,
1141
+
* MW4).
1142
+
*/
1143
+
static void epf_ntb_free_peer_mem(struct epf_ntb_epc *ntb_epc)
1144
+
{
1145
+
struct pci_epf_bar *epf_bar;
1146
+
void __iomem *mw_addr;
1147
+
phys_addr_t phys_addr;
1148
+
enum epf_ntb_bar bar;
1149
+
enum pci_barno barno;
1150
+
struct pci_epc *epc;
1151
+
size_t size;
1152
+
1153
+
epc = ntb_epc->epc;
1154
+
1155
+
for (bar = BAR_DB_MW1; bar < BAR_MW4; bar++) {
1156
+
barno = ntb_epc->epf_ntb_bar[bar];
1157
+
mw_addr = ntb_epc->mw_addr[barno];
1158
+
epf_bar = &ntb_epc->epf_bar[barno];
1159
+
phys_addr = epf_bar->phys_addr;
1160
+
size = epf_bar->size;
1161
+
if (mw_addr) {
1162
+
pci_epc_mem_free_addr(epc, phys_addr, mw_addr, size);
1163
+
ntb_epc->mw_addr[barno] = NULL;
1164
+
}
1165
+
}
1166
+
}
1167
+
1168
+
/**
1169
+
* epf_ntb_db_mw_bar_clear() - Clear doorbell and memory BAR
1170
+
* @ntb_epc: EPC associated with one of the HOST which holds peer's outbound
1171
+
* address
1172
+
*
1173
+
* +-----------------+ +---->+----------------+-----------+-----------------+
1174
+
* | BAR0 | | | Doorbell 1 +-----------> MSI|X ADDRESS 1 |
1175
+
* +-----------------+ | +----------------+ +-----------------+
1176
+
* | BAR1 | | | Doorbell 2 +---------+ | |
1177
+
* +-----------------+----+ +----------------+ | | |
1178
+
* | BAR2 | | Doorbell 3 +-------+ | +-----------------+
1179
+
* +-----------------+----+ +----------------+ | +-> MSI|X ADDRESS 2 |
1180
+
* | BAR3 | | | Doorbell 4 +-----+ | +-----------------+
1181
+
* +-----------------+ | |----------------+ | | | |
1182
+
* | BAR4 | | | | | | +-----------------+
1183
+
* +-----------------+ | | MW1 +---+ | +-->+ MSI|X ADDRESS 3||
1184
+
* | BAR5 | | | | | | +-----------------+
1185
+
* +-----------------+ +---->-----------------+ | | | |
1186
+
* EP CONTROLLER 1 | | | | +-----------------+
1187
+
* | | | +---->+ MSI|X ADDRESS 4 |
1188
+
* +----------------+ | +-----------------+
1189
+
* (A) EP CONTROLLER 2 | | |
1190
+
* (OB SPACE) | | |
1191
+
* +-------> MW1 |
1192
+
* | |
1193
+
* | |
1194
+
* (B) +-----------------+
1195
+
* | |
1196
+
* | |
1197
+
* | |
1198
+
* | |
1199
+
* | |
1200
+
* +-----------------+
1201
+
* PCI Address Space
1202
+
* (Managed by HOST2)
1203
+
*
1204
+
* Clear doorbell and memory BARs (remove inbound ATU configuration). In the above
1205
+
* diagram it clears BAR2 TO BAR5 of EP CONTROLLER 1 (Doorbell BAR, MW1 BAR, MW2
1206
+
* BAR, MW3 BAR and MW4 BAR).
1207
+
*/
1208
+
static void epf_ntb_db_mw_bar_clear(struct epf_ntb_epc *ntb_epc)
1209
+
{
1210
+
struct pci_epf_bar *epf_bar;
1211
+
enum epf_ntb_bar bar;
1212
+
enum pci_barno barno;
1213
+
struct pci_epc *epc;
1214
+
u8 func_no;
1215
+
1216
+
epc = ntb_epc->epc;
1217
+
1218
+
func_no = ntb_epc->func_no;
1219
+
1220
+
for (bar = BAR_DB_MW1; bar < BAR_MW4; bar++) {
1221
+
barno = ntb_epc->epf_ntb_bar[bar];
1222
+
epf_bar = &ntb_epc->epf_bar[barno];
1223
+
pci_epc_clear_bar(epc, func_no, epf_bar);
1224
+
}
1225
+
}
1226
+
1227
+
/**
1228
+
* epf_ntb_db_mw_bar_cleanup() - Clear doorbell/memory BAR and free memory
1229
+
* allocated in peers outbound address space
1230
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1231
+
* @type: PRIMARY interface or SECONDARY interface
1232
+
*
1233
+
* Wrapper for epf_ntb_db_mw_bar_clear() to clear HOST1's BAR and
1234
+
* epf_ntb_free_peer_mem() which frees up HOST2 outbound memory.
1235
+
*/
1236
+
static void epf_ntb_db_mw_bar_cleanup(struct epf_ntb *ntb,
1237
+
enum pci_epc_interface_type type)
1238
+
{
1239
+
struct epf_ntb_epc *peer_ntb_epc, *ntb_epc;
1240
+
1241
+
ntb_epc = ntb->epc[type];
1242
+
peer_ntb_epc = ntb->epc[!type];
1243
+
1244
+
epf_ntb_db_mw_bar_clear(ntb_epc);
1245
+
epf_ntb_free_peer_mem(peer_ntb_epc);
1246
+
}
1247
+
1248
+
/**
1249
+
* epf_ntb_configure_interrupt() - Configure MSI/MSI-X capaiblity
1250
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1251
+
* @type: PRIMARY interface or SECONDARY interface
1252
+
*
1253
+
* Configure MSI/MSI-X capability for each interface with number of
1254
+
* interrupts equal to "db_count" configfs entry.
1255
+
*/
1256
+
static int epf_ntb_configure_interrupt(struct epf_ntb *ntb,
1257
+
enum pci_epc_interface_type type)
1258
+
{
1259
+
const struct pci_epc_features *epc_features;
1260
+
bool msix_capable, msi_capable;
1261
+
struct epf_ntb_epc *ntb_epc;
1262
+
struct pci_epc *epc;
1263
+
struct device *dev;
1264
+
u32 db_count;
1265
+
u8 func_no;
1266
+
int ret;
1267
+
1268
+
ntb_epc = ntb->epc[type];
1269
+
dev = &ntb->epf->dev;
1270
+
1271
+
epc_features = ntb_epc->epc_features;
1272
+
msix_capable = epc_features->msix_capable;
1273
+
msi_capable = epc_features->msi_capable;
1274
+
1275
+
if (!(msix_capable || msi_capable)) {
1276
+
dev_err(dev, "MSI or MSI-X is required for doorbell\n");
1277
+
return -EINVAL;
1278
+
}
1279
+
1280
+
func_no = ntb_epc->func_no;
1281
+
1282
+
db_count = ntb->db_count;
1283
+
if (db_count > MAX_DB_COUNT) {
1284
+
dev_err(dev, "DB count cannot be more than %d\n", MAX_DB_COUNT);
1285
+
return -EINVAL;
1286
+
}
1287
+
1288
+
ntb->db_count = db_count;
1289
+
epc = ntb_epc->epc;
1290
+
1291
+
if (msi_capable) {
1292
+
ret = pci_epc_set_msi(epc, func_no, db_count);
1293
+
if (ret) {
1294
+
dev_err(dev, "%s intf: MSI configuration failed\n",
1295
+
pci_epc_interface_string(type));
1296
+
return ret;
1297
+
}
1298
+
}
1299
+
1300
+
if (msix_capable) {
1301
+
ret = pci_epc_set_msix(epc, func_no, db_count,
1302
+
ntb_epc->msix_bar,
1303
+
ntb_epc->msix_table_offset);
1304
+
if (ret) {
1305
+
dev_err(dev, "MSI configuration failed\n");
1306
+
return ret;
1307
+
}
1308
+
}
1309
+
1310
+
return 0;
1311
+
}
1312
+
1313
+
/**
1314
+
* epf_ntb_alloc_peer_mem() - Allocate memory in peer's outbound address space
1315
+
* @ntb_epc: EPC associated with one of the HOST whose BAR holds peer's outbound
1316
+
* address
1317
+
* @bar: BAR of @ntb_epc in for which memory has to be allocated (could be
1318
+
* BAR_DB_MW1, BAR_MW2, BAR_MW3, BAR_MW4)
1319
+
* @peer_ntb_epc: EPC associated with HOST whose outbound address space is
1320
+
* used by @ntb_epc
1321
+
* @size: Size of the address region that has to be allocated in peers OB SPACE
1322
+
*
1323
+
*
1324
+
* +-----------------+ +---->+----------------+-----------+-----------------+
1325
+
* | BAR0 | | | Doorbell 1 +-----------> MSI|X ADDRESS 1 |
1326
+
* +-----------------+ | +----------------+ +-----------------+
1327
+
* | BAR1 | | | Doorbell 2 +---------+ | |
1328
+
* +-----------------+----+ +----------------+ | | |
1329
+
* | BAR2 | | Doorbell 3 +-------+ | +-----------------+
1330
+
* +-----------------+----+ +----------------+ | +-> MSI|X ADDRESS 2 |
1331
+
* | BAR3 | | | Doorbell 4 +-----+ | +-----------------+
1332
+
* +-----------------+ | |----------------+ | | | |
1333
+
* | BAR4 | | | | | | +-----------------+
1334
+
* +-----------------+ | | MW1 +---+ | +-->+ MSI|X ADDRESS 3||
1335
+
* | BAR5 | | | | | | +-----------------+
1336
+
* +-----------------+ +---->-----------------+ | | | |
1337
+
* EP CONTROLLER 1 | | | | +-----------------+
1338
+
* | | | +---->+ MSI|X ADDRESS 4 |
1339
+
* +----------------+ | +-----------------+
1340
+
* (A) EP CONTROLLER 2 | | |
1341
+
* (OB SPACE) | | |
1342
+
* +-------> MW1 |
1343
+
* | |
1344
+
* | |
1345
+
* (B) +-----------------+
1346
+
* | |
1347
+
* | |
1348
+
* | |
1349
+
* | |
1350
+
* | |
1351
+
* +-----------------+
1352
+
* PCI Address Space
1353
+
* (Managed by HOST2)
1354
+
*
1355
+
* Allocate memory in OB space of EP CONTROLLER 2 in the above diagram. Allocate
1356
+
* for Doorbell 1, Doorbell 2, Doorbell 3, Doorbell 4, MW1 (and MW2, MW3, MW4).
1357
+
*/
1358
+
static int epf_ntb_alloc_peer_mem(struct device *dev,
1359
+
struct epf_ntb_epc *ntb_epc,
1360
+
enum epf_ntb_bar bar,
1361
+
struct epf_ntb_epc *peer_ntb_epc,
1362
+
size_t size)
1363
+
{
1364
+
const struct pci_epc_features *epc_features;
1365
+
struct pci_epf_bar *epf_bar;
1366
+
struct pci_epc *peer_epc;
1367
+
phys_addr_t phys_addr;
1368
+
void __iomem *mw_addr;
1369
+
enum pci_barno barno;
1370
+
size_t align;
1371
+
1372
+
epc_features = ntb_epc->epc_features;
1373
+
align = epc_features->align;
1374
+
1375
+
if (size < 128)
1376
+
size = 128;
1377
+
1378
+
if (align)
1379
+
size = ALIGN(size, align);
1380
+
else
1381
+
size = roundup_pow_of_two(size);
1382
+
1383
+
peer_epc = peer_ntb_epc->epc;
1384
+
mw_addr = pci_epc_mem_alloc_addr(peer_epc, &phys_addr, size);
1385
+
if (!mw_addr) {
1386
+
dev_err(dev, "%s intf: Failed to allocate OB address\n",
1387
+
pci_epc_interface_string(peer_ntb_epc->type));
1388
+
return -ENOMEM;
1389
+
}
1390
+
1391
+
barno = ntb_epc->epf_ntb_bar[bar];
1392
+
epf_bar = &ntb_epc->epf_bar[barno];
1393
+
ntb_epc->mw_addr[barno] = mw_addr;
1394
+
1395
+
epf_bar->phys_addr = phys_addr;
1396
+
epf_bar->size = size;
1397
+
epf_bar->barno = barno;
1398
+
epf_bar->flags = PCI_BASE_ADDRESS_MEM_TYPE_32;
1399
+
1400
+
return 0;
1401
+
}
1402
+
1403
+
/**
1404
+
* epf_ntb_db_mw_bar_init() - Configure Doorbell and Memory window BARs
1405
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1406
+
* @type: PRIMARY interface or SECONDARY interface
1407
+
*
1408
+
* Wrapper for epf_ntb_alloc_peer_mem() and pci_epc_set_bar() that allocates
1409
+
* memory in OB address space of HOST2 and configures BAR of HOST1
1410
+
*/
1411
+
static int epf_ntb_db_mw_bar_init(struct epf_ntb *ntb,
1412
+
enum pci_epc_interface_type type)
1413
+
{
1414
+
const struct pci_epc_features *epc_features;
1415
+
struct epf_ntb_epc *peer_ntb_epc, *ntb_epc;
1416
+
struct pci_epf_bar *epf_bar;
1417
+
struct epf_ntb_ctrl *ctrl;
1418
+
u32 num_mws, db_count;
1419
+
enum epf_ntb_bar bar;
1420
+
enum pci_barno barno;
1421
+
struct pci_epc *epc;
1422
+
struct device *dev;
1423
+
size_t align;
1424
+
int ret, i;
1425
+
u8 func_no;
1426
+
u64 size;
1427
+
1428
+
ntb_epc = ntb->epc[type];
1429
+
peer_ntb_epc = ntb->epc[!type];
1430
+
1431
+
dev = &ntb->epf->dev;
1432
+
epc_features = ntb_epc->epc_features;
1433
+
align = epc_features->align;
1434
+
func_no = ntb_epc->func_no;
1435
+
epc = ntb_epc->epc;
1436
+
num_mws = ntb->num_mws;
1437
+
db_count = ntb->db_count;
1438
+
1439
+
for (bar = BAR_DB_MW1, i = 0; i < num_mws; bar++, i++) {
1440
+
if (bar == BAR_DB_MW1) {
1441
+
align = align ? align : 4;
1442
+
size = db_count * align;
1443
+
size = ALIGN(size, ntb->mws_size[i]);
1444
+
ctrl = ntb_epc->reg;
1445
+
ctrl->mw1_offset = size;
1446
+
size += ntb->mws_size[i];
1447
+
} else {
1448
+
size = ntb->mws_size[i];
1449
+
}
1450
+
1451
+
ret = epf_ntb_alloc_peer_mem(dev, ntb_epc, bar,
1452
+
peer_ntb_epc, size);
1453
+
if (ret) {
1454
+
dev_err(dev, "%s intf: DoorBell mem alloc failed\n",
1455
+
pci_epc_interface_string(type));
1456
+
goto err_alloc_peer_mem;
1457
+
}
1458
+
1459
+
barno = ntb_epc->epf_ntb_bar[bar];
1460
+
epf_bar = &ntb_epc->epf_bar[barno];
1461
+
1462
+
ret = pci_epc_set_bar(epc, func_no, epf_bar);
1463
+
if (ret) {
1464
+
dev_err(dev, "%s intf: DoorBell BAR set failed\n",
1465
+
pci_epc_interface_string(type));
1466
+
goto err_alloc_peer_mem;
1467
+
}
1468
+
}
1469
+
1470
+
return 0;
1471
+
1472
+
err_alloc_peer_mem:
1473
+
epf_ntb_db_mw_bar_cleanup(ntb, type);
1474
+
1475
+
return ret;
1476
+
}
1477
+
1478
+
/**
1479
+
* epf_ntb_epc_destroy_interface() - Cleanup NTB EPC interface
1480
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1481
+
* @type: PRIMARY interface or SECONDARY interface
1482
+
*
1483
+
* Unbind NTB function device from EPC and relinquish reference to pci_epc
1484
+
* for each of the interface.
1485
+
*/
1486
+
static void epf_ntb_epc_destroy_interface(struct epf_ntb *ntb,
1487
+
enum pci_epc_interface_type type)
1488
+
{
1489
+
struct epf_ntb_epc *ntb_epc;
1490
+
struct pci_epc *epc;
1491
+
struct pci_epf *epf;
1492
+
1493
+
if (type < 0)
1494
+
return;
1495
+
1496
+
epf = ntb->epf;
1497
+
ntb_epc = ntb->epc[type];
1498
+
if (!ntb_epc)
1499
+
return;
1500
+
epc = ntb_epc->epc;
1501
+
pci_epc_remove_epf(epc, epf, type);
1502
+
pci_epc_put(epc);
1503
+
}
1504
+
1505
+
/**
1506
+
* epf_ntb_epc_destroy() - Cleanup NTB EPC interface
1507
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1508
+
*
1509
+
* Wrapper for epf_ntb_epc_destroy_interface() to cleanup all the NTB interfaces
1510
+
*/
1511
+
static void epf_ntb_epc_destroy(struct epf_ntb *ntb)
1512
+
{
1513
+
enum pci_epc_interface_type type;
1514
+
1515
+
for (type = PRIMARY_INTERFACE; type <= SECONDARY_INTERFACE; type++)
1516
+
epf_ntb_epc_destroy_interface(ntb, type);
1517
+
}
1518
+
1519
+
/**
1520
+
* epf_ntb_epc_create_interface() - Create and initialize NTB EPC interface
1521
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1522
+
* @epc: struct pci_epc to which a particular NTB interface should be associated
1523
+
* @type: PRIMARY interface or SECONDARY interface
1524
+
*
1525
+
* Allocate memory for NTB EPC interface and initialize it.
1526
+
*/
1527
+
static int epf_ntb_epc_create_interface(struct epf_ntb *ntb,
1528
+
struct pci_epc *epc,
1529
+
enum pci_epc_interface_type type)
1530
+
{
1531
+
const struct pci_epc_features *epc_features;
1532
+
struct pci_epf_bar *epf_bar;
1533
+
struct epf_ntb_epc *ntb_epc;
1534
+
struct pci_epf *epf;
1535
+
struct device *dev;
1536
+
u8 func_no;
1537
+
1538
+
dev = &ntb->epf->dev;
1539
+
1540
+
ntb_epc = devm_kzalloc(dev, sizeof(*ntb_epc), GFP_KERNEL);
1541
+
if (!ntb_epc)
1542
+
return -ENOMEM;
1543
+
1544
+
epf = ntb->epf;
1545
+
if (type == PRIMARY_INTERFACE) {
1546
+
func_no = epf->func_no;
1547
+
epf_bar = epf->bar;
1548
+
} else {
1549
+
func_no = epf->sec_epc_func_no;
1550
+
epf_bar = epf->sec_epc_bar;
1551
+
}
1552
+
1553
+
ntb_epc->linkup = false;
1554
+
ntb_epc->epc = epc;
1555
+
ntb_epc->func_no = func_no;
1556
+
ntb_epc->type = type;
1557
+
ntb_epc->epf_bar = epf_bar;
1558
+
ntb_epc->epf_ntb = ntb;
1559
+
1560
+
epc_features = pci_epc_get_features(epc, func_no);
1561
+
if (!epc_features)
1562
+
return -EINVAL;
1563
+
ntb_epc->epc_features = epc_features;
1564
+
1565
+
ntb->epc[type] = ntb_epc;
1566
+
1567
+
return 0;
1568
+
}
1569
+
1570
+
/**
1571
+
* epf_ntb_epc_create() - Create and initialize NTB EPC interface
1572
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1573
+
*
1574
+
* Get a reference to EPC device and bind NTB function device to that EPC
1575
+
* for each of the interface. It is also a wrapper to
1576
+
* epf_ntb_epc_create_interface() to allocate memory for NTB EPC interface
1577
+
* and initialize it
1578
+
*/
1579
+
static int epf_ntb_epc_create(struct epf_ntb *ntb)
1580
+
{
1581
+
struct pci_epf *epf;
1582
+
struct device *dev;
1583
+
int ret;
1584
+
1585
+
epf = ntb->epf;
1586
+
dev = &epf->dev;
1587
+
1588
+
ret = epf_ntb_epc_create_interface(ntb, epf->epc, PRIMARY_INTERFACE);
1589
+
if (ret) {
1590
+
dev_err(dev, "PRIMARY intf: Fail to create NTB EPC\n");
1591
+
return ret;
1592
+
}
1593
+
1594
+
ret = epf_ntb_epc_create_interface(ntb, epf->sec_epc,
1595
+
SECONDARY_INTERFACE);
1596
+
if (ret) {
1597
+
dev_err(dev, "SECONDARY intf: Fail to create NTB EPC\n");
1598
+
goto err_epc_create;
1599
+
}
1600
+
1601
+
return 0;
1602
+
1603
+
err_epc_create:
1604
+
epf_ntb_epc_destroy_interface(ntb, PRIMARY_INTERFACE);
1605
+
1606
+
return ret;
1607
+
}
1608
+
1609
+
/**
1610
+
* epf_ntb_init_epc_bar_interface() - Identify BARs to be used for each of
1611
+
* the NTB constructs (scratchpad region, doorbell, memorywindow)
1612
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1613
+
* @type: PRIMARY interface or SECONDARY interface
1614
+
*
1615
+
* Identify the free BARs to be used for each of BAR_CONFIG, BAR_PEER_SPAD,
1616
+
* BAR_DB_MW1, BAR_MW2, BAR_MW3 and BAR_MW4.
1617
+
*/
1618
+
static int epf_ntb_init_epc_bar_interface(struct epf_ntb *ntb,
1619
+
enum pci_epc_interface_type type)
1620
+
{
1621
+
const struct pci_epc_features *epc_features;
1622
+
struct epf_ntb_epc *ntb_epc;
1623
+
enum pci_barno barno;
1624
+
enum epf_ntb_bar bar;
1625
+
struct device *dev;
1626
+
u32 num_mws;
1627
+
int i;
1628
+
1629
+
barno = BAR_0;
1630
+
ntb_epc = ntb->epc[type];
1631
+
num_mws = ntb->num_mws;
1632
+
dev = &ntb->epf->dev;
1633
+
epc_features = ntb_epc->epc_features;
1634
+
1635
+
/* These are required BARs which are mandatory for NTB functionality */
1636
+
for (bar = BAR_CONFIG; bar <= BAR_DB_MW1; bar++, barno++) {
1637
+
barno = pci_epc_get_next_free_bar(epc_features, barno);
1638
+
if (barno < 0) {
1639
+
dev_err(dev, "%s intf: Fail to get NTB function BAR\n",
1640
+
pci_epc_interface_string(type));
1641
+
return barno;
1642
+
}
1643
+
ntb_epc->epf_ntb_bar[bar] = barno;
1644
+
}
1645
+
1646
+
/* These are optional BARs which don't impact NTB functionality */
1647
+
for (bar = BAR_MW2, i = 1; i < num_mws; bar++, barno++, i++) {
1648
+
barno = pci_epc_get_next_free_bar(epc_features, barno);
1649
+
if (barno < 0) {
1650
+
ntb->num_mws = i;
1651
+
dev_dbg(dev, "BAR not available for > MW%d\n", i + 1);
1652
+
}
1653
+
ntb_epc->epf_ntb_bar[bar] = barno;
1654
+
}
1655
+
1656
+
return 0;
1657
+
}
1658
+
1659
+
/**
1660
+
* epf_ntb_init_epc_bar() - Identify BARs to be used for each of the NTB
1661
+
* constructs (scratchpad region, doorbell, memorywindow)
1662
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1663
+
* @type: PRIMARY interface or SECONDARY interface
1664
+
*
1665
+
* Wrapper to epf_ntb_init_epc_bar_interface() to identify the free BARs
1666
+
* to be used for each of BAR_CONFIG, BAR_PEER_SPAD, BAR_DB_MW1, BAR_MW2,
1667
+
* BAR_MW3 and BAR_MW4 for all the interfaces.
1668
+
*/
1669
+
static int epf_ntb_init_epc_bar(struct epf_ntb *ntb)
1670
+
{
1671
+
enum pci_epc_interface_type type;
1672
+
struct device *dev;
1673
+
int ret;
1674
+
1675
+
dev = &ntb->epf->dev;
1676
+
for (type = PRIMARY_INTERFACE; type <= SECONDARY_INTERFACE; type++) {
1677
+
ret = epf_ntb_init_epc_bar_interface(ntb, type);
1678
+
if (ret) {
1679
+
dev_err(dev, "Fail to init EPC bar for %s interface\n",
1680
+
pci_epc_interface_string(type));
1681
+
return ret;
1682
+
}
1683
+
}
1684
+
1685
+
return 0;
1686
+
}
1687
+
1688
+
/**
1689
+
* epf_ntb_epc_init_interface() - Initialize NTB interface
1690
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1691
+
* @type: PRIMARY interface or SECONDARY interface
1692
+
*
1693
+
* Wrapper to initialize a particular EPC interface and start the workqueue
1694
+
* to check for commands from host. This function will write to the
1695
+
* EP controller HW for configuring it.
1696
+
*/
1697
+
static int epf_ntb_epc_init_interface(struct epf_ntb *ntb,
1698
+
enum pci_epc_interface_type type)
1699
+
{
1700
+
struct epf_ntb_epc *ntb_epc;
1701
+
struct pci_epc *epc;
1702
+
struct pci_epf *epf;
1703
+
struct device *dev;
1704
+
u8 func_no;
1705
+
int ret;
1706
+
1707
+
ntb_epc = ntb->epc[type];
1708
+
epf = ntb->epf;
1709
+
dev = &epf->dev;
1710
+
epc = ntb_epc->epc;
1711
+
func_no = ntb_epc->func_no;
1712
+
1713
+
ret = epf_ntb_config_sspad_bar_set(ntb->epc[type]);
1714
+
if (ret) {
1715
+
dev_err(dev, "%s intf: Config/self SPAD BAR init failed\n",
1716
+
pci_epc_interface_string(type));
1717
+
return ret;
1718
+
}
1719
+
1720
+
ret = epf_ntb_peer_spad_bar_set(ntb, type);
1721
+
if (ret) {
1722
+
dev_err(dev, "%s intf: Peer SPAD BAR init failed\n",
1723
+
pci_epc_interface_string(type));
1724
+
goto err_peer_spad_bar_init;
1725
+
}
1726
+
1727
+
ret = epf_ntb_configure_interrupt(ntb, type);
1728
+
if (ret) {
1729
+
dev_err(dev, "%s intf: Interrupt configuration failed\n",
1730
+
pci_epc_interface_string(type));
1731
+
goto err_peer_spad_bar_init;
1732
+
}
1733
+
1734
+
ret = epf_ntb_db_mw_bar_init(ntb, type);
1735
+
if (ret) {
1736
+
dev_err(dev, "%s intf: DB/MW BAR init failed\n",
1737
+
pci_epc_interface_string(type));
1738
+
goto err_db_mw_bar_init;
1739
+
}
1740
+
1741
+
ret = pci_epc_write_header(epc, func_no, epf->header);
1742
+
if (ret) {
1743
+
dev_err(dev, "%s intf: Configuration header write failed\n",
1744
+
pci_epc_interface_string(type));
1745
+
goto err_write_header;
1746
+
}
1747
+
1748
+
INIT_DELAYED_WORK(&ntb->epc[type]->cmd_handler, epf_ntb_cmd_handler);
1749
+
queue_work(kpcintb_workqueue, &ntb->epc[type]->cmd_handler.work);
1750
+
1751
+
return 0;
1752
+
1753
+
err_write_header:
1754
+
epf_ntb_db_mw_bar_cleanup(ntb, type);
1755
+
1756
+
err_db_mw_bar_init:
1757
+
epf_ntb_peer_spad_bar_clear(ntb->epc[type]);
1758
+
1759
+
err_peer_spad_bar_init:
1760
+
epf_ntb_config_sspad_bar_clear(ntb->epc[type]);
1761
+
1762
+
return ret;
1763
+
}
1764
+
1765
+
/**
1766
+
* epf_ntb_epc_cleanup_interface() - Cleanup NTB interface
1767
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1768
+
* @type: PRIMARY interface or SECONDARY interface
1769
+
*
1770
+
* Wrapper to cleanup a particular NTB interface.
1771
+
*/
1772
+
static void epf_ntb_epc_cleanup_interface(struct epf_ntb *ntb,
1773
+
enum pci_epc_interface_type type)
1774
+
{
1775
+
struct epf_ntb_epc *ntb_epc;
1776
+
1777
+
if (type < 0)
1778
+
return;
1779
+
1780
+
ntb_epc = ntb->epc[type];
1781
+
cancel_delayed_work(&ntb_epc->cmd_handler);
1782
+
epf_ntb_db_mw_bar_cleanup(ntb, type);
1783
+
epf_ntb_peer_spad_bar_clear(ntb_epc);
1784
+
epf_ntb_config_sspad_bar_clear(ntb_epc);
1785
+
}
1786
+
1787
+
/**
1788
+
* epf_ntb_epc_cleanup() - Cleanup all NTB interfaces
1789
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1790
+
*
1791
+
* Wrapper to cleanup all NTB interfaces.
1792
+
*/
1793
+
static void epf_ntb_epc_cleanup(struct epf_ntb *ntb)
1794
+
{
1795
+
enum pci_epc_interface_type type;
1796
+
1797
+
for (type = PRIMARY_INTERFACE; type <= SECONDARY_INTERFACE; type++)
1798
+
epf_ntb_epc_cleanup_interface(ntb, type);
1799
+
}
1800
+
1801
+
/**
1802
+
* epf_ntb_epc_init() - Initialize all NTB interfaces
1803
+
* @ntb: NTB device that facilitates communication between HOST1 and HOST2
1804
+
*
1805
+
* Wrapper to initialize all NTB interface and start the workqueue
1806
+
* to check for commands from host.
1807
+
*/
1808
+
static int epf_ntb_epc_init(struct epf_ntb *ntb)
1809
+
{
1810
+
enum pci_epc_interface_type type;
1811
+
struct device *dev;
1812
+
int ret;
1813
+
1814
+
dev = &ntb->epf->dev;
1815
+
1816
+
for (type = PRIMARY_INTERFACE; type <= SECONDARY_INTERFACE; type++) {
1817
+
ret = epf_ntb_epc_init_interface(ntb, type);
1818
+
if (ret) {
1819
+
dev_err(dev, "%s intf: Failed to initialize\n",
1820
+
pci_epc_interface_string(type));
1821
+
goto err_init_type;
1822
+
}
1823
+
}
1824
+
1825
+
return 0;
1826
+
1827
+
err_init_type:
1828
+
epf_ntb_epc_cleanup_interface(ntb, type - 1);
1829
+
1830
+
return ret;
1831
+
}
1832
+
1833
+
/**
1834
+
* epf_ntb_bind() - Initialize endpoint controller to provide NTB functionality
1835
+
* @epf: NTB endpoint function device
1836
+
*
1837
+
* Initialize both the endpoint controllers associated with NTB function device.
1838
+
* Invoked when a primary interface or secondary interface is bound to EPC
1839
+
* device. This function will succeed only when EPC is bound to both the
1840
+
* interfaces.
1841
+
*/
1842
+
static int epf_ntb_bind(struct pci_epf *epf)
1843
+
{
1844
+
struct epf_ntb *ntb = epf_get_drvdata(epf);
1845
+
struct device *dev = &epf->dev;
1846
+
int ret;
1847
+
1848
+
if (!epf->epc) {
1849
+
dev_dbg(dev, "PRIMARY EPC interface not yet bound\n");
1850
+
return 0;
1851
+
}
1852
+
1853
+
if (!epf->sec_epc) {
1854
+
dev_dbg(dev, "SECONDARY EPC interface not yet bound\n");
1855
+
return 0;
1856
+
}
1857
+
1858
+
ret = epf_ntb_epc_create(ntb);
1859
+
if (ret) {
1860
+
dev_err(dev, "Failed to create NTB EPC\n");
1861
+
return ret;
1862
+
}
1863
+
1864
+
ret = epf_ntb_init_epc_bar(ntb);
1865
+
if (ret) {
1866
+
dev_err(dev, "Failed to create NTB EPC\n");
1867
+
goto err_bar_init;
1868
+
}
1869
+
1870
+
ret = epf_ntb_config_spad_bar_alloc_interface(ntb);
1871
+
if (ret) {
1872
+
dev_err(dev, "Failed to allocate BAR memory\n");
1873
+
goto err_bar_alloc;
1874
+
}
1875
+
1876
+
ret = epf_ntb_epc_init(ntb);
1877
+
if (ret) {
1878
+
dev_err(dev, "Failed to initialize EPC\n");
1879
+
goto err_bar_alloc;
1880
+
}
1881
+
1882
+
epf_set_drvdata(epf, ntb);
1883
+
1884
+
return 0;
1885
+
1886
+
err_bar_alloc:
1887
+
epf_ntb_config_spad_bar_free(ntb);
1888
+
1889
+
err_bar_init:
1890
+
epf_ntb_epc_destroy(ntb);
1891
+
1892
+
return ret;
1893
+
}
1894
+
1895
+
/**
1896
+
* epf_ntb_unbind() - Cleanup the initialization from epf_ntb_bind()
1897
+
* @epf: NTB endpoint function device
1898
+
*
1899
+
* Cleanup the initialization from epf_ntb_bind()
1900
+
*/
1901
+
static void epf_ntb_unbind(struct pci_epf *epf)
1902
+
{
1903
+
struct epf_ntb *ntb = epf_get_drvdata(epf);
1904
+
1905
+
epf_ntb_epc_cleanup(ntb);
1906
+
epf_ntb_config_spad_bar_free(ntb);
1907
+
epf_ntb_epc_destroy(ntb);
1908
+
}
1909
+
1910
+
#define EPF_NTB_R(_name) \
1911
+
static ssize_t epf_ntb_##_name##_show(struct config_item *item, \
1912
+
char *page) \
1913
+
{ \
1914
+
struct config_group *group = to_config_group(item); \
1915
+
struct epf_ntb *ntb = to_epf_ntb(group); \
1916
+
\
1917
+
return sprintf(page, "%d\n", ntb->_name); \
1918
+
}
1919
+
1920
+
#define EPF_NTB_W(_name) \
1921
+
static ssize_t epf_ntb_##_name##_store(struct config_item *item, \
1922
+
const char *page, size_t len) \
1923
+
{ \
1924
+
struct config_group *group = to_config_group(item); \
1925
+
struct epf_ntb *ntb = to_epf_ntb(group); \
1926
+
u32 val; \
1927
+
int ret; \
1928
+
\
1929
+
ret = kstrtou32(page, 0, &val); \
1930
+
if (ret) \
1931
+
return ret; \
1932
+
\
1933
+
ntb->_name = val; \
1934
+
\
1935
+
return len; \
1936
+
}
1937
+
1938
+
#define EPF_NTB_MW_R(_name) \
1939
+
static ssize_t epf_ntb_##_name##_show(struct config_item *item, \
1940
+
char *page) \
1941
+
{ \
1942
+
struct config_group *group = to_config_group(item); \
1943
+
struct epf_ntb *ntb = to_epf_ntb(group); \
1944
+
int win_no; \
1945
+
\
1946
+
sscanf(#_name, "mw%d", &win_no); \
1947
+
\
1948
+
return sprintf(page, "%lld\n", ntb->mws_size[win_no - 1]); \
1949
+
}
1950
+
1951
+
#define EPF_NTB_MW_W(_name) \
1952
+
static ssize_t epf_ntb_##_name##_store(struct config_item *item, \
1953
+
const char *page, size_t len) \
1954
+
{ \
1955
+
struct config_group *group = to_config_group(item); \
1956
+
struct epf_ntb *ntb = to_epf_ntb(group); \
1957
+
struct device *dev = &ntb->epf->dev; \
1958
+
int win_no; \
1959
+
u64 val; \
1960
+
int ret; \
1961
+
\
1962
+
ret = kstrtou64(page, 0, &val); \
1963
+
if (ret) \
1964
+
return ret; \
1965
+
\
1966
+
if (sscanf(#_name, "mw%d", &win_no) != 1) \
1967
+
return -EINVAL; \
1968
+
\
1969
+
if (ntb->num_mws < win_no) { \
1970
+
dev_err(dev, "Invalid num_nws: %d value\n", ntb->num_mws); \
1971
+
return -EINVAL; \
1972
+
} \
1973
+
\
1974
+
ntb->mws_size[win_no - 1] = val; \
1975
+
\
1976
+
return len; \
1977
+
}
1978
+
1979
+
static ssize_t epf_ntb_num_mws_store(struct config_item *item,
1980
+
const char *page, size_t len)
1981
+
{
1982
+
struct config_group *group = to_config_group(item);
1983
+
struct epf_ntb *ntb = to_epf_ntb(group);
1984
+
u32 val;
1985
+
int ret;
1986
+
1987
+
ret = kstrtou32(page, 0, &val);
1988
+
if (ret)
1989
+
return ret;
1990
+
1991
+
if (val > MAX_MW)
1992
+
return -EINVAL;
1993
+
1994
+
ntb->num_mws = val;
1995
+
1996
+
return len;
1997
+
}
1998
+
1999
+
EPF_NTB_R(spad_count)
2000
+
EPF_NTB_W(spad_count)
2001
+
EPF_NTB_R(db_count)
2002
+
EPF_NTB_W(db_count)
2003
+
EPF_NTB_R(num_mws)
2004
+
EPF_NTB_MW_R(mw1)
2005
+
EPF_NTB_MW_W(mw1)
2006
+
EPF_NTB_MW_R(mw2)
2007
+
EPF_NTB_MW_W(mw2)
2008
+
EPF_NTB_MW_R(mw3)
2009
+
EPF_NTB_MW_W(mw3)
2010
+
EPF_NTB_MW_R(mw4)
2011
+
EPF_NTB_MW_W(mw4)
2012
+
2013
+
CONFIGFS_ATTR(epf_ntb_, spad_count);
2014
+
CONFIGFS_ATTR(epf_ntb_, db_count);
2015
+
CONFIGFS_ATTR(epf_ntb_, num_mws);
2016
+
CONFIGFS_ATTR(epf_ntb_, mw1);
2017
+
CONFIGFS_ATTR(epf_ntb_, mw2);
2018
+
CONFIGFS_ATTR(epf_ntb_, mw3);
2019
+
CONFIGFS_ATTR(epf_ntb_, mw4);
2020
+
2021
+
static struct configfs_attribute *epf_ntb_attrs[] = {
2022
+
&epf_ntb_attr_spad_count,
2023
+
&epf_ntb_attr_db_count,
2024
+
&epf_ntb_attr_num_mws,
2025
+
&epf_ntb_attr_mw1,
2026
+
&epf_ntb_attr_mw2,
2027
+
&epf_ntb_attr_mw3,
2028
+
&epf_ntb_attr_mw4,
2029
+
NULL,
2030
+
};
2031
+
2032
+
static const struct config_item_type ntb_group_type = {
2033
+
.ct_attrs = epf_ntb_attrs,
2034
+
.ct_owner = THIS_MODULE,
2035
+
};
2036
+
2037
+
/**
2038
+
* epf_ntb_add_cfs() - Add configfs directory specific to NTB
2039
+
* @epf: NTB endpoint function device
2040
+
*
2041
+
* Add configfs directory specific to NTB. This directory will hold
2042
+
* NTB specific properties like db_count, spad_count, num_mws etc.,
2043
+
*/
2044
+
static struct config_group *epf_ntb_add_cfs(struct pci_epf *epf,
2045
+
struct config_group *group)
2046
+
{
2047
+
struct epf_ntb *ntb = epf_get_drvdata(epf);
2048
+
struct config_group *ntb_group = &ntb->group;
2049
+
struct device *dev = &epf->dev;
2050
+
2051
+
config_group_init_type_name(ntb_group, dev_name(dev), &ntb_group_type);
2052
+
2053
+
return ntb_group;
2054
+
}
2055
+
2056
+
/**
2057
+
* epf_ntb_probe() - Probe NTB function driver
2058
+
* @epf: NTB endpoint function device
2059
+
*
2060
+
* Probe NTB function driver when endpoint function bus detects a NTB
2061
+
* endpoint function.
2062
+
*/
2063
+
static int epf_ntb_probe(struct pci_epf *epf)
2064
+
{
2065
+
struct epf_ntb *ntb;
2066
+
struct device *dev;
2067
+
2068
+
dev = &epf->dev;
2069
+
2070
+
ntb = devm_kzalloc(dev, sizeof(*ntb), GFP_KERNEL);
2071
+
if (!ntb)
2072
+
return -ENOMEM;
2073
+
2074
+
epf->header = &epf_ntb_header;
2075
+
ntb->epf = epf;
2076
+
epf_set_drvdata(epf, ntb);
2077
+
2078
+
return 0;
2079
+
}
2080
+
2081
+
static struct pci_epf_ops epf_ntb_ops = {
2082
+
.bind = epf_ntb_bind,
2083
+
.unbind = epf_ntb_unbind,
2084
+
.add_cfs = epf_ntb_add_cfs,
2085
+
};
2086
+
2087
+
static const struct pci_epf_device_id epf_ntb_ids[] = {
2088
+
{
2089
+
.name = "pci_epf_ntb",
2090
+
},
2091
+
{},
2092
+
};
2093
+
2094
+
static struct pci_epf_driver epf_ntb_driver = {
2095
+
.driver.name = "pci_epf_ntb",
2096
+
.probe = epf_ntb_probe,
2097
+
.id_table = epf_ntb_ids,
2098
+
.ops = &epf_ntb_ops,
2099
+
.owner = THIS_MODULE,
2100
+
};
2101
+
2102
+
static int __init epf_ntb_init(void)
2103
+
{
2104
+
int ret;
2105
+
2106
+
kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM |
2107
+
WQ_HIGHPRI, 0);
2108
+
ret = pci_epf_register_driver(&epf_ntb_driver);
2109
+
if (ret) {
2110
+
destroy_workqueue(kpcintb_workqueue);
2111
+
pr_err("Failed to register pci epf ntb driver --> %d\n", ret);
2112
+
return ret;
2113
+
}
2114
+
2115
+
return 0;
2116
+
}
2117
+
module_init(epf_ntb_init);
2118
+
2119
+
static void __exit epf_ntb_exit(void)
2120
+
{
2121
+
pci_epf_unregister_driver(&epf_ntb_driver);
2122
+
destroy_workqueue(kpcintb_workqueue);
2123
+
}
2124
+
module_exit(epf_ntb_exit);
2125
+
2126
+
MODULE_DESCRIPTION("PCI EPF NTB DRIVER");
2127
+
MODULE_AUTHOR("Kishon Vijay Abraham I <kishon@ti.com>");
2128
+
MODULE_LICENSE("GPL v2");
+9
-4
drivers/pci/endpoint/functions/pci-epf-test.c
+9
-4
drivers/pci/endpoint/functions/pci-epf-test.c
···
619
619
620
620
if (epf_test->reg[bar]) {
621
621
pci_epc_clear_bar(epc, epf->func_no, epf_bar);
622
-
pci_epf_free_space(epf, epf_test->reg[bar], bar);
622
+
pci_epf_free_space(epf, epf_test->reg[bar], bar,
623
+
PRIMARY_INTERFACE);
623
624
}
624
625
}
625
626
}
···
652
651
653
652
ret = pci_epc_set_bar(epc, epf->func_no, epf_bar);
654
653
if (ret) {
655
-
pci_epf_free_space(epf, epf_test->reg[bar], bar);
654
+
pci_epf_free_space(epf, epf_test->reg[bar], bar,
655
+
PRIMARY_INTERFACE);
656
656
dev_err(dev, "Failed to set BAR%d\n", bar);
657
657
if (bar == test_reg_bar)
658
658
return ret;
···
773
771
}
774
772
775
773
base = pci_epf_alloc_space(epf, test_reg_size, test_reg_bar,
776
-
epc_features->align);
774
+
epc_features->align, PRIMARY_INTERFACE);
777
775
if (!base) {
778
776
dev_err(dev, "Failed to allocated register space\n");
779
777
return -ENOMEM;
···
791
789
continue;
792
790
793
791
base = pci_epf_alloc_space(epf, bar_size[bar], bar,
794
-
epc_features->align);
792
+
epc_features->align,
793
+
PRIMARY_INTERFACE);
795
794
if (!base)
796
795
dev_err(dev, "Failed to allocate space for BAR%d\n",
797
796
bar);
···
837
834
linkup_notifier = epc_features->linkup_notifier;
838
835
core_init_notifier = epc_features->core_init_notifier;
839
836
test_reg_bar = pci_epc_get_first_free_bar(epc_features);
837
+
if (test_reg_bar < 0)
838
+
return -EINVAL;
840
839
pci_epf_configure_bar(epf, epc_features);
841
840
}
842
841
+173
-3
drivers/pci/endpoint/pci-ep-cfs.c
+173
-3
drivers/pci/endpoint/pci-ep-cfs.c
···
21
21
22
22
struct pci_epf_group {
23
23
struct config_group group;
24
+
struct config_group primary_epc_group;
25
+
struct config_group secondary_epc_group;
26
+
struct delayed_work cfs_work;
24
27
struct pci_epf *epf;
25
28
int index;
26
29
};
···
42
39
static inline struct pci_epc_group *to_pci_epc_group(struct config_item *item)
43
40
{
44
41
return container_of(to_config_group(item), struct pci_epc_group, group);
42
+
}
43
+
44
+
static int pci_secondary_epc_epf_link(struct config_item *epf_item,
45
+
struct config_item *epc_item)
46
+
{
47
+
int ret;
48
+
struct pci_epf_group *epf_group = to_pci_epf_group(epf_item->ci_parent);
49
+
struct pci_epc_group *epc_group = to_pci_epc_group(epc_item);
50
+
struct pci_epc *epc = epc_group->epc;
51
+
struct pci_epf *epf = epf_group->epf;
52
+
53
+
ret = pci_epc_add_epf(epc, epf, SECONDARY_INTERFACE);
54
+
if (ret)
55
+
return ret;
56
+
57
+
ret = pci_epf_bind(epf);
58
+
if (ret) {
59
+
pci_epc_remove_epf(epc, epf, SECONDARY_INTERFACE);
60
+
return ret;
61
+
}
62
+
63
+
return 0;
64
+
}
65
+
66
+
static void pci_secondary_epc_epf_unlink(struct config_item *epc_item,
67
+
struct config_item *epf_item)
68
+
{
69
+
struct pci_epf_group *epf_group = to_pci_epf_group(epf_item->ci_parent);
70
+
struct pci_epc_group *epc_group = to_pci_epc_group(epc_item);
71
+
struct pci_epc *epc;
72
+
struct pci_epf *epf;
73
+
74
+
WARN_ON_ONCE(epc_group->start);
75
+
76
+
epc = epc_group->epc;
77
+
epf = epf_group->epf;
78
+
pci_epf_unbind(epf);
79
+
pci_epc_remove_epf(epc, epf, SECONDARY_INTERFACE);
80
+
}
81
+
82
+
static struct configfs_item_operations pci_secondary_epc_item_ops = {
83
+
.allow_link = pci_secondary_epc_epf_link,
84
+
.drop_link = pci_secondary_epc_epf_unlink,
85
+
};
86
+
87
+
static const struct config_item_type pci_secondary_epc_type = {
88
+
.ct_item_ops = &pci_secondary_epc_item_ops,
89
+
.ct_owner = THIS_MODULE,
90
+
};
91
+
92
+
static struct config_group
93
+
*pci_ep_cfs_add_secondary_group(struct pci_epf_group *epf_group)
94
+
{
95
+
struct config_group *secondary_epc_group;
96
+
97
+
secondary_epc_group = &epf_group->secondary_epc_group;
98
+
config_group_init_type_name(secondary_epc_group, "secondary",
99
+
&pci_secondary_epc_type);
100
+
configfs_register_group(&epf_group->group, secondary_epc_group);
101
+
102
+
return secondary_epc_group;
103
+
}
104
+
105
+
static int pci_primary_epc_epf_link(struct config_item *epf_item,
106
+
struct config_item *epc_item)
107
+
{
108
+
int ret;
109
+
struct pci_epf_group *epf_group = to_pci_epf_group(epf_item->ci_parent);
110
+
struct pci_epc_group *epc_group = to_pci_epc_group(epc_item);
111
+
struct pci_epc *epc = epc_group->epc;
112
+
struct pci_epf *epf = epf_group->epf;
113
+
114
+
ret = pci_epc_add_epf(epc, epf, PRIMARY_INTERFACE);
115
+
if (ret)
116
+
return ret;
117
+
118
+
ret = pci_epf_bind(epf);
119
+
if (ret) {
120
+
pci_epc_remove_epf(epc, epf, PRIMARY_INTERFACE);
121
+
return ret;
122
+
}
123
+
124
+
return 0;
125
+
}
126
+
127
+
static void pci_primary_epc_epf_unlink(struct config_item *epc_item,
128
+
struct config_item *epf_item)
129
+
{
130
+
struct pci_epf_group *epf_group = to_pci_epf_group(epf_item->ci_parent);
131
+
struct pci_epc_group *epc_group = to_pci_epc_group(epc_item);
132
+
struct pci_epc *epc;
133
+
struct pci_epf *epf;
134
+
135
+
WARN_ON_ONCE(epc_group->start);
136
+
137
+
epc = epc_group->epc;
138
+
epf = epf_group->epf;
139
+
pci_epf_unbind(epf);
140
+
pci_epc_remove_epf(epc, epf, PRIMARY_INTERFACE);
141
+
}
142
+
143
+
static struct configfs_item_operations pci_primary_epc_item_ops = {
144
+
.allow_link = pci_primary_epc_epf_link,
145
+
.drop_link = pci_primary_epc_epf_unlink,
146
+
};
147
+
148
+
static const struct config_item_type pci_primary_epc_type = {
149
+
.ct_item_ops = &pci_primary_epc_item_ops,
150
+
.ct_owner = THIS_MODULE,
151
+
};
152
+
153
+
static struct config_group
154
+
*pci_ep_cfs_add_primary_group(struct pci_epf_group *epf_group)
155
+
{
156
+
struct config_group *primary_epc_group = &epf_group->primary_epc_group;
157
+
158
+
config_group_init_type_name(primary_epc_group, "primary",
159
+
&pci_primary_epc_type);
160
+
configfs_register_group(&epf_group->group, primary_epc_group);
161
+
162
+
return primary_epc_group;
45
163
}
46
164
47
165
static ssize_t pci_epc_start_store(struct config_item *item, const char *page,
···
218
94
struct pci_epc *epc = epc_group->epc;
219
95
struct pci_epf *epf = epf_group->epf;
220
96
221
-
ret = pci_epc_add_epf(epc, epf);
97
+
ret = pci_epc_add_epf(epc, epf, PRIMARY_INTERFACE);
222
98
if (ret)
223
99
return ret;
224
100
225
101
ret = pci_epf_bind(epf);
226
102
if (ret) {
227
-
pci_epc_remove_epf(epc, epf);
103
+
pci_epc_remove_epf(epc, epf, PRIMARY_INTERFACE);
228
104
return ret;
229
105
}
230
106
···
244
120
epc = epc_group->epc;
245
121
epf = epf_group->epf;
246
122
pci_epf_unbind(epf);
247
-
pci_epc_remove_epf(epc, epf);
123
+
pci_epc_remove_epf(epc, epf, PRIMARY_INTERFACE);
248
124
}
249
125
250
126
static struct configfs_item_operations pci_epc_item_ops = {
···
490
366
.release = pci_epf_release,
491
367
};
492
368
369
+
static struct config_group *pci_epf_type_make(struct config_group *group,
370
+
const char *name)
371
+
{
372
+
struct pci_epf_group *epf_group = to_pci_epf_group(&group->cg_item);
373
+
struct config_group *epf_type_group;
374
+
375
+
epf_type_group = pci_epf_type_add_cfs(epf_group->epf, group);
376
+
return epf_type_group;
377
+
}
378
+
379
+
static void pci_epf_type_drop(struct config_group *group,
380
+
struct config_item *item)
381
+
{
382
+
config_item_put(item);
383
+
}
384
+
385
+
static struct configfs_group_operations pci_epf_type_group_ops = {
386
+
.make_group = &pci_epf_type_make,
387
+
.drop_item = &pci_epf_type_drop,
388
+
};
389
+
493
390
static const struct config_item_type pci_epf_type = {
391
+
.ct_group_ops = &pci_epf_type_group_ops,
494
392
.ct_item_ops = &pci_epf_ops,
495
393
.ct_attrs = pci_epf_attrs,
496
394
.ct_owner = THIS_MODULE,
497
395
};
396
+
397
+
static void pci_epf_cfs_work(struct work_struct *work)
398
+
{
399
+
struct pci_epf_group *epf_group;
400
+
struct config_group *group;
401
+
402
+
epf_group = container_of(work, struct pci_epf_group, cfs_work.work);
403
+
group = pci_ep_cfs_add_primary_group(epf_group);
404
+
if (IS_ERR(group)) {
405
+
pr_err("failed to create 'primary' EPC interface\n");
406
+
return;
407
+
}
408
+
409
+
group = pci_ep_cfs_add_secondary_group(epf_group);
410
+
if (IS_ERR(group)) {
411
+
pr_err("failed to create 'secondary' EPC interface\n");
412
+
return;
413
+
}
414
+
}
498
415
499
416
static struct config_group *pci_epf_make(struct config_group *group,
500
417
const char *name)
···
575
410
goto free_name;
576
411
}
577
412
413
+
epf->group = &epf_group->group;
578
414
epf_group->epf = epf;
579
415
580
416
kfree(epf_name);
417
+
418
+
INIT_DELAYED_WORK(&epf_group->cfs_work, pci_epf_cfs_work);
419
+
queue_delayed_work(system_wq, &epf_group->cfs_work,
420
+
msecs_to_jiffies(1));
581
421
582
422
return &epf_group->group;
583
423
+111
-19
drivers/pci/endpoint/pci-epc-core.c
+111
-19
drivers/pci/endpoint/pci-epc-core.c
···
87
87
* pci_epc_get_first_free_bar() - helper to get first unreserved BAR
88
88
* @epc_features: pci_epc_features structure that holds the reserved bar bitmap
89
89
*
90
-
* Invoke to get the first unreserved BAR that can be used for endpoint
90
+
* Invoke to get the first unreserved BAR that can be used by the endpoint
91
91
* function. For any incorrect value in reserved_bar return '0'.
92
92
*/
93
-
unsigned int pci_epc_get_first_free_bar(const struct pci_epc_features
94
-
*epc_features)
93
+
enum pci_barno
94
+
pci_epc_get_first_free_bar(const struct pci_epc_features *epc_features)
95
95
{
96
-
int free_bar;
96
+
return pci_epc_get_next_free_bar(epc_features, BAR_0);
97
+
}
98
+
EXPORT_SYMBOL_GPL(pci_epc_get_first_free_bar);
99
+
100
+
/**
101
+
* pci_epc_get_next_free_bar() - helper to get unreserved BAR starting from @bar
102
+
* @epc_features: pci_epc_features structure that holds the reserved bar bitmap
103
+
* @bar: the starting BAR number from where unreserved BAR should be searched
104
+
*
105
+
* Invoke to get the next unreserved BAR starting from @bar that can be used
106
+
* for endpoint function. For any incorrect value in reserved_bar return '0'.
107
+
*/
108
+
enum pci_barno pci_epc_get_next_free_bar(const struct pci_epc_features
109
+
*epc_features, enum pci_barno bar)
110
+
{
111
+
unsigned long free_bar;
97
112
98
113
if (!epc_features)
99
-
return 0;
114
+
return BAR_0;
100
115
101
-
free_bar = ffz(epc_features->reserved_bar);
116
+
/* If 'bar - 1' is a 64-bit BAR, move to the next BAR */
117
+
if ((epc_features->bar_fixed_64bit << 1) & 1 << bar)
118
+
bar++;
119
+
120
+
/* Find if the reserved BAR is also a 64-bit BAR */
121
+
free_bar = epc_features->reserved_bar & epc_features->bar_fixed_64bit;
122
+
123
+
/* Set the adjacent bit if the reserved BAR is also a 64-bit BAR */
124
+
free_bar <<= 1;
125
+
free_bar |= epc_features->reserved_bar;
126
+
127
+
free_bar = find_next_zero_bit(&free_bar, 6, bar);
102
128
if (free_bar > 5)
103
-
return 0;
129
+
return NO_BAR;
104
130
105
131
return free_bar;
106
132
}
107
-
EXPORT_SYMBOL_GPL(pci_epc_get_first_free_bar);
133
+
EXPORT_SYMBOL_GPL(pci_epc_get_next_free_bar);
108
134
109
135
/**
110
136
* pci_epc_get_features() - get the features supported by EPC
···
229
203
return ret;
230
204
}
231
205
EXPORT_SYMBOL_GPL(pci_epc_raise_irq);
206
+
207
+
/**
208
+
* pci_epc_map_msi_irq() - Map physical address to MSI address and return
209
+
* MSI data
210
+
* @epc: the EPC device which has the MSI capability
211
+
* @func_no: the physical endpoint function number in the EPC device
212
+
* @phys_addr: the physical address of the outbound region
213
+
* @interrupt_num: the MSI interrupt number
214
+
* @entry_size: Size of Outbound address region for each interrupt
215
+
* @msi_data: the data that should be written in order to raise MSI interrupt
216
+
* with interrupt number as 'interrupt num'
217
+
* @msi_addr_offset: Offset of MSI address from the aligned outbound address
218
+
* to which the MSI address is mapped
219
+
*
220
+
* Invoke to map physical address to MSI address and return MSI data. The
221
+
* physical address should be an address in the outbound region. This is
222
+
* required to implement doorbell functionality of NTB wherein EPC on either
223
+
* side of the interface (primary and secondary) can directly write to the
224
+
* physical address (in outbound region) of the other interface to ring
225
+
* doorbell.
226
+
*/
227
+
int pci_epc_map_msi_irq(struct pci_epc *epc, u8 func_no, phys_addr_t phys_addr,
228
+
u8 interrupt_num, u32 entry_size, u32 *msi_data,
229
+
u32 *msi_addr_offset)
230
+
{
231
+
int ret;
232
+
233
+
if (IS_ERR_OR_NULL(epc))
234
+
return -EINVAL;
235
+
236
+
if (!epc->ops->map_msi_irq)
237
+
return -EINVAL;
238
+
239
+
mutex_lock(&epc->lock);
240
+
ret = epc->ops->map_msi_irq(epc, func_no, phys_addr, interrupt_num,
241
+
entry_size, msi_data, msi_addr_offset);
242
+
mutex_unlock(&epc->lock);
243
+
244
+
return ret;
245
+
}
246
+
EXPORT_SYMBOL_GPL(pci_epc_map_msi_irq);
232
247
233
248
/**
234
249
* pci_epc_get_msi() - get the number of MSI interrupt numbers allocated
···
534
467
* pci_epc_add_epf() - bind PCI endpoint function to an endpoint controller
535
468
* @epc: the EPC device to which the endpoint function should be added
536
469
* @epf: the endpoint function to be added
470
+
* @type: Identifies if the EPC is connected to the primary or secondary
471
+
* interface of EPF
537
472
*
538
473
* A PCI endpoint device can have one or more functions. In the case of PCIe,
539
474
* the specification allows up to 8 PCIe endpoint functions. Invoke
540
475
* pci_epc_add_epf() to add a PCI endpoint function to an endpoint controller.
541
476
*/
542
-
int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf)
477
+
int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf,
478
+
enum pci_epc_interface_type type)
543
479
{
480
+
struct list_head *list;
544
481
u32 func_no;
545
482
int ret = 0;
546
483
547
-
if (epf->epc)
484
+
if (IS_ERR_OR_NULL(epc))
485
+
return -EINVAL;
486
+
487
+
if (type == PRIMARY_INTERFACE && epf->epc)
548
488
return -EBUSY;
549
489
550
-
if (IS_ERR(epc))
551
-
return -EINVAL;
490
+
if (type == SECONDARY_INTERFACE && epf->sec_epc)
491
+
return -EBUSY;
552
492
553
493
mutex_lock(&epc->lock);
554
494
func_no = find_first_zero_bit(&epc->function_num_map,
···
572
498
}
573
499
574
500
set_bit(func_no, &epc->function_num_map);
575
-
epf->func_no = func_no;
576
-
epf->epc = epc;
501
+
if (type == PRIMARY_INTERFACE) {
502
+
epf->func_no = func_no;
503
+
epf->epc = epc;
504
+
list = &epf->list;
505
+
} else {
506
+
epf->sec_epc_func_no = func_no;
507
+
epf->sec_epc = epc;
508
+
list = &epf->sec_epc_list;
509
+
}
577
510
578
-
list_add_tail(&epf->list, &epc->pci_epf);
579
-
511
+
list_add_tail(list, &epc->pci_epf);
580
512
ret:
581
513
mutex_unlock(&epc->lock);
582
514
···
597
517
*
598
518
* Invoke to remove PCI endpoint function from the endpoint controller.
599
519
*/
600
-
void pci_epc_remove_epf(struct pci_epc *epc, struct pci_epf *epf)
520
+
void pci_epc_remove_epf(struct pci_epc *epc, struct pci_epf *epf,
521
+
enum pci_epc_interface_type type)
601
522
{
523
+
struct list_head *list;
524
+
u32 func_no = 0;
525
+
602
526
if (!epc || IS_ERR(epc) || !epf)
603
527
return;
604
528
529
+
if (type == PRIMARY_INTERFACE) {
530
+
func_no = epf->func_no;
531
+
list = &epf->list;
532
+
} else {
533
+
func_no = epf->sec_epc_func_no;
534
+
list = &epf->sec_epc_list;
535
+
}
536
+
605
537
mutex_lock(&epc->lock);
606
-
clear_bit(epf->func_no, &epc->function_num_map);
607
-
list_del(&epf->list);
538
+
clear_bit(func_no, &epc->function_num_map);
539
+
list_del(list);
608
540
epf->epc = NULL;
609
541
mutex_unlock(&epc->lock);
610
542
}
+73
-32
drivers/pci/endpoint/pci-epf-core.c
+73
-32
drivers/pci/endpoint/pci-epf-core.c
···
21
21
static const struct device_type pci_epf_type;
22
22
23
23
/**
24
+
* pci_epf_type_add_cfs() - Help function drivers to expose function specific
25
+
* attributes in configfs
26
+
* @epf: the EPF device that has to be configured using configfs
27
+
* @group: the parent configfs group (corresponding to entries in
28
+
* pci_epf_device_id)
29
+
*
30
+
* Invoke to expose function specific attributes in configfs. If the function
31
+
* driver does not have anything to expose (attributes configured by user),
32
+
* return NULL.
33
+
*/
34
+
struct config_group *pci_epf_type_add_cfs(struct pci_epf *epf,
35
+
struct config_group *group)
36
+
{
37
+
struct config_group *epf_type_group;
38
+
39
+
if (!epf->driver) {
40
+
dev_err(&epf->dev, "epf device not bound to driver\n");
41
+
return NULL;
42
+
}
43
+
44
+
if (!epf->driver->ops->add_cfs)
45
+
return NULL;
46
+
47
+
mutex_lock(&epf->lock);
48
+
epf_type_group = epf->driver->ops->add_cfs(epf, group);
49
+
mutex_unlock(&epf->lock);
50
+
51
+
return epf_type_group;
52
+
}
53
+
EXPORT_SYMBOL_GPL(pci_epf_type_add_cfs);
54
+
55
+
/**
24
56
* pci_epf_unbind() - Notify the function driver that the binding between the
25
57
* EPF device and EPC device has been lost
26
58
* @epf: the EPF device which has lost the binding with the EPC device
···
106
74
* @epf: the EPF device from whom to free the memory
107
75
* @addr: the virtual address of the PCI EPF register space
108
76
* @bar: the BAR number corresponding to the register space
77
+
* @type: Identifies if the allocated space is for primary EPC or secondary EPC
109
78
*
110
79
* Invoke to free the allocated PCI EPF register space.
111
80
*/
112
-
void pci_epf_free_space(struct pci_epf *epf, void *addr, enum pci_barno bar)
81
+
void pci_epf_free_space(struct pci_epf *epf, void *addr, enum pci_barno bar,
82
+
enum pci_epc_interface_type type)
113
83
{
114
84
struct device *dev = epf->epc->dev.parent;
85
+
struct pci_epf_bar *epf_bar;
86
+
struct pci_epc *epc;
115
87
116
88
if (!addr)
117
89
return;
118
90
119
-
dma_free_coherent(dev, epf->bar[bar].size, addr,
120
-
epf->bar[bar].phys_addr);
91
+
if (type == PRIMARY_INTERFACE) {
92
+
epc = epf->epc;
93
+
epf_bar = epf->bar;
94
+
} else {
95
+
epc = epf->sec_epc;
96
+
epf_bar = epf->sec_epc_bar;
97
+
}
121
98
122
-
epf->bar[bar].phys_addr = 0;
123
-
epf->bar[bar].addr = NULL;
124
-
epf->bar[bar].size = 0;
125
-
epf->bar[bar].barno = 0;
126
-
epf->bar[bar].flags = 0;
99
+
dev = epc->dev.parent;
100
+
dma_free_coherent(dev, epf_bar[bar].size, addr,
101
+
epf_bar[bar].phys_addr);
102
+
103
+
epf_bar[bar].phys_addr = 0;
104
+
epf_bar[bar].addr = NULL;
105
+
epf_bar[bar].size = 0;
106
+
epf_bar[bar].barno = 0;
107
+
epf_bar[bar].flags = 0;
127
108
}
128
109
EXPORT_SYMBOL_GPL(pci_epf_free_space);
129
110
···
146
101
* @size: the size of the memory that has to be allocated
147
102
* @bar: the BAR number corresponding to the allocated register space
148
103
* @align: alignment size for the allocation region
104
+
* @type: Identifies if the allocation is for primary EPC or secondary EPC
149
105
*
150
106
* Invoke to allocate memory for the PCI EPF register space.
151
107
*/
152
108
void *pci_epf_alloc_space(struct pci_epf *epf, size_t size, enum pci_barno bar,
153
-
size_t align)
109
+
size_t align, enum pci_epc_interface_type type)
154
110
{
155
-
void *space;
156
-
struct device *dev = epf->epc->dev.parent;
111
+
struct pci_epf_bar *epf_bar;
157
112
dma_addr_t phys_addr;
113
+
struct pci_epc *epc;
114
+
struct device *dev;
115
+
void *space;
158
116
159
117
if (size < 128)
160
118
size = 128;
···
167
119
else
168
120
size = roundup_pow_of_two(size);
169
121
122
+
if (type == PRIMARY_INTERFACE) {
123
+
epc = epf->epc;
124
+
epf_bar = epf->bar;
125
+
} else {
126
+
epc = epf->sec_epc;
127
+
epf_bar = epf->sec_epc_bar;
128
+
}
129
+
130
+
dev = epc->dev.parent;
170
131
space = dma_alloc_coherent(dev, size, &phys_addr, GFP_KERNEL);
171
132
if (!space) {
172
133
dev_err(dev, "failed to allocate mem space\n");
173
134
return NULL;
174
135
}
175
136
176
-
epf->bar[bar].phys_addr = phys_addr;
177
-
epf->bar[bar].addr = space;
178
-
epf->bar[bar].size = size;
179
-
epf->bar[bar].barno = bar;
180
-
epf->bar[bar].flags |= upper_32_bits(size) ?
137
+
epf_bar[bar].phys_addr = phys_addr;
138
+
epf_bar[bar].addr = space;
139
+
epf_bar[bar].size = size;
140
+
epf_bar[bar].barno = bar;
141
+
epf_bar[bar].flags |= upper_32_bits(size) ?
181
142
PCI_BASE_ADDRESS_MEM_TYPE_64 :
182
143
PCI_BASE_ADDRESS_MEM_TYPE_32;
183
144
···
338
281
return epf;
339
282
}
340
283
EXPORT_SYMBOL_GPL(pci_epf_create);
341
-
342
-
const struct pci_epf_device_id *
343
-
pci_epf_match_device(const struct pci_epf_device_id *id, struct pci_epf *epf)
344
-
{
345
-
if (!id || !epf)
346
-
return NULL;
347
-
348
-
while (*id->name) {
349
-
if (strcmp(epf->name, id->name) == 0)
350
-
return id;
351
-
id++;
352
-
}
353
-
354
-
return NULL;
355
-
}
356
-
EXPORT_SYMBOL_GPL(pci_epf_match_device);
357
284
358
285
static void pci_epf_dev_release(struct device *dev)
359
286
{
+35
-4
include/linux/pci-epc.h
+35
-4
include/linux/pci-epc.h
···
13
13
14
14
struct pci_epc;
15
15
16
+
enum pci_epc_interface_type {
17
+
UNKNOWN_INTERFACE = -1,
18
+
PRIMARY_INTERFACE,
19
+
SECONDARY_INTERFACE,
20
+
};
21
+
16
22
enum pci_epc_irq_type {
17
23
PCI_EPC_IRQ_UNKNOWN,
18
24
PCI_EPC_IRQ_LEGACY,
19
25
PCI_EPC_IRQ_MSI,
20
26
PCI_EPC_IRQ_MSIX,
21
27
};
28
+
29
+
static inline const char *
30
+
pci_epc_interface_string(enum pci_epc_interface_type type)
31
+
{
32
+
switch (type) {
33
+
case PRIMARY_INTERFACE:
34
+
return "primary";
35
+
case SECONDARY_INTERFACE:
36
+
return "secondary";
37
+
default:
38
+
return "UNKNOWN interface";
39
+
}
40
+
}
22
41
23
42
/**
24
43
* struct pci_epc_ops - set of function pointers for performing EPC operations
···
55
36
* @get_msix: ops to get the number of MSI-X interrupts allocated by the RC
56
37
* from the MSI-X capability register
57
38
* @raise_irq: ops to raise a legacy, MSI or MSI-X interrupt
39
+
* @map_msi_irq: ops to map physical address to MSI address and return MSI data
58
40
* @start: ops to start the PCI link
59
41
* @stop: ops to stop the PCI link
60
42
* @owner: the module owner containing the ops
···
78
58
int (*get_msix)(struct pci_epc *epc, u8 func_no);
79
59
int (*raise_irq)(struct pci_epc *epc, u8 func_no,
80
60
enum pci_epc_irq_type type, u16 interrupt_num);
61
+
int (*map_msi_irq)(struct pci_epc *epc, u8 func_no,
62
+
phys_addr_t phys_addr, u8 interrupt_num,
63
+
u32 entry_size, u32 *msi_data,
64
+
u32 *msi_addr_offset);
81
65
int (*start)(struct pci_epc *epc);
82
66
void (*stop)(struct pci_epc *epc);
83
67
const struct pci_epc_features* (*get_features)(struct pci_epc *epc,
···
199
175
struct module *owner);
200
176
void devm_pci_epc_destroy(struct device *dev, struct pci_epc *epc);
201
177
void pci_epc_destroy(struct pci_epc *epc);
202
-
int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf);
178
+
int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf,
179
+
enum pci_epc_interface_type type);
203
180
void pci_epc_linkup(struct pci_epc *epc);
204
181
void pci_epc_init_notify(struct pci_epc *epc);
205
-
void pci_epc_remove_epf(struct pci_epc *epc, struct pci_epf *epf);
182
+
void pci_epc_remove_epf(struct pci_epc *epc, struct pci_epf *epf,
183
+
enum pci_epc_interface_type type);
206
184
int pci_epc_write_header(struct pci_epc *epc, u8 func_no,
207
185
struct pci_epf_header *hdr);
208
186
int pci_epc_set_bar(struct pci_epc *epc, u8 func_no,
···
221
195
int pci_epc_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts,
222
196
enum pci_barno, u32 offset);
223
197
int pci_epc_get_msix(struct pci_epc *epc, u8 func_no);
198
+
int pci_epc_map_msi_irq(struct pci_epc *epc, u8 func_no,
199
+
phys_addr_t phys_addr, u8 interrupt_num,
200
+
u32 entry_size, u32 *msi_data, u32 *msi_addr_offset);
224
201
int pci_epc_raise_irq(struct pci_epc *epc, u8 func_no,
225
202
enum pci_epc_irq_type type, u16 interrupt_num);
226
203
int pci_epc_start(struct pci_epc *epc);
227
204
void pci_epc_stop(struct pci_epc *epc);
228
205
const struct pci_epc_features *pci_epc_get_features(struct pci_epc *epc,
229
206
u8 func_no);
230
-
unsigned int pci_epc_get_first_free_bar(const struct pci_epc_features
231
-
*epc_features);
207
+
enum pci_barno
208
+
pci_epc_get_first_free_bar(const struct pci_epc_features *epc_features);
209
+
enum pci_barno pci_epc_get_next_free_bar(const struct pci_epc_features
210
+
*epc_features, enum pci_barno bar);
232
211
struct pci_epc *pci_epc_get(const char *epc_name);
233
212
void pci_epc_put(struct pci_epc *epc);
234
213
+24
-4
include/linux/pci-epf.h
+24
-4
include/linux/pci-epf.h
···
9
9
#ifndef __LINUX_PCI_EPF_H
10
10
#define __LINUX_PCI_EPF_H
11
11
12
+
#include <linux/configfs.h>
12
13
#include <linux/device.h>
13
14
#include <linux/mod_devicetable.h>
14
15
#include <linux/pci.h>
15
16
16
17
struct pci_epf;
18
+
enum pci_epc_interface_type;
17
19
18
20
enum pci_notify_event {
19
21
CORE_INIT,
···
23
21
};
24
22
25
23
enum pci_barno {
24
+
NO_BAR = -1,
26
25
BAR_0,
27
26
BAR_1,
28
27
BAR_2,
···
63
60
* @bind: ops to perform when a EPC device has been bound to EPF device
64
61
* @unbind: ops to perform when a binding has been lost between a EPC device
65
62
* and EPF device
63
+
* @add_cfs: ops to initialize function specific configfs attributes
66
64
*/
67
65
struct pci_epf_ops {
68
66
int (*bind)(struct pci_epf *epf);
69
67
void (*unbind)(struct pci_epf *epf);
68
+
struct config_group *(*add_cfs)(struct pci_epf *epf,
69
+
struct config_group *group);
70
70
};
71
71
72
72
/**
···
124
118
* @list: to add pci_epf as a list of PCI endpoint functions to pci_epc
125
119
* @nb: notifier block to notify EPF of any EPC events (like linkup)
126
120
* @lock: mutex to protect pci_epf_ops
121
+
* @sec_epc: the secondary EPC device to which this EPF device is bound
122
+
* @sec_epc_list: to add pci_epf as list of PCI endpoint functions to secondary
123
+
* EPC device
124
+
* @sec_epc_bar: represents the BAR of EPF device associated with secondary EPC
125
+
* @sec_epc_func_no: unique (physical) function number within the secondary EPC
126
+
* @group: configfs group associated with the EPF device
127
127
*/
128
128
struct pci_epf {
129
129
struct device dev;
···
146
134
struct notifier_block nb;
147
135
/* mutex to protect against concurrent access of pci_epf_ops */
148
136
struct mutex lock;
137
+
138
+
/* Below members are to attach secondary EPC to an endpoint function */
139
+
struct pci_epc *sec_epc;
140
+
struct list_head sec_epc_list;
141
+
struct pci_epf_bar sec_epc_bar[6];
142
+
u8 sec_epc_func_no;
143
+
struct config_group *group;
149
144
};
150
145
151
146
/**
···
183
164
return dev_get_drvdata(&epf->dev);
184
165
}
185
166
186
-
const struct pci_epf_device_id *
187
-
pci_epf_match_device(const struct pci_epf_device_id *id, struct pci_epf *epf);
188
167
struct pci_epf *pci_epf_create(const char *name);
189
168
void pci_epf_destroy(struct pci_epf *epf);
190
169
int __pci_epf_register_driver(struct pci_epf_driver *driver,
191
170
struct module *owner);
192
171
void pci_epf_unregister_driver(struct pci_epf_driver *driver);
193
172
void *pci_epf_alloc_space(struct pci_epf *epf, size_t size, enum pci_barno bar,
194
-
size_t align);
195
-
void pci_epf_free_space(struct pci_epf *epf, void *addr, enum pci_barno bar);
173
+
size_t align, enum pci_epc_interface_type type);
174
+
void pci_epf_free_space(struct pci_epf *epf, void *addr, enum pci_barno bar,
175
+
enum pci_epc_interface_type type);
196
176
int pci_epf_bind(struct pci_epf *epf);
197
177
void pci_epf_unbind(struct pci_epf *epf);
178
+
struct config_group *pci_epf_type_add_cfs(struct pci_epf *epf,
179
+
struct config_group *group);
198
180
#endif /* __LINUX_PCI_EPF_H */
+1
include/linux/pci_ids.h
+1
include/linux/pci_ids.h