tjh.dev
Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
1perf-c2c(1)
2===========
3
4NAME
5----
6perf-c2c - Shared Data C2C/HITM Analyzer.
7
8SYNOPSIS
9--------
10[verse]
11'perf c2c record' [<options>] <command>
12'perf c2c record' [<options>] \-- [<record command options>] <command>
13'perf c2c report' [<options>]
14
15DESCRIPTION
16-----------
17C2C stands for Cache To Cache.
18
19The perf c2c tool provides means for Shared Data C2C/HITM analysis. It allows
20you to track down the cacheline contentions.
21
22On Intel, the tool is based on load latency and precise store facility events
23provided by Intel CPUs. On PowerPC, the tool uses random instruction sampling
24with thresholding feature. On AMD, the tool uses IBS op pmu (due to hardware
25limitations, perf c2c is not supported on Zen3 cpus). On Arm64 it uses SPE to
26sample load and store operations, therefore hardware and kernel support is
27required. See linkperf:perf-arm-spe[1] for a setup guide. Due to the
28statistical nature of Arm SPE sampling, not every memory operation will be
29sampled.
30
31These events provide:
32 - memory address of the access
33 - type of the access (load and store details)
34 - latency (in cycles) of the load access
35
36The c2c tool provide means to record this data and report back access details
37for cachelines with highest contention - highest number of HITM accesses.
38
39The basic workflow with this tool follows the standard record/report phase.
40User uses the record command to record events data and report command to
41display it.
42
43
44RECORD OPTIONS
45--------------
46-e::
47--event=::
48 Select the PMU event. Use 'perf c2c record -e list'
49 to list available events.
50
51-v::
52--verbose::
53 Be more verbose (show counter open errors, etc).
54
55-l::
56--ldlat::
57 Configure mem-loads latency. Supported on Intel, Arm64 and some AMD
58 processors. Ignored on other archs.
59
60 On supported AMD processors:
61 - /sys/bus/event_source/devices/ibs_op/caps/ldlat file contains '1'.
62 - Supported latency values are 128 to 2048 (both inclusive).
63 - Latency value which is a multiple of 128 incurs a little less profiling
64 overhead compared to other values.
65 - Load latency filtering is disabled by default.
66
67-k::
68--all-kernel::
69 Configure all used events to run in kernel space.
70
71-u::
72--all-user::
73 Configure all used events to run in user space.
74
75REPORT OPTIONS
76--------------
77-k::
78--vmlinux=<file>::
79 vmlinux pathname
80
81-v::
82--verbose::
83 Be more verbose (show counter open errors, etc).
84
85-i::
86--input::
87 Specify the input file to process.
88
89-N::
90--node-info::
91 Show extra node info in report (see NODE INFO section)
92
93-c::
94--coalesce::
95 Specify sorting fields for single cacheline display.
96 Following fields are available: tid,pid,iaddr,dso
97 (see COALESCE)
98
99-g::
100--call-graph::
101 Setup callchains parameters.
102 Please refer to perf-report man page for details.
103
104--stdio::
105 Force the stdio output (see STDIO OUTPUT)
106
107--stats::
108 Display only statistic tables and force stdio mode.
109
110--full-symbols::
111 Display full length of symbols.
112
113--no-source::
114 Do not display Source:Line column.
115
116--show-all::
117 Show all captured HITM lines, with no regard to HITM % 0.0005 limit.
118
119-f::
120--force::
121 Don't do ownership validation.
122
123-d::
124--display::
125 Switch to HITM type (rmt, lcl) or peer snooping type (peer) to display
126 and sort on. Total HITMs (tot) as default, except Arm64 uses peer mode
127 as default.
128
129--stitch-lbr::
130 Show callgraph with stitched LBRs, which may have more complete
131 callgraph. The perf.data file must have been obtained using
132 perf c2c record --call-graph lbr.
133 Disabled by default. In common cases with call stack overflows,
134 it can recreate better call stacks than the default lbr call stack
135 output. But this approach is not foolproof. There can be cases
136 where it creates incorrect call stacks from incorrect matches.
137 The known limitations include exception handing such as
138 setjmp/longjmp will have calls/returns not match.
139
140--double-cl::
141 Group the detection of shared cacheline events into double cacheline
142 granularity. Some architectures have an Adjacent Cacheline Prefetch
143 feature, which causes cacheline sharing to behave like the cacheline
144 size is doubled.
145
146C2C RECORD
147----------
148The perf c2c record command setup options related to HITM cacheline analysis
149and calls standard perf record command.
150
151Following perf record options are configured by default:
152(check perf record man page for details)
153
154 -W,-d,--phys-data,--sample-cpu
155
156Unless specified otherwise with '-e' option, following events are monitored by
157default on Intel:
158
159 cpu/mem-loads,ldlat=30/P
160 cpu/mem-stores/P
161
162following on AMD:
163
164 ibs_op//
165
166and following on PowerPC:
167
168 cpu/mem-loads/
169 cpu/mem-stores/
170
171User can pass any 'perf record' option behind '--' mark, like (to enable
172callchains and system wide monitoring):
173
174 $ perf c2c record -- -g -a
175
176Please check RECORD OPTIONS section for specific c2c record options.
177
178C2C REPORT
179----------
180The perf c2c report command displays shared data analysis. It comes in two
181display modes: stdio and tui (default).
182
183The report command workflow is following:
184 - sort all the data based on the cacheline address
185 - store access details for each cacheline
186 - sort all cachelines based on user settings
187 - display data
188
189In general perf report output consist of 2 basic views:
190 1) most expensive cachelines list
191 2) offsets details for each cacheline
192
193For each cacheline in the 1) list we display following data:
194(Both stdio and TUI modes follow the same fields output)
195
196 Index
197 - zero based index to identify the cacheline
198
199 Cacheline
200 - cacheline address (hex number)
201
202 Rmt/Lcl Hitm (Display with HITM types)
203 - cacheline percentage of all Remote/Local HITM accesses
204
205 Peer Snoop (Display with peer type)
206 - cacheline percentage of all peer accesses
207
208 LLC Load Hitm - Total, LclHitm, RmtHitm (For display with HITM types)
209 - count of Total/Local/Remote load HITMs
210
211 Load Peer - Total, Local, Remote (For display with peer type)
212 - count of Total/Local/Remote load from peer cache or DRAM
213
214 Total records
215 - sum of all cachelines accesses
216
217 Total loads
218 - sum of all load accesses
219
220 Total stores
221 - sum of all store accesses
222
223 Store Reference - L1Hit, L1Miss, N/A
224 L1Hit - store accesses that hit L1
225 L1Miss - store accesses that missed L1
226 N/A - store accesses with memory level is not available
227
228 Core Load Hit - FB, L1, L2
229 - count of load hits in FB (Fill Buffer), L1 and L2 cache
230
231 LLC Load Hit - LlcHit, LclHitm
232 - count of LLC load accesses, includes LLC hits and LLC HITMs
233
234 RMT Load Hit - RmtHit, RmtHitm
235 - count of remote load accesses, includes remote hits and remote HITMs;
236 on Arm neoverse cores, RmtHit is used to account remote accesses,
237 includes remote DRAM or any upward cache level in remote node
238
239 Load Dram - Lcl, Rmt
240 - count of local and remote DRAM accesses
241
242For each offset in the 2) list we display following data:
243
244 HITM - Rmt, Lcl (Display with HITM types)
245 - % of Remote/Local HITM accesses for given offset within cacheline
246
247 Peer Snoop - Rmt, Lcl (Display with peer type)
248 - % of Remote/Local peer accesses for given offset within cacheline
249
250 Store Refs - L1 Hit, L1 Miss, N/A
251 - % of store accesses that hit L1, missed L1 and N/A (no available) memory
252 level for given offset within cacheline
253
254 Data address - Offset
255 - offset address
256
257 Pid
258 - pid of the process responsible for the accesses
259
260 Tid
261 - tid of the process responsible for the accesses
262
263 Code address
264 - code address responsible for the accesses
265
266 cycles - rmt hitm, lcl hitm, load (Display with HITM types)
267 - sum of cycles for given accesses - Remote/Local HITM and generic load
268
269 cycles - rmt peer, lcl peer, load (Display with peer type)
270 - sum of cycles for given accesses - Remote/Local peer load and generic load
271
272 cpu cnt
273 - number of cpus that participated on the access
274
275 Symbol
276 - code symbol related to the 'Code address' value
277
278 Shared Object
279 - shared object name related to the 'Code address' value
280
281 Source:Line
282 - source information related to the 'Code address' value
283
284 Node
285 - nodes participating on the access (see NODE INFO section)
286
287NODE INFO
288---------
289The 'Node' field displays nodes that accesses given cacheline
290offset. Its output comes in 3 flavors:
291 - node IDs separated by ','
292 - node IDs with stats for each ID, in following format:
293 Node{cpus %hitms %stores} (Display with HITM types)
294 Node{cpus %peers %stores} (Display with peer type)
295 - node IDs with list of affected CPUs in following format:
296 Node{cpu list}
297
298User can switch between above flavors with -N option or
299use 'n' key to interactively switch in TUI mode.
300
301COALESCE
302--------
303User can specify how to sort offsets for cacheline.
304
305Following fields are available and governs the final
306output fields set for cacheline offsets output:
307
308 tid - coalesced by process TIDs
309 pid - coalesced by process PIDs
310 iaddr - coalesced by code address, following fields are displayed:
311 Code address, Code symbol, Shared Object, Source line
312 dso - coalesced by shared object
313
314By default the coalescing is setup with 'pid,iaddr'.
315
316STDIO OUTPUT
317------------
318The stdio output displays data on standard output.
319
320Following tables are displayed:
321 Trace Event Information
322 - overall statistics of memory accesses
323
324 Global Shared Cache Line Event Information
325 - overall statistics on shared cachelines
326
327 Shared Data Cache Line Table
328 - list of most expensive cachelines
329
330 Shared Cache Line Distribution Pareto
331 - list of all accessed offsets for each cacheline
332
333TUI OUTPUT
334----------
335The TUI output provides interactive interface to navigate
336through cachelines list and to display offset details.
337
338For details please refer to the help window by pressing '?' key.
339
340CREDITS
341-------
342Although Don Zickus, Dick Fowles and Joe Mario worked together
343to get this implemented, we got lots of early help from Arnaldo
344Carvalho de Melo, Stephane Eranian, Jiri Olsa and Andi Kleen.
345
346C2C BLOG
347--------
348Check Joe's blog on c2c tool for detailed use case explanation:
349 https://joemario.github.io/blog/2016/09/01/c2c-blog/
350
351SEE ALSO
352--------
353linkperf:perf-record[1], linkperf:perf-mem[1], linkperf:perf-arm-spe[1]