You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 13:37
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack5slot0.osadl.org (updated Sun Apr 28, 2024 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1593621300,3sleep01460999cyclictest22:16:390
3172121220,0sleep131720-21sshd22:07:391
433621210,3sleep11461099cyclictest22:37:041
1338221150,2sleep213379-21sshd21:36:212
14225211240,44sleep30-21swapper/319:07:223
2858421100,3sleep21461199cyclictest22:14:322
43192980,3sleep20-21swapper/222:29:552
44202950,2sleep30-21swapper/300:04:353
219572910,1sleep021932-21sshd22:35:220
25042900,2sleep22505-21sshd23:49:262
1426428933,11sleep20-21swapper/219:07:532
71552880,0sleep00-21swapper/000:16:140
50692880,0sleep00-21swapper/000:04:400
122442880,2sleep01460999cyclictest00:13:050
216532870,2sleep30-21swapper/321:19:273
124502850,2sleep212449-21sshd21:54:412
1437628437,43sleep10-21swapper/119:09:161
283452830,2sleep328335-21sshd00:15:003
265262820,2sleep21876-21nfsd22:35:532
51722790,2sleep20-21swapper/223:24:222
287542790,6sleep1138550irq/27-eth000:22:401
109752790,1sleep12121-21runrttasks21:46:581
35562770,4sleep21461199cyclictest21:10:162
177822770,1sleep20-21swapper/223:43:532
221742760,0sleep222175-21sshd21:33:492
232772750,1sleep00-21swapper/021:30:180
177612730,1sleep017750-21bash21:55:150
28812720,1sleep30-21swapper/300:38:503
145462700,4sleep21461199cyclictest21:25:432
145462700,4sleep21461199cyclictest21:25:432
156852690,1sleep20-21swapper/222:56:202
185642610,1sleep11461099cyclictest00:29:091
1433125936,11sleep00-21swapper/019:08:430
245772360,2sleep30-21swapper/322:06:483
287592330,31sleep1138550irq/27-eth000:15:031
245692320,30sleep328750irq/26-ahci[00021:26:533
245692320,30sleep328750irq/26-ahci[00021:26:533
236512320,2sleep10-21swapper/122:50:071
236512320,2sleep10-21swapper/122:50:061
256642290,2sleep325668-21sshd00:26:023
138062290,1sleep013805-21seq22:52:340
138062290,1sleep013805-21seq22:52:340
260592280,1sleep00-21swapper/023:59:300
87442270,2sleep10-21swapper/100:35:301
132422270,2sleep00-21swapper/023:17:580
88922260,2sleep28891-21bash23:35:322
75702260,2sleep20-21swapper/223:13:442
75702260,2sleep20-21swapper/223:13:442
72132250,2sleep11921-21nscd22:44:421
69372250,23sleep1138550irq/27-eth023:46:141
32112250,2sleep10-21swapper/123:53:061
290552252,21sleep2138550irq/27-eth022:03:442
110942250,2sleep20-21swapper/200:16:432
247992240,3sleep20-21swapper/221:59:422
1460999243,14cyclictest23873-21bash22:57:170
141212240,22sleep2138550irq/27-eth022:09:202
118602242,20sleep3138550irq/27-eth023:25:093
164272230,1sleep10-21swapper/123:07:261
117382230,2sleep111740-21sshd21:39:531
40732220,20sleep3138550irq/27-eth023:42:193
30152220,2sleep12121-21runrttasks00:31:021
20012220,2sleep20-21swapper/200:30:542
177072220,1sleep017705-21bash23:40:160
51462210,1sleep00-21swapper/021:53:500
1460999211,13cyclictest0-21swapper/021:26:460
1460999211,13cyclictest0-21swapper/021:26:460
217462200,1sleep00-21swapper/023:26:090
14609992010,8cyclictest3-21ksoftirqd/000:21:260
114282200,1sleep10-21swapper/120:20:191
322272190,16sleep310950irq/23-ehci_hcd22:58:153
14612991914,3cyclictest1877-21nfsd21:49:203
1460999197,11cyclictest3-21ksoftirqd/021:10:140
1460999196,11cyclictest0-21swapper/000:37:420
1460999194,13cyclictest0-21swapper/022:06:000
1460999191,16cyclictest0-21swapper/022:21:270
1460999190,2cyclictest0-21swapper/021:40:210
1460999190,17cyclictest0-21swapper/022:44:380
1460999190,13cyclictest0-21swapper/000:34:230
144252190,17sleep1138550irq/27-eth021:51:121
60392180,16sleep3138550irq/27-eth022:19:153
325242182,14sleep128750irq/26-ahci[00022:00:261
247572180,2sleep00-21swapper/023:04:450
247572180,2sleep00-21swapper/023:04:440
1461199182,10cyclictest0-21swapper/219:56:332
1460999189,5cyclictest0-21swapper/023:20:140
1460999181,11cyclictest0-21swapper/020:15:010
1460999180,9cyclictest19732-21sshd00:06:170
1460999180,4cyclictest0-21swapper/022:12:010
1460999180,12cyclictest10490-21sshd23:39:290
37212170,15sleep2138550irq/27-eth023:16:532
1461299170,14cyclictest18029-21sshd21:40:273
1461199174,12cyclictest0-21swapper/222:23:532
1461199172,7cyclictest138550irq/27-eth000:15:012
1461199172,13cyclictest18353-21sshd23:00:172
1461199172,13cyclictest18353-21sshd23:00:162
1461199172,10cyclictest0-21swapper/221:01:392
1461199171,7cyclictest138550irq/27-eth023:07:132
1461199171,13cyclictest26893-21sshd21:16:212
1461199171,13cyclictest0-21swapper/220:22:352
1461099175,11cyclictest10950irq/23-ehci_hcd20:44:061
1461099173,8cyclictest10463-21sshd21:58:041
14610991713,2cyclictest0-21swapper/121:40:111
1461099170,12cyclictest1876-21nfsd00:14:311
1460999178,8cyclictest1987-21sshd21:36:310
1460999178,8cyclictest0-21swapper/021:05:410
1460999178,5cyclictest5902-21sshd23:06:110
1460999178,5cyclictest0-21swapper/022:31:460
1460999174,9cyclictest29220-21kworker/0:119:20:190
1460999174,5cyclictest20068-21sshd22:46:010
1460999174,2cyclictest3704-21bash21:20:560
1460999173,5cyclictest24110-21bash23:30:090
1460999172,13cyclictest0-21swapper/023:50:460
1460999170,16cyclictest0-21swapper/022:00:280
1460999170,16cyclictest0-21swapper/021:00:580
1460999170,16cyclictest0-21swapper/000:25:500
1460999170,14cyclictest0-21swapper/020:35:160
1460999170,14cyclictest0-21swapper/019:37:120
1460999170,12cyclictest10027-21sshd22:26:520
52772160,14sleep2138550irq/27-eth022:19:102
304772160,2sleep20-21swapper/223:52:322
229332160,8sleep20-21swapper/200:21:572
204702160,2sleep10-21swapper/123:22:251
1461299169,0cyclictest22483-21sshd21:12:203
1461299166,5cyclictest2349-21sshd00:19:353
1461299162,2cyclictest21877-21sshd21:59:213
1461299160,14cyclictest16044-21bash23:00:043
1461299160,14cyclictest16044-21bash23:00:033
1461299160,14cyclictest13875-21rm23:10:423
1461299160,14cyclictest13875-21rm23:10:423
1461199169,6cyclictest26838-21sshd21:45:082
1461199163,8cyclictest0-21swapper/220:40:172
1461199163,6cyclictest138550irq/27-eth000:29:502
1461199163,11cyclictest0-21swapper/220:03:452
1461199162,6cyclictest0-21swapper/200:07:022
1461199162,11cyclictest0-21swapper/222:48:152
1461199161,12cyclictest23963-21sshd21:41:082
1461199161,12cyclictest0-21swapper/223:34:282
1461199160,11cyclictest23052-21sshd22:53:382
1461199160,11cyclictest23052-21sshd22:53:372
1461199160,11cyclictest0-21swapper/220:37:152
1461099167,7cyclictest0-21swapper/123:30:141
1461099164,11cyclictest0-21swapper/123:37:431
1461099162,12cyclictest0-21swapper/122:45:201
14610991611,4cyclictest18415-21bash21:33:221
14610991610,4cyclictest9882-21bash22:12:231
1461099160,14cyclictest17388-21id23:00:111
1461099160,14cyclictest17388-21id23:00:111
1461099160,14cyclictest0-21swapper/123:29:261
1461099160,14cyclictest0-21swapper/120:05:141
1460999167,8cyclictest0-21swapper/020:47:020
1460999167,5cyclictest0-21swapper/021:46:430
1460999166,6cyclictest0-21swapper/023:10:400
1460999166,6cyclictest0-21swapper/023:10:390
1460999160,5cyclictest31485-21sh19:55:000
1460999160,15cyclictest7804-21sshd23:46:190
1460999160,15cyclictest0-21swapper/021:15:480
1460999160,15cyclictest0-21swapper/020:55:300
1460999160,15cyclictest0-21swapper/020:25:090
1460999160,15cyclictest0-21swapper/020:19:340
1461299157,4cyclictest0-21swapper/322:10:133
1461299155,1cyclictest0-21swapper/323:08:103
1461299153,4cyclictest0-21swapper/323:50:353
14612991512,2cyclictest0-21swapper/323:23:243
14612991511,3cyclictest27941-21sshd23:48:403
1461299150,6cyclictest0-21swapper/322:03:193
1461299150,6cyclictest0-21swapper/321:52:213
1461299150,3cyclictest32380-21sshd22:40:153
1461299150,14cyclictest21937-21sshd00:33:203
1461299150,14cyclictest1987-21sshd23:39:363
1461299150,14cyclictest0-21swapper/322:22:413
1461299150,14cyclictest0-21swapper/321:37:363
1461299150,13cyclictest31164-21bash21:20:243
1461299150,12cyclictest0-21swapper/320:54:083
1461299150,10cyclictest0-21swapper/323:15:463
1461199157,6cyclictest0-21swapper/221:21:152
1461199155,0cyclictest32008-21sshd00:38:242
1461199153,7cyclictest0-21swapper/220:18:192
1461199153,10cyclictest138550irq/27-eth019:45:172
1461199152,12cyclictest0-21swapper/220:12:332
1461199152,10cyclictest0-21swapper/220:50:192
1461199151,4cyclictest138550irq/27-eth023:56:092
1461199151,12cyclictest28750irq/26-ahci[00000:02:272
1461199150,13cyclictest14741-21sshd22:30:582
1461199150,13cyclictest0-21swapper/223:25:572
1461199150,13cyclictest0-21swapper/220:47:592
1461199150,11cyclictest8798-21sshd22:44:522
1461099157,3cyclictest0-21swapper/121:28:531
1461099157,3cyclictest0-21swapper/121:28:531
1461099152,4cyclictest17018-21sshd22:20:231
1461099152,11cyclictest3999-21sshd00:00:331
14610991512,2cyclictest0-21swapper/122:27:051
1461099151,13cyclictest0-21swapper/121:11:191
1461099151,12cyclictest22265-21nfsd23:15:191
14610991510,4cyclictest12099-21sshd23:14:161
14610991510,4cyclictest12099-21sshd23:14:161
1461099150,3cyclictest20973-21sshd23:44:151
1461099150,13cyclictest12351-21sshd22:55:571
1460999157,7cyclictest0-21swapper/020:51:520
1460999157,7cyclictest0-21swapper/020:06:460
1460999157,7cyclictest0-21swapper/019:31:020
1460999151,13cyclictest0-21swapper/019:45:140
1460999150,1cyclictest0-21swapper/020:30:440
1460999150,13cyclictest0-21swapper/019:15:120
1461299142,7cyclictest26077-21sshd00:07:033
1461299140,5cyclictest8634-21sshd22:30:203
1461299140,5cyclictest0-21swapper/321:30:463
1461299140,13cyclictest2121-21runrttasks20:25:253
1461299140,13cyclictest1987-21sshd00:21:253
1461299140,13cyclictest0-21swapper/322:27:453
1461299140,13cyclictest0-21swapper/320:15:253
1461299140,12cyclictest29920-21snmpd20:20:083
1461299140,10cyclictest138550irq/27-eth023:55:033
1461199143,7cyclictest138550irq/27-eth020:09:372
1461199143,10cyclictest10950irq/23-ehci_hcd19:34:532
1461199142,9cyclictest0-21swapper/220:26:332
1461199142,7cyclictest0-21swapper/219:24:332
1461099149,4cyclictest138550irq/27-eth000:06:301
1461099143,2cyclictest0-21swapper/123:55:181
1461099142,3cyclictest18072-21bash21:15:221
1461099141,12cyclictest15583-21sshd22:16:371
1461099141,12cyclictest0-21swapper/122:30:081
14610991410,3cyclictest0-21swapper/120:47:321
1461099140,3cyclictest138550irq/27-eth020:25:591
1461099140,11cyclictest0-21swapper/121:01:051
1460999141,1cyclictest138550irq/27-eth019:10:190
1460999140,1cyclictest0-21swapper/019:55:480
1460999140,1cyclictest0-21swapper/019:40:010
1460999140,1cyclictest0-21swapper/019:28:360
1460999140,13cyclictest29920-21snmpd20:20:070
1460999140,13cyclictest0-21swapper/020:41:030
1460999140,13cyclictest0-21swapper/020:00:390
1461299137,5cyclictest10950irq/23-ehci_hcd23:31:243
1461299135,4cyclictest28972-21sshd22:36:103
1461299132,10cyclictest138550irq/27-eth020:06:593
1461299131,11cyclictest1987-21sshd22:46:303
14612991311,1cyclictest0-21swapper/320:48:373
14612991311,1cyclictest0-21swapper/319:56:343
1461299130,4cyclictest20351-21hwlatdetect20:40:163
1461299130,12cyclictest28750irq/26-ahci[00020:10:043
1461299130,12cyclictest1877-21nfsd19:31:523
1461299130,12cyclictest0-21swapper/320:57:033
1461299130,12cyclictest0-21swapper/320:35:313
1461299130,12cyclictest0-21swapper/319:40:083
1461299130,12cyclictest0-21swapper/319:35:083
1461299130,12cyclictest0-21swapper/319:25:523
1461299130,12cyclictest0-21swapper/319:23:403
1461299130,12cyclictest0-21swapper/319:15:323
1461299130,12cyclictest0-21swapper/319:13:333
1461299130,11cyclictest30938-21munin-node21:05:183
1461299130,11cyclictest26326-21sshd22:50:193
1461299130,11cyclictest26326-21sshd22:50:193
1461299130,11cyclictest15925-21iostat_ios20:30:143
1461199138,4cyclictest2121-21runrttasks20:30:172
1461199138,4cyclictest1955-21snmpd21:05:102
1461199133,7cyclictest0-21swapper/219:36:332
1461199131,4cyclictest138550irq/27-eth019:44:362
1461199130,12cyclictest0-21swapper/220:56:152
1461199130,12cyclictest0-21swapper/219:10:072
1461099138,4cyclictest2375-21if_err_eth020:00:181
1461099133,9cyclictest0-21swapper/119:42:121
1461099130,5cyclictest0-21swapper/119:35:021
1461099130,4cyclictest9076-21munin-node20:15:161
1461099130,4cyclictest2121-21runrttasks19:55:261
1461099130,4cyclictest1875-21nfsd21:20:401
1461099130,12cyclictest19052-21uptime20:35:291
1461099130,12cyclictest0-21swapper/120:55:031
1461099130,12cyclictest0-21swapper/120:31:101
1461099130,12cyclictest0-21swapper/120:11:451
1461099130,12cyclictest0-21swapper/119:50:091
1461099130,12cyclictest0-21swapper/119:48:111
1461099130,12cyclictest0-21swapper/119:25:501
1461099130,11cyclictest14972-21df19:10:081
1461299127,4cyclictest0-21swapper/320:00:243
1461299120,3cyclictest29088-21df19:45:083
1461299120,3cyclictest0-21swapper/321:00:213
1461199127,4cyclictest0-21swapper/219:51:472
1461199120,3cyclictest22182-21iostat19:25:172
1461199120,3cyclictest17624-21hwlatdetect19:15:142
1461099127,4cyclictest23702-21munin-node19:30:221
1461099120,3cyclictest20826-21users19:20:311
1461099120,11cyclictest0-21swapper/119:15:051
1461099120,10cyclictest24854-21munin-node20:50:211
1461099120,10cyclictest2121-21runrttasks21:05:021
1461299110,10cyclictest0-21swapper/319:50:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional