You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 05:46
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot2.osadl.org (updated Sat Nov 22, 2025 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
289512204188,11sleep10-21swapper/119:09:001
290172195178,12sleep00-21swapper/019:09:510
288142191175,11sleep30-21swapper/319:07:133
29019211497,12sleep20-21swapper/219:09:532
249522480,0sleep30-21swapper/300:09:273
133382450,0sleep10-21swapper/123:46:351
285042440,0sleep30-21swapper/322:32:513
181672440,0sleep30-21swapper/300:21:483
14352440,0sleep00-21swapper/019:15:190
14552430,0sleep20-21swapper/221:14:352
223162400,0sleep10-21swapper/123:18:531
130832370,0sleep30-21swapper/323:30:163
150402290,0sleep340-21ksoftirqd/300:36:293
68042120,0sleep00-21swapper/022:54:540
61032100,0sleep10-21swapper/122:54:271
2921699102,4cyclictest4831-21munin-run20:30:000
13251290,0sleep00-21swapper/019:40:170
292169981,3cyclictest16887-21cron23:00:000
292169973,4cyclictest32304-21munin-run20:20:000
292169972,5cyclictest19565-21munin-run22:45:000
292169971,3cyclictest18704-21perf23:50:000
292239960,5cyclictest0-21swapper/321:55:143
292239950,5cyclictest27107-21perl00:10:143
292239950,5cyclictest0-21swapper/322:49:503
292239950,5cyclictest0-21swapper/320:54:263
292239950,4cyclictest29465-21ssh00:27:393
292239950,4cyclictest22385-21expr23:35:133
292239950,4cyclictest0-21swapper/321:40:513
292199954,1cyclictest0-21swapper/221:58:042
292199950,5cyclictest0-21swapper/222:03:192
292199950,5cyclictest0-21swapper/219:30:142
292199950,5cyclictest0-21swapper/200:25:372
292199950,5cyclictest0-21swapper/200:08:532
292199950,0cyclictest0-21swapper/222:14:312
292179955,0cyclictest0-21swapper/123:57:191
292179951,2cyclictest0-21swapper/122:20:111
292179951,2cyclictest0-21swapper/121:25:131
292179951,2cyclictest0-21swapper/119:40:151
292169955,0cyclictest0-21swapper/021:54:590
292169950,5cyclictest0-21swapper/021:37:050
292169950,4cyclictest0-21swapper/022:35:010
292169950,0cyclictest0-21swapper/023:50:050
13946250,0sleep20-21swapper/220:45:202
11652250,0sleep00-21swapper/022:08:060
292239944,0cyclictest0-21swapper/323:10:033
292239944,0cyclictest0-21swapper/322:02:413
292239942,1cyclictest5359-21ssh22:20:443
292239941,2cyclictest22828-21aten2_r5power_p21:25:143
292239940,4cyclictest6292-21irqstats19:25:183
292239940,4cyclictest0-21swapper/323:55:103
292239940,4cyclictest0-21swapper/323:50:103
292239940,4cyclictest0-21swapper/323:42:413
292239940,4cyclictest0-21swapper/322:55:553
292239940,4cyclictest0-21swapper/322:50:143
292239940,4cyclictest0-21swapper/322:10:123
292239940,4cyclictest0-21swapper/322:05:113
292239940,4cyclictest0-21swapper/321:45:113
292239940,4cyclictest0-21swapper/321:33:493
292239940,4cyclictest0-21swapper/320:30:133
292239940,4cyclictest0-21swapper/320:15:113
292239940,4cyclictest0-21swapper/319:55:173
292239940,4cyclictest0-21swapper/319:40:153
292239940,4cyclictest0-21swapper/300:00:123
292239940,3cyclictest9885-21rm22:40:003
292239940,3cyclictest4824-21latency_hist20:30:003
292239940,3cyclictest3861-21sh00:30:413
292239940,3cyclictest3146-21df19:20:153
292239940,3cyclictest293072sleep319:10:053
292239940,3cyclictest25932-21hddtemp_smartct20:05:153
292239940,3cyclictest2499-21ssh21:15:053
292239940,3cyclictest23647-21ntp_states21:05:183
292239940,3cyclictest22398-21ssh23:02:113
292239940,3cyclictest19667-21latency_hist21:00:003
292239940,3cyclictest18975-21ssh21:22:593
292239940,1cyclictest9329-21/usr/sbin/munin23:45:133
292239940,1cyclictest3886-21/usr/sbin/munin00:15:183
292239940,1cyclictest27562-21/usr/sbin/munin20:10:113
292239940,1cyclictest27325-21/usr/sbin/munin23:05:203
292239940,1cyclictest16120-21ntp_states23:15:173
292239940,1cyclictest10233-21/usr/sbin/munin19:35:133
292239940,0cyclictest0-21swapper/323:25:113
292239940,0cyclictest0-21swapper/323:21:243
292199943,1cyclictest30795-21cut22:50:172
292199941,3cyclictest714-21snmpd22:20:102
292199941,3cyclictest6397-21ssh23:26:452
292199941,3cyclictest25592-21rm21:42:212
292199941,2cyclictest0-21swapper/200:20:112
292199941,1cyclictest61-21kswapd022:05:112
292199941,1cyclictest0-21swapper/219:25:142
292199940,4cyclictest4455-21df_inode00:15:142
292199940,4cyclictest26307-21python23:20:192
292199940,4cyclictest24622-21diskmemload22:15:122
292199940,4cyclictest0-21swapper/223:55:122
292199940,4cyclictest0-21swapper/223:51:562
292199940,4cyclictest0-21swapper/223:47:152
292199940,4cyclictest0-21swapper/223:40:022
292199940,4cyclictest0-21swapper/223:16:272
292199940,4cyclictest0-21swapper/223:05:062
292199940,4cyclictest0-21swapper/223:00:132
292199940,4cyclictest0-21swapper/222:55:122
292199940,4cyclictest0-21swapper/222:46:402
292199940,4cyclictest0-21swapper/222:40:472
292199940,4cyclictest0-21swapper/222:30:122
292199940,4cyclictest0-21swapper/221:50:132
292199940,4cyclictest0-21swapper/221:45:112
292199940,4cyclictest0-21swapper/221:21:172
292199940,4cyclictest0-21swapper/221:15:032
292199940,4cyclictest0-21swapper/220:00:132
292199940,4cyclictest0-21swapper/219:13:212
292199940,4cyclictest0-21swapper/200:36:462
292199940,4cyclictest0-21swapper/200:30:122
292199940,4cyclictest0-21swapper/200:12:222
292199940,4cyclictest0-21swapper/200:00:122
292199940,3cyclictest7444-21ssh21:33:272
292199940,3cyclictest6935-21grep23:10:252
292199940,3cyclictest32259-21cat22:35:002
292199940,3cyclictest28850-21ssh23:38:262
292199940,3cyclictest2555-21cat20:25:002
292199940,3cyclictest2371-21latency_hist19:19:592
292199940,3cyclictest15628-21ssh22:25:582
292199940,3cyclictest121062sleep220:43:202
292199940,3cyclictest0-21swapper/219:50:002
292179944,0cyclictest0-21swapper/123:28:301
292179944,0cyclictest0-21swapper/123:24:431
292179944,0cyclictest0-21swapper/121:58:181
292179944,0cyclictest0-21swapper/121:35:531
292179944,0cyclictest0-21swapper/100:16:221
292179941,2cyclictest0-21swapper/123:00:131
292179941,2cyclictest0-21swapper/120:40:091
292179941,2cyclictest0-21swapper/119:45:111
292179941,2cyclictest0-21swapper/119:20:131
292179940,4cyclictest0-21swapper/123:05:051
292179940,4cyclictest0-21swapper/122:13:161
292179940,4cyclictest0-21swapper/121:53:031
292179940,0cyclictest0-21swapper/123:50:281
292179940,0cyclictest0-21swapper/123:43:381
292179940,0cyclictest0-21swapper/121:47:491
292179940,0cyclictest0-21swapper/121:16:411
292179940,0cyclictest0-21swapper/100:25:321
292169944,0cyclictest29299-21ssh21:28:130
292169944,0cyclictest0-21swapper/023:31:530
292169944,0cyclictest0-21swapper/023:28:060
292169944,0cyclictest0-21swapper/023:24:190
292169944,0cyclictest0-21swapper/022:00:130
292169944,0cyclictest0-21swapper/000:17:260
292169944,0cyclictest0-21swapper/000:10:350
292169944,0cyclictest0-21swapper/000:00:420
292169943,1cyclictest4875-21munin-run19:25:000
292169942,1cyclictest0-21swapper/000:39:550
292169941,2cyclictest0-21swapper/020:55:110
292169941,2cyclictest0-21swapper/019:50:120
292169941,1cyclictest0-21swapper/022:30:120
292169940,4cyclictest3543-21hddtemp_smartct22:20:130
292169940,4cyclictest29821-21munin-run20:15:000
292169940,4cyclictest25506-21ssh23:36:190
292169940,4cyclictest24622-21diskmemload21:48:050
292169940,4cyclictest0-21swapper/023:10:160
292169940,4cyclictest0-21swapper/023:05:110
292169940,4cyclictest0-21swapper/022:10:110
292169940,4cyclictest0-21swapper/021:57:490
292169940,4cyclictest0-21swapper/021:20:130
292169940,4cyclictest0-21swapper/021:15:140
292169940,4cyclictest0-21swapper/020:30:110
292169940,4cyclictest0-21swapper/000:25:100
292169940,3cyclictest7690-21cat19:30:000
292169940,3cyclictest5299-21ssh00:31:350
292169940,3cyclictest32542-21apt-get20:20:140
292169940,3cyclictest31603-21ssh22:18:090
292169940,3cyclictest26056-21cat00:10:000
292169940,3cyclictest25982-21ssh22:47:470
292169940,3cyclictest25139-21ssh21:10:130
292169940,3cyclictest24806-21grep21:10:010
292169940,3cyclictest225-21systemd-journal20:05:000
292169940,3cyclictest20341-21/usr/sbin/munin21:40:120
292169940,3cyclictest17336-21sort20:55:000
292239932,1cyclictest725-21tune2fs20:20:163
292239930,3cyclictest61-21kswapd022:35:013
292239930,3cyclictest31846-21munin-node19:15:013
292239930,3cyclictest25131-21apt21:10:123
292239930,3cyclictest25072-21ssh22:15:103
292239930,3cyclictest22629-21apt-get20:00:133
292239930,3cyclictest20167-21tr21:00:123
292239930,3cyclictest13015-21diskstats20:45:143
292239930,3cyclictest0-21swapper/322:25:123
292239930,3cyclictest0-21swapper/321:50:053
292239930,3cyclictest0-21swapper/321:35:143
292239930,3cyclictest0-21swapper/320:40:093
292239930,3cyclictest0-21swapper/320:35:023
292239930,3cyclictest0-21swapper/319:50:023
292239930,3cyclictest0-21swapper/319:45:123
292239930,3cyclictest0-21swapper/319:30:133
292199932,1cyclictest714-21snmpd20:55:022
292199930,3cyclictest8269-21perl20:35:162
292199930,3cyclictest23213-21hddtemp_smartct21:05:142
292199930,3cyclictest22530-21ssh21:25:062
292199930,3cyclictest18026-21grep19:50:132
292199930,3cyclictest16268-21cut20:50:172
292199930,3cyclictest0-21swapper/221:35:062
292199930,3cyclictest0-21swapper/221:00:122
292199930,3cyclictest0-21swapper/220:30:112
292199930,3cyclictest0-21swapper/220:25:112
292199930,3cyclictest0-21swapper/220:15:112
292199930,3cyclictest0-21swapper/220:10:102
292199930,3cyclictest0-21swapper/220:05:122
292199930,3cyclictest0-21swapper/219:55:112
292199930,3cyclictest0-21swapper/219:40:152
292199930,3cyclictest0-21swapper/219:35:112
292199930,3cyclictest0-21swapper/219:20:132
292199930,2cyclictest11799-21ssh23:30:042
292179933,0cyclictest0-21swapper/123:39:511
292179933,0cyclictest0-21swapper/121:08:321
292179933,0cyclictest0-21swapper/121:00:221
292179933,0cyclictest0-21swapper/120:23:251
292179933,0cyclictest0-21swapper/100:13:191
292179931,1cyclictest95892chrt23:12:041
292179931,1cyclictest263682chrt22:31:291
292179931,1cyclictest0-21swapper/123:30:121
292179931,1cyclictest0-21swapper/122:45:151
292179931,1cyclictest0-21swapper/122:40:131
292179931,1cyclictest0-21swapper/122:35:131
292179931,1cyclictest0-21swapper/122:25:121
292179931,1cyclictest0-21swapper/122:05:121
292179931,1cyclictest0-21swapper/122:00:131
292179931,1cyclictest0-21swapper/121:40:111
292179931,1cyclictest0-21swapper/121:30:021
292179931,1cyclictest0-21swapper/120:45:121
292179931,1cyclictest0-21swapper/120:35:141
292179931,1cyclictest0-21swapper/120:15:111
292179931,1cyclictest0-21swapper/120:10:151
292179931,1cyclictest0-21swapper/120:05:121
292179931,1cyclictest0-21swapper/119:50:121
292179931,1cyclictest0-21swapper/119:35:111
292179931,1cyclictest0-21swapper/100:35:121
292179931,1cyclictest0-21swapper/100:20:111
292179931,1cyclictest0-21swapper/100:05:121
292179931,1cyclictest0-21swapper/100:00:111
292179930,2cyclictest27722-21taskset21:10:421
292179930,2cyclictest0-21swapper/119:25:011
292169932,1cyclictest31682-21ssh23:40:050
292169931,1cyclictest0-21swapper/019:10:100
292169930,3cyclictest7772-21apt-get19:30:130
292169930,3cyclictest29050-21ssh23:55:100
292169930,3cyclictest24622-21diskmemload21:30:020
292169930,3cyclictest17169-21ssh23:00:010
292169930,3cyclictest10012-21apt-get20:40:090
292169930,3cyclictest0-21swapper/023:15:110
292169930,3cyclictest0-21swapper/021:00:110
292169930,3cyclictest0-21swapper/020:45:120
292169930,3cyclictest0-21swapper/020:35:140
292169930,3cyclictest0-21swapper/020:00:130
292169930,3cyclictest0-21swapper/019:55:110
292169930,3cyclictest0-21swapper/019:45:110
292169930,3cyclictest0-21swapper/000:20:100
292169930,2cyclictest12314-21ssh22:25:010
292169930,2cyclictest10233-21/usr/sbin/munin19:35:110
14834230,0sleep10-21swapper/121:20:231
292179922,0cyclictest0-21swapper/120:50:021
292179922,0cyclictest0-21swapper/119:15:191
292179921,1cyclictest0-21swapper/120:30:111
292179921,1cyclictest0-21swapper/119:55:111
292179921,1cyclictest0-21swapper/100:30:121
292179920,1cyclictest7770-21chrt19:30:131
292179920,1cyclictest25004-21chrt22:15:071
292179920,1cyclictest24489-21chrt20:00:231
292179920,1cyclictest0-21swapper/122:55:321
292179920,1cyclictest0-21swapper/120:55:111
292179920,1cyclictest0-21swapper/120:25:111
292179920,1cyclictest0-21swapper/119:10:101
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional