You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-29 - 02:53
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot2.osadl.org (updated Sun Jun 29, 2025 00:44:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
154622208192,11sleep10-21swapper/119:09:241
154212186169,12sleep00-21swapper/019:08:500
151672185168,12sleep30-21swapper/319:05:363
153472126109,12sleep20-21swapper/219:07:572
62092460,0sleep20-21swapper/200:23:192
262222450,0sleep20-21swapper/223:27:512
179082440,0sleep30-21swapper/321:13:513
300052420,0sleep10-21swapper/122:24:591
139942420,0sleep00-21swapper/000:10:240
127162130,0sleep00-21swapper/022:32:000
266992100,0sleep10-21swapper/120:35:141
156839980,4cyclictest3926-21munin-run19:50:010
156839970,3cyclictest31425-21perf19:40:000
156839962,4cyclictest2646-21awk23:00:000
4514250,0sleep10-21swapper/123:33:221
156939955,0cyclictest0-21swapper/322:25:553
156939953,1cyclictest15509-21/usr/sbin/munin22:50:203
156939950,4cyclictest23311-21memory00:15:193
156879955,0cyclictest31-21ksoftirqd/223:39:592
156879955,0cyclictest0-21swapper/222:11:152
156879954,1cyclictest31-21ksoftirqd/221:50:132
156879952,3cyclictest31-21ksoftirqd/200:30:182
156849955,0cyclictest0-21swapper/123:49:261
156849951,3cyclictest0-21swapper/121:55:131
156849951,2cyclictest0-21swapper/123:00:181
156849951,2cyclictest0-21swapper/120:20:131
156849950,5cyclictest0-21swapper/121:35:441
156839955,0cyclictest0-21swapper/023:55:570
156839955,0cyclictest0-21swapper/020:40:160
156839950,5cyclictest0-21swapper/022:15:150
156839950,5cyclictest0-21swapper/021:33:480
32420240,0sleep00-21swapper/021:36:500
30769240,0sleep00-21swapper/022:40:490
2509240,0sleep00-21swapper/023:32:060
15816240,0sleep30-21swapper/322:00:443
156939944,0cyclictest0-21swapper/323:55:333
156939944,0cyclictest0-21swapper/321:56:243
156939944,0cyclictest0-21swapper/321:31:573
156939944,0cyclictest0-21swapper/300:37:553
156939943,1cyclictest24498-21ntp_states23:10:223
156939942,2cyclictest3378-21cat20:55:133
156939942,2cyclictest19477-21tr20:20:143
156939942,1cyclictest30997-21snmpd22:12:393
156939942,1cyclictest2604-21ssh23:48:463
156939941,2cyclictest0-21swapper/322:05:133
156939941,1cyclictest0-21swapper/320:40:143
156939940,4cyclictest11524-21users00:25:253
156939940,4cyclictest0-21swapper/323:50:173
156939940,4cyclictest0-21swapper/323:40:253
156939940,4cyclictest0-21swapper/323:36:443
156939940,4cyclictest0-21swapper/322:41:243
156939940,4cyclictest0-21swapper/321:45:213
156939940,4cyclictest0-21swapper/321:20:143
156939940,4cyclictest0-21swapper/320:30:223
156939940,4cyclictest0-21swapper/319:49:563
156939940,4cyclictest0-21swapper/319:15:013
156939940,4cyclictest0-21swapper/300:20:063
156939940,4cyclictest0-21swapper/300:05:123
156939940,3cyclictest29077-21sed19:35:013
156939940,3cyclictest27623-21date22:40:013
156939940,3cyclictest27292-21which21:50:223
156939940,3cyclictest23030-21ssh22:20:363
156939940,3cyclictest1578-21ssh23:31:283
156939940,3cyclictest13149-21sh23:05:163
156939940,3cyclictest10582-21/usr/sbin/munin20:00:203
156939940,3cyclictest10111-21rm23:20:023
156939940,3cyclictest0-21swapper/322:55:143
156939940,3cyclictest0-21swapper/322:45:253
156939940,3cyclictest0-21swapper/320:25:143
156939940,1cyclictest5596-21/usr/sbin/munin21:40:153
156939940,1cyclictest15797-21/usr/sbin/munin19:10:123
156939940,1cyclictest14257-21/usr/sbin/munin20:10:213
156939940,1cyclictest14195-21grep00:10:253
156939940,1cyclictest0-21swapper/323:25:133
156939940,1cyclictest0-21swapper/319:55:153
156879944,0cyclictest31-21ksoftirqd/221:25:172
156879944,0cyclictest31-21ksoftirqd/219:45:012
156879944,0cyclictest0-21swapper/223:55:182
156879943,1cyclictest31-21ksoftirqd/223:50:202
156879943,1cyclictest14722-21ssh22:00:252
156879942,1cyclictest18032-21ssh22:35:132
156879942,1cyclictest17678-21cat22:35:002
156879941,3cyclictest29244-21ssh23:13:032
156879941,2cyclictest0-21swapper/220:55:102
156879941,2cyclictest0-21swapper/219:25:152
156879941,1cyclictest0-21swapper/222:25:162
156879940,4cyclictest9440-21ssh23:03:022
156879940,4cyclictest28045-21ssh22:40:142
156879940,4cyclictest0-21swapper/223:47:062
156879940,4cyclictest0-21swapper/223:44:142
156879940,4cyclictest0-21swapper/223:30:132
156879940,4cyclictest0-21swapper/223:05:172
156879940,4cyclictest0-21swapper/222:55:152
156879940,4cyclictest0-21swapper/221:55:142
156879940,4cyclictest0-21swapper/221:46:412
156879940,4cyclictest0-21swapper/221:32:212
156879940,4cyclictest0-21swapper/221:15:122
156879940,4cyclictest0-21swapper/220:05:172
156879940,4cyclictest0-21swapper/219:50:122
156879940,4cyclictest0-21swapper/200:37:062
156879940,4cyclictest0-21swapper/200:27:382
156879940,4cyclictest0-21swapper/200:15:142
156879940,4cyclictest0-21swapper/200:10:162
156879940,4cyclictest0-21swapper/200:05:112
156879940,4cyclictest0-21swapper/200:01:252
156879940,3cyclictest8727-21ssh21:41:012
156879940,3cyclictest529-21ssh23:15:162
156879940,3cyclictest30052-21munin-run22:25:012
156879940,3cyclictest28715-21ssh22:07:362
156879940,3cyclictest27713-21sed19:30:152
156879940,3cyclictest18581-21memory19:15:002
156879940,3cyclictest0-21swapper/221:20:142
156879940,3cyclictest0-21swapper/221:05:152
156849941,2cyclictest0-21swapper/123:25:141
156849941,2cyclictest0-21swapper/122:40:161
156849941,2cyclictest0-21swapper/122:00:161
156849941,2cyclictest0-21swapper/121:30:161
156849941,2cyclictest0-21swapper/121:25:131
156849941,2cyclictest0-21swapper/121:20:141
156849941,2cyclictest0-21swapper/121:00:111
156849941,2cyclictest0-21swapper/120:45:131
156849941,2cyclictest0-21swapper/119:50:131
156849941,2cyclictest0-21swapper/119:35:151
156849941,2cyclictest0-21swapper/119:25:141
156849941,2cyclictest0-21swapper/100:10:151
156849941,2cyclictest0-21swapper/100:00:131
156849941,1cyclictest0-21swapper/100:30:131
156849940,4cyclictest0-21swapper/123:50:101
156849940,4cyclictest0-21swapper/123:08:421
156849940,4cyclictest0-21swapper/122:59:061
156849940,4cyclictest0-21swapper/121:50:071
156849940,0cyclictest0-21swapper/122:32:021
156839944,0cyclictest0-21swapper/023:01:250
156839944,0cyclictest0-21swapper/022:48:540
156839944,0cyclictest0-21swapper/022:25:360
156839944,0cyclictest0-21swapper/019:28:060
156839943,1cyclictest13826-21uniq22:00:170
156839942,1cyclictest18042-21munin-run19:15:000
156839940,4cyclictest2712-21diskmemload00:37:500
156839940,4cyclictest22971-21ntp_states20:25:200
156839940,4cyclictest18023-21apt-get22:35:140
156839940,4cyclictest1403-21apt-get20:50:000
156839940,4cyclictest0-21swapper/023:51:510
156839940,4cyclictest0-21swapper/023:41:140
156839940,4cyclictest0-21swapper/023:27:450
156839940,4cyclictest0-21swapper/023:15:160
156839940,4cyclictest0-21swapper/023:14:360
156839940,4cyclictest0-21swapper/021:40:140
156839940,4cyclictest0-21swapper/021:14:010
156839940,4cyclictest0-21swapper/020:17:250
156839940,4cyclictest0-21swapper/020:14:030
156839940,4cyclictest0-21swapper/019:30:080
156839940,4cyclictest0-21swapper/000:27:160
156839940,3cyclictest9672-21diskmemload23:05:400
156839940,3cyclictest31959-21/usr/sbin/munin00:20:150
156839940,3cyclictest28204-21df23:45:140
156839940,3cyclictest21379-21ssh00:30:220
156839940,3cyclictest17241-21cut23:40:000
156839940,3cyclictest17073-21ntp_states21:45:210
156839940,3cyclictest16430-21sh23:22:540
156839940,3cyclictest0-21swapper/000:00:470
156839940,1cyclictest19997-21grep20:20:160
156839940,1cyclictest14021-21latency_hist20:10:000
156839940,0cyclictest0-21swapper/022:50:200
156839940,0cyclictest0-21swapper/022:23:250
156839940,0cyclictest0-21swapper/021:20:170
10135240,3sleep20-21swapper/221:10:112
156939932,1cyclictest31635-21/usr/sbin/munin19:40:133
156939931,1cyclictest0-21swapper/300:30:133
156939930,3cyclictest7887-21sh22:30:043
156939930,3cyclictest24694-21sh00:00:063
156939930,3cyclictest21132-21grep21:15:163
156939930,3cyclictest1772-21dpkg20:50:003
156939930,3cyclictest0-21swapper/323:15:033
156939930,3cyclictest0-21swapper/322:15:033
156939930,3cyclictest0-21swapper/321:35:143
156939930,3cyclictest0-21swapper/321:25:073
156939930,3cyclictest0-21swapper/321:05:193
156939930,3cyclictest0-21swapper/321:00:073
156939930,3cyclictest0-21swapper/320:45:133
156939930,3cyclictest0-21swapper/320:35:133
156939930,3cyclictest0-21swapper/320:15:143
156939930,3cyclictest0-21swapper/320:05:153
156939930,3cyclictest0-21swapper/319:50:023
156939930,3cyclictest0-21swapper/319:35:143
156939930,3cyclictest0-21swapper/319:25:023
156939930,3cyclictest0-21swapper/319:20:143
156939930,2cyclictest3250-21ssh23:00:163
156879931,1cyclictest0-21swapper/220:35:142
156879931,1cyclictest0-21swapper/220:20:132
156879930,3cyclictest30421-21ntpq19:35:222
156879930,3cyclictest27808-21ssh21:35:052
156879930,3cyclictest10310-21ssh22:15:022
156879930,3cyclictest0-21swapper/223:20:132
156879930,3cyclictest0-21swapper/222:50:062
156879930,3cyclictest0-21swapper/222:45:062
156879930,3cyclictest0-21swapper/221:00:122
156879930,3cyclictest0-21swapper/220:50:062
156879930,3cyclictest0-21swapper/220:45:122
156879930,3cyclictest0-21swapper/220:40:072
156879930,3cyclictest0-21swapper/220:30:122
156879930,3cyclictest0-21swapper/220:25:052
156879930,3cyclictest0-21swapper/220:15:112
156879930,3cyclictest0-21swapper/220:10:132
156879930,3cyclictest0-21swapper/220:00:072
156879930,3cyclictest0-21swapper/219:55:062
156879930,3cyclictest0-21swapper/219:45:122
156879930,3cyclictest0-21swapper/219:20:132
156879930,3cyclictest0-21swapper/219:10:122
156849933,0cyclictest0-21swapper/121:49:231
156849933,0cyclictest0-21swapper/121:17:151
156849932,1cyclictest0-21swapper/123:40:171
156849931,2cyclictest0-21swapper/122:25:161
156849931,2cyclictest0-21swapper/120:40:151
156849931,2cyclictest0-21swapper/119:40:131
156849931,2cyclictest0-21swapper/119:30:101
156849931,1cyclictest0-21swapper/123:55:171
156849931,1cyclictest0-21swapper/123:35:171
156849931,1cyclictest0-21swapper/123:20:171
156849931,1cyclictest0-21swapper/122:50:161
156849931,1cyclictest0-21swapper/122:45:171
156849931,1cyclictest0-21swapper/122:35:141
156849931,1cyclictest0-21swapper/122:15:151
156849931,1cyclictest0-21swapper/122:05:141
156849931,1cyclictest0-21swapper/121:40:141
156849931,1cyclictest0-21swapper/121:10:141
156849931,1cyclictest0-21swapper/121:05:151
156849931,1cyclictest0-21swapper/120:55:091
156849931,1cyclictest0-21swapper/120:10:131
156849931,1cyclictest0-21swapper/120:00:131
156849931,1cyclictest0-21swapper/119:55:141
156849931,1cyclictest0-21swapper/119:45:131
156849931,1cyclictest0-21swapper/119:15:021
156849931,1cyclictest0-21swapper/119:10:121
156849931,1cyclictest0-21swapper/100:35:121
156849931,1cyclictest0-21swapper/100:25:151
156849931,1cyclictest0-21swapper/100:20:141
156849931,1cyclictest0-21swapper/100:15:141
156849931,1cyclictest0-21swapper/100:05:111
156849930,3cyclictest0-21swapper/120:15:061
156849930,2cyclictest0-21swapper/120:25:131
156839932,1cyclictest7694-21grep21:05:150
156839932,0cyclictest0-21swapper/000:15:130
156839931,1cyclictest18025-21cron19:15:010
156839930,3cyclictest6448-21which21:00:190
156839930,3cyclictest4678-21diskstats19:50:150
156839930,3cyclictest0-21swapper/022:05:140
156839930,3cyclictest0-21swapper/021:55:070
156839930,3cyclictest0-21swapper/021:50:030
156839930,3cyclictest0-21swapper/021:25:140
156839930,3cyclictest0-21swapper/021:15:020
156839930,3cyclictest0-21swapper/020:55:100
156839930,3cyclictest0-21swapper/020:50:010
156839930,3cyclictest0-21swapper/020:35:130
156839930,3cyclictest0-21swapper/020:30:120
156839930,3cyclictest0-21swapper/020:00:020
156839930,3cyclictest0-21swapper/019:40:050
156839930,3cyclictest0-21swapper/019:21:110
156839930,2cyclictest9672-21diskmemload00:05:030
156839930,2cyclictest7006-21cstates19:55:160
156839930,2cyclictest0-21swapper/022:10:020
156849921,1cyclictest117882chrt20:05:161
156849921,1cyclictest0-21swapper/123:15:151
156849921,1cyclictest0-21swapper/123:10:091
156849920,1cyclictest556-21chrt22:10:111
156849920,1cyclictest0-21swapper/120:50:001
156849920,1cyclictest0-21swapper/120:30:121
156849920,1cyclictest0-21swapper/119:20:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional