You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 13:02
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot2.osadl.org (updated Fri Nov 21, 2025 00:44:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
311522198182,11sleep10-21swapper/119:09:331
289662186170,11sleep30-21swapper/319:05:043
309572182164,13sleep00-21swapper/019:06:590
289652173154,14sleep20-21swapper/219:05:032
102742440,0sleep30-21swapper/300:32:163
212622430,0sleep20-21swapper/223:32:352
48052420,0sleep20-21swapper/219:20:002
3137299113,4cyclictest7718-21munin-run22:05:020
313729993,3cyclictest16897-21munin-run20:49:590
13534290,0sleep10-21swapper/121:19:181
313729982,3cyclictest22952-21apt-get22:45:010
313729962,4cyclictest11961-21sendmail-msp20:40:000
313729960,3cyclictest6993-21munin-run00:15:010
313869955,0cyclictest2055-21kworker/3:000:20:163
313869953,1cyclictest22849-21/usr/sbin/munin23:50:183
313869950,5cyclictest0-21swapper/322:29:553
313839954,1cyclictest23433-21ssh00:22:442
313839950,5cyclictest0-21swapper/222:19:132
313839950,5cyclictest0-21swapper/200:34:052
313839950,4cyclictest32722-21ssh21:45:042
313839950,0cyclictest26786-21diskmemload22:44:142
313839950,0cyclictest0-21swapper/219:50:212
313779951,3cyclictest0-21swapper/119:40:081
313729955,0cyclictest0-21swapper/000:38:010
313729954,1cyclictest13706-21ssh21:50:590
313729953,1cyclictest13002-21munin-run22:40:010
313729952,1cyclictest0-21swapper/023:05:140
313729951,4cyclictest19607-21munin-run19:50:000
313729950,5cyclictest0-21swapper/023:58:360
313729950,5cyclictest0-21swapper/023:34:180
313729950,5cyclictest0-21swapper/022:55:140
313729950,4cyclictest31665-21apt-get19:10:020
313729950,4cyclictest31665-21apt-get19:10:020
313729950,1cyclictest0-21swapper/023:05:010
14971250,0sleep00-21swapper/022:24:500
10847250,0sleep10-21swapper/123:43:591
313869944,0cyclictest2055-21kworker/3:022:39:303
313869944,0cyclictest0-21swapper/322:23:033
313869944,0cyclictest0-21swapper/322:16:133
313869944,0cyclictest0-21swapper/319:15:133
313869944,0cyclictest0-21swapper/300:05:293
313869943,1cyclictest19593-21/usr/sbin/munin20:55:193
313869942,1cyclictest15906-21perl00:35:153
313869941,3cyclictest1161-21tail23:55:163
313869941,2cyclictest711-21runrttasks22:53:043
313869941,1cyclictest0-21swapper/319:50:113
313869940,4cyclictest0-21swapper/323:34:343
313869940,4cyclictest0-21swapper/323:15:113
313869940,4cyclictest0-21swapper/323:00:123
313869940,4cyclictest0-21swapper/322:30:133
313869940,4cyclictest0-21swapper/322:12:553
313869940,4cyclictest0-21swapper/322:05:123
313869940,4cyclictest0-21swapper/322:00:133
313869940,4cyclictest0-21swapper/321:26:473
313869940,4cyclictest0-21swapper/321:17:563
313869940,4cyclictest0-21swapper/320:42:573
313869940,4cyclictest0-21swapper/320:10:153
313869940,4cyclictest0-21swapper/320:05:123
313869940,4cyclictest0-21swapper/300:15:023
313869940,3cyclictest8912-21expr23:10:123
313869940,3cyclictest7178-21cat00:15:003
313869940,3cyclictest5383-21sh21:30:383
313869940,3cyclictest4505-21ssh23:24:323
313869940,3cyclictest32404-21sh23:38:213
313869940,3cyclictest31481-21ssh21:11:413
313869940,3cyclictest3120-21sh23:40:123
313869940,3cyclictest23421-21sh22:45:033
313869940,3cyclictest17833-21expr19:45:133
313869940,3cyclictest15642-21ssh00:02:293
313869940,3cyclictest15169-21df20:45:163
313869940,3cyclictest12932-21/usr/sbin/munin21:35:133
313869940,3cyclictest11979-21/usr/sbin/munin21:50:183
313869940,3cyclictest1143-21ssh23:05:433
313869940,1cyclictest7187-21ntp_states23:25:183
313869940,1cyclictest4989-21/usr/sbin/munin19:20:143
313869940,1cyclictest22955-21/usr/sbin/munin21:40:213
313869940,1cyclictest15269-21/usr/sbin/munin21:20:223
313869940,1cyclictest1275-21fschecks_time21:45:153
313869940,1cyclictest12495-21ntp_states22:55:203
313869940,1cyclictest0-21swapper/300:25:003
313839944,0cyclictest0-21swapper/223:35:012
313839944,0cyclictest0-21swapper/222:58:372
313839944,0cyclictest0-21swapper/222:33:362
313839944,0cyclictest0-21swapper/221:37:352
313839944,0cyclictest0-21swapper/220:00:142
313839942,1cyclictest495-21nscd00:15:002
313839941,1cyclictest0-21swapper/219:55:132
313839940,4cyclictest20647-21/usr/sbin/munin21:55:212
313839940,4cyclictest13356-21ssh00:01:022
313839940,4cyclictest0-21swapper/223:53:112
313839940,4cyclictest0-21swapper/223:40:132
313839940,4cyclictest0-21swapper/223:15:112
313839940,4cyclictest0-21swapper/223:11:132
313839940,4cyclictest0-21swapper/223:05:152
313839940,4cyclictest0-21swapper/223:04:172
313839940,4cyclictest0-21swapper/222:35:022
313839940,4cyclictest0-21swapper/222:25:132
313839940,4cyclictest0-21swapper/222:14:322
313839940,4cyclictest0-21swapper/222:07:562
313839940,4cyclictest0-21swapper/221:40:132
313839940,4cyclictest0-21swapper/221:20:152
313839940,4cyclictest0-21swapper/221:10:122
313839940,4cyclictest0-21swapper/220:34:342
313839940,3cyclictest4517-21latency_hist20:25:012
313839940,3cyclictest32755-21ssh23:55:132
313839940,3cyclictest30846-21ssh00:26:012
313839940,3cyclictest29650-21cat20:10:002
313839940,3cyclictest25078-21df_inode21:05:142
313839940,3cyclictest15675-21ssh00:35:142
313839940,3cyclictest14048-21irqstats23:45:142
313839940,2cyclictest0-21swapper/222:48:392
313839940,2cyclictest0-21swapper/200:13:462
313839940,0cyclictest0-21swapper/222:54:502
313779944,0cyclictest0-21swapper/122:49:201
313779944,0cyclictest0-21swapper/122:33:191
313779942,1cyclictest0-21swapper/100:20:121
313779941,2cyclictest0-21swapper/123:50:121
313779941,2cyclictest0-21swapper/123:30:111
313779941,2cyclictest0-21swapper/123:05:141
313779941,2cyclictest0-21swapper/122:05:111
313779941,2cyclictest0-21swapper/121:50:141
313779941,2cyclictest0-21swapper/121:45:131
313779941,2cyclictest0-21swapper/120:25:141
313779941,2cyclictest0-21swapper/120:05:111
313779941,2cyclictest0-21swapper/119:30:131
313779941,2cyclictest0-21swapper/119:20:101
313779941,2cyclictest0-21swapper/119:15:121
313779941,2cyclictest0-21swapper/100:05:111
313779940,4cyclictest0-21swapper/123:35:011
313779940,4cyclictest0-21swapper/123:25:071
313779940,4cyclictest0-21swapper/122:17:201
313779940,3cyclictest0-21swapper/122:50:131
313779940,3cyclictest0-21swapper/119:25:121
313779940,2cyclictest0-21swapper/121:35:131
313779940,2cyclictest0-21swapper/120:35:111
313779940,0cyclictest0-21swapper/100:34:241
313729944,0cyclictest0-21swapper/000:30:270
313729944,0cyclictest0-21swapper/000:26:400
313729944,0cyclictest0-21swapper/000:02:220
313729942,2cyclictest1731-21kworker/0:122:30:580
313729942,2cyclictest1731-21kworker/0:122:19:570
313729942,2cyclictest1731-21kworker/0:120:55:240
313729942,1cyclictest84552sleep023:10:110
313729942,1cyclictest714-21snmpd21:26:010
313729941,3cyclictest4742-21grep21:30:230
313729941,3cyclictest28106-21ssh21:42:170
313729941,2cyclictest0-21swapper/021:00:130
313729941,2cyclictest0-21swapper/019:35:140
313729941,1cyclictest0-21swapper/021:10:120
313729940,4cyclictest32048-21ssh23:54:550
313729940,4cyclictest2285-21apt-get20:20:120
313729940,4cyclictest13450-21ntp_states20:40:210
313729940,4cyclictest0-21swapper/023:24:290
313729940,4cyclictest0-21swapper/022:45:120
313729940,4cyclictest0-21swapper/022:05:120
313729940,4cyclictest0-21swapper/021:55:140
313729940,4cyclictest0-21swapper/021:45:130
313729940,4cyclictest0-21swapper/021:35:140
313729940,4cyclictest0-21swapper/021:21:570
313729940,4cyclictest0-21swapper/021:15:120
313729940,4cyclictest0-21swapper/020:25:210
313729940,4cyclictest0-21swapper/000:23:010
313729940,4cyclictest0-21swapper/000:15:140
313729940,3cyclictest32729-21df_inode20:15:150
313729940,3cyclictest3107-21apt23:40:120
313729940,3cyclictest26828-21cut21:10:020
313729940,3cyclictest20070-21python22:10:250
313729940,3cyclictest18906-21df23:15:120
313729940,3cyclictest15809-21/usr/sbin/munin22:25:220
28518240,0sleep00-21swapper/000:09:240
20680240,0sleep10-21swapper/122:27:081
17140240,0sleep017138-21perf19:45:010
313869932,1cyclictest24861-21grep21:05:143
313869931,1cyclictest0-21swapper/320:00:133
313869931,1cyclictest0-21swapper/319:40:093
313869930,3cyclictest31213-21apt-get19:10:023
313869930,3cyclictest31213-21apt-get19:10:013
313869930,3cyclictest2285-21apt-get20:20:123
313869930,3cyclictest22076-21/usr/sbin/munin21:00:173
313869930,3cyclictest20849-21grep21:55:143
313869930,3cyclictest14601-21modprobe22:40:163
313869930,3cyclictest0-21swapper/323:45:103
313869930,3cyclictest0-21swapper/320:50:123
313869930,3cyclictest0-21swapper/320:35:063
313869930,3cyclictest0-21swapper/320:30:173
313869930,3cyclictest0-21swapper/320:25:143
313869930,3cyclictest0-21swapper/320:15:143
313869930,3cyclictest0-21swapper/319:55:123
313869930,3cyclictest0-21swapper/319:35:053
313869930,3cyclictest0-21swapper/319:30:123
313869930,3cyclictest0-21swapper/319:25:123
313839933,0cyclictest0-21swapper/220:55:142
313839932,1cyclictest714-21snmpd20:45:042
313839932,1cyclictest714-21snmpd19:35:572
313839932,1cyclictest2633-21ssh21:30:032
313839931,2cyclictest26786-21diskmemload21:50:102
313839931,1cyclictest4989-21/usr/sbin/munin19:20:112
313839931,1cyclictest0-21swapper/220:50:132
313839931,1cyclictest0-21swapper/220:25:142
313839931,1cyclictest0-21swapper/219:25:132
313839930,3cyclictest5850-21apt22:20:142
313839930,3cyclictest32440-21cut20:15:152
313839930,3cyclictest28383-21ssh23:20:142
313839930,3cyclictest25072-21rm21:25:052
313839930,3cyclictest12939-21grep20:40:162
313839930,3cyclictest0-21swapper/223:25:062
313839930,3cyclictest0-21swapper/222:00:022
313839930,3cyclictest0-21swapper/221:15:022
313839930,3cyclictest0-21swapper/221:00:112
313839930,3cyclictest0-21swapper/220:35:112
313839930,3cyclictest0-21swapper/220:10:142
313839930,3cyclictest0-21swapper/219:45:052
313839930,3cyclictest0-21swapper/219:40:082
313839930,3cyclictest0-21swapper/219:30:122
313839930,3cyclictest0-21swapper/219:10:012
313839930,3cyclictest0-21swapper/219:10:012
313839930,3cyclictest0-21swapper/200:05:022
313779933,0cyclictest0-21swapper/122:14:161
313779933,0cyclictest0-21swapper/121:40:041
313779933,0cyclictest0-21swapper/119:10:031
313779933,0cyclictest0-21swapper/119:10:021
313779931,2cyclictest0-21swapper/121:05:121
313779931,2cyclictest0-21swapper/120:55:181
313779931,2cyclictest0-21swapper/120:45:141
313779931,1cyclictest0-21swapper/123:55:121
313779931,1cyclictest0-21swapper/123:45:111
313779931,1cyclictest0-21swapper/122:35:121
313779931,1cyclictest0-21swapper/122:20:151
313779931,1cyclictest0-21swapper/121:55:141
313779931,1cyclictest0-21swapper/121:20:151
313779931,1cyclictest0-21swapper/121:10:121
313779931,1cyclictest0-21swapper/120:50:121
313779931,1cyclictest0-21swapper/120:30:181
313779931,1cyclictest0-21swapper/120:20:121
313779931,1cyclictest0-21swapper/120:10:151
313779931,1cyclictest0-21swapper/119:55:131
313779931,1cyclictest0-21swapper/119:50:111
313779931,1cyclictest0-21swapper/100:15:131
313779931,1cyclictest0-21swapper/100:10:141
313779931,1cyclictest0-21swapper/100:00:121
313779930,2cyclictest0-21swapper/123:00:111
313779930,2cyclictest0-21swapper/121:30:141
313779930,2cyclictest0-21swapper/121:00:121
313729931,1cyclictest0-21swapper/020:30:170
313729931,1cyclictest0-21swapper/020:05:120
313729931,1cyclictest0-21swapper/019:30:120
313729930,3cyclictest5521-21diskstats19:20:120
313729930,3cyclictest30719-21irqstats20:10:190
313729930,3cyclictest26786-21diskmemload23:35:080
313729930,3cyclictest0-21swapper/023:45:110
313729930,3cyclictest0-21swapper/022:50:080
313729930,3cyclictest0-21swapper/020:50:120
313729930,3cyclictest0-21swapper/020:00:130
313729930,3cyclictest0-21swapper/019:55:120
313729930,3cyclictest0-21swapper/019:50:040
313729930,3cyclictest0-21swapper/019:25:120
313729930,3cyclictest0-21swapper/019:15:130
313729930,2cyclictest5481-21ssh23:25:010
313779921,1cyclictest0-21swapper/123:15:121
313779921,1cyclictest0-21swapper/119:35:131
313779920,1cyclictest12187-21chrt20:40:141
313779920,1cyclictest0-21swapper/123:20:141
313779920,1cyclictest0-21swapper/123:10:121
313779920,1cyclictest0-21swapper/122:55:081
313779920,1cyclictest0-21swapper/122:40:051
313779920,1cyclictest0-21swapper/122:00:121
313779920,1cyclictest0-21swapper/121:25:121
313779920,1cyclictest0-21swapper/120:15:131
313779920,1cyclictest0-21swapper/120:00:131
313779920,1cyclictest0-21swapper/119:45:111
313779920,1cyclictest0-21swapper/100:35:131
313779920,1cyclictest0-21swapper/100:25:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional