You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 12:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot2.osadl.org (updated Thu Dec 04, 2025 00:44:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
162282225208,12sleep10-21swapper/119:08:541
162722190174,12sleep00-21swapper/019:09:290
161162182166,12sleep30-21swapper/319:07:273
162392176160,11sleep20-21swapper/219:09:032
87372460,0sleep10-21swapper/100:21:351
14932410,0sleep20-21swapper/219:40:162
119642330,0sleep00-21swapper/023:02:080
199902320,0sleep219989-21ssh22:50:142
165689960,6cyclictest0-21swapper/223:50:182
165689960,1cyclictest8950-21ssh22:28:322
165669961,3cyclictest0-21swapper/123:10:131
165669961,3cyclictest0-21swapper/119:40:111
165669961,3cyclictest0-21swapper/100:00:131
165669961,2cyclictest0-21swapper/121:05:121
165659964,2cyclictest0-21swapper/021:16:290
165729955,0cyclictest56750irq/130-eth021:56:173
165729955,0cyclictest56750irq/130-eth021:56:173
165729954,1cyclictest0-21swapper/300:02:093
165729951,2cyclictest0-21swapper/320:15:113
165729950,5cyclictest0-21swapper/322:18:253
165729950,5cyclictest0-21swapper/300:30:573
165729950,5cyclictest0-21swapper/300:16:343
165729950,4cyclictest28511-21users22:05:243
165689955,0cyclictest0-21swapper/221:51:512
165689950,5cyclictest0-21swapper/223:35:012
165689950,4cyclictest11834-21apt-get22:30:132
165669952,2cyclictest0-21swapper/123:20:121
165669951,3cyclictest0-21swapper/123:45:101
165669951,3cyclictest0-21swapper/120:40:121
165669951,3cyclictest0-21swapper/120:10:111
165669951,3cyclictest0-21swapper/119:45:111
165669951,2cyclictest0-21swapper/123:50:121
165669951,2cyclictest0-21swapper/123:00:131
165669951,2cyclictest0-21swapper/122:50:111
165669951,2cyclictest0-21swapper/122:25:121
165669951,2cyclictest0-21swapper/121:40:121
165669951,2cyclictest0-21swapper/120:15:101
165669951,2cyclictest0-21swapper/100:25:021
165669951,1cyclictest0-21swapper/122:15:151
165659955,0cyclictest0-21swapper/023:48:150
165659950,5cyclictest12357-21diskmemload23:20:020
165659950,5cyclictest0-21swapper/021:35:330
165659950,4cyclictest29459-21ssh22:55:120
165659950,4cyclictest0-21swapper/019:35:120
6986240,0sleep1201rcuc/122:10:571
165729944,0cyclictest0-21swapper/322:58:523
165729944,0cyclictest0-21swapper/321:26:363
165729944,0cyclictest0-21swapper/320:54:523
165729944,0cyclictest0-21swapper/319:37:543
165729943,1cyclictest32715-21sed22:40:143
165729943,1cyclictest22690-21/usr/sbin/munin20:25:123
165729942,2cyclictest31161-21wget21:35:133
165729942,1cyclictest13577-21swap22:30:193
165729942,1cyclictest11303-21ssh23:18:173
165729942,0cyclictest0-21swapper/319:13:583
165729941,3cyclictest517-21nscd23:25:133
165729941,2cyclictest0-21swapper/323:45:113
165729941,2cyclictest0-21swapper/322:25:133
165729941,2cyclictest0-21swapper/320:10:123
165729941,1cyclictest0-21swapper/323:00:123
165729941,1cyclictest0-21swapper/321:05:123
165729940,4cyclictest22553-21/usr/sbin/munin19:20:133
165729940,4cyclictest0-21swapper/323:55:113
165729940,4cyclictest0-21swapper/323:50:133
165729940,4cyclictest0-21swapper/323:40:153
165729940,4cyclictest0-21swapper/323:35:533
165729940,4cyclictest0-21swapper/323:10:123
165729940,4cyclictest0-21swapper/322:50:123
165729940,4cyclictest0-21swapper/322:45:123
165729940,4cyclictest0-21swapper/321:50:133
165729940,4cyclictest0-21swapper/321:45:133
165729940,4cyclictest0-21swapper/321:34:243
165729940,4cyclictest0-21swapper/321:10:123
165729940,4cyclictest0-21swapper/319:59:523
165729940,4cyclictest0-21swapper/319:45:113
165729940,4cyclictest0-21swapper/319:15:133
165729940,3cyclictest27639-21apt-get20:35:133
165729940,3cyclictest26539-21ssh22:36:583
165729940,3cyclictest23965-21apt-get22:20:023
165729940,3cyclictest17062-21ssh23:05:123
165729940,3cyclictest16919-21diskstats22:00:133
165729940,3cyclictest0-21swapper/321:20:133
165729940,3cyclictest0-21swapper/320:05:003
165729940,2cyclictest8042-21ssh22:11:363
165729940,2cyclictest0-21swapper/323:20:133
165729940,2cyclictest0-21swapper/300:20:113
165729940,1cyclictest32639-21/usr/sbin/munin19:40:163
165729940,1cyclictest25111-21/usr/sbin/munin19:25:163
165729940,1cyclictest23074-21grep21:15:133
165689944,0cyclictest0-21swapper/221:35:512
165689943,1cyclictest9861-21dump-pmu-power20:00:002
165689942,2cyclictest808-21snmpd20:08:572
165689942,1cyclictest808-21snmpd19:15:542
165689942,1cyclictest808-21snmpd00:34:322
165689941,1cyclictest0-21swapper/219:50:112
165689940,4cyclictest7648-21apt-get21:00:112
165689940,4cyclictest14584-21awk21:10:202
165689940,4cyclictest1296-21rm22:56:452
165689940,4cyclictest0-21swapper/223:55:532
165689940,4cyclictest0-21swapper/223:45:112
165689940,4cyclictest0-21swapper/223:20:132
165689940,4cyclictest0-21swapper/223:15:112
165689940,4cyclictest0-21swapper/222:35:122
165689940,4cyclictest0-21swapper/222:10:142
165689940,4cyclictest0-21swapper/221:55:112
165689940,4cyclictest0-21swapper/221:55:112
165689940,4cyclictest0-21swapper/221:45:042
165689940,4cyclictest0-21swapper/221:30:082
165689940,4cyclictest0-21swapper/221:25:122
165689940,4cyclictest0-21swapper/221:20:132
165689940,4cyclictest0-21swapper/221:19:202
165689940,4cyclictest0-21swapper/221:05:122
165689940,4cyclictest0-21swapper/220:20:152
165689940,4cyclictest0-21swapper/219:30:122
165689940,4cyclictest0-21swapper/200:35:142
165689940,4cyclictest0-21swapper/200:15:152
165689940,4cyclictest0-21swapper/200:10:132
165689940,3cyclictest7266-21cat23:00:142
165689940,3cyclictest32596-21ls19:40:002
165689940,3cyclictest27042-21ssh23:10:122
165689940,3cyclictest27009-21users20:30:202
165689940,3cyclictest25085-21sed20:30:012
165689940,3cyclictest17626-21df_inode23:05:132
165689940,3cyclictest16901-21df22:00:122
165689940,3cyclictest16600-21taskset19:10:032
165689940,3cyclictest14814-21ssh00:08:542
165689940,3cyclictest14073-21ssh22:15:152
165689940,1cyclictest2305-21cat23:30:122
165689940,0cyclictest0-21swapper/222:40:442
165689940,0cyclictest0-21swapper/200:25:572
165689940,0cyclictest0-21swapper/200:00:552
165669942,1cyclictest0-21swapper/123:40:131
165669941,3cyclictest0-21swapper/119:35:121
165669941,2cyclictest0-21swapper/123:35:151
165669941,2cyclictest0-21swapper/123:25:111
165669941,2cyclictest0-21swapper/122:35:131
165669941,2cyclictest0-21swapper/122:20:011
165669941,2cyclictest0-21swapper/121:30:121
165669941,2cyclictest0-21swapper/121:20:131
165669941,2cyclictest0-21swapper/121:15:131
165669941,2cyclictest0-21swapper/120:59:521
165669941,2cyclictest0-21swapper/120:45:101
165669941,2cyclictest0-21swapper/120:30:121
165669941,2cyclictest0-21swapper/120:25:101
165669941,2cyclictest0-21swapper/120:20:151
165669941,2cyclictest0-21swapper/119:55:131
165669941,2cyclictest0-21swapper/119:15:121
165669941,2cyclictest0-21swapper/100:35:151
165669941,1cyclictest0-21swapper/123:05:131
165669941,1cyclictest0-21swapper/119:30:111
165669940,4cyclictest0-21swapper/120:00:191
165669940,3cyclictest0-21swapper/121:10:121
165669940,2cyclictest0-21swapper/122:55:121
165669940,2cyclictest0-21swapper/122:45:101
165669940,2cyclictest0-21swapper/121:50:121
165669940,0cyclictest0-21swapper/123:31:101
165659944,0cyclictest0-21swapper/023:10:520
165659941,3cyclictest21870-21apt22:35:120
165659940,4cyclictest28916-21perf22:55:010
165659940,4cyclictest21294-21ntp_states20:20:200
165659940,4cyclictest0-21swapper/023:40:140
165659940,4cyclictest0-21swapper/023:30:100
165659940,4cyclictest0-21swapper/023:05:120
165659940,4cyclictest0-21swapper/022:45:110
165659940,4cyclictest0-21swapper/022:30:120
165659940,4cyclictest0-21swapper/022:25:120
165659940,4cyclictest0-21swapper/021:40:220
165659940,4cyclictest0-21swapper/021:30:120
165659940,4cyclictest0-21swapper/021:28:420
165659940,4cyclictest0-21swapper/019:55:180
165659940,4cyclictest0-21swapper/019:50:110
165659940,4cyclictest0-21swapper/000:10:140
165659940,4cyclictest0-21swapper/000:05:150
165659940,3cyclictest7614-21date21:00:010
165659940,3cyclictest6398-21python00:20:190
165659940,3cyclictest5090-21ssh00:04:040
165659940,3cyclictest5014-21cut20:55:000
165659940,3cyclictest4581-21apt23:15:120
165659940,3cyclictest31366-21ntpq20:40:180
165659940,3cyclictest30161-21ntp_states21:50:190
165659940,3cyclictest28910-21ntp_states20:35:190
165659940,3cyclictest28201-21df_inode19:30:130
165659940,3cyclictest2791-21users21:20:220
165659940,3cyclictest27523-21irqstats22:05:180
165659940,3cyclictest27147-21apt-get00:15:150
165659940,3cyclictest23433-21ssh23:56:280
165659940,3cyclictest17946-21expr20:15:120
165659940,3cyclictest17883-21ntp_states19:10:170
165659940,3cyclictest17736-21ntp_states22:00:170
165659940,3cyclictest14580-21sh00:25:000
165659940,3cyclictest12157-21/usr/sbin/munin23:35:190
165659940,2cyclictest0-21swapper/022:15:150
165659940,0cyclictest0-21swapper/021:45:260
165659940,0cyclictest0-21swapper/000:35:150
165729932,1cyclictest5186-21wget19:50:113
165729930,3cyclictest5114-21apt-get20:55:123
165729930,3cyclictest3047-21df_inode00:35:163
165729930,3cyclictest2037-21rm23:30:063
165729930,3cyclictest14458-21apt-config00:25:003
165729930,3cyclictest0-21swapper/321:40:123
165729930,3cyclictest0-21swapper/321:00:213
165729930,3cyclictest0-21swapper/320:40:133
165729930,3cyclictest0-21swapper/320:30:123
165729930,3cyclictest0-21swapper/320:20:063
165729930,3cyclictest0-21swapper/300:10:063
165729930,2cyclictest0-21swapper/320:05:143
165729930,1cyclictest6939-21taskset00:05:063
165729930,1cyclictest0-21swapper/319:30:113
165729930,0cyclictest0-21swapper/320:45:113
165689932,1cyclictest808-21snmpd19:20:032
165689931,1cyclictest21908-21ssh23:40:032
165689930,3cyclictest9424-21apt-get22:45:102
165689930,3cyclictest24331-21ssh23:25:022
165689930,3cyclictest11634-21python20:00:202
165689930,3cyclictest0-21swapper/222:20:012
165689930,3cyclictest0-21swapper/222:05:132
165689930,3cyclictest0-21swapper/221:40:092
165689930,3cyclictest0-21swapper/220:55:122
165689930,3cyclictest0-21swapper/220:50:132
165689930,3cyclictest0-21swapper/220:45:112
165689930,3cyclictest0-21swapper/220:40:132
165689930,3cyclictest0-21swapper/220:35:122
165689930,3cyclictest0-21swapper/220:15:112
165689930,3cyclictest0-21swapper/220:10:112
165689930,3cyclictest0-21swapper/219:45:112
165689930,3cyclictest0-21swapper/219:25:102
165689930,3cyclictest0-21swapper/200:20:032
165669931,1cyclictest0-21swapper/122:40:111
165669931,1cyclictest0-21swapper/121:55:111
165669931,1cyclictest0-21swapper/121:55:111
165669931,1cyclictest0-21swapper/121:45:131
165669931,1cyclictest0-21swapper/121:00:121
165669931,1cyclictest0-21swapper/120:50:131
165669931,1cyclictest0-21swapper/119:20:121
165669931,1cyclictest0-21swapper/100:30:121
165669931,1cyclictest0-21swapper/100:15:141
165669931,1cyclictest0-21swapper/100:05:151
165669930,2cyclictest15887-21chrt21:27:171
165669930,2cyclictest0-21swapper/123:55:121
165669930,2cyclictest0-21swapper/123:15:111
165669930,2cyclictest0-21swapper/119:50:111
165669930,2cyclictest0-21swapper/119:10:111
165669930,2cyclictest0-21swapper/100:10:131
165669930,1cyclictest125922sleep120:05:141
165659933,0cyclictest32211-21tr22:40:120
165659932,1cyclictest808-21snmpd23:50:010
165659932,0cyclictest24553-21apt-get23:25:100
165659931,1cyclictest6304-21apt-get21:55:110
165659931,1cyclictest6304-21apt-get21:55:110
165659930,3cyclictest674-21df20:45:110
165659930,3cyclictest4010-21ssh22:10:140
165659930,3cyclictest26082-21sed00:30:170
165659930,3cyclictest25642-21diskstats19:25:110
165659930,3cyclictest23861-21cat22:20:020
165659930,3cyclictest22673-21gpgv20:25:000
165659930,3cyclictest19863-21open_inodes19:15:010
165659930,3cyclictest16351-21open_inodes20:10:180
165659930,3cyclictest13444-21diskstats21:10:140
165659930,3cyclictest12589-21apt-get20:05:140
165659930,3cyclictest10663-21fschecks_count20:00:140
165659930,3cyclictest0-21swapper/021:05:120
165659930,3cyclictest0-21swapper/021:00:120
165659930,3cyclictest0-21swapper/020:30:120
165659930,3cyclictest0-21swapper/019:40:110
165659930,3cyclictest0-21swapper/019:20:120
165659930,0cyclictest0-21swapper/019:45:110
165669921,1cyclictest0-21swapper/122:30:131
165669921,1cyclictest0-21swapper/121:35:121
165669921,1cyclictest0-21swapper/120:35:121
165669920,1cyclictest27275-21chrt19:28:111
165669920,1cyclictest0-21swapper/122:05:091
165669920,1cyclictest0-21swapper/122:00:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional