You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-03 - 02:29
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot2.osadl.org (updated Sun Dec 03, 2023 00:44:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
192762203186,12sleep10-21swapper/119:05:301
193252194179,11sleep30-21swapper/319:06:093
195782185155,11sleep00-21swapper/019:09:240
195922161143,13sleep20-21swapper/219:09:332
220312460,0sleep10-21swapper/121:52:291
186742440,0sleep20-21swapper/220:10:212
70412120,0sleep10-21swapper/121:24:261
68382100,1sleep26841-21pmu-power00:25:362
197959982,3cyclictest12758-21munin-run21:05:020
197959981,3cyclictest31351-21awk22:40:010
197959962,4cyclictest32450-21munin-run22:20:010
198099955,0cyclictest0-21swapper/323:40:533
198099955,0cyclictest0-21swapper/323:32:013
198099950,5cyclictest0-21swapper/323:10:203
198099950,5cyclictest0-21swapper/319:10:323
198099950,0cyclictest15221-21diskmemload21:25:353
198099950,0cyclictest0-21swapper/323:49:453
198099950,0cyclictest0-21swapper/321:45:133
198099950,0cyclictest0-21swapper/300:33:423
198049951,2cyclictest0-21swapper/223:16:592
198049950,5cyclictest0-21swapper/222:05:032
198009951,2cyclictest0-21swapper/123:40:171
198009951,2cyclictest0-21swapper/123:20:221
198009951,2cyclictest0-21swapper/122:55:221
198009951,2cyclictest0-21swapper/121:15:181
198009950,5cyclictest0-21swapper/123:05:521
198009950,5cyclictest0-21swapper/122:49:011
198009950,3cyclictest0-21swapper/119:35:211
197959955,0cyclictest3-21ksoftirqd/021:50:020
197959955,0cyclictest3-21ksoftirqd/021:35:010
197959950,5cyclictest18472-21ssh00:34:120
197959950,4cyclictest7826-21apt-get20:55:200
197959950,4cyclictest0-21swapper/023:35:010
2602240,0sleep20-21swapper/223:22:542
24683240,0sleep10-21swapper/121:35:041
198099944,0cyclictest0-21swapper/323:08:443
198099944,0cyclictest0-21swapper/322:20:433
198099944,0cyclictest0-21swapper/322:08:483
198099941,2cyclictest0-21swapper/300:25:183
198099940,4cyclictest30951-21apt-get23:00:153
198099940,4cyclictest30147-21ssh23:59:033
198099940,4cyclictest15221-21diskmemload00:12:363
198099940,4cyclictest13719-21munin-run00:10:003
198099940,4cyclictest0-21swapper/323:50:233
198099940,4cyclictest0-21swapper/323:25:213
198099940,4cyclictest0-21swapper/323:15:193
198099940,4cyclictest0-21swapper/322:51:093
198099940,4cyclictest0-21swapper/322:39:553
198099940,4cyclictest0-21swapper/322:30:163
198099940,4cyclictest0-21swapper/322:17:083
198099940,4cyclictest0-21swapper/322:10:083
198099940,4cyclictest0-21swapper/321:40:243
198099940,4cyclictest0-21swapper/321:40:233
198099940,4cyclictest0-21swapper/321:21:393
198099940,4cyclictest0-21swapper/321:15:173
198099940,4cyclictest0-21swapper/320:10:183
198099940,4cyclictest0-21swapper/320:00:183
198099940,4cyclictest0-21swapper/319:47:563
198099940,4cyclictest0-21swapper/300:36:073
198099940,3cyclictest5881-21df_inode20:50:223
198099940,3cyclictest588-21hddtemp_smartct22:00:173
198099940,3cyclictest3665-21ssh22:42:123
198099940,3cyclictest32129-21awk20:35:353
198099940,3cyclictest28695-21latency_hist00:20:003
198099940,3cyclictest18090-21diskstats21:50:213
198099940,3cyclictest15880-21cstates20:05:203
198099940,3cyclictest0-21swapper/322:55:223
198049944,0cyclictest22368-21kworker/2:023:08:002
198049944,0cyclictest22368-21kworker/2:022:56:472
198049944,0cyclictest22368-21kworker/2:021:16:022
198049944,0cyclictest22368-21kworker/2:019:35:172
198049944,0cyclictest22368-21kworker/2:019:24:062
198049944,0cyclictest22368-21kworker/2:000:15:092
198049943,1cyclictest32705-21grep23:40:272
198049943,1cyclictest30343-21/usr/sbin/munin20:35:342
198049940,4cyclictest360-21perl22:40:282
198049940,4cyclictest0-21swapper/223:50:042
198049940,4cyclictest0-21swapper/223:30:022
198049940,4cyclictest0-21swapper/223:10:162
198049940,4cyclictest0-21swapper/222:50:142
198049940,4cyclictest0-21swapper/222:10:222
198049940,4cyclictest0-21swapper/221:55:382
198049940,4cyclictest0-21swapper/221:40:112
198049940,4cyclictest0-21swapper/221:40:112
198049940,4cyclictest0-21swapper/221:38:572
198049940,4cyclictest0-21swapper/221:34:152
198049940,4cyclictest0-21swapper/221:00:212
198049940,4cyclictest0-21swapper/220:45:192
198049940,4cyclictest0-21swapper/220:18:522
198049940,4cyclictest0-21swapper/219:15:182
198049940,4cyclictest0-21swapper/200:35:132
198049940,4cyclictest0-21swapper/200:32:372
198049940,4cyclictest0-21swapper/200:12:202
198049940,4cyclictest0-21swapper/200:01:582
198049940,3cyclictest8675-21sh23:45:322
198049940,3cyclictest4782-21latency_hist00:25:002
198049940,3cyclictest30314-21cut20:35:002
198049940,3cyclictest16236-21cat21:30:022
198049940,2cyclictest0-21swapper/222:47:022
198049940,1cyclictest5882-21/usr/sbin/munin23:25:312
198049940,1cyclictest31854-21hddtemp_smartct23:00:182
198049940,1cyclictest22885-21/usr/sbin/munin23:35:322
198049940,1cyclictest18081-21df_inode21:50:212
198049940,1cyclictest10537-21/usr/sbin/munin19:55:272
198009944,0cyclictest0-21swapper/121:55:281
198009944,0cyclictest0-21swapper/121:26:121
198009941,2cyclictest0-21swapper/123:50:221
198009941,2cyclictest0-21swapper/123:30:191
198009941,2cyclictest0-21swapper/123:15:201
198009941,2cyclictest0-21swapper/121:45:131
198009941,2cyclictest0-21swapper/121:05:211
198009941,2cyclictest0-21swapper/120:15:191
198009941,2cyclictest0-21swapper/120:05:181
198009941,2cyclictest0-21swapper/119:30:201
198009941,2cyclictest0-21swapper/100:30:011
198009941,2cyclictest0-21swapper/100:15:021
198009941,2cyclictest0-21swapper/100:10:171
198009941,2cyclictest0-21swapper/100:05:141
198009941,1cyclictest0-21swapper/120:30:201
198009941,1cyclictest0-21swapper/120:20:211
198009941,1cyclictest0-21swapper/119:45:171
198009940,4cyclictest0-21swapper/122:35:191
198009940,4cyclictest0-21swapper/122:06:311
198009940,4cyclictest0-21swapper/120:52:271
198009940,4cyclictest0-21swapper/120:35:071
198009940,4cyclictest0-21swapper/119:41:261
198009940,3cyclictest0-21swapper/120:55:191
198009940,2cyclictest0-21swapper/123:35:221
198009940,0cyclictest0-21swapper/123:04:161
197959944,0cyclictest0-21swapper/021:10:300
197959943,1cyclictest25810-21grep23:55:360
197959941,3cyclictest759-21snmpd22:45:560
197959941,2cyclictest0-21swapper/020:25:210
197959940,4cyclictest9143-21ssh00:27:040
197959940,4cyclictest26386-21ntp_states21:35:250
197959940,4cyclictest15496-21apt-get20:05:190
197959940,4cyclictest15221-21diskmemload21:15:350
197959940,4cyclictest10258-21ssh23:27:250
197959940,4cyclictest0-21swapper/023:41:100
197959940,4cyclictest0-21swapper/023:10:150
197959940,4cyclictest0-21swapper/023:00:150
197959940,4cyclictest0-21swapper/022:50:130
197959940,4cyclictest0-21swapper/022:40:240
197959940,4cyclictest0-21swapper/022:30:230
197959940,4cyclictest0-21swapper/022:05:140
197959940,4cyclictest0-21swapper/021:55:410
197959940,4cyclictest0-21swapper/021:28:330
197959940,4cyclictest0-21swapper/021:20:240
197959940,4cyclictest0-21swapper/019:45:270
197959940,4cyclictest0-21swapper/000:35:150
197959940,3cyclictest6523-21taskset23:05:010
197959940,3cyclictest5111-21latency_hist20:50:000
197959940,3cyclictest3419-21ssh22:21:070
197959940,3cyclictest32130-21/usr/sbin/munin22:00:190
197959940,3cyclictest2501-21ssh00:02:400
197959940,3cyclictest24986-21ssh22:55:380
197959940,3cyclictest24840-21ssh00:16:570
197959940,3cyclictest24311-21sed22:15:020
197959940,3cyclictest2258-21ssh23:22:370
197959940,3cyclictest22495-21perl19:15:010
197959940,3cyclictest21922-21ssh21:52:230
197959940,3cyclictest19056-21memory20:10:240
197959940,3cyclictest1716-21grep21:40:270
197959940,3cyclictest1716-21grep21:40:270
197959940,3cyclictest14323-21ssh23:49:460
197959940,3cyclictest13790-21find23:30:010
197959940,3cyclictest0-21swapper/020:00:180
197959940,1cyclictest16890-21ntp_states23:50:310
198099933,0cyclictest0-21swapper/319:35:173
198099932,1cyclictest25496-21aten2_r5power_v20:25:223
198099931,2cyclictest0-21swapper/321:55:123
198099931,1cyclictest0-21swapper/319:50:213
198099930,3cyclictest29771-21df00:20:173
198099930,3cyclictest28575-21perl20:30:233
198099930,3cyclictest22896-21munin-node19:15:013
198099930,3cyclictest15616-21ssh21:10:113
198099930,3cyclictest15221-21diskmemload00:00:023
198099930,3cyclictest10883-21tr19:55:233
198099930,3cyclictest0-21swapper/323:35:173
198099930,3cyclictest0-21swapper/323:20:103
198099930,3cyclictest0-21swapper/322:45:093
198099930,3cyclictest0-21swapper/322:25:193
198099930,3cyclictest0-21swapper/321:35:023
198099930,3cyclictest0-21swapper/321:30:053
198099930,3cyclictest0-21swapper/321:05:053
198099930,3cyclictest0-21swapper/321:00:023
198099930,3cyclictest0-21swapper/320:55:203
198099930,3cyclictest0-21swapper/320:45:053
198099930,3cyclictest0-21swapper/320:40:193
198099930,3cyclictest0-21swapper/320:20:043
198099930,3cyclictest0-21swapper/320:15:163
198099930,3cyclictest0-21swapper/319:40:223
198099930,3cyclictest0-21swapper/319:30:073
198099930,3cyclictest0-21swapper/319:25:203
198099930,3cyclictest0-21swapper/319:20:183
198049931,2cyclictest57250irq/125-eth022:30:032
198049931,1cyclictest759-21snmpd20:55:042
198049931,1cyclictest0-21swapper/222:15:192
198049931,1cyclictest0-21swapper/220:00:172
198049930,3cyclictest5850-21chrt00:05:102
198049930,3cyclictest367-21/usr/sbin/munin20:40:222
198049930,3cyclictest32686-21ssh22:20:012
198049930,3cyclictest23669-21apt-get22:35:202
198049930,3cyclictest20448-21df_inode19:10:202
198049930,3cyclictest13727-21tune2fs21:05:232
198049930,3cyclictest0-21swapper/223:55:232
198049930,3cyclictest0-21swapper/222:25:442
198049930,3cyclictest0-21swapper/222:00:142
198049930,3cyclictest0-21swapper/221:45:072
198049930,3cyclictest0-21swapper/221:20:062
198049930,3cyclictest0-21swapper/221:10:062
198049930,3cyclictest0-21swapper/220:50:192
198049930,3cyclictest0-21swapper/220:25:002
198049930,3cyclictest0-21swapper/220:20:072
198049930,3cyclictest0-21swapper/220:05:052
198049930,3cyclictest0-21swapper/219:50:022
198049930,3cyclictest0-21swapper/219:45:052
198049930,3cyclictest0-21swapper/219:40:222
198049930,3cyclictest0-21swapper/219:30:152
198049930,3cyclictest0-21swapper/219:25:212
198009933,0cyclictest0-21swapper/122:01:001
198009931,1cyclictest114812chrt22:27:251
198009931,1cyclictest0-21swapper/123:45:241
198009931,1cyclictest0-21swapper/123:25:211
198009931,1cyclictest0-21swapper/123:10:151
198009931,1cyclictest0-21swapper/122:50:141
198009931,1cyclictest0-21swapper/122:40:231
198009931,1cyclictest0-21swapper/122:30:161
198009931,1cyclictest0-21swapper/122:10:221
198009931,1cyclictest0-21swapper/121:40:251
198009931,1cyclictest0-21swapper/121:40:251
198009931,1cyclictest0-21swapper/121:30:241
198009931,1cyclictest0-21swapper/120:45:191
198009931,1cyclictest0-21swapper/120:40:211
198009931,1cyclictest0-21swapper/120:25:211
198009931,1cyclictest0-21swapper/120:00:181
198009931,1cyclictest0-21swapper/119:50:221
198009931,1cyclictest0-21swapper/119:25:211
198009931,1cyclictest0-21swapper/119:15:171
198009931,1cyclictest0-21swapper/119:10:181
198009931,1cyclictest0-21swapper/100:35:161
198009931,1cyclictest0-21swapper/100:25:171
198009930,2cyclictest0-21swapper/122:15:181
198009930,2cyclictest0-21swapper/121:10:201
198009930,2cyclictest0-21swapper/120:10:171
198009930,1cyclictest0-21swapper/100:00:001
198009930,0cyclictest0-21swapper/119:21:211
197959931,2cyclictest0-21swapper/000:05:000
197959930,3cyclictest28844-21perl19:25:230
197959930,3cyclictest21605-21ssh23:15:160
197959930,3cyclictest0-21swapper/022:25:020
197959930,3cyclictest0-21swapper/021:05:190
197959930,3cyclictest0-21swapper/020:50:200
197959930,3cyclictest0-21swapper/020:40:190
197959930,3cyclictest0-21swapper/020:35:040
197959930,3cyclictest0-21swapper/020:30:200
197959930,3cyclictest0-21swapper/020:20:090
197959930,3cyclictest0-21swapper/020:15:180
197959930,3cyclictest0-21swapper/019:55:100
197959930,3cyclictest0-21swapper/019:50:210
197959930,3cyclictest0-21swapper/019:40:220
197959930,3cyclictest0-21swapper/019:35:110
197959930,3cyclictest0-21swapper/019:30:200
197959930,3cyclictest0-21swapper/019:20:080
197959930,3cyclictest0-21swapper/019:10:190
197959930,3cyclictest0-21swapper/000:20:150
197959930,3cyclictest0-21swapper/000:10:150
198009922,0cyclictest0-21swapper/100:20:041
198009920,1cyclictest323942sleep100:00:401
198009920,1cyclictest0-21swapper/122:20:241
198009920,1cyclictest0-21swapper/121:00:011
198009920,1cyclictest0-21swapper/119:55:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional