You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 09:56
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot2.osadl.org (updated Sun Apr 28, 2024 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305312199184,11sleep10-21swapper/119:07:471
306072194178,12sleep00-21swapper/019:08:480
303602175159,12sleep30-21swapper/319:05:383
306762159141,13sleep20-21swapper/219:09:412
247162510,0sleep00-21swapper/022:27:080
207592440,0sleep00-21swapper/021:45:470
12062440,0sleep10-21swapper/122:32:321
57832130,0sleep20-21swapper/221:18:162
207792100,0sleep00-21swapper/023:22:570
308639970,6cyclictest61-21kswapd023:30:132
308679962,3cyclictest3602-21snmpd00:12:393
308679961,5cyclictest0-21swapper/322:10:113
308519960,6cyclictest0-21swapper/000:28:440
308519960,5cyclictest0-21swapper/023:25:110
308519960,3cyclictest17257-21apt-get19:50:010
308679953,1cyclictest40-21ksoftirqd/322:10:013
308679950,5cyclictest0-21swapper/322:28:353
308679950,5cyclictest0-21swapper/322:19:593
308679950,5cyclictest0-21swapper/321:56:173
308679950,5cyclictest0-21swapper/300:09:263
308679950,4cyclictest0-21swapper/322:35:103
308679950,4cyclictest0-21swapper/321:20:093
308679950,0cyclictest0-21swapper/321:26:443
308639952,3cyclictest7763-21sort23:15:112
308639950,5cyclictest17675-21apt-get19:50:012
308549950,5cyclictest0-21swapper/121:35:311
308519951,4cyclictest32067-21munin-run21:15:020
308519950,5cyclictest0-21swapper/022:33:560
308519950,5cyclictest0-21swapper/000:04:260
308519950,4cyclictest16212-21apt-get20:55:200
308519950,4cyclictest0-21swapper/000:05:120
308519950,1cyclictest5550-21munin-run22:55:010
308519950,0cyclictest0-21swapper/021:15:040
10236250,1sleep210235-21apt00:35:002
32628240,0sleep00-21swapper/023:10:240
308679944,0cyclictest0-21swapper/323:51:313
308679944,0cyclictest0-21swapper/323:10:113
308679942,1cyclictest3602-21snmpd20:00:013
308679941,2cyclictest0-21swapper/322:40:143
308679940,4cyclictest7578-21ntp_states20:35:243
308679940,4cyclictest5154-21ssh23:32:593
308679940,4cyclictest19797-21kworker/3:219:55:153
308679940,4cyclictest0-21swapper/323:55:113
308679940,4cyclictest0-21swapper/323:45:063
308679940,4cyclictest0-21swapper/323:40:113
308679940,4cyclictest0-21swapper/323:25:333
308679940,4cyclictest0-21swapper/323:20:113
308679940,4cyclictest0-21swapper/323:18:283
308679940,4cyclictest0-21swapper/322:50:123
308679940,4cyclictest0-21swapper/322:45:143
308679940,4cyclictest0-21swapper/322:24:053
308679940,4cyclictest0-21swapper/321:51:533
308679940,4cyclictest0-21swapper/321:51:523
308679940,4cyclictest0-21swapper/321:45:053
308679940,4cyclictest0-21swapper/321:30:113
308679940,4cyclictest0-21swapper/321:10:163
308679940,4cyclictest0-21swapper/321:00:113
308679940,4cyclictest0-21swapper/320:34:493
308679940,4cyclictest0-21swapper/300:30:113
308679940,4cyclictest0-21swapper/300:22:193
308679940,4cyclictest0-21swapper/300:03:493
308679940,3cyclictest8115-21/usr/sbin/munin23:35:163
308679940,3cyclictest6758-21/usr/sbin/munin19:25:163
308679940,3cyclictest32278-21ssh21:15:013
308679940,3cyclictest30704-21threads22:30:263
308679940,3cyclictest1888-21diskstats20:25:133
308679940,3cyclictest14368-21ssh23:00:093
308679940,3cyclictest13729-21/usr/sbin/munin20:50:283
308679940,3cyclictest11224-21date20:45:013
308679940,3cyclictest10968-21memory21:40:183
308679940,3cyclictest10714-21ntp_states19:30:233
308679940,3cyclictest10095-21ssh00:15:083
308679940,3cyclictest0-21swapper/321:35:133
308679940,3cyclictest0-21swapper/320:05:093
308679940,3cyclictest0-21swapper/319:15:153
308639944,0cyclictest0-21swapper/222:00:122
308639943,1cyclictest4291-21/usr/sbin/munin19:20:282
308639943,1cyclictest31215-21/usr/sbin/munin20:20:162
308639942,2cyclictest9669-21sh21:40:072
308639942,2cyclictest548-21in:imuxsock21:40:022
308639942,2cyclictest31-21ksoftirqd/222:05:132
308639942,1cyclictest9295-21date19:30:012
308639942,1cyclictest0-21swapper/222:55:122
308639941,3cyclictest22328-21cat22:25:232
308639941,2cyclictest0-21swapper/222:15:102
308639941,2cyclictest0-21swapper/221:25:102
308639940,4cyclictest8786-21/usr/sbin/munin20:40:162
308639940,4cyclictest31506-21ssh22:50:222
308639940,4cyclictest0-21swapper/223:50:092
308639940,4cyclictest0-21swapper/223:35:082
308639940,4cyclictest0-21swapper/223:25:122
308639940,4cyclictest0-21swapper/223:05:552
308639940,4cyclictest0-21swapper/222:45:322
308639940,4cyclictest0-21swapper/222:21:432
308639940,4cyclictest0-21swapper/221:56:422
308639940,4cyclictest0-21swapper/221:50:122
308639940,4cyclictest0-21swapper/221:50:122
308639940,4cyclictest0-21swapper/221:13:032
308639940,4cyclictest0-21swapper/220:35:132
308639940,4cyclictest0-21swapper/220:30:082
308639940,4cyclictest0-21swapper/219:10:062
308639940,4cyclictest0-21swapper/200:25:052
308639940,4cyclictest0-21swapper/200:11:262
308639940,4cyclictest0-21swapper/200:00:132
308639940,3cyclictest30078-21/usr/sbin/munin22:30:202
308639940,3cyclictest26158-21/usr/sbin/munin20:10:222
308639940,3cyclictest23787-21grep21:10:012
308639940,3cyclictest2315-21df19:15:122
308639940,3cyclictest19137-21grep00:20:132
308639940,3cyclictest15669-21tune2fs19:45:092
308639940,3cyclictest10578-21irqstats23:55:192
308639940,1cyclictest8445-21/usr/sbin/munin21:20:212
308639940,1cyclictest26662-21diskstats00:05:142
308639940,1cyclictest16940-21ntp_states23:20:212
308639940,1cyclictest15175-21sed22:40:292
308639940,1cyclictest0-21swapper/223:45:242
308639940,1cyclictest0-21swapper/222:10:152
308639940,0cyclictest0-21swapper/200:38:552
308549944,0cyclictest0-21swapper/122:16:081
308549944,0cyclictest0-21swapper/120:40:221
308549944,0cyclictest0-21swapper/100:02:381
308549941,2cyclictest0-21swapper/123:20:101
308549941,2cyclictest0-21swapper/122:35:111
308549941,2cyclictest0-21swapper/122:05:121
308549941,2cyclictest0-21swapper/121:40:101
308549941,2cyclictest0-21swapper/121:25:111
308549941,2cyclictest0-21swapper/121:10:151
308549941,2cyclictest0-21swapper/120:35:141
308549940,4cyclictest0-21swapper/122:48:161
308549940,4cyclictest0-21swapper/121:59:141
308549940,4cyclictest0-21swapper/121:50:491
308549940,4cyclictest0-21swapper/121:50:491
308549940,4cyclictest0-21swapper/119:30:321
308549940,4cyclictest0-21swapper/119:10:441
308549940,0cyclictest0-21swapper/100:17:191
308549940,0cyclictest0-21swapper/100:13:151
308519944,0cyclictest0-21swapper/023:47:430
308519944,0cyclictest0-21swapper/023:03:020
308519944,0cyclictest0-21swapper/022:17:290
308519944,0cyclictest0-21swapper/021:25:430
308519942,1cyclictest220-21systemd-journal22:00:020
308519941,2cyclictest0-21swapper/020:50:130
308519940,4cyclictest0-21swapper/023:35:080
308519940,4cyclictest0-21swapper/022:40:200
308519940,4cyclictest0-21swapper/022:07:220
308519940,4cyclictest0-21swapper/021:20:090
308519940,4cyclictest0-21swapper/020:15:080
308519940,4cyclictest0-21swapper/000:20:090
308519940,3cyclictest8173-21ssh22:36:240
308519940,3cyclictest5811-21ssh22:55:020
308519940,3cyclictest3601-21ssh21:35:310
308519940,3cyclictest31381-21ssh22:11:370
308519940,3cyclictest31218-21apt-get20:20:120
308519940,3cyclictest2634-21rm21:55:120
308519940,3cyclictest24353-21tune2fs20:05:150
308519940,3cyclictest2405-21apt-get00:30:110
308519940,3cyclictest21404-21chrt22:45:010
308519940,3cyclictest1936-21ssh00:10:110
308519940,3cyclictest18507-21/usr/sbin/munin19:55:060
308519940,3cyclictest17718-21if_eth423:40:200
308519940,3cyclictest15073-21ssh00:37:170
308519940,3cyclictest13629-21sed20:50:000
308519940,3cyclictest12630-21df22:20:130
308519940,3cyclictest10295-21df_inode21:40:120
308519940,3cyclictest10224-21ssh23:16:150
308519940,0cyclictest0-21swapper/023:58:020
308679932,1cyclictest3602-21snmpd20:55:223
308679932,1cyclictest3602-21snmpd20:15:023
308679930,3cyclictest5991-21rm22:55:123
308679930,3cyclictest31594-21grep19:10:103
308679930,3cyclictest22635-21ssh23:05:113
308679930,3cyclictest17672-21rm19:50:013
308679930,3cyclictest10877-21grep00:35:133
308679930,3cyclictest0-21swapper/322:00:113
308679930,3cyclictest0-21swapper/321:05:133
308679930,3cyclictest0-21swapper/320:45:103
308679930,3cyclictest0-21swapper/320:20:133
308679930,3cyclictest0-21swapper/320:10:083
308679930,3cyclictest0-21swapper/319:40:493
308679930,3cyclictest0-21swapper/319:35:023
308679930,3cyclictest0-21swapper/319:20:123
308679930,3cyclictest0-21swapper/300:25:103
308679930,2cyclictest15538-21df_inode19:45:103
308639930,3cyclictest6758-21/usr/sbin/munin19:25:112
308639930,3cyclictest18481-21cat19:55:012
308639930,3cyclictest13733-21apt-get20:50:142
308639930,3cyclictest0-21swapper/223:40:022
308639930,3cyclictest0-21swapper/223:10:002
308639930,3cyclictest0-21swapper/223:00:102
308639930,3cyclictest0-21swapper/221:45:112
308639930,3cyclictest0-21swapper/221:30:082
308639930,3cyclictest0-21swapper/221:00:092
308639930,3cyclictest0-21swapper/220:55:052
308639930,3cyclictest0-21swapper/220:25:122
308639930,3cyclictest0-21swapper/220:15:042
308639930,3cyclictest0-21swapper/220:05:092
308639930,3cyclictest0-21swapper/220:00:122
308639930,3cyclictest0-21swapper/219:41:372
308639930,3cyclictest0-21swapper/219:35:112
308639930,3cyclictest0-21swapper/200:15:022
308639930,1cyclictest4766-21ssh22:35:102
308639930,1cyclictest11250-21/usr/sbin/munin20:45:112
308549933,0cyclictest0-21swapper/122:10:451
308549933,0cyclictest0-21swapper/121:24:551
308549933,0cyclictest0-21swapper/120:55:041
308549931,1cyclictest0-21swapper/123:40:121
308549931,1cyclictest0-21swapper/123:35:091
308549931,1cyclictest0-21swapper/123:35:011
308549931,1cyclictest0-21swapper/123:25:111
308549931,1cyclictest0-21swapper/123:15:101
308549931,1cyclictest0-21swapper/123:10:021
308549931,1cyclictest0-21swapper/123:00:091
308549931,1cyclictest0-21swapper/122:20:111
308549931,1cyclictest0-21swapper/122:00:101
308549931,1cyclictest0-21swapper/121:30:101
308549931,1cyclictest0-21swapper/121:15:091
308549931,1cyclictest0-21swapper/120:20:121
308549931,1cyclictest0-21swapper/120:10:181
308549931,1cyclictest0-21swapper/120:05:101
308549931,1cyclictest0-21swapper/120:00:131
308549930,3cyclictest0-21swapper/119:26:361
308549930,2cyclictest6033-21chrt23:52:351
308549930,2cyclictest249942sleep121:49:061
308549930,2cyclictest12614-21chrt22:58:591
308549930,2cyclictest10662sleep122:51:301
308549930,2cyclictest0-21swapper/122:25:101
308549930,2cyclictest0-21swapper/120:15:111
308549930,2cyclictest0-21swapper/100:05:021
308519932,1cyclictest18951-21grep21:00:130
308519931,1cyclictest0-21swapper/020:35:140
308519931,1cyclictest0-21swapper/020:30:080
308519931,1cyclictest0-21swapper/019:15:110
308519931,1cyclictest0-21swapper/000:15:100
308519930,3cyclictest0-21swapper/023:50:010
308519930,3cyclictest0-21swapper/023:30:000
308519930,3cyclictest0-21swapper/023:05:120
308519930,3cyclictest0-21swapper/021:50:130
308519930,3cyclictest0-21swapper/021:50:120
308519930,3cyclictest0-21swapper/021:30:110
308519930,3cyclictest0-21swapper/021:05:130
308519930,3cyclictest0-21swapper/020:40:120
308519930,3cyclictest0-21swapper/020:25:110
308519930,3cyclictest0-21swapper/020:10:180
308519930,3cyclictest0-21swapper/020:00:120
308519930,3cyclictest0-21swapper/019:50:470
308519930,3cyclictest0-21swapper/019:43:190
308519930,3cyclictest0-21swapper/019:35:180
308519930,3cyclictest0-21swapper/019:30:000
308519930,3cyclictest0-21swapper/019:25:100
308519930,3cyclictest0-21swapper/019:20:120
308519930,3cyclictest0-21swapper/019:10:030
308549921,1cyclictest233162chrt21:06:331
308549921,1cyclictest0-21swapper/122:40:151
308549921,1cyclictest0-21swapper/120:45:091
308549921,1cyclictest0-21swapper/119:35:121
308549921,1cyclictest0-21swapper/119:20:111
308549921,0cyclictest0-21swapper/123:05:131
308549920,1cyclictest14708-21chrt19:44:111
308549920,1cyclictest0-21swapper/123:55:101
308549920,1cyclictest0-21swapper/123:45:091
308549920,1cyclictest0-21swapper/121:00:111
308549920,1cyclictest0-21swapper/120:50:141
308549920,1cyclictest0-21swapper/120:30:091
308549920,1cyclictest0-21swapper/120:25:121
308549920,1cyclictest0-21swapper/119:55:001
308549920,1cyclictest0-21swapper/119:50:001
308549920,1cyclictest0-21swapper/119:50:001
308549920,1cyclictest0-21swapper/119:15:101
308549920,1cyclictest0-21swapper/100:35:231
308549920,1cyclictest0-21swapper/100:30:121
308549920,1cyclictest0-21swapper/100:25:111
308549920,1cyclictest0-21swapper/100:20:091
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional