You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-28 - 14:42

x86 Intel Core i3-8145UE @2200 MHz, Linux 5.10.27-rt36 (Profile)

Latency plot of system in rack #e, slot #5
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5.osadl.org (updated Tue Mar 28, 2023 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21297993129,1cyclictest18727-21hddtemp_smartct22:20:163
2129799310,30cyclictest28802-21cstates22:30:203
2129799310,30cyclictest2196-21/usr/sbin/munin21:30:203
2129399310,30cyclictest9883-21/usr/sbin/munin23:15:212
2128999310,30cyclictest12849-21diskmemload21:59:551
2128599310,30cyclictest9344-21latency23:15:170
2128599310,30cyclictest4303-21runrttasks22:05:260
2128599310,30cyclictest12264-21cat21:40:250
21297993028,1cyclictest12849-21diskmemload21:15:043
2129799300,29cyclictest9322-21missed_timers00:20:193
2129799300,29cyclictest307-21systemd-journal22:05:013
2129799300,29cyclictest13055-21munin-run23:20:013
2129799300,29cyclictest12849-21diskmemload23:31:193
2129799300,29cyclictest12849-21diskmemload22:25:013
2129799300,29cyclictest12849-21diskmemload21:25:183
2129799300,29cyclictest0-21swapper/322:45:143
2129799300,29cyclictest0-21swapper/321:53:043
2129799300,1cyclictest14729-21taskset21:43:083
2129799300,0cyclictest12849-21diskmemload23:04:493
21293993029,1cyclictest12849-21diskmemload22:35:582
21289993028,1cyclictest0-21swapper/122:50:001
2128999300,29cyclictest12849-21diskmemload22:40:241
2128999300,29cyclictest12849-21diskmemload21:16:371
2128999300,29cyclictest11373-21ssh21:40:211
2128999300,29cyclictest0-21swapper/123:40:131
2128999300,29cyclictest0-21swapper/121:05:021
2128999300,1cyclictest12849-21diskmemload00:20:211
2128599300,30cyclictest0-21swapper/023:55:210
2128599300,29cyclictest12849-21diskmemload22:03:510
2128599300,29cyclictest12849-21diskmemload21:30:180
2128599300,29cyclictest12849-21diskmemload21:15:280
2128599300,29cyclictest12849-21diskmemload00:35:480
2128599300,29cyclictest0-21swapper/023:50:590
2128599300,29cyclictest0-21swapper/023:25:030
2128599300,29cyclictest0-21swapper/022:45:180
2128599300,29cyclictest0-21swapper/022:32:220
2128599300,29cyclictest0-21swapper/021:21:420
2128599300,29cyclictest0-21swapper/000:28:330
2128599300,1cyclictest25981-21cstates00:05:220
21297992928,1cyclictest12849-21diskmemload21:35:433
21297992927,1cyclictest17070-21apt-get20:10:143
2129799290,29cyclictest12849-21diskmemload23:52:533
2129799290,29cyclictest12849-21diskmemload22:37:053
2129799290,29cyclictest12849-21diskmemload21:23:543
2129799290,29cyclictest0-21swapper/323:59:323
2129799290,29cyclictest0-21swapper/322:05:583
2129799290,29cyclictest0-21swapper/300:37:133
2129799290,29cyclictest0-21swapper/300:15:173
2129799290,29cyclictest0-21swapper/300:07:433
2129799290,29cyclictest0-21swapper/300:04:353
2129799290,28cyclictest0-21swapper/319:20:203
2129799290,0cyclictest12849-21diskmemload23:45:243
2129799290,0cyclictest12849-21diskmemload23:35:353
2129799290,0cyclictest12849-21diskmemload23:25:233
2129799290,0cyclictest12849-21diskmemload23:11:513
2129799290,0cyclictest12849-21diskmemload22:57:423
2129799290,0cyclictest12849-21diskmemload22:13:223
2129799290,0cyclictest12849-21diskmemload21:45:533
2129799290,0cyclictest12849-21diskmemload21:11:023
2129799290,0cyclictest12441-21apt-get20:00:183
2129799290,0cyclictest0-21swapper/323:44:413
2129799290,0cyclictest0-21swapper/323:21:393
2129799290,0cyclictest0-21swapper/323:05:163
2129799290,0cyclictest0-21swapper/300:26:533
2129799290,0cyclictest0-21swapper/300:10:353
21293992928,1cyclictest26016-21cstates00:05:212
21293992928,1cyclictest18204-21sh23:25:032
21293992928,1cyclictest12849-21diskmemload22:51:032
21293992928,1cyclictest12849-21diskmemload00:39:052
2129399290,29cyclictest12849-21diskmemload23:41:072
2129399290,29cyclictest12849-21diskmemload22:12:252
2129399290,29cyclictest12849-21diskmemload22:06:132
2129399290,29cyclictest12849-21diskmemload21:55:012
2129399290,29cyclictest12849-21diskmemload21:35:052
2129399290,29cyclictest12849-21diskmemload21:20:402
2129399290,29cyclictest12849-21diskmemload00:16:252
2129399290,29cyclictest12849-21diskmemload00:01:352
2129399290,29cyclictest0-21swapper/223:49:142
2129399290,29cyclictest0-21swapper/223:36:392
2129399290,29cyclictest0-21swapper/223:30:062
2129399290,29cyclictest0-21swapper/223:12:292
2129399290,29cyclictest0-21swapper/223:05:162
2129399290,29cyclictest0-21swapper/223:00:152
2129399290,29cyclictest0-21swapper/222:45:062
2129399290,29cyclictest0-21swapper/222:43:422
2129399290,29cyclictest0-21swapper/222:31:002
2129399290,29cyclictest0-21swapper/221:47:562
2129399290,29cyclictest0-21swapper/221:42:342
2129399290,29cyclictest0-21swapper/221:33:272
2129399290,29cyclictest0-21swapper/221:26:392
2129399290,29cyclictest0-21swapper/221:11:152
2129399290,29cyclictest0-21swapper/200:32:152
2129399290,29cyclictest0-21swapper/200:20:562
2129399290,29cyclictest0-21swapper/200:11:152
2129399290,28cyclictest14522-21/usr/sbin/munin23:20:202
2129399290,1cyclictest13423-21ssh23:51:492
2129399290,0cyclictest12849-21diskmemload22:57:042
2129399290,0cyclictest12849-21diskmemload22:26:092
2129399290,0cyclictest12849-21diskmemload21:54:232
2129399290,0cyclictest12849-21diskmemload21:17:572
2129399290,0cyclictest0-21swapper/222:01:062
2129399290,0cyclictest0-21swapper/220:05:122
2129399290,0cyclictest0-21swapper/200:25:512
21289992929,0cyclictest12849-21diskmemload23:20:111
21289992929,0cyclictest12849-21diskmemload22:15:291
21289992929,0cyclictest0-21swapper/121:20:221
21289992927,1cyclictest14750-21apt-get20:05:131
2128999290,29cyclictest12849-21diskmemload23:47:521
2128999290,29cyclictest12849-21diskmemload21:51:591
2128999290,29cyclictest12849-21diskmemload21:25:021
2128999290,29cyclictest0-21swapper/123:55:561
2128999290,29cyclictest0-21swapper/123:35:391
2128999290,29cyclictest0-21swapper/123:33:521
2128999290,29cyclictest0-21swapper/123:25:121
2128999290,29cyclictest0-21swapper/123:15:511
2128999290,29cyclictest0-21swapper/123:13:441
2128999290,29cyclictest0-21swapper/123:05:521
2128999290,29cyclictest0-21swapper/122:58:591
2128999290,29cyclictest0-21swapper/122:50:521
2128999290,29cyclictest0-21swapper/122:31:291
2128999290,29cyclictest0-21swapper/122:25:461
2128999290,29cyclictest0-21swapper/122:20:071
2128999290,29cyclictest0-21swapper/122:05:111
2128999290,29cyclictest0-21swapper/122:04:271
2128999290,29cyclictest0-21swapper/121:46:101
2128999290,29cyclictest0-21swapper/121:33:021
2128999290,29cyclictest0-21swapper/100:15:341
2128999290,29cyclictest0-21swapper/100:09:501
2128999290,0cyclictest12849-21diskmemload22:11:141
2128999290,0cyclictest12849-21diskmemload21:10:191
2128999290,0cyclictest0-21swapper/123:00:071
2128999290,0cyclictest0-21swapper/122:37:231
2128999290,0cyclictest0-21swapper/121:35:161
2128999290,0cyclictest0-21swapper/100:39:241
2128999290,0cyclictest0-21swapper/100:30:391
2128999290,0cyclictest0-21swapper/100:26:161
2128999290,0cyclictest0-21swapper/100:11:401
2128999290,0cyclictest0-21swapper/100:01:451
21285992929,0cyclictest13-21ksoftirqd/022:10:160
21285992929,0cyclictest0-21swapper/023:00:350
21285992929,0cyclictest0-21swapper/021:55:140
21285992929,0cyclictest0-21swapper/021:45:130
21285992928,1cyclictest12849-21diskmemload23:21:490
2128599290,29cyclictest0-21swapper/023:43:390
2128599290,29cyclictest0-21swapper/023:37:050
2128599290,29cyclictest0-21swapper/023:32:260
2128599290,29cyclictest0-21swapper/022:36:370
2128599290,29cyclictest0-21swapper/022:25:190
2128599290,29cyclictest0-21swapper/022:21:060
2128599290,29cyclictest0-21swapper/021:27:300
2128599290,29cyclictest0-21swapper/000:10:450
2128599290,29cyclictest0-21swapper/000:02:230
2128599290,28cyclictest0-21swapper/019:40:150
2128599290,0cyclictest12849-21diskmemload23:45:170
2128599290,0cyclictest12849-21diskmemload21:36:200
2128599290,0cyclictest0-21swapper/022:55:410
2128599290,0cyclictest0-21swapper/022:51:020
2128599290,0cyclictest0-21swapper/022:15:160
2128599290,0cyclictest0-21swapper/021:50:090
2128599290,0cyclictest0-21swapper/000:31:560
2128599290,0cyclictest0-21swapper/000:20:320
2128599290,0cyclictest0-21swapper/000:15:140
21297992828,0cyclictest0-21swapper/322:40:283
21297992828,0cyclictest0-21swapper/300:30:203
2129799280,28cyclictest0-21swapper/320:00:013
2129799280,0cyclictest0-21swapper/321:55:153
2129799280,0cyclictest0-21swapper/319:35:273
21293992828,0cyclictest3143-21apt-get19:40:162
21293992828,0cyclictest12849-21diskmemload22:21:392
21293992828,0cyclictest0-21swapper/223:56:432
2129399280,0cyclictest26208-21apt-get19:20:192
21289992828,0cyclictest0-21swapper/120:20:141
2128999280,28cyclictest0-21swapper/120:55:141
2128999280,0cyclictest0-21swapper/119:20:191
21285992828,0cyclictest0-21swapper/023:10:380
21285992826,1cyclictest0-21swapper/020:55:020
2128599280,28cyclictest0-21swapper/022:40:220
21297992727,0cyclictest0-21swapper/319:25:133
21289992727,0cyclictest0-21swapper/120:45:181
21285992727,0cyclictest28518-21apt-get19:25:140
21285992727,0cyclictest0-21swapper/020:15:020
21285992727,0cyclictest0-21swapper/020:05:130
21285992727,0cyclictest0-21swapper/020:00:190
21297992626,0cyclictest0-21swapper/321:05:123
21297992626,0cyclictest0-21swapper/319:40:173
21289992626,0cyclictest0-21swapper/119:55:131
21297992525,0cyclictest0-21swapper/319:55:013
21293992524,1cyclictest19121-21perf20:15:022
21293992523,1cyclictest5592-21cron20:55:012
21289992522,2cyclictest30545-21sh19:30:011
21285992524,1cyclictest12141-21sendmail-msp20:00:010
21297992424,0cyclictest0-21swapper/320:05:123
21293992424,0cyclictest0-21swapper/220:00:182
21293992424,0cyclictest0-21swapper/219:25:132
21289992424,0cyclictest0-21swapper/120:10:141
21289992424,0cyclictest0-21swapper/119:10:121
21285992424,0cyclictest0-21swapper/020:30:120
21285992424,0cyclictest0-21swapper/019:35:270
21285992422,1cyclictest0-21swapper/020:30:000
2129799225,4cyclictest0-21swapper/319:10:113
21293992221,1cyclictest812-21apt-get19:35:282
21293992220,1cyclictest0-21swapper/220:00:002
21289992222,0cyclictest0-21swapper/120:15:141
21285992222,0cyclictest0-21swapper/021:05:120
21297992118,2cyclictest307-21systemd-journal19:20:013
21293992121,0cyclictest0-21swapper/220:30:132
21293992120,1cyclictest10815-21apt-get21:05:132
21293992119,1cyclictest26377-21munin-run20:30:012
21289992121,0cyclictest0-21swapper/120:30:121
21297992019,1cyclictest470-21cron19:35:003
21289991919,0cyclictest0-21swapper/119:40:171
21289991919,0cyclictest0-21swapper/119:30:141
21289991817,1cyclictest12441-21apt-get20:00:181
21285991818,0cyclictest0-21swapper/019:20:200
21285991717,0cyclictest0-21swapper/020:20:010
21285991716,1cyclictest21540-21apt-get19:10:120
21297991616,0cyclictest0-21swapper/320:45:183
21289991616,0cyclictest0-21swapper/120:50:141
2128999150,14cyclictest0-21swapper/119:50:191
21285991515,0cyclictest1253-21apt-get20:45:190
2128599150,1cyclictest0-21swapper/021:00:010
2128599150,14cyclictest0-21swapper/019:50:210
21297991414,0cyclictest0-21swapper/321:01:033
21297991412,1cyclictest0-21swapper/320:20:163
2129799140,14cyclictest0-21swapper/320:40:253
2129799140,14cyclictest0-21swapper/320:35:113
2129799140,14cyclictest0-21swapper/320:30:243
2129799140,14cyclictest0-21swapper/320:25:423
2129799140,14cyclictest0-21swapper/320:15:533
2129799140,0cyclictest0-21swapper/320:56:073
2129799140,0cyclictest0-21swapper/320:50:023
2129799140,0cyclictest0-21swapper/319:47:233
21293991414,0cyclictest0-21swapper/220:20:022
21293991414,0cyclictest0-21swapper/219:10:132
21293991413,1cyclictest8579-21irqstats19:50:222
2129399140,14cyclictest0-21swapper/221:00:032
2129399140,14cyclictest0-21swapper/220:58:452
2129399140,14cyclictest0-21swapper/220:45:152
2129399140,14cyclictest0-21swapper/220:40:062
2129399140,14cyclictest0-21swapper/220:16:172
2129399140,14cyclictest0-21swapper/219:33:062
2129399140,14cyclictest0-21swapper/219:15:072
2129399140,0cyclictest0-21swapper/220:35:332
21289991414,0cyclictest0-21swapper/121:00:531
21289991414,0cyclictest0-21swapper/120:36:251
21289991414,0cyclictest0-21swapper/119:15:401
2128999140,14cyclictest0-21swapper/120:28:511
2128999140,14cyclictest0-21swapper/119:36:161
2128999140,0cyclictest0-21swapper/120:40:561
2128999140,0cyclictest0-21swapper/119:46:391
21285991414,0cyclictest0-21swapper/019:45:120
2128599140,14cyclictest0-21swapper/021:00:450
2128599140,14cyclictest0-21swapper/020:42:090
2128599140,14cyclictest0-21swapper/020:35:200
2128599140,14cyclictest0-21swapper/020:16:020
2128599140,0cyclictest0-21swapper/019:31:160
2128599140,0cyclictest0-21swapper/019:15:450
21293991313,0cyclictest0-21swapper/219:45:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional