You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Fri May 02, 2025 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2056499250,24cyclictest313-21systemd-journal23:50:002
2056499250,24cyclictest313-21systemd-journal23:50:002
2056499240,23cyclictest5770-21idleruntime-cro21:59:592
2055799240,1cyclictest313-21systemd-journal21:05:000
20561992320,2cyclictest25390-21sendmail22:40:001
2056199230,2cyclictest30134-21cut19:30:001
2056199230,22cyclictest23799-21idleruntime-cro21:29:591
2056199230,22cyclictest18723-21perf00:40:001
20557992320,2cyclictest2493-21cron19:40:000
2055799230,22cyclictest4133-21basename20:50:010
2055799230,21cyclictest0-21swapper/000:15:000
2056799221,1cyclictest10205-21munin-plugin-st23:15:013
2056799220,21cyclictest794-21apt-get22:59:363
2056799220,21cyclictest31605-21apt-get21:49:383
2056799220,21cyclictest0-21swapper/319:24:363
2056799220,1cyclictest7146-21apt-get20:59:393
20564992222,0cyclictest10318-21apt-get19:59:392
20564992220,1cyclictest8918-21ls21:00:002
20564992220,1cyclictest23029-21munin-run22:35:002
20564992220,1cyclictest1932-21latency_hist00:05:012
20564992219,2cyclictest9095-21dump-pmu-power00:20:002
2056499220,21cyclictest3133-21apt-get19:44:362
2056499220,21cyclictest27664-21chrt23:54:362
2056499220,21cyclictest0-21swapper/221:15:002
2056499220,21cyclictest0-21swapper/221:04:382
2056499220,1cyclictest0-21swapper/200:40:002
20561992222,0cyclictest0-21swapper/123:54:361
20561992221,1cyclictest0-21swapper/121:14:401
2056199220,21cyclictest0-21swapper/123:59:371
2056199220,1cyclictest16899-21perf20:10:001
2056199220,1cyclictest0-21swapper/122:59:361
20557992222,0cyclictest0-21swapper/019:44:340
20557992220,1cyclictest26183-21grep21:35:010
20557992220,1cyclictest21137-21apt-get19:14:340
2055799220,21cyclictest0-21swapper/023:59:380
2055799220,21cyclictest0-21swapper/023:10:020
2055799220,21cyclictest0-21swapper/021:59:370
2055799220,21cyclictest0-21swapper/020:39:370
2055799220,21cyclictest0-21swapper/020:19:380
2055799220,21cyclictest0-21swapper/019:54:420
2055799220,1cyclictest0-21swapper/022:05:000
20567992121,0cyclictest22005-21apt-get21:29:353
20567992121,0cyclictest0-21swapper/323:54:363
20567992121,0cyclictest0-21swapper/322:49:353
20567992120,1cyclictest32371-21apt-get20:44:353
20567992120,1cyclictest14745-21apt-get21:14:403
20567992119,1cyclictest313-21systemd-journal22:40:003
2056799211,1cyclictest8186-21latency_hist22:04:593
2056799210,20cyclictest689-21apt-get19:39:353
2056799210,20cyclictest3227-21apt-get23:04:373
2056799210,20cyclictest0-21swapper/322:34:363
2056799210,20cyclictest0-21swapper/320:39:373
2056799210,1cyclictest9557-21apt-get21:04:373
2056799210,1cyclictest30130-21perf19:29:593
2056799210,1cyclictest10804-21apt-get23:19:373
2056799210,0cyclictest16915-21awk20:10:003
20564992121,0cyclictest0-21swapper/220:04:362
20564992121,0cyclictest0-21swapper/200:09:342
20564992120,1cyclictest29959-21apt-get20:39:372
20564992119,1cyclictest26541-21apt-get20:30:022
20564992119,1cyclictest19962-21apt-get20:19:382
2056499210,20cyclictest19610-21apt-get21:24:392
2056499210,20cyclictest18060-21apt-get23:34:372
2056499210,20cyclictest0-21swapper/223:05:002
2056499210,20cyclictest0-21swapper/222:59:362
2056499210,20cyclictest0-21swapper/222:05:022
2056499210,1cyclictest7388-21turbostat.cron23:10:012
2056499210,1cyclictest17138-21apt-get21:19:362
2056499210,1cyclictest16897-21turbostat20:10:002
2056499210,1cyclictest16381-21apt-get22:24:342
2056499210,0cyclictest0-21swapper/222:44:362
2056499210,0cyclictest0-21swapper/221:39:382
20561992120,1cyclictest2539-21apt-get00:09:351
20561992119,1cyclictest0-21swapper/121:55:001
2056199210,21cyclictest0-21swapper/122:49:361
2056199210,20cyclictest0-21swapper/121:04:381
2056199210,20cyclictest0-21swapper/119:19:341
2056199210,0cyclictest0-21swapper/123:04:381
2056199210,0cyclictest0-21swapper/122:09:361
2056199210,0cyclictest0-21swapper/121:49:381
2056199210,0cyclictest0-21swapper/119:49:331
2056199210,0cyclictest0-21swapper/119:49:331
2056199210,0cyclictest0-21swapper/119:39:351
20557992121,0cyclictest0-21swapper/022:34:360
20557992121,0cyclictest0-21swapper/019:59:400
20557992121,0cyclictest0-21swapper/019:19:350
20557992120,0cyclictest14357-21apt-get21:10:010
2055799210,21cyclictest0-21swapper/021:24:380
2055799210,20cyclictest0-21swapper/022:59:360
2055799210,20cyclictest0-21swapper/022:39:370
2055799210,20cyclictest0-21swapper/020:24:370
2055799210,1cyclictest27567-21apt-get20:34:380
2055799210,1cyclictest0-21swapper/021:00:000
2055799210,0cyclictest0-21swapper/023:29:370
20567992020,0cyclictest20449-21apt-get23:39:353
20567992020,0cyclictest0-21swapper/323:59:383
20567992020,0cyclictest0-21swapper/321:54:373
20567992020,0cyclictest0-21swapper/320:19:383
20567992020,0cyclictest0-21swapper/300:09:353
20567992019,1cyclictest9184-21apt-get22:09:353
2056799200,20cyclictest0-21swapper/322:19:393
2056799200,1cyclictest4915-21turbostat19:45:003
2056799200,0cyclictest2354-21apt-get20:49:353
2056799200,0cyclictest0-21swapper/321:09:393
2056799200,0cyclictest0-21swapper/300:04:393
20564992020,0cyclictest6723-21perf00:14:592
20564992020,0cyclictest0-21swapper/223:39:362
20564992020,0cyclictest0-21swapper/222:49:352
20564992020,0cyclictest0-21swapper/221:44:402
20564992020,0cyclictest0-21swapper/220:44:352
20564992020,0cyclictest0-21swapper/219:29:362
2056499200,20cyclictest0-21swapper/219:54:422
2056499200,19cyclictest0-21swapper/223:19:352
20561992020,0cyclictest0-21swapper/123:39:361
20561992020,0cyclictest0-21swapper/123:14:361
20561992020,0cyclictest0-21swapper/122:54:351
20561992020,0cyclictest0-21swapper/122:19:381
20561992020,0cyclictest0-21swapper/120:44:351
20561992020,0cyclictest0-21swapper/119:57:541
20561992020,0cyclictest0-21swapper/119:24:361
20561992020,0cyclictest0-21swapper/100:19:361
20561992020,0cyclictest0-21swapper/100:04:381
20561992019,1cyclictest16317-21perf00:35:001
20561992019,1cyclictest0-21swapper/121:44:401
20561992019,0cyclictest0-21swapper/123:34:371
20561992019,0cyclictest0-21swapper/121:59:371
2056199200,20cyclictest0-21swapper/120:19:381
2056199200,0cyclictest0-21swapper/119:44:351
20557992020,0cyclictest27057-21perf23:50:000
20557992020,0cyclictest27057-21perf23:49:590
20557992020,0cyclictest0-21swapper/023:39:360
20557992020,0cyclictest0-21swapper/022:14:360
20557992020,0cyclictest0-21swapper/019:49:340
20557992020,0cyclictest0-21swapper/019:49:340
20557992020,0cyclictest0-21swapper/000:19:370
20557992017,2cyclictest32514-21dump-pmu-power19:35:000
2055799200,20cyclictest0-21swapper/022:19:380
2055799200,20cyclictest0-21swapper/021:55:000
2055799200,20cyclictest0-21swapper/020:09:370
20567991919,0cyclictest0-21swapper/323:09:363
20567991919,0cyclictest0-21swapper/321:19:373
20567991919,0cyclictest0-21swapper/319:19:353
20567991919,0cyclictest0-21swapper/300:19:353
20567991918,1cyclictest21450irq/143-nvme0q400:35:003
20567991918,1cyclictest0-21swapper/323:34:363
20567991918,0cyclictest0-21swapper/319:59:403
20564991919,0cyclictest24775-21apt-get20:29:382
20564991919,0cyclictest0-21swapper/222:04:362
20564991919,0cyclictest0-21swapper/219:24:362
20561991919,0cyclictest0-21swapper/122:44:371
20561991919,0cyclictest0-21swapper/122:04:371
20561991919,0cyclictest0-21swapper/121:24:391
20561991919,0cyclictest0-21swapper/121:19:371
20561991919,0cyclictest0-21swapper/120:29:381
20561991919,0cyclictest0-21swapper/100:29:351
20561991919,0cyclictest0-21swapper/100:14:361
20561991918,1cyclictest26796-21apt-get21:39:381
20557991919,0cyclictest0-21swapper/022:24:330
20557991919,0cyclictest0-21swapper/020:44:350
20557991919,0cyclictest0-21swapper/000:24:360
20557991918,0cyclictest27661-21apt-get23:54:350
2055799190,1cyclictest8798-21apt-get22:05:020
2055799190,1cyclictest4984-21cron23:04:590
20567991818,0cyclictest30828-21apt-get22:54:343
20567991818,0cyclictest29213-21apt-get21:44:403
20567991818,0cyclictest0-21swapper/322:44:373
20567991818,0cyclictest0-21swapper/321:24:393
20567991818,0cyclictest0-21swapper/300:29:353
20567991818,0cyclictest0-21swapper/300:14:363
20567991816,1cyclictest5800-21awk22:00:003
2056499182,1cyclictest26175-21apt21:35:012
20564991818,0cyclictest4754-21apt-get20:54:412
20564991818,0cyclictest13737-21perf21:10:002
20564991818,0cyclictest0-21swapper/223:44:382
20564991818,0cyclictest0-21swapper/223:29:362
20564991818,0cyclictest0-21swapper/221:54:362
20564991818,0cyclictest0-21swapper/221:29:352
20564991818,0cyclictest0-21swapper/220:14:372
20564991817,0cyclictest0-21swapper/223:10:022
20564991816,1cyclictest30058-21apt-get23:59:382
20561991817,1cyclictest0-21swapper/122:29:351
20561991817,1cyclictest0-21swapper/119:14:351
20557991818,0cyclictest0-21swapper/023:44:380
20557991818,0cyclictest0-21swapper/019:29:350
20567991717,0cyclictest16916-21apt-get00:39:363
20567991717,0cyclictest0-21swapper/321:39:393
20567991717,0cyclictest0-21swapper/320:29:383
20567991716,1cyclictest0-21swapper/319:14:353
2056799170,16cyclictest0-21swapper/320:24:383
20564991716,1cyclictest30735-21apt-get19:34:352
20564991715,1cyclictest0-21swapper/222:39:592
20564991715,1cyclictest0-21swapper/222:30:012
20561991717,0cyclictest0-21swapper/120:39:361
20561991717,0cyclictest0-21swapper/120:24:381
20561991717,0cyclictest0-21swapper/120:04:371
20561991717,0cyclictest0-21swapper/119:34:351
20561991715,1cyclictest15013-21munin-run23:25:001
2056199170,16cyclictest0-21swapper/123:49:361
2056199170,16cyclictest0-21swapper/123:49:361
20557991714,2cyclictest15011-21awk23:25:000
20557991713,3cyclictest12132-21apt-get00:29:350
20567991616,0cyclictest0-21swapper/322:29:363
20567991616,0cyclictest0-21swapper/319:34:343
20567991615,0cyclictest0-21swapper/320:04:373
20567991614,1cyclictest27178-21apt-get20:30:023
20567991614,1cyclictest0-21swapper/323:25:003
2056799160,15cyclictest7324-21munin-run19:50:003
2056799160,15cyclictest7324-21munin-run19:50:003
2056799160,15cyclictest18323-21munin-run22:25:013
20564991616,0cyclictest13974-21apt-get22:19:382
20564991616,0cyclictest0-21swapper/219:39:352
20564991615,1cyclictest13217-21apt-get23:24:352
20564991615,0cyclictest22384-21apt-get20:24:382
2056499160,15cyclictest782-21sshd21:48:002
20561991616,0cyclictest0-21swapper/122:14:361
20561991616,0cyclictest0-21swapper/120:30:011
2056199160,1cyclictest0-21swapper/122:25:011
2055799162,13cyclictest28433-21apt-get22:49:350
20557991616,0cyclictest0-21swapper/023:09:360
20557991616,0cyclictest0-21swapper/021:09:390
20557991616,0cyclictest0-21swapper/020:29:370
20557991616,0cyclictest0-21swapper/020:04:360
20557991616,0cyclictest0-21swapper/000:09:350
20557991616,0cyclictest0-21swapper/000:04:390
20557991614,1cyclictest27722-21turbostat.cron19:25:000
2055799160,8cyclictest0-21swapper/021:49:380
2055799160,1cyclictest12584-21perf23:20:000
2055799160,15cyclictest19777-21users23:34:470
20567991515,0cyclictest0-21swapper/322:14:363
20567991515,0cyclictest0-21swapper/320:14:363
20567991514,1cyclictest7927-21apt-get19:54:423
20567991514,1cyclictest22876-21apt-get23:44:383
20564991514,0cyclictest9741-21apt-get00:24:362
20564991513,1cyclictest14526-21apt-get00:34:382
2056499150,1cyclictest0-21swapper/220:49:372
2056499150,14cyclictest0-21swapper/222:54:452
2056199159,6cyclictest1686-21apache223:25:411
20561991515,0cyclictest0-21swapper/123:44:381
20561991515,0cyclictest0-21swapper/123:09:351
20561991515,0cyclictest0-21swapper/120:59:401
20561991515,0cyclictest0-21swapper/119:54:411
20561991513,1cyclictest13715-21idleruntime-cro21:10:001
20557991515,0cyclictest26043-21apt-get22:44:360
20557991515,0cyclictest0-21swapper/022:54:350
20557991515,0cyclictest0-21swapper/021:44:400
20567991414,0cyclictest0-21swapper/323:45:053
20567991414,0cyclictest0-21swapper/323:45:053
20567991414,0cyclictest0-21swapper/300:20:083
2056799140,0cyclictest0-21swapper/323:25:163
2056799140,0cyclictest0-21swapper/321:32:173
2056799140,0cyclictest0-21swapper/320:50:343
2056499140,14cyclictest0-21swapper/219:10:332
2056499140,14cyclictest0-21swapper/200:26:552
2056499140,0cyclictest0-21swapper/222:10:012
2056499140,0cyclictest0-21swapper/219:46:572
2056499140,0cyclictest0-21swapper/219:46:572
2056499140,0cyclictest0-21swapper/219:15:302
20561991414,0cyclictest0-21swapper/123:16:401
2056199140,14cyclictest0-21swapper/100:20:091
2056199140,0cyclictest0-21swapper/122:30:301
2056199140,0cyclictest0-21swapper/121:30:511
2056199140,0cyclictest0-21swapper/120:52:281
20557991414,0cyclictest0-21swapper/020:50:020
20557991414,0cyclictest0-21swapper/000:30:220
2055799140,14cyclictest0-21swapper/022:28:340
2055799140,14cyclictest0-21swapper/020:10:260
2055799140,0cyclictest0-21swapper/021:35:040
2055799140,0cyclictest0-21swapper/021:25:500
2055799140,0cyclictest0-21swapper/000:36:350
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional