You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 12:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Thu Dec 04, 2025 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12303993635,1cyclictest22740-21lspci19:30:302
1230199360,35cyclictest0-21swapper/019:30:290
1230499250,24cyclictest15382-21apt-get23:43:123
1230199250,24cyclictest14888-21idleruntime-cro23:40:010
12302992421,2cyclictest5966-21turbostat.cron22:15:001
1230199240,2cyclictest25962-21awk21:50:000
1230199240,23cyclictest11515-21idleruntime-cro21:20:010
12304992323,0cyclictest19209-21apt-get21:38:113
12304992321,1cyclictest4845-21cron23:19:593
12304992320,2cyclictest17105-21perf19:20:003
1230499230,22cyclictest612-21gdbus20:45:003
12303992321,1cyclictest4306-21munin-run21:05:002
12303992320,2cyclictest24727-21apache200:00:002
12303992320,2cyclictest11508-21perf21:20:012
1230399230,22cyclictest25960-21sh21:50:002
1230299230,22cyclictest27633-21apt23:00:001
1230199230,2cyclictest22239-21awk20:35:010
12304992220,1cyclictest21909-21munin-run19:30:003
12304992220,1cyclictest0-21swapper/323:25:003
1230499220,2cyclictest9372-21awk00:35:003
1230499220,21cyclictest593-21dbus-daemon21:20:013
1230499220,21cyclictest15002-21latency_hist20:20:003
1230499220,21cyclictest10785-21grep22:25:003
1230499220,21cyclictest0-21swapper/323:25:033
1230499220,21cyclictest0-21swapper/320:58:083
1230499220,21cyclictest0-21swapper/320:48:103
1230499220,1cyclictest22575-21apt-get23:58:113
12303992222,0cyclictest0-21swapper/222:55:002
12303992222,0cyclictest0-21swapper/219:45:012
12303992221,1cyclictest14885-21grep23:40:012
1230399220,21cyclictest22725-21apt-get20:38:122
1230399220,21cyclictest2182-21munin-run00:20:002
1230399220,21cyclictest0-21swapper/221:28:122
1230299220,21cyclictest9592-21apt-get21:18:101
1230299220,21cyclictest27390-21idleruntime-cro00:05:001
1230299220,21cyclictest0-21swapper/123:08:121
1230299220,21cyclictest0-21swapper/120:03:111
1230299220,21cyclictest0-21swapper/119:45:021
1230299220,1cyclictest0-21swapper/123:20:001
12301992220,1cyclictest598-21rs:main0
1230199220,2cyclictest1876-21turbostat21:00:000
1230199220,21cyclictest2393-21apt-get21:03:100
1230199220,21cyclictest0-21swapper/022:43:130
1230199220,21cyclictest0-21swapper/021:33:110
1230199220,21cyclictest0-21swapper/019:53:110
1230199220,21cyclictest0-21swapper/019:23:100
1230199220,1cyclictest25476-21apt-get00:03:100
1230199220,1cyclictest24765-21apache200:00:000
1230199220,1cyclictest0-21swapper/021:30:000
1230199220,0cyclictest0-21swapper/023:23:110
12304992121,0cyclictest28218-21apt-get19:43:103
12304992120,1cyclictest16090-21apt-get22:38:143
12304992119,1cyclictest30052-21latency_hist23:05:003
1230499210,21cyclictest19516-21perf19:25:003
1230499210,21cyclictest0-21swapper/321:03:103
1230499210,20cyclictest16313-21latency_hist21:30:003
1230499210,20cyclictest0-21swapper/319:53:113
12303992121,0cyclictest17925-21apt-get20:28:112
12303992120,1cyclictest27719-21perf19:40:012
12303992120,1cyclictest21625-21apt-get21:43:102
12303992120,0cyclictest6469-21apt-get22:18:122
12303992120,0cyclictest27878-21apt-get00:08:142
1230399210,20cyclictest5071-21apt-get00:28:112
1230399210,20cyclictest18718-21perf21:35:002
1230399210,20cyclictest0-21swapper/223:03:102
1230399210,20cyclictest0-21swapper/220:53:142
1230399210,20cyclictest0-21swapper/219:58:092
1230399210,1cyclictest12011-21apt-get21:23:122
1230399210,0cyclictest1668-21apt-get22:08:112
12302992121,0cyclictest0-21swapper/122:48:141
12302992121,0cyclictest0-21swapper/100:28:111
12302992120,1cyclictest25980-21turbostat21:50:001
12302992119,1cyclictest313-21systemd-journal20:35:011
12302992119,1cyclictest24489-40mandb00:00:001
1230299210,21cyclictest0-21swapper/121:38:111
1230299210,20cyclictest16090-21apt-get22:38:141
12301992120,0cyclictest12789-21apt-get19:13:130
12301992120,0cyclictest0-21swapper/022:03:120
12301992120,0cyclictest0-21swapper/019:43:100
12301992119,1cyclictest0-21swapper/021:53:120
12301992118,2cyclictest25251-21sh22:55:010
1230199210,21cyclictest0-21swapper/020:53:130
1230199210,20cyclictest0-21swapper/022:58:110
1230199210,20cyclictest0-21swapper/019:38:120
1230199210,0cyclictest0-21swapper/021:43:110
1230199210,0cyclictest0-21swapper/020:23:110
12304992020,0cyclictest5389-21perf20:00:003
12304992020,0cyclictest31011-21apt-get19:48:113
12304992020,0cyclictest13687-21apt-get22:33:103
12304992020,0cyclictest0-21swapper/323:38:123
12304992020,0cyclictest0-21swapper/322:43:133
12304992020,0cyclictest0-21swapper/322:28:103
12304992020,0cyclictest0-21swapper/322:13:113
12304992020,0cyclictest0-21swapper/321:43:113
12304992019,0cyclictest17782-21apt-get23:48:113
12304992017,2cyclictest313-21systemd-journal21:50:003
1230499200,20cyclictest0-21swapper/320:08:113
1230499200,0cyclictest26871-21apt-get21:53:113
1230499200,0cyclictest0-21swapper/323:53:133
12303992020,0cyclictest31685-21apt-get22:03:122
12303992020,0cyclictest0-21swapper/223:33:122
12303992020,0cyclictest0-21swapper/223:13:102
12303992020,0cyclictest0-21swapper/222:58:102
12303992020,0cyclictest0-21swapper/222:38:142
12303992020,0cyclictest0-21swapper/221:38:102
12303992020,0cyclictest0-21swapper/219:13:132
12303992019,1cyclictest1875-21idleruntime-cro21:00:002
12303992019,0cyclictest2949-21apt-get23:18:102
1230399200,0cyclictest30564-21apt-get23:08:122
1230399200,0cyclictest17602-21apt-get19:23:102
1230399200,0cyclictest15188-21apt-get19:18:132
1230399200,0cyclictest0-21swapper/223:43:122
1230399200,0cyclictest0-21swapper/222:33:102
12302992020,0cyclictest0-21swapper/122:08:111
12302992020,0cyclictest0-21swapper/122:03:121
12302992020,0cyclictest0-21swapper/121:15:001
12302992020,0cyclictest0-21swapper/120:08:111
12302992020,0cyclictest0-21swapper/119:43:101
12302992020,0cyclictest0-21swapper/100:33:121
12302992020,0cyclictest0-21swapper/100:08:141
12302992019,0cyclictest0-21swapper/121:08:121
12302992019,0cyclictest0-21swapper/120:58:091
1230299200,20cyclictest0-21swapper/123:13:101
1230299200,20cyclictest0-21swapper/122:53:111
1230299200,20cyclictest0-21swapper/121:03:101
1230299200,20cyclictest0-21swapper/120:38:131
1230299200,0cyclictest0-21swapper/121:58:121
1230299200,0cyclictest0-21swapper/120:48:101
1230299200,0cyclictest0-21swapper/120:13:101
1230299200,0cyclictest0-21swapper/119:53:111
12301992020,0cyclictest0-21swapper/023:18:110
12301992020,0cyclictest0-21swapper/023:13:100
12301992020,0cyclictest0-21swapper/022:48:140
12301992020,0cyclictest0-21swapper/022:33:110
12301992020,0cyclictest0-21swapper/022:28:100
12301992020,0cyclictest0-21swapper/022:23:120
12301992020,0cyclictest0-21swapper/022:13:100
12301992020,0cyclictest0-21swapper/020:18:090
12301992020,0cyclictest0-21swapper/019:45:020
12301992020,0cyclictest0-21swapper/019:18:130
12301992019,0cyclictest0-21swapper/021:38:110
12301992018,1cyclictest27076-21cron20:45:000
1230199200,20cyclictest0-21swapper/023:48:120
1230199200,0cyclictest32667-21apt-get00:18:100
1230199200,0cyclictest0-21swapper/023:08:110
1230199200,0cyclictest0-21swapper/020:38:130
1230199200,0cyclictest0-21swapper/020:03:110
12304991919,0cyclictest23352-21apt-get19:33:123
12304991919,0cyclictest0-21swapper/322:18:123
12304991919,0cyclictest0-21swapper/321:13:123
12304991919,0cyclictest0-21swapper/319:13:133
12304991918,0cyclictest20904-21apt-get22:48:143
1230499190,0cyclictest0-21swapper/323:33:123
12303991919,0cyclictest8311-21apt-get20:08:112
12303991919,0cyclictest15529-21apt-get20:23:112
12303991919,0cyclictest0-21swapper/223:48:112
12303991919,0cyclictest0-21swapper/223:23:112
12303991919,0cyclictest0-21swapper/222:28:102
12303991919,0cyclictest0-21swapper/221:08:122
12303991919,0cyclictest0-21swapper/219:48:112
12303991919,0cyclictest0-21swapper/200:03:102
12303991918,1cyclictest0-21swapper/222:48:142
12302991919,0cyclictest10562-21apt-get23:33:121
12302991919,0cyclictest0-21swapper/123:43:121
12302991919,0cyclictest0-21swapper/122:43:131
12302991919,0cyclictest0-21swapper/119:13:121
12302991919,0cyclictest0-21swapper/100:38:141
12302991917,1cyclictest32165-21dump-pmu-power00:15:001
12302991917,1cyclictest19999-21apt-get19:28:111
12301991919,0cyclictest6971-21perf00:30:010
12301991918,0cyclictest0-21swapper/023:03:110
12301991917,1cyclictest8392-21perf22:19:590
12304991818,0cyclictest0-21swapper/322:53:113
12304991818,0cyclictest0-21swapper/322:03:113
12304991818,0cyclictest0-21swapper/320:33:123
12304991816,1cyclictest2164-21perf00:20:003
12304991816,1cyclictest18719-21munin-run21:35:003
12303991818,0cyclictest5884-21apt-get20:03:112
12303991818,0cyclictest0-21swapper/222:43:132
12303991817,1cyclictest5938-21cron22:15:002
12303991816,1cyclictest21888-21grep19:30:002
12302991818,0cyclictest0-21swapper/123:28:121
12302991816,1cyclictest8385-21awk22:20:001
12302991815,2cyclictest19488-21idleruntime-cro19:25:001
12301991818,0cyclictest0-21swapper/020:28:120
12301991816,1cyclictest6663-21cron21:10:000
12304991717,0cyclictest0-21swapper/322:58:123
12304991717,0cyclictest0-21swapper/322:08:113
12304991717,0cyclictest0-21swapper/321:23:123
12304991717,0cyclictest0-21swapper/321:08:123
12304991717,0cyclictest0-21swapper/300:13:143
12303991717,0cyclictest1027-21apt-get19:53:112
12303991717,0cyclictest0-21swapper/221:58:112
12303991716,0cyclictest8889-21apt-get22:23:122
1230399170,1cyclictest10064-21perf23:30:002
12302991717,0cyclictest0-21swapper/123:48:101
12302991717,0cyclictest0-21swapper/119:33:121
12302991717,0cyclictest0-21swapper/119:18:131
12301991717,0cyclictest27589-21apt-get20:48:100
12301991717,0cyclictest0-21swapper/020:13:100
12301991717,0cyclictest0-21swapper/020:08:110
12301991717,0cyclictest0-21swapper/019:58:090
12301991715,1cyclictest11761-21perf00:40:000
12301991715,1cyclictest0-21swapper/022:40:000
1230199170,16cyclictest0-21swapper/023:54:470
12304991616,0cyclictest29990-21apt-get20:53:143
12304991616,0cyclictest0-21swapper/323:08:123
12304991616,0cyclictest0-21swapper/320:38:133
12304991616,0cyclictest0-21swapper/300:23:123
12304991615,1cyclictest0-21swapper/320:28:113
12304991614,1cyclictest11803-21latency_hist00:40:003
1230499160,15cyclictest960-21diskstats23:13:113
12303991616,0cyclictest20323-21apt-get20:33:112
1230399160,0cyclictest10704-21apt-get20:13:102
12302991616,0cyclictest0-21swapper/119:58:081
1230299160,0cyclictest0-21swapper/123:38:121
12301991615,0cyclictest0-21swapper/023:43:110
12301991614,1cyclictest9880-21users23:28:210
12304991515,0cyclictest0-21swapper/319:40:013
12304991515,0cyclictest0-21swapper/300:28:113
12304991513,1cyclictest277022sleep300:06:223
12303991515,0cyclictest13111-21apt-get20:18:102
12303991515,0cyclictest0-21swapper/221:53:122
12303991515,0cyclictest0-21swapper/220:43:092
12302991515,0cyclictest0-21swapper/121:33:111
12302991515,0cyclictest0-21swapper/120:23:101
12302991515,0cyclictest0-21swapper/119:38:121
12302991515,0cyclictest0-21swapper/100:23:121
12302991513,1cyclictest22808-21grep21:43:171
12302991513,1cyclictest0-21swapper/123:03:131
1230299150,15cyclictest0-21swapper/121:23:181
1230299150,14cyclictest0-21swapper/123:24:501
12301991515,0cyclictest0-21swapper/023:33:110
12301991515,0cyclictest0-21swapper/000:23:120
12301991515,0cyclictest0-21swapper/000:08:130
1230499140,14cyclictest0-21swapper/321:57:503
1230499140,14cyclictest0-21swapper/320:10:133
1230499140,14cyclictest0-21swapper/320:02:083
1230499140,14cyclictest0-21swapper/300:00:333
1230499140,0cyclictest0-21swapper/320:22:463
1230399140,14cyclictest0-21swapper/221:13:442
1230399140,14cyclictest0-21swapper/200:36:292
1230399140,14cyclictest0-21swapper/200:20:392
1230399140,14cyclictest0-21swapper/200:10:442
1230399140,0cyclictest0-21swapper/223:53:312
1230399140,0cyclictest0-21swapper/220:48:462
1230399140,0cyclictest0-21swapper/200:31:422
12302991414,0cyclictest0-21swapper/121:28:111
1230299140,14cyclictest0-21swapper/123:50:021
1230299140,14cyclictest0-21swapper/121:50:221
1230299140,14cyclictest0-21swapper/120:41:361
1230299140,14cyclictest0-21swapper/100:15:351
1230299140,0cyclictest0-21swapper/122:32:441
1230299140,0cyclictest0-21swapper/120:51:531
1230299140,0cyclictest0-21swapper/120:25:311
1230199140,14cyclictest0-21swapper/022:05:100
1230199140,14cyclictest0-21swapper/021:13:160
1230199140,14cyclictest0-21swapper/000:33:220
1230199140,0cyclictest0-21swapper/021:23:040
1230199140,0cyclictest0-21swapper/000:10:510
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional