You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 13:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Tue Nov 25, 2025 00:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29228992623,2cyclictest2192-21cron22:40:002
29226992621,2cyclictest25732-21grep21:15:000
29228992523,1cyclictest11819-21cron23:00:002
2922899250,24cyclictest0-21swapper/200:30:002
2922799250,1cyclictest0-21swapper/119:25:001
29230992422,1cyclictest0-21swapper/322:05:003
29230992422,1cyclictest0-21swapper/322:05:003
2923099240,1cyclictest0-21swapper/322:47:243
29226992421,2cyclictest15817-21chrt22:02:250
29226992421,2cyclictest15817-21chrt22:02:240
2922699240,23cyclictest9239-21apt-get19:37:250
2923099230,22cyclictest23352-21perf21:09:593
2922899230,2cyclictest313-21systemd-journal19:35:002
2922899230,2cyclictest313-21systemd-journal19:35:002
2922899230,22cyclictest28504-21cron21:20:002
2922899230,21cyclictest313-21systemd-journal00:35:002
29227992323,0cyclictest0-21swapper/121:17:251
29227992321,1cyclictest32219-21idleruntime-cro22:35:001
2922799230,2cyclictest11261-21awk20:44:591
2922799230,22cyclictest7472-21apt-get22:52:231
29226992321,1cyclictest3986-21awk19:25:000
29226992321,1cyclictest0-21swapper/020:45:000
29230992220,1cyclictest29840-21perf22:30:003
29230992220,1cyclictest0-21swapper/300:20:003
29230992219,2cyclictest31602-21apt19:14:593
2923099220,21cyclictest9321-21apt-get20:42:233
2923099220,21cyclictest593-21dbus-daemon22:55:003
2923099220,21cyclictest0-21swapper/300:27:253
2923099220,21cyclictest0-21swapper/300:02:253
2923099220,1cyclictest4356-21apt-get23:52:233
29228992222,0cyclictest0-21swapper/223:37:232
29228992220,1cyclictest18417-21cron00:20:002
2922899220,21cyclictest6203-21apt-get21:42:252
2922899220,21cyclictest11213-21perf00:05:002
2922899220,21cyclictest0-21swapper/220:20:002
2922899220,1cyclictest13394-21apt-get21:57:242
29227992221,0cyclictest3783-21apt-get21:37:231
29227992221,0cyclictest0-21swapper/123:12:231
29227992220,1cyclictest15369-21perf22:00:001
29227992219,2cyclictest6450-21cron20:35:011
2922799220,2cyclictest2235-21latency_hist22:40:001
2922799220,21cyclictest26068-21grep20:10:001
2922799220,1cyclictest313-21systemd-journal00:34:591
2922799220,1cyclictest0-21swapper/100:22:251
2922799220,0cyclictest0-21swapper/121:07:241
29226992220,1cyclictest21486-21apt-get23:20:010
2922699220,21cyclictest7020-21idleruntime-cro22:49:590
2922699220,21cyclictest25604-21awk00:35:000
2922699220,21cyclictest20231-21perf22:10:000
2922699220,21cyclictest0-21swapper/021:17:240
2922699220,21cyclictest0-21swapper/020:52:250
2922699220,1cyclictest3319-21cron21:35:000
2922699220,1cyclictest30846-21cron20:20:000
2922699220,1cyclictest0-21swapper/022:30:000
29230992121,0cyclictest16514-21apt-get20:57:243
29230992119,1cyclictest0-21swapper/319:45:013
2923099210,20cyclictest23663-21apt-get00:32:253
2923099210,20cyclictest2106-21apt-get20:27:223
2923099210,20cyclictest16004-21apt-get19:50:023
2923099210,20cyclictest0-21swapper/323:22:233
2923099210,20cyclictest0-21swapper/300:22:253
2923099210,0cyclictest0-21swapper/322:42:243
2923099210,0cyclictest0-21swapper/321:52:233
29228992121,0cyclictest0-21swapper/222:32:232
29228992119,1cyclictest20932-21idleruntime-cro21:05:002
29228992119,1cyclictest0-21swapper/223:29:592
2922899210,21cyclictest0-21swapper/222:07:252
2922899210,20cyclictest25471-21apt-get22:22:212
2922899210,20cyclictest0-21swapper/223:52:242
2922899210,0cyclictest14116-21apt-get20:52:242
2922899210,0cyclictest0-21swapper/222:17:252
29227992121,0cyclictest6756-21apt-get23:57:251
29227992120,1cyclictest10993-21apt-get21:52:241
29227992120,0cyclictest0-21swapper/100:37:261
2922799210,21cyclictest0-21swapper/120:25:031
2922799210,21cyclictest0-21swapper/119:37:241
2922799210,20cyclictest9263-21apt-get00:02:261
2922799210,20cyclictest0-21swapper/119:17:251
2922799210,0cyclictest0-21swapper/100:07:261
29226992121,0cyclictest0-21swapper/023:57:250
29226992119,1cyclictest9444-21munin-run22:54:590
29226992119,1cyclictest0-21swapper/019:32:220
29226992119,1cyclictest0-21swapper/019:32:210
2922699210,21cyclictest0-21swapper/023:32:260
2922699210,21cyclictest0-21swapper/023:27:250
2922699210,21cyclictest0-21swapper/000:40:000
2922699210,20cyclictest0-21swapper/019:27:230
2922699210,0cyclictest0-21swapper/021:52:240
29230992020,0cyclictest0-21swapper/323:17:253
29230992020,0cyclictest0-21swapper/321:32:233
29230992020,0cyclictest0-21swapper/319:37:253
29230992020,0cyclictest0-21swapper/319:32:233
29230992020,0cyclictest0-21swapper/319:32:233
29230992020,0cyclictest0-21swapper/300:37:263
29230992019,1cyclictest6903-21apt-get20:37:233
29230992018,1cyclictest13603-21awk00:10:003
2923099200,0cyclictest0-21swapper/323:32:263
2923099200,0cyclictest0-21swapper/321:17:253
29228992020,0cyclictest4442-21apt-get19:27:242
29228992020,0cyclictest23779-21apt-get21:12:252
29228992020,0cyclictest0-21swapper/223:47:252
29228992020,0cyclictest0-21swapper/222:52:232
29228992020,0cyclictest0-21swapper/222:42:242
29228992020,0cyclictest0-21swapper/221:27:252
29228992019,1cyclictest9239-21apt-get19:37:252
29228992018,1cyclictest3353-21turbostat.cron21:35:002
29228992018,1cyclictest23337-21wc21:10:002
29228992017,2cyclictest10565-21latency_hist21:50:002
2922899200,20cyclictest0-21swapper/219:12:232
2922899200,0cyclictest28991-21apt-get21:22:232
2922899200,0cyclictest0-21swapper/223:42:232
2922899200,0cyclictest0-21swapper/223:12:242
2922899200,0cyclictest0-21swapper/222:12:252
2922899200,0cyclictest0-21swapper/220:32:242
2922899200,0cyclictest0-21swapper/200:12:242
29227992020,0cyclictest0-21swapper/122:12:241
29227992020,0cyclictest0-21swapper/119:35:001
29227992020,0cyclictest0-21swapper/119:34:591
29227992020,0cyclictest0-21swapper/100:12:241
29227992019,1cyclictest31968-21apt-get23:42:241
29227992019,0cyclictest23057-21apt-get22:17:251
29227992019,0cyclictest22348-21apt-get23:22:221
29227992019,0cyclictest1372-21apt-get21:32:211
2922799200,20cyclictest0-21swapper/123:37:241
2922799200,20cyclictest0-21swapper/119:57:231
29226992020,0cyclictest0-21swapper/020:57:240
29226992020,0cyclictest0-21swapper/020:25:020
29226992020,0cyclictest0-21swapper/020:22:250
29226992020,0cyclictest0-21swapper/019:57:210
29226992020,0cyclictest0-21swapper/000:27:250
29226992019,1cyclictest0-21swapper/021:07:240
29226992019,0cyclictest9873-21apt-get22:57:230
2922699200,20cyclictest0-21swapper/021:02:230
29230991919,0cyclictest20659-21apt-get22:12:253
29230991919,0cyclictest0-21swapper/323:57:253
29230991919,0cyclictest0-21swapper/322:57:243
29230991918,1cyclictest0-21swapper/320:22:253
29230991917,1cyclictest26694-21grep23:30:003
29228991919,0cyclictest0-21swapper/223:22:242
29228991919,0cyclictest0-21swapper/222:47:242
29228991919,0cyclictest0-21swapper/222:02:252
29228991919,0cyclictest0-21swapper/222:02:252
29228991919,0cyclictest0-21swapper/220:20:022
29228991919,0cyclictest0-21swapper/220:02:252
29227991919,0cyclictest27869-21apt-get22:27:241
29227991919,0cyclictest0-21swapper/123:52:241
29227991919,0cyclictest0-21swapper/121:27:231
29227991919,0cyclictest0-21swapper/121:12:241
29227991919,0cyclictest0-21swapper/120:52:251
29227991919,0cyclictest0-21swapper/119:52:241
29227991919,0cyclictest0-21swapper/119:42:231
29227991918,0cyclictest0-21swapper/120:57:241
29226991919,0cyclictest0-21swapper/023:12:240
29226991919,0cyclictest0-21swapper/022:42:240
29226991919,0cyclictest0-21swapper/019:42:240
29226991919,0cyclictest0-21swapper/000:07:260
29226991918,0cyclictest0-21swapper/019:50:020
29226991917,1cyclictest583-21avahi-daemon20:10:000
29226991916,2cyclictest313-21systemd-journal22:35:000
29230991818,0cyclictest17084-21apt-get23:12:243
29230991818,0cyclictest0-21swapper/323:47:253
29230991818,0cyclictest0-21swapper/322:22:223
29230991818,0cyclictest0-21swapper/321:57:243
29230991818,0cyclictest0-21swapper/321:22:233
29230991817,0cyclictest0-21swapper/323:37:233
29230991816,1cyclictest25740-21awk21:15:003
29228991818,0cyclictest0-21swapper/220:47:252
29228991818,0cyclictest0-21swapper/219:52:222
29227991818,0cyclictest0-21swapper/123:02:231
29227991818,0cyclictest0-21swapper/122:42:241
29227991818,0cyclictest0-21swapper/121:47:251
29227991818,0cyclictest0-21swapper/120:37:231
29227991818,0cyclictest0-21swapper/120:20:021
29227991818,0cyclictest0-21swapper/119:27:241
29227991817,0cyclictest0-21swapper/123:27:251
29227991816,1cyclictest21486-21apt-get23:19:591
29226991818,0cyclictest0-21swapper/022:12:260
29226991818,0cyclictest0-21swapper/021:42:240
29226991818,0cyclictest0-21swapper/000:22:240
29226991818,0cyclictest0-21swapper/000:17:230
29226991814,3cyclictest31395-21apt-get21:27:240
29230991717,0cyclictest4508-21apt-get20:32:243
29230991717,0cyclictest0-21swapper/323:07:253
29230991717,0cyclictest0-21swapper/321:37:233
29230991717,0cyclictest0-21swapper/321:27:243
29230991717,0cyclictest0-21swapper/320:07:253
2922899176,11cyclictest28-21ksoftirqd/222:27:232
29228991717,0cyclictest0-21swapper/220:27:232
29228991716,1cyclictest19539-21apt-get23:17:252
29228991715,1cyclictest0-21swapper/220:57:242
29227991717,0cyclictest0-21swapper/121:22:241
29227991717,0cyclictest0-21swapper/121:02:241
29227991715,1cyclictest0-21swapper/119:47:231
29227991714,2cyclictest22416-21cut00:27:301
29226991717,0cyclictest0-21swapper/019:13:370
29226991717,0cyclictest0-21swapper/000:02:260
29226991716,1cyclictest29551-21apt-get23:37:240
2922699170,16cyclictest1962-21apt-get23:47:260
29230991616,0cyclictest18209-21apt-get22:07:253
29230991616,0cyclictest0-21swapper/322:17:253
29230991616,0cyclictest0-21swapper/320:52:253
29230991616,0cyclictest0-21swapper/320:47:263
29230991616,0cyclictest0-21swapper/300:12:243
29230991615,0cyclictest0-21swapper/319:17:253
29230991614,1cyclictest781-21lldpd22:36:323
29230991614,1cyclictest12289-21apt-get23:02:233
29230991614,1cyclictest0-21swapper/319:47:273
2923099160,15cyclictest22138-21df_inode20:02:253
29228991616,0cyclictest24116-21apt-get20:07:262
29228991616,0cyclictest0-21swapper/221:52:242
29228991616,0cyclictest0-21swapper/220:42:242
29228991616,0cyclictest0-21swapper/200:07:262
29228991615,1cyclictest26060-21apt-get00:37:252
29228991614,1cyclictest0-21swapper/220:40:012
29228991613,2cyclictest20818-21grep00:25:002
29227991616,0cyclictest0-21swapper/123:47:261
29227991616,0cyclictest0-21swapper/123:32:261
29227991616,0cyclictest0-21swapper/123:07:241
29227991614,1cyclictest0-21swapper/123:00:001
2922799160,1cyclictest31588-21latency_hist19:15:001
2922699163,12cyclictest578-21irqbalance23:50:250
29226991616,0cyclictest0-21swapper/020:12:250
29226991614,1cyclictest15368-21sendmail-msp22:00:000
2922699160,7cyclictest0-21swapper/019:17:250
2922699160,15cyclictest0-21swapper/023:02:230
29230991515,0cyclictest0-21swapper/322:32:223
29230991514,1cyclictest26510-21apt-get20:12:253
29230991514,1cyclictest18931-21apt-get21:02:233
29228991515,0cyclictest0-21swapper/223:02:232
29228991515,0cyclictest0-21swapper/219:22:272
29228991515,0cyclictest0-21swapper/219:17:262
29228991514,0cyclictest0-21swapper/223:07:242
29228991513,2cyclictest19247-21apt-get19:57:232
2922899150,14cyclictest0-21swapper/219:42:252
29227991515,0cyclictest0-21swapper/122:22:221
29227991515,0cyclictest0-21swapper/122:07:241
29227991515,0cyclictest0-21swapper/122:02:251
29227991515,0cyclictest0-21swapper/122:02:251
29226991515,0cyclictest0-21swapper/021:37:230
29226991515,0cyclictest0-21swapper/000:12:230
29226991514,1cyclictest11722-21apt-get20:47:260
29226991513,1cyclictest6453-21awk20:35:000
2923099140,0cyclictest0-21swapper/323:40:013
2923099140,0cyclictest0-21swapper/321:46:103
2923099140,0cyclictest0-21swapper/321:44:453
2923099140,0cyclictest0-21swapper/320:15:033
2923099140,0cyclictest0-21swapper/319:28:333
2923099140,0cyclictest0-21swapper/319:21:063
29228991414,0cyclictest0-21swapper/220:10:012
2922899140,14cyclictest0-21swapper/223:32:132
2922899140,14cyclictest0-21swapper/219:47:162
2922899140,0cyclictest0-21swapper/223:58:372
29227991414,0cyclictest0-21swapper/120:11:511
2922799140,14cyclictest0-21swapper/122:45:321
2922799140,14cyclictest0-21swapper/120:18:391
2922799140,0cyclictest0-21swapper/100:15:121
29226991414,0cyclictest0-21swapper/023:40:210
29226991414,0cyclictest0-21swapper/023:05:440
29226991414,0cyclictest0-21swapper/022:15:050
2922699140,14cyclictest0-21swapper/022:21:260
2922699140,14cyclictest0-21swapper/020:35:350
2922699140,14cyclictest0-21swapper/019:45:040
2922699140,13cyclictest0-21swapper/021:22:240
2922699140,0cyclictest0-21swapper/021:47:010
29230991313,0cyclictest0-21swapper/319:55:183
29228991313,0cyclictest0-21swapper/221:36:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional