You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-10 - 06:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot4s.osadl.org (updated Thu Jul 10, 2025 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2881222228191,24sleep20-21swapper/219:07:592
2881812227170,45sleep50-21swapper/519:08:525
2881652226188,24sleep70-21swapper/719:08:377
2880972222164,17sleep40-21swapper/419:07:384
2879932218184,21sleep30-21swapper/319:06:083
2882042213179,22sleep10-21swapper/119:09:121
2880792209175,21sleep60-21swapper/619:07:246
2879992203176,17sleep00-21swapper/019:06:130
4489242650,0sleep7851rcuc/723:45:187
28849999280,27cyclictest444949-21turbostat23:40:005
28850599270,22cyclictest325284-21ntpq20:10:187
28850599270,21cyclictest0-21swapper/700:40:007
288499992620,5cyclictest0-21swapper/521:30:005
28849999260,24cyclictest0-21swapper/519:20:125
288486992624,1cyclictest361851-21apt21:15:000
288499992523,1cyclictest428237-21apt-get23:10:115
28849999250,1cyclictest342044-21apt-get20:40:125
28848899250,25cyclictest361855-21dump-pmu-power21:15:001
28850399240,2cyclictest344490-21basename20:45:006
28850399240,23cyclictest320808-21awk20:05:006
28850399240,22cyclictest368003-21apt-get21:25:126
28849999240,24cyclictest0-21swapper/520:14:595
28849999240,1cyclictest407980-21apt-get22:35:115
288492992421,3cyclictest335659-21basename20:30:002
28848899240,23cyclictest479285-21apt-get00:40:001
28848899240,1cyclictest0-21swapper/123:45:161
288505992321,1cyclictest0-21swapper/720:00:007
288505992320,3cyclictest0-21swapper/722:45:137
288505992319,3cyclictest388007-21apt-get22:00:167
28850599230,22cyclictest425128-21apt-get23:05:137
28850599230,22cyclictest0-21swapper/723:55:137
28850599230,22cyclictest0-21swapper/721:35:007
288503992322,1cyclictest422276-21apt-get23:00:126
28850399230,22cyclictest0-21swapper/622:50:136
28850399230,0cyclictest0-21swapper/623:15:166
28849999230,22cyclictest0-21swapper/519:45:135
28849999230,22cyclictest0-21swapper/500:35:135
28849999230,20cyclictest0-21swapper/521:45:115
288497992321,1cyclictest456206-21fwupd23:57:544
288497992321,1cyclictest0-21swapper/422:15:014
288494992323,0cyclictest0-21swapper/322:55:163
288494992322,1cyclictest0-21swapper/319:50:003
288494992321,2cyclictest433963-21apt-get23:20:113
28849299230,22cyclictest427551-21apt-get23:10:022
28849299230,1cyclictest442100-21munin-run23:35:002
28849299230,1cyclictest393704-21apt-get22:10:132
288488992321,1cyclictest327173-21apt-get20:15:151
288488992321,1cyclictest0-21swapper/120:10:121
28848899230,22cyclictest0-21swapper/121:45:181
288486992320,2cyclictest315075-21awk19:55:000
28848699230,22cyclictest329795-21apt-key20:20:000
28848699230,22cyclictest0-21swapper/019:55:140
288505992220,2cyclictest0-21swapper/722:20:147
288505992220,1cyclictest0-21swapper/700:25:137
288505992219,2cyclictest356579-21apt-get21:05:137
28850599220,21cyclictest0-21swapper/723:50:107
28850599220,21cyclictest0-21swapper/722:15:137
28850599220,21cyclictest0-21swapper/721:55:157
28850599220,21cyclictest0-21swapper/720:30:117
288503992221,1cyclictest306931-21apt-get19:40:086
28850399220,22cyclictest0-21swapper/622:05:016
28850399220,22cyclictest0-21swapper/620:30:006
28850399220,22cyclictest0-21swapper/619:50:006
28850399220,21cyclictest393704-21apt-get22:10:136
28850399220,21cyclictest0-21swapper/622:30:166
28850399220,21cyclictest0-21swapper/620:55:126
28850399220,21cyclictest0-21swapper/619:15:196
28850399220,21cyclictest0-21swapper/600:15:136
28850399220,1cyclictest390853-21apt-get22:05:126
28850399220,1cyclictest0-21swapper/623:29:596
28850399220,1cyclictest0-21swapper/620:40:006
288499992221,1cyclictest842-21runrttasks19:25:145
288499992221,1cyclictest0-21swapper/521:00:145
288499992221,1cyclictest0-21swapper/500:10:145
288499992218,0cyclictest0-21swapper/523:30:125
28849999220,21cyclictest416134-21awk22:50:015
28849999220,21cyclictest0-21swapper/500:00:125
28849999220,1cyclictest416553-21apt-get22:50:135
28849999220,1cyclictest0-21swapper/520:15:155
28849999220,19cyclictest0-21swapper/520:05:135
288497992221,0cyclictest298359-21apt-get19:25:134
288497992220,1cyclictest451089-21apt-get23:50:104
288497992220,1cyclictest376149-21run-parts21:40:004
28849799220,22cyclictest0-21swapper/422:35:014
28849799220,21cyclictest335628-21apt-get20:30:014
28849799220,21cyclictest0-21swapper/420:35:134
28849799220,1cyclictest0-21swapper/420:05:134
28849799220,0cyclictest0-21swapper/423:20:124
288494992221,1cyclictest410455-21awk22:40:013
288494992221,1cyclictest0-21swapper/300:20:123
288494992221,0cyclictest0-21swapper/323:25:113
288494992220,1cyclictest376130-21idleruntime-cro21:40:003
288494992220,1cyclictest0-21swapper/300:20:003
28849499220,22cyclictest453530-21turbostat23:55:003
288492992221,1cyclictest453943-21apt-get23:55:132
288492992221,1cyclictest445392-21apt-get23:40:132
288492992220,2cyclictest418990-21perf22:55:012
288492992220,1cyclictest301211-21apt-get19:30:142
28849299220,1cyclictest0-21swapper/219:20:012
288488992221,1cyclictest320779-21idleruntime-cro20:04:591
288488992221,1cyclictest0-21swapper/120:20:011
288488992220,1cyclictest342044-21apt-get20:40:121
288488992220,1cyclictest0-21swapper/123:40:121
288488992220,1cyclictest0-21swapper/120:40:001
28848899220,21cyclictest428237-21apt-get23:10:111
28848899220,21cyclictest353468-21apt-get21:00:151
28848899220,21cyclictest0-21swapper/123:00:121
28848899220,21cyclictest0-21swapper/122:40:131
28848899220,21cyclictest0-21swapper/122:20:141
28848899220,21cyclictest0-21swapper/120:55:121
28848899220,21cyclictest0-21swapper/120:50:111
28848899220,21cyclictest0-21swapper/119:10:121
288486992221,1cyclictest479306-21systemctl00:39:590
28848699220,21cyclictest0-21swapper/023:50:100
28848699220,21cyclictest0-21swapper/023:25:110
28848699220,21cyclictest0-21swapper/020:25:140
28848699220,21cyclictest0-21swapper/000:20:130
28848699220,21cyclictest0-21swapper/000:05:130
288505992120,1cyclictest468311-21apt-get00:20:127
288505992120,1cyclictest0-21swapper/721:35:147
288505992120,1cyclictest0-21swapper/720:00:127
288505992120,0cyclictest0-21swapper/723:10:117
288505992119,2cyclictest0-21swapper/722:30:157
288505992119,1cyclictest419407-21apt-get22:55:167
28850599210,0cyclictest0-21swapper/720:05:137
288503992121,0cyclictest0-21swapper/623:55:136
288503992121,0cyclictest0-21swapper/620:45:136
288503992121,0cyclictest0-21swapper/600:25:136
288503992120,1cyclictest761rcuc/623:00:006
288503992120,1cyclictest0-21swapper/600:35:006
288503992119,2cyclictest336319-21apt-get20:30:116
28850399210,20cyclictest0-21swapper/622:20:136
28850399210,20cyclictest0-21swapper/619:10:136
28850399210,0cyclictest0-21swapper/619:20:136
28850399210,0cyclictest0-21swapper/600:10:136
288499992120,1cyclictest365152-21apt-get21:20:125
288499992120,1cyclictest365152-21apt-get21:20:115
288499992120,1cyclictest353037-21systemctl21:00:005
288499992118,0cyclictest0-21swapper/500:30:135
28849999210,21cyclictest0-21swapper/522:40:135
28849999210,21cyclictest0-21swapper/500:20:125
28849999210,20cyclictest0-21swapper/520:00:135
28849999210,0cyclictest0-21swapper/523:40:135
288497992121,0cyclictest0-21swapper/421:40:134
288497992121,0cyclictest0-21swapper/419:15:134
288497992120,1cyclictest312635-21apt-get19:50:124
288497992119,1cyclictest335681-21systemd-run20:30:004
28849799210,20cyclictest0-21swapper/420:10:124
28849799210,0cyclictest0-21swapper/423:00:124
28849799210,0cyclictest0-21swapper/422:20:144
288494992121,0cyclictest0-21swapper/300:00:113
288494992120,1cyclictest359011-21munin-run21:10:003
288494992120,1cyclictest0-21swapper/322:50:013
28849499210,21cyclictest0-21swapper/323:10:013
28849499210,21cyclictest0-21swapper/319:20:133
28849499210,20cyclictest364702-21cron21:20:003
28849499210,20cyclictest364702-21cron21:20:003
28849499210,20cyclictest0-21swapper/322:20:133
28849499210,0cyclictest0-21swapper/322:00:173
288492992121,0cyclictest0-21swapper/221:30:142
288492992121,0cyclictest0-21swapper/220:45:132
288492992121,0cyclictest0-21swapper/200:35:132
288492992120,1cyclictest376580-21apt-get21:40:122
288492992120,1cyclictest338758-21perf20:35:002
288492992120,1cyclictest0-21swapper/221:35:152
28849299210,21cyclictest399028-21sendmail-msp22:20:012
28849299210,21cyclictest0-21swapper/219:25:202
28849299210,21cyclictest0-21swapper/219:15:132
28849299210,0cyclictest0-21swapper/221:25:122
28849299210,0cyclictest0-21swapper/221:20:122
28849299210,0cyclictest0-21swapper/221:20:122
288488992121,0cyclictest0-21swapper/119:45:131
288488992120,1cyclictest0-21swapper/123:30:121
288488992120,1cyclictest0-21swapper/120:45:131
288488992120,0cyclictest0-21swapper/122:25:101
28848899210,21cyclictest0-21swapper/123:05:131
28848899210,20cyclictest0-21swapper/121:30:151
28848899210,0cyclictest0-21swapper/121:25:121
288486992121,0cyclictest0-21swapper/022:40:130
288486992120,1cyclictest0-21swapper/019:10:130
288486992119,2cyclictest306552-21kworker/u16:3+ext4-rsv-conversion20:05:000
288486992119,1cyclictest370859-21apt-get21:30:160
288486992119,1cyclictest362279-21apt-get21:15:130
288486992119,1cyclictest362279-21apt-get21:15:120
28848699210,21cyclictest0-21swapper/021:45:000
28848699210,21cyclictest0-21swapper/019:15:230
28848699210,20cyclictest0-21swapper/022:00:160
28848699210,20cyclictest0-21swapper/020:10:120
28848699210,1cyclictest295507-21apt-get19:20:130
288505992020,0cyclictest0-21swapper/720:55:127
288505992020,0cyclictest0-21swapper/719:35:147
288505992020,0cyclictest0-21swapper/719:25:147
288505992020,0cyclictest0-21swapper/719:10:137
288505992019,1cyclictest0-21swapper/723:00:127
288503992020,0cyclictest0-21swapper/619:55:136
288503992020,0cyclictest0-21swapper/600:20:136
288503992019,1cyclictest0-21swapper/622:45:136
288503992019,1cyclictest0-21swapper/619:35:156
28850399200,20cyclictest0-21swapper/620:09:596
288499992019,1cyclictest0-21swapper/522:30:165
28849999200,20cyclictest0-21swapper/519:15:135
288497992020,0cyclictest0-21swapper/422:55:154
288497992020,0cyclictest0-21swapper/421:00:004
288497992018,1cyclictest467901-21sendmail00:19:594
28849799200,0cyclictest0-21swapper/420:55:004
288494992020,0cyclictest0-21swapper/323:55:123
288494992020,0cyclictest0-21swapper/322:15:133
288494992020,0cyclictest0-21swapper/320:10:123
288494992020,0cyclictest0-21swapper/319:35:153
288494992020,0cyclictest0-21swapper/319:35:003
28849499200,20cyclictest0-21swapper/320:30:013
288492992020,0cyclictest0-21swapper/223:15:162
28849299200,0cyclictest0-21swapper/223:00:122
288488992020,0cyclictest0-21swapper/119:25:141
288488992019,1cyclictest0-21swapper/121:20:121
288488992019,1cyclictest0-21swapper/121:20:121
288488992019,1cyclictest0-21swapper/121:05:131
288488992019,0cyclictest0-21swapper/123:20:111
28848899200,0cyclictest0-21swapper/123:25:111
288486992020,0cyclictest0-21swapper/023:40:130
288486992020,0cyclictest0-21swapper/023:10:110
288486992020,0cyclictest0-21swapper/023:05:130
288486992020,0cyclictest0-21swapper/022:45:130
288486992020,0cyclictest0-21swapper/022:20:140
288486992020,0cyclictest0-21swapper/021:25:120
288486992020,0cyclictest0-21swapper/021:00:150
288486992020,0cyclictest0-21swapper/020:55:110
288486992020,0cyclictest0-21swapper/020:50:120
288486992020,0cyclictest0-21swapper/020:20:150
288486992019,1cyclictest0-21swapper/022:15:130
288505991919,0cyclictest0-21swapper/721:40:117
288505991919,0cyclictest0-21swapper/719:50:127
288505991919,0cyclictest0-21swapper/719:30:157
288505991918,1cyclictest0-21swapper/721:10:167
288503991919,0cyclictest0-21swapper/623:30:126
288503991919,0cyclictest0-21swapper/622:40:136
288503991919,0cyclictest0-21swapper/621:10:166
288503991919,0cyclictest0-21swapper/621:05:136
288503991919,0cyclictest0-21swapper/600:39:596
288503991917,2cyclictest329640-21sendmail20:20:006
288499991919,0cyclictest0-21swapper/523:20:125
288499991919,0cyclictest0-21swapper/523:05:145
288499991919,0cyclictest0-21swapper/521:10:005
288499991919,0cyclictest0-21swapper/520:45:135
288499991919,0cyclictest0-21swapper/520:30:125
288499991919,0cyclictest0-21swapper/519:35:155
288499991919,0cyclictest0-21swapper/500:05:135
288499991916,2cyclictest1896684-21systemd-journal23:20:005
28849999190,18cyclictest0-21swapper/500:25:135
288497991919,0cyclictest0-21swapper/423:40:134
288497991919,0cyclictest0-21swapper/421:55:154
288497991919,0cyclictest0-21swapper/421:45:134
288497991919,0cyclictest0-21swapper/420:00:134
288497991919,0cyclictest0-21swapper/419:40:104
288494991919,0cyclictest0-21swapper/323:30:123
288494991919,0cyclictest0-21swapper/322:45:013
288494991919,0cyclictest0-21swapper/320:30:123
288494991919,0cyclictest0-21swapper/319:40:093
288494991917,2cyclictest344476-21munin-run20:45:003
288492991919,0cyclictest0-21swapper/221:45:132
288492991918,1cyclictest448244-21apt-get23:45:152
288492991918,1cyclictest350594-21apt-get20:55:132
288492991918,1cyclictest0-21swapper/221:15:132
288492991918,1cyclictest0-21swapper/221:15:132
288488991919,0cyclictest0-21swapper/100:25:131
288488991917,1cyclictest444985-21basename23:40:001
288488991917,1cyclictest0-21swapper/121:50:131
288486991919,0cyclictest0-21swapper/023:00:120
288486991919,0cyclictest0-21swapper/021:55:150
288486991918,1cyclictest0-21swapper/020:40:000
288486991918,1cyclictest0-21swapper/020:30:000
288486991917,1cyclictest365152-21apt-get21:20:120
288486991917,1cyclictest365152-21apt-get21:20:120
288486991917,1cyclictest298359-21apt-get19:25:140
288505991818,0cyclictest0-21swapper/722:35:137
288505991818,0cyclictest0-21swapper/720:15:017
288505991818,0cyclictest0-21swapper/700:30:127
288505991816,1cyclictest379429-21apt-get21:45:137
288505991816,1cyclictest297929-21turbostat.cron19:25:007
288505991815,3cyclictest0-21swapper/700:00:127
288505991814,3cyclictest0-21swapper/721:19:237
288505991814,3cyclictest0-21swapper/721:19:227
28850599181,0cyclictest0-21swapper/723:35:597
28850599181,0cyclictest0-21swapper/721:02:137
28850599180,1cyclictest0-21swapper/719:45:207
28850599180,17cyclictest0-21swapper/720:23:027
288503991818,0cyclictest0-21swapper/621:00:206
288503991817,1cyclictest0-21swapper/621:55:016
288499991818,0cyclictest0-21swapper/522:10:135
288497991818,0cyclictest0-21swapper/423:05:134
288497991818,0cyclictest0-21swapper/422:45:134
288497991818,0cyclictest0-21swapper/422:40:124
288497991818,0cyclictest0-21swapper/419:45:134
288494991818,0cyclictest425128-21apt-get23:05:123
288494991818,0cyclictest0-21swapper/323:45:163
288494991818,0cyclictest0-21swapper/320:19:593
288494991818,0cyclictest0-21swapper/319:25:143
288494991816,1cyclictest341646-21sendmail-msp20:40:003
288494991815,2cyclictest391075-21cpuspeed_turbos22:05:143
288492991818,0cyclictest0-21swapper/220:50:122
288492991818,0cyclictest0-21swapper/220:00:132
288492991818,0cyclictest0-21swapper/219:45:142
288492991818,0cyclictest0-21swapper/200:20:132
288492991818,0cyclictest0-21swapper/200:00:112
288492991817,1cyclictest0-21swapper/222:00:002
288488991818,0cyclictest0-21swapper/123:55:131
288488991818,0cyclictest0-21swapper/121:55:151
288488991818,0cyclictest0-21swapper/119:30:151
288488991818,0cyclictest0-21swapper/119:20:011
288488991818,0cyclictest0-21swapper/119:15:191
288488991818,0cyclictest0-21swapper/100:20:131
288488991817,1cyclictest465018-21awk00:15:001
288488991817,1cyclictest0-21swapper/120:25:121
28848899180,0cyclictest0-21swapper/120:30:011
288486991818,0cyclictest0-21swapper/023:30:120
288486991818,0cyclictest0-21swapper/022:25:110
288486991818,0cyclictest0-21swapper/020:45:130
288486991818,0cyclictest0-21swapper/020:40:120
288486991818,0cyclictest0-21swapper/019:45:130
288505991717,0cyclictest431093-21apt-get23:15:157
288505991717,0cyclictest0-21swapper/723:40:137
288505991716,1cyclictest0-21swapper/723:24:597
28850599171,4cyclictest395230-21cut22:10:227
28850599170,16cyclictest0-21swapper/720:47:497
28850599170,15cyclictest0-21swapper/721:20:417
28850599170,15cyclictest0-21swapper/721:20:417
28850599170,0cyclictest0-21swapper/722:29:117
288503991716,1cyclictest0-21swapper/620:20:156
288503991716,0cyclictest324058-21apt-get20:10:126
28850399170,1cyclictest0-21swapper/622:25:116
288499991717,0cyclictest0-21swapper/521:50:135
288499991717,0cyclictest0-21swapper/520:50:125
288499991717,0cyclictest0-21swapper/520:24:035
288499991716,1cyclictest448244-21apt-get23:45:165
288499991715,1cyclictest404717-21perf22:30:015
288499991715,1cyclictest401861-21munin-run22:25:015
28849999170,16cyclictest0-21swapper/522:55:165
288497991717,0cyclictest0-21swapper/420:45:134
288497991717,0cyclictest0-21swapper/419:30:154
288497991717,0cyclictest0-21swapper/400:00:124
288497991716,1cyclictest396553-21apt-get22:15:124
288497991716,1cyclictest0-21swapper/419:20:134
288494991717,0cyclictest0-21swapper/321:20:123
288494991717,0cyclictest0-21swapper/321:20:123
288494991716,1cyclictest0-21swapper/323:15:163
288494991716,1cyclictest0-21swapper/322:00:013
288494991716,1cyclictest0-21swapper/320:05:003
288492991717,0cyclictest0-21swapper/220:40:122
288492991717,0cyclictest0-21swapper/220:15:012
288492991717,0cyclictest0-21swapper/219:35:152
28849299170,16cyclictest0-21swapper/221:50:132
288488991717,0cyclictest0-21swapper/122:45:131
288486991717,0cyclictest0-21swapper/023:20:100
288486991717,0cyclictest0-21swapper/021:05:130
288486991716,1cyclictest0-21swapper/000:25:130
28848699170,16cyclictest453911-21/usr/sbin/munin23:55:120
288505991616,0cyclictest0-21swapper/721:25:127
288505991615,1cyclictest0-21swapper/723:26:337
288505991614,1cyclictest382281-21apt-get21:50:137
288505991611,4cyclictest0-21swapper/720:40:127
28850599160,16cyclictest0-21swapper/700:06:227
28850599160,15cyclictest306897-21/usr/sbin/munin19:40:187
28850599160,15cyclictest0-21swapper/720:39:277
28850599160,15cyclictest0-21swapper/700:15:577
28850599160,15cyclictest0-21swapper/700:10:117
28850599160,14cyclictest0-21swapper/719:16:037
28850599160,0cyclictest0-21swapper/722:09:577
288503991616,0cyclictest0-21swapper/622:35:126
288503991616,0cyclictest0-21swapper/621:30:156
288503991616,0cyclictest0-21swapper/620:50:126
288503991615,1cyclictest0-21swapper/619:35:006
288499991616,0cyclictest0-21swapper/523:55:135
288499991616,0cyclictest0-21swapper/522:05:135
288499991616,0cyclictest0-21swapper/522:00:175
288499991616,0cyclictest0-21swapper/519:55:145
288497991616,0cyclictest0-21swapper/423:35:124
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional