You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-28 - 06:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Dec 28, 2025 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5521210167,9sleep20-21swapper/219:07:282
560329076,10sleep00-21swapper/019:08:300
541328060,9sleep30-21swapper/319:06:043
538627767,6sleep10-21swapper/119:05:421
580799291,9cyclictest3418-21apt-get23:35:143
579899250,4cyclictest0-21swapper/122:25:131
580799230,3cyclictest0-21swapper/322:00:173
579899231,4cyclictest27010-21grep19:50:261
5798992221,1cyclictest0-21swapper/100:06:121
5803992119,1cyclictest0-21swapper/220:40:142
5803992119,1cyclictest0-21swapper/220:40:132
579499211,14cyclictest0-21swapper/023:35:140
579899200,1cyclictest0-21swapper/100:05:011
579499206,7cyclictest18946-21basename23:00:010
580799189,3cyclictest7794-21sed21:25:283
580799189,3cyclictest32540-21wc21:10:223
579899181,8cyclictest0-21swapper/121:35:151
5798991816,1cyclictest22700-21awk23:05:211
580799179,2cyclictest22505-21ls21:59:593
5807991710,2cyclictest11607-21cat23:50:173
5803991715,1cyclictest13188-21ntp_states20:30:242
580399170,3cyclictest0-21swapper/223:50:132
580799169,2cyclictest1524-21sed22:20:303
5798991611,3cyclictest14323-21ls21:40:191
579899160,12cyclictest0-21swapper/121:25:151
579899151,5cyclictest225-21systemd-journal21:10:001
579499156,8cyclictest21312-21munin-run23:05:000
579499156,7cyclictest10626-21sendmail-msp19:20:010
5794991511,3cyclictest20095-21cron19:40:000
580799140,13cyclictest4354-21cron22:29:593
580799140,13cyclictest3138-21cron23:35:003
5803991410,2cyclictest11270-21grep20:25:292
579899143,5cyclictest0-21swapper/121:50:161
5798991411,2cyclictest8270-21cat20:20:191
5798991410,3cyclictest6798-21sh22:35:001
5798991410,3cyclictest21584-21grep20:50:151
5798991410,3cyclictest10161-21grep19:15:281
5798991410,2cyclictest24970-21ls23:10:181
579899140,13cyclictest0-21swapper/119:40:151
579499141,8cyclictest0-21swapper/020:30:170
5794991411,2cyclictest1666-21cut23:30:170
5794991410,3cyclictest24601-21grep19:45:290
5794991410,2cyclictest32729-21cat21:10:250
580799139,3cyclictest29958-21cat20:00:003
580799139,3cyclictest29942-21head23:20:223
5807991310,2cyclictest9215-21basename22:40:013
5807991310,2cyclictest9215-21basename22:40:003
5807991310,2cyclictest8765-21grep19:15:163
5807991310,2cyclictest5698-21seq22:30:203
5807991310,2cyclictest32362-21basename00:35:153
5807991310,2cyclictest30684-21grep22:15:173
5807991310,2cyclictest26608-21tr21:00:163
5807991310,2cyclictest21055-21cut00:10:183
5807991310,2cyclictest20153-21cron00:09:593
5807991310,2cyclictest18730-21grep21:50:203
5807991310,2cyclictest17742-21latency_hist21:49:593
580399139,3cyclictest6099-21tr23:40:142
5803991310,2cyclictest9246-21grep19:15:192
5803991310,2cyclictest598-21tail22:20:212
5803991310,2cyclictest31527-21grep21:10:152
5803991310,2cyclictest26037-21cron23:15:002
5803991310,2cyclictest23967-21wc19:45:222
5803991310,2cyclictest23800-21wc00:15:192
5803991310,2cyclictest15182-21sed22:50:202
5803991310,2cyclictest0-21swapper/220:20:152
580399130,12cyclictest16016-21sort00:00:182
579899139,3cyclictest27312-21cron00:24:591
579899139,2cyclictest5517-21cron23:40:001
579899139,2cyclictest2658-21wc21:15:251
579899139,2cyclictest12303-21grep23:50:261
579899138,4cyclictest16510-21cron20:40:001
5798991310,2cyclictest9182-21sh22:40:011
5798991310,2cyclictest9182-21sh22:40:001
5798991310,2cyclictest8241-21wc19:15:001
5798991310,2cyclictest23581-21sed00:15:161
5798991310,2cyclictest20126-21cut20:45:201
579499139,3cyclictest31818-21grep23:25:200
5794991310,2cyclictest9696-21chrt23:45:280
5794991310,2cyclictest31120-21grep22:15:210
5794991310,2cyclictest26355-21grep00:20:200
5794991310,2cyclictest24938-21awk00:20:000
580799129,2cyclictest6633-21tail23:40:173
580799129,2cyclictest27403-21grep23:15:213
580799129,2cyclictest13491-21grep21:40:143
580799129,2cyclictest12999-21perf21:40:003
5807991210,1cyclictest317-21sed23:25:303
580399129,2cyclictest9174-21sort21:30:172
580399129,2cyclictest6292-21grep19:10:162
580399129,2cyclictest3152-21grep20:10:192
580399129,2cyclictest29882-21wc20:00:002
580399129,2cyclictest27577-21wc21:00:232
580399129,2cyclictest27350-21grep00:25:002
580399129,2cyclictest20879-21head20:45:282
580399129,2cyclictest18729-21grep21:50:202
580399129,2cyclictest15376-21turbostat21:45:002
580399129,2cyclictest11601-21grep21:35:192
580399128,3cyclictest29258-21df23:20:152
580399128,2cyclictest22557-21sleep00:15:002
580399125,4cyclictest311rcuc/220:55:162
580399122,6cyclictest0-21swapper/222:35:002
579899129,2cyclictest9924-21grep20:25:171
579899129,2cyclictest5570-21cut20:15:181
579899129,2cyclictest4058-21grep20:10:281
579899129,2cyclictest32316-21wc21:10:191
579899129,2cyclictest30414-21head23:20:271
579899129,2cyclictest22698-21cat22:00:001
579899129,2cyclictest1786-21grep20:05:291
579899128,3cyclictest29557-21taskset00:28:241
579899124,4cyclictest225-21systemd-journal23:50:001
579899123,6cyclictest28006-21unixbench_singl21:00:281
579899120,3cyclictest0-21swapper/122:40:151
579899120,10cyclictest0-21swapper/120:40:161
579899120,10cyclictest0-21swapper/120:40:151
579499129,2cyclictest358-21grep00:35:160
579499129,2cyclictest29001-21cat00:25:270
579499129,2cyclictest24708-21sed20:55:180
579499129,2cyclictest24434-21grep22:00:300
579499129,2cyclictest18410-21expr21:50:180
579499129,2cyclictest15213-21head22:50:190
579499129,2cyclictest12977-21sh19:25:000
579499128,3cyclictest29639-21grep21:05:180
579499123,8cyclictest1370-21apache200:06:360
580799119,1cyclictest21333-21wc23:05:003
580799118,2cyclictest6926-21taskset20:19:143
580799118,2cyclictest32742-21tr20:05:183
580799118,2cyclictest28817-21tr00:25:233
580799118,2cyclictest2677-21grep20:10:163
580799118,2cyclictest25599-21expr22:05:173
580799118,2cyclictest25501-21cut00:20:133
580799118,2cyclictest22261-21grep20:50:173
580799118,2cyclictest20961-21cat19:40:173
580799118,2cyclictest19148-21ntpq19:35:203
580799118,2cyclictest18732-21chrt20:42:523
580799118,2cyclictest18732-21chrt20:42:523
580799118,2cyclictest18053-21wc22:55:233
580799118,2cyclictest14654-21grep20:35:163
580799118,2cyclictest13698-21expr19:25:143
580799118,2cyclictest12232-21expr22:45:133
580799118,2cyclictest12172-21wget20:30:173
580799110,10cyclictest23988-21cut20:55:163
580399118,2cyclictest6806-21grep21:25:182
580399118,2cyclictest32-21ksoftirqd/219:35:142
580399118,2cyclictest31834-21grep23:25:192
580399118,2cyclictest31782-21grep20:00:302
580399118,2cyclictest30718-21tail00:30:182
580399118,2cyclictest30718-21tail00:30:172
580399118,2cyclictest30615-21grep22:15:172
580399118,2cyclictest28874-21taskset21:05:092
580399118,2cyclictest27718-21taskset22:10:142
580399118,2cyclictest22491-21sh22:00:002
580399118,2cyclictest22433-21taskset19:41:352
580399118,2cyclictest19455-21tr23:00:132
580399118,2cyclictest16755-21taskset22:55:042
580399118,2cyclictest1470-21tr21:15:142
580399118,2cyclictest11839-21apt-get22:45:112
580399118,2cyclictest11522-21cut19:20:182
580399118,2cyclictest10236-21tail22:40:192
5803991110,1cyclictest0-21swapper/221:20:162
580399110,11cyclictest0-21swapper/223:15:142
580399110,10cyclictest418-21df00:35:172
579899118,2cyclictest593-21head22:20:211
579899118,2cyclictest31355-21head20:00:251
579899118,2cyclictest30039-21grep22:15:151
579899118,2cyclictest29865-21basename20:00:001
579899118,2cyclictest24003-21aten2_r7power_p20:55:161
579899118,2cyclictest19615-21grep19:35:271
579899118,2cyclictest17397-21sort22:55:171
579899118,2cyclictest15204-21cut22:50:201
579899118,2cyclictest1175-21grep23:30:151
579899117,3cyclictest13358-21fgrep20:30:251
579899117,2cyclictest13279-21tr23:55:151
579899116,3cyclictest20689-21tr00:10:161
579899110,5cyclictest11549-21fschecks_time19:20:181
579499118,2cyclictest843-21head20:05:200
579499118,2cyclictest6835-21grep19:10:190
579499118,2cyclictest6549-21expr21:25:160
579499118,2cyclictest6544-21sed20:15:270
579499118,2cyclictest4121-21grep20:10:280
579499118,2cyclictest30562-21sort00:30:170
579499118,2cyclictest30562-21sort00:30:160
579499118,2cyclictest30348-21grep23:20:270
579499118,2cyclictest28308-21head22:10:180
579499118,2cyclictest22797-21grep23:05:220
579499118,2cyclictest21933-21mailstats19:40:230
579499118,2cyclictest21553-21tr00:10:210
579499118,2cyclictest21136-21head21:55:190
579499118,2cyclictest17240-21chrt21:45:280
579499118,2cyclictest16121-21grep00:00:180
579499118,2cyclictest14349-21cut23:55:230
579499118,2cyclictest10709-21cat21:35:000
579499117,3cyclictest12569-21sed21:35:280
579499115,2cyclictest225-21systemd-journal20:50:010
580799108,2cyclictest24641-21seq00:16:343
580799108,1cyclictest9980-21expr20:25:183
580799108,1cyclictest9651-21ntpq21:30:213
580799108,1cyclictest72842sleep320:20:133
580799108,1cyclictest6467-21expr19:10:163
580799108,1cyclictest31491-21tr20:00:263
580799108,1cyclictest26736-21sensors19:50:233
580799108,1cyclictest21805-21expr23:05:153
580799108,1cyclictest19504-21expr20:45:163
580799108,1cyclictest16812-21cat00:00:263
580799108,1cyclictest15948-21expr19:30:163
580799108,1cyclictest1523-21grep21:15:133
580799108,1cyclictest14769-21cat22:50:163
580799107,2cyclictest9561-21cut22:40:163
580799107,2cyclictest8989-21grep23:45:203
580799107,2cyclictest4021-21expr21:20:183
580799107,2cyclictest30417-21expr00:30:163
580799107,2cyclictest30417-21expr00:30:153
580799107,2cyclictest29257-21expr21:05:143
580799107,2cyclictest29228-21grep22:10:263
580799107,2cyclictest24409-21expr23:10:133
580799107,2cyclictest15097-21cron00:00:003
580799107,2cyclictest11144-21grep19:20:163
580799100,9cyclictest23448-21expr19:45:183
580399108,1cyclictest9912sleep223:30:082
580399108,1cyclictest6910-21taskset20:19:012
580399108,1cyclictest28547-21wc00:25:202
580399108,1cyclictest27042-21chrt22:07:222
580399108,1cyclictest25651-21cat19:50:142
580399108,1cyclictest2307-21cut22:25:142
580399108,1cyclictest22879-21hostname22:00:172
580399108,1cyclictest22583-21ls20:50:192
580399108,1cyclictest17801-21wget00:05:152
580399108,1cyclictest17479-21taskset19:31:572
580399108,1cyclictest15285-21taskset19:29:162
580399108,1cyclictest15139-21tail20:35:192
580399108,1cyclictest100562sleep223:46:192
580399107,2cyclictest8292-21grep22:35:222
580399107,2cyclictest8292-21grep22:35:222
580399107,2cyclictest5060-21grep23:35:262
580399107,2cyclictest1998-21taskset20:07:482
580399107,2cyclictest15147-21lockfile-create23:59:592
579899108,1cyclictest9044-21grep21:30:161
579899108,1cyclictest3741-21cut21:20:161
579899108,1cyclictest30916-21wc00:30:201
579899108,1cyclictest30916-21wc00:30:191
579899108,1cyclictest23598-21sort22:00:201
579899108,1cyclictest23352-21expr19:45:181
579899108,1cyclictest1807-21seq00:37:251
579899108,1cyclictest17541-21chrt21:47:321
579899108,1cyclictest12147-21wc22:45:141
579899107,2cyclictest5772-21taskset23:40:051
579899107,2cyclictest31120-21cut23:25:171
579899107,2cyclictest27618-21aten2_r7power_p22:10:151
579899107,2cyclictest26844-21awk22:05:281
579899107,2cyclictest26420-21cut23:15:151
579899103,3cyclictest273-21systemd-udevd19:25:031
579899101,5cyclictest225-21systemd-journal19:35:001
579499108,1cyclictest6616-21expr23:40:180
579499108,1cyclictest4589-21taskset22:30:050
579499108,1cyclictest3393-21grep22:25:210
579499108,1cyclictest30076-21taskset20:00:050
579499108,1cyclictest25728-21df_inode22:05:170
579499108,1cyclictest24138-21grep23:10:130
579499108,1cyclictest16507-21cat19:30:190
579499108,1cyclictest152502sleep021:43:360
579499107,2cyclictest9912-21grep20:25:170
579499107,2cyclictest8075-21which20:20:170
579499107,2cyclictest7350-21tr22:35:150
579499107,2cyclictest7350-21tr22:35:140
579499107,2cyclictest442-21sort22:20:200
579499107,2cyclictest27242-21cut23:15:190
579499107,2cyclictest26602-21tr21:00:160
579499107,2cyclictest26284-21uname19:50:200
579499107,2cyclictest1999-21head21:15:170
579499107,2cyclictest17218-21expr20:40:160
579499107,2cyclictest17218-21expr20:40:150
579499107,2cyclictest14816-21ls19:25:240
579499107,2cyclictest10174-21head22:40:180
579499100,10cyclictest0-21swapper/023:53:130
58039997,1cyclictest21572-21dpkg23:05:132
57989996,2cyclictest19915-21grep23:00:151
57949997,2cyclictest5088-21wc21:20:280
57949997,1cyclictest27677-21chrt19:55:010
57949997,1cyclictest21631-21cut20:50:150
57949997,1cyclictest15307-21cat20:35:200
57949997,1cyclictest11815-21cat22:45:100
58039986,1cyclictest16590-21grep21:45:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional