You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 05:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Mar 25, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
250632211158,40sleep20-21swapper/219:09:532
248742203168,23sleep00-21swapper/019:07:250
248122203166,25sleep30-21swapper/319:06:363
247312199163,24sleep10-21swapper/119:05:331
2327121580,4sleep12517099cyclictest22:45:001
478321510,2sleep02516399cyclictest22:00:220
2518099105102,2cyclictest0-21swapper/300:34:413
25180999894,3cyclictest0-21swapper/321:28:563
25180999693,2cyclictest0-21swapper/321:02:403
25180999187,3cyclictest0-21swapper/321:23:413
25180998076,3cyclictest0-21swapper/319:17:333
25180996158,2cyclictest0-21swapper/300:26:403
132372470,1sleep30-21swapper/321:05:273
25180994542,2cyclictest0-21swapper/323:30:263
113572450,1sleep10-21swapper/119:50:121
25180993734,2cyclictest0-21swapper/322:30:423
2516399347,13cyclictest21077-21munin-run22:40:010
183572320,4sleep12517099cyclictest21:20:011
25180992926,2cyclictest0-21swapper/320:38:303
2518099292,22cyclictest23183-21kworker/3:219:37:053
2518099292,17cyclictest23183-21kworker/3:223:08:173
2516399291,14cyclictest0-21swapper/022:08:170
2518099282,16cyclictest23183-21kworker/3:200:38:253
2517099281,11cyclictest0-21swapper/122:59:181
2516399280,13cyclictest15109-21perf23:40:010
2516399280,13cyclictest15109-21perf23:40:010
2518099272,17cyclictest23183-21kworker/3:222:58:373
2517599271,11cyclictest0-21swapper/219:28:282
2516399273,12cyclictest378-40dnf20:37:500
2516399272,12cyclictest30511-21munin-run00:15:010
2518099261,9cyclictest23183-21kworker/3:221:59:373
2517599261,5cyclictest0-21swapper/221:32:462
2517599261,11cyclictest0-21swapper/220:15:252
2517099263,9cyclictest0-21swapper/123:04:251
2517099261,11cyclictest0-21swapper/121:41:341
73132250,2sleep30-21swapper/322:05:253
324132250,4sleep232416-21cpuspeed_turbos23:05:122
25180992513,9cyclictest23867-40updatedb00:00:003
2517099258,3cyclictest0-21swapper/100:36:061
2517099252,8cyclictest0-21swapper/121:49:461
2517099251,6cyclictest0-21swapper/123:15:451
2517099251,11cyclictest14668-21fschecks_time21:10:161
2517099251,11cyclictest0-21swapper/122:10:091
2517099251,11cyclictest0-21swapper/121:21:581
2517099251,10cyclictest0-21swapper/121:35:481
2516399252,11cyclictest32647-21/usr/sbin/munin19:25:160
2516399252,11cyclictest0-21swapper/021:40:010
2516399251,12cyclictest25365-21/usr/sbin/munin19:10:250
318652240,8sleep230-21rcuc/221:50:122
316782241,3sleep131863-21ls21:50:121
2518099242,9cyclictest23183-21kworker/3:222:48:293
2518099242,8cyclictest23183-21kworker/3:222:04:203
2518099242,8cyclictest23183-21kworker/3:200:21:413
2518099242,8cyclictest23183-21kworker/3:200:07:333
2518099242,15cyclictest23183-21kworker/3:223:22:333
2517599242,12cyclictest6299-21taskset23:20:012
2517599241,13cyclictest0-21swapper/222:26:572
2517599241,13cyclictest0-21swapper/222:26:572
2517099245,10cyclictest0-21swapper/122:48:011
2517099241,16cyclictest0-21swapper/122:08:551
2517099241,11cyclictest1856-21df_inode21:55:141
2517099241,10cyclictest0-21swapper/123:46:081
2517099241,10cyclictest0-21swapper/123:35:291
2517099241,10cyclictest0-21swapper/123:35:291
2517099241,10cyclictest0-21swapper/122:52:441
2517099241,10cyclictest0-21swapper/122:19:581
2516399247,9cyclictest11882-21gltestperf19:50:150
2516399247,14cyclictest11064-21idleruntime-cro19:50:010
2516399242,7cyclictest3278-21kworker/u8:222:15:170
2516399241,3cyclictest0-21swapper/023:52:450
2516399240,22cyclictest0-21swapper/020:15:160
2518099232,8cyclictest23183-21kworker/3:222:13:213
2518099232,7cyclictest23183-21kworker/3:222:41:213
2518099232,7cyclictest23183-21kworker/3:222:37:573
2518099231,4cyclictest0-21swapper/322:28:283
2518099231,4cyclictest0-21swapper/322:28:283
2517599236,10cyclictest20406-21kworker/u8:121:35:402
2517599232,10cyclictest580-21systemd-journal20:30:012
2517599231,3cyclictest86550irq/126-enp1s0-23:41:262
2517099239,3cyclictest13860-21sh21:10:001
2517099233,6cyclictest0-21swapper/100:02:381
2517099233,10cyclictest32483-21cstates23:05:131
25170992317,3cyclictest5731-21/usr/sbin/munin00:30:161
2517099231,3cyclictest30803-21chrt20:34:471
25170992312,3cyclictest19135-21smart_sda20:05:241
2517099231,13cyclictest10065-21timerandwakeup23:25:261
2517099231,10cyclictest527-21df19:25:131
2517099231,10cyclictest0-21swapper/100:10:011
2516399237,4cyclictest2126-21sed00:20:270
2516399231,5cyclictest7520-21grep23:20:190
2516399231,5cyclictest32668-21grep21:50:190
2516399231,10cyclictest30546-21ntp_states21:45:220
25163992310,3cyclictest86750irq/128-enp1s0-21:28:150
2518099221,8cyclictest23183-21kworker/3:221:49:133
2518099221,4cyclictest17655-21cpuspeed_turbos23:45:123
2517599225,9cyclictest833-21runrttasks22:11:212
2517599223,10cyclictest13865-21diskmemload21:15:202
2517599221,7cyclictest14476-21crond22:25:012
2517599221,5cyclictest13875-21perf21:10:012
2517599221,14cyclictest0-21swapper/222:02:172
2517599221,13cyclictest0-21swapper/222:31:412
2517599221,12cyclictest0-21swapper/200:10:552
2517599220,10cyclictest0-21swapper/220:02:092
2517099226,8cyclictest0-21swapper/122:21:001
25170992217,3cyclictest0-21swapper/123:52:151
25170992215,4cyclictest28649-21awk20:30:011
2517099221,3cyclictest21142-21ntp_states20:10:241
2517099221,19cyclictest0-21swapper/120:53:591
2517099221,18cyclictest15053-21taskset19:55:251
2517099221,12cyclictest0-21swapper/121:31:161
2517099221,11cyclictest19098-21/usr/sbin/munin22:35:211
2517099221,10cyclictest27414-21latency_hist19:15:011
2517099221,10cyclictest0-21swapper/123:43:151
25170992210,3cyclictest32325-21users00:15:281
2516399228,3cyclictest17078-21sshd23:42:260
2516399222,7cyclictest4684-21/usr/sbin/munin19:35:260
2516399221,5cyclictest23051-21pmu-power23:55:230
2516399221,3cyclictest18863-21grep20:05:200
2516399221,19cyclictest0-21swapper/022:10:220
2516399220,20cyclictest0-21swapper/023:05:160
2516399220,20cyclictest0-21swapper/000:25:200
2518099217,2cyclictest23183-21kworker/3:223:25:143
2518099213,7cyclictest0-21swapper/321:37:303
2518099212,3cyclictest12050irq/123-ahci[0020:35:013
2518099212,11cyclictest10311-21/usr/sbin/munin22:15:183
2518099211,9cyclictest23183-21kworker/3:220:54:013
2518099211,11cyclictest23183-21kworker/3:219:25:453
2517599216,3cyclictest86850irq/129-enp1s0-19:40:192
2517599215,8cyclictest86550irq/126-enp1s0-22:15:122
2517599214,8cyclictest12050irq/123-ahci[0019:50:012
2517599214,4cyclictest7987-21fschecks_time20:55:132
2517599214,4cyclictest16058-21crond21:15:012
2517599213,9cyclictest7886-21latency_hist22:10:012
2517599213,9cyclictest3002-21cat20:45:022
2517599213,5cyclictest19518-21fschecks_count22:35:152
2517599211,9cyclictest8630-21cat23:25:012
2517599211,8cyclictest31040-21unixbench_singl21:45:262
2517599211,7cyclictest2547-21perf00:25:012
2517599211,11cyclictest28355-21ntp_states21:40:222
2517599211,11cyclictest21597-21memory21:25:182
2517099218,3cyclictest0-21swapper/122:25:211
2517099218,3cyclictest0-21swapper/122:25:211
2517099217,7cyclictest0-21swapper/121:28:511
2517099216,3cyclictest16018-21diskstats20:00:151
2517099212,17cyclictest0-21swapper/119:49:521
2517099211,9cyclictest0-21swapper/123:21:071
25170992117,3cyclictest0-21swapper/119:17:031
2517099211,18cyclictest0-21swapper/122:30:191
2517099211,18cyclictest0-21swapper/120:44:321
2517099211,11cyclictest5017-21df19:35:141
2516399219,9cyclictest25336-21sshd22:48:260
2516399217,3cyclictest30093-21/usr/sbin/munin23:00:140
2516399214,8cyclictest14864-21/usr/sbin/munin21:10:170
2516399213,8cyclictest4109-21munin-run23:15:010
2516399213,8cyclictest0-21swapper/022:59:450
2516399212,9cyclictest24579-21fschecks_time00:00:130
2516399212,16cyclictest5954-21strings00:30:180
2516399212,11cyclictest14707-21/usr/sbin/munin22:25:230
2516399212,11cyclictest14707-21/usr/sbin/munin22:25:220
2516399212,10cyclictest0-21swapper/021:40:220
113242210,1sleep00-21swapper/023:30:150
2518099209,3cyclictest14480-21munin-run22:25:013
2518099208,2cyclictest23183-21kworker/3:223:50:293
2518099207,2cyclictest23183-21kworker/3:220:29:493
2518099205,2cyclictest23183-21kworker/3:220:42:053
2518099204,2cyclictest23183-21kworker/3:219:53:413
2518099203,7cyclictest0-21swapper/323:35:413
2518099203,7cyclictest0-21swapper/323:35:403
2518099202,9cyclictest23183-21kworker/3:220:17:333
2518099202,7cyclictest23183-21kworker/3:221:52:133
2518099202,4cyclictest12050irq/123-ahci[0019:35:013
2518099201,9cyclictest23183-21kworker/3:220:07:443
2518099201,7cyclictest23183-21kworker/3:220:10:263
2518099201,6cyclictest23183-21kworker/3:219:24:573
2518099201,6cyclictest23183-21kworker/3:219:12:333
2518099201,11cyclictest29041-21sshd21:42:263
2518099201,11cyclictest13865-21diskmemload00:15:143
2518099201,11cyclictest0-21swapper/323:41:573
2518099200,9cyclictest23183-21kworker/3:220:57:173
2517599205,3cyclictest86550irq/126-enp1s0-23:00:122
2517599204,9cyclictest12050irq/123-ahci[0023:35:012
2517599201,7cyclictest32713-21taskset20:36:072
2517599201,6cyclictest6213-21awk19:35:252
2517599201,6cyclictest25944-21cut19:10:162
2517599201,5cyclictest2625-21grep19:30:112
2517599201,11cyclictest24112-21acpi00:00:102
2517599201,11cyclictest14642-21ps19:55:222
2517599201,10cyclictest28311-21latency_hist00:10:022
2517599201,10cyclictest2201-21awk23:10:112
2517599201,10cyclictest20542-21cat21:25:002
2517599201,10cyclictest19837-21latency_hist20:10:012
2517599201,10cyclictest0-21swapper/223:39:242
2517599201,10cyclictest0-21swapper/223:39:242
2517099209,3cyclictest0-21swapper/120:21:301
2517099208,3cyclictest2484-21diskstats23:10:131
2517099206,5cyclictest0-21swapper/123:31:051
2517099206,11cyclictest2993-21expr00:25:141
2517099202,10cyclictest0-21swapper/120:35:511
25170992016,3cyclictest0-21swapper/120:49:301
25170992014,3cyclictest2958-21grep19:30:151
2517099201,3cyclictest23048-21pmu-power23:55:231
2517099201,17cyclictest0-21swapper/121:02:271
2517099201,17cyclictest0-21swapper/119:23:031
2517099201,17cyclictest0-21swapper/100:11:271
2517099201,16cyclictest4522-21irqstats22:00:171
2516399209,8cyclictest9075-21sshd00:39:260
2516399204,9cyclictest21750-21sshd20:11:260
2516399204,3cyclictest3632-21head19:30:220
2516399202,9cyclictest9063-21df23:25:140
2516399201,8cyclictest31878-21pmu-power00:15:230
2516399201,6cyclictest26761-21fschecks_count00:05:150
2516399200,7cyclictest12056-21kworker/u8:022:41:210
2516399200,1cyclictest0-21swapper/022:50:150
2516399200,19cyclictest0-21swapper/021:55:220
2516399200,18cyclictest0-21swapper/019:59:100
236332201,3sleep20-21swapper/223:57:032
295542190,3sleep230-21rcuc/220:30:182
2518099197,2cyclictest23183-21kworker/3:219:49:493
2518099196,2cyclictest23183-21kworker/3:221:18:393
2518099195,6cyclictest4716-21fschecks_count23:15:133
2518099195,2cyclictest23183-21kworker/3:220:04:413
2518099193,5cyclictest0-21swapper/323:14:573
2518099192,2cyclictest86750irq/128-enp1s0-19:40:173
25180991914,3cyclictest13602-21grep19:55:113
2518099191,10cyclictest27535-21sshd22:53:263
2517599194,9cyclictest0-21swapper/223:50:052
2517599191,7cyclictest23742-21cstates22:45:122
2517599191,7cyclictest0-21swapper/222:52:082
2517599191,6cyclictest3491-21if_enp1s000:25:182
2517599191,6cyclictest2995-21grep21:55:262
2517599191,6cyclictest17862-21diskstats23:45:142
2517599191,6cyclictest0-21swapper/223:25:162
2517599191,6cyclictest0-21swapper/222:55:152
2517599191,5cyclictest7509-21diskstats00:35:142
2517599191,11cyclictest0-21swapper/222:40:032
2517599191,10cyclictest21332-21sensors20:10:262
2517599190,9cyclictest0-21swapper/219:53:052
2517099191,16cyclictest0-21swapper/120:56:591
2517099191,16cyclictest0-21swapper/120:15:031
2517099191,16cyclictest0-21swapper/119:40:491
2516399196,3cyclictest86750irq/128-enp1s0-23:15:190
2516399194,8cyclictest7-21ksoftirqd/020:45:160
2516399193,7cyclictest5476-21cstates20:50:120
2516399192,8cyclictest786-21NetworkManager21:04:120
2516399192,4cyclictest27159-21tune2fs20:25:150
2516399192,4cyclictest15669-21/usr/sbin/munin20:00:140
2516399191,9cyclictest7489-21/usr/sbin/munin20:55:120
2516399191,9cyclictest23694-21irqstats21:30:180
2516399191,8cyclictest13253-21sed21:05:270
2516399191,6cyclictest2579-21sshd20:41:250
2516399191,6cyclictest0-21swapper/021:20:530
2516399191,5cyclictest7024-21tr19:40:120
2516399190,6cyclictest12056-21kworker/u8:022:30:230
260282181,2sleep30-21swapper/300:04:143
2518099189,2cyclictest86550irq/126-enp1s0-20:49:093
2518099181,5cyclictest0-21swapper/300:13:573
2518099181,3cyclictest24318-21timerwakeupswit21:30:253
25180991812,3cyclictest25536-21ntpq20:20:213
2517599181,9cyclictest30736-21/usr/sbin/munin00:15:292
2517599181,6cyclictest0-21swapper/200:31:232
2517599181,5cyclictest5998-21/usr/sbin/munin20:50:162
2517599181,5cyclictest24462-21/usr/sbin/munin20:20:122
2517599181,5cyclictest11832-21cat21:05:012
2517099182,4cyclictest501-21cpu00:20:131
2516399186,10cyclictest86550irq/126-enp1s0-20:23:260
2516399183,6cyclictest17515-21/usr/sbin/munin23:45:170
2516399181,8cyclictest30594-21expr19:20:120
2516399181,8cyclictest29050-21cstates20:30:140
2516399181,4cyclictest851-21sssd_nss22:20:200
2518099177,7cyclictest31558-21unixbench-2d23:00:253
2518099173,8cyclictest0-21swapper/321:12:283
2517599171,6cyclictest0-21swapper/220:46:512
2517599171,6cyclictest0-21swapper/219:15:442
2517599171,5cyclictest30453-21/usr/sbin/munin19:20:222
2516399171,3cyclictest0-21swapper/021:15:120
25163991712,3cyclictest28534-21expr19:15:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional