You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-12 - 23:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Dec 12, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17182210781,9sleep30-21swapper/319:06:163
1742629977,17sleep10-21swapper/119:09:231
1533728467,13sleep20-21swapper/219:05:062
1736328265,13sleep00-21swapper/019:08:340
1755799290,28cyclictest0-21swapper/023:36:380
17557992625,1cyclictest0-21swapper/023:58:400
1756799230,22cyclictest0-21swapper/323:40:153
1755799230,21cyclictest0-21swapper/021:42:160
1756399211,1cyclictest28733-21sh22:35:192
1756399210,21cyclictest0-21swapper/222:15:462
1756799200,19cyclictest0-21swapper/323:45:143
1755799202,13cyclictest0-21swapper/023:15:130
1755799195,8cyclictest19990-21perf00:30:000
1756799181,14cyclictest0-21swapper/300:15:163
17563991815,2cyclictest32-21ksoftirqd/221:20:542
1756399181,13cyclictest10616-21diskmemload23:13:182
1755799184,7cyclictest10616-21diskmemload23:05:000
1756799173,11cyclictest10083-21kworker/u8:000:10:033
1756799172,4cyclictest141rcu_preempt23:35:503
1756799170,17cyclictest0-21swapper/321:42:293
1756799170,17cyclictest0-21swapper/300:03:043
1756799170,0cyclictest0-21swapper/322:40:093
17563991714,3cyclictest32-21ksoftirqd/223:59:412
1756099172,14cyclictest0-21swapper/121:44:461
1755799170,15cyclictest698-21ls23:50:190
17567991614,0cyclictest0-21swapper/323:02:253
17563991613,2cyclictest0-21swapper/221:55:382
1756099163,7cyclictest224-21systemd-journal19:55:001
17560991613,3cyclictest0-21swapper/123:49:171
1756099160,16cyclictest0-21swapper/123:34:251
1755799163,9cyclictest0-21swapper/023:25:160
1755799161,8cyclictest21122-21munin-run20:25:000
1756799150,1cyclictest0-21swapper/300:26:503
1756099155,5cyclictest10616-21diskmemload23:11:411
1756099154,8cyclictest472-21avahi-daemon00:05:251
17560991512,2cyclictest14517-21cut00:00:191
1756099150,2cyclictest141rcu_preempt20:05:271
17557991511,3cyclictest22115-21rm22:30:190
1756799142,9cyclictest26894-21ssh21:45:193
1756799141,5cyclictest0-21swapper/323:30:153
1756799141,3cyclictest224-21systemd-journal22:39:553
17567991413,1cyclictest0-21swapper/322:30:053
17567991413,1cyclictest0-21swapper/322:06:283
1756399148,3cyclictest0-21swapper/221:31:542
17563991414,0cyclictest0-21swapper/200:32:182
17563991410,3cyclictest7353-21head21:00:242
17563991410,3cyclictest37850irq/131-snd_hda23:05:152
1756399140,1cyclictest21518-21apt22:30:172
1756099149,4cyclictest25-21ksoftirqd/121:35:121
17560991411,2cyclictest2764-21wc20:50:281
17560991411,2cyclictest2764-21wc20:50:281
17560991410,3cyclictest24962-21wc20:30:201
1756099140,13cyclictest0-21swapper/123:15:131
1755799147,3cyclictest5891-21apt-get21:30:140
17557991411,2cyclictest28845-21ssh00:35:470
1755799140,14cyclictest0-21swapper/000:31:340
17567991310,2cyclictest9044-21grep22:20:183
17567991310,2cyclictest4953-21grep20:55:223
17567991310,2cyclictest24789-21cat19:25:023
17567991310,2cyclictest18932-21ls20:20:003
1756799130,12cyclictest31888-21awk19:39:593
1756399139,3cyclictest25083-21uname20:30:212
1756399134,2cyclictest0-21swapper/200:04:422
1756399133,7cyclictest0-21swapper/223:38:252
17563991312,1cyclictest0-21swapper/223:40:482
17563991311,1cyclictest0-21swapper/219:35:182
17563991310,2cyclictest798-21tr21:50:182
17563991310,2cyclictest3466-21basename20:55:002
17563991310,2cyclictest3466-21basename20:55:002
17563991310,2cyclictest24674-21cron19:25:002
1756399130,11cyclictest5955-21cat21:00:002
1756099139,3cyclictest17669-21wc20:15:201
1756099139,2cyclictest9065-21head22:20:181
1756099139,2cyclictest8778-21tr21:05:201
17560991310,2cyclictest19430-21grep19:10:271
1755799139,3cyclictest25585-21ssh00:08:250
1755799139,3cyclictest21857-21grep19:15:270
1755799139,2cyclictest11622-21wc23:35:000
1755799135,7cyclictest15556-21munin-run22:50:000
1755799134,5cyclictest14776-21grep20:10:160
17557991310,2cyclictest9373-21tail21:55:310
17557991310,2cyclictest32420-21grep21:25:210
17557991310,2cyclictest28664-21cron20:40:000
17557991310,2cyclictest2055-21tail20:50:190
17557991310,2cyclictest2055-21tail20:50:190
17557991310,2cyclictest17469-21wc22:50:270
17557991310,2cyclictest17397-21apt21:15:000
17557991310,2cyclictest10010-21grep22:20:270
1755799130,12cyclictest19488-21grep19:10:270
1755799130,12cyclictest0-21swapper/022:19:380
1756799129,2cyclictest9171-21cat20:00:003
1756799129,2cyclictest9150-21cat21:55:303
1756799129,2cyclictest3713-21grep19:45:253
1756799129,2cyclictest3572-21sed22:15:303
1756799129,2cyclictest30700-21taskset22:11:523
1756799129,2cyclictest22488-21grep20:25:233
1756799129,2cyclictest12193-21cat21:35:003
1756799129,2cyclictest10150-21grep21:05:323
17567991212,0cyclictest0-21swapper/322:55:203
1756799121,1cyclictest0-21swapper/321:51:063
17567991210,1cyclictest32494-21grep20:45:243
17567991210,1cyclictest16920-21head22:50:223
1756799120,4cyclictest10616-21diskmemload23:19:203
1756799120,11cyclictest11482-21cat20:05:003
1756399129,2cyclictest7983-21tr00:20:172
1756399129,2cyclictest5084-21uniq19:50:182
1756399129,2cyclictest3417-21uname22:40:232
1756399129,2cyclictest32383-21grep19:40:162
1756399129,2cyclictest255802sleep222:56:072
1756399129,2cyclictest22053-21ssh22:05:252
1756399129,2cyclictest19989-21sed21:40:202
1756399129,2cyclictest17501-21cat21:15:002
1756399129,2cyclictest16888-21tr20:15:152
1756399129,2cyclictest16666-21grep22:25:292
1756399129,2cyclictest14639-21head00:25:212
1756399129,2cyclictest1130-21sed21:25:302
1756399129,2cyclictest10620-21perf21:10:002
1756399129,2cyclictest10488-21ssh22:20:472
1756399128,3cyclictest29318-21grep00:10:312
1756399128,3cyclictest1271-21sh00:15:142
1756399128,1cyclictest4661-21ssh23:29:572
1756399127,4cyclictest21188-21wc20:25:002
1756399126,2cyclictest224-21systemd-journal23:47:182
1756099129,2cyclictest4753-21ssh00:17:171
1756099129,2cyclictest27750-21egrep00:10:181
1756099129,2cyclictest27283-21tail20:35:211
1756099129,2cyclictest25073-21grep23:20:211
1756099129,2cyclictest2372-21grep21:50:311
1756099129,2cyclictest22527-21date22:55:011
1756099129,2cyclictest20554-21wc20:20:291
1756099129,2cyclictest1230-21grep22:15:001
1756099129,2cyclictest11056-21head22:45:261
1756099128,3cyclictest7602-21grep19:55:171
1756099128,3cyclictest6812-21cut21:00:191
1756099128,3cyclictest30057-21grep22:35:301
1756099128,3cyclictest15393-21grep22:00:211
1756099126,2cyclictest224-21systemd-journal22:20:011
1755799129,2cyclictest7945-21ssh23:07:320
1755799129,2cyclictest6994-21grep19:55:150
1755799129,2cyclictest31622-21grep20:45:170
1755799129,2cyclictest29984-21cut19:35:140
1755799129,2cyclictest28409-21head00:10:220
1755799129,2cyclictest25344-21ssh23:45:120
1755799129,2cyclictest24281-21munin-run21:20:000
1755799129,2cyclictest15720-21sed00:00:310
1755799129,2cyclictest14984-21chrt22:25:150
1755799129,2cyclictest13457-21sh21:35:170
1755799123,6cyclictest0-21swapper/019:45:000
1755799121,6cyclictest27079-21idleruntime-cro19:30:000
1755799121,6cyclictest11376-21munin-run20:05:000
1755799120,1cyclictest0-21swapper/023:13:380
1755799120,11cyclictest29314-21sed22:10:280
309672110,3sleep30-21swapper/321:24:483
1756799119,1cyclictest8052-21grep23:55:233
1756799119,1cyclictest2368-21seq23:51:023
1756799118,2cyclictest9230-21basename22:45:133
1756799118,2cyclictest5811-21wc23:05:283
1756799118,2cyclictest26745-21apache_accesses00:35:153
1756799118,2cyclictest26544-21ssh23:20:543
1756799118,2cyclictest24453-21ssh00:32:503
1756799118,2cyclictest21111-21cron20:25:003
1756799118,2cyclictest17308-21rm21:14:533
1756799113,4cyclictest224-21systemd-journal00:08:033
1756799110,10cyclictest6850-21head21:00:203
1756399118,2cyclictest9047-21ssh22:45:072
1756399118,2cyclictest7436-21ssh23:30:472
1756399118,2cyclictest4228-21sleep19:50:002
1756399118,2cyclictest3832-21ssh23:52:282
1756399118,2cyclictest32598-21ssh23:01:252
1756399118,2cyclictest32312-21sh21:50:002
1756399118,2cyclictest29969-21ssh00:37:132
1756399118,2cyclictest29656-21ssh23:24:082
1756399118,2cyclictest27868-21uname20:35:262
1756399118,2cyclictest27559-21grep19:30:152
1756399118,2cyclictest22538-21sed00:05:282
1756399118,2cyclictest20874-21ssh22:53:282
1756399118,2cyclictest20863-21cut19:15:172
1756399118,2cyclictest18303-21sed23:15:202
1756399118,2cyclictest13945-21ssh22:00:022
1756399118,2cyclictest12721-21tr21:35:132
1756399118,2cyclictest1237-21cat20:50:002
1756399118,2cyclictest11821-21acpi20:05:132
1756399117,3cyclictest22358-21ssh21:18:052
1756399116,4cyclictest30136-21ssh22:11:152
1756399116,4cyclictest22819-21switchtime20:25:292
1756099119,1cyclictest7115-21tr23:55:171
1756099118,2cyclictest7701-21ssh22:43:591
1756099118,2cyclictest7534-21ssh21:55:181
1756099118,2cyclictest680-21sed20:45:291
1756099118,2cyclictest32425-21tr23:50:171
1756099118,2cyclictest31752-21taskset19:38:411
1756099118,2cyclictest27089-21cut00:35:201
1756099118,2cyclictest26992-21taskset19:29:181
1756099118,2cyclictest26833-21grep21:45:181
1756099118,2cyclictest25615-21grep21:20:211
1756099118,2cyclictest24973-21ssh00:33:221
1756099118,2cyclictest24708-21sh22:07:391
1756099118,2cyclictest23303-21grep22:30:311
1756099118,2cyclictest22018-21chrt19:16:481
1756099118,2cyclictest20554-21ssh22:29:331
1756099118,2cyclictest18738-21sed21:15:191
1756099118,2cyclictest1347-21rm21:25:421
1756099118,2cyclictest12891-21ssh00:24:271
1756099118,2cyclictest11717-21cut21:10:181
1756099118,2cyclictest10762-21wc20:00:251
1756099118,2cyclictest10010-21ssh23:09:491
1756099117,3cyclictest32151-21uname23:25:221
1756099117,3cyclictest30088-21tr23:00:191
1756099117,2cyclictest12152-21rm23:35:161
1756099113,2cyclictest141rcu_preempt22:55:151
1756099111,2cyclictest20237-21sensors_temp23:40:241
1755799118,2cyclictest9046-21df21:05:210
1755799118,2cyclictest7663-21acpi00:20:130
1755799118,2cyclictest4669-21ssh21:52:440
1755799118,2cyclictest3216-21vmstat00:15:310
1755799118,2cyclictest30961-21ssh22:36:190
1755799118,2cyclictest30664-21wc23:25:000
1755799118,2cyclictest27923-21tail21:45:280
1755799118,2cyclictest2667-21tr22:40:180
1755799118,2cyclictest24989-21ssh22:07:570
1755799118,2cyclictest24813-21rm21:20:160
1755799118,2cyclictest23797-21head22:55:200
1755799118,2cyclictest20493-21grep23:40:260
1755799118,2cyclictest14492-21ssh22:00:160
1755799118,2cyclictest12476-21sort20:05:180
1756799108,2cyclictest5290-21cat19:50:203
1756799108,1cyclictest7567-21ssh00:20:063
1756799108,1cyclictest32416-21cut19:40:173
1756799108,1cyclictest28056-21grep19:30:173
1756799108,1cyclictest26940-21cat20:35:193
1756799108,1cyclictest24509-21cat21:20:013
1756799108,1cyclictest17989-21ssh22:02:303
1756799108,1cyclictest12659-21cat20:05:193
1756799107,2cyclictest32457-21sed21:25:213
1756799107,2cyclictest32328-21wc23:25:243
1756799107,2cyclictest2053-21grep20:50:203
1756799107,2cyclictest2053-21grep20:50:193
1756799107,2cyclictest14755-21ssh22:25:103
1756799107,2cyclictest14694-21expr20:10:163
1756799107,2cyclictest13646-21grep21:35:183
1756799107,2cyclictest12216-21sh23:10:263
1756399108,1cyclictest303712sleep220:40:232
1756399108,1cyclictest18795-21cat19:10:202
1756399107,2cyclictest7296-21expr19:55:152
1756399107,2cyclictest14517-21cut20:10:152
1756399107,2cyclictest11357-21cron20:05:002
1756399106,2cyclictest25978-21cat19:25:192
1756099108,1cyclictest15452-21grep20:10:231
1756099107,2cyclictest9487-21ssh21:32:151
1756099107,2cyclictest3116-21sed19:45:181
1756099107,2cyclictest21451-21cut20:25:171
1756099107,2cyclictest2013-21cat19:45:001
1756099107,2cyclictest13676-21ssh00:25:051
1756099106,2cyclictest29120-21grep20:40:131
1756099103,6cyclictest4476-21gltestperf20:55:181
1756099102,4cyclictest224-21systemd-journal19:20:111
1755799108,1cyclictest5807-21taskset20:59:550
1755799108,1cyclictest4448-21taskset19:50:100
1755799108,1cyclictest29753-21grep20:40:180
1755799108,1cyclictest27686-21cat19:30:150
1755799108,1cyclictest24762-21hddtemp_smartct20:30:180
1755799107,2cyclictest2963-21cat19:45:170
1755799107,2cyclictest18736-21turbostat.cron20:20:000
1755799105,2cyclictest493-21systemd-logind19:20:240
1755799100,9cyclictest21861-21expr20:25:180
175679998,1cyclictest269372sleep319:28:363
175679998,1cyclictest24115-21grep20:30:153
175679997,1cyclictest17939-21grep19:10:163
175679996,2cyclictest20307-21hostname19:15:153
175679990,9cyclictest0-21swapper/320:40:133
175609990,5cyclictest224-21systemd-journal19:30:131
175579996,2cyclictest6364-21grep21:00:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional