You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-04 - 04:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon May 04, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
704521080,0sleep20-21swapper/222:38:232
2761529569,8sleep00-21swapper/019:07:260
2748229582,10sleep30-21swapper/319:05:433
2761729280,8sleep20-21swapper/219:07:272
2765128665,6sleep10-21swapper/119:07:551
304002620,0sleep20-21swapper/200:29:122
252772610,0sleep20-21swapper/221:31:202
2790299256,18cyclictest0-21swapper/220:31:312
27900992522,2cyclictest25-21ksoftirqd/122:20:041
2789599240,2cyclictest0-21swapper/023:40:470
2790299230,23cyclictest0-21swapper/220:16:152
27900992222,0cyclictest0-21swapper/100:39:471
2790599213,17cyclictest0-21swapper/300:03:323
2790599210,1cyclictest0-21swapper/322:32:503
2790299210,21cyclictest0-21swapper/222:48:002
2790599200,2cyclictest0-21swapper/321:49:083
2790299200,20cyclictest0-21swapper/200:16:462
2790099203,17cyclictest0-21swapper/123:42:221
2790099201,6cyclictest25-21ksoftirqd/122:57:311
2790099200,1cyclictest0-21swapper/121:21:051
2790299191,18cyclictest0-21swapper/223:36:132
27900991919,0cyclictest0-21swapper/122:15:581
27900991918,1cyclictest25-21ksoftirqd/122:14:531
2790099191,1cyclictest30085-21df_inode23:50:161
2790099190,19cyclictest0-21swapper/122:01:111
2789599190,1cyclictest0-21swapper/000:01:000
27902991818,0cyclictest0-21swapper/221:37:332
2790099189,8cyclictest25-21ksoftirqd/120:55:131
27900991816,1cyclictest0-21swapper/120:50:171
2790099181,2cyclictest0-21swapper/122:45:401
2790099180,18cyclictest0-21swapper/123:04:511
2789599183,1cyclictest141rcu_preempt23:54:380
213362180,0sleep20-21swapper/223:25:212
213362180,0sleep20-21swapper/223:25:202
2790599174,11cyclictest0-21swapper/323:10:123
2790599170,2cyclictest0-21swapper/300:37:073
2790599170,17cyclictest0-21swapper/322:12:193
2790299175,12cyclictest0-21swapper/219:30:512
27902991717,0cyclictest0-21swapper/222:43:242
2790099171,5cyclictest25-21ksoftirqd/121:35:471
27900991710,6cyclictest20413-21diskmemload21:17:431
2790099170,0cyclictest0-21swapper/121:53:511
2789599170,1cyclictest0-21swapper/021:22:570
2789599170,17cyclictest0-21swapper/023:22:060
2790599164,1cyclictest11613-21apt-get23:20:133
2790299162,7cyclictest0-21swapper/222:30:172
27902991612,2cyclictest9732-21turbostat19:40:002
2790299160,16cyclictest0-21swapper/222:54:592
2790299160,15cyclictest0-21swapper/222:25:122
2790299160,0cyclictest0-21swapper/222:10:542
27900991613,3cyclictest25-21ksoftirqd/100:23:281
2790099160,4cyclictest141rcu_preempt22:38:571
2790099160,1cyclictest0-21swapper/121:55:141
27895991615,1cyclictest0-21swapper/022:47:410
27895991615,1cyclictest0-21swapper/021:37:350
27905991512,1cyclictest0-21swapper/323:15:123
2790299153,12cyclictest0-21swapper/219:53:332
27902991515,0cyclictest0-21swapper/221:40:132
2790299150,14cyclictest0-21swapper/200:13:342
2790299150,12cyclictest0-21swapper/221:16:272
27900991512,3cyclictest25-21ksoftirqd/123:26:531
27900991512,3cyclictest25-21ksoftirqd/123:26:521
27900991510,3cyclictest0-21swapper/120:10:131
2790099150,14cyclictest0-21swapper/100:25:581
2789599153,6cyclictest29111-21apt-get20:20:140
2789599153,6cyclictest13251-21idleruntime-cro00:00:010
2789599150,2cyclictest0-21swapper/023:37:160
2789599150,13cyclictest0-21swapper/022:25:290
2790599144,6cyclictest456-21apt-get19:20:133
2790599143,8cyclictest31106-21switchtime23:50:253
27905991411,2cyclictest9411-21cut20:45:173
27905991410,3cyclictest8795-21sed00:15:253
27905991410,2cyclictest14280-21sed20:55:153
2790599140,13cyclictest0-21swapper/322:20:493
2790299149,3cyclictest15795-21ssh00:40:002
27902991414,0cyclictest0-21swapper/200:34:432
27902991411,2cyclictest12529-21grep23:00:262
27902991410,3cyclictest10432-21grep20:45:262
2790299140,5cyclictest0-21swapper/220:20:142
2790299140,1cyclictest0-21swapper/223:10:002
2790299140,14cyclictest0-21swapper/221:25:392
2790099148,2cyclictest25-21ksoftirqd/100:33:451
2790099142,2cyclictest141rcu_preempt23:23:031
2790099141,6cyclictest225-21systemd-journal00:00:001
2790099141,1cyclictest20413-21diskmemload00:02:481
27900991410,3cyclictest3789-21cron23:35:001
27900991410,3cyclictest17981-21sort22:25:181
2790099140,14cyclictest0-21swapper/123:39:451
2790099140,13cyclictest20413-21diskmemload21:34:181
2790099140,12cyclictest22353-21uname23:45:191
2789599149,3cyclictest15216-21basename22:04:590
2789599146,7cyclictest4989-21awk19:30:000
2789599146,7cyclictest31432-21munin-run00:30:000
2789599142,6cyclictest12125-21munin-run19:45:000
27895991411,2cyclictest11869-21head20:50:170
2789599141,12cyclictest0-21swapper/019:20:120
27895991410,3cyclictest4726-21idleruntime-cro21:20:000
27895991410,3cyclictest23071-21cron00:24:590
2790599139,3cyclictest14039-21sed19:45:263
2790599139,3cyclictest1249-21ssh22:35:123
2790599139,2cyclictest27137-21ls22:50:183
2790599139,2cyclictest24355-21grep20:10:143
2790599139,2cyclictest10513-21grep22:40:193
27905991311,2cyclictest0-21swapper/323:36:463
27905991311,1cyclictest20413-21diskmemload22:49:483
27905991310,2cyclictest6982-21ssh00:34:403
27905991310,2cyclictest32464-21ssh23:32:043
27905991310,2cyclictest30776-21grep19:15:143
27905991310,2cyclictest22208-21grep20:05:123
27905991310,2cyclictest22208-21grep20:05:123
27905991310,2cyclictest21584-21sed21:10:153
2790599130,13cyclictest0-21swapper/323:06:303
2790599130,13cyclictest0-21swapper/321:35:293
2790299139,3cyclictest3952-21wc23:35:012
27902991310,2cyclictest4558-21grep19:25:262
27902991310,2cyclictest27650-21tr23:10:142
27902991310,2cyclictest13463-21dpkg00:00:002
2790299130,12cyclictest16597-21cat22:25:002
2790299130,12cyclictest0-21swapper/220:50:152
2790099139,2cyclictest25-21ksoftirqd/100:16:531
2790099138,4cyclictest7882-21tr19:35:151
2790099138,1cyclictest225-21systemd-journal20:05:001
2790099134,8cyclictest0-21swapper/123:19:521
2790099133,1cyclictest141rcu_preempt23:09:081
27900991310,3cyclictest0-21swapper/100:09:541
27900991310,2cyclictest5273-21cut19:30:141
27900991310,2cyclictest17857-21grep19:55:171
27900991310,2cyclictest15332-21sed21:45:151
2790099130,13cyclictest0-21swapper/121:25:091
2790099130,12cyclictest668-21sed20:25:251
2789599139,3cyclictest729-21idleruntime-cro22:35:000
2789599138,3cyclictest32102-21ls00:10:200
2789599138,3cyclictest18443-21grep21:05:170
2789599132,10cyclictest25374-21ntp_states22:10:210
2789599131,8cyclictest0-21swapper/022:10:010
27895991310,2cyclictest31282-21sed19:15:170
27895991310,2cyclictest17890-21sed19:55:160
27895991310,2cyclictest15633-21sh21:00:000
2789599130,13cyclictest0-21swapper/021:49:420
2789599130,12cyclictest0-21swapper/021:40:340
2789599130,0cyclictest0-21swapper/022:54:520
2790599129,2cyclictest9706-21cron19:40:003
2790599129,2cyclictest7665-21grep22:00:143
2790599129,2cyclictest6204-21grep23:55:163
2790599129,2cyclictest30586-21wc20:20:253
2790599129,2cyclictest23911-21ssh23:46:043
2790599129,2cyclictest14663-21sed23:40:283
2790599129,2cyclictest11231-21tr20:50:143
2790599129,2cyclictest10787-21grep23:00:143
2790599128,3cyclictest18629-21wc19:55:253
2790599122,8cyclictest0-21swapper/322:08:453
2790299129,2cyclictest7666-21tr22:00:142
2790299129,2cyclictest3977-21sed22:55:252
2790299129,2cyclictest31766-21grep21:55:152
2790299129,2cyclictest28734-21uniq19:10:152
2790299129,2cyclictest14791-21sed00:00:172
2790299125,6cyclictest30996-21awk23:50:232
2790299121,10cyclictest11155-21ntp_states19:40:202
2790299120,4cyclictest0-21swapper/220:55:152
2790299120,1cyclictest0-21swapper/221:52:032
2790299120,11cyclictest23552-21grep20:05:242
2790299120,11cyclictest23552-21grep20:05:242
2790299120,11cyclictest13001-21grep23:40:152
2790099129,2cyclictest7188-21sed20:40:161
2790099129,2cyclictest32128-21ssh00:10:211
2790099129,2cyclictest10178-21expr22:40:161
2790099128,3cyclictest18797-21grep21:05:181
2790099128,3cyclictest16998-21tr22:05:211
2790099128,1cyclictest7697-21chrt21:40:211
2790099127,4cyclictest6090-21taskset20:39:401
2790099126,4cyclictest10681-21sort19:40:171
2790099124,6cyclictest25-21ksoftirqd/121:10:511
2790099121,5cyclictest225-21systemd-journal20:16:101
27900991210,1cyclictest3538-21apt20:35:011
2790099120,1cyclictest0-21swapper/119:45:131
2789599129,2cyclictest6882-21basename22:00:000
2789599129,2cyclictest6659-21grep20:40:140
2789599129,2cyclictest6004-21grep19:30:160
2789599129,2cyclictest4736-21head20:35:180
2789599129,2cyclictest4129-21sed23:15:170
2789599129,2cyclictest16860-21uname21:00:170
2789599129,2cyclictest15696-21grep21:25:260
2789599129,2cyclictest1159-21dump-pmu-power20:30:010
2789599128,3cyclictest14796-21cat00:20:000
2789599123,7cyclictest23560-21/usr/sbin/munin00:05:200
2789599120,6cyclictest14503-21idleruntime-cro19:50:000
2789599120,12cyclictest0-21swapper/000:36:450
2789599120,11cyclictest3542-21turbostat.cron20:35:010
2790599118,2cyclictest6600-21readlink19:30:233
2790599118,2cyclictest3542-21uniq19:25:173
2790599118,2cyclictest32579-21grep20:25:193
2790599118,2cyclictest29076-21sh21:15:033
2790599118,2cyclictest28901-21sed19:10:163
2790599118,2cyclictest24801-21ssh00:25:193
2790599118,2cyclictest24516-21wc21:50:223
2790599118,2cyclictest2433-21tr22:55:143
2790599118,2cyclictest24001-21ssh00:05:243
2790599118,2cyclictest22883-21ssh21:30:173
2790599118,2cyclictest20635-21sort23:25:163
2790599118,2cyclictest20635-21sort23:25:163
2790599118,2cyclictest1865-21ssh21:56:043
2790599118,2cyclictest17684-21ssh21:26:583
2790599118,2cyclictest17056-21cat00:20:263
2790599118,2cyclictest16851-21uname21:00:183
2790599118,2cyclictest16824-21ssh22:25:093
2790599118,2cyclictest11880-21sh21:43:113
2790599118,2cyclictest10791-21expr19:40:173
2790599117,3cyclictest21287-21apache200:10:103
2790599117,2cyclictest7191-21wc21:20:243
2790299119,1cyclictest65302sleep221:20:172
2790299118,2cyclictest444-21sh22:15:142
2790299118,2cyclictest4246-21if_enp0s31f623:15:192
2790299118,2cyclictest25994-21sed20:10:262
2790299118,2cyclictest25664-21ssh00:06:272
2790299118,2cyclictest21951-21tail23:45:172
2790299118,2cyclictest18160-21ssh22:05:472
2790299118,2cyclictest17842-21head19:55:172
2790299118,2cyclictest16314-21ssh21:45:232
2790299118,2cyclictest15897-21basename21:00:102
2790299118,2cyclictest15812-21expr00:20:152
2790299118,2cyclictest12919-21ssh23:20:202
2790299118,2cyclictest12631-21cut19:45:142
2790299111,9cyclictest4019-21apache_processe20:35:132
2790299110,8cyclictest0-21swapper/219:20:132
2790099118,2cyclictest31416-21ssh22:52:571
2790099118,2cyclictest29678-21ssh22:32:171
2790099118,2cyclictest28583-21cat19:10:141
2790099118,2cyclictest2593-21sh19:24:591
2790099116,4cyclictest30255-21kworker/1:219:15:261
2790099115,5cyclictest20413-21diskmemload23:11:161
2789599119,1cyclictest26017-21grep20:10:270
2789599118,2cyclictest5396-21ssh22:56:270
2789599118,2cyclictest5355-21ssh22:37:060
2789599118,2cyclictest32229-21ssh23:31:540
2789599118,2cyclictest31841-21ssh00:30:090
2789599118,2cyclictest31072-21ssh23:11:470
2789599118,2cyclictest26301-21ssh23:28:470
2789599118,2cyclictest26301-21ssh23:28:470
2789599118,2cyclictest23949-21taskset20:08:490
2789599118,2cyclictest23949-21taskset20:08:480
2789599118,2cyclictest22050-21ssh21:30:110
2789599118,2cyclictest21023-21ssh23:45:110
2789599118,2cyclictest20908-21ssh21:10:120
2789599118,2cyclictest19677-21cut23:05:170
2789599118,2cyclictest15973-21seq23:03:030
2789599118,2cyclictest14840-21cut19:50:140
2789599118,2cyclictest14763-21ssh22:42:540
2789599118,2cyclictest11975-21ssh22:21:320
2789599118,2cyclictest10000-21chrt20:45:220
2790599108,1cyclictest7237-21basename20:40:173
2790599108,1cyclictest3455-21taskset20:34:043
2790599108,1cyclictest32364-21apache_processe22:15:113
2790599107,2cyclictest5004-21wc20:35:203
2790599107,2cyclictest27601-21cat20:15:193
2790599107,2cyclictest18449-21egrep21:05:173
2790599104,5cyclictest20092-21diskstats20:00:163
2790299108,1cyclictest30953-21expr19:15:152
2790299108,1cyclictest21026-21cat20:00:262
2790299108,1cyclictest20828-21sh21:10:092
2790299107,2cyclictest31616-21cut20:25:142
2790299100,9cyclictest6747-21ls20:40:142
2790099108,1cyclictest9164-21expr20:45:141
2790099108,1cyclictest4378-21cat19:25:261
2790099108,1cyclictest30454-21cut20:20:231
2790099100,9cyclictest23268-21expr20:05:201
2790099100,9cyclictest23268-21expr20:05:191
2790099100,9cyclictest16362-21cat21:00:141
2789599108,1cyclictest7703-21basename19:35:150
2789599108,1cyclictest30429-21wc19:15:000
2789599108,1cyclictest22945-21rm21:50:070
2789599107,2cyclictest32142-21ssh22:15:020
2789599107,2cyclictest19785-21tr20:00:150
279059997,1cyclictest165682sleep319:51:273
279029997,1cyclictest19348-21cat21:05:222
279009997,1cyclictest15177-21expr19:50:141
278959997,1cyclictest27262-21sed20:15:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional