You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-16 - 21:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun Aug 16, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
326521020,0sleep00-21swapper/010:34:390
30321210061,34sleep00-21swapper/007:07:140
2839828947,38sleep10-21swapper/107:05:031
3022628774,9sleep30-21swapper/307:06:003
2840528370,9sleep20-21swapper/207:05:082
179622760,0sleep30-21swapper/312:20:003
3062699260,22cyclictest0-21swapper/211:50:012
3062499250,23cyclictest0-21swapper/109:10:211
16182240,0sleep20-21swapper/210:52:312
3061999218,7cyclictest16869-21munin-run12:00:000
30619992112,8cyclictest13-21ksoftirqd/012:07:500
3062699200,19cyclictest0-21swapper/210:11:212
30624992019,1cyclictest0-21swapper/110:33:071
3062499200,19cyclictest0-21swapper/111:21:381
30619992017,2cyclictest13-21ksoftirqd/010:58:080
30619992017,2cyclictest13-21ksoftirqd/010:58:070
3062499192,14cyclictest0-21swapper/108:40:141
30624991917,1cyclictest0-21swapper/111:56:061
30624991913,5cyclictest25-21ksoftirqd/109:51:531
3062499190,1cyclictest0-21swapper/109:28:401
30626991812,4cyclictest537-21snmpd07:25:182
30626991812,4cyclictest537-21snmpd07:25:172
3062699180,1cyclictest0-21swapper/211:21:582
3062499181,5cyclictest0-21swapper/110:29:321
3062499180,17cyclictest8804-21ssh12:14:251
3062999170,2cyclictest141rcu_preempt09:37:173
3062999170,16cyclictest49-21kcompactd009:15:243
3062999170,16cyclictest49-21kcompactd009:15:243
30626991717,0cyclictest0-21swapper/210:38:412
30626991714,3cyclictest0-21swapper/211:19:582
3062699170,17cyclictest0-21swapper/212:31:592
3062699170,16cyclictest0-21swapper/209:59:472
3062499173,12cyclictest3735-21sensors09:55:241
3062499170,17cyclictest0-21swapper/112:38:511
30629991615,1cyclictest0-21swapper/309:50:573
30629991614,1cyclictest0-21swapper/311:43:423
30626991615,1cyclictest0-21swapper/210:33:272
30626991610,1cyclictest0-21swapper/211:11:412
3062499165,3cyclictest23010-21diskmemload11:25:021
30624991614,1cyclictest0-21swapper/109:15:141
30624991614,1cyclictest0-21swapper/109:15:141
30624991612,4cyclictest0-21swapper/111:17:141
3061999167,7cyclictest19642-21cron07:55:000
3061999164,6cyclictest22835-21apt-get11:25:000
3061999163,10cyclictest30172-21/usr/sbin/munin10:50:170
3061999160,1cyclictest141rcu_preempt11:18:570
3061999160,14cyclictest0-21swapper/011:44:150
30629991515,0cyclictest0-21swapper/310:17:113
30626991514,1cyclictest0-21swapper/211:38:082
30626991511,3cyclictest9218-21tr08:40:152
30626991511,3cyclictest14418-21grep08:50:172
3062699150,3cyclictest0-21swapper/210:00:152
30624991514,1cyclictest0-21swapper/111:00:341
30624991510,4cyclictest0-21swapper/110:22:411
3062999149,4cyclictest16688-21cat11:20:263
30629991413,1cyclictest39-21ksoftirqd/309:22:023
30629991410,3cyclictest1469-21sed11:50:183
3062699145,5cyclictest225-21systemd-journal10:17:012
3062699142,11cyclictest0-21swapper/212:25:162
3062699141,12cyclictest17651-21grep09:45:152
3062499148,2cyclictest225-21systemd-journal10:09:591
3062499146,4cyclictest26754-21/usr/sbin/munin12:25:131
30624991410,3cyclictest30532-21cat08:15:221
30624991410,3cyclictest30029-21wc11:10:011
3061999148,6cyclictest8023-21apache209:57:410
3061999142,6cyclictest3914-21munin-run08:30:000
30619991413,1cyclictest26677-21diskmemload11:07:310
3061999140,13cyclictest14390-21grep07:40:260
116132140,0chrt0-21swapper/110:39:361
3062999139,3cyclictest9285-21tr08:40:153
3062999139,3cyclictest6128-21wc10:35:283
3062999139,3cyclictest4793-21apt10:55:003
3062999139,3cyclictest21921-21grep10:45:203
30629991310,2cyclictest9660-21sh12:15:003
30629991310,2cyclictest4166-21grep12:30:213
30629991310,2cyclictest2608-21expr09:55:153
30629991310,2cyclictest18561-21cut12:20:143
3062999130,1cyclictest0-21swapper/311:08:173
3062699139,2cyclictest6575-21grep08:35:132
3062699138,3cyclictest17289-21basename07:50:002
30626991312,1cyclictest23010-21diskmemload12:17:192
30626991310,2cyclictest32491-21grep08:20:192
30626991310,2cyclictest26582-21sed12:05:182
30626991310,2cyclictest22931-21sed11:05:182
30626991310,2cyclictest21377-21ls10:25:222
30626991310,2cyclictest18560-21grep09:25:262
3062699130,3cyclictest0-21swapper/210:09:272
3062499139,3cyclictest6149-21sed10:55:181
3062499139,3cyclictest6149-21sed10:55:171
3062499138,4cyclictest16980-21taskset07:46:241
3062499131,9cyclictest0-21swapper/107:35:131
30624991310,2cyclictest8747-21tr08:40:001
30624991310,2cyclictest8244-21tr07:30:141
30624991310,2cyclictest22561-21tr08:00:141
30624991310,2cyclictest14053-21ls10:40:221
3062499130,2cyclictest0-21swapper/108:20:171
3061999139,3cyclictest29180-21sleep08:14:590
3061999139,3cyclictest25655-21grep09:10:260
3061999139,3cyclictest1281-21sh11:50:170
3061999134,7cyclictest13-21ksoftirqd/009:40:400
3061999133,7cyclictest25989-21runrttasks09:50:160
3061999131,6cyclictest18387-21munin-run10:04:590
30619991311,2cyclictest8023-21apache212:39:170
30619991310,2cyclictest3960-21sed07:20:180
30619991310,2cyclictest16768-21cut08:55:170
30619991310,2cyclictest15744-21cron11:39:590
30619991310,2cyclictest1570-21grep07:15:180
3061999130,7cyclictest32706-21idleruntime-cro09:35:000
3062999129,2cyclictest770-21seq07:15:123
3062999129,2cyclictest5829-21tr07:25:163
3062999129,2cyclictest5829-21tr07:25:163
3062999129,2cyclictest28126-21cut10:10:173
3062999129,2cyclictest25124-21grep09:10:233
3062999129,2cyclictest2341-21df08:25:153
3062999129,2cyclictest19787-21cut10:05:183
3062999129,2cyclictest19241-21grep09:00:173
3062999129,2cyclictest16378-21tr08:55:153
3062999129,2cyclictest12993-21tr10:40:173
3062999129,2cyclictest11208-21ssh12:35:123
3062999129,2cyclictest1114-21sed08:20:293
3062999128,4cyclictest8023-21apache210:01:403
3062999128,2cyclictest17674-21tr12:00:163
3062999127,3cyclictest5399-21head08:30:233
3062999124,6cyclictest6569-21ssh11:15:143
3062999124,4cyclictest225-21systemd-journal08:04:593
3062999120,9cyclictest0-21swapper/311:00:393
3062999120,0cyclictest0-21swapper/311:56:043
3062699129,2cyclictest8023-21apache210:22:322
3062699129,2cyclictest5580-21ssh10:55:152
3062699129,2cyclictest5580-21ssh10:55:142
3062699129,2cyclictest3454-21sed08:25:272
3062699129,2cyclictest31393-21latency_hist11:30:002
3062699129,2cyclictest25449-21sed08:05:172
3062699129,2cyclictest24207-21wc11:45:002
3062699129,2cyclictest20592-21sort07:55:172
3062699129,2cyclictest14407-21cat10:40:272
3062699129,2cyclictest13464-21sort07:40:162
3062699129,2cyclictest12447-21sed12:35:202
3062699129,2cyclictest0-21swapper/209:50:512
3062699128,3cyclictest3062-21pmu-power12:10:222
3062699128,3cyclictest25163-21sh09:30:162
3062699126,6cyclictest0-21swapper/209:15:032
3062699126,6cyclictest0-21swapper/209:15:032
3062699121,8cyclictest225-21systemd-journal09:10:002
3062699120,2cyclictest0-21swapper/209:38:062
3062699120,11cyclictest19165-21grep09:00:172
3062699120,11cyclictest18396-21cat09:00:002
3062499129,2cyclictest9617-21cat11:35:261
3062499129,2cyclictest4992-21ssh10:15:291
3062499129,2cyclictest28111-21sed10:10:171
3062499129,2cyclictest19310-21grep12:20:171
3062499129,2cyclictest1049-21tr07:15:161
3062499129,2cyclictest10376-21wc09:40:201
3062499129,2cyclictest0-21swapper/111:34:481
3062499128,3cyclictest19300-21sed09:00:181
3062499128,3cyclictest19141-21cat12:00:271
3062499128,3cyclictest1053-21tr09:35:151
3062499120,8cyclictest0-21swapper/109:20:131
3061999129,2cyclictest32712-21head11:30:170
3061999129,2cyclictest2381-21ssh12:30:010
3061999129,2cyclictest20627-21apt09:05:010
3061999129,2cyclictest19274-21tr10:05:160
3061999129,2cyclictest15856-21ls08:55:000
3061999129,2cyclictest12120-21head08:45:170
3061999129,2cyclictest12092-21head07:35:260
3061999129,2cyclictest10963-21sed12:15:180
3061999128,3cyclictest24926-21grep08:05:150
3061999128,3cyclictest13953-21grep10:40:210
3061999127,5cyclictest13-21ksoftirqd/011:28:410
3061999124,3cyclictest225-21systemd-journal10:13:250
30619991210,1cyclictest16219-21cat09:25:000
3062999118,2cyclictest9844-21chrt07:31:293
3062999118,2cyclictest8875-21cat08:40:003
3062999118,2cyclictest7629-21ssh11:35:043
3062999118,2cyclictest5603-21tr10:55:143
3062999118,2cyclictest5603-21tr10:55:143
3062999118,2cyclictest540-21awk07:15:003
3062999118,2cyclictest5382-21ssh11:33:293
3062999118,2cyclictest30220-21sed10:30:253
3062999118,2cyclictest30104-21grep08:15:193
3062999118,2cyclictest29957-21sh12:07:113
3062999118,2cyclictest28999-21seq12:25:343
3062999118,2cyclictest28560-21sh09:31:443
3062999118,2cyclictest26546-21ssh10:29:023
3062999118,2cyclictest23304-21sh11:25:023
3062999118,2cyclictest23116-21cut09:10:003
3062999118,2cyclictest21531-21sed07:55:263
3062999118,2cyclictest21218-21ssh09:46:453
3062999118,2cyclictest14629-21cat08:50:193
3062999118,2cyclictest14132-21ssh09:42:443
3062999118,2cyclictest14126-21ssh10:20:493
3062999117,2cyclictest3248-21tr07:20:143
3062999116,4cyclictest39-21ksoftirqd/311:10:123
3062699118,2cyclictest9644-21expr09:40:152
3062699118,2cyclictest8886-21ssh11:55:082
3062699118,2cyclictest801-21chrt07:15:142
3062699118,2cyclictest4790-21ssh11:33:032
3062699118,2cyclictest464-21taskset07:14:362
3062699118,2cyclictest29719-21grep08:15:162
3062699118,2cyclictest28879-21ssh09:12:552
3062699118,2cyclictest24451-21cat08:04:592
3062699118,2cyclictest24360-21ssh11:45:022
3062699118,2cyclictest23997-21ssh12:23:112
3062699118,2cyclictest20811-21ssh10:45:042
3062699118,2cyclictest13584-21ssh11:00:072
3062699113,8cyclictest56550irq/124-enp0s3109:21:142
3062699110,10cyclictest17132-21ssh12:00:022
3062499118,2cyclictest6026-21expr07:25:171
3062499118,2cyclictest6026-21expr07:25:161
3062499118,2cyclictest5969-21taskset08:31:121
3062499118,2cyclictest5476-21seq12:30:591
3062499118,2cyclictest3261-21ssh10:53:491
3062499118,2cyclictest30673-21ssh11:10:131
3062499118,2cyclictest27361-21cat08:10:151
3062499118,2cyclictest25674-21sh12:05:111
3062499118,2cyclictest25161-21expr11:45:161
3062499118,2cyclictest23494-21ssh09:48:311
3062499118,2cyclictest2235-21smart_sda11:50:261
3062499118,2cyclictest21558-21tune2fs09:05:161
3062499118,2cyclictest18325-21head07:50:161
3062499118,2cyclictest17643-21ssh10:04:261
3062499118,2cyclictest17062-21expr11:40:151
3062499118,2cyclictest10956-21sed12:15:181
3062499117,3cyclictest3021-21tr08:25:221
3062499117,3cyclictest14435-21grep07:40:261
3062499117,2cyclictest10877-21kworker/1:108:50:011
3062499116,4cyclictest29929-21ssh09:32:501
30624991110,1cyclictest0-21swapper/110:46:591
3062499110,9cyclictest24226-21kworker/1:108:05:141
3061999119,1cyclictest31412-21head11:10:160
3061999119,1cyclictest20694-21sort10:25:170
3061999118,2cyclictest9095-21sed07:30:190
3061999118,2cyclictest7567-21ssh10:36:280
3061999118,2cyclictest6392-21hostname07:25:190
3061999118,2cyclictest6392-21hostname07:25:190
3061999118,2cyclictest606-21cat07:15:000
3061999118,2cyclictest5708-21ssh12:31:100
3061999118,2cyclictest4297-21ssh12:10:560
3061999118,2cyclictest3765-21ssh09:36:130
3061999118,2cyclictest3700-21expr10:15:180
3061999118,2cyclictest26801-21ssh11:45:350
3061999118,2cyclictest21734-21sh10:45:190
3061999118,2cyclictest20377-21ssh09:46:050
3061999118,2cyclictest2017-21ssh09:15:430
3061999118,2cyclictest2017-21ssh09:15:420
3061999118,2cyclictest19298-21grep12:20:170
3061999118,2cyclictest172912sleep012:00:090
3061999118,2cyclictest17056-21expr09:25:150
3061999117,3cyclictest24413-21cron08:04:590
3061999110,10cyclictest0-21swapper/010:22:160
3062999108,1cyclictest27061-21apache_processe08:10:133
3062999108,1cyclictest25770-21wc11:45:203
3062999108,1cyclictest18869-21awk07:50:223
3062999108,1cyclictest170352sleep307:47:073
3062999108,1cyclictest16980-21expr09:25:153
3062999108,1cyclictest147072sleep307:42:433
3062999107,2cyclictest13120-21taskset08:45:323
3062699108,1cyclictest50142sleep207:21:142
3062699108,1cyclictest18902-21readlink07:50:232
3062699108,1cyclictest11707-21expr08:45:162
3062699108,1cyclictest10360-21chrt07:35:082
3062699107,2cyclictest4177-21aten2_r7power_c08:30:152
3062699107,2cyclictest27683-21fschecks_count08:10:172
3062699107,2cyclictest0-21swapper/207:30:012
3062499108,1cyclictest19930-21basename07:55:141
3062499107,2cyclictest3482-21grep07:20:151
3062499107,2cyclictest31502-21grep07:10:171
3062499101,7cyclictest15456-21sed08:50:271
3061999108,1cyclictest7031-21grep08:35:160
3061999108,1cyclictest30936-21sed08:15:280
3061999108,1cyclictest22017-21ntp_kernel_pll_09:05:200
3061999108,1cyclictest1569-21cat08:25:010
3061999107,2cyclictest21867-21chrt07:58:100
3061999107,2cyclictest15892-21ssh11:00:340
3061999106,3cyclictest10610-21grep08:40:270
3061999100,9cyclictest4251-21cut08:30:140
306299997,1cyclictest265082sleep308:06:273
306249997,1cyclictest16119-21apache_volume08:55:131
306199996,2cyclictest15788-21grep07:45:170
306299986,1cyclictest11884-21cat07:35:263
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional