You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 11:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Mar 16, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1222229583,8sleep00-21swapper/019:08:180
1224228774,9sleep20-21swapper/219:08:322
1213828659,9sleep30-21swapper/319:07:133
1213127959,16sleep10-21swapper/119:07:051
87242540,0sleep30-21swapper/323:45:323
1244599301,1cyclictest13423-21ssh22:50:271
1244899280,26cyclictest0-21swapper/223:27:112
1244599252,22cyclictest9305-21ssh22:10:141
1244899240,22cyclictest0-21swapper/200:21:132
12445992422,1cyclictest25-21ksoftirqd/123:21:481
1244199240,1cyclictest0-21swapper/023:51:020
1244199230,23cyclictest0-21swapper/023:55:370
1244899210,21cyclictest0-21swapper/222:34:362
1244599210,3cyclictest0-21swapper/122:55:191
12441992117,3cyclictest13-21ksoftirqd/022:30:130
1244199210,20cyclictest0-21swapper/022:27:170
1245099200,20cyclictest0-21swapper/323:40:163
12441992019,1cyclictest0-21swapper/023:37:510
1244199200,2cyclictest0-21swapper/022:02:320
1245099196,11cyclictest24902-21apt-get22:00:143
1244599195,14cyclictest9602-21ssh21:50:211
1244199197,6cyclictest14428-21munin-run23:50:010
1244199190,2cyclictest0-21swapper/022:47:370
1244199190,18cyclictest0-21swapper/000:14:410
1244199190,17cyclictest0-21swapper/023:33:510
1244199190,17cyclictest0-21swapper/023:33:510
1244899189,4cyclictest225-21systemd-journal22:00:012
1244899180,18cyclictest0-21swapper/200:38:192
12445991816,2cyclictest25-21ksoftirqd/122:30:121
12445991816,2cyclictest0-21swapper/122:39:281
12445991816,1cyclictest25-21ksoftirqd/121:29:001
12445991811,2cyclictest0-21swapper/120:25:151
1244199183,11cyclictest25487-21hddtemp_smartct00:15:180
12441991817,1cyclictest0-21swapper/021:29:100
12441991813,4cyclictest13904-21sh22:50:440
1244199181,17cyclictest0-21swapper/023:25:020
1244899170,16cyclictest0-21swapper/222:48:082
1244599175,9cyclictest25-21ksoftirqd/100:36:511
12445991715,2cyclictest5242-21diskmemload00:27:381
1244599170,2cyclictest0-21swapper/123:16:451
12441991713,2cyclictest0-21swapper/000:02:180
1244199170,1cyclictest0-21swapper/023:20:110
1245099164,11cyclictest22280-21munin-node22:05:213
12450991610,5cyclictest4973-21apache221:57:263
12448991615,1cyclictest32-21ksoftirqd/221:12:122
1244899160,16cyclictest0-21swapper/223:55:422
1244899160,14cyclictest0-21swapper/200:12:322
12445991614,1cyclictest1487-21cut21:00:161
12445991613,2cyclictest25-21ksoftirqd/100:20:031
1244599160,16cyclictest0-21swapper/123:58:271
1244599160,15cyclictest0-21swapper/122:19:041
1244599160,14cyclictest0-21swapper/122:09:101
1244199160,2cyclictest0-21swapper/000:24:490
1244199160,16cyclictest0-21swapper/022:57:080
12450991512,2cyclictest2563-21kworker/3:021:20:123
1245099150,13cyclictest17960-21cat00:10:233
1244899155,9cyclictest22280-21munin-node23:16:512
1244899150,2cyclictest0-21swapper/223:44:522
1244899150,2cyclictest0-21swapper/200:00:132
1244899150,15cyclictest0-21swapper/221:18:232
1244899150,14cyclictest0-21swapper/223:05:202
1244899150,12cyclictest0-21swapper/200:15:272
1244599158,2cyclictest25-21ksoftirqd/121:20:211
1244599153,4cyclictest0-21swapper/100:15:211
12445991514,1cyclictest25-21ksoftirqd/121:35:341
12445991513,1cyclictest31279-21ssh21:44:431
12445991512,2cyclictest0-21swapper/100:08:381
1244599150,1cyclictest0-21swapper/100:11:401
1244599150,14cyclictest0-21swapper/123:47:331
1244599150,11cyclictest38150irq/131-snd_hda23:32:311
1244599150,11cyclictest38150irq/131-snd_hda23:32:311
1244199150,3cyclictest28426-21aten2_r7power_v23:00:130
1244199150,0cyclictest0-21swapper/022:17:420
1245099149,3cyclictest21063-21wc22:55:203
1245099143,6cyclictest22201-21apt-get19:30:133
1245099141,4cyclictest0-21swapper/300:25:133
1245099141,12cyclictest20710-21latency_hist23:15:003
12450991411,2cyclictest11908-21tune2fs20:15:173
1245099141,10cyclictest0-21swapper/321:49:213
12450991410,3cyclictest24240-21grep23:35:283
12450991410,3cyclictest14888-21cron21:35:003
1244899146,5cyclictest17224-21apt-get00:30:132
12448991413,1cyclictest0-21swapper/221:50:552
12448991412,1cyclictest0-21swapper/222:55:272
1244899140,14cyclictest0-21swapper/219:34:262
1244599148,3cyclictest0-21swapper/121:58:541
12445991410,3cyclictest3662-21sed22:45:141
1244599140,2cyclictest0-21swapper/123:01:311
12441991413,1cyclictest181irq_work/023:08:550
12441991410,3cyclictest18226-21wc21:55:270
12441991410,3cyclictest13-21ksoftirqd/000:30:140
1244199140,14cyclictest0-21swapper/021:39:400
1245099139,3cyclictest3147-21wc19:55:243
1245099139,3cyclictest24854-21grep19:35:153
1245099139,3cyclictest24599-21ssh00:15:133
1245099139,3cyclictest11840-21tr22:50:153
1245099139,2cyclictest3441-21pluginstate22:25:213
1245099133,7cyclictest15965-21cron20:25:003
1245099133,4cyclictest0-21swapper/322:30:163
1245099131,3cyclictest0-21swapper/322:16:263
12450991310,2cyclictest4785-21grep20:00:183
12450991310,2cyclictest372-21taskset00:00:223
12450991310,2cyclictest32450-21sed21:25:213
12450991310,2cyclictest30597-21grep20:55:123
12450991310,2cyclictest27066-21gzip20:45:213
12450991310,2cyclictest22641-21grep20:35:263
12450991310,2cyclictest19112-21grep22:35:123
12450991310,2cyclictest10017-21grep22:10:163
12448991310,2cyclictest25236-21wc21:40:212
12448991310,2cyclictest23512-21cat23:35:222
12448991310,2cyclictest15684-21grep23:50:162
12448991310,2cyclictest15491-21sed20:20:252
12448991310,2cyclictest14798-21awk19:15:002
1244899130,1cyclictest0-21swapper/222:22:432
1244899130,1cyclictest0-21swapper/221:39:302
1244599135,7cyclictest6222-21cron20:05:011
12445991312,1cyclictest25-21ksoftirqd/123:35:331
12445991310,3cyclictest25-21ksoftirqd/121:30:241
12445991310,2cyclictest11839-21grep20:15:171
1244599130,11cyclictest0-21swapper/121:17:161
1244599130,0cyclictest0-21swapper/122:44:291
1244199139,3cyclictest22897-21wc23:15:240
1244199139,3cyclictest13655-21cat20:20:010
1244199139,3cyclictest10488-21uname22:10:200
1244199139,2cyclictest19599-21wc20:30:200
1244199139,2cyclictest15783-21grep19:15:160
12441991310,2cyclictest25601-21wc19:35:200
12441991310,2cyclictest23418-21tune2fs21:20:160
12441991310,2cyclictest19459-21cut22:35:140
12441991310,2cyclictest1694-21grep21:45:280
12441991310,2cyclictest13898-21grep19:10:210
18112120,2sleep10-21swapper/122:25:011
1245099129,2cyclictest3854-21grep22:45:143
1245099129,2cyclictest3293-21grep21:05:143
1245099129,2cyclictest23411-21grep20:40:123
1245099129,2cyclictest23070-21grep23:15:253
1245099129,2cyclictest19393-21grep20:30:183
1245099129,2cyclictest16973-21grep20:25:183
1245099129,2cyclictest15939-21ssh23:50:183
1245099123,7cyclictest17236-21ssh00:30:133
1245099120,11cyclictest5799-21ssh23:05:233
1245099120,11cyclictest0-21swapper/323:59:103
1244899129,2cyclictest7222-21head20:05:162
1244899129,2cyclictest4165-21cut21:05:192
1244899129,2cyclictest31901-21tail21:25:172
1244899129,2cyclictest30535-21sh21:25:002
1244899129,2cyclictest21507-21grep19:25:262
1244899129,2cyclictest18025-21grep22:15:142
1244899129,2cyclictest16159-21cut19:15:182
1244899129,2cyclictest11530-21tr20:15:152
1244899129,2cyclictest10637-21head22:10:202
1244899120,12cyclictest0-21swapper/200:07:142
1244899120,11cyclictest0-21swapper/220:50:142
1244599129,2cyclictest7913-21wc20:05:251
1244599129,2cyclictest5851-21cut23:25:131
1244599129,2cyclictest25-21ksoftirqd/119:15:211
1244599129,2cyclictest241rcuc/119:10:261
1244599129,2cyclictest21924-21cron19:30:001
1244599129,2cyclictest20266-21grep20:30:271
1244599129,2cyclictest13167-21df23:10:151
1244599128,4cyclictest25-21ksoftirqd/100:30:261
1244599128,3cyclictest30632-21cut20:55:121
1244599126,2cyclictest25-21ksoftirqd/122:04:431
1244599124,5cyclictest5677-21ssh23:05:231
1244599123,1cyclictest141rcu_preempt21:47:491
12445991211,1cyclictest25-21ksoftirqd/123:52:261
12445991210,1cyclictest15091-21tr20:20:211
1244599120,0cyclictest0-21swapper/120:50:211
1244199129,2cyclictest8964-21grep00:25:130
1244199129,2cyclictest6531-21cut20:05:130
1244199129,2cyclictest6388-21ssh21:10:160
1244199129,2cyclictest4096-21grep20:00:140
1244199129,2cyclictest32542-21grep23:40:260
1244199129,2cyclictest27358-21grep20:45:260
1244199129,2cyclictest13815-21chrt20:20:020
1244199129,2cyclictest13456-21head23:10:170
1244199128,3cyclictest1213-21cut21:00:140
1244199121,9cyclictest13040-31find00:09:050
1244199120,7cyclictest8757-21perf22:10:010
1244199120,7cyclictest10996-21idleruntime-cro20:15:000
1245099118,2cyclictest8580-21ssh23:26:043
1245099118,2cyclictest8336-21ssh21:10:523
1245099118,2cyclictest3926-21ssh00:21:333
1245099118,2cyclictest31036-21sed19:45:263
1245099118,2cyclictest29951-21cat23:00:253
1245099118,2cyclictest28028-21ssh00:35:503
1245099118,2cyclictest27983-21head22:40:163
1245099118,2cyclictest25198-21grep21:40:213
1245099118,2cyclictest18368-21ssh21:36:113
1245099118,2cyclictest18216-21ssh23:32:113
1245099118,2cyclictest18216-21ssh23:32:103
1245099118,2cyclictest17933-21sh21:16:523
1245099118,2cyclictest14442-21ssh21:53:473
1245099118,2cyclictest1421-21basename19:55:003
1245099118,2cyclictest1421-21basename19:55:003
1245099117,3cyclictest27682-21awk22:20:223
1245099112,5cyclictest0-21swapper/320:05:123
12450991110,1cyclictest5242-21diskmemload00:05:223
1244899118,2cyclictest7467-21sh22:09:022
1244899118,2cyclictest7392-21grep23:45:162
1244899118,2cyclictest7185-21cut21:30:142
1244899118,2cyclictest689-21grep21:00:122
1244899118,2cyclictest4541-21df20:00:162
1244899118,2cyclictest3549-21ssh21:46:512
1244899118,2cyclictest3228-21ssh23:23:292
1244899118,2cyclictest32159-21expr19:50:152
1244899118,2cyclictest32159-21expr19:50:142
1244899118,2cyclictest32063-21rm22:04:272
1244899118,2cyclictest31696-21ssh22:42:242
1244899118,2cyclictest2383-21tr22:25:142
1244899118,2cyclictest23733-21cat20:40:132
1244899118,2cyclictest21660-21sort20:35:172
1244899118,2cyclictest1797-21sh23:03:302
1244899118,2cyclictest17776-21expr19:20:132
1244899118,2cyclictest15442-21ssh23:30:212
1244899118,2cyclictest15442-21ssh23:30:212
1244899118,2cyclictest15265-21ssh23:10:562
1244899118,2cyclictest14645-21ssh22:51:192
1244899118,2cyclictest12516-21sh00:26:452
1244899112,5cyclictest0-21swapper/219:55:132
1244599118,2cyclictest9931-21cat20:10:201
1244599118,2cyclictest6614-21expr21:10:161
1244599118,2cyclictest3683-21ssh00:02:261
1244599118,2cyclictest3305-21grep21:05:141
1244599118,2cyclictest27598-21grep22:20:231
1244599112,5cyclictest31079-21cat23:40:151
1244599111,9cyclictest20640-21kworker/1:020:35:121
1244199118,2cyclictest7583-21ssh21:30:150
1244199118,2cyclictest3778-21sh20:00:010
1244199118,2cyclictest31229-21grep20:55:150
1244199118,2cyclictest29930-21chrt20:51:140
1244199118,2cyclictest29440-21wget19:45:140
1244199118,2cyclictest27691-21head19:40:150
1244199118,2cyclictest26297-21chrt22:20:150
1244199118,2cyclictest26253-21sort00:35:170
1244199118,2cyclictest25478-21cron20:45:000
1244199118,2cyclictest24560-21cut21:40:160
1244199118,2cyclictest19820-21apache_accesses19:25:120
1244199118,2cyclictest15344-21grep21:15:190
1244199117,3cyclictest18140-21fschecks_time19:20:150
1244199115,5cyclictest13-21ksoftirqd/021:50:180
1245099108,1cyclictest31606-21cat23:20:263
1245099108,1cyclictest22252sleep321:00:363
1245099108,1cyclictest20893-21sed19:25:193
1245099108,1cyclictest169992sleep319:18:013
1245099107,2cyclictest9727-21if_enp2s020:10:193
1245099107,2cyclictest28373-21expr20:50:153
1245099107,2cyclictest19329-21chrt19:22:113
1245099107,2cyclictest13219-21df19:10:153
1244899108,1cyclictest29818-21df19:45:142
1244899108,1cyclictest27557-21taskset20:46:142
1244899108,1cyclictest20795-21cut20:35:002
1244899108,1cyclictest18686-21ssh22:35:072
1244899107,2cyclictest9925-21cat20:10:212
1244899107,2cyclictest30803-21cat20:55:132
1244899107,2cyclictest26554-21taskset19:38:322
1244599108,1cyclictest31978-21cut19:50:151
1244599108,1cyclictest31978-21cut19:50:141
1244599108,1cyclictest23679-21cat20:40:131
1244599107,2cyclictest29566-21grep19:45:141
1244599107,2cyclictest27784-21basename19:40:161
1244599107,2cyclictest25-21ksoftirqd/119:40:001
1244599107,2cyclictest22905-21grep19:30:171
1244599107,1cyclictest25-21ksoftirqd/120:45:141
1244599100,9cyclictest3419-21chrt19:55:351
1244199108,1cyclictest2881-21apt-config21:05:000
1244199108,1cyclictest23627-21sed19:30:250
1244199108,1cyclictest18023-21taskset20:26:000
1244199107,2cyclictest32488-21cut19:50:170
1244199107,2cyclictest32488-21cut19:50:170
1244199104,3cyclictest27176-21apt-get22:40:130
124509997,1cyclictest26959-21taskset19:40:113
124489998,1cyclictest27172-21grep19:40:142
124489998,1cyclictest16863-21uniq20:25:172
124459996,2cyclictest17554-21tr19:20:121
124419997,1cyclictest229222sleep020:37:560
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional