You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 13:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat May 23, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
826229379,10sleep20-21swapper/219:09:182
826229379,10sleep20-21swapper/219:09:182
815128876,8sleep00-21swapper/019:07:510
815128876,8sleep00-21swapper/019:07:510
803828465,15sleep10-21swapper/119:06:231
803828465,15sleep10-21swapper/119:06:231
804528368,9sleep30-21swapper/319:06:293
804528368,9sleep30-21swapper/319:06:283
840699280,28cyclictest0-21swapper/200:15:092
839799280,27cyclictest0-21swapper/022:27:380
839799280,26cyclictest0-21swapper/023:44:590
839799280,21cyclictest225-21systemd-journal21:02:340
839799280,1cyclictest0-21swapper/023:48:480
839799271,23cyclictest2592-21diskmemload21:42:130
839799262,23cyclictest2592-21diskmemload22:08:460
839799260,25cyclictest0-21swapper/021:55:000
839799250,24cyclictest2592-21diskmemload23:10:150
839799250,24cyclictest2592-21diskmemload22:12:420
839799250,24cyclictest0-21swapper/021:45:150
839799250,24cyclictest0-21swapper/021:15:130
839799250,1cyclictest0-21swapper/019:20:120
840199240,24cyclictest0-21swapper/122:30:271
840199240,24cyclictest0-21swapper/122:30:271
839799240,23cyclictest0-21swapper/023:28:500
839799240,1cyclictest0-21swapper/023:55:130
839799240,1cyclictest0-21swapper/021:35:120
839799240,0cyclictest0-21swapper/023:18:010
839799240,0cyclictest0-21swapper/022:38:560
8401992321,2cyclictest26047-21kworker/u8:221:02:011
839799231,21cyclictest2592-21diskmemload00:33:110
839799231,20cyclictest13750-21grep23:30:260
839799230,22cyclictest0-21swapper/022:41:520
839799230,22cyclictest0-21swapper/021:05:130
839799230,22cyclictest0-21swapper/019:25:130
839799230,20cyclictest2592-21diskmemload22:52:480
840899222,2cyclictest0-21swapper/300:38:063
840899220,20cyclictest0-21swapper/323:42:173
840899220,16cyclictest0-21swapper/300:11:573
840199223,15cyclictest2592-21diskmemload21:45:041
839799222,19cyclictest5494-21chrt00:24:100
839799222,18cyclictest2592-21diskmemload23:36:140
839799222,18cyclictest2592-21diskmemload22:04:480
839799221,19cyclictest2592-21diskmemload22:23:100
839799221,18cyclictest0-21swapper/021:10:140
839799220,21cyclictest0-21swapper/023:01:520
839799220,21cyclictest0-21swapper/000:00:140
840699210,21cyclictest0-21swapper/200:08:272
840199211,4cyclictest141rcu_preempt22:38:111
840199210,21cyclictest0-21swapper/123:03:261
840199210,0cyclictest0-21swapper/121:36:391
839799211,4cyclictest2592-21diskmemload23:07:540
839799211,20cyclictest0-21swapper/021:22:510
839799211,1cyclictest2592-21diskmemload21:32:110
839799210,20cyclictest2592-21diskmemload00:14:170
839799210,20cyclictest0-21swapper/022:32:090
839799210,20cyclictest0-21swapper/022:32:080
839799210,1cyclictest2592-21diskmemload00:25:140
839799210,1cyclictest0-21swapper/019:40:160
839799210,18cyclictest0-21swapper/000:36:320
840899200,19cyclictest0-21swapper/321:11:453
840699200,19cyclictest0-21swapper/200:24:162
840699200,18cyclictest0-21swapper/222:31:452
840699200,18cyclictest0-21swapper/222:31:452
840199200,6cyclictest0-21swapper/122:14:031
839799202,16cyclictest0-21swapper/021:30:010
839799201,16cyclictest27628-21ssh23:20:150
839799200,2cyclictest2592-21diskmemload00:19:460
839799200,1cyclictest0-21swapper/023:51:190
839799200,1cyclictest0-21swapper/022:19:320
839799200,17cyclictest0-21swapper/022:55:420
8408991919,0cyclictest0-21swapper/322:43:093
840899190,1cyclictest0-21swapper/321:35:053
8406991918,1cyclictest0-21swapper/221:28:352
8406991912,5cyclictest17491-21apache223:05:132
840699190,19cyclictest0-21swapper/222:38:022
8401991917,1cyclictest0-21swapper/122:55:081
840199190,3cyclictest0-21swapper/123:46:021
840199190,19cyclictest0-21swapper/122:19:111
840199190,18cyclictest0-21swapper/122:23:091
840199190,17cyclictest0-21swapper/121:55:491
839799190,1cyclictest0-21swapper/022:49:570
839799190,1cyclictest0-21swapper/000:08:420
839799190,15cyclictest10318-21head19:10:260
840899180,1cyclictest0-21swapper/323:57:313
840899180,0cyclictest0-21swapper/321:53:093
8406991818,0cyclictest0-21swapper/221:39:042
840199183,10cyclictest4593-21pmu-power23:25:201
840199181,17cyclictest0-21swapper/123:22:531
8408991717,0cyclictest0-21swapper/322:23:363
8408991715,2cyclictest0-21swapper/321:23:523
840899170,1cyclictest0-21swapper/321:44:373
840699171,1cyclictest141rcu_preempt21:49:152
8401991713,3cyclictest30562-21sh00:19:591
839799178,6cyclictest9445-21taskset20:18:540
839799173,7cyclictest13137-21munin-run19:20:010
839799172,14cyclictest2592-21diskmemload21:55:070
839799170,1cyclictest0-21swapper/020:40:150
8408991614,2cyclictest39-21ksoftirqd/323:00:133
840899160,1cyclictest0-21swapper/322:31:143
840899160,1cyclictest0-21swapper/322:31:133
840699162,4cyclictest24859-21/usr/sbin/munin22:40:182
8406991616,0cyclictest0-21swapper/221:15:452
840699160,16cyclictest0-21swapper/222:52:202
8401991615,1cyclictest0-21swapper/100:11:571
8401991614,1cyclictest0-21swapper/121:10:341
8401991613,2cyclictest22730-21sendmail19:40:011
840199160,15cyclictest0-21swapper/122:43:521
840199160,15cyclictest0-21swapper/121:51:091
839799161,10cyclictest225-21systemd-journal19:51:030
839799160,0cyclictest0-21swapper/019:35:120
8408991514,1cyclictest0-21swapper/322:36:093
840899150,2cyclictest0-21swapper/323:21:453
840699156,5cyclictest225-21systemd-journal19:20:002
840699150,15cyclictest0-21swapper/221:43:432
840699150,0cyclictest0-21swapper/222:15:242
8401991511,1cyclictest6204-21ntp_states20:10:201
839799159,5cyclictest28074-21wc20:55:240
839799152,11cyclictest21923-21/usr/sbin/munin20:45:130
839799151,13cyclictest18898-21fschecks_time19:30:160
839799150,13cyclictest25418-21proc_pri20:50:370
840899148,5cyclictest11832-21sh22:50:523
840899144,6cyclictest22340-21aten2_r7power_p22:00:143
840899141,13cyclictest0-21swapper/300:21:563
8408991411,2cyclictest6539-21sed23:45:263
8408991411,2cyclictest31875-21systemd21:02:203
840699143,9cyclictest4984-21ssh22:28:052
8406991414,0cyclictest0-21swapper/223:45:132
8406991413,1cyclictest0-21swapper/221:52:462
840699140,9cyclictest0-21swapper/223:36:562
840699140,14cyclictest0-21swapper/200:31:082
840699140,12cyclictest0-21swapper/222:57:182
840699140,12cyclictest0-21swapper/221:55:122
840699140,11cyclictest0-21swapper/222:20:022
8401991414,0cyclictest0-21swapper/123:41:211
8401991413,1cyclictest0-21swapper/122:28:551
8401991412,1cyclictest2592-21diskmemload00:29:091
8401991410,3cyclictest25803-21grep20:50:411
8401991410,3cyclictest16028-21tr19:25:141
8401991410,3cyclictest15084-21grep19:20:261
840199140,2cyclictest0-21swapper/123:14:241
8397991410,2cyclictest2968-21grep20:05:150
8408991310,2cyclictest6673-21grep20:10:253
8408991310,2cyclictest3514-21head23:05:213
8408991310,2cyclictest29400-21grep21:25:183
8408991310,2cyclictest28690-21cat20:59:593
8408991310,2cyclictest24920-21grep20:50:323
8408991310,2cyclictest23025-1kworker/u9:021:30:123
8408991310,2cyclictest15889-21grep19:25:133
8408991310,2cyclictest13249-21grep23:50:163
8408991310,2cyclictest11303-21grep23:10:163
840699139,3cyclictest23241-21grep19:40:152
840699139,3cyclictest19726-21grep23:15:152
840699139,2cyclictest29212-21grep21:00:152
840699136,5cyclictest10355-21chrt20:20:132
8406991312,1cyclictest0-21swapper/223:11:382
8406991310,2cyclictest31416-21ssh22:05:162
840199138,4cyclictest26766-21turbostat.cron23:19:591
840199133,8cyclictest38150irq/131-snd_hda21:15:171
840199131,8cyclictest23459-21latency_hist00:35:011
8401991311,2cyclictest0-21swapper/123:55:451
8401991310,2cyclictest23089-21sed22:00:171
8401991310,2cyclictest16288-21grep20:30:261
8401991310,2cyclictest10531-21tr22:50:191
840199130,1cyclictest141rcu_preempt19:50:141
840199130,1cyclictest0-21swapper/123:06:551
840199130,13cyclictest0-21swapper/123:39:471
840199130,13cyclictest0-21swapper/100:24:221
840199130,12cyclictest2592-21diskmemload21:27:451
839799139,3cyclictest26990-21grep19:45:250
839799137,5cyclictest7346-21cat20:14:590
839799137,5cyclictest32015-21taskset19:57:280
839799131,9cyclictest1616-21/usr/sbin/munin20:00:220
840899129,2cyclictest7967-21sh22:10:193
840899129,2cyclictest32630-21df22:25:143
840899129,2cyclictest20488-21cut23:35:123
840899129,2cyclictest1664-21grep21:05:213
840899129,2cyclictest1556-21wc20:00:213
840899129,2cyclictest12051-21tr23:30:143
840899121,10cyclictest0-21swapper/300:25:133
8408991210,1cyclictest16187-21grep20:30:253
840899120,11cyclictest30368-21grep19:55:143
840699129,2cyclictest7458-21sort22:10:162
840699129,2cyclictest3851-21head21:10:152
840699129,2cyclictest30726-21tail00:00:172
840699129,2cyclictest23000-21grep20:45:212
840699129,2cyclictest19934-21grep19:30:252
840699126,5cyclictest22630-21munin-node20:37:142
840699125,3cyclictest0-21swapper/221:31:012
840699123,5cyclictest225-21systemd-journal23:29:592
840699120,3cyclictest0-21swapper/223:44:152
840699120,2cyclictest0-21swapper/221:23:322
840699120,11cyclictest7605-21grep00:25:152
840199129,2cyclictest3431-21tail20:05:181
840199129,2cyclictest31810-21grep19:55:251
840199129,2cyclictest31585-21ssh22:05:171
840199129,2cyclictest21686-21grep21:40:141
840199129,2cyclictest13009-21cut23:30:191
840199129,2cyclictest1049-21cut20:00:161
840199129,1cyclictest0-21swapper/121:31:051
840199120,8cyclictest38150irq/131-snd_hda00:00:141
840199120,11cyclictest1578-21chrt21:05:201
839799127,4cyclictest10142-21ls20:20:110
839799126,5cyclictest13654-21sort20:25:260
839799120,11cyclictest0-21swapper/020:35:120
840899119,2cyclictest39-21ksoftirqd/321:17:403
840899119,1cyclictest24095-21cut19:40:193
840899119,1cyclictest19688-21grep20:40:143
840899118,2cyclictest8892-21taskset00:06:143
840899118,2cyclictest30908-21cut22:05:143
840899118,2cyclictest30288-21grep21:45:183
840899118,2cyclictest30083-21ssh00:19:383
840899118,2cyclictest2979-21ssh00:03:103
840899118,2cyclictest2905-21cut20:05:143
840899118,2cyclictest24106-21cut20:50:013
840899118,2cyclictest20810-21sed23:15:253
840899118,2cyclictest20198-21taskset19:33:293
840899118,2cyclictest19248-21cat20:40:003
840899118,2cyclictest19081-21rm00:31:393
840899118,2cyclictest18316-21ssh21:57:213
840899118,2cyclictest17654-21cat22:55:113
840899118,2cyclictest11281-21tr19:15:153
840899115,2cyclictest25370-21apt-get19:45:123
840899114,1cyclictest27991-21cut19:50:143
8408991111,0cyclictest0-21swapper/323:25:173
840899110,4cyclictest225-21systemd-journal19:40:013
840699118,2cyclictest31237-21ntpq19:55:192
840699118,2cyclictest29333-21ssh23:01:202
840699118,2cyclictest28018-21sort20:55:232
840699118,2cyclictest27750-21ssh23:58:482
840699118,2cyclictest27234-21ssh23:20:112
840699118,2cyclictest26142-21grep19:45:162
840699118,2cyclictest26074-21taskset20:52:392
840699118,2cyclictest2573-21cron21:10:002
840699118,2cyclictest24893-21ssh22:00:572
840699118,2cyclictest24079-21ssh00:35:122
840699118,2cyclictest20453-21cat20:40:192
840699118,2cyclictest1569-21cut22:45:162
840699118,2cyclictest15255-21taskset19:21:442
840699118,2cyclictest15124-21df20:30:142
840699118,2cyclictest14220-21bc00:10:112
840699118,2cyclictest12985-21sh23:50:132
840699118,2cyclictest12894-21ssh23:30:192
840199119,1cyclictest25-21ksoftirqd/100:36:151
840199119,1cyclictest0-21swapper/120:55:121
840199118,2cyclictest954-21ssh22:45:131
840199118,2cyclictest9536-21cron20:20:001
840199118,2cyclictest25655-21tr19:45:141
840199118,2cyclictest24564-21ssh21:22:271
840199118,2cyclictest23813-21sort19:40:181
840199118,2cyclictest13654-21grep23:50:161
840199118,2cyclictest10701-21chrt19:14:371
840199118,2cyclictest10214-21expr20:20:111
840199117,3cyclictest22769-21wc20:45:171
840199117,3cyclictest19629-21tr20:40:141
840199116,4cyclictest6691-21tr00:05:171
840199116,4cyclictest14270-21chrt20:29:371
839799119,1cyclictest15303-21grep20:30:170
840899108,2cyclictest10575-21taskset19:12:593
840899108,1cyclictest9126-21grep20:15:263
840899108,1cyclictest14454-21wc20:30:003
840899107,2cyclictest2154-21ntpq22:45:193
840899107,2cyclictest13399-21basename19:20:123
840899107,2cyclictest11958-21sleep20:25:003
840899102,5cyclictest16833-21timerandwakeup22:15:253
840699108,1cyclictest9792-21ntpq19:10:212
840699108,1cyclictest7493-21cut20:15:132
840699108,1cyclictest3369-21cut20:05:172
840699108,1cyclictest29447-21awk19:50:262
840699108,1cyclictest226552sleep219:39:342
840699108,1cyclictest17134-21cut19:25:212
840699107,2cyclictest5312-21tr20:10:142
840699107,2cyclictest403-21grep20:00:142
840699107,2cyclictest13603-21smartctl20:25:252
840199108,1cyclictest18971-21taskset20:38:131
840199108,1cyclictest11229-21ls19:15:151
84019986,1cyclictest18809-21forks19:30:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional