You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-13 - 06:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Sep 13, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1725129262,12sleep30-21swapper/319:07:313
1716529070,15sleep10-21swapper/119:06:251
1724128874,9sleep00-21swapper/019:07:250
1582728066,9sleep20-21swapper/219:05:132
1752899348,24cyclictest24442-21gltestperf22:35:190
1752899290,5cyclictest4428-21df_abs21:25:150
1752899273,17cyclictest18868-21cron20:20:000
1752899271,4cyclictest10753-21diskmemload00:01:450
1752899271,4cyclictest10753-21diskmemload00:01:450
1752899270,3cyclictest0-21swapper/021:16:140
1752899261,4cyclictest0-21swapper/022:30:120
17534992418,5cyclictest12643-21latency_hist00:25:012
17534992411,12cyclictest25804-21idleruntime22:55:222
1752899244,2cyclictest0-21swapper/021:32:190
1752899242,3cyclictest6098-21head19:50:230
1752899241,22cyclictest0-21swapper/000:05:130
1752899240,4cyclictest0-21swapper/021:45:280
1753799231,2cyclictest141rcu_preempt20:40:143
1753499237,15cyclictest6460-21switchtime21:45:262
1753499232,1cyclictest141rcu_preempt23:14:072
1753499230,19cyclictest0-21swapper/221:30:172
17531992322,1cyclictest0-21swapper/123:20:561
1753799220,21cyclictest14513-21pmu-power21:50:223
1753799220,1cyclictest0-21swapper/300:30:233
1752899221,3cyclictest18236-21ssh22:50:320
1752899221,20cyclictest0-21swapper/023:50:160
1752899220,3cyclictest0-21swapper/023:05:520
17534992118,2cyclictest22007-21cstates00:30:202
17534992118,2cyclictest14631-21cstates22:10:162
1753199214,1cyclictest2645-21apt-get23:40:131
1752899212,3cyclictest17836-21munin-run23:29:590
1752899210,5cyclictest0-21swapper/021:39:090
1753799200,19cyclictest0-21swapper/320:30:133
17534992017,2cyclictest1142-21runrttasks22:04:522
1753499201,4cyclictest32-21ksoftirqd/223:53:122
1753199200,2cyclictest0-21swapper/122:49:581
1752899208,9cyclictest20541-21sh00:10:130
17528992014,6cyclictest13-21ksoftirqd/021:40:350
1752899200,5cyclictest31998-21aten2_r7power_c22:40:140
1752899200,4cyclictest10753-21diskmemload21:22:110
1752899200,3cyclictest0-21swapper/023:14:260
1753799192,0cyclictest141rcu_preempt22:57:223
17537991917,1cyclictest39-21ksoftirqd/321:50:003
1753499198,1cyclictest32-21ksoftirqd/222:42:212
17534991917,1cyclictest22869-21pmu-power21:55:222
17531991918,1cyclictest0-21swapper/123:49:401
1752899194,1cyclictest10753-21diskmemload22:00:490
1752899191,2cyclictest2197-21tr20:50:180
1752899191,2cyclictest1693-21aten2_r7power_c23:20:120
1752899191,2cyclictest0-21swapper/019:10:150
1752899190,3cyclictest0-21swapper/022:19:450
1752899190,2cyclictest0-21swapper/022:56:030
1752899190,1cyclictest0-21swapper/023:42:190
1753799182,1cyclictest141rcu_preempt23:23:243
17537991813,5cyclictest39-21ksoftirqd/323:38:073
17534991816,1cyclictest0-21swapper/222:22:532
17534991814,1cyclictest32-21ksoftirqd/222:28:342
17531991817,1cyclictest0-21swapper/122:11:241
17531991816,1cyclictest20515-21apt-get21:35:151
1753199180,2cyclictest0-21swapper/122:21:151
1753199180,17cyclictest0-21swapper/122:36:291
1752899188,6cyclictest27496-21sh00:33:330
1752899186,6cyclictest1009-21apt-get20:50:000
1752899184,6cyclictest477-21rs:main0
1752899184,2cyclictest0-21swapper/020:24:240
1752899184,2cyclictest0-21swapper/020:24:230
1752899183,2cyclictest10753-21diskmemload23:55:500
1752899183,14cyclictest0-21swapper/000:35:260
17528991810,7cyclictest18942-21ssh23:30:150
1753799173,2cyclictest141rcu_preempt22:54:133
1753799172,3cyclictest141rcu_preempt23:12:113
17537991715,1cyclictest39-21ksoftirqd/321:25:113
17534991716,1cyclictest0-21swapper/222:35:342
17531991711,5cyclictest0-21swapper/123:53:301
1753199170,1cyclictest0-21swapper/123:31:311
1752899178,5cyclictest9172-21taskset19:57:530
1752899171,3cyclictest0-21swapper/023:35:190
17528991710,6cyclictest31266-21tr22:20:130
1752899170,3cyclictest0-21swapper/020:00:210
1753799164,5cyclictest39-21ksoftirqd/322:04:023
1753799162,1cyclictest141rcu_preempt23:40:463
1753799161,1cyclictest141rcu_preempt00:18:173
1753799161,0cyclictest141rcu_preempt22:49:223
17537991610,2cyclictest141rcu_preempt22:11:253
17537991610,2cyclictest0-21swapper/321:19:203
1753799160,14cyclictest0-21swapper/321:20:153
17534991614,1cyclictest32-21ksoftirqd/221:14:522
17534991612,1cyclictest32-21ksoftirqd/222:06:012
1753499160,1cyclictest0-21swapper/223:04:002
1753499160,11cyclictest0-21swapper/200:05:142
1753199161,15cyclictest16716-21ssh21:51:431
17531991611,4cyclictest28604-21sh21:20:171
1753199160,16cyclictest0-21swapper/123:14:151
1752899169,6cyclictest8385-21munin-run21:05:000
1752899169,6cyclictest10761-21munin-run21:09:590
1752899165,2cyclictest225-21systemd-journal20:57:280
1752899164,1cyclictest10753-21diskmemload22:11:560
1752899160,16cyclictest0-21swapper/021:11:500
1752899160,15cyclictest30250-21ps20:40:190
1753799158,6cyclictest7075-21cpu22:25:133
17537991515,0cyclictest0-21swapper/321:59:103
1753799151,13cyclictest0-21swapper/323:28:123
17537991511,1cyclictest39-21ksoftirqd/322:17:393
1753499157,2cyclictest27473-21munin-run21:19:592
1753499152,3cyclictest141rcu_preempt23:40:152
17534991512,2cyclictest1235-21ntpd23:23:152
1753499151,1cyclictest0-21swapper/222:32:542
17534991511,3cyclictest4582-21grep20:55:182
1753199150,1cyclictest12810-21kworker/1:021:59:091
1753199150,1cyclictest0-21swapper/123:36:341
1753199150,1cyclictest0-21swapper/123:06:001
1753199150,15cyclictest0-21swapper/123:01:051
1752899159,5cyclictest15103-21grep20:10:180
1752899158,6cyclictest2019-21basename19:44:590
1752899157,7cyclictest7962-21ssh22:45:100
1752899157,7cyclictest27949-21expr19:30:150
1752899157,7cyclictest21033-21date21:55:010
1752899157,7cyclictest13999-21sed00:25:160
1752899157,7cyclictest10690-21sh22:26:500
1752899156,2cyclictest7502-21readlink22:05:220
1752899156,2cyclictest25320-21cut19:25:170
1752899155,2cyclictest24437-21tr20:30:140
1752899152,2cyclictest225-21systemd-journal20:40:010
1752899150,4cyclictest225-21systemd-journal22:00:000
1752899150,14cyclictest13984-21taskset20:08:250
1753799149,2cyclictest0-21swapper/322:05:133
1753799147,2cyclictest39-21ksoftirqd/300:07:133
1753799146,7cyclictest0-21swapper/300:27:383
17537991411,2cyclictest39-21ksoftirqd/323:55:233
17534991412,2cyclictest32-21ksoftirqd/223:16:562
17534991412,1cyclictest32-21ksoftirqd/223:39:442
1753499140,3cyclictest0-21swapper/222:50:212
1753499140,1cyclictest0-21swapper/222:19:362
17531991412,1cyclictest10753-21diskmemload22:55:491
1753199140,7cyclictest0-21swapper/122:31:521
1753199140,0cyclictest0-21swapper/121:17:391
1752899147,6cyclictest30022-21ssh23:17:140
1752899147,6cyclictest24901-21taskset19:24:550
1752899147,6cyclictest22924-21chrt20:25:220
1752899142,2cyclictest10753-21diskmemload00:20:110
17528991410,3cyclictest30943-21grep00:15:290
1753799138,4cyclictest21296-21unixbench_singl23:50:293
1753799138,2cyclictest39-21ksoftirqd/323:07:283
17537991310,2cyclictest5516-21grep20:55:283
17537991310,2cyclictest39-21ksoftirqd/320:50:013
1753799130,4cyclictest141rcu_preempt21:30:183
1753799130,13cyclictest0-21swapper/321:42:073
1753499134,8cyclictest1887-21ssh21:43:072
1753499131,1cyclictest141rcu_preempt00:10:552
17534991310,3cyclictest32-21ksoftirqd/223:55:232
17534991310,2cyclictest7873-21grep21:00:282
17534991310,2cyclictest7381-21grep19:55:172
17534991310,2cyclictest3989-21cpu21:25:122
17534991310,2cyclictest27277-21grep20:35:182
17534991310,2cyclictest19062-21grep20:20:012
17534991310,2cyclictest14110-21cron20:10:012
1753499130,13cyclictest0-21swapper/200:35:212
1753199131,12cyclictest24915-21ssh00:12:181
17531991311,1cyclictest0-21swapper/120:30:131
17531991310,2cyclictest31787-21grep22:00:271
17531991310,2cyclictest21485-21turbostat20:25:011
17531991310,2cyclictest21485-21turbostat20:25:011
17531991310,2cyclictest13903-21grep00:25:161
1753199130,2cyclictest4823-21if_err_enp2s021:25:171
1753199130,1cyclictest141rcu_preempt00:15:141
1753199130,13cyclictest0-21swapper/123:16:331
1752899134,2cyclictest0-21swapper/023:00:160
1752899132,2cyclictest225-21systemd-journal19:45:010
1752899131,4cyclictest30494-21df_abs19:35:140
1752899130,2cyclictest141rcu_preempt19:20:000
1753799129,2cyclictest31250-21cut22:20:143
1753799129,2cyclictest28342-21tail19:30:183
1753799129,2cyclictest26450-21cut23:15:183
1753799128,3cyclictest15519-21cpuspeed_turbos22:30:153
1753799128,2cyclictest55650irq/127-enp2s0-23:33:533
1753799127,1cyclictest39-21ksoftirqd/300:20:163
1753799126,2cyclictest39-21ksoftirqd/322:42:383
1753799125,3cyclictest39-21ksoftirqd/321:10:183
1753799122,7cyclictest10956-21/usr/sbin/munin23:45:263
17537991210,1cyclictest9735-21grep20:00:123
17537991210,1cyclictest23050-21grep19:20:183
17537991210,1cyclictest10808-21sed21:10:003
1753799120,3cyclictest0-21swapper/322:40:003
1753799120,11cyclictest22016-21tr20:25:163
1753499129,2cyclictest23291-21cat20:25:282
1753499129,2cyclictest1626-21sed19:40:272
1753499129,2cyclictest15120-21grep20:10:182
1753499129,2cyclictest14686-21cut21:50:232
1753499129,2cyclictest11733-21cron20:04:592
1753499129,2cyclictest10755-21awk21:09:592
1753499126,5cyclictest14600-21processes00:25:212
1753499125,1cyclictest30214-21/usr/sbin/munin00:15:222
17534991210,1cyclictest0-21swapper/223:31:232
1753499120,3cyclictest141rcu_preempt19:25:192
1753499120,1cyclictest141rcu_preempt20:55:002
1753199129,2cyclictest13107-21head21:30:201
1753199126,4cyclictest2218-21rm21:43:221
1753199123,5cyclictest10753-21diskmemload00:00:151
1753199123,5cyclictest10753-21diskmemload00:00:141
1753199122,9cyclictest21361-21ssh00:30:051
1753199122,8cyclictest141rcu_preempt00:06:361
1753199121,2cyclictest141rcu_preempt21:11:091
1753199121,1cyclictest6878-21users00:20:281
1753199121,1cyclictest16896-21ssh23:29:181
1753199121,11cyclictest10753-21diskmemload22:06:531
17531991210,1cyclictest10350-21grep20:00:161
1753199120,12cyclictest8902-21grep21:05:151
1753799119,2cyclictest30057-21cut00:35:143
1753799119,1cyclictest7914-21grep21:00:283
1753799119,1cyclictest2777-21tr19:45:173
1753799119,1cyclictest22525-21grep21:35:283
1753799119,1cyclictest2230-21grep20:50:173
1753799119,1cyclictest14486-21grep20:10:143
1753799118,2cyclictest3153-21ssh23:00:483
1753799118,2cyclictest27334-21grep20:35:183
1753799118,2cyclictest22323-21chrt19:17:373
1753799118,2cyclictest10526-21ssh00:04:183
1753799118,2cyclictest10526-21ssh00:04:173
1753799117,2cyclictest10753-21diskmemload00:10:163
1753799116,1cyclictest39-21ksoftirqd/320:00:013
1753799110,10cyclictest17079-21expr20:15:173
1753499119,1cyclictest32-21ksoftirqd/221:36:012
1753499118,2cyclictest7863-21rm22:45:042
1753499118,2cyclictest6065-21ssh00:00:472
1753499118,2cyclictest6065-21ssh00:00:462
1753499118,2cyclictest29059-21aten2_r7power_v20:40:112
1753499118,2cyclictest28816-21tail21:20:192
1753499118,2cyclictest26249-21seq20:34:352
1753499118,1cyclictest9726-21ssh23:25:012
1753499117,3cyclictest9890-21sed23:05:182
1753499117,3cyclictest23575-21head19:20:212
1753499117,3cyclictest17989-21tr19:10:172
1753499110,3cyclictest0-21swapper/223:46:222
1753199119,1cyclictest29650-21grep20:40:141
1753199118,2cyclictest7046-21sed21:00:191
1753199118,2cyclictest3887-21hostname20:55:151
1753199118,2cyclictest29704-21ssh00:35:061
1753199118,2cyclictest28033-21grep23:55:151
1753199118,2cyclictest25-21ksoftirqd/121:50:001
1753199118,2cyclictest23592-21grep19:20:201
1753199118,2cyclictest21807-21taskset20:25:161
1753199110,7cyclictest15576-21ls20:10:221
1753199110,3cyclictest27006-21chrt19:26:291
1753199110,1cyclictest0-21swapper/122:15:091
1753799108,1cyclictest27179-21taskset19:28:413
1753799108,1cyclictest20529-21ntp_states20:20:233
1753799108,1cyclictest20529-21ntp_states20:20:223
1753799108,1cyclictest126822sleep320:05:183
1753799107,2cyclictest31674-21taskset19:35:293
1753799107,2cyclictest19157-21python19:10:273
1753799102,2cyclictest0-21swapper/319:40:143
1753499108,1cyclictest3978-21grep19:45:282
1753499108,1cyclictest27996-21expr19:30:152
1753499108,1cyclictest21589-21awk19:15:222
1753499107,2cyclictest32649-21find20:45:242
1753499107,2cyclictest19659-21grep20:20:172
1753499107,2cyclictest19659-21grep20:20:162
1753499100,9cyclictest6831-21chrt19:53:182
1753199108,1cyclictest29506-21taskset19:33:071
1753199108,1cyclictest16441-21ssh22:50:161
1753199107,2cyclictest8245-21grep22:25:201
1753199107,2cyclictest30194-21tr19:35:141
1753199107,2cyclictest17051-21cut20:15:161
1753199100,9cyclictest4559-21ssh22:42:491
1753199100,6cyclictest3314-21sshd20:51:241
1753199100,1cyclictest0-21swapper/119:50:221
175379997,1cyclictest66142sleep319:50:303
175349997,1cyclictest31834-21chrt19:37:322
175319998,1cyclictest2693-21grep19:45:171
175319998,1cyclictest20541-21awk19:15:141
175319998,1cyclictest19737-21taskset19:13:291
175319998,1cyclictest1282-21grep19:40:231
175319997,1cyclictest9077-21taskset19:56:401
175319997,1cyclictest31610-21grep20:45:171
175319997,1cyclictest27054-21sed20:35:171
175319997,1cyclictest12000-21basename20:05:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional