You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-08 - 06:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Jun 08, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23678210289,9sleep00-21swapper/019:08:420
2216629876,16sleep30-21swapper/319:05:153
2370529464,18sleep20-21swapper/219:09:042
2141028567,13sleep10-21swapper/119:05:011
2386899265,6cyclictest141rcu_preempt23:00:130
2387199250,23cyclictest0-21swapper/123:21:491
2386899254,9cyclictest16929-21diskmemload22:10:140
2387199240,1cyclictest0-21swapper/120:45:131
23871992320,2cyclictest17358-21/usr/sbin/munin21:10:261
2387199230,21cyclictest0-21swapper/123:08:491
2387799225,7cyclictest39-21ksoftirqd/300:17:243
23871992221,1cyclictest0-21swapper/122:59:361
23871992220,1cyclictest31102-21/usr/sbin/munin23:15:181
23871992219,2cyclictest791-21/usr/sbin/munin23:55:171
23868992222,0cyclictest0-21swapper/021:25:510
23868992219,2cyclictest2531-21/usr/sbin/munin21:40:270
2387799211,3cyclictest39-21ksoftirqd/300:10:203
2387799211,3cyclictest39-21ksoftirqd/300:10:203
23875992119,1cyclictest15472-21/usr/sbin/munin23:25:182
2386899214,6cyclictest141rcu_preempt21:55:370
23868992119,1cyclictest29415-21/usr/sbin/munin22:35:190
23868992119,1cyclictest21101-21/usr/sbin/munin22:30:150
23868992119,1cyclictest17421-21/usr/sbin/munin00:05:150
2386899210,1cyclictest0-21swapper/022:01:320
2387799206,2cyclictest381rcuc/322:30:143
23877992015,5cyclictest27503-21kworker/u8:023:09:183
23877992015,2cyclictest381rcuc/322:45:253
23875992017,2cyclictest17421-21/usr/sbin/munin00:05:272
23871992018,1cyclictest9182-21/usr/sbin/munin00:00:261
23871992011,1cyclictest141rcu_preempt22:33:231
2387199200,1cyclictest141rcu_preempt22:06:491
2387199200,0cyclictest141rcu_preempt21:29:431
23868992020,0cyclictest0-21swapper/021:17:380
2386899200,3cyclictest0-21swapper/023:23:560
2386899200,0cyclictest0-21swapper/021:23:320
23877991918,1cyclictest381rcuc/300:28:503
23877991918,1cyclictest0-21swapper/323:13:043
23877991916,2cyclictest32138-21/usr/sbin/munin23:35:143
23877991916,2cyclictest28491-21/usr/sbin/munin22:15:213
23877991916,1cyclictest18794-21http21:50:003
23877991914,5cyclictest39-21ksoftirqd/322:01:383
2387799190,3cyclictest0-21swapper/323:53:413
2387799190,1cyclictest141rcu_preempt00:00:583
23875991918,1cyclictest0-21swapper/220:38:592
23875991917,1cyclictest10078-21/usr/sbin/munin00:20:172
2387199193,10cyclictest141rcu_preempt21:39:591
2387199191,2cyclictest141rcu_preempt22:46:371
2387199190,1cyclictest0-21swapper/121:19:591
115242190,0sleep20-21swapper/222:43:412
23877991816,1cyclictest381rcuc/319:40:013
23877991816,1cyclictest16929-21diskmemload21:17:573
23877991816,1cyclictest10078-21/usr/sbin/munin00:20:203
23877991815,2cyclictest16929-21diskmemload22:27:563
2387799180,1cyclictest0-21swapper/323:00:253
2387599183,2cyclictest141rcu_preempt21:32:272
23875991816,1cyclictest1510-21/usr/sbin/munin21:20:182
23875991816,1cyclictest13656-21/usr/sbin/munin22:45:152
23871991817,1cyclictest0-21swapper/123:52:231
23871991814,3cyclictest25-21ksoftirqd/122:50:201
23871991810,5cyclictest25-21ksoftirqd/122:04:571
23868991815,2cyclictest1815-21/usr/sbin/munin00:15:170
2386899180,2cyclictest0-21swapper/023:06:070
2387799179,8cyclictest0-21swapper/322:24:343
23877991716,1cyclictest2549-21/usr/sbin/munin00:35:203
23877991716,1cyclictest0-21swapper/300:30:483
23877991715,1cyclictest17497-21apt-get00:05:143
23877991715,1cyclictest0-21swapper/323:31:393
23877991712,4cyclictest39-21ksoftirqd/321:23:463
23877991710,6cyclictest9096-21apache221:41:373
2387799170,1cyclictest0-21swapper/322:11:283
2387599171,13cyclictest0-21swapper/220:25:152
2387599170,13cyclictest0-21swapper/219:40:132
2387199173,0cyclictest141rcu_preempt23:35:351
23871991715,1cyclictest12831-21/usr/sbin/munin22:25:201
23871991710,5cyclictest16455-21ssh23:45:121
2387199170,16cyclictest0-21swapper/100:18:061
2386899175,7cyclictest15961-21munin-run23:45:000
23868991716,1cyclictest0-21swapper/023:27:180
23868991715,1cyclictest32138-21/usr/sbin/munin23:35:180
2386899170,17cyclictest0-21swapper/021:33:530
2387799163,11cyclictest0-21swapper/322:38:373
23877991615,1cyclictest16929-21diskmemload21:32:263
23877991615,1cyclictest13007-21kworker/u8:121:57:513
23877991614,1cyclictest791-21/usr/sbin/munin23:55:203
23877991610,5cyclictest16929-21diskmemload22:06:453
23875991614,1cyclictest27773-21/usr/sbin/munin21:55:232
23875991614,1cyclictest22838-21/usr/sbin/munin23:10:202
2387199161,0cyclictest141rcu_preempt00:37:551
2387199160,1cyclictest0-21swapper/122:10:491
2386899161,7cyclictest4976-21munin-run20:45:000
23868991616,0cyclictest0-21swapper/000:29:090
23868991612,3cyclictest13-21ksoftirqd/022:20:110
2386899160,16cyclictest0-21swapper/021:12:410
2387799157,8cyclictest0-21swapper/322:58:273
23877991512,2cyclictest16929-21diskmemload23:40:243
23877991510,5cyclictest39-21ksoftirqd/322:52:543
2387799150,2cyclictest0-21swapper/320:00:193
2387799150,14cyclictest11116-21wc21:25:193
23875991513,1cyclictest26651-21/usr/sbin/munin21:35:162
23875991512,2cyclictest1815-21/usr/sbin/munin00:15:242
2387199159,2cyclictest6486-21grep23:00:141
2387199155,2cyclictest141rcu_preempt21:48:111
2387199152,12cyclictest1357-21if_err_enp2s020:35:191
2387199151,1cyclictest16929-21diskmemload00:09:381
2387199150,14cyclictest0-21swapper/100:25:031
2386899151,8cyclictest16966-21munin-run00:05:000
2386899151,2cyclictest537-21snmpd00:31:480
23868991510,4cyclictest0-21swapper/000:10:140
23868991510,4cyclictest0-21swapper/000:10:140
2387799144,2cyclictest225-21systemd-journal20:45:003
23877991412,1cyclictest19421-21/usr/sbin/munin21:50:243
23877991411,2cyclictest2581-21sendmail20:40:003
2387599149,4cyclictest21452-21tr22:30:152
2387599149,4cyclictest16892-21sh00:04:582
2387599142,5cyclictest0-21swapper/222:10:212
2387599142,12cyclictest0-21swapper/200:37:152
23875991411,2cyclictest25518-21ls19:10:252
23875991410,3cyclictest18677-21grep21:10:202
2387599140,2cyclictest0-21swapper/223:45:132
2387599140,1cyclictest0-21swapper/221:54:522
2387599140,14cyclictest0-21swapper/223:21:462
23871991411,2cyclictest1375-21sh00:15:001
23871991411,2cyclictest1375-21sh00:15:001
2387199141,12cyclictest10972-21kworker/1:222:20:161
23871991411,2cyclictest0-21swapper/123:25:451
2387199140,13cyclictest24070-21cat19:10:101
2386899147,5cyclictest16395-21ssh22:45:590
23868991412,1cyclictest785-21kworker/u8:122:40:590
2387799139,3cyclictest24508-21sed20:15:263
2387799136,6cyclictest0-21swapper/323:20:243
2387799133,8cyclictest39-21ksoftirqd/321:10:133
23877991311,2cyclictest0-21swapper/319:15:133
23877991310,2cyclictest6199-21tr19:40:153
23877991310,2cyclictest381rcuc/323:20:003
23877991310,2cyclictest28887-21sed20:25:183
23877991310,2cyclictest18122-21ssh23:45:253
23877991310,2cyclictest16929-21diskmemload23:29:493
2387799130,4cyclictest0-21swapper/322:40:143
2387599139,3cyclictest23035-21wc20:15:152
2387599137,4cyclictest8070-21/usr/sbin/munin23:40:162
2387599135,4cyclictest225-21systemd-journal22:20:002
2387599132,1cyclictest141rcu_preempt22:00:502
23875991311,2cyclictest23862-21cyclictest22:26:562
23875991310,2cyclictest18822-21wc21:50:002
2387599130,11cyclictest0-21swapper/200:10:142
2387599130,11cyclictest0-21swapper/200:10:142
23871991311,1cyclictest25-21ksoftirqd/122:38:311
23871991310,2cyclictest6322-21ssh21:41:491
23871991310,2cyclictest3304-21cron19:35:001
23871991310,2cyclictest30235-21cron20:30:001
23871991310,2cyclictest20326-21grep21:50:171
2387199130,4cyclictest9445-21perf21:25:001
2386899139,3cyclictest5240-21grep19:35:250
2386899135,4cyclictest225-21systemd-journal20:50:000
2386899132,10cyclictest26480-21apache_volume19:15:130
2386899131,6cyclictest25033-21sh23:10:330
23868991310,2cyclictest4478-21ssh00:35:260
23868991310,2cyclictest30631-21taskset21:36:590
2386899130,2cyclictest141rcu_preempt22:16:560
2387799129,2cyclictest9507-21grep19:45:193
2387799129,2cyclictest8345-21tail20:50:173
2387799129,2cyclictest8345-21tail20:50:173
2387799129,2cyclictest29135-21tr19:20:153
2387799129,2cyclictest2814-21sed19:30:263
2387799129,2cyclictest24832-21head19:10:163
2387799129,2cyclictest18653-21tail20:05:173
2387799129,2cyclictest16530-21sed21:05:253
2387799127,1cyclictest28001-21ntpq21:35:213
2387799126,2cyclictest39-21ksoftirqd/320:00:003
2387599129,2cyclictest6425-21sed22:20:252
2387599129,2cyclictest32328-21wc23:15:182
2387599129,2cyclictest2686-21cut21:40:152
2387599129,2cyclictest15411-21sort21:05:152
2387599129,2cyclictest15409-21head23:05:182
2387599129,2cyclictest12813-21grep22:05:182
2387599124,6cyclictest537-21snmpd00:25:132
23875991210,1cyclictest55650irq/127-enp2s0-22:54:512
2387599120,12cyclictest0-21swapper/221:16:382
2387599120,11cyclictest8828-21grep20:50:242
2387599120,11cyclictest8828-21grep20:50:242
2387199129,2cyclictest23645-21head20:15:171
2387199129,2cyclictest12095-21grep19:50:211
2387199128,3cyclictest32716-21cat19:25:261
2387199128,1cyclictest141rcu_preempt22:18:451
2387199121,4cyclictest225-21systemd-journal22:00:001
23871991212,0cyclictest0-21swapper/121:33:351
23871991210,1cyclictest9293-21grep20:50:291
23871991210,1cyclictest9293-21grep20:50:291
2386899129,2cyclictest8337-21grep20:50:170
2386899129,2cyclictest8337-21grep20:50:170
2386899129,2cyclictest348-21cat23:15:240
2386899129,2cyclictest31058-21grep20:30:150
2386899129,2cyclictest25587-21grep23:50:170
2386899129,2cyclictest23655-21grep20:15:170
2386899129,2cyclictest18246-21cat23:45:260
2386899129,2cyclictest11683-21chrt21:45:160
2386899128,3cyclictest16424-21sed21:05:250
2386899124,6cyclictest25693-21/usr/sbin/munin20:20:150
2386899122,1cyclictest16929-21diskmemload22:55:140
2386899121,5cyclictest2590-21munin-run20:40:000
2386899120,1cyclictest0-21swapper/020:00:140
2386899120,12cyclictest0-21swapper/023:33:320
2387799119,1cyclictest12883-21cron19:55:003
2387799117,3cyclictest10144-21tr20:55:143
2387799115,4cyclictest25798-21tr20:20:153
2387599119,1cyclictest6926-21sed20:45:262
2387599118,2cyclictest9343-21tr19:45:182
2387599118,2cyclictest8104-21rm23:00:262
2387599118,2cyclictest5474-21ssh23:57:312
2387599118,2cyclictest3795-21sed20:40:202
2387599118,2cyclictest3526-21cat19:35:002
2387599118,2cyclictest32015-21sed19:25:182
2387599118,2cyclictest32005-21ssh23:35:052
2387599118,2cyclictest31192-21sed22:55:182
2387599118,2cyclictest30285-21grep22:35:172
2387599118,2cyclictest25454-21fschecks_count23:50:162
2387599118,2cyclictest20974-21grep20:10:162
2387599118,2cyclictest18180-21tr20:05:142
2387599118,2cyclictest17663-21latency_hist20:05:002
2387599118,2cyclictest1590-21ssh00:34:362
2387599118,2cyclictest13191-21cut21:00:172
2387599118,2cyclictest12947-21cat19:55:002
2387599117,3cyclictest55650irq/127-enp2s0-23:30:182
2387599113,6cyclictest32634-21perf20:35:002
2387599110,10cyclictest10041-21apache_volume20:55:132
2387199119,2cyclictest25-21ksoftirqd/100:25:001
2387199119,1cyclictest30761-21grep20:30:141
2387199119,1cyclictest10768-21head20:55:171
2387199118,2cyclictest9853-21ls19:45:241
2387199118,2cyclictest8940-21ssh23:40:181
2387199118,2cyclictest25222-21ssh23:30:211
2387199118,2cyclictest22684-21rm23:10:051
2387199118,2cyclictest10630-21sh22:42:581
2387199117,4cyclictest3891-21ls19:35:141
2387199111,2cyclictest141rcu_preempt20:10:251
2387199110,10cyclictest27117-21expr00:30:151
2386899118,2cyclictest9178-21tail19:45:160
2386899118,2cyclictest7204-21grep19:40:220
2386899118,2cyclictest327-21wc19:25:270
2386899118,2cyclictest28134-21apache_accesses20:25:130
2386899118,2cyclictest26414-21ssh22:52:180
2386899118,2cyclictest19463-21ssh21:50:120
2386899118,2cyclictest14043-21grep21:00:260
2386899118,2cyclictest13789-21sort22:25:190
2386899118,2cyclictest11845-21ssh22:05:080
2386899118,2cyclictest11327-21sed19:50:140
2386899117,3cyclictest10809-21fschecks_count00:20:170
2386899115,5cyclictest16929-21diskmemload23:57:580
2387799108,1cyclictest31142-21cut20:30:163
2387799107,2cyclictest629-21chrt19:26:193
2387799107,2cyclictest5370-21grep20:45:143
2387799107,2cyclictest20614-21seq20:10:143
2387799102,4cyclictest13786-21mailstats21:00:223
2387599108,1cyclictest29237-21cat19:20:152
2387599108,1cyclictest27118-21cat20:20:272
2387599107,2cyclictest5730-21sendmail-msp19:40:002
2387599107,2cyclictest28173-21sed19:15:262
2387599107,2cyclictest10610-21grep21:25:162
2387199108,2cyclictest8068-21chrt19:44:561
2387199108,1cyclictest273122sleep120:20:421
2387199101,3cyclictest141rcu_preempt20:00:191
2387199100,9cyclictest3313-21expr20:40:151
2387199100,9cyclictest26668-21egrep19:15:151
2386899108,1cyclictest24644-21sed19:10:140
2386899108,1cyclictest10331-21expr20:55:150
2386899107,2cyclictest14500-21tr19:55:220
2386899107,2cyclictest1392-21egrep19:30:140
2386899100,9cyclictest22538-21cut20:15:000
238759997,1cyclictest13718-21sed19:55:152
238719998,1cyclictest29991-21nfs4_client19:20:201
238719998,1cyclictest12887-21expr21:00:151
238719997,2cyclictest16204-21smart_sda21:05:241
238719996,2cyclictest13917-21grep19:55:171
238689997,1cyclictest308332sleep019:23:110
238689997,1cyclictest18137-21cat20:05:150
238719987,1cyclictest18126-21grep20:05:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional