You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 08:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed May 06, 2026 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
643621750,1sleep10-21swapper/121:38:431
1947921670,1sleep00-21swapper/022:56:550
109522166115,28sleep20-21swapper/219:08:262
108962165110,28sleep10-21swapper/119:07:391
109632163105,28sleep30-21swapper/319:08:343
107712161102,29sleep00-21swapper/019:06:030
728921530,2sleep20-21swapper/222:02:072
58182730,1sleep20-21swapper/223:34:092
222622720,3sleep122-21ksoftirqd/123:45:241
16092720,2sleep10-21swapper/123:30:231
112812660,2chrt0-21swapper/022:50:560
217392650,1sleep30-21swapper/300:31:323
108452580,2sleep010847-21grep00:00:240
3972540,6sleep30-21swapper/300:16:083
13422540,2sleep10-21swapper/123:07:111
127092490,1sleep20-21swapper/222:29:532
127092490,1sleep20-21swapper/222:29:522
111802480,1sleep20-21swapper/200:24:462
163492470,1sleep10-21swapper/122:31:401
118352460,1chrt11836-21rm22:05:212
81422440,1sleep08143-21sh21:40:060
37212440,2chrt0-21swapper/023:09:270
242612430,10sleep228-21ksoftirqd/223:24:062
54792420,1sleep30-21swapper/323:33:493
1134899379,7cyclictest9262-21cat00:00:010
1134899379,7cyclictest9262-21cat00:00:010
11348993611,5cyclictest25148-21grep00:10:240
163752340,3chrt0-21swapper/300:28:043
1135099342,16cyclictest24770-21ssh22:15:132
1134899342,9cyclictest15250-21chrt21:45:110
11348993410,6cyclictest14978-21date19:15:250
11348993410,6cyclictest14978-21date19:15:240
209062330,2chrt0-21swapper/023:45:030
1134999331,28cyclictest31241-21diskmemload00:36:131
1134899332,5cyclictest49982sleep000:20:130
11348993311,5cyclictest18671-21seq23:43:050
11348993311,5cyclictest15715-21seq20:34:040
1134899331,12cyclictest11728-21taskset23:15:130
11351993213,5cyclictest34-21ksoftirqd/322:45:123
1134899329,5cyclictest7446-21cat20:15:100
1134899328,6cyclictest29494-21sed23:50:240
1134899322,10cyclictest19954-21/usr/sbin/munin21:25:110
1134899319,5cyclictest30702-21seq19:50:350
1134899318,5cyclictest25769-21cat20:55:180
1134899318,5cyclictest25769-21cat20:55:180
1134899312,7cyclictest22207-21apt-get21:50:130
1134899312,10cyclictest640-21snmpd23:35:230
11348993119,10cyclictest0-21swapper/000:33:000
1134999300,26cyclictest28517-21fschecks_time23:50:151
1134899309,5cyclictest1490-21cut20:00:140
1134899302,8cyclictest640-21snmpd21:20:120
1134899301,7cyclictest2637-21/usr/sbin/munin21:35:240
11348993014,3cyclictest17245-21timerwakeupswit19:20:250
1134899301,11cyclictest6950-21grep22:25:170
1134899301,11cyclictest6950-21grep22:25:160
1134899301,10cyclictest17028-21fschecks_time00:05:170
11348993010,7cyclictest2623-21latency22:45:190
11348993010,5cyclictest21097-21cut22:35:190
11348993010,4cyclictest31241-21diskmemload21:30:150
1135199291,4cyclictest28520-21fschecks_time23:50:153
11351992913,5cyclictest34-21ksoftirqd/323:20:153
1135099291,25cyclictest16561-21diskstats22:55:152
1134899298,5cyclictest874-21awk23:30:150
1134899292,12cyclictest25680-21ssh00:35:120
1134899291,14cyclictest26775-21ssh22:15:420
11348992910,3cyclictest31241-21diskmemload23:04:210
11351992822,4cyclictest34-21ksoftirqd/300:40:003
11351992811,5cyclictest34-21ksoftirqd/300:11:393
11350992821,5cyclictest29153-21ssh00:37:052
1134999284,10cyclictest22-21ksoftirqd/100:26:111
1134899285,9cyclictest0-21swapper/022:42:110
1134899281,4cyclictest3912-21cpuspeed_turbos22:00:140
11351992717,8cyclictest10795-21ssh22:28:033
11351992717,8cyclictest10795-21ssh22:28:023
11351992711,3cyclictest141rcu_preempt22:20:333
11350992723,2cyclictest14087-21apt-get23:40:112
1134999271,2cyclictest16164-21apt-get22:55:111
1134999271,24cyclictest0-21swapper/123:40:291
11348992713,8cyclictest31241-21diskmemload23:25:120
25012260,2sleep20-21swapper/221:11:552
217182260,3sleep10-21swapper/122:35:251
1135199266,11cyclictest34-21ksoftirqd/322:31:573
11351992623,2cyclictest0-21swapper/323:40:143
11351992620,4cyclictest34-21ksoftirqd/323:59:593
11351992620,4cyclictest34-21ksoftirqd/300:00:003
1135199261,2cyclictest0-21swapper/300:25:003
1135199261,23cyclictest0-21swapper/323:45:043
11351992611,5cyclictest34-21ksoftirqd/322:37:493
11351992610,7cyclictest141rcu_preempt22:15:233
11351992610,7cyclictest141rcu_preempt20:00:173
11351992610,3cyclictest141rcu_preempt23:25:113
11350992622,2cyclictest0-21swapper/200:26:012
1135099261,23cyclictest0-21swapper/200:32:042
1134999262,10cyclictest22-21ksoftirqd/121:44:381
1134999261,24cyclictest0-21swapper/100:30:011
1134999261,1cyclictest0-21swapper/100:25:001
11349992611,12cyclictest14198-21ntpq21:20:201
1134999260,24cyclictest14679-21ssh22:54:121
11348992623,2cyclictest0-21swapper/021:15:170
1134899261,24cyclictest0-21swapper/000:26:060
11348992612,3cyclictest5280-21ssh21:14:380
84062250,11sleep228-21ksoftirqd/223:59:222
84062250,11sleep228-21ksoftirqd/223:59:212
313582250,11sleep334-21ksoftirqd/321:55:503
1135199259,8cyclictest141rcu_preempt00:00:183
1135199259,7cyclictest141rcu_preempt22:14:573
1135199259,3cyclictest34-21ksoftirqd/321:12:203
1135199258,4cyclictest34-21ksoftirqd/323:01:283
1135199258,4cyclictest34-21ksoftirqd/321:35:303
1135199251,5cyclictest5334-21taskset20:10:133
11351992511,8cyclictest34-21ksoftirqd/323:14:093
11351992510,4cyclictest34-21ksoftirqd/300:05:193
1135099251,23cyclictest0-21swapper/222:54:292
1135099251,1cyclictest0-21swapper/223:45:322
1135099250,2cyclictest28986-21strings23:50:192
1134999256,6cyclictest22-21ksoftirqd/121:55:241
1134999256,11cyclictest22-21ksoftirqd/121:33:151
1134999252,13cyclictest22-21ksoftirqd/121:46:201
1134999251,2cyclictest141rcu_preempt00:04:161
1134999251,13cyclictest22-21ksoftirqd/122:06:011
1135199249,4cyclictest34-21ksoftirqd/323:36:043
1135199249,3cyclictest141rcu_preempt22:45:013
1135199249,3cyclictest141rcu_preempt22:03:583
1135199248,5cyclictest34-21ksoftirqd/321:15:163
1135199248,4cyclictest34-21ksoftirqd/321:54:473
1135199247,4cyclictest34-21ksoftirqd/322:05:193
1135199245,8cyclictest34-21ksoftirqd/320:35:223
1135199245,5cyclictest34-21ksoftirqd/321:20:453
1135199243,3cyclictest141rcu_preempt20:05:133
1135199243,3cyclictest141rcu_preempt20:05:123
11351992421,2cyclictest0-21swapper/322:55:193
11351992411,7cyclictest141rcu_preempt19:36:123
11351992410,3cyclictest141rcu_preempt20:15:203
11351992410,2cyclictest141rcu_preempt23:17:423
11350992422,1cyclictest0-21swapper/222:47:222
1134999249,5cyclictest22-21ksoftirqd/119:55:221
1134999248,8cyclictest141rcu_preempt23:28:291
1134999248,4cyclictest22-21ksoftirqd/121:28:431
1134999247,4cyclictest22-21ksoftirqd/121:50:241
1134999246,8cyclictest22-21ksoftirqd/100:12:501
214172230,8sleep2271rcuc/221:25:232
148262230,2sleep10-21swapper/123:16:461
1135199239,4cyclictest34-21ksoftirqd/322:50:203
1135199236,7cyclictest141rcu_preempt21:28:473
1135199233,4cyclictest34-21ksoftirqd/319:30:223
11351992310,3cyclictest141rcu_preempt21:40:173
1135099235,16cyclictest27775-21needreboot21:30:192
11350992321,1cyclictest0-21swapper/223:39:222
11350992321,1cyclictest0-21swapper/221:15:202
1135099231,5cyclictest0-21swapper/223:10:382
1135099231,5cyclictest0-21swapper/221:46:592
1134999239,9cyclictest22-21ksoftirqd/100:15:121
1134999238,4cyclictest22-21ksoftirqd/123:10:441
1134999234,6cyclictest22-21ksoftirqd/122:22:591
1134999232,14cyclictest2560-21ssh23:55:111
1134999232,14cyclictest2560-21ssh23:55:101
1134999231,12cyclictest22-21ksoftirqd/122:44:081
1135199227,2cyclictest141rcu_preempt19:25:233
1135199224,8cyclictest34-21ksoftirqd/319:12:323
11350992217,3cyclictest271rcuc/221:40:002
11350992212,7cyclictest25376-21df_inode20:55:142
11350992212,7cyclictest25376-21df_inode20:55:132
1134999227,11cyclictest141rcu_preempt19:20:111
1134999226,7cyclictest141rcu_preempt22:04:321
1134999226,3cyclictest141rcu_preempt21:15:541
1134999226,14cyclictest22-21ksoftirqd/123:00:251
1134999225,5cyclictest141rcu_preempt23:35:151
11349992213,6cyclictest9973-21cat20:20:161
11349992211,2cyclictest141rcu_preempt23:21:061
1134899222,4cyclictest0-21swapper/000:15:140
1135199219,3cyclictest141rcu_preempt19:41:483
1135199219,2cyclictest141rcu_preempt21:30:013
1135199217,2cyclictest141rcu_preempt21:05:243
1135199216,4cyclictest34-21ksoftirqd/323:07:143
1135199216,4cyclictest34-21ksoftirqd/319:55:113
1135199216,3cyclictest141rcu_preempt21:45:143
1135199216,3cyclictest141rcu_preempt21:00:153
1135199216,3cyclictest141rcu_preempt20:21:183
1135099217,2cyclictest141rcu_preempt19:35:182
1135099211,5cyclictest0-21swapper/200:03:242
11350992113,6cyclictest31802-21ls23:05:282
11350992113,6cyclictest29511-21entropy19:50:152
11350992113,6cyclictest22714-21ssh00:09:432
11350992112,7cyclictest3447-21sed22:00:012
11350992111,7cyclictest1515-21perl20:00:162
1134999217,5cyclictest22-21ksoftirqd/122:10:241
1134999217,12cyclictest211rcuc/122:15:581
11349992113,6cyclictest18394-21seq20:40:071
11349992112,7cyclictest23138-21perl20:50:151
11349992110,2cyclictest141rcu_preempt22:45:351
1134899212,13cyclictest18427-21apt-get20:40:130
11348992113,6cyclictest13536-21seq20:28:360
1135199208,2cyclictest141rcu_preempt20:55:243
1135199208,2cyclictest141rcu_preempt20:55:233
1135199207,7cyclictest141rcu_preempt20:42:373
1135199207,3cyclictest141rcu_preempt19:50:163
1135199205,3cyclictest141rcu_preempt20:51:163
11351992014,4cyclictest34-21ksoftirqd/319:20:013
11351992014,4cyclictest34-21ksoftirqd/319:20:003
11351992012,6cyclictest16690-21sed19:20:193
1135099201,4cyclictest24311-21munin-run22:14:592
11350992013,5cyclictest23990-21sh21:50:272
11350992012,6cyclictest31651-21sendmail_mailqu00:15:232
11350992012,6cyclictest21210-21ls22:35:212
11350992012,6cyclictest20648-21fw_packets19:30:152
11350992012,6cyclictest16130-21expr19:20:132
11350992011,6cyclictest20471-21cat20:45:012
11350992011,6cyclictest13730-21acpi19:15:122
11350992011,6cyclictest13730-21acpi19:15:112
1134999209,3cyclictest22-21ksoftirqd/120:30:111
1134999203,3cyclictest141rcu_preempt19:50:161
1134999201,5cyclictest22-21ksoftirqd/121:13:391
11349992014,4cyclictest22-21ksoftirqd/121:10:001
11349992013,5cyclictest22-21ksoftirqd/119:35:011
11349992012,6cyclictest17923-21seq20:37:131
11349992011,6cyclictest560-21dbus-daemon22:25:241
11349992011,6cyclictest560-21dbus-daemon22:25:231
1134899209,9cyclictest18767-21cpuspeed_turbos23:20:110
11348992011,6cyclictest22222-21sed19:35:000
11348992010,7cyclictest25810-21ntp_states19:40:180
1135199195,3cyclictest141rcu_preempt19:45:253
11351991911,6cyclictest14672-21memory20:30:183
1135099198,3cyclictest9482cat20:00:002
1135099191,5cyclictest0-21swapper/222:32:182
11350991914,3cyclictest29160-21sh23:27:072
11350991913,4cyclictest28-21ksoftirqd/219:45:002
11350991912,5cyclictest4203-21seq22:23:232
11350991912,5cyclictest25039-21ls00:10:242
11350991911,6cyclictest28110-21memory22:40:172
11350991911,5cyclictest10210-21idleruntime20:20:162
11350991910,7cyclictest14482-21cat20:30:162
11350991910,6cyclictest52950irq/141-enp3s0-23:00:112
11350991910,6cyclictest27106-21cat19:45:142
1135099190,7cyclictest8837-21gltestperf21:40:172
1134999192,12cyclictest13745-21apt-get19:15:131
1134999192,12cyclictest13745-21apt-get19:15:121
11349991912,5cyclictest624-21gdbus20:30:001
11349991912,5cyclictest27161-21date21:00:011
11349991912,5cyclictest27161-21date21:00:001
11349991912,4cyclictest18307-21sh00:05:271
11349991911,6cyclictest19354-21ls19:25:241
1134999190,3cyclictest141rcu_preempt19:35:111
11348991914,3cyclictest21734-21switchtime20:45:240
11348991913,4cyclictest19020-21sh22:34:120
11348991912,5cyclictest10409-21cat20:20:180
11348991911,6cyclictest18622-21grep19:25:160
11348991911,6cyclictest11630-21memory22:05:170
11348991910,7cyclictest13083-21seq19:12:390
11348991910,6cyclictest32079-21missed_timers19:55:190
1134899190,5cyclictest23315-21sed19:35:180
183252180,1sleep00-21swapper/022:10:160
1135199186,4cyclictest34-21ksoftirqd/320:25:173
11350991813,3cyclictest4267-21rt-features20:05:242
11350991813,3cyclictest4267-21rt-features20:05:232
11350991812,4cyclictest16625-21sh21:22:082
11350991811,5cyclictest6077-21expr20:10:182
11350991811,5cyclictest29600-21expr21:05:152
11350991811,5cyclictest11628-21cat19:10:142
1134999183,2cyclictest141rcu_preempt19:40:161
11349991812,4cyclictest22-21ksoftirqd/120:15:011
11349991812,4cyclictest21143-21idleruntime20:45:171
11349991811,5cyclictest3593-21tune2fs20:05:171
11349991811,5cyclictest3593-21tune2fs20:05:161
11349991811,5cyclictest27389-21expr21:00:131
11349991810,6cyclictest27214-21ls19:45:131
11348991812,4cyclictest29727-21sh21:55:150
11348991811,5cyclictest5844-21sed20:10:160
11348991811,5cyclictest3644-21cut20:05:170
11348991811,5cyclictest3644-21cut20:05:160
11348991811,5cyclictest29018-21cut19:50:000
11348991810,5cyclictest29748-21fschecks_time21:05:170
1135199173,3cyclictest141rcu_preempt20:45:233
1135099179,6cyclictest7682-21cat20:15:132
1135099179,6cyclictest16827-21smartctl20:35:162
1135099179,6cyclictest12211-21perl23:15:162
11350991711,4cyclictest23825-21sendmail_mailtr20:50:232
11350991711,4cyclictest21226-21uname20:45:172
11350991710,5cyclictest18031-21find19:25:112
11349991710,5cyclictest2412-21cat20:00:241
11349991710,5cyclictest11704-21cat19:10:151
11348991712,3cyclictest31720-21/usr/sbin/munin22:20:120
1135099161,5cyclictest0-21swapper/220:25:372
11350991611,3cyclictest27652-21fschecks_time21:00:152
1134999161,3cyclictest141rcu_preempt20:15:171
1134899168,6cyclictest16730-21gltestperf20:35:160
1134899161,13cyclictest5280-21tail23:10:150
11348991610,4cyclictest27282-21cpuspeed_turbos21:00:120
11348991610,4cyclictest22684-21acpi20:50:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional