You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-16 - 13:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun Aug 16, 2026 00:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
282192177103,25sleep00-21swapper/019:06:580
422621630,2sleep20-21swapper/223:47:132
156321630,2sleep10-21swapper/119:20:181
283592147110,25sleep10-21swapper/119:08:471
284432139104,24sleep30-21swapper/319:09:543
284492138100,27sleep20-21swapper/219:09:572
11032610,3sleep10-21swapper/120:35:171
321992600,2sleep20-21swapper/221:21:092
259482590,1sleep00-21swapper/020:15:260
230212590,3sleep10-21swapper/122:25:331
134742580,2sleep20-21swapper/221:05:002
147632540,1sleep30-21swapper/323:55:163
136972540,9sleep012-21ksoftirqd/022:43:150
80582520,1sleep00-21swapper/020:50:170
86632490,2sleep122-21ksoftirqd/123:26:371
49102490,1sleep20-21swapper/223:24:372
96442470,1sleep10-21swapper/120:55:121
2872599479,22cyclictest10905-21seq20:55:320
18172470,3sleep1211rcuc/121:46:281
121862470,1sleep00-21swapper/000:17:000
121862470,1sleep00-21swapper/000:17:000
167432460,3chrt34-21ksoftirqd/323:09:253
289172450,1sleep00-21swapper/023:41:290
2872599427,21cyclictest18850-21sh22:46:250
2872599427,21cyclictest18850-21sh22:46:240
28725994112,16cyclictest26143-21sh21:16:560
2872599361,22cyclictest0-21swapper/023:18:540
2872599361,21cyclictest15565-21diskmemload00:35:350
68432330,10sleep334-21ksoftirqd/323:01:233
2872599338,22cyclictest19261-21cat23:10:250
2872599337,23cyclictest31003-21grep19:15:140
28725993315,15cyclictest17820-21chrt22:22:070
2872599331,29cyclictest1723-21rm21:23:030
319842320,4sleep0131rcuc/000:07:530
2872599329,20cyclictest2520-21expr22:35:140
2872599328,21cyclictest18197-21expr20:00:140
2872599328,21cyclictest10714-21seq23:51:560
2872599327,22cyclictest23966-21seq22:26:280
28725993214,15cyclictest32109-21sh22:56:040
28725993214,15cyclictest17940-21sh21:10:370
2872599321,26cyclictest0-21swapper/022:10:110
28725993210,19cyclictest1819-21cat20:35:250
2872599320,4cyclictest12962-21grep19:45:250
2872599320,2cyclictest0-21swapper/021:30:170
2872599318,20cyclictest8482-21seq23:03:000
2872599311,4cyclictest30003-21ssh21:44:010
28725993114,14cyclictest24782-21sh22:50:320
28725993113,16cyclictest20230-21ls20:05:130
28725993112,16cyclictest2925-21sh21:47:340
28725993110,19cyclictest31799-21fschecks_count23:20:160
2872599310,6cyclictest15120-21grep19:50:250
2872599310,5cyclictest17628-21ssh23:57:050
2872599310,3cyclictest21545-21grep00:00:170
2872599310,28cyclictest3195-21rm23:46:120
2872599309,18cyclictest11354-21wc19:45:000
2872599307,20cyclictest29411-21interrupts19:10:180
2872599307,20cyclictest17892-21cut20:00:020
28725993014,14cyclictest22183-21sh23:36:350
2872599301,26cyclictest2651-21fschecks_time00:10:190
28725993012,15cyclictest15630-21/usr/sbin/munin00:20:180
2872599300,3cyclictest8859-21ssh21:51:410
2872599300,3cyclictest772-21ssh22:33:530
2872599300,3cyclictest5588-21grep20:45:170
170242300,2sleep00-21swapper/021:58:040
2872599296,20cyclictest27774-21expr22:05:230
2872599295,20cyclictest7270-21timerwakeupswit23:25:250
28725992912,15cyclictest30071-21sh00:30:210
28725992912,14cyclictest17595-21sh23:33:450
28725992912,14cyclictest17595-21sh23:33:440
28725992911,15cyclictest26510-21sh00:27:470
28725992911,15cyclictest23676-21sed20:10:250
2872599290,4cyclictest30996-21expr20:30:130
2872599290,3cyclictest5670-21head19:30:170
2872599290,3cyclictest23567-21rm22:02:450
2872599290,3cyclictest23567-21rm22:02:450
2872599290,3cyclictest10237-21ssh22:16:260
2872599287,18cyclictest7470-21expr19:35:140
2872599282,20cyclictest0-21swapper/020:40:120
2872599280,3cyclictest12447-21timerandwakeup23:05:250
2872599280,3cyclictest1112-21cat19:20:130
2872599280,2cyclictest17791-21apt-get21:35:130
2872599280,2cyclictest0-21swapper/021:26:490
12422280,2sleep30-21swapper/322:34:233
87392270,3sleep10-21swapper/100:38:531
2872799277,3cyclictest141rcu_preempt22:58:392
2872699271,24cyclictest0-21swapper/119:13:301
2872599272,22cyclictest27080-21gltestperf20:20:150
2872599271,23cyclictest29783-21processes20:25:250
2872599271,23cyclictest29783-21processes20:25:250
2872799261,24cyclictest0-21swapper/219:13:302
37322250,2sleep30-21swapper/319:25:173
2872799251,5cyclictest141rcu_preempt22:15:122
2872799251,22cyclictest1197-21diskstats19:20:152
2872599254,18cyclictest13953-21cstates21:05:130
240012250,2sleep30-21swapper/322:26:313
146162250,1sleep30-21swapper/323:30:483
146162250,1sleep30-21swapper/323:30:483
118312250,3sleep1211rcuc/119:45:141
2872899244,4cyclictest15565-21diskmemload21:49:343
28728992422,1cyclictest0-21swapper/319:12:573
2872899242,18cyclictest27831-21ssh00:05:153
2872699245,9cyclictest141rcu_preempt00:10:211
2872699242,16cyclictest21069-21ssh00:00:141
2872699241,13cyclictest0-21swapper/121:57:181
2872599241,5cyclictest3359-21/usr/sbin/munin19:25:150
2872599240,3cyclictest12170-21if_err_enp3s021:00:170
219982240,11sleep334-21ksoftirqd/322:49:313
219982240,11sleep334-21ksoftirqd/322:49:313
37482230,5chrt3743-21awk23:00:001
2872899235,3cyclictest15565-21diskmemload22:57:503
2872899234,3cyclictest15565-21diskmemload23:20:473
2872899233,6cyclictest34-21ksoftirqd/300:18:363
2872899233,6cyclictest34-21ksoftirqd/300:18:363
2872899232,17cyclictest19323-21ssh23:35:143
28728992315,5cyclictest53050irq/142-enp3s0-21:20:243
28728992314,6cyclictest53050irq/142-enp3s0-00:25:113
28728992314,6cyclictest24999-21tune2fs21:40:133
28728992313,6cyclictest53050irq/142-enp3s0-22:15:123
2872899231,10cyclictest153542ssh21:56:233
2872799232,15cyclictest2497-21cpuspeed_turbos22:35:142
28726992316,5cyclictest4853-21sh20:45:001
28726992316,4cyclictest15565-21diskmemload23:20:231
254232231,2sleep20-21swapper/200:26:432
2872899221,4cyclictest141rcu_preempt23:10:213
28728992214,6cyclictest17124-21awk21:10:243
2872799226,3cyclictest294452cat00:30:152
28727992215,5cyclictest28-21ksoftirqd/223:35:002
28727992215,5cyclictest28-21ksoftirqd/223:35:002
28727992214,6cyclictest16054-21diskstats00:20:162
28727992213,6cyclictest21532-21diskstats22:25:142
2872699221,7cyclictest15565-21diskmemload00:24:551
28726992216,3cyclictest657-21sshd22:11:021
28726992215,5cyclictest52950irq/141-enp3s0-23:11:581
28726992214,6cyclictest17269-21pmu-power22:45:231
28726992214,6cyclictest17269-21pmu-power22:45:231
28726992213,7cyclictest29021-21ntp_states22:30:211
28726992213,7cyclictest18766-21ls20:00:201
28726992212,6cyclictest52950irq/141-enp3s0-22:00:231
28726992212,6cyclictest52950irq/141-enp3s0-22:00:221
2872699221,18cyclictest31003-21awk21:44:591
279212220,2sleep3331rcuc/322:05:253
122452220,2sleep30-21swapper/323:53:283
2872899214,5cyclictest640-21snmpd00:23:173
2872899211,6cyclictest34-21ksoftirqd/322:12:393
28728992113,6cyclictest25449-21sed23:15:213
28728992112,7cyclictest29078-21diskstats20:25:183
28728992112,7cyclictest29078-21diskstats20:25:183
28728992111,6cyclictest53050irq/142-enp3s0-00:35:123
2872799218,6cyclictest28-21ksoftirqd/200:04:222
2872799217,4cyclictest15565-21diskmemload00:35:162
2872799217,3cyclictest15565-21diskmemload23:27:142
2872799215,10cyclictest21117-1kworker/u9:223:13:462
28727992116,3cyclictest15558-21sh21:10:002
28727992115,3cyclictest15565-21diskmemload23:00:412
2872799211,17cyclictest52750irq/139-enp3s0-23:40:152
28727992110,6cyclictest10949-21ssh21:30:122
2872799210,10cyclictest66050lldpd20:23:032
2872699217,9cyclictest9167-21ssh22:40:121
2872699217,3cyclictest141rcu_preempt21:25:161
2872699215,3cyclictest141rcu_preempt00:10:001
2872699213,15cyclictest22378-21tr20:10:131
2872699212,17cyclictest12226-21awk21:55:001
2872699212,13cyclictest512-21ssh23:45:031
2872699212,11cyclictest141rcu_preempt22:06:311
28726992113,5cyclictest636-21runrttasks23:44:371
28726992113,5cyclictest15565-21diskmemload00:25:141
28726992112,6cyclictest17726-21seq19:59:571
2872699211,11cyclictest0-21swapper/123:56:261
2872699211,10cyclictest0-21swapper/123:16:191
28726992110,9cyclictest5166-21ntp_states23:00:181
28212210,1sleep10-21swapper/121:24:091
2872899203,14cyclictest7963-21wc20:50:163
28728992013,5cyclictest8834-21sed22:40:003
28728992012,6cyclictest19475-21/usr/sbin/munin22:00:143
28728992012,6cyclictest19475-21/usr/sbin/munin22:00:143
2872899201,16cyclictest2183-21ssh00:10:163
2872799204,3cyclictest15565-21diskmemload23:57:052
2872799203,15cyclictest6980-21pmu-power21:50:212
2872799203,15cyclictest4878-21sh21:25:172
28727992014,4cyclictest31380-21grep19:15:162
28727992014,3cyclictest15565-21diskmemload22:23:182
28727992014,3cyclictest15565-21diskmemload00:19:462
28727992014,3cyclictest15565-21diskmemload00:19:452
28727992014,3cyclictest15565-21diskmemload00:12:352
28727992013,4cyclictest15565-21diskmemload22:45:222
28727992013,4cyclictest15565-21diskmemload22:45:222
28727992012,6cyclictest9671-21perl22:40:152
28727992012,6cyclictest7747-21perl19:35:172
28727992012,6cyclictest29633-21ntpq20:25:242
28727992012,6cyclictest29633-21ntpq20:25:232
28727992012,6cyclictest10137-21memory20:55:172
28727992012,5cyclictest20968-21sed22:00:292
28727992012,5cyclictest20968-21sed22:00:282
2872799201,16cyclictest29696-21/usr/sbin/munin22:30:262
28727992011,3cyclictest11352-21sh23:05:152
2872799200,18cyclictest22964-21cut21:15:132
2872799200,11cyclictest244752diskmemload21:40:052
2872699202,9cyclictest141rcu_preempt22:53:201
28726992015,3cyclictest52950irq/141-enp3s0-00:30:191
28726992014,3cyclictest15565-21diskmemload23:34:311
28726992014,3cyclictest15565-21diskmemload23:34:311
28726992014,3cyclictest15565-21diskmemload22:36:431
28726992013,4cyclictest15565-21diskmemload21:12:091
28726992012,6cyclictest15471-21seq19:53:271
28726992012,6cyclictest12860-21seq00:17:411
28726992012,6cyclictest12860-21seq00:17:401
28726992011,7cyclictest20376-21awk23:35:221
28726992011,6cyclictest15597-21grep21:10:001
2872699201,11cyclictest0-21swapper/122:16:251
2872699201,11cyclictest0-21swapper/121:31:141
2872899199,6cyclictest53050irq/142-enp3s0-21:35:143
2872899194,3cyclictest141rcu_preempt22:22:233
2872899193,4cyclictest34-21ksoftirqd/322:41:443
2872899193,14cyclictest22341-21cut00:00:253
2872899193,12cyclictest4140-21ssh21:25:123
2872899192,3cyclictest15565-21diskmemload23:49:463
2872899192,14cyclictest3081-21cpuspeed_turbos20:40:123
28728991914,3cyclictest3123-21turbostat00:35:013
28728991914,3cyclictest15565-21diskmemload22:52:043
28728991912,5cyclictest23813-21cut21:15:193
28728991911,6cyclictest15550-21seq21:09:593
28728991911,6cyclictest12176-21cut23:30:003
28728991910,7cyclictest21656-21users20:05:253
28728991910,6cyclictest53050irq/142-enp3s0-21:50:113
2872799195,12cyclictest23199-21ntpq20:10:202
2872799193,14cyclictest25526-21/usr/sbin/munin20:15:232
2872799193,11cyclictest26253-21apt-get22:05:112
2872799192,12cyclictest5147-21apt-get19:30:132
2872799192,11cyclictest3007-21apt-get19:25:122
28727991914,3cyclictest20807-21kernelversion20:05:172
2872799191,3cyclictest31427-21apt-get21:45:122
28727991913,4cyclictest6313-21sh22:14:162
28727991913,4cyclictest18035-21sh21:10:432
28727991913,4cyclictest17785-21sh21:35:132
28727991913,3cyclictest15565-21diskmemload00:08:112
28727991912,5cyclictest23971-21sh23:38:212
28727991912,5cyclictest23574-21smartctl22:50:172
28727991912,5cyclictest16807-21pluginstate19:55:222
28727991912,4cyclictest15565-21diskmemload23:52:592
28727991912,4cyclictest15565-21diskmemload21:59:062
28727991910,7cyclictest15594-21cron19:55:002
2872799190,17cyclictest15565-21diskmemload23:15:112
28726991914,3cyclictest7361-21rm23:50:071
28726991912,4cyclictest15565-21diskmemload23:07:431
28726991911,6cyclictest6978-21seq19:34:291
28726991911,6cyclictest2999-21awk19:25:111
28726991911,6cyclictest23378-21tune2fs21:15:151
28726991911,5cyclictest9789-21cat19:40:151
28726991910,7cyclictest8653-21/usr/sbin/munin20:50:231
28726991910,6cyclictest24972-21sed20:15:161
2872699190,10cyclictest0-21swapper/122:22:241
2872899189,6cyclictest14226-21gltestperf19:50:163
2872899189,5cyclictest12410-21timerandwakeup21:30:243
2872899182,13cyclictest27282-21wc20:20:163
2872899182,11cyclictest582-21apt-get20:35:143
2872899182,10cyclictest18017-21apt-get20:00:143
28728991812,4cyclictest15565-21diskmemload23:42:193
28728991811,5cyclictest16569-21latency19:55:193
28728991811,5cyclictest11414-21sed21:00:003
28728991810,6cyclictest7365-21wc19:35:143
2872799182,13cyclictest2754-21cut20:40:012
28727991811,5cyclictest5388-21cat20:45:152
28727991811,5cyclictest32174-21cat20:30:252
28727991810,6cyclictest560-21dbus-daemon19:45:232
28727991810,6cyclictest3886-21ntp_states20:40:202
28727991810,5cyclictest9848-21grep19:40:172
2872699183,13cyclictest28825-21sort20:25:171
2872699183,13cyclictest28825-21sort20:25:161
2872699182,10cyclictest52950irq/141-enp3s0-21:35:131
28726991811,5cyclictest28638-21sed20:25:001
28726991811,5cyclictest22206-21date20:10:001
2872899179,6cyclictest22858-21sed20:10:173
2872899179,5cyclictest11520-21sort21:00:113
2872899178,7cyclictest31381-21gltestperf19:15:163
2872899173,8cyclictest861-21apt-get19:20:123
2872899172,12cyclictest9825-21cut19:40:163
28727991710,5cyclictest18186-21cat20:00:142
2872699179,6cyclictest8203-21ntpq19:35:231
2872699179,5cyclictest30997-21grep19:15:141
28726991710,4cyclictest6286-21switchtime20:45:251
2872899169,5cyclictest13632-21sed19:50:023
28728991612,2cyclictest30853-21/usr/sbin/munin20:30:173
28728991611,3cyclictest6205-21sendmail_mailst20:45:243
28728991610,4cyclictest636-21runrttasks19:33:293
28726991610,4cyclictest392-21sed20:35:001
245432160,3sleep30-21swapper/320:15:133
2872799158,5cyclictest7992-21ls20:50:162
2872699159,4cyclictest12120-21idleruntime21:00:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional