You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 19:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue Sep 08, 2026 12:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21282163102,26sleep30-21swapper/307:09:073
18752162111,27sleep20-21swapper/207:05:492
20952155100,27sleep00-21swapper/007:08:400
21312150102,26sleep10-21swapper/107:09:101
122172830,2sleep20-21swapper/210:37:372
188322630,2sleep10-21swapper/112:19:241
100752620,2sleep3331rcuc/309:48:373
103632610,2sleep010367-21sed07:25:250
227062590,1sleep20-21swapper/211:33:582
159102580,1sleep00-21swapper/011:52:430
159102580,1sleep00-21swapper/011:52:430
325482560,1sleep10-21swapper/110:05:131
6132550,2sleep10-21swapper/108:20:151
30062550,2sleep10-21swapper/109:43:291
326882540,1sleep00-21swapper/011:16:430
285542510,2sleep30-21swapper/309:14:583
155002510,2sleep00-21swapper/011:28:310
320322480,2sleep1211rcuc/110:28:401
315982480,1sleep10-21swapper/108:15:211
22702480,2sleep32269-21hddtemp_smartct10:30:193
323462470,1sleep30-21swapper/310:28:583
47322440,1sleep30-21swapper/309:20:243
302312440,1sleep30-21swapper/310:03:223
314462410,2sleep20-21swapper/212:04:482
70732400,2chrt0-21swapper/111:45:331
140032380,2chrt0-21swapper/207:35:182
56972370,10sleep012-21ksoftirqd/008:30:170
248232360,4chrt271rcuc/211:59:552
6542350,1chrt0-21swapper/109:17:451
2474993222,2cyclictest0-21swapper/307:18:283
2472993224,1cyclictest0-21swapper/107:18:281
34992280,8sleep012-21ksoftirqd/010:30:360
247299281,25cyclictest0-21swapper/107:21:291
247499271,23cyclictest0-21swapper/307:26:463
2472992716,9cyclictest779-21ssh11:17:141
2471992715,7cyclictest52750irq/139-enp3s0-09:45:020
247199271,24cyclictest0-21swapper/007:17:580
65372260,3chrt0-21swapper/311:45:233
2473992623,2cyclictest0-21swapper/207:22:012
247399261,23cyclictest0-21swapper/207:18:152
247299262,16cyclictest11917-21ssh11:50:111
247299262,16cyclictest11917-21ssh11:50:111
247299261,1cyclictest0-21swapper/107:26:461
247199261,24cyclictest0-21swapper/007:23:250
199542260,2sleep10-21swapper/110:20:021
68042250,7sleep26821-21ssh12:34:392
250632250,8sleep228-21ksoftirqd/208:00:222
247499251,22cyclictest0-21swapper/307:22:343
247399250,23cyclictest0-21swapper/207:26:462
247299251,9cyclictest23601-21python309:10:221
2472992519,4cyclictest12964-21cron10:15:011
247199252,16cyclictest25375-21ssh12:00:120
323542240,2sleep30-21swapper/312:29:303
247499249,13cyclictest28509-21apt10:50:123
2474992416,6cyclictest20327-21hddtemp_smartct07:50:143
247199243,15cyclictest22264-21ssh09:10:110
2471992419,3cyclictest30147-21sh09:40:000
212562240,3sleep321261-21pmu-power10:20:193
66562230,2sleep10-21swapper/112:34:291
38822230,2sleep00-21swapper/010:07:400
299192230,2sleep10-21swapper/112:27:031
257842230,1sleep10-21swapper/109:35:391
2473992313,7cyclictest16368-21seq10:40:232
247199232,15cyclictest17431-21ssh11:30:120
186092230,4sleep1211rcuc/109:30:251
170002230,2sleep30-21swapper/309:30:073
149452230,2sleep00-21swapper/011:04:070
2474992213,7cyclictest34-21ksoftirqd/310:40:003
2474992213,6cyclictest11727-21ntp_kernel_pll_11:25:193
247399221,6cyclictest5414-21ssh09:20:452
2473992212,4cyclictest5057-21sh11:20:202
247299225,3cyclictest21844-21diskmemload10:36:021
247299224,16cyclictest9500-21grep12:35:251
2472992219,2cyclictest0-21swapper/107:32:351
2472992213,7cyclictest2966-21python310:30:261
247199222,14cyclictest636-21runrttasks09:30:130
125792220,2sleep20-21swapper/208:45:242
78872211,2sleep20-21swapper/210:35:002
299502210,1sleep00-21swapper/011:39:300
247499212,17cyclictest20226-21sed09:55:253
247499211,5cyclictest25933-21chrt11:11:403
2474992113,6cyclictest9708-21fw_packets08:40:183
2474992113,6cyclictest278-21systemd-journal08:10:013
2474992113,6cyclictest23292-21unixbench-2d07:55:253
2474992113,6cyclictest13058-21seq12:38:583
2474992112,6cyclictest14428-21sensors_temp12:15:223
247499211,16cyclictest23844-21apt-get09:35:153
2473992112,7cyclictest16639-21cron09:30:002
2473992112,6cyclictest7841-21memory08:35:192
2473992111,7cyclictest12942-21ntp_offset11:50:192
2473992111,7cyclictest12942-21ntp_offset11:50:192
247399211,16cyclictest0-21swapper/211:40:122
247299218,8cyclictest75-21kswapd012:20:091
247299212,9cyclictest979-21python312:05:231
247299212,6cyclictest32173-21munin-node11:26:581
247299211,5cyclictest6761-21df_inode12:10:141
247299211,4cyclictest0-21swapper/108:33:461
2472992113,6cyclictest23065-21seq09:58:091
2472992113,6cyclictest20705-21awk07:50:201
2472992112,6cyclictest25520-21grep08:00:251
2472992110,8cyclictest24556-21df_inode11:35:151
2472992110,3cyclictest162972irqstats08:55:151
247199213,13cyclictest75-21kswapd012:05:120
247199212,14cyclictest3413-21taskset09:20:130
2471992114,5cyclictest19822-21/usr/sbin/munin12:20:150
2471992113,6cyclictest22663-21seq09:57:460
2471992113,5cyclictest636-21runrttasks11:47:020
2471992113,5cyclictest636-21runrttasks11:24:220
2471992112,7cyclictest9911-21sed08:40:200
2471992112,7cyclictest376-21cat08:20:130
2471992111,7cyclictest15972-21df08:55:110
247199211,14cyclictest29045-21ssh09:15:130
147772210,7sleep0131rcuc/009:51:350
251302200,2sleep10-21swapper/110:47:021
247499208,10cyclictest34-21ksoftirqd/309:19:083
247499204,3cyclictest21844-21diskmemload09:42:243
247499203,3cyclictest21844-21diskmemload10:16:513
247499202,12cyclictest15504-21apt-get07:40:123
247499201,6cyclictest22436-21/usr/sbin/munin10:45:163
2474992015,3cyclictest5676-21sed11:20:243
2474992014,3cyclictest21844-21diskmemload09:54:333
2474992013,5cyclictest53050irq/142-enp3s0-11:57:293
2474992013,5cyclictest4135-21sensors10:55:233
2474992013,5cyclictest13036-21cat10:15:003
2474992012,6cyclictest32565-21perl12:05:163
2474992012,6cyclictest13312-21seq09:26:403
2474992012,5cyclictest21844-21diskmemload12:10:123
2474992012,5cyclictest21844-21diskmemload10:40:093
2474992011,7cyclictest3586-21rt-features07:10:223
2474992011,7cyclictest20373-21fw_packets12:20:143
247499201,16cyclictest26208-21ssh12:00:183
2474992010,7cyclictest30213-21perl11:15:153
247399208,10cyclictest30073-21sed10:50:252
2473992013,5cyclictest28-21ksoftirqd/207:15:002
2473992013,5cyclictest27628-21awk08:05:252
2473992012,6cyclictest6961-21date08:35:002
2473992012,6cyclictest20688-21ntpq07:50:202
247399201,16cyclictest24190-21ssh11:35:132
247299204,3cyclictest21844-21diskmemload11:33:201
247299203,5cyclictest22-21ksoftirqd/111:24:141
247299203,3cyclictest141rcu_preempt10:55:251
247299202,6cyclictest3386-21apt-get09:20:121
2472992015,3cyclictest10460-21cpuspeed_turbos09:25:131
2472992014,4cyclictest22503-21idleruntime07:55:171
2472992014,4cyclictest1517-21sh11:41:431
2472992013,5cyclictest17175-21seq07:42:261
2472992012,6cyclictest16934-21if_enp3s011:05:161
2472992012,5cyclictest278-21systemd-journal08:45:001
2472992012,5cyclictest14752-21threads08:50:221
2472992011,7cyclictest6540-21wc09:45:241
2472992011,7cyclictest4267-21seq07:13:291
2472992011,6cyclictest19678-21cat07:50:011
2472992011,6cyclictest13420-21apt10:15:141
247199201,9cyclictest624-21gdbus12:35:020
247199201,6cyclictest21844-21diskmemload10:22:590
2471992014,4cyclictest19229-21grep11:55:170
2471992013,5cyclictest18428-21sed07:45:180
2471992013,4cyclictest16221-21/usr/sbin/munin11:05:190
2471992012,6cyclictest8228-21seq10:59:090
2471992012,6cyclictest14773-21seq07:35:360
2471992011,7cyclictest23581-21sed11:10:150
247499197,4cyclictest53050irq/142-enp3s0-11:05:233
247499193,3cyclictest21844-21diskmemload10:07:393
247499193,3cyclictest141rcu_preempt11:35:133
247499191,8cyclictest31277-21df_inode11:40:153
2474991911,6cyclictest29108-21memory08:10:193
2474991911,6cyclictest17976-21grep11:30:163
2474991911,5cyclictest9719-21grep11:00:153
2474991911,5cyclictest12088-21grep11:50:133
2474991911,5cyclictest12088-21grep11:50:123
247399199,7cyclictest22923-21cut10:45:212
247399198,6cyclictest52950irq/141-enp3s0-10:05:122
247399198,3cyclictest0-21swapper/210:26:002
2473991914,4cyclictest28-21ksoftirqd/210:20:002
2473991914,3cyclictest7680-21cut10:10:232
2473991914,3cyclictest23243-21sh09:34:512
2473991914,3cyclictest20351-21sh11:07:492
2473991913,4cyclictest8768-21sh09:47:182
2473991913,4cyclictest7841-21sh10:58:472
2473991913,4cyclictest32549-21sh12:29:412
2473991912,5cyclictest25537-21apt-get10:00:102
2473991912,4cyclictest21844-21diskmemload09:13:192
2473991911,6cyclictest5401-21expr11:45:132
2473991911,6cyclictest29428-21date11:15:002
2473991911,6cyclictest20755-21sed12:20:172
2473991911,6cyclictest12514-21seq11:25:312
247399191,13cyclictest29697-21rm11:15:132
2473991910,7cyclictest682-21ntpq12:05:202
2473991910,6cyclictest17881-21cat07:45:132
247299192,12cyclictest15083-21ssh10:40:131
2472991914,3cyclictest11982-21cpuspeed_turbos09:50:131
2472991913,4cyclictest29355-21sh12:02:421
2472991913,4cyclictest20256-21unixbench_multi11:55:251
2472991912,4cyclictest28371-21sh10:01:321
2472991911,6cyclictest9824-21cat11:00:161
2472991911,6cyclictest24628-21grep11:10:241
2472991911,6cyclictest20263-21expr09:05:141
2472991911,6cyclictest13056-21seq08:47:301
247299191,15cyclictest4093-21users08:25:261
247199198,8cyclictest15065-21apt-get10:40:130
247199195,11cyclictest959-21latency_hist10:30:000
247199193,9cyclictest21844-21diskmemload12:25:130
2471991913,4cyclictest14438-21sh12:15:220
2471991912,4cyclictest6683-21/usr/sbin/munin10:10:230
2471991912,4cyclictest32624-21grep11:40:270
2471991911,6cyclictest6882-21perl12:10:150
2471991910,7cyclictest24029-21sed08:00:000
2471991910,6cyclictest18522-21latency09:00:170
2471991910,5cyclictest52750irq/139-enp3s0-10:50:130
247499189,6cyclictest16116-21gltestperf08:55:143
2474991816,1cyclictest0-21swapper/307:32:353
247499181,5cyclictest0-21swapper/312:30:373
2474991813,3cyclictest18180-21grep07:45:163
2474991812,4cyclictest7426-21forks08:35:173
2474991811,5cyclictest3721-21readlink08:25:243
2474991811,5cyclictest13846-21if_enp3s007:35:163
2474991811,5cyclictest13739-21expr08:50:123
2474991810,6cyclictest5704-21expr08:30:173
2474991810,6cyclictest25212-21tr08:00:233
2474991810,6cyclictest21602-21seq09:07:143
2474991810,6cyclictest12308-21awk08:45:223
247399183,4cyclictest28-21ksoftirqd/212:35:122
2473991816,1cyclictest0-21swapper/207:34:152
2473991813,3cyclictest26324-21sh09:36:112
2473991813,3cyclictest21788-21sh09:56:522
2473991813,3cyclictest1977-21sh09:19:042
2473991813,3cyclictest16743-21threads07:40:242
2473991813,3cyclictest12840-21sh11:02:002
2473991812,4cyclictest23951-21sh10:22:302
2473991812,4cyclictest15574-21sh12:16:092
2473991811,5cyclictest304862sleep208:15:112
2473991811,5cyclictest28-21ksoftirqd/208:40:232
2473991811,4cyclictest0-21swapper/209:50:132
2473991810,6cyclictest22474-21smartctl07:55:172
2473991810,6cyclictest20649-21sed09:05:182
2472991813,3cyclictest26587-21fschecks_count08:05:151
2472991813,3cyclictest13568-21sh07:35:131
2472991811,5cyclictest28659-21expr10:50:141
247299181,12cyclictest17810-21apt-get09:00:111
2472991810,6cyclictest9106-21cut08:40:001
247199189,7cyclictest20854-21ntpq09:05:200
247199189,6cyclictest278-21systemd-journal09:25:000
247199182,5cyclictest13421-21apt-get10:15:140
2471991811,5cyclictest23342-21sed10:45:240
2471991810,6cyclictest9860-21cat10:35:260
2471991810,6cyclictest8727-21seq08:37:070
2471991810,6cyclictest4528-21date07:15:010
2471991810,6cyclictest20370-21cat07:50:150
2471991810,5cyclictest2725-21expr08:25:140
2474991710,5cyclictest31282-21ls08:15:193
2474991710,5cyclictest18736-21uname09:00:203
247499170,15cyclictest646-21expr08:20:153
247399179,6cyclictest3670-21gzip08:25:232
247399179,6cyclictest31695-21open_inodes09:40:212
247399178,6cyclictest16943-21cat08:55:222
247399171,7cyclictest17785-21/usr/sbin/munin09:00:252
2473991712,3cyclictest13488-21/usr/sbin/munin08:50:202
247399171,11cyclictest6378-21apt-get12:10:122
247299179,6cyclictest28889-21sed08:10:161
247199179,6cyclictest13079-21cut07:35:010
247199179,6cyclictest12952-21seq08:46:100
247199178,6cyclictest25706-21grep10:00:110
247199178,6cyclictest1949-21ntpq12:30:200
247199173,12cyclictest28311-21seq08:10:030
247199172,3cyclictest30607-21apt-get09:40:120
2471991712,3cyclictest30458-21/usr/sbin/munin08:15:120
2471991711,4cyclictest14254-21sed08:50:170
2471991710,5cyclictest24519-21cut08:00:150
247399169,5cyclictest1398-21sed08:20:232
247199169,5cyclictest15728-21expr07:40:140
2471991611,3cyclictest28095-21latency_hist08:10:000
247399158,5cyclictest30265-21ls08:15:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional