You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-10 - 02:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Jul 09, 2026 12:45:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
62752244190,29sleep30-21swapper/307:07:043
6388216296,59sleep20-21swapper/207:08:322
65022158102,28sleep00-21swapper/007:09:580
62782157104,28sleep10-21swapper/107:07:061
167052750,1sleep30-21swapper/310:37:263
196512640,3sleep2271rcuc/211:50:262
36992600,2sleep00-21swapper/009:40:210
299582580,2sleep20-21swapper/211:35:002
51092550,6sleep1211rcuc/109:17:511
80992520,3sleep1211rcuc/110:07:151
80992520,3sleep1211rcuc/110:07:141
289252490,2sleep10-21swapper/111:57:431
95182480,2sleep00-21swapper/009:20:260
319762480,1sleep131977-21sh10:00:321
206012480,1sleep10-21swapper/112:39:401
50402470,1sleep00-21swapper/008:20:160
256982470,1sleep00-21swapper/009:32:550
134182470,1sleep00-21swapper/010:10:470
90282460,2sleep00-21swapper/010:08:110
90282460,2sleep00-21swapper/010:08:100
251122460,1chrt0-21swapper/012:19:000
177382440,3sleep117716-21munin-run11:50:001
7172430,2chrt0-21swapper/012:00:230
179632400,1chrt0-21swapper/110:38:411
63832380,1sleep10-21swapper/109:42:271
227632380,1chrt0-21swapper/311:29:593
281012370,4chrt560-21dbus-daemon08:00:003
677799301,26cyclictest17063-21taskset07:32:142
6775993014,10cyclictest14647-21ssh09:25:120
6775992911,4cyclictest12-21ksoftirqd/009:13:590
6775992910,11cyclictest12-21ksoftirqd/011:55:260
221092290,10sleep012-21ksoftirqd/007:45:120
677699281,25cyclictest0-21swapper/107:21:331
6775992814,4cyclictest12-21ksoftirqd/009:58:570
6775992813,8cyclictest12-21ksoftirqd/009:54:430
274942280,3chrt0-21swapper/212:20:192
677699271,24cyclictest0-21swapper/107:25:541
677599278,8cyclictest141rcu_preempt11:47:490
677599276,4cyclictest12-21ksoftirqd/009:48:120
6775992715,2cyclictest141rcu_preempt11:05:090
6775992712,4cyclictest12-21ksoftirqd/011:24:560
6775992711,4cyclictest12-21ksoftirqd/009:39:040
306822270,2chrt0-21swapper/312:22:493
677899261,23cyclictest0-21swapper/307:20:053
6777992619,5cyclictest25905-21sh09:56:182
677799261,23cyclictest0-21swapper/207:20:332
677799261,23cyclictest0-21swapper/207:18:232
6776992624,1cyclictest0-21swapper/107:33:011
6776992624,1cyclictest0-21swapper/107:20:001
6775992620,4cyclictest131rcuc/010:15:180
6775992615,2cyclictest141rcu_preempt11:51:370
677599261,1cyclictest0-21swapper/007:20:340
677599261,1cyclictest0-21swapper/007:19:000
6775992610,9cyclictest12-21ksoftirqd/010:28:350
6775992610,4cyclictest12-21ksoftirqd/011:33:000
677599260,24cyclictest0-21swapper/007:32:140
677599260,24cyclictest0-21swapper/007:26:060
135502260,2sleep20-21swapper/211:45:512
6778992522,2cyclictest0-21swapper/307:26:143
6778992512,11cyclictest6553-21ssh10:54:083
6778992512,11cyclictest20434-21ssh11:50:583
677899251,1cyclictest0-21swapper/307:32:103
677899250,23cyclictest0-21swapper/307:17:263
6777992523,1cyclictest0-21swapper/207:25:182
677599255,9cyclictest12-21ksoftirqd/011:04:050
677599254,13cyclictest12-21ksoftirqd/010:24:300
677599252,11cyclictest12-21ksoftirqd/011:15:520
6775992511,8cyclictest12-21ksoftirqd/010:02:330
6775992510,9cyclictest12-21ksoftirqd/012:10:260
6775992510,4cyclictest12-21ksoftirqd/010:54:140
6777992414,3cyclictest26140-21diskmemload11:55:152
6776992417,5cyclictest22-21ksoftirqd/111:20:011
677599249,8cyclictest12-21ksoftirqd/010:32:280
677599249,4cyclictest12-21ksoftirqd/011:14:190
677599249,4cyclictest12-21ksoftirqd/009:15:040
677599249,3cyclictest12-21ksoftirqd/011:44:070
677599248,5cyclictest12-21ksoftirqd/011:36:500
677599246,2cyclictest141rcu_preempt12:27:530
677599246,2cyclictest141rcu_preempt12:27:520
677599244,6cyclictest12-21ksoftirqd/012:06:370
145762240,1sleep20-21swapper/209:25:102
6778992314,5cyclictest26140-21diskmemload10:45:033
677799237,6cyclictest28-21ksoftirqd/211:44:192
6777992313,7cyclictest15970-21awk11:00:222
677699239,12cyclictest22131-21ssh11:29:241
6776992313,7cyclictest1840-21sed10:50:191
677599238,8cyclictest12-21ksoftirqd/011:27:310
677599238,3cyclictest141rcu_preempt12:22:010
677599236,4cyclictest12-21ksoftirqd/012:30:400
6775992315,6cyclictest12-21ksoftirqd/008:40:000
6775992314,6cyclictest14958-21df_inode12:35:130
62102230,2sleep30-21swapper/309:42:163
182152230,3chrt0-21swapper/111:02:001
677899223,16cyclictest12018-21gltestperf11:45:173
677899223,14cyclictest12826-21ssh12:10:123
677899222,14cyclictest26588-21ssh09:10:123
677899221,5cyclictest26140-21diskmemload11:44:193
6778992212,9cyclictest22539-21ssh10:18:063
6777992213,7cyclictest32007-21df_inode12:00:122
6777992213,6cyclictest7108-21awk10:30:182
677699221,15cyclictest9492-21ssh09:45:041
677699220,10cyclictest66050lldpd07:50:541
677599228,4cyclictest12-21ksoftirqd/010:35:090
677599227,8cyclictest12-21ksoftirqd/010:41:540
677599223,13cyclictest7460-21apt-get10:55:040
6775992210,4cyclictest12-21ksoftirqd/008:29:210
269752220,9sleep122-21ksoftirqd/111:32:151
677899211,5cyclictest0-21swapper/311:38:493
6778992114,4cyclictest0-21swapper/308:00:123
6778992111,7cyclictest14010-21gpgv12:35:013
6777992113,6cyclictest18531-21ntp_states08:50:212
6777992113,6cyclictest1735-21ls12:25:192
6777992113,6cyclictest1735-21ls12:25:182
6777992112,7cyclictest14628-21users08:40:252
6777992112,5cyclictest53050irq/142-enp3s0-12:30:112
6777992111,7cyclictest560-21dbus-daemon09:25:012
6777992111,6cyclictest53050irq/142-enp3s0-12:10:102
6776992111,7cyclictest21224-21sed09:30:011
6776992111,6cyclictest52850irq/140-enp3s0-10:25:131
677599217,4cyclictest12-21ksoftirqd/010:45:170
6775992116,4cyclictest12-21ksoftirqd/008:00:010
6775992112,6cyclictest278-21systemd-journal09:10:000
677899207,8cyclictest52950irq/141-enp3s0-09:15:143
677899202,13cyclictest23262-21ssh11:30:033
677899201,6cyclictest4871-21diskstats10:05:143
677899201,6cyclictest4871-21diskstats10:05:143
677899201,4cyclictest0-21swapper/309:03:393
6778992013,4cyclictest31235-21idleruntime-cro12:00:003
6778992011,7cyclictest14388-21sed12:35:003
6778992011,6cyclictest6535-21cut10:30:153
677899201,16cyclictest17706-21grep08:50:133
677799206,9cyclictest53050irq/142-enp3s0-09:30:072
677799202,12cyclictest16358-21apt-get11:25:132
6777992014,4cyclictest6106-21cpuspeed_turbos12:05:112
6777992014,4cyclictest10312-21hwlatdetect09:45:192
6777992013,5cyclictest28701-21sh09:10:482
6777992013,5cyclictest12049-21interrupts10:10:172
6777992013,4cyclictest10707-21sh10:09:502
6777992013,4cyclictest10707-21sh10:09:502
6777992012,6cyclictest6632-21cut08:25:002
6777992012,6cyclictest21311-21users07:40:262
6777992012,6cyclictest21311-21users07:40:262
6777992012,6cyclictest13965-21ntpq10:35:202
6777992011,6cyclictest3191-21cat08:15:192
6777992011,6cyclictest25943-21netstat10:20:222
6777992011,5cyclictest53050irq/142-enp3s0-10:50:182
6776992015,3cyclictest21682-21cron07:45:011
6776992015,3cyclictest21682-21cron07:45:001
6776992015,3cyclictest19146-21sh12:14:591
6776992014,4cyclictest52850irq/140-enp3s0-09:13:051
6776992012,6cyclictest28812-21perl08:00:151
6776992012,6cyclictest2834-21fw_packets08:15:171
6776992012,6cyclictest27348-21readlink07:55:221
6776992012,5cyclictest52850irq/140-enp3s0-12:25:111
6776992012,5cyclictest52850irq/140-enp3s0-12:25:111
6776992011,7cyclictest7983-21sed08:25:251
6776992011,7cyclictest27917-21processes10:45:231
6776992011,6cyclictest26716-21grep12:20:131
6776992011,6cyclictest2380-21cron11:15:001
677699201,14cyclictest6029-21ssh12:05:111
6776992010,7cyclictest560-21dbus-daemon12:15:001
677599204,6cyclictest12-21ksoftirqd/008:52:130
6775992015,4cyclictest12-21ksoftirqd/007:45:010
6775992015,4cyclictest12-21ksoftirqd/007:45:000
6775992014,4cyclictest12-21ksoftirqd/008:14:590
6775992012,6cyclictest14118-21grep08:40:220
6775992010,4cyclictest12-21ksoftirqd/009:00:190
677899198,7cyclictest3578-21chrt10:04:323
677899197,9cyclictest560-21dbus-daemon09:05:013
677899196,10cyclictest10528-21ssh09:45:193
677899191,8cyclictest6381-21diskstats12:05:133
677899191,7cyclictest21853-21grep11:05:123
677899191,5cyclictest0-21swapper/311:04:503
6778991914,3cyclictest30326-21sh10:24:013
6778991911,6cyclictest9302-21sed09:20:253
6778991911,6cyclictest4480-21grep11:15:253
6778991911,5cyclictest52950irq/141-enp3s0-09:30:003
6778991910,5cyclictest52950irq/141-enp3s0-09:30:113
677799198,7cyclictest22232-21df_inode09:00:142
677799197,9cyclictest0-21swapper/209:50:022
677799197,10cyclictest27585-21memory10:45:202
677799196,10cyclictest0-21swapper/210:00:142
677799191,9cyclictest8107-21ssh10:55:122
677799191,8cyclictest26140-21diskmemload10:26:512
677799191,7cyclictest26140-21diskmemload10:42:532
6777991914,3cyclictest18609-21processes07:35:232
6777991913,3cyclictest22194-21sh12:16:062
6777991912,5cyclictest9832-21expr11:20:122
6777991912,5cyclictest53050irq/142-enp3s0-11:15:252
6777991912,4cyclictest3551-21sh09:16:182
6777991911,6cyclictest9037-21expr08:30:142
6777991911,6cyclictest9037-21expr08:30:132
6777991911,6cyclictest7142-21fw_packets08:25:162
6777991911,6cyclictest31758-21ls11:35:242
6777991911,6cyclictest29318-21cat11:10:202
6777991911,6cyclictest24643-21smartctl07:50:162
6777991911,6cyclictest22198-21tune2fs11:05:152
6777991911,5cyclictest22891-21awk07:45:222
6777991911,5cyclictest1339-21mailstats08:10:222
6777991910,7cyclictest560-21dbus-daemon08:00:002
6777991910,7cyclictest28797-21tune2fs08:00:152
6777991910,6cyclictest3176-21expr09:40:152
6777991910,6cyclictest11116-21expr08:35:122
677699199,8cyclictest14180-21cron09:25:001
677699198,9cyclictest23790-21turbostat.cron09:05:001
677699198,7cyclictest52850irq/140-enp3s0-11:20:101
6776991913,4cyclictest31958-21sh11:35:261
6776991913,4cyclictest22890-21sh10:18:261
6776991913,4cyclictest20027-21sh11:50:351
6776991912,5cyclictest5413-21ntp_kernel_err08:20:191
6776991911,6cyclictest6887-21cat07:10:111
6776991911,6cyclictest25109-21cut10:20:141
6776991911,6cyclictest17864-21cat08:50:151
6776991911,6cyclictest17756-21run-parts09:50:241
6776991911,6cyclictest17418-21wc08:50:001
6776991911,6cyclictest10106-21seq11:44:101
6776991911,5cyclictest25504-21timerandwakeup09:05:241
6776991911,5cyclictest22-21ksoftirqd/109:35:101
6776991910,6cyclictest22355-21cut07:45:151
677599198,4cyclictest12-21ksoftirqd/008:04:300
677599197,3cyclictest12-21ksoftirqd/007:50:550
6775991914,4cyclictest12-21ksoftirqd/008:35:000
6775991914,4cyclictest12-21ksoftirqd/008:35:000
6775991914,4cyclictest12-21ksoftirqd/007:15:000
6775991911,5cyclictest18340-21missed_timers07:35:180
677599190,10cyclictest636-21runrttasks08:55:300
677899187,7cyclictest560-21dbus-daemon10:55:013
677899186,8cyclictest6674-21apt-get08:25:133
677899185,9cyclictest2389-21ssh12:25:263
677899185,9cyclictest2389-21ssh12:25:253
6778991813,3cyclictest5079-21sshd10:29:243
6778991813,3cyclictest28069-21sh09:35:023
6778991812,4cyclictest31571-21sh12:00:083
6778991812,4cyclictest21689-21latency_hist07:45:013
6778991812,4cyclictest21689-21latency_hist07:45:003
6778991812,3cyclictest636-21runrttasks08:10:163
6778991811,5cyclictest23886-21sed07:50:003
6778991811,4cyclictest11131-21sh10:10:023
6778991810,6cyclictest9433-21packagekitd09:50:063
6778991810,6cyclictest24195-21sed09:55:203
6778991810,6cyclictest14235-21tr08:40:233
6778991810,5cyclictest18883-21timerwakeupswit07:35:243
677799189,7cyclictest6814-21apt-get07:10:112
677799181,6cyclictest26140-21diskmemload10:18:132
6777991812,4cyclictest16297-21sh12:35:252
6777991812,3cyclictest28751-21sh09:35:142
6777991811,5cyclictest20064-21expr08:55:132
677699189,7cyclictest1915-21seq08:12:331
677699189,6cyclictest21163-21grep08:55:231
677699186,4cyclictest12957-21grep10:10:251
6776991813,3cyclictest9865-21sh10:55:281
6776991813,3cyclictest10945-21acpi08:35:111
6776991812,4cyclictest9951-21sh10:32:291
6776991812,4cyclictest28079-21sh09:58:261
6776991811,5cyclictest31974-21ls12:00:111
6776991811,5cyclictest22441-21cut11:05:171
6776991810,6cyclictest9326-21expr08:30:151
6776991810,6cyclictest9326-21expr08:30:151
6776991810,6cyclictest278-21systemd-journal08:10:001
6776991810,6cyclictest17658-21grep07:35:121
677599183,7cyclictest12-21ksoftirqd/008:05:090
6775991811,5cyclictest3213-21grep08:15:190
284472180,2sleep30-21swapper/311:10:133
677899178,6cyclictest25382-21cat07:50:243
677899176,8cyclictest0-21swapper/312:18:403
677899173,9cyclictest19744-21apt-get08:55:113
6778991713,2cyclictest6789-21/usr/sbin/munin07:10:163
6778991712,3cyclictest13784-21sh11:22:523
6778991711,4cyclictest9176-21sort08:30:153
6778991711,4cyclictest9176-21sort08:30:153
6778991710,5cyclictest21189-21cat10:40:243
6778991710,4cyclictest31032-21grep08:05:163
677799178,7cyclictest15664-21expr08:45:132
6777991710,5cyclictest31957-21awk08:05:252
6777991710,5cyclictest25482-21date09:05:232
6776991712,3cyclictest21329-21sh09:30:071
6776991712,3cyclictest20768-21sh10:40:191
6776991711,4cyclictest9867-21sh12:30:551
677599171,7cyclictest17291-21sshd08:49:230
677899169,5cyclictest0-21swapper/308:20:163
6778991610,4cyclictest4260-21sendmail08:19:593
6778991610,4cyclictest11651-21grep08:35:183
6776991610,4cyclictest14242-21mailstats08:40:221
6778991510,3cyclictest15272-21apt-key08:45:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional