You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 07:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed May 13, 2026 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
313672152117,25sleep10-21swapper/119:07:531
311982149127,15sleep30-21swapper/319:05:423
313982141106,24sleep20-21swapper/219:08:182
312252139102,26sleep00-21swapper/019:06:020
136482670,3sleep122-21ksoftirqd/121:53:281
75772640,1sleep00-21swapper/022:12:420
61532530,2sleep30-21swapper/322:11:163
230732520,2sleep10-21swapper/121:35:551
50772500,2sleep10-21swapper/119:20:231
189642490,1sleep10-21swapper/100:20:401
186992490,1sleep10-21swapper/121:56:451
180452480,1sleep00-21swapper/023:55:430
126052480,1sleep00-21swapper/023:28:140
91862460,2sleep20-21swapper/223:01:312
127392440,1sleep20-21swapper/200:16:072
25052430,1sleep00-21swapper/020:30:170
254882370,2sleep20-21swapper/223:37:382
183202370,2chrt0-21swapper/321:56:223
49352340,5chrt0-21swapper/122:59:021
294302300,3chrt0-21swapper/322:29:143
257522300,2chrt0-21swapper/123:14:381
211942300,2sleep00-21swapper/021:35:130
3180399296,12cyclictest12-21ksoftirqd/021:52:170
303182290,1sleep30-21swapper/321:41:223
172432290,2sleep10-21swapper/122:20:131
158772290,1chrt0-21swapper/322:42:463
31803992811,8cyclictest12-21ksoftirqd/022:28:100
152472280,1sleep10-21swapper/122:18:361
3180399274,7cyclictest12-21ksoftirqd/022:05:230
3180499261,1cyclictest0-21swapper/119:13:351
3180399266,7cyclictest0-21swapper/023:33:080
3180399262,8cyclictest12-21ksoftirqd/022:47:120
31803992624,1cyclictest0-21swapper/019:14:360
31803992610,7cyclictest12-21ksoftirqd/000:10:540
164202260,8sleep228-21ksoftirqd/223:07:012
145552260,2sleep20-21swapper/219:45:012
91702250,1sleep10-21swapper/121:25:351
31806992523,1cyclictest0-21swapper/319:14:183
3180699252,15cyclictest9315-21ssh23:50:123
31805992523,1cyclictest0-21swapper/219:13:122
31805992514,6cyclictest7294-21apt-get21:25:132
3180399257,9cyclictest12-21ksoftirqd/022:00:560
3180399252,7cyclictest12-21ksoftirqd/023:00:510
3180399252,7cyclictest12-21ksoftirqd/021:41:150
3180399252,6cyclictest26773-21apt-get23:15:130
31803992521,3cyclictest131rcuc/022:24:190
3180399251,8cyclictest12-21ksoftirqd/023:24:580
3180399250,6cyclictest2035-21apt-get21:45:110
210112250,1sleep20-21swapper/223:34:542
156872250,9sleep122-21ksoftirqd/100:19:081
3180699241,10cyclictest64050snmpd20:15:173
3180399246,10cyclictest12-21ksoftirqd/000:30:570
3180399245,11cyclictest12-21ksoftirqd/023:52:260
3180399243,6cyclictest12-21ksoftirqd/022:57:140
3180399243,5cyclictest12-21ksoftirqd/023:15:010
3180399242,6cyclictest12-21ksoftirqd/021:12:570
31803992421,2cyclictest131rcuc/022:36:500
31803992420,3cyclictest131rcuc/021:27:330
31803992419,3cyclictest131rcuc/000:37:420
31803992419,3cyclictest131rcuc/000:00:200
3180399241,8cyclictest12-21ksoftirqd/022:50:320
3180399241,6cyclictest10236-21ssh00:15:070
27082240,2sleep10-21swapper/121:45:181
115772240,2sleep30-21swapper/321:51:263
102342240,16sleep228-21ksoftirqd/220:50:012
31805992312,8cyclictest560-21dbus-daemon21:20:002
3180499232,9cyclictest29051-21munin-run00:05:011
31804992318,3cyclictest7643-21idleruntime-cro23:24:591
3180399232,6cyclictest12-21ksoftirqd/022:42:140
3180399232,12cyclictest0-21swapper/021:59:430
31803992320,2cyclictest131rcuc/000:27:530
3180399231,9cyclictest12-21ksoftirqd/023:43:220
31803992319,3cyclictest131rcuc/021:34:420
31803992318,3cyclictest131rcuc/020:38:140
31803992318,3cyclictest131rcuc/019:38:190
3180399231,7cyclictest12-21ksoftirqd/000:21:380
3180399231,7cyclictest12-21ksoftirqd/000:05:120
31803992315,6cyclictest19581-21/usr/sbin/munin19:55:170
31803992310,4cyclictest244492cat20:05:230
3180399230,5cyclictest0-21swapper/023:05:560
242342230,1sleep30-21swapper/300:00:283
208132230,2sleep10-21swapper/122:45:571
19062230,2sleep30-21swapper/322:08:463
31806992216,4cyclictest34-21ksoftirqd/300:40:003
31806992214,5cyclictest18657-21diskmemload23:35:253
31806992213,6cyclictest52950irq/141-enp3s0-21:20:103
3180399229,7cyclictest141rcu_preempt23:45:210
3180399228,2cyclictest141rcu_preempt21:05:170
3180399223,5cyclictest0-21swapper/023:35:310
3180399221,9cyclictest12-21ksoftirqd/021:16:510
31803992218,3cyclictest131rcuc/022:17:480
31803992217,3cyclictest131rcuc/021:21:120
309112220,2sleep20-21swapper/222:05:252
296052220,3sleep318657-21diskmemload00:05:133
161012220,2sleep30-21swapper/323:06:423
3180699218,3cyclictest141rcu_preempt21:09:593
31806992113,6cyclictest20898-21ntpq23:10:223
31806992113,6cyclictest15384-21mailstats21:30:243
31806992113,6cyclictest13803-21seq23:29:273
3180599219,7cyclictest52850irq/140-enp3s0-21:50:122
3180599219,10cyclictest636-21runrttasks22:39:282
3180599212,13cyclictest4580-21ssh00:35:132
3180599212,13cyclictest1164-21apt-get23:20:042
31805992115,4cyclictest28-21ksoftirqd/221:10:012
31805992114,5cyclictest27195-21cron21:40:012
31805992112,7cyclictest24149-21tune2fs00:25:162
31805992112,6cyclictest20005-21grep23:10:152
3180499213,11cyclictest21559-21ssh23:35:121
3180499212,13cyclictest6411-21ssh23:00:131
31804992115,4cyclictest636-21runrttasks19:20:011
31804992112,6cyclictest29028-21cron00:05:011
31803992117,3cyclictest131rcuc/019:20:050
31803992113,6cyclictest25837-21fw_packets20:10:170
31803992110,9cyclictest131rcuc/020:46:000
3180399210,8cyclictest18537-21taskset19:53:160
3180399210,6cyclictest131rcuc/022:30:590
15412210,2sleep30-21swapper/300:32:223
31806992014,4cyclictest5502-21sh23:46:373
31806992014,4cyclictest3020-21timerandwakeup20:30:233
31806992013,4cyclictest4998-21mailstats00:10:253
31806992012,6cyclictest7887-21ls23:00:273
31806992012,6cyclictest560-21dbus-daemon21:19:593
31806992012,6cyclictest2244-21sed23:20:193
31806992012,6cyclictest16410-21entropy23:55:153
31806992012,5cyclictest18657-21diskmemload22:54:043
31806992012,5cyclictest18657-21diskmemload22:47:433
31806992011,6cyclictest20288-21cat19:55:243
31806992011,6cyclictest1768-21acpi19:15:123
3180699201,14cyclictest10331-21ssh22:15:113
3180699200,17cyclictest0-21swapper/319:40:193
3180599207,4cyclictest141rcu_preempt21:55:252
3180599202,14cyclictest19129-21ssh21:10:032
3180599201,8cyclictest8066-21apt-get23:25:112
31805992014,4cyclictest28-21ksoftirqd/220:30:002
31805992013,5cyclictest9079-21sed23:50:002
31805992012,6cyclictest20269-21cat19:55:242
31805992011,7cyclictest278-21systemd-journal22:20:002
31805992011,7cyclictest16160-21users19:45:252
31805992010,7cyclictest3980-21cat22:10:152
3180599200,17cyclictest28729-21pluginstate21:40:212
3180499203,11cyclictest13096-21ssh23:05:111
31804992014,4cyclictest17328-21hwlatdetect21:05:161
31804992013,5cyclictest10085-21munin-run20:50:001
31804992012,6cyclictest560-21dbus-daemon00:00:001
31804992012,6cyclictest29920-21seq22:29:431
31804992012,6cyclictest1076-21smartctl21:20:141
31804992012,5cyclictest30936-21tail00:30:181
31804992011,7cyclictest4624-21seq22:10:211
31804992011,5cyclictest23683-21sh00:25:141
3180399205,6cyclictest12-21ksoftirqd/020:16:150
3180399204,9cyclictest12-21ksoftirqd/019:29:370
3180399202,7cyclictest12-21ksoftirqd/019:15:260
3180399202,13cyclictest6053-21apt-get20:40:140
3180399201,7cyclictest12-21ksoftirqd/019:41:230
31803992011,6cyclictest30346-21fw_packets20:20:160
31803992011,6cyclictest21429-21diskstats20:00:130
3180399200,7cyclictest12-21ksoftirqd/021:00:410
3180399200,7cyclictest12-21ksoftirqd/020:26:160
3180399200,7cyclictest12-21ksoftirqd/019:48:150
3180699197,3cyclictest91722ssh21:25:343
3180699195,9cyclictest24024-21df00:25:153
3180699195,2cyclictest141rcu_preempt23:40:483
3180699192,12cyclictest29882-21apt-get20:20:123
31806991914,3cyclictest6990-21open_files20:40:233
31806991911,6cyclictest9138-21uname19:30:203
31806991911,6cyclictest27444-21basename23:15:173
31806991911,6cyclictest22832-21forks22:00:163
31806991911,6cyclictest15389-21sed19:45:173
31806991911,5cyclictest7575-21grep19:25:263
31806991911,5cyclictest22465-21sh22:24:013
3180699191,13cyclictest5053-21apt-get22:35:133
31806991910,7cyclictest11815-21cat00:15:253
31806991910,6cyclictest8282-21expr20:45:133
31806991910,6cyclictest827-21cat20:25:253
31806991910,6cyclictest16781-21acpi19:50:113
3180599193,4cyclictest28-21ksoftirqd/200:14:192
3180599193,4cyclictest141rcu_preempt21:00:122
3180599192,12cyclictest32060-21ssh22:55:132
31805991913,4cyclictest28-21ksoftirqd/222:35:002
31805991913,3cyclictest12211-21sh20:55:002
31805991912,5cyclictest20278-21users22:45:262
31805991912,4cyclictest22441-21sh22:00:132
31805991911,6cyclictest560-21dbus-daemon19:29:592
31805991911,6cyclictest31118-21ls20:20:232
31805991911,6cyclictest21421-21df_inode20:00:122
31805991911,6cyclictest13238-21sed20:55:192
3180499199,7cyclictest53050irq/142-enp3s0-21:15:141
3180499191,3cyclictest0-21swapper/100:35:231
31804991913,4cyclictest19189-21sh23:33:051
31804991912,5cyclictest11851-21awk19:35:251
31804991912,5cyclictest11655-21sed20:50:251
31804991912,4cyclictest8882-21sh23:50:001
31804991911,6cyclictest28696-21cat20:15:231
3180499191,15cyclictest9131-21chrt23:50:021
3180499191,14cyclictest640-21snmpd19:40:101
31803991911,6cyclictest12011-21seq20:52:360
31803991910,6cyclictest8214-21cat19:30:110
3180399190,7cyclictest12-21ksoftirqd/020:55:320
244472190,7sleep228-21ksoftirqd/220:05:232
3180699189,7cyclictest11166-21cat19:35:193
3180699189,6cyclictest25962-21idleruntime20:10:183
3180699189,6cyclictest24588-21cat20:05:243
3180699185,3cyclictest141rcu_preempt21:10:243
31806991813,3cyclictest3864-21/usr/sbin/munin20:35:213
31806991813,3cyclictest2964-21sh22:33:253
31806991812,4cyclictest3924-21sh22:58:013
31806991812,4cyclictest16919-21sh00:20:083
31806991812,4cyclictest14678-21sh23:30:053
31806991811,4cyclictest4673-21sh21:46:183
3180699181,12cyclictest14625-21apt-get21:00:133
3180699181,12cyclictest12462-21apt-get20:55:123
3180599187,9cyclictest0-21swapper/223:58:092
3180599187,9cyclictest0-21swapper/222:21:142
3180599187,6cyclictest52850irq/140-enp3s0-21:20:102
3180599182,12cyclictest28277-21apt-get23:40:132
3180599181,8cyclictest13972-21ssh21:30:072
3180599181,7cyclictest24003-21ssh00:00:242
31805991813,3cyclictest16964-21/usr/sbin/munin00:20:162
31805991812,4cyclictest636-21runrttasks22:26:582
31805991811,5cyclictest8425-21cat19:30:142
31805991811,5cyclictest30126-21sh22:53:332
31805991811,5cyclictest25548-21ls20:10:142
31805991811,5cyclictest2186-21perl20:30:132
31805991811,5cyclictest10348-21find19:35:112
31805991810,6cyclictest16790-21basename19:50:112
3180499182,11cyclictest26137-21apt-get23:15:041
3180499182,10cyclictest25288-21apt-get22:50:111
3180499181,8cyclictest8596-21perl23:25:161
3180499181,8cyclictest22963-21/usr/sbin/munin22:00:171
3180499181,8cyclictest14726-21hddtemp_smartct21:30:181
31804991813,3cyclictest1467-21sh22:08:201
31804991812,4cyclictest9061-21sh00:14:091
31804991812,4cyclictest14050-21sh22:40:561
31804991811,5cyclictest23975-21sed20:05:171
31804991811,5cyclictest16622-21date19:50:001
31804991811,5cyclictest15717-21which21:00:231
31804991811,5cyclictest1494-21cron20:29:591
31804991810,6cyclictest6045-21awk20:40:131
31804991810,6cyclictest31953-21tr22:30:201
31804991810,5cyclictest26626-21timerandwakeup20:10:261
31804991810,3cyclictest5633-21gltestperf22:35:161
3180699172,10cyclictest10327-21apt-get20:50:133
31806991711,4cyclictest21278-21fschecks_count21:35:143
3180699171,11cyclictest3946-21apt-get19:20:143
3180599179,6cyclictest27810-21cat23:15:202
3180599179,4cyclictest29600-21cpuspeed_turbos00:05:122
3180599178,7cyclictest4028-21sort19:20:142
3180599172,10cyclictest11820-21sh22:40:132
31805991712,3cyclictest13058-21sh23:52:282
31805991711,4cyclictest6751-21sed20:40:192
31805991711,4cyclictest53050irq/142-enp3s0-20:35:222
31805991710,5cyclictest2309-21expr21:45:142
31805991710,5cyclictest1717-21sed19:15:002
3180599170,8cyclictest4261-21cat00:35:012
3180499179,6cyclictest19185-21ls19:55:121
3180499179,5cyclictest1845-21grep20:30:111
31804991712,3cyclictest28479-21cstates23:40:141
31804991712,3cyclictest27991-21sh21:40:131
31804991712,3cyclictest18657-21diskmemload21:10:441
3180499171,11cyclictest16796-21apt-get19:50:131
31804991710,5cyclictest31111-21sed20:20:241
3180699169,5cyclictest23082-21date20:05:013
3180599162,10cyclictest27510-21apt-get20:15:132
3180499169,5cyclictest21849-21cut20:00:171
3180499161,6cyclictest12437-21/usr/sbin/munin20:55:141
31804991612,2cyclictest8529-21munin-node20:35:091
31804991611,3cyclictest6966-21ntp_kernel_err19:25:211
3180499148,4cyclictest8858-21interrupts19:30:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional