You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 16:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri May 08, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
272252240163,24sleep30-21swapper/307:08:223
271672237162,25sleep20-21swapper/207:07:352
270262236162,23sleep10-21swapper/107:05:461
271752235159,24sleep00-21swapper/007:07:420
27447991091,3cyclictest0-21swapper/209:30:232
274479910392,9cyclictest0-21swapper/208:57:412
274479910197,3cyclictest0-21swapper/208:15:542
51182530,1sleep20-21swapper/212:25:122
247532530,2sleep10-21swapper/108:15:131
28372510,4sleep339-21rcuc/309:50:223
269512500,2sleep30-21swapper/308:20:123
2745499421,28cyclictest8202-21/usr/sbin/munin10:05:153
2745499421,25cyclictest958-21/usr/sbin/munin08:35:133
249142420,4sleep39-21rcu_preempt08:15:153
2745499411,32cyclictest5374-21/usr/sbin/munin08:45:233
2745499411,26cyclictest19802-21/usr/sbin/munin11:45:133
2744299411,30cyclictest18208-21gltestperf11:40:151
2744299411,27cyclictest8202-21/usr/sbin/munin10:05:201
2745499409,22cyclictest7571-21/usr/sbin/munin08:50:233
2744299407,23cyclictest18338-21latency_hist09:15:021
2744299407,23cyclictest18338-21latency_hist09:15:021
2744299401,29cyclictest473-21perl11:00:151
2744299401,29cyclictest11355-21/usr/sbin/munin07:45:121
2745499398,22cyclictest19539-21entropy10:30:153
2744299396,22cyclictest15470-21/usr/sbin/munin10:20:151
2744299391,34cyclictest9768-21gltestperf07:40:141
2744299391,31cyclictest11442-21cat12:40:001
2744299391,28cyclictest5659-21sed07:30:171
2744299391,24cyclictest1594-21/usr/sbin/munin09:50:191
27442993910,21cyclictest11734-21latency_hist09:00:011
2745499388,20cyclictest22964-21/usr/sbin/munin09:25:243
2745499387,20cyclictest26396-21/usr/sbin/munin12:00:193
2744299381,2cyclictest0-21swapper/109:55:171
2744299381,28cyclictest15402-21/usr/sbin/munin11:35:161
2744299381,24cyclictest1537-21crond08:05:011
2744299381,24cyclictest1537-21crond08:05:011
2744299381,24cyclictest0-21swapper/110:47:401
2744299375,21cyclictest0-21swapper/111:10:121
2744299373,24cyclictest8814-21/usr/sbin/munin11:20:251
2745499362,30cyclictest30530-21/usr/sbin/munin07:15:123
2745499361,24cyclictest658-21latency_hist08:35:023
2744299367,26cyclictest23-21ksoftirqd/108:45:071
2744299361,32cyclictest24951-21awk09:30:011
2744299361,24cyclictest30803-21/usr/sbin/munin12:10:151
2744299361,24cyclictest27983-21/usr/sbin/munin10:50:221
2744299361,24cyclictest20154-21entropy11:45:151
2744299361,24cyclictest17136-21tr10:25:101
27435993610,13cyclictest22744-21munin-run09:25:010
2745499354,21cyclictest0-21swapper/309:25:013
2745499351,4cyclictest5995-21/usr/sbin/munin10:00:143
2745499351,24cyclictest1402-21cat09:50:023
2744299356,20cyclictest18790-21expr09:15:131
2744299353,23cyclictest87350irq/126-enp1s0-10:35:121
2744299353,14cyclictest0-21swapper/109:35:241
2744299352,30cyclictest1122-21taskset09:47:101
2744299351,27cyclictest0-21swapper/107:28:201
2744299351,23cyclictest12050irq/123-ahci[0011:55:021
2744299350,1cyclictest0-21swapper/108:00:011
2745499343,22cyclictest12050irq/123-ahci[0011:45:023
2745499341,7cyclictest27360-21/usr/sbin/munin09:35:203
2745499341,6cyclictest9769-21/usr/sbin/munin08:55:163
2745499340,17cyclictest3177-21/usr/sbin/munin08:40:183
2744299345,17cyclictest13218-21grep10:15:131
27442993418,13cyclictest22341-21grep09:20:261
2744299341,3cyclictest26838-21cpuspeed_turbos08:20:121
2744299341,30cyclictest8587-21ls11:20:011
2744299341,2cyclictest24486-21sed08:15:021
2744299341,29cyclictest827-21runrttasks11:58:341
2744299341,24cyclictest4058-21awk12:20:211
2744299341,23cyclictest545-21/usr/sbin/munin12:15:171
2744299341,21cyclictest12362-21ls10:15:001
2744299341,18cyclictest7230-21/usr/sbin/munin12:30:221
2744299341,18cyclictest0-21swapper/109:45:021
27442993410,21cyclictest32493-21latency_hist07:20:021
2745499337,14cyclictest0-21swapper/308:10:223
2745499332,3cyclictest87350irq/126-enp1s0-10:40:313
2745499332,27cyclictest20503-21diskstats08:05:143
2745499332,17cyclictest16370-21/usr/sbin/munin09:10:213
2745499332,17cyclictest16370-21/usr/sbin/munin09:10:213
2745499331,5cyclictest0-21swapper/312:05:113
2745499331,20cyclictest5035-21/usr/sbin/munin12:25:143
2744299338,17cyclictest0-21swapper/108:35:161
2744299332,28cyclictest5848-21irqstats12:25:161
2744299331,8cyclictest30533-21sshd10:55:151
2744299331,6cyclictest13551-21/usr/sbin/munin07:50:121
2744299331,29cyclictest9093-21cat07:40:001
2744299331,29cyclictest5142-21latency_hist08:45:011
2744299331,22cyclictest29613-21gltestperf08:25:151
2744299331,21cyclictest13545-21df11:30:131
2744299331,21cyclictest13545-21df11:30:131
2745499323,19cyclictest17968-21sed10:25:183
2745499321,17cyclictest27130-21latency_hist09:35:023
2745499321,17cyclictest16140-21diskmemload11:13:373
2745499321,17cyclictest0-21swapper/312:30:223
2745499320,5cyclictest0-21swapper/310:55:023
2744299324,19cyclictest87650irq/129-enp1s0-07:20:161
2744299322,26cyclictest7065-21ntp_states10:00:221
2744299322,26cyclictest3449-21grep11:05:221
2744299321,28cyclictest31884-21if_enp1s008:30:161
2744299321,28cyclictest0-21swapper/107:15:021
2743599329,11cyclictest32168-21perf11:00:020
2745499313,2cyclictest87550irq/128-enp1s0-12:35:183
2745499312,3cyclictest11103-40dnf07:44:533
2745499311,5cyclictest26903-21/usr/sbin/munin10:45:213
2745499311,26cyclictest9230-21sshd11:20:143
2745499311,15cyclictest0-21swapper/310:10:153
2744299311,4cyclictest808-21snmpd11:25:141
2744299311,27cyclictest26743-21diskstats12:00:141
2744299311,18cyclictest19585-21grep10:30:141
2744299310,20cyclictest26605-21timerwakeupswit09:30:241
2744299310,1cyclictest0-21swapper/108:05:121
2743599314,13cyclictest24940-21munin-run09:30:010
154672310,2sleep20-21swapper/211:35:112
2745499309,18cyclictest8439-21unixbench-2d07:35:253
2745499309,18cyclictest16264-21sort11:35:173
2745499309,17cyclictest18739-21cstates09:15:123
2745499302,3cyclictest87350irq/126-enp1s0-10:55:143
2745499301,5cyclictest0-21swapper/311:05:243
2745499301,4cyclictest18575-21/usr/sbin/munin08:00:163
2745499301,4cyclictest18575-21/usr/sbin/munin08:00:163
2745499301,25cyclictest25262-21ntp_states11:55:203
2745499301,25cyclictest12958-21strings09:00:173
2745499301,18cyclictest827-21runrttasks11:51:283
2745499301,18cyclictest1745-21taskset11:01:503
2744299309,18cyclictest14683-21fschecks_time09:05:151
2744299301,4cyclictest24571-21strings10:40:191
2745499299,17cyclictest12632-21switchtime07:45:233
2745499298,18cyclictest29609-21dump-pmu-power07:15:003
2745499293,3cyclictest87650irq/129-enp1s0-10:15:243
2745499292,24cyclictest15245-21ntp_states09:05:203
2745499291,4cyclictest87350irq/126-enp1s0-12:23:323
2745499291,4cyclictest0-21swapper/309:44:563
2745499291,3cyclictest839-21lldpd11:27:383
2745499291,25cyclictest13208-21/usr/sbin/munin11:30:223
2745499291,25cyclictest13208-21/usr/sbin/munin11:30:213
2745499291,24cyclictest21715-21df10:35:123
2745499291,20cyclictest808-21snmpd08:27:273
2745499291,18cyclictest3959-21tr09:55:113
27454992910,16cyclictest5565-21interrupts07:30:173
2745499290,3cyclictest87650irq/129-enp1s0-12:15:253
2744299291,24cyclictest30209-21/usr/sbin/munin12:05:261
27442992910,16cyclictest14002-21date09:05:011
2745499286,19cyclictest7528-21load11:15:173
2745499286,19cyclictest15883-21tr07:55:123
2745499283,22cyclictest4536-21munin-run07:30:023
2745499282,4cyclictest1958-21users07:20:253
2745499281,6cyclictest0-21swapper/312:11:013
27454992810,15cyclictest14609-21ntp_offset07:50:193
2744299281,24cyclictest7572-21acpi08:50:111
2743599281,13cyclictest2964-21awk08:40:020
2745499271,2cyclictest14787-21/usr/sbin/munin10:20:113
2743599275,11cyclictest0-21swapper/011:45:000
2743599263,11cyclictest29624-21idleruntime-cro07:15:010
2743599258,15cyclictest16160-21perf09:10:010
242732240,2sleep20-21swapper/209:25:252
2743599236,14cyclictest19932-21munin-run08:05:010
2743599236,14cyclictest19932-21munin-run08:05:010
2743599231,6cyclictest0-21swapper/009:42:310
2744799221,5cyclictest0-21swapper/210:31:342
27447992215,5cyclictest16982-21find07:55:212
2743599224,3cyclictest5384-21users09:55:230
2743599221,9cyclictest0-21swapper/012:33:320
2743599221,10cyclictest16363-21kworker/u8:009:30:160
2744799211,5cyclictest0-21swapper/211:18:552
2744799211,5cyclictest0-21swapper/210:44:262
2743599216,6cyclictest0-21swapper/011:54:530
2743599212,11cyclictest21381-21/usr/sbin/munin10:35:160
2743599211,6cyclictest0-21swapper/012:24:210
27435992114,4cyclictest28273-21grep08:20:250
2743599211,11cyclictest27983-21/usr/sbin/munin10:50:120
2744799202,7cyclictest15524-21munin-run07:55:012
2744799201,5cyclictest0-21swapper/211:33:512
2744799201,5cyclictest0-21swapper/211:33:512
2744799201,4cyclictest0-21swapper/210:49:592
2744799201,10cyclictest16140-21diskmemload10:25:182
2743599209,3cyclictest24197-21/usr/sbin/munin11:55:180
2743599206,6cyclictest0-21swapper/011:25:540
2743599206,5cyclictest0-21swapper/010:02:530
2743599205,14cyclictest0-21swapper/011:10:010
2743599204,7cyclictest0-21swapper/010:46:400
2743599204,13cyclictest14211-21cut07:50:140
2743599202,10cyclictest545-21/usr/sbin/munin12:15:170
2743599202,10cyclictest30803-21/usr/sbin/munin12:10:190
2743599201,3cyclictest0-21swapper/010:06:470
2743599201,10cyclictest4759-21/usr/sbin/munin07:30:180
2743599201,10cyclictest0-21swapper/011:15:540
240582201,2sleep20-21swapper/208:11:452
2744799195,4cyclictest12590-21/usr/sbin/munin10:15:242
2744799192,5cyclictest20967-21irqstats08:05:172
2744799191,8cyclictest8219-21sensors_temp07:35:232
2744799191,6cyclictest32766-21crond12:15:022
2744799191,4cyclictest0-21swapper/212:35:282
27447991913,3cyclictest6808-21irqstats10:00:182
2743599197,6cyclictest0-21swapper/011:03:130
2743599194,6cyclictest0-21swapper/009:19:340
2743599193,3cyclictest16742-21fschecks_count09:10:140
2743599193,3cyclictest16742-21fschecks_count09:10:140
2743599193,10cyclictest0-21swapper/011:48:500
2743599192,14cyclictest5029-21grep11:10:150
2743599191,5cyclictest0-21swapper/010:29:130
2743599191,12cyclictest0-21swapper/010:40:160
2743599191,11cyclictest0-21swapper/012:25:210
2743599191,10cyclictest28123-21awk09:35:160
2743599190,9cyclictest0-21swapper/011:31:320
2743599190,9cyclictest0-21swapper/011:31:320
129982191,14sleep013032-21ntp_states09:00:190
2744799187,3cyclictest9829-21sa107:40:162
2744799186,4cyclictest5013-21processes09:55:212
2744799183,3cyclictest87350irq/126-enp1s0-10:05:142
2744799182,9cyclictest0-21swapper/211:01:502
2744799181,9cyclictest0-21swapper/209:22:252
2744799181,8cyclictest8230-21grep08:50:142
2744799181,7cyclictest579-21systemd-journal08:30:012
2744799181,7cyclictest15440-21cut10:20:152
2744799181,7cyclictest0-21swapper/210:51:442
2744799181,7cyclictest0-21swapper/209:35:192
2744799181,6cyclictest0-21swapper/211:58:162
2744799181,6cyclictest0-21swapper/209:42:202
2744799181,5cyclictest4629-21taskset12:22:502
2744799181,5cyclictest16743-21fschecks_count09:10:142
2744799181,5cyclictest16743-21fschecks_count09:10:142
2744799181,3cyclictest324812threads08:30:232
2744799180,15cyclictest27743-21strings08:20:192
2743599188,2cyclictest808-21snmpd07:38:100
2743599183,5cyclictest19570-21sshd10:30:150
2743599182,7cyclictest7571-21/usr/sbin/munin08:50:230
2743599181,8cyclictest15750-21/usr/sbin/munin07:55:230
2743599181,6cyclictest0-21swapper/012:01:150
2743599181,3cyclictest0-21swapper/008:30:150
27435991812,3cyclictest32095-21df09:45:120
2743599181,11cyclictest0-21swapper/012:36:350
2743599181,11cyclictest0-21swapper/009:54:440
2743599181,10cyclictest0-21swapper/010:12:200
2743599180,10cyclictest0-21swapper/011:35:310
126962181,2sleep20-21swapper/211:26:292
42472171,2sleep00-21swapper/007:26:290
2744799178,3cyclictest0-21swapper/211:52:142
2744799175,4cyclictest808-21snmpd11:22:592
2744799175,3cyclictest87650irq/129-enp1s0-10:35:112
2744799174,4cyclictest20907-21pmu-power11:45:232
2744799172,5cyclictest12359-21latency_hist10:15:012
2744799171,9cyclictest16363-21kworker/u8:009:49:132
2744799171,7cyclictest4397-21ps08:40:212
2744799171,6cyclictest0-21swapper/212:08:192
2744799171,6cyclictest0-21swapper/211:14:232
2744799171,4cyclictest0-21swapper/212:00:202
27447991711,3cyclictest2477-21unixbench_multi08:35:252
2744799171,10cyclictest0-21swapper/210:58:492
27447991710,4cyclictest19183-21tail09:15:162
27447991710,4cyclictest0-21swapper/209:50:172
2743599171,5cyclictest0-21swapper/008:25:120
2743599171,3cyclictest32724-21/usr/sbin/munin07:20:220
27435991713,3cyclictest0-21swapper/010:18:180
27435991712,3cyclictest29394-21/usr/sbin/munin12:05:150
2743599171,14cyclictest0-21swapper/008:55:490
27435991711,3cyclictest6413-21ntpq08:45:200
2743599171,10cyclictest0-21swapper/011:24:200
2743599171,10cyclictest0-21swapper/007:45:190
261902171,2sleep00-21swapper/008:15:570
43672160,2sleep20-21swapper/207:28:032
2744799163,11cyclictest30663-21cpuspeed_turbos07:15:112
2744799162,5cyclictest9205-21munin-run12:35:012
2744799161,7cyclictest19378-21/usr/sbin/munin08:00:242
2744799161,7cyclictest19378-21/usr/sbin/munin08:00:242
2744799161,7cyclictest0-21swapper/211:05:032
2744799161,6cyclictest7391-21cat08:50:012
2744799161,2cyclictest0-21swapper/209:03:392
27447991612,3cyclictest0-21swapper/211:41:092
27447991612,3cyclictest0-21swapper/207:46:362
27447991611,3cyclictest32717-21seq07:20:062
27447991611,3cyclictest16021-21seq09:08:372
2744799161,10cyclictest0-21swapper/212:16:422
27447991610,3cyclictest28320-21hwlatdetect07:10:152
2743599169,4cyclictest23347-21grep08:10:190
2743599165,3cyclictest30451-21missed_timers07:15:050
2743599161,9cyclictest0-21swapper/010:22:070
2743599161,9cyclictest0-21swapper/007:40:190
2744799151,6cyclictest0-21swapper/207:31:522
2743599151,6cyclictest0-21swapper/008:05:230
27435991511,3cyclictest0-21swapper/008:42:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional