You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 00:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Dec 09, 2025 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30932239164,24sleep10-21swapper/107:09:171
31402237162,24sleep30-21swapper/307:09:523
30712232163,55sleep00-21swapper/007:09:000
28242203165,25sleep20-21swapper/207:05:492
3170821790,3sleep0324799cyclictest09:25:160
1832321660,1sleep00-21swapper/012:35:150
3260991141,111cyclictest0-21swapper/211:39:032
324002530,4sleep123-21ksoftirqd/110:40:161
37032460,1sleep30-21swapper/310:50:013
242752420,5sleep3326499cyclictest10:20:263
3264993713,2cyclictest16392-21/usr/sbin/munin10:05:203
3247993711,12cyclictest0-21swapper/011:10:010
140642370,1sleep30-21swapper/311:10:243
326499365,11cyclictest0-21swapper/312:15:213
326499364,8cyclictest27098-40dnf10:29:333
326499363,16cyclictest8491-21grep08:30:253
3264993615,4cyclictest12607-21grep12:20:253
3264993613,2cyclictest21717-21/usr/sbin/munin07:50:153
3264993512,3cyclictest5392-21/usr/sbin/munin09:40:153
326499351,15cyclictest0-21swapper/309:45:273
326499351,11cyclictest32588-21/usr/sbin/munin10:40:173
326499348,4cyclictest29586-21/usr/sbin/munin10:35:253
326499341,5cyclictest15557-21cpuspeed_turbos12:30:133
325399341,8cyclictest0-21swapper/109:35:521
325399341,11cyclictest14634-21snmpd11:08:541
3247993410,11cyclictest27169-21awk10:30:010
326499338,3cyclictest9793-21smart_sda07:20:243
326499333,8cyclictest12412-40dnf07:26:323
3264993311,4cyclictest6486-21tr07:15:123
325399331,9cyclictest0-21swapper/109:50:301
326499328,8cyclictest16662-21sshd11:18:313
3264993213,3cyclictest21599-21kernelversion10:15:183
325399321,29cyclictest0-21swapper/111:45:181
325399320,12cyclictest0-21swapper/108:31:581
249922320,2sleep10-21swapper/109:10:161
326499317,3cyclictest18594-21/usr/sbin/munin10:10:183
326499311,4cyclictest4467-21cpu12:05:103
3264993113,3cyclictest31332-21cat09:25:133
325399311,8cyclictest0-21swapper/109:49:101
325399311,13cyclictest0-21swapper/111:16:461
325399311,12cyclictest0-21swapper/111:12:061
325399311,10cyclictest0-21swapper/110:02:221
325399310,12cyclictest0-21swapper/108:45:261
325399310,12cyclictest0-21swapper/108:45:261
324799313,13cyclictest5160-21sadc09:40:000
3264993017,3cyclictest21405-21smart_sda09:00:233
326499301,13cyclictest24972-21/usr/sbin/munin09:10:153
325399301,13cyclictest0-21swapper/111:54:141
325399301,13cyclictest0-21swapper/108:27:341
325399301,13cyclictest0-21swapper/108:13:101
325399301,12cyclictest0-21swapper/111:59:341
325399301,12cyclictest0-21swapper/110:32:461
325399301,11cyclictest0-21swapper/108:50:221
325399300,12cyclictest0-21swapper/111:21:581
3247993010,16cyclictest12705-21munin-run07:30:000
66922290,5sleep3326499cyclictest12:10:173
326499291,10cyclictest28009-21/usr/sbin/munin11:45:193
326499290,10cyclictest0-21swapper/308:13:473
325399291,8cyclictest0-21swapper/112:00:221
325399291,13cyclictest0-21swapper/110:38:221
325399291,13cyclictest0-21swapper/109:07:261
325399290,17cyclictest0-21swapper/111:00:041
326499287,3cyclictest10344-21sed11:05:013
326499281,5cyclictest23915-21/usr/sbin/munin07:55:153
3264992810,3cyclictest28840-21cut08:05:153
325399281,12cyclictest0-21swapper/112:27:101
325399281,12cyclictest0-21swapper/110:49:181
325399281,11cyclictest0-21swapper/107:48:381
325399280,13cyclictest0-21swapper/109:23:101
326499272,2cyclictest86750irq/128-enp1s0-09:35:473
326499270,11cyclictest0-21swapper/312:03:073
325399271,12cyclictest0-21swapper/112:20:061
325399271,12cyclictest0-21swapper/107:40:301
324799272,12cyclictest7-21ksoftirqd/011:45:000
326499269,3cyclictest0-21swapper/311:32:233
326499263,11cyclictest86650irq/127-enp1s0-08:27:273
326499262,2cyclictest86650irq/127-enp1s0-11:51:313
3264992615,3cyclictest0-21swapper/311:43:073
326499261,11cyclictest2906-21diskstats08:20:133
325399261,10cyclictest0-21swapper/108:36:301
325399260,12cyclictest0-21swapper/110:50:381
325399260,12cyclictest0-21swapper/110:50:381
324799261,12cyclictest7377-21perf09:45:000
326499252,6cyclictest0-21swapper/309:22:353
326499252,3cyclictest22689-21df09:05:133
325399257,9cyclictest0-21swapper/111:35:131
325399251,17cyclictest25192-21smart_sda07:55:351
325399251,17cyclictest0-21swapper/112:15:291
325399251,10cyclictest0-21swapper/112:05:261
325399251,10cyclictest0-21swapper/110:23:101
325399250,9cyclictest0-21swapper/110:11:021
325399250,13cyclictest0-21swapper/111:33:581
325399250,10cyclictest0-21swapper/109:16:221
324799256,16cyclictest4537-21munin-run08:25:010
324799252,12cyclictest13331-21munin-run08:45:000
326499241,2cyclictest86550irq/126-enp1s0-11:05:233
326499241,12cyclictest0-21swapper/310:33:353
3264992410,8cyclictest0-21swapper/309:58:573
325399241,9cyclictest0-21swapper/109:42:301
325399241,19cyclictest12412-40dnf07:26:341
324799246,15cyclictest17105-21munin-run07:40:000
324799243,2cyclictest86550irq/126-enp1s0-09:15:150
326499233,8cyclictest0-21swapper/308:38:113
3264992316,4cyclictest5411-21/usr/sbin/munin10:50:273
3264992316,4cyclictest5411-21/usr/sbin/munin10:50:273
326499231,13cyclictest14260-21sed07:30:233
326499230,11cyclictest0-21swapper/308:41:233
325399231,19cyclictest8349-21chrt12:11:371
325399231,13cyclictest0-21swapper/108:57:181
325399231,13cyclictest0-21swapper/107:33:341
324799238,3cyclictest84832tail08:30:250
324799231,3cyclictest14422-21proc_pri12:25:240
326499229,10cyclictest86750irq/128-enp1s0-07:36:353
326499227,8cyclictest0-21swapper/308:46:413
326499227,8cyclictest0-21swapper/308:46:413
326499224,3cyclictest1-21systemd07:15:023
326499223,5cyclictest17416-1kworker/u9:110:05:013
326499222,2cyclictest86750irq/128-enp1s0-12:28:433
326499221,3cyclictest17990-21diskstats12:35:133
326499221,2cyclictest86850irq/129-enp1s0-11:35:473
3264992212,8cyclictest86750irq/128-enp1s0-08:55:353
325399226,13cyclictest0-21swapper/108:09:371
325399221,9cyclictest0-21swapper/110:59:501
325399221,9cyclictest0-21swapper/110:05:561
325399221,3cyclictest848-21sssd_nss10:25:261
325399221,18cyclictest19541-21expr11:25:161
325399221,16cyclictest0-21swapper/108:42:071
325399221,12cyclictest0-21swapper/108:24:461
325399220,16cyclictest0-21swapper/109:25:051
85622210,1sleep20-21swapper/208:30:282
326499210,3cyclictest0-21swapper/311:29:393
326099212,16cyclictest3605-21ntp_kernel_pll_08:20:192
3260992114,4cyclictest24672-21tr09:10:132
3253992113,3cyclictest0-21swapper/110:16:541
3253992113,3cyclictest0-21swapper/108:00:071
325399211,15cyclictest0-21swapper/109:33:181
325399211,13cyclictest0-21swapper/107:14:211
325399211,12cyclictest0-21swapper/111:40:141
325399211,12cyclictest0-21swapper/107:20:061
325399211,11cyclictest6043-21users07:15:031
325399210,14cyclictest0-21swapper/107:50:141
324799211,9cyclictest14412-21kworker/0:211:18:030
3247992116,3cyclictest3091-21grep10:45:260
324799211,11cyclictest27411-21switchtime08:00:300
324799211,10cyclictest1204-21kworker/u8:211:13:270
324799210,10cyclictest0-21swapper/011:58:120
326499207,3cyclictest0-21swapper/309:54:573
326499202,10cyclictest17329-21/usr/sbin/munin07:40:173
3264992014,4cyclictest1444-21cut09:30:173
3264992013,3cyclictest6440-21grep10:55:173
326499201,2cyclictest86650irq/127-enp1s0-09:18:193
326099202,8cyclictest0-21swapper/210:29:332
326099201,7cyclictest0-21swapper/211:47:352
326099201,6cyclictest22765-21idleruntime-cro10:20:012
326099201,12cyclictest0-21swapper/210:41:262
325399201,18cyclictest0-21swapper/109:00:341
325399201,18cyclictest0-21swapper/107:36:391
325399201,11cyclictest1052-21idleruntime08:15:181
325399200,15cyclictest0-21swapper/109:58:511
324799209,3cyclictest19219-21/usr/sbin/munin11:25:220
324799208,3cyclictest2971-21idleruntime-cro09:35:000
324799206,5cyclictest0-21swapper/011:01:350
324799201,4cyclictest23695-21perf07:55:010
3247992014,3cyclictest17056-40dnf12:30:310
324799201,11cyclictest4576-21grep12:05:100
324799201,11cyclictest24556-21/usr/sbin/munin09:10:130
324799201,10cyclictest18074-21sshd10:06:310
324799201,10cyclictest0-21swapper/011:36:470
324799201,10cyclictest0-21swapper/008:16:070
324799200,10cyclictest0-21swapper/011:54:210
75832191,3sleep00-21swapper/009:45:080
326499197,3cyclictest12050irq/123-ahci[0008:05:013
326499191,3cyclictest0-21swapper/311:57:453
326499191,10cyclictest32721-21/usr/sbin/munin08:15:233
326499190,2cyclictest86750irq/128-enp1s0-11:23:473
326099199,3cyclictest86850irq/129-enp1s0-12:00:122
326099196,2cyclictest0-21swapper/211:33:052
326099193,3cyclictest17416-1kworker/u9:111:02:472
326099192,5cyclictest0-21swapper/209:20:152
326099192,15cyclictest0-21swapper/212:25:412
326099192,14cyclictest27357-21find08:00:292
3260991914,3cyclictest4904-21grep08:25:122
3260991914,3cyclictest15977-21grep12:30:162
326099191,15cyclictest13857-21find11:10:222
326099191,12cyclictest0-21swapper/209:44:312
325399191,8cyclictest0-21swapper/112:37:101
325399191,17cyclictest0-21swapper/112:33:241
324799198,10cyclictest0-21swapper/009:05:010
324799195,3cyclictest24331-21munin-run09:10:010
324799195,3cyclictest0-21swapper/009:54:140
324799193,9cyclictest792-21avahi-daemon10:33:330
324799191,9cyclictest0-21swapper/010:12:280
3247991913,3cyclictest24500-21grep07:55:160
3247991912,4cyclictest7981-21latency_hist11:00:010
324799191,11cyclictest0-21swapper/012:15:140
324799191,10cyclictest8532-21/usr/sbin/munin07:20:170
324799191,10cyclictest2476-21expr12:00:120
324799191,10cyclictest0-21swapper/010:53:040
324799191,10cyclictest0-21swapper/010:53:040
267452191,3sleep20-21swapper/209:15:082
54382181,2sleep20-21swapper/210:50:372
54382181,2sleep20-21swapper/210:50:372
326499181,14cyclictest19521-21/usr/sbin/munin07:45:273
326499180,2cyclictest86750irq/128-enp1s0-08:50:333
326099186,5cyclictest24328-21diskmemload11:07:282
326099186,3cyclictest86850irq/129-enp1s0-09:45:292
326099185,3cyclictest86850irq/129-enp1s0-12:20:152
326099184,4cyclictest15817-21grep11:15:222
326099183,7cyclictest31153-21/usr/sbin/munin09:25:262
326099181,8cyclictest0-21swapper/210:58:302
326099181,7cyclictest4183-21needreboot09:35:202
326099181,7cyclictest0-21swapper/212:14:372
326099181,7cyclictest0-21swapper/211:21:392
326099181,7cyclictest0-21swapper/209:30:252
326099181,7cyclictest0-21swapper/209:00:152
326099181,6cyclictest0-21swapper/211:40:032
3260991813,3cyclictest14995-21irqstats10:00:192
324799186,3cyclictest0-21swapper/010:01:540
324799184,7cyclictest16778-21kworker/u8:011:34:370
324799184,4cyclictest0-21swapper/011:49:120
324799183,14cyclictest0-21swapper/010:16:160
324799183,12cyclictest30525-21/usr/sbin/munin08:10:190
324799182,3cyclictest17725-21munin-run08:55:010
324799181,9cyclictest0-21swapper/012:13:400
326099176,3cyclictest12050irq/123-ahci[0008:20:022
326099174,4cyclictest30793-21taskset11:50:172
326099171,7cyclictest28974-21users10:30:262
326099171,7cyclictest0-21swapper/212:06:472
326099171,6cyclictest13951-21latency_hist10:00:022
326099171,5cyclictest0-21swapper/209:50:232
326099171,5cyclictest0-21swapper/208:41:282
3260991713,3cyclictest0-21swapper/207:35:402
3260991713,2cyclictest23613-21sensors_temp09:05:272
3260991712,3cyclictest8298-21latency_hist07:20:022
3260991712,3cyclictest32403-21/usr/sbin/munin11:55:252
3260991712,3cyclictest2879-21smartctl10:45:242
3260991712,3cyclictest23070-21ls10:20:122
3260991712,3cyclictest19564-21sh11:25:162
3260991712,3cyclictest17853-21unixbench-2d10:05:282
3260991712,3cyclictest15885-21grep08:50:132
326099171,14cyclictest18941-21strings08:55:232
3260991711,3cyclictest836-21runrttasks07:25:052
3260991711,3cyclictest21830-21grep07:50:102
3260991711,3cyclictest17915-21grep07:40:172
326099171,10cyclictest0-21swapper/210:39:102
324799174,3cyclictest0-21swapper/010:35:160
324799171,9cyclictest0-21swapper/012:22:450
3247991712,3cyclictest9781-21gltestperf08:35:150
3247991712,3cyclictest29766-21irqstats09:20:180
3247991712,3cyclictest14028-21date10:00:010
324799171,15cyclictest0-21swapper/008:55:370
324799171,14cyclictest0-21swapper/007:10:020
3247991711,3cyclictest5028-21expr08:25:110
3247991710,4cyclictest0-21swapper/010:20:270
317672171,2sleep00-21swapper/010:40:030
300042171,2sleep00-21swapper/008:06:280
51332161,2sleep20-21swapper/207:11:352
326099165,3cyclictest86850irq/129-enp1s0-08:10:222
326099162,5cyclictest25886-21munin-run08:00:012
326099161,6cyclictest19069-21timerwakeupswit12:35:242
326099161,2cyclictest0-21swapper/207:31:492
3260991612,3cyclictest0-21swapper/208:05:412
3260991611,3cyclictest9571-21ntp_kernel_pll_07:20:202
3260991611,3cyclictest9518-21cut12:15:162
3260991611,3cyclictest18950-21entropy10:10:162
3260991610,4cyclictest0-21swapper/207:45:192
3260991610,3cyclictest14252-21idleruntime08:45:182
3260991610,3cyclictest14252-21idleruntime08:45:182
324799161,6cyclictest18500-21grep07:40:250
324799161,2cyclictest0-21swapper/011:20:160
324799161,2cyclictest0-21swapper/008:45:070
324799161,2cyclictest0-21swapper/008:45:070
324799161,14cyclictest0-21swapper/007:46:340
3247991610,3cyclictest836-21runrttasks07:15:040
3247991610,3cyclictest13616-21hostname07:30:150
3260991510,3cyclictest9683-21fschecks_time08:35:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional