You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-28 - 00:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Aug 27, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
174592234162,23sleep20-21swapper/207:08:482
172922232156,24sleep00-21swapper/007:06:370
172162203166,24sleep10-21swapper/107:05:381
172082201163,25sleep30-21swapper/307:05:323
695021740,4sleep31765499cyclictest11:35:253
17649997061,3cyclictest0-21swapper/211:37:362
323392580,2sleep00-21swapper/012:35:130
189352560,1sleep20-21swapper/209:35:242
324732550,5sleep01763899cyclictest08:55:130
1763899421,37cyclictest25279-21pmu-power09:50:230
17649994138,2cyclictest0-21swapper/207:46:422
1763899401,23cyclictest27318-21df_inode11:10:140
1765499390,28cyclictest0-21swapper/311:40:113
1763899391,32cyclictest6938-21/usr/sbin/munin11:35:240
1763899391,27cyclictest12191-21/usr/sbin/munin11:50:220
1763899381,25cyclictest12709-21ls10:35:230
1763899381,23cyclictest1155-21gltestperf10:10:170
1763899381,23cyclictest1155-21gltestperf10:10:170
1763899373,22cyclictest27932-21/usr/sbin/munin08:45:250
1763899371,20cyclictest87350irq/126-enp1s0-11:40:250
1765499350,26cyclictest0-21swapper/310:17:433
1763899358,13cyclictest20776-21awk12:10:000
1763899353,23cyclictest28896-21dnf10:00:140
1763899351,28cyclictest87350irq/126-enp1s0-12:04:360
1763899351,23cyclictest22931-21/usr/sbin/munin07:20:180
1763899351,22cyclictest4371-21/usr/sbin/munin09:05:170
1763899351,22cyclictest23297-21cpuspeed_turbos12:15:120
1763899351,22cyclictest17555-21/usr/sbin/munin09:35:200
1765499340,26cyclictest0-21swapper/310:08:353
1765499340,11cyclictest0-21swapper/307:49:013
1763899341,24cyclictest0-21swapper/009:55:040
1763899341,22cyclictest10281-21cat08:05:020
1763899341,18cyclictest18177-21/usr/sbin/munin10:50:190
1765499331,21cyclictest16883-21memory10:45:193
1765499331,10cyclictest0-21swapper/312:11:313
1763899336,17cyclictest87550irq/128-enp1s0-12:10:150
1763899331,28cyclictest24733-21sort11:05:010
1763899331,23cyclictest32736-21basename10:10:010
17638993312,18cyclictest20987-21grep10:55:150
1763899331,20cyclictest4736-21/usr/sbin/munin11:30:240
1763899331,20cyclictest0-21swapper/010:25:190
1763899331,19cyclictest7454-21grep09:10:190
1763899331,19cyclictest14998-21gltestperf11:55:160
1763899331,18cyclictest22095-21expr09:45:140
1763899330,25cyclictest0-21swapper/008:30:020
1765499321,23cyclictest0-21swapper/310:27:353
1763899322,26cyclictest27320-40dnf07:30:090
1763899322,20cyclictest1490950kworker/u8:009:42:120
1763899322,13cyclictest87350irq/126-enp1s0-11:10:010
1763899321,28cyclictest27583-21/usr/sbin/munin12:25:120
1763899321,21cyclictest17016-21sshd09:31:100
1763899321,21cyclictest1535-21/usr/sbin/munin07:45:130
1763899321,20cyclictest17557-21users10:45:270
1763899321,19cyclictest6676-21if_enp1s007:55:170
1763899321,19cyclictest6676-21if_enp1s007:55:170
1763899321,19cyclictest12541-21sshd08:10:090
293242310,3sleep31765499cyclictest08:45:243
1765499311,23cyclictest0-21swapper/311:11:463
1765499310,23cyclictest0-21swapper/311:34:193
17644993110,18cyclictest15357-21users10:40:231
1763899313,18cyclictest9544-21if_err_enp1s009:15:160
1763899311,27cyclictest2076-21/usr/sbin/munin11:25:180
1763899311,21cyclictest0-21swapper/007:40:230
17638993110,18cyclictest6979-21wc10:25:010
1765499301,8cyclictest0-21swapper/308:26:193
1765499301,23cyclictest0-21swapper/307:53:353
1765499301,12cyclictest0-21swapper/307:20:273
1765499301,11cyclictest0-21swapper/312:29:193
1763899302,25cyclictest9380-21/usr/sbin/munin10:30:350
1763899301,26cyclictest3754-21/usr/sbin/munin07:50:230
1763899301,26cyclictest21345-21/usr/sbin/munin08:30:140
1763899301,25cyclictest19821-21idleruntime-cro07:15:010
1763899301,19cyclictest827-21runrttasks10:40:090
1763899301,16cyclictest30591-21irqstats12:30:180
1763899300,15cyclictest0-21swapper/011:45:180
1765499291,12cyclictest0-21swapper/309:45:353
1765499291,11cyclictest0-21swapper/312:15:153
1765499290,23cyclictest0-21swapper/309:41:553
1765499290,22cyclictest0-21swapper/312:37:033
1765499290,22cyclictest0-21swapper/309:50:113
1765499290,13cyclictest0-21swapper/312:21:033
1765499290,12cyclictest0-21swapper/311:50:313
1765499290,12cyclictest0-21swapper/311:06:023
1765499290,12cyclictest0-21swapper/308:14:593
1764499294,18cyclictest16927-21df_inode12:00:141
1763899292,24cyclictest11624-21/usr/sbin/munin08:05:230
1763899292,23cyclictest26998-21smart_sda08:40:240
1763899291,25cyclictest25390-21/usr/sbin/munin12:20:220
1763899291,25cyclictest25126-21/usr/sbin/munin07:25:140
1763899291,25cyclictest20739-21/usr/sbin/munin07:15:220
1763899291,25cyclictest14747-21/usr/sbin/munin08:15:270
1763899291,25cyclictest14160-21grep09:25:200
1763899291,25cyclictest10964-21/usr/sbin/munin09:21:230
1763899291,24cyclictest30615-21fschecks_time08:50:150
17638992911,16cyclictest3126-21df10:15:140
1765499281,9cyclictest0-21swapper/308:55:473
1765499281,12cyclictest0-21swapper/310:31:473
1765499281,11cyclictest15356-21users10:40:233
1765499281,11cyclictest0-21swapper/307:43:593
1765499281,10cyclictest16271-21memory09:30:193
1765499281,10cyclictest0-21swapper/308:39:303
1765499280,22cyclictest0-21swapper/307:25:453
1765499280,18cyclictest0-21swapper/310:54:193
1765499280,17cyclictest0-21swapper/309:22:353
1765499280,13cyclictest0-21swapper/308:30:153
17644992812,4cyclictest31941-21fschecks_time11:20:161
17644992812,4cyclictest31941-21fschecks_time11:20:161
1764499281,22cyclictest87650irq/129-enp1s0-09:30:121
17644992810,14cyclictest15129-21munin-run09:30:011
1763899281,24cyclictest31366-21/usr/sbin/munin11:20:160
1763899281,24cyclictest31366-21/usr/sbin/munin11:20:160
1763899281,24cyclictest29953-21/usr/sbin/munin11:15:180
1763899281,24cyclictest29519-21/usr/sbin/munin07:35:140
1763899281,24cyclictest24244-21/usr/sbin/munin08:35:150
1763899281,24cyclictest17299-21/usr/sbin/munin08:20:140
1763899281,23cyclictest2319-21cpuspeed_turbos09:00:120
1765499271,11cyclictest0-21swapper/311:46:313
1765499271,10cyclictest0-21swapper/309:14:113
1765499270,18cyclictest0-21swapper/309:36:433
1765499270,13cyclictest0-21swapper/309:56:113
1765499270,11cyclictest0-21swapper/308:22:553
1765499270,10cyclictest0-21swapper/311:58:473
1765499270,10cyclictest0-21swapper/307:19:313
17649992716,6cyclictest810-21sshd07:30:122
1764499278,15cyclictest5945-21diskstats11:35:141
1764499274,19cyclictest8356-21ps10:25:231
1765499261,12cyclictest0-21swapper/311:19:263
1765499261,10cyclictest0-21swapper/310:38:513
17654992610,3cyclictest0-21swapper/310:04:193
1765499260,9cyclictest0-21swapper/309:26:393
1765499260,8cyclictest0-21swapper/310:12:553
1765499260,8cyclictest0-21swapper/310:12:553
1765499260,11cyclictest0-21swapper/309:08:393
1765499260,10cyclictest0-21swapper/311:25:433
1764999263,8cyclictest0-21swapper/210:31:252
1764999261,3cyclictest0-21swapper/211:05:012
1764499268,15cyclictest16705-21latency_hist08:20:021
1764499264,19cyclictest6117-21grep10:20:231
1764499261,17cyclictest5038-21sshd11:30:551
1765499251,8cyclictest0-21swapper/309:19:073
1765499251,10cyclictest0-21swapper/307:34:393
1765499251,10cyclictest0-21swapper/307:11:223
17654992510,3cyclictest0-21swapper/308:41:463
1765499250,9cyclictest0-21swapper/308:50:463
1765499250,8cyclictest0-21swapper/310:20:193
1765499250,11cyclictest0-21swapper/310:57:313
1765499250,10cyclictest0-21swapper/312:05:073
1765499250,10cyclictest0-21swapper/308:02:073
1765499250,10cyclictest0-21swapper/307:36:193
1764999258,3cyclictest0-21swapper/211:48:252
1764499258,14cyclictest12323-21awk08:10:011
1764499257,4cyclictest12665-21tr08:10:101
1764499255,17cyclictest9768-21munin-run11:45:011
1764499252,16cyclictest2013-21irqstats11:25:181
1764499251,16cyclictest21183-21irqstats10:55:171
1764499251,15cyclictest26697-21diskstats09:55:131
1764499251,11cyclictest12839-21chrt09:23:591
17644992510,8cyclictest8564-1kworker/u9:110:35:571
1764499250,22cyclictest21958-21/usr/sbin/munin09:45:251
1765499240,11cyclictest0-21swapper/308:09:393
1765499240,10cyclictest0-21swapper/312:02:273
1764499247,14cyclictest27675-21diskstats07:30:141
1764499245,16cyclictest32628-21runrttasks10:08:581
1764499244,17cyclictest7399-21kernelversion09:10:201
1764499243,15cyclictest1295-21if_enp1s010:10:181
1764499243,15cyclictest1295-21if_enp1s010:10:171
17644992412,9cyclictest3518-21latency_hist07:50:011
1764499241,20cyclictest23271-21df_inode07:20:131
1764499241,17cyclictest87650irq/129-enp1s0-10:16:131
1765499237,2cyclictest7169-21rt-features07:55:233
1765499237,2cyclictest7169-21rt-features07:55:233
1765499230,9cyclictest0-21swapper/308:17:233
1765499230,8cyclictest0-21swapper/312:34:313
1765499230,11cyclictest0-21swapper/311:01:033
1764999231,6cyclictest0-21swapper/210:48:112
1764999231,5cyclictest32-21ksoftirqd/209:41:122
1764999231,3cyclictest0-21swapper/212:25:242
1764499235,15cyclictest19309-21grep08:25:121
1764499235,15cyclictest14983-21grep11:55:171
1764499233,17cyclictest17124-21cstates08:20:121
1764499233,17cyclictest1035-21awk08:55:221
17644992318,3cyclictest7089-21pmu-power07:55:231
17644992318,3cyclictest7089-21pmu-power07:55:231
1764499231,20cyclictest0-21swapper/109:05:181
1764499231,13cyclictest27219-21expr11:10:131
1764499230,16cyclictest0-21swapper/112:08:341
1765499227,3cyclictest0-21swapper/309:04:493
1765499221,11cyclictest0-21swapper/311:21:273
1765499221,11cyclictest0-21swapper/311:21:263
1764999221,5cyclictest11628-21kworker/u8:112:10:282
1764499225,14cyclictest19818-21munin-run07:15:011
1764499223,16cyclictest31483-21turbostat07:40:011
1764499222,3cyclictest10303-21memory10:30:271
1764499222,17cyclictest31291-21proc_pri08:50:221
17644992213,8cyclictest0-21swapper/110:45:151
1764499220,19cyclictest0-21swapper/109:55:001
321642210,4sleep21764999cyclictest12:35:112
1764999211,5cyclictest11926-21diskstats10:35:132
1764999211,5cyclictest0-21swapper/210:04:392
1764999211,3cyclictest0-21swapper/211:06:262
1764999211,2cyclictest0-21swapper/209:46:512
1764499219,2cyclictest6339-21diskmemload12:36:011
1764499217,7cyclictest0-21swapper/111:17:171
1764499213,15cyclictest1300-21latency_hist07:45:011
1764499212,16cyclictest22485-21unixbench_multi12:10:241
1764499211,3cyclictest25993-21expr08:40:131
17644992113,5cyclictest31292-21unixbench_multi12:30:261
1764499211,17cyclictest25992-21gltestperf12:20:161
17644992111,3cyclictest27087-21latency_hist07:30:001
1764499211,12cyclictest0-21swapper/109:00:111
1764499210,6cyclictest0-21swapper/111:00:071
1764999209,3cyclictest26332-21missed_timers12:20:202
1764999206,5cyclictest16009-21smart_sda08:15:242
1764999201,6cyclictest3851-21ntp_states10:15:212
17649992010,2cyclictest5731-21awk07:55:012
1764499209,2cyclictest6339-21diskmemload10:04:281
1764499208,9cyclictest25101-21unixbench_singl08:35:251
1764499206,4cyclictest20739-21/usr/sbin/munin07:15:131
1764499205,13cyclictest0-21swapper/109:15:191
1764499203,10cyclictest25691-21memory11:05:181
1764499202,6cyclictest27890-21cstates12:25:121
17644992017,2cyclictest0-21swapper/107:50:131
1764499201,2cyclictest0-21swapper/111:50:131
1764499200,13cyclictest20100-21df09:40:121
1764999198,3cyclictest87350irq/126-enp1s0-08:30:232
1764999193,2cyclictest87650irq/129-enp1s0-09:10:012
1764999192,8cyclictest2213350kworker/u8:011:14:032
1764999191,9cyclictest8215-21fschecks_count11:40:152
1764999191,7cyclictest13885-21cpuspeed_turbos10:40:092
1764999191,6cyclictest1657-21ntp_states10:10:222
1764999191,6cyclictest1657-21ntp_states10:10:222
1764499195,12cyclictest0-21swapper/109:35:031
1764499191,3cyclictest0-21swapper/111:46:081
1764499191,16cyclictest0-21swapper/110:52:151
1764499191,16cyclictest0-21swapper/108:46:081
1764499191,13cyclictest0-21swapper/112:19:041
275102181,2sleep20-21swapper/208:42:402
1764999182,7cyclictest30116-21df12:30:142
1764999182,6cyclictest21286-21memory10:55:182
1764999181,8cyclictest0-21swapper/211:25:372
1764999181,7cyclictest9146-21latency_hist10:30:012
1764999181,7cyclictest27103-21awk07:30:012
1764999181,7cyclictest0-21swapper/210:05:342
1764999181,6cyclictest0-21swapper/209:18:112
1764999181,6cyclictest0-21swapper/207:56:562
1764999181,6cyclictest0-21swapper/207:56:562
1764999181,15cyclictest0-21swapper/208:35:292
17649991811,4cyclictest24757-21grep12:15:262
1764999181,10cyclictest0-21swapper/210:53:422
1764999181,10cyclictest0-21swapper/207:15:212
1764499182,13cyclictest8955-21/usr/sbin/munin08:00:181
17644991814,3cyclictest0-21swapper/108:30:121
295812171,2sleep20-21swapper/208:46:052
1764999176,3cyclictest87650irq/129-enp1s0-11:15:212
1764999175,4cyclictest6339-21diskmemload10:20:322
1764999173,6cyclictest0-21swapper/211:55:392
1764999171,7cyclictest4150-21munin-run09:05:012
1764999171,7cyclictest0-21swapper/209:53:352
1764999171,6cyclictest23033-21grep07:20:122
1764999171,6cyclictest12632-21users09:21:282
1764999171,6cyclictest0-21swapper/211:30:182
1764999171,6cyclictest0-21swapper/209:56:262
1764999171,6cyclictest0-21swapper/209:29:002
1764999171,5cyclictest8564-1kworker/u9:112:02:022
1764999171,5cyclictest0-21swapper/211:22:102
1764999171,5cyclictest0-21swapper/211:22:102
1764999171,3cyclictest0-21swapper/207:11:552
1764999171,15cyclictest0-21swapper/207:40:192
17649991711,3cyclictest15365-21acpi09:30:122
1764999162,6cyclictest30128-21/usr/sbin/munin08:50:142
1764999161,9cyclictest0-21swapper/209:14:062
1764999161,9cyclictest0-21swapper/207:38:042
1764999161,7cyclictest0-21swapper/212:06:302
1764999161,6cyclictest13260-21ntp_states11:50:222
17649991612,2cyclictest19798-21gltestperf08:25:162
17649991611,3cyclictest9293-21tr08:00:212
17649991611,3cyclictest18550-21vmstat08:20:252
17649991611,3cyclictest13060-21cut08:10:122
17649991610,3cyclictest399-21fschecks_time08:55:162
17649991510,3cyclictest10357-21acpi08:05:102
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional