You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 15:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Thu Apr 16, 2026 12:43:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2184741995835,20cyclictest2369587-21munin-run12:20:011
2184741995530,5cyclictest2211709-21munin-plugin-st08:40:011
218474199542,29cyclictest2221912-21sh09:10:371
2184741995327,5cyclictest0-21swapper/111:55:371
2184741995327,5cyclictest0-21swapper/111:55:371
2184741995323,6cyclictest1368112-21cups-browsed09:50:421
2184742995143,4cyclictest0-21swapper/210:45:392
2184741995130,5cyclictest2303503-21grep10:50:491
2184740995141,7cyclictest1368112-21cups-browsed10:20:490
2184742994940,5cyclictest2339815-21sendmail11:40:012
2184741994941,5cyclictest2339803-21awk11:40:011
2184741994923,4cyclictest1051-21mta-sts-daemon08:15:391
218474299484,31cyclictest2348049-21ssh11:50:432
2184742994841,4cyclictest0-21swapper/212:15:372
2184741994825,4cyclictest2245580-21aten_r2power_cu09:40:261
218474199482,23cyclictest2330252-21sshd11:25:381
2184742994534,5cyclictest0-21swapper/211:45:412
2184741994534,8cyclictest1368112-21cups-browsed10:15:391
2184741994534,8cyclictest1368112-21cups-browsed10:15:391
2184740994535,8cyclictest2204194-21kworker/u8:0-nfsiod08:36:020
2184742994437,2cyclictest0-21swapper/208:36:042
218474299442,33cyclictest2295088-21ssh10:40:432
2184740994434,7cyclictest2286948-21ssh10:30:400
2184742994335,5cyclictest421-21dbus-daemon11:05:022
2184742994335,5cyclictest421-21dbus-daemon11:05:022
2184742994332,7cyclictest261-21systemd-journal12:33:372
2184741994334,5cyclictest2213254-21systemd-run08:45:021
2184740994336,5cyclictest1-21systemd09:35:410
2184740994234,3cyclictest0-21swapper/011:50:360
2184740994233,6cyclictest2297770-21cron10:45:010
2184742994133,5cyclictest2357808-21cron12:05:002
2184741994134,4cyclictest2313802-21cron11:05:011
2184741994134,4cyclictest2313802-21cron11:05:011
2184741994131,7cyclictest2350357-21sh11:55:011
2184741994130,8cyclictest2187725-21idleruntime-cro07:20:001
2184741994128,9cyclictest2271112-21timerandwakeup10:10:481
2184740994135,3cyclictest0-21swapper/009:00:010
2184740994129,4cyclictest0-21swapper/009:20:400
2184742994034,3cyclictest0-21swapper/212:35:342
2184742994032,5cyclictest2309833-21systemctl11:00:012
2184742994031,6cyclictest624-21lldpd12:05:432
2184742994030,7cyclictest548-21gdbus11:25:472
2184741994034,3cyclictest0-21swapper/109:15:371
2184741994032,5cyclictest2217708-21munin-run09:00:011
2184741994031,5cyclictest2301773-21munin-run10:50:011
2184741994028,8cyclictest2374478-21kernelversion12:25:311
2184740994033,4cyclictest2319092-21proc_pri11:10:440
218474299396,25cyclictest2226100-21ssh09:15:402
2184742993931,4cyclictest0-21swapper/209:20:382
2184742993931,3cyclictest0-21swapper/211:05:392
218474299391,32cyclictest2286843-21ssh10:30:362
2184741993931,5cyclictest2343254-21cron11:45:011
2184740993932,4cyclictest2321629-21kworker/u8:14+xprtiod12:25:010
2184740993927,3cyclictest0-21swapper/010:30:020
2184742993832,3cyclictest0-21swapper/210:05:012
2184742993830,5cyclictest2323308-21unixbench-2d11:15:492
2184742993830,3cyclictest0-21swapper/209:55:392
2184741993832,3cyclictest0-21swapper/112:15:011
2184741993830,3cyclictest0-21swapper/112:30:371
2184741993829,6cyclictest1-21systemd10:40:011
2184740993828,7cyclictest12350irq/32-eth007:15:390
2184740993825,10cyclictest2226295-21mailstats09:15:460
2184740993819,16cyclictest2350852-21aten_r2power_en11:55:270
2184740993819,16cyclictest2350852-21aten_r2power_en11:55:270
2184742993732,2cyclictest0-21swapper/208:05:012
2184742993730,3cyclictest0-21swapper/211:30:362
2184742993729,5cyclictest1054-21mta-sts-daemon10:50:372
2184742993725,8cyclictest1051-21mta-sts-daemon09:05:292
218474299372,26cyclictest2242363-21ssh09:35:452
2184741993725,9cyclictest2261327-21munin-run10:00:011
218474099373,30cyclictest2222008-21ssh09:10:400
2184740993726,8cyclictest2306860-21missed_timers10:55:350
2184742993629,3cyclictest2300561-21kworker/u8:6-xprtiod11:45:012
2184742993627,6cyclictest1051-21mta-sts-daemon10:05:502
2184742993627,5cyclictest2321742-21latency_hist11:15:022
218474299362,27cyclictest2234184-21ssh09:25:372
2184741993628,5cyclictest2354846-21if_eth012:00:311
2184741993628,5cyclictest2191251-21fschecks_count07:30:291
2184741993626,5cyclictest0-21swapper/109:05:411
2184741993625,8cyclictest1-21systemd12:24:371
218474099363,30cyclictest2366854-21ssh12:15:380
2184740993629,3cyclictest12350irq/32-eth010:35:380
2184740993624,9cyclictest2311450-21timerandwakeup11:00:490
2184740993624,9cyclictest2311450-21timerandwakeup11:00:490
2184740993624,9cyclictest2219107-21unixbench_multi09:00:450
2184742993527,5cyclictest2216135-21head08:50:482
2184742993525,5cyclictest2371397-21unixbench_multi12:20:502
218474299352,25cyclictest2238329-21sh09:30:452
218474299352,25cyclictest2238329-21sh09:30:452
2184741993528,3cyclictest0-21swapper/109:20:381
2184741993526,6cyclictest2282646-21missed_timers10:25:331
2184740993528,5cyclictest1-21systemd10:20:010
2184740993528,5cyclictest1-21systemd10:20:010
2184740993528,4cyclictest2327150-21unixbench-2d11:20:560
2184740993526,6cyclictest2199741-21sh08:00:010
218474099352,30cyclictest2246284-21ssh09:40:370
2184742993427,4cyclictest1054-21mta-sts-daemon11:57:212
2184742993427,4cyclictest1054-21mta-sts-daemon11:57:212
2184742993426,5cyclictest2190733-21munin-plugin-st07:30:002
2184742993425,6cyclictest2377768-21latency_hist12:30:032
2184742993425,6cyclictest1368112-21cups-browsed10:23:482
2184742993423,8cyclictest2362521-21if_eth012:10:302
2184741993426,5cyclictest2197117-21aten_r2power_vo07:50:271
2184741993425,5cyclictest2306929-21needreboot10:55:361
2184740993426,6cyclictest516-21in:imuxsock08:13:290
218474099342,13cyclictest2238284-21ntpq09:30:430
218474099342,13cyclictest2238284-21ntpq09:30:430
2184742993326,3cyclictest0-21swapper/208:45:022
218474299332,26cyclictest2246490-21ssh09:40:442
218474199333,25cyclictest2262475-21ssh10:00:341
2184741993328,2cyclictest0-21swapper/108:00:011
2184741993321,9cyclictest2278565-21latency10:20:311
218474099339,18cyclictest2250386-21ssh09:45:380
218474099335,19cyclictest2337532-21ssh11:35:440
2184740993327,4cyclictest2261314-21ssh10:00:010
2184740993324,6cyclictest151ktimers/009:05:370
2184740993318,3cyclictest0-21swapper/010:05:340
218474099331,28cyclictest2362834-21ssh12:10:350
218474099331,23cyclictest2355154-21ssh12:00:370
2184742993225,4cyclictest2198204-21cron07:55:012
2184742993224,5cyclictest2273549-21sh10:15:022
2184742993224,5cyclictest2254265-21needreboot09:50:322
2184742993223,5cyclictest0-21swapper/211:25:022
2184742993219,9cyclictest0-21swapper/210:25:372
2184741993223,5cyclictest2241158-21latency_hist09:35:031
2184741993223,5cyclictest2241158-21latency_hist09:35:031
2184741993222,8cyclictest2322794-21ntp_kernel_pll_11:15:401
218474099325,23cyclictest2322816-21ssh11:15:390
2184740993227,3cyclictest2344255-21missed_timers11:45:340
2184740993226,3cyclictest0-21swapper/008:20:010
2184740993225,4cyclictest2336255-21cut11:35:010
2184740993225,4cyclictest2188011-21aten_r2power_cu07:20:250
2184740993225,4cyclictest2188011-21aten_r2power_cu07:20:250
2184740993212,4cyclictest0-21swapper/010:50:010
218474299312,5cyclictest171rcu_preempt08:10:402
2184742993123,3cyclictest0-21swapper/210:15:422
2184742993123,3cyclictest0-21swapper/210:15:422
218474299312,23cyclictest2250333-21ssh09:45:362
2184741993126,2cyclictest0-21swapper/111:50:021
2184741993124,4cyclictest1055-21mta-sts-daemon07:25:391
2184741993124,3cyclictest0-21swapper/111:20:561
2184741993123,5cyclictest2193726-21idleruntime-cro07:40:011
2184741993123,3cyclictest0-21swapper/111:30:401
218474199312,23cyclictest2314936-21ssh11:05:361
2184741993121,7cyclictest695-21cron08:05:011
2184740993124,5cyclictest0-21swapper/011:10:020
2184740993124,4cyclictest2257294-21ssh09:55:010
2184740993123,5cyclictest439-21systemd-logind09:30:010
2184740993123,5cyclictest1368112-21cups-browsed12:25:280
2184740993121,2cyclictest0-21swapper/010:15:010
2184740993119,9cyclictest0-21swapper/011:40:280
218474099311,24cyclictest2330282-21ssh11:25:390
2184742993024,4cyclictest2184739-21cyclictest08:35:002
2184742993023,3cyclictest0-21swapper/209:10:362
2184742993023,3cyclictest0-21swapper/207:20:002
2184742993020,7cyclictest2218569-21ntp_kernel_err09:00:332
218474199308,15cyclictest1475125-21munin-node08:45:431
2184741993025,3cyclictest2219212-21kworker/u8:0-xprtiod09:35:291
2184741993023,4cyclictest2176781-21kworker/u8:9-xprtiod07:25:011
2184741993023,4cyclictest2176781-21kworker/u8:9-xprtiod07:25:011
2184741993023,3cyclictest0-21swapper/111:10:271
2184741993022,5cyclictest2286452-21ssh10:30:311
2184741993020,6cyclictest2234162-21ntp_kernel_pll_09:25:361
2184740993023,4cyclictest2381682-21latency_hist12:35:020
2184740993023,3cyclictest0-21swapper/007:40:010
2184740993013,11cyclictest2358982-21rm12:05:370
2184742992921,5cyclictest2290183-21/usr/sbin/munin10:35:322
2184742992921,5cyclictest2189261-21systemd-run07:25:012
2184742992921,5cyclictest2189261-21systemd-run07:25:012
2184742992919,7cyclictest2207927-21latency08:25:332
2184742992919,7cyclictest2207927-21latency08:25:332
218474199293,20cyclictest2382730-21ssh12:35:381
2184741992923,3cyclictest0-21swapper/112:10:011
2184741992922,4cyclictest0-21swapper/109:00:391
218474199292,21cyclictest2294990-21ssh10:40:391
2184741992921,4cyclictest1056-21mta-sts-daemon09:50:011
2184740992920,5cyclictest0-21swapper/010:00:260
2184740992915,11cyclictest421-21dbus-daemon08:30:440
2184740992914,12cyclictest12450irq/33-eth012:35:370
218474299285,12cyclictest1475125-21munin-node08:55:352
2184742992821,3cyclictest0-21swapper/208:45:392
2184742992820,5cyclictest439-21systemd-logind08:20:012
2184741992819,7cyclictest2265989-21sh10:05:281
2184740992820,5cyclictest1368112-21cups-browsed07:42:380
2184740992817,8cyclictest2302862-21needreboot10:50:350
2184740992815,11cyclictest547-21gmain08:20:430
2184740992815,11cyclictest547-21gmain08:20:430
2184742992721,3cyclictest0-21swapper/207:10:012
2184742992720,4cyclictest2186256-21latency_hist07:15:012
2184742992720,4cyclictest1051-21mta-sts-daemon08:20:162
2184742992720,4cyclictest1051-21mta-sts-daemon08:20:162
2184742992719,4cyclictest0-21swapper/207:55:362
2184741992720,4cyclictest2201741-21fschecks_count08:05:311
2184741992720,4cyclictest1054-21mta-sts-daemon07:46:511
2184741992719,5cyclictest2216097-21tail08:50:481
218474099272,20cyclictest1017-21sendmail-mta08:45:390
2184742992618,5cyclictest2202612-21unixbench_multi08:05:492
2184742992618,5cyclictest2191257-21fschecks_time07:30:292
218474199263,17cyclictest2203216-21sshd08:10:291
2184740992619,4cyclictest2213215-21cron08:45:010
218474299254,11cyclictest428-21mta-sts-daemon07:35:402
2184740992516,6cyclictest2192214-21cron07:35:000
2184740992515,7cyclictest0-21swapper/007:50:270
2184741992419,3cyclictest1054-21mta-sts-daemon08:29:371
2184741992419,3cyclictest1054-21mta-sts-daemon08:29:371
2184741992418,3cyclictest0-21swapper/108:30:411
2184740992418,3cyclictest2202607-21tail08:05:490
2184742992317,3cyclictest1054-21mta-sts-daemon07:45:432
2184741992317,4cyclictest0-21swapper/107:10:351
218474099232,13cyclictest1054-21mta-sts-daemon07:10:400
2184740992318,3cyclictest0-21swapper/008:50:290
2184740992316,5cyclictest548-21gdbus07:29:020
218474299223,15cyclictest2194257-21/usr/sbin/munin07:40:292
2184741992214,5cyclictest0-21swapper/108:25:001
2184741992214,5cyclictest0-21swapper/108:25:001
2184741992213,6cyclictest2193983-21aten_r2power_cu07:40:251
2184740992216,4cyclictest2201110-21unixbench_multi08:00:470
2184740992211,8cyclictest421-21dbus-daemon07:50:010
218474399215,8cyclictest0-21swapper/312:15:373
218474399214,11cyclictest0-21swapper/311:45:453
218474399213,11cyclictest0-21swapper/310:40:423
218474399213,10cyclictest0-21swapper/310:30:373
2184740992116,3cyclictest2207747-21fschecks_count08:25:310
2184740992116,3cyclictest2207747-21fschecks_count08:25:310
218474399202,12cyclictest0-21swapper/311:20:433
218474399202,11cyclictest0-21swapper/309:40:393
218474399193,9cyclictest0-21swapper/309:20:373
218474399193,10cyclictest0-21swapper/312:25:373
218474399193,10cyclictest0-21swapper/310:55:513
218474399193,10cyclictest0-21swapper/309:55:363
218474399192,9cyclictest0-21swapper/312:00:363
218474399192,13cyclictest0-21swapper/310:25:373
218474399191,9cyclictest0-21swapper/310:35:403
218474399191,11cyclictest0-21swapper/312:05:373
218474399183,9cyclictest0-21swapper/310:50:283
218474399183,9cyclictest0-21swapper/309:50:413
218474399183,8cyclictest0-21swapper/309:30:413
218474399183,8cyclictest0-21swapper/309:30:413
218474399183,8cyclictest0-21swapper/309:05:393
218474399183,7cyclictest0-21swapper/311:30:363
218474399183,10cyclictest0-21swapper/311:00:363
218474399183,10cyclictest0-21swapper/311:00:363
218474399182,9cyclictest0-21swapper/311:55:343
218474399182,9cyclictest0-21swapper/311:55:343
218474399182,9cyclictest0-21swapper/311:50:363
218474399182,10cyclictest0-21swapper/311:25:383
218474399182,10cyclictest0-21swapper/311:05:403
218474399182,10cyclictest0-21swapper/310:20:383
218474399182,10cyclictest0-21swapper/310:15:433
218474399182,10cyclictest0-21swapper/310:15:433
218474399182,10cyclictest0-21swapper/310:10:393
218474399182,10cyclictest0-21swapper/310:00:373
218474399182,10cyclictest0-21swapper/309:25:453
218474399182,10cyclictest0-21swapper/309:10:443
218474399182,10cyclictest0-21swapper/308:36:033
2184741991812,3cyclictest0-21swapper/107:10:011
218474399174,7cyclictest0-21swapper/311:40:383
218474399173,9cyclictest0-21swapper/311:15:443
218474399173,9cyclictest0-21swapper/310:05:383
218474399173,9cyclictest0-21swapper/309:35:383
218474399173,7cyclictest0-21swapper/312:10:343
218474399173,10cyclictest0-21swapper/309:15:313
218474399172,9cyclictest0-21swapper/312:30:353
218474399172,9cyclictest0-21swapper/307:55:423
218474399172,8cyclictest0-21swapper/312:20:373
218474399172,7cyclictest0-21swapper/308:50:383
218474399172,7cyclictest0-21swapper/308:00:373
218474399172,10cyclictest0-21swapper/311:10:383
218474399172,10cyclictest0-21swapper/310:45:383
218474099171,12cyclictest2184692-21munin-run07:10:010
218474399163,8cyclictest0-21swapper/312:35:373
218474399163,8cyclictest0-21swapper/311:35:303
218474399162,9cyclictest0-21swapper/309:45:433
218474399155,8cyclictest0-21swapper/307:35:373
218474399153,7cyclictest0-21swapper/309:00:013
218474399153,7cyclictest0-21swapper/308:30:413
218474399152,8cyclictest0-21swapper/307:10:383
218474399152,7cyclictest0-21swapper/307:30:003
218474399152,6cyclictest0-21swapper/308:20:383
218474399152,6cyclictest0-21swapper/308:20:383
218474399151,9cyclictest0-21swapper/309:00:363
218474399145,6cyclictest0-21swapper/308:15:373
218474399144,7cyclictest0-21swapper/307:15:393
218474399143,5cyclictest0-21swapper/308:05:383
218474399134,4cyclictest0-21swapper/307:20:343
218474399134,4cyclictest0-21swapper/307:20:343
218474399133,6cyclictest0-21swapper/308:25:383
218474399133,6cyclictest0-21swapper/308:25:383
218474399132,6cyclictest0-21swapper/307:45:353
218474399132,6cyclictest0-21swapper/307:40:363
218474399132,6cyclictest0-21swapper/307:30:353
218474399131,7cyclictest0-21swapper/308:45:393
218474399131,7cyclictest0-21swapper/308:10:353
218474399122,7cyclictest0-21swapper/308:40:293
218474399111,6cyclictest0-21swapper/307:50:363
21847439992,5cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional