You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 05:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot4s.osadl.org (updated Wed Mar 25, 2026 00:44:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
260668821740,3sleep375-21kworker/3:1+events23:25:013
26155452630,6chrt0-21swapper/223:31:422
2351231995242,8cyclictest2516123-21ntpq21:51:410
2351231994940,6cyclictest2472549-21latency_hist21:10:010
2351238994837,8cyclictest2561724-21awk22:40:002
2351236994841,4cyclictest2398218-21latency_hist19:55:011
2351236994840,5cyclictest2611439-21latency_hist23:30:011
2351231994740,4cyclictest2615558-21kworker/u8:2+flush-8:000:26:330
2351236994634,9cyclictest280-21systemd-journal00:30:001
2351236994431,11cyclictest280-21systemd-journal19:27:401
2351231994232,7cyclictest2631255-21latency_hist23:50:010
26745902400,1sleep02674594-21grep00:31:390
2351236994029,8cyclictest280-21systemd-journal19:12:311
2351231994030,7cyclictest2413015-21latency_hist20:10:010
235123699391,23cyclictest589-21runrttasks23:41:461
2351236993828,8cyclictest2650200-21ntpq00:06:391
2351236993828,8cyclictest2541681-21apt22:20:001
2351238993730,5cyclictest2601779-21kworker/u8:0+flush-8:023:56:442
2351236993710,16cyclictest421rcuog/222:36:091
2351231993728,6cyclictest2487351-21latency_hist21:25:010
2351231993725,9cyclictest280-21systemd-journal21:05:000
235123699369,17cyclictest421rcuog/220:23:301
235123699365,16cyclictest2499730-21tr21:36:371
2351236993629,5cyclictest2393049-21latency_hist19:50:011
2351231993630,4cyclictest2377039-21kworker/u8:3+flush-8:021:26:310
2351231993630,4cyclictest2322637-21kworker/u8:1+flush-8:020:11:320
2351231993630,4cyclictest2322637-21kworker/u8:1+flush-8:000:06:300
25712052350,3chrt75-21kworker/3:1+events22:46:483
2351236993527,5cyclictest2531768-21latency_hist22:10:001
2351236993525,8cyclictest2482767-21apt-get21:21:271
2351236993524,9cyclictest2507118-21systemd-run21:45:011
2351236993524,9cyclictest2399164-21irqcore19:56:351
2351236993518,14cyclictest2680079-21ntpq00:36:411
2351231993531,2cyclictest280-21systemd-journal20:40:000
2351231993530,4cyclictest2322637-21kworker/u8:1+flush-8:020:16:350
2351231993529,4cyclictest2507089-21cron21:45:000
2351231993529,4cyclictest2322637-21kworker/u8:1+flush-8:020:31:320
2351231993527,6cyclictest2601292-21latency_hist23:20:000
23465162354,10sleep10-21swapper/119:06:161
235123699347,18cyclictest421rcuog/221:59:251
235123699347,17cyclictest0-21swapper/121:15:111
2351236993421,11cyclictest2601705-21apt-key23:20:011
235123199341,31cyclictest1-21systemd22:30:010
2351236993322,9cyclictest2404055-21irqcore20:01:331
2351236993319,12cyclictest2555527-21/usr/sbin/munin22:31:401
2351236993316,15cyclictest2566985-21/usr/sbin/munin22:46:471
2351231993327,5cyclictest2313455-21kworker/u8:0+flush-8:019:26:320
2351231993327,4cyclictest2636630-21grep23:56:290
2351231993327,4cyclictest2601779-21kworker/u8:0+flush-8:023:36:330
2351231993326,5cyclictest280-21systemd-journal20:27:460
2351231993326,5cyclictest2565799-21ntpq22:41:410
2351231993323,7cyclictest2571574-21latency_hist22:50:010
2351238993226,4cyclictest2511733-21unixbench_singl21:46:482
235123699325,18cyclictest0-21swapper/119:39:081
2351236993225,5cyclictest2606504-21latency_hist23:25:011
2351236993217,13cyclictest2612684-21kernelversion23:31:381
235123699316,17cyclictest421rcuog/221:33:371
235123699315,17cyclictest0-21swapper/121:08:211
235123699313,25cyclictest0-21swapper/122:41:291
2351236993126,3cyclictest2662221-21irqrtprio00:21:371
2351236993126,3cyclictest2627396-21irqrtprio23:46:351
2351236993120,9cyclictest2591443-21systemd-run23:10:001
2351236993119,10cyclictest2536448-21unixbench_singl22:11:501
2351236993119,10cyclictest2536448-21unixbench_singl22:11:501
2351236993118,11cyclictest2443810-21irqcore20:41:351
2351236993118,10cyclictest0-21swapper/119:31:411
235123699303,19cyclictest421rcuog/220:39:071
2351236993020,9cyclictest2388097-21cron19:45:001
2351236993020,7cyclictest280-21systemd-journal22:57:401
2351236993019,9cyclictest0-21swapper/123:36:451
2351231993026,3cyclictest2383118-21latency_hist19:40:010
2351231993026,2cyclictest2482415-21sendmail21:20:000
2351231993026,2cyclictest0-21swapper/000:40:000
2351231993024,4cyclictest2362976-21sendmail19:20:000
26605192290,5sleep10-21swapper/100:16:471
2351238992920,7cyclictest280-21systemd-journal21:21:402
235123699292,21cyclictest407-21mta-sts-daemon19:16:291
2351236992917,10cyclictest2592499-21irqrtprio23:11:351
2351236992913,14cyclictest2507437-21find21:46:281
235123699291,24cyclictest583-21snmpd19:21:431
235123699291,23cyclictest2472908-21apt-get21:11:311
2351236992912,14cyclictest2437165-21ntpq20:31:441
2351236992911,14cyclictest2551572-21latency_hist22:30:011
2351231992925,3cyclictest2601779-21kworker/u8:0+events_unbound23:26:380
2351231992924,3cyclictest2537317-21cpuspeed_turbos22:16:320
2351231992924,3cyclictest2377039-21kworker/u8:3+flush-8:019:46:470
2351231992920,7cyclictest2363302-21/usr/sbin/munin19:21:400
2351231992918,8cyclictest280-21systemd-journal23:05:000
2351238992825,2cyclictest2432466-21unixbench_singl20:26:482
2351238992820,5cyclictest2622313-21irqcore23:41:362
2351238992819,7cyclictest2372881-21cat19:30:002
235123899280,18cyclictest2520495-21kthreadcore21:56:372
235123699288,17cyclictest280-21systemd-journal20:16:401
235123699282,3cyclictest0-21swapper/122:01:411
235123699282,3cyclictest0-21swapper/120:26:421
2351236992823,3cyclictest2516992-21systemd-run21:55:001
235123699282,23cyclictest0-21swapper/122:21:281
235123699282,22cyclictest2641487-21apt-get00:01:301
2351236992817,9cyclictest0-21swapper/121:30:011
235123699281,23cyclictest2631609-21apt-get23:51:291
2351231992824,3cyclictest2531755-21cron22:10:000
2351231992824,3cyclictest2377039-21kworker/u8:3+flush-8:020:01:310
2351231992824,2cyclictest0-21swapper/023:00:000
2351231992822,4cyclictest2601779-21kworker/u8:0+flush-8:023:21:460
2351231992818,8cyclictest2387299-21ntpq19:41:430
2351231992818,7cyclictest2526851-21latency_hist22:05:010
235123199281,25cyclictest2398797-21cpuspeed_turbos19:56:300
2351238992723,2cyclictest2591118-21unixbench_singl23:06:482
2351238992723,2cyclictest2571962-21aten2_r5power_c22:51:292
235123699277,18cyclictest0-21swapper/120:11:331
2351236992723,2cyclictest885-21mta-sts-daemon00:31:311
2351236992723,2cyclictest0-21swapper/100:00:001
235123699272,22cyclictest2461814-21ntpq20:56:401
235123699272,22cyclictest0-21swapper/122:51:281
2351231992723,2cyclictest2398213-21cron19:55:000
2351231992722,4cyclictest2361213-21kworker/u8:2+events_unbound22:51:470
2351231992720,5cyclictest2507487-21wget21:46:300
235123899266,3cyclictest25805182wc22:56:422
2351238992623,2cyclictest2446963-21grep20:41:402
235123699263,20cyclictest0-21swapper/120:46:281
235123699263,20cyclictest0-21swapper/120:46:281
235123699262,3cyclictest0-21swapper/120:06:301
235123699262,3cyclictest0-21swapper/100:11:421
235123699262,12cyclictest0-21swapper/123:00:491
235123699261,7cyclictest2456870-21ntpq20:51:401
235123699261,22cyclictest2463749-21sed21:01:341
2351231992621,3cyclictest2645781-21unixbench_singl00:01:490
2351231992618,6cyclictest2423508-21diskstats20:21:290
2351231992616,7cyclictest2373492-21diskstats19:31:320
2351231992615,9cyclictest2451957-21ntpq20:46:410
2351231992615,9cyclictest2451957-21ntpq20:46:410
26458092250,2sleep3481ktimers/300:01:483
2351238992522,2cyclictest2531407-21unixbench_multi22:06:482
2351238992522,2cyclictest2468190-21cpuspeed_turbos21:06:312
2351238992521,2cyclictest2473174-21fschecks_count21:11:352
2351238992520,3cyclictest2367943-21cron19:25:002
2351231992520,3cyclictest2556932-21apt-key22:35:010
2351231992519,4cyclictest2615558-21kworker/u8:2+flush-8:000:16:460
2351238992415,7cyclictest2631247-21turbostat.cron23:50:002
2351238992415,7cyclictest2596354-21awk23:15:002
2351231992420,2cyclictest0-21swapper/021:36:380
2351231992419,4cyclictest2377039-21kworker/u8:3+flush-8:021:11:400
2351231992419,4cyclictest2377039-21kworker/u8:3+flush-8:020:56:320
2351231992419,3cyclictest2542350-21fschecks_count22:21:310
2351231992415,6cyclictest2655932-21apt-get00:15:010
2351238992319,2cyclictest2671585-21cpuspeed_turbos00:31:322
2351238992319,2cyclictest2448428-21cpuspeed_turbos20:46:302
2351238992319,2cyclictest2448428-21cpuspeed_turbos20:46:302
2351238992318,3cyclictest2398204-21cron19:55:002
235123899230,10cyclictest23740032cut19:31:362
2351231992320,2cyclictest0-21swapper/020:55:000
2351231992319,2cyclictest0-21swapper/020:41:310
2351231992317,4cyclictest2601779-21kworker/u8:0+flush-8:023:31:520
2351231992316,5cyclictest2361213-21kworker/u8:2+flush-8:023:11:460
2351238992218,2cyclictest2676494-21cpuspeed_turbos00:36:292
2351238992218,2cyclictest2567932-21tail22:46:372
2351238992218,2cyclictest2562162-21aten2_r5power_p22:41:302
2351238992218,2cyclictest2477865-21aten2_r5power_c21:16:302
2351238992218,2cyclictest2383097-21latency_hist19:40:012
2351238992217,4cyclictest2437616-21unixbench_singl20:31:482
2351238992215,5cyclictest2412662-21grep20:06:492
235123199222,14cyclictest2496377-21chrt21:31:410
2351231992218,3cyclictest401-21dbus-daemon23:55:010
2351231992218,2cyclictest0-21swapper/019:11:370
23464682223,13sleep00-21swapper/019:05:390
25275002210,3sleep3451irq_work/322:06:313
2351238992118,2cyclictest2552016-21aten2_r5power_p22:31:312
2351238992117,2cyclictest0-21swapper/200:15:012
2351238992116,4cyclictest0-21swapper/223:20:002
2351238992116,3cyclictest2428048-21latency_hist20:25:012
2351238992114,5cyclictest2393420-21base6419:50:012
2351231992119,1cyclictest0-21swapper/023:07:400
2351231992117,3cyclictest2377039-21kworker/u8:3+flush-8:021:56:320
2351231992116,3cyclictest0-21swapper/022:40:010
2351231992111,8cyclictest2665263-21ntpq00:21:430
235123899208,4cyclictest0-21swapper/220:51:272
235123899207,4cyclictest0-21swapper/221:51:472
2351238992017,2cyclictest2522505-21cpuspeed_turbos22:01:312
2351238992017,2cyclictest2458342-21fschecks_count20:56:322
2351238992016,2cyclictest0-21swapper/223:21:442
2351238992016,2cyclictest0-21swapper/223:01:292
2351231992013,5cyclictest2615558-21kworker/u8:2+flush-8:023:41:340
23508112204,12sleep20-21swapper/219:06:492
2351238991915,3cyclictest20977-21cron22:20:002
2351238991915,2cyclictest2403557-21aten2_r5power_c20:01:282
2351238991914,3cyclictest2547562-21sed22:26:342
2351238991914,3cyclictest0-21swapper/220:16:402
2351238991913,4cyclictest2501801-21unixbench_multi21:36:482
2351231991915,2cyclictest0-21swapper/022:11:500
2351231991915,2cyclictest0-21swapper/022:11:500
25835212180,4sleep3481ktimers/323:01:373
235123899182,12cyclictest2502526-21apt-get21:41:292
2351238991815,2cyclictest2467291-21unixbench_singl21:01:462
2351238991815,2cyclictest2413685-21fschecks_time20:11:322
2351238991814,2cyclictest2631831-21tr23:51:302
2351238991814,2cyclictest0-21swapper/223:26:292
2351238991814,2cyclictest0-21swapper/222:13:492
2351238991814,2cyclictest0-21swapper/222:13:492
23464462182,9sleep30-21swapper/319:05:233
25759582177,6sleep30-21swapper/322:51:473
235123899178,2cyclictest0-21swapper/221:31:442
2351238991715,1cyclictest0-21swapper/219:56:462
2351238991713,2cyclictest0-21swapper/220:36:412
2351238991713,2cyclictest0-21swapper/219:10:182
2351238991713,2cyclictest0-21swapper/200:03:522
2351238991712,3cyclictest0-21swapper/221:26:292
235123899167,7cyclictest2648190-21tr00:06:352
235123899162,2cyclictest0-21swapper/200:21:272
235123899162,13cyclictest0-21swapper/222:20:292
235123899162,13cyclictest0-21swapper/219:40:422
2351238991613,2cyclictest0-21swapper/200:16:282
2351238991611,3cyclictest0-21swapper/200:26:382
235123899157,6cyclictest2617561-21sed23:36:362
235123899156,6cyclictest2359627-21tr19:16:372
26800532132,7sleep30-21swapper/300:36:413
26701512132,7sleep30-21swapper/300:26:423
26106282134,5sleep30-21swapper/323:26:423
24865612132,7sleep30-21swapper/321:21:423
24667502132,7sleep30-21swapper/321:01:403
24568632132,7sleep30-21swapper/320:51:403
24469672132,7sleep30-21swapper/320:41:413
23930352132,7sleep30-21swapper/319:50:013
23828682131,6sleep30-21swapper/319:40:013
26619092122,7sleep30-21swapper/300:21:363
26568912122,6sleep30-21swapper/300:16:343
26502952122,6sleep30-21swapper/300:06:433
26320322122,6sleep30-21swapper/323:51:343
26303312122,6sleep30-21swapper/323:46:413
26261772123,6sleep30-21swapper/323:43:243
26122362122,6sleep30-21swapper/323:31:363
25927122122,6sleep30-21swapper/323:11:373
25648602122,6sleep30-21swapper/322:41:403
25571042123,6sleep30-21swapper/322:36:303
25409882122,6sleep30-21swapper/322:16:483
25358452122,6sleep30-21swapper/322:11:423
25358452122,6sleep30-21swapper/322:11:423
25251042122,5sleep30-21swapper/322:01:393
25169542122,5sleep30-21swapper/321:55:013
25074632122,6sleep30-21swapper/321:46:303
24980382120,8sleep30-21swapper/321:36:353
24921592122,6sleep30-21swapper/321:28:403
24818232122,6sleep30-21swapper/321:16:483
24682162122,6sleep30-21swapper/321:06:333
24422112122,6sleep30-21swapper/320:36:483
24187162122,6sleep30-21swapper/320:16:373
24126912122,6sleep30-21swapper/320:06:503
24049042122,6sleep30-21swapper/320:01:363
23587412122,6sleep30-21swapper/319:16:343
23555362122,6sleep30-21swapper/319:11:483
26717572112,5sleep30-21swapper/300:31:353
26512522113,5sleep30-21swapper/300:10:123
26374042111,6sleep30-21swapper/323:56:363
26163992112,5sleep30-21swapper/323:35:013
25965782112,5sleep30-21swapper/323:15:023
25867572113,5sleep30-21swapper/323:05:563
25768012112,5sleep30-21swapper/322:55:403
25520462112,6sleep30-21swapper/322:31:323
25474942112,5sleep30-21swapper/322:26:353
25457842112,5sleep30-21swapper/322:21:413
25173302112,5sleep30-21swapper/321:56:293
25024952112,5sleep30-21swapper/321:41:243
24968502112,5sleep30-21swapper/321:31:493
24728822112,5sleep30-21swapper/321:11:313
24580822112,6sleep30-21swapper/320:56:303
24486832112,5sleep30-21swapper/320:46:353
24486832112,5sleep30-21swapper/320:46:353
24332892112,5sleep30-21swapper/320:30:503
24281202112,5sleep30-21swapper/320:25:483
24271082112,5sleep30-21swapper/320:21:433
24133442112,5sleep30-21swapper/320:11:273
23984642113,5sleep30-21swapper/319:55:223
23843842112,5sleep30-21swapper/319:41:383
23686592112,5sleep30-21swapper/319:26:333
23666632112,5sleep30-21swapper/319:21:393
23940092102,5sleep30-21swapper/319:51:333
23731122102,5sleep30-21swapper/319:30:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional