You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-31 - 04:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Fri Oct 31, 2025 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
222866121010,5chrt0-21swapper/123:41:361
20112382700,3sleep130-21ksoftirqd/120:01:361
1955971996559,4cyclictest2146235-21kworker/u8:0+flush-8:022:30:001
1955966996452,8cyclictest2106401-21apt-get21:40:000
1955966996346,14cyclictest2240746-21latency_hist23:55:010
22490082620,2chrt201rcuc/000:01:370
20124892580,4sleep00-21swapper/020:05:000
1955971995650,4cyclictest2206078-21latency_hist23:20:011
1955971995445,7cyclictest2141506-21latency_hist22:15:011
1955971995245,5cyclictest2221216-21kworker/u8:1+events_unbound00:01:301
1955966995235,15cyclictest2270677-21latency_hist00:25:010
1955971995143,6cyclictest1962995-21kworker/u8:0+events_unbound19:46:281
1955971995043,5cyclictest1982393-21kworker/u8:3+events_unbound20:21:281
1955971995043,5cyclictest1982393-21kworker/u8:3+events_unbound20:21:281
1955971995043,5cyclictest1897890-21kworker/u8:3+events_unbound19:15:051
1955971994947,1cyclictest2255009-21kworker/u8:0+events_unbound00:06:461
1955971994841,5cyclictest1962995-21kworker/u8:0+events_unbound20:41:391
1955966994825,20cyclictest2085674-21ntpq21:16:370
1955971994743,3cyclictest2086112-21kworker/u8:1+flush-8:021:26:461
1955971994743,3cyclictest1982393-21kworker/u8:3+events_unbound20:36:501
1955971994742,3cyclictest2138917-21kworker/u8:3+flush-8:022:16:331
1955971994739,6cyclictest1962995-21kworker/u8:0+events_unbound19:36:381
1955971994738,7cyclictest282-21systemd-journal22:10:001
1955966994738,6cyclictest2225913-21latency_hist23:40:010
1955966994733,12cyclictest1982579-21gpgv19:35:000
1955971994640,5cyclictest1950670-21kworker/u8:1+events_unbound19:26:291
1955971994541,3cyclictest2138917-21kworker/u8:3+flush-8:023:30:511
1955971994541,3cyclictest2138917-21kworker/u8:3+events_unbound23:56:461
1955971994541,3cyclictest2117008-21kworker/u8:1+events_unbound21:51:451
1955971994540,4cyclictest1923836-21kworker/u8:2+flush-8:019:31:441
1955971994539,4cyclictest2221216-21kworker/u8:1+flush-8:023:46:331
1955971994438,5cyclictest2206347-21kworker/u8:1+events_unbound23:26:281
1955971994438,4cyclictest2052230-21kworker/u8:2+flush-8:020:51:381
1955971994435,3cyclictest2221216-21kworker/u8:1+events_unbound00:26:501
1955966994430,12cyclictest2022004-21grep20:11:480
1955966994427,14cyclictest2076324-21grep21:06:460
195492224412,7sleep01954972-21collect219:06:380
1955971994339,3cyclictest2206347-21kworker/u8:1+events_unbound23:23:521
1955971994339,3cyclictest2057193-21kworker/u8:3+events_unbound21:06:451
1955971994339,3cyclictest1962995-21kworker/u8:0+flush-8:021:16:281
1955971994338,3cyclictest2052230-21kworker/u8:2+events_unbound20:46:401
1955971994338,3cyclictest1962995-21kworker/u8:0+events_unbound21:11:451
1955971994337,5cyclictest2126410-21kworker/u8:0+flush-8:022:01:381
1955971994336,5cyclictest2138917-21kworker/u8:3+flush-8:000:31:411
1955971994336,5cyclictest2057193-21kworker/u8:3+flush-8:020:56:411
1955971994332,8cyclictest2006604-21ntpq19:56:371
1955966994336,5cyclictest2115689-21ntpq21:46:400
1955966994329,11cyclictest2138917-21kworker/u8:3+flush-8:000:36:480
1955971994238,3cyclictest2052230-21kworker/u8:2+flush-8:022:46:291
1955971994238,3cyclictest2022515-21kworker/u8:2+flush-8:020:31:471
1955966994228,12cyclictest2187017-21fschecks_time23:01:280
1955966994228,12cyclictest2187017-21fschecks_time23:01:280
1955971994132,7cyclictest2101446-21latency_hist21:35:011
1955966994133,6cyclictest282-21systemd-journal19:18:290
1955966994123,16cyclictest405-21dbus-daemon20:50:010
1955966994120,18cyclictest2220971-21latency_hist23:35:010
1955966994117,20cyclictest2061874-21latency_hist20:55:010
1955971994035,3cyclictest2052230-21kworker/u8:2+events_unbound22:21:481
1955971994030,7cyclictest2201137-21latency_hist23:15:011
1955971993935,3cyclictest1923836-21kworker/u8:2+events_unbound19:21:431
1955971993934,3cyclictest1982393-21kworker/u8:3+flush-8:019:41:451
1955971993933,5cyclictest2138917-21kworker/u8:3+events_unbound23:36:271
1955971993933,5cyclictest2057193-21kworker/u8:3+events_unbound21:01:291
1955966993928,8cyclictest282-21systemd-journal23:48:510
1955966993926,11cyclictest2280505-21unixbench_singl00:31:550
1955971993834,3cyclictest2255009-21kworker/u8:0+flush-8:000:16:451
1955971993834,3cyclictest2138917-21kworker/u8:3+flush-8:000:11:421
1955971993833,4cyclictest2221216-21kworker/u8:1+events_unbound00:20:071
1955966993829,7cyclictest282-21systemd-journal22:20:000
1955966993822,14cyclictest0-21swapper/022:36:350
1955966993822,14cyclictest0-21swapper/022:36:350
195596699382,14cyclictest584-21snmpd21:31:280
1955966993820,15cyclictest0-21swapper/020:40:000
20029092370,6sleep3451irq_work/319:56:253
1955971993732,4cyclictest2052230-21kworker/u8:2+events_unbound22:36:301
1955971993732,4cyclictest2052230-21kworker/u8:2+events_unbound22:36:301
1955971993732,4cyclictest1962995-21kworker/u8:0+flush-8:019:51:451
195596699373,15cyclictest584-21snmpd20:31:290
1955966993728,6cyclictest2027284-21true20:20:000
1955966993725,9cyclictest2150697-21ntpq22:21:430
1955966993721,13cyclictest0-21swapper/019:40:000
195596699371,22cyclictest2275838-21sshd00:29:510
1955971993631,4cyclictest2138917-21kworker/u8:3+events_unbound22:51:441
1955971993631,4cyclictest1962995-21kworker/u8:0+events_unbound20:16:441
1955971993630,4cyclictest2138917-21kworker/u8:3+events_unbound22:56:301
1955971993629,5cyclictest2122092-21apt-get21:56:251
1955971993627,7cyclictest2106858-21base6421:40:011
1955971993627,6cyclictest2116545-21latency_hist21:50:011
1955966993619,15cyclictest2007470-21systemctl20:00:000
1955966993618,16cyclictest2111612-21systemd-run21:45:010
1955966993612,20cyclictest0-21swapper/021:21:280
1955973993524,9cyclictest2027249-21awk20:20:002
1955971993531,3cyclictest2138917-21kworker/u8:3+flush-8:023:01:281
1955971993531,3cyclictest2138917-21kworker/u8:3+flush-8:023:01:281
1955971993530,4cyclictest2146235-21kworker/u8:0+flush-8:022:31:301
1955971993530,4cyclictest2052230-21kworker/u8:2+events_unbound21:35:281
1955966993527,5cyclictest2216012-21latency_hist23:30:010
1955966993521,11cyclictest833-21mta-sts-daemon23:16:390
1955966993519,14cyclictest0-21swapper/021:01:430
1955966993518,15cyclictest0-21swapper/023:41:300
1955966993516,16cyclictest421rcuog/222:41:390
1955966993516,16cyclictest421rcuog/222:41:390
1955966993514,19cyclictest2186330-21systemd-run23:00:000
1955966993511,9cyclictest1956481-21diskstats19:11:280
1955973993428,5cyclictest2052230-21kworker/u8:2+flush-8:021:16:392
1955971993430,3cyclictest2161574-21kworker/u8:1+flush-8:022:41:331
1955971993430,3cyclictest2161574-21kworker/u8:1+flush-8:022:41:331
1955971993429,4cyclictest1897890-21kworker/u8:3+flush-8:019:11:431
1955966993427,5cyclictest2156084-21grep22:26:450
1955966993415,17cyclictest0-21swapper/023:56:370
1955973993328,4cyclictest2196203-21kworker/u8:0+flush-8:023:12:022
1955973993324,7cyclictest2241191-21cp23:55:012
1955971993328,4cyclictest1962995-21kworker/u8:0+events_unbound20:05:091
195596699334,2cyclictest21369452aten2_r5power_e22:11:260
1955966993316,15cyclictest2016534-21ntpq20:06:440
1955966993313,18cyclictest2156729-21acpi22:31:250
1955973993228,3cyclictest2221216-21kworker/u8:1+flush-8:000:01:372
1955973993225,5cyclictest2091196-21grep21:21:482
1955971993228,3cyclictest2017114-21kworker/u8:1+flush-8:020:11:451
1955971993221,9cyclictest2036295-21ntpq20:26:421
195596699323,2cyclictest0-21swapper/019:41:280
1955966993223,7cyclictest2027597-21/usr/sbin/munin20:21:380
1955966993223,7cyclictest2027597-21/usr/sbin/munin20:21:380
1955966993220,9cyclictest2122341-21diskstats21:56:290
1955966993213,2cyclictest0-21swapper/020:26:370
1955966993212,2cyclictest0-21swapper/022:51:460
1955966993212,2cyclictest0-21swapper/000:11:370
1955966993212,1cyclictest0-21swapper/022:07:360
1955973993125,4cyclictest2240752-21apt-get23:55:002
1955971993127,3cyclictest2221216-21kworker/u8:1+events_unbound23:54:211
1955971993126,4cyclictest2052230-21kworker/u8:2+flush-8:023:07:011
1955971993126,3cyclictest2086112-21kworker/u8:1+events_unbound21:21:351
195596699318,6cyclictest1999686-21tr19:51:340
195596699311,9cyclictest1972112-21python319:21:440
195596699311,8cyclictest584-21snmpd21:11:250
1955966993113,2cyclictest2051087-21needreboot20:41:360
1955971993026,3cyclictest2138917-21kworker/u8:3+events_unbound00:37:271
195596699309,5cyclictest2066498-21grep20:56:480
195596699302,12cyclictest2264967-21ntpq00:16:470
1955966993018,10cyclictest610-21sshd19:45:500
1955966993012,1cyclictest0-21swapper/023:06:250
1955966993010,6cyclictest2200172-21sed23:11:390
195596699297,5cyclictest2120160-21kthreadcore21:51:360
195596699291,5cyclictest2251233-21/usr/sbin/munin00:06:320
1955966992913,3cyclictest2096141-21unixbench_multi21:26:460
195596699291,11cyclictest2130779-21ntpq22:01:380
195596699290,4cyclictest21721092chrt22:46:290
1955973992822,4cyclictest2146472-21sh22:20:002
1955973992820,6cyclictest2231211-21apt-get23:46:252
195596699288,6cyclictest2210044-21cat23:21:360
1955973992723,2cyclictest2067240-21aten2_r5power_e21:01:282
1955973992623,2cyclictest2256430-21fschecks_count00:11:282
1955973992623,2cyclictest2117425-21fschecks_count21:51:302
1955973992623,2cyclictest1923836-21kworker/u8:2+flush-8:019:31:342
195597399262,2cyclictest0-21swapper/222:36:462
195597399262,2cyclictest0-21swapper/222:36:462
1955973992622,3cyclictest1962995-21kworker/u8:0+events_unbound19:51:452
1955973992617,7cyclictest2275866-21turbostat.cron00:30:002
1955966992620,4cyclictest832-21mta-sts-daemon19:26:360
1955973992515,8cyclictest2255820-21awk00:10:002
1955973992514,8cyclictest2266325-21diskstats00:21:322
1955973992420,3cyclictest2052230-21kworker/u8:2+flush-8:022:56:422
1955973992419,3cyclictest0-21swapper/221:15:002
1955973992415,7cyclictest2017685-21munin-node20:11:272
195597399238,2cyclictest20112442kthreadcore20:01:362
1955973992320,2cyclictest2191953-21fschecks_time23:06:302
1955973992319,2cyclictest1963327-21cpuspeed_turbos19:16:292
1955973992319,2cyclictest0-21swapper/223:20:002
1955973992318,4cyclictest2138917-21kworker/u8:3+events_unbound23:33:522
195597399230,10cyclictest20545042kthreadcore20:46:352
1955973992219,2cyclictest1960502-21unixbench_multi19:11:452
1955973992217,4cyclictest2062598-21cut20:56:322
1955973992217,3cyclictest2206533-21aten2_r5power_p23:21:272
195597399216,3cyclictest21488512cat22:21:372
1955973992118,2cyclictest2215661-21unixbench_multi23:26:442
1955973992118,2cyclictest1972675-21cron19:25:002
1955973992117,3cyclictest2285732-21turbostat.cron00:40:002
1955973992116,4cyclictest2138917-21kworker/u8:3+flush-8:023:36:382
1955973992116,3cyclictest2091268-21kworker/u8:3+flush-8:021:55:012
1955973992116,3cyclictest2036822-21unixbench_singl20:26:502
19512422214,11sleep10-21swapper/119:06:231
20461432203,6sleep30-21swapper/320:36:433
195597399202,12cyclictest1993054-21apt-get19:46:272
1955973992016,2cyclictest0-21swapper/219:41:282
1955973992016,2cyclictest0-21swapper/219:36:362
1955973992012,6cyclictest2169708-21seq22:41:352
1955973992012,6cyclictest2169708-21seq22:41:352
1955973992011,7cyclictest2152542-21sed22:26:322
1955973991915,2cyclictest0-21swapper/222:15:002
1955973991915,2cyclictest0-21swapper/220:51:372
1955973991911,6cyclictest2175552-21grep22:46:372
195597399189,7cyclictest2108803-21tr21:41:342
195597399185,11cyclictest161rcu_preempt20:06:332
1955973991814,3cyclictest0-21swapper/221:40:002
1955973991813,3cyclictest0-21swapper/222:51:482
195597399181,2cyclictest21309722/usr/sbin/munin22:01:432
195597399181,12cyclictest2047401-21apt-get20:41:262
1955973991810,6cyclictest2229211-21cut23:41:362
19557522184,10sleep20-21swapper/219:09:252
19511872183,8sleep30-21swapper/319:05:413
195597399177,8cyclictest2132578-21sed22:06:322
195597399177,8cyclictest2031517-21cut20:21:432
195597399177,8cyclictest2031517-21cut20:21:432
195597399175,4cyclictest0-21swapper/222:35:002
195597399172,10cyclictest2279983-21ntpq00:31:422
1955973991712,3cyclictest0-21swapper/220:31:372
1955973991711,4cyclictest2116908-21apt-config21:50:012
195597399171,12cyclictest2261130-21apt-get00:16:322
195597399167,7cyclictest2096935-21cut21:31:302
195597399167,7cyclictest2095503-21grep21:26:372
195597399167,7cyclictest2042721-21fschecks_count20:36:332
195597399161,11cyclictest2072115-21apt-get21:06:262
19776252150,2sleep3451irq_work/319:30:003
195597399152,12cyclictest0-21swapper/219:56:432
1955973991511,2cyclictest1973070-21aten2_r5power_c19:26:272
1955973991510,3cyclictest0-21swapper/223:01:322
1955973991510,3cyclictest0-21swapper/223:01:322
22706592130,8sleep30-21swapper/300:25:013
22271602132,7sleep30-21swapper/323:41:353
22117752134,5sleep30-21swapper/323:26:303
22051442132,7sleep30-21swapper/323:16:393
21886252133,5sleep30-21swapper/323:01:343
21886252133,5sleep30-21swapper/323:01:343
21365722132,6sleep30-21swapper/322:10:013
21225812132,7sleep30-21swapper/321:56:333
20273402132,6sleep30-21swapper/320:20:013
20273402132,6sleep30-21swapper/320:20:013
19917102132,7sleep30-21swapper/319:41:373
22607952122,5sleep30-21swapper/300:15:013
22517652122,6sleep30-21swapper/300:06:353
22466622122,6sleep30-21swapper/300:01:333
22412012122,7sleep30-21swapper/323:55:033
22358062122,6sleep30-21swapper/323:50:013
22227792122,6sleep30-21swapper/323:36:333
21969082122,6sleep30-21swapper/323:11:333
21916072122,6sleep30-21swapper/323:06:273
21724402122,6sleep30-21swapper/322:46:333
21704292122,6sleep30-21swapper/322:41:393
21704292122,6sleep30-21swapper/322:41:393
21613712122,6sleep30-21swapper/322:35:013
21613712122,6sleep30-21swapper/322:35:013
21527502122,7sleep30-21swapper/322:26:343
21170632122,6sleep30-21swapper/321:50:513
21119512122,6sleep30-21swapper/321:46:293
20977592122,6sleep30-21swapper/321:31:363
20918082122,6sleep30-21swapper/321:25:513
20872792122,6sleep30-21swapper/321:21:333
20714812123,5sleep30-21swapper/321:01:473
20568342122,6sleep30-21swapper/320:48:513
20371312122,6sleep30-21swapper/320:30:013
20325652122,6sleep30-21swapper/320:26:313
20182042121,7sleep30-21swapper/320:11:353
20145952122,6sleep30-21swapper/320:06:413
20115612122,6sleep30-21swapper/320:01:393
20003972122,6sleep30-21swapper/319:51:363
19973602122,6sleep30-21swapper/319:46:513
19835822122,6sleep30-21swapper/319:36:313
19776252122,5sleep30-21swapper/319:30:013
19718252122,6sleep30-21swapper/319:21:383
22761772113,5sleep30-21swapper/300:31:153
22712682112,5sleep30-21swapper/300:26:293
22597982111,6sleep30-21swapper/300:11:363
22356792112,5sleep30-21swapper/323:48:493
22180762112,5sleep30-21swapper/323:31:353
22095612112,5sleep30-21swapper/323:21:363
21859462112,5sleep30-21swapper/322:56:483
21767802112,5sleep30-21swapper/322:51:263
21566142112,6sleep30-21swapper/322:30:033
21468582112,5sleep30-21swapper/322:21:293
21417882112,5sleep30-21swapper/322:15:513
21342992112,5sleep30-21swapper/322:06:363
21270482112,5sleep30-21swapper/322:01:263
21114972112,6sleep30-21swapper/321:44:013
21062092112,5sleep30-21swapper/321:37:463
20818782113,5sleep30-21swapper/321:15:393
20814812112,6sleep30-21swapper/321:13:353
20722122112,5sleep30-21swapper/321:06:283
20626002112,6sleep30-21swapper/320:56:333
20572032113,5sleep30-21swapper/320:50:353
20473752112,5sleep30-21swapper/320:41:243
20234832112,5sleep30-21swapper/320:16:363
19634862112,6sleep30-21swapper/319:16:333
19561162113,5sleep30-21swapper/319:10:153
22819682100,6sleep30-21swapper/300:36:363
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional