You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 12:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot4s.osadl.org (updated Thu Sep 18, 2025 00:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34916692730,2chrt662-21kworker/3:2-events21:20:003
35898802640,5chrt0-21swapper/023:00:530
34467672590,6sleep20-21swapper/220:30:442
3364702995949,7cyclictest3566857-21latency_hist22:40:011
34444262520,3chrt662-21kworker/3:2+events20:25:513
3364702995243,7cyclictest3562125-21kworker/u8:2+flush-8:022:51:001
3364702995144,5cyclictest3451416-21kworker/u8:1+flush-8:020:51:031
3364702995143,6cyclictest3471463-21/usr/sbin/munin20:55:581
3364702995139,4cyclictest3611641-21kworker/u8:3+events_unbound00:05:501
3364702995040,3cyclictest3307827-21kworker/u8:1+events_unbound19:30:461
3364702994943,5cyclictest3451416-21kworker/u8:1+flush-8:022:06:011
3364702994843,4cyclictest3400909-21kworker/u8:2+flush-8:019:45:461
3364702994841,5cyclictest2521540-21kworker/u8:3+flush-8:020:40:591
3364702994641,4cyclictest3451416-21kworker/u8:1+events_unbound22:15:441
3364698994630,14cyclictest3411326-21sh19:55:000
3364698994626,18cyclictest3496552-21latency_hist21:25:010
3364702994540,4cyclictest2521540-21kworker/u8:3+events_unbound20:45:491
3364698994537,6cyclictest3385496-21ntpq19:25:520
3364698994527,16cyclictest3556521-21unixbench_multi22:26:000
3364702994440,3cyclictest2521540-21kworker/u8:3+events_unbound20:30:521
3364702994439,3cyclictest3486868-21kworker/u8:0+flush-8:021:40:511
3364698994431,11cyclictest3425308-21ntpq20:05:550
3364698994416,21cyclictest421rcuog/220:51:040
3364702994339,3cyclictest3616610-21kworker/u8:2+events_unbound23:41:001
3364702994337,5cyclictest3611641-21kworker/u8:3+events_unbound00:25:481
3364698994332,8cyclictest3461068-21apt-get20:45:010
3364702994237,4cyclictest3451416-21kworker/u8:1+events_unbound22:25:441
3364702994237,4cyclictest2521540-21kworker/u8:3+events_unbound21:30:501
3364702994236,5cyclictest3307827-21kworker/u8:1+flush-8:019:16:031
3364698994238,3cyclictest3451416-21kworker/u8:1+events_unbound22:15:480
3364698994225,15cyclictest3576352-21unixbench_multi22:46:010
336469899420,25cyclictest3369737-21install19:15:010
3364702994137,3cyclictest2521540-21kworker/u8:3+events_unbound19:25:461
3364702994135,5cyclictest2521540-21kworker/u8:3+events_unbound21:15:461
3364702994134,5cyclictest3461543-21kworker/u8:2+flush-8:021:10:521
3364698994129,10cyclictest3481977-21apt-get21:10:380
3364698994124,15cyclictest0-21swapper/020:00:470
3364698994124,15cyclictest0-21swapper/020:00:470
3364702994036,3cyclictest3426198-21kworker/u8:0+events_unbound20:20:011
3364702994034,4cyclictest3441099-21gpgv20:25:001
3364698994011,21cyclictest0-21swapper/000:18:400
3364705993933,4cyclictest3521522-21gpgv21:50:002
3364705993933,4cyclictest3521522-21gpgv21:50:002
3364702993934,4cyclictest3611641-21kworker/u8:3+flush-8:023:25:501
3364702993934,4cyclictest3572014-21kworker/u8:1+flush-8:000:15:591
3364702993934,4cyclictest3451416-21kworker/u8:1+events_unbound22:20:471
3364702993933,4cyclictest3451416-21kworker/u8:1+flush-8:022:11:001
3364702993933,4cyclictest2521540-21kworker/u8:3+flush-8:019:55:531
3364702993932,5cyclictest3616646-21apt-get23:30:381
3364698993923,14cyclictest3521498-21latency_hist21:50:000
3364698993923,14cyclictest3521498-21latency_hist21:50:000
3364698993923,14cyclictest3452377-21kthreadcore20:35:490
3364698993922,15cyclictest3645938-21grep23:56:020
3364702993833,4cyclictest3562125-21kworker/u8:2+events_unbound23:05:481
3364702993833,4cyclictest3411354-21kworker/u8:0+events_unbound20:00:421
3364702993833,4cyclictest3411354-21kworker/u8:0+events_unbound20:00:421
3364702993831,5cyclictest3567171-21apt-get22:40:421
3364702993831,5cyclictest2521540-21kworker/u8:3+events_unbound20:25:451
3364702993830,6cyclictest3636193-21latency_hist23:50:011
3364702993828,7cyclictest3640245-21ntpq23:50:551
3364705993730,5cyclictest3451416-21kworker/u8:1+flush-8:021:06:092
3364702993732,4cyclictest3562125-21kworker/u8:2+flush-8:023:11:031
3364702993731,5cyclictest3420937-21kworker/u8:1+flush-8:020:10:491
3364702993731,4cyclictest3322966-21kworker/u8:2+events_unbound19:21:021
3364702993730,5cyclictest3611641-21kworker/u8:3+events_unbound00:10:481
3364702993632,3cyclictest3611641-21kworker/u8:3+flush-8:023:36:031
3364702993632,3cyclictest3572014-21kworker/u8:1+flush-8:023:21:011
3364702993631,3cyclictest3611641-21kworker/u8:3+events_unbound00:35:021
3364702993630,4cyclictest3400909-21kworker/u8:2+events_unbound19:50:591
3364702993628,7cyclictest3451416-21kworker/u8:1+flush-8:021:00:551
3364702993628,6cyclictest3527288-21latency_hist22:00:011
3364698993629,5cyclictest3562125-21kworker/u8:2+flush-8:022:35:470
336469899362,22cyclictest0-21swapper/022:05:060
3364698993621,13cyclictest3435233-21ntpq20:15:560
3364698993621,13cyclictest3426714-21tr20:10:440
3364702993528,5cyclictest3572014-21kworker/u8:1+flush-8:022:45:531
3364698993530,3cyclictest3441638-21aten2_r5power_c20:25:430
336469899352,29cyclictest3390452-21ntpq19:30:560
336469899352,28cyclictest3621630-21apt-get23:35:430
3364698993518,15cyclictest3627112-21irqcore23:40:490
3364698993518,15cyclictest3462326-21tr20:45:480
3364698993516,16cyclictest3506869-21latency_hist21:35:010
336470599349,12cyclictest3621630-21apt-get23:35:442
3364702993429,4cyclictest3562125-21kworker/u8:2+flush-8:023:15:361
3364702993429,4cyclictest3451416-21kworker/u8:1+flush-8:022:00:561
3364702993429,4cyclictest2521540-21kworker/u8:3+events_unbound20:35:371
3364702993429,3cyclictest3486868-21kworker/u8:0+events_unbound22:30:441
3364702993429,3cyclictest3451416-21kworker/u8:1+events_unbound21:25:491
336469899344,23cyclictest0-21swapper/000:35:410
336469899343,24cyclictest0-21swapper/023:20:430
336469899342,2cyclictest0-21swapper/019:45:420
3364698993422,9cyclictest3401216-21apt-get19:45:000
3364698993416,16cyclictest3671226-21latency_hist00:25:000
3364698993415,17cyclictest3596133-21unixbench_multi23:06:020
336443923421,9sleep20-21swapper/219:08:552
3364702993327,5cyclictest3486868-21kworker/u8:0+flush-8:021:20:551
336469899334,24cyclictest0-21swapper/023:45:430
336469899333,23cyclictest0-21swapper/019:55:410
336469899333,23cyclictest0-21swapper/000:25:410
3364698993325,5cyclictest3511617-21latency_hist21:40:010
3364698993318,12cyclictest0-21swapper/023:15:420
3364698993317,14cyclictest3381452-21latency_hist19:25:010
3364698993317,12cyclictest0-21swapper/021:15:420
3364698993316,14cyclictest0-21swapper/000:30:570
3364698993315,16cyclictest3615834-21sort23:26:020
3364698993314,16cyclictest3481677-21latency_hist21:10:010
3364698993312,18cyclictest2583070-21systemd-journal22:15:000
336470599326,16cyclictest3377165-21perl19:20:462
3364702993229,2cyclictest3426204-21systemd-run20:10:001
3364702993228,3cyclictest3307827-21kworker/u8:1+flush-8:019:35:491
3364702993227,4cyclictest3516584-21kworker/u8:2+flush-8:021:45:471
3364702993227,4cyclictest3516584-21kworker/u8:2+flush-8:021:45:471
3364702993226,5cyclictest3572014-21kworker/u8:1+events_unbound23:00:471
336469899324,24cyclictest0-21swapper/023:30:390
336469899323,24cyclictest0-21swapper/020:20:410
336469899322,26cyclictest3500858-21ntpq21:25:560
336469899321,3cyclictest3660410-21ntpq00:10:590
336469899321,3cyclictest3650523-21ntpq00:00:540
3364705993125,5cyclictest2521540-21kworker/u8:3+flush-8:020:14:102
3364705993117,9cyclictest3391600-21apt-get19:35:422
3364705993110,4cyclictest3480192-21ntpq21:00:562
3364702993127,3cyclictest3611641-21kworker/u8:3+events_unbound00:35:141
3364702993126,4cyclictest3486868-21kworker/u8:0+flush-8:021:35:581
3364702993125,4cyclictest3665874-21kworker/u8:0+events_unbound00:20:571
336469899313,2cyclictest0-21swapper/022:50:420
336469899313,25cyclictest0-21swapper/023:10:440
336469899313,25cyclictest0-21swapper/022:20:440
336469899313,24cyclictest0-21swapper/020:30:410
336469899313,24cyclictest0-21swapper/019:35:410
336469899313,23cyclictest0-21swapper/021:55:380
336469899312,23cyclictest660-21snmpd19:15:440
3364698993113,14cyclictest0-21swapper/023:00:000
336469899311,26cyclictest3480243-21ntpq21:00:570
336469899310,8cyclictest3475577-21/usr/sbin/munin20:56:000
336470599309,6cyclictest3515583-21sed21:40:532
3364705993026,3cyclictest3675721-21unixbench_singl00:26:022
3364705993021,7cyclictest3685934-21awk00:40:002
3364705993021,7cyclictest3661836-21fschecks_count00:15:492
3364705993020,5cyclictest3616646-21apt-get23:30:402
3364705993016,10cyclictest3652220-21ps00:05:512
3364705993015,2cyclictest34678142tr20:50:532
3364705993015,11cyclictest3412012-21/usr/sbin/munin19:55:452
3364702993025,3cyclictest3537169-21kworker/u8:3+events_unbound22:55:521
336469899303,24cyclictest0-21swapper/021:40:420
336469899303,23cyclictest0-21swapper/022:40:420
3364698993026,2cyclictest0-21swapper/022:01:010
3364698993022,6cyclictest3522354-21latency_hist21:55:000
336469899301,4cyclictest3561039-21ntpq22:30:560
336469899301,3cyclictest3655466-21ntpq00:05:580
336470599298,5cyclictest3640211-21ntpq23:50:542
336470599298,5cyclictest3537451-21awk22:10:412
336470599298,5cyclictest3492681-21cut21:20:482
3364705992918,8cyclictest3528369-21cut22:00:492
3364705992917,8cyclictest3365059-21/usr/sbin/munin19:10:442
3364705992917,10cyclictest3420782-21/usr/sbin/munin20:00:592
3364705992917,10cyclictest3420782-21/usr/sbin/munin20:00:592
3364705992916,11cyclictest3526344-21/usr/sbin/munin21:55:502
3364705992915,11cyclictest3435605-21/usr/sbin/munin20:16:012
3364705992910,3cyclictest3551154-21ntpq22:20:582
3364702992924,3cyclictest3481666-21cron21:10:001
336469899291,4cyclictest3640237-21ntpq23:50:550
336470599287,5cyclictest3670267-21ntpq00:20:582
336470599287,5cyclictest3546178-21ntpq22:15:542
3364705992824,2cyclictest3369926-21irqrtprio19:15:022
3364705992823,3cyclictest3641950-21fschecks_count23:55:472
3364705992821,5cyclictest3633706-21kthreadcore23:45:522
3364705992820,6cyclictest3587216-21fschecks_count23:00:472
3364705992816,9cyclictest3456479-21/usr/sbin/munin20:40:432
3364702992822,4cyclictest3611641-21kworker/u8:3+events_unbound23:55:441
3364702992822,4cyclictest2583070-21systemd-journal19:15:011
336469899281,11cyclictest3364701-21sed19:10:010
3364705992721,4cyclictest3611977-21fschecks_count23:25:462
3364705992721,4cyclictest3567485-21fschecks_time22:40:472
3364705992721,4cyclictest3562420-21cpuspeed_turbos22:35:452
3364705992719,6cyclictest3646974-21fschecks_time00:00:432
3364705992716,8cyclictest3471412-21seq20:55:022
3364705992715,9cyclictest3582372-21/usr/sbin/munin22:55:482
3364705992714,10cyclictest3676324-21seq00:30:132
3364705992712,12cyclictest3580789-21grep22:50:542
3364702992723,3cyclictest3400909-21kworker/u8:2+events_unbound19:43:151
3364702992722,4cyclictest3572014-21kworker/u8:1+events_unbound00:00:531
336470599267,4cyclictest3602031-21df23:15:432
3364705992623,2cyclictest3482292-21fschecks_time21:10:432
3364705992614,10cyclictest3630422-21runrttasks23:40:572
3364705992614,10cyclictest3501030-21/usr/sbin/munin21:26:002
3364705992614,10cyclictest3425133-21/usr/sbin/munin20:05:542
3364705992613,11cyclictest3601451-21taskset23:14:342
3364705992613,10cyclictest3487027-21/usr/sbin/munin21:15:452
3364705992612,12cyclictest3456054-21chrt20:39:432
3364705992611,12cyclictest3606752-21chrt23:20:412
3364705992521,2cyclictest3441455-21latency_hist20:25:012
3364705992518,3cyclictest3536273-21ntpq22:05:562
3364705992516,7cyclictest3660728-21/usr/sbin/munin00:11:042
3364705992516,7cyclictest3576020-21/usr/sbin/munin22:45:582
3364705992516,7cyclictest3410953-21/usr/sbin/munin19:51:032
3364705992515,8cyclictest3405711-21/usr/sbin/munin19:45:592
3364705992515,6cyclictest3382034-21/usr/sbin/munin19:25:422
3364705992514,8cyclictest3507566-21/usr/sbin/munin21:35:502
3364705992513,9cyclictest3595621-21awk23:05:552
3364705992513,8cyclictest3461606-21/usr/sbin/munin20:45:432
3364705992513,10cyclictest3506518-21taskset21:33:042
3364705992512,10cyclictest3522322-21chrt21:54:462
3364705992510,11cyclictest3552291-21chrt22:25:362
336435122510,9sleep10-21swapper/119:07:461
34478342240,4sleep3481ktimers/320:30:493
3364705992412,9cyclictest3396498-21chrt19:40:162
3364705992411,11cyclictest3443974-21tr20:25:512
3364705992318,3cyclictest3386883-21cpuspeed_turbos19:30:452
3364705992312,9cyclictest3561818-21chrt22:34:112
3364702992011,7cyclictest3521942-21cat21:50:011
33643082183,8sleep30-21swapper/319:07:123
34825612156,5sleep30-21swapper/321:10:463
36783612144,6sleep30-21swapper/300:30:533
36056662142,8sleep30-21swapper/323:15:583
36015082140,2sleep3481ktimers/323:15:003
35119362132,7sleep30-21swapper/321:40:443
34712142132,6sleep30-21swapper/320:55:023
34670452132,7sleep30-21swapper/320:50:523
34537232134,5sleep30-21swapper/320:35:523
36584972123,5sleep30-21swapper/300:10:553
36513472122,5sleep30-21swapper/300:05:013
36505332122,6sleep30-21swapper/300:00:563
36357632122,6sleep30-21swapper/323:46:033
36262972122,5sleep30-21swapper/323:40:013
36254662122,6sleep30-21swapper/323:36:003
36202822122,6sleep30-21swapper/323:30:503
36117712123,5sleep30-21swapper/323:25:453
36104662123,6sleep30-21swapper/323:20:553
35917122122,5sleep30-21swapper/323:05:023
35819722122,6sleep30-21swapper/322:55:433
35618792122,5sleep30-21swapper/322:35:013
35615722122,6sleep30-21swapper/322:31:043
35520992122,5sleep30-21swapper/322:25:023
35443572122,5sleep30-21swapper/322:15:513
35379482122,6sleep30-21swapper/322:10:493
35363562122,6sleep30-21swapper/322:06:003
35318632122,6sleep30-21swapper/322:01:023
35249252122,6sleep30-21swapper/321:55:493
34964952122,8sleep30-21swapper/321:24:273
34383282122,6sleep30-21swapper/320:20:513
34319342122,6sleep30-21swapper/320:15:493
34264852122,6sleep30-21swapper/320:10:443
34064012122,5sleep30-21swapper/319:50:013
34003052122,6sleep30-21swapper/319:40:543
33921232122,6sleep30-21swapper/319:35:493
33814712122,5sleep30-21swapper/319:25:013
33698012122,5sleep30-21swapper/319:15:023
36812992112,5sleep30-21swapper/300:35:433
36714572112,5sleep30-21swapper/300:25:443
36664352111,7sleep30-21swapper/300:20:153
36651342112,5sleep30-21swapper/300:15:563
36415802112,5sleep30-21swapper/323:55:023
35866642112,5sleep30-21swapper/323:00:013
35769562112,5sleep30-21swapper/322:50:023
35754512112,5sleep30-21swapper/322:45:523
35670942112,5sleep30-21swapper/322:40:023
35472922112,5sleep30-21swapper/322:20:043
35165902112,5sleep30-21swapper/321:45:013
35165902112,5sleep30-21swapper/321:45:013
35069012112,5sleep30-21swapper/321:35:023
35023012112,5sleep30-21swapper/321:30:493
34970882112,5sleep30-21swapper/321:25:433
34767932112,5sleep30-21swapper/321:00:473
34619702112,5sleep30-21swapper/320:45:473
34566562112,5sleep30-21swapper/320:40:463
34215212112,5sleep30-21swapper/320:05:423
34165382112,5sleep30-21swapper/320:00:043
34165382112,5sleep30-21swapper/320:00:043
34120582112,5sleep30-21swapper/319:55:473
34043692112,5sleep30-21swapper/319:45:523
33869182112,5sleep30-21swapper/319:30:473
33772202112,5sleep30-21swapper/319:20:483
36364172102,5sleep30-21swapper/323:50:043
35219542102,5sleep30-21swapper/321:50:033
34812792102,4sleep30-21swapper/321:05:053
33650582102,5sleep30-21swapper/319:10:453
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional