You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 07:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot4s.osadl.org (updated Wed Sep 17, 2025 00:44:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
385217721780,3sleep3662-21kworker/3:2+events00:25:573
38438602770,3sleep3662-21kworker/3:2+events00:15:533
3539479996553,8cyclictest2583070-21systemd-journal00:40:001
3539475996139,19cyclictest2583070-21systemd-journal00:20:580
38075072590,3sleep3662-21kworker/3:2+events23:40:543
3539479995950,6cyclictest3507771-21kworker/u8:3+flush-8:020:05:511
3539479995950,6cyclictest3507771-21kworker/u8:3+flush-8:020:05:511
35528722580,3sleep13552878-21tr19:20:511
3539475995832,23cyclictest3679464-21ntpq21:26:010
353947599530,38cyclictest1-21systemd20:40:000
3539479995246,4cyclictest3581360-21kworker/u8:0+flush-8:020:35:011
3539475995245,5cyclictest3839992-21latency_hist00:15:010
3539475995126,22cyclictest3824288-21ntpq23:55:550
3539479994942,5cyclictest3581360-21kworker/u8:0+flush-8:019:55:501
3539479994935,10cyclictest3654480-21ntpq21:00:571
3539479994842,4cyclictest3685147-21kworker/u8:1+flush-8:000:05:011
3539475994819,10cyclictest3604962-21ntpq20:10:550
3539479994742,4cyclictest3685147-21kworker/u8:1+flush-8:022:20:591
3539475994738,6cyclictest3600906-21latency_hist20:10:000
3539475994738,6cyclictest3600906-21latency_hist20:10:000
3539475994726,18cyclictest3543452-21ntpq19:10:550
3539479994640,4cyclictest3555533-21kworker/u8:2+flush-8:023:00:461
3539482994536,6cyclictest2583070-21systemd-journal22:30:572
3539475994537,5cyclictest3810328-21latency_hist23:45:000
3539475994535,7cyclictest3704329-21ntpq21:50:540
3539479994335,5cyclictest3620672-21latency_hist20:30:011
3539475994331,8cyclictest2583070-21systemd-journal21:30:010
3539475994327,14cyclictest3794068-21needreboot23:25:540
3539475994322,19cyclictest3720662-21latency_hist22:15:010
3539479994238,3cyclictest3507771-21kworker/u8:3+flush-8:020:10:461
3539475994235,5cyclictest3685147-21kworker/u8:1+events_unbound00:26:060
3539475994225,15cyclictest3710141-21cron22:00:000
3539475994224,15cyclictest3660324-21latency_hist21:10:010
3539475994220,20cyclictest21855-21cron20:30:000
3539475994217,9cyclictest2583070-21systemd-journal22:40:000
3539482994131,8cyclictest2583070-21systemd-journal22:10:002
3539479994137,3cyclictest3507771-21kworker/u8:3+flush-8:019:11:011
3539479994136,4cyclictest3640120-21kworker/u8:3+flush-8:022:36:031
3539479994136,4cyclictest3640120-21kworker/u8:3+flush-8:022:15:471
3539479994129,9cyclictest3680301-21latency_hist21:30:011
3539482994035,4cyclictest3825434-21kworker/u8:3+flush-8:000:31:042
3539479994033,5cyclictest3805676-21apt-get23:40:451
3539479994030,8cyclictest3834199-21ntpq00:05:571
3539479994030,8cyclictest3754578-21ntpq22:45:581
3539475994031,6cyclictest3635514-21latency_hist20:45:010
3539475994021,17cyclictest3780199-21idleruntime-cro23:15:000
3539479993934,4cyclictest3555533-21kworker/u8:2+flush-8:020:20:461
3539479993931,5cyclictest3800625-21cat23:35:011
3539475993912,7cyclictest3646234-21/usr/sbin/munin20:55:500
3539479993833,4cyclictest3581360-21kworker/u8:0+flush-8:019:50:461
3539479993833,4cyclictest3555533-21kworker/u8:2+flush-8:019:31:051
3539479993832,4cyclictest3640120-21kworker/u8:3+flush-8:022:00:471
3539475993827,8cyclictest3544579-21munin-node19:15:010
3539475993822,14cyclictest3695017-21unixbench_singl21:41:030
3539475993817,4cyclictest3645024-21unixbench_multi20:51:020
3539479993731,4cyclictest3735790-21apt-get22:30:411
3539479993731,4cyclictest3640120-21kworker/u8:3+flush-8:023:06:001
3539479993728,6cyclictest2583070-21systemd-journal23:46:591
3539479993726,7cyclictest2583070-21systemd-journal23:16:581
3539479993724,9cyclictest2583070-21systemd-journal20:01:561
3539475993718,5cyclictest3790065-21latency_hist23:25:000
3539475993718,15cyclictest0-21swapper/021:20:440
3539475993717,2cyclictest0-21swapper/023:40:000
3539475993717,2cyclictest0-21swapper/023:35:000
3539475993716,4cyclictest3769939-21unixbench_multi23:01:010
3539475993715,5cyclictest3844926-21latency_hist00:20:010
3539482993630,4cyclictest3507771-21kworker/u8:3+flush-8:020:11:012
3539482993629,5cyclictest2583070-21systemd-journal19:30:562
3539479993630,4cyclictest3640120-21kworker/u8:3+events_unbound21:45:001
3539479993627,6cyclictest3854790-21latency_hist00:30:011
3539479993627,6cyclictest3539317-21latency_hist19:10:011
3539475993628,5cyclictest3665472-21latency_hist21:15:010
353947599361,9cyclictest3669645-21ntpq21:15:590
353947599361,9cyclictest35528782tr19:20:510
3539475993619,2cyclictest0-21swapper/022:35:010
3539475993618,3cyclictest3581960-21sort19:50:480
3539475993618,2cyclictest0-21swapper/019:50:000
3539475993617,2cyclictest3825652-21cpuspeed00:00:450
37860942350,4sleep10-21swapper/123:20:501
3539482993530,4cyclictest3555533-21kworker/u8:2+flush-8:000:01:032
3539479993531,3cyclictest3685147-21kworker/u8:1+flush-8:023:55:471
3539479993530,4cyclictest3581360-21kworker/u8:0+flush-8:020:15:491
3539475993519,3cyclictest3655015-21sed21:01:040
3539475993516,3cyclictest0-21swapper/022:55:010
3539475993513,6cyclictest691-21lldpd00:05:430
37308462340,1sleep03730849-21acpi22:25:440
3539479993428,4cyclictest3640120-21kworker/u8:3+flush-8:023:25:511
3539479993428,4cyclictest3555533-21kworker/u8:2+flush-8:021:05:481
3539479993425,6cyclictest2583070-21systemd-journal20:59:561
3539479993424,8cyclictest3625855-21/usr/sbin/munin20:35:551
3539479993424,7cyclictest3780196-21latency_hist23:15:011
3539475993417,2cyclictest0-21swapper/019:35:500
3539479993329,3cyclictest3555533-21kworker/u8:2+flush-8:000:20:481
3539479993328,4cyclictest3645106-21kworker/u8:0+flush-8:021:30:481
3539479993328,4cyclictest3507771-21kworker/u8:3+flush-8:019:35:501
3539479993328,3cyclictest3555533-21kworker/u8:2+events_unbound20:50:011
3539479993327,4cyclictest3685147-21kworker/u8:1+events_unbound21:45:471
3539479993326,5cyclictest3670432-21latency_hist21:20:011
3539479993321,9cyclictest3858824-21ntpq00:30:571
353947599335,9cyclictest544-21mta-sts-daemon22:15:430
3539475993318,2cyclictest0-21swapper/023:00:010
3539482993227,4cyclictest3645106-21kworker/u8:0+events_unbound21:30:572
3539479993227,4cyclictest3507771-21kworker/u8:3+flush-8:019:45:581
35391052324,10sleep20-21swapper/219:07:282
37270042310,3sleep3662-21kworker/3:2+events22:20:493
3539482993125,4cyclictest2583070-21systemd-journal19:45:562
3539479993124,5cyclictest3670752-21apt-get21:20:431
3539475993124,5cyclictest3555533-21kworker/u8:2+flush-8:023:15:460
3539482993024,4cyclictest3555533-21kworker/u8:2+events_unbound23:05:012
3539479993025,4cyclictest3640120-21kworker/u8:3+flush-8:022:56:011
3539479993024,4cyclictest3685147-21kworker/u8:1+flush-8:023:50:481
3539479993024,4cyclictest3640120-21kworker/u8:3+events_unbound21:50:521
353947599309,2cyclictest0-21swapper/019:59:050
3539475993025,4cyclictest2637019-21kworker/u8:1+flush-8:019:27:570
3539475993024,4cyclictest3730557-21latency_hist22:25:010
3539482992926,2cyclictest3784682-21unixbench_multi23:16:012
3539482992926,2cyclictest3571617-21fschecks_count19:40:472
3539482992926,2cyclictest3571617-21fschecks_count19:40:462
3539482992925,3cyclictest3625601-21kworker/u8:1+flush-8:020:45:572
3539482992922,4cyclictest3679329-21tail21:25:582
3539479992923,4cyclictest3640120-21kworker/u8:3+events_unbound20:50:461
3539479992922,6cyclictest3685147-21kworker/u8:1+events_unbound22:26:011
3539479992922,5cyclictest3640120-21kworker/u8:3+events_unbound22:10:441
3539479992922,5cyclictest2637019-21kworker/u8:1+flush-8:019:25:551
3539479992919,8cyclictest3759535-21ntpq22:50:551
3539482992824,2cyclictest3710694-21cpuspeed_turbos22:00:452
3539479992824,3cyclictest3706262-21irqrtprio21:55:491
3539479992818,7cyclictest3640120-21kworker/u8:3+flush-8:021:35:571
3539482992723,2cyclictest3790472-21aten2_r5power_v23:25:452
3539479992722,4cyclictest3685147-21kworker/u8:1+events_unbound00:15:421
353947599275,19cyclictest3566112-21kworker/0:1+events19:40:420
353947599275,19cyclictest3566112-21kworker/0:1+events19:40:420
3539475992722,4cyclictest3555533-21kworker/u8:2+events_unbound20:20:540
3539482992621,3cyclictest3700283-21ls21:50:002
3539479992622,3cyclictest3835642-21cut00:10:471
353947599262,23cyclictest0-21swapper/023:05:580
37575122250,6chrt451irq_work/322:50:513
3539482992520,4cyclictest3555533-21kworker/u8:2+events_unbound21:10:562
3539479992517,6cyclictest3640120-21kworker/u8:3+events_unbound23:30:011
3539479992515,8cyclictest3571303-21/usr/sbin/munin19:41:001
3539479992515,8cyclictest3571303-21/usr/sbin/munin19:41:001
35392772252,5sleep00-21swapper/019:09:430
37960782240,6sleep3451irq_work/323:30:513
353948299249,8cyclictest39-21ksoftirqd/221:15:582
3539482992418,4cyclictest3645044-21unixbench_singl20:51:022
353948299236,3cyclictest37308492acpi22:25:442
3539482992319,3cyclictest3581360-21kworker/u8:0+events_unbound20:06:042
3539482992319,3cyclictest3581360-21kworker/u8:0+events_unbound20:06:042
3539482992318,4cyclictest3555533-21kworker/u8:2+flush-8:023:55:512
3539479992318,3cyclictest0-21swapper/120:45:011
353947599235,16cyclictest3600931-21kworker/0:2+events22:00:540
353947599234,17cyclictest3600931-21kworker/0:2+events22:45:500
3539475992319,3cyclictest3863742-21ntp_kernel_pll_00:35:580
3539475992318,3cyclictest3815262-21apt23:50:000
3539475992318,3cyclictest3746524-21sort22:40:490
3539475992318,3cyclictest0-21swapper/020:30:440
3539475992317,4cyclictest3607743-21kthreadcore20:15:530
353948299222,14cyclictest3556456-21apt-get19:25:412
3539482992217,4cyclictest0-21swapper/219:15:002
353948299220,7cyclictest3620046-21smart_sda20:26:022
353947599226,15cyclictest3600931-21kworker/0:2+events21:45:460
353948299212,13cyclictest3655666-21apt-get21:05:452
353948299212,12cyclictest3725919-21apt-get22:20:432
353948299211,7cyclictest37860972cut23:20:502
3539482992117,2cyclictest3820184-21cron23:55:002
3539482992117,2cyclictest3645998-21fschecks_count20:55:472
3539482992117,2cyclictest0-21swapper/220:40:002
3539482992116,3cyclictest3585646-21unixbench_multi19:51:012
3539482992114,5cyclictest2583070-21systemd-journal00:25:002
3539479992116,3cyclictest0-21swapper/121:11:011
3539479992115,5cyclictest3555533-21kworker/u8:2+flush-8:022:40:561
3539479992115,4cyclictest3545578-21kthreadcore19:15:021
3539475992117,2cyclictest0-21swapper/023:50:550
3539475992117,2cyclictest0-21swapper/000:30:470
3539482992016,3cyclictest2637019-21kworker/u8:1+flush-8:019:16:192
3539482992016,2cyclictest0-21swapper/219:25:002
3539482992015,4cyclictest3779776-21unixbench_multi23:11:062
3539482992015,4cyclictest3764929-21unixbench_multi22:56:032
3539482992015,4cyclictest0-21swapper/223:45:002
353948299200,9cyclictest36081681kthreadcore20:15:532
3539479992015,3cyclictest3685147-21kworker/u8:1+events_unbound22:10:001
3539475992018,1cyclictest0-21swapper/021:36:010
3539475992014,4cyclictest3562340-21kthreadcore19:30:540
353948299199,2cyclictest0-21swapper/220:40:502
353948299192,12cyclictest3566380-21apt-get19:35:482
3539482991916,2cyclictest0-21swapper/200:10:002
3539482991915,2cyclictest3721187-21cpuspeed_turbos22:15:452
3539482991915,2cyclictest0-21swapper/223:35:002
3539482991915,2cyclictest0-21swapper/222:50:532
3539482991914,3cyclictest0-21swapper/200:35:472
353947599192,16cyclictest0-21swapper/020:01:000
3539475991917,1cyclictest0-21swapper/020:46:000
3539475991914,3cyclictest0-21swapper/022:10:000
35392552193,9sleep30-21swapper/319:09:263
353948299189,7cyclictest3594106-21tr20:00:522
3539482991814,3cyclictest0-21swapper/221:40:002
3539482991814,2cyclictest3625167-21unixbench_multi20:31:012
3539482991814,2cyclictest0-21swapper/222:10:462
3539482991814,2cyclictest0-21swapper/221:41:022
3539482991813,4cyclictest3640120-21kworker/u8:3+flush-8:022:35:312
3539482991810,6cyclictest3655269-21sshd21:03:562
38636162170,3sleep3662-21kworker/3:2+events00:35:563
353948299179,6cyclictest3805359-21awk23:40:002
353948299178,7cyclictest3671185-21awk21:20:482
3539482991713,3cyclictest0-21swapper/222:41:022
3539482991713,2cyclictest0-21swapper/222:46:282
353948299168,6cyclictest3813708-21tr23:45:502
353948299168,6cyclictest3706298-21sed21:55:492
353948299167,7cyclictest3837166-21cut00:10:522
3539482991612,2cyclictest0-21swapper/220:20:442
3539482991611,3cyclictest0-21swapper/221:50:522
353948299161,10cyclictest3589576-21kthreadcore19:55:562
353948299160,3cyclictest3844753-21sshd00:17:592
353948299156,6cyclictest3773137-21cut23:05:502
3539482991510,3cyclictest371rcuc/200:25:562
38251972143,6sleep30-21swapper/300:00:023
37517452145,5sleep30-21swapper/322:45:543
36843902132,7sleep30-21swapper/321:30:553
36355012132,7sleep30-21swapper/320:45:003
36150772132,7sleep30-21swapper/320:21:013
35611022132,5sleep30-21swapper/319:30:013
38573472122,6sleep30-21swapper/300:30:553
38248312122,6sleep30-21swapper/323:56:043
38143532122,6sleep30-21swapper/323:45:533
37851392122,5sleep30-21swapper/323:20:013
37826362122,6sleep30-21swapper/323:15:523
37708652122,6sleep30-21swapper/323:05:443
37604452122,5sleep30-21swapper/322:55:013
37354762122,5sleep30-21swapper/322:30:013
37342752122,6sleep30-21swapper/322:25:563
37167682122,7sleep30-21swapper/322:10:473
37121252122,6sleep30-21swapper/322:00:523
37101402122,6sleep30-21swapper/322:00:003
37002742122,5sleep30-21swapper/321:50:013
36905212122,5sleep30-21swapper/321:40:023
36857462122,6sleep30-21swapper/321:35:443
36757642122,6sleep30-21swapper/321:25:493
36645982122,6sleep30-21swapper/321:10:563
36509462122,6sleep30-21swapper/321:00:493
36258822124,5sleep30-21swapper/320:35:423
36255682122,5sleep30-21swapper/320:35:013
36157472122,6sleep30-21swapper/320:25:023
36011822122,6sleep30-21swapper/320:10:413
36006562122,6sleep30-21swapper/320:06:563
36006562122,6sleep30-21swapper/320:06:563
35913392122,6sleep30-21swapper/320:00:423
35836802122,6sleep30-21swapper/319:50:513
35770022122,6sleep30-21swapper/319:45:503
35750702120,8sleep30-21swapper/319:40:563
35750702120,8sleep30-21swapper/319:40:563
35604962122,6sleep30-21swapper/319:25:593
35513012122,5sleep30-21swapper/319:20:023
38458222112,5sleep30-21swapper/300:20:523
38375052112,5sleep30-21swapper/300:10:533
38342432112,5sleep30-21swapper/300:05:593
38154582112,5sleep30-21swapper/323:50:023
38006912112,5sleep30-21swapper/323:35:423
37950022111,6sleep30-21swapper/323:30:003
37792802112,5sleep30-21swapper/323:10:583
37666682112,5sleep30-21swapper/323:00:513
37459502112,5sleep30-21swapper/322:40:453
37413672112,5sleep30-21swapper/322:35:513
37209682112,5sleep30-21swapper/322:15:453
37152982112,5sleep30-21swapper/322:05:013
36959012112,5sleep30-21swapper/321:45:473
36716382112,5sleep30-21swapper/321:20:513
36655892112,5sleep30-21swapper/321:15:023
36556402112,5sleep30-21swapper/321:05:433
36465942112,5sleep30-21swapper/320:55:523
36405822112,5sleep30-21swapper/320:50:023
36404122112,5sleep30-21swapper/320:49:423
36066022112,5sleep30-21swapper/320:15:513
35662992112,5sleep30-21swapper/319:35:023
35465432112,5sleep30-21swapper/319:15:443
35396032112,5sleep30-21swapper/319:10:023
35868432102,5sleep30-21swapper/319:55:533
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional