You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 01:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Fri May 15, 2026 00:43:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3976509995546,5cyclictest4064811-21ssh22:15:011
3976509995339,7cyclictest0-21swapper/122:41:001
397651099504,34cyclictest4070149-21ssh22:20:512
3976509995021,11cyclictest4068874-21munin-run22:20:011
3976508994942,5cyclictest421-21dbus-daemon19:20:010
3976508994940,6cyclictest4085066-21systemd22:40:010
3976509994838,5cyclictest0-21swapper/122:25:561
3976510994740,4cyclictest3986994-21latency_hist19:45:012
3976509994734,10cyclictest4025133-21/usr/sbin/munin21:25:521
3976509994635,6cyclictest439-21systemd-logind19:50:011
3976510994534,5cyclictest0-21swapper/223:35:552
3976509994535,6cyclictest4038595-21grep21:41:051
3976509994535,6cyclictest1055-21mta-sts-daemon20:20:551
3976509994435,5cyclictest4109337-21systemd-run23:10:021
3976509994336,4cyclictest4036600-21munin-plugin-st21:40:011
3976508994330,9cyclictest4130735-21needreboot23:35:510
3976509994234,5cyclictest4009511-21systemd-run21:00:011
3976509994233,5cyclictest4135527-21unixbench_singl23:41:081
3976509994232,5cyclictest0-21swapper/122:55:511
3976509994231,7cyclictest2127839-21cupsd23:23:011
3976509994218,14cyclictest4147122-21ssh23:56:001
3976509994216,20cyclictest4041920-21ssh21:45:561
3976509994214,10cyclictest4079047-21ssh22:31:181
3976509994214,10cyclictest4079047-21ssh22:31:181
397650899424,21cyclictest4138858-21sh23:45:510
3976508994232,6cyclictest4072917-21kworker/u8:12+xprtiod22:45:000
3976508994225,11cyclictest12350irq/32-eth023:30:530
3976510994133,5cyclictest4054586-21grep22:01:062
3976510994133,5cyclictest4034306-21timerandwakeup21:36:052
3976509994133,5cyclictest3985536-21latency_hist19:40:021
3976509994132,6cyclictest1-21systemd20:00:011
3976509994116,9cyclictest4083058-21sh22:36:141
3976508994134,4cyclictest4097154-21systemd-run22:55:010
3976508994132,7cyclictest4158425-21/usr/sbin/munin00:10:470
3976508994132,7cyclictest4158425-21/usr/sbin/munin00:10:470
3976508994130,8cyclictest4021123-21/usr/sbin/munin21:20:490
3976510994032,5cyclictest4076936-21ssh22:30:012
3976510994030,6cyclictest420-21avahi-daemon23:58:382
3976509994031,6cyclictest4179626-21processes00:36:041
3976509994031,6cyclictest4029798-21sh21:30:521
3976509994030,8cyclictest3992995-21/usr/sbin/munin20:05:011
3976509994030,7cyclictest4151625-21sed00:01:041
3976509994030,7cyclictest4151625-21sed00:01:041
3976509994018,15cyclictest4053944-21ssh22:00:511
3976509994016,8cyclictest4172468-21ssh00:27:001
3976508994031,6cyclictest2127843-21cups-browsed23:17:540
3976508994026,10cyclictest12350irq/32-eth022:10:590
3976508994025,13cyclictest0-21swapper/021:40:020
3976510993931,5cyclictest420-21avahi-daemon23:15:082
3976510993930,5cyclictest4089070-21cron22:45:002
3976510993929,5cyclictest0-21swapper/222:15:552
3976510993926,9cyclictest421-21dbus-daemon22:38:162
3976509993929,6cyclictest2127843-21cups-browsed22:06:311
3976509993924,8cyclictest4010342-21ntpq21:00:511
3976509993915,9cyclictest4071317-21sh22:21:591
3976509993915,9cyclictest4011656-21kernelversion21:05:471
3976509993915,7cyclictest4047207-21sh21:52:221
3976509993915,7cyclictest4047207-21sh21:52:221
3976510993831,4cyclictest3991509-21kworker/u8:6-xprtiod20:26:292
3976510993831,3cyclictest0-21swapper/223:50:562
3976510993830,5cyclictest4137566-21cron23:45:012
3976510993829,6cyclictest420-21avahi-daemon22:05:482
3976510993829,3cyclictest0-21swapper/200:30:582
3976510993828,7cyclictest4174031-21ssh00:30:002
3976510993826,9cyclictest4178647-21/usr/sbin/munin00:36:022
3976509993833,2cyclictest0-21swapper/120:40:011
3976509993830,5cyclictest4139494-21unixbench_multi23:46:051
3976509993826,9cyclictest4117401-21munin-run23:20:001
3976509993825,9cyclictest4049709-21latency21:55:481
3976509993825,8cyclictest3995337-21ntpq20:10:491
3976509993824,8cyclictest3993958-21ntpq20:05:581
3976509993822,12cyclictest4128062-21ssh23:32:151
3976509993822,10cyclictest4102709-21ps23:01:001
3976509993821,12cyclictest3981957-21ntpq19:25:551
3976509993813,8cyclictest3980709-21cat19:21:011
3976509993812,9cyclictest3999840-21ntpq20:25:491
3976508993832,4cyclictest4171274-21sh00:25:520
3976508993829,6cyclictest4042376-21timerwakeupswit21:46:060
3976508993823,12cyclictest4110226-21fschecks_count23:10:490
3976510993730,4cyclictest4032568-21kworker/u8:7-xprtiod23:30:592
3976510993729,5cyclictest4021770-21needreboot21:20:502
3976510993728,6cyclictest420-21avahi-daemon00:17:242
3976510993728,6cyclictest2127843-21cups-browsed23:01:012
3976510993728,6cyclictest0-21swapper/219:40:012
3976509993729,5cyclictest4006498-21true20:50:011
3976509993728,6cyclictest4175376-21ntp_kernel_pll_00:30:541
3976509993715,6cyclictest4132600-21ssh23:38:131
397650999371,28cyclictest4090421-21ssh22:45:571
3976508993730,4cyclictest4151709-21unixbench_singl00:01:040
3976508993730,4cyclictest4151709-21unixbench_singl00:01:040
3976508993722,8cyclictest420-21avahi-daemon23:05:030
3976510993629,4cyclictest4017733-21missed_timers21:15:522
3976510993628,5cyclictest421-21dbus-daemon22:50:012
3976510993628,5cyclictest4157842-21cron00:10:002
3976510993628,5cyclictest4157842-21cron00:10:002
3976510993628,5cyclictest4113352-21sh23:15:012
3976510993628,5cyclictest3992018-21fschecks_time20:00:462
3976510993627,6cyclictest2127843-21cups-browsed21:56:592
397651099362,28cyclictest4045990-21ssh21:50:582
397651099362,28cyclictest4045990-21ssh21:50:582
397650999366,21cyclictest428-21mta-sts-daemon00:11:001
397650999366,21cyclictest428-21mta-sts-daemon00:11:001
3976509993629,4cyclictest0-21swapper/119:50:551
3976509993629,4cyclictest0-21swapper/119:50:551
3976509993625,7cyclictest3986456-21ntpq19:40:561
3976509993624,9cyclictest3996788-21needreboot20:15:491
3976509993624,8cyclictest4122748-21ntpq23:25:561
3976509993623,7cyclictest4167381-21ntpq00:20:561
3976509993622,9cyclictest4021974-21ntpq21:20:571
3976509993618,4cyclictest0-21swapper/123:50:481
3976509993615,7cyclictest3977412-21ntpq19:10:531
3976508993630,4cyclictest4149765-21sendmail00:00:010
3976508993627,6cyclictest4149711-21cups-browsed00:17:020
3976508993627,6cyclictest2127839-21cupsd23:27:380
3976508993626,7cyclictest4025604-21kernelversion21:25:520
3976508993620,13cyclictest3991527-21latency_hist20:00:010
3976508993617,10cyclictest12350irq/32-eth022:30:530
3976508993617,10cyclictest12350irq/32-eth022:30:530
397650899361,27cyclictest4134850-21ssh23:40:530
397650899361,26cyclictest4175380-21ssh00:30:540
3976510993529,3cyclictest0-21swapper/223:25:002
3976510993528,3cyclictest0-21swapper/223:25:592
3976510993527,6cyclictest4101171-21sendmail23:00:012
3976510993527,5cyclictest4040542-21cron21:45:012
3976510993526,6cyclictest3980993-21cron19:25:012
397650999359,10cyclictest4154112-21rm00:05:101
397650999359,10cyclictest4154112-21rm00:05:101
3976509993522,9cyclictest3979130-21switchtime19:16:021
3976509993520,12cyclictest4007368-21ntpq20:50:521
3976509993514,7cyclictest4165584-21diskmemload00:19:201
3976508993529,4cyclictest4182106-21sendmail00:40:010
3976508993529,4cyclictest4032593-21latency_hist21:35:020
3976508993527,6cyclictest4000493-21cut20:30:020
3976508993527,5cyclictest421-21dbus-daemon00:06:450
3976508993527,5cyclictest421-21dbus-daemon00:06:450
3976508993527,5cyclictest4017145-21/usr/sbin/munin21:15:430
3976508993527,5cyclictest2127843-21cups-browsed21:54:020
3976508993527,5cyclictest2127843-21cups-browsed21:54:020
3976508993525,7cyclictest151ktimers/020:05:540
3976508993523,9cyclictest4056729-21systemd-run22:05:010
3976510993427,3cyclictest0-21swapper/200:15:012
3976510993427,3cyclictest0-21swapper/200:15:012
3976510993426,5cyclictest4028627-21latency_hist21:30:022
3976510993426,4cyclictest0-21swapper/222:50:532
3976510993425,4cyclictest0-21swapper/222:10:522
397650999348,19cyclictest4094414-21ssh22:50:541
3976509993423,7cyclictest3983456-21ntpq19:30:551
3976509993422,7cyclictest4004449-21ntpq20:40:551
3976509993421,9cyclictest4013667-21missed_timers21:10:501
3976509993419,10cyclictest4112225-21ssh23:12:521
3976509993412,7cyclictest4017114-21ssh21:15:401
3976508993428,4cyclictest4037520-21fschecks_time21:40:450
3976508993427,5cyclictest4119407-21unixbench_singl23:21:050
3976508993427,5cyclictest261-21systemd-journal20:40:010
3976508993423,8cyclictest4065883-21latency22:15:480
3976510993325,5cyclictest4006504-21systemd-run20:50:012
3976510993324,6cyclictest4008084-21latency_hist20:55:022
3976509993323,7cyclictest4001347-21ntpq20:30:511
3976508993325,5cyclictest4149711-21cups-browsed00:24:000
3976508993324,2cyclictest0-21swapper/023:00:010
3976508993323,7cyclictest4014107-21sendmail_mailtr21:11:030
3976508993322,9cyclictest4011511-21fschecks_count21:05:450
397651099323,23cyclictest4138946-21ssh23:45:542
3976510993225,4cyclictest3977145-21kernelversion19:10:472
3976510993225,4cyclictest1056-21mta-sts-daemon20:40:542
3976510993225,3cyclictest0-21swapper/221:31:002
3976510993220,9cyclictest4151360-21mailstats00:01:002
3976510993220,9cyclictest4151360-21mailstats00:01:002
3976508993226,3cyclictest0-21swapper/023:51:050
3976508993225,4cyclictest4060763-21systemd-run22:10:010
397650899322,23cyclictest659-21snmpd20:55:530
3976508993221,9cyclictest4076935-21idleruntime-cro22:30:010
3976510993123,3cyclictest0-21swapper/221:10:592
3976510993120,8cyclictest4077991-21kernelversion22:30:482
3976510993120,8cyclictest4077991-21kernelversion22:30:482
3976508993125,4cyclictest4011010-21systemd-run21:05:010
3976508993124,5cyclictest4007909-21sed20:51:030
3976508993123,5cyclictest420-21avahi-daemon21:55:340
3976508993123,5cyclictest2127843-21cups-browsed22:46:420
3976508993121,8cyclictest3998034-21fschecks_time20:20:440
3976508993120,8cyclictest0-21swapper/019:40:010
3976508993117,11cyclictest4106019-21aten2.4-expect23:05:430
3976510993024,3cyclictest0-21swapper/220:40:012
3976510993023,4cyclictest4107219-21sed23:06:052
3976510993020,7cyclictest4167705-21timerandwakeup00:21:042
3976508993019,3cyclictest0-21swapper/022:21:030
3976510992923,3cyclictest0-21swapper/221:00:012
397650899292,21cyclictest659-21snmpd20:45:510
3976510992822,3cyclictest0-21swapper/221:47:032
3976510992820,4cyclictest4001998-21systemd-run20:35:012
3976510992819,5cyclictest3978289-21aten_r2power_en19:15:412
3976508992822,4cyclictest3977020-21grep19:10:450
3976510992721,3cyclictest0-21swapper/219:55:012
3976510992721,3cyclictest0-21swapper/219:55:012
3976510992720,4cyclictest3990794-21needreboot19:55:532
3976510992716,8cyclictest3982685-21latency_hist19:30:032
3976509992716,8cyclictest3976521-21munin-run19:10:021
3976508992723,2cyclictest0-21swapper/019:45:020
3976510992621,2cyclictest0-21swapper/220:20:012
3976508992619,4cyclictest3988852-21aten2.4-expect19:50:430
3976508992619,4cyclictest3988852-21aten2.4-expect19:50:430
3976508992619,3cyclictest0-21swapper/020:10:530
3976508992618,6cyclictest3983303-21needreboot19:30:480
3976508992616,2cyclictest0-21swapper/020:40:420
3976510992519,3cyclictest0-21swapper/221:10:012
3976510992518,4cyclictest427-21polkitd19:48:102
3976510992517,5cyclictest261-21systemd-journal21:05:012
3976508992520,3cyclictest1051-21mta-sts-daemon19:49:160
3976508992518,5cyclictest2127843-21cups-browsed20:30:550
3976508992518,4cyclictest0-21swapper/020:15:560
3976508992512,11cyclictest0-21swapper/019:25:020
397651099242,16cyclictest3993965-21ntp_states20:06:002
3976510992416,5cyclictest1-21systemd19:10:012
3976510992319,2cyclictest0-21swapper/219:35:022
3976510992318,2cyclictest0-21swapper/220:15:022
3976510992316,4cyclictest3997961-21cpuspeed20:20:422
3976508992318,2cyclictest0-21swapper/019:10:010
397650899222,15cyclictest3992374-21ntpq20:00:530
397651199215,10cyclictest0-21swapper/322:46:023
397651199212,11cyclictest0-21swapper/323:15:573
397651199212,11cyclictest0-21swapper/300:35:563
3976508992115,3cyclictest0-21swapper/019:25:530
397651199203,11cyclictest0-21swapper/300:20:523
397651199203,10cyclictest0-21swapper/323:45:553
397651199202,10cyclictest0-21swapper/323:10:553
397651199202,10cyclictest0-21swapper/321:00:513
397651199193,9cyclictest0-21swapper/322:25:493
397651199193,11cyclictest0-21swapper/322:50:593
397651199192,11cyclictest0-21swapper/323:50:593
397651199192,11cyclictest0-21swapper/323:40:563
397651199192,11cyclictest0-21swapper/323:35:533
397651199192,11cyclictest0-21swapper/322:30:533
397651199192,11cyclictest0-21swapper/322:30:533
397651199192,11cyclictest0-21swapper/321:55:513
397651199192,10cyclictest0-21swapper/322:15:553
397651199185,8cyclictest0-21swapper/321:45:573
397651199183,9cyclictest0-21swapper/322:10:513
397651199183,9cyclictest0-21swapper/300:05:593
397651199183,9cyclictest0-21swapper/300:05:593
397651199183,10cyclictest0-21swapper/323:01:003
397651199183,10cyclictest0-21swapper/321:10:543
397651199182,11cyclictest0-21swapper/300:30:583
397651199182,10cyclictest0-21swapper/321:40:543
397651199182,10cyclictest0-21swapper/300:15:513
397651199181,10cyclictest0-21swapper/322:55:593
397651199173,9cyclictest0-21swapper/321:50:523
397651199173,9cyclictest0-21swapper/321:50:523
397651199173,8cyclictest0-21swapper/319:35:513
397651199173,10cyclictest0-21swapper/323:55:453
397651199173,10cyclictest0-21swapper/321:30:563
397651199173,10cyclictest0-21swapper/321:25:013
397651199172,9cyclictest0-21swapper/323:05:583
397651199172,9cyclictest0-21swapper/322:35:543
397651199172,7cyclictest0-21swapper/320:30:503
397651199172,11cyclictest0-21swapper/322:00:503
397651199172,10cyclictest0-21swapper/322:20:503
397651199172,10cyclictest0-21swapper/322:05:573
397651199172,10cyclictest0-21swapper/300:25:503
397651199172,10cyclictest0-21swapper/300:10:543
397651199172,10cyclictest0-21swapper/300:10:543
397651199171,11cyclictest0-21swapper/300:00:533
397651199171,11cyclictest0-21swapper/300:00:533
397651199163,9cyclictest0-21swapper/321:25:553
397651199162,9cyclictest0-21swapper/323:30:533
397651199162,10cyclictest0-21swapper/322:40:533
397651199162,10cyclictest0-21swapper/321:15:543
397651199161,7cyclictest0-21swapper/320:25:503
397651199154,7cyclictest0-21swapper/320:45:513
397651199153,8cyclictest0-21swapper/323:20:423
397651199153,6cyclictest0-21swapper/319:30:493
397651199152,9cyclictest0-21swapper/323:25:493
397651199152,9cyclictest0-21swapper/321:35:433
397651199152,8cyclictest0-21swapper/321:10:013
397651199152,8cyclictest0-21swapper/320:50:523
397651199152,8cyclictest0-21swapper/319:10:523
397651199152,7cyclictest0-21swapper/319:50:523
397651199152,7cyclictest0-21swapper/319:50:523
397651199152,6cyclictest0-21swapper/319:45:553
397651199152,6cyclictest0-21swapper/319:25:503
397651199151,9cyclictest0-21swapper/319:15:513
397651199151,6cyclictest0-21swapper/320:20:493
397651199143,7cyclictest0-21swapper/320:55:533
397651199143,5cyclictest0-21swapper/320:15:523
397651199142,7cyclictest0-21swapper/319:40:583
397651199141,7cyclictest0-21swapper/320:10:493
397651199141,7cyclictest0-21swapper/320:05:523
397651199141,6cyclictest0-21swapper/319:20:503
397651199132,8cyclictest0-21swapper/320:35:463
397651199131,8cyclictest0-21swapper/319:56:063
397651199131,7cyclictest0-21swapper/320:00:513
397651199122,7cyclictest0-21swapper/320:40:513
397651199121,7cyclictest0-21swapper/319:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional