You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-26 - 01:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Tue May 26, 2026 00:43:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
752333995542,7cyclictest0-21swapper/223:40:572
752331995344,7cyclictest812479-21munin-run21:40:000
752333995142,5cyclictest0-21swapper/200:20:572
752332995142,5cyclictest930328-21latency_hist00:05:021
752331995136,12cyclictest774849-21systemd-run20:25:010
75233299503,38cyclictest907200-21ssh23:35:591
752331994943,3cyclictest0-21swapper/023:15:530
752333994740,4cyclictest752342-21systemd-run19:10:012
752333994739,5cyclictest1-21systemd20:10:012
752333994737,6cyclictest828753-21systemd-run22:00:012
752333994736,7cyclictest1054-21mta-sts-daemon00:10:572
752331994739,5cyclictest1-21systemd00:15:020
752332994640,3cyclictest0-21swapper/100:25:551
752332994640,3cyclictest0-21swapper/100:25:551
752333994535,6cyclictest420-21avahi-daemon21:46:072
752332994536,5cyclictest755369-21latency_hist19:20:021
752333994436,5cyclictest948490-21unixbench_singl00:26:122
752333994436,5cyclictest948490-21unixbench_singl00:26:122
752333994436,5cyclictest889601-21systemd-run23:15:012
752333994436,4cyclictest0-21swapper/222:20:592
752333994436,4cyclictest0-21swapper/222:20:592
752333994434,6cyclictest802409-21unixbench_singl21:26:132
752333994434,6cyclictest802409-21unixbench_singl21:26:132
752331994337,4cyclictest767340-21systemd-run20:00:010
752331994337,4cyclictest767340-21systemd-run20:00:010
752331994337,4cyclictest1-21systemd23:15:020
752331994332,8cyclictest911660-21timerandwakeup23:41:090
752332994236,3cyclictest0-21swapper/120:40:011
752331994228,11cyclictest824634-21cron21:55:010
752332994131,6cyclictest0-21swapper/100:21:071
75233299411,6cyclictest171rcu_preempt22:56:031
75233299411,6cyclictest171rcu_preempt22:56:031
752331994132,6cyclictest420-21avahi-daemon23:59:080
752331994131,6cyclictest0-21swapper/021:26:070
752331994131,6cyclictest0-21swapper/021:26:070
752333994031,5cyclictest0-21swapper/221:21:052
752332994032,5cyclictest897694-21basename23:25:011
752332994032,5cyclictest765803-21latency_hist19:55:001
752332994032,5cyclictest765803-21latency_hist19:55:001
752332994032,5cyclictest548-21gdbus20:11:051
752332994029,8cyclictest885512-21idleruntime-cro23:10:001
752332994028,9cyclictest923728-21mailstats23:56:091
752332994028,9cyclictest838698-21timerwakeupswit22:11:091
752331994034,3cyclictest0-21swapper/023:06:040
752331994028,8cyclictest955899-21ssh00:35:580
752333993931,4cyclictest788314-21munin-plugin-st21:10:002
752333993931,4cyclictest788314-21munin-plugin-st21:10:002
752333993929,6cyclictest914561-21acpi23:45:482
752332993931,5cyclictest806410-21sed21:31:111
752332993931,5cyclictest806410-21sed21:31:111
752332993930,5cyclictest873311-21cron22:55:011
752332993930,5cyclictest801692-21needreboot21:25:581
752332993930,5cyclictest801692-21needreboot21:25:581
752331993932,5cyclictest0-21swapper/022:35:020
752331993930,6cyclictest756819-21cron19:25:010
75233199392,31cyclictest943849-21ssh00:21:030
752333993831,5cyclictest752330-21cyclictest23:00:032
752333993831,5cyclictest752330-21cyclictest23:00:032
752333993830,5cyclictest942396-21sendmail00:20:012
752333993830,5cyclictest767296-21cron20:00:002
752333993830,5cyclictest767296-21cron20:00:002
752333993828,7cyclictest824635-21cron21:55:012
752332993829,6cyclictest421-21dbus-daemon22:04:401
752332993829,5cyclictest922047-21systemd-run23:55:011
752332993829,5cyclictest0-21swapper/122:05:591
752332993825,10cyclictest866968-21switchtime22:46:101
752331993830,5cyclictest421-21dbus-daemon22:05:020
752331993820,13cyclictest12350irq/32-eth023:51:020
75233199381,32cyclictest817882-21ssh21:45:580
752333993730,4cyclictest1055-21mta-sts-daemon20:21:052
752333993729,5cyclictest926155-21modprobe00:00:012
752333993729,5cyclictest261-21systemd-journal20:45:012
752333993728,5cyclictest849902-21cpuspeed22:25:502
752333993728,5cyclictest849902-21cpuspeed22:25:502
752333993727,4cyclictest0-21swapper/223:16:002
752333993725,8cyclictest927282-21latency00:00:552
75233399372,27cyclictest809772-21ssh21:35:582
75233399372,26cyclictest261-21systemd-journal21:31:052
75233399372,26cyclictest261-21systemd-journal21:31:052
752332993731,3cyclictest0-21swapper/122:15:521
752332993731,3cyclictest0-21swapper/122:15:521
752332993729,5cyclictest942381-21idleruntime-cro00:20:011
752332993729,5cyclictest421-21dbus-daemon23:45:011
752332993728,5cyclictest890550-21gltestperf23:15:521
752332993722,4cyclictest0-21swapper/122:41:001
752331993728,6cyclictest836961-21latency_hist22:10:020
752331993726,9cyclictest12350irq/32-eth022:46:040
752331993724,10cyclictest0-21swapper/022:30:010
752331993724,10cyclictest0-21swapper/022:30:010
752331993724,10cyclictest0-21swapper/000:15:530
752331993711,5cyclictest14-21ksoftirqd/023:31:000
752333993629,3cyclictest0-21swapper/222:00:582
752333993628,3cyclictest0-21swapper/200:06:062
752333993624,8cyclictest854896-21timerandwakeup22:31:102
75233399362,27cyclictest838165-21ssh22:10:572
752332993630,3cyclictest0-21swapper/121:36:051
752332993624,9cyclictest931997-21switchtime00:06:171
752331993629,4cyclictest951240-21aten_r2power_fr00:30:480
752331993628,5cyclictest421-21dbus-daemon20:05:010
752331993628,5cyclictest421-21dbus-daemon20:05:010
752331993627,6cyclictest765412-21mailstats19:51:060
752331993627,6cyclictest765412-21mailstats19:51:060
752331993626,7cyclictest828698-21cron22:00:010
752331993624,9cyclictest695-21cron19:45:000
75233399353,24cyclictest842211-21ssh22:15:562
75233399353,24cyclictest842211-21ssh22:15:562
752333993528,4cyclictest518-21rs:main2
752333993527,5cyclictest956292-21switchtime00:36:092
75233399351,27cyclictest895055-21ssh23:21:002
752332993529,3cyclictest0-21swapper/100:35:001
752332993528,4cyclictest783335-21processes20:51:051
752332993525,6cyclictest3179076-21cups-browsed22:27:131
752332993525,6cyclictest3179076-21cups-browsed22:27:131
752332993518,13cyclictest271ktimers/123:30:581
75233199354,4cyclictest842375-21ntp_states22:16:020
75233199354,4cyclictest842375-21ntp_states22:16:020
752331993530,3cyclictest798299-21timerwakeupswit21:21:160
752331993529,3cyclictest0-21swapper/022:55:010
752331993528,5cyclictest1-21systemd20:31:050
752331993528,4cyclictest861205-21cat22:40:010
752331993528,4cyclictest796644-21latency_hist21:20:030
752331993526,6cyclictest808561-21latency_hist21:35:020
752331993526,6cyclictest808561-21latency_hist21:35:020
752331993523,8cyclictest420-21avahi-daemon00:06:030
752331993514,5cyclictest14-21ksoftirqd/022:22:580
752331993514,5cyclictest14-21ksoftirqd/022:22:580
752331993513,5cyclictest14-21ksoftirqd/023:26:000
75233199351,28cyclictest874798-21sh22:56:020
75233199351,28cyclictest874798-21sh22:56:020
752333993427,4cyclictest765365-21processes19:51:042
752333993427,4cyclictest765365-21processes19:51:042
752333993426,4cyclictest755334-21sendmail19:20:012
752333993425,5cyclictest906742-21sort23:35:512
752332993428,3cyclictest0-21swapper/121:45:011
752332993426,4cyclictest0-21swapper/120:55:581
752332993426,4cyclictest0-21swapper/120:55:581
752332993425,5cyclictest861152-21idleruntime-cro22:40:001
752332993422,8cyclictest788317-21diskmemload23:00:591
75233299342,27cyclictest935734-21sh00:11:011
75233199342,24cyclictest878931-21ssh23:01:040
752331993414,5cyclictest14-21ksoftirqd/000:00:590
752333993325,5cyclictest861133-21ssh22:40:002
752332993328,2cyclictest0-21swapper/120:20:011
752332993327,3cyclictest0-21swapper/122:31:111
752332993326,3cyclictest0-21swapper/123:10:571
752332993325,4cyclictest0-21swapper/100:35:591
752332993324,6cyclictest767347-21unin-run20:00:011
752332993324,6cyclictest767347-21unin-run20:00:011
752331993320,11cyclictest0-21swapper/021:40:530
752331993310,11cyclictest14-21ksoftirqd/021:11:030
752333993226,4cyclictest0-21swapper/222:41:082
752333993225,3cyclictest0-21swapper/222:10:012
752333993224,5cyclictest781357-21gltestperf20:45:532
752333993223,7cyclictest884910-21sh23:08:522
752333993223,6cyclictest3179076-21cups-browsed20:00:542
752333993223,6cyclictest3179076-21cups-browsed20:00:542
75233399321,23cyclictest878930-21sh23:01:042
752332993224,4cyclictest818504-21unixbench_multi21:46:111
752332993223,6cyclictest914602-21aten_r2power_cu23:45:481
752332993223,6cyclictest0-21swapper/123:26:041
75233199329,4cyclictest14-21ksoftirqd/023:20:590
75233199322,22cyclictest907171-21ssh23:35:580
752331993215,6cyclictest780754-21unixbench_singl20:41:110
752331993214,14cyclictest862352-21sed22:40:560
752331993210,6cyclictest14-21ksoftirqd/023:45:410
752331993210,11cyclictest14-21ksoftirqd/000:26:020
752331993210,11cyclictest14-21ksoftirqd/000:26:020
75233399314,20cyclictest903062-21ssh23:30:562
752333993125,3cyclictest0-21swapper/219:40:012
752333993123,5cyclictest793935-21/usr/sbin/munin21:16:062
752333993122,6cyclictest3179073-21cupsd22:51:052
752332993125,3cyclictest0-21swapper/121:01:081
752332993125,3cyclictest0-21swapper/121:01:081
752332993123,5cyclictest821245-21tail21:50:471
752332993123,5cyclictest764319-21munin-plugin-st19:50:001
752332993123,5cyclictest764319-21munin-plugin-st19:50:001
752332993123,4cyclictest0-21swapper/121:56:361
752332993123,3cyclictest0-21swapper/121:11:001
752332993122,6cyclictest768741-21unixbench_singl20:01:131
752332993122,6cyclictest768741-21unixbench_singl20:01:131
752332993122,5cyclictest788317-21diskmemload22:21:051
752332993122,5cyclictest788317-21diskmemload22:21:051
75233199318,4cyclictest14-21ksoftirqd/019:26:010
75233199311,26cyclictest838174-21ntpq22:10:570
75233399307,17cyclictest899126-21rm23:26:012
75233399303,19cyclictest3179076-21cups-browsed20:31:012
752333993024,3cyclictest0-21swapper/222:46:112
752333993023,4cyclictest786763-21unixbench_singl21:01:112
752333993023,4cyclictest786763-21unixbench_singl21:01:112
752333993022,5cyclictest920015-21head23:51:122
752333993022,5cyclictest813157-21head21:40:472
75233299306,16cyclictest428-21mta-sts-daemon21:15:591
752332993022,4cyclictest761308-21idleruntime-cro19:40:011
752332993019,7cyclictest788317-21diskmemload21:21:061
75233199309,3cyclictest14-21ksoftirqd/021:06:080
75233199309,3cyclictest14-21ksoftirqd/021:06:080
75233199308,3cyclictest151ktimers/021:01:030
75233199308,3cyclictest151ktimers/021:01:030
75233199304,8cyclictest14-21ksoftirqd/020:15:540
752331993020,7cyclictest771800-21cron20:15:010
752331993010,3cyclictest14-21ksoftirqd/019:35:570
752333992922,4cyclictest755869-21fschecks_time19:20:512
752333992921,5cyclictest792371-21grep21:15:002
75233399290,4cyclictest171rcu_preempt20:56:022
75233399290,4cyclictest171rcu_preempt20:56:022
75233199298,4cyclictest14-21ksoftirqd/019:30:530
752331992922,4cyclictest753816-21grep19:15:010
752331992921,5cyclictest1056-21mta-sts-daemon20:05:580
75233399282,21cyclictest951840-21ssh00:30:572
752333992821,4cyclictest780833-21kworker/u8:14-nfsiod20:55:012
752333992820,5cyclictest753806-21cron19:15:012
752332992821,4cyclictest0-21swapper/120:05:531
752332992820,5cyclictest774756-21unixbench_singl20:21:101
75233199289,5cyclictest14-21ksoftirqd/020:50:500
75233199288,4cyclictest14-21ksoftirqd/020:26:050
752331992810,4cyclictest0-21swapper/020:55:580
752331992810,4cyclictest0-21swapper/020:55:580
752333992720,4cyclictest779356-21latency_hist20:40:022
752332992720,3cyclictest0-21swapper/119:10:491
752332992719,5cyclictest757162-21aten_r2power_fr19:25:491
752332992719,5cyclictest3179073-21cupsd21:06:041
752332992719,5cyclictest3179073-21cupsd21:06:041
752332992716,8cyclictest781716-21ntp_kernel_pll_20:46:011
75233199278,3cyclictest14-21ksoftirqd/019:15:580
75233199274,5cyclictest14-21ksoftirqd/019:45:560
75233199274,5cyclictest14-21ksoftirqd/019:45:560
752331992720,4cyclictest781332-21fschecks_count20:45:520
752333992621,2cyclictest0-21swapper/220:30:022
752333992619,4cyclictest758960-21kernelversion19:30:522
752333992618,4cyclictest0-21swapper/219:46:012
752333992618,4cyclictest0-21swapper/219:46:012
752333992614,9cyclictest421-21dbus-daemon20:15:022
752332992620,3cyclictest0-21swapper/119:25:011
752332992617,7cyclictest775591-21missed_timers20:25:591
752332992617,6cyclictest420-21avahi-daemon19:41:031
752332992617,5cyclictest0-21swapper/120:31:011
752331992612,3cyclictest171rcu_preempt20:35:500
75233399257,5cyclictest171rcu_preempt20:15:572
752332992516,7cyclictest759496-21switchtime19:31:071
75233299244,14cyclictest428-21mta-sts-daemon20:41:021
752331992214,5cyclictest752280-21latency_hist19:10:020
752333992115,3cyclictest757669-21nfs_client19:25:572
75233499172,6cyclictest0-21swapper/321:15:593
75233499163,7cyclictest0-21swapper/322:56:043
75233499163,7cyclictest0-21swapper/322:56:043
75233499162,8cyclictest0-21swapper/322:06:023
75233499161,9cyclictest0-21swapper/322:46:003
75233499161,7cyclictest0-21swapper/323:00:583
75233499161,7cyclictest0-21swapper/321:56:043
75233299161,11cyclictest752286-21/usr/sbin/munin19:10:021
75233499153,6cyclictest0-21swapper/322:15:013
75233499152,7cyclictest0-21swapper/323:11:023
75233499152,7cyclictest0-21swapper/321:21:063
75233499152,6cyclictest0-21swapper/320:51:063
75233499152,6cyclictest0-21swapper/300:15:583
75233499151,8cyclictest0-21swapper/323:40:583
75233499151,8cyclictest0-21swapper/322:26:003
75233499151,8cyclictest0-21swapper/322:26:003
75233499151,8cyclictest0-21swapper/321:45:583
75233499151,8cyclictest0-21swapper/300:11:043
75233499151,7cyclictest0-21swapper/322:30:583
75233499151,7cyclictest0-21swapper/321:50:583
75233499143,6cyclictest0-21swapper/323:40:023
75233499143,6cyclictest0-21swapper/300:26:073
75233499143,6cyclictest0-21swapper/300:26:073
75233499143,4cyclictest0-21swapper/320:41:023
75233499142,8cyclictest0-21swapper/322:15:583
75233499142,8cyclictest0-21swapper/322:15:583
75233499142,7cyclictest0-21swapper/300:06:053
75233499142,6cyclictest0-21swapper/321:30:573
75233499142,6cyclictest0-21swapper/321:30:573
75233499142,4cyclictest0-21swapper/320:26:023
75233499141,8cyclictest0-21swapper/323:25:583
75233499141,8cyclictest0-21swapper/300:36:003
75233499141,7cyclictest0-21swapper/319:25:583
75233499141,6cyclictest0-21swapper/319:30:563
75233499133,6cyclictest0-21swapper/300:25:013
75233499132,7cyclictest0-21swapper/323:55:593
75233499132,7cyclictest0-21swapper/322:51:043
75233499132,7cyclictest0-21swapper/322:40:533
75233499132,5cyclictest0-21swapper/320:45:583
75233499132,5cyclictest0-21swapper/319:20:583
75233499131,7cyclictest0-21swapper/323:45:503
75233499131,7cyclictest0-21swapper/323:20:573
75233499131,7cyclictest0-21swapper/322:21:043
75233499131,7cyclictest0-21swapper/322:21:043
75233499131,7cyclictest0-21swapper/322:00:593
75233499131,7cyclictest0-21swapper/321:26:013
75233499131,7cyclictest0-21swapper/321:26:013
75233499131,6cyclictest0-21swapper/322:36:013
75233499131,6cyclictest0-21swapper/320:21:003
75233499131,6cyclictest0-21swapper/320:10:573
75233499131,6cyclictest0-21swapper/320:06:013
75233499131,6cyclictest0-21swapper/319:41:003
75233499131,5cyclictest0-21swapper/321:05:593
75233499131,5cyclictest0-21swapper/321:05:593
75233499131,5cyclictest0-21swapper/320:55:573
75233499131,5cyclictest0-21swapper/320:55:573
75233499123,6cyclictest0-21swapper/319:35:553
75233499122,6cyclictest0-21swapper/323:50:473
75233499122,5cyclictest0-21swapper/300:00:523
75233499121,7cyclictest0-21swapper/321:41:003
75233499121,7cyclictest0-21swapper/321:36:053
75233499121,6cyclictest0-21swapper/320:15:523
75233499121,5cyclictest0-21swapper/321:01:033
75233499121,5cyclictest0-21swapper/321:01:033
75233499112,5cyclictest0-21swapper/323:15:573
75233499111,6cyclictest0-21swapper/323:05:553
75233499111,6cyclictest0-21swapper/320:31:013
75233499111,6cyclictest0-21swapper/319:10:493
75233499111,6cyclictest0-21swapper/300:30:463
75233499111,5cyclictest0-21swapper/319:55:593
75233499111,5cyclictest0-21swapper/319:55:593
75233499111,5cyclictest0-21swapper/319:15:583
75233499101,6cyclictest0-21swapper/321:10:473
75233499101,5cyclictest0-21swapper/320:00:523
75233499101,5cyclictest0-21swapper/320:00:523
75233499101,5cyclictest0-21swapper/319:45:563
75233499101,5cyclictest0-21swapper/319:45:563
7523349993,4cyclictest0-21swapper/320:35:583
7523349993,4cyclictest0-21swapper/319:51:033
7523349993,4cyclictest0-21swapper/319:51:033
7523349991,6cyclictest0-21swapper/323:30:473
7523349971,3cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional