You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 00:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Tue Jun 02, 2026 12:42:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2314579996555,7cyclictest1055-21mta-sts-daemon12:36:031
2314578996151,8cyclictest2371629-40mandb09:42:040
2314579995835,9cyclictest2445614-21idleruntime-cro11:40:011
2314579995444,5cyclictest0-21swapper/110:56:041
2314580995339,10cyclictest2354742-21latency09:16:032
2314578995244,5cyclictest1-21systemd12:00:010
2314578995020,22cyclictest2405624-21ssh10:36:050
2314580994638,5cyclictest2335573-21awk08:20:012
2314579994638,4cyclictest0-21swapper/108:46:051
2314578994635,8cyclictest2382212-21latency_hist10:00:020
2314578994622,4cyclictest171rcu_preempt12:21:020
2314579994437,3cyclictest0-21swapper/110:00:011
2314579994435,5cyclictest2323580-21sendmail07:40:011
2314578994336,3cyclictest0-21swapper/012:31:110
2314578994236,3cyclictest0-21swapper/011:41:050
2314578994227,11cyclictest0-21swapper/009:31:010
2314580994134,5cyclictest2371650-21ssh09:42:042
2314580994132,5cyclictest2338586-21systemd-run08:30:012
2314579994130,7cyclictest420-21avahi-daemon11:45:291
2314579994130,7cyclictest2396739-21vmstat10:21:151
2314580994034,3cyclictest0-21swapper/210:21:072
231458099403,31cyclictest2440522-21rm11:31:042
2314580994032,5cyclictest2316083-21systemd-run07:15:012
2314580994032,5cyclictest2316083-21systemd-run07:15:012
231457999403,31cyclictest2377013-21ssh09:50:591
2314579994032,5cyclictest2431203-21modprobe11:16:091
2314579994031,6cyclictest2332583-21systemd-run08:10:011
2314578994030,7cyclictest2350573-21munin-plugin-st09:10:010
2314580993931,5cyclictest2343066-21cron08:45:002
2314580993931,5cyclictest2317444-21tail07:16:132
2314580993931,5cyclictest2317444-21tail07:16:132
2314579993931,5cyclictest0-21swapper/111:21:031
2314579993930,6cyclictest2349067-21cron09:05:011
2314579993928,7cyclictest439-21systemd-logind10:10:001
2314579993927,9cyclictest2405059-21aten_r2power_vo10:35:541
2314578993931,6cyclictest0-21swapper/010:24:150
2314578993919,3cyclictest171rcu_preempt08:21:050
231457899390,7cyclictest14-21ksoftirqd/008:06:040
2314580993828,6cyclictest261-21systemd-journal12:25:012
231457999383,30cyclictest2358085-21ssh09:21:061
231457999383,25cyclictest2427999-21ssh11:11:081
231457999383,25cyclictest2427999-21ssh11:11:081
2314579993831,4cyclictest2456862-21unixbench_multi11:56:161
2314579993830,3cyclictest2369553-21kworker/u8:1-rpciod09:42:051
2314579993829,6cyclictest0-21swapper/112:35:011
2314579993827,8cyclictest2462006-21aten_r2power_en12:05:521
2314579993819,7cyclictest1054-21mta-sts-daemon10:31:061
2314578993831,4cyclictest2351985-21proc_pri09:11:090
2314578993827,8cyclictest2437131-21latency11:26:020
231457899382,28cyclictest2421448-21sh11:01:020
231457899382,28cyclictest2421448-21sh11:01:020
2314580993726,8cyclictest2465526-21interrupts12:10:572
231457899372,28cyclictest2361310-21ssh09:26:050
2314580993629,4cyclictest2384005-21grep10:01:142
2314580993624,8cyclictest2374368-21timerwakeupswit09:46:132
231458099362,29cyclictest2462645-21ssh12:06:022
2314579993631,2cyclictest0-21swapper/108:25:011
2314579993628,4cyclictest0-21swapper/110:25:581
2314579993628,4cyclictest0-21swapper/109:16:011
2314579993626,6cyclictest0-21swapper/109:10:011
2314578993630,4cyclictest2374447-21grep09:46:140
2314578993626,7cyclictest2354270-21acpi09:15:520
2314578993623,3cyclictest171rcu_preempt11:21:060
2314578993616,3cyclictest171rcu_preempt12:06:070
2314578993614,4cyclictest0-21swapper/011:06:090
2314578993614,4cyclictest0-21swapper/011:06:090
2314580993527,5cyclictest2366459-21latency_hist09:35:022
2314580993526,4cyclictest0-21swapper/207:31:072
2314580993522,10cyclictest2380692-21timerwakeupswit09:56:152
231457999353,26cyclictest2389692-21ssh10:11:011
231457999353,21cyclictest2468970-21ssh12:16:011
2314579993527,5cyclictest2442426-21cron11:35:001
2314579993527,5cyclictest2340068-21munin-plugin-st08:35:001
2314579993527,5cyclictest2340068-21munin-plugin-st08:35:001
2314579993526,5cyclictest1-21systemd09:35:011
2314579993522,9cyclictest2393381-21timerwakeupswit10:16:141
2314578993529,4cyclictest1054-21mta-sts-daemon08:26:070
2314578993528,4cyclictest2317564-21idleruntime-cro07:20:000
2314578993528,4cyclictest2317564-21idleruntime-cro07:20:000
231457899352,6cyclictest14-21ksoftirqd/007:26:030
2314578993525,7cyclictest0-21swapper/009:55:030
2314578993523,3cyclictest171rcu_preempt11:51:060
231457899352,28cyclictest2469028-21ssh12:16:030
2314578993520,4cyclictest171rcu_preempt10:41:030
231457899351,28cyclictest2481647-21ssh12:36:040
231457899351,28cyclictest2399302-21ssh10:26:050
231457899351,25cyclictest2386534-21ssh10:06:000
2314578993512,3cyclictest171rcu_preempt07:21:010
2314578993511,3cyclictest171rcu_preempt07:51:040
2314580993428,3cyclictest0-21swapper/210:36:072
2314580993428,2cyclictest0-21swapper/210:26:022
2314580993426,5cyclictest2350570-21idleruntime-cro09:10:012
2314580993426,5cyclictest2350566-21diskmemload12:35:532
2314580993426,4cyclictest0-21swapper/210:11:032
2314580993425,7cyclictest2358349-21mailstats09:21:172
2314580993423,8cyclictest2351804-21sh09:11:032
2314580993423,8cyclictest2319892-21needreboot07:25:592
231458099342,27cyclictest2468938-21ssh12:16:002
231457999343,26cyclictest2424695-21ssh11:06:051
231457999343,26cyclictest2424695-21ssh11:06:051
2314579993428,3cyclictest271ktimers/107:26:061
2314579993427,5cyclictest0-21swapper/112:01:001
2314579993425,6cyclictest2334131-21latency_hist08:15:021
2314579993424,7cyclictest1017-21sendmail-mta11:01:371
2314579993424,7cyclictest1017-21sendmail-mta11:01:371
231457999341,25cyclictest659-21snmpd08:01:031
231457999341,25cyclictest2446909-21ssh11:41:051
231457899347,6cyclictest0-21swapper/007:56:020
2314578993429,2cyclictest0-21swapper/009:40:010
2314578993427,4cyclictest2344678-21latency_hist08:50:020
2314578993427,3cyclictest0-21swapper/011:16:030
2314578993425,3cyclictest171rcu_preempt11:32:420
2314578993418,14cyclictest600-21gmain12:03:210
2314580993326,3cyclictest0-21swapper/211:41:042
231458099331,25cyclictest2424636-21ssh11:06:032
231458099331,25cyclictest2424636-21ssh11:06:032
2314579993327,3cyclictest0-21swapper/109:00:011
2314579993325,5cyclictest2373631-21kernelversion09:45:571
2314579993324,6cyclictest624-21lldpd09:37:361
2314579993324,4cyclictest0-21swapper/112:15:011
2314578993327,2cyclictest0-21swapper/012:26:050
2314578993323,7cyclictest12350irq/32-eth010:01:090
2314578993320,6cyclictest171rcu_preempt11:36:040
2314578993316,3cyclictest171rcu_preempt08:11:010
2314578993316,3cyclictest0-21swapper/008:51:050
2314578993312,2cyclictest171rcu_preempt08:01:000
231457899330,12cyclictest2339548-21ntpq08:31:070
231457899330,12cyclictest2339548-21ntpq08:31:070
2314580993225,4cyclictest0-21swapper/210:45:032
2314580993225,3cyclictest0-21swapper/210:51:022
231458099322,22cyclictest2361236-21ssh09:26:022
231457999326,17cyclictest1056-21mta-sts-daemon08:26:031
2314579993222,7cyclictest1-21systemd12:26:591
2314579993222,4cyclictest0-21swapper/109:25:081
2314578993219,5cyclictest2393534-21head10:16:150
2314578993217,3cyclictest171rcu_preempt11:48:500
2314578993214,3cyclictest0-21swapper/010:56:480
2314580993126,2cyclictest0-21swapper/210:35:012
2314580993125,2cyclictest0-21swapper/211:01:032
2314580993125,2cyclictest0-21swapper/211:01:032
2314580993124,4cyclictest1055-21mta-sts-daemon09:35:582
2314580993123,5cyclictest2332571-21munin-plugin-st08:10:012
2314580993123,4cyclictest0-21swapper/211:55:022
2314580993122,6cyclictest2334010-21unixbench_singl08:11:102
231458099312,20cyclictest2386599-21ssh10:06:032
231457999313,22cyclictest2383519-21sshd10:01:071
2314579993126,2cyclictest0-21swapper/111:55:011
2314579993123,4cyclictest0-21swapper/110:41:051
231457999312,22cyclictest2472275-21ssh12:21:071
2314579993121,7cyclictest421-21dbus-daemon08:00:011
231457999311,10cyclictest2327544-21ntpq07:51:071
2314578993125,3cyclictest0-21swapper/007:31:040
2314578993124,4cyclictest0-21swapper/012:11:070
2314578993124,3cyclictest0-21swapper/007:36:070
2314578993123,4cyclictest151ktimers/011:15:010
2314578993123,4cyclictest151ktimers/011:15:010
2314578993123,4cyclictest0-21swapper/010:51:060
2314578993119,8cyclictest2358094-21sh09:21:060
2314578993118,3cyclictest171rcu_preempt07:45:590
2314578993118,10cyclictest0-21swapper/009:01:010
2314578993117,6cyclictest2388782-21ssh10:10:090
2314578993116,3cyclictest171rcu_preempt10:31:000
2314578993111,4cyclictest0-21swapper/007:11:060
2314578993111,4cyclictest0-21swapper/007:11:060
231458099302,8cyclictest171rcu_preempt10:16:032
2314580993024,3cyclictest0-21swapper/212:01:062
2314580993024,3cyclictest0-21swapper/210:46:022
2314580993023,4cyclictest2332796-21kworker/u8:5-xprtiod08:36:132
2314580993023,4cyclictest2332796-21kworker/u8:5-xprtiod08:36:132
2314580993021,6cyclictest0-21swapper/211:46:042
2314580993019,8cyclictest695-21cron09:00:012
231457999309,15cyclictest421-21dbus-daemon07:41:101
2314579993023,3cyclictest0-21swapper/109:15:011
2314579993021,6cyclictest2335405-21unixbench-2d08:16:121
231457999301,8cyclictest2315499-21ntpq07:11:081
231457999301,8cyclictest2315499-21ntpq07:11:081
2314578993018,3cyclictest171rcu_preempt08:36:050
2314578993018,3cyclictest171rcu_preempt08:36:050
2314578993012,13cyclictest2413878-21munin-plugin-st10:50:010
2314578993011,4cyclictest0-21swapper/008:56:020
2314580992922,4cyclictest2433799-21fschecks_count11:20:562
2314580992922,4cyclictest2377620-21unixbench_multi09:51:112
231458099292,22cyclictest2418250-21diskmemload10:56:022
231458099292,21cyclictest243105750ssh11:16:052
2314580992920,6cyclictest1525705-21cups-browsed11:59:362
2314579992921,5cyclictest2318119-21fschecks_time07:20:551
2314579992921,3cyclictest0-21swapper/110:51:061
231457999291,9cyclictest2342557-21ntpq08:41:071
2314579992919,5cyclictest0-21swapper/107:20:011
2314579992919,5cyclictest0-21swapper/107:20:011
231457999291,8cyclictest2341048-21ntpq08:36:071
231457999291,8cyclictest2341048-21ntpq08:36:071
2314578992914,3cyclictest171rcu_preempt08:16:040
2314578992914,2cyclictest171rcu_preempt07:40:580
2314578992913,15cyclictest57-21kcompactd008:40:330
2314580992822,3cyclictest0-21swapper/211:40:012
2314580992821,4cyclictest2437385-21ntp_kernel_pll_11:26:072
2314580992820,5cyclictest2345110-21gltestperf08:50:562
2314580992820,3cyclictest0-21swapper/208:31:062
2314580992820,3cyclictest0-21swapper/208:31:062
231457999284,15cyclictest428-21mta-sts-daemon07:31:021
2314579992820,5cyclictest1525705-21cups-browsed08:53:571
2314579992819,6cyclictest0-21swapper/111:25:531
2314579992818,7cyclictest2411823-21missed_timers10:46:001
231458099275,13cyclictest428-21mta-sts-daemon07:46:002
2314580992722,2cyclictest0-21swapper/207:40:012
231458099272,19cyclictest2475396-21ntpq12:26:072
2314580992720,3cyclictest0-21swapper/212:35:012
2314580992719,5cyclictest2348801-21timerandwakeup09:01:122
2314580992616,4cyclictest1055-21mta-sts-daemon11:14:472
2314580992616,4cyclictest1055-21mta-sts-daemon11:14:472
2314579992619,4cyclictest2326148-21sendmail_mailst07:46:071
2314580992518,5cyclictest0-21swapper/207:45:022
2314580992518,3cyclictest0-21swapper/208:21:112
2314580992516,6cyclictest420-21avahi-daemon08:46:152
231458099242,18cyclictest1056-21mta-sts-daemon08:01:082
2314580992417,3cyclictest0-21swapper/207:51:042
2314578992410,3cyclictest0-21swapper/007:10:010
2314578992410,3cyclictest0-21swapper/007:10:010
2314580992317,3cyclictest0-21swapper/207:10:002
2314580992317,3cyclictest0-21swapper/207:10:002
2314580992316,5cyclictest0-21swapper/208:00:012
2314580992310,10cyclictest1055-21mta-sts-daemon07:21:022
2314579992212,6cyclictest548-21gdbus07:10:011
2314579992212,6cyclictest548-21gdbus07:10:011
231458199211,9cyclictest0-21swapper/309:26:023
231458199181,8cyclictest0-21swapper/307:46:063
231458199174,10cyclictest0-21swapper/307:41:073
231458199173,7cyclictest0-21swapper/309:36:053
231458199172,8cyclictest0-21swapper/308:51:083
231458199164,7cyclictest0-21swapper/310:36:063
231458199163,9cyclictest0-21swapper/312:21:023
231458199163,7cyclictest0-21swapper/311:06:073
231458199163,7cyclictest0-21swapper/311:06:073
231458199161,8cyclictest0-21swapper/310:46:043
231458199161,8cyclictest0-21swapper/310:01:043
231458199153,7cyclictest0-21swapper/311:41:013
231458199153,6cyclictest0-21swapper/310:30:593
231458199152,7cyclictest0-21swapper/312:31:053
231458199152,7cyclictest0-21swapper/309:51:003
231458199152,6cyclictest0-21swapper/307:36:073
231458199152,10cyclictest0-21swapper/309:42:033
231458199151,8cyclictest0-21swapper/310:41:093
231458199151,7cyclictest0-21swapper/309:01:023
231458199151,7cyclictest0-21swapper/307:16:043
231458199151,7cyclictest0-21swapper/307:16:043
231458199144,5cyclictest0-21swapper/312:36:043
231458199143,8cyclictest0-21swapper/309:16:093
231458199143,6cyclictest0-21swapper/311:16:023
231458199143,5cyclictest0-21swapper/312:16:023
231458199142,7cyclictest0-21swapper/310:51:013
231458199141,7cyclictest0-21swapper/312:11:013
231458199141,7cyclictest0-21swapper/310:26:033
231458199141,7cyclictest0-21swapper/309:11:013
231458199141,7cyclictest0-21swapper/308:21:043
231458199141,6cyclictest0-21swapper/311:56:043
231458199141,6cyclictest0-21swapper/311:51:083
231458199141,6cyclictest0-21swapper/310:56:013
231458199141,6cyclictest0-21swapper/310:21:013
231458199141,6cyclictest0-21swapper/310:16:043
231458199141,6cyclictest0-21swapper/308:46:033
231458199141,6cyclictest0-21swapper/307:11:053
231458199141,6cyclictest0-21swapper/307:11:053
231458199132,7cyclictest0-21swapper/312:01:003
231458199132,6cyclictest0-21swapper/310:10:533
231458199132,5cyclictest0-21swapper/311:35:563
231458199132,5cyclictest0-21swapper/308:26:033
231458199132,5cyclictest0-21swapper/308:20:003
231458199131,8cyclictest0-21swapper/309:21:063
231458199131,7cyclictest0-21swapper/311:31:103
231458199131,7cyclictest0-21swapper/311:26:073
231458199131,7cyclictest0-21swapper/311:01:083
231458199131,7cyclictest0-21swapper/311:01:083
231458199131,6cyclictest0-21swapper/308:56:053
231458199131,6cyclictest0-21swapper/308:31:083
231458199131,6cyclictest0-21swapper/308:31:083
231458199131,6cyclictest0-21swapper/308:01:023
231458199131,6cyclictest0-21swapper/307:56:043
231458199131,5cyclictest0-21swapper/308:41:013
231458199123,5cyclictest0-21swapper/310:06:003
231458199122,6cyclictest0-21swapper/312:06:013
231458199122,5cyclictest0-21swapper/307:25:563
231458199121,7cyclictest0-21swapper/309:56:083
231458199121,6cyclictest0-21swapper/311:46:013
231458199121,6cyclictest0-21swapper/309:31:013
231458199121,5cyclictest0-21swapper/308:36:013
231458199121,5cyclictest0-21swapper/308:36:013
231458199121,5cyclictest0-21swapper/307:51:023
231458199112,4cyclictest0-21swapper/308:10:583
231458199111,6cyclictest0-21swapper/312:26:053
231458199111,6cyclictest0-21swapper/307:31:013
231458199110,5cyclictest0-21swapper/308:05:593
231458199102,5cyclictest0-21swapper/309:06:073
231458199101,6cyclictest0-21swapper/311:20:573
231458199101,5cyclictest0-21swapper/309:45:523
231458199101,5cyclictest0-21swapper/307:21:003
231458199100,6cyclictest0-21swapper/311:10:553
231458199100,6cyclictest0-21swapper/311:10:553
23145819981,4cyclictest0-21swapper/307:10:023
23145819981,4cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional