You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-14 - 18:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Mon Sep 14, 2026 12:42:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3698165995437,14cyclictest3740263-21ssh09:20:012
3698165995431,20cyclictest3713135-21munin-run08:00:002
3698165995431,20cyclictest3713135-21munin-run08:00:002
3698163994841,4cyclictest3776758-21munin-run10:20:010
3698163994639,4cyclictest3764853-21kworker/u8:3+xprtiod11:00:020
3698163994639,4cyclictest3728250-21latency_hist08:50:020
3698165994535,6cyclictest265-21systemd-journal12:09:592
3698163994538,4cyclictest3747467-21missed_timers09:31:080
3698165994435,4cyclictest3819567-21kworker/u8:9-xprtiod12:01:162
369816599443,33cyclictest3726056-21ntpq08:41:112
369816599441,36cyclictest3808502-21ntpq11:11:142
3698165994334,6cyclictest3807315-21latency_hist11:10:022
3698165994334,6cyclictest0-21swapper/212:40:012
369816599431,39cyclictest3708127-21ntpq07:41:162
369816599431,37cyclictest3844985-21ntpq12:11:132
369816599431,37cyclictest3724504-21ntpq08:36:112
369816599431,36cyclictest3727495-21ntpq08:46:092
3698164994333,7cyclictest0-21swapper/111:00:011
369816599422,35cyclictest3720008-21ntpq08:21:132
369816599421,33cyclictest3714129-21ntpq08:01:152
3698164994233,5cyclictest0-21swapper/111:06:181
3698163994228,11cyclictest3833308-21grep11:51:230
3698165994135,3cyclictest0-21swapper/210:40:012
3698165994127,11cyclictest0-21swapper/211:20:012
369816599412,35cyclictest3814525-21ntpq11:21:122
369816599412,34cyclictest3721539-21ntpq08:26:122
369816599411,35cyclictest3717132-21ntpq08:11:162
369816599411,24cyclictest3789190-21sh10:40:072
3698164994135,3cyclictest0-21swapper/108:00:001
3698164994135,3cyclictest0-21swapper/108:00:001
3698163994135,4cyclictest3777348-21aten_r2power_cu10:21:000
3698163994130,8cyclictest3766285-21timerandwakeup10:01:240
3698163994128,10cyclictest2910165-21cups-browsed09:53:380
3698163994127,12cyclictest1001-21mta-sts-daemon10:30:010
369816599403,33cyclictest3733626-21ntpq09:06:172
3698165994033,3cyclictest0-21swapper/210:51:152
369816599401,35cyclictest3699137-21ntpq07:11:172
369816599401,32cyclictest3730479-21ntpq08:56:082
369816599400,37cyclictest3702061-21ntpq07:21:122
369816499403,29cyclictest3802453-21ssh11:01:181
3698164994031,6cyclictest3780391-21aten_r2power_cu10:26:021
3698164994031,5cyclictest3767658-21/usr/sbin/munin10:05:011
3698163994030,8cyclictest3852807-21sh12:25:010
3698163994027,10cyclictest3735441-21ntpq09:11:180
3698163994027,10cyclictest3735441-21ntpq09:11:180
3698165993931,5cyclictest3776751-21cron10:20:002
369816599392,33cyclictest3835757-21ntpq11:56:102
369816599392,31cyclictest3772025-21ntpq10:11:142
369816599392,31cyclictest3706533-21ntpq07:36:162
369816599391,35cyclictest3826650-21ntpq11:41:112
369816599391,35cyclictest3820585-21ntpq11:31:122
369816599391,35cyclictest3750112-21aten_r2power_vo09:36:022
369816599391,34cyclictest3703639-21ntpq07:26:212
369816599391,34cyclictest3703639-21ntpq07:26:212
369816599391,32cyclictest3753813-21ntpq09:41:172
369816599391,32cyclictest3753813-21ntpq09:41:172
3698164993932,4cyclictest3734145-21latency_hist09:10:001
3698164993931,5cyclictest694-21cron11:40:011
3698164993931,5cyclictest1-21systemd12:40:021
3698164993927,9cyclictest3707974-21needreboot07:41:101
369816399396,26cyclictest3808350-21ssh11:11:090
369816399394,32cyclictest3719106-21kworker/0:1+events_power_efficient08:21:230
3698163993932,4cyclictest0-21swapper/012:20:010
3698163993929,7cyclictest3802247-21needreboot11:01:100
3698163993929,7cyclictest3700176-21fschecks_time07:16:070
3698163993927,9cyclictest3768822-21needreboot10:06:100
3698165993830,4cyclictest0-21swapper/211:01:152
3698165993828,7cyclictest3710960-21needreboot07:51:132
3698165993828,7cyclictest3710960-21needreboot07:51:132
369816599382,32cyclictest3729003-21ntpq08:51:092
369816599382,31cyclictest3854022-21ntpq12:26:102
3698165993822,13cyclictest3766391-21unixbench-2d10:01:242
3698165993819,15cyclictest0-21swapper/212:20:012
369816599381,34cyclictest3723075-21ntpq08:31:142
369816599381,33cyclictest3744555-21ntpq09:26:122
369816599381,33cyclictest3718617-21ntpq08:16:172
369816599381,32cyclictest3762804-21ntpq09:56:132
369816599381,32cyclictest3731402-21/usr/sbin/munin09:01:012
369816599381,32cyclictest3709622-21ntpq07:46:152
369816599381,32cyclictest2910165-21cups-browsed12:33:552
369816599380,32cyclictest3817501-21ssh11:26:102
3698164993830,5cyclictest439-21systemd-logind09:55:011
3698164993830,5cyclictest3737384-21latency_hist09:15:021
3698164993830,5cyclictest3737384-21latency_hist09:15:021
3698164993830,5cyclictest0-21swapper/111:20:011
3698163993832,3cyclictest0-21swapper/010:51:000
3698163993831,3cyclictest0-21swapper/011:16:140
3698163993830,5cyclictest420-21avahi-daemon07:10:020
3698163993828,6cyclictest3766978-21kworker/u8:4-xprtiod11:56:180
3698163993828,4cyclictest3764623-21sendmail10:00:010
369816399382,31cyclictest3716375-21kworker/0:0+events_power_efficient08:16:170
3698163993821,13cyclictest12650irq/32-eth012:06:120
3698163993820,12cyclictest12650irq/32-eth011:41:150
3698163993815,20cyclictest3750629-21ld09:36:110
3698165993728,6cyclictest3741742-21ssh09:21:222
3698165993718,15cyclictest0-21swapper/212:21:162
369816599371,32cyclictest3777996-21ntpq10:21:102
369816599371,32cyclictest3715521-21ntpq08:06:122
369816599370,32cyclictest3768139-21ssh10:05:502
369816599370,23cyclictest3734143-21diskmemload09:30:162
369816499378,20cyclictest3793345-21ssh10:46:191
3698164993730,5cyclictest3698162-21cyclictest10:31:011
3698164993725,9cyclictest3796620-21timerandwakeup10:51:211
3698163993730,4cyclictest3844308-21aten_r2power_fr12:11:000
3698163993727,7cyclictest3754218-21timerwakeupswit09:41:240
3698163993727,7cyclictest3754218-21timerwakeupswit09:41:240
3698163993724,10cyclictest3743296-21idleruntime-cro09:25:010
3698163993719,12cyclictest151ktimers/008:26:160
3698165993623,9cyclictest265-21systemd-journal11:35:552
369816599362,30cyclictest3832751-21ntpq11:51:112
369816599362,29cyclictest3781056-21ntpq10:26:152
3698165993621,12cyclictest0-21swapper/210:50:012
3698165993619,13cyclictest3831092-21ssh11:48:442
369816599361,32cyclictest3760704-21ssh09:52:192
369816599361,32cyclictest3734732-21/usr/sbin/munin09:11:022
369816599361,32cyclictest3734732-21/usr/sbin/munin09:11:022
369816599361,30cyclictest3705058-21ntpq07:31:122
369816599361,30cyclictest3705058-21ntpq07:31:122
369816599361,30cyclictest3699904-21/usr/sbin/munin07:16:012
369816499363,24cyclictest3842007-21ssh12:06:161
3698164993630,4cyclictest3698162-21cyclictest09:35:011
3698164993629,4cyclictest3837593-21sh12:00:001
3698164993627,6cyclictest3840645-21cron12:05:011
3698164993625,7cyclictest3744777-21mailstats09:26:191
3698163993626,7cyclictest3790030-21missed_timers10:41:110
369816599351,31cyclictest3799229-21ntpq10:56:102
369816599351,31cyclictest3784787-21ssh10:31:362
3698164993528,4cyclictest0-21swapper/109:36:121
3698164993522,9cyclictest3820985-21timerandwakeup11:31:221
3698163993526,6cyclictest3839087-21sendmail_mailst12:01:180
369816599340,20cyclictest375592650ssh09:45:402
369816599340,20cyclictest375592650ssh09:45:402
3698164993426,5cyclictest3814591-21ssh11:21:151
3698164993425,5cyclictest3830224-21unixbench-2d11:46:221
3698164993412,6cyclictest28-21ksoftirqd/112:11:091
3698163993425,7cyclictest3715470-21needreboot08:06:100
369816399342,30cyclictest14-21ksoftirqd/010:36:190
3698163993414,15cyclictest3764853-21kworker/u8:3+rpciod11:21:110
3698164993326,4cyclictest3707135-21cron07:40:001
3698164993326,3cyclictest0-21swapper/110:16:011
3698164993325,5cyclictest265-21systemd-journal08:15:011
3698164993325,4cyclictest0-21swapper/111:11:111
3698164993323,6cyclictest421-21dbus-daemon09:17:531
3698164993323,5cyclictest0-21swapper/108:56:111
3698164993321,9cyclictest3756531-21latency09:46:061
3698164993321,9cyclictest3756531-21latency09:46:061
369816499330,28cyclictest3817559-21ssh11:26:121
3698163993325,5cyclictest421-21dbus-daemon08:41:080
3698163993323,4cyclictest3853689-21tr12:26:050
3698163993320,5cyclictest265-21systemd-journal11:40:020
3698163993318,12cyclictest151ktimers/008:31:100
3698163993318,10cyclictest12650irq/32-eth007:36:180
3698163993316,12cyclictest12650irq/32-eth009:16:150
3698163993314,9cyclictest12650irq/32-eth011:06:150
369816399331,25cyclictest3857150-21ssh12:31:130
369816599321,28cyclictest3698119-21munin-run07:10:012
369816499329,4cyclictest28-21ksoftirqd/111:51:111
3698164993226,4cyclictest3698162-21cyclictest09:05:021
3698164993224,5cyclictest3740865-21aten_r2power_en09:21:021
3698164993224,5cyclictest3725677-21fschecks_count08:41:031
3698164993224,5cyclictest3711634-21cron07:55:001
3698164993224,5cyclictest3711634-21cron07:55:001
3698164993224,5cyclictest1-21systemd12:35:011
3698164993224,5cyclictest0-21swapper/110:40:011
3698164993223,6cyclictest0-21swapper/112:21:011
3698164993222,3cyclictest0-21swapper/110:41:211
3698164993221,8cyclictest3763319-21timerwakeupswit09:56:251
3698164993215,10cyclictest3778895-21ssh10:22:071
3698164993210,5cyclictest28-21ksoftirqd/112:26:151
369816399323,23cyclictest3784078-21ssh10:31:110
369816399321,24cyclictest3829729-21ssh11:46:110
3698164993126,2cyclictest0-21swapper/108:20:011
3698164993125,3cyclictest0-21swapper/110:10:001
3698164993110,4cyclictest28-21ksoftirqd/107:31:111
3698164993110,4cyclictest28-21ksoftirqd/107:31:111
3698163993125,4cyclictest3819409-21latency_hist11:30:020
3698163993125,4cyclictest3795002-21ssh10:50:020
3698163993124,4cyclictest3734146-21munin-run09:10:000
3698163993124,4cyclictest3728457-21aten_r2power_fr08:51:000
3698163993116,3cyclictest171rcu_preempt12:36:140
369816399311,4cyclictest171rcu_preempt07:11:170
369816499308,5cyclictest28-21ksoftirqd/111:41:121
3698164993018,7cyclictest3771276-21aten_r2power_en10:11:001
3698163993018,4cyclictest3701441-21aten_r2power_cu07:21:010
369816499298,6cyclictest28-21ksoftirqd/108:21:131
369816499298,4cyclictest28-21ksoftirqd/108:01:101
369816499297,5cyclictest28-21ksoftirqd/108:46:091
369816499297,5cyclictest28-21ksoftirqd/108:26:161
3698164992921,5cyclictest3699227-21systemctl07:11:181
3698164992917,9cyclictest3753959-21mailstats09:41:201
3698164992917,9cyclictest3753959-21mailstats09:41:201
369816399292,22cyclictest3771945-21ssh10:11:110
3698164992823,2cyclictest0-21swapper/112:20:011
3698164992820,5cyclictest3722675-21fschecks_time08:31:051
3698164992820,5cyclictest3710081-21unixbench_singl07:46:221
3698164992820,5cyclictest3700672-21proc_pri07:16:211
3698163992823,2cyclictest0-21swapper/011:31:160
369816499279,4cyclictest28-21ksoftirqd/107:21:151
369816499278,4cyclictest28-21ksoftirqd/108:51:131
369816499276,4cyclictest28-21ksoftirqd/108:36:141
3698164992719,5cyclictest439-21systemd-logind07:30:011
3698164992719,5cyclictest439-21systemd-logind07:30:011
369816399273,5cyclictest171rcu_preempt08:56:100
369816399272,21cyclictest3756729-21ssh09:46:100
369816399272,21cyclictest3756729-21ssh09:46:100
3698163992721,4cyclictest3698162-21cyclictest07:49:400
369816399264,14cyclictest428-21mta-sts-daemon09:26:170
3698163992617,7cyclictest3705028-21ntp_kernel_pll_07:31:120
3698163992617,7cyclictest3705028-21ntp_kernel_pll_07:31:120
3698163992519,3cyclictest0-21swapper/007:30:010
3698163992519,3cyclictest0-21swapper/007:30:010
3698163992517,6cyclictest485-21in:imuxsock08:00:000
3698163992517,6cyclictest485-21in:imuxsock08:00:000
369816399251,19cyclictest665-21snmpd07:41:120
3698163992510,10cyclictest601-21gmain07:51:200
3698163992510,10cyclictest601-21gmain07:51:200
369816399247,13cyclictest151ktimers/009:01:140
3698163992418,4cyclictest2910165-21cups-browsed08:14:200
369816499235,3cyclictest28-21ksoftirqd/108:06:041
3698164992315,4cyclictest0-21swapper/107:10:011
3698163992313,2cyclictest191rcuc/008:01:070
3698163992216,4cyclictest2910165-21cups-browsed08:35:170
369816699182,7cyclictest0-21swapper/310:46:113
369816699182,7cyclictest0-21swapper/309:31:123
369816699181,8cyclictest0-21swapper/311:56:133
369816699172,5cyclictest0-21swapper/307:51:143
369816699172,5cyclictest0-21swapper/307:51:143
369816699171,7cyclictest0-21swapper/308:06:143
369816699153,6cyclictest0-21swapper/312:11:003
369816699152,8cyclictest0-21swapper/312:31:223
369816699152,8cyclictest0-21swapper/308:30:013
369816699152,7cyclictest0-21swapper/311:21:113
369816699151,8cyclictest0-21swapper/311:11:133
369816699151,8cyclictest0-21swapper/310:41:133
369816699151,7cyclictest0-21swapper/310:01:093
369816699142,7cyclictest0-21swapper/309:55:013
369816699142,7cyclictest0-21swapper/309:26:103
369816699142,7cyclictest0-21swapper/309:16:113
369816699142,7cyclictest0-21swapper/309:06:183
369816699142,7cyclictest0-21swapper/308:36:123
369816699142,6cyclictest0-21swapper/308:31:133
369816699142,5cyclictest0-21swapper/308:16:143
369816699141,8cyclictest0-21swapper/309:41:163
369816699141,8cyclictest0-21swapper/309:41:163
369816699141,7cyclictest0-21swapper/311:41:103
369816699141,7cyclictest0-21swapper/310:36:173
369816699141,7cyclictest0-21swapper/310:26:063
369816699141,7cyclictest0-21swapper/310:06:143
369816699141,6cyclictest0-21swapper/308:51:093
369816699141,6cyclictest0-21swapper/308:01:143
369816699141,6cyclictest0-21swapper/307:41:143
369816699141,6cyclictest0-21swapper/307:11:173
369816699136,4cyclictest0-21swapper/311:31:113
369816699135,5cyclictest0-21swapper/310:11:123
369816699134,5cyclictest0-21swapper/310:31:113
369816699133,7cyclictest0-21swapper/312:26:123
369816699133,4cyclictest0-21swapper/310:21:053
369816699132,7cyclictest0-21swapper/309:56:073
369816699132,6cyclictest0-21swapper/309:50:023
369816699132,6cyclictest0-21swapper/309:50:023
369816699132,6cyclictest0-21swapper/309:20:513
369816699132,5cyclictest0-21swapper/309:36:023
369816699131,7cyclictest0-21swapper/312:06:103
369816699131,7cyclictest0-21swapper/311:16:133
369816699131,7cyclictest0-21swapper/310:51:133
369816699131,7cyclictest0-21swapper/310:16:153
369816699131,7cyclictest0-21swapper/309:11:143
369816699131,7cyclictest0-21swapper/309:11:143
369816699131,6cyclictest0-21swapper/307:46:113
369816699131,5cyclictest0-21swapper/308:46:093
369816699131,5cyclictest0-21swapper/307:36:153
369816699131,5cyclictest0-21swapper/307:21:143
369816699123,7cyclictest0-21swapper/311:06:033
369816699123,5cyclictest0-21swapper/311:36:253
369816699122,6cyclictest0-21swapper/312:21:243
369816699122,6cyclictest0-21swapper/308:56:123
369816699121,7cyclictest0-21swapper/312:36:133
369816699121,6cyclictest0-21swapper/311:01:103
369816699121,6cyclictest0-21swapper/310:56:123
369816699121,6cyclictest0-21swapper/309:01:113
369816699121,5cyclictest0-21swapper/307:16:133
369816699113,5cyclictest0-21swapper/307:56:123
369816699113,5cyclictest0-21swapper/307:56:123
369816699112,5cyclictest0-21swapper/308:41:093
369816699111,7cyclictest0-21swapper/312:01:003
369816699111,7cyclictest0-21swapper/311:26:103
369816699111,6cyclictest0-21swapper/311:46:093
369816699111,6cyclictest0-21swapper/308:21:133
369816699102,5cyclictest0-21swapper/307:31:103
369816699102,5cyclictest0-21swapper/307:31:103
369816699101,6cyclictest0-21swapper/312:16:013
369816699101,5cyclictest0-21swapper/308:11:053
36981669993,5cyclictest0-21swapper/307:10:013
36981669991,5cyclictest0-21swapper/311:51:113
36981669991,5cyclictest0-21swapper/307:26:043
36981669991,5cyclictest0-21swapper/307:26:043
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional