You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-10 - 20:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Apr 10, 2026 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2187899611,47cyclictest11272-21ssh11:00:121
2183599603,46cyclictest29758-21ssh09:40:120
2187899572,42cyclictest8027-21ssh10:40:121
2183599573,37cyclictest20764-21ssh09:35:120
21878995538,9cyclictest0-21swapper/110:05:121
21878995538,9cyclictest0-21swapper/110:05:111
21908995422,21cyclictest28366irq/58-eth010:00:133
2187899536,5cyclictest131rcu_preempt10:00:121
2190099482,25cyclictest24366-21ssh09:55:142
2187899484,32cyclictest31501-21ssh10:35:141
2187899484,32cyclictest31501-21ssh10:35:131
2183599481,9cyclictest131rcu_preempt09:05:130
2190899473,25cyclictest28366irq/58-eth010:30:103
2187899472,4cyclictest131rcu_preempt11:15:121
2190899467,24cyclictest28366irq/58-eth012:20:113
2187899442,8cyclictest0-21swapper/110:55:151
21878994417,21cyclictest21-21ksoftirqd/107:20:121
2190899436,21cyclictest28366irq/58-eth009:40:113
2190899428,22cyclictest28366irq/58-eth009:55:133
2183599422,32cyclictest26277-21ssh09:20:110
2183599414,26cyclictest13553-21ssh10:25:140
2190899403,20cyclictest28366irq/58-eth011:40:123
2190899403,20cyclictest28366irq/58-eth011:40:123
2190899394,22cyclictest28366irq/58-eth009:45:123
21900993918,15cyclictest27-21ksoftirqd/212:15:132
2187899393,20cyclictest3852-21ssh11:50:131
21878993929,3cyclictest0-21swapper/111:10:131
2187899391,27cyclictest20307-21ssh11:05:141
2187899391,13cyclictest11398-21sendmail-msp08:40:011
2190099373,23cyclictest9316-21ssh11:35:122
2187899372,10cyclictest5717-21kthreadcore09:05:201
21835993714,2cyclictest131rcu_preempt07:15:130
21835993714,2cyclictest131rcu_preempt07:15:120
2190099362,16cyclictest8378-21ssh09:10:112
2187899361,12cyclictest30443-21unixbench_singl09:20:271
2187899361,11cyclictest9009-21cut07:30:141
2187899361,11cyclictest11017-21ssh09:10:191
2187899361,10cyclictest27292-21kthreadcore08:55:191
2187899361,10cyclictest17310-21/usr/sbin/munin09:15:231
2187899357,4cyclictest27988-21tail11:25:311
2187899353,9cyclictest31151-21if_err_eth112:05:141
2187899351,9cyclictest29933-21fschecks_count09:40:151
2187899351,9cyclictest25621-21fschecks_time12:20:141
2187899351,9cyclictest24435-21ntp_192.168.11409:35:241
2187899351,9cyclictest19403-21cron12:35:011
2187899351,9cyclictest19403-21cron12:35:001
2187899351,9cyclictest13650-21tr09:30:201
2187899351,11cyclictest19490-21kthreadcore10:45:211
2187899351,10cyclictest7982-21irqcore12:10:181
2187899351,10cyclictest13141-21cron10:25:011
2183599353,23cyclictest12-21ksoftirqd/008:50:120
2187899343,9cyclictest16170-21kthreadcore08:10:181
2187899342,20cyclictest21853-21ssh12:00:111
2187899341,9cyclictest23536-21fschecks_time07:45:141
2187899341,9cyclictest15990-21irqrtprio09:50:181
2187899341,9cyclictest14228-21sed07:35:171
2187899341,9cyclictest12789-21apt-get11:55:101
2187899341,9cyclictest10976-21kernelversion08:05:181
2187899341,8cyclictest9294-21/usr/sbin/munin11:35:301
2187899341,8cyclictest1866-21latency_hist08:30:021
2187899341,7cyclictest18426-21kthreadcore12:15:211
2187899341,11cyclictest23395-21needreboot12:35:241
2187899341,10cyclictest3813-21kthreadcore12:25:231
2187899341,10cyclictest21088-21kthreadcore08:15:201
21878993410,17cyclictest2916-21ssh09:25:141
21878993410,17cyclictest2916-21ssh09:25:131
2190099333,19cyclictest22346irq/40-dwc2_hso12:05:122
2190099331,24cyclictest327-21ssh11:30:112
2187899339,5cyclictest25450-21ntp_kernel_pll_07:10:231
2187899331,9cyclictest6315-21/usr/sbin/munin09:45:121
2187899331,9cyclictest4364-21timerandwakeup07:55:271
2187899331,9cyclictest29452-21latency_hist08:25:001
2187899331,9cyclictest15120-21ntp_states08:40:241
2187899331,8cyclictest31737-21ssh10:15:251
2187899331,7cyclictest8934-21sh11:35:011
2187899331,12cyclictest24475-21diskstats09:55:141
2190899324,15cyclictest28366irq/58-eth010:20:143
2190099322,20cyclictest22346irq/40-dwc2_hso10:15:112
2190099321,22cyclictest5830-21ssh11:15:122
2187899327,3cyclictest0-21swapper/111:46:021
2187899321,9cyclictest8001-21diskmemload10:50:161
2187899321,9cyclictest417-21latency_hist07:55:011
2187899321,9cyclictest23165-21latency_hist07:45:011
2187899321,9cyclictest21953-21tr08:50:171
2187899321,8cyclictest22045-21ntpq11:40:251
2187899321,8cyclictest22045-21ntpq11:40:241
2187899321,8cyclictest14925-21fschecks_count11:20:151
2187899321,8cyclictest13503-21/usr/sbin/munin10:25:261
2187899321,10cyclictest2101-21diskstats08:30:141
2190899313,5cyclictest28366irq/58-eth009:05:123
2190899312,15cyclictest28366irq/58-eth011:10:133
2187899317,4cyclictest0-21swapper/110:30:291
2187899311,8cyclictest27423-21irqrtprio07:15:181
2187899311,8cyclictest27423-21irqrtprio07:15:171
2187899311,10cyclictest32145-21kthreadcore09:00:181
2183599313,21cyclictest7452-21ssh12:10:130
2183599312,25cyclictest2186366ssh12:00:120
2183599311,23cyclictest16410-21ssh12:15:120
2187899301,9cyclictest9877-21usage_EthcTask08:00:311
2187899301,10cyclictest21527-21kthreadcore10:10:211
2183599304,13cyclictest18881-21ssh10:10:120
2183599303,20cyclictest11281-21ssh11:00:130
21835993016,4cyclictest12-21ksoftirqd/011:15:120
2190099293,17cyclictest22346irq/40-dwc2_hso08:55:122
2190099292,18cyclictest22346irq/40-dwc2_hso09:50:112
2187899291,10cyclictest6406-21kthreadcore07:25:211
2187899290,8cyclictest16437-21/usr/sbin/munin08:45:221
2190899285,16cyclictest28366irq/58-eth008:10:133
2190899283,16cyclictest0-21swapper/311:33:153
2190899283,15cyclictest28366irq/58-eth011:20:113
2190899283,14cyclictest0-21swapper/309:25:133
2190899283,14cyclictest0-21swapper/309:25:133
2190899282,9cyclictest0-21swapper/312:08:163
21908992810,3cyclictest0-21swapper/309:32:063
2183599281,9cyclictest8149-21diskstats10:40:140
2190899273,16cyclictest0-21swapper/312:10:173
2190899272,6cyclictest0-21swapper/311:27:153
2190899272,14cyclictest28366irq/58-eth011:50:123
2190099272,19cyclictest0-21swapper/208:40:132
2190099272,10cyclictest0-21swapper/210:57:152
21900992714,7cyclictest27-21ksoftirqd/210:40:122
21900992710,5cyclictest0-21swapper/207:55:182
21908992621,3cyclictest28366irq/58-eth007:55:143
2190099265,13cyclictest21479-21munin-run12:00:022
2190099264,16cyclictest22346irq/40-dwc2_hso09:35:142
2190099263,18cyclictest22346irq/40-dwc2_hso08:50:122
2190099262,15cyclictest22346irq/40-dwc2_hso09:05:112
21900992614,5cyclictest27-21ksoftirqd/208:25:152
2190099261,15cyclictest13819-21grep07:35:162
2183599264,4cyclictest131rcu_preempt09:25:140
2183599264,4cyclictest131rcu_preempt09:25:130
2183599261,18cyclictest9332-21ssh11:35:130
2190899259,8cyclictest0-21swapper/310:55:143
2190899255,11cyclictest28366irq/58-eth012:30:123
2190899255,11cyclictest28366irq/58-eth012:30:123
2190899253,15cyclictest0-21swapper/308:00:143
2190099251,7cyclictest7858-21wc11:50:272
21900992513,10cyclictest20836-21memory09:15:232
2190099251,10cyclictest5726-21ls08:00:162
2183599252,20cyclictest12814-21ssh11:55:110
2183599250,2cyclictest131rcu_preempt11:50:110
2190899243,9cyclictest0-21swapper/312:18:153
2190099249,6cyclictest31088-21wc11:45:282
2190099244,16cyclictest31504-21diskmemload10:35:142
2190099244,16cyclictest31504-21diskmemload10:35:142
2190099243,16cyclictest18896-21ssh10:10:132
2190099242,16cyclictest22346irq/40-dwc2_hso08:10:142
2190099241,17cyclictest0-21swapper/211:12:152
2190099241,13cyclictest11815-21ssh09:30:132
2183599241,16cyclictest27896-21ssh10:15:120
2190099239,7cyclictest14837-21expr11:00:222
2190099231,9cyclictest0-21swapper/210:49:182
2190099231,7cyclictest0-21swapper/211:28:152
21900992313,8cyclictest0-21swapper/210:00:252
2190099231,14cyclictest0-21swapper/210:30:222
2190099231,12cyclictest22346irq/40-dwc2_hso10:20:122
21835992317,3cyclictest0-21swapper/010:50:140
2183599231,17cyclictest11829-21ssh09:30:130
21835992311,10cyclictest12-21ksoftirqd/009:50:110
2190899222,6cyclictest0-21swapper/309:23:153
2190099228,7cyclictest8001-21diskmemload12:29:042
2190099225,9cyclictest29242-21kthreadcore07:50:182
2190099222,7cyclictest0-21swapper/210:05:432
2190099222,7cyclictest0-21swapper/210:05:422
2190099222,6cyclictest0-21swapper/212:11:172
2190099222,4cyclictest271ksoftirqd/210:50:582
2190099221,9cyclictest0-21swapper/212:30:052
2190099221,9cyclictest0-21swapper/212:30:042
2190099221,8cyclictest0-21swapper/212:04:172
21900992212,8cyclictest25366-21ssh11:05:432
21900992211,9cyclictest25008-21ssh12:35:552
2190099221,15cyclictest928-21ssh12:24:172
2190099221,12cyclictest13554-21sh10:25:142
2183599222,15cyclictest12144-21ssh09:10:260
21835992213,6cyclictest12-21ksoftirqd/012:35:120
2183599221,18cyclictest20568-21missed_timers10:45:240
2183599221,15cyclictest4632-21ls10:20:130
2190899213,5cyclictest0-21swapper/307:40:283
2190899211,17cyclictest0-21swapper/310:15:273
2190099212,15cyclictest24790-21apt-get08:20:132
2190099212,10cyclictest131rcu_preempt11:42:202
2190099212,10cyclictest131rcu_preempt11:42:192
2190099211,8cyclictest0-21swapper/211:21:182
2190099211,6cyclictest6569-21kworker/2:008:35:172
21900992110,9cyclictest0-21swapper/207:45:272
2183599219,8cyclictest30953-21ntpq11:45:250
2183599212,16cyclictest951-21apt-get10:00:120
2183599212,12cyclictest342-21ssh11:30:130
21835992111,7cyclictest2152-21open_files12:05:250
2183599211,13cyclictest19692-21latency_hist08:15:020
2190899203,12cyclictest0-21swapper/310:05:133
2190899203,12cyclictest0-21swapper/310:05:133
21908992011,6cyclictest0-21swapper/309:10:253
2190899201,13cyclictest28366irq/58-eth010:40:123
21908992010,4cyclictest0-21swapper/310:50:313
2190099208,6cyclictest134-21systemd-journal07:21:382
2190099207,11cyclictest31573-21latency_hist07:20:022
2190099207,11cyclictest31573-21latency_hist07:20:012
2190099202,12cyclictest7949-21cut09:45:192
2190099201,9cyclictest30081-21ssh09:40:162
2190099201,9cyclictest2883-21apt-get09:25:122
2190099201,9cyclictest2883-21apt-get09:25:122
2190099201,16cyclictest4181-21fschecks_time07:25:172
2183599208,3cyclictest131rcu_preempt11:25:230
2183599204,6cyclictest21815-21cyclictest08:05:120
2183599201,16cyclictest24340-21apt-get09:55:120
2183599201,15cyclictest29626-21apt-get08:25:120
2183599201,12cyclictest29338-21ssh12:20:270
2190899198,8cyclictest0-21swapper/310:27:153
2190899198,6cyclictest0-21swapper/312:01:213
2190899193,11cyclictest0-21swapper/311:09:153
2190899193,11cyclictest0-21swapper/309:39:153
2190899192,11cyclictest0-21swapper/307:50:053
2190899192,10cyclictest0-21swapper/307:20:133
2190899191,12cyclictest0-21swapper/307:10:213
21908991910,6cyclictest0-21swapper/311:47:363
2190099194,7cyclictest27-21ksoftirqd/209:00:182
2190099192,9cyclictest26256-21/usr/sbin/munin09:20:142
2190099191,9cyclictest25333-21missed_timers07:10:222
2190099191,9cyclictest20106-21fschecks_time08:15:172
2190099191,9cyclictest16791-21grep08:45:172
2190099191,4cyclictest0-21swapper/207:40:102
2190099191,10cyclictest10532-21hwlatdetect08:05:162
2190099190,13cyclictest8604-21kworker/2:207:30:292
2183599198,8cyclictest21501-21unixbench_singl09:15:280
2183599198,8cyclictest10062-21df10:05:140
2183599198,8cyclictest10062-21df10:05:140
2183599197,3cyclictest131rcu_preempt09:45:140
2183599196,4cyclictest131rcu_preempt08:00:160
2183599195,9cyclictest0-21swapper/012:30:130
2183599195,9cyclictest0-21swapper/012:30:120
2183599191,9cyclictest4063-21netstat07:55:230
2183599191,5cyclictest22444-21apt-get10:30:100
2183599191,14cyclictest3981-21users10:35:320
2183599191,14cyclictest3981-21users10:35:310
2183599190,13cyclictest29235-21find11:10:120
2190899188,6cyclictest0-21swapper/311:01:493
2190899187,6cyclictest0-21swapper/310:10:183
2190899182,11cyclictest0-21swapper/309:15:143
2190899182,10cyclictest0-21swapper/310:35:053
2190899182,10cyclictest0-21swapper/310:35:043
2190899182,10cyclictest0-21swapper/308:45:143
2190899182,10cyclictest0-21swapper/307:45:133
21908991812,3cyclictest0-21swapper/312:25:293
2190899181,12cyclictest0-21swapper/312:00:013
21908991810,6cyclictest0-21swapper/311:35:403
2190099181,8cyclictest2190-21fschecks_time08:30:162
2183599186,9cyclictest30809-21latency_hist09:00:020
2183599185,10cyclictest22762-21kthreadcore08:15:210
2183599182,13cyclictest25532-21kthreadcore08:20:200
2183599182,13cyclictest1965-21ssh12:25:140
2183599181,4cyclictest0-21swapper/008:39:100
2183599181,15cyclictest12409-21sendmail_mailst07:30:270
2183599180,12cyclictest23797-21latency11:05:230
2190899179,6cyclictest0-21swapper/310:47:343
2190899178,6cyclictest0-21swapper/311:15:113
2190899178,4cyclictest0-21swapper/312:35:353
2190899172,10cyclictest0-21swapper/308:35:143
2190899172,10cyclictest0-21swapper/307:30:123
21908991710,4cyclictest0-21swapper/309:50:263
2183599179,6cyclictest171-21systemd-network10:57:360
2183599178,7cyclictest31184-21perl09:00:160
2183599171,6cyclictest8001-21diskmemload11:40:040
2183599171,6cyclictest8001-21diskmemload11:40:030
2183599171,5cyclictest27947-21munin-run07:50:020
2183599171,3cyclictest131rcu_preempt08:40:110
2183599171,14cyclictest14677-21dpkg11:20:050
2183599171,12cyclictest21678-21apt-get07:10:030
2190899162,9cyclictest0-21swapper/308:30:133
2190899162,9cyclictest0-21swapper/308:25:143
2187899161,11cyclictest21956-21latency_hist07:10:031
2183599166,7cyclictest29963-21cut07:50:190
2183599162,12cyclictest15552-21kthreadcore07:35:200
2183599162,10cyclictest0-21swapper/007:25:150
2183599161,9cyclictest20188-21timerandwakeup08:45:280
2183599161,8cyclictest1988-21/usr/sbin/munin08:30:170
2183599161,12cyclictest20763-21cut07:40:200
2183599161,11cyclictest32521-21tr07:20:180
2190899154,5cyclictest0-21swapper/307:35:123
2190899151,11cyclictest0-21swapper/308:20:173
2190899144,7cyclictest0-21swapper/307:15:133
2190899144,7cyclictest0-21swapper/307:15:133
2190899143,8cyclictest0-21swapper/308:05:143
2190899141,10cyclictest0-21swapper/309:00:163
2190899141,10cyclictest0-21swapper/308:55:203
2190099131,9cyclictest21952-21latency_hist07:10:022
2183599131,9cyclictest21708-21apt-key07:10:010
2190899127,3cyclictest0-21swapper/308:50:113
2190899121,9cyclictest0-21swapper/307:25:233
2190899121,8cyclictest0-21swapper/308:40:163
2190899121,8cyclictest0-21swapper/308:15:133
2190899121,8cyclictest0-21swapper/307:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional