You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-04 - 21:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Aug 04, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29339998425,44cyclictest131rcu_preempt10:40:132
29319997427,7cyclictest131rcu_preempt11:00:150
29319997427,7cyclictest131rcu_preempt11:00:140
2935099734,64cyclictest35966irq/58-eth011:20:163
29350996551,10cyclictest35966irq/58-eth011:05:133
2933999591,36cyclictest15908-21ssh11:35:152
2933199583,42cyclictest5796-21ssh10:35:141
2933199574,48cyclictest31265-21ssh09:55:131
2931999563,46cyclictest27834-21ssh09:35:140
2931999563,44cyclictest23571-21ssh10:45:140
2931999542,41cyclictest20242-21ssh10:25:130
2933199524,42cyclictest30434-21ssh11:25:151
2933199523,46cyclictest23159-21ssh12:15:121
2933999512,31cyclictest930-21snmpd09:50:132
29331995016,24cyclictest21-21ksoftirqd/112:10:121
2933199501,41cyclictest25653-21ssh10:10:131
2931999501,35cyclictest3387-21ssh11:10:100
2935099492,39cyclictest0-21swapper/311:45:123
2933999491,28cyclictest930-21snmpd10:50:112
2935099483,27cyclictest35966irq/58-eth012:00:133
2935099473,27cyclictest35966irq/58-eth011:15:133
2933199433,32cyclictest26957-21ssh11:05:141
2933999424,17cyclictest131rcu_preempt12:04:152
2935099413,23cyclictest35966irq/58-eth011:25:143
2933999405,23cyclictest0-21swapper/209:35:152
2933999401,29cyclictest3395-21ssh11:10:102
2933199408,24cyclictest664-21ntpd08:15:141
2931999401,34cyclictest10416-21ssh11:50:140
2933999390,20cyclictest8848-21ssh12:25:142
2931999391,29cyclictest0-21swapper/010:55:140
2931999387,24cyclictest12-21ksoftirqd/011:35:140
2935099371,24cyclictest35966irq/58-eth012:25:133
29331993724,9cyclictest10404-21apt-get11:50:141
29350993620,11cyclictest35966irq/58-eth010:45:133
2933999362,18cyclictest24387-21rm10:08:582
29331993621,9cyclictest0-21swapper/108:55:131
29350993526,5cyclictest35966irq/58-eth009:35:133
2935099351,17cyclictest35966irq/58-eth009:20:143
2933999355,18cyclictest27-21ksoftirqd/211:45:122
2931999358,17cyclictest12-21ksoftirqd/010:30:120
29319993519,10cyclictest14908-21diskmemload10:08:010
2931999351,24cyclictest14713-21sh10:40:130
2931999351,24cyclictest0-21swapper/007:35:140
29331993426,3cyclictest12600-21apt-get08:00:131
29331993425,5cyclictest22192-21apt-get08:10:131
29331993425,4cyclictest4199-21apt-get08:25:141
29331993420,10cyclictest11266-21apt-get10:20:131
29331993417,12cyclictest15401-21apt-get09:10:161
2935099333,26cyclictest35966irq/58-eth010:10:133
2935099331,29cyclictest0-21swapper/311:35:143
2933999331,25cyclictest0-21swapper/210:05:012
2933199333,12cyclictest30712-21cat12:35:311
29331993317,10cyclictest28158-21latency_hist12:00:011
2931999338,3cyclictest131rcu_preempt09:30:120
2935099323,19cyclictest35966irq/58-eth008:40:133
2935099321,25cyclictest0-21swapper/310:15:163
2935099321,25cyclictest0-21swapper/310:15:153
2933999324,18cyclictest24925-21ssh11:40:152
2933199324,11cyclictest10865-21latency_hist10:20:021
2933199324,11cyclictest10865-21latency_hist10:20:011
29331993222,5cyclictest3030-21apt-get07:50:121
2935099311,27cyclictest0-21swapper/311:40:153
2933999315,15cyclictest27346irq/40-dwc2_hso08:35:122
2933999314,16cyclictest27346irq/40-dwc2_hso07:20:132
2933999312,22cyclictest27346irq/40-dwc2_hso08:00:142
2933999311,22cyclictest25767-1kworker/u9:111:05:142
29331993123,4cyclictest23427-21apt-get08:45:121
2933199312,23cyclictest6951-21ssh11:30:141
2935099305,12cyclictest35966irq/58-eth009:15:133
2935099301,22cyclictest0-21swapper/311:50:143
2935099301,21cyclictest0-21swapper/310:55:133
2933999302,11cyclictest30108-21cat11:25:012
2933199302,22cyclictest30109-21grep11:25:021
2933199302,11cyclictest4097-21cat08:25:011
2933199301,23cyclictest13301-21ssh09:45:141
2931999305,18cyclictest12-21ksoftirqd/007:50:140
2931999301,25cyclictest22331-21ssh09:50:150
2935099294,19cyclictest35966irq/58-eth011:00:153
2935099294,19cyclictest35966irq/58-eth011:00:143
2935099293,22cyclictest36066irq/59-eth011:30:143
2935099291,23cyclictest0-21swapper/312:30:143
29350992911,12cyclictest0-21swapper/312:10:133
2933999294,15cyclictest27346irq/40-dwc2_hso07:35:132
2933999293,17cyclictest27346irq/40-dwc2_hso08:10:142
2933999293,11cyclictest0-21swapper/210:27:012
2933999292,20cyclictest27346irq/40-dwc2_hso08:50:132
2933199293,19cyclictest16635-21ssh10:05:121
2933199292,21cyclictest16333-21latency_hist10:05:021
2933199292,10cyclictest10960-21irqrtprio09:05:201
29331992919,6cyclictest3385-21apt-get11:10:111
2935099285,15cyclictest0-21swapper/310:50:143
2935099282,22cyclictest0-21swapper/310:00:153
2935099282,21cyclictest0-21swapper/309:30:133
2933999284,13cyclictest27346irq/40-dwc2_hso08:30:142
2933999283,6cyclictest0-21swapper/207:50:142
2933999282,20cyclictest0-21swapper/207:55:142
2933199286,8cyclictest13816-21apt-get08:35:121
2933199284,10cyclictest26229-21ssh12:34:531
2933199283,9cyclictest0-21swapper/107:55:221
2933199282,8cyclictest0-21swapper/109:20:281
29331992816,8cyclictest22306-21apt-get09:50:131
29331992815,8cyclictest6599-21apt-get07:20:121
2931999284,18cyclictest32315-21ssh12:20:150
2931999284,18cyclictest32315-21ssh12:20:140
2931999282,22cyclictest17790-21ssh12:30:150
2935099274,19cyclictest0-21swapper/309:50:143
29350992716,7cyclictest35966irq/58-eth009:55:143
29350992716,5cyclictest35966irq/58-eth011:10:123
2935099271,23cyclictest0-21swapper/312:20:133
2935099271,23cyclictest0-21swapper/312:20:123
2935099271,22cyclictest0-21swapper/307:25:143
2935099271,21cyclictest0-21swapper/308:25:153
2933999273,8cyclictest0-21swapper/210:14:232
29339992714,9cyclictest9046-21diskmemload10:55:152
2933999271,19cyclictest16086-21sed12:10:202
2933999271,18cyclictest10638-21ntp_states11:30:262
2933999270,22cyclictest31256-21ssh09:55:142
2933199276,7cyclictest20025-21ssh11:18:531
2933199275,9cyclictest29644-21grep08:50:211
2933199272,11cyclictest4017-21latency_hist09:40:011
29331992714,9cyclictest9938-21ssh09:25:141
2931999272,19cyclictest1283-21ssh10:14:240
2935099262,19cyclictest0-21swapper/310:40:133
2935099262,19cyclictest0-21swapper/309:10:163
2935099261,21cyclictest0-21swapper/312:35:173
2935099261,21cyclictest0-21swapper/308:10:133
2935099261,20cyclictest0-21swapper/309:25:143
2933999267,5cyclictest15423-21kthreadcore11:15:232
2933999262,9cyclictest31341-21ssh10:49:022
2933999262,16cyclictest27346irq/40-dwc2_hso11:50:132
2933999261,5cyclictest24156-21ssh09:15:012
2933999261,3cyclictest131rcu_preempt12:35:212
2933199265,7cyclictest14908-21diskmemload09:31:211
29331992620,3cyclictest14708-21apt-get10:40:151
29331992619,3cyclictest17942-21apt-get11:00:151
29331992619,3cyclictest17942-21apt-get11:00:141
29331992614,8cyclictest11403-21apt-get07:25:141
2933199261,11cyclictest23748-21ssh10:25:231
2931999262,18cyclictest844-21munin-node08:25:160
2931999262,14cyclictest24919-21ssh11:40:150
2935099254,16cyclictest36066irq/59-eth012:15:123
2935099252,18cyclictest0-21swapper/307:50:123
2935099251,20cyclictest0-21swapper/308:45:113
2933999253,9cyclictest17786-21ssh12:30:142
2933999253,15cyclictest27346irq/40-dwc2_hso07:15:132
2933999253,13cyclictest27346irq/40-dwc2_hso07:10:132
2933999253,13cyclictest27346irq/40-dwc2_hso07:10:122
2933999251,9cyclictest8812-21latency_hist08:30:012
2933999251,5cyclictest14280-21ssh09:25:332
2933999251,4cyclictest0-21swapper/212:05:292
2933999251,16cyclictest9452-21ssh10:35:272
2933999251,16cyclictest0-21swapper/211:27:292
2933999251,16cyclictest0-21swapper/210:30:072
2933999251,14cyclictest0-21swapper/209:40:202
2933999250,6cyclictest8631-21latency_hist12:25:022
2933999250,6cyclictest8631-21latency_hist12:25:012
2933199255,8cyclictest2051-21unixbench_multi07:45:311
2933199252,7cyclictest0-21swapper/112:05:281
29331992519,2cyclictest29210-21apt-get10:30:131
29331992518,4cyclictest24908-21apt-get11:40:161
2935099242,18cyclictest0-21swapper/309:45:143
2935099241,19cyclictest0-21swapper/307:55:153
2933999245,8cyclictest14916-21apt-get09:10:022
2933999245,13cyclictest19251-21kthreadcore07:30:232
2933999241,5cyclictest0-21swapper/210:15:132
2933999241,5cyclictest0-21swapper/210:15:122
2933999241,5cyclictest0-21swapper/209:48:272
2933999241,16cyclictest0-21swapper/208:20:312
2933999240,6cyclictest1157-21kthreadcore07:45:232
2933199249,8cyclictest0-21swapper/108:05:131
2933199243,7cyclictest0-21swapper/109:40:341
2933199242,5cyclictest3760-21ntpq10:50:261
29331992419,3cyclictest23550-21apt-get10:45:131
2933199241,8cyclictest9157-21df_inode10:55:151
29331992415,6cyclictest14908-21diskmemload09:15:151
29319992415,6cyclictest12-21ksoftirqd/009:10:160
2935099231,19cyclictest0-21swapper/311:55:043
2935099231,19cyclictest0-21swapper/308:05:123
2935099231,18cyclictest0-21swapper/310:20:113
2935099231,17cyclictest0-21swapper/308:00:133
2935099231,17cyclictest0-21swapper/307:10:133
2935099231,17cyclictest0-21swapper/307:10:123
2935099231,15cyclictest0-21swapper/310:30:153
2935099230,20cyclictest0-21swapper/310:35:143
2933999232,16cyclictest0-21swapper/211:01:022
2933999232,16cyclictest0-21swapper/211:01:022
2933999231,5cyclictest19382-21sh11:55:032
2933999231,5cyclictest0-21swapper/209:20:032
29339992312,9cyclictest8235-21ssh09:23:282
2933999231,11cyclictest21905-21sort08:40:222
2933199234,3cyclictest0-21swapper/107:10:301
2933199234,3cyclictest0-21swapper/107:10:301
2933199232,8cyclictest0-21swapper/112:00:131
2933199231,7cyclictest0-21swapper/111:45:151
29331992317,3cyclictest21025-21apt-get07:35:141
2933199231,16cyclictest9047-21grep12:25:171
2931999233,17cyclictest1641-21ssh11:25:270
29319992312,8cyclictest1412-21apt-get11:45:110
2935099224,14cyclictest0-21swapper/307:40:133
2935099221,18cyclictest0-21swapper/308:15:133
2935099221,17cyclictest0-21swapper/312:05:123
2935099221,17cyclictest0-21swapper/309:05:143
2933999221,6cyclictest18441-21cut08:05:192
2933999221,15cyclictest27346irq/40-dwc2_hso10:20:142
29339992211,4cyclictest131rcu_preempt09:30:052
2933999220,8cyclictest23021-21ssh12:15:032
2933199227,4cyclictest0-21swapper/107:30:151
2933199225,7cyclictest20986-21sh11:35:461
2933199223,12cyclictest32319-21ssh12:20:151
2933199223,12cyclictest32319-21ssh12:20:141
2931999221,3cyclictest131rcu_preempt09:10:010
2931999221,17cyclictest28551-21apt-get12:00:130
29319992211,3cyclictest131rcu_preempt11:05:210
2935099211,18cyclictest0-21swapper/310:05:113
2935099211,17cyclictest0-21swapper/310:25:243
2935099211,17cyclictest0-21swapper/309:40:073
2935099211,17cyclictest0-21swapper/309:00:263
2935099211,17cyclictest0-21swapper/308:55:133
2935099211,15cyclictest0-21swapper/307:15:143
2933999219,10cyclictest31684-21latency_hist08:20:012
2933999217,8cyclictest5934-21cut09:00:182
2933999211,9cyclictest0-21swapper/207:40:202
2933199217,9cyclictest25837-21cpu07:40:151
2931999218,4cyclictest131rcu_preempt12:25:260
2931999212,12cyclictest844-21munin-node07:20:140
2931999212,11cyclictest0-21swapper/008:10:140
2931999211,4cyclictest23342-21ssh12:15:140
2931999211,3cyclictest131rcu_preempt07:40:160
2931999211,15cyclictest11283-21ssh10:20:120
2935099201,16cyclictest0-21swapper/308:30:143
2935099201,16cyclictest0-21swapper/308:20:163
2935099201,16cyclictest0-21swapper/307:45:303
2935099201,16cyclictest0-21swapper/307:35:123
2935099201,16cyclictest0-21swapper/307:20:233
2935099200,17cyclictest0-21swapper/307:30:123
2935099200,16cyclictest0-21swapper/308:35:133
2933999206,5cyclictest27-21ksoftirqd/207:25:152
2933999205,12cyclictest1357-21irqrtprio08:55:202
2933199208,9cyclictest29389-21grep07:10:011
29331992015,3cyclictest201rcuc/108:40:161
2933199200,17cyclictest6548-21ls07:20:011
2931999201,4cyclictest0-21swapper/012:39:140
29319992011,6cyclictest5967-21kthreadcore12:05:190
2931999201,16cyclictest28288-21switchtime09:15:300
2931999200,3cyclictest131rcu_preempt10:53:020
2935099191,15cyclictest0-21swapper/308:50:123
2933999191,10cyclictest23620-21fschecks_time08:45:142
2933199195,3cyclictest131rcu_preempt09:00:201
2931999196,4cyclictest0-21swapper/009:25:250
2931999194,5cyclictest12-21ksoftirqd/010:02:480
2931999194,4cyclictest131rcu_preempt08:20:160
2931999193,11cyclictest0-21swapper/008:00:140
2931999192,7cyclictest131rcu_preempt12:10:290
2931999192,3cyclictest8246-21ssh09:23:300
2931999192,3cyclictest131rcu_preempt08:15:130
2931999191,8cyclictest16004-21/usr/sbin/munin11:15:250
2931999191,4cyclictest131rcu_preempt10:35:260
2931999191,3cyclictest131rcu_preempt11:23:220
2931999190,11cyclictest131rcu_preempt10:15:280
2931999190,11cyclictest131rcu_preempt10:15:270
2933199181,14cyclictest10328-21kthreadcore08:30:211
2931999189,7cyclictest28013-21sshd08:48:030
2931999186,4cyclictest131rcu_preempt11:30:230
2931999182,3cyclictest131rcu_preempt07:10:250
2931999182,3cyclictest131rcu_preempt07:10:240
2931999181,3cyclictest131rcu_preempt07:45:320
2931999181,2cyclictest131rcu_preempt07:30:290
29319991812,4cyclictest0-21swapper/007:25:190
29319991811,4cyclictest14908-21diskmemload09:40:150
2931999181,13cyclictest18398-21sendmail-msp08:40:010
29319991810,6cyclictest23170-21sshd08:41:010
2933999171,13cyclictest29200-21latency_hist07:10:022
2931999178,2cyclictest12-21ksoftirqd/008:35:000
2931999176,3cyclictest131rcu_preempt08:50:220
2931999176,2cyclictest131rcu_preempt09:49:590
2931999174,11cyclictest14908-21diskmemload11:55:150
2931999172,4cyclictest131rcu_preempt09:00:220
2931999171,11cyclictest1196-21sed08:55:200
29319991710,4cyclictest2076-21if_err_eth007:15:180
2931999163,10cyclictest18271-21kthreadcore08:05:190
2931999162,2cyclictest131rcu_preempt07:55:150
29319991610,4cyclictest0-21swapper/009:55:190
2935099151,11cyclictest0-21swapper/307:10:013
2931999142,9cyclictest29308-21cyclictest07:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional