You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-05 - 21:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Sat Sep 05, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1725899564,12cyclictest27346irq/40-dwc2_hso11:00:122
1725899504,39cyclictest21499-21ssh10:50:152
17249995026,8cyclictest12-21ksoftirqd/010:25:110
1725899482,35cyclictest16039-21ssh11:05:142
1725899479,24cyclictest27-21ksoftirqd/209:15:142
17266994527,8cyclictest35966irq/58-eth012:05:153
1726699431,27cyclictest35966irq/58-eth009:40:133
1725899432,33cyclictest22252-21sshd07:15:132
1726699413,23cyclictest35966irq/58-eth009:15:123
1725899402,21cyclictest19452-21ssh11:40:532
17249993916,21cyclictest12-21ksoftirqd/012:35:140
1725899363,24cyclictest630-21Async09:36:312
1725899362,26cyclictest22920-21apt-get11:45:052
17258993617,13cyclictest0-21swapper/210:20:142
1725099362,2cyclictest131rcu_preempt12:25:121
1725899351,21cyclictest30419-21diskmemload09:25:142
1725099351,31cyclictest8951-21ssh10:25:111
1725099350,22cyclictest23402-21ssh11:45:131
1726699343,19cyclictest35966irq/58-eth012:25:113
1725099331,23cyclictest3682-21ssh12:10:151
1726699327,13cyclictest35966irq/58-eth010:40:133
1725899324,18cyclictest27346irq/40-dwc2_hso08:10:132
1725899323,22cyclictest0-21swapper/211:25:132
1725899322,25cyclictest13062-21ssh11:39:052
17249993215,10cyclictest664-21ntpd12:00:130
1725099314,3cyclictest131rcu_preempt09:55:151
1725099312,25cyclictest1603-21ssh11:15:131
1725099311,27cyclictest18038-21ssh12:00:141
1725899302,21cyclictest0-21swapper/212:21:052
17249993016,9cyclictest12-21ksoftirqd/008:15:110
1724999301,23cyclictest28700-21ssh11:30:150
1726699292,18cyclictest35966irq/58-eth007:15:113
1725099292,22cyclictest18032-21ssh10:30:171
1726699282,17cyclictest35966irq/58-eth012:10:163
1725899284,20cyclictest17248-21cyclictest12:03:052
1725899284,13cyclictest27346irq/40-dwc2_hso08:30:142
1725899281,20cyclictest3521-21ssh10:40:122
1724999288,11cyclictest630-21Async09:36:310
1724999285,15cyclictest12-21ksoftirqd/011:15:140
1726699270,18cyclictest35966irq/58-eth009:10:143
1726699270,18cyclictest35966irq/58-eth009:10:143
17258992720,3cyclictest536-21usb-storage07:50:112
17258992711,12cyclictest877-21lldpd12:05:152
17250992713,11cyclictest1715-21ssh09:45:121
1725099271,18cyclictest27085-21sshd07:20:131
17266992611,9cyclictest35966irq/58-eth012:30:163
17266992611,9cyclictest35966irq/58-eth012:30:153
1726699261,16cyclictest35966irq/58-eth010:15:133
1725899262,8cyclictest20488-21ssh12:19:062
1725899261,15cyclictest21409-21ssh09:20:122
17249992611,8cyclictest12-21ksoftirqd/008:25:150
1726699259,10cyclictest0-21swapper/307:45:133
1726699252,15cyclictest35966irq/58-eth008:20:133
1726699251,15cyclictest35966irq/58-eth011:10:153
1725899252,20cyclictest0-21swapper/208:35:152
1725899252,19cyclictest0-21swapper/207:30:142
17250992510,11cyclictest30415-21ssh10:55:141
1726699242,13cyclictest35966irq/58-eth010:55:133
17266992413,8cyclictest35966irq/58-eth010:30:183
17266992413,8cyclictest35966irq/58-eth010:25:133
1725899242,17cyclictest8976-21ssh12:11:042
1725899242,16cyclictest0-21swapper/208:40:042
17258992412,8cyclictest0-21swapper/209:50:192
17258992412,8cyclictest0-21swapper/209:50:182
1725899240,11cyclictest13552-21latency_hist07:40:022
1724999242,17cyclictest3008-21diskmemload12:23:050
17266992314,6cyclictest0-21swapper/312:20:223
17266992313,8cyclictest0-21swapper/310:15:003
1726699231,12cyclictest35966irq/58-eth011:25:123
1725899233,14cyclictest27346irq/40-dwc2_hso10:15:142
1725899232,6cyclictest0-21swapper/212:28:052
1725899232,16cyclictest0-21swapper/212:37:062
17250992311,3cyclictest3008-21diskmemload11:20:121
17250992311,3cyclictest3008-21diskmemload11:20:121
1724999238,9cyclictest0-21swapper/007:20:110
17249992310,8cyclictest151rcuc/012:08:050
1726699224,12cyclictest35966irq/58-eth008:55:123
1726699222,12cyclictest35966irq/58-eth011:45:143
1725899223,13cyclictest27346irq/40-dwc2_hso09:00:152
1725899222,15cyclictest0-21swapper/212:34:052
1725899222,15cyclictest0-21swapper/212:34:042
1725899221,18cyclictest930-21snmpd08:50:202
1725899221,17cyclictest930-21snmpd10:45:142
1725899221,17cyclictest5463-21sh10:05:142
1725899221,11cyclictest0-21swapper/209:05:132
1725099223,4cyclictest21-21ksoftirqd/108:50:161
1725099221,15cyclictest8982-21apt-get07:35:131
1725099221,15cyclictest16058-21ssh11:05:151
1724999227,3cyclictest131rcu_preempt11:00:180
1724999225,9cyclictest0-21swapper/010:55:150
1724999222,17cyclictest10341-21apt-get08:10:120
17249992212,7cyclictest28446-21cron10:00:010
17249992211,8cyclictest19652-21ntpq09:35:260
17249992211,4cyclictest3008-21diskmemload10:05:130
1726699212,14cyclictest35966irq/58-eth011:00:123
17266992115,4cyclictest35966irq/58-eth010:02:553
17266992111,4cyclictest0-21swapper/310:45:143
1725899219,10cyclictest32442-21ssh11:50:142
1725899215,5cyclictest25389-21irqcore11:10:192
1725899214,14cyclictest4599-21head08:00:302
17258992116,3cyclictest7282-21ssh09:10:292
17258992116,3cyclictest7282-21ssh09:10:282
17258992114,4cyclictest17610-21sendmail12:00:012
17258992112,6cyclictest0-21swapper/208:05:182
17258992112,6cyclictest0-21swapper/207:55:202
1725899211,12cyclictest31642-21cut11:30:232
1725899211,11cyclictest31037-21irqrtprio10:55:192
1725899211,11cyclictest30135-21cut10:35:242
1725899211,10cyclictest16504-21kthreadcore10:10:242
17258992110,9cyclictest21359-21ssh09:36:092
1725099219,9cyclictest8741-21cron07:35:011
1725099216,3cyclictest3008-21diskmemload11:10:231
1725099213,3cyclictest131rcu_preempt09:30:251
1725099212,14cyclictest27806-21ssh12:22:041
1725099211,17cyclictest3131-21timerandwakeup07:25:291
1725099211,16cyclictest25119-21ssh09:40:141
1724999217,8cyclictest12-21ksoftirqd/008:00:150
1724999215,13cyclictest21483-21ssh10:50:150
1724999212,15cyclictest12661-21ssh12:15:140
17249992113,5cyclictest0-21swapper/008:05:180
17249992111,8cyclictest3008-21diskmemload11:25:130
1724999211,15cyclictest17965-21missed_timers10:10:260
1724999211,15cyclictest1721-21ssh09:45:140
1726699208,9cyclictest0-21swapper/309:20:213
1726699203,12cyclictest0-21swapper/312:19:063
1726699203,11cyclictest35966irq/58-eth008:05:143
1726699203,11cyclictest0-21swapper/311:40:523
1726699202,12cyclictest35966irq/58-eth011:15:123
17266992015,3cyclictest36066irq/59-eth009:53:393
17266992015,3cyclictest36066irq/59-eth009:53:383
17266992013,4cyclictest0-21swapper/309:35:123
1726699201,14cyclictest0-21swapper/310:35:143
1725899208,5cyclictest13585-21kthreadcore08:45:202
1725899208,4cyclictest10987-21ssh11:20:182
1725899208,4cyclictest10987-21ssh11:20:172
1725899203,8cyclictest22285-21kthreadcore08:20:222
1725899201,9cyclictest21418-21fschecks_time08:55:162
1725899201,9cyclictest15669-21latency_hist09:35:012
1725899201,16cyclictest0-21swapper/210:25:212
1725899201,15cyclictest18803-21fschecks_time07:45:172
1725899201,12cyclictest14957-21kthreadcore07:40:222
1725899201,11cyclictest3404-21ssh11:15:212
1725899201,10cyclictest27264-21fschecks_time07:20:152
1725899201,10cyclictest1376-21latency_hist09:45:012
17258992010,7cyclictest18653-21irqrtprio10:30:222
1725899200,18cyclictest5734-21timerwakeupswit09:45:302
1725099209,9cyclictest7683-1kworker/u9:010:10:191
1725099208,3cyclictest131rcu_preempt09:15:131
1725099207,10cyclictest27900-21kthreadcore12:05:191
1725099204,8cyclictest0-21swapper/111:25:121
1725099204,13cyclictest24297-21tr10:15:201
1725099202,9cyclictest0-21swapper/110:10:021
1725099201,7cyclictest13599-21latency08:10:221
1725099201,15cyclictest28699-21sh11:30:151
1725099201,11cyclictest1587-21ntp_192.168.11409:25:251
1725099201,10cyclictest18001-21ssh09:35:201
1724999206,4cyclictest131rcu_preempt08:25:010
1724999202,9cyclictest0-21swapper/010:30:160
1724999202,5cyclictest3683-21apt-get12:10:150
1724999202,3cyclictest0-21swapper/010:02:550
1724999202,15cyclictest8971-21ssh11:55:130
1724999202,14cyclictest3424-21ssh09:10:140
1724999202,14cyclictest3424-21ssh09:10:140
1724999201,9cyclictest3274-21kthreadcore08:35:210
17249992014,4cyclictest0-21swapper/010:15:150
17249992014,4cyclictest0-21swapper/007:40:180
1724999201,14cyclictest0-21swapper/008:50:150
1724999201,10cyclictest25574-21missed_timers07:15:210
17249992010,7cyclictest13854-21cron11:40:020
1726699198,7cyclictest35966irq/58-eth012:00:153
1726699192,11cyclictest0-21swapper/307:30:153
17266991912,5cyclictest0-21swapper/310:20:203
17266991911,6cyclictest0-21swapper/311:50:243
17266991911,6cyclictest0-21swapper/310:05:033
1726699191,12cyclictest0-21swapper/309:05:163
17266991910,6cyclictest0-21swapper/310:51:563
1725899196,11cyclictest18654-21kthreadcore07:10:212
1725899192,11cyclictest0-21swapper/208:25:142
1725899191,9cyclictest31850-21cpu07:25:162
1725899191,9cyclictest18959-21tail08:15:272
1725899191,7cyclictest28455-21munin-run10:00:002
1725899191,15cyclictest17310-21latency_hist07:10:022
1725099196,10cyclictest5476-21grep11:35:181
1725099195,3cyclictest131rcu_preempt08:20:021
1725099193,10cyclictest18014-21ssh11:40:271
1725099191,7cyclictest14485-21kthreadcore12:15:211
1725099191,6cyclictest3139-21awk10:40:001
1725099191,16cyclictest32733-21grep09:05:211
1725099191,15cyclictest32360-21apt-get10:20:131
1725099191,13cyclictest28882-21ssh10:00:141
1724999197,9cyclictest19679-21ntpq11:05:260
1724999192,15cyclictest14275-21ssh11:40:130
1724999192,14cyclictest7223-21ssh12:30:160
1724999192,14cyclictest7223-21ssh12:30:160
1724999191,9cyclictest25989-21latency_hist09:00:010
1724999191,9cyclictest11704-21cpuspeed08:45:140
1724999191,4cyclictest26948-21apt-get10:35:140
17249991913,3cyclictest12-21ksoftirqd/007:55:120
17249991913,3cyclictest12-21ksoftirqd/007:25:290
17249991911,5cyclictest3008-21diskmemload09:25:200
17249991911,5cyclictest25265-21mailstats09:20:270
1724999191,14cyclictest28457-21kthreadcore11:10:240
1724999191,14cyclictest10820-21ssh09:50:180
1724999191,14cyclictest10820-21ssh09:50:180
1724999191,13cyclictest32377-21ssh10:20:130
17249991910,6cyclictest26298-21cut09:40:190
17249991910,5cyclictest13130-21kthreadcore11:20:220
17249991910,5cyclictest13130-21kthreadcore11:20:220
1726699189,6cyclictest0-21swapper/312:37:273
1726699183,6cyclictest0-21swapper/311:35:143
1726699182,13cyclictest0-21swapper/311:20:203
1726699182,13cyclictest0-21swapper/311:20:193
1726699182,10cyclictest0-21swapper/308:30:163
1726699182,10cyclictest0-21swapper/307:25:153
1726699181,9cyclictest35966irq/58-eth011:05:133
1725099189,7cyclictest514-21ntpq08:30:261
1725099186,8cyclictest21-21ksoftirqd/111:50:141
1725099183,12cyclictest9874-21kthreadcore12:30:241
1725099183,12cyclictest9874-21kthreadcore12:30:231
1725099182,12cyclictest9599-21irqrtprio11:55:191
1725099181,5cyclictest437-21latency_hist08:00:021
1725099181,5cyclictest3008-21diskmemload11:04:081
17250991813,3cyclictest0-21swapper/108:00:121
1725099181,14cyclictest6535-21apt-get08:40:011
1725099181,13cyclictest7480-21timerandwakeup10:40:291
1725099181,10cyclictest3008-21diskmemload09:20:411
17250991810,3cyclictest91-21pr/ttyS007:50:011
1725099180,16cyclictest3594-21cut09:10:171
1725099180,16cyclictest3594-21cut09:10:161
1724999189,5cyclictest453-21cut11:50:170
1724999189,5cyclictest0-21swapper/010:50:010
1724999185,3cyclictest131rcu_preempt10:40:260
1724999182,13cyclictest397-21kthreadcore12:25:200
1724999182,13cyclictest12411-21apt-get09:15:140
1724999182,11cyclictest0-21swapper/011:45:050
17249991810,5cyclictest0-21swapper/007:50:160
1726699179,6cyclictest0-21swapper/309:47:243
1726699178,6cyclictest0-21swapper/311:30:053
1726699177,8cyclictest0-21swapper/309:55:023
1726699175,9cyclictest0-21swapper/311:55:213
1726699172,10cyclictest0-21swapper/309:00:163
1726699172,10cyclictest0-21swapper/308:45:133
1726699172,10cyclictest0-21swapper/308:40:153
1726699172,10cyclictest0-21swapper/308:10:133
17266991711,4cyclictest35966irq/58-eth009:35:023
1726699171,11cyclictest0-21swapper/309:25:153
1725099176,4cyclictest25665-21ntp_192.168.11407:15:241
1725099174,3cyclictest131rcu_preempt10:45:241
1725099172,10cyclictest0-21swapper/108:45:141
1725099171,12cyclictest29866-21grep09:00:291
1725099171,12cyclictest25751-21kthreadcore08:25:211
1725099171,12cyclictest14692-21switchtime09:50:331
1725099171,12cyclictest14692-21switchtime09:50:331
1725099171,10cyclictest0-21swapper/110:51:061
1724999178,7cyclictest30626-21cron09:05:010
1724999172,11cyclictest10661-21ntp_states09:30:290
1724999171,12cyclictest3016-21munin-plugin-st09:10:010
1725099168,6cyclictest17394-21cat07:10:021
1725099162,10cyclictest0-21swapper/108:05:151
17250991611,3cyclictest16871-21kernelversion12:35:201
1725099161,12cyclictest24024-21irqrtprio07:50:161
1725099161,12cyclictest21441-21fschecks_time08:55:161
1725099161,12cyclictest14483-21sort07:40:211
1725099161,10cyclictest844-21munin-node08:40:151
17250991610,4cyclictest0-21swapper/107:10:071
1724999168,5cyclictest7604-21ntp_kernel_err07:30:250
1724999168,5cyclictest393-21missed_timers08:30:230
1724999163,11cyclictest9122-21fschecks_count07:35:160
1724999162,11cyclictest19473-21kthreadcore07:45:180
1724999161,11cyclictest7069-21perl08:40:170
1725099151,11cyclictest21010-21kthreadcore08:20:201
1726699149,3cyclictest0-21swapper/307:50:223
1726699141,7cyclictest0-21swapper/307:35:133
1724999142,10cyclictest17248-21cyclictest07:10:020
1726699138,3cyclictest0-21swapper/308:35:263
1726699138,3cyclictest0-21swapper/308:15:163
1726699131,8cyclictest0-21swapper/307:40:163
1726699131,6cyclictest0-21swapper/308:50:223
1726699124,5cyclictest0-21swapper/307:10:153
1726699121,9cyclictest0-21swapper/307:20:183
1726699121,3cyclictest0-21swapper/308:00:173
1726699115,4cyclictest0-21swapper/307:55:223
1726699111,7cyclictest0-21swapper/308:25:223
1726699111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional