You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 04:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Thu May 28, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24453995911,10cyclictest131rcu_preempt20:25:111
2444599571,3cyclictest0-21swapper/021:30:110
2445399552,8cyclictest131rcu_preempt23:00:111
2445399521,38cyclictest5653-21rm23:50:121
2444599521,6cyclictest0-21swapper/020:55:120
2444599521,45cyclictest0-21swapper/021:15:130
2446299491,40cyclictest0-21swapper/223:45:132
2444599491,3cyclictest0-21swapper/023:30:130
2444599491,3cyclictest0-21swapper/023:30:120
2444599482,7cyclictest0-21swapper/020:35:130
2444599482,2cyclictest2835-21ssh22:00:120
2444599471,38cyclictest0-21swapper/021:10:130
2447099463,27cyclictest35966irq/58-eth023:45:133
2447099463,24cyclictest35966irq/58-eth020:25:113
2446299461,25cyclictest20101-21ssh23:40:132
2447099441,26cyclictest35966irq/58-eth022:30:123
2444599441,3cyclictest0-21swapper/022:25:120
2444599441,3cyclictest0-21swapper/000:30:120
2444599421,3cyclictest0-21swapper/019:50:110
2444599421,2cyclictest23531-21apt-get20:50:130
24462994115,7cyclictest131rcu_preempt20:35:132
2444599412,2cyclictest31885-21apt-get20:25:120
2444599402,4cyclictest5628-21apt-get23:50:130
2444599401,2cyclictest0-21swapper/023:10:120
2447099392,23cyclictest35966irq/58-eth023:15:133
2444599391,2cyclictest0-21swapper/000:20:130
2444599391,27cyclictest26252-21ssh21:55:130
2446299387,20cyclictest27527-21ssh22:50:132
2444599388,26cyclictest0-21swapper/022:35:120
2444599383,31cyclictest0-21swapper/022:30:120
2444599381,16cyclictest17449-21apt-get20:10:110
2446299371,26cyclictest0-21swapper/223:20:122
2445399374,28cyclictest27344-21ssh00:20:121
2444599372,31cyclictest11827-21ssh22:05:130
2447099365,21cyclictest35966irq/58-eth019:50:113
2447099361,23cyclictest35966irq/58-eth021:45:133
2446299362,18cyclictest22491-21ssh22:28:352
2444599363,2cyclictest1788-21apt-get19:20:120
2444599362,5cyclictest0-21swapper/020:15:130
2444599361,5cyclictest0-21swapper/022:20:130
2444599361,3cyclictest0-21swapper/022:45:120
2444599360,4cyclictest29667-21apt22:15:120
2444599360,2cyclictest20958-21apt-get19:40:130
2446299355,13cyclictest261rcuc/223:05:112
2444599352,7cyclictest0-21swapper/019:55:110
2444599352,3cyclictest0-21swapper/020:45:100
2444599352,2cyclictest29379-21apt-get19:15:140
2444599351,3cyclictest0-21swapper/019:25:110
2444599351,3cyclictest0-21swapper/019:10:120
2444599351,30cyclictest14775-21ntpq23:35:260
2444599351,2cyclictest0-21swapper/020:40:120
2444599350,5cyclictest31872-21apt-get21:40:100
2446299345,16cyclictest27346irq/40-dwc2_hso19:15:132
2444599342,9cyclictest13126-21df23:00:140
2444599341,3cyclictest0-21swapper/022:50:120
2444599341,11cyclictest28635-21fschecks_time21:20:140
2446299334,20cyclictest27-21ksoftirqd/219:20:112
2445399332,23cyclictest4062-21ssh00:25:141
2444599334,26cyclictest1665-21cut21:00:210
2444599332,2cyclictest0-21swapper/021:45:130
2444599331,9cyclictest18010-21grep21:50:170
2444599331,9cyclictest11329-21ntp_states20:00:260
2444599331,8cyclictest29700-21sed23:45:170
2444599331,8cyclictest22102-21awk23:05:130
2444599331,8cyclictest15958-21/usr/sbin/munin20:05:230
2444599331,7cyclictest20340-21fschecks_time23:40:160
2444599331,2cyclictest0-21swapper/023:25:120
2444599331,11cyclictest7741-21ntp_states22:55:260
2444599331,10cyclictest14781-21ntpq19:30:260
2444599330,9cyclictest25920-21ssh22:10:550
2444599330,3cyclictest4953-21apt-get21:25:110
2445399327,4cyclictest21-21ksoftirqd/120:05:131
2445399320,22cyclictest20711-21ssh22:10:111
2444599325,24cyclictest17248-21cut19:35:190
2444599321,8cyclictest23484-21cut21:35:200
2444599321,8cyclictest19453-21tr23:20:190
2444599321,8cyclictest13421-21cut22:40:280
2444599321,5cyclictest5627-21diskstats21:05:150
2444599321,3cyclictest18410-21df_inode00:15:150
2444599321,3cyclictest0-21swapper/000:25:130
2444599321,10cyclictest24393-21ssh00:00:190
2444599321,10cyclictest12459-21users23:15:320
2444599320,6cyclictest5002-21ps20:30:200
2444599320,6cyclictest5002-21ps20:30:190
2444599320,2cyclictest9299-21ssh00:10:120
2447099311,17cyclictest35966irq/58-eth022:05:123
2447099311,16cyclictest35966irq/58-eth022:35:123
2445399315,7cyclictest131rcu_preempt23:55:141
2445399313,23cyclictest21999-21ssh23:05:121
2444599311,4cyclictest22154-21df_inode00:35:160
2444599311,2cyclictest0-21swapper/000:05:140
2444599310,2cyclictest14632-21apt-get23:55:130
24470993012,10cyclictest35966irq/58-eth023:30:133
24470993012,10cyclictest35966irq/58-eth023:30:123
24470993010,14cyclictest35966irq/58-eth022:10:123
2446299302,20cyclictest27346irq/40-dwc2_hso23:00:132
2446299302,17cyclictest4969-21ssh21:25:102
2446299302,13cyclictest17518-21ssh00:14:332
2445399302,20cyclictest10064-21diskmemload23:35:141
2445399301,24cyclictest0-21swapper/123:45:131
2444599301,9cyclictest27919-21tr20:20:170
2444599301,6cyclictest29476-21awk19:45:260
2447099292,14cyclictest35966irq/58-eth022:45:113
24462992810,13cyclictest8910-21cron20:35:012
24462992810,13cyclictest8910-21cron20:35:002
2445399281,22cyclictest20120-21ssh23:40:131
2447099273,11cyclictest35966irq/58-eth023:55:143
24470992710,9cyclictest35966irq/58-eth000:30:133
2445399271,22cyclictest514-21ssh22:35:121
24453992710,13cyclictest21-21ksoftirqd/120:30:131
24453992710,13cyclictest21-21ksoftirqd/120:30:131
2447099261,16cyclictest35966irq/58-eth020:05:133
2447099260,11cyclictest35966irq/58-eth019:25:113
2446299263,17cyclictest2111-21ssh23:30:132
2446299263,17cyclictest2111-21ssh23:30:132
2446299261,21cyclictest22839-21ssh21:35:132
2445399261,20cyclictest73-21kswapd023:20:111
2447099254,12cyclictest0-21swapper/322:50:133
2447099251,16cyclictest35966irq/58-eth023:00:133
2447099251,16cyclictest35966irq/58-eth021:15:133
24462992520,3cyclictest930-21snmpd20:15:132
2445399257,8cyclictest27619-21/usr/sbin/munin22:50:131
2445399255,14cyclictest13024-21diskmemload00:30:141
2444599251,4cyclictest24373-21latency_hist19:10:020
2447099249,10cyclictest0-21swapper/300:04:323
2447099243,14cyclictest0-21swapper/300:22:553
2447099242,17cyclictest35966irq/58-eth023:05:123
2447099242,15cyclictest35966irq/58-eth023:35:123
2446299246,5cyclictest0-21swapper/223:55:122
24462992412,9cyclictest0-21swapper/222:35:292
2446299241,14cyclictest31895-21ssh21:40:132
2445399241,4cyclictest131rcu_preempt20:40:121
2445399241,15cyclictest22059-21ssh00:35:151
24470992311,8cyclictest35966irq/58-eth021:55:263
2446299237,7cyclictest9615-21kthreadcore19:25:212
2446299237,14cyclictest5636-21ssh23:50:122
2446299236,5cyclictest0-21swapper/200:39:472
2446299235,4cyclictest18366-21cpuspeed00:15:142
2446299233,16cyclictest10064-21diskmemload00:30:142
2446299233,15cyclictest27346irq/40-dwc2_hso19:40:132
2446299232,18cyclictest6255-21ssh22:20:122
2446299232,17cyclictest18560-21ssh22:45:142
2446299232,12cyclictest29695-21ssh22:15:142
24462992315,5cyclictest25397-21ssh22:10:292
24462992312,9cyclictest0-21swapper/222:40:132
2446299231,19cyclictest9259-21timerwakeupswit21:05:272
2446299231,19cyclictest15036-21timerandwakeup19:30:292
2446299231,12cyclictest930-21snmpd23:15:132
2447099225,11cyclictest35966irq/58-eth021:20:103
2447099222,13cyclictest0-21swapper/322:55:133
24470992211,9cyclictest0-21swapper/323:20:193
24470992211,9cyclictest0-21swapper/322:20:193
2446299228,11cyclictest42-21kauditd20:30:012
2446299226,7cyclictest19680-21grep19:35:272
2446299225,14cyclictest4225-21kthreadcore19:55:172
24462992212,8cyclictest0-21swapper/221:46:402
24462992211,9cyclictest1115-21ssh23:29:172
2446299221,17cyclictest983-21kthreadcore23:10:212
2446299221,13cyclictest31102-21ssh00:20:262
2446299221,12cyclictest930-21snmpd21:20:262
2446299221,12cyclictest4462-21head00:05:292
24462992210,10cyclictest10853-21ssh00:27:512
2445399223,13cyclictest0-21swapper/119:20:131
2445399223,10cyclictest9638-21tr19:25:211
2445399221,7cyclictest10094-21kthreadcore00:10:181
24453992214,5cyclictest0-21swapper/121:16:241
24453992213,6cyclictest0-21swapper/120:55:261
24453992213,6cyclictest0-21swapper/119:45:271
2445399221,15cyclictest6384-21ls19:55:221
2445399221,10cyclictest0-21swapper/122:05:021
24453992210,9cyclictest5932-21tr22:55:211
2447099218,6cyclictest35966irq/58-eth020:15:133
2447099213,13cyclictest0-21swapper/322:28:343
2446299219,9cyclictest32202-21unixbench_singl20:55:272
2446299219,9cyclictest271ksoftirqd/221:30:202
2446299217,11cyclictest0-21swapper/222:05:192
2446299216,12cyclictest28440-21unixbench_singl19:10:272
24462992116,3cyclictest628-21tcf-agent20:05:182
2446299211,2cyclictest131rcu_preempt22:30:222
2446299211,16cyclictest25998-21kthreadcore00:00:222
2446299211,12cyclictest17956-21cut21:50:172
2446299211,11cyclictest29963-21sed21:55:252
2446299211,11cyclictest19518-21diskstats21:15:152
2446299211,11cyclictest1499-21latency19:50:222
2446299211,11cyclictest1178-21irqcore21:00:192
2446299211,10cyclictest4838-21kthreadcore22:55:182
2445399212,14cyclictest26559-21ssh00:19:341
2445399211,9cyclictest0-21swapper/121:10:291
24453992112,5cyclictest0-21swapper/120:20:261
2445399211,17cyclictest22349-21ls20:45:261
2445399211,15cyclictest7624-21ssh23:15:131
2445399210,17cyclictest1540-21latency_hist19:20:011
2447099209,8cyclictest0-21swapper/300:07:213
2447099207,7cyclictest35966irq/58-eth021:35:143
24470992010,8cyclictest0-21swapper/322:19:013
24470992010,6cyclictest35966irq/58-eth022:00:133
2446299208,5cyclictest2735-21ssh22:00:062
2446299206,7cyclictest27413-21grep20:50:292
2446299204,12cyclictest27346irq/40-dwc2_hso20:00:132
2446299202,12cyclictest27346irq/40-dwc2_hso20:10:132
2446299201,9cyclictest23403-21latency_hist20:50:022
2446299201,16cyclictest14815-21kthreadcore20:40:182
2446299201,11cyclictest24552-21latency_hist19:10:022
2446299201,11cyclictest13608-21cut23:35:232
2446299201,10cyclictest27166-21kthreadcore19:45:192
2446299200,11cyclictest10835-21grep21:10:182
2445399207,9cyclictest13397-21/usr/sbin/munin22:40:281
2445399204,6cyclictest21-21ksoftirqd/122:27:451
2445399202,2cyclictest131rcu_preempt22:05:121
2445399202,13cyclictest6977-21ssh00:07:431
24453992012,5cyclictest0-21swapper/119:30:231
24453992011,6cyclictest10064-21diskmemload21:45:171
2445399201,15cyclictest25942-21ls21:55:021
24453992010,8cyclictest26279-21irqstats23:25:171
2445399200,16cyclictest23639-21cut20:50:151
2447099198,8cyclictest0-21swapper/321:45:013
2447099193,11cyclictest0-21swapper/300:17:343
2447099192,11cyclictest0-21swapper/320:40:133
2447099191,13cyclictest0-21swapper/319:55:153
2447099191,12cyclictest0-21swapper/323:25:223
2447099191,12cyclictest0-21swapper/321:00:233
2447099191,12cyclictest0-21swapper/320:45:253
24470991910,6cyclictest0-21swapper/321:51:453
2446299191,11cyclictest27131-21ls20:20:132
2445399199,7cyclictest28154-21cat00:00:301
2445399191,7cyclictest5601-21memory23:30:221
2445399191,7cyclictest5601-21memory23:30:211
2445399191,7cyclictest22831-21apt-get21:35:141
2445399191,2cyclictest131rcu_preempt22:30:201
24453991912,4cyclictest0-21swapper/121:25:121
24453991911,6cyclictest25503-21cat20:15:221
24453991911,6cyclictest0-21swapper/121:55:261
2445399191,13cyclictest31414-21apt-get21:40:031
2447099189,6cyclictest0-21swapper/319:15:273
2447099186,9cyclictest0-21swapper/300:12:413
2447099183,10cyclictest0-21swapper/300:37:433
2447099182,10cyclictest0-21swapper/322:40:053
2447099182,10cyclictest0-21swapper/320:55:133
2447099182,10cyclictest0-21swapper/320:35:133
2447099182,10cyclictest0-21swapper/320:30:053
2447099182,10cyclictest0-21swapper/320:30:043
2447099182,10cyclictest0-21swapper/320:20:123
2447099182,10cyclictest0-21swapper/320:10:133
24470991810,6cyclictest0-21swapper/321:34:503
24470991810,5cyclictest35966irq/58-eth023:50:133
2445399189,6cyclictest23499-21kthreadcore19:40:201
2445399182,3cyclictest131rcu_preempt21:20:261
2445399181,7cyclictest8179-21sed20:00:171
24453991811,5cyclictest0-21swapper/122:45:181
24453991811,4cyclictest0-21swapper/122:20:161
2447099179,6cyclictest0-21swapper/321:27:593
2447099179,3cyclictest0-21swapper/323:42:353
2447099179,3cyclictest0-21swapper/300:28:233
2447099178,6cyclictest0-21swapper/321:10:153
24470991711,4cyclictest0-21swapper/323:10:183
2447099171,11cyclictest0-21swapper/319:20:273
2445399179,5cyclictest830-21cut19:50:201
2445399177,8cyclictest2863-21cat23:10:291
2445399177,4cyclictest21-21ksoftirqd/120:10:131
2445399173,3cyclictest8400-21kthreadcore21:05:211
2445399173,2cyclictest10064-21diskmemload21:30:301
2445399172,12cyclictest3123-21kthreadcore21:00:221
2445399171,8cyclictest30529-21cut22:15:181
24453991711,4cyclictest0-21swapper/119:10:011
2445399171,12cyclictest24606-21apt-get19:10:121
2447099161,9cyclictest0-21swapper/319:40:163
2445399161,12cyclictest9141-21apt-get20:35:141
2445399161,12cyclictest19527-21netstat19:35:241
2447099152,9cyclictest0-21swapper/319:10:143
2447099152,10cyclictest0-21swapper/319:35:133
2447099151,11cyclictest0-21swapper/321:05:223
2447099149,3cyclictest0-21swapper/319:48:333
2447099131,7cyclictest0-21swapper/319:30:133
2447099127,3cyclictest0-21swapper/320:00:263
2447099122,9cyclictest0-21swapper/320:50:193
2447099101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional