You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-16 - 06:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Sep 16, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22680991012,23cyclictest12-21ksoftirqd/022:35:130
2268099757,13cyclictest12-21ksoftirqd/023:20:120
2270399644,41cyclictest23742-21ssh00:00:142
22711996342,12cyclictest35966irq/58-eth022:15:133
2268099606,46cyclictest14763-21ssh23:55:130
2269399552,7cyclictest131rcu_preempt21:00:151
2269399552,7cyclictest131rcu_preempt21:00:151
2269399544,37cyclictest2201-21ssh23:30:121
2268099501,38cyclictest30565-21ssh21:40:140
2268099501,38cyclictest30565-21ssh21:40:140
2270399485,15cyclictest27346irq/40-dwc2_hso00:30:162
2268099481,37cyclictest29256-21ssh23:45:130
2271199465,26cyclictest35966irq/58-eth022:40:133
2269399463,4cyclictest131rcu_preempt00:30:161
2270399456,15cyclictest27346irq/40-dwc2_hso21:25:162
22693994516,20cyclictest21-21ksoftirqd/120:50:131
2268099454,12cyclictest12-21ksoftirqd/000:05:130
2268099446,22cyclictest12-21ksoftirqd/021:50:130
22680994416,20cyclictest151rcuc/022:25:120
2271199433,21cyclictest35966irq/58-eth000:25:123
2270399430,5cyclictest22344-21ssh21:35:232
2268099427,23cyclictest12-21ksoftirqd/023:05:130
22703994122,13cyclictest0-21swapper/222:45:132
2268099413,32cyclictest21595-21rm21:35:150
2270399403,15cyclictest20758-21grep23:20:282
2268099401,29cyclictest30879-21ssh23:10:130
2271199394,21cyclictest35966irq/58-eth020:15:123
2268099386,4cyclictest131rcu_preempt22:10:140
2269399372,27cyclictest18236-21ssh22:45:141
2269399363,26cyclictest0-21swapper/120:40:141
2269399361,28cyclictest0-21swapper/120:20:151
2268099364,25cyclictest25228-21ssh21:55:120
2269399355,22cyclictest18363-21ssh00:15:131
2269399351,27cyclictest3510-21ssh21:25:141
2268099352,19cyclictest1835-21ssh22:00:140
2271199334,23cyclictest35966irq/58-eth021:40:143
2271199334,23cyclictest35966irq/58-eth021:40:133
2271199332,24cyclictest35966irq/58-eth023:40:133
22711993315,14cyclictest0-21swapper/321:15:133
2269399331,25cyclictest28856-21ssh22:15:131
2268099331,7cyclictest3504-21apt-get21:25:150
2268099331,11cyclictest12-21ksoftirqd/020:15:120
22711993218,10cyclictest0-21swapper/323:10:143
2271199321,17cyclictest35966irq/58-eth000:05:133
2270399325,17cyclictest27346irq/40-dwc2_hso22:15:132
22680993224,3cyclictest0-21swapper/022:40:120
22711993014,10cyclictest0-21swapper/320:00:133
2271199301,17cyclictest35966irq/58-eth000:00:143
2271199301,15cyclictest35966irq/58-eth000:20:123
22711993010,11cyclictest35966irq/58-eth021:50:133
2270399305,14cyclictest27346irq/40-dwc2_hso20:40:142
2268099301,21cyclictest26938-21ssh21:20:130
2270399293,16cyclictest27346irq/40-dwc2_hso20:55:132
2270399292,20cyclictest0-21swapper/200:15:132
2268099294,16cyclictest12-21ksoftirqd/022:55:170
2268099294,16cyclictest12-21ksoftirqd/022:55:170
2271199286,14cyclictest35966irq/58-eth000:15:123
2271199282,12cyclictest35966irq/58-eth019:40:133
2271199281,15cyclictest35966irq/58-eth021:30:143
2270399282,21cyclictest27-21ksoftirqd/221:15:142
2270399282,20cyclictest0-21swapper/221:00:172
2270399282,20cyclictest0-21swapper/221:00:172
2271199273,12cyclictest0-21swapper/300:10:113
2271199272,16cyclictest35966irq/58-eth022:50:123
22711992717,6cyclictest35966irq/58-eth022:45:153
2270399272,11cyclictest0-21swapper/220:20:152
2271199261,16cyclictest35966irq/58-eth023:25:153
2270399265,12cyclictest27346irq/40-dwc2_hso20:45:142
2270399264,11cyclictest27346irq/40-dwc2_hso22:35:102
2270399262,16cyclictest11455-21irqrtprio20:05:192
2270399262,14cyclictest27346irq/40-dwc2_hso23:25:152
2270399261,15cyclictest29411-21fschecks_count23:45:172
2269399262,18cyclictest5787-21ssh23:50:141
2268099265,13cyclictest5423-21ssh22:20:100
2268099264,8cyclictest12-21ksoftirqd/023:30:140
2268099262,18cyclictest27246-21ssh22:50:130
2271199252,14cyclictest35966irq/58-eth022:35:123
2271199252,14cyclictest35966irq/58-eth019:15:133
2270399253,18cyclictest14164-21kthreadcore23:35:242
2270399252,20cyclictest5006-21fschecks_count19:25:162
2270399251,19cyclictest2214-21sh23:30:142
2269399253,15cyclictest9160-21ssh22:40:111
2269399252,18cyclictest7566-21ssh00:25:241
2271199243,15cyclictest0-21swapper/322:29:373
2271199243,13cyclictest35966irq/58-eth021:55:113
2271199241,15cyclictest35966irq/58-eth020:55:143
2270399244,11cyclictest26037-21kthreadcore21:55:182
2270399243,18cyclictest9530-21fschecks_time00:10:162
2270399242,17cyclictest0-21swapper/222:30:362
2270399242,15cyclictest19037-21cat19:40:012
2270399242,13cyclictest24268-21ssh23:40:282
2270399241,7cyclictest727466ssh21:45:142
22703992413,9cyclictest5700-21ssh21:43:442
22703992413,9cyclictest5700-21ssh21:43:442
2270399241,12cyclictest2292166ssh22:29:482
2270399240,12cyclictest320266ssh00:24:362
2269399244,13cyclictest25699-21ssh23:25:151
2269399241,7cyclictest19671-21kthreadcore23:20:201
2269399241,19cyclictest14478-21apt-get19:35:111
2268099244,4cyclictest25676-21apt-get23:25:140
2268099242,3cyclictest131rcu_preempt19:55:140
2268099242,19cyclictest12912-21ssh23:00:110
2268099242,16cyclictest23747-21ssh00:00:140
2268099242,15cyclictest7183-21ssh21:45:150
2268099241,18cyclictest20235-21ssh23:40:130
2271199232,15cyclictest35966irq/58-eth022:30:123
2270399233,12cyclictest356-21irqcore19:20:182
2270399232,18cyclictest8535-21latency_hist21:10:022
2270399232,18cyclictest29758-21kthreadcore19:15:222
2270399232,18cyclictest16115-21kthreadcore23:55:222
2270399232,16cyclictest0-21swapper/222:50:372
22703992316,4cyclictest930-21snmpd22:05:262
22703992316,4cyclictest930-21snmpd22:05:252
22703992312,9cyclictest28553-21ssh23:07:422
2270399231,17cyclictest12590-21ssh21:30:142
22703992311,5cyclictest11516-21ssh00:28:452
2269399231,19cyclictest9669-21apt-get19:30:121
2269399231,18cyclictest2665-21cut20:30:161
2268099237,8cyclictest12-21ksoftirqd/020:35:300
2268099236,4cyclictest131rcu_preempt00:24:500
2268099232,17cyclictest12-21ksoftirqd/022:05:130
2268099232,17cyclictest12-21ksoftirqd/022:05:120
22680992317,4cyclictest12-21ksoftirqd/021:15:120
22680992312,2cyclictest12-21ksoftirqd/022:15:140
22680992311,4cyclictest12-21ksoftirqd/019:10:260
2270399229,10cyclictest11113-21kthreadcore19:30:202
2270399224,12cyclictest8800-21/usr/sbin/munin21:10:122
2270399223,12cyclictest26424-21ls20:55:012
2270399222,17cyclictest18521-21kthreadcore21:50:212
2270399222,15cyclictest0-21swapper/222:43:382
22703992211,9cyclictest27938-21ssh22:14:252
2270399221,18cyclictest1364-21cut00:05:192
2270399221,14cyclictest8425-21diskmemload23:02:472
2270399221,14cyclictest252766ssh21:24:212
2269399222,17cyclictest20065-21grep21:50:291
2269399222,17cyclictest19681-21kthreadcore21:15:211
2269399221,18cyclictest23486-21ls00:00:011
2269399221,17cyclictest26968-21irqcore20:55:181
2269399221,15cyclictest30696-21ls00:40:021
2269399221,15cyclictest12594-21sh21:30:141
2268099225,9cyclictest151rcuc/000:10:120
22680992214,5cyclictest0-21swapper/019:30:250
22680992214,4cyclictest12-21ksoftirqd/019:40:150
22680992210,3cyclictest12-21ksoftirqd/000:25:360
2268099220,19cyclictest2642-21cut20:30:160
2271199219,8cyclictest33-21ksoftirqd/322:10:143
2271199211,3cyclictest0-21swapper/322:20:103
22711992110,8cyclictest0-21swapper/322:02:123
2270399217,3cyclictest16766systemd-network20:30:082
2270399213,7cyclictest0-21swapper/223:50:132
2270399212,12cyclictest19569-21tail20:10:312
2270399211,11cyclictest4156-21fschecks_count22:55:202
2270399211,11cyclictest4156-21fschecks_count22:55:192
2269399215,3cyclictest131rcu_preempt22:40:011
2269399213,3cyclictest131rcu_preempt22:59:401
2269399213,3cyclictest131rcu_preempt22:59:391
2269399212,16cyclictest9319-21kthreadcore20:35:211
2269399212,16cyclictest6251-21cut20:00:161
2269399212,16cyclictest4347-21kthreadcore19:55:241
2269399212,16cyclictest22171-21fschecks_time23:05:161
2269399212,15cyclictest21747-21latency_hist23:05:011
2269399212,15cyclictest11388-21irqcore20:05:191
2269399212,14cyclictest8425-21diskmemload00:05:131
2269399211,6cyclictest29977-21irqstats23:45:191
2269399211,17cyclictest17351-21irqcore20:45:181
2269399211,17cyclictest14299-21kthreadcore23:35:231
2269399210,6cyclictest21-21ksoftirqd/122:27:171
2268099219,3cyclictest12-21ksoftirqd/023:19:010
2268099219,3cyclictest12-21ksoftirqd/022:46:330
2268099219,3cyclictest12-21ksoftirqd/022:30:510
2268099216,3cyclictest131rcu_preempt21:00:180
2268099216,3cyclictest131rcu_preempt21:00:170
2268099215,3cyclictest131rcu_preempt20:50:260
22680992112,6cyclictest16560-21ssh23:36:070
2268099211,17cyclictest14609-21cut19:35:140
2271199209,8cyclictest0-21swapper/323:17:363
2271199202,11cyclictest35966irq/58-eth023:30:133
2271199202,10cyclictest35966irq/58-eth000:30:153
22711992013,4cyclictest0-21swapper/321:45:273
2270399203,9cyclictest11483-21ssh23:15:302
2270399203,5cyclictest0-21swapper/220:25:142
2270399203,14cyclictest7448-21fschecks_count20:35:152
2270399202,9cyclictest27214-21kthreadcore19:45:222
2270399202,16cyclictest23937-21kthreadcore19:10:192
2270399202,15cyclictest7372-21kthreadcore20:00:212
2270399202,15cyclictest32691-21kthreadcore23:10:202
2270399202,14cyclictest5537-21df22:20:132
2270399201,9cyclictest5897-21tail22:00:312
2270399201,16cyclictest21987-21/usr/sbin/munin00:35:282
2270399201,16cyclictest1712-21cut19:55:192
2270399201,16cyclictest0-21swapper/219:40:142
2270399201,11cyclictest23887-21cat20:15:222
2269399202,16cyclictest16067-21irqcore20:10:191
2269399202,14cyclictest29637-21kthreadcore19:50:191
22693992013,5cyclictest0-21swapper/119:10:271
2269399201,16cyclictest31035-21kthreadcore20:25:211
2269399201,16cyclictest27894-21kthreadcore21:55:201
2269399201,14cyclictest2125-21ssh23:10:251
2268099209,7cyclictest0-21swapper/019:45:120
2268099209,2cyclictest12-21ksoftirqd/019:30:010
2268099208,6cyclictest131rcu_preempt20:40:310
2268099208,2cyclictest12-21ksoftirqd/021:10:250
2268099207,7cyclictest12-21ksoftirqd/000:18:270
2268099201,9cyclictest15412-21ssh00:30:230
22680992012,5cyclictest6304-21irqcore23:50:190
2268099201,11cyclictest22021-21ssh00:35:130
22680992010,6cyclictest151rcuc/020:20:150
2271199199,8cyclictest0-21swapper/323:09:473
2271199191,14cyclictest35966irq/58-eth023:45:133
2271199191,12cyclictest0-21swapper/320:50:153
2271199191,11cyclictest0-21swapper/319:40:023
22711991910,6cyclictest0-21swapper/323:57:553
22711991910,6cyclictest0-21swapper/323:21:533
2270399191,15cyclictest1064-21latency_hist19:55:012
2269399198,3cyclictest21-21ksoftirqd/121:24:231
2269399192,14cyclictest30450-21kthreadcore19:15:221
2269399191,8cyclictest24005-21tr21:35:241
2269399191,4cyclictest12421-21ntp_192.168.11421:10:251
2269399191,2cyclictest131rcu_preempt00:10:111
2269399191,15cyclictest25428-21kthreadcore19:45:201
2269399191,15cyclictest20962-21kthreadcore23:40:191
2269399191,15cyclictest1655-21kthreadcore19:20:201
22693991910,6cyclictest3433-21cron22:55:011
22693991910,6cyclictest21302-21kthreadcore20:15:191
22680991912,4cyclictest12-21ksoftirqd/020:45:280
2268099191,15cyclictest3356-21latency19:20:220
2271199189,6cyclictest0-21swapper/323:51:393
2271199182,10cyclictest0-21swapper/321:00:183
2271199182,10cyclictest0-21swapper/321:00:173
2271199182,10cyclictest0-21swapper/320:45:153
22711991813,3cyclictest33-21ksoftirqd/323:35:213
22711991812,4cyclictest0-21swapper/322:55:163
22711991812,4cyclictest0-21swapper/322:55:163
2271199181,11cyclictest0-21swapper/319:45:223
22711991810,6cyclictest0-21swapper/322:08:393
22711991810,6cyclictest0-21swapper/322:08:383
22711991810,5cyclictest0-21swapper/321:35:163
2270399181,8cyclictest22801-21latency_hist19:10:022
2269399183,12cyclictest2700-21usage_EthcTask21:40:321
2269399183,12cyclictest2700-21usage_EthcTask21:40:311
2269399182,6cyclictest0-21swapper/100:00:141
2269399182,3cyclictest131rcu_preempt21:45:131
2269399182,14cyclictest1994-21sh22:00:181
2269399182,13cyclictest22765-21wc19:10:021
2269399182,12cyclictest4503-21kworker/u8:123:15:041
2269399181,7cyclictest23466-21/usr/sbin/munin22:30:171
2269399181,15cyclictest20795-21kthreadcore19:40:201
2269399181,15cyclictest11289-21ssh22:05:191
2269399181,15cyclictest11289-21ssh22:05:181
2269399181,14cyclictest7481-21kthreadcore22:20:191
2269399181,14cyclictest4077-21hwlatdetect21:05:151
2269399181,14cyclictest28755-21kthreadcore00:20:211
2268099185,3cyclictest131rcu_preempt20:05:180
2268099183,10cyclictest0-21swapper/021:05:130
2268099181,14cyclictest30301-21wc20:55:280
2271199179,6cyclictest0-21swapper/321:10:113
2271199178,6cyclictest35966irq/58-eth023:00:123
2271199178,6cyclictest35966irq/58-eth021:20:143
2271199178,6cyclictest0-21swapper/321:25:563
2271199172,9cyclictest0-21swapper/319:25:143
2271199172,10cyclictest0-21swapper/320:20:153
2271199172,10cyclictest0-21swapper/319:50:143
2271199172,10cyclictest0-21swapper/319:10:153
2271199172,10cyclictest0-21swapper/300:35:043
22711991712,3cyclictest33-21ksoftirqd/319:20:193
2269399171,13cyclictest20195-21if_eth022:10:181
2268099179,6cyclictest32345-21ntpq19:50:260
2268099171,13cyclictest8119-21kthreadcore20:00:220
2268099171,12cyclictest12773-21grep21:30:160
2271199162,9cyclictest0-21swapper/319:30:153
2269399165,7cyclictest21-21ksoftirqd/119:25:141
2268099163,10cyclictest16136-21irqcore20:10:190
2268099161,8cyclictest27848-21cut19:15:170
2268099161,13cyclictest0-21swapper/020:25:130
2271199152,10cyclictest0-21swapper/320:35:143
2271199151,9cyclictest0-21swapper/319:55:153
2271199141,10cyclictest0-21swapper/320:10:183
2268099141,10cyclictest22733-21latency_hist19:10:020
2271199132,9cyclictest0-21swapper/320:25:153
2271199132,8cyclictest0-21swapper/320:05:123
2271199128,2cyclictest33-21ksoftirqd/320:30:233
2271199127,3cyclictest0-21swapper/321:05:153
2271199121,4cyclictest0-21swapper/320:40:193
2271199101,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional