You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 19:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Mar 25, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
567899602,45cyclictest14028-21ssh11:45:142
5662996010,10cyclictest131rcu_preempt11:10:110
5678995944,10cyclictest26557-21sh10:05:132
5686995839,11cyclictest28366irq/58-eth010:55:133
567899581,5cyclictest0-21swapper/208:05:142
567099581,45cyclictest31718-21ssh09:50:141
567899571,42cyclictest924-21snmpd12:05:122
566299552,7cyclictest131rcu_preempt07:45:120
567099533,40cyclictest27980-21ssh09:30:131
566299513,45cyclictest11396-21ssh10:50:120
567899501,44cyclictest0-21swapper/211:10:112
567899501,3cyclictest0-21swapper/208:30:132
567899501,38cyclictest924-21snmpd11:00:122
567099492,7cyclictest131rcu_preempt09:55:111
567099492,7cyclictest131rcu_preempt09:55:111
566299493,10cyclictest131rcu_preempt08:10:120
567899481,2cyclictest27622-21apt-get08:40:132
566299484,38cyclictest10282-21ssh11:25:110
566299484,38cyclictest10282-21ssh11:25:110
567899471,3cyclictest0-21swapper/207:45:132
567899462,34cyclictest0-21swapper/209:40:112
567899461,3cyclictest0-21swapper/208:35:132
567899461,29cyclictest27965-21ssh09:30:122
566299465,35cyclictest6477-21ssh11:05:110
567899452,2cyclictest22346irq/40-dwc2_hso08:25:162
567899452,2cyclictest22346irq/40-dwc2_hso08:25:152
567899451,3cyclictest20529-21ssh10:55:112
567899451,3cyclictest0-21swapper/210:30:092
567899451,2cyclictest8366-21apt-get09:55:112
567899451,2cyclictest8366-21apt-get09:55:112
567099455,6cyclictest131rcu_preempt11:40:111
566299452,33cyclictest31727-21ssh09:50:140
566299451,34cyclictest4616-21ssh09:35:140
567899441,6cyclictest0-21swapper/208:45:132
567899440,39cyclictest12338-21apt-get10:15:152
567899440,2cyclictest4595-21ssh09:35:122
566299441,33cyclictest25579-21ssh10:40:110
567899431,7cyclictest0-21swapper/207:30:132
567899431,37cyclictest0-21swapper/212:20:132
567899431,37cyclictest0-21swapper/211:25:122
567899431,37cyclictest0-21swapper/211:25:112
567099431,28cyclictest2303-21ssh10:45:121
566299433,37cyclictest12461-21ssh12:20:130
567899422,2cyclictest28323-21apt-get11:35:122
567099423,35cyclictest828-21ssh09:15:121
5670994227,7cyclictest531-21usb-storage11:00:131
567899411,2cyclictest15398-21apt-get07:20:142
567899402,32cyclictest22346irq/40-dwc2_hso08:20:152
567899400,4cyclictest20206-21apt-get07:25:102
567899400,3cyclictest0-21swapper/207:55:122
5670994014,19cyclictest21-21ksoftirqd/107:50:141
566299405,25cyclictest16504-21ssh10:35:130
566299405,25cyclictest16504-21ssh10:35:130
5662994032,4cyclictest20544-21ssh10:55:130
567899391,2cyclictest5784-21apt-get07:10:122
567899390,4cyclictest7259-21apt-get12:35:142
568699387,16cyclictest28366irq/58-eth011:20:123
567899381,3cyclictest0-21swapper/210:50:132
567899381,3cyclictest0-21swapper/209:00:142
566299381,29cyclictest26558-21ssh10:05:130
566299381,26cyclictest24282-21rm09:10:140
568699372,20cyclictest28366irq/58-eth010:40:123
567899372,3cyclictest0-21swapper/212:00:132
567899371,8cyclictest0-21swapper/210:10:122
567899371,4cyclictest816-21apt-get09:15:132
567899371,29cyclictest19254-21apt-get09:05:122
567899370,33cyclictest17422-21ssh10:00:112
567099373,29cyclictest21534-21ssh10:20:131
566299375,23cyclictest7304-21sh10:30:100
566299372,26cyclictest32048-21rm11:55:130
5662993713,14cyclictest8378-21ssh09:55:100
5662993713,14cyclictest8378-21ssh09:55:100
567899361,31cyclictest9951-21rm11:40:382
567899361,30cyclictest0-21swapper/208:10:122
566299364,25cyclictest14008-21ssh11:45:120
566299361,27cyclictest8640-21ssh12:00:110
567899353,5cyclictest25259-21ssh09:45:212
567899351,7cyclictest1566-21ssh10:44:362
567899351,5cyclictest21531-21apt-get10:20:142
567899351,3cyclictest0-21swapper/207:40:122
567899351,23cyclictest32567-21cut11:55:182
567899350,3cyclictest3289-21apt-get12:15:132
567899350,2cyclictest19268-21ssh11:30:132
567099355,26cyclictest7268-21ssh12:35:111
567099352,29cyclictest26681-21ssh12:10:131
567899343,28cyclictest22346irq/40-dwc2_hso11:50:122
567899341,7cyclictest4835-21/usr/sbin/munin08:50:182
567899341,6cyclictest1244-21grep07:35:282
567899340,2cyclictest21504-21apt-get12:25:142
5662993417,14cyclictest0-21swapper/010:15:140
567899336,24cyclictest12937-21cut07:15:222
567899333,27cyclictest22346irq/40-dwc2_hso07:50:142
567899331,9cyclictest24387-21df_inode09:10:162
567899331,7cyclictest2201-21timerwakeupswit12:30:282
567899331,6cyclictest1193-21/usr/sbin/munin11:20:242
567899331,5cyclictest13512-21unixbench_singl08:55:272
567899331,28cyclictest22178-21kthreadcore08:00:182
567899330,4cyclictest30666-21apt-get10:25:122
567899330,3cyclictest18894-21apt-get09:25:122
567099336,22cyclictest21-21ksoftirqd/112:15:131
566299336,21cyclictest666-21ntpd08:30:120
566299334,21cyclictest12-21ksoftirqd/010:25:130
567899321,6cyclictest3111-21awk09:50:292
567899321,6cyclictest12968-21cut09:20:232
567899321,27cyclictest0-21swapper/210:35:132
567899321,27cyclictest0-21swapper/210:35:122
567899320,7cyclictest6572-21df11:05:142
567099322,21cyclictest21420-21apt-get08:00:111
5670993216,10cyclictest21-21ksoftirqd/107:20:141
567899311,3cyclictest0-21swapper/210:45:112
567899311,2cyclictest3579-21apt-get08:15:132
567899310,7cyclictest31639-21ssh11:18:062
566299311,20cyclictest9912-21ssh09:20:130
5686993017,9cyclictest28366irq/58-eth009:35:133
567899305,23cyclictest26561-21ssh12:10:062
566299302,22cyclictest29638-21ssh12:29:220
568699292,14cyclictest28366irq/58-eth012:25:133
567099296,16cyclictest5643-21cyclictest10:40:121
567099292,22cyclictest19746-21ssh12:23:231
5670992914,6cyclictest131rcu_preempt12:05:121
566299295,21cyclictest12-21ksoftirqd/009:05:140
566299293,20cyclictest32037-21ssh11:35:250
568699282,16cyclictest28366irq/58-eth008:35:113
5686992814,9cyclictest28366irq/58-eth009:00:133
5686992812,9cyclictest0-21swapper/310:50:133
568699281,14cyclictest28366irq/58-eth012:10:133
568699273,14cyclictest28366irq/58-eth010:45:113
567099279,11cyclictest24565-21ssh11:15:121
567099271,2cyclictest131rcu_preempt10:15:141
566299277,9cyclictest0-21swapper/012:10:130
566299272,19cyclictest6529-21ssh12:34:360
5662992712,10cyclictest0-21swapper/008:40:120
568699264,12cyclictest28366irq/58-eth007:30:133
568699262,15cyclictest28366irq/58-eth009:50:143
568699261,21cyclictest28366irq/58-eth011:30:123
567099262,20cyclictest17444-21ssh10:00:141
567099261,18cyclictest19277-21ssh11:30:131
5670992611,3cyclictest131rcu_preempt11:50:311
566299262,19cyclictest26283-21ssh09:45:270
566299261,21cyclictest18909-21ssh09:25:140
568699253,14cyclictest28366irq/58-eth007:35:123
568699251,15cyclictest0-21swapper/312:23:223
567099252,18cyclictest14442-21apt-get09:00:131
566299256,12cyclictest23862-21diskmemload10:05:000
5686992413,9cyclictest0-21swapper/310:30:203
567899244,17cyclictest5714-21latency_hist07:10:022
5670992412,9cyclictest4529-21cron11:40:011
566299244,14cyclictest10229-21kworker/u8:211:50:130
568699233,13cyclictest0-21swapper/312:34:373
5686992316,4cyclictest0-21swapper/311:35:123
567099233,16cyclictest22564-21ssh09:45:121
567099231,3cyclictest131rcu_preempt08:15:121
566299232,16cyclictest3217-21ssh10:10:120
566299231,3cyclictest131rcu_preempt11:00:250
568699223,16cyclictest28366irq/58-eth009:20:133
567099229,10cyclictest4537-21cron08:50:011
567099221,17cyclictest20144-21apt-config10:55:021
566299222,15cyclictest750-21sendmail-mta09:00:140
566299221,16cyclictest7291-21sh12:35:130
566299221,16cyclictest13360-21cut08:25:170
566299221,16cyclictest13360-21cut08:25:170
566299221,13cyclictest5341-21ssh10:45:220
568699217,11cyclictest28366irq/58-eth012:00:123
568699213,10cyclictest28366irq/58-eth011:25:113
568699213,10cyclictest28366irq/58-eth011:25:103
5686992111,7cyclictest28366irq/58-eth010:10:123
5686992110,8cyclictest0-21swapper/310:03:053
5686992110,8cyclictest0-21swapper/309:11:493
567099218,3cyclictest131rcu_preempt11:45:151
567099217,10cyclictest30336-21latency_hist12:30:021
567099214,11cyclictest0-21swapper/107:30:121
567099211,7cyclictest20883-21ssh09:43:101
5670992111,8cyclictest9950-21sshd11:05:221
567099211,13cyclictest24246-21find09:10:121
566299219,8cyclictest27795-21sshd11:15:220
566299215,9cyclictest0-21swapper/011:30:120
566299213,12cyclictest0-21swapper/011:20:120
568699209,7cyclictest28366irq/58-eth011:55:333
5686992012,5cyclictest0-21swapper/312:35:183
5686992011,6cyclictest28366irq/58-eth010:35:143
5686992011,6cyclictest28366irq/58-eth010:35:133
568699201,13cyclictest0-21swapper/308:30:173
568699201,13cyclictest0-21swapper/307:55:153
568699201,12cyclictest28366irq/58-eth011:00:133
568699201,12cyclictest0-21swapper/309:25:233
567099206,11cyclictest31399-21ssh10:05:311
567099202,3cyclictest131rcu_preempt10:35:201
567099202,3cyclictest131rcu_preempt10:35:201
567099201,9cyclictest30676-21sort12:30:141
567099201,8cyclictest15181-21sendmail-msp07:20:011
567099201,8cyclictest10989-21kthreadcore09:20:201
5670992013,4cyclictest0-21swapper/107:40:281
566299209,7cyclictest12-21ksoftirqd/007:20:130
566299208,8cyclictest21167-21cron08:00:010
566299207,9cyclictest29940-21df_inode07:35:140
566299207,10cyclictest26690-21grep08:35:280
566299202,15cyclictest11651-21cat08:20:230
566299202,13cyclictest31623-21ntp_192.168.11409:30:260
566299201,16cyclictest21562-21sh10:20:150
566299200,13cyclictest13315-21ls08:55:260
568699192,13cyclictest28366irq/58-eth012:05:123
568699191,13cyclictest0-21swapper/308:45:233
568699191,12cyclictest0-21swapper/308:25:293
568699191,12cyclictest0-21swapper/308:25:293
5686991910,6cyclictest0-21swapper/309:44:153
567099194,10cyclictest0-21swapper/109:05:141
567099192,6cyclictest53146usb-storage08:50:041
567099191,16cyclictest9504-21ls07:10:281
5670991911,6cyclictest23594-21sshd11:14:201
5670991911,6cyclictest21154-21sshd07:58:471
5670991911,6cyclictest13316-21cron09:40:011
567099191,14cyclictest30694-21ssh10:25:141
566299191,8cyclictest12013-21fschecks_time07:50:170
566299191,5cyclictest0-21swapper/009:16:030
566299191,15cyclictest8163-21cron08:20:010
566299191,15cyclictest28914-21grep07:30:270
5662991910,7cyclictest9372-21cat11:40:280
5662991910,6cyclictest0-21swapper/007:25:140
568699189,6cyclictest0-21swapper/310:25:183
568699188,3cyclictest0-21swapper/311:48:493
568699187,7cyclictest28366irq/58-eth011:05:123
5686991811,5cyclictest28366irq/58-eth011:20:013
568699181,13cyclictest0-21swapper/308:10:163
568699181,13cyclictest0-21swapper/307:25:133
5686991810,6cyclictest28366irq/58-eth011:50:133
5686991810,6cyclictest28366irq/58-eth009:30:123
567099188,8cyclictest18299-21/usr/sbin/munin08:30:171
567099184,10cyclictest2004-21kthreadcore11:20:181
567099184,10cyclictest0-21swapper/108:40:131
5670991813,3cyclictest20549-21cpu10:55:131
567099181,14cyclictest765-21kthreadcore08:10:201
567099181,14cyclictest15058-21kthreadcore08:25:221
567099181,14cyclictest15058-21kthreadcore08:25:221
5670991810,5cyclictest7889-21ssh10:30:151
567099180,16cyclictest18939-21ls11:30:011
567099180,16cyclictest18939-21ls11:30:011
566299183,4cyclictest131rcu_preempt08:50:300
566299183,10cyclictest6971-21ssh12:15:240
566299182,11cyclictest0-21swapper/008:00:140
5662991811,4cyclictest19460-21kthreadcore12:05:190
568699179,6cyclictest28366irq/58-eth009:15:133
568699179,6cyclictest0-21swapper/312:17:173
568699179,6cyclictest0-21swapper/311:40:403
568699179,6cyclictest0-21swapper/310:20:463
568699179,6cyclictest0-21swapper/310:06:033
568699172,9cyclictest0-21swapper/308:15:043
568699172,9cyclictest0-21swapper/307:50:143
568699172,9cyclictest0-21swapper/307:45:133
568699172,9cyclictest0-21swapper/307:40:133
568699172,11cyclictest0-21swapper/309:55:113
568699172,11cyclictest0-21swapper/309:55:113
568699171,12cyclictest0-21swapper/307:20:193
568699171,11cyclictest0-21swapper/309:45:293
568699171,10cyclictest0-21swapper/311:10:283
567099177,7cyclictest12340-21cut08:55:201
567099174,10cyclictest1993-21munin-run07:40:011
567099173,9cyclictest0-21swapper/109:25:131
567099171,7cyclictest26664-21sed08:05:181
567099171,12cyclictest32157-21diskstats11:55:161
567099170,12cyclictest8829-21ssh12:00:151
566299179,5cyclictest23862-21diskmemload09:40:110
566299174,10cyclictest2776-21kthreadcore08:45:210
566299171,13cyclictest30789-21apt-get08:10:010
566299171,12cyclictest10372-21munin-run07:15:010
568699168,3cyclictest0-21swapper/310:15:513
568699163,7cyclictest0-21swapper/309:05:123
568699162,9cyclictest0-21swapper/308:40:133
568699162,9cyclictest0-21swapper/308:00:133
568699162,9cyclictest0-21swapper/307:10:153
568699161,9cyclictest0-21swapper/308:50:043
568699161,9cyclictest0-21swapper/308:10:023
567099162,10cyclictest3416-21fschecks_time10:10:151
567099161,8cyclictest10733-21find07:45:281
567099161,12cyclictest20388-21cut07:25:151
567099161,11cyclictest25253-21kthreadcore08:35:201
567099160,9cyclictest10356-21cut08:20:221
566299162,11cyclictest11164-21irqrtprio07:15:190
566299161,11cyclictest5196-21kthreadcore07:40:210
568699152,9cyclictest0-21swapper/308:55:123
568699151,11cyclictest0-21swapper/307:20:013
5686991510,3cyclictest28466irq/59-eth008:20:163
567099142,9cyclictest5733-21wc07:10:011
566299141,10cyclictest5660-21cat07:10:010
56869991,5cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional