You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-26 - 21:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Apr 26, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
623699585,9cyclictest131rcu_preempt08:35:120
6267995534,10cyclictest35966irq/58-eth012:10:143
6267995534,10cyclictest35966irq/58-eth012:10:133
624899552,43cyclictest21910-21ssh12:25:141
624899511,38cyclictest20660-21ssh10:55:131
624899502,39cyclictest1331-21ssh11:20:141
6259994632,8cyclictest27-21ksoftirqd/211:40:132
6259994632,8cyclictest27-21ksoftirqd/211:40:122
623699449,9cyclictest131rcu_preempt10:40:120
626799425,22cyclictest35966irq/58-eth011:30:143
625999412,31cyclictest27048-21ssh12:10:112
625999412,31cyclictest27048-21ssh12:10:102
624899415,26cyclictest21-21ksoftirqd/112:15:141
623699411,33cyclictest14198-21ssh09:40:140
625999401,26cyclictest28533-21ssh11:35:142
626799389,14cyclictest35966irq/58-eth009:40:143
626799382,19cyclictest35966irq/58-eth009:35:143
625999382,28cyclictest15303-21ssh11:10:022
6259993821,9cyclictest0-21swapper/209:10:112
6259993812,4cyclictest131rcu_preempt10:40:122
6259993810,23cyclictest131rcu_preempt08:45:132
6236993813,16cyclictest12-21ksoftirqd/011:40:120
6236993813,16cyclictest12-21ksoftirqd/011:40:120
626799373,22cyclictest35966irq/58-eth012:05:133
626799373,22cyclictest35966irq/58-eth012:05:123
625999372,29cyclictest9267-21ssh10:47:442
625999372,18cyclictest31639-21ssh11:54:432
625999372,18cyclictest20645-21ssh11:10:452
626799356,14cyclictest35966irq/58-eth011:45:143
626799353,22cyclictest35966irq/58-eth010:00:133
6267993512,16cyclictest0-21swapper/310:05:133
624899353,23cyclictest211ksoftirqd/107:45:121
624899353,23cyclictest211ksoftirqd/107:45:111
626799344,19cyclictest35966irq/58-eth010:30:123
625999344,20cyclictest27346irq/40-dwc2_hso11:15:142
625999341,17cyclictest1289-21ssh09:15:142
623699345,9cyclictest131rcu_preempt12:30:110
626799333,21cyclictest35966irq/58-eth009:50:133
626799331,20cyclictest35966irq/58-eth012:35:143
624899331,29cyclictest21890-21sh10:20:101
623699331,29cyclictest7687-21rm10:30:130
623699331,24cyclictest21923-21ssh10:20:130
6267993216,11cyclictest35966irq/58-eth007:45:123
6267993216,11cyclictest35966irq/58-eth007:45:123
624899322,27cyclictest12823-21rm12:20:151
624899321,24cyclictest12764-21ssh10:15:141
624899321,24cyclictest12764-21ssh10:15:131
6236993215,5cyclictest131rcu_preempt08:45:130
625999313,19cyclictest0-21swapper/211:20:132
624899314,21cyclictest844-21munin-node08:50:161
624899312,26cyclictest7659-21ssh12:35:121
6236993122,4cyclictest131rcu_preempt12:05:140
6236993122,4cyclictest131rcu_preempt12:05:140
6267993015,3cyclictest0-21swapper/310:15:133
6267993015,3cyclictest0-21swapper/310:15:133
625999303,18cyclictest27346irq/40-dwc2_hso07:25:152
625999302,19cyclictest27346irq/40-dwc2_hso07:30:122
625999302,11cyclictest0-21swapper/209:05:132
624899301,25cyclictest29780-21ssh11:00:121
623699302,21cyclictest7678-21ssh12:35:140
623699301,21cyclictest12780-21find12:20:130
625999293,21cyclictest0-21swapper/211:58:442
625999292,21cyclictest0-21swapper/209:00:112
625999292,20cyclictest0-21swapper/207:35:132
625999292,11cyclictest15729-21ssh12:02:432
624899292,19cyclictest32199-21ssh11:55:071
623699292,7cyclictest131rcu_preempt09:05:140
623699291,25cyclictest17929-21ssh10:00:120
623699291,25cyclictest16825-21ssh10:35:140
626799283,18cyclictest0-21swapper/311:20:443
625999283,20cyclictest0-21swapper/211:04:442
625999283,15cyclictest27346irq/40-dwc2_hso07:55:122
625999281,12cyclictest0-21swapper/211:28:442
623699282,23cyclictest31027-21ssh10:25:140
6236992817,9cyclictest6231-21cyclictest07:50:130
626799274,14cyclictest35966irq/58-eth009:30:123
626799273,16cyclictest35966irq/58-eth010:50:143
626799273,16cyclictest35966irq/58-eth010:50:133
626799273,16cyclictest0-21swapper/311:25:043
626799273,16cyclictest0-21swapper/311:10:013
6267992715,7cyclictest35966irq/58-eth008:00:123
625999272,20cyclictest0-21swapper/207:10:142
624899276,16cyclictest8853-21ssh09:55:121
624899271,21cyclictest32211-21apt-get09:50:131
624899271,20cyclictest15600-21ssh11:10:131
623699274,4cyclictest131rcu_preempt10:50:140
623699274,4cyclictest131rcu_preempt10:50:130
623699272,20cyclictest0-21swapper/009:30:130
623699272,19cyclictest11111-21ssh10:49:530
6236992712,10cyclictest12-21ksoftirqd/007:20:130
626799264,8cyclictest0-21swapper/310:55:123
626799262,17cyclictest0-21swapper/312:04:443
6267992612,6cyclictest36066irq/59-eth010:25:133
625999263,17cyclictest0-21swapper/210:10:122
625999262,9cyclictest24506-21ssh11:30:452
623699267,15cyclictest12-21ksoftirqd/009:10:130
623699265,17cyclictest10391-21ssh09:20:140
626799252,17cyclictest0-21swapper/311:15:433
625999259,12cyclictest0-21swapper/209:45:192
625999253,17cyclictest7675-21ssh10:30:122
625999252,20cyclictest0-21swapper/208:50:162
6259992511,6cyclictest30985-21ssh12:30:132
624899252,17cyclictest3699-21ssh10:10:131
623699259,11cyclictest12-21ksoftirqd/011:01:440
623699252,19cyclictest32228-21ssh09:50:140
626799243,13cyclictest0-21swapper/311:55:073
626799243,13cyclictest0-21swapper/310:47:443
6267992414,7cyclictest0-21swapper/309:25:283
625999245,10cyclictest10655-21sh12:35:232
625999243,6cyclictest0-21swapper/211:45:072
625999243,14cyclictest27346irq/40-dwc2_hso08:15:132
625999242,8cyclictest0-21swapper/212:15:132
625999242,7cyclictest27-21ksoftirqd/210:52:452
625999242,7cyclictest27-21ksoftirqd/210:52:452
625999241,6cyclictest22073-21fschecks_count12:25:162
625999241,19cyclictest27055-21sh10:05:142
624899243,16cyclictest25868-21ssh10:40:121
624899242,16cyclictest6467-21ssh11:05:141
624899241,19cyclictest11642-21ssh10:50:151
624899241,19cyclictest11642-21ssh10:50:151
623699245,14cyclictest0-21swapper/011:05:130
623699242,16cyclictest3719-21ssh10:10:140
623699241,18cyclictest27056-21ssh10:05:140
626799233,13cyclictest0-21swapper/311:50:453
626799231,14cyclictest0-21swapper/310:10:133
625999232,6cyclictest0-21swapper/210:56:442
625999232,15cyclictest0-21swapper/212:05:102
625999232,15cyclictest0-21swapper/212:05:102
625999231,13cyclictest28447-21ssh09:30:122
625999230,13cyclictest30625-21ssh10:25:012
624899238,7cyclictest21-21ksoftirqd/109:00:111
624899233,12cyclictest21-21ksoftirqd/107:40:141
624899232,19cyclictest24282-21diskmemload12:00:151
624899231,19cyclictest201rcuc/111:40:121
624899231,19cyclictest201rcuc/111:40:111
624899231,17cyclictest30971-21ssh12:30:121
623699236,11cyclictest0-21swapper/008:00:140
623699233,13cyclictest1288-21sh09:15:140
623699232,18cyclictest5134-21ssh09:35:140
623699232,18cyclictest30710-21cut07:35:150
626799229,7cyclictest0-21swapper/307:40:143
626799223,13cyclictest0-21swapper/311:10:453
626799223,12cyclictest0-21swapper/311:35:063
6267992212,7cyclictest0-21swapper/311:40:123
6267992212,7cyclictest0-21swapper/311:40:123
6267992211,6cyclictest0-21swapper/308:35:123
626799221,13cyclictest0-21swapper/309:10:143
625999222,15cyclictest0-21swapper/212:23:442
625999222,12cyclictest7484-21latency_hist10:30:012
6259992211,9cyclictest20137-21ssh10:18:302
6259992211,9cyclictest20137-21ssh10:18:292
6259992210,5cyclictest28655-21tr08:05:202
625999220,19cyclictest14523-21ls07:15:242
624899222,4cyclictest131rcu_preempt08:15:141
624899221,18cyclictest2522-21ssh10:45:151
624899220,16cyclictest10242-21cut08:55:151
6236992214,5cyclictest0-21swapper/012:15:160
623699221,19cyclictest10380-21ssh11:25:130
626799213,9cyclictest0-21swapper/312:20:123
6267992114,3cyclictest0-21swapper/308:45:133
625999215,13cyclictest18235-21sh08:30:012
625999214,9cyclictest1568-21kthreadcore09:50:202
625999212,9cyclictest12514-21ssh09:55:252
625999211,2cyclictest131rcu_preempt10:00:122
625999211,16cyclictest12781-21ssh09:20:212
625999211,11cyclictest14153-21/usr/sbin/munin09:40:142
625999211,11cyclictest11085-21kthreadcore08:55:192
625999211,10cyclictest5386-21hwlatdetect09:35:172
623699212,12cyclictest23256-21ssh11:50:140
623699211,6cyclictest23152-21ntp_states11:30:270
6236992112,5cyclictest32210-21ntp_states11:35:250
623699211,16cyclictest32277-21apt-get11:55:130
623699211,16cyclictest19412-21ssh09:25:130
623699210,18cyclictest12134-21ls08:20:230
623699210,18cyclictest11436-21ls07:15:170
623699210,17cyclictest31821-21cut08:10:160
626799209,6cyclictest0-21swapper/312:25:143
626799202,9cyclictest35966irq/58-eth009:15:143
6267992015,3cyclictest0-21swapper/309:20:123
6267992013,3cyclictest0-21swapper/309:05:143
6267992011,6cyclictest35966irq/58-eth010:40:113
626799201,15cyclictest35966irq/58-eth011:00:143
625999206,11cyclictest13581-21latency_hist08:25:022
625999202,9cyclictest15808-21latency07:50:232
625999202,11cyclictest23268-21apt-get08:35:122
625999201,16cyclictest32653-21munin-run08:45:012
6259992011,6cyclictest0-21swapper/208:00:182
625999201,15cyclictest7874-21fschecks_count07:45:162
625999201,15cyclictest7874-21fschecks_count07:45:162
625999201,11cyclictest19652-21tune2fs09:25:162
625999201,10cyclictest31862-21tune2fs08:10:162
625999201,10cyclictest19440-21kthreadcore08:30:192
6259992010,3cyclictest6538-21swap07:40:282
624899207,3cyclictest131rcu_preempt11:26:471
624899202,16cyclictest2820-21ssh11:37:441
624899202,15cyclictest14177-21ssh11:45:121
6248992011,6cyclictest30320-21cron07:35:021
6248992011,5cyclictest0-21swapper/111:15:121
624899201,13cyclictest1942566ssh11:30:141
624899200,17cyclictest26753-21ls09:45:241
623699208,10cyclictest12-21ksoftirqd/011:24:440
623699203,14cyclictest24630-21wc07:25:290
623699202,16cyclictest29667-21ssh11:15:430
623699202,16cyclictest14063-21ssh11:45:070
623699202,13cyclictest26908-21ssh09:45:280
623699201,5cyclictest0-21swapper/012:14:240
623699201,5cyclictest0-21swapper/012:14:240
623699201,2cyclictest131rcu_preempt10:15:150
623699201,2cyclictest131rcu_preempt10:15:140
6236992011,5cyclictest11003-21kthreadcore12:00:230
623699201,13cyclictest7768-21timerandwakeup08:15:290
623699200,15cyclictest9032-21cut09:55:160
626799193,11cyclictest0-21swapper/312:30:143
626799192,10cyclictest0-21swapper/307:20:143
6267991913,3cyclictest0-21swapper/308:40:133
6267991912,2cyclictest0-21swapper/309:55:133
6267991911,3cyclictest0-21swapper/309:45:143
625999191,9cyclictest16247-21fschecks_count07:20:162
625999191,15cyclictest17259-21sh10:35:172
624899196,4cyclictest21-21ksoftirqd/110:05:211
6248991911,4cyclictest21-21ksoftirqd/109:25:121
624899190,16cyclictest6365-21ls07:10:011
623699191,3cyclictest131rcu_preempt08:30:280
6236991912,4cyclictest0-21swapper/007:40:270
623699191,15cyclictest6231-21cyclictest07:10:020
623699191,15cyclictest25532-21ssh12:25:250
626799189,6cyclictest0-21swapper/310:36:433
626799189,6cyclictest0-21swapper/310:22:243
626799183,10cyclictest0-21swapper/312:19:443
626799182,10cyclictest0-21swapper/307:30:133
626799181,9cyclictest0-21swapper/308:55:123
626799181,9cyclictest0-21swapper/308:00:023
6267991811,4cyclictest0-21swapper/307:15:133
624899189,6cyclictest14050-21awk09:20:261
624899182,12cyclictest32646-21apt-get08:45:021
624899182,12cyclictest21589-21missed_timers12:05:231
624899182,12cyclictest21589-21missed_timers12:05:221
624899182,11cyclictest0-21swapper/108:35:131
624899181,6cyclictest24287-21cron09:10:001
624899181,14cyclictest3965-21ls08:15:021
624899181,14cyclictest1485-21tr10:25:231
624899181,13cyclictest20399-21ntp_192.168.11410:35:251
623699189,6cyclictest29571-21ssh11:00:010
623699183,4cyclictest18248-21missed_timers09:00:210
623699181,9cyclictest30326-21latency_hist07:35:020
623699181,15cyclictest15894-21ls11:10:170
626799172,9cyclictest0-21swapper/308:20:143
626799172,9cyclictest0-21swapper/308:15:153
626799172,9cyclictest0-21swapper/307:25:153
626799172,10cyclictest0-21swapper/308:50:163
6267991711,2cyclictest0-21swapper/308:30:133
624899179,6cyclictest21845-21sshd07:55:441
624899179,4cyclictest27080-21/usr/sbin/munin12:10:171
624899179,4cyclictest27080-21/usr/sbin/munin12:10:161
624899176,8cyclictest23517-21wc11:50:161
624899172,11cyclictest0-21swapper/108:30:141
624899172,11cyclictest0-21swapper/108:20:131
624899172,10cyclictest0-21swapper/108:40:141
624899171,7cyclictest2695-21ls07:40:001
624899171,4cyclictest6913-21irqrtprio07:10:181
624899171,12cyclictest7703-21diskmemload10:30:141
624899171,12cyclictest21663-21ntp_states10:00:251
623699177,4cyclictest7122-21kthreadcore07:10:190
623699173,11cyclictest31463-21munin-run08:10:010
623699172,2cyclictest131rcu_preempt08:25:140
623699171,8cyclictest19272-21kthreadcore07:55:200
626799162,9cyclictest0-21swapper/309:00:123
626799162,9cyclictest0-21swapper/308:25:143
6267991611,2cyclictest0-21swapper/307:50:133
624899168,6cyclictest16256-21kthreadcore09:40:221
624899168,6cyclictest15772-21cron07:20:041
624899167,6cyclictest28056-21kthreadcore09:10:231
624899163,10cyclictest15795-21tr07:20:011
624899162,10cyclictest0-21swapper/107:50:141
624899162,10cyclictest0-21swapper/107:25:161
624899161,12cyclictest29939-21cut09:30:191
624899161,11cyclictest5291-21fschecks_count09:35:151
624899161,11cyclictest13746-21diskstats08:25:151
623699169,5cyclictest0-21swapper/008:55:180
623699163,10cyclictest31028-21kthreadcore08:40:210
623699163,10cyclictest11550-21unixbench_multi07:45:280
623699163,10cyclictest11550-21unixbench_multi07:45:270
623699162,11cyclictest5321-21apt-get08:50:130
626799159,3cyclictest0-21swapper/307:35:123
626799156,6cyclictest0-21swapper/308:05:173
626799153,8cyclictest0-21swapper/307:10:163
625999150,13cyclictest6272-21latency_hist07:10:022
624899152,9cyclictest0-21swapper/108:05:151
624899152,11cyclictest22789-21kthreadcore08:00:191
624899151,5cyclictest1280-21apt-get09:15:141
6267991410,2cyclictest0-21swapper/308:10:133
626799111,8cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional