You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-31 - 18:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Aug 31, 2026 12:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6315998349,19cyclictest35966irq/58-eth011:00:133
628499593,43cyclictest10806-21ssh10:50:141
628499593,43cyclictest10806-21ssh10:50:131
626799592,41cyclictest10383-21ssh09:20:100
626799582,44cyclictest6153-21ssh12:35:130
628499574,40cyclictest16125-21ssh10:35:141
626799573,46cyclictest32678-21ssh11:20:120
628499526,35cyclictest0-21swapper/110:30:151
628499495,36cyclictest13028-21ssh11:45:121
628499494,37cyclictest28386-21ssh09:30:121
628499481,43cyclictest14649-21ssh11:10:111
626799482,41cyclictest14082-21ssh09:40:130
631599463,26cyclictest35966irq/58-eth012:15:143
6293994510,18cyclictest12477-21ssh10:15:122
626799452,33cyclictest19402-21ssh09:25:120
626799451,29cyclictest22156-21ssh11:50:140
628499443,38cyclictest2473-21ssh12:15:131
628499432,33cyclictest0-21swapper/109:25:101
629399412,20cyclictest30142-21ssh11:54:162
628499411,4cyclictest0-21swapper/107:45:141
628499404,25cyclictest5544-21ssh11:05:131
6267994019,17cyclictest12-21ksoftirqd/008:05:110
6267994012,21cyclictest12-21ksoftirqd/007:35:130
628499391,11cyclictest30533-21apt-get07:35:141
626799381,30cyclictest7898-21ssh12:00:130
626799381,29cyclictest19859-21ssh10:55:140
631599374,20cyclictest35966irq/58-eth010:15:113
626799376,22cyclictest12-21ksoftirqd/007:40:140
631599361,20cyclictest35966irq/58-eth010:30:153
631599360,22cyclictest35966irq/58-eth007:40:133
628499361,19cyclictest0-21swapper/109:40:141
628499352,29cyclictest536-21usb-storage09:15:111
628499352,13cyclictest6098-21/usr/sbin/munin12:35:321
628499351,9cyclictest29572-21unixbench_multi07:30:291
628499351,9cyclictest26607-21sort09:10:211
628499351,9cyclictest20956-21tr10:55:201
628499351,9cyclictest11229-21kthreadcore09:55:211
628499351,11cyclictest29982-21ssh11:00:201
628499351,11cyclictest17137-21kthreadcore07:20:191
6267993513,17cyclictest12-21ksoftirqd/008:55:120
6315993418,11cyclictest35966irq/58-eth010:55:153
631599341,20cyclictest35966irq/58-eth011:50:133
628499346,3cyclictest0-21swapper/111:34:101
628499341,9cyclictest484-21kernelversion09:50:191
628499341,9cyclictest15185-21irqcore09:00:181
628499341,9cyclictest10012-21/usr/sbin/munin08:55:141
628499341,7cyclictest26960-21cpuspeed08:05:141
628499341,7cyclictest1705-21/usr/sbin/munin10:45:241
628499341,10cyclictest8060-21cut12:00:171
628499341,10cyclictest4515-21sed08:15:191
628499341,10cyclictest20305-21ssh12:25:011
628499335,5cyclictest23772-21sh10:20:221
628499331,9cyclictest26777-21awk09:45:251
628499331,8cyclictest5718-21tr07:40:221
628499331,8cyclictest5382-21irqcore09:35:181
628499331,8cyclictest17814-21kthreadcore12:05:191
628499331,7cyclictest22860-21irqstats11:50:201
628499331,10cyclictest8293-21cut08:50:211
628499331,10cyclictest31560-21cat10:05:301
628499330,10cyclictest17189-21ls07:55:011
626799333,23cyclictest8858-21ssh09:55:130
629399323,18cyclictest10800-21sh10:50:142
629399323,18cyclictest10800-21sh10:50:132
629399322,22cyclictest29555-21ssh12:30:132
628499323,13cyclictest0-21swapper/110:10:111
628499321,9cyclictest24759-21tr08:35:201
628499321,9cyclictest20647-21cut08:30:211
628499321,8cyclictest7645-21latency_hist12:00:011
628499321,8cyclictest13745-21cut10:15:191
628499321,7cyclictest21489-21sed07:25:201
628499321,6cyclictest10829-21sed09:20:151
628499321,10cyclictest15214-21unixbench_singl07:15:301
628499320,9cyclictest3932-21/usr/sbin/munin11:40:301
628499320,9cyclictest25944-21kthreadcore11:15:231
631599314,16cyclictest35966irq/58-eth010:05:113
631599313,16cyclictest35966irq/58-eth011:10:103
6293993125,3cyclictest24215-21diskmemload11:00:122
629399312,22cyclictest2630-21grep12:15:172
629399312,17cyclictest28387-21ssh09:30:112
628499311,8cyclictest9260-21/usr/sbin/munin11:25:131
628499311,8cyclictest31441-21grep08:40:231
628499311,8cyclictest25257-21df_inode10:40:151
628499311,7cyclictest31190-21switchtime11:35:271
628499311,7cyclictest31190-21switchtime11:35:271
628499311,7cyclictest26534-21sed12:10:201
628499311,7cyclictest17933-21df10:00:131
628499311,6cyclictest30627-21diskstats10:25:151
626799313,21cyclictest31160-21ssh11:55:140
631599303,17cyclictest35966irq/58-eth011:55:123
629399303,17cyclictest27346irq/40-dwc2_hso11:45:132
629399302,18cyclictest27312-21ssh11:35:152
629399302,18cyclictest27312-21ssh11:35:142
628499302,9cyclictest786-21irqcore11:20:171
628499301,9cyclictest3911-21ntpq08:45:261
628499301,9cyclictest20545-21acpi12:25:131
628499301,7cyclictest29565-21ssh12:30:121
628499300,8cyclictest6471-21cut07:10:031
626799302,21cyclictest4996-21ssh09:35:150
629399293,20cyclictest27346irq/40-dwc2_hso11:05:132
629399292,20cyclictest0-21swapper/208:10:142
628499291,9cyclictest25666-21mailstats08:00:271
628499291,8cyclictest32098-21irqcore08:10:171
628499291,8cyclictest20831-21grep07:55:281
628499291,8cyclictest14378-21kthreadcore08:25:191
628499291,8cyclictest11229-21tr08:20:211
631599283,18cyclictest0-21swapper/312:30:573
6315992821,5cyclictest0-21swapper/312:35:143
629399283,18cyclictest27346irq/40-dwc2_hso09:05:132
6293992823,3cyclictest261rcuc/209:35:152
629399282,18cyclictest27346irq/40-dwc2_hso10:00:112
629399282,16cyclictest27346irq/40-dwc2_hso11:10:102
629399282,16cyclictest27346irq/40-dwc2_hso07:25:132
628499281,9cyclictest6540-21apt-get07:10:121
628499281,7cyclictest21898-21cut09:05:211
631599271,15cyclictest35966irq/58-eth007:15:133
629399273,14cyclictest18336-21ssh11:30:142
629399272,19cyclictest0-21swapper/209:00:142
629399272,19cyclictest0-21swapper/208:25:132
629399271,18cyclictest27346irq/40-dwc2_hso08:50:132
626799273,20cyclictest23152-21ssh09:45:140
626799272,19cyclictest2362-21ssh12:30:580
6315992612,9cyclictest35966irq/58-eth012:00:123
629399263,20cyclictest8849-21ssh09:55:142
629399263,18cyclictest27346irq/40-dwc2_hso11:15:162
629399262,16cyclictest27346irq/40-dwc2_hso10:35:132
629399262,15cyclictest27346irq/40-dwc2_hso08:30:142
631599252,18cyclictest0-21swapper/312:13:163
629399255,15cyclictest24623-21ssh09:10:152
629399254,9cyclictest0-21swapper/212:25:142
629399253,16cyclictest27346irq/40-dwc2_hso12:05:142
6293992513,9cyclictest27565-21ssh10:58:542
629399251,14cyclictest27346irq/40-dwc2_hso07:20:132
629399251,11cyclictest852366ssh09:18:152
629399250,15cyclictest25846-21find12:10:122
626799259,7cyclictest12-21ksoftirqd/011:27:450
6267992511,10cyclictest6256-21cyclictest11:37:170
6267992511,10cyclictest6256-21cyclictest11:37:160
6315992412,9cyclictest0-21swapper/311:05:153
631599241,17cyclictest35966irq/58-eth009:50:133
629399249,6cyclictest1867-21diskstats10:45:152
629399243,17cyclictest11492-21ssh12:20:132
629399242,16cyclictest0-21swapper/211:40:172
629399241,8cyclictest14145-21ssh12:39:162
6293992411,5cyclictest19301-21ssh09:25:062
629399241,14cyclictest31150-21ssh11:55:132
626799242,17cyclictest28894-21ssh11:00:120
6267992418,2cyclictest0-21swapper/008:00:130
6267992415,7cyclictest12-21ksoftirqd/012:15:130
631599238,7cyclictest35966irq/58-eth009:10:143
631599231,11cyclictest35966irq/58-eth008:30:133
629399239,11cyclictest23120-21apt-get09:45:142
629399233,15cyclictest27346irq/40-dwc2_hso10:40:152
629399233,15cyclictest27346irq/40-dwc2_hso08:40:132
629399233,14cyclictest0-21swapper/210:30:152
629399232,17cyclictest30439-21netstat10:05:222
629399232,14cyclictest27346irq/40-dwc2_hso10:25:112
6293992312,9cyclictest17513-21ssh09:23:132
6293992312,9cyclictest11601-21ssh10:14:262
6293992311,9cyclictest16097-21kthreadcore09:40:202
629399231,18cyclictest13004-21ntpq11:25:282
6293992311,10cyclictest14639-21ssh12:02:492
626799238,6cyclictest131rcu_preempt11:45:310
626799238,4cyclictest12-21ksoftirqd/010:32:050
626799233,15cyclictest16844-21apt-get12:05:130
6267992311,6cyclictest12-21ksoftirqd/010:05:100
631599223,11cyclictest35966irq/58-eth008:00:143
6315992212,7cyclictest35966irq/58-eth009:35:253
631599221,13cyclictest0-21swapper/310:35:133
629399224,14cyclictest27346irq/40-dwc2_hso08:45:142
629399222,18cyclictest23969-21kthreadcore08:35:182
629399222,15cyclictest0-21swapper/209:50:132
629399221,13cyclictest21554-21ssh10:20:142
629399221,12cyclictest32752-21sort11:20:132
629399220,18cyclictest25835-21grep07:30:172
626799224,13cyclictest0-21swapper/010:15:110
626799223,14cyclictest12-21ksoftirqd/007:10:120
626799222,16cyclictest3432-21apt-get10:10:120
626799222,14cyclictest24808-21cut09:10:160
6267992212,6cyclictest24215-21diskmemload11:15:260
6267992210,3cyclictest12-21ksoftirqd/010:35:290
631599213,6cyclictest0-21swapper/311:37:173
631599213,6cyclictest0-21swapper/311:37:163
631599212,12cyclictest35966irq/58-eth009:25:123
629399217,7cyclictest27-21ksoftirqd/208:00:132
629399214,12cyclictest0-21swapper/208:20:132
629399213,15cyclictest27346irq/40-dwc2_hso07:50:142
629399211,16cyclictest2921-21apt-get07:40:132
629399211,10cyclictest15920-21sendmail07:20:012
626799219,4cyclictest12-21ksoftirqd/010:41:180
626799219,3cyclictest12-21ksoftirqd/009:17:280
6267992115,3cyclictest12-21ksoftirqd/007:30:290
6267992113,5cyclictest0-21swapper/007:10:020
626799211,17cyclictest19629-21apt-get09:05:110
6267992111,7cyclictest12-21ksoftirqd/009:50:110
626799211,15cyclictest13758-21cpuspeed08:25:130
626799211,10cyclictest32275-21sed08:10:180
631599209,6cyclictest0-21swapper/311:43:023
631599201,9cyclictest35966irq/58-eth011:30:153
631599201,15cyclictest0-21swapper/309:40:213
631599201,13cyclictest35966irq/58-eth007:25:133
631599201,12cyclictest0-21swapper/310:50:153
631599201,12cyclictest0-21swapper/310:50:143
629399203,14cyclictest12367-21latency_hist07:50:012
629399202,12cyclictest0-21swapper/207:55:132
629399201,17cyclictest0-21swapper/207:10:132
629399201,16cyclictest29308-21kthreadcore08:05:202
629399201,10cyclictest6763-21kthreadcore08:15:212
626799209,8cyclictest21169-21grep07:55:290
626799208,9cyclictest18575-21fw_packets11:30:180
626799207,10cyclictest12-21ksoftirqd/008:30:260
626799203,2cyclictest131rcu_preempt10:50:420
626799203,2cyclictest131rcu_preempt10:50:410
626799203,2cyclictest131rcu_preempt09:30:210
626799202,2cyclictest131rcu_preempt12:20:080
626799202,15cyclictest630-21Async09:36:310
626799202,14cyclictest10473-21ssh11:05:340
6267992011,6cyclictest12610-21sshd07:50:170
626799201,15cyclictest24304-21/usr/sbin/munin12:25:260
626799201,14cyclictest5464-21ntp_states10:45:270
6267992010,6cyclictest25208-21ntp_kernel_pll_10:20:270
631599198,8cyclictest0-21swapper/312:08:493
631599193,11cyclictest0-21swapper/311:45:173
631599192,8cyclictest0-21swapper/309:45:133
631599192,12cyclictest0-21swapper/312:27:163
631599192,12cyclictest0-21swapper/312:20:573
6315991913,4cyclictest0-21swapper/310:20:213
6315991911,6cyclictest35966irq/58-eth011:20:133
6315991911,5cyclictest35966irq/58-eth010:45:143
6315991911,5cyclictest35966irq/58-eth009:30:383
631599191,13cyclictest0-21swapper/309:00:293
631599191,12cyclictest0-21swapper/307:10:023
631599191,11cyclictest0-21swapper/308:55:013
629399196,10cyclictest6358-21latency_hist07:10:012
629399191,9cyclictest30654-21fschecks_count07:35:162
629399191,9cyclictest10159-21fschecks_count08:55:152
626799198,2cyclictest12-21ksoftirqd/010:01:370
626799196,10cyclictest21094-21grep07:25:160
626799191,2cyclictest131rcu_preempt11:10:090
6267991910,6cyclictest5410-21fschecks_count08:50:160
626799190,3cyclictest131rcu_preempt12:15:010
631599189,6cyclictest0-21swapper/309:55:403
631599182,10cyclictest0-21swapper/308:20:133
6315991811,5cyclictest0-21swapper/310:40:313
631599181,13cyclictest0-21swapper/311:15:313
631599181,13cyclictest0-21swapper/307:15:023
631599181,11cyclictest0-21swapper/308:25:133
631599181,11cyclictest0-21swapper/308:10:133
631599181,10cyclictest0-21swapper/307:20:133
6315991810,6cyclictest35966irq/58-eth009:20:113
6315991810,6cyclictest0-21swapper/311:25:063
6315991810,6cyclictest0-21swapper/310:01:343
6315991810,6cyclictest0-21swapper/309:18:043
6315991810,5cyclictest35966irq/58-eth010:10:173
626799189,6cyclictest14955-21fschecks_count09:00:170
626799184,9cyclictest0-21swapper/008:35:120
626799183,13cyclictest18590-21kthreadcore07:20:220
626799181,9cyclictest30494-21apt-get10:25:120
626799181,3cyclictest131rcu_preempt08:20:120
626799180,9cyclictest1124-21kernelversion08:45:190
631599179,6cyclictest0-21swapper/310:25:103
631599171,9cyclictest0-21swapper/308:40:023
631599171,10cyclictest0-21swapper/307:55:153
626799179,6cyclictest12630-21cut07:15:200
626799174,3cyclictest12-21ksoftirqd/007:45:120
6267991712,3cyclictest0-21swapper/008:15:250
626799171,12cyclictest28215-21diskstats08:40:160
631599161,12cyclictest0-21swapper/309:05:283
631599161,10cyclictest0-21swapper/308:45:143
631599151,10cyclictest0-21swapper/308:40:143
631599141,9cyclictest35966irq/58-eth008:55:133
631599141,9cyclictest0-21swapper/308:05:133
631599141,7cyclictest0-21swapper/307:35:173
631599141,10cyclictest35966irq/58-eth007:45:133
631599132,6cyclictest0-21swapper/307:30:133
631599131,7cyclictest0-21swapper/307:50:163
631599131,10cyclictest0-21swapper/308:15:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional