You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-20 - 02:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Sep 20, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
976799604,44cyclictest9648-21ssh23:40:120
9795995924,24cyclictest35966irq/58-eth021:45:133
977699595,7cyclictest131rcu_preempt19:35:131
977699576,37cyclictest26168-21ssh22:20:131
978799535,34cyclictest22609-21ssh00:05:142
978799532,19cyclictest19147-21ssh00:37:472
9795995012,25cyclictest35966irq/58-eth021:50:153
977699483,31cyclictest29043-21ssh23:15:131
9795994730,10cyclictest35966irq/58-eth022:10:163
978799473,22cyclictest13149-21ssh21:55:132
977699473,4cyclictest131rcu_preempt00:35:151
977699461,37cyclictest9053-21ssh21:35:141
9767994529,8cyclictest0-21swapper/019:55:150
977699444,27cyclictest32365-21ssh21:30:141
976799444,30cyclictest17505-21ssh00:20:140
9767994420,14cyclictest12-21ksoftirqd/019:15:140
978799431,26cyclictest17052-21ssh22:15:142
978799431,26cyclictest17052-21ssh22:15:142
976799436,31cyclictest23282-21ssh21:25:140
976799432,30cyclictest31322-21ssh22:05:140
979599414,22cyclictest35966irq/58-eth019:15:133
977699411,5cyclictest131rcu_preempt21:55:131
977699411,37cyclictest18774-21ssh23:45:151
9776994111,19cyclictest21-21ksoftirqd/119:30:141
979599391,23cyclictest35966irq/58-eth022:25:123
978799392,18cyclictest30215-21sh00:25:252
979599382,22cyclictest35966irq/58-eth021:20:153
979599375,20cyclictest35966irq/58-eth022:55:143
977699372,7cyclictest131rcu_preempt19:40:141
977699361,26cyclictest13521-21ssh00:00:131
979599353,20cyclictest35966irq/58-eth000:30:133
979599351,23cyclictest35966irq/58-eth023:50:133
9787993521,8cyclictest14171-21ssh21:20:162
977699354,18cyclictest16040-21ssh22:50:151
977699351,30cyclictest10979-21ssh23:05:141
9787993418,11cyclictest32379-21sh21:30:162
977699344,19cyclictest10930-21kworker/u8:023:10:151
977699340,21cyclictest21-21ksoftirqd/122:25:141
9787993326,3cyclictest24546irq/40-fe98000019:20:132
978799325,16cyclictest27346irq/40-dwc2_hso21:35:152
978799322,14cyclictest20244-21kworker/u8:000:22:442
976799325,16cyclictest844-21munin-node19:50:130
976799324,19cyclictest8017-21ssh22:10:170
976799322,22cyclictest26614-21ssh00:25:150
9795993115,8cyclictest0-21swapper/323:10:133
9787993117,10cyclictest21120-21sh22:35:142
978799311,23cyclictest20077-21ssh23:10:152
977699313,19cyclictest27745-21ssh23:50:131
976799312,5cyclictest0-21swapper/020:05:150
976799312,5cyclictest0-21swapper/020:05:150
976799311,20cyclictest392-21ssh23:35:110
976799311,20cyclictest0-21swapper/021:50:150
978799305,16cyclictest27346irq/40-dwc2_hso20:30:132
976799307,13cyclictest630-21Async09:36:310
9795992921,3cyclictest33-21ksoftirqd/322:45:143
979599291,20cyclictest35966irq/58-eth021:40:153
978799299,10cyclictest27-21ksoftirqd/219:50:132
978799292,21cyclictest0-21swapper/200:10:152
978799292,21cyclictest0-21swapper/200:10:152
978799291,19cyclictest27346irq/40-dwc2_hso23:55:132
9776992912,11cyclictest0-21swapper/120:50:131
9776992912,11cyclictest0-21swapper/120:50:121
976799292,21cyclictest1896-21ssh23:00:140
976799292,18cyclictest31694-21ssh00:10:140
976799292,18cyclictest31694-21ssh00:10:130
9795992816,4cyclictest0-21swapper/321:35:013
9776992814,6cyclictest21-21ksoftirqd/119:10:121
9767992817,6cyclictest0-21swapper/000:05:140
9767992813,3cyclictest0-21swapper/019:45:160
976799281,21cyclictest0-21swapper/022:00:110
979599273,14cyclictest35966irq/58-eth019:50:123
978799273,16cyclictest27346irq/40-dwc2_hso23:20:142
978799273,15cyclictest22176-21ssh22:00:122
978799272,20cyclictest0-21swapper/223:15:142
978799272,17cyclictest27346irq/40-dwc2_hso22:20:142
978799272,16cyclictest27346irq/40-dwc2_hso21:10:132
977699276,17cyclictest17495-21ssh00:20:131
977699272,19cyclictest15117-21ssh00:17:421
977699271,21cyclictest4479-21ssh23:55:151
977699271,18cyclictest2611-21ssh22:05:281
977699271,18cyclictest21123-21ssh22:35:141
9767992718,4cyclictest0-21swapper/021:00:140
9767992716,5cyclictest0-21swapper/020:10:120
976799271,19cyclictest8366-21ssh00:15:150
979599261,19cyclictest0-21swapper/323:30:143
978799264,16cyclictest27346irq/40-dwc2_hso19:30:132
978799263,14cyclictest27346irq/40-dwc2_hso00:15:142
978799261,20cyclictest24072-21ls19:55:252
976799269,6cyclictest0-21swapper/023:25:130
976799269,6cyclictest0-21swapper/023:25:120
976799261,21cyclictest3266-21ssh00:30:140
979599251,16cyclictest0-21swapper/322:05:143
978799255,13cyclictest27346irq/40-dwc2_hso23:00:142
978799252,9cyclictest0-21swapper/220:45:142
977699253,13cyclictest0-21swapper/123:35:121
9767992520,3cyclictest12-21ksoftirqd/022:20:180
9767992514,8cyclictest27733-21apt-get23:50:130
9767992512,3cyclictest0-21swapper/022:41:590
976799251,17cyclictest27340-21ssh21:45:130
979599242,10cyclictest0-21swapper/300:22:443
9795992416,6cyclictest0-21swapper/323:40:133
9795992414,7cyclictest35966irq/58-eth000:05:153
979599241,15cyclictest35966irq/58-eth023:15:143
979599241,12cyclictest35966irq/58-eth021:55:143
978799242,11cyclictest25181-21ssh22:55:152
9787992416,5cyclictest19230-21cron19:20:012
976799243,9cyclictest131rcu_preempt22:53:300
976799241,18cyclictest24738-21sshd22:35:240
979599233,15cyclictest0-21swapper/300:25:143
978799234,12cyclictest0-21swapper/221:25:142
978799232,6cyclictest0-21swapper/200:31:252
978799232,16cyclictest27346irq/40-dwc2_hso20:50:112
978799232,16cyclictest27346irq/40-dwc2_hso20:50:102
978799231,8cyclictest14775-21ssh23:25:142
978799231,8cyclictest14775-21ssh23:25:142
9787992311,9cyclictest13390-21ssh00:00:052
978799231,11cyclictest28179-21ssh21:45:192
9787992310,10cyclictest18741-21apt-get23:45:152
978799230,20cyclictest28020-21ls23:50:162
977699234,12cyclictest0-21swapper/119:20:131
977699232,16cyclictest23849-21ssh23:30:151
977699231,10cyclictest2702766ssh22:00:301
976799239,9cyclictest8952-21apt-get20:50:120
976799239,9cyclictest8952-21apt-get20:50:120
976799237,8cyclictest12-21ksoftirqd/023:14:560
976799235,9cyclictest12-21ksoftirqd/020:25:170
976799233,5cyclictest12-21ksoftirqd/000:00:250
976799232,3cyclictest131rcu_preempt23:55:250
976799232,15cyclictest18773-21diskmemload23:45:150
976799232,14cyclictest18200-21ssh21:40:150
9767992315,4cyclictest32353-21apt-get21:30:140
978799224,12cyclictest27346irq/40-dwc2_hso21:05:132
978799222,17cyclictest28842-21cron19:30:002
9787992211,9cyclictest0-21swapper/220:05:282
9787992211,9cyclictest0-21swapper/220:05:282
978799221,16cyclictest4017-21acpi21:50:122
978799221,11cyclictest32121-21ssh22:05:202
978799221,11cyclictest13465-21tr22:30:202
9787992210,5cyclictest23408-21ssh21:40:582
978799220,15cyclictest0-21swapper/223:40:052
977699225,9cyclictest0-21swapper/122:15:131
977699225,9cyclictest0-21swapper/122:15:131
977699221,18cyclictest5905-21ls23:00:301
977699220,6cyclictest27276-21irqrtprio00:25:191
976799225,6cyclictest12-21ksoftirqd/000:37:030
976799225,2cyclictest131rcu_preempt22:48:590
976799223,5cyclictest12-21ksoftirqd/023:30:150
976799223,3cyclictest131rcu_preempt23:15:150
976799221,16cyclictest5750-21ssh23:20:140
979599216,7cyclictest35966irq/58-eth020:40:133
979599212,13cyclictest0-21swapper/300:37:473
979599211,14cyclictest0-21swapper/322:20:143
978799219,4cyclictest10231-21irqcore19:10:182
978799213,10cyclictest5429-21irqcore21:15:192
978799211,9cyclictest27790-21timerandwakeup23:30:302
9787992113,5cyclictest0-21swapper/220:35:192
978799211,12cyclictest32310-21cut20:40:182
978799211,11cyclictest8011-21ssh22:10:162
978799211,11cyclictest7557-21sed22:45:192
977699214,3cyclictest131rcu_preempt21:16:051
977699212,15cyclictest25155-21ssh22:55:131
9776992112,4cyclictest15613-21sh22:30:241
976799218,7cyclictest12-21ksoftirqd/020:30:280
976799218,5cyclictest12-21ksoftirqd/023:05:140
976799218,5cyclictest12-21ksoftirqd/021:11:560
976799217,4cyclictest12-21ksoftirqd/020:45:010
976799216,9cyclictest12-21ksoftirqd/022:25:220
976799214,6cyclictest12-21ksoftirqd/022:57:540
9767992114,2cyclictest5012-21apt-get21:15:140
976799211,15cyclictest15626-21ssh22:30:250
979599209,6cyclictest35966irq/58-eth000:10:143
979599209,6cyclictest35966irq/58-eth000:10:143
979599202,12cyclictest0-21swapper/300:15:253
9795992012,5cyclictest35966irq/58-eth023:57:253
978799202,9cyclictest27331-21kthreadcore20:00:212
978799202,9cyclictest20056-21kthreadcore21:00:222
978799202,15cyclictest636-21fschecks_time23:35:152
978799202,10cyclictest12241-21apt-key20:20:022
978799201,9cyclictest3734-21kthreadcore19:35:222
978799201,9cyclictest30442-21sort22:40:172
978799201,9cyclictest15737-21kthreadcore20:55:212
978799201,9cyclictest13439-21kthreadcore23:05:232
978799201,16cyclictest19741-21kthreadcore20:25:192
978799201,16cyclictest16330-21ssh22:50:192
978799201,10cyclictest6913-21timerwakeupswit22:25:292
978799201,10cyclictest13334-21kthreadcore20:20:222
978799201,10cyclictest11288-21grep19:45:192
978799201,10cyclictest10879-21latency_hist19:45:012
977699202,13cyclictest30248-21ssh22:40:131
9776992012,4cyclictest0-21swapper/121:05:141
977699201,15cyclictest9264-21kthreadcore20:15:201
977699201,14cyclictest27946-21diskmemload22:45:151
977699201,13cyclictest7454-21ls20:15:011
9776992010,7cyclictest2597-21cron20:10:011
9776992010,7cyclictest2597-21cron20:10:011
977699200,15cyclictest28913-21ls19:30:011
976799204,9cyclictest12-21ksoftirqd/021:55:460
976799204,5cyclictest12-21ksoftirqd/019:25:290
976799203,3cyclictest131rcu_preempt20:55:260
976799201,5cyclictest12-21ksoftirqd/019:40:240
9767992010,3cyclictest0-21swapper/021:23:490
979599199,8cyclictest0-21swapper/323:37:253
979599197,9cyclictest0-21swapper/323:49:013
9795991913,4cyclictest0-21swapper/321:15:303
9795991911,6cyclictest0-21swapper/322:32:393
9795991910,6cyclictest0-21swapper/321:28:183
978799191,8cyclictest3631-21kthreadcore20:10:192
978799191,15cyclictest9832-21latency_hist19:10:012
977699192,12cyclictest844-21munin-node21:20:161
9776991913,4cyclictest27299-21/usr/sbin/munin21:45:171
9776991911,6cyclictest9762-21cyclictest00:10:141
9776991911,6cyclictest9762-21cyclictest00:10:141
9776991911,5cyclictest11959-21latency_hist00:35:021
977699191,14cyclictest18387-21ssh23:25:251
977699191,14cyclictest18387-21ssh23:25:251
976799198,3cyclictest131rcu_preempt22:15:430
976799198,3cyclictest131rcu_preempt22:15:430
976799197,3cyclictest131rcu_preempt21:35:170
976799196,4cyclictest131rcu_preempt19:30:260
976799196,3cyclictest131rcu_preempt20:05:020
976799196,11cyclictest4817-21sed20:45:190
976799194,8cyclictest12-21ksoftirqd/021:05:170
976799191,5cyclictest12-21ksoftirqd/019:20:250
976799191,3cyclictest131rcu_preempt20:35:270
9767991911,5cyclictest151rcuc/019:35:150
976799190,14cyclictest9663-21latency_hist19:10:010
979599184,10cyclictest35966irq/58-eth022:50:153
979599182,10cyclictest0-21swapper/321:00:153
979599182,10cyclictest0-21swapper/320:55:153
979599182,10cyclictest0-21swapper/320:30:153
9795991812,4cyclictest0-21swapper/322:18:343
9795991812,4cyclictest0-21swapper/322:18:333
9795991812,4cyclictest0-21swapper/322:00:203
9795991812,4cyclictest0-21swapper/300:00:273
9795991811,4cyclictest0-21swapper/321:35:143
979599181,12cyclictest0-21swapper/323:05:023
979599181,12cyclictest0-21swapper/320:25:023
9795991810,6cyclictest0-21swapper/322:36:263
9795991810,6cyclictest0-21swapper/321:11:343
977699188,7cyclictest26567-21sed00:05:301
977699186,3cyclictest131rcu_preempt21:40:211
977699185,10cyclictest15823-21latency_hist19:50:021
977699182,14cyclictest18112-21kthreadcore20:25:171
977699181,8cyclictest21542-21kthreadcore19:55:211
977699181,7cyclictest10031-21/usr/sbin/munin23:40:171
9776991812,4cyclictest26749-21munin-plugin-st20:35:011
977699181,14cyclictest4624-21irqrtprio21:50:191
977699180,12cyclictest23577-21idleruntime21:25:181
976799186,3cyclictest131rcu_preempt20:20:280
976799182,8cyclictest12-21ksoftirqd/020:15:180
979599178,6cyclictest0-21swapper/322:41:023
979599171,9cyclictest35966irq/58-eth023:25:133
979599171,9cyclictest35966irq/58-eth023:25:133
9795991711,4cyclictest0-21swapper/323:05:183
979599171,12cyclictest33-21ksoftirqd/320:05:153
979599171,12cyclictest33-21ksoftirqd/320:05:153
979599171,10cyclictest0-21swapper/320:25:113
979599171,10cyclictest0-21swapper/320:15:133
979599171,10cyclictest0-21swapper/320:05:023
979599171,10cyclictest0-21swapper/319:10:233
9795991710,5cyclictest0-21swapper/323:23:263
977699174,10cyclictest4557-21irqcore20:45:181
977699171,8cyclictest17505-21timerwakeupswit20:55:291
977699171,4cyclictest53646usb-storage20:20:051
9776991711,4cyclictest0-21swapper/121:10:281
977699171,13cyclictest9375-21ssh23:20:261
977699171,12cyclictest9061-21kthreadcore22:10:231
977699171,12cyclictest30143-21awk20:05:011
9776991710,5cyclictest0-21swapper/120:40:281
979599168,3cyclictest0-21swapper/319:55:153
979599161,10cyclictest0-21swapper/321:05:143
979599161,10cyclictest0-21swapper/319:30:143
977699162,12cyclictest17000-21kthreadcore19:50:191
977699162,11cyclictest27307-21ps20:35:181
977699162,11cyclictest21998-21ntp_states21:00:271
977699161,12cyclictest14792-21cut19:15:171
979599159,4cyclictest0-21swapper/319:40:303
979599150,10cyclictest35966irq/58-eth020:35:133
979599149,3cyclictest0-21swapper/320:10:163
979599149,3cyclictest0-21swapper/319:45:263
979599141,9cyclictest0-21swapper/319:20:143
977699141,9cyclictest9834-21latency_hist19:10:011
979599138,3cyclictest0-21swapper/319:25:253
979599127,3cyclictest0-21swapper/319:35:133
979599121,7cyclictest0-21swapper/320:45:123
979599111,8cyclictest0-21swapper/320:50:153
979599111,8cyclictest0-21swapper/320:50:153
979599101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional