You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 22:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat Mar 21, 2026 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
375399742,30cyclictest24199-21ssh10:05:132
374299561,46cyclictest16887-21ssh12:25:121
3742995327,14cyclictest131rcu_preempt09:55:161
375399511,27cyclictest2312-21ssh09:35:122
374299494,11cyclictest131rcu_preempt12:35:121
3765994825,18cyclictest28366irq/58-eth011:40:123
375399484,16cyclictest22346irq/40-dwc2_hso08:40:122
375399482,27cyclictest31062-21ssh09:15:132
374299483,37cyclictest22212-21ssh12:10:141
374299451,32cyclictest7651-21ssh09:20:111
374299448,7cyclictest21-21ksoftirqd/109:15:131
373399443,16cyclictest0-21swapper/008:45:130
375399439,22cyclictest633-21Async12:11:532
374299423,31cyclictest4206-21sh12:00:121
374299422,7cyclictest131rcu_preempt10:20:131
375399413,19cyclictest22448-21ssh10:40:132
3753994114,13cyclictest633-21Async12:11:532
374299417,23cyclictest21-21ksoftirqd/108:25:131
374299416,25cyclictest21-21ksoftirqd/109:50:121
374299414,25cyclictest11701-21ssh11:10:131
376599405,22cyclictest28366irq/58-eth011:15:143
373399402,27cyclictest16687-21rm09:25:130
3733993926,10cyclictest21672-21diskmemload12:15:100
3733993926,10cyclictest21672-21diskmemload12:15:100
376599382,23cyclictest28366irq/58-eth011:50:143
3765993811,23cyclictest28366irq/58-eth011:05:133
375399385,22cyclictest24334-21ssh11:35:132
373399381,31cyclictest9766-21ssh10:15:130
3742993626,7cyclictest21-21ksoftirqd/107:50:131
3733993613,19cyclictest12-21ksoftirqd/012:30:140
375399352,28cyclictest20337-21ssh10:37:562
375399351,15cyclictest924-21snmpd08:45:142
376599342,18cyclictest28366irq/58-eth010:30:133
376599342,18cyclictest28366irq/58-eth010:30:123
3753993411,20cyclictest27-21ksoftirqd/207:45:112
374299345,6cyclictest131rcu_preempt08:55:121
374299340,27cyclictest0-21swapper/111:00:131
376599331,20cyclictest28366irq/58-eth010:05:133
375399334,13cyclictest15205-21ssh10:00:142
375399331,21cyclictest9767-21ssh11:45:132
375399331,21cyclictest9767-21ssh11:45:122
374299334,7cyclictest131rcu_preempt11:17:561
374299332,20cyclictest21-21ksoftirqd/111:25:141
375399324,9cyclictest633-21Async12:11:532
3753993217,9cyclictest22346irq/40-dwc2_hso09:40:112
3753993216,8cyclictest25759-21ssh09:30:132
375399321,15cyclictest978166ssh10:15:132
375399313,14cyclictest8710-21sshd07:15:142
374299314,19cyclictest27897-21diskmemload10:25:151
374299311,18cyclictest29848-21ssh11:20:111
373399317,18cyclictest13192-21diskmemload12:05:120
373399313,19cyclictest31371-21ssh10:45:110
373399312,23cyclictest10058-21ssh12:38:540
3733993112,15cyclictest12-21ksoftirqd/011:15:120
375399302,20cyclictest22346irq/40-dwc2_hso07:55:132
374299306,9cyclictest131rcu_preempt09:45:141
373399302,22cyclictest11878-21rm12:03:550
375399293,15cyclictest22346irq/40-dwc2_hso12:35:122
375399292,20cyclictest0-21swapper/210:20:142
374299291,24cyclictest2651-21ssh11:05:141
374299291,19cyclictest4389-21ssh10:30:131
374299291,19cyclictest4389-21ssh10:30:121
373399294,19cyclictest27543-21ssh11:55:120
375399284,16cyclictest22346irq/40-dwc2_hso07:25:122
375399283,18cyclictest22346irq/40-dwc2_hso08:30:112
375399283,18cyclictest22346irq/40-dwc2_hso08:30:102
375399282,11cyclictest9296-21ssh11:44:552
374299282,20cyclictest25883-21ssh12:30:161
373399285,4cyclictest131rcu_preempt10:35:110
3733992815,6cyclictest0-21swapper/011:21:500
376599272,15cyclictest28366irq/58-eth009:15:123
375399273,8cyclictest0-21swapper/211:15:552
375399272,20cyclictest924-21snmpd11:01:552
375399272,16cyclictest21672-21diskmemload11:30:132
375399272,10cyclictest0-21swapper/207:35:112
375399271,21cyclictest0-21swapper/210:45:122
375399271,16cyclictest7998-21ssh10:50:152
374299273,15cyclictest16998-21rm10:55:151
373399277,11cyclictest12-21ksoftirqd/010:25:040
3733992712,10cyclictest7617-21/usr/sbin/munin09:20:280
376599263,17cyclictest0-21swapper/310:54:553
376599263,15cyclictest0-21swapper/310:37:553
376599262,17cyclictest0-21swapper/311:13:553
375399264,14cyclictest22346irq/40-dwc2_hso07:10:132
375399261,9cyclictest924-21snmpd12:05:112
373399263,19cyclictest22071-21ssh09:10:140
373399263,13cyclictest21672-21diskmemload09:55:140
3733992614,5cyclictest0-21swapper/010:00:140
3733992613,8cyclictest0-21swapper/008:20:120
376599255,11cyclictest0-21swapper/312:10:153
376599252,16cyclictest0-21swapper/309:40:123
375399253,14cyclictest22346irq/40-dwc2_hso11:55:122
375399252,9cyclictest0-21swapper/208:20:142
375399252,20cyclictest0-21swapper/209:10:132
374299257,11cyclictest21-21ksoftirqd/111:50:131
373399252,16cyclictest22453-21ssh10:40:130
376599243,12cyclictest28366irq/58-eth011:20:123
3765992415,6cyclictest28366irq/58-eth009:50:123
374299243,2cyclictest131rcu_preempt11:30:281
374299242,17cyclictest24379-21apt-get08:05:121
374299241,18cyclictest1-21systemd07:15:151
373399248,3cyclictest131rcu_preempt10:30:270
373399248,3cyclictest131rcu_preempt10:30:260
373399247,12cyclictest12-21ksoftirqd/011:33:550
373399246,13cyclictest12-21ksoftirqd/007:20:120
373399244,4cyclictest12-21ksoftirqd/008:30:150
373399244,4cyclictest12-21ksoftirqd/008:30:140
373399242,15cyclictest7929-21ssh12:20:170
373399241,17cyclictest741-21ssh10:10:120
3733992411,10cyclictest2636-21ssh11:05:120
3733992410,8cyclictest0-21swapper/009:52:190
376599234,12cyclictest28366irq/58-eth011:35:133
376599232,11cyclictest28366irq/58-eth011:25:153
375399235,15cyclictest5667-21kthreadcore11:05:212
375399233,16cyclictest22346irq/40-dwc2_hso09:55:142
375399233,12cyclictest22346irq/40-dwc2_hso09:20:112
375399232,16cyclictest0-21swapper/211:14:562
375399231,14cyclictest0-21swapper/208:50:122
375399231,10cyclictest452666ssh10:30:162
375399231,10cyclictest452666ssh10:30:152
374299239,4cyclictest131rcu_preempt11:35:151
374299239,4cyclictest131rcu_preempt08:35:221
374299231,18cyclictest9747-21ssh11:45:121
374299231,18cyclictest9747-21ssh11:45:111
374299231,17cyclictest21672-21diskmemload10:36:071
3742992311,3cyclictest0-21swapper/108:00:311
374299230,11cyclictest25885-21latency10:40:211
373399235,8cyclictest12-21ksoftirqd/008:05:180
373399234,14cyclictest2321-21ssh09:35:130
373399232,3cyclictest131rcu_preempt09:30:050
373399232,15cyclictest22397-21ssh10:56:100
3733992312,6cyclictest0-21swapper/008:50:140
3733992311,8cyclictest15013-21cron11:30:020
373399231,17cyclictest1-21systemd07:45:280
3733992310,5cyclictest0-21swapper/009:48:230
376599223,13cyclictest0-21swapper/310:56:113
376599223,11cyclictest0-21swapper/311:45:123
376599223,11cyclictest0-21swapper/311:45:113
376599222,14cyclictest28366irq/58-eth009:45:133
3765992214,5cyclictest0-21swapper/310:20:223
375399224,14cyclictest27-21ksoftirqd/212:15:112
375399224,14cyclictest27-21ksoftirqd/212:15:102
375399224,10cyclictest22346irq/40-dwc2_hso09:50:122
375399222,15cyclictest0-21swapper/212:02:542
375399222,14cyclictest22346irq/40-dwc2_hso08:00:132
375399222,14cyclictest22346irq/40-dwc2_hso07:20:132
3753992214,5cyclictest924-21snmpd12:10:212
374299227,4cyclictest131rcu_preempt10:50:171
374299226,3cyclictest131rcu_preempt09:30:151
374299222,16cyclictest18325-21apt-get07:25:131
3742992213,6cyclictest3715-21sh12:15:311
3742992213,6cyclictest3715-21sh12:15:301
374299221,14cyclictest201rcuc/110:10:111
374299221,10cyclictest5658-21tr07:10:201
373399228,6cyclictest131rcu_preempt11:52:530
373399228,4cyclictest12-21ksoftirqd/011:38:230
373399225,13cyclictest0-21swapper/011:10:130
373399225,10cyclictest12-21ksoftirqd/009:44:070
373399224,9cyclictest12-21ksoftirqd/011:42:380
373399222,4cyclictest12-21ksoftirqd/007:50:120
373399221,7cyclictest2279-21ssh09:15:240
3733992212,8cyclictest11538-21sed10:50:240
373399221,17cyclictest12549-21ls09:00:160
3765992110,6cyclictest28366irq/58-eth009:20:133
375399218,11cyclictest9371-21kthreadcore08:55:212
375399214,13cyclictest22346irq/40-dwc2_hso09:25:142
375399213,9cyclictest6170-21latency_hist08:20:012
375399212,15cyclictest0-21swapper/210:56:552
375399212,12cyclictest0-21swapper/208:05:132
375399211,2cyclictest0-21swapper/208:35:142
375399211,16cyclictest0-21swapper/211:20:172
375399211,12cyclictest14111-21kthreadcore09:00:202
375399211,10cyclictest28427-21tr12:30:222
375399211,10cyclictest10086-21grep07:50:182
374299217,4cyclictest131rcu_preempt08:00:011
374299214,14cyclictest31366-21ssh10:45:121
374299213,4cyclictest131rcu_preempt10:00:131
374299213,11cyclictest0-21swapper/109:40:111
374299212,5cyclictest131rcu_preempt12:05:121
374299211,9cyclictest18757-21cut09:25:231
374299211,9cyclictest131rcu_preempt09:35:291
374299211,18cyclictest3894-21switchtime07:40:271
374299211,17cyclictest27896-21apt-get07:35:111
373399217,7cyclictest12-21ksoftirqd/012:25:040
373399217,5cyclictest12-21ksoftirqd/011:02:560
3733992114,4cyclictest0-21swapper/012:11:180
376599209,9cyclictest0-21swapper/309:10:303
3765992013,3cyclictest33-21ksoftirqd/310:45:203
3765992011,6cyclictest28366irq/58-eth012:35:123
375399204,13cyclictest22346irq/40-dwc2_hso07:30:152
375399203,12cyclictest0-21swapper/208:10:132
375399202,9cyclictest28045-21fschecks_count10:25:172
375399201,9cyclictest17126-21cpu09:05:132
375399201,8cyclictest14545-21wc08:25:232
375399201,16cyclictest2408-21kthreadcore10:10:202
375399201,15cyclictest20620-21grep09:45:162
374299206,3cyclictest131rcu_preempt08:10:161
374299201,12cyclictest1225-21missed_timers08:45:241
374299201,10cyclictest3741-21tr08:15:191
373399207,5cyclictest12-21ksoftirqd/010:22:330
373399206,8cyclictest12-21ksoftirqd/007:15:280
373399205,4cyclictest131rcu_preempt07:40:110
3733992014,4cyclictest0-21swapper/008:15:140
3733992014,4cyclictest0-21swapper/007:35:090
373399201,12cyclictest25565-21find08:40:130
3733992010,8cyclictest0-21swapper/011:47:470
3733992010,8cyclictest0-21swapper/011:47:460
3733992010,4cyclictest12-21ksoftirqd/010:08:170
373399200,14cyclictest22026-21ls07:25:280
376599199,4cyclictest0-21swapper/309:55:283
376599193,11cyclictest0-21swapper/312:34:563
376599193,11cyclictest0-21swapper/312:03:553
376599193,11cyclictest0-21swapper/311:01:113
3765991913,4cyclictest33-21ksoftirqd/307:25:163
3765991911,4cyclictest0-21swapper/310:40:303
376599191,13cyclictest0-21swapper/309:05:013
376599191,13cyclictest0-21swapper/308:20:223
376599191,12cyclictest0-21swapper/308:50:173
376599191,10cyclictest0-21swapper/309:10:013
3765991910,6cyclictest0-21swapper/311:59:043
3765991910,6cyclictest0-21swapper/311:30:283
375399197,5cyclictest19175-21cut11:50:172
375399192,10cyclictest874-21irqrtprio07:40:192
374299194,4cyclictest131rcu_preempt10:15:131
374299192,13cyclictest32442-21users11:55:331
374299191,9cyclictest27844-21grep07:35:011
374299191,8cyclictest23202-21kthreadcore09:10:191
374299191,3cyclictest0-21swapper/108:50:281
3742991911,6cyclictest21648-21sshd09:05:481
374299191,13cyclictest10143-21timerwakeupswit08:20:281
3742991910,6cyclictest967-21cut11:40:151
373399198,5cyclictest12-21ksoftirqd/008:35:270
373399195,3cyclictest131rcu_preempt08:10:110
373399193,12cyclictest0-21swapper/008:00:040
3733991913,4cyclictest0-21swapper/007:30:150
373399191,14cyclictest0-21swapper/007:55:150
376599189,6cyclictest0-21swapper/312:28:413
376599181,13cyclictest0-21swapper/310:25:183
376599181,13cyclictest0-21swapper/308:35:023
376599181,13cyclictest0-21swapper/308:35:013
376599181,13cyclictest0-21swapper/308:00:153
3765991810,6cyclictest0-21swapper/312:20:203
374299183,10cyclictest0-21swapper/108:30:131
374299183,10cyclictest0-21swapper/108:30:121
374299181,8cyclictest15848-21kthreadcore07:20:191
374299181,7cyclictest8633-21irqstats12:20:211
3742991811,5cyclictest24150-21/usr/sbin/munin10:05:241
374299181,13cyclictest12442-21cut09:00:141
3733991812,4cyclictest0-21swapper/009:05:140
3733991812,4cyclictest0-21swapper/008:55:230
373399180,11cyclictest3987-21cut07:10:160
376599179,6cyclictest0-21swapper/310:17:333
376599179,6cyclictest0-21swapper/310:10:083
376599179,6cyclictest0-21swapper/309:35:093
376599179,5cyclictest28366irq/58-eth012:05:123
376599179,5cyclictest28366irq/58-eth010:00:133
376599179,5cyclictest28366irq/58-eth009:25:143
376599178,6cyclictest0-21swapper/309:30:263
3765991710,4cyclictest0-21swapper/312:15:273
3765991710,4cyclictest0-21swapper/312:15:263
374299171,5cyclictest25778-21/usr/sbin/munin08:40:161
374299171,3cyclictest131rcu_preempt07:45:161
3733991712,3cyclictest0-21swapper/008:25:230
376599161,9cyclictest0-21swapper/308:45:233
3765991611,3cyclictest33-21ksoftirqd/307:10:283
376599161,12cyclictest0-21swapper/307:40:273
376599161,10cyclictest0-21swapper/307:50:133
3765991610,4cyclictest33-21ksoftirqd/308:40:303
374299161,12cyclictest3820-21latency_hist07:10:021
376599153,9cyclictest0-21swapper/307:15:153
376599151,11cyclictest0-21swapper/308:10:273
376599142,9cyclictest0-21swapper/308:35:143
376599141,8cyclictest0-21swapper/307:55:283
376599141,8cyclictest0-21swapper/307:35:113
376599141,8cyclictest0-21swapper/307:10:023
376599141,7cyclictest0-21swapper/308:25:183
373399141,10cyclictest3736-21latency_hist07:10:000
376599138,3cyclictest0-21swapper/307:22:533
376599133,6cyclictest28366irq/58-eth008:05:243
376599131,7cyclictest0-21swapper/308:20:033
376599131,7cyclictest0-21swapper/307:30:143
375399131,9cyclictest3848-21latency_hist07:10:012
376599121,9cyclictest0-21swapper/308:55:203
376599121,7cyclictest0-21swapper/307:45:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional