You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 00:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Thu Aug 06, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1912996638,16cyclictest35966irq/58-eth009:20:143
1892996453,8cyclictest10254-21ssh12:05:141
188499625,46cyclictest1841-21ssh10:30:120
189299616,8cyclictest131rcu_preempt09:25:121
189299604,46cyclictest6789-21ssh11:45:111
1900995718,30cyclictest27-21ksoftirqd/212:25:142
1900995718,30cyclictest27-21ksoftirqd/212:25:132
1900995539,10cyclictest3820-21ssh09:55:142
190099533,37cyclictest27267-21ssh09:50:112
189299525,8cyclictest131rcu_preempt12:25:131
189299525,8cyclictest131rcu_preempt12:25:131
189299521,38cyclictest9375-21ssh09:40:111
190099506,39cyclictest21231-21ssh11:35:122
190099502,32cyclictest367-21ssh09:35:122
1912994634,7cyclictest35966irq/58-eth012:00:113
1900994513,25cyclictest0-21swapper/210:50:132
1892994517,18cyclictest21-21ksoftirqd/112:15:121
191299444,31cyclictest35966irq/58-eth009:50:113
189299448,9cyclictest21-21ksoftirqd/109:50:121
189299444,33cyclictest28814-21ssh10:45:121
189299441,36cyclictest5381-21ssh10:50:121
188499441,30cyclictest15792-21ssh11:50:130
188499441,30cyclictest15792-21ssh11:50:120
188499430,35cyclictest6796-21sh11:45:130
191299412,20cyclictest0-21swapper/311:35:123
189299411,27cyclictest12860-21ssh10:00:131
191299394,22cyclictest35966irq/58-eth012:05:133
190099372,18cyclictest4706-21sh11:42:592
190099372,18cyclictest26747-21ssh09:49:502
1912993617,12cyclictest35966irq/58-eth012:10:133
188499362,24cyclictest30780-21ssh10:10:120
190099352,17cyclictest23226-21sh09:29:502
1900993518,9cyclictest0-21swapper/209:20:132
188499353,21cyclictest753-21sendmail-mta10:20:120
188499351,28cyclictest22659-21ssh12:30:120
191299344,19cyclictest0-21swapper/311:15:133
190099333,24cyclictest0-21swapper/209:30:502
1900993315,11cyclictest27-21ksoftirqd/208:40:132
189299331,22cyclictest3818-21sh09:55:141
190099322,18cyclictest9384-21ssh09:40:132
1900993215,11cyclictest0-21swapper/211:10:142
189299322,23cyclictest19210-21ssh12:10:141
188499323,20cyclictest25285-21ssh10:25:140
188499323,20cyclictest1266-21ssh12:00:120
190099312,22cyclictest17878-21ssh11:15:152
190099312,21cyclictest3321-21ssh11:25:112
189299312,21cyclictest28978-21rm09:30:501
189299311,23cyclictest20378-21ssh09:10:141
189299311,17cyclictest26818-21ssh11:20:131
188499314,16cyclictest19915-21sh10:40:110
188499312,23cyclictest32289-21ssh09:34:500
1884993118,6cyclictest0-21swapper/007:10:110
1912993011,12cyclictest35966irq/58-eth012:15:123
1900993013,12cyclictest10253-21sh12:05:132
190099301,25cyclictest24765-21ssh11:55:132
189299306,13cyclictest21-21ksoftirqd/107:40:101
188499302,22cyclictest12632-21ssh09:22:510
188499302,21cyclictest12858-21sh10:00:140
1884993018,6cyclictest3812-21apt-get09:55:130
191299296,14cyclictest35966irq/58-eth010:50:133
191299293,19cyclictest0-21swapper/312:25:213
191299293,19cyclictest0-21swapper/312:25:213
1912992913,7cyclictest0-21swapper/309:05:133
1900992919,8cyclictest0-21swapper/211:20:142
189299292,22cyclictest4707-21ssh11:43:001
189299292,21cyclictest1085566ssh10:35:131
1892992915,10cyclictest5861-21ssh09:20:131
1912992814,11cyclictest35966irq/58-eth010:25:143
190099283,17cyclictest0-21swapper/207:15:122
190099283,17cyclictest0-21swapper/207:15:122
190099282,21cyclictest22769-21ssh10:59:512
189299284,18cyclictest1829-21apt-get10:30:121
1884992811,12cyclictest12-21ksoftirqd/011:42:000
1912992712,10cyclictest35966irq/58-eth011:25:103
190099271,16cyclictest32344-21ls08:15:172
189299278,11cyclictest0-21swapper/109:05:121
189299275,9cyclictest131rcu_preempt12:35:121
188499279,6cyclictest12-21ksoftirqd/009:46:270
188499272,18cyclictest31750-21ssh12:15:210
191299263,11cyclictest0-21swapper/309:55:133
191299262,14cyclictest35966irq/58-eth011:50:133
191299262,14cyclictest35966irq/58-eth011:50:123
190099263,13cyclictest27346irq/40-dwc2_hso07:55:142
190099262,20cyclictest5513-21ssh12:37:202
190099261,3cyclictest0-21swapper/210:10:132
190099261,21cyclictest19922-21ssh10:40:132
190099261,14cyclictest22670-21sh12:30:132
189299269,12cyclictest201rcuc/110:05:141
1892992618,6cyclictest31865-21cron08:15:011
188499263,16cyclictest844-21munin-node09:00:130
191299255,12cyclictest0-21swapper/312:20:123
191299253,16cyclictest0-21swapper/309:30:503
191299253,15cyclictest0-21swapper/311:43:003
191299252,13cyclictest35966irq/58-eth010:35:123
190099258,14cyclictest23788-21cat12:10:312
190099253,15cyclictest27346irq/40-dwc2_hso11:30:112
190099251,16cyclictest29336-21ls10:25:302
190099251,14cyclictest20455-21cpuspeed09:10:142
189299255,14cyclictest14361-21ssh10:55:121
1892992515,6cyclictest0-21swapper/109:35:141
1892992511,6cyclictest131rcu_preempt08:15:121
188499257,4cyclictest131rcu_preempt11:35:270
188499255,3cyclictest131rcu_preempt08:55:170
188499254,16cyclictest19195-21ssh12:10:130
188499253,16cyclictest32102-21apt-get08:15:120
188499252,18cyclictest23344-21ssh11:00:140
188499252,17cyclictest0-21swapper/010:55:130
1884992510,4cyclictest12-21ksoftirqd/011:11:280
191299243,13cyclictest35966irq/58-eth011:00:123
191299243,13cyclictest0-21swapper/309:35:123
191299242,10cyclictest0-21swapper/309:29:493
190099246,13cyclictest27-21ksoftirqd/208:30:132
190099243,14cyclictest27346irq/40-dwc2_hso10:05:152
190099243,13cyclictest27346irq/40-dwc2_hso07:40:122
190099242,17cyclictest27346irq/40-dwc2_hso09:15:102
190099242,13cyclictest28986-21fschecks_count10:45:162
190099241,15cyclictest27346irq/40-dwc2_hso07:30:112
190099241,11cyclictest6755-21rm11:45:112
189299242,18cyclictest3310-21apt-get11:25:111
188499246,7cyclictest12-21ksoftirqd/010:50:530
188499246,3cyclictest131rcu_preempt11:56:310
1884992413,8cyclictest30586-21cron07:40:010
188499241,18cyclictest26823-21ssh11:20:140
191299233,14cyclictest35966irq/58-eth009:45:123
191299232,15cyclictest35966irq/58-eth009:40:143
191299232,15cyclictest35966irq/58-eth008:30:103
191299232,13cyclictest35966irq/58-eth010:40:123
190099237,10cyclictest0-21swapper/208:35:152
190099236,14cyclictest15497-21grep09:05:162
190099235,14cyclictest27346irq/40-dwc2_hso07:20:122
190099232,6cyclictest0-21swapper/210:33:502
190099232,14cyclictest0-21swapper/208:45:132
190099231,8cyclictest2978-21ssh12:18:212
1900992317,4cyclictest16485-21ssh10:20:152
190099231,18cyclictest5139-21irqcore12:20:172
190099231,12cyclictest9227-21cut10:15:202
189299231,18cyclictest4758-21ssh12:20:131
189299231,17cyclictest28706-21cut08:45:161
189299231,16cyclictest24784-21ssh11:55:151
188499238,7cyclictest12-21ksoftirqd/011:33:550
188499238,5cyclictest12-21ksoftirqd/010:46:180
188499237,6cyclictest131rcu_preempt09:37:350
188499236,7cyclictest12-21ksoftirqd/011:05:560
188499236,7cyclictest12-21ksoftirqd/010:15:580
188499236,7cyclictest12-21ksoftirqd/007:40:260
188499236,4cyclictest12-21ksoftirqd/012:27:520
188499236,4cyclictest12-21ksoftirqd/012:27:510
1884992312,3cyclictest0-21swapper/012:08:510
188499231,17cyclictest29499-21cut09:15:140
1884992310,5cyclictest12-21ksoftirqd/010:07:420
1884992310,4cyclictest12-21ksoftirqd/009:27:580
190099229,10cyclictest1-21systemd08:05:142
190099227,6cyclictest2960-21kthreadcore08:50:202
190099227,12cyclictest27-21ksoftirqd/210:00:212
190099224,11cyclictest27346irq/40-dwc2_hso12:00:132
190099223,11cyclictest4626-21fschecks_count08:20:162
1900992215,5cyclictest27-21ksoftirqd/211:00:122
1900992212,8cyclictest14565-21sh10:35:262
1900992211,9cyclictest6538-21sshd07:14:402
190099221,16cyclictest15796-21ssh11:50:132
190099221,16cyclictest15796-21ssh11:50:122
190099221,11cyclictest28476-21kthreadcore07:35:172
190099221,11cyclictest2308-21kthreadcore11:05:212
189299221,8cyclictest1530-21perl12:00:161
189299221,5cyclictest21-21ksoftirqd/111:30:201
189299221,16cyclictest23733-21apt-get08:40:131
1892992210,3cyclictest131rcu_preempt11:50:281
1892992210,3cyclictest131rcu_preempt11:50:271
188499228,4cyclictest12-21ksoftirqd/011:28:300
188499228,4cyclictest12-21ksoftirqd/009:52:510
188499227,6cyclictest12-21ksoftirqd/012:37:160
188499225,6cyclictest12-21ksoftirqd/007:30:010
188499224,4cyclictest131rcu_preempt11:20:000
188499224,3cyclictest131rcu_preempt10:38:210
188499223,8cyclictest12-21ksoftirqd/009:41:240
188499223,4cyclictest12-21ksoftirqd/007:30:150
188499221,17cyclictest4731-21apt-get12:20:100
1884992211,5cyclictest12-21ksoftirqd/008:10:140
191299217,11cyclictest0-21swapper/310:45:233
1912992112,6cyclictest0-21swapper/312:35:203
190099211,8cyclictest11894-21grep07:50:292
190099211,5cyclictest0-21swapper/208:00:052
190099211,16cyclictest12230-21kthreadcore08:25:202
190099211,10cyclictest4122-21cut07:45:192
190099211,10cyclictest29072-21kthreadcore08:10:202
190099211,10cyclictest19058-21kthreadcore07:25:212
189299212,14cyclictest14353-21kworker/u8:011:03:531
1892992113,6cyclictest29796-21sshd12:33:201
189299211,15cyclictest19968-21diskmemload10:20:131
189299211,14cyclictest16471-21ls07:25:151
1892992110,9cyclictest23485-21cron08:40:011
188499211,16cyclictest18918-21apt-get08:35:130
191299208,6cyclictest35966irq/58-eth010:10:133
1912992011,6cyclictest35966irq/58-eth010:20:123
190099202,9cyclictest6057-21irqcore08:55:172
190099202,14cyclictest10682-21grep09:00:162
189299201,7cyclictest6049-21irqcore08:55:171
1892992012,6cyclictest424-21ssh09:15:221
189299201,16cyclictest3482-21ls07:45:171
1892992011,6cyclictest26475-21ssh11:20:011
189299201,13cyclictest21-21ksoftirqd/107:50:121
188499209,8cyclictest12-21ksoftirqd/008:45:130
188499206,8cyclictest12-21ksoftirqd/008:00:220
188499204,3cyclictest131rcu_preempt07:10:020
188499203,8cyclictest12-21ksoftirqd/009:12:160
191299199,7cyclictest0-21swapper/310:05:203
191299198,7cyclictest35966irq/58-eth012:30:133
191299198,7cyclictest0-21swapper/307:10:273
191299198,6cyclictest0-21swapper/311:05:073
191299197,9cyclictest0-21swapper/309:19:263
191299193,11cyclictest0-21swapper/311:13:553
191299191,13cyclictest0-21swapper/307:40:263
191299191,13cyclictest0-21swapper/307:30:023
1912991910,7cyclictest0-21swapper/311:23:173
189299199,7cyclictest13620-21ssh11:10:321
189299193,3cyclictest131rcu_preempt09:45:231
1892991911,6cyclictest32326-21ssh11:05:121
188499199,7cyclictest11841-21hwlatdetect07:20:160
188499198,3cyclictest131rcu_preempt08:30:200
188499196,3cyclictest131rcu_preempt08:40:250
188499193,5cyclictest12-21ksoftirqd/009:05:160
191299189,6cyclictest35966irq/58-eth011:30:133
191299183,13cyclictest0-21swapper/310:30:193
191299182,10cyclictest0-21swapper/308:35:153
191299182,10cyclictest0-21swapper/308:15:143
191299182,10cyclictest0-21swapper/308:00:143
191299182,10cyclictest0-21swapper/307:30:133
191299181,13cyclictest0-21swapper/308:45:163
191299181,12cyclictest0-21swapper/307:20:163
191299181,12cyclictest0-21swapper/307:15:123
191299181,12cyclictest0-21swapper/307:15:123
191299181,11cyclictest0-21swapper/308:25:123
1912991810,6cyclictest0-21swapper/311:56:483
1912991810,6cyclictest0-21swapper/310:57:553
1912991810,6cyclictest0-21swapper/309:14:393
189299189,7cyclictest7554-21fw_packets10:15:161
189299181,8cyclictest4130-21tr08:50:221
189299181,8cyclictest26144-21grep07:35:131
189299181,14cyclictest0-21swapper/107:24:391
189299180,15cyclictest1869-21ssh10:10:221
188499189,3cyclictest131rcu_preempt08:25:270
188499187,9cyclictest151rcuc/008:10:010
188499182,5cyclictest12-21ksoftirqd/007:45:280
188499181,14cyclictest15800-21kthreadcore07:55:220
188499181,11cyclictest4566-21switchtime08:50:270
191299179,6cyclictest0-21swapper/311:45:093
191299179,6cyclictest0-21swapper/310:15:133
191299178,6cyclictest0-21swapper/307:45:273
191299172,10cyclictest0-21swapper/308:40:133
191299172,10cyclictest0-21swapper/308:05:133
191299171,13cyclictest0-21swapper/308:10:283
190099175,5cyclictest1901-21latency_hist07:10:022
189299179,5cyclictest21798-21irqrtprio11:35:171
189299179,5cyclictest0-21swapper/107:15:131
189299179,5cyclictest0-21swapper/107:15:121
189299178,6cyclictest25797-21cut10:25:191
189299173,11cyclictest15275-21grep09:05:021
189299173,11cyclictest0-21swapper/107:10:131
189299171,9cyclictest9111-21latency_hist08:25:011
1892991711,4cyclictest0-21swapper/108:00:251
189299171,13cyclictest23469-21kthreadcore07:30:181
189299171,12cyclictest19459-21apt-get10:40:041
189299171,12cyclictest14250-21uniq08:30:151
189299170,15cyclictest26186-21ls08:05:271
188499174,8cyclictest12-21ksoftirqd/008:20:160
188499173,5cyclictest12-21ksoftirqd/007:15:220
188499173,5cyclictest12-21ksoftirqd/007:15:220
191299168,5cyclictest35966irq/58-eth007:50:143
191299167,7cyclictest0-21swapper/310:03:113
191299162,10cyclictest0-21swapper/309:00:143
189299168,6cyclictest13903-21cron08:30:011
189299162,9cyclictest0-21swapper/107:55:151
188499167,3cyclictest12-21ksoftirqd/007:50:160
191299152,9cyclictest0-21swapper/308:50:133
191299152,10cyclictest0-21swapper/307:35:113
191299151,8cyclictest0-21swapper/307:55:223
189299131,9cyclictest1931-21latency_hist07:10:011
191299122,7cyclictest0-21swapper/308:20:163
191299121,9cyclictest0-21swapper/308:55:203
191299121,8cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional