You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-28 - 05:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Jul 28, 2026 00:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2239499594,43cyclictest30210-21ssh23:45:130
2239499562,43cyclictest8121-21ssh23:15:140
2240899558,6cyclictest131rcu_preempt00:25:131
22394995132,9cyclictest12-21ksoftirqd/021:10:130
22394995132,9cyclictest12-21ksoftirqd/021:10:130
2240899502,38cyclictest13351-21ssh23:00:141
2240899501,37cyclictest7072-21ssh21:45:121
2242899464,27cyclictest35966irq/58-eth000:20:143
2242899451,27cyclictest35966irq/58-eth023:15:133
2239499441,32cyclictest23946-21rm22:30:140
2239499435,30cyclictest27628-21ssh22:50:120
22428994131,7cyclictest0-21swapper/321:45:123
2242899401,22cyclictest35966irq/58-eth021:10:133
2242899401,22cyclictest35966irq/58-eth021:10:133
2239499407,18cyclictest5498-21ssh00:25:130
2239499402,23cyclictest26596-21ssh21:20:140
2241799391,28cyclictest930-21snmpd22:40:132
2241799391,28cyclictest930-21snmpd22:40:132
2239499393,31cyclictest1905-21ssh22:00:130
2242899384,24cyclictest35966irq/58-eth022:35:133
2242899381,21cyclictest35966irq/58-eth021:40:123
2242899381,21cyclictest35966irq/58-eth021:40:123
2241799383,21cyclictest5492-21ssh00:25:132
22408993823,7cyclictest7940-21diskmemload00:20:131
22408993812,19cyclictest21-21ksoftirqd/120:50:131
2240899367,19cyclictest21-21ksoftirqd/120:30:131
2240899362,27cyclictest569-21ssh22:35:141
2242899357,16cyclictest35966irq/58-eth019:40:133
2241799353,25cyclictest0-21swapper/200:22:462
2241799350,31cyclictest0-21swapper/223:05:142
2239499352,27cyclictest17206-21ssh23:20:120
22394993414,12cyclictest0-21swapper/022:20:120
2239499331,20cyclictest19729-21ssh00:15:130
2239499331,20cyclictest19729-21ssh00:15:130
2239499322,21cyclictest31485-21ssh23:10:150
22394993219,6cyclictest0-21swapper/021:40:120
22394993219,6cyclictest0-21swapper/021:40:120
2242899312,19cyclictest35966irq/58-eth022:10:123
2239499312,21cyclictest14946-21ssh22:25:150
22394993112,12cyclictest12-21ksoftirqd/019:30:140
2242899301,21cyclictest35966irq/58-eth021:00:143
2242899301,16cyclictest35966irq/58-eth021:05:143
2241799304,15cyclictest27346irq/40-dwc2_hso21:35:132
22417993018,6cyclictest2496146ssh00:00:132
2240899302,23cyclictest16124-21ssh21:50:141
22408993021,6cyclictest26430-21ssh23:25:151
22408993021,6cyclictest26430-21ssh23:25:151
2239499302,22cyclictest4886-21ssh00:24:460
2241799295,14cyclictest27346irq/40-dwc2_hso20:40:122
2241799293,16cyclictest27346irq/40-dwc2_hso19:40:152
2239499292,20cyclictest30397-21ssh00:37:450
2239499292,20cyclictest13176-21ssh22:40:250
2239499292,20cyclictest13176-21ssh22:40:250
2242899286,14cyclictest35966irq/58-eth021:50:133
2241799284,17cyclictest3038-21ssh23:30:152
2241799283,15cyclictest11003-21ssh22:05:132
2241799282,21cyclictest20961-21ssh23:55:462
2241799282,21cyclictest15260-21ssh23:54:462
2241799282,20cyclictest0-21swapper/223:40:142
22417992812,7cyclictest27-21ksoftirqd/220:30:122
2240899286,4cyclictest131rcu_preempt23:20:131
2239499284,17cyclictest6813-21ssh23:50:140
2239499283,18cyclictest22418-21ssh23:05:140
2239499282,20cyclictest25227-21ssh21:55:160
22394992814,8cyclictest22384-21cyclictest19:25:130
2241799275,15cyclictest27346irq/40-dwc2_hso23:25:152
2241799275,15cyclictest27346irq/40-dwc2_hso23:25:152
2241799274,16cyclictest27346irq/40-dwc2_hso23:35:152
2241799273,15cyclictest27346irq/40-dwc2_hso00:15:112
2241799273,15cyclictest27346irq/40-dwc2_hso00:15:112
2241799273,10cyclictest27-21ksoftirqd/222:10:132
2241799272,17cyclictest930-21snmpd23:45:152
2241799271,11cyclictest17435-21ssh00:12:462
22408992720,4cyclictest21-21ksoftirqd/121:20:131
22428992615,7cyclictest35966irq/58-eth022:45:143
2239499263,17cyclictest3197-21ssh21:25:140
22428992516,6cyclictest0-21swapper/322:55:143
2241799253,17cyclictest0-21swapper/200:06:462
22417992516,4cyclictest27-21ksoftirqd/222:49:072
22417992513,9cyclictest0-21swapper/220:45:132
22417992512,5cyclictest0-21swapper/219:45:282
2241799251,15cyclictest8363-21ssh21:10:142
2241799251,15cyclictest8363-21ssh21:10:142
2240899254,9cyclictest21-21ksoftirqd/122:20:121
22408992510,6cyclictest131rcu_preempt21:17:551
2239499255,13cyclictest0-21swapper/020:20:140
2239499251,18cyclictest20096-21ssh22:10:130
2241799245,12cyclictest27346irq/40-dwc2_hso23:10:142
2241799242,6cyclictest0-21swapper/200:38:462
22417992414,5cyclictest27-21ksoftirqd/222:34:062
2241799241,13cyclictest29178-21ssh22:15:132
22417992411,10cyclictest0-21swapper/219:30:282
2240899249,5cyclictest21-21ksoftirqd/121:25:201
22408992413,8cyclictest21-21ksoftirqd/120:35:131
2240899241,18cyclictest21368-21ssh21:35:141
22408992411,7cyclictest0-21swapper/123:36:441
22408992411,7cyclictest0-21swapper/122:09:071
22408992411,3cyclictest0-21swapper/122:49:071
2239499243,12cyclictest12-21ksoftirqd/019:45:130
2239499242,17cyclictest17459-21rm21:15:150
2242899235,10cyclictest35966irq/58-eth019:50:123
2241799237,4cyclictest27-21ksoftirqd/221:18:342
2241799234,13cyclictest27346irq/40-dwc2_hso19:50:132
2241799234,11cyclictest27346irq/40-dwc2_hso22:35:142
2241799233,16cyclictest22569-21apt-get19:10:132
2241799232,11cyclictest7260-21grep21:25:312
2241799231,19cyclictest23678-21ls22:30:012
2241799231,11cyclictest0-21swapper/223:20:132
2240899232,17cyclictest29817-21apt-get20:25:131
2240899231,17cyclictest11975-21ssh21:10:261
2240899231,17cyclictest11975-21ssh21:10:261
2240899231,14cyclictest0-21swapper/123:30:151
22408992310,3cyclictest21-21ksoftirqd/123:18:171
2239499232,16cyclictest7940-21diskmemload00:09:460
22394992314,5cyclictest0-21swapper/020:25:140
2239499231,18cyclictest1297-21irqstats22:35:190
2239499231,16cyclictest5840-21apt-get20:00:130
22394992310,4cyclictest131rcu_preempt23:25:200
22394992310,4cyclictest131rcu_preempt23:25:200
2241799227,10cyclictest0-21swapper/221:00:152
2241799224,12cyclictest27-21ksoftirqd/221:20:182
2241799223,12cyclictest27346irq/40-dwc2_hso20:50:142
2241799222,13cyclictest27346irq/40-dwc2_hso21:45:122
22417992216,4cyclictest28728-21ssh22:50:202
2241799221,5cyclictest32492-21ls19:20:152
22417992211,9cyclictest20495-21sshd00:31:452
2241799221,16cyclictest7936-21cron21:10:012
2241799221,11cyclictest25797-21cut21:55:212
2241799221,11cyclictest14003-21irqrtprio23:00:192
2240899229,3cyclictest21-21ksoftirqd/123:50:041
2240899229,3cyclictest21-21ksoftirqd/123:45:461
2240899229,3cyclictest131rcu_preempt19:55:181
2240899229,11cyclictest0-21swapper/100:30:321
2240899228,3cyclictest0-21swapper/122:34:061
2240899227,6cyclictest131rcu_preempt22:53:471
2240899227,11cyclictest21-21ksoftirqd/122:10:121
22408992212,7cyclictest21-21ksoftirqd/100:10:141
22408992210,8cyclictest22727-21if_err_eth123:05:171
2239499222,16cyclictest3029-21apt-get23:30:140
2239499221,13cyclictest10709-21ssh00:10:150
2242899219,9cyclictest33-21ksoftirqd/323:05:133
2242899215,9cyclictest35966irq/58-eth022:05:143
2242899211,15cyclictest35966irq/58-eth023:00:143
2241799218,11cyclictest21440-21ssh21:51:052
2241799217,5cyclictest8262-21kthreadcore20:35:212
2241799213,8cyclictest13781-21rm21:30:202
2241799212,17cyclictest6386-21ps22:20:182
2241799211,9cyclictest2025-21latency_hist20:30:022
22417992112,6cyclictest0-21swapper/221:40:112
22417992112,6cyclictest0-21swapper/221:40:112
2241799211,17cyclictest16785-21cron23:20:012
2241799211,16cyclictest6179-21kthreadcore19:25:202
2241799211,16cyclictest5954-21fschecks_count20:00:172
2241799211,16cyclictest4452-21kthreadcore22:00:212
2241799211,10cyclictest18665-21latency20:10:232
22417992110,9cyclictest0-21swapper/219:15:492
22417992110,8cyclictest0-21swapper/220:55:202
2240899219,3cyclictest21-21ksoftirqd/100:15:121
2240899219,3cyclictest21-21ksoftirqd/100:15:121
2240899214,13cyclictest21-21ksoftirqd/123:10:151
2240899212,15cyclictest12263-21ssh21:30:131
22408992112,5cyclictest24030-21irqcore00:35:181
22408992111,7cyclictest2462-21ps22:00:181
2240899211,16cyclictest14943-21sh22:25:151
2239499217,5cyclictest0-21swapper/021:05:140
2239499213,11cyclictest7940-21diskmemload23:55:140
2239499212,14cyclictest32552-21ssh00:03:460
2239499212,14cyclictest21178-21ssh23:40:130
2239499211,16cyclictest8336-21ls22:55:310
2239499211,13cyclictest16402-21ls21:50:170
2242899206,10cyclictest0-21swapper/300:15:163
2242899206,10cyclictest0-21swapper/300:15:163
22428992010,8cyclictest0-21swapper/300:31:313
22428992010,8cyclictest0-21swapper/300:13:473
2241799201,9cyclictest4463-21ssh22:55:182
2241799201,9cyclictest1733-21kthreadcore19:55:212
22417992012,5cyclictest0-21swapper/219:35:182
2241799201,15cyclictest10730-21cut20:05:162
2241799201,10cyclictest25422-21irqcore20:20:182
2241799201,10cyclictest20378-21fschecks_time20:15:162
2241799201,10cyclictest20378-21fschecks_time20:15:162
2240899209,8cyclictest9384-21apt-get19:30:131
2240899208,2cyclictest21-21ksoftirqd/120:10:251
2240899206,5cyclictest21-21ksoftirqd/100:05:151
2240899204,8cyclictest0-21swapper/121:59:011
22408992014,3cyclictest21-21ksoftirqd/100:00:111
22408992013,4cyclictest21-21ksoftirqd/122:43:521
22408992013,4cyclictest21-21ksoftirqd/122:43:521
22408992012,4cyclictest25690-21usage_EthcTask23:40:311
2239499202,11cyclictest16032-21kworker/u8:222:15:150
2242899199,6cyclictest35966irq/58-eth022:30:153
2242899199,6cyclictest35966irq/58-eth022:15:153
2242899197,9cyclictest0-21swapper/323:46:063
2242899193,11cyclictest0-21swapper/323:59:463
2242899193,11cyclictest0-21swapper/300:37:463
2242899193,11cyclictest0-21swapper/300:06:463
22428991911,6cyclictest0-21swapper/321:30:213
2242899191,12cyclictest0-21swapper/323:20:123
2242899191,11cyclictest0-21swapper/319:20:273
22428991910,6cyclictest0-21swapper/321:17:093
22428991910,4cyclictest0-21swapper/319:35:273
2240899199,2cyclictest131rcu_preempt23:59:391
2240899194,2cyclictest21-21ksoftirqd/121:41:101
2240899194,2cyclictest21-21ksoftirqd/121:41:101
22408991913,4cyclictest201rcuc/122:55:141
2240899191,13cyclictest32348-21diskstats19:20:141
2239499191,9cyclictest13623-21cut23:00:170
22394991912,5cyclictest0-21swapper/019:10:290
22394991911,5cyclictest17685-21pluginstate19:35:270
22394991911,5cyclictest13790-21kthreadcore21:30:200
2239499191,13cyclictest14711-21ssh22:05:270
2239499191,12cyclictest24891-21missed_timers21:35:240
2242899189,7cyclictest0-21swapper/322:40:533
2242899189,7cyclictest0-21swapper/322:40:533
2242899189,6cyclictest0-21swapper/323:37:443
2242899188,6cyclictest0-21swapper/319:25:193
2242899187,8cyclictest35966irq/58-eth022:50:133
2242899183,10cyclictest0-21swapper/300:27:463
2242899183,10cyclictest0-21swapper/300:00:463
2242899182,11cyclictest0-21swapper/323:54:463
2242899182,10cyclictest0-21swapper/319:15:043
2242899181,13cyclictest0-21swapper/321:25:243
22428991810,6cyclictest35966irq/58-eth023:30:133
22428991810,6cyclictest0-21swapper/322:26:023
22428991810,6cyclictest0-21swapper/321:39:163
2240899189,7cyclictest31710-21cut22:15:231
2240899185,10cyclictest18046-21grep19:35:291
2240899183,3cyclictest131rcu_preempt20:00:181
2240899181,3cyclictest131rcu_preempt19:45:131
22408991813,3cyclictest0-21swapper/120:45:161
2239499189,7cyclictest27436-21apt-get19:15:350
2239499182,12cyclictest0-21swapper/000:30:150
2239499181,9cyclictest22689-21awk19:40:290
22394991812,4cyclictest0-21swapper/019:50:160
2239499181,14cyclictest31195-21cut21:00:170
22394991810,5cyclictest19149-21irqcore22:45:180
2242899179,6cyclictest0-21swapper/323:42:223
2242899179,6cyclictest0-21swapper/323:29:203
2242899179,6cyclictest0-21swapper/323:29:203
2242899179,6cyclictest0-21swapper/322:22:393
2242899179,6cyclictest0-21swapper/322:04:573
2242899172,10cyclictest0-21swapper/320:50:143
22428991711,4cyclictest0-21swapper/323:12:423
2242899171,10cyclictest0-21swapper/320:10:153
2240899178,6cyclictest28479-21ntp_states20:20:261
2240899174,8cyclictest0-21swapper/119:15:361
2240899172,12cyclictest6216-21kthreadcore21:05:231
2240899172,11cyclictest32433-21kthreadcore21:00:211
2240899172,10cyclictest0-21swapper/119:50:141
2240899171,6cyclictest22331-21latency_hist19:10:011
2240899171,6cyclictest22331-21latency_hist19:10:011
2240899171,6cyclictest16452-21latency_hist20:45:011
22408991712,3cyclictest0-21swapper/120:55:241
2240899171,10cyclictest21-21ksoftirqd/119:40:151
22394991712,3cyclictest12377-21ls23:35:180
2239499171,14cyclictest490-21kthreadcore19:20:180
22394991711,4cyclictest0-21swapper/020:40:160
2239499171,12cyclictest7778-21irqstats21:45:170
22394991710,3cyclictest0-21swapper/020:50:250
2242899168,6cyclictest0-21swapper/321:55:363
2242899161,10cyclictest0-21swapper/321:20:043
2241799161,12cyclictest22504-21latency_hist19:10:012
2241799161,12cyclictest22504-21latency_hist19:10:012
2240899167,6cyclictest10729-21fschecks_count20:05:161
2240899161,11cyclictest22695-21fschecks_count19:10:171
22408991610,4cyclictest0-21swapper/120:15:181
22408991610,4cyclictest0-21swapper/120:15:181
22408991610,3cyclictest21-21ksoftirqd/119:25:131
2239499168,6cyclictest2717-21sed20:30:180
2239499162,10cyclictest0-21swapper/020:10:150
2239499161,11cyclictest20417-21fschecks_time20:15:160
2239499161,11cyclictest20417-21fschecks_time20:15:160
2239499161,11cyclictest12-21ksoftirqd/020:05:120
22394991610,4cyclictest0-21swapper/019:55:170
2239499160,13cyclictest8472-21kthreadcore20:35:210
2242899152,10cyclictest0-21swapper/320:40:133
2242899151,9cyclictest0-21swapper/319:45:283
2242899151,9cyclictest0-21swapper/319:30:283
2242899151,8cyclictest0-21swapper/320:05:233
2242899151,11cyclictest0-21swapper/319:10:163
2239499156,6cyclictest26814-21irqrtprio20:55:200
2239499151,11cyclictest18603-21tr20:45:220
2242899149,3cyclictest0-21swapper/320:25:153
2242899142,8cyclictest0-21swapper/320:00:133
2242899141,8cyclictest0-21swapper/320:50:013
2242899141,10cyclictest0-21swapper/320:55:213
2239499142,9cyclictest22400-21sort19:10:010
2239499142,9cyclictest22400-21sort19:10:010
2242899138,3cyclictest33-21ksoftirqd/319:57:513
2242899131,9cyclictest0-21swapper/320:35:203
2242899127,3cyclictest0-21swapper/320:20:283
2242899121,4cyclictest0-21swapper/320:16:463
2242899121,4cyclictest0-21swapper/320:16:463
2242899112,5cyclictest0-21swapper/320:30:143
224289991,5cyclictest0-21swapper/319:10:013
224289991,5cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional