You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-27 - 10:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Apr 27, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17949998534,30cyclictest131rcu_preempt00:15:150
17953997410,57cyclictest26815-21ssh00:05:111
1795399674,52cyclictest325-21ssh22:20:141
17963995918,31cyclictest27-21ksoftirqd/222:40:142
1795399576,35cyclictest23804-21ssh22:15:121
1794999544,11cyclictest131rcu_preempt20:35:110
1794999543,45cyclictest15624-21ssh00:35:120
1797499525,31cyclictest35966irq/58-eth000:05:133
17963994942,4cyclictest29089-21ssh23:30:122
1795399492,39cyclictest16425-21ssh23:05:121
1794999483,9cyclictest131rcu_preempt20:00:130
1794999481,10cyclictest131rcu_preempt19:55:130
1794999474,39cyclictest3357-21ssh00:10:110
1794999472,35cyclictest26824-21ssh00:05:120
1795399461,3cyclictest0-21swapper/120:55:151
1795399461,2cyclictest17083-21apt-get20:50:131
1795399451,4cyclictest0-21swapper/121:35:111
1795399451,4cyclictest0-21swapper/121:05:141
1794999456,4cyclictest131rcu_preempt22:20:140
1794999452,35cyclictest22055-21ssh21:20:140
1797499443,25cyclictest35966irq/58-eth022:25:133
17949994410,26cyclictest131rcu_preempt20:50:140
1797499432,29cyclictest35966irq/58-eth000:00:143
1794999432,34cyclictest16577-21ssh21:35:140
1796399425,29cyclictest27-21ksoftirqd/222:25:132
1795399426,32cyclictest5875-21ssh22:05:121
1795399422,27cyclictest11232-21ssh23:20:171
1795399421,2cyclictest7575-21ssh21:30:141
1794999427,23cyclictest3619-21diskmemload22:45:140
1795399413,2cyclictest9295-21apt-get22:25:131
1796399408,13cyclictest261rcuc/221:20:152
1795399402,2cyclictest28961-21apt-get19:55:141
1795399402,2cyclictest12318-21apt-get00:15:141
1795399401,8cyclictest0-21swapper/119:50:151
1795399401,8cyclictest0-21swapper/119:50:141
1795399401,3cyclictest20216-21apt-get21:55:131
1795399401,2cyclictest12970-21apt-get21:15:141
1795399401,2cyclictest0-21swapper/123:15:131
1795399401,2cyclictest0-21swapper/122:30:111
17963993925,6cyclictest27-21ksoftirqd/222:15:132
1796399392,34cyclictest7409-21ssh23:00:142
1795399394,3cyclictest4003-21apt-get21:10:111
1795399393,2cyclictest0-21swapper/123:45:121
1795399392,3cyclictest30752-21apt-get22:55:141
1795399391,3cyclictest3787-21apt-get22:40:131
1796399383,19cyclictest29239-21ssh22:00:122
1795399383,2cyclictest0-21swapper/121:40:121
1795399381,4cyclictest29240-21apt-get22:00:141
1795399381,4cyclictest21242-21apt-get00:20:131
1795399381,3cyclictest0-21swapper/120:40:121
1795399381,3cyclictest0-21swapper/120:05:141
1794999383,25cyclictest12-21ksoftirqd/019:35:120
1795399370,14cyclictest14874-21ssh22:10:141
1795399362,3cyclictest0-21swapper/121:00:131
1795399361,3cyclictest0-21swapper/121:45:111
1795399361,3cyclictest0-21swapper/120:20:131
1795399361,32cyclictest14482-21sh23:40:141
1795399361,32cyclictest14482-21sh23:40:141
1795399361,31cyclictest5584-21ssh23:35:111
17949993613,19cyclictest12-21ksoftirqd/021:00:140
1797499354,19cyclictest35966irq/58-eth023:30:123
17974993512,13cyclictest35966irq/58-eth021:30:143
1795399353,2cyclictest0-21swapper/123:30:131
1795399353,24cyclictest11178-21sh21:50:151
1795399351,3cyclictest0-21swapper/119:20:131
1795399351,3cyclictest0-21swapper/100:35:131
1795399351,3cyclictest0-21swapper/100:30:151
17953993511,16cyclictest32387-21ssh23:50:131
1795399342,6cyclictest0-21swapper/123:00:131
1795399341,4cyclictest0-21swapper/100:25:141
1795399341,2cyclictest2582-21apt-get20:35:121
1795399341,2cyclictest0-21swapper/123:10:151
1797499333,18cyclictest35966irq/58-eth021:10:113
1795399332,3cyclictest1339-21apt-get20:00:131
1795399331,7cyclictest11597-21kthreadcore20:10:201
1795399331,4cyclictest22981-21apt-get19:15:141
1795399331,10cyclictest28020-21tr20:25:231
1795399331,10cyclictest21548-21awk00:00:271
1795399330,2cyclictest4947-21apt-get19:30:141
1794999334,17cyclictest12-21ksoftirqd/022:55:140
1796399327,16cyclictest11806-21kworker/2:121:15:132
1796399323,18cyclictest27346irq/40-dwc2_hso20:35:122
1796399322,19cyclictest25503-21ssh23:10:142
17963993217,12cyclictest27-21ksoftirqd/220:40:132
1795399325,24cyclictest22121-21cut19:45:211
1795399321,9cyclictest32089-21users22:35:331
1795399321,9cyclictest13225-21ntp_states19:35:251
1795399321,8cyclictest3700-21if_eth100:10:171
1795399321,7cyclictest12711-21sed20:45:181
1795399321,6cyclictest10573-21tr23:55:201
1795399320,5cyclictest32583-21apt-get19:25:151
1795399320,5cyclictest14677-21diskstats19:40:161
1797499313,21cyclictest35966irq/58-eth023:05:143
1797499312,22cyclictest35966irq/58-eth022:55:143
1796399313,22cyclictest27-21ksoftirqd/222:50:142
1795399311,8cyclictest21960-21/usr/sbin/munin22:50:161
1795399311,7cyclictest18749-21cut19:10:171
1795399311,7cyclictest15863-21fschecks_time20:15:171
1795399311,7cyclictest15863-21fschecks_time20:15:161
1795399311,7cyclictest13206-21sed22:45:191
1795399311,3cyclictest0-21swapper/120:30:131
1795399311,3cyclictest0-21swapper/120:30:131
1795399311,11cyclictest557-21tr21:25:211
1795399311,10cyclictest21814-21dpkg21:20:031
17963993014,13cyclictest17941-21cyclictest20:20:142
1795399301,7cyclictest24024-21sendmail_mailtr23:25:291
1794999303,21cyclictest20243-21ssh21:55:140
1796399292,25cyclictest14258-21sh22:09:462
1796399292,16cyclictest27346irq/40-dwc2_hso19:55:132
1794999292,21cyclictest26884-21sh22:50:460
1794999291,22cyclictest30189-21sh00:25:160
1797499281,14cyclictest35966irq/58-eth020:00:133
1796399282,18cyclictest27346irq/40-dwc2_hso20:05:132
1796399281,12cyclictest27-21ksoftirqd/221:42:462
1797499276,12cyclictest0-21swapper/321:20:133
1797499274,16cyclictest35966irq/58-eth022:40:133
1796399276,11cyclictest0-21swapper/220:45:172
1796399275,12cyclictest2869-21tail23:15:202
1796399273,19cyclictest0-21swapper/200:39:462
1796399272,19cyclictest18312-21ssh22:30:142
1796399271,12cyclictest21253-21ssh00:20:122
17974992619,4cyclictest35966irq/58-eth022:05:133
1796399269,8cyclictest27-21ksoftirqd/200:30:182
1796399264,14cyclictest27346irq/40-dwc2_hso20:30:132
1796399264,14cyclictest27346irq/40-dwc2_hso20:30:122
1796399264,14cyclictest27346irq/40-dwc2_hso00:00:142
1796399263,21cyclictest0-21swapper/221:37:462
1796399263,16cyclictest27346irq/40-dwc2_hso20:10:132
1796399262,9cyclictest5277-21ssh00:28:472
1796399262,15cyclictest23431-21ssh23:45:142
1796399261,9cyclictest8925-21ssh23:55:122
1796399261,6cyclictest32408-21ssh23:50:142
1796399261,17cyclictest7572-21apt-get21:30:132
1795399261,6cyclictest18096-21latency_hist19:10:011
1794999262,19cyclictest31337-21ssh22:18:470
17949992613,8cyclictest5872-21apt-get22:05:130
1797499253,16cyclictest0-21swapper/322:18:463
1796399253,11cyclictest14476-21ssh23:40:132
1796399253,11cyclictest14476-21ssh23:40:122
1796399252,20cyclictest0-21swapper/200:15:162
1796399252,16cyclictest27346irq/40-dwc2_hso21:00:152
1796399252,15cyclictest27346irq/40-dwc2_hso21:10:122
1794999251,19cyclictest3619-21diskmemload21:30:140
1794999251,18cyclictest9322-21ssh22:25:130
17974992416,5cyclictest35966irq/58-eth023:55:143
17974992411,9cyclictest0-21swapper/322:10:223
1796399243,14cyclictest27346irq/40-dwc2_hso19:10:112
1796399243,11cyclictest24749-21kthreadcore20:55:222
1796399242,17cyclictest27346irq/40-dwc2_hso23:35:132
1796399242,14cyclictest11905-21ssh00:15:002
1796399241,8cyclictest3619-21diskmemload22:59:452
1796399241,18cyclictest22372-21ssh22:13:452
17963992410,11cyclictest16402-21/usr/sbin/munin23:05:242
17949992411,10cyclictest12-21ksoftirqd/000:20:120
17974992312,8cyclictest0-21swapper/321:55:123
1796399236,9cyclictest27-21ksoftirqd/221:55:132
1796399234,12cyclictest27346irq/40-dwc2_hso19:30:142
1796399232,15cyclictest27346irq/40-dwc2_hso19:15:132
17963992312,9cyclictest1385-21ssh22:37:362
1796399231,13cyclictest31028-21ssh21:25:152
1794999239,3cyclictest131rcu_preempt22:00:170
17949992315,6cyclictest0-21swapper/021:45:140
1794999231,19cyclictest7392-21find20:40:120
1797499222,13cyclictest35966irq/58-eth021:50:153
1796399222,9cyclictest13217-21cut22:45:182
1796399221,9cyclictest3803-21wc22:20:232
17963992214,5cyclictest20196-21ssh23:25:162
1796399221,18cyclictest30962-21grep00:05:302
1796399221,17cyclictest10762-21kthreadcore19:35:192
1794999224,12cyclictest31012-21ssh21:25:130
1794999222,6cyclictest0-21swapper/023:03:450
1794999221,18cyclictest23042-21ls19:45:260
1794999221,16cyclictest12988-21ssh21:15:130
1794999221,14cyclictest12178-21kworker/u8:022:10:120
1797499214,13cyclictest0-21swapper/323:00:133
1797499213,12cyclictest0-21swapper/323:10:473
17974992113,5cyclictest35966irq/58-eth021:35:133
1797499211,13cyclictest35966irq/58-eth022:00:133
1797499211,13cyclictest0-21swapper/322:45:143
1796399212,16cyclictest19597-21kthreadcore20:50:232
1796399211,10cyclictest2885-21tail21:05:282
1796399211,10cyclictest15090-21irqrtprio19:40:192
1796399211,10cyclictest13831-21tr21:50:232
1794999218,4cyclictest131rcu_preempt23:05:250
1794999213,10cyclictest0-21swapper/019:25:120
1794999212,3cyclictest131rcu_preempt23:35:130
1794999212,2cyclictest131rcu_preempt23:20:170
1794999212,15cyclictest25556-21apt-get21:40:110
1794999212,14cyclictest17880-21ssh00:00:140
17949992114,4cyclictest592-21timerwakeupswit23:30:290
17949992113,5cyclictest26469-21cut23:45:220
1794999211,17cyclictest31692-21ls21:05:160
1794999211,17cyclictest1493-21ls20:30:280
1794999211,17cyclictest1493-21ls20:30:270
1794999211,17cyclictest11154-21find21:50:140
1797499202,10cyclictest0-21swapper/320:35:113
17974992012,5cyclictest0-21swapper/322:30:233
17974992011,4cyclictest0-21swapper/323:45:123
1797499201,14cyclictest0-21swapper/322:50:453
1797499201,12cyclictest35966irq/58-eth023:20:153
1796399206,5cyclictest5905-21cron20:05:012
1796399204,10cyclictest438-21grep19:25:192
1796399203,9cyclictest17570-21kthreadcore20:15:212
1796399203,9cyclictest17570-21kthreadcore20:15:202
1796399201,9cyclictest31323-21wc19:20:272
1796399201,9cyclictest27870-21cat19:50:292
1796399201,9cyclictest27870-21cat19:50:282
1796399201,9cyclictest11371-21fschecks_count23:20:182
17963992014,4cyclictest0-21swapper/221:45:152
1796399201,10cyclictest22516-21kthreadcore19:45:202
1794999203,13cyclictest19702-21apt-get23:25:030
1794999201,16cyclictest27576-21ls22:35:160
1797499199,7cyclictest35966irq/58-eth023:15:133
1797499192,9cyclictest35966irq/58-eth000:35:143
17974991913,3cyclictest33-21ksoftirqd/323:50:133
1797499191,13cyclictest0-21swapper/300:20:133
1797499191,12cyclictest0-21swapper/320:25:293
1796399191,15cyclictest0-21swapper/220:25:132
1794999197,6cyclictest12576-21cat20:45:160
1794999192,12cyclictest17941-21cyclictest19:40:140
1794999191,6cyclictest25096-21apt-get20:25:010
1794999191,3cyclictest0-21swapper/023:50:080
1794999191,2cyclictest131rcu_preempt20:10:010
17949991911,6cyclictest1-21systemd23:17:170
1794999191,15cyclictest15675-21latency_hist20:15:020
1797499189,6cyclictest0-21swapper/321:15:573
1797499189,5cyclictest35966irq/58-eth023:35:123
1797499188,6cyclictest35966irq/58-eth023:25:143
1797499186,5cyclictest0-21swapper/322:20:153
1797499183,10cyclictest0-21swapper/300:33:473
1797499182,10cyclictest0-21swapper/300:25:153
17974991812,4cyclictest33-21ksoftirqd/300:15:323
17974991811,4cyclictest35966irq/58-eth021:44:383
17974991811,4cyclictest33-21ksoftirqd/319:10:263
1797499181,13cyclictest0-21swapper/320:30:293
1797499181,13cyclictest0-21swapper/320:30:283
1797499181,12cyclictest0-21swapper/319:40:183
1797499181,11cyclictest0-21swapper/320:40:283
1794999186,3cyclictest131rcu_preempt23:40:230
1794999186,3cyclictest131rcu_preempt23:40:220
1794999184,4cyclictest131rcu_preempt22:30:230
1794999183,10cyclictest0-21swapper/023:13:460
1794999182,3cyclictest91-21pr/ttyS019:55:010
1794999182,3cyclictest91-21pr/ttyS019:55:010
1794999182,11cyclictest4014-21ssh21:10:120
17949991811,5cyclictest0-21swapper/019:20:280
1794999181,14cyclictest3797-21ssh22:40:140
1794999181,11cyclictest12634-21ntp_states23:55:270
17949991810,5cyclictest27534-21sh19:20:000
1797499179,6cyclictest35966irq/58-eth000:10:133
1797499179,6cyclictest0-21swapper/323:41:503
1797499179,6cyclictest0-21swapper/323:41:493
1797499179,6cyclictest0-21swapper/321:48:203
1797499178,6cyclictest0-21swapper/321:25:093
17974991711,4cyclictest0-21swapper/322:35:313
1797499171,13cyclictest0-21swapper/319:45:133
17974991710,2cyclictest0-21swapper/319:20:173
1794999179,5cyclictest25683-21grep20:55:290
1794999175,9cyclictest30182-21latency_hist20:30:020
1794999172,4cyclictest0-21swapper/000:35:020
1794999171,8cyclictest5595-21uname19:30:200
1797499162,9cyclictest0-21swapper/319:25:143
1797499161,9cyclictest35966irq/58-eth020:15:153
1797499161,9cyclictest35966irq/58-eth020:15:143
1796399162,11cyclictest18070-21latency_hist19:10:022
1794999163,10cyclictest19274-21awk20:15:280
1794999163,10cyclictest19274-21awk20:15:270
1794999161,11cyclictest20517-21kthreadcore19:10:200
1797499152,9cyclictest0-21swapper/319:30:153
1797499151,9cyclictest0-21swapper/320:50:163
1797499151,8cyclictest0-21swapper/319:40:013
1794999153,10cyclictest17990-21grep19:10:010
1797499142,8cyclictest0-21swapper/320:10:153
1797499142,7cyclictest0-21swapper/319:55:123
1797499141,7cyclictest0-21swapper/320:20:163
1797499139,2cyclictest33-21ksoftirqd/321:00:123
1797499138,3cyclictest33-21ksoftirqd/319:50:293
1797499138,3cyclictest33-21ksoftirqd/319:50:283
1797499138,3cyclictest0-21swapper/320:05:283
1797499133,8cyclictest0-21swapper/320:45:163
1797499131,9cyclictest0-21swapper/319:15:203
1797499123,7cyclictest35966irq/58-eth020:55:143
1797499121,4cyclictest0-21swapper/321:10:023
1797499101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional