You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-21 - 08:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Apr 21, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
325399682,48cyclictest9263-21ssh23:45:122
325399593,49cyclictest25060-21apt-get20:40:132
325399571,49cyclictest0-21swapper/223:05:122
326199568,29cyclictest35966irq/58-eth022:40:153
325399552,48cyclictest0-21swapper/219:50:132
325399511,45cyclictest0-21swapper/220:55:142
325399501,2cyclictest0-21swapper/221:15:132
325399481,3cyclictest0-21swapper/219:30:142
325399472,24cyclictest18470-21ssh22:20:162
325399471,3cyclictest0-21swapper/220:35:132
325399471,36cyclictest9277-21ssh22:15:132
325399462,40cyclictest0-21swapper/220:30:152
325399461,36cyclictest0-21swapper/222:05:142
325399451,5cyclictest0-21swapper/220:45:132
325399451,39cyclictest0-21swapper/220:50:102
3253994423,14cyclictest0-21swapper/223:40:132
324499443,28cyclictest16549-21ssh22:55:131
325399432,37cyclictest0-21swapper/223:30:152
325399431,38cyclictest0-21swapper/222:50:122
324499435,6cyclictest131rcu_preempt23:10:121
324499431,30cyclictest30666-21ssh00:15:111
325399421,3cyclictest0-21swapper/220:20:132
325399421,3cyclictest0-21swapper/220:10:122
325399421,2cyclictest20001-21apt-get21:45:132
325399420,4cyclictest17823-21apt-get19:25:142
325399420,2cyclictest25296-21apt-get00:30:132
324499422,33cyclictest21972-21ssh22:40:141
3261994121,10cyclictest35966irq/58-eth023:55:133
325399411,34cyclictest0-21swapper/223:55:112
325399411,2cyclictest3965-21apt-get22:30:132
326199404,20cyclictest35966irq/58-eth021:55:143
325399401,35cyclictest0-21swapper/219:45:112
325399401,31cyclictest27346irq/40-dwc2_hso22:00:132
325399401,31cyclictest27346irq/40-dwc2_hso22:00:122
3261993915,16cyclictest35966irq/58-eth019:55:123
325399391,34cyclictest0-21swapper/222:45:122
326199381,25cyclictest0-21swapper/323:05:123
325399382,31cyclictest0-21swapper/223:12:552
325399382,31cyclictest0-21swapper/219:55:122
325399382,29cyclictest11051-21ssh21:40:142
325399381,4cyclictest0-21swapper/219:40:142
325399381,3cyclictest0-21swapper/221:00:122
325399381,35cyclictest0-21swapper/223:00:142
325399381,2cyclictest2030-21apt-get21:35:122
325399381,2cyclictest0-21swapper/221:20:142
324499382,27cyclictest16359-21ssh21:25:141
326199371,22cyclictest35966irq/58-eth021:50:123
325399372,2cyclictest5699-21apt-get23:25:152
325399372,2cyclictest5699-21apt-get23:25:142
325399371,7cyclictest23654-21munin-run20:05:002
325399371,6cyclictest1415-21usage_EthcTask23:20:322
323299375,23cyclictest7298-21ssh00:20:130
326199363,22cyclictest0-21swapper/321:40:133
3261993620,13cyclictest0-21swapper/321:25:133
325399361,32cyclictest0-21swapper/222:25:142
325399361,32cyclictest0-21swapper/222:25:142
325399361,31cyclictest20349-21fschecks_time23:15:152
323299361,25cyclictest23734-21rm23:35:110
325399351,3cyclictest0-21swapper/219:15:132
325399351,3cyclictest0-21swapper/219:15:122
325399351,30cyclictest24428-21sed23:35:172
325399351,2cyclictest3806-21apt-get00:00:132
325399351,25cyclictest18238-21/usr/sbin/munin23:50:262
3244993521,8cyclictest7357-21ssh21:20:121
323299351,30cyclictest14648-21sh23:30:140
3261993418,12cyclictest0-21swapper/323:30:153
325399341,4cyclictest16257-21ls00:05:222
325399341,28cyclictest0-21swapper/219:35:142
325399341,25cyclictest13233-21fschecks_time22:35:192
325399340,2cyclictest27346irq/40-dwc2_hso20:25:122
325399340,2cyclictest23878-21apt-get20:05:142
325399340,24cyclictest25437-21/usr/sbin/munin21:30:152
325399332,9cyclictest0-21swapper/222:10:152
325399332,27cyclictest30653-21ssh00:15:112
325399331,28cyclictest8191-21kthreadcore00:20:202
325399331,28cyclictest26702-21kworker/u8:121:57:092
325399331,28cyclictest21708-21ssh21:10:142
325399330,30cyclictest21877-21rm21:26:192
325399321,2cyclictest0-21swapper/220:15:142
325399320,2cyclictest0-21swapper/221:50:132
325399320,29cyclictest6921-21ssh00:35:382
324499322,22cyclictest5909-21fw_packets21:55:161
3232993224,2cyclictest12-21ksoftirqd/022:10:160
326199313,14cyclictest35966irq/58-eth000:25:143
326199312,16cyclictest35966irq/58-eth019:15:123
326199312,16cyclictest35966irq/58-eth019:15:113
325399315,23cyclictest20355-21switchtime21:05:282
325399314,24cyclictest24835-21cut00:10:232
325399310,4cyclictest21660-21ssh00:26:092
325399310,2cyclictest30449-21ssh22:44:542
3232993123,5cyclictest12-21ksoftirqd/021:55:130
323299312,26cyclictest30910-21ssh22:45:100
323299311,26cyclictest12806-21ssh00:05:130
326199302,16cyclictest35966irq/58-eth000:10:133
325399301,5cyclictest3258-21latency_hist19:10:022
325399301,26cyclictest6433-21kthreadcore19:10:202
3244993022,5cyclictest0-21swapper/123:25:141
3244993022,5cyclictest0-21swapper/123:25:141
323299302,21cyclictest73-21kswapd023:55:100
3232993019,8cyclictest12-21ksoftirqd/022:20:140
326199296,16cyclictest0-21swapper/322:20:173
3261992922,4cyclictest0-21swapper/323:45:123
326199292,20cyclictest35966irq/58-eth021:00:143
3261992917,7cyclictest0-21swapper/322:15:123
325399291,25cyclictest15480-21kthreadcore19:20:212
325399291,24cyclictest21068-21usage_EthcTask22:55:322
324499292,21cyclictest21349-21ssh22:02:421
324499292,21cyclictest21349-21ssh22:02:411
323299293,21cyclictest12-21ksoftirqd/019:55:130
323299292,22cyclictest30757-21ssh21:15:150
3232992913,12cyclictest12-21ksoftirqd/019:35:140
326199283,15cyclictest35966irq/58-eth000:15:133
3261992812,10cyclictest35966irq/58-eth022:30:143
324499287,14cyclictest0-21swapper/123:40:121
323299283,20cyclictest27382-21ssh22:25:150
323299283,20cyclictest27382-21ssh22:25:140
3232992817,8cyclictest0-21swapper/023:00:130
326199273,16cyclictest35966irq/58-eth021:20:143
326199273,16cyclictest0-21swapper/322:10:163
3261992722,3cyclictest0-21swapper/319:45:113
3261992719,6cyclictest0-21swapper/320:50:113
3261992719,5cyclictest0-21swapper/323:40:143
3261992718,4cyclictest0-21swapper/320:35:133
3261992718,3cyclictest0-21swapper/322:00:143
3261992718,3cyclictest0-21swapper/322:00:133
3261992715,9cyclictest0-21swapper/320:30:153
324499278,12cyclictest23721-21apt23:35:121
324499272,20cyclictest11056-21ssh21:40:151
324499272,19cyclictest8972-21sh23:07:551
324499272,19cyclictest18480-21fschecks_count22:20:171
3244992721,3cyclictest21-21ksoftirqd/120:40:131
3244992714,4cyclictest131rcu_preempt21:45:301
323299274,20cyclictest1966-21ssh00:35:150
323299272,21cyclictest3819-21ssh00:00:140
323299272,21cyclictest12-21ksoftirqd/019:15:120
323299272,21cyclictest12-21ksoftirqd/019:15:110
323299270,24cyclictest18272-21ssh23:50:140
3261992620,3cyclictest0-21swapper/323:10:123
3261992619,4cyclictest0-21swapper/321:15:133
3261992618,4cyclictest0-21swapper/320:55:133
324499262,18cyclictest29141-21ssh23:20:131
324499262,18cyclictest23706-21ssh22:05:151
324499262,18cyclictest13077-21ssh22:51:161
323299264,12cyclictest12-21ksoftirqd/022:00:130
323299264,12cyclictest12-21ksoftirqd/022:00:120
323299261,17cyclictest20031-21ssh21:45:140
3261992519,4cyclictest35966irq/58-eth019:25:133
3261992519,3cyclictest0-21swapper/322:50:133
3261992513,8cyclictest0-21swapper/321:30:023
3261992513,8cyclictest0-21swapper/319:35:143
324499259,12cyclictest0-21swapper/121:30:131
324499257,12cyclictest2036-21ssh21:35:121
324499251,20cyclictest7317-21ssh00:20:151
323299252,19cyclictest7573-21ssh22:50:140
3232992519,3cyclictest12-21ksoftirqd/022:30:120
3261992420,2cyclictest0-21swapper/320:20:143
3261992419,2cyclictest0-21swapper/322:55:123
3261992418,3cyclictest0-21swapper/323:00:143
3261992418,3cyclictest0-21swapper/322:05:123
3261992418,3cyclictest0-21swapper/319:50:143
3261992416,5cyclictest0-21swapper/323:25:143
3261992416,5cyclictest0-21swapper/323:25:133
3261992415,4cyclictest0-21swapper/322:45:113
3261992415,4cyclictest0-21swapper/319:10:143
3261992414,7cyclictest0-21swapper/320:15:133
3261992413,7cyclictest0-21swapper/323:50:313
326199241,15cyclictest0-21swapper/322:25:153
326199241,15cyclictest0-21swapper/322:25:143
324499241,17cyclictest1980-21memory22:45:201
324499241,16cyclictest25392-21ssh21:10:261
323299248,11cyclictest12-21ksoftirqd/000:10:140
323299242,19cyclictest151rcuc/023:45:110
3261992317,3cyclictest0-21swapper/323:20:123
3261992317,2cyclictest0-21swapper/300:35:133
3261992316,3cyclictest0-21swapper/322:35:153
3261992316,3cyclictest0-21swapper/320:25:123
3261992315,4cyclictest0-21swapper/300:30:133
3261992314,6cyclictest0-21swapper/300:05:123
3261992314,5cyclictest0-21swapper/321:45:143
3261992311,7cyclictest0-21swapper/320:10:133
326199231,13cyclictest0-21swapper/321:10:143
324499232,13cyclictest0-21swapper/120:20:141
3244992310,9cyclictest21-21ksoftirqd/100:25:121
324499230,21cyclictest0-21swapper/123:50:131
323299232,19cyclictest9542-21idleruntime22:15:160
3261992218,2cyclictest0-21swapper/319:30:113
3261992217,3cyclictest0-21swapper/300:00:133
3261992217,2cyclictest0-21swapper/320:05:123
3261992212,7cyclictest0-21swapper/300:20:123
324499225,11cyclictest0-21swapper/122:30:141
324499220,17cyclictest30791-21ssh00:31:151
323299227,12cyclictest12-21ksoftirqd/000:33:140
323299222,16cyclictest23681-21apt-get22:05:120
323299222,15cyclictest12-21ksoftirqd/020:55:140
323299221,4cyclictest131rcu_preempt00:25:240
326199212,11cyclictest35966irq/58-eth023:15:133
3261992116,3cyclictest0-21swapper/320:45:123
3261992116,2cyclictest0-21swapper/320:40:113
3261992112,7cyclictest0-21swapper/319:40:133
3261992112,6cyclictest0-21swapper/323:35:293
3261992111,7cyclictest0-21swapper/321:35:193
324499219,9cyclictest10323-21cut23:45:191
324499214,11cyclictest0-21swapper/121:05:141
324499212,13cyclictest29053-21ssh21:50:121
3232992114,5cyclictest0-21swapper/023:10:320
3232992113,4cyclictest0-21swapper/020:30:150
323299211,15cyclictest2135-21ssh23:05:130
323299210,18cyclictest29998-21cut20:45:160
326199207,10cyclictest0-21swapper/321:05:123
3261992015,2cyclictest0-21swapper/319:20:113
324499202,10cyclictest21-21ksoftirqd/119:40:151
324499201,3cyclictest7131-21apt-get20:55:131
3244992013,3cyclictest9267-21ssh22:15:131
324499201,16cyclictest20073-21ls20:35:021
324499200,16cyclictest30835-21ls21:15:161
323299205,9cyclictest0-21swapper/019:25:150
323299203,12cyclictest628-21tcf-agent22:35:160
323299202,15cyclictest20239-21apt-get20:35:130
3232992013,5cyclictest26134-21usage_Anybus22:40:310
3232992013,3cyclictest12-21ksoftirqd/019:45:110
3232992012,5cyclictest16324-21/usr/sbin/munin21:25:290
3232992012,5cyclictest0-21swapper/020:55:020
323299201,17cyclictest8868-21cron23:45:020
323299201,16cyclictest21309-21diskmemload21:35:170
323299201,15cyclictest30963-21ssh00:15:160
3232992011,4cyclictest0-21swapper/023:25:140
3232992011,4cyclictest0-21swapper/023:25:130
323299201,13cyclictest32263-21ls20:10:280
323299201,13cyclictest25492-21ssh21:30:150
323299201,13cyclictest20051-21missed_timers21:05:220
3261991914,2cyclictest0-21swapper/320:00:133
324499197,6cyclictest21-21ksoftirqd/122:35:141
324499196,4cyclictest131rcu_preempt00:10:231
324499194,6cyclictest21-21ksoftirqd/100:00:231
324499192,12cyclictest7962-21munin-run19:15:011
324499191,4cyclictest8516-21irqcore19:15:171
324499191,4cyclictest8516-21irqcore19:15:161
324499191,4cyclictest0-21swapper/119:25:191
324499191,10cyclictest2728-21irqrtprio20:50:161
3244991910,6cyclictest27647-21hwlatdetect22:25:181
3244991910,6cyclictest27647-21hwlatdetect22:25:171
323299196,3cyclictest131rcu_preempt21:40:210
323299193,5cyclictest14244-21grep20:25:270
323299192,3cyclictest131rcu_preempt23:15:130
323299191,9cyclictest3440-21wc19:40:270
323299191,9cyclictest25635-21ssh21:10:290
323299191,10cyclictest3389-21/usr/sbin/munin19:10:150
3232991910,6cyclictest32587-21latency23:20:220
324499189,7cyclictest2050-21df00:35:141
324499188,7cyclictest16331-21memory19:20:211
324499188,7cyclictest1-21systemd20:37:241
324499187,8cyclictest18156-21memory23:30:231
3244991811,5cyclictest28382-21sshd20:06:161
324499181,13cyclictest13345-21head19:50:311
324499181,10cyclictest20284-21fschecks_count23:15:161
3244991810,5cyclictest27563-21kthreadcore23:00:211
323299187,8cyclictest12-21ksoftirqd/020:00:280
323299185,10cyclictest1859-21kthreadcore20:15:190
323299183,12cyclictest16511-21/usr/sbin/munin22:55:180
323299181,13cyclictest0-21swapper/021:00:140
323299181,12cyclictest7425-21ssh21:20:150
3232991810,5cyclictest0-21swapper/020:10:010
323299180,13cyclictest29219-21ssh21:50:150
326199177,6cyclictest0-21swapper/319:10:023
324499174,3cyclictest131rcu_preempt19:35:031
324499172,10cyclictest0-21swapper/120:10:141
324499171,14cyclictest48-21kcompactd000:09:001
324499171,13cyclictest317-21diskstats22:10:151
3244991710,5cyclictest27178-21cron19:35:021
3244991710,4cyclictest4724-21cat20:15:271
324499170,11cyclictest27191-21ssh23:55:111
3232991711,4cyclictest0-21swapper/019:50:190
324499163,10cyclictest5838-21tr19:45:181
324499163,10cyclictest20970-21kthreadcore20:00:211
324499161,11cyclictest15597-21grep21:00:271
323299164,9cyclictest28140-21kthreadcore20:40:210
323299162,11cyclictest13019-21apt-get19:20:120
323299161,4cyclictest0-21swapper/020:22:410
323299161,12cyclictest26433-21grep19:30:270
324499157,6cyclictest12639-21tr20:25:191
324499152,9cyclictest0-21swapper/120:45:131
324499151,11cyclictest15289-21kthreadcore19:55:181
324499151,10cyclictest3278-21latency_hist19:10:011
323299151,4cyclictest131rcu_preempt19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional