You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-04 - 06:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat Apr 04, 2026 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16002997023,10cyclictest131rcu_preempt21:05:130
16034996332,22cyclictest28366irq/58-eth000:15:133
16034996145,11cyclictest28366irq/58-eth023:00:123
1602299582,39cyclictest15282-21ssh22:30:122
1600299561,41cyclictest25249-21ssh00:25:130
1600299542,12cyclictest131rcu_preempt00:10:110
16002995219,24cyclictest12-21ksoftirqd/020:00:120
1600299511,8cyclictest131rcu_preempt20:35:130
1600299503,8cyclictest131rcu_preempt20:25:130
1602299492,35cyclictest24688-21ssh23:30:132
1600299493,36cyclictest18917-21ssh23:45:130
1601399473,39cyclictest13251-21ssh00:00:121
16022994615,24cyclictest27-21ksoftirqd/223:10:122
1601399431,35cyclictest14330-21ssh21:35:111
16022994021,15cyclictest261rcuc/221:25:132
1601399405,26cyclictest27831-21ssh23:50:141
1601399391,28cyclictest6373-21ssh22:25:121
1603499383,21cyclictest28366irq/58-eth023:15:123
1603499381,22cyclictest28366irq/58-eth023:05:133
1601399383,22cyclictest16396-21ssh00:20:151
1603499371,21cyclictest28366irq/58-eth022:20:133
1603499371,21cyclictest28366irq/58-eth021:20:143
1600299374,28cyclictest15785-21ssh23:25:130
16002993720,9cyclictest131rcu_preempt23:30:140
1602299360,23cyclictest26635-21ssh22:00:132
1601399367,19cyclictest5344-21ssh21:30:131
1601399363,23cyclictest23394-21ssh21:40:141
1600299354,21cyclictest24202-21ssh22:35:120
1602299342,25cyclictest1654-21ssh23:35:162
1601399343,21cyclictest10733-21ssh00:35:141
1601399343,21cyclictest10733-21ssh00:35:131
1600299342,6cyclictest1751-21apt-get00:30:120
16034993316,12cyclictest28366irq/58-eth000:05:243
1603499331,21cyclictest28366irq/58-eth019:25:113
1602299331,15cyclictest12681-21ssh23:05:142
1600299332,19cyclictest12-21ksoftirqd/020:50:130
1603499323,21cyclictest28366irq/58-eth022:10:133
1601399322,22cyclictest1777-21ssh00:30:121
16002993214,9cyclictest12-21ksoftirqd/023:10:110
16022993123,4cyclictest531-21usb-storage00:10:112
1601399315,4cyclictest131rcu_preempt22:40:141
1600299313,22cyclictest28845-21ssh21:25:150
1600299312,16cyclictest12-21ksoftirqd/020:10:110
1600299312,16cyclictest12-21ksoftirqd/020:10:110
16034993018,4cyclictest28366irq/58-eth000:20:153
1602299305,9cyclictest131rcu_preempt23:45:282
1602299302,21cyclictest0-21swapper/219:50:052
1601399302,21cyclictest2015-21ssh21:10:131
1602299292,11cyclictest13977-21sh23:23:152
1601399292,20cyclictest12690-21diskmemload23:05:151
1600299294,13cyclictest12-21ksoftirqd/019:55:120
1603499284,13cyclictest28366irq/58-eth023:55:123
1603499283,18cyclictest0-21swapper/323:39:173
16034992812,9cyclictest28366irq/58-eth000:30:133
1603499281,17cyclictest28366irq/58-eth019:30:113
1602299285,13cyclictest22346irq/40-dwc2_hso19:10:122
1602299283,20cyclictest0-21swapper/223:53:152
1602299282,21cyclictest30386-21ssh00:09:512
1602299282,11cyclictest12024-21apt22:10:132
1601399283,17cyclictest10027-21ssh23:40:131
1601399282,20cyclictest30382-21ssh23:15:111
1600299288,9cyclictest131rcu_preempt20:20:140
1600299283,19cyclictest12051-21ssh22:10:140
1600299282,19cyclictest1166-21ssh23:35:120
1603499273,16cyclictest0-21swapper/323:45:163
1602299272,19cyclictest0-21swapper/219:45:132
1601399274,16cyclictest17747-21ssh21:55:121
1601399271,16cyclictest27397-21ssh22:55:141
16013992711,5cyclictest21-21ksoftirqd/122:15:251
1600299277,11cyclictest0-21swapper/019:50:140
1603499263,14cyclictest28366irq/58-eth022:30:123
1602299262,19cyclictest3860-21ssh23:00:132
1601399263,5cyclictest21-21ksoftirqd/120:50:261
1601399262,18cyclictest32334-21ssh21:45:131
16002992613,3cyclictest131rcu_preempt21:10:150
1603499251,13cyclictest28366irq/58-eth022:15:123
1602299257,11cyclictest131rcu_preempt21:30:182
1602299254,4cyclictest131rcu_preempt21:15:222
16022992517,5cyclictest13065-21ssh22:45:222
1601399251,3cyclictest131rcu_preempt21:20:161
1601399251,18cyclictest6923-21ssh23:20:121
16013992511,4cyclictest0-21swapper/122:06:071
1600299253,14cyclictest22111-21ssh00:05:130
1600299251,9cyclictest28441-21munin-run21:25:020
1603499242,13cyclictest28366irq/58-eth020:25:113
1603499241,13cyclictest28366irq/58-eth020:35:113
1602299242,15cyclictest22346irq/40-dwc2_hso21:05:122
16022992415,5cyclictest29890-21ssh22:20:132
1602299241,4cyclictest0-21swapper/221:10:182
1602299241,18cyclictest27-21ksoftirqd/223:55:122
16022992410,11cyclictest12584-21apt-get19:40:132
1601399248,5cyclictest21-21ksoftirqd/100:27:171
1601399247,12cyclictest21-21ksoftirqd/123:49:161
1601399246,9cyclictest8845-21ssh21:50:111
1601399243,4cyclictest21-21ksoftirqd/119:55:241
1601399243,3cyclictest131rcu_preempt00:10:241
1601399242,4cyclictest21-21ksoftirqd/123:56:311
1601399242,17cyclictest18461-21ssh22:50:141
16013992416,3cyclictest10258-21apt-get20:45:111
16013992414,6cyclictest0-21swapper/100:15:141
1600299242,17cyclictest9525-21ssh22:45:120
16002992415,5cyclictest0-21swapper/022:20:140
1603499233,13cyclictest28366irq/58-eth023:30:133
16034992312,7cyclictest28366irq/58-eth022:00:133
1602299237,13cyclictest21726-21kthreadcore22:15:182
1602299234,11cyclictest131rcu_preempt23:15:262
1602299232,6cyclictest633-21Async12:11:532
1602299232,2cyclictest0-21swapper/223:40:252
1602299232,13cyclictest131rcu_preempt19:30:252
16022992312,3cyclictest0-21swapper/220:55:272
16022992310,9cyclictest32284-21/usr/sbin/munin21:45:292
1602299230,7cyclictest4213-21latency22:40:242
1601399239,8cyclictest21-21ksoftirqd/122:22:431
1601399235,8cyclictest21-21ksoftirqd/121:25:141
1601399235,3cyclictest131rcu_preempt00:07:561
1601399235,14cyclictest0-21swapper/120:15:121
1601399234,3cyclictest131rcu_preempt22:35:251
1601399233,4cyclictest21-21ksoftirqd/123:10:151
1601399231,3cyclictest131rcu_preempt21:15:171
16013992311,10cyclictest0-21swapper/122:47:391
16013992310,4cyclictest21-21ksoftirqd/123:33:061
1600299237,11cyclictest151rcuc/000:15:120
1603499222,15cyclictest28366irq/58-eth022:55:143
1603499222,15cyclictest28366irq/58-eth022:45:123
1603499222,14cyclictest0-21swapper/321:35:113
1602299229,8cyclictest11074-21sh00:15:252
1602299229,5cyclictest0-21swapper/222:10:012
1602299223,13cyclictest131rcu_preempt20:00:012
1602299222,5cyclictest0-21swapper/200:29:512
1602299222,15cyclictest0-21swapper/200:33:512
1602299221,7cyclictest19298-21cut21:55:202
1602299221,6cyclictest26804-21ssh21:40:232
16022992212,4cyclictest0-21swapper/221:50:252
1601399229,7cyclictest131rcu_preempt19:20:151
1601399228,7cyclictest21-21ksoftirqd/123:36:081
1601399227,8cyclictest0-21swapper/120:55:131
1601399225,9cyclictest21-21ksoftirqd/119:55:011
1601399221,2cyclictest131rcu_preempt22:30:271
16013992210,4cyclictest21-21ksoftirqd/120:40:241
1600299228,10cyclictest15397-21cron23:25:010
1600299223,7cyclictest131rcu_preempt21:00:120
1600299222,16cyclictest7759-21apt-get19:35:120
1600299221,3cyclictest131rcu_preempt23:50:180
16002992211,8cyclictest6499-21diskstats22:25:140
16002992211,3cyclictest131rcu_preempt21:35:250
1603499219,9cyclictest0-21swapper/321:28:553
1603499219,9cyclictest0-21swapper/300:14:293
16034992110,6cyclictest0-21swapper/323:23:003
1602299218,7cyclictest1606-21diskmemload21:35:202
1602299217,5cyclictest0-21swapper/220:55:002
1602299217,11cyclictest24851-21cut22:35:172
1602299211,9cyclictest6944-21ssh22:25:162
1602299211,5cyclictest20047-21cut20:20:202
1602299211,4cyclictest0-21swapper/221:24:122
1602299211,16cyclictest23540-21fschecks_count20:25:152
1602299211,16cyclictest18777-21cut00:20:232
1602299211,15cyclictest14201-21latency00:35:242
1602299211,15cyclictest14201-21latency00:35:242
1602299211,12cyclictest0-21swapper/219:20:212
1602299211,11cyclictest15793-21sh23:25:132
1602299211,10cyclictest5108-21apt-get20:40:022
1602299211,10cyclictest15070-21kthreadcore20:15:202
1601399215,3cyclictest131rcu_preempt22:10:151
1601399214,3cyclictest131rcu_preempt23:00:211
1601399213,3cyclictest131rcu_preempt22:00:131
1601399211,13cyclictest21078-21grep19:15:161
1600299218,9cyclictest9036-21ntpq21:30:270
1600299211,7cyclictest13032-21latency_hist00:00:020
1600299211,3cyclictest131rcu_preempt23:40:110
1600299211,2cyclictest131rcu_preempt23:15:140
1603499209,8cyclictest0-21swapper/322:51:223
1603499202,10cyclictest28366irq/58-eth021:15:133
16034992011,6cyclictest28366irq/58-eth023:10:113
16034992010,8cyclictest0-21swapper/322:06:053
1602299207,6cyclictest19690-21cut22:50:202
1602299205,12cyclictest0-21swapper/220:10:192
1602299205,12cyclictest0-21swapper/220:10:182
1602299202,12cyclictest0-21swapper/220:40:032
1602299201,8cyclictest0-21swapper/220:45:112
1602299201,7cyclictest28509-21kthreadcore22:55:212
1602299201,4cyclictest31916-21sort20:00:142
1601399205,3cyclictest131rcu_preempt23:25:391
1601399202,5cyclictest21-21ksoftirqd/119:25:261
1601399202,4cyclictest21-21ksoftirqd/119:40:191
1601399201,3cyclictest131rcu_preempt20:05:251
16013992011,7cyclictest31493-21memory20:30:221
1601399201,11cyclictest11833-21cut20:10:201
1601399201,11cyclictest11833-21cut20:10:191
1600299201,2cyclictest131rcu_preempt22:50:270
16002992011,6cyclictest27908-21cat21:40:310
1603499199,3cyclictest0-21swapper/322:29:233
1603499193,10cyclictest0-21swapper/323:50:143
1603499192,12cyclictest28366irq/58-eth021:55:123
1603499192,10cyclictest0-21swapper/321:05:123
1603499192,10cyclictest0-21swapper/320:55:133
16034991912,5cyclictest28366irq/58-eth021:45:113
1603499191,13cyclictest0-21swapper/300:05:023
1603499191,12cyclictest28366irq/58-eth023:40:133
1603499191,12cyclictest28366irq/58-eth019:10:133
1603499191,12cyclictest0-21swapper/322:35:123
1603499191,12cyclictest0-21swapper/320:15:013
1603499191,12cyclictest0-21swapper/320:15:013
1603499191,12cyclictest0-21swapper/319:15:283
16034991910,6cyclictest28366irq/58-eth023:25:133
16034991910,6cyclictest0-21swapper/300:35:273
16034991910,6cyclictest0-21swapper/300:35:273
16034991910,6cyclictest0-21swapper/300:26:463
1602299199,7cyclictest28325-21fschecks_count20:30:152
1602299191,9cyclictest7949-21fschecks_time19:35:152
1602299191,9cyclictest4862-21kernelversion20:05:202
1602299191,9cyclictest21065-21fschecks_count19:15:162
16022991913,4cyclictest0-21swapper/219:25:142
1601399195,3cyclictest131rcu_preempt20:35:211
1601399195,3cyclictest131rcu_preempt19:35:011
1601399193,13cyclictest25945-21kthreadcore20:25:211
16013991912,5cyclictest0-21swapper/120:00:301
1601399191,15cyclictest7750-21find19:35:111
1601399191,15cyclictest22258-21ls20:20:281
1600299199,8cyclictest8846-21apt-get21:50:120
1600299199,7cyclictest1-21systemd19:46:040
1600299196,4cyclictest131rcu_preempt20:56:180
1600299194,3cyclictest131rcu_preempt22:05:160
1600299193,3cyclictest0-21swapper/022:30:140
1600299192,14cyclictest1168-21kthreadcore21:45:190
1600299191,9cyclictest540-21kthreadcore19:25:190
1600299191,15cyclictest10712-21apt-get00:35:130
1600299191,15cyclictest10712-21apt-get00:35:130
1600299191,13cyclictest805-21df22:40:150
1600299191,13cyclictest30899-21sh22:55:250
1600299191,13cyclictest30222-21netstat22:00:230
16002991910,6cyclictest17722-21ssh21:55:110
1603499188,8cyclictest0-21swapper/322:40:023
1603499183,9cyclictest28466irq/59-eth021:50:113
1603499183,9cyclictest0-21swapper/320:40:133
1603499182,10cyclictest0-21swapper/320:50:133
1603499182,10cyclictest0-21swapper/320:45:133
1603499182,10cyclictest0-21swapper/319:55:133
1603499182,10cyclictest0-21swapper/319:35:133
16034991812,4cyclictest0-21swapper/321:30:273
1602299189,6cyclictest0-21swapper/219:10:022
1602299181,9cyclictest24693-21sort21:00:142
1601399184,11cyclictest20916-21sendmail_mailst19:45:261
1601399181,3cyclictest131rcu_preempt21:05:121
1601399181,3cyclictest131rcu_preempt19:10:271
16013991811,5cyclictest24845-21fw_packets21:00:161
1600299189,6cyclictest20710-21cron19:15:010
1600299185,4cyclictest9161-21timerandwakeup20:40:270
1600299182,12cyclictest20523-21apt-get22:15:030
1600299181,8cyclictest6382-21kthreadcore23:00:210
16002991812,4cyclictest0-21swapper/000:20:150
16002991811,5cyclictest24481-21awk19:15:260
1600299181,14cyclictest17066-21latency20:15:220
16002991810,5cyclictest16594-21switchtime23:05:290
1603499179,6cyclictest0-21swapper/321:42:073
1603499175,6cyclictest28366irq/58-eth021:10:133
1603499171,13cyclictest0-21swapper/320:00:223
1603499171,12cyclictest0-21swapper/319:20:283
1603499171,11cyclictest0-21swapper/320:20:233
1603499171,10cyclictest0-21swapper/321:00:133
1600299174,8cyclictest0-21swapper/019:30:120
1600299172,11cyclictest8025-21tail20:05:290
1600299171,13cyclictest10966-21ssh21:15:140
1600299171,12cyclictest15977-21kthreadcore00:00:220
1600299170,2cyclictest131rcu_preempt19:40:140
1603499162,9cyclictest0-21swapper/320:05:153
1603499162,9cyclictest0-21swapper/319:45:133
1603499161,12cyclictest0-21swapper/319:45:013
1600299167,7cyclictest25927-21diskstats19:20:160
1600299163,10cyclictest10416-21fschecks_time20:45:150
1600299162,9cyclictest0-21swapper/020:30:130
1600299151,10cyclictest16041-21latency_hist19:10:020
1603499142,10cyclictest0-21swapper/320:30:133
1603499138,3cyclictest0-21swapper/320:18:043
1601399131,9cyclictest16057-21latency_hist19:10:021
1603499121,5cyclictest0-21swapper/319:50:143
1603499101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional