You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 14:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Mar 27, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3245199553,41cyclictest30903-21ssh22:00:111
3245199494,35cyclictest22782-21ssh21:30:111
3245199434,32cyclictest6546-21ssh22:30:141
3246099421,21cyclictest10685-21ssh00:20:132
3245199424,31cyclictest24966-21ssh00:10:111
3244099421,37cyclictest24741-21ssh22:40:140
3244099421,36cyclictest8540-21ssh21:40:130
3245199417,27cyclictest18429-21diskmemload00:30:141
3244099411,36cyclictest6943-21ssh00:00:140
3244099401,35cyclictest20894-21ssh22:20:160
3244099382,30cyclictest15589-21ssh22:35:140
3244099382,29cyclictest13679-21ssh21:25:110
3246099379,13cyclictest27-21ksoftirqd/200:27:282
3245199373,24cyclictest8706-21ssh23:25:111
3246099362,29cyclictest9061-21ssh00:18:342
32440993632,2cyclictest531-21usb-storage20:10:120
3244099362,28cyclictest15955-21ssh00:05:140
3244099361,25cyclictest0-21swapper/019:50:120
3246899353,21cyclictest28366irq/58-eth021:35:133
3246899352,20cyclictest28366irq/58-eth000:00:123
3244099353,8cyclictest12-21ksoftirqd/021:15:130
32440993522,7cyclictest0-21swapper/023:50:120
3244099352,23cyclictest12-21ksoftirqd/021:10:120
3246099341,27cyclictest924-21snmpd20:40:112
3244099341,28cyclictest23112-21ssh23:15:140
3244099341,24cyclictest5356-21sh00:35:140
3246899334,16cyclictest28366irq/58-eth021:30:123
3245199333,24cyclictest32087-21ssh23:20:121
3244099335,21cyclictest30918-21ssh22:00:130
3244099334,22cyclictest1368-21ssh22:45:140
3244099332,23cyclictest19429-21ssh22:55:140
3244099331,19cyclictest5087-21sh23:05:130
3246899323,16cyclictest28366irq/58-eth023:40:123
3246899323,16cyclictest28366irq/58-eth023:40:123
3244099326,3cyclictest131rcu_preempt00:25:140
3244099322,22cyclictest17709-21ssh23:30:130
32468993118,8cyclictest28366irq/58-eth023:00:133
3246099311,24cyclictest6934-21ssh00:00:132
3244099312,23cyclictest8763-21ssh00:18:130
3246899302,17cyclictest28366irq/58-eth022:50:123
3246899302,17cyclictest28366irq/58-eth022:50:113
3245199301,21cyclictest0-21swapper/119:35:151
3246099299,16cyclictest27-21ksoftirqd/221:05:112
3246099293,18cyclictest22346irq/40-dwc2_hso23:25:112
32460992912,14cyclictest9535-21ssh00:35:282
3245199292,20cyclictest12343-21ssh23:45:121
3245199292,19cyclictest5424-21ssh00:15:281
32468992817,7cyclictest28366irq/58-eth023:25:123
32468992815,8cyclictest0-21swapper/323:20:123
3246099282,20cyclictest0-21swapper/219:35:152
3246099281,20cyclictest30429-21ssh23:55:172
32451992813,3cyclictest21-21ksoftirqd/100:20:131
3246899277,11cyclictest28366irq/58-eth022:30:143
32460992713,8cyclictest27-21ksoftirqd/200:30:142
32460992712,7cyclictest28529-21forks23:00:162
3245199276,11cyclictest32436-21cyclictest21:35:131
3244099271,9cyclictest12-21ksoftirqd/023:55:160
3246899262,18cyclictest0-21swapper/300:13:283
32468992614,5cyclictest0-21swapper/321:15:113
3246099261,2cyclictest131rcu_preempt22:25:142
3246099261,17cyclictest30410-21ls20:15:012
3246099261,13cyclictest6562-21sh22:30:152
3245199262,19cyclictest28401-21ssh23:00:141
3244099262,18cyclictest4593-21ssh21:20:140
3244099262,17cyclictest15748-21kworker/u8:023:40:140
3244099262,17cyclictest15748-21kworker/u8:023:40:130
3246899253,17cyclictest0-21swapper/300:17:283
32468992519,3cyclictest28366irq/58-eth022:55:143
3246099254,16cyclictest22346irq/40-dwc2_hso19:45:132
3246099254,16cyclictest22346irq/40-dwc2_hso19:45:132
3246099254,13cyclictest22346irq/40-dwc2_hso21:00:112
3246099253,17cyclictest633-21Async12:11:532
32460992512,6cyclictest28222-21memory22:40:222
32460992511,3cyclictest18429-21diskmemload23:24:322
3245199257,9cyclictest0-21swapper/123:55:141
3244099252,16cyclictest12-21ksoftirqd/019:30:120
32440992516,6cyclictest0-21swapper/023:35:120
3244099251,18cyclictest0-21swapper/019:25:130
3244099251,17cyclictest29952-21sh22:25:160
3246899242,12cyclictest28366irq/58-eth022:00:123
3246099244,16cyclictest22346irq/40-dwc2_hso21:15:132
3246099242,18cyclictest0-21swapper/219:10:142
32460992411,10cyclictest7162-21cron22:05:012
32451992412,3cyclictest131rcu_preempt20:20:021
3244099243,13cyclictest14126-21ssh23:10:120
32460992312,9cyclictest9789-21ssh21:20:532
32460992312,9cyclictest5002-21ssh22:45:252
3246099231,14cyclictest0-21swapper/221:37:102
32460992311,3cyclictest27450-21kernelversion23:35:192
32460992311,10cyclictest32389-21memory19:40:222
3245199238,4cyclictest131rcu_preempt21:15:151
32440992312,3cyclictest12-21ksoftirqd/021:33:120
32468992212,4cyclictest0-21swapper/322:17:453
32468992210,9cyclictest0-21swapper/321:45:293
32460992211,9cyclictest24167-21ssh21:52:482
32460992211,9cyclictest23342-21ssh23:31:282
32460992211,9cyclictest21963-21ssh23:14:072
3246099221,13cyclictest0-21swapper/221:40:322
3246099221,11cyclictest924-21snmpd23:06:362
32460992210,9cyclictest28021-21ssh23:15:372
32460992210,9cyclictest26993-21ssh21:55:532
3246099220,13cyclictest0-21swapper/223:49:082
3245199229,3cyclictest131rcu_preempt22:25:021
3245199222,14cyclictest23536-21ssh00:25:291
3245199221,6cyclictest19114-21memory22:35:241
32451992210,2cyclictest21-21ksoftirqd/120:50:111
3244099222,11cyclictest151rcuc/019:20:100
3246899219,8cyclictest0-21swapper/322:45:173
3246899219,7cyclictest0-21swapper/323:15:043
3246899213,14cyclictest0-21swapper/322:35:133
32468992114,4cyclictest28366irq/58-eth022:25:163
3246099219,5cyclictest2708-21sshd20:18:172
3246099213,4cyclictest27-21ksoftirqd/222:55:302
3246099211,5cyclictest7578-21ls20:25:012
3246099211,16cyclictest10411-21ssh22:50:132
3246099211,16cyclictest10411-21ssh22:50:122
3246099211,10cyclictest6839-21missed_timers23:40:242
3246099211,10cyclictest6839-21missed_timers23:40:242
32460992110,9cyclictest23315-21ssh00:08:302
32460992110,9cyclictest22094-21ssh23:50:182
32460992110,9cyclictest20573-21ssh22:35:392
32460992110,9cyclictest15496-21ssh22:14:272
32460992110,9cyclictest15496-21ssh22:14:272
3245199219,9cyclictest27969-21cron23:00:011
3245199211,14cyclictest839-21munin-node20:40:131
3244099219,3cyclictest12-21ksoftirqd/022:30:250
3244099214,10cyclictest0-21swapper/020:30:140
32440992115,3cyclictest12-21ksoftirqd/000:30:220
32440992112,6cyclictest6065-21cron19:50:010
32440992112,6cyclictest6065-21cron19:50:000
32440992111,3cyclictest12-21ksoftirqd/020:15:230
32440992110,3cyclictest12-21ksoftirqd/021:35:110
32440992110,2cyclictest12-21ksoftirqd/022:51:020
32440992110,2cyclictest12-21ksoftirqd/022:51:020
3246899209,6cyclictest0-21swapper/322:40:413
3246899208,7cyclictest28366irq/58-eth023:45:123
3246899201,10cyclictest0-21swapper/300:20:123
32468992010,4cyclictest0-21swapper/321:13:013
3246099209,9cyclictest10877-21ssh22:09:092
3246099209,5cyclictest9597-21cut20:25:202
3246099209,5cyclictest24236-21pluginstate20:05:292
3246099207,10cyclictest22798-21diskmemload21:30:122
3246099206,7cyclictest16019-21df_inode20:00:132
3246099201,11cyclictest22669-21/usr/sbin/munin21:10:272
3246099201,10cyclictest6413-21sort19:50:152
3246099201,10cyclictest21282-21grep20:35:292
3246099200,17cyclictest21720-21kthreadcore19:30:212
3245199207,3cyclictest131rcu_preempt00:35:211
3245199202,13cyclictest3334-21ssh23:40:131
3245199202,13cyclictest3334-21ssh23:40:121
32451992012,6cyclictest29877-21sshd21:59:171
32451992012,6cyclictest11102-21memory20:25:231
3245199201,11cyclictest20655-21kworker/1:023:52:151
3245199200,13cyclictest18428-21latency_hist21:10:011
3244099209,6cyclictest0-21swapper/000:13:340
3244099209,3cyclictest12-21ksoftirqd/021:55:390
3244099209,3cyclictest12-21ksoftirqd/021:05:280
3244099208,3cyclictest12-21ksoftirqd/019:45:010
3244099208,2cyclictest12-21ksoftirqd/022:10:000
3244099206,12cyclictest12-21ksoftirqd/019:55:120
3244099203,3cyclictest131rcu_preempt20:05:160
3244099202,13cyclictest13044-21irqstats23:45:180
32440992014,3cyclictest12-21ksoftirqd/019:40:000
32440992012,4cyclictest12-21ksoftirqd/021:00:020
32440992011,6cyclictest11238-21ps00:20:190
32440992010,2cyclictest12-21ksoftirqd/023:25:110
3246899199,7cyclictest0-21swapper/300:05:273
3246899198,8cyclictest0-21swapper/322:11:133
3246899198,8cyclictest0-21swapper/322:11:123
3246899198,8cyclictest0-21swapper/321:26:023
3246899195,11cyclictest0-21swapper/323:05:203
3246899193,11cyclictest0-21swapper/300:26:293
3246899192,13cyclictest28466irq/59-eth023:35:123
32468991914,3cyclictest33-21ksoftirqd/323:10:173
32468991914,3cyclictest28366irq/58-eth021:20:133
32468991911,6cyclictest0-21swapper/321:45:023
32468991910,6cyclictest0-21swapper/322:20:253
3246099192,9cyclictest15705-21awk20:00:022
3246099191,9cyclictest20890-21ssh22:20:172
3246099191,9cyclictest17365-21grep20:35:012
3246099191,9cyclictest0-21swapper/222:18:072
3246099191,8cyclictest7227-21kthreadcore19:15:282
3246099191,8cyclictest29607-21kthreadcore20:45:222
3246099191,8cyclictest29607-21kthreadcore20:45:222
3246099191,15cyclictest4427-21uniq20:55:172
3246099191,15cyclictest21633-21cron21:50:022
3246099191,15cyclictest14421-21kernelversion21:25:192
3246099191,15cyclictest14418-21idleruntime-cro19:25:002
3246099191,10cyclictest32614-21kthreadcore20:50:182
3246099191,10cyclictest16439-21kthreadcore19:25:192
3245199196,3cyclictest131rcu_preempt21:25:121
3245199194,10cyclictest16123-21fschecks_count00:05:171
3245199193,8cyclictest131rcu_preempt21:49:401
3245199192,9cyclictest18429-21diskmemload21:10:311
3245199191,9cyclictest9461-21kthreadcore21:40:181
3245199191,14cyclictest1166-21ssh22:25:261
3245199191,10cyclictest19398-21latency_hist19:30:021
3244099196,3cyclictest0-21swapper/022:17:100
3244099191,8cyclictest8337-21munin-run23:25:010
3244099191,3cyclictest0-21swapper/023:00:150
32440991913,3cyclictest12-21ksoftirqd/019:10:240
3244099191,2cyclictest131rcu_preempt20:20:140
32440991911,5cyclictest12-21ksoftirqd/021:00:210
3246899182,10cyclictest0-21swapper/323:55:163
3246899182,10cyclictest0-21swapper/319:45:133
3246899182,10cyclictest0-21swapper/319:45:123
3246899182,10cyclictest0-21swapper/319:40:143
3246899182,10cyclictest0-21swapper/319:35:153
32468991812,3cyclictest33-21ksoftirqd/320:30:293
32468991810,6cyclictest0-21swapper/322:08:273
32468991810,5cyclictest28366irq/58-eth000:35:143
3246099181,8cyclictest32523-21latency_hist19:10:022
3245199189,6cyclictest10096-21wc19:20:141
3245199188,4cyclictest18429-21diskmemload22:07:131
3245199184,6cyclictest295711tr23:35:211
3245199183,12cyclictest23379-21sed23:15:161
3245199181,9cyclictest15912-21/usr/sbin/munin20:00:291
3245199181,7cyclictest5145-21/usr/sbin/munin22:45:271
32451991811,5cyclictest0-21swapper/119:15:291
3245199181,14cyclictest2009-21kthreadcore19:10:201
32451991810,6cyclictest9797-21awk19:50:271
3244099182,12cyclictest9626-21munin-run19:20:010
3244099181,4cyclictest131rcu_preempt20:37:480
32440991810,6cyclictest26054-21cron21:55:010
3246899179,6cyclictest0-21swapper/323:50:483
3246899179,6cyclictest0-21swapper/323:30:053
3246899179,6cyclictest0-21swapper/321:56:433
3246899179,6cyclictest0-21swapper/321:50:153
3246899172,10cyclictest0-21swapper/321:00:133
3246899171,13cyclictest28366irq/58-eth000:30:133
3246899171,12cyclictest0-21swapper/319:50:163
3246899171,11cyclictest0-21swapper/320:45:183
3246899171,11cyclictest0-21swapper/320:45:173
3245199178,6cyclictest14333-21sshd22:13:021
3245199178,6cyclictest14333-21sshd22:13:011
3245199178,5cyclictest21-21ksoftirqd/123:05:301
3245199177,7cyclictest13285-21cron21:25:011
3245199173,12cyclictest12758-21kthreadcore19:55:191
3245199172,12cyclictest17699-21apt-get23:30:141
3245199171,6cyclictest24243-21apt-get22:40:031
3245199171,2cyclictest131rcu_preempt19:30:131
3245199171,13cyclictest3603-21kthreadcore19:45:201
3245199171,13cyclictest3603-21kthreadcore19:45:191
3245199171,13cyclictest14239-21diskstats23:10:151
3245199171,13cyclictest11129-21grep00:00:331
32451991710,5cyclictest8810-21sshd20:59:151
32451991710,5cyclictest22024-21sshd20:37:551
32451991710,5cyclictest10471-21awk22:50:141
32451991710,5cyclictest10471-21awk22:50:131
3244099178,4cyclictest12-21ksoftirqd/020:25:120
3244099172,12cyclictest27158-21diskstats20:45:160
3244099172,12cyclictest27158-21diskstats20:45:150
32440991711,3cyclictest12-21ksoftirqd/022:11:520
32440991711,3cyclictest12-21ksoftirqd/022:11:510
32440991710,4cyclictest12-21ksoftirqd/021:48:070
32468991611,3cyclictest33-21ksoftirqd/320:25:293
3246899161,12cyclictest0-21swapper/320:40:013
3245199162,11cyclictest17142-21apt-get20:35:021
3245199161,7cyclictest29424-21tail19:40:171
3245199161,7cyclictest27441-21kthreadcore20:10:201
3245199161,12cyclictest22309-21tr20:05:231
32451991610,4cyclictest0-21swapper/122:15:261
3245199160,13cyclictest2970-21apt-get20:20:131
3244099169,5cyclictest22941-21sed20:40:180
3244099169,4cyclictest12-21ksoftirqd/020:50:100
3244099161,12cyclictest19773-21grep20:00:290
3245199159,4cyclictest0-21swapper/121:53:431
3245199152,10cyclictest201rcuc/121:00:121
3245199151,11cyclictest27588-21sed20:45:191
3245199151,11cyclictest27588-21sed20:45:181
3246899149,3cyclictest33-21ksoftirqd/319:10:113
3246899148,4cyclictest33-21ksoftirqd/319:30:173
3246899142,8cyclictest0-21swapper/320:40:123
3246899141,10cyclictest0-21swapper/319:20:143
3244099141,11cyclictest32443-21grep19:10:020
3246899139,2cyclictest33-21ksoftirqd/320:55:233
3246899138,3cyclictest0-21swapper/320:20:183
3246899133,8cyclictest0-21swapper/319:25:143
3246899132,9cyclictest0-21swapper/320:10:053
3246899132,8cyclictest28366irq/58-eth019:15:123
3246899131,9cyclictest0-21swapper/320:05:143
3245199130,10cyclictest32519-21latency_hist19:10:011
3246899128,2cyclictest33-21ksoftirqd/319:56:423
3246899126,4cyclictest28366irq/58-eth020:16:163
3246899124,6cyclictest28366irq/58-eth020:50:113
3246899122,6cyclictest0-21swapper/321:05:123
3246899122,6cyclictest0-21swapper/320:00:133
3246899101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional