You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 18:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Nov 28, 2025 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2761399693,42cyclictest8870-21ssh12:00:122
2760599686,48cyclictest4663-21ssh10:30:151
27621996450,5cyclictest29566irq/58-eth011:30:133
2760599614,45cyclictest14982-21ssh12:25:101
2761399607,15cyclictest131rcu_preempt09:00:132
2761399573,17cyclictest131rcu_preempt11:05:132
2760599540,44cyclictest7579-21ssh12:20:141
2760599523,36cyclictest13495-21ssh11:20:121
2760599492,33cyclictest4602-21ssh09:25:121
2761399483,20cyclictest27-21ksoftirqd/208:40:122
27613994810,31cyclictest27-21ksoftirqd/207:30:122
2761399461,30cyclictest32463-21ssh11:10:122
2760599464,34cyclictest18375-21ssh11:45:161
27613994526,11cyclictest0-21swapper/211:30:132
2761399441,27cyclictest27120-21ssh10:45:152
2761399441,24cyclictest11179-21ssh11:40:132
27605994311,20cyclictest21-21ksoftirqd/107:50:131
2759499413,34cyclictest19393-21ssh09:35:150
2760599405,26cyclictest13850-21diskmemload12:05:161
2761399393,13cyclictest0-21swapper/212:25:122
2761399392,19cyclictest6996-21ps10:10:192
27594993814,19cyclictest14209-21ssh09:10:100
27613993722,10cyclictest31164-21ssh12:13:392
2761399372,18cyclictest15771-21ssh10:35:262
2761399372,18cyclictest14318-21cat10:55:292
27594993719,11cyclictest12-21ksoftirqd/008:35:110
27594993714,17cyclictest32377-21ssh12:15:130
27594993616,15cyclictest12-21ksoftirqd/011:05:120
27621993515,10cyclictest29566irq/58-eth010:50:143
2761399352,17cyclictest21511-21awk11:00:302
27613993513,16cyclictest27-21ksoftirqd/209:45:132
2760599356,19cyclictest73-21kswapd011:00:141
27605993426,3cyclictest32623-21apt-get07:15:121
27605993421,9cyclictest32362-21apt-get12:15:111
2759499343,20cyclictest12-21ksoftirqd/009:50:130
27621993317,9cyclictest29566irq/58-eth011:45:153
2761399322,19cyclictest26764-21ssh09:40:142
2761399322,19cyclictest26764-21ssh09:40:132
2761399322,14cyclictest25449-21ssh10:42:392
27605993225,3cyclictest6915-21apt-get11:15:131
27605993222,5cyclictest1605-21apt-get09:45:141
2762199315,21cyclictest29566irq/58-eth009:30:133
2762199313,12cyclictest29566irq/58-eth011:20:143
2761399313,17cyclictest26346irq/40-dwc2_hso09:55:132
27613993117,7cyclictest629-21Async14:00:482
2762199301,19cyclictest29566irq/58-eth008:10:113
27613993023,2cyclictest0-21swapper/209:20:122
2761399302,21cyclictest0-21swapper/210:20:122
2761399302,21cyclictest0-21swapper/207:35:132
2761399302,15cyclictest3620-21ssh11:35:132
2760599304,8cyclictest29409-21ntp_192.168.11411:50:241
2760599301,21cyclictest2231-21ssh10:50:141
27594993016,9cyclictest23869-21ssh10:00:140
2759499301,22cyclictest1219-21ssh11:55:120
27594993010,14cyclictest31445-21diskmemload10:05:140
2762199294,20cyclictest29566irq/58-eth012:00:143
2762199292,17cyclictest29566irq/58-eth011:55:123
2761399292,11cyclictest5936-21ssh12:17:392
27605992922,4cyclictest24991-21apt-get11:05:121
2762199283,16cyclictest29566irq/58-eth009:10:103
2761399283,20cyclictest0-21swapper/210:50:392
2760599289,15cyclictest0-21swapper/109:00:121
27605992821,4cyclictest27811-21apt-get07:10:141
27605992821,4cyclictest27811-21apt-get07:10:131
2762199272,18cyclictest0-21swapper/310:44:403
2761399273,17cyclictest26346irq/40-dwc2_hso07:25:132
2761399272,20cyclictest0-21swapper/207:10:152
2761399272,20cyclictest0-21swapper/207:10:142
2760599274,19cyclictest29533-21ssh10:25:141
2760599274,15cyclictest19733-1kworker/u9:211:25:151
27605992722,2cyclictest25748-21apt-get08:15:131
2760599272,19cyclictest19593-21ssh10:40:131
27605992718,5cyclictest14200-21apt-get09:10:111
27605992716,7cyclictest21874-21apt-get09:15:151
2759499272,13cyclictest21881-21ssh09:15:140
2762199263,15cyclictest0-21swapper/310:35:253
2761399262,15cyclictest26346irq/40-dwc2_hso07:55:112
2761399262,14cyclictest14224-21ssh09:10:112
27613992613,8cyclictest27-21ksoftirqd/210:05:122
2760599263,9cyclictest25540-21wc08:15:011
27605992620,4cyclictest31427-21apt-get10:05:131
27605992618,4cyclictest24324-21apt-get07:40:161
27605992618,4cyclictest24324-21apt-get07:40:151
27605992613,10cyclictest3611-21apt-get11:35:151
2759499266,12cyclictest0-21swapper/011:50:130
2759499262,16cyclictest4668-21ssh10:30:150
27594992614,6cyclictest131rcu_preempt11:35:120
2761399259,12cyclictest26346irq/40-dwc2_hso11:55:112
2761399253,17cyclictest26346irq/40-dwc2_hso07:15:132
2761399253,15cyclictest26346irq/40-dwc2_hso07:40:142
2761399253,15cyclictest26346irq/40-dwc2_hso07:40:132
2761399253,14cyclictest26346irq/40-dwc2_hso09:35:152
2761399252,7cyclictest0-21swapper/209:50:152
2761399252,18cyclictest7586-21sh12:20:142
2761399252,16cyclictest0-21swapper/208:20:042
2761399251,17cyclictest25781-21find11:50:112
2761399251,11cyclictest29404-21ls08:15:272
2760599252,8cyclictest25287-21ntp_192.168.11410:20:221
27605992520,3cyclictest3074-21apt-get08:25:111
27605992520,2cyclictest5036-21apt-get07:20:141
27605992519,3cyclictest1198-21apt-get11:55:131
2760599251,11cyclictest1932-21kthreadcore11:10:201
2759499253,17cyclictest14200-21ssh10:15:120
2759499252,5cyclictest131rcu_preempt12:05:150
2762199249,11cyclictest29566irq/58-eth009:15:153
2762199241,13cyclictest29566irq/58-eth010:15:133
2761399243,14cyclictest26346irq/40-dwc2_hso08:05:122
2761399242,19cyclictest13850-21diskmemload11:25:162
2761399242,12cyclictest0-21swapper/209:05:162
2761399242,12cyclictest0-21swapper/209:05:152
2761399241,17cyclictest3184-21uniq08:25:152
27605992418,3cyclictest8863-21apt-get12:00:131
27605992412,9cyclictest16113-21ntp_192.168.11408:35:211
2760599241,17cyclictest29447-21ssh09:20:141
27605992411,7cyclictest12028-21apt-get09:30:131
2760599241,17cyclictest11302-21find08:00:111
2760599241,12cyclictest0-21swapper/109:39:211
2759499241,18cyclictest709-21ssh10:47:390
2762199238,10cyclictest0-21swapper/310:10:393
27621992313,5cyclictest29566irq/58-eth009:25:123
27621992311,7cyclictest29566irq/58-eth012:35:263
2761399233,16cyclictest0-21swapper/210:18:382
2761399232,6cyclictest0-21swapper/210:04:402
2761399231,8cyclictest31769-21latency_hist08:55:012
27613992312,7cyclictest27-21ksoftirqd/207:20:132
27613992312,5cyclictest22732-21ssh12:07:532
2761399230,10cyclictest15844-21ls09:30:282
2760599236,5cyclictest0-21swapper/109:40:131
2760599236,5cyclictest0-21swapper/109:40:121
27605992317,3cyclictest28438-21apt-get11:30:131
27605992311,9cyclictest11159-21ssh11:40:131
2759499232,15cyclictest28462-21ssh11:30:150
2759499231,10cyclictest15333-21ps12:25:160
2762199224,13cyclictest0-21swapper/309:45:153
2761399224,8cyclictest26018-21missed_timers12:30:262
2761399221,9cyclictest0-21swapper/208:55:122
2761399221,6cyclictest0-21swapper/209:25:132
27613992210,9cyclictest3265-21ssh12:37:032
2760599229,9cyclictest3399-21grep08:55:281
2760599228,10cyclictest12455-21cron08:35:001
27605992212,7cyclictest21-21ksoftirqd/108:50:141
2760599221,18cyclictest22697-21sh12:30:191
27605992211,7cyclictest15431-21kthreadcore10:35:201
2760599221,10cyclictest0-21swapper/110:15:131
27594992212,7cyclictest24070-21cron07:40:000
2762199219,9cyclictest0-21swapper/310:29:383
2762199213,11cyclictest0-21swapper/310:00:143
2761399214,9cyclictest15119-21grep08:00:282
2761399212,4cyclictest0-21swapper/208:10:112
2761399212,15cyclictest0-21swapper/210:30:152
2761399211,9cyclictest9775-21kthreadcore11:15:212
2761399211,6cyclictest21886-21ssh09:15:152
2761399211,17cyclictest14256-21kthreadcore08:35:182
2761399211,11cyclictest27758-21cat07:10:012
2761399211,11cyclictest1305-21munin-run07:50:012
2760599219,9cyclictest19673-21find08:05:271
2760599218,9cyclictest0-21swapper/107:25:121
2760599218,10cyclictest1882-21latency_hist10:50:011
2760599218,10cyclictest0-21swapper/112:35:161
2760599213,7cyclictest13850-21diskmemload12:11:031
27605992115,3cyclictest8878-21apt-get09:50:141
2760599210,9cyclictest22695-21sh09:58:341
2759499213,16cyclictest29655-21fschecks_time09:20:160
2759499212,14cyclictest842-21munin-node11:10:140
2759499211,16cyclictest8531-21latency_hist09:50:020
2759499211,15cyclictest19705-21ssh10:40:140
2762199202,12cyclictest0-21swapper/311:35:133
27621992011,3cyclictest29566irq/58-eth009:20:133
2761399207,10cyclictest1838-21wc07:50:172
2761399201,16cyclictest10974-21kthreadcore08:30:202
2761399201,10cyclictest31820-21kthreadcore10:25:222
2761399201,10cyclictest15522-21kthreadcore11:20:202
2761399200,17cyclictest22357-21find08:45:122
2760599206,11cyclictest16283-21ssh10:58:411
2760599204,4cyclictest0-21swapper/110:12:021
27605992011,6cyclictest0-21swapper/107:45:261
2759499209,8cyclictest7002-21cpuspeed11:15:130
2759499202,15cyclictest6648-21ssh10:10:140
2759499202,13cyclictest1-21systemd07:15:120
27594992012,6cyclictest135-21systemd-journal07:55:010
27594992012,5cyclictest30774-21mailstats08:50:280
27594992011,7cyclictest17571-21grep11:00:180
2759499201,16cyclictest6911-21ps07:20:220
2759499201,16cyclictest17044-21awk11:00:000
27594992011,6cyclictest0-21swapper/009:05:330
27594992011,6cyclictest0-21swapper/009:05:320
2759499201,10cyclictest18601-21/usr/sbin/munin11:45:180
2762199199,8cyclictest0-21swapper/312:29:203
2762199199,6cyclictest0-21swapper/310:21:053
2762199194,9cyclictest0-21swapper/309:50:143
2762199193,11cyclictest0-21swapper/310:45:393
2762199193,10cyclictest0-21swapper/307:25:263
2762199192,13cyclictest0-21swapper/310:05:143
2762199192,11cyclictest0-21swapper/308:20:053
27621991911,6cyclictest0-21swapper/312:30:223
27621991911,6cyclictest0-21swapper/310:30:283
27621991911,5cyclictest0-21swapper/312:16:553
27621991911,3cyclictest29566irq/58-eth011:25:143
2762199191,12cyclictest0-21swapper/309:00:233
2762199191,12cyclictest0-21swapper/307:50:163
2762199191,12cyclictest0-21swapper/307:35:023
2760599198,8cyclictest9884-21irqstats09:05:191
2760599198,8cyclictest9884-21irqstats09:05:191
2760599198,8cyclictest19580-21df_inode07:35:151
2760599197,9cyclictest6816-21cut07:55:161
2760599197,9cyclictest0-21swapper/107:30:111
2760599195,11cyclictest17673-21fschecks_count08:40:161
2760599192,12cyclictest23840-21/usr/sbin/munin10:00:321
2759499199,7cyclictest26317-21usage_EthcTask10:20:280
2759499192,13cyclictest8869-21sh12:00:140
2759499192,13cyclictest0-21swapper/007:45:140
2759499191,9cyclictest3056-21/usr/sbin/munin08:25:160
27594991913,4cyclictest26621-21uptime12:30:320
27594991913,4cyclictest0-21swapper/010:00:020
27594991911,6cyclictest26937-21cron08:50:010
27594991911,5cyclictest4839-21kthreadcore10:50:220
2759499191,14cyclictest29548-21ssh10:25:160
2759499191,14cyclictest28243-21cat11:30:020
2762199189,6cyclictest0-21swapper/311:43:503
2762199189,6cyclictest0-21swapper/311:05:333
2762199187,8cyclictest0-21swapper/309:40:193
2762199187,8cyclictest0-21swapper/309:40:183
2762199182,8cyclictest0-21swapper/308:50:133
2762199182,13cyclictest0-21swapper/311:10:133
2762199182,11cyclictest0-21swapper/311:02:393
2762199182,11cyclictest0-21swapper/310:57:383
2762199182,10cyclictest0-21swapper/309:05:163
2762199182,10cyclictest0-21swapper/309:05:153
2762199181,11cyclictest0-21swapper/307:35:223
2760599185,9cyclictest1720-21awk08:20:251
2759499186,9cyclictest18008-21cron11:45:010
2759499185,10cyclictest9776-21kthreadcore08:30:200
2759499185,10cyclictest25914-21kthreadcore07:40:210
2759499185,10cyclictest25914-21kthreadcore07:40:210
2759499182,13cyclictest32024-21cpu08:55:130
2759499182,13cyclictest24543-21fschecks_time12:10:150
2759499182,10cyclictest0-21swapper/008:15:130
2759499181,15cyclictest29847-21ssh12:35:170
2759499181,14cyclictest7540-21/usr/sbin/munin12:20:290
2759499181,13cyclictest6474-21apt-get07:55:100
2759499181,13cyclictest14472-21tr10:35:200
27594991810,6cyclictest9007-21cron09:05:010
27594991810,5cyclictest0-21swapper/008:10:140
2762199179,6cyclictest0-21swapper/311:16:293
2762199178,6cyclictest0-21swapper/311:50:083
2762199174,8cyclictest29566irq/58-eth009:35:153
2762199172,9cyclictest0-21swapper/308:15:133
2762199172,12cyclictest0-21swapper/312:05:223
2762199172,11cyclictest0-21swapper/312:12:383
2762199172,11cyclictest0-21swapper/307:15:123
2762199172,10cyclictest0-21swapper/308:30:133
2762199172,10cyclictest0-21swapper/308:25:133
2762199171,13cyclictest0-21swapper/309:55:243
2762199171,12cyclictest0-21swapper/307:15:023
2762199171,12cyclictest0-21swapper/307:15:013
2762199171,10cyclictest0-21swapper/307:40:183
2762199171,10cyclictest0-21swapper/307:40:173
2760599173,5cyclictest23026-21tr08:45:181
2759499179,5cyclictest5315-21kthreadcore09:25:190
2759499179,5cyclictest0-21swapper/008:05:270
2759499178,6cyclictest29830-21kthreadcore07:10:230
2759499178,6cyclictest29830-21kthreadcore07:10:220
2759499178,6cyclictest19400-21kthreadcore08:40:200
2759499177,7cyclictest13028-21kthreadcore07:25:210
2759499172,12cyclictest825-21kthreadcore08:20:220
2759499171,7cyclictest26922-21grep09:40:150
2759499171,7cyclictest26922-21grep09:40:150
2759499171,6cyclictest0-21swapper/007:30:180
2759499171,5cyclictest0-21swapper/011:20:240
2759499171,12cyclictest12040-21ssh09:30:130
2762199161,9cyclictest0-21swapper/307:20:153
27621991611,3cyclictest0-21swapper/312:20:133
2762199161,12cyclictest0-21swapper/308:55:183
2762199161,12cyclictest0-21swapper/308:05:273
2762199161,12cyclictest0-21swapper/307:55:143
2759499165,7cyclictest27548-21latency_hist07:10:020
2759499162,11cyclictest11647-21irqcore08:00:180
2762199151,9cyclictest0-21swapper/308:40:123
2762199151,12cyclictest0-21swapper/307:45:173
2762199151,11cyclictest0-21swapper/308:45:213
2762199151,11cyclictest0-21swapper/308:00:213
2760599141,9cyclictest27721-21latency_hist07:10:011
2762199133,7cyclictest0-21swapper/308:40:003
2762199111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional