You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 18:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Nov 24, 2025 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18471998251,21cyclictest29566irq/58-eth011:50:133
1846399553,27cyclictest9377-21ssh09:35:142
1845499535,33cyclictest15013-21ssh12:10:151
1844699531,37cyclictest21257-21ssh12:35:140
1844699524,6cyclictest0-21swapper/010:15:100
1845499512,38cyclictest23497-21ssh10:50:131
1846399506,14cyclictest27-21ksoftirqd/212:00:122
1846399502,31cyclictest10715-21ssh10:20:132
1844699482,3cyclictest25790-21ssh10:30:150
1844699470,5cyclictest6240-21apt-get11:00:130
1844699461,5cyclictest0-21swapper/007:30:130
1844699461,5cyclictest0-21swapper/007:30:130
1844699453,35cyclictest863-21ssh10:35:150
1844699451,5cyclictest0-21swapper/012:20:120
1846399431,26cyclictest6244-21ssh11:00:142
1845499432,7cyclictest131rcu_preempt11:20:121
1844699431,6cyclictest11048-21apt-get11:25:130
1844699431,5cyclictest0-21swapper/008:40:150
1847199422,21cyclictest29566irq/58-eth010:20:133
1844699418,6cyclictest131rcu_preempt10:00:150
1844699411,5cyclictest0-21swapper/008:25:140
1844699411,5cyclictest0-21swapper/008:25:140
1844699411,4cyclictest0-21swapper/008:35:130
1844699411,4cyclictest0-21swapper/007:20:110
1846399404,28cyclictest15843-21ssh10:45:142
1845499401,32cyclictest19677-21ssh09:20:131
1844699401,7cyclictest0-21swapper/011:35:130
1844699401,32cyclictest12460-21ssh09:15:130
1844699401,30cyclictest15004-21ssh12:10:140
1844699400,5cyclictest20789-21apt-get10:05:140
1847199392,23cyclictest29566irq/58-eth007:10:133
18463993931,4cyclictest4760-21diskmemload09:25:152
18471993810,18cyclictest29566irq/58-eth008:20:143
1846399382,28cyclictest15711-21cat11:05:212
1847199373,19cyclictest29566irq/58-eth011:40:133
1844699372,4cyclictest131rcu_preempt12:00:110
1844699372,25cyclictest16274-21sh11:50:130
1844699371,9cyclictest11858-21ntp_192.168.11410:40:250
1845499361,28cyclictest31635-21ssh12:00:121
1844699362,3cyclictest0-21swapper/009:45:150
1844699361,9cyclictest19659-21cut08:15:220
1844699361,10cyclictest30803-21cat09:25:300
18463993523,8cyclictest261rcuc/208:55:122
1845499353,29cyclictest13471-21ssh12:30:131
1845499352,30cyclictest1638-21ssh11:40:131
1844699356,25cyclictest23918-21ssh11:55:130
1844699351,4cyclictest0-21swapper/008:00:140
1844699351,3cyclictest0-21swapper/007:50:130
1847199343,18cyclictest29566irq/58-eth011:05:133
1845499343,23cyclictest14138-21ssh11:05:161
1845499341,27cyclictest26729-21ssh11:35:121
1844699341,9cyclictest3571-21tr11:40:210
1844699341,7cyclictest20076-21diskstats07:45:150
1844699341,4cyclictest0-21swapper/011:05:150
1844699341,4cyclictest0-21swapper/009:10:120
1844699341,3cyclictest0-21swapper/008:05:140
1844699341,30cyclictest21807-21grep10:25:280
1844699340,8cyclictest16870-21cat12:30:210
1846399335,16cyclictest26346irq/40-dwc2_hso10:05:142
18454993321,8cyclictest21-21ksoftirqd/111:45:131
18454993320,9cyclictest0-21swapper/111:00:141
1844699331,9cyclictest27343-21ls10:50:270
1844699331,8cyclictest8993-21tr12:25:230
1844699331,7cyclictest18488-21load07:40:220
1844699331,7cyclictest10711-21apt-get10:20:120
1844699331,6cyclictest32763-21kthreadcore07:55:220
1844699331,6cyclictest16670-21ntp_states08:45:270
1844699331,4cyclictest21578-21ntp_states08:50:240
1844699331,4cyclictest0-21swapper/009:00:120
1844699331,28cyclictest8990-21sed11:45:190
1844699331,10cyclictest13711-21ntp_192.168.11407:35:240
1844699330,8cyclictest2229-21kthreadcore07:25:210
1844699330,6cyclictest16642-21apt-get09:40:120
1846399325,13cyclictest628-21Async14:00:482
1844699326,23cyclictest22275-21irqrtprio11:10:210
1844699326,23cyclictest22275-21irqrtprio11:10:210
1844699321,9cyclictest9261-21ssh11:22:180
1844699321,9cyclictest8097-21tr12:05:200
1844699321,9cyclictest31492-21/usr/sbin/munin09:50:180
1844699321,9cyclictest16554-21kernelversion10:45:200
1844699321,8cyclictest10095-21kthreadcore09:35:210
1844699321,7cyclictest3594-21ntp_192.168.11409:05:260
1844699321,4cyclictest23336-21ntp_states09:20:270
1844699321,4cyclictest11844-21df_inode08:10:140
1844699320,7cyclictest31359-21fw_packets08:30:160
1847199312,17cyclictest29566irq/58-eth011:10:153
1847199312,17cyclictest29566irq/58-eth011:10:143
1846399313,21cyclictest0-21swapper/208:20:152
1846399312,22cyclictest4760-21diskmemload09:34:212
1845499311,21cyclictest19194-1kworker/u9:109:50:131
1844699318,21cyclictest26404-21ntp_states08:55:280
1844699312,4cyclictest0-21swapper/007:15:130
1844699311,8cyclictest31218-21ssh10:55:160
1844699311,8cyclictest30954-21kthreadcore10:10:240
1844699311,8cyclictest28426-21/usr/sbin/munin11:15:250
1844699311,7cyclictest8992-21tr09:55:220
1844699311,7cyclictest26454-21ntp_192.168.11412:15:240
1844699311,7cyclictest21187-21kthreadcore07:10:210
1844699311,6cyclictest20008-21kthreadcore11:30:200
1846399305,12cyclictest26346irq/40-dwc2_hso07:50:132
1846399303,18cyclictest26346irq/40-dwc2_hso08:00:122
18463993013,11cyclictest0-21swapper/211:45:142
18454993017,7cyclictest21-21ksoftirqd/110:10:151
1844699300,8cyclictest2338-21sed09:30:160
1844699300,6cyclictest24924-21ntpq08:20:260
1847199293,17cyclictest29566irq/58-eth011:25:153
1847199293,15cyclictest0-21swapper/310:00:143
1845499292,19cyclictest18931-21ssh11:30:131
18454992920,6cyclictest14337-21kworker/1:111:50:131
1846399284,14cyclictest26346irq/40-dwc2_hso07:55:142
1846399282,19cyclictest0-21swapper/208:25:152
1846399282,19cyclictest0-21swapper/208:25:152
18463992816,9cyclictest28455-21ssh11:15:122
1846399281,17cyclictest932-21snmpd07:30:122
1846399281,17cyclictest932-21snmpd07:30:112
1847199275,15cyclictest29566irq/58-eth012:20:113
1847199273,20cyclictest29566irq/58-eth010:45:133
1847199273,17cyclictest29566irq/58-eth012:35:143
1846399275,15cyclictest27-21ksoftirqd/208:50:172
1846399274,15cyclictest26346irq/40-dwc2_hso08:15:132
1846399273,17cyclictest22867-21ssh12:15:112
1846399272,19cyclictest0-21swapper/208:05:152
18463992714,4cyclictest0-21swapper/209:45:162
1846399271,20cyclictest932-21snmpd08:10:132
1846399271,15cyclictest8013-21ls10:40:022
1845499273,16cyclictest851-21ssh10:35:131
1845499272,20cyclictest7090-21ssh12:05:141
1845499272,19cyclictest20805-21ssh10:05:151
18454992714,5cyclictest23925-21ssh11:55:141
1844699271,7cyclictest18460-21latency_hist07:10:010
1847199264,17cyclictest0-21swapper/310:25:143
1846399264,15cyclictest0-21swapper/210:00:152
1846399260,19cyclictest13456-21ssh12:30:122
1847199252,19cyclictest29566irq/58-eth012:25:133
1847199252,17cyclictest0-21swapper/310:16:023
18471992515,6cyclictest29566irq/58-eth009:35:153
1846399259,6cyclictest131rcu_preempt12:24:462
1846399258,5cyclictest27-21ksoftirqd/209:15:152
1846399252,9cyclictest0-21swapper/208:40:162
1846399251,18cyclictest20185-21timerandwakeup11:50:292
1845499255,15cyclictest21-21ksoftirqd/111:25:141
1845499252,16cyclictest4760-21diskmemload09:55:131
1847199243,13cyclictest29566irq/58-eth011:00:143
1847199242,11cyclictest0-21swapper/311:45:143
18471992412,7cyclictest0-21swapper/310:05:153
1846399243,14cyclictest27-21ksoftirqd/212:11:462
1846399243,13cyclictest26346irq/40-dwc2_hso08:30:132
1846399242,19cyclictest31188-21apt-get09:50:142
1846399242,15cyclictest26346irq/40-dwc2_hso11:20:132
18463992415,4cyclictest23613-21sh10:50:162
18463992413,9cyclictest10521-21ssh09:11:432
18463992412,9cyclictest0-21swapper/207:15:182
18463992411,9cyclictest13418-21sshd07:35:222
1845499241,17cyclictest30832-21ssh12:20:141
18471992312,6cyclictest29566irq/58-eth009:40:153
1847199230,17cyclictest29566irq/58-eth010:55:133
1846399238,9cyclictest131rcu_preempt10:15:442
1846399233,14cyclictest26346irq/40-dwc2_hso08:45:152
1846399232,6cyclictest0-21swapper/207:40:142
18463992311,9cyclictest10977-21ssh09:55:482
1846399231,18cyclictest21727-21apt-get11:10:142
1846399231,18cyclictest21727-21apt-get11:10:142
1846399231,15cyclictest26346irq/40-dwc2_hso09:40:142
1846399231,15cyclictest23866-21kworker/u8:111:35:132
1846399231,13cyclictest23172-21idleruntime-cro07:15:012
1845499238,10cyclictest0-21swapper/107:25:131
1845499232,15cyclictest8697-21ssh10:16:211
18454992315,6cyclictest8147-21cron08:40:001
18454992313,7cyclictest21-21ksoftirqd/109:45:151
1845499231,18cyclictest12958-21ssh09:35:251
1847199226,7cyclictest0-21swapper/309:25:153
1847199221,14cyclictest0-21swapper/310:35:153
1846399225,9cyclictest131rcu_preempt12:38:532
1846399225,9cyclictest131rcu_preempt11:26:132
1846399221,14cyclictest26573-21ls11:35:022
18463992210,9cyclictest24543-21sshd07:45:352
1846399220,16cyclictest4753-21cron09:10:012
1845499228,9cyclictest0-21swapper/108:20:131
1845499223,12cyclictest0-21swapper/110:25:131
18454992213,6cyclictest5112-21ssh09:10:141
18454992210,9cyclictest15183-21sshd10:00:211
1847199213,12cyclictest29566irq/58-eth011:30:123
1847199212,11cyclictest29566irq/58-eth011:55:143
18471992113,5cyclictest29566irq/58-eth012:06:353
18471992113,5cyclictest0-21swapper/310:30:143
18471992111,6cyclictest0-21swapper/311:15:123
1846399217,2cyclictest131rcu_preempt12:26:202
1846399216,6cyclictest27-21ksoftirqd/208:35:132
1846399213,14cyclictest4020-21kthreadcore11:40:222
1846399212,10cyclictest19150-21kthreadcore10:25:202
1846399211,17cyclictest0-21swapper/212:05:172
1846399211,11cyclictest365-21cron07:25:012
1846399211,10cyclictest19838-21fschecks_count09:20:162
1846399211,10cyclictest1695-21kthreadcore07:25:192
1846399210,18cyclictest29263-21missed_timers10:30:252
1845499213,10cyclictest0-21swapper/107:10:131
1845499212,12cyclictest21715-21ssh11:10:141
1845499212,12cyclictest21715-21ssh11:10:141
18454992112,6cyclictest8142-21ssh10:40:051
1845499211,16cyclictest31888-21ls07:20:251
1845499211,16cyclictest1870-21ssh09:30:131
1847199209,9cyclictest0-21swapper/312:00:303
1847199209,7cyclictest0-21swapper/312:13:523
1847199201,14cyclictest0-21swapper/312:30:133
1846399206,11cyclictest24291-21ssh11:55:192
1846399201,15cyclictest8239-21ssh10:40:142
1846399201,15cyclictest31889-21ntpq10:10:282
1845499202,14cyclictest24877-21apt-get07:50:121
18454992012,6cyclictest15908-21sshd10:21:211
18454992011,6cyclictest4764-21idleruntime-cro09:10:011
1845499201,10cyclictest600-21missed_timers07:55:231
1847199198,6cyclictest0-21swapper/311:39:523
1847199194,12cyclictest29566irq/58-eth009:10:143
1847199192,13cyclictest0-21swapper/310:00:023
1847199192,12cyclictest0-21swapper/311:20:123
1847199192,11cyclictest0-21swapper/310:40:153
1847199192,11cyclictest0-21swapper/309:30:133
1847199192,10cyclictest0-21swapper/308:50:033
1847199191,13cyclictest0-21swapper/309:20:403
1847199191,13cyclictest0-21swapper/308:05:173
18471991910,6cyclictest0-21swapper/310:53:083
1846399195,9cyclictest0-21swapper/209:00:122
1845499198,8cyclictest488-21ssh10:55:211
1845499193,12cyclictest16663-21ssh09:40:151
18454991912,4cyclictest19707-21threads10:45:291
1845499190,14cyclictest31393-21ls08:30:171
1847199182,10cyclictest0-21swapper/312:15:133
1847199182,10cyclictest0-21swapper/309:05:153
1847199182,10cyclictest0-21swapper/308:00:143
1847199182,10cyclictest0-21swapper/307:55:143
1847199181,12cyclictest0-21swapper/309:20:023
1847199181,12cyclictest0-21swapper/307:35:163
18471991810,6cyclictest0-21swapper/310:13:373
18471991810,5cyclictest29566irq/58-eth007:15:133
1846399181,9cyclictest18493-21latency_hist07:10:012
1845499188,7cyclictest19028-21cut08:15:211
1845499182,13cyclictest10284-21kthreadcore08:40:231
1845499181,9cyclictest24998-21rm12:35:291
1845499181,15cyclictest22860-21apt-get12:15:111
18454991811,4cyclictest27334-21kthreadcore10:30:221
1845499181,14cyclictest14989-21latency_hist07:40:011
1845499181,13cyclictest26317-21apt-get08:25:131
1845499181,13cyclictest26317-21apt-get08:25:121
18454991810,5cyclictest29170-21kthreadcore11:15:201
18454991810,5cyclictest12478-21sh09:15:141
18454991810,5cyclictest0-21swapper/109:00:171
1845499180,4cyclictest53046usb-storage08:50:041
1847199176,7cyclictest29566irq/58-eth009:45:163
1847199173,9cyclictest0-21swapper/309:50:133
1847199173,9cyclictest0-21swapper/308:25:153
1847199173,9cyclictest0-21swapper/308:25:153
1847199172,10cyclictest0-21swapper/308:35:143
1847199172,10cyclictest0-21swapper/307:50:143
1847199171,13cyclictest0-21swapper/308:15:163
1845499172,12cyclictest5576-21ntp_kernel_pll_08:00:241
1845499171,9cyclictest15247-21cut07:40:161
1845499171,13cyclictest29255-21cut09:25:231
1845499171,12cyclictest5945-21ssh12:25:141
18454991710,4cyclictest11848-21sh08:10:141
1847199169,5cyclictest0-21swapper/308:45:273
1847199162,9cyclictest0-21swapper/308:10:133
1847199162,10cyclictest0-21swapper/308:30:143
1847199161,9cyclictest0-21swapper/307:20:153
1845499168,6cyclictest19983-21apt-get07:45:121
1845499162,8cyclictest0-21swapper/107:15:131
1845499162,11cyclictest9373-21unixbench_singl07:30:291
1845499162,11cyclictest9373-21unixbench_singl07:30:291
1845499162,11cyclictest15077-21sort08:45:221
1845499161,11cyclictest8960-21kthreadcore08:05:221
1845499160,13cyclictest24021-21kthreadcore08:55:191
1847199152,9cyclictest0-21swapper/307:50:013
1847199149,3cyclictest0-21swapper/307:40:273
1847199142,9cyclictest0-21swapper/308:40:163
1845499142,10cyclictest18434-21cyclictest07:10:011
1847199131,9cyclictest0-21swapper/307:30:013
1847199131,10cyclictest0-21swapper/307:30:213
1847199131,10cyclictest0-21swapper/307:30:213
1847199123,7cyclictest0-21swapper/308:55:223
1847199121,8cyclictest0-21swapper/309:00:113
1847199101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional