You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 00:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Fri Jan 09, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1084499693,49cyclictest8254-21ssh10:30:120
10861995741,9cyclictest22346irq/40-dwc2_hso10:35:112
1084499575,8cyclictest131rcu_preempt12:05:110
10869995333,15cyclictest28366irq/58-eth009:50:123
1086199502,36cyclictest8248-21ssh10:30:122
1084499492,3cyclictest131rcu_preempt09:40:120
1086199481,24cyclictest924-21snmpd10:50:142
1086199471,24cyclictest582-21ssh09:30:132
1084499472,33cyclictest13952-21ssh12:25:140
1084499471,8cyclictest0-21swapper/008:55:120
10853994312,24cyclictest21-21ksoftirqd/111:05:131
1086199411,21cyclictest13958-21ssh12:25:142
1085399401,6cyclictest131rcu_preempt11:50:111
1086199392,16cyclictest11850-21ssh09:55:122
1086999378,24cyclictest28366irq/58-eth011:10:143
1084499371,32cyclictest11296-21cut10:50:190
1084499360,26cyclictest17023-21ssh10:35:120
1085399351,28cyclictest26736-21ssh11:55:131
1084499352,5cyclictest0-21swapper/012:17:550
10844993524,6cyclictest24280-21ssh09:25:140
1084499351,9cyclictest581-21kthreadcore11:20:210
1084499351,9cyclictest1643-21awk10:45:010
1084499351,9cyclictest1121-21cpuspeed08:40:130
1084499351,8cyclictest17835-21cat11:10:310
1084499351,11cyclictest5758-21df_inode10:10:160
1084499350,9cyclictest23775-21memory09:00:210
1086199341,24cyclictest0-21swapper/211:10:122
1084499345,6cyclictest687-21in:imuxsock07:30:000
1084499344,6cyclictest4467-21ssh11:05:120
1084499341,9cyclictest6719-21kthreadcore08:45:170
1084499341,9cyclictest28260-21fschecks_count11:00:150
1084499341,9cyclictest24704-21unixbench_multi10:00:280
1084499341,9cyclictest19594-21timerandwakeup07:15:280
1084499341,9cyclictest1322-21grep12:35:200
1084499341,9cyclictest12783-21fschecks_time07:45:140
1084499341,9cyclictest12690-21ssh09:17:040
1084499341,8cyclictest30586-21kthreadcore10:05:320
1084499341,8cyclictest25369-21diskstats09:05:140
1084499341,10cyclictest9767-21irqcore11:45:170
1084499340,9cyclictest23131-21/usr/sbin/munin10:20:270
1084499340,9cyclictest12863-21ls08:15:280
1084499340,10cyclictest18965-21cpuspeed08:25:140
1084499340,10cyclictest18965-21cpuspeed08:25:140
1086999334,17cyclictest28366irq/58-eth011:50:123
1086199339,19cyclictest27-21ksoftirqd/211:00:132
1085399333,25cyclictest22618-21rm12:30:131
1085399332,28cyclictest9280-21ssh11:45:141
1085399331,29cyclictest31228-21ssh12:35:121
1084499336,19cyclictest12-21ksoftirqd/011:40:140
1084499332,10cyclictest10058-21users12:20:320
1084499331,9cyclictest30679-21sed09:10:170
1084499331,9cyclictest30647-21fschecks_count07:30:140
1084499331,9cyclictest27943-21ntp_kernel_pll_11:35:240
1084499331,9cyclictest27943-21ntp_kernel_pll_11:35:230
1084499331,9cyclictest2024-21cut08:05:190
1084499331,9cyclictest1725-21kthreadcore10:25:200
1084499331,9cyclictest10577-21kthreadcore09:35:200
1084499331,8cyclictest8944-21latency_hist09:35:020
1084499331,8cyclictest24008-21ntpq12:10:250
1084499331,8cyclictest22007-21latency_hist07:55:010
1084499331,8cyclictest2045-21cpu10:45:140
1084499331,7cyclictest22440-21cut07:55:170
1084499331,6cyclictest22627-21sed11:15:180
1084499330,9cyclictest28982-21kthreadcore08:00:190
1084499330,9cyclictest28982-21kthreadcore08:00:190
1084499330,9cyclictest18326-21if_eth111:50:160
1084499330,8cyclictest9700-21ssh09:52:430
1084499330,8cyclictest3073-21fschecks_count07:35:150
1084499330,8cyclictest19627-21ssh10:15:580
1084499330,7cyclictest32135-21ntp_states08:35:230
1084499330,10cyclictest21377-21irqrtprio07:20:170
1086999322,18cyclictest28366irq/58-eth010:40:133
10861993214,13cyclictest27-21ksoftirqd/208:05:102
10853993226,3cyclictest531-21usb-storage07:20:111
10853993221,8cyclictest21-21ksoftirqd/108:05:101
1084499326,6cyclictest7430-21irqrtprio11:25:180
1084499326,4cyclictest28730-21ssh12:32:110
1084499321,9cyclictest26747-21cpu11:55:140
1084499321,8cyclictest24385-21kthreadcore08:30:180
1084499320,9cyclictest13780-21kthreadcore08:50:190
1085399315,3cyclictest131rcu_preempt11:25:141
1085399314,23cyclictest25766-21ssh10:40:121
1084499311,9cyclictest11961-21df_inode09:55:130
1084499311,8cyclictest6715-21ntp_kernel_err12:00:240
1084499311,8cyclictest14138-21df_inode08:20:140
1084499311,6cyclictest4965-21cut08:10:170
1084499311,10cyclictest27784-21kthreadcore09:45:180
10869993025,2cyclictest0-21swapper/309:40:123
1086199303,18cyclictest19302-21ssh10:55:132
1086199303,17cyclictest22346irq/40-dwc2_hso08:00:132
1086199303,17cyclictest22346irq/40-dwc2_hso08:00:122
1086199302,21cyclictest0-21swapper/211:20:132
10861993014,12cyclictest0-21swapper/210:20:132
1086199301,21cyclictest924-21snmpd09:40:132
10853993022,5cyclictest21-21ksoftirqd/107:35:111
1085399301,26cyclictest23157-21ssh10:20:111
1084499302,24cyclictest15488-21ssh09:20:140
1084499300,9cyclictest15571-21/usr/sbin/munin11:30:110
1084499300,8cyclictest11260-21ntp_states07:40:240
1086999292,17cyclictest28366irq/58-eth011:55:143
1086199292,8cyclictest5197-21ssh12:20:132
1086199292,20cyclictest13163-21ssh10:13:552
1086199292,20cyclictest0-21swapper/207:55:142
1086199292,18cyclictest22346irq/40-dwc2_hso08:25:132
1086199292,18cyclictest22346irq/40-dwc2_hso08:25:132
10861992918,5cyclictest0-21swapper/207:45:112
1086199291,17cyclictest3092-21ssh12:00:132
1086199291,16cyclictest6880-21ssh11:25:142
1085399292,7cyclictest0-21swapper/110:50:141
1085399292,22cyclictest11866-21ssh09:55:121
1084499290,9cyclictest11146-21fschecks_count07:10:150
1084499290,8cyclictest26386-21ssh10:58:200
1086999284,17cyclictest28366irq/58-eth009:35:133
1086999283,16cyclictest28366irq/58-eth011:45:123
1086999281,16cyclictest28366irq/58-eth011:05:133
1086199282,21cyclictest26012-21ssh11:54:372
1085399283,20cyclictest21-21ksoftirqd/108:25:131
1085399283,20cyclictest21-21ksoftirqd/108:25:131
10853992814,8cyclictest531-21usb-storage09:00:111
1086999273,16cyclictest0-21swapper/312:09:373
10869992710,8cyclictest28366irq/58-eth010:15:113
1086199274,9cyclictest27-21ksoftirqd/209:20:222
1086199274,13cyclictest22346irq/40-dwc2_hso08:55:132
1086199273,9cyclictest0-21swapper/210:09:552
1086199272,18cyclictest0-21swapper/208:45:132
1086199272,17cyclictest22346irq/40-dwc2_hso08:40:122
1086999264,11cyclictest0-21swapper/310:28:553
1086999262,17cyclictest28366irq/58-eth009:20:133
1086999262,17cyclictest0-21swapper/310:24:323
1086999262,15cyclictest28466irq/59-eth012:35:123
1086199267,16cyclictest0-21swapper/212:30:142
1086199263,15cyclictest22346irq/40-dwc2_hso07:40:122
1086199262,9cyclictest3245-21ssh12:17:562
1086199262,10cyclictest0-21swapper/210:15:132
10861992612,9cyclictest0-21swapper/208:20:132
1086199261,14cyclictest17668-21ls07:50:152
1086199261,14cyclictest11795-21ssh12:05:122
1085399262,18cyclictest19695-21ssh12:09:371
10853992615,9cyclictest633-21Async12:11:531
1086999252,17cyclictest0-21swapper/310:30:133
10869992516,6cyclictest28366irq/58-eth010:55:133
1086199252,15cyclictest22346irq/40-dwc2_hso08:50:102
10861992513,10cyclictest3539-21unixbench_multi10:25:282
10861992510,10cyclictest0-21swapper/209:35:132
1085399255,13cyclictest21-21ksoftirqd/108:50:121
1085399251,9cyclictest2644-21users09:10:311
10853992517,5cyclictest0-21swapper/111:30:131
1086999244,12cyclictest0-21swapper/310:00:123
1086199246,4cyclictest27-21ksoftirqd/211:57:122
1086199242,8cyclictest27-21ksoftirqd/211:49:382
1086199242,6cyclictest0-21swapper/211:33:552
1086199242,15cyclictest0-21swapper/208:35:042
1085399247,14cyclictest21-21ksoftirqd/110:15:121
1085399246,10cyclictest10834-21cyclictest08:15:131
1085399242,19cyclictest7006-21ssh10:28:551
10853992415,6cyclictest13165-21ssh11:10:131
1085399241,17cyclictest6675-21ssh09:15:141
1086999235,14cyclictest0-21swapper/312:25:143
1086999235,10cyclictest28366irq/58-eth010:05:113
10869992313,7cyclictest0-21swapper/310:45:273
10869992311,9cyclictest28366irq/58-eth011:15:143
1086199238,9cyclictest131rcu_preempt11:38:222
1086199238,9cyclictest131rcu_preempt11:38:222
1086199237,6cyclictest131rcu_preempt11:16:132
1086199234,3cyclictest131rcu_preempt09:12:332
1086199232,16cyclictest0-21swapper/212:39:382
1086199232,15cyclictest0-21swapper/207:30:052
1086199231,8cyclictest26733-21/usr/sbin/munin09:45:252
10861992312,8cyclictest0-21swapper/210:40:412
10861992311,9cyclictest14985-21cron09:20:012
1086199231,13cyclictest2023-21ssh10:45:132
1086199231,12cyclictest21199-21irqrtprio10:00:162
1085399232,7cyclictest3687-21irqcore09:50:191
1085399231,15cyclictest24313-21ssh11:35:101
1085399231,15cyclictest24313-21ssh11:35:101
1086999223,14cyclictest28366irq/58-eth007:15:113
1086999223,14cyclictest0-21swapper/311:30:133
1086999223,11cyclictest0-21swapper/311:35:103
1086999223,11cyclictest0-21swapper/311:35:103
1086999222,15cyclictest28366irq/58-eth010:50:133
1086999222,15cyclictest0-21swapper/311:25:153
1086199227,6cyclictest27-21ksoftirqd/209:50:272
1086199222,6cyclictest0-21swapper/209:25:152
1086199222,17cyclictest22346irq/40-dwc2_hso09:05:122
1086199221,4cyclictest25724-21cut07:25:162
10861992211,8cyclictest0-21swapper/211:05:132
1085399222,16cyclictest5659-21ssh10:10:141
10853992214,6cyclictest20568-21ssh10:00:111
1085399221,17cyclictest9220-21ssh09:35:111
1086999213,15cyclictest28366irq/58-eth007:55:133
1086999212,12cyclictest0-21swapper/312:30:133
10869992116,2cyclictest0-21swapper/309:05:133
1086199212,13cyclictest0-21swapper/208:10:122
10861992112,7cyclictest0-21swapper/212:10:312
10861992112,7cyclictest0-21swapper/208:30:232
10861992111,8cyclictest0-21swapper/211:40:312
1086199211,16cyclictest9309-21grep08:15:152
1085399212,7cyclictest16585-21munin-run10:35:011
1085399212,14cyclictest18781-21irqstats09:40:181
1085399211,5cyclictest13935-21apt-get12:25:131
10853992112,6cyclictest6582-21ssh12:00:221
1085399211,15cyclictest4375-21apt-get08:10:111
1085399211,15cyclictest28524-21latency09:05:211
1086999203,14cyclictest28366irq/58-eth009:25:153
1086999202,2cyclictest0-21swapper/312:20:133
1086999201,15cyclictest0-21swapper/307:25:143
1086999201,14cyclictest28366irq/58-eth010:10:123
1086999201,13cyclictest0-21swapper/307:30:263
1086199205,12cyclictest14590-21cat07:10:262
1086199203,13cyclictest10964-21latency_hist07:10:022
1086199202,15cyclictest3298-21irqcore07:35:162
1086199202,14cyclictest25529-21latency_hist07:25:022
1086199201,10cyclictest21693-21kthreadcore09:00:182
1085399208,9cyclictest11090-21df_inode07:10:141
1085399205,7cyclictest6220-21irqcore08:45:161
1085399203,15cyclictest21-21ksoftirqd/112:10:141
1085399201,7cyclictest32678-21awk09:30:011
1085399201,7cyclictest26890-21diskstats09:45:131
10853992012,5cyclictest23925-21kthreadcore11:15:211
10853992011,7cyclictest19282-21apt-get10:55:121
1085399201,10cyclictest29860-21kthreadcore08:00:211
1085399201,10cyclictest29860-21kthreadcore08:00:201
10853992010,7cyclictest5764-21processes10:45:271
1085399200,14cyclictest17749-21grep07:50:161
1086999195,12cyclictest28366irq/58-eth008:35:093
1086999193,11cyclictest0-21swapper/311:24:353
1086999193,11cyclictest0-21swapper/309:55:123
1086999192,10cyclictest0-21swapper/307:50:133
1086999192,10cyclictest0-21swapper/307:10:133
10869991913,3cyclictest0-21swapper/311:00:203
1086999191,13cyclictest0-21swapper/308:15:173
1086199191,9cyclictest16035-21hwlatdetect07:15:162
1085399198,8cyclictest15572-21df09:20:141
10853991912,4cyclictest17765-21kthreadcore10:35:181
10853991911,6cyclictest2174-21ssh11:20:281
1085399191,15cyclictest29370-21tr11:00:191
10853991911,5cyclictest0-21swapper/107:15:121
1086999189,7cyclictest28366irq/58-eth012:15:123
1086999183,10cyclictest0-21swapper/312:00:133
1086999183,10cyclictest0-21swapper/311:40:133
1086999182,7cyclictest0-21swapper/309:45:143
1086999182,10cyclictest0-21swapper/309:00:133
1086999182,10cyclictest0-21swapper/308:50:123
1086999182,10cyclictest0-21swapper/308:25:153
1086999182,10cyclictest0-21swapper/308:25:143
1086999182,10cyclictest0-21swapper/308:05:123
1086999181,9cyclictest0-21swapper/308:20:113
1086999181,13cyclictest0-21swapper/309:30:163
1086999181,13cyclictest0-21swapper/308:45:163
1086999181,11cyclictest0-21swapper/309:15:153
1086999181,11cyclictest0-21swapper/307:20:133
10869991810,6cyclictest0-21swapper/312:10:253
10869991810,6cyclictest0-21swapper/310:39:403
10869991810,6cyclictest0-21swapper/309:12:163
1085399184,11cyclictest5842-21sed12:20:181
1085399182,13cyclictest3213-21kthreadcore08:40:201
1085399181,8cyclictest26842-21cron08:00:011
1085399181,6cyclictest544-21/usr/sbin/munin09:30:181
1085399181,13cyclictest4254-21ssh11:40:261
1085399181,13cyclictest29941-21sed10:05:301
1085399180,10cyclictest24690-21cut08:30:181
1086999172,9cyclictest0-21swapper/308:55:133
1086999172,10cyclictest0-21swapper/308:30:123
1086999172,10cyclictest0-21swapper/308:00:133
1086999172,10cyclictest0-21swapper/308:00:133
1086999172,10cyclictest0-21swapper/307:45:133
1086999171,10cyclictest0-21swapper/308:10:263
1085399173,11cyclictest31290-21kthreadcore08:35:181
1085399172,10cyclictest0-21swapper/107:25:141
1085399171,4cyclictest3224146kthreadcore07:30:191
1085399171,12cyclictest7966-21grep07:40:141
1085399164,9cyclictest14914-21kthreadcore07:45:191
1085399161,11cyclictest0-21swapper/108:20:131
10853991610,4cyclictest15626-21sh08:55:141
1085399151,12cyclictest10950-21cat07:10:021
1084499142,9cyclictest10914-21latency_hist07:10:020
1086999131,5cyclictest0-21swapper/308:40:133
1086999122,8cyclictest0-21swapper/307:40:123
1086999122,6cyclictest0-21swapper/307:35:143
1086999111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional