You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 14:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Nov 19, 2025 12:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
180399668,10cyclictest12-21ksoftirqd/009:25:120
180399621,7cyclictest131rcu_preempt09:55:130
180399593,9cyclictest12-21ksoftirqd/010:50:140
180499543,42cyclictest21626-21ssh11:00:151
1816995324,5cyclictest27-21ksoftirqd/212:15:152
1803995312,9cyclictest12-21ksoftirqd/010:40:150
180499501,7cyclictest131rcu_preempt07:10:121
181699473,34cyclictest1237-21ssh11:30:132
1804994619,17cyclictest21-21ksoftirqd/111:40:131
180499451,31cyclictest29052-21ssh10:00:131
180399451,5cyclictest131rcu_preempt10:45:130
182599422,25cyclictest29566irq/58-eth009:55:143
1804994117,18cyclictest21-21ksoftirqd/111:15:161
182599403,23cyclictest29566irq/58-eth009:25:133
180399381,31cyclictest18695-21rm12:25:160
180499362,26cyclictest7112-21ssh09:45:151
1825993421,9cyclictest29566irq/58-eth012:20:153
182599342,19cyclictest29566irq/58-eth012:05:133
180499332,23cyclictest28508-21ssh09:15:151
1804993319,9cyclictest0-21swapper/111:20:171
1804993312,16cyclictest21-21ksoftirqd/112:35:161
182599322,18cyclictest29566irq/58-eth010:40:143
181699321,11cyclictest89266lldpd10:00:152
180399321,19cyclictest4007-21ssh10:05:150
181699314,15cyclictest26346irq/40-dwc2_hso07:50:132
180399312,21cyclictest991-21ssh10:25:170
180499305,15cyclictest14427-21ssh09:50:151
1825992916,7cyclictest29566irq/58-eth009:10:143
181699294,17cyclictest26346irq/40-dwc2_hso11:50:172
181699293,17cyclictest26346irq/40-dwc2_hso07:40:132
181699292,18cyclictest26346irq/40-dwc2_hso08:25:142
180499295,15cyclictest18479-21ssh10:15:141
182599282,13cyclictest29566irq/58-eth012:25:153
181699283,18cyclictest26346irq/40-dwc2_hso08:45:132
181699282,19cyclictest2864-21ssh09:20:162
180499282,19cyclictest22997-21ssh11:45:171
180499282,19cyclictest22997-21ssh11:45:171
1803992816,7cyclictest0-21swapper/008:45:130
180399281,3cyclictest131rcu_preempt11:45:150
180399281,3cyclictest131rcu_preempt11:45:140
182599276,17cyclictest29566irq/58-eth008:40:143
181699273,15cyclictest26346irq/40-dwc2_hso11:35:142
181699272,19cyclictest0-21swapper/212:10:152
181699272,17cyclictest26346irq/40-dwc2_hso09:35:152
181699271,16cyclictest26346irq/40-dwc2_hso10:10:152
180499272,19cyclictest11090-21diskmemload12:20:171
180399273,17cyclictest12-21ksoftirqd/010:10:140
181699261,16cyclictest21042-21ls09:30:302
180499269,10cyclictest201rcuc/108:05:131
180499263,3cyclictest131rcu_preempt10:25:181
1804992618,4cyclictest201rcuc/109:25:151
180399261,18cyclictest24647-21ssh09:35:150
181699252,13cyclictest26346irq/40-dwc2_hso12:05:142
181699251,8cyclictest969266ssh10:07:322
181699251,11cyclictest631066ssh10:26:372
182599243,14cyclictest29566irq/58-eth009:35:153
182599241,13cyclictest29566irq/58-eth010:25:183
181699243,5cyclictest23463-21ssh10:40:142
181699241,6cyclictest2100066rm09:54:012
1816992416,6cyclictest18420-21ssh12:25:012
1816992416,5cyclictest5844-21ssh11:55:182
181699241,14cyclictest18702-21ssh12:25:162
181699241,11cyclictest1991166ssh12:03:462
180499242,16cyclictest21-21ksoftirqd/111:25:151
182599234,15cyclictest29566irq/58-eth011:45:163
182599234,15cyclictest29566irq/58-eth011:45:153
182599231,12cyclictest0-21swapper/312:35:153
1825992310,10cyclictest0-21swapper/310:30:213
181699236,12cyclictest32063-21sh09:40:142
181699233,15cyclictest26346irq/40-dwc2_hso08:35:132
181699231,7cyclictest2849266ssh09:15:132
1816992311,5cyclictest7555-21ssh12:38:342
180499238,2cyclictest21-21ksoftirqd/112:00:371
180499237,2cyclictest21-21ksoftirqd/112:32:041
1804992314,4cyclictest4006-21sh10:05:161
1804992311,3cyclictest21-21ksoftirqd/109:58:271
1803992314,6cyclictest25274-21ssh11:00:290
1803992314,6cyclictest1293-21ssh11:30:150
180399231,18cyclictest14464-21apt-get08:30:150
180399231,15cyclictest26175-21ssh11:25:130
182599224,13cyclictest0-21swapper/311:55:163
182599223,11cyclictest29566irq/58-eth009:45:153
182599222,15cyclictest29566irq/58-eth010:05:163
182599222,13cyclictest0-21swapper/310:50:163
181699222,7cyclictest0-21swapper/210:45:122
181699221,6cyclictest2861766ssh11:47:102
181699221,6cyclictest2861766ssh11:47:102
181699221,6cyclictest2103166ssh11:41:472
181699221,6cyclictest1475266ssh10:34:042
181699221,6cyclictest10785-21usage_EthcTask10:50:332
181699221,5cyclictest362566ssh11:09:182
181699221,16cyclictest11726-21apt-get11:15:152
181699221,13cyclictest2607466ssh11:25:032
1804992210,5cyclictest0-21swapper/107:15:191
180399224,5cyclictest12-21ksoftirqd/008:30:010
180399221,17cyclictest3393-21apt-get07:45:130
180399221,15cyclictest842-21munin-node08:35:150
1803992211,3cyclictest12-21ksoftirqd/010:35:170
182599218,9cyclictest0-21swapper/311:40:183
182599213,10cyclictest29566irq/58-eth010:00:143
182599212,13cyclictest0-21swapper/310:15:153
182599211,12cyclictest0-21swapper/311:00:163
1825992110,7cyclictest29566irq/58-eth011:15:163
181699219,5cyclictest5871-21cron08:55:012
181699218,10cyclictest4500-21munin-run08:20:002
181699217,11cyclictest21416-21df07:30:172
181699217,11cyclictest21416-21df07:30:162
181699215,8cyclictest15109-21kthreadcore10:55:202
181699214,9cyclictest12173-21ssh09:25:222
181699213,9cyclictest29408-21kthreadcore10:20:242
181699211,6cyclictest2792466ssh11:03:202
181699211,6cyclictest2189366ssh10:38:092
181699211,6cyclictest1060866ssh11:13:072
181699211,6cyclictest1060866ssh11:13:072
181699211,5cyclictest3232466ssh12:33:192
1816992111,8cyclictest0-21swapper/210:15:152
181699211,17cyclictest27161-21kthreadcore08:40:252
181699211,16cyclictest10954-21apt-get09:00:162
181699210,15cyclictest7983-21tr08:55:242
180499219,3cyclictest21-21ksoftirqd/110:20:151
180499218,7cyclictest21-21ksoftirqd/109:11:231
180499212,14cyclictest19251-21ntp_192.168.11410:35:291
1804992115,4cyclictest0-21swapper/107:50:181
180399216,3cyclictest131rcu_preempt10:55:190
180399214,5cyclictest662-21ntpd09:17:110
180399213,12cyclictest842-21munin-node09:20:160
180399211,17cyclictest26146-21df10:20:160
180399211,14cyclictest5723-21ssh11:55:160
182599209,8cyclictest29566irq/58-eth011:05:173
182599201,13cyclictest0-21swapper/309:45:023
181699208,10cyclictest22535-21cat08:05:022
181699202,9cyclictest16815-21kthreadcore09:05:192
181699202,15cyclictest14968-21kthreadcore07:55:212
181699202,10cyclictest5776-21timerwakeupswit07:10:292
181699201,9cyclictest28780-21kthreadcore08:10:212
181699201,9cyclictest21031-21fschecks_time09:10:182
181699201,9cyclictest18918-21cpu11:20:172
181699201,9cyclictest18206-21kthreadcore07:25:222
181699201,9cyclictest10551-21wc07:15:372
181699201,16cyclictest91-21pr/ttyS008:35:022
181699201,15cyclictest25219-21kthreadcore08:05:212
181699201,11cyclictest3385-21acpi07:45:122
1816992010,8cyclictest0-21swapper/209:56:122
180499209,9cyclictest0-21swapper/111:57:121
180499209,3cyclictest21-21ksoftirqd/110:40:131
180499209,2cyclictest21-21ksoftirqd/108:25:291
180499209,2cyclictest21-21ksoftirqd/108:00:261
180499207,4cyclictest21-21ksoftirqd/110:52:491
180499202,15cyclictest13999-21kthreadcore07:20:221
180499202,13cyclictest31018-21ssh10:45:141
1804992012,4cyclictest0-21swapper/109:30:151
180499201,15cyclictest32325-21ls09:40:191
1804992010,3cyclictest21-21ksoftirqd/107:30:201
1804992010,3cyclictest21-21ksoftirqd/107:30:191
1804992010,2cyclictest21-21ksoftirqd/112:27:551
1804992010,2cyclictest21-21ksoftirqd/110:30:391
180399207,5cyclictest1995766kworker/u8:212:10:190
180399205,3cyclictest131rcu_preempt12:40:010
180399202,9cyclictest11276-21fschecks_time12:20:190
180399201,3cyclictest0-21swapper/011:36:130
180399201,3cyclictest0-21swapper/009:10:180
1803992012,6cyclictest24595-21sshd11:22:360
180399201,14cyclictest15317-21ssh11:15:270
1803992010,7cyclictest1-21systemd09:34:120
182599198,7cyclictest29566irq/58-eth011:30:143
182599192,10cyclictest0-21swapper/308:35:153
182599191,9cyclictest29566irq/58-eth008:20:143
182599191,13cyclictest0-21swapper/307:55:303
182599191,12cyclictest0-21swapper/307:40:293
182599191,12cyclictest0-21swapper/307:25:173
181699196,11cyclictest9937-21kthreadcore09:45:242
181699191,9cyclictest5249-21irqcore08:20:192
181699191,9cyclictest26788-21kernelversion07:35:202
181699191,8cyclictest11838-21fschecks_time07:20:182
180499199,2cyclictest21-21ksoftirqd/111:33:551
180499198,9cyclictest21-21ksoftirqd/110:57:071
180499198,2cyclictest21-21ksoftirqd/110:10:251
180499197,5cyclictest131rcu_preempt12:18:081
180499196,3cyclictest21-21ksoftirqd/109:35:141
180499195,11cyclictest13217-21fschecks_time07:55:181
180499192,14cyclictest28404-21apt-get12:10:141
1804991913,4cyclictest0-21swapper/107:45:191
1804991913,4cyclictest0-21swapper/107:25:371
1804991913,3cyclictest21-21ksoftirqd/111:10:311
1804991913,3cyclictest21-21ksoftirqd/111:10:301
180499191,15cyclictest8606-21find11:35:131
180499191,15cyclictest21690-21kthreadcore12:05:211
1804991911,5cyclictest0-21swapper/107:40:171
1804991910,6cyclictest0-21swapper/108:55:291
180499190,17cyclictest29004-21uniq08:45:161
180399195,3cyclictest131rcu_preempt12:15:180
180399194,2cyclictest131rcu_preempt12:05:150
180399191,9cyclictest0-21swapper/011:52:310
180399191,15cyclictest6755-21cron09:45:010
180399191,10cyclictest0-21swapper/011:12:280
180399191,10cyclictest0-21swapper/011:12:270
1803991910,7cyclictest22058-21ssh10:15:260
182599189,6cyclictest0-21swapper/312:17:363
182599188,6cyclictest0-21swapper/310:14:503
182599182,13cyclictest0-21swapper/309:30:243
182599182,10cyclictest0-21swapper/312:30:163
182599182,10cyclictest0-21swapper/311:35:153
182599182,10cyclictest0-21swapper/311:25:153
182599182,10cyclictest0-21swapper/310:55:163
182599182,10cyclictest0-21swapper/308:10:143
182599182,10cyclictest0-21swapper/307:10:143
1825991811,5cyclictest0-21swapper/309:15:133
182599181,12cyclictest0-21swapper/309:20:313
182599181,11cyclictest0-21swapper/307:35:153
182599181,11cyclictest0-21swapper/307:15:163
1825991810,6cyclictest0-21swapper/312:13:223
181699181,8cyclictest1757-21latency_hist07:10:022
180499189,7cyclictest20453-21cron09:10:011
180499187,3cyclictest21-21ksoftirqd/111:08:371
180499182,13cyclictest2638-21kthreadcore08:15:241
1804991812,3cyclictest21-21ksoftirqd/108:10:151
1804991811,5cyclictest5872-21cron08:55:011
1804991810,6cyclictest1-21systemd09:24:101
180499180,2cyclictest131rcu_preempt11:51:301
180399188,8cyclictest26144-21apt-get07:35:150
180399187,8cyclictest3400-21cat09:40:320
180399186,9cyclictest1-21systemd12:04:300
180399183,12cyclictest24431-21sed08:40:210
180399183,11cyclictest8313-21/usr/sbin/munin10:30:210
180399182,4cyclictest12-21ksoftirqd/007:35:010
180399182,4cyclictest12-21ksoftirqd/007:35:010
180399182,2cyclictest0-21swapper/009:50:100
180399181,9cyclictest0-21swapper/011:40:440
180399181,8cyclictest5892-21head07:10:290
180399181,3cyclictest0-21swapper/007:43:570
1803991812,4cyclictest11418-21sh07:20:010
182599179,6cyclictest0-21swapper/311:23:053
182599179,6cyclictest0-21swapper/311:11:203
182599179,6cyclictest0-21swapper/311:11:203
182599179,6cyclictest0-21swapper/310:20:103
182599179,6cyclictest0-21swapper/309:52:413
182599172,10cyclictest0-21swapper/312:00:143
182599172,10cyclictest0-21swapper/310:45:143
182599172,10cyclictest0-21swapper/310:35:193
182599172,10cyclictest0-21swapper/308:50:133
182599172,10cyclictest0-21swapper/308:15:163
1825991711,3cyclictest0-21swapper/311:50:273
180499178,7cyclictest19229-21apt-get08:35:131
180499177,3cyclictest21-21ksoftirqd/108:30:151
180499172,11cyclictest0-21swapper/108:20:041
180499172,10cyclictest0-21swapper/109:00:181
1804991712,3cyclictest0-21swapper/108:40:221
1804991711,4cyclictest30745-21cron07:40:011
180399179,6cyclictest10678-21sshd08:55:370
180399173,11cyclictest22406-21vmstat08:00:380
180399172,10cyclictest0-21swapper/012:30:160
180399172,10cyclictest0-21swapper/007:50:140
1803991711,4cyclictest4624-21apt-config08:20:020
180399171,13cyclictest2388-21ssh10:02:360
180399171,13cyclictest16017-21fschecks_count09:05:170
180399171,13cyclictest14894-21tr07:20:230
180399170,11cyclictest29235-21ssh11:05:150
182599162,10cyclictest0-21swapper/309:05:043
180399162,12cyclictest13133-21cut09:00:240
1803991611,3cyclictest23166-21irqrtprio08:05:190
180399161,13cyclictest0-21swapper/007:25:150
180399161,12cyclictest29161-21cut08:10:210
182599152,10cyclictest0-21swapper/308:00:173
182599151,11cyclictest0-21swapper/308:45:153
180399157,6cyclictest12-21ksoftirqd/008:20:170
180399154,9cyclictest1419-21fschecks_time08:50:150
180399151,11cyclictest14891-21kthreadcore07:55:220
182599148,4cyclictest0-21swapper/307:50:053
182599146,5cyclictest29566irq/58-eth007:20:153
182599141,10cyclictest0-21swapper/307:45:203
180499141,10cyclictest1824-21latency_hist07:10:011
180399141,10cyclictest1967-21cat07:10:010
182599138,3cyclictest0-21swapper/308:30:253
182599132,8cyclictest0-21swapper/308:10:023
182599131,10cyclictest0-21swapper/308:55:313
182599131,10cyclictest0-21swapper/308:25:233
182599131,10cyclictest0-21swapper/307:30:253
182599131,10cyclictest0-21swapper/307:30:253
182599121,9cyclictest0-21swapper/309:00:233
182599111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional