You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 00:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Nov 17, 2025 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
47499574,36cyclictest25178-21ssh11:40:140
47499564,39cyclictest13393-21ssh12:30:160
495995331,15cyclictest0-21swapper/210:40:132
48499521,43cyclictest24971-21ssh09:55:161
49599514,16cyclictest131rcu_preempt09:20:172
49599514,16cyclictest131rcu_preempt09:20:162
484994918,20cyclictest21-21ksoftirqd/108:05:161
48499472,7cyclictest0-21swapper/107:50:151
48499472,31cyclictest9122-21ssh11:50:141
50499454,21cyclictest29566irq/58-eth011:00:163
504994518,19cyclictest29566irq/58-eth010:00:163
48499434,28cyclictest1593-21ssh09:40:141
47499421,33cyclictest1814-21sh11:25:140
474994114,7cyclictest131rcu_preempt08:50:130
48499405,28cyclictest25843-21diskmemload12:00:151
49599393,11cyclictest131rcu_preempt10:10:182
50499376,12cyclictest29666irq/59-eth010:40:143
504993715,10cyclictest29566irq/58-eth007:25:143
49599371,17cyclictest18978-21ssh12:15:162
50499363,19cyclictest29566irq/58-eth010:05:153
50499363,19cyclictest29566irq/58-eth010:05:153
48499367,20cyclictest13902-21ssh10:30:141
48499366,6cyclictest6944-21kthreadcore10:25:211
484993627,5cyclictest0-21swapper/111:55:161
484993627,5cyclictest0-21swapper/111:55:151
48499361,9cyclictest27868-21grep08:45:191
48499361,11cyclictest27399-21latency_hist08:45:011
48499361,10cyclictest2170-21fschecks_time07:45:181
48499361,10cyclictest17161-21kthreadcore07:25:221
48499352,8cyclictest0-21swapper/110:40:131
48499351,10cyclictest12925-21tr11:10:221
47499353,25cyclictest17166-21ssh09:50:150
49599346,8cyclictest0-21swapper/211:25:162
495993421,3cyclictest0-21swapper/209:35:142
48499342,11cyclictest7105-21kthreadcore10:45:241
484993412,19cyclictest6008-21sed10:25:011
474993427,3cyclictest131rcu_preempt11:45:150
49599331,21cyclictest26354-21ssh11:20:132
48499339,3cyclictest512-21sh11:02:221
48499337,4cyclictest4864-21kthreadcore08:20:221
48499332,22cyclictest749-21ssh11:45:151
48499331,26cyclictest26159-21ssh09:35:141
48499331,10cyclictest1885-21latency_hist07:45:011
48499331,10cyclictest12263-21kthreadcore09:45:221
47499331,25cyclictest27488-21ssh09:15:150
49599324,9cyclictest261rcuc/210:50:162
48499328,3cyclictest19156-21diskmemload10:51:281
48499327,4cyclictest19098-21fschecks_count12:15:181
48499321,9cyclictest9565-21latency_hist09:00:021
48499321,9cyclictest4844-21irqcore12:25:201
48499321,9cyclictest4844-21irqcore12:25:191
48499321,9cyclictest3210-21latency_hist08:20:021
48499321,9cyclictest2206-21cut12:05:221
48499321,9cyclictest16087-21irqrtprio10:10:221
48499321,9cyclictest11188-21netstat10:05:271
48499321,9cyclictest11188-21netstat10:05:261
48499321,8cyclictest20389-21kthreadcore11:35:241
48499321,11cyclictest8357-21fschecks_count08:25:181
48499321,10cyclictest9849-21ssh11:30:181
48499321,10cyclictest15829-21kthreadcore09:05:221
48499320,10cyclictest13195-21ssh09:25:241
48499320,10cyclictest13195-21ssh09:25:231
50499312,13cyclictest29566irq/58-eth011:05:133
48499318,4cyclictest11791-21kthreadcore07:20:221
48499317,5cyclictest19156-21diskmemload09:15:471
48499313,10cyclictest26730-21ssh11:40:201
48499311,9cyclictest20365-21kthreadcore09:30:221
48499311,10cyclictest25049-21tr12:35:271
48499311,10cyclictest14553-21cut12:30:221
48499311,10cyclictest13300-21hwlatdetect08:30:181
50499304,15cyclictest29566irq/58-eth012:25:143
50499304,15cyclictest29566irq/58-eth012:25:133
49599304,16cyclictest26346irq/40-dwc2_hso09:40:152
495993022,4cyclictest2746ksoftirqd/207:45:152
48499302,20cyclictest32437-21ssh10:00:171
48499301,9cyclictest643-21/usr/sbin/munin07:10:131
48499301,9cyclictest18697-21kthreadcore08:35:211
48499301,11cyclictest3912-21grep08:50:291
47499304,21cyclictest12-21ksoftirqd/012:20:150
47499304,21cyclictest12-21ksoftirqd/012:20:150
474993013,11cyclictest12-21ksoftirqd/011:15:150
49599293,19cyclictest26346irq/40-dwc2_hso08:25:142
49599293,15cyclictest26346irq/40-dwc2_hso09:55:182
49599293,15cyclictest26346irq/40-dwc2_hso07:55:162
49599291,20cyclictest18427-21ssh09:30:142
49599291,18cyclictest27-21ksoftirqd/207:10:132
48499291,9cyclictest29549-21latency_hist07:40:011
48499291,9cyclictest26492-21tune2fs11:20:151
48499291,11cyclictest28438-21kthreadcore08:10:211
48499291,11cyclictest21650-21ssh10:35:151
48499291,10cyclictest13179-21tr12:10:241
484992910,5cyclictest26082-21ssh09:13:211
48499290,10cyclictest13688-21cut07:55:241
47499296,20cyclictest12-21ksoftirqd/011:05:150
474992915,9cyclictest12-21ksoftirqd/010:30:140
47499291,22cyclictest6217-21ssh10:25:150
50499283,16cyclictest29566irq/58-eth009:35:153
50499282,17cyclictest0-21swapper/311:50:153
49599284,15cyclictest26346irq/40-dwc2_hso08:35:142
49599283,17cyclictest457-21cyclictest07:15:142
49599283,14cyclictest27084-21ssh11:00:162
49599281,21cyclictest30786-21ssh10:20:152
48499286,5cyclictest17081-21irqrtprio08:00:191
48499281,9cyclictest5026-21kthreadcore09:20:241
48499281,9cyclictest5026-21kthreadcore09:20:241
48499281,9cyclictest21553-21kthreadcore10:55:251
48499281,9cyclictest21553-21kthreadcore10:55:241
484992811,4cyclictest17883-21kthreadcore09:50:221
48499281,10cyclictest7084-21kthreadcore07:15:201
48499280,9cyclictest20286-21ssh11:15:231
47499284,20cyclictest12-21ksoftirqd/011:55:160
47499284,20cyclictest12-21ksoftirqd/011:55:150
49599273,20cyclictest0-21swapper/208:50:152
49599272,19cyclictest13223-21ssh12:30:052
49599272,10cyclictest1547-21ssh12:05:172
49599271,17cyclictest26346irq/40-dwc2_hso08:30:162
48499279,3cyclictest30173-21sh10:19:401
48499272,9cyclictest3358-21kthreadcore11:25:231
47499271,18cyclictest25850-21ssh12:00:150
504992611,9cyclictest29566irq/58-eth009:20:293
504992611,9cyclictest29566irq/58-eth009:20:283
49599264,15cyclictest26346irq/40-dwc2_hso11:35:152
49599263,8cyclictest457-21cyclictest10:55:162
49599263,8cyclictest457-21cyclictest10:55:152
49599263,16cyclictest26346irq/40-dwc2_hso07:40:152
49599260,11cyclictest3774-21rm10:45:142
47499262,18cyclictest10665-21ssh11:10:140
50499255,11cyclictest0-21swapper/309:30:143
49599252,17cyclictest7668-21ssh10:05:172
49599252,17cyclictest7668-21ssh10:05:162
49599252,17cyclictest0-21swapper/211:40:142
495992511,6cyclictest614-21systemd-logind12:10:122
48499251,9cyclictest27404-21apt-get12:20:051
48499251,9cyclictest27404-21apt-get12:20:051
47499251,17cyclictest19178-21ssh10:55:170
47499251,17cyclictest19178-21ssh10:55:160
50499245,12cyclictest0-21swapper/312:30:153
50499242,14cyclictest0-21swapper/312:05:163
49599246,14cyclictest19714-21ls07:30:022
49599243,16cyclictest26346irq/40-dwc2_hso07:30:132
49599242,14cyclictest26346irq/40-dwc2_hso11:55:142
49599242,14cyclictest26346irq/40-dwc2_hso11:55:132
49599241,16cyclictest22877-21grep08:40:162
48499241,9cyclictest9721-21/usr/sbin/munin09:00:171
47499246,14cyclictest17344-21ssh11:35:140
50499232,11cyclictest29566irq/58-eth010:25:153
504992316,5cyclictest33-21ksoftirqd/310:55:163
504992316,5cyclictest33-21ksoftirqd/310:55:153
49599239,9cyclictest0-21swapper/209:15:132
49599237,10cyclictest0-21swapper/209:05:152
49599234,12cyclictest26346irq/40-dwc2_hso08:45:142
49599233,7cyclictest0-21swapper/210:30:142
495992313,7cyclictest29344-21cron07:40:012
495992312,9cyclictest32506-21ssh12:03:262
49599231,17cyclictest26346irq/40-dwc2_hso11:15:162
49599231,16cyclictest26346irq/40-dwc2_hso08:05:152
48499234,12cyclictest2646-21sh11:05:131
47499239,6cyclictest0-21swapper/012:08:400
47499234,16cyclictest151rcuc/010:00:170
47499231,9cyclictest25870-21kthreadcore10:15:240
47499231,18cyclictest25088-21ls10:35:230
50499223,11cyclictest29566irq/58-eth010:30:153
49599223,14cyclictest26346irq/40-dwc2_hso09:45:152
49599222,12cyclictest22241-21ssh12:35:182
49599221,18cyclictest10015-21ssh10:25:302
47499222,15cyclictest10178-21ssh12:10:160
47499221,3cyclictest131rcu_preempt12:29:330
47499221,3cyclictest131rcu_preempt12:29:320
474992211,8cyclictest27351-21cron08:45:020
474992211,8cyclictest12-21ksoftirqd/011:30:270
50499211,16cyclictest29566irq/58-eth011:25:143
49599212,9cyclictest32412-21/usr/sbin/munin10:00:292
49599212,4cyclictest0-21swapper/209:50:142
49599212,10cyclictest4348-21awk11:45:252
49599211,9cyclictest1889-21latency08:15:262
49599211,7cyclictest1377066ssh11:50:332
49599211,5cyclictest21776-21grep10:35:162
495992111,8cyclictest27-21ksoftirqd/212:20:222
495992111,8cyclictest27-21ksoftirqd/212:20:212
49599211,15cyclictest26346irq/40-dwc2_hso08:20:152
49599211,12cyclictest24609-21kthreadcore10:15:232
495992110,8cyclictest29752-1kworker/2:0H12:25:182
495992110,8cyclictest29752-1kworker/2:0H12:25:172
47499214,8cyclictest21786-21kthreadcore09:30:230
47499212,16cyclictest11437-21grep07:55:020
47499211,14cyclictest10884-21irqcore09:25:200
47499211,14cyclictest10884-21irqcore09:25:190
47499211,12cyclictest24548-21latency_hist07:35:010
50499209,6cyclictest0-21swapper/309:40:383
50499204,9cyclictest0-21swapper/310:35:143
504992011,6cyclictest29566irq/58-eth011:40:153
49599205,12cyclictest26919-21kthreadcore08:10:192
49599202,2cyclictest0-21swapper/209:00:162
49599202,2cyclictest0-21swapper/207:55:022
49599201,15cyclictest14532-21timerwakeupswit09:25:302
49599201,15cyclictest14532-21timerwakeupswit09:25:302
49599201,15cyclictest13169-21cut11:10:212
49599201,15cyclictest10292-21cron11:10:022
49599201,14cyclictest10299-21acpi07:20:122
49599201,10cyclictest8558-21switchtime08:55:282
47499207,5cyclictest22875-21cut08:05:210
47499205,10cyclictest12-21ksoftirqd/011:20:130
47499203,12cyclictest22224-21ssh12:35:160
47499202,3cyclictest131rcu_preempt10:40:350
47499202,2cyclictest131rcu_preempt08:35:130
474992012,6cyclictest16241-21sshd07:55:370
47499201,14cyclictest9110-21apt-get11:50:150
47499200,15cyclictest10445-21cut07:20:170
50499199,6cyclictest0-21swapper/312:02:573
50499193,8cyclictest29566irq/58-eth010:50:153
50499192,10cyclictest0-21swapper/310:15:173
50499192,10cyclictest0-21swapper/307:40:173
50499191,14cyclictest29566irq/58-eth009:55:173
50499191,13cyclictest0-21swapper/307:45:163
50499191,12cyclictest0-21swapper/309:50:153
50499191,12cyclictest0-21swapper/308:45:303
50499191,12cyclictest0-21swapper/307:30:303
50499191,10cyclictest0-21swapper/308:15:023
50499191,10cyclictest0-21swapper/307:10:013
504991910,7cyclictest0-21swapper/311:13:373
49599196,9cyclictest10694-21kthreadcore11:30:222
49599193,13cyclictest586-21latency_hist07:10:022
495991911,5cyclictest19508-21ssh09:10:132
49599191,10cyclictest20011-21ntp_states08:00:272
47499193,12cyclictest21027-21kthreadcore09:10:210
47499193,11cyclictest0-21swapper/008:45:140
47499192,2cyclictest131rcu_preempt10:45:280
47499192,2cyclictest131rcu_preempt09:47:390
47499191,9cyclictest10068-21ps09:00:200
47499191,3cyclictest131rcu_preempt07:35:150
47499191,2cyclictest131rcu_preempt10:09:300
47499191,2cyclictest131rcu_preempt10:09:290
47499191,13cyclictest5183-21sh09:40:260
474991910,7cyclictest27192-21diskstats11:00:180
50499189,6cyclictest0-21swapper/312:17:523
50499187,5cyclictest0-21swapper/311:45:143
50499185,9cyclictest29566irq/58-eth009:15:143
50499182,12cyclictest0-21swapper/311:55:203
50499182,12cyclictest0-21swapper/311:55:203
50499182,10cyclictest0-21swapper/310:10:043
50499182,10cyclictest0-21swapper/309:10:153
50499182,10cyclictest0-21swapper/308:40:133
50499182,10cyclictest0-21swapper/308:35:163
50499182,10cyclictest0-21swapper/308:15:183
504991811,4cyclictest0-21swapper/312:35:253
50499181,13cyclictest0-21swapper/309:00:313
50499181,12cyclictest0-21swapper/308:00:183
50499181,12cyclictest0-21swapper/307:25:013
50499181,12cyclictest0-21swapper/307:15:173
504991810,6cyclictest0-21swapper/311:20:273
504991810,5cyclictest29566irq/58-eth012:10:303
47499184,11cyclictest14606-21cpuspeed09:05:160
47499183,10cyclictest5639-21ls07:45:300
47499181,9cyclictest8533-21wc08:55:290
47499181,5cyclictest0-21swapper/010:51:140
47499181,2cyclictest131rcu_preempt10:13:550
47499181,14cyclictest5985-21cut09:20:260
47499181,14cyclictest5985-21cut09:20:250
47499181,11cyclictest29636-21missed_timers09:35:250
47499180,2cyclictest131rcu_preempt10:23:090
47499180,2cyclictest131rcu_preempt09:55:390
50499179,6cyclictest0-21swapper/312:23:523
50499179,6cyclictest0-21swapper/312:23:513
50499177,7cyclictest0-21swapper/311:15:233
50499173,6cyclictest0-21swapper/311:30:163
50499172,11cyclictest0-21swapper/310:45:153
50499172,10cyclictest0-21swapper/308:20:173
50499171,8cyclictest29566irq/58-eth009:25:163
50499171,8cyclictest29566irq/58-eth009:25:153
50499171,13cyclictest0-21swapper/311:35:293
50499171,11cyclictest0-21swapper/310:20:153
504991710,3cyclictest0-21swapper/309:46:143
48499172,13cyclictest23788-21grep07:30:301
47499179,6cyclictest12-21ksoftirqd/008:25:340
47499178,5cyclictest26473-21wc08:10:180
47499177,7cyclictest121ksoftirqd/008:20:320
47499171,2cyclictest131rcu_preempt07:10:120
47499171,12cyclictest19072-21df_inode12:15:170
47499170,12cyclictest13215-21fschecks_count08:30:180
50499162,10cyclictest0-21swapper/309:05:163
50499162,10cyclictest0-21swapper/308:50:143
50499162,10cyclictest0-21swapper/307:35:153
50499161,12cyclictest0-21swapper/309:00:023
47499162,11cyclictest29922-21grep07:40:210
47499162,11cyclictest20045-21ps08:00:280
47499162,10cyclictest473-21latency_hist07:10:010
47499161,12cyclictest15139-21apt-get07:25:150
474991610,4cyclictest5825-21irqcore07:15:180
47499160,8cyclictest1917-21wc08:15:270
50499152,9cyclictest0-21swapper/308:25:143
50499152,10cyclictest0-21swapper/308:05:043
50499152,10cyclictest0-21swapper/307:10:153
48499158,5cyclictest623-21latency_hist07:10:011
50499149,3cyclictest0-21swapper/307:56:373
50499141,7cyclictest0-21swapper/307:50:303
50499124,5cyclictest0-21swapper/308:30:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional