You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 09:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Thu Nov 20, 2025 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3067999532,39cyclictest20551-21ssh21:55:140
3070699507,26cyclictest29566irq/58-eth023:50:163
30706994925,13cyclictest29566irq/58-eth023:20:133
30706994925,13cyclictest29566irq/58-eth023:20:123
3070699483,26cyclictest29566irq/58-eth021:40:113
3067999480,36cyclictest30140-21ssh23:00:130
3070699464,27cyclictest29566irq/58-eth021:15:123
3067999423,6cyclictest131rcu_preempt21:40:110
3070699412,22cyclictest29566irq/58-eth000:10:143
3070699412,22cyclictest29566irq/58-eth000:10:133
3070699402,23cyclictest29566irq/58-eth021:45:113
3070699394,20cyclictest29566irq/58-eth022:45:123
3069899392,19cyclictest13504-21ssh23:47:182
3067999383,3cyclictest131rcu_preempt21:20:110
3070699352,21cyclictest29566irq/58-eth021:10:153
3070699342,15cyclictest29566irq/58-eth000:00:133
30689993325,5cyclictest0-21swapper/123:45:181
3067999331,24cyclictest16843-21ssh00:10:140
3067999331,24cyclictest16843-21ssh00:10:130
3070699323,19cyclictest29566irq/58-eth000:20:133
3067999322,7cyclictest131rcu_preempt20:05:130
3070699312,14cyclictest29566irq/58-eth021:50:133
30706993117,10cyclictest29566irq/58-eth022:30:143
3070699311,18cyclictest29566irq/58-eth023:45:173
3068999313,25cyclictest23028-21ssh23:35:121
3067999311,25cyclictest11624-21ssh21:10:150
30679993110,15cyclictest12-21ksoftirqd/023:15:140
3070699302,16cyclictest29566irq/58-eth000:35:143
3069899301,24cyclictest28512-21ssh21:40:122
3068999302,25cyclictest15575-21ssh00:30:171
3070699294,12cyclictest0-21swapper/323:35:113
3069899297,8cyclictest29292-21latency_hist22:40:012
3069899293,17cyclictest26346irq/40-dwc2_hso21:50:152
3069899291,20cyclictest14434-21latency_hist23:30:002
3068999291,25cyclictest4935-21ssh22:25:131
3067999291,20cyclictest7409-21ssh23:45:180
3069899283,10cyclictest0-21swapper/223:43:282
3069899282,20cyclictest0-21swapper/220:20:052
3068999285,20cyclictest22478-21ssh23:15:131
3070699277,13cyclictest0-21swapper/323:55:133
3067999272,19cyclictest21761-21ssh23:14:280
3069899261,5cyclictest10773-21ssh23:05:302
30698992614,5cyclictest0-21swapper/200:20:022
3069899261,20cyclictest0-21swapper/200:39:292
3069899261,12cyclictest5725-21grep22:45:172
30689992611,13cyclictest21-21ksoftirqd/100:25:121
30679992611,9cyclictest12-21ksoftirqd/000:00:130
3069899251,6cyclictest21785-21ssh21:35:192
30698992512,5cyclictest31102-21irqcore23:20:172
30698992512,5cyclictest31102-21irqcore23:20:162
3069899251,21cyclictest32416-21ls00:20:022
3069899251,17cyclictest5186-21ls22:25:152
3069899251,17cyclictest0-21swapper/222:42:092
3069899251,15cyclictest23594-21kthreadcore22:15:222
3069899251,14cyclictest21657-21kthreadcore20:45:202
3068999253,3cyclictest131rcu_preempt23:50:151
30679992510,9cyclictest12-21ksoftirqd/022:45:130
3070699248,9cyclictest29566irq/58-eth000:15:153
3070699243,14cyclictest0-21swapper/322:35:143
3070699243,13cyclictest29566irq/58-eth023:05:113
3070699243,11cyclictest29566irq/58-eth022:50:123
3069899241,9cyclictest0-21swapper/223:17:292
3069899241,5cyclictest0-21swapper/220:15:142
30698992411,9cyclictest0-21swapper/220:05:182
3069899241,17cyclictest0-21swapper/222:56:582
3069899241,16cyclictest18518-21fschecks_count20:10:182
3069899241,15cyclictest0-21swapper/223:30:322
3069899241,15cyclictest0-21swapper/221:55:222
3069899241,15cyclictest0-21swapper/221:17:322
3069899241,12cyclictest32765-21missed_timers20:55:212
30689992417,3cyclictest131rcu_preempt23:10:131
3068999241,20cyclictest12114-21ssh21:30:121
3068999241,13cyclictest131rcu_preempt21:00:231
3070699233,13cyclictest0-21swapper/323:14:283
3070699232,16cyclictest0-21swapper/322:00:143
3069899233,15cyclictest26346irq/40-dwc2_hso22:00:162
3069899232,9cyclictest28233-21ssh21:20:132
3069899232,6cyclictest0-21swapper/223:10:272
30698992311,9cyclictest0-21swapper/200:30:162
3069899231,16cyclictest11223-21diskmemload22:52:582
3069899231,15cyclictest0-21swapper/223:03:572
3069899231,14cyclictest23732-21irqstats23:35:172
3069899231,14cyclictest0-21swapper/200:12:192
3069899231,14cyclictest0-21swapper/200:12:192
3069899231,12cyclictest32456-21sh00:00:142
3069899231,12cyclictest11618-21ssh21:10:142
3069899231,11cyclictest24100-21ssh23:55:142
3068999237,7cyclictest131rcu_preempt00:24:141
3068999232,17cyclictest29095-21ssh22:20:121
3068999230,19cyclictest24232-21uniq23:55:171
3067999238,12cyclictest12-21ksoftirqd/019:55:050
30679992315,5cyclictest0-21swapper/000:35:310
3067999231,17cyclictest30743-21ssh23:20:130
3067999231,17cyclictest30743-21ssh23:20:120
30706992212,7cyclictest0-21swapper/322:30:023
3069899227,5cyclictest131rcu_preempt19:26:262
3069899224,10cyclictest11103-21kworker/2:021:31:032
3069899221,9cyclictest3592-21apt-config19:55:012
3069899221,9cyclictest0-21swapper/221:25:162
3069899221,5cyclictest0-21swapper/200:26:012
3069899221,15cyclictest0-21swapper/222:33:482
3069899221,13cyclictest0-21swapper/223:50:162
3069899220,9cyclictest9256-21kthreadcore00:05:212
3068999227,7cyclictest131rcu_preempt22:10:401
3068999222,9cyclictest131rcu_preempt21:49:461
3068999222,17cyclictest6599-21ssh23:25:171
3068999222,16cyclictest14798-21ssh23:30:141
3068999222,12cyclictest131rcu_preempt22:00:391
3068999221,9cyclictest131rcu_preempt22:53:201
3068999221,9cyclictest131rcu_preempt21:51:211
30689992215,4cyclictest629-21Async14:00:481
30689992212,7cyclictest11221-21cron21:10:011
3068999221,14cyclictest131rcu_preempt21:12:141
3067999221,9cyclictest28731-21/usr/sbin/munin22:00:330
30679992213,6cyclictest0-21swapper/020:00:370
3070699212,12cyclictest0-21swapper/323:15:283
3070699211,14cyclictest0-21swapper/321:55:143
3069899214,12cyclictest0-21swapper/220:50:152
3069899211,9cyclictest0-21swapper/222:21:362
3069899211,8cyclictest11223-21diskmemload22:14:452
30698992111,8cyclictest0-21swapper/221:45:202
3068999219,10cyclictest0-21swapper/100:10:101
3068999219,10cyclictest0-21swapper/100:10:091
3068999218,9cyclictest131rcu_preempt21:40:111
3068999217,2cyclictest131rcu_preempt21:24:181
3068999216,6cyclictest131rcu_preempt21:18:421
3068999215,3cyclictest21-21ksoftirqd/120:30:171
3068999215,12cyclictest21-21ksoftirqd/100:08:281
3068999213,13cyclictest21-21ksoftirqd/123:05:121
30689992110,6cyclictest32101-21kthreadcore23:40:161
3067999216,3cyclictest131rcu_preempt00:05:130
3067999215,7cyclictest131rcu_preempt22:53:350
3067999212,14cyclictest32559-21ssh00:20:120
3070699203,12cyclictest29566irq/58-eth021:25:173
30706992011,6cyclictest0-21swapper/323:00:183
3069899206,12cyclictest0-21swapper/220:00:202
3069899201,3cyclictest0-21swapper/222:05:192
3068999207,2cyclictest131rcu_preempt22:05:391
3068999202,13cyclictest26869-21memory00:35:241
3068999201,9cyclictest131rcu_preempt21:38:201
30689992014,4cyclictest0-21swapper/100:00:141
30689992011,6cyclictest30609-21irqcore23:00:171
30689992010,2cyclictest131rcu_preempt23:20:091
30689992010,2cyclictest131rcu_preempt23:20:081
3067999209,3cyclictest131rcu_preempt22:10:110
3067999203,8cyclictest131rcu_preempt20:20:230
30679992015,3cyclictest530-21usb-storage22:25:110
3067999201,3cyclictest0-21swapper/022:39:440
30679992013,4cyclictest0-21swapper/020:30:300
3070699199,6cyclictest29566irq/58-eth023:40:113
3070699192,11cyclictest0-21swapper/320:45:143
30706991911,6cyclictest0-21swapper/322:40:283
30706991911,5cyclictest0-21swapper/322:55:193
3070699191,14cyclictest29566irq/58-eth000:30:173
3070699191,12cyclictest0-21swapper/323:25:173
3070699191,12cyclictest0-21swapper/319:55:123
3070699191,12cyclictest0-21swapper/319:55:023
3069899198,4cyclictest27-21ksoftirqd/219:55:032
3069899198,3cyclictest31081-21sendmail19:20:012
3069899197,5cyclictest7480-21kthreadcore20:30:242
3069899197,10cyclictest3822-21memory20:25:232
3069899191,9cyclictest6802-21cut21:05:162
3069899191,9cyclictest5090-21grep21:00:232
3069899191,9cyclictest32484-21idleruntime-cro19:50:012
3069899191,10cyclictest18374-21ntp_192.168.11420:40:252
3069899191,10cyclictest13109-21cut20:35:232
3069899191,10cyclictest13109-21cut20:35:222
3068999192,12cyclictest0-21swapper/122:40:151
3068999192,11cyclictest0-21swapper/120:00:171
3068999191,8cyclictest131rcu_preempt20:55:161
3068999191,3cyclictest131rcu_preempt22:15:151
30689991913,4cyclictest0-21swapper/121:55:181
3068999191,15cyclictest4110-21ls19:55:131
3068999191,14cyclictest21408-21ssh22:35:141
30689991910,6cyclictest0-21swapper/120:40:281
3067999196,4cyclictest131rcu_preempt21:45:140
3067999195,4cyclictest0-21swapper/021:35:210
3067999194,9cyclictest131rcu_preempt23:50:220
3067999194,7cyclictest131rcu_preempt20:10:140
3067999193,5cyclictest12-21ksoftirqd/022:15:210
3067999193,4cyclictest131rcu_preempt23:35:110
3067999192,8cyclictest131rcu_preempt21:17:520
3067999192,13cyclictest15805-21ntp_states21:30:240
3067999191,9cyclictest0-21swapper/000:32:120
3067999191,8cyclictest10695-21sed20:35:190
3067999191,8cyclictest10695-21sed20:35:180
3067999191,7cyclictest18363-21ntp_192.168.11423:30:260
3067999191,4cyclictest11117-21sh21:29:040
3067999191,3cyclictest131rcu_preempt23:40:100
30679991913,4cyclictest0-21swapper/023:55:120
3067999191,2cyclictest0-21swapper/000:25:180
30679991912,5cyclictest0-21swapper/023:08:340
30679991911,6cyclictest31771-21cron19:35:010
3067999191,14cyclictest8294-1kworker/u9:121:50:130
3067999191,14cyclictest14970-21apt-get20:40:140
3067999191,11cyclictest32664-21ls19:50:020
30679991910,3cyclictest131rcu_preempt20:20:020
3070699189,6cyclictest29566irq/58-eth023:30:143
3070699189,6cyclictest0-21swapper/322:20:213
3070699189,6cyclictest0-21swapper/322:12:083
3070699188,6cyclictest0-21swapper/300:28:093
3070699182,10cyclictest0-21swapper/320:40:153
3070699182,10cyclictest0-21swapper/320:15:163
3070699181,12cyclictest0-21swapper/321:20:143
3070699181,12cyclictest0-21swapper/320:00:163
3070699181,11cyclictest0-21swapper/321:30:153
3070699181,10cyclictest0-21swapper/320:25:183
3069899181,8cyclictest32175-21latency_hist19:40:022
3069899181,14cyclictest32259-21grep19:45:002
3069899181,14cyclictest31880-21latency_hist19:35:012
3069899181,10cyclictest31324-21munin-run19:25:002
3068999187,6cyclictest32006-21ls19:40:011
3068999183,3cyclictest131rcu_preempt20:35:191
3068999183,3cyclictest131rcu_preempt20:35:181
3068999182,13cyclictest15292-21kthreadcore20:05:191
3068999182,12cyclictest22053-21df22:55:171
3068999181,8cyclictest7301-21kthreadcore22:45:211
3068999181,2cyclictest131rcu_preempt00:20:011
3068999181,14cyclictest5993-21cut21:25:231
3068999181,14cyclictest24844-21ls20:50:201
3067999189,6cyclictest28960-21unixbench_multi00:15:310
3067999187,6cyclictest131rcu_preempt22:20:120
3067999187,2cyclictest131rcu_preempt21:05:180
3067999182,4cyclictest91-21pr/ttyS019:55:010
3067999182,2cyclictest131rcu_preempt20:55:010
3067999181,4cyclictest0-21swapper/022:41:320
3067999181,3cyclictest0-21swapper/022:33:070
3067999181,3cyclictest0-21swapper/019:24:490
30679991812,4cyclictest0-21swapper/022:55:310
30679991812,4cyclictest0-21swapper/020:55:250
30679991811,5cyclictest14437-21cat23:30:010
3070699179,6cyclictest0-21swapper/322:05:403
3070699179,6cyclictest0-21swapper/321:39:523
3070699175,9cyclictest0-21swapper/322:15:223
3070699172,11cyclictest0-21swapper/300:05:143
3070699172,10cyclictest0-21swapper/321:00:143
3070699172,10cyclictest0-21swapper/320:55:123
3070699171,10cyclictest0-21swapper/320:30:173
3069899171,13cyclictest30840-21ls19:15:002
3068999174,10cyclictest32260-21munin-plugin-st19:45:011
3068999174,10cyclictest27917-21ls20:20:011
3068999172,3cyclictest131rcu_preempt20:25:241
3068999172,3cyclictest131rcu_preempt20:20:251
3068999171,6cyclictest622-21irqcore19:52:061
3068999171,3cyclictest131rcu_preempt20:10:201
3067999178,3cyclictest131rcu_preempt22:05:160
3067999172,12cyclictest4019-21ps20:25:270
3067999171,14cyclictest31075-21cron19:20:010
30679991711,4cyclictest0-21swapper/021:00:270
3067999171,13cyclictest20283-21sed20:45:180
3070699162,10cyclictest0-21swapper/321:05:153
3070699161,8cyclictest0-21swapper/319:15:023
3069899161,12cyclictest30703-21latency_hist19:10:022
3068999167,6cyclictest31550-21ls19:30:011
3068999167,6cyclictest0-21swapper/119:45:321
30689991610,4cyclictest19760-21/usr/sbin/munin20:45:181
3067999161,3cyclictest0-21swapper/019:39:410
3070699151,7cyclictest0-21swapper/320:55:013
3068999151,11cyclictest31320-21idleruntime-cro19:25:011
3068999151,11cyclictest1-21systemd19:15:291
3068999151,10cyclictest31778-21munin-plugin-st19:35:021
3068999151,10cyclictest30708-21cat19:10:011
3068999150,12cyclictest30832-21cron19:15:011
3070699149,3cyclictest0-21swapper/320:35:163
3070699149,3cyclictest0-21swapper/320:35:153
3070699149,3cyclictest0-21swapper/320:10:303
3070699149,3cyclictest0-21swapper/319:39:013
3070699147,4cyclictest0-21swapper/320:05:283
3070699141,8cyclictest0-21swapper/319:45:023
3067999145,6cyclictest12-21ksoftirqd/019:10:010
3067999141,4cyclictest32247-21cron19:45:000
3067999141,10cyclictest31607-21latency_hist19:30:020
3067999141,10cyclictest30842-21munin-run19:15:000
3070699138,3cyclictest0-21swapper/319:48:313
3070699132,9cyclictest0-21swapper/320:20:213
3070699131,8cyclictest0-21swapper/319:20:013
3070699122,7cyclictest0-21swapper/319:35:023
3070699121,7cyclictest0-21swapper/319:30:033
3070699116,3cyclictest0-21swapper/319:22:363
3070699101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional