You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-08 - 22:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Thu Jan 08, 2026 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
190999673,37cyclictest13164-21ssh10:25:120
190999524,8cyclictest131rcu_preempt09:35:130
1932995037,4cyclictest13179-21ssh10:25:142
191899471,7cyclictest13207-21ssh11:05:141
191899451,5cyclictest0-21swapper/107:20:131
194099445,24cyclictest28366irq/58-eth009:40:133
191899442,4cyclictest0-21swapper/108:35:121
191899431,8cyclictest12823-21apt-get09:45:121
191899431,34cyclictest24235-21ssh10:50:111
191899422,6cyclictest26720-21apt-get11:50:121
191899422,10cyclictest1464-21ssh09:20:111
191899421,22cyclictest10554-21ssh10:05:141
191899421,22cyclictest10554-21ssh10:05:131
191899411,4cyclictest0-21swapper/112:25:121
191899403,3cyclictest0-21swapper/112:15:141
1909994015,18cyclictest12-21ksoftirqd/008:45:120
1940993818,12cyclictest28366irq/58-eth011:35:133
191899382,8cyclictest0-21swapper/109:00:131
191899381,6cyclictest15993-21diskmemload12:20:221
191899375,27cyclictest0-21swapper/111:20:131
191899372,4cyclictest0-21swapper/110:00:081
193299361,18cyclictest6640-21ssh09:20:512
191899361,5cyclictest24456-21apt-get08:45:111
191899361,32cyclictest0-21swapper/110:10:131
191899360,6cyclictest16533-21apt-get07:25:131
191899360,6cyclictest16533-21apt-get07:25:121
191899360,5cyclictest25075-21apt-get09:15:131
194099351,21cyclictest28366irq/58-eth010:05:133
194099351,21cyclictest28366irq/58-eth010:05:123
191899359,23cyclictest23696-21cut10:30:181
191899352,28cyclictest6945-21ssh10:40:111
191899351,6cyclictest27907-21ssh10:15:101
190999353,17cyclictest12840-21ssh09:45:130
194099342,22cyclictest28366irq/58-eth010:40:123
1940993413,16cyclictest28366irq/58-eth011:20:123
193299341,15cyclictest12846-21ssh09:45:132
191899341,9cyclictest32574-21cat08:50:221
191899341,6cyclictest0-21swapper/107:35:121
191899341,5cyclictest0-21swapper/108:40:121
191899341,3cyclictest18041-21ssh11:45:111
191899341,13cyclictest30997-21if_err_eth010:35:171
191899341,11cyclictest31385-21ntp_192.168.11409:35:241
1909993423,6cyclictest15648-21apt-get11:25:120
194099339,15cyclictest28366irq/58-eth007:50:133
194099333,17cyclictest28366irq/58-eth010:10:133
191899339,21cyclictest0-21swapper/108:00:231
191899337,24cyclictest24304-21sh11:30:101
191899333,2cyclictest15993-21diskmemload11:36:051
191899331,9cyclictest18652-21awk09:30:011
191899331,8cyclictest24147-21cut09:50:221
191899331,8cyclictest1876-21ntp_192.168.11411:15:231
191899331,8cyclictest1876-21ntp_192.168.11411:15:221
191899331,8cyclictest13283-21kthreadcore12:00:191
191899331,7cyclictest21973-21df11:10:141
191899331,7cyclictest11803-21ssh11:03:411
191899331,6cyclictest15929-21kthreadcore07:55:211
191899331,6cyclictest15929-21kthreadcore07:55:201
191899331,5cyclictest4307-21ssh10:20:131
191899331,4cyclictest32764-21apt-get08:20:101
191899331,12cyclictest5141-21latency_hist12:15:021
191899331,11cyclictest3639-21ssh12:30:311
191899331,11cyclictest15677-21df10:45:121
191899330,7cyclictest5025-21memory08:55:231
191899330,5cyclictest4600-21sed09:40:191
194099324,24cyclictest28366irq/58-eth011:45:113
1932993210,14cyclictest27-21ksoftirqd/211:45:142
191899323,5cyclictest8539-21missed_timers08:25:231
191899323,5cyclictest8539-21missed_timers08:25:221
191899321,9cyclictest10923-21grep07:50:211
191899321,8cyclictest9486-21kthreadcore07:15:221
191899321,8cyclictest5077-21kthreadcore11:55:201
191899321,5cyclictest6944-21grep07:45:271
191899321,24cyclictest9336-21ssh11:40:121
191899320,7cyclictest12710-21apt-get10:25:031
191899316,22cyclictest0-21swapper/108:30:221
191899311,9cyclictest23951-21cut08:10:141
191899311,9cyclictest23945-21ntp_192.168.11412:05:231
191899311,8cyclictest2037-21/usr/sbin/munin07:10:161
191899311,8cyclictest19034-21ssh09:30:121
191899311,8cyclictest17132-21cut09:10:201
191899311,6cyclictest23593-21ssh11:29:301
191899311,6cyclictest2056-21ntpq07:40:231
191899311,2cyclictest0-21swapper/112:35:141
193299309,14cyclictest27-21ksoftirqd/212:04:442
193299309,13cyclictest630-21tcf-agent10:40:122
1932993011,10cyclictest0-21swapper/208:35:102
191899301,8cyclictest21809-21sed07:30:181
191899300,9cyclictest31197-21ssh09:56:341
191899300,7cyclictest2390-21ssh10:57:501
190999302,22cyclictest24272-21ssh09:30:590
194099293,11cyclictest0-21swapper/309:20:123
1940992914,9cyclictest28366irq/58-eth011:50:323
193299295,9cyclictest27-21ksoftirqd/209:29:492
193299293,21cyclictest0-21swapper/210:19:502
191899291,8cyclictest27889-21/usr/sbin/munin08:15:171
191899291,8cyclictest11939-21irqrtprio09:05:161
194099287,15cyclictest28366irq/58-eth011:55:133
193299289,4cyclictest27-21ksoftirqd/211:19:442
193299289,4cyclictest27-21ksoftirqd/211:19:432
193299287,9cyclictest27-21ksoftirqd/211:29:092
193299283,10cyclictest131rcu_preempt09:30:212
193299282,19cyclictest26624-21rm11:50:042
193299282,19cyclictest0-21swapper/207:45:132
193299282,11cyclictest0-21swapper/207:50:122
1932992815,9cyclictest0-21swapper/210:45:102
193299281,4cyclictest271ksoftirqd/210:12:272
1932992812,13cyclictest32507-21cron08:20:012
1932992812,12cyclictest0-21swapper/208:55:172
191899281,5cyclictest23015-21latency_hist08:10:021
1909992819,7cyclictest12-21ksoftirqd/011:10:110
194099273,20cyclictest28366irq/58-eth011:40:123
194099273,16cyclictest0-21swapper/312:26:303
194099271,14cyclictest28366irq/58-eth012:15:123
193299279,2cyclictest131rcu_preempt11:57:262
193299278,7cyclictest27-21ksoftirqd/209:39:092
193299278,5cyclictest0-21swapper/210:30:042
193299274,5cyclictest27-21ksoftirqd/211:35:172
193299273,14cyclictest22346irq/40-dwc2_hso07:55:122
193299273,14cyclictest22346irq/40-dwc2_hso07:55:112
193299272,19cyclictest0-21swapper/208:10:102
193299271,5cyclictest271ksoftirqd/212:25:212
1932992710,5cyclictest27-21ksoftirqd/212:06:202
1932992710,11cyclictest131rcu_preempt12:24:162
190999272,19cyclictest15531-21ssh09:25:590
1909992720,3cyclictest26186-21apt-get07:35:110
194099265,12cyclictest0-21swapper/312:35:143
1940992617,5cyclictest28366irq/58-eth012:00:113
194099261,19cyclictest0-21swapper/309:15:243
193299268,11cyclictest27-21ksoftirqd/210:09:492
193299268,11cyclictest27-21ksoftirqd/210:09:482
193299267,4cyclictest0-21swapper/208:50:122
193299266,9cyclictest27-21ksoftirqd/212:30:122
193299266,9cyclictest27-21ksoftirqd/208:10:002
193299263,15cyclictest131rcu_preempt10:36:342
193299263,14cyclictest22346irq/40-dwc2_hso08:20:122
193299261,13cyclictest0-21swapper/212:15:112
190999267,5cyclictest12-21ksoftirqd/007:50:020
190999265,5cyclictest12-21ksoftirqd/009:57:520
190999264,4cyclictest12-21ksoftirqd/010:11:510
190999261,7cyclictest12-21ksoftirqd/011:55:110
193299258,11cyclictest27-21ksoftirqd/209:15:252
193299257,7cyclictest27-21ksoftirqd/210:22:512
193299253,3cyclictest131rcu_preempt11:20:202
1932992511,8cyclictest27-21ksoftirqd/209:59:282
1932992511,3cyclictest131rcu_preempt11:00:342
193299250,15cyclictest7627-21find12:35:112
191899251,3cyclictest0-21swapper/107:10:021
190999255,4cyclictest12-21ksoftirqd/009:42:240
194099243,16cyclictest0-21swapper/310:20:123
193299247,11cyclictest27-21ksoftirqd/210:50:192
193299246,8cyclictest27-21ksoftirqd/212:11:592
193299246,12cyclictest27-21ksoftirqd/211:12:322
193299246,11cyclictest27-21ksoftirqd/209:53:162
193299246,10cyclictest131rcu_preempt07:40:292
193299244,4cyclictest27-21ksoftirqd/207:20:122
193299244,11cyclictest27-21ksoftirqd/211:05:302
193299242,15cyclictest27-21ksoftirqd/211:44:392
1932992414,7cyclictest0-21swapper/210:00:222
1932992412,10cyclictest24219-21cron08:45:022
1932992411,8cyclictest0-21swapper/207:15:122
1932992411,10cyclictest0-21swapper/208:00:182
1932992410,8cyclictest27-21ksoftirqd/210:59:322
190999245,4cyclictest151rcuc/010:18:320
190999245,15cyclictest0-21swapper/011:35:130
190999244,4cyclictest12-21ksoftirqd/008:30:010
190999243,9cyclictest12-21ksoftirqd/012:39:550
190999243,9cyclictest12-21ksoftirqd/012:05:130
1909992419,4cyclictest151rcuc/010:30:520
190999241,8cyclictest27002-21idleruntime11:50:160
1909992418,4cyclictest151rcuc/011:32:490
190999241,7cyclictest15951-21/usr/sbin/munin10:45:140
190999241,10cyclictest12-21ksoftirqd/012:25:160
194099234,13cyclictest0-21swapper/309:10:143
194099233,13cyclictest0-21swapper/311:30:113
194099232,14cyclictest0-21swapper/310:50:123
194099232,12cyclictest0-21swapper/311:05:153
193299234,11cyclictest131rcu_preempt07:40:012
193299233,8cyclictest27-21ksoftirqd/211:30:512
193299233,13cyclictest27-21ksoftirqd/209:41:192
193299232,5cyclictest27-21ksoftirqd/209:10:572
193299231,4cyclictest616-21dbus-daemon09:00:542
1932992312,3cyclictest27-21ksoftirqd/207:30:502
190999234,6cyclictest12-21ksoftirqd/010:00:300
190999234,5cyclictest12-21ksoftirqd/012:11:190
190999234,5cyclictest12-21ksoftirqd/012:00:120
190999234,5cyclictest12-21ksoftirqd/009:12:360
190999234,4cyclictest12-21ksoftirqd/011:40:280
190999234,4cyclictest12-21ksoftirqd/010:40:200
190999234,4cyclictest12-21ksoftirqd/010:08:400
190999234,4cyclictest12-21ksoftirqd/010:08:400
190999233,6cyclictest12-21ksoftirqd/010:52:480
190999233,6cyclictest12-21ksoftirqd/008:20:120
190999233,5cyclictest12-21ksoftirqd/012:21:060
190999233,4cyclictest12-21ksoftirqd/011:04:120
190999232,6cyclictest12-21ksoftirqd/010:20:120
190999232,5cyclictest12-21ksoftirqd/008:00:180
1909992317,4cyclictest14809-21apt-get08:35:110
190999231,6cyclictest12-21ksoftirqd/010:36:040
1909992316,3cyclictest2055-21apt-get07:10:120
190999231,5cyclictest10610-21ntp_192.168.11411:20:240
1909992315,3cyclictest21521-21apt-get09:50:110
194099223,8cyclictest0-21swapper/312:10:113
193299228,7cyclictest27-21ksoftirqd/208:30:312
1932992212,3cyclictest0-21swapper/207:25:142
1932992212,3cyclictest0-21swapper/207:25:142
190999225,4cyclictest12-21ksoftirqd/009:00:280
190999225,3cyclictest131rcu_preempt12:20:010
190999224,5cyclictest12-21ksoftirqd/009:15:230
190999224,4cyclictest12-21ksoftirqd/009:20:420
190999223,5cyclictest12-21ksoftirqd/010:58:460
190999223,4cyclictest12-21ksoftirqd/012:30:320
190999223,4cyclictest12-21ksoftirqd/011:45:240
190999223,4cyclictest12-21ksoftirqd/011:06:130
190999222,5cyclictest12-21ksoftirqd/008:25:180
190999222,4cyclictest12-21ksoftirqd/011:15:180
190999222,4cyclictest12-21ksoftirqd/011:15:170
190999221,15cyclictest839-21munin-node08:50:120
1940992113,5cyclictest28366irq/58-eth010:45:123
193299213,5cyclictest27-21ksoftirqd/209:10:012
190999213,5cyclictest12-21ksoftirqd/008:55:240
194099204,13cyclictest0-21swapper/309:25:143
194099203,11cyclictest0-21swapper/309:55:513
1940992014,4cyclictest0-21swapper/312:20:293
1940992011,7cyclictest0-21swapper/311:10:263
1940992011,5cyclictest28366irq/58-eth009:30:123
193299204,12cyclictest22346irq/40-dwc2_hso08:45:112
193299201,5cyclictest5475-21interrupts08:25:162
193299201,5cyclictest5475-21interrupts08:25:162
193299201,10cyclictest4228-21kthreadcore07:10:202
190999204,4cyclictest12-21ksoftirqd/008:10:110
190999203,5cyclictest12-21ksoftirqd/007:40:140
190999203,4cyclictest12-21ksoftirqd/009:05:240
190999203,4cyclictest12-21ksoftirqd/007:55:210
190999203,4cyclictest12-21ksoftirqd/007:55:210
190999201,6cyclictest24030-21cat08:40:300
190999201,3cyclictest151rcuc/008:30:240
190999201,3cyclictest0-21swapper/007:20:160
190999200,4cyclictest10294-21ntp_192.168.11407:15:240
194099197,9cyclictest0-21swapper/311:04:503
194099195,10cyclictest28366irq/58-eth009:45:123
194099193,11cyclictest0-21swapper/310:19:503
194099193,10cyclictest0-21swapper/309:50:133
194099192,5cyclictest0-21swapper/312:30:113
1940991912,5cyclictest0-21swapper/312:05:283
194099191,12cyclictest0-21swapper/307:20:143
190999193,4cyclictest12-21ksoftirqd/008:15:290
190999193,4cyclictest12-21ksoftirqd/007:25:170
190999193,4cyclictest12-21ksoftirqd/007:25:160
194099189,6cyclictest0-21swapper/310:32:033
194099185,9cyclictest0-21swapper/310:25:133
194099183,10cyclictest0-21swapper/311:15:133
194099183,10cyclictest0-21swapper/311:15:123
194099182,11cyclictest0-21swapper/310:58:513
194099182,10cyclictest0-21swapper/308:40:123
194099182,10cyclictest0-21swapper/308:15:143
194099182,10cyclictest0-21swapper/307:15:143
194099182,10cyclictest0-21swapper/307:10:123
194099181,6cyclictest0-21swapper/309:00:123
1940991810,6cyclictest28366irq/58-eth011:25:133
190999183,4cyclictest12-21ksoftirqd/007:50:160
190999181,6cyclictest23764-21kthreadcore07:30:220
194099179,6cyclictest0-21swapper/309:35:363
194099178,5cyclictest28366irq/58-eth010:35:133
194099173,9cyclictest0-21swapper/308:00:043
194099172,10cyclictest0-21swapper/308:35:133
194099171,10cyclictest0-21swapper/310:00:123
194099171,10cyclictest0-21swapper/308:50:143
194099161,10cyclictest0-21swapper/309:05:173
194099155,6cyclictest0-21swapper/308:55:133
194099153,8cyclictest0-21swapper/307:45:123
194099153,6cyclictest0-21swapper/307:25:143
194099153,6cyclictest0-21swapper/307:25:143
194099142,8cyclictest0-21swapper/307:35:123
194099141,7cyclictest0-21swapper/308:45:153
194099141,10cyclictest0-21swapper/308:25:013
190999147,2cyclictest12-21ksoftirqd/008:05:320
194099136,5cyclictest0-21swapper/307:30:293
194099133,6cyclictest0-21swapper/308:30:143
194099132,6cyclictest0-21swapper/308:10:093
194099131,8cyclictest0-21swapper/307:55:123
194099131,8cyclictest0-21swapper/307:55:113
194099131,6cyclictest0-21swapper/308:25:143
194099131,6cyclictest0-21swapper/308:25:133
194099131,10cyclictest0-21swapper/307:40:193
193299131,9cyclictest1962-21latency_hist07:10:012
190999131,9cyclictest1900-21cyclictest07:10:000
194099121,6cyclictest0-21swapper/308:10:013
194099111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional