You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-15 - 16:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat Aug 15, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31366999028,47cyclictest131rcu_preempt11:20:132
3136699607,10cyclictest131rcu_preempt10:20:142
3135799584,50cyclictest18165-21ssh10:40:131
3136699563,29cyclictest19848-21ssh10:05:132
31357995421,25cyclictest21-21ksoftirqd/111:50:151
3135799501,38cyclictest23472-21ssh10:25:111
31346995027,12cyclictest131rcu_preempt07:30:130
31357994734,7cyclictest7197-21apt-get09:40:151
3134699471,36cyclictest7238-21ssh11:10:140
3138199463,31cyclictest35966irq/58-eth010:10:123
31357994637,5cyclictest9996-21apt-get07:55:131
3135799462,34cyclictest16222-21ssh11:15:141
3134699462,5cyclictest131rcu_preempt10:15:120
3138199454,25cyclictest35966irq/58-eth009:35:123
31357994538,3cyclictest333-21apt-get07:45:141
3134699451,37cyclictest28869-21ssh10:10:120
3134699451,35cyclictest10871-21ssh10:00:160
31357994418,16cyclictest3942-21grep08:55:201
3135799441,32cyclictest27515-21ssh12:15:131
3138199435,26cyclictest35966irq/58-eth009:15:123
31357994337,2cyclictest19665-21apt-get08:05:151
31357994332,7cyclictest32673-21apt-get12:00:151
3135799432,32cyclictest21614-21ssh11:00:131
31357994317,15cyclictest2771-21cut08:20:201
31357994317,15cyclictest2771-21cut08:20:191
31357994316,8cyclictest27107-21idleruntime-cro12:15:021
31357994314,11cyclictest3540-21ssh10:50:031
3134699434,34cyclictest14535-21ssh10:20:150
3134699433,33cyclictest9331-21ssh12:05:130
31357994215,11cyclictest27375-21ssh12:30:581
3134699422,31cyclictest7218-21sh09:40:160
3138199412,24cyclictest35966irq/58-eth009:20:133
3138199412,24cyclictest35966irq/58-eth009:20:133
3135799412,28cyclictest29009-21ssh11:40:131
31357994116,9cyclictest16745-21sed09:45:191
31357994115,12cyclictest28731-21sed11:40:021
31357994115,12cyclictest28731-21sed11:40:011
3134699415,28cyclictest25289-21ssh11:20:140
31346994122,7cyclictest131rcu_preempt11:05:120
3134699411,32cyclictest31173-21ssh12:35:150
31357994031,5cyclictest23156-21apt-get07:35:131
31357994019,17cyclictest0-21swapper/108:45:131
31357994017,13cyclictest1657-21ls08:20:021
31357994016,18cyclictest5615-21ssh11:45:151
31357994016,10cyclictest23617-21kthreadcore08:40:211
31357994015,11cyclictest23055-21ssh10:24:591
31357994011,22cyclictest15003-21sed10:00:311
3138199392,19cyclictest35966irq/58-eth011:25:143
3138199391,31cyclictest35966irq/58-eth009:30:143
31357993932,3cyclictest28012-21apt-get07:40:121
31357993932,3cyclictest28012-21apt-get07:40:111
31357993916,10cyclictest2814-21ls08:55:011
31357993915,10cyclictest11208-21ssh10:16:331
31357993914,8cyclictest1326-21sh10:47:381
31357993914,8cyclictest12711-21ssh12:05:221
31357993914,11cyclictest26906-21sh10:08:081
31357993913,9cyclictest27794-21sed11:55:301
31357993913,11cyclictest31123-21/usr/sbin/munin12:35:191
31357993913,11cyclictest14055-21cut11:30:241
31357993913,11cyclictest14055-21cut11:30:231
31357993912,11cyclictest31427-21date07:10:021
3134699391,31cyclictest32683-21ssh12:00:130
3138199383,26cyclictest35966irq/58-eth011:05:113
3136699386,8cyclictest0-21swapper/209:25:152
31357993819,14cyclictest0-21swapper/109:55:131
31357993815,9cyclictest4194-21ssh11:06:521
31357993811,10cyclictest18947-21cat08:00:281
3134699383,23cyclictest21605-21ssh11:00:120
31346993818,11cyclictest131rcu_preempt08:20:140
31346993818,11cyclictest131rcu_preempt08:20:140
31357993717,17cyclictest0-21swapper/112:20:141
31357993714,7cyclictest17222-21diskmemload09:26:001
31357993714,10cyclictest30918-21ssh09:15:291
31357993713,10cyclictest6173-21cut09:20:221
31357993713,10cyclictest6173-21cut09:20:211
31357993712,8cyclictest12550-21kworker/u8:011:20:241
3138199363,19cyclictest35966irq/58-eth011:45:153
31357993613,11cyclictest25966-21cut09:50:191
31357993613,11cyclictest25966-21cut09:50:181
31357993613,10cyclictest9661-21cut08:25:231
31357993612,8cyclictest4938-21cut07:15:211
31357993612,11cyclictest22207-21irqstats09:30:201
31357993612,10cyclictest11481-21fschecks_count08:30:161
31357993612,10cyclictest10918-21ssh10:35:201
3136699353,15cyclictest261rcuc/210:50:132
3136699352,28cyclictest18203-21ssh10:03:312
3136699352,25cyclictest30802-21ntp_states10:45:262
3135799357,7cyclictest0-21swapper/110:55:291
31357993515,12cyclictest1640-21ssh09:35:231
31357993514,9cyclictest7469-21sh11:26:191
31357993513,8cyclictest18036-21ssh09:10:181
31381993422,8cyclictest0-21swapper/312:35:123
31357993425,6cyclictest8666-21apt-get07:20:121
31357993418,12cyclictest0-21swapper/110:30:141
31357993414,7cyclictest0-21swapper/112:29:171
31357993412,10cyclictest11283-21awk09:00:231
31357993411,9cyclictest7254-21ssh11:10:151
3138199334,24cyclictest35966irq/58-eth009:45:123
31381993318,5cyclictest35966irq/58-eth010:05:123
31381993316,9cyclictest35966irq/58-eth011:30:153
31381993316,9cyclictest35966irq/58-eth011:30:143
3138199322,20cyclictest35966irq/58-eth008:55:123
3136699321,20cyclictest31157-21sh12:35:142
3136699321,20cyclictest27-21ksoftirqd/208:40:152
31357993220,9cyclictest2489-21grep07:10:241
31357993215,14cyclictest13515-21/usr/sbin/munin07:25:241
31357993212,6cyclictest0-21swapper/109:05:181
31357993211,8cyclictest9069-21unixbench_singl07:50:291
3134699322,23cyclictest19863-21ssh10:05:140
31346993214,5cyclictest131rcu_preempt12:10:130
3138199313,16cyclictest35966irq/58-eth009:40:153
3136699312,23cyclictest26693-21ssh09:15:122
31366993115,12cyclictest30987-1kworker/u9:209:40:162
3135799318,8cyclictest0-21swapper/108:35:291
3135799318,10cyclictest21825-21ntpq07:30:251
3135799316,11cyclictest28842-21/usr/sbin/munin10:10:161
3138199301,17cyclictest35966irq/58-eth011:20:143
3136699304,21cyclictest27-21ksoftirqd/209:32:302
3136699303,19cyclictest27346irq/40-dwc2_hso08:05:132
3136699302,8cyclictest30660-21ssh11:05:132
3135799303,17cyclictest21-21ksoftirqd/108:10:131
3138199295,12cyclictest35966irq/58-eth010:30:133
3138199292,19cyclictest35966irq/58-eth010:00:173
3136699292,20cyclictest27346irq/40-dwc2_hso07:30:132
3136699292,20cyclictest0-21swapper/207:35:142
3136699290,22cyclictest9153-21ssh10:35:132
3134699292,20cyclictest17654-21ssh09:10:150
3136699284,19cyclictest27346irq/40-dwc2_hso11:35:142
3136699284,19cyclictest27346irq/40-dwc2_hso11:35:132
3136699283,11cyclictest0-21swapper/209:58:312
3136699282,17cyclictest27346irq/40-dwc2_hso07:20:142
31366992818,4cyclictest27-21ksoftirqd/212:33:352
31366992818,4cyclictest27-21ksoftirqd/212:19:402
31366992818,2cyclictest27-21ksoftirqd/210:40:212
3134699286,7cyclictest12-21ksoftirqd/007:15:140
31346992810,8cyclictest31332-21cyclictest11:35:130
31346992810,8cyclictest31332-21cyclictest11:35:120
31381992716,7cyclictest35966irq/58-eth010:55:133
3136699278,8cyclictest27-21ksoftirqd/212:05:132
3136699273,21cyclictest5648-21sh10:15:162
3136699271,9cyclictest131rcu_preempt12:10:202
31366992711,13cyclictest0-21swapper/208:55:262
3138199263,16cyclictest0-21swapper/311:00:323
31381992612,9cyclictest35966irq/58-eth012:00:143
3136699267,7cyclictest4636-21cut09:20:202
3136699267,7cyclictest4636-21cut09:20:192
3136699262,3cyclictest131rcu_preempt09:50:222
3136699262,3cyclictest131rcu_preempt09:50:212
31366992617,2cyclictest27-21ksoftirqd/211:55:112
31366992616,4cyclictest27-21ksoftirqd/211:30:022
31366992616,4cyclictest27-21ksoftirqd/209:45:042
31366992616,4cyclictest27-21ksoftirqd/208:25:322
31366992614,3cyclictest13082-21/usr/sbin/munin12:25:232
3134699261,9cyclictest12-21ksoftirqd/007:40:120
3134699261,9cyclictest12-21ksoftirqd/007:40:110
3134699261,21cyclictest151rcuc/010:50:140
3136699251,9cyclictest16418-21timerwakeupswit09:05:282
3136699251,16cyclictest0-21swapper/210:10:292
31366992511,12cyclictest27550-21sed10:25:292
3134699259,3cyclictest131rcu_preempt10:45:140
31346992511,4cyclictest12-21ksoftirqd/011:30:160
31346992511,4cyclictest12-21ksoftirqd/011:30:160
3138199245,10cyclictest0-21swapper/308:10:133
3136699245,4cyclictest0-21swapper/209:00:162
3136699244,3cyclictest27-21ksoftirqd/212:00:142
31366992415,4cyclictest27-21ksoftirqd/211:31:142
31366992415,4cyclictest27-21ksoftirqd/211:31:132
3136699241,4cyclictest0-21swapper/207:55:122
3136699241,2cyclictest131rcu_preempt09:10:152
3136699241,2cyclictest131rcu_preempt08:45:222
3136699241,13cyclictest131rcu_preempt07:50:302
31366992411,10cyclictest7612-21irqcore11:10:192
3134699242,7cyclictest131rcu_preempt11:44:320
31346992410,7cyclictest12-21ksoftirqd/012:20:150
3138199239,9cyclictest0-21swapper/310:17:563
3138199232,10cyclictest0-21swapper/310:45:253
31381992313,6cyclictest35966irq/58-eth011:10:153
3136699238,4cyclictest131rcu_preempt08:50:222
3136699234,9cyclictest0-21swapper/208:20:212
3136699234,9cyclictest0-21swapper/208:20:202
3136699233,16cyclictest27346irq/40-dwc2_hso07:45:142
3136699232,6cyclictest0-21swapper/211:44:322
3136699232,16cyclictest0-21swapper/211:18:322
3136699231,8cyclictest131rcu_preempt08:30:152
3136699231,7cyclictest995-21grep09:35:222
3136699231,2cyclictest131rcu_preempt11:45:302
31366992312,9cyclictest7148-21ssh10:33:042
3136699230,2cyclictest131rcu_preempt11:00:192
3134699236,3cyclictest131rcu_preempt12:15:140
3134699233,8cyclictest131rcu_preempt08:35:130
3134699232,17cyclictest5605-21ssh11:45:140
3134699231,17cyclictest8821-21cut07:20:170
3138199222,13cyclictest35966irq/58-eth011:15:123
3136699221,3cyclictest131rcu_preempt11:54:142
3136699221,2cyclictest131rcu_preempt12:20:392
31366992212,4cyclictest0-21swapper/210:55:272
3134699221,11cyclictest23617-21ssh11:55:130
3138199213,13cyclictest0-21swapper/309:25:153
3136699215,8cyclictest0-21swapper/207:45:012
3136699215,8cyclictest0-21swapper/207:45:012
3136699211,7cyclictest131rcu_preempt07:10:292
31366992110,4cyclictest4944-21kthreadcore07:15:202
3134699213,11cyclictest13105-21apt-get12:25:130
3134699211,18cyclictest10583-21ls11:30:010
3134699211,17cyclictest30349-21ls08:50:000
3134699210,16cyclictest16301-21ls11:15:140
3138199209,8cyclictest0-21swapper/310:40:303
3138199209,6cyclictest0-21swapper/312:13:023
31381992012,6cyclictest0-21swapper/312:15:153
31381992010,8cyclictest0-21swapper/312:32:253
31381992010,8cyclictest0-21swapper/310:35:163
3136699207,10cyclictest18084-21kthreadcore08:00:222
3136699201,2cyclictest131rcu_preempt08:15:222
31366992012,6cyclictest18136-21cron07:30:012
3136699201,12cyclictest16134-21apt-get08:35:132
31366992010,8cyclictest0-21swapper/207:10:002
31366992010,7cyclictest29038-21cron08:15:022
3134699209,8cyclictest25275-21apt-get09:50:140
3134699209,8cyclictest25275-21apt-get09:50:130
3134699207,3cyclictest131rcu_preempt09:00:150
3134699204,13cyclictest22184-21ssh12:30:120
3134699203,3cyclictest131rcu_preempt10:30:140
3134699201,9cyclictest131rcu_preempt10:42:150
3138199198,8cyclictest0-21swapper/312:08:253
3138199193,11cyclictest35966irq/58-eth009:50:153
3138199193,11cyclictest35966irq/58-eth009:50:153
3138199192,13cyclictest0-21swapper/311:50:253
3138199192,11cyclictest0-21swapper/309:05:143
3138199191,13cyclictest35966irq/58-eth012:25:143
3138199191,12cyclictest0-21swapper/310:20:303
3138199191,11cyclictest0-21swapper/307:45:283
3134699199,8cyclictest0-21swapper/008:50:150
3134699199,7cyclictest1809-21ntp_kernel_pll_09:35:250
3134699194,5cyclictest0-21swapper/008:30:230
3134699191,6cyclictest25623-21kthreadcore08:10:200
3134699191,2cyclictest131rcu_preempt09:20:140
3134699191,2cyclictest131rcu_preempt09:20:140
31346991911,6cyclictest1-21systemd09:45:150
3134699191,15cyclictest1832-21ssh09:55:140
31346991910,7cyclictest8506-21memory07:50:240
31346991910,7cyclictest21523-21sh09:30:160
3134699190,12cyclictest23456-21find10:25:120
3138199189,6cyclictest0-21swapper/311:35:303
3138199189,6cyclictest0-21swapper/311:35:303
3138199183,9cyclictest0-21swapper/309:10:153
3138199182,10cyclictest0-21swapper/307:50:153
3138199181,12cyclictest0-21swapper/310:30:013
31381991810,3cyclictest36066irq/59-eth012:20:153
3134699184,11cyclictest10198-21switchtime08:25:280
3134699184,11cyclictest0-21swapper/009:15:110
3134699182,2cyclictest131rcu_preempt08:05:140
3134699182,11cyclictest0-21swapper/007:25:050
3134699181,5cyclictest13148-21irqstats09:25:210
3134699181,13cyclictest18227-21ntp_192.168.11411:50:250
3138199179,6cyclictest0-21swapper/311:40:093
3138199178,6cyclictest0-21swapper/311:55:273
3138199172,11cyclictest0-21swapper/309:55:143
3138199172,10cyclictest0-21swapper/308:40:153
3138199172,10cyclictest0-21swapper/308:25:143
3138199172,10cyclictest0-21swapper/308:00:143
3138199171,12cyclictest0-21swapper/310:50:183
3138199171,12cyclictest0-21swapper/307:40:163
3138199171,12cyclictest0-21swapper/307:40:153
3134699178,6cyclictest3614-21munin-plugin-st07:15:020
3134699174,4cyclictest9243-21ssh10:35:150
3134699172,10cyclictest0-21swapper/007:45:140
3134699172,10cyclictest0-21swapper/007:35:150
3134699171,3cyclictest131rcu_preempt08:40:150
31346991712,3cyclictest0-21swapper/008:55:230
3134699171,13cyclictest31228-21tr08:15:210
3134699171,13cyclictest12741-21diskstats09:05:140
3134699171,13cyclictest12652-21ssh10:55:120
31346991710,5cyclictest18462-21sshd08:00:270
3134699170,12cyclictest12-21ksoftirqd/007:55:120
3138199162,9cyclictest0-21swapper/307:25:053
3138199152,10cyclictest0-21swapper/307:10:143
3138199151,9cyclictest0-21swapper/307:55:123
3138199151,8cyclictest0-21swapper/307:35:283
3138199151,11cyclictest0-21swapper/308:35:203
3138199141,8cyclictest0-21swapper/307:30:183
3134699141,9cyclictest31371-21latency_hist07:10:000
3138199138,3cyclictest0-21swapper/307:20:263
3138199138,3cyclictest0-21swapper/307:18:333
3138199132,7cyclictest0-21swapper/308:20:153
3138199132,7cyclictest0-21swapper/308:20:143
3138199127,3cyclictest0-21swapper/308:50:293
3138199121,9cyclictest0-21swapper/308:15:223
3138199121,8cyclictest0-21swapper/308:45:283
3138199121,7cyclictest0-21swapper/308:05:153
3138199111,7cyclictest0-21swapper/308:30:193
3138199111,5cyclictest0-21swapper/309:00:123
3138199101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional