You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 15:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Mar 20, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1893099662,55cyclictest0-21swapper/209:15:112
1893099631,50cyclictest0-21swapper/208:05:122
1893099574,35cyclictest20280-21ssh10:30:112
18939995537,11cyclictest28366irq/58-eth011:30:123
1891599551,35cyclictest16579-21ssh10:10:141
18939995335,12cyclictest28366irq/58-eth009:40:143
18915995134,12cyclictest0-21swapper/111:50:121
1893099491,29cyclictest12722-21ssh09:50:122
1890699496,6cyclictest131rcu_preempt12:15:120
1893999464,27cyclictest28366irq/58-eth009:10:133
1893999463,26cyclictest28366irq/58-eth010:05:173
1893999463,26cyclictest28366irq/58-eth010:05:163
1893099466,37cyclictest22346irq/40-dwc2_hso07:45:142
1893099462,38cyclictest0-21swapper/212:15:122
1893099461,40cyclictest0-21swapper/211:10:122
1893099461,40cyclictest0-21swapper/210:15:122
1893099441,36cyclictest0-21swapper/212:30:122
1893999433,24cyclictest28366irq/58-eth011:15:153
1893999433,24cyclictest28366irq/58-eth011:15:153
1893999432,26cyclictest28366irq/58-eth011:40:103
1893099430,37cyclictest0-21swapper/208:20:122
1891599431,30cyclictest28149-21ssh11:10:131
1890699433,31cyclictest20251-21ssh12:35:120
1893099421,36cyclictest0-21swapper/209:30:132
1893099421,34cyclictest0-21swapper/210:40:112
1893999419,20cyclictest28366irq/58-eth009:20:123
1893099411,35cyclictest20231-21apt-get12:35:122
1893099410,35cyclictest0-21swapper/211:40:122
1893099401,35cyclictest26063-21kthreadcore07:15:192
1893099400,36cyclictest0-21swapper/210:45:112
1893099400,34cyclictest8711-21ssh11:35:142
18915994022,7cyclictest21-21ksoftirqd/112:05:081
1893099391,34cyclictest0-21swapper/211:55:112
1893099390,34cyclictest0-21swapper/211:20:142
1893099390,34cyclictest0-21swapper/209:55:122
1893099382,33cyclictest22346irq/40-dwc2_hso08:50:122
1893999371,20cyclictest28366irq/58-eth009:25:123
1893099371,18cyclictest9961-21ssh11:00:112
1893099361,31cyclictest15445-21apt-get07:40:122
1893099361,31cyclictest13962-21/usr/sbin/munin07:35:222
1893099361,30cyclictest23018-21ssh11:25:132
1893099361,27cyclictest29978-21fschecks_time07:55:162
1893099360,33cyclictest0-21swapper/207:20:112
1893099360,29cyclictest0-21swapper/207:25:122
1893099360,27cyclictest13904-21cut08:10:202
1893099360,24cyclictest4451-21diskmemload10:27:022
1893099360,23cyclictest0-21swapper/209:25:102
1891599364,28cyclictest0-21swapper/108:40:111
1891599362,25cyclictest8776-21ssh09:30:141
1893099353,9cyclictest0-21swapper/210:54:582
1893099352,29cyclictest30820-21apt-get10:00:122
1893099351,31cyclictest27460-21apt-get09:00:052
1893099351,31cyclictest2037-21apt-get12:25:142
1893099351,29cyclictest22566-21apt-get08:55:142
1893099351,29cyclictest19115-21apt-get11:05:112
1893099351,29cyclictest0-21swapper/208:40:112
1893099351,24cyclictest32060-21ssh11:30:142
1893099350,31cyclictest2316-21entropy08:00:152
1893099350,31cyclictest2316-21entropy08:00:152
1893999343,20cyclictest0-21swapper/310:30:113
1893099341,30cyclictest6065-21idleruntime07:30:162
1893099341,29cyclictest16643-21df08:15:142
1893099341,26cyclictest7879-21users08:35:302
1893099340,32cyclictest31347-21ssh09:24:362
1893099331,29cyclictest30849-21ssh12:21:132
1893099330,31cyclictest24412-21ssh09:37:232
1893099330,31cyclictest24412-21ssh09:37:232
1893099330,30cyclictest10601-21ssh10:24:422
1893099330,29cyclictest11994-21ssh11:54:512
18939993210,18cyclictest0-21swapper/312:30:123
1893099321,29cyclictest7592-21ssh10:57:412
1893099321,28cyclictest30861-21timerwakeupswit11:45:282
1893099321,28cyclictest30861-21timerwakeupswit11:45:282
1893099321,28cyclictest28685-21kworker/u8:212:02:572
1893099321,28cyclictest11216-21ssh11:17:222
1893099321,28cyclictest11216-21ssh11:17:212
1893099321,27cyclictest4729-21rm09:10:062
1893099320,29cyclictest25244-21cut07:50:132
1891599323,21cyclictest4451-21diskmemload11:15:151
1891599323,21cyclictest4451-21diskmemload11:15:141
18939993114,10cyclictest0-21swapper/311:55:123
1893099311,28cyclictest9113-21ssh09:46:142
1893099311,27cyclictest30576-21ssh10:35:202
1893099311,27cyclictest30576-21ssh10:35:202
1893099311,27cyclictest21594-21sshd10:10:442
1893099311,26cyclictest31596-21tr12:05:172
1893099311,26cyclictest11647-21awk10:05:322
1893099311,26cyclictest11647-21awk10:05:322
1893099310,27cyclictest31073-21diskstats08:30:152
1893099310,27cyclictest26561-21sed08:25:172
1893099310,27cyclictest20906-21kthreadcore07:10:202
1891599312,23cyclictest22356-21ssh11:24:441
1891599312,23cyclictest1397-21ssh12:24:441
1890699313,7cyclictest131rcu_preempt08:45:110
1893999302,17cyclictest28366irq/58-eth011:00:123
1893099301,26cyclictest26960-21ssh09:40:072
1891599302,21cyclictest26729-21ssh11:45:071
1891599302,21cyclictest26729-21ssh11:45:061
1893999293,16cyclictest28366irq/58-eth010:15:123
1893099291,25cyclictest12158-21ssh12:10:342
1893099290,26cyclictest17503-21sshd08:49:312
1891599292,21cyclictest399-21ssh10:54:581
1890699291,26cyclictest17951-21ssh09:35:140
1890699291,26cyclictest17951-21ssh09:35:130
18939992811,11cyclictest28366irq/58-eth009:15:123
1893099281,24cyclictest18892-21cyclictest07:10:022
1891599283,11cyclictest23019-21ssh11:25:141
18939992713,10cyclictest0-21swapper/311:50:133
1893099272,22cyclictest22346irq/40-dwc2_hso09:05:152
1891599274,15cyclictest633-21Async12:11:531
1891599272,19cyclictest31340-21ssh09:24:371
1891599272,19cyclictest26190-21ssh11:44:451
1890699276,18cyclictest12-21ksoftirqd/011:50:130
1890699272,19cyclictest25437-21ssh12:20:110
1890699272,19cyclictest23749-21ssh10:49:580
1890699271,24cyclictest883-21ssh10:55:130
1890699271,21cyclictest32075-21ssh09:25:120
1890699271,20cyclictest21691-21ssh12:00:130
1893999263,15cyclictest0-21swapper/312:24:443
1893999262,17cyclictest0-21swapper/311:20:033
18906992617,6cyclictest12-21ksoftirqd/010:50:120
18906992616,5cyclictest131rcu_preempt08:50:130
1893999254,14cyclictest28366irq/58-eth010:35:143
1893999254,14cyclictest28366irq/58-eth010:35:143
1893999252,12cyclictest28366irq/58-eth011:25:133
1891599251,19cyclictest13883-21ssh09:15:111
1890699253,21cyclictest12-21ksoftirqd/011:10:110
1890699252,20cyclictest29303-21ssh10:35:140
1890699252,20cyclictest29303-21ssh10:35:140
18906992513,9cyclictest6769-21cron08:05:010
18906992513,9cyclictest6769-21cron08:05:000
1893999243,14cyclictest0-21swapper/310:49:583
1893999243,13cyclictest0-21swapper/312:00:053
1893999242,14cyclictest0-21swapper/311:45:073
1893999242,14cyclictest0-21swapper/311:45:073
18939992410,9cyclictest28366irq/58-eth009:35:143
18939992410,9cyclictest28366irq/58-eth009:35:143
1891599243,14cyclictest32044-21ssh11:30:131
1891599242,16cyclictest7257-21ssh12:10:141
1891599241,18cyclictest2237-21sh10:20:131
1890699245,17cyclictest12-21ksoftirqd/008:55:140
18906992411,10cyclictest12-21ksoftirqd/008:20:120
18939992312,7cyclictest0-21swapper/309:55:123
1891599231,19cyclictest29279-21ssh10:35:121
1891599231,19cyclictest29279-21ssh10:35:111
1891599231,18cyclictest27101-21ssh09:40:151
1891599231,16cyclictest7242-21ls08:05:151
1890699236,3cyclictest151rcuc/011:25:210
1890699235,15cyclictest12-21ksoftirqd/008:10:110
1890699231,4cyclictest131rcu_preempt07:55:130
1890699231,16cyclictest9717-21ssh11:15:350
1890699231,16cyclictest9717-21ssh11:15:350
1890699231,16cyclictest30845-21ssh10:00:140
1890699231,12cyclictest10826-21kthreadcore11:00:170
1893999224,11cyclictest28366irq/58-eth012:10:123
1891599222,15cyclictest12513-21apt-get11:55:121
18915992216,4cyclictest17330-21sendmail11:40:011
1891599221,17cyclictest11053-21ssh12:30:121
1890699222,4cyclictest131rcu_preempt09:20:130
1890699222,17cyclictest4451-21diskmemload11:05:130
1890699222,17cyclictest26820-21ssh11:45:130
1890699222,17cyclictest26820-21ssh11:45:120
1890699222,17cyclictest12706-21ssh09:50:100
18906992212,7cyclictest18668-21cat11:20:320
18906992210,9cyclictest30406-21cron10:00:010
18939992115,2cyclictest0-21swapper/309:45:123
1893999211,12cyclictest0-21swapper/310:20:113
1891599219,9cyclictest15293-21sed10:25:261
1891599212,14cyclictest20286-21ssh10:30:121
1891599212,13cyclictest21-21ksoftirqd/110:15:121
1891599211,2cyclictest131rcu_preempt08:30:121
1891599211,17cyclictest7550-21ssh10:05:171
1891599211,17cyclictest7550-21ssh10:05:171
18915992110,8cyclictest1-21systemd12:35:141
1891599210,18cyclictest9737-21ls11:00:021
1891599210,16cyclictest9977-21ssh11:00:121
1890699219,9cyclictest7932-21ps10:05:210
1890699219,9cyclictest7932-21ps10:05:210
18906992114,5cyclictest0-21swapper/011:55:130
18906992111,5cyclictest0-21swapper/009:05:140
1890699211,14cyclictest25122-21ssh10:30:320
18906992110,8cyclictest17827-21df11:40:130
1893999209,8cyclictest0-21swapper/311:08:413
1893999203,12cyclictest0-21swapper/310:58:443
1893999203,11cyclictest0-21swapper/310:10:143
18939992010,5cyclictest0-21swapper/311:10:123
1891599209,3cyclictest131rcu_preempt07:45:011
1891599204,5cyclictest21-21ksoftirqd/109:25:281
1891599202,14cyclictest21188-21apt-get12:00:051
18915992013,5cyclictest0-21swapper/109:50:171
18915992012,5cyclictest0-21swapper/108:50:231
18915992011,6cyclictest4451-21diskmemload10:00:191
1891599200,6cyclictest25087-21ssh09:38:111
1891599200,6cyclictest25087-21ssh09:38:101
1891599200,13cyclictest23968-21ls10:50:021
1890699205,9cyclictest0-21swapper/012:10:140
1890699203,15cyclictest3649-21ssh09:45:120
1890699202,16cyclictest0-21swapper/010:41:440
1890699202,12cyclictest16435-21ssh09:33:440
1890699201,4cyclictest131rcu_preempt09:17:500
1890699201,4cyclictest131rcu_preempt07:20:140
1890699201,13cyclictest2234-21ssh10:20:110
1893999198,8cyclictest0-21swapper/311:37:343
1893999193,11cyclictest0-21swapper/310:50:443
1893999193,11cyclictest0-21swapper/309:30:443
1893999192,11cyclictest0-21swapper/310:40:443
18939991913,4cyclictest0-21swapper/309:05:133
1891599193,4cyclictest21-21ksoftirqd/109:11:241
1891599191,16cyclictest1383-21irqcore07:25:161
1890699192,14cyclictest4840-21apt-get09:10:130
1890699191,7cyclictest12349-21grep11:35:250
18906991911,5cyclictest344-21kthreadcore11:30:200
1890699191,13cyclictest11271-21ssh10:25:120
1890699191,12cyclictest25795-21grep10:15:160
18906991910,7cyclictest15210-21awk07:40:010
1890699190,16cyclictest12813-21ls08:45:020
1890699190,10cyclictest712-21cron08:20:010
1893999189,6cyclictest0-21swapper/312:29:253
1893999189,5cyclictest28366irq/58-eth010:00:143
1893999183,9cyclictest0-21swapper/308:30:143
1893999182,9cyclictest0-21swapper/308:50:143
1893999182,9cyclictest0-21swapper/308:45:133
1893999182,12cyclictest0-21swapper/312:15:123
1893999182,11cyclictest0-21swapper/310:28:443
1893999182,10cyclictest0-21swapper/309:50:133
1893999182,10cyclictest0-21swapper/309:00:053
1893999182,10cyclictest0-21swapper/307:25:133
1893999182,10cyclictest0-21swapper/307:20:123
1893999182,10cyclictest0-21swapper/307:15:133
1893999182,10cyclictest0-21swapper/307:10:133
18939991812,4cyclictest0-21swapper/307:55:133
1893999181,13cyclictest0-21swapper/307:30:273
1893999181,11cyclictest0-21swapper/308:40:153
1891599189,6cyclictest9979-21wc10:40:251
1891599189,6cyclictest30208-21kthreadcore07:20:181
1891599186,7cyclictest21-21ksoftirqd/107:35:111
1891599182,13cyclictest18657-21apt-get11:05:021
1891599182,12cyclictest23941-21ssh12:18:431
18915991811,5cyclictest0-21swapper/108:20:181
1890699187,3cyclictest131rcu_preempt10:10:250
1890699186,9cyclictest31370-21grep09:00:280
1890699184,11cyclictest21343-21kthreadcore07:10:200
1890699183,5cyclictest27827-21kthreadcore09:40:190
1890699182,12cyclictest30701-21df_inode12:05:120
18906991812,4cyclictest0-21swapper/012:30:100
1890699181,14cyclictest1999-21ntp_states08:30:240
1893999179,6cyclictest0-21swapper/312:35:123
1893999179,6cyclictest0-21swapper/312:06:193
1893999173,6cyclictest28366irq/58-eth008:00:133
1893999173,6cyclictest28366irq/58-eth008:00:133
1893999172,9cyclictest0-21swapper/308:35:143
1893999172,9cyclictest0-21swapper/308:25:133
1893999172,9cyclictest0-21swapper/307:40:143
1893999172,9cyclictest0-21swapper/307:35:133
18939991711,4cyclictest0-21swapper/307:45:283
1893999171,13cyclictest0-21swapper/308:05:213
18939991711,2cyclictest0-21swapper/308:15:123
1893999171,10cyclictest0-21swapper/307:50:253
1891599179,5cyclictest5521-21wc08:00:231
1891599179,5cyclictest5521-21wc08:00:231
1891599179,5cyclictest22641-21kthreadcore07:45:211
1891599173,3cyclictest131rcu_preempt09:45:191
1891599172,12cyclictest9628-21grep07:30:271
1891599172,10cyclictest0-21swapper/108:15:141
1891599172,10cyclictest0-21swapper/107:50:121
1891599171,4cyclictest0-21swapper/112:27:371
1891599171,13cyclictest25760-21tr08:55:221
1891599171,13cyclictest15831-21kthreadcore08:45:191
1891599171,12cyclictest21769-21ssh09:55:121
18915991710,5cyclictest0-21swapper/109:00:281
1890699179,6cyclictest0-21swapper/007:50:240
1890699178,6cyclictest2947-21sh12:25:190
1890699174,10cyclictest8042-21latency_hist08:40:020
1890699171,3cyclictest131rcu_preempt07:40:140
1890699171,13cyclictest22073-21tr07:45:210
18906991710,5cyclictest1-21systemd07:19:180
18906991710,4cyclictest26277-21cut08:25:150
1891599168,6cyclictest27674-21kthreadcore08:25:181
1891599167,6cyclictest32316-21cut07:55:211
1891599161,7cyclictest19224-21fschecks_time07:10:151
1891599161,13cyclictest13360-21kthreadcore08:10:191
1890699165,9cyclictest9322-21wc07:30:250
1890699162,11cyclictest10874-21grep08:05:280
1893999152,9cyclictest0-21swapper/308:20:133
1891599151,11cyclictest404-21irqrtprio09:05:181
1891599151,11cyclictest19022-21latency_hist07:10:011
1891599151,10cyclictest24155-21sed07:15:151
1890699151,11cyclictest1129-21fschecks_count07:25:150
1890699151,10cyclictest18943-21latency_hist07:10:020
1893999149,3cyclictest0-21swapper/308:55:143
1893999133,5cyclictest0-21swapper/308:10:113
1893999101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional