You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-01 - 18:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Fri May 01, 2026 12:44:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1189321100,3sleep30-21swapper/312:07:313
1189321100,3sleep30-21swapper/312:07:313
1189321100,3sleep30-21swapper/312:07:313
770721090,3sleep30-21swapper/307:37:253
838821070,2sleep30-21swapper/309:07:143
838821070,2sleep30-21swapper/309:07:143
41862990,3chrt0-21swapper/311:47:293
41862990,3chrt0-21swapper/311:47:293
132072830,3chrt0-21swapper/209:17:342
132072830,3chrt0-21swapper/209:17:342
132072830,3chrt0-21swapper/209:17:342
23112740,2sleep10-21swapper/107:22:291
23112740,2sleep10-21swapper/107:22:291
176702720,2chrt17669-21/usr/sbin/munin12:22:321
176702720,2chrt17669-21/usr/sbin/munin12:22:321
176702720,2chrt17669-21/usr/sbin/munin12:22:321
298892700,3sleep30-21swapper/311:27:353
298892700,3sleep30-21swapper/311:27:353
298892700,3sleep30-21swapper/311:27:353
326422690,1sleep00-21swapper/008:42:340
326422690,1sleep00-21swapper/008:42:340
326422690,1sleep00-21swapper/008:42:340
56942680,2sleep30-21swapper/308:57:333
56942680,2sleep30-21swapper/308:57:333
150012660,2sleep0171ktimers/009:22:300
150012660,2sleep0171ktimers/009:22:300
150012660,2sleep0171ktimers/009:22:300
202572650,2sleep30-21swapper/311:02:353
202572650,2sleep30-21swapper/311:02:353
185732640,3chrt0-21swapper/212:27:162
185732640,3chrt0-21swapper/212:27:162
185732640,3chrt0-21swapper/212:27:162
266162580,3sleep20-21swapper/211:22:162
266162580,3sleep20-21swapper/211:22:162
35482570,3sleep30-21swapper/311:47:173
35482570,3sleep30-21swapper/311:47:173
231662560,2chrt23165-21sendmail_mailtr08:17:340
231662560,2chrt23165-21sendmail_mailtr08:17:340
231662560,2chrt23165-21sendmail_mailtr08:17:340
65402530,1sleep00-21swapper/010:27:260
65402530,1sleep00-21swapper/010:27:260
299042510,2sleep10-21swapper/108:37:261
299042510,2sleep10-21swapper/108:37:261
292412490,1sleep00-21swapper/011:27:280
292412490,1sleep00-21swapper/011:27:280
292412490,1sleep00-21swapper/011:27:280
135322490,3sleep30-21swapper/310:47:193
135322490,3sleep30-21swapper/310:47:193
19992460,2chrt0-21swapper/207:22:252
19992460,2chrt0-21swapper/207:22:252
17042460,2sleep01709-21unixbench_multi10:12:350
17042460,2sleep01709-21unixbench_multi10:12:350
237122430,3sleep022446-21/usr/sbin/munin12:37:350
104362430,4sleep20-21swapper/207:42:352
104362430,4sleep20-21swapper/207:42:352
274422410,4chrt0-21swapper/111:22:311
274422410,4chrt0-21swapper/111:22:311
248572410,2sleep20-21swapper/211:17:212
203952390,2sleep20-21swapper/209:37:282
203952390,2sleep20-21swapper/209:37:282
85092380,2sleep10-21swapper/110:32:261
30418993829,7cyclictest31772-21netstat10:07:301
30418993829,7cyclictest31772-21netstat10:07:301
30418993828,6cyclictest17758-21sh09:32:131
30418993828,6cyclictest17758-21sh09:32:131
30418993828,6cyclictest17758-21sh09:32:131
175302380,3sleep30-21swapper/310:57:243
175302380,3sleep30-21swapper/310:57:243
175302380,3sleep30-21swapper/310:57:243
30415993720,14cyclictest16446-21systemctl08:02:120
30415993720,14cyclictest16446-21systemctl08:02:120
277562370,3chrt0-21swapper/209:57:262
277562370,3chrt0-21swapper/209:57:262
27062360,2sleep12714-21idleruntime-cro08:52:131
27062360,2sleep12714-21idleruntime-cro08:52:131
59912340,3sleep15994-21unixbench_multi08:57:351
59912340,3sleep15994-21unixbench_multi08:57:351
30418993428,5cyclictest18062-21needreboot10:57:301
30418993428,5cyclictest18062-21needreboot10:57:301
30418993428,5cyclictest18062-21needreboot10:57:301
263552340,2sleep30-21swapper/309:52:323
263552340,2sleep30-21swapper/309:52:323
30415993325,6cyclictest774-21gltestperf10:12:250
192852330,3chrt0-21swapper/308:07:323
192852330,3chrt0-21swapper/308:07:323
30418993228,3cyclictest30414-21cyclictest10:27:241
30418993228,3cyclictest30414-21cyclictest10:27:241
30415993222,8cyclictest12599-21processes10:42:310
30415993222,8cyclictest12599-21processes10:42:310
30418993122,7cyclictest30212-21threads10:02:331
30418993122,7cyclictest30212-21threads10:02:331
30418993122,7cyclictest27498-21fschecks_count09:57:241
30418993122,7cyclictest27498-21fschecks_count09:57:241
30415993127,3cyclictest16915-21fschecks_time08:02:250
30415993127,3cyclictest16915-21fschecks_time08:02:250
30418993025,3cyclictest19214-21needreboot08:07:311
30418993025,3cyclictest19214-21needreboot08:07:311
30418993025,3cyclictest11377-21tail07:47:261
30418993023,6cyclictest9406-21processes09:07:341
30418993023,5cyclictest9691-21idleruntime-cro10:37:121
30415993024,5cyclictest8384-21latency_hist09:07:140
30415993024,5cyclictest8384-21latency_hist09:07:140
30415993021,7cyclictest31320-21fschecks_time10:07:260
10882300,3sleep11092-21unixbench-2d11:37:341
10882300,3sleep11092-21unixbench-2d11:37:341
30418992923,5cyclictest9846-21kernelversion12:02:271
30418992923,5cyclictest9846-21kernelversion12:02:271
30418992923,4cyclictest1705-21unixbench_multi10:12:351
30418992923,4cyclictest1705-21unixbench_multi10:12:351
30418992922,4cyclictest0-21swapper/112:32:341
30418992922,4cyclictest0-21swapper/112:32:341
30418992921,6cyclictest3835-21cut08:52:341
30418992921,6cyclictest3835-21cut08:52:341
30418992921,6cyclictest25229-21latency_hist09:52:241
30418992921,6cyclictest25229-21latency_hist09:52:241
30415992923,5cyclictest20897-21sshd11:06:480
30415992923,5cyclictest20897-21sshd11:06:480
3041899284,10cyclictest15766-21ps12:17:311
3041899284,10cyclictest15766-21ps12:17:311
30418992823,4cyclictest20762-21mailstats09:37:341
30418992823,4cyclictest20762-21mailstats09:37:341
30418992823,3cyclictest30414-21cyclictest10:52:291
30418992820,6cyclictest4704-21threads11:47:361
30418992820,6cyclictest4704-21threads11:47:361
30418992819,7cyclictest23283-21cut11:12:251
30418992819,7cyclictest23283-21cut11:12:251
30415992822,5cyclictest8906-21latency_hist07:42:130
30415992822,5cyclictest27274-21kernelversion11:22:290
30415992822,5cyclictest27274-21kernelversion11:22:290
30415992822,4cyclictest1346-21avahi-daemon10:07:330
30415992819,7cyclictest3740-21processes08:52:330
30415992819,7cyclictest3740-21processes08:52:330
30415992819,7cyclictest18512-21timerwakeupswit10:57:340
30415992819,7cyclictest18512-21timerwakeupswit10:57:340
30415992819,7cyclictest18512-21timerwakeupswit10:57:340
30415992819,6cyclictest11194-21/usr/sbin/munin12:07:240
30415992819,6cyclictest11194-21/usr/sbin/munin12:07:240
23202280,2sleep02446-21missed_timers11:42:290
23202280,2sleep02446-21missed_timers11:42:290
3041899276,7cyclictest4712-21cat10:22:271
3041899274,11cyclictest2865-21sed11:42:351
3041899274,11cyclictest2865-21sed11:42:351
30418992721,5cyclictest8773-21sshd07:40:371
30418992721,4cyclictest13394-21kernelversion07:52:271
30418992721,4cyclictest0-21swapper/111:07:251
30415992721,4cyclictest0-21swapper/009:32:350
30415992721,4cyclictest0-21swapper/009:32:350
30415992721,4cyclictest0-21swapper/007:22:220
30415992721,4cyclictest0-21swapper/007:22:220
30415992720,5cyclictest29074-21latency_hist10:02:170
30415992720,5cyclictest29074-21latency_hist10:02:170
30415992720,5cyclictest21216-21sshd09:38:480
30415992720,5cyclictest21216-21sshd09:38:480
30415992720,5cyclictest19088-21head12:27:260
30415992720,5cyclictest19088-21head12:27:260
30415992720,5cyclictest19088-21head12:27:260
30415992720,5cyclictest13300-21cut12:12:250
30415992720,5cyclictest13300-21cut12:12:250
30415992720,5cyclictest13300-21cut12:12:250
3041899263,9cyclictest2488-21df10:17:251
30418992622,3cyclictest30414-21cyclictest08:07:131
30418992622,3cyclictest30414-21cyclictest08:07:131
3041899262,11cyclictest749-21runrttasks08:34:431
3041899262,11cyclictest749-21runrttasks08:34:431
3041899262,11cyclictest749-21runrttasks08:19:231
3041899262,11cyclictest749-21runrttasks08:19:231
3041899262,11cyclictest749-21runrttasks08:19:231
30418992619,5cyclictest11618-21unixbench-2d09:12:341
30418992619,5cyclictest11618-21unixbench-2d09:12:341
30418992617,7cyclictest21100-21netstat08:12:301
30418992617,7cyclictest21100-21netstat08:12:301
30418992611,4cyclictest90-1pr/ttyS010:32:181
30415992621,4cyclictest30414-21cyclictest08:12:280
30415992621,4cyclictest30414-21cyclictest08:12:280
30415992621,3cyclictest30414-21cyclictest11:17:240
30415992621,3cyclictest30414-21cyclictest11:17:240
30415992619,5cyclictest30795-21timerandwakeup08:37:350
30415992619,5cyclictest30795-21timerandwakeup08:37:350
30415992617,7cyclictest23345-21tail11:12:250
30415992616,8cyclictest32529-21cut11:37:240
30415992616,8cyclictest32529-21cut11:37:240
223672260,3sleep30-21swapper/308:17:253
223672260,3sleep30-21swapper/308:17:253
223672260,3sleep30-21swapper/308:17:253
3041899255,10cyclictest16216-21df_inode09:27:241
3041899255,10cyclictest16216-21df_inode09:27:241
3041899254,7cyclictest11761-21ls10:42:211
3041899254,7cyclictest11761-21ls10:42:211
30418992519,5cyclictest19224-21kernelversion12:27:281
30418992519,5cyclictest19224-21kernelversion12:27:281
30418992519,5cyclictest19224-21kernelversion12:27:281
30418992517,6cyclictest19995-21netstat11:02:311
30418992516,6cyclictest9275-21df07:42:231
30418992516,6cyclictest9275-21df07:42:231
30418992514,2cyclictest181rcu_preempt08:32:061
30418992514,2cyclictest181rcu_preempt08:32:061
30415992520,3cyclictest13681-21apache210:06:110
30415992520,3cyclictest0-21swapper/008:27:240
30415992520,3cyclictest0-21swapper/008:27:240
30415992520,3cyclictest0-21swapper/007:52:330
30415992520,2cyclictest0-21swapper/008:32:350
30415992520,2cyclictest0-21swapper/008:32:350
30415992519,5cyclictest17071-21switchtime09:27:330
30415992519,5cyclictest17071-21switchtime09:27:330
30415992519,5cyclictest17071-21switchtime09:27:330
30415992519,4cyclictest0-21swapper/012:12:290
30415992519,4cyclictest0-21swapper/012:12:290
30415992519,3cyclictest0-21swapper/008:47:290
30415992519,3cyclictest0-21swapper/008:47:290
30415992518,5cyclictest30414-21cyclictest11:12:280
30415992518,5cyclictest30414-21cyclictest11:12:280
30415992518,5cyclictest30414-21cyclictest07:27:290
30415992518,5cyclictest0-21swapper/012:32:360
30415992518,5cyclictest0-21swapper/012:32:360
30415992518,4cyclictest0-21swapper/010:52:310
30415992516,7cyclictest24578-21ls09:47:360
30415992516,7cyclictest24578-21ls09:47:360
30418992419,4cyclictest30414-21cyclictest11:34:321
30418992419,4cyclictest30414-21cyclictest11:34:321
30418992418,5cyclictest11952-21ntp_offset12:07:321
30418992418,5cyclictest11952-21ntp_offset12:07:321
30418992418,5cyclictest11952-21ntp_offset12:07:321
30418992418,3cyclictest0-21swapper/110:42:231
30418992418,3cyclictest0-21swapper/110:42:231
30418992417,5cyclictest30414-21cyclictest07:32:421
30418992417,5cyclictest30414-21cyclictest07:32:421
30418992417,4cyclictest0-21swapper/107:27:291
30418992416,2cyclictest181rcu_preempt09:37:011
30418992416,2cyclictest181rcu_preempt09:37:011
30418992412,3cyclictest90-1pr/ttyS009:24:571
30418992411,3cyclictest90-1pr/ttyS011:17:471
30418992411,3cyclictest90-1pr/ttyS011:17:471
3041899240,7cyclictest24504-21seq11:15:181
3041899240,7cyclictest24504-21seq11:15:181
30415992420,3cyclictest22537-21processes09:42:320
30415992420,3cyclictest22537-21processes09:42:320
30415992419,4cyclictest749-21runrttasks10:22:290
30415992419,4cyclictest749-21runrttasks10:22:290
30415992419,4cyclictest749-21runrttasks10:22:290
30415992419,3cyclictest0-21swapper/009:07:280
30415992418,3cyclictest0-21swapper/011:47:260
30415992418,3cyclictest0-21swapper/011:47:260
3041899239,2cyclictest181rcu_preempt12:14:501
3041899239,2cyclictest181rcu_preempt12:14:501
3041899239,2cyclictest181rcu_preempt08:00:231
3041899239,2cyclictest181rcu_preempt08:00:231
3041899234,3cyclictest281ktimers/111:29:581
3041899234,3cyclictest281ktimers/111:29:581
3041899234,3cyclictest281ktimers/111:29:581
30418992319,2cyclictest0-21swapper/111:57:261
30418992319,2cyclictest0-21swapper/111:57:261
30418992319,2cyclictest0-21swapper/108:42:331
30418992319,2cyclictest0-21swapper/108:42:331
30418992319,2cyclictest0-21swapper/108:42:331
30418992318,3cyclictest0-21swapper/109:17:291
30418992318,3cyclictest0-21swapper/109:17:291
30418992318,3cyclictest0-21swapper/109:17:291
30418992316,2cyclictest181rcu_preempt09:04:481
30418992316,2cyclictest181rcu_preempt09:04:481
30418992311,9cyclictest7187-21sshd11:56:481
30418992311,9cyclictest7187-21sshd11:56:481
30418992311,9cyclictest25287-21/usr/sbin/munin08:22:361
30418992311,9cyclictest25287-21/usr/sbin/munin08:22:361
30418992310,3cyclictest181rcu_preempt09:57:211
30418992310,3cyclictest181rcu_preempt09:57:211
30415992319,3cyclictest30414-21cyclictest08:27:310
30415992319,3cyclictest30414-21cyclictest08:27:310
30415992319,2cyclictest0-21swapper/011:07:340
30415992319,2cyclictest0-21swapper/011:07:340
30415992317,5cyclictest11535-21kernelversion07:47:280
30415992316,5cyclictest18502-21latency_hist12:27:130
30415992316,5cyclictest18502-21latency_hist12:27:130
30415992316,5cyclictest18502-21latency_hist12:27:130
30415992315,6cyclictest9875-21netstat07:42:290
30415992315,6cyclictest9875-21netstat07:42:290
127052230,3chrt0-21swapper/307:52:133
30418992217,3cyclictest0-21swapper/112:37:321
30418992217,3cyclictest0-21swapper/111:09:321
30418992217,3cyclictest0-21swapper/109:42:261
30418992217,3cyclictest0-21swapper/109:42:261
30418992214,2cyclictest181rcu_preempt10:51:261
30418992214,2cyclictest181rcu_preempt10:51:261
30415992217,3cyclictest0-21swapper/012:17:270
30415992217,3cyclictest0-21swapper/012:17:270
30415992217,3cyclictest0-21swapper/011:52:270
30415992217,3cyclictest0-21swapper/011:52:270
30415992217,3cyclictest0-21swapper/009:12:250
30415992217,3cyclictest0-21swapper/009:12:250
30415992216,5cyclictest31379-21sendmail_mailst11:32:310
30415992216,5cyclictest31379-21sendmail_mailst11:32:310
30415992216,5cyclictest14400-21ntp_kernel_pll_10:47:310
30415992216,5cyclictest14400-21ntp_kernel_pll_10:47:310
30415992216,4cyclictest8679-21head11:57:360
30415992216,4cyclictest8679-21head11:57:360
30415992216,4cyclictest0-21swapper/007:32:340
30415992216,4cyclictest0-21swapper/007:32:340
30415992215,5cyclictest27273-21latency_hist09:57:210
30415992215,5cyclictest27273-21latency_hist09:57:210
30415992214,6cyclictest12497-21cut09:17:250
30415992214,6cyclictest12497-21cut09:17:250
30415992214,6cyclictest12497-21cut09:17:250
30415992116,3cyclictest0-21swapper/008:57:300
30415992116,3cyclictest0-21swapper/008:57:300
30415992113,6cyclictest3400-21sort10:17:350
3041599201,9cyclictest10173-21fschecks_time10:37:250
3041599201,9cyclictest10173-21fschecks_time10:37:250
30415992016,2cyclictest0-21swapper/008:07:310
30415992016,2cyclictest0-21swapper/008:07:310
322992190,3sleep30-21swapper/308:42:293
322992190,3sleep30-21swapper/308:42:293
322992190,3sleep30-21swapper/308:42:293
3042099193,9cyclictest0-21swapper/309:17:483
3042099193,9cyclictest0-21swapper/309:17:483
3042099193,9cyclictest0-21swapper/309:17:483
3042099192,13cyclictest0-21swapper/311:22:163
3042099192,13cyclictest0-21swapper/311:22:163
3041999191,13cyclictest16086-21chrt09:27:232
3041999191,13cyclictest16086-21chrt09:27:232
3041999191,13cyclictest16086-21chrt09:27:232
3041599191,15cyclictest8648-21ls10:32:270
118052190,3chrt0-21swapper/207:47:332
3042099183,9cyclictest0-21swapper/308:53:483
3042099183,9cyclictest0-21swapper/308:53:483
3042099181,13cyclictest16301-21chrt10:52:303
3042099181,13cyclictest15593-21chrt07:57:313
3042099181,13cyclictest15593-21chrt07:57:313
3042099181,12cyclictest6026-21chrt10:27:173
3042099181,12cyclictest6026-21chrt10:27:173
3042099181,12cyclictest6026-21chrt10:27:173
3042099173,9cyclictest0-21swapper/310:32:483
3042099173,9cyclictest0-21swapper/309:49:483
3042099173,9cyclictest0-21swapper/309:49:483
3042099171,11cyclictest9303-21taskset07:42:243
3042099171,11cyclictest9303-21taskset07:42:243
3042099171,11cyclictest0-21swapper/312:32:283
3042099171,11cyclictest0-21swapper/312:32:283
3041999172,10cyclictest395-21taskset10:12:212
3041999172,10cyclictest395-21taskset10:12:212
3041999171,13cyclictest4588-21chrt07:27:352
30419991710,5cyclictest2715-21chrt08:52:132
30419991710,5cyclictest2715-21chrt08:52:132
3042099163,9cyclictest0-21swapper/312:22:483
3042099163,9cyclictest0-21swapper/312:22:483
3042099163,9cyclictest0-21swapper/312:22:483
3042099163,9cyclictest0-21swapper/310:07:483
3042099163,9cyclictest0-21swapper/310:07:483
3042099163,9cyclictest0-21swapper/310:02:483
3042099163,9cyclictest0-21swapper/310:02:483
3042099163,9cyclictest0-21swapper/308:38:483
3042099163,9cyclictest0-21swapper/308:38:483
3042099161,9cyclictest0-21swapper/312:02:283
3042099161,9cyclictest0-21swapper/312:02:283
3042099161,11cyclictest26795-21chrt11:22:253
3042099161,11cyclictest26795-21chrt11:22:253
3042099161,11cyclictest23608-21chrt11:12:283
3042099161,11cyclictest23608-21chrt11:12:283
3042099161,11cyclictest20807-21chrt08:12:273
3042099161,11cyclictest20807-21chrt08:12:273
3042099161,11cyclictest16964-21chrt08:02:263
3042099161,11cyclictest16964-21chrt08:02:263
3042099161,11cyclictest0-21swapper/310:17:233
3042099161,11cyclictest0-21swapper/309:27:273
3042099161,11cyclictest0-21swapper/309:27:273
3042099161,11cyclictest0-21swapper/309:27:273
3042099161,11cyclictest0-21swapper/308:22:283
3042099161,11cyclictest0-21swapper/308:22:283
3042099161,10cyclictest8756-21chrt09:07:263
3042099161,10cyclictest2118-21chrt07:22:263
3042099161,10cyclictest2118-21chrt07:22:263
3042099161,10cyclictest0-21swapper/308:47:343
3042099161,10cyclictest0-21swapper/308:47:343
3041999163,9cyclictest0-21swapper/210:43:482
3041999163,9cyclictest0-21swapper/210:43:482
3041999163,9cyclictest0-21swapper/210:23:482
3041999163,9cyclictest0-21swapper/210:23:482
3041999163,9cyclictest0-21swapper/210:23:482
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional