You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-27 - 09:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon Apr 27, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22602164108,28sleep20-21swapper/219:06:452
24462162109,28sleep30-21swapper/319:09:113
22792161103,28sleep10-21swapper/119:06:591
23402159102,29sleep00-21swapper/019:07:470
16022680,2chrt0-21swapper/223:19:282
60192600,2sleep30-21swapper/319:15:203
231492590,1sleep20-21swapper/223:10:262
220982560,1sleep10-21swapper/100:22:281
274722550,2sleep00-21swapper/023:14:370
321232500,2sleep20-21swapper/200:30:162
262932500,1sleep20-21swapper/221:12:312
137462490,4sleep00-21swapper/000:15:490
300932470,2sleep10-21swapper/123:15:301
236652460,2sleep10-21swapper/121:59:181
124312450,2sleep30-21swapper/321:25:573
295882400,1sleep10-21swapper/122:27:111
43182320,2sleep10-21swapper/122:32:361
9072300,2sleep30-21swapper/300:06:213
299199303,11cyclictest0-21swapper/200:05:122
299199302,11cyclictest10872-21sh22:37:292
299199301,11cyclictest10360-21cpuspeed_turbos21:25:142
2987993013,7cyclictest6209-21ls21:45:240
29782300,2sleep10-21swapper/123:20:161
293702300,9sleep228-21ksoftirqd/221:39:202
64092290,3sleep32984-21cyclictest22:10:153
299199292,10cyclictest20668-21rt-features22:20:232
299199290,17cyclictest19669-21sh23:08:292
17222290,3chrt0-21swapper/321:42:223
299199284,6cyclictest27143-21/usr/sbin/munin22:25:202
299199282,8cyclictest25870-21latency00:25:192
299199280,11cyclictest0-21swapper/222:41:292
221162280,2sleep2271rcuc/200:22:292
2994992710,7cyclictest24770-21seq22:24:053
299199273,8cyclictest9197-21ssh23:49:152
298899271,24cyclictest0-21swapper/122:05:251
118952270,2sleep30-21swapper/323:02:243
299199264,8cyclictest25870-21ssh00:00:472
299199263,8cyclictest2702-21ssh23:44:302
299199263,8cyclictest23657-21ssh22:46:502
299199262,9cyclictest14118-21sh21:51:222
299199262,12cyclictest28-21ksoftirqd/200:18:312
2987992624,1cyclictest0-21swapper/022:05:240
2987992618,6cyclictest31612-21ssh22:53:060
299199254,7cyclictest0-21swapper/222:16:262
299199254,19cyclictest6630-21head00:10:252
299199253,8cyclictest17776-21ssh23:30:302
299199253,8cyclictest17776-21ssh23:30:302
299199253,6cyclictest21532-21ssh21:57:072
299199253,6cyclictest19651-21ssh23:56:182
299199252,7cyclictest8838-21sh21:47:492
299199252,6cyclictest5062-21ssh23:21:102
2991992515,6cyclictest0-21swapper/222:00:032
299199250,11cyclictest8637-21sh22:11:202
15132250,2sleep10-21swapper/121:18:281
55122240,1sleep20-21swapper/219:15:152
299499241,13cyclictest0-21swapper/322:35:153
299199247,6cyclictest28-21ksoftirqd/221:15:332
299199243,6cyclictest3373-21ssh21:44:022
299199243,6cyclictest2765-21ssh22:07:092
299199242,9cyclictest9458-21sh21:24:422
270732240,2sleep122-21ksoftirqd/120:05:131
232162240,3chrt0-21swapper/021:10:170
140582240,2sleep30-21swapper/300:16:083
47842230,2sleep00-21swapper/021:20:240
299499232,14cyclictest11489-21ssh21:50:143
2991992318,3cyclictest271rcuc/220:55:212
299199231,6cyclictest15786-21sh23:54:082
299199230,9cyclictest10999-21sh23:01:302
299199230,6cyclictest23604-21ls23:35:172
299199230,10cyclictest21208-21sh21:33:032
298899237,9cyclictest22-21ksoftirqd/122:57:111
298899234,4cyclictest141rcu_preempt21:30:131
298899232,14cyclictest8223-21diskmemload23:00:121
298899232,14cyclictest22057-21diskmemload23:10:121
2988992317,5cyclictest22-21ksoftirqd/100:05:001
2988992315,6cyclictest5808-21ntp_states21:45:201
298899231,10cyclictest301662cat20:10:231
298899231,10cyclictest299612diskmemload21:39:561
2988992310,11cyclictest9466-21rm21:24:421
92702221,4sleep334-21ksoftirqd/323:25:123
299499223,16cyclictest27090-21grep22:25:203
2994992215,5cyclictest29632-21fschecks_count21:15:163
2994992213,7cyclictest10397-21pluginstate20:40:223
299199225,9cyclictest28-21ksoftirqd/222:59:582
299199221,5cyclictest0-21swapper/220:30:112
299199221,17cyclictest1215-21sort22:55:012
2988992213,6cyclictest25859-21python320:00:221
298799224,12cyclictest75-21kswapd022:35:110
298799223,3cyclictest303872ssh21:40:060
298799222,4cyclictest0-21swapper/021:29:360
294432220,5sleep029445-21sh23:40:130
286652220,2sleep10-21swapper/100:27:221
161662221,2sleep116186-21ssh22:41:041
14632220,2sleep00-21swapper/022:55:110
299499213,13cyclictest0-21swapper/300:20:113
299499211,4cyclictest0-21swapper/319:25:133
2994992114,5cyclictest34-21ksoftirqd/322:20:013
2994992114,5cyclictest20528-21grep21:05:153
2994992114,4cyclictest25012-21/usr/sbin/munin22:00:143
2994992113,6cyclictest24195-21users21:10:253
2994992113,6cyclictest17567-21perl23:55:163
299199214,9cyclictest6037-21ssh00:35:122
299199212,6cyclictest28-21ksoftirqd/219:25:162
2991992117,3cyclictest271rcuc/223:27:362
2991992113,6cyclictest17233-21ls19:40:252
2991992113,6cyclictest14649-21ntp_states20:50:222
299199211,16cyclictest3998-21tr20:25:222
298899219,3cyclictest141rcu_preempt23:55:001
2988992115,5cyclictest22-21ksoftirqd/100:10:001
2988992115,4cyclictest22-21ksoftirqd/100:00:001
2988992113,6cyclictest20721-21smartctl21:05:161
2988992112,7cyclictest10412-21pluginstate23:25:231
2988992112,6cyclictest30925-21grep23:40:261
298799215,4cyclictest66050lldpd19:45:560
298799212,13cyclictest5186-21chrt19:15:130
2987992115,4cyclictest0-21swapper/023:28:050
2987992112,6cyclictest29302-21df20:10:140
2987992111,7cyclictest17425-21df23:55:140
103562210,1sleep10-21swapper/122:36:581
2994992014,4cyclictest25730-21sh23:36:463
2994992012,6cyclictest560-21dbus-daemon20:40:003
2994992012,6cyclictest5379-21seq22:09:483
2994992012,6cyclictest31116-21apt20:15:123
2994992012,6cyclictest23824-21apt-get00:00:133
2994992012,6cyclictest11681-21tr23:50:233
2994992011,6cyclictest27994-21sensors_temp20:05:233
2994992011,6cyclictest21187-21pluginstate23:10:013
2994992011,6cyclictest13914-21grep19:35:133
2994992011,6cyclictest13914-21grep19:35:133
299499200,8cyclictest219742sh23:10:153
299199202,5cyclictest28-21ksoftirqd/222:33:022
2991992012,6cyclictest14316-21expr19:35:162
2991992012,6cyclictest14316-21expr19:35:162
2991992011,6cyclictest334-21seq20:17:322
2991992011,6cyclictest11803-21expr20:45:132
2991992010,7cyclictest19299-21wc21:00:252
298899202,2cyclictest141rcu_preempt23:37:221
298899201,5cyclictest0-21swapper/123:46:481
2988992012,6cyclictest17351-21cat23:30:231
2988992012,6cyclictest17351-21cat23:30:221
2988992012,6cyclictest16721-21netstat20:55:211
2988992012,6cyclictest13414-21seq20:49:251
2988992011,7cyclictest6778-21seq20:31:531
2988992011,7cyclictest5975-21ntpq00:10:181
2988992011,7cyclictest28588-21cat22:50:251
2988992011,6cyclictest801-21cut20:20:001
298799209,9cyclictest20020-21sh23:32:460
298799209,9cyclictest20020-21sh23:32:460
298799208,10cyclictest23442-21latency_hist21:35:000
298799208,10cyclictest0-21swapper/021:15:170
298799202,13cyclictest22695-21ssh23:35:120
298799201,9cyclictest6887-21memory22:10:190
2987992015,3cyclictest2116-21sh23:20:000
2987992012,6cyclictest6839-21seq20:32:370
2987992012,6cyclictest23910-21cat00:00:150
2987992012,6cyclictest22259-21sed21:10:000
2987992011,6cyclictest12-21ksoftirqd/023:05:190
2987992010,6cyclictest31920-21cut00:05:220
235922200,3sleep323590-21sh22:46:473
299499192,12cyclictest3533-21ssh21:20:133
299499192,12cyclictest24705-21apt-get20:00:123
299499192,12cyclictest18134-21apt-get21:55:123
299499192,12cyclictest18060-21apt-get19:45:103
2994991914,3cyclictest499-21sh23:42:183
2994991914,3cyclictest21887-21sh23:34:393
2994991914,3cyclictest21887-21sh23:34:393
2994991914,3cyclictest2139-21vmstat22:30:253
2994991914,3cyclictest15891-21sh22:40:473
2994991913,4cyclictest8488-21sh23:48:323
2994991913,4cyclictest8230-21sh00:11:583
2994991913,4cyclictest4442-21sh22:56:383
2994991913,4cyclictest30388-21sh23:15:483
2994991912,5cyclictest6475-21seq23:22:363
2994991912,5cyclictest24297-21uniq21:35:153
2994991912,5cyclictest16764-21ntp_kernel_pll_20:55:213
2994991912,4cyclictest22057-21diskmemload21:32:233
2994991911,6cyclictest26367-21readlink00:25:243
2994991910,7cyclictest387-21pluginstate00:30:223
299199199,7cyclictest7757-21cut19:20:152
2991991911,6cyclictest25899-21cat20:00:232
2991991911,5cyclictest23293-21expr19:55:192
299199190,5cyclictest29885-21ntpq20:10:212
298899196,3cyclictest141rcu_preempt00:19:091
2988991913,4cyclictest30442-21sh21:40:101
2988991913,4cyclictest22-21ksoftirqd/122:05:001
2988991913,4cyclictest22-21ksoftirqd/121:15:001
2988991913,4cyclictest20695-21sendmail_mailst22:20:231
2988991913,4cyclictest17439-21vmstat19:40:271
2988991912,5cyclictest22208-21seq19:54:051
2988991911,6cyclictest16005-21cron22:17:001
2988991911,6cyclictest15717-21grep20:55:011
2988991911,6cyclictest12434-21sed19:30:201
2988991910,7cyclictest14269-21expr19:35:161
2988991910,7cyclictest14269-21expr19:35:151
2988991910,6cyclictest52850irq/140-enp3s0-23:05:131
2988991910,6cyclictest16500-21grep21:30:011
298799199,7cyclictest31751-21ssh00:30:130
298799192,12cyclictest640-21snmpd19:40:260
298799192,12cyclictest23803-21ssh21:35:120
2987991914,3cyclictest12085-21uptime23:50:250
2987991912,5cyclictest14382-21expr22:40:160
2987991911,6cyclictest31578-21perl20:15:150
2987991911,6cyclictest14623-21ntp_kernel_pll_20:50:220
2987991911,5cyclictest52750irq/139-enp3s0-22:25:220
2987991911,5cyclictest4313-21timerandwakeup19:10:220
2987991910,6cyclictest52750irq/139-enp3s0-22:15:100
2987991910,6cyclictest4119-21tune2fs23:45:150
2987991910,6cyclictest25842-21latency22:00:190
2987991910,6cyclictest12-21ksoftirqd/019:50:200
32302180,2chrt0-21swapper/022:31:310
299499186,10cyclictest4897-21cron20:30:003
299499185,9cyclictest6040-21apt-get00:35:133
299499182,11cyclictest7345-21apt-get19:20:133
2994991813,3cyclictest5779-21if_err_enp3s020:30:163
2994991813,3cyclictest28963-21acpi20:10:123
2994991812,4cyclictest19264-21find21:00:253
2994991811,5cyclictest4910-21expr21:45:133
2994991811,5cyclictest28047-21ls22:50:193
2994991811,5cyclictest12862-21find19:30:253
2994991810,6cyclictest21784-21ls19:50:263
299199182,4cyclictest0-21swapper/220:44:552
2991991810,6cyclictest941-21sort20:20:142
2991991810,5cyclictest20450-21expr21:05:132
299199180,7cyclictest28-21ksoftirqd/219:50:192
299199180,6cyclictest28-21ksoftirqd/219:30:252
298899181,8cyclictest11165-21ssh22:13:531
298899181,5cyclictest0-21swapper/100:31:171
2988991813,3cyclictest8721-21sh00:36:251
2988991812,4cyclictest22-21ksoftirqd/119:19:591
2988991812,4cyclictest12539-21sh21:50:231
2988991811,5cyclictest9988-21expr19:25:171
2988991811,5cyclictest23273-21sed19:55:181
2988991811,5cyclictest10122-21cut20:40:181
2988991811,4cyclictest20539-21sh22:45:121
2988991810,6cyclictest3836-21cat19:10:181
298799187,7cyclictest3011-21apt-get20:25:130
298799187,3cyclictest141rcu_preempt21:57:360
298799181,8cyclictest8234-21taskset23:00:120
298799181,8cyclictest12648-21ps21:50:240
2987991813,3cyclictest22860-21sh00:23:140
2987991811,5cyclictest8997-21seq20:37:540
2987991811,5cyclictest7324-21grep00:35:240
2987991810,6cyclictest818-21seq20:20:010
2987991810,6cyclictest20677-21grep22:45:140
2987991810,6cyclictest19812-21diskstats22:20:150
2987991810,6cyclictest16601-21sed20:55:190
2987991810,6cyclictest14681-21uname19:35:210
2987991810,6cyclictest14681-21uname19:35:200
2987991810,6cyclictest1-21systemd20:45:220
2987991810,5cyclictest9431-21find20:40:100
2987991810,5cyclictest26654-21grep00:25:270
299499179,6cyclictest23916-21cat19:55:243
2994991710,5cyclictest4331-21cat19:10:233
2994991710,5cyclictest11893-21expr20:45:133
299199171,5cyclictest28-21ksoftirqd/220:35:212
2991991711,4cyclictest3317-21expr19:10:122
2991991711,4cyclictest19342-21cat19:45:232
298899172,2cyclictest141rcu_preempt19:45:181
2988991710,5cyclictest8066-21sed20:35:181
2988991710,5cyclictest3797-21sed20:25:201
2988991710,5cyclictest2135-21sed20:20:251
298799179,6cyclictest7680-21diskstats19:20:160
298799179,6cyclictest6609-21tail00:10:250
298799179,6cyclictest20033-21date21:05:000
298799172,11cyclictest2616-21apt-get23:20:150
2987991712,3cyclictest26817-21/usr/sbin/munin20:05:150
2987991712,3cyclictest25308-21if_enp3s020:00:160
2987991712,3cyclictest22522-21/usr/sbin/munin19:55:130
2987991710,5cyclictest11193-21seq19:27:570
299499161,11cyclictest13727-21apt-get20:50:143
2994991610,5cyclictest34-21ksoftirqd/320:25:003
2994991610,4cyclictest17819-21latency_hist19:45:013
299199161,10cyclictest26842-21apt-get20:05:132
298899168,6cyclictest18097-21cat21:00:141
2988991611,3cyclictest8028-21latency19:20:191
298799169,5cyclictest12364-21cat19:30:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional