You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-29 - 06:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Aug 29, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2810521280,0sleep20-21swapper/222:40:502
10189210188,9sleep00-21swapper/019:05:300
1050228370,9sleep20-21swapper/219:09:342
1037228265,13sleep10-21swapper/119:07:521
1042627966,9sleep30-21swapper/319:08:333
179732530,0sleep10-21swapper/121:18:171
1061599271,1cyclictest141rcu_preempt21:24:081
10624992624,1cyclictest0-21swapper/321:50:123
10620992322,1cyclictest7954-21kworker/u8:022:35:102
10620992322,1cyclictest7954-21kworker/u8:022:35:092
1061599230,1cyclictest141rcu_preempt23:01:001
10620992119,1cyclictest32-21ksoftirqd/200:24:292
10620992117,3cyclictest32-21ksoftirqd/222:10:452
10620992110,1cyclictest32-21ksoftirqd/222:59:002
1061599210,20cyclictest55650irq/127-enp2s0-21:10:131
1062499200,3cyclictest29969-21apache_processe21:45:123
10615992015,2cyclictest25-21ksoftirqd/122:50:171
1061199200,0cyclictest0-21swapper/020:01:410
293252190,0sleep00-21swapper/021:44:530
10620991914,1cyclictest32-21ksoftirqd/200:29:032
10620991914,1cyclictest32-21ksoftirqd/200:29:022
10615991917,1cyclictest25-21ksoftirqd/123:12:501
10615991917,1cyclictest25-21ksoftirqd/122:04:301
1062499180,1cyclictest0-21swapper/300:07:443
1062099187,2cyclictest141rcu_preempt21:20:182
1062099180,1cyclictest0-21swapper/221:58:342
10615991817,1cyclictest0-21swapper/100:22:001
10615991812,6cyclictest38150irq/131-snd_hda00:09:101
1061199180,1cyclictest0-21swapper/021:13:070
1061199180,17cyclictest0-21swapper/023:09:070
1062499170,16cyclictest15094-21python23:50:253
1062499170,12cyclictest0-21swapper/320:10:133
1062099171,2cyclictest141rcu_preempt22:22:432
1061599175,1cyclictest25-21ksoftirqd/121:32:521
1061599170,2cyclictest141rcu_preempt21:52:511
1061599170,16cyclictest0-21swapper/121:58:341
10611991715,2cyclictest13-21ksoftirqd/022:54:270
1061199170,1cyclictest0-21swapper/023:04:010
1062499168,3cyclictest225-21systemd-journal20:33:293
1062499161,12cyclictest0-21swapper/323:00:143
1062499160,13cyclictest0-21swapper/322:57:023
1062499160,0cyclictest0-21swapper/323:45:453
1062499160,0cyclictest0-21swapper/322:13:593
1062099164,5cyclictest19809-21munin-node22:50:292
1062099164,2cyclictest32-21ksoftirqd/200:32:012
10620991614,2cyclictest32-21ksoftirqd/222:03:162
1062099161,2cyclictest141rcu_preempt00:01:372
1062099160,4cyclictest0-21swapper/221:25:252
1062099160,3cyclictest0-21swapper/222:25:282
1062099160,16cyclictest0-21swapper/221:31:102
1061599164,8cyclictest17859-21/usr/sbin/munin22:35:151
1061599164,8cyclictest17859-21/usr/sbin/munin22:35:141
1061599161,4cyclictest141rcu_preempt21:25:141
10615991614,2cyclictest25-21ksoftirqd/122:20:001
1061599161,1cyclictest141rcu_preempt22:35:001
1061599161,0cyclictest141rcu_preempt23:49:541
1061199163,7cyclictest3428-21diskmemload22:50:010
10611991610,5cyclictest15827-21apache221:50:170
1062499159,6cyclictest15851-21apache223:19:103
10624991512,2cyclictest22697-21apt-config00:15:003
1062099159,1cyclictest141rcu_preempt22:05:182
1062099153,1cyclictest141rcu_preempt00:36:392
10620991513,2cyclictest9406-21kworker/u8:020:36:152
10620991513,1cyclictest32-21ksoftirqd/200:07:392
10620991512,2cyclictest32-21ksoftirqd/221:45:182
10620991511,3cyclictest5238-21sh23:45:122
10620991511,3cyclictest3839-21head20:00:202
10620991511,3cyclictest23298-21wget00:15:142
10620991511,2cyclictest32-21ksoftirqd/223:40:132
10620991510,2cyclictest32-21ksoftirqd/223:20:142
10615991513,1cyclictest241rcuc/123:40:131
1061599151,2cyclictest141rcu_preempt22:09:521
10615991511,3cyclictest24018-21ls19:35:181
1061599150,14cyclictest0-21swapper/100:34:451
1061199151,7cyclictest5475-21munin-run21:50:000
10611991514,1cyclictest0-21swapper/000:26:030
10611991514,1cyclictest0-21swapper/000:26:020
10611991514,1cyclictest0-21swapper/000:21:110
10611991512,1cyclictest0-21swapper/000:30:130
1062499140,2cyclictest0-21swapper/321:37:583
1062099148,4cyclictest32-21ksoftirqd/221:15:132
1062099148,2cyclictest32-21ksoftirqd/222:30:422
1062099143,8cyclictest32-21ksoftirqd/223:04:572
1062099141,2cyclictest141rcu_preempt21:44:492
10620991412,1cyclictest32-21ksoftirqd/222:17:372
10620991411,3cyclictest32-21ksoftirqd/221:50:312
10620991411,2cyclictest32-21ksoftirqd/223:07:082
10620991411,2cyclictest28715-21basename20:55:002
10620991410,3cyclictest7297-21ssh21:11:302
1062099140,12cyclictest0-21swapper/223:32:302
1061599148,2cyclictest25-21ksoftirqd/100:04:501
1061599148,2cyclictest225-21systemd-journal23:26:571
1061599142,0cyclictest141rcu_preempt23:07:051
1061599141,13cyclictest3428-21diskmemload23:21:081
1061599141,13cyclictest0-21swapper/122:23:201
10615991411,2cyclictest14280-21head23:30:251
1061199147,6cyclictest23956-21munin-run20:45:010
1061199140,13cyclictest17117-21basename20:30:130
1061199140,13cyclictest0-21swapper/022:59:300
1061199140,13cyclictest0-21swapper/000:00:100
1061199140,0cyclictest0-21swapper/000:12:150
1062499131,1cyclictest0-21swapper/321:57:513
10624991310,2cyclictest32459-21cut22:05:193
10624991310,2cyclictest20749-21ssh21:20:123
1062499130,2cyclictest0-21swapper/322:28:213
1062099139,3cyclictest5339-21tr20:05:142
1062099139,3cyclictest17210-21grep20:30:132
1062099135,7cyclictest1142-21runrttasks23:25:062
1062099133,3cyclictest225-21systemd-journal23:20:002
1062099132,5cyclictest24297-21sh23:36:362
10620991310,2cyclictest22799-21wc23:55:192
10620991310,2cyclictest16678-21grep00:10:262
10620991310,2cyclictest15410-21grep20:25:182
1061599138,4cyclictest27234-21ssh21:43:131
1061599138,1cyclictest25-21ksoftirqd/119:11:071
1061599136,6cyclictest25-21ksoftirqd/123:50:141
1061599135,4cyclictest25-21ksoftirqd/121:35:191
1061599133,10cyclictest241rcuc/122:10:031
10615991312,1cyclictest0-21swapper/100:15:341
10615991310,2cyclictest373-21tr19:55:141
10615991310,2cyclictest25-21ksoftirqd/123:17:121
10615991310,2cyclictest15583-21wc19:20:011
1061599130,13cyclictest0-21swapper/100:28:201
1061599130,13cyclictest0-21swapper/100:28:191
1061599130,12cyclictest0-21swapper/122:46:371
1061599130,12cyclictest0-21swapper/121:46:061
1061599130,12cyclictest0-21swapper/100:36:371
1061199138,3cyclictest3428-21diskmemload23:30:180
10611991311,2cyclictest13-21ksoftirqd/021:20:580
10611991310,2cyclictest32209-21grep22:05:170
10611991310,2cyclictest2479-21cat20:00:000
10611991310,2cyclictest24571-21cat19:35:250
10611991310,2cyclictest2309-21grep22:25:270
10611991310,2cyclictest22727-21grep23:35:250
1061199130,12cyclictest7730-21grep20:10:130
1061199130,11cyclictest0-21swapper/023:17:520
1062499129,2cyclictest9350-21grep00:25:273
1062499129,2cyclictest9350-21grep00:25:263
1062499129,2cyclictest30435-21grep21:25:213
1062499129,2cyclictest29939-21ssh00:00:003
1062499129,2cyclictest29682-21cron00:00:003
1062499129,2cyclictest2930-21cut23:05:143
1062499129,2cyclictest27500-21sed22:40:253
1062499129,2cyclictest26123-21rm00:16:093
1062499129,2cyclictest24865-21cut00:35:163
1062499129,2cyclictest22930-21tr21:40:193
1062499129,2cyclictest16674-21head19:20:173
1062499129,2cyclictest14395-21sleep20:24:593
1062499129,2cyclictest13984-21grep19:15:173
1062499129,2cyclictest12637-21tr23:30:123
1062499129,2cyclictest12401-21grep21:15:123
1062499129,2cyclictest12111-21grep19:10:353
1062499128,3cyclictest17273-21cut22:35:133
1062499128,3cyclictest17273-21cut22:35:133
1062499127,3cyclictest1867-21tr22:45:153
10624991210,1cyclictest24989-21head20:45:163
10624991210,1cyclictest24989-21head20:45:163
1062499120,1cyclictest141rcu_preempt23:24:403
1062499120,11cyclictest32133-21grep21:00:163
1062099129,2cyclictest3425-21sh21:10:002
1062099129,2cyclictest32096-21tail21:00:162
1062099129,2cyclictest29693-21head20:55:182
1062099129,2cyclictest14255-21grep23:50:172
1062099126,5cyclictest32-21ksoftirqd/222:45:162
1062099125,1cyclictest32-21ksoftirqd/223:10:112
1061599129,2cyclictest22985-21tr20:40:361
1061599129,2cyclictest2260-21wc22:25:271
1061599129,2cyclictest20692-21tr19:30:141
1061599129,2cyclictest1383-21grep21:05:121
1061599128,3cyclictest29565-21grep20:55:161
1061599128,1cyclictest25-21ksoftirqd/119:45:001
1061599126,1cyclictest25-21ksoftirqd/123:55:231
1061599125,5cyclictest25-21ksoftirqd/120:25:151
1061599124,7cyclictest25-21ksoftirqd/100:10:211
1061599124,7cyclictest11998-21munin-run20:20:001
10615991210,2cyclictest25-21ksoftirqd/122:56:431
1061199129,2cyclictest9620-21grep22:30:160
1061199129,2cyclictest6944-21grep00:05:170
1061199129,2cyclictest5253-21tr21:30:140
1061199129,2cyclictest31091-21tail23:40:260
1061199129,2cyclictest25713-21grep19:40:140
1061199129,2cyclictest25207-21cut22:20:160
1061199129,2cyclictest25086-21sh22:40:010
1061199129,2cyclictest25086-21sh22:40:000
1061199129,2cyclictest22341-21cat21:59:590
1061199129,2cyclictest20163-21cron21:20:010
1061199129,2cyclictest19729-21grep20:35:130
1061199124,5cyclictest23939-21cron22:20:000
10611991210,2cyclictest13-21ksoftirqd/022:11:060
1061199120,11cyclictest29911-21grep23:20:270
1062499118,2cyclictest9676-21sed22:30:173
1062499118,2cyclictest5836-21head20:05:153
1062499118,2cyclictest5485-21df21:30:153
1062499118,2cyclictest5344-21ssh23:25:213
1062499118,2cyclictest4723-21grep21:10:173
1062499118,2cyclictest3926-21ls20:00:213
1062499118,2cyclictest31865-21tr00:20:143
1062499118,2cyclictest28294-21ssh22:21:523
1062499118,2cyclictest22711-21ssh22:00:083
1062499118,2cyclictest22469-21cat23:35:253
1062499118,2cyclictest20885-21ssh00:32:483
1062499118,2cyclictest18339-21basename19:25:143
1062499118,2cyclictest15015-21ssh23:11:513
10624991110,1cyclictest0-21swapper/322:15:403
1062499110,10cyclictest30471-21taskset20:55:303
1062099118,2cyclictest28569-21cut19:45:152
1062099116,4cyclictest32-21ksoftirqd/220:45:212
1062099116,4cyclictest32-21ksoftirqd/220:45:202
1062099116,4cyclictest32-21ksoftirqd/220:15:222
1062099116,4cyclictest32-21ksoftirqd/219:55:142
1062099116,4cyclictest32-21ksoftirqd/219:40:152
1062099116,1cyclictest32-21ksoftirqd/221:35:132
1062099116,1cyclictest32-21ksoftirqd/220:20:202
1062099116,1cyclictest32-21ksoftirqd/219:20:162
1062099116,1cyclictest32-21ksoftirqd/219:15:262
10620991110,1cyclictest32-21ksoftirqd/219:35:222
1061599118,2cyclictest9312-21chrt20:11:291
1061599118,2cyclictest32383-21wc21:00:191
1061599118,2cyclictest31406-21grep19:50:221
1061599118,2cyclictest21724-21grep20:40:001
1061599118,2cyclictest19001-21head19:25:171
1061599117,3cyclictest27566-21head22:40:251
1061599116,4cyclictest25-21ksoftirqd/119:45:151
1061599113,4cyclictest225-21systemd-journal20:31:221
10615991110,1cyclictest25-21ksoftirqd/123:35:091
1061599110,10cyclictest16483-21df19:20:151
1061199118,2cyclictest6143-21uname23:45:180
1061199118,2cyclictest5983-21cat20:05:170
1061199118,2cyclictest30955-21grep19:50:170
1061199118,2cyclictest27907-21ssh22:40:400
1061199118,2cyclictest25780-21rm23:56:570
1061199118,2cyclictest25268-21ssh00:15:260
1061199118,2cyclictest2045-21grep21:05:150
1061199118,2cyclictest17105-21ssh23:13:250
1061199118,2cyclictest16336-21ssh23:51:050
1061199118,2cyclictest1031-21cron21:05:010
1061199118,1cyclictest0-21swapper/023:25:170
1061199117,3cyclictest29776-21grep21:25:170
1061199117,2cyclictest14394-21ntp_states19:15:210
1061199110,1cyclictest0-21swapper/000:37:050
1061199110,10cyclictest16805-21ssh21:36:370
1062499109,1cyclictest0-21swapper/323:40:083
1062499108,1cyclictest9653-21ssh22:50:013
1062499108,1cyclictest25779-21expr19:40:143
1062499108,1cyclictest248882sleep319:36:553
1062499108,1cyclictest225472sleep319:32:213
1062499108,1cyclictest21838-21apache_accesses20:40:123
1062499108,1cyclictest19650-21grep20:35:143
1062499108,1cyclictest1398-21hostname21:05:123
1062499108,1cyclictest10504-21fschecks_time20:15:173
1062499107,2cyclictest28470-21seq20:52:113
1062499102,4cyclictest537-21snmpd19:55:163
1062099108,1cyclictest187652sleep219:25:162
1062099107,2cyclictest8210-21tail20:10:172
1062099107,2cyclictest32093-21taskset19:52:422
1062099106,1cyclictest32-21ksoftirqd/219:10:112
1062099105,1cyclictest32-21ksoftirqd/219:30:272
1062099101,4cyclictest225-21systemd-journal20:45:012
1061599108,1cyclictest3105-21expr20:00:141
1061599107,2cyclictest6647-21wc20:05:251
1061599107,2cyclictest27631-21sed20:50:191
1061599107,2cyclictest13587-21wc20:20:211
1061599104,5cyclictest25-21ksoftirqd/120:45:171
1061599104,5cyclictest25-21ksoftirqd/120:45:161
1061199108,1cyclictest23161-21expr22:00:140
1061199108,1cyclictest21038-21expr19:30:150
1061199108,1cyclictest17814-21taskset19:22:310
1061199107,2cyclictest29590-21grep20:55:160
1061199107,2cyclictest26731-21grep20:50:140
1061199107,2cyclictest24814-21grep20:45:150
1061199107,2cyclictest24814-21grep20:45:140
1061199107,2cyclictest18904-21grep19:25:170
1061199107,2cyclictest12573-21cstates20:20:140
1061199107,2cyclictest11198-21expr19:10:130
1061199107,2cyclictest11168-21cut20:15:220
106249997,1cyclictest27911-21aten2_r7power_v19:45:143
106249997,1cyclictest16659-21chrt20:28:153
106249996,2cyclictest30834-21cut19:50:173
106119996,2cyclictest27879-21basename19:45:130
106119980,7cyclictest0-21swapper/020:25:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional