You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-03 - 00:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri May 02, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
819921220,0sleep20-21swapper/209:16:182
6221210591,9sleep10-21swapper/107:06:471
641729177,9sleep30-21swapper/307:09:223
642128265,13sleep20-21swapper/207:09:252
635228168,8sleep00-21swapper/007:08:310
655999220,14cyclictest0-21swapper/112:05:121
655999211,14cyclictest0-21swapper/111:05:151
655999210,17cyclictest0-21swapper/112:37:371
656299200,19cyclictest0-21swapper/212:35:162
6565991912,5cyclictest24263-21ls10:50:283
6559991817,1cyclictest0-21swapper/110:31:251
6556991818,0cyclictest0-21swapper/012:13:400
656599170,13cyclictest0-21swapper/311:05:123
6562991714,2cyclictest500-21in:imuxsock10:40:002
6562991713,2cyclictest0-21swapper/207:15:132
655699174,7cyclictest21442-21awk11:40:000
656599165,10cyclictest8907-21idleruntime-cro07:15:013
6559991616,0cyclictest0-21swapper/111:45:131
655999160,16cyclictest0-21swapper/109:34:421
6565991511,3cyclictest39-21ksoftirqd/309:49:183
656299152,12cyclictest311rcuc/210:45:172
655699155,4cyclictest4150-21apache_processe10:35:150
655699152,6cyclictest25832-21munin-run10:00:000
656599143,5cyclictest11836-21ssh11:55:163
6565991410,3cyclictest4587-21grep08:10:313
6565991410,3cyclictest24280-21cat08:55:003
656599140,8cyclictest0-21swapper/310:40:173
656599140,3cyclictest0-21swapper/310:55:243
656599140,1cyclictest0-21swapper/310:20:173
656599140,1cyclictest0-21swapper/310:20:173
656299149,3cyclictest9991-21cat08:25:002
656299142,1cyclictest141rcu_preempt12:31:322
6562991410,3cyclictest21094-21cut11:15:192
6562991410,3cyclictest20939-21grep09:55:232
656299140,14cyclictest0-21swapper/210:22:492
656299140,14cyclictest0-21swapper/210:22:492
656299140,13cyclictest12468-21head09:20:192
655999148,6cyclictest31057-21apache210:20:431
655999148,6cyclictest31057-21apache210:20:421
6559991413,1cyclictest0-21swapper/109:35:151
6559991410,3cyclictest27778-21grep11:20:151
6559991410,3cyclictest12820-21grep12:15:151
655699143,5cyclictest0-21swapper/010:25:220
6556991412,1cyclictest32444-21sh12:26:260
6556991411,2cyclictest9149-21grep11:05:180
6556991410,2cyclictest16544-21grep07:30:150
655699140,5cyclictest0-21swapper/011:23:470
655699140,2cyclictest0-21swapper/012:33:130
655699140,2cyclictest0-21swapper/011:25:540
655699140,11cyclictest0-21swapper/009:10:140
655699140,10cyclictest0-21swapper/009:05:150
655699140,0cyclictest0-21swapper/011:43:170
24082140,0sleep20-21swapper/210:06:312
656599139,3cyclictest28613-21grep12:05:133
656599139,3cyclictest16551-21tr07:30:153
656599139,3cyclictest14271-21grep12:35:213
6565991310,2cyclictest8382-21tr11:30:173
6565991310,2cyclictest6526-21pluginstate08:15:233
6565991310,2cyclictest30593-21tr12:25:243
6565991310,2cyclictest22781-21grep12:20:263
656599130,12cyclictest19368-21grep08:45:003
656299139,3cyclictest23726-21grep07:45:252
656299139,3cyclictest17916-21basename10:20:002
656299131,9cyclictest37850irq/131-snd_hda08:10:152
6562991310,2cyclictest31375-21sort09:10:002
6562991310,2cyclictest30741-21grep10:55:302
6562991310,2cyclictest28643-21grep12:05:142
6562991310,2cyclictest20087-21cut12:00:142
656299130,1cyclictest0-21swapper/207:50:252
656299130,13cyclictest0-21swapper/212:18:362
656299130,12cyclictest0-21swapper/211:02:522
656299130,11cyclictest0-21swapper/212:10:132
655999139,3cyclictest8284-21basename09:45:181
655999139,3cyclictest5988-21basename10:10:161
6559991312,1cyclictest0-21swapper/112:20:131
6559991310,2cyclictest4382-21cat08:10:311
6559991310,2cyclictest21867-21grep09:55:321
655999130,1cyclictest0-21swapper/112:02:151
655999130,13cyclictest0-21swapper/111:58:401
655999130,12cyclictest4016-21cat11:00:281
655999130,12cyclictest12752-21grep10:15:211
655699139,3cyclictest13380-21head12:15:180
655699139,3cyclictest11798-21tr07:20:150
655699134,5cyclictest224-21systemd-journal08:00:000
655699132,6cyclictest2698-21munin-run08:10:000
6556991310,2cyclictest30768-21grep10:55:300
6556991310,2cyclictest27540-21sed09:00:200
6556991310,2cyclictest14451-21grep09:50:170
6556991310,2cyclictest14291-21grep12:35:200
655699130,7cyclictest8061-21perf11:04:590
655699130,3cyclictest0-21swapper/011:32:000
656599129,2cyclictest8416-21cut08:20:203
656599129,2cyclictest28330-21cut07:55:183
656599129,2cyclictest26536-21ls11:20:003
656599129,2cyclictest24685-21head07:45:323
656599129,2cyclictest2306-21head08:05:273
656599129,2cyclictest22889-21tail11:40:203
656599129,2cyclictest21267-21grep08:45:273
656599129,2cyclictest18974-21tr07:35:193
656599129,2cyclictest16984-21perf08:40:003
656599129,2cyclictest14165-21tr07:25:163
656599129,2cyclictest12027-21cron12:15:003
656599129,2cyclictest1190-21wc10:05:283
656599129,2cyclictest11851-21grep07:20:153
656599129,2cyclictest11533-21cat09:20:003
656599129,2cyclictest11267-21cron11:55:003
656599128,3cyclictest8029-21cron11:04:593
656599126,2cyclictest224-21systemd-journal12:20:003
656599120,1cyclictest0-21swapper/309:43:383
656599120,1cyclictest0-21swapper/309:20:053
656599120,12cyclictest0-21swapper/310:31:053
656599120,11cyclictest0-21swapper/311:39:333
656299129,2cyclictest523-21uname09:10:202
656299129,2cyclictest28273-21wc11:45:012
656299129,2cyclictest23947-21wc10:50:242
656299129,2cyclictest18352-21grep08:40:242
656299129,2cyclictest16570-21tr07:30:152
656299129,2cyclictest16254-21grep11:35:222
656299128,3cyclictest7338-21taskset08:18:422
656299128,3cyclictest263912sleep210:25:592
656299128,3cyclictest15080-21tr08:35:172
656299126,2cyclictest493-21systemd-logind09:01:572
656299125,1cyclictest1291-21runrttasks09:31:282
6562991210,1cyclictest32-21ksoftirqd/211:35:002
656299120,2cyclictest0-21swapper/207:10:162
656299120,1cyclictest0-21swapper/211:26:312
656299120,11cyclictest28951-21tr11:45:142
656299120,11cyclictest22719-21grep08:50:162
655999129,2cyclictest9872-21grep07:15:171
655999129,2cyclictest4341-21grep11:50:171
655999129,2cyclictest31830-21ls08:00:221
655999129,2cyclictest30620-21cat10:55:291
655999129,2cyclictest26060-21tr07:50:171
655999129,2cyclictest24818-21grep10:25:191
655999129,2cyclictest22796-21uniq11:40:191
655999129,2cyclictest17929-21grep07:30:271
655999129,2cyclictest17670-21taskset11:36:111
655999129,2cyclictest11332-21grep08:25:241
655999129,2cyclictest10887-21grep10:40:211
655999128,3cyclictest4505-21grep12:10:141
6559991210,1cyclictest32371-21ssh09:10:161
655999120,11cyclictest5370-21tr08:15:151
655699129,2cyclictest7389-21sort07:10:190
655699129,2cyclictest5995-21expr09:15:160
655699129,2cyclictest31727-21wc10:05:000
655699129,2cyclictest25493-21tr08:55:230
655699129,2cyclictest20002-21ls10:20:290
655699129,2cyclictest20002-21ls10:20:280
655699129,2cyclictest12650-21sed11:55:180
655699128,3cyclictest3705-21apt10:35:000
655699128,3cyclictest3396-21cat09:40:220
655699128,3cyclictest31414-21sed08:00:180
655699128,2cyclictest10777-21grep07:15:260
655699124,5cyclictest0-21swapper/010:15:160
655699121,9cyclictest23183-21munin-run07:45:010
6556991210,1cyclictest16559-21grep08:35:300
655699120,12cyclictest0-21swapper/010:48:300
656599119,1cyclictest24150-21grep09:30:293
656599118,2cyclictest8782-21ssh12:31:463
656599118,2cyclictest7591-21sort10:10:273
656599118,2cyclictest643-21wc09:10:203
656599118,2cyclictest31115-21sh11:45:523
656599118,2cyclictest29406-21ssh11:20:573
656599118,2cyclictest29171-21aten2_r7power_e09:05:153
656599118,2cyclictest28894-21grep09:35:163
656599118,2cyclictest28029-21sed10:00:303
656599118,2cyclictest26472-21latency_hist09:00:003
656599118,2cyclictest25895-21tail10:25:293
656599118,2cyclictest21480-21wc09:55:273
656599118,2cyclictest20323-21grep12:00:153
656599118,2cyclictest19574-21cron09:55:003
656599118,2cyclictest1925-21ssh11:25:203
656599118,2cyclictest17064-21chrt10:19:063
656599118,2cyclictest15720-21ssh11:11:073
656599118,2cyclictest11817-21head08:25:293
656599116,3cyclictest0-21swapper/309:25:293
656599115,2cyclictest224-21systemd-journal08:00:403
656299118,2cyclictest7422-21ssh11:51:572
656299118,2cyclictest6030-21sh09:42:592
656299118,2cyclictest32052-21sort10:30:302
656299118,2cyclictest30034-21chrt07:56:492
656299118,2cyclictest29283-21cut12:25:162
656299118,2cyclictest28638-21cat11:20:232
656299118,2cyclictest25915-21sed08:55:282
656299118,2cyclictest25807-21ssh12:22:462
656299118,2cyclictest22940-21taskset07:42:052
656299118,2cyclictest18001-21sed09:25:202
656299118,2cyclictest13239-21grep08:30:182
656299118,2cyclictest12653-21head11:55:192
656299118,2cyclictest12277-21sh09:48:252
656299118,2cyclictest11874-21chrt08:25:422
656299118,2cyclictest11842-21cut07:20:152
656299117,3cyclictest7674-21tail10:10:282
656299112,8cyclictest291irq_work/210:00:182
656299111,8cyclictest28802-21ssh09:35:152
655999119,1cyclictest17421-21grep09:25:171
655999118,2cyclictest9582-21sh12:32:241
655999118,2cyclictest8459-21grep07:10:301
655999118,2cyclictest807-21tr08:05:171
655999118,2cyclictest7387-21grep09:15:271
655999118,2cyclictest5301-21ssh11:27:341
655999118,2cyclictest4941-21grep10:35:191
655999118,2cyclictest31948-21ssh10:05:131
655999118,2cyclictest25074-21grep07:45:361
655999118,2cyclictest23774-21seq08:51:001
655999118,2cyclictest23311-21ssh11:16:331
655999118,2cyclictest2271-21tr09:40:151
655999118,2cyclictest21935-21cat08:50:001
655999118,2cyclictest21295-21ssh10:48:591
655999118,2cyclictest20791-21dump-pmu-power07:40:001
655999118,2cyclictest18478-21head08:40:251
655999118,2cyclictest17568-21ssh09:52:091
655999118,2cyclictest13916-21ssh11:34:321
655999118,2cyclictest12721-21cut08:30:151
655999117,2cyclictest26595-21tr10:00:161
655999112,7cyclictest14411-21apache_processe11:10:131
655999110,11cyclictest0-21swapper/110:53:291
655999110,10cyclictest16495-21taskset09:24:341
655699119,1cyclictest22593-21chrt11:15:470
655699119,1cyclictest175292sleep011:12:250
655699118,2cyclictest9422-21ntpq09:45:240
655699118,2cyclictest7545-21ssh11:52:030
655699118,2cyclictest5081-21munin-plugin-st08:15:000
655699118,2cyclictest29493-21wc12:05:180
655699118,2cyclictest28602-21ssh11:45:090
655699118,2cyclictest26084-21tr07:50:170
655699118,2cyclictest25458-21ssh09:31:270
655699118,2cyclictest24327-21grep10:50:290
655699118,2cyclictest23460-21ssh12:20:550
655699118,2cyclictest13442-21grep09:20:280
655699118,2cyclictest12622-21grep08:30:150
655699117,3cyclictest30141-21proc_pri09:35:240
655699117,3cyclictest20010-21apache_accesses12:00:120
655699115,5cyclictest16065-21awk07:30:000
655699110,10cyclictest8174-21sh10:10:580
655699110,10cyclictest31835-21sh10:05:060
655699110,10cyclictest19575-21ssh09:26:130
656599108,1cyclictest6712-21ssh10:36:133
656599107,2cyclictest28325-21wc09:00:293
656599107,2cyclictest27417-21ls07:50:293
656599107,2cyclictest16960-21cut10:45:183
656599106,2cyclictest12532-21basename08:30:153
656299108,1cyclictest18773-21grep07:35:182
656299108,1cyclictest16871-21ssh09:51:212
656299108,1cyclictest144542sleep211:10:192
656299108,1cyclictest12169-21grep10:40:322
656299107,2cyclictest14727-21which07:25:202
656299103,5cyclictest62050irq/128-enp2s0-11:05:152
656299100,9cyclictest32580-21chrt08:03:022
655999108,1cyclictest8596-21if_enp0s31f608:20:221
655999108,1cyclictest31387-21ssh12:25:361
655999108,1cyclictest15278-21expr08:35:171
655999108,1cyclictest13415-21seq07:21:531
655999107,2cyclictest29086-21taskset09:05:041
655999100,9cyclictest253842sleep108:55:221
655699108,1cyclictest95322sleep008:21:080
655699108,1cyclictest7276-21chrt08:17:540
655699108,1cyclictest18961-21grep08:40:300
655699108,1cyclictest10168-21ssh10:40:060
655699107,2cyclictest22379-21expr08:50:140
655699107,2cyclictest20347-21sed07:35:300
655699107,2cyclictest19902-21tr08:45:140
655699107,2cyclictest10335-21grep08:25:170
65659990,8cyclictest9495-21cstates07:15:153
65659990,5cyclictest224-21systemd-journal07:45:003
65629997,1cyclictest20046-21expr08:45:162
65629997,1cyclictest1176-21fschecks_count08:05:192
65599998,1cyclictest28389-21tr07:55:181
65599997,1cyclictest27507-21tail09:00:211
65599997,1cyclictest158592sleep107:27:481
65569996,2cyclictest23585-21cut07:45:240
65599987,1cyclictest22208-21grep07:40:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional