You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 03:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Nov 17, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4813210980,10sleep10-21swapper/119:08:201
4689210561,39sleep30-21swapper/319:06:453
480229075,10sleep00-21swapper/019:08:110
400228565,15sleep20-21swapper/219:05:242
503499330,1cyclictest0-21swapper/322:51:093
503499330,1cyclictest0-21swapper/322:47:093
503499330,1cyclictest0-21swapper/321:47:493
503099330,3cyclictest23562-21fschecks_time00:15:192
503499320,32cyclictest0-21swapper/322:20:493
503499320,1cyclictest0-21swapper/323:44:093
503499320,1cyclictest0-21swapper/321:51:093
503499320,1cyclictest0-21swapper/300:01:293
5034992725,1cyclictest0-21swapper/319:50:153
503099272,1cyclictest0-21swapper/221:40:192
503099241,2cyclictest0-21swapper/221:55:202
503099232,3cyclictest12078-21awk23:20:002
503099220,1cyclictest0-21swapper/223:37:232
502299225,10cyclictest224-21systemd-journal20:35:000
503099215,8cyclictest37850irq/131-snd_hda21:25:172
5028992113,5cyclictest224-21systemd-journal00:23:411
5028992113,5cyclictest224-21systemd-journal00:23:411
313222210,1sleep30-21swapper/321:10:183
503499202,16cyclictest0-21swapper/321:40:153
503499200,1cyclictest0-21swapper/321:58:493
503099202,3cyclictest29471-21latency_hist22:45:002
502299205,12cyclictest27910-21python23:30:300
503499193,15cyclictest3453-21kworker/u8:023:27:363
503499191,16cyclictest0-21swapper/300:21:083
503499191,16cyclictest0-21swapper/300:21:083
503099191,6cyclictest37850irq/131-snd_hda23:27:282
503099190,19cyclictest0-21swapper/222:34:232
502899190,16cyclictest0-21swapper/122:00:161
503499183,6cyclictest4509-21cat00:25:163
503099182,1cyclictest19495-21ntp_offset23:00:232
503099180,18cyclictest0-21swapper/222:00:532
503099179,7cyclictest27351-21fgrep21:30:242
503099178,1cyclictest6225-21grep22:50:282
503099175,2cyclictest224-21systemd-journal21:55:002
503099173,4cyclictest224-21systemd-journal23:55:002
503099172,1cyclictest0-21swapper/219:35:232
5030991714,3cyclictest0-21swapper/200:22:472
5030991714,3cyclictest0-21swapper/200:22:472
503099170,3cyclictest0-21swapper/221:35:222
503099170,0cyclictest0-21swapper/223:07:312
5028991715,2cyclictest25-21ksoftirqd/122:12:541
502299179,5cyclictest20220-21tune2fs23:25:180
502299171,3cyclictest141rcu_preempt22:49:500
503499168,7cyclictest20679-21munin-run23:50:003
5034991614,1cyclictest0-21swapper/300:17:203
503099168,2cyclictest9918-21rm21:18:062
503099168,2cyclictest8479-21expr22:05:182
503099168,2cyclictest4901-21ssh23:14:362
503099168,2cyclictest19248-21ssh00:11:422
503099168,2cyclictest14138-21ssh23:45:082
503099168,1cyclictest4116-21cat00:00:272
503099163,2cyclictest31652-21idleruntime-cro20:05:002
5030991615,1cyclictest0-21swapper/222:29:212
503099160,10cyclictest1061-21ssh22:47:182
502899163,11cyclictest19973-21cron21:50:001
37462160,0sleep10-21swapper/100:00:231
503499151,9cyclictest0-21swapper/322:44:503
503099158,2cyclictest16982-21ssh21:46:542
503099158,1cyclictest7073-21ssh00:26:312
503099158,1cyclictest5073-21cut21:15:012
503099158,1cyclictest30127-21cat21:05:282
503099157,2cyclictest9104-21cat20:25:002
503099157,2cyclictest31445-21ssh22:21:302
503099157,2cyclictest30676-21rm23:56:122
503099157,2cyclictest19006-21ssh22:12:402
503099157,2cyclictest18885-21ssh00:36:102
503099157,2cyclictest14826-21ssh23:20:512
503099156,2cyclictest21227-21grep19:40:272
503099153,3cyclictest224-21systemd-journal00:08:222
5030991513,1cyclictest0-21swapper/220:40:162
503099151,2cyclictest224-21systemd-journal19:45:002
503099150,9cyclictest14820-21ssh21:21:032
503099150,1cyclictest0-21swapper/222:35:152
502899154,8cyclictest30660-21diskmemload23:37:461
502899153,5cyclictest0-21swapper/120:23:591
502899150,2cyclictest0-21swapper/121:35:321
502299155,5cyclictest224-21systemd-journal22:00:000
5022991514,1cyclictest0-21swapper/022:11:300
5022991511,2cyclictest19287-21wc23:00:210
5022991510,3cyclictest22384-21wc20:50:210
503499144,9cyclictest32630-21/usr/sbin/munin21:35:263
503499143,11cyclictest0-21swapper/300:06:283
5034991410,3cyclictest9247-21grep19:15:273
503499140,10cyclictest0-21swapper/320:05:173
503099148,1cyclictest21974-21diskstats20:50:172
503099148,1cyclictest14504-21ssh00:33:242
503099148,1cyclictest112172sleep220:27:542
503099148,1cyclictest10750-21ssh23:41:552
503099147,3cyclictest15160-21tr20:35:232
503099147,2cyclictest9312-21grep19:15:272
503099147,1cyclictest27957-21ssh23:30:302
503099147,1cyclictest24931-21egrep19:50:162
503099141,1cyclictest224-21systemd-journal22:56:372
503099141,11cyclictest0-21swapper/220:45:262
503099140,3cyclictest15445-21grep19:30:192
5028991414,0cyclictest0-21swapper/121:27:261
5028991413,1cyclictest0-21swapper/121:51:111
5028991411,2cyclictest3202-21apache222:36:321
5022991413,1cyclictest0-21swapper/000:06:110
5022991410,3cyclictest29766-21grep00:20:190
5022991410,3cyclictest29766-21grep00:20:190
5022991410,3cyclictest23547-21cut00:15:190
5022991410,2cyclictest29714-21cut22:20:200
5034991310,2cyclictest10386-21sed22:30:213
503499130,8cyclictest0-21swapper/323:10:233
503499130,12cyclictest0-21swapper/322:35:153
503099138,1cyclictest2700-21cut20:10:182
503099137,1cyclictest72822sleep219:14:012
503099137,1cyclictest28171-21missed_timers19:55:222
503099137,1cyclictest10166-21aten2_r7power_v19:20:132
503099136,1cyclictest24060-21ssh22:15:552
502899137,4cyclictest16137-21cron20:40:001
502899136,2cyclictest493-21systemd-logind23:14:221
502899132,7cyclictest10625-21wget00:30:191
502299139,3cyclictest890-21sed23:10:310
502299139,3cyclictest32640-21grep20:05:210
502299139,2cyclictest20163-21wc21:50:000
5022991310,2cyclictest8185-21cut23:40:170
5022991310,2cyclictest4824-21tr22:50:160
502299130,1cyclictest0-21swapper/022:35:150
502299130,1cyclictest0-21swapper/021:40:500
503499129,2cyclictest9070-21sed22:05:203
503499129,2cyclictest5514-21cut19:10:183
503499129,2cyclictest30162-21ssh23:32:293
503499129,2cyclictest2943-21egrep22:25:153
503499129,2cyclictest28025-21ssh21:30:393
503499129,2cyclictest27855-21ssh23:06:373
503499129,2cyclictest26896-21head21:00:193
503499129,2cyclictest21666-21grep20:50:163
503499129,2cyclictest21073-21ssh22:14:493
503499129,2cyclictest18529-21cron20:45:003
503499129,2cyclictest16441-21grep19:30:283
503499129,2cyclictest13835-21ssh00:32:293
503499129,2cyclictest12427-21ls22:55:263
503499129,2cyclictest12208-21cat23:20:003
503499129,2cyclictest11666-21aten2_r7power_c20:30:153
503499129,2cyclictest10028-21grep20:25:193
5034991212,0cyclictest0-21swapper/323:24:293
503499120,9cyclictest0-21swapper/322:00:163
503099129,2cyclictest5158-21grep20:15:182
502899129,2cyclictest25519-21sed19:50:191
502899129,2cyclictest17991-21cat00:35:301
502899129,2cyclictest17763-21perf23:00:001
502899129,2cyclictest10144-21grep19:20:131
502899129,2cyclictest10086-21tail22:05:301
502899128,3cyclictest9829-21ssh23:40:551
502899128,3cyclictest9142-21chrt21:17:221
502899128,3cyclictest29959-21sh23:55:311
502899128,3cyclictest28359-21ssh23:07:091
502899128,3cyclictest15627-21ssh23:21:431
502899128,3cyclictest15028-21cut19:30:171
502899126,3cyclictest30660-21diskmemload21:32:541
502899126,1cyclictest30660-21diskmemload00:17:131
502899124,4cyclictest224-21systemd-journal23:35:001
502899124,2cyclictest224-21systemd-journal19:27:211
5028991210,1cyclictest12369-21head20:30:181
502899120,11cyclictest9826-21ssh22:53:491
502899120,11cyclictest21202-21wc23:25:281
502899120,11cyclictest1160-21chrt21:11:001
502299129,2cyclictest8543-21tr22:05:180
502299129,2cyclictest6644-21basename20:20:000
502299129,2cyclictest4777-21ssh00:25:170
502299129,2cyclictest31485-21tr21:10:190
502299129,2cyclictest29346-21cut23:55:250
502299129,2cyclictest27685-21grep21:30:270
502299129,2cyclictest25163-21chrt20:55:250
502299129,2cyclictest22614-21grep19:45:160
502299129,2cyclictest17889-21ssh23:00:010
502299129,2cyclictest12032-21cron23:20:000
502299129,2cyclictest11384-21sed22:30:300
502299128,3cyclictest18632-21basename21:25:010
502299128,3cyclictest11880-21wc00:30:270
502299128,2cyclictest2007-21wc21:35:300
502299127,3cyclictest20048-21wc20:45:200
502299126,6cyclictest3207-21apache223:21:560
502299124,4cyclictest224-21systemd-journal19:14:550
502299123,2cyclictest224-21systemd-journal19:18:470
502299120,11cyclictest1166-21ssh23:35:170
503499119,1cyclictest8303-21python20:20:273
503499119,1cyclictest5904-21python20:15:263
503499119,1cyclictest5536-21tr21:15:173
503499119,1cyclictest344-21seq23:35:053
503499119,1cyclictest28926-21sort23:55:203
503499119,1cyclictest28504-21taskset21:05:043
503499119,1cyclictest25601-21ssh22:17:283
503499119,1cyclictest21394-21ssh21:25:523
503499119,1cyclictest19685-21cut20:45:173
503499119,1cyclictest11139-21grep19:20:193
503499118,2cyclictest39-21ksoftirqd/300:36:453
503499118,2cyclictest21712-21grep19:40:323
503499118,2cyclictest17400-21hostname19:35:163
503499118,2cyclictest17018-21grep00:10:213
503499118,2cyclictest14485-21ssh21:20:443
503499114,6cyclictest20602-21ssh23:00:553
503499113,4cyclictest14105-21wget20:35:163
502899119,1cyclictest19069-21gltestperf23:00:181
502899118,2cyclictest9490-21sh23:17:151
502899118,2cyclictest9301-21sh22:30:041
502899118,2cyclictest7345-21tr21:40:171
502899118,2cyclictest5931-21sed00:25:271
502899118,2cyclictest5865-21cut19:10:201
502899118,2cyclictest29789-21cut22:45:141
502899118,2cyclictest28521-21basename21:05:131
502899118,2cyclictest28338-21sed19:55:241
502899118,2cyclictest2754-21ssh22:25:121
502899118,2cyclictest24650-21chrt22:16:321
502899118,2cyclictest24539-21cut20:55:171
502899118,2cyclictest24052-21sed19:45:281
502899118,2cyclictest23829-21ssh23:51:151
502899118,2cyclictest236542sleep122:40:161
502899118,2cyclictest208232sleep120:46:161
502899118,2cyclictest20487-21grep19:40:191
502899118,2cyclictest19305-21chrt19:37:441
502899118,2cyclictest18017-21grep00:10:311
502899118,2cyclictest16890-21ssh23:46:101
502899118,2cyclictest16785-21cat20:40:171
502899118,2cyclictest12324-21sh21:20:161
502899117,3cyclictest31373-21ssh21:57:591
5028991110,1cyclictest0-21swapper/122:20:261
502899110,10cyclictest8993-21sh00:05:021
502299119,1cyclictest7626-21grep20:20:180
502299118,2cyclictest7703-21chrt21:15:520
502299118,2cyclictest4283-21grep00:00:280
502299118,2cyclictest4277-21wc22:25:250
502299118,2cyclictest28872-21ssh23:07:410
502299118,2cyclictest26615-21ssh22:42:130
502299118,2cyclictest22490-21wc23:50:230
502299118,2cyclictest22066-21rm00:14:420
502299118,2cyclictest21815-21cut22:15:140
502299118,2cyclictest20202-21ssh21:50:020
502299118,2cyclictest2009-21test20:10:010
502299118,2cyclictest19522-21expr21:25:170
502299118,2cyclictest18110-21grep19:35:190
502299118,2cyclictest17923-21grep00:35:290
502299118,2cyclictest17139-21grep20:40:190
502299118,2cyclictest14996-21sort19:30:170
502299118,2cyclictest14485-21taskset19:29:370
502299118,2cyclictest14267-21tr20:35:170
502299118,2cyclictest10975-21uname19:20:180
502299117,3cyclictest26846-21sort21:00:190
502299117,3cyclictest20033-21tr19:40:180
503499109,1cyclictest29928-21expr20:00:183
503499109,1cyclictest23185-21ssh23:50:353
503499108,1cyclictest2860-21smartctl20:10:193
503499108,1cyclictest14520-21cron19:30:003
503499102,6cyclictest0-21swapper/319:55:173
503099108,1cyclictest26890-21head21:00:192
503099108,1cyclictest258692sleep221:00:002
503099107,2cyclictest32253-21expr20:05:192
503099107,2cyclictest12303-21grep20:30:182
503099100,9cyclictest13744-21expr19:25:222
502899108,1cyclictest93962sleep119:15:371
502899108,1cyclictest30596-21sed20:00:241
502899108,1cyclictest23057-21grep20:50:281
502899108,1cyclictest1062-21sed20:05:301
502899107,2cyclictest6341-21chrt20:16:391
502899107,2cyclictest40392sleep120:12:381
502899107,2cyclictest26495-21expr21:00:171
502899107,2cyclictest11088-21chrt20:26:171
502299108,1cyclictest28066-21cut19:55:210
502299108,1cyclictest1461-21awk22:00:170
502299107,2cyclictest9554-21expr20:25:170
502299107,2cyclictest28798-21grep21:05:160
502299107,2cyclictest25094-21ls19:50:160
502299107,2cyclictest14561-21ssh23:45:190
50349997,2cyclictest22681-21wc19:45:163
50349997,1cyclictest23728-21seq20:55:053
50229997,1cyclictest29843-21expr20:00:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional