You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-23 - 19:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jun 23, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2542529682,9sleep20-21swapper/207:07:072
2531929265,9sleep30-21swapper/307:05:433
2537728468,12sleep10-21swapper/107:06:301
2351328268,9sleep00-21swapper/007:05:040
2574099262,14cyclictest2277-21ssh11:53:370
2574099260,26cyclictest0-21swapper/010:11:080
2574699250,25cyclictest0-21swapper/211:43:542
2575199232,19cyclictest0-21swapper/310:14:143
25751992218,3cyclictest39-21ksoftirqd/312:31:383
2575199221,18cyclictest0-21swapper/309:14:433
25743992221,1cyclictest0-21swapper/110:18:331
2574399210,20cyclictest18745-21diskmemload11:51:011
2574399200,20cyclictest0-21swapper/112:28:051
2574399200,20cyclictest0-21swapper/111:03:161
21572200,0sleep20-21swapper/212:12:342
25751991912,3cyclictest537-21snmpd11:16:223
2574699191,17cyclictest0-21swapper/208:00:002
25751991817,1cyclictest11026-21kworker/3:112:27:423
2575199180,18cyclictest11026-21kworker/3:112:38:333
2575199180,18cyclictest11026-21kworker/3:110:40:433
2575199180,15cyclictest0-21swapper/309:59:393
2575199180,0cyclictest0-21swapper/312:22:013
2574399180,18cyclictest0-21swapper/110:47:031
2574099182,8cyclictest28188-21munin-run09:35:000
25740991817,1cyclictest0-21swapper/010:03:460
2575199170,17cyclictest22532-21kworker/3:209:46:333
25746991715,2cyclictest0-21swapper/211:12:442
2574699171,2cyclictest141rcu_preempt12:24:222
2574699170,3cyclictest0-21swapper/211:02:412
2574399170,3cyclictest0-21swapper/112:11:391
2574399170,17cyclictest0-21swapper/112:03:371
224272170,0sleep20-21swapper/210:09:482
2575199163,9cyclictest0-21swapper/308:00:003
25751991615,1cyclictest0-21swapper/310:29:443
25751991615,1cyclictest0-21swapper/310:19:553
2574699162,13cyclictest0-21swapper/211:08:492
2574699160,1cyclictest0-21swapper/211:53:372
2574399169,5cyclictest537-21snmpd10:59:051
25743991615,1cyclictest0-21swapper/109:31:521
25743991614,1cyclictest0-21swapper/107:50:201
2574399160,16cyclictest0-21swapper/110:04:541
25740991616,0cyclictest0-21swapper/011:45:510
25740991616,0cyclictest0-21swapper/011:28:160
2574099161,1cyclictest0-21swapper/010:50:130
267222150,4sleep10-21swapper/109:52:391
2575199155,7cyclictest13668-21apt12:19:593
2575199151,13cyclictest0-21swapper/311:46:513
25746991511,2cyclictest10158-21head08:50:152
2574699150,1cyclictest0-21swapper/209:50:582
25743991513,2cyclictest0-21swapper/111:44:501
25743991511,2cyclictest30705-21grep08:25:181
25743991511,2cyclictest30705-21grep08:25:171
2574399150,2cyclictest0-21swapper/111:05:501
2574399150,15cyclictest0-21swapper/111:18:171
2574399150,13cyclictest0-21swapper/109:44:061
2574099153,7cyclictest22715-21ssh10:10:010
2574099150,15cyclictest0-21swapper/012:35:330
25751991410,3cyclictest5604-21latency_hist07:35:003
25751991410,3cyclictest29686-21cron08:25:013
25751991410,3cyclictest22569-21ls12:05:223
2575199140,14cyclictest0-21swapper/310:48:203
25746991412,1cyclictest0-21swapper/210:28:362
25746991411,2cyclictest3043-21run-parts09:20:002
25746991411,2cyclictest26149-21wc08:15:192
25746991411,2cyclictest14156-21ssh09:45:162
25746991410,3cyclictest32056-21cron08:30:012
25746991410,3cyclictest32056-21cron08:30:002
25746991410,3cyclictest22535-21cron08:10:002
25746991410,3cyclictest18745-21diskmemload10:16:092
2574699140,5cyclictest0-21swapper/209:23:282
25743991413,1cyclictest0-21swapper/109:39:331
2574399141,13cyclictest0-21swapper/109:14:191
25743991410,2cyclictest0-21swapper/110:25:501
2574399140,1cyclictest0-21swapper/109:23:371
2574399140,1cyclictest0-21swapper/109:15:551
2574399140,13cyclictest0-21swapper/110:30:191
2574099142,6cyclictest13674-21cron12:19:590
25740991410,2cyclictest24944-21grep10:30:160
2574099140,1cyclictest0-21swapper/010:47:570
2575199139,3cyclictest28602-21grep11:10:273
2575199138,2cyclictest14148-21date10:05:013
2575199131,12cyclictest0-21swapper/309:41:113
25751991310,2cyclictest8142-21grep08:45:203
25751991310,2cyclictest7586-21head07:35:273
25751991310,2cyclictest31553-21sort07:20:163
25751991310,2cyclictest30206-21cut12:10:143
25751991310,2cyclictest16323-21perf09:04:593
2575199130,12cyclictest29493-21grep09:35:163
25746991311,1cyclictest0-21swapper/210:04:182
25746991310,2cyclictest29245-21date12:10:002
25746991310,2cyclictest27279-21wc09:14:592
25746991310,2cyclictest27012-21ssh10:50:222
25746991310,2cyclictest22897-21basename08:10:142
25746991310,2cyclictest1579-21grep08:30:252
2574699130,13cyclictest0-21swapper/210:24:432
2574699130,11cyclictest0-21swapper/210:10:032
2574399139,3cyclictest8110-21cat08:45:211
2574399139,3cyclictest24082-21grep10:10:171
2574399134,9cyclictest21983-21munin-node11:24:461
2574399131,11cyclictest0-21swapper/109:27:591
25743991310,2cyclictest5166-21grep07:30:261
25743991310,2cyclictest4608-21sh09:59:121
25743991310,2cyclictest4383-21cron08:40:001
25743991310,2cyclictest31093-21tr07:20:141
25743991310,2cyclictest29685-21cron08:25:001
25743991310,2cyclictest29411-21wc12:10:001
25743991310,2cyclictest24020-21ls08:10:221
25743991310,2cyclictest20680-21grep11:45:161
25743991310,2cyclictest15176-21cat09:00:161
2574399130,1cyclictest0-21swapper/111:13:271
2574399130,12cyclictest32588-21tr08:30:131
2574399130,12cyclictest27570-21taskset07:10:241
2574099139,3cyclictest7306-21grep08:45:160
2574099139,3cyclictest20157-21cron08:05:010
2574099139,3cyclictest18199-21wc09:05:230
2574099139,2cyclictest14414-21tr09:00:110
2574099137,4cyclictest25732-21cyclictest10:23:030
2574099131,6cyclictest2039-21awk08:35:010
25740991311,2cyclictest13-21ksoftirqd/012:32:420
25740991310,2cyclictest29289-21grep07:15:170
25740991310,2cyclictest2312-21tail10:55:170
25740991310,2cyclictest19838-21cron09:29:590
25740991310,2cyclictest14793-21wc07:50:260
25740991310,2cyclictest1168-21taskset11:14:180
25740991310,2cyclictest1163-21grep10:35:160
2574099130,7cyclictest3059-21awk11:35:010
2574099130,7cyclictest17804-21awk08:00:000
2575199129,2cyclictest623-21cut10:35:143
2575199129,2cyclictest3716-21grep11:35:153
2575199129,2cyclictest3716-21grep11:35:143
2575199129,2cyclictest29542-21grep09:15:273
2575199129,2cyclictest25783-21tr10:50:153
2575199129,2cyclictest25245-21head10:30:173
2575199129,2cyclictest24524-21head08:10:273
2575199129,2cyclictest18274-21tail09:05:243
2575199129,2cyclictest12545-21sed08:55:163
2575199129,2cyclictest11897-21ssh11:40:113
2575199128,3cyclictest32581-21tr08:30:133
2575199128,2cyclictest3596-21ssh09:20:123
2575199121,6cyclictest22730-21cron10:10:013
2575199121,2cyclictest0-21swapper/311:30:123
2574699129,2cyclictest9393-21cat11:00:002
2574699129,2cyclictest7700-21grep08:45:172
2574699129,2cyclictest3766-21grep07:30:142
2574699129,2cyclictest3765-21cat08:35:272
2574699129,2cyclictest29549-21head09:35:162
2574699129,2cyclictest2837-21grep11:15:142
2574699129,2cyclictest27812-21tr08:20:142
2574699129,2cyclictest22284-21grep11:45:282
2574699129,2cyclictest16641-21latency_hist10:45:002
2574699129,2cyclictest163452sleep211:23:042
2574699129,2cyclictest1617-21head07:25:162
2574699128,3cyclictest23886-21grep12:25:192
2574699128,3cyclictest20625-21tr09:30:152
2574699128,2cyclictest31094-21grep12:30:142
2574399129,2cyclictest32249-21grep12:30:191
2574399129,2cyclictest26944-21tr10:50:221
2574399129,2cyclictest15310-21grep09:45:261
2574399127,2cyclictest22491-21sh10:09:501
2574399124,6cyclictest0-21swapper/107:55:121
25743991211,1cyclictest0-21swapper/110:23:041
25743991210,1cyclictest18745-21diskmemload11:27:511
2574399120,11cyclictest18739-21idleruntime-cro09:10:001
2574099129,2cyclictest8230-21sort07:40:010
2574099129,2cyclictest7071-21basename10:20:000
2574099129,2cyclictest4838-21tr11:55:120
2574099129,2cyclictest3979-21grep11:15:210
2574099129,2cyclictest31023-21grep07:20:120
2574099129,2cyclictest23748-21grep09:50:260
2574099129,2cyclictest23573-21grep08:10:170
2574099129,2cyclictest13026-21grep08:55:200
2574099129,2cyclictest11116-21tr07:45:140
2574099126,3cyclictest16762-21apache209:42:520
2574099123,4cyclictest0-21swapper/008:25:150
2574099123,4cyclictest0-21swapper/008:25:140
2574099122,9cyclictest9653-21sh10:40:160
2574099120,1cyclictest0-21swapper/012:23:110
2574099120,11cyclictest21660-21grep08:05:210
2575199119,1cyclictest0-21swapper/311:07:013
2575199118,2cyclictest9578-21grep07:40:193
2575199118,2cyclictest5720-21cat08:40:203
2575199118,2cyclictest4582-21cut11:55:123
2575199118,2cyclictest30573-21sed11:50:253
2575199118,2cyclictest30226-21cut08:25:153
2575199118,2cyclictest30226-21cut08:25:153
2575199118,2cyclictest3005-21grep08:35:173
2575199118,2cyclictest2821-21taskset07:27:383
2575199118,2cyclictest26215-21cut07:10:143
2575199118,2cyclictest25894-21grep08:15:173
2575199118,2cyclictest25461-21ssh11:28:383
2575199118,2cyclictest23734-21cat09:50:273
2575199118,2cyclictest21810-21ssh09:30:233
2575199118,2cyclictest14308-21grep12:00:213
2575199118,2cyclictest13938-21tail07:50:163
2575199118,2cyclictest13449-21ssh09:25:233
2575199118,2cyclictest12104-21ssh11:00:383
2575199118,2cyclictest10758-21ssh11:20:033
2575199117,3cyclictest8759-21wc10:20:193
2575199117,1cyclictest0-21swapper/310:57:163
2574699119,1cyclictest2243-21ssh09:57:182
2574699118,2cyclictest9865-21ssh12:36:082
2574699118,2cyclictest8713-21cut07:40:132
2574699118,2cyclictest8317-21sh09:41:342
2574699118,2cyclictest8160-21ssh12:15:412
2574699118,2cyclictest7275-21ssh11:36:482
2574699118,2cyclictest7275-21ssh11:36:482
2574699118,2cyclictest4927-21cut08:40:152
2574699118,2cyclictest29891-21ssh11:30:332
2574699118,2cyclictest24429-21ssh10:30:142
2574699118,2cyclictest22032-21ssh11:25:572
2574699118,2cyclictest18833-21head08:00:172
2574699118,2cyclictest18090-21head10:45:182
2574699118,2cyclictest16050-21ssh12:01:192
2574699118,2cyclictest13453-21tail08:55:252
2574699118,2cyclictest11932-21cat07:45:202
2574699118,2cyclictest11095-21ssh11:59:002
2574699114,6cyclictest11889-21/usr/sbin/munin09:25:212
2574699114,5cyclictest14227-21aten2_r7power_c09:00:112
2574699113,4cyclictest225-21systemd-journal10:36:372
2574699112,5cyclictest30685-21munin-run07:19:592
2574399118,2cyclictest8017-21basename07:40:001
2574399118,2cyclictest656-21grep10:35:141
2574399118,2cyclictest5749-21cat08:40:191
2574399118,2cyclictest4593-21basename11:55:121
2574399118,2cyclictest3025-21dump-pmu-power07:30:001
2574399118,2cyclictest29792-21grep07:15:211
2574399118,2cyclictest25440-21tr08:15:141
2574399118,2cyclictest21235-21hostname08:05:171
2574399118,2cyclictest18319-21tr08:00:151
2574399118,2cyclictest15995-21wc12:20:281
2574399118,2cyclictest12032-21grep08:55:131
2574399118,2cyclictest11052-21seq10:40:391
2574399111,9cyclictest18745-21diskmemload12:40:001
2574099118,2cyclictest4404-21turbostat.cron08:40:010
2574099118,2cyclictest29725-21ssh09:15:310
2574099118,2cyclictest26249-21cut07:10:140
2574099118,2cyclictest25401-21sort08:15:150
2574099118,2cyclictest24965-21ssh12:25:390
2574099118,2cyclictest24313-21ssh09:12:500
2574099118,2cyclictest22157-21grep12:05:180
2574099118,2cyclictest2003-21cat07:25:200
2574099118,2cyclictest189762sleep012:03:340
2574099118,2cyclictest18565-21grep11:05:130
2574099118,2cyclictest17066-21ssh11:04:280
2574099118,2cyclictest15324-21sed09:45:270
2574099118,2cyclictest13982-21ssh11:21:130
2574099118,2cyclictest11868-21ssh11:40:100
2574099118,2cyclictest10403-21ssh09:24:130
2574099117,3cyclictest30104-21head09:35:210
2574099117,1cyclictest141rcu_preempt09:55:120
2574099110,10cyclictest31763-21unixbench-2d12:10:270
2575199107,2cyclictest9704-21cut08:50:143
2575199107,2cyclictest28937-21cat07:15:143
2575199107,2cyclictest21180-21cut08:05:163
2574699108,1cyclictest26384-21cat07:10:152
2574699108,1cyclictest14623-21swap07:50:242
2574699107,2cyclictest31346-21expr07:20:142
2574699107,2cyclictest16912-21tr09:05:122
2574399108,1cyclictest11257-21cat07:45:151
2574399107,2cyclictest6059-21tr12:15:131
2574399107,2cyclictest3274-21ssh11:35:011
2574399107,2cyclictest3274-21ssh11:35:001
2574399107,2cyclictest27844-21grep11:30:121
2574399107,2cyclictest10157-21tail08:50:151
2574399100,9cyclictest8920-21cat07:40:141
2574099108,1cyclictest55182sleep007:34:010
2574099108,1cyclictest42572sleep011:35:170
2574099108,1cyclictest42572sleep011:35:160
2574099108,1cyclictest15751-21ssh10:25:030
2574099107,2cyclictest8347-21sed07:40:000
2574099106,3cyclictest29701-21munin-run08:25:000
257519997,1cyclictest19978-21chrt08:02:463
257469997,1cyclictest5911-21awk07:35:142
257409996,2cyclictest5534-21if_enp0s31f608:40:170
257409996,2cyclictest10644-21grep08:50:200
257519985,2cyclictest11689-21if_enp0s31f607:45:183
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional