You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-15 - 07:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Dec 15, 2025 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2583329565,10sleep00-21swapper/019:05:260
2587229267,9sleep20-21swapper/219:05:552
2547528874,9sleep30-21swapper/319:05:203
2524727960,17sleep10-21swapper/119:05:181
2626499250,4cyclictest0-21swapper/019:50:140
2626699190,18cyclictest0-21swapper/220:05:132
26266991712,2cyclictest0-21swapper/221:25:142
2626699170,13cyclictest0-21swapper/221:30:132
2626699160,15cyclictest0-21swapper/221:15:152
2626499168,7cyclictest25210-21munin-run22:30:010
2626499161,11cyclictest0-21swapper/021:25:140
26266991511,3cyclictest4750-21wc00:00:192
26266991511,3cyclictest20493-21cut22:20:002
2626599150,14cyclictest0-21swapper/121:00:131
26267991413,1cyclictest20726-21apt-get22:20:153
26266991410,3cyclictest9017-21tr00:10:142
26266991410,3cyclictest4727-21tr19:30:202
2626499141,9cyclictest0-21swapper/023:55:150
221892140,1sleep0161rcuc/022:20:260
2626799138,2cyclictest32638-21apt-get20:30:123
26267991310,2cyclictest9829-21grep00:10:203
26267991310,2cyclictest508-21sed19:20:283
26267991310,2cyclictest2872-21grep20:35:213
26267991310,2cyclictest19793-21sed22:15:273
26267991310,2cyclictest11833-21tail00:15:173
2626799130,12cyclictest11361-21sed20:50:273
2626699139,3cyclictest826-21grep20:30:162
2626699139,3cyclictest8256-21grep23:00:182
2626699139,3cyclictest2467-21tr23:55:222
2626699139,3cyclictest19483-21tr00:30:222
2626699139,3cyclictest13217-21cron00:20:002
2626699136,3cyclictest225-21systemd-journal00:09:432
26266991312,1cyclictest0-21swapper/221:05:202
26266991310,2cyclictest7079-21cron20:45:002
26266991310,2cyclictest1709-21tr21:40:142
26266991310,2cyclictest14007-21grep23:10:282
26266991310,2cyclictest12080-21latency_hist23:10:002
2626699130,12cyclictest25903-21sed21:20:262
2626699130,12cyclictest16300-21cut19:55:232
2626699130,11cyclictest31544-21tr19:20:172
2626599139,3cyclictest9044-21sed20:45:251
2626599139,3cyclictest10636-21cron22:00:001
26265991310,2cyclictest8761-21grep23:00:221
26265991310,2cyclictest5476-21grep21:45:301
26265991310,2cyclictest5299-21grep19:30:271
26265991310,2cyclictest407-21tr20:30:141
26265991310,2cyclictest31758-21tr21:35:451
26265991310,2cyclictest23942-21wc20:10:231
26265991310,2cyclictest20358-21sh00:35:001
26265991310,2cyclictest18616-21cat21:05:261
2626599130,12cyclictest3361-21sleep19:30:001
2626499139,3cyclictest6035-21cat20:40:200
2626499131,6cyclictest7260-21munin-run23:00:000
26264991310,2cyclictest25241-21cat21:20:190
26264991310,2cyclictest18343-21grep23:20:210
2626799129,2cyclictest9029-21grep20:45:253
2626799129,2cyclictest5719-21awk19:35:003
2626799129,2cyclictest3633-21cat21:45:003
2626799129,2cyclictest31099-21cron23:50:003
2626799129,2cyclictest28755-21awk23:45:003
2626799129,2cyclictest2714-21wc19:25:263
2626799129,2cyclictest25188-21cron22:30:003
2626799129,2cyclictest20913-21cut00:35:173
2626799129,2cyclictest20526-21wc00:35:003
2626799129,2cyclictest15983-21tr00:25:133
2626799128,3cyclictest6374-21head22:55:213
2626799121,9cyclictest381rcuc/321:10:133
2626699129,2cyclictest32112-21sort21:35:472
2626699129,2cyclictest30744-21cut20:25:182
2626699129,2cyclictest26822-21wc20:15:262
2626699129,2cyclictest21127-21tr00:35:182
2626699129,2cyclictest18651-21grep20:00:162
2626699129,2cyclictest1524-21tr22:45:192
2626699129,2cyclictest1524-21tr22:45:192
2626699129,2cyclictest11442-21head20:50:272
2626699128,3cyclictest9842-21awk19:40:262
2626699120,11cyclictest0-21swapper/219:17:382
2626599129,2cyclictest9425-21cat19:40:201
2626599129,2cyclictest6873-21grep21:50:181
2626599129,2cyclictest6641-21sed00:05:171
2626599129,2cyclictest5372-21grep00:00:261
2626599129,2cyclictest23114-21basename22:25:121
2626599129,2cyclictest20859-21cut22:20:151
2626599129,2cyclictest17397-21tr23:20:141
2626599129,2cyclictest16964-21tr22:10:241
2626599129,2cyclictest1629-21grep22:45:221
2626599129,2cyclictest1629-21grep22:45:211
2626599129,2cyclictest13776-21tr00:20:171
2626599129,2cyclictest13230-21munin-run00:20:001
2626599129,2cyclictest12605-21tr20:55:151
2626599129,2cyclictest11882-21cron20:55:001
2626599128,3cyclictest24368-21grep23:35:141
2626599128,3cyclictest14483-21wc23:15:001
2626499129,2cyclictest7640-21sed19:35:270
2626499129,2cyclictest5741-21cat19:35:000
2626499129,2cyclictest24500-21grep23:35:140
2626499129,2cyclictest20992-21cat21:10:240
2626499129,2cyclictest18752-21grep21:05:260
2626499129,2cyclictest1456-21grep19:25:150
2626499129,2cyclictest13216-21cron00:20:000
2626499128,3cyclictest31639-21grep23:50:150
2626499120,7cyclictest12101-21basename23:10:000
2626799119,1cyclictest593-21grep23:50:293
2626799119,1cyclictest25925-21grep21:20:263
2626799119,1cyclictest15804-21grep23:15:203
2626799119,1cyclictest15076-21cat00:20:273
2626799119,1cyclictest13450-21grep19:50:153
2626799119,1cyclictest127332sleep322:00:303
2626799118,2cyclictest6299-21cat19:35:163
2626799118,2cyclictest5018-21tr20:40:143
2626799118,2cyclictest32045-21taskset20:28:433
2626799118,2cyclictest30588-21grep22:40:143
2626799118,2cyclictest29635-21cat22:35:273
2626799118,2cyclictest29464-21cat21:30:143
2626799118,2cyclictest29185-21cat19:15:133
2626799118,2cyclictest25926-21sed23:35:263
2626799118,2cyclictest23648-21wc20:10:203
2626799118,2cyclictest19211-21cron23:24:593
2626799118,2cyclictest17500-21chrt19:58:143
2626799118,2cyclictest13120-21head20:55:183
2626799118,2cyclictest1145-21systemctl21:39:593
2626799118,2cyclictest1012-21grep22:45:163
2626799118,2cyclictest1012-21grep22:45:163
26267991110,1cyclictest0-21swapper/322:05:283
2626799110,2cyclictest0-21swapper/319:10:103
2626699119,1cyclictest28276-21tail23:40:252
2626699118,2cyclictest7357-21chrt20:45:122
2626699118,2cyclictest5693-21grep22:55:152
2626699118,2cyclictest560-21grep23:50:282
2626699118,2cyclictest3995-21tr22:50:222
2626699118,2cyclictest31118-21munin-run23:50:002
2626699118,2cyclictest28761-21tr22:35:192
2626699118,2cyclictest24308-21tr22:25:192
2626699118,2cyclictest1695-21cat19:25:152
2626699118,2cyclictest15871-21ntpq23:15:212
2626699118,2cyclictest14704-21sed00:20:232
2626699118,2cyclictest13564-21sed20:55:222
2626699115,2cyclictest225-21systemd-journal23:40:002
2626699111,8cyclictest10661-21perf22:00:012
2626599119,1cyclictest1972-21cut23:55:181
2626599118,2cyclictest8135-21wc19:39:591
2626599118,2cyclictest32085-21head23:50:171
2626599118,2cyclictest31249-21cat21:35:011
2626599118,2cyclictest31206-21ntp_states20:25:221
2626599118,2cyclictest27152-21df21:25:161
2626599118,2cyclictest26926-21tail20:15:261
2626599118,2cyclictest26405-21grep22:30:181
2626599118,2cyclictest20879-21chrt23:25:271
2626599118,2cyclictest20312-21wc22:20:001
2626599118,2cyclictest15473-21sed19:55:001
2626599118,2cyclictest14093-21head22:05:321
2626599118,2cyclictest13092-21cron22:05:001
2626599117,3cyclictest7087-21sh20:45:001
2626599117,3cyclictest25763-21wc21:20:261
2626599115,2cyclictest225-21systemd-journal21:44:541
2626599111,6cyclictest28385-21grep20:20:181
2626599110,10cyclictest32061-21grep22:40:261
2626499118,2cyclictest9395-21chrt20:49:230
2626499118,2cyclictest8933-21cat21:55:150
2626499118,2cyclictest8778-21expr19:40:150
2626499118,2cyclictest7833-21grep21:50:270
2626499118,2cyclictest4270-21expr21:45:200
2626499118,2cyclictest31890-21cat22:40:250
2626499118,2cyclictest27424-21awk20:20:000
2626499118,2cyclictest22508-21sort23:30:170
2626499118,2cyclictest2097-21sort21:40:160
2626499118,2cyclictest20372-21perf00:35:000
2626499118,2cyclictest18347-21cut22:15:160
2626499118,2cyclictest15880-21chrt00:25:110
2626499118,2cyclictest14992-21grep22:05:410
2626499118,2cyclictest14685-21chrt23:15:110
2626499117,3cyclictest17199-21sed19:55:300
2626499110,10cyclictest27853-21tr23:40:220
2626799108,2cyclictest6917-21ls00:05:193
2626799108,2cyclictest17568-21expr21:05:153
2626799108,1cyclictest8726-21awk21:55:143
2626799108,1cyclictest6992-21basename21:50:193
2626799108,1cyclictest5305-21sed00:00:263
2626799108,1cyclictest4652-21chrt22:52:143
2626799108,1cyclictest4635-21if_enp0s31f621:45:223
2626799108,1cyclictest29706-21seq20:24:113
2626799108,1cyclictest28300-21sed21:25:263
2626799108,1cyclictest26542-21cat22:30:193
2626799108,1cyclictest20617-21cat20:05:163
2626799108,1cyclictest2046-21grep23:55:183
2626799108,1cyclictest20211-21expr23:25:193
2626799108,1cyclictest17407-21grep22:10:283
2626799108,1cyclictest14770-21aten2_r7power_c21:00:143
2626799108,1cyclictest12184-21sensors19:45:233
2626799108,1cyclictest10049-21wget23:05:143
2626799107,2cyclictest7889-21expr23:00:163
2626799107,2cyclictest26059-21head20:15:173
2626799107,2cyclictest23983-21awk23:35:003
2626799107,2cyclictest22175-21tr21:15:153
2626799107,2cyclictest13181-21cat23:10:183
2626799100,9cyclictest197292sleep320:01:063
2626699108,1cyclictest78672sleep219:37:252
2626699108,1cyclictest22300-21sed22:20:262
2626699108,1cyclictest21401-21chrt21:12:212
2626699108,1cyclictest18060-21cut23:20:192
2626699108,1cyclictest129962sleep222:03:492
2626699107,2cyclictest6732-21sed21:50:172
2626699107,2cyclictest4025-21grep21:45:192
2626699107,2cyclictest3818-21head20:35:292
2626699107,2cyclictest31680-21tr22:40:222
2626699107,2cyclictest28378-21taskset19:12:072
2626699107,2cyclictest28180-21sed20:20:162
2626699107,2cyclictest26251-21sed22:30:172
2626699107,2cyclictest14831-21grep21:00:142
2626699107,2cyclictest12471-21tail19:45:262
2626699105,1cyclictest23052-21pmu-power23:30:222
2626699104,5cyclictest32-21ksoftirqd/220:10:162
2626599108,1cyclictest72202sleep122:59:461
2626599108,1cyclictest3324-21grep22:50:161
2626599108,1cyclictest31380-21cut19:20:161
2626599108,1cyclictest29869-21grep19:15:171
2626599108,1cyclictest26574-21cut19:10:111
2626599108,1cyclictest2608-21cut20:35:201
2626599108,1cyclictest236562sleep121:15:481
2626599108,1cyclictest220572sleep120:05:281
2626599108,1cyclictest20199-21grep21:10:161
2626599108,1cyclictest178982sleep120:00:031
2626599108,1cyclictest17362-21chrt19:56:271
2626599107,2cyclictest29701-21tail22:35:271
2626599107,2cyclictest29155-21wc23:45:161
2626599107,2cyclictest28433-21chrt23:41:131
2626599107,2cyclictest23839-21sshd23:33:321
2626599107,2cyclictest22721-21chrt00:37:181
2626599107,2cyclictest16417-21head23:15:271
2626599107,2cyclictest16026-21wc00:25:131
2626599102,7cyclictest10620-21grep23:05:171
2626599101,5cyclictest225-21systemd-journal00:15:001
2626499108,2cyclictest19963-21chrt20:04:030
2626499108,1cyclictest9114-21expr00:10:150
2626499108,1cyclictest7534-21apache_accesses23:00:140
2626499108,1cyclictest595-21sshd19:20:310
2626499108,1cyclictest4693-21taskset22:52:460
2626499108,1cyclictest319-21taskset22:45:090
2626499108,1cyclictest319-21taskset22:45:090
2626499108,1cyclictest30968-21uname20:25:190
2626499108,1cyclictest277302sleep019:10:180
2626499108,1cyclictest24134-21ls20:10:260
2626499108,1cyclictest21618-21sed00:35:210
2626499108,1cyclictest19904-21expr23:25:170
2626499108,1cyclictest11263-21expr22:00:150
2626499108,1cyclictest11146-21expr19:45:160
2626499107,2cyclictest7743-21sed00:05:270
2626499107,2cyclictest30612-21cat21:30:250
2626499107,2cyclictest20558-21tr20:05:150
2626499107,2cyclictest12439-21tr20:55:140
2626499107,2cyclictest10960-21tr20:50:230
2626499100,9cyclictest31434-21chrt21:35:260
2626499100,9cyclictest26834-21ntpq22:30:220
262679996,2cyclictest8582-21ls19:40:143
262669997,2cyclictest21201-21awk23:25:292
262669997,1cyclictest14406-21awk19:50:222
262669997,1cyclictest133412sleep222:05:182
262669996,2cyclictest16117-21grep00:25:132
262669996,2cyclictest15743-21apt-get22:10:152
262659997,1cyclictest11558-21ls19:45:181
262649998,1cyclictest30974-21chrt23:48:250
262649998,1cyclictest30964-21taskset19:19:470
262649998,1cyclictest28698-21cut20:20:200
262649998,1cyclictest15236-21expr21:00:150
262649997,1cyclictest658-21expr20:30:140
262649997,1cyclictest36892sleep000:00:110
262649997,1cyclictest28336-21cut22:35:160
262649997,1cyclictest2596-21taskset20:35:190
262649997,1cyclictest15735-21apache_accesses22:10:140
262649997,1cyclictest13646-21cut23:10:230
262649992,3cyclictest225-21systemd-journal00:25:000
262649987,1cyclictest21960-21grep21:15:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional