You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 23:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Dec 06, 2025 12:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2720229885,9sleep30-21swapper/307:07:373
2729529582,9sleep10-21swapper/107:08:481
2738128666,9sleep00-21swapper/007:09:550
2726327868,6sleep20-21swapper/207:08:242
2747699250,23cyclictest0-21swapper/210:47:442
2747699220,20cyclictest0-21swapper/211:54:022
2747699210,20cyclictest0-21swapper/212:09:322
2746899211,3cyclictest0-21swapper/012:09:190
2747699200,20cyclictest0-21swapper/212:02:472
27480991919,0cyclictest0-21swapper/312:01:223
2747699190,1cyclictest0-21swapper/212:30:162
2747699190,19cyclictest0-21swapper/211:55:212
2747399190,13cyclictest0-21swapper/110:20:141
27480991814,3cyclictest94-21kswapd009:54:003
2748099180,2cyclictest0-21swapper/310:40:003
2746899180,4cyclictest0-21swapper/010:20:090
27476991715,1cyclictest0-21swapper/209:42:192
2747699170,16cyclictest0-21swapper/210:07:302
2748099160,1cyclictest0-21swapper/311:06:403
2748099160,1cyclictest0-21swapper/310:40:153
27476991613,2cyclictest32742-21munin-plugin-st09:20:002
27476991612,2cyclictest516-21gdbus08:10:252
2747699160,16cyclictest0-21swapper/211:17:572
2747399162,14cyclictest0-21swapper/111:53:431
2746899160,6cyclictest0-21swapper/012:20:150
111262160,0chrt0-21swapper/111:57:531
2748099154,7cyclictest0-21swapper/307:35:153
27480991510,4cyclictest20476-21diskmemload11:11:583
27476991512,3cyclictest32-21ksoftirqd/209:31:422
2747699150,15cyclictest0-21swapper/212:24:502
2747699150,15cyclictest0-21swapper/209:28:232
2747699150,0cyclictest0-21swapper/210:31:222
27473991513,1cyclictest20476-21diskmemload12:09:581
27473991511,3cyclictest19173-21munin-run09:59:591
27473991510,5cyclictest24866-21apache211:19:591
27468991514,1cyclictest0-21swapper/012:27:320
2746899150,8cyclictest19201-21idleruntime-cro09:59:590
27480991410,3cyclictest14694-21cut08:55:213
2748099140,1cyclictest141rcu_preempt10:55:063
2748099140,12cyclictest20583-21sort09:10:003
2747699144,5cyclictest5069-21/usr/sbin/munin10:40:232
27476991413,1cyclictest0-21swapper/209:10:272
2747699141,2cyclictest20476-21diskmemload10:54:212
27476991410,3cyclictest5601-21sed11:30:232
2747699140,1cyclictest0-21swapper/212:12:162
2747699140,14cyclictest0-21swapper/211:46:232
27473991411,2cyclictest28782-21wc08:20:001
27468991411,2cyclictest25028-21basename10:30:010
27468991411,2cyclictest23828-21perf08:09:590
27468991410,3cyclictest20581-21grep10:00:180
27468991410,3cyclictest20552-21tr08:00:220
2748099139,3cyclictest31547-21grep08:25:153
2748099137,3cyclictest6633-21fschecks_count10:15:163
27480991311,1cyclictest17035-21sh09:57:403
27480991310,2cyclictest6662-21grep12:15:173
27480991310,2cyclictest32129-21grep11:25:423
27480991310,2cyclictest31723-21wc11:00:293
27480991310,2cyclictest31057-21grep12:10:163
27480991310,2cyclictest25758-21grep10:30:183
27480991310,2cyclictest21683-21grep10:00:283
2748099130,1cyclictest0-21swapper/311:42:523
2748099130,12cyclictest0-21swapper/310:51:143
2747699139,3cyclictest4798-21grep08:35:172
2747699138,4cyclictest9376-21sed08:45:182
2747699132,6cyclictest224-21systemd-journal08:08:112
2747699131,11cyclictest0-21swapper/210:15:142
27476991310,2cyclictest32267-21grep08:25:182
27476991310,2cyclictest2510-21grep09:20:262
27476991310,2cyclictest2510-21grep09:20:262
27476991310,2cyclictest19644-21tr09:35:172
27476991310,2cyclictest17638-21grep07:55:192
2747699130,12cyclictest32647-21tr09:45:152
2747399139,3cyclictest31734-21sh12:10:211
2747399139,3cyclictest26296-21tail10:30:201
2747399139,2cyclictest4110-21grep11:05:151
2747399136,4cyclictest18825-21/usr/sbin/munin11:40:221
27473991310,2cyclictest9113-21grep08:45:171
27473991310,2cyclictest7158-21tail10:15:201
27473991310,2cyclictest301-21wc08:25:241
27473991310,2cyclictest13514-21sort12:20:001
2747399130,12cyclictest20990-21grep09:10:151
2747399130,12cyclictest11763-21grep11:10:221
2746899139,3cyclictest29407-21ssh09:42:200
2746899139,3cyclictest28163-21grep09:15:180
2746899132,5cyclictest1291-21runrttasks11:21:290
27468991310,2cyclictest5836-21ssh09:49:430
27468991310,2cyclictest30740-21grep11:00:200
27468991310,2cyclictest18705-21cat11:40:210
2746899130,7cyclictest6729-21perf09:25:000
2746899130,7cyclictest6729-21perf09:25:000
2746899130,12cyclictest10571-21grep07:40:170
2748099129,2cyclictest7423-21cat11:55:193
2748099129,2cyclictest5750-21grep08:35:273
2748099129,2cyclictest32042-21sed10:10:003
2748099129,2cyclictest32010-21grep12:30:293
2748099129,2cyclictest31513-21apt11:50:013
2748099129,2cyclictest28714-21awk09:15:213
2748099129,2cyclictest26760-21tr08:15:163
2748099129,2cyclictest23444-21grep12:25:243
2748099129,2cyclictest22639-21uname08:05:193
2748099129,2cyclictest22293-21ssh12:05:163
2748099129,2cyclictest19118-21grep11:15:273
2748099129,2cyclictest15660-21grep12:20:253
2748099129,2cyclictest13398-21grep10:20:183
2748099129,2cyclictest11329-21grep11:35:183
2748099128,3cyclictest7213-21sed08:40:173
2748099128,3cyclictest32611-21tr09:45:153
2748099128,3cyclictest18149-21ls07:55:243
2748099126,5cyclictest24866-21apache207:15:163
2748099121,0cyclictest0-21swapper/311:32:553
2747699129,2cyclictest8169-21cut07:35:182
2747699129,2cyclictest10882-21grep11:35:162
2747699128,3cyclictest8338-21taskset08:42:142
2747699128,3cyclictest4197-21wc07:25:262
2747699128,3cyclictest30796-21cut11:00:202
2747699128,3cyclictest14304-21ntpq09:55:222
2747699127,4cyclictest23046-21ssh11:43:522
2747699127,2cyclictest224-21systemd-journal11:24:212
2747699124,4cyclictest224-21systemd-journal07:33:052
2747699123,5cyclictest224-21systemd-journal11:15:002
2747699121,2cyclictest4165-21rm10:13:112
27476991210,2cyclictest32-21ksoftirqd/212:29:192
2747699120,11cyclictest0-21swapper/209:00:182
2747399129,2cyclictest7656-21cut08:40:231
2747399129,2cyclictest6372-21wc08:40:001
2747399129,2cyclictest3325-21sh10:12:091
2747399129,2cyclictest30986-21tail12:30:211
2747399129,2cyclictest30471-21ssh11:25:301
2747399129,2cyclictest29607-21munin-run12:30:001
2747399129,2cyclictest25041-21cpuspeed_turbos11:45:181
2747399129,2cyclictest23845-21expr10:55:191
2747399129,2cyclictest20671-21sed10:00:181
2747399129,2cyclictest1995-21grep07:20:261
2747399129,2cyclictest18581-21tr09:05:171
2747399129,2cyclictest17636-21cut10:50:181
2747399129,2cyclictest14555-21grep12:20:151
2747399129,2cyclictest11073-21tr10:45:151
2747399128,3cyclictest8469-21wc07:35:211
2747399128,3cyclictest13410-21grep07:45:221
2747399128,2cyclictest14515-21grep12:00:171
27473991211,1cyclictest25-21ksoftirqd/112:37:111
2746899129,2cyclictest8557-21grep09:25:230
2746899129,2cyclictest7389-21ssh12:35:380
2746899129,2cyclictest6750-21grep07:30:280
2746899129,2cyclictest6400-21grep10:40:260
2746899129,2cyclictest32513-21tr10:10:190
2746899129,2cyclictest24646-21ssh10:55:240
2746899129,2cyclictest15311-21grep07:50:180
2746899129,2cyclictest14745-21grep09:30:330
2746899129,2cyclictest11395-21cut11:10:200
2746899128,3cyclictest11970-21head08:50:230
2746899128,2cyclictest6918-21ssh12:15:190
2746899126,5cyclictest24855-21apache211:09:050
2746899124,5cyclictest17272-21ssh11:15:140
2746899120,7cyclictest30531-21python08:20:250
2746899120,12cyclictest0-21swapper/012:12:040
2748099119,1cyclictest3387-21grep07:25:173
2748099118,2cyclictest7389-21tr09:25:153
2748099118,2cyclictest731-21ssh09:20:113
2748099118,2cyclictest731-21ssh09:20:113
2748099118,2cyclictest5805-21ssh12:35:193
2748099118,2cyclictest5666-21date10:15:003
2748099118,2cyclictest31748-21ssh11:50:023
2748099118,2cyclictest26588-21expr09:40:193
2748099118,2cyclictest23757-21apt11:20:133
2748099118,2cyclictest21737-21head09:10:183
2748099118,2cyclictest21186-21chrt10:25:513
2748099118,2cyclictest1697-21wc07:20:253
2748099118,2cyclictest15562-21ssh09:31:233
2748099118,2cyclictest12931-21tail07:45:163
2748099118,2cyclictest11694-21df_abs08:50:223
2748099118,2cyclictest11663-21sed10:45:183
2748099118,2cyclictest11627-21taskset07:41:273
2748099111,5cyclictest224-21systemd-journal08:50:003
2747699118,3cyclictest29721-21kworker/2:007:15:132
2747699118,2cyclictest8480-21chrt12:36:052
2747699118,2cyclictest7775-21tr09:50:212
2747699118,2cyclictest6936-21cat12:15:202
2747699118,2cyclictest32240-21uptime11:25:432
2747699118,2cyclictest25279-21cat10:55:312
2747699118,2cyclictest19976-21sort08:00:172
2747699118,2cyclictest1870-21ssh10:36:472
2747699118,2cyclictest16207-21grep07:50:272
2747699118,2cyclictest14859-21ssh10:20:472
2747699117,1cyclictest13192-21chrt08:53:342
2747699116,2cyclictest493-21systemd-logind07:12:212
2747699110,10cyclictest0-21swapper/210:03:582
2747399119,1cyclictest15002-21ls08:55:271
2747399118,2cyclictest8825-21rm11:33:471
2747399118,2cyclictest6970-21expr09:50:161
2747399118,2cyclictest5186-21basename10:40:171
2747399118,2cyclictest3418-21ssh09:21:201
2747399118,2cyclictest3418-21ssh09:21:191
2747399118,2cyclictest30989-21taskset08:21:481
2747399118,2cyclictest30008-21ssh11:00:161
2747399118,2cyclictest27712-21dpkg07:10:131
2747399118,2cyclictest27497-21ssh09:15:151
2747399118,2cyclictest22250-21df08:05:151
2747399118,2cyclictest20457-21ssh10:25:281
2747399118,2cyclictest19038-21cron08:00:001
2747399118,2cyclictest17651-21sed09:00:281
2747399118,2cyclictest17460-21chrt09:33:281
2747399118,2cyclictest16683-21basename07:55:001
2747399118,2cyclictest1175-21ssh09:45:211
2747399118,2cyclictest10846-21tr11:35:161
2747399117,3cyclictest11480-21grep08:50:201
2747399114,6cyclictest26197-21/usr/sbin/munin10:05:161
2747399112,3cyclictest20476-21diskmemload09:35:191
2746899119,1cyclictest320-21grep11:50:180
2746899118,2cyclictest9827-21ssh10:17:340
2746899118,2cyclictest8575-21ssh09:50:340
2746899118,2cyclictest8557-21cron08:45:010
2746899118,2cyclictest7235-21grep11:55:170
2746899118,2cyclictest6180-21grep11:30:270
2746899118,2cyclictest4761-21idleruntime-cro07:30:000
2746899118,2cyclictest32333-21ssh12:30:390
2746899118,2cyclictest31761-21cat07:15:250
2746899118,2cyclictest31028-21grep11:25:330
2746899118,2cyclictest28579-21taskset10:05:560
2746899118,2cyclictest28222-21grep08:15:280
2746899118,2cyclictest26949-21wc10:30:260
2746899118,2cyclictest24499-21expr08:10:160
2746899118,2cyclictest23462-21ssh09:38:000
2746899118,2cyclictest18999-21chrt07:59:380
2746899118,2cyclictest18613-21cut09:05:160
2746899118,2cyclictest17351-21basename09:00:250
2746899118,2cyclictest16419-21ssh10:50:000
2746899118,2cyclictest15124-21sed12:00:190
2746899118,2cyclictest13861-21tr08:55:170
2746899118,2cyclictest13839-21sed07:45:260
2746899118,2cyclictest10280-21cat08:45:280
2746899110,3cyclictest0-21swapper/009:10:140
2746899110,10cyclictest31075-21sh10:35:070
2746899110,10cyclictest19993-21ssh10:51:400
2746899110,10cyclictest12619-21rm11:35:330
2748099108,1cyclictest6870-21chrt07:31:423
2748099108,1cyclictest15448-21if_err_enp2s007:50:193
2748099107,2cyclictest3392-21sed08:30:283
2748099107,2cyclictest29497-21expr08:20:153
2748099107,2cyclictest24462-21expr08:10:163
2748099107,2cyclictest20704-21ssh09:35:243
2748099107,2cyclictest16319-21expr09:00:173
2748099100,4cyclictest0-21swapper/307:10:143
2747699107,2cyclictest3839-21sh11:05:032
2747699107,2cyclictest2903-21grep08:30:232
2747699107,2cyclictest27616-21cut08:15:212
2747699107,2cyclictest18767-21chrt10:25:022
2747699107,2cyclictest18605-21tr09:05:172
2747699107,2cyclictest15712-21turbostat.cron09:00:002
2747699107,2cyclictest12854-21fschecks_time07:45:162
2747699103,4cyclictest278-21systemd-udevd07:24:212
2747399108,1cyclictest7768-21forks09:25:171
2747399108,1cyclictest6796-21chrt07:30:431
2747399108,1cyclictest314172sleep107:15:201
2747399108,1cyclictest28000-21ssh09:40:461
2747399108,1cyclictest20682-21sendmail_mailtr08:00:231
2747399108,1cyclictest1550-21ssh10:36:251
2747399108,1cyclictest1523-21cat08:30:001
2747399107,2cyclictest26827-21ssh11:21:521
2747399107,2cyclictest24048-21acpi08:10:111
2747399107,2cyclictest11428-21grep07:40:261
2746899108,1cyclictest7389-21basename07:35:130
2746899108,1cyclictest32661-21ntpq08:25:220
2746899108,1cyclictest24459-21chrt11:45:020
2746899107,2cyclictest577-21ls07:20:150
2746899107,2cyclictest1921-21tr08:30:160
274809997,1cyclictest20299-21cat08:00:203
274769997,1cyclictest29757-21expr08:20:172
274769996,2cyclictest11250-21wc07:40:252
274739997,1cyclictest3023-21expr07:25:151
274689997,1cyclictest4245-21awk08:35:150
274689996,2cyclictest29320-21grep07:10:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional