You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-16 - 20:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot7s.osadl.org (updated Mon Jun 16, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1281210086,9sleep30-21swapper/307:05:433
133429986,9sleep10-21swapper/107:06:221
146129369,8sleep00-21swapper/007:07:570
135227867,6sleep20-21swapper/207:06:352
234432620,0sleep30-21swapper/310:58:203
1710992723,3cyclictest3163-21fschecks_count09:45:172
189292240,0sleep10-21swapper/112:34:111
171399230,7cyclictest0-21swapper/310:45:153
171399221,3cyclictest0-21swapper/311:10:413
171399216,3cyclictest5098-21ssh11:40:003
171399213,3cyclictest0-21swapper/311:55:183
171099210,21cyclictest0-21swapper/210:14:502
171099210,21cyclictest0-21swapper/209:55:482
171099210,19cyclictest0-21swapper/208:25:002
1710992017,3cyclictest32-21ksoftirqd/210:06:442
171099200,20cyclictest0-21swapper/211:18:412
170499205,8cyclictest15971-21awk07:40:000
170499200,20cyclictest0-21swapper/012:15:450
101022200,0sleep20-21swapper/209:23:382
171399199,2cyclictest29094-21grep09:40:173
171399191,2cyclictest0-21swapper/307:35:133
1713991910,2cyclictest6374-21sort11:40:223
1713991910,2cyclictest28915-21wc09:10:333
171099190,1cyclictest0-21swapper/210:46:192
171099190,19cyclictest0-21swapper/210:35:512
170499190,1cyclictest0-21swapper/011:28:170
171399189,2cyclictest31800-21grep10:10:173
171399188,3cyclictest28235-21wc09:40:003
1713991810,2cyclictest7703-21wc07:20:213
1708991817,1cyclictest0-21swapper/111:09:381
170499184,13cyclictest2924-21apt-get08:20:160
171399179,2cyclictest6976-21cut09:20:243
171399179,2cyclictest5524-21grep08:25:153
171399174,3cyclictest0-21swapper/308:30:033
171399174,3cyclictest0-21swapper/308:30:033
171099171,15cyclictest22559-21/usr/sbin/munin12:15:302
1708991717,0cyclictest0-21swapper/112:28:571
170499174,6cyclictest4053-21munin-run07:15:000
171399168,2cyclictest25007-21sed07:55:273
171399162,13cyclictest0-21swapper/312:25:163
171099160,7cyclictest0-21swapper/212:34:292
170899160,16cyclictest0-21swapper/109:48:531
170499162,7cyclictest18590-21perf11:25:000
170499160,1cyclictest0-21swapper/010:30:110
1710991514,1cyclictest0-21swapper/209:44:382
1710991510,3cyclictest32398-21cut11:35:222
1708991514,1cyclictest0-21swapper/111:19:421
77592140,0sleep00-21swapper/010:45:150
319022140,1sleep20-21swapper/211:05:332
1713991414,0cyclictest0-21swapper/312:21:163
1713991412,1cyclictest0-21swapper/311:46:023
1713991411,2cyclictest13277-21cut08:40:193
171099149,2cyclictest9305-21cat12:05:222
1710991413,1cyclictest37850irq/131-snd_hda11:40:192
1710991411,2cyclictest21093-21uname10:00:232
1710991410,2cyclictest21219-21wc08:55:282
171099140,1cyclictest0-21swapper/211:58:022
171099140,14cyclictest0-21swapper/212:12:052
171099140,13cyclictest0-21swapper/211:53:092
170899142,12cyclictest0-21swapper/109:25:151
1708991414,0cyclictest0-21swapper/109:18:191
1708991410,3cyclictest3495-21df11:10:221
170499149,3cyclictest5863-21ls12:25:240
170499142,6cyclictest18614-21apt-get07:45:140
1704991413,1cyclictest0-21swapper/009:23:420
170499140,8cyclictest5132-21systemctl11:40:000
170499140,5cyclictest0-21swapper/012:35:120
171399138,4cyclictest29553-21cron11:05:003
171399136,6cyclictest7744-21snmpd09:25:043
1713991310,2cyclictest6003-21sed07:15:283
1713991310,2cyclictest28088-21sleep10:35:013
1713991310,2cyclictest27338-21grep10:05:303
1713991310,2cyclictest19166-21grep11:25:173
1713991310,2cyclictest11700-21grep07:30:153
1713991310,2cyclictest11220-21grep10:20:193
171399130,12cyclictest2457-21cron11:10:003
171099139,3cyclictest27397-21wc08:00:382
171099139,3cyclictest24709-21sleep11:00:002
171099138,4cyclictest26823-21rm12:39:052
171099138,4cyclictest24829-21tr09:05:192
171099138,4cyclictest18450-21cat07:45:002
171099136,5cyclictest32-21ksoftirqd/210:25:002
171099131,1cyclictest0-21swapper/211:45:172
1710991310,2cyclictest4131-21tr11:10:272
1710991310,2cyclictest18189-21grep10:25:252
171099130,11cyclictest0-21swapper/210:18:332
171099130,11cyclictest0-21swapper/210:18:332
170899139,3cyclictest6992-21sed08:25:261
170899139,3cyclictest4796-21cat10:15:001
170899139,2cyclictest26476-21cat10:05:221
170899131,0cyclictest0-21swapper/112:18:111
1708991310,2cyclictest9824-21grep07:25:171
1708991310,2cyclictest32206-21sed11:35:201
1708991310,2cyclictest19202-21grep11:25:171
1708991310,2cyclictest18950-21grep09:30:311
1708991310,2cyclictest15098-21wc07:35:211
170899130,5cyclictest5190-21ssh10:15:161
170899130,5cyclictest5190-21ssh10:15:161
170499139,3cyclictest15890-21tr12:10:250
170499135,7cyclictest17156-21perf08:50:000
170499135,5cyclictest11533-21df_abs10:20:190
1704991310,2cyclictest8331-21head11:15:230
1704991310,2cyclictest27324-21grep10:05:300
1704991310,2cyclictest1675-21grep09:15:300
1704991310,2cyclictest14681-21grep09:55:190
171399129,2cyclictest5483-21tr10:15:173
171399129,2cyclictest5483-21tr10:15:173
171399129,2cyclictest2098-21wget10:40:163
171399129,2cyclictest17637-21tail10:25:203
171399129,2cyclictest15899-21sh09:55:293
171399129,2cyclictest15770-21ssh10:50:293
171399129,2cyclictest10975-21tail08:35:213
171399129,2cyclictest10087-21sed09:50:293
171399129,2cyclictest0-21swapper/311:22:433
171399128,2cyclictest30599-21cut08:10:163
171399128,2cyclictest14990-21wc12:10:193
1713991211,1cyclictest0-21swapper/309:19:093
171099129,2cyclictest8621-21grep08:30:192
171099129,2cyclictest8621-21grep08:30:182
171099129,2cyclictest8593-21tr09:50:182
171099129,2cyclictest5300-21awk09:20:002
171099129,2cyclictest5235-21sort12:25:202
171099129,2cyclictest29171-21grep12:20:152
171099129,2cyclictest2686-21tr10:40:182
171099129,2cyclictest24280-21sed09:35:252
171099129,2cyclictest19076-21sed08:50:312
171099129,2cyclictest10961-21grep08:35:212
171099129,2cyclictest10705-21sed07:25:272
171099128,3cyclictest6751-21cut07:20:162
171099128,3cyclictest23115-21ssh11:28:112
171099125,3cyclictest27940-21fschecks_time09:10:202
171099124,4cyclictest224-21systemd-journal07:14:592
1710991210,1cyclictest13699-21tr11:20:182
171099120,12cyclictest0-21swapper/210:33:022
170899129,2cyclictest9561-21sed08:30:281
170899129,2cyclictest9561-21sed08:30:281
170899129,2cyclictest3534-21grep08:20:321
170899129,2cyclictest2234-21tr10:40:161
170899129,2cyclictest21361-21ssh12:35:201
170899129,2cyclictest20707-21sort11:50:261
170899129,2cyclictest2006-21grep07:10:141
170899129,2cyclictest16882-21sort07:40:171
170899129,2cyclictest15788-21head08:45:211
170899128,2cyclictest14231-21tr10:50:171
1708991211,0cyclictest0-21swapper/110:00:161
1708991210,1cyclictest28884-21cut09:40:151
170899120,12cyclictest0-21swapper/111:24:561
170499129,2cyclictest8475-21grep08:30:170
170499129,2cyclictest8475-21grep08:30:170
170499129,2cyclictest5955-21grep10:15:200
170499129,2cyclictest5955-21grep10:15:200
170499129,2cyclictest5028-21tail07:15:190
170499129,2cyclictest30312-21awk08:10:000
170499129,2cyclictest28234-21sort11:55:310
170499129,2cyclictest26520-21grep08:00:300
170499129,2cyclictest23496-21tr07:55:160
170499129,2cyclictest22205-21grep07:50:240
170499129,2cyclictest18513-21uname08:50:250
170499129,2cyclictest14865-21cat08:45:000
170499128,3cyclictest21147-21grep10:55:280
170499128,3cyclictest1948-21grep12:00:200
170499128,2cyclictest18738-21wc09:30:300
1704991211,1cyclictest0-21swapper/007:40:140
170499120,2cyclictest0-21swapper/009:39:510
170499120,11cyclictest28998-21ssh09:40:160
171399118,2cyclictest8815-21sh11:15:273
171399118,2cyclictest3174-21expr08:20:303
171399118,2cyclictest3126-21ssh12:00:303
171399118,2cyclictest30090-21taskset08:07:313
171399118,2cyclictest28660-21grep10:35:203
171399118,2cyclictest24195-21ssh10:04:043
171399118,2cyclictest23085-21sh07:55:003
171399118,2cyclictest22901-21grep09:00:223
171399118,2cyclictest21691-21ssh11:50:453
171399118,2cyclictest21122-21ssh12:15:183
171399118,2cyclictest18206-21cut09:30:243
171399118,2cyclictest17566-21systemctl07:40:233
171399118,2cyclictest15785-21sed08:45:213
171399118,2cyclictest13298-21grep12:30:213
171399118,2cyclictest1197-21tune2fs08:15:183
171399118,2cyclictest10865-21chrt07:25:553
171399114,5cyclictest496-21dbus-daemon11:30:273
171399111,7cyclictest21081-21df12:35:193
171099118,2cyclictest5943-21grep07:15:272
171099118,2cyclictest21732-21tail07:50:192
171099118,2cyclictest13396-21taskset07:32:522
171099118,2cyclictest13146-21mailstats09:25:252
171099117,3cyclictest7476-21sort08:30:002
171099117,3cyclictest25847-21grep11:30:202
171099117,3cyclictest18736-21cat09:30:302
171099115,5cyclictest15240-21cut10:50:242
171099113,4cyclictest224-21systemd-journal08:50:002
171099112,5cyclictest0-21swapper/212:00:182
170899119,1cyclictest29302-21sort10:35:241
170899119,1cyclictest28895-21grep08:05:181
170899118,2cyclictest9712-21ssh11:42:371
170899118,2cyclictest8999-21cut09:50:211
170899118,2cyclictest559-21hostname08:15:141
170899118,2cyclictest31986-21sort08:10:281
170899118,2cyclictest31633-21ssh12:21:051
170899118,2cyclictest30968-21ssh11:57:561
170899118,2cyclictest29861-21ssh09:11:521
170899118,2cyclictest28205-21basename09:40:001
170899118,2cyclictest26872-21cat09:10:001
170899118,2cyclictest25995-21grep11:00:201
170899118,2cyclictest22601-21tr10:30:161
170899118,2cyclictest22480-21cat09:00:201
170899118,2cyclictest2068-21head12:00:201
170899118,2cyclictest19736-21chrt08:55:081
170899118,2cyclictest19360-21head07:45:181
170899118,2cyclictest17860-21uname10:25:221
170899118,2cyclictest17263-21ssh11:48:161
170899118,2cyclictest16772-21ssh12:10:441
170899118,2cyclictest14938-21ssh09:55:211
170899118,2cyclictest13362-21head08:40:201
170899118,2cyclictest13211-21chrt10:49:451
170899118,2cyclictest12343-21taskset08:39:541
170899118,2cyclictest10232-21ssh12:05:361
170899117,3cyclictest27923-21apt08:05:001
170899110,10cyclictest24678-21ssh10:59:591
170899110,10cyclictest121352sleep110:20:231
170499119,1cyclictest3587-21cat10:40:270
170499119,1cyclictest13357-21head11:45:230
170499118,2cyclictest9719-21sh11:42:370
170499118,2cyclictest7206-21grep07:20:170
170499118,2cyclictest7113-21sh11:15:000
170499118,2cyclictest32012-21ssh10:10:190
170499118,2cyclictest30945-21taskset11:05:190
170499118,2cyclictest30734-21ssh09:12:530
170499118,2cyclictest29712-21tail12:20:180
170499118,2cyclictest24999-21cat11:30:150
170499118,2cyclictest22764-21ssh11:51:480
170499118,2cyclictest20490-21head08:55:200
170499118,2cyclictest17699-21sed10:25:200
170499118,2cyclictest17531-21ssh12:33:020
170499118,2cyclictest13765-21ssh10:50:050
170499118,2cyclictest12447-21tail09:25:190
170499118,2cyclictest11885-21expr07:30:170
170499118,2cyclictest10640-21ssh09:51:000
171399108,1cyclictest26311-21sed08:00:283
171399108,1cyclictest25475-21cut09:05:243
171399108,1cyclictest18545-21chrt08:50:263
171399107,2cyclictest8022-21ssh12:05:043
171399107,2cyclictest2544-21cut09:45:143
171399107,2cyclictest19464-21if_enp0s31f607:45:193
171399106,3cyclictest2209-21tr07:10:143
171099108,1cyclictest27319-21chrt11:00:442
171099108,1cyclictest23462sleep208:16:312
171099108,1cyclictest22805-21grep09:00:222
171099108,1cyclictest18960-21expr07:45:162
171099108,1cyclictest13369-21expr08:40:192
171099107,2cyclictest32132-21cat08:10:282
171099107,2cyclictest15484-21tail07:35:262
171099106,3cyclictest24090-21head07:55:182
170899108,1cyclictest7755-21ssh09:20:381
170899108,1cyclictest21016-21wget07:50:151
170899108,1cyclictest12780-21grep07:30:231
170899107,2cyclictest4292-21basename07:15:131
170899107,2cyclictest25382-21cat11:30:181
170499108,1cyclictest8196-21ssh12:05:140
170499108,1cyclictest27689-21sh11:01:150
170499108,1cyclictest23184-21load09:00:260
170499107,2cyclictest2368-21sh09:45:030
170499107,2cyclictest21761-21grep10:00:300
170499106,3cyclictest9806-21grep07:25:170
170499106,3cyclictest30426-21grep10:35:340
170499104,3cyclictest224-21systemd-journal08:15:010
170499100,9cyclictest7410-21/usr/sbin/munin08:30:000
170499100,9cyclictest2095-21cat08:15:260
17139997,1cyclictest19978-21grep08:55:173
17109997,1cyclictest29931-21chrt08:05:292
17089997,1cyclictest7572-21cat07:20:201
17089997,1cyclictest23418-21basename07:55:151
17089997,1cyclictest18579-21open_files08:50:271
17049997,1cyclictest24876-21expr09:05:190
17049996,2cyclictest10288-21wget08:35:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional