You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-31 - 20:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Dec 31, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
70092236162,24sleep30-21swapper/307:06:433
70062235162,60sleep00-21swapper/007:06:400
53052203165,25sleep20-21swapper/207:05:092
71752199164,23sleep10-21swapper/107:08:511
242221640,4sleep3737199cyclictest11:50:163
2993121290,3sleep3737199cyclictest09:10:283
1555421160,4sleep3737199cyclictest11:05:213
215732640,1sleep00-21swapper/007:40:110
29492580,2sleep00-21swapper/011:50:210
737199421,23cyclictest17083-21turbostat12:25:013
737199381,25cyclictest29642-21latency_hist11:40:023
737199381,24cyclictest21195-21latency_hist07:40:013
737199380,15cyclictest87250irq/129-enp1s0-09:49:103
737199372,24cyclictest24887-40dnf11:25:313
737199366,13cyclictest12050irq/123-ahci[0009:00:013
7371993618,15cyclictest28418-21grep09:10:013
737199361,26cyclictest1727-21ntp_states09:20:223
737199361,21cyclictest28414-21diskmemload12:35:263
737199361,21cyclictest28414-21diskmemload12:35:263
737199354,21cyclictest12050irq/123-ahci[0009:30:013
737199353,22cyclictest12050irq/123-ahci[0012:05:023
737199351,4cyclictest22666-21/usr/sbin/munin10:10:173
737199351,26cyclictest23684-21cpuspeed_turbos07:45:103
737199351,22cyclictest30212-21/usr/sbin/munin08:00:163
737199351,21cyclictest0-21swapper/308:25:133
737199351,18cyclictest29981-21latency_hist08:00:013
7354993511,12cyclictest0-21swapper/008:00:010
737199345,19cyclictest751-21rngd07:30:393
737199345,16cyclictest12050irq/123-ahci[0012:10:013
737199342,29cyclictest16837-21tr11:10:143
737199341,24cyclictest16065-21/usr/sbin/munin09:55:233
735499346,13cyclictest18664-21munin-run11:15:010
737199333,26cyclictest30329-21ntp_states10:25:233
737199332,21cyclictest32001-21latency_hist11:45:023
737199331,2cyclictest0-21swapper/308:40:113
737199331,29cyclictest17647-21/usr/sbin/munin08:45:203
737199331,28cyclictest29010-40dnf10:24:553
737199331,27cyclictest22493-21ntp_states07:40:203
737199331,24cyclictest25747-21latency10:15:193
737199331,24cyclictest25747-21latency10:15:193
737199331,20cyclictest12246-21unixbench_multi12:10:273
737199330,23cyclictest0-21swapper/310:10:013
737199322,22cyclictest86950irq/126-enp1s0-11:00:263
737199321,28cyclictest7889-21/usr/sbin/munin10:50:223
7358993214,6cyclictest14480-40dnf07:23:331
737199317,14cyclictest0-21swapper/309:40:203
737199316,11cyclictest28414-21diskmemload09:35:153
737199314,14cyclictest87250irq/129-enp1s0-09:50:153
737199311,6cyclictest26994-21processes07:50:243
737199311,6cyclictest26994-21processes07:50:243
7371993115,13cyclictest1124-21wc11:45:253
737199311,22cyclictest16101-21smart_sda07:25:253
735499314,12cyclictest6438-21awk08:20:010
735499313,13cyclictest6094-21idleruntime-cro12:00:010
96622300,14sleep00-21swapper/012:05:220
737199306,11cyclictest87250irq/129-enp1s0-12:15:203
737199302,3cyclictest32423-21/usr/sbin/munin08:05:153
737199301,3cyclictest19288-21/usr/sbin/munin10:00:203
737199301,26cyclictest22420-21/usr/sbin/munin11:20:263
737199301,25cyclictest5169-21ntp_states11:55:223
735499304,12cyclictest19279-21munin-run12:30:010
735499301,20cyclictest10152-21kworker/0:109:58:270
737199292,24cyclictest6944-21smart_sda10:45:233
737199292,23cyclictest9688-21irqstats07:15:023
737199292,16cyclictest87250irq/129-enp1s0-10:41:543
737199291,4cyclictest5076-21/usr/sbin/munin09:30:173
737199291,24cyclictest6651-21/usr/sbin/munin08:20:263
737199291,24cyclictest12058-21latency_hist11:00:023
737199291,17cyclictest87250irq/129-enp1s0-10:30:033
7371992911,15cyclictest24794-21fschecks_time09:00:173
7371992910,16cyclictest13088-21cut07:20:143
735499293,13cyclictest14621-21munin-run07:25:010
737199288,17cyclictest1424-21cut10:35:123
737199288,17cyclictest12409-21latency_hist07:20:023
737199287,18cyclictest14384-21ls08:35:223
737199287,15cyclictest751-21rngd08:51:333
737199282,23cyclictest5372-21memory08:15:203
737199282,23cyclictest18376-21ntp_states12:25:243
737199281,4cyclictest31192-21/usr/sbin/munin09:15:153
737199281,3cyclictest2257-21/usr/sbin/munin08:10:123
737199281,3cyclictest19679-21/usr/sbin/munin11:15:173
7371992812,13cyclictest19696-21cat12:30:133
735899281,11cyclictest24328-21kworker/1:007:40:231
735499282,12cyclictest0-21swapper/009:00:010
735499281,12cyclictest12699-21awk12:15:010
737199272,22cyclictest26034-21perl11:30:173
737199271,23cyclictest11614-21perl08:30:163
735499273,12cyclictest31849-21munin-run11:45:010
735499273,12cyclictest19008-21munin-run07:35:010
735499271,13cyclictest24638-21munin-run10:15:010
736499268,10cyclictest0-21swapper/209:25:142
735899261,16cyclictest26164-21switchtime10:15:251
735899261,16cyclictest26164-21switchtime10:15:251
735899261,15cyclictest0-21swapper/112:36:531
735899261,15cyclictest0-21swapper/112:36:531
735499261,13cyclictest31236-21munin-run10:30:010
735899241,17cyclictest0-21swapper/111:26:551
735899241,16cyclictest9190-21sshd09:39:301
735899241,15cyclictest0-21swapper/111:49:021
7358992410,4cyclictest11236-21awk10:55:251
735499241,12cyclictest14282-21awk11:05:010
735499240,12cyclictest22453-21perf10:10:010
735899231,4cyclictest2451-21taskset09:22:391
7358992312,6cyclictest29010-40dnf10:24:581
735899231,15cyclictest0-21swapper/112:20:301
7354992310,5cyclictest0-21swapper/011:47:120
273292230,4sleep011-21rcuc/007:50:260
273292230,4sleep011-21rcuc/007:50:260
735899221,7cyclictest0-21swapper/107:35:191
735899221,19cyclictest0-21swapper/108:45:171
735899221,18cyclictest6875-21expr08:20:121
735899221,15cyclictest0-21swapper/111:38:511
735899221,15cyclictest0-21swapper/111:11:161
735499221,8cyclictest0-21swapper/008:45:130
262892221,3sleep126392-21memory11:30:201
736499211,7cyclictest0-21swapper/211:08:152
7364992116,3cyclictest28414-21diskmemload10:14:402
736499211,5cyclictest0-21swapper/210:50:052
735899218,5cyclictest23-21ksoftirqd/112:25:371
735899211,9cyclictest0-21swapper/112:01:531
735899211,17cyclictest19087-21cstates11:15:131
735899211,15cyclictest0-21swapper/110:42:051
735899211,14cyclictest0-21swapper/112:12:261
735899211,11cyclictest19520-21smart_sda10:00:241
735899211,11cyclictest16459-21sh11:10:011
735499216,6cyclictest0-21swapper/011:38:250
735499216,6cyclictest0-21swapper/010:36:090
735499211,12cyclictest26030-21wc10:15:230
735499211,12cyclictest26030-21wc10:15:230
736499204,4cyclictest0-21swapper/212:39:562
736499204,4cyclictest0-21swapper/212:39:562
736499202,15cyclictest28329-21cut11:35:172
736499201,6cyclictest0-21swapper/210:29:212
735899209,3cyclictest14064-21irqstats08:35:181
735899209,3cyclictest10386-21memory09:40:191
735899202,10cyclictest6122-21latency_hist12:00:021
735899201,8cyclictest0-21swapper/111:43:001
7358992017,2cyclictest0-21swapper/108:29:501
7358992014,3cyclictest0-21swapper/110:53:281
735899201,2cyclictest20563-21ntpq12:30:211
7358992012,5cyclictest15418-21grep09:50:261
735899201,18cyclictest0-21swapper/110:05:241
735899201,18cyclictest0-21swapper/108:50:221
735899201,14cyclictest0-21swapper/110:47:481
735899201,14cyclictest0-21swapper/110:14:531
735899201,10cyclictest320-21awk08:05:151
735899201,10cyclictest25755-21cat07:50:011
735899201,10cyclictest17369-21df07:30:151
735899201,10cyclictest15903-21ntp_states07:25:221
735899200,14cyclictest87250irq/129-enp1s0-09:15:541
735499207,5cyclictest0-21swapper/010:30:050
735499205,6cyclictest0-21swapper/010:24:140
735499202,3cyclictest0-21swapper/010:59:500
735499202,11cyclictest29291-21kworker/u8:112:01:040
7354992015,3cyclictest31684-21grep08:00:260
7354992014,3cyclictest13583-21head12:15:160
735499201,10cyclictest2362-21sshd09:21:300
304442201,2sleep20-21swapper/209:13:002
736499195,5cyclictest26540-21ntpq11:30:232
736499194,3cyclictest28414-21diskmemload09:20:352
736499191,8cyclictest3206-21cat08:10:182
736499191,6cyclictest0-21swapper/211:29:272
736499191,6cyclictest0-21swapper/210:05:252
736499191,5cyclictest0-21swapper/210:58:022
735899192,3cyclictest87150irq/128-enp1s0-07:10:131
735899191,7cyclictest0-21swapper/108:12:331
7358991916,2cyclictest0-21swapper/108:41:051
7358991915,3cyclictest0-21swapper/111:24:081
7358991915,3cyclictest0-21swapper/109:58:431
7358991915,3cyclictest0-21swapper/108:18:241
7358991915,3cyclictest0-21swapper/107:54:341
7358991915,3cyclictest0-21swapper/107:54:341
7358991912,2cyclictest0-21swapper/111:02:311
7358991912,2cyclictest0-21swapper/110:36:131
735899191,17cyclictest0-21swapper/107:58:561
735899191,16cyclictest0-21swapper/108:55:021
7358991911,5cyclictest2547-21sed11:50:171
735899191,10cyclictest6137-21/usr/sbin/munin09:30:221
735899191,10cyclictest32039-21fw_packets10:30:171
735899191,10cyclictest30554-21df_inode08:00:141
735899191,10cyclictest13655-21if_enp2s012:15:171
735899190,18cyclictest0-21swapper/110:27:361
735899190,10cyclictest23-21ksoftirqd/109:10:201
735499198,3cyclictest14496-21/usr/sbin/munin11:05:270
735499196,5cyclictest0-21swapper/009:32:390
735499195,5cyclictest0-21swapper/010:46:310
735499193,9cyclictest11080-21sshd09:40:300
735499191,8cyclictest28414-21diskmemload11:18:060
735499191,5cyclictest0-21swapper/011:25:390
7354991912,4cyclictest18864-21head10:00:160
735499191,11cyclictest0-21swapper/009:47:150
735499191,11cyclictest0-21swapper/009:25:170
735499191,10cyclictest2257-21/usr/sbin/munin08:10:130
735499191,10cyclictest21703-21/usr/sbin/munin12:35:210
735499191,10cyclictest21703-21/usr/sbin/munin12:35:210
735499191,10cyclictest15170-21sshd08:39:300
735499191,10cyclictest0-21swapper/009:35:020
170642191,3sleep00-21swapper/012:24:570
736499188,3cyclictest87250irq/129-enp1s0-10:30:162
736499187,3cyclictest9259-21awk09:40:012
736499187,3cyclictest87250irq/129-enp1s0-09:45:262
736499187,3cyclictest86950irq/126-enp1s0-11:10:162
736499181,8cyclictest9820-21dnf09:40:142
736499181,7cyclictest17026-21sshd12:24:302
736499181,6cyclictest0-21swapper/210:22:232
7364991813,3cyclictest13066-21expr12:15:122
735899188,4cyclictest0-21swapper/109:47:071
7358991815,2cyclictest0-21swapper/109:25:171
735899181,16cyclictest0-21swapper/109:00:131
735899181,16cyclictest0-21swapper/107:16:321
735499184,4cyclictest28414-21diskmemload12:31:540
735499183,3cyclictest0-21swapper/011:22:200
735499183,11cyclictest87250irq/129-enp1s0-09:15:150
735499182,13cyclictest9522-21dump-pmu-power07:15:010
735499181,9cyclictest10088-21gzip08:25:240
735499181,10cyclictest13564-21kworker/u8:009:52:460
245852180,2sleep20-21swapper/209:00:162
79852171,2sleep20-21swapper/212:01:162
736499173,4cyclictest6084-21perf12:00:012
736499173,3cyclictest86950irq/126-enp1s0-12:15:012
736499172,7cyclictest4456-21/usr/sbin/munin08:15:142
736499171,9cyclictest0-21swapper/208:28:302
736499171,8cyclictest22056-21cpu08:55:142
736499171,8cyclictest0-21swapper/212:26:182
736499171,6cyclictest27505-21ntp_states09:05:202
736499171,6cyclictest25860-21cpu07:50:122
736499171,6cyclictest25860-21cpu07:50:122
736499171,6cyclictest0-21swapper/209:34:282
736499171,4cyclictest0-21swapper/210:40:132
7364991712,3cyclictest14324-21sshd07:21:302
736499171,15cyclictest0-21swapper/211:02:182
7364991711,4cyclictest29450-21sort07:55:242
736499171,14cyclictest10039-21grep12:05:252
736499171,14cyclictest0-21swapper/207:10:332
736499171,10cyclictest0-21swapper/212:33:062
736499171,10cyclictest0-21swapper/208:00:182
7364991710,5cyclictest11083-21head07:15:152
7358991714,2cyclictest0-21swapper/108:32:051
735899171,2cyclictest0-21swapper/109:05:131
735899170,15cyclictest0-21swapper/112:06:001
735499177,3cyclictest87150irq/128-enp1s0-10:50:240
735499175,3cyclictest0-21swapper/010:40:080
735499173,3cyclictest0-21swapper/011:31:460
735499171,4cyclictest0-21swapper/009:05:010
735499171,3cyclictest0-21swapper/007:25:220
7354991713,3cyclictest0-21swapper/009:11:460
735499171,2cyclictest0-21swapper/007:45:230
735499171,14cyclictest0-21swapper/008:50:460
735499171,10cyclictest0-21swapper/007:15:190
735499170,4cyclictest0-21swapper/008:05:220
216182171,2sleep20-21swapper/208:52:422
207242170,3sleep20-21swapper/211:18:302
154862170,2sleep20-21swapper/209:50:412
151632171,2sleep20-21swapper/208:39:252
736499166,3cyclictest86950irq/126-enp1s0-11:20:192
736499165,3cyclictest87250irq/129-enp1s0-11:50:052
736499161,6cyclictest6022-21df10:45:122
736499161,6cyclictest31194-21entropy09:15:152
736499161,6cyclictest24332-21basename07:45:142
736499161,6cyclictest1271-21/usr/sbin/munin10:35:232
736499161,6cyclictest11638-21perl08:30:162
736499161,6cyclictest0-21swapper/207:40:142
736499161,5cyclictest577-21systemd-journal07:35:012
736499161,5cyclictest0-21swapper/210:03:052
7364991612,3cyclictest0-21swapper/208:43:082
736499161,14cyclictest0-21swapper/208:46:412
736499161,14cyclictest0-21swapper/208:22:252
736499161,14cyclictest0-21swapper/208:09:072
7364991611,3cyclictest25228-21/usr/sbin/munin10:15:152
7364991611,3cyclictest25228-21/usr/sbin/munin10:15:152
7364991611,3cyclictest19514-21expr07:35:132
7364991611,3cyclictest16911-21latency09:55:202
735499165,3cyclictest11-21rcuc/007:35:150
735499161,6cyclictest15783-21df08:40:130
7354991610,3cyclictest26441-21/usr/sbin/munin09:05:230
7354991610,3cyclictest11211-21cut08:30:130
13062161,2sleep20-21swapper/211:45:382
736499151,5cyclictest31461-21users11:40:272
7364991510,3cyclictest16427-21users07:25:272
7354991510,3cyclictest6655-21cat08:20:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional