You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-21 - 09:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 21, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
75552231163,55sleep20-21swapper/219:07:382
75202222193,19sleep10-21swapper/119:07:121
76412205131,61sleep00-21swapper/019:08:440
74972199163,24sleep30-21swapper/319:06:543
3168321800,5sleep0782299cyclictest23:40:250
78349910197,3cyclictest0-21swapper/221:36:382
7834997066,3cyclictest0-21swapper/221:20:442
45702520,3sleep081ktimersoftd/023:55:010
7834993935,3cyclictest0-21swapper/220:22:062
783999384,2cyclictest87150irq/128-enp1s0-21:40:133
782299389,13cyclictest15088-21munin-run19:25:010
7839993712,22cyclictest16538-21/usr/sbin/munin21:55:173
783999371,11cyclictest9010-21/usr/sbin/munin00:05:163
7839993612,21cyclictest3358-21/usr/sbin/munin21:25:253
7839993612,21cyclictest3358-21/usr/sbin/munin21:25:253
7834993633,2cyclictest0-21swapper/219:26:502
7839993513,19cyclictest17164-21/usr/sbin/munin23:10:213
783999341,7cyclictest19475-21latency_hist19:35:023
783999341,6cyclictest9928-21grep20:25:203
7839993412,19cyclictest6019-21/usr/sbin/munin21:30:163
7839993411,3cyclictest19334-21processes20:45:233
783999339,3cyclictest12766-21/usr/sbin/munin23:00:253
783999338,4cyclictest0-21swapper/322:35:283
783999334,4cyclictest25201-21fschecks_time21:00:163
783999332,5cyclictest1309-40dnf22:32:303
783999331,6cyclictest4702-21sh20:15:013
783999331,6cyclictest4702-21sh20:15:013
7839993311,3cyclictest891-21latency_hist21:20:013
7839993311,3cyclictest10920-21/usr/sbin/munin19:15:213
7839993310,3cyclictest4586-21/usr/sbin/munin23:55:173
782299336,12cyclictest15711-21idleruntime-cro20:40:010
783999329,4cyclictest0-21swapper/322:40:143
783999328,2cyclictest1133-21/usr/sbin/munin21:20:183
783999324,14cyclictest751-21rngd19:29:193
783999321,6cyclictest21808-21cat19:40:033
7839993216,13cyclictest1329-21irqstats20:05:183
783999321,5cyclictest3491-21/usr/sbin/munin23:50:203
783999321,5cyclictest3491-21/usr/sbin/munin23:50:203
783999321,23cyclictest0-21swapper/320:04:293
783999319,3cyclictest800-21sshd19:48:263
783999319,3cyclictest7830-21ntpq00:00:223
783999319,2cyclictest9961-21vmstat22:50:283
783999317,3cyclictest6167-21sendmail_mailqu20:15:263
783999311,8cyclictest21396-21timerandwakeup00:30:263
783999311,5cyclictest0-21swapper/320:55:183
783999311,3cyclictest801-21snmpd22:45:363
7839993112,3cyclictest0-21swapper/323:25:033
7839993111,3cyclictest25675-21sh22:15:153
783999311,10cyclictest17806-21/usr/sbin/munin00:25:183
7839993110,3cyclictest30194-21open_files21:10:213
783999309,4cyclictest13184-21munin-plugin-st00:15:013
783999308,4cyclictest14386-21/usr/sbin/munin19:20:243
783999308,3cyclictest28261-21latency_hist19:55:013
783999308,3cyclictest15066-21threads20:35:253
783999308,3cyclictest14342-21/usr/sbin/munin21:50:183
783999305,4cyclictest86950irq/126-enp1s0-23:35:153
783999304,7cyclictest16031-21grep20:40:123
783999301,3cyclictest801-21snmpd22:55:373
7839993010,3cyclictest15850-21grep23:05:203
783999299,4cyclictest7915-21cut21:35:133
783999299,4cyclictest0-21swapper/322:10:243
783999298,3cyclictest28495-21/usr/sbin/munin19:55:203
783999298,3cyclictest13629-21date20:35:013
783999292,5cyclictest1241-21ntp_states23:45:253
783999291,7cyclictest0-21swapper/320:50:493
783999291,6cyclictest13781-21fschecks_count00:15:153
783999291,4cyclictest0-21swapper/323:23:313
7839992910,3cyclictest19787-21ntp_offset22:00:203
783999288,3cyclictest835-21runrttasks19:11:443
783999288,3cyclictest13152-21netstat21:45:203
783999287,3cyclictest31612-21swap23:40:243
783999281,6cyclictest26895-21missed_timers23:30:193
7834992824,3cyclictest0-21swapper/200:29:532
782299285,11cyclictest27916-21crond23:35:000
783999279,3cyclictest28596-21ntp_states22:20:223
783999271,3cyclictest86950irq/126-enp1s0-23:15:123
783999271,3cyclictest86950irq/126-enp1s0-22:25:143
782299271,12cyclictest21985-21perf00:35:010
783999269,3cyclictest22452-21sed22:05:283
783999269,2cyclictest24115-21sshd00:39:303
783999263,12cyclictest751-21rngd19:42:003
783999261,6cyclictest8288-21awk20:20:253
782899267,9cyclictest0-21swapper/121:45:231
782299262,12cyclictest23545-21idleruntime-cro23:25:010
782299261,12cyclictest0-21swapper/020:10:020
104862260,2sleep20-21swapper/220:25:272
783999259,3cyclictest0-21swapper/321:07:453
783999251,4cyclictest0-21swapper/300:21:263
7828992512,11cyclictest18490-21kworker/u8:119:36:581
782899251,11cyclictest9992-21sed00:05:211
151322250,5sleep1782899cyclictest23:05:141
782899246,7cyclictest0-21swapper/121:14:091
782899245,8cyclictest0-21swapper/122:28:441
7834992313,2cyclictest22914-21awk22:10:012
782899231,11cyclictest0-21swapper/123:52:491
782899231,11cyclictest0-21swapper/123:52:491
782299234,13cyclictest32508-21/usr/sbin/munin22:30:160
782299233,11cyclictest24713-21/usr/sbin/munin21:00:180
782299230,21cyclictest0-21swapper/000:15:230
261462232,17sleep126249-21memory22:15:191
783499227,3cyclictest86950irq/126-enp1s0-21:50:272
783499224,10cyclictest0-21swapper/222:00:022
782899224,7cyclictest0-21swapper/100:25:181
782899221,19cyclictest0-21swapper/122:07:491
782899221,18cyclictest9936-21latency_hist21:40:011
782899220,10cyclictest0-21swapper/123:33:381
782299223,11cyclictest24742-21kworker/u8:122:28:270
782299222,8cyclictest2225-21ntp_states21:20:220
782299222,11cyclictest86950irq/126-enp1s0-23:55:120
783499212,8cyclictest775-21dbus-daemon19:30:272
782899218,3cyclictest13679-21memory23:00:211
782899215,6cyclictest0-21swapper/100:31:471
782899213,2cyclictest22118-21systemd-journal20:53:001
782899213,1cyclictest0-21swapper/100:37:261
782899211,9cyclictest11621-21/usr/sbin/munin22:55:221
782899211,9cyclictest0-21swapper/121:23:451
782899211,3cyclictest32603-21hddtemp_smartct22:30:171
782899211,11cyclictest9920-21fw_packets20:25:201
782899211,11cyclictest8398-21smart_sda20:20:261
782899211,11cyclictest31326-21acpi21:15:141
782899211,11cyclictest28156-21/usr/sbin/munin23:35:301
782899211,11cyclictest21901-21df_inode23:20:161
782899211,11cyclictest18379-21latency19:30:201
782899211,11cyclictest17435-21grep20:40:261
782899211,10cyclictest15606-21/usr/sbin/munin00:20:121
782899211,10cyclictest15385-21chrt20:36:101
782299216,6cyclictest0-21swapper/022:13:280
782299216,6cyclictest0-21swapper/000:11:010
782299215,8cyclictest2904-21cpuspeed_turbos20:10:160
782299215,8cyclictest2904-21cpuspeed_turbos20:10:150
782299213,12cyclictest0-21swapper/000:27:380
782299213,11cyclictest21183-21kworker/u8:021:42:590
782299213,11cyclictest21051-21kworker/u8:200:05:510
782299212,12cyclictest0-21swapper/023:06:420
782299212,12cyclictest0-21swapper/022:41:460
782299212,10cyclictest18734-21/usr/sbin/munin22:00:170
7822992115,4cyclictest17365-21sshd00:22:290
782299210,12cyclictest0-21swapper/021:31:180
783499204,13cyclictest31090-21ls21:15:002
782899204,4cyclictest28760-21sshd21:08:301
782899202,8cyclictest20425-21ntp_states23:15:211
782899202,16cyclictest0-21swapper/119:45:041
782899201,8cyclictest0-21swapper/122:03:281
782899201,8cyclictest0-21swapper/121:56:021
782899201,8cyclictest0-21swapper/100:10:201
7828992017,2cyclictest0-21swapper/121:27:061
7828992017,2cyclictest0-21swapper/121:27:061
7828992016,3cyclictest0-21swapper/119:14:241
782899201,2cyclictest11712-21cstates20:30:151
782899201,18cyclictest0-21swapper/120:07:591
782899201,18cyclictest0-21swapper/119:58:121
782899201,17cyclictest0-21swapper/123:45:491
782899201,11cyclictest22296-21fschecks_count19:40:151
782899201,10cyclictest5703-21cut21:30:131
782899201,10cyclictest4375-21chrt20:10:531
782899201,10cyclictest4375-21chrt20:10:531
782899201,10cyclictest24904-21ls21:00:141
782899201,10cyclictest24832-21awk23:25:231
782899201,10cyclictest14680-21df21:50:141
782899201,10cyclictest14672-21/usr/sbin/munin00:15:231
782899201,10cyclictest10541-21sed22:55:011
782899200,18cyclictest0-21swapper/123:55:251
782899200,12cyclictest0-21swapper/100:02:501
782299205,2cyclictest0-21swapper/021:46:450
782299202,12cyclictest0-21swapper/022:15:030
782299202,12cyclictest0-21swapper/021:52:070
782299202,11cyclictest1820-21kworker/u8:123:45:520
782299201,9cyclictest86950irq/126-enp1s0-23:35:280
7822992015,3cyclictest21687-21awk19:40:010
782299201,12cyclictest0-21swapper/023:50:420
782299201,12cyclictest0-21swapper/023:18:060
782299201,12cyclictest0-21swapper/022:53:130
782299201,12cyclictest0-21swapper/021:59:490
782299201,12cyclictest0-21swapper/000:01:270
783499197,7cyclictest0-21swapper/200:19:452
783499192,5cyclictest986-1kworker/u9:100:11:302
783499192,15cyclictest20892-21sed23:15:252
783499191,6cyclictest0-21swapper/223:45:272
783499191,6cyclictest0-21swapper/200:30:232
783499191,4cyclictest0-21swapper/200:20:032
7834991912,4cyclictest5326-21munin-run21:30:022
7834991912,4cyclictest5326-21munin-run21:30:022
783499191,10cyclictest13925-21uname19:20:192
782899194,7cyclictest87250irq/129-enp1s0-22:45:171
782899192,16cyclictest0-21swapper/120:04:191
782899191,9cyclictest3035-21smart_sda22:35:251
782899191,8cyclictest0-21swapper/122:42:351
7828991914,2cyclictest11394-21fschecks_time19:15:171
782899191,3cyclictest0-21swapper/122:24:231
782899191,17cyclictest0-21swapper/123:40:461
782899191,17cyclictest0-21swapper/119:52:121
7828991911,5cyclictest23139-21grep20:55:201
782899190,18cyclictest0-21swapper/121:43:271
782299191,3cyclictest12956-21kworker/u8:222:10:000
782299191,13cyclictest0-21swapper/022:49:570
782299191,11cyclictest0-21swapper/022:55:170
782299191,11cyclictest0-21swapper/021:19:440
782299190,8cyclictest4208-21latency21:25:210
782299190,8cyclictest4208-21latency21:25:210
782299190,12cyclictest24742-21kworker/u8:122:36:210
263402190,4sleep011-21rcuc/019:50:130
7834991813,3cyclictest24618-21tune2fs19:45:152
7834991812,4cyclictest7519-21idleruntime-cro21:35:022
7834991812,4cyclictest19801-21ntpq22:00:212
783499181,15cyclictest0-21swapper/219:10:152
7834991811,4cyclictest24901-21head23:25:232
783499181,10cyclictest0-21swapper/222:29:582
7828991814,3cyclictest0-21swapper/123:10:221
782899181,3cyclictest0-21swapper/119:26:051
782899181,3cyclictest0-21swapper/119:21:561
782899181,16cyclictest0-21swapper/122:11:041
782899181,16cyclictest0-21swapper/120:17:041
782899180,16cyclictest0-21swapper/120:45:181
782299187,3cyclictest801-21snmpd19:45:010
782299186,4cyclictest13083-21grep20:30:270
782299185,6cyclictest751-21rngd21:11:550
782299184,8cyclictest0-21swapper/023:04:150
782299183,7cyclictest16001-21cat20:40:120
782299181,5cyclictest9720-21sh21:40:010
782299181,5cyclictest0-21swapper/022:20:140
782299181,4cyclictest9942-21tail20:25:200
7822991813,3cyclictest20685-21cut20:50:140
782299181,11cyclictest0-21swapper/020:21:120
782299180,6cyclictest7-21ksoftirqd/021:10:000
77722181,2sleep20-21swapper/222:45:292
159182181,2sleep20-21swapper/220:40:082
783499177,3cyclictest86950irq/126-enp1s0-23:11:232
783499174,4cyclictest10495-21idleruntime19:15:032
783499173,4cyclictest0-21swapper/223:36:022
783499172,6cyclictest10915-21diskstats22:55:152
783499172,5cyclictest22292-21munin-run20:55:012
783499172,5cyclictest20347-21gltestperf19:35:172
783499172,3cyclictest0-21swapper/221:49:452
783499171,7cyclictest986-1kworker/u9:121:40:182
783499171,7cyclictest2757-21sort23:50:142
783499171,7cyclictest2757-21sort23:50:142
783499171,7cyclictest1946-21cat20:05:262
783499171,7cyclictest0-21swapper/223:20:102
783499171,5cyclictest0-21swapper/223:00:192
783499171,5cyclictest0-21swapper/222:50:272
783499171,5cyclictest0-21swapper/222:34:022
783499171,4cyclictest0-21swapper/220:55:292
7834991712,3cyclictest23254-21cut22:10:132
783499171,15cyclictest0-21swapper/221:19:312
7834991711,4cyclictest23430-21fgrep00:35:232
7834991711,4cyclictest0-21swapper/222:20:152
7834991711,3cyclictest5483-21sed22:40:252
783499171,10cyclictest0-21swapper/219:50:272
7834991710,4cyclictest0-21swapper/200:00:132
782299177,2cyclictest87150irq/128-enp1s0-20:00:160
782299174,4cyclictest19003-21sshd23:13:300
782299171,8cyclictest22480-21expr00:35:120
782299171,3cyclictest0-21swapper/020:19:360
7822991712,3cyclictest19482-21munin-run19:35:010
68272171,2sleep20-21swapper/220:19:142
34962170,3sleep20-21swapper/222:37:032
257882171,3sleep00-21swapper/019:46:330
242262170,4sleep024229-21fschecks_time23:25:160
181112171,3sleep00-21swapper/020:45:040
139912171,2sleep20-21swapper/220:35:152
783499165,3cyclictest986-1kworker/u9:123:05:132
783499164,5cyclictest1-21systemd23:30:292
783499164,3cyclictest86950irq/126-enp1s0-00:05:142
783499163,4cyclictest30470-21munin-run20:00:002
783499161,9cyclictest0-21swapper/221:02:292
783499161,6cyclictest3809-21ntp_states20:10:242
783499161,6cyclictest3809-21ntp_states20:10:242
7834991611,3cyclictest20221-21sed20:50:022
7834991611,3cyclictest11530-21/usr/sbin/munin20:30:232
7834991610,4cyclictest25801-21tune2fs22:15:152
7834991610,3cyclictest30609-21expr23:40:132
782299164,3cyclictest0-21swapper/019:26:050
782299161,8cyclictest9089-21ntp_states19:10:230
782299161,4cyclictest0-21swapper/019:55:230
7822991611,3cyclictest10449-21sensors_temp19:15:030
235552160,2sleep20-21swapper/219:41:052
7834991510,3cyclictest835-21runrttasks20:01:012
7834991510,3cyclictest5202-21head23:55:162
7834991510,3cyclictest26921-21/usr/sbin/munin21:05:142
782299151,12cyclictest22519-21/usr/sbin/munin20:55:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional