You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-21 - 16:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Aug 21, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291662241163,25sleep00-21swapper/007:08:150
293012216163,39sleep20-21swapper/207:09:582
289922215141,24sleep10-21swapper/107:05:591
289942202164,25sleep30-21swapper/307:06:013
29403991081,2cyclictest0-21swapper/209:24:302
29403996662,2cyclictest26408-21/usr/sbin/munin09:25:252
29403996662,2cyclictest26408-21/usr/sbin/munin09:25:252
9122570,2sleep20-21swapper/208:30:142
93742540,4sleep123-21ksoftirqd/110:00:301
29403995350,2cyclictest0-21swapper/211:30:582
242802530,5sleep32940799cyclictest10:35:193
135402510,1sleep10-21swapper/112:40:011
2940399422,3cyclictest26496-21memory10:40:192
2939699421,29cyclictest4229-21/usr/sbin/munin11:05:271
2940399415,29cyclictest31204-40dnf07:10:332
2939699411,23cyclictest11219-21dump-pmu-power12:35:011
2940399404,6cyclictest14137-40dnf10:12:242
2939699402,35cyclictest20979-21ntp_states08:00:251
2939699401,27cyclictest0-21swapper/107:15:261
2939699391,29cyclictest6687-21expr11:10:141
29403993822,5cyclictest18049-21diskstats07:55:142
2940399381,5cyclictest13088-21awk11:25:162
2939699381,27cyclictest24427-21/usr/sbin/munin10:35:211
2939699381,26cyclictest18351-21/usr/sbin/munin09:10:231
2939699381,23cyclictest1130-21sh09:45:021
2940399373,3cyclictest86850irq/129-enp1s0-10:35:262
2940399371,7cyclictest29629-21memory08:20:192
2939699373,26cyclictest8170-21grep12:25:251
2939699371,28cyclictest4543-21grep07:25:141
2939699371,25cyclictest814-21sshd08:35:321
2939699370,27cyclictest0-21swapper/112:20:141
21972370,7sleep07-21ksoftirqd/009:45:180
2939699361,3cyclictest8687-21munin-run07:35:011
2939699361,21cyclictest0-21swapper/110:30:161
2939699361,21cyclictest0-21swapper/110:30:151
2940399353,5cyclictest5779-21/usr/sbin/munin09:55:192
2940399353,3cyclictest2363-21needreboot09:45:202
2940399351,5cyclictest4629-21latency_hist12:20:022
2940399351,31cyclictest9213-21find10:00:272
29396993519,13cyclictest13210-21grep10:10:191
29396993517,15cyclictest6421-21expr09:55:171
2939699351,3cyclictest14984-21latency_hist11:30:021
2939699350,24cyclictest0-21swapper/108:20:011
2940399343,6cyclictest4016-21ntp_states08:35:222
2940399341,6cyclictest0-21swapper/212:30:192
2940399341,5cyclictest3996-21latency_hist11:05:012
2940399341,4cyclictest0-21swapper/210:50:272
2940399341,30cyclictest20896-21diskstats09:15:152
2940399341,18cyclictest31299-21cat09:39:562
2939699341,4cyclictest27558-21awk10:45:011
2939699341,3cyclictest581-21systemd-journal08:05:211
2939699341,3cyclictest24331-21uptime09:20:271
2939699341,3cyclictest2295-40dnf12:13:221
2939699341,2cyclictest26212-21/usr/sbin/munin11:55:151
2939699341,27cyclictest0-21swapper/107:36:121
2939699341,23cyclictest29589-21taskset07:10:091
2939699341,23cyclictest0-21swapper/111:02:001
2939699341,19cyclictest22280-21fschecks_count11:45:151
2939699340,27cyclictest0-21swapper/108:12:441
2940399338,21cyclictest24124-21grep11:50:142
2940399334,3cyclictest86750irq/128-enp1s0-11:45:122
2940399332,5cyclictest13654-21diskstats07:45:162
2940399331,6cyclictest5207-21sshd09:50:422
2940399331,6cyclictest0-21swapper/208:00:162
2940399331,5cyclictest0-21swapper/210:45:162
2939699337,14cyclictest31352-21if_enp1s012:05:141
2939699335,20cyclictest0-21swapper/111:17:471
2939699334,26cyclictest848-21sssd_nss10:50:571
2940399324,2cyclictest86850irq/129-enp1s0-11:58:092
2940399322,5cyclictest1371-21smart_sda07:15:252
2940399322,3cyclictest8193-40dnf11:12:312
2940399322,27cyclictest10144-21grep07:35:252
2940399321,5cyclictest0-21swapper/212:00:482
2940399321,4cyclictest19230-21cstates10:25:132
2940399321,3cyclictest28473-21latency_hist08:20:022
2940399321,2cyclictest0-21swapper/212:05:232
2940399321,28cyclictest20612-21grep11:40:202
29403993211,18cyclictest4905-21hddtemp_smartct11:05:182
2939699326,18cyclictest0-21swapper/107:21:321
2939699323,21cyclictest0-21swapper/109:31:001
2939699322,26cyclictest20685-21ntp_states11:40:221
2939699322,22cyclictest15220-21/usr/sbin/munin11:30:121
2939699321,6cyclictest14583-21/usr/sbin/munin10:15:261
2939699321,3cyclictest28600-21cut10:45:181
2939699321,3cyclictest22010-21cat09:15:261
2939699321,23cyclictest0-21swapper/109:35:441
2939699321,22cyclictest0-21swapper/111:51:371
2939699320,25cyclictest0-21swapper/109:46:581
2940399314,2cyclictest12050irq/123-ahci[0012:35:262
2940399313,3cyclictest86550irq/126-enp1s0-11:20:202
2940399312,5cyclictest6216-21ntp_states08:40:232
2940399312,4cyclictest22453-21smart_sda10:30:292
2940399312,4cyclictest22453-21smart_sda10:30:292
2940399311,5cyclictest12805-21ntp_states08:55:242
2940399311,4cyclictest86550irq/126-enp1s0-08:05:152
2940399311,3cyclictest8150-21irqstats08:45:192
2940399311,3cyclictest3392-21ntp_states07:20:222
2940399311,3cyclictest20038-40dnf09:11:312
2940399311,27cyclictest24861-21fschecks_time08:10:162
29403993110,18cyclictest17192-21ntpq09:05:212
2939699317,20cyclictest10957-21grep11:20:131
2939699313,3cyclictest2917-21cat08:35:021
2939699313,23cyclictest5492-21diskstats08:40:151
2939699311,4cyclictest18125-21awk09:10:011
29396993114,14cyclictest836-21runrttasks10:25:181
2939699311,3cyclictest5630-21cat09:55:021
2939699311,2cyclictest0-21swapper/112:00:291
2939699311,2cyclictest0-21swapper/107:40:141
2939699311,22cyclictest28708-21/usr/sbin/munin08:20:191
2939699311,21cyclictest784-21NetworkManager08:45:121
2939699311,18cyclictest17435-21cpu11:35:111
2939699311,18cyclictest17435-21cpu11:35:111
2939699310,5cyclictest0-21swapper/110:20:171
2939199312,13cyclictest4638-21munin-run12:20:010
2940399308,19cyclictest29122-21idleruntime-cro09:35:012
2940399305,3cyclictest86550irq/126-enp1s0-10:05:132
2940399305,3cyclictest4879-21/usr/sbin/munin07:25:162
2940399303,3cyclictest86850irq/129-enp1s0-12:25:142
2940399302,5cyclictest11459-21diskstats07:40:162
2940399302,4cyclictest731-21diskstats12:10:142
2940399302,4cyclictest7060-21df_inode07:30:152
2940399301,4cyclictest32135-21/usr/sbin/munin08:25:262
2940399301,4cyclictest0-21swapper/210:24:382
2940399301,3cyclictest8973-21diskstats11:15:152
2940399301,11cyclictest0-21swapper/209:03:042
29403993011,16cyclictest14845-21expr10:15:132
2939699308,18cyclictest12325-21grep08:55:181
2939699301,5cyclictest32181-21/usr/sbin/munin10:55:261
2939699301,3cyclictest14744-21timerwakeupswit07:45:271
2939699301,26cyclictest18278-21fschecks_time07:55:161
2939699301,23cyclictest0-21swapper/112:16:331
2940399297,19cyclictest31695-21cpuspeed_turbos09:40:122
2940399291,5cyclictest32181-21/usr/sbin/munin10:55:152
2940399291,5cyclictest17748-21df11:35:122
2940399291,5cyclictest17748-21df11:35:122
2940399291,4cyclictest9959-21uniq08:50:162
2940399291,4cyclictest17004-21/usr/sbin/munin07:50:272
2940399291,3cyclictest6031-21/usr/sbin/munin12:20:252
2939699291,3cyclictest26179-21fgrep09:25:231
2939699291,3cyclictest26179-21fgrep09:25:231
29396992912,12cyclictest0-21swapper/109:00:151
2939699291,17cyclictest0-21swapper/110:05:381
2939699282,4cyclictest10800-21smart_sda08:50:251
29396992812,13cyclictest15654-21cpuspeed_turbos07:50:131
2939199284,12cyclictest10600-21munin-run11:20:010
29391992810,15cyclictest11228-21munin-run12:35:010
2939699277,17cyclictest836-21runrttasks08:29:321
2939199271,13cyclictest1804-21basename11:00:010
29407992417,4cyclictest24716-21dump-pmu-power09:25:013
2940799240,8cyclictest0-21swapper/308:35:123
2940799239,3cyclictest86550irq/126-enp1s0-11:25:283
2940799238,6cyclictest86550irq/126-enp1s0-09:05:163
2940799231,6cyclictest7493-21ntp_states11:10:223
2940799231,6cyclictest401ktimersoftd/310:00:023
29407992316,4cyclictest19354-21cut10:25:153
29407992311,5cyclictest12050irq/123-ahci[0007:45:013
2940799231,12cyclictest18118-21diskmemload10:49:273
2939199239,3cyclictest38272unixbench_multi07:20:260
2939199232,11cyclictest4013-21awk11:05:010
2940799229,5cyclictest86850irq/129-enp1s0-10:15:263
2940799229,4cyclictest86550irq/126-enp1s0-12:05:233
2940799228,5cyclictest401ktimersoftd/311:00:013
2940799227,5cyclictest86550irq/126-enp1s0-10:10:253
2940799226,5cyclictest86550irq/126-enp1s0-11:05:143
2940799226,12cyclictest86850irq/129-enp1s0-11:42:043
2939199221,4cyclictest13424-21munin-run12:40:010
2940799218,2cyclictest86850irq/129-enp1s0-10:00:193
2940799218,2cyclictest86550irq/126-enp1s0-10:20:143
2940799218,2cyclictest401ktimersoftd/312:20:253
2940799216,3cyclictest401ktimersoftd/309:30:193
2940799215,5cyclictest86850irq/129-enp1s0-09:25:223
2940799215,5cyclictest86850irq/129-enp1s0-09:25:223
2940799215,4cyclictest0-21swapper/312:02:393
2940799214,7cyclictest401ktimersoftd/309:50:303
2940799211,12cyclictest748-21sshd09:40:553
2939199214,4cyclictest22364-21grep10:30:270
2939199214,4cyclictest22364-21grep10:30:270
2939199212,11cyclictest24357-21df_inode11:50:150
2940799209,3cyclictest30915-21/usr/sbin/munin08:25:213
2940799209,3cyclictest17414-21/usr/sbin/munin11:35:223
2940799209,3cyclictest17414-21/usr/sbin/munin11:35:223
2940799208,2cyclictest88050irq/133-enp2s0-11:15:143
2940799207,4cyclictest86550irq/126-enp1s0-12:35:143
2940799202,8cyclictest25871-21grep08:10:263
2940799202,7cyclictest41-21ksoftirqd/310:05:253
2940799201,6cyclictest39-21rcuc/309:35:163
2940799201,14cyclictest0-21swapper/309:15:213
2940799201,12cyclictest0-21swapper/310:30:043
2940799201,12cyclictest0-21swapper/310:30:043
2940799201,11cyclictest13933-21/usr/sbin/munin09:00:243
2939199209,6cyclictest0-21swapper/008:19:560
2939199209,3cyclictest23382-21/usr/sbin/munin10:35:260
2939199204,8cyclictest0-21swapper/008:02:210
2939199203,14cyclictest18306-21tail07:55:160
2939199202,3cyclictest28854-21ntp_states10:45:260
2939199202,10cyclictest7340-21/usr/sbin/munin08:45:190
2939199201,6cyclictest0-21swapper/011:58:000
29391992014,3cyclictest26920-21munin-run09:30:010
29391992014,3cyclictest26920-21munin-run09:30:010
2939199201,11cyclictest5146-21/usr/sbin/munin08:40:180
2939199201,11cyclictest22747-21/usr/sbin/munin09:20:150
2939199201,11cyclictest18351-21/usr/sbin/munin09:10:280
2939199201,11cyclictest13062-21sh10:10:180
2939199201,10cyclictest0-21swapper/010:26:010
26252201,3sleep00-21swapper/008:33:580
89932191,2sleep00-21swapper/012:29:460
2940799199,3cyclictest14634-21snmpd12:14:213
2940799199,2cyclictest12050irq/123-ahci[0010:45:013
2940799198,3cyclictest12050irq/123-ahci[0008:50:023
2940799194,8cyclictest86550irq/126-enp1s0-12:30:143
2940799194,8cyclictest39-21rcuc/307:20:203
2940799193,7cyclictest86850irq/129-enp1s0-12:15:153
2940799192,8cyclictest86850irq/129-enp1s0-12:25:143
29407991913,3cyclictest1364-21gltestperf08:30:173
2940799191,2cyclictest86550irq/126-enp1s0-11:55:213
2940799191,11cyclictest6720-21/usr/sbin/munin07:30:243
2940799191,11cyclictest39-21rcuc/308:20:153
2940799191,10cyclictest22112-21/usr/sbin/munin08:05:253
2940799191,10cyclictest2194-21irqstats09:45:183
2940799191,10cyclictest21816-21/usr/sbin/munin11:45:183
2940799190,12cyclictest0-21swapper/309:12:403
2939199195,4cyclictest14157-21pmu-power11:25:260
2939199195,3cyclictest11147-21expr11:20:140
2939199195,3cyclictest0-21swapper/011:30:110
2939199193,2cyclictest0-21swapper/010:16:170
2939199191,6cyclictest0-21swapper/010:50:350
2939199191,6cyclictest0-21swapper/010:00:210
2939199191,5cyclictest15923-21perf09:05:010
2939199191,4cyclictest0-21swapper/009:32:350
2939199191,15cyclictest10724-21grep10:05:170
2939199191,12cyclictest0-21swapper/012:21:100
2939199191,11cyclictest18539-21sshd10:22:200
2939199191,10cyclictest5779-21/usr/sbin/munin09:55:180
2939199191,10cyclictest18118-21diskmemload09:35:110
2939199191,10cyclictest14634-21snmpd08:27:570
2939199191,10cyclictest10568-21/usr/sbin/munin08:50:210
2939199191,10cyclictest0-21swapper/011:44:230
2940799188,3cyclictest8915-21/usr/sbin/munin07:35:163
2940799185,5cyclictest86550irq/126-enp1s0-11:00:163
2940799184,6cyclictest12050irq/123-ahci[0007:29:263
2940799182,2cyclictest0-21swapper/310:50:253
2940799181,12cyclictest0-21swapper/311:32:403
2939199189,2cyclictest0-21swapper/008:55:180
2939199185,4cyclictest0-21swapper/011:35:540
2939199185,4cyclictest0-21swapper/011:35:540
2939199184,7cyclictest0-21swapper/012:10:110
2939199184,6cyclictest0-21swapper/009:42:570
2939199181,9cyclictest0-21swapper/009:19:130
29391991814,3cyclictest0-21swapper/011:49:220
29391991812,3cyclictest836-21runrttasks07:30:200
29391991811,4cyclictest0-21swapper/011:05:080
2939199181,11cyclictest0-21swapper/010:40:130
2939199181,10cyclictest0-21swapper/012:05:350
2940799178,2cyclictest86550irq/126-enp1s0-11:20:143
2940799177,3cyclictest14634-21snmpd07:17:443
2940799174,5cyclictest24712-21/usr/sbin/munin11:50:183
2940799171,7cyclictest848-21sssd_nss08:15:213
2940799171,7cyclictest15505-21/usr/sbin/munin07:50:213
2940799171,6cyclictest0-21swapper/307:45:243
29407991712,3cyclictest9710-21expr08:50:133
29407991712,3cyclictest21173-21smartctl08:00:273
29407991711,3cyclictest5854-21basename08:40:183
2940799171,13cyclictest17702-21/usr/sbin/munin07:55:123
2940799171,11cyclictest0-21swapper/307:10:153
2939199177,7cyclictest6563-21cpuspeed_turbos11:10:130
2939199177,7cyclictest0-21swapper/007:35:170
2939199172,2cyclictest86550irq/126-enp1s0-12:00:180
2939199171,15cyclictest0-21swapper/007:48:550
29391991710,4cyclictest4347-21awk09:50:210
29391991710,4cyclictest31109-21unixbench_multi07:10:270
2939199170,15cyclictest0-21swapper/007:25:070
29407991612,3cyclictest0-21swapper/308:56:353
2939199169,4cyclictest24334-21cpu08:10:120
2939199166,3cyclictest0-21swapper/008:05:270
2939199162,11cyclictest2947-21/usr/sbin/munin08:35:170
29391991611,3cyclictest31701-21munin-node07:15:030
29391991611,3cyclictest16164-21cpu09:05:130
29391991611,3cyclictest11110-21/usr/sbin/munin07:40:160
29391991611,2cyclictest15946-21fschecks_count07:50:160
29391991610,3cyclictest29298-21grep08:20:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional