You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 19:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Nov 22, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
296842272195,65sleep00-21swapper/007:09:520
296842272195,65sleep00-21swapper/007:09:520
294742236160,24sleep10-21swapper/107:07:111
294742236160,24sleep10-21swapper/107:07:111
293502211179,21sleep20-21swapper/207:05:352
293502211179,21sleep20-21swapper/207:05:342
296002203132,58sleep30-21swapper/307:08:493
296002203132,58sleep30-21swapper/307:08:493
2286121560,3sleep02978499cyclictest10:30:250
29797997168,2cyclictest0-21swapper/208:08:252
29797997168,2cyclictest0-21swapper/208:08:252
286562510,2sleep30-21swapper/309:30:233
29784994013,13cyclictest31068-40dnf08:25:010
2978499371,17cyclictest12136-21sed10:05:270
2978499331,7cyclictest0-21swapper/010:48:250
2978499321,14cyclictest0-21swapper/008:37:310
2978499310,7cyclictest0-21swapper/012:26:530
2978499310,7cyclictest0-21swapper/012:26:530
2978499301,13cyclictest0-21swapper/011:45:170
2978499301,13cyclictest0-21swapper/008:45:130
2978499301,12cyclictest0-21swapper/009:50:370
2978499301,12cyclictest0-21swapper/009:32:410
29784993011,16cyclictest13279-21munin-run11:25:000
29784993011,16cyclictest13279-21munin-run11:25:000
2978499300,6cyclictest0-21swapper/011:11:170
2980199291,21cyclictest86550irq/126-enp1s0-09:40:273
2978499291,6cyclictest0-21swapper/010:26:510
2978499291,6cyclictest0-21swapper/010:04:410
2978499291,12cyclictest0-21swapper/011:42:130
2978499290,13cyclictest0-21swapper/011:32:130
2978499281,12cyclictest0-21swapper/009:42:390
2978499281,11cyclictest0-21swapper/009:10:250
2978499280,15cyclictest0-21swapper/011:05:250
2978499280,13cyclictest0-21swapper/008:30:170
2978499280,11cyclictest0-21swapper/012:30:030
2978499280,11cyclictest0-21swapper/010:42:090
64312270,3sleep30-21swapper/307:25:273
2980199271,24cyclictest0-21swapper/311:30:203
2978499273,11cyclictest19211-21idleruntime-cro10:25:000
2978499271,7cyclictest0-21swapper/009:48:130
2978499271,11cyclictest0-21swapper/012:37:510
2978499270,12cyclictest0-21swapper/011:01:130
2978499270,12cyclictest0-21swapper/008:09:500
2978499270,12cyclictest0-21swapper/008:09:490
157842270,2sleep2311ktimersoftd/210:15:192
2980199260,24cyclictest0-21swapper/310:10:223
2979099262,20cyclictest16767-21ntp_states11:30:221
29790992620,3cyclictest25284-21fschecks_time08:10:161
29790992615,3cyclictest3075-21cat12:15:021
2978499263,18cyclictest27705-21kworker/0:010:51:580
2978499261,12cyclictest25115-21munin-run09:25:010
2978499261,11cyclictest0-21swapper/011:37:350
2978499261,11cyclictest0-21swapper/010:13:100
2978499261,11cyclictest0-21swapper/009:16:490
2978499260,6cyclictest0-21swapper/007:53:460
2980199253,6cyclictest21049-40dnf10:25:443
2980199251,12cyclictest23823-21/usr/sbin/munin10:35:253
2980199251,11cyclictest2723-21/usr/sbin/munin07:20:163
29790992511,11cyclictest31110-21/usr/sbin/munin12:05:171
2978499257,15cyclictest26474-21munin-run11:55:010
2978499251,12cyclictest16327-21awk09:05:000
2978499251,10cyclictest0-21swapper/012:21:330
2978499250,6cyclictest0-21swapper/007:55:490
2978499250,13cyclictest0-21swapper/010:37:230
2978499250,12cyclictest0-21swapper/011:18:030
2978499250,12cyclictest0-21swapper/009:39:090
2980199245,7cyclictest0-21swapper/311:15:213
2979099241,17cyclictest0-21swapper/107:40:241
2979099241,15cyclictest7295-21ntp_states09:55:241
2979099241,14cyclictest0-21swapper/111:28:001
2979099241,10cyclictest20077-21perf08:00:001
2978499241,9cyclictest0-21swapper/011:59:270
2978499241,14cyclictest0-21swapper/007:11:110
2978499240,9cyclictest0-21swapper/007:16:370
2980199232,10cyclictest9940-21/usr/sbin/munin08:50:163
29801992315,5cyclictest4882-21grep08:35:433
2980199231,20cyclictest86550irq/126-enp1s0-11:00:243
2980199231,20cyclictest0-21swapper/307:15:283
2980199231,11cyclictest14634-21snmpd10:33:143
2979799239,3cyclictest215412sendmail_mailst08:00:252
2979799233,4cyclictest0-21swapper/211:10:032
2979099239,3cyclictest4630-21gltestperf09:50:191
2979099239,3cyclictest26465-21sh11:55:021
2979099237,9cyclictest0-21swapper/108:40:311
2979099234,3cyclictest0-21swapper/107:19:551
2979099232,4cyclictest19203-21munin-run10:25:011
29790992317,3cyclictest8774-21entropy10:00:161
29790992317,3cyclictest22447-21expr11:45:181
29790992316,4cyclictest14059-21diskstats07:45:181
29790992314,2cyclictest10995-21kworker/u8:211:35:091
2979099231,20cyclictest0-21swapper/109:45:291
2979099231,16cyclictest0-21swapper/112:00:001
2979099231,16cyclictest0-21swapper/108:54:091
2979099231,14cyclictest0-21swapper/112:15:031
2979099231,14cyclictest0-21swapper/109:20:281
2979099231,10cyclictest31959-21ls07:15:011
2978499231,7cyclictest0-21swapper/012:08:030
2978499231,14cyclictest0-21swapper/007:34:520
2978499230,11cyclictest0-21swapper/008:43:050
2980199228,3cyclictest26355-21sshd11:53:433
2980199228,3cyclictest17225-21/usr/sbin/munin10:20:213
2980199221,8cyclictest0-21swapper/312:19:593
2980199221,6cyclictest31068-40dnf08:25:023
2980199221,4cyclictest0-21swapper/309:45:413
2980199221,19cyclictest0-21swapper/311:45:373
2980199221,19cyclictest0-21swapper/311:25:063
2980199221,18cyclictest5477-21irqstats11:05:193
2980199221,11cyclictest0-21swapper/309:50:223
2980199221,10cyclictest9235-21NetworkManager12:26:363
2980199221,10cyclictest9235-21NetworkManager12:26:363
2980199221,10cyclictest16888-21needreboot07:50:243
2980199220,10cyclictest0-21swapper/311:40:073
2979799222,5cyclictest0-21swapper/212:15:132
2979099228,9cyclictest0-21swapper/109:38:011
2979099228,4cyclictest768-21in:imjournal10:40:021
2979099228,3cyclictest28403-21cstates10:45:151
2979099227,9cyclictest0-21swapper/107:24:461
2979099222,12cyclictest0-21swapper/111:05:261
2979099222,11cyclictest5252-21df07:25:131
2979099222,11cyclictest23765-21sensors_temp08:05:251
2979099222,11cyclictest23765-21sensors_temp08:05:251
29790992218,3cyclictest0-21swapper/109:32:021
29790992218,3cyclictest0-21swapper/108:20:251
29790992218,3cyclictest0-21swapper/107:39:411
2979099221,5cyclictest0-21swapper/112:38:141
2979099221,5cyclictest0-21swapper/109:17:551
2979099221,3cyclictest8505-21timerandwakeup07:30:251
2979099221,19cyclictest0-21swapper/108:56:431
2979099221,18cyclictest20525-21chrt09:12:441
2979099221,16cyclictest23-21ksoftirqd/110:50:231
2979099221,16cyclictest0-21swapper/109:28:521
2979099221,16cyclictest0-21swapper/109:02:471
2979099221,15cyclictest0-21swapper/111:10:191
2979099221,15cyclictest0-21swapper/110:05:241
2979099221,14cyclictest0-21swapper/110:15:341
29790992210,3cyclictest5692-21diskstats12:20:151
2978499221,6cyclictest0-21swapper/010:55:570
2978499221,14cyclictest0-21swapper/007:42:080
2978499221,13cyclictest0-21swapper/012:10:250
2978499221,13cyclictest0-21swapper/008:00:050
2978499221,13cyclictest0-21swapper/007:26:320
2978499220,9cyclictest0-21swapper/008:59:510
2978499220,14cyclictest0-21swapper/012:15:070
2980199212,7cyclictest15027-21/usr/sbin/munin10:15:263
2980199212,5cyclictest7234-21df_inode11:10:143
29801992116,3cyclictest25629-21cat08:10:213
2980199211,4cyclictest0-21swapper/309:10:263
2980199211,12cyclictest0-21swapper/308:00:543
2980199211,10cyclictest14634-21snmpd08:57:473
2980199211,10cyclictest0-21swapper/307:35:143
2979799211,5cyclictest0-21swapper/210:25:142
29797992115,3cyclictest12166-21sed07:40:182
2979099218,3cyclictest86750irq/128-enp1s0-11:00:161
2979099213,6cyclictest31310-21/usr/sbin/munin08:25:191
2979099212,6cyclictest32623-21/usr/sbin/munin10:55:131
29790992119,1cyclictest0-21swapper/111:15:221
29790992117,3cyclictest0-21swapper/108:01:381
2979099211,19cyclictest0-21swapper/110:42:431
2979099211,15cyclictest0-21swapper/110:12:111
2979099211,12cyclictest18518-21diskmemload12:30:451
2979099211,12cyclictest0-21swapper/109:43:231
2979099211,11cyclictest8605-21ntp_states12:25:221
2979099211,11cyclictest8605-21ntp_states12:25:221
2979099211,11cyclictest30162-21smart_sda12:00:251
2978499217,2cyclictest26153-21irqstats09:25:180
2978499211,12cyclictest0-21swapper/009:55:190
2978499211,12cyclictest0-21swapper/008:26:230
2978499211,11cyclictest0-21swapper/011:28:590
2978499210,7cyclictest0-21swapper/007:35:560
2980199205,6cyclictest0-21swapper/309:21:063
2980199205,3cyclictest28166-21smartctl08:15:253
2980199204,4cyclictest10936-21expr10:05:143
2980199204,3cyclictest86550irq/126-enp1s0-09:15:253
2980199202,15cyclictest32045-21awk08:25:193
2980199201,8cyclictest0-21swapper/308:30:193
2980199201,6cyclictest18518-21diskmemload12:05:223
2980199201,6cyclictest15147-21irqstats09:00:203
2980199201,16cyclictest842-21lldpd12:30:383
2980199201,16cyclictest6225-21/usr/sbin/munin09:55:203
2980199201,16cyclictest20049-21sshd07:59:553
2980199201,10cyclictest0-21swapper/312:10:133
2980199201,10cyclictest0-21swapper/307:40:213
29797992017,2cyclictest0-21swapper/211:41:112
2979799201,5cyclictest0-21swapper/212:02:522
29797992014,3cyclictest8353-21latency_hist10:00:012
2979799201,12cyclictest0-21swapper/211:50:332
2979099201,7cyclictest22439-21irqstats10:30:211
29790992017,2cyclictest0-21swapper/108:36:371
29790992014,3cyclictest28490-21users08:15:261
2979099201,17cyclictest0-21swapper/110:25:171
2979099201,11cyclictest17882-21awk07:55:001
2979099201,10cyclictest9793-21cat08:50:001
2979099201,10cyclictest3237-21cat08:35:011
2979099201,10cyclictest0-21swapper/111:40:431
2979099200,15cyclictest0-21swapper/109:08:151
2979099200,15cyclictest0-21swapper/109:08:141
2978499202,10cyclictest24706-21/usr/sbin/munin08:10:180
2978499201,12cyclictest0-21swapper/007:48:160
2978499201,11cyclictest0-21swapper/007:21:160
2980199197,3cyclictest28731-1kworker/u9:210:53:393
2980199195,4cyclictest842-21lldpd07:10:033
2980199195,3cyclictest11305-21/usr/sbin/munin11:20:163
2980199195,3cyclictest11305-21/usr/sbin/munin11:20:163
2980199194,4cyclictest30506-21sshd12:00:293
2980199193,7cyclictest0-21swapper/311:38:583
2980199192,4cyclictest18518-21diskmemload09:25:183
2980199191,6cyclictest29744-21/usr/sbin/munin09:35:293
2980199191,5cyclictest26371-21diskstats10:40:153
2980199191,3cyclictest29284-21ntp_states10:45:243
2980199191,3cyclictest23318-21irqstats08:05:203
2980199191,3cyclictest23318-21irqstats08:05:203
2980199191,10cyclictest0-21swapper/311:59:123
2979799196,4cyclictest22407-21grep11:45:182
2979799193,9cyclictest28731-1kworker/u9:209:13:292
2979799191,8cyclictest0-21swapper/207:26:372
29797991914,3cyclictest27343-21gltestperf11:55:172
29797991913,3cyclictest26023-21/usr/sbin/munin10:40:212
2979799190,5cyclictest0-21swapper/210:45:432
2979099191,3cyclictest12543-21find11:20:251
2979099191,3cyclictest12543-21find11:20:251
2978499198,3cyclictest26903-21/usr/sbin/munin08:15:190
2978499191,12cyclictest0-21swapper/009:05:410
2978499191,12cyclictest0-21swapper/009:05:410
2978499190,10cyclictest0-21swapper/012:00:090
2980199184,4cyclictest0-21swapper/310:00:143
2980199181,6cyclictest0-21swapper/312:35:163
2980199181,10cyclictest0-21swapper/312:21:383
2979799189,2cyclictest1586-21awk09:45:022
2979799188,3cyclictest3132-21awk08:35:022
2979799184,3cyclictest28731-1kworker/u9:210:07:482
2979799183,4cyclictest0-21swapper/211:30:442
2979799182,6cyclictest32099-21munin-node07:45:332
2979799181,8cyclictest9237-21irqstats10:00:202
2979799181,7cyclictest4781-21timerwakeupswit08:35:432
2979799181,7cyclictest0-21swapper/212:22:402
2979799181,6cyclictest0-21swapper/212:08:192
2979799181,6cyclictest0-21swapper/211:35:592
2979799181,6cyclictest0-21swapper/209:27:382
2979799181,5cyclictest31682-21sensors10:50:252
2979799181,5cyclictest0-21swapper/211:24:202
2979799181,5cyclictest0-21swapper/211:24:202
29797991813,3cyclictest6658-21idleruntime-cro11:10:012
29797991813,3cyclictest19160-21ntp_offset07:55:232
29797991812,3cyclictest2725-21acpi07:20:152
29797991812,3cyclictest12170-21cat12:35:152
29797991811,4cyclictest14737-21fgrep11:25:262
2978499181,15cyclictest11913-21munin-run08:55:000
2980199173,12cyclictest32623-21/usr/sbin/munin10:55:253
2980199171,14cyclictest401ktimersoftd/308:45:213
29801991711,3cyclictest7254-21grep07:30:133
2980199171,10cyclictest0-21swapper/308:40:353
2980199170,9cyclictest0-21swapper/309:05:283
2980199170,9cyclictest0-21swapper/309:05:283
2979799178,7cyclictest5680-21cpuspeed_turbos08:40:142
2979799174,4cyclictest8969-21sshd07:33:422
2979799172,2cyclictest86850irq/129-enp1s0-10:56:352
2979799171,7cyclictest9961-21grep11:15:212
2979799171,7cyclictest29106-21/usr/sbin/munin08:20:242
2979799171,7cyclictest18518-21diskmemload10:38:332
2979799171,7cyclictest0-21swapper/212:13:502
2979799171,6cyclictest0-21swapper/210:10:082
2979799171,6cyclictest0-21swapper/209:33:032
2979799171,6cyclictest0-21swapper/209:23:022
2979799171,3cyclictest21985-21tr09:15:232
29797991712,3cyclictest3425-21sshd09:45:422
29797991712,3cyclictest31467-21grep08:25:142
29797991712,3cyclictest31178-21timerwakeupswit09:35:302
29797991712,3cyclictest27048-21tr08:15:132
29797991712,3cyclictest18653-21date09:10:022
29797991712,3cyclictest18653-21date09:10:022
29797991712,3cyclictest10338-21nfs4_client07:35:202
2979799171,15cyclictest0-21swapper/210:23:572
29797991711,3cyclictest21629-21/usr/sbin/munin10:30:172
29797991711,3cyclictest10053-21cat12:30:152
29797991710,4cyclictest3720-21smart_sda11:00:252
2978499171,15cyclictest0-21swapper/010:15:250
29801991610,3cyclictest14064-21dnf07:45:203
29797991612,3cyclictest0-21swapper/208:53:532
29797991612,3cyclictest0-21swapper/208:49:352
29797991612,3cyclictest0-21swapper/207:18:202
29797991612,2cyclictest7880-21df12:25:152
29797991612,2cyclictest7880-21df12:25:152
29797991611,3cyclictest30295-21cat07:10:152
29797991611,3cyclictest24841-21expr08:10:142
29797991611,3cyclictest16582-21hddtemp_smartct07:50:212
29797991611,3cyclictest14866-21grep09:00:182
29797991611,3cyclictest13229-21open_inodes08:55:242
29797991610,3cyclictest4170-21grep09:50:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional