You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 17:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Nov 29, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
169172236163,23sleep30-21swapper/307:07:233
170222230156,22sleep10-21swapper/107:08:461
170932204166,26sleep00-21swapper/007:09:370
168162202165,25sleep20-21swapper/207:06:052
1942721750,5sleep31723699cyclictest08:25:183
264572640,4sleep226459-21fschecks_count09:55:182
184602620,8sleep011-21rcuc/012:05:010
71312600,2sleep011-21rcuc/009:10:190
321692540,1sleep20-21swapper/211:20:242
75552510,3sleep3401ktimersoftd/308:00:013
36572480,2sleep339-21rcuc/311:30:173
181202420,2sleep00-21swapper/009:35:200
17229993834,3cyclictest0-21swapper/212:16:192
1721899338,12cyclictest16038-21idleruntime-cro12:00:000
1721899338,12cyclictest16038-21idleruntime-cro12:00:000
1721899328,11cyclictest5996-21awk09:10:010
1722999283,11cyclictest25077-40dnf08:40:022
1721899287,12cyclictest25099-21munin-run08:40:000
1721899287,12cyclictest25099-21munin-run08:40:000
113672280,3sleep29-21rcu_preempt09:20:172
1721899279,15cyclictest13861-21idleruntime-cro11:55:000
1722499261,20cyclictest0-21swapper/111:27:301
17224992611,13cyclictest28584-21kworker/u8:211:41:331
1721899261,12cyclictest8812-21munin-run10:30:000
1721899260,18cyclictest0-21swapper/011:48:290
1722499251,18cyclictest0-21swapper/112:28:411
1722499251,17cyclictest836-21runrttasks07:49:041
1721899251,5cyclictest0-21swapper/011:24:480
263572241,3sleep126391-21ntp_states08:40:231
17236992417,4cyclictest28159-21cut08:45:163
1723699241,11cyclictest13435-21/usr/sbin/munin10:40:143
17229992411,4cyclictest0-21swapper/209:35:152
1722499242,19cyclictest27712-21irqstats07:30:181
1722499242,19cyclictest27712-21irqstats07:30:181
17224992414,3cyclictest0-21swapper/109:00:291
1722499241,17cyclictest0-21swapper/108:07:091
1722499241,15cyclictest0-21swapper/109:40:041
1722499240,17cyclictest0-21swapper/108:26:231
1721899241,12cyclictest13221-21awk10:40:000
8512231,2sleep3891-21latency_hist07:45:003
1723699237,7cyclictest0-21swapper/309:30:273
1723699233,11cyclictest16273-21/usr/sbin/munin12:00:293
17236992315,5cyclictest25288-21latency_hist08:40:013
1722999233,5cyclictest9083-40dnf11:40:332
1722499231,16cyclictest0-21swapper/109:59:491
1722499231,15cyclictest0-21swapper/107:42:271
1721899237,3cyclictest6158-21tail10:20:270
1721899232,18cyclictest3077-21/usr/sbin/munin11:30:190
1721899231,11cyclictest24486-21awk07:25:010
1721899231,11cyclictest14634-21snmpd07:48:300
119762231,3sleep112048-21irqstats10:35:201
267082220,5sleep011-21rcuc/011:10:130
225272225,2sleep222689-21cstates07:20:152
17236992215,4cyclictest31646-21head11:20:173
1723699221,2cyclictest0-21swapper/311:40:333
1722999221,4cyclictest27722-21kworker/u8:010:11:232
1722499229,10cyclictest19365-21cat07:14:561
1722499221,18cyclictest18487-21/usr/sbin/munin12:05:221
1722499221,18cyclictest13563-21timerwakeupswit08:10:281
1722499221,18cyclictest10213-21ps10:30:241
1722499221,16cyclictest0-21swapper/107:25:361
1722499221,15cyclictest0-21swapper/110:10:301
1722499221,15cyclictest0-21swapper/109:50:341
1722499221,15cyclictest0-21swapper/108:55:321
1722499221,15cyclictest0-21swapper/108:32:181
1722499220,16cyclictest0-21swapper/110:04:111
1721899221,7cyclictest4323-21needreboot07:50:200
256632210,1sleep00-21swapper/011:05:250
231202211,2sleep10-21swapper/108:35:131
231202211,2sleep10-21swapper/108:35:131
193022211,3sleep019428-21users10:50:270
193022211,3sleep019428-21users10:50:270
182792210,3sleep30-21swapper/309:35:243
177882210,3sleep117792-21gltestperf09:35:181
1723699218,4cyclictest22883-21/usr/sbin/munin12:15:233
1723699212,17cyclictest15839-21sed08:15:263
17236992115,3cyclictest10723-21kworker/u8:007:19:223
1723699211,11cyclictest28621-21fschecks_count10:00:163
1723699211,11cyclictest14634-21snmpd09:02:593
1722999216,6cyclictest179842grep10:50:142
1722999216,6cyclictest179842grep10:50:142
17229992114,4cyclictest3412-21df11:30:142
17229992112,2cyclictest86850irq/129-enp1s0-09:10:212
1722499219,3cyclictest30808-21latency_hist11:20:021
1722499218,4cyclictest0-21swapper/109:25:121
17224992114,2cyclictest0-21swapper/107:36:411
17224992113,3cyclictest0-21swapper/109:06:431
1722499211,18cyclictest0-21swapper/110:46:111
1722499211,18cyclictest0-21swapper/110:15:201
1722499211,15cyclictest0-21swapper/110:20:471
1722499211,15cyclictest0-21swapper/109:48:021
1722499211,15cyclictest0-21swapper/108:49:191
1722499211,15cyclictest0-21swapper/108:22:431
1722499211,15cyclictest0-21swapper/108:16:021
1722499211,11cyclictest17350-21ntp_states12:00:241
17224992110,3cyclictest13841-21latency_hist11:55:021
17224992110,3cyclictest10942-21smart_sda11:45:281
1721899216,8cyclictest0-21swapper/008:59:290
1721899214,6cyclictest14199-21mii-tool10:40:180
1721899213,8cyclictest9934-21/usr/sbin/munin08:05:250
1721899210,6cyclictest0-21swapper/011:26:130
1721899210,5cyclictest0-21swapper/010:12:520
1723699208,3cyclictest6840-21/usr/sbin/munin10:25:233
1723699208,3cyclictest11875-21/usr/sbin/munin11:50:173
1722999205,9cyclictest25077-40dnf08:40:012
1722999202,8cyclictest14998-21/usr/sbin/munin09:30:172
1722999201,6cyclictest14634-21snmpd11:18:232
1722999201,6cyclictest0-21swapper/211:27:092
17229992013,4cyclictest836-21runrttasks07:15:132
1722999201,13cyclictest0-21swapper/210:01:112
1722499208,3cyclictest25101-21egrep12:20:131
1722499202,12cyclictest0-21swapper/111:30:361
1722499202,10cyclictest24942-21cut12:20:011
1722499201,9cyclictest0-21swapper/109:30:581
17224992014,3cyclictest14547-21ls10:40:251
17224992013,2cyclictest0-21swapper/107:19:441
1722499201,2cyclictest86750irq/128-enp1s0-10:05:371
1722499201,2cyclictest0-21swapper/108:50:461
1722499201,18cyclictest0-21swapper/111:23:031
1722499201,17cyclictest0-21swapper/111:56:211
1722499201,17cyclictest0-21swapper/111:56:211
1722499201,17cyclictest0-21swapper/108:00:591
1722499201,14cyclictest0-21swapper/109:13:341
1722499201,14cyclictest0-21swapper/107:55:081
1722499201,14cyclictest0-21swapper/107:20:031
1722499201,10cyclictest10948-21diskstats09:20:141
1722499200,15cyclictest0-21swapper/112:11:091
1721899208,3cyclictest26898-21/usr/sbin/munin07:30:120
1721899208,3cyclictest26898-21/usr/sbin/munin07:30:120
1721899207,7cyclictest86550irq/126-enp1s0-12:17:220
1721899206,4cyclictest0-21swapper/009:45:400
1721899206,3cyclictest86750irq/128-enp1s0-10:05:200
1721899206,3cyclictest6131-21gltestperf07:55:170
1721899205,5cyclictest759-21dbus-daemon07:39:310
1721899205,4cyclictest27280-21/usr/sbin/munin12:25:160
1721899201,6cyclictest8989-21grep11:40:280
1721899201,6cyclictest14325-21/usr/sbin/munin08:15:200
1721899201,5cyclictest13048-21sshd08:10:200
1721899201,12cyclictest0-21swapper/009:34:580
1721899201,11cyclictest8404-21/usr/sbin/munin09:15:170
1721899201,11cyclictest17414-21/usr/sbin/munin07:10:250
1721899201,11cyclictest14634-21snmpd11:37:200
1721899201,10cyclictest30354-21tail11:15:270
1721899201,10cyclictest27557-21unixbench_singl09:55:280
1721899200,11cyclictest0-21swapper/012:23:020
1723699199,3cyclictest836-21runrttasks08:55:003
1723699198,3cyclictest23920-21sshd11:01:223
1723699192,9cyclictest16523-21/usr/sbin/munin08:20:273
1723699191,9cyclictest27296-21cpu12:25:153
1723699191,9cyclictest12131-21/usr/sbin/munin08:10:203
1723699191,9cyclictest0-21swapper/311:14:213
1723699191,7cyclictest3360-21memory10:15:213
1723699191,4cyclictest784-21NetworkManager08:09:553
1723699191,3cyclictest21444-21timerwakeupswit10:55:273
17236991913,3cyclictest17761-21sh07:10:143
17236991913,3cyclictest11375-21cpuspeed_turbos10:35:143
1723699191,15cyclictest26226-21cstates09:55:153
1723699191,10cyclictest14634-21snmpd11:38:273
1723699191,10cyclictest0-21swapper/309:16:473
1722999197,3cyclictest0-21swapper/211:48:142
1722999195,8cyclictest25077-40dnf08:40:022
1722999192,6cyclictest30811-21fschecks_count10:05:172
1722999192,6cyclictest17202-1kworker/u9:210:49:492
1722999191,7cyclictest0-21swapper/211:50:152
1722999191,5cyclictest0-21swapper/212:14:402
17229991913,3cyclictest19474-21needreboot12:05:232
1722999191,12cyclictest0-21swapper/211:06:092
1722499194,4cyclictest0-21swapper/109:15:521
17224991916,2cyclictest0-21swapper/111:06:301
1722499191,2cyclictest0-21swapper/110:50:501
1722499191,2cyclictest0-21swapper/110:50:501
1722499191,10cyclictest27623-21/usr/sbin/munin11:10:221
1722499191,10cyclictest1-21systemd10:25:211
1722499190,17cyclictest0-21swapper/112:37:091
1721899196,4cyclictest0-21swapper/010:16:290
1721899193,8cyclictest86750irq/128-enp1s0-12:10:180
1721899193,3cyclictest28120-21gltestperf08:45:160
1721899192,6cyclictest842-21lldpd09:22:400
1721899192,5cyclictest17202-1kworker/u9:208:52:030
1721899191,4cyclictest29820-21df_inode12:30:150
17218991913,3cyclictest0-21swapper/010:55:160
1721899191,10cyclictest9038-21/usr/sbin/munin10:30:200
1721899191,10cyclictest748-21unixbench_multi12:35:270
1721899190,6cyclictest0-21swapper/010:04:440
56922181,2sleep10-21swapper/111:35:171
272702181,2sleep00-21swapper/008:44:550
1723699184,4cyclictest25260-21cat08:40:013
1723699181,8cyclictest7095-21kworker/3:009:20:153
1723699181,8cyclictest0-21swapper/311:55:383
1723699181,8cyclictest0-21swapper/311:55:383
1723699181,7cyclictest9324-21expr10:30:143
1723699181,7cyclictest32231-21cut12:35:173
1723699181,7cyclictest29652-21irqstats11:15:203
1723699181,7cyclictest24668-21cat07:25:013
1723699181,6cyclictest25043-21grep11:05:173
17236991813,3cyclictest4987-21sh10:20:163
17236991812,4cyclictest25659-21df08:40:143
17236991812,3cyclictest481-21df10:10:153
17236991812,3cyclictest30404-21/usr/sbin/munin10:05:143
17236991812,3cyclictest21118-21fschecks_count12:10:173
17236991812,3cyclictest14634-21snmpd07:45:053
1723699181,15cyclictest0-21swapper/307:52:473
1723699181,14cyclictest12798-21/usr/sbin/munin09:25:133
1723699181,13cyclictest18487-21/usr/sbin/munin12:05:153
1723699181,10cyclictest0-21swapper/312:21:483
1722999186,3cyclictest19719-21cat09:40:132
1722999186,11cyclictest311ktimersoftd/208:50:262
1722999183,6cyclictest16523-21/usr/sbin/munin08:20:192
1722999183,6cyclictest0-21swapper/210:31:032
1722999183,12cyclictest8857-21grep08:00:242
1722999182,14cyclictest0-21swapper/212:01:172
1722999181,8cyclictest545-21ntp_states08:55:222
1722999181,7cyclictest0-21swapper/212:36:482
1722999181,7cyclictest0-21swapper/211:39:262
1722999181,7cyclictest0-21swapper/209:51:292
1722999181,7cyclictest0-21swapper/209:27:082
1722999181,6cyclictest17202-1kworker/u9:210:15:242
1722999181,6cyclictest0-21swapper/212:32:222
17229991813,3cyclictest10072-21sshd09:16:192
17229991812,3cyclictest14948-21unixbench_multi10:40:282
1722999181,11cyclictest0-21swapper/211:03:452
1722999181,11cyclictest0-21swapper/210:26:302
1722999181,10cyclictest5977-21diskmemload10:56:142
1722499181,9cyclictest20708-21smartctl10:55:181
17224991812,3cyclictest29815-21sh12:30:141
17224991812,3cyclictest22234-21/usr/sbin/munin11:00:181
1721899183,4cyclictest17210-21cyclictest08:20:040
1721899182,3cyclictest86550irq/126-enp1s0-09:29:350
1721899181,9cyclictest18493-21acpi12:05:140
1721899181,5cyclictest22248-21sshd07:19:520
1721899181,10cyclictest0-21swapper/011:00:140
1721899180,11cyclictest0-21swapper/010:48:500
96542171,2sleep30-21swapper/308:04:353
64852170,3sleep20-21swapper/210:23:332
1723699178,2cyclictest0-21swapper/311:25:273
1723699177,2cyclictest29100-21/usr/sbin/munin07:35:303
1723699175,6cyclictest5977-21diskmemload09:48:013
1723699175,4cyclictest25067-21gmain09:40:203
1723699172,12cyclictest29480-21/usr/sbin/munin12:30:203
1723699171,7cyclictest0-21swapper/311:48:413
17236991712,3cyclictest31938-21/usr/sbin/munin08:55:183
17236991712,3cyclictest24699-21/usr/sbin/munin07:25:153
17236991712,3cyclictest24644-21kernelversion09:50:203
1723699171,15cyclictest0-21swapper/310:50:343
1723699171,15cyclictest0-21swapper/310:50:343
1723699171,14cyclictest24762-21kworker/u8:209:10:063
1723699171,14cyclictest0-21swapper/309:09:063
17236991710,4cyclictest21904-21needreboot08:30:193
17236991710,4cyclictest16001-21/usr/sbin/munin10:45:163
1722999172,5cyclictest17202-1kworker/u9:210:35:202
1722999171,7cyclictest0-21swapper/211:13:382
1722999171,7cyclictest0-21swapper/208:25:262
1722999171,6cyclictest30-21rcuc/208:30:162
1722999171,6cyclictest27250-21entropy07:30:152
1722999171,6cyclictest27250-21entropy07:30:152
17229991713,3cyclictest0-21swapper/208:48:522
17229991713,3cyclictest0-21swapper/207:27:592
17229991712,3cyclictest25861-21if_err_enp1s012:20:202
17229991712,3cyclictest22615-21netstat09:45:212
17229991712,3cyclictest14510-21ls08:15:142
1722999171,15cyclictest0-21swapper/212:25:232
1722999171,15cyclictest0-21swapper/207:47:302
17229991710,4cyclictest17808-21grep07:10:152
1722499172,12cyclictest4005-21hddtemp_smartct07:50:181
1721899173,3cyclictest86750irq/128-enp1s0-07:25:220
1721899171,4cyclictest20330-21sshd09:40:180
1721899171,14cyclictest0-21swapper/008:25:010
1721899171,12cyclictest5977-21diskmemload09:50:150
17218991710,4cyclictest21603-21sh08:30:150
151762171,2sleep215294-21rt-features11:55:262
151762171,2sleep215294-21rt-features11:55:262
17236991611,3cyclictest27173-21cat07:30:123
17236991611,3cyclictest27173-21cat07:30:123
1722999161,7cyclictest3747-21fschecks_count07:50:172
1722999161,6cyclictest4007-21/usr/sbin/munin09:05:232
1722999161,6cyclictest0-21swapper/209:00:172
1722999161,6cyclictest0-21swapper/207:35:222
1722999161,14cyclictest0-21swapper/207:44:062
17229991611,3cyclictest12131-21/usr/sbin/munin08:10:262
17229991611,3cyclictest10762-21sed08:05:202
17218991611,3cyclictest7708-21date08:00:010
1721899161,12cyclictest1797-21/usr/sbin/munin09:00:140
1721899160,14cyclictest0-21swapper/007:40:140
17229991511,2cyclictest5565-21cpu07:55:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional