You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-18 - 23:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 18, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
213402222194,19sleep10-21swapper/107:07:281
212102204167,26sleep00-21swapper/007:05:460
213372203167,24sleep30-21swapper/307:07:273
212122202166,24sleep20-21swapper/207:05:492
2763821440,2sleep32164299cyclictest11:00:223
1296821400,5sleep02162599cyclictest10:27:560
21635998481,2cyclictest0-21swapper/212:08:582
111182470,1sleep20-21swapper/210:25:022
2164299347,12cyclictest30965-21/usr/sbin/munin11:10:233
21642993310,13cyclictest5898-21ls10:10:253
21642993310,13cyclictest5898-21ls10:10:253
317062320,2sleep230-21rcuc/208:45:002
2164299326,12cyclictest827-21runrttasks09:51:203
2164299324,20cyclictest0-21swapper/310:55:203
21642993210,8cyclictest2160-21munin-run10:05:003
2162999311,24cyclictest0-21swapper/112:35:211
2162599311,24cyclictest0-21swapper/009:33:390
2164299309,6cyclictest19553-21runrttasks10:42:503
2164299309,10cyclictest790-21in:imjournal07:23:203
2164299308,7cyclictest3157-21cpuspeed_turbos11:20:123
2164299301,10cyclictest20170-21kworker/u8:012:15:383
2163599301,7cyclictest0-21swapper/209:56:452
2162999304,13cyclictest6205-21systemd-journal10:10:411
2162999304,13cyclictest6205-21systemd-journal10:10:411
2164299294,15cyclictest21751-21expr09:35:103
2164299293,16cyclictest0-21swapper/311:30:013
2164299293,14cyclictest23744-21/usr/sbin/munin09:40:223
2164299290,17cyclictest0-21swapper/307:20:013
2162599294,12cyclictest20353-21munin-run12:00:010
2164299288,6cyclictest10327-21diskmemload12:05:173
2164299286,12cyclictest8144-21if_enp2s011:30:173
2164299281,6cyclictest0-21swapper/310:37:563
21642992810,7cyclictest19971-21/usr/sbin/munin11:55:263
21642992810,6cyclictest10327-21diskmemload11:45:263
21642992810,3cyclictest28770-21/usr/sbin/munin11:05:143
21629992818,2cyclictest0-21swapper/111:40:111
2162599281,12cyclictest7149-21cut09:00:220
2162599281,12cyclictest7149-21cut09:00:220
2162599281,12cyclictest11580-21idleruntime-cro11:40:010
2164299276,9cyclictest3634-21cpu12:35:143
2164299276,8cyclictest10566-21gltestperf07:55:163
2164299276,10cyclictest808-21snmpd07:14:593
2164299274,9cyclictest14052-21runrttasks08:04:273
21642992716,7cyclictest10327-21diskmemload09:30:233
2163599276,9cyclictest25040-1kworker/2:1H09:05:202
2162999273,8cyclictest20532-21processes09:30:241
2162599271,10cyclictest15134-21pmu-power11:45:230
2164299268,15cyclictest14949-21/usr/sbin/munin09:20:233
2164299267,7cyclictest9874-21sed07:55:023
2164299267,7cyclictest9874-21sed07:55:023
2164299267,7cyclictest8865-21cat10:20:013
2164299267,5cyclictest10327-21diskmemload10:07:383
2164299267,5cyclictest10327-21diskmemload09:16:013
2164299265,8cyclictest28069-21sed09:50:013
2164299265,18cyclictest23124-21/usr/sbin/munin08:25:173
2164299265,11cyclictest29962-21cstates08:40:123
2164299264,13cyclictest10327-21diskmemload11:19:583
2164299263,9cyclictest18555-21timerwakeupswit09:25:243
2164299263,11cyclictest19447-21kworker/u8:210:54:263
2164299261,6cyclictest0-21swapper/312:14:103
2164299261,6cyclictest0-21swapper/310:24:253
2164299260,15cyclictest0-21swapper/310:25:193
2163599266,8cyclictest10327-21diskmemload12:20:142
2163599264,8cyclictest3105-40dnf12:31:332
2163599263,9cyclictest15060-21expr09:20:122
21629992616,4cyclictest0-21swapper/108:00:131
21629992610,4cyclictest26499-21cut09:45:161
2162599262,12cyclictest7170-21munin-run11:30:010
2162599262,11cyclictest6141-21grep10:10:270
2162599262,11cyclictest6141-21grep10:10:270
2162599261,16cyclictest0-21swapper/010:00:530
2164299259,6cyclictest12783-21grep11:40:203
2164299259,3cyclictest3343-21/usr/sbin/munin07:40:163
2164299259,13cyclictest10732-21pmu-power11:35:223
2164299258,6cyclictest27294-21latency_hist08:35:013
2164299257,6cyclictest0-21swapper/312:29:213
2164299251,9cyclictest0-21swapper/307:25:223
21642992514,7cyclictest827-21runrttasks12:31:343
2164299250,10cyclictest0-21swapper/308:15:163
2163599256,9cyclictest87350irq/126-enp1s0-11:05:132
2163599256,7cyclictest311ktimersoftd/210:10:212
2163599256,7cyclictest311ktimersoftd/210:10:212
2163599255,8cyclictest3618-21/usr/sbin/munin12:35:142
2163599251,8cyclictest0-21swapper/210:06:072
21635992512,5cyclictest311ktimersoftd/211:55:132
2163599251,17cyclictest0-21swapper/212:27:522
21635992511,6cyclictest12177-21netstat10:25:222
21629992511,7cyclictest18245-1kworker/u9:210:09:381
2162599259,3cyclictest20295-21df10:45:140
2162599257,4cyclictest0-21swapper/011:18:310
2162599251,16cyclictest0-21swapper/012:00:190
2162599251,11cyclictest0-21swapper/010:18:410
2162599251,10cyclictest0-21swapper/009:37:530
2164299247,6cyclictest87350irq/126-enp1s0-11:50:203
2164299245,16cyclictest3957-21/usr/sbin/munin08:55:253
2164299242,19cyclictest8357-21/usr/sbin/munin09:05:203
21642992414,6cyclictest14632-21smart_sda10:30:243
2163599244,9cyclictest18941-21chrt09:27:542
2163599244,8cyclictest0-21swapper/210:50:162
2163599241,8cyclictest0-21swapper/210:43:592
2163599241,8cyclictest0-21swapper/210:35:512
2163599241,4cyclictest0-21swapper/211:28:462
2163599241,14cyclictest0-21swapper/212:17:092
2162999245,4cyclictest0-21swapper/111:12:301
21629992418,3cyclictest2100-21df_inode08:50:141
21629992416,3cyclictest0-21swapper/108:45:151
2162999240,19cyclictest0-21swapper/109:35:101
2162599243,4cyclictest0-21swapper/011:06:320
2162599241,9cyclictest0-21swapper/012:28:350
2162599241,4cyclictest0-21swapper/012:18:520
2162599241,22cyclictest0-21swapper/008:20:150
2162599241,21cyclictest0-21swapper/011:13:550
2162599241,15cyclictest0-21swapper/011:54:310
21625992411,4cyclictest0-21swapper/009:46:280
2162599241,12cyclictest0-21swapper/011:40:180
2162599241,11cyclictest29562-21cstates12:20:140
2162599241,10cyclictest8337-21tail07:50:150
2162599241,10cyclictest8337-21tail07:50:150
2162599241,10cyclictest0-21swapper/010:53:100
2162599241,10cyclictest0-21swapper/010:37:310
2162599241,10cyclictest0-21swapper/009:27:080
2162599240,20cyclictest25513-21pmu-power10:55:240
2164299239,2cyclictest7517-21awk07:50:013
2164299236,4cyclictest30340-21/usr/sbin/munin09:55:183
2164299234,8cyclictest755-21rngd12:22:203
2164299234,8cyclictest755-21rngd12:03:523
2164299233,12cyclictest0-21swapper/307:35:003
21642992315,5cyclictest27862-21df_inode08:35:113
21642992315,5cyclictest10910-21diskstats09:10:163
21642992313,7cyclictest6830-21hddtemp_smartct09:00:183
21642992313,7cyclictest6830-21hddtemp_smartct09:00:183
2164299230,9cyclictest17394-21latency08:10:213
2163599237,7cyclictest87650irq/129-enp1s0-10:30:212
2163599234,8cyclictest25353-21cut10:55:202
2163599234,7cyclictest24986-21/usr/sbin/munin12:10:172
2163599234,3cyclictest870-21turbostat.cron07:35:012
2163599231,7cyclictest0-21swapper/211:52:422
2163599231,6cyclictest0-21swapper/209:34:222
2163599231,5cyclictest0-21swapper/209:13:382
2163599231,12cyclictest12784-21needreboot11:40:212
2163599230,5cyclictest2436-21kworker/u8:111:22:272
2163599230,13cyclictest1135-21diskstats11:15:152
2162999231,7cyclictest0-21swapper/111:45:181
2162999231,10cyclictest16012-21fschecks_count10:35:161
2162999230,5cyclictest0-21swapper/108:25:201
2162999230,11cyclictest0-21swapper/110:03:261
2162599239,3cyclictest32305-21grep07:30:200
2162599238,3cyclictest26137-21pmu-power12:10:240
2162599231,17cyclictest0-21swapper/010:07:150
2162599231,11cyclictest3618-21/usr/sbin/munin12:35:160
2162599231,11cyclictest29718-21/usr/sbin/munin08:40:240
2162599231,11cyclictest22788-21/usr/sbin/munin12:05:210
2162599231,11cyclictest0-21swapper/009:42:080
2162599231,10cyclictest0-21swapper/011:31:320
2164299229,3cyclictest31916-21/usr/sbin/munin08:45:183
2164299224,8cyclictest87650irq/129-enp1s0-08:05:213
21642992213,6cyclictest2838-21ntpq08:50:223
21642992213,6cyclictest21752-21/usr/sbin/munin08:20:193
2163599225,9cyclictest29927-21taskset09:52:472
2163599225,4cyclictest0-21swapper/211:10:042
2163599222,3cyclictest32-21ksoftirqd/212:00:062
2163599221,9cyclictest0-21swapper/211:37:162
2163599221,6cyclictest9715-21awk07:55:012
2163599221,6cyclictest9715-21awk07:55:012
2163599221,6cyclictest0-21swapper/209:48:512
2163599221,13cyclictest0-21swapper/210:00:162
21635992210,5cyclictest12050irq/123-ahci[0009:20:012
2162999228,4cyclictest19456-21open_files11:55:221
2162999224,11cyclictest0-21swapper/112:31:101
2162999223,7cyclictest18245-1kworker/u9:211:19:561
2162999221,9cyclictest20710-21awk08:20:011
2162999221,7cyclictest30371-21grep12:20:201
2162999221,6cyclictest25707-21needreboot07:15:171
21629992216,1cyclictest0-21swapper/111:35:021
21629992215,4cyclictest6500-21diskstats09:00:151
21629992215,4cyclictest6500-21diskstats09:00:151
21629992215,2cyclictest0-21swapper/111:05:011
21629992215,2cyclictest0-21swapper/109:55:191
21629992212,3cyclictest28773-21users12:15:271
2162999221,18cyclictest8406-21chrt10:15:451
2162999221,11cyclictest20170-21kworker/u8:012:08:301
2162999221,10cyclictest14729-21perf09:20:021
2162599228,3cyclictest16535-21/usr/sbin/munin08:10:220
2162599225,6cyclictest0-21swapper/011:22:570
2162599221,9cyclictest23240-21kworker/u8:010:30:170
2162599221,9cyclictest0-21swapper/009:17:200
2162599221,8cyclictest0-21swapper/008:50:250
21642992113,5cyclictest1460-21df07:35:143
21642992111,7cyclictest19962-21/usr/sbin/munin10:45:163
2163599217,3cyclictest10327-21diskmemload11:45:122
2163599215,8cyclictest5240-21chrt07:44:172
2163599215,3cyclictest19962-21/usr/sbin/munin10:45:172
2163599214,10cyclictest22975-21timerwakeupswit09:35:242
2163599212,3cyclictest29590-21cat08:40:012
2163599211,3cyclictest25040-1kworker/2:1H08:30:012
2163599211,3cyclictest0-21swapper/211:31:212
2163599211,3cyclictest0-21swapper/210:15:182
2163599211,12cyclictest0-21swapper/209:00:552
2163599211,12cyclictest0-21swapper/209:00:552
2163599211,11cyclictest0-21swapper/211:00:122
2162999219,2cyclictest17144-21/usr/sbin/munin09:25:121
2162999216,4cyclictest0-21swapper/112:10:401
2162999215,4cyclictest4072-21ntpq11:20:221
2162999212,7cyclictest0-21swapper/112:25:431
2162999211,8cyclictest10422-21sed11:35:191
2162999211,8cyclictest0-21swapper/110:30:041
2162999211,3cyclictest0-21swapper/108:31:171
21629992112,3cyclictest11824-21smart_sda09:10:271
2162999211,18cyclictest0-21swapper/109:40:071
2162999211,14cyclictest0-21swapper/111:05:261
21629992111,3cyclictest0-21swapper/110:50:191
21629992111,3cyclictest0-21swapper/108:40:001
2162999210,8cyclictest0-21swapper/107:40:001
2162999210,19cyclictest0-21swapper/107:50:121
2162999210,19cyclictest0-21swapper/107:50:121
2162599218,3cyclictest28670-21head08:35:210
2162599215,4cyclictest6299-21if_enp2s007:45:170
2162599215,3cyclictest87550irq/128-enp1s0-07:20:180
2162599213,3cyclictest0-21swapper/010:21:500
2162599211,9cyclictest808-21snmpd08:32:440
2162599211,9cyclictest1751-21df12:30:130
2162599211,4cyclictest0-21swapper/009:22:200
2162599211,3cyclictest87350irq/126-enp1s0-10:41:510
2162599211,17cyclictest29732-21users09:50:250
215622210,4sleep123-21ksoftirqd/108:20:151
2163599203,3cyclictest25227-21grep09:40:272
2163599202,6cyclictest5920-21latency_hist09:00:002
2163599201,5cyclictest29109-21/usr/sbin/munin07:25:252
2162999206,4cyclictest10176-21cstates07:55:121
2162999206,3cyclictest0-21swapper/109:53:001
2162999202,6cyclictest0-21swapper/112:04:251
2162999201,7cyclictest26913-21/usr/sbin/munin07:20:141
21629992016,3cyclictest0-21swapper/107:40:191
21629992016,2cyclictest14727-21tune2fs08:05:151
2162999201,17cyclictest0-21swapper/110:20:251
2162999201,17cyclictest0-21swapper/108:44:131
2162999201,16cyclictest10326-21sh09:10:021
2162999201,14cyclictest0-21swapper/110:59:411
21629992011,3cyclictest23211-21sed07:10:261
2162999201,11cyclictest16196-21acpi11:50:121
2162999201,10cyclictest1028-21cat07:35:011
21629992010,3cyclictest16615-21grep08:10:141
2162999200,8cyclictest0-21swapper/110:27:551
2162999200,4cyclictest0-21swapper/111:27:191
2162599205,3cyclictest10327-21diskmemload09:11:500
2162599204,9cyclictest0-21swapper/008:46:120
2162599203,3cyclictest12516-21cut08:00:150
2162599201,10cyclictest0-21swapper/009:56:270
2163599191,9cyclictest21259-21df08:20:132
2163599191,4cyclictest0-21swapper/207:35:032
2163599191,10cyclictest0-21swapper/208:50:292
2162999192,8cyclictest0-21swapper/110:47:191
21629991916,2cyclictest0-21swapper/107:45:161
21629991912,3cyclictest0-21swapper/109:23:001
2162999190,17cyclictest0-21swapper/108:55:221
2162599198,1cyclictest0-21swapper/009:05:260
2162599192,5cyclictest23469-21df_inode08:25:140
2162599191,4cyclictest24715-21/usr/sbin/munin07:15:120
2162599191,3cyclictest0-21swapper/011:03:130
2162599191,15cyclictest1311-21ls07:35:120
307122180,2sleep10-21swapper/107:25:261
2163599185,2cyclictest10879-21missed_timers07:55:192
2163599181,6cyclictest23943-21munin-node07:45:292
2163599181,14cyclictest634-21ps08:45:212
2163599181,11cyclictest0-21swapper/208:06:342
2163599181,10cyclictest0-21swapper/207:20:052
2163599180,4cyclictest0-21swapper/208:30:142
2162999185,7cyclictest0-21swapper/110:40:171
2162599185,3cyclictest87550irq/128-enp1s0-07:40:180
2162599182,4cyclictest20133-21timerwakeupswit08:15:250
2162599181,4cyclictest14336-21/usr/sbin/munin08:05:140
21625991813,3cyclictest9933-21/usr/sbin/munin07:55:110
170052180,4sleep230-21rcuc/208:10:172
2163599172,2cyclictest87350irq/126-enp1s0-07:15:022
2163599171,5cyclictest0-21swapper/207:10:172
2163599171,4cyclictest0-21swapper/208:15:182
2163599171,14cyclictest0-21swapper/208:00:062
2162599172,3cyclictest87550irq/128-enp1s0-08:55:250
2162599172,12cyclictest29749-21cut07:25:140
21625991612,2cyclictest21825-21/usr/sbin/munin07:10:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional