You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-04 - 21:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Aug 04, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
156152238163,24sleep20-21swapper/207:07:572
157282232162,21sleep30-21swapper/307:09:233
155732232163,57sleep00-21swapper/007:07:220
154642203166,25sleep10-21swapper/107:05:591
1937721640,2sleep31587799cyclictest07:15:173
15870998582,2cyclictest0-21swapper/210:41:092
231932740,1sleep10-21swapper/111:05:131
227012640,7sleep2311ktimersoftd/212:15:212
262012520,1sleep339-21rcuc/311:10:173
233752490,1sleep10-21swapper/107:25:131
15860994220,11cyclictest7-21ksoftirqd/011:15:020
15870994138,2cyclictest0-21swapper/209:53:332
1586599351,2cyclictest19511-21expr10:55:191
1586599351,10cyclictest0-21swapper/110:47:071
1586099346,13cyclictest15600-21munin-run09:35:010
1586099337,12cyclictest6827-21perf09:15:020
1586099318,11cyclictest14656-21basename12:00:010
1586599301,9cyclictest0-21swapper/107:43:471
1586599301,10cyclictest0-21swapper/110:31:151
1586599300,13cyclictest0-21swapper/112:35:231
1586099301,14cyclictest12460-21perf11:55:010
15860993011,16cyclictest8056-21munin-run11:45:000
276872290,4sleep230-21rcuc/207:35:012
1586599291,9cyclictest0-21swapper/110:09:491
1586599291,5cyclictest0-21swapper/109:50:541
1586599291,12cyclictest0-21swapper/111:48:181
1586599291,11cyclictest0-21swapper/109:35:301
1586599290,7cyclictest12804-21timerandwakeup09:25:271
1586099289,16cyclictest30368-21idleruntime-cro08:55:010
158092280,5sleep230-21rcuc/208:20:162
1586599271,9cyclictest0-21swapper/112:27:421
1586099279,15cyclictest25653-21munin-run12:25:010
1586099271,7cyclictest0-21swapper/010:49:010
129152270,5sleep011-21rcuc/009:25:290
1587799261,11cyclictest18346-21smart_sda12:05:243
1587799261,11cyclictest18346-21smart_sda12:05:243
1586599263,3cyclictest4617-21diskmemload11:10:241
1586599261,9cyclictest0-21swapper/112:08:371
1586599261,9cyclictest0-21swapper/112:08:371
1586599261,9cyclictest0-21swapper/110:40:051
1586599261,9cyclictest0-21swapper/109:55:101
1586599261,8cyclictest0-21swapper/107:46:251
1586599261,14cyclictest0-21swapper/111:50:371
1586599261,13cyclictest4617-21diskmemload12:04:571
1586599261,11cyclictest32159-21expr11:25:151
1586599261,11cyclictest0-21swapper/112:10:411
1586599261,11cyclictest0-21swapper/110:15:451
1586599261,11cyclictest0-21swapper/108:02:141
1586599261,10cyclictest0-21swapper/112:17:391
1586599261,10cyclictest0-21swapper/111:38:061
1586599261,10cyclictest0-21swapper/110:53:291
1586099269,15cyclictest29725-21awk07:40:020
1587799253,11cyclictest16457-21/usr/sbin/munin10:50:183
1587799251,17cyclictest0-21swapper/310:45:163
1587799251,12cyclictest0-21swapper/312:29:013
1587799251,10cyclictest21362-21pmu-power09:45:223
15870992510,9cyclictest909-21sssd_nss12:35:252
1586599251,9cyclictest0-21swapper/109:40:421
1586599251,12cyclictest8112-21ntp_states09:15:231
1586599251,12cyclictest0-21swapper/111:31:301
1586599251,10cyclictest0-21swapper/111:18:371
1586599251,10cyclictest0-21swapper/110:37:451
1586599251,10cyclictest0-21swapper/110:27:251
1586599251,10cyclictest0-21swapper/110:20:091
1586599250,9cyclictest30959-21kworker/u8:009:10:011
1587799242,18cyclictest0-21swapper/309:34:043
15870992411,4cyclictest32-21ksoftirqd/209:03:372
1586599241,20cyclictest108812sleep109:20:591
1586599241,11cyclictest3954-21idleruntime-cro07:55:011
1586599241,11cyclictest0-21swapper/111:55:341
1586599241,10cyclictest0-21swapper/110:00:501
1586599241,10cyclictest0-21swapper/109:49:341
1586599240,7cyclictest0-21swapper/108:32:231
1586599240,12cyclictest0-21swapper/108:35:131
1586099241,12cyclictest19067-21awk12:10:010
1586099241,12cyclictest19067-21awk12:10:010
1586099241,12cyclictest11219-21awk09:25:010
1587799233,17cyclictest31538-21gunzip12:35:243
15877992317,3cyclictest1803-21latency_hist07:50:013
1587799231,4cyclictest0-21swapper/310:30:073
1587799231,16cyclictest0-21swapper/308:06:073
1587799231,12cyclictest0-21swapper/309:40:153
1587799231,10cyclictest10672-21irqstats10:35:203
1587099233,7cyclictest15513-40dnf09:34:052
1587099231,6cyclictest0-21swapper/207:15:252
1586599238,3cyclictest0-21swapper/108:55:251
1586599231,9cyclictest0-21swapper/112:31:411
1586599231,9cyclictest0-21swapper/107:31:031
1586599231,8cyclictest0-21swapper/109:14:091
15865992317,3cyclictest1485-21ntp_states09:00:221
1586599231,6cyclictest0-21swapper/107:59:331
1586599231,5cyclictest0-21swapper/112:22:451
1586599231,16cyclictest0-21swapper/108:22:241
1586599231,16cyclictest0-21swapper/108:09:551
1586599231,16cyclictest0-21swapper/107:39:161
1586599231,11cyclictest0-21swapper/111:23:451
1586099231,4cyclictest4617-21diskmemload10:05:110
1586099231,4cyclictest0-21swapper/010:50:210
1587799229,2cyclictest401ktimersoftd/308:24:583
1587799227,3cyclictest0-21swapper/308:50:083
1587799222,12cyclictest28921-21unixbench-2d11:15:253
15877992215,4cyclictest13582-21sed10:40:283
1587799221,17cyclictest29157-21ntp_states12:30:223
1587799221,13cyclictest0-21swapper/309:03:093
1587799221,11cyclictest0-21swapper/310:23:293
1587799221,11cyclictest0-21swapper/309:25:533
1587799221,10cyclictest22735-21smart_sda12:15:223
1587799221,10cyclictest2073-21fschecks_count11:30:143
1587799221,10cyclictest10824-21users09:20:253
1587799221,10cyclictest0-21swapper/311:09:043
1587799221,10cyclictest0-21swapper/309:10:013
1587799220,9cyclictest0-21swapper/312:23:263
1587099221,4cyclictest0-21swapper/209:46:442
1586599221,8cyclictest0-21swapper/108:51:291
1586599221,6cyclictest0-21swapper/111:01:021
1586599221,6cyclictest0-21swapper/111:01:021
1586599221,16cyclictest0-21swapper/108:13:531
1586599221,12cyclictest18177-21munin-node08:15:281
1586099222,6cyclictest18245-1kworker/u9:209:43:460
1586099221,5cyclictest0-21swapper/011:23:490
1586099221,5cyclictest0-21swapper/010:16:260
1586099221,5cyclictest0-21swapper/009:39:180
1587799219,2cyclictest401ktimersoftd/308:38:283
1587799219,2cyclictest401ktimersoftd/307:41:543
1587799218,3cyclictest0-21swapper/310:12:093
1587799218,3cyclictest0-21swapper/308:31:063
1587799217,4cyclictest0-21swapper/311:58:433
1587799215,4cyclictest0-21swapper/312:14:473
1587799211,9cyclictest0-21swapper/312:01:143
1587799211,9cyclictest0-21swapper/311:53:063
1587799211,9cyclictest0-21swapper/311:35:043
1587799211,9cyclictest0-21swapper/310:27:283
1587799211,9cyclictest0-21swapper/310:15:433
1587799211,6cyclictest30085-21ntpq10:05:193
1587799211,6cyclictest0-21swapper/311:29:563
1587799211,10cyclictest25616-21cpuspeed_turbos07:30:133
1587799211,10cyclictest0-21swapper/311:43:563
1587799211,10cyclictest0-21swapper/309:13:173
1587799211,10cyclictest0-21swapper/308:59:583
1587799211,10cyclictest0-21swapper/308:42:283
1587799211,10cyclictest0-21swapper/308:42:283
1587099219,3cyclictest0-21swapper/208:05:202
1587099213,3cyclictest1438-21crond11:30:012
1587099212,7cyclictest29030-21/usr/sbin/munin10:05:222
1587099211,5cyclictest11242-21if_enp2s011:50:172
1586599219,3cyclictest26833-21hddtemp_smartct08:45:141
1586599210,15cyclictest0-21swapper/111:41:311
1586099213,9cyclictest2948450kworker/u8:211:26:130
1586099212,5cyclictest18245-1kworker/u9:211:30:360
1586099212,16cyclictest8320-21cut10:30:150
1586099212,11cyclictest3886-21/usr/sbin/munin11:35:250
1586099211,5cyclictest0-21swapper/011:49:370
1586099211,5cyclictest0-21swapper/010:36:340
1586099211,5cyclictest0-21swapper/010:03:560
1586099211,5cyclictest0-21swapper/009:47:590
1586099211,5cyclictest0-21swapper/009:18:560
1586099211,4cyclictest0-21swapper/012:37:080
1586099211,11cyclictest28095-21/usr/sbin/munin12:30:180
1586099211,11cyclictest1485-21ntp_states09:00:220
1586099210,4cyclictest16499-21kworker/u8:112:14:180
1587799208,3cyclictest0-21swapper/311:21:043
1587799208,1cyclictest0-21swapper/309:15:223
1587799206,6cyclictest0-21swapper/311:00:243
1587799206,6cyclictest0-21swapper/311:00:243
1587799206,12cyclictest22668-21grep07:20:263
1587799201,9cyclictest0-21swapper/308:17:263
1587799201,6cyclictest10778-21/usr/sbin/munin08:10:193
1587799201,5cyclictest17426-21users09:35:273
1587799201,10cyclictest0-21swapper/310:55:273
1587799201,10cyclictest0-21swapper/308:45:263
1587799200,17cyclictest22188-21kworker/u8:009:51:573
1587099203,3cyclictest87650irq/129-enp1s0-09:25:142
1587099201,7cyclictest989-21gzip07:45:252
1587099201,7cyclictest17373-21/usr/sbin/munin08:25:222
1587099201,6cyclictest22897-21pmu-power08:35:232
1587099201,5cyclictest0-21swapper/209:20:512
1586599207,3cyclictest0-21swapper/107:18:431
1586599206,2cyclictest0-21swapper/108:44:171
1586599206,2cyclictest0-21swapper/108:44:171
1586599201,14cyclictest0-21swapper/108:25:541
1586599201,11cyclictest31821-21gltestperf10:10:151
1586599201,10cyclictest18074-21munin-node07:15:011
1586599200,4cyclictest0-21swapper/109:31:221
1586099209,3cyclictest25874-21/usr/sbin/munin12:25:130
1586099202,8cyclictest19270-21kworker/0:009:58:090
1586099202,5cyclictest18245-1kworker/u9:210:23:390
1586099201,8cyclictest0-21swapper/012:16:260
1586099201,6cyclictest0-21swapper/010:11:110
1586099201,5cyclictest0-21swapper/011:06:450
1586099201,4cyclictest0-21swapper/011:19:490
1586099201,4cyclictest0-21swapper/009:50:330
1586099201,11cyclictest8581-21/usr/sbin/munin08:05:210
1586099201,11cyclictest13777-21sshd10:42:010
291922191,2sleep20-21swapper/208:50:192
1587799194,3cyclictest87650irq/129-enp1s0-09:55:223
1587799193,14cyclictest18896-21kworker/3:007:25:273
1587799191,6cyclictest827-21runrttasks08:25:153
1587799191,4cyclictest10078-21sshd11:48:003
1587799191,15cyclictest28922-21sed10:05:013
1587099194,3cyclictest87650irq/129-enp1s0-11:15:172
1587099194,2cyclictest12050irq/123-ahci[0009:10:022
1587099193,9cyclictest755-21rngd10:02:422
1587099191,6cyclictest7427-21grep09:15:162
1587099191,6cyclictest31618-21latency_hist11:25:022
1587099191,6cyclictest22440-21users11:00:262
1587099191,6cyclictest22440-21users11:00:262
1587099191,6cyclictest0-21swapper/210:53:262
1587099191,6cyclictest0-21swapper/209:35:532
1587099191,5cyclictest6434-21runrttasks11:40:162
1587099191,4cyclictest20778-21latency_hist11:00:012
1587099191,4cyclictest0-21swapper/210:32:282
1587099191,2cyclictest0-21swapper/212:30:142
1587099191,2cyclictest0-21swapper/211:10:012
1587099191,15cyclictest26217-21df_inode12:25:152
1587099190,5cyclictest0-21swapper/207:13:192
1586599191,11cyclictest21160-21/usr/sbin/munin07:20:131
1586099191,6cyclictest0-21swapper/011:04:380
1586099191,6cyclictest0-21swapper/011:04:380
1586099191,6cyclictest0-21swapper/010:26:260
1586099191,4cyclictest810-21sshd08:03:020
1586099191,15cyclictest81ktimersoftd/008:35:160
1586099191,10cyclictest15170-21/usr/sbin/munin08:20:130
197032181,2sleep20-21swapper/209:41:162
1587799182,15cyclictest0-21swapper/307:54:083
1587799181,7cyclictest0-21swapper/308:02:123
1587799181,7cyclictest0-21swapper/307:55:223
15877991814,2cyclictest17326-21smart_sda07:10:243
1587099181,7cyclictest0-21swapper/207:39:022
1587099181,6cyclictest7931-21sed08:00:262
1587099181,6cyclictest4617-21diskmemload11:10:552
1587099181,6cyclictest30603-21/usr/sbin/munin08:55:132
1587099181,6cyclictest0-21swapper/211:36:542
1587099181,5cyclictest1983-21/usr/sbin/munin07:50:152
1587099181,5cyclictest0-21swapper/211:56:002
1587099181,5cyclictest0-21swapper/211:45:502
1587099181,2cyclictest87550irq/128-enp1s0-10:37:432
15870991812,3cyclictest23697-21df_inode07:25:152
1587099181,15cyclictest0-21swapper/212:10:182
1587099181,15cyclictest0-21swapper/209:55:082
15870991811,4cyclictest20387-21irqstats08:30:192
1586099187,3cyclictest808-21snmpd07:17:060
1586099185,4cyclictest12039-21smart_sda08:10:260
1586099181,9cyclictest808-21snmpd08:27:200
1586099181,9cyclictest0-21swapper/007:40:230
1586099181,8cyclictest0-21swapper/010:56:300
1586099181,6cyclictest0-21swapper/012:00:290
15860991813,3cyclictest21055-21grep08:30:260
1586099181,15cyclictest0-21swapper/008:17:170
1586099181,10cyclictest30603-21/usr/sbin/munin08:55:220
166932170,3sleep20-21swapper/212:03:062
15877991713,3cyclictest0-21swapper/307:35:293
1587099174,7cyclictest0-21swapper/209:14:202
1587099171,6cyclictest0-21swapper/210:11:072
1587099171,5cyclictest18138-21ntpq12:05:222
1587099171,5cyclictest18138-21ntpq12:05:222
1587099171,5cyclictest0-21swapper/212:21:002
1587099171,5cyclictest0-21swapper/210:25:442
1587099171,4cyclictest14933-21dump-pmu-power08:20:002
1587099171,3cyclictest0-21swapper/210:46:122
15870991711,3cyclictest3653-21sh11:35:012
1587099171,11cyclictest0-21swapper/210:15:332
1586099171,9cyclictest808-21snmpd08:42:020
1586099171,9cyclictest808-21snmpd08:42:020
1586099171,9cyclictest0-21swapper/009:05:280
1586099171,7cyclictest1983-21/usr/sbin/munin07:50:240
15860991711,3cyclictest24126-21if_err_enp1s007:25:190
1586099171,10cyclictest0-21swapper/007:55:240
1587099164,2cyclictest87550irq/128-enp1s0-08:10:152
1587099163,2cyclictest87650irq/129-enp1s0-10:20:152
1587099162,3cyclictest12050irq/123-ahci[0008:40:062
1587099162,3cyclictest12050irq/123-ahci[0008:40:062
1587099161,5cyclictest30759-21irqstats07:40:192
1587099161,5cyclictest0-21swapper/207:55:212
1587099160,6cyclictest0-21swapper/208:45:112
1586099162,5cyclictest17651-21users07:10:260
1586099161,6cyclictest26179-21chrt08:45:100
1586099161,3cyclictest32145-21/usr/sbin/munin07:45:160
15860991610,3cyclictest26624-21ntpq07:30:230
1587099158,4cyclictest22233-21ntpq07:20:222
15860991510,3cyclictest21160-21/usr/sbin/munin07:20:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional