You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-14 - 01:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Aug 13, 2026 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
320302234163,58sleep20-21swapper/207:05:452
320532207170,25sleep00-21swapper/007:06:030
320862203167,24sleep30-21swapper/307:06:313
320222202166,23sleep10-21swapper/107:05:391
2594521860,4sleep03244899cyclictest11:45:180
2965321340,4sleep03244899cyclictest11:55:130
32459998784,2cyclictest0-21swapper/207:10:052
32459998683,2cyclictest0-21swapper/210:13:442
40622580,4sleep13245399cyclictest10:55:231
22592550,4sleep230-21rcuc/210:50:262
32459994845,2cyclictest0-21swapper/211:24:132
32459994845,2cyclictest0-21swapper/211:24:132
104602480,1sleep00-21swapper/011:10:190
32459994441,2cyclictest0-21swapper/210:49:032
32459993612,9cyclictest24283-21users11:40:282
3245999355,18cyclictest5018-21ls11:00:012
3245999336,10cyclictest17430-21wc11:25:242
32459993312,10cyclictest14314-40dnf10:05:232
32459993312,10cyclictest14314-40dnf10:05:232
3245999323,15cyclictest1324-21latency_hist12:05:022
32459993214,8cyclictest16016-21/usr/sbin/munin12:35:272
3245999319,11cyclictest3181-21/usr/sbin/munin07:15:162
3245999315,10cyclictest24029-21grep10:30:012
32459993118,7cyclictest827-21runrttasks11:05:452
3245399311,22cyclictest0-21swapper/108:05:051
3245399310,19cyclictest0-21swapper/111:38:541
3245399310,12cyclictest0-21swapper/109:57:571
3245999307,12cyclictest13271-21memory12:30:182
3245999307,12cyclictest13271-21memory12:30:182
3245999303,16cyclictest8266-21expr12:20:132
3245399300,23cyclictest0-21swapper/111:59:331
3244899302,13cyclictest19612-21munin-run10:20:020
32448993011,16cyclictest23391-21munin-run09:15:010
3245999296,8cyclictest19620-21awk10:20:012
3245999296,7cyclictest22592-21grep09:10:222
32459992920,5cyclictest24866-40dnf08:04:242
32459992910,4cyclictest31424-21gunzip09:30:232
3245399291,8cyclictest0-21swapper/110:45:171
3245399291,13cyclictest0-21swapper/110:13:591
3245399291,11cyclictest0-21swapper/112:19:411
3245399290,22cyclictest0-21swapper/111:27:461
3245399290,22cyclictest0-21swapper/109:18:461
3245399290,17cyclictest0-21swapper/111:05:421
3245399290,12cyclictest0-21swapper/110:54:591
3245399290,11cyclictest0-21swapper/112:02:451
3245999288,7cyclictest6412-21latency_hist09:50:012
3245399281,12cyclictest0-21swapper/109:07:341
3245399281,11cyclictest0-21swapper/108:17:031
3245399281,10cyclictest0-21swapper/110:01:291
3245399280,9cyclictest0-21swapper/109:02:381
3245399280,12cyclictest0-21swapper/107:43:571
3245399280,12cyclictest0-21swapper/107:43:571
3245399280,10cyclictest0-21swapper/112:37:031
3244899284,12cyclictest9244-21munin-run11:10:010
3244899283,12cyclictest6420-21munin-run09:50:000
3246599274,7cyclictest8890-40dnf11:05:443
3245999277,7cyclictest29078-21ntp_states09:25:222
3245999277,11cyclictest827-21runrttasks11:10:232
3245999276,5cyclictest16353-21/usr/sbin/munin07:45:172
3245999275,10cyclictest0-21swapper/211:15:172
3245999274,13cyclictest0-21swapper/212:28:222
32459992718,6cyclictest3014-40dnf12:05:512
32459992718,5cyclictest9246-21ntp_states08:40:202
32459992718,5cyclictest3511-21smart_sda09:40:262
32459992712,6cyclictest3120-21unixbench_singl08:25:242
3245399271,9cyclictest0-21swapper/111:34:181
3245399271,9cyclictest0-21swapper/110:36:291
3245399271,7cyclictest0-21swapper/107:59:051
3245399271,7cyclictest0-21swapper/107:45:371
3245399271,11cyclictest0-21swapper/109:47:431
3245399271,11cyclictest0-21swapper/108:56:141
3245399271,10cyclictest0-21swapper/112:30:571
3245399271,10cyclictest0-21swapper/112:30:571
3245399271,10cyclictest0-21swapper/110:21:551
3245399271,10cyclictest0-21swapper/108:40:101
3245399271,10cyclictest0-21swapper/107:19:451
3245399270,11cyclictest0-21swapper/112:28:331
3245399270,11cyclictest0-21swapper/111:46:021
3245399270,11cyclictest0-21swapper/111:17:261
3245399270,10cyclictest0-21swapper/109:35:331
3244899274,11cyclictest20541-21perf07:55:010
3245999267,5cyclictest5771-21awk08:35:002
3245999266,10cyclictest87350irq/126-enp1s0-09:35:122
3245999265,13cyclictest27066-21sendmail_mailtr09:20:242
3245999264,8cyclictest28217-21sshd10:37:492
3245999262,10cyclictest18552-21/usr/sbin/munin07:50:232
32459992612,11cyclictest27433-21chrt11:50:152
32459992610,13cyclictest7744-21ls12:20:012
3245399261,9cyclictest0-21swapper/109:21:351
3245399261,8cyclictest0-21swapper/108:02:331
3245399261,8cyclictest0-21swapper/107:31:581
3245399261,11cyclictest0-21swapper/111:13:521
3245399261,10cyclictest0-21swapper/108:54:001
3245399261,10cyclictest0-21swapper/108:54:001
3245399261,10cyclictest0-21swapper/108:36:541
3245399261,10cyclictest0-21swapper/108:30:381
3245399260,15cyclictest0-21swapper/107:38:211
3245399260,13cyclictest0-21swapper/110:25:311
3245399260,12cyclictest0-21swapper/112:11:151
3245399260,12cyclictest0-21swapper/111:53:111
3245399260,12cyclictest0-21swapper/111:01:531
3245399260,10cyclictest0-21swapper/108:28:551
3244899267,16cyclictest25591-21munin-run09:20:010
3244899262,10cyclictest30311-21kworker/0:010:47:420
3244899261,13cyclictest7347-21munin-run07:25:010
3246599257,7cyclictest41-21ksoftirqd/311:43:333
3245999253,11cyclictest87650irq/129-enp1s0-07:25:152
3245999252,7cyclictest26148-21smart_sda11:45:222
32459992515,6cyclictest9187-21diskstats09:55:132
32459992515,6cyclictest10116-21diskstats07:30:152
32459992511,6cyclictest5127-21cpuspeed_turbos11:00:122
3245999250,10cyclictest0-21swapper/207:55:192
3245399251,8cyclictest6446-21ntp_states07:20:221
3245399251,8cyclictest0-21swapper/107:11:571
3245399251,13cyclictest0-21swapper/112:07:451
3245399251,11cyclictest0-21swapper/108:11:331
3245399250,8cyclictest0-21swapper/108:23:491
3245399250,5cyclictest0-21swapper/110:08:411
3245399250,5cyclictest0-21swapper/110:08:411
3245399250,12cyclictest0-21swapper/110:15:031
3245399250,11cyclictest0-21swapper/107:26:001
3245399250,10cyclictest0-21swapper/109:52:471
3244899259,14cyclictest17428-21perf10:15:020
3244899252,11cyclictest18058-21awk11:30:010
3244899252,11cyclictest1765-21kworker/0:009:57:410
3244899251,9cyclictest27773-21kworker/0:209:30:330
3244899250,9cyclictest21068-21kworker/0:109:21:410
3244899250,11cyclictest31221-21kworker/0:208:20:140
237772250,1sleep339-21rcuc/309:15:123
137552250,3sleep339-21rcuc/311:20:013
3245999249,6cyclictest20184-21dnf10:20:142
3245999248,7cyclictest30799-21cat10:45:022
3245999246,4cyclictest13253-21sed07:35:252
3245999245,11cyclictest2557-21kworker/u8:111:35:122
32459992411,10cyclictest4213-21gltestperf12:10:182
3245399244,3cyclictest0-21swapper/111:40:101
3245399241,9cyclictest0-21swapper/109:10:161
3245399241,9cyclictest0-21swapper/108:47:221
3245399241,9cyclictest0-21swapper/107:51:571
3245399241,11cyclictest0-21swapper/111:20:541
3245399241,11cyclictest0-21swapper/111:20:541
3245399240,9cyclictest0-21swapper/112:22:211
3245399240,8cyclictest0-21swapper/109:33:171
3245399240,11cyclictest0-21swapper/109:25:051
3244899242,9cyclictest11833-21kworker/0:108:54:170
3244899242,9cyclictest11833-21kworker/0:108:54:170
3244899241,9cyclictest31871-21kworker/0:309:38:050
3244899241,11cyclictest24664-21munin-run11:45:010
3244899240,10cyclictest1018-21kworker/0:312:06:170
3246599239,3cyclictest8362expr10:50:123
3246599238,3cyclictest41-21ksoftirqd/308:46:133
3246599237,5cyclictest87350irq/126-enp1s0-12:25:163
3246599233,9cyclictest39-21rcuc/309:55:023
3246599231,6cyclictest0-21swapper/310:07:583
3246599231,6cyclictest0-21swapper/310:07:583
3246599231,10cyclictest0-21swapper/311:53:093
3245999236,14cyclictest25747-21unixbench_multi10:30:262
3245999234,10cyclictest0-21swapper/211:31:542
3245999231,5cyclictest0-21swapper/209:17:402
32459992315,5cyclictest7711-21ntp_states09:50:202
32459992315,5cyclictest20479-21smart_sda09:05:222
32459992313,6cyclictest14494-21/usr/sbin/munin07:40:152
32459992313,6cyclictest14494-21/usr/sbin/munin07:40:152
3245399231,9cyclictest0-21swapper/109:41:571
3245399230,8cyclictest0-21swapper/110:31:491
3245399230,10cyclictest0-21swapper/110:43:591
3244899239,2cyclictest28097-21kworker/0:310:40:220
3244899232,9cyclictest31221-21kworker/0:208:34:220
3244899231,12cyclictest21209-21perf09:10:010
32459992214,5cyclictest5723-21diskstats07:20:142
32459992214,5cyclictest15483-21hddtemp_smartct08:55:152
32459992214,5cyclictest11183-21irqstats08:45:162
32459992213,6cyclictest12167-21ls10:00:232
32459992212,7cyclictest27701-21diskstats08:10:132
3244899229,2cyclictest32564-21kworker/0:110:56:100
3244899228,3cyclictest10493-21kworker/0:110:05:300
3244899228,3cyclictest10493-21kworker/0:110:05:290
3244899227,4cyclictest27039-21kworker/0:008:16:140
3244899222,10cyclictest15814-21kworker/0:108:10:580
32448992211,3cyclictest1018-21kworker/0:312:10:010
3244899221,10cyclictest5472-21kworker/0:308:45:140
3244899221,10cyclictest31221-21kworker/0:208:35:580
3244899221,10cyclictest2460-21kworker/0:210:53:540
28782220,4sleep339-21rcuc/309:40:173
3246599217,4cyclictest5058-21grep12:10:273
3246599215,6cyclictest41-21ksoftirqd/311:23:413
3246599215,6cyclictest41-21ksoftirqd/311:23:413
3246599212,2cyclictest41-21ksoftirqd/310:20:013
3246599211,6cyclictest0-21swapper/310:44:173
32465992115,3cyclictest21189-21dump-pmu-power09:10:013
32465992111,2cyclictest87650irq/129-enp1s0-09:45:113
32459992114,4cyclictest556-21smart_sda08:20:242
32459992113,5cyclictest13659-21ntp_states08:50:192
32459992113,5cyclictest13659-21ntp_states08:50:192
32459992112,6cyclictest30561-21/usr/sbin/munin11:55:242
32459992112,6cyclictest25154-21/usr/sbin/munin08:05:102
32459992112,6cyclictest18041-21/usr/sbin/munin09:00:202
3244899219,2cyclictest11800-21kworker/0:112:30:290
3244899219,2cyclictest11800-21kworker/0:112:30:290
3244899217,3cyclictest15117-21kworker/0:010:25:220
3246599208,3cyclictest0-21swapper/308:16:573
3246599207,2cyclictest87350irq/126-enp1s0-08:04:253
3246599203,9cyclictest39-21rcuc/310:00:153
3246599202,6cyclictest30641-21threads11:55:243
3246599201,9cyclictest0-21swapper/309:10:453
3246599201,8cyclictest0-21swapper/309:20:163
3246599201,7cyclictest0-21swapper/310:30:443
3246599201,13cyclictest0-21swapper/310:25:223
3246599201,11cyclictest26429-21/usr/sbin/munin10:35:143
3246599201,11cyclictest12359-21/usr/sbin/munin12:30:213
3246599201,11cyclictest12359-21/usr/sbin/munin12:30:213
3246599201,11cyclictest0-21swapper/309:51:463
3246599201,10cyclictest401ktimersoftd/309:30:013
32459992013,4cyclictest6677-21/usr/sbin/munin08:35:152
32459992012,5cyclictest29894-21diskstats08:15:132
3244899209,4cyclictest0-21swapper/011:20:010
3244899206,4cyclictest1765-21kworker/0:009:52:010
3244899202,10cyclictest14329-21kworker/0:008:55:170
3244899201,12cyclictest16361-21kworker/0:112:39:570
3244899201,11cyclictest3181-21/usr/sbin/munin07:15:200
3244899200,8cyclictest8564-1kworker/u9:109:29:590
3244899200,10cyclictest7708-21kworker/0:212:26:050
221212201,2sleep30-21swapper/311:35:473
205562200,4sleep011-21rcuc/011:35:110
3246599195,6cyclictest41-21ksoftirqd/308:00:023
3246599195,5cyclictest87650irq/129-enp1s0-08:55:143
3246599195,2cyclictest0-21swapper/310:48:043
3246599194,8cyclictest87350irq/126-enp1s0-07:50:183
3246599194,6cyclictest41-21ksoftirqd/312:15:203
3246599193,6cyclictest41-21ksoftirqd/312:05:163
3246599191,7cyclictest0-21swapper/308:30:203
3246599191,10cyclictest32404-21/usr/sbin/munin09:35:263
3244899197,2cyclictest9496-21kworker/0:007:30:170
3244899195,3cyclictest0-21swapper/011:33:160
3244899192,12cyclictest9496-21kworker/0:007:37:010
3244899191,11cyclictest5063-21/usr/sbin/munin11:00:250
3244899191,10cyclictest5778-21/usr/sbin/munin12:15:150
3244899191,10cyclictest0-21swapper/008:25:440
265642191,2sleep30-21swapper/311:46:233
32712181,2sleep30-21swapper/308:26:293
3246599187,3cyclictest5374-21/usr/sbin/munin07:20:223
3246599187,2cyclictest8564-1kworker/u9:111:30:193
3246599185,3cyclictest10809-21ntpq07:30:203
3246599184,5cyclictest87350irq/126-enp1s0-10:55:123
3246599182,9cyclictest30201-21/usr/sbin/munin09:30:163
3246599182,6cyclictest41-21ksoftirqd/307:35:243
3246599182,5cyclictest7970-21/usr/sbin/munin12:20:153
3246599181,7cyclictest0-21swapper/312:35:083
3246599181,7cyclictest0-21swapper/308:40:253
3246599181,6cyclictest0-21swapper/311:00:193
3246599181,3cyclictest0-21swapper/312:00:363
32465991811,4cyclictest16663-21grep11:25:153
3246599180,10cyclictest0-21swapper/310:24:553
3244899187,3cyclictest32645-21/usr/sbin/munin07:10:130
3244899187,2cyclictest24462-21kworker/0:011:53:090
3244899186,3cyclictest10531-21kworker/0:011:20:210
3244899186,3cyclictest10531-21kworker/0:011:20:210
3244899183,6cyclictest0-21swapper/009:40:470
3244899182,13cyclictest14802-21tail07:40:160
3244899182,13cyclictest14802-21tail07:40:160
3244899182,10cyclictest2099-21kworker/0:207:25:450
3244899181,11cyclictest0-21swapper/012:04:390
3244899181,10cyclictest19500-21kworker/u8:110:20:210
3244899181,10cyclictest0-21swapper/008:42:540
251452181,2sleep00-21swapper/008:05:030
172332180,2sleep30-21swapper/310:12:533
3246599171,7cyclictest17189-21cstates09:00:133
3246599171,14cyclictest11054-21users11:10:253
3246599171,14cyclictest0-21swapper/307:10:363
3246599170,15cyclictest0-21swapper/308:53:223
3246599170,15cyclictest0-21swapper/308:53:223
3244899178,3cyclictest27657-21sendmail_mailqu10:35:230
3244899178,3cyclictest0-21swapper/012:23:030
3244899172,6cyclictest0-21swapper/010:30:300
3244899171,9cyclictest0-21swapper/009:00:570
3244899171,10cyclictest6089-21kworker/u8:010:01:060
181762170,2sleep30-21swapper/307:48:183
3246599165,4cyclictest87350irq/126-enp1s0-08:20:213
3246599162,3cyclictest7571-21/usr/sbin/munin07:25:133
32465991612,3cyclictest0-21swapper/307:40:103
32465991612,3cyclictest0-21swapper/307:40:093
3246599161,14cyclictest0-21swapper/308:38:173
3246599161,13cyclictest0-21swapper/308:05:103
3246599161,12cyclictest27352-21/usr/sbin/munin08:10:213
3246599160,4cyclictest2478-21systemctl07:15:023
3244899162,12cyclictest20749-21/usr/sbin/munin07:55:110
3244899160,7cyclictest0-21swapper/008:00:130
32448991510,3cyclictest17833-21unixbench-2d07:45:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional