You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-05 - 15:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Fri Jun 05, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
238092241165,25sleep00-21swapper/007:09:540
237012237163,24sleep20-21swapper/207:08:312
237912235163,23sleep30-21swapper/307:09:403
236382204167,25sleep10-21swapper/107:07:421
23914998675,9cyclictest12609-21diskmemload11:26:312
10112800,3sleep081ktimersoftd/011:10:120
23914995956,2cyclictest0-21swapper/211:41:272
197342530,4sleep02390199cyclictest09:25:130
196522530,4sleep121-21rcuc/108:10:171
248372520,2sleep20-21swapper/212:04:562
248372520,2sleep20-21swapper/212:04:562
35132500,3sleep10-21swapper/107:35:151
201062490,4sleep22391499cyclictest11:50:512
224312480,1sleep00-21swapper/010:45:120
2391899381,24cyclictest0-21swapper/308:04:293
2391899381,20cyclictest0-21swapper/311:15:413
2391899371,22cyclictest0-21swapper/311:30:333
2391499373,6cyclictest16016-40dnf08:00:342
2391499362,7cyclictest6856-21/usr/sbin/munin10:10:142
23914993613,20cyclictest6230-21/usr/sbin/munin08:55:142
23901993611,12cyclictest19822-21munin-run10:40:010
23918993516,3cyclictest0-21swapper/307:35:133
2391499356,4cyclictest12608-21sh09:10:012
2391499355,4cyclictest12609-21diskmemload09:44:292
23914993513,19cyclictest5285-21/usr/sbin/munin11:20:152
2391499346,4cyclictest31034-21/usr/sbin/munin08:35:242
2391499346,4cyclictest31034-21/usr/sbin/munin08:35:242
2391499342,9cyclictest808-21snmpd11:12:432
23914993413,18cyclictest8430-21/usr/sbin/munin09:00:132
23918993324,2cyclictest0-21swapper/308:25:163
23918993318,6cyclictest0-21swapper/312:01:063
23918993318,6cyclictest0-21swapper/312:01:063
2391499339,3cyclictest827-21runrttasks09:48:372
2391499339,3cyclictest827-21runrttasks09:48:372
23914993311,3cyclictest28866-21users12:10:252
2391499329,3cyclictest24446-21/usr/sbin/munin10:50:242
2391499327,4cyclictest0-21swapper/208:05:272
2391499324,4cyclictest16177-21pmu-power09:15:252
2391499322,5cyclictest5009-21df_inode10:05:142
2391499321,5cyclictest0-21swapper/212:31:062
2391499321,4cyclictest22633-21dump-pmu-power12:00:012
23914993211,3cyclictest4035-21/usr/sbin/munin08:50:252
23914993211,3cyclictest25321-21processes07:10:242
23914993210,3cyclictest29628-21if_err_enp2s011:00:172
2390199327,12cyclictest29582-21munin-run08:35:010
2391899311,13cyclictest0-21swapper/311:01:293
2391499319,3cyclictest827-21runrttasks12:07:262
2391499319,3cyclictest827-21runrttasks12:07:262
2391499319,3cyclictest29474-21/usr/sbin/munin12:15:182
2391499319,2cyclictest3427-21/usr/sbin/munin07:35:282
2391499318,3cyclictest17849-21/usr/sbin/munin10:35:192
23914993114,14cyclictest7139-21grep07:40:252
2391499311,3cyclictest0-21swapper/209:30:122
23914993111,3cyclictest827-21runrttasks08:40:222
2391499307,3cyclictest26645-21/usr/sbin/munin10:55:182
2391499307,3cyclictest26645-21/usr/sbin/munin10:55:182
2391499305,4cyclictest1232-21cpu07:30:112
2391499305,3cyclictest12050irq/123-ahci[0007:50:012
2391499304,3cyclictest12248-21needreboot10:20:192
2391499303,4cyclictest32423-21iwlist12:20:152
2391499303,3cyclictest87650irq/129-enp1s0-11:45:122
2391499302,7cyclictest21930-21taskset10:44:042
2391499301,4cyclictest790-21in:imjournal08:00:022
2391499301,3cyclictest0-21swapper/210:15:132
2391499301,2cyclictest87350irq/126-enp1s0-08:10:102
23914993010,4cyclictest0-21swapper/212:35:172
2391499300,5cyclictest0-21swapper/209:10:142
2391899299,4cyclictest7827-21df_inode11:25:133
2391899299,11cyclictest25941-21sort09:40:013
23918992911,3cyclictest29010-21chrt12:12:083
2391899291,12cyclictest0-21swapper/308:13:423
2391499299,3cyclictest827-21runrttasks10:27:332
2391499299,3cyclictest1649-21cut12:25:112
2391499298,3cyclictest30417-21/usr/sbin/munin09:50:222
2391499295,3cyclictest18264-21grep09:20:192
2391499292,5cyclictest808-21snmpd10:30:112
2391499291,6cyclictest579-21systemd-journal08:20:002
2391499291,6cyclictest23933-21cut09:35:122
2391499291,5cyclictest0-21swapper/210:46:182
23914992910,2cyclictest827-21runrttasks09:59:172
2391499290,1cyclictest0-21swapper/211:17:462
2390199299,16cyclictest30833-21munin-run11:05:010
2391899281,9cyclictest0-21swapper/307:51:073
2391899281,12cyclictest0-21swapper/308:43:093
2391499289,7cyclictest0-21swapper/207:50:322
2391499289,2cyclictest26055-21grep08:25:162
2391499288,2cyclictest827-21runrttasks07:15:552
2391499287,3cyclictest9682-21/usr/sbin/munin11:30:242
2391499285,4cyclictest31376-21cstates11:05:132
2391499281,8cyclictest0-21swapper/209:25:212
2391499281,5cyclictest13743-21chrt11:38:522
2390799280,6cyclictest0-21swapper/110:55:131
2390799280,6cyclictest0-21swapper/110:55:131
2391899279,3cyclictest0-21swapper/310:48:333
2391899278,8cyclictest0-21swapper/307:47:173
23918992711,3cyclictest14522-21ntp_states10:25:223
2391899271,13cyclictest0-21swapper/312:05:063
2391899271,13cyclictest0-21swapper/312:05:063
2391899271,12cyclictest0-21swapper/310:21:543
2391899271,12cyclictest0-21swapper/307:43:493
23918992710,9cyclictest0-21swapper/311:21:033
2391499278,3cyclictest31722-21df07:25:132
2391499278,3cyclictest28649-21ntpq08:30:192
23914992724,2cyclictest0-21swapper/208:20:032
2391499271,6cyclictest2455-21/usr/sbin/munin10:00:242
2391499271,6cyclictest1855-21/usr/sbin/munin08:45:122
2390799271,11cyclictest31944-21cat08:40:011
2390799271,11cyclictest31944-21cat08:40:011
2390799271,11cyclictest17336-21ntp_states11:45:211
2391899269,9cyclictest12609-21diskmemload10:18:583
23918992616,3cyclictest0-21swapper/309:22:533
2391899261,13cyclictest0-21swapper/311:12:493
23918992611,3cyclictest0-21swapper/311:05:253
2391899261,13cyclictest0-21swapper/307:57:293
2391899261,12cyclictest0-21swapper/310:32:053
2391899261,10cyclictest0-21swapper/309:18:333
23918992610,8cyclictest0-21swapper/311:56:543
23918992610,8cyclictest0-21swapper/311:42:173
23918992610,3cyclictest0-21swapper/310:39:333
23918992610,3cyclictest0-21swapper/307:23:013
2391899260,13cyclictest0-21swapper/308:30:123
2391499261,2cyclictest0-21swapper/207:20:182
2390799266,17cyclictest0-21swapper/111:05:471
2390799261,16cyclictest0-21swapper/112:37:281
2391899259,10cyclictest0-21swapper/307:12:383
2391899258,8cyclictest0-21swapper/309:01:573
2391899258,3cyclictest0-21swapper/308:38:053
2391899258,3cyclictest0-21swapper/308:38:053
2391899258,3cyclictest0-21swapper/307:19:093
23918992511,4cyclictest0-21swapper/309:56:273
2391899251,13cyclictest0-21swapper/310:42:533
2391899251,13cyclictest0-21swapper/310:07:383
2391899251,13cyclictest0-21swapper/309:34:293
2391899251,12cyclictest0-21swapper/311:53:023
2391899251,12cyclictest0-21swapper/310:11:543
2391899251,12cyclictest0-21swapper/309:49:253
2391899251,12cyclictest0-21swapper/309:49:253
2391899251,12cyclictest0-21swapper/309:06:053
2391899251,12cyclictest0-21swapper/308:53:253
2391899251,12cyclictest0-21swapper/307:31:013
2391899251,11cyclictest0-21swapper/312:21:063
2391899251,11cyclictest0-21swapper/309:50:533
2391899251,11cyclictest0-21swapper/308:46:013
2391899251,11cyclictest0-21swapper/308:16:223
2391899251,11cyclictest0-21swapper/307:27:253
23918992510,3cyclictest0-21swapper/311:45:183
2390799251,16cyclictest30606-40dnf11:02:251
2390799251,16cyclictest0-21swapper/111:52:071
2390799251,16cyclictest0-21swapper/110:18:101
2390199258,15cyclictest3821-21awk08:50:020
2391899249,8cyclictest0-21swapper/308:56:013
2391899249,3cyclictest24922-21fschecks_time10:50:153
2391899248,3cyclictest0-21swapper/310:57:453
2391899248,3cyclictest0-21swapper/310:57:453
2391899247,3cyclictest0-21swapper/309:12:493
2391899246,11cyclictest0-21swapper/309:27:493
2391899245,10cyclictest0-21swapper/310:03:373
2391899241,12cyclictest0-21swapper/312:27:223
2390799241,18cyclictest0-21swapper/109:18:471
2390799241,11cyclictest5634-21diskstats11:20:131
2390799241,11cyclictest1993-21if_err_enp1s007:30:171
2390799241,11cyclictest16554-21cat08:05:021
2390799241,11cyclictest14803-21latency_hist09:15:031
2390799240,17cyclictest0-21swapper/111:56:351
2390799240,17cyclictest0-21swapper/110:29:101
2390799240,16cyclictest0-21swapper/112:15:201
23901992412,2cyclictest24655-1kworker/u9:012:18:410
2391899239,3cyclictest0-21swapper/312:33:383
2391899239,3cyclictest0-21swapper/308:23:143
2391899239,3cyclictest0-21swapper/308:05:453
2391899237,3cyclictest0-21swapper/312:17:423
23918992310,8cyclictest0-21swapper/311:36:333
2390799238,3cyclictest3812-21munin-run08:50:011
2390799232,18cyclictest909-21sssd_nss08:25:241
2390799231,3cyclictest26422-21munin-run10:55:021
2390799231,1cyclictest0-21swapper/112:11:251
2390799231,17cyclictest0-21swapper/112:08:051
2390799231,17cyclictest0-21swapper/112:08:051
2390799231,10cyclictest10403-21crond09:05:011
2390799230,5cyclictest0-21swapper/109:20:261
2390799230,17cyclictest0-21swapper/111:35:271
2390799230,15cyclictest0-21swapper/111:19:511
23918992210,3cyclictest0-21swapper/312:36:083
2390799226,7cyclictest0-21swapper/112:26:161
2390799224,7cyclictest0-21swapper/109:38:291
23907992217,2cyclictest24742-21chrt12:03:421
23907992217,2cyclictest24742-21chrt12:03:421
23907992216,4cyclictest12820-21sed10:20:251
2390799221,19cyclictest0-21swapper/112:30:111
2390799221,19cyclictest0-21swapper/111:25:171
2390799221,19cyclictest0-21swapper/109:31:241
2390199223,3cyclictest0-21swapper/010:29:340
2390199221,13cyclictest24777-21missed_timers09:35:190
2390799217,5cyclictest2471-21cpu10:00:121
2390799213,6cyclictest0-21swapper/109:26:311
2390799211,18cyclictest0-21swapper/109:40:091
2390799211,18cyclictest0-21swapper/109:05:171
2390799211,18cyclictest0-21swapper/107:25:441
2390799211,14cyclictest87650irq/129-enp1s0-10:37:201
2390199218,4cyclictest14954-21latency11:40:170
2390199216,2cyclictest12947-21ntp_states11:35:210
2390199214,3cyclictest0-21swapper/011:45:110
2390199213,2cyclictest0-21swapper/010:31:440
2390199212,4cyclictest13414-21perl09:10:140
2390199211,3cyclictest23363-21fschecks_time12:00:130
2390199211,3cyclictest23363-21fschecks_time12:00:130
23901992112,3cyclictest25646-21fschecks_time12:05:140
23901992112,3cyclictest25646-21fschecks_time12:05:140
166162211,2sleep116702-21cpuspeed_turbos08:05:131
309442200,3sleep00-21swapper/009:50:140
275702200,4sleep02390199cyclictest10:55:190
275702200,4sleep02390199cyclictest10:55:190
2391899201,10cyclictest26022-21/usr/sbin/munin09:40:233
2390799208,3cyclictest27808-21expr08:30:111
2390799202,2cyclictest0-21swapper/107:10:121
2390799202,16cyclictest0-21swapper/108:20:041
23907992016,3cyclictest0-21swapper/109:54:421
23907992013,4cyclictest8158-21df_inode07:45:161
2390799201,18cyclictest0-21swapper/108:18:321
2390799201,10cyclictest8828-21latency_hist10:15:011
2390799201,10cyclictest579-21systemd-journal07:55:011
2390799201,10cyclictest4208-21cstates08:50:111
2390799201,10cyclictest32387-21dump-pmu-power09:59:561
2390799201,10cyclictest29031-21irqstats09:45:181
2390799201,10cyclictest29031-21irqstats09:45:171
2390799201,10cyclictest21632-21users10:40:251
2390799200,10cyclictest0-21swapper/112:24:041
2390199209,3cyclictest6856-21/usr/sbin/munin10:10:130
2390199208,3cyclictest4495-21ntp_states07:35:240
2390199207,2cyclictest827-21runrttasks08:36:060
2390199207,2cyclictest827-21runrttasks08:36:050
2390199206,4cyclictest13285-21date10:25:010
2390199205,9cyclictest0-21swapper/011:05:500
2390199205,8cyclictest24552-21cpuspeed_turbos10:50:120
2390199205,3cyclictest0-21swapper/009:41:190
2390199204,3cyclictest0-21swapper/009:48:440
2390199204,3cyclictest0-21swapper/009:48:440
2390199204,3cyclictest0-21swapper/009:17:080
2390199204,2cyclictest0-21swapper/012:33:040
2390199202,11cyclictest0-21swapper/011:26:460
2390199202,10cyclictest21009-21/usr/sbin/munin08:15:130
23901992015,3cyclictest10381-21grep07:50:160
23901992012,6cyclictest2447-21kworker/u8:007:17:400
2390199201,16cyclictest976-21munin-run07:30:010
2390199201,13cyclictest0-21swapper/012:29:100
2390199201,12cyclictest26816-21kworker/u8:312:11:060
2390199201,11cyclictest7818-21/usr/sbin/munin07:45:190
2390199201,10cyclictest9682-21/usr/sbin/munin11:30:270
2390199200,5cyclictest0-21swapper/009:56:500
225712201,2sleep013774-21kworker/0:011:59:210
2390799194,5cyclictest0-21swapper/108:55:291
2390799192,2cyclictest87650irq/129-enp1s0-10:45:481
2390799192,16cyclictest0-21swapper/110:32:001
2390799191,9cyclictest1212-21df_inode11:10:141
23907991916,2cyclictest0-21swapper/111:40:111
23907991915,3cyclictest0-21swapper/107:21:451
23907991913,4cyclictest0-21swapper/110:05:151
2390799191,16cyclictest0-21swapper/107:16:431
2390799191,11cyclictest0-21swapper/108:43:321
2390799191,10cyclictest827-21runrttasks07:40:291
2390799190,13cyclictest87650irq/129-enp1s0-11:31:131
2390199197,3cyclictest32675-21hddtemp_smartct08:40:170
2390199196,3cyclictest0-21swapper/010:44:590
2390199194,9cyclictest0-21swapper/010:06:280
2390199194,8cyclictest6230-21/usr/sbin/munin08:55:180
2390199193,3cyclictest12050irq/123-ahci[0010:20:010
2390199193,3cyclictest0-21swapper/012:20:220
2390199191,3cyclictest23130-21grep09:30:230
2390199191,3cyclictest0-21swapper/009:20:510
23901991913,4cyclictest0-21swapper/012:35:180
23901991913,3cyclictest6356-21ntp_states11:20:210
2390199191,13cyclictest0-21swapper/011:54:050
2390199191,11cyclictest0-21swapper/008:25:170
2390799186,3cyclictest0-21swapper/107:55:171
2390199188,3cyclictest1204-21/usr/sbin/munin07:30:190
2390199183,3cyclictest0-21swapper/011:16:110
2390199181,9cyclictest18811-21/usr/sbin/munin08:10:200
2390199181,14cyclictest7148-21grep07:40:250
2390199181,14cyclictest42782sleep010:03:050
2390199177,2cyclictest25334-21gzip07:10:230
2390199174,6cyclictest4035-21/usr/sbin/munin08:50:140
23901991710,4cyclictest14179-21dump-pmu-power08:00:010
2390199161,4cyclictest16617-21/usr/sbin/munin08:05:160
2390199161,3cyclictest0-21swapper/007:20:120
2390199161,2cyclictest0-21swapper/009:05:140
23901991611,3cyclictest23990-21if_err_enp1s008:20:170
2390199151,3cyclictest0-21swapper/008:00:110
2390199150,10cyclictest0-21swapper/009:00:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional