You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 03:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 09, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
118852237164,24sleep30-21swapper/319:07:023
120632226153,24sleep20-21swapper/219:09:222
117852202167,23sleep10-21swapper/119:05:431
118672201165,25sleep00-21swapper/019:06:470
1705021650,4sleep21220799cyclictest00:10:242
3235721530,4sleep31221499cyclictest19:51:273
224022670,4sleep022400-21grep21:55:260
59352630,4sleep25929-21munin-run22:35:012
12207996057,2cyclictest0-21swapper/222:39:262
65982590,4sleep31221499cyclictest21:20:213
40692540,2sleep121-21rcuc/121:15:181
198432500,2sleep00-21swapper/019:25:120
240422490,3sleep230-21rcuc/219:35:022
313492470,2sleep20-21swapper/219:50:152
51382460,3sleep339-21rcuc/320:05:133
39682360,4sleep01219599cyclictest21:15:170
1219599349,12cyclictest30105-21munin-run23:30:020
1219599341,19cyclictest13933-21kworker/0:021:40:570
1220799311,21cyclictest0-21swapper/223:40:212
1220799311,14cyclictest0-21swapper/223:34:252
12195993120,3cyclictest24948-21kworker/0:222:15:000
12195993110,7cyclictest17069-21kworker/0:119:20:490
1220799301,12cyclictest0-21swapper/221:50:172
1220799301,12cyclictest0-21swapper/220:30:132
1220799300,21cyclictest0-21swapper/219:42:432
1219599307,11cyclictest24480-21basename20:50:010
273202290,2sleep30-21swapper/322:10:023
1220799291,18cyclictest0-21swapper/200:19:012
1220799291,13cyclictest0-21swapper/219:16:392
1220299291,3cyclictest4337-21latency_hist23:45:011
1219599292,25cyclictest18927-21kworker/0:323:05:250
12195992920,3cyclictest19393-21kworker/0:300:29:210
1219599291,10cyclictest2986-21kworker/0:221:25:130
1219599291,10cyclictest14932-21kworker/0:222:58:530
1220799281,21cyclictest0-21swapper/223:46:172
1220799281,17cyclictest0-21swapper/221:41:212
1220799281,14cyclictest0-21swapper/222:59:172
1220799281,14cyclictest0-21swapper/200:06:252
1220799281,13cyclictest0-21swapper/223:57:212
1220799281,12cyclictest14765-21ntpq20:25:212
1220799281,12cyclictest14765-21ntpq20:25:212
1220799281,12cyclictest0-21swapper/222:22:252
1220799281,12cyclictest0-21swapper/219:10:212
1220799281,12cyclictest0-21swapper/219:10:212
1220799281,11cyclictest0-21swapper/223:35:172
1220799281,11cyclictest0-21swapper/220:07:312
1220799281,10cyclictest0-21swapper/222:16:132
1220799280,13cyclictest0-21swapper/221:22:412
1220799280,12cyclictest0-21swapper/223:27:392
1220799280,12cyclictest0-21swapper/221:34:132
1219599286,11cyclictest87350irq/126-enp1s0-00:21:130
1219599281,13cyclictest11900-21munin-run21:35:010
1220799279,3cyclictest0-21swapper/200:30:172
1220799271,9cyclictest0-21swapper/220:49:192
1220799271,13cyclictest0-21swapper/222:46:172
1220799271,13cyclictest0-21swapper/221:55:372
1220799271,12cyclictest0-21swapper/222:52:172
1220799271,12cyclictest0-21swapper/222:52:172
1220799271,12cyclictest0-21swapper/222:02:052
1220799271,12cyclictest0-21swapper/200:26:332
1220799271,11cyclictest0-21swapper/221:14:402
1220799271,10cyclictest0-21swapper/219:38:032
1220799270,2cyclictest0-21swapper/219:24:152
1220799270,13cyclictest0-21swapper/200:39:572
1220299277,10cyclictest18779-21smartctl20:35:151
1219599279,15cyclictest31692-21munin-run22:20:010
12195992713,3cyclictest13933-21kworker/0:021:53:410
1219599271,12cyclictest5619-21kworker/0:222:35:290
1220799261,12cyclictest0-21swapper/223:14:162
1220799261,11cyclictest0-21swapper/223:53:492
1220799260,13cyclictest0-21swapper/222:14:492
1220799260,12cyclictest0-21swapper/222:07:052
1220299260,17cyclictest0-21swapper/122:44:281
1219599262,10cyclictest596-21kworker/0:121:08:570
1219599262,10cyclictest23656-21kworker/0:019:42:090
1219599262,10cyclictest21601-21kworker/0:000:32:130
1219599261,9cyclictest6487-21kworker/0:100:15:090
1219599261,9cyclictest30382-21kworker/0:119:55:170
1219599261,20cyclictest3364-21kworker/0:122:30:410
1219599261,13cyclictest9082-21munin-run20:15:010
1219599261,11cyclictest17809-21kworker/0:120:40:330
1219599261,10cyclictest24948-21kworker/0:222:04:490
1219599261,10cyclictest18394-21kworker/0:221:49:330
12214992520,3cyclictest14124-21cat00:05:163
1220799251,13cyclictest0-21swapper/200:20:292
1220799251,12cyclictest0-21swapper/221:28:012
1220799251,10cyclictest0-21swapper/221:37:332
1220799251,10cyclictest0-21swapper/221:37:332
1220799250,17cyclictest0-21swapper/223:22:192
1219599259,12cyclictest31323-21strings23:30:190
1219599258,3cyclictest22302-21kworker/0:020:50:130
1219599255,3cyclictest14932-21kworker/0:223:14:090
1219599252,9cyclictest22622-21kworker/0:023:19:330
1219599252,8cyclictest30785-21kworker/0:021:11:530
1219599251,10cyclictest1978-21kworker/0:023:39:250
1221499247,7cyclictest23403-21expr22:00:123
1220799241,9cyclictest0-21swapper/221:08:392
1220799241,15cyclictest0-21swapper/222:41:292
1220799241,10cyclictest0-21swapper/223:16:452
1220799240,12cyclictest0-21swapper/221:16:282
1219599248,3cyclictest14058-21kworker/0:019:14:370
1219599248,3cyclictest14058-21kworker/0:019:14:360
1219599247,9cyclictest13252-21kworker/0:120:24:200
1219599245,4cyclictest6487-21kworker/0:123:57:170
1219599244,9cyclictest11664-21kworker/0:321:37:530
1219599244,9cyclictest11664-21kworker/0:321:37:530
1219599244,7cyclictest2084-21kworker/0:223:47:490
1219599242,10cyclictest6487-21kworker/0:100:14:530
1219599241,9cyclictest2986-21kworker/0:221:24:090
1219599241,8cyclictest24948-21kworker/0:222:20:360
12195992411,2cyclictest14058-21kworker/0:019:15:250
1221499238,4cyclictest87350irq/126-enp1s0-21:15:323
1220799231,9cyclictest0-21swapper/223:06:192
1220799230,9cyclictest0-21swapper/220:50:032
12202992317,3cyclictest10997-21ntp_states21:30:201
1220299231,16cyclictest0-21swapper/121:14:301
12202992310,11cyclictest27311-21kworker/u8:220:23:341
12202992310,11cyclictest25866-21kworker/u8:119:49:331
1220299230,14cyclictest0-21swapper/100:00:401
1220299230,14cyclictest0-21swapper/100:00:401
1219599239,3cyclictest11028-21kworker/0:220:17:370
1219599238,3cyclictest11028-21kworker/0:220:35:130
1219599236,8cyclictest21809-21kworker/0:122:06:000
1219599236,3cyclictest28057-21kworker/0:319:45:230
1219599232,11cyclictest25369-21kworker/0:123:20:130
1219599231,9cyclictest561-21kworker/0:022:29:130
1219599230,8cyclictest12798-21kworker/0:300:07:050
1219599230,11cyclictest4704-21idleruntime-cro20:05:010
1221499221,5cyclictest0-21swapper/323:19:223
1221499221,10cyclictest547-21sa122:20:213
1220799221,11cyclictest0-21swapper/200:02:092
1220799221,11cyclictest0-21swapper/200:02:092
1220299228,12cyclictest13770-21kworker/u8:221:43:001
1220299226,3cyclictest0-21swapper/122:22:341
12202992213,3cyclictest0-21swapper/120:25:091
12202992213,3cyclictest0-21swapper/120:25:091
1220299221,18cyclictest12526-21perf22:50:001
1220299221,16cyclictest0-21swapper/121:01:031
1220299221,16cyclictest0-21swapper/120:10:221
1220299221,15cyclictest0-21swapper/122:25:301
1220299221,15cyclictest0-21swapper/120:42:041
12202992210,4cyclictest1403-21memory19:55:181
1220299220,8cyclictest89250irq/135-enp2s0-20:31:311
1220299220,6cyclictest0-21swapper/121:26:561
1220299220,16cyclictest0-21swapper/122:18:401
1219599226,4cyclictest26321-21kworker/0:221:03:410
1219599226,12cyclictest87650irq/129-enp1s0-22:53:330
1219599226,12cyclictest87650irq/129-enp1s0-22:53:330
1219599225,2cyclictest7996-21kworker/0:022:48:330
1219599221,9cyclictest13252-21kworker/0:120:25:290
1219599221,9cyclictest13252-21kworker/0:120:25:290
1219599220,9cyclictest26321-21kworker/0:220:55:090
1221499216,3cyclictest808-21snmpd22:45:203
1221499211,6cyclictest0-21swapper/322:28:583
1221499211,12cyclictest0-21swapper/321:38:453
1221499211,12cyclictest0-21swapper/321:38:443
1221499211,12cyclictest0-21swapper/300:17:443
1221499211,10cyclictest20336-21grep23:05:193
1220799217,3cyclictest0-21swapper/222:27:172
1220799216,3cyclictest28774-21chrt20:59:052
1220799211,11cyclictest19409-21switchtime20:35:232
1220799210,9cyclictest0-21swapper/220:12:572
1220299212,14cyclictest87650irq/129-enp1s0-22:51:101
1220299212,14cyclictest87650irq/129-enp1s0-22:51:091
1220299211,3cyclictest13076-21hddtemp_smartct19:10:151
1220299211,3cyclictest13076-21hddtemp_smartct19:10:141
12202992113,3cyclictest0-21swapper/121:50:321
12202992113,3cyclictest0-21swapper/119:38:481
1220299211,18cyclictest0-21swapper/100:15:121
1220299211,15cyclictest0-21swapper/122:30:251
1220299211,14cyclictest0-21swapper/122:04:141
1220299211,11cyclictest21850-21sed19:30:021
1220299210,16cyclictest0-21swapper/119:34:221
1220299210,15cyclictest0-21swapper/120:56:121
1219599218,2cyclictest11028-21kworker/0:220:32:410
1219599216,6cyclictest19293-21kworker/0:219:30:570
1219599216,6cyclictest0-21swapper/023:00:040
1219599215,4cyclictest6487-21kworker/0:123:51:370
1219599211,8cyclictest6487-21kworker/0:100:01:200
1219599211,8cyclictest6487-21kworker/0:100:01:200
1221499202,10cyclictest4569-21/usr/sbin/munin23:45:153
1221499201,11cyclictest26576-21/usr/sbin/munin00:35:163
1221499201,11cyclictest20917-21/usr/sbin/munin21:55:263
1221499201,11cyclictest11505-21/usr/sbin/munin20:20:223
1220799206,3cyclictest0-21swapper/221:49:352
1220799205,3cyclictest18636-21unixbench_multi23:00:272
1220799203,7cyclictest89150irq/134-enp2s0-19:55:092
1220299202,4cyclictest6539-21munin-run23:50:011
1220299202,16cyclictest0-21swapper/120:48:421
12202992016,3cyclictest0-21swapper/123:17:301
12202992016,3cyclictest0-21swapper/100:27:071
12202992016,2cyclictest13056-21sort21:35:171
12202992016,2cyclictest13056-21sort21:35:171
12202992012,3cyclictest0-21swapper/119:18:421
1220299201,14cyclictest0-21swapper/122:09:221
1220299201,14cyclictest0-21swapper/121:05:031
1220299201,14cyclictest0-21swapper/120:02:011
1220299201,14cyclictest0-21swapper/119:42:141
1220299201,11cyclictest29394-21smart_sda23:25:231
1220299201,11cyclictest17773-21cstates19:20:121
1220299201,11cyclictest15922-21df00:10:121
1220299201,10cyclictest28019-21fschecks_time22:10:151
1220299201,10cyclictest10943-21perf00:00:011
1220299200,15cyclictest0-21swapper/122:39:531
1220299200,15cyclictest0-21swapper/119:52:211
1219599209,3cyclictest27639-21ntp_states00:35:210
1219599205,7cyclictest30382-21kworker/0:119:51:450
1219599205,3cyclictest7996-21kworker/0:022:41:490
12195992015,3cyclictest6880-21sh20:10:010
199802191,2sleep30-21swapper/300:20:093
189432190,3sleep30-21swapper/323:02:523
1221499198,3cyclictest12745-21/usr/sbin/munin22:50:213
1221499198,3cyclictest12745-21/usr/sbin/munin22:50:213
1221499191,9cyclictest13657-21mailstats19:10:223
1221499191,9cyclictest13657-21mailstats19:10:223
1221499191,4cyclictest0-21swapper/321:44:583
12214991912,4cyclictest16733-21grep00:10:213
1221499191,16cyclictest0-21swapper/320:40:143
1221499191,10cyclictest7108-21/usr/sbin/munin20:10:183
12214991910,7cyclictest41-21ksoftirqd/319:30:363
1220799195,3cyclictest0-21swapper/219:26:572
1220799190,10cyclictest0-21swapper/220:00:052
1220299197,3cyclictest15358-21crond00:10:011
1220299191,8cyclictest0-21swapper/123:20:051
12202991912,2cyclictest0-21swapper/121:22:131
1220299191,17cyclictest0-21swapper/122:55:241
1220299191,17cyclictest0-21swapper/100:21:001
12202991911,3cyclictest0-21swapper/121:47:081
1220299191,12cyclictest0-21swapper/123:00:541
1220299191,12cyclictest0-21swapper/100:30:411
1220299191,10cyclictest8732-21latency_hist23:55:011
1220299190,2cyclictest862-21diskmemload23:10:571
1220299190,17cyclictest0-21swapper/123:09:271
1220299190,14cyclictest0-21swapper/120:17:421
1219599195,9cyclictest87350irq/126-enp1s0-19:40:010
1219599191,10cyclictest2368-21/usr/sbin/munin23:40:200
1221499188,3cyclictest87350irq/126-enp1s0-22:35:283
1221499186,4cyclictest9708-21awk21:30:013
1221499185,3cyclictest25866-21kworker/u8:121:50:353
1221499181,9cyclictest808-21snmpd22:10:143
1221499181,7cyclictest0-21swapper/321:47:133
1221499181,6cyclictest0-21swapper/300:31:363
12214991812,3cyclictest22492-21/usr/sbin/munin20:45:103
12214991812,3cyclictest11088-21sed00:00:013
12214991811,4cyclictest8964-21grep22:40:153
12214991811,4cyclictest742-21sed23:35:143
12214991810,5cyclictest8733-21sh23:55:013
1221499180,10cyclictest0-21swapper/322:18:343
1220799180,7cyclictest0-21swapper/219:45:232
1220799180,11cyclictest0-21swapper/221:04:042
12202991815,2cyclictest0-21swapper/123:35:101
12202991812,2cyclictest0-21swapper/120:50:481
12202991811,2cyclictest0-21swapper/120:06:491
1220299181,10cyclictest862-21diskmemload23:30:221
1220299180,1cyclictest0-21swapper/121:56:271
1220299180,17cyclictest0-21swapper/100:36:471
1221499179,5cyclictest909-21sssd_nss22:55:233
1221499178,3cyclictest12050irq/123-ahci[0019:23:593
1221499175,3cyclictest862-21diskmemload23:32:313
1221499172,6cyclictest0-21swapper/300:25:193
1221499172,5cyclictest16537-40dnf19:15:233
1221499171,7cyclictest0-21swapper/322:30:073
1221499171,5cyclictest0-21swapper/321:31:503
1221499171,4cyclictest0-21swapper/323:22:393
1221499171,3cyclictest26891-21/usr/sbin/munin20:55:153
12214991712,3cyclictest17492-21users20:30:253
1221499171,14cyclictest0-21swapper/320:25:093
1221499171,14cyclictest0-21swapper/320:25:093
12214991711,3cyclictest29063-21gltestperf19:45:143
1221499171,10cyclictest839-21lldpd00:03:063
1221499171,10cyclictest839-21lldpd00:03:063
12214991710,4cyclictest3372-21gltestperf20:00:153
1221499170,7cyclictest0-21swapper/320:15:243
1221499170,14cyclictest29126-21needreboot23:25:193
1220799171,9cyclictest0-21swapper/220:43:272
1220799171,7cyclictest9307-21/usr/sbin/munin20:15:182
1220799170,11cyclictest0-21swapper/220:20:092
1221499169,4cyclictest909-21sssd_nss20:50:143
1221499169,4cyclictest26917-21gltestperf19:40:153
1221499161,9cyclictest0-21swapper/321:11:123
1221499161,6cyclictest30685-21vmstat21:00:263
12214991612,3cyclictest0-21swapper/319:37:143
12214991611,3cyclictest3005-21grep23:40:143
12214991611,3cyclictest22911-21timerandwakeup23:10:243
12214991611,3cyclictest19688-21/usr/sbin/munin19:25:203
12214991611,3cyclictest18101-21/usr/sbin/munin20:35:153
12214991611,2cyclictest31292-21/usr/sbin/munin21:05:113
1221499150,14cyclictest0-21swapper/319:56:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional