You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 15:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed May 06, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
270572238162,27sleep00-21swapper/007:09:140
270572238162,27sleep00-21swapper/007:09:140
268362228161,54sleep30-21swapper/307:06:233
268362228161,54sleep30-21swapper/307:06:223
269432217165,39sleep20-21swapper/207:07:472
269432217165,39sleep20-21swapper/207:07:472
268092202166,24sleep10-21swapper/107:06:011
268092202166,24sleep10-21swapper/107:06:011
651321390,5sleep12719999cyclictest12:26:461
2112521390,4sleep12719999cyclictest10:35:021
27207999289,2cyclictest0-21swapper/209:48:372
27207997370,2cyclictest0-21swapper/211:54:202
197072640,5sleep219709-21timerandwakeup09:15:242
320572390,2sleep132060-21latency_hist11:00:021
27207993927,3cyclictest0-21swapper/208:55:532
2719199378,13cyclictest24088-21munin-run08:15:010
67172360,12sleep339-21rcuc/311:15:133
2719999367,13cyclictest23-21ksoftirqd/111:25:471
2719199368,13cyclictest2388-21munin-run12:20:020
2719199327,11cyclictest0-21swapper/008:50:020
2721199310,15cyclictest13318-21/usr/sbin/munin07:50:113
2721199299,18cyclictest0-21swapper/310:35:143
27211992912,5cyclictest0-21swapper/311:50:173
27207992926,2cyclictest0-21swapper/209:41:212
2721199283,12cyclictest1545-21cstates09:50:113
27211992819,6cyclictest29778-21munin-node07:15:033
27211992810,6cyclictest0-21swapper/308:05:003
27211992810,5cyclictest0-21swapper/307:10:223
2719199287,10cyclictest0-21swapper/010:00:020
2719199281,13cyclictest28503-21perf08:25:010
286042270,4sleep07-21ksoftirqd/010:50:180
27211992711,4cyclictest0-21swapper/309:30:243
27211992711,4cyclictest0-21swapper/309:30:243
27211992711,14cyclictest0-21swapper/311:30:143
2721199270,14cyclictest9201-21/usr/sbin/munin12:35:173
2721199270,11cyclictest6512-40dnf12:26:463
2720799271,13cyclictest0-21swapper/212:00:512
2721199269,15cyclictest0-21swapper/307:35:133
2721199268,16cyclictest0-21swapper/309:00:123
2721199262,20cyclictest0-21swapper/307:41:463
27211992619,4cyclictest6820-21/usr/sbin/munin10:00:203
27211992619,4cyclictest5457-21/usr/sbin/munin11:10:243
27211992619,4cyclictest3717-21/usr/sbin/munin12:20:213
27211992619,4cyclictest31574-21/usr/sbin/munin12:10:183
27211992619,4cyclictest3078-21/usr/sbin/munin07:25:153
27211992619,4cyclictest30652-21/usr/sbin/munin10:55:163
27211992619,4cyclictest23448-21/usr/sbin/munin08:10:223
27211992619,4cyclictest22787-21/usr/sbin/munin09:25:113
27211992619,4cyclictest21155-21/usr/sbin/munin09:20:153
27211992619,4cyclictest17306-21/usr/sbin/munin09:10:223
27211992618,6cyclictest27123-21/usr/sbin/munin09:35:133
27211992618,5cyclictest5023-21/usr/sbin/munin09:55:253
27211992618,5cyclictest20711-21/usr/sbin/munin08:05:153
27211992610,14cyclictest0-21swapper/312:30:103
2721199260,17cyclictest0-21swapper/309:45:183
2721199260,10cyclictest26216-21cpu12:00:123
2719999262,20cyclictest19345-21munin-run11:45:001
27211992519,4cyclictest5489-21/usr/sbin/munin08:45:123
27211992518,5cyclictest32492-21/usr/sbin/munin07:20:193
27211992518,4cyclictest5553-21/usr/sbin/munin07:30:193
27211992518,4cyclictest3964-21/usr/sbin/munin08:40:183
27211992518,4cyclictest29454-21/usr/sbin/munin08:25:153
27211992518,4cyclictest27425-21/usr/sbin/munin08:20:183
27211992518,4cyclictest24031-21/usr/sbin/munin11:55:123
27211992518,4cyclictest24031-21/usr/sbin/munin11:55:113
27211992518,4cyclictest18409-21/usr/sbin/munin11:40:203
27211992518,4cyclictest17104-21/usr/sbin/munin10:25:113
27211992518,4cyclictest13413-21/usr/sbin/munin10:15:213
27211992518,4cyclictest11944-21/usr/sbin/munin11:25:213
2721199250,8cyclictest16354-21kworker/u8:110:48:103
2721199250,12cyclictest29322-21/usr/sbin/munin09:40:243
253722250,5sleep22720799cyclictest08:15:212
2721199248,8cyclictest31746-21kworker/u8:111:08:533
2721199248,10cyclictest0-21swapper/308:55:253
2721199247,15cyclictest0-21swapper/309:05:173
2721199241,9cyclictest0-21swapper/311:48:363
27211992419,3cyclictest9030-21/usr/sbin/munin10:05:203
27211992418,4cyclictest7680-21/usr/sbin/munin08:50:133
27211992418,4cyclictest31590-21/usr/sbin/munin08:30:153
27211992418,4cyclictest15873-21/usr/sbin/munin11:35:173
27211992418,4cyclictest11764-21/usr/sbin/munin10:10:263
27211992417,4cyclictest27836-21/usr/sbin/munin10:50:123
27211992410,11cyclictest23350-21/usr/sbin/munin10:40:223
27211992410,11cyclictest23350-21/usr/sbin/munin10:40:223
2721199240,15cyclictest28377-21/usr/sbin/munin12:05:173
2721199240,14cyclictest847-21fschecks_time12:15:143
27207992421,2cyclictest0-21swapper/208:37:592
2720799241,9cyclictest0-21swapper/208:45:512
2719999241,17cyclictest0-21swapper/107:45:411
2719999241,11cyclictest0-21swapper/112:37:421
169982240,1sleep10-21swapper/110:25:101
2721199235,16cyclictest0-21swapper/308:35:243
2721199234,4cyclictest87650irq/129-enp1s0-11:23:073
2721199232,6cyclictest15554-21needreboot10:20:203
2721199232,12cyclictest87650irq/129-enp1s0-10:30:193
27211992317,4cyclictest12719-21/usr/sbin/munin07:45:263
2721199230,16cyclictest0-21swapper/311:00:263
2719999232,11cyclictest1775-21munin-run11:05:011
2719999231,16cyclictest0-21swapper/107:20:351
2719999230,15cyclictest0-21swapper/108:00:221
170522230,4sleep22720799cyclictest09:10:192
27211992214,6cyclictest24893-21/usr/sbin/munin08:15:153
2721199220,9cyclictest20124-21seq09:17:513
2720799221,5cyclictest0-21swapper/211:08:472
2719999223,2cyclictest87450irq/127-enp1s0-12:06:331
2719999222,2cyclictest0-21swapper/111:18:471
2719999221,5cyclictest0-21swapper/111:53:561
27199992214,2cyclictest0-21swapper/109:27:051
2719999221,3cyclictest25353-21cut09:30:151
2719999221,3cyclictest25353-21cut09:30:141
27199992212,3cyclictest0-21swapper/112:30:131
2719999221,18cyclictest13677-21chrt09:04:441
2719999220,16cyclictest0-21swapper/112:22:561
2719199221,12cyclictest23031-21smart_sda11:50:230
2719199220,11cyclictest0-21swapper/009:32:540
2719199220,11cyclictest0-21swapper/009:32:530
2719199220,10cyclictest0-21swapper/010:20:490
27211992112,7cyclictest0-21swapper/308:00:003
2720799218,4cyclictest0-21swapper/209:35:182
2720799211,7cyclictest0-21swapper/212:32:512
2720799211,5cyclictest17437-21awk11:40:112
2719999217,5cyclictest87650irq/129-enp1s0-09:35:121
2719999211,6cyclictest0-21swapper/111:56:341
2719999211,6cyclictest0-21swapper/111:56:341
2719999211,6cyclictest0-21swapper/111:09:201
2719999211,6cyclictest0-21swapper/110:46:331
2719999211,18cyclictest0-21swapper/107:50:141
2719999211,15cyclictest0-21swapper/112:17:581
2719999211,15cyclictest0-21swapper/109:48:301
2719999211,14cyclictest0-21swapper/109:06:281
2719999211,11cyclictest15980-21tune2fs07:55:141
27199992110,3cyclictest30394-21ntp_states09:40:201
304752200,5sleep232-21ksoftirqd/212:10:022
2720799202,7cyclictest87650irq/129-enp1s0-10:10:252
2720799201,12cyclictest0-21swapper/212:20:192
2720799201,10cyclictest6024-21rt-features12:25:222
2719999205,4cyclictest0-21swapper/111:32:001
27199992016,3cyclictest0-21swapper/110:03:481
2719999201,4cyclictest9606-21chrt07:40:161
27199992013,3cyclictest0-21swapper/108:09:151
27199992013,2cyclictest0-21swapper/112:10:211
2719999201,17cyclictest0-21swapper/110:22:451
2719999201,17cyclictest0-21swapper/107:33:141
2719999201,15cyclictest0-21swapper/108:12:151
2719999201,14cyclictest0-21swapper/110:16:321
2719999201,14cyclictest0-21swapper/109:56:511
2719999201,14cyclictest0-21swapper/108:54:491
2719999201,14cyclictest0-21swapper/108:40:451
2719999201,14cyclictest0-21swapper/107:29:101
2719999201,11cyclictest9036-21ntp_states10:05:201
2719999201,11cyclictest19923-21users09:15:261
2719999201,11cyclictest10636-21open_inodes08:55:221
2719999201,10cyclictest31054-21tr08:30:121
27199992010,2cyclictest87650irq/129-enp1s0-11:10:121
2719999200,5cyclictest0-21swapper/109:50:361
2719999200,15cyclictest0-21swapper/110:35:381
2719999200,14cyclictest0-21swapper/109:20:061
2719999200,14cyclictest0-21swapper/108:18:411
2719199206,3cyclictest0-21swapper/011:57:060
2719199206,3cyclictest0-21swapper/011:57:060
2719199206,3cyclictest0-21swapper/010:37:390
2719199204,2cyclictest0-21swapper/009:29:190
2719199202,2cyclictest0-21swapper/010:49:000
2719199202,10cyclictest11122-21/usr/sbin/munin07:45:250
2719199201,8cyclictest13929-21kworker/0:109:19:060
2719199201,7cyclictest18300-21latency_hist09:15:000
2719199201,12cyclictest0-21swapper/010:15:180
2719199201,11cyclictest0-21swapper/011:38:200
2719199201,11cyclictest0-21swapper/009:50:020
2719199201,10cyclictest2427-21ntp_states09:50:200
212722201,2sleep20-21swapper/211:46:462
2720799196,5cyclictest755-21rngd10:55:202
2720799191,5cyclictest0-21swapper/209:51:412
2719999198,6cyclictest0-21swapper/111:35:151
27199991916,2cyclictest0-21swapper/108:46:061
2719999191,5cyclictest0-21swapper/110:50:131
2719999191,2cyclictest0-21swapper/108:27:041
27199991912,2cyclictest0-21swapper/108:36:531
2719999191,17cyclictest0-21swapper/112:01:521
2719999191,14cyclictest0-21swapper/107:12:271
2719999191,10cyclictest24613-21smart_sda10:40:251
2719999191,10cyclictest24613-21smart_sda10:40:251
2719999190,4cyclictest0-21swapper/107:35:171
2719999190,14cyclictest0-21swapper/109:11:391
2719199198,10cyclictest0-21swapper/008:00:010
2719199196,4cyclictest0-21swapper/012:32:160
2719199196,4cyclictest0-21swapper/010:08:010
2719199195,7cyclictest20162-21kworker/u8:309:23:540
2719199194,8cyclictest0-21swapper/011:45:470
2719199194,5cyclictest0-21swapper/010:59:080
2719199193,4cyclictest12453-40dnf11:26:250
2719199192,9cyclictest808-21snmpd07:12:010
2719199191,9cyclictest5104-21memory11:10:180
2719199191,9cyclictest21460-21kworker/u8:312:23:560
2719199191,6cyclictest0-21swapper/009:36:470
2719199191,3cyclictest0-21swapper/012:03:510
27191991912,4cyclictest18726-21crond10:30:010
2719199191,16cyclictest30352-21munin-run12:10:010
2719199191,15cyclictest8298-21seq11:19:260
2719199191,11cyclictest0-21swapper/011:32:400
2719199191,10cyclictest9201-21/usr/sbin/munin12:35:150
2719199191,10cyclictest0-21swapper/011:01:200
2719199190,6cyclictest0-21swapper/011:24:190
2719199190,11cyclictest0-21swapper/011:05:170
215182190,4sleep230-21rcuc/210:35:152
74212180,1sleep20-21swapper/208:50:132
2720799187,3cyclictest12050irq/123-ahci[0010:05:012
2720799186,4cyclictest15920-21munin-run09:10:012
2720799181,7cyclictest4173-21grep09:55:152
2720799181,7cyclictest0-21swapper/210:30:242
2720799181,7cyclictest0-21swapper/209:21:342
2720799181,6cyclictest0-21swapper/209:28:502
2720799181,5cyclictest15732-21processes10:20:222
27207991815,2cyclictest87350irq/126-enp1s0-07:18:142
2720799181,15cyclictest0-21swapper/207:41:482
2720799181,15cyclictest0-21swapper/207:38:512
2720799181,11cyclictest0-21swapper/208:05:172
2720799181,10cyclictest827-21runrttasks12:37:172
27199991815,2cyclictest0-21swapper/111:48:091
2719999181,16cyclictest0-21swapper/111:22:171
2719999181,16cyclictest0-21swapper/108:20:271
2719999180,2cyclictest0-21swapper/110:10:171
2719999180,10cyclictest0-21swapper/107:15:031
2719199188,3cyclictest8925-21/usr/sbin/munin07:40:250
2719199181,9cyclictest0-21swapper/010:42:280
2719199181,9cyclictest0-21swapper/010:42:280
2719199181,9cyclictest0-21swapper/010:05:020
2719199181,8cyclictest0-21swapper/011:42:160
2719199181,6cyclictest0-21swapper/009:00:230
2719199181,5cyclictest0-21swapper/009:05:240
2719199181,15cyclictest0-21swapper/008:37:170
2719199181,10cyclictest15909-21diskmemload09:40:360
2719199181,10cyclictest0-21swapper/012:10:230
2719199181,10cyclictest0-21swapper/010:30:250
2719199180,6cyclictest0-21swapper/012:28:330
314792171,2sleep20-21swapper/212:10:192
2720799175,3cyclictest13695-21idleruntime11:30:162
2720799173,5cyclictest15909-21diskmemload10:15:132
2720799171,9cyclictest0-21swapper/211:20:122
2720799171,7cyclictest25055-21/usr/sbin/munin11:55:202
2720799171,7cyclictest25055-21/usr/sbin/munin11:55:202
2720799171,7cyclictest1737-21switchtime12:15:252
2720799171,6cyclictest17612-21cat08:00:012
2720799171,6cyclictest0-21swapper/210:45:062
27207991713,2cyclictest24415-21ntp_states10:40:222
27207991713,2cyclictest24415-21ntp_states10:40:222
27207991712,3cyclictest9035-21ntpq10:05:192
27207991712,3cyclictest31975-21ntp_states08:30:212
27207991712,3cyclictest28804-21ntpq10:50:202
27207991712,3cyclictest14655-21threads07:50:232
27207991712,3cyclictest0-21swapper/211:00:352
2720799171,14cyclictest0-21swapper/208:42:362
2720799171,14cyclictest0-21swapper/208:26:022
27207991711,3cyclictest4870-21df07:30:132
2720799171,10cyclictest0-21swapper/211:29:372
2720799171,10cyclictest0-21swapper/210:25:142
2719199177,6cyclictest0-21swapper/007:54:570
2719199176,3cyclictest12050irq/123-ahci[0008:55:020
2719199172,6cyclictest7074-21diskstats07:35:130
2719199172,2cyclictest7705-21kworker/0:207:24:170
2719199171,15cyclictest0-21swapper/010:12:400
2720799161,7cyclictest11122-21/usr/sbin/munin07:45:122
2720799161,6cyclictest15909-21diskmemload09:32:412
2720799161,6cyclictest15909-21diskmemload09:32:412
2720799161,6cyclictest0-21swapper/211:15:342
2720799161,5cyclictest0-21swapper/208:21:392
2720799161,3cyclictest0-21swapper/211:10:182
2720799161,2cyclictest0-21swapper/208:12:082
27207991611,3cyclictest18452-21if_enp1s008:00:172
27207991611,3cyclictest1396-21wc07:20:232
27207991610,3cyclictest15177-21/usr/sbin/munin11:35:182
27207991610,3cyclictest13848-21date09:05:022
2720799160,15cyclictest0-21swapper/207:13:172
2719199166,2cyclictest30907-21/usr/sbin/munin08:30:140
2719199164,3cyclictest19269-21head08:00:260
2719199163,4cyclictest11473-21sshd08:59:450
2719199161,14cyclictest0-21swapper/007:27:460
27191991611,3cyclictest30646-21diskstats07:15:160
27191991611,3cyclictest30333-21seq08:25:420
27191991611,3cyclictest25677-21sed08:15:250
27191991611,3cyclictest21171-21mailstats08:05:220
2719199160,15cyclictest0-21swapper/007:32:210
2720799151,5cyclictest2336-21/usr/sbin/munin07:25:122
27191991510,3cyclictest2943-21/usr/sbin/munin08:40:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional