You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-30 - 22:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat May 30, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27122236162,23sleep20-21swapper/207:06:472
28212203170,22sleep30-21swapper/307:08:123
27112202166,24sleep10-21swapper/107:06:461
26472199163,25sleep00-21swapper/007:05:550
3239621420,4sleep0304499cyclictest09:25:250
295042600,4sleep3306199cyclictest08:05:273
304499346,13cyclictest1150-21munin-run10:45:010
305099330,25cyclictest0-21swapper/111:54:161
305099321,22cyclictest0-21swapper/110:53:531
305099310,24cyclictest0-21swapper/109:35:291
304499314,12cyclictest4341-21idleruntime-cro08:25:010
305099301,17cyclictest0-21swapper/109:58:401
305099300,11cyclictest0-21swapper/110:59:291
305099291,12cyclictest0-21swapper/108:24:291
305099290,17cyclictest0-21swapper/111:37:211
305099290,12cyclictest0-21swapper/107:21:091
305099281,9cyclictest0-21swapper/107:50:401
305099281,9cyclictest0-21swapper/107:13:251
305099281,11cyclictest0-21swapper/111:01:011
305099281,11cyclictest0-21swapper/108:04:521
305099281,10cyclictest0-21swapper/109:10:051
305099281,10cyclictest0-21swapper/108:29:571
305099280,12cyclictest0-21swapper/110:32:291
304499284,12cyclictest21933-21awk09:05:010
304499283,12cyclictest3385-21perf10:50:010
304499281,13cyclictest2131-21munin-run08:20:020
305099271,16cyclictest0-21swapper/107:30:411
305099271,10cyclictest0-21swapper/109:52:081
305099270,18cyclictest0-21swapper/108:33:131
305099270,12cyclictest0-21swapper/109:20:251
305099270,12cyclictest0-21swapper/108:49:411
305099270,12cyclictest0-21swapper/108:49:411
305099270,12cyclictest0-21swapper/108:43:291
305099270,10cyclictest0-21swapper/111:47:001
304499278,8cyclictest3045-21/usr/sbin/munin12:00:190
306199262,12cyclictest25158-21irqstats09:10:193
306199261,9cyclictest0-21swapper/309:59:513
306199261,10cyclictest0-21swapper/312:21:593
306199261,10cyclictest0-21swapper/308:30:033
305099261,9cyclictest0-21swapper/112:31:531
305099261,9cyclictest0-21swapper/110:45:411
305099261,9cyclictest0-21swapper/108:38:271
305099261,9cyclictest0-21swapper/108:11:411
305099261,16cyclictest0-21swapper/108:59:531
305099261,11cyclictest7326-21sort09:45:001
305099261,10cyclictest0-21swapper/110:04:161
305099261,10cyclictest0-21swapper/108:53:291
305099260,9cyclictest0-21swapper/107:40:571
305099260,8cyclictest0-21swapper/107:25:571
305099260,11cyclictest0-21swapper/109:03:091
304499264,14cyclictest13925-21/usr/sbin/munin11:10:240
3044992616,6cyclictest13737-40dnf10:00:030
3044992615,3cyclictest18433-21cpuspeed_turbos10:10:130
304499260,12cyclictest5232-21idleruntime-cro07:15:010
306199251,9cyclictest0-21swapper/307:57:393
3061992512,6cyclictest0-21swapper/309:45:533
306199251,11cyclictest0-21swapper/307:38:553
306199251,10cyclictest0-21swapper/309:07:273
306199251,10cyclictest0-21swapper/307:24:593
306199250,9cyclictest0-21swapper/311:08:553
306199250,9cyclictest0-21swapper/311:08:553
306199250,10cyclictest0-21swapper/312:25:353
305799252,5cyclictest0-21swapper/209:52:512
305099251,9cyclictest0-21swapper/110:35:491
305099251,9cyclictest0-21swapper/107:38:291
305099251,7cyclictest0-21swapper/110:22:091
305099251,12cyclictest0-21swapper/111:05:451
305099251,12cyclictest0-21swapper/111:05:451
305099251,10cyclictest0-21swapper/109:26:531
305099250,9cyclictest0-21swapper/112:20:051
305099250,9cyclictest0-21swapper/110:25:171
304499258,8cyclictest0-21swapper/009:35:260
304499255,5cyclictest87350irq/126-enp1s0-09:25:020
304499255,10cyclictest30315-21/usr/sbin/munin08:10:170
304499252,6cyclictest87350irq/126-enp1s0-10:29:260
304499251,12cyclictest14697-21munin-run07:35:000
306199249,3cyclictest0-21swapper/309:16:393
306199247,3cyclictest0-21swapper/307:40:593
306199246,3cyclictest0-21swapper/310:21:513
306199241,4cyclictest0-21swapper/311:12:223
306199241,11cyclictest0-21swapper/310:03:593
306199241,11cyclictest0-21swapper/308:16:393
306199241,10cyclictest0-21swapper/308:55:513
306199241,10cyclictest0-21swapper/308:13:273
306199240,12cyclictest0-21swapper/311:53:433
306199240,12cyclictest0-21swapper/310:06:553
305099247,6cyclictest0-21swapper/107:55:081
305099246,7cyclictest0-21swapper/112:00:091
305099241,8cyclictest0-21swapper/109:06:371
305099241,11cyclictest0-21swapper/112:36:531
305099241,11cyclictest0-21swapper/110:43:451
305099241,10cyclictest0-21swapper/107:16:391
305099240,8cyclictest0-21swapper/112:07:171
305099240,15cyclictest0-21swapper/111:29:531
305099240,11cyclictest0-21swapper/112:28:291
304499249,7cyclictest7717-21diskstats09:45:150
304499248,8cyclictest24656-21/usr/sbin/munin11:35:250
304499243,12cyclictest0-21swapper/010:57:400
304499241,11cyclictest19106-21awk07:45:010
163382241,19sleep216511-21df_inode10:05:142
306199239,3cyclictest30171-21irqstats10:35:193
306199239,3cyclictest0-21swapper/312:38:423
306199238,3cyclictest0-21swapper/312:04:393
306199231,9cyclictest0-21swapper/309:33:353
306199231,7cyclictest0-21swapper/307:31:313
306199230,9cyclictest0-21swapper/309:51:033
306199230,10cyclictest0-21swapper/312:14:153
305099238,3cyclictest28463-21df_inode08:05:151
305099237,10cyclictest0-21swapper/109:31:251
305099234,3cyclictest0-21swapper/111:15:481
305099231,8cyclictest20380-21ntp_states07:45:201
305099231,4cyclictest26881-21df09:15:151
305099231,3cyclictest9342-21munin-run09:50:001
305099231,16cyclictest0-21swapper/112:15:211
305099231,12cyclictest0-21swapper/111:55:531
305099231,11cyclictest0-21swapper/110:05:561
305099231,10cyclictest0-21swapper/112:12:211
304499239,2cyclictest8606-21kworker/0:008:35:530
304499236,4cyclictest11533-21sh09:55:010
3044992311,3cyclictest10251-21kworker/0:312:30:010
304499231,10cyclictest12864-21sed12:25:010
3044992310,5cyclictest22869-21sshd11:31:330
306199229,3cyclictest0-21swapper/311:40:113
306199229,2cyclictest0-21swapper/311:34:153
306199228,3cyclictest0-21swapper/307:52:233
306199227,3cyclictest0-21swapper/311:55:393
306199224,4cyclictest7756-21sh11:00:003
306199221,9cyclictest0-21swapper/308:43:553
306199221,5cyclictest0-21swapper/312:17:053
306199221,10cyclictest0-21swapper/308:25:473
306199220,5cyclictest0-21swapper/311:25:593
306199220,11cyclictest0-21swapper/310:15:553
306199220,10cyclictest0-21swapper/311:20:113
305799222,4cyclictest4173-21cat12:05:012
305099228,3cyclictest0-21swapper/108:15:591
305099228,2cyclictest0-21swapper/110:18:091
305099221,7cyclictest0-21swapper/110:13:571
305099221,18cyclictest25908-21df11:40:131
304499229,2cyclictest8118-21perf07:20:010
304499226,13cyclictest808-21snmpd12:13:310
304499225,6cyclictest9028-21fschecks_count12:15:140
304499225,5cyclictest13112-21latency_hist08:45:020
304499222,7cyclictest24107-21crond09:10:000
304499221,9cyclictest16677-21cat11:20:010
304499221,5cyclictest0-21swapper/011:42:170
306199218,3cyclictest0-21swapper/310:50:583
306199218,2cyclictest0-21swapper/310:32:313
306199217,3cyclictest0-21swapper/310:11:353
306199217,3cyclictest0-21swapper/308:48:583
306199217,3cyclictest0-21swapper/308:48:583
306199216,3cyclictest0-21swapper/311:15:033
306199212,4cyclictest0-21swapper/311:01:213
306199211,11cyclictest0-21swapper/311:49:103
306199211,10cyclictest0-21swapper/311:39:113
306199211,10cyclictest0-21swapper/308:50:033
306199210,9cyclictest0-21swapper/310:46:143
305799219,3cyclictest0-21swapper/207:54:172
305799216,7cyclictest0-21swapper/209:40:362
305799210,11cyclictest0-21swapper/211:20:592
305099211,1cyclictest0-21swapper/111:14:411
304499218,3cyclictest23716-21/usr/sbin/munin07:55:150
304499216,7cyclictest18245-1kworker/u9:211:58:050
304499214,4cyclictest87550irq/128-enp1s0-09:10:170
304499213,15cyclictest2757-21basename09:35:010
304499212,13cyclictest0-21swapper/009:59:200
304499211,4cyclictest0-21swapper/012:08:370
3044992113,5cyclictest28417-21smart_sda10:30:210
304499211,12cyclictest0-21swapper/012:31:030
304499211,11cyclictest17398-21/usr/sbin/munin12:35:120
306199208,3cyclictest0-21swapper/307:10:113
306199206,2cyclictest0-21swapper/308:35:393
306199201,4cyclictest0-21swapper/309:42:433
306199201,10cyclictest0-21swapper/307:25:513
306199200,10cyclictest0-21swapper/309:21:033
305799205,6cyclictest755-21rngd08:28:132
305799201,8cyclictest0-21swapper/211:55:012
305799200,17cyclictest7759-21sadc11:00:012
305099207,3cyclictest0-21swapper/111:31:271
305099201,9cyclictest0-21swapper/111:21:431
304499205,11cyclictest16974-21grep08:50:260
304499204,6cyclictest87350irq/126-enp1s0-09:17:320
304499204,13cyclictest20359-21tr07:45:200
304499203,5cyclictest0-21swapper/010:50:250
304499201,7cyclictest27773-21/usr/sbin/munin11:45:230
304499201,7cyclictest20566-21/usr/sbin/munin10:15:210
3044992013,5cyclictest15787-21latency07:35:170
304499201,12cyclictest0-21swapper/010:24:140
304499201,12cyclictest0-21swapper/009:40:120
304499200,17cyclictest8335-21/usr/sbin/munin07:20:130
306199199,2cyclictest19318-21/usr/sbin/munin07:45:173
306199198,4cyclictest839-21lldpd08:21:483
306199194,4cyclictest0-21swapper/310:40:453
306199191,9cyclictest0-21swapper/312:07:593
306199191,15cyclictest5601-21munin-node07:15:013
306199191,11cyclictest0-21swapper/309:25:143
305799197,9cyclictest30-21rcuc/207:20:162
305799194,6cyclictest0-21swapper/212:37:332
305799194,5cyclictest30-21rcuc/210:50:132
305799191,8cyclictest28708-21cat09:20:022
3057991915,3cyclictest32-21ksoftirqd/207:35:242
305799191,15cyclictest12436-21hddtemp_smartct09:55:162
305799191,11cyclictest0-21swapper/211:38:402
305799191,11cyclictest0-21swapper/209:20:102
305799191,10cyclictest0-21swapper/209:35:152
305799191,10cyclictest0-21swapper/209:32:272
304499196,5cyclictest0-21swapper/010:35:180
304499195,11cyclictest8518-21fschecks_time11:00:150
304499191,9cyclictest0-21swapper/011:21:310
304499191,5cyclictest5768-21processes08:25:220
304499191,10cyclictest17734-21/usr/sbin/munin08:55:150
3044991910,6cyclictest8014-21/usr/sbin/munin08:30:230
304499190,16cyclictest18982-21/usr/sbin/munin11:25:120
58992181,2sleep20-21swapper/212:06:262
306199183,7cyclictest3159-21cstates09:35:143
306199181,5cyclictest0-21swapper/310:26:273
305799188,2cyclictest32-21ksoftirqd/210:15:462
305799182,13cyclictest31553-21find08:10:242
305799181,6cyclictest29741-21fschecks_count10:35:162
305799181,4cyclictest18245-1kworker/u9:211:11:222
305799181,4cyclictest0-21swapper/211:56:022
305799181,15cyclictest0-21swapper/208:24:272
305799181,11cyclictest0-21swapper/212:30:392
305799181,10cyclictest0-21swapper/211:15:262
304499189,6cyclictest30321-21diskstats11:50:150
304499189,6cyclictest22585-21ntp_states07:50:240
304499185,10cyclictest27905-21awk08:05:010
304499184,11cyclictest16643-21fschecks_time10:05:150
304499181,5cyclictest0-21swapper/007:28:360
3044991810,5cyclictest11072-21latency11:05:200
3044991810,5cyclictest11072-21latency11:05:200
229412181,2sleep20-21swapper/211:32:312
169612180,2sleep30-21swapper/312:32:263
3061991712,3cyclictest19937-21/usr/sbin/munin09:00:193
305799176,3cyclictest87350irq/126-enp1s0-09:25:162
305799175,4cyclictest27218-21egrep10:30:102
305799171,9cyclictest0-21swapper/211:05:272
305799171,9cyclictest0-21swapper/211:05:272
305799171,7cyclictest4627-21timerandwakeup07:10:252
305799171,7cyclictest22426-21ls09:05:132
305799171,6cyclictest790-21in:imjournal08:05:022
305799171,5cyclictest32-21ksoftirqd/209:05:012
305799171,4cyclictest0-21swapper/211:44:052
305799171,3cyclictest0-21swapper/208:52:072
3057991712,3cyclictest5835-21cpuspeed_turbos07:15:032
3057991712,3cyclictest32-21ksoftirqd/209:14:142
305799171,15cyclictest0-21swapper/212:29:402
305799171,15cyclictest0-21swapper/212:21:042
305799171,14cyclictest0-21swapper/208:47:412
305799171,14cyclictest0-21swapper/208:47:412
3057991711,3cyclictest827-21runrttasks08:40:032
3057991711,3cyclictest1-21systemd10:45:332
305799171,13cyclictest0-21swapper/211:25:292
305799171,10cyclictest0-21swapper/212:18:092
305799171,10cyclictest0-21swapper/210:40:162
304499171,4cyclictest87350irq/126-enp1s0-08:45:120
304499171,4cyclictest87350irq/126-enp1s0-08:45:120
3044991712,3cyclictest28123-21/usr/sbin/munin08:05:190
203102170,2sleep20-21swapper/210:14:492
3061991611,3cyclictest26499-21gltestperf08:00:163
305799161,9cyclictest0-21swapper/211:47:512
305799161,6cyclictest14341-21sshd07:30:332
305799161,6cyclictest0-21swapper/212:11:062
305799161,6cyclictest0-21swapper/210:20:132
305799161,4cyclictest19651-21df07:45:132
305799161,2cyclictest0-21swapper/208:35:412
3057991612,3cyclictest0-21swapper/210:01:112
3057991612,3cyclictest0-21swapper/208:17:402
3057991612,2cyclictest30-21rcuc/208:00:012
305799161,14cyclictest0-21swapper/210:26:372
3057991611,3cyclictest8807-21cut11:00:182
305799161,12cyclictest6921-21expr08:30:112
3057991610,3cyclictest17117-21/usr/sbin/munin07:40:222
305799151,7cyclictest0-21swapper/208:08:382
305799151,7cyclictest0-21swapper/207:28:332
305799151,5cyclictest8046-21hddtemp_smartct09:45:172
305799151,5cyclictest19363-21sshd08:55:442
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional