You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 04:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri May 15, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
195212230163,54sleep30-21swapper/319:06:513
194432226149,24sleep20-21swapper/219:05:502
194302203166,25sleep00-21swapper/019:05:400
195942201165,24sleep10-21swapper/119:07:461
2644721860,4sleep31986699cyclictest21:50:123
19860999390,2cyclictest0-21swapper/200:16:232
19860999281,3cyclictest0-21swapper/200:27:372
286422680,4sleep1808-21snmpd20:40:161
11752560,3sleep00-21swapper/022:05:150
19860993611,3cyclictest7821-21/usr/sbin/munin23:35:222
19866993527,3cyclictest0-21swapper/320:36:393
1986699351,20cyclictest0-21swapper/320:26:343
19860993512,20cyclictest9404-21/usr/sbin/munin22:25:102
1985199358,13cyclictest3547-21munin-run19:45:020
136422350,5sleep31986699cyclictest21:20:153
19860993412,19cyclictest14760-21/usr/sbin/munin20:10:152
1986099341,19cyclictest0-21swapper/220:26:532
19860993411,20cyclictest7203-21/usr/sbin/munin22:20:212
19860993411,20cyclictest27339-21/usr/sbin/munin19:25:202
19860993410,5cyclictest0-21swapper/222:15:192
1986699331,22cyclictest0-21swapper/321:46:183
1986699331,21cyclictest0-21swapper/320:14:543
1986099339,4cyclictest15719-21/usr/sbin/munin23:50:252
1986099331,5cyclictest808-21snmpd21:35:252
19860993311,3cyclictest12568-21/usr/sbin/munin20:05:262
19860993310,3cyclictest5621-21/usr/sbin/munin23:30:262
19866993214,8cyclictest0-21swapper/319:52:303
1986699321,21cyclictest0-21swapper/320:18:143
1986699321,21cyclictest0-21swapper/320:05:423
1986699321,19cyclictest0-21swapper/300:13:473
1986699321,14cyclictest0-21swapper/322:27:233
1986699321,13cyclictest0-21swapper/320:55:233
1986699321,11cyclictest9136-21diskstats21:10:153
1986099326,4cyclictest10865-21/usr/sbin/munin23:40:192
1986099321,6cyclictest4584-21sshd22:12:372
19860993212,17cyclictest21036-21/usr/sbin/munin00:05:232
1986099321,14cyclictest0-21swapper/221:15:132
1986099321,11cyclictest10337-21unixbench_singl21:10:272
19860993211,18cyclictest1868-21/usr/sbin/munin00:35:182
19860993210,3cyclictest1207-21/usr/sbin/munin23:20:252
19860993210,2cyclictest31442-21hddtemp_smartct22:00:162
1986699319,9cyclictest0-21swapper/319:25:313
19866993117,4cyclictest0-21swapper/320:04:103
19866993116,10cyclictest0-21swapper/321:19:153
1986699311,24cyclictest0-21swapper/320:49:593
1986699311,14cyclictest0-21swapper/300:27:483
1986699311,12cyclictest0-21swapper/323:50:063
1986699311,12cyclictest0-21swapper/322:24:073
1986699311,12cyclictest0-21swapper/321:28:553
1986699311,12cyclictest0-21swapper/321:28:553
1986099312,10cyclictest20874-1kworker/2:2H00:03:332
1986099311,6cyclictest0-21swapper/223:15:112
1986099311,13cyclictest0-21swapper/221:49:452
19860993110,3cyclictest2805-21grep20:55:162
19860993110,3cyclictest22156-21cstates21:40:112
19860993110,3cyclictest15377-21/usr/sbin/munin21:25:182
19860993110,3cyclictest15377-21/usr/sbin/munin21:25:182
1985199318,11cyclictest32767-21perf22:05:020
1985199313,13cyclictest4796-21awk22:15:000
1986699308,14cyclictest0-21swapper/321:55:303
19866993017,3cyclictest0-21swapper/320:23:423
1986699301,26cyclictest18870-21hddtemp_smartct22:45:173
1986699301,18cyclictest0-21swapper/323:08:593
1986699301,13cyclictest0-21swapper/323:49:063
1986699301,13cyclictest0-21swapper/323:36:343
1986699301,11cyclictest0-21swapper/320:34:123
1986099308,3cyclictest10359-21/usr/sbin/munin20:00:212
1986099308,2cyclictest4270-21latency23:25:192
1986099308,2cyclictest16957-21/usr/sbin/munin20:15:112
1986099306,3cyclictest18837-21smart_sda21:30:242
1986099305,4cyclictest0-21swapper/223:45:262
1986099303,5cyclictest30796-21smart_sda19:30:222
1986099303,2cyclictest87550irq/128-enp1s0-00:30:192
1986099302,17cyclictest4527-1kworker/2:1H23:01:412
1986099301,6cyclictest18651-21cat00:00:012
1986099301,11cyclictest0-21swapper/219:57:332
19860993011,16cyclictest1620-21cpu19:40:112
1986099300,4cyclictest0-21swapper/222:30:122
1986099300,20cyclictest0-21swapper/220:51:532
19866992918,3cyclictest0-21swapper/319:14:073
19866992916,3cyclictest0-21swapper/320:41:593
1986699291,19cyclictest0-21swapper/319:39:113
1986699291,13cyclictest0-21swapper/323:03:393
1986699291,12cyclictest0-21swapper/300:19:313
1986699291,11cyclictest0-21swapper/322:01:183
1986099299,3cyclictest13470-21expr21:20:132
1986099298,3cyclictest21166-21if_err_enp1s022:50:182
1986099298,2cyclictest23558-21/usr/sbin/munin20:30:202
1986099296,4cyclictest30460-21/usr/sbin/munin23:10:262
1986099296,4cyclictest28868-21grep00:20:232
19860992913,3cyclictest18046-21latency_hist22:45:012
19860992913,3cyclictest18046-21latency_hist22:45:012
1986099291,2cyclictest0-21swapper/220:35:182
19860992911,3cyclictest6533-21cut21:05:012
1986099291,13cyclictest0-21swapper/222:05:372
1986099291,12cyclictest29829-21kworker/u8:119:48:582
1986099291,12cyclictest0-21swapper/220:48:052
19860992911,15cyclictest25048-21cut19:20:022
19860992910,3cyclictest29096-21grep20:40:222
1985199291,13cyclictest19546-21munin-run21:35:010
1985199291,13cyclictest11991-21munin-run23:45:010
312432280,4sleep121-21rcuc/122:00:151
1986699289,3cyclictest0-21swapper/323:24:393
19866992818,3cyclictest0-21swapper/319:34:153
1986699281,13cyclictest0-21swapper/322:39:193
1986699281,13cyclictest0-21swapper/300:08:073
1986699281,12cyclictest0-21swapper/323:29:503
1986699281,12cyclictest0-21swapper/322:57:073
1986699281,12cyclictest0-21swapper/322:11:473
1986699280,14cyclictest0-21swapper/300:01:263
1986099289,3cyclictest14271-21grep22:35:162
1986099288,3cyclictest21456-21cat19:10:242
1986099288,3cyclictest19927-21cat20:20:162
1986099286,3cyclictest0-21swapper/223:05:122
1986099284,5cyclictest0-21swapper/222:55:162
1986099281,6cyclictest8383-21sshd21:07:542
1986099281,5cyclictest27090-21sshd19:24:542
19860992812,13cyclictest29904-21threads21:55:242
19860992810,3cyclictest32303-21cut19:35:142
1985199283,12cyclictest23021-21munin-run00:10:000
1986699271,13cyclictest0-21swapper/323:44:463
1986699271,12cyclictest0-21swapper/321:44:473
1986699271,11cyclictest0-21swapper/319:17:013
1986099278,3cyclictest24433-21processes00:10:232
1986099278,2cyclictest6385-21fschecks_count19:50:142
1986099271,6cyclictest27288-21memory21:50:202
1985199270,12cyclictest27126-21awk19:25:010
1986699261,9cyclictest0-21swapper/321:39:213
19866992615,3cyclictest0-21swapper/321:00:513
19866992614,3cyclictest0-21swapper/319:49:593
1986699261,12cyclictest0-21swapper/322:32:553
1986699261,11cyclictest0-21swapper/323:59:263
1986699261,11cyclictest0-21swapper/323:17:113
1986699261,11cyclictest0-21swapper/320:52:353
1986699261,10cyclictest0-21swapper/322:16:473
1986699260,16cyclictest0-21swapper/319:59:203
1986099261,6cyclictest19420-21rt-features22:45:252
1986699259,9cyclictest0-21swapper/300:31:033
1986699251,23cyclictest0-21swapper/321:06:393
1986699251,13cyclictest0-21swapper/319:22:353
1986699251,12cyclictest0-21swapper/323:33:543
1986699251,12cyclictest0-21swapper/300:37:033
1986699251,11cyclictest0-21swapper/322:50:533
1986699251,11cyclictest0-21swapper/322:44:193
1986699251,11cyclictest0-21swapper/322:44:193
1986699251,10cyclictest0-21swapper/321:32:553
143862250,4sleep011-21rcuc/023:50:020
298082240,3sleep11985699cyclictest00:25:021
1986699241,14cyclictest0-21swapper/319:42:113
1986699241,11cyclictest0-21swapper/323:12:553
1985699245,14cyclictest30363-40dnf21:57:571
1985699242,16cyclictest0-21swapper/122:34:091
1985199241,11cyclictest21752cstates00:35:140
1986699231,11cyclictest0-21swapper/322:09:113
1985699235,5cyclictest0-21swapper/121:50:161
1985699235,11cyclictest87550irq/128-enp1s0-23:40:241
19856992316,4cyclictest7657-21ntp_states21:05:201
19856992315,3cyclictest0-21swapper/122:08:181
1985699231,20cyclictest0-21swapper/122:46:221
1985699231,16cyclictest0-21swapper/122:57:061
1985699231,16cyclictest0-21swapper/122:18:031
1985699231,16cyclictest0-21swapper/100:18:491
1985699231,12cyclictest31510-21munin-run19:35:021
1985699231,12cyclictest0-21swapper/123:27:281
1985699230,16cyclictest0-21swapper/121:41:031
1985699230,15cyclictest0-21swapper/122:20:191
1985199232,8cyclictest18804-21kworker/0:100:23:180
1985199232,8cyclictest10090-21kworker/0:320:11:550
1986699225,8cyclictest0-21swapper/300:21:273
1985699228,4cyclictest0-21swapper/122:12:461
1985699222,15cyclictest0-21swapper/120:54:251
19856992217,3cyclictest12607-21taskset21:15:441
19856992214,7cyclictest0-21swapper/100:27:491
1985699221,3cyclictest25696-21perl19:20:131
1985699221,18cyclictest22105-21ntp_states00:05:241
1985699221,16cyclictest0-21swapper/123:54:531
1985699221,16cyclictest0-21swapper/121:49:301
1985699221,10cyclictest0-21swapper/122:38:001
19856992210,3cyclictest31213-21ntp_states20:45:211
1985699220,20cyclictest0-21swapper/119:13:191
1985199222,8cyclictest26572-21kworker/0:123:19:290
1985199221,6cyclictest0-21swapper/023:39:220
85852210,1sleep00-21swapper/021:10:010
1985699218,3cyclictest0-21swapper/121:20:521
19856992117,3cyclictest0-21swapper/121:27:511
19856992117,3cyclictest0-21swapper/121:27:511
19856992114,2cyclictest0-21swapper/121:14:041
1985699211,15cyclictest0-21swapper/121:32:491
1985699211,15cyclictest0-21swapper/100:35:201
1985699211,11cyclictest16949-21df23:55:141
1985699211,11cyclictest0-21swapper/122:41:251
1985699211,11cyclictest0-21swapper/122:41:251
1985699211,11cyclictest0-21swapper/121:37:301
1985699211,10cyclictest0-21swapper/123:12:291
1985699211,10cyclictest0-21swapper/100:30:291
1985699210,4cyclictest8904-21kworker/u8:023:19:351
1985699210,14cyclictest0-21swapper/123:08:551
1985199215,14cyclictest26789-21perf23:05:010
1985199211,9cyclictest28982-21kworker/0:319:29:350
1985199211,7cyclictest13248-21kworker/0:022:42:140
1985199211,7cyclictest13248-21kworker/0:022:42:140
1985199211,11cyclictest13800-21/usr/sbin/munin22:35:120
1985199210,12cyclictest27148-1kworker/0:0H19:40:020
146482210,5sleep07-21ksoftirqd/020:10:010
1985699209,9cyclictest0-21swapper/119:35:261
1985699209,3cyclictest9871-21cat23:40:011
1985699209,3cyclictest2533-21diskstats20:55:141
1985699209,3cyclictest0-21swapper/122:30:011
1985699202,14cyclictest0-21swapper/123:31:311
1985699201,14cyclictest0-21swapper/123:49:391
1985699201,10cyclictest0-21swapper/120:25:141
1985199208,2cyclictest10473-21kworker/0:221:19:110
1985199202,10cyclictest7817-21kworker/0:119:55:470
1985199201,7cyclictest0-21swapper/021:13:130
1985199201,6cyclictest0-21swapper/021:56:040
1985199201,5cyclictest20233-1kworker/u9:122:18:390
1985199201,11cyclictest5621-21/usr/sbin/munin23:30:280
1985199201,11cyclictest25914-21kworker/0:021:49:270
1985199201,10cyclictest1207-21/usr/sbin/munin23:20:260
1985199200,15cyclictest4104-21kworker/0:021:00:240
1985699195,5cyclictest8162-21/usr/sbin/munin19:55:161
19856991915,3cyclictest0-21swapper/120:00:531
19856991915,3cyclictest0-21swapper/119:19:221
1985699191,17cyclictest0-21swapper/121:02:151
1985699191,17cyclictest0-21swapper/120:39:071
1985699191,17cyclictest0-21swapper/119:42:341
1985699191,17cyclictest0-21swapper/119:25:241
1985699191,16cyclictest0-21swapper/120:15:101
1985699191,16cyclictest0-21swapper/119:50:091
1985699191,10cyclictest23590-21diskstats00:10:151
1985699191,10cyclictest20981-21gltestperf22:50:151
1985699191,10cyclictest19001-21expr00:00:111
1985699191,10cyclictest16350-21users20:10:251
1985699191,10cyclictest14531-21latency_hist20:10:011
1985699191,10cyclictest0-21swapper/123:21:411
1985699190,9cyclictest6627-21kworker/u8:223:01:361
1985199198,3cyclictest25039-21sshd00:12:530
1985199195,2cyclictest24221-21kworker/0:023:00:010
1985199192,8cyclictest2492-21kworker/0:119:15:220
1985199192,15cyclictest27741-21idleruntime-cro20:40:000
1985199191,8cyclictest7817-21kworker/0:120:03:150
1985199191,8cyclictest15021-21kworker/0:121:24:350
1985199191,4cyclictest0-21swapper/023:26:170
19851991913,3cyclictest21188-21ls19:10:220
1985199191,10cyclictest29533-21/usr/sbin/munin19:30:230
19856991815,2cyclictest0-21swapper/120:20:091
1985699181,3cyclictest0-21swapper/119:46:471
1985699181,16cyclictest0-21swapper/120:32:181
1985199189,2cyclictest29832-21/usr/sbin/munin00:25:250
1985199188,5cyclictest7-21ksoftirqd/022:54:560
1985199183,2cyclictest26572-21kworker/0:123:08:310
1985199182,5cyclictest20233-1kworker/u9:121:36:550
19851991813,3cyclictest11755-21grep22:30:120
1985199181,10cyclictest0-21swapper/021:51:130
1985199180,9cyclictest13967-21kworker/0:323:59:150
1985199176,3cyclictest19154-21/usr/sbin/munin20:20:180
1985199171,7cyclictest30148-21/usr/sbin/munin20:45:230
1985199171,6cyclictest0-21swapper/020:25:130
1985199171,3cyclictest18192-21/usr/sbin/munin22:45:240
19851991712,3cyclictest23568-21vmstat21:40:260
19851991712,3cyclictest15377-21/usr/sbin/munin21:25:180
19851991712,3cyclictest15377-21/usr/sbin/munin21:25:180
19851991711,3cyclictest412-21grep20:50:140
1985199164,4cyclictest8563-21diskmemload00:00:250
1985199161,6cyclictest808-21snmpd19:50:060
1985199161,6cyclictest7203-21/usr/sbin/munin22:20:180
1985199161,6cyclictest14416-21/usr/sbin/munin23:50:130
1985199161,6cyclictest0-21swapper/023:11:190
1985199161,6cyclictest0-21swapper/000:30:300
1985199161,4cyclictest0-21swapper/000:16:270
19851991612,3cyclictest0-21swapper/020:55:130
19851991612,3cyclictest0-21swapper/020:33:030
19851991611,3cyclictest9489-21cpuspeed_turbos22:25:090
19851991611,3cyclictest4009-21ls19:45:130
1985199153,9cyclictest28597-21gltestperf20:40:150
1985199151,5cyclictest808-21snmpd20:15:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional