You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 12:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 15, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
141432239163,24sleep30-21swapper/319:08:503
140502205166,26sleep20-21swapper/219:07:362
141152203166,26sleep00-21swapper/019:08:260
122712200166,23sleep10-21swapper/119:05:051
2981921470,3sleep11432799cyclictest20:55:181
14332998683,2cyclictest0-21swapper/221:40:482
142432650,1sleep20-21swapper/221:35:122
14327994518,18cyclictest22838-21/usr/sbin/munin19:25:201
1432799419,13cyclictest13392-21latency_hist20:20:021
14327993815,12cyclictest25636-21awk23:15:011
14327993814,20cyclictest4548-21taskset23:40:121
14327993813,13cyclictest14630-21sh22:50:011
5642370,1sleep00-21swapper/019:50:110
1432799377,15cyclictest0-21swapper/123:18:131
1432799376,23cyclictest0-21swapper/122:25:161
1432099378,13cyclictest23445-21idleruntime-cro23:10:010
1432799369,12cyclictest31729-21cat22:15:011
1432799367,21cyclictest0-21swapper/123:30:011
14327993616,12cyclictest14922-21hddtemp_smartct21:35:161
14327993616,11cyclictest21814-21diskstats23:05:141
14327993613,19cyclictest8601-21grep20:05:251
14327993613,12cyclictest10251-21awk22:40:011
14327993518,13cyclictest7331-21/usr/sbin/munin22:30:241
14327993516,11cyclictest25675-21/usr/sbin/munin20:45:221
14327993513,13cyclictest22819-21awk21:55:011
14327993510,22cyclictest5667-21kernelversion20:00:181
1432799347,17cyclictest0-21swapper/100:00:171
14327993414,6cyclictest8980-21grep23:50:101
14327993413,18cyclictest13051-21munin-run00:00:021
1433999331,25cyclictest0-21swapper/319:50:043
1432799338,12cyclictest0-21swapper/122:20:281
14327993315,4cyclictest11276-21/usr/sbin/munin22:40:181
14327993314,16cyclictest19667-21turbostat.cron00:15:011
14327993313,17cyclictest28405-21diskstats23:20:151
14327993313,17cyclictest28405-21diskstats23:20:151
14327993310,20cyclictest17759-21df_inode19:15:151
14327993310,13cyclictest20157-21cat20:35:011
1433999321,23cyclictest0-21swapper/319:55:403
1433999321,20cyclictest0-21swapper/319:14:063
1432799329,14cyclictest27850-21chrt20:50:201
1432799329,14cyclictest23271-21ps00:20:231
1432799329,13cyclictest0-21swapper/119:36:291
1432799329,12cyclictest20223-21/usr/sbin/munin20:35:261
1432799328,12cyclictest28469-21perf00:35:011
1432799328,12cyclictest18408-21latency_hist21:45:011
1432799327,13cyclictest29870-21processes00:35:211
1432799326,11cyclictest0-21swapper/121:45:281
1432799323,20cyclictest0-21swapper/122:56:491
14327993216,4cyclictest6516-21ntp_states21:15:231
14327993212,17cyclictest14482-21latency20:20:181
14327993212,11cyclictest3784-21fschecks_time21:10:161
14327993211,7cyclictest19258-21/usr/sbin/munin23:00:121
14327993210,19cyclictest9624-21munin-run21:25:011
14339993115,3cyclictest0-21swapper/319:37:043
1433999311,24cyclictest0-21swapper/320:11:463
1433999311,21cyclictest0-21swapper/323:57:363
1432799316,15cyclictest27655-21cstates22:05:121
14327993113,5cyclictest15828-21/usr/sbin/munin20:25:151
14327993113,15cyclictest2946-21cat19:55:121
14327993112,5cyclictest25142-21pmu-power19:30:221
14327993112,11cyclictest1615-1kworker/u9:121:28:301
14327993111,11cyclictest16477-21needreboot00:05:201
14327993110,9cyclictest579-21systemd-journal19:55:011
14327993110,6cyclictest87350irq/126-enp1s0-22:05:011
1433999309,8cyclictest0-21swapper/321:21:123
1433999308,9cyclictest0-21swapper/319:29:203
1433999301,18cyclictest0-21swapper/300:05:123
1433999301,13cyclictest0-21swapper/321:17:243
1433999301,13cyclictest0-21swapper/320:45:043
1433999301,13cyclictest0-21swapper/319:19:323
1433999301,10cyclictest0-21swapper/321:49:283
14339993010,8cyclictest0-21swapper/323:06:243
1433999300,22cyclictest0-21swapper/321:28:003
1432799306,22cyclictest0-21swapper/123:30:151
14327993020,6cyclictest1205-40dnf22:17:261
14327993015,12cyclictest3353-21ntp_states23:35:201
14327993013,14cyclictest30595-21/usr/sbin/munin19:45:231
14327993011,11cyclictest327-21users21:00:271
14327993010,5cyclictest738-21dump-pmu-power21:09:561
1432099307,11cyclictest16848-21awk22:55:020
1432099306,11cyclictest7-21ksoftirqd/000:25:010
1433999291,20cyclictest0-21swapper/322:39:083
1433999291,19cyclictest0-21swapper/322:45:443
1433999291,14cyclictest0-21swapper/320:30:083
14339992910,7cyclictest0-21swapper/322:33:243
1433299291,3cyclictest0-21swapper/222:55:222
1432799299,11cyclictest14880-21bc22:50:121
1432799298,9cyclictest12048-21/usr/sbin/munin21:30:141
1432799298,19cyclictest0-21swapper/100:30:011
1432799298,19cyclictest0-21swapper/100:30:011
14327992912,9cyclictest28737-21df19:40:141
14327992912,14cyclictest20730-21pmu-power19:20:221
1433999289,3cyclictest0-21swapper/323:10:403
1433999287,8cyclictest0-21swapper/323:45:523
1433999287,3cyclictest0-21swapper/323:01:563
1433999281,17cyclictest0-21swapper/320:36:023
1433999280,21cyclictest0-21swapper/300:22:203
1432799289,7cyclictest14524-21/usr/sbin/munin19:10:201
1432799285,9cyclictest0-21swapper/121:55:171
14327992811,10cyclictest3028-21diskmemload00:16:101
76442271,3sleep27709-21/usr/sbin/munin21:20:122
1433999277,3cyclictest0-21swapper/320:00:503
1433999271,18cyclictest0-21swapper/321:42:103
1433999271,18cyclictest0-21swapper/320:58:563
1433999271,18cyclictest0-21swapper/300:34:263
1433999271,13cyclictest0-21swapper/321:58:003
1433999271,12cyclictest0-21swapper/322:52:403
1433999271,12cyclictest0-21swapper/321:09:123
1433999271,11cyclictest0-21swapper/300:15:183
1433999271,11cyclictest0-21swapper/300:12:083
1433999271,10cyclictest29757-21ntp_states00:35:203
1433999270,13cyclictest0-21swapper/322:05:283
1433999270,13cyclictest0-21swapper/319:31:483
14327992719,5cyclictest22769-21diskstats20:40:151
14327992718,6cyclictest10008-21/usr/sbin/munin20:10:171
1432099273,11cyclictest1440-21idleruntime-cro22:20:010
1433999268,3cyclictest0-21swapper/321:39:323
1433999268,10cyclictest0-21swapper/321:52:063
1433999267,3cyclictest0-21swapper/319:41:283
14339992614,2cyclictest0-21swapper/321:10:163
1433999261,11cyclictest0-21swapper/322:04:323
1433999261,11cyclictest0-21swapper/320:18:043
1433999261,10cyclictest0-21swapper/323:50:283
1433999261,10cyclictest0-21swapper/323:39:163
1433999261,10cyclictest0-21swapper/323:34:283
1433999261,10cyclictest0-21swapper/323:16:083
1433999260,9cyclictest0-21swapper/322:25:243
1433999260,12cyclictest0-21swapper/300:04:023
1433999260,11cyclictest14307-21cyclictest22:55:223
14327992617,6cyclictest6686-21/usr/sbin/munin23:45:181
1432099261,13cyclictest21874-21munin-run00:20:010
1433999256,10cyclictest0-21swapper/319:23:023
1433999252,9cyclictest0-21swapper/323:27:503
1433999251,9cyclictest0-21swapper/322:15:443
1433999251,11cyclictest0-21swapper/319:49:403
1433999251,10cyclictest0-21swapper/323:43:263
1433999251,10cyclictest0-21swapper/300:25:123
1433999251,10cyclictest0-21swapper/300:25:123
1433999250,11cyclictest0-21swapper/320:21:363
1433299251,6cyclictest0-21swapper/223:26:342
1433999247,9cyclictest0-21swapper/321:34:503
1433999247,10cyclictest0-21swapper/323:20:543
1433999247,10cyclictest0-21swapper/323:20:543
1433999241,9cyclictest0-21swapper/322:20:283
1433999241,9cyclictest0-21swapper/321:02:263
1433999241,8cyclictest0-21swapper/322:10:563
1433999241,10cyclictest0-21swapper/320:40:503
1432099249,12cyclictest21595-21awk19:25:000
1432099241,12cyclictest765-21idleruntime-cro21:05:010
1433999231,11cyclictest827-21runrttasks20:25:543
1433999230,16cyclictest0-21swapper/322:41:183
1433299231,5cyclictest0-21swapper/200:06:402
1432099231,10cyclictest15598-21munin-run20:25:010
1433999226,3cyclictest0-21swapper/320:54:563
1433299226,4cyclictest28754-21cat22:05:242
1433299221,7cyclictest0-21swapper/222:45:542
1433299221,5cyclictest0-21swapper/221:31:162
1432099227,3cyclictest26450-21cat00:30:010
1432099227,3cyclictest26450-21cat00:30:010
1433299211,4cyclictest0-21swapper/200:01:442
14332992113,4cyclictest7028-21grep21:15:282
1433299211,12cyclictest0-21swapper/222:10:552
1432099215,6cyclictest0-21swapper/022:55:230
1432099212,11cyclictest20838-21/usr/sbin/munin21:50:250
1432099211,6cyclictest0-21swapper/022:25:320
14320992111,3cyclictest87550irq/128-enp1s0-21:40:510
14320992110,3cyclictest87550irq/128-enp1s0-21:30:360
1433299203,4cyclictest5146-21sshd21:14:112
1433299201,6cyclictest0-21swapper/222:01:082
1433299201,6cyclictest0-21swapper/221:27:332
1433299201,6cyclictest0-21swapper/200:17:462
1433299201,5cyclictest1779-21tr22:20:132
1433299201,5cyclictest0-21swapper/223:44:402
1432099202,10cyclictest14524-21/usr/sbin/munin19:10:260
1432099201,9cyclictest11065-21gltestperf22:40:160
1432099201,5cyclictest0-21swapper/023:16:190
1432099201,12cyclictest0-21swapper/000:01:110
1432099201,10cyclictest19784-21head21:45:230
14320992010,3cyclictest87550irq/128-enp1s0-21:25:220
1432099200,7cyclictest23414-21needreboot20:40:200
14339991910,7cyclictest8924-21sshd20:09:103
1433299199,7cyclictest19310-21gltestperf21:45:162
1433299194,10cyclictest12122sleep222:17:262
1433299192,14cyclictest19755-21tr19:20:132
1433299191,7cyclictest0-21swapper/223:47:292
1433299191,6cyclictest0-21swapper/223:20:022
1433299191,6cyclictest0-21swapper/223:20:022
1433299191,6cyclictest0-21swapper/200:32:382
1433299191,5cyclictest21185-21df_inode21:50:142
1433299191,5cyclictest10848-21latency_hist23:55:022
1433299191,5cyclictest0-21swapper/223:35:112
1433299191,5cyclictest0-21swapper/200:29:152
1433299191,5cyclictest0-21swapper/200:29:152
1433299191,11cyclictest0-21swapper/200:22:382
1432099198,3cyclictest32448-21/usr/sbin/munin23:30:160
1432099191,7cyclictest6219-21tr22:30:130
1432099191,6cyclictest0-21swapper/022:05:060
1432099191,5cyclictest0-21swapper/000:09:410
14320991914,3cyclictest6925-21sed21:15:270
14320991914,3cyclictest28170-21grep19:40:010
1432099191,10cyclictest808-21snmpd19:49:100
1432099191,10cyclictest14240-21/usr/sbin/munin21:35:200
95352181,2sleep00-21swapper/021:24:070
1433299185,5cyclictest755-21rngd00:11:222
1433299185,3cyclictest87350irq/126-enp1s0-23:55:212
1433299183,5cyclictest30972-21sadc21:00:012
1433299183,3cyclictest87650irq/129-enp1s0-22:40:202
1433299183,2cyclictest87650irq/129-enp1s0-19:30:252
1433299182,3cyclictest87650irq/129-enp1s0-20:34:112
1433299181,6cyclictest0-21swapper/222:50:292
1433299181,6cyclictest0-21swapper/222:31:102
1433299181,5cyclictest0-21swapper/223:33:242
1433299181,5cyclictest0-21swapper/223:10:072
1433299181,5cyclictest0-21swapper/222:36:042
1433299181,5cyclictest0-21swapper/219:40:232
14332991813,3cyclictest6322-21grep20:00:252
14332991812,3cyclictest1368-21df21:05:152
1433299181,15cyclictest0-21swapper/223:15:202
1432099189,2cyclictest0-21swapper/023:00:170
1432099187,3cyclictest2629-21/usr/sbin/munin19:55:110
1432099187,2cyclictest17787-21cpuspeed_turbos00:10:130
1432099182,6cyclictest16636-21munin-node23:35:080
1432099181,7cyclictest30368-21perf19:45:010
1432099181,7cyclictest0-21swapper/022:35:090
1432099181,5cyclictest0-21swapper/023:26:350
1432099181,5cyclictest0-21swapper/022:13:270
1432099181,5cyclictest0-21swapper/021:55:210
1432099181,10cyclictest28228-21sshd00:32:120
1432099181,10cyclictest0-21swapper/000:37:310
1433299174,4cyclictest28998-21expr00:35:112
1433299171,6cyclictest9231-21/usr/sbin/munin20:10:142
1433299171,6cyclictest24382-21latency_hist20:45:012
1433299171,6cyclictest0-21swapper/223:00:162
1433299171,5cyclictest0-21swapper/222:27:082
1433299171,5cyclictest0-21swapper/220:18:282
1433299171,3cyclictest0-21swapper/223:05:452
1433299171,13cyclictest4196-21head19:55:252
14332991711,3cyclictest18478-21ntp_states19:15:222
14332991711,3cyclictest1017-21gltestperf19:50:142
1433299170,5cyclictest25202-21sed22:00:012
1433299170,5cyclictest0-21swapper/219:10:232
1432099178,2cyclictest0-21swapper/022:00:220
1432099171,9cyclictest0-21swapper/020:45:190
1432099171,7cyclictest28787-21munin-run20:55:010
1432099171,6cyclictest0-21swapper/023:55:380
1432099171,6cyclictest0-21swapper/019:25:170
1432099171,4cyclictest0-21swapper/022:20:230
14320991713,3cyclictest0-21swapper/020:32:090
14320991711,4cyclictest7147-21grep20:05:120
1432099171,11cyclictest0-21swapper/021:10:150
14320991710,5cyclictest9871-21cut20:10:150
1432099170,6cyclictest0-21swapper/019:30:220
1432099170,11cyclictest0-21swapper/020:55:150
10192171,2sleep00-21swapper/021:05:050
1433299166,8cyclictest17325-21unixbench_multi20:25:262
1433299165,3cyclictest87650irq/129-enp1s0-20:20:242
1433299165,3cyclictest87650irq/129-enp1s0-19:35:132
1433299164,3cyclictest87350irq/126-enp1s0-20:45:172
1433299163,3cyclictest87650irq/129-enp1s0-19:45:182
1433299162,11cyclictest7447-21tune2fs20:05:142
14332991611,4cyclictest9665-21kworker/u8:119:25:202
14332991611,3cyclictest32461-21mailstats21:00:242
1432099168,3cyclictest0-21swapper/023:20:150
1432099168,3cyclictest0-21swapper/023:20:150
1432099166,3cyclictest14358-21kworker/u8:222:46:580
1432099161,6cyclictest20223-21/usr/sbin/munin20:35:240
1432099161,6cyclictest11426-21/usr/sbin/munin20:15:200
1432099161,5cyclictest829150kworker/u8:223:48:140
14320991611,3cyclictest17160-21sed20:25:250
14320991610,3cyclictest8902-21cpu23:50:100
1433299154,3cyclictest87550irq/128-enp1s0-20:35:172
1433299151,7cyclictest0-21swapper/220:50:142
1432099151,5cyclictest17415-21/usr/sbin/munin19:15:130
1432099151,4cyclictest0-21swapper/023:40:040
14320991510,3cyclictest5968-21awk20:00:220
14320991510,3cyclictest23820-21grep23:10:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional