You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-03 - 00:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Aug 02, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
129502240163,23sleep10-21swapper/107:07:481
129042231162,57sleep20-21swapper/207:07:122
129622204167,25sleep30-21swapper/307:07:573
128672201165,24sleep00-21swapper/007:06:420
81721680,5sleep31322499cyclictest09:05:203
132179910092,2cyclictest0-21swapper/208:33:472
13217998582,2cyclictest0-21swapper/208:19:582
13217996461,2cyclictest0-21swapper/210:26:172
13217996461,2cyclictest0-21swapper/210:26:172
158542570,3sleep115858-21unixbench-2d12:05:251
166262470,1sleep10-21swapper/107:15:141
13212994410,26cyclictest2139-21grep10:20:271
13212993810,18cyclictest12001-21ntp_states09:30:211
13212993714,19cyclictest12573-21/usr/sbin/munin08:20:141
1321299368,19cyclictest18916-21timerandwakeup09:45:251
1321299366,13cyclictest87550irq/128-enp1s0-10:05:251
13212993615,9cyclictest32262-21/usr/sbin/munin10:15:261
13212993513,19cyclictest15387-21munin-run07:15:011
13212993512,14cyclictest7090-21grep11:45:261
13212993416,15cyclictest24939-21irqstats10:00:181
13212993414,11cyclictest13842-21cpuspeed_turbos10:50:141
13212993413,12cyclictest4024-21sensors_temp10:25:251
13212993413,12cyclictest4024-21sensors_temp10:25:251
13212993411,15cyclictest7177-21cpu10:35:131
13212993410,21cyclictest28016-21ntp_states11:20:201
1321299337,11cyclictest0-21swapper/111:05:161
13212993315,15cyclictest5352-21munin-run11:45:011
13212993310,20cyclictest18972-21kernelversion11:00:181
1322499320,17cyclictest0-21swapper/311:14:163
1321299325,16cyclictest0-21swapper/111:50:301
1321299322,19cyclictest89250irq/135-enp2s0-07:43:391
13212993215,14cyclictest14051-21memory09:35:181
13212993213,16cyclictest6098-21grep10:30:231
13212993210,18cyclictest17471-21sed10:55:251
1322499310,23cyclictest0-21swapper/312:29:563
13212993115,13cyclictest12225-21hddtemp_smartct10:45:161
13212993114,14cyclictest17609-21nfs4_client12:10:221
13212993112,7cyclictest2147-21/usr/sbin/munin09:10:261
13212993111,17cyclictest827-21runrttasks09:25:171
13212993110,12cyclictest523-21hddtemp_smartct09:05:171
13212993110,10cyclictest19118-21/usr/sbin/munin08:35:201
1322499301,17cyclictest0-21swapper/309:35:163
1321299309,18cyclictest12391-21sort08:20:021
1321299309,13cyclictest11321-21latency_hist10:45:021
1321299306,7cyclictest29697-21/usr/sbin/munin10:10:231
1321299306,15cyclictest0-21swapper/108:00:191
13212993015,4cyclictest27918-21diskstats12:35:131
13212993015,12cyclictest23505-21/usr/sbin/munin08:45:131
13212993013,14cyclictest20480-21perf07:25:011
1322499291,11cyclictest0-21swapper/310:36:323
1322499290,11cyclictest0-21swapper/309:42:553
1321299299,8cyclictest27906-21/usr/sbin/munin08:55:171
1321299299,5cyclictest25672-21/usr/sbin/munin11:15:191
1321299299,17cyclictest15206-21grep08:25:171
1321299298,18cyclictest6321-21perf09:20:011
1321299298,18cyclictest6321-21perf09:20:001
1321299298,12cyclictest29482-21/usr/sbin/munin07:45:171
13212992921,5cyclictest32606-21smart_sda11:30:241
13212992920,6cyclictest22994-21ntp_states09:55:231
13212992915,11cyclictest23513-21df12:25:151
13212992914,13cyclictest3168-21awk11:40:011
13212992912,14cyclictest22746-21cat07:30:011
13212992912,14cyclictest21686-21cat09:54:561
13212992911,15cyclictest11953-21idleruntime-cro12:00:021
13212992911,10cyclictest29210-21grep11:25:121
13212992910,16cyclictest17304-21latency_hist09:45:011
162262280,4sleep2311ktimersoftd/209:40:192
1322499280,9cyclictest0-21swapper/308:25:263
1321299287,7cyclictest0-21swapper/111:10:121
13212992813,12cyclictest6648-21grep09:20:131
13212992811,9cyclictest19843-21ntp_offset12:15:231
1322499271,18cyclictest0-21swapper/308:33:433
1322499271,18cyclictest0-21swapper/307:47:553
1322499270,9cyclictest0-21swapper/312:04:393
1322499270,19cyclictest0-21swapper/312:12:593
1322499270,17cyclictest0-21swapper/309:26:073
1322499270,11cyclictest0-21swapper/310:43:403
1321299279,15cyclictest26893-21taskset07:38:051
1321299279,15cyclictest21033-21cpu12:20:131
1321299277,17cyclictest31748-21wc07:50:141
1321299277,13cyclictest0-21swapper/107:55:161
1321299276,10cyclictest12266-21cpuspeed_turbos12:00:141
13212992718,6cyclictest6999-21ntp_states08:05:271
13212992718,6cyclictest21667-21/usr/sbin/munin08:40:151
13212992718,5cyclictest26967-21smart_sda08:50:241
13212992718,5cyclictest18181-21smart_sda08:30:261
1320799270,12cyclictest6330-21idleruntime-cro09:20:000
1320799270,12cyclictest6330-21idleruntime-cro09:20:000
1320799270,12cyclictest32714-21perf10:20:010
1322499261,11cyclictest0-21swapper/307:35:213
1322499260,9cyclictest0-21swapper/308:37:593
1322499260,10cyclictest0-21swapper/312:33:563
1322499260,10cyclictest0-21swapper/312:08:393
1322499260,10cyclictest0-21swapper/309:13:273
1322499260,10cyclictest0-21swapper/308:44:143
1322499260,10cyclictest0-21swapper/307:19:233
1321799261,12cyclictest0-21swapper/208:53:002
13212992618,5cyclictest9181-21ntp_states08:10:211
13212992618,5cyclictest23239-21diskstats07:30:141
13212992617,6cyclictest25376-21/usr/sbin/munin12:30:191
13212992617,5cyclictest31361-21smart_sda09:00:241
1320799261,2cyclictest20960-21kworker/0:008:52:590
284732250,2sleep30-21swapper/312:35:173
1322499258,8cyclictest0-21swapper/307:43:553
1322499251,9cyclictest19125-21/usr/sbin/munin12:15:153
1322499251,8cyclictest0-21swapper/308:07:583
1322499251,13cyclictest703-21sed07:50:263
1322499251,11cyclictest0-21swapper/311:58:033
1322499251,11cyclictest0-21swapper/310:32:283
1322499251,11cyclictest0-21swapper/309:56:043
1322499251,11cyclictest0-21swapper/308:49:403
1322499251,10cyclictest0-21swapper/310:17:023
1322499250,12cyclictest0-21swapper/310:49:293
1320799251,18cyclictest0-21swapper/010:05:170
1322499249,5cyclictest27919-21sed11:20:193
1322499248,3cyclictest23882-21strings07:30:193
1322499241,9cyclictest0-21swapper/308:23:323
1322499241,10cyclictest0-21swapper/307:14:063
1322499240,12cyclictest0-21swapper/311:37:403
1322499240,11cyclictest0-21swapper/310:07:403
1322499240,10cyclictest0-21swapper/311:48:083
1322499240,10cyclictest0-21swapper/307:57:383
1321799242,7cyclictest28925-21munin-run11:25:002
1321799240,16cyclictest0-21swapper/210:56:122
1320799241,16cyclictest22395-21kworker/0:207:30:070
1322499238,3cyclictest0-21swapper/308:00:083
1322499238,3cyclictest0-21swapper/307:20:393
1322499236,3cyclictest0-21swapper/308:10:593
1322499231,9cyclictest0-21swapper/310:01:023
1322499231,8cyclictest0-21swapper/308:16:513
1322499231,8cyclictest0-21swapper/307:28:013
1322499231,12cyclictest0-21swapper/311:25:443
1322499230,12cyclictest0-21swapper/310:52:003
1322499230,12cyclictest0-21swapper/309:34:473
13207992314,3cyclictest0-21swapper/011:38:080
1320799231,11cyclictest30517-21basename10:15:010
1322499229,4cyclictest0-21swapper/310:27:423
1322499229,4cyclictest0-21swapper/310:27:423
1322499227,3cyclictest0-21swapper/311:51:073
1322499227,3cyclictest0-21swapper/308:50:443
1322499226,11cyclictest0-21swapper/311:33:363
1322499225,3cyclictest0-21swapper/311:07:243
1322499223,11cyclictest15954-21/usr/sbin/munin10:55:263
1322499222,11cyclictest17536-21/usr/sbin/munin09:45:143
1322499221,3cyclictest0-21swapper/308:56:543
1322499220,9cyclictest0-21swapper/310:20:163
1322499220,10cyclictest0-21swapper/312:20:353
1320799221,16cyclictest0-21swapper/011:40:280
293372210,2sleep30-21swapper/310:10:193
1322499213,3cyclictest0-21swapper/311:19:293
1321799211,13cyclictest0-21swapper/210:11:002
1320799219,7cyclictest0-21swapper/012:37:450
1320799219,3cyclictest3835-21ntp_states10:25:230
1320799219,3cyclictest3835-21ntp_states10:25:230
1320799218,5cyclictest0-21swapper/011:18:160
1320799216,6cyclictest0-21swapper/010:34:420
1320799216,4cyclictest10024-21hddtemp_smartct10:40:140
1320799211,9cyclictest0-21swapper/012:13:190
1320799211,4cyclictest0-21swapper/009:22:430
13207992112,2cyclictest0-21swapper/009:44:590
1322499205,3cyclictest0-21swapper/311:41:323
1321799203,11cyclictest3922-21taskset09:12:382
1321799202,7cyclictest1269-21munin-run07:55:002
1321799202,5cyclictest4693-21diskstats09:15:132
1321799202,5cyclictest4693-21diskstats09:15:132
1321799202,14cyclictest0-21swapper/212:14:282
1321799201,7cyclictest0-21swapper/210:52:512
1321799201,3cyclictest0-21swapper/211:38:072
13217992013,4cyclictest13754-21df_inode07:10:152
1321799201,15cyclictest26409-21grep12:30:212
1320799209,3cyclictest20547-21sshd12:17:340
1320799209,3cyclictest1916-21diskmemload10:35:160
1320799209,3cyclictest0-21swapper/010:55:400
1320799208,5cyclictest0-21swapper/011:13:020
1320799208,4cyclictest0-21swapper/012:07:060
1320799208,3cyclictest0-21swapper/011:21:070
1320799205,6cyclictest0-21swapper/012:04:230
1320799201,12cyclictest0-21swapper/010:45:200
1320799201,11cyclictest19738-21/usr/sbin/munin09:50:180
1320799201,10cyclictest26642-21grep07:35:260
1320799201,10cyclictest25061-21head08:45:230
1320799201,10cyclictest0-21swapper/012:22:070
237502191,2sleep20-21swapper/209:58:172
142852191,3sleep20-21swapper/208:22:342
1322499197,3cyclictest0-21swapper/309:00:593
1322499195,3cyclictest0-21swapper/309:21:323
1322499190,9cyclictest0-21swapper/311:05:003
1321799196,4cyclictest10602-21/usr/sbin/munin10:40:232
1321799191,7cyclictest0-21swapper/211:53:112
13217991913,3cyclictest28099-21fschecks_time12:35:142
1321799191,16cyclictest0-21swapper/207:45:262
1320799199,3cyclictest18155-21/usr/sbin/munin11:00:180
1320799197,3cyclictest0-21swapper/009:46:400
1320799194,2cyclictest3905-21kworker/0:009:33:320
1320799192,10cyclictest808-21snmpd08:22:280
1320799191,10cyclictest21386-21nfs4_client11:05:200
1320799191,10cyclictest0-21swapper/011:27:300
78792180,2sleep00-21swapper/008:10:000
291402181,2sleep20-21swapper/211:25:052
248622181,2sleep20-21swapper/212:26:342
18762181,2sleep00-21swapper/009:09:390
1322499181,6cyclictest0-21swapper/309:50:223
1322499181,3cyclictest0-21swapper/309:19:163
1322499181,3cyclictest0-21swapper/309:19:163
1321799186,3cyclictest87350irq/126-enp1s0-07:55:152
1321799183,6cyclictest32-21ksoftirqd/210:35:272
1321799183,5cyclictest6389-21irqstats11:45:172
1321799182,8cyclictest87350irq/126-enp1s0-10:00:182
1321799181,7cyclictest20732-21needreboot09:50:222
1321799181,6cyclictest0-21swapper/212:02:402
1321799181,6cyclictest0-21swapper/210:20:042
1321799181,6cyclictest0-21swapper/209:31:142
1321799181,5cyclictest19016-21unixbench-2d09:45:272
13217991813,3cyclictest8954-21kernelversion08:10:182
13217991812,3cyclictest16960-21gltestperf07:15:172
1321799181,15cyclictest0-21swapper/210:09:382
13217991811,4cyclictest21781-21grep08:40:152
1320799184,3cyclictest11-21rcuc/010:00:270
1320799183,2cyclictest3136-21grep09:10:200
1320799182,7cyclictest31680-21/usr/sbin/munin07:50:270
1320799181,9cyclictest808-21snmpd08:17:270
1320799181,9cyclictest5578-21/usr/sbin/munin11:45:140
1320799181,8cyclictest2133-21unixbench_singl10:20:270
1320799181,6cyclictest0-21swapper/009:28:360
1320799181,5cyclictest11042-21ntp_states11:55:230
13207991814,3cyclictest0-21swapper/007:25:200
13207991812,4cyclictest0-21swapper/012:30:140
13207991812,3cyclictest19650-21head07:20:240
13207991812,3cyclictest14893-21grep07:10:260
1320799181,12cyclictest0-21swapper/011:51:320
1320799181,10cyclictest0-21swapper/012:27:040
1320799180,8cyclictest0-21swapper/009:04:340
51112170,2sleep20-21swapper/211:42:072
156912171,2sleep00-21swapper/010:54:410
1321799179,5cyclictest909-21sssd_nss08:45:172
1321799176,3cyclictest87350irq/126-enp1s0-12:15:172
1321799174,4cyclictest1916-21diskmemload10:31:282
1321799174,11cyclictest24469-21head07:30:252
1321799173,3cyclictest22206-21gzip12:20:242
1321799172,4cyclictest13017-21grep10:45:252
1321799171,7cyclictest24758-21/usr/sbin/munin11:15:172
1321799171,7cyclictest0-21swapper/211:55:092
1321799171,7cyclictest0-21swapper/209:37:432
1321799171,7cyclictest0-21swapper/207:44:592
1321799171,6cyclictest31410-21smartctl10:15:162
1321799171,6cyclictest16682-21crond08:30:012
1321799171,6cyclictest10248-21sed09:25:262
13217991712,3cyclictest7615-21ntp_states09:20:242
13217991712,3cyclictest31346-21/usr/sbin/munin11:30:222
13217991711,3cyclictest14382-21/usr/sbin/munin12:05:142
1321799170,7cyclictest19980-21unixbench-2d07:20:272
1320799175,3cyclictest87550irq/128-enp1s0-07:45:120
1320799175,3cyclictest15017-21ls08:25:140
1320799173,9cyclictest0-21swapper/009:55:290
1320799172,3cyclictest87550irq/128-enp1s0-08:55:210
1320799171,4cyclictest1916-21diskmemload11:30:200
1320799171,4cyclictest16922-21/usr/sbin/munin08:30:150
1320799171,13cyclictest13259-21tr09:35:130
1321799168,5cyclictest1156-21/usr/sbin/munin09:05:242
1321799163,4cyclictest32-21ksoftirqd/211:05:232
1321799161,2cyclictest0-21swapper/208:57:372
13217991612,3cyclictest0-21swapper/208:05:072
1321799161,14cyclictest0-21swapper/211:11:542
1321799161,14cyclictest0-21swapper/207:38:462
13217991611,3cyclictest4028-21cat08:00:142
13217991611,3cyclictest21465-21if_err_enp1s007:25:162
13217991610,3cyclictest18837-21hwlatdetect11:00:172
1320799162,8cyclictest0-21swapper/008:41:380
1320799162,5cyclictest8927-21irqstats08:10:170
13207991612,3cyclictest0-21swapper/007:55:330
13207991611,3cyclictest3718-21/usr/sbin/munin08:00:200
13207991611,3cyclictest27528-21expr07:40:110
13207991611,3cyclictest15520-21munin-node07:15:030
1321799154,4cyclictest19926-21interrupts08:35:192
1321799150,8cyclictest0-21swapper/209:00:182
13207991510,3cyclictest19628-21cut08:35:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional