You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-24 - 22:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Mon Aug 24, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
271702236165,58sleep10-21swapper/107:06:231
271972228163,23sleep30-21swapper/307:06:453
273832221191,20sleep20-21swapper/207:09:092
271122202164,26sleep00-21swapper/007:05:370
27549996663,2cyclictest0-21swapper/207:38:252
239732620,5sleep023974-21unixbench_multi08:10:270
89992600,1sleep30-21swapper/308:50:263
251992510,1sleep10-21swapper/109:30:021
230792470,2sleep00-21swapper/010:35:260
2754499437,29cyclictest20948-21ntp_kernel_pll_11:45:201
27549994112,15cyclictest20869-21/usr/sbin/munin09:20:132
27549994018,12cyclictest16879-21awk10:25:012
2754499391,10cyclictest25984-21threads08:15:251
27549993816,9cyclictest22446-21/usr/sbin/munin11:50:142
27549993813,15cyclictest1119-21fschecks_count12:15:152
27549993812,14cyclictest24229-21ps09:25:232
2754499386,4cyclictest18362open_files12:15:221
2754499381,11cyclictest31037-21timerandwakeup09:40:251
27549993714,8cyclictest10913-21grep08:55:212
27549993712,15cyclictest22401-21memory10:35:182
27549993712,15cyclictest11353-21latency10:10:182
27549993711,15cyclictest6259-21sshd11:12:112
2754499371,12cyclictest1-21systemd09:00:011
27549993614,10cyclictest26329-21ntp_states09:30:222
2754499361,12cyclictest29881-21ps12:05:231
27549993515,8cyclictest12068-21sshd09:00:102
27549993515,7cyclictest17710-21/usr/sbin/munin11:40:192
27549993512,8cyclictest6823-21latency_hist07:35:022
27549993512,13cyclictest26257-21latency_hist12:00:022
27549993511,10cyclictest10518-21cut12:35:202
27549993510,15cyclictest19929-21mailstats09:15:242
27549993510,10cyclictest1441-21crond09:54:572
2754499357,4cyclictest5551-21fschecks_count12:25:141
27544993512,20cyclictest26497-21/usr/sbin/munin12:00:181
244282350,10sleep121-21rcuc/108:15:011
27549993417,5cyclictest87350irq/126-enp1s0-11:20:212
27549993413,11cyclictest32047-21cat09:45:142
27549993412,9cyclictest14898-21awk11:30:262
27549993412,13cyclictest9276-21sed10:05:202
27549993411,8cyclictest0-21swapper/209:35:242
27549993410,14cyclictest4266-21fschecks_count09:55:162
2754499348,4cyclictest23610-21cat10:40:021
2754499348,23cyclictest25471-21processes11:55:241
2754499342,28cyclictest20641-21munin-run09:20:011
2754499341,9cyclictest32640-21cpuspeed_turbos11:00:131
27544993411,3cyclictest7918-21gltestperf12:30:161
27544993411,20cyclictest17711-21acpi11:40:131
27544993410,3cyclictest13995-21df_inode07:50:121
27544993410,2cyclictest0-21swapper/108:36:451
2754999338,14cyclictest360-21diskstats11:00:142
2754999338,11cyclictest87350irq/126-enp1s0-09:40:272
27549993314,7cyclictest17155-21taskset10:25:132
27549993312,8cyclictest0-21swapper/209:14:112
27549993311,8cyclictest24486-21interrupts10:40:182
27549993311,6cyclictest11880-21if_enp2s011:25:172
27549993311,10cyclictest3751-21irqstats12:20:172
27549993310,9cyclictest504-21cpuspeed_turbos07:20:132
2754499331,16cyclictest8151-21latency_hist10:05:001
2754499331,10cyclictest15490-21date11:35:021
27544993310,2cyclictest6143-21hddtemp_smartct08:45:171
2754499330,4cyclictest0-21swapper/110:31:541
27539993321,5cyclictest0-21swapper/012:20:260
2753999330,23cyclictest25477-21processes11:55:240
2754999329,8cyclictest21006-21runrttasks10:31:542
2754999329,14cyclictest87350irq/126-enp1s0-12:25:202
27549993215,9cyclictest17821-21latency_hist08:00:002
27549993214,7cyclictest16243-21diskmemload11:35:462
27549993214,7cyclictest16243-21diskmemload11:35:462
27549993211,9cyclictest27498-21chrt10:45:392
27549993211,8cyclictest21996-21latency_hist11:50:022
27549993211,10cyclictest7470-21if_enp2s011:15:182
2754499326,5cyclictest13343-21sshd07:49:101
2754499326,4cyclictest0-21swapper/111:55:011
2754499325,3cyclictest32255-21munin-run11:00:021
2754499321,5cyclictest4524-21/usr/sbin/munin11:10:151
2754499321,3cyclictest26079-21cat10:45:131
2754499321,27cyclictest4201-21smart_sda12:20:231
2754499321,15cyclictest29431-21awk09:40:011
2754499321,12cyclictest2033-21df09:50:131
2754499321,10cyclictest16243-21diskmemload09:55:151
27544993210,3cyclictest6719-21/usr/sbin/munin11:15:161
27544993210,3cyclictest11567-21ntp_states10:10:201
2754499320,9cyclictest0-21swapper/111:25:121
2754499319,3cyclictest30021-21grep07:15:021
2754499318,4cyclictest10168-21/usr/sbin/munin07:40:171
2754499318,3cyclictest16820-21df_inode09:10:131
27544993113,15cyclictest27647-21cat08:20:171
2754499311,27cyclictest13042-21diskstats10:15:111
27544993112,16cyclictest2329-21/usr/sbin/munin11:05:251
27544993110,3cyclictest827-21runrttasks08:26:531
27544993110,3cyclictest827-21runrttasks07:12:531
27544993110,3cyclictest15089-21cat10:20:111
2754499310,3cyclictest1242-21/usr/sbin/munin07:20:181
2754999309,11cyclictest29678-21vmstat10:50:242
27549993022,5cyclictest27568-21ntp_states12:00:222
27549993019,7cyclictest14911-21smart_sda07:50:212
27549993013,6cyclictest30593-21cstates10:55:122
27549993013,5cyclictest1058-21/usr/sbin/munin08:35:192
27549993011,5cyclictest18050-21/usr/sbin/munin08:00:242
2754499303,5cyclictest9565-21smart_sda10:05:241
2754499302,4cyclictest21515-21smart_sda08:05:251
2754499302,4cyclictest21515-21smart_sda08:05:251
2754499302,3cyclictest20021-21crond08:05:011
2754499301,4cyclictest9832-21memory11:20:191
2754499301,27cyclictest0-21swapper/107:35:121
2754499301,12cyclictest17766-21hddtemp_smartct10:25:161
27544993010,3cyclictest827-21runrttasks12:37:511
2754499300,2cyclictest0-21swapper/109:20:291
2753999301,19cyclictest0-21swapper/010:10:250
27549992920,6cyclictest8014-21/usr/sbin/munin12:30:172
27549992912,4cyclictest30635-21users08:25:272
2754499294,5cyclictest12663-21head09:00:161
2754499293,3cyclictest16247-21munin-run09:10:011
2754499291,8cyclictest0-21swapper/112:10:121
2754499291,6cyclictest3897-21tail08:40:151
2754499291,6cyclictest0-21swapper/110:40:231
27544992911,2cyclictest810-21sshd07:25:091
27544992910,3cyclictest31987-21expr09:45:121
27549992821,4cyclictest3241-21memory11:05:202
27549992821,4cyclictest21843-21users08:05:272
27549992821,4cyclictest21843-21users08:05:262
27549992820,5cyclictest9601-21diskstats07:40:112
27549992820,5cyclictest28113-21smart_sda08:20:242
27549992819,6cyclictest3011-21diskstats07:25:152
27549992819,6cyclictest25562-21memory08:15:192
27549992818,7cyclictest8941-21/usr/sbin/munin08:50:262
27549992818,7cyclictest31847-21/usr/sbin/munin07:15:212
27549992818,7cyclictest29799-21/usr/sbin/munin12:05:232
27549992818,7cyclictest27745-21/usr/sbin/munin07:10:212
27549992818,7cyclictest14625-21diskstats09:05:142
27549992818,6cyclictest6265-21/usr/sbin/munin08:45:182
27549992811,5cyclictest13028-21df10:15:112
2754499287,3cyclictest4859-21/usr/sbin/munin07:30:161
2754499282,2cyclictest87350irq/126-enp1s0-08:55:011
27544992810,2cyclictest28083-21/usr/sbin/munin10:50:251
2753999287,13cyclictest8145-21irqstats12:30:190
2753999285,15cyclictest18943-21wc11:40:230
2753999282,13cyclictest25651-21munin-run10:45:010
27539992811,8cyclictest13646-21/usr/sbin/munin07:50:210
2753999281,13cyclictest27868-21perf10:50:010
2753999280,13cyclictest0-21swapper/009:45:010
27549992720,4cyclictest4089-21irqstats08:40:172
27549992719,5cyclictest31609-21/usr/sbin/munin08:30:152
27549992718,6cyclictest11803-21diskstats07:45:142
27549992717,7cyclictest22818-21/usr/sbin/munin08:10:162
27549992713,11cyclictest30980-21cpuspeed_turbos12:10:132
2754499277,4cyclictest909-21sssd_nss09:30:101
2754499277,3cyclictest17169-21switchtime07:55:241
2754499277,2cyclictest31754-21fschecks_time08:30:161
2754499276,3cyclictest17486-21munin-run11:40:011
2754499276,3cyclictest17486-21munin-run11:40:011
27539992710,8cyclictest7133-21latency_hist12:30:010
272342270,2sleep00-21swapper/012:00:170
27549992617,6cyclictest6443-21/usr/sbin/munin10:00:152
2753999265,11cyclictest15570-21hddtemp_smartct10:20:160
2753999262,15cyclictest87350irq/126-enp1s0-09:15:130
2753999262,12cyclictest9642-21munin-run08:55:010
2753999261,17cyclictest0-21swapper/011:30:010
2753999261,12cyclictest8076-21munin-run10:05:010
2753999259,6cyclictest28223-21awk09:35:150
2753999253,10cyclictest1461-21perf09:50:020
2753999251,15cyclictest16252-21perf09:10:010
2753999251,12cyclictest26623-21awk08:20:010
27539992510,9cyclictest81ktimersoftd/009:59:260
2753999244,10cyclictest0-21swapper/010:30:160
2753999243,9cyclictest31978-21ntp_states12:10:220
2753999242,10cyclictest20247-21sshd08:05:080
2753999242,10cyclictest20247-21sshd08:05:080
2753999241,15cyclictest87350irq/126-enp1s0-11:35:020
2753999241,14cyclictest21849-21needreboot09:20:170
27539992410,4cyclictest87350irq/126-enp1s0-11:01:390
2753999234,10cyclictest0-21swapper/011:45:160
2753999233,2cyclictest17976-21latency_hist08:00:010
2753999232,18cyclictest17610-21grep09:10:220
2753999232,16cyclictest87350irq/126-enp1s0-12:05:270
2753999232,15cyclictest7-21ksoftirqd/009:34:070
27539992314,3cyclictest0-21swapper/011:15:550
2753999224,8cyclictest6486-21latency_hist11:15:010
2753999224,11cyclictest755-21rngd08:30:450
2753999223,8cyclictest11455-21/usr/sbin/munin07:45:140
2753999223,3cyclictest6580-21kworker/0:207:35:250
2753999223,10cyclictest808-21snmpd08:22:470
2753999222,15cyclictest87350irq/126-enp1s0-10:50:130
27539992212,2cyclictest18050-21/usr/sbin/munin08:00:150
2753999221,15cyclictest0-21swapper/011:07:140
2753999221,15cyclictest0-21swapper/010:56:160
2753999220,3cyclictest0-21swapper/010:15:230
2753999220,13cyclictest30627-21/usr/sbin/munin07:15:230
2755499218,4cyclictest20263-21chrt09:15:273
2755499211,12cyclictest31936-21cpuspeed_turbos09:45:133
2755499211,12cyclictest0-21swapper/309:40:243
2755499211,11cyclictest362-21/usr/sbin/munin07:20:123
2755499210,9cyclictest0-21swapper/308:19:213
2753999217,3cyclictest0-21swapper/010:08:330
2753999215,8cyclictest87550irq/128-enp1s0-11:50:140
2753999211,9cyclictest0-21swapper/009:26:100
2753999211,11cyclictest1117-21kworker/u8:209:50:360
2753999210,3cyclictest88950irq/132-enp2s0-08:38:450
2753999210,11cyclictest1072-21diskstats12:15:150
2755499208,3cyclictest1708-21users11:00:273
2755499208,3cyclictest0-21swapper/308:43:343
2755499206,3cyclictest0-21swapper/308:05:283
2755499206,3cyclictest0-21swapper/308:05:283
2755499201,9cyclictest0-21swapper/311:50:173
2755499201,5cyclictest8885-21grep12:30:263
2755499201,3cyclictest0-21swapper/309:54:553
2755499201,11cyclictest21486-21/usr/sbin/munin10:35:193
2755499201,11cyclictest13922-21grep10:15:213
2755499201,10cyclictest19310-21smart_sda08:00:243
2755499201,10cyclictest0-21swapper/308:57:143
2755499200,11cyclictest0-21swapper/312:03:353
2753999209,6cyclictest0-21swapper/011:20:360
2753999206,5cyclictest4188-21/usr/sbin/munin08:40:180
2753999203,4cyclictest394-21cpu07:20:130
2753999201,14cyclictest0-21swapper/010:28:200
2753999201,13cyclictest87350irq/126-enp1s0-12:35:290
2753999201,11cyclictest9870-21/usr/sbin/munin08:55:140
2753999200,14cyclictest87350irq/126-enp1s0-11:35:320
2753999200,14cyclictest87350irq/126-enp1s0-11:35:320
2755499198,3cyclictest19902-21/usr/sbin/munin11:45:203
2755499194,12cyclictest41-21ksoftirqd/307:35:163
2755499192,3cyclictest9891-21users10:05:273
2755499192,13cyclictest39-21rcuc/308:35:163
2755499191,6cyclictest0-21swapper/312:36:243
2755499191,5cyclictest0-21swapper/310:10:283
2755499191,4cyclictest0-21swapper/312:13:483
2755499191,4cyclictest0-21swapper/309:55:163
2755499191,15cyclictest5853-21basename12:25:163
2755499191,10cyclictest6149-21cat07:30:233
2755499191,10cyclictest25038-21cat10:40:263
2755499190,7cyclictest16719-21kworker/u8:010:45:133
2753999197,3cyclictest5927-21ntp_states07:30:200
2753999192,6cyclictest29706-21latency_hist07:15:020
2753999191,10cyclictest29048-21/usr/sbin/munin08:25:270
2755499185,11cyclictest16262-21if_enp2s011:35:183
2755499185,11cyclictest16262-21if_enp2s011:35:183
2755499184,4cyclictest6444-21diskstats10:00:153
2755499184,11cyclictest39-21rcuc/308:45:123
2755499182,7cyclictest16243-21diskmemload11:56:393
2755499182,6cyclictest16634-21sshd10:22:103
2755499181,8cyclictest1136-21grep12:15:153
2755499181,7cyclictest0-21swapper/309:39:523
2755499181,6cyclictest0-21swapper/311:26:253
27554991815,2cyclictest0-21swapper/307:55:223
2755499181,3cyclictest29126-21ntp_kernel_pll_10:50:193
2755499181,15cyclictest0-21swapper/309:23:393
2755499181,14cyclictest24528-21grep09:25:253
2755499181,12cyclictest87350irq/126-enp1s0-09:30:153
2755499181,11cyclictest41-21ksoftirqd/310:30:253
2755499181,10cyclictest27697-21grep08:20:183
2755499180,10cyclictest0-21swapper/307:20:023
2753999188,2cyclictest10914-21sshd07:41:100
2753999183,8cyclictest421-21kworker/0:008:45:490
2753999181,5cyclictest0-21swapper/009:00:500
44542171,2sleep30-21swapper/307:27:573
2755499173,11cyclictest41-21ksoftirqd/309:05:193
2755499172,12cyclictest839-21lldpd11:06:213
2755499172,12cyclictest7528-21irqstats11:15:173
2755499172,12cyclictest4893-21fschecks_count11:10:143
2755499171,8cyclictest27741-21kworker/u8:007:41:213
2755499171,7cyclictest14650-21grep11:30:243
2755499171,6cyclictest28770-21cpu12:05:133
2755499171,6cyclictest15822-21cat07:55:013
2755499171,5cyclictest0-21swapper/309:10:163
27554991712,3cyclictest24474-21date08:15:013
2755499171,14cyclictest0-21swapper/310:56:493
27554991711,3cyclictest309-21sed08:30:263
2755499171,11cyclictest0-21swapper/311:41:023
2755499171,10cyclictest0-21swapper/310:25:103
2755499171,10cyclictest0-21swapper/309:01:023
2755499170,15cyclictest0-21swapper/307:13:213
2753999178,6cyclictest2665-21/usr/sbin/munin07:25:200
2755499161,6cyclictest11541-21cpuspeed_turbos07:45:133
2755499161,5cyclictest9255-21df11:20:133
2755499161,2cyclictest0-21swapper/308:25:403
27554991612,3cyclictest0-21swapper/312:20:403
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional