You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-21 - 09:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Oct 21, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
279522241163,25sleep00-21swapper/019:08:590
279502238161,23sleep30-21swapper/319:08:573
276782217137,20sleep10-21swapper/119:05:301
277752202164,25sleep20-21swapper/219:06:432
28133991171,114cyclictest0-21swapper/200:01:482
281339910197,3cyclictest0-21swapper/221:46:072
28133999693,2cyclictest0-21swapper/223:11:412
28133997268,2cyclictest0-21swapper/222:05:172
28133995645,10cyclictest0-21swapper/220:52:332
28139994013,14cyclictest13194-21kernelversion23:30:193
49462380,4sleep339-21rcuc/322:00:023
28139993729,5cyclictest30325-21df00:10:163
28139993710,13cyclictest9609-21latency_hist19:40:023
28139993611,15cyclictest7480-21timerwakeupswit20:45:263
28139993516,9cyclictest24376-21/usr/sbin/munin23:55:203
28139993515,11cyclictest8616-21/usr/sbin/munin00:35:193
28139993512,14cyclictest31955-21ntp_states21:45:223
28139993510,13cyclictest26918-21latency_hist22:50:013
28122993510,12cyclictest7108-21munin-run22:05:010
2813999344,18cyclictest780-21rngd00:09:043
28139993415,15cyclictest15369-21fschecks_time21:05:173
28139993413,12cyclictest26570-21netstat00:00:223
28139993412,19cyclictest4381-21tr00:25:163
28139993411,10cyclictest20768-21runrttasks23:48:113
28139993410,12cyclictest0-21swapper/323:00:143
28133993418,7cyclictest0-21swapper/200:31:132
2812299347,12cyclictest16543-21perf23:40:020
2813999339,15cyclictest21973-21pmu-power20:05:243
2813999338,15cyclictest16618-21fschecks_time22:25:203
28139993314,10cyclictest19278-21/usr/sbin/munin21:15:263
28139993311,13cyclictest14556-21/usr/sbin/munin23:35:283
28139993310,9cyclictest2180-21irqstats23:05:203
28139993310,10cyclictest14845-21missed_timers22:20:203
28139993310,10cyclictest13147-21timerwakeupswit22:15:263
2812299339,11cyclictest19049-21munin-run21:15:000
2812299334,13cyclictest18095-21munin-run22:30:000
28122993314,16cyclictest27550-21munin-run00:05:010
322312320,3sleep07-21ksoftirqd/021:45:250
2813999329,14cyclictest7785-21/usr/sbin/munin00:30:253
2813999328,10cyclictest0-21swapper/323:10:193
2813999327,14cyclictest2616-21gltestperf00:20:193
28139993215,13cyclictest0-21swapper/321:25:013
28139993212,11cyclictest16758-21/usr/sbin/munin23:40:263
28139993212,11cyclictest16758-21/usr/sbin/munin23:40:253
28139993210,11cyclictest17500-21sort21:10:163
187042320,4sleep02812299cyclictest20:00:120
2813999318,12cyclictest22719-21/usr/sbin/munin22:40:183
2813999312,16cyclictest780-21rngd20:44:323
28139993116,6cyclictest27465-21diskstats22:50:143
28139993111,6cyclictest28321-21/usr/sbin/munin19:10:133
28139993110,8cyclictest8400-21ntp_states22:05:243
2813999308,19cyclictest10266-21tr23:25:183
28139993021,5cyclictest9025-21ntp_states23:20:253
28139993017,10cyclictest29672-21diskstats22:55:173
28139993013,14cyclictest14634-21snmpd21:35:203
28139993011,7cyclictest11208-21runrttasks22:11:283
28139993010,17cyclictest8275-21/usr/sbin/munin20:50:273
28139993010,11cyclictest21585-21/usr/sbin/munin22:35:253
28139993010,11cyclictest21585-21/usr/sbin/munin22:35:243
2813999299,11cyclictest0-21swapper/320:02:323
2813999296,10cyclictest0-21swapper/319:25:123
2813999294,10cyclictest780-21rngd19:23:483
2813999294,10cyclictest780-21rngd19:23:483
28139992920,5cyclictest3362-40dnf20:36:313
28139992920,5cyclictest10818-21diskstats20:55:163
28139992916,6cyclictest2316-21users21:50:273
28139992913,12cyclictest23041-21/usr/sbin/munin20:10:203
28139992910,16cyclictest5759-21/usr/sbin/munin23:15:173
28139992910,12cyclictest25293-21sshd21:25:423
2813999289,7cyclictest29763-21ntpq21:40:203
2813999289,5cyclictest86750irq/128-enp1s0-19:40:143
2813999288,4cyclictest0-21swapper/320:17:163
2813999287,11cyclictest0-21swapper/323:50:173
2813999286,9cyclictest0-21swapper/322:00:243
2813999285,10cyclictest12043-21/usr/sbin/munin19:45:213
2813999283,10cyclictest0-21swapper/322:30:243
28139992821,4cyclictest1089-21taskset20:30:493
28139992820,5cyclictest32473-21smart_sda19:15:263
2813999281,10cyclictest27095-21gunzip21:30:263
2813999281,10cyclictest27095-21gunzip21:30:263
28139992810,10cyclictest780-21rngd20:28:443
2812799280,22cyclictest0-21swapper/123:31:211
2813999277,9cyclictest32242-21cpu00:15:133
2813999277,17cyclictest28513-21ntp_states20:20:253
28139992719,5cyclictest14586-21diskstats19:50:133
28139992719,5cyclictest13018-21diskstats21:00:153
197582270,4sleep230-21rcuc/220:00:222
28139992617,6cyclictest5450-21/usr/sbin/munin19:30:263
2812799261,17cyclictest11212-21if_enp2s020:55:181
2812799261,14cyclictest0-21swapper/120:36:321
2812299269,15cyclictest0-21swapper/023:10:020
2812299262,13cyclictest8394-21munin-run00:35:000
2812299260,12cyclictest0-21swapper/020:45:010
278882260,5sleep121-21rcuc/100:05:121
115062261,3sleep211533-21ntp_states20:55:222
28139992517,5cyclictest16442-21/usr/sbin/munin19:55:153
28127992512,6cyclictest13361-21kworker/u8:022:25:571
2812299251,11cyclictest20939-21awk23:50:020
15382251,3sleep11729-21hddtemp_smartct19:20:191
15382251,3sleep11729-21hddtemp_smartct19:20:191
2813399241,10cyclictest0-21swapper/223:29:052
2812799246,10cyclictest0-21swapper/122:12:531
2812799242,19cyclictest1714-21diskstats23:05:161
279622240,5sleep232-21ksoftirqd/222:50:182
2812799239,12cyclictest1492-21kworker/u8:100:38:171
2812799239,11cyclictest10344-21cstates23:25:171
2812799232,11cyclictest25325-21munin-run00:00:011
2812799232,11cyclictest16855-21munin-run21:10:011
28127992314,3cyclictest0-21swapper/121:00:231
2812799231,3cyclictest0-21swapper/100:28:421
2812799231,19cyclictest5420-21sed19:30:011
2812799231,19cyclictest21336-21/usr/sbin/munin22:35:211
2812799231,19cyclictest21336-21/usr/sbin/munin22:35:211
2812799231,19cyclictest21179-21df20:05:141
2812799231,19cyclictest15044-21cut22:20:241
2812799231,16cyclictest0-21swapper/120:30:321
2812799230,18cyclictest0-21swapper/100:31:471
2812799230,17cyclictest0-21swapper/121:30:291
2812799230,17cyclictest0-21swapper/121:30:291
2812299231,11cyclictest27212-21munin-run20:20:000
20762230,2sleep232-21ksoftirqd/221:50:252
40312220,2sleep20-21swapper/221:55:272
2813399228,4cyclictest8633-21needreboot19:35:202
2813399221,8cyclictest22252expr00:20:152
2813399221,4cyclictest17357-21cstates21:10:132
2812799221,3cyclictest5382-21taskset23:13:131
2812799221,3cyclictest31400-21wc19:15:131
2812799221,18cyclictest26525-21sed00:00:201
2812799221,18cyclictest0-21swapper/120:00:011
2812799221,15cyclictest0-21swapper/121:38:521
2812299221,10cyclictest0-21swapper/021:17:480
29922211,3sleep00-21swapper/019:24:490
29922211,3sleep00-21swapper/019:24:480
2813399216,5cyclictest0-21swapper/222:35:272
2813399216,5cyclictest0-21swapper/222:35:272
2813399212,6cyclictest7977-21acpi23:20:152
2813399211,6cyclictest0-21swapper/223:42:022
2813399211,6cyclictest0-21swapper/223:42:022
2812799219,6cyclictest5047-21kworker/u8:119:40:101
2812799216,6cyclictest0-21swapper/123:03:301
2812799214,5cyclictest9109-21unixbench-2d19:35:261
2812799213,6cyclictest16316-40dnf23:37:321
28127992117,3cyclictest0-21swapper/100:15:371
28127992113,3cyclictest0-21swapper/121:20:531
2812799211,18cyclictest0-21swapper/120:25:251
2812799211,15cyclictest0-21swapper/122:15:521
2812799211,15cyclictest0-21swapper/122:01:031
2812799211,15cyclictest0-21swapper/121:52:421
2812799211,15cyclictest0-21swapper/121:49:501
2812799211,15cyclictest0-21swapper/121:42:181
2812799211,15cyclictest0-21swapper/121:27:591
2812799211,15cyclictest0-21swapper/121:18:061
2812799211,15cyclictest0-21swapper/120:16:141
2812799211,11cyclictest4896-21munin-run22:00:011
2812799211,11cyclictest29090-21latency_hist22:55:011
2812299217,5cyclictest0-21swapper/022:17:360
2812299216,6cyclictest0-21swapper/000:08:230
2812299214,10cyclictest768-21in:imjournal19:30:200
2812299212,3cyclictest32235-21idleruntime23:00:200
2812299211,11cyclictest29390-21ntp_states19:10:210
2812299211,10cyclictest0-21swapper/023:58:500
77262201,2sleep10-21swapper/120:46:061
311482201,2sleep10-21swapper/122:58:211
2813399209,2cyclictest24589-21/usr/sbin/munin23:55:232
2813399204,7cyclictest16136-21uptime23:35:272
2813399202,5cyclictest12860-21kworker/2:120:32:432
2813399201,7cyclictest24517-21grep21:25:202
2813399201,6cyclictest0-21swapper/223:03:252
2813399201,6cyclictest0-21swapper/222:17:242
2813399201,6cyclictest0-21swapper/221:16:562
2812799209,3cyclictest3283-21sensors_temp00:20:261
2812799208,3cyclictest30580-21grep00:10:171
28127992013,2cyclictest0-21swapper/121:12:041
2812799201,18cyclictest0-21swapper/123:53:491
2812799201,17cyclictest0-21swapper/123:47:301
2812799201,17cyclictest0-21swapper/120:04:261
2812799201,11cyclictest7410-21latency_hist19:35:011
2812799201,10cyclictest26650-21chrt22:47:081
2812799201,10cyclictest20385-21cat22:35:011
2812799200,5cyclictest0-21swapper/123:21:441
2812799200,18cyclictest16856-21diskmemload23:17:031
2812799200,18cyclictest0-21swapper/123:40:411
2812799200,18cyclictest0-21swapper/123:40:411
2812799200,14cyclictest0-21swapper/122:05:071
2812299209,3cyclictest23670-21/usr/sbin/munin21:25:180
2812299209,3cyclictest14879-21/usr/sbin/munin21:05:240
2812299207,2cyclictest14343-21perf23:35:010
2812299206,5cyclictest0-21swapper/000:20:080
2812299205,5cyclictest0-21swapper/023:15:310
2812299204,6cyclictest0-21swapper/021:54:280
2812299203,2cyclictest0-21swapper/000:16:350
2812299201,6cyclictest0-21swapper/023:27:420
2812299201,3cyclictest112072sleep022:11:270
28122992013,4cyclictest32373-21grep20:30:180
2812299201,11cyclictest7962-21/usr/sbin/munin23:20:210
2812299201,11cyclictest21472-21/usr/sbin/munin21:20:140
2812299201,11cyclictest14634-21snmpd21:00:120
28122992010,3cyclictest24087-21timerandwakeup22:40:270
2812299200,11cyclictest0-21swapper/023:42:310
2812299200,11cyclictest0-21swapper/023:42:310
143872201,2sleep20-21swapper/221:01:482
290102191,2sleep00-21swapper/022:54:070
2813399192,7cyclictest0-21swapper/220:35:182
2813399191,7cyclictest0-21swapper/222:49:422
2813399191,6cyclictest0-21swapper/223:52:212
2813399191,6cyclictest0-21swapper/220:29:472
28133991913,3cyclictest13928-21sed23:30:262
28133991912,4cyclictest27861-21grep20:20:182
28127991915,3cyclictest0-21swapper/120:50:191
2812799191,2cyclictest0-21swapper/119:47:261
28127991912,2cyclictest0-21swapper/120:41:421
2812799191,10cyclictest23323-21head22:40:161
2812799190,2cyclictest0-21swapper/120:22:191
2812799190,18cyclictest0-21swapper/119:13:541
2812299199,2cyclictest0-21swapper/022:05:200
2812299198,9cyclictest28473-21turbostat21:40:010
2812299196,5cyclictest0-21swapper/000:26:570
2812299196,4cyclictest0-21swapper/022:46:560
2812299196,3cyclictest0-21swapper/000:12:450
2812299195,6cyclictest0-21swapper/023:51:300
2812299191,6cyclictest0-21swapper/022:36:390
2812299191,6cyclictest0-21swapper/022:36:390
2812299191,5cyclictest0-21swapper/021:32:110
2812299191,5cyclictest0-21swapper/021:32:110
2812299191,10cyclictest29646-21/usr/sbin/munin20:25:210
2812299191,10cyclictest2930-21/usr/sbin/munin21:55:190
2812299191,10cyclictest1679-21/usr/sbin/munin20:35:250
2812299191,10cyclictest14634-21snmpd20:55:100
2812299191,10cyclictest0-21swapper/022:55:580
251902190,2sleep00-21swapper/020:15:010
207592191,3sleep20-21swapper/223:48:042
140302191,2sleep10-21swapper/119:50:021
57132181,2sleep20-21swapper/220:43:262
2813399184,4cyclictest6681-21sed22:00:282
2813399182,7cyclictest27432-21sed21:30:292
2813399182,7cyclictest27432-21sed21:30:292
2813399182,7cyclictest16906-21chrt22:25:212
2813399181,7cyclictest0-21swapper/221:23:442
2813399181,3cyclictest2123-21ntp_states19:20:252
2813399181,3cyclictest2123-21ntp_states19:20:252
28133991811,4cyclictest13302-21sensors_temp19:45:282
2813399180,3cyclictest0-21swapper/220:15:252
2812799186,4cyclictest86750irq/128-enp1s0-20:10:251
2812299185,5cyclictest0-21swapper/021:44:470
2812299185,3cyclictest0-21swapper/000:37:350
2812299181,9cyclictest16856-21diskmemload22:22:150
2812299181,9cyclictest14634-21snmpd19:39:550
2812299181,6cyclictest4245-21hddtemp_smartct23:10:180
28122991811,4cyclictest32250-21ntp_kernel_pll_19:15:210
28122991811,4cyclictest15306-21ntpq19:50:190
2812299181,10cyclictest0-21swapper/022:30:570
115002181,2sleep00-21swapper/019:41:100
103882181,2sleep20-21swapper/200:37:412
58912171,3sleep20-21swapper/200:26:212
306222171,2sleep20-21swapper/221:44:402
2813399175,2cyclictest86550irq/126-enp1s0-19:25:172
2813399172,6cyclictest0-21swapper/222:12:422
2813399171,9cyclictest0-21swapper/222:20:402
2813399171,9cyclictest0-21swapper/220:05:402
2813399171,9cyclictest0-21swapper/200:15:032
2813399171,6cyclictest28107-21sshd21:35:412
2813399171,6cyclictest0-21swapper/223:06:142
2813399171,6cyclictest0-21swapper/222:41:352
2813399171,4cyclictest0-21swapper/222:30:222
2813399171,4cyclictest0-21swapper/221:05:482
28133991712,3cyclictest6265-21ls20:45:122
28133991711,3cyclictest23266-21expr20:10:122
2813399171,10cyclictest0-21swapper/219:55:092
2813399171,10cyclictest0-21swapper/219:54:182
2812299171,9cyclictest2274-21kworker/u8:020:53:140
2812299171,7cyclictest3246-21/usr/sbin/munin19:25:170
28122991712,3cyclictest6075-21/usr/sbin/munin20:45:200
28122991712,3cyclictest12714-21expr19:45:200
28122991711,3cyclictest16548-21expr19:55:150
2813399162,6cyclictest29325-21/usr/sbin/munin22:55:192
2813399161,9cyclictest0-21swapper/219:16:292
28133991612,3cyclictest0-21swapper/223:17:202
28133991612,3cyclictest0-21swapper/200:10:352
2813399161,13cyclictest6327-21latency19:30:212
28133991611,3cyclictest28487-21cstates19:10:132
28133991611,3cyclictest10808-21sed19:40:182
28133991611,2cyclictest28383-21gltestperf00:05:152
28122991611,3cyclictest29445-21sort20:25:010
2812299159,4cyclictest0-21swapper/020:05:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional