You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-11 - 19:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Nov 11, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
64792239162,25sleep30-21swapper/307:05:053
84132238164,24sleep00-21swapper/007:09:360
84362236164,58sleep10-21swapper/107:09:531
80992204165,26sleep20-21swapper/207:05:312
876221350,2sleep10-21swapper/112:00:241
8554991091,106cyclictest0-21swapper/212:28:262
8554999188,2cyclictest0-21swapper/207:39:212
8554997167,3cyclictest0-21swapper/208:57:502
8554995855,2cyclictest0-21swapper/211:21:262
34732520,1sleep10-21swapper/108:10:011
210402510,1sleep00-21swapper/012:30:130
8554994138,2cyclictest0-21swapper/207:56:522
8554994138,2cyclictest0-21swapper/207:56:512
23552320,4sleep121-21rcuc/110:35:011
855899310,23cyclictest19006-21kworker/u8:009:58:413
855899291,25cyclictest22036-21kworker/u8:111:25:033
854799292,12cyclictest9881-21munin-run08:25:011
854799291,26cyclictest0-21swapper/109:58:201
8541992912,14cyclictest2362-21perf10:35:010
855899284,17cyclictest29677-21diskmemload11:44:483
855899284,17cyclictest29677-21diskmemload10:27:423
854799281,25cyclictest0-21swapper/111:53:261
854199285,15cyclictest31411-21unixbench_multi09:10:280
855899275,14cyclictest86550irq/126-enp1s0-10:56:003
8554992715,3cyclictest30-21rcuc/210:20:272
854799276,10cyclictest14634-21snmpd10:44:221
854799272,23cyclictest0-21swapper/108:42:361
854799271,17cyclictest0-21swapper/107:13:101
854799271,16cyclictest0-21swapper/110:35:291
854799271,16cyclictest0-21swapper/107:30:021
854199271,14cyclictest29691-21idleruntime-cro09:10:010
854199271,14cyclictest28113-21munin-run10:20:010
198272270,2sleep00-21swapper/012:25:240
159622270,3sleep0854199cyclictest12:17:210
855899269,4cyclictest0-21swapper/309:30:293
855899261,21cyclictest18741-21cut12:25:143
855899261,17cyclictest29677-21diskmemload10:47:393
855499264,4cyclictest768-21in:imjournal12:23:432
854799264,11cyclictest0-21swapper/111:21:261
854799262,11cyclictest23430-21irqstats07:40:211
854799261,2cyclictest0-21swapper/112:13:141
854799261,2cyclictest0-21swapper/112:13:141
854799261,2cyclictest0-21swapper/109:08:571
854799261,23cyclictest0-21swapper/111:37:481
854799261,20cyclictest0-21swapper/111:07:441
854799261,20cyclictest0-21swapper/111:07:441
854799261,17cyclictest0-21swapper/108:32:031
854199261,13cyclictest2979-21munin-run11:50:010
855899254,18cyclictest780-21rngd11:31:413
8558992514,9cyclictest14244-21cpu12:15:133
855899251,21cyclictest29677-21diskmemload11:08:163
855899251,21cyclictest29677-21diskmemload11:08:153
855899251,21cyclictest29677-21diskmemload10:39:183
854799259,3cyclictest14908-21awk09:50:011
8547992514,3cyclictest0-21swapper/112:35:281
8547992514,3cyclictest0-21swapper/110:10:001
854799251,2cyclictest15942-21/usr/sbin/munin09:50:191
854799251,2cyclictest0-21swapper/111:33:251
854799251,21cyclictest10381-21seq09:38:321
854199258,14cyclictest9603-21perf12:05:010
854199251,12cyclictest30945-21munin-run11:40:010
855899247,3cyclictest21084-21unixbench_singl10:00:293
855899242,19cyclictest24065-21ntp_states12:35:223
855899241,7cyclictest0-21swapper/309:40:463
855899241,21cyclictest0-21swapper/307:40:253
855899241,20cyclictest30293-21kworker/u8:210:32:443
855899241,20cyclictest29677-21diskmemload11:29:363
855899241,20cyclictest29677-21diskmemload10:50:303
855899241,20cyclictest29677-21diskmemload10:20:383
855899241,20cyclictest23289-21/usr/sbin/munin08:55:183
855899241,18cyclictest0-21swapper/311:12:003
855899241,11cyclictest20802-21/usr/sbin/munin12:30:163
855899241,10cyclictest11072-21sensors_temp12:05:243
8554992417,3cyclictest4536-21turbostat.cron10:40:012
8547992411,3cyclictest86750irq/128-enp1s0-11:58:571
854799241,11cyclictest23069-21munin-run08:55:011
854199248,3cyclictest4770-21/usr/sbin/munin10:40:210
854199246,3cyclictest21388-21kworker/0:310:12:170
854199241,6cyclictest9749-21perl10:50:170
8541992411,3cyclictest86750irq/128-enp1s0-10:37:180
855899238,3cyclictest13827-21/usr/sbin/munin07:20:173
855899234,10cyclictest86550irq/126-enp1s0-11:35:163
855899233,5cyclictest0-21swapper/311:58:273
8558992312,7cyclictest23378-21systemd-journal10:05:583
855899231,20cyclictest19006-21kworker/u8:011:00:053
855899231,20cyclictest11710-21kworker/u8:309:20:413
855899231,12cyclictest0-21swapper/311:51:403
855499234,8cyclictest86550irq/126-enp1s0-08:40:232
855499233,10cyclictest32-21ksoftirqd/211:25:202
855499231,10cyclictest11579-21grep08:25:282
854799239,3cyclictest29275-21grep07:55:151
854799239,3cyclictest29275-21grep07:55:151
854799232,3cyclictest0-21swapper/111:19:201
854799232,2cyclictest86750irq/128-enp1s0-10:47:431
854799231,4cyclictest2739-21interrupts09:20:181
854799231,3cyclictest30876-21df_inode10:25:151
854799231,3cyclictest16621-21expr12:20:141
854799231,2cyclictest0-21swapper/111:01:281
854799231,21cyclictest0-21swapper/109:43:191
854799231,16cyclictest0-21swapper/110:03:241
854799231,15cyclictest0-21swapper/110:20:211
854799231,11cyclictest28102-21latency_hist10:20:021
854799230,15cyclictest0-21swapper/108:00:331
109712230,4sleep339-21rcuc/308:25:213
855899227,3cyclictest31608-21sshd09:11:433
855899226,6cyclictest0-21swapper/307:55:263
855899226,6cyclictest0-21swapper/307:55:263
855899221,5cyclictest0-21swapper/310:10:183
855899221,19cyclictest3603-21kworker/u8:109:36:013
855899221,18cyclictest86550irq/126-enp1s0-12:20:163
855899221,12cyclictest19006-21kworker/u8:011:45:463
855899221,10cyclictest29677-21diskmemload12:04:253
855899221,10cyclictest29677-21diskmemload09:28:173
855899221,10cyclictest18219-21/usr/sbin/munin07:30:223
855899221,10cyclictest14634-21snmpd08:33:323
855899221,10cyclictest0-21swapper/310:45:033
855499221,7cyclictest22798-21ntp_states10:05:212
855499221,7cyclictest21227-21tr08:50:112
855499221,12cyclictest0-21swapper/211:40:282
855499221,11cyclictest8664-21if_enp2s008:20:182
855499221,11cyclictest15223-21sshd11:01:022
854799226,7cyclictest780-21rngd11:40:161
854799222,3cyclictest0-21swapper/108:49:051
854799222,3cyclictest0-21swapper/108:35:551
8547992213,3cyclictest0-21swapper/107:15:111
854799221,2cyclictest0-21swapper/109:26:391
854799221,19cyclictest0-21swapper/111:46:091
854799221,16cyclictest0-21swapper/108:19:591
854799221,15cyclictest0-21swapper/112:25:351
854799221,15cyclictest0-21swapper/107:51:351
854799221,15cyclictest0-21swapper/107:29:121
854799221,14cyclictest0-21swapper/107:20:081
854799221,10cyclictest0-21swapper/110:58:311
854199227,3cyclictest86750irq/128-enp1s0-09:25:180
854199226,8cyclictest0-21swapper/011:02:230
854199222,9cyclictest12068-21dump-pmu-power08:30:010
854199222,11cyclictest15806-21hddtemp_smartct09:50:180
854199221,7cyclictest16701-21/usr/sbin/munin08:40:160
855899217,4cyclictest21225-21grep08:50:113
855899217,3cyclictest21020-21grep07:35:163
8558992116,3cyclictest12592-21grep12:10:173
8558992116,3cyclictest12592-21grep12:10:173
855899211,2cyclictest0-21swapper/308:45:433
855899211,10cyclictest0-21swapper/309:15:133
855899210,4cyclictest0-21swapper/310:20:023
855499212,10cyclictest25000-21ntp_states10:10:232
855499211,7cyclictest0-21swapper/209:00:262
855499211,5cyclictest2970-21latency_hist11:50:022
855499211,11cyclictest24065-21ntp_states12:35:222
855499211,10cyclictest3282-21perf08:14:572
855499211,10cyclictest0-21swapper/211:51:572
855499211,10cyclictest0-21swapper/211:32:242
855499211,10cyclictest0-21swapper/209:15:032
8547992113,3cyclictest0-21swapper/112:32:201
854799211,18cyclictest0-21swapper/109:30:151
854799211,15cyclictest0-21swapper/110:10:581
854799211,14cyclictest0-21swapper/108:26:481
854799210,18cyclictest25643-21ntp_states11:25:241
854799210,15cyclictest0-21swapper/107:35:311
854199219,3cyclictest0-21swapper/007:33:440
854199214,8cyclictest5093-21users08:10:280
854199213,9cyclictest20171-21/usr/sbin/munin11:15:270
854199212,10cyclictest18320-21diskstats11:10:150
8541992117,3cyclictest0-21swapper/008:21:230
854199211,6cyclictest0-21swapper/009:15:480
854199211,12cyclictest0-21swapper/012:11:000
854199211,12cyclictest0-21swapper/012:11:000
218172210,2sleep30-21swapper/311:16:043
855899206,2cyclictest28872-21wc09:05:233
855899202,15cyclictest3461-21sort08:10:013
855899201,6cyclictest28121-21ntp_states07:50:223
855899201,12cyclictest0-21swapper/309:51:363
855499205,9cyclictest14005-40dnf08:31:472
855499205,7cyclictest0-21swapper/212:00:132
855499201,9cyclictest0-21swapper/210:48:012
855499201,9cyclictest0-21swapper/209:25:202
855499201,7cyclictest27727-21users10:15:302
855499201,6cyclictest15468-21df09:50:152
8554992015,3cyclictest32095-21gltestperf08:00:182
8554992014,3cyclictest29908-21acpi09:10:132
8554992013,4cyclictest17090-21sh07:25:222
855499201,12cyclictest0-21swapper/212:05:062
855499201,11cyclictest0-21swapper/207:45:192
855499201,10cyclictest4770-21/usr/sbin/munin10:40:192
854799202,17cyclictest0-21swapper/112:09:411
854799202,17cyclictest0-21swapper/109:11:191
854799202,17cyclictest0-21swapper/107:48:381
854799201,17cyclictest0-21swapper/109:15:051
854799201,17cyclictest0-21swapper/109:01:501
854799201,10cyclictest5480-21perf08:15:011
854799201,10cyclictest18156-21cstates11:10:141
854799201,10cyclictest0-21swapper/110:52:131
854199203,9cyclictest10878-21ntp_states12:05:210
854199203,2cyclictest3205-21/usr/sbin/munin11:50:170
854199202,10cyclictest1939-21/usr/sbin/munin09:20:180
854199201,8cyclictest24799-21unixbench_multi08:55:260
8541992018,1cyclictest0-21swapper/007:10:130
854199201,6cyclictest0-21swapper/009:47:510
854199201,6cyclictest0-21swapper/009:33:520
854199201,11cyclictest8537-21/usr/sbin/munin09:35:300
854199201,11cyclictest23002-21/usr/sbin/munin12:35:220
854199201,10cyclictest29597-21smart_sda10:20:250
854199201,10cyclictest26531-21latency_hist11:30:020
7412201,3sleep30-21swapper/308:01:543
855899195,3cyclictest86850irq/129-enp1s0-09:00:213
855899192,4cyclictest15762-21smart_sda08:35:243
855899191,6cyclictest17048-21diskstats08:40:143
855899191,5cyclictest13996-21ntp_states09:45:213
855499196,4cyclictest0-21swapper/211:05:262
855499196,4cyclictest0-21swapper/211:05:262
855499193,4cyclictest19521-21sadc10:00:092
855499192,14cyclictest13594-21cut09:45:162
855499191,5cyclictest11947-21perl10:55:182
855499191,4cyclictest0-21swapper/209:05:142
855499191,10cyclictest0-21swapper/211:13:112
855499191,10cyclictest0-21swapper/209:22:192
854799190,17cyclictest0-21swapper/112:15:361
854199195,3cyclictest86550irq/126-enp1s0-10:45:270
854199192,7cyclictest14634-21snmpd08:45:400
854199192,5cyclictest15772-21/usr/sbin/munin11:05:290
854199192,5cyclictest15772-21/usr/sbin/munin11:05:290
854199192,4cyclictest32122-21users10:25:280
854199191,6cyclictest13827-21/usr/sbin/munin07:20:130
8541991916,2cyclictest0-21swapper/007:40:250
854199191,5cyclictest0-21swapper/011:23:290
854199191,4cyclictest19529-21/usr/sbin/munin10:00:150
854199191,4cyclictest0-21swapper/011:41:090
8541991914,2cyclictest27001-21grep09:00:260
854199191,10cyclictest27058-21/usr/sbin/munin07:50:230
854199191,10cyclictest26768-21/usr/sbin/munin11:30:170
854199191,10cyclictest0-21swapper/010:55:250
854199191,10cyclictest0-21swapper/010:05:380
854199191,10cyclictest0-21swapper/009:44:150
854199191,10cyclictest0-21swapper/008:15:160
227852190,13sleep232-21ksoftirqd/212:35:012
163932191,3sleep00-21swapper/012:20:030
155062191,3sleep20-21swapper/207:21:262
855899182,13cyclictest4640-21tr08:10:243
855899181,6cyclictest17579-21sed07:25:273
855899181,5cyclictest8737-21/usr/sbin/munin07:10:253
855899181,4cyclictest11628-21/usr/sbin/munin07:15:163
8558991812,3cyclictest6769-21sh08:15:233
855499186,3cyclictest86550irq/126-enp1s0-11:15:142
855499185,3cyclictest0-21swapper/207:31:082
855499183,6cyclictest14501-21/usr/sbin/munin08:35:172
855499181,9cyclictest0-21swapper/212:16:412
855499181,9cyclictest0-21swapper/209:58:572
855499181,7cyclictest0-21swapper/211:35:222
855499181,7cyclictest0-21swapper/208:09:412
855499181,5cyclictest7664-21latency_hist08:20:032
855499181,4cyclictest0-21swapper/210:50:252
855499181,4cyclictest0-21swapper/210:25:142
8554991812,3cyclictest23893-21swap07:40:282
8554991811,4cyclictest5885-21fschecks_time11:55:162
855499181,11cyclictest0-21swapper/208:45:212
855499181,10cyclictest0-21swapper/209:34:022
854199187,3cyclictest1282-21/usr/sbin/munin08:05:220
854199183,9cyclictest0-21swapper/011:55:330
854199181,8cyclictest17242-21gunzip07:25:250
854199181,6cyclictest0-21swapper/008:01:230
854199181,5cyclictest21092-21/usr/sbin/munin08:50:160
854199181,5cyclictest12302-21/usr/sbin/munin08:30:120
8541991814,3cyclictest0-21swapper/008:35:140
8558991712,3cyclictest8197-21ls08:20:133
8558991712,3cyclictest25016-21cstates07:45:143
855499176,3cyclictest86550irq/126-enp1s0-10:32:432
855499171,9cyclictest0-21swapper/209:35:222
855499171,10cyclictest0-21swapper/209:42:552
8547991711,3cyclictest23289-21/usr/sbin/munin08:55:201
854199171,5cyclictest21330-21memory07:35:190
854199171,4cyclictest26822-21latency_hist07:50:010
854199171,2cyclictest0-21swapper/009:55:200
8541991712,3cyclictest13050-21wc07:15:390
103442171,2sleep20-21swapper/207:10:302
855499161,7cyclictest0-21swapper/212:10:192
855499161,7cyclictest0-21swapper/212:10:192
855499161,6cyclictest0-21swapper/207:18:112
855499161,4cyclictest27065-21acpi07:50:132
854199161,5cyclictest0-21swapper/007:55:170
854199161,5cyclictest0-21swapper/007:55:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional