You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 11:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Dec 04, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
314012233162,58sleep20-21swapper/219:05:382
317052228157,21sleep30-21swapper/319:09:333
313982228196,21sleep10-21swapper/119:05:381
315452209135,60sleep00-21swapper/019:07:300
629621630,4sleep03183399cyclictest19:20:250
31844991091,106cyclictest0-21swapper/221:00:572
154472590,5sleep33185199cyclictest22:10:183
154472590,5sleep33185199cyclictest22:10:183
135682490,1sleep20-21swapper/223:20:142
154492460,4sleep339-21rcuc/320:55:233
31833994010,14cyclictest24517-21munin-run20:05:010
3183399387,15cyclictest16080-21munin-run00:40:000
31833993518,9cyclictest7-21ksoftirqd/021:10:010
31833993518,9cyclictest7-21ksoftirqd/021:10:010
82902340,5sleep121-21rcuc/100:20:181
82902340,5sleep121-21rcuc/100:20:181
36612290,2sleep20-21swapper/220:30:122
3183399291,13cyclictest9768-21basename20:45:020
3183399283,12cyclictest26730-21awk20:10:010
3183799271,18cyclictest0-21swapper/123:30:161
3183799261,22cyclictest0-21swapper/100:38:491
3183799261,16cyclictest27419-21latency_hist21:25:011
3183399269,14cyclictest9483-21munin-run00:25:000
3183399269,14cyclictest9483-21munin-run00:25:000
3183399262,12cyclictest12608-21awk22:05:010
3185199259,3cyclictest9458-21sshd20:41:243
3184499256,6cyclictest13523-21idleruntime-cro19:40:022
3184499251,7cyclictest0-21swapper/200:09:532
3183799257,10cyclictest0-21swapper/121:00:201
3183799251,3cyclictest3187-21sort20:30:011
3183799251,17cyclictest29469-21gltestperf23:55:171
3183799251,16cyclictest0-21swapper/122:40:101
3183399250,11cyclictest5086-21munin-run00:15:000
31851992410,5cyclictest1993-21expr21:40:133
31851992410,12cyclictest86550irq/126-enp1s0-21:38:393
3184499249,4cyclictest30534-21if_enp2s021:30:182
31837992418,3cyclictest17564-21df_inode22:15:161
3183799241,19cyclictest14520-21ntp_states23:20:231
235162240,2sleep03183399cyclictest23:40:260
3185199237,12cyclictest0-21swapper/319:40:143
3184499234,7cyclictest1-21systemd19:14:222
31844992310,3cyclictest86850irq/129-enp1s0-23:16:362
31844992310,3cyclictest86850irq/129-enp1s0-23:16:362
3183799236,3cyclictest30131-21diskstats21:30:141
3183799234,10cyclictest86850irq/129-enp1s0-19:25:361
3183799232,18cyclictest9708-21cut23:10:171
3183799231,3cyclictest7284-21ntp_states21:50:221
3183799231,19cyclictest16589-21unixbench_singl22:10:281
3183799231,19cyclictest16589-21unixbench_singl22:10:281
3183799231,18cyclictest0-21swapper/121:55:261
3183799231,17cyclictest0-21swapper/100:09:561
3183799231,16cyclictest0-21swapper/121:46:551
3183799231,16cyclictest0-21swapper/121:16:541
31837992310,3cyclictest28620-21munin-run23:55:021
3183399231,11cyclictest1611-21awk21:40:020
3183399230,11cyclictest13242-21awk23:20:000
3183399230,11cyclictest13242-21awk23:20:000
3185199226,3cyclictest12050irq/123-ahci[0019:25:023
3185199222,10cyclictest1732-21grep22:50:263
3185199221,9cyclictest0-21swapper/323:51:543
3185199221,9cyclictest0-21swapper/300:35:573
3185199220,7cyclictest0-21swapper/321:15:183
3184499226,4cyclictest10934-21uptime19:30:272
3184499226,3cyclictest18761-21gltestperf19:50:152
3184499225,6cyclictest780-21rngd21:16:472
3184499224,4cyclictest24466-21cpu23:45:122
3184499221,2cyclictest0-21swapper/222:55:272
31837992218,3cyclictest0-21swapper/123:30:011
3183799221,3cyclictest0-21swapper/123:18:431
3183799221,3cyclictest0-21swapper/123:18:431
3183799221,18cyclictest0-21swapper/122:55:271
3183799221,16cyclictest0-21swapper/119:20:001
3183799221,15cyclictest0-21swapper/122:51:431
3183799221,15cyclictest0-21swapper/122:05:011
3183799221,15cyclictest0-21swapper/122:05:011
3183799221,15cyclictest0-21swapper/120:55:071
3183799221,15cyclictest0-21swapper/120:30:331
3183799221,15cyclictest0-21swapper/120:01:541
3183799221,15cyclictest0-21swapper/100:30:181
3183799221,14cyclictest11529-21chrt00:28:171
31837992211,3cyclictest840-21sensors_temp21:35:241
3183799220,8cyclictest0-21swapper/100:14:051
3183399221,12cyclictest20506-21taskset22:20:210
3183399221,12cyclictest0-21swapper/021:25:020
3185199212,10cyclictest26017-21/usr/sbin/munin22:35:203
3185199211,7cyclictest6401-21cstates21:50:143
31851992115,3cyclictest27679-21cpuspeed_turbos21:25:133
3185199211,11cyclictest0-21swapper/320:25:233
3185199211,10cyclictest3993-21head20:30:163
3185199211,10cyclictest0-21swapper/321:06:183
3185199211,10cyclictest0-21swapper/321:06:183
3185199210,5cyclictest0-21swapper/300:30:053
3184499219,5cyclictest22552-1kworker/u9:122:41:052
3184499218,3cyclictest3825-21sort21:45:022
3184499215,4cyclictest0-21swapper/200:00:192
3184499214,6cyclictest780-21rngd00:19:002
3184499212,5cyclictest9756-21munin-run20:45:022
3184499211,14cyclictest0-21swapper/223:59:312
3184499211,11cyclictest0-21swapper/220:24:502
3183799217,8cyclictest0-21swapper/122:00:201
3183799211,9cyclictest17591-21kworker/u8:123:36:491
3183799211,9cyclictest0-21swapper/122:35:231
3183799211,9cyclictest0-21swapper/121:10:201
3183799211,4cyclictest1-21systemd21:10:001
3183799211,4cyclictest1-21systemd21:10:001
31837992113,3cyclictest0-21swapper/120:53:291
31837992113,3cyclictest0-21swapper/120:41:381
3183799211,2cyclictest13695-21cat19:40:011
31837992112,3cyclictest0-21swapper/120:49:271
3183799211,17cyclictest14634-21snmpd20:19:381
3183799211,15cyclictest0-21swapper/123:09:311
3183799211,15cyclictest0-21swapper/122:45:271
3183799211,15cyclictest0-21swapper/122:27:381
3183799211,15cyclictest0-21swapper/120:36:011
3183799211,15cyclictest0-21swapper/119:13:031
3183799211,14cyclictest0-21swapper/100:15:251
3183799211,11cyclictest994-21cat20:25:011
3183799211,11cyclictest2503-21hddtemp_smartct21:40:161
31837992110,3cyclictest31757-21basename00:00:181
3183799210,2cyclictest0-21swapper/119:20:131
3183799210,15cyclictest0-21swapper/123:47:501
3183799210,15cyclictest0-21swapper/123:44:241
3183799210,15cyclictest0-21swapper/120:05:121
3183399216,6cyclictest0-21swapper/023:02:090
3183399215,6cyclictest0-21swapper/023:31:020
3183399212,10cyclictest6861-21/usr/sbin/munin23:05:160
3183399211,5cyclictest27989-21munin-run22:40:010
3183399211,10cyclictest0-21swapper/021:33:180
3183399210,15cyclictest0-21swapper/023:36:040
3185199209,3cyclictest31051-21/usr/sbin/munin00:00:273
3185199208,3cyclictest14634-21snmpd22:55:493
3185199208,3cyclictest0-21swapper/321:01:173
3185199202,11cyclictest0-21swapper/321:21:383
3185199201,8cyclictest21133-21NetworkManager22:21:513
3185199201,7cyclictest0-21swapper/323:33:243
31851992013,4cyclictest13052-21grep19:35:273
3185199201,15cyclictest31887-21uniq20:20:183
3185199201,10cyclictest8755-21sshd19:26:113
3184499206,3cyclictest20749-21diskmemload23:06:512
3184499205,3cyclictest24870-21ntp_offset22:30:212
3184499204,6cyclictest780-21rngd22:49:282
3184499203,8cyclictest3768-21cut19:15:242
3184499203,8cyclictest26283-21sed21:20:202
3184499203,8cyclictest22834-21sshd21:13:392
3184499202,9cyclictest5001-21diskstats23:00:152
3184499202,9cyclictest1190-21users21:35:272
3184499201,6cyclictest6874-21perf19:25:022
31844992013,4cyclictest8601-21grep19:25:242
3183799208,3cyclictest4988-21df23:00:151
31837992015,2cyclictest16338-21fschecks_count19:45:141
3183799201,15cyclictest0-21swapper/122:20:531
3183799201,12cyclictest0-21swapper/121:25:501
3183799200,3cyclictest0-21swapper/119:52:451
3183399208,2cyclictest1530-21kworker/0:219:28:500
3183399201,9cyclictest0-21swapper/022:47:390
31833992014,3cyclictest11505-21grep20:45:250
3183399201,3cyclictest86550irq/126-enp1s0-22:28:220
109542201,2sleep10-21swapper/119:30:311
3185199198,2cyclictest21042-21grep23:35:223
3185199196,3cyclictest31843-21latency_hist21:35:023
3185199194,3cyclictest0-21swapper/321:10:183
3185199191,9cyclictest9062-21/usr/sbin/munin23:10:163
3185199191,8cyclictest2042-21pmu-power00:05:263
3185199191,6cyclictest0-21swapper/322:15:063
3185199191,4cyclictest13457-21/usr/sbin/munin23:20:153
31851991914,3cyclictest10630-21wc00:25:213
3185199191,11cyclictest0-21swapper/319:16:003
3185199191,10cyclictest0-21swapper/323:40:173
3185199191,10cyclictest0-21swapper/320:45:183
3184499196,3cyclictest86850irq/129-enp1s0-22:05:212
3184499196,3cyclictest86850irq/129-enp1s0-22:05:212
3184499195,6cyclictest780-21rngd00:26:072
3184499195,3cyclictest86550irq/126-enp1s0-20:05:182
3184499194,6cyclictest780-21rngd22:04:492
3184499194,4cyclictest23326-21ntpq23:40:242
3184499194,3cyclictest86550irq/126-enp1s0-23:50:172
3184499193,7cyclictest17830-21chrt19:49:092
3184499193,5cyclictest2864-21sshd20:26:202
3184499193,5cyclictest16722-21sshd22:11:422
3184499193,5cyclictest16722-21sshd22:11:422
3184499192,5cyclictest505-21df_inode22:50:142
3184499192,5cyclictest21965-21diskstats22:25:152
3184499192,5cyclictest20756-21grep21:10:012
3184499192,5cyclictest20756-21grep21:10:012
3184499192,5cyclictest17572-21diskstats22:15:182
3184499191,9cyclictest27766-21cstates21:25:142
3184499191,9cyclictest26017-21/usr/sbin/munin22:35:252
3184499191,7cyclictest0-21swapper/200:12:052
3184499191,6cyclictest22533-21/usr/sbin/munin20:00:222
3184499191,5cyclictest16000-21diskstats23:25:142
3184499191,5cyclictest11531-21grep20:45:252
3184499191,3cyclictest20334-21cstates23:35:162
3184499191,11cyclictest0-21swapper/220:51:162
3184499190,1cyclictest0-21swapper/223:30:262
3183799196,3cyclictest0-21swapper/122:30:201
31837991915,3cyclictest0-21swapper/119:57:511
31837991915,2cyclictest287522sleep120:13:011
3183799191,17cyclictest0-21swapper/119:44:481
3183399198,5cyclictest22552-1kworker/u9:123:25:280
3183399198,3cyclictest23818-21/usr/sbin/munin22:30:190
3183399196,4cyclictest0-21swapper/021:49:210
3183399196,4cyclictest0-21swapper/000:00:090
3183399195,6cyclictest780-21rngd21:40:150
3183399194,9cyclictest0-21swapper/023:45:470
3183399193,10cyclictest0-21swapper/023:50:160
3183399191,5cyclictest0-21swapper/023:55:120
31833991914,3cyclictest7-21ksoftirqd/000:05:250
3183399191,10cyclictest2455-21/usr/sbin/munin22:55:140
3183399191,10cyclictest15025-21/usr/sbin/munin22:10:250
3183399191,10cyclictest15025-21/usr/sbin/munin22:10:240
3183399191,10cyclictest0-21swapper/000:31:270
3183399190,8cyclictest0-21swapper/000:15:180
151362191,2sleep00-21swapper/023:21:280
3185199189,3cyclictest0-21swapper/322:46:403
3185199184,3cyclictest836-21runrttasks23:00:193
3185199183,12cyclictest39-21rcuc/322:40:253
3185199181,7cyclictest32753-21if_enp1s019:10:163
3185199181,6cyclictest0-21swapper/323:57:083
3185199181,6cyclictest0-21swapper/323:45:163
3185199181,6cyclictest0-21swapper/323:06:303
3185199181,6cyclictest0-21swapper/321:57:013
3185199181,6cyclictest0-21swapper/300:21:483
3185199181,6cyclictest0-21swapper/300:21:483
3185199181,5cyclictest25408-21users22:30:273
3185199181,5cyclictest0-21swapper/323:28:103
3185199181,4cyclictest5810-21cut00:15:183
31851991813,3cyclictest17441-21grep19:45:253
3185199181,14cyclictest25352-21gltestperf20:05:183
31851991811,4cyclictest22760-21grep22:25:233
3185199181,10cyclictest0-21swapper/322:00:263
3184499187,3cyclictest86550irq/126-enp1s0-00:30:222
3184499184,3cyclictest9395-21uname21:55:192
3184499183,3cyclictest22552-1kworker/u9:100:21:512
3184499183,3cyclictest22552-1kworker/u9:100:21:512
3184499182,7cyclictest14376-21/usr/sbin/munin20:55:212
3184499181,9cyclictest836-21runrttasks00:37:512
3184499181,5cyclictest9409-21diskstats23:10:152
3184499181,4cyclictest7679-21cat21:50:272
3184499181,4cyclictest13742-21/usr/sbin/munin19:40:112
31844991812,3cyclictest5092-21ntpq21:45:232
3183399187,2cyclictest24911-21kworker/0:221:17:140
3183399186,4cyclictest0-21swapper/000:25:040
3183399185,2cyclictest86550irq/126-enp1s0-21:10:250
3183399184,6cyclictest0-21swapper/021:56:170
3183399184,3cyclictest86650irq/127-enp1s0-19:40:020
3183399182,15cyclictest0-21swapper/023:13:170
3183399181,8cyclictest28213-21/usr/sbin/munin22:40:220
31833991814,3cyclictest7-21ksoftirqd/019:15:010
3183399181,3cyclictest21143-21irqstats19:55:190
3183399181,3cyclictest0-21swapper/019:41:220
31833991812,3cyclictest18244-21tr19:50:130
31833991811,4cyclictest2566-21sort20:25:240
3183399181,12cyclictest0-21swapper/021:51:140
3183399181,11cyclictest0-21swapper/022:07:140
3183399181,11cyclictest0-21swapper/022:07:140
3183399181,10cyclictest0-21swapper/022:19:170
285792181,2sleep20-21swapper/220:10:502
270152181,2sleep00-21swapper/021:20:500
5022171,2sleep00-21swapper/020:20:330
3185199172,7cyclictest41-21ksoftirqd/320:35:193
3185199172,4cyclictest0-21swapper/323:18:173
3185199172,4cyclictest0-21swapper/323:18:173
3185199172,12cyclictest5519-21grep21:45:273
3185199171,7cyclictest12824-21/usr/sbin/munin22:05:213
3185199171,7cyclictest12824-21/usr/sbin/munin22:05:213
3185199171,15cyclictest0-21swapper/319:59:353
31851991711,3cyclictest14634-21snmpd20:10:203
3184499175,2cyclictest86550irq/126-enp1s0-22:20:422
3184499171,2cyclictest0-21swapper/220:15:142
31844991711,4cyclictest6150-21cut20:35:162
3184499171,14cyclictest0-21swapper/219:55:102
31833991713,3cyclictest0-21swapper/019:46:240
31833991711,3cyclictest32622-21cat22:50:110
3183399171,10cyclictest0-21swapper/020:50:150
3183399171,10cyclictest0-21swapper/020:35:160
181782171,3sleep00-21swapper/021:00:310
111232171,2sleep00-21swapper/019:32:400
3185199161,4cyclictest12197-21egrep20:50:133
3185199161,14cyclictest0-21swapper/319:33:163
31851991611,3cyclictest3230-21expr00:10:143
31851991611,3cyclictest18545-21cut19:50:153
3185199161,12cyclictest29159-21cpu20:15:123
31851991610,3cyclictest22620-21cpuspeed_turbos20:00:143
3183399165,3cyclictest86750irq/128-enp1s0-20:15:110
3183399164,2cyclictest22070-21kworker/0:220:12:180
3183399161,6cyclictest15368-21cut20:55:210
3183399161,5cyclictest3547-21cpuspeed_turbos20:30:130
3183399159,3cyclictest2021-21timerandwakeup19:15:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional