You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-02 - 02:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sun Nov 02, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
101992240162,25sleep00-21swapper/019:08:220
102972239163,23sleep20-21swapper/219:09:372
102892236163,59sleep10-21swapper/119:09:321
100342234162,59sleep30-21swapper/319:06:143
1203021600,3sleep21042999cyclictest00:05:162
2963621570,4sleep21042999cyclictest22:16:102
1081221550,4sleep21042999cyclictest20:20:242
1042999107104,2cyclictest0-21swapper/221:42:592
223092640,2sleep00-21swapper/023:15:130
121892520,2sleep20-21swapper/220:25:142
52022320,4sleep230-21rcuc/220:10:002
1042099311,14cyclictest14539-21idleruntime-cro21:45:020
1042499304,10cyclictest0-21swapper/123:35:001
78962280,4sleep1848-21sssd_nss23:55:151
1042099281,12cyclictest13925-21awk20:30:010
1042099272,15cyclictest0-21swapper/022:25:150
1042099271,13cyclictest32153-21munin-run22:25:010
1042099271,13cyclictest32153-21munin-run22:25:010
1043499262,11cyclictest23579-21/usr/sbin/munin22:05:213
1043499261,22cyclictest784-21NetworkManager00:13:133
1042499261,4cyclictest17043-21cpuspeed_turbos21:50:131
1042499261,17cyclictest0-21swapper/122:44:331
1042499260,22cyclictest837-21unixbench_multi21:10:281
1043499251,21cyclictest576-21switchtime21:10:263
1043499251,12cyclictest14634-21snmpd20:50:213
1042499257,9cyclictest0-21swapper/121:25:131
10424992514,8cyclictest0-21swapper/123:22:541
10424992514,3cyclictest836-21runrttasks19:24:061
1042499250,23cyclictest0-21swapper/121:18:271
1043499248,10cyclictest0-21swapper/320:11:373
1043499246,6cyclictest32376-21/usr/sbin/munin22:25:243
1043499241,21cyclictest401ktimersoftd/323:21:583
1043499241,20cyclictest14634-21snmpd21:55:323
1043499241,11cyclictest12573-21/usr/sbin/munin21:40:183
1043499241,10cyclictest19189-21memory00:20:213
1043499240,4cyclictest28116-21kworker/u8:123:29:053
1042999246,7cyclictest17346-21fschecks_count21:50:162
1042499241,16cyclictest0-21swapper/123:27:061
106702230,12sleep339-21rcuc/321:35:173
1043499236,4cyclictest5083-21switchtime21:20:263
1043499232,10cyclictest30811-21/usr/sbin/munin23:35:203
1043499231,5cyclictest581-21systemd-journal22:15:013
1043499231,19cyclictest14634-21snmpd20:45:203
1043499231,11cyclictest20472-21/usr/sbin/munin00:25:183
10434992310,3cyclictest86850irq/129-enp1s0-23:00:123
1043499230,2cyclictest0-21swapper/320:55:263
10424992318,2cyclictest6375-21turbostat.cron22:40:001
1042499231,7cyclictest24867-21/usr/sbin/munin00:35:221
10424992316,4cyclictest13644-21latency_hist00:10:021
1042499231,3cyclictest25568-21ntp_states19:40:241
1042499231,19cyclictest14634-21snmpd22:45:391
1042099236,3cyclictest28902-21wc22:15:210
42382221,2sleep30-21swapper/320:05:263
1043499227,3cyclictest4414-21/usr/sbin/munin22:35:163
1043499227,12cyclictest24867-21/usr/sbin/munin00:35:223
1043499226,4cyclictest965-21df23:40:143
1043499225,4cyclictest0-21swapper/322:20:263
1043499225,4cyclictest0-21swapper/322:20:263
1043499224,7cyclictest18875-21cat19:25:213
1043499221,3cyclictest30455-21grep19:50:273
1043499221,10cyclictest27335-21/usr/sbin/munin21:00:173
1043499221,10cyclictest13214-21/usr/sbin/munin22:55:163
1042999222,5cyclictest86850irq/129-enp1s0-21:47:342
1042499227,9cyclictest0-21swapper/120:25:221
1042499227,3cyclictest0-21swapper/122:15:201
1042499225,10cyclictest0-21swapper/123:54:181
1042499224,10cyclictest0-21swapper/122:03:291
1042499224,10cyclictest0-21swapper/121:44:381
1042499223,10cyclictest0-21swapper/122:34:261
1042499222,16cyclictest11188-21irqstats21:35:211
1042499221,7cyclictest19344-21ntp_states00:20:231
1042499221,6cyclictest31098-21/usr/sbin/munin19:55:151
10424992216,3cyclictest16327-21memory23:00:201
1042499221,19cyclictest0-21swapper/120:15:201
1042499221,19cyclictest0-21swapper/120:15:201
1042499221,19cyclictest0-21swapper/100:12:461
1042499221,18cyclictest4421-21awk20:05:271
1042499221,16cyclictest0-21swapper/121:30:231
1042499221,15cyclictest0-21swapper/119:15:451
1042499221,11cyclictest2851-21acpi23:45:111
1042499221,11cyclictest20594-21tr00:25:141
10424992210,3cyclictest308-21latency_hist23:40:011
1042499220,16cyclictest0-21swapper/100:00:111
1042099226,6cyclictest0-21swapper/000:09:560
1043499218,3cyclictest13515-21/usr/sbin/munin19:15:283
1043499213,4cyclictest16531-40dnf21:47:343
1043499212,3cyclictest12050irq/123-ahci[0023:35:013
1043499211,11cyclictest0-21swapper/321:25:273
1043499211,10cyclictest17779-21taskset00:16:463
1043499211,10cyclictest17620-21/usr/sbin/munin23:05:193
1042999212,11cyclictest19344-21ntp_states00:20:242
1042999211,8cyclictest0-21swapper/223:15:272
10429992115,3cyclictest19833-21gltestperf21:55:202
1042499218,3cyclictest0-21swapper/123:06:271
1042499216,9cyclictest0-21swapper/123:12:131
1042499215,5cyclictest23781-21sshd23:17:331
1042499215,3cyclictest13287-21cpuspeed_turbos22:55:131
10424992117,3cyclictest0-21swapper/122:20:271
10424992117,3cyclictest0-21swapper/122:20:271
1042499211,6cyclictest3911-21expr21:20:141
1042499211,18cyclictest0-21swapper/122:54:501
1042499211,18cyclictest0-21swapper/122:27:211
1042499211,18cyclictest0-21swapper/121:45:551
1042499211,18cyclictest0-21swapper/120:55:241
1042499211,18cyclictest0-21swapper/120:45:531
1042499211,18cyclictest0-21swapper/120:35:141
1042499211,18cyclictest0-21swapper/120:31:161
1042499211,18cyclictest0-21swapper/119:31:401
1042499211,15cyclictest0-21swapper/120:13:571
1042499211,13cyclictest0-21swapper/100:16:451
1042499211,11cyclictest12626-21munin-node19:15:011
1042499211,10cyclictest2070-21pmu-power20:00:251
1042499210,15cyclictest0-21swapper/100:30:031
1042499210,15cyclictest0-21swapper/100:30:031
1042099215,4cyclictest16116-21munin-run20:35:010
1042099212,8cyclictest0-21swapper/000:27:410
1042099210,12cyclictest0-21swapper/000:23:490
118372201,2sleep20-21swapper/219:10:252
1043499206,3cyclictest22300-21/usr/sbin/munin19:35:223
1043499205,3cyclictest5750irq/9-acpi21:50:163
1043499203,4cyclictest9450-21/usr/sbin/munin00:00:123
1043499201,9cyclictest0-21swapper/320:26:523
1043499200,11cyclictest0-21swapper/322:41:323
1042999202,7cyclictest22710-21munin-run20:50:012
1042999201,13cyclictest0-21swapper/219:35:172
1042499205,3cyclictest24644-21ntp_states22:05:231
10424992017,2cyclictest0-21swapper/119:35:401
1042499201,5cyclictest29711-21irqstats19:50:191
10424992015,2cyclictest20162-21strings21:55:241
1042499201,18cyclictest0-21swapper/120:50:041
1042499201,17cyclictest0-21swapper/122:11:081
1042499201,13cyclictest0-21swapper/123:43:091
1042499201,11cyclictest11043-21switchtime20:20:281
1042499201,10cyclictest19666-21pmu-power20:40:241
1042499200,18cyclictest0-21swapper/121:05:291
1042499200,18cyclictest0-21swapper/119:46:271
1042099209,3cyclictest10619-21/usr/sbin/munin19:10:150
1042099207,5cyclictest0-21swapper/021:38:160
1042099207,3cyclictest31515-21diskmemload00:04:540
1042099206,5cyclictest28102-21sshd23:26:340
1042099204,8cyclictest31515-21diskmemload23:24:330
1042099204,3cyclictest11-21rcuc/021:10:250
1042099201,9cyclictest0-21swapper/022:05:300
1042099201,13cyclictest0-21swapper/023:36:300
1042099201,12cyclictest0-21swapper/022:59:020
1042099201,11cyclictest14634-21snmpd19:30:060
1042099201,10cyclictest0-21swapper/023:55:150
1042099200,14cyclictest86550irq/126-enp1s0-21:20:280
1043499196,7cyclictest0-21swapper/322:48:013
1043499195,3cyclictest19361-21kernelversion20:40:193
1043499195,2cyclictest2051-1kworker/u9:022:15:143
1043499194,3cyclictest11085-21grep20:20:293
1043499192,5cyclictest22269-21cat19:35:023
10434991913,3cyclictest21132-21sort23:10:263
1043499191,10cyclictest0-21swapper/323:46:533
1043499190,9cyclictest0-21swapper/320:15:173
1043499190,9cyclictest0-21swapper/320:15:173
1042999191,8cyclictest23311-21sed00:30:192
1042999191,8cyclictest23311-21sed00:30:192
1042999191,6cyclictest0-21swapper/219:25:392
10429991913,3cyclictest23091-21taskset22:01:532
10429991913,3cyclictest11014-21/usr/sbin/munin22:50:272
1042999191,11cyclictest0-21swapper/222:10:172
1042999191,11cyclictest0-21swapper/220:13:382
1042499191,6cyclictest27335-21/usr/sbin/munin21:00:161
1042499191,17cyclictest0-21swapper/119:29:541
1042099199,2cyclictest1818-21/usr/sbin/munin23:40:240
1042099197,3cyclictest31515-21diskmemload00:35:250
1042099197,3cyclictest0-21swapper/022:36:500
1042099196,4cyclictest0-21swapper/000:33:200
1042099196,4cyclictest0-21swapper/000:33:200
1042099196,3cyclictest20498-21crond20:45:010
1042099196,3cyclictest0-21swapper/022:43:380
1042099194,7cyclictest86550irq/126-enp1s0-22:00:170
1042099194,6cyclictest0-21swapper/023:07:530
1042099194,4cyclictest21336-21tail20:45:170
1042099193,6cyclictest0-21swapper/021:46:590
1042099193,3cyclictest16014-21gltestperf23:00:180
1042099193,3cyclictest0-21swapper/021:31:430
1042099192,9cyclictest0-21swapper/021:15:060
1042099191,6cyclictest0-21swapper/022:30:190
1042099191,4cyclictest7788-21chrt21:28:110
1042099191,12cyclictest0-21swapper/022:47:270
1042099190,5cyclictest0-21swapper/022:55:020
1042099190,5cyclictest0-21swapper/021:54:410
240482181,2sleep00-21swapper/019:37:210
1043499187,2cyclictest908-21/usr/sbin/munin20:00:223
1043499185,3cyclictest86550irq/126-enp1s0-21:05:203
1043499184,4cyclictest12523-21grep22:50:283
1043499183,3cyclictest23344-21latency_hist22:05:013
1043499181,7cyclictest22617-21gltestperf23:15:163
1043499181,6cyclictest4180-21sh22:35:013
1043499181,5cyclictest713-21cat20:00:013
1043499181,5cyclictest5049-21/usr/sbin/munin23:50:193
1043499181,5cyclictest27384-21/usr/sbin/munin19:45:193
1043499181,4cyclictest18114-21sshd20:37:403
10434991812,3cyclictest836-21runrttasks20:30:183
10434991812,3cyclictest7243-21/usr/sbin/munin23:55:153
10434991812,3cyclictest23226-21perl00:30:193
10434991812,3cyclictest23226-21perl00:30:193
10434991811,4cyclictest1-21systemd19:15:013
1042999187,3cyclictest0-21swapper/222:34:362
1042999183,3cyclictest0-21swapper/221:34:562
1042999182,5cyclictest22075-21munin-run19:35:002
1042999182,10cyclictest0-21swapper/223:51:312
1042999181,7cyclictest5107-21sed21:20:252
1042999181,7cyclictest0-21swapper/223:40:462
1042999181,7cyclictest0-21swapper/222:36:252
10429991814,3cyclictest32-21ksoftirqd/221:40:012
10429991813,3cyclictest16284-21taskset19:20:152
1042999181,10cyclictest0-21swapper/223:30:592
1042999180,7cyclictest0-21swapper/200:14:172
1042099185,3cyclictest13880-21/usr/sbin/munin00:10:280
1042099184,8cyclictest0-21swapper/022:11:010
1042099184,3cyclictest86550irq/126-enp1s0-20:20:260
1042099183,4cyclictest31515-21diskmemload23:11:410
1042099182,10cyclictest86550irq/126-enp1s0-00:16:200
1042099181,6cyclictest0-21swapper/020:57:320
10420991812,3cyclictest24822-21expr19:40:160
10420991812,3cyclictest14344-21sed19:15:210
1042099181,11cyclictest0-21swapper/020:50:190
1042099180,5cyclictest0-21swapper/023:33:220
82522171,2sleep20-21swapper/222:41:002
258832171,2sleep20-21swapper/223:21:182
1043499171,4cyclictest8170-21/usr/sbin/munin21:30:263
10434991713,3cyclictest0-21swapper/319:24:213
10434991711,3cyclictest24706-21expr19:40:163
1042999176,3cyclictest86850irq/129-enp1s0-23:25:222
1042999175,6cyclictest0-21swapper/220:55:222
1042999171,8cyclictest0-21swapper/222:20:222
1042999171,8cyclictest0-21swapper/222:20:222
1042999171,6cyclictest14515-21fschecks_count20:30:172
1042999171,6cyclictest0-21swapper/223:05:202
1042999171,6cyclictest0-21swapper/220:51:212
1042999171,6cyclictest0-21swapper/219:47:482
1042999171,6cyclictest0-21swapper/200:35:042
1042999171,6cyclictest0-21swapper/200:25:102
1042999171,6cyclictest0-21swapper/200:15:182
1042999171,5cyclictest0-21swapper/222:47:302
1042999171,4cyclictest0-21swapper/221:25:032
10429991713,3cyclictest32-21ksoftirqd/220:40:002
10429991712,3cyclictest20751-21sed23:10:212
10429991712,3cyclictest16471-21ntp_offset23:00:232
1042999171,15cyclictest0-21swapper/222:25:332
1042999171,15cyclictest0-21swapper/220:17:102
1042999171,15cyclictest0-21swapper/220:17:102
1042999171,14cyclictest0-21swapper/221:15:262
1042999171,14cyclictest0-21swapper/219:58:322
1042999171,14cyclictest0-21swapper/200:00:032
10429991710,4cyclictest26469-21dump-pmu-power19:45:012
1042099177,3cyclictest16340-21/usr/sbin/munin20:35:250
1042099172,6cyclictest5724-21hddtemp_smartct23:50:180
1042099171,7cyclictest29543-21/usr/sbin/munin21:05:290
1042099171,4cyclictest0-21swapper/023:45:140
10420991711,3cyclictest7737-21expr20:15:150
10420991711,3cyclictest7737-21expr20:15:150
10420991711,3cyclictest5350-21/usr/sbin/munin20:10:260
1042099171,13cyclictest1335-21grep20:00:170
1042099171,10cyclictest0-21swapper/021:56:150
1042099170,12cyclictest0-21swapper/019:20:170
1043499161,9cyclictest0-21swapper/300:05:143
1043499160,10cyclictest0-21swapper/321:15:183
1042999169,2cyclictest0-21swapper/222:55:182
1042999166,2cyclictest30-21rcuc/222:05:192
1042999161,9cyclictest0-21swapper/221:00:242
1042999161,5cyclictest0-21swapper/221:10:382
1042999161,5cyclictest0-21swapper/219:52:232
1042999161,2cyclictest0-21swapper/223:55:102
1042999161,2cyclictest0-21swapper/219:18:012
10429991612,3cyclictest0-21swapper/223:35:112
10429991611,3cyclictest3783-21sed23:45:192
10429991611,3cyclictest1615-21sh20:00:172
10429991610,3cyclictest29849-21expr21:05:152
1042099165,3cyclictest14634-21snmpd19:25:050
1042099161,3cyclictest0-21swapper/021:00:140
1042099161,14cyclictest0-21swapper/019:50:590
10420991611,3cyclictest3823-21hddtemp_smartct20:05:190
10420991611,3cyclictest32263-21proc_pri19:55:250
10420991611,3cyclictest27703-21netstat19:45:220
1042999150,10cyclictest0-21swapper/220:40:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional