You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 07:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Nov 25, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
102782237164,24sleep10-21swapper/119:09:081
101992233159,60sleep00-21swapper/019:08:070
101262233163,57sleep20-21swapper/219:07:112
101512202165,24sleep30-21swapper/319:07:293
2125521810,5sleep21045499cyclictest19:30:242
2490521490,3sleep01044299cyclictest23:20:150
2490521490,3sleep01044299cyclictest23:20:150
10454991101,107cyclictest0-21swapper/221:00:452
10454996358,3cyclictest0-21swapper/219:24:292
298732590,3sleep121-21rcuc/121:05:121
97842570,5sleep31045899cyclictest21:30:283
1045899393,28cyclictest31534-21diskmemload23:52:423
1044299311,14cyclictest651-21idleruntime-cro20:00:000
1044799304,18cyclictest0-21swapper/123:52:441
1044299303,13cyclictest20538-21awk20:45:010
1044299284,11cyclictest16141-21idleruntime-cro20:35:010
1045899272,13cyclictest39-21rcuc/321:55:193
1045499275,8cyclictest11023-40dnf22:50:122
1044799274,15cyclictest29144-21sshd21:02:461
1044799271,5cyclictest1737-21ntp_states23:40:211
1044299271,14cyclictest6403-21munin-run22:40:010
10454992622,3cyclictest0-21swapper/220:08:352
10454992613,3cyclictest12050irq/123-ahci[0022:35:022
10447992614,3cyclictest0-21swapper/123:18:471
1044799261,18cyclictest0-21swapper/121:55:571
1045899259,3cyclictest22321-21/usr/sbin/munin19:35:153
10458992510,3cyclictest86550irq/126-enp1s0-22:20:213
1044799254,3cyclictest0-21swapper/121:19:311
1044799254,12cyclictest0-21swapper/100:08:281
1044799253,15cyclictest86750irq/128-enp1s0-23:25:131
1044799252,4cyclictest6879-21sed20:10:261
1044799251,16cyclictest16407-21/usr/sbin/munin19:20:171
1044799251,16cyclictest0-21swapper/123:39:231
1045899241,4cyclictest784-21NetworkManager21:07:533
1045899241,11cyclictest9012-21sshd23:57:223
10458992410,3cyclictest86550irq/126-enp1s0-22:00:253
1045899240,6cyclictest0-21swapper/300:15:013
1045499243,5cyclictest27903-21kworker/2:121:30:282
1045499241,5cyclictest0-21swapper/200:23:012
1045499241,2cyclictest87950irq/132-enp2s0-20:27:522
1045499241,10cyclictest32-21ksoftirqd/220:33:122
1045499241,10cyclictest0-21swapper/223:13:112
1044799242,11cyclictest24293-21munin-run19:40:011
1044799241,17cyclictest0-21swapper/119:10:021
1044799241,10cyclictest31534-21diskmemload22:20:131
1044799240,17cyclictest0-21swapper/120:41:201
1044299241,8cyclictest0-21swapper/022:50:110
217252230,5sleep121-21rcuc/100:25:231
217252230,5sleep121-21rcuc/100:25:231
1045899238,3cyclictest11686-21/usr/sbin/munin00:05:163
1045899234,6cyclictest11023-40dnf22:50:123
1045899232,10cyclictest31534-21diskmemload23:17:063
1045899231,4cyclictest0-21swapper/321:29:463
1045899231,3cyclictest31534-21diskmemload00:19:573
1045899231,19cyclictest32420-21expr21:10:183
1045899231,19cyclictest32420-21expr21:10:183
1045899231,13cyclictest5498-21ntp_states22:35:223
1045899231,11cyclictest9469-21/usr/sbin/munin00:00:203
1045899231,11cyclictest31534-21diskmemload22:43:593
1045899231,11cyclictest27694-21smart_sda23:25:253
1045899231,10cyclictest30833-21/usr/sbin/munin23:35:153
1045899231,10cyclictest28562-40dnf19:48:323
1045499238,2cyclictest27903-21kworker/2:122:45:282
1045499237,3cyclictest18280-21grep23:05:152
1045499235,8cyclictest14389-21taskset21:42:552
10454992312,3cyclictest12050irq/123-ahci[0021:40:022
10454992310,5cyclictest86550irq/126-enp1s0-21:10:272
10454992310,5cyclictest86550irq/126-enp1s0-21:10:272
1045499230,12cyclictest28919-21/usr/sbin/munin19:50:282
1044799239,9cyclictest0-21swapper/100:36:341
1044799236,9cyclictest0-21swapper/121:42:231
1044799231,3cyclictest3508-21tail23:45:151
1044799231,19cyclictest11395-21cut21:35:211
1044799231,19cyclictest11023-40dnf22:50:091
1044799231,16cyclictest10988-21kworker/u8:223:23:321
1044799231,16cyclictest10988-21kworker/u8:223:23:321
1044799231,16cyclictest0-21swapper/121:52:361
1044799231,16cyclictest0-21swapper/120:58:391
1044799231,16cyclictest0-21swapper/120:30:411
1044799231,16cyclictest0-21swapper/120:22:511
1044799230,5cyclictest0-21swapper/100:17:301
1044799230,18cyclictest0-21swapper/123:07:501
57032221,2sleep20-21swapper/220:10:162
1045899229,2cyclictest39-21rcuc/320:10:193
1045899223,11cyclictest10988-21kworker/u8:223:41:153
1045899222,9cyclictest31534-21diskmemload22:26:033
1045899222,17cyclictest2654-21uniq22:30:143
1045899221,4cyclictest0-21swapper/323:03:353
1045899221,11cyclictest14634-21snmpd20:18:223
1045899221,10cyclictest17925-21/usr/sbin/munin19:25:203
1045899220,7cyclictest0-21swapper/321:40:303
1045499227,2cyclictest0-21swapper/222:05:182
1045499223,16cyclictest20700-21unixbench_multi21:55:272
1045499221,9cyclictest0-21swapper/220:53:482
10454992210,4cyclictest3008-21cat21:15:272
1044799229,4cyclictest21195-21ntp_states19:30:221
1044799222,15cyclictest28559-40dnf19:48:331
1044799222,11cyclictest9173-21needreboot21:30:201
10447992218,3cyclictest0-21swapper/121:11:181
10447992218,3cyclictest0-21swapper/121:11:181
1044799221,3cyclictest23018-21expr00:30:141
1044799221,3cyclictest18613-21cstates00:20:151
1044799221,15cyclictest30547-21kworker/u8:123:57:011
1044799221,15cyclictest0-21swapper/123:12:491
1044799221,15cyclictest0-21swapper/120:16:001
1044799221,14cyclictest0-21swapper/121:22:051
1044799221,11cyclictest0-21swapper/122:38:211
1044799221,10cyclictest0-21swapper/122:59:161
1044799221,10cyclictest0-21swapper/122:06:531
1044799221,10cyclictest0-21swapper/121:27:031
1044299223,15cyclictest0-21swapper/019:48:330
250162210,3sleep011-21rcuc/022:05:240
1045899217,4cyclictest0-21swapper/322:09:103
1045899216,4cyclictest14158-21/usr/sbin/munin20:30:213
1045899214,14cyclictest12365-21ls21:40:023
1045899211,9cyclictest31534-21diskmemload00:28:313
1045899211,9cyclictest31534-21diskmemload00:28:313
1045899211,11cyclictest20900-21ntp_states23:10:213
1045899211,10cyclictest31534-21diskmemload23:33:093
1045499217,3cyclictest0-21swapper/220:20:322
1045499215,8cyclictest86550irq/126-enp1s0-21:25:132
1045499214,6cyclictest780-21rngd00:08:002
1045499211,9cyclictest0-21swapper/220:40:162
1045499211,6cyclictest0-21swapper/222:36:572
1045499211,10cyclictest0-21swapper/223:17:592
1044799217,8cyclictest0-21swapper/122:30:121
1044799212,11cyclictest29704-21ntp_states23:30:231
1044799211,7cyclictest21760-21fschecks_count22:00:141
10447992114,2cyclictest0-21swapper/120:28:501
10447992114,2cyclictest0-21swapper/119:42:391
10447992113,3cyclictest0-21swapper/119:16:491
1044799211,18cyclictest0-21swapper/122:47:551
1044799211,15cyclictest0-21swapper/122:10:451
1044799211,15cyclictest0-21swapper/120:52:341
1044799211,15cyclictest0-21swapper/120:45:321
1044799211,12cyclictest0-21swapper/122:17:451
1044799211,11cyclictest581-21systemd-journal22:45:011
1044799211,11cyclictest16819-21cat21:50:011
1044799211,10cyclictest0-21swapper/100:11:061
1044299217,11cyclictest836-21runrttasks23:10:250
181052201,3sleep30-21swapper/320:37:223
171642201,3sleep117267-21memory20:35:181
1045899209,3cyclictest0-21swapper/319:43:103
1045899208,3cyclictest26044-21sshd23:23:083
1045899208,3cyclictest26044-21sshd23:23:083
1045899207,3cyclictest9427-21head22:45:173
1045899206,5cyclictest0-21swapper/321:53:443
1045899205,3cyclictest0-21swapper/300:39:213
1045899201,9cyclictest780-21rngd21:22:503
1045899201,4cyclictest28919-21/usr/sbin/munin19:50:143
1045899201,4cyclictest25800-21/usr/sbin/munin22:10:163
1045899201,11cyclictest424-21sshd19:57:353
1045499209,3cyclictest86850irq/129-enp1s0-00:25:182
1045499209,3cyclictest86850irq/129-enp1s0-00:25:182
1045499206,3cyclictest0-21swapper/219:11:452
1045499205,6cyclictest86550irq/126-enp1s0-00:00:242
1045499204,8cyclictest86850irq/129-enp1s0-21:21:392
1045499203,14cyclictest27323-21tail22:10:262
1045499202,10cyclictest14792-21/usr/sbin/munin21:45:182
1045499201,7cyclictest0-21swapper/200:32:042
10454992014,3cyclictest4411-21grep23:45:262
10454992013,4cyclictest15421-21tail00:10:262
1045499200,9cyclictest32-21ksoftirqd/220:40:022
1044799209,3cyclictest19514-21users19:25:271
1044799206,5cyclictest0-21swapper/100:00:151
1044799201,17cyclictest0-21swapper/120:00:311
1044799201,10cyclictest15777-21diskstats23:00:151
1044799200,5cyclictest0-21swapper/122:25:081
1044299201,11cyclictest14792-21/usr/sbin/munin21:45:240
1045899195,8cyclictest86550irq/126-enp1s0-21:15:213
1045899192,10cyclictest7812-21kworker/u8:021:47:043
1045899191,9cyclictest15058-21sshd22:58:223
1045899191,11cyclictest0-21swapper/323:08:243
1045899191,10cyclictest24734-21sshd20:52:463
1045499199,2cyclictest27903-21kworker/2:123:20:162
1045499199,2cyclictest27903-21kworker/2:123:20:162
1045499197,3cyclictest0-21swapper/222:55:042
1045499196,2cyclictest30-21rcuc/219:15:032
1045499193,3cyclictest12050irq/123-ahci[0021:10:012
1045499192,8cyclictest22030-21expr22:00:162
1045499192,3cyclictest29969-21awk22:20:012
1045499192,14cyclictest26629-21sed20:55:272
1045499191,8cyclictest30769-21kworker/2:020:45:142
1045499191,6cyclictest0-21swapper/219:25:232
10454991913,3cyclictest31886-21ntpq23:35:232
1044799192,2cyclictest0-21swapper/119:50:091
10447991915,3cyclictest0-21swapper/120:08:491
10447991915,3cyclictest0-21swapper/119:58:271
1044299198,3cyclictest32395-21/usr/sbin/munin22:25:160
1044299197,7cyclictest0-21swapper/020:39:560
1044299197,3cyclictest14634-21snmpd00:27:210
1044299197,3cyclictest14634-21snmpd00:27:210
1044299191,9cyclictest86550irq/126-enp1s0-21:55:140
1044299191,5cyclictest0-21swapper/022:46:200
1044299191,2cyclictest86550irq/126-enp1s0-19:52:430
10442991911,5cyclictest19559-21/usr/sbin/munin00:20:260
1044299191,11cyclictest0-21swapper/021:43:500
1044299191,10cyclictest27996-21/usr/sbin/munin22:15:190
321192181,2sleep00-21swapper/022:24:320
220402181,2sleep30-21swapper/319:34:323
179912181,2sleep00-21swapper/000:19:130
1045899187,3cyclictest18297-21/usr/sbin/munin00:20:203
1045899187,3cyclictest15725-21/usr/sbin/munin19:20:173
1045899186,5cyclictest0-21swapper/322:17:093
1045899183,8cyclictest10988-21kworker/u8:200:32:123
1045899182,13cyclictest25151-21/usr/sbin/munin20:55:163
1045899181,3cyclictest13611-21cpuspeed_turbos19:15:123
10458991813,3cyclictest11745-21pmu-power19:10:233
10458991812,4cyclictest13112-21awk20:25:223
10458991812,3cyclictest1150-21expr20:00:143
10458991812,3cyclictest10424-21sed20:20:183
1045499189,2cyclictest86850irq/129-enp1s0-20:15:172
1045499185,2cyclictest30-21rcuc/223:25:182
1045499184,3cyclictest0-21swapper/219:42:482
1045499184,12cyclictest6146-21open_files23:50:232
1045499182,8cyclictest86850irq/129-enp1s0-19:55:172
1045499182,7cyclictest86850irq/129-enp1s0-20:00:242
1045499181,7cyclictest31369-21ps22:20:222
1045499181,3cyclictest16991-21/usr/sbin/munin21:50:232
1045499181,15cyclictest0-21swapper/223:00:142
1045499181,10cyclictest0-21swapper/222:25:162
1045499181,10cyclictest0-21swapper/219:45:212
1044299189,2cyclictest14634-21snmpd22:31:520
1044299184,7cyclictest780-21rngd23:10:010
1044299182,5cyclictest9265-21ntp_states21:30:220
1044299181,9cyclictest14634-21snmpd19:36:190
1044299181,8cyclictest16767-21/usr/sbin/munin23:00:260
1044299181,7cyclictest0-21swapper/023:35:370
1044299181,7cyclictest0-21swapper/000:08:040
1044299181,5cyclictest0-21swapper/023:50:380
1044299181,5cyclictest0-21swapper/021:18:110
1044299181,3cyclictest26493-21munin-run19:45:000
10442991813,3cyclictest31046-21unixbench-2d21:05:240
10442991813,3cyclictest14006-21if_err_enp1s022:55:180
10442991812,3cyclictest14237-21df_inode00:10:130
10442991812,3cyclictest1065-21expr20:00:130
1044299181,14cyclictest12730-21sort19:15:020
1044299181,10cyclictest11957-21/usr/sbin/munin20:25:200
246272171,3sleep00-21swapper/020:51:230
1045899177,2cyclictest5069-21sshd20:09:213
1045899171,6cyclictest3621-21if_enp2s023:45:173
1045899171,14cyclictest19222-21sed20:40:173
10458991711,3cyclictest28613-21smart_sda21:00:243
10458991711,3cyclictest20895-21cpuspeed_turbos20:45:143
1045499171,7cyclictest0-21swapper/200:17:312
1045499171,6cyclictest0-21swapper/222:40:132
10454991711,3cyclictest29251-21head23:30:182
1045499171,10cyclictest0-21swapper/219:39:242
1044299172,8cyclictest0-21swapper/023:49:050
1044299172,12cyclictest9755-21/usr/sbin/munin20:20:190
1044299171,7cyclictest22848-21tr00:30:140
1044299171,7cyclictest0-21swapper/021:20:250
1044299171,6cyclictest0-21swapper/023:30:240
1044299171,6cyclictest0-21swapper/000:35:100
1044299171,5cyclictest0-21swapper/022:40:070
10442991712,3cyclictest8290-21nfs4_client23:55:200
10442991712,3cyclictest5709-21diskstats20:10:140
10442991712,3cyclictest26432-21/usr/sbin/munin23:25:200
1044299171,15cyclictest0-21swapper/023:18:230
1044299171,14cyclictest0-21swapper/022:13:270
1044299171,14cyclictest0-21swapper/021:10:090
1044299171,14cyclictest0-21swapper/021:10:090
10442991711,3cyclictest836-21runrttasks19:30:140
10442991711,3cyclictest674-21wc23:40:120
10442991711,3cyclictest10764-21grep21:35:160
10442991711,3cyclictest10031-21cut00:00:160
1044299171,10cyclictest0-21swapper/022:00:110
1045499161,6cyclictest0-21swapper/200:35:182
1045499161,2cyclictest0-21swapper/223:40:102
10454991611,3cyclictest7275-21cat23:55:132
1044299161,6cyclictest13250-21gltestperf19:15:030
1044299161,4cyclictest0-21swapper/020:45:160
10442991611,3cyclictest18996-21ntp_states19:25:220
10442991611,3cyclictest18095-21awk21:50:200
10442991611,3cyclictest17574-21seq19:23:350
10442991610,3cyclictest5267-21cut20:10:010
10442991610,3cyclictest27683-21sh21:00:120
1044299160,5cyclictest0-21swapper/021:25:140
10442991511,2cyclictest7565-21acpi20:15:130
10442991510,3cyclictest26260-21grep20:55:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional