You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri May 09, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
318412238164,25sleep30-21swapper/319:07:293
317492232162,56sleep00-21swapper/019:06:170
317512218188,20sleep20-21swapper/219:06:192
317552204167,24sleep10-21swapper/119:06:221
2706821370,5sleep03212799cyclictest23:50:140
2053621100,3sleep33214299cyclictest22:20:213
3213899108105,2cyclictest0-21swapper/220:59:172
3213899107104,2cyclictest0-21swapper/223:48:022
4022640,4sleep13213399cyclictest22:50:141
96152590,6sleep03212799cyclictest23:10:160
307132480,1sleep30-21swapper/321:30:193
262472420,1sleep30-21swapper/321:20:203
32127993912,13cyclictest2242-21awk22:55:020
32138993330,2cyclictest0-21swapper/200:38:462
3213399321,23cyclictest0-21swapper/123:35:231
3212799326,13cyclictest4968-21memory21:45:200
3212799315,11cyclictest3265-21/usr/sbin/munin22:55:190
303102300,2sleep00-21swapper/020:15:220
3213899291,11cyclictest32623-21/usr/sbin/munin22:50:212
3213399281,18cyclictest17586-21diskstats22:15:151
3213399281,18cyclictest17586-21diskstats22:15:151
3212799288,16cyclictest20805-21diskmemload21:18:140
3212799283,11cyclictest12050irq/123-ahci[0000:30:010
3212799283,11cyclictest12050irq/123-ahci[0000:30:010
3212799281,18cyclictest13266-21cat23:20:010
3212799274,11cyclictest3237-21awk20:30:010
32127992711,6cyclictest6350-21ntp_states00:15:240
32127992710,3cyclictest26031-21/usr/sbin/munin22:35:220
3212799270,12cyclictest20206-21perf19:55:010
3213899261,12cyclictest0-21swapper/223:18:432
3213399264,19cyclictest848-21sssd_nss00:35:201
32133992613,10cyclictest4635-21systemd-journal23:00:021
3212799268,15cyclictest1021-21awk20:25:010
3212799265,10cyclictest13154-21switchtime00:30:260
3212799263,13cyclictest10040-21/usr/sbin/munin20:45:190
3213899253,8cyclictest21402-21latency_hist22:25:012
3213399251,4cyclictest14621-21proc_pri23:20:281
3213399251,4cyclictest14621-21proc_pri23:20:281
3213399250,6cyclictest0-21swapper/121:10:301
3213399250,6cyclictest0-21swapper/121:10:301
3212799255,3cyclictest12050irq/123-ahci[0021:30:010
3212799255,10cyclictest26246-21irqstats21:20:200
3212799254,9cyclictest31442-21/usr/sbin/munin22:45:210
3212799253,9cyclictest0-21swapper/021:00:200
3214299241,6cyclictest0-21swapper/300:15:303
3213899242,10cyclictest0-21swapper/200:23:502
3213899241,5cyclictest0-21swapper/221:45:252
32138992414,3cyclictest12050irq/123-ahci[0023:55:002
3213399244,4cyclictest9246-21runrttasks00:22:331
3213399243,16cyclictest25043-40dnf21:18:151
3213399241,16cyclictest0-21swapper/122:30:311
3213399240,16cyclictest0-21swapper/100:06:191
3212799243,8cyclictest0-21swapper/022:10:190
3212799242,18cyclictest2316-21fschecks_count21:40:150
3212799242,12cyclictest0-21swapper/023:06:340
3212799242,11cyclictest20419-21/usr/sbin/munin19:55:180
3212799242,10cyclictest32018-21/usr/sbin/munin21:35:240
3212799241,6cyclictest0-21swapper/000:10:150
3212799241,10cyclictest31446-21kworker/u8:221:55:240
3213899239,5cyclictest86850irq/129-enp1s0-21:30:172
3213899233,2cyclictest88150irq/134-enp2s0-19:45:022
32133992317,3cyclictest19898-21ntp_states21:05:221
32133992314,3cyclictest0-21swapper/122:21:441
3213399231,3cyclictest12003-21munin-run20:50:021
3213399231,19cyclictest22846-21processes22:25:241
3213399231,19cyclictest15959-21memory22:10:191
3213399231,16cyclictest0-21swapper/122:57:031
3213399231,16cyclictest0-21swapper/119:27:441
3213399231,10cyclictest1143-21unixbench_multi21:35:271
3212799232,9cyclictest14169-21kworker/0:320:59:460
3212799232,11cyclictest21046-21netstat23:35:210
3212799232,11cyclictest19436-21/usr/sbin/munin22:20:110
3212799231,7cyclictest0-21swapper/023:22:020
3212799231,7cyclictest0-21swapper/023:22:020
3212799231,4cyclictest0-21swapper/023:49:240
32127992313,7cyclictest5643-21/usr/sbin/munin20:35:140
3212799230,13cyclictest835-21/usr/sbin/munin00:05:150
3213899228,7cyclictest18220-21kworker/2:120:20:592
3213899227,5cyclictest780-21rngd21:40:212
3213899224,3cyclictest24699-21unixbench-2d21:15:252
3213899222,7cyclictest26684-21cut22:35:182
3213899221,8cyclictest0-21swapper/200:20:002
3213899221,7cyclictest20805-21diskmemload22:40:292
3213899221,3cyclictest88250irq/135-enp2s0-20:44:412
32138992212,7cyclictest21033-21systemd-journal21:10:092
32138992212,7cyclictest21033-21systemd-journal21:10:092
3213899221,13cyclictest0-21swapper/223:20:562
3213899221,13cyclictest0-21swapper/223:20:552
3213899221,12cyclictest0-21swapper/223:14:112
32138992210,5cyclictest86750irq/128-enp1s0-22:55:172
32138992210,4cyclictest86750irq/128-enp1s0-22:15:232
32138992210,4cyclictest86750irq/128-enp1s0-22:15:232
3213399227,13cyclictest14758-21kworker/u8:023:06:501
32133992216,3cyclictest6759-21grep00:15:291
3213399221,18cyclictest564-21cat19:10:161
3213399221,15cyclictest0-21swapper/122:41:261
3212799226,6cyclictest23774-21/usr/sbin/munin23:40:260
3212799225,4cyclictest0-21swapper/022:00:410
3212799221,6cyclictest0-21swapper/021:51:040
32127992213,6cyclictest4594-40dnf19:17:110
32127992213,5cyclictest10704-21smart_sda19:30:250
3212799221,12cyclictest7249-21/usr/sbin/munin19:25:180
3212799221,10cyclictest28226-21/usr/sbin/munin22:40:210
127822220,4sleep339-21rcuc/323:15:253
67142211,2sleep30-21swapper/320:35:213
37062210,5sleep33214299cyclictest20:30:143
3214299213,10cyclictest11638-21/usr/sbin/munin19:35:253
3214299212,16cyclictest5690-21tail19:20:133
3214299212,10cyclictest26031-21/usr/sbin/munin22:35:153
32142992114,4cyclictest17290-21gltestperf21:00:183
3214299211,11cyclictest8624-21cstates21:55:113
3214299210,9cyclictest0-21swapper/323:23:323
3214299210,9cyclictest0-21swapper/323:23:323
3213899218,3cyclictest12050irq/123-ahci[0022:00:002
3213899215,9cyclictest13155-21switchtime00:30:252
3213899214,7cyclictest86550irq/126-enp1s0-19:35:162
3213899213,9cyclictest25317-21sed22:30:252
3213899212,12cyclictest86850irq/129-enp1s0-22:10:132
3213899211,8cyclictest18220-21kworker/2:120:25:072
32138992111,7cyclictest86850irq/129-enp1s0-20:06:262
32138992111,7cyclictest86850irq/129-enp1s0-20:06:262
32138992111,3cyclictest12050irq/123-ahci[0020:00:162
3213899211,13cyclictest0-21swapper/200:13:212
3213899211,12cyclictest0-21swapper/221:25:262
32133992117,3cyclictest0-21swapper/121:46:501
3213399211,18cyclictest0-21swapper/120:33:521
3213399211,18cyclictest0-21swapper/120:29:421
3213399211,15cyclictest0-21swapper/122:01:361
3213399211,15cyclictest0-21swapper/119:21:071
3213399211,11cyclictest25974-21/usr/sbin/munin23:45:271
3213399210,15cyclictest0-21swapper/123:10:351
3212799218,2cyclictest20016-21kworker/0:120:13:040
3212799217,7cyclictest0-21swapper/000:03:490
3212799217,11cyclictest8827-21timerandwakeup00:20:260
3212799216,2cyclictest3160-21kworker/0:220:33:390
3212799215,6cyclictest0-21swapper/023:25:030
3212799212,5cyclictest0-21swapper/023:30:260
3212799212,16cyclictest19543-21idleruntime21:05:180
32127992113,5cyclictest13302-21ntp_states20:50:200
3212799211,11cyclictest0-21swapper/021:14:230
3212799211,11cyclictest0-21swapper/021:14:230
3212799210,7cyclictest0-21swapper/022:15:460
3212799210,7cyclictest0-21swapper/022:15:460
3212799210,11cyclictest86850irq/129-enp1s0-22:30:100
309512210,4sleep121-21rcuc/122:45:161
152932211,16sleep315419-21users19:40:253
118572210,5sleep121-21rcuc/123:15:151
3214299207,4cyclictest0-21swapper/322:42:243
3214299206,4cyclictest31741-21basename00:00:203
3214299204,9cyclictest0-21swapper/322:05:343
3214299201,8cyclictest0-21swapper/321:27:453
32142992012,4cyclictest836-21runrttasks20:05:263
32142992012,4cyclictest836-21runrttasks20:05:263
3214299201,11cyclictest1779-21sshd19:13:113
3213899207,5cyclictest86850irq/129-enp1s0-23:39:212
3213899204,9cyclictest0-21swapper/223:58:502
3213899204,10cyclictest32-21ksoftirqd/222:30:012
3213899203,7cyclictest0-21swapper/221:10:012
3213899202,6cyclictest32-21ksoftirqd/222:46:562
3213899201,8cyclictest581-21systemd-journal21:05:012
3213899201,8cyclictest11032-21cut20:45:222
3213899201,13cyclictest0-21swapper/221:35:212
3213899201,12cyclictest0-21swapper/221:51:102
3213399206,3cyclictest22947-21basename23:40:171
32133992016,3cyclictest0-21swapper/121:23:541
32133992016,3cyclictest0-21swapper/120:09:561
32133992016,3cyclictest0-21swapper/120:09:561
3213399201,3cyclictest0-21swapper/119:15:491
3213399201,2cyclictest0-21swapper/119:31:201
3213399201,18cyclictest0-21swapper/123:50:151
3213399201,18cyclictest0-21swapper/120:57:411
3213399201,18cyclictest0-21swapper/120:53:081
3213399201,11cyclictest848-21sssd_nss00:10:261
3213399201,11cyclictest28957-21cut23:55:131
3213399201,10cyclictest9990-21unixbench_singl21:55:241
3213399201,10cyclictest18198-21cat19:50:011
32133992010,3cyclictest8213-21latency_hist21:55:021
3213399200,18cyclictest0-21swapper/100:00:281
3213399200,14cyclictest0-21swapper/122:05:061
3212799207,4cyclictest0-21swapper/022:06:390
3212799206,2cyclictest11521-21kworker/0:100:38:020
3212799206,2cyclictest0-21swapper/019:39:450
3212799203,2cyclictest20016-21kworker/0:120:03:380
3212799203,11cyclictest0-21swapper/022:25:150
3212799203,10cyclictest88050irq/133-enp2s0-21:33:200
3212799201,6cyclictest0-21swapper/019:20:210
32127992012,6cyclictest14227-21fschecks_count19:40:140
32127992012,5cyclictest8192-21diskstats20:40:140
224922200,4sleep341-21ksoftirqd/321:10:283
224922200,4sleep341-21ksoftirqd/321:10:283
3214299198,3cyclictest9056-21/usr/sbin/munin23:10:243
3214299193,5cyclictest12017-21awk20:50:023
3214299192,14cyclictest25782-21wc23:45:263
32142991913,3cyclictest19774-21head19:50:253
3214299191,11cyclictest14634-21snmpd21:37:333
3214299191,10cyclictest10642-21/usr/sbin/munin22:00:293
3213899196,5cyclictest86850irq/129-enp1s0-23:40:182
3213899195,7cyclictest86550irq/126-enp1s0-21:22:182
3213899193,2cyclictest88050irq/133-enp2s0-20:13:042
3213899192,10cyclictest28601-21kworker/2:019:18:252
3213899191,7cyclictest2157-21smart_sda19:15:022
3213899191,15cyclictest29347-21cpuspeed_turbos20:15:112
3213899191,13cyclictest0-21swapper/200:27:452
3213899191,13cyclictest0-21swapper/200:27:452
3213899191,12cyclictest0-21swapper/223:28:282
3213899191,12cyclictest0-21swapper/200:05:062
3213899191,11cyclictest0-21swapper/222:08:532
32138991910,2cyclictest86550irq/126-enp1s0-00:00:202
3213399193,7cyclictest7845-21/usr/sbin/munin20:40:251
32133991916,2cyclictest0-21swapper/100:31:491
32133991915,3cyclictest0-21swapper/123:30:311
3213399191,4cyclictest7231-21users20:35:241
32133991913,2cyclictest0-21swapper/123:26:161
3213399191,2cyclictest0-21swapper/119:35:331
3213399191,17cyclictest0-21swapper/120:21:451
3213399191,16cyclictest0-21swapper/121:25:381
3213399191,16cyclictest0-21swapper/100:26:351
3213399191,16cyclictest0-21swapper/100:26:351
3213399191,15cyclictest31383-21grep21:30:271
3213399191,14cyclictest0-21swapper/122:35:431
3213399190,2cyclictest0-21swapper/121:42:101
3213399190,18cyclictest0-21swapper/119:51:021
3213399190,17cyclictest0-21swapper/119:40:151
3212799199,7cyclictest32394-21cpuspeed_turbos19:10:120
3212799191,2cyclictest0-21swapper/019:45:250
32127991912,4cyclictest25007-21cstates20:05:140
32127991912,4cyclictest25007-21cstates20:05:140
3212799191,15cyclictest4651-21/usr/sbin/munin23:00:190
29882191,3sleep30-21swapper/320:27:103
218212190,4sleep230-21rcuc/219:55:252
68582181,2sleep20-21swapper/223:05:132
3214299189,8cyclictest86550irq/126-enp1s0-21:45:153
3214299184,3cyclictest86550irq/126-enp1s0-23:55:173
3214299182,9cyclictest0-21swapper/322:20:003
3214299182,9cyclictest0-21swapper/322:20:003
3214299181,9cyclictest0-21swapper/322:30:053
3214299181,7cyclictest9539-21latency_hist00:25:013
3214299181,7cyclictest581-21systemd-journal00:40:013
3214299181,5cyclictest20638-21head23:35:173
3214299181,4cyclictest14634-21snmpd20:10:283
3214299181,4cyclictest0-21swapper/322:47:193
3214299181,3cyclictest0-21swapper/319:31:303
32142991813,3cyclictest13292-21ntp_offset20:50:193
32142991812,3cyclictest29220-21/usr/sbin/munin20:15:243
3214299181,10cyclictest14634-21snmpd23:51:333
3213899185,5cyclictest86550irq/126-enp1s0-22:00:212
3213899182,6cyclictest0-21swapper/220:35:002
3213899182,2cyclictest88250irq/135-enp2s0-20:36:402
3213899181,6cyclictest0-21swapper/223:32:122
3213899181,6cyclictest0-21swapper/219:30:132
32138991812,3cyclictest6115-21sh19:20:182
3213899181,10cyclictest0-21swapper/223:01:112
3213399181,8cyclictest29220-21/usr/sbin/munin20:15:171
32133991815,2cyclictest0-21swapper/120:00:411
32133991814,3cyclictest0-21swapper/120:11:361
3213399181,16cyclictest0-21swapper/119:57:311
3213399181,11cyclictest0-21swapper/121:00:141
3212799188,7cyclictest29340-21tune2fs23:55:160
3214299176,3cyclictest86850irq/129-enp1s0-23:05:153
3214299174,10cyclictest0-21swapper/321:51:183
3214299172,7cyclictest20805-21diskmemload23:43:583
3214299171,6cyclictest3438-21diskstats00:10:163
3214299171,6cyclictest19802-21sh23:35:013
3214299171,6cyclictest16391-21latency_hist21:00:013
3214299171,6cyclictest0-21swapper/300:30:123
3214299171,6cyclictest0-21swapper/300:25:113
3214299171,6cyclictest0-21swapper/300:25:113
3214299171,3cyclictest0-21swapper/300:05:323
32142991712,3cyclictest7949-21tr20:40:123
32142991712,3cyclictest323-21cstates22:50:143
32142991712,3cyclictest3083-21expr19:15:133
3214299171,15cyclictest0-21swapper/323:02:233
3214299171,14cyclictest0-21swapper/319:48:303
3214299171,14cyclictest0-21swapper/319:26:193
3214299171,13cyclictest2457-21/usr/sbin/munin22:55:173
32142991710,4cyclictest16924-21/usr/sbin/munin23:25:283
3213899171,9cyclictest0-21swapper/219:45:182
234072171,2sleep30-21swapper/322:27:413
207412171,2sleep30-21swapper/321:09:253
168102170,3sleep30-21swapper/322:12:353
3214299161,6cyclictest0-21swapper/321:40:023
3214299161,6cyclictest0-21swapper/321:15:183
3214299161,14cyclictest0-21swapper/320:03:003
32142991611,3cyclictest31574-21expr20:20:103
3213899162,3cyclictest86550irq/126-enp1s0-19:25:222
32138991612,3cyclictest0-21swapper/219:50:132
32138991610,3cyclictest12293-21cpu20:50:112
220582160,2sleep30-21swapper/319:55:593
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional