You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-24 - 21:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 24, 2024 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
177692204166,25sleep20-21swapper/207:06:412
177222204164,26sleep10-21swapper/107:06:051
178542202165,24sleep00-21swapper/007:07:490
177192201164,25sleep30-21swapper/307:06:023
2427321420,2sleep21812199cyclictest11:15:192
1812199105101,3cyclictest0-21swapper/209:10:352
18121998986,2cyclictest0-21swapper/211:26:572
18121997268,3cyclictest0-21swapper/208:53:522
37172610,4sleep21812199cyclictest08:00:002
227102580,5sleep11811699cyclictest11:10:281
93022520,1sleep00-21swapper/010:40:270
1812699452,27cyclictest1042-21diskstats10:20:133
1812699411,24cyclictest28098-21latency_hist12:40:013
1811699411,30cyclictest0-21swapper/112:17:151
1811699411,13cyclictest0-21swapper/111:20:281
1812699401,25cyclictest581-21systemd-journal11:50:093
266582390,11sleep339-21rcuc/308:50:193
1812699396,21cyclictest12937-21/usr/sbin/munin12:05:263
1812699391,29cyclictest19834-21memory11:05:203
1812699391,27cyclictest19880-21/usr/sbin/munin09:50:243
1811699391,32cyclictest0-21swapper/109:41:561
1811699391,28cyclictest0-21swapper/112:39:111
1811699391,11cyclictest0-21swapper/110:02:401
1812699381,29cyclictest0-21swapper/310:15:153
1812699381,26cyclictest0-21swapper/310:35:133
1812699381,19cyclictest1834-21latency_hist11:40:013
1812699381,19cyclictest1834-21latency_hist11:40:013
1812699373,20cyclictest21012-21grep12:20:303
1812699371,33cyclictest28057-21/usr/sbin/munin08:55:193
1812699371,20cyclictest21229-21tr11:10:113
1810799379,13cyclictest24047-21munin-run10:00:010
1812699361,21cyclictest28801-21latency_hist07:40:013
1812699361,21cyclictest25815-21runrttasks12:33:593
1811699361,18cyclictest0-21swapper/109:47:241
1811699360,11cyclictest0-21swapper/107:50:271
201642350,1sleep10-21swapper/111:05:261
1812699355,20cyclictest12050irq/123-ahci[0008:25:013
1812699352,29cyclictest17675-21/usr/sbin/munin09:45:133
18126993514,18cyclictest20794-21irqstats07:15:043
18126993514,18cyclictest20794-21irqstats07:15:043
1812699351,25cyclictest16549-21ntp_states09:40:223
1812699351,21cyclictest3039-21open_inodes11:40:233
1812699351,21cyclictest0-21swapper/309:15:253
1812699350,27cyclictest0-21swapper/311:45:233
1811699350,8cyclictest0-21swapper/111:43:401
1810799358,12cyclictest2702-21perf10:25:000
1812699342,28cyclictest842-21lldpd08:11:573
1812699342,27cyclictest15777-21smart_sda10:55:283
1812699342,23cyclictest21465-21/usr/sbin/munin08:40:123
1812699341,30cyclictest25516-21/usr/sbin/munin11:20:293
1812699341,29cyclictest29904-21/usr/sbin/munin11:30:163
1812699341,29cyclictest14634-21snmpd12:17:153
1812699341,24cyclictest86550irq/126-enp1s0-09:10:113
1812699341,23cyclictest13632-21diskstats09:35:133
1812699341,21cyclictest26060-21taskset10:02:403
1812699341,20cyclictest17101-21cpu08:30:153
1812699341,19cyclictest814-21sshd08:19:473
1812699341,19cyclictest30629-21taskset10:14:533
1812699341,19cyclictest15881-21netstat08:25:233
1812699333,16cyclictest86550irq/126-enp1s0-11:55:203
1812699331,29cyclictest364-21needreboot07:50:183
1812699331,19cyclictest16704-21/usr/sbin/munin11:00:243
1812699330,5cyclictest0-21swapper/312:25:203
1811699330,15cyclictest0-21swapper/112:03:541
1810799337,12cyclictest16483-21munin-run11:00:010
1810799334,13cyclictest19048-21munin-run08:35:020
1812699323,25cyclictest28972-21smart_sda11:25:223
1812699322,27cyclictest21819-21/usr/sbin/munin07:20:103
1812699322,26cyclictest20260-40dnf07:14:503
1812699321,23cyclictest784-21NetworkManager08:36:323
1812699321,23cyclictest14278-21latency_hist10:55:013
1812699321,22cyclictest6699-21/usr/sbin/munin09:20:163
1812699321,22cyclictest6699-21/usr/sbin/munin09:20:163
1812699321,17cyclictest3592-21hddtemp_smartct10:25:173
1812699320,30cyclictest0-21swapper/308:05:133
1811699320,6cyclictest0-21swapper/111:55:001
1811699320,16cyclictest0-21swapper/111:00:241
1811699320,13cyclictest0-21swapper/111:45:321
1812699314,19cyclictest86850irq/129-enp1s0-09:30:263
1812699312,20cyclictest86550irq/126-enp1s0-07:34:463
1812699312,18cyclictest86550irq/126-enp1s0-11:15:093
1812699311,27cyclictest780-21rngd07:47:063
1812699310,23cyclictest0-21swapper/310:41:423
1811699311,15cyclictest0-21swapper/110:11:221
1811699311,12cyclictest0-21swapper/108:25:201
1811699310,12cyclictest0-21swapper/108:01:061
1812699301,26cyclictest5084-21/usr/sbin/munin08:00:233
1812699301,26cyclictest30253-21/usr/sbin/munin09:00:203
1812699301,26cyclictest2022-21diskstats07:55:143
1812699301,26cyclictest11080-21/usr/sbin/munin12:00:143
1812699301,25cyclictest22421-21diskstats09:55:123
1812699301,19cyclictest86850irq/129-enp1s0-09:05:153
18126993010,17cyclictest24203-21cstates07:25:113
1811699301,7cyclictest0-21swapper/110:59:001
1811699301,12cyclictest0-21swapper/109:56:541
1811699300,13cyclictest0-21swapper/111:58:021
1811699300,12cyclictest0-21swapper/112:13:091
1810799306,12cyclictest6102-21idleruntime-cro11:50:010
1812699291,25cyclictest26481-21/usr/sbin/munin10:05:153
1812699291,25cyclictest23663-21/usr/sbin/munin08:45:173
1812699291,25cyclictest15135-21/usr/sbin/munin12:10:103
1812699291,25cyclictest10117-21/usr/sbin/munin10:45:173
1812699291,16cyclictest0-21swapper/307:42:013
1811699291,13cyclictest30664-21nfs4_client07:45:151
1811699291,12cyclictest0-21swapper/109:54:241
1811699291,11cyclictest0-21swapper/109:27:061
1811699291,10cyclictest0-21swapper/112:32:491
1811699290,6cyclictest0-21swapper/110:33:521
1811699290,6cyclictest0-21swapper/110:33:521
1811699290,6cyclictest0-21swapper/108:17:461
1812699281,24cyclictest8894-21/usr/sbin/munin09:25:123
1812699280,13cyclictest12050irq/123-ahci[0010:35:013
1812699280,13cyclictest12050irq/123-ahci[0010:35:013
1811699281,6cyclictest0-21swapper/108:21:541
1811699281,13cyclictest0-21swapper/107:55:241
1811699281,12cyclictest0-21swapper/112:27:391
1811699281,12cyclictest0-21swapper/112:23:291
1811699281,12cyclictest0-21swapper/112:09:561
1811699281,12cyclictest0-21swapper/110:20:301
1811699281,11cyclictest0-21swapper/111:34:441
1811699281,11cyclictest0-21swapper/109:07:401
1811699281,10cyclictest0-21swapper/110:27:541
1811699280,6cyclictest0-21swapper/110:52:521
1811699280,5cyclictest0-21swapper/109:16:461
1810799281,20cyclictest0-21swapper/012:30:000
1810799281,19cyclictest0-21swapper/011:22:120
1811699276,3cyclictest0-21swapper/107:16:371
1811699276,3cyclictest0-21swapper/107:16:371
18116992711,3cyclictest0-21swapper/111:35:401
18116992711,3cyclictest0-21swapper/111:35:401
1811699271,12cyclictest0-21swapper/111:16:521
1811699271,12cyclictest0-21swapper/107:35:051
1811699271,11cyclictest0-21swapper/109:23:461
1811699271,11cyclictest0-21swapper/109:23:461
1811699271,10cyclictest0-21swapper/110:17:401
1811699271,10cyclictest0-21swapper/109:35:541
1811699270,5cyclictest0-21swapper/110:49:401
1811699270,12cyclictest0-21swapper/110:40:101
1811699270,12cyclictest0-21swapper/109:33:281
1811699270,10cyclictest0-21swapper/109:03:161
1810799271,20cyclictest0-21swapper/009:16:160
1810799271,19cyclictest0-21swapper/010:28:440
1811699261,9cyclictest0-21swapper/108:36:341
1811699261,13cyclictest0-21swapper/108:06:421
1811699261,11cyclictest0-21swapper/110:08:001
1811699261,11cyclictest0-21swapper/108:33:121
1811699261,11cyclictest0-21swapper/107:29:091
1811699261,10cyclictest0-21swapper/110:35:561
1811699261,10cyclictest0-21swapper/108:13:561
1811699260,10cyclictest0-21swapper/109:11:561
1810799261,19cyclictest0-21swapper/012:37:420
1810799261,18cyclictest0-21swapper/010:32:510
1810799261,18cyclictest0-21swapper/010:32:510
1810799261,17cyclictest0-21swapper/009:10:220
1812199251,12cyclictest0-21swapper/209:40:282
18121992510,3cyclictest8909-21cpu09:25:132
1811699250,11cyclictest0-21swapper/108:45:561
1810799251,4cyclictest1715-21munin-run11:40:000
1810799251,4cyclictest1715-21munin-run11:40:000
1810799251,19cyclictest0-21swapper/011:50:230
1810799251,18cyclictest0-21swapper/012:30:350
1810799251,18cyclictest0-21swapper/011:18:190
1810799251,18cyclictest0-21swapper/011:06:500
1810799251,18cyclictest0-21swapper/009:31:500
1810799251,17cyclictest0-21swapper/012:10:220
1810799251,17cyclictest0-21swapper/010:50:110
1810799251,17cyclictest0-21swapper/010:03:570
1810799251,17cyclictest0-21swapper/009:48:270
1810799251,16cyclictest0-21swapper/012:16:480
1810799251,16cyclictest0-21swapper/011:40:030
1810799251,15cyclictest18208-21/usr/sbin/munin11:00:260
1810799251,12cyclictest23795-21munin-run07:25:010
1810799251,11cyclictest21525-21basename12:25:000
1812199241,21cyclictest0-21swapper/212:21:152
1811699241,10cyclictest0-21swapper/108:53:001
1810799245,6cyclictest3183-21unixbench_multi07:55:280
1810799241,18cyclictest0-21swapper/011:55:560
1810799241,18cyclictest0-21swapper/010:06:150
1810799241,17cyclictest0-21swapper/012:00:320
1810799241,17cyclictest0-21swapper/010:38:050
1810799241,17cyclictest0-21swapper/009:50:340
1810799241,17cyclictest0-21swapper/009:38:380
1810799241,17cyclictest0-21swapper/009:22:360
1810799241,17cyclictest0-21swapper/009:22:360
1811699236,3cyclictest0-21swapper/107:32:171
1811699231,12cyclictest0-21swapper/108:40:501
1810799231,9cyclictest25228-21sed08:45:330
18107992312,3cyclictest24966-21kworker/u8:011:26:290
1810799231,17cyclictest0-21swapper/011:12:530
1810799231,17cyclictest0-21swapper/009:26:080
1810799231,16cyclictest0-21swapper/012:05:070
1810799231,16cyclictest0-21swapper/009:40:020
1810799231,15cyclictest0-21swapper/010:48:400
1812199222,17cyclictest27152-21cut10:05:182
1812199221,20cyclictest0-21swapper/211:34:212
1811699226,3cyclictest0-21swapper/107:20:211
1811699221,18cyclictest291992sleep107:42:211
1811699220,6cyclictest0-21swapper/111:28:421
1810799229,3cyclictest138362proc_pri08:20:260
1810799228,2cyclictest973-21unixbench_multi07:50:320
1810799222,11cyclictest26830-21/usr/sbin/munin07:35:210
1810799221,15cyclictest0-21swapper/008:30:010
1810799221,13cyclictest31576-21kworker/u8:111:31:470
1810799220,20cyclictest0-21swapper/010:10:220
1812199218,3cyclictest32-21ksoftirqd/209:50:282
1812199214,5cyclictest780-21rngd07:49:512
1812199212,5cyclictest6685-21df_inode11:50:212
1812199211,6cyclictest14903-21sh12:10:022
18116992116,3cyclictest0-21swapper/107:10:251
1811699210,19cyclictest0-21swapper/108:55:121
1810799211,7cyclictest3408-21kworker/0:008:00:100
18107992115,3cyclictest8279-21/usr/sbin/munin08:10:280
1810799211,14cyclictest0-21swapper/010:15:010
1812199209,2cyclictest86550irq/126-enp1s0-08:29:352
1812199208,3cyclictest0-21swapper/210:49:542
1812199207,3cyclictest86850irq/129-enp1s0-11:57:392
1812199206,3cyclictest25380-21timerandwakeup07:25:262
1812199202,6cyclictest5714-21/usr/sbin/munin10:35:182
1812199201,8cyclictest0-21swapper/210:55:152
1812199201,7cyclictest0-21swapper/210:50:562
1812199201,6cyclictest21798-21df08:40:122
1812199201,6cyclictest0-21swapper/208:04:002
1812199201,5cyclictest0-21swapper/208:45:262
18121992014,3cyclictest31246-21needreboot09:00:232
1812199201,17cyclictest25914-21perf12:35:012
1812199201,10cyclictest0-21swapper/209:07:042
1810799207,3cyclictest30628-21needreboot07:45:150
1810799206,9cyclictest0-21swapper/007:16:000
1810799206,9cyclictest0-21swapper/007:15:590
1810799201,6cyclictest11-21rcuc/008:35:160
1810799201,6cyclictest10811-21df08:15:130
1810799201,10cyclictest25079-21ntpq07:25:220
1810799200,14cyclictest0-21swapper/007:14:140
1812199191,7cyclictest5661-21ps09:15:222
1812199191,7cyclictest1394-21chrt07:54:462
1812199191,6cyclictest2311-21users10:20:302
1812199191,6cyclictest19454-21grep07:10:242
1812199191,6cyclictest18927-21users12:15:282
1812199191,6cyclictest0-21swapper/211:37:022
1812199191,6cyclictest0-21swapper/211:37:022
18121991916,2cyclictest0-21swapper/209:39:272
1812199191,5cyclictest7756-21ntp_offset09:20:252
1812199191,5cyclictest7756-21ntp_offset09:20:252
1812199191,5cyclictest22658-21fw_packets09:55:152
1812199191,5cyclictest17108-21munin-run12:15:022
18121991914,3cyclictest26952-21tr07:35:072
18121991913,3cyclictest25851-21df11:20:142
1812199191,12cyclictest0-21swapper/211:41:122
1812199191,11cyclictest0-21swapper/209:46:212
1812199191,10cyclictest0-21swapper/208:34:282
1810799197,3cyclictest0-21swapper/008:55:020
1810799195,3cyclictest22521-21ntpq08:40:210
1810799191,6cyclictest6075-21/usr/sbin/munin08:05:250
1810799191,13cyclictest0-21swapper/009:09:500
1810799191,10cyclictest26466-21grep08:50:180
1812199184,4cyclictest23359-21kworker/u8:112:36:272
1812199183,3cyclictest5124-21chrt10:30:232
1812199183,3cyclictest5124-21chrt10:30:232
1812199181,7cyclictest0-21swapper/211:12:122
1812199181,6cyclictest30875-21/usr/sbin/munin10:15:282
1812199181,6cyclictest27448-1kworker/u9:110:03:582
1812199181,5cyclictest14634-21snmpd07:31:582
1812199181,5cyclictest11115-21grep12:00:162
1812199181,5cyclictest0-21swapper/210:41:102
1812199181,4cyclictest3989-21ntp_states10:25:262
18121991813,3cyclictest4319-21cat11:45:152
18121991813,3cyclictest22417-21hwlatdetect12:25:162
18121991812,3cyclictest11086-21/usr/sbin/munin09:30:102
1812199181,11cyclictest0-21swapper/208:21:472
1810799188,1cyclictest0-21swapper/009:00:210
1812199173,3cyclictest86750irq/128-enp1s0-07:20:082
1812199172,4cyclictest14634-21snmpd07:42:002
1812199171,6cyclictest792-21avahi-daemon11:00:512
1812199171,6cyclictest29526-21kernelversion10:10:192
1812199171,5cyclictest0-21swapper/211:09:072
18121991712,3cyclictest29126-21sh08:55:252
18121991712,3cyclictest20748-21hddtemp_smartct07:15:042
18121991712,3cyclictest20748-21hddtemp_smartct07:15:032
1812199171,15cyclictest0-21swapper/208:05:252
18121991711,4cyclictest9652-21/usr/sbin/munin08:10:292
18121991710,4cyclictest10890-21/usr/sbin/munin08:15:142
1810799175,3cyclictest0-21swapper/007:32:570
18121991610,4cyclictest19851-21/usr/sbin/munin08:35:152
1810799161,2cyclictest0-21swapper/007:41:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional