You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-25 - 07:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Jun 25, 2024 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
276082238164,24sleep30-21swapper/319:09:013
276322235164,58sleep20-21swapper/219:09:202
257192203166,24sleep10-21swapper/119:05:051
273842202164,25sleep00-21swapper/019:06:080
27793999188,2cyclictest0-21swapper/200:38:572
27793998884,3cyclictest0-21swapper/221:29:412
27793998882,3cyclictest19811-21tail20:00:312
295242580,4sleep02778099cyclictest20:25:120
111702560,4sleep12778699cyclictest20:55:271
27793995551,3cyclictest0-21swapper/221:47:042
27793995451,2cyclictest0-21swapper/223:52:432
27793995046,3cyclictest0-21swapper/222:20:182
2779899421,19cyclictest0-21swapper/300:33:503
2779899400,18cyclictest0-21swapper/320:01:083
2778699401,29cyclictest21188-21/usr/sbin/munin21:20:111
2778699392,23cyclictest26983-21hddtemp_smartct22:45:181
2778699391,25cyclictest30850-40dnf00:06:441
2778699387,3cyclictest14476-21if_enp2s023:30:161
2778699381,29cyclictest0-21swapper/121:00:591
2778699380,30cyclictest0-21swapper/120:50:551
2778699371,8cyclictest24716-21runrttasks23:55:071
2778699371,5cyclictest836-21runrttasks22:09:301
2778699371,22cyclictest31309-21gltestperf22:55:161
2779899361,18cyclictest0-21swapper/321:50:263
2778699363,25cyclictest780-21rngd19:31:241
27786993615,18cyclictest21091-21grep20:05:211
2778699360,30cyclictest0-21swapper/122:43:271
2778699355,4cyclictest30794-21ntp_states19:15:051
2778699351,4cyclictest768-21in:imjournal21:30:011
2778699351,4cyclictest5395-21ntp_states21:55:281
2778699351,27cyclictest0-21swapper/122:31:401
27786993510,21cyclictest6916-21idleruntime-cro23:15:011
27786993420,6cyclictest8833-21kworker/u8:123:21:181
2778699341,25cyclictest0-21swapper/100:19:121
2778699341,24cyclictest0-21swapper/123:46:321
27786993411,14cyclictest26150-21smart_sda20:15:321
2778699341,10cyclictest0-21swapper/123:00:431
2778699340,14cyclictest0-21swapper/119:58:161
2778699337,3cyclictest3277-21turbostat20:40:021
2778699332,4cyclictest9117-21munin-run23:20:011
2778699332,4cyclictest9117-21munin-run23:20:011
2778699331,4cyclictest10447-21tune2fs00:35:171
27786993312,18cyclictest18812-21sort22:25:331
27786993312,18cyclictest12601-21memory19:45:231
27786993310,19cyclictest22682-21/usr/sbin/munin20:10:171
2778699330,2cyclictest23-21ksoftirqd/122:53:181
2778699329,20cyclictest5588-21/usr/sbin/munin00:25:241
2778699329,20cyclictest5588-21/usr/sbin/munin00:25:241
2778699327,4cyclictest0-21swapper/123:53:381
2778699326,2cyclictest23-21ksoftirqd/123:06:361
2778699324,3cyclictest19320-21taskset20:00:241
2778699322,26cyclictest4098-21munin-run21:55:011
27786993221,3cyclictest0-21swapper/100:13:481
2778699321,7cyclictest0-21swapper/123:40:101
2778699321,5cyclictest7287-21/usr/sbin/munin19:35:271
2778699321,4cyclictest4857-21crond19:30:001
2778699321,4cyclictest31472-21/usr/sbin/munin20:30:321
2778699321,4cyclictest14634-21snmpd19:20:421
2778699321,28cyclictest17302-21awk22:25:011
2778699321,23cyclictest0-21swapper/121:12:091
2778699321,13cyclictest86650irq/127-enp1s0-21:06:571
2778699321,13cyclictest86650irq/127-enp1s0-21:06:571
27786993211,18cyclictest10926-21/usr/sbin/munin22:10:301
27786993211,18cyclictest10926-21/usr/sbin/munin22:10:301
2778099327,12cyclictest4726-21munin-run23:10:010
2778099327,12cyclictest15850-21munin-run19:55:010
2779899311,22cyclictest0-21swapper/321:01:103
2779899311,12cyclictest0-21swapper/321:19:183
2779899310,20cyclictest0-21swapper/321:30:543
2778699319,19cyclictest28517-21timerwakeupswit20:20:301
2778699316,2cyclictest0-21swapper/123:25:131
2778699311,4cyclictest5708-21/usr/sbin/munin20:45:311
2778699311,2cyclictest86750irq/128-enp1s0-21:46:321
27786993111,17cyclictest30387-21/usr/sbin/munin20:25:261
27786993110,18cyclictest14696-21uname19:50:221
2778699310,2cyclictest23716-21kworker/u8:221:37:481
2779899301,22cyclictest0-21swapper/321:11:423
2779899301,19cyclictest16559-21diskmemload22:47:103
2779899301,19cyclictest0-21swapper/320:41:583
2779899301,12cyclictest15663-21df_inode22:20:143
2779899300,23cyclictest0-21swapper/320:52:143
2778699304,3cyclictest5108-21vmstat20:40:341
2778699303,5cyclictest30100-21diskstats19:15:021
2778699302,17cyclictest0-21swapper/119:40:561
2778699301,10cyclictest0-21swapper/100:20:311
27786993010,17cyclictest16686-21if_enp2s023:35:171
127332300,4sleep32779899cyclictest19:45:253
2779899291,8cyclictest16559-21diskmemload00:26:103
2779899291,8cyclictest16559-21diskmemload00:26:103
2779899291,13cyclictest0-21swapper/321:38:303
2779899291,12cyclictest0-21swapper/320:38:463
2779899290,13cyclictest0-21swapper/320:29:083
2778699299,17cyclictest836-21runrttasks21:15:501
2778699299,17cyclictest22812-21latency22:35:211
2778699298,17cyclictest30195-21cpuspeed_turbos21:40:111
2778699295,5cyclictest0-21swapper/121:31:101
2778699294,2cyclictest7867-21cat22:00:311
2778699292,3cyclictest8866-21ntp_states00:30:311
27786992910,16cyclictest28043-21open_inodes00:00:301
2778099291,13cyclictest5490-21awk20:45:010
2779899286,3cyclictest0-21swapper/319:58:163
2779899281,12cyclictest0-21swapper/323:01:583
2779899281,12cyclictest0-21swapper/300:23:183
2779899281,11cyclictest0-21swapper/319:11:343
2779899280,21cyclictest0-21swapper/323:41:503
2779899280,11cyclictest0-21swapper/323:56:303
2778699285,3cyclictest0-21swapper/122:15:181
2779899276,3cyclictest0-21swapper/323:34:463
2779899271,9cyclictest0-21swapper/322:18:543
27798992718,2cyclictest0-21swapper/323:25:343
2779899271,12cyclictest0-21swapper/320:34:063
27798992711,2cyclictest0-21swapper/319:26:343
2779899271,12cyclictest0-21swapper/300:11:023
2779899271,11cyclictest0-21swapper/322:58:543
2779899271,11cyclictest0-21swapper/322:25:103
2779899271,11cyclictest0-21swapper/320:56:223
2779899271,11cyclictest0-21swapper/320:21:343
2779899271,11cyclictest0-21swapper/300:18:143
2779899271,10cyclictest0-21swapper/320:07:343
2779899270,13cyclictest0-21swapper/321:56:143
2779899270,11cyclictest0-21swapper/323:09:023
2779899270,11cyclictest0-21swapper/322:12:063
2779899270,11cyclictest0-21swapper/322:12:063
27793992723,3cyclictest0-21swapper/221:43:562
2779899269,3cyclictest20538-21kernelversion22:30:223
2779899265,8cyclictest0-21swapper/319:21:563
2779899261,12cyclictest0-21swapper/323:52:063
27798992611,2cyclictest0-21swapper/323:10:303
2779899261,12cyclictest0-21swapper/322:09:183
2779899261,11cyclictest0-21swapper/322:39:343
2779899261,10cyclictest0-21swapper/319:16:343
27798992610,3cyclictest25606-21unixbench_multi22:40:303
2779899260,11cyclictest0-21swapper/320:48:303
2779899260,10cyclictest0-21swapper/321:43:483
2779899260,10cyclictest0-21swapper/319:50:443
2779899255,3cyclictest23755-21ntp_states20:10:263
2779899251,17cyclictest0-21swapper/323:47:423
2779899251,17cyclictest0-21swapper/300:00:303
2779899251,12cyclictest0-21swapper/323:38:063
2779899251,11cyclictest0-21swapper/321:09:583
2779899251,11cyclictest0-21swapper/321:09:583
2779899251,11cyclictest0-21swapper/319:41:423
2779399251,9cyclictest0-21swapper/219:12:042
76042240,4sleep230-21rcuc/200:30:012
76042240,4sleep230-21rcuc/200:30:012
2779899247,3cyclictest0-21swapper/322:03:503
2779899242,15cyclictest32576-21df_inode21:45:183
2779899241,11cyclictest0-21swapper/320:19:583
2779899241,10cyclictest0-21swapper/300:36:543
2779899240,9cyclictest0-21swapper/321:28:283
2779899240,15cyclictest0-21swapper/322:50:283
2779899233,3cyclictest86850irq/129-enp1s0-23:21:243
2779899231,10cyclictest0-21swapper/323:15:583
2779899231,10cyclictest0-21swapper/323:15:583
2779899231,10cyclictest0-21swapper/321:23:263
2778099231,3cyclictest0-21swapper/022:15:420
27780992311,5cyclictest836-21runrttasks22:42:030
2778099231,11cyclictest32692-21idleruntime-cro23:00:010
2779899220,7cyclictest0-21swapper/319:37:543
2779399220,13cyclictest0-21swapper/221:24:202
6922210,2sleep20-21swapper/200:12:432
302852210,4sleep011-21rcuc/021:40:110
2779399216,9cyclictest0-21swapper/220:10:522
2779399213,3cyclictest88050irq/133-enp2s0-20:08:002
2779399212,3cyclictest15314-21kworker/2:000:00:022
2779399211,13cyclictest0-21swapper/222:44:482
2779399211,12cyclictest0-21swapper/220:34:242
2779399211,11cyclictest11686-21runrttasks22:10:242
2779399211,11cyclictest11686-21runrttasks22:10:242
2778099211,7cyclictest0-21swapper/023:29:540
2778099211,6cyclictest8999-1kworker/u9:123:46:430
2778099211,11cyclictest0-21swapper/022:31:040
318242201,15sleep031891-21fschecks_count20:30:150
2779899200,8cyclictest0-21swapper/319:31:363
2779899200,2cyclictest0-21swapper/300:07:143
2779399209,2cyclictest88050irq/133-enp2s0-20:25:322
2779399207,2cyclictest25346-21kworker/2:021:57:262
2779399201,9cyclictest30-21rcuc/200:15:342
2779399201,4cyclictest0-21swapper/200:05:172
2778099203,5cyclictest0-21swapper/020:35:240
2778099202,7cyclictest7561-21munin-run00:30:000
2778099202,7cyclictest7561-21munin-run00:30:000
2778099202,10cyclictest15928-21/usr/sbin/munin23:35:210
2778099201,9cyclictest26445-21grep20:15:350
2778099201,5cyclictest0-21swapper/000:08:120
2778099201,11cyclictest24724-21/usr/sbin/munin23:55:210
2778099201,11cyclictest14634-21snmpd21:11:000
2778099201,10cyclictest9309-21grep00:30:370
2778099201,10cyclictest11681-21sshd00:36:430
2778099201,10cyclictest10342-21needreboot23:20:230
2779399197,3cyclictest86550irq/126-enp1s0-22:45:252
27793991915,3cyclictest32-21ksoftirqd/200:04:332
27793991913,3cyclictest13461-21df22:15:162
2779399191,15cyclictest22907-21head22:35:222
2779399191,12cyclictest0-21swapper/223:30:252
2779399191,11cyclictest0-21swapper/221:52:002
2778099199,2cyclictest26937-21/usr/sbin/munin00:00:260
2778099198,8cyclictest24970-21unixbench_singl21:25:370
2778099197,3cyclictest86750irq/128-enp1s0-22:45:120
2778099192,3cyclictest11136-21expr22:10:160
2778099192,3cyclictest11136-21expr22:10:160
2778099191,8cyclictest24166-21sed20:10:320
2778099191,5cyclictest14634-21snmpd19:40:430
2778099191,5cyclictest0-21swapper/021:33:020
2778099191,4cyclictest608-21/usr/sbin/munin19:20:130
27780991914,3cyclictest1782-21find23:00:250
27780991911,5cyclictest19848-21kworker/0:023:44:350
2778099191,10cyclictest27075-21/usr/sbin/munin20:20:170
2778099191,10cyclictest14634-21snmpd22:36:150
2778099191,10cyclictest14634-21snmpd00:11:350
2778099191,10cyclictest10237-21unixbench_multi22:05:310
2779399188,2cyclictest86550irq/126-enp1s0-23:45:272
2779399186,3cyclictest0-21swapper/223:00:122
2779399186,3cyclictest0-21swapper/220:21:282
2779399186,2cyclictest87950irq/132-enp2s0-21:01:162
2779399182,3cyclictest12050irq/123-ahci[0019:40:252
2779399181,7cyclictest9254-21latency_hist19:40:012
2779399181,7cyclictest17668-21cpuspeed_turbos22:25:132
2779399181,6cyclictest18348-21expr23:40:112
2779399181,6cyclictest0-21swapper/223:35:232
2779399181,6cyclictest0-21swapper/222:55:362
27793991815,2cyclictest88150irq/134-enp2s0-19:23:322
2779399181,4cyclictest20952-21crond21:20:012
27793991813,3cyclictest6423-21grep23:10:312
27793991811,4cyclictest7184-21cut22:00:172
2779399181,11cyclictest0-21swapper/223:18:182
2779399181,11cyclictest0-21swapper/223:18:182
2779399181,11cyclictest0-21swapper/200:33:112
2778099188,2cyclictest14634-21snmpd21:56:080
2778099187,2cyclictest4965-21sed00:20:360
2778099187,2cyclictest0-21swapper/019:37:420
2778099186,5cyclictest1613-21sshd21:47:180
2778099183,3cyclictest29100-21smart_sda21:35:260
2778099183,12cyclictest2449-21wc00:15:350
2778099181,9cyclictest14634-21snmpd21:51:070
2778099181,8cyclictest14081-21diskstats23:30:110
2778099181,8cyclictest0-21swapper/022:50:200
2778099181,7cyclictest0-21swapper/022:24:530
2778099181,7cyclictest0-21swapper/021:22:520
2778099181,4cyclictest14634-21snmpd21:05:590
2778099181,4cyclictest14634-21snmpd21:05:590
27780991813,3cyclictest7079-21awk19:35:020
27780991812,3cyclictest7295-21cpuspeed_turbos23:15:130
27780991812,3cyclictest7295-21cpuspeed_turbos23:15:130
2778099181,15cyclictest9884-21perf20:55:010
27780991810,7cyclictest496-21jbd2/dm-0-819:57:470
2778099180,9cyclictest0-21swapper/023:50:200
2779399179,5cyclictest848-21sssd_nss22:50:222
2779399175,4cyclictest12346-21gltestperf19:45:172
2779399174,6cyclictest32-21ksoftirqd/223:20:252
2779399172,12cyclictest1774-21grep20:35:142
2779399171,9cyclictest0-21swapper/223:05:022
2779399171,9cyclictest0-21swapper/222:30:162
2779399171,6cyclictest0-21swapper/221:30:242
2779399171,6cyclictest0-21swapper/219:50:322
2779399171,6cyclictest0-21swapper/200:20:142
2779399171,5cyclictest0-21swapper/221:05:312
2779399171,5cyclictest0-21swapper/221:05:312
27793991713,3cyclictest0-21swapper/221:38:382
27793991713,2cyclictest30-21rcuc/220:50:012
27793991712,3cyclictest4909-21cat20:40:332
27793991712,3cyclictest12052-21fschecks_time23:25:162
27793991711,3cyclictest3265-21fschecks_count19:25:162
2779399171,11cyclictest0-21swapper/222:05:132
2778099178,3cyclictest0-21swapper/021:16:440
2778099176,3cyclictest86550irq/126-enp1s0-20:05:260
2778099172,14cyclictest0-21swapper/019:20:010
2778099171,7cyclictest0-21swapper/022:27:090
2778099171,2cyclictest13199-21grep19:45:330
27780991712,3cyclictest13884-21uptime21:00:300
27780991712,3cyclictest10788-21hwlatdetect20:55:200
2778099171,13cyclictest6516-21/usr/sbin/munin22:00:170
27780991711,3cyclictest2909-21bc19:25:110
2778099171,10cyclictest0-21swapper/019:10:170
200672171,2sleep00-21swapper/020:02:450
58622161,2sleep20-21swapper/219:30:202
2779399164,2cyclictest88250irq/135-enp2s0-20:54:262
2779399161,7cyclictest0-21swapper/221:10:212
2779399161,6cyclictest10098-21/usr/sbin/munin20:55:152
27793991612,3cyclictest0-21swapper/220:18:202
2778099161,5cyclictest0-21swapper/023:12:000
27780991611,3cyclictest7192-21sed20:45:410
2779399159,3cyclictest30440-21cstates19:15:032
2779399158,4cyclictest17333-21sendmail_mailtr19:55:312
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional