You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 01:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sun May 04, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
295842236166,57sleep20-21swapper/207:09:252
294622234162,58sleep10-21swapper/107:07:511
293572233160,60sleep00-21swapper/007:06:290
295502218145,25sleep30-21swapper/307:08:583
235362640,3sleep19-21rcu_preempt08:05:231
189362570,4sleep121-21rcuc/107:55:201
62972530,2sleep00-21swapper/009:55:170
2971999399,14cyclictest2110-21munin-run11:00:000
2971999380,15cyclictest0-21swapper/010:37:410
2971999377,14cyclictest31679-21munin-run09:40:010
29719993722,3cyclictest13108-21munin-run11:25:010
29736993510,15cyclictest14189-21cat09:00:013
2972599353,23cyclictest7396-1kworker/1:2H11:20:561
2971999357,13cyclictest24404-21awk08:10:020
2973699343,22cyclictest16557-21kworker/u8:010:22:393
2971999341,26cyclictest0-21swapper/010:26:530
2971999340,14cyclictest0-21swapper/007:16:060
2973699332,22cyclictest17462-21kworker/u8:011:35:433
2972599331,21cyclictest0-21swapper/110:35:011
2971999336,12cyclictest445-21awk12:10:000
2971999331,13cyclictest8351-21kworker/0:211:34:150
2971999330,14cyclictest0-21swapper/010:23:170
2972599325,18cyclictest780-21rngd08:42:081
2972599325,18cyclictest780-21rngd08:42:081
2972599322,21cyclictest0-21swapper/107:54:401
2971999327,12cyclictest27289-21awk09:30:010
2971999321,15cyclictest22535-21kworker/0:309:32:420
2971999321,15cyclictest16167-21kworker/0:209:10:340
2973699314,16cyclictest13672-21runrttasks12:39:433
2972599314,15cyclictest780-21rngd12:16:441
2971999311,15cyclictest7825-21kworker/0:310:00:230
2971999310,24cyclictest22535-21kworker/0:309:20:520
2971999310,13cyclictest30611-21kworker/0:012:25:150
2971999310,12cyclictest20989-21kworker/0:310:33:310
2973699305,15cyclictest29242-21grep12:00:193
2973699301,13cyclictest0-21swapper/309:30:193
2971999302,11cyclictest5934-21cat12:20:190
2971999301,14cyclictest2152-21kworker/0:307:28:360
2971999301,14cyclictest13727-21awk12:40:010
2971999301,13cyclictest9650-21tune2fs07:35:180
2971999301,11cyclictest0-21swapper/007:44:300
2971999300,12cyclictest9414-21kworker/0:009:00:360
2971999300,11cyclictest30969-21kworker/0:008:35:360
2973699299,3cyclictest1176-21cpuspeed_turbos08:30:143
2973699293,8cyclictest15015-40dnf11:26:333
2973699291,10cyclictest0-21swapper/308:35:033
29725992920,2cyclictest7759-21kworker/u8:209:55:501
2972599291,21cyclictest0-21swapper/112:06:331
2971999291,22cyclictest0-21swapper/011:07:490
2971999291,15cyclictest0-21swapper/008:33:260
2971999291,13cyclictest12507-21munin-run10:10:000
2971999290,15cyclictest8853-21kworker/0:007:33:460
2971999290,15cyclictest8853-21kworker/0:007:33:460
2971999290,13cyclictest14509-21kworker/0:010:17:210
2971999290,12cyclictest30611-21kworker/0:012:30:230
2973699288,3cyclictest0-21swapper/311:50:403
2973699285,15cyclictest2208-21wc11:00:023
29725992818,2cyclictest6163-21kworker/u8:011:12:421
2972599281,6cyclictest0-21swapper/109:37:431
2972599281,10cyclictest0-21swapper/108:55:541
29719992810,15cyclictest24101-21idleruntime-cro11:50:000
2971999280,13cyclictest848-21sssd_nss08:25:000
2971999280,13cyclictest0-21swapper/010:46:130
2971999280,12cyclictest32081-21kworker/0:210:53:050
2971999280,12cyclictest32081-21kworker/0:210:53:050
2973699277,3cyclictest0-21swapper/308:10:563
2973699274,3cyclictest0-21swapper/307:38:493
2973699271,9cyclictest0-21swapper/307:53:223
29725992718,3cyclictest0-21swapper/110:44:511
2972599270,12cyclictest0-21swapper/107:40:161
2971999271,13cyclictest0-21swapper/008:56:140
2971999271,11cyclictest0-21swapper/012:18:450
2971999271,11cyclictest0-21swapper/012:13:330
2971999270,25cyclictest25996-21kworker/0:111:57:060
2971999270,25cyclictest25996-21kworker/0:111:57:050
2971999270,12cyclictest29219-21kworker/0:209:42:580
2971999270,12cyclictest19604-21kworker/0:011:42:490
2971999270,11cyclictest8351-21kworker/0:211:28:310
2973699261,8cyclictest0-21swapper/308:52:413
2973699261,23cyclictest0-21swapper/311:55:263
2973699261,23cyclictest0-21swapper/311:55:263
2973699261,20cyclictest0-21swapper/310:18:053
2973699261,10cyclictest0-21swapper/309:11:033
2972999261,11cyclictest0-21swapper/207:11:402
29725992620,3cyclictest30701-21perf12:05:011
2972599261,9cyclictest0-21swapper/110:46:321
29725992617,3cyclictest0-21swapper/111:59:341
29725992617,3cyclictest0-21swapper/111:59:341
2972599261,19cyclictest0-21swapper/112:10:201
2972599261,17cyclictest19499-21taskset11:37:361
2972599261,12cyclictest0-21swapper/111:08:321
2972599261,12cyclictest0-21swapper/107:37:541
29719992611,1cyclictest0-21swapper/011:15:250
2971999260,12cyclictest19604-21kworker/0:011:54:590
2971999260,10cyclictest28337-21kworker/0:212:02:550
2971999260,10cyclictest12294-21kworker/0:110:11:230
45662252,3sleep14569-40dnf07:24:341
2973699257,7cyclictest18547-21cat09:10:013
2973699255,3cyclictest0-21swapper/308:47:453
2973699253,9cyclictest8329-21/usr/sbin/munin10:00:223
2973699253,10cyclictest12730-21/usr/sbin/munin10:10:213
2973699252,14cyclictest5499-21diskstats12:20:163
29736992512,8cyclictest14634-21snmpd08:25:033
2973699251,21cyclictest0-21swapper/311:14:203
2973699251,20cyclictest0-21swapper/310:30:133
2973699250,23cyclictest0-21swapper/311:20:273
2972599258,9cyclictest0-21swapper/110:17:401
2972599257,6cyclictest0-21swapper/110:21:141
2972599255,5cyclictest0-21swapper/111:48:181
2972599255,12cyclictest780-21rngd08:33:141
2972599254,11cyclictest20931-40dnf10:25:351
2972599251,17cyclictest0-21swapper/107:10:211
2972599251,16cyclictest0-21swapper/110:55:511
2972599251,11cyclictest0-21swapper/107:16:421
2971999251,12cyclictest29219-21kworker/0:209:50:260
2971999251,11cyclictest25434-21kworker/0:111:03:390
2971999251,11cyclictest17739-21kworker/0:308:00:280
2971999251,11cyclictest0-21swapper/011:10:370
2973699248,3cyclictest5004-21ntp_states09:50:233
2973699246,7cyclictest22328-21cat08:05:023
2973699245,8cyclictest18023-21/usr/sbin/munin07:55:233
2973699241,9cyclictest0-21swapper/309:40:273
29736992415,6cyclictest20931-40dnf10:25:343
2973699241,20cyclictest0-21swapper/312:32:253
2973699241,20cyclictest0-21swapper/312:15:533
2973699241,20cyclictest0-21swapper/312:14:283
2973699241,14cyclictest0-21swapper/307:11:083
2973699241,13cyclictest0-21swapper/310:50:283
2973699241,13cyclictest0-21swapper/310:50:283
2973699241,11cyclictest2341-21/usr/sbin/munin11:00:183
2973699241,11cyclictest12068-21unixbench_singl10:05:273
2972599245,11cyclictest29832-21/usr/sbin/munin08:20:201
2972599241,9cyclictest780-21rngd09:05:241
2972599241,4cyclictest811-21latency_hist08:30:011
2972599241,21cyclictest0-21swapper/109:45:201
2972599241,21cyclictest0-21swapper/109:45:201
2972599241,20cyclictest24175-21ntp_states09:20:241
2972599241,12cyclictest0-21swapper/109:01:201
2972599241,12cyclictest0-21swapper/108:13:301
2972599241,11cyclictest836-21runrttasks08:37:041
2972599241,10cyclictest19319-21tail09:10:191
2971999241,7cyclictest14634-21snmpd07:20:350
2971999241,12cyclictest0-21swapper/008:48:380
2971999241,11cyclictest0-21swapper/008:50:380
2971999240,9cyclictest17739-21kworker/0:308:15:230
2971999240,12cyclictest22535-21kworker/0:309:17:360
2971999240,11cyclictest29219-21kworker/0:209:47:220
2971999240,11cyclictest29219-21kworker/0:209:47:220
2971999240,10cyclictest0-21swapper/011:35:490
2971999240,10cyclictest0-21swapper/008:42:380
2971999240,10cyclictest0-21swapper/008:42:380
2973699238,3cyclictest0-21swapper/310:42:163
2973699236,8cyclictest0-21swapper/310:45:563
2973699236,3cyclictest0-21swapper/309:03:113
2973699235,3cyclictest0-21swapper/307:25:313
2973699231,5cyclictest0-21swapper/311:05:243
2973699231,3cyclictest0-21swapper/308:08:123
2973699231,19cyclictest18488-21diskmemload09:49:123
2973699231,19cyclictest18488-21diskmemload09:49:123
2973699231,19cyclictest0-21swapper/309:36:113
2973699231,11cyclictest0-21swapper/311:34:433
2973699231,10cyclictest0-21swapper/312:06:273
2973699231,10cyclictest0-21swapper/310:37:423
2973699231,10cyclictest0-21swapper/309:20:553
29729992310,4cyclictest18499-21basename09:10:012
2972599239,3cyclictest0-21swapper/107:45:201
2972599238,4cyclictest0-21swapper/112:35:541
2972599237,7cyclictest0-21swapper/108:15:401
2972599234,9cyclictest780-21rngd09:41:481
2972599231,4cyclictest0-21swapper/111:44:381
2972599231,19cyclictest0-21swapper/109:16:511
2972599231,16cyclictest0-21swapper/110:14:361
2972599231,16cyclictest0-21swapper/110:03:531
2972599231,13cyclictest0-21swapper/109:25:231
2972599231,11cyclictest0-21swapper/107:26:081
2972599231,10cyclictest0-21swapper/110:51:141
2972599231,10cyclictest0-21swapper/110:51:141
2972599230,17cyclictest0-21swapper/107:30:161
2972599230,17cyclictest0-21swapper/107:30:161
2971999231,13cyclictest17492-21kworker/0:107:57:080
2971999231,12cyclictest23192-21kworker/0:010:40:410
2971999231,11cyclictest0-21swapper/008:29:500
2971999230,14cyclictest27376-21kworker/0:007:15:000
2971999230,13cyclictest13929-21kworker/0:109:09:480
2971999230,10cyclictest0-21swapper/008:10:260
2973699223,8cyclictest88050irq/133-enp2s0-07:49:393
2973699223,10cyclictest26165-21sed09:25:193
2973699221,2cyclictest0-21swapper/311:46:343
2973699221,12cyclictest0-21swapper/308:41:273
2973699221,12cyclictest0-21swapper/308:41:273
2973699221,10cyclictest19930-21/usr/sbin/munin11:40:213
2973699221,10cyclictest18488-21diskmemload09:55:093
2973699220,6cyclictest0-21swapper/308:20:083
2972999221,9cyclictest0-21swapper/208:31:402
2972599229,2cyclictest88050irq/133-enp2s0-08:00:041
2972599227,9cyclictest0-21swapper/109:35:001
2972599226,6cyclictest8451-21pmu-power12:25:231
2972599225,9cyclictest0-21swapper/111:03:031
2972599225,3cyclictest0-21swapper/112:30:561
2972599224,9cyclictest0-21swapper/111:50:161
2972599222,10cyclictest15015-40dnf11:26:311
2972599221,11cyclictest0-21swapper/110:39:481
2972599220,15cyclictest86750irq/128-enp1s0-10:08:121
2971999221,11cyclictest0-21swapper/007:48:540
321952210,3sleep230-21rcuc/209:40:182
2973699214,6cyclictest8936-21/usr/sbin/munin11:15:203
2973699214,3cyclictest0-21swapper/307:40:353
2973699212,15cyclictest21016-21cpuspeed_turbos09:15:133
2973699211,14cyclictest0-21swapper/308:17:523
2973699210,12cyclictest0-21swapper/307:19:543
2972599215,3cyclictest0-21swapper/108:47:321
2972599212,10cyclictest10878-21sed08:50:211
29725992115,4cyclictest0-21swapper/112:20:281
2972599211,18cyclictest0-21swapper/111:15:061
2972599211,12cyclictest0-21swapper/111:31:211
2971999210,12cyclictest17492-21kworker/0:107:52:420
2973699204,3cyclictest7693-21diskstats12:25:153
2972599201,10cyclictest0-21swapper/109:50:271
2973699194,3cyclictest0-21swapper/307:33:383
2973699194,3cyclictest0-21swapper/307:33:373
2973699191,15cyclictest2649-21/usr/sbin/munin07:20:193
2972999191,9cyclictest0-21swapper/210:48:242
2972999191,6cyclictest32-21ksoftirqd/208:23:082
29729991912,4cyclictest13219-21pluginstate08:55:232
287332190,5sleep232-21ksoftirqd/212:00:162
243492191,2sleep224444-21if_enp1s010:35:192
39342181,2sleep20-21swapper/209:50:152
2972999189,5cyclictest848-21sssd_nss08:10:172
2972999184,3cyclictest32-21ksoftirqd/207:54:392
2972999181,9cyclictest0-21swapper/210:00:302
2972999181,8cyclictest0-21swapper/211:54:172
2972999181,7cyclictest7262-21grep09:55:282
2972999181,6cyclictest780-21rngd08:09:332
2972999181,6cyclictest0-21swapper/212:26:092
2972999181,4cyclictest0-21swapper/210:05:052
29729991813,3cyclictest3991-21ntp_kernel_pll_12:15:212
29729991813,3cyclictest3715-21basename09:50:012
29729991813,3cyclictest3715-21basename09:50:012
29729991813,3cyclictest31177-21expr12:05:142
29729991813,3cyclictest27510-21unixbench_singl10:40:432
29729991812,3cyclictest8581-21unixbench_singl07:30:282
29729991812,3cyclictest8581-21unixbench_singl07:30:282
29729991812,3cyclictest19868-21cat11:40:012
29729991811,4cyclictest19093-21grep10:25:022
2972999181,10cyclictest0-21swapper/211:07:022
139682181,2sleep20-21swapper/207:45:182
2972999174,7cyclictest31001-40dnf08:25:032
2972999174,5cyclictest8956-21sendmail_mailtr08:45:252
2972999173,5cyclictest31827-1kworker/u9:209:20:462
2972999172,7cyclictest780-21rngd08:00:402
2972999171,8cyclictest0-21swapper/211:48:312
2972999171,7cyclictest10524-21users11:15:282
2972999171,6cyclictest0-21swapper/211:01:552
2972999171,6cyclictest0-21swapper/208:40:212
2972999171,6cyclictest0-21swapper/208:40:212
2972999171,4cyclictest31827-1kworker/u9:209:10:082
2972999171,4cyclictest0-21swapper/212:30:482
2972999171,3cyclictest0-21swapper/210:26:042
29729991713,2cyclictest9913-21hddtemp_smartct07:35:192
29729991713,2cyclictest2116-21perf11:00:012
29729991712,3cyclictest6279-21cat07:25:282
29729991712,3cyclictest20582-21gltestperf11:40:172
29729991712,3cyclictest1703-21sendmail_mailtr07:15:252
2972999171,15cyclictest0-21swapper/212:36:292
29729991711,3cyclictest2832-21cstates07:20:152
29729991711,3cyclictest13806-21open_files10:10:232
2972999171,11cyclictest0-21swapper/210:35:012
2972999171,10cyclictest0-21swapper/211:30:162
2972999171,10cyclictest0-21swapper/211:21:112
2972999166,3cyclictest86550irq/126-enp1s0-08:35:122
2972999166,3cyclictest86550irq/126-enp1s0-07:40:142
2972999162,6cyclictest0-21swapper/212:10:142
2972999161,6cyclictest14141-21irqstats11:25:182
2972999161,6cyclictest0-21swapper/212:20:152
2972999161,6cyclictest0-21swapper/208:15:342
2972999161,5cyclictest0-21swapper/209:28:552
2972999161,3cyclictest0-21swapper/209:39:232
2972999161,2cyclictest0-21swapper/209:19:292
29729991612,2cyclictest158750irq/12-ttyS011:55:222
29729991612,2cyclictest158750irq/12-ttyS011:55:222
29729991611,3cyclictest18338-21ls07:55:142
29729991611,3cyclictest16229-21sed10:15:272
29729991611,3cyclictest15684-21timerandwakeup09:00:272
29729991611,2cyclictest836-21runrttasks09:30:102
2972999160,7cyclictest0-21swapper/211:10:142
29729991511,2cyclictest30984-21hddtemp_smartct10:50:182
29729991511,2cyclictest30984-21hddtemp_smartct10:50:182
29729991510,3cyclictest9936-21cpu08:50:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional