You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-03 - 00:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri May 02, 2025 12:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24242240164,23sleep00-21swapper/007:08:070
23032237164,24sleep30-21swapper/307:06:343
25312236164,58sleep10-21swapper/107:09:301
23092232164,55sleep20-21swapper/207:06:382
1011721440,4sleep2266999cyclictest08:35:292
2669999390,2cyclictest0-21swapper/211:14:112
2669997269,2cyclictest0-21swapper/210:38:042
169282540,2sleep29-21rcu_preempt12:35:012
145192470,4sleep3267599cyclictest11:15:153
173842450,2sleep00-21swapper/011:20:180
107062450,1sleep00-21swapper/011:05:200
2657994010,14cyclictest19955-21munin-run10:15:020
265799331,18cyclictest27244-21kworker/0:308:11:200
265799321,15cyclictest22157-21awk10:20:020
265799320,12cyclictest25772-21kworker/0:209:20:180
265799316,12cyclictest17758-21munin-run10:10:020
265799312,14cyclictest1729-21munin-run08:20:010
265799311,13cyclictest22489-21kworker/0:011:34:400
265799301,18cyclictest4241-21kworker/0:309:42:120
265799301,11cyclictest14904-21kworker/0:211:25:240
266599292,17cyclictest1010-21hddtemp_smartct09:30:211
265799291,13cyclictest10520-21awk08:40:000
265799283,12cyclictest25298-21awk08:00:020
2657992811,15cyclictest12369-21munin-run12:25:020
265799281,10cyclictest6471-21kworker/0:010:04:200
2657992810,16cyclictest14311-21munin-run07:35:010
265799280,9cyclictest1153-21kworker/0:212:06:200
265799271,11cyclictest17688-21kworker/0:110:37:240
265799271,10cyclictest7766-21kworker/0:312:16:100
265799264,10cyclictest32497-21perf09:30:000
265799261,15cyclictest0-21swapper/007:45:260
265799261,10cyclictest3517-21kworker/0:112:10:240
2675992510,9cyclictest759-21dbus-daemon11:04:363
2665992513,3cyclictest0-21swapper/110:00:151
265799259,3cyclictest8220-21kworker/0:108:45:140
265799251,16cyclictest17688-21kworker/0:110:42:100
2657992511,3cyclictest23548-21kworker/0:009:10:160
2657992510,3cyclictest31261-21kworker/0:011:56:320
2657992510,3cyclictest31261-21kworker/0:011:56:320
93742240,12sleep10-21swapper/111:02:431
267599245,10cyclictest25838-21swap10:25:253
266599247,4cyclictest8831-21sshd09:48:361
266599247,4cyclictest8831-21sshd09:48:361
266599244,12cyclictest86750irq/128-enp1s0-12:20:111
266599242,3cyclictest5786-21munin-run12:10:011
266599241,7cyclictest23701-21diskmemload10:30:241
266599241,22cyclictest0-21swapper/109:39:531
266599241,17cyclictest0-21swapper/112:19:071
266599241,17cyclictest0-21swapper/108:45:161
266599241,16cyclictest0-21swapper/112:10:441
266599241,16cyclictest0-21swapper/110:58:461
266599241,13cyclictest30936-21perf10:40:011
265799247,3cyclictest22489-21kworker/0:011:35:220
265799241,10cyclictest24697-21kworker/0:111:50:360
267599236,3cyclictest0-21swapper/311:26:523
267599234,10cyclictest19515-21kworker/3:111:30:293
267599231,4cyclictest0-21swapper/309:31:403
267599231,13cyclictest0-21swapper/308:15:123
267599231,10cyclictest2579-21users10:45:273
267599230,11cyclictest365-21open_files11:55:243
267599230,11cyclictest365-21open_files11:55:243
266999232,1cyclictest0-21swapper/209:37:212
2669992317,3cyclictest10477-21cat12:20:112
266599232,11cyclictest29698-21munin-run08:10:021
2665992317,3cyclictest10002-21unixbench-2d08:35:291
266599231,4cyclictest22059-21sed07:50:181
2665992314,2cyclictest23701-21diskmemload11:32:401
266599231,20cyclictest0-21swapper/112:39:301
266599231,16cyclictest0-21swapper/112:34:471
266599231,16cyclictest0-21swapper/111:17:481
265799238,3cyclictest6471-21kworker/0:009:51:400
265799238,3cyclictest11934-21kworker/0:207:38:300
265799238,3cyclictest11934-21kworker/0:207:38:300
265799237,7cyclictest4998-21kworker/0:110:53:120
265799237,7cyclictest4998-21kworker/0:110:53:120
265799237,3cyclictest8220-21kworker/0:108:59:040
265799232,10cyclictest1979-21kworker/0:209:36:180
265799231,12cyclictest23417-21ntpq10:20:220
2657992310,2cyclictest385-21kworker/0:010:59:020
267599228,4cyclictest0-21swapper/310:42:483
267599226,4cyclictest0-21swapper/310:57:093
267599222,18cyclictest39-21rcuc/311:45:163
267599221,8cyclictest0-21swapper/310:13:513
267599221,5cyclictest0-21swapper/311:10:133
267599221,12cyclictest18928-21/usr/sbin/munin07:45:273
267599221,11cyclictest15786-21/usr/sbin/munin10:05:133
267599221,10cyclictest23321-21/usr/sbin/munin11:35:173
267599221,10cyclictest10332-21latency_hist12:20:023
267599221,10cyclictest0-21swapper/311:44:123
267599220,8cyclictest0-21swapper/310:38:333
266999221,6cyclictest0-21swapper/212:16:382
2669992215,4cyclictest27493-21tr11:45:122
266999221,3cyclictest0-21swapper/210:17:122
2669992211,3cyclictest31827-1kworker/u9:211:23:542
2669992210,3cyclictest31827-1kworker/u9:210:07:532
266599222,4cyclictest0-21swapper/110:49:141
2665992218,3cyclictest0-21swapper/108:02:391
266599221,6cyclictest0-21swapper/109:50:521
266599221,5cyclictest0-21swapper/110:41:201
2665992213,3cyclictest0-21swapper/107:45:231
266599221,18cyclictest30607-21wc09:25:141
266599221,16cyclictest0-21swapper/111:54:181
266599221,16cyclictest0-21swapper/111:29:551
266599221,15cyclictest0-21swapper/108:31:001
266599221,15cyclictest0-21swapper/108:23:591
266599221,12cyclictest11130-21grep08:40:181
266599221,12cyclictest11130-21grep08:40:181
266599221,10cyclictest0-21swapper/112:29:091
266599221,10cyclictest0-21swapper/111:55:111
266599221,10cyclictest0-21swapper/111:55:111
266599221,10cyclictest0-21swapper/111:10:061
265799229,4cyclictest7614-21kworker/0:007:21:260
265799229,3cyclictest4241-21kworker/0:309:45:300
265799229,3cyclictest4241-21kworker/0:309:45:300
265799229,3cyclictest31261-21kworker/0:012:00:460
265799229,2cyclictest4517-21kworker/0:207:16:460
265799226,4cyclictest28001-21kworker/0:009:19:520
265799224,3cyclictest11695-21kworker/0:111:15:200
265799222,9cyclictest12591-21kworker/0:212:30:540
265799222,10cyclictest17688-21kworker/0:110:45:420
265799221,13cyclictest0-21swapper/008:20:210
265799221,10cyclictest4998-21kworker/0:111:00:160
265799221,10cyclictest14272-21kworker/0:007:40:140
265799220,11cyclictest19049-21kworker/0:009:01:540
267599211,8cyclictest0-21swapper/309:21:483
267599211,7cyclictest1967-21fschecks_count12:00:163
267599211,3cyclictest814-21sshd08:55:353
267599211,11cyclictest15171-21users10:00:313
267599211,11cyclictest11558-21sshd11:07:363
267599211,10cyclictest3724-21ps09:35:243
267599210,11cyclictest12220-21kernelversion09:55:223
267599210,11cyclictest11457-21idleruntime08:40:193
267599210,11cyclictest11457-21idleruntime08:40:193
267599210,10cyclictest0-21swapper/310:20:143
266999215,7cyclictest19894-21fschecks_count09:00:182
266999211,6cyclictest0-21swapper/211:34:072
266599217,8cyclictest0-21swapper/111:45:481
266599212,10cyclictest581-21systemd-journal09:10:021
2665992116,3cyclictest0-21swapper/107:20:221
2665992114,2cyclictest0-21swapper/107:59:191
266599211,3cyclictest0-21swapper/112:00:431
266599211,17cyclictest0-21swapper/107:28:051
266599211,15cyclictest0-21swapper/111:22:441
266599211,15cyclictest0-21swapper/111:06:591
266599211,15cyclictest0-21swapper/109:56:351
266599211,15cyclictest0-21swapper/109:10:281
266599211,15cyclictest0-21swapper/109:03:381
266599211,11cyclictest768-21in:imjournal10:10:021
266599211,11cyclictest20527-21diskstats10:15:171
266599210,19cyclictest0-21swapper/107:30:251
266599210,15cyclictest0-21swapper/110:50:281
266599210,15cyclictest0-21swapper/110:50:281
266599210,14cyclictest8858-21kworker/u8:010:23:501
265799218,3cyclictest8220-21kworker/0:109:07:480
265799218,3cyclictest27244-21kworker/0:308:02:200
265799217,3cyclictest26333-21kworker/0:010:30:580
265799216,3cyclictest16531-21kworker/0:112:36:280
265799212,10cyclictest11003-21kworker/0:209:58:140
265799211,13cyclictest0-21swapper/009:33:500
265799211,12cyclictest24074-21kworker/0:210:25:140
265799211,11cyclictest8220-21kworker/0:108:44:200
265799211,11cyclictest8220-21kworker/0:108:44:190
267599209,2cyclictest23947-21egrep09:10:123
267599207,3cyclictest0-21swapper/310:50:563
267599207,3cyclictest0-21swapper/310:50:563
267599207,2cyclictest5414-40dnf12:05:323
267599205,6cyclictest0-21swapper/312:30:133
267599205,4cyclictest0-21swapper/309:27:243
267599204,10cyclictest0-21swapper/309:45:303
267599204,10cyclictest0-21swapper/309:45:303
267599203,2cyclictest0-21swapper/312:13:593
267599201,9cyclictest10172-21needreboot09:50:213
267599201,9cyclictest0-21swapper/310:30:073
267599201,8cyclictest0-21swapper/309:44:403
267599201,10cyclictest31513-21users08:10:323
267599201,10cyclictest2864-21/usr/sbin/munin07:10:173
267599200,4cyclictest0-21swapper/311:51:353
266999208,3cyclictest31827-1kworker/u9:211:29:392
266999207,3cyclictest1673-21cat12:00:122
266999206,7cyclictest26124-21/usr/sbin/munin09:15:212
266999205,8cyclictest24116-21chrt10:22:142
266999205,7cyclictest0-21swapper/211:57:132
266999205,7cyclictest0-21swapper/211:57:132
266999205,4cyclictest14655-21cpuspeed_turbos07:35:162
266999205,4cyclictest14655-21cpuspeed_turbos07:35:162
266999202,3cyclictest86550irq/126-enp1s0-11:15:152
2669992015,3cyclictest365-21cut09:30:172
266999201,4cyclictest17140-21grep12:35:162
266999201,10cyclictest31084-21tail11:50:252
266599201,6cyclictest0-21swapper/109:43:031
266599201,18cyclictest0-21swapper/108:25:321
266599201,15cyclictest0-21swapper/111:43:551
266599201,14cyclictest0-21swapper/111:36:381
266599201,14cyclictest0-21swapper/108:57:411
266599201,14cyclictest0-21swapper/108:54:531
266599201,14cyclictest0-21swapper/108:17:451
266599201,14cyclictest0-21swapper/108:14:301
266599201,10cyclictest0-21swapper/109:19:021
266599200,15cyclictest0-21swapper/109:21:271
265799208,2cyclictest24697-21kworker/0:111:47:260
265799204,3cyclictest0-21swapper/008:31:490
265799201,7cyclictest11695-21kworker/0:111:12:420
265799201,10cyclictest12596-21/usr/sbin/munin12:25:190
265799200,10cyclictest0-21swapper/007:51:180
126442201,15sleep212744-21timerandwakeup09:55:312
267599199,8cyclictest9563-21netstat08:35:223
267599197,3cyclictest12342-21/usr/sbin/munin07:30:163
267599196,5cyclictest0-21swapper/312:20:313
267599196,3cyclictest86550irq/126-enp1s0-10:15:143
267599192,6cyclictest6345-21/usr/sbin/munin08:30:223
267599191,9cyclictest0-21swapper/312:25:423
267599191,9cyclictest0-21swapper/311:21:573
267599191,6cyclictest25515-21/usr/sbin/munin08:00:253
2675991914,3cyclictest8092-21tr07:20:123
267599191,10cyclictest23701-21diskmemload09:15:363
266999197,3cyclictest29467-21pmu-power09:20:242
266999196,3cyclictest28305-21/usr/sbin/munin10:30:302
266999196,3cyclictest22963-21sshd07:53:352
266999195,7cyclictest5837-21sshd08:26:362
266999193,4cyclictest9422-21grep07:20:252
266999192,6cyclictest23695-21sadc09:10:022
266999192,4cyclictest28440-21basename08:05:182
266999191,9cyclictest0-21swapper/211:42:202
266999191,8cyclictest6459-21ntp_states10:55:232
266999191,6cyclictest25518-21users09:10:272
266999191,5cyclictest0-21swapper/210:51:552
266999191,5cyclictest0-21swapper/210:51:552
2669991913,3cyclictest19444-21unixbench-2d10:10:282
2669991912,4cyclictest30988-21ntpq08:10:232
2669991911,2cyclictest31827-1kworker/u9:210:41:522
266599191,9cyclictest0-21swapper/110:10:131
2665991912,2cyclictest0-21swapper/107:36:221
2665991912,2cyclictest0-21swapper/107:36:221
266599191,17cyclictest0-21swapper/110:26:361
266599191,16cyclictest0-21swapper/107:10:221
266599191,14cyclictest17798-21ntp_states07:40:221
265799198,2cyclictest8220-21kworker/0:108:50:220
265799194,3cyclictest4594-21kworker/0:307:25:060
265799192,4cyclictest2260-21kworker/0:107:11:590
265799191,10cyclictest24697-21kworker/0:111:44:240
176492190,4sleep339-21rcuc/312:35:193
267599188,2cyclictest86550irq/126-enp1s0-09:04:283
267599182,5cyclictest17111-21basename08:55:003
267599182,13cyclictest13569-21gltestperf08:45:153
267599181,6cyclictest23085-21latency_hist07:55:013
267599181,6cyclictest0-21swapper/307:19:513
267599181,10cyclictest11906-21sshd07:27:343
266999186,2cyclictest15128-21/usr/sbin/munin08:50:122
266999185,3cyclictest6992-21/usr/sbin/munin09:45:232
266999185,3cyclictest6992-21/usr/sbin/munin09:45:232
266999185,3cyclictest4772-21/usr/sbin/munin09:40:142
266999185,3cyclictest24561-21users11:35:282
266999185,3cyclictest23701-21diskmemload11:07:232
266999185,3cyclictest13724-21cat08:45:182
266999184,3cyclictest12249-21sed08:40:292
266999184,3cyclictest12249-21sed08:40:292
266999183,8cyclictest6345-21/usr/sbin/munin08:30:202
266999183,4cyclictest17345-21cpu08:55:152
266999182,5cyclictest86550irq/126-enp1s0-09:51:372
266999181,7cyclictest0-21swapper/210:01:122
266999181,5cyclictest3338-21kworker/u8:112:05:262
266999181,2cyclictest0-21swapper/212:10:202
266999181,2cyclictest0-21swapper/210:45:082
2669991812,3cyclictest836-21runrttasks09:25:172
2669991812,3cyclictest7810-21cat11:00:122
266999181,10cyclictest0-21swapper/210:25:442
2665991812,3cyclictest5525-21cpuspeed_turbos07:15:041
2657991812,3cyclictest4376-21expr08:25:150
265799181,10cyclictest22932-21kworker/0:108:05:140
267599171,6cyclictest5318-21proc_pri08:25:253
267599171,4cyclictest16352-21taskset07:38:153
267599171,4cyclictest16352-21taskset07:38:153
2675991711,3cyclictest23512-21cat07:55:153
2675991710,4cyclictest28152-21fschecks_count08:05:153
266999171,9cyclictest0-21swapper/207:10:272
266999171,8cyclictest24672-21timerandwakeup07:55:282
266999171,6cyclictest17643-21memory07:40:192
266999171,5cyclictest0-21swapper/212:26:032
266999171,4cyclictest0-21swapper/207:45:232
266999171,3cyclictest0-21swapper/207:35:012
266999171,14cyclictest0-21swapper/208:18:132
2669991711,3cyclictest25767-21cat08:00:142
267599161,8cyclictest0-21swapper/307:40:183
267599161,5cyclictest22640-21memory09:05:203
267599161,5cyclictest2136-21cstates08:20:153
266999162,7cyclictest6206-21fschecks_count07:15:142
2669991611,3cyclictest10850-21sort07:25:192
2669991610,4cyclictest0-21swapper/208:20:272
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional