You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 01:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Nov 16, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
56524040,5sleep30-21swapper/307:09:273
3532233161,59sleep00-21swapper/007:06:520
3962214162,39sleep10-21swapper/107:07:261
4862204166,25sleep20-21swapper/207:08:312
2306921760,4sleep070499cyclictest10:25:140
119522800,4sleep339-21rcuc/307:30:273
119522800,4sleep339-21rcuc/307:30:273
257022730,1sleep10-21swapper/108:05:011
718996864,3cyclictest0-21swapper/208:09:362
718995855,2cyclictest0-21swapper/211:30:342
104182550,3sleep30-21swapper/311:10:133
718994535,9cyclictest0-21swapper/208:31:482
718994535,9cyclictest0-21swapper/208:31:482
60122430,2sleep30-21swapper/309:45:203
324332410,1sleep032432-21sed09:30:270
70499397,29cyclictest14354-21munin-run11:20:010
70499371,11cyclictest3994-21munin-run12:10:010
704993710,4cyclictest26795-21/usr/sbin/munin08:05:220
14072360,3sleep070499cyclictest09:35:170
70499359,3cyclictest27142-21/usr/sbin/munin10:35:230
70499358,9cyclictest24266-21gltestperf08:00:190
704993510,3cyclictest30273-21/usr/sbin/munin08:15:250
70499348,4cyclictest14164-21awk12:30:250
70499343,8cyclictest15505-21sort11:20:190
70499341,6cyclictest768-21in:imjournal10:05:010
704993414,3cyclictest17064-21grep08:55:270
704993411,3cyclictest29743-21ntp_kernel_err09:25:240
704993311,3cyclictest26156-21gltestperf11:45:160
704993311,3cyclictest22535-21ntp_states07:55:240
704993311,3cyclictest19913-21/usr/sbin/munin09:05:260
704993310,3cyclictest8619-21/usr/sbin/munin12:20:170
70499329,3cyclictest5779-21/usr/sbin/munin11:00:200
70499328,4cyclictest28137-21/usr/sbin/munin11:50:160
70499322,10cyclictest14634-21snmpd10:12:340
704993212,3cyclictest9184-21seq09:53:310
70499321,10cyclictest15984-21latency_hist10:10:010
72399311,7cyclictest21880-21sadc09:10:013
70499314,12cyclictest31319-21awk10:45:010
70499314,11cyclictest0-21swapper/012:15:010
70499313,2cyclictest86550irq/126-enp1s0-09:10:220
70499312,13cyclictest1627-21unixbench_multi08:20:270
70499311,6cyclictest0-21swapper/009:15:380
70499311,5cyclictest17072-21/usr/sbin/munin07:45:160
704993111,3cyclictest3182-21sed07:15:010
704993110,3cyclictest836-21runrttasks09:01:390
704993110,3cyclictest2809-21cat10:50:270
704993110,3cyclictest24255-21sort11:40:200
704993110,3cyclictest24255-21sort11:40:200
70499309,3cyclictest27580-21ntp_states09:20:240
70499309,3cyclictest22240-21seq10:21:410
70499309,3cyclictest14484-21processes08:50:240
70499302,4cyclictest2960-21latency_hist07:15:010
70499301,6cyclictest11438-21gltestperf12:25:210
70499301,11cyclictest21878-21diskmemload12:36:200
70499301,11cyclictest0-21swapper/010:48:300
704993010,3cyclictest8867-21gltestperf07:25:170
704993010,3cyclictest24946-21/usr/sbin/munin10:30:290
704993010,3cyclictest16248-21timerandwakeup07:40:280
70499296,3cyclictest86750irq/128-enp1s0-12:15:200
70499296,2cyclictest0-21swapper/010:15:210
70499294,4cyclictest0-21swapper/010:55:510
70499291,9cyclictest20150-21/usr/sbin/munin11:30:240
70499291,8cyclictest21171-21/usr/sbin/munin11:35:330
70499291,5cyclictest10484-21/usr/sbin/munin07:30:170
70499291,5cyclictest10484-21/usr/sbin/munin07:30:160
704992910,3cyclictest6491-21munin-run08:35:010
704992910,3cyclictest6491-21munin-run08:35:010
704992910,2cyclictest5193-21kworker/0:211:25:150
718992824,3cyclictest0-21swapper/208:23:412
70499289,3cyclictest5858-21sort09:45:180
70499289,3cyclictest3145-21kernelversion08:25:200
70499288,3cyclictest21324-21cut07:55:010
70499281,8cyclictest0-21swapper/011:07:000
70499281,10cyclictest0-21swapper/009:58:340
704992810,3cyclictest9230-21expr08:40:160
704992810,2cyclictest28788-21seq08:10:180
20102281,3sleep22107-21/usr/sbin/munin10:50:182
70499275,3cyclictest86750irq/128-enp1s0-07:35:210
70499272,4cyclictest30909-21/usr/sbin/munin11:55:200
70499271,4cyclictest0-21swapper/012:00:540
70499271,11cyclictest0-21swapper/011:13:460
288742270,4sleep339-21rcuc/309:25:173
70499261,11cyclictest0-21swapper/009:43:170
136102260,4sleep121-21rcuc/108:50:141
72399251,5cyclictest842-21lldpd09:40:143
712992519,3cyclictest10455-21memory09:55:201
70499250,9cyclictest0-21swapper/008:35:170
38542250,15sleep121878-21diskmemload09:40:191
72399241,4cyclictest14634-21snmpd07:23:173
71899241,6cyclictest32-21ksoftirqd/211:23:462
71299242,3cyclictest12446-40dnf12:25:501
71299241,20cyclictest30985-21cat09:30:121
71899231,9cyclictest17251-21open_inodes10:10:232
71299232,3cyclictest17453-21hddtemp_smartct11:25:191
71299231,3cyclictest19358-21cpuspeed_turbos07:50:131
71299231,3cyclictest18052sleep108:21:311
71299231,19cyclictest4257-21hddtemp_smartct10:55:171
71299231,14cyclictest1682-21sshd12:03:471
72399221,6cyclictest0-21swapper/312:15:353
723992216,3cyclictest27114-21grep09:20:173
72399221,14cyclictest0-21swapper/311:55:253
72399221,12cyclictest842-21lldpd08:37:123
72399221,11cyclictest15647-21ntp_states11:20:223
71899229,7cyclictest21878-21diskmemload09:24:512
71899221,6cyclictest0-21swapper/211:10:202
71899221,6cyclictest0-21swapper/210:38:572
71899221,5cyclictest0-21swapper/209:55:282
71899221,3cyclictest0-21swapper/208:55:272
71299221,3cyclictest27842-21latency_hist08:10:021
71299221,19cyclictest0-21swapper/107:35:291
71299221,18cyclictest3266-21mailstats12:05:261
252682220,4sleep271899cyclictest09:15:202
72399211,9cyclictest0-21swapper/311:42:073
72399211,9cyclictest0-21swapper/311:42:063
712992118,2cyclictest0-21swapper/107:55:361
712992117,3cyclictest0-21swapper/111:18:271
712992117,3cyclictest0-21swapper/109:25:421
71299211,18cyclictest0-21swapper/111:35:181
71299211,18cyclictest0-21swapper/110:49:031
71299211,18cyclictest0-21swapper/107:15:041
71299211,15cyclictest86850irq/129-enp1s0-12:15:511
71299211,15cyclictest0-21swapper/110:14:281
71299211,11cyclictest23813-21ntp_states10:25:241
71299211,11cyclictest19913-21/usr/sbin/munin09:05:281
71299211,11cyclictest14634-21snmpd10:33:511
71299211,11cyclictest14634-21snmpd08:29:491
71299211,11cyclictest10521-21users08:40:441
712992110,3cyclictest23720-21diskstats11:40:161
712992110,3cyclictest23720-21diskstats11:40:161
712992110,3cyclictest13075-21latency_hist08:50:011
67812211,2sleep10-21swapper/109:45:501
72399209,3cyclictest4224-21/usr/sbin/munin12:10:273
72399209,3cyclictest32165-21/usr/sbin/munin12:00:243
72399208,4cyclictest842-21lldpd08:28:123
72399202,15cyclictest30694-21sort08:15:163
72399202,10cyclictest18346-21/usr/sbin/munin10:15:163
71899204,4cyclictest0-21swapper/211:55:382
71899202,15cyclictest8850-21grep09:50:272
71899201,7cyclictest31310-21munin-run10:45:012
71899201,6cyclictest0-21swapper/207:51:002
71899201,5cyclictest0-21swapper/211:29:392
71299209,3cyclictest7977-21smart_sda08:35:261
71299202,8cyclictest14880-21/usr/sbin/munin07:40:171
71299202,14cyclictest21210-21gltestperf10:20:171
712992014,3cyclictest20026-21ntp_kernel_pll_11:30:221
712992013,4cyclictest17986-21memory07:45:201
71299201,18cyclictest0-21swapper/110:02:491
71299201,18cyclictest0-21swapper/108:13:361
71299201,17cyclictest0-21swapper/112:31:331
71299201,17cyclictest0-21swapper/111:46:341
71299201,17cyclictest0-21swapper/111:00:481
71299201,17cyclictest0-21swapper/110:39:491
71299201,17cyclictest0-21swapper/109:36:401
71299201,14cyclictest0-21swapper/110:16:361
71299201,11cyclictest28834-21ntp_states11:50:221
71299201,11cyclictest0-21swapper/110:44:391
71299201,10cyclictest5736-21grep12:10:281
71299201,10cyclictest16974-21unixbench-2d08:55:261
71299201,10cyclictest0-21swapper/109:51:201
712992010,3cyclictest2647-21smart_sda10:50:261
71299200,18cyclictest0-21swapper/107:30:191
71299200,18cyclictest0-21swapper/107:30:191
72399198,3cyclictest22743-21/usr/sbin/munin10:25:183
72399193,7cyclictest27771-21/usr/sbin/munin11:50:173
72399192,7cyclictest23299-1kworker/u9:212:30:243
72399192,3cyclictest27046-21ps08:05:273
72399191,9cyclictest0-21swapper/310:45:213
72399191,7cyclictest16765-21gltestperf10:10:183
72399191,6cyclictest145872grep10:05:183
72399191,6cyclictest0-21swapper/310:53:493
723991914,3cyclictest23053-21users07:55:283
723991913,3cyclictest3199-21munin-node07:35:153
723991913,3cyclictest1927-21processes09:35:243
723991913,3cyclictest10156-21grep09:55:183
72399191,11cyclictest20897-21sshd07:50:483
72399191,11cyclictest0-21swapper/311:06:113
72399191,10cyclictest8916-21/usr/sbin/munin08:40:433
72399191,10cyclictest3880-21sshd12:08:483
72399191,10cyclictest14634-21snmpd11:29:013
71899192,5cyclictest8007-21users12:15:282
71899191,6cyclictest0-21swapper/209:40:212
71899191,15cyclictest7076-21grep07:20:202
71299197,4cyclictest15986-21if_err_enp1s012:35:201
71299194,9cyclictest0-21swapper/111:23:371
71299191,9cyclictest0-21swapper/112:20:131
71299191,7cyclictest0-21swapper/109:22:421
712991916,2cyclictest0-21swapper/111:08:421
712991915,3cyclictest0-21swapper/109:16:281
712991915,3cyclictest0-21swapper/109:11:111
71299191,4cyclictest18190-21grep09:00:161
71299191,17cyclictest0-21swapper/111:13:221
71299191,17cyclictest0-21swapper/107:24:481
71299191,17cyclictest0-21swapper/107:12:301
71299191,16cyclictest21878-21diskmemload10:07:131
71299191,10cyclictest29966-21/usr/sbin/munin11:55:131
70499199,3cyclictest6085-21/usr/sbin/munin07:20:260
27102191,2sleep22716-21latency_hist09:40:012
72399187,3cyclictest27142-21/usr/sbin/munin10:35:273
72399187,3cyclictest14634-21snmpd11:49:053
72399185,4cyclictest4544-21cat10:55:213
72399183,12cyclictest971-21/usr/sbin/munin07:10:163
72399181,9cyclictest0-21swapper/311:30:073
72399181,9cyclictest0-21swapper/308:58:483
72399181,7cyclictest8284-21/usr/sbin/munin07:25:223
72399181,6cyclictest24785-21fschecks_time09:15:153
72399181,6cyclictest0-21swapper/312:36:413
723991812,3cyclictest5795-21awk08:30:283
723991812,3cyclictest5795-21awk08:30:283
723991812,3cyclictest28081-21/usr/sbin/munin08:10:163
723991812,3cyclictest23140-21ntp_kernel_err09:10:223
723991812,3cyclictest13960-21unixbench_singl11:15:283
72399181,15cyclictest0-21swapper/307:40:113
723991811,4cyclictest24339-21/usr/sbin/munin08:00:193
72399181,10cyclictest0-21swapper/310:00:173
72399180,9cyclictest0-21swapper/309:00:133
71899185,2cyclictest0-21swapper/207:15:232
71899184,5cyclictest0-21swapper/209:25:232
71899182,6cyclictest0-21swapper/209:30:132
71899182,3cyclictest0-21swapper/207:30:272
71899182,3cyclictest0-21swapper/207:30:272
71899182,10cyclictest0-21swapper/210:59:312
71899181,7cyclictest2700-21sh12:05:182
71899181,7cyclictest0-21swapper/212:14:372
71899181,7cyclictest0-21swapper/211:05:542
71899181,6cyclictest0-21swapper/210:30:052
71899181,5cyclictest0-21swapper/211:49:552
718991813,3cyclictest28829-21ntp_offset11:50:222
718991811,4cyclictest12051-21fgrep12:25:282
71899181,12cyclictest0-21swapper/212:03:402
71899181,11cyclictest0-21swapper/210:49:382
71299184,11cyclictest30882-21tail08:15:161
71299181,16cyclictest0-21swapper/108:30:031
71299181,16cyclictest0-21swapper/108:30:031
71299181,16cyclictest0-21swapper/107:27:461
267072181,3sleep30-21swapper/310:32:343
223282181,2sleep30-21swapper/310:22:513
187642181,2sleep20-21swapper/209:00:232
72399175,11cyclictest86550irq/126-enp1s0-08:45:143
72399172,6cyclictest7348-21/usr/sbin/munin09:50:273
72399171,6cyclictest31215-21expr09:30:143
72399171,6cyclictest0-21swapper/312:25:183
72399171,5cyclictest6854-21ntpq11:00:233
72399171,3cyclictest13308-21/usr/sbin/munin08:50:173
72399171,2cyclictest29509-21/usr/sbin/munin10:40:153
723991712,3cyclictest9723-21pmu-power12:20:243
72399171,13cyclictest32473-21/usr/sbin/munin08:20:193
72399171,13cyclictest21171-21/usr/sbin/munin11:35:333
72399171,13cyclictest17072-21/usr/sbin/munin07:45:143
71899176,3cyclictest86550irq/126-enp1s0-08:45:252
71899172,6cyclictest5151-21/usr/sbin/munin09:45:202
71899172,14cyclictest0-21swapper/211:04:162
71899171,7cyclictest30273-21/usr/sbin/munin08:15:202
71899171,7cyclictest23923-21tune2fs11:40:172
71899171,7cyclictest23923-21tune2fs11:40:172
71899171,6cyclictest0-21swapper/212:35:242
71899171,6cyclictest0-21swapper/211:15:222
71899171,6cyclictest0-21swapper/210:05:042
71899171,6cyclictest0-21swapper/208:51:122
718991713,3cyclictest0-21swapper/210:00:132
718991712,3cyclictest2518-21grep07:10:262
718991712,3cyclictest22508-40dnf10:25:012
718991712,3cyclictest22445-21smart_sda11:35:322
718991712,3cyclictest14880-21/usr/sbin/munin07:40:232
71899171,15cyclictest0-21swapper/208:26:282
71899171,13cyclictest9053-21grep08:40:152
718991711,3cyclictest23176-21sh09:10:232
718991711,3cyclictest18131-21ntp_offset07:45:222
71899171,10cyclictest0-21swapper/210:15:222
71899171,10cyclictest0-21swapper/209:05:052
70499171,6cyclictest0-21swapper/008:45:280
72399161,5cyclictest5475-21sed07:15:423
71899163,3cyclictest0-21swapper/207:27:232
71899161,6cyclictest23959-21expr08:00:162
718991612,2cyclictest311ktimersoftd/208:10:572
71899161,14cyclictest0-21swapper/212:31:492
718991611,3cyclictest10004-21sed12:20:262
718991610,3cyclictest22887-21grep10:25:142
71899160,15cyclictest0-21swapper/208:35:112
71899160,15cyclictest0-21swapper/207:55:332
718991510,3cyclictest12682-21/usr/sbin/munin07:35:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional