You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 06:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sun Nov 23, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
271192233163,56sleep20-21swapper/219:07:062
270532233163,56sleep10-21swapper/119:06:171
269982211179,21sleep30-21swapper/319:05:343
262332204166,25sleep00-21swapper/019:05:200
27443997572,2cyclictest0-21swapper/222:59:362
321122630,2sleep20-21swapper/220:30:202
292872600,4sleep011-21rcuc/020:25:160
181692540,2sleep20-21swapper/223:40:162
134202540,1sleep20-21swapper/223:30:132
311822520,1sleep10-21swapper/119:15:171
133682520,1sleep20-21swapper/219:50:012
284922490,2sleep30-21swapper/300:05:013
275762480,4sleep232-21ksoftirqd/220:20:202
27449994525,3cyclictest8898-21hddtemp_smartct22:05:153
2744999346,10cyclictest22222-21latency_hist20:10:023
2744999334,17cyclictest3699-21grep21:55:013
2744999326,14cyclictest15677-21grep22:20:203
2744999326,14cyclictest15677-21grep22:20:193
2743899321,6cyclictest0-21swapper/123:27:011
112972320,4sleep011-21rcuc/019:45:010
27449993117,9cyclictest4752-40dnf19:30:123
195902310,3sleep339-21rcuc/321:15:223
195902310,3sleep339-21rcuc/321:15:223
2744999306,10cyclictest393-21ntp_states21:45:243
2743899302,1cyclictest88250irq/135-enp2s0-00:18:021
2744999299,10cyclictest3841-21memory00:20:223
2743299291,13cyclictest5798-21munin-run22:00:010
2744999285,14cyclictest17662-21/usr/sbin/munin23:40:263
2744999282,14cyclictest32724-21/usr/sbin/munin19:20:193
2744999282,14cyclictest19859-21/usr/sbin/munin23:45:213
27449992816,7cyclictest20908-40dnf22:31:333
27449992814,4cyclictest780-21rngd21:06:123
2744999276,17cyclictest2711-21grep19:25:153
2744999272,13cyclictest0-21swapper/320:53:083
2744999271,11cyclictest0-21swapper/319:48:323
2743899271,24cyclictest0-21swapper/119:36:401
2743899271,11cyclictest0-21swapper/123:11:171
2744999266,14cyclictest0-21swapper/300:19:433
2744999266,11cyclictest5118-21/usr/sbin/munin00:25:273
2744999266,11cyclictest5118-21/usr/sbin/munin00:25:273
2744999265,11cyclictest780-21rngd20:47:463
2744999265,10cyclictest4940-21grep23:10:153
2744999264,10cyclictest9583-21cpuspeed_turbos00:35:133
2744999263,10cyclictest25318-21cut21:30:123
2744999261,11cyclictest0-21swapper/322:58:343
2743299267,16cyclictest8642-21munin-run23:20:000
2744999255,5cyclictest86850irq/129-enp1s0-22:05:003
2744999254,13cyclictest8952-21chrt00:30:513
2744999253,10cyclictest780-21rngd19:54:483
2744999252,9cyclictest0-21swapper/322:25:383
2744999252,11cyclictest780-21rngd19:13:033
2744999252,10cyclictest0-21swapper/320:11:363
2744999252,10cyclictest0-21swapper/320:11:363
2744999251,14cyclictest0-21swapper/320:23:123
27438992511,3cyclictest16728-21df21:10:131
2744999247,3cyclictest0-21swapper/323:35:143
2744999246,9cyclictest4601-21/usr/sbin/munin21:55:183
2744999246,12cyclictest0-21swapper/321:40:003
2744999245,17cyclictest86750irq/128-enp1s0-20:28:323
2744999244,17cyclictest8104-21awk19:35:283
2744999243,11cyclictest9782-21/usr/sbin/munin20:55:273
2744999241,9cyclictest0-21swapper/322:47:343
2744999241,10cyclictest30105-21fschecks_time21:40:173
2744999241,10cyclictest18759-21irqstats20:00:203
2743899242,12cyclictest26812-40dnf21:30:441
27438992413,4cyclictest0-21swapper/123:35:081
27438992411,3cyclictest27276-21irqstats00:00:201
2743899241,10cyclictest0-21swapper/123:01:191
2743299241,11cyclictest16163-21munin-run21:10:000
325102230,4sleep02743299cyclictest19:20:010
2744999236,9cyclictest22974-21missed_timers23:50:193
2744999236,7cyclictest0-21swapper/323:25:013
2744999236,7cyclictest0-21swapper/323:25:013
2744999233,9cyclictest780-21rngd20:34:083
2744999232,10cyclictest0-21swapper/323:55:273
27449992314,7cyclictest25496-21kworker/u8:223:00:143
2744999230,9cyclictest0-21swapper/319:44:403
2743899234,7cyclictest0-21swapper/120:17:531
2743899233,3cyclictest27420-21cyclictest00:23:551
2743899232,4cyclictest9561-21munin-run20:55:011
2743899231,9cyclictest7668-21diskstats00:30:141
2743899231,6cyclictest0-21swapper/121:54:481
2743899231,19cyclictest27113-21switchtime22:45:241
2743899230,17cyclictest0-21swapper/122:50:211
2744999227,5cyclictest780-21rngd19:57:443
2744999226,14cyclictest86550irq/126-enp1s0-21:25:163
2744999225,5cyclictest848-21sssd_nss23:31:443
2744999223,16cyclictest780-21rngd20:16:103
2744999221,8cyclictest0-21swapper/320:37:443
2744999221,8cyclictest0-21swapper/300:09:323
2744999221,6cyclictest0-21swapper/300:11:303
27449992213,6cyclictest12348-21grep21:00:163
27449992213,5cyclictest7730-21ntp_states23:15:223
2744999221,11cyclictest780-21rngd19:15:233
2744999221,10cyclictest780-21rngd23:26:123
2744999221,10cyclictest780-21rngd23:07:273
2744399225,7cyclictest0-21swapper/200:15:142
2744399221,3cyclictest0-21swapper/223:47:082
2743899223,6cyclictest0-21swapper/123:40:221
27438992211,9cyclictest2000-21kworker/u8:120:45:131
2743899221,19cyclictest0-21swapper/119:55:201
2743899221,16cyclictest0-21swapper/123:54:301
2744999216,3cyclictest11765-21grep22:10:253
2744999213,15cyclictest21424-21/usr/sbin/munin22:35:233
2744999211,9cyclictest0-21swapper/321:22:003
27449992114,4cyclictest18367-21munin-run21:15:013
2743899216,13cyclictest7282-21kworker/u8:021:01:061
2743899211,9cyclictest20691-21grep22:30:251
2743899211,2cyclictest0-21swapper/100:35:221
2743899211,19cyclictest0-21swapper/120:57:451
2743899211,18cyclictest0-21swapper/122:21:501
2743899211,18cyclictest0-21swapper/122:21:501
2743899211,18cyclictest0-21swapper/120:40:011
2743899211,15cyclictest0-21swapper/123:58:481
2743899211,15cyclictest0-21swapper/123:06:011
2743899211,15cyclictest0-21swapper/100:08:531
2743899211,11cyclictest22694-21df_inode20:10:141
2743899211,11cyclictest22694-21df_inode20:10:141
27438992110,9cyclictest0-21swapper/121:46:021
27438992110,2cyclictest581-21systemd-journal20:30:301
2743899210,15cyclictest0-21swapper/121:21:031
2743299216,5cyclictest0-21swapper/022:12:440
2743299211,8cyclictest27290-21kernelversion00:00:210
27432992112,2cyclictest0-21swapper/023:47:070
2744999207,2cyclictest0-21swapper/320:41:523
27449992013,4cyclictest13729-21/usr/sbin/munin22:15:223
27449992012,5cyclictest28376-21diskstats22:50:173
27449992010,7cyclictest24989-21/usr/sbin/munin22:40:263
2744399201,6cyclictest0-21swapper/200:35:182
27443992016,3cyclictest0-21swapper/220:10:572
27443992016,3cyclictest0-21swapper/220:10:572
2743899207,11cyclictest307-21cpu19:20:121
2743899205,7cyclictest13076-21systemd-journal19:47:031
2743899201,8cyclictest0-21swapper/123:16:551
27438992016,3cyclictest0-21swapper/119:50:181
2743899201,3cyclictest0-21swapper/119:27:271
2743899201,18cyclictest0-21swapper/121:27:001
2743899201,18cyclictest0-21swapper/121:09:561
2743899201,18cyclictest0-21swapper/120:29:581
2743899201,18cyclictest0-21swapper/120:22:271
2743899201,18cyclictest0-21swapper/100:27:171
2743899201,18cyclictest0-21swapper/100:27:171
2743899201,11cyclictest18243-21expr20:00:141
2743899201,11cyclictest0-21swapper/121:55:221
2743899201,10cyclictest30737-21pmu-power21:40:231
2743899201,10cyclictest29387-21wc21:40:011
2743899201,10cyclictest19765-21ps21:15:241
2743899201,10cyclictest19765-21ps21:15:241
2743899200,15cyclictest0-21swapper/123:34:101
2743899200,15cyclictest0-21swapper/122:58:121
2743899200,15cyclictest0-21swapper/122:43:241
2743299209,3cyclictest637-21/usr/sbin/munin00:15:250
2743299207,3cyclictest0-21swapper/023:52:140
2743299206,3cyclictest0-21swapper/000:29:340
2743299206,3cyclictest0-21swapper/000:29:340
2743299205,3cyclictest2920-21kworker/0:200:39:210
2743299204,5cyclictest16159-21diskmemload23:20:270
2743299204,5cyclictest16159-21diskmemload23:20:270
2743299202,7cyclictest14634-21snmpd21:30:170
2743299201,10cyclictest0-21swapper/022:54:200
188582201,2sleep10-21swapper/122:28:261
159262201,3sleep20-21swapper/221:07:142
112342200,1sleep20-21swapper/223:25:162
5332190,3sleep011-21rcuc/021:45:240
276412191,3sleep00-21swapper/022:48:080
2744399197,3cyclictest86550irq/126-enp1s0-22:31:342
2744399195,6cyclictest0-21swapper/200:09:352
2744399193,4cyclictest0-21swapper/223:10:172
2744399191,9cyclictest0-21swapper/222:50:282
2744399191,6cyclictest3309-21cut00:20:182
2743899197,2cyclictest86550irq/126-enp1s0-19:30:151
2743899191,9cyclictest20154-21/usr/sbin/munin20:05:211
27438991915,3cyclictest0-21swapper/123:45:431
27438991915,3cyclictest0-21swapper/122:11:461
27438991915,3cyclictest0-21swapper/119:12:131
27438991915,2cyclictest0-21swapper/120:40:091
27438991913,4cyclictest9824-21cut19:40:171
27438991913,3cyclictest31536-21sed00:10:171
2743899191,17cyclictest0-21swapper/122:01:531
2743899191,10cyclictest8339-21cpuspeed_turbos22:05:111
2743899190,14cyclictest0-21swapper/123:20:501
2743899190,14cyclictest0-21swapper/123:20:501
2743299198,3cyclictest30406-21cstates22:55:130
2743299198,3cyclictest2051-1kworker/u9:023:05:140
2743299197,3cyclictest0-21swapper/023:35:400
2743299191,8cyclictest5272-21interrupts23:10:180
2743299191,8cyclictest0-21swapper/023:25:080
27432991913,3cyclictest2960-21munin-run20:40:020
27432991912,4cyclictest4740-21unixbench_singl20:40:260
2743299191,11cyclictest0-21swapper/022:25:590
2743299191,10cyclictest26752-21/usr/sbin/munin20:20:250
2743299191,10cyclictest0-21swapper/023:30:150
2743299190,10cyclictest20835-21kworker/u8:022:32:010
198282191,2sleep00-21swapper/020:03:530
291012181,3sleep20-21swapper/221:36:492
2744399186,3cyclictest12050irq/123-ahci[0023:35:142
2744399184,3cyclictest86550irq/126-enp1s0-22:48:082
2744399183,4cyclictest9-21rcu_preempt21:40:192
2744399183,3cyclictest5157-21crond20:45:012
2744399182,6cyclictest23121-21ntp_states23:50:212
2744399181,7cyclictest7928-21smartctl23:15:262
2744399181,7cyclictest14634-21snmpd21:20:152
2744399181,6cyclictest32139-21diskstats21:45:172
2744399181,6cyclictest22006-21fw_packets22:35:182
27443991813,3cyclictest8377-21ntp_offset00:30:222
27443991813,3cyclictest4147-21users19:25:282
27443991813,3cyclictest14591-21turbostat.cron22:20:012
27443991813,3cyclictest11100-21hddtemp_smartct22:10:172
27443991812,3cyclictest30198-21hwlatdetect19:15:042
27443991812,3cyclictest1695-21idleruntime20:35:182
27443991812,3cyclictest15514-21latency_hist19:55:022
27443991811,4cyclictest31306-21fschecks_count00:10:162
27443991811,4cyclictest23979-21head21:25:212
2744399181,10cyclictest0-21swapper/223:06:572
2743899181,9cyclictest0-21swapper/122:15:261
27438991815,2cyclictest0-21swapper/122:36:331
2743299187,3cyclictest14634-21snmpd19:59:590
2743299187,3cyclictest0-21swapper/000:07:490
2743299186,3cyclictest0-21swapper/021:44:260
2743299184,5cyclictest23152-21kworker/u8:122:37:550
2743299184,3cyclictest6624-21grep22:00:160
2743299181,9cyclictest0-21swapper/022:21:200
2743299181,9cyclictest0-21swapper/022:21:200
2743299181,9cyclictest0-21swapper/000:33:230
2743299181,6cyclictest32312-21sed00:10:260
27432991813,3cyclictest27927-21grep21:35:180
27432991812,5cyclictest27185-21kworker/u8:221:00:030
27432991812,4cyclictest1278-21/usr/sbin/munin23:00:230
27432991812,3cyclictest9258-21ntpq22:05:190
2743299181,15cyclictest0-21swapper/000:20:050
27432991811,4cyclictest29795-21latency_hist19:15:020
2743299181,10cyclictest0-21swapper/021:26:430
2743299181,10cyclictest0-21swapper/021:20:330
2743299180,3cyclictest20575-21awk21:20:010
2743299180,3cyclictest20575-21awk21:20:010
202252181,3sleep20-21swapper/221:15:532
202252181,3sleep20-21swapper/221:15:532
165622180,2sleep00-21swapper/021:10:120
43102171,2sleep00-21swapper/019:27:200
311422170,2sleep00-21swapper/020:30:070
2744399175,3cyclictest86850irq/129-enp1s0-20:50:272
2744399171,9cyclictest0-21swapper/222:25:052
2744399171,9cyclictest0-21swapper/222:08:552
2744399171,7cyclictest0-21swapper/200:25:142
2744399171,7cyclictest0-21swapper/200:25:142
2744399171,6cyclictest9759-21grep19:40:172
2744399171,6cyclictest0-21swapper/223:00:262
2744399171,4cyclictest27083-21gltestperf00:00:192
27443991712,3cyclictest26363-21processes21:30:212
27443991712,3cyclictest1841-21expr21:50:142
27443991711,3cyclictest9947-21open_files23:20:242
27443991711,3cyclictest9947-21open_files23:20:242
27443991711,3cyclictest5386-21/usr/sbin/munin20:45:122
27443991711,3cyclictest29249-21expr20:25:162
27443991711,3cyclictest25139-21wc23:55:222
27443991711,3cyclictest24390-21if_err_enp1s022:40:182
27443991711,3cyclictest15039-21expr22:20:142
27443991711,3cyclictest15039-21expr22:20:142
27443991711,3cyclictest13707-21cut19:50:142
2744399171,10cyclictest0-21swapper/221:57:472
2744399171,10cyclictest0-21swapper/221:03:412
2743299176,3cyclictest86750irq/128-enp1s0-20:45:240
2743299176,3cyclictest17110-21kworker/u8:123:40:140
2743299176,3cyclictest14634-21snmpd20:55:100
2743299175,4cyclictest24333-21awk20:15:000
2743299175,4cyclictest24333-21awk20:15:000
2743299171,9cyclictest0-21swapper/019:35:120
2743299171,5cyclictest14212-21users22:15:270
27432991712,3cyclictest9113-21awk20:50:260
27432991712,3cyclictest23622-21/usr/sbin/munin22:40:250
27432991712,3cyclictest12717-21timerandwakeup19:45:250
2743299171,10cyclictest0-21swapper/021:52:190
21542171,2sleep00-21swapper/019:22:510
153672171,2sleep00-21swapper/019:53:130
2744399163,4cyclictest14634-21snmpd19:39:562
2744399161,5cyclictest0-21swapper/222:00:132
27443991612,3cyclictest0-21swapper/220:06:002
27443991612,3cyclictest0-21swapper/219:14:492
27443991611,3cyclictest9782-21/usr/sbin/munin20:55:172
27443991611,3cyclictest5053-21expr19:30:152
27443991611,3cyclictest26011-21sed20:15:272
27443991611,3cyclictest19210-21smartctl20:00:262
27443991611,3cyclictest1047-21idleruntime19:20:182
27443991610,3cyclictest16534-21cut21:10:122
2743299166,7cyclictest0-21swapper/020:05:260
27432991612,3cyclictest0-21swapper/019:30:120
27432991611,3cyclictest25660-21pmu-power20:15:250
27432991610,3cyclictest24388-21cut23:55:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional