You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 20:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Sun Nov 30, 2025 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1569499602,47cyclictest6895-21ssh09:55:150
1570499595,6cyclictest131rcu_preempt12:35:141
15694995415,30cyclictest32592-21ssh09:30:140
15694995415,30cyclictest32592-21ssh09:30:130
1570499515,6cyclictest131rcu_preempt09:05:121
1570499503,8cyclictest131rcu_preempt09:55:151
1572099496,26cyclictest29566irq/58-eth011:20:153
1571299471,38cyclictest0-21swapper/212:15:142
1571299471,38cyclictest0-21swapper/212:15:132
15720994610,24cyclictest29566irq/58-eth011:00:133
1570499462,33cyclictest21188-21ssh10:25:151
15694994613,20cyclictest0-21swapper/007:35:160
15694994613,20cyclictest0-21swapper/007:35:160
1571299453,17cyclictest28555-21ssh11:35:152
1570499453,7cyclictest131rcu_preempt07:50:141
15704994528,9cyclictest0-21swapper/111:50:131
1570499452,35cyclictest31743-21sshd08:00:131
15712994420,6cyclictest0-21swapper/207:35:162
15712994420,6cyclictest0-21swapper/207:35:152
1571299441,26cyclictest28705-21ssh10:30:152
1570499412,27cyclictest28954-21ssh12:15:141
1570499412,27cyclictest28954-21ssh12:15:131
1569499414,22cyclictest18843-21ssh10:45:140
15694994131,5cyclictest28278-21apt-get08:30:130
1570499401,28cyclictest26233-21ssh10:50:151
15694994016,6cyclictest22716-21apt-get10:05:150
1572099392,25cyclictest29566irq/58-eth008:30:133
15712993927,5cyclictest0-21swapper/210:00:162
1570499392,26cyclictest5678-21ssh10:15:131
1570499383,26cyclictest23654-21ssh09:45:141
1572099361,21cyclictest29566irq/58-eth011:05:133
15694993622,10cyclictest23451-21apt-get08:25:130
15704993518,9cyclictest21-21ksoftirqd/111:25:131
15694993525,4cyclictest15793-21apt-get09:40:110
1572099343,16cyclictest29566irq/58-eth007:20:173
15694993426,4cyclictest1334-21apt-get10:55:150
15694993421,9cyclictest26222-21apt-get10:50:150
15694993418,12cyclictest4213-21ssh12:00:150
1571299331,27cyclictest31346-21ssh09:50:122
1569499338,16cyclictest7915-21ssh09:12:100
15694993327,2cyclictest9644-21apt-get09:15:150
1572099323,18cyclictest0-21swapper/312:15:143
1572099323,18cyclictest0-21swapper/312:15:143
1570499323,19cyclictest24782-21sshd09:00:141
1570499323,19cyclictest24782-21sshd09:00:141
1569499328,11cyclictest30327-21ntp_192.168.11407:55:250
15694993225,3cyclictest22088-21apt-get07:50:130
1572099312,18cyclictest29566irq/58-eth009:35:123
1569499313,9cyclictest11387-21cat10:40:010
15694993123,4cyclictest30456-21apt-get07:25:140
15720993010,9cyclictest29566irq/58-eth012:10:153
1571299304,2cyclictest131rcu_preempt09:15:172
1571299303,18cyclictest26346irq/40-dwc2_hso08:55:142
1571299302,22cyclictest20673-21ssh12:10:162
1571299302,21cyclictest0-21swapper/208:00:152
1571299302,17cyclictest26346irq/40-dwc2_hso09:30:132
1571299302,17cyclictest26346irq/40-dwc2_hso09:30:122
1571299301,17cyclictest5683-21ssh10:15:142
1569499307,14cyclictest0-21swapper/011:35:150
1569499302,23cyclictest30667-21ssh11:15:150
1569499302,23cyclictest30667-21ssh11:15:140
1569499302,21cyclictest5674-21rm11:20:130
15694993019,9cyclictest28941-21apt-get12:15:140
15694993019,9cyclictest28941-21apt-get12:15:130
15694993014,12cyclictest0-21swapper/007:30:130
1571299293,10cyclictest0-21swapper/209:20:132
1571299292,19cyclictest26346irq/40-dwc2_hso08:05:142
1570499293,18cyclictest22729-21ssh10:05:161
1570499292,19cyclictest24892-21sshd09:25:141
1569499297,7cyclictest1770-21diskmemload10:22:570
1569499294,11cyclictest13153-21/usr/sbin/munin12:25:150
1571299282,19cyclictest0-21swapper/209:45:142
1571299282,18cyclictest0-21swapper/211:15:162
1571299282,18cyclictest0-21swapper/211:15:162
15712992820,4cyclictest26346irq/40-dwc2_hso11:00:122
1571299281,2cyclictest0-21swapper/208:15:162
1571299281,16cyclictest26346irq/40-dwc2_hso10:50:142
15712992810,13cyclictest0-21swapper/210:45:162
1570499282,19cyclictest1341-21ssh10:55:131
1570499281,19cyclictest4840-21ssh12:20:141
1569499286,8cyclictest14729-21ssh09:38:520
1569499285,11cyclictest17691-21grep08:15:300
1569499283,11cyclictest17863-21ssh12:06:190
15694992811,12cyclictest0-21swapper/008:05:130
1572099278,11cyclictest29566irq/58-eth011:50:143
15720992715,7cyclictest29566irq/58-eth010:00:163
1571299278,12cyclictest27-21ksoftirqd/211:28:122
1571299274,15cyclictest27-21ksoftirqd/209:13:502
1571299272,19cyclictest0-21swapper/207:25:142
1571299272,17cyclictest26346irq/40-dwc2_hso11:20:152
1571299272,11cyclictest0-21swapper/210:05:162
15712992713,12cyclictest27-21ksoftirqd/207:30:172
1569499274,20cyclictest9799-21unixbench_singl10:15:280
15694992712,11cyclictest21222-21apt-get12:30:120
15720992612,9cyclictest29566irq/58-eth012:20:273
1571299262,9cyclictest0-21swapper/208:10:152
1571299262,21cyclictest0-21swapper/210:40:142
1571299262,19cyclictest0-21swapper/208:25:162
1571299261,4cyclictest27-21ksoftirqd/210:30:012
15712992610,13cyclictest20512-21awk07:10:322
15712992610,13cyclictest20512-21awk07:10:322
1570499262,17cyclictest7845-21ssh09:35:131
1570499261,22cyclictest2151-21sshd09:10:151
1569499264,17cyclictest0-21swapper/008:20:130
15694992620,3cyclictest24778-21apt-get09:00:130
15694992620,3cyclictest24778-21apt-get09:00:120
15694992615,8cyclictest19219-21netstat11:05:240
1572099257,6cyclictest29566irq/58-eth010:55:143
1572099253,14cyclictest29566irq/58-eth010:25:143
1571299257,6cyclictest131rcu_preempt12:37:362
1571299256,3cyclictest131rcu_preempt11:45:012
1571299253,16cyclictest28385-21ssh11:55:162
1571299253,15cyclictest26346irq/40-dwc2_hso07:40:152
1571299251,7cyclictest131rcu_preempt10:10:202
1571299251,13cyclictest26346irq/40-dwc2_hso08:30:132
15712992511,11cyclictest26168-21ssh12:30:372
1570499252,18cyclictest856-21sshd11:05:131
1569499259,12cyclictest0-21swapper/011:55:150
1569499254,9cyclictest28807-21ssh11:11:530
1569499253,14cyclictest14752-21sh10:00:150
1569499252,18cyclictest23546-21ssh11:50:240
15694992519,3cyclictest10319-21apt-get08:45:150
15694992519,2cyclictest29492-21apt-get12:35:140
15694992511,10cyclictest4705-21tr11:40:210
1571299247,8cyclictest27-21ksoftirqd/207:50:382
1571299247,6cyclictest6888-21ssh09:55:152
1571299247,11cyclictest27-21ksoftirqd/211:14:322
1571299243,9cyclictest27-21ksoftirqd/210:38:002
1571299243,5cyclictest27-21ksoftirqd/211:54:282
1571299241,16cyclictest13151-21sh12:25:152
15704992413,8cyclictest0-21swapper/107:45:141
15704992411,8cyclictest0-21swapper/112:30:131
1569499243,9cyclictest16829-21ssh10:41:440
1569499241,8cyclictest24215-21ntpq07:15:270
15694992416,6cyclictest20521-21sshd07:14:030
15694992416,6cyclictest20521-21sshd07:14:020
1572099233,14cyclictest29666irq/59-eth010:30:153
1572099233,13cyclictest0-21swapper/310:45:153
1572099232,12cyclictest29566irq/58-eth008:40:143
15720992310,7cyclictest29566irq/58-eth009:55:143
1571299239,8cyclictest27-21ksoftirqd/208:20:272
1571299238,6cyclictest4575-21ssh12:00:202
1571299236,14cyclictest612-21dbus-daemon11:46:012
1571299236,14cyclictest612-21dbus-daemon11:46:002
1571299236,11cyclictest27-21ksoftirqd/210:59:182
15712992314,6cyclictest0-21swapper/207:50:012
1571299231,12cyclictest11906-21tail09:35:302
15712992311,10cyclictest26845-21ssh11:32:132
1571299230,12cyclictest27-21ksoftirqd/210:23:132
1569499236,5cyclictest0-21swapper/010:10:150
1569499236,4cyclictest0-21swapper/009:48:210
1569499231,7cyclictest5507-21ssh09:53:180
1569499231,7cyclictest1770-21diskmemload11:00:120
15694992315,6cyclictest1-21systemd07:40:140
15694992311,9cyclictest0-21swapper/007:45:130
1569499231,16cyclictest8800-21ls12:20:280
15694992310,10cyclictest0-21swapper/009:20:130
1572099223,12cyclictest0-21swapper/312:25:143
1571299228,7cyclictest27-21ksoftirqd/212:05:492
1571299227,7cyclictest131rcu_preempt09:28:102
1571299224,13cyclictest261rcuc/208:35:302
1571299223,9cyclictest21375-21ssh09:41:452
1571299223,5cyclictest27-21ksoftirqd/208:45:252
1571299221,9cyclictest25783-21fschecks_time07:20:212
15712992212,7cyclictest0-21swapper/208:40:192
15712992211,8cyclictest0-21swapper/211:05:132
1571299221,10cyclictest28578-21kthreadcore07:55:202
1570499229,10cyclictest11235-21cron11:45:011
1570499221,18cyclictest8999-21uniq11:00:171
1570499221,17cyclictest30746-21ls11:15:161
1570499221,17cyclictest30746-21ls11:15:151
1570499221,16cyclictest13150-21rm12:25:141
1569499222,15cyclictest28538-21ssh09:25:270
15694992216,3cyclictest29615-21apt-get09:05:150
15694992212,7cyclictest15795-21irqstats08:50:180
15694992210,9cyclictest21171-21kthreadcore08:55:210
1569499220,11cyclictest18858-21ssh11:26:380
1571299213,12cyclictest27-21ksoftirqd/209:00:262
1571299213,12cyclictest27-21ksoftirqd/209:00:252
1571299211,6cyclictest27-21ksoftirqd/207:15:272
1571299211,14cyclictest27-21ksoftirqd/212:21:212
1571299211,11cyclictest15867-21kthreadcore08:50:192
1571299211,10cyclictest502-21grep09:05:242
1570499213,15cyclictest6677-21latency_hist09:55:021
15704992111,7cyclictest20285-21cron12:10:011
1570499211,15cyclictest23097-21tr11:30:221
1570499211,15cyclictest10376-21ls08:45:141
1569499218,10cyclictest0-21swapper/011:45:150
1569499218,10cyclictest0-21swapper/011:45:140
1569499212,4cyclictest12-21ksoftirqd/011:30:180
15694992110,9cyclictest21071-21ssh10:25:040
15694992110,8cyclictest13734-21cat08:15:020
1569499210,8cyclictest26664-21rm12:12:200
1572099209,8cyclictest0-21swapper/311:11:083
1572099207,9cyclictest321rcuc/310:40:143
1572099203,11cyclictest0-21swapper/309:40:133
1572099202,10cyclictest0-21swapper/308:45:153
15720992011,6cyclictest0-21swapper/309:53:173
1570499202,14cyclictest24304-21ssh12:10:271
1570499202,13cyclictest0-21swapper/107:15:131
15704992011,5cyclictest10832-21sh09:15:221
1569499208,8cyclictest5485-21/usr/sbin/munin08:40:140
1569499207,9cyclictest2796-21ntpq08:00:260
15694992011,7cyclictest28811-21diskstats10:30:170
15694992010,7cyclictest0-21swapper/007:20:160
1572099199,8cyclictest0-21swapper/312:37:043
1572099192,11cyclictest0-21swapper/308:10:053
1572099192,10cyclictest0-21swapper/307:35:153
1572099192,10cyclictest0-21swapper/307:35:153
15720991913,4cyclictest0-21swapper/311:30:043
15720991913,4cyclictest0-21swapper/310:05:263
15720991912,5cyclictest0-21swapper/309:25:273
15720991912,4cyclictest29566irq/58-eth012:05:133
15720991912,4cyclictest0-21swapper/312:30:293
15720991912,4cyclictest0-21swapper/310:35:283
15720991911,6cyclictest0-21swapper/309:17:033
15720991911,5cyclictest29566irq/58-eth010:50:153
1572099191,11cyclictest0-21swapper/307:10:173
1572099191,11cyclictest0-21swapper/307:10:163
15720991910,7cyclictest0-21swapper/309:10:153
15720991910,6cyclictest29566irq/58-eth011:55:153
15720991910,6cyclictest0-21swapper/311:47:133
15720991910,6cyclictest0-21swapper/311:47:123
15720991910,6cyclictest0-21swapper/311:44:403
15720991910,6cyclictest0-21swapper/310:11:403
15720991910,3cyclictest0-21swapper/311:15:143
15720991910,3cyclictest0-21swapper/311:15:133
1570499199,7cyclictest32475-21tail11:55:311
1570499191,9cyclictest18886-21ssh10:00:321
1570499191,15cyclictest4251-21cut08:05:171
1570499191,13cyclictest0-21swapper/107:25:151
1570499191,10cyclictest5647-21/usr/sbin/munin11:20:301
1570499190,14cyclictest6183-21ps12:00:221
1570499190,13cyclictest22757-21ls10:45:311
1572099189,6cyclictest0-21swapper/309:20:443
1572099182,10cyclictest0-21swapper/310:15:133
1572099182,10cyclictest0-21swapper/308:55:153
1572099182,10cyclictest0-21swapper/308:25:153
1572099182,10cyclictest0-21swapper/308:15:163
15720991812,4cyclictest0-21swapper/312:00:143
15720991812,4cyclictest0-21swapper/309:45:163
15720991812,4cyclictest0-21swapper/307:30:283
1572099181,11cyclictest0-21swapper/311:25:313
1572099181,10cyclictest0-21swapper/308:40:023
15720991810,6cyclictest0-21swapper/310:23:183
15720991810,5cyclictest29566irq/58-eth011:35:153
1570499189,6cyclictest19449-21sshd08:50:301
1570499188,7cyclictest643-21sshd08:35:141
1570499188,6cyclictest30809-21irqcore10:10:191
1570499186,3cyclictest131rcu_preempt07:20:231
1570499185,10cyclictest20981-21kthreadcore08:55:191
1570499182,11cyclictest15809-21ssh09:40:131
1570499181,9cyclictest19322-21kernelversion08:20:191
1570499181,6cyclictest12278-21latency_hist07:40:011
1570499181,6cyclictest12278-21latency_hist07:40:001
1570499181,12cyclictest31522-21idleruntime-cro08:00:011
15704991810,5cyclictest28678-21/usr/sbin/munin10:30:271
15704991810,5cyclictest10187-21latency_hist08:45:011
1569499181,14cyclictest725-21ls08:35:150
1572099179,6cyclictest0-21swapper/309:30:093
1572099179,6cyclictest0-21swapper/309:30:083
1572099179,5cyclictest29566irq/58-eth009:05:143
1572099172,10cyclictest0-21swapper/308:00:153
1572099172,10cyclictest0-21swapper/307:50:143
15720991711,4cyclictest0-21swapper/308:50:283
15720991711,4cyclictest0-21swapper/308:20:283
15720991711,4cyclictest0-21swapper/308:05:143
1570499178,6cyclictest0-21swapper/110:25:001
1570499174,11cyclictest30391-21latency_hist11:15:021
1570499174,10cyclictest27273-21timerwakeupswit08:25:301
1570499173,9cyclictest0-21swapper/110:35:161
1570499173,4cyclictest18401-21munin-run10:45:021
1570499173,10cyclictest13968-21ssh11:45:221
1570499173,10cyclictest13968-21ssh11:45:221
1570499172,11cyclictest7553-21latency_hist07:35:021
1570499171,5cyclictest31744-21ntp_states08:30:261
1570499171,13cyclictest28545-21apt-get11:35:141
1570499171,13cyclictest19425-21ntp_states07:10:261
1570499171,13cyclictest19425-21ntp_states07:10:251
1570499171,12cyclictest3583-21memory09:30:231
1570499171,12cyclictest3583-21memory09:30:231
1570499171,12cyclictest20423-21tr09:20:211
1572099161,9cyclictest29566irq/58-eth009:00:143
1572099161,9cyclictest29566irq/58-eth009:00:133
1570499168,6cyclictest9167-21tune2fs08:10:161
1570499164,8cyclictest21-21ksoftirqd/108:15:151
1570499160,8cyclictest12555-21fschecks_count07:40:181
1572099155,7cyclictest0-21swapper/307:25:163
1571299151,10cyclictest15811-21latency_hist07:10:012
1569499155,8cyclictest15685-21cyclictest07:10:010
1572099145,4cyclictest33-21ksoftirqd/307:55:183
1572099141,11cyclictest0-21swapper/307:45:003
1570499142,10cyclictest15723-21latency_hist07:10:021
1572099126,4cyclictest0-21swapper/307:45:163
1572099115,4cyclictest0-21swapper/307:10:023
1572099113,6cyclictest0-21swapper/307:15:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional