You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 20:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Nov 29, 2025 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1103399541,37cyclictest24045-21ssh10:30:141
1102499533,7cyclictest131rcu_preempt10:45:160
1103399502,6cyclictest131rcu_preempt08:00:161
1104199491,35cyclictest30005-21ssh09:10:132
1104199452,37cyclictest9146-21ssh12:05:162
1103399453,36cyclictest8531-21ssh10:20:161
1105299444,24cyclictest29566irq/58-eth009:25:133
1103399403,34cyclictest5716-21ssh11:00:141
1104199392,34cyclictest0-21swapper/212:20:152
11033993828,6cyclictest18396-21sh12:30:151
11041993717,10cyclictest27-21ksoftirqd/211:50:142
11041993712,16cyclictest27-21ksoftirqd/211:45:172
11052993625,6cyclictest0-21swapper/310:10:143
1104199341,31cyclictest1815-21ssh09:55:152
1104199333,7cyclictest0-21swapper/208:30:122
1104199333,18cyclictest28401-21sshd08:35:142
1104199322,19cyclictest13355-21ssh09:20:142
1102499323,23cyclictest5529-21ssh09:15:140
1104199313,21cyclictest0-21swapper/211:30:052
1104199312,20cyclictest26346irq/40-dwc2_hso08:15:142
1102499317,3cyclictest12-21ksoftirqd/008:05:150
1104199304,17cyclictest26346irq/40-dwc2_hso07:25:142
11041993017,8cyclictest27-21ksoftirqd/207:10:132
11052992922,4cyclictest29566irq/58-eth012:10:163
1104199299,12cyclictest0-21swapper/207:50:152
1104199295,16cyclictest261rcuc/210:35:142
11041992910,15cyclictest20719-21sshd08:25:222
1104199282,14cyclictest25811-21sh12:15:172
1104199282,14cyclictest25811-21sh12:15:162
11041992813,9cyclictest0-21swapper/211:55:152
1104199281,21cyclictest3708-21sh09:35:152
11041992812,13cyclictest27-21ksoftirqd/211:05:132
1104199281,18cyclictest1495-21diskmemload12:00:152
1102499282,21cyclictest18403-21ssh12:30:160
1102499282,19cyclictest9755-21rm10:00:150
1105299272,13cyclictest29566irq/58-eth010:15:153
1104199273,13cyclictest26346irq/40-dwc2_hso11:00:152
1104199272,14cyclictest22597-21ssh09:45:252
11041992714,9cyclictest0-21swapper/210:10:302
1102499272,23cyclictest28627-21ssh11:15:140
1102499272,18cyclictest856-21sshd08:00:150
1104199264,14cyclictest26346irq/40-dwc2_hso11:35:132
1104199264,14cyclictest26346irq/40-dwc2_hso11:10:152
1104199263,16cyclictest26346irq/40-dwc2_hso08:05:142
1104199262,21cyclictest30228-21switchtime09:50:302
1104199262,20cyclictest0-21swapper/207:15:142
1104199262,11cyclictest0-21swapper/210:30:142
11041992615,5cyclictest27229-21ssh12:35:152
1104199261,13cyclictest932-21snmpd08:50:152
1104199253,17cyclictest26346irq/40-dwc2_hso08:20:132
1104199253,15cyclictest2393-21ssh11:40:172
1104199252,4cyclictest0-21swapper/207:35:182
1104199251,16cyclictest26346irq/40-dwc2_hso09:40:122
1102499258,9cyclictest0-21swapper/010:15:150
1105299242,13cyclictest29566irq/58-eth010:45:153
11052992411,10cyclictest0-21swapper/310:50:213
11041992415,5cyclictest27-21ksoftirqd/210:05:152
11041992413,7cyclictest23671-21cron10:30:012
11041992412,9cyclictest13892-21ssh10:21:342
1104199241,12cyclictest9711-21ssh12:25:162
1104199240,12cyclictest3939-21grep11:20:172
1103399241,19cyclictest22569-21ssh10:50:151
1102499242,17cyclictest25428-21ssh10:10:160
1102499242,17cyclictest11323-21ssh09:40:140
1104199236,9cyclictest23120-21cut10:50:202
1104199231,19cyclictest30265-21find10:55:132
1102499231,16cyclictest7979-21cut07:40:170
1105299224,12cyclictest0-21swapper/308:00:153
1105299222,15cyclictest0-21swapper/310:20:163
1105299221,14cyclictest0-21swapper/309:15:143
1104199227,12cyclictest27-21ksoftirqd/210:45:072
1104199224,11cyclictest261rcuc/212:10:142
1104199223,8cyclictest9212-21kthreadcore10:40:232
1104199223,8cyclictest9212-21kthreadcore10:40:232
1104199222,17cyclictest969-21fschecks_time10:15:192
11041992210,9cyclictest18508-21ssh12:30:172
1103399222,15cyclictest662-21ntpd10:25:161
11033992212,7cyclictest612-21dbus-daemon11:35:521
11033992212,4cyclictest6794-21cut09:35:231
1103399220,2cyclictest26384-21ssh09:50:141
1102499222,13cyclictest0-21swapper/011:10:150
1102499221,9cyclictest28011-21ssh12:35:210
1102499221,8cyclictest13699-21munin-run08:20:010
11024992215,4cyclictest26396-21sh09:50:150
1102499221,14cyclictest25768-21find12:15:140
1102499221,14cyclictest25768-21find12:15:140
1102499221,11cyclictest17901-21kthreadcore10:25:220
1105299212,10cyclictest0-21swapper/312:30:143
11052992114,4cyclictest29566irq/58-eth012:15:163
11052992114,4cyclictest29566irq/58-eth012:15:163
11052992111,7cyclictest0-21swapper/310:25:213
1104199217,6cyclictest15229-21/usr/sbin/munin11:25:302
1104199217,5cyclictest20947-21cpuspeed09:25:142
1104199215,14cyclictest21308-21irqcore07:20:202
1104199213,15cyclictest30719-21grep07:30:162
1104199212,9cyclictest6698-21kthreadcore08:45:202
1104199212,10cyclictest0-21swapper/209:05:172
1104199211,18cyclictest12922-21sed07:45:182
1104199211,17cyclictest31799-21latency_hist08:05:022
1104199211,16cyclictest15504-21cut08:55:162
1104199211,11cyclictest5793-21idleruntime09:15:172
1104199211,10cyclictest24817-21awk09:05:012
1103399212,14cyclictest744-21ssh10:15:161
1103399211,9cyclictest30333-21latency_hist07:30:011
1103399211,7cyclictest7257-21tr09:15:211
11033992113,5cyclictest856-21sshd09:46:191
11033992111,8cyclictest19288-21df11:30:151
1103399211,16cyclictest16273-21cut07:15:161
1102499217,4cyclictest131rcu_preempt12:20:300
1102499211,15cyclictest31784-21apt-get10:35:140
1102499210,16cyclictest4345-21ls08:10:160
1105299203,11cyclictest0-21swapper/311:15:153
11052992013,5cyclictest29566irq/58-eth009:35:153
11052992012,5cyclictest29566irq/58-eth009:45:143
11052992011,7cyclictest0-21swapper/309:20:253
1104199201,9cyclictest22299-21sort07:55:152
1104199201,14cyclictest29051-21irqcore11:15:192
1104199201,12cyclictest3692-21kthreadcore08:40:232
1104199201,11cyclictest11121-21kthreadcore10:00:212
1104199201,10cyclictest29087-21irqrtprio09:30:192
1104199201,10cyclictest10811-21kthreadcore07:40:232
1103399205,10cyclictest1585-21diskmemload12:20:161
1103399201,16cyclictest20833-21ssh11:10:141
1103399201,10cyclictest1063366ssh10:40:271
1103399201,10cyclictest1063366ssh10:40:271
11033992010,7cyclictest17040-21ntp_states09:20:271
1102499209,8cyclictest5720-21df08:45:150
1102499209,8cyclictest13342-21apt-get09:20:120
1102499201,6cyclictest512-21munin-run08:40:010
11024992014,3cyclictest0-21swapper/012:10:150
11024992013,4cyclictest131rcu_preempt08:40:140
11024992013,4cyclictest12731-21ntp_192.168.11412:05:270
11024992012,5cyclictest0-21swapper/011:30:020
1102499201,16cyclictest5835-21ls09:55:310
1102499201,16cyclictest10069-21uniq11:45:190
1105299193,12cyclictest29566irq/58-eth011:10:143
1105299192,10cyclictest0-21swapper/309:30:143
1105299191,11cyclictest0-21swapper/309:05:193
11052991910,5cyclictest29566irq/58-eth011:25:163
1104199191,10cyclictest4377-21grep08:10:182
1103399197,9cyclictest12328-21kthreadcore08:15:231
1103399193,3cyclictest131rcu_preempt11:05:211
1103399193,11cyclictest30281-21ssh10:55:131
1103399192,3cyclictest0-21swapper/112:05:161
1103399192,13cyclictest14843-21ssh10:45:151
1103399191,9cyclictest0-21swapper/108:25:221
1103399191,4cyclictest131rcu_preempt07:50:281
11033991912,4cyclictest25975-21if_eth111:55:191
1103399191,15cyclictest17394-21apt-get12:10:141
1103399191,13cyclictest11313-21ssh09:40:131
1103399191,11cyclictest29646-21diskmemload09:13:291
1103399191,10cyclictest30273-21tail08:35:211
11033991910,7cyclictest32000-21awk08:05:171
11033991910,7cyclictest2975-21ntp_states10:35:271
1103399190,15cyclictest20951-21ls11:50:241
1102499197,3cyclictest131rcu_preempt09:10:210
1102499191,9cyclictest24878-21unixbench_multi09:25:300
1102499191,8cyclictest9671-21/usr/sbin/munin12:25:300
11024991912,5cyclictest0-21swapper/011:50:240
11024991911,6cyclictest28451-21sshd10:52:210
1102499191,15cyclictest26985-21ssh11:35:150
1102499191,11cyclictest31613-21date10:35:030
1102499191,10cyclictest6975-21ssh10:40:140
1102499191,10cyclictest6975-21ssh10:40:140
1105299189,6cyclictest0-21swapper/311:33:423
1105299189,6cyclictest0-21swapper/311:05:433
1105299184,8cyclictest0-21swapper/310:40:143
1105299184,8cyclictest0-21swapper/310:40:143
11052991812,4cyclictest0-21swapper/310:35:213
1105299181,12cyclictest0-21swapper/308:55:283
11052991810,3cyclictest0-21swapper/310:56:163
11052991810,3cyclictest0-21swapper/310:06:033
1104199182,13cyclictest11064-21latency_hist07:10:012
1103399187,9cyclictest28931-21grep09:05:321
1103399184,4cyclictest21-21ksoftirqd/110:10:321
1103399181,9cyclictest29646-21diskmemload11:28:101
1103399181,8cyclictest11183-21kthreadcore12:25:221
1103399181,3cyclictest131rcu_preempt08:55:191
1103399181,13cyclictest1833-21ssh09:55:161
11033991810,6cyclictest6780-21sshd12:01:211
11033991810,5cyclictest9754-21sh10:00:151
11033991810,5cyclictest2746-21interrupts11:40:201
1103399180,12cyclictest28536-21cpuspeed09:30:151
1103399180,11cyclictest20819-21ssh09:25:071
1102499189,7cyclictest20895-21ps09:00:210
1102499188,7cyclictest30367-21kthreadcore09:30:210
1102499187,8cyclictest25652-21diskmemload11:55:150
1102499184,11cyclictest13494-21grep11:05:180
1102499181,8cyclictest22898-21uname07:55:190
1102499181,7cyclictest17606-21/usr/sbin/munin10:05:160
11024991812,4cyclictest15376-21cpu08:55:140
11024991812,4cyclictest12118-21nfs_client10:20:260
1102499181,14cyclictest2394-21ssh11:40:170
1102499181,13cyclictest11183-21apt-get07:10:120
1102499181,11cyclictest1716-21uniq12:00:180
11024991810,6cyclictest29634-21cron09:10:010
1105299179,6cyclictest0-21swapper/312:29:463
1105299179,6cyclictest0-21swapper/311:35:423
1105299179,6cyclictest0-21swapper/310:33:373
1105299179,6cyclictest0-21swapper/309:53:063
1105299179,5cyclictest0-21swapper/310:00:223
1105299177,7cyclictest29566irq/58-eth012:05:163
1105299177,6cyclictest0-21swapper/311:40:173
11052991710,4cyclictest29566irq/58-eth012:35:183
1103399179,6cyclictest8869-21cron08:15:011
1103399176,8cyclictest31071-21sed07:30:191
1103399175,2cyclictest91-21pr/ttyS009:05:011
1103399173,9cyclictest0-21swapper/111:15:151
1103399172,13cyclictest11453-21switchtime07:40:281
1103399171,4cyclictest0-21swapper/111:24:011
1103399171,3cyclictest131rcu_preempt12:16:191
1103399171,3cyclictest131rcu_preempt12:16:191
1103399171,13cyclictest2507-21kthreadcore08:40:221
1103399171,12cyclictest14703-21users11:45:341
1103399171,11cyclictest27287-21ls12:35:161
1103399171,10cyclictest5821-21cut08:45:171
1103399171,10cyclictest0-21swapper/108:50:051
1102499177,7cyclictest20631-21users07:15:310
1102499173,11cyclictest27625-21ps07:25:210
1102499172,12cyclictest8787-21kthreadcore11:00:230
1102499172,12cyclictest4448-21irqrtprio11:20:190
1102499172,12cyclictest32228-21kthreadcore10:55:220
1102499171,6cyclictest0-21swapper/008:20:190
1102499171,2cyclictest514146kthreadcore07:35:250
1102499171,13cyclictest3707-21ssh09:35:150
1102499171,12cyclictest19199-21ssh11:30:140
1105299168,6cyclictest0-21swapper/312:20:143
1105299168,6cyclictest0-21swapper/312:00:233
1105299168,6cyclictest0-21swapper/308:27:273
1105299168,5cyclictest29566irq/58-eth009:55:153
1105299168,5cyclictest29566irq/58-eth009:40:123
1105299168,5cyclictest29566irq/58-eth009:10:143
1105299161,12cyclictest0-21swapper/308:35:243
11052991610,4cyclictest0-21swapper/311:51:123
11052991610,4cyclictest0-21swapper/311:21:433
11052991610,4cyclictest0-21swapper/311:01:383
1103399169,5cyclictest15074-21grep07:10:281
1103399168,6cyclictest25512-21sshd07:24:521
1103399161,7cyclictest24685-21kthreadcore08:30:191
11033991611,3cyclictest12837-21idleruntime07:45:171
11033991610,3cyclictest21355-21switchtime10:05:301
1102499162,11cyclictest13296-21kthreadcore07:45:190
1102499161,12cyclictest30994-21sed07:30:170
1102499161,12cyclictest25784-21kthreadcore08:30:200
1102499161,12cyclictest22163-21kthreadcore07:20:220
1102499161,12cyclictest19173-21sed08:25:190
1102499161,12cyclictest19011-21ssh09:45:150
1102499161,12cyclictest18888-21kthreadcore07:50:210
1102499161,12cyclictest10875-21ps08:50:190
1105299157,6cyclictest0-21swapper/308:45:313
1105299157,6cyclictest0-21swapper/307:30:283
1105299153,9cyclictest0-21swapper/311:55:143
1105299152,8cyclictest0-21swapper/307:40:143
11052991510,3cyclictest0-21swapper/311:45:313
1103399158,5cyclictest4208-21kthreadcore07:35:241
1103399156,6cyclictest14464-21irqrtprio08:20:181
1103399151,11cyclictest22262-21cpu07:55:141
1105299149,3cyclictest0-21swapper/307:25:223
1105299149,3cyclictest0-21swapper/307:10:253
1105299141,8cyclictest0-21swapper/308:40:203
1105299141,6cyclictest0-21swapper/308:15:223
1105299141,10cyclictest0-21swapper/308:50:193
1103399141,9cyclictest11088-21latency_hist07:10:011
1102499141,9cyclictest11091-21wc07:10:010
1105299138,3cyclictest0-21swapper/307:54:363
1105299132,7cyclictest0-21swapper/307:45:123
1105299131,9cyclictest0-21swapper/307:25:013
1105299131,9cyclictest0-21swapper/307:10:013
1105299131,7cyclictest0-21swapper/308:05:243
1105299131,5cyclictest0-21swapper/308:10:313
1105299131,10cyclictest0-21swapper/308:30:193
1105299131,10cyclictest0-21swapper/308:20:173
1105299123,7cyclictest0-21swapper/307:55:233
1105299122,7cyclictest0-21swapper/307:40:013
1105299121,8cyclictest0-21swapper/309:00:183
1105299111,7cyclictest0-21swapper/307:15:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional