You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 04:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Wed Dec 03, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16730996838,18cyclictest29566irq/58-eth023:55:133
1670399612,52cyclictest12790-21ssh00:10:120
16730995429,12cyclictest29566irq/58-eth021:40:163
1670399471,7cyclictest131rcu_preempt21:40:150
16714994614,24cyclictest21-21ksoftirqd/119:35:151
1670399464,33cyclictest865-21ssh22:45:130
1670399462,32cyclictest12842-21ssh21:35:140
1671499452,33cyclictest1833-21rm23:05:131
1670399453,32cyclictest4189-21ssh00:25:150
1670399432,6cyclictest131rcu_preempt19:25:120
1672299405,24cyclictest0-21swapper/220:45:152
1670399403,33cyclictest19982-21ssh00:35:150
1673099394,24cyclictest29566irq/58-eth000:05:153
1673099392,20cyclictest29566irq/58-eth021:35:143
1672299392,22cyclictest27-21ksoftirqd/223:15:132
1671499390,34cyclictest28625-21ssh00:20:141
1670399397,18cyclictest12-21ksoftirqd/000:15:130
1673099381,22cyclictest29566irq/58-eth000:35:143
1671499381,33cyclictest10204-21ssh23:10:131
1672299372,18cyclictest23186-21cut23:35:242
1672299362,28cyclictest5145-21ssh22:26:242
1672299353,21cyclictest26346irq/40-dwc2_hso20:05:152
16714993523,8cyclictest21-21ksoftirqd/119:25:131
16703993518,10cyclictest14114-21cut23:30:230
1672299344,9cyclictest9193-21ssh22:50:142
1671499343,25cyclictest21276-21ssh21:40:161
1671499342,25cyclictest16452-21ssh22:35:161
1673099333,18cyclictest29566irq/58-eth000:30:133
1671499332,23cyclictest5696-21ssh21:50:161
1671499332,23cyclictest5696-21ssh21:50:151
16714993310,4cyclictest131rcu_preempt00:30:131
1670399331,22cyclictest28348-21sh21:25:140
16722993216,2cyclictest0-21swapper/223:20:122
1671499323,20cyclictest2792-21diskmemload23:55:151
1671499322,23cyclictest15337-21ssh22:15:161
16722993122,6cyclictest0-21swapper/222:40:152
16703993113,8cyclictest12-21ksoftirqd/023:20:110
1673099302,18cyclictest29566irq/58-eth021:05:143
1673099302,16cyclictest29566irq/58-eth021:55:133
1672299307,15cyclictest26346irq/40-dwc2_hso21:55:122
1672299305,15cyclictest26346irq/40-dwc2_hso20:30:132
1672299303,19cyclictest3009-21ssh23:25:132
1672299302,20cyclictest5331-21memory23:05:242
1671499302,22cyclictest32717-21ssh23:03:241
1671499302,22cyclictest32717-21ssh23:03:231
1670399304,5cyclictest131rcu_preempt22:25:120
1670399302,23cyclictest23083-21ssh22:38:000
16703993022,2cyclictest0-21swapper/000:30:130
1673099293,19cyclictest0-21swapper/323:08:233
1672299295,15cyclictest26346irq/40-dwc2_hso21:30:132
1671499292,21cyclictest17774-21ssh23:32:231
1671499292,20cyclictest19985-21ssh21:20:151
16714992914,7cyclictest16690-21cyclictest19:30:121
16730992810,8cyclictest29566irq/58-eth022:25:113
1673099280,16cyclictest29566irq/58-eth023:20:113
1672299286,16cyclictest15777-21kworker/2:020:50:162
1672299283,20cyclictest0-21swapper/222:14:222
1672299282,19cyclictest28388-21ssh23:40:132
1672299282,18cyclictest22974-21sh22:56:232
1672299282,11cyclictest22213-21ssh22:18:222
1672299282,11cyclictest0-21swapper/222:31:232
1672299282,10cyclictest7474-21ssh22:48:022
1672299281,19cyclictest23765-21ssh22:20:132
1671499284,16cyclictest21-21ksoftirqd/119:40:131
16714992814,11cyclictest0-21swapper/122:40:131
1670399282,19cyclictest31878-21ssh23:40:230
1673099273,16cyclictest0-21swapper/323:30:233
1673099273,16cyclictest0-21swapper/323:15:233
1673099272,19cyclictest0-21swapper/322:38:013
1672299274,16cyclictest27-21ksoftirqd/219:50:152
1672299273,15cyclictest26346irq/40-dwc2_hso20:10:142
1672299272,11cyclictest5008-21ssh22:07:552
1671499279,2cyclictest21-21ksoftirqd/122:13:031
1671499272,19cyclictest18599-21ssh23:15:121
1670399272,20cyclictest22178-21/usr/sbin/munin23:15:230
1670399272,19cyclictest13434-21ssh22:12:570
1672299263,20cyclictest0-21swapper/223:32:222
1672299263,19cyclictest0-21swapper/221:50:162
1672299263,19cyclictest0-21swapper/221:50:152
1672299263,16cyclictest27-21ksoftirqd/200:05:142
1672299263,15cyclictest26346irq/40-dwc2_hso19:55:142
1671499263,17cyclictest28373-21ssh23:40:131
1671499261,19cyclictest3008-21sh23:25:121
1670399263,18cyclictest2792-21diskmemload22:30:140
1670399262,15cyclictest0-21swapper/019:15:150
1670399262,15cyclictest0-21swapper/019:15:140
1673099253,16cyclictest0-21swapper/323:35:243
16730992513,8cyclictest29566irq/58-eth021:45:133
1673099251,17cyclictest29566irq/58-eth023:45:133
1672299254,16cyclictest26346irq/40-dwc2_hso20:35:162
1672299253,19cyclictest0-21swapper/223:03:242
1672299253,19cyclictest0-21swapper/223:03:242
1672299253,16cyclictest26346irq/40-dwc2_hso20:55:122
1672299252,19cyclictest24086-21ssh22:39:232
1672299251,17cyclictest26346irq/40-dwc2_hso20:40:122
1672299251,14cyclictest21156-21sh23:55:152
1671499259,11cyclictest21-21ksoftirqd/122:23:231
1671499252,20cyclictest28475-21ssh21:25:171
1673099243,14cyclictest0-21swapper/323:14:233
16730992412,8cyclictest29566irq/58-eth022:50:143
1672299241,20cyclictest316-21kthreadcore21:05:202
1671499242,16cyclictest21-21ksoftirqd/120:00:131
1671499242,16cyclictest11550-21ssh21:15:141
16714992413,8cyclictest11720-21cron00:30:001
16714992412,9cyclictest20724-21cron20:55:011
1671499241,14cyclictest4797-21ssh00:05:141
1670399246,9cyclictest131rcu_preempt23:35:150
16703992417,3cyclictest0-21swapper/020:35:140
1673099233,13cyclictest0-21swapper/323:40:233
1673099232,10cyclictest0-21swapper/322:43:023
16730992313,7cyclictest29566irq/58-eth022:15:153
1672299235,13cyclictest42-21kauditd00:30:012
1672299232,16cyclictest0-21swapper/223:10:232
1671499239,6cyclictest21-21ksoftirqd/122:53:111
1671499237,12cyclictest21-21ksoftirqd/122:49:231
1671499232,15cyclictest3475-21kworker/u8:222:25:131
1671499231,19cyclictest2792-21diskmemload21:45:131
16714992310,7cyclictest0-21swapper/123:20:141
1670399231,18cyclictest12828-21apt-get23:50:150
1673099226,10cyclictest29566irq/58-eth022:45:143
16730992212,7cyclictest0-21swapper/300:00:143
16730992210,7cyclictest0-21swapper/321:25:133
1672299222,15cyclictest26346irq/40-dwc2_hso20:15:142
1672299222,14cyclictest26346irq/40-dwc2_hso21:40:152
1672299221,5cyclictest0-21swapper/221:20:132
16722992215,4cyclictest626-21tcf-agent19:40:312
16722992212,6cyclictest0-21swapper/219:10:262
1672299221,11cyclictest932-21snmpd00:00:122
16714992213,5cyclictest16958-21usage_Anybus23:50:321
16714992212,5cyclictest21-21ksoftirqd/121:35:121
1671499221,15cyclictest22543-21ssh22:00:131
1670399229,9cyclictest1824-21apt-get23:05:140
1670399228,5cyclictest12-21ksoftirqd/021:55:130
1673099218,7cyclictest29566irq/58-eth023:50:293
1672299219,10cyclictest20393-21sshd21:39:212
1672299217,6cyclictest12985-21fschecks_count23:50:182
1672299214,8cyclictest20181-21sh00:35:182
1672299212,13cyclictest26346irq/40-dwc2_hso20:20:142
16722992111,8cyclictest0-21swapper/222:00:122
1672299211,16cyclictest3772-21irqrtprio21:10:192
1672299211,16cyclictest3772-21irqrtprio21:10:182
1672299211,16cyclictest12086-21ssh00:30:132
1672299211,10cyclictest8551-21tail23:45:292
1672299211,10cyclictest29051-21sh21:25:202
1672299211,10cyclictest23919-21kthreadcore00:15:232
1672299211,10cyclictest22853-21idleruntime-cro19:50:022
1672299211,10cyclictest13381-21cat19:40:022
1671499219,6cyclictest0-21swapper/100:00:131
1671499212,2cyclictest131rcu_preempt21:11:131
1671499212,2cyclictest131rcu_preempt21:11:131
16714992113,4cyclictest21-21ksoftirqd/122:55:181
16714992111,5cyclictest21-21ksoftirqd/121:00:131
1671499211,14cyclictest14090-21find21:55:121
16714992110,2cyclictest21-21ksoftirqd/122:08:081
1670399218,10cyclictest0-21swapper/019:50:220
1670399212,14cyclictest29611-21sh21:45:130
1670399212,13cyclictest6621-21ntp_192.168.11423:25:230
1670399211,17cyclictest26644-21cut19:20:190
1670399211,16cyclictest24915-21find22:40:130
1673099201,5cyclictest0-21swapper/321:10:143
1673099201,5cyclictest0-21swapper/321:10:133
16730992011,4cyclictest0-21swapper/323:25:263
1672299208,10cyclictest6075-21kthreadcore19:30:222
1672299208,10cyclictest13908-21kthreadcore21:15:212
1672299201,9cyclictest29630-21cpu21:45:142
1672299201,9cyclictest26857-21sed19:20:212
1672299201,9cyclictest16743-21cat19:10:022
1672299201,9cyclictest13644-21cut00:10:182
1672299201,3cyclictest0-21swapper/200:20:292
1672299201,17cyclictest0-21swapper/219:15:182
1672299201,17cyclictest0-21swapper/219:15:182
1672299201,15cyclictest0-21swapper/220:25:182
16714992014,4cyclictest0-21swapper/119:45:181
16714992014,4cyclictest0-21swapper/100:35:261
16714992013,3cyclictest21-21ksoftirqd/119:10:141
1671499201,14cyclictest19354-21latency_hist20:20:021
1671499201,12cyclictest131rcu_preempt00:15:571
1670399209,9cyclictest16690-21cyclictest19:10:020
1670399207,4cyclictest131rcu_preempt20:10:140
1670399202,14cyclictest23783-21sh22:20:140
1670399202,13cyclictest32514-21ssh23:03:060
1670399202,13cyclictest32514-21ssh23:03:050
1670399201,11cyclictest1971-21wc22:05:220
1673099199,7cyclictest29566irq/58-eth022:05:133
1673099193,11cyclictest0-21swapper/322:24:243
1673099192,7cyclictest0-21swapper/323:00:293
1673099192,7cyclictest0-21swapper/323:00:293
1673099192,13cyclictest0-21swapper/322:55:273
1673099192,11cyclictest0-21swapper/320:00:153
16730991911,5cyclictest29566irq/58-eth022:10:143
1673099191,13cyclictest0-21swapper/320:25:253
1673099191,13cyclictest0-21swapper/319:15:013
1673099191,12cyclictest0-21swapper/319:25:293
1673099191,12cyclictest0-21swapper/300:25:023
1672299191,8cyclictest419-21fschecks_count20:00:172
1672299191,10cyclictest31544-21fschecks_time19:25:172
1672299191,10cyclictest28538-21kthreadcore21:00:232
1671499198,2cyclictest21-21ksoftirqd/122:32:161
16714991913,4cyclictest0-21swapper/120:10:191
16714991912,5cyclictest0-21swapper/120:40:181
16714991912,4cyclictest0-21swapper/120:35:291
16714991911,5cyclictest0-21swapper/120:25:281
16714991911,4cyclictest20744-21kthreadcore23:35:211
1670399193,11cyclictest22553-21ssh22:00:140
1670399192,12cyclictest5686-21ssh21:50:150
1670399192,12cyclictest5686-21ssh21:50:150
1670399192,11cyclictest4437-21ssh23:45:140
1670399191,5cyclictest131rcu_preempt00:00:210
1670399191,5cyclictest0-21swapper/022:17:380
16703991911,6cyclictest11021-21cron20:45:020
1670399191,15cyclictest8061-21ssh21:30:270
1670399191,14cyclictest20959-21apt-get20:55:110
1670399191,14cyclictest17541-21ssh22:55:150
1670399191,14cyclictest11255-21apt-get20:45:140
1670399191,13cyclictest24743-21ssh23:55:260
1670399191,12cyclictest8792-21ls20:05:270
1670399191,11cyclictest3234-21cpuspeed21:10:150
1670399191,11cyclictest3234-21cpuspeed21:10:150
1673099188,6cyclictest0-21swapper/300:29:423
1673099188,3cyclictest0-21swapper/321:23:433
1673099183,10cyclictest0-21swapper/322:30:233
1673099182,10cyclictest0-21swapper/321:50:163
1673099182,10cyclictest0-21swapper/321:50:153
1673099182,10cyclictest0-21swapper/320:55:143
1673099182,10cyclictest0-21swapper/320:40:143
1673099182,10cyclictest0-21swapper/320:10:163
1673099182,10cyclictest0-21swapper/300:10:043
1673099181,9cyclictest0-21swapper/320:05:143
16730991812,4cyclictest0-21swapper/321:15:273
1673099181,13cyclictest0-21swapper/322:00:163
1673099181,12cyclictest0-21swapper/319:35:193
1673099181,11cyclictest0-21swapper/321:30:153
1671499186,9cyclictest21-21ksoftirqd/100:12:251
1671499181,3cyclictest21-21ksoftirqd/121:30:131
16714991812,4cyclictest0-21swapper/120:55:181
16714991812,4cyclictest0-21swapper/120:20:271
16714991812,3cyclictest21-21ksoftirqd/120:50:021
16714991811,5cyclictest0-21swapper/119:15:281
16714991811,5cyclictest0-21swapper/119:15:281
16714991811,3cyclictest21-21ksoftirqd/119:55:141
1670399188,7cyclictest14737-21apt-get20:15:150
1670399181,4cyclictest0-21swapper/023:12:270
1673099172,9cyclictest0-21swapper/320:50:053
1673099172,9cyclictest0-21swapper/320:35:153
1673099172,9cyclictest0-21swapper/320:20:153
1673099172,10cyclictest0-21swapper/320:45:163
1673099172,10cyclictest0-21swapper/300:15:143
1673099171,13cyclictest0-21swapper/319:55:023
1673099171,13cyclictest0-21swapper/319:40:183
1671499175,9cyclictest4289-21ssh23:45:021
16714991712,3cyclictest0-21swapper/119:20:171
16714991711,4cyclictest0-21swapper/119:50:201
1670399179,6cyclictest17340-21cut20:50:220
1670399175,9cyclictest28305-21tr21:00:230
1670399172,13cyclictest402-21fschecks_count20:00:170
1670399172,10cyclictest0-21swapper/021:05:150
1670399171,9cyclictest20591-21kthreadcore20:20:210
1670399171,8cyclictest9340-21fschecks_count22:50:170
1670399171,13cyclictest17007-21uniq19:10:160
1670399171,12cyclictest8356-21ntp_192.168.11400:05:260
16703991710,4cyclictest24260-21usage_Anybus21:20:320
1673099161,12cyclictest0-21swapper/320:15:253
1673099161,12cyclictest0-21swapper/319:35:013
1671499163,4cyclictest29529-21ps20:30:171
1671499162,11cyclictest30829-21tune2fs21:05:171
16714991610,4cyclictest5460-21ps20:05:191
1670399168,6cyclictest20669-21cut19:45:200
1670399167,6cyclictest6554-21cut19:30:220
1670399166,8cyclictest12-21ksoftirqd/020:25:310
1670399166,7cyclictest28679-21cpu00:20:150
1670399162,11cyclictest11919-21sed21:15:180
1670399161,7cyclictest842-21munin-node20:30:130
1670399161,12cyclictest13582-21tune2fs19:40:180
1670399161,10cyclictest30472-21kthreadcore19:55:220
1670399151,4cyclictest0-21swapper/019:35:190
1673099141,8cyclictest0-21swapper/319:15:233
1673099141,8cyclictest0-21swapper/319:15:233
1671499146,6cyclictest16742-21latency_hist19:10:021
1673099131,9cyclictest0-21swapper/319:20:213
1673099131,5cyclictest0-21swapper/319:45:193
1673099131,10cyclictest0-21swapper/320:30:293
1673099121,9cyclictest0-21swapper/320:00:013
1673099121,7cyclictest0-21swapper/319:10:023
1673099121,5cyclictest0-21swapper/321:00:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional