You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-13 - 09:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Nov 13, 2025 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1103299624,6cyclictest131rcu_preempt23:30:160
1105399591,38cyclictest22065-21ssh00:25:162
1103299594,48cyclictest14424-21ssh23:20:140
1103299585,41cyclictest31164-21ssh00:10:130
1103299521,10cyclictest131rcu_preempt19:25:170
11053995018,24cyclictest23691-21ssh21:45:142
1106199482,31cyclictest29566irq/58-eth023:55:163
11053994711,22cyclictest27-21ksoftirqd/222:15:152
1104299454,29cyclictest22963-21ssh00:05:141
11042994227,10cyclictest14400-21apt-get23:20:141
1104299421,5cyclictest131rcu_preempt22:15:141
1106199416,17cyclictest0-21swapper/323:20:143
1106199413,25cyclictest29566irq/58-eth021:10:163
1106199413,24cyclictest29566irq/58-eth022:45:173
11042994135,3cyclictest15545-21apt-get20:55:141
1106199405,22cyclictest29566irq/58-eth021:20:133
1104299393,25cyclictest5901-21ssh21:15:121
1106199382,22cyclictest0-21swapper/321:40:133
1106199382,22cyclictest0-21swapper/321:40:133
1105399382,28cyclictest31035-21diskmemload21:30:182
1105399381,19cyclictest6463-21ssh23:55:152
11042993831,4cyclictest15319-21apt-get21:40:141
11042993831,4cyclictest15319-21apt-get21:40:141
1103299387,23cyclictest8758-1kworker/u9:221:10:150
11032993815,17cyclictest12-21ksoftirqd/021:20:130
1104299379,10cyclictest22884-21ssh22:04:041
11042993719,11cyclictest0-21swapper/123:05:131
1106199368,22cyclictest0-21swapper/300:30:143
1106199365,19cyclictest29566irq/58-eth021:50:153
1106199363,24cyclictest29566irq/58-eth022:05:153
11053993615,17cyclictest0-21swapper/200:10:142
11042993613,10cyclictest10036-21ntp_192.168.11400:15:241
1103299363,24cyclictest23831-21ssh22:05:140
11042993510,11cyclictest5129-21ssh21:14:231
11042993510,10cyclictest10332-21rm00:36:051
11042993510,10cyclictest10332-21rm00:36:051
1104299349,8cyclictest30476-21ssh21:29:511
1104299349,10cyclictest21989-21ssh23:03:591
1104299349,10cyclictest12124-21sh21:36:031
1104299345,11cyclictest22474-21ssh00:04:521
11042993414,16cyclictest0-21swapper/123:35:141
11042993410,8cyclictest20017-21ssh21:21:501
1103299344,20cyclictest29814-21ssh00:30:140
1103299344,19cyclictest12-21ksoftirqd/020:35:160
1106199334,19cyclictest29566irq/58-eth023:50:133
1106199333,21cyclictest29566irq/58-eth021:30:173
1105399336,22cyclictest5208-21ssh22:32:422
1105399336,20cyclictest20549-21kworker/u8:021:10:152
11053993319,9cyclictest20378-21ssh23:22:212
1104299339,8cyclictest29809-21diskmemload22:47:271
1104299333,20cyclictest9650irq/52-mmc019:20:131
1105399323,18cyclictest26346irq/40-dwc2_hso19:50:142
1104299329,7cyclictest0-21swapper/123:54:591
1104299327,9cyclictest9345-21processes20:45:261
1104299324,18cyclictest31021-21ssh21:30:171
11042993227,3cyclictest15434-21apt-get22:20:131
11042993225,4cyclictest28565-21apt-get20:35:161
11042993211,7cyclictest31350-21ssh22:09:501
1103299323,18cyclictest23357-21ssh22:45:170
1106199315,15cyclictest29566irq/58-eth022:55:133
1106199312,16cyclictest29566irq/58-eth023:10:133
1105399312,24cyclictest22606-21ssh21:25:152
1104299319,8cyclictest14392-21ssh22:39:231
1104299319,13cyclictest22350-21ntp_192.168.11420:25:291
1104299318,9cyclictest27026-21ntp_192.168.11422:25:271
1104299318,8cyclictest29809-21diskmemload22:44:151
1104299318,10cyclictest30828-21irqcore19:30:171
1104299317,7cyclictest0-21swapper/123:31:131
1104299316,7cyclictest0-21swapper/121:45:051
11042993125,2cyclictest6451-21apt-get23:55:171
11042993110,7cyclictest29809-21diskmemload00:33:071
11061993016,10cyclictest29566irq/58-eth021:15:123
11061993013,8cyclictest29566irq/58-eth000:25:153
1105399308,18cyclictest11019-21cyclictest23:35:142
1105399305,19cyclictest26346irq/40-dwc2_hso00:35:152
1105399305,19cyclictest26346irq/40-dwc2_hso00:35:152
1105399304,14cyclictest26346irq/40-dwc2_hso19:10:142
1105399302,17cyclictest6498-21ssh00:15:142
1104299309,9cyclictest30868-21wc20:00:281
1104299306,7cyclictest0-21swapper/122:56:141
1104299305,11cyclictest5388-21rm22:53:021
1104299302,19cyclictest31024-21ssh23:10:141
11042993010,9cyclictest6322-21diskmemload22:13:491
1103299303,20cyclictest31018-21ssh23:10:130
11032993019,8cyclictest0-21swapper/000:35:150
11032993019,8cyclictest0-21swapper/000:35:150
1106199294,13cyclictest29566irq/58-eth021:35:143
1105399293,13cyclictest26346irq/40-dwc2_hso20:00:132
1105399292,12cyclictest0-21swapper/223:50:132
1105399291,19cyclictest15341-21ssh21:40:142
1105399291,19cyclictest15341-21ssh21:40:142
1104299297,8cyclictest0-21swapper/121:57:521
1104299296,7cyclictest29809-21diskmemload00:12:111
1104299292,19cyclictest14073-21ssh23:40:101
1104299292,18cyclictest22353-21ssh23:45:121
11042992916,10cyclictest25670-21apt-get19:25:181
11042992914,12cyclictest0-21swapper/123:15:141
11042992914,10cyclictest0-21swapper/100:25:141
11042992912,11cyclictest11212-21apt-get19:10:131
1103299294,21cyclictest23701-21ssh21:45:150
1103299291,22cyclictest22449-21ssh23:25:180
1105399283,10cyclictest0-21swapper/223:12:112
1105399282,19cyclictest0-21swapper/220:50:142
1104299287,8cyclictest4437-21grep20:10:151
1104299282,9cyclictest0-21swapper/121:54:231
11042992821,5cyclictest14263-21apt-get00:20:151
11042992820,5cyclictest22367-21df19:55:121
11042992815,10cyclictest0-21swapper/119:45:151
11042992814,9cyclictest0-21swapper/119:50:131
11042992813,10cyclictest3015-21grep19:35:171
1103299288,4cyclictest131rcu_preempt22:00:150
11032992822,4cyclictest0-21swapper/000:25:140
1106199273,15cyclictest29566irq/58-eth000:05:143
1106199273,15cyclictest0-21swapper/322:10:153
1106199273,15cyclictest0-21swapper/300:10:133
1105399272,16cyclictest26346irq/40-dwc2_hso20:10:142
1104299278,7cyclictest27777-21sh23:26:231
1104299276,7cyclictest14394-21/usr/sbin/munin20:20:191
11042992716,7cyclictest1089-21diskstats20:40:161
11042992714,9cyclictest28604-21needreboot21:05:231
11042992714,9cyclictest28604-21needreboot21:05:231
1103299272,19cyclictest14027-21ssh21:38:420
1106199266,12cyclictest0-21swapper/321:55:133
1106199262,14cyclictest29566irq/58-eth021:45:143
11061992612,9cyclictest29566irq/58-eth023:00:153
1105399262,9cyclictest0-21swapper/223:40:122
1105399261,20cyclictest23048-21ssh23:05:142
1105399261,14cyclictest932-21snmpd20:05:152
11042992620,3cyclictest3030-21ntp_192.168.11420:05:241
11042992614,9cyclictest0-21swapper/122:30:171
11032992618,4cyclictest0-21swapper/020:00:140
1106199253,18cyclictest0-21swapper/322:40:143
1106199253,15cyclictest0-21swapper/300:35:143
1106199253,15cyclictest0-21swapper/300:35:133
11061992520,3cyclictest0-21swapper/320:20:133
11061992512,8cyclictest29566irq/58-eth022:35:043
1105399256,14cyclictest26346irq/40-dwc2_hso20:55:152
1105399254,13cyclictest26346irq/40-dwc2_hso00:20:142
1105399252,16cyclictest26346irq/40-dwc2_hso19:55:102
1105399251,19cyclictest750-21munin-run20:40:012
1105399251,17cyclictest26346irq/40-dwc2_hso19:45:152
11042992514,9cyclictest16220-21fschecks_time19:15:151
11042992513,9cyclictest9589-21interrupts20:15:201
11042992513,9cyclictest20486-21entropy21:00:151
11042992513,8cyclictest23753-21/usr/sbin/munin20:30:131
1103299252,17cyclictest14280-21ssh00:20:150
1103299252,16cyclictest6130-21ssh23:35:150
1105399242,18cyclictest22470-21ssh23:04:422
1105399242,12cyclictest23445-21ssh22:25:162
1105399241,12cyclictest23365-21sh22:45:192
1105399241,11cyclictest3019-21grep19:35:172
11032992413,3cyclictest12-21ksoftirqd/022:30:170
1105399232,6cyclictest0-21swapper/221:38:432
1105399231,13cyclictest5911-21ssh21:15:122
11042992314,6cyclictest7827-21apt-get19:40:151
11042992312,8cyclictest11070-21latency_hist19:10:011
11042992312,8cyclictest10729-21df_inode20:50:151
1103299232,18cyclictest31909-21ssh22:10:160
1103299231,18cyclictest11692-21ls21:55:290
1106199224,10cyclictest0-21swapper/322:15:133
1106199222,11cyclictest29666irq/59-eth022:20:133
1105399224,15cyclictest17812-21kthreadcore22:00:232
1105399224,11cyclictest26346irq/40-dwc2_hso20:20:132
1105399222,15cyclictest0-21swapper/222:24:422
1105399221,19cyclictest25334-21grep21:05:162
1105399221,19cyclictest25334-21grep21:05:162
11053992211,9cyclictest13774-21ssh23:19:352
1105399221,13cyclictest22957-21sh00:05:142
11053992210,5cyclictest3658-21ssh00:33:002
1103299223,16cyclictest7022-21sh22:55:140
1103299222,16cyclictest14997-21ssh23:00:150
1103299221,4cyclictest131rcu_preempt23:15:130
1103299221,18cyclictest6485-21apt-get00:15:120
1103299221,18cyclictest23255-21fschecks_time23:05:170
11032992211,8cyclictest15444-21ssh22:20:130
1105399217,11cyclictest20541-21fschecks_count21:00:172
1105399215,13cyclictest612-21dbus-daemon20:47:072
1105399211,7cyclictest10163-21kthreadcore22:55:232
11053992116,3cyclictest22305-21ssh21:25:022
1105399211,16cyclictest9390-21fschecks_count20:15:202
1105399211,15cyclictest31996-21grep22:50:172
1105399211,12cyclictest8840-21cut21:55:212
1105399211,10cyclictest30557-21cpuspeed19:30:142
1103299217,6cyclictest0-21swapper/021:32:420
1103299213,13cyclictest30629-21ssh23:50:130
1106199204,7cyclictest0-21swapper/320:45:153
1106199203,12cyclictest0-21swapper/322:32:423
1106199203,11cyclictest0-21swapper/323:15:143
1105399205,12cyclictest24427-21kthreadcore20:30:192
1105399203,14cyclictest1260-21kthreadcore22:10:222
1105399201,9cyclictest22197-21grep20:25:272
1105399201,9cyclictest15474-21kthreadcore00:00:192
1105399201,16cyclictest20990-21cut19:20:162
1105399201,15cyclictest1690-21ntp_kernel_err23:30:282
1105399201,15cyclictest15366-21tune2fs22:40:162
1105399201,14cyclictest0-21swapper/219:40:042
1105399201,11cyclictest25382-21ssh23:45:222
1105399201,11cyclictest23820-21apt-get22:05:142
1105399201,10cyclictest26498-21timerwakeupswit23:25:342
1105399201,10cyclictest1894-21cut21:50:222
1103299209,8cyclictest24012-21cut21:25:210
1103299206,11cyclictest15381-21gpgv20:55:010
1103299204,14cyclictest15317-21ssh21:40:130
1103299204,14cyclictest15317-21ssh21:40:130
1103299203,12cyclictest6468-21ssh23:55:160
1103299201,13cyclictest15205-21ssh22:40:140
1103299201,13cyclictest10411-21latency_hist20:50:010
1103299201,12cyclictest19156-21ls20:25:220
1103299200,6cyclictest318-21tr22:50:190
1106199192,11cyclictest0-21swapper/320:50:143
1106199192,10cyclictest0-21swapper/320:05:153
11061991911,3cyclictest0-21swapper/322:25:293
1106199191,11cyclictest0-21swapper/320:35:213
1106199191,11cyclictest0-21swapper/319:40:143
11061991910,6cyclictest29566irq/58-eth023:05:143
1105399191,9cyclictest25819-21fschecks_count19:25:202
1105399191,9cyclictest16004-21/usr/sbin/munin19:15:132
1105399191,10cyclictest7071-21apt-get22:35:162
1103299193,12cyclictest32036-21ssh21:50:150
1103299191,15cyclictest32556-21irqrtprio20:05:190
1103299191,12cyclictest6635-21kthreadcore21:15:170
11032991910,6cyclictest28255-21vmstat22:25:350
1103299190,16cyclictest24973-21latency_hist21:05:010
1106199189,6cyclictest0-21swapper/322:52:453
1106199189,6cyclictest0-21swapper/300:21:283
1106199188,7cyclictest0-21swapper/323:35:143
1106199182,11cyclictest0-21swapper/321:29:423
1106199182,11cyclictest0-21swapper/319:15:123
1106199182,10cyclictest29566irq/58-eth000:15:133
1106199182,10cyclictest0-21swapper/323:40:123
1106199182,10cyclictest0-21swapper/321:05:153
1106199182,10cyclictest0-21swapper/321:05:143
1106199182,10cyclictest0-21swapper/320:30:143
1106199182,10cyclictest0-21swapper/320:15:043
1106199182,10cyclictest0-21swapper/319:55:123
1106199182,10cyclictest0-21swapper/319:50:153
1106199182,10cyclictest0-21swapper/319:30:143
1106199181,13cyclictest0-21swapper/323:45:143
1106199181,12cyclictest0-21swapper/320:25:193
1106199181,11cyclictest0-21swapper/323:35:013
11061991810,6cyclictest0-21swapper/300:01:003
1105399182,13cyclictest2382-21kthreadcore20:40:202
1105399181,9cyclictest11079-21latency_hist19:10:012
1103299187,3cyclictest131rcu_preempt22:40:010
1103299185,10cyclictest16129-21fschecks_count19:15:130
1103299184,5cyclictest20992-21processes19:50:270
1103299182,13cyclictest0-21swapper/019:35:150
11032991812,4cyclictest0-21swapper/020:15:020
1103299181,15cyclictest15075-21kthreadcore19:45:240
1103299181,13cyclictest25530-21sed21:05:180
1103299181,13cyclictest25530-21sed21:05:180
1103299181,12cyclictest17544-21memory23:40:210
11032991810,6cyclictest135-21systemd-journal19:25:010
1106199179,6cyclictest0-21swapper/322:00:073
1106199172,9cyclictest0-21swapper/321:00:143
1106199172,10cyclictest0-21swapper/319:45:173
1106199171,13cyclictest0-21swapper/323:25:253
1106199171,13cyclictest0-21swapper/319:10:283
1106199171,12cyclictest33-21ksoftirqd/320:10:133
1103299179,5cyclictest11864-21usage_EthcTask22:15:310
1103299178,6cyclictest15490-21kthreadcore20:20:200
1103299175,9cyclictest11699-21grep19:40:320
1103299175,8cyclictest0-21swapper/020:40:130
1103299171,13cyclictest9598-21ps20:15:210
1103299171,13cyclictest23700-21sed00:05:200
1103299171,13cyclictest17655-21tr00:00:230
1106199162,9cyclictest0-21swapper/320:55:043
1106199161,12cyclictest0-21swapper/320:40:153
1103299163,10cyclictest23736-21/usr/sbin/munin20:30:170
1103299162,9cyclictest0-21swapper/020:55:150
1103299162,10cyclictest22273-21acpi19:55:100
1103299161,12cyclictest23806-21cut23:45:200
1103299161,12cyclictest15056-21sed19:10:290
1103299151,6cyclictest30830-21ps19:30:180
1103299151,10cyclictest11159-21cat19:10:010
1106199141,6cyclictest0-21swapper/319:20:203
1106199141,10cyclictest0-21swapper/319:35:163
1106199131,6cyclictest0-21swapper/320:00:293
1106199123,3cyclictest0-21swapper/319:25:173
1106199111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional