You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-25 - 08:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Apr 25, 2025 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21582996311,13cyclictest131rcu_preempt21:50:101
2158299586,9cyclictest131rcu_preempt19:10:131
21591995721,21cyclictest22146irq/40-dwc2_hso22:00:132
21582995729,18cyclictest16952-21sendmail-msp00:20:151
2160399524,28cyclictest25866irq/58-eth023:15:143
2157199511,42cyclictest20931-21sendmail-msp00:00:140
2159199491,30cyclictest15986-21sh21:40:132
2158299492,40cyclictest7808-21sendmail-msp22:20:131
21582994815,22cyclictest21-21ksoftirqd/123:55:131
2158299481,42cyclictest16856-21ssh23:35:131
2157199481,38cyclictest3899-21sendmail-msp22:40:140
21591994714,26cyclictest27-21ksoftirqd/223:55:152
2159199471,29cyclictest20931-21sendmail-msp00:00:152
2158299450,35cyclictest8980-21ssh00:15:121
2157199452,34cyclictest28466-21sendmail-msp23:20:120
2157199423,22cyclictest29103-21ssh00:30:100
2157199421,32cyclictest32556-21sendmail-msp20:00:150
2158299412,28cyclictest19699-21rm21:20:191
2157199413,31cyclictest27758-21sh21:50:070
2157199411,29cyclictest24880-21sendmail-msp23:40:150
2158299409,6cyclictest131rcu_preempt22:30:091
2158299403,4cyclictest131rcu_preempt22:35:111
2160399391,24cyclictest25866irq/58-eth022:30:083
2158299391,31cyclictest19364-21sendmail-msp20:20:171
2157199392,19cyclictest4913-21ssh00:35:130
2157199383,24cyclictest19301-21sendmail-msp21:20:200
21582993714,8cyclictest21-21ksoftirqd/121:47:391
2157199353,20cyclictest7589-21ssh21:35:120
2157199351,26cyclictest20014-21ssh22:30:100
2159199342,26cyclictest6971-21ssh21:34:392
2159199342,25cyclictest0-21swapper/221:50:092
2157199333,22cyclictest16233-21ssh22:50:100
2157199333,22cyclictest16233-21ssh22:50:100
2157199331,27cyclictest6180-21sendmail-msp20:40:130
2160399315,16cyclictest0-21swapper/321:55:143
2159199314,19cyclictest25389-21sendmail-msp21:00:182
2157199312,23cyclictest31034-21sendmail-msp19:20:180
2160399304,16cyclictest0-21swapper/322:15:143
2159199304,18cyclictest22146irq/40-dwc2_hso22:15:152
2158299302,20cyclictest19408-21ssh22:29:381
2159199292,21cyclictest3715-21ssh21:55:172
2159199292,19cyclictest22146irq/40-dwc2_hso19:10:142
2160399281,23cyclictest25866irq/58-eth023:35:153
2158299281,16cyclictest601-21rngd22:15:131
21571992811,8cyclictest12-21ksoftirqd/019:35:130
2160399274,11cyclictest25866irq/58-eth021:20:203
2160399272,19cyclictest0-21swapper/300:00:393
2159199279,10cyclictest22146irq/40-dwc2_hso00:20:132
2159199278,8cyclictest16952-21sendmail-msp00:20:022
2159199272,13cyclictest710-21rs:main2
2158299275,17cyclictest522-21usb-storage21:00:191
2157199272,19cyclictest25389-21sendmail-msp21:00:390
2159199263,7cyclictest0-21swapper/221:47:402
2159199262,9cyclictest0-21swapper/220:50:182
2157199262,20cyclictest20465-21ssh23:15:150
2157199262,17cyclictest601-21rngd23:59:390
2157199261,19cyclictest31642-21ssh21:30:100
2157199261,15cyclictest12-21ksoftirqd/019:40:170
2159199253,18cyclictest28466-21sendmail-msp23:20:262
2159199253,18cyclictest19416-21wc20:20:022
21591992513,9cyclictest12037-21ssh23:54:152
2159199251,13cyclictest9120-21cut22:20:212
21591992511,11cyclictest18198-21fschecks_count19:40:202
2158299255,4cyclictest131rcu_preempt21:15:021
2158299252,14cyclictest15987-21ssh21:40:121
2158299251,18cyclictest11649-21sendmail-msp22:00:131
2160399246,11cyclictest0-21swapper/321:40:113
2159199248,5cyclictest7330-21kthreadcore20:40:322
2159199242,6cyclictest0-21swapper/200:25:402
2159199241,8cyclictest2599-21diskmemload22:25:402
2159199241,18cyclictest14555-21ssh21:38:382
2159199241,17cyclictest3899-21sendmail-msp22:40:122
2159199241,15cyclictest28567-21uniq22:35:152
21591992411,5cyclictest22962-21cron19:50:022
2158299241,17cyclictest32323-21sendmail-msp23:00:161
2158299241,15cyclictest32556-21sendmail-msp20:00:341
2158299241,15cyclictest21318-21ssh00:00:171
2157199242,19cyclictest7808-21sendmail-msp22:21:390
2157199241,18cyclictest30088-21ssh22:12:380
2160399233,13cyclictest0-21swapper/321:34:393
2160399231,14cyclictest0-21swapper/323:23:393
2159199238,12cyclictest23505-21kthreadcore20:55:222
2159199237,13cyclictest4080-21proc_pri23:00:372
2159199235,14cyclictest13785-21ps19:35:192
2159199232,6cyclictest0-21swapper/222:09:392
2159199232,6cyclictest0-21swapper/221:10:392
21591992314,7cyclictest11387-21sh21:15:152
21591992314,6cyclictest23802-21ssh22:10:102
21591992313,8cyclictest0-21swapper/219:20:372
21591992312,8cyclictest32520-21cron20:00:012
21591992310,5cyclictest22146irq/40-dwc2_hso23:40:172
2158299238,9cyclictest601-21rngd21:59:391
2158299231,19cyclictest2770-21ssh21:30:191
2157199231,17cyclictest3038-21rm21:10:170
21603992212,7cyclictest25866irq/58-eth021:50:093
2159199229,6cyclictest8817-21cat20:05:282
2159199229,6cyclictest12603-21ssh22:45:162
2159199229,10cyclictest3592-21ssh23:49:022
2159199229,10cyclictest17510-21ssh23:11:242
2159199228,6cyclictest19364-21sendmail-msp20:20:332
2159199227,11cyclictest0-21swapper/222:30:112
2159199223,15cyclictest24984-21irqrtprio22:55:162
2159199222,15cyclictest0-21swapper/200:05:382
2159199221,9cyclictest29061-21/usr/sbin/munin00:30:072
21591992210,10cyclictest2595-21cron21:10:012
2159199220,11cyclictest4910-21ls20:35:242
2158299227,11cyclictest28466-21sendmail-msp23:20:211
2158299221,7cyclictest21-21ksoftirqd/119:20:231
2158299221,15cyclictest6180-21sendmail-msp20:40:541
2157199229,10cyclictest11649-21sendmail-msp22:00:460
21571992211,9cyclictest32323-21sendmail-msp23:00:250
2157199221,15cyclictest28390-21rm22:35:120
21571992210,3cyclictest12-21ksoftirqd/021:40:530
21603992113,5cyclictest0-21swapper/321:15:143
2159199219,5cyclictest23269-21cut23:15:232
2159199218,10cyclictest12996-21http19:35:012
2159199217,7cyclictest27-21ksoftirqd/219:55:022
2159199217,6cyclictest22853-21ssh22:53:122
2159199217,6cyclictest22853-21ssh22:53:112
2159199212,12cyclictest628-21ssh00:10:092
2159199211,9cyclictest6445-21cut00:35:232
2159199211,9cyclictest27325-21kthreadcore19:15:212
2159199211,8cyclictest11431-21tune2fs20:45:202
2159199211,17cyclictest10101-21awk20:10:202
2158299219,9cyclictest9761-21cron20:10:001
2158299219,8cyclictest24880-21sendmail-msp23:40:251
2158299211,3cyclictest131rcu_preempt20:45:161
2157199212,14cyclictest16952-21sendmail-msp00:20:380
2160399203,11cyclictest0-21swapper/321:38:393
2160399203,11cyclictest0-21swapper/321:04:383
2160399203,11cyclictest0-21swapper/320:49:383
2160399202,11cyclictest0-21swapper/321:47:403
2160399202,10cyclictest0-21swapper/322:25:393
2160399202,10cyclictest0-21swapper/320:50:183
2159199201,9cyclictest1376-21munin-run20:35:012
2159199201,7cyclictest836766ssh23:05:032
2159199201,16cyclictest29019-21munin-plugin-st20:30:012
2159199201,15cyclictest18035-21kthreadcore23:35:202
2159199201,15cyclictest0-21swapper/223:30:102
2158299201,4cyclictest131rcu_preempt23:15:181
2158299200,9cyclictest601-21rngd00:28:301
2157199209,8cyclictest12905-21tr21:15:210
2157199202,12cyclictest6807-21ssh22:18:390
2157199201,15cyclictest24429-21ssh22:55:100
2160399193,11cyclictest0-21swapper/322:46:393
2160399193,11cyclictest0-21swapper/322:02:393
2160399193,11cyclictest0-21swapper/300:27:393
2160399193,11cyclictest0-21swapper/300:20:163
2160399193,11cyclictest0-21swapper/300:07:403
2160399192,12cyclictest0-21swapper/321:10:183
2160399192,11cyclictest0-21swapper/320:55:403
2160399192,10cyclictest0-21swapper/320:15:153
21603991913,4cyclictest33-21ksoftirqd/322:55:293
2160399191,13cyclictest0-21swapper/323:03:383
21603991911,3cyclictest0-21swapper/321:29:123
2160399191,13cyclictest0-21swapper/320:05:023
21603991910,6cyclictest25866irq/58-eth000:15:123
21603991910,6cyclictest0-21swapper/322:21:073
2159199199,3cyclictest0-21swapper/221:25:272
2159199191,9cyclictest845-21munin-node20:00:132
2158299199,7cyclictest3899-21sendmail-msp22:41:151
2158299198,5cyclictest601-21rngd20:50:161
2158299191,4cyclictest131rcu_preempt23:25:511
2158299191,4cyclictest131rcu_preempt21:15:071
2158299191,3cyclictest131rcu_preempt23:14:291
2158299191,3cyclictest131rcu_preempt21:40:021
2158299191,14cyclictest4492-21ls00:10:251
2157199193,10cyclictest12746-21ls23:55:010
2157199192,11cyclictest18717-21ssh22:28:390
2157199191,14cyclictest601-21rngd23:35:130
2157199191,13cyclictest15789-21latency_hist20:50:010
21571991910,6cyclictest0-21swapper/020:15:190
2160399189,6cyclictest25866irq/58-eth022:10:093
2160399189,6cyclictest0-21swapper/323:10:183
2160399189,6cyclictest0-21swapper/322:40:553
2160399189,6cyclictest0-21swapper/300:36:373
2160399182,13cyclictest0-21swapper/300:30:093
2160399182,11cyclictest0-21swapper/323:08:393
2160399182,11cyclictest0-21swapper/322:06:403
21603991812,4cyclictest25966irq/59-eth022:35:293
21603991811,5cyclictest0-21swapper/323:31:013
2160399181,12cyclictest0-21swapper/323:55:153
2160399181,10cyclictest0-21swapper/319:20:163
21603991810,6cyclictest0-21swapper/323:26:063
21603991810,5cyclictest33-21ksoftirqd/319:40:223
21603991810,5cyclictest0-21swapper/322:50:173
21603991810,5cyclictest0-21swapper/322:50:173
2159199181,9cyclictest3743-21diskstats19:25:172
2159199181,6cyclictest540166ssh23:26:122
2159199180,9cyclictest21557-21cyclictest19:10:022
2158299189,6cyclictest21608-21ntp_offset19:40:341
2158299189,6cyclictest201rcuc/120:15:131
2158299187,8cyclictest32754-21cron00:10:021
2158299181,4cyclictest131rcu_preempt23:06:391
2158299181,3cyclictest131rcu_preempt00:40:021
21582991812,4cyclictest0-21swapper/122:11:231
21582991811,5cyclictest0-21swapper/120:25:211
21582991811,4cyclictest32402-21rm23:00:021
2158299181,11cyclictest601-21rngd20:55:141
2158299181,11cyclictest28940-21ssh21:26:391
2157199189,7cyclictest21732-21apt-get19:10:140
2157199182,13cyclictest7807-21unixbench_singl21:55:330
2157199182,11cyclictest0-21swapper/020:35:150
2157199181,9cyclictest7170-21find19:25:290
2157199181,4cyclictest52246usb-storage20:20:250
2157199181,13cyclictest8404-21/usr/sbin/munin23:30:260
2160399178,6cyclictest0-21swapper/323:51:273
2160399173,10cyclictest0-21swapper/323:47:393
2160399172,10cyclictest0-21swapper/321:05:393
21603991712,3cyclictest33-21ksoftirqd/320:20:343
21603991711,4cyclictest0-21swapper/300:10:503
21603991710,4cyclictest0-21swapper/323:41:213
2158299178,6cyclictest10152-21fschecks_time20:10:211
2158299177,7cyclictest21-21ksoftirqd/123:50:021
2158299176,8cyclictest3650-21apt-get19:25:131
2158299175,9cyclictest502-21tr21:05:221
2158299171,7cyclictest30833-21kthreadcore00:30:171
21582991711,4cyclictest0-21swapper/123:31:491
21582991711,4cyclictest0-21swapper/119:30:201
2158299171,13cyclictest601-21rngd22:50:021
2158299171,13cyclictest4227-21apt-get23:50:051
2158299171,12cyclictest23194-21apt-get19:50:081
2158299171,10cyclictest0-21swapper/119:49:391
2157199179,6cyclictest28789-21basename00:30:020
2157199173,11cyclictest8922-21cut19:30:210
2157199172,11cyclictest601-21rngd23:09:390
2157199172,11cyclictest601-21rngd21:46:400
2157199172,11cyclictest601-21rngd21:26:380
2157199172,11cyclictest18361-21ssh23:12:380
2157199172,10cyclictest0-21swapper/020:05:150
2157199171,13cyclictest25129-21kthreadcore20:25:210
2157199171,13cyclictest11863-21sed20:10:250
2157199171,10cyclictest601-21rngd21:05:160
2157199171,10cyclictest601-21rngd20:30:200
2160399164,7cyclictest0-21swapper/319:10:143
2158299165,9cyclictest13284-21sort19:35:121
2158299162,10cyclictest601-21rngd22:09:401
2158299161,3cyclictest131rcu_preempt20:35:241
2158299161,12cyclictest32545-21sendmail20:00:021
2158299161,12cyclictest29849-21ntp_192.168.11419:15:271
2158299161,11cyclictest1372-21munin-plugin-st20:35:011
2158299161,10cyclictest601-21rngd22:50:131
2158299161,10cyclictest601-21rngd22:50:121
2157199165,7cyclictest12-21ksoftirqd/000:10:090
2157199164,9cyclictest23690-21grep19:50:140
2157199162,10cyclictest601-21rngd22:05:390
2157199162,10cyclictest601-21rngd00:09:390
2157199161,9cyclictest601-21rngd00:15:230
2157199161,13cyclictest24518-21timerandwakeup20:55:290
2157199161,12cyclictest19414-21ntp_192.168.11420:50:280
21571991610,4cyclictest0-21swapper/019:15:180
2157199160,11cyclictest601-21rngd19:55:140
2160399151,9cyclictest0-21swapper/320:25:173
2160399151,12cyclictest0-21swapper/320:35:013
21603991510,3cyclictest0-21swapper/319:50:243
2157199151,11cyclictest4226-21idleruntime-cro23:50:010
2157199151,11cyclictest15899-21cron22:50:000
2160399149,3cyclictest33-21ksoftirqd/320:40:443
2160399149,3cyclictest0-21swapper/319:30:333
2160399141,7cyclictest0-21swapper/320:10:323
2160399141,10cyclictest0-21swapper/320:35:193
2160399141,10cyclictest0-21swapper/319:15:163
2160399141,10cyclictest0-21swapper/319:10:013
2157199143,3cyclictest131rcu_preempt23:25:310
2157199141,10cyclictest21648-21latency_hist19:10:020
2157199140,10cyclictest22964-21apt19:50:020
2160399139,2cyclictest33-21ksoftirqd/319:25:173
2160399138,3cyclictest33-21ksoftirqd/319:55:183
2160399138,3cyclictest0-21swapper/319:49:383
2160399131,10cyclictest0-21swapper/319:35:173
2160399111,7cyclictest0-21swapper/320:05:183
2158299101,6cyclictest601-21rngd19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional