You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-18 - 17:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Apr 18, 2025 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
246799842,14cyclictest21-21ksoftirqd/110:35:121
247299802,51cyclictest10857-21ssh11:30:102
247299711,7cyclictest0-21swapper/208:45:162
246799696,19cyclictest21-21ksoftirqd/110:30:081
246799625,12cyclictest21-21ksoftirqd/108:45:151
245999615,7cyclictest131rcu_preempt09:00:160
247299583,3cyclictest0-21swapper/207:25:162
247299571,36cyclictest8454-21ssh12:30:102
2472995422,12cyclictest29112-21sendmail-msp09:40:142
2479995330,19cyclictest0-21swapper/310:55:143
245999531,27cyclictest4701-21sendmail-msp08:20:160
2479995138,10cyclictest25866irq/58-eth010:15:143
2479995044,3cyclictest25866irq/58-eth009:40:143
2472995026,16cyclictest31072-21sendmail-msp07:40:142
245999502,4cyclictest131rcu_preempt07:45:130
247299471,5cyclictest0-21swapper/209:30:102
245999475,36cyclictest27087-21sendmail-msp12:20:150
245999441,35cyclictest22747-21sendmail-msp10:00:170
245999441,31cyclictest11896-21sendmail-msp07:20:140
247299434,16cyclictest632-21Async17:44:202
247999412,24cyclictest25866irq/58-eth009:50:093
2472994114,17cyclictest632-21Async17:44:202
247299402,31cyclictest28743-21ssh09:39:292
247299392,29cyclictest3175-21ssh12:24:292
247299391,27cyclictest0-21swapper/207:10:142
245999393,25cyclictest9620-21sendmail-msp10:40:160
2459993928,2cyclictest131rcu_preempt10:35:120
2459993916,17cyclictest12-21ksoftirqd/008:40:140
247299382,28cyclictest27668-21sendmail-msp12:01:292
247299372,28cyclictest10836-21processes07:15:282
2472993710,14cyclictest17956-21sendmail-msp08:00:142
2467993725,8cyclictest201rcuc/109:00:161
246799372,26cyclictest30345-21sendmail-msp11:20:151
247299365,21cyclictest632-21Async17:44:202
247299362,21cyclictest22146irq/40-dwc2_hso11:10:092
247299361,9cyclictest31069-21kthreadcore11:40:292
247299361,9cyclictest1369-21apt-get08:50:172
247299361,10cyclictest15677-21sed09:55:212
2467993614,15cyclictest27668-21sendmail-msp12:00:161
245999365,24cyclictest2733-21sendmail-msp09:20:170
2459993618,12cyclictest28636-21sendmail-msp11:40:190
247299352,26cyclictest25466-21ssh12:39:292
247299351,9cyclictest4275-21kthreadcore09:20:252
246799351,25cyclictest16481-21sendmail-msp10:20:151
246799351,22cyclictest20339-21diskmemload09:10:161
246799351,22cyclictest20339-21diskmemload09:10:161
247999341,17cyclictest25866irq/58-eth011:35:123
247299343,7cyclictest9775-21sh12:10:152
247299341,9cyclictest23253-21rm10:00:182
247299341,4cyclictest1817-21memory11:20:292
245999343,25cyclictest30345-21sendmail-msp11:20:160
247299332,6cyclictest27857-21ssh09:15:132
247299332,6cyclictest27857-21ssh09:15:132
247299331,9cyclictest25159-21ssh09:10:342
247299331,9cyclictest25159-21ssh09:10:342
247299331,8cyclictest24775-21kthreadcore08:40:232
247299331,8cyclictest1405-21kthreadcore08:15:212
247299331,8cyclictest10714-21tr09:00:012
247999321,16cyclictest25866irq/58-eth008:10:173
247299322,22cyclictest16920-21sh10:20:172
247299321,9cyclictest26650-21awk08:05:292
247299321,9cyclictest25340-21sh11:57:172
247299321,9cyclictest23070-21kthreadcore11:35:232
247299321,9cyclictest2241-21irqstats10:10:152
247299321,9cyclictest10800-21kthreadcore09:50:192
247299321,8cyclictest7035-21kthreadcore11:00:302
247299321,28cyclictest0-21swapper/210:07:182
247299321,27cyclictest3806-21kthreadcore10:35:192
246799322,23cyclictest2305-21nfs_client07:40:291
245999321,28cyclictest9131-21ssh12:10:100
247299312,5cyclictest0-21swapper/209:05:142
247299311,8cyclictest30390-21latency_hist11:20:032
247299311,7cyclictest24362-21uname10:50:342
247299311,6cyclictest19056-21kthreadcore12:15:202
247299310,10cyclictest19315-21acpi08:35:122
246799312,22cyclictest629-21tcf-agent07:25:291
246799310,3cyclictest601-21rngd07:30:171
246799310,3cyclictest601-21rngd07:30:171
245999312,21cyclictest933-21ssh10:33:290
247299304,22cyclictest8234-21ssh11:27:162
247299302,9cyclictest13981-21kthreadcore07:20:232
247299302,23cyclictest8870-21ls12:10:022
247299301,9cyclictest14061-21kthreadcore09:00:272
247299301,7cyclictest8508-21mailstats08:20:342
247299301,7cyclictest27014-21apt-get10:30:082
247299301,7cyclictest10335-21irqcore10:40:222
247299301,6cyclictest14599-21cpuspeed08:30:192
247299301,4cyclictest0-21swapper/207:45:132
247299300,5cyclictest13626-21grep08:25:302
245999302,15cyclictest17956-21sendmail-msp08:00:290
247999295,13cyclictest25866irq/58-eth012:30:083
247999294,16cyclictest0-21swapper/307:15:283
247299296,2cyclictest0-21swapper/211:47:352
247299293,20cyclictest0-21swapper/210:49:292
247299291,8cyclictest27958-21fschecks_time08:10:202
247299291,7cyclictest12918-21ssh11:08:222
247299291,6cyclictest29170-21kthreadcore07:35:242
247299291,11cyclictest26285-21latency_hist07:35:012
247299291,11cyclictest26285-21latency_hist07:35:012
247299290,5cyclictest9823-21ssh11:50:082
246799292,20cyclictest23884-21sendmail-msp08:40:281
245999294,14cyclictest12-21ksoftirqd/011:10:090
245999292,19cyclictest2008-21sh09:41:290
245999292,19cyclictest18960-21sh10:47:290
245999292,19cyclictest1106-21ssh10:56:290
247999283,16cyclictest0-21swapper/310:30:303
247299281,16cyclictest28246-21ssh10:55:132
247299280,7cyclictest3955-21ssh12:25:032
247299280,7cyclictest12040-21ssh09:27:222
246799287,13cyclictest21-21ksoftirqd/108:10:171
245999282,19cyclictest25180-21ntp_states07:30:290
245999282,19cyclictest25180-21ntp_states07:30:290
245999282,17cyclictest522-21usb-storage11:00:150
2479992711,10cyclictest25866irq/58-eth007:25:293
247299271,3cyclictest5133-21ssh09:46:052
246799272,19cyclictest17956-21sendmail-msp08:01:291
246799272,17cyclictest601-21rngd09:41:291
246799272,16cyclictest0-21swapper/108:05:141
246799272,15cyclictest3652-21ssh11:00:161
246799271,20cyclictest27087-21sendmail-msp12:20:301
246799271,18cyclictest3103-21ssh09:20:171
245999279,13cyclictest27668-21sendmail-msp12:00:180
2479992618,5cyclictest0-21swapper/312:00:263
2479992612,8cyclictest25866irq/58-eth012:35:143
247299261,6cyclictest0-21swapper/210:28:402
2467992611,4cyclictest21-21ksoftirqd/109:30:101
245999261,21cyclictest7616-21ssh09:50:110
247999254,13cyclictest0-21swapper/309:20:293
2479992510,10cyclictest0-21swapper/311:34:303
247299250,7cyclictest2489-21latency_hist07:10:022
246799252,19cyclictest22747-21sendmail-msp10:00:321
246799252,18cyclictest11896-21sendmail-msp07:20:331
246799252,16cyclictest601-21rngd10:56:291
2467992513,3cyclictest21-21ksoftirqd/112:10:101
245999252,7cyclictest131rcu_preempt11:55:290
2459992511,5cyclictest131rcu_preempt09:15:140
2459992511,5cyclictest131rcu_preempt09:15:130
246799242,8cyclictest52246usb-storage10:40:261
246799242,14cyclictest601-21rngd10:47:291
246799241,19cyclictest28636-21sendmail-msp11:40:341
2467992411,4cyclictest21-21ksoftirqd/111:56:371
245999241,5cyclictest131rcu_preempt11:50:090
245999241,11cyclictest26841-21ls10:30:010
247999233,13cyclictest0-21swapper/307:40:293
245999238,9cyclictest12-21ksoftirqd/011:35:130
245999232,16cyclictest22041-21ssh12:15:280
247999223,14cyclictest0-21swapper/310:35:123
247999222,8cyclictest0-21swapper/311:15:133
247999222,12cyclictest0-21swapper/309:30:093
2479992212,8cyclictest0-21swapper/310:20:373
2467992211,2cyclictest21-21ksoftirqd/108:55:301
245999221,14cyclictest26586-21switchtime08:05:280
247999219,9cyclictest0-21swapper/311:59:313
2479992115,4cyclictest0-21swapper/312:20:263
2479992112,6cyclictest0-21swapper/310:01:533
246799219,3cyclictest21-21ksoftirqd/107:35:181
246799216,9cyclictest21-21ksoftirqd/112:19:291
246799216,8cyclictest0-21swapper/112:05:081
246799216,4cyclictest21-21ksoftirqd/109:59:041
246799212,13cyclictest601-21rngd07:15:281
246799211,4cyclictest131rcu_preempt08:35:141
245999212,14cyclictest0-21swapper/012:06:290
247999208,5cyclictest0-21swapper/312:10:193
247999203,11cyclictest0-21swapper/312:18:293
247999203,11cyclictest0-21swapper/311:05:293
247999203,10cyclictest0-21swapper/310:47:293
246799209,2cyclictest21-21ksoftirqd/109:19:581
246799209,2cyclictest21-21ksoftirqd/109:19:581
246799206,5cyclictest131rcu_preempt11:30:011
246799201,3cyclictest131rcu_preempt09:35:151
246799201,16cyclictest9819-21apt-get11:50:081
246799200,5cyclictest5920-21latency_hist08:55:011
245999209,8cyclictest31072-21sendmail-msp07:40:170
245999206,3cyclictest131rcu_preempt10:10:140
245999203,11cyclictest601-21rngd09:05:140
245999201,5cyclictest601-21rngd11:15:170
2459992011,7cyclictest16481-21sendmail-msp10:21:010
245999201,16cyclictest31008-21ls08:10:260
247999193,11cyclictest0-21swapper/309:16:293
247999193,11cyclictest0-21swapper/309:16:293
247999193,10cyclictest0-21swapper/308:01:293
247999192,11cyclictest0-21swapper/309:01:293
247999192,11cyclictest0-21swapper/308:40:283
2479991911,6cyclictest0-21swapper/312:05:253
2479991911,6cyclictest0-21swapper/311:21:033
2479991911,6cyclictest0-21swapper/310:25:163
247999191,11cyclictest0-21swapper/308:40:023
247999191,11cyclictest0-21swapper/307:30:263
247999191,11cyclictest0-21swapper/307:30:263
2479991910,7cyclictest0-21swapper/311:49:563
2479991910,6cyclictest0-21swapper/311:41:503
2479991910,6cyclictest0-21swapper/309:35:163
2479991910,3cyclictest0-21swapper/311:01:203
246799197,2cyclictest21-21ksoftirqd/111:45:481
246799193,2cyclictest131rcu_preempt09:51:351
246799192,11cyclictest601-21rngd10:51:291
2467991913,4cyclictest21-21ksoftirqd/110:15:271
2467991913,3cyclictest21-21ksoftirqd/111:18:551
2467991912,4cyclictest21-21ksoftirqd/109:46:541
246799191,10cyclictest3785-21/usr/sbin/munin07:45:151
246799190,10cyclictest601-21rngd07:50:221
245999198,8cyclictest7448-21df_inode07:15:170
245999191,5cyclictest601-21rngd08:55:160
245999191,3cyclictest131rcu_preempt09:50:010
245999191,14cyclictest4821-21sshd08:50:290
245999191,10cyclictest9905-21fschecks_count08:25:170
2459991910,6cyclictest13867-21apt-get09:30:100
245999190,6cyclictest26105-21ssh12:40:020
247999189,6cyclictest0-21swapper/311:13:533
247999189,6cyclictest0-21swapper/309:26:113
247999188,8cyclictest321rcuc/309:10:173
247999188,8cyclictest321rcuc/309:10:173
247999183,10cyclictest0-21swapper/311:28:293
247999183,10cyclictest0-21swapper/310:50:293
247999183,10cyclictest0-21swapper/307:10:293
247999182,11cyclictest0-21swapper/309:46:293
247999182,10cyclictest0-21swapper/310:43:283
247999182,10cyclictest0-21swapper/308:45:053
2479991812,4cyclictest0-21swapper/312:30:023
2479991811,5cyclictest0-21swapper/308:20:243
2479991810,6cyclictest0-21swapper/311:50:143
2479991810,6cyclictest0-21swapper/310:10:123
2479991810,6cyclictest0-21swapper/310:09:343
246799189,6cyclictest15974-21cat11:10:171
246799187,3cyclictest21-21ksoftirqd/108:15:151
246799182,11cyclictest601-21rngd11:05:291
246799182,11cyclictest3649-21apt-get12:25:041
2467991813,3cyclictest21-21ksoftirqd/110:06:191
2467991812,3cyclictest21-21ksoftirqd/112:34:271
2467991812,3cyclictest21-21ksoftirqd/111:36:161
2467991811,4cyclictest21-21ksoftirqd/110:26:051
245999182,4cyclictest12-21ksoftirqd/012:25:130
245999182,13cyclictest24703-21tr09:35:250
247999172,9cyclictest0-21swapper/308:15:153
247999172,10cyclictest0-21swapper/309:55:173
247999172,10cyclictest0-21swapper/308:50:173
247999171,12cyclictest0-21swapper/308:30:193
246799179,5cyclictest13678-21/usr/sbin/munin07:55:181
246799179,3cyclictest21-21ksoftirqd/108:30:171
246799178,6cyclictest5104-21fschecks_time08:20:201
246799178,2cyclictest21-21ksoftirqd/110:13:361
246799177,3cyclictest21-21ksoftirqd/112:35:101
2467991712,2cyclictest21-21ksoftirqd/111:31:301
246799171,13cyclictest6388-21sed07:10:291
246799171,11cyclictest13570-21unixbench_multi08:25:301
246799170,3cyclictest601-21rngd09:27:171
245999171,8cyclictest4701-21sendmail-msp08:20:010
245999171,8cyclictest16926-21/usr/sbin/munin07:25:330
245999171,7cyclictest28717-21kthreadcore07:35:230
245999171,3cyclictest131rcu_preempt08:30:300
245999171,13cyclictest14494-21ntp_192.168.11411:30:220
245999171,12cyclictest10524-21latency_hist11:30:020
2459991710,5cyclictest0-21swapper/010:00:010
247999161,10cyclictest0-21swapper/307:35:183
246799169,5cyclictest20223-21cat09:05:311
245999168,6cyclictest1249-21cron10:10:010
245999166,7cyclictest8588-21apt-get07:50:180
245999165,9cyclictest8855-21irqcore12:30:150
245999162,10cyclictest601-21rngd09:28:290
245999162,10cyclictest601-21rngd09:10:290
245999162,10cyclictest601-21rngd09:10:290
245999161,12cyclictest8847-21apt-get10:15:130
245999161,12cyclictest13986-21apt11:10:020
245999161,11cyclictest9499-21munin-plugin-st11:50:010
247999157,5cyclictest25866irq/58-eth008:25:203
247999151,7cyclictest0-21swapper/308:10:013
245999159,4cyclictest20253-21kthreadcore08:35:200
245999152,10cyclictest601-21rngd07:10:120
245999151,12cyclictest22835-21kthreadcore10:50:160
2459991510,3cyclictest28960-21apt-config08:45:020
245999150,4cyclictest601-21rngd07:55:230
247999149,3cyclictest0-21swapper/307:55:193
247999142,8cyclictest0-21swapper/308:55:133
247999141,10cyclictest0-21swapper/309:05:223
247999141,10cyclictest0-21swapper/307:50:223
247999141,10cyclictest0-21swapper/307:20:373
246799141,10cyclictest2526-21wc07:10:021
247999111,6cyclictest0-21swapper/307:45:193
247999101,6cyclictest0-21swapper/307:10:023
245999101,7cyclictest601-21rngd07:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional