You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-19 - 20:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Wed Mar 19, 2025 12:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23979996619,7cyclictest131rcu_preempt07:50:160
2398199654,29cyclictest30539-21ssh09:30:082
2397999602,15cyclictest12-21ksoftirqd/008:05:130
23980995729,20cyclictest1178-21sendmail-msp07:20:121
2398099542,44cyclictest7425-21sendmail-msp08:00:151
2397999543,7cyclictest131rcu_preempt08:50:150
23981994712,28cyclictest22146irq/40-dwc2_hso11:40:152
23981994631,9cyclictest23318-21ssh09:50:102
23979994615,18cyclictest12-21ksoftirqd/009:40:160
2398099454,10cyclictest131rcu_preempt09:30:101
2398099452,32cyclictest5474-21sendmail-msp10:00:121
2397999454,11cyclictest131rcu_preempt08:10:170
2397999441,36cyclictest24913-21sendmail-msp10:40:110
2397999431,7cyclictest131rcu_preempt07:25:130
2397999431,7cyclictest131rcu_preempt07:25:120
2398299423,24cyclictest25866irq/58-eth010:50:083
2397999421,30cyclictest16444-21ssh10:10:090
2398099411,30cyclictest30502-21ssh12:15:131
2398099411,30cyclictest30502-21ssh12:15:131
2398199406,10cyclictest21472-21ssh12:10:112
2398199402,29cyclictest30865-21ssh09:55:112
2398199402,29cyclictest30865-21ssh09:55:102
2398199401,17cyclictest31670-21ssh11:55:132
23981993926,8cyclictest27-21ksoftirqd/212:30:112
2397999392,27cyclictest31613-21sendmail-msp10:20:120
2397999372,26cyclictest30522-21ssh12:15:140
2397999372,26cyclictest30522-21ssh12:15:130
2398099361,28cyclictest13694-21sendmail-msp08:40:161
2397999364,26cyclictest20485-21sendmail-msp07:40:130
23979993612,16cyclictest11703-21sendmail-msp11:20:190
2398099359,22cyclictest11703-21sendmail-msp11:20:161
2398299343,18cyclictest25866irq/58-eth012:15:133
2398299343,18cyclictest25866irq/58-eth012:15:133
2397999346,17cyclictest12-21ksoftirqd/008:30:160
2398299339,19cyclictest0-21swapper/311:55:123
2398299334,24cyclictest25866irq/58-eth012:30:143
2397999335,16cyclictest12-21ksoftirqd/011:00:130
2397999332,22cyclictest5979-21ssh09:10:100
2398199323,15cyclictest0-21swapper/208:00:152
2397999324,25cyclictest26775-21sendmail-msp08:20:150
2397999323,21cyclictest29369-21ssh12:35:110
2398199314,9cyclictest23458-21ssh11:30:092
2397999311,20cyclictest24175-21ssh10:15:140
2398299302,16cyclictest25866irq/58-eth011:50:103
23982993015,8cyclictest0-21swapper/309:30:093
2398199302,22cyclictest0-21swapper/207:10:142
2398199301,15cyclictest13393-21ssh09:40:152
2398099302,4cyclictest131rcu_preempt12:30:151
2398099301,21cyclictest22594-21ssh11:50:101
2398299293,15cyclictest25866irq/58-eth008:25:143
23982992917,8cyclictest25866irq/58-eth012:35:353
2398199298,7cyclictest10758-1kworker/2:1H10:00:142
2398099292,18cyclictest12971-21sendmail-msp09:40:141
2397999292,20cyclictest5653-21diskmemload11:14:410
2398099282,20cyclictest482-21sendmail-msp09:00:151
2397999282,21cyclictest20442-21ssh12:30:150
2397999282,19cyclictest18878-21ssh09:17:090
2398299273,15cyclictest25866irq/58-eth007:45:113
2398199271,11cyclictest3992-21ssh11:15:132
2398099271,21cyclictest20536-21sendmail-msp09:20:161
2397999272,17cyclictest20536-21sendmail-msp09:20:130
2397999272,16cyclictest845-21munin-node08:35:130
2397999272,16cyclictest845-21munin-node08:35:130
23979992716,8cyclictest7753-21sendmail-msp12:00:150
2397999271,21cyclictest7037-21ssh12:20:160
2397999271,19cyclictest21314-21ssh11:27:110
2398299262,18cyclictest0-21swapper/310:24:103
2398299262,17cyclictest0-21swapper/311:44:113
2398299262,15cyclictest25866irq/58-eth008:45:153
2398299262,12cyclictest25866irq/58-eth011:35:143
23982992612,9cyclictest25866irq/58-eth012:10:113
23981992615,8cyclictest0-21swapper/207:45:122
23981992614,9cyclictest0-21swapper/209:15:292
2398199261,15cyclictest1811-21ssh12:15:292
2398199261,15cyclictest1811-21ssh12:15:282
2398099262,16cyclictest601-21rngd12:14:521
2397999269,12cyclictest5474-21sendmail-msp10:00:150
2397999262,18cyclictest8680-21sendmail-msp11:40:150
23982992516,5cyclictest0-21swapper/309:50:083
23982992514,8cyclictest0-21swapper/309:35:133
2398199252,18cyclictest31268-21ssh10:19:302
2398199252,17cyclictest601-21rngd09:00:142
23981992513,9cyclictest5365-21ssh12:39:542
2398199250,18cyclictest601-21rngd10:55:142
2398099251,15cyclictest8680-21sendmail-msp11:40:151
23979992519,3cyclictest32468-21ssh11:35:150
23979992512,4cyclictest12-21ksoftirqd/012:05:110
23982992411,10cyclictest25866irq/58-eth012:09:533
2398199243,5cyclictest26978-21acpi08:20:132
2398199243,17cyclictest0-21swapper/208:25:142
23981992412,9cyclictest5351-21ssh10:22:242
23981992412,9cyclictest23358-21ssh10:37:142
23981992410,7cyclictest11703-21sendmail-msp11:20:362
2398199240,21cyclictest30437-21ls09:30:022
2397999240,4cyclictest131rcu_preempt09:35:120
2398299233,13cyclictest0-21swapper/310:35:133
2398299233,13cyclictest0-21swapper/309:44:103
2398199233,13cyclictest27-21ksoftirqd/207:50:162
23981992315,5cyclictest7312-21ssh12:20:212
23981992312,9cyclictest22927-21ssh09:49:532
23981992312,9cyclictest22927-21ssh09:49:522
2398199231,11cyclictest32477-21ssh11:35:162
23981992311,10cyclictest10900-21ssh09:36:212
23979992317,4cyclictest12-21ksoftirqd/010:50:090
23982992215,3cyclictest0-21swapper/310:15:113
2398199227,12cyclictest29092-21mailstats10:40:312
2398199225,13cyclictest27447-21awk11:50:292
2398199222,5cyclictest0-21swapper/212:29:562
2398199222,15cyclictest0-21swapper/210:48:102
2398199221,8cyclictest2297066ssh11:01:062
23981992210,9cyclictest22183-21ssh10:12:092
23981992210,9cyclictest20467-21grep07:40:022
2398099223,5cyclictest21-21ksoftirqd/112:00:201
2398099223,13cyclictest665-21ntpd11:35:161
2398099221,16cyclictest6635-21sendmail-msp12:20:141
2397999222,14cyclictest29990-21ssh12:14:530
2397999222,13cyclictest845-21munin-node07:35:120
2398299213,12cyclictest0-21swapper/309:47:113
2398299213,12cyclictest0-21swapper/309:47:113
2398299213,11cyclictest0-21swapper/310:55:153
23982992116,2cyclictest0-21swapper/308:10:153
23982992113,3cyclictest0-21swapper/308:15:123
23982992111,7cyclictest0-21swapper/309:55:123
23982992111,7cyclictest0-21swapper/309:55:113
2398199219,9cyclictest0-21swapper/212:05:102
2398199217,7cyclictest20536-21sendmail-msp09:20:172
2398199217,7cyclictest20485-21sendmail-msp07:40:162
2398199217,11cyclictest1178-21sendmail-msp07:21:242
2398199213,10cyclictest5348-21kthreadcore10:50:172
23981992113,5cyclictest8194-21ssh12:00:152
23981992110,8cyclictest0-21swapper/208:40:092
2398099219,9cyclictest13040-21cron09:15:001
2398099217,9cyclictest21-21ksoftirqd/107:30:171
2398099211,15cyclictest31934-21ssh10:20:131
2398099211,15cyclictest17668-21ssh10:35:141
2397999212,9cyclictest27550-21unixbench_singl11:30:250
23979992111,7cyclictest7425-21sendmail-msp08:00:180
2398299206,11cyclictest25866irq/58-eth011:06:483
2398299204,11cyclictest0-21swapper/312:00:153
2398299202,13cyclictest0-21swapper/311:19:443
23982992015,2cyclictest0-21swapper/312:20:153
23982992013,4cyclictest0-21swapper/308:05:123
2398299201,12cyclictest0-21swapper/311:30:103
2398199206,5cyclictest3656-21latency_hist11:15:002
2398199205,11cyclictest27-21ksoftirqd/207:55:122
2398199203,9cyclictest19218-21kthreadcore08:10:252
2398199201,9cyclictest29215-21hwlatdetect07:15:172
2398199201,11cyclictest7145-21ssh10:25:052
2398199201,11cyclictest15633-21cron07:35:012
2398099203,5cyclictest21-21ksoftirqd/111:00:391
2398099201,15cyclictest24913-21sendmail-msp10:40:291
2398099201,15cyclictest20485-21sendmail-msp07:40:331
2398099201,14cyclictest10343-21apt-get10:55:141
2398099200,16cyclictest30355-21ls08:20:281
2397999201,6cyclictest52246usb-storage08:40:240
23979992013,4cyclictest0-21swapper/010:55:150
2398299198,8cyclictest0-21swapper/310:40:103
2398299193,12cyclictest0-21swapper/311:00:163
2398299193,11cyclictest0-21swapper/310:49:343
2398299193,11cyclictest0-21swapper/309:29:113
2398299193,10cyclictest0-21swapper/312:29:543
2398299192,11cyclictest0-21swapper/311:27:113
23982991914,2cyclictest0-21swapper/310:10:103
23982991914,2cyclictest0-21swapper/309:10:083
2398299191,13cyclictest0-21swapper/308:35:273
2398299191,13cyclictest0-21swapper/308:35:263
2398299191,12cyclictest25866irq/58-eth011:10:153
2398299191,11cyclictest0-21swapper/307:15:223
2398299191,10cyclictest25866irq/58-eth007:35:113
23982991910,6cyclictest0-21swapper/311:49:243
23982991910,6cyclictest0-21swapper/309:15:573
23982991910,6cyclictest0-21swapper/308:30:233
2398199196,10cyclictest9079-21sshd08:35:112
2398199196,10cyclictest9079-21sshd08:35:112
2398199196,10cyclictest4578-21irqcore08:30:222
2398199193,13cyclictest16283-21ls10:10:012
2398199191,9cyclictest10532-21sed10:30:132
2398199191,8cyclictest5657-21cut09:10:012
2398199191,8cyclictest24994-21kthreadcore08:15:212
2398199191,6cyclictest2218866ssh11:49:582
2398199191,15cyclictest6514-21ps09:10:152
2398199191,15cyclictest28522-21fschecks_time08:55:192
2398199191,14cyclictest6769-21irqrtprio07:25:192
2398199191,14cyclictest6769-21irqrtprio07:25:182
2398199191,10cyclictest20232-21cut08:45:222
2398099192,7cyclictest131rcu_preempt09:15:241
2398099190,13cyclictest73-21kswapd007:10:261
2397999199,2cyclictest12-21ksoftirqd/007:10:020
2397999195,9cyclictest12-21ksoftirqd/008:45:140
23979991914,3cyclictest12-21ksoftirqd/009:00:110
23979991910,6cyclictest167-21systemd-network10:27:140
2397999190,3cyclictest131rcu_preempt10:37:380
2398299189,6cyclictest0-21swapper/310:30:133
2398299187,7cyclictest0-21swapper/307:40:243
2398299182,11cyclictest0-21swapper/310:06:113
2398299182,11cyclictest0-21swapper/310:00:103
2398299182,10cyclictest0-21swapper/308:20:113
2398299182,10cyclictest0-21swapper/308:01:113
23982991814,2cyclictest0-21swapper/307:20:133
23982991813,2cyclictest0-21swapper/311:20:163
23982991813,2cyclictest0-21swapper/308:40:153
23982991812,3cyclictest0-21swapper/309:00:123
23982991810,6cyclictest0-21swapper/307:10:303
2398199189,7cyclictest13115-21irqrtprio08:05:192
2398199181,9cyclictest24567-21kthreadcore08:50:222
2398199181,10cyclictest23172-21latency_hist11:30:012
2398099189,7cyclictest20756-21ntpq08:10:281
2398099188,8cyclictest5341-21tr08:30:241
2398099183,5cyclictest21-21ksoftirqd/107:15:151
23980991812,4cyclictest0-21swapper/108:55:221
23980991811,5cyclictest3966-21sshd08:26:101
2398099181,11cyclictest601-21rngd11:48:101
2398099180,13cyclictest16326-21ls08:05:401
2397999188,7cyclictest10533-21irqrtprio10:30:130
2397999188,2cyclictest12-21ksoftirqd/007:20:140
23979991813,3cyclictest0-21swapper/007:10:240
23979991813,2cyclictest12-21ksoftirqd/012:30:020
23979991812,4cyclictest0-21swapper/009:50:110
23979991812,3cyclictest12-21ksoftirqd/011:59:330
2397999181,14cyclictest22571-21apt-get11:50:090
2397999180,16cyclictest5944-21kthreadcore11:15:190
2398299177,7cyclictest0-21swapper/307:25:263
2398299177,7cyclictest0-21swapper/307:25:263
2398299174,10cyclictest0-21swapper/310:25:033
23982991713,2cyclictest0-21swapper/309:20:133
23982991712,3cyclictest0-21swapper/307:50:153
2398199171,9cyclictest0-21swapper/211:05:212
2398099179,6cyclictest167-21systemd-network09:51:541
2398099179,6cyclictest13553-21netstat10:30:211
2398099174,3cyclictest131rcu_preempt07:10:011
2398099174,2cyclictest131rcu_preempt11:32:201
2398099172,10cyclictest601-21rngd11:25:111
2398099172,10cyclictest601-21rngd09:45:111
2398099172,10cyclictest601-21rngd09:45:111
2398099171,7cyclictest18187-21kthreadcore07:35:201
2398099171,7cyclictest10850-21idleruntime-cro07:30:011
2398099171,7cyclictest10850-21idleruntime-cro07:30:011
23980991712,3cyclictest0-21swapper/108:45:171
2398099171,11cyclictest601-21rngd11:15:111
2398099171,10cyclictest601-21rngd10:50:181
2397999172,8cyclictest151rcuc/007:15:130
2397999172,2cyclictest131rcu_preempt09:29:460
2397999172,2cyclictest131rcu_preempt08:15:250
2397999172,2cyclictest131rcu_preempt07:30:240
2397999171,3cyclictest131rcu_preempt09:55:170
2397999171,3cyclictest131rcu_preempt09:55:160
23979991712,3cyclictest0-21swapper/010:06:100
23979991711,3cyclictest12-21ksoftirqd/010:45:420
23979991710,5cyclictest0-21swapper/011:05:150
23979991710,4cyclictest12-21ksoftirqd/009:49:490
23979991710,4cyclictest12-21ksoftirqd/009:49:480
2398299168,6cyclictest0-21swapper/307:30:343
23982991611,2cyclictest0-21swapper/308:55:153
2398099169,5cyclictest9760-21sed10:30:001
2398099168,6cyclictest3508-21cut07:55:191
2398099167,7cyclictest30607-21tune2fs07:50:181
2398099166,7cyclictest13169-21sed09:40:021
2398099162,10cyclictest601-21rngd12:29:551
2398099162,10cyclictest601-21rngd12:08:111
2398099162,10cyclictest601-21rngd10:18:111
2398099162,10cyclictest601-21rngd10:14:121
2398099161,3cyclictest131rcu_preempt12:40:021
23980991611,3cyclictest5439-21sh10:00:021
23980991611,3cyclictest5439-21sh10:00:021
2398099161,11cyclictest9099-21apt-get08:35:131
2398099161,11cyclictest9099-21apt-get08:35:121
2398099161,11cyclictest32246-21sed11:55:191
2398099161,11cyclictest22651-21sed08:15:191
2398099161,10cyclictest601-21rngd10:06:111
2398099160,4cyclictest601-21rngd11:10:321
2397999169,3cyclictest12-21ksoftirqd/009:30:210
23979991611,3cyclictest0-21swapper/011:45:320
23979991611,3cyclictest0-21swapper/008:25:160
2398299155,7cyclictest0-21swapper/308:50:163
2398299151,10cyclictest0-21swapper/309:10:013
23982991510,2cyclictest0-21swapper/307:55:123
2398199151,7cyclictest24148-21sort07:10:022
2398099156,6cyclictest28128-21cron08:55:001
2398099152,10cyclictest25691-21apt-get07:45:111
2398099151,10cyclictest601-21rngd09:28:101
2398099150,4cyclictest601-21rngd11:06:101
2397999151,2cyclictest131rcu_preempt07:55:150
2398099148,4cyclictest5653-21diskmemload10:48:071
2398099141,10cyclictest5736-21latency_hist09:10:011
2397999142,8cyclictest601-21rngd07:45:120
2397999141,10cyclictest5649-21basename09:10:010
2397999140,2cyclictest131rcu_preempt08:55:150
2398299112,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional