You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 01:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Sun May 04, 2025 12:45:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11839996125,24cyclictest27-21ksoftirqd/209:50:162
11816996029,23cyclictest7520-21sendmail-msp12:00:170
1183999566,34cyclictest27-21ksoftirqd/208:00:172
1183999542,41cyclictest0-21swapper/208:50:152
11839994932,11cyclictest27-21ksoftirqd/208:40:142
1181699481,31cyclictest20339-21sendmail-msp09:00:140
1182899472,38cyclictest14220-21sendmail-msp08:20:141
1183999461,38cyclictest0-21swapper/210:25:142
1182899432,11cyclictest131rcu_preempt10:50:131
11839994226,10cyclictest261rcuc/211:50:142
11839994222,16cyclictest27-21ksoftirqd/211:40:142
11839994216,22cyclictest27-21ksoftirqd/212:05:142
11839994216,22cyclictest27-21ksoftirqd/212:05:142
1182899423,35cyclictest12356-21sendmail-msp09:20:171
1183999411,12cyclictest601-21rngd09:00:142
1182899413,35cyclictest4141-21sendmail-msp10:40:161
1182899411,37cyclictest2141-21sendmail-msp11:00:161
1182899411,37cyclictest2141-21sendmail-msp11:00:161
1184899402,34cyclictest0-21swapper/307:15:153
1183999401,34cyclictest4783-21apt-get08:10:152
11828994027,10cyclictest21-21ksoftirqd/109:10:171
1183999391,26cyclictest31265-21apt-get11:55:142
1182899395,5cyclictest131rcu_preempt10:20:131
1184899381,21cyclictest25866irq/58-eth011:05:163
1183999381,32cyclictest0-21swapper/207:30:152
1183999381,26cyclictest0-21swapper/210:35:142
11839993812,21cyclictest0-21swapper/209:45:152
11839993812,21cyclictest0-21swapper/209:45:152
1182899383,4cyclictest131rcu_preempt07:50:161
1182899381,26cyclictest31277-21ssh11:55:141
1181699382,29cyclictest27348-21sendmail-msp08:00:150
1183999371,31cyclictest0-21swapper/209:10:172
1183999371,29cyclictest0-21swapper/210:50:142
1183999370,33cyclictest22765-21apt-get07:55:142
1182899374,27cyclictest21-21ksoftirqd/107:45:171
1182899372,32cyclictest3833-21sendmail-msp11:40:141
1183999361,23cyclictest30500-21ssh11:16:142
1183999360,32cyclictest10214-21ssh11:05:152
1183999360,21cyclictest9823-21ssh11:25:132
1182899362,24cyclictest21283-21sendmail-msp07:20:171
1183999351,31cyclictest29305-21kthreadcore10:55:222
1181699352,19cyclictest11038-21sendmail-msp12:20:170
11816993518,9cyclictest10783-21sendmail-msp09:40:180
11816993518,9cyclictest10783-21sendmail-msp09:40:180
11839993414,17cyclictest0-21swapper/207:15:152
1181699349,7cyclictest8145-21sendmail-msp07:40:350
1181699343,18cyclictest6228-21sendmail-msp10:20:150
1184899331,26cyclictest25866irq/58-eth010:55:133
11839993313,5cyclictest522-21usb-storage07:20:152
1183999331,29cyclictest12662-21cut08:15:272
1183999331,26cyclictest0-21swapper/209:30:162
1183999330,29cyclictest9508-21ssh09:37:412
1183999330,29cyclictest20318-21apt-get12:25:132
1181699330,16cyclictest27800-21ls10:10:250
1184899321,26cyclictest0-21swapper/310:10:133
1183999328,21cyclictest7389-21cat11:00:422
1183999328,21cyclictest7389-21cat11:00:422
1183999327,17cyclictest7520-21sendmail-msp12:00:222
1183999322,27cyclictest2074-21kthreadcore08:05:222
1183999322,26cyclictest1657-21apt-get09:55:182
1183999322,26cyclictest1657-21apt-get09:55:182
1183999321,28cyclictest20231-21idleruntime-cro09:25:012
1183999321,28cyclictest15923-21usage_EthcTask09:40:442
1183999321,28cyclictest15923-21usage_EthcTask09:40:442
1183999321,25cyclictest17794-21apt-get11:10:132
1183999320,27cyclictest6438-21ssh10:20:022
1183999320,27cyclictest0-21swapper/210:15:152
1181699322,20cyclictest8947-21sendmail-msp10:00:170
11816993219,7cyclictest7573-21kworker/u8:007:20:150
1184899312,19cyclictest0-21swapper/310:25:133
1184899311,24cyclictest0-21swapper/311:55:143
1184899311,24cyclictest0-21swapper/311:10:133
1183999311,27cyclictest6858-21ssh11:21:582
1183999311,27cyclictest29028-21ssh10:34:212
1183999311,27cyclictest24168-21latency_hist10:10:022
1183999311,27cyclictest15672-21/usr/sbin/munin07:10:312
1183999311,27cyclictest13541-21ssh12:38:492
1183999311,27cyclictest13541-21ssh12:38:492
1183999311,27cyclictest10091-21kthreadcore10:00:252
1183999311,26cyclictest0-21swapper/209:25:132
1183999311,23cyclictest30440-21cat12:10:342
1183999311,22cyclictest27730-21apt-get11:35:152
1183999311,22cyclictest10746-21awk08:50:012
1183999311,21cyclictest23045-21expr08:25:292
1182899315,19cyclictest2425-21ssh12:15:131
1181699311,14cyclictest2141-21sendmail-msp11:00:190
1181699311,14cyclictest2141-21sendmail-msp11:00:190
11848993015,11cyclictest25866irq/58-eth009:45:153
11848993015,11cyclictest25866irq/58-eth009:45:153
1183999303,24cyclictest14220-21sendmail-msp08:20:292
1183999301,27cyclictest29022-21cat10:10:342
1183999301,26cyclictest6946-21nfs_client07:35:262
1183999301,26cyclictest4349-21/usr/sbin/munin10:40:222
1183999301,26cyclictest29211-21unixbench_multi09:05:302
1183999301,26cyclictest20474-21latency_hist09:00:022
1183999301,26cyclictest12794-21ssh10:45:212
1183999301,24cyclictest18784-21apt-get11:30:132
1183999301,21cyclictest1120-21awk08:40:012
1183999300,21cyclictest5102-21apt-get09:15:152
11828993014,7cyclictest21-21ksoftirqd/110:35:141
1181699302,9cyclictest27762-21ssh11:35:150
1183999297,15cyclictest17072-21sort11:45:322
1183999291,25cyclictest1804-21usage_EthcTask12:30:352
1183999291,25cyclictest10500-21kthreadcore07:40:272
1183999291,24cyclictest11645-21fschecks_time12:20:202
1183999291,20cyclictest28778-21latency_hist08:35:012
1183999291,20cyclictest0-21swapper/212:15:112
1181699292,23cyclictest612-21sendmail-msp11:20:130
1184899282,17cyclictest25866irq/58-eth012:05:143
1184899282,17cyclictest25866irq/58-eth012:05:143
1184899281,24cyclictest0-21swapper/309:20:153
1184899281,23cyclictest0-21swapper/310:20:133
1183999281,24cyclictest19022-21kthreadcore07:50:222
1183999280,25cyclictest0-21swapper/207:45:162
1182899286,12cyclictest21-21ksoftirqd/112:20:151
1182899282,21cyclictest1032-21ssh11:20:151
11828992812,12cyclictest8947-21sendmail-msp10:00:361
11828992810,6cyclictest131rcu_preempt10:55:141
1181699282,22cyclictest4276-21ssh11:40:130
1181699282,19cyclictest12356-21sendmail-msp09:20:340
1184899271,19cyclictest0-21swapper/307:35:153
1184899270,22cyclictest0-21swapper/308:35:123
1183999271,23cyclictest30974-21sshd07:25:392
1183999271,23cyclictest11846-21latency_hist07:10:012
1183999271,23cyclictest11846-21latency_hist07:10:012
1182899271,20cyclictest17817-21sh11:10:141
1184899261,22cyclictest0-21swapper/310:35:143
1184899261,22cyclictest0-21swapper/308:45:153
1184899261,21cyclictest0-21swapper/309:00:143
1182899262,18cyclictest22100-21ssh10:30:141
11828992612,5cyclictest131rcu_preempt09:15:141
1184899251,21cyclictest0-21swapper/312:30:143
1184899250,21cyclictest0-21swapper/307:30:143
11828992519,2cyclictest601-21rngd08:25:151
1181699254,18cyclictest31296-21ssh11:55:150
1181699252,19cyclictest11776-21ssh10:45:150
1181699251,19cyclictest18806-21ssh11:30:140
1184899241,20cyclictest0-21swapper/312:20:013
1184899241,20cyclictest0-21swapper/308:25:133
1184899241,19cyclictest0-21swapper/309:10:163
1184899241,19cyclictest0-21swapper/308:05:133
1184899241,18cyclictest0-21swapper/310:15:153
1184899241,18cyclictest0-21swapper/307:45:163
11828992414,7cyclictest7520-21sendmail-msp12:00:361
1184899231,20cyclictest0-21swapper/310:30:133
1184899231,19cyclictest0-21swapper/312:25:133
1184899231,19cyclictest0-21swapper/311:45:233
1184899231,18cyclictest0-21swapper/310:50:143
1184899231,18cyclictest0-21swapper/310:05:143
1184899231,18cyclictest0-21swapper/308:40:153
1184899231,18cyclictest0-21swapper/307:10:143
1184899230,18cyclictest0-21swapper/308:10:153
11816992310,9cyclictest4141-21sendmail-msp10:41:590
1184899223,16cyclictest0-21swapper/311:35:013
1184899223,12cyclictest0-21swapper/312:10:063
11848992212,7cyclictest0-21swapper/311:40:273
1184899221,18cyclictest0-21swapper/311:22:373
1184899221,18cyclictest0-21swapper/311:15:183
1184899221,18cyclictest0-21swapper/310:00:163
1184899221,18cyclictest0-21swapper/309:57:223
1184899221,18cyclictest0-21swapper/309:57:213
1184899221,18cyclictest0-21swapper/309:15:263
1184899221,18cyclictest0-21swapper/308:30:133
1184899221,18cyclictest0-21swapper/308:20:143
1184899221,17cyclictest0-21swapper/311:50:123
1184899221,17cyclictest0-21swapper/307:50:163
1184899220,19cyclictest0-21swapper/309:50:153
1182899221,13cyclictest845-21munin-node11:05:161
1181699223,13cyclictest5104-21ssh09:15:130
1181699222,13cyclictest601-21rngd10:25:050
1181699221,17cyclictest21351-21apt-config07:20:020
1181699220,19cyclictest10367-21cut11:05:190
11848992112,7cyclictest0-21swapper/309:41:043
11848992112,7cyclictest0-21swapper/309:41:043
1184899211,17cyclictest0-21swapper/312:22:283
1184899211,17cyclictest0-21swapper/312:00:153
1184899211,17cyclictest0-21swapper/311:25:243
1184899211,17cyclictest0-21swapper/311:00:153
1184899211,17cyclictest0-21swapper/311:00:153
1184899211,17cyclictest0-21swapper/310:45:273
1184899211,17cyclictest0-21swapper/309:35:213
1184899211,17cyclictest0-21swapper/309:30:213
1184899211,17cyclictest0-21swapper/309:25:123
1184899211,17cyclictest0-21swapper/309:05:153
1184899211,17cyclictest0-21swapper/308:15:173
1184899211,17cyclictest0-21swapper/307:55:153
1184899211,17cyclictest0-21swapper/307:20:153
1184899211,16cyclictest0-21swapper/308:55:133
1184899210,17cyclictest0-21swapper/307:25:143
1182899212,13cyclictest601-21rngd09:30:161
1182899212,11cyclictest601-21rngd11:45:151
11828992113,5cyclictest0-21swapper/110:15:171
1182899211,14cyclictest6024-21ssh12:35:131
1182899211,14cyclictest6024-21ssh12:35:131
1181699215,12cyclictest3474-21ssh09:35:140
1181699212,16cyclictest13197-21sh11:45:150
1181699211,7cyclictest30860-21kthreadcore08:35:200
11816992114,5cyclictest0-21swapper/012:10:030
1181699211,17cyclictest1573-21ls08:40:190
1181699211,14cyclictest13213-21acpi07:45:160
1184899203,15cyclictest0-21swapper/307:40:173
1184899201,16cyclictest0-21swapper/311:35:143
1184899201,16cyclictest0-21swapper/310:40:413
1184899200,17cyclictest0-21swapper/308:50:133
1182899207,10cyclictest24017-21pluginstate10:10:011
1182899202,12cyclictest601-21rngd09:35:151
1182899202,10cyclictest601-21rngd11:25:141
1182899201,8cyclictest21-21ksoftirqd/111:55:011
1182899201,15cyclictest26281-21apt-get09:50:161
1182899201,12cyclictest5452-21kthreadcore08:10:211
1181699201,16cyclictest1672-21ssh09:55:200
1181699201,16cyclictest1672-21ssh09:55:190
1181699201,15cyclictest14660-21ls12:40:020
1181699201,15cyclictest14660-21ls12:40:020
1184899191,16cyclictest0-21swapper/312:35:173
1184899191,16cyclictest0-21swapper/312:35:173
1184899191,15cyclictest0-21swapper/308:00:153
1182899199,8cyclictest29440-21ssh12:30:141
1182899196,4cyclictest131rcu_preempt10:30:011
1182899195,11cyclictest1117-21sendmail-msp08:40:351
1182899193,10cyclictest11812-21cyclictest09:05:141
1182899191,5cyclictest52246usb-storage09:01:161
1182899191,5cyclictest52246usb-storage07:40:281
1182899190,6cyclictest52246usb-storage12:10:101
1182899190,5cyclictest52246usb-storage09:40:321
1182899190,5cyclictest52246usb-storage09:40:321
1181699192,14cyclictest0-21swapper/010:05:140
1181699190,4cyclictest601-21rngd12:26:400
1182899187,8cyclictest27341-21cron11:35:021
1182899186,9cyclictest10298-21kthreadcore08:15:251
1182899183,12cyclictest27278-21kthreadcore07:25:211
1182899181,14cyclictest3853-21ssh11:40:021
1182899181,13cyclictest1863-21fschecks_time09:55:211
1182899181,13cyclictest1863-21fschecks_time09:55:211
1181699188,7cyclictest14220-21sendmail-msp08:21:110
1181699185,7cyclictest12-21ksoftirqd/007:55:150
1181699184,12cyclictest31082-21sed10:55:310
1181699181,9cyclictest11927-21/usr/sbin/munin07:10:300
1181699181,8cyclictest601-21rngd10:15:210
1182899179,6cyclictest15510-21sshd08:51:391
1182899171,3cyclictest601-21rngd09:45:311
1182899171,3cyclictest601-21rngd09:45:311
1182899171,13cyclictest12958-21tr10:45:221
1182899171,11cyclictest1201-21latency_hist08:40:011
1182899170,10cyclictest9934-21timerwakeupswit08:45:311
1181699179,5cyclictest11000-21cron12:20:010
1181699178,7cyclictest21893-21cut08:25:230
1181699178,6cyclictest4086-21irqrtprio07:35:210
1181699173,6cyclictest1517-21kthreadcore07:30:240
1181699173,11cyclictest30335-21grep09:50:340
1181699172,12cyclictest1398-21ssh10:35:330
1181699171,3cyclictest131rcu_preempt12:10:030
1181699171,3cyclictest131rcu_preempt12:10:020
11816991711,4cyclictest0-21swapper/009:45:170
11816991711,4cyclictest0-21swapper/009:45:170
11816991711,4cyclictest0-21swapper/008:55:120
1181699171,13cyclictest11016-21kthreadcore11:25:200
1181699171,12cyclictest24824-21kthreadcore11:50:230
1181699171,11cyclictest26430-21acpi07:25:140
1181699171,10cyclictest601-21rngd11:15:140
1181699170,11cyclictest4893-21uniq08:10:170
1184899161,12cyclictest0-21swapper/307:10:023
1184899161,12cyclictest0-21swapper/307:10:013
1182899169,4cyclictest0-21swapper/112:25:141
1182899165,8cyclictest27605-21cut11:15:221
1182899164,4cyclictest131rcu_preempt08:55:141
1182899164,10cyclictest17427-21cut12:05:201
1182899164,10cyclictest17427-21cut12:05:201
1182899163,10cyclictest24442-21sed08:30:191
1182899161,11cyclictest601-21rngd10:10:141
11828991610,4cyclictest27548-21/usr/sbin/munin08:00:171
1182899160,13cyclictest3276-21ls08:05:241
1181699168,6cyclictest29966-21cron09:10:010
1181699163,9cyclictest12586-21kthreadcore08:15:270
1181699161,7cyclictest30302-21/usr/sbin/munin09:10:180
1181699161,6cyclictest19372-21apt-get10:50:130
1181699161,4cyclictest17985-21apt-get07:50:160
1181699161,12cyclictest18796-21kthreadcore11:10:200
1181699161,11cyclictest20574-21apt-get09:25:130
1181699160,11cyclictest601-21rngd10:35:010
1182899152,10cyclictest27475-21wc08:00:021
1182899151,9cyclictest21261-21sendmail07:20:011
1182899151,11cyclictest20956-21cut09:25:181
1182899151,10cyclictest8118-21munin-run07:40:011
1182899150,5cyclictest601-21rngd07:15:021
1181699153,9cyclictest15527-21latency_hist08:55:010
1181699151,9cyclictest29299-21ssh12:30:060
1181699151,9cyclictest28439-21acpi09:30:140
1182899140,10cyclictest3349-21idleruntime-cro07:35:011
1181699141,9cyclictest6502-21irqcore08:45:190
1181699141,9cyclictest24331-21fschecks_count08:30:170
1181699141,10cyclictest32408-21/usr/sbin/munin08:05:130
1182899131,9cyclictest11855-21latency_hist07:10:011
1182899131,9cyclictest11855-21latency_hist07:10:011
1181699130,10cyclictest11812-21cyclictest07:10:010
1181699130,10cyclictest11812-21cyclictest07:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional