You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-03-03 - 23:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Mar 03, 2024 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
734999591,45cyclictest0-21swapper/211:25:192
734999591,45cyclictest0-21swapper/211:25:192
734399573,41cyclictest788-21ssh09:55:211
734399564,11cyclictest131rcu_preempt08:50:191
734399552,45cyclictest27683-21ssh10:55:201
733999534,46cyclictest8837-21ssh10:00:260
734999511,27cyclictest0-21swapper/212:00:222
734399491,38cyclictest1855-21ssh12:00:211
734399464,36cyclictest18597-21ssh09:25:241
734399453,14cyclictest21-21ksoftirqd/111:50:191
735699433,28cyclictest20166irq/58-eth010:20:203
734999412,22cyclictest0-21swapper/210:20:172
7349994010,14cyclictest24524-1kworker/u9:110:10:222
733999406,22cyclictest23108-21kworker/u8:110:10:220
7356993917,9cyclictest20166irq/58-eth011:25:193
7356993917,9cyclictest20166irq/58-eth011:25:193
734399385,4cyclictest131rcu_preempt07:35:161
734399382,26cyclictest10348-21ssh11:05:241
733999382,28cyclictest32425-21ssh10:15:210
733999381,27cyclictest25914-21ssh11:55:200
7339993727,5cyclictest0-21swapper/011:45:190
735699362,21cyclictest20166irq/58-eth011:40:223
734399368,18cyclictest21-21ksoftirqd/107:40:211
733999361,26cyclictest17966-21ssh11:10:190
735699343,21cyclictest20166irq/58-eth007:45:223
7349993424,4cyclictest10342-21ssh11:05:242
734399347,18cyclictest21-21ksoftirqd/109:35:221
734399347,18cyclictest21-21ksoftirqd/109:35:221
735699334,17cyclictest20166irq/58-eth012:15:203
735699333,18cyclictest20166irq/58-eth010:35:213
734999333,12cyclictest27-21ksoftirqd/211:30:182
734399331,23cyclictest26441-21ssh09:30:261
734399330,6cyclictest131rcu_preempt10:05:211
734999322,21cyclictest9120-21ssh11:45:192
734399329,7cyclictest131rcu_preempt09:54:511
734999317,19cyclictest24113-21ssh10:09:542
734999314,18cyclictest21846irq/40-dwc2_hso08:05:202
734999313,17cyclictest0-21swapper/211:40:222
734999311,18cyclictest17976-21ssh11:10:202
734999302,10cyclictest18617-21ssh12:10:222
735699294,16cyclictest0-21swapper/309:10:193
734999293,20cyclictest0-21swapper/210:15:172
734999293,17cyclictest21846irq/40-dwc2_hso10:45:212
7343992911,8cyclictest131rcu_preempt08:00:191
7339992912,8cyclictest0-21swapper/011:40:220
733999291,21cyclictest5581-21sh10:40:210
735699286,15cyclictest20166irq/58-eth010:00:253
735699282,12cyclictest20166irq/58-eth012:25:223
7356992811,8cyclictest20166irq/58-eth010:10:223
734999286,15cyclictest0-21swapper/208:15:222
734999283,20cyclictest21846irq/40-dwc2_hso09:00:212
734999282,20cyclictest0-21swapper/207:30:192
734999282,20cyclictest0-21swapper/207:25:222
734399282,18cyclictest17490-21ssh09:45:181
733999287,13cyclictest12-21ksoftirqd/012:30:220
733999282,19cyclictest4846-21ssh10:39:170
734999273,18cyclictest21846irq/40-dwc2_hso10:50:202
734999273,17cyclictest21846irq/40-dwc2_hso08:45:202
734999273,17cyclictest21846irq/40-dwc2_hso07:50:212
734999272,19cyclictest0-21swapper/208:10:222
7339992717,6cyclictest0-21swapper/011:30:180
733999271,18cyclictest27061-21ssh12:15:200
733999271,18cyclictest24954-21ssh11:35:230
735699263,17cyclictest0-21swapper/309:57:173
734999263,15cyclictest21846irq/40-dwc2_hso07:45:212
734999262,17cyclictest21846irq/40-dwc2_hso08:20:212
734999260,19cyclictest27070-21sh12:15:202
733999269,8cyclictest12-21ksoftirqd/011:06:260
733999268,12cyclictest12-21ksoftirqd/009:59:170
733999261,22cyclictest28292-21ssh12:35:230
7339992611,8cyclictest12-21ksoftirqd/012:20:350
735699253,18cyclictest0-21swapper/311:55:213
734999255,10cyclictest21846irq/40-dwc2_hso08:30:202
734999253,16cyclictest0-21swapper/210:36:172
734999252,7cyclictest131rcu_preempt09:59:172
734999252,18cyclictest30171-21ssh10:34:582
734999252,16cyclictest21846irq/40-dwc2_hso11:15:202
734999251,9cyclictest2635366ssh09:50:312
7349992517,5cyclictest4358-21ssh09:35:312
7349992517,5cyclictest4358-21ssh09:35:312
7339992510,9cyclictest0-21swapper/011:29:070
7339992510,9cyclictest0-21swapper/011:29:070
7339992510,4cyclictest12-21ksoftirqd/008:15:350
7356992411,9cyclictest0-21swapper/310:15:203
734999245,14cyclictest21846irq/40-dwc2_hso08:25:202
734999243,15cyclictest21846irq/40-dwc2_hso11:55:192
734999242,18cyclictest21846irq/40-dwc2_hso09:45:192
734999242,15cyclictest21846irq/40-dwc2_hso09:05:202
7343992413,7cyclictest0-21swapper/108:40:191
733999249,7cyclictest131rcu_preempt09:10:140
733999249,7cyclictest12-21ksoftirqd/009:20:190
733999249,2cyclictest131rcu_preempt10:45:220
733999248,5cyclictest12-21ksoftirqd/010:53:330
733999247,6cyclictest12-21ksoftirqd/011:50:040
733999242,17cyclictest22502-21ssh10:07:180
7339992418,4cyclictest151rcuc/007:40:200
7339992417,4cyclictest0-21swapper/009:15:220
735699236,12cyclictest0-21swapper/311:05:243
734999233,16cyclictest21846irq/40-dwc2_hso07:15:202
734999233,11cyclictest29213-21kthreadcore09:10:272
734999232,6cyclictest0-21swapper/210:00:172
734999232,16cyclictest0-21swapper/210:25:172
734999231,15cyclictest21846irq/40-dwc2_hso12:25:222
734999231,11cyclictest976-21/usr/sbin/munin11:20:402
734399237,12cyclictest21-21ksoftirqd/110:25:251
734399231,15cyclictest25126-21cut11:35:251
7343992311,3cyclictest26995-21diskmemload10:36:311
733999239,8cyclictest12-21ksoftirqd/009:32:020
733999234,10cyclictest12-21ksoftirqd/010:58:380
733999233,5cyclictest12-21ksoftirqd/010:30:300
7339992313,7cyclictest131-21systemd-journal12:25:020
733999231,17cyclictest9693-21ssh09:40:190
7339992311,4cyclictest12-21ksoftirqd/009:45:360
7339992311,3cyclictest0-21swapper/011:00:230
735699223,12cyclictest0-21swapper/311:30:193
735699222,12cyclictest20166irq/58-eth011:45:193
734999223,15cyclictest0-21swapper/210:55:202
734999222,15cyclictest0-21swapper/207:10:172
734999222,14cyclictest21846irq/40-dwc2_hso08:40:172
7349992217,3cyclictest27-21ksoftirqd/208:00:042
7349992211,8cyclictest0-21swapper/208:55:242
734999221,14cyclictest21846irq/40-dwc2_hso08:35:192
734999221,12cyclictest19917-21ssh12:30:222
734399225,10cyclictest0-21swapper/108:30:211
734399221,16cyclictest28305-21sh12:35:251
733999229,7cyclictest12-21ksoftirqd/007:10:300
733999228,4cyclictest12-21ksoftirqd/009:50:250
733999226,9cyclictest12-21ksoftirqd/008:30:260
733999224,8cyclictest12-21ksoftirqd/012:10:160
7339992216,4cyclictest0-21swapper/011:15:190
733999221,18cyclictest1948-21ls12:00:230
735699219,9cyclictest0-21swapper/309:38:153
735699219,9cyclictest0-21swapper/309:38:153
735699217,6cyclictest20166irq/58-eth011:10:203
735699214,11cyclictest0-21swapper/311:35:243
735699213,13cyclictest20166irq/58-eth011:50:213
735699212,13cyclictest20166irq/58-eth009:50:253
735699212,12cyclictest0-21swapper/307:35:183
734999214,14cyclictest32003-21fschecks_count07:35:202
734999213,9cyclictest7880-21wc07:40:362
734999213,13cyclictest21846irq/40-dwc2_hso07:20:202
734999211,9cyclictest25078-21fschecks_count11:35:252
7349992116,3cyclictest909-21snmpd12:06:242
734999211,12cyclictest909-21snmpd07:55:232
734999211,12cyclictest909-21snmpd07:55:232
734999211,10cyclictest28889-21cut12:35:302
734999211,10cyclictest10126-21kthreadcore08:50:292
7349992110,7cyclictest6877-21usage_Anybus09:15:402
734399219,7cyclictest131rcu_preempt10:21:171
734399212,14cyclictest11434-21ssh12:25:231
7343992113,5cyclictest0-21swapper/110:30:221
733999219,4cyclictest12-21ksoftirqd/007:35:330
733999217,8cyclictest12-21ksoftirqd/007:55:240
733999217,8cyclictest12-21ksoftirqd/007:55:240
733999217,3cyclictest131rcu_preempt07:30:200
733999214,4cyclictest12-21ksoftirqd/010:20:260
733999213,12cyclictest626-21tcf-agent08:35:190
7339992113,4cyclictest12-21ksoftirqd/011:20:210
7339992112,5cyclictest0-21swapper/008:00:190
733999211,17cyclictest1927-21cut09:35:250
733999211,17cyclictest1927-21cut09:35:250
735699208,8cyclictest20166irq/58-eth010:50:213
735699204,9cyclictest20166irq/58-eth007:20:183
735699203,14cyclictest0-21swapper/310:05:213
7356992016,3cyclictest321rcuc/308:55:223
7356992013,4cyclictest0-21swapper/309:15:223
734999208,9cyclictest7133-21unixbench_multi12:20:392
734999208,10cyclictest0-21swapper/210:40:272
734999202,9cyclictest21340-21kthreadcore09:25:352
734999201,9cyclictest19676-21kthreadcore11:50:302
734999201,16cyclictest28406-21kthreadcore09:30:332
734999201,15cyclictest13009-21cut09:20:322
734999201,10cyclictest4237-21cut11:00:282
734999201,10cyclictest12579-21kthreadcore09:40:282
734399204,6cyclictest21-21ksoftirqd/109:10:371
734399203,10cyclictest0-21swapper/107:55:221
734399203,10cyclictest0-21swapper/107:55:221
7343992011,6cyclictest0-21swapper/110:15:301
734399201,13cyclictest6538-21missed_timers12:20:321
733999209,3cyclictest131rcu_preempt08:50:190
733999208,4cyclictest12-21ksoftirqd/007:25:330
733999207,5cyclictest12-21ksoftirqd/007:45:370
733999204,2cyclictest131rcu_preempt10:25:250
733999202,5cyclictest12-21ksoftirqd/009:25:320
7339992013,4cyclictest0-21swapper/008:45:200
7339992012,5cyclictest0-21swapper/009:05:230
733999201,14cyclictest842-21munin-node08:25:210
733999201,10cyclictest14172-21fschecks_count07:50:240
735699199,6cyclictest0-21swapper/312:23:473
735699198,8cyclictest0-21swapper/312:05:253
735699198,8cyclictest0-21swapper/310:41:333
735699193,11cyclictest0-21swapper/310:34:593
7356991914,3cyclictest0-21swapper/310:59:103
7356991913,4cyclictest0-21swapper/311:21:533
735699191,12cyclictest0-21swapper/309:05:233
735699191,12cyclictest0-21swapper/308:10:313
734999191,15cyclictest7367-21wc07:10:012
734999191,15cyclictest7367-21wc07:10:012
734399199,7cyclictest14438-21ssh09:20:361
734399197,3cyclictest131rcu_preempt11:40:311
734399193,2cyclictest131rcu_preempt09:05:191
734399191,5cyclictest52346usb-storage11:20:061
734399191,15cyclictest31409-21kthreadcore08:05:281
733999197,6cyclictest12-21ksoftirqd/008:40:320
733999196,7cyclictest12-21ksoftirqd/008:20:420
733999192,12cyclictest842-21munin-node08:55:230
7339991913,4cyclictest0-21swapper/012:05:180
735699183,9cyclictest0-21swapper/308:15:223
735699183,11cyclictest0-21swapper/309:30:263
735699182,7cyclictest0-21swapper/312:35:253
735699182,10cyclictest0-21swapper/311:15:203
735699182,10cyclictest0-21swapper/309:00:223
735699182,10cyclictest0-21swapper/308:00:043
7356991812,4cyclictest0-21swapper/312:31:153
7356991812,4cyclictest0-21swapper/310:28:143
7356991812,4cyclictest0-21swapper/309:43:453
7356991812,4cyclictest0-21swapper/309:27:373
735699181,13cyclictest20166irq/58-eth009:45:183
735699181,13cyclictest0-21swapper/310:45:363
735699181,13cyclictest0-21swapper/308:35:183
735699181,12cyclictest0-21swapper/311:00:373
735699181,12cyclictest0-21swapper/307:25:203
734399189,7cyclictest5731-21sshd08:12:161
734399187,9cyclictest3439-21grep08:45:251
734399186,3cyclictest131rcu_preempt12:30:301
734399184,4cyclictest21-21ksoftirqd/109:15:311
734399181,9cyclictest19993-21latency_hist10:50:011
734399181,7cyclictest18582-21/usr/sbin/munin12:10:371
734399181,15cyclictest73-21kswapd011:00:211
7343991811,4cyclictest29070-21kthreadcore12:15:291
734399181,13cyclictest9692-21diskmemload09:40:191
734399181,12cyclictest16732-21kthreadcore08:25:281
7343991810,6cyclictest18612-21cron07:55:011
7343991810,5cyclictest9551-21ps10:00:311
733999189,7cyclictest28766-21df_inode08:05:210
733999181,6cyclictest12-21ksoftirqd/007:25:010
7339991812,4cyclictest0-21swapper/007:15:240
735699179,3cyclictest0-21swapper/308:54:073
735699179,3cyclictest0-21swapper/308:07:563
735699177,7cyclictest20166irq/58-eth009:20:233
7356991712,3cyclictest0-21swapper/308:30:273
7356991711,4cyclictest0-21swapper/312:11:443
7356991711,4cyclictest0-21swapper/312:00:273
7356991711,4cyclictest0-21swapper/307:55:333
7356991711,4cyclictest0-21swapper/307:55:333
735699171,12cyclictest0-21swapper/307:20:023
735699171,10cyclictest0-21swapper/308:45:053
734399179,6cyclictest20305-21sshd10:50:171
734399179,6cyclictest16924-21fw_packets11:30:221
734399179,5cyclictest21-21ksoftirqd/111:55:261
734399178,7cyclictest12429-21awk07:15:191
734399178,4cyclictest27776-21kthreadcore11:15:291
734399177,7cyclictest9452-21cat11:45:231
734399174,3cyclictest131rcu_preempt09:00:301
734399174,10cyclictest29255-21timerwakeupswit08:35:411
734399174,10cyclictest20332-21kthreadcore07:20:271
734399173,11cyclictest9682-21irqrtprio07:45:271
734399172,9cyclictest15157-21ssh12:05:401
734399171,9cyclictest9088-21ssh11:25:221
734399171,9cyclictest9088-21ssh11:25:221
734399171,8cyclictest7777-21kthreadcore08:15:291
734399171,7cyclictest21866-21find11:10:371
734399170,15cyclictest27174-21cut07:30:211
734399170,14cyclictest25431-21ls07:25:321
733999175,5cyclictest12-21ksoftirqd/008:10:220
733999171,7cyclictest7365-21latency_hist07:10:010
733999171,7cyclictest7365-21latency_hist07:10:010
733999171,11cyclictest842-21munin-node09:00:220
735699169,3cyclictest0-21swapper/308:40:133
7356991611,3cyclictest321rcuc/307:40:263
7356991611,3cyclictest0-21swapper/308:28:173
7356991611,3cyclictest0-21swapper/308:20:363
734399168,6cyclictest24864-21ssh10:10:261
734399163,10cyclictest5990-21sed10:40:261
734399162,11cyclictest0-21swapper/108:20:211
734399161,12cyclictest12812-21tune2fs08:55:251
734399161,12cyclictest11773-21cat07:10:371
735699152,10cyclictest0-21swapper/307:10:173
7356991510,3cyclictest0-21swapper/307:50:313
734399151,11cyclictest7445-21wc07:10:011
734399151,11cyclictest7445-21wc07:10:011
735699149,3cyclictest0-21swapper/307:30:113
735699111,6cyclictest0-21swapper/307:10:013
735699111,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional