You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 22:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Tue Dec 02, 2025 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
728599611,48cyclictest30151-21apt-get11:00:142
728599611,48cyclictest30151-21apt-get11:00:142
727799583,45cyclictest26267-21ssh09:10:161
727799572,45cyclictest31087-21ssh12:25:151
727799572,45cyclictest31087-21ssh12:25:151
728599561,50cyclictest0-21swapper/212:15:132
728599531,33cyclictest23027-21ssh12:20:152
728599511,44cyclictest0-21swapper/208:30:122
7293994929,12cyclictest29566irq/58-eth011:20:123
728599491,41cyclictest0-21swapper/211:50:152
728599491,39cyclictest0-21swapper/208:15:122
727799491,34cyclictest0-21swapper/112:00:121
728599481,42cyclictest28006-21apt-get09:55:122
728599471,42cyclictest8903-21apt-get10:25:142
728599470,42cyclictest0-21swapper/208:10:142
7269994719,22cyclictest12-21ksoftirqd/011:15:150
728599461,39cyclictest17130-21apt-get07:20:132
728599451,39cyclictest0-21swapper/211:30:132
728599451,39cyclictest0-21swapper/211:15:152
728599451,39cyclictest0-21swapper/209:50:132
726999451,36cyclictest5766-21ssh09:40:140
7293994431,9cyclictest29566irq/58-eth009:15:143
728599440,38cyclictest0-21swapper/211:35:142
728599432,37cyclictest0-21swapper/207:40:112
728599431,36cyclictest0-21swapper/211:10:152
728599422,35cyclictest0-21swapper/210:10:132
728599422,33cyclictest0-21swapper/212:30:132
728599421,27cyclictest9329-21ntp_states09:40:352
729399411,3cyclictest0-21swapper/309:10:153
728599411,36cyclictest0-21swapper/209:00:132
728599410,37cyclictest21917-21apt-get07:25:142
728599410,32cyclictest7437-21apt-get07:10:112
727799411,34cyclictest26831-21ssh11:40:161
729399403,25cyclictest29566irq/58-eth009:35:153
728599401,35cyclictest10983-21kthreadcore12:10:252
728599401,25cyclictest30897-21sh10:39:102
728599401,25cyclictest30897-21sh10:39:102
728599401,24cyclictest25809-21diskmemload10:56:382
727799401,20cyclictest131rcu_preempt10:55:141
728599392,30cyclictest0-21swapper/211:55:152
728599391,35cyclictest26960-21fschecks_time07:30:192
728599391,26cyclictest1289-21ssh10:20:122
728599391,23cyclictest18956-21ssh09:48:352
726999392,15cyclictest12-21ksoftirqd/011:55:140
729399383,21cyclictest29566irq/58-eth010:20:133
728599381,34cyclictest5530-21apt-get11:05:142
728599381,34cyclictest27319-21ps09:30:352
728599381,34cyclictest11564-21kthreadcore08:20:192
728599381,33cyclictest358-21wc08:10:012
728599381,28cyclictest0-21swapper/209:15:132
728599381,24cyclictest932-21snmpd10:54:582
727799381,30cyclictest32699-21ssh12:05:161
727799381,28cyclictest8861-21ssh09:20:141
726999383,26cyclictest12-21ksoftirqd/010:40:150
729399374,23cyclictest29566irq/58-eth012:10:163
728599376,28cyclictest27088-21kthreadcore12:00:192
728599371,33cyclictest7345-21grep11:25:262
728599371,33cyclictest7345-21grep11:25:252
728599371,32cyclictest8783-21/usr/sbin/munin07:45:252
728599371,32cyclictest29497-21/usr/sbin/munin08:40:282
728599371,32cyclictest25564-21kthreadcore08:35:202
728599371,32cyclictest23529-21sort09:30:012
728599371,31cyclictest0-21swapper/208:55:142
728599371,31cyclictest0-21swapper/208:00:112
728599371,28cyclictest18099-21tr10:30:182
728599371,28cyclictest15290-21fschecks_count12:35:192
728599371,27cyclictest11246-21fschecks_time10:05:172
728599370,20cyclictest5049-21ssh09:39:082
727799373,30cyclictest15062-21ssh12:15:131
729399362,22cyclictest29566irq/58-eth011:30:133
729399361,22cyclictest29566irq/58-eth011:35:153
728599362,30cyclictest26818-21apt-get11:40:152
728599361,26cyclictest12085-21sh09:20:222
728599360,31cyclictest26346irq/40-dwc2_hso07:15:122
729399355,22cyclictest29566irq/58-eth010:30:133
728599352,27cyclictest9039-21kthreadcore10:45:212
728599351,31cyclictest6736-21acpi08:50:132
728599351,31cyclictest6711-21ssh10:44:472
728599351,31cyclictest1897-21/usr/sbin/munin11:45:292
728599351,30cyclictest4025-21irqrtprio10:00:172
728599351,30cyclictest18531-21ntp_states08:25:262
727799350,28cyclictest28004-21ssh09:55:121
726999352,4cyclictest131rcu_preempt10:20:130
729399341,3cyclictest0-21swapper/311:55:143
729399341,3cyclictest0-21swapper/308:30:123
728599341,2cyclictest0-21swapper/212:05:182
728599341,20cyclictest0-21swapper/209:13:462
728599340,26cyclictest30123-21ssh10:15:282
727799342,23cyclictest9809-21ssh11:50:151
728599337,23cyclictest28664-21irqcore11:20:172
728599331,28cyclictest5765-21grep08:45:312
728599331,28cyclictest5765-21grep08:45:312
728599330,30cyclictest31051-21sort12:25:132
728599330,30cyclictest31051-21sort12:25:132
728599330,29cyclictest31560-21sshd07:35:152
728599330,29cyclictest31560-21sshd07:35:152
726999337,10cyclictest31571-21grep11:20:210
7269993323,7cyclictest12-21ksoftirqd/010:55:140
728599322,26cyclictest24879-21grep09:05:282
7277993214,11cyclictest0-21swapper/110:00:131
726999326,20cyclictest151rcuc/010:35:140
726999326,20cyclictest151rcuc/010:35:140
726999323,16cyclictest12-21ksoftirqd/012:30:130
726999321,21cyclictest12-21ksoftirqd/007:40:130
729399313,16cyclictest29566irq/58-eth011:00:143
729399313,16cyclictest29566irq/58-eth011:00:143
729399311,25cyclictest0-21swapper/311:50:153
728599310,28cyclictest14008-21irqcore07:50:152
728599310,26cyclictest26346irq/40-dwc2_hso07:55:192
727799312,22cyclictest25757-21grep10:35:221
727799312,22cyclictest25757-21grep10:35:221
729399302,2cyclictest0-21swapper/309:20:133
729399301,4cyclictest0-21swapper/310:35:143
729399301,4cyclictest0-21swapper/310:35:143
729399301,23cyclictest0-21swapper/310:50:143
728599301,24cyclictest7304-21sort07:10:012
727799302,20cyclictest16406-21ssh10:30:131
726999309,14cyclictest12-21ksoftirqd/011:30:130
729399292,6cyclictest0-21swapper/307:55:183
729399292,22cyclictest0-21swapper/309:45:143
729399291,5cyclictest0-21swapper/312:20:143
729399291,4cyclictest0-21swapper/311:15:153
729399291,24cyclictest0-21swapper/308:40:153
727799292,24cyclictest31414-21ssh11:00:211
727799292,24cyclictest31414-21ssh11:00:211
727799292,15cyclictest21925-21sshd07:25:131
726999293,20cyclictest5551-21ssh11:05:160
726999292,21cyclictest31423-21kthreadcore11:00:210
726999292,21cyclictest31423-21kthreadcore11:00:210
726999292,20cyclictest15061-21ssh12:15:130
729399282,2cyclictest0-21swapper/308:55:153
729399282,23cyclictest0-21swapper/311:05:143
7293992815,8cyclictest29566irq/58-eth012:00:133
727799282,11cyclictest3876-21kthreadcore11:45:211
727799281,18cyclictest11369-21ssh11:30:141
726999285,13cyclictest0-21swapper/008:10:150
726999284,14cyclictest14978-21ssh10:50:140
726999282,15cyclictest7164-21ssh10:45:130
729399272,18cyclictest0-21swapper/311:11:223
727799274,17cyclictest7648-21kworker/u8:112:30:141
727799272,22cyclictest913-21cut10:40:211
727799272,20cyclictest17627-21cut10:50:211
727799272,16cyclictest0-21swapper/112:20:151
729399262,18cyclictest0-21swapper/310:59:213
729399261,4cyclictest0-21swapper/312:15:143
729399261,4cyclictest0-21swapper/310:15:143
729399261,4cyclictest0-21swapper/310:10:143
729399261,3cyclictest0-21swapper/309:30:133
729399261,3cyclictest0-21swapper/308:10:143
729399261,3cyclictest0-21swapper/307:40:133
729399261,2cyclictest0-21swapper/307:35:153
729399261,2cyclictest0-21swapper/307:35:153
729399260,3cyclictest0-21swapper/309:50:133
727799269,12cyclictest31348-1kworker/u9:111:20:111
727799261,16cyclictest21-21ksoftirqd/109:35:141
726999262,9cyclictest0-21swapper/011:11:000
726999262,14cyclictest12998-21sh09:45:130
726999261,19cyclictest11051-21ssh10:05:150
729399251,6cyclictest0-21swapper/310:45:213
729399251,4cyclictest0-21swapper/312:30:133
729399251,3cyclictest0-21swapper/311:29:273
729399251,3cyclictest0-21swapper/311:29:273
729399251,3cyclictest0-21swapper/308:35:143
729399251,2cyclictest0-21swapper/310:45:013
729399242,3cyclictest0-21swapper/307:20:133
729399241,6cyclictest0-21swapper/312:27:393
729399241,6cyclictest0-21swapper/312:27:393
729399241,5cyclictest0-21swapper/309:55:133
729399241,4cyclictest0-21swapper/308:50:153
729399241,3cyclictest0-21swapper/312:35:013
729399241,3cyclictest0-21swapper/310:25:143
729399241,3cyclictest0-21swapper/310:05:023
729399241,3cyclictest0-21swapper/310:00:573
729399241,3cyclictest0-21swapper/309:40:123
729399241,3cyclictest0-21swapper/308:25:133
729399241,3cyclictest0-21swapper/308:15:123
729399241,3cyclictest0-21swapper/308:05:213
729399241,3cyclictest0-21swapper/308:00:273
729399241,2cyclictest0-21swapper/311:40:153
729399241,20cyclictest0-21swapper/311:45:293
727799248,8cyclictest7258-21cyclictest10:45:141
726999248,4cyclictest12-21ksoftirqd/009:35:040
726999243,16cyclictest151rcuc/009:50:120
726999242,14cyclictest12-21ksoftirqd/011:25:110
726999242,14cyclictest12-21ksoftirqd/011:25:110
7269992417,4cyclictest12-21ksoftirqd/007:35:150
7269992417,4cyclictest12-21ksoftirqd/007:35:150
729399234,16cyclictest29666irq/59-eth012:06:223
729399231,3cyclictest0-21swapper/309:05:133
729399231,3cyclictest0-21swapper/307:25:273
729399231,2cyclictest0-21swapper/309:25:543
729399231,2cyclictest0-21swapper/309:00:243
729399231,2cyclictest0-21swapper/307:50:263
729399231,2cyclictest0-21swapper/307:45:253
727799233,15cyclictest3549-21ssh11:25:131
727799233,15cyclictest3549-21ssh11:25:131
7277992311,8cyclictest9107-21ntpq11:05:261
727799231,17cyclictest1928-21apt-get08:45:151
727799231,17cyclictest1928-21apt-get08:45:151
727799231,17cyclictest10282-21ls08:20:131
727799231,16cyclictest9525-21ssh09:40:371
726999232,16cyclictest842-21munin-node11:35:150
7269992312,3cyclictest12-21ksoftirqd/010:14:500
7269992310,4cyclictest12-21ksoftirqd/011:42:400
729399221,3cyclictest0-21swapper/308:45:313
729399221,3cyclictest0-21swapper/308:45:313
729399221,3cyclictest0-21swapper/307:30:263
729399221,3cyclictest0-21swapper/307:10:173
729399221,2cyclictest0-21swapper/308:20:153
727799222,14cyclictest0-21swapper/107:50:101
727799221,16cyclictest22664-21ssh11:35:271
726999222,12cyclictest318-21df12:05:160
7269992213,5cyclictest0-21swapper/010:25:160
726999221,16cyclictest28021-21sh09:55:140
729399211,3cyclictest0-21swapper/307:15:123
727799217,4cyclictest4481-21irqrtprio07:40:181
727799212,17cyclictest20109-21ls08:30:161
727799211,15cyclictest1383-21cut09:15:171
727799211,14cyclictest26990-21ls07:30:181
726999219,5cyclictest0-21swapper/011:52:230
726999216,8cyclictest0-21swapper/012:23:280
726999213,12cyclictest0-21swapper/008:50:350
726999211,10cyclictest0-21swapper/009:16:480
7269992110,9cyclictest0-21swapper/010:01:140
726999210,18cyclictest8983-21cut07:45:160
727799205,12cyclictest14562-21rm11:10:231
727799203,11cyclictest0-21swapper/107:10:121
727799202,12cyclictest17542-21ssh11:55:141
7277992012,5cyclictest13009-21kthreadcore10:05:231
726999209,6cyclictest0-21swapper/008:45:160
726999209,6cyclictest0-21swapper/008:45:160
7269992013,5cyclictest0-21swapper/010:19:100
7269992012,5cyclictest26976-21kthreadcore09:10:210
7269992011,4cyclictest12-21ksoftirqd/009:00:150
726999201,14cyclictest12-21ksoftirqd/007:10:120
7269992010,8cyclictest0-21swapper/010:33:140
7269992010,7cyclictest12375-21sed09:20:220
7269992010,4cyclictest0-21swapper/012:26:150
7269992010,4cyclictest0-21swapper/012:26:150
727799198,8cyclictest8970-21cpuspeed10:25:161
727799196,10cyclictest27846-21sed09:55:021
727799195,3cyclictest131rcu_preempt07:25:011
727799191,16cyclictest23071-21awk08:00:011
727799191,12cyclictest24502-21diskmemload11:15:291
7277991910,6cyclictest0-21swapper/110:15:181
726999192,2cyclictest131rcu_preempt11:49:330
726999192,12cyclictest15137-21ssh12:35:170
726999191,8cyclictest0-21swapper/012:12:020
726999191,15cyclictest16200-21apt-get09:25:130
726999191,10cyclictest28555-21cut08:05:210
726999190,13cyclictest13945-21ls08:20:270
729399181,3cyclictest0-21swapper/307:10:013
727799187,8cyclictest3969-21sshd08:10:251
727799183,9cyclictest21-21ksoftirqd/112:35:161
727799183,12cyclictest31865-21tr08:40:231
727799182,14cyclictest26853-21kthreadcore09:30:221
727799181,14cyclictest9254-21cut12:10:221
7277991811,4cyclictest12392-21sendmail_mailtr07:45:261
727799181,13cyclictest25937-21cmp09:10:011
726999189,4cyclictest131rcu_preempt08:15:120
726999189,2cyclictest12-21ksoftirqd/012:03:520
726999181,2cyclictest131rcu_preempt07:55:330
7269991811,5cyclictest0-21swapper/008:35:280
726999181,10cyclictest0-21swapper/009:32:240
727799174,10cyclictest16186-21grep07:15:281
727799172,12cyclictest25049-21tr08:00:191
727799172,12cyclictest16212-21ssh09:25:131
727799171,13cyclictest24666-21/usr/sbin/munin08:35:251
727799171,13cyclictest1856-21tr07:35:241
727799171,13cyclictest1856-21tr07:35:241
727799171,13cyclictest12993-21apt-get09:45:131
727799171,12cyclictest6801-21sort08:50:151
727799171,11cyclictest18739-21/usr/sbin/munin10:10:291
727799171,10cyclictest18754-21ls08:25:281
7277991710,5cyclictest16168-21cron09:00:011
726999179,5cyclictest30795-21kthreadcore08:40:210
726999177,7cyclictest12-21ksoftirqd/007:25:160
726999175,9cyclictest23427-21fschecks_count08:00:150
726999173,11cyclictest17283-21fschecks_time07:20:170
726999173,11cyclictest15274-21kthreadcore07:15:210
726999171,8cyclictest0-21swapper/008:25:190
726999171,8cyclictest0-21swapper/007:54:460
726999171,14cyclictest27191-21irqcore07:30:210
727799163,10cyclictest1695-21irqcore10:20:191
727799162,10cyclictest0-21swapper/108:05:171
727799161,7cyclictest16769-21irqcore09:00:181
727799161,11cyclictest5552-21sort08:15:151
726999169,4cyclictest20187-21irqcore08:30:170
726999162,9cyclictest0-21swapper/009:05:140
726999161,11cyclictest15004-21ntp_192.168.11408:55:270
726999152,11cyclictest7299-21latency_hist07:10:010
727799141,10cyclictest7300-21grep07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional