You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 08:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Nov 06, 2025 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1141599626,11cyclictest12-21ksoftirqd/022:00:120
11437996032,18cyclictest131rcu_preempt00:25:142
1143799602,37cyclictest0-21swapper/219:35:122
11437995524,12cyclictest131rcu_preempt23:25:142
1143799511,41cyclictest9127-21ssh00:30:122
1143799511,26cyclictest26652-21ssh22:05:132
1142599471,39cyclictest30493-21ssh21:10:121
1144699462,28cyclictest29566irq/58-eth022:45:173
11437994619,18cyclictest0-21swapper/219:50:162
11446994214,18cyclictest29566irq/58-eth000:05:133
1143799412,29cyclictest0-21swapper/221:25:132
1142599412,27cyclictest4095-21ssh22:30:111
1144699402,23cyclictest29566irq/58-eth000:25:143
1143799401,33cyclictest0-21swapper/200:10:142
1142599406,24cyclictest21-21ksoftirqd/100:25:141
1142599401,24cyclictest23606-21ssh21:25:121
1143799391,27cyclictest0-21swapper/219:45:122
1142599394,29cyclictest8180-21ssh21:35:131
1142599394,29cyclictest8180-21ssh21:35:131
1144699361,19cyclictest29566irq/58-eth022:15:143
1142599361,23cyclictest21-21ksoftirqd/120:35:131
1144699352,18cyclictest29566irq/58-eth022:00:123
1143799357,15cyclictest0-21swapper/221:15:142
1142599351,29cyclictest9701-21ssh21:55:161
1144699344,19cyclictest29566irq/58-eth022:10:153
1141599341,5cyclictest0-21swapper/019:35:100
1144699331,18cyclictest29566irq/58-eth023:40:123
1143799334,19cyclictest26346irq/40-dwc2_hso21:10:112
1143799331,22cyclictest2718-21ssh22:10:152
11415993327,3cyclictest30108-21diskmemload00:05:120
11446993223,5cyclictest29566irq/58-eth021:20:123
1141599322,27cyclictest0-21swapper/020:40:150
1141599315,19cyclictest17987-21ssh23:55:150
1141599311,20cyclictest15111-21ssh21:20:130
1144699301,14cyclictest29566irq/58-eth021:00:143
1143799303,20cyclictest26346irq/40-dwc2_hso22:35:132
1141599301,7cyclictest22855-21apt-get19:55:110
1141599301,25cyclictest19611-21ssh22:20:090
1144699291,15cyclictest29566irq/58-eth021:15:133
1143799296,12cyclictest26346irq/40-dwc2_hso21:05:162
1142599291,23cyclictest26355-21ssh00:00:121
1141599298,10cyclictest12-21ksoftirqd/022:44:200
1143799283,17cyclictest26346irq/40-dwc2_hso19:25:142
1143799283,16cyclictest26346irq/40-dwc2_hso19:40:092
1143799281,20cyclictest23662-21ssh23:20:112
11425992814,8cyclictest21-21ksoftirqd/122:40:121
1141599282,22cyclictest12536-21ssh22:35:130
1144699275,19cyclictest29566irq/58-eth023:00:133
1144699271,13cyclictest29566irq/58-eth021:30:123
1143799273,9cyclictest0-21swapper/223:50:162
1143799272,10cyclictest0-21swapper/220:25:162
1143799271,21cyclictest16711-21ssh21:40:142
1141599279,3cyclictest131rcu_preempt00:00:130
1141599275,2cyclictest131rcu_preempt00:30:120
1144699263,12cyclictest29566irq/58-eth023:30:123
1143799264,15cyclictest26346irq/40-dwc2_hso20:10:142
1143799262,9cyclictest0-21swapper/219:55:032
1143799262,19cyclictest0-21swapper/220:40:162
1143799262,16cyclictest26346irq/40-dwc2_hso22:15:142
1143799261,20cyclictest0-21swapper/220:15:132
1142599269,9cyclictest11404-21cyclictest23:30:111
1141599268,3cyclictest131rcu_preempt23:25:140
1141599267,4cyclictest131rcu_preempt21:30:190
1141599265,12cyclictest0-21swapper/020:50:130
1141599263,20cyclictest16702-21ssh21:40:130
1141599262,4cyclictest12-21ksoftirqd/019:30:130
11415992621,4cyclictest151rcuc/000:15:150
1143799255,7cyclictest131rcu_preempt23:30:122
1143799255,11cyclictest26346irq/40-dwc2_hso00:00:122
11437992517,5cyclictest19819-21ssh00:15:222
11437992513,5cyclictest13765-21ssh21:36:312
11437992513,5cyclictest13765-21ssh21:36:312
1143799251,17cyclictest7866-21ls21:35:002
1143799251,15cyclictest21074-21ssh22:40:122
1142599255,2cyclictest21-21ksoftirqd/122:17:141
1142599251,16cyclictest842-21munin-node23:35:161
1141599258,12cyclictest12-21ksoftirqd/023:40:110
1141599254,5cyclictest12-21ksoftirqd/021:25:210
1141599251,18cyclictest9700-21ssh21:55:160
1142599245,13cyclictest15091-21ssh21:20:121
1142599244,12cyclictest629-21Async14:00:481
1142599241,16cyclictest10261-21ssh00:10:151
1141599249,4cyclictest12-21ksoftirqd/023:19:240
1141599248,4cyclictest131rcu_preempt19:15:250
1141599243,6cyclictest131rcu_preempt22:10:000
11415992419,3cyclictest0-21swapper/022:15:150
11415992413,3cyclictest12-21ksoftirqd/022:55:010
11415992411,7cyclictest0-21swapper/021:39:280
11415992411,7cyclictest0-21swapper/021:39:280
1141599241,16cyclictest8110-21ssh23:30:130
11415992410,10cyclictest0-21swapper/023:50:160
1143799236,11cyclictest26346irq/40-dwc2_hso21:45:142
1143799233,16cyclictest26346irq/40-dwc2_hso00:35:132
1143799233,16cyclictest26346irq/40-dwc2_hso00:35:132
1143799233,13cyclictest26346irq/40-dwc2_hso19:20:122
1143799232,16cyclictest26346irq/40-dwc2_hso21:00:142
1143799232,14cyclictest26346irq/40-dwc2_hso00:05:112
1143799231,13cyclictest18781-21ssh21:20:252
1143799230,8cyclictest0-21swapper/222:45:202
1142599237,3cyclictest131rcu_preempt23:45:001
1142599233,15cyclictest21921-21ssh00:15:291
1142599233,12cyclictest629-21Async14:00:481
11425992311,4cyclictest0-21swapper/120:40:181
1141599238,4cyclictest131rcu_preempt21:05:220
1141599237,8cyclictest12-21ksoftirqd/021:45:080
1141599234,3cyclictest131rcu_preempt21:17:260
1141599233,5cyclictest131rcu_preempt23:45:260
1141599233,5cyclictest131rcu_preempt23:45:260
1141599232,3cyclictest131rcu_preempt20:15:130
11415992312,4cyclictest12-21ksoftirqd/019:20:250
1141599231,18cyclictest27656-21find20:00:130
1144699229,8cyclictest29566irq/58-eth023:15:143
1144699222,16cyclictest33-21ksoftirqd/300:15:153
11446992214,5cyclictest29566irq/58-eth000:35:273
11446992214,5cyclictest29566irq/58-eth000:35:273
1143799226,9cyclictest32537-21sshd00:22:592
1143799225,13cyclictest26346irq/40-dwc2_hso20:35:122
1143799222,7cyclictest30950-21ls20:00:232
1143799222,13cyclictest26346irq/40-dwc2_hso20:20:122
1143799221,14cyclictest24180-21find20:30:122
1143799221,11cyclictest6739-21ssh23:10:152
1142599228,3cyclictest131rcu_preempt21:30:191
1142599222,15cyclictest26190-21ssh00:20:151
1142599222,14cyclictest0-21swapper/119:20:241
1142599221,3cyclictest131rcu_preempt22:04:381
1142599221,15cyclictest6312-21uniq20:45:151
1142599221,15cyclictest6312-21uniq20:45:151
11425992210,10cyclictest0-21swapper/121:44:051
1142599220,2cyclictest131rcu_preempt00:30:381
1141599228,3cyclictest131rcu_preempt22:55:140
1141599227,11cyclictest151rcuc/000:28:110
1141599224,5cyclictest32061-21ssh23:05:220
1141599224,5cyclictest12-21ksoftirqd/022:45:240
1141599224,4cyclictest131rcu_preempt00:10:150
1141599222,3cyclictest131rcu_preempt21:10:230
1141599222,15cyclictest23661-21sh23:20:110
11415992212,4cyclictest0-21swapper/022:25:130
11415992210,4cyclictest12-21ksoftirqd/020:07:070
1144699218,8cyclictest29566irq/58-eth023:20:223
1143799219,7cyclictest32459-21kthreadcore23:05:222
1143799217,11cyclictest23674-21kthreadcore23:00:202
1143799212,16cyclictest16755-21irqcore19:15:182
1143799211,9cyclictest18836-21irqrtprio22:00:182
11437992116,3cyclictest23217-21sh22:20:212
11437992111,8cyclictest0-21swapper/221:50:112
1143799211,11cyclictest4062-21acpi22:30:102
1142599219,3cyclictest21-21ksoftirqd/119:30:001
1142599213,8cyclictest8226-21apt-get19:40:091
1142599213,8cyclictest131rcu_preempt22:22:391
1142599213,14cyclictest9453-21apt20:15:011
1142599212,2cyclictest131rcu_preempt23:12:201
1142599211,3cyclictest131rcu_preempt20:00:291
1142599211,15cyclictest201rcuc/123:05:151
1142599211,12cyclictest21-21ksoftirqd/120:20:121
11425992110,9cyclictest0-21swapper/100:09:231
11425992110,7cyclictest2869-21fschecks_count22:10:171
1141599219,6cyclictest12-21ksoftirqd/019:50:160
1141599219,3cyclictest0-21swapper/000:21:230
1141599218,7cyclictest12-21ksoftirqd/021:50:100
1141599217,8cyclictest12-21ksoftirqd/022:30:550
1141599214,9cyclictest12-21ksoftirqd/023:12:530
1141599214,3cyclictest131rcu_preempt19:10:220
1141599214,2cyclictest131rcu_preempt23:35:440
1141599213,6cyclictest12-21ksoftirqd/022:10:170
1141599213,14cyclictest22347-21ssh23:00:130
1144699202,12cyclictest29566irq/58-eth021:55:163
1144699201,9cyclictest29566irq/58-eth022:35:123
11446992012,5cyclictest29566irq/58-eth022:30:123
11446992011,6cyclictest29566irq/58-eth022:55:153
11446992011,5cyclictest29566irq/58-eth021:35:133
11446992011,5cyclictest29566irq/58-eth021:35:133
11446992010,8cyclictest0-21swapper/323:45:263
11446992010,8cyclictest0-21swapper/323:45:263
11446992010,8cyclictest0-21swapper/322:29:593
1143799206,6cyclictest27-21ksoftirqd/223:55:232
1143799204,8cyclictest15571-21awk20:50:322
1143799202,9cyclictest25698-21irqrtprio23:40:172
1143799201,9cyclictest2114-21wc19:30:262
1143799201,16cyclictest6428-21sed20:45:162
1143799201,16cyclictest6428-21sed20:45:162
1143799201,15cyclictest24965-21ssh23:40:012
1143799201,11cyclictest29301-21cut22:25:202
1143799201,10cyclictest15670-21kthreadcore22:55:232
1143799201,10cyclictest1554-21ps23:45:182
1143799201,10cyclictest1554-21ps23:45:182
1143799200,10cyclictest11892-21cut19:10:172
1142599209,8cyclictest22891-21irqrtprio23:00:181
1142599209,4cyclictest131rcu_preempt23:45:221
1142599209,4cyclictest131rcu_preempt23:45:221
1142599209,3cyclictest0-21swapper/121:54:301
1142599208,7cyclictest131rcu_preempt19:45:211
1142599208,4cyclictest21-21ksoftirqd/119:55:111
1142599208,3cyclictest25658-21kthreadcore23:20:181
1142599207,9cyclictest13924-21ssh22:55:161
1142599202,4cyclictest131rcu_preempt23:18:121
1142599201,2cyclictest131rcu_preempt23:57:291
1142599201,2cyclictest131rcu_preempt23:28:081
1142599201,2cyclictest131rcu_preempt19:30:141
11425992011,6cyclictest0-21swapper/119:55:011
1142599201,14cyclictest26662-21ssh22:05:131
1142599200,17cyclictest16439-21ls22:35:281
1141599209,3cyclictest131rcu_preempt20:45:210
1141599209,3cyclictest131rcu_preempt20:45:210
1141599204,6cyclictest12-21ksoftirqd/020:30:160
1141599203,4cyclictest131rcu_preempt20:25:250
11415992013,4cyclictest12-21ksoftirqd/020:20:290
11415992012,5cyclictest0-21swapper/020:55:170
1144699199,6cyclictest0-21swapper/322:53:433
1144699191,12cyclictest0-21swapper/320:20:273
1144699191,11cyclictest0-21swapper/319:25:013
11446991910,7cyclictest0-21swapper/321:45:293
11446991910,6cyclictest29566irq/58-eth022:05:143
1143799196,5cyclictest6626-21kthreadcore22:50:182
1143799192,9cyclictest358-21irqcore20:05:182
1143799192,12cyclictest0-21swapper/223:15:132
1143799191,9cyclictest16012-21cut20:55:162
1143799191,9cyclictest10502-21kthreadcore21:55:212
1142599199,7cyclictest31950-21mailstats22:25:281
1142599198,8cyclictest131rcu_preempt21:00:201
1142599198,5cyclictest22944-21rm00:39:111
1142599198,5cyclictest22944-21rm00:39:111
1142599193,4cyclictest131rcu_preempt22:45:231
1142599192,3cyclictest131rcu_preempt21:15:211
1142599192,2cyclictest131rcu_preempt21:46:001
1142599192,14cyclictest9655-21apt-get23:50:151
11425991914,3cyclictest30103-21cron21:10:001
1142599191,2cyclictest131rcu_preempt22:53:281
11425991912,4cyclictest0-21swapper/120:25:211
1142599190,15cyclictest32582-21uniq20:05:161
1141599199,4cyclictest12-21ksoftirqd/019:40:210
1141599197,3cyclictest131rcu_preempt20:10:280
1141599196,3cyclictest131rcu_preempt21:05:010
1141599194,4cyclictest12-21ksoftirqd/019:25:220
1141599193,3cyclictest131rcu_preempt19:45:120
1141599191,3cyclictest131rcu_preempt20:35:290
11415991910,7cyclictest21841-21sshd00:36:590
11415991910,7cyclictest21841-21sshd00:36:590
1144699189,6cyclictest29566irq/58-eth021:10:123
1144699187,7cyclictest0-21swapper/323:35:253
1144699186,9cyclictest0-21swapper/323:30:013
1144699182,10cyclictest0-21swapper/320:55:143
1144699182,10cyclictest0-21swapper/320:00:153
1144699182,10cyclictest0-21swapper/319:30:133
11446991812,4cyclictest0-21swapper/300:20:413
11446991811,4cyclictest0-21swapper/321:25:243
1144699181,13cyclictest0-21swapper/322:20:203
1144699181,13cyclictest0-21swapper/319:35:203
1144699181,13cyclictest0-21swapper/319:10:273
1144699181,10cyclictest0-21swapper/319:45:033
11446991810,6cyclictest0-21swapper/323:58:383
11446991810,6cyclictest0-21swapper/322:40:163
1142599187,4cyclictest131rcu_preempt19:35:241
1142599183,10cyclictest0-21swapper/120:30:141
1142599182,13cyclictest13521-21unixbench_multi20:15:271
1144699179,6cyclictest0-21swapper/323:10:513
1144699179,6cyclictest0-21swapper/323:07:503
1144699179,6cyclictest0-21swapper/300:01:033
1144699178,6cyclictest0-21swapper/321:40:313
1144699178,3cyclictest0-21swapper/321:51:373
11446991711,4cyclictest0-21swapper/323:52:453
1144699171,12cyclictest0-21swapper/320:25:183
1144699171,11cyclictest0-21swapper/320:05:283
1144699171,10cyclictest0-21swapper/320:45:133
1144699171,10cyclictest0-21swapper/320:45:133
1144699171,10cyclictest0-21swapper/319:40:113
11446991710,4cyclictest29566irq/58-eth000:30:183
1143799171,7cyclictest11458-21latency_hist19:10:002
1142599177,7cyclictest15923-21diskstats20:55:151
1142599172,11cyclictest0-21swapper/119:10:131
1144699161,11cyclictest0-21swapper/320:15:193
1144699161,10cyclictest0-21swapper/320:10:153
11446991610,4cyclictest0-21swapper/300:10:113
1141599163,10cyclictest11489-21latency_hist19:10:000
1144699157,5cyclictest29566irq/58-eth019:55:123
1144699152,9cyclictest0-21swapper/320:30:143
1144699152,10cyclictest0-21swapper/319:50:163
1144699151,9cyclictest0-21swapper/321:05:163
1144699151,10cyclictest0-21swapper/320:40:163
1144699149,3cyclictest0-21swapper/319:15:153
1144699140,10cyclictest0-21swapper/319:25:143
1142599144,7cyclictest11475-21cat19:10:001
1144699131,8cyclictest0-21swapper/320:50:143
1144699131,7cyclictest0-21swapper/320:35:143
114469991,6cyclictest0-21swapper/319:10:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional