You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 10:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Fri Nov 14, 2025 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1056299576,3cyclictest131rcu_preempt22:30:141
1056299573,40cyclictest20768-21ssh00:00:111
1055399572,34cyclictest15771-21ssh22:00:140
1055399541,40cyclictest31487-21ssh21:50:140
1056299512,37cyclictest8232-21ssh22:15:121
10553994529,11cyclictest0-21swapper/022:35:140
1058099443,22cyclictest29566irq/58-eth000:00:143
1057199442,33cyclictest0-21swapper/223:30:172
10580994014,19cyclictest29566irq/58-eth000:25:163
1055399405,21cyclictest3318-21ssh00:30:150
1055399403,28cyclictest29152-21ssh00:05:150
10571993914,11cyclictest876-21sh22:30:142
1056299391,29cyclictest13984-21ssh21:20:121
1055399395,26cyclictest24944-21ssh22:25:120
1056299383,22cyclictest20265-21ssh23:40:151
1056299383,22cyclictest20265-21ssh23:40:141
1056299381,5cyclictest131rcu_preempt22:55:141
1058099374,22cyclictest29566irq/58-eth023:50:163
1056299371,30cyclictest4394-21sh23:50:171
1055399376,21cyclictest12-21ksoftirqd/020:00:130
10553993613,18cyclictest12-21ksoftirqd/022:10:140
1056299355,18cyclictest28319-21ssh00:25:181
1058099333,21cyclictest0-21swapper/321:30:163
1056299334,9cyclictest10883-21ssh23:15:141
1055399335,16cyclictest27722-21ssh23:25:170
1057199322,21cyclictest4382-21ssh23:50:162
1056299323,18cyclictest21-21ksoftirqd/120:55:131
10553993215,13cyclictest22364-21sh21:25:110
1058099316,17cyclictest33-21ksoftirqd/323:20:123
1057199314,18cyclictest10540-21cyclictest21:25:132
10580993023,4cyclictest29566irq/58-eth000:30:153
10571993011,14cyclictest261rcuc/222:10:142
10553993016,6cyclictest0-21swapper/021:05:140
10553993016,6cyclictest0-21swapper/021:05:130
1058099297,9cyclictest0-21swapper/322:15:113
1057199292,19cyclictest0-21swapper/219:40:152
1058099283,20cyclictest29566irq/58-eth022:25:113
1057199289,7cyclictest27-21ksoftirqd/219:45:132
1057199289,7cyclictest27-21ksoftirqd/219:45:122
1057199282,21cyclictest27778-21ssh00:24:432
10571992814,9cyclictest3333-21ssh00:30:152
1057199281,21cyclictest30707-21ssh21:30:172
10562992819,4cyclictest0-21swapper/122:20:131
1055399283,17cyclictest28510-21ssh23:45:140
1055399281,23cyclictest2489-21ssh23:10:130
1057199272,19cyclictest0-21swapper/220:50:152
1057199272,19cyclictest0-21swapper/219:10:132
1057199272,10cyclictest0-21swapper/220:20:132
1056299274,12cyclictest21-21ksoftirqd/120:50:151
1056299272,18cyclictest24571-21sh22:43:431
1056299271,20cyclictest5614-21rm21:15:151
1055399272,20cyclictest14919-21ssh21:40:170
1055399272,19cyclictest22511-21sh22:40:430
1058099268,13cyclictest0-21swapper/323:45:143
1056299264,10cyclictest629-21Async14:00:481
1055399263,3cyclictest131rcu_preempt20:45:150
1058099257,12cyclictest29566irq/58-eth023:15:153
1058099254,13cyclictest29566irq/58-eth023:55:143
10580992516,6cyclictest29566irq/58-eth022:10:153
10580992514,8cyclictest0-21swapper/321:15:133
1057199252,8cyclictest0-21swapper/220:15:172
10571992514,7cyclictest17490-21/usr/sbin/munin22:40:192
1057199251,16cyclictest28496-21ssh23:45:142
10571992510,7cyclictest23447-21uptime23:20:302
1056299251,3cyclictest131rcu_preempt23:00:131
1055399259,4cyclictest12-21ksoftirqd/022:20:010
1055399258,4cyclictest12-21ksoftirqd/022:30:200
1055399253,5cyclictest12-21ksoftirqd/000:15:010
1055399253,16cyclictest0-21swapper/000:20:180
1058099246,14cyclictest0-21swapper/321:55:133
1058099245,10cyclictest0-21swapper/322:30:153
1058099244,15cyclictest0-21swapper/300:05:163
1057199241,19cyclictest23516-21cut19:55:202
10562992414,7cyclictest31369-21ssh23:45:241
1055399249,5cyclictest12-21ksoftirqd/023:40:430
1055399249,5cyclictest12-21ksoftirqd/023:40:420
1055399249,4cyclictest12-21ksoftirqd/021:35:230
10553992419,3cyclictest0-21swapper/023:00:120
10580992314,5cyclictest0-21swapper/322:35:143
1058099231,15cyclictest0-21swapper/322:55:143
1057199238,6cyclictest447-21cron22:30:012
1057199237,13cyclictest0-21swapper/222:50:142
1057199233,13cyclictest26346irq/40-dwc2_hso20:55:132
1056299238,11cyclictest4832-21ssh23:30:231
1056299231,17cyclictest0-21swapper/122:28:431
1055399237,8cyclictest12-21ksoftirqd/000:19:090
1055399237,8cyclictest12-21ksoftirqd/000:19:090
1055399236,8cyclictest12-21ksoftirqd/022:55:170
1055399231,7cyclictest12-21ksoftirqd/019:50:200
1055399231,20cyclictest29313-21diskmemload23:17:430
1058099224,13cyclictest0-21swapper/321:50:163
1058099221,6cyclictest0-21swapper/322:20:143
1057199229,5cyclictest1609-21kthreadcore20:40:232
1057199222,17cyclictest16651-21ssh22:20:132
1057199222,16cyclictest26346irq/40-dwc2_hso00:05:162
1057199222,15cyclictest0-21swapper/223:16:432
10571992212,8cyclictest0-21swapper/220:30:332
1057199221,17cyclictest31193-21cut21:10:222
1057199221,12cyclictest26700-21ssh23:05:142
1056299221,16cyclictest3547-21ssh22:10:271
1055399228,8cyclictest12-21ksoftirqd/022:24:350
1055399228,8cyclictest12-21ksoftirqd/021:34:270
1055399227,6cyclictest131rcu_preempt23:31:140
1055399227,4cyclictest12-21ksoftirqd/020:10:350
1055399227,4cyclictest12-21ksoftirqd/019:55:230
1055399222,10cyclictest0-21swapper/021:15:120
10580992111,6cyclictest29566irq/58-eth023:40:163
10580992111,6cyclictest29566irq/58-eth023:40:163
10580992110,8cyclictest0-21swapper/322:40:133
1057199219,9cyclictest25020-21sshd19:23:422
1057199219,9cyclictest25020-21sshd19:23:412
1057199217,5cyclictest30387-21ps19:30:192
1057199215,11cyclictest0-21swapper/222:15:122
1057199213,9cyclictest30021-21latency_hist19:30:002
1057199213,15cyclictest26346irq/40-dwc2_hso23:55:152
1057199213,10cyclictest8634-21kthreadcore20:45:262
1057199213,10cyclictest10391-21kthreadcore21:55:222
1057199212,9cyclictest15246-21if_eth021:40:212
1057199212,15cyclictest26346irq/40-dwc2_hso00:00:132
1057199211,9cyclictest29745-21kthreadcore23:25:242
1057199211,9cyclictest24751-21ssh22:05:212
1057199211,16cyclictest5615-21sh21:15:152
1057199211,15cyclictest12687-21ssh00:15:112
1057199211,15cyclictest12687-21ssh00:15:102
1057199211,11cyclictest29974-21kthreadcore00:25:252
1057199211,11cyclictest15751-21grep19:15:172
1057199211,11cyclictest14611-21latency_hist21:40:022
1057199211,10cyclictest13386-21ssh23:35:232
10571992110,9cyclictest28284-21ssh21:45:532
1056299211,9cyclictest0-21swapper/123:25:241
1056299211,5cyclictest2222566ssh23:20:221
10562992113,5cyclictest16159-21ssh23:55:261
1056299211,10cyclictest10663-21kthreadcore21:55:221
1055399215,7cyclictest12-21ksoftirqd/000:27:420
1055399215,3cyclictest131rcu_preempt23:23:550
1055399214,2cyclictest131rcu_preempt22:07:490
1055399212,3cyclictest131rcu_preempt23:35:160
1055399212,12cyclictest842-21munin-node19:20:160
1055399212,12cyclictest842-21munin-node19:20:160
1055399211,17cyclictest31621-21cut20:05:190
10553992111,4cyclictest12-21ksoftirqd/021:58:470
1055399210,18cyclictest28072-21ls20:35:020
1058099209,6cyclictest0-21swapper/300:24:443
1058099203,11cyclictest29566irq/58-eth021:45:143
10580992012,5cyclictest29566irq/58-eth023:00:143
1058099201,14cyclictest0-21swapper/321:40:183
1057199209,8cyclictest0-21swapper/222:40:022
1057199202,15cyclictest30300-21kthreadcore20:35:272
1057199202,12cyclictest0-21swapper/222:55:142
1057199201,9cyclictest6930-21kthreadcore00:10:242
1057199201,9cyclictest2169-21kthreadcore20:05:252
1057199201,9cyclictest18419-21/usr/sbin/munin20:25:252
1057199201,7cyclictest0-21swapper/223:40:172
1057199201,7cyclictest0-21swapper/223:40:172
10571992013,5cyclictest15892-21ssh22:00:172
10571992011,7cyclictest23586-21ssh23:01:082
1057199201,15cyclictest3324-21kthreadcore19:35:192
1057199201,11cyclictest28249-21wc21:05:272
1057199201,11cyclictest28249-21wc21:05:272
1057199201,11cyclictest28157-21kthreadcore20:00:192
1057199201,10cyclictest31882-21irqcore21:50:202
1057199201,10cyclictest26119-21tune2fs22:45:202
1057199201,10cyclictest17418-21irqrtprio19:50:192
1057199201,10cyclictest14084-21df_inode21:20:162
1057199201,10cyclictest10677-21latency_hist23:15:022
1057199201,10cyclictest10494-21latency_hist19:10:012
1057199201,10cyclictest10494-21latency_hist19:10:012
10571992010,8cyclictest14014-21sh00:35:302
1056299208,3cyclictest29313-21diskmemload23:11:281
1056299204,9cyclictest0-21swapper/120:20:141
1056299201,9cyclictest13091-21find22:35:291
10562992012,5cyclictest4859-21ssh00:10:161
10562992012,5cyclictest30760-21ssh21:30:171
10562992011,7cyclictest6800-21sshd21:54:431
10562992011,6cyclictest19494-21ssh22:00:301
10562992011,6cyclictest0-21swapper/100:35:141
1055399209,5cyclictest131rcu_preempt22:54:050
1055399208,3cyclictest131rcu_preempt19:10:130
1055399207,8cyclictest12-21ksoftirqd/021:45:560
10553992014,4cyclictest0-21swapper/020:40:260
10553992013,5cyclictest0-21swapper/000:01:280
10553992012,6cyclictest2930-21sshd21:11:430
10553992012,5cyclictest0-21swapper/019:30:130
1055399201,15cyclictest12124-21apt-get19:45:120
1055399201,15cyclictest12124-21apt-get19:45:120
1055399201,14cyclictest13990-21ssh21:20:140
1058099199,6cyclictest29566irq/58-eth023:05:143
1058099196,10cyclictest0-21swapper/322:45:153
1058099193,14cyclictest0-21swapper/323:29:443
1058099192,11cyclictest0-21swapper/320:35:043
1058099192,10cyclictest0-21swapper/320:20:143
10580991911,6cyclictest0-21swapper/300:15:263
10580991911,6cyclictest0-21swapper/300:15:253
1058099191,14cyclictest0-21swapper/319:20:013
1058099191,12cyclictest0-21swapper/323:10:173
10580991910,6cyclictest0-21swapper/300:12:473
1057199192,14cyclictest4762-21kthreadcore20:10:242
1057199191,8cyclictest20089-21fschecks_time21:00:202
1056299192,13cyclictest22373-21ssh21:25:131
1056299191,9cyclictest0-21swapper/100:20:541
1056299191,8cyclictest0-21swapper/121:47:541
1056299191,14cyclictest1420-21munin-run22:50:011
1056299191,12cyclictest29313-21diskmemload21:10:141
1056299191,10cyclictest8516-21kthreadcore21:35:221
10562991910,6cyclictest11974-21ssh23:35:141
1055399198,5cyclictest0-21swapper/023:07:250
1055399198,4cyclictest151rcuc/019:35:160
1055399198,4cyclictest12-21ksoftirqd/019:25:230
1055399195,8cyclictest12-21ksoftirqd/022:45:020
1055399193,9cyclictest12-21ksoftirqd/023:56:300
10553991913,4cyclictest0-21swapper/020:25:020
1058099189,6cyclictest0-21swapper/322:50:553
1058099189,6cyclictest0-21swapper/321:25:123
1058099189,6cyclictest0-21swapper/321:20:383
1058099182,13cyclictest29566irq/58-eth020:10:183
1058099182,10cyclictest0-21swapper/321:00:053
1058099182,10cyclictest0-21swapper/320:30:163
1058099182,10cyclictest0-21swapper/320:15:043
1058099182,10cyclictest0-21swapper/320:05:163
1058099182,10cyclictest0-21swapper/319:45:143
1058099182,10cyclictest0-21swapper/319:45:143
1058099182,10cyclictest0-21swapper/319:20:163
1058099182,10cyclictest0-21swapper/319:20:163
1058099182,10cyclictest0-21swapper/319:10:143
1058099181,14cyclictest29566irq/58-eth023:35:163
10580991811,4cyclictest0-21swapper/323:30:353
1058099181,13cyclictest0-21swapper/300:40:013
1056299188,7cyclictest4143-21cut22:50:241
1056299188,7cyclictest26814-21cut22:05:231
1056299186,4cyclictest131rcu_preempt00:05:341
1056299182,3cyclictest131rcu_preempt20:35:171
1056299181,9cyclictest0-21swapper/123:06:091
1056299181,5cyclictest21-21ksoftirqd/121:40:261
10562991811,5cyclictest9841-21ssh00:34:341
1056299181,12cyclictest16163-21latency00:15:221
1056299181,12cyclictest16163-21latency00:15:221
10562991810,6cyclictest2209-21sshd19:31:421
1055399189,7cyclictest10284-21df20:50:170
1055399189,2cyclictest12-21ksoftirqd/023:54:510
1055399187,4cyclictest12-21ksoftirqd/019:40:180
1055399183,10cyclictest0-21swapper/019:15:040
1055399182,11cyclictest0-21swapper/021:00:040
1055399182,11cyclictest0-21swapper/000:35:160
1055399181,15cyclictest18615-21ls20:55:280
1055399181,14cyclictest27847-21apt-get20:35:040
1058099179,3cyclictest0-21swapper/322:04:383
1058099179,3cyclictest0-21swapper/321:37:503
1058099172,9cyclictest0-21swapper/320:50:163
1058099172,10cyclictest0-21swapper/322:05:143
1058099172,10cyclictest0-21swapper/319:30:153
1058099171,9cyclictest0-21swapper/321:05:143
1058099171,9cyclictest0-21swapper/321:05:143
1058099171,13cyclictest0-21swapper/321:10:303
1058099171,13cyclictest0-21swapper/320:40:193
1056299175,9cyclictest20623-21threads19:50:301
1056299175,5cyclictest27868-21gpgv20:35:011
1056299174,9cyclictest0-21swapper/120:15:161
1056299171,8cyclictest11895-21latency_hist19:45:011
1055399174,10cyclictest11062-21kthreadcore20:15:230
1058099162,9cyclictest0-21swapper/320:25:133
1058099162,10cyclictest0-21swapper/319:40:153
1058099161,9cyclictest0-21swapper/319:55:133
1058099161,12cyclictest0-21swapper/319:25:183
1056299167,7cyclictest623-21df_inode20:40:171
1056299166,7cyclictest28620-21kthreadcore20:00:201
1056299166,7cyclictest21-21ksoftirqd/120:10:241
1056299163,11cyclictest20747-21if_eth119:20:201
1056299163,11cyclictest20747-21if_eth119:20:191
1056299163,10cyclictest2785-21cut19:35:181
1056299163,10cyclictest24934-21fschecks_time21:05:171
1056299163,10cyclictest24934-21fschecks_time21:05:161
1056299163,10cyclictest20228-21sendmail-msp19:20:001
1056299161,12cyclictest19571-21apt-get21:00:051
1056299161,12cyclictest18437-21apt-get20:25:131
1056299161,12cyclictest14504-21cut19:45:221
1056299161,12cyclictest14504-21cut19:45:221
1056299161,12cyclictest11181-21sed19:10:181
1056299161,11cyclictest7067-21kthreadcore20:45:241
1056299161,11cyclictest10671-21latency_hist19:10:021
1056299161,11cyclictest10671-21latency_hist19:10:011
1056299161,10cyclictest25765-21ps19:25:181
1055399161,12cyclictest19148-21tr20:25:210
1058099151,7cyclictest0-21swapper/319:55:013
1058099151,11cyclictest0-21swapper/319:40:023
1056299151,11cyclictest22560-21kthreadcore19:55:181
1058099141,9cyclictest0-21swapper/320:00:283
1058099141,6cyclictest0-21swapper/320:45:153
1058099134,5cyclictest0-21swapper/320:55:153
1055399130,9cyclictest10554-21latency_hist19:10:010
1055399130,9cyclictest10554-21latency_hist19:10:000
1058099111,7cyclictest0-21swapper/319:10:023
1058099111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional