You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 12:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue Sep 16, 2025 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15858997437,26cyclictest25366irq/58-eth000:10:163
15847995913,19cyclictest1283-21ssh21:30:142
15847995517,17cyclictest19290-21apt-get23:40:132
15858995437,10cyclictest25366irq/58-eth022:00:173
1584199543,33cyclictest9513-21ssh21:35:121
1583799482,35cyclictest9533-21ssh21:35:140
1584199468,28cyclictest131rcu_preempt20:40:131
15847994522,8cyclictest0-21swapper/220:40:132
15847994431,9cyclictest9507-21apt-get21:35:122
1583799422,28cyclictest10208-21ssh22:35:150
1585899412,26cyclictest25366irq/58-eth021:50:153
1585899412,26cyclictest25366irq/58-eth021:50:143
15847994126,12cyclictest0-21swapper/200:30:132
1584799412,26cyclictest17507-21ssh21:20:132
1583799418,8cyclictest131rcu_preempt23:05:150
15847994028,8cyclictest6919-21apt-get00:35:152
15847993824,9cyclictest19017-21apt-get23:20:132
1584199381,27cyclictest19311-21ssh23:40:161
15847993728,6cyclictest23239-21apt-get20:25:142
15847993725,7cyclictest10286-21apt-get22:55:032
1584199371,21cyclictest0-21swapper/100:30:141
1584799364,19cyclictest22555-21grep22:40:312
1584799364,18cyclictest14931-21switchtime23:35:302
15847993621,11cyclictest21846irq/40-dwc2_hso20:10:162
15847993620,12cyclictest0-21swapper/222:30:122
15837993619,10cyclictest0-21swapper/000:10:150
1584799359,13cyclictest19945-21ps20:55:172
15847993516,15cyclictest0-21swapper/200:00:142
1584199352,29cyclictest26462-21ssh22:25:131
1584799349,10cyclictest7943-21grep20:05:322
1584799348,16cyclictest8811-21rm21:53:562
1584799348,16cyclictest8811-21rm21:53:562
1584799348,14cyclictest30729-21tune2fs19:25:182
1584799345,10cyclictest1672-21ssh21:10:142
1584799345,10cyclictest1672-21ssh21:10:132
15847993423,8cyclictest27793-21apt-get00:05:152
15847993417,8cyclictest0-21swapper/223:30:102
15847993415,16cyclictest0-21swapper/220:45:132
1584799341,22cyclictest10626-21ssh00:15:152
15847993324,4cyclictest2435-21apt-get22:50:142
15847993316,12cyclictest3350-21ssh23:50:122
1583799336,5cyclictest131rcu_preempt19:20:140
1584799327,7cyclictest0-21swapper/221:55:252
1584799326,13cyclictest22601-21cat00:20:312
1584799326,10cyclictest0-21swapper/220:50:302
1584799323,12cyclictest0-21swapper/220:35:262
15847993225,3cyclictest2970-21apt-get00:10:152
15841993218,6cyclictest2240-21wc22:30:171
1584799318,7cyclictest10915-21cat22:15:222
1584799317,10cyclictest18326-21ssh23:19:332
1584799314,10cyclictest11836-21kthreadcore22:35:232
15847993125,3cyclictest26440-21apt-get22:25:142
15847993124,3cyclictest13624-21apt-get20:15:122
15841993110,17cyclictest21-21ksoftirqd/123:05:151
15847993023,3cyclictest18221-21apt-get22:20:122
1584799296,8cyclictest29658-21cat21:25:302
1584799293,8cyclictest0-21swapper/219:30:162
1584799292,20cyclictest0-21swapper/223:05:052
15847992921,3cyclictest18137-21apt-get22:00:172
15847992915,11cyclictest0-21swapper/219:15:132
15847992915,10cyclictest0-21swapper/220:00:132
15847992914,12cyclictest9932-21if_err_eth021:15:222
1584199294,4cyclictest131rcu_preempt19:40:141
15837992913,4cyclictest131rcu_preempt22:55:130
1585899284,12cyclictest25366irq/58-eth023:15:133
1584799289,9cyclictest26580-21/usr/sbin/munin22:45:282
1584799289,12cyclictest25557-21munin-run21:45:012
1584799287,11cyclictest9923-21ssh23:14:092
1584799284,14cyclictest21846irq/40-dwc2_hso19:35:142
1584799282,13cyclictest21846irq/40-dwc2_hso22:05:122
15847992812,12cyclictest31362-21/usr/sbin/munin23:45:282
1583799283,4cyclictest131rcu_preempt23:50:120
1585899273,13cyclictest0-21swapper/323:05:153
1584799279,12cyclictest0-21swapper/221:45:132
1584799277,17cyclictest12522-21/usr/sbin/munin19:40:292
15847992712,12cyclictest20748-21latency_hist19:15:012
15847992712,10cyclictest0-21swapper/200:25:142
15847992711,13cyclictest1711-21sh23:28:462
15847992710,9cyclictest15235-21awk23:55:282
1584199272,16cyclictest1314-21diskmemload23:45:131
1585899263,13cyclictest25366irq/58-eth023:40:143
1584799264,13cyclictest21846irq/40-dwc2_hso19:20:142
1584799263,4cyclictest18764-21ps20:20:182
15847992618,5cyclictest21846irq/40-dwc2_hso19:55:112
15847992613,9cyclictest1968-21ssh22:10:142
1583799267,3cyclictest131rcu_preempt19:35:140
1583799263,12cyclictest6925-21ssh00:35:140
1583799261,19cyclictest2461-21ssh22:50:160
1584799259,12cyclictest32591-21sort20:35:022
1584799259,12cyclictest28103-21timerandwakeup21:00:302
1584799259,10cyclictest17326-21apt-get19:45:122
1584799255,16cyclictest30501-21cut21:05:182
15847992519,3cyclictest21846irq/40-dwc2_hso23:00:142
1584199253,18cyclictest10077-21ssh22:15:151
1584199252,14cyclictest0-21swapper/123:52:391
1584199251,9cyclictest131rcu_preempt22:05:281
1583799254,17cyclictest18235-21ssh22:20:140
15837992514,8cyclictest0-21swapper/021:50:150
15837992514,8cyclictest0-21swapper/021:50:140
1584199247,9cyclictest28857-21kthreadcore22:45:211
1584199243,19cyclictest2457-21ssh22:50:151
1584199240,11cyclictest26754-21grep20:25:281
1585899232,15cyclictest25366irq/58-eth021:35:133
1584799234,11cyclictest21846irq/40-dwc2_hso19:50:142
1584199237,4cyclictest131rcu_preempt21:50:121
1584199237,4cyclictest131rcu_preempt21:50:121
1584199232,16cyclictest10871-21ssh23:15:131
15841992315,6cyclictest21-21ksoftirqd/100:10:161
1583799237,4cyclictest131rcu_preempt21:10:290
1583799237,4cyclictest131rcu_preempt21:10:290
1585899222,16cyclictest0-21swapper/322:15:153
1585899222,13cyclictest25366irq/58-eth000:20:153
1584199229,10cyclictest26074-21idleruntime-cro22:25:011
1584199228,6cyclictest0-21swapper/123:59:351
1584199226,7cyclictest0-21swapper/119:15:021
1584199220,8cyclictest25434-21ssh21:44:511
1584199220,2cyclictest131rcu_preempt00:38:501
1583799223,3cyclictest131rcu_preempt23:25:210
1583799222,15cyclictest18948-21ssh23:00:140
1583799222,15cyclictest0-21swapper/019:15:130
15837992215,4cyclictest0-21swapper/020:55:110
1585899218,10cyclictest0-21swapper/322:50:013
15858992112,4cyclictest0-21swapper/322:40:223
15858992110,6cyclictest0-21swapper/321:42:063
1584199219,9cyclictest29724-21/usr/sbin/munin21:45:281
1584199213,5cyclictest23041-21unixbench_singl23:00:321
1584199212,2cyclictest131rcu_preempt23:30:111
1584199211,4cyclictest131rcu_preempt00:05:221
1584199211,16cyclictest1969-21ssh22:10:141
1584199211,16cyclictest11160-21uniq23:35:171
1584199211,14cyclictest9933-21ssh21:55:171
1584199210,12cyclictest25451-21vmstat19:15:311
1583799216,11cyclictest1472-21latency_hist21:50:020
1583799214,5cyclictest12-21ksoftirqd/000:20:170
1583799213,5cyclictest131rcu_preempt20:00:130
1585899208,8cyclictest25366irq/58-eth023:00:163
1585899203,12cyclictest25366irq/58-eth022:55:123
1585899201,15cyclictest25366irq/58-eth000:00:163
15858992010,8cyclictest0-21swapper/323:20:513
1584199207,10cyclictest20189-21kthreadcore23:20:211
1584199202,4cyclictest21-21ksoftirqd/121:00:021
1584199202,2cyclictest131rcu_preempt21:25:291
1584199201,6cyclictest32641-21sed20:35:011
1584199201,4cyclictest131rcu_preempt20:15:011
1584199201,4cyclictest131rcu_preempt00:15:221
1584199201,16cyclictest18158-21ssh22:00:171
1584199200,3cyclictest131rcu_preempt19:55:281
1583799208,10cyclictest16941-1kworker/u9:122:00:250
1583799206,4cyclictest131rcu_preempt22:40:180
15837992012,5cyclictest0-21swapper/000:05:180
1583799201,15cyclictest30072-21cat22:45:230
1583799201,15cyclictest13219-21ssh21:15:290
1583799201,14cyclictest29185-21sh00:25:250
1583799201,14cyclictest12-21ksoftirqd/022:25:140
1583799201,10cyclictest876-21irqrtprio20:35:200
15837992010,7cyclictest14969-21cat00:15:320
1585899195,8cyclictest0-21swapper/322:20:133
1585899192,11cyclictest0-21swapper/319:25:083
15858991911,6cyclictest0-21swapper/321:55:183
15858991910,7cyclictest0-21swapper/300:15:143
1584799191,9cyclictest15959-21latency_hist19:10:022
1584199199,7cyclictest24378-21kthreadcore19:50:221
1584199197,10cyclictest15956-21apt-key19:10:011
1584199194,11cyclictest23595-21grep00:00:301
1584199191,4cyclictest131rcu_preempt21:15:181
1584199191,3cyclictest0-21swapper/121:24:011
15841991912,5cyclictest7484-21sshd19:32:501
1584199191,16cyclictest21730-21ls20:20:211
15841991911,6cyclictest21181-21cut22:40:231
1584199191,11cyclictest21621-21rm00:20:251
1584199191,10cyclictest8397-21kthreadcore19:35:191
15841991910,6cyclictest15577-21kthreadcore20:15:211
1584199190,2cyclictest131rcu_preempt23:10:141
1584199190,2cyclictest131rcu_preempt21:10:131
1584199190,2cyclictest131rcu_preempt21:10:121
1583799192,2cyclictest131rcu_preempt21:32:320
1583799192,13cyclictest0-21swapper/023:45:130
1583799191,9cyclictest26854-21kthreadcore21:25:210
1583799191,7cyclictest12-21ksoftirqd/021:40:250
1583799191,3cyclictest0-21swapper/021:20:470
15837991912,4cyclictest10799-21kthreadcore22:15:210
1583799191,10cyclictest10981-21/usr/sbin/munin23:35:320
1583799190,7cyclictest25991-21rm23:23:470
1583799190,16cyclictest18307-21ls20:20:020
1585899189,6cyclictest0-21swapper/322:37:203
15858991811,5cyclictest0-21swapper/323:49:313
15858991811,4cyclictest33-21ksoftirqd/322:05:283
1585899181,14cyclictest0-21swapper/321:25:193
1585899181,11cyclictest0-21swapper/320:55:273
15858991810,6cyclictest0-21swapper/322:27:043
15858991810,6cyclictest0-21swapper/322:12:223
15858991810,6cyclictest0-21swapper/300:38:233
15858991810,5cyclictest25366irq/58-eth023:50:143
1584199189,3cyclictest3100-21/usr/sbin/munin20:00:301
1584199185,5cyclictest131rcu_preempt22:37:581
1584199182,3cyclictest9850irq/52-mmc019:20:211
1584199181,9cyclictest32583-21ssh23:26:261
1584199181,6cyclictest20275-21kthreadcore19:45:221
1584199181,6cyclictest19445-21latency_hist20:55:011
1584199181,2cyclictest131rcu_preempt22:55:011
1584199181,13cyclictest25647-21ssh00:25:131
1584199181,13cyclictest1308-21ssh21:30:161
15841991810,6cyclictest31076-21sed19:25:191
1583799187,8cyclictest18435-21apt-get20:20:120
1583799185,10cyclictest3339-21kthreadcore22:10:220
1583799183,9cyclictest0-21swapper/020:25:140
1583799183,12cyclictest6319-21ssh23:10:270
1583799181,9cyclictest0-21swapper/021:57:070
1583799181,6cyclictest9930-21/usr/sbin/munin20:45:280
1583799181,6cyclictest18280-21ssh23:58:240
1583799181,5cyclictest22954-21ntp_kernel_pll_23:40:270
1583799181,3cyclictest131rcu_preempt20:05:330
1583799181,2cyclictest52846usb-storage19:25:080
15837991812,4cyclictest26791-21irqrtprio22:05:170
15837991811,5cyclictest10842-21/usr/sbin/munin23:15:260
15837991811,5cyclictest0-21swapper/020:40:250
15837991810,6cyclictest3590-21cut22:30:200
1583799180,6cyclictest1872-21kthreadcore00:30:220
1583799180,2cyclictest131rcu_preempt19:10:020
1585899179,6cyclictest0-21swapper/323:37:133
1585899179,6cyclictest0-21swapper/323:10:313
1585899179,6cyclictest0-21swapper/321:16:423
1585899179,6cyclictest0-21swapper/300:28:533
1585899179,6cyclictest0-21swapper/300:07:383
1585899179,5cyclictest25366irq/58-eth022:30:143
1585899177,8cyclictest0-21swapper/323:30:403
1585899177,8cyclictest0-21swapper/321:48:263
1585899177,7cyclictest0-21swapper/321:10:143
1585899177,7cyclictest0-21swapper/321:10:133
15858991711,4cyclictest0-21swapper/323:57:043
1585899171,12cyclictest0-21swapper/321:20:143
1584199171,13cyclictest410-21df20:35:171
1584199171,13cyclictest10004-21sort20:45:161
1584199170,2cyclictest131rcu_preempt20:10:011
1583799174,11cyclictest28762-21kthreadcore19:55:200
1583799173,11cyclictest6583-21ps23:30:250
1583799173,11cyclictest17176-21kthreadcore19:10:200
1583799173,11cyclictest12878-21irqcore19:40:180
1583799172,12cyclictest20685-21missed_timers19:45:240
1583799172,12cyclictest16898-21kthreadcore20:50:250
1583799171,4cyclictest0-21swapper/019:30:590
1583799171,3cyclictest131rcu_preempt21:00:180
15837991711,4cyclictest12600-21timerwakeupswit20:10:310
1583799171,13cyclictest28525-21irqrtprio20:30:150
1583799171,12cyclictest31664-21kthreadcore21:05:200
15837991710,4cyclictest0-21swapper/019:50:180
1585899169,5cyclictest33-21ksoftirqd/319:30:183
1585899168,6cyclictest0-21swapper/300:32:083
1585899168,5cyclictest25366irq/58-eth022:50:153
1585899161,12cyclictest0-21swapper/319:50:033
1584199161,7cyclictest0-21swapper/121:00:171
1584199161,11cyclictest29350-21fschecks_time21:05:151
1583799166,7cyclictest23219-21cut00:00:240
1585899158,4cyclictest25366irq/58-eth023:25:143
1585899157,5cyclictest33-21ksoftirqd/320:15:173
1585899151,8cyclictest0-21swapper/320:25:153
1585899151,11cyclictest0-21swapper/319:45:013
15858991510,3cyclictest0-21swapper/321:30:163
15858991510,3cyclictest0-21swapper/319:36:233
1585899149,3cyclictest33-21ksoftirqd/320:35:293
1585899149,3cyclictest33-21ksoftirqd/320:10:193
1585899149,3cyclictest0-21swapper/319:15:163
1585899141,6cyclictest0-21swapper/319:50:193
1585899141,10cyclictest0-21swapper/319:20:213
15858991410,2cyclictest33-21ksoftirqd/321:05:283
15858991410,2cyclictest33-21ksoftirqd/320:00:133
1585899131,8cyclictest0-21swapper/319:55:123
1585899131,5cyclictest0-21swapper/320:05:263
1585899128,2cyclictest33-21ksoftirqd/319:10:293
1585899127,3cyclictest0-21swapper/320:53:023
1585899122,8cyclictest0-21swapper/320:50:013
1585899121,9cyclictest0-21swapper/320:20:183
1585899111,8cyclictest0-21swapper/320:40:163
1585899111,7cyclictest0-21swapper/321:00:203
1585899111,7cyclictest0-21swapper/320:30:163
1585899111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional