You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 12:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Sep 08, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
687899635,45cyclictest5721-21ssh21:35:151
688699521,28cyclictest19938-21ssh21:25:132
687899483,35cyclictest19006-21ssh23:30:111
687099482,8cyclictest131rcu_preempt19:10:140
688699464,19cyclictest27-21ksoftirqd/223:05:132
6878994533,9cyclictest14729-21apt-get21:40:141
6870994519,18cyclictest12-21ksoftirqd/021:30:170
687099431,2cyclictest0-21swapper/022:10:150
689799422,26cyclictest35966irq/58-eth022:45:133
689799419,27cyclictest35966irq/58-eth022:30:133
687099413,24cyclictest6455-21ssh00:35:140
6870994117,21cyclictest12-21ksoftirqd/023:00:130
687099401,29cyclictest17262-21ssh00:05:120
6878993825,10cyclictest6428-21apt-get00:35:131
689799373,28cyclictest35966irq/58-eth021:25:123
6878993729,5cyclictest32409-21apt-get20:10:121
687099372,25cyclictest2529-21ssh22:45:140
6870993619,7cyclictest12-21ksoftirqd/019:15:130
689799352,20cyclictest35966irq/58-eth023:25:133
6878993528,3cyclictest26231-21apt-get19:30:141
687099352,24cyclictest27323-21ssh22:05:150
688699344,19cyclictest0-21swapper/220:00:132
688699343,11cyclictest19005-21ssh23:30:122
6886993416,10cyclictest26319-21apt-get00:10:152
688699340,14cyclictest2860-21ssh00:15:132
6878993427,3cyclictest31211-21apt-get22:25:141
6878993420,10cyclictest872-21apt-get23:20:141
6878993420,10cyclictest872-21apt-get23:20:141
689799332,20cyclictest35966irq/58-eth022:35:153
688699332,21cyclictest27346irq/40-dwc2_hso19:55:142
688699332,18cyclictest0-21swapper/219:15:122
687899337,8cyclictest28039-21ntp_states23:15:271
687899334,10cyclictest14804-21grep00:20:221
6878993325,5cyclictest7026-21apt-get19:10:141
6878993325,5cyclictest19175-21apt-get20:30:121
6878993317,12cyclictest15391-21ssh23:10:161
6878993310,8cyclictest858-21users20:40:311
687099332,21cyclictest25901-21ssh22:40:130
6870993314,16cyclictest151rcuc/021:35:130
6886993225,2cyclictest0-21swapper/222:35:132
687899327,10cyclictest410-21ssh22:06:241
687899327,10cyclictest10683-21ssh00:19:031
687899322,24cyclictest6249-21ssh23:05:141
6878993213,13cyclictest0-21swapper/120:00:121
687099322,22cyclictest4563-21ssh00:33:160
687899316,7cyclictest3983-21apt-get22:10:161
6878993124,3cyclictest16641-21apt-get19:20:121
6878993114,12cyclictest9322-21diskmemload21:55:131
6878993114,12cyclictest9322-21diskmemload21:55:121
6878993113,15cyclictest0-21swapper/120:55:111
6870993121,5cyclictest121ksoftirqd/020:30:120
6897993023,5cyclictest321rcuc/320:05:153
6897993023,5cyclictest321rcuc/320:05:153
688699305,15cyclictest0-21swapper/221:35:152
6886993015,11cyclictest29935-21rm00:30:172
687899306,8cyclictest3487-21cat19:40:021
687899305,10cyclictest20986-21kthreadcore21:05:221
6878993024,3cyclictest4770-21apt-get20:15:141
687099302,19cyclictest23735-21ssh21:45:140
687099301,20cyclictest22771-21ssh23:50:170
687099301,19cyclictest20916-21ssh00:25:140
689799293,19cyclictest35966irq/58-eth021:30:163
6897992912,11cyclictest35966irq/58-eth000:30:173
688699298,7cyclictest27-21ksoftirqd/221:40:132
688699292,16cyclictest27346irq/40-dwc2_hso22:05:132
688699291,2cyclictest0-21swapper/200:20:122
6878992923,3cyclictest9911-21apt-get23:25:131
6878992922,4cyclictest8307-21apt-get00:00:141
6878992918,6cyclictest32677-21apt-get21:50:131
6870992910,6cyclictest12-21ksoftirqd/023:25:130
688699287,15cyclictest25889-21apt-get22:40:122
688699285,17cyclictest11437-21diskmemload22:30:252
688699282,5cyclictest271ksoftirqd/220:45:162
688699282,16cyclictest9323-21ssh21:55:142
688699282,16cyclictest9323-21ssh21:55:132
688699281,18cyclictest18481-21cut22:00:142
6886992810,15cyclictest9927-21ssh23:25:142
687899286,10cyclictest28821-21latency_hist21:30:021
6878992812,12cyclictest8630-21if_eth119:45:181
687099283,18cyclictest12-21ksoftirqd/019:40:120
687099281,21cyclictest25314-21ssh21:10:150
687099281,20cyclictest24368-21ssh23:15:130
688699274,6cyclictest27-21ksoftirqd/200:07:282
688699274,15cyclictest27346irq/40-dwc2_hso19:35:132
688699273,14cyclictest27346irq/40-dwc2_hso20:40:132
688699273,14cyclictest27-21ksoftirqd/222:50:142
688699273,14cyclictest0-21swapper/223:57:552
688699273,13cyclictest27-21ksoftirqd/200:00:212
688699272,5cyclictest271ksoftirqd/221:20:182
6886992721,3cyclictest29641-21apt-get23:00:132
688699272,13cyclictest27-21ksoftirqd/223:15:012
688699272,13cyclictest27-21ksoftirqd/222:25:212
6886992716,8cyclictest13119-21apt-get19:50:142
688699271,15cyclictest27-21ksoftirqd/220:05:322
688699271,15cyclictest27-21ksoftirqd/220:05:322
688699270,12cyclictest13973-21grep22:15:182
687899275,8cyclictest25408-21cat00:25:301
687899274,10cyclictest29076-21ssh22:23:041
6878992715,5cyclictest29926-21sh00:30:161
6878992712,12cyclictest13613-21apt-get23:45:121
687099271,20cyclictest8324-21ssh00:00:130
687099271,19cyclictest11664-21ssh22:50:180
6897992615,8cyclictest36066irq/59-eth023:30:123
688699268,9cyclictest11100-21cron22:50:012
688699264,8cyclictest27-21ksoftirqd/223:24:442
688699264,8cyclictest27-21ksoftirqd/223:24:442
688699263,11cyclictest27-21ksoftirqd/223:46:072
688699262,9cyclictest26425-21ssh00:26:192
688699262,16cyclictest27346irq/40-dwc2_hso20:55:122
687899269,8cyclictest27849-21cut22:40:201
687899269,12cyclictest0-21swapper/122:35:151
687099269,9cyclictest121ksoftirqd/020:15:140
6870992613,8cyclictest0-21swapper/021:15:140
687099261,16cyclictest28000-21ssh23:35:110
689799253,13cyclictest35966irq/58-eth022:15:133
689799251,18cyclictest0-21swapper/323:35:113
688699259,4cyclictest0-21swapper/221:05:162
688699258,15cyclictest261rcuc/223:16:142
688699257,10cyclictest0-21swapper/219:45:142
688699253,4cyclictest271ksoftirqd/222:12:132
688699253,4cyclictest271ksoftirqd/221:30:032
688699253,13cyclictest27-21ksoftirqd/220:35:292
688699252,16cyclictest27346irq/40-dwc2_hso21:00:152
688699252,16cyclictest27-21ksoftirqd/223:36:102
688699251,4cyclictest24894-21diskmemload21:49:212
6886992512,5cyclictest0-21swapper/223:40:142
688699251,15cyclictest22856-21ls23:50:182
687899253,7cyclictest27994-21apt-get23:35:121
6878992515,6cyclictest22240-21apt-get23:50:041
6878992513,9cyclictest6918-21ssh23:58:511
6878992511,8cyclictest2667-21ssh22:45:161
6878992510,12cyclictest0-21swapper/122:30:131
6878992510,12cyclictest0-21swapper/100:10:161
689799244,12cyclictest321rcuc/320:50:113
689799243,11cyclictest35966irq/58-eth000:15:133
6897992411,10cyclictest33-21ksoftirqd/322:00:123
688699246,5cyclictest27-21ksoftirqd/222:55:132
688699244,4cyclictest27-21ksoftirqd/222:22:472
688699243,4cyclictest271ksoftirqd/220:20:272
688699242,3cyclictest27-21ksoftirqd/200:38:082
687899249,12cyclictest0-21swapper/121:10:131
687899245,15cyclictest23718-21tr19:25:211
687899243,8cyclictest23900-21ssh21:45:171
687899243,7cyclictest29125-21apt-get21:30:161
687899242,17cyclictest13034-21ssh22:15:141
687899241,7cyclictest0-21swapper/100:06:321
6878992415,5cyclictest0-21swapper/122:00:111
6878992412,9cyclictest20065-21sshd21:02:171
6878992412,9cyclictest1921-21apt-get21:15:131
6878992410,11cyclictest14138-21kthreadcore21:20:251
687099242,18cyclictest0-21swapper/000:15:140
689799232,11cyclictest35966irq/58-eth021:35:133
689799231,15cyclictest35966irq/58-eth023:05:133
688699235,7cyclictest27-21ksoftirqd/220:30:142
6886992314,4cyclictest25290-21apt-get21:10:142
688699231,18cyclictest0-21swapper/221:15:132
6886992311,7cyclictest27346irq/40-dwc2_hso19:40:112
687899236,7cyclictest954266kworker/u8:123:41:461
6878992315,4cyclictest11809-21apt-get19:15:121
6878992313,8cyclictest20588-21ssh22:55:131
6878992312,8cyclictest28513-21sshd20:38:181
6878992312,8cyclictest18931-21sshd20:27:511
6878992312,8cyclictest18931-21sshd20:27:511
6878992311,9cyclictest15310-21ntpq22:50:301
6878992311,8cyclictest5839-21munin-run23:05:011
687099231,7cyclictest14894-21http23:10:020
687099231,15cyclictest11915-21ssh00:20:130
687099230,12cyclictest30425-21grep00:10:330
6897992215,4cyclictest0-21swapper/323:45:153
6897992212,7cyclictest0-21swapper/322:25:293
688699222,4cyclictest27-21ksoftirqd/219:25:202
688699222,11cyclictest131rcu_preempt21:50:022
688699221,17cyclictest18187-21grep20:25:292
688699221,17cyclictest18187-21grep20:25:282
687899229,10cyclictest7319-21cat19:40:271
687899225,14cyclictest6452-21sed20:50:171
6878992211,8cyclictest5801-21latency_hist20:50:011
6878992211,8cyclictest13101-21/usr/sbin/munin19:50:261
6878992211,7cyclictest0-21swapper/120:05:131
6878992211,7cyclictest0-21swapper/120:05:121
687099222,15cyclictest31215-21ssh22:25:140
687099221,15cyclictest15423-21sh23:10:180
689799217,8cyclictest35966irq/58-eth000:20:133
6897992112,5cyclictest321rcuc/323:20:123
6897992112,5cyclictest321rcuc/323:20:123
6897992110,9cyclictest0-21swapper/300:05:153
688699218,9cyclictest9363-21sh20:20:002
688699217,10cyclictest30829-21cron19:35:012
688699212,4cyclictest27-21ksoftirqd/219:10:212
688699211,5cyclictest271ksoftirqd/220:50:252
6878992114,5cyclictest9609-21apt-get20:20:131
687099216,9cyclictest12-21ksoftirqd/019:25:120
687099213,15cyclictest22110-21ssh22:20:140
687099211,7cyclictest4543-21munin-run20:15:010
6870992113,6cyclictest0-21swapper/022:15:290
6870992113,3cyclictest12-21ksoftirqd/021:40:120
6870992111,7cyclictest12-21ksoftirqd/020:20:130
689799209,8cyclictest0-21swapper/300:35:493
689799202,12cyclictest35966irq/58-eth021:15:143
689799202,11cyclictest35966irq/58-eth021:55:123
689799202,11cyclictest35966irq/58-eth021:55:123
6897992014,4cyclictest0-21swapper/323:55:233
6897992014,4cyclictest0-21swapper/322:55:133
6897992013,5cyclictest0-21swapper/323:50:173
6897992012,5cyclictest0-21swapper/300:10:263
689799201,10cyclictest35966irq/58-eth021:50:113
6897992010,6cyclictest0-21swapper/322:40:303
688699202,15cyclictest0-21swapper/220:10:122
687099207,3cyclictest131rcu_preempt21:55:230
687099207,3cyclictest131rcu_preempt21:55:230
687099205,4cyclictest131rcu_preempt20:00:010
687099203,12cyclictest0-21swapper/019:30:150
687099200,16cyclictest18838-21ls21:00:250
687099200,14cyclictest29256-21latency_hist23:00:020
689799192,12cyclictest0-21swapper/300:28:183
689799192,11cyclictest0-21swapper/320:20:133
6897991914,2cyclictest0-21swapper/300:00:123
6897991913,4cyclictest0-21swapper/323:04:103
6897991913,4cyclictest0-21swapper/322:50:093
6897991913,4cyclictest0-21swapper/322:10:063
6897991913,4cyclictest0-21swapper/321:45:263
6897991913,4cyclictest0-21swapper/321:40:293
6897991913,4cyclictest0-21swapper/321:20:293
6897991913,4cyclictest0-21swapper/319:50:303
6897991912,5cyclictest0-21swapper/323:40:143
689799191,12cyclictest0-21swapper/320:45:273
689799191,11cyclictest0-21swapper/320:35:013
688699191,14cyclictest17000-21irqcore19:20:162
687899194,12cyclictest20574-21kthreadcore19:55:221
687099199,8cyclictest12-21ksoftirqd/023:20:330
687099199,8cyclictest12-21ksoftirqd/023:20:320
687099193,11cyclictest28620-21apt-get21:30:020
6870991911,6cyclictest29081-21sshd20:40:180
687099191,15cyclictest16825-21find22:35:130
687099191,14cyclictest8335-21apt-get19:45:130
687099190,3cyclictest131rcu_preempt20:35:140
687099190,16cyclictest7959-21ls00:00:020
689799182,10cyclictest0-21swapper/320:55:133
6897991813,3cyclictest0-21swapper/323:15:233
6897991812,4cyclictest0-21swapper/322:20:173
6897991812,4cyclictest0-21swapper/322:06:153
6897991812,4cyclictest0-21swapper/321:10:273
6897991812,4cyclictest0-21swapper/319:45:163
6897991811,3cyclictest0-21swapper/321:00:143
689799181,11cyclictest0-21swapper/319:40:143
689799181,10cyclictest0-21swapper/319:55:133
688699181,13cyclictest6973-21cat19:10:022
687899181,7cyclictest6964-21latency_hist19:10:021
687099184,5cyclictest12-21ksoftirqd/023:40:270
687099184,10cyclictest3828-21rm21:50:220
687099183,9cyclictest10956-21ssh21:20:160
687099181,8cyclictest13793-21kthreadcore19:50:200
687099181,7cyclictest9831-21kthreadcore22:30:210
687099181,4cyclictest0-21swapper/022:02:470
6870991811,5cyclictest21194-21sshd19:21:180
6870991810,6cyclictest13737-21diskstats23:45:160
6870991810,5cyclictest32245-21latency_hist20:10:010
6870991810,5cyclictest32245-21latency_hist20:10:010
689799172,10cyclictest0-21swapper/323:10:053
689799172,10cyclictest0-21swapper/320:15:153
689799171,12cyclictest0-21swapper/319:30:173
689799171,10cyclictest0-21swapper/319:35:283
687099178,6cyclictest16549-21cut20:25:210
687099178,6cyclictest16549-21cut20:25:210
687099175,3cyclictest131rcu_preempt23:30:250
687099174,10cyclictest26397-21timerandwakeup20:00:280
687099171,13cyclictest8117-21tr20:50:200
687099171,13cyclictest13888-21cut20:55:210
689799162,10cyclictest0-21swapper/320:00:133
689799162,10cyclictest0-21swapper/319:20:123
689799162,10cyclictest0-21swapper/319:15:133
689799161,9cyclictest0-21swapper/320:45:023
687099163,10cyclictest20484-21fschecks_time21:05:180
687099162,11cyclictest3672-21kthreadcore20:45:210
687099160,13cyclictest31699-21irqrtprio19:35:190
689799159,4cyclictest0-21swapper/320:10:123
689799151,6cyclictest0-21swapper/320:25:203
689799151,6cyclictest0-21swapper/320:25:193
689799151,11cyclictest0-21swapper/319:15:013
6897991510,3cyclictest0-21swapper/321:05:303
6897991510,3cyclictest0-21swapper/319:25:263
689799149,3cyclictest0-21swapper/320:35:343
687099140,9cyclictest6891-21latency_hist19:10:010
689799101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional