You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 17:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Mon Dec 01, 2025 12:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6720996044,9cyclictest29566irq/58-eth009:20:133
671199572,33cyclictest5082-21ssh12:30:142
671199572,33cyclictest5082-21ssh12:30:132
665899575,7cyclictest131rcu_preempt07:10:140
667999511,45cyclictest10542-21apt-get11:10:131
667999491,6cyclictest0-21swapper/111:15:141
665899487,4cyclictest131rcu_preempt10:05:150
667999471,41cyclictest21051-21apt-get12:20:131
667999461,39cyclictest0-21swapper/107:55:151
667999461,39cyclictest0-21swapper/107:55:141
672099456,27cyclictest29566irq/58-eth010:15:133
665899456,5cyclictest131rcu_preempt08:15:130
667999442,37cyclictest4041-21apt-get09:40:131
667999440,4cyclictest6844-21apt-get07:10:141
671199431,35cyclictest19376-21ssh09:50:132
667999431,3cyclictest0-21swapper/107:45:151
667999430,2cyclictest29013-21ssh12:05:141
6658994324,9cyclictest12-21ksoftirqd/009:50:130
671199423,32cyclictest13561-21ssh11:55:122
667999421,34cyclictest0-21swapper/108:40:111
667999421,2cyclictest31400-21apt-get11:45:141
667999421,2cyclictest0-21swapper/110:55:141
665899427,23cyclictest32247-21ssh10:20:120
665899421,31cyclictest26818-21ssh09:55:120
672099412,25cyclictest29566irq/58-eth011:55:123
667999411,36cyclictest0-21swapper/108:50:121
667999410,3cyclictest0-21swapper/108:25:141
672099404,24cyclictest29566irq/58-eth010:25:153
6720994015,11cyclictest29566irq/58-eth012:00:133
671199402,30cyclictest31417-21ssh11:45:162
671199392,33cyclictest22825-21ssh10:35:132
672099383,22cyclictest29566irq/58-eth011:40:163
667999382,32cyclictest0-21swapper/111:25:141
667999382,31cyclictest0-21swapper/110:50:141
6711993720,14cyclictest27-21ksoftirqd/209:10:122
667999373,28cyclictest9615-21apt-get10:05:161
667999372,2cyclictest10872-21apt-get08:55:131
667999371,2cyclictest0-21swapper/109:10:111
667999370,7cyclictest11282-21ssh11:50:491
667999370,1cyclictest0-21swapper/111:20:121
667999370,1cyclictest0-21swapper/110:10:131
667999363,27cyclictest26811-21apt-get09:55:131
667999362,2cyclictest28057-21apt-get11:00:141
667999361,5cyclictest0-21swapper/107:35:121
667999361,3cyclictest0-21swapper/108:10:131
667999361,2cyclictest0-21swapper/111:40:151
667999353,28cyclictest8356-21apt-get11:30:131
667999351,6cyclictest13330-21cut07:15:201
667999351,2cyclictest2787-21apt-get11:05:131
667999351,2cyclictest16008-21apt-get11:35:111
667999351,29cyclictest22935-21entropy10:35:161
667999350,3cyclictest0-21swapper/112:15:141
667999341,4cyclictest16664-21ntp_192.168.11412:35:241
667999341,3cyclictest0-21swapper/108:00:131
6658993422,8cyclictest0-21swapper/008:30:140
671199332,24cyclictest0-21swapper/212:05:132
667999332,2cyclictest19369-21apt-get09:50:121
667999332,21cyclictest30432-21ssh10:40:191
667999331,4cyclictest19813-21cat07:20:221
667999331,2cyclictest13555-21apt-get11:55:111
667999331,2cyclictest0-21swapper/109:15:131
667999331,2cyclictest0-21swapper/107:50:121
667999331,27cyclictest5071-21sed08:45:311
667999330,6cyclictest10385-21cat08:20:201
667999330,2cyclictest2104-21ssh10:00:121
665899332,6cyclictest131rcu_preempt11:00:140
671199322,24cyclictest17141-21sh10:10:142
671199321,25cyclictest0-21swapper/209:55:122
667999321,3cyclictest0-21swapper/107:25:211
667999321,2cyclictest0-21swapper/109:25:131
667999321,28cyclictest31271-21cut12:25:221
667999320,4cyclictest5075-21apt-get12:30:151
667999320,4cyclictest5075-21apt-get12:30:141
667999320,3cyclictest0-21swapper/109:05:151
665899322,24cyclictest626-21tcf-agent11:10:140
665899322,22cyclictest18113-21ssh12:16:170
672099313,17cyclictest29566irq/58-eth011:10:153
671199313,16cyclictest26346irq/40-dwc2_hso10:05:162
671199312,17cyclictest17879-21ssh11:15:162
667999311,9cyclictest29927-21ssh10:16:281
667999311,7cyclictest6817-21latency_hist07:10:021
667999311,6cyclictest17222-21ssh09:46:511
667999311,5cyclictest8935-21usage_Anybus12:10:291
667999311,4cyclictest21605-21cat12:00:181
667999311,3cyclictest0-21swapper/108:30:141
667999311,26cyclictest30399-21ssh09:35:211
667999310,8cyclictest31043-21sed08:05:321
667999310,6cyclictest15680-21/usr/sbin/munin09:00:281
667999310,4cyclictest9647-21kthreadcore10:25:231
667999310,3cyclictest12485-21ssh09:22:071
667999310,2cyclictest73-21kswapd010:30:181
667999310,2cyclictest73-21kswapd010:30:171
667999310,27cyclictest9523-21latency_hist08:20:021
6658993110,6cyclictest131rcu_preempt08:10:110
672099302,17cyclictest29566irq/58-eth012:35:133
671199305,15cyclictest26346irq/40-dwc2_hso11:05:132
671199303,17cyclictest26346irq/40-dwc2_hso10:15:132
671199302,21cyclictest20700-21ssh12:00:122
667999301,7cyclictest27531-21processes08:35:291
667999301,7cyclictest23946-21cut09:30:211
667999300,2cyclictest0-21swapper/110:45:131
665899302,21cyclictest28975-21ssh12:05:120
665899301,26cyclictest4611-21sh12:10:140
6720992914,10cyclictest29566irq/58-eth010:45:153
672099291,13cyclictest29566irq/58-eth011:05:123
671199293,17cyclictest26346irq/40-dwc2_hso08:15:142
671199292,16cyclictest26346irq/40-dwc2_hso08:55:122
671199291,18cyclictest26346irq/40-dwc2_hso07:20:112
667999291,3cyclictest29675-21/usr/sbin/munin07:30:281
667999291,25cyclictest4003-21kthreadcore07:40:181
672099289,13cyclictest321rcuc/308:50:133
6720992810,13cyclictest29566irq/58-eth010:50:283
671199283,21cyclictest0-21swapper/212:25:122
671199283,18cyclictest26346irq/40-dwc2_hso08:05:182
671199283,16cyclictest26346irq/40-dwc2_hso07:40:122
671199283,15cyclictest26346irq/40-dwc2_hso08:45:152
671199282,19cyclictest26346irq/40-dwc2_hso07:25:142
667999281,9cyclictest5868-21ssh10:22:401
665899283,18cyclictest20737-21ssh12:00:140
672099271,18cyclictest0-21swapper/312:15:153
672099271,17cyclictest29566irq/58-eth011:45:143
671199275,11cyclictest26346irq/40-dwc2_hso08:25:122
671199273,17cyclictest26346irq/40-dwc2_hso09:30:132
671199263,16cyclictest26346irq/40-dwc2_hso10:00:142
671199263,15cyclictest26346irq/40-dwc2_hso12:20:122
671199263,14cyclictest26346irq/40-dwc2_hso11:40:152
671199261,15cyclictest18740-21ssh10:30:262
671199261,15cyclictest18740-21ssh10:30:262
665899266,13cyclictest5675-21ssh10:45:150
665899262,18cyclictest4058-21ssh09:40:140
6658992612,9cyclictest3666-21cron09:40:010
665899261,18cyclictest5094-21sh12:30:140
665899261,18cyclictest5094-21sh12:30:140
672099259,10cyclictest0-21swapper/309:40:143
672099252,12cyclictest29666irq/59-eth010:55:143
6720992519,4cyclictest0-21swapper/312:28:133
672099251,14cyclictest29566irq/58-eth012:30:133
672099251,14cyclictest29566irq/58-eth012:30:133
671199257,10cyclictest6623-21cyclictest11:35:122
671199252,8cyclictest4588-21apt-get12:10:122
671199252,17cyclictest8372-21ssh11:30:152
671199252,16cyclictest0-21swapper/211:50:122
6711992516,4cyclictest27-21ksoftirqd/210:45:162
671199251,13cyclictest4059-21sh09:40:152
6711992510,5cyclictest2277-21kthreadcore10:20:222
665899256,11cyclictest0-21swapper/011:05:120
672099243,13cyclictest0-21swapper/311:30:163
671199247,13cyclictest2585-21ssh11:05:012
671199242,16cyclictest0-21swapper/212:35:132
671199241,9cyclictest25082-21diskmemload09:20:122
671199241,19cyclictest6792-21ls07:10:022
671199241,12cyclictest12763-21ssh12:15:162
6658992415,6cyclictest21057-21sshd12:20:130
672099238,10cyclictest29566irq/58-eth012:20:133
672099238,10cyclictest0-21swapper/312:06:233
672099233,13cyclictest0-21swapper/310:20:143
671199232,15cyclictest31787-1kworker/u9:010:25:152
671199232,14cyclictest26346irq/40-dwc2_hso07:35:122
6711992312,9cyclictest30961-21ssh11:21:402
6711992310,10cyclictest20307-21latency_hist09:05:002
665899232,15cyclictest5554-21ssh11:49:120
665899231,16cyclictest16671-21ssh12:35:240
672099229,9cyclictest0-21swapper/310:00:193
672099223,13cyclictest0-21swapper/312:13:123
672099223,12cyclictest0-21swapper/311:15:153
672099222,15cyclictest0-21swapper/309:50:143
6720992211,6cyclictest29566irq/58-eth011:20:143
671199229,10cyclictest0-21swapper/209:49:002
671199225,10cyclictest26346irq/40-dwc2_hso07:50:122
671199222,16cyclictest26346irq/40-dwc2_hso11:10:132
671199222,14cyclictest26346irq/40-dwc2_hso08:50:132
671199221,11cyclictest8135-21kthreadcore07:10:202
665899220,17cyclictest30822-21ls07:35:020
672099212,13cyclictest0-21swapper/310:35:133
671199213,7cyclictest0-21swapper/207:45:142
671199213,3cyclictest0-21swapper/207:55:162
671199213,3cyclictest0-21swapper/207:55:152
671199213,15cyclictest22941-21kthreadcore10:55:242
671199212,11cyclictest22504-21missed_timers08:30:232
671199211,9cyclictest93266snmpd10:53:382
671199211,10cyclictest32646-21timerandwakeup09:35:282
671199211,10cyclictest32477-21fschecks_time09:15:162
671199211,10cyclictest29046-21fschecks_time08:40:152
671199211,10cyclictest26229-21tune2fs07:30:162
671199210,8cyclictest20591-21ls09:05:162
665899212,15cyclictest29253-21ssh12:25:140
665899212,10cyclictest3808-21tr08:45:240
665899211,16cyclictest653-21ssh11:25:150
665899211,16cyclictest25384-21ssh09:10:040
665899211,15cyclictest30422-21ssh10:40:170
6658992110,3cyclictest131rcu_preempt11:55:170
6720992014,4cyclictest0-21swapper/309:57:153
671199203,9cyclictest30708-21sh10:40:212
671199203,9cyclictest16094-21kthreadcore09:25:202
671199201,16cyclictest32410-21uniq08:10:152
671199201,11cyclictest16255-21tr07:20:012
671199201,10cyclictest27530-21wc08:35:292
671199201,10cyclictest1233-21cut11:25:202
665899203,14cyclictest25504-21ssh11:20:130
665899202,15cyclictest11022-21kthreadcore07:45:230
665899201,14cyclictest24241-21latency_hist10:15:010
665899201,14cyclictest20821-21sh10:55:150
6658992010,8cyclictest21465-21apt-get09:30:130
665899200,15cyclictest0-21swapper/010:50:130
672099199,8cyclictest0-21swapper/310:40:213
672099192,11cyclictest0-21swapper/311:50:133
6720991913,4cyclictest0-21swapper/310:05:173
6720991912,4cyclictest0-21swapper/310:10:133
6720991911,5cyclictest0-21swapper/309:30:133
672099191,13cyclictest0-21swapper/308:35:023
672099191,13cyclictest0-21swapper/307:15:223
672099191,12cyclictest0-21swapper/311:25:133
672099191,12cyclictest0-21swapper/310:30:303
672099191,12cyclictest0-21swapper/310:30:303
672099191,12cyclictest0-21swapper/308:35:303
672099191,11cyclictest0-21swapper/311:00:163
672099191,11cyclictest0-21swapper/308:20:303
6720991910,6cyclictest0-21swapper/311:39:593
671199191,9cyclictest10450-21kthreadcore08:20:202
671199191,8cyclictest22792-21fschecks_time08:00:142
665899193,3cyclictest131rcu_preempt10:33:030
665899193,3cyclictest131rcu_preempt10:33:020
665899191,15cyclictest8362-21ssh11:30:140
665899191,11cyclictest15199-21cut08:25:190
672099188,8cyclictest0-21swapper/309:39:573
672099182,10cyclictest0-21swapper/309:00:143
672099182,10cyclictest0-21swapper/308:55:133
672099182,10cyclictest0-21swapper/307:40:143
6720991812,4cyclictest0-21swapper/309:25:293
672099181,13cyclictest0-21swapper/308:45:303
672099181,13cyclictest0-21swapper/307:55:183
672099181,13cyclictest0-21swapper/307:55:183
672099181,13cyclictest0-21swapper/307:25:243
672099181,13cyclictest0-21swapper/307:25:013
672099181,10cyclictest0-21swapper/309:45:133
665899188,7cyclictest8386-21kthreadcore10:25:200
665899187,8cyclictest14279-21ntp_192.168.11408:55:230
665899186,8cyclictest12-21ksoftirqd/007:55:160
665899186,8cyclictest12-21ksoftirqd/007:55:150
665899183,10cyclictest0-21swapper/008:20:160
665899182,11cyclictest12-21ksoftirqd/011:52:110
665899181,9cyclictest5169-21kthreadcore07:40:220
665899181,9cyclictest26876-21cut08:35:240
665899181,9cyclictest22071-21kthreadcore07:25:200
665899181,15cyclictest2470-21irqcore10:00:170
665899180,11cyclictest7054-21ls09:20:150
672099179,6cyclictest0-21swapper/309:16:563
672099172,10cyclictest0-21swapper/309:10:133
672099171,13cyclictest0-21swapper/307:10:143
672099171,12cyclictest0-21swapper/307:55:023
665899179,5cyclictest21196-21cron09:30:010
665899179,5cyclictest15182-21needreboot09:45:250
665899176,8cyclictest27497-21df_inode08:05:190
665899175,9cyclictest25240-21sed09:10:020
665899172,12cyclictest23497-21irqstats10:35:200
665899171,8cyclictest27526-21kthreadcore10:15:210
665899171,8cyclictest12129-21sed07:15:170
665899171,13cyclictest26007-21ntp_192.168.11408:00:220
665899171,13cyclictest19595-21grep09:00:300
665899171,12cyclictest3426-21ntp_states09:15:270
6658991710,4cyclictest31522-21uname07:35:180
672099161,10cyclictest0-21swapper/308:10:163
672099161,10cyclictest0-21swapper/307:35:163
665899169,5cyclictest14236-21kthreadcore07:50:210
665899165,9cyclictest7187-21kthreadcore08:50:210
665899163,4cyclictest18149-21ls11:15:180
665899162,10cyclictest16028-21ssh11:35:130
665899161,12cyclictest29542-21kthreadcore08:40:190
665899160,2cyclictest131rcu_preempt11:40:110
672099152,10cyclictest0-21swapper/308:05:183
672099151,9cyclictest0-21swapper/307:45:243
665899152,10cyclictest16544-21ls07:20:140
6658991510,3cyclictest6523-21latency_hist07:10:010
672099149,3cyclictest0-21swapper/308:15:283
672099149,3cyclictest0-21swapper/307:30:263
672099142,9cyclictest0-21swapper/308:25:123
672099141,10cyclictest0-21swapper/309:05:243
672099132,7cyclictest321rcuc/308:40:123
672099121,7cyclictest0-21swapper/308:00:113
672099102,5cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional