You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 01:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon Jun 30, 2025 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17499996848,15cyclictest32466irq/58-eth011:45:173
1748099626,42cyclictest17348-21ssh09:40:191
17491995731,20cyclictest9327-21apt-get12:35:162
17480995117,24cyclictest21-21ksoftirqd/108:40:171
17491994925,17cyclictest11442-21apt-get10:40:222
1748099482,10cyclictest131rcu_preempt08:15:151
17491994620,17cyclictest19884-21ssh09:20:202
1747199461,39cyclictest17358-21ssh09:40:200
1747199452,36cyclictest3220-21ssh11:55:150
1747199452,36cyclictest3220-21ssh11:55:140
17471994332,7cyclictest0-21swapper/012:15:140
1747199431,38cyclictest21029-21ssh12:05:190
1749999424,24cyclictest32466irq/58-eth011:35:163
1749999422,25cyclictest32466irq/58-eth011:15:153
1747199422,32cyclictest0-21swapper/009:30:210
1747199422,32cyclictest0-21swapper/009:30:210
17491994125,10cyclictest24159-21apt-get12:25:152
17499994016,15cyclictest32466irq/58-eth010:15:193
1748099407,5cyclictest131rcu_preempt10:00:171
1747199403,19cyclictest12-21ksoftirqd/009:10:160
1749999397,21cyclictest32466irq/58-eth012:35:153
1749199393,30cyclictest32321-21ssh09:50:182
17491993919,14cyclictest0-21swapper/209:40:182
17491993823,9cyclictest26830-21apt-get11:50:172
1748099381,24cyclictest11450-21ssh10:40:201
1747199385,24cyclictest12069-21ssh12:00:150
17491993729,3cyclictest21149-21apt-get10:25:192
17491993717,15cyclictest27-21ksoftirqd/211:20:162
1748099372,19cyclictest26490-21ssh12:06:211
17491993619,13cyclictest0-21swapper/211:00:152
17491993617,14cyclictest0-21swapper/208:00:152
1748099363,22cyclictest27355-21ssh09:25:161
1748099361,22cyclictest520-21ssh12:30:161
17491993514,10cyclictest0-21swapper/211:10:172
1747199352,26cyclictest12061-21rm09:15:180
1749999342,17cyclictest32466irq/58-eth010:10:173
1749999342,17cyclictest32466irq/58-eth010:10:173
1749199348,10cyclictest8401-21ssh12:34:212
17491993424,7cyclictest24515-21apt-get09:45:172
1748099341,27cyclictest17054-21ssh11:05:151
1748099341,27cyclictest17054-21ssh11:05:141
17471993425,4cyclictest12-21ksoftirqd/012:25:150
1749999334,17cyclictest32466irq/58-eth008:20:163
1749999333,20cyclictest32466irq/58-eth011:40:133
17491993322,8cyclictest5578-21apt-get10:15:182
17491993316,13cyclictest0-21swapper/209:15:172
17471993327,3cyclictest12-21ksoftirqd/012:35:140
1749999327,15cyclictest32466irq/58-eth010:25:173
1749199326,14cyclictest11250-21ntp_states09:55:262
1749199323,22cyclictest0-21swapper/212:06:212
17491993216,8cyclictest0-21swapper/208:10:122
17491993216,12cyclictest0-21swapper/207:40:142
17491993215,13cyclictest0-21swapper/208:45:152
17491993215,13cyclictest0-21swapper/208:45:152
1748099321,25cyclictest9576-21ssh09:35:191
1749999312,18cyclictest32466irq/58-eth009:20:193
1749199316,11cyclictest9065-21sh11:40:102
17491993124,3cyclictest9483-21apt-get07:35:122
17491993123,5cyclictest3208-21apt-get11:55:172
17491993123,5cyclictest3208-21apt-get11:55:162
17491993121,6cyclictest15789-21ssh11:25:172
17491993121,3cyclictest2758-21apt-get08:35:152
17491993116,12cyclictest26964-21ls09:00:182
1747199311,21cyclictest3904-21diskmemload10:10:180
1747199311,21cyclictest3904-21diskmemload10:10:180
1749199305,18cyclictest23346irq/40-dwc2_hso10:30:172
1749199305,18cyclictest23346irq/40-dwc2_hso10:30:172
1749199303,18cyclictest23346irq/40-dwc2_hso12:00:152
1749199303,18cyclictest23346irq/40-dwc2_hso07:10:122
17491993024,3cyclictest23346irq/40-dwc2_hso11:35:152
17491993023,3cyclictest23346irq/40-dwc2_hso10:00:182
17491993019,6cyclictest6352-21ssh12:15:142
17491993017,10cyclictest28851-21/usr/sbin/munin07:55:322
17491993014,10cyclictest0-21swapper/208:15:142
17491993013,13cyclictest27472-21ntp_192.168.11411:30:272
17491993010,18cyclictest32396-21uname09:05:232
1748099302,21cyclictest29884-21ssh12:10:161
1749999293,19cyclictest0-21swapper/312:11:423
1749199298,17cyclictest32545-21/usr/sbin/munin11:15:292
1749199298,17cyclictest17237-21fschecks_time11:05:192
1749199298,17cyclictest17237-21fschecks_time11:05:182
1749199297,9cyclictest3904-21diskmemload10:45:332
1749199296,7cyclictest0-21swapper/210:35:352
1749199295,8cyclictest31177-21kthreadcore10:10:242
1749199295,8cyclictest31177-21kthreadcore10:10:232
17491992920,7cyclictest13387-21apt-get10:20:172
17491992915,9cyclictest0-21swapper/209:35:202
17491992915,12cyclictest28865-21ntp_192.168.11408:25:262
17491992910,11cyclictest8091-21mailstats09:10:322
17480992918,5cyclictest0-21swapper/112:35:151
1747199298,13cyclictest17459-21cyclictest07:50:180
1747199292,20cyclictest15311-21ssh10:00:200
1747199291,21cyclictest13410-21ssh10:20:170
1749999283,14cyclictest0-21swapper/309:15:183
17499992814,10cyclictest32466irq/58-eth009:30:213
17499992814,10cyclictest32466irq/58-eth009:30:203
1749199283,17cyclictest23346irq/40-dwc2_hso12:20:202
17491992822,3cyclictest17233-21apt-get08:50:172
17491992817,3cyclictest6109-21apt-get08:05:152
17491992813,12cyclictest0-21swapper/208:20:162
17491992812,12cyclictest31929-21sshd07:22:122
17491992811,13cyclictest24172-21fschecks_count07:50:202
1749999272,4cyclictest0-21swapper/310:35:173
1749999271,15cyclictest32466irq/58-eth011:55:163
1749999271,15cyclictest32466irq/58-eth011:55:163
1749199274,18cyclictest23346irq/40-dwc2_hso11:45:192
1749199272,20cyclictest0-21swapper/207:30:182
17491992717,7cyclictest0-21swapper/209:25:142
17491992712,12cyclictest11511-21usage_Anybus08:40:332
17491992711,13cyclictest916-21snmpd07:17:572
17491992711,13cyclictest916-21snmpd07:17:562
17491992710,14cyclictest9318-21sort09:35:022
17491992710,14cyclictest9318-21sort09:35:012
1747199272,18cyclictest7596-21ssh09:55:130
1749999263,15cyclictest32466irq/58-eth010:45:173
1749199269,9cyclictest2671-21kthreadcore07:25:212
1749199265,7cyclictest0-21swapper/210:54:172
1749199262,15cyclictest23346irq/40-dwc2_hso08:55:182
17491992620,3cyclictest19183-21apt-get07:45:152
17491992613,11cyclictest8357-21rm10:58:362
17491992611,12cyclictest27553-21vmstat10:05:312
1748099262,19cyclictest8393-21ssh11:39:431
1747199264,14cyclictest26841-21ssh11:50:160
1747199263,16cyclictest32626-21ssh11:35:180
1749199251,19cyclictest29887-21sh12:10:162
1748099252,18cyclictest12063-21ssh09:15:181
1748099251,19cyclictest24677-21cut09:45:211
1748099251,19cyclictest22729-21ssh10:05:141
1748099251,18cyclictest24189-21ssh12:25:161
1748099241,17cyclictest30343-21ssh10:10:181
1748099241,17cyclictest30343-21ssh10:10:171
17471992411,10cyclictest17169-21ssh11:25:250
1749999233,16cyclictest0-21swapper/312:25:043
1749999233,16cyclictest0-21swapper/312:06:213
1749999232,10cyclictest0-21swapper/312:16:223
1749999231,17cyclictest0-21swapper/310:55:153
1748099239,11cyclictest12129-21grep11:20:321
1748099238,13cyclictest21-21ksoftirqd/110:25:181
1748099235,10cyclictest9313-21ssh11:00:161
1748099233,15cyclictest9144-21ssh11:40:141
1747199231,19cyclictest17224-21find08:50:160
1749999223,13cyclictest32466irq/58-eth012:00:123
1749999222,14cyclictest0-21swapper/310:05:133
1749999222,13cyclictest32466irq/58-eth012:20:183
17499992211,4cyclictest0-21swapper/309:55:313
1749999221,13cyclictest0-21swapper/309:50:183
1749199222,17cyclictest32304-21kthreadcore08:30:222
1748099229,10cyclictest599-21systemd-resolve10:40:001
1748099224,13cyclictest3229-21ssh11:55:161
1748099224,13cyclictest3229-21ssh11:55:161
1748099222,17cyclictest3904-21diskmemload09:55:151
1748099221,17cyclictest25596-21cut08:25:191
1748099220,18cyclictest26520-21ls11:50:021
1747199222,15cyclictest9299-21apt-get11:00:140
17471992213,6cyclictest20952-21threads11:05:310
17471992213,6cyclictest20952-21threads11:05:310
1747199221,16cyclictest1780-21ssh10:55:160
1747199221,15cyclictest27505-21cut07:20:160
1749999212,14cyclictest0-21swapper/310:30:183
1749999212,14cyclictest0-21swapper/310:30:173
17499992113,5cyclictest33-21ksoftirqd/307:55:203
1748099213,12cyclictest0-21swapper/112:16:211
17480992112,6cyclictest2141-21sh10:55:201
1748099211,18cyclictest3550-21ls10:35:011
1748099211,18cyclictest3550-21ls10:35:011
17480992110,8cyclictest13791-21kthreadcore12:00:201
1748099210,18cyclictest29050-21grep07:55:211
1747199217,2cyclictest131rcu_preempt08:25:240
17471992111,7cyclictest31510-21cron09:05:010
1749999208,10cyclictest0-21swapper/309:10:233
17499992014,4cyclictest0-21swapper/311:10:183
17499992011,6cyclictest32466irq/58-eth009:35:193
1749999201,13cyclictest32466irq/58-eth011:25:183
1748099208,9cyclictest9704-21cat08:05:301
1748099202,11cyclictest0-21swapper/107:40:141
17480992014,4cyclictest23739-21sh12:25:011
1748099201,10cyclictest9234-21ntpq10:15:301
1747199205,10cyclictest629-21Async09:12:570
1747199202,5cyclictest24774-21cat08:20:340
1747199202,13cyclictest26398-21ssh10:05:260
1747199202,12cyclictest15263-21ssh12:20:170
17471992012,6cyclictest3516-21cron10:35:010
17471992012,6cyclictest3516-21cron10:35:000
17471992012,5cyclictest12439-21tr10:40:260
1749999199,6cyclictest0-21swapper/311:21:203
1749999193,9cyclictest0-21swapper/309:45:173
1749999193,11cyclictest0-21swapper/311:54:423
17499991911,6cyclictest0-21swapper/310:40:523
1749999191,12cyclictest0-21swapper/308:10:153
1749999191,12cyclictest0-21swapper/307:50:013
17499991910,6cyclictest32466irq/58-eth009:40:203
17499991910,6cyclictest0-21swapper/311:09:523
17499991910,6cyclictest0-21swapper/311:09:523
1748099199,8cyclictest1349-21df_inode08:00:181
1748099192,11cyclictest26556-21ssh10:50:181
1748099192,11cyclictest0-21swapper/108:55:181
17480991911,5cyclictest31015-21usage_Anybus11:50:351
1748099191,15cyclictest23918-21ssh11:30:141
1747199199,7cyclictest27347-21apt-get09:25:150
1747199198,8cyclictest12-21ksoftirqd/010:50:160
1747199198,6cyclictest0-21swapper/012:11:420
1747199197,9cyclictest22054-21apt-get08:55:190
1747199196,11cyclictest8466-21grep07:30:330
1747199193,10cyclictest0-21swapper/008:30:050
17471991911,6cyclictest10720-21sshd08:08:320
1747199191,12cyclictest7814-21idleruntime08:40:200
17471991910,6cyclictest18292-21/usr/sbin/munin11:45:220
1749999189,6cyclictest0-21swapper/310:20:243
1749999185,10cyclictest0-21swapper/311:00:163
1749999182,12cyclictest0-21swapper/312:30:153
1749999182,10cyclictest0-21swapper/309:05:183
1749999182,10cyclictest0-21swapper/308:40:173
1749999182,10cyclictest0-21swapper/307:35:053
1749999182,10cyclictest0-21swapper/307:15:163
1749999182,10cyclictest0-21swapper/307:15:153
1749999182,10cyclictest0-21swapper/307:10:143
1749999181,13cyclictest0-21swapper/308:45:183
1749999181,13cyclictest0-21swapper/308:45:183
1749999181,11cyclictest0-21swapper/311:30:173
1749999181,10cyclictest0-21swapper/307:25:023
1748099189,6cyclictest27803-21kthreadcore11:10:251
1748099187,3cyclictest17166systemd-network07:23:021
1748099183,12cyclictest7795-21cat11:20:021
1748099181,9cyclictest17377-21fschecks_count08:50:191
1748099181,8cyclictest20897-21grep10:25:021
1748099181,4cyclictest53046usb-storage08:30:111
17480991812,4cyclictest20977-21kthreadcore10:45:241
1748099181,14cyclictest15774-21ssh11:25:171
1748099181,13cyclictest32317-21apt-get09:50:171
1748099181,11cyclictest11117-21cut08:10:161
1748099180,10cyclictest24798-21kthreadcore07:15:231
1748099180,10cyclictest24798-21kthreadcore07:15:221
1748099180,10cyclictest22834-21kthreadcore09:20:281
1747199184,11cyclictest25144-21kthreadcore07:15:230
1747199184,11cyclictest25144-21kthreadcore07:15:230
1747199183,12cyclictest22257-21ls07:15:000
1747199183,10cyclictest0-21swapper/008:45:160
1747199183,10cyclictest0-21swapper/008:45:150
1747199182,2cyclictest131rcu_preempt10:25:130
1747199182,2cyclictest0-21swapper/011:43:200
1747199182,11cyclictest1127-21irqrtprio12:30:200
1747199181,9cyclictest12857-21kthreadcore08:10:200
1747199181,14cyclictest2831-21cut08:00:220
1747199181,13cyclictest1044-21idleruntime-cro08:00:020
17471991810,5cyclictest3847-21/usr/sbin/munin10:35:320
1747199180,15cyclictest27135-21ls09:25:020
1749999179,6cyclictest0-21swapper/310:50:243
1749999179,3cyclictest0-21swapper/308:05:153
1749999172,10cyclictest0-21swapper/308:55:203
1749999172,10cyclictest0-21swapper/308:15:163
1749999172,10cyclictest0-21swapper/307:25:143
1749999171,9cyclictest0-21swapper/307:55:013
17499991711,4cyclictest0-21swapper/310:00:343
17499991711,4cyclictest0-21swapper/309:25:293
1749999171,13cyclictest0-21swapper/308:50:273
1749999171,11cyclictest0-21swapper/308:00:223
1748099177,8cyclictest22752-21wc07:45:301
1748099176,8cyclictest1-21systemd07:25:131
1748099171,5cyclictest9170-21apt-get07:35:001
1748099171,14cyclictest1573-21kthreadcore09:05:261
17480991711,3cyclictest5012-21kernelversion09:10:231
1748099171,13cyclictest20772-21cut08:20:201
1748099171,12cyclictest7549-21latency_hist08:40:021
1747199178,7cyclictest19720-21kthreadcore10:45:230
1747199174,9cyclictest8434-21irqcore11:20:210
1747199173,11cyclictest14324-21acpi07:40:140
1747199172,12cyclictest4262-21grep11:15:310
1747199172,12cyclictest27821-21timerandwakeup11:30:310
1747199171,7cyclictest19354-21fschecks_time07:45:190
1747199171,4cyclictest0-21swapper/009:54:200
1747199171,14cyclictest13205-21ls07:35:290
1747199171,12cyclictest20442-21sendmail-msp08:20:020
1749999162,9cyclictest0-21swapper/308:40:023
1748099167,7cyclictest20995-21memory07:10:221
1748099162,11cyclictest5235-21kthreadcore09:30:301
1748099162,11cyclictest5235-21kthreadcore09:30:301
1748099162,11cyclictest27574-21proc_pri07:50:311
1748099161,12cyclictest17007-21munin-run08:50:011
1748099161,12cyclictest17007-21munin-run08:50:001
1748099161,11cyclictest9872-21irqcore07:35:191
1748099161,10cyclictest0-21swapper/109:00:181
1747199167,7cyclictest32177-21apt-get07:25:140
1747199162,11cyclictest0-21swapper/009:05:180
1747199161,13cyclictest28439-21sed09:45:330
1747199161,12cyclictest6533-21grep08:35:310
1747199161,12cyclictest28392-21ntp_192.168.11411:10:280
1747199161,11cyclictest9676-21df09:35:200
1749999154,5cyclictest0-21swapper/307:30:163
1749999149,3cyclictest0-21swapper/309:00:383
1749999141,7cyclictest0-21swapper/307:40:173
1749199145,7cyclictest17496-21cut07:10:022
1748099141,10cyclictest17508-21latency_hist07:10:011
1747199141,10cyclictest17523-21latency_hist07:10:020
1749999132,6cyclictest0-21swapper/308:30:043
1749999122,7cyclictest0-21swapper/308:25:323
1749999101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional