You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 02:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Jul 06, 2025 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
450799552,44cyclictest3557-21ssh12:00:160
450799542,44cyclictest27402-21ssh09:55:120
4525995229,20cyclictest0-21swapper/211:10:152
452599517,24cyclictest27-21ksoftirqd/208:50:162
450799512,44cyclictest15745-21ssh10:30:140
453399502,29cyclictest32466irq/58-eth010:10:133
452599465,19cyclictest27-21ksoftirqd/212:05:142
453399452,22cyclictest32466irq/58-eth010:55:143
4533994418,16cyclictest32466irq/58-eth011:20:143
4517994416,23cyclictest21-21ksoftirqd/109:20:191
451799422,30cyclictest3547-21ssh12:00:161
450799422,6cyclictest0-21swapper/011:00:140
453399393,23cyclictest32466irq/58-eth012:25:143
453399392,23cyclictest32466irq/58-eth010:00:143
450799391,12cyclictest17101-21ssh11:10:250
452599387,23cyclictest6114-21ssh11:25:162
4525993812,10cyclictest4496-21cyclictest12:15:152
453399362,20cyclictest32466irq/58-eth012:30:133
452599362,27cyclictest29907-21kthreadcore11:55:232
450799361,10cyclictest29801-21grep11:20:000
453399356,18cyclictest32466irq/58-eth009:45:153
453399356,18cyclictest32466irq/58-eth009:45:153
452599352,17cyclictest12560-21latency11:45:242
452599352,17cyclictest12560-21latency11:45:232
451799353,9cyclictest131rcu_preempt11:45:141
451799353,9cyclictest131rcu_preempt11:45:131
450799352,2cyclictest0-21swapper/007:35:150
450799351,9cyclictest26342-21kthreadcore09:10:220
450799351,8cyclictest2559-21apt-config08:15:010
450799351,8cyclictest2559-21apt-config08:15:010
450799351,10cyclictest856-21sshd12:16:230
450799351,10cyclictest1326-21tr11:40:210
451799345,20cyclictest31212-21ssh09:15:151
450799347,4cyclictest5525-21cpuspeed10:45:140
450799341,9cyclictest17856-21irqrtprio08:30:180
450799341,9cyclictest11761-21grep10:05:200
450799341,8cyclictest1410-21/usr/sbin/munin07:40:160
450799341,8cyclictest12931-21ntp_192.168.11407:15:230
450799341,8cyclictest10369-21cut10:25:210
450799341,6cyclictest25422-21latency_hist08:05:020
450799341,10cyclictest25656-21grep08:05:170
450799341,10cyclictest24148-21kthreadcore10:35:180
450799340,8cyclictest14016-21tr09:45:230
450799340,8cyclictest14016-21tr09:45:220
451799335,25cyclictest12537-21ssh12:05:151
451799333,20cyclictest21540-21ssh12:10:161
450799331,9cyclictest6482-21kthreadcore09:40:230
450799331,9cyclictest5780-21kthreadcore08:50:240
450799331,9cyclictest18985-21kthreadcore11:50:220
450799331,8cyclictest32408-21latency_hist11:40:010
450799331,8cyclictest23013-21cut08:35:200
450799331,7cyclictest6220-21ssh11:25:180
450799331,7cyclictest5436-21ntpq12:35:250
450799331,7cyclictest25297-21ntp_states09:30:240
450799331,7cyclictest17431-21cut07:20:240
450799331,7cyclictest17431-21cut07:20:230
450799331,14cyclictest31221-21ssh09:15:150
450799331,10cyclictest4086-21ntp_192.168.11410:20:280
450799331,10cyclictest4086-21ntp_192.168.11410:20:270
450799331,10cyclictest20792-21cut10:10:240
450799330,9cyclictest8207-21kthreadcore11:05:210
450799330,9cyclictest16053-21fschecks_count07:55:190
450799330,8cyclictest21707-21diskstats10:55:160
453399323,18cyclictest32466irq/58-eth009:55:113
452599327,7cyclictest0-21swapper/211:35:152
452599322,25cyclictest8176-21ssh12:37:242
4517993210,19cyclictest18550-21kworker/u8:211:25:191
450799327,4cyclictest29414-21unixbench_singl10:15:300
450799326,5cyclictest18134-21kthreadcore12:25:200
450799321,9cyclictest379-21kthreadcore10:40:230
450799321,9cyclictest11581-21sed07:50:190
450799321,8cyclictest19975-21if_eth109:50:200
450799320,9cyclictest30199-21cpu11:20:160
452599319,6cyclictest0-21swapper/208:05:152
452599314,17cyclictest0-21swapper/210:55:132
452599313,17cyclictest23346irq/40-dwc2_hso10:50:152
452599313,11cyclictest23100-21ssh11:15:222
451799318,16cyclictest4037-21ssh09:40:161
451799313,25cyclictest15737-21sh10:30:141
450799318,5cyclictest0-21swapper/008:25:280
450799311,9cyclictest17987-21timerwakeupswit09:25:320
450799311,9cyclictest14993-21df_inode11:30:150
450799311,7cyclictest29886-21ssh12:14:440
450799311,7cyclictest29326-21sed09:35:190
450799311,7cyclictest12649-21ntp_states08:55:270
450799311,6cyclictest8202-21ntp_states07:10:270
450799311,6cyclictest22262-21ntp_192.168.11409:05:250
450799311,6cyclictest13458-21ssh10:50:150
450799310,9cyclictest25178-21cpu12:30:140
450799310,8cyclictest9122-21/usr/sbin/munin11:45:160
450799310,8cyclictest9122-21/usr/sbin/munin11:45:150
450799310,7cyclictest12171-21ssh09:22:250
453399302,14cyclictest32466irq/58-eth008:45:153
453399302,14cyclictest32466irq/58-eth008:45:153
452599308,15cyclictest0-21swapper/209:35:152
452599305,16cyclictest27-21ksoftirqd/210:30:142
452599302,21cyclictest0-21swapper/212:30:142
452599302,21cyclictest0-21swapper/207:35:092
452599302,11cyclictest6046-21cut12:00:232
4525993010,11cyclictest27-21ksoftirqd/212:25:122
450799302,23cyclictest26947-21ssh11:55:150
450799301,9cyclictest20561-21kthreadcore07:25:220
450799301,7cyclictest10842-21sed12:20:270
450799301,5cyclictest17927-21grep09:00:300
452599295,14cyclictest23346irq/40-dwc2_hso07:40:162
452599292,12cyclictest12222-21ssh12:20:482
452599292,11cyclictest23036-21ssh11:50:432
451799297,14cyclictest21-21ksoftirqd/110:15:151
451799294,21cyclictest23777-21ssh09:10:121
450799291,9cyclictest9367-21kthreadcore07:45:220
450799291,9cyclictest16634-21unixbench_multi12:05:290
450799291,7cyclictest23991-21apt-get07:30:140
450799291,6cyclictest6602-21threads10:00:330
452599282,20cyclictest614-21systemd-logind09:40:152
4525992813,11cyclictest23426-21diskmemload10:35:132
452599281,21cyclictest0-21swapper/208:00:132
451799283,21cyclictest25169-21ssh12:30:131
450799281,9cyclictest27118-21fschecks_count08:40:150
450799281,6cyclictest4510-21cat07:10:020
450799280,9cyclictest6310-21awk08:15:270
453399273,16cyclictest0-21swapper/311:15:233
451799272,22cyclictest30547-21ssh12:15:161
451799272,21cyclictest1782-21rm12:35:141
450799271,6cyclictest2992-21grep08:45:290
450799271,6cyclictest2992-21grep08:45:280
450799270,10cyclictest10685-21cut08:20:200
450799270,10cyclictest10685-21cut08:20:190
453399263,15cyclictest0-21swapper/311:58:233
4533992613,9cyclictest32466irq/58-eth011:10:173
453399260,16cyclictest32466irq/58-eth010:15:153
452599265,15cyclictest23346irq/40-dwc2_hso08:55:152
452599262,19cyclictest26610-21ssh12:10:462
452599261,22cyclictest15861-21ls09:45:312
452599261,22cyclictest15861-21ls09:45:312
451799261,22cyclictest32711-21ssh11:40:131
453399253,15cyclictest0-21swapper/312:37:233
453399252,17cyclictest0-21swapper/312:08:223
452599254,15cyclictest23346irq/40-dwc2_hso11:40:142
452599252,15cyclictest23346irq/40-dwc2_hso07:25:152
452599252,13cyclictest842-21munin-node10:10:152
452599251,5cyclictest0-21swapper/210:20:172
452599251,5cyclictest0-21swapper/210:20:162
452599251,17cyclictest30195-21sh11:20:162
4525992510,6cyclictest9335-21timerandwakeup11:05:282
453399242,14cyclictest32466irq/58-eth009:35:153
452599241,9cyclictest25891-21timerandwakeup08:35:292
451799243,18cyclictest21-21ksoftirqd/111:15:171
451799242,18cyclictest5469-21ssh10:45:151
453399238,7cyclictest33-21ksoftirqd/310:40:133
453399238,10cyclictest32466irq/58-eth011:35:153
453399233,15cyclictest0-21swapper/311:05:243
453399233,13cyclictest0-21swapper/311:50:223
452599236,4cyclictest4496-21cyclictest10:45:142
452599232,16cyclictest0-21swapper/211:31:232
4525992311,5cyclictest0-21swapper/209:25:132
452599231,11cyclictest20500-21munin-run08:00:022
4517992311,6cyclictest530-21usb-storage07:35:141
453399222,12cyclictest32466irq/58-eth012:00:163
4533992214,5cyclictest32466irq/58-eth009:20:183
4533992212,7cyclictest0-21swapper/312:10:163
4533992212,7cyclictest0-21swapper/311:00:233
452599223,14cyclictest23346irq/40-dwc2_hso08:20:112
452599223,14cyclictest23346irq/40-dwc2_hso08:20:112
452599222,5cyclictest0-21swapper/210:05:132
452599221,11cyclictest28885-21tr09:55:202
452599221,11cyclictest27749-21timerandwakeup07:30:302
451799222,14cyclictest30194-21ssh11:20:141
451799222,14cyclictest18809-21ssh11:50:231
4517992211,8cyclictest23959-21fschecks_count11:35:181
4533992116,3cyclictest321rcuc/309:17:053
4533992116,3cyclictest321rcuc/309:14:373
4533992115,4cyclictest0-21swapper/309:51:433
4533992115,4cyclictest0-21swapper/309:40:343
4533992114,5cyclictest0-21swapper/310:30:293
4533992113,5cyclictest0-21swapper/310:05:433
4533992111,6cyclictest32466irq/58-eth011:45:163
4533992111,6cyclictest32466irq/58-eth011:45:163
452599218,10cyclictest30008-21fschecks_time11:00:172
452599213,10cyclictest17780-21ntp_192.168.11407:20:262
452599213,10cyclictest17780-21ntp_192.168.11407:20:252
452599212,10cyclictest14191-21kthreadcore08:25:212
452599211,18cyclictest5211-21irqrtprio07:10:192
452599211,17cyclictest8322-21sort10:25:152
452599211,16cyclictest27121-21kthreadcore10:15:212
452599211,13cyclictest3169-21tr08:15:212
452599211,12cyclictest2399-21ntp_192.168.11409:15:262
452599211,11cyclictest7601-21cut09:20:252
452599211,11cyclictest24317-21tr09:10:192
452599211,11cyclictest23043-21ls09:50:252
452599211,10cyclictest31598-21grep08:45:012
451799212,17cyclictest29910-21kthreadcore11:55:221
451799212,15cyclictest444-21diskmemload10:20:171
451799212,15cyclictest444-21diskmemload10:20:161
451799212,14cyclictest17086-21ssh10:50:251
4517992113,5cyclictest0-21swapper/110:55:151
451799211,17cyclictest9312-21cut08:55:181
451799211,15cyclictest19848-21ssh12:25:261
451799211,14cyclictest16039-21cut07:55:181
4517992111,4cyclictest0-21swapper/110:35:111
4517992110,8cyclictest7457-21cron08:20:011
4533992014,4cyclictest321rcuc/311:43:303
4533992014,4cyclictest0-21swapper/312:20:113
4533992014,4cyclictest0-21swapper/308:05:153
4533992013,5cyclictest0-21swapper/312:15:143
4533992012,6cyclictest0-21swapper/310:50:313
452599209,8cyclictest21877-21idleruntime09:30:152
452599205,12cyclictest23346irq/40-dwc2_hso07:15:142
452599203,8cyclictest447-21kthreadcore08:10:232
452599203,8cyclictest447-21kthreadcore08:10:222
452599202,15cyclictest9454-21kthreadcore07:45:232
452599202,15cyclictest22024-21latency_hist08:35:022
452599201,9cyclictest3219-21irqcore10:00:192
452599201,9cyclictest14602-21ps07:50:352
452599201,8cyclictest3020366ssh10:40:042
452599201,15cyclictest20909-21kthreadcore09:05:222
451799202,14cyclictest6274-21apt-get07:45:121
451799201,9cyclictest9539-21acpi07:15:121
4517992013,4cyclictest15403-21latency_hist10:30:001
4517992012,5cyclictest9762-21idleruntime-cro10:05:011
4533991913,4cyclictest0-21swapper/311:25:183
4533991913,4cyclictest0-21swapper/310:45:233
4533991913,4cyclictest0-21swapper/310:35:243
4533991913,4cyclictest0-21swapper/310:25:323
4533991913,4cyclictest0-21swapper/309:30:243
4533991913,4cyclictest0-21swapper/309:00:323
4533991913,4cyclictest0-21swapper/308:10:183
4533991913,4cyclictest0-21swapper/308:10:183
4533991913,4cyclictest0-21swapper/307:15:173
4533991913,4cyclictest0-21swapper/307:10:273
4533991912,5cyclictest0-21swapper/308:55:173
4533991912,5cyclictest0-21swapper/308:15:323
4533991912,5cyclictest0-21swapper/307:35:063
4533991911,6cyclictest0-21swapper/311:32:443
4533991910,6cyclictest0-21swapper/310:20:183
4533991910,6cyclictest0-21swapper/310:20:183
452599191,9cyclictest32009-21fschecks_time08:45:182
452599191,9cyclictest32009-21fschecks_time08:45:172
452599191,9cyclictest14015-21cpu09:00:162
451799194,13cyclictest21-21ksoftirqd/108:00:301
451799194,10cyclictest0-21swapper/109:00:131
451799192,15cyclictest1-21systemd12:20:471
451799191,9cyclictest14855-21/usr/sbin/munin11:30:161
4517991910,6cyclictest30873-21kthreadcore09:35:221
453399182,10cyclictest0-21swapper/308:00:143
4533991813,3cyclictest0-21swapper/307:50:213
4533991812,4cyclictest0-21swapper/308:40:263
4533991812,4cyclictest0-21swapper/308:35:133
4533991812,4cyclictest0-21swapper/307:30:153
4533991812,3cyclictest0-21swapper/307:20:203
4533991812,3cyclictest0-21swapper/307:20:203
453399181,11cyclictest0-21swapper/309:25:173
4533991810,6cyclictest0-21swapper/309:10:033
451799189,7cyclictest31814-21apt-get08:45:151
451799189,7cyclictest31814-21apt-get08:45:141
451799189,6cyclictest10264-21fschecks_time10:05:161
451799186,9cyclictest18338-21sed10:10:191
451799183,12cyclictest15620-21sshd07:20:221
451799183,12cyclictest15620-21sshd07:20:211
451799182,13cyclictest21700-21ssh09:30:141
451799181,9cyclictest30370-21acpi08:10:141
451799181,9cyclictest30370-21acpi08:10:131
4517991812,4cyclictest0-21swapper/109:45:341
4517991812,4cyclictest0-21swapper/109:45:331
451799181,15cyclictest22195-21kthreadcore07:25:221
451799181,15cyclictest11878-21kthreadcore07:50:201
451799181,14cyclictest9097-21ntp_states11:05:271
4517991810,5cyclictest1-21systemd09:25:131
451799180,12cyclictest4868-21fschecks_time07:10:181
4533991712,3cyclictest0-21swapper/307:40:223
4533991711,4cyclictest321rcuc/308:50:163
4533991711,4cyclictest0-21swapper/308:30:293
4533991711,4cyclictest0-21swapper/308:25:193
4533991711,4cyclictest0-21swapper/308:20:193
4533991711,4cyclictest0-21swapper/308:20:183
4533991711,4cyclictest0-21swapper/307:45:123
451799179,6cyclictest0-21swapper/107:30:161
451799173,9cyclictest19459-21kthreadcore08:30:211
451799171,14cyclictest27581-21fschecks_time09:55:161
451799171,13cyclictest30608-21if_eth010:40:191
451799171,12cyclictest31145-21cut11:00:211
451799171,12cyclictest25650-21ntp_states08:35:261
4533991611,3cyclictest0-21swapper/307:25:203
4533991610,4cyclictest0-21swapper/307:55:203
451799168,6cyclictest4496-21cyclictest07:40:171
451799162,9cyclictest0-21swapper/109:05:161
451799162,9cyclictest0-21swapper/108:40:151
451799162,12cyclictest7831-21fschecks_count08:20:151
451799162,12cyclictest7831-21fschecks_count08:20:141
451799162,10cyclictest4367-21fschecks_count08:50:191
451799161,12cyclictest13731-21sort11:10:161
451799161,11cyclictest16007-21ntp_states08:25:251
451799161,10cyclictest19224-21apt-get09:50:031
451799151,10cyclictest25911-21irqcore08:05:181
451799141,9cyclictest4594-21latency_hist07:10:011
452599131,9cyclictest4559-21latency_hist07:10:022
453399111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional