You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-18 - 22:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun May 18, 2025 12:45:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22474997331,28cyclictest25866irq/58-eth011:50:163
2245699545,6cyclictest131rcu_preempt08:15:161
2246599493,25cyclictest27-21ksoftirqd/212:30:152
22446994830,10cyclictest0-21swapper/011:35:140
2246599470,29cyclictest900-21snmpd11:35:162
2245699462,37cyclictest16489-21ssh09:15:141
2244699461,35cyclictest28615-21ssh11:45:150
22474994529,10cyclictest25866irq/58-eth009:10:133
2244699459,24cyclictest12-21ksoftirqd/012:35:160
2245699441,33cyclictest19646-21ssh11:40:141
2247499428,20cyclictest25866irq/58-eth012:35:173
22474994223,12cyclictest25866irq/58-eth012:10:203
2246599415,20cyclictest27-21ksoftirqd/208:35:162
2246599412,18cyclictest31555-21ssh10:50:182
2247499401,23cyclictest25866irq/58-eth011:20:133
2244699398,19cyclictest601-21rngd10:20:170
2247499384,22cyclictest25866irq/58-eth009:30:143
22474993723,6cyclictest0-21swapper/311:00:163
2246599360,29cyclictest0-21swapper/211:20:132
22446993617,14cyclictest601-21rngd07:55:130
2247499352,14cyclictest25866irq/58-eth010:55:143
22474993516,13cyclictest25866irq/58-eth011:10:173
2246599351,18cyclictest27-21ksoftirqd/210:15:172
2246599350,17cyclictest29500-21ssh09:45:152
2247499322,19cyclictest25866irq/58-eth010:25:163
2247499322,16cyclictest25866irq/58-eth010:35:163
2247499314,18cyclictest0-21swapper/310:40:173
2246599311,22cyclictest12518-21ssh09:55:182
2246599311,13cyclictest629-21tcf-agent09:40:132
22446993121,5cyclictest13379-21kworker/u8:011:50:140
2244699309,10cyclictest601-21rngd07:40:150
2244699305,12cyclictest30816-21munin-run11:10:010
2244699301,23cyclictest19664-21rm11:40:170
2246599293,16cyclictest22146irq/40-dwc2_hso08:15:162
22465992911,11cyclictest24546-21sh12:00:232
22446992921,3cyclictest12154-21apt-get08:40:150
2247499286,13cyclictest0-21swapper/309:20:123
2246599281,14cyclictest27982-21ssh10:05:172
22474992716,8cyclictest0-21swapper/309:25:163
2247499271,20cyclictest0-21swapper/310:10:183
2246599274,11cyclictest27-21ksoftirqd/207:35:152
2247499262,17cyclictest0-21swapper/311:40:163
2246599261,17cyclictest7268-21ls08:35:022
2244699263,9cyclictest2753-21sed09:25:310
2244699261,17cyclictest4681-21ssh09:50:190
2247499252,12cyclictest25866irq/58-eth009:35:153
22474992515,7cyclictest0-21swapper/309:50:183
22474992511,9cyclictest25866irq/58-eth012:25:213
22474992511,9cyclictest25866irq/58-eth012:25:203
22474992510,10cyclictest25866irq/58-eth011:35:163
22465992513,5cyclictest16418-21ssh09:15:072
22465992512,5cyclictest13537-21ssh10:36:012
22465992511,9cyclictest0-21swapper/211:30:142
2245699252,16cyclictest845-21munin-node08:55:161
2244699254,8cyclictest6498-21sed12:25:380
2244699254,8cyclictest6498-21sed12:25:380
22474992411,8cyclictest25866irq/58-eth010:05:163
22465992412,9cyclictest26646-21ssh10:03:242
22465992412,9cyclictest20929-21ssh10:41:032
22465992412,9cyclictest13504-21ssh12:11:092
2244699242,7cyclictest4201-21cat12:05:390
2244699242,18cyclictest1713-21apt-get11:30:140
22446992414,5cyclictest601-21rngd08:05:180
2244699241,16cyclictest601-21rngd12:10:200
2247499233,11cyclictest0-21swapper/312:05:183
2246599239,6cyclictest261rcuc/210:20:172
2246599231,7cyclictest16971-21ssh12:15:162
22465992314,7cyclictest0-21swapper/209:35:292
22465992311,9cyclictest25328-21ssh12:35:512
2246599231,19cyclictest18933-21kthreadcore08:45:222
22465992310,10cyclictest31156-21ssh11:26:462
2246599230,11cyclictest131rcu_preempt12:20:172
2245699232,16cyclictest601-21rngd11:30:141
22456992316,3cyclictest21-21ksoftirqd/108:45:141
2245699231,2cyclictest131rcu_preempt09:40:121
2244699236,4cyclictest7993-21apt-get10:35:040
2244699233,4cyclictest601-21rngd09:35:160
2244699233,4cyclictest601-21rngd09:15:250
2244699232,7cyclictest601-21rngd11:30:020
22446992317,4cyclictest601-21rngd11:55:150
22474992213,6cyclictest0-21swapper/310:30:283
2247499221,12cyclictest25866irq/58-eth008:45:123
2246599229,5cyclictest2768-21ssh10:10:062
2246599222,13cyclictest601-21rngd08:40:142
2246599221,9cyclictest2578366ssh10:25:162
2246599221,17cyclictest0-21swapper/208:55:152
22465992210,9cyclictest4637-21ssh09:27:082
22465992210,10cyclictest31489-21perl11:10:202
2245699223,6cyclictest131rcu_preempt09:50:171
2244699224,8cyclictest0-21swapper/009:30:390
2244699223,4cyclictest601-21rngd10:10:140
2244699223,3cyclictest0-21swapper/012:15:220
2244699223,10cyclictest0-21swapper/007:10:120
22446992216,3cyclictest601-21rngd10:00:160
2244699221,18cyclictest29379-21ls12:20:260
2244699220,6cyclictest601-21rngd10:30:350
2247499211,9cyclictest0-21swapper/308:05:163
22474992115,4cyclictest0-21swapper/310:16:073
22474992114,5cyclictest0-21swapper/310:21:253
22474992114,4cyclictest0-21swapper/307:50:313
22474992114,2cyclictest25866irq/58-eth012:30:153
22474992112,7cyclictest0-21swapper/311:56:593
22474992112,7cyclictest0-21swapper/310:50:183
22474992110,7cyclictest25866irq/58-eth011:45:163
2246599219,9cyclictest7848-21ssh10:34:472
2246599219,9cyclictest7372-21ssh12:25:482
2246599219,9cyclictest7372-21ssh12:25:482
2246599219,4cyclictest0-21swapper/212:05:192
2246599218,11cyclictest0-21swapper/211:00:012
22465992115,4cyclictest16737-21ssh11:55:242
22465992113,5cyclictest15311-21ssh11:19:002
22465992113,5cyclictest15200-21ssh09:13:122
2246599211,15cyclictest0-21swapper/207:40:152
2245699216,4cyclictest601-21rngd12:30:201
22456992115,4cyclictest21-21ksoftirqd/112:25:031
22456992115,4cyclictest21-21ksoftirqd/112:25:021
22456992114,5cyclictest1111-21sh10:30:171
2245699211,15cyclictest4060-21cpuspeed09:05:161
2245699211,14cyclictest15378-21latency_hist08:10:001
2244699218,8cyclictest0-21swapper/007:50:130
2244699214,3cyclictest601-21rngd10:45:160
2244699213,6cyclictest25609-21irqcore08:20:230
2244699213,4cyclictest151rcuc/009:40:130
2244699213,12cyclictest0-21swapper/007:20:160
2244699212,4cyclictest31494-21memory10:05:260
2244699211,5cyclictest601-21rngd11:20:160
22446992111,8cyclictest601-21rngd07:25:130
2244699211,16cyclictest11349-21cut12:30:180
22446992110,7cyclictest601-21rngd10:55:140
22474992014,4cyclictest0-21swapper/311:31:033
22474992014,4cyclictest0-21swapper/309:40:273
22474992013,5cyclictest0-21swapper/312:20:133
22474992013,5cyclictest0-21swapper/312:00:033
22474992013,4cyclictest0-21swapper/309:15:163
22474992012,6cyclictest0-21swapper/312:16:433
22474992011,6cyclictest0-21swapper/309:55:293
2247499201,12cyclictest25866irq/58-eth007:40:163
22474992010,6cyclictest321rcuc/308:30:183
2246599208,9cyclictest5322-21kthreadcore07:25:222
2246599208,9cyclictest21272-21ssh11:03:222
2246599207,5cyclictest28602-21latency_hist07:50:012
2246599204,8cyclictest28674-21kthreadcore07:15:212
2246599203,14cyclictest7398-21ssh11:50:252
2246599203,14cyclictest7115-21kthreadcore09:50:272
2246599201,16cyclictest30218-21ssh11:45:222
2246599201,12cyclictest6831-21kthreadcore08:00:182
2246599201,12cyclictest6831-21kthreadcore08:00:172
2246599201,10cyclictest7246-21ssh09:30:182
2246599201,10cyclictest31425-21kthreadcore08:25:212
2245699207,4cyclictest131rcu_preempt11:50:151
2245699202,13cyclictest31325-21ssh09:25:161
22456992012,6cyclictest22805-21apt-get11:05:171
22456992012,5cyclictest22478-21cron12:00:011
2244699207,9cyclictest20044-21cat10:40:370
2244699207,8cyclictest21931-21apt-get08:50:150
2244699205,4cyclictest601-21rngd07:35:180
2244699204,9cyclictest0-21swapper/008:10:140
2244699204,3cyclictest601-21rngd09:45:290
2244699203,4cyclictest601-21rngd11:11:480
2244699203,4cyclictest601-21rngd07:30:160
2244699202,5cyclictest12-21ksoftirqd/010:52:270
2244699202,3cyclictest601-21rngd12:00:220
2244699202,3cyclictest601-21rngd11:15:080
2244699201,4cyclictest601-21rngd10:25:200
22446992012,5cyclictest26771-21apt-get08:55:140
2244699201,15cyclictest4007-21apt-get09:05:150
2244699200,13cyclictest2592-21cpuspeed08:30:200
22474991913,4cyclictest0-21swapper/311:15:203
22474991913,4cyclictest0-21swapper/310:45:063
22474991913,4cyclictest0-21swapper/309:45:163
22474991912,5cyclictest0-21swapper/308:35:223
22474991912,5cyclictest0-21swapper/307:15:163
22474991911,5cyclictest0-21swapper/308:15:313
2247499191,14cyclictest25866irq/58-eth011:25:203
22474991910,6cyclictest25866irq/58-eth011:05:163
22474991910,6cyclictest0-21swapper/310:02:333
2246599192,9cyclictest27560-21kthreadcore08:20:262
2246599192,14cyclictest32372-21kthreadcore09:00:202
2246599191,9cyclictest900-21snmpd08:10:012
2246599191,9cyclictest31171-21munin-run10:50:002
2246599191,9cyclictest28211-21munin-plugin-st11:45:012
2246599191,4cyclictest28149-21ssh09:20:312
22465991912,5cyclictest7451-21ntp_kernel_err09:05:282
2246599191,15cyclictest30048-21cut07:50:222
2246599191,13cyclictest0-21swapper/208:50:152
2246599191,10cyclictest24328-21kthreadcore07:10:182
2246599191,10cyclictest22821-21ssh11:05:172
2246599191,10cyclictest16152-21cut08:10:202
2245699199,6cyclictest0-21swapper/109:35:371
2245699196,3cyclictest601-21rngd10:00:291
2245699191,16cyclictest28634-21latency_hist12:40:021
2245699191,15cyclictest31515-21hwlatdetect11:10:201
22456991910,6cyclictest19453-21cat11:00:361
2244699197,10cyclictest10269-21kthreadcore09:10:220
2244699194,10cyclictest12628-1kworker/u9:111:00:160
2244699193,3cyclictest12-21ksoftirqd/010:16:210
2244699192,13cyclictest16755-21apt-get08:45:050
22446991910,6cyclictest25024-21sshd08:16:350
2247499183,7cyclictest0-21swapper/307:30:153
2247499182,10cyclictest0-21swapper/308:55:163
2247499182,10cyclictest0-21swapper/308:50:043
2247499182,10cyclictest0-21swapper/308:40:163
2247499182,10cyclictest0-21swapper/308:10:163
22474991812,4cyclictest0-21swapper/307:45:203
2247499181,11cyclictest0-21swapper/307:35:153
2246599181,9cyclictest9588-21fschecks_count07:30:162
2246599181,9cyclictest1628-21grep07:55:182
2246599181,8cyclictest32224-21/usr/sbin/munin07:20:142
2245699183,9cyclictest0-21swapper/108:40:161
2245699182,3cyclictest131rcu_preempt12:00:171
22456991812,4cyclictest21-21ksoftirqd/109:55:351
22456991812,4cyclictest0-21swapper/109:33:411
2245699181,11cyclictest601-21rngd07:45:011
22456991810,6cyclictest16108-21cron11:20:021
2245699180,10cyclictest0-21swapper/110:20:021
2244699188,6cyclictest0-21swapper/007:45:160
2244699186,8cyclictest0-21swapper/009:00:120
2244699185,9cyclictest30730-21load07:15:240
2244699182,3cyclictest12-21ksoftirqd/009:20:160
22446991812,3cyclictest601-21rngd09:55:170
2244699181,12cyclictest1185-21ntp_states08:25:290
22446991810,5cyclictest601-21rngd08:35:150
2247499173,9cyclictest25866irq/58-eth007:55:123
22474991711,4cyclictest0-21swapper/309:05:203
22474991711,4cyclictest0-21swapper/308:25:303
22474991711,4cyclictest0-21swapper/308:05:013
22474991711,4cyclictest0-21swapper/308:05:013
22474991711,4cyclictest0-21swapper/307:20:183
22474991710,5cyclictest0-21swapper/307:10:163
2246599171,7cyclictest22486-21latency_hist07:10:022
2245699179,6cyclictest12728-21swap09:10:311
2245699174,5cyclictest131rcu_preempt10:45:291
2245699174,10cyclictest26410-21grep10:25:211
2245699173,7cyclictest601-21rngd08:20:181
2245699172,12cyclictest25795-21ssh12:20:151
2245699172,11cyclictest3252-21ntp_192.168.11407:20:261
2245699171,8cyclictest7989-21kthreadcore10:55:201
2245699171,7cyclictest4787-21ls11:50:011
22456991713,2cyclictest0-21swapper/109:20:121
22456991712,3cyclictest12445-21timerwakeupswit10:35:341
2245699171,13cyclictest17973-21apt-get10:20:161
2245699171,12cyclictest7317-21apt-get08:35:151
2245699171,12cyclictest4625-21ntp_states07:55:251
2245699171,12cyclictest2900-21ssh10:10:171
2245699171,10cyclictest601-21rngd08:10:301
22456991710,3cyclictest32001-21idleruntime-cro07:20:011
2244699171,12cyclictest9444-21ntp_states08:00:250
2244699171,12cyclictest9444-21ntp_states08:00:250
2245699169,5cyclictest711-21cron07:15:011
2245699168,6cyclictest23211-21cron10:45:001
2245699167,7cyclictest4971-21irqcore07:25:191
2245699166,8cyclictest2667-21tune2fs08:30:221
2245699163,11cyclictest12491-21ssh12:10:371
2245699162,11cyclictest1258-21grep12:05:251
2245699161,6cyclictest17006-21cpu12:15:171
2245699161,4cyclictest131rcu_preempt11:20:291
2245699161,11cyclictest654-21ntp_192.168.11409:45:291
22456991610,3cyclictest1324-21cron11:30:011
2245699160,12cyclictest14300-21apt-get07:35:151
2245699160,11cyclictest11422-21kthreadcore11:35:221
2245699160,10cyclictest601-21rngd10:05:271
2247499159,4cyclictest0-21swapper/309:00:183
2247499157,5cyclictest0-21swapper/308:20:163
2247499151,9cyclictest0-21swapper/307:25:173
2245699159,3cyclictest21-21ksoftirqd/108:00:171
2245699159,3cyclictest21-21ksoftirqd/108:00:171
2245699158,5cyclictest601-21rngd08:50:011
2245699152,11cyclictest11264-21kthreadcore07:30:211
2245699151,12cyclictest24584-21/usr/sbin/munin07:45:221
2245699151,10cyclictest31932-21grep10:50:221
2245699151,10cyclictest28875-21fschecks_count07:50:181
2244699153,9cyclictest22437-21cyclictest07:10:020
2247499149,3cyclictest0-21swapper/307:10:023
2245699141,10cyclictest31756-21fschecks_time09:00:171
2245699141,10cyclictest1588-21sed08:25:321
224569991,5cyclictest601-21rngd07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional