You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 22:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Dec 08, 2025 12:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32224997635,9cyclictest12-21ksoftirqd/010:20:140
32262996746,14cyclictest29566irq/58-eth012:05:173
3222499575,6cyclictest131rcu_preempt12:20:160
32224995433,15cyclictest18635-21diskmemload11:30:140
32262995130,14cyclictest29566irq/58-eth011:30:143
3223899512,35cyclictest6489-21ssh12:15:141
3223899504,29cyclictest27389-21ssh11:25:151
3223899492,40cyclictest9166-21ssh10:30:151
32238994919,11cyclictest131rcu_preempt09:40:151
3222499491,35cyclictest23362-21ssh09:35:150
3223899480,40cyclictest1430-21ssh09:20:141
3223899463,27cyclictest21-21ksoftirqd/107:50:131
3222499461,35cyclictest19780-21ssh11:20:140
3226299455,25cyclictest29566irq/58-eth011:15:133
3226299453,27cyclictest29566irq/58-eth008:35:123
3222499441,32cyclictest22943-21ssh12:05:150
3222499431,33cyclictest6913-21ssh12:35:140
3226299412,23cyclictest29566irq/58-eth008:40:133
3226299411,26cyclictest29566irq/58-eth011:00:153
32251994024,13cyclictest0-21swapper/211:40:162
3226299383,21cyclictest29566irq/58-eth010:10:143
3226299362,17cyclictest29566irq/58-eth009:50:123
3225199367,18cyclictest32162-21kworker/2:011:50:142
3223899362,26cyclictest30633-21ssh12:10:121
3222499365,23cyclictest662-21ntpd12:30:150
3225199352,13cyclictest261rcuc/208:40:132
3223899356,17cyclictest21-21ksoftirqd/109:45:151
3223899343,24cyclictest13004-21ssh09:50:131
3223899342,23cyclictest6928-21ssh12:35:151
32251993325,6cyclictest932-21snmpd09:25:142
3223899333,21cyclictest21-21ksoftirqd/110:15:161
3223899332,24cyclictest28291-21ssh10:00:151
3225199324,15cyclictest0-21swapper/208:55:132
3225199311,23cyclictest16983-21ssh10:35:152
3225199311,13cyclictest5320-21ssh09:45:132
3223899312,21cyclictest21-21ksoftirqd/110:40:141
3222499312,22cyclictest22766-21ssh12:25:130
3225199301,19cyclictest26346irq/40-dwc2_hso10:30:142
3223899303,19cyclictest17705-21ssh11:40:161
3223899301,21cyclictest29869-21ssh11:05:141
32251992919,6cyclictest0-21swapper/211:35:142
3226299282,17cyclictest29566irq/58-eth008:15:133
3225199285,14cyclictest26346irq/40-dwc2_hso11:00:172
3225199282,19cyclictest30204-21ssh09:40:152
3223899282,14cyclictest19151-21kworker/1:207:30:201
32224992817,5cyclictest0-21swapper/010:45:160
3222499281,19cyclictest25389-21ssh11:45:140
3226299274,18cyclictest29566irq/58-eth010:15:163
3225199274,16cyclictest26346irq/40-dwc2_hso09:00:162
3225199273,19cyclictest0-21swapper/207:10:142
3225199273,10cyclictest0-21swapper/208:05:042
3225199272,20cyclictest0-21swapper/212:10:132
3225199272,19cyclictest0-21swapper/208:10:122
3225199271,20cyclictest19767-21ssh11:20:132
3225199271,17cyclictest26346irq/40-dwc2_hso09:20:152
3225199263,17cyclictest26346irq/40-dwc2_hso12:15:142
3225199262,16cyclictest26346irq/40-dwc2_hso11:10:142
32251992618,6cyclictest0-21swapper/210:20:122
3225199261,14cyclictest12195-21ssh11:15:132
3223899264,6cyclictest131rcu_preempt07:10:141
3225199252,9cyclictest0-21swapper/207:15:132
3225199252,14cyclictest26346irq/40-dwc2_hso07:40:132
3223899253,15cyclictest20602-21ssh10:35:251
3222499251,19cyclictest6272-21apt-get07:50:140
3226299242,9cyclictest29566irq/58-eth009:15:153
3225199242,16cyclictest0-21swapper/210:40:142
3225199241,20cyclictest18367-21ls07:25:292
32251992410,10cyclictest932-21snmpd11:25:142
3222499248,7cyclictest12-21ksoftirqd/010:35:140
3222499248,4cyclictest131rcu_preempt11:55:270
3222499241,15cyclictest12-21ksoftirqd/009:40:150
32262992312,7cyclictest29566irq/58-eth011:05:133
3225199239,11cyclictest6359-21df_inode07:50:162
3225199232,9cyclictest0-21swapper/207:20:132
3225199232,4cyclictest0-21swapper/210:00:152
3225199232,13cyclictest0-21swapper/208:50:162
32251992312,9cyclictest19765-21ssh12:20:572
32251992312,9cyclictest17189-21sshd08:30:472
3225199231,14cyclictest135-21systemd-journal09:20:002
3223899232,15cyclictest842-21munin-node12:00:141
3223899232,12cyclictest21-21ksoftirqd/109:05:131
32238992311,9cyclictest27137-21sed11:25:011
3223899231,16cyclictest8340-21ssh11:10:251
3222499231,19cyclictest12983-21find09:50:120
3222499231,17cyclictest7728-21ls10:50:160
3222499231,17cyclictest14032-21apt-get09:05:130
3226299221,12cyclictest0-21swapper/312:30:133
32262992210,4cyclictest0-21swapper/310:35:143
3225199228,7cyclictest25854-21cut12:25:212
3225199227,7cyclictest30961-21tr11:05:192
3225199224,6cyclictest0-21swapper/210:45:162
3225199223,10cyclictest1638-21uniq07:45:152
3225199222,14cyclictest23361-21sh09:35:142
3225199221,9cyclictest243566ssh11:30:142
32251992211,7cyclictest19701-21kthreadcore10:15:242
3223899223,9cyclictest201rcuc/108:30:141
3223899222,17cyclictest25808-21ls08:10:111
3223899222,16cyclictest0-21swapper/107:40:141
32238992212,7cyclictest28194-21sshd12:06:231
3222499229,4cyclictest131rcu_preempt11:10:220
3222499222,15cyclictest668-21ssh11:50:150
3222499221,19cyclictest15258-21ssh10:55:150
3222499221,19cyclictest15258-21ssh10:55:150
3222499221,17cyclictest20778-21apt-get09:55:130
3225199219,5cyclictest13322-21/usr/sbin/munin09:50:182
3225199217,6cyclictest18221-1kworker/u9:209:55:302
3225199216,12cyclictest4398-21kthreadcore10:05:222
3225199216,12cyclictest4398-21kthreadcore10:05:222
3225199211,8cyclictest1967766ssh09:10:192
3225199211,14cyclictest18635-21diskmemload12:00:152
3225199211,10cyclictest13150-21cut10:10:212
3225199210,5cyclictest19381-21find07:30:122
3223899211,15cyclictest15502-21ssh10:10:321
3223899211,14cyclictest15574-21latency_hist09:30:021
3222499210,17cyclictest2276-21ls11:30:020
3222499210,17cyclictest15889-21ls09:30:150
3226299209,8cyclictest0-21swapper/311:56:453
3226299204,6cyclictest29666irq/59-eth010:05:153
3226299204,6cyclictest29666irq/59-eth010:05:143
3226299204,14cyclictest0-21swapper/311:45:133
32262992010,8cyclictest0-21swapper/311:42:073
3225199207,4cyclictest28713-21kthreadcore11:45:222
3225199207,10cyclictest11077-21latency08:25:192
3225199206,11cyclictest30560-21apt-get08:15:122
3225199201,9cyclictest25353-21ssh12:05:242
3225199201,9cyclictest2476-21kthreadcore10:25:212
3225199201,9cyclictest19091-21kthreadcore08:00:222
3225199201,9cyclictest14937-21latency_hist10:55:002
3225199201,8cyclictest6508-21cron12:35:012
3225199201,16cyclictest10349-21latency12:35:232
3225199201,15cyclictest17601-21fschecks_time08:35:152
3225199201,11cyclictest18835-21ntp_192.168.11410:55:262
3225199201,11cyclictest18835-21ntp_192.168.11410:55:252
3225199201,10cyclictest31754-21latency_hist08:50:012
3225199200,17cyclictest24469-21ls07:35:162
3223899202,3cyclictest131rcu_preempt08:50:151
3223899202,3cyclictest131rcu_preempt08:05:171
32238992013,5cyclictest0-21swapper/109:00:181
3223899201,2cyclictest131rcu_preempt11:58:231
3223899201,2cyclictest131rcu_preempt11:17:561
3223899201,17cyclictest27053-21ssh09:35:301
3223899201,16cyclictest2395-21ntp_states12:30:271
3223899201,14cyclictest17487-21wc11:40:021
3223899201,11cyclictest3903-21tr07:45:211
32238992010,7cyclictest21-21ksoftirqd/111:00:051
3222499206,3cyclictest131rcu_preempt11:05:220
3222499203,12cyclictest12174-21ssh11:15:120
3222499202,15cyclictest5617-21kthreadcore08:55:210
3222499201,16cyclictest1634-21cut09:20:170
3222499201,15cyclictest29065-21apt-get07:40:130
3222499201,15cyclictest15416-21apt-get12:00:140
32224992010,7cyclictest4940-21cron09:45:020
3226299191,13cyclictest0-21swapper/309:15:023
3226299191,13cyclictest0-21swapper/308:25:263
3225199199,8cyclictest0-21swapper/209:05:042
3225199192,9cyclictest13253-21kthreadcore07:55:232
3225199191,8cyclictest6508-21grep08:20:282
3225199191,7cyclictest1580366ssh09:30:042
3225199191,15cyclictest8275-21fschecks_count11:55:182
3223899196,2cyclictest131rcu_preempt10:54:151
3223899195,10cyclictest22433-21http11:00:021
3223899195,10cyclictest22433-21http11:00:021
3223899192,4cyclictest53046usb-storage07:20:111
3223899191,9cyclictest131rcu_preempt10:27:391
3223899191,8cyclictest131rcu_preempt11:33:391
32238991913,4cyclictest0-21swapper/107:15:161
32238991911,6cyclictest21994-21cron08:40:011
32238991911,5cyclictest25358-21acpi11:45:131
3223899191,14cyclictest30116-21ntp_192.168.11409:15:281
3223899191,13cyclictest0-21swapper/108:20:151
3223899191,11cyclictest842-21munin-node09:55:141
3223899190,13cyclictest7918-21grep08:25:141
3222499195,11cyclictest13325-21cut11:35:230
3222499192,14cyclictest1541-21diskstats10:25:170
3222499191,3cyclictest131rcu_preempt10:30:160
32224991911,6cyclictest1-21systemd09:10:530
3222499191,14cyclictest27075-21apt-get08:45:140
3222499191,14cyclictest26549-21ssh09:15:170
32224991910,7cyclictest28268-21sed10:40:240
3226299188,8cyclictest0-21swapper/310:51:193
3226299188,8cyclictest0-21swapper/310:43:043
3226299184,7cyclictest29566irq/58-eth010:30:143
3226299182,10cyclictest0-21swapper/308:55:143
3226299182,10cyclictest0-21swapper/307:20:043
32262991811,5cyclictest29566irq/58-eth012:00:143
3226299181,12cyclictest0-21swapper/311:25:203
3226299181,12cyclictest0-21swapper/308:50:183
3226299181,12cyclictest0-21swapper/308:00:163
3226299181,11cyclictest29566irq/58-eth010:00:133
3226299181,11cyclictest0-21swapper/311:35:153
3226299181,10cyclictest0-21swapper/307:55:203
32262991810,6cyclictest0-21swapper/312:13:523
32262991810,6cyclictest0-21swapper/310:46:333
32262991810,6cyclictest0-21swapper/310:29:123
32262991810,6cyclictest0-21swapper/309:22:533
32262991810,5cyclictest29566irq/58-eth012:25:123
3223899188,6cyclictest358866sshd10:05:101
3223899188,6cyclictest358866sshd10:05:101
3223899183,2cyclictest131rcu_preempt09:14:381
3223899182,12cyclictest760-21df11:50:151
3223899181,9cyclictest131rcu_preempt10:47:011
3223899181,4cyclictest131rcu_preempt12:29:191
32238991810,6cyclictest20933-21sshd09:31:101
3222499189,6cyclictest18635-21diskmemload11:45:020
3222499188,7cyclictest31885-21apt-get08:50:150
3222499188,7cyclictest19490-21diskstats07:30:150
3222499188,3cyclictest12-21ksoftirqd/008:40:120
3222499186,8cyclictest12-21ksoftirqd/008:05:020
3222499185,10cyclictest32119-21sh12:10:200
3222499183,12cyclictest7813-21cpu08:25:120
3222499182,9cyclictest0-21swapper/007:15:130
3222499182,14cyclictest17270-21latency_hist08:35:010
3222499182,13cyclictest25065-21kthreadcore07:35:200
3222499182,12cyclictest18635-21diskmemload10:15:160
3222499181,14cyclictest31969-21ntp_states10:00:270
32224991810,5cyclictest11976-21open_files09:25:280
3222499180,12cyclictest14582-21acpi07:25:130
3226299179,6cyclictest0-21swapper/312:19:483
3226299179,6cyclictest0-21swapper/311:50:193
3226299179,6cyclictest0-21swapper/311:13:523
3226299179,6cyclictest0-21swapper/310:56:073
3226299179,6cyclictest0-21swapper/310:56:063
3226299179,6cyclictest0-21swapper/309:44:093
3226299179,6cyclictest0-21swapper/309:37:343
3226299179,6cyclictest0-21swapper/309:26:333
3226299172,9cyclictest0-21swapper/309:30:143
3226299172,10cyclictest0-21swapper/312:20:173
3226299172,10cyclictest0-21swapper/310:20:143
3226299172,10cyclictest0-21swapper/308:05:053
32262991711,4cyclictest29666irq/59-eth011:20:173
3226299171,11cyclictest0-21swapper/309:55:243
3226299171,11cyclictest0-21swapper/307:45:233
3226299171,10cyclictest0-21swapper/309:00:173
3226299171,10cyclictest0-21swapper/307:10:033
3225199171,9cyclictest32250-21touch07:10:002
32238991711,4cyclictest0-21swapper/108:45:201
3223899171,13cyclictest4956-21grep08:55:201
3223899171,12cyclictest25698-21apt-get10:20:041
3223899170,10cyclictest11428-21ls07:55:201
3222499179,5cyclictest13365-21kthreadcore07:55:240
3222499175,9cyclictest4560-21kthreadcore07:45:220
3222499173,10cyclictest12-21ksoftirqd/008:05:050
3222499172,10cyclictest0-21swapper/008:20:150
3222499171,8cyclictest6497-21wc12:15:140
3222499171,7cyclictest463-21irqrtprio07:10:190
3222499171,13cyclictest28705-21tr08:10:200
32224991711,3cyclictest18635-21diskmemload10:10:270
32224991710,5cyclictest9651-21sed09:00:210
32224991710,4cyclictest19290-21kthreadcore08:35:200
3222499170,10cyclictest3644-21ssh10:05:150
3222499170,10cyclictest3644-21ssh10:05:140
3226299161,9cyclictest0-21swapper/309:05:283
3226299161,12cyclictest0-21swapper/308:20:173
32262991610,3cyclictest0-21swapper/312:35:073
3223899163,11cyclictest17598-21tr08:00:191
3223899161,12cyclictest30877-21ps08:15:181
3223899161,11cyclictest24921-21kthreadcore07:35:191
3223899161,11cyclictest14724-21fschecks_count07:25:171
3223899161,11cyclictest14647-21/usr/sbin/munin12:20:281
3223899161,10cyclictest22400-21cut08:40:151
3222499166,7cyclictest10920-21kthreadcore07:20:180
3222499162,10cyclictest0-21swapper/008:15:140
3222499160,13cyclictest23406-21sed11:00:220
3226299151,10cyclictest0-21swapper/307:50:153
32262991510,3cyclictest0-21swapper/309:46:133
3223899151,10cyclictest32131-21apt-key07:10:011
3222499152,10cyclictest32232-21grep07:10:010
3226299149,3cyclictest0-21swapper/307:18:493
3226299141,9cyclictest0-21swapper/308:30:303
3226299141,7cyclictest0-21swapper/308:10:253
3226299131,6cyclictest0-21swapper/307:40:283
3226299127,3cyclictest0-21swapper/307:30:213
3226299122,6cyclictest0-21swapper/307:25:143
3226299116,3cyclictest0-21swapper/308:47:593
3226299111,7cyclictest0-21swapper/307:35:193
3226299101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional