You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 15:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Nov 26, 2025 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30410995423,9cyclictest131rcu_preempt08:55:141
3039599504,5cyclictest131rcu_preempt10:00:110
3041099462,38cyclictest9397-21ssh12:15:141
30395994328,11cyclictest0-21swapper/010:40:150
3042099411,29cyclictest32573-21ssh09:20:122
3041099402,31cyclictest1510-21rm11:50:141
3041099394,23cyclictest21-21ksoftirqd/109:00:111
3041099394,23cyclictest21-21ksoftirqd/109:00:111
3041099391,30cyclictest17026-21ssh12:00:121
3043299389,15cyclictest29566irq/58-eth012:15:163
3043299382,24cyclictest29566irq/58-eth012:05:143
3041099382,32cyclictest13480-21ssh10:55:151
3041099379,22cyclictest21-21ksoftirqd/112:05:151
3042099362,27cyclictest25795-21apt-get12:25:042
3041099355,20cyclictest4395-21ssh11:10:121
3042099336,16cyclictest26346irq/40-dwc2_hso08:45:142
3042099325,15cyclictest26346irq/40-dwc2_hso07:50:122
3042099313,15cyclictest18848-21ssh09:55:132
30410993117,9cyclictest0-21swapper/109:35:131
3039599312,23cyclictest26256-21ssh12:25:140
3039599312,22cyclictest23380-21sh12:02:490
3042099305,19cyclictest24875-21diskmemload11:20:492
3042099305,15cyclictest26346irq/40-dwc2_hso07:35:122
3042099304,20cyclictest26346irq/40-dwc2_hso11:00:132
3042099301,17cyclictest1599-21ssh12:30:132
3041099302,22cyclictest16540-21ssh12:37:491
30410993016,10cyclictest0-21swapper/108:35:121
3043299296,14cyclictest29666irq/59-eth012:20:133
3042099295,16cyclictest26346irq/40-dwc2_hso08:50:122
3042099293,20cyclictest0-21swapper/212:01:492
3042099292,20cyclictest0-21swapper/209:15:152
3039599292,2cyclictest131rcu_preempt11:00:130
3039599292,19cyclictest18853-21ssh09:55:140
3042099283,18cyclictest26346irq/40-dwc2_hso10:40:152
3042099282,20cyclictest1676-21ssh11:05:492
3043299277,11cyclictest0-21swapper/310:35:133
3042099273,16cyclictest26346irq/40-dwc2_hso09:25:162
3042099272,9cyclictest0-21swapper/209:05:132
3042099271,16cyclictest26047-21ssh10:00:142
3041099279,12cyclictest21-21ksoftirqd/112:30:051
3041099272,20cyclictest31390-21ssh10:25:141
3041099272,17cyclictest4917-21ssh10:05:261
3041099271,21cyclictest5726-21ssh10:50:141
3039599271,20cyclictest25294-21ssh09:15:150
3042099262,19cyclictest0-21swapper/211:15:492
3041099269,12cyclictest21-21ksoftirqd/111:55:151
3041099263,13cyclictest16090-21diskmemload10:15:131
3041099263,13cyclictest16090-21diskmemload10:15:131
3041099262,19cyclictest19558-21ssh11:19:571
3041099262,14cyclictest0-21swapper/110:35:131
3039599262,19cyclictest30998-21ssh12:06:490
3039599262,18cyclictest26547-21ssh11:45:140
3039599261,3cyclictest131rcu_preempt08:40:150
3043299252,17cyclictest0-21swapper/311:35:143
3043299252,17cyclictest0-21swapper/311:35:143
3043299252,17cyclictest0-21swapper/311:35:003
3043299252,17cyclictest0-21swapper/311:35:003
30432992510,11cyclictest0-21swapper/311:20:173
3042099257,13cyclictest27-21ksoftirqd/208:25:132
3042099252,8cyclictest31612-21ssh12:07:492
3042099251,9cyclictest0-21swapper/211:25:152
3041099257,2cyclictest131rcu_preempt10:24:491
3041099252,8cyclictest0-21swapper/111:26:481
30410992510,9cyclictest0-21swapper/112:20:131
3043299243,14cyclictest0-21swapper/312:37:493
3042099243,6cyclictest0-21swapper/211:53:502
3042099243,15cyclictest26346irq/40-dwc2_hso11:45:132
3042099242,17cyclictest1205-21ssh12:10:142
3042099242,16cyclictest0-21swapper/211:10:502
3042099242,15cyclictest0-21swapper/212:20:042
3042099241,18cyclictest18708-21ssh11:40:142
3042099241,12cyclictest1548066ssh11:35:312
3042099241,12cyclictest1548066ssh11:35:312
3041099248,4cyclictest21-21ksoftirqd/108:00:171
3041099246,15cyclictest21-21ksoftirqd/111:05:171
3041099240,13cyclictest2782-21latency_hist08:55:021
3039599240,3cyclictest131rcu_preempt07:45:130
3043299233,14cyclictest0-21swapper/312:00:123
3043299233,13cyclictest0-21swapper/311:45:013
3042099238,12cyclictest135-21systemd-journal10:06:032
3042099236,8cyclictest28781-21apt-key08:15:012
3042099232,6cyclictest0-21swapper/211:59:492
3042099232,16cyclictest6830-21ssh10:30:052
3041099231,17cyclictest12697-21apt-get09:05:121
3041099231,16cyclictest6942-21ssh10:30:151
3041099230,10cyclictest21-21ksoftirqd/110:10:131
3039599232,14cyclictest0-21swapper/007:35:130
30395992312,6cyclictest0-21swapper/011:20:490
3039599231,18cyclictest4951-21ls11:50:230
3039599231,18cyclictest4355-21cut09:45:180
3043299224,11cyclictest33-21ksoftirqd/310:10:133
3043299222,12cyclictest29566irq/58-eth010:55:143
30432992212,7cyclictest29566irq/58-eth011:45:143
3042099229,8cyclictest6730-21latency_hist10:30:022
3042099224,9cyclictest5353-21cron10:50:002
3042099222,6cyclictest0-21swapper/212:35:052
3042099222,15cyclictest0-21swapper/212:19:492
3042099221,11cyclictest21174-21grep07:30:262
3042099221,10cyclictest10431-21apt-config11:35:022
3042099221,10cyclictest10431-21apt-config11:35:012
3041099222,15cyclictest0-21swapper/111:36:501
3041099222,15cyclictest0-21swapper/111:36:491
3041099221,16cyclictest11591-21ls09:50:171
30410992210,8cyclictest29907-21sed10:00:271
3039599224,14cyclictest6952-21ssh10:30:170
3039599222,15cyclictest11334-21ssh09:50:140
30432992113,5cyclictest29566irq/58-eth012:30:133
30432992111,4cyclictest0-21swapper/311:10:273
3043299211,13cyclictest29566irq/58-eth010:15:133
3043299211,13cyclictest29566irq/58-eth010:15:133
3042099214,14cyclictest20609-21cron08:40:012
3042099212,9cyclictest12771-21wc07:55:222
3042099212,14cyclictest4183-21ssh09:45:152
3042099212,11cyclictest21624-21munin-run09:35:022
3042099212,10cyclictest16846-21kthreadcore10:35:202
3042099211,9cyclictest23785-21cron08:10:012
3042099211,9cyclictest15113-21mailstats09:50:292
3042099211,8cyclictest17670-21ssh09:10:152
3041099218,4cyclictest131rcu_preempt08:40:281
3041099211,9cyclictest131rcu_preempt09:10:161
3041099211,2cyclictest131rcu_preempt07:20:321
30410992110,8cyclictest4902-21sh12:10:251
30410992110,8cyclictest18845-21apt-get09:55:131
3041099210,2cyclictest131rcu_preempt11:42:051
3039599212,14cyclictest8842-21ssh12:14:490
3039599211,8cyclictest0-21swapper/009:35:430
3039599211,17cyclictest3688-21grep07:15:210
30395992111,6cyclictest17302-21diskmemload11:15:140
3039599211,14cyclictest18351-21munin-run11:40:010
3039599211,14cyclictest18351-21munin-run11:40:010
3039599211,12cyclictest12837-21fschecks_count07:25:140
3043299209,9cyclictest0-21swapper/311:02:443
3043299204,11cyclictest0-21swapper/310:50:123
3043299203,12cyclictest0-21swapper/311:19:583
30432992012,5cyclictest0-21swapper/309:55:023
30432992011,6cyclictest29566irq/58-eth010:40:163
3043299201,13cyclictest0-21swapper/309:35:163
3042099206,6cyclictest24031-21hwlatdetect10:20:172
3042099203,14cyclictest8462-21/usr/sbin/munin10:10:182
3042099202,9cyclictest932-21snmpd07:15:152
3042099202,9cyclictest17310-21apt-get07:30:022
3042099202,4cyclictest0-21swapper/207:45:142
3042099202,10cyclictest30825-21kthreadcore08:15:202
3042099202,10cyclictest22222-21kthreadcore08:40:242
3042099202,10cyclictest15439-21kthreadcore10:55:212
3042099201,9cyclictest30442-21grep07:10:022
3042099201,12cyclictest7845-21apt-get07:20:142
3042099201,11cyclictest14539-21grep08:00:172
3042099201,10cyclictest22514-21sed09:35:172
30420992010,7cyclictest0-21swapper/207:40:282
3042099200,12cyclictest30727-21cut07:10:182
3041099208,8cyclictest21-21ksoftirqd/108:10:131
3041099207,11cyclictest25587-21tr10:40:241
3041099207,10cyclictest29553-21tail08:45:301
3041099202,9cyclictest131rcu_preempt11:31:271
3041099202,9cyclictest131rcu_preempt11:31:261
3041099202,15cyclictest8325-21find07:50:421
3041099201,9cyclictest131rcu_preempt12:25:371
3041099201,2cyclictest131rcu_preempt09:20:361
3041099201,16cyclictest17605-21acpi07:30:121
30410992010,8cyclictest12090-21cat09:25:351
3039599204,4cyclictest131rcu_preempt08:35:260
3039599203,5cyclictest12-21ksoftirqd/007:55:290
3039599202,14cyclictest8145-21munin-run10:10:010
3039599201,9cyclictest0-21swapper/012:19:180
3039599201,7cyclictest10721-21tr09:25:250
3039599201,3cyclictest0-21swapper/011:10:360
3039599201,2cyclictest131rcu_preempt10:35:130
3039599201,15cyclictest9568-21cut11:55:170
30395992011,5cyclictest121ksoftirqd/011:25:140
30395992010,7cyclictest8047-21perl07:20:180
3043299199,8cyclictest0-21swapper/309:46:503
3043299198,8cyclictest0-21swapper/311:50:213
3043299193,7cyclictest29566irq/58-eth007:10:163
3043299193,11cyclictest0-21swapper/310:25:143
30432991912,3cyclictest0-21swapper/309:20:123
3043299191,13cyclictest0-21swapper/312:25:203
3043299191,13cyclictest0-21swapper/308:45:313
3043299191,13cyclictest0-21swapper/307:50:013
30432991910,6cyclictest0-21swapper/311:25:313
30432991910,6cyclictest0-21swapper/310:30:203
3042099199,3cyclictest5719-21sh10:50:122
3042099191,9cyclictest5774-21kthreadcore08:55:222
3042099191,9cyclictest3175-21kthreadcore08:20:222
3042099191,8cyclictest13298-21kthreadcore08:30:222
3042099191,15cyclictest8360-21grep09:00:162
3042099191,15cyclictest8360-21grep09:00:162
3042099191,11cyclictest16084-21apt-get10:15:122
3042099191,11cyclictest16084-21apt-get10:15:122
3042099190,9cyclictest30137-21kthreadcore09:40:212
3041099199,7cyclictest8377-21cat09:45:311
3041099195,11cyclictest28631-21kthreadcore09:15:231
3041099193,9cyclictest0-21swapper/111:50:031
3041099193,12cyclictest13241-21timerwakeupswit07:55:281
3041099191,9cyclictest131rcu_preempt11:22:171
30410991912,4cyclictest0-21swapper/111:00:151
30410991911,6cyclictest6451-21cut08:25:151
30410991911,6cyclictest1-21systemd07:19:151
30410991910,6cyclictest0-21swapper/108:05:261
30395991910,4cyclictest12-21ksoftirqd/007:10:140
3039599190,2cyclictest131rcu_preempt10:15:200
3039599190,2cyclictest131rcu_preempt10:15:200
3039599190,16cyclictest3088-21ls08:55:140
3043299188,6cyclictest0-21swapper/310:09:253
3043299183,12cyclictest0-21swapper/311:55:133
3043299183,10cyclictest0-21swapper/312:10:033
3043299182,9cyclictest0-21swapper/308:40:153
3043299182,12cyclictest0-21swapper/310:45:223
3043299182,11cyclictest0-21swapper/311:07:483
3043299182,10cyclictest0-21swapper/309:40:143
3043299181,13cyclictest0-21swapper/308:20:173
3043299181,10cyclictest0-21swapper/308:55:013
3041099188,7cyclictest14879-21cut09:30:161
3041099183,12cyclictest29352-21kthreadcore07:40:211
3041099182,10cyclictest0-21swapper/109:40:141
30410991813,3cyclictest0-21swapper/107:10:021
30410991811,5cyclictest4399-21idleruntime-cro07:50:011
3041099181,13cyclictest26361-21unixbench_singl07:35:281
3041099181,13cyclictest1154-21run-parts08:20:011
3041099180,3cyclictest131rcu_preempt10:48:371
3039599182,11cyclictest7391-21ssh12:31:490
3039599182,10cyclictest0-21swapper/009:05:130
3039599181,9cyclictest3310-21irqrtprio11:30:170
3039599181,9cyclictest3310-21irqrtprio11:30:170
3039599181,8cyclictest29712-21irqrtprio11:05:170
3039599181,14cyclictest18707-21ssh11:40:140
30395991810,6cyclictest29011-21sed08:45:220
30395991810,6cyclictest17982-21cut09:30:220
30395991810,5cyclictest32543-21/usr/sbin/munin09:20:300
3039599180,13cyclictest5848-21uniq10:50:170
3043299179,6cyclictest0-21swapper/310:21:353
3043299179,6cyclictest0-21swapper/309:30:033
3043299175,6cyclictest0-21swapper/309:10:143
3043299172,9cyclictest0-21swapper/308:15:133
3043299172,8cyclictest0-21swapper/309:05:123
3043299172,11cyclictest0-21swapper/309:55:193
3043299172,10cyclictest0-21swapper/309:00:123
3043299172,10cyclictest0-21swapper/309:00:123
3043299172,10cyclictest0-21swapper/308:25:043
3043299171,8cyclictest0-21swapper/307:35:023
3043299171,12cyclictest0-21swapper/310:00:203
30432991710,4cyclictest0-21swapper/309:25:253
3041099171,2cyclictest131rcu_preempt08:30:151
3041099171,13cyclictest2659-21kthreadcore08:20:201
3041099171,12cyclictest17314-21latency_hist07:30:021
3039599179,6cyclictest28222-21cat08:10:290
3039599175,3cyclictest131rcu_preempt10:55:160
3039599172,3cyclictest12-21ksoftirqd/012:35:120
3039599172,10cyclictest0-21swapper/008:30:140
3039599171,8cyclictest17604-21/usr/sbin/munin07:30:270
3039599171,3cyclictest131rcu_preempt10:25:140
30395991711,3cyclictest18042-21fschecks_count12:20:150
3039599171,12cyclictest29187-21df_inode09:40:150
3039599170,12cyclictest6504-21uniq08:25:160
3043299168,5cyclictest29566irq/58-eth009:15:143
3043299165,7cyclictest33-21ksoftirqd/308:35:153
3043299162,9cyclictest0-21swapper/308:00:143
3043299161,12cyclictest33-21ksoftirqd/307:55:153
3039599169,5cyclictest19021-21cut08:05:010
3039599169,4cyclictest4780-21fschecks_time07:50:160
3039599169,4cyclictest0-21swapper/009:10:170
3039599167,7cyclictest7866-21apt-get09:00:120
3039599167,7cyclictest7866-21apt-get09:00:110
3039599163,8cyclictest0-21swapper/008:50:110
3039599163,10cyclictest9412-21tr10:10:190
3039599162,9cyclictest0-21swapper/010:45:140
30395991610,4cyclictest0-21swapper/008:20:240
3043299152,9cyclictest0-21swapper/308:05:153
3043299151,6cyclictest0-21swapper/308:30:293
3043299151,10cyclictest0-21swapper/307:35:143
3039599159,4cyclictest19986-21cut08:05:210
3039599156,6cyclictest29785-21kthreadcore08:15:190
3039599151,10cyclictest23935-21cut10:20:160
3039599150,12cyclictest27334-21find07:40:120
3043299149,3cyclictest33-21ksoftirqd/307:54:203
3043299149,3cyclictest0-21swapper/308:10:113
3043299142,7cyclictest0-21swapper/309:00:023
3041099141,10cyclictest30440-21latency_hist07:10:011
3039599144,7cyclictest30399-21latency_hist07:10:010
3043299138,3cyclictest33-21ksoftirqd/307:18:233
3043299134,7cyclictest33-21ksoftirqd/307:25:253
3043299131,10cyclictest0-21swapper/307:20:213
3043299121,8cyclictest0-21swapper/307:45:013
3043299101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional