You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-04 - 07:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Nov 04, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15129997358,10cyclictest25366irq/58-eth021:30:173
15129995722,22cyclictest25366irq/58-eth000:25:153
1511099470,4cyclictest17328-21apt-get21:20:161
1511099452,2cyclictest12650-21apt-get20:15:151
1511099432,37cyclictest18678-21apt-get20:55:141
15120994215,24cyclictest0-21swapper/221:20:162
1510299411,28cyclictest890-21ssh23:10:140
1510299410,34cyclictest26864-21ssh23:45:160
1510299410,34cyclictest26864-21ssh23:45:160
1512099407,17cyclictest9131-21ssh21:15:202
1511099404,32cyclictest0-21swapper/123:05:141
1511099401,34cyclictest0-21swapper/100:30:121
1512999392,22cyclictest25366irq/58-eth022:00:133
15120993930,6cyclictest535-21apt-get21:50:142
1511099391,34cyclictest7618-21ssh22:35:131
1511099391,34cyclictest0-21swapper/121:35:141
1511099391,2cyclictest0-21swapper/121:45:141
1511099391,25cyclictest19610-21kthreadcore00:00:231
1511099390,7cyclictest21455-21if_eth119:50:181
15102993912,21cyclictest12-21ksoftirqd/019:30:140
1511099381,34cyclictest0-21swapper/123:50:141
1511099381,33cyclictest0-21swapper/121:25:151
15120993721,12cyclictest0-21swapper/221:45:142
1511099371,24cyclictest15125-21ssh00:39:451
1512999363,25cyclictest25366irq/58-eth023:20:143
15120993627,6cyclictest10684-21apt-get23:55:172
1511099361,30cyclictest0-21swapper/122:40:131
1511099361,2cyclictest27065-21apt-get20:30:141
1511099361,25cyclictest16450-21df_inode19:45:161
1511099361,23cyclictest357-21ssh23:47:201
1511099361,23cyclictest357-21ssh23:47:201
1511099360,2cyclictest17422-21ssh23:20:131
1511099360,10cyclictest1584-21apt-get23:30:141
1510299361,25cyclictest16574-21ssh22:00:130
1510299361,23cyclictest31955-21ssh22:30:160
1512999351,19cyclictest25366irq/58-eth022:20:143
1512099358,17cyclictest16536-21ssh22:00:112
15120993522,8cyclictest32627-21ssh00:30:132
15120993521,9cyclictest16866-21ssh23:38:282
15120993521,11cyclictest21846irq/40-dwc2_hso20:05:132
1511099351,6cyclictest14327-21sed20:50:201
1511099350,2cyclictest0-21swapper/123:55:171
15102993520,12cyclictest25174-21apt-get21:45:140
1512099344,14cyclictest8732-21irqrtprio22:15:212
15120993422,9cyclictest0-21swapper/222:40:132
1511099344,28cyclictest0-21swapper/119:25:111
1511099342,29cyclictest24690-21apt-get22:05:141
1511099341,6cyclictest11470-21latency_hist19:40:011
1511099341,30cyclictest25817-21df23:25:141
1511099341,30cyclictest0-21swapper/122:15:151
1511099341,2cyclictest20058-21apt-get19:15:141
1511099341,2cyclictest20058-21apt-get19:15:141
1511099341,20cyclictest0-21swapper/121:15:431
1511099340,2cyclictest559-21ssh21:50:141
15129993324,6cyclictest321rcuc/321:55:143
1512099337,12cyclictest0-21swapper/200:07:252
1511099333,25cyclictest31930-21apt-get20:35:141
1511099331,3cyclictest0-21swapper/122:30:191
1511099331,19cyclictest31818-21ssh00:29:101
1511099331,13cyclictest17480-21apt-get20:20:141
1511099330,2cyclictest0-21swapper/123:00:161
1512099323,13cyclictest0-21swapper/219:20:272
15120993224,6cyclictest7890-21apt-get00:35:142
15120993215,10cyclictest0-21swapper/223:05:142
1511099325,23cyclictest19826-21kthreadcore23:40:191
1511099322,25cyclictest22258-21apt-get20:25:141
1511099321,7cyclictest30766-21apt-get20:00:131
1511099321,6cyclictest19763-21kthreadcore22:00:221
1511099321,2cyclictest9953-21apt-get23:35:141
1511099321,2cyclictest15884-21apt-get22:20:141
1511099321,2cyclictest0-21swapper/121:13:141
1511099321,28cyclictest0-21swapper/119:10:161
1511099321,27cyclictest26587-21cut19:55:201
1511099321,27cyclictest15033-21ntp_states19:40:281
1511099320,2cyclictest0-21swapper/119:20:121
15102993219,7cyclictest0-21swapper/000:25:150
15102993215,12cyclictest778-21apt-get21:10:150
1512099314,13cyclictest10265-21ntpq19:35:272
1512099313,17cyclictest0-21swapper/222:51:172
15120993121,8cyclictest17411-21apt-get23:20:132
1511099311,9cyclictest15655-21ssh22:59:041
1511099311,9cyclictest13860-21ssh00:15:351
1511099311,6cyclictest16702-21ssh23:19:351
1511099311,3cyclictest4630-21ntp_states21:30:311
1511099311,2cyclictest28283-21apt-get21:05:141
1511099311,27cyclictest6026-21sendmail_mailst00:10:301
1511099311,27cyclictest1590-21/usr/sbin/munin23:10:191
1511099311,27cyclictest11272-21ntp_192.168.11420:10:251
1511099311,26cyclictest17600-21cut00:20:181
1511099311,25cyclictest20694-21ntp_192.168.11421:40:251
1511099310,3cyclictest0-21swapper/122:45:141
1511099310,2cyclictest32470-21apt-get22:10:151
1511099310,2cyclictest0-21swapper/122:25:151
1511099310,28cyclictest2247-21/usr/sbin/munin19:30:181
15129993019,8cyclictest25366irq/58-eth023:10:143
1512099305,15cyclictest0-21swapper/200:02:082
15120993024,2cyclictest16363-21apt-get19:45:142
15120993021,6cyclictest877-21apt-get23:10:142
15120993021,6cyclictest31947-21apt-get22:30:182
15120993020,7cyclictest2586-21apt-get23:50:142
15120993011,8cyclictest31400-21ssh00:28:272
15120993010,16cyclictest0-21swapper/220:30:152
15120993010,11cyclictest370-21diskmemload00:12:552
1511099301,5cyclictest27142-21idleruntime00:05:171
1511099301,26cyclictest8661-21fw_packets21:55:171
1511099300,6cyclictest6735-21ssh22:53:081
1510299304,17cyclictest15907-21ssh22:20:150
1512099296,11cyclictest7269-21ssh23:31:432
1512099293,12cyclictest0-21swapper/223:00:442
1512099292,17cyclictest0-21swapper/221:30:082
15120992920,6cyclictest23875-21apt-get22:25:162
15120992919,7cyclictest0-21swapper/220:35:132
15120992916,10cyclictest0-21swapper/219:40:172
15120992915,7cyclictest26687-21latency_hist23:45:022
15120992913,8cyclictest17462-21munin-node20:20:142
15120992912,8cyclictest23904-21head19:15:302
15120992912,8cyclictest23904-21head19:15:302
15120992912,14cyclictest31746-21ssh21:28:112
15120992912,14cyclictest11934-21kthreadcore00:15:222
15120992911,16cyclictest0-21swapper/220:25:142
1511099291,2cyclictest0-21swapper/121:00:121
1511099291,25cyclictest3210-21df_inode20:05:141
1511099291,24cyclictest4358-21ls20:40:161
1511099290,6cyclictest15093-21cyclictest19:10:011
1512999282,14cyclictest25366irq/58-eth023:40:123
1512999281,17cyclictest25366irq/58-eth022:55:163
1512999281,15cyclictest25366irq/58-eth000:20:143
1512099284,15cyclictest15978-21ssh21:59:432
1512099284,12cyclictest0-21swapper/219:25:142
15120992818,7cyclictest25735-21ssh23:25:132
15120992813,12cyclictest2997-21sort20:05:012
15120992813,12cyclictest28231-21grep22:45:302
15120992813,12cyclictest19911-21ssh22:20:312
15120992813,12cyclictest10640-21kthreadcore23:15:212
15120992812,7cyclictest19949-21latency_hist19:15:012
15120992812,13cyclictest4287-21cat21:10:252
15120992812,13cyclictest32167-21uptime21:05:302
15120992812,12cyclictest4527-21wc20:40:182
15120992811,8cyclictest11420-21cut22:55:252
1511099280,9cyclictest9066-21grep20:45:151
1511099280,9cyclictest9066-21grep20:45:151
1512999273,18cyclictest0-21swapper/300:08:333
1512999273,15cyclictest0-21swapper/322:05:153
15120992712,13cyclictest0-21swapper/222:05:162
15120992711,8cyclictest23381-21wc21:00:032
15120992711,13cyclictest32649-21fschecks_count22:10:172
1510299275,6cyclictest31804-21latency_hist20:35:010
1510299275,13cyclictest629-21Async08:52:160
1510299275,13cyclictest629-21Async08:52:160
1510299270,8cyclictest3492-21cut21:50:230
1512999261,15cyclictest25366irq/58-eth020:00:133
1512099266,17cyclictest0-21swapper/219:55:142
15120992612,11cyclictest11330-21ntp_states20:10:262
15120992611,8cyclictest7983-21if_eth122:35:192
15120992611,7cyclictest16245-21ssh21:39:242
15120992611,12cyclictest6698-21cat19:35:022
15120992611,12cyclictest23683-21rm00:23:442
15120992611,12cyclictest21149-21/usr/sbin/munin19:50:242
15120992610,8cyclictest26715-21rm23:45:042
15120992610,8cyclictest26715-21rm23:45:042
15120992610,8cyclictest22146-21ssh21:40:522
1510299265,5cyclictest12-21ksoftirqd/000:39:410
1510299262,19cyclictest32737-21ssh23:28:420
1512099259,13cyclictest21846irq/40-dwc2_hso20:50:162
1512099259,13cyclictest16082-21ntpq20:15:262
1512099259,12cyclictest0-21swapper/221:00:132
1510299253,6cyclictest3767-21kthreadcore21:30:260
1510299253,6cyclictest12-21ksoftirqd/023:55:160
15102992520,3cyclictest8987-21apt-get21:35:140
1510299251,18cyclictest9782-21sh00:15:140
1512999242,15cyclictest25366irq/58-eth022:40:143
1510299249,9cyclictest12-21ksoftirqd/021:19:200
1510299244,5cyclictest12-21ksoftirqd/022:05:270
1510299244,5cyclictest12-21ksoftirqd/020:10:200
1510299244,4cyclictest13465-21tr20:15:210
1510299243,5cyclictest12-21ksoftirqd/022:55:190
1510299243,5cyclictest12-21ksoftirqd/000:25:020
1510299241,4cyclictest12-21ksoftirqd/023:30:490
15102992413,8cyclictest23475-21apt-get21:00:120
1510299241,19cyclictest9969-21sh23:35:140
1512999232,15cyclictest25366irq/58-eth022:10:153
1510299235,14cyclictest0-21swapper/023:20:130
1510299234,7cyclictest868-21cut22:50:190
1510299233,6cyclictest12-21ksoftirqd/022:17:100
1510299233,5cyclictest12-21ksoftirqd/022:25:200
1510299233,4cyclictest12-21ksoftirqd/022:35:150
15102992317,3cyclictest24971-21apt-get23:05:140
15102992312,2cyclictest131rcu_preempt23:03:190
1512999229,10cyclictest0-21swapper/323:50:013
1512999229,10cyclictest0-21swapper/323:50:013
1512999223,15cyclictest25366irq/58-eth000:35:143
1512999223,13cyclictest0-21swapper/323:33:273
1512099228,10cyclictest9042-21/usr/sbin/munin20:45:132
1512099228,10cyclictest9042-21/usr/sbin/munin20:45:132
1510299226,5cyclictest12-21ksoftirqd/022:00:010
1510299224,5cyclictest28121-21/usr/sbin/munin22:45:300
1510299224,5cyclictest12-21ksoftirqd/023:15:290
1510299224,4cyclictest18794-21cut20:55:160
1510299224,4cyclictest12-21ksoftirqd/021:40:290
1510299224,4cyclictest12-21ksoftirqd/020:50:010
1510299224,4cyclictest12-21ksoftirqd/020:50:010
1510299223,6cyclictest4052-21timerwakeupswit22:10:300
1510299223,5cyclictest12-21ksoftirqd/000:30:490
1510299223,4cyclictest12-21ksoftirqd/023:50:590
1510299223,4cyclictest12-21ksoftirqd/023:41:180
1510299222,6cyclictest0-21swapper/000:12:420
1510299222,4cyclictest12-21ksoftirqd/019:35:170
1510299221,5cyclictest15503-21usage_Anybus19:40:320
1510299217,11cyclictest21167-21apt-get19:50:140
1510299216,12cyclictest15897-21apt-get22:40:130
1510299213,6cyclictest4477-21fschecks_time20:40:180
1510299213,6cyclictest28606-21timerwakeupswit19:20:280
1510299213,6cyclictest24389-21ssh00:01:380
1510299213,5cyclictest12-21ksoftirqd/021:20:040
1510299213,4cyclictest12-21ksoftirqd/021:25:210
1510299211,16cyclictest32113-21cut20:35:170
1510299211,13cyclictest28268-21find21:05:130
1510299210,6cyclictest12-21ksoftirqd/020:10:010
1512999209,8cyclictest0-21swapper/322:31:053
1512999205,10cyclictest0-21swapper/323:28:423
15129992014,4cyclictest25366irq/58-eth023:35:123
15129992011,6cyclictest0-21swapper/322:35:193
15129992010,8cyclictest0-21swapper/321:29:273
1510299204,4cyclictest12-21ksoftirqd/019:55:250
1510299203,6cyclictest1682-21cat20:00:240
1510299203,5cyclictest12-21ksoftirqd/020:50:150
1510299203,4cyclictest12-21ksoftirqd/020:25:300
1510299203,4cyclictest12-21ksoftirqd/019:45:220
1510299201,6cyclictest12-21ksoftirqd/000:05:130
1512999199,8cyclictest0-21swapper/323:52:403
1512999199,8cyclictest0-21swapper/323:09:403
1512999199,6cyclictest25366irq/58-eth021:15:173
1512999198,8cyclictest0-21swapper/323:56:523
1512999198,8cyclictest0-21swapper/322:53:473
1512999193,11cyclictest0-21swapper/300:16:423
1512999192,10cyclictest0-21swapper/320:15:163
15129991913,4cyclictest0-21swapper/319:30:323
15129991913,4cyclictest0-21swapper/300:10:273
1512999191,13cyclictest0-21swapper/319:20:173
15129991910,6cyclictest0-21swapper/321:46:463
15129991910,6cyclictest0-21swapper/321:36:163
15120991910,7cyclictest15127-21latency_hist19:10:012
1510299197,3cyclictest12-21ksoftirqd/019:13:410
1510299191,6cyclictest20875-21ntp_192.168.11420:20:260
1510299191,6cyclictest12-21ksoftirqd/019:25:140
1512999183,10cyclictest0-21swapper/300:31:423
1512999182,10cyclictest0-21swapper/320:25:163
15129991812,4cyclictest0-21swapper/321:21:573
1512999181,13cyclictest0-21swapper/319:25:213
1512999181,12cyclictest0-21swapper/323:15:173
1512999181,12cyclictest0-21swapper/320:30:233
1512999181,11cyclictest0-21swapper/322:20:013
1512999181,10cyclictest0-21swapper/323:00:173
15129991810,6cyclictest0-21swapper/321:11:333
15129991810,6cyclictest0-21swapper/300:00:313
1512999179,6cyclictest0-21swapper/322:26:223
1512999179,6cyclictest0-21swapper/321:50:083
1512999179,6cyclictest0-21swapper/321:43:373
1512999178,6cyclictest25366irq/58-eth022:45:133
1512999172,10cyclictest0-21swapper/321:05:153
1512999172,10cyclictest0-21swapper/321:00:133
1512999172,10cyclictest0-21swapper/320:55:153
1512999172,10cyclictest0-21swapper/320:45:153
1512999172,10cyclictest0-21swapper/320:45:153
1512999172,10cyclictest0-21swapper/320:20:163
1512999172,10cyclictest0-21swapper/320:10:143
1512999172,10cyclictest0-21swapper/320:05:133
1512999172,10cyclictest0-21swapper/319:35:153
1512999172,10cyclictest0-21swapper/319:10:183
1512999170,13cyclictest25366irq/58-eth019:55:143
1512999162,10cyclictest0-21swapper/319:45:153
1512999151,12cyclictest0-21swapper/319:45:013
1512999151,11cyclictest0-21swapper/320:40:193
1512999143,6cyclictest0-21swapper/319:50:133
1510299141,10cyclictest15195-21latency_hist19:10:020
1512999131,10cyclictest0-21swapper/320:50:303
1512999122,6cyclictest0-21swapper/320:35:043
1512999111,7cyclictest0-21swapper/319:15:193
1512999111,7cyclictest0-21swapper/319:15:193
1512999111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional