You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 06:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue Dec 30, 2025 00:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1705499665,50cyclictest11367-21ssh22:10:130
1705499646,40cyclictest8593-21ssh21:50:140
17084996129,19cyclictest28366irq/58-eth023:05:143
1707699585,39cyclictest17429-21ssh21:55:152
1707699531,28cyclictest0-21swapper/222:25:132
1707699524,30cyclictest2831-21ssh21:10:122
17076995026,16cyclictest27-21ksoftirqd/222:05:152
1706599502,6cyclictest19942-21ssh23:10:131
1707699496,18cyclictest29380-21ssh21:25:162
1706599491,8cyclictest0-21swapper/120:55:131
1706599491,8cyclictest0-21swapper/120:55:131
17084994824,17cyclictest28366irq/58-eth000:20:153
1706599482,2cyclictest6343-21apt-get20:40:141
1705499481,33cyclictest23443-21ssh21:40:150
1706599472,3cyclictest17414-21apt-get21:55:141
1706599470,14cyclictest53146usb-storage19:40:121
1706599461,3cyclictest30112-21apt-get00:30:151
1705499465,5cyclictest131rcu_preempt20:00:120
1705499465,5cyclictest131rcu_preempt20:00:120
1706599452,3cyclictest0-21swapper/120:05:131
1706599451,5cyclictest0-21swapper/122:30:131
1706599450,11cyclictest17118-21apt-get19:10:131
1705499455,6cyclictest131rcu_preempt00:35:160
1708499446,25cyclictest28366irq/58-eth023:00:123
17076994429,7cyclictest22346irq/40-dwc2_hso23:05:132
1706599441,4cyclictest0-21swapper/100:10:151
1706599441,4cyclictest0-21swapper/100:10:151
1706599440,4cyclictest0-21swapper/121:00:161
1708499432,25cyclictest28366irq/58-eth021:55:153
1706599432,2cyclictest15909-21apt-get20:50:141
1706599422,6cyclictest0-21swapper/119:35:151
1706599422,5cyclictest0-21swapper/120:45:161
1706599421,3cyclictest0-21swapper/120:25:141
1708499414,22cyclictest28366irq/58-eth000:10:143
1708499414,22cyclictest28366irq/58-eth000:10:143
1707699412,29cyclictest924-21snmpd22:40:132
1706599411,3cyclictest0-21swapper/119:55:141
1706599411,3cyclictest0-21swapper/119:55:141
1706599410,4cyclictest14800-21apt-get20:15:161
1707699401,22cyclictest31468-21ssh23:35:152
1707699401,21cyclictest20127-21ssh22:15:122
1707699401,21cyclictest20127-21ssh22:15:122
1706599402,2cyclictest31828-21apt-get22:40:131
1706599400,5cyclictest28973-21apt-get22:20:161
1706599391,8cyclictest1835266rm22:13:111
1706599391,5cyclictest11099-21ssh23:05:121
1706599390,14cyclictest11641-21apt-get21:15:131
1705499393,27cyclictest16471-21ssh23:45:120
1707699384,25cyclictest27690-21ssh00:10:142
1707699384,25cyclictest27690-21ssh00:10:142
1706599381,2cyclictest21927-21apt-get19:15:141
1708499374,22cyclictest28366irq/58-eth019:25:143
1707699372,19cyclictest32648-21ssh22:02:322
1706599371,2cyclictest2571-21apt-get22:05:151
1707699362,28cyclictest28147-21ssh00:27:582
1706599363,5cyclictest0-21swapper/121:35:151
1706599363,2cyclictest31441-21apt-get23:35:131
1706599361,3cyclictest18399-21apt-get19:45:111
1706599361,2cyclictest2372-21apt-get23:00:121
1705499361,25cyclictest20488-21ssh21:20:110
1708499352,19cyclictest28366irq/58-eth021:35:143
17076993513,7cyclictest27-21ksoftirqd/223:40:302
17076993513,7cyclictest27-21ksoftirqd/223:40:302
1706599356,25cyclictest8597-21if_eth122:45:171
1706599351,2cyclictest26732-21apt-get19:20:161
1706599350,4cyclictest2833-21apt-get21:10:141
1705499353,23cyclictest1485-21ssh23:55:150
1708499342,18cyclictest28366irq/58-eth022:25:143
1706599341,4cyclictest29372-21apt-get21:25:161
1706599341,4cyclictest0-21swapper/120:30:111
1706599341,3cyclictest21407-21apt-get00:25:141
1706599341,3cyclictest0-21swapper/120:35:151
1706599341,3cyclictest0-21swapper/119:30:131
1706599341,2cyclictest31539-21apt-get19:25:151
1706599341,12cyclictest24965-21ssh22:54:181
1706599341,11cyclictest5080-21ssh23:20:131
1706599341,11cyclictest25026-21cat23:50:011
17054993419,12cyclictest12-21ksoftirqd/023:15:150
1708499331,18cyclictest28366irq/58-eth021:10:123
1706599336,23cyclictest8719-21entropy21:50:161
1706599331,7cyclictest26234-21cut21:40:231
1706599331,2cyclictest30320-21apt-get21:05:151
1706599330,3cyclictest6437-21apt-get00:35:181
1705499331,27cyclictest12636-21ssh00:20:140
1706599321,6cyclictest5958-21forks21:30:161
1706599321,4cyclictest32204-21apt-get21:45:111
1706599321,3cyclictest28739-21apt-get23:15:151
1706599321,10cyclictest5851-21/usr/sbin/munin23:55:331
17054993217,11cyclictest12-21ksoftirqd/019:20:150
1705499321,25cyclictest32222-21ssh21:45:120
1708499314,19cyclictest28366irq/58-eth022:45:133
1707699315,17cyclictest22346irq/40-dwc2_hso20:15:132
1706599317,15cyclictest19056-21ssh00:05:141
1706599314,24cyclictest28587-21tr22:20:031
1706599314,24cyclictest28587-21tr22:20:021
1706599311,7cyclictest29579-21ntp_192.168.11422:55:261
1706599311,7cyclictest23863-21tr23:30:221
1706599311,7cyclictest1268-21latency_hist23:55:021
1706599311,7cyclictest10455-21entropy00:00:161
1706599311,6cyclictest27186-21cut22:00:201
1706599310,7cyclictest27835-21ssh21:23:351
1706599310,4cyclictest19611-21apt-get20:20:141
1705499317,18cyclictest19064-21rm00:05:140
1705499312,26cyclictest31836-21ssh22:40:130
17054993120,8cyclictest0-21swapper/021:30:130
17054993113,16cyclictest12-21ksoftirqd/022:45:150
1705499310,6cyclictest131rcu_preempt23:00:120
1708499301,18cyclictest28366irq/58-eth021:50:123
1707699305,13cyclictest22346irq/40-dwc2_hso00:30:152
1707699304,19cyclictest2381-21diskmemload23:00:122
1707699304,15cyclictest22346irq/40-dwc2_hso23:15:142
1707699303,19cyclictest22346irq/40-dwc2_hso19:50:132
1706599302,6cyclictest17044-21cyclictest19:10:011
1706599302,22cyclictest11759-21ssh22:27:371
1706599301,8cyclictest7969-21diskstats23:40:161
1706599301,8cyclictest7969-21diskstats23:40:161
1706599301,8cyclictest13892-21ls23:25:121
1706599301,6cyclictest3911-21/usr/sbin/munin00:15:211
1706599301,6cyclictest10200-21hwlatdetect20:10:181
1706599301,6cyclictest10200-21hwlatdetect20:10:181
1706599300,9cyclictest18362-21ssh00:21:361
1706599300,7cyclictest23139-21/usr/sbin/munin19:50:241
17054993024,2cyclictest0-21swapper/020:15:130
17054993021,7cyclictest12-21ksoftirqd/022:55:140
1708499294,15cyclictest0-21swapper/322:40:133
1708499292,18cyclictest28366irq/58-eth000:00:143
1708499291,16cyclictest28366irq/58-eth000:30:143
1707699296,16cyclictest27-21ksoftirqd/200:23:312
1707699291,22cyclictest11667-21sh21:15:152
1706599291,7cyclictest23294-21ssh22:35:181
1706599291,5cyclictest32716-21/usr/sbin/munin20:00:251
1706599291,5cyclictest32716-21/usr/sbin/munin20:00:251
1705499292,22cyclictest30139-21ssh00:30:150
1705499291,21cyclictest21429-21ssh00:25:140
1707699288,7cyclictest27-21ksoftirqd/221:31:522
1707699288,12cyclictest27-21ksoftirqd/221:52:242
1707699282,18cyclictest29405-21kworker/u8:122:50:052
1706599281,5cyclictest17044-21cyclictest19:10:011
1705499283,20cyclictest13814-21ssh23:25:120
1705499282,22cyclictest22614-21apt-get23:30:150
1705499281,25cyclictest5095-21ssh23:20:140
1708499273,18cyclictest28366irq/58-eth000:05:133
1708499272,16cyclictest0-21swapper/322:31:383
1707699273,14cyclictest22346irq/40-dwc2_hso19:25:162
1707699272,16cyclictest22346irq/40-dwc2_hso20:25:142
17076992710,8cyclictest27-21ksoftirqd/221:22:362
17076992710,11cyclictest27-21ksoftirqd/222:45:252
1705499274,15cyclictest0-21swapper/022:35:150
1705499272,22cyclictest14606-21ssh21:35:140
17054992714,9cyclictest12-21ksoftirqd/019:15:140
1708499269,11cyclictest0-21swapper/322:35:383
1708499263,14cyclictest28366irq/58-eth023:40:133
1708499263,14cyclictest28366irq/58-eth023:40:133
1707699267,7cyclictest27-21ksoftirqd/221:45:172
1707699266,15cyclictest22346irq/40-dwc2_hso20:20:132
1707699263,9cyclictest0-21swapper/223:31:382
1707699262,9cyclictest0-21swapper/223:56:382
1705499267,11cyclictest12-21ksoftirqd/019:25:140
1705499265,11cyclictest5309-21ssh22:25:140
1705499262,19cyclictest19907-21ssh22:31:380
1708499251,14cyclictest28366irq/58-eth000:25:143
1707699257,4cyclictest27-21ksoftirqd/200:38:522
1707699257,11cyclictest27-21ksoftirqd/223:54:242
1707699256,7cyclictest27-21ksoftirqd/223:46:182
1707699253,13cyclictest22346irq/40-dwc2_hso21:00:162
1707699251,9cyclictest22303-21ssh22:34:382
17076992513,9cyclictest0-21swapper/200:05:162
1707699251,18cyclictest31115-21ssh22:39:382
17076992511,7cyclictest27-21ksoftirqd/222:58:392
1705499252,19cyclictest25527-21sh21:59:380
1705499251,21cyclictest25243-21ssh23:50:130
1708499243,10cyclictest0-21swapper/322:06:383
1707699249,9cyclictest10201-21ssh00:00:052
1707699249,3cyclictest27-21ksoftirqd/222:24:402
1707699244,11cyclictest27-21ksoftirqd/223:14:222
17076992412,5cyclictest22147-21ssh21:38:552
1707699241,16cyclictest23545-21ssh21:40:162
1705499242,16cyclictest11658-21ssh21:15:140
1705499241,17cyclictest2597-21ssh22:05:160
1708499231,16cyclictest0-21swapper/321:25:153
1707699237,8cyclictest27-21ksoftirqd/223:22:272
1707699237,8cyclictest27-21ksoftirqd/219:55:272
1707699237,8cyclictest27-21ksoftirqd/219:55:272
1707699237,10cyclictest131rcu_preempt00:17:122
1707699234,4cyclictest27-21ksoftirqd/222:10:052
17076992313,8cyclictest0-21swapper/220:00:242
17076992313,8cyclictest0-21swapper/220:00:242
1705499233,12cyclictest0-21swapper/022:50:130
1708499223,13cyclictest0-21swapper/322:02:323
1708499223,13cyclictest0-21swapper/319:45:123
1708499223,10cyclictest28366irq/58-eth023:25:103
1707699228,8cyclictest27-21ksoftirqd/219:15:002
1707699228,6cyclictest22346irq/40-dwc2_hso19:45:122
1707699228,11cyclictest27-21ksoftirqd/219:20:022
1707699226,4cyclictest27-21ksoftirqd/223:27:532
1707699221,11cyclictest7442-21kthreadcore20:05:212
1705499229,4cyclictest12-21ksoftirqd/021:10:190
1705499229,10cyclictest12-21ksoftirqd/020:10:150
1705499229,10cyclictest12-21ksoftirqd/020:10:140
1708499216,12cyclictest28366irq/58-eth023:55:153
1707699214,14cyclictest13255-21latency20:10:242
1707699214,14cyclictest13255-21latency20:10:242
1707699212,15cyclictest1767-21wc20:35:182
1707699212,12cyclictest22346irq/40-dwc2_hso20:30:112
17076992111,8cyclictest0-21swapper/220:55:232
17076992111,8cyclictest0-21swapper/220:55:232
1707699211,10cyclictest13749-21fschecks_time19:40:152
17076992110,4cyclictest0-21swapper/219:30:252
1705499218,9cyclictest12-21ksoftirqd/022:20:160
1705499213,11cyclictest0-21swapper/020:40:140
1705499212,14cyclictest3860-21users23:35:340
17054992114,4cyclictest0-21swapper/000:15:150
17054992113,5cyclictest27663-21/usr/sbin/munin00:10:330
17054992113,5cyclictest27663-21/usr/sbin/munin00:10:330
17054992110,8cyclictest11268-21grep23:05:170
1708499203,12cyclictest0-21swapper/323:31:383
1708499202,11cyclictest0-21swapper/323:45:123
1708499202,11cyclictest0-21swapper/322:10:043
17084992013,5cyclictest0-21swapper/322:18:213
17084992013,5cyclictest0-21swapper/322:18:213
17084992012,6cyclictest0-21swapper/322:53:543
1707699202,9cyclictest13458-21kthreadcore20:45:232
1707699202,15cyclictest0-21swapper/219:35:162
1707699201,9cyclictest30430-21fschecks_count21:05:182
1707699201,5cyclictest0-21swapper/219:20:192
1707699201,15cyclictest11020-21latency_hist20:45:022
1705499209,8cyclictest26348-21df_inode22:00:150
1705499209,8cyclictest11225-21df_inode20:45:170
1705499202,15cyclictest26578-21kthreadcore21:00:230
1705499201,16cyclictest10313-21apt-get00:00:130
1705499201,10cyclictest21727-21cut23:10:210
1708499199,8cyclictest0-21swapper/322:23:583
1708499199,6cyclictest0-21swapper/323:13:293
1708499199,6cyclictest0-21swapper/321:22:163
1708499193,11cyclictest0-21swapper/300:35:163
1708499193,10cyclictest0-21swapper/320:00:143
1708499193,10cyclictest0-21swapper/320:00:143
1708499191,13cyclictest0-21swapper/319:15:173
17084991910,6cyclictest0-21swapper/323:35:233
1707699191,9cyclictest16135-21hwlatdetect20:50:172
1705499194,11cyclictest0-21swapper/020:20:130
1705499193,11cyclictest0-21swapper/021:25:160
1705499192,5cyclictest53146usb-storage20:05:100
17054991910,6cyclictest20566-21sed22:15:160
17054991910,6cyclictest20566-21sed22:15:160
1708499187,6cyclictest0-21swapper/300:15:063
1708499185,9cyclictest28366irq/58-eth023:50:143
1708499182,10cyclictest0-21swapper/321:00:173
1708499182,10cyclictest0-21swapper/319:10:153
1708499181,13cyclictest0-21swapper/319:55:013
1708499181,12cyclictest0-21swapper/322:55:203
1708499181,12cyclictest0-21swapper/320:45:243
1705499189,6cyclictest9016-21kthreadcore23:40:200
1705499189,6cyclictest9016-21kthreadcore23:40:200
1705499186,7cyclictest25261-21cron21:00:020
1705499186,7cyclictest25261-21cron21:00:010
1705499185,8cyclictest0-21swapper/019:10:140
1705499181,8cyclictest7753-21unixbench_singl19:30:290
17054991810,5cyclictest6102-21cron20:40:010
1708499179,6cyclictest0-21swapper/321:46:083
1708499173,6cyclictest0-21swapper/320:15:143
1708499172,10cyclictest0-21swapper/321:30:143
1708499172,10cyclictest0-21swapper/319:20:163
17084991711,4cyclictest0-21swapper/323:20:153
17084991711,4cyclictest0-21swapper/321:40:353
1708499171,12cyclictest28466irq/59-eth023:15:143
1708499171,10cyclictest0-21swapper/320:40:293
17084991710,5cyclictest28366irq/58-eth021:15:143
1705499179,5cyclictest28087-21fschecks_count19:55:170
1705499179,5cyclictest28087-21fschecks_count19:55:160
1705499178,7cyclictest8911-21fw_packets19:35:180
1705499176,8cyclictest13818-21wc19:40:160
1705499172,12cyclictest24933-21kthreadcore19:50:200
1705499172,12cyclictest19360-21ntp_kernel_pll_20:50:250
1705499172,10cyclictest0-21swapper/019:45:130
1705499171,7cyclictest31987-21kthreadcore21:05:220
1708499162,9cyclictest0-21swapper/320:55:143
1708499162,9cyclictest0-21swapper/320:55:143
1708499162,10cyclictest0-21swapper/319:30:153
1708499161,12cyclictest0-21swapper/320:10:163
1708499161,12cyclictest0-21swapper/320:10:163
1707699161,12cyclictest17095-21latency_hist19:10:012
1705499166,8cyclictest12-21ksoftirqd/020:25:150
1705499163,10cyclictest31036-21kthreadcore20:30:190
1708499152,7cyclictest0-21swapper/320:25:133
1705499151,11cyclictest17093-21latency_hist19:10:010
1708499149,3cyclictest0-21swapper/319:39:533
1708499143,7cyclictest33-21ksoftirqd/319:40:153
1708499141,7cyclictest0-21swapper/320:50:143
1708499141,7cyclictest0-21swapper/320:30:153
1708499138,3cyclictest0-21swapper/320:35:073
1708499138,3cyclictest0-21swapper/320:08:073
1708499131,10cyclictest0-21swapper/320:20:213
1705499133,8cyclictest17064-21latency_hist19:10:010
1708499122,7cyclictest0-21swapper/321:10:013
1708499121,9cyclictest0-21swapper/319:55:253
1708499121,9cyclictest0-21swapper/319:55:253
1707699101,2cyclictest0-21swapper/219:10:012
170849971,4cyclictest0-21swapper/319:10:013
none12:11:533
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional