You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 21:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Sat Dec 27, 2025 12:45:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2359599695,48cyclictest20199-21ssh11:25:140
2361599604,44cyclictest9429-21ssh10:05:152
2359599602,44cyclictest23776-21ssh10:50:130
2361599572,45cyclictest15407-21ssh09:50:132
23595995711,9cyclictest131rcu_preempt11:55:130
2360399563,45cyclictest30227-21ssh09:40:161
2361599522,31cyclictest27367-21ssh09:20:142
2360399502,42cyclictest31372-21apt-get11:50:131
23595994913,10cyclictest131rcu_preempt10:20:160
2360399482,38cyclictest0-21swapper/109:25:131
2360399481,41cyclictest0-21swapper/109:55:141
2360399481,41cyclictest0-21swapper/109:55:141
2360399471,37cyclictest19094-21ssh12:20:151
2360399471,36cyclictest0-21swapper/111:10:141
2360399462,37cyclictest0-21swapper/109:30:131
2360399462,35cyclictest0-21swapper/110:30:151
2360399461,38cyclictest2225-21apt-get07:55:121
2360399461,37cyclictest0-21swapper/111:55:131
2360399461,35cyclictest10564-21apt-get07:30:121
2360399452,40cyclictest0-21swapper/108:45:131
2360399451,39cyclictest0-21swapper/111:45:171
2360399451,39cyclictest0-21swapper/111:45:171
2360399451,38cyclictest0-21swapper/110:20:171
2360399451,37cyclictest0-21swapper/110:45:121
2360399450,27cyclictest6085-21/usr/sbin/munin07:25:181
2360399442,37cyclictest32686-21apt-get10:55:151
2360399441,39cyclictest0-21swapper/108:35:111
2360399441,38cyclictest22787-21apt-get08:50:131
2362499433,25cyclictest28366irq/58-eth010:55:143
2361599430,34cyclictest924-21snmpd09:00:132
2360399432,36cyclictest32404-21apt-get09:00:131
2360399431,37cyclictest951-21apt-get07:20:141
2360399431,36cyclictest4144-21apt-get12:30:111
2359599432,30cyclictest8971-21ssh11:00:140
2360399424,32cyclictest5255-21apt-get11:35:131
2360399421,35cyclictest27820-21apt-get12:25:131
2360399421,34cyclictest9414-21ssh10:05:141
2360399420,39cyclictest4790-21apt-get09:05:121
2359599421,35cyclictest19089-21ssh12:20:140
2360399411,24cyclictest10030-21ssh11:18:411
2362499401,23cyclictest28366irq/58-eth011:15:163
2360399402,34cyclictest6587-21apt-get09:45:131
2360399402,34cyclictest31107-21apt-get08:25:121
2360399401,24cyclictest27049-21diskstats10:15:161
2360399400,36cyclictest13930-21ssh11:40:121
2361599391,23cyclictest18594-21ssh09:15:132
2360399391,35cyclictest12931-21apt-get12:35:141
2360399391,35cyclictest12931-21apt-get12:35:131
2360399391,33cyclictest10282-21apt-get12:15:141
2360399390,27cyclictest26024-21ssh11:10:001
2360399390,27cyclictest26024-21ssh11:10:001
2359599391,30cyclictest6597-21ssh09:45:130
2361599383,27cyclictest261rcuc/209:30:132
2361599382,18cyclictest6453-21ps07:25:192
2360399382,32cyclictest0-21swapper/107:15:141
2360399381,29cyclictest25048-21apt-get07:45:141
2360399381,29cyclictest25048-21apt-get07:45:141
2360399380,30cyclictest21310-21ssh09:35:121
2360399380,25cyclictest21955-21sh12:01:211
2361599372,12cyclictest32236-21find07:15:292
2360399371,30cyclictest0-21swapper/110:10:131
2359599375,24cyclictest32694-21ssh10:55:140
2361599363,26cyclictest0-21swapper/211:24:302
23615993624,4cyclictest28920-21ssh11:30:142
2360399361,31cyclictest9813-21tr11:00:211
2360399361,30cyclictest15395-21apt-get09:50:121
2360399361,29cyclictest0-21swapper/108:00:131
2360399361,22cyclictest3450-21ssh11:33:111
2360399361,1cyclictest0-21swapper/109:20:141
2361599352,17cyclictest10974-21cat09:45:302
2360399351,31cyclictest25234-21apt-get12:05:151
2360399351,30cyclictest3534-21cut12:10:211
2360399351,2cyclictest0-21swapper/110:40:121
2360399351,29cyclictest23766-21apt-get10:50:141
2360399351,25cyclictest28142-21cat07:10:291
2360399351,25cyclictest28142-21cat07:10:291
2360399351,24cyclictest19089-21ssh09:15:191
2359599352,8cyclictest131rcu_preempt07:45:140
2359599352,8cyclictest131rcu_preempt07:45:140
23595993517,14cyclictest12-21ksoftirqd/011:10:130
2361599342,26cyclictest0-21swapper/211:35:142
23615993422,4cyclictest4163-21ssh12:30:122
2361599342,24cyclictest0-21swapper/207:10:162
2361599342,24cyclictest0-21swapper/207:10:162
2360399341,30cyclictest27577-21apt-get08:55:151
2360399341,2cyclictest0-21swapper/108:10:141
2360399341,28cyclictest24908-21latency_hist07:45:021
2360399341,22cyclictest0-21swapper/110:02:371
2360399341,20cyclictest4060-21ssh10:37:531
2360399340,30cyclictest22848-21kthreadcore11:25:251
2360399340,30cyclictest0-21swapper/108:30:131
2359599343,22cyclictest12-21ksoftirqd/012:10:130
2360399332,27cyclictest15169-21ntp_states11:20:291
23595993319,9cyclictest0-21swapper/008:40:120
2360399321,28cyclictest15737-21kthreadcore08:40:211
2360399321,26cyclictest22490-21kthreadcore08:15:191
2360399320,28cyclictest18256-21sed09:15:011
2360399320,28cyclictest13287-21cat08:05:211
23595993217,9cyclictest27895-21cat07:10:290
23595993217,9cyclictest27895-21cat07:10:290
2360399311,27cyclictest20021-21latency_hist07:40:011
2360399311,26cyclictest933-21ps07:50:271
2360399311,26cyclictest29567-21latency08:20:221
2360399310,27cyclictest12520-21cut10:25:181
23595993117,10cyclictest0-21swapper/007:55:120
2361599302,12cyclictest6107-21ssh11:53:302
2360399301,26cyclictest23585-21cyclictest07:10:011
2362499291,17cyclictest28366irq/58-eth008:55:143
2361599295,14cyclictest0-21swapper/207:40:132
2361599293,21cyclictest0-21swapper/212:39:302
2361599293,21cyclictest0-21swapper/212:39:302
2361599293,18cyclictest0-21swapper/209:40:302
2361599292,21cyclictest26404-21ntp_states11:45:302
2361599292,21cyclictest26404-21ntp_states11:45:302
2361599292,20cyclictest0-21swapper/208:15:142
2361599292,18cyclictest24214-21ssh09:55:142
2361599292,18cyclictest24214-21ssh09:55:142
2359599293,20cyclictest16515-21ssh12:00:140
23595992923,5cyclictest121ksoftirqd/009:30:480
2359599292,22cyclictest12-21ksoftirqd/008:50:130
23595992920,7cyclictest12-21ksoftirqd/010:30:150
2362499284,14cyclictest28366irq/58-eth008:25:123
2362499282,14cyclictest28366irq/58-eth012:10:123
2362499281,15cyclictest28366irq/58-eth010:30:133
2361599286,16cyclictest14933-21sh10:45:122
2361599281,19cyclictest27-21ksoftirqd/208:25:122
2361599281,16cyclictest2755-21ssh11:15:152
2359599283,7cyclictest131rcu_preempt11:50:120
2359599282,20cyclictest29376-21sh09:39:300
2359599282,18cyclictest1236-21cat07:20:180
2361599274,6cyclictest3345-21apt-get10:20:162
2361599272,18cyclictest0-21swapper/207:45:052
2361599272,18cyclictest0-21swapper/207:45:052
23615992713,8cyclictest27-21ksoftirqd/211:00:162
2359599275,6cyclictest12-21ksoftirqd/012:35:320
2359599275,6cyclictest12-21ksoftirqd/012:35:320
2359599272,19cyclictest10977-21users10:40:300
2359599272,16cyclictest9404-21ssh10:05:140
23595992718,5cyclictest0-21swapper/011:05:130
23595992718,5cyclictest0-21swapper/011:05:130
2361599266,14cyclictest271ksoftirqd/211:55:122
2361599263,15cyclictest22346irq/40-dwc2_hso08:40:132
2361599263,15cyclictest22346irq/40-dwc2_hso08:30:152
2361599263,14cyclictest22346irq/40-dwc2_hso10:25:132
2361599261,8cyclictest0-21swapper/208:20:132
2361599261,13cyclictest3725-21ssh09:25:132
2359599261,2cyclictest131rcu_preempt09:10:250
23595992610,3cyclictest12-21ksoftirqd/011:45:340
23595992610,3cyclictest12-21ksoftirqd/011:45:340
2362499253,14cyclictest0-21swapper/309:45:303
2361599251,18cyclictest6180-21rm10:40:142
2359599252,17cyclictest12-21ksoftirqd/012:30:110
2359599251,20cyclictest15422-21ssh09:50:140
2362499246,15cyclictest33-21ksoftirqd/311:12:183
2362499243,13cyclictest0-21swapper/309:39:303
2361599241,9cyclictest27838-21ssh12:25:132
2361599241,8cyclictest9347-21diskmemload12:08:302
23615992413,8cyclictest27-21ksoftirqd/211:10:142
2359599246,9cyclictest12-21ksoftirqd/011:35:140
2359599243,16cyclictest0-21swapper/007:25:140
2362499236,15cyclictest33-21ksoftirqd/311:57:373
2362499233,14cyclictest0-21swapper/309:50:303
2361599234,4cyclictest881-21lldpd12:12:262
2361599233,3cyclictest131rcu_preempt07:20:142
2361599232,6cyclictest0-21swapper/210:03:302
2361599232,6cyclictest0-21swapper/208:50:042
2361599232,16cyclictest0-21swapper/210:18:302
2361599232,15cyclictest22346irq/40-dwc2_hso08:35:122
2361599231,19cyclictest10547-21acpi07:30:122
2359599235,4cyclictest131rcu_preempt12:05:220
2362499225,14cyclictest33-21ksoftirqd/312:02:013
2362499221,14cyclictest0-21swapper/310:50:143
2361599221,18cyclictest32210-21apt-get10:55:012
2361599221,17cyclictest22346irq/40-dwc2_hso12:15:142
2361599220,8cyclictest0-21swapper/207:50:142
2359599227,4cyclictest12-21ksoftirqd/010:19:250
2359599221,19cyclictest12198-21ls10:25:150
2359599221,18cyclictest11933-21apt-get08:05:140
2359599221,17cyclictest2769-21sh11:15:170
2362499219,9cyclictest0-21swapper/310:01:243
2362499216,10cyclictest0-21swapper/309:43:303
2362499212,11cyclictest28366irq/58-eth012:30:113
23624992111,5cyclictest28466irq/59-eth010:15:133
2361599219,4cyclictest10871-21cut09:10:202
2361599214,14cyclictest17782-21kthreadcore08:10:212
2361599213,9cyclictest21114-21fschecks_count10:30:172
2361599212,16cyclictest13000-21kthreadcore08:05:212
2361599212,15cyclictest0-21swapper/209:36:302
2361599212,13cyclictest22346irq/40-dwc2_hso08:45:122
23615992115,4cyclictest366-21ssh10:35:232
2361599211,17cyclictest20158-21kthreadcore11:05:212
2361599211,17cyclictest20158-21kthreadcore11:05:212
2361599211,10cyclictest19174-21grep07:35:282
2359599214,14cyclictest11469-21ssh11:20:150
23595992113,4cyclictest12-21ksoftirqd/008:25:130
23595992112,6cyclictest0-21swapper/009:05:300
2359599211,18cyclictest30110-21grep10:35:190
2359599211,16cyclictest16724-21apt-get08:10:140
2359599211,15cyclictest12-21ksoftirqd/009:40:150
2359599211,15cyclictest0-21swapper/007:30:130
2362499209,6cyclictest0-21swapper/310:49:283
2362499204,11cyclictest28366irq/58-eth011:30:143
2362499202,11cyclictest0-21swapper/310:20:183
23624992012,5cyclictest28366irq/58-eth009:55:143
23624992012,5cyclictest28366irq/58-eth009:55:143
23624992011,6cyclictest0-21swapper/309:10:223
2361599203,9cyclictest22873-21sendmail_mailqu12:20:302
2361599201,9cyclictest8288-21kthreadcore08:00:202
2361599201,9cyclictest6915-21kthreadcore09:05:202
2361599201,9cyclictest3010-21kthreadcore07:55:182
2361599201,9cyclictest23396-21ssh11:25:252
2361599201,8cyclictest512-21sort10:55:182
2361599201,12cyclictest16768-21ls12:00:172
2361599201,10cyclictest29943-21kthreadcore08:55:222
2361599201,10cyclictest16581-21ssh11:40:212
2359599209,3cyclictest0-21swapper/011:42:490
2359599208,9cyclictest18161-21apt-get10:10:130
2359599207,4cyclictest12-21ksoftirqd/007:15:180
2359599203,3cyclictest131rcu_preempt09:20:010
2359599203,2cyclictest131rcu_preempt09:25:140
2359599202,2cyclictest131rcu_preempt10:49:100
23595992014,4cyclictest0-21swapper/011:30:130
23595992014,3cyclictest12-21ksoftirqd/009:55:210
23595992014,3cyclictest12-21ksoftirqd/009:55:210
2359599201,15cyclictest27847-21ssh12:25:130
2359599201,14cyclictest8415-21cut08:35:140
2359599201,13cyclictest30963-21ssh09:20:250
2359599201,10cyclictest30337-21irqrtprio07:50:180
2362499198,9cyclictest0-21swapper/312:26:483
2362499193,11cyclictest0-21swapper/312:39:303
2362499193,11cyclictest0-21swapper/312:39:303
2362499193,11cyclictest0-21swapper/311:28:303
2362499192,11cyclictest28366irq/58-eth012:15:153
2362499191,13cyclictest0-21swapper/310:40:173
2362499191,13cyclictest0-21swapper/308:50:283
2362499191,12cyclictest0-21swapper/309:00:163
23624991910,6cyclictest28366irq/58-eth009:25:133
23624991910,6cyclictest0-21swapper/311:06:303
23624991910,6cyclictest0-21swapper/311:06:303
2361599191,9cyclictest18319-21fschecks_count10:10:172
23595991910,3cyclictest4221-21ntpq10:00:250
2362499189,6cyclictest0-21swapper/311:54:453
2362499183,10cyclictest0-21swapper/309:30:483
2362499183,10cyclictest0-21swapper/307:25:183
2362499182,13cyclictest0-21swapper/311:00:203
23624991812,4cyclictest0-21swapper/310:05:153
23624991812,3cyclictest0-21swapper/310:35:233
23624991811,5cyclictest28366irq/58-eth011:20:153
23624991811,4cyclictest28366irq/58-eth011:41:393
23624991811,4cyclictest0-21swapper/312:20:153
2362499181,11cyclictest0-21swapper/308:05:163
2362499181,11cyclictest0-21swapper/307:20:173
23624991810,6cyclictest28366irq/58-eth010:14:303
2359599184,4cyclictest131rcu_preempt08:20:200
2359599183,13cyclictest25359-21sed08:15:290
2359599182,2cyclictest131rcu_preempt09:00:170
2359599181,15cyclictest23209-21kthreadcore07:40:210
23595991811,5cyclictest11021-21sshd08:00:300
2359599180,15cyclictest32154-21latency_hist09:00:020
2362499179,6cyclictest0-21swapper/310:28:203
2362499179,6cyclictest0-21swapper/309:16:513
2362499177,7cyclictest0-21swapper/311:35:243
2362499172,9cyclictest0-21swapper/307:45:153
2362499172,9cyclictest0-21swapper/307:45:153
2362499172,10cyclictest0-21swapper/307:55:133
23624991711,3cyclictest0-21swapper/312:07:163
23624991710,5cyclictest28366irq/58-eth011:45:343
23624991710,5cyclictest28366irq/58-eth011:45:343
23624991710,5cyclictest0-21swapper/309:20:213
2359599175,3cyclictest12-21ksoftirqd/008:45:210
2359599171,2cyclictest131rcu_preempt08:30:210
23595991711,3cyclictest12-21ksoftirqd/012:15:220
2359599171,12cyclictest18838-21grep07:35:260
23624991611,3cyclictest0-21swapper/308:00:193
2361599161,12cyclictest23673-21latency_hist07:10:022
2362499153,7cyclictest0-21swapper/307:10:293
2362499153,7cyclictest0-21swapper/307:10:293
2362499151,8cyclictest0-21swapper/308:30:243
2362499151,10cyclictest28366irq/58-eth007:30:123
2362499149,3cyclictest0-21swapper/308:20:223
2362499149,3cyclictest0-21swapper/307:35:263
2362499147,5cyclictest0-21swapper/308:40:123
2362499146,6cyclictest33-21ksoftirqd/307:20:013
2362499142,10cyclictest0-21swapper/308:45:183
2359599146,5cyclictest0-21swapper/007:10:010
2362499138,3cyclictest0-21swapper/307:50:303
2362499131,9cyclictest0-21swapper/309:05:233
2362499131,7cyclictest0-21swapper/308:15:173
2362499131,7cyclictest0-21swapper/308:10:163
2362499131,6cyclictest0-21swapper/307:40:203
2362499127,3cyclictest0-21swapper/308:35:233
2362499101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional