You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-08 - 16:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Apr 08, 2026 12:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
300599632,37cyclictest7808-21ssh12:20:112
297499564,43cyclictest18773-21ssh11:50:111
297499511,38cyclictest7302-21ssh09:20:141
300599472,29cyclictest31200-21ssh12:15:122
3024994623,14cyclictest28366irq/58-eth011:45:123
2952994523,4cyclictest12-21ksoftirqd/012:00:100
302499443,27cyclictest28366irq/58-eth012:30:123
3024994425,10cyclictest28366irq/58-eth009:55:133
300599447,28cyclictest27-21ksoftirqd/211:20:132
302499411,22cyclictest28366irq/58-eth011:55:123
300599413,26cyclictest5732-21ssh09:55:112
2952994112,10cyclictest131rcu_preempt07:35:110
302499402,25cyclictest28366irq/58-eth009:10:123
297499409,22cyclictest21-21ksoftirqd/109:05:121
297499402,2cyclictest131rcu_preempt10:55:121
2952994017,16cyclictest28461-1kworker/u9:012:25:130
297499383,7cyclictest131rcu_preempt12:00:091
297499383,28cyclictest14783-21ssh10:00:121
302499372,24cyclictest28366irq/58-eth010:45:133
300599377,24cyclictest23064-21kworker/u8:208:20:122
300599372,19cyclictest24134-21ssh11:35:132
295299371,27cyclictest32721-21ssh10:10:130
300599363,15cyclictest20038-21ssh09:45:122
297499363,21cyclictest21-21ksoftirqd/110:25:141
297499353,18cyclictest21-21ksoftirqd/108:25:151
295299355,7cyclictest131rcu_preempt08:30:110
295299352,27cyclictest7803-21ssh12:20:120
2974993418,9cyclictest0-21swapper/111:00:111
2974993418,9cyclictest0-21swapper/111:00:111
302499331,19cyclictest28366irq/58-eth010:25:133
300599332,24cyclictest924-21snmpd10:30:292
297499330,25cyclictest16295-21ssh09:25:111
300599328,16cyclictest22548-21kworker/u8:111:30:132
300599322,24cyclictest11126-21ssh09:40:122
297499323,4cyclictest131rcu_preempt11:40:111
302499316,15cyclictest28366irq/58-eth010:35:133
300599316,15cyclictest22346irq/40-dwc2_hso10:40:112
300599315,16cyclictest22346irq/40-dwc2_hso11:10:122
300599314,17cyclictest22346irq/40-dwc2_hso09:30:132
300599314,17cyclictest22346irq/40-dwc2_hso08:25:132
300599313,20cyclictest0-21swapper/211:43:282
295299311,24cyclictest12-21ksoftirqd/009:45:120
3024993013,10cyclictest28366irq/58-eth009:50:133
300599303,19cyclictest22346irq/40-dwc2_hso07:40:122
300599302,21cyclictest0-21swapper/212:05:132
300599302,21cyclictest0-21swapper/212:03:402
300599301,5cyclictest271ksoftirqd/212:30:282
295299301,25cyclictest9297-21ssh10:15:110
2952993011,8cyclictest12-21ksoftirqd/011:15:110
295299301,16cyclictest2540-21ssh12:35:120
300599293,17cyclictest22346irq/40-dwc2_hso09:05:142
300599293,16cyclictest22346irq/40-dwc2_hso08:10:122
300599293,14cyclictest12929-21ssh10:35:122
300599292,16cyclictest22346irq/40-dwc2_hso12:10:132
300599292,13cyclictest14764-21ssh10:00:122
295299293,20cyclictest31031-21ssh10:45:120
295299292,21cyclictest25377-21ssh09:30:130
302499281,17cyclictest28366irq/58-eth007:25:133
300599285,14cyclictest22346irq/40-dwc2_hso10:25:122
300599282,20cyclictest21077-21diskmemload11:49:282
300599282,20cyclictest0-21swapper/207:45:132
300599282,11cyclictest23169-21ssh10:57:282
3005992811,11cyclictest0-21swapper/211:19:102
297499282,18cyclictest1918-21ssh11:20:291
2974992813,6cyclictest2915-21cyclictest10:20:131
295299284,17cyclictest21077-21diskmemload09:55:130
3024992715,7cyclictest321rcuc/308:30:123
3024992715,7cyclictest0-21swapper/310:15:133
3024992711,10cyclictest28366irq/58-eth011:23:313
300599276,12cyclictest27-21ksoftirqd/211:25:122
300599274,8cyclictest27-21ksoftirqd/209:25:552
300599273,15cyclictest27-21ksoftirqd/208:35:312
3005992711,9cyclictest25789-21apt-get11:00:132
3005992711,9cyclictest25789-21apt-get11:00:122
295299272,19cyclictest31573-21ssh11:38:330
295299272,18cyclictest15200-21ssh11:30:140
302499263,14cyclictest28366irq/58-eth008:10:113
300599263,17cyclictest23710-21ssh10:05:132
300599263,15cyclictest22346irq/40-dwc2_hso08:30:132
300599262,18cyclictest13282-21ssh10:51:292
300599262,15cyclictest22346irq/40-dwc2_hso11:05:122
3005992614,8cyclictest9262-21/usr/sbin/munin10:15:272
3005992611,10cyclictest29132-21apt-get09:50:132
297499264,17cyclictest633-21Async12:11:531
302499251,16cyclictest28366irq/58-eth007:10:123
300599253,4cyclictest27-21ksoftirqd/209:35:202
300599252,18cyclictest26064-21ssh11:53:292
300599252,18cyclictest21557-21ssh09:10:132
3005992512,9cyclictest5323-21cron08:20:012
300599251,12cyclictest2398-21processes10:45:252
297499256,14cyclictest20466-21ssh11:15:111
295299254,16cyclictest25928-21ssh12:30:110
295299253,7cyclictest131rcu_preempt11:53:370
295299251,6cyclictest15035-21kworker/u8:010:20:140
295299251,21cyclictest16654-21ssh10:55:110
2952992511,6cyclictest0-21swapper/011:10:130
300599249,8cyclictest4316-21ssh09:17:062
300599246,7cyclictest27-21ksoftirqd/209:21:292
300599242,9cyclictest27-21ksoftirqd/211:55:212
300599242,14cyclictest0-21swapper/207:30:122
3005992412,10cyclictest10024-21ssh12:38:422
3005992410,11cyclictest26721-21kthreadcore08:05:202
297499241,19cyclictest30578-21ssh09:15:121
295299242,17cyclictest27751-21ssh11:55:130
3024992312,5cyclictest0-21swapper/311:30:273
302499231,11cyclictest28366irq/58-eth011:25:133
3024992310,8cyclictest28366irq/58-eth012:35:133
300599237,6cyclictest18622-21latency_hist08:00:022
300599235,3cyclictest131rcu_preempt09:00:272
300599232,6cyclictest0-21swapper/208:55:132
297499232,16cyclictest5832-21sshd09:35:271
2974992313,7cyclictest17936-21sshd11:49:291
2974992311,3cyclictest131rcu_preempt12:15:141
2952992314,5cyclictest29542-21interrupts09:50:200
295299231,17cyclictest25820-21ssh11:00:140
295299231,17cyclictest25820-21ssh11:00:130
295299231,10cyclictest131rcu_preempt10:50:120
302499223,10cyclictest28366irq/58-eth007:15:133
3024992213,5cyclictest0-21swapper/312:10:133
300599227,6cyclictest17678-21kthreadcore12:25:202
300599227,12cyclictest6794-21latency_hist08:55:012
300599225,13cyclictest382-21sh10:10:162
300599223,10cyclictest8533-21irqrtprio07:15:172
300599222,16cyclictest15492-21kthreadcore07:20:212
300599221,7cyclictest0-21swapper/208:40:132
300599221,17cyclictest19120-21grep08:00:172
300599221,11cyclictest30094-21kthreadcore07:35:202
3005992210,10cyclictest2813-21cron07:10:032
300599220,3cyclictest18278-21apt-get10:20:142
297499221,18cyclictest25435-21ssh10:40:201
295299221,18cyclictest19437-21kthreadcore07:25:210
3024992116,3cyclictest28466irq/59-eth010:11:353
3024992114,5cyclictest0-21swapper/311:35:123
3024992114,5cyclictest0-21swapper/311:15:153
3024992114,5cyclictest0-21swapper/310:05:193
3024992113,5cyclictest0-21swapper/311:05:183
3024992110,6cyclictest0-21swapper/311:40:123
300599211,16cyclictest30005-21tr08:45:172
300599211,16cyclictest30005-21tr08:45:162
297499218,4cyclictest131rcu_preempt12:35:251
297499214,14cyclictest26304-21kthreadcore08:40:191
297499213,14cyclictest0-21swapper/110:30:121
297499212,8cyclictest25216-21ntp_states09:10:261
297499212,14cyclictest29164-21ssh09:50:141
2974992114,5cyclictest0-21swapper/112:30:301
2974992112,6cyclictest15278-21apt-get08:30:121
297499211,17cyclictest29329-21ls09:30:291
297499211,11cyclictest22295-21kthreadcore08:35:211
295299218,10cyclictest5511-21gpgv08:20:020
295299216,4cyclictest131rcu_preempt10:30:190
295299212,15cyclictest839-21munin-node10:35:130
295299212,14cyclictest839-21munin-node10:40:130
295299211,7cyclictest28028-21/usr/sbin/munin10:25:190
2952992113,6cyclictest6890-21apt-get08:55:110
2952992113,6cyclictest1741-21usage_EthcTask11:20:300
295299211,16cyclictest17314-21ls07:55:210
302499208,9cyclictest0-21swapper/309:20:213
302499207,8cyclictest28366irq/58-eth012:00:113
3024992014,4cyclictest0-21swapper/309:45:083
3024992013,4cyclictest0-21swapper/309:30:433
3024992012,5cyclictest28366irq/58-eth012:05:133
3024992012,5cyclictest0-21swapper/312:23:113
300599202,15cyclictest0-21swapper/207:25:132
300599201,10cyclictest9942-21kthreadcore07:50:172
297499201,5cyclictest25570-21ls12:30:011
2974992012,4cyclictest0-21swapper/112:05:121
295299201,16cyclictest5909-21missed_timers11:05:220
295299201,14cyclictest1961-21ssh09:35:050
295299201,14cyclictest17353-21latency_hist07:25:010
295299200,2cyclictest131rcu_preempt07:30:110
295299200,17cyclictest23664-21find10:05:110
302499193,11cyclictest0-21swapper/310:53:273
302499193,10cyclictest0-21swapper/311:00:053
302499193,10cyclictest0-21swapper/311:00:043
302499192,12cyclictest0-21swapper/311:52:283
3024991913,4cyclictest0-21swapper/311:11:383
3024991913,4cyclictest0-21swapper/310:55:143
3024991913,4cyclictest0-21swapper/310:41:263
3024991912,5cyclictest0-21swapper/312:15:123
3024991912,5cyclictest0-21swapper/310:30:143
3024991912,5cyclictest0-21swapper/309:25:283
302499191,13cyclictest0-21swapper/310:20:233
302499191,13cyclictest0-21swapper/308:40:263
3024991910,6cyclictest0-21swapper/309:42:003
297499197,8cyclictest16540-21sed12:25:021
297499196,10cyclictest6040-21cat09:55:171
297499191,9cyclictest31023-21cut11:55:211
2974991912,5cyclictest31003-21apt10:45:121
2974991911,5cyclictest15150-21/usr/sbin/munin11:30:291
2974991910,7cyclictest25678-21memory07:30:211
2974991910,6cyclictest0-21swapper/111:35:241
297499190,12cyclictest32213-21ls07:40:171
295299196,10cyclictest8112-21grep07:15:160
295299195,3cyclictest131rcu_preempt09:20:210
295299193,12cyclictest1898-21kthreadcore11:40:190
295299192,4cyclictest53146usb-storage07:10:050
2952991912,4cyclictest21077-21diskmemload09:10:200
295299191,12cyclictest13542-21sshd11:28:310
2952991910,6cyclictest21845-21sh12:10:010
2952991910,5cyclictest2915-21cyclictest10:00:140
302499182,10cyclictest0-21swapper/309:05:133
302499182,10cyclictest0-21swapper/307:45:123
302499182,10cyclictest0-21swapper/307:35:113
3024991812,4cyclictest0-21swapper/310:00:113
3024991812,4cyclictest0-21swapper/308:55:143
3024991811,4cyclictest0-21swapper/309:16:303
302499181,12cyclictest0-21swapper/308:00:023
297499184,5cyclictest633-21Async12:11:531
297499183,4cyclictest131rcu_preempt10:10:211
297499183,11cyclictest4370-21apt-get07:45:111
297499181,9cyclictest4386-21kthreadcore11:05:191
297499181,8cyclictest30009-21irqcore08:45:171
297499181,8cyclictest30009-21irqcore08:45:161
2974991811,5cyclictest0-21swapper/108:00:151
2974991811,4cyclictest11853-21kthreadcore09:40:191
297499181,13cyclictest25000-21ssh09:45:341
297499181,11cyclictest15363-21switchtime09:00:281
297499181,10cyclictest23817-21diskstats10:05:151
297499180,7cyclictest24051-21sed08:05:171
295299189,6cyclictest0-21swapper/009:25:120
295299183,8cyclictest0-21swapper/008:40:130
295299181,7cyclictest22334-21df_inode12:10:140
295299181,3cyclictest0-21swapper/009:18:060
2952991811,4cyclictest21104-21apt-key09:10:020
295299181,12cyclictest0-21swapper/007:45:130
2952991810,6cyclictest18385-21cron11:50:010
302499179,5cyclictest28366irq/58-eth008:15:293
302499172,9cyclictest0-21swapper/308:05:123
302499172,10cyclictest0-21swapper/309:00:133
302499172,10cyclictest0-21swapper/308:25:153
3024991711,4cyclictest0-21swapper/312:25:163
3024991711,4cyclictest0-21swapper/309:35:303
3024991710,4cyclictest0-21swapper/308:50:193
300599172,3cyclictest131rcu_preempt07:10:022
297499179,6cyclictest10186-21sshd08:21:281
297499179,5cyclictest5936-21grep08:50:281
297499179,5cyclictest12895-21/usr/sbin/munin10:35:171
297499176,3cyclictest131rcu_preempt10:50:131
297499175,3cyclictest131rcu_preempt11:25:041
297499173,11cyclictest9282-21grep07:50:151
297499173,11cyclictest11908-21irqcore11:10:171
297499171,8cyclictest12767-21/usr/sbin/munin07:20:161
2974991711,4cyclictest7761-21idleruntime-cro07:15:011
297499171,13cyclictest18064-21irqcore07:25:171
297499170,9cyclictest27660-21irqcore07:35:151
295299179,6cyclictest12626-21ntpq07:50:240
295299174,8cyclictest31168-21acpi12:15:120
295299173,7cyclictest0-21swapper/008:00:130
295299173,10cyclictest2849-21apt-key07:10:020
295299172,11cyclictest11812-21diskstats09:00:150
295299172,10cyclictest0-21swapper/008:35:130
295299171,6cyclictest11274-21fschecks_count09:40:140
295299171,4cyclictest0-21swapper/008:20:120
295299171,14cyclictest12992-21kthreadcore08:25:210
302499162,9cyclictest0-21swapper/308:45:133
302499162,9cyclictest0-21swapper/308:45:133
302499162,9cyclictest0-21swapper/307:20:123
302499162,10cyclictest0-21swapper/307:50:133
302499162,10cyclictest0-21swapper/307:40:133
297499162,11cyclictest9492-21kthreadcore07:15:191
297499161,12cyclictest724-21apt-get08:15:121
297499161,12cyclictest16061-21kthreadcore07:55:191
297499161,11cyclictest31863-21proc_pri08:10:251
297499161,11cyclictest0-21swapper/108:55:121
295299169,4cyclictest32102-21cut07:40:150
295299169,4cyclictest30047-21cut08:45:170
295299169,4cyclictest30047-21cut08:45:160
295299163,10cyclictest23676-21fschecks_count08:05:140
295299162,11cyclictest2304-21grep08:50:150
295299161,10cyclictest28852-21grep08:10:180
302499152,7cyclictest321rcuc/308:35:133
302499152,10cyclictest0-21swapper/307:30:123
3024991510,3cyclictest0-21swapper/308:00:283
297499143,9cyclictest2915-21cyclictest07:10:031
302499126,4cyclictest0-21swapper/307:10:033
302499121,5cyclictest0-21swapper/308:20:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional