You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 11:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Mar 21, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3037199564,40cyclictest16796-21ssh22:40:111
3037199563,37cyclictest22072-21ssh22:25:131
3037199533,6cyclictest131rcu_preempt21:50:161
3036399533,39cyclictest1919-21ssh21:20:130
3037199483,33cyclictest30999-21ssh00:00:131
3037199483,33cyclictest30999-21ssh00:00:131
3036399474,35cyclictest924-21snmpd22:05:140
3036399461,35cyclictest15049-21ssh23:15:130
3038399451,27cyclictest13021-21ssh22:20:132
3038399444,25cyclictest18378-21ssh22:05:132
3037199442,30cyclictest16367-21ssh21:10:121
3037199441,34cyclictest2506-21ssh22:50:141
3039199433,28cyclictest28366irq/58-eth023:10:103
3038399423,19cyclictest27-21ksoftirqd/222:10:142
3038399423,19cyclictest27-21ksoftirqd/222:10:132
30391994124,8cyclictest28366irq/58-eth023:40:133
30383994121,13cyclictest16514-21sh00:10:122
3037199413,28cyclictest32760-21ssh21:55:121
3037199407,24cyclictest21-21ksoftirqd/123:00:131
3039199392,25cyclictest28366irq/58-eth022:40:123
3037199391,32cyclictest557-21ssh23:25:121
3036399391,32cyclictest16515-21rm00:10:130
30391993817,12cyclictest28366irq/58-eth021:45:103
30363993817,15cyclictest12-21ksoftirqd/020:45:120
3038399372,18cyclictest17867-21ssh23:34:432
3037199373,19cyclictest13012-21ssh22:20:121
30391993615,12cyclictest28366irq/58-eth023:30:133
3038399362,18cyclictest26086-21ssh21:31:532
3038399362,17cyclictest31920-21diskmemload22:46:582
3038399352,17cyclictest8907-21ssh23:29:442
3038399334,21cyclictest27482-21ssh23:40:132
30391993214,10cyclictest28366irq/58-eth021:25:123
3038399322,11cyclictest18524-21ssh23:35:142
3038399322,11cyclictest18524-21ssh23:35:142
3037199322,22cyclictest29516-21ssh23:05:121
30371993221,5cyclictest7542-21ssh00:05:111
3036399322,20cyclictest31158-21ssh22:30:100
30363993216,8cyclictest30353-21cyclictest19:20:110
3039199312,18cyclictest28366irq/58-eth023:50:133
3039199311,13cyclictest28366irq/58-eth020:45:123
3038399315,16cyclictest22346irq/40-dwc2_hso23:15:132
3037199312,26cyclictest22015-21ssh23:55:121
3037199301,24cyclictest0-21swapper/121:35:141
3036399307,18cyclictest12-21ksoftirqd/020:50:120
3039199297,10cyclictest28366irq/58-eth020:30:133
3039199292,21cyclictest28366irq/58-eth020:55:123
3039199291,14cyclictest28366irq/58-eth022:25:133
30391992911,11cyclictest28366irq/58-eth022:20:123
3038399292,21cyclictest5025-21ssh21:39:442
3038399292,16cyclictest24114-21ssh23:20:132
3037199292,17cyclictest25505-21ssh00:15:131
30371992911,10cyclictest21-21ksoftirqd/120:25:131
30371992911,10cyclictest21-21ksoftirqd/120:25:121
3038399282,18cyclictest22346irq/40-dwc2_hso20:45:112
3038399282,11cyclictest0-21swapper/221:41:422
3037199283,15cyclictest28126-21kworker/u8:122:15:121
3037199282,18cyclictest25891-21ssh22:45:131
3037199281,21cyclictest14062-21ssh21:44:431
3039199272,16cyclictest0-21swapper/323:29:433
3039199272,16cyclictest0-21swapper/321:31:423
3038399276,11cyclictest15968-21diskmemload00:05:012
3038399276,11cyclictest15968-21diskmemload00:05:002
3038399272,15cyclictest10978-21ssh00:25:122
3038399271,19cyclictest0-21swapper/219:55:132
3038399271,18cyclictest22346irq/40-dwc2_hso19:20:132
3037199273,15cyclictest0-21swapper/120:05:141
3037199272,19cyclictest22374-21ssh00:11:431
3037199272,16cyclictest9342-21ssh22:00:151
3037199271,22cyclictest13000-21ssh23:50:141
3037199271,21cyclictest20112-21ssh21:30:121
3037199271,17cyclictest21-21ksoftirqd/119:50:121
3039199263,17cyclictest0-21swapper/300:13:463
3038399262,14cyclictest31182-21ssh22:30:122
3038399262,10cyclictest0-21swapper/220:15:142
3038399261,21cyclictest22292-21grep22:25:172
3038399261,19cyclictest0-21swapper/220:10:132
3038399261,15cyclictest22346-21cut19:35:162
3037199261,18cyclictest14642-21ssh21:45:111
3039199252,17cyclictest0-21swapper/321:39:443
3038399252,14cyclictest22346irq/40-dwc2_hso20:50:112
3038399251,9cyclictest0-21swapper/221:05:132
3038399251,4cyclictest0-21swapper/221:17:292
3037199253,12cyclictest30353-21cyclictest21:15:121
3037199252,18cyclictest3442-21ssh23:44:441
30371992515,6cyclictest0-21swapper/120:10:131
3036399251,8cyclictest20719-21cat22:40:260
3039199245,10cyclictest0-21swapper/300:00:123
3039199245,10cyclictest0-21swapper/300:00:113
30391992411,10cyclictest0-21swapper/322:30:223
3038399242,18cyclictest0-21swapper/200:05:142
30383992413,9cyclictest12196-21ssh23:12:042
3038399240,10cyclictest7751-21ssh22:35:142
3037199248,11cyclictest21-21ksoftirqd/119:30:131
3037199243,13cyclictest0-21swapper/122:35:111
3037199242,8cyclictest16981-21ssh21:26:521
3037199242,16cyclictest27422-21rm22:10:141
3037199242,16cyclictest27422-21rm22:10:141
30371992415,5cyclictest0-21swapper/119:40:121
3037199241,19cyclictest17460-21ls20:00:271
3036399245,5cyclictest131rcu_preempt00:15:140
3038399239,11cyclictest17383-21apt-get19:30:132
3038399238,11cyclictest24359-21sh20:45:012
3038399236,8cyclictest20721-21wc22:40:262
3038399235,8cyclictest16754-21grep21:10:172
3038399234,13cyclictest22346irq/40-dwc2_hso20:35:132
3038399233,6cyclictest0-21swapper/221:26:512
3038399233,10cyclictest31495-21kthreadcore23:05:202
3038399232,6cyclictest0-21swapper/221:47:432
3038399232,6cyclictest0-21swapper/200:18:552
30383992310,4cyclictest15441-21sh23:50:212
3037199234,9cyclictest21-21ksoftirqd/121:00:121
3037199233,12cyclictest19961-21sh00:30:141
3037199232,14cyclictest6003-21sh23:10:111
3037199231,18cyclictest1753-21find20:55:121
3036399235,8cyclictest0-21swapper/020:35:130
3036399233,13cyclictest0-21swapper/020:20:130
3036399231,18cyclictest22193-21cut23:55:170
3039199225,12cyclictest0-21swapper/321:44:433
3039199223,10cyclictest28366irq/58-eth021:15:123
3039199221,4cyclictest0-21swapper/321:10:133
3038399222,17cyclictest20386-21ssh00:30:192
3038399221,8cyclictest10376-21cut20:30:162
3038399221,16cyclictest28871-21ssh00:35:122
3038399221,13cyclictest3982-21ssh22:15:132
3037199222,4cyclictest131rcu_preempt22:55:141
30371992215,4cyclictest0-21swapper/119:15:151
3036399228,10cyclictest18710-21grep23:35:170
3036399228,10cyclictest18710-21grep23:35:160
3036399224,14cyclictest12-21ksoftirqd/019:10:120
3036399221,17cyclictest9290-21find22:00:130
3036399221,16cyclictest550-21apt-get23:25:120
30363992210,3cyclictest131rcu_preempt00:00:160
30363992210,3cyclictest131rcu_preempt00:00:150
3039199216,9cyclictest28366irq/58-eth023:45:253
3039199215,9cyclictest0-21swapper/323:35:133
3039199215,9cyclictest0-21swapper/323:35:123
3039199213,13cyclictest0-21swapper/323:20:143
3039199212,11cyclictest28366irq/58-eth021:55:123
3038399218,4cyclictest25315-21kthreadcore21:50:212
3038399216,5cyclictest26536-1kworker/u9:123:55:232
3038399212,12cyclictest15968-21diskmemload23:48:582
3038399212,10cyclictest0-21swapper/221:20:132
3038399211,9cyclictest8910-21sendmail22:00:012
30383992111,8cyclictest0-21swapper/222:03:402
3038399211,16cyclictest2722-21ls19:15:022
3038399211,10cyclictest7721-21grep19:50:272
3037199219,10cyclictest0-21swapper/123:22:501
3037199212,15cyclictest15029-21apt-get20:35:131
3037199211,9cyclictest19936-21kthreadcore23:35:201
3037199211,9cyclictest19936-21kthreadcore23:35:191
3037199211,16cyclictest32030-21kthreadcore20:50:191
3037199211,10cyclictest2757-21latency_hist19:15:011
30371992110,8cyclictest18483-21sh22:05:141
30371992110,4cyclictest21-21ksoftirqd/100:20:171
3037199210,16cyclictest7698-21find19:20:111
3036399219,8cyclictest12978-21apt-get23:50:130
3036399212,8cyclictest5775-21df21:40:130
3036399212,14cyclictest20109-21rm21:30:110
3036399211,17cyclictest11253-21ls21:25:160
3039199202,9cyclictest28366irq/58-eth019:40:133
3039199202,13cyclictest0-21swapper/322:05:133
3038399203,8cyclictest15708-21tr20:00:202
3038399201,8cyclictest616566ssh22:50:252
3038399201,6cyclictest1156066ssh22:55:152
3038399201,16cyclictest10288-21timerandwakeup21:00:292
3038399201,15cyclictest30875-21grep19:40:292
3038399201,11cyclictest4251-21tr20:55:202
3037199202,13cyclictest881-21lldpd20:30:131
3037199201,9cyclictest11132-21fschecks_count00:25:141
30371992014,4cyclictest0-21swapper/121:20:251
30371992012,5cyclictest4060-21/usr/sbin/munin23:45:281
3037199201,12cyclictest28435-21cut20:15:151
3036399208,3cyclictest131rcu_preempt21:10:300
3036399207,11cyclictest26091-21kthreadcore23:20:210
3036399202,12cyclictest0-21swapper/023:10:120
3036399201,8cyclictest29048-21ntp_states21:15:250
30363992013,5cyclictest21404-21ssh22:24:420
3036399201,16cyclictest25881-21ssh22:45:130
3036399201,15cyclictest32632-21missed_timers21:35:250
3036399201,10cyclictest10675-21kthreadcore23:30:190
3039199199,5cyclictest0-21swapper/323:15:213
3039199198,8cyclictest0-21swapper/300:25:553
3039199193,11cyclictest0-21swapper/300:08:393
3039199192,13cyclictest0-21swapper/323:05:113
3039199191,12cyclictest0-21swapper/321:05:223
3039199191,12cyclictest0-21swapper/319:35:023
3039199191,11cyclictest0-21swapper/322:50:153
30391991910,6cyclictest0-21swapper/322:17:433
30391991910,6cyclictest0-21swapper/300:30:183
30391991910,5cyclictest0-21swapper/322:35:123
3038399191,9cyclictest5629-21kthreadcore19:15:212
3038399191,9cyclictest3953-21grep20:20:222
3038399191,9cyclictest12819-21wc19:25:172
3038399191,8cyclictest18870-21wc20:05:172
3038399191,15cyclictest6885-21tr20:25:202
3038399191,15cyclictest6885-21tr20:25:192
3038399191,15cyclictest32042-21ls19:45:152
3038399191,11cyclictest2062-21apt-get00:20:112
3038399190,6cyclictest15968-21diskmemload23:00:112
3037199193,2cyclictest131rcu_preempt00:38:101
3037199193,12cyclictest11549-21tr23:30:201
3037199192,11cyclictest23660-21apt-config23:20:011
3037199191,10cyclictest8963-21/usr/sbin/munin19:55:171
3036399197,9cyclictest12-21ksoftirqd/000:25:130
3036399197,3cyclictest131rcu_preempt22:15:190
3036399193,8cyclictest8136-21irqcore22:35:180
3036399192,14cyclictest22093-21ssh22:25:140
3036399192,12cyclictest781-21ntp_states23:05:250
3036399191,8cyclictest32445-21latency_hist21:55:020
3036399191,7cyclictest32268-21users22:10:310
3036399191,7cyclictest32268-21users22:10:300
30363991911,6cyclictest25749-21sshd00:31:430
3039199189,6cyclictest0-21swapper/323:57:243
3039199188,6cyclictest0-21swapper/300:18:433
3039199185,9cyclictest28366irq/58-eth022:45:133
3039199182,11cyclictest0-21swapper/321:54:443
3039199182,10cyclictest0-21swapper/320:35:153
3039199182,10cyclictest0-21swapper/319:55:143
3039199182,10cyclictest0-21swapper/319:25:043
3039199182,10cyclictest0-21swapper/319:10:123
30391991811,4cyclictest0-21swapper/321:20:163
3039199181,13cyclictest0-21swapper/320:05:293
3039199181,13cyclictest0-21swapper/300:35:163
30391991810,6cyclictest0-21swapper/300:20:393
3038399181,8cyclictest30420-21latency_hist19:10:022
3037199189,6cyclictest13770-21kthreadcore19:25:201
3037199185,10cyclictest1-21systemd20:21:361
30371991811,3cyclictest21-21ksoftirqd/122:30:121
3037199181,10cyclictest21-21ksoftirqd/121:05:131
3036399188,7cyclictest329-21cron20:20:020
3036399186,9cyclictest20676-21fschecks_count23:00:150
3036399185,10cyclictest3095-21fschecks_time19:15:150
3036399184,11cyclictest19294-21kernelversion20:05:210
3036399183,12cyclictest2783-21irqstats00:20:170
3036399182,13cyclictest2293-21kthreadcore21:55:210
3036399182,12cyclictest12346-21users00:05:320
3036399181,7cyclictest27594-21diskstats23:40:140
30363991811,4cyclictest0-21swapper/023:45:140
30363991810,5cyclictest32199-21irqcore19:45:160
3039199179,6cyclictest0-21swapper/322:55:453
3039199179,6cyclictest0-21swapper/322:10:283
3039199179,6cyclictest0-21swapper/322:10:273
3039199179,3cyclictest0-21swapper/323:02:183
3039199172,10cyclictest0-21swapper/320:15:143
3039199172,10cyclictest0-21swapper/319:45:133
3039199171,13cyclictest28366irq/58-eth022:00:133
3039199171,13cyclictest0-21swapper/319:20:133
3037199173,10cyclictest24728-21fschecks_count20:45:151
3037199172,10cyclictest0-21swapper/120:40:141
3036399177,7cyclictest6754-21fschecks_time21:00:170
3036399173,6cyclictest29266-21irqcore00:35:170
3036399171,7cyclictest26803-21sendmail-msp19:40:000
3036399171,6cyclictest2096-21irqcore20:55:170
3036399171,14cyclictest27690-21kthreadcore19:40:200
30363991711,3cyclictest2514-21sh22:50:150
3036399171,13cyclictest15980-21kthreadcore21:45:180
3036399171,13cyclictest11611-21grep21:05:170
3036399171,13cyclictest0-21swapper/020:00:120
3036399171,12cyclictest14080-21sh22:55:210
3039199162,9cyclictest0-21swapper/320:50:123
3039199162,9cyclictest0-21swapper/320:00:133
3039199161,9cyclictest0-21swapper/320:20:163
30391991611,3cyclictest33-21ksoftirqd/321:00:293
3037199163,10cyclictest22353-21fschecks_time19:35:161
3037199162,10cyclictest31812-21apt-get19:45:121
3036399168,6cyclictest9997-21sshd20:26:440
3036399168,6cyclictest9997-21sshd20:26:430
3036399163,10cyclictest10010-21kthreadcore19:55:200
3036399162,11cyclictest30444-21latency_hist19:10:010
3036399161,11cyclictest4197-21apt-get19:50:140
3036399161,11cyclictest23735-21grep20:10:160
3036399161,11cyclictest13476-21tr19:25:180
3036399161,11cyclictest11091-21cut20:30:190
3039199152,9cyclictest0-21swapper/320:10:123
3039199152,10cyclictest0-21swapper/319:50:133
3036399152,10cyclictest17407-21cpu19:30:140
3036399151,9cyclictest19962-21cut20:40:170
3039199141,11cyclictest0-21swapper/319:35:163
3039199141,11cyclictest0-21swapper/319:20:013
3039199138,3cyclictest0-21swapper/320:40:243
3037199131,9cyclictest30421-21cat19:10:021
3039199121,9cyclictest0-21swapper/320:25:163
3039199121,9cyclictest0-21swapper/320:25:163
3039199101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional