You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-25 - 19:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Dec 25, 2025 12:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
451129650,42sleep20-21swapper/207:08:292
444729263,9sleep00-21swapper/007:07:380
459428465,15sleep30-21swapper/307:09:343
428327759,14sleep10-21swapper/107:05:311
79062740,1sleep3381rcuc/311:44:593
4713992926,2cyclictest13-21ksoftirqd/012:30:000
471799252,13cyclictest0-21swapper/111:55:121
471399210,1cyclictest0-21swapper/012:00:140
471999190,0cyclictest0-21swapper/207:10:122
471399196,6cyclictest32722-21apt-get09:15:010
4713991917,1cyclictest30356-21munin-run09:10:000
472399180,17cyclictest0-21swapper/308:40:143
4719991817,1cyclictest0-21swapper/212:34:472
4713991816,1cyclictest9942-21apt-get07:20:160
471799170,1cyclictest0-21swapper/110:25:151
471399175,6cyclictest11490-21perf10:45:000
471999162,1cyclictest141rcu_preempt10:10:172
471799160,15cyclictest0-21swapper/107:50:121
471799160,0cyclictest0-21swapper/110:30:151
471399161,1cyclictest141rcu_preempt08:10:260
4713991611,4cyclictest13-21ksoftirqd/008:55:270
4723991511,3cyclictest31520-21sed12:30:283
472399151,12cyclictest0-21swapper/311:05:143
471799155,5cyclictest17122-21apt-get07:35:141
4717991513,1cyclictest0-21swapper/110:00:141
4717991511,3cyclictest25-21ksoftirqd/110:20:171
4717991510,3cyclictest18237-21grep10:55:281
471399152,3cyclictest13-21ksoftirqd/010:35:110
4713991515,0cyclictest0-21swapper/012:10:120
4713991513,2cyclictest13-21ksoftirqd/012:35:130
4723991411,2cyclictest7870-21cron09:30:003
4723991410,3cyclictest27667-21grep11:15:263
4723991410,3cyclictest14396-21tr10:50:133
4723991410,3cyclictest13220-21tr11:55:133
4723991410,3cyclictest11464-21cron10:45:013
4723991410,3cyclictest10242-21cron11:50:003
4723991410,2cyclictest6595-21cut08:20:153
4723991410,2cyclictest26505-21sed12:20:263
471999140,14cyclictest291irq_work/208:40:172
471799149,4cyclictest29895-21grep10:15:141
4717991410,3cyclictest25831-21head10:05:181
471399143,10cyclictest13-21ksoftirqd/011:00:130
4713991410,3cyclictest13468-21sh08:35:010
471399140,4cyclictest13-21ksoftirqd/011:50:210
472399139,3cyclictest20858-21head09:55:163
472399139,3cyclictest13145-21sed07:25:163
4723991310,2cyclictest29885-21cut08:00:183
4723991310,2cyclictest29561-21head11:20:203
4723991310,2cyclictest27804-21gpgv09:05:003
4723991310,2cyclictest2760-21grep10:25:153
4723991310,2cyclictest27573-21grep07:55:153
4723991310,2cyclictest25881-21head10:05:183
4723991310,2cyclictest25376-21turbostat.cron09:00:003
4723991310,2cyclictest12680-21sh11:54:593
472399130,12cyclictest29608-21cat12:30:003
471999139,3cyclictest2884-21cron09:20:002
471999139,3cyclictest15776-21grep10:50:242
471999137,4cyclictest32-21ksoftirqd/208:35:122
4719991310,2cyclictest32-21ksoftirqd/212:20:012
4719991310,2cyclictest13849-21cat07:25:242
471999130,13cyclictest32421-21sshd09:11:202
471999130,12cyclictest27781-21http09:05:002
471799139,3cyclictest27749-21sh09:05:001
471799139,3cyclictest15583-21cat10:50:241
4717991310,2cyclictest5808-21grep07:10:151
4717991310,2cyclictest28865-21grep12:25:261
4717991310,2cyclictest23360-21wc07:45:231
4717991310,2cyclictest21965-21grep08:50:191
4717991310,2cyclictest20247-21grep07:40:151
4717991310,2cyclictest1737-21head11:30:171
4717991310,2cyclictest17030-21grep12:00:281
471799130,7cyclictest23539-21grep08:55:161
471399139,3cyclictest7160-21sort08:20:180
471399136,5cyclictest13-21ksoftirqd/009:35:260
471399133,8cyclictest22473-21apt-get12:15:130
471399132,7cyclictest13-21ksoftirqd/007:35:000
471399132,4cyclictest0-21swapper/009:15:130
4713991311,1cyclictest28463-21apt-get11:20:130
4713991310,2cyclictest6032-21grep11:40:120
4713991310,2cyclictest27120-21wc11:15:200
4713991310,2cyclictest22434-21ntpq11:05:200
4713991310,2cyclictest18826-21grep07:35:260
471399130,12cyclictest0-21swapper/008:00:280
472399129,2cyclictest32725-21sleep09:15:003
472399129,2cyclictest32012-21lockfile-create10:20:003
472399129,2cyclictest18640-21cut08:45:143
472399128,3cyclictest23201-21tail12:15:173
472399128,3cyclictest17443-21cron12:05:003
472399120,11cyclictest9781-21grep09:30:283
471999129,2cyclictest715-21sh11:30:002
471999129,2cyclictest3671-21grep09:20:152
471999129,2cyclictest28988-21head09:05:162
471999129,2cyclictest28783-21wc12:25:262
471999129,2cyclictest28171-21idleruntime-cro11:20:002
471999129,2cyclictest19750-21grep07:40:122
471999129,2cyclictest16442-21sed07:30:292
471999129,2cyclictest13645-21grep09:40:162
471999129,2cyclictest13335-21cat10:45:242
471999129,2cyclictest12807-21cat08:30:242
471999129,2cyclictest11731-21ls09:35:222
471999128,4cyclictest19697-21munin-node07:54:552
4719991210,1cyclictest14665-21grep11:55:252
471999120,11cyclictest22064-21grep11:05:162
471799129,2cyclictest8207-21grep08:20:281
471799129,2cyclictest6537-21grep11:40:141
471799129,2cyclictest29960-21grep08:00:191
471799129,2cyclictest28813-21wc10:10:221
471799129,2cyclictest22884-21ls11:05:251
471799129,2cyclictest21232-21tr12:10:191
471799129,2cyclictest15446-21sed08:35:241
471799129,2cyclictest13105-21grep07:25:161
471799129,2cyclictest11293-21grep11:50:181
471799129,2cyclictest10087-21chrt08:25:201
471799128,2cyclictest3971-21sort11:35:161
4717991210,1cyclictest28548-21grep07:55:241
471399129,2cyclictest8426-21tr09:30:150
471399129,2cyclictest8288-21cut07:15:160
471399129,2cyclictest30881-21taskset10:15:190
471399129,2cyclictest29140-21tail10:10:260
471399129,2cyclictest23941-21tr11:10:150
471399129,2cyclictest2248-21grep10:25:120
471399129,2cyclictest21162-21sed07:40:240
471399129,2cyclictest17304-21cut10:55:180
471399129,2cyclictest15589-21cut09:45:140
471399129,2cyclictest1534-21grep10:20:280
471399129,2cyclictest14666-21grep09:40:260
471399129,2cyclictest13049-21grep10:45:210
471399129,2cyclictest10347-21wc08:25:240
471399128,3cyclictest17949-21tr12:05:140
471399122,6cyclictest13-21ksoftirqd/010:50:120
472399118,2cyclictest7184-21chrt07:14:053
472399118,2cyclictest6280-21grep10:30:273
472399118,2cyclictest5263-21taskset09:22:033
472399118,2cyclictest3488-21cut11:35:143
472399118,2cyclictest28908-21expr10:10:263
472399118,2cyclictest24843-21cron10:05:003
472399118,2cyclictest24177-21grep11:10:143
472399118,2cyclictest23353-21grep07:45:233
472399118,2cyclictest18463-21tail09:50:163
472399118,2cyclictest1347-21expr11:30:153
472399118,2cyclictest12114-21cat10:45:143
472399118,2cyclictest10562-21basename09:35:133
472399118,2cyclictest10082-21ntpq08:25:203
472399117,3cyclictest3886-21idleruntime-cro08:15:003
472399116,3cyclictest16851-21sh07:35:013
472399115,2cyclictest225-21systemd-journal07:16:223
472399112,5cyclictest1436-21sendmail-mta09:05:033
472399110,10cyclictest24615-21egrep07:50:133
472399110,10cyclictest14776-21taskset09:41:153
471999119,1cyclictest6288-21grep10:30:272
471999119,1cyclictest25405-21grep10:05:162
471999119,1cyclictest1142-21runrttasks08:16:282
471999119,1cyclictest1060-21head10:20:232
471999118,2cyclictest7553-21taskset11:40:552
471999118,2cyclictest7378-21grep09:25:252
471999118,2cyclictest3610-21grep11:35:142
471999118,2cyclictest30569-21head10:15:172
471999118,2cyclictest25967-21grep12:20:202
471999118,2cyclictest23121-21sshd07:45:212
471999118,2cyclictest19949-21cat11:00:192
471999118,2cyclictest18910-21ls12:05:202
471999118,2cyclictest18791-21grep08:45:142
471999118,2cyclictest16577-21head09:45:212
471999118,2cyclictest1248-21tr11:30:152
471999110,10cyclictest8901-21tail09:30:182
471799118,2cyclictest9844-21sed11:45:271
471799118,2cyclictest7642-21grep07:15:131
471799118,2cyclictest4292-21tr08:15:151
471799118,2cyclictest32322-21head08:05:141
471799118,2cyclictest32257-21cut12:35:131
471799118,2cyclictest32238-21cut11:25:201
471799118,2cyclictest24062-21cat11:10:151
471799118,2cyclictest1914-21tr09:15:191
471799118,2cyclictest18945-21grep09:50:221
471799118,2cyclictest18759-21expr08:45:141
471799118,2cyclictest16402-21grep08:40:151
471799118,2cyclictest12111-21grep07:25:011
471799118,2cyclictest12099-21grep08:30:151
471799118,2cyclictest11223-21head09:35:171
471799117,3cyclictest3814-21taskset08:14:131
471799115,5cyclictest0-21swapper/111:15:141
471799112,6cyclictest30519-21latency_hist09:10:001
471399118,3cyclictest13-21ksoftirqd/010:30:240
471399118,2cyclictest7479-21grep09:25:270
471399118,2cyclictest5230-21cut07:10:120
471399118,2cyclictest28557-21grep07:55:240
471399118,2cyclictest26877-21chrt12:23:560
471399118,2cyclictest26388-21grep09:00:160
471399118,2cyclictest25792-21cut10:05:190
471399117,3cyclictest161rcuc/008:50:160
471399117,3cyclictest13-21ksoftirqd/011:35:150
471399114,3cyclictest30568-21apt-get11:25:030
472399108,1cyclictest98912sleep307:20:023
472399108,1cyclictest7302sleep309:15:103
472399108,1cyclictest495-21systemctl08:05:203
472399108,1cyclictest32351-21taskset10:20:163
472399108,1cyclictest32210-21cut12:35:133
472399108,1cyclictest221032sleep312:13:523
472399108,1cyclictest21661-21tail08:50:183
472399108,1cyclictest213262sleep307:41:223
472399108,1cyclictest18080-21cat12:05:133
472399108,1cyclictest157742sleep308:39:163
472399108,1cyclictest12193-21basename08:30:163
472399107,2cyclictest4952-21which08:15:173
472399107,2cyclictest31021-21cat11:25:093
472399107,2cyclictest17991-21sort10:55:273
472399107,2cyclictest17835-21grep07:35:163
472399106,3cyclictest19171-21tr11:00:153
472399100,9cyclictest8298-21mailstats10:35:213
471999108,2cyclictest27254-21expr07:55:122
471999108,1cyclictest9686-21tail08:25:162
471999108,1cyclictest28887-21chrt11:20:152
471999108,1cyclictest256142sleep211:12:502
471999108,1cyclictest23403-21cut10:00:182
471999108,1cyclictest17328-21cpuspeed_turbos07:35:142
471999108,1cyclictest1589-21date08:10:012
471999108,1cyclictest12243-21sed11:50:272
471999107,2cyclictest8461-21seq11:45:152
471999107,2cyclictest7070-21wget10:35:132
471999107,2cyclictest4051-21taskset10:26:392
471999107,2cyclictest21702-21sed12:10:252
471999107,0cyclictest0-21swapper/212:00:152
471999106,3cyclictest8852-21wc08:25:002
471999104,5cyclictest19099-21/usr/sbin/munin09:50:232
471999102,7cyclictest19697-21munin-node12:35:132
471799108,1cyclictest298252sleep112:30:141
471799108,1cyclictest23603-21ntpq12:15:211
471799108,1cyclictest20058-21cat09:55:111
471799108,1cyclictest15536-21hddtemp_smartct07:30:201
471799108,1cyclictest11787-21cut10:45:131
471799107,2cyclictest8445-21wc10:35:241
471799107,2cyclictest3677-21grep09:20:151
471799107,2cyclictest31338-21grep09:10:181
471799107,2cyclictest19663-21expr11:00:171
471799107,2cyclictest13902-21grep09:40:181
471799107,2cyclictest10239-21cron09:35:001
471799103,3cyclictest0-21swapper/111:20:141
471799102,4cyclictest225-21systemd-journal12:05:061
471399108,1cyclictest264012sleep007:52:540
471399108,1cyclictest22192-21cut07:45:140
471399108,1cyclictest200442sleep009:55:020
471399108,1cyclictest19510-21taskset09:51:090
471399107,2cyclictest748-21wc08:05:230
471399107,2cyclictest4134-21grep09:20:170
471399107,2cyclictest22811-21cut10:00:140
471399107,2cyclictest16415-21tr08:40:150
471399107,2cyclictest1605-21cut11:30:160
471399106,3cyclictest13-21ksoftirqd/008:20:000
471399105,2cyclictest14067-21head07:25:250
471399101,2cyclictest20619-21munin-run08:50:000
471399100,4cyclictest0-21swapper/012:30:150
47239997,1cyclictest16067-21expr09:45:173
47199998,1cyclictest21793-21irqstats08:50:182
47199998,1cyclictest118642sleep207:22:512
47199997,1cyclictest9864-21cut07:20:012
47199997,1cyclictest31663-21aten2_r7power_e08:05:112
47199997,1cyclictest20317-21tr09:55:142
47199997,1cyclictest16578-21tr10:55:152
47199996,2cyclictest9603-21tr10:40:142
47199996,2cyclictest29504-21cut08:00:172
47199996,2cyclictest23515-21grep08:55:162
47179997,1cyclictest5772-21taskset09:25:131
47179997,1cyclictest10495-21ntpq10:40:201
47179996,2cyclictest25510-21cut12:20:171
47179996,2cyclictest16438-21grep09:45:191
47139997,1cyclictest13824-21grep08:35:120
47139993,1cyclictest161rcuc/011:49:180
47139991,5cyclictest14257-21grep11:55:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional