You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 23:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Dec 29, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3000729885,9sleep00-21swapper/007:07:550
3008729066,8sleep20-21swapper/207:08:572
3010328168,9sleep30-21swapper/307:09:093
2988928164,13sleep10-21swapper/107:06:231
30265992917,12cyclictest0-21swapper/309:10:333
3026599212,19cyclictest20390-21kworker/3:009:15:023
3026599212,19cyclictest20390-21kworker/3:009:09:223
3026099210,20cyclictest0-21swapper/109:20:161
3026099204,8cyclictest0-21swapper/109:00:141
30260992019,1cyclictest0-21swapper/110:05:141
30265991910,6cyclictest381rcuc/308:45:203
30265991816,1cyclictest14719-21apt-get10:00:143
30265991814,3cyclictest39-21ksoftirqd/308:15:143
3025699185,6cyclictest13674-21munin-run11:05:010
3026599173,3cyclictest141rcu_preempt09:32:553
30260991716,0cyclictest0-21swapper/109:50:261
3026099170,0cyclictest0-21swapper/111:05:061
3025699175,6cyclictest14692-21munin-run07:44:590
3025699164,6cyclictest7325-21awk09:45:000
3026599150,2cyclictest0-21swapper/310:50:123
30260991511,3cyclictest15250-21sh12:15:001
3025699153,6cyclictest12060-21munin-run09:54:590
30265991411,2cyclictest12973-21wc11:00:283
3026599140,3cyclictest0-21swapper/307:14:573
3026599140,14cyclictest0-21swapper/310:30:173
30263991411,2cyclictest28291-21grep10:25:282
30260991412,1cyclictest11286-21find11:00:001
30260991410,2cyclictest29873-21grep08:15:161
30256991410,3cyclictest8103-21cron12:00:000
3026599139,3cyclictest29125-21tr11:35:153
3026599134,6cyclictest39-21ksoftirqd/308:40:013
3026599134,6cyclictest39-21ksoftirqd/308:35:003
3026599133,6cyclictest39-21ksoftirqd/311:35:003
3026599131,11cyclictest39-21ksoftirqd/308:20:143
30265991310,2cyclictest28844-21wc10:30:003
30265991310,2cyclictest24346-21grep12:30:273
30265991310,2cyclictest20239-21sed10:10:173
30265991310,2cyclictest17185-21grep12:15:283
3026599130,1cyclictest0-21swapper/307:50:203
3026399139,3cyclictest19802-21grep07:55:152
3026399135,4cyclictest225-21systemd-journal09:44:592
3026399131,8cyclictest21571-21turbostat.cron10:15:002
30263991310,2cyclictest5247-21tr08:30:242
30263991310,2cyclictest24036-21cat10:20:002
30263991310,2cyclictest21212-21cat11:20:002
30263991310,2cyclictest17143-21grep12:15:282
30263991310,2cyclictest13666-21cut08:50:152
30263991310,2cyclictest11626-21grep11:00:172
30263991310,2cyclictest11007-21cut12:05:152
3026099139,2cyclictest3900-21tr11:50:161
30260991310,2cyclictest666-21grep07:15:141
30260991310,2cyclictest6455-21wc10:50:011
30260991310,2cyclictest21677-21cat10:15:001
30260991310,2cyclictest15804-21tr12:15:161
30260991310,2cyclictest13631-21grep08:50:151
3026099130,13cyclictest0-21swapper/108:25:061
3025699139,3cyclictest7853-21grep09:45:140
3025699139,3cyclictest19642-21head11:15:170
3025699138,3cyclictest1895-21cut10:40:160
30256991310,2cyclictest31579-21wc09:25:210
30256991310,2cyclictest27196-21cron12:40:000
30256991310,2cyclictest26188-21cut09:15:190
30256991310,2cyclictest20914-21grep12:25:150
30256991310,2cyclictest19022-21grep07:50:250
30256991310,2cyclictest1018-21grep09:30:180
3025699130,12cyclictest4909-21grep10:45:160
3025699130,12cyclictest27415-21sed10:25:190
3026599129,2cyclictest7855-21tr09:45:143
3026599129,2cyclictest7697-21head11:55:263
3026599129,2cyclictest4955-21head10:45:163
3026599129,2cyclictest4230-21cut11:50:193
3026599129,2cyclictest381rcuc/309:00:133
3026599129,2cyclictest3400-21grep07:20:163
3026599129,2cyclictest32232-21cut10:35:163
3026599129,2cyclictest2898-21wc08:25:243
3026599129,2cyclictest23948-21apt10:20:003
3026599129,2cyclictest23287-21sed08:00:203
3026599129,2cyclictest21934-21tail12:25:253
3026599129,2cyclictest19132-21tr11:15:133
3026599129,2cyclictest18649-21grep12:20:173
3026599129,2cyclictest14258-21cut12:10:203
3026599129,2cyclictest13641-21wc07:40:243
3026599128,3cyclictest10693-21wc12:05:003
3026599120,8cyclictest0-21swapper/309:50:143
3026599120,11cyclictest2922-21grep10:40:213
3026399129,2cyclictest6297-21cron10:50:012
3026399129,2cyclictest4513-21grep07:20:272
3026399129,2cyclictest3923-21cron10:45:002
3026399129,2cyclictest3352-21grep09:35:162
3026399129,2cyclictest31480-21grep09:25:222
3026399129,2cyclictest31009-21sh11:39:592
3026399129,2cyclictest27172-21cut08:10:172
3026399129,2cyclictest25402-21uname08:05:192
3026399129,2cyclictest24591-21grep11:25:192
3026399129,2cyclictest24507-21grep10:20:122
3026399129,2cyclictest23765-21tail08:00:252
3026399129,2cyclictest21515-21tr11:20:152
3026399129,2cyclictest18844-21grep09:00:152
3026399129,2cyclictest16150-21tr08:55:162
3026399129,2cyclictest16150-21tr08:55:162
3026399129,2cyclictest13316-21tail07:40:192
3026399128,3cyclictest1558-21cat08:25:002
3026399126,2cyclictest225-21systemd-journal11:55:002
3026399126,2cyclictest225-21systemd-journal07:54:592
3026399124,4cyclictest225-21systemd-journal07:26:322
3026099129,2cyclictest8477-21grep07:30:181
3026099129,2cyclictest4754-21head08:30:191
3026099129,2cyclictest26932-21cat09:15:261
3026099129,2cyclictest2594-21grep10:40:191
3026099129,2cyclictest13349-21head07:40:191
3026099129,2cyclictest13083-21head09:55:171
3026099129,2cyclictest10720-21taskset08:42:591
3026099128,3cyclictest7331-21grep10:50:151
3026099128,3cyclictest25151-21cron09:15:001
3026099126,2cyclictest225-21systemd-journal12:00:001
3026099126,2cyclictest225-21systemd-journal11:39:591
30260991210,1cyclictest22114-21tr10:15:151
30260991210,1cyclictest17999-21cron09:00:001
30260991210,1cyclictest17999-21cron09:00:001
3025699129,2cyclictest4972-21basename07:25:010
3025699129,2cyclictest454-21grep08:20:210
3025699129,2cyclictest3451-21grep09:35:170
3025699129,2cyclictest32150-21grep07:10:260
3025699129,2cyclictest2291-21sort08:25:200
3025699129,2cyclictest20558-21cat10:10:200
3025699129,2cyclictest1523-21tr11:45:140
3025699129,2cyclictest14978-21tr10:00:140
3025699121,8cyclictest17639-21apt12:20:000
3025699120,11cyclictest21217-21df09:05:150
3026599119,2cyclictest39-21ksoftirqd/311:11:033
3026599119,1cyclictest3905-21grep09:35:213
3026599119,1cyclictest381rcuc/308:40:173
3026599118,2cyclictest5258-21cut09:40:153
3026599118,2cyclictest31860-21grep09:25:253
3026599118,2cyclictest2947-21grep11:45:253
3026599118,2cyclictest29000-21cron08:15:003
3026599118,2cyclictest15495-21which11:05:213
3026599118,2cyclictest0-21swapper/310:20:193
3026599117,3cyclictest31549-21tr11:40:173
3026599117,3cyclictest25327-21tr12:35:153
3026599116,4cyclictest9231-21cpuspeed_turbos10:55:183
3026599114,5cyclictest39-21ksoftirqd/311:28:183
3026599114,5cyclictest39-21ksoftirqd/311:22:053
3026599113,8cyclictest39-21ksoftirqd/310:05:133
3026599113,6cyclictest39-21ksoftirqd/308:50:273
3026399118,2cyclictest9580-21taskset07:30:472
3026399118,2cyclictest8213-21cut09:45:162
3026399118,2cyclictest7006-21cut08:35:182
3026399118,2cyclictest32712-21wc07:15:002
3026399118,2cyclictest31904-21tr10:35:142
3026399118,2cyclictest30053-21wc09:25:002
3026399118,2cyclictest29413-21grep08:15:142
3026399118,2cyclictest26051-21sort09:15:172
3026399118,2cyclictest23677-21sort09:10:182
3026399118,2cyclictest23391-21tail12:30:182
3026399118,2cyclictest22369-21sed09:05:262
3026399118,2cyclictest21959-21grep12:25:252
3026399118,2cyclictest1947-21cat08:25:172
3026399118,2cyclictest16790-21tr11:10:152
3026399118,2cyclictest15672-21tail07:45:172
3026399118,2cyclictest14877-21grep11:05:162
3026399118,2cyclictest14838-21head12:10:262
3026399118,2cyclictest10010-21sed12:00:282
3026399114,6cyclictest29011-21/usr/sbin/munin10:30:222
3026399110,10cyclictest9173-21expr10:55:172
3026099119,1cyclictest32506-21apt-config09:30:011
3026099118,2cyclictest898-21grep08:20:251
3026099118,2cyclictest5707-21grep07:25:151
3026099118,2cyclictest5482-21grep09:40:161
3026099118,2cyclictest3093-21cut07:20:151
3026099118,2cyclictest30829-21expr07:10:141
3026099118,2cyclictest29309-21cut10:30:171
3026099118,2cyclictest28035-21ntpq08:10:231
3026099118,2cyclictest23197-21head08:00:191
3026099118,2cyclictest21453-21grep09:05:161
3026099118,2cyclictest21209-21cat07:55:261
3026099118,2cyclictest18684-21head12:20:181
3026099118,2cyclictest16648-21grep07:45:281
3026099118,2cyclictest12909-21cat12:10:011
3026099117,3cyclictest25097-21grep11:25:231
3026099115,5cyclictest17990-21tune2fs07:50:161
3025699118,2cyclictest9028-21ls07:30:230
3025699118,2cyclictest6851-21grep10:50:130
3025699118,2cyclictest4229-21grep11:50:180
3025699118,2cyclictest30129-21grep10:30:220
3025699118,2cyclictest29496-21sort11:35:170
3025699118,2cyclictest29415-21tr08:15:140
3025699118,2cyclictest28529-21cut09:20:160
3025699118,2cyclictest27877-21cat11:30:270
3025699118,2cyclictest27854-21cut08:10:210
3025699118,2cyclictest24705-21cat11:25:190
3025699118,2cyclictest23599-21uname12:30:180
3025699118,2cyclictest22003-21grep11:20:170
3025699118,2cyclictest21402-21head07:55:260
3025699118,2cyclictest18281-21ls10:05:210
3025699118,2cyclictest15703-21head07:45:180
3025699118,2cyclictest15191-21sed08:50:260
3025699118,2cyclictest1509-21wc07:15:190
3025699118,2cyclictest14481-21expr11:05:140
3025699118,2cyclictest14332-21taskset09:58:560
3025699118,2cyclictest12210-21wc08:45:190
3025699118,2cyclictest10179-21python08:40:280
3025699110,10cyclictest9758-21run-parts12:00:240
3026599107,2cyclictest5720-21tr07:25:153
3026599107,2cyclictest28367-21sed09:20:143
3026599107,2cyclictest21780-21chrt07:59:503
3026599107,2cyclictest16838-21chrt07:47:193
3026599107,2cyclictest16445-21df_abs08:55:173
3026599107,2cyclictest16445-21df_abs08:55:173
3026599103,5cyclictest39-21ksoftirqd/307:30:153
3026599102,4cyclictest25224-21gltestperf08:05:173
3026599101,5cyclictest225-21systemd-journal07:20:013
3026599100,9cyclictest12430-21awk09:55:143
3026399108,1cyclictest841-21chrt11:43:262
3026399108,1cyclictest7971-21chrt11:58:252
3026399108,1cyclictest26969-21grep11:30:172
3026399108,1cyclictest2226-21taskset09:31:082
3026399108,1cyclictest19789-21taskset12:22:062
3026399108,1cyclictest16788-21taskset10:04:422
3026399107,2cyclictest443-21grep07:15:152
3026399107,2cyclictest17341-21tr10:05:152
3026399107,2cyclictest12601-21grep09:55:142
3026399107,2cyclictest11787-21taskset09:51:442
3026399106,3cyclictest11921-21sed08:45:162
3026399103,2cyclictest225-21systemd-journal12:35:002
3026399102,4cyclictest1269-21cut11:45:132
3026399100,9cyclictest9829-21sed08:40:232
3026399100,0cyclictest141rcu_preempt10:50:142
3026099108,2cyclictest24835-21expr08:05:151
3026099108,2cyclictest11407-21expr08:45:141
3026099108,1cyclictest6753-21expr08:35:161
3026099108,1cyclictest48692sleep109:39:361
3026099108,1cyclictest32704-21cat09:30:011
3026099108,1cyclictest26385-21cut11:30:151
3026099108,1cyclictest25029-21basename12:35:121
3026099108,1cyclictest21418-21grep12:25:191
3026099108,1cyclictest1975-21grep11:45:161
3026099108,1cyclictest17268-21sed11:10:171
3026099107,2cyclictest9146-21tail12:00:191
3026099107,2cyclictest7695-21aten2_r7power_v09:45:141
3026099107,2cyclictest31769-21wget10:35:131
3026099107,2cyclictest20020-21tr11:15:191
3026099107,2cyclictest11784-21grep11:00:181
3026099100,9cyclictest31674-21expr11:40:181
3026099100,9cyclictest26840-21cut10:25:161
3026099100,9cyclictest10500-21expr07:35:131
3026099100,6cyclictest16618-21sshd10:02:321
3025699108,1cyclictest6149-21grep07:25:180
3025699108,1cyclictest31876-21sh11:40:190
3025699108,1cyclictest24679-21expr10:20:140
3025699108,1cyclictest23196-21egrep09:10:150
3025699108,1cyclictest21835-21apache_processe10:15:120
3025699108,1cyclictest10728-21taskset10:55:580
3025699108,1cyclictest10616-21expr07:35:140
3025699107,2cyclictest32764-21ntpq10:35:210
3025699107,2cyclictest24502-21wget08:05:140
3025699107,2cyclictest23812-21sed08:00:250
3025699107,2cyclictest18839-21uname12:20:200
3025699107,2cyclictest18265-21/usr/sbin/munin09:00:150
3025699107,2cyclictest17066-21df11:10:150
3025699107,2cyclictest16892-21ls08:55:200
3025699107,2cyclictest16892-21ls08:55:200
3025699107,2cyclictest12604-21taskset12:06:350
3025699104,5cyclictest8487-21cron08:40:000
3025699102,7cyclictest14138-21latency12:10:190
302659997,1cyclictest10671-21sed07:35:143
302659996,2cyclictest11504-21head12:05:163
302639996,2cyclictest10285-21grep07:35:132
302609997,1cyclictest26288-21taskset10:24:301
302609997,1cyclictest21486-21wc11:20:151
302609996,2cyclictest22920-21cut12:30:141
302569996,2cyclictest4595-21dnf08:30:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional