You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-25 - 07:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Dec 25, 2025 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2992029077,9sleep00-21swapper/019:06:330
3007928560,9sleep30-21swapper/319:08:383
2996428471,9sleep20-21swapper/219:07:072
2994827958,6sleep10-21swapper/119:06:541
3027199260,26cyclictest0-21swapper/121:34:071
3027199241,1cyclictest11744-21ls19:35:261
30275992019,1cyclictest0-21swapper/219:59:472
3026899190,18cyclictest0-21swapper/021:50:140
3026899190,16cyclictest0-21swapper/020:45:140
30278991812,5cyclictest39-21ksoftirqd/320:23:203
3027899171,12cyclictest0-21swapper/323:55:143
3027899166,4cyclictest18076-21apt-get23:15:143
30275991614,1cyclictest0-21swapper/221:50:142
3027899156,8cyclictest21235-21apache219:40:173
3027899149,3cyclictest30587-21tr21:25:133
30278991410,3cyclictest9738-21awk21:50:013
30278991410,3cyclictest28817-21turbostat.cron22:30:003
3027599140,13cyclictest16578-21cut20:55:172
3027199142,5cyclictest7089-21munin-run00:00:001
30271991412,1cyclictest9737-21apt-get00:05:131
30271991411,2cyclictest19459-21perf19:55:011
3026899149,2cyclictest11863-31su00:08:560
3026899140,2cyclictest0-21swapper/020:05:160
3027899139,3cyclictest26641-21sh20:10:003
3027899138,3cyclictest18375-21grep21:00:133
30278991310,2cyclictest6572-21tr20:35:143
30278991310,2cyclictest4519-21grep21:35:273
30278991310,2cyclictest3825-21wc20:30:003
30278991310,2cyclictest25989-21grep00:35:253
30278991310,2cyclictest22486-21grep21:05:273
30278991310,2cyclictest15673-21head19:45:173
3027599138,3cyclictest30047-21grep21:25:012
3027599136,7cyclictest0-21swapper/219:34:082
3027599133,6cyclictest21273-21cups-browsed00:09:042
30275991310,2cyclictest8556-21ls00:00:252
30275991310,2cyclictest24256-21grep21:10:212
30275991310,2cyclictest16906-21sendmail-msp00:20:002
3027199139,3cyclictest8499-21grep19:30:141
3027199139,3cyclictest15019-21tr00:15:141
3027199138,4cyclictest5381-21grep20:30:241
3027199137,5cyclictest16510-21ls19:45:251
3027199132,5cyclictest452-21systemd-logind20:23:211
30271991310,2cyclictest8692-21mailstats00:00:261
30271991310,2cyclictest30537-21grep23:40:151
30271991310,2cyclictest29323-21grep22:30:181
30271991310,2cyclictest2762-21wc22:40:221
30271991310,2cyclictest23852-21sed20:00:291
30271991310,2cyclictest18965-21sort21:00:161
3026899139,1cyclictest27629-21cron21:20:000
3027899129,2cyclictest9072-21grep20:40:133
3027899129,2cyclictest8555-21grep19:30:153
3027899129,2cyclictest8007-21cut00:00:203
3027899129,2cyclictest7846-21grep22:50:273
3027899129,2cyclictest29908-21cut20:15:173
3027899129,2cyclictest29802-21tail22:30:193
3027899129,2cyclictest26209-21head23:30:193
3027899129,2cyclictest23186-21grep23:25:163
3027899129,2cyclictest20178-21cut22:10:163
3027899129,2cyclictest18408-21wc00:20:203
3027899129,2cyclictest13602-21sed00:10:233
3027899128,3cyclictest29992-21awk23:40:003
3027899128,3cyclictest15829-21wc20:55:013
3027899127,3cyclictest31952-21wc19:10:253
3027899127,3cyclictest1138-21sh22:39:593
3027599129,2cyclictest7135-21grep20:35:162
3027599129,2cyclictest31500-21ls21:25:202
3027599129,2cyclictest27942-21cat20:10:192
3027599129,2cyclictest26850-21find21:15:222
3027599129,2cyclictest26155-21grep20:05:292
3027599129,2cyclictest25240-21awk23:30:002
3027599129,2cyclictest22879-21cron21:10:002
3027599129,2cyclictest19595-21grep00:25:142
3027599129,2cyclictest19106-21head21:00:162
3027599129,2cyclictest17898-21grep22:05:162
3027599129,2cyclictest10671-21munin-plugin-st23:00:002
3027599129,2cyclictest1023-21grep21:30:172
3027599122,9cyclictest0-21swapper/220:20:132
30275991210,1cyclictest1861-21grep23:45:262
3027599120,9cyclictest56550irq/124-enp0s3123:42:282
3027599120,11cyclictest15700-21grep19:45:162
3027199129,2cyclictest4037-21tr22:45:151
3027199129,2cyclictest3579-21grep19:20:181
3027199129,2cyclictest29031-21ls20:14:591
3027199129,2cyclictest28802-21cron22:30:001
3027199129,2cyclictest24809-21head23:25:281
3027199129,2cyclictest13614-21grep23:05:151
3027199128,3cyclictest19783-21grep00:25:151
3027199128,3cyclictest1419-21cut23:45:211
3027199128,3cyclictest13121-21grep21:55:161
30271991210,1cyclictest0-21swapper/122:05:131
3026899129,2cyclictest9271-21tail21:45:240
3026899129,2cyclictest5340-21cut19:25:130
3026899129,2cyclictest3561-21cut19:20:160
3026899129,2cyclictest17151-21chrt00:20:100
3026899129,2cyclictest15236-21tr19:45:140
3026899129,2cyclictest13779-21wc21:55:240
3026899129,2cyclictest13591-21tr23:05:150
3026899129,2cyclictest12611-21tr00:10:150
3026899128,3cyclictest24344-21head23:25:240
3026899128,2cyclictest29624-21expr20:15:150
3026899125,6cyclictest9948-21munin-run19:35:000
3026899122,7cyclictest23157-21apache_volume21:10:140
30268991210,1cyclictest20722-21sed22:10:200
30268991210,1cyclictest16017-21wc00:15:220
3027899118,2cyclictest9164-21cut22:55:173
3027899118,2cyclictest739-21grep21:30:163
3027899118,2cyclictest7297-21taskset21:44:283
3027899118,2cyclictest6126-21taskset20:34:373
3027899118,2cyclictest444-21aten2_r7power_c19:15:133
3027899118,2cyclictest32045-21chrt23:40:593
3027899118,2cyclictest29243-21grep21:20:223
3027899118,2cyclictest2882-21tr23:50:143
3027899118,2cyclictest27218-21cat20:10:153
3027899118,2cyclictest2537-21grep22:40:203
3027899118,2cyclictest24002-21chrt20:01:423
3027899118,2cyclictest23574-21sed22:15:263
3027899118,2cyclictest21943-21aten2_r7power_e00:30:143
3027899118,2cyclictest19283-21chrt19:53:053
3027899118,2cyclictest14844-21cat23:05:263
3027899118,2cyclictest11976-21grep20:45:173
3027899118,2cyclictest11310-21wc21:50:213
3027899118,2cyclictest10231-21cut19:35:143
3027899117,3cyclictest16855-21cron22:05:013
3027899113,4cyclictest225-21systemd-journal23:15:003
3027599119,1cyclictest27597-21turbostat.cron23:35:012
3027599119,1cyclictest12839-21grep19:40:152
3027599118,3cyclictest3038-21seq21:35:152
3027599118,2cyclictest7080-21ls00:00:012
3027599118,2cyclictest4515-21grep22:45:162
3027599118,2cyclictest31707-21tr22:35:142
3027599118,2cyclictest26993-21tr22:25:142
3027599118,2cyclictest26312-21taskset22:23:302
3027599118,2cyclictest21159-21tail22:10:252
3027599118,2cyclictest12589-21tr21:55:132
3027599114,4cyclictest225-21systemd-journal00:15:002
3027599113,4cyclictest225-21systemd-journal23:55:002
3027599110,10cyclictest1706-21grep22:40:152
3027199119,1cyclictest11432sleep119:15:161
3027199118,2cyclictest9638-21wc20:40:161
3027199118,2cyclictest8236-21chrt20:36:131
3027199118,2cyclictest6468-21tr19:25:201
3027199118,2cyclictest6344-21grep21:40:201
3027199118,2cyclictest3653-21taskset20:28:191
3027199118,2cyclictest31292-21cat21:25:191
3027199118,2cyclictest29958-21cron23:40:011
3027199118,2cyclictest25572-21grep20:05:221
3027199118,2cyclictest22699-21tail22:15:161
3027199118,2cyclictest21385-21grep19:55:271
3027199118,2cyclictest17864-21grep00:20:151
3027199118,2cyclictest16532-21cat20:55:161
3027199118,2cyclictest14715-21perf19:45:011
3027199118,2cyclictest12453-21tr00:10:151
3027199118,2cyclictest10614-21sort21:50:171
3027199117,3cyclictest11466-21grep20:45:151
3027199110,10cyclictest4495-21sed21:35:271
3026899119,2cyclictest1483-21head21:30:210
3026899119,1cyclictest8963-21grep20:40:120
3026899119,1cyclictest7415-21tr22:50:220
3026899119,1cyclictest30971-21tail23:40:170
3026899119,1cyclictest2790-21tr23:50:150
3026899119,1cyclictest15836-21grep23:10:150
3026899118,2cyclictest9026-21tail00:00:300
3026899118,2cyclictest694-21tr19:15:150
3026899118,2cyclictest5699-21grep23:55:170
3026899118,2cyclictest4897-21tail22:45:200
3026899118,2cyclictest4787-21tail20:30:180
3026899118,2cyclictest4391-21wc21:35:260
3026899118,2cyclictest29620-21grep21:20:260
3026899118,2cyclictest29511-21tail23:35:270
3026899118,2cyclictest28122-21wc22:25:240
3026899118,2cyclictest25936-21tail00:35:240
3026899118,2cyclictest2437-21tail20:25:170
3026899118,2cyclictest22303-21wget20:00:160
3026899118,2cyclictest20547-21cat21:05:010
3026899118,2cyclictest19266-21ls23:15:220
3026899118,2cyclictest18620-21sed22:05:250
3026899118,2cyclictest1716-21expr22:40:150
3026899118,2cyclictest11705-21sed23:00:170
3026899118,2cyclictest10202-21grep22:55:280
3026899117,3cyclictest19784-21tr00:25:150
3026899115,5cyclictest24051-21sendmail22:20:000
3026899113,5cyclictest0-21swapper/020:35:130
3026899110,10cyclictest19161-21chrt19:51:300
3027899108,1cyclictest4228-21cat19:20:233
3027899108,1cyclictest4139-21expr22:45:143
3027899108,1cyclictest268332sleep321:15:223
3027899108,1cyclictest23267-21awk21:10:143
3027899108,1cyclictest20218-21tune2fs00:25:173
3027899108,1cyclictest17938-21seq20:58:413
3027899108,1cyclictest15203-21cat00:15:143
3027899108,1cyclictest13724-21systemctl21:55:213
3027899108,1cyclictest12108-21ntpq23:00:203
3027899107,2cyclictest5884-21cut19:25:153
3027899107,2cyclictest25115-21grep22:20:163
3027899107,2cyclictest20832-21wc23:20:143
3027899107,2cyclictest11712-21chrt00:08:313
3027899107,2cyclictest1161-21grep23:45:193
3027899103,3cyclictest225-21systemd-journal22:05:203
3027599108,2cyclictest6262-21cat20:35:002
3027599108,2cyclictest4350-21expr19:20:262
3027599108,2cyclictest29347-21grep22:30:182
3027599108,2cyclictest23718-21taskset00:31:252
3027599108,2cyclictest1070-21grep19:15:152
3027599108,1cyclictest25942-21grep00:35:242
3027599108,1cyclictest21900-21chrt22:15:082
3027599108,1cyclictest20968-21cut23:20:152
3027599108,1cyclictest19596-21cat19:55:012
3027599108,1cyclictest17017-21cat22:05:012
3027599107,2cyclictest29997-21tr23:40:012
3027599107,2cyclictest18863-21sed00:20:242
3027599107,2cyclictest11739-21cat19:35:262
3027599105,5cyclictest0-21swapper/221:44:542
3027599105,2cyclictest225-21systemd-journal21:47:272
3027599104,3cyclictest452-21systemd-logind20:49:042
3027599103,6cyclictest7578-21python22:50:262
3027599103,3cyclictest225-21systemd-journal20:54:592
3027599103,2cyclictest225-21systemd-journal23:11:032
3027599100,9cyclictest51962sleep219:25:032
3027199108,1cyclictest9571-21load22:55:211
3027199108,1cyclictest30951-21tail20:15:261
3027199108,1cyclictest296952sleep121:20:501
3027199108,1cyclictest261122sleep122:20:551
3027199108,1cyclictest25995-21expr21:15:151
3027199108,1cyclictest25339-21cat00:35:171
3027199108,1cyclictest21402-21sort21:05:161
3027199108,1cyclictest19872-21expr22:10:131
3027199108,1cyclictest15913-21ls23:10:161
3027199107,2cyclictest7881-21sed22:50:261
3027199107,2cyclictest32594-21cat19:14:551
3027199107,2cyclictest2736-21grep23:50:141
3027199107,2cyclictest25723-21grep23:30:161
3027199107,2cyclictest23114-21chrt21:10:041
3027199107,2cyclictest22622-21head00:30:181
3027199107,2cyclictest21495-21grep23:20:171
3027199107,2cyclictest19395-21wc23:15:221
3027199107,2cyclictest15116-21expr22:00:161
3027199107,2cyclictest11742-21sh23:00:171
3027199107,2cyclictest1161-21turbostat22:39:591
3027199103,7cyclictest0-21swapper/121:50:001
3026899108,1cyclictest31988-21wc19:10:250
3026899108,1cyclictest28622-21grep20:10:260
3026899108,1cyclictest24640-21cat22:20:140
3026899108,1cyclictest17459-21sort20:55:260
3026899108,1cyclictest15787-21wc22:00:210
3026899108,1cyclictest144712sleep019:42:120
3026899107,2cyclictest857-21chrt22:36:330
3026899107,2cyclictest5449-21cat21:40:160
3026899107,2cyclictest32153-21expr20:20:150
3026899107,2cyclictest25761-21tr23:30:150
3026899107,2cyclictest22173-21tr00:30:160
3026899107,2cyclictest2127-21chrt23:47:380
3026899107,2cyclictest20687-21apt-get23:20:140
3026899107,2cyclictest15514-21chrt20:52:590
3026899107,2cyclictest11016-21basename19:35:180
3026899105,2cyclictest225-21systemd-journal22:35:000
3026899100,9cyclictest216692sleep019:57:270
302789996,2cyclictest19760-21cut19:55:153
302759998,1cyclictest310682sleep220:15:522
302759998,1cyclictest22517-21expr20:00:182
302759998,1cyclictest1877-21grep20:25:152
302759998,1cyclictest18533-21expr23:15:152
302759998,1cyclictest11369-21expr23:00:142
302759997,2cyclictest13588-21expr23:05:152
302759997,1cyclictest30810-21cat19:10:142
302759997,1cyclictest10225-21smartctl20:40:232
302719996,2cyclictest13916-21expr20:50:161
302689996,2cyclictest20965-21tr21:05:140
302689995,3cyclictest30269-21/usr/sbin/munin21:25:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional