You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-17 - 21:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Dec 17, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3269929076,9sleep20-21swapper/207:08:132
3046728873,10sleep00-21swapper/007:05:010
37528164,13sleep10-21swapper/107:10:011
3253127865,9sleep30-21swapper/307:06:053
47999220,1cyclictest0-21swapper/210:40:142
479992018,1cyclictest3055-21/usr/sbin/munin09:30:192
47299193,12cyclictest0-21swapper/109:50:121
486991815,2cyclictest39-21ksoftirqd/307:54:593
47999171,14cyclictest0-21swapper/207:25:132
472991716,1cyclictest0-21swapper/109:27:111
46999175,6cyclictest12562-21munin-run12:05:000
47999169,6cyclictest455-21dbus-daemon07:30:222
47999169,6cyclictest0-21swapper/207:35:162
47299163,2cyclictest225-21systemd-journal08:14:591
47999153,4cyclictest225-21systemd-journal08:05:002
479991511,3cyclictest584-21ls09:25:002
47999150,14cyclictest56550irq/124-enp0s3110:56:292
46999156,8cyclictest9991-21apt-get07:30:000
46999151,7cyclictest0-21swapper/007:10:210
47999141,12cyclictest29365-21latency_hist08:09:592
479991411,2cyclictest18144-21head10:00:162
479991410,3cyclictest24589-21wc09:05:202
47999140,1cyclictest0-21swapper/212:35:132
47299148,5cyclictest11788-21fgrep07:30:221
472991410,3cyclictest26119-21wc10:15:261
46999149,3cyclictest8084-21grep11:55:130
46999149,3cyclictest14126-21basename11:00:140
469991412,1cyclictest13-21ksoftirqd/010:40:180
469991411,2cyclictest13-21ksoftirqd/007:15:110
469991410,3cyclictest16513-21basename08:50:150
469991410,3cyclictest16513-21basename08:50:140
46999140,7cyclictest22169-21awk07:55:000
48699139,3cyclictest14717-21perf09:54:593
48699139,2cyclictest30479-21turbostat.cron09:20:013
48699132,8cyclictest0-21swapper/308:05:003
486991310,2cyclictest30188-21sort08:10:153
486991310,2cyclictest29260-21basename12:40:013
486991310,2cyclictest28246-21wc08:05:183
486991310,2cyclictest23555-21grep11:20:013
486991310,2cyclictest18044-21grep08:50:263
486991310,2cyclictest18044-21grep08:50:263
486991310,2cyclictest14951-21cat12:09:593
486991310,2cyclictest12695-21cat10:55:193
47999139,3cyclictest29337-21grep11:30:172
47999139,2cyclictest29682-21grep10:25:122
47999134,8cyclictest17166-21latency_hist10:00:002
47999133,9cyclictest0-21swapper/212:00:002
479991310,2cyclictest640-21munin-plugin-st11:39:592
479991310,2cyclictest5784-21tr07:20:152
479991310,2cyclictest21934-21grep09:00:172
479991310,2cyclictest19788-21wc07:50:002
479991310,2cyclictest15906-21grep12:10:162
47999130,3cyclictest0-21swapper/208:50:142
47999130,3cyclictest0-21swapper/208:50:142
47999130,12cyclictest0-21swapper/207:40:132
47999130,11cyclictest15246-21pmu-power08:45:212
47299139,3cyclictest9949-21cron09:44:591
47299139,3cyclictest1509-21grep07:10:241
47299139,3cyclictest12609-21wc08:40:191
47299131,2cyclictest0-21swapper/107:25:121
472991310,2cyclictest6416-21sleep08:30:001
472991310,2cyclictest22927-21tail10:10:171
472991310,2cyclictest17587-21tr10:00:141
46999139,3cyclictest7116-21grep07:20:260
46999136,6cyclictest6417-21apt-get08:30:000
46999135,7cyclictest29266-21munin-run12:40:000
469991310,2cyclictest979-21head10:30:210
48699129,2cyclictest9985-21grep10:50:163
48699129,2cyclictest5401-21cron11:50:003
48699129,2cyclictest415-21grep10:30:163
48699129,2cyclictest22646-21grep07:55:163
48699129,2cyclictest20576-21grep12:20:153
48699129,2cyclictest19539-21grep08:55:173
48699129,2cyclictest16133-21cron11:05:013
48699129,2cyclictest14075-21grep07:35:233
48699129,2cyclictest12054-21grep12:00:253
48699129,2cyclictest10723-21tr07:30:143
486991210,1cyclictest31717-21grep11:35:173
486991210,1cyclictest25548-21grep12:30:163
486991210,1cyclictest23038-21grep12:25:183
47999129,2cyclictest8809-21cat09:40:182
47999129,2cyclictest3819-21grep07:15:162
47999129,2cyclictest343-21cut10:30:162
47999129,2cyclictest21501-21wc12:20:232
47999129,2cyclictest20733-21grep07:50:172
47999129,2cyclictest20509-21grep10:05:162
47999129,2cyclictest18840-21grep08:55:142
47999129,2cyclictest14460-21grep12:05:262
47999129,2cyclictest12424-21grep08:40:172
47999128,3cyclictest5691-21grep09:35:132
47999128,3cyclictest29998-21sed09:15:252
47999128,3cyclictest25769-21grep10:15:212
47999128,3cyclictest10378-21tr08:35:192
47999120,3cyclictest4047-21perf08:24:592
47999120,11cyclictest12564-21perf12:05:002
47299129,2cyclictest7111-21grep09:35:241
47299129,2cyclictest32229-21tr08:15:151
47299129,2cyclictest21064-21head12:20:181
47299129,2cyclictest18398-21cut07:45:171
47299129,2cyclictest17167-21cut11:05:171
47299129,2cyclictest14741-21tail11:00:161
47299128,3cyclictest8527-21grep08:30:271
47299128,3cyclictest22648-21tr07:55:151
47299128,2cyclictest12915-21grep07:35:151
47299126,4cyclictest15729-21df_abs07:40:151
47299124,4cyclictest225-21systemd-journal09:35:001
47299121,9cyclictest32745-21diskstats10:30:151
46999129,2cyclictest4326-21wc10:40:000
46999129,2cyclictest31732-21sed11:35:180
46999129,2cyclictest3148-21grep08:20:200
46999129,2cyclictest29706-21tr11:30:200
46999129,2cyclictest26952-21cat08:05:000
46999129,2cyclictest26939-21grep10:20:000
46999129,2cyclictest24923-21cat11:20:210
46999129,2cyclictest15720-21head09:55:170
46999129,2cyclictest11977-21cat07:30:240
46999125,2cyclictest141rcu_preempt07:55:260
469991210,1cyclictest29313-21idleruntime-cro10:24:590
46999120,7cyclictest24301-21awk10:15:000
48699119,1cyclictest9273-21grep11:55:213
48699118,2cyclictest7185-21who09:35:253
48699118,2cyclictest27352-21grep11:25:193
48699118,2cyclictest24517-21awk09:05:193
48699118,2cyclictest24216-21expr11:20:153
48699118,2cyclictest17708-21grep12:15:123
48699118,2cyclictest16753-21expr11:05:153
48699118,2cyclictest15020-21cat08:45:193
48699118,2cyclictest13337-21grep08:40:263
48699118,2cyclictest10771-21df09:45:153
48699116,2cyclictest225-21systemd-journal08:24:593
48699111,1cyclictest0-21swapper/309:25:143
48699110,10cyclictest17120-21awk10:00:003
47999118,2cyclictest6410-21idleruntime-cro08:30:002
47999118,2cyclictest31465-21chrt08:12:312
47999118,2cyclictest27508-21cat10:20:142
47999118,2cyclictest24886-21wc11:20:212
47999118,2cyclictest23072-21sort12:25:182
47999118,2cyclictest2163-21wget10:35:142
47999118,2cyclictest2037-21cut11:40:182
47999118,2cyclictest20135-21ntpq11:10:182
47999118,2cyclictest18333-21tail12:15:142
47999118,2cyclictest12869-21tr09:50:132
47999118,2cyclictest1272-21taskset08:15:312
47999113,4cyclictest225-21systemd-journal11:50:002
47299119,2cyclictest38150irq/131-snd_hda09:20:181
47299119,2cyclictest3590-21sed10:35:251
47299118,2cyclictest9150-21grep11:55:201
47299118,2cyclictest4024-21cron08:24:591
47299118,2cyclictest30201-21sed11:30:261
47299118,2cyclictest29291-21cron08:09:591
47299118,2cyclictest27696-21grep10:20:141
47299118,2cyclictest27633-21grep12:35:141
47299118,2cyclictest26734-21sed09:10:171
47299118,2cyclictest23899-21cut11:20:151
47299118,2cyclictest21847-21cut09:00:171
47299118,2cyclictest21755-21cat11:15:161
47299118,2cyclictest12346-21grep10:55:181
47299118,2cyclictest10581-21grep12:00:141
47299118,2cyclictest10006-21cut08:35:161
47299116,4cyclictest17661-21cut08:50:221
47299116,4cyclictest17661-21cut08:50:211
47299116,2cyclictest225-21systemd-journal10:30:001
47299113,5cyclictest24496-21cron12:30:011
47299110,10cyclictest23654-21cut09:05:151
46999119,1cyclictest31666-21sh08:14:590
46999119,1cyclictest2432-21sed09:25:260
46999119,1cyclictest1361-21taskset08:16:410
46999118,2cyclictest31854-21grep09:20:210
46999118,2cyclictest30456-21cron09:20:010
46999118,2cyclictest26383-21grep12:30:250
46999118,2cyclictest21398-21grep09:00:140
46999118,2cyclictest20387-21grep10:05:150
46999118,2cyclictest19491-21chrt07:47:130
46999118,2cyclictest17631-21grep12:15:120
46999118,2cyclictest17005-21chrt12:11:220
46999118,2cyclictest13770-21sleep11:00:010
46999118,2cyclictest13248-21expr12:05:140
46999110,10cyclictest11058-21cat09:45:170
48699108,1cyclictest8327-21expr07:25:143
48699108,1cyclictest3245-21basename07:15:143
48699108,1cyclictest31113-21expr09:20:153
48699108,1cyclictest29586-21aten2_r7power_c10:25:123
48699108,1cyclictest26369-21taskset10:16:283
48699108,1cyclictest2419-21sed07:10:333
48699108,1cyclictest18484-21cat07:45:183
48699108,1cyclictest15518-21cut07:40:143
48699108,1cyclictest1452-21taskset08:17:483
48699107,2cyclictest8618-21expr09:40:163
48699107,2cyclictest6334-21seq10:41:213
48699107,2cyclictest5990-21sed08:25:273
48699107,2cyclictest2986-21cat10:35:183
48699107,2cyclictest2900-21chrt11:43:243
48699107,2cyclictest27265-21cut10:20:133
48699107,2cyclictest23398-21awk10:10:223
48699107,2cyclictest21448-21grep10:05:263
48699107,2cyclictest19204-21cat11:10:123
48699107,2cyclictest17552-21egrep10:00:133
48699105,2cyclictest452-21systemd-logind08:39:033
48699104,2cyclictest225-21systemd-journal09:33:113
47999108,1cyclictest8494-21unixbench-2d10:45:242
47999108,1cyclictest7541-21taskset11:52:022
47999108,1cyclictest26069-21aten2_r7power_p09:10:142
47999108,1cyclictest25210-21cat12:30:142
47999108,1cyclictest182952sleep211:07:172
47999108,1cyclictest15293-21grep11:00:222
47999108,1cyclictest1403-21cut09:25:172
47999108,1cyclictest10426-21cat09:45:142
47999107,2cyclictest8003-21ntp_kernel_err08:30:212
47999107,2cyclictest26242-21awk11:25:122
47999107,2cyclictest22886-21expr07:55:152
47999107,2cyclictest22244-21grep10:10:142
47999106,2cyclictest22136-21grep11:15:182
47999103,3cyclictest225-21systemd-journal10:50:112
47999100,9cyclictest2131-21sensors07:10:302
47299108,1cyclictest63142sleep110:41:071
47299108,1cyclictest5769-21cut11:50:121
47299108,1cyclictest5080-21chrt07:18:161
47299108,1cyclictest304322sleep109:19:461
47299108,1cyclictest25730-21awk08:00:191
47299108,1cyclictest25486-21tail12:30:161
47299108,1cyclictest18776-21apt-get08:55:131
47299108,1cyclictest13479-21cut12:05:161
47299107,2cyclictest6197-21cut07:20:171
47299107,2cyclictest4465-21ntpq11:45:201
47299107,2cyclictest26876-21cut11:25:151
47299107,2cyclictest21504-21taskset10:05:431
47299107,2cyclictest20692-21grep07:50:161
47299107,2cyclictest1574-21cut11:40:151
47299107,2cyclictest10414-21egrep09:45:141
47299106,3cyclictest15973-21cut12:10:161
47299105,2cyclictest225-21systemd-journal08:48:541
47299102,1cyclictest17567-21apt-get12:15:121
47299100,9cyclictest352-21chrt11:37:091
47299100,9cyclictest21101-21cron11:15:001
46999108,1cyclictest8064-21ls09:40:130
46999108,1cyclictest6116-21expr11:50:130
46999108,1cyclictest5711-21tr09:35:140
46999108,1cyclictest5318-21cat09:35:000
46999108,1cyclictest28268-21taskset11:29:380
46999108,1cyclictest253902sleep009:06:060
46999108,1cyclictest218192sleep012:21:100
46999108,1cyclictest15681-21ls07:40:140
46999108,1cyclictest12945-21expr09:50:130
46999107,2cyclictest9820-21sed08:35:150
46999107,2cyclictest5403-21cron11:50:000
46999107,2cyclictest23174-21chrt11:15:270
46999107,2cyclictest20692-21taskset08:57:490
46999107,2cyclictest19712-21head11:10:160
46999107,2cyclictest18664-21uname10:00:210
46999107,2cyclictest16460-21cut11:05:140
46999107,2cyclictest14787-21tail08:45:180
46999107,2cyclictest13040-21sensors_temp08:40:240
46999107,2cyclictest1156-21expr11:40:130
46999101,2cyclictest25984-21apt-get09:10:130
46999100,9cyclictest6917-21basename10:45:120
4869998,1cyclictest29881-21gzip11:30:223
4869998,1cyclictest15624-21expr12:10:143
4869997,1cyclictest7471-21diskstats10:45:143
4869997,1cyclictest6679-21date08:30:003
4869997,1cyclictest5981-21cat11:50:123
4869997,1cyclictest5659-21grep07:20:143
4869994,4cyclictest21945-21seq09:00:173
4869990,9cyclictest278422sleep309:12:013
4729997,1cyclictest10166-21if_err_enp2s010:50:181
4729996,2cyclictest15449-21df09:55:151
4729992,5cyclictest6963-21cut10:45:121
4699997,1cyclictest8886-21chrt08:33:490
4699997,1cyclictest31157-21cat10:25:240
4699997,1cyclictest27784-21fschecks_count08:05:160
4699997,1cyclictest13417-21df_inode07:35:170
4699996,2cyclictest9757-21grep10:50:150
4699996,2cyclictest23252-21cat12:25:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional