You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 16:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri May 08, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2480229380,9sleep20-21swapper/207:07:122
2491429264,9sleep00-21swapper/007:08:390
2282529172,14sleep10-21swapper/107:05:071
2493728666,9sleep30-21swapper/307:08:583
2511199270,3cyclictest0-21swapper/111:44:381
2511399250,24cyclictest0-21swapper/210:17:572
25111992524,1cyclictest0-21swapper/112:13:111
2511199230,21cyclictest0-21swapper/110:33:091
2511799200,2cyclictest0-21swapper/311:48:423
230202200,0chrt0-21swapper/109:31:321
25117991918,1cyclictest0-21swapper/311:08:473
25117991917,1cyclictest39-21ksoftirqd/311:14:123
2511799190,0cyclictest0-21swapper/310:25:093
2511399190,1cyclictest0-21swapper/209:47:052
2511199190,4cyclictest0-21swapper/111:45:431
2511199190,15cyclictest0-21swapper/109:27:401
25108991913,2cyclictest0-21swapper/012:39:010
2511799180,18cyclictest30739-21kworker/3:110:32:123
2511399180,1cyclictest0-21swapper/212:04:452
2511199189,2cyclictest0-21swapper/112:30:561
25111991814,4cyclictest38150irq/131-snd_hda10:56:301
2511199180,16cyclictest0-21swapper/112:07:271
2510899181,11cyclictest1120-21ssh11:15:130
25117991716,1cyclictest30739-21kworker/3:110:42:523
25117991716,1cyclictest30739-21kworker/3:110:35:123
25117991710,2cyclictest225-21systemd-journal12:25:523
2511399170,17cyclictest0-21swapper/212:17:492
2511199172,1cyclictest141rcu_preempt12:38:111
2511199170,1cyclictest0-21swapper/112:00:241
2511199170,14cyclictest0-21swapper/110:53:391
25117991611,3cyclictest39-21ksoftirqd/309:41:183
25111991615,1cyclictest0-21swapper/109:58:311
25111991612,3cyclictest0-21swapper/109:21:171
25111991611,3cyclictest16070-21cron11:05:001
25117991514,1cyclictest30739-21kworker/3:109:27:123
25117991514,1cyclictest0-21swapper/311:01:403
25117991513,0cyclictest0-21swapper/311:50:133
2511399153,7cyclictest19463-21/usr/sbin/munin09:30:132
25113991513,1cyclictest0-21swapper/209:16:122
2511199159,5cyclictest29288-21wc07:15:241
25111991511,2cyclictest6693-21wc10:00:241
2511199150,15cyclictest0-21swapper/112:29:161
2510899159,4cyclictest4548-21ssh11:55:190
25108991512,3cyclictest13-21ksoftirqd/011:02:310
2510899150,8cyclictest0-21swapper/008:00:110
2510899150,2cyclictest0-21swapper/009:49:250
2510899150,11cyclictest0-21swapper/009:23:570
250452150,3sleep10-21swapper/111:29:301
2511799146,8cyclictest30739-21kworker/3:110:56:183
25117991413,1cyclictest30739-21kworker/3:112:04:323
25117991413,1cyclictest30739-21kworker/3:110:20:313
25117991413,1cyclictest30739-21kworker/3:109:51:123
25117991413,1cyclictest30739-21kworker/3:109:38:313
25117991413,1cyclictest30739-21kworker/3:109:13:223
25117991410,3cyclictest3466-21wc08:40:003
25117991410,3cyclictest2353-21grep11:15:193
25117991410,3cyclictest20925-21grep09:30:213
2511799140,2cyclictest0-21swapper/310:02:323
2511799140,12cyclictest0-21swapper/312:38:063
2511399148,6cyclictest1132-21munin-node10:57:192
2511399148,4cyclictest6481-21df10:20:142
2511399143,9cyclictest0-21swapper/210:31:382
2511399143,8cyclictest28365-21wc11:50:192
2511399142,7cyclictest3654-21/usr/sbin/munin11:35:222
25113991413,1cyclictest0-21swapper/212:14:552
2511399140,1cyclictest0-21swapper/209:10:252
2511399140,13cyclictest0-21swapper/211:07:422
2511399140,12cyclictest0-21swapper/212:20:332
25111991410,3cyclictest28049-21wc08:20:251
25111991410,3cyclictest13901-21wc10:25:001
25111991410,2cyclictest0-21swapper/110:45:591
2511199140,1cyclictest17662-21diskmemload11:08:271
2511199140,1cyclictest0-21swapper/109:13:211
2511199140,13cyclictest0-21swapper/110:27:211
2510899141,10cyclictest0-21swapper/007:30:130
2510899140,8cyclictest9999-21awk11:40:000
2510899140,3cyclictest0-21swapper/011:29:240
2510899140,2cyclictest0-21swapper/010:31:380
2511799134,1cyclictest0-21swapper/312:34:323
2511799133,6cyclictest273-21systemd-udevd09:22:043
25117991312,1cyclictest30739-21kworker/3:112:11:023
25117991312,1cyclictest30739-21kworker/3:111:38:523
25113991312,1cyclictest0-21swapper/211:32:042
25113991311,1cyclictest0-21swapper/210:45:542
25113991310,2cyclictest3379-21grep07:30:162
25113991310,2cyclictest18040-21sort11:25:142
2511399130,2cyclictest0-21swapper/209:50:122
2511399130,1cyclictest0-21swapper/211:41:322
2511399130,1cyclictest0-21swapper/209:21:342
2511199139,3cyclictest24422-21tr08:15:151
2511199138,3cyclictest7127-21cron07:39:591
25111991313,0cyclictest0-21swapper/110:16:271
25111991312,1cyclictest0-21swapper/109:53:251
25111991310,2cyclictest482-21sed08:30:251
25111991310,2cyclictest3774-21wc11:35:241
2511199130,12cyclictest2408-21taskset11:15:201
2511199130,12cyclictest0-21swapper/111:30:091
2510899138,2cyclictest23875-21systemd-cgroups08:14:560
2510899136,6cyclictest3650-21perf12:15:000
2510899136,6cyclictest18824-21apt-get09:30:000
2510899135,7cyclictest26320-21grep08:20:000
25108991312,1cyclictest25102-21cyclictest11:43:570
25108991310,2cyclictest4656-21head09:40:160
2510899130,9cyclictest26023-21taskset11:30:040
2510899130,5cyclictest0-21swapper/010:08:250
2511799129,2cyclictest5767-21sh08:45:003
2511799129,2cyclictest31280-21cat10:15:183
2511799129,2cyclictest28286-21sed08:20:243
2511799129,2cyclictest25825-21grep08:15:273
2511799129,2cyclictest24785-21head10:50:163
2511799129,2cyclictest16368-21cut10:45:153
2511799129,2cyclictest12785-21cut07:50:153
2511799121,7cyclictest10503-21cron08:55:003
2511799120,11cyclictest14828-21grep07:55:143
2511799120,11cyclictest10540-21grep07:45:153
2511399129,2cyclictest3574-21expr11:55:132
2511399129,2cyclictest3388-21tr09:40:002
2511399129,2cyclictest31955-21grep07:20:252
2511399129,2cyclictest30498-21grep12:30:202
2511399129,2cyclictest16248-21grep10:25:252
2511399129,2cyclictest15159-21tune2fs07:55:162
2511399129,2cyclictest10883-21grep11:20:212
2511399129,2cyclictest10020-21grep08:50:242
2511399126,5cyclictest7619-21df10:40:152
2511399123,7cyclictest0-21swapper/211:03:472
2511399123,4cyclictest0-21swapper/208:45:132
2511399123,4cyclictest0-21swapper/208:45:132
2511399122,4cyclictest0-21swapper/208:00:122
2511399122,1cyclictest0-21swapper/211:46:442
25113991210,1cyclictest26973-21grep07:10:242
2511399120,2cyclictest0-21swapper/210:03:392
2511399120,12cyclictest0-21swapper/212:37:122
2511399120,12cyclictest0-21swapper/211:11:522
2511399120,10cyclictest0-21swapper/210:14:112
2511199129,2cyclictest674-21grep10:35:241
2511199129,2cyclictest4368-21grep08:40:171
2511199129,2cyclictest3855-21cut11:55:141
2511199129,2cyclictest28271-21wc09:15:261
2511199129,2cyclictest17222-21tail09:05:261
2511199129,2cyclictest15287-21grep07:55:161
2511199129,2cyclictest14220-21sed10:05:171
2511199129,2cyclictest12904-21perf09:00:001
2511199129,2cyclictest12718-21tr09:45:141
2511199129,2cyclictest12633-21cut12:20:121
2511199129,2cyclictest11697-21turbostat09:45:001
2511199128,3cyclictest4765-21dump-pmu-power07:35:001
2511199126,5cyclictest38150irq/131-snd_hda07:20:141
2511199126,2cyclictest9466-21ssh11:20:121
25111991212,0cyclictest0-21swapper/111:10:101
2510899129,2cyclictest30569-21tr07:20:130
2510899129,2cyclictest28492-21chrt07:15:160
2510899128,3cyclictest17582-21ssh12:03:390
2510899127,4cyclictest28405-21rm11:50:190
2510899122,8cyclictest1132-21munin-node11:05:100
2510899122,7cyclictest22707-21ssh12:25:260
2510899121,2cyclictest0-21swapper/012:07:270
2510899121,10cyclictest17662-21diskmemload09:12:010
25108991210,1cyclictest0-21swapper/010:16:070
2510899120,11cyclictest6749-21head07:35:260
2510899120,11cyclictest32221-21sh10:55:080
2510899120,11cyclictest23138-21grep10:10:210
2510899120,11cyclictest15171-21cat10:45:010
2511799118,2cyclictest8168-21grep07:40:163
2511799118,2cyclictest6372-21ssh12:15:373
2511799118,2cyclictest5495-21cat07:35:153
2511799118,2cyclictest4506-21ls11:55:203
2511799118,2cyclictest31781-21expr08:30:143
2511799118,2cyclictest30044-21ssh09:16:353
2511799118,2cyclictest29149-21grep09:55:123
2511799118,2cyclictest26690-21expr11:30:143
2511799118,2cyclictest26661-21expr07:10:203
2511799118,2cyclictest2619-21taskset07:30:053
2511799118,2cyclictest24803-21ssh11:29:183
2511799118,2cyclictest20471-21ssh12:05:153
2511799118,2cyclictest20034-21head08:05:173
2511799118,2cyclictest19443-21ssh12:24:243
2511799118,2cyclictest17401-21sh09:47:473
2511799118,2cyclictest17355-21ssh10:06:593
2511799117,3cyclictest31084-21grep07:20:153
25117991110,1cyclictest0-21swapper/307:19:583
2511799110,1cyclictest0-21swapper/310:11:093
2511799110,10cyclictest10513-21ssh11:40:133
2511399119,1cyclictest8207-21grep07:40:162
2511399119,1cyclictest4725-21grep09:40:162
2511399119,1cyclictest4564-21taskset10:00:002
2511399119,1cyclictest30048-21cut08:25:192
2511399118,2cyclictest549-21sed08:30:252
2511399118,2cyclictest30743-21ssh10:54:092
2511399118,2cyclictest22666-21grep12:25:262
2511399118,2cyclictest14939-21ssh09:26:572
2511399118,2cyclictest14521-21ssh10:05:192
2511399118,2cyclictest1388-21tr11:15:142
2511399118,2cyclictest12260-21cat08:55:252
2511399117,3cyclictest9797-21apache_volume07:45:122
2511399111,4cyclictest582-21cstates07:25:142
25113991110,1cyclictest0-21swapper/210:35:242
2511399110,1cyclictest0-21swapper/212:05:122
2511399110,10cyclictest20974-21grep08:05:262
2511199119,1cyclictest22484-21tail10:10:151
2511199118,2cyclictest9094-21grep07:40:261
2511199118,2cyclictest6307-21tr08:45:141
2511199118,2cyclictest6307-21tr08:45:131
2511199118,2cyclictest29504-21ssh11:50:381
2511199118,2cyclictest28157-21egrep09:35:141
2511199118,2cyclictest23805-21seq08:14:341
2511199118,2cyclictest14571-21ssh10:44:461
2510899119,1cyclictest32372-21wc10:35:190
2510899119,1cyclictest19669-21grep11:45:180
2510899119,1cyclictest19552-21tr08:05:140
2510899119,1cyclictest17491-21grep10:45:250
2510899118,2cyclictest7094-21grep08:45:190
2510899118,2cyclictest7094-21grep08:45:190
2510899118,2cyclictest6621-21ssh12:15:490
2510899118,2cyclictest5267-21wc10:00:130
2510899118,2cyclictest5205-21wc08:40:260
2510899118,2cyclictest29108-21wc09:35:190
2510899118,2cyclictest28109-21grep08:20:250
2510899118,2cyclictest26441-21head07:10:180
2510899118,2cyclictest24976-21sh09:33:040
2510899118,2cyclictest23531-21sh09:50:590
2510899118,2cyclictest16839-21ssh10:25:510
2510899118,2cyclictest12342-21sed08:55:250
2510899115,5cyclictest3385-21sendmail08:39:590
2510899115,2cyclictest225-21systemd-journal08:55:000
2510899114,4cyclictest10410-21idleruntime11:20:170
2510899110,10cyclictest27191-21users11:10:260
2511799108,1cyclictest6344-21cstates08:45:153
2511799108,1cyclictest6344-21cstates08:45:153
2511799108,1cyclictest386-21egrep07:25:143
2511799108,1cyclictest22000-21cat08:10:143
2511799108,1cyclictest11087-21cat08:55:143
2511799108,1cyclictest10785-21ssh11:20:203
2511799107,2cyclictest17197-21grep08:00:143
2511799107,2cyclictest13517-21cat09:00:123
2511399108,2cyclictest6902-21taskset07:37:072
2511399108,2cyclictest13215-21basename09:00:122
2511399108,1cyclictest1219-21seq08:35:012
2511399107,2cyclictest28675-21cat07:15:172
2511399107,2cyclictest12919-21head07:50:152
2511399101,4cyclictest23627-21sshd08:12:142
2511199108,1cyclictest26648-21grep07:10:191
2511199108,1cyclictest200222sleep108:05:171
2511199108,1cyclictest10699-21cat07:45:181
2511199107,2cyclictest3966-21ssh12:15:061
2511199107,2cyclictest17673-21tail08:00:161
2511199106,2cyclictest1047-21cut07:25:171
2511199105,2cyclictest14829-21sed09:00:251
2510899108,1cyclictest8147-21expr07:40:160
2510899108,1cyclictest5860-21ssh10:20:080
2510899108,1cyclictest30835-21sed12:30:250
2510899108,1cyclictest29107-21rm09:55:110
2510899108,1cyclictest24150-21grep10:50:140
2510899108,1cyclictest22582sleep007:28:260
2510899108,1cyclictest149362sleep009:00:420
2510899108,1cyclictest12364-21sh12:20:070
2510899108,1cyclictest10769-21sed07:45:180
2510899107,2cyclictest29182-21tr08:25:140
2510899107,2cyclictest12304-21grep07:50:130
2510899105,4cyclictest13-21ksoftirqd/007:55:110
2510899101,6cyclictest28173-21ssh09:15:240
2510899100,2cyclictest0-21swapper/008:30:120
251179997,1cyclictest15855-21cat09:05:143
251179996,2cyclictest30167-21head08:25:213
251139998,1cyclictest3866-21grep08:40:142
251139998,1cyclictest27694-21ntp_kernel_pll_08:20:192
251139998,1cyclictest25959-21chrt08:16:042
251139998,1cyclictest16362-21grep09:05:172
251119997,1cyclictest14039-21chrt07:52:071
251119997,1cyclictest10000-21sed08:50:251
251119994,4cyclictest38150irq/131-snd_hda08:25:191
251089997,1cyclictest15696-21egrep09:05:140
251119986,1cyclictest1589-21expr08:35:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional