You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 00:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed May 13, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3184629582,9sleep00-21swapper/007:09:350
3158029481,9sleep20-21swapper/207:06:072
3177328666,16sleep10-21swapper/107:08:391
3172028472,8sleep30-21swapper/307:07:573
3196999280,23cyclictest0-21swapper/211:43:162
3196999270,27cyclictest0-21swapper/212:34:372
256402270,2chrt25638-21cron09:29:592
31965992624,1cyclictest0-21swapper/110:21:371
31971992322,1cyclictest0-21swapper/309:32:593
3197199230,1cyclictest0-21swapper/309:38:313
3196999231,2cyclictest21481-21wc10:05:262
3196999230,23cyclictest0-21swapper/211:30:412
3196999230,23cyclictest0-21swapper/211:30:402
3196999230,1cyclictest0-21swapper/208:10:132
3196999220,22cyclictest0-21swapper/210:21:372
3196599222,12cyclictest1457-21grep12:10:151
3196599221,12cyclictest24773-21diskmemload09:15:061
31963992221,1cyclictest0-21swapper/011:47:380
31969992120,1cyclictest0-21swapper/211:52:392
3196999210,21cyclictest0-21swapper/209:18:072
31963992112,2cyclictest0-21swapper/012:05:010
31971992019,1cyclictest0-21swapper/310:17:593
3197199200,1cyclictest0-21swapper/311:01:183
3196999200,20cyclictest0-21swapper/211:46:532
3196999200,20cyclictest0-21swapper/211:26:082
3196999200,20cyclictest0-21swapper/210:12:342
3196999200,20cyclictest0-21swapper/209:46:192
3196999200,1cyclictest0-21swapper/212:24:372
3196999200,1cyclictest0-21swapper/212:09:272
3196599203,13cyclictest24773-21diskmemload12:27:161
3196599200,20cyclictest0-21swapper/109:48:491
3196399206,7cyclictest17346-21munin-run09:25:000
3197199195,12cyclictest0-21swapper/310:30:143
3196999190,19cyclictest0-21swapper/210:03:252
3196999190,17cyclictest0-21swapper/209:12:112
31965991915,1cyclictest0-21swapper/112:04:481
3196399191,17cyclictest0-21swapper/011:30:140
3196399191,17cyclictest0-21swapper/011:30:140
3196399190,2cyclictest0-21swapper/012:25:320
243712190,2sleep110646-21kworker/1:110:07:341
31971991816,1cyclictest13368-21cstates10:40:143
3196999180,1cyclictest141rcu_preempt12:37:112
3196999180,18cyclictest0-21swapper/210:18:312
3196599180,18cyclictest0-21swapper/109:22:271
3196399186,6cyclictest5139-21munin-run10:55:000
224352180,0chrt0-21swapper/311:24:453
31969991714,2cyclictest32-21ksoftirqd/211:35:152
3196599172,8cyclictest0-21swapper/108:16:541
3196399179,6cyclictest13-21ksoftirqd/012:30:170
31963991716,1cyclictest0-21swapper/011:00:120
69032160,0sleep00-21swapper/012:13:200
31971991616,0cyclictest0-21swapper/311:52:113
3197199160,1cyclictest0-21swapper/310:28:583
3197199160,15cyclictest0-21swapper/311:46:153
3196999161,1cyclictest141rcu_preempt09:39:082
3196999161,14cyclictest0-21swapper/210:30:142
3196999160,16cyclictest0-21swapper/211:59:472
3196999160,16cyclictest0-21swapper/209:43:382
31965991615,1cyclictest0-21swapper/112:30:321
3196399160,1cyclictest0-21swapper/010:18:020
3197199153,5cyclictest356-21runrttasks09:14:473
3197199152,1cyclictest0-21swapper/308:55:133
3196999159,3cyclictest1370-21apache212:27:202
3196999155,3cyclictest0-21swapper/210:55:002
3196599158,6cyclictest11710-21grep07:35:141
3196599152,10cyclictest537-21snmpd12:09:481
31965991514,1cyclictest0-21swapper/111:23:431
3196399156,3cyclictest0-21swapper/007:40:200
3196399150,3cyclictest0-21swapper/011:58:010
3196399150,15cyclictest0-21swapper/011:38:400
31971991410,4cyclictest13490-21apache210:05:533
31971991410,2cyclictest5772-21grep08:30:143
3197199140,14cyclictest0-21swapper/312:00:293
3197199140,14cyclictest0-21swapper/310:10:293
3196999148,2cyclictest141rcu_preempt08:10:012
31969991410,2cyclictest32-21ksoftirqd/208:30:142
3196999140,14cyclictest0-21swapper/211:06:062
3196599146,5cyclictest505-21polkitd12:20:221
3196599141,2cyclictest0-21swapper/109:25:111
31965991412,1cyclictest0-21swapper/107:55:141
31965991411,2cyclictest1004-21rm11:50:171
3196599140,1cyclictest0-21swapper/109:56:311
3196599140,14cyclictest0-21swapper/110:58:171
3196599140,12cyclictest19475-21sort07:50:171
3196399149,4cyclictest996-21sh09:15:050
3196399146,7cyclictest12562-21sendmail-msp10:40:000
3197199139,3cyclictest7497-21grep10:55:253
3197199139,3cyclictest31054-21grep08:15:153
3197199139,3cyclictest2385-21grep07:15:143
3197199131,8cyclictest27326-21cpuspeed_turbos09:50:133
31971991313,0cyclictest0-21swapper/312:30:003
31971991312,0cyclictest0-21swapper/312:15:123
31971991310,2cyclictest6848-21cat07:25:003
31971991310,2cyclictest2462-21grep12:10:203
31971991310,2cyclictest19615-21head07:50:183
31971991310,2cyclictest19058-21grep09:45:133
31971991310,2cyclictest17581-21wc11:40:253
31971991310,2cyclictest14424-21latency_hist11:20:003
3196999138,3cyclictest32-21ksoftirqd/210:26:582
3196999131,6cyclictest13935-21ssh11:19:382
31969991310,2cyclictest21753-21wc09:00:252
3196999130,13cyclictest0-21swapper/210:35:362
3196999130,12cyclictest18040-21wc12:00:182
3196999130,12cyclictest0-21swapper/212:13:172
3196999130,0cyclictest141rcu_preempt12:17:062
3196599139,3cyclictest5309-21sed07:20:171
3196599139,3cyclictest31878-21tr11:30:151
3196599139,3cyclictest31878-21tr11:30:151
3196599139,3cyclictest25050-21wc11:25:251
3196599136,6cyclictest537-21snmpd09:54:201
3196599132,3cyclictest273-21systemd-udevd11:12:041
31965991310,2cyclictest24953-21cat09:10:001
31965991310,2cyclictest11547-21head12:35:161
31963991312,1cyclictest0-21swapper/008:45:130
31963991311,2cyclictest13-21ksoftirqd/009:26:440
31963991310,2cyclictest7890-21cron11:55:010
31963991310,2cyclictest7362-21grep11:15:170
31963991310,2cyclictest3639-21ssh09:35:240
31963991310,2cyclictest23877-21tr09:05:180
3196399130,12cyclictest27147-21grep09:10:260
3197199129,2cyclictest8690-21cut11:55:143
3197199129,2cyclictest26287-21tr08:05:143
3197199129,2cyclictest23592-21wc11:25:143
3197199129,2cyclictest21873-21sort07:55:163
3197199129,2cyclictest12156-21sed08:40:263
3197199128,4cyclictest39-21ksoftirqd/310:52:133
3197199128,3cyclictest3263-21grep09:55:153
3197199128,3cyclictest13503-21apache209:25:293
3197199128,3cyclictest10358-21cat07:30:203
3197199126,2cyclictest225-21systemd-journal12:09:593
3197199120,11cyclictest0-21swapper/310:35:033
3196999129,2cyclictest7170-21tr07:25:152
3196999129,2cyclictest5230-21cut07:20:172
3196999129,2cyclictest2440-21tr07:15:132
3196999129,2cyclictest22922-21grep09:05:122
3196999129,2cyclictest14153-21basename11:00:142
3196999129,2cyclictest13661-21grep08:45:162
3196999129,2cyclictest10796-21awk10:00:012
31969991210,1cyclictest311rcuc/208:40:002
31969991210,1cyclictest0-21swapper/210:44:012
3196999120,11cyclictest15256-21tr11:20:152
3196599129,2cyclictest8352-21ssh11:55:101
3196599129,2cyclictest3855-21tail08:25:161
3196599129,2cyclictest21399-21apache_accesses10:45:121
3196599129,2cyclictest17388-21cron08:54:591
3196599129,2cyclictest12797-21wc10:00:211
3196599129,2cyclictest12592-21ls08:45:001
3196599128,4cyclictest26284-21ssh11:06:571
3196599128,3cyclictest30509-21dump-pmu-power08:15:011
3196599128,3cyclictest20031-21sh11:42:081
3196599128,3cyclictest14398-21tr07:40:141
3196599128,3cyclictest11240-21grep09:40:171
3196599128,3cyclictest10004-21run-parts08:40:001
3196599127,4cyclictest32191-21ssh09:33:371
3196599122,6cyclictest225-21systemd-journal11:49:441
3196599122,6cyclictest2088-21ssh09:35:141
3196599121,9cyclictest31224-21ssh10:11:241
3196599120,11cyclictest0-21swapper/110:44:281
3196399129,2cyclictest5662-21sh10:55:130
3196399129,2cyclictest29589-21grep10:30:160
3196399129,2cyclictest28601-21ssh10:49:310
3196399129,2cyclictest21923-21sed09:00:260
3196399129,2cyclictest19756-21cut10:05:130
3196399129,2cyclictest11964-21cat08:40:250
3196399121,7cyclictest24773-21diskmemload12:38:330
31963991211,1cyclictest0-21swapper/011:28:480
3196399120,1cyclictest0-21swapper/011:23:220
3197199118,2cyclictest8806-21ntpq11:35:223
3197199118,2cyclictest8196-21cat08:35:133
3197199118,2cyclictest7714-21tail07:25:163
3197199118,2cyclictest31551-21ssh11:30:123
3197199118,2cyclictest31551-21ssh11:30:123
3197199118,2cyclictest31007-21cut11:10:173
3197199118,2cyclictest30082-21sed08:10:263
3197199118,2cyclictest28398-21ssh11:08:373
3197199118,2cyclictest2687-21fgrep09:15:233
3197199118,2cyclictest26074-21ssh12:25:053
3197199118,2cyclictest22785-21cut09:05:113
3197199118,2cyclictest21021-21ssh12:21:173
3197199118,2cyclictest17907-21swap07:45:233
3197199118,2cyclictest16772-21ssh10:22:173
3197199118,2cyclictest12802-21ntp_kernel_pll_07:35:213
3197199118,2cyclictest12781-21sh09:40:453
3197199118,2cyclictest12144-21sed10:00:173
3197199118,2cyclictest12121-21ssh09:20:563
3197199118,2cyclictest11098-21cat12:35:143
3197199116,2cyclictest24773-21diskmemload10:45:133
3197199113,4cyclictest225-21systemd-journal07:15:003
3196999118,2cyclictest9576-21tr07:30:162
3196999118,2cyclictest7578-21head10:55:252
3196999118,2cyclictest3211-21basename08:25:142
3196999118,2cyclictest31502-21grep08:15:162
3196999118,2cyclictest30982-21tr11:10:162
3196999118,2cyclictest2866-21turbostat.cron08:25:012
3196999118,2cyclictest21732-21tr10:45:142
3196999118,2cyclictest21525-21tr07:55:142
3196999118,2cyclictest18394-21grep08:55:162
3196999118,2cyclictest18109-21sed07:45:242
3196999118,2cyclictest16672-21cut08:50:232
3196999118,2cyclictest11702-21basename07:35:142
3196999118,2cyclictest11556-21ssh09:20:302
3196999116,2cyclictest225-21systemd-journal07:15:002
3196999110,3cyclictest32-21ksoftirqd/208:04:432
3196599118,2cyclictest9251-21rm10:37:261
3196599118,2cyclictest7342-21ssh11:35:071
3196599118,2cyclictest6998-21grep07:25:141
3196599118,2cyclictest6601-21grep08:30:191
3196599118,2cyclictest3691-21sh10:15:061
3196599118,2cyclictest32196-21sh10:50:481
3196599118,2cyclictest30575-21awk10:30:251
3196599118,2cyclictest21058-21tail09:00:171
3196599118,2cyclictest17947-21tr08:55:141
3196599118,2cyclictest14934-21hwlatdetect11:00:171
3196599118,2cyclictest10229-21fschecks_time12:15:151
3196599117,3cyclictest6612-21sh11:15:131
3196599112,4cyclictest27823-21taskset09:10:521
3196599110,1cyclictest0-21swapper/110:26:151
3196399119,2cyclictest22585-21grep10:25:240
3196399119,1cyclictest3290-21cut09:55:150
3196399119,1cyclictest22444-21sed07:55:210
3196399119,1cyclictest20069-21ls07:50:220
3196399119,1cyclictest11400-21cron07:34:590
3196399118,2cyclictest9391-21ssh12:15:070
3196399118,2cyclictest8307-21expr08:35:130
3196399118,2cyclictest28342-21cut08:10:000
3196399118,2cyclictest26438-21sh11:07:040
3196399118,2cyclictest19772-21sh09:00:000
3196399118,2cyclictest18965-21ssh09:45:140
3196399118,2cyclictest13325-21cut10:40:140
3196399118,2cyclictest13065-21uniq10:20:160
3196399114,6cyclictest24773-21diskmemload12:05:130
3196399113,6cyclictest10390-21apt-get09:40:130
3196399110,11cyclictest0-21swapper/010:04:500
3196399110,10cyclictest7559-21taskset08:34:320
3197199108,1cyclictest17368-21taskset08:54:453
3197199108,1cyclictest1661-21cat08:20:193
3197199108,1cyclictest14922-21sh07:40:173
3197199107,2cyclictest24063-21cat08:00:143
3197199107,2cyclictest20639-21expr09:00:153
3197199107,2cyclictest13674-21head08:45:163
3196999108,1cyclictest26143-21ssh09:30:122
3196999108,1cyclictest160852sleep207:43:142
3196999107,2cyclictest26827-21ssh09:50:042
3196999107,2cyclictest20308-21sed07:50:252
3196599108,1cyclictest17980-21grep07:45:241
3196599108,1cyclictest149532sleep108:49:161
3196599107,2cyclictest9851-21grep07:30:161
3196599107,2cyclictest32224-21cut07:10:151
3196599107,2cyclictest27441-21expr08:05:241
3196599106,3cyclictest2824-21grep07:15:161
3196599101,4cyclictest273-21systemd-udevd08:04:041
3196399108,2cyclictest4654-21rm11:13:550
3196399108,1cyclictest3309-21ntp_states07:15:210
3196399108,1cyclictest26070-21ssh09:30:090
3196399108,1cyclictest16006-21head08:50:160
3196399108,1cyclictest15728-21ssh11:40:070
3196399107,2cyclictest7268-21cat07:25:150
3196399107,2cyclictest726-21ssh09:53:390
3196399107,2cyclictest1887-21idleruntime-cro07:15:000
3196399107,2cyclictest18263-21tr12:20:140
3196399105,2cyclictest225-21systemd-journal08:15:010
3196399100,9cyclictest11683-21dpkg07:35:140
3196399100,10cyclictest30669-21rm10:10:590
319719997,1cyclictest3795-21tune2fs08:25:153
319699997,1cyclictest98592sleep208:38:352
319659996,2cyclictest883-21cut08:20:151
319639997,1cyclictest5319-21head07:20:170
319639997,1cyclictest30809-21tr08:15:140
319639996,2cyclictest3224-21basename08:25:140
319639996,2cyclictest16936-21expr07:45:130
319639990,9cyclictest1460-21grep08:20:180
319639987,1cyclictest25045-21smartctl08:00:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional