You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-24 - 13:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Dec 24, 2025 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
694629363,9sleep00-21swapper/019:09:460
669329167,19sleep20-21swapper/219:06:312
669928371,8sleep30-21swapper/319:06:363
687028164,13sleep10-21swapper/119:08:471
705799225,8cyclictest0-21swapper/223:50:132
705499210,18cyclictest24645-21forks19:45:221
7054991815,2cyclictest0-21swapper/120:55:181
705299186,6cyclictest19887-21sendmail22:59:590
705799171,15cyclictest28469-21if_enp2s021:00:232
7054991716,1cyclictest0-21swapper/120:25:161
705299168,7cyclictest843-21munin-run22:19:590
705299164,3cyclictest225-21systemd-journal00:10:000
705299162,7cyclictest18970-21munin-run19:35:010
7061991511,3cyclictest22007-21wc20:45:263
7061991511,2cyclictest14270-21cut21:40:133
705299152,6cyclictest27360-21sendmail-msp21:00:000
7052991510,3cyclictest17738-21cron20:40:000
7061991411,2cyclictest23786-21awk00:15:003
7061991410,3cyclictest31360-21grep23:20:263
7057991411,2cyclictest16284-21cron23:59:592
7057991411,2cyclictest15859-21grep21:40:262
7057991410,3cyclictest10628-21grep20:25:002
705499140,12cyclictest0-21swapper/120:30:131
705299149,3cyclictest7200-21grep23:40:160
705299149,3cyclictest383-21tail20:00:300
7052991410,3cyclictest8238-21latency_hist20:20:000
7052991410,3cyclictest6792-21basename23:40:010
7052991410,3cyclictest616-21grep00:30:270
7052991410,2cyclictest2964-21grep00:35:250
706199139,3cyclictest30214-21wc19:55:263
706199139,3cyclictest2954-21sed00:35:253
706199139,3cyclictest17046-21grep22:50:253
7061991312,1cyclictest0-21swapper/320:35:153
7061991310,2cyclictest9428-21wc19:15:003
7061991310,2cyclictest672-21sed00:30:273
7061991310,2cyclictest26636-21tr22:05:173
7061991310,2cyclictest13430-21grep23:50:263
7061991310,2cyclictest11362-21grep22:40:183
705799132,9cyclictest11909-21aten2_r7power_e21:35:142
7057991310,2cyclictest32152-21cat20:00:222
7057991310,2cyclictest25599-21cut23:10:182
7057991310,2cyclictest25599-21cut23:10:182
7057991310,2cyclictest13321-21wc19:20:212
7057991310,2cyclictest10623-21cat19:15:202
7057991310,2cyclictest10623-21cat19:15:192
705499137,3cyclictest0-21swapper/100:00:121
705499134,4cyclictest225-21systemd-journal00:25:171
705499131,9cyclictest38150irq/131-snd_hda23:35:151
7054991311,1cyclictest0-21swapper/123:40:211
7054991310,2cyclictest28073-21grep19:50:271
7054991310,2cyclictest26646-21tail23:10:281
7054991310,2cyclictest26646-21tail23:10:281
7054991310,2cyclictest19414-21grep22:55:261
7054991310,2cyclictest15854-21cat19:25:291
7054991310,2cyclictest15773-21grep20:35:151
7054991310,2cyclictest13742-21grep22:45:181
7054991310,2cyclictest13664-21grep19:20:251
7054991310,2cyclictest10620-21sh20:25:001
705299139,3cyclictest30966-21tr23:20:220
705299139,3cyclictest21730-21cut00:10:140
705299139,3cyclictest13319-21cat21:35:260
705299139,2cyclictest9607-21cut21:30:160
705299139,2cyclictest20739-21grep20:45:130
705299131,6cyclictest11564-21basename23:50:000
7052991310,2cyclictest9232-21wc21:30:010
7052991310,2cyclictest24952-21apt20:55:000
7052991310,2cyclictest10618-21cron20:25:000
706199129,2cyclictest7771-21grep23:40:173
706199129,2cyclictest340-21grep22:15:273
706199129,2cyclictest15200-21sed19:25:193
706199129,2cyclictest14963-21grep23:55:193
706199129,2cyclictest12827-21grep19:20:173
706199129,2cyclictest10601-21tr21:30:213
706199128,3cyclictest32086-21ls21:10:003
706199128,3cyclictest24765-21head19:45:233
706199128,2cyclictest3439-21cat20:10:003
706199126,2cyclictest29480-21basename23:20:003
706199123,5cyclictest225-21systemd-journal00:08:573
7061991211,1cyclictest0-21swapper/321:55:003
705799129,2cyclictest5755-21cat22:29:592
705799129,2cyclictest31493-21grep00:30:142
705799129,2cyclictest31187-21cat23:20:252
705799129,2cyclictest30898-21cron20:00:002
705799129,2cyclictest2782-21tail23:30:182
705799129,2cyclictest25110-21grep22:00:202
705799129,2cyclictest21011-21grep20:45:152
705799129,2cyclictest11660-21grep20:25:182
705799128,2cyclictest27457-21cat22:05:222
705499129,2cyclictest4553-21cat21:20:011
705499129,2cyclictest3814-21grep20:10:181
705499129,2cyclictest31628-21sed21:05:271
705499129,2cyclictest29560-21cut22:10:201
705499129,2cyclictest28682-21cat22:10:001
705499129,2cyclictest24934-21cron20:55:001
705499129,2cyclictest23795-21turbostat.cron19:45:011
705499129,2cyclictest22198-21sed21:55:181
705499129,2cyclictest1837-21wc23:30:011
705499129,2cyclictest17837-21grep21:45:211
705499129,2cyclictest12561-21head23:50:171
705499129,2cyclictest10266-21sort19:15:181
705499129,2cyclictest10266-21sort19:15:171
705499128,3cyclictest992-21cat22:20:001
705499128,2cyclictest4836-21basename21:20:141
705499127,4cyclictest20918-21chrt00:07:371
705499127,3cyclictest5716-21cat22:30:001
705299129,2cyclictest5818-21head21:20:180
705299129,2cyclictest5199-21grep20:10:300
705299129,2cyclictest3920-21grep21:15:260
705299129,2cyclictest31236-21wc21:05:220
705299129,2cyclictest30903-21sh20:00:000
705299129,2cyclictest29531-21tail00:25:170
705299129,2cyclictest28374-21cat23:15:190
705299129,2cyclictest2801-21head23:30:170
705299129,2cyclictest26660-21cut19:50:160
705299129,2cyclictest26539-21cut22:05:170
705299129,2cyclictest21399-21sendmail19:40:010
705299129,2cyclictest16716-21grep00:00:160
705299129,2cyclictest16527-21cat21:45:010
705299129,2cyclictest16165-21sed22:50:170
705299129,2cyclictest13091-21cut22:45:140
705299129,2cyclictest12517-21grep23:50:170
705299128,3cyclictest29006-21tr22:10:180
705299128,3cyclictest14199-21wc19:25:000
705299122,8cyclictest11807-21idleruntime-cro19:20:000
705299122,8cyclictest11807-21idleruntime-cro19:20:000
706199118,2cyclictest9135-21grep20:20:193
706199118,2cyclictest5485-21taskset20:13:173
706199118,2cyclictest5307-21fschecks_time23:35:183
706199118,2cyclictest433-21grep23:25:153
706199118,2cyclictest3464-21chrt22:25:103
706199118,2cyclictest3079-21grep21:15:173
706199118,2cyclictest27764-21run-parts00:20:223
706199118,2cyclictest27535-21date21:00:003
706199118,2cyclictest2528-21cat23:30:163
706199118,2cyclictest24284-21head23:05:283
706199118,2cyclictest21546-21fgrep23:00:263
706199118,2cyclictest1864-21expr22:20:173
706199118,2cyclictest17230-21grep21:45:173
706199118,2cyclictest1481-21wc21:10:283
706199118,2cyclictest1481-21wc21:10:283
706199118,2cyclictest14609-21cat22:45:273
706199118,2cyclictest12548-21sed20:25:273
706199118,2cyclictest10457-21/usr/sbin/munin23:45:213
706199118,2cyclictest10041-21taskset22:36:023
706199117,3cyclictest24678-21grep00:15:163
706199117,3cyclictest12932-21tr21:35:203
706199113,4cyclictest225-21systemd-journal20:00:173
705799119,1cyclictest19688-21grep20:40:282
705799118,2cyclictest9352-21cat22:35:212
705799118,2cyclictest7591-21grep22:30:272
705799118,2cyclictest687-21head21:10:182
705799118,2cyclictest687-21head21:10:172
705799118,2cyclictest6862-21grep20:15:202
705799118,2cyclictest6646-21seq23:38:362
705799118,2cyclictest6359-21sed21:20:262
705799118,2cyclictest4947-21ls20:10:292
705799118,2cyclictest3449-21cron00:39:592
705799118,2cyclictest29584-21head00:25:182
705799118,2cyclictest2797-21tail20:05:262
705799118,2cyclictest27764-21expr23:15:152
705799118,2cyclictest23317-21tail23:05:162
705799118,2cyclictest22622-21wc00:10:202
705799118,2cyclictest20024-21sed00:05:222
705799118,2cyclictest19875-21sh23:00:002
705799118,2cyclictest18789-21cat00:05:002
705799118,2cyclictest17545-21grep19:30:172
705799118,2cyclictest14408-21ntp_offset20:30:212
705799118,2cyclictest11296-21fschecks_time22:40:172
705799118,2cyclictest10163-21grep23:45:192
705799112,2cyclictest225-21systemd-journal21:55:002
705799111,2cyclictest0-21swapper/219:10:142
705499118,2cyclictest9225-21basename21:30:011
705499118,2cyclictest8876-21cut22:35:181
705499118,2cyclictest4399-21sort23:35:011
705499118,2cyclictest32622-21grep21:10:161
705499118,2cyclictest32622-21grep21:10:151
705499118,2cyclictest31295-21grep20:00:161
705499118,2cyclictest30924-21perf20:00:001
705499118,2cyclictest25138-21nfs4_client22:00:191
705499118,2cyclictest24171-21tr00:15:141
705499118,2cyclictest20924-21grep19:35:261
705499118,2cyclictest15851-21sed23:55:281
705499118,2cyclictest11846-21sed22:40:211
705499118,2cyclictest10235-21head21:30:191
705499117,3cyclictest284022sleep100:23:241
705499117,2cyclictest3269-21perf20:10:001
7054991110,1cyclictest0-21swapper/121:50:271
705299119,1cyclictest25233-21tr23:10:160
705299119,1cyclictest25233-21tr23:10:160
705299118,2cyclictest8881-21grep22:35:180
705299118,2cyclictest29469-21taskset21:02:040
705299118,2cyclictest24320-21tr00:15:140
705299118,2cyclictest24193-21grep19:45:200
705299118,2cyclictest23520-21apt-get22:00:000
705299118,2cyclictest1828-21grep22:20:170
705299118,2cyclictest18256-21tr20:40:170
705299118,2cyclictest14474-21tr23:55:170
705299118,2cyclictest11973-21fgrep22:40:220
705299117,3cyclictest2818-21tail20:05:250
705299115,2cyclictest225-21systemd-journal21:15:000
705299115,2cyclictest225-21systemd-journal21:15:000
705299111,3cyclictest9402-21munin-run19:15:000
706199108,1cyclictest89532sleep321:27:003
706199108,1cyclictest8073-21taskset20:18:093
706199108,1cyclictest77272sleep322:31:473
706199108,1cyclictest5345-21cut21:20:163
706199108,1cyclictest29011-21cat00:25:163
706199108,1cyclictest28726-21taskset22:10:013
706199108,1cyclictest281952sleep319:50:523
706199108,1cyclictest25408-21taskset22:00:233
706199108,1cyclictest25160-21grep23:10:163
706199108,1cyclictest25160-21grep23:10:163
706199108,1cyclictest24894-21chrt20:54:313
706199108,1cyclictest16536-21taskset00:00:053
706199108,1cyclictest13845-21dnf20:30:173
706199107,2cyclictest21373-21chrt21:55:053
706199107,2cyclictest18486-21expr20:40:163
706199107,2cyclictest10768-21strings19:15:213
706199107,2cyclictest10768-21strings19:15:213
706199106,3cyclictest17997-21tr22:55:153
706199105,3cyclictest27611-21apache_volume21:00:173
706199102,6cyclictest22365-21fschecks_time19:40:163
705799108,2cyclictest14143-21ntp_kernel_err22:45:212
705799108,1cyclictest7313-21cut23:40:152
705799108,1cyclictest30528-21taskset22:10:382
705799108,1cyclictest30112-21hostname21:05:152
705799108,1cyclictest263842sleep200:20:002
705799108,1cyclictest24266-21tr00:15:142
705799108,1cyclictest22000-21entropy21:55:172
705799108,1cyclictest20509-21cat23:00:182
705799108,1cyclictest16980-21expr21:45:142
705799107,2cyclictest9702-21grep21:30:152
705799107,2cyclictest32736-21grep23:25:152
705799107,2cyclictest27127-21grep19:50:182
705799107,2cyclictest2593-21grep21:15:152
705799107,2cyclictest25276-21tr20:55:112
705799107,2cyclictest23065-21grep20:50:162
705799107,2cyclictest20214-21sort19:35:182
705799107,2cyclictest16108-21grep22:50:162
705799107,2cyclictest15762-21fgrep19:25:262
705799107,2cyclictest1394-21grep22:20:142
705799100,9cyclictest260642sleep219:48:592
705499108,1cyclictest6958-21cat22:30:201
705499108,1cyclictest27645-21wget21:00:181
705499108,1cyclictest22135-21expr00:10:151
705499108,1cyclictest21089-21uniq20:45:161
705499108,1cyclictest20522-21ls23:00:181
705499108,1cyclictest16446-21sed22:50:191
705499108,1cyclictest161462sleep121:42:371
705499107,2cyclictest867-21taskset00:32:361
705499107,2cyclictest6366-21cut20:15:171
705499107,2cyclictest30342-21grep23:20:171
705499107,2cyclictest24647-21taskset23:09:241
705499107,2cyclictest1728-21expr00:35:121
705499107,2cyclictest1555-21expr22:20:151
705499106,3cyclictest17897-21grep19:30:201
705499105,4cyclictest29461-21ls23:19:591
705499100,9cyclictest9361-21seq19:14:471
705499100,9cyclictest11235-21taskset23:46:131
705299108,1cyclictest24693-21fschecks_time22:00:160
705299108,1cyclictest221072sleep023:02:170
705299108,1cyclictest18549-21chrt21:48:010
705299108,1cyclictest15753-21grep19:25:240
705299108,1cyclictest11529-21cut20:25:180
705299107,2cyclictest6135-21cut22:30:150
705299107,2cyclictest4819-21wc22:25:220
705299107,2cyclictest26708-21tr00:20:150
705299107,2cyclictest14652-21sensors_fan20:30:240
705299106,3cyclictest23382-21sed23:05:170
705299100,9cyclictest32680-21sed23:25:140
705299100,9cyclictest23521-21chrt19:41:470
70619997,1cyclictest21238-21chrt19:38:133
70579997,2cyclictest15837-21grep20:35:162
70579997,1cyclictest8547-21cat21:25:252
70579997,1cyclictest32529-21taskset22:15:232
70579994,2cyclictest225-21systemd-journal19:41:332
70549998,1cyclictest12162-21expr21:35:151
70549997,1cyclictest20310-21date20:45:001
70529997,1cyclictest21326-21date21:55:000
70619982,3cyclictest17546-21grep19:30:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional