You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-24 - 05:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Apr 24, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1530321140,0sleep00-21swapper/021:17:350
821429986,9sleep00-21swapper/019:06:010
848829170,16sleep20-21swapper/219:09:332
837328666,15sleep10-21swapper/119:08:031
835028559,9sleep30-21swapper/319:07:463
204322470,0sleep20-21swapper/221:59:532
861299221,20cyclictest4875-21latency_hist22:10:011
861299220,22cyclictest0-21swapper/100:16:491
861799201,1cyclictest0-21swapper/321:37:013
861799200,17cyclictest0-21swapper/323:21:113
8612992020,0cyclictest0-21swapper/123:22:151
3432200,0sleep20-21swapper/200:03:472
8617991912,3cyclictest9137-21ssh23:10:203
861499191,18cyclictest0-21swapper/222:37:292
861299191,15cyclictest38150irq/131-snd_hda23:40:141
861299191,15cyclictest38150irq/131-snd_hda23:40:131
861499180,17cyclictest0-21swapper/222:44:572
8617991716,1cyclictest0-21swapper/300:11:143
861799170,2cyclictest0-21swapper/300:18:443
8612991715,2cyclictest25-21ksoftirqd/121:54:031
861299170,1cyclictest0-21swapper/120:30:131
861299170,17cyclictest0-21swapper/100:00:451
861299170,16cyclictest0-21swapper/122:59:421
860799170,17cyclictest0-21swapper/000:38:380
861799164,11cyclictest0-21swapper/321:19:223
861799161,14cyclictest27966-21ssh22:23:323
8614991615,1cyclictest0-21swapper/222:20:372
8614991614,1cyclictest0-21swapper/222:25:132
861499160,16cyclictest0-21swapper/200:21:152
861499160,13cyclictest0-21swapper/220:00:122
861499160,11cyclictest0-21swapper/200:35:192
861299160,8cyclictest537-21snmpd21:26:591
861299160,16cyclictest0-21swapper/123:55:171
860799164,6cyclictest20626-21sendmail22:00:010
860799161,14cyclictest0-21swapper/021:30:150
861799151,11cyclictest0-21swapper/320:40:123
861799150,15cyclictest0-21swapper/323:47:563
861499157,5cyclictest11098-21cron21:35:002
861499151,13cyclictest351-21apache_processe23:25:122
8614991511,2cyclictest10522-21sh00:10:002
860799151,7cyclictest5423-21munin-run20:10:010
860799150,1cyclictest0-21swapper/000:17:040
861799142,12cyclictest0-21swapper/323:58:483
8617991411,2cyclictest20754-21grep00:35:163
861799140,14cyclictest0-21swapper/322:58:283
861799140,12cyclictest2119-21/usr/sbin/munin21:10:123
861499140,14cyclictest0-21swapper/200:11:362
861499140,13cyclictest0-21swapper/221:52:332
861299149,4cyclictest31589-21turbostat23:05:001
861299149,3cyclictest21014-21wc20:40:181
861299148,5cyclictest38150irq/131-snd_hda23:31:071
861299142,11cyclictest1142-21runrttasks22:27:401
8612991413,1cyclictest0-21swapper/121:14:511
8612991411,2cyclictest12057-21wc20:20:261
8612991410,3cyclictest22521-21cut22:00:211
8612991410,3cyclictest18290-21grep23:15:261
861299140,13cyclictest0-21swapper/122:37:141
861299140,13cyclictest0-21swapper/100:28:591
860799142,6cyclictest0-21swapper/021:05:130
8607991411,2cyclictest24943-21wc19:40:250
8607991410,3cyclictest396-21grep20:00:010
8607991410,3cyclictest1856-21wc23:25:220
8607991410,3cyclictest14399-21wc20:25:270
860799140,12cyclictest31663-21sleep21:05:010
861799139,3cyclictest22362-21cron22:40:003
861799136,2cyclictest39-21ksoftirqd/322:32:003
861799131,5cyclictest0-21swapper/300:05:133
8617991310,2cyclictest3595-21grep21:30:163
8617991310,2cyclictest31436-21wc22:25:193
8617991310,2cyclictest20923-21grep21:40:173
8617991310,2cyclictest16624-21grep23:15:133
8617991310,2cyclictest0-21swapper/300:00:233
861799130,3cyclictest0-21swapper/300:25:393
861799130,13cyclictest0-21swapper/322:12:323
861499139,3cyclictest32326-21cat22:45:202
8614991310,2cyclictest21515-21cron22:20:012
8614991310,2cyclictest20090-21head00:15:152
861499130,12cyclictest0-21swapper/223:34:032
861299139,3cyclictest7560-21wc23:10:001
861299139,3cyclictest5667-21ssh00:06:191
861299139,2cyclictest8322-21tr20:15:141
8612991310,2cyclictest31287-21tr22:45:141
8612991310,2cyclictest29456-21grep19:50:191
8612991310,2cyclictest14569-21grep22:15:191
8612991310,2cyclictest13597-21head20:25:181
8612991310,2cyclictest10632-21basename21:15:151
861299130,0cyclictest0-21swapper/123:29:031
860799139,3cyclictest10512-21cron00:10:000
860799136,6cyclictest16550-21munin-run23:35:010
860799131,10cyclictest13892-21apt-get19:20:120
8607991310,2cyclictest9038-21grep22:50:250
8607991310,2cyclictest27207-21grep23:40:250
8607991310,2cyclictest27207-21grep23:40:250
8607991310,2cyclictest20980-21sed21:20:260
8607991310,2cyclictest16739-21tr22:55:210
8607991310,2cyclictest15675-21grep22:15:280
8607991310,2cyclictest10942-21cron19:15:000
861799129,2cyclictest694-21grep23:25:163
861799129,2cyclictest30402-21grep21:45:273
861799129,2cyclictest30402-21grep21:45:263
861799129,2cyclictest28284-21tr20:55:213
861799129,2cyclictest24545-21grep23:00:163
861799129,2cyclictest17033-21grep19:25:173
861799129,2cyclictest1638-21wc21:10:003
861799129,2cyclictest13016-21basename20:25:143
861799129,2cyclictest11692-21tr19:15:153
861799129,2cyclictest10451-21grep19:10:253
861799128,3cyclictest9771-21cut23:30:193
861799128,3cyclictest2587-21sed20:00:263
861799121,2cyclictest5778-21ssh22:49:223
8617991212,0cyclictest0-21swapper/323:53:193
861799120,5cyclictest29717-21kworker/3:121:57:553
861799120,2cyclictest0-21swapper/323:42:323
861799120,2cyclictest0-21swapper/323:42:323
861799120,1cyclictest0-21swapper/322:16:233
861799120,11cyclictest13024-21cut00:30:173
861499129,2cyclictest8240-21grep23:10:152
861499129,2cyclictest3338-21sed23:45:242
861499129,2cyclictest31438-21grep19:55:172
861499129,2cyclictest2976-21sed21:10:172
861499129,2cyclictest28589-21cat20:55:252
861499129,2cyclictest26550-21grep19:45:162
861499129,2cyclictest26155-21grep23:40:162
861499129,2cyclictest26155-21grep23:40:162
861499129,2cyclictest21305-21cut21:40:202
861499129,2cyclictest19642-21grep23:55:222
861499129,2cyclictest18890-21wc20:35:212
861499129,2cyclictest12039-21df19:15:152
861499121,5cyclictest0-21swapper/222:15:002
861499120,12cyclictest0-21swapper/223:37:332
861299129,2cyclictest9946-21cut19:10:191
861299129,2cyclictest4698-21grep21:30:231
861299129,2cyclictest4035-21head20:05:161
861299129,2cyclictest24178-21sed19:40:161
861299129,2cyclictest23152-21tail20:45:181
861299129,2cyclictest2262-21sed23:45:141
861299128,3cyclictest12212-21tail19:15:161
861299128,2cyclictest30703-21cron22:45:011
861299121,8cyclictest38150irq/131-snd_hda20:35:131
861299120,1cyclictest0-21swapper/123:14:091
860799129,2cyclictest5716-21tr21:50:210
860799129,2cyclictest27313-21cut00:00:170
860799129,2cyclictest26025-21ps20:50:210
860799129,2cyclictest25068-21rm23:40:000
860799129,2cyclictest23622-21grep20:45:220
860799129,2cyclictest2022-21grep20:00:190
860799129,2cyclictest20106-21taskset00:35:000
860799129,2cyclictest16516-21grep19:25:140
860799129,2cyclictest14864-21cut22:35:150
860799129,2cyclictest12653-21wc20:25:000
860799128,3cyclictest24584-21tail23:00:160
860799128,3cyclictest20740-21sh19:35:000
860799127,1cyclictest0-21swapper/022:07:420
860799125,6cyclictest24107-21munin-run23:20:000
860799121,9cyclictest21817-21sensors21:40:240
8607991211,1cyclictest181irq_work/023:22:270
860799120,2cyclictest0-21swapper/022:04:320
861799119,1cyclictest30167-21chrt19:51:173
861799119,1cyclictest17862-21grep23:35:173
861799118,2cyclictest8723-21tune2fs20:15:163
861799118,2cyclictest607-21ssh21:28:193
861799118,2cyclictest5692-21cut21:50:213
861799118,2cyclictest31941-21sh23:05:073
861799118,2cyclictest31920-21ssh22:05:503
861799118,2cyclictest31335-21grep19:55:163
861799118,2cyclictest29684-21egrep21:00:143
861799118,2cyclictest26710-21ssh22:41:503
861799118,2cyclictest25252-21ssh22:02:073
861799118,2cyclictest24523-21awk20:50:003
861799118,2cyclictest22816-21ssh21:21:483
861799118,2cyclictest21757-21grep19:35:173
861799118,2cyclictest2138-21ssh00:24:173
861799118,2cyclictest19361-21who20:35:263
861799118,2cyclictest12557-21taskset20:24:203
861799118,2cyclictest11290-21ssh22:52:113
861499119,1cyclictest22140-21ls20:44:592
861499118,2cyclictest9742-21cat19:10:172
861499118,2cyclictest7623-21ssh00:27:002
861499118,2cyclictest6800-21tr20:10:202
861499118,2cyclictest3576-21ssh23:06:562
861499118,2cyclictest29510-21cat21:00:112
861499118,2cyclictest29201-21ssh22:05:052
861499118,2cyclictest28841-21wc21:25:232
861499118,2cyclictest27264-21runrttasks23:20:562
861499118,2cyclictest25519-21ssh22:02:202
861499118,2cyclictest24026-21sed20:45:282
861499118,2cyclictest22923-21rm23:19:052
861499118,2cyclictest21067-21sed21:20:262
861499118,2cyclictest15288-21rm22:55:062
861499118,2cyclictest14122-21grep19:20:122
861499118,2cyclictest12138-21ssh00:30:122
861499118,2cyclictest11716-21ssh23:50:262
861499118,2cyclictest10837-21grep21:15:162
861499118,2cyclictest10375-21cat20:20:002
861499117,3cyclictest7374-21tr22:50:132
861499112,4cyclictest0-21swapper/221:45:132
861499112,4cyclictest0-21swapper/221:45:132
861499110,10cyclictest6709-21expr22:30:142
861299118,2cyclictest7905-21grep22:50:161
861299118,2cyclictest32437-21head19:55:271
861299118,2cyclictest31558-21chrt21:03:581
861299118,2cyclictest30771-21ssh21:45:381
861299118,2cyclictest30771-21ssh21:45:381
861299118,2cyclictest29180-21systemctl00:20:221
861299118,2cyclictest28311-21ssh22:23:491
861299118,2cyclictest23512-21ssh21:22:201
861299118,2cyclictest22959-21taskset19:38:061
861299118,2cyclictest22952-21ssh21:41:011
861299118,2cyclictest21061-21chrt23:37:011
861299118,2cyclictest15676-21ssh21:56:091
861299118,2cyclictest1511-21taskset21:08:571
861299118,2cyclictest13939-21ssh00:11:091
861299118,2cyclictest13724-21tr00:30:231
861299118,2cyclictest12834-21ssh22:34:021
861299114,3cyclictest0-21swapper/100:35:161
861299110,10cyclictest5743-21cut20:10:141
860799119,1cyclictest21990-21/usr/sbin/munin22:20:130
860799118,2cyclictest8629-21ssh22:11:290
860799118,2cyclictest7696-21ssh22:30:230
860799118,2cyclictest7319-21ssh21:12:540
860799118,2cyclictest678-21ssh21:28:230
860799118,2cyclictest4924-21ssh23:08:010
860799118,2cyclictest32515-21tr22:45:210
860799118,2cyclictest28815-21tr21:45:140
860799118,2cyclictest28815-21tr21:45:140
860799118,2cyclictest28430-21grep19:50:130
860799118,2cyclictest23816-21ssh23:58:180
860799118,2cyclictest20575-21grep20:40:140
860799118,2cyclictest13972-21ssh23:13:330
860799118,2cyclictest11722-21cut00:10:160
860799118,2cyclictest11407-21ssh23:50:250
860799117,3cyclictest17670-21cut20:35:140
860799113,4cyclictest225-21systemd-journal00:28:270
860799110,9cyclictest0-21swapper/000:20:130
860799110,2cyclictest0-21swapper/023:48:490
860799110,10cyclictest16164-21ssh21:37:240
861799108,2cyclictest6804-21ntp_kernel_pll_20:10:213
861799108,1cyclictest3509-21cat20:05:143
861799108,1cyclictest15398-21tr20:30:133
861799107,2cyclictest24750-21acpi20:50:103
861799107,2cyclictest15590-21sed19:20:243
861799100,9cyclictest25742-21taskset19:45:103
861799100,9cyclictest25368-21seq19:43:203
861499108,1cyclictest32673-21tail21:05:172
861499108,1cyclictest23984-21entropy19:40:172
861499108,1cyclictest19489-21if_enp0s31f619:30:162
861499108,1cyclictest11344-21ssh21:35:022
861499107,2cyclictest29667-21gzip19:50:212
861499107,2cyclictest25722-21tail23:00:252
861499107,2cyclictest16388-21aten2_r7power_v19:25:142
861499107,2cyclictest13670-21grep20:25:172
861499107,2cyclictest10922-21expr20:20:152
861299108,1cyclictest5151-21ssh22:10:061
861299108,1cyclictest28081-21sed19:50:001
861299108,1cyclictest24815-21awk20:50:131
861299108,1cyclictest17240-21sed19:25:191
861299108,1cyclictest11334-21ssh21:35:021
861299107,2cyclictest19284-21sort19:30:161
861299107,2cyclictest14645-21head19:20:141
861299107,2cyclictest13270-21ssh23:51:331
861299106,3cyclictest3006-21taskset20:04:451
861299101,5cyclictest27858-21grep20:55:161
860799107,2cyclictest8789-21tail20:15:160
860799107,2cyclictest32502-21ssh22:25:330
860799107,2cyclictest28916-21chrt20:55:480
860799107,2cyclictest23401-21expr22:40:150
860799107,2cyclictest22667-21tail19:35:260
860799100,9cyclictest7009-21run-parts20:10:230
86179997,1cyclictest19066-21expr19:30:143
86149998,1cyclictest26334-21sed20:50:252
86149997,2cyclictest21597-21entropy19:35:162
86149997,1cyclictest3759-21expr20:05:142
86149996,2cyclictest15518-21tr20:30:132
86079997,1cyclictest26321-21sed19:45:140
86079997,1cyclictest17530-21sed20:35:000
86079997,1cyclictest133252sleep019:16:350
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional