You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-05 - 18:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Sep 05, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3139429368,8sleep20-21swapper/207:07:392
3123429371,17sleep10-21swapper/107:05:331
56002920,0sleep00-21swapper/010:54:340
3124328760,8sleep00-21swapper/007:05:410
3149828370,9sleep30-21swapper/307:09:013
25082630,0sleep30-21swapper/312:29:263
3166299292,26cyclictest0-21swapper/107:35:191
3166299260,26cyclictest0-21swapper/110:27:561
31658992621,4cyclictest5406-21apache210:20:040
31662992423,1cyclictest24684-21diskmemload11:23:361
3166299241,1cyclictest0-21swapper/112:19:271
31658992421,2cyclictest5417-21apache212:10:080
31667992322,1cyclictest0-21swapper/309:15:543
3166799231,5cyclictest141rcu_preempt11:44:593
3166799230,21cyclictest13587-21pmu-power12:35:233
31664992320,2cyclictest5406-21apache210:38:232
31662992319,1cyclictest0-21swapper/112:20:281
31658992320,2cyclictest5417-21apache209:35:360
31658992320,2cyclictest5406-21apache209:45:150
3166799220,2cyclictest0-21swapper/310:29:543
3166799220,16cyclictest0-21swapper/309:05:133
31664992220,1cyclictest5417-21apache211:07:512
31664992220,1cyclictest5417-21apache209:18:272
31664992220,1cyclictest5406-21apache211:13:532
31664992220,1cyclictest5406-21apache210:23:342
31664992219,2cyclictest5406-21apache211:19:262
3166299220,1cyclictest0-21swapper/109:29:281
31658992220,1cyclictest5417-21apache211:13:060
31658992220,1cyclictest5406-21apache212:26:280
31658992220,1cyclictest5406-21apache209:41:230
31664992118,2cyclictest5417-21apache212:21:242
3166299210,1cyclictest0-21swapper/109:15:541
31658992120,1cyclictest0-21swapper/010:55:140
31658992119,1cyclictest5417-21apache211:33:060
31658992119,1cyclictest5417-21apache209:29:570
31658992119,1cyclictest5406-21apache209:23:260
31667992020,0cyclictest0-21swapper/311:56:013
31667992020,0cyclictest0-21swapper/310:07:043
3166799200,1cyclictest0-21swapper/311:37:293
3166799200,1cyclictest0-21swapper/310:32:223
3166299200,1cyclictest0-21swapper/110:05:521
31658992018,1cyclictest5417-21apache212:15:060
31658992018,1cyclictest5417-21apache210:31:080
31658992018,1cyclictest5406-21apache211:58:410
3165899201,18cyclictest0-21swapper/008:50:140
31664991917,1cyclictest5417-21apache212:15:312
31664991917,1cyclictest5406-21apache209:31:052
31662991918,1cyclictest24684-21diskmemload11:41:031
3166299190,3cyclictest0-21swapper/110:58:191
31658991917,1cyclictest5406-21apache211:19:020
31658991916,2cyclictest5417-21apache210:26:460
3166499184,11cyclictest8906-21ssh11:35:142
31664991816,1cyclictest5417-21apache212:04:122
31664991816,1cyclictest5417-21apache211:46:192
3166299182,15cyclictest0-21swapper/109:10:171
31662991816,1cyclictest0-21swapper/112:13:091
3166299180,1cyclictest0-21swapper/110:31:271
3166299180,18cyclictest0-21swapper/112:38:271
31658991816,1cyclictest5406-21apache209:53:070
31658991815,2cyclictest5406-21apache210:11:580
3165899180,2cyclictest0-21swapper/010:41:240
3166799170,17cyclictest0-21swapper/310:53:083
31664991715,1cyclictest5417-21apache211:44:022
31664991715,1cyclictest5417-21apache209:44:452
31664991715,1cyclictest5406-21apache209:24:342
31664991711,5cyclictest13365-21munin-node11:04:312
3166499170,1cyclictest0-21swapper/210:42:082
3166299172,15cyclictest24684-21diskmemload09:37:381
31662991716,1cyclictest0-21swapper/107:23:021
31658991716,1cyclictest0-21swapper/012:06:280
31658991715,1cyclictest5417-21apache212:36:070
3166799165,8cyclictest25184-21apache_accesses09:10:133
31664991614,1cyclictest5417-21apache212:10:042
31664991612,3cyclictest5417-21apache209:39:352
3166499160,16cyclictest0-21swapper/209:13:432
31662991614,2cyclictest0-21swapper/112:08:431
3166299160,3cyclictest0-21swapper/112:27:581
3166299160,2cyclictest0-21swapper/110:47:391
3166299160,16cyclictest0-21swapper/112:34:051
3166299160,16cyclictest0-21swapper/111:56:521
3166299160,15cyclictest0-21swapper/110:14:281
31658991614,1cyclictest5417-21apache211:49:450
31658991614,1cyclictest5417-21apache209:19:550
3166799150,5cyclictest0-21swapper/311:46:223
3166799150,14cyclictest0-21swapper/309:29:163
3166799150,13cyclictest0-21swapper/312:13:093
3166499153,5cyclictest24672-21munin-run09:10:002
3166499152,12cyclictest0-21swapper/209:51:482
3166499150,1cyclictest0-21swapper/211:30:472
3166499150,15cyclictest0-21swapper/210:59:022
3166499150,15cyclictest0-21swapper/210:30:562
3166299150,4cyclictest0-21swapper/110:39:421
3166299150,14cyclictest0-21swapper/109:50:081
31658991513,1cyclictest5417-21apache209:30:380
31658991513,1cyclictest31652-21cyclictest10:01:010
3165899150,1cyclictest0-21swapper/010:16:080
3166799143,8cyclictest32342-21cron11:30:003
3166799140,9cyclictest0-21swapper/311:22:323
3166799140,1cyclictest0-21swapper/311:51:143
3166799140,13cyclictest0-21swapper/312:22:213
3166799140,12cyclictest0-21swapper/309:54:293
31664991414,0cyclictest0-21swapper/212:38:242
31664991412,1cyclictest5417-21apache210:15:042
31664991411,2cyclictest6732-21wc08:30:242
31664991411,2cyclictest19267-21grep07:50:152
31664991410,3cyclictest29573-21wc08:10:252
3166299143,6cyclictest30195-21cron08:15:001
31662991414,0cyclictest0-21swapper/108:34:561
3166299140,4cyclictest0-21swapper/111:08:041
3166299140,1cyclictest0-21swapper/109:49:271
3166299140,13cyclictest0-21swapper/109:33:101
3165899145,5cyclictest225-21systemd-journal09:00:010
3165899141,5cyclictest32353-21munin-run11:29:590
31658991411,2cyclictest8785-21cron07:30:000
31658991410,2cyclictest32386-21df07:10:160
31667991310,2cyclictest7883-21wget11:15:153
31667991310,2cyclictest20231-21sed07:50:243
3166499139,3cyclictest545-21grep07:10:212
3166499132,9cyclictest2687-21ssh10:14:072
3166499131,3cyclictest141rcu_preempt11:30:002
31664991311,1cyclictest5417-21apache211:52:342
31664991311,1cyclictest5406-21apache209:55:572
31664991310,2cyclictest4515-21grep07:20:142
31664991310,2cyclictest27843-21apt-get12:25:122
31664991310,2cyclictest26495-21grep08:05:172
31664991310,2cyclictest17966-21grep11:20:252
3166499130,13cyclictest0-21swapper/209:46:552
3166499130,12cyclictest18001-21run-parts12:00:002
31662991312,1cyclictest0-21swapper/111:30:371
31662991310,2cyclictest27478-21cat11:45:251
31662991310,2cyclictest1080-21grep07:10:271
3166299130,3cyclictest28176-21sh11:26:451
3166299130,1cyclictest0-21swapper/111:54:271
3166299130,0cyclictest0-21swapper/109:57:331
3165899139,3cyclictest3957-21cron07:20:000
3165899139,2cyclictest24037-21wc09:05:260
3165899134,8cyclictest9902-21sh11:35:190
31658991311,1cyclictest5406-21apache210:36:200
31658991310,2cyclictest25908-21cut09:10:160
31658991310,2cyclictest19738-21grep12:20:170
31658991310,2cyclictest16082-21wc07:45:000
31658991310,2cyclictest14269-21wc08:45:250
238322130,0chrt0-21swapper/011:24:550
3166799129,2cyclictest7786-21sed08:35:013
3166799129,2cyclictest7534-21wc10:35:283
3166799129,2cyclictest7204-21expr10:55:153
3166799129,2cyclictest6959-21grep07:25:153
3166799129,2cyclictest6734-21ssh10:15:423
3166799129,2cyclictest5023-21ssh12:30:213
3166799129,2cyclictest4904-21wc09:55:243
3166799129,2cyclictest32240-21cat07:10:163
3166799129,2cyclictest26945-21head08:05:233
3166799129,2cyclictest26196-21ssh10:46:523
3166799129,2cyclictest25831-21ssh11:05:453
3166799129,2cyclictest2331-21rm11:11:133
3166799129,2cyclictest21571-21grep09:00:283
3166799129,2cyclictest20191-21wc12:00:273
3166799129,2cyclictest17941-21ssh11:01:083
3166799129,2cyclictest14783-21ssh09:22:593
3166799129,2cyclictest13593-21grep10:20:173
3166799129,2cyclictest13449-21cut08:45:153
3166799129,2cyclictest11538-21ssh10:00:043
3166799129,2cyclictest11215-21tail08:40:183
3166799123,8cyclictest10894-21ssh09:40:143
3166799123,2cyclictest0-21swapper/309:35:133
3166799121,11cyclictest0-21swapper/312:18:283
3166799121,10cyclictest29487-21if_enp2s010:10:223
3166499129,2cyclictest7918-21grep07:25:232
3166499129,2cyclictest3663-21grep08:25:182
3166499129,2cyclictest3663-21grep08:25:172
3166499129,2cyclictest32396-21sed08:15:232
3166499129,2cyclictest25445-21cron08:05:002
3166499129,2cyclictest21948-21grep10:25:172
3166499129,2cyclictest21335-21cut10:05:202
3166499129,2cyclictest17301-21cron08:55:002
31664991210,1cyclictest5406-21apache212:34:572
31664991210,1cyclictest17958-21aten2_r7power_e09:25:152
3166299129,2cyclictest20724-21cut09:00:181
3166299129,2cyclictest19400-21basename07:50:171
3166299129,2cyclictest19191-21ssh10:23:531
3166299129,2cyclictest15582-21taskset07:40:371
3166299129,2cyclictest12493-21sed10:20:011
3166299129,2cyclictest11639-21tail09:40:171
3166299129,2cyclictest10813-21grep11:35:271
3166299128,3cyclictest913-21mailstats11:10:231
3166299124,2cyclictest452-21systemd-logind09:20:211
3166299123,6cyclictest7851-21apt-get08:35:151
3166299121,7cyclictest24684-21diskmemload12:00:181
3166299120,1cyclictest0-21swapper/111:16:571
3166299120,1cyclictest0-21swapper/110:50:391
3166299120,12cyclictest0-21swapper/111:00:311
3165899129,2cyclictest6004-21grep07:20:240
3165899129,2cyclictest22558-21grep07:55:260
3165899129,2cyclictest17139-21uname07:45:180
3165899129,2cyclictest11780-21ssh12:35:000
3165899128,3cyclictest23101-21sh11:43:180
31658991210,1cyclictest5406-21apache212:01:350
3166799119,2cyclictest21542-21ssh09:45:463
3166799119,1cyclictest9879-21taskset08:39:143
3166799119,1cyclictest5358-21cut07:20:193
3166799119,1cyclictest4723-21sed08:25:283
3166799119,1cyclictest4723-21sed08:25:273
3166799119,1cyclictest432-21ssh11:30:133
3166799119,1cyclictest30568-21apt-config08:15:003
3166799119,1cyclictest27048-21cpu12:05:163
3166799119,1cyclictest26056-21appstreamcli09:30:003
3166799119,1cyclictest25630-21cat08:05:013
3166799119,1cyclictest2175-21expr07:15:133
3166799119,1cyclictest181202sleep307:47:443
3166799119,1cyclictest14250-21cat07:40:163
3166799118,2cyclictest29087-21sed08:10:193
3166799118,2cyclictest18628-21cat08:55:203
3166799118,2cyclictest17136-21sh10:41:183
3166799118,2cyclictest15472-21cut08:50:163
3166799118,2cyclictest12768-21systemctl07:35:233
3166499119,1cyclictest5417-21apache210:51:472
3166499118,2cyclictest27729-21head12:05:192
3166499118,2cyclictest26308-21ssh10:46:572
3166499118,2cyclictest21380-21expr07:55:162
3166499118,2cyclictest16431-21tr07:45:152
3166499118,2cyclictest15561-21ssh10:01:522
3166499118,2cyclictest14588-21head07:40:182
3166499118,2cyclictest12374-21cut07:35:192
3166299119,1cyclictest30737-21basename08:15:111
3166299119,1cyclictest25796-21awk08:05:141
3166299119,1cyclictest24674-21which08:00:241
3166299119,1cyclictest23374-21basename09:05:181
3166299119,1cyclictest21437-21df07:55:151
3166299119,1cyclictest18501-21uname08:55:191
3166299119,1cyclictest1814-21taskset07:15:051
3166299119,1cyclictest16057-21ssh10:40:271
3166299119,1cyclictest158422sleep110:02:041
3166299118,2cyclictest952-21expr08:20:161
3166299118,2cyclictest10685-21tr08:40:151
3166299118,2cyclictest0-21swapper/107:30:011
3166299110,10cyclictest16494-21cstates07:45:151
3165899118,2cyclictest9712-21cut07:30:170
3165899118,2cyclictest5595-21aten2_r7power_v08:30:160
3165899118,2cyclictest3116-21ssh09:55:030
3165899118,2cyclictest28754-21ssh10:48:530
3165899118,2cyclictest23984-21cut08:00:180
3165899118,2cyclictest21790-21ssh10:05:260
3165899118,2cyclictest21171-21cut09:00:220
3165899118,2cyclictest1625-21ssh11:50:080
3165899118,2cyclictest13-21ksoftirqd/008:20:160
3165899118,2cyclictest11098-21cut08:40:160
3165899117,3cyclictest26468-21sed08:05:170
3165899117,3cyclictest14958-21/usr/sbin/munin11:00:240
3165899114,3cyclictest0-21swapper/011:09:370
3166799109,1cyclictest20949-21apache_volume07:55:143
3166799108,1cyclictest606-21chrt08:20:013
3166799100,10cyclictest9085-21aten2_r7power_c07:30:153
3166499108,1cyclictest9716-21chrt08:37:062
3166499108,1cyclictest3833-21taskset07:18:282
3166499108,1cyclictest19525-21taskset08:58:102
3166499108,1cyclictest1199-21sed08:20:162
3166499108,1cyclictest10595-21egrep08:40:152
3166499107,2cyclictest9285-21grep07:30:152
3166499107,2cyclictest19941-21cat09:00:122
3166499107,2cyclictest13912-21grep08:45:192
3166299108,1cyclictest4926-21taskset08:27:191
3166299108,1cyclictest4926-21taskset08:27:181
3166299108,1cyclictest145992sleep108:45:521
3166299104,5cyclictest7924-21head07:25:231
3166299104,5cyclictest15467-21expr08:50:151
3165899108,1cyclictest29861-21taskset08:10:480
3165899108,1cyclictest20506-21taskset07:52:540
3165899107,2cyclictest5127-21chrt08:29:560
3165899107,2cyclictest5127-21chrt08:29:550
3165899106,3cyclictest30906-21grep08:15:130
316589997,1cyclictest9916-21taskset08:39:450
316589997,1cyclictest11916-21taskset07:35:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional