You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 23:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Apr 05, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2478829585,6sleep00-21swapper/007:07:030
2468228561,8sleep30-21swapper/307:05:413
2341028265,12sleep20-21swapper/207:05:162
2481427760,13sleep10-21swapper/107:07:241
271472660,0sleep00-21swapper/009:15:120
2510499270,4cyclictest0-21swapper/009:50:140
2510699260,25cyclictest0-21swapper/109:55:161
25106992524,1cyclictest0-21swapper/111:58:301
2510699250,25cyclictest0-21swapper/112:14:091
2511099230,22cyclictest0-21swapper/211:23:322
2510699232,20cyclictest0-21swapper/112:30:121
2510699230,1cyclictest0-21swapper/107:45:181
25112992119,2cyclictest39-21ksoftirqd/311:46:563
2510499210,2cyclictest0-21swapper/010:10:140
25112992018,1cyclictest39-21ksoftirqd/309:49:313
25112992016,3cyclictest39-21ksoftirqd/311:11:563
2511099200,19cyclictest0-21swapper/209:37:362
25106992017,2cyclictest25-21ksoftirqd/108:30:001
2510499200,1cyclictest0-21swapper/010:18:410
25112991917,1cyclictest39-21ksoftirqd/311:37:343
2511299191,0cyclictest141rcu_preempt10:26:283
25106991913,1cyclictest17901-21diskmemload10:48:321
25104991919,0cyclictest0-21swapper/011:05:030
2510499190,1cyclictest0-21swapper/011:22:550
2510499190,0cyclictest0-21swapper/011:57:200
25112991816,1cyclictest39-21ksoftirqd/310:54:533
2511299180,1cyclictest141rcu_preempt10:41:313
2511299180,15cyclictest0-21swapper/310:15:113
2511299180,10cyclictest0-21swapper/310:55:003
2510699180,3cyclictest0-21swapper/109:13:061
2510699180,16cyclictest0-21swapper/112:28:171
2510499186,6cyclictest8381-21munin-run08:50:000
2511299170,6cyclictest141rcu_preempt12:01:403
2511299170,6cyclictest141rcu_preempt12:01:403
25110991717,0cyclictest0-21swapper/212:02:212
25110991717,0cyclictest0-21swapper/212:02:212
2510699170,17cyclictest0-21swapper/110:29:491
2510699170,16cyclictest0-21swapper/111:41:191
2510499171,15cyclictest1253-21sh12:33:030
25112991611,2cyclictest39-21ksoftirqd/309:25:003
25110991615,1cyclictest0-21swapper/211:16:132
2511099160,1cyclictest0-21swapper/211:12:422
2511099160,16cyclictest0-21swapper/211:00:432
2510699160,16cyclictest0-21swapper/108:30:011
2510499160,16cyclictest0-21swapper/011:36:450
2510499160,13cyclictest0-21swapper/010:42:390
2511299157,6cyclictest26941-21df_abs11:50:163
2511299153,11cyclictest11202-21apt-get09:25:133
25112991514,1cyclictest0-21swapper/309:36:093
25112991511,3cyclictest6538-21wc10:00:243
25112991511,2cyclictest15804-21cut10:45:163
2511099155,9cyclictest8637-21apache_accesses08:50:112
25110991514,1cyclictest0-21swapper/212:19:202
25110991512,1cyclictest0-21swapper/211:31:492
2511099150,15cyclictest0-21swapper/209:54:562
25106991514,1cyclictest17901-21diskmemload12:39:471
25106991514,1cyclictest0-21swapper/110:38:121
2510699151,1cyclictest10208-21ssh10:22:141
2510699150,14cyclictest17901-21diskmemload10:59:561
25104991511,3cyclictest18392-21latency_hist12:05:010
25104991511,3cyclictest18392-21latency_hist12:05:010
2510499150,1cyclictest0-21swapper/011:04:360
27762140,0sleep30-21swapper/311:15:423
2511299149,4cyclictest39-21ksoftirqd/309:30:133
2511299146,4cyclictest39-21ksoftirqd/309:55:123
2511299140,1cyclictest141rcu_preempt12:34:193
2511299140,13cyclictest31729-21grep07:20:263
25110991413,1cyclictest55650irq/127-enp2s0-09:18:542
25110991411,3cyclictest55650irq/127-enp2s0-09:30:482
2510699146,8cyclictest25096-21cyclictest11:21:401
25106991414,0cyclictest0-21swapper/111:07:291
25106991410,3cyclictest22585-21ls10:10:201
2510499149,3cyclictest5147-21tr07:35:140
2510499144,5cyclictest26724-21idleruntime-cro12:10:000
2510499140,13cyclictest15997-21sed07:55:260
2510499140,13cyclictest0-21swapper/012:15:140
2511299139,3cyclictest26466-21ssh12:09:483
2511299136,6cyclictest17901-21diskmemload11:57:123
25112991310,2cyclictest9745-21tr07:45:133
25112991310,2cyclictest6980-21grep08:45:173
25112991310,2cyclictest39-21ksoftirqd/307:35:123
25112991310,2cyclictest20266-21grep08:05:213
25112991310,2cyclictest18036-21wc08:00:213
25112991310,2cyclictest11761-21grep08:55:173
2511099139,4cyclictest22691-21kworker/u8:211:51:252
2511099139,3cyclictest20159-21cat09:10:262
2511099138,4cyclictest15159-21ssh10:25:192
2511099137,3cyclictest8377-21sh08:50:002
2511099136,3cyclictest225-21systemd-journal11:40:002
25110991310,2cyclictest8106-21ntpq10:40:212
25110991310,2cyclictest5683-21grep12:35:202
25110991310,2cyclictest5517-21grep08:40:252
25110991310,2cyclictest5517-21grep08:40:252
25110991310,2cyclictest29986-21grep10:15:132
25110991310,2cyclictest29415-21grep07:15:262
25110991310,2cyclictest19968-21cat09:50:002
2511099130,1cyclictest0-21swapper/212:34:472
2510699139,2cyclictest28392-21grep07:15:171
2510699132,3cyclictest19179-21/usr/sbin/munin11:45:191
25106991312,1cyclictest38150irq/131-snd_hda10:19:071
25106991310,2cyclictest7975-21grep08:45:261
25106991310,2cyclictest27339-21awk09:35:001
25106991310,2cyclictest11763-21cut08:55:171
2510699130,1cyclictest0-21swapper/112:07:401
2510699130,1cyclictest0-21swapper/110:05:191
2510699130,13cyclictest0-21swapper/110:30:591
2510699130,0cyclictest0-21swapper/111:54:021
2510699130,0cyclictest0-21swapper/110:02:541
2510499139,3cyclictest304-21tr07:25:140
2510499135,4cyclictest225-21systemd-journal08:05:000
25104991311,1cyclictest5925-21ssh10:20:120
25104991310,2cyclictest7503-21tr07:40:150
25104991310,2cyclictest25751-21grep08:15:240
25104991310,2cyclictest15111-21head10:05:260
25104991310,2cyclictest10869-21grep07:45:200
2510499130,2cyclictest0-21swapper/011:51:590
2510499130,1cyclictest0-21swapper/009:39:480
2510499130,12cyclictest29539-21tune2fs08:25:150
2510499130,10cyclictest0-21swapper/009:47:060
2510499130,0cyclictest0-21swapper/012:35:400
2511299129,2cyclictest9895-21wc08:50:213
2511299129,2cyclictest8257-21wc07:40:203
2511299129,2cyclictest29617-21tail08:25:163
2511299129,2cyclictest25096-21cyclictest09:05:163
2511299129,2cyclictest20713-21cut09:50:153
2511299129,2cyclictest19209-21ssh11:06:203
2511299129,2cyclictest11822-21grep12:20:153
2511299129,1cyclictest141rcu_preempt12:19:313
2511299128,3cyclictest13680-21tr09:00:143
2511299124,6cyclictest0-21swapper/310:20:113
2511299123,8cyclictest24945-21ls11:30:003
25112991210,1cyclictest39-21ksoftirqd/310:35:173
25112991210,1cyclictest0-21swapper/311:40:133
2511299120,11cyclictest225342sleep310:30:133
2511299120,11cyclictest13998-21sed10:05:173
2511099129,2cyclictest4510-21sort09:40:182
2511099129,2cyclictest24499-21grep10:30:272
2511099129,2cyclictest22550-21cat10:10:202
2511099129,2cyclictest20874-21sed12:05:282
2511099129,2cyclictest16548-21sed09:05:172
2511099129,2cyclictest16389-21ntpq10:45:192
2511099129,2cyclictest13101-21sed09:25:262
2511099129,2cyclictest11571-21sed11:40:252
2511099129,2cyclictest10965-21expr07:45:212
2511099121,9cyclictest12636-21cron10:05:002
25110991211,1cyclictest32-21ksoftirqd/211:26:092
2511099120,3cyclictest0-21swapper/210:50:242
2510699129,2cyclictest5900-21ssh12:16:001
2510699129,2cyclictest3412-21head09:20:161
2510699129,2cyclictest21201-21head09:50:161
2510699129,2cyclictest18786-21grep11:25:231
2510699129,2cyclictest15019-21grep09:00:261
2510699129,2cyclictest12268-21tr07:50:141
2510699129,2cyclictest11673-21aten2_r7power_p12:20:151
2510699129,2cyclictest10891-21tr12:00:141
2510699129,2cyclictest10891-21tr12:00:141
2510699127,4cyclictest8245-21sh09:42:221
2510699127,4cyclictest5538-21ssh11:37:091
2510699127,4cyclictest29685-21ssh09:16:121
2510699123,3cyclictest25-21ksoftirqd/111:14:311
25106991211,1cyclictest0-21swapper/107:10:201
2510699120,11cyclictest0-21swapper/111:02:421
2510499129,2cyclictest5448-21ssh10:00:150
2510499129,2cyclictest3206-21sed07:30:170
2510499129,2cyclictest31088-21tr10:35:120
2510499129,2cyclictest27950-21grep07:15:150
2510499129,2cyclictest25608-21cut07:10:140
2510499129,2cyclictest24410-21grep11:10:140
2510499129,2cyclictest19199-21tail09:10:170
2510499129,2cyclictest17585-21cat11:45:000
2510499128,3cyclictest9391-21cut08:50:170
2510499128,3cyclictest20880-21sed08:05:280
2510499128,3cyclictest1258-21grep11:15:180
2510499122,8cyclictest11136-21/usr/sbin/munin09:25:250
25104991212,0cyclictest0-21swapper/012:23:460
2510499120,11cyclictest18996-21tail11:25:230
181772120,1sleep10-21swapper/109:48:411
2511299118,2cyclictest9507-21taskset09:43:223
2511299118,2cyclictest9329-21ssh12:37:483
2511299118,2cyclictest5879-21taskset08:43:353
2511299118,2cyclictest5879-21taskset08:43:353
2511299118,2cyclictest4055-21grep07:30:263
2511299118,2cyclictest39-21ksoftirqd/308:40:003
2511299118,2cyclictest29047-21ssh09:15:423
2511299118,2cyclictest28819-21ssh12:10:243
2511299118,2cyclictest27998-21grep07:15:153
2511299118,2cyclictest27844-21fgrep08:20:223
2511299118,2cyclictest25727-21tr11:30:143
2511299118,2cyclictest24386-21ssh10:11:113
2511299118,2cyclictest20326-21ls12:25:153
2511299118,2cyclictest19044-21diskstats09:10:163
2511299118,2cyclictest14361-21ssh11:23:143
2511299118,2cyclictest12366-21ls07:50:153
2511299116,2cyclictest225-21systemd-journal08:15:003
2511299114,7cyclictest25341-21apt-get07:10:133
2511099119,1cyclictest26816-21tr08:20:142
2511099118,2cyclictest7966-21ssh10:20:292
2511099118,2cyclictest32015-21ssh10:35:182
2511099118,2cyclictest3185-21grep09:20:142
2511099118,2cyclictest31696-21sh10:55:102
2511099118,2cyclictest27505-21cut12:10:142
2511099118,2cyclictest2548-21expr11:55:142
2511099118,2cyclictest23867-21turbostat08:15:002
2511099118,2cyclictest19805-21grep08:05:182
2511099118,2cyclictest18563-21rm11:45:152
2511099118,2cyclictest18544-21ssh11:05:502
2511099118,2cyclictest18262-21cat08:00:252
2511099118,2cyclictest15276-21ssh10:05:332
2511099118,2cyclictest11526-21acpi12:20:122
2511099111,9cyclictest0-21swapper/212:28:592
2510699119,1cyclictest2117-21grep08:35:171
2510699118,3cyclictest38150irq/131-snd_hda11:31:501
2510699118,2cyclictest31737-21grep07:20:261
2510699118,2cyclictest31373-21cat10:55:001
2510699118,2cyclictest30707-21ssh09:36:081
2510699118,2cyclictest14493-21tr07:55:141
2510699118,2cyclictest10479-21taskset08:51:411
2510699116,4cyclictest4039-21date07:30:261
2510699116,2cyclictest14200-21ssh09:26:131
2510699110,10cyclictest2593-21ssh11:15:321
2510499118,2cyclictest8256-21ssh09:42:220
2510499118,2cyclictest4589-21cut08:40:160
2510499118,2cyclictest4589-21cut08:40:160
2510499118,2cyclictest32417-21ssh09:56:390
2510499118,2cyclictest31347-21grep07:20:220
2510499118,2cyclictest28369-21ssh10:52:440
2510499118,2cyclictest2829-21ssh10:56:350
2510499118,2cyclictest27890-21cut12:10:160
2510499118,2cyclictest27239-21grep08:20:170
2510499118,2cyclictest26525-21ssh09:34:220
2510499118,2cyclictest24298-21wc10:30:260
2510499118,2cyclictest22479-21grep08:10:180
2510499118,2cyclictest2041-21cut08:35:170
2510499118,2cyclictest19685-21sh12:25:010
2510499118,2cyclictest18776-21head11:45:160
2510499118,2cyclictest17413-21ssh10:26:220
2510499118,2cyclictest15861-21cut10:45:160
2510499118,2cyclictest12974-21wc07:50:190
2511299108,1cyclictest32626-21awk08:30:193
2511299108,1cyclictest32514-21cut07:25:133
2511299108,1cyclictest24352-21grep08:15:133
2511299108,1cyclictest164142sleep307:59:063
2511299107,2cyclictest8017-21tr11:00:133
2511099108,2cyclictest31624-21ls07:20:262
2511099108,1cyclictest8077-21cat07:40:182
2511099108,1cyclictest5607-21grep07:35:172
2511099108,1cyclictest33912sleep208:38:082
2511099108,1cyclictest29474-21df08:25:152
2511099108,1cyclictest28462-21ssh09:55:002
2511099108,1cyclictest162762sleep207:57:212
2511099108,1cyclictest1056-21cat07:25:192
2511099107,2cyclictest31710-21cut08:30:132
2511099107,2cyclictest27279-21chrt07:13:022
2511099107,2cyclictest2680-21cut07:30:142
2511099107,2cyclictest24868-21tail08:15:152
2511099107,2cyclictest15109-21sed09:00:262
2511099107,2cyclictest11231-21wc08:55:152
2511099100,9cyclictest12491-21expr07:50:152
2510699108,1cyclictest8023-21grep07:40:181
2510699108,1cyclictest6628-21taskset07:35:351
2510699108,1cyclictest28318-21chrt08:21:041
2510699108,1cyclictest22150-21expr08:10:161
2510699108,1cyclictest20955-21seq08:06:091
2510699108,1cyclictest17492-21expr08:00:161
2510699107,2cyclictest6615-21ssh10:40:031
2510699107,2cyclictest26235-21sh08:20:001
2510699107,2cyclictest15921-21wget09:05:151
25106991010,0cyclictest0-21swapper/107:25:261
2510499108,1cyclictest3226-21ssh09:20:150
2510499108,1cyclictest10969-21chrt08:55:010
2510499107,2cyclictest32086-21sort08:30:150
2510499107,2cyclictest25505-21awk11:30:130
2510499107,2cyclictest16719-21grep09:05:190
251069997,1cyclictest4036-21cat08:40:141
251069997,1cyclictest4036-21cat08:40:141
251049997,1cyclictest14152-21tr09:00:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional