You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-24 - 01:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Sep 23, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
145362940,0sleep20-21swapper/211:27:052
2029529264,8sleep00-21swapper/007:07:380
2023728764,6sleep20-21swapper/207:06:522
2037228471,9sleep30-21swapper/307:08:373
2045327565,6sleep10-21swapper/107:09:401
112182630,0sleep10-21swapper/109:27:391
20587992519,6cyclictest32-21ksoftirqd/209:17:532
20587992519,6cyclictest32-21ksoftirqd/209:17:532
2058799250,25cyclictest0-21swapper/211:47:072
20589992320,2cyclictest6473-21/usr/sbin/munin09:25:243
2058299230,22cyclictest0-21swapper/012:35:110
20589992219,2cyclictest3531-21/usr/sbin/munin12:00:163
2058799220,2cyclictest0-21swapper/211:02:332
2058799211,1cyclictest13361-21diskmemload09:39:422
20583992118,2cyclictest4170-21/usr/sbin/munin12:20:181
2058299213,3cyclictest141rcu_preempt09:36:500
20582992119,1cyclictest27035-21/usr/sbin/munin11:35:280
2058299210,17cyclictest0-21swapper/010:05:310
2058299210,0cyclictest0-21swapper/010:49:180
20589992019,1cyclictest0-21swapper/310:39:533
20587992019,1cyclictest55850irq/129-enp2s0-12:05:182
20587992018,1cyclictest9789-21/usr/sbin/munin11:05:182
20583992018,2cyclictest55550irq/126-enp2s0-11:25:271
20583992018,1cyclictest8820-21/usr/sbin/munin10:05:131
224602190,0sleep30-21swapper/309:53:533
20587991917,1cyclictest25510-21/usr/sbin/munin10:15:112
20587991917,1cyclictest25510-21/usr/sbin/munin10:15:112
2058799190,1cyclictest0-21swapper/211:34:432
2058799190,18cyclictest0-21swapper/211:18:132
2058399190,1cyclictest0-21swapper/112:35:431
2058399190,17cyclictest0-21swapper/112:07:161
2058299190,0cyclictest0-21swapper/009:31:020
2058999180,17cyclictest0-21swapper/309:42:483
20587991817,1cyclictest0-21swapper/210:00:032
20587991817,1cyclictest0-21swapper/209:34:532
2058799180,18cyclictest0-21swapper/209:25:232
20583991816,1cyclictest2850-21/usr/sbin/munin11:40:201
2058399180,2cyclictest0-21swapper/107:35:001
2058399180,18cyclictest0-21swapper/110:51:431
2058399180,17cyclictest0-21swapper/109:10:101
2058399180,16cyclictest0-21swapper/111:46:421
2058299187,5cyclictest8411-21sendmail-msp09:00:000
2058299182,1cyclictest141rcu_preempt10:55:290
2058299180,2cyclictest0-21swapper/011:20:110
2058999173,9cyclictest983-21apt-get10:20:023
2058999170,17cyclictest0-21swapper/311:50:183
20587991716,1cyclictest0-21swapper/211:41:432
20587991716,1cyclictest0-21swapper/211:23:332
20587991716,1cyclictest0-21swapper/210:52:332
20587991716,1cyclictest0-21swapper/210:24:032
20587991716,1cyclictest0-21swapper/210:10:332
20587991716,1cyclictest0-21swapper/209:54:232
20587991716,1cyclictest0-21swapper/209:42:232
20587991716,1cyclictest0-21swapper/209:24:032
20587991715,1cyclictest24624-21/usr/sbin/munin09:55:182
2058799170,17cyclictest0-21swapper/212:20:232
2058799170,13cyclictest0-21swapper/210:06:172
20583991715,1cyclictest22145-21/usr/sbin/munin09:15:131
20583991715,1cyclictest22145-21/usr/sbin/munin09:15:131
2058299170,0cyclictest0-21swapper/007:35:140
2058999163,1cyclictest141rcu_preempt09:10:243
2058999162,11cyclictest18730-21ssh11:30:143
20589991614,1cyclictest28332-21/usr/sbin/munin12:15:233
20589991613,2cyclictest55850irq/129-enp2s0-09:55:203
2058999160,8cyclictest0-21swapper/310:34:273
2058799162,1cyclictest0-21swapper/210:42:452
20587991615,1cyclictest0-21swapper/212:30:032
20587991615,1cyclictest0-21swapper/212:27:132
20587991615,1cyclictest0-21swapper/210:59:532
20587991615,1cyclictest0-21swapper/209:12:232
2058799160,2cyclictest0-21swapper/211:59:232
2058799160,15cyclictest0-21swapper/212:11:532
20583991613,2cyclictest2553-21/usr/sbin/munin10:40:261
2058299160,1cyclictest0-21swapper/012:34:390
2058299160,16cyclictest0-21swapper/012:27:270
20589991514,1cyclictest0-21swapper/310:11:043
20587991515,0cyclictest0-21swapper/211:36:432
20587991512,2cyclictest28192-21/usr/sbin/munin07:25:152
2058799150,15cyclictest0-21swapper/212:38:132
2058399159,4cyclictest0-21swapper/110:45:151
2058399155,4cyclictest27246-21munin-run11:55:001
2058399152,12cyclictest15794-21if_err_enp2s009:30:201
20583991511,3cyclictest19833-21tr08:15:151
2058299150,1cyclictest0-21swapper/011:11:350
2058299150,0cyclictest0-21swapper/010:19:220
2058299150,0cyclictest0-21swapper/010:19:220
2058999141,7cyclictest0-21swapper/311:35:143
20589991411,2cyclictest3138-21grep08:45:253
20589991411,2cyclictest27023-21df11:15:173
20589991410,3cyclictest1754-21cut11:00:143
20589991410,3cyclictest16183-21ls09:30:233
20589991410,2cyclictest1272-21tail07:35:173
20587991410,3cyclictest16381-21wc08:05:282
2058799140,13cyclictest0-21swapper/209:47:582
2058399146,4cyclictest19066-21cut11:30:171
2058399141,9cyclictest14842-21apt-get08:05:151
20583991412,1cyclictest16182-21apt-get09:50:171
20583991410,3cyclictest4555-21grep07:40:251
20583991410,2cyclictest26770-21basename10:35:141
2058399140,3cyclictest0-21swapper/109:38:291
2058399140,13cyclictest0-21swapper/112:12:371
20582991414,0cyclictest0-21swapper/012:05:120
20582991412,1cyclictest24458-21ssh09:55:030
2058299140,2cyclictest0-21swapper/012:24:090
2058299140,13cyclictest11539-21tr09:05:140
2058999139,4cyclictest24341-21apache209:21:343
2058999131,2cyclictest13361-21diskmemload11:48:323
2058999131,1cyclictest0-21swapper/312:25:143
20589991310,2cyclictest806-21sed08:40:273
20589991310,2cyclictest3598-21tr11:20:213
20589991310,2cyclictest11972-21ssh10:45:193
2058999130,13cyclictest0-21swapper/311:10:133
2058799139,3cyclictest2935-21wc08:45:252
2058799139,2cyclictest28016-21sed08:30:232
2058799136,5cyclictest0-21swapper/210:34:252
20587991312,1cyclictest0-21swapper/207:20:002
2058399139,3cyclictest29210-21wc07:25:191
2058399138,4cyclictest4731-21wc12:00:211
20583991313,0cyclictest0-21swapper/111:55:021
2058399131,11cyclictest0-21swapper/112:15:141
20583991310,2cyclictest4936-21chrt09:24:101
20583991310,2cyclictest3982-21rm10:20:381
20583991310,2cyclictest14592-21grep12:25:261
2058399130,12cyclictest25070-21sh10:15:001
2058299139,3cyclictest1179-21chrt09:21:120
2058299139,2cyclictest9024-21ntpq09:45:190
20582991310,2cyclictest3234-21grep11:40:150
20582991310,2cyclictest26678-21tr11:15:160
20582991310,2cyclictest12004-21ssh09:28:160
2058299130,13cyclictest0-21swapper/011:28:290
2058299130,12cyclictest20082-21grep10:50:200
2058299130,12cyclictest0-21swapper/011:53:160
2058999129,2cyclictest9649-21grep10:05:153
2058999129,2cyclictest30095-21grep12:35:163
2058999129,2cyclictest27063-21expr10:15:223
2058999129,2cyclictest27063-21expr10:15:223
2058999129,2cyclictest26078-21tr10:55:153
2058999129,2cyclictest14560-21cron08:05:003
2058999129,2cyclictest13458-21cat12:05:243
2058999129,2cyclictest13347-21cron09:10:003
2058999129,2cyclictest1321-21sort10:00:183
2058999129,2cyclictest11177-21cut07:55:203
2058999128,3cyclictest22211-21ssh09:15:143
2058999128,3cyclictest22211-21ssh09:15:143
20589991211,1cyclictest0-21swapper/311:57:063
20589991210,2cyclictest39-21ksoftirqd/312:22:053
2058999120,2cyclictest0-21swapper/310:41:563
2058799129,2cyclictest19389-21wc08:15:002
2058799129,2cyclictest18314-21cut11:10:192
2058799129,2cyclictest11220-21tr10:45:152
2058799128,2cyclictest9829-21cat09:00:192
20587991211,1cyclictest0-21swapper/207:40:232
20587991210,1cyclictest0-21swapper/210:37:532
2058799120,11cyclictest0-21swapper/210:29:312
2058399129,2cyclictest4660-21grep08:50:181
2058399129,2cyclictest26581-21grep07:20:161
2058399129,2cyclictest26581-21grep07:20:161
2058399129,2cyclictest26403-21sed10:15:161
2058399129,2cyclictest26403-21sed10:15:161
2058399129,2cyclictest26121-21tr10:55:151
2058399129,2cyclictest12299-21sh10:25:391
2058399128,3cyclictest4356-21ssh10:01:501
2058399128,3cyclictest26408-21wc09:55:261
2058399128,3cyclictest23644-21grep07:15:141
2058399128,2cyclictest10791-21grep07:55:171
2058399124,7cyclictest9504-21runrttasks11:24:401
2058399121,9cyclictest38150irq/131-snd_hda08:25:131
20583991211,1cyclictest0-21swapper/108:10:141
2058399120,12cyclictest0-21swapper/112:30:131
2058299129,2cyclictest9996-21tr11:05:140
2058299129,2cyclictest28557-21tail11:55:170
2058299129,2cyclictest2755-21cut10:40:130
2058299129,2cyclictest24459-21grep08:25:130
2058299129,2cyclictest18595-21ssh11:49:450
2058299128,3cyclictest9024-21taskset10:24:370
2058299128,3cyclictest26047-21tr07:20:140
2058299128,3cyclictest26047-21tr07:20:140
2058299120,11cyclictest9155-21tr09:00:150
2058299120,11cyclictest19490-21rm09:51:330
2058299120,11cyclictest10892-21tail10:25:170
2058999118,2cyclictest8495-21hostname07:50:173
2058999118,2cyclictest4169-21awk07:40:213
2058999118,2cyclictest3823-21wc11:40:183
2058999118,2cyclictest29273-21wget08:35:143
2058999118,2cyclictest28222-21apache_processe07:25:123
2058999118,2cyclictest26830-21cut08:30:153
2058999118,2cyclictest24014-21ssh12:31:173
2058999118,2cyclictest22992-21ssh09:35:123
2058999118,2cyclictest21995-21sed12:10:303
2058999118,2cyclictest20611-21ntpq10:50:253
2058999118,2cyclictest16074-21tr08:05:243
2058999118,2cyclictest12789-21sh11:25:413
2058999118,2cyclictest11346-21grep11:05:223
2058999118,2cyclictest10168-21cut10:25:143
2058999112,6cyclictest24429-21apt-get08:25:133
2058999110,10cyclictest8372-21sh09:45:153
2058799118,2cyclictest7005-21grep08:55:202
2058799118,2cyclictest4165-21tr08:50:152
2058799118,2cyclictest28685-21tr12:15:142
2058799118,2cyclictest27455-21sed07:20:252
2058799118,2cyclictest27455-21sed07:20:252
2058799118,2cyclictest25132-21tail08:25:162
2058799118,2cyclictest22552-21wc07:10:252
2058799118,2cyclictest20683-21cut08:15:202
2058799118,2cyclictest19230-21rm11:50:052
2058799118,2cyclictest10212-21egrep07:55:142
2058799117,3cyclictest3715-21cut12:00:152
2058399118,2cyclictest6457-21egrep08:55:171
2058399118,2cyclictest6408-21grep07:45:201
2058399118,2cyclictest32286-21ssh11:18:331
2058399118,2cyclictest31714-21basename09:40:141
2058399118,2cyclictest27287-21df08:30:171
2058399118,2cyclictest22101-21ssh10:31:481
2058399118,2cyclictest21762-21head07:10:171
2058399118,2cyclictest21317-21ssh11:11:251
2058399118,2cyclictest1509-21basename08:45:111
2058399118,2cyclictest13480-21date09:10:001
2058399117,3cyclictest10115-21ssh11:05:141
2058299119,1cyclictest26551-21sh11:35:000
2058299118,2cyclictest7211-21ssh11:03:200
2058299118,2cyclictest5157-21ssh10:02:280
2058299118,2cyclictest4046-21cat12:00:160
2058299118,2cyclictest3711-21cut07:40:160
2058299118,2cyclictest344-21cat09:40:200
2058299118,2cyclictest32554-21chrt07:33:100
2058299118,2cyclictest31808-21ssh12:16:360
2058299118,2cyclictest28899-21cron08:35:000
2058299118,2cyclictest23055-21grep09:15:180
2058299118,2cyclictest23055-21grep09:15:180
2058299118,2cyclictest21267-21cut07:10:140
2058299118,2cyclictest20956-21grep12:10:210
2058299118,2cyclictest18881-21grep08:10:260
2058299118,2cyclictest17366-21wget10:10:170
2058299117,3cyclictest30332-21sh07:30:000
2058299116,2cyclictest225-21systemd-journal08:20:000
2058299112,6cyclictest0-21swapper/008:05:150
2058299110,8cyclictest0-21swapper/010:30:070
2058999108,1cyclictest9134-21expr09:00:153
2058999108,1cyclictest80322sleep308:55:473
2058999108,1cyclictest5508-21egrep07:45:143
2058999108,1cyclictest324582sleep307:31:573
2058999108,1cyclictest22959-21chrt07:13:053
2058999107,2cyclictest26597-21grep07:20:173
2058999107,2cyclictest26597-21grep07:20:173
2058999107,2cyclictest21465-21taskset08:17:053
2058999107,2cyclictest17894-21cut08:10:173
2058999103,3cyclictest225-21systemd-journal08:24:423
2058999102,7cyclictest4383-21sed08:50:163
2058999101,5cyclictest225-21systemd-journal07:20:003
2058799108,1cyclictest94442sleep207:50:352
2058799108,1cyclictest1600-21sed07:35:202
2058799108,1cyclictest1457-21date08:45:002
2058799108,1cyclictest14230-21chrt08:00:492
2058799107,2cyclictest30828-21grep07:30:162
2058799107,2cyclictest29436-21tr08:35:152
2058799107,2cyclictest11318-21grep09:05:152
2058799100,6cyclictest225-21systemd-journal07:47:472
2058399108,1cyclictest238512sleep108:21:591
2058399108,1cyclictest13446-21ls08:00:171
2058399108,1cyclictest1172-21chrt08:44:111
2058399107,2cyclictest7830-21grep07:50:131
2058399107,2cyclictest29359-21egrep08:35:151
2058399107,2cyclictest1458-21ssh11:00:071
2058399102,4cyclictest225-21systemd-journal07:37:571
2058399100,9cyclictest8188-21expr09:45:141
2058399100,9cyclictest28170-21cat11:35:201
2058299108,2cyclictest464-21which08:40:230
2058299108,1cyclictest72232sleep007:47:320
2058299108,1cyclictest30991-21chrt08:36:290
2058299108,1cyclictest26531-21sh10:35:040
2058299108,1cyclictest10507-21expr07:55:150
2058299107,2cyclictest5309-21expr08:50:260
2058299107,2cyclictest25557-21cron07:20:000
2058299107,2cyclictest23139-21ntp_states08:20:190
2058299107,2cyclictest1524-21dpkg08:45:120
2058299107,2cyclictest12527-21tr08:00:110
2058299100,9cyclictest13900-21ssh09:10:140
205879997,1cyclictest23907-21taskset08:22:412
205839996,2cyclictest8876-21apt09:00:131
205829997,1cyclictest7749-21aten2_r7power_e07:50:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional