You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-15 - 16:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Jun 15, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
867329282,6sleep00-21swapper/019:09:150
866129166,9sleep30-21swapper/319:09:063
849529072,14sleep10-21swapper/119:06:551
859328975,10sleep20-21swapper/219:08:122
101082700,0sleep20-21swapper/222:32:032
310402630,1sleep30-21swapper/322:45:103
240502520,1sleep339-21ksoftirqd/321:42:243
311992510,0sleep30-21swapper/323:04:343
881799280,1cyclictest0-21swapper/121:04:371
8817992624,1cyclictest29923-21cat00:20:221
8817992624,1cyclictest29923-21cat00:20:211
882299240,24cyclictest0-21swapper/223:12:212
882499231,20cyclictest0-21swapper/320:45:123
882499230,21cyclictest0-21swapper/322:31:363
882499220,22cyclictest0-21swapper/300:24:243
882499220,22cyclictest0-21swapper/300:24:233
881799221,21cyclictest0-21swapper/123:47:541
881799220,21cyclictest22825-21sendmail_mailqu00:35:231
881799210,21cyclictest0-21swapper/123:30:581
8824992019,1cyclictest0-21swapper/321:38:303
881799200,20cyclictest0-21swapper/122:37:591
8813992017,2cyclictest13-21ksoftirqd/000:35:220
8824991912,6cyclictest0-21swapper/322:42:123
8824991912,6cyclictest0-21swapper/322:42:123
882499180,16cyclictest0-21swapper/321:25:303
882299187,11cyclictest32-21ksoftirqd/221:15:232
8822991816,2cyclictest32-21ksoftirqd/223:39:322
882299180,18cyclictest0-21swapper/223:52:522
881799181,17cyclictest0-21swapper/123:03:041
881799181,16cyclictest0-21swapper/123:20:131
881799180,4cyclictest0-21swapper/121:31:581
881799180,3cyclictest0-21swapper/119:20:131
881799180,1cyclictest0-21swapper/121:11:121
881799180,17cyclictest0-21swapper/123:29:351
881799180,16cyclictest0-21swapper/100:34:391
882499172,1cyclictest0-21swapper/300:32:433
881799170,15cyclictest0-21swapper/122:31:361
882499169,5cyclictest0-21swapper/300:25:143
8824991615,1cyclictest0-21swapper/323:16:383
882499160,15cyclictest0-21swapper/319:40:133
882299165,10cyclictest22721-21/usr/sbin/munin22:40:202
882299165,10cyclictest22721-21/usr/sbin/munin22:40:202
882299160,15cyclictest0-21swapper/221:51:442
882299160,14cyclictest0-21swapper/222:15:192
881799165,8cyclictest11926-21wc21:55:001
8817991615,1cyclictest0-21swapper/122:51:451
881799160,2cyclictest0-21swapper/100:15:471
881799160,16cyclictest0-21swapper/123:35:271
881799160,16cyclictest0-21swapper/121:44:001
881799160,13cyclictest0-21swapper/120:45:121
881399161,8cyclictest7760-21tune2fs22:50:150
881399161,1cyclictest0-21swapper/022:42:140
881399161,1cyclictest0-21swapper/022:42:130
882499156,8cyclictest13790-21apache223:26:013
8824991514,1cyclictest0-21swapper/321:14:383
882499150,15cyclictest0-21swapper/300:09:423
8822991511,3cyclictest32-21ksoftirqd/223:35:002
882299150,1cyclictest0-21swapper/222:09:522
881799156,4cyclictest13590-21/usr/sbin/munin21:55:201
881799150,1cyclictest0-21swapper/122:45:121
8813991512,2cyclictest8804-21cyclictest00:10:030
8813991511,3cyclictest13-21ksoftirqd/023:48:300
881399150,11cyclictest0-21swapper/023:25:180
882499141,9cyclictest0-21swapper/320:25:133
8824991412,1cyclictest0-21swapper/323:45:093
8824991412,1cyclictest0-21swapper/323:39:033
882499140,1cyclictest0-21swapper/322:18:353
882499140,1cyclictest0-21swapper/322:12:423
882499140,1cyclictest0-21swapper/321:53:093
882299147,6cyclictest0-21swapper/220:30:132
882299141,3cyclictest141rcu_preempt21:35:002
8822991411,2cyclictest47472sleep220:07:062
8822991410,3cyclictest17355-21grep19:25:172
882299140,5cyclictest0-21swapper/223:23:222
882299140,12cyclictest0-21swapper/223:04:142
881799145,5cyclictest10775-21df21:15:161
881799142,12cyclictest0-21swapper/121:37:261
8817991410,4cyclictest0-21swapper/122:16:361
881399147,6cyclictest6510-21idleruntime-cro22:50:010
881399147,1cyclictest23664-21grep22:20:270
881399141,6cyclictest32642-21munin-run20:00:000
8813991411,2cyclictest12705-21wc21:35:230
8813991410,2cyclictest9245-21grep20:15:270
8813991410,2cyclictest13233-21tr00:30:130
25132140,0sleep00-21swapper/000:04:540
882499132,10cyclictest0-21swapper/322:38:313
8824991312,1cyclictest0-21swapper/321:55:213
8824991311,1cyclictest0-21swapper/300:39:533
8824991310,2cyclictest29798-21cut22:05:183
882499130,7cyclictest0-21swapper/300:12:423
882499130,1cyclictest0-21swapper/323:44:313
882499130,13cyclictest0-21swapper/322:25:373
882499130,13cyclictest0-21swapper/321:17:433
882299138,2cyclictest15978-21ssh21:56:412
882299133,9cyclictest6745-31find00:06:382
882299132,1cyclictest141rcu_preempt00:18:492
8822991310,2cyclictest29214-21awk21:45:212
8822991310,2cyclictest24406-21cat20:50:012
8822991310,2cyclictest23576-21taskset23:19:182
8822991310,2cyclictest13660-21wc20:25:222
882299130,13cyclictest0-21swapper/222:57:552
882299130,13cyclictest0-21swapper/222:39:272
882299130,12cyclictest404-21seq00:22:092
882299130,12cyclictest404-21seq00:22:082
882299130,12cyclictest18879-21grep21:20:122
882299130,0cyclictest0-21swapper/200:10:312
881799138,4cyclictest20795-21latency20:40:191
881799137,5cyclictest537-21snmpd00:26:151
881799136,6cyclictest537-21snmpd23:54:521
8817991310,2cyclictest2604-21cron20:05:001
8817991310,2cyclictest23067-21ntpq22:20:211
8817991310,2cyclictest13537-21sh19:20:011
8813991310,2cyclictest4499-21grep20:05:280
8813991310,2cyclictest29052-21munin-plugin-st21:00:000
8813991310,2cyclictest23415-21cron23:00:010
8813991310,2cyclictest2308-21cron21:30:000
8813991310,2cyclictest19741-21tr23:55:170
881399130,12cyclictest20622-21grep00:15:150
58332130,0sleep00-21swapper/000:05:590
882499129,2cyclictest930-21grep21:05:263
882499129,2cyclictest31628-21head19:55:213
882499129,2cyclictest22852-21cat22:20:193
882499129,2cyclictest21775-21cut19:35:173
882499129,2cyclictest20329-21head19:30:273
882499129,2cyclictest16535-21cut19:25:123
882499129,2cyclictest13532-21cron19:20:013
882499126,3cyclictest10176-21wc20:20:143
882499123,7cyclictest23753-21ssh21:22:543
8824991210,1cyclictest28525-21cut00:00:213
8824991210,1cyclictest25814-21grep23:20:183
8824991210,1cyclictest10701-21grep19:10:253
882499120,11cyclictest10750-21sh22:51:483
882299129,2cyclictest29036-21cron21:00:002
882299129,2cyclictest2678-21df23:45:162
882299129,2cyclictest2227-21tr21:10:152
882299129,2cyclictest19877-21grep21:40:152
882299129,2cyclictest14114-21tr19:20:142
882299129,2cyclictest14025-21cut00:30:172
882299129,2cyclictest1263-21head23:05:232
882299128,3cyclictest19714-21head23:55:182
882299126,2cyclictest225-21systemd-journal00:05:002
882299122,1cyclictest141rcu_preempt19:10:172
882299120,9cyclictest0-21swapper/200:35:172
882299120,12cyclictest0-21swapper/222:51:432
882299120,12cyclictest0-21swapper/221:38:232
882299120,11cyclictest22695-21grep19:35:262
881799129,2cyclictest18523-21grep23:15:261
881799124,2cyclictest475-21in:imuxsock19:44:341
881799121,11cyclictest0-21swapper/123:57:291
8817991211,1cyclictest0-21swapper/100:06:551
8817991210,1cyclictest30239-21cut22:25:131
8817991210,1cyclictest1406-21diskmemload21:20:221
881799120,12cyclictest0-21swapper/123:08:091
881799120,11cyclictest29332-21grep21:45:231
881399129,2cyclictest9714-21cron21:15:000
881399129,2cyclictest6872-21grep22:10:270
881399129,2cyclictest5890-21ssh22:30:070
881399129,2cyclictest3721-21chrt21:50:010
881399129,2cyclictest25265-21grep23:20:150
881399129,2cyclictest21044-21tr22:00:130
881399129,2cyclictest19634-21wc23:35:250
881399129,2cyclictest18605-21grep23:15:260
881399129,2cyclictest14819-21wc22:15:210
881399129,2cyclictest12193-21tail19:15:170
881399129,2cyclictest10803-21sed20:20:170
881399129,2cyclictest10773-21grep23:50:150
881399128,3cyclictest25656-21wc23:00:240
881399128,3cyclictest22320-21wget20:45:120
881399128,3cyclictest21276-21tr19:35:150
881399128,3cyclictest19024-21grep20:35:270
881399128,3cyclictest15533-21grep19:20:260
881399126,2cyclictest225-21systemd-journal20:30:000
8813991212,0cyclictest0-21swapper/021:56:520
8813991212,0cyclictest0-21swapper/021:22:380
8813991210,1cyclictest15472-21ssh22:35:180
881399120,9cyclictest0-21swapper/023:44:500
881399120,11cyclictest10618-21sed23:30:170
881399120,11cyclictest10525-21tr21:15:160
882499119,1cyclictest4043-21wc21:30:193
882499119,1cyclictest15782-21wc22:55:143
882499118,2cyclictest9292-21sh23:30:063
882499118,2cyclictest7866-21tr20:15:163
882499118,2cyclictest6568-21wc20:10:213
882499118,2cyclictest30171-21taskset19:54:023
882499118,2cyclictest29554-21tr21:00:153
882499118,2cyclictest27764-21ssh21:45:073
882499118,2cyclictest24734-21ssh00:17:213
882499118,2cyclictest2181-21grep20:00:263
882499118,2cyclictest2168-21ssh23:05:413
882499118,2cyclictest20995-21grep22:00:133
882499118,2cyclictest19184-21tr23:55:153
882499118,2cyclictest14163-21ssh23:51:413
882499118,2cyclictest12528-21ssh23:12:133
882499110,10cyclictest18124-21grep20:35:183
882299118,2cyclictest8408-21sed20:15:182
882299118,2cyclictest7644-21ssh22:10:572
882299118,2cyclictest5476-21grep20:10:142
882299118,2cyclictest31859-21grep22:45:152
882299118,2cyclictest30448-21ssh23:42:102
882299118,2cyclictest30067-21grep21:00:182
882299118,2cyclictest28817-21tune2fs19:50:162
882299118,2cyclictest27657-21grep21:25:172
882299118,2cyclictest24597-21sed19:40:212
882299118,2cyclictest20543-21ssh22:00:052
882299118,2cyclictest18895-21expr19:30:162
882299118,2cyclictest10219-21ssh00:28:152
882299115,2cyclictest225-21systemd-journal19:15:012
882299110,2cyclictest0-21swapper/222:26:212
882299110,1cyclictest0-21swapper/223:25:052
882299110,10cyclictest19502-21taskset20:39:512
881799119,1cyclictest16562-21if_enp2s022:55:181
881799119,1cyclictest10004-21grep23:10:251
881799118,3cyclictest9144-21rm22:12:081
881799118,2cyclictest9445-21cat19:10:131
881799118,2cyclictest720-21cat21:05:261
881799118,2cyclictest5895-21grep20:10:161
881799118,2cyclictest4989-21basename20:10:001
881799118,2cyclictest32623-21cron20:00:001
881799118,2cyclictest30422-21ssh23:42:091
881799118,2cyclictest28412-21ps21:25:231
881799118,2cyclictest26255-21grep20:50:281
881799118,2cyclictest16702-21sed20:30:271
881799118,2cyclictest13552-21cat00:10:281
881799118,2cyclictest10701-21grep20:20:171
881799117,4cyclictest28265-21cat00:00:201
881399118,2cyclictest5974-21ssh00:25:230
881399118,2cyclictest4334-21gzip21:30:230
881399118,2cyclictest32296-21ssh22:06:250
881399118,2cyclictest30894-21grep22:25:170
881399118,2cyclictest30591-21wc21:00:230
881399118,2cyclictest28738-21ssh21:45:190
881399118,2cyclictest2853-21ssh00:24:030
881399118,2cyclictest2853-21ssh00:24:020
881399118,2cyclictest25762-21cut20:50:220
881399118,2cyclictest24653-21proc_pri19:40:210
881399118,2cyclictest24319-21ssh21:42:360
881399118,2cyclictest1435-21ssh23:05:240
881399118,2cyclictest10265-21ssh23:10:290
881399117,3cyclictest28773-21grep19:50:160
881399110,3cyclictest0-21swapper/019:10:140
882499108,1cyclictest2877-21apache_processe20:05:143
882499108,1cyclictest28193-21wc20:55:213
882499108,1cyclictest25857-21cut19:45:143
882499107,2cyclictest21601-21taskset20:41:023
882499107,2cyclictest16600-21sed20:30:273
882499107,2cyclictest14485-21sort19:20:163
882499104,2cyclictest225-21systemd-journal20:52:113
882299108,1cyclictest26260-21df19:45:162
882299108,1cyclictest1386-21cat20:00:182
882299108,1cyclictest10368-21expr20:20:152
882299107,2cyclictest31809-21wget21:05:142
882299107,2cyclictest22695-21basename22:20:182
882299107,2cyclictest20953-21awk20:40:202
882299100,9cyclictest24698-21cut20:50:142
881799108,2cyclictest27520-21chrt19:45:291
881799108,1cyclictest29034-21cron21:00:001
881799108,1cyclictest21870-21wc22:00:181
881799108,1cyclictest21211-21grep19:35:141
881799108,1cyclictest18985-21expr19:30:161
881799108,1cyclictest17608-21egrep20:35:161
881799108,1cyclictest12675-21grep20:25:151
881799107,3cyclictest300942sleep119:53:021
881799107,2cyclictest24795-21who22:40:271
881799107,2cyclictest24795-21who22:40:261
881799106,3cyclictest28819-21rm22:05:021
881399108,1cyclictest6103-21if_enp0s31f620:10:170
881399108,1cyclictest26913-21ntpq19:45:210
881399108,1cyclictest20106-21cstates20:40:150
881399108,1cyclictest197912sleep019:30:220
881399107,2cyclictest2473-21chrt20:03:230
881399107,2cyclictest16360-21cut19:25:110
881399107,2cyclictest15773-21expr20:30:170
881399102,7cyclictest1394-21idleruntime-cro21:10:010
88229998,1cyclictest31878-21sendmail_mailtr19:55:232
88179998,1cyclictest8890-21grep20:15:231
88179997,1cyclictest16265-21aten2_r7power_c19:25:101
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional