You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 23:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Nov 28, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9010210679,9sleep10-21swapper/107:09:271
883329585,6sleep30-21swapper/307:07:093
827329251,35sleep20-21swapper/207:05:282
904528269,9sleep00-21swapper/007:09:510
107662660,0sleep20-21swapper/211:32:592
913999280,1cyclictest0-21swapper/212:35:162
9139992423,1cyclictest0-21swapper/210:10:102
913999240,2cyclictest0-21swapper/210:53:582
913999220,22cyclictest0-21swapper/211:10:192
913999220,22cyclictest0-21swapper/211:10:182
913499220,22cyclictest0-21swapper/109:34:381
914399210,17cyclictest0-21swapper/308:20:153
914399210,14cyclictest0-21swapper/308:05:133
913999204,16cyclictest0-21swapper/212:18:562
913999192,3cyclictest1957-21diskmemload11:01:392
9130991918,1cyclictest0-21swapper/009:23:460
9130991918,1cyclictest0-21swapper/009:23:450
913099191,3cyclictest141rcu_preempt11:25:290
9143991816,1cyclictest472-21avahi-daemon10:15:243
9143991812,2cyclictest2792-21sh10:05:003
9139991816,1cyclictest0-21swapper/209:30:112
9139991812,2cyclictest32-21ksoftirqd/211:23:112
913999180,17cyclictest0-21swapper/211:59:022
913999180,17cyclictest0-21swapper/210:39:062
9134991818,0cyclictest0-21swapper/110:19:411
9143991712,3cyclictest11036-21idleruntime-cro11:04:593
914399170,15cyclictest500-21in:imuxsock12:04:493
913999176,10cyclictest9129-21cyclictest12:33:222
913999170,17cyclictest13700-21users09:45:282
19512170,0chrt0-21swapper/012:19:130
9143991614,1cyclictest472-21avahi-daemon10:09:263
9143991613,2cyclictest472-21avahi-daemon10:24:223
914399160,16cyclictest0-21swapper/311:21:233
913499161,1cyclictest606-21ntp_states10:55:201
913499160,1cyclictest0-21swapper/111:20:131
913499160,16cyclictest0-21swapper/107:35:191
913099165,2cyclictest141rcu_preempt12:35:000
913099161,12cyclictest0-21swapper/007:30:130
9143991514,1cyclictest39-21ksoftirqd/311:18:583
9143991513,1cyclictest1957-21diskmemload11:55:153
9143991512,1cyclictest0-21swapper/308:45:223
913999150,14cyclictest0-21swapper/210:55:132
913499156,4cyclictest224-21systemd-journal09:04:381
913499155,6cyclictest224-21systemd-journal09:16:281
913499150,1cyclictest0-21swapper/111:58:051
914399145,5cyclictest224-21systemd-journal07:25:003
9143991411,2cyclictest2839-21sed10:30:233
913499142,3cyclictest224-21systemd-journal12:09:591
9134991410,3cyclictest25-21ksoftirqd/110:35:091
9134991410,3cyclictest12436-21grep09:45:181
9134991410,2cyclictest25838-21ls07:45:011
913499140,2cyclictest0-21swapper/109:43:491
9130991411,1cyclictest0-21swapper/011:06:290
913099140,2cyclictest0-21swapper/011:58:410
913099140,13cyclictest0-21swapper/009:10:560
914399132,7cyclictest1148-21ssh11:25:203
9143991310,2cyclictest25317-21grep07:40:283
9143991310,2cyclictest19952-21tr10:45:153
9143991310,2cyclictest1950-21grep09:35:283
9143991310,2cyclictest15545-21grep10:40:263
9143991310,2cyclictest10400-21ssh09:15:273
914399130,13cyclictest0-21swapper/312:13:453
913999139,3cyclictest24429-21sed08:45:262
9139991310,2cyclictest26111-21taskset10:23:352
9139991310,2cyclictest20706-21sh09:25:112
9139991310,2cyclictest19176-21cat08:35:232
9139991310,2cyclictest10902-21grep12:00:182
913999130,13cyclictest0-21swapper/209:12:022
913499139,3cyclictest29134-21wc10:00:001
913499139,2cyclictest27509-21grep08:55:131
913499139,2cyclictest17592-21grep11:10:161
913499139,2cyclictest17592-21grep11:10:151
913499138,4cyclictest61750irq/125-enp2s0-09:54:141
913499138,3cyclictest20227-21cron12:35:001
913499137,5cyclictest1378-21ntpd11:38:001
913499135,2cyclictest224-21systemd-journal08:30:281
9134991310,2cyclictest6191-21grep11:00:181
9134991310,2cyclictest387-21grep09:35:171
9134991310,2cyclictest3687-21tr08:05:161
9134991310,2cyclictest31131-21grep07:55:121
9134991310,2cyclictest28712-21head10:25:191
9134991310,2cyclictest19592-21grep07:30:161
9134991310,2cyclictest17981-21grep08:35:121
913499130,11cyclictest0-21swapper/111:28:591
913099139,3cyclictest28191-21perf07:50:000
913099138,3cyclictest0-21swapper/010:28:370
913099131,11cyclictest6165-21ssh09:40:140
9130991311,1cyclictest10865-21cut12:00:180
9130991310,2cyclictest17594-21wc07:25:200
913099130,12cyclictest0-21swapper/011:40:010
221132130,1sleep2311rcuc/209:53:572
914399129,2cyclictest8901-21grep08:15:173
914399129,2cyclictest6472-21sort08:10:193
914399129,2cyclictest567-21sed09:05:193
914399129,2cyclictest3251-21sed09:10:153
914399129,2cyclictest30526-21tr11:50:143
914399129,2cyclictest30106-21cut09:00:233
914399129,2cyclictest30095-21grep12:15:163
914399129,2cyclictest27935-21wc09:30:243
914399129,2cyclictest24035-21basename11:45:013
914399129,2cyclictest21403-21sort12:35:193
914399129,2cyclictest13913-21grep07:19:593
914399128,3cyclictest1681-21grep08:00:183
914399124,6cyclictest14370-21sh11:06:423
914399124,3cyclictest17456-21cron08:35:003
914399123,5cyclictest0-21swapper/310:25:143
914399120,11cyclictest13873-21ls11:35:203
913999129,3cyclictest26458-21cat08:50:222
913999129,2cyclictest8389-21cat09:15:002
913999129,2cyclictest24358-21cat09:55:222
913999128,3cyclictest657-21taskset07:59:312
913999127,3cyclictest0-21swapper/211:50:282
913999124,4cyclictest224-21systemd-journal10:30:162
913999122,9cyclictest0-21swapper/210:00:152
9139991210,1cyclictest1520-21grep09:35:242
913999120,3cyclictest0-21swapper/211:42:122
913999120,11cyclictest1957-21diskmemload10:18:132
913499129,2cyclictest8593-21taskset11:30:271
913499129,2cyclictest6601-21head08:10:211
913499129,2cyclictest5187-21grep10:05:271
913499129,2cyclictest30284-21grep07:50:311
913499129,2cyclictest19276-21turbostat10:45:001
913499129,2cyclictest1755-21sed08:00:171
913499128,3cyclictest8036-21ssh09:15:001
913499122,7cyclictest1957-21diskmemload10:50:151
913499122,6cyclictest2622-21sendmail12:20:001
913099129,2cyclictest9542-21cut09:15:190
913099129,2cyclictest4127-21grep10:05:180
913099129,2cyclictest3216-21wc10:30:270
913099129,2cyclictest31532-21cat10:55:000
913099129,2cyclictest30110-21tr09:00:230
913099129,2cyclictest2651-21grep08:00:260
913099129,2cyclictest26096-21wc11:45:340
913099129,2cyclictest25836-21grep08:50:170
913099129,2cyclictest17668-21tr12:05:230
913099129,2cyclictest14658-21grep11:35:290
913099128,3cyclictest6176-21ssh12:22:060
913099128,3cyclictest30453-21chrt07:51:100
913099128,3cyclictest26319-21ssh11:19:180
913099128,3cyclictest25774-21taskset07:44:480
913099128,3cyclictest19638-21chrt08:37:300
913099128,3cyclictest15234-21wc08:29:590
913099125,6cyclictest22925-21munin-run09:55:010
913099123,5cyclictest224-21systemd-journal10:00:070
913099123,3cyclictest13-21ksoftirqd/012:12:370
913099122,7cyclictest353-21df_inode09:05:160
913099121,5cyclictest32224-21ssh09:35:000
913099120,12cyclictest0-21swapper/012:30:170
913099120,11cyclictest29679-21ssh11:21:180
913099120,11cyclictest25468-21taskset10:22:530
914399119,1cyclictest57072sleep312:21:323
914399118,2cyclictest9751-21ssh09:43:083
914399118,2cyclictest9583-21tune2fs12:25:153
914399118,2cyclictest6845-21ssh11:30:153
914399118,2cyclictest31787-21ssh10:55:113
914399118,2cyclictest30735-21ssh10:54:163
914399118,2cyclictest30495-21taskset07:51:423
914399118,2cyclictest29263-21chrt08:56:043
914399118,2cyclictest27495-21cat07:45:263
914399118,2cyclictest26410-21ssh09:56:573
914399118,2cyclictest25197-21ssh11:45:203
914399118,2cyclictest22444-21apt-config08:45:003
914399118,2cyclictest21317-21sh09:53:033
914399118,2cyclictest20524-21ssh09:25:013
914399118,2cyclictest19866-21ssh12:34:383
914399118,2cyclictest19746-21chrt12:07:053
914399118,2cyclictest15834-21cat09:20:193
914399118,2cyclictest15834-21cat09:20:193
914399118,2cyclictest13119-21cut08:25:143
914399118,2cyclictest10486-21uname07:10:223
914399114,3cyclictest22068-21dump-pmu-power11:15:013
914399114,3cyclictest22068-21dump-pmu-power11:15:013
913999119,2cyclictest8700-21rm09:41:162
913999119,2cyclictest3670-21rm11:26:452
913999119,2cyclictest22873-21egrep12:10:142
913999119,2cyclictest15473-21grep09:20:172
913999119,2cyclictest15473-21grep09:20:172
913999119,2cyclictest14875-21rm11:35:422
913999119,2cyclictest11901-21cat11:05:152
913999119,1cyclictest7221-21smart_sda08:10:292
913999119,1cyclictest32679-21chrt10:29:032
913999119,1cyclictest2840-21taskset12:20:072
913999119,1cyclictest28233-21head08:55:172
913999119,1cyclictest24139-21cat11:15:252
913999119,1cyclictest14659-21cat10:40:182
913999119,1cyclictest11914-21gzip08:20:242
913999119,1cyclictest10066-21cut12:25:182
913999118,3cyclictest22180-21chrt08:44:252
913999118,2cyclictest17018-21df12:05:172
913999118,2cyclictest13029-21cut08:25:122
913999118,1cyclictest25918-21which11:45:312
913999115,5cyclictest32507-21grep09:05:152
913999113,5cyclictest500-21in:imuxsock07:14:402
913999111,2cyclictest0-21swapper/210:45:142
913999110,10cyclictest0-21swapper/207:30:122
913499118,2cyclictest9865-21df07:10:161
913499118,2cyclictest748-21ssh10:02:261
913499118,2cyclictest6974-21ssh12:23:081
913499118,2cyclictest31742-21ssh11:50:251
913499118,2cyclictest28257-21cat07:50:001
913499118,2cyclictest26550-21ssh12:12:301
913499118,2cyclictest25718-21ssh09:29:331
913499118,2cyclictest24245-21ssh11:45:021
913499118,2cyclictest23970-21ssh12:36:471
913499118,2cyclictest22138-21chrt08:43:511
913499118,2cyclictest21395-21ssh11:41:091
913499118,2cyclictest19846-21grep10:45:161
913499118,2cyclictest14866-21grep07:20:181
913499118,2cyclictest14030-21ssh10:13:401
913499118,2cyclictest13792-21ssh12:02:251
913499118,2cyclictest13050-21cut08:25:121
913499118,2cyclictest11584-21aten2_r7power_p11:05:141
913499117,3cyclictest24157-21cat11:15:251
913499112,8cyclictest8452-21cron12:29:551
913499110,9cyclictest0-21swapper/110:32:021
913099119,1cyclictest22019-21grep07:35:170
913099119,1cyclictest15564-21grep10:40:260
913099118,2cyclictest8980-21grep12:25:130
913099118,2cyclictest7611-21sh10:35:060
913099118,2cyclictest7372-21cut11:30:170
913099118,2cyclictest6202-21cut11:00:190
913099118,2cyclictest396-21wc10:55:190
913099118,2cyclictest30551-21cut11:50:150
913099118,2cyclictest23974-21wc08:45:210
913099118,2cyclictest23938-21sort09:55:180
913099118,2cyclictest20368-21cut10:45:180
913099118,2cyclictest20362-21tr08:40:140
913099118,2cyclictest17656-21ssh10:15:470
913099118,2cyclictest17306-21rm11:10:130
913099118,2cyclictest17306-21rm11:10:120
913099118,2cyclictest16122-21chrt07:23:160
913099118,2cyclictest1436-21ssh09:35:230
913099118,2cyclictest11862-21wget07:15:150
913099118,2cyclictest11668-21ssh09:45:110
913099118,2cyclictest10725-21cut08:20:160
913099118,2cyclictest10578-21sort10:10:200
913099117,3cyclictest20833-21ssh09:25:140
913099113,2cyclictest224-21systemd-journal08:05:280
913099110,10cyclictest31022-21bc07:55:100
914399108,1cyclictest9404-21ssh10:10:083
914399108,1cyclictest21745-21expr07:35:153
914399108,1cyclictest18403-21cut08:35:133
914399107,2cyclictest87592sleep310:35:173
914399107,2cyclictest31470-21ls07:55:123
914399107,2cyclictest25334-21chrt08:50:143
914399107,2cyclictest19641-21grep07:30:163
914399107,2cyclictest16794-21grep07:25:143
914399107,2cyclictest11851-21ssh09:45:153
913999109,1cyclictest9237-21cat08:15:202
913999109,1cyclictest4168-21hddtemp_smartct10:05:172
913999109,1cyclictest22715-21cat07:35:252
913999109,1cyclictest17900-21readlink07:25:222
913999108,1cyclictest16697-21mailstats08:30:292
913999108,1cyclictest15809-21grep07:20:282
913999108,1cyclictest12851-21tr07:15:202
913999107,2cyclictest25702-21taskset07:43:562
913999101,7cyclictest1291-21runrttasks08:05:162
913499108,1cyclictest26922-21taskset08:51:301
913499108,1cyclictest14539-21ssh09:20:011
913499108,1cyclictest14539-21ssh09:20:001
913499107,2cyclictest9838-21grep08:15:261
913499107,2cyclictest23546-21chrt10:20:321
913499107,2cyclictest11863-21grep07:15:151
913499104,5cyclictest519-21gmain08:45:021
913499101,2cyclictest0-21swapper/107:25:131
913099108,1cyclictest10184-21chrt08:18:290
913099107,2cyclictest9343-21chrt07:10:110
913099107,2cyclictest7164-21tr08:10:270
913099107,2cyclictest27541-21cut08:55:130
913099107,2cyclictest15721-21expr08:30:150
91399998,1cyclictest30561-21taskset07:52:332
91399998,1cyclictest30336-21cat09:00:232
91399998,1cyclictest27190-21ntpq07:45:202
91399997,1cyclictest9562sleep208:00:052
91349997,1cyclictest1248-21cat09:05:261
91349996,2cyclictest10581-21basename08:20:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional