You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 23:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jun 28, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12525210086,9sleep10-21swapper/107:06:231
1279529986,9sleep30-21swapper/307:09:553
1066928269,9sleep00-21swapper/007:05:050
1265027867,6sleep20-21swapper/207:08:012
171772570,0sleep10-21swapper/111:58:021
12892992826,1cyclictest0-21swapper/111:55:011
12898992521,2cyclictest5898-21diskmemload10:15:183
1289699240,23cyclictest29791-21ssh12:25:312
1289699220,1cyclictest0-21swapper/210:20:162
12889992216,5cyclictest22254-21unixbench_singl07:25:270
1289899215,16cyclictest0-21swapper/312:29:403
1289899211,1cyclictest0-21swapper/312:00:143
1289899210,21cyclictest0-21swapper/307:25:143
12892992120,1cyclictest0-21swapper/110:24:491
12889992119,1cyclictest5898-21diskmemload10:17:340
1289899208,7cyclictest0-21swapper/310:55:153
12898992017,2cyclictest5898-21diskmemload11:04:423
12898992016,3cyclictest28351-21head11:15:373
1289899200,19cyclictest0-21swapper/311:45:163
1289699200,19cyclictest0-21swapper/209:28:442
1289299200,1cyclictest141rcu_preempt11:10:011
12889992017,3cyclictest13-21ksoftirqd/011:33:180
12889992014,4cyclictest28350-21grep11:15:370
12889992011,8cyclictest0-21swapper/008:40:150
12889992011,8cyclictest0-21swapper/008:35:190
165672190,0sleep30-21swapper/310:10:533
12898991917,2cyclictest0-21swapper/309:50:183
12892991918,1cyclictest0-21swapper/110:45:051
12892991916,2cyclictest25-21ksoftirqd/109:20:011
12889991917,1cyclictest5898-21diskmemload12:39:440
12889991914,4cyclictest0-21swapper/007:25:000
12898991815,3cyclictest39-21ksoftirqd/311:14:113
12898991814,3cyclictest0-21swapper/309:10:003
1288999186,6cyclictest4525-21awk08:00:010
12889991816,1cyclictest0-21swapper/012:30:150
12889991816,1cyclictest0-21swapper/009:22:060
12889991811,5cyclictest0-21swapper/010:50:390
1289899176,8cyclictest0-21swapper/311:06:393
12898991715,2cyclictest0-21swapper/310:42:533
12898991715,1cyclictest5898-21diskmemload10:50:393
12898991714,2cyclictest5898-21diskmemload10:02:043
12898991714,2cyclictest0-21swapper/311:29:393
12898991714,1cyclictest0-21swapper/312:19:483
12898991713,3cyclictest20290-21munin-run07:25:003
1289699173,1cyclictest141rcu_preempt12:36:332
12896991716,1cyclictest0-21swapper/211:58:152
1289299173,4cyclictest2467-21apache_volume10:00:181
1289299172,2cyclictest141rcu_preempt09:30:001
1289299172,11cyclictest9905-21latency_hist09:40:001
1289299170,16cyclictest0-21swapper/112:02:111
1288999177,5cyclictest0-21swapper/007:40:150
1288999177,4cyclictest0-21swapper/010:24:280
1288999176,9cyclictest0-21swapper/010:40:230
12889991716,1cyclictest0-21swapper/008:30:160
12889991714,2cyclictest5898-21diskmemload11:14:110
12889991710,6cyclictest0-21swapper/011:37:370
12889991710,5cyclictest0-21swapper/011:54:200
1289899167,6cyclictest1660-21cpuspeed_turbos09:00:183
1289899166,5cyclictest17218-21/usr/sbin/munin09:45:213
1289899166,10cyclictest0-21swapper/310:07:163
1289899163,12cyclictest28196-21ssh12:05:173
1289899162,3cyclictest39-21ksoftirqd/310:20:113
12898991614,1cyclictest5898-21diskmemload11:39:123
12898991613,2cyclictest0-21swapper/308:30:163
12898991613,1cyclictest31505-21switchtime07:45:283
12898991612,3cyclictest17703-21sendmail09:20:003
1289699162,13cyclictest0-21swapper/209:45:442
1289699161,2cyclictest21528-21kworker/u8:211:47:142
1289699160,1cyclictest0-21swapper/211:53:372
1289699160,1cyclictest0-21swapper/211:39:212
1289699160,13cyclictest0-21swapper/212:33:492
12892991614,2cyclictest26291-21latency_hist08:45:001
1289299160,2cyclictest0-21swapper/109:20:131
1288999162,7cyclictest9818-21grep09:40:000
12889991614,1cyclictest5898-21diskmemload11:46:520
12889991614,1cyclictest5898-21diskmemload10:57:090
12889991614,1cyclictest0-21swapper/010:02:040
12889991613,2cyclictest0-21swapper/008:15:260
1289899155,5cyclictest224-21systemd-journal12:34:553
12898991512,2cyclictest5898-21diskmemload09:11:343
12898991512,1cyclictest39-21ksoftirqd/310:45:103
12898991511,3cyclictest5898-21diskmemload10:33:173
1289699151,3cyclictest0-21swapper/209:50:172
12896991512,2cyclictest14283-21sendmail-msp08:20:002
12896991512,2cyclictest0-21swapper/207:40:242
12896991512,0cyclictest0-21swapper/211:40:222
1289699151,13cyclictest343-21cstates07:50:222
1289299153,1cyclictest18791-21cut07:20:201
12892991512,2cyclictest25-21ksoftirqd/112:30:271
12892991512,2cyclictest25-21ksoftirqd/109:35:001
1289299151,0cyclictest141rcu_preempt10:26:101
1288999158,5cyclictest0-21swapper/009:19:420
1288999157,2cyclictest0-21swapper/009:57:450
1288999154,2cyclictest0-21swapper/010:35:190
1288999154,2cyclictest0-21swapper/010:35:190
12889991514,1cyclictest5898-21diskmemload12:17:360
12889991513,2cyclictest13-21ksoftirqd/010:26:420
12889991512,2cyclictest5016-21head09:05:250
12889991512,2cyclictest0-21swapper/012:10:580
12889991511,3cyclictest5898-21diskmemload12:08:280
12889991511,2cyclictest0-21swapper/012:24:410
12889991510,3cyclictest662-21fschecks_time07:50:240
1289899149,5cyclictest5898-21diskmemload12:37:473
1289899149,5cyclictest5898-21diskmemload11:30:283
1289899149,4cyclictest0-21swapper/308:40:153
1289899148,2cyclictest39-21ksoftirqd/309:41:583
1289899146,6cyclictest0-21swapper/309:57:453
1289899142,12cyclictest0-21swapper/312:12:273
12898991413,1cyclictest39-21ksoftirqd/312:24:303
12898991413,1cyclictest39-21ksoftirqd/309:25:363
1289699142,10cyclictest5898-21diskmemload09:24:382
12896991412,1cyclictest0-21swapper/212:15:182
1289299143,4cyclictest0-21swapper/107:50:191
1289299142,3cyclictest141rcu_preempt10:40:211
1289299141,2cyclictest16902-21ntp_kernel_pll_07:15:221
12892991412,1cyclictest25-21ksoftirqd/111:43:391
12892991412,1cyclictest25-21ksoftirqd/109:51:221
12892991410,3cyclictest20714-21tail11:35:301
1289299140,1cyclictest0-21swapper/110:15:421
1289299140,1cyclictest0-21swapper/109:47:521
1288999148,5cyclictest5898-21diskmemload12:28:270
1288999148,5cyclictest27959-21cat08:45:280
1288999147,5cyclictest0-21swapper/009:50:020
1288999143,3cyclictest5898-21diskmemload12:01:130
1288999142,3cyclictest1291-21runrttasks11:56:490
12889991412,2cyclictest0-21swapper/010:30:290
12889991411,3cyclictest13-21ksoftirqd/011:02:070
12889991411,2cyclictest32476-21sed08:55:210
12889991411,2cyclictest0-21swapper/010:05:160
12889991410,3cyclictest1663-21tr09:00:180
12889991410,1cyclictest5898-21diskmemload09:25:140
1289899139,3cyclictest2630-21tr07:55:163
1289899138,5cyclictest5898-21diskmemload11:58:063
1289899134,8cyclictest0-21swapper/308:40:003
1289899134,5cyclictest0-21swapper/310:25:583
12898991310,2cyclictest27925-21wc08:45:283
12898991310,2cyclictest23024-21cut07:30:163
12898991310,2cyclictest15523-21wc08:20:253
12898991310,2cyclictest12401-21tr08:15:193
1289899130,5cyclictest0-21swapper/308:05:163
1289899130,12cyclictest27452-21wc07:40:003
1289699139,3cyclictest9147-21grep11:00:242
1289699139,3cyclictest6993-21grep10:30:282
1289699139,3cyclictest4718-21head12:10:202
1289699139,3cyclictest25341-21tr08:40:332
1289699139,3cyclictest14954-21grep10:10:192
1289699138,4cyclictest21031-21grep08:30:292
1289699137,2cyclictest224-21systemd-journal10:05:002
1289699131,1cyclictest5898-21diskmemload11:10:132
1289699131,12cyclictest5898-21diskmemload10:45:252
12896991310,2cyclictest604-21grep08:55:282
12896991310,2cyclictest30634-21wc08:50:282
12896991310,2cyclictest18822-21sort07:20:212
1289699130,1cyclictest0-21swapper/210:51:192
1289699130,13cyclictest21187-21ntpq12:20:242
1289699130,12cyclictest24177-21wc07:30:252
1289299139,1cyclictest141rcu_preempt08:20:221
12892991311,1cyclictest25-21ksoftirqd/112:05:041
12892991311,1cyclictest25-21ksoftirqd/111:00:001
12892991310,2cyclictest27528-21cat07:40:001
12892991310,2cyclictest20214-21grep12:20:181
1289299130,3cyclictest0-21swapper/108:30:201
1289299130,12cyclictest8215-21sed09:10:291
1288999139,3cyclictest6895-21ls08:05:000
1288999139,3cyclictest5898-21diskmemload09:30:120
1288999139,3cyclictest13904-21sed07:10:220
1288999137,5cyclictest0-21swapper/010:45:230
1288999137,5cyclictest0-21swapper/008:50:180
1288999135,4cyclictest224-21systemd-journal07:34:590
1288999133,6cyclictest5898-21diskmemload09:46:160
1288999131,9cyclictest0-21swapper/011:20:210
12889991312,1cyclictest0-21swapper/009:41:310
12889991311,2cyclictest0-21swapper/011:25:200
12889991311,1cyclictest5898-21diskmemload11:05:070
12889991310,2cyclictest17436-21grep07:15:280
1289899129,2cyclictest31575-21grep09:30:263
1289899129,2cyclictest27290-21sed11:40:273
1289899129,2cyclictest0-21swapper/308:25:173
1289899128,3cyclictest663-21fschecks_time07:50:243
1289899128,2cyclictest11150-21grep08:10:313
1289899124,7cyclictest0-21swapper/309:36:223
12898991211,1cyclictest5898-21diskmemload11:54:203
12898991211,1cyclictest0-21swapper/310:36:593
12898991211,1cyclictest0-21swapper/310:36:583
1289699129,2cyclictest7926-21grep08:05:202
1289699129,2cyclictest5918-21awk09:10:012
1289699129,2cyclictest5898-21diskmemload10:35:272
1289699129,2cyclictest5898-21diskmemload10:35:272
1289699129,2cyclictest5388-21chrt09:35:302
1289699129,2cyclictest4701-21cron10:30:002
1289699129,2cyclictest30355-21tr07:45:182
1289699129,2cyclictest29587-21df09:55:182
1289699129,2cyclictest12261-21ssh09:15:172
1289699128,3cyclictest8416-21cat11:25:312
1289699128,3cyclictest6672-21tr09:10:172
1289699128,3cyclictest4434-21tail10:55:302
1289699128,3cyclictest30635-21ssh11:18:122
1289699127,4cyclictest18392-21cat08:25:342
1289699121,5cyclictest22642-21cron07:29:592
1289299129,2cyclictest5545-21head08:00:191
1289299129,2cyclictest32237-21basename11:20:001
1289299129,2cyclictest29712-21cut08:50:191
1289299129,2cyclictest21760-21tr11:10:261
1289299129,2cyclictest13189-21basename07:10:191
1289299129,2cyclictest13167-21grep12:15:251
1289299124,1cyclictest141rcu_preempt09:10:011
1289299120,3cyclictest141rcu_preempt12:13:441
1289299120,1cyclictest141rcu_preempt11:05:011
1289299120,11cyclictest11576-21tr12:35:171
1288999126,6cyclictest0-21swapper/010:11:120
1288999126,5cyclictest0-21swapper/011:40:220
1288999122,9cyclictest18995-21taskset08:29:150
1288999122,4cyclictest0-21swapper/007:45:160
12889991210,1cyclictest0-21swapper/009:10:140
1289899118,2cyclictest630-21ssh11:20:203
1289899118,2cyclictest29741-21sort08:50:193
1289899118,2cyclictest21219-21sh09:21:593
1289899118,2cyclictest14900-21chrt07:10:553
1289899117,3cyclictest31684-21tr08:55:163
1289699119,1cyclictest19328-21unixbench_singl10:40:302
1289699118,2cyclictest8386-21cat10:05:172
1289699118,2cyclictest552-21grep11:20:212
1289699118,2cyclictest30817-21ssh12:06:022
1289699118,2cyclictest27394-21cron07:40:002
1289699118,2cyclictest26691-21cat08:45:172
1289699118,2cyclictest23429-21sh12:01:352
1289699118,2cyclictest1482-21grep09:00:172
1289699118,2cyclictest11477-21needreboot09:40:252
1289699117,3cyclictest20913-21cut10:15:202
1289699110,10cyclictest14982-21taskset07:11:572
1289299118,2cyclictest6540-21ssh10:30:231
1289299118,2cyclictest32537-21rm09:57:381
1289299118,2cyclictest30739-21ssh12:26:141
1289299118,2cyclictest28878-21ssh10:50:141
1289299118,2cyclictest22991-21tr07:30:161
1289299118,2cyclictest2249-21ssh11:46:541
1289299118,2cyclictest2143-21grep11:20:321
1289299118,2cyclictest20648-21grep07:25:151
1289299118,2cyclictest15060-21tail10:10:191
1289299118,2cyclictest10452-21taskset09:40:181
1289299117,3cyclictest30830-21head07:45:201
1289299113,4cyclictest25-21ksoftirqd/107:45:001
1289299111,1cyclictest141rcu_preempt10:07:001
1289299110,1cyclictest141rcu_preempt10:35:291
1289299110,1cyclictest141rcu_preempt10:35:291
1289299110,10cyclictest12050-21cat08:15:011
1288999118,2cyclictest8564-21swap08:05:280
1288999118,2cyclictest11294-21sshd08:10:400
1288999111,2cyclictest224-21systemd-journal07:37:400
1289899108,1cyclictest4814-21cut08:00:163
1289899108,1cyclictest17118-21tr07:15:243
1289899102,6cyclictest29792-21cron07:45:013
1289699108,1cyclictest66292sleep208:01:452
1289699108,1cyclictest41422sleep207:55:362
1289699108,1cyclictest31680-21ssh09:30:282
1289699108,1cyclictest16574-21taskset08:23:362
1289699108,1cyclictest15654-21gpg-connect-age07:15:012
1289699108,1cyclictest14536-21dpkg11:05:192
1289699108,1cyclictest13625-21basename11:30:212
1289699107,2cyclictest9624-21hostname08:10:192
1289299108,2cyclictest25-21ksoftirqd/108:10:011
1289299108,1cyclictest7545-21tail11:25:221
1289299108,1cyclictest21900-21egrep08:35:201
1289299107,2cyclictest27612-21ntpq08:45:241
1289299107,2cyclictest12746-21aten2_r7power_c11:30:161
1289299107,2cyclictest10307-21head08:10:221
1288999108,1cyclictest15908-21basename08:20:290
128969990,4cyclictest0-21swapper/208:35:192
128929997,1cyclictest32221-21grep08:55:191
128929997,1cyclictest2804-21cat07:55:161
128929997,1cyclictest175432sleep108:25:241
128929996,2cyclictest2804-21sensors_temp09:00:281
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional