You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 05:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Jul 07, 2025 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7055210277,9sleep30-21swapper/319:07:173
7006210264,33sleep10-21swapper/119:06:391
582227360,8sleep00-21swapper/019:05:210
695527161,6sleep20-21swapper/219:05:592
735399282,3cyclictest0-21swapper/322:15:223
735399282,25cyclictest6250-21kworker/u8:023:43:093
735399274,2cyclictest4760-21head20:10:183
735399273,4cyclictest19432-21idleruntime-cro19:35:003
735399270,2cyclictest0-21swapper/320:35:163
735399270,26cyclictest0-21swapper/323:50:263
735399270,26cyclictest0-21swapper/322:50:223
7342992626,0cyclictest0-21swapper/121:43:071
735399253,1cyclictest19274-21chrt22:59:173
735399251,1cyclictest0-21swapper/321:18:313
735399250,3cyclictest32293-21diskmemload21:32:203
735399250,24cyclictest0-21swapper/322:05:153
735399244,18cyclictest16456-21cpuspeed_turbos23:45:133
734999241,23cyclictest0-21swapper/222:40:342
735399234,1cyclictest32293-21diskmemload00:36:423
735399231,4cyclictest24200-21idleruntime-cro19:45:013
7353992314,8cyclictest27597-21cron21:30:013
735399231,21cyclictest3152-21cron00:25:003
735399230,4cyclictest0-21swapper/322:30:223
735399230,3cyclictest0-21swapper/322:41:003
735399230,3cyclictest0-21swapper/300:01:413
735399222,2cyclictest32293-21diskmemload21:59:003
735399220,7cyclictest0-21swapper/323:17:443
734999220,13cyclictest0-21swapper/219:15:152
734999220,13cyclictest0-21swapper/219:15:142
735399211,4cyclictest27423-21cstates23:05:193
735399211,1cyclictest32293-21diskmemload23:34:503
735399210,5cyclictest3016-21df_abs21:35:163
735399210,1cyclictest0-21swapper/300:10:123
735399204,1cyclictest32293-21diskmemload21:42:133
735399202,1cyclictest14231-21sed19:20:273
735399201,3cyclictest32293-21diskmemload23:01:103
735399201,3cyclictest32293-21diskmemload23:01:103
734999200,0cyclictest0-21swapper/221:44:222
735399198,6cyclictest20398-21ssh21:48:123
735399191,2cyclictest0-21swapper/322:37:133
735399190,9cyclictest0-21swapper/322:12:073
7342991917,2cyclictest0-21swapper/121:53:211
735399189,7cyclictest17366-21wc00:10:003
735399182,2cyclictest32293-21diskmemload21:14:563
735399182,2cyclictest32293-21diskmemload21:14:563
735399181,2cyclictest8744-21/usr/sbin/munin20:20:253
735399181,16cyclictest0-21swapper/300:17:583
735399180,3cyclictest0-21swapper/323:10:203
735399180,1cyclictest0-21swapper/322:25:153
734999180,18cyclictest0-21swapper/200:19:582
734999180,18cyclictest0-21swapper/200:11:482
734999180,18cyclictest0-21swapper/200:09:152
734299180,18cyclictest0-21swapper/122:59:491
735399177,6cyclictest27318-21taskset20:57:313
735399177,5cyclictest7076-21ssh00:29:013
735399177,5cyclictest5639-21ssh23:36:443
735399177,5cyclictest11983-21taskset19:16:133
735399177,5cyclictest11983-21taskset19:16:133
735399175,11cyclictest4883-21load22:00:203
735399171,15cyclictest9708-21perf19:15:003
735399170,2cyclictest0-21swapper/323:20:153
735399170,16cyclictest29598-21expr19:55:173
735399170,16cyclictest23293-21wc23:25:273
7349991715,1cyclictest22871-21taskset23:25:232
7349991711,5cyclictest37850irq/131-snd_hda21:54:352
734999170,17cyclictest0-21swapper/223:54:102
734999170,16cyclictest0-21swapper/222:46:412
734299171,2cyclictest141rcu_preempt23:44:011
734299170,1cyclictest0-21swapper/100:25:171
273612170,2sleep3381rcuc/319:50:153
735399169,6cyclictest15900-21rm21:20:263
735399168,7cyclictest1094-21chrt23:57:113
735399163,2cyclictest20487-21latency_hist20:45:003
734999160,2cyclictest0-21swapper/222:51:152
734299164,8cyclictest496-21dbus-daemon22:30:241
734299161,1cyclictest32293-21diskmemload22:38:021
734299160,2cyclictest0-21swapper/121:36:301
735399158,6cyclictest8099-21apache_processe00:30:143
735399158,6cyclictest26398-21chrt19:47:523
735399158,6cyclictest14855-21mailstats20:30:223
735399154,3cyclictest23328-21tr20:50:163
735399151,13cyclictest0-21swapper/322:20:173
735399150,4cyclictest32073-21sshd21:07:223
735399150,1cyclictest0-21swapper/322:48:533
7349991513,1cyclictest0-21swapper/223:15:172
7349991511,2cyclictest0-21swapper/221:48:132
734999150,15cyclictest0-21swapper/222:27:442
734999150,14cyclictest0-21swapper/222:05:522
735399148,5cyclictest6593-21cut20:15:153
735399147,6cyclictest31616-21aten2_r7power_c20:00:143
735399147,6cyclictest12803-21tail20:25:273
735399142,4cyclictest14957-21apt-get19:25:143
735399142,3cyclictest224-21systemd-journal20:49:593
735399141,3cyclictest224-21systemd-journal21:55:003
7353991410,2cyclictest29356-21cat21:00:303
735399140,4cyclictest224-21systemd-journal20:10:003
734999142,1cyclictest0-21swapper/221:30:142
734999140,12cyclictest0-21swapper/200:30:042
7342991413,1cyclictest25-21ksoftirqd/121:26:251
7342991411,2cyclictest8337-21grep21:15:211
734299140,2cyclictest0-21swapper/123:39:461
734299140,12cyclictest0-21swapper/100:05:071
734099143,2cyclictest8185-21cat21:15:200
734099141,12cyclictest7089-21/usr/sbin/munin22:50:280
735399131,3cyclictest224-21systemd-journal19:36:223
734999139,3cyclictest28261-21wc19:50:252
734999139,3cyclictest15859-21grep21:20:262
734999139,2cyclictest20771-21grep20:45:162
734999138,4cyclictest25240-21sed22:15:212
734999133,10cyclictest32293-21diskmemload23:20:362
734999132,9cyclictest0-21swapper/219:20:142
7349991310,2cyclictest31062-21sed21:55:272
7349991310,2cyclictest30041-21grep23:30:282
7349991310,2cyclictest241162sleep222:39:052
7349991310,2cyclictest17750-21ls22:10:142
7349991310,2cyclictest17514-21grep20:35:282
7349991310,2cyclictest16579-21sed19:25:272
7349991310,2cyclictest12251-21grep20:25:202
734999130,12cyclictest22531-21uname21:25:222
734999130,12cyclictest10394-21date00:05:002
734999130,0cyclictest0-21swapper/222:23:032
734299139,3cyclictest977-21chrt23:33:571
7342991310,2cyclictest9374-21wc23:15:241
7342991310,2cyclictest27929-21head23:05:221
7342991310,2cyclictest26179-21tail20:55:181
7342991310,2cyclictest21798-21cron19:40:011
734099139,3cyclictest8035-21sed20:15:260
734099139,3cyclictest11057-21sed21:40:280
734099139,2cyclictest29978-21cut23:55:170
734099138,3cyclictest24643-21grep20:50:270
734099133,6cyclictest10461-21apt-config00:05:000
734099132,8cyclictest14692-21turbostat.cron19:25:000
734099131,7cyclictest0-21swapper/021:35:150
734099131,1cyclictest32293-21diskmemload22:18:200
7340991310,2cyclictest3855-21grep22:00:150
7340991310,2cyclictest305-21cut22:45:140
7340991310,2cyclictest29438-21ls21:30:240
7340991310,2cyclictest28482-21wc23:05:260
7340991310,2cyclictest27101-21sed20:55:270
7340991310,2cyclictest16134-21grep20:35:160
734099130,1cyclictest0-21swapper/023:02:150
734099130,1cyclictest0-21swapper/023:02:140
734099130,12cyclictest27391-21grep19:50:160
734999129,2cyclictest9833-21date19:15:002
734999129,2cyclictest384-21grep20:00:232
734999129,2cyclictest3176-21grep23:10:262
734999129,2cyclictest23744-21tail19:40:292
734999129,2cyclictest2062-21df20:05:222
734999129,2cyclictest20377-21sort19:35:182
734999129,2cyclictest19903-21sed20:40:302
734999129,2cyclictest18002-21cut19:30:182
734999126,5cyclictest0-21swapper/220:50:142
7349991211,1cyclictest0-21swapper/221:12:262
7349991211,1cyclictest0-21swapper/221:12:262
7349991210,2cyclictest0-21swapper/223:37:462
734999120,9cyclictest0-21swapper/221:38:562
734999120,1cyclictest0-21swapper/200:35:172
734299129,2cyclictest7212-21cat21:15:011
734299129,2cyclictest7212-21cat21:15:001
734299129,2cyclictest3875-21wc20:09:591
734299129,2cyclictest31170-21head00:20:191
734299129,2cyclictest22405-21sed23:25:181
734299129,2cyclictest12498-21grep22:05:251
734299129,2cyclictest10384-21ssh00:05:001
734299128,3cyclictest6507-21ntpq22:25:221
734299128,3cyclictest22342-21grep19:40:181
734299124,7cyclictest11726-21packagekitd19:57:221
734299120,1cyclictest0-21swapper/121:22:121
734299120,11cyclictest8287-21grep19:10:181
734299120,11cyclictest0-21swapper/120:35:161
734099129,2cyclictest8184-21basename00:30:170
734099129,2cyclictest32711-21strings20:00:200
734099129,2cyclictest29771-21cat23:30:280
734099129,2cyclictest27736-21cat23:30:010
734099129,2cyclictest2715-21tr23:35:130
734099129,2cyclictest24515-21sed21:50:280
734099129,2cyclictest23851-21cut23:50:190
734099129,2cyclictest23539-21sh00:15:000
734099129,2cyclictest22537-21basename23:50:000
734099129,2cyclictest21395-21head20:45:200
734099129,2cyclictest19007-21grep20:40:210
734099129,2cyclictest18720-21ls22:10:210
734099128,3cyclictest16204-21cat21:45:150
734099124,4cyclictest224-21systemd-journal20:10:000
734099124,4cyclictest224-21systemd-journal19:29:590
734099123,5cyclictest32293-21diskmemload00:05:160
734099120,8cyclictest0-21swapper/019:10:150
734999118,2cyclictest7596-21grep20:15:222
734999118,2cyclictest4902-21grep00:25:412
734999118,2cyclictest31205-21grep23:55:252
734999118,2cyclictest30870-21grep19:55:282
734999118,2cyclictest30789-21ssh23:07:212
734999118,2cyclictest26343-21cut20:55:202
734999118,2cyclictest17533-21grep23:45:212
734999118,2cyclictest15959-21ssh22:55:392
734999118,2cyclictest12350-21tr22:30:152
734999118,2cyclictest10610-21taskset20:21:382
734999118,2cyclictest10576-21grep23:40:232
734999118,2cyclictest10467-21wc22:05:002
734999117,3cyclictest14280-21cut20:30:182
734999117,2cyclictest21860-21fgrep23:00:262
734999117,2cyclictest21860-21fgrep23:00:262
734299118,2cyclictest9467-21tune2fs20:20:171
734299118,2cyclictest8340-21tr22:50:191
734299118,2cyclictest7566-21sleep23:15:001
734299118,2cyclictest4757-21sed20:10:181
734299118,2cyclictest31904-21ssh21:56:101
734299118,2cyclictest31045-21grep23:55:231
734299118,2cyclictest29492-21ssh22:42:201
734299118,2cyclictest27903-21ssh22:17:261
734299118,2cyclictest26260-21chrt19:46:051
734299118,2cyclictest25904-21ssh23:04:251
734299118,2cyclictest25904-21ssh23:04:241
734299118,2cyclictest25598-21ssh00:15:271
734299118,2cyclictest2321-21ssh22:45:371
734299118,2cyclictest20450-21ssh23:47:491
734299118,2cyclictest18808-21ntpq00:10:201
734299118,2cyclictest18698-21ssh23:22:391
734299118,2cyclictest17080-21tr21:45:211
734299118,2cyclictest13623-21grep00:35:171
734299118,2cyclictest12868-21grep20:25:271
734299118,2cyclictest12798-21tr19:20:151
734299118,2cyclictest10273-21ssh00:30:571
734299118,2cyclictest10009-21readlink19:15:011
734299118,2cyclictest10009-21readlink19:15:001
734299117,3cyclictest1938-21cron21:35:011
734299110,10cyclictest31182-21cat22:20:161
734299110,10cyclictest30673-21df21:05:161
734099119,1cyclictest461-21tr21:10:180
734099119,1cyclictest461-21tr21:10:170
734099119,1cyclictest25986-21grep22:40:160
734099118,2cyclictest9047-21tr20:20:140
734099118,2cyclictest7985-21ssh23:15:150
734099118,2cyclictest5143-21sh22:25:100
734099118,2cyclictest4752-21cut00:25:390
734099118,2cyclictest29289-21ssh00:19:210
734099118,2cyclictest25597-21tr19:45:200
734099118,2cyclictest2452-21sh22:22:380
734099118,2cyclictest21619-21grep21:25:170
734099118,2cyclictest21608-21chrt19:37:350
734099118,2cyclictest16582-21sh23:20:290
734099118,2cyclictest1584-21tr23:10:140
734099118,2cyclictest15257-21rm23:44:310
734099118,2cyclictest14052-21grep00:35:190
734099118,2cyclictest13877-21ssh22:30:280
734099118,2cyclictest11668-21ssh22:05:180
734099118,2cyclictest11235-21cat19:15:210
734099118,2cyclictest11235-21cat19:15:210
734099117,3cyclictest15882-21ssh21:20:270
734099110,11cyclictest0-21swapper/022:58:480
734099110,10cyclictest6132-21wc20:15:000
734099110,10cyclictest29755-21sh21:55:160
734999107,2cyclictest864-21ssh00:21:382
734999107,2cyclictest7606-21ssh21:15:172
734999107,2cyclictest32127-21taskset21:08:022
734999107,2cyclictest28078-21tr21:00:182
734999106,3cyclictest4719-21grep20:10:182
734299108,1cyclictest8336-21taskset20:18:121
734299108,1cyclictest3512-21rm22:00:001
734299108,1cyclictest22864-21ssh23:50:061
734299108,1cyclictest22751-21cat20:49:551
734299108,1cyclictest190772sleep119:30:351
734299108,1cyclictest18259-21taskset20:40:121
734299107,2cyclictest29691-21chrt21:02:131
734299107,2cyclictest23639-21sort20:50:181
734299107,2cyclictest17336-21ssh22:10:101
734299107,2cyclictest15023-21tr19:25:141
734299107,2cyclictest13711-21egrep20:30:151
734099108,1cyclictest32069-21taskset21:07:190
734099108,1cyclictest31166-21grep00:20:180
734099108,1cyclictest19271-21ssh22:35:160
734099107,2cyclictest31310-21cron20:00:000
734099107,2cyclictest29861-21taskset21:04:250
734099107,2cyclictest23817-21sed19:40:290
734099107,2cyclictest13783-21tr20:30:150
73499997,1cyclictest244232sleep219:45:022
73429998,1cyclictest32629-21memory20:00:201
73429997,1cyclictest27673-21cat19:50:181
73409997,1cyclictest17651-21cat19:30:160
73409996,2cyclictest11224-21aten2_r7power_p20:25:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional