You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 01:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Nov 23, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5700210389,9sleep30-21swapper/307:05:093
757229666,9sleep00-21swapper/007:06:420
772229581,9sleep10-21swapper/107:08:401
772328964,17sleep20-21swapper/207:08:412
289902580,0sleep10-21swapper/109:33:291
7917992624,1cyclictest0-21swapper/012:19:060
792499240,24cyclictest0-21swapper/210:35:412
792499220,1cyclictest6726-21grep11:49:592
7926992020,0cyclictest0-21swapper/310:45:133
7924992013,7cyclictest0-21swapper/211:37:342
792499195,11cyclictest61850irq/126-enp2s0-08:20:192
792499194,2cyclictest0-21swapper/209:22:032
792099190,19cyclictest0-21swapper/111:54:321
792099180,18cyclictest0-21swapper/110:43:241
792499179,2cyclictest10705-21cat09:45:172
792499174,4cyclictest141rcu_preempt12:10:172
7924991712,2cyclictest0-21swapper/207:35:132
792499171,13cyclictest0-21swapper/208:35:132
7920991716,1cyclictest0-21swapper/109:22:211
792099170,0cyclictest0-21swapper/111:14:481
791799170,0cyclictest0-21swapper/010:10:310
301372170,0sleep30-21swapper/310:50:273
7924991616,0cyclictest0-21swapper/210:30:132
7924991615,1cyclictest697-21diskmemload10:43:302
7924991615,1cyclictest0-21swapper/210:10:312
792499161,3cyclictest0-21swapper/207:50:142
7924991613,2cyclictest37850irq/131-snd_hda08:50:162
792499160,1cyclictest0-21swapper/209:50:442
792099160,1cyclictest0-21swapper/110:13:581
792699157,7cyclictest24288-21munin-node10:59:373
792699150,14cyclictest9791-21sh09:45:003
792499151,13cyclictest0-21swapper/209:10:432
792499150,1cyclictest0-21swapper/209:33:072
792499150,14cyclictest0-21swapper/211:12:432
791799158,6cyclictest17071-21ntp_states09:50:270
791799150,3cyclictest0-21swapper/012:03:120
7926991412,2cyclictest39-21ksoftirqd/310:09:283
792499149,3cyclictest21422-21taskset12:00:482
792499144,2cyclictest0-21swapper/211:06:012
792499140,3cyclictest0-21swapper/212:35:542
792499140,1cyclictest0-21swapper/211:19:432
792499140,1cyclictest0-21swapper/210:59:212
792499140,1cyclictest0-21swapper/207:15:152
792499140,13cyclictest0-21swapper/210:46:222
792099149,4cyclictest12052-21sed09:45:301
792099149,3cyclictest27528-21ssh11:40:181
7920991411,2cyclictest1652-21awk10:55:011
7920991410,3cyclictest29012-21grep10:00:141
791799146,7cyclictest697-21diskmemload11:57:240
791799140,7cyclictest0-21swapper/011:20:120
791799140,4cyclictest0-21swapper/012:09:380
792699139,3cyclictest17669-21sed09:50:323
792699131,2cyclictest141rcu_preempt09:49:533
7926991310,2cyclictest26068-21grep12:05:153
7926991310,2cyclictest22676-21chrt12:02:213
7926991310,2cyclictest20754-21grep08:40:273
7926991310,2cyclictest20359-21tr09:25:233
7926991310,2cyclictest14363-21basename12:20:133
792699130,12cyclictest1810-21cron10:05:003
792499139,3cyclictest23941-21grep09:55:262
792499139,3cyclictest13080-21cat09:20:012
7924991310,2cyclictest32186-21latency_hist12:09:592
792499130,1cyclictest0-21swapper/212:15:142
792499130,13cyclictest0-21swapper/212:20:472
792499130,13cyclictest0-21swapper/209:29:592
792099139,3cyclictest8275-21rm11:25:171
792099139,3cyclictest3773-21grep08:05:251
792099139,2cyclictest22746-21grep07:40:151
792099134,6cyclictest9319-21tail11:00:191
7920991310,2cyclictest29728-21wc10:25:281
7920991310,2cyclictest29528-21cut12:30:191
7920991310,2cyclictest2881-21grep10:30:171
7920991310,2cyclictest21552-21grep10:20:171
7920991310,2cyclictest1416-21grep08:00:251
7920991310,2cyclictest1291-21runrttasks09:55:031
791799131,9cyclictest697-21diskmemload10:40:220
7917991310,3cyclictest13-21ksoftirqd/009:19:150
7917991310,2cyclictest19397-21grep09:25:180
791799130,11cyclictest0-21swapper/007:40:140
792699129,2cyclictest9074-21grep11:50:283
792699129,2cyclictest487-21cut12:10:173
792699129,2cyclictest3714-21cat10:30:253
792699129,2cyclictest22889-21head09:55:183
792699129,2cyclictest21995-21grep10:20:183
792699129,2cyclictest1845-21tail09:10:173
792699129,2cyclictest15934-21cut07:25:163
792699129,2cyclictest11468-21cron08:25:003
792699128,3cyclictest21206-21apt-get10:45:003
792699128,3cyclictest11285-21grep07:15:183
792699127,5cyclictest5284-21apache211:18:413
792699126,2cyclictest224-21systemd-journal08:00:003
792699122,3cyclictest224-21systemd-journal07:45:003
7926991210,1cyclictest22247-21grep08:45:193
792699120,1cyclictest0-21swapper/310:36:243
792499129,2cyclictest30899-21cat07:55:212
792499129,2cyclictest22680-21ls08:45:232
792499129,2cyclictest21334-21tr10:20:162
792499129,2cyclictest18734-21cat07:30:352
792499129,2cyclictest15550-21cat11:30:262
792499129,2cyclictest15404-21basename07:25:132
792499128,3cyclictest3742-21awk09:39:592
792499126,5cyclictest0-21swapper/211:52:222
792499124,1cyclictest0-21swapper/211:02:022
7924991210,1cyclictest2915-21grep08:05:162
792499120,3cyclictest3568-21ssh11:21:102
792499120,0cyclictest0-21swapper/212:32:312
792099129,2cyclictest9078-21cron08:19:591
792099129,2cyclictest7325-21tr09:15:141
792099129,2cyclictest31787-21grep09:35:171
792099129,2cyclictest30512-21grep07:55:191
792099129,2cyclictest27454-21wc12:05:251
792099129,2cyclictest2650-21grep11:45:261
792099129,2cyclictest25761-21tail07:45:181
792099129,2cyclictest20683-21ssh11:35:171
792099129,2cyclictest19842-21tail08:40:181
792099129,2cyclictest15680-21cut11:05:421
792099129,2cyclictest15115-21tail11:55:261
792099129,2cyclictest14838-21grep08:30:171
792099129,2cyclictest10755-21tr07:15:161
792099128,3cyclictest9619-21grep08:20:161
792099128,3cyclictest8728-21ssh09:43:491
792099128,3cyclictest28531-21grep07:50:201
792099128,3cyclictest20974-21ls12:25:001
792099125,6cyclictest5299-21cups-browsed12:10:251
791799129,2cyclictest8198-21sort10:10:000
791799129,2cyclictest32419-21tr08:00:140
791799129,2cyclictest24514-21ssh12:26:340
791799129,2cyclictest24514-21ssh12:26:330
791799129,2cyclictest2139-21cut10:30:140
791799129,2cyclictest19247-21wc07:30:410
791799129,2cyclictest15182-21wc08:30:200
791799127,5cyclictest5295-21apache212:22:100
791799121,9cyclictest0-21swapper/008:05:130
7917991210,1cyclictest531-21grep12:10:170
7917991210,1cyclictest21691-21grep11:35:240
791799120,7cyclictest0-21swapper/011:44:260
791799120,11cyclictest17632-21chrt09:23:340
791799120,11cyclictest12778-21wc08:25:200
149572121,1sleep014962-21apache_volume11:05:130
792699119,2cyclictest29435-21ssh11:41:153
792699119,1cyclictest8501-21uname11:25:203
792699119,1cyclictest32484-21grep08:00:143
792699119,1cyclictest30276-21grep09:00:263
792699119,1cyclictest14184-21head09:20:193
792699119,1cyclictest12415-21grep08:25:163
792699118,2cyclictest9253-21grep07:10:253
792699118,2cyclictest873-21seq11:20:033
792699118,2cyclictest8386-21cut12:15:193
792699118,2cyclictest6246-21sed08:10:283
792699118,2cyclictest6161-21ssh12:36:233
792699118,2cyclictest5485-21ssh11:48:373
792699118,2cyclictest32757-21grep09:35:273
792699118,2cyclictest31245-21tr09:05:153
792699118,2cyclictest29916-21ssh12:30:233
792699118,2cyclictest28740-21grep10:25:193
792699118,2cyclictest27648-21sort07:50:153
792699118,2cyclictest26597-21wc07:45:263
792699118,2cyclictest25577-21ssh11:13:413
792699118,2cyclictest24289-21cat08:50:133
792699118,2cyclictest22776-21ssh12:25:203
792699118,2cyclictest22776-21ssh12:25:193
792699118,2cyclictest20565-21ssh10:19:463
792699118,2cyclictest20167-21ssh11:35:093
792699118,2cyclictest19422-21grep07:30:433
792699118,2cyclictest18578-21taskset08:37:053
792699118,2cyclictest16493-21taskset11:05:503
792699118,2cyclictest15047-21turbostat07:25:013
792699118,2cyclictest14090-21taskset10:14:133
792699118,2cyclictest12025-21sh09:19:003
792699118,2cyclictest10234-21timerwakeupswit11:00:283
792499119,1cyclictest28552-21df10:25:172
792499119,1cyclictest28381-21tr09:00:012
792499119,1cyclictest14578-21grep07:20:242
792499118,2cyclictest8407-21grep07:10:182
792499118,2cyclictest6350-21taskset08:10:432
792499118,2cyclictest5407-21taskset09:40:222
792499118,2cyclictest489-21head08:00:162
792499118,2cyclictest31802-21chrt10:01:392
792499118,2cyclictest31007-21chrt09:05:132
792499118,2cyclictest28326-21tr10:50:142
792499118,2cyclictest25319-21cut07:45:162
792499118,2cyclictest24828-21cat07:45:002
792499118,2cyclictest22275-21sh12:25:162
792499118,2cyclictest22275-21sh12:25:162
792499118,2cyclictest16643-21tr10:15:242
792499118,2cyclictest16640-21ssh11:57:072
792499118,2cyclictest15604-21cat08:30:262
792499118,2cyclictest12710-21ls08:25:182
792499117,3cyclictest11333-21taskset11:27:352
792499110,3cyclictest0-21swapper/210:06:532
792099118,2cyclictest7658-21cut12:15:151
792099118,2cyclictest5053-21ssh11:22:481
792099118,2cyclictest4794-21tr08:10:161
792099118,2cyclictest4605-21ssh10:56:041
792099118,2cyclictest3008-21sh12:35:011
792099118,2cyclictest29757-21awk09:00:211
792099118,2cyclictest28156-21turbostat12:30:011
792099118,2cyclictest28156-21turbostat12:30:001
792099118,2cyclictest27997-21grep11:15:181
792099118,2cyclictest2391-21wget10:05:131
792099118,2cyclictest23012-21ssh12:02:461
792099118,2cyclictest20968-21tune2fs07:35:161
792099118,2cyclictest18988-21ssh09:51:551
792099118,2cyclictest18923-21ssh11:34:051
792099118,2cyclictest18811-21head07:30:351
792099118,2cyclictest17802-21tail08:35:191
792099118,2cyclictest13155-21ssh10:38:301
792099117,3cyclictest10267-21ls07:15:001
791799119,1cyclictest9283-21cut11:00:190
791799119,1cyclictest4884-21grep09:40:170
791799119,1cyclictest29367-21head09:00:180
791799119,1cyclictest27196-21uname08:55:360
791799119,1cyclictest24644-21cut08:50:160
791799119,1cyclictest22130-21cut08:45:170
791799119,1cyclictest21994-21head07:35:250
791799119,1cyclictest17158-21grep10:15:280
791799118,2cyclictest961-21ssh10:04:010
791799118,2cyclictest9337-21taskset08:20:100
791799118,2cyclictest875-21cut11:45:140
791799118,2cyclictest5078-21tail12:35:280
791799118,2cyclictest3707-21ssh09:11:220
791799118,2cyclictest32327-21wc09:35:220
791799118,2cyclictest29551-21ssh12:30:200
791799118,2cyclictest2939-21cut10:55:190
791799118,2cyclictest27385-21cut11:15:150
791799118,2cyclictest27286-21ssh09:30:370
791799118,2cyclictest17697-21wc08:35:190
791799118,2cyclictest15448-21grep07:25:140
791799118,2cyclictest14234-21ssh09:48:440
791799118,2cyclictest13133-21sh11:29:440
791799115,1cyclictest697-21diskmemload10:45:100
791799114,5cyclictest7642-21cut08:15:170
791799110,1cyclictest0-21swapper/011:50:200
791799110,10cyclictest30327-21sh10:50:400
791799110,10cyclictest21518-21cat10:20:170
791799110,10cyclictest12993-21cut07:20:130
792699108,1cyclictest28542-21sed09:00:013
792699108,1cyclictest26564-21ls09:30:233
792699108,1cyclictest21391-21taskset07:35:193
792699108,1cyclictest13714-21ssh11:30:073
792699107,2cyclictest3194-21sed08:05:193
792699107,2cyclictest15464-21ssh11:55:473
792699107,2cyclictest14671-21df08:30:163
792499108,1cyclictest286272sleep209:00:112
792499107,2cyclictest26995-21grep11:40:152
792099108,1cyclictest27160-21seq08:55:371
792099108,1cyclictest23965-21ssh10:45:401
792099108,1cyclictest2329-21awk09:10:211
792099108,1cyclictest15311-21apache_volume07:25:121
792099107,2cyclictest24131-21tr08:50:141
792099107,2cyclictest22627-21awk08:45:211
792099107,2cyclictest15831-21df10:15:171
792099107,2cyclictest13651-21grep07:20:151
792099107,2cyclictest12007-21tr08:25:141
792099106,3cyclictest19227-21grep09:25:171
792099100,9cyclictest31443-21expr09:05:161
791799109,1cyclictest28213-21cut07:50:170
791799108,1cyclictest268302sleep007:45:530
791799108,1cyclictest22704-21ssh11:10:270
791799108,1cyclictest22443-21expr09:55:160
791799108,1cyclictest11995-21sed07:15:270
791799107,2cyclictest31800-21chrt07:58:470
791799107,2cyclictest31662-21fschecks_time09:05:170
791799107,2cyclictest29228-21cat10:25:210
791799107,2cyclictest19747-21cut08:40:170
791799107,2cyclictest11292-21ssh10:36:330
791799102,5cyclictest14460-21df_inode11:30:150
791799100,6cyclictest0-21swapper/007:10:140
79269996,2cyclictest8284-21gzip08:15:243
79249998,1cyclictest8119-21ntpq08:15:212
79249998,1cyclictest208972sleep208:41:042
79179997,1cyclictest4612-21basename08:10:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional