You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 04:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Nov 22, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32296210476,9sleep10-21swapper/119:08:041
3214929682,9sleep30-21swapper/319:06:093
3243128366,13sleep20-21swapper/219:09:492
3211228369,9sleep00-21swapper/019:05:420
32531992322,1cyclictest37850irq/131-snd_hda22:36:312
3253199230,22cyclictest0-21swapper/222:00:072
3253199230,22cyclictest0-21swapper/221:47:332
3253199220,22cyclictest0-21swapper/223:41:052
134282220,1sleep125-21ksoftirqd/121:23:501
3252799215,12cyclictest0-21swapper/121:05:131
3252799210,21cyclictest0-21swapper/100:13:591
32535992018,2cyclictest94-21kswapd021:45:103
3253599193,15cyclictest6197-21sh21:18:493
32531991917,1cyclictest0-21swapper/221:30:132
32535991818,0cyclictest0-21swapper/300:03:083
3253199180,1cyclictest0-21swapper/223:50:362
32531991715,1cyclictest12479-21fschecks_count22:30:162
3253199170,17cyclictest0-21swapper/200:18:082
3252799172,7cyclictest224-21systemd-journal23:11:241
32527991711,2cyclictest0-21swapper/122:34:351
3253599161,12cyclictest0-21swapper/322:30:133
32531991614,2cyclictest0-21swapper/200:35:082
32527991610,6cyclictest0-21swapper/121:45:121
3252799160,1cyclictest0-21swapper/122:35:331
3252299165,8cyclictest25532-21diskmemload21:11:520
3252299160,2cyclictest0-21swapper/021:57:130
3253599152,12cyclictest0-21swapper/300:14:223
32535991512,2cyclictest14735-21sh22:55:003
32535991512,1cyclictest0-21swapper/323:08:443
3253199159,3cyclictest23899-21latency_hist20:00:002
32527991513,2cyclictest25-21ksoftirqd/100:19:071
3252799150,7cyclictest224-21systemd-journal22:30:001
3252299159,3cyclictest4067-21cron23:10:000
3252299150,14cyclictest0-21swapper/021:52:480
32535991410,3cyclictest520-21sh22:00:003
32535991410,3cyclictest21628-21sh22:14:313
32535991410,2cyclictest10015-21cut19:30:153
32531991411,2cyclictest23109-21cstates22:15:152
3253199140,11cyclictest0-21swapper/223:25:162
3252799144,2cyclictest224-21systemd-journal19:50:001
32527991413,1cyclictest0-21swapper/100:03:391
32527991411,2cyclictest25685-21apt21:55:001
32527991410,3cyclictest11125-21grep00:00:001
3252299149,4cyclictest1692-21ssh23:30:150
32522991410,3cyclictest26355-21basename22:40:140
32522991410,3cyclictest22686-21sed21:00:260
32522991410,3cyclictest22686-21sed21:00:260
32522991410,2cyclictest11957-21tr21:45:140
3252299140,7cyclictest969-21idleruntime-cro23:30:000
3253599139,3cyclictest1048-21sort23:30:003
3253599139,2cyclictest4719-21tr21:40:143
3253599135,7cyclictest13546-21munin-node22:17:583
32535991310,2cyclictest8761-21wc19:25:243
32535991310,2cyclictest5240-21cat00:20:013
32535991310,2cyclictest25157-21chrt23:24:183
32535991310,2cyclictest24591-21grep23:45:273
3253599130,12cyclictest3485-21sed19:15:183
3253599130,12cyclictest21750-21ssh21:29:553
3253199139,3cyclictest11066-21cron00:00:002
3253199134,3cyclictest224-21systemd-journal00:30:442
3253199131,4cyclictest224-21systemd-journal20:35:002
32531991310,2cyclictest694-21awk22:45:002
32531991310,2cyclictest3399-21cut19:15:182
32531991310,2cyclictest17036-21head21:25:282
3253199130,2cyclictest0-21swapper/223:04:062
3253199130,12cyclictest24198-21grep21:05:172
3253199130,12cyclictest13960-21grep20:45:162
3252799139,3cyclictest30928-21ntpq21:35:231
3252799139,3cyclictest30707-21tr23:50:161
3252799139,3cyclictest14571-21cat19:40:011
3252799139,3cyclictest12599-21grep23:15:161
3252799134,6cyclictest23890-21awk20:00:001
32527991310,2cyclictest29929-21grep23:05:141
32527991310,2cyclictest28481-21grep20:05:261
32527991310,2cyclictest2546-21grep21:15:281
32527991310,2cyclictest24801-21sh00:10:001
32527991310,2cyclictest11031-21perf20:40:001
3252799130,4cyclictest0-21swapper/121:44:181
3252299139,3cyclictest23904-21wc23:45:200
3252299139,3cyclictest17202-21grep22:10:260
3252299139,3cyclictest15794-21ls20:50:000
3252299139,2cyclictest9618-21grep20:35:190
3252299139,2cyclictest4408-21grep19:15:270
3252299139,2cyclictest25522-21tr00:10:150
3252299131,6cyclictest8647-21munin-run20:35:000
3252299131,1cyclictest1291-21runrttasks00:02:500
32522991310,2cyclictest19948-21sed23:20:160
32522991310,2cyclictest15424-21grep21:25:160
32522991310,2cyclictest13472-21grep00:25:260
32522991310,2cyclictest10625-21grep23:35:260
3253599129,2cyclictest8860-21cat21:20:193
3253599129,2cyclictest7252-21grep20:30:173
3253599129,2cyclictest6371-21sed23:55:243
3253599129,2cyclictest24178-21sed21:05:173
3253599129,2cyclictest2399-21grep22:00:233
3253599129,2cyclictest22399-21grep19:55:163
3253599129,2cyclictest20617-21ls19:50:223
3253599129,2cyclictest20411-21cat00:05:273
3253599129,2cyclictest18670-21grep00:30:213
3253599129,2cyclictest16487-21tail23:40:203
3253599129,2cyclictest14406-21grep19:40:013
3253599128,2cyclictest31765-21grep22:20:253
3253599127,4cyclictest8980-21tr23:35:153
3253599121,11cyclictest0-21swapper/300:23:233
32535991210,1cyclictest0-21swapper/322:44:473
32535991210,1cyclictest0-21swapper/321:39:063
3253599120,11cyclictest15143-21df19:40:203
3253199129,2cyclictest8153-21chrt00:21:192
3253199129,2cyclictest5883-21head19:20:192
3253199129,2cyclictest5598-21ssh21:18:152
3253199129,2cyclictest32078-21grep20:15:142
3253199129,2cyclictest31068-21tr21:35:242
3253199129,2cyclictest29119-21cat20:10:002
3253199129,2cyclictest26686-21fgrep00:10:232
3253199129,2cyclictest14381-21awk19:40:002
3253199129,2cyclictest13407-21sh20:45:002
3253199129,2cyclictest11006-21cron20:40:002
3253199128,3cyclictest5109-21cat20:25:182
3253199128,3cyclictest23147-21sh21:05:002
3253199128,3cyclictest23147-21sh21:05:002
3253199128,3cyclictest14324-21sh21:24:422
3253199125,5cyclictest10029-21unixbench_multi22:05:262
3253199121,4cyclictest224-21systemd-journal23:15:002
32531991211,1cyclictest61750irq/125-enp2s0-22:47:542
3253199120,0cyclictest141rcu_preempt21:58:472
3252799129,2cyclictest8162-21ssh22:05:121
3252799129,2cyclictest7250-21awk19:25:001
3252799129,2cyclictest704-21cat22:00:001
3252799129,2cyclictest3348-21cut19:15:181
3252799129,2cyclictest32559-21sed20:15:161
3252799129,2cyclictest28412-21unixbench_multi23:25:291
3252799129,2cyclictest26291-21tr21:10:151
3252799129,2cyclictest18271-21grep00:30:181
3252799129,2cyclictest15672-21grep22:10:151
3252799129,2cyclictest11198-21wc00:25:011
3252799124,3cyclictest224-21systemd-journal21:31:301
3252799121,7cyclictest17033-21latency20:50:181
3252799120,11cyclictest8213-21cron23:35:001
3252299129,2cyclictest6319-21grep23:55:240
3252299129,2cyclictest3433-21grep20:20:290
3252299129,2cyclictest30918-21sed20:10:240
3252299129,2cyclictest30742-21head23:50:160
3252299129,2cyclictest30731-21tail22:20:160
3252299129,2cyclictest30511-21grep21:35:200
3252299129,2cyclictest24250-21grep21:30:280
3252299129,2cyclictest22996-21grep19:55:210
3252299129,2cyclictest19653-21grep19:50:150
3252299129,2cyclictest19007-21grep00:05:160
3252299129,2cyclictest1754-21cut22:00:180
3252299129,2cyclictest15693-21grep22:55:150
3252299129,2cyclictest12286-21grep19:35:130
3252299129,2cyclictest11086-21dump-pmu-power00:25:000
3252299128,3cyclictest29236-21cron23:05:000
3252299128,3cyclictest13-21ksoftirqd/000:17:170
3252299125,6cyclictest7439-21cron21:20:000
3252299121,8cyclictest0-21swapper/020:15:130
3253599119,2cyclictest0-21swapper/300:25:023
3253599118,2cyclictest8369-21grep22:05:143
3253599118,2cyclictest7040-21ssh23:33:513
3253599118,2cyclictest6014-21cat19:20:203
3253599118,2cyclictest5430-21head23:10:223
3253599118,2cyclictest4427-21tr20:25:143
3253599118,2cyclictest3987-21ssh22:46:183
3253599118,2cyclictest29747-21ssh23:50:053
3253599118,2cyclictest29293-21cut20:10:123
3253599118,2cyclictest28947-21ssh21:11:233
3253599118,2cyclictest28634-21taskset20:06:163
3253599118,2cyclictest27805-21ssh23:03:403
3253599118,2cyclictest22972-21sh22:37:193
3253599118,2cyclictest22174-21ssh21:30:063
3253599118,2cyclictest20480-21seq20:59:253
3253599118,2cyclictest20480-21seq20:59:253
3253599118,2cyclictest16763-21grep22:55:223
3253599118,2cyclictest12148-21cat20:40:163
3253599118,2cyclictest1202-21taskset20:16:373
3253599118,2cyclictest10192-21sh22:28:503
3253599117,3cyclictest2463-21turbostat19:15:003
3253599117,2cyclictest9507-21sort20:35:193
3253599110,1cyclictest0-21swapper/323:15:063
3253599110,10cyclictest19949-21ssh21:50:203
3253199118,2cyclictest4808-21aten2_r7power_c22:25:142
3253199118,2cyclictest4580-21cut21:40:142
3253199118,2cyclictest29351-21sh23:49:552
3253199118,2cyclictest2909-21ssh22:23:402
3253199118,2cyclictest27397-21grep21:10:222
3253199118,2cyclictest23738-21ssh00:09:012
3253199118,2cyclictest22052-21ssh22:14:552
3253199118,2cyclictest21501-21cron19:55:002
3253199118,2cyclictest1891-21seq23:07:562
3253199118,2cyclictest18875-21sh21:50:122
3253199118,2cyclictest17854-21cat19:45:172
3253199118,2cyclictest16934-21ssh00:03:422
3253199118,2cyclictest13435-21ssh23:38:192
3253199117,3cyclictest2164-21taskset19:11:272
3253199117,3cyclictest13731-21sed23:15:262
3253199113,4cyclictest224-21systemd-journal20:05:002
3253199110,9cyclictest0-21swapper/200:25:262
3252799118,2cyclictest7639-21grep19:25:161
3252799118,2cyclictest5798-21tail20:25:251
3252799118,2cyclictest3807-21ssh22:01:151
3252799118,2cyclictest32006-21ssh22:20:321
3252799118,2cyclictest26292-21apt-get20:05:001
3252799118,2cyclictest26103-21ssh22:40:031
3252799118,2cyclictest25227-21ssh22:16:061
3252799118,2cyclictest24339-21ssh00:35:171
3252799118,2cyclictest23392-21sed23:00:191
3252799118,2cyclictest23019-21tr23:45:151
3252799118,2cyclictest2268-21grep20:20:181
3252799118,2cyclictest20580-21awk23:20:211
3252799118,2cyclictest18826-21sh22:56:521
3252799118,2cyclictest18591-21ssh21:26:531
3252799118,2cyclictest1743-21df22:45:181
3252799118,2cyclictest15963-21cut20:50:001
3252799118,2cyclictest15422-21ssh23:40:011
3252799118,2cyclictest10999-21ssh23:35:481
3252799111,5cyclictest224-21systemd-journal20:45:001
3252799111,5cyclictest0-21swapper/100:25:361
32527991110,1cyclictest0-21swapper/122:50:161
3252299119,1cyclictest16848-21cat23:40:220
3252299118,2cyclictest9401-21chrt19:27:250
3252299118,2cyclictest9286-21cut21:20:230
3252299118,2cyclictest8202-21grep22:05:130
3252299118,2cyclictest6079-21open_inodes22:25:210
3252299118,2cyclictest5635-21needreboot21:40:190
3252299118,2cyclictest5615-21df19:20:170
3252299118,2cyclictest5296-21ssh22:47:360
3252299118,2cyclictest4655-21ssh23:10:170
3252299118,2cyclictest29496-21ssh00:39:500
3252299118,2cyclictest25185-21taskset21:05:470
3252299118,2cyclictest24408-21grep20:00:170
3252299118,2cyclictest22905-21ssh22:37:150
3252299118,2cyclictest19203-21grep20:55:170
3252299118,2cyclictest19203-21grep20:55:170
3252299118,2cyclictest15334-21sort19:40:210
3252299118,2cyclictest1451-21grep19:10:180
3252299118,2cyclictest14300-21taskset23:15:530
3252299118,2cyclictest14113-21ssh22:30:450
3252299118,2cyclictest12110-21sshd22:52:240
3252299115,2cyclictest19146-21latency_hist19:50:000
32522991110,1cyclictest13-21ksoftirqd/022:18:030
3252299110,10cyclictest20107-21ssh00:31:170
3253599108,1cyclictest36322sleep320:21:543
3253599108,1cyclictest26355-21sh00:36:103
3253599108,1cyclictest16071-21aten2_r7power_c20:50:123
3253599108,1cyclictest14641-21ls20:45:203
3253599107,2cyclictest26195-21chrt20:03:593
3253599107,2cyclictest22546-21wc21:00:263
3253599107,2cyclictest22546-21wc21:00:263
3253599107,2cyclictest17836-21wc19:45:173
3253199108,1cyclictest31057-21taskset20:11:522
3253199108,1cyclictest19176-21basename23:20:132
3253199108,1cyclictest16383-21ls20:50:132
3253199108,1cyclictest15702-21cut22:55:152
3253199108,1cyclictest14893-21expr19:40:192
3253199108,1cyclictest1248-21ssh23:30:032
3253199108,1cyclictest10920-21ssh22:51:142
3253199107,2cyclictest19461-21sort20:55:202
3253199107,2cyclictest19461-21sort20:55:202
3253199100,9cyclictest2186-21expr20:20:182
3252799108,1cyclictest8410-21seq20:32:131
3252799108,1cyclictest291612sleep120:10:061
3252799108,1cyclictest1857-21sed19:10:231
3252799107,2cyclictest22828-21taskset21:01:011
3252799107,2cyclictest22828-21taskset21:01:011
3252799107,2cyclictest19935-21wc20:55:261
3252799107,2cyclictest19935-21wc20:55:261
3252799107,2cyclictest19636-21tr19:50:151
3252799107,2cyclictest14911-21expr19:40:191
3252799107,2cyclictest10551-21cut19:30:181
3252299108,1cyclictest16906-21cat20:50:170
3252299108,1cyclictest10197-21expr19:30:160
3252299107,2cyclictest5531-21cut20:25:220
3252299107,2cyclictest12246-21cut20:40:170
325319997,1cyclictest8547-21cat19:25:212
325319990,1cyclictest0-21swapper/219:30:182
325229997,1cyclictest28656-21chrt20:06:320
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional