You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-19 - 08:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Jun 19, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23317210187,9sleep10-21swapper/119:06:111
2339129885,9sleep30-21swapper/319:07:103
2337828169,8sleep00-21swapper/019:06:580
2353527868,6sleep20-21swapper/219:09:012
2370099270,26cyclictest0-21swapper/123:55:121
2370099240,24cyclictest0-21swapper/100:33:031
2370599220,11cyclictest0-21swapper/319:50:143
2370299221,3cyclictest23534-21sh21:36:472
2370299200,19cyclictest0-21swapper/223:21:132
2369599200,18cyclictest0-21swapper/021:15:060
23702991918,1cyclictest0-21swapper/223:10:192
2370099191,13cyclictest0-21swapper/122:40:131
2370599186,10cyclictest21684-21munin-node21:45:043
23702991815,2cyclictest0-21swapper/222:45:412
2370299180,18cyclictest0-21swapper/222:00:232
2370299180,17cyclictest0-21swapper/223:15:202
2369599180,2cyclictest0-21swapper/000:10:400
2370299172,1cyclictest141rcu_preempt23:06:482
23702991713,3cyclictest8804-21cut00:30:002
2370299170,17cyclictest0-21swapper/221:52:032
2370099172,8cyclictest496-21dbus-daemon20:00:001
2369599176,6cyclictest26048-21munin-run19:15:010
2369599171,13cyclictest6222-21apache_processe19:40:130
225902170,0sleep10-21swapper/123:51:531
2370599160,15cyclictest0-21swapper/321:37:463
2370299169,6cyclictest9288-21tr00:30:172
2370299160,15cyclictest0-21swapper/221:44:582
2370299160,14cyclictest0-21swapper/223:00:052
23700991614,1cyclictest0-21swapper/100:05:221
2369599162,7cyclictest3578-21munin-run19:35:010
2369599160,1cyclictest0-21swapper/019:55:170
2369599160,14cyclictest0-21swapper/021:50:140
23705991514,1cyclictest0-21swapper/323:47:583
23705991513,1cyclictest0-21swapper/300:05:133
23705991511,2cyclictest13015-21tr21:30:143
2370599150,14cyclictest0-21swapper/323:58:103
2370299158,6cyclictest3753-21wc19:35:002
23702991514,1cyclictest0-21swapper/221:26:402
2370299150,14cyclictest0-21swapper/223:58:102
2370099153,7cyclictest4244-21cat23:39:591
2370099150,1cyclictest0-21swapper/100:17:111
2369599154,6cyclictest7969-21sed23:20:000
2370599143,1cyclictest21290-21ssh23:28:533
23705991411,2cyclictest21497-21sed22:20:253
23705991410,2cyclictest15732-21cron22:39:593
2370599140,13cyclictest0-21swapper/322:01:343
2370299149,4cyclictest17156-21wc23:25:262
23702991413,1cyclictest0-21swapper/221:31:022
23702991411,2cyclictest31183-21head20:30:182
23702991410,3cyclictest25553-21sed19:10:282
23702991410,3cyclictest23174-21wc22:45:012
23702991410,2cyclictest2023-21grep22:30:142
2370299140,14cyclictest0-21swapper/223:49:162
2370299140,14cyclictest0-21swapper/221:58:022
2370299140,12cyclictest13551-21sort19:55:182
2370099149,3cyclictest3614-21wc22:30:251
2370099145,5cyclictest224-21systemd-journal23:00:011
2370099140,11cyclictest0-21swapper/120:50:141
23695991410,3cyclictest16716-21tr22:40:140
2369599140,8cyclictest0-21swapper/021:45:140
2369599140,0cyclictest0-21swapper/023:40:060
2370599139,3cyclictest24855-21cut21:15:173
2370599136,3cyclictest22283-21idleruntime-cro00:15:013
23705991312,1cyclictest7203-21ssh00:04:473
23705991312,1cyclictest0-21swapper/321:25:003
23705991311,2cyclictest39-21ksoftirqd/321:10:383
23705991310,2cyclictest5779-21tail22:55:193
23705991310,2cyclictest17415-21grep20:00:263
23705991310,1cyclictest18929-21sed20:05:193
2370599130,13cyclictest0-21swapper/322:54:243
2370599130,12cyclictest20962-21cat23:50:273
2370299139,3cyclictest5081-21cpuspeed_turbos20:45:172
2370299139,3cyclictest4087-21tr19:35:172
2370299139,3cyclictest29295-21basename00:20:182
2370299139,3cyclictest28671-21wc20:25:222
2370299139,3cyclictest16560-21cut20:00:172
2370299139,2cyclictest0-21swapper/222:15:192
23702991310,2cyclictest32255-21tail22:50:252
23702991310,2cyclictest11172-21sed20:55:322
2370299130,13cyclictest0-21swapper/221:22:342
2370299130,12cyclictest12700-21cut21:00:192
2370099135,8cyclictest0-21swapper/121:16:341
2370099132,10cyclictest18628-21apt-config23:50:001
23700991311,2cyclictest25-21ksoftirqd/122:53:561
23700991310,2cyclictest9255-21grep23:20:241
23700991310,2cyclictest736-21cron23:15:011
23700991310,2cyclictest27053-21grep20:20:271
23700991310,2cyclictest25567-21grep19:10:281
23700991310,2cyclictest23996-21wc20:15:211
23700991310,2cyclictest17419-21grep20:00:271
23700991310,2cyclictest14498-21wc21:30:221
23700991310,2cyclictest10581-21wc21:50:231
2370099130,12cyclictest5528-21grep19:35:291
2370099130,12cyclictest0-21swapper/122:35:451
2369599139,3cyclictest10874-21wc20:55:280
2369599132,8cyclictest4797-21apache_processe20:45:140
23695991310,2cyclictest8750-21tr23:20:200
23695991310,2cyclictest8228-21cat20:50:220
23695991310,2cyclictest30962-21ssh00:20:300
23695991310,2cyclictest25557-21grep20:20:160
23695991310,2cyclictest25041-21cron20:20:000
23695991310,2cyclictest2135-21ls00:00:270
23695991310,2cyclictest13000-21sh22:15:170
2370599129,2cyclictest9689-21cut20:55:203
2370599129,2cyclictest9654-21strings23:20:263
2370599129,2cyclictest8602-21sh00:30:003
2370599129,2cyclictest6752-21grep23:40:273
2370599129,2cyclictest4221-21cron23:39:593
2370599129,2cyclictest31035-21ls19:25:003
2370599129,2cyclictest3103-21grep19:30:283
2370599129,2cyclictest28444-21sed23:10:323
2370599129,2cyclictest28211-21sed19:15:283
2370599129,2cyclictest24610-21sed20:15:283
2370599129,2cyclictest23413-21grep00:15:193
2370599129,2cyclictest16687-21date21:10:013
2370599129,2cyclictest12658-21head21:00:203
2370599128,3cyclictest29976-21munin-plugin-st22:50:003
2370599125,4cyclictest224-21systemd-journal20:20:263
23705991211,1cyclictest61750irq/125-enp2s0-22:14:383
2370599120,11cyclictest0-21swapper/321:40:453
2370299129,2cyclictest9289-21ls20:54:592
2370299129,2cyclictest6453-21tr19:40:142
2370299129,2cyclictest5040-21sleep22:09:592
2370299129,2cyclictest30243-21ssh23:35:192
2370299129,2cyclictest28958-21cut19:20:142
2370299129,2cyclictest27288-21grep19:15:192
2370299129,2cyclictest24600-21grep20:15:282
2370299129,2cyclictest20847-21cut22:20:192
2370299129,2cyclictest19352-21awk00:39:592
2370299129,2cyclictest15659-21cut00:10:182
2370299129,2cyclictest12692-21grep19:50:272
2370299129,2cyclictest10580-21sensors22:35:252
2370299128,3cyclictest23054-21tr00:15:172
2370299127,2cyclictest224-21systemd-journal21:20:012
2370099129,2cyclictest6137-21grep21:25:201
2370099129,2cyclictest5719-21grep23:40:181
2370099129,2cyclictest30872-21grep22:05:161
2370099129,2cyclictest2728-21cut21:45:181
2370099129,2cyclictest22151-21head21:35:291
2370099129,2cyclictest19326-21grep23:05:131
2370099129,2cyclictest16557-21sed23:25:211
2370099129,2cyclictest13587-21grep21:00:281
2370099129,2cyclictest13293-21df22:15:171
2370099128,3cyclictest801-21cut20:35:201
2370099128,3cyclictest12669-21ssh23:00:211
2370099128,3cyclictest11645-21cron21:00:011
2370099127,2cyclictest224-21systemd-journal20:50:001
2370099121,6cyclictest10725-21idleruntime-cro19:50:001
23700991210,1cyclictest21044-21grep20:10:151
2370099120,11cyclictest17453-21grep21:10:171
2369599129,2cyclictest4438-21cron22:55:000
2369599129,2cyclictest4324-21grep19:35:180
2369599129,2cyclictest32220-21grep19:25:170
2369599129,2cyclictest30863-21cut22:05:160
2369599129,2cyclictest28467-21sed20:25:200
2369599129,2cyclictest28180-21grep21:40:190
2369599129,2cyclictest24760-21cat22:00:200
2369599129,2cyclictest23818-21grep23:30:220
2369599129,2cyclictest17472-21chrt20:00:270
2369599129,2cyclictest17429-21cat23:25:310
2369599129,2cyclictest16516-21basename21:55:160
2369599129,2cyclictest14653-21cut21:05:160
2369599128,3cyclictest18866-21rm21:34:130
2369599121,8cyclictest14014-21unixbench_singl23:45:330
23695991210,2cyclictest13-21ksoftirqd/022:45:290
2369599120,1cyclictest0-21swapper/023:39:560
2369599120,1cyclictest0-21swapper/020:10:140
2370599119,1cyclictest16718-21cut22:40:153
2370599119,1cyclictest12311-21grep23:00:193
2370599118,2cyclictest8873-21grep20:50:283
2370599118,2cyclictest7458-21sendmail_mailtr21:25:273
2370599118,2cyclictest5751-21taskset19:37:443
2370599118,2cyclictest4198-21taskset20:40:513
2370599118,2cyclictest4044-21ssh22:09:033
2370599118,2cyclictest29901-21cut00:20:203
2370599118,2cyclictest28424-21grep20:25:203
2370599118,2cyclictest24170-21cat23:30:253
2370599118,2cyclictest2128-21cron20:40:003
2370599118,2cyclictest20188-21cat23:05:183
2370599118,2cyclictest18352-21sed21:55:273
2370599118,2cyclictest1757-21ssh22:30:073
2370599118,2cyclictest16251-21ssh00:35:313
2370599118,2cyclictest12522-21ssh00:32:293
2370599118,2cyclictest11857-21ssh21:51:103
2370599117,3cyclictest1213-21wc19:30:013
2370599111,9cyclictest2136-21hddtemp_smartct23:15:203
2370599110,10cyclictest0-21swapper/322:25:203
2370299118,2cyclictest9555-21ssh22:58:042
2370299118,2cyclictest8705-21cron21:49:592
2370299118,2cyclictest8403-21cut00:05:152
2370299118,2cyclictest28109-21head22:25:202
2370299118,2cyclictest25555-21ssh23:31:052
2370299118,2cyclictest2062-21taskset20:39:112
2370299118,2cyclictest1103-21tr00:00:192
2370299117,3cyclictest226202sleep220:14:352
2370299113,7cyclictest16503-21diskmemload22:10:332
2370299112,5cyclictest19372-21aten2_r7power_v23:50:172
2370299112,3cyclictest224-21systemd-journal23:40:012
2370099118,2cyclictest759-21ssh21:20:271
2370099118,2cyclictest6229-21ssh23:18:241
2370099118,2cyclictest5026-21sh00:25:411
2370099118,2cyclictest3940-21sh00:01:401
2370099118,2cyclictest3582-21awk19:35:001
2370099118,2cyclictest31740-21grep20:30:251
2370099118,2cyclictest29831-21grep00:20:201
2370099118,2cyclictest29374-21tail20:25:291
2370099118,2cyclictest289212sleep119:20:141
2370099118,2cyclictest23909-21cut22:45:181
2370099118,2cyclictest22727-21sh23:30:031
2370099118,2cyclictest18284-21grep20:05:161
2370099118,2cyclictest17601-21taskset00:37:161
2370099118,2cyclictest16524-21grep21:55:161
2370099118,2cyclictest16051-21tail21:05:281
2370099118,2cyclictest12175-21cut19:50:221
2370099118,2cyclictest10696-21ssh22:13:341
2370099115,2cyclictest224-21systemd-journal00:10:011
2370099112,5cyclictest28183-21unixbench_multi19:15:281
2370099110,11cyclictest0-21swapper/122:26:121
2370099110,10cyclictest26750-21ssh22:01:341
2369599118,2cyclictest9804-21cut22:35:180
2369599118,2cyclictest8130-21ssh00:05:130
2369599118,2cyclictest5439-21ssh22:10:100
2369599118,2cyclictest4794-21ssh22:55:070
2369599118,2cyclictest3241-21tr22:30:210
2369599118,2cyclictest30517-21date19:20:270
2369599118,2cyclictest29367-21sh23:56:390
2369599118,2cyclictest28711-21head22:25:260
2369599118,2cyclictest23348-21ssh21:36:370
2369599118,2cyclictest23176-21expr00:15:170
2369599118,2cyclictest22443-21ssh22:20:480
2369599118,2cyclictest20764-21ssh21:12:070
2369599118,2cyclictest19897-21head23:05:150
2369599118,2cyclictest15259-21ssh23:01:470
2369599118,2cyclictest14785-21grep00:35:180
2369599118,2cyclictest12321-21ssh00:32:130
2369599114,5cyclictest26471-21/usr/sbin/munin23:10:210
2369599110,10cyclictest13174-21cut19:55:000
133262111,5sleep313437-21tune2fs22:15:183
2370599108,1cyclictest5246-21expr20:45:173
2370599108,1cyclictest30069-21cat20:30:103
2370599108,1cyclictest24078-21grep19:10:163
2370599108,1cyclictest13385-21wget19:55:173
2370599107,2cyclictest8577-21basename19:45:133
2370599107,2cyclictest6936-21grep19:40:173
2370599100,9cyclictest22874-21cat20:14:593
2370299108,1cyclictest199672sleep220:06:022
2370299108,1cyclictest17174-21ssh21:10:142
2370299107,2cyclictest31821-21wc19:25:162
2370299107,2cyclictest26461-21awk20:20:212
2370299106,2cyclictest4418-21chrt20:43:402
2370099108,1cyclictest20279-21chrt22:20:171
2370099107,2cyclictest6191-21taskset19:40:101
2370099107,2cyclictest3037-21grep20:40:151
2370099107,2cyclictest28059-21grep21:40:191
2370099107,2cyclictest1011-21taskset19:27:571
2369599108,1cyclictest8799-21sort19:45:160
2369599108,1cyclictest4843-21ssh00:25:290
2369599108,1cyclictest31220-21chrt21:20:120
2369599108,1cyclictest282892sleep019:15:430
2369599108,1cyclictest2397-21cat20:40:110
2369599108,1cyclictest19052-21rm23:50:060
2369599108,1cyclictest18144-21chrt20:05:080
2369599108,1cyclictest12420-21expr21:00:170
2369599107,2cyclictest9528-21ssh21:27:070
2369599107,2cyclictest730-21grep20:35:200
2369599107,2cyclictest32336-21wc20:35:010
237029996,2cyclictest10394-21chrt19:46:072
237029991,3cyclictest15575-21pmu-power21:05:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional