You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-21 - 16:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Aug 21, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2112429884,9sleep30-21swapper/307:08:063
2117429683,9sleep10-21swapper/107:08:431
2125528565,16sleep00-21swapper/007:09:460
2095328365,13sleep20-21swapper/207:05:522
21352992624,1cyclictest0-21swapper/011:27:180
2136199252,22cyclictest15502-21taskset11:51:062
2136199250,1cyclictest0-21swapper/211:09:322
21361992420,3cyclictest94-21kswapd009:55:012
2136199220,18cyclictest0-21swapper/207:20:132
2135299221,1cyclictest0-21swapper/010:49:010
21361992120,1cyclictest0-21swapper/210:00:012
2136199210,20cyclictest0-21swapper/212:13:352
297462200,0sleep10-21swapper/110:17:241
2136199200,19cyclictest0-21swapper/212:28:302
2136199191,18cyclictest559-21kworker/2:109:56:472
2136199190,19cyclictest0-21swapper/211:22:522
21356991913,6cyclictest25-21ksoftirqd/110:30:161
21352991916,1cyclictest0-21swapper/012:01:090
2135299190,18cyclictest198-21jbd2/sda2-807:20:130
2136399181,15cyclictest23813-21kworker/u8:012:00:033
2135299182,15cyclictest472-21avahi-daemon07:31:010
2135299181,16cyclictest25378-21python07:15:250
2135299180,17cyclictest0-21swapper/007:11:340
21363991717,0cyclictest0-21swapper/312:07:243
2136199170,17cyclictest0-21swapper/209:16:462
2135299174,12cyclictest0-21swapper/007:25:380
2135299163,4cyclictest224-21systemd-journal11:14:590
21352991615,1cyclictest0-21swapper/012:18:450
21352991615,1cyclictest0-21swapper/009:57:240
21352991612,3cyclictest11326-21apache209:54:440
2136199158,7cyclictest11339-21apache210:13:292
2136199150,15cyclictest18025-21kworker/2:010:40:142
2136199150,15cyclictest14390-21kworker/2:009:14:382
2136199150,15cyclictest0-21swapper/212:02:412
2135699150,15cyclictest0-21swapper/110:11:331
21352991510,3cyclictest20952-21wc11:05:220
21363991411,2cyclictest7689-21grep08:55:183
21363991411,2cyclictest2991-21cat09:55:003
21363991410,3cyclictest25842-21wc11:35:183
21363991410,2cyclictest21448-21sed08:15:183
21363991410,2cyclictest13469-21wc09:35:253
2136199148,4cyclictest32-21ksoftirqd/210:23:482
2136199146,8cyclictest1695-21ssh12:23:422
2136199143,7cyclictest0-21swapper/211:15:152
21361991413,1cyclictest26739-21kworker/2:009:49:172
21361991410,3cyclictest21873-21ls08:15:232
2135699149,3cyclictest25-21ksoftirqd/109:40:011
2135699146,5cyclictest22400-21/usr/sbin/munin09:15:231
21356991413,1cyclictest25-21ksoftirqd/112:18:021
21356991413,1cyclictest0-21swapper/110:06:451
21356991411,2cyclictest30611-21ls10:45:351
21356991411,2cyclictest17524-21grep08:05:281
21356991410,1cyclictest0-21swapper/112:32:301
2135699140,1cyclictest0-21swapper/111:00:041
2135299149,3cyclictest13045-21cat08:00:010
2135299142,3cyclictest1291-21runrttasks10:07:470
21352991412,1cyclictest0-21swapper/009:26:020
21352991410,3cyclictest2096-21head08:40:270
2135299140,13cyclictest7830-21sort09:30:260
2136399139,3cyclictest5972-21wget11:20:153
2136399139,3cyclictest32752-21ssh11:15:173
2136399139,2cyclictest29016-21grep09:20:263
2136399134,5cyclictest224-21systemd-journal11:00:003
21363991310,2cyclictest8184-21dpkg07:50:003
21363991310,2cyclictest27213-21munin-run12:40:003
21363991310,2cyclictest26225-21wc07:20:003
21363991310,2cyclictest2485-21grep09:25:263
21363991310,2cyclictest20570-21cut11:05:203
21363991310,2cyclictest17737-21tr10:35:153
21363991310,2cyclictest13559-21wc12:30:273
21363991310,2cyclictest12136-21grep07:55:223
2136399130,5cyclictest224-21systemd-journal10:05:203
2136399130,11cyclictest0-21swapper/308:35:133
2136199139,3cyclictest15768-21wc11:00:262
21361991310,2cyclictest29194-21grep08:30:212
21361991310,2cyclictest27050-21grep10:15:162
21361991310,2cyclictest11898-21cat07:55:192
21361991310,2cyclictest0-21swapper/209:30:412
2136199130,1cyclictest0-21swapper/209:25:402
2136199130,13cyclictest0-21swapper/209:35:052
2136199130,12cyclictest0-21swapper/210:59:442
2136199130,12cyclictest0-21swapper/210:35:252
2135699139,4cyclictest19302sleep110:21:011
2135699139,2cyclictest9269-21ssh10:00:161
2135699133,4cyclictest0-21swapper/109:25:141
21356991311,2cyclictest25-21ksoftirqd/111:56:181
21356991310,2cyclictest4906-21ls08:50:001
21356991310,2cyclictest31613-21cron11:14:591
21356991310,2cyclictest24702-21grep07:15:191
21356991310,2cyclictest24387-21gzip08:20:211
21356991310,2cyclictest0-21swapper/109:13:381
2135699130,12cyclictest6256-21grep10:25:171
2135699130,12cyclictest0-21swapper/109:53:561
2135299139,3cyclictest4688-21cut12:25:180
2135299139,3cyclictest13427-21sort09:05:200
2135299134,8cyclictest5435-21awk11:20:000
2135299132,7cyclictest472-21avahi-daemon11:58:520
21352991310,2cyclictest9202-21grep07:50:170
21352991310,2cyclictest5498-21grep12:05:160
21352991310,2cyclictest26632-21tr10:15:150
21352991310,2cyclictest19123-21grep09:40:200
21352991310,2cyclictest14695-21grep11:50:270
21352991310,2cyclictest11007-21wc09:00:200
2136399129,2cyclictest7457-21grep10:25:263
2136399129,2cyclictest6825-21grep08:50:283
2136399129,2cyclictest32311-21tr10:20:163
2136399129,2cyclictest31578-21sh10:20:003
2136399129,2cyclictest27657-21wc08:30:003
2136399129,2cyclictest26283-21cut11:10:153
2136399129,2cyclictest23297-21grep07:10:263
2136399129,2cyclictest21867-21head11:55:173
2136399129,2cyclictest20301-21cron11:55:013
2136399129,2cyclictest19611-21wc08:10:233
2136399129,2cyclictest18658-21sort11:30:173
2136399129,2cyclictest14029-21ls11:25:293
2136399129,2cyclictest13116-21grep09:05:173
2136399129,2cyclictest12322-21cat11:50:003
2136399129,2cyclictest11915-21wc10:30:173
2136399128,3cyclictest9985-21wc07:50:273
2136399128,3cyclictest529-21grep08:40:163
2136399128,3cyclictest29003-21grep12:20:183
2136399128,3cyclictest20513-21tr12:15:153
2136399128,2cyclictest29255-21df07:25:183
2136399120,2cyclictest0-21swapper/309:10:563
2136199129,2cyclictest27857-21gpg-connect-age08:30:012
2136199129,2cyclictest26483-21sort11:35:242
2136199129,2cyclictest22718-21tr07:10:212
2136199129,2cyclictest1892-21head07:35:202
2136199129,2cyclictest18573-21tr08:10:142
2136199129,2cyclictest15713-21cat08:05:012
2136199129,2cyclictest15713-21cat08:05:002
2136199129,2cyclictest13479-21cat11:25:232
2136199128,1cyclictest0-21swapper/210:05:212
21361991210,1cyclictest9140-21cut07:50:182
21361991210,1cyclictest16671-21head08:05:192
2136199120,3cyclictest224-21systemd-journal11:35:012
2136199120,1cyclictest0-21swapper/212:35:152
2135699129,2cyclictest7489-21grep07:45:281
2135699129,2cyclictest480-21tr08:40:151
2135699129,2cyclictest29492-21cat09:50:001
2135699129,2cyclictest26260-21chrt08:25:201
2135699129,2cyclictest23715-21munin-run07:15:001
2135699129,2cyclictest20780-21cat11:05:221
2135699129,2cyclictest20764-21seq09:41:451
2135699129,2cyclictest20340-21cat11:55:011
2135699129,2cyclictest19489-21sed10:35:271
2135699129,2cyclictest11692-21sed09:00:271
2135699129,2cyclictest1131-21tr07:35:171
2135699128,3cyclictest28092-21tr08:30:141
2135699128,3cyclictest11818-21turbostat12:09:591
2135699123,8cyclictest7302-21sh09:58:031
2135699120,1cyclictest0-21swapper/111:49:391
2135299129,2cyclictest9681-21perf09:00:000
2135299129,2cyclictest6481-21sort07:45:190
2135299129,2cyclictest24804-21chrt12:37:590
2135299129,2cyclictest21709-21wc08:15:210
2135299129,2cyclictest14307-21cut08:00:200
2135299129,2cyclictest14307-21cut08:00:200
2135299129,2cyclictest1415-21grep10:20:260
2135299129,2cyclictest12271-21basename11:50:010
2135299128,3cyclictest19782-21wc12:15:000
2135299128,2cyclictest13240-21wc12:30:240
2136399118,2cyclictest9035-21ssh09:31:423
2136399118,2cyclictest5365-21taskset10:50:513
2136399118,2cyclictest4672-21grep12:25:193
2136399118,2cyclictest4289-21ssh09:55:203
2136399118,2cyclictest31543-21tr11:40:143
2136399118,2cyclictest28787-21tail08:30:183
2136399118,2cyclictest26552-21ssh10:41:513
2136399118,2cyclictest2557-21wc07:35:273
2136399118,2cyclictest24388-21grep08:20:223
2136399118,2cyclictest23327-21ssh10:11:173
2136399118,2cyclictest20264-21ssh09:41:083
2136399118,2cyclictest18760-21ssh12:14:123
2136399118,2cyclictest141192sleep311:00:133
2136399118,2cyclictest14080-21sed08:00:193
2136399118,2cyclictest14080-21sed08:00:183
2136399118,2cyclictest11992-21ssh10:01:263
2136399117,3cyclictest23966-21grep09:45:153
2136399114,5cyclictest29672-21cpu10:45:303
2136399113,4cyclictest224-21systemd-journal07:24:593
2136399111,7cyclictest0-21swapper/309:15:243
2136399110,10cyclictest3505-21head08:45:183
2136199118,2cyclictest7754-21cat08:55:182
2136199118,2cyclictest6672-21timerandwakeup08:50:272
2136199118,2cyclictest5593-21rm11:45:022
2136199118,2cyclictest4208-21cat08:45:262
2136199118,2cyclictest25183-21cron08:25:002
2136199118,2cyclictest24917-21uname07:15:212
2136199118,2cyclictest23763-21ssh12:16:472
2136199118,2cyclictest19341-21cat12:34:592
2136199118,2cyclictest13837-21cat09:05:252
2136199118,2cyclictest12272-21cat09:05:002
2136199118,2cyclictest12029-21sh12:09:592
2136199115,2cyclictest224-21systemd-journal10:49:202
2136199113,7cyclictest24610-21sh11:56:462
2136199113,4cyclictest224-21systemd-journal11:45:012
2136199110,11cyclictest0-21swapper/209:41:362
2135699118,2cyclictest5921-21aten2_r7power_c11:20:141
2135699118,2cyclictest5797-21grep12:25:281
2135699118,2cyclictest3737-21tr07:40:161
2135699118,2cyclictest31684-21ssh09:23:471
2135699118,2cyclictest29578-21grep12:20:241
2135699118,2cyclictest27185-21ssh11:35:561
2135699118,2cyclictest24699-21ssh10:40:221
2135699118,2cyclictest2333-21ssh11:41:501
2135699118,2cyclictest21412-21head08:15:191
2135699118,2cyclictest21050-21ssh12:35:221
2135699118,2cyclictest20944-21chrt11:31:321
2135699118,2cyclictest15318-21ssh11:27:051
2135699118,2cyclictest1205-21ssh12:02:481
2135699118,2cyclictest11240-21cat07:55:151
2135699117,2cyclictest13671-21tr08:00:181
2135699117,2cyclictest13671-21tr08:00:171
21356991111,0cyclictest0-21swapper/110:58:311
2135299118,2cyclictest6887-21grep08:50:270
2135299118,2cyclictest4177-21cut07:40:180
2135299118,2cyclictest30160-21cron08:35:000
2135299118,2cyclictest27935-21ssh12:20:030
2135299118,2cyclictest26323-21ssh10:41:370
2135299118,2cyclictest25496-21wget08:25:170
2135299118,2cyclictest2548-21ssh11:42:030
2135299118,2cyclictest24066-21grep09:45:140
2135299118,2cyclictest22553-21grep10:10:270
2135299118,2cyclictest19369-21sh09:13:290
2135299118,2cyclictest18824-21cut11:30:160
2135299118,2cyclictest16669-21taskset11:01:150
2135299118,2cyclictest13-21ksoftirqd/010:00:270
2135299118,2cyclictest11107-21ssh10:56:520
2135299115,2cyclictest224-21systemd-journal09:24:200
2135299111,2cyclictest0-21swapper/009:15:130
2136399108,1cyclictest53282sleep307:41:273
2136399107,2cyclictest31861-21tail07:30:203
2136399107,2cyclictest17486-21wc08:05:273
2136199108,2cyclictest0-21swapper/208:40:202
2136199108,1cyclictest4215-21uname10:50:212
2136199108,1cyclictest3832-21expr07:40:172
2136199108,1cyclictest30654-21grep08:35:152
2136199108,1cyclictest27801-21sed09:20:152
2136199107,2cyclictest13965-21ssh10:31:072
2136199106,1cyclictest29083-21ssh11:11:572
2136199104,2cyclictest224-21systemd-journal07:29:212
2136199100,10cyclictest0-21swapper/210:25:142
2135699108,1cyclictest65832sleep110:52:211
2135699108,1cyclictest29224-21expr07:25:181
2135699108,1cyclictest26802-21expr07:20:151
2135699108,1cyclictest201582sleep108:11:571
2135699108,1cyclictest1752-21sh11:15:431
2135699108,1cyclictest12510-21cut09:05:141
2135699107,2cyclictest8313-21cat07:50:011
2135699107,2cyclictest5455-21expr08:50:161
2135699107,2cyclictest32086-21uname07:30:211
2135699107,2cyclictest12161-21rm12:10:071
2135699100,9cyclictest6502-21expr09:30:151
2135299108,1cyclictest6810-21ntp_states10:25:220
2135299108,1cyclictest24948-21taskset08:21:590
2135299108,1cyclictest17580-21sh10:35:070
2135299107,2cyclictest9136-21ssh11:22:060
2135299107,2cyclictest5532-21ssh10:51:060
2135299107,2cyclictest4849-21chrt08:49:270
2135299107,2cyclictest30392-21taskset08:35:050
2135299107,2cyclictest25961-21ntp_kernel_pll_11:35:200
2135299107,2cyclictest14011-21ssh10:31:080
2135299107,2cyclictest1168-21grep07:35:170
2135299106,2cyclictest12863-21tail09:35:200
2135299103,3cyclictest224-21systemd-journal08:10:010
213639997,1cyclictest9902-21seq09:00:013
213619997,1cyclictest32080-21uname07:30:212
213619992,1cyclictest74952unixbench-2d07:45:282
213569997,1cyclictest8157-21uniq08:55:211
213569996,2cyclictest30741-21cstates08:35:151
213529997,1cyclictest20306-21taskset08:13:520
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional