You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 09:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Dec 27, 2025 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2260229482,8sleep20-21swapper/219:06:312
2282328875,9sleep00-21swapper/019:09:220
2275428673,9sleep30-21swapper/319:08:293
2263328064,12sleep10-21swapper/119:06:541
22967992423,1cyclictest39-21ksoftirqd/320:55:143
2296799230,4cyclictest0-21swapper/320:40:133
2296299231,21cyclictest38150irq/131-snd_hda23:40:131
22967991918,1cyclictest0-21swapper/321:00:143
22967991810,8cyclictest0-21swapper/321:15:133
2295799186,6cyclictest1410-21munin-run20:40:000
22962991715,1cyclictest16774-21ntpq22:15:211
22962991713,3cyclictest18621-21cpuspeed_turbos21:15:141
2295799170,11cyclictest0-21swapper/021:15:130
22967991614,1cyclictest2883-21apt-get19:35:133
22967991612,4cyclictest39-21ksoftirqd/319:55:183
22964991616,0cyclictest0-21swapper/222:40:132
22962991615,1cyclictest0-21swapper/119:15:201
22957991611,3cyclictest4103-21munin-run00:05:000
22967991513,2cyclictest0-21swapper/322:05:143
2296499151,8cyclictest225-21systemd-journal20:21:292
2296499151,8cyclictest225-21systemd-journal20:21:292
22967991410,3cyclictest22286-21grep22:25:273
2296499149,3cyclictest29814-21tr23:50:162
22964991413,1cyclictest0-21swapper/220:45:132
22964991410,3cyclictest21971-21ls21:20:192
2296299140,6cyclictest225-21systemd-journal20:30:001
22957991410,2cyclictest15930-21wc21:10:000
2296799139,3cyclictest27695-21cron22:40:003
2296799139,3cyclictest16573-21wc23:20:243
2296799138,4cyclictest12320-21df_abs00:20:143
22967991310,2cyclictest30366-21wc20:30:213
22967991310,2cyclictest29825-21grep23:50:163
22967991310,2cyclictest28440-21cut21:35:173
22967991310,2cyclictest23996-21sed22:30:193
22967991310,2cyclictest14861-21tail00:25:173
22967991310,2cyclictest1460-21sed21:45:173
22967991310,2cyclictest11943-21grep19:50:263
2296499139,3cyclictest26869-21grep19:15:192
2296499139,2cyclictest31308-21grep22:45:192
2296499135,4cyclictest0-21swapper/223:20:002
22964991313,0cyclictest0-21swapper/200:15:152
22964991310,2cyclictest8094-21seq20:50:272
22964991310,2cyclictest7311-21grep21:55:252
22964991310,2cyclictest22858-21grep20:15:172
2296299139,3cyclictest12768-21grep19:55:141
2296299133,9cyclictest0-21swapper/119:40:231
22962991310,2cyclictest29795-21grep19:20:261
22962991310,2cyclictest19580-21dump-pmu-power23:30:001
22962991310,2cyclictest19553-21grep20:10:001
22962991310,2cyclictest12515-21wc21:00:221
2295799139,3cyclictest1494-21wc22:50:190
2295799135,4cyclictest225-21systemd-journal00:00:000
2295799134,5cyclictest225-21systemd-journal00:35:180
22957991310,2cyclictest6957-21sed19:40:260
22957991310,2cyclictest3148-21tr19:35:140
22957991310,2cyclictest30045-21grep20:30:180
22957991310,2cyclictest29783-21grep19:20:260
22957991310,2cyclictest28371-21grep21:35:170
22957991310,2cyclictest11809-21cut00:20:120
2295799130,12cyclictest8138-21grep20:50:270
2296799129,2cyclictest3133-21grep22:55:153
2296799129,2cyclictest19876-21cut20:10:133
2296799129,2cyclictest19148-21grep22:20:223
2296799129,2cyclictest16581-21uname22:15:203
2296799127,3cyclictest381rcuc/321:40:113
2296799123,6cyclictest39-21ksoftirqd/319:25:003
2296799123,4cyclictest0-21swapper/300:00:183
22967991211,1cyclictest39-21ksoftirqd/322:45:013
22967991210,1cyclictest24830-21apt-get23:40:133
22967991210,1cyclictest1984-21timerandwakeup19:30:253
2296799120,5cyclictest39-21ksoftirqd/319:13:513
2296499129,2cyclictest7744-21grep00:10:182
2296499129,2cyclictest6729-21python19:40:252
2296499129,2cyclictest5008-21sort00:05:182
2296499129,2cyclictest31401-21grep21:40:152
2296499129,2cyclictest25622-21cat19:15:002
2296499129,2cyclictest25085-21sed21:25:252
2296499129,2cyclictest2460-21head20:40:172
2296499129,2cyclictest18819-21head22:20:202
2296499129,2cyclictest16768-21sed21:10:192
2296499129,2cyclictest13870-21tr21:05:162
2296299129,2cyclictest9976-21cat19:50:001
2296299129,2cyclictest31976-21tr20:35:141
2296299129,2cyclictest23839-21sed20:15:271
2296299129,2cyclictest18737-21cat00:35:011
2296299129,2cyclictest17384-21cat23:25:001
2296299129,2cyclictest1728-21ls22:50:211
2296299129,2cyclictest16270-21tr21:10:161
2296299129,2cyclictest13791-21cat23:15:201
2296299129,2cyclictest13163-21tr22:10:141
2296299129,2cyclictest12222-21apt23:15:001
2296299129,2cyclictest11554-21wc19:50:211
2296299129,2cyclictest10719-21tr22:05:161
2296299128,3cyclictest26832-21grep22:35:221
2296299125,3cyclictest225-21systemd-journal20:15:001
2296299122,4cyclictest225-21systemd-journal00:06:181
2296299120,6cyclictest0-21swapper/122:55:141
2295799129,2cyclictest4615-21grep22:55:260
2295799129,2cyclictest29374-21cat23:49:590
2295799129,2cyclictest25112-21grep19:10:260
2295799129,2cyclictest25066-21tr23:40:140
2295799129,2cyclictest24048-21cut21:25:160
2295799128,3cyclictest17456-21basename20:05:120
2295799128,2cyclictest631-21grep19:30:130
2295799128,2cyclictest22643-21tr23:35:120
2295799126,2cyclictest225-21systemd-journal22:45:000
2295799122,9cyclictest12633-21/usr/sbin/munin19:55:290
22957991210,1cyclictest7765-21tail00:10:180
2296799118,2cyclictest8265-21wc00:10:233
2296799118,2cyclictest5699-21apache_volume21:55:113
2296799118,2cyclictest28263-21cut22:40:143
2296799118,2cyclictest27946-21sed23:45:173
2296799118,2cyclictest26380-21sort19:15:163
2296799118,2cyclictest24620-21grep20:20:133
2296799118,2cyclictest24620-21grep20:20:133
2296799118,2cyclictest24084-21tail23:35:233
2296799118,2cyclictest24033-21chrt20:16:443
2296799118,2cyclictest2178-21grep22:50:263
2296799118,2cyclictest18939-21cut00:35:133
2296799118,2cyclictest16769-21grep00:30:153
2296799118,2cyclictest1208-21grep23:55:273
2296799118,2cyclictest11877-21taskset23:10:483
2296799118,2cyclictest11384-21taskset00:18:493
2296799118,1cyclictest22551-21taskset21:20:503
2296799117,3cyclictest7023-21tail23:00:273
2296799117,1cyclictest28994-21taskset20:29:143
2296799116,2cyclictest381rcuc/320:05:153
2296799112,5cyclictest6262-21systemd00:06:183
2296799112,5cyclictest39-21ksoftirqd/320:35:173
2296799111,8cyclictest8906-21pmu-power19:45:243
2296799111,8cyclictest39-21ksoftirqd/323:25:143
2296799110,6cyclictest39-21ksoftirqd/320:45:213
2296499119,1cyclictest29582-21tr20:30:162
2296499119,1cyclictest12892-21tail21:00:272
2296499118,2cyclictest9090-21tr20:55:162
2296499118,2cyclictest4589-21sed19:35:252
2296499118,2cyclictest3767-21cat21:50:132
2296499118,2cyclictest28526-21expr19:20:152
2296499118,2cyclictest27538-21chrt20:25:162
2296499118,2cyclictest26355-21head22:35:182
2296499118,2cyclictest25961-21ls21:30:152
2296499118,2cyclictest23019-21cp22:30:012
2296499118,2cyclictest21480-21tail20:10:252
2296499118,2cyclictest20301-21expr23:30:142
2296499118,2cyclictest1858-21expr22:50:222
2296499118,2cyclictest18380-21apache_processe21:15:122
2296499118,2cyclictest16672-21grep20:00:262
2296499118,2cyclictest13657-21sort22:10:162
2296499118,2cyclictest12884-21uname00:20:192
2296499118,2cyclictest10749-21tr22:05:162
2296499117,3cyclictest20994-21cron00:40:012
2296499117,3cyclictest12217-21grep23:15:002
2296499111,6cyclictest225-21systemd-journal21:50:002
2296299119,1cyclictest7236-21grep20:50:181
2296299119,1cyclictest28306-21head23:45:211
2296299119,1cyclictest18761-21tail22:20:201
2296299118,2cyclictest8760-21cat23:05:181
2296299118,2cyclictest30956-21tr21:40:131
2296299118,2cyclictest28333-21apache_accesses21:35:151
2296299118,2cyclictest19129-21tr00:35:131
2296299118,2cyclictest1256-21grep19:30:161
2296299118,2cyclictest10204-21latency_hist22:05:001
2296299117,2cyclictest28241-21tr22:40:141
2296299114,6cyclictest8402-21python00:10:251
2296299110,10cyclictest30912-21expr19:25:151
2296299110,10cyclictest1934-21grep20:40:141
2295799118,2cyclictest8408-21cut23:05:150
2295799118,2cyclictest7044-21sed23:00:270
2295799118,2cyclictest32300-21head22:45:290
2295799118,2cyclictest31452-21grep21:40:150
2295799118,2cyclictest31272-21tail19:25:170
2295799118,2cyclictest28952-21taskset20:28:400
2295799118,2cyclictest27730-21taskset21:33:330
2295799118,2cyclictest22730-21cron22:29:590
2295799118,2cyclictest18666-21cut22:20:190
2295799118,2cyclictest15973-21taskset21:10:080
2295799118,2cyclictest15732-21cut23:20:150
2295799118,2cyclictest14504-21grep22:10:250
2295799118,2cyclictest12529-21readlink23:15:000
2295799118,2cyclictest12341-21sort23:15:000
2295799118,2cyclictest11317-21chrt00:17:580
2295799110,10cyclictest9471-21taskset19:45:430
2296799108,1cyclictest8671-21cut23:05:173
2296799108,1cyclictest8258-21taskset20:50:493
2296799108,1cyclictest26450-21tr21:30:173
2296799108,1cyclictest15612-21sed20:00:163
2296799108,1cyclictest12740-21cut23:15:143
2296799107,2cyclictest381rcuc/321:50:183
2296799107,2cyclictest31276-21grep19:25:173
2296799107,2cyclictest20335-21tr23:30:153
2296799104,4cyclictest13004-21gdbus22:10:013
2296799102,5cyclictest39-21ksoftirqd/321:09:293
2296799101,6cyclictest39-21ksoftirqd/319:40:203
2296499108,1cyclictest850-21cat19:30:142
2296499108,1cyclictest6186-21basename23:00:192
2296499108,1cyclictest3280-21sendmail_mailtr00:00:232
2296499108,1cyclictest28885-21sed21:35:182
2296499108,1cyclictest26224-21smartctl23:40:242
2296499108,1cyclictest25076-21taskset22:31:512
2296499108,1cyclictest243642sleep223:37:532
2296499108,1cyclictest1569-21cat20:40:012
2296499108,1cyclictest15585-21sh22:15:012
2296499107,2cyclictest9256-21ls19:45:282
2296499107,2cyclictest8509-21expr22:00:152
2296499107,2cyclictest7740-21chrt23:05:122
2296499107,2cyclictest31324-21head19:25:172
2296499107,2cyclictest18113-21sort23:25:172
2296499107,2cyclictest17561-21cat00:30:202
2296499107,2cyclictest16002-21taskset00:26:532
2296499107,2cyclictest15234-21egrep23:20:132
2296499107,2cyclictest13211-21df19:55:162
2296499107,2cyclictest12261-21seq19:53:212
2296499105,2cyclictest452-21systemd-logind23:55:242
2296499100,9cyclictest27772-21df_inode23:45:162
2296299108,1cyclictest9950-21grep20:55:211
2296299108,1cyclictest7620-21chrt21:57:431
2296299108,1cyclictest746-21grep23:55:231
2296299108,1cyclictest5753-21expr23:00:161
2296299108,1cyclictest24812-21expr20:20:141
2296299108,1cyclictest24812-21expr20:20:141
2296299108,1cyclictest23756-21cat19:10:151
2296299108,1cyclictest23735-21cat21:25:141
2296299108,1cyclictest22609-21chrt21:21:331
2296299108,1cyclictest19888-21find23:30:001
2296299108,1cyclictest168502sleep120:01:031
2296299108,1cyclictest158982sleep100:25:321
2296299108,1cyclictest12077-21cat00:20:131
2296299107,2cyclictest4816-21grep20:45:171
2296299107,2cyclictest31136-21taskset20:31:131
2296299107,2cyclictest30992-21cat22:45:171
2296299107,2cyclictest26471-21grep21:30:171
2296299107,2cyclictest21039-21expr22:25:161
2296299107,2cyclictest13890-21cut21:05:161
2296299107,2cyclictest10026-21grep00:15:181
2296299101,8cyclictest0-21swapper/122:30:261
2296299100,9cyclictest3230-21chrt21:50:061
2295799108,1cyclictest8291-21tr22:00:140
2295799108,1cyclictest5460-21grep20:45:220
2295799108,1cyclictest3267-21apache_volume21:50:100
2295799108,1cyclictest251652sleep022:33:000
2295799108,1cyclictest21493-21sed20:10:250
2295799108,1cyclictest175962sleep022:18:060
2295799108,1cyclictest121782sleep019:52:170
2295799107,2cyclictest5757-21cut21:55:130
2295799107,2cyclictest31451-21chrt23:52:140
2295799107,2cyclictest26333-21grep22:35:180
2295799107,2cyclictest22838-21cut20:15:170
2295799107,2cyclictest1806-21ntpq21:45:200
2295799107,2cyclictest10342-21wc20:55:270
2295799104,2cyclictest225-21systemd-journal20:04:550
2295799103,3cyclictest452-21systemd-logind00:06:180
2295799102,2cyclictest27431-21tail19:15:260
2295799101,5cyclictest21771-21head23:30:270
2295799100,9cyclictest15670-21sed00:25:250
229679997,1cyclictest8095-21apache_volume22:00:133
229679997,1cyclictest252942sleep321:27:443
229679997,1cyclictest15974-21chrt21:10:093
229649997,1cyclictest3346-21expr22:55:152
229649997,1cyclictest17466-21wget20:05:122
229629998,1cyclictest3339-21ls19:35:141
229629998,1cyclictest2298-21expr00:00:151
229629997,2cyclictest30027-21expr23:50:161
229629997,1cyclictest27072sleep121:49:061
229629997,1cyclictest22808-21ls23:35:121
229579997,1cyclictest226542sleep021:22:080
229579997,1cyclictest18043-21diskstats23:25:160
229579997,1cyclictest130352sleep021:00:570
229579997,1cyclictest11794-21find22:05:230
229579996,2cyclictest4009-21sed20:45:000
229579996,2cyclictest25873-21chrt20:20:220
229579996,2cyclictest25873-21chrt20:20:220
229579996,2cyclictest17919-21taskset00:30:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional