You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-15 - 22:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Dec 15, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2828121330,0sleep20-21swapper/208:25:192
2215929685,6sleep00-21swapper/007:05:560
2244528875,9sleep20-21swapper/207:09:392
2229428158,19sleep10-21swapper/107:07:411
2231127865,9sleep30-21swapper/307:07:553
22566992825,2cyclictest18510-21cpuspeed_turbos10:20:141
22566992624,1cyclictest27273-21cron07:19:591
22566992617,7cyclictest24599-21fschecks_count11:40:161
22566992321,1cyclictest10287-21df_inode11:10:151
22566992217,4cyclictest23727-21run-parts11:40:001
22566992116,4cyclictest7974-21wc12:15:001
22566992112,6cyclictest9066-21mktemp08:55:011
22566992111,9cyclictest21185-21grep08:10:181
2257299200,17cyclictest0-21swapper/308:40:133
22566992017,3cyclictest0-21swapper/110:00:131
22566992017,2cyclictest0-21swapper/109:10:131
22566992015,4cyclictest31816-21grep07:25:261
22566992015,4cyclictest0-21swapper/110:35:171
22566991913,3cyclictest30432-21sort08:30:171
22566991913,3cyclictest0-21swapper/108:25:181
22568991814,2cyclictest0-21swapper/208:05:002
22566991816,1cyclictest18747-21kworker/1:011:30:001
22566991812,5cyclictest0-21swapper/108:45:001
2256399186,6cyclictest24605-21basename08:20:000
2256399180,16cyclictest0-21swapper/007:35:150
2256699178,5cyclictest0-21swapper/111:55:151
22566991715,1cyclictest0-21swapper/110:05:131
22566991714,2cyclictest0-21swapper/110:55:141
22566991714,1cyclictest21475-21apt-get09:20:131
22566991712,4cyclictest0-21swapper/109:00:151
22566991711,4cyclictest30884-21latency_hist11:55:001
2256699170,9cyclictest0-21swapper/110:45:141
2256699169,6cyclictest0-21swapper/112:35:171
2256699169,5cyclictest14352-21latency_hist11:20:001
2256699167,7cyclictest0-21swapper/107:55:151
22566991616,0cyclictest0-21swapper/109:45:121
22566991611,4cyclictest31156-21sort09:40:001
22566991611,4cyclictest0-21swapper/112:15:221
22566991610,5cyclictest0-21swapper/107:10:131
2256699160,0cyclictest0-21swapper/111:45:271
2256399162,8cyclictest16608-21awk11:25:000
2256399162,7cyclictest5579-21munin-run12:10:000
2256699159,5cyclictest0-21swapper/109:35:001
2256699154,4cyclictest0-21swapper/110:50:161
22566991512,2cyclictest14454-21/usr/sbin/munin11:20:191
22566991512,2cyclictest0-21swapper/111:00:211
22566991510,4cyclictest29045-21pmu-power10:40:221
22566991510,4cyclictest0-21swapper/110:35:001
2256399151,13cyclictest6671-21fgrep07:40:220
22563991510,3cyclictest13358-21grep09:00:270
2256399150,2cyclictest0-21swapper/011:55:140
22572991411,2cyclictest22146-21cat10:25:253
22572991411,2cyclictest19812-21cron08:10:003
22568991411,2cyclictest4570-21grep12:05:202
22568991411,2cyclictest20094-21cron12:40:002
22568991410,3cyclictest7347-21wc12:10:252
2256699142,11cyclictest19837-21cat09:15:181
22566991412,1cyclictest38150irq/131-snd_hda09:30:001
22566991411,1cyclictest0-21swapper/108:00:141
2256399149,3cyclictest26713-21cat07:15:260
2256399141,7cyclictest29683-21munin-run07:25:000
22563991410,3cyclictest23265-21sed11:35:260
2257299139,3cyclictest8030-21sed11:05:163
2257299139,3cyclictest39-21ksoftirqd/308:45:153
2257299139,3cyclictest16618-21grep10:15:173
2257299138,4cyclictest1035-21packagekitd11:46:093
2257299137,3cyclictest537-21snmpd10:59:463
2257299133,2cyclictest225-21systemd-journal11:30:003
22572991310,2cyclictest5760-21wc12:10:003
22572991310,2cyclictest5189-21ls09:50:243
2257299130,12cyclictest7535-21sed12:10:263
2256899139,3cyclictest9484-21grep07:45:242
2256899139,3cyclictest4456-21head09:50:152
2256899139,3cyclictest30417-21grep10:45:152
2256899139,3cyclictest28052-21tr10:40:152
2256899139,3cyclictest22207-21grep09:20:162
2256899134,5cyclictest225-21systemd-journal10:25:002
22568991310,2cyclictest3335-21cut08:40:222
22568991310,2cyclictest31707-21cron08:35:002
22568991310,2cyclictest2602-21grep09:45:202
22568991310,2cyclictest19420-21grep08:05:272
22568991310,2cyclictest16157-21grep10:15:142
22568991310,2cyclictest14347-21grep07:55:262
2256899130,2cyclictest0-21swapper/210:55:142
2256699139,3cyclictest15828-21grep09:05:271
2256699138,4cyclictest9882-21idleruntime-cro07:50:001
2256699136,6cyclictest3196-21latency_hist12:05:001
2256699135,8cyclictest0-21swapper/110:25:441
22566991311,1cyclictest14340-21/usr/sbin/munin10:10:181
22566991310,2cyclictest5846-21sh09:55:001
22566991310,2cyclictest4069-21fschecks_count12:05:171
22566991310,2cyclictest24748-21wc08:20:001
22566991310,2cyclictest20483-21wc11:30:201
22566991310,2cyclictest0-21swapper/108:10:001
2256399134,8cyclictest29924-21munin-run10:45:000
22563991310,2cyclictest3176-21grep08:40:210
22563991310,2cyclictest28665-21cat11:50:000
22563991310,2cyclictest27115-21grep10:35:260
22563991310,2cyclictest2597-21grep09:45:200
22563991310,2cyclictest23090-21grep09:20:250
2257299129,2cyclictest9206-21head12:15:183
2257299129,2cyclictest8878-21basename08:55:003
2257299129,2cyclictest5621-21tr07:40:143
2257299129,2cyclictest3439-21head07:35:183
2257299129,2cyclictest32342-21grep11:55:223
2257299129,2cyclictest30444-21grep11:50:263
2257299129,2cyclictest28348-21head07:20:153
2257299129,2cyclictest27370-21uname09:30:183
2257299129,2cyclictest27274-21tr08:25:133
2257299129,2cyclictest24112-21cut09:25:143
2257299129,2cyclictest23116-21sed09:20:253
2257299129,2cyclictest18915-21cut10:20:163
2257299129,2cyclictest13354-21grep12:25:143
2257299129,2cyclictest12555-21grep10:05:253
2257299129,2cyclictest12371-21grep11:15:153
2257299128,3cyclictest24174-21tr10:30:213
2257299128,3cyclictest12452-21grep09:00:183
2257299128,2cyclictest25720-21df07:15:163
2257299121,8cyclictest0-21swapper/311:40:133
2257299120,2cyclictest0-21swapper/311:30:123
2256899129,2cyclictest8851-21tr10:00:142
2256899129,2cyclictest3192-21cron12:05:002
2256899129,2cyclictest30868-21grep07:25:172
2256899129,2cyclictest25971-21awk09:30:002
2256899129,2cyclictest22386-21sed10:25:262
2256899129,2cyclictest10877-21tail07:50:162
2256899128,3cyclictest21346-21sleep11:35:002
2256899128,3cyclictest17955-21grep11:25:172
2256899128,3cyclictest13815-21grep10:10:152
22568991212,0cyclictest0-21swapper/211:35:272
2256699129,2cyclictest5091-21awk07:40:001
2256699129,2cyclictest15866-21tr12:30:151
2256699129,2cyclictest0-21swapper/107:50:111
2256699124,7cyclictest2418-21cron07:35:001
2256699121,10cyclictest267242sleep108:21:581
2256699120,8cyclictest0-21swapper/107:40:171
2256399129,2cyclictest9084-21grep12:15:170
2256399129,2cyclictest7351-21sort08:50:160
2256399129,2cyclictest32699-21grep08:35:160
2256399129,2cyclictest27342-21tr08:25:140
2256399129,2cyclictest27165-21grep09:30:160
2256399129,2cyclictest21736-21grep08:10:240
2256399129,2cyclictest18607-21munin-run09:15:000
2256399129,2cyclictest18285-21grep12:35:150
2256399129,2cyclictest13-21ksoftirqd/012:00:190
2256399129,2cyclictest12824-21grep11:15:180
2256399129,2cyclictest10473-21head11:10:170
2256399129,2cyclictest1024-21grep07:30:170
2256399128,3cyclictest15855-21grep08:00:170
2256399121,7cyclictest0-21swapper/010:30:000
2256399120,11cyclictest30883-21head07:25:170
2257299118,2cyclictest8055-21expr07:45:133
2257299118,2cyclictest6660-21taskset11:00:353
2257299118,2cyclictest575-21grep07:30:143
2257299118,2cyclictest31058-21cat08:30:243
2257299118,2cyclictest30466-21tr10:45:153
2257299118,2cyclictest29746-21sed09:35:203
2257299118,2cyclictest28075-21grep10:40:153
2257299118,2cyclictest27390-21chrt10:38:233
2257299118,2cyclictest24118-21awk07:10:183
2257299118,2cyclictest22718-21cut08:15:143
2257299118,2cyclictest22338-21grep11:35:173
2257299118,2cyclictest20367-21cut08:10:133
2257299118,2cyclictest19553-21tune2fs09:15:163
2257299118,2cyclictest15632-21sh10:15:003
2257299118,2cyclictest1252-21uname10:50:223
2257299118,2cyclictest10178-21expr11:10:143
2257299118,2cyclictest10073-21head08:55:173
2257299117,3cyclictest1668-21sh08:40:003
2256899118,2cyclictest7416-21tail08:50:172
2256899118,2cyclictest6351-21tr09:55:142
2256899118,2cyclictest5454-21grep07:40:142
2256899118,2cyclictest5052-21head08:45:182
2256899118,2cyclictest4582-21chrt07:37:192
2256899118,2cyclictest32729-21sed08:35:162
2256899118,2cyclictest31423-21chrt09:40:122
2256899118,2cyclictest30909-21cron09:40:002
2256899118,2cyclictest30106-21run-parts11:50:222
2256899118,2cyclictest27180-21taskset10:35:392
2256899118,2cyclictest25465-21sort08:20:162
2256899118,2cyclictest25211-21cut07:15:142
2256899118,2cyclictest24673-21sed10:30:262
2256899118,2cyclictest21189-21grep08:10:182
2256899118,2cyclictest1736-21sed10:50:282
2256899118,2cyclictest17236-21grep09:10:162
2256899118,2cyclictest16301-21sort12:30:172
2256899118,2cyclictest14821-21grep09:05:182
2256899118,2cyclictest13118-21wc11:15:202
2256899118,2cyclictest12827-21taskset12:23:062
2256899118,2cyclictest10401-21grep11:10:162
2256899115,2cyclictest225-21systemd-journal11:45:002
2256899113,4cyclictest225-21systemd-journal11:59:552
2256699117,3cyclictest0-21swapper/110:15:131
2256699116,5cyclictest0-21swapper/112:20:121
2256699115,5cyclictest11461-21basename09:00:001
2256699115,4cyclictest8225-21latency_hist10:00:001
2256699111,9cyclictest4885-21sed08:45:161
2256699111,6cyclictest0-21swapper/109:40:021
2256399119,2cyclictest0-21swapper/008:30:130
2256399118,2cyclictest8472-21tr11:05:190
2256399118,2cyclictest8095-21expr07:45:130
2256399118,2cyclictest6922-21taskset11:03:570
2256399118,2cyclictest31872-21sed10:45:270
2256399118,2cyclictest29522-21grep11:50:170
2256399118,2cyclictest28854-21aten2_r7power_e09:35:140
2256399118,2cyclictest26215-21wc11:45:000
2256399118,2cyclictest2439-21cat10:55:010
2256399118,2cyclictest24261-21awk07:10:210
2256399118,2cyclictest19395-21grep10:20:190
2256399118,2cyclictest16827-21basename11:25:090
2256399118,2cyclictest16409-21cat09:10:010
2256399118,2cyclictest13631-21cut10:10:140
2256399118,2cyclictest12320-21cut10:05:220
2256399118,2cyclictest11115-21tr12:20:130
2256399117,3cyclictest3162-21grep10:55:170
2256399110,1cyclictest0-21swapper/009:40:120
2256399110,10cyclictest13447-21expr07:55:170
2257299108,1cyclictest30471-21ls07:25:153
2257299108,1cyclictest24841-21cat08:20:113
2257299108,1cyclictest18756-21expr12:35:173
2257299108,1cyclictest16538-21cut09:10:133
2257299108,1cyclictest15615-21cat12:30:123
2257299108,1cyclictest14521-21taskset07:56:023
2257299108,1cyclictest14067-21taskset09:05:023
2257299107,2cyclictest7601-21cat09:55:253
2257299107,2cyclictest3391-21taskset09:49:033
2257299107,2cyclictest17314-21grep08:05:003
2257299107,2cyclictest14546-21basename11:20:143
2257299107,2cyclictest11480-21sort12:20:153
2257299107,2cyclictest10215-21grep10:00:273
2257299100,9cyclictest31564-21wc09:40:133
2256899108,1cyclictest498-21grep07:30:142
2256899108,1cyclictest282872sleep209:31:502
2256899108,1cyclictest24734-21chrt07:13:032
2256899108,1cyclictest24321-21taskset08:16:522
2256899108,1cyclictest116382sleep210:05:162
2256899108,1cyclictest102482sleep212:15:462
2256899107,2cyclictest8087-21hostname11:05:172
2256899107,2cyclictest28314-21grep07:20:152
2256899107,2cyclictest19093-21grep09:15:132
2256899107,2cyclictest13355-21cut12:25:142
2256899104,2cyclictest225-21systemd-journal11:45:002
2256899103,2cyclictest0-21swapper/211:20:142
2256699108,1cyclictest27866-21wc07:20:131
2256699103,3cyclictest225-21systemd-journal11:05:071
2256699102,4cyclictest739-21sshd08:35:211
2256399108,1cyclictest8634-21cut10:00:140
2256399108,1cyclictest7840-21chrt12:13:480
2256399108,1cyclictest6553-21cat09:55:150
2256399108,1cyclictest5396-21cat08:45:200
2256399108,1cyclictest3704-21taskset09:50:060
2256399108,1cyclictest18940-21cut09:15:130
2256399108,1cyclictest16245-21expr10:15:150
2256399107,2cyclictest9450-21seq08:55:140
2256399107,2cyclictest24284-21cat09:25:140
2256399107,2cyclictest17433-21taskset12:31:180
2256399107,2cyclictest13304-21cut12:25:130
2256399107,2cyclictest11644-21cat07:50:240
2256399106,1cyclictest195812sleep008:07:020
2256399100,6cyclictest24151-21needreboot10:30:200
225729997,1cyclictest1806-21tail12:00:163
225729996,2cyclictest11524-21grep07:50:213
225689997,1cyclictest12889-21ntpq09:00:212
225689996,2cyclictest6002-21cat11:00:192
225689990,7cyclictest0-21swapper/208:55:132
225669996,2cyclictest13582-21expr12:25:141
225639997,1cyclictest248312sleep008:20:080
225639990,8cyclictest13-21ksoftirqd/011:30:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional