You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-14 - 16:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Dec 14, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2754328874,9sleep00-21swapper/007:09:060
2756128774,9sleep30-21swapper/307:09:213
2728928644,38sleep20-21swapper/207:05:472
2727827659,13sleep10-21swapper/107:05:381
2770799320,31cyclictest0-21swapper/309:40:133
2770499300,28cyclictest0-21swapper/209:40:132
2770499292,26cyclictest30193-21idleruntime-cro09:30:002
2770499290,28cyclictest0-21swapper/207:35:142
2770799283,24cyclictest30192-21cron09:30:003
2770799280,25cyclictest0-21swapper/307:35:143
2770499280,28cyclictest0-21swapper/210:35:132
2770799270,24cyclictest11855-21fschecks_time12:10:163
2770799270,1cyclictest0-21swapper/309:20:143
2770799261,24cyclictest25222-21munin-run09:20:003
2770799261,24cyclictest17355-21/usr/sbin/munin11:15:133
2770799260,1cyclictest0-21swapper/307:20:153
2770799250,1cyclictest0-21swapper/310:35:133
2770799241,21cyclictest1417-21latency_hist08:30:013
2770799240,23cyclictest0-21swapper/308:00:123
2770499240,18cyclictest0-21swapper/207:20:152
2770799231,21cyclictest14901-21wc08:55:223
2770799230,21cyclictest23726-21wc11:25:273
27704992315,7cyclictest0-21swapper/211:15:132
2770499230,0cyclictest0-21swapper/209:20:142
2770299230,23cyclictest0-21swapper/111:16:291
2770799224,15cyclictest30261-21awk10:35:013
2770799224,13cyclictest0-21swapper/311:05:133
2770799221,20cyclictest13330-21latency_hist08:55:003
2770799220,21cyclictest2520-21ls09:40:003
2770499223,18cyclictest1605-21grep09:35:222
2770499222,16cyclictest14900-21fgrep08:55:222
2770499221,19cyclictest17455-21wc09:00:262
2770299221,20cyclictest0-21swapper/109:40:181
2770799211,19cyclictest9283-21threads10:55:253
2770799210,16cyclictest11654-21gzip09:55:233
2770799210,16cyclictest11654-21gzip09:55:223
2770499212,18cyclictest25292-21perf10:25:002
2770499211,18cyclictest0-21swapper/207:50:252
2770499210,20cyclictest0-21swapper/208:00:122
2770499210,18cyclictest0-21swapper/212:10:162
2770299210,20cyclictest0-21swapper/111:00:141
2770799208,9cyclictest30996-21taskset08:22:243
2770799200,19cyclictest0-21swapper/307:25:163
2770499201,18cyclictest9284-21awk10:55:252
2770499200,19cyclictest0-21swapper/207:45:122
2770799191,17cyclictest10249-21/usr/sbin/munin08:45:253
2770799190,1cyclictest0-21swapper/311:35:133
2770799190,18cyclictest23208-21wget10:20:153
2770799190,18cyclictest17111-21awk09:00:213
2770799190,18cyclictest0-21swapper/312:00:273
2770799190,16cyclictest0-21swapper/307:40:153
2770499192,16cyclictest0-21swapper/210:00:002
2770499192,16cyclictest0-21swapper/210:00:002
2770799183,14cyclictest0-21swapper/307:45:123
2770499189,8cyclictest1235-21ntpd08:40:212
2770499182,14cyclictest0-21swapper/212:00:262
2770499180,2cyclictest0-21swapper/208:25:152
27702991811,6cyclictest30027-21cron07:14:591
2770799171,15cyclictest9463-21fw_packets12:05:183
2770799171,15cyclictest5629-21grep08:35:263
2770799171,15cyclictest26028-21/usr/sbin/munin11:30:243
2770799170,1cyclictest20265-21df11:20:163
2770799170,1cyclictest0-21swapper/312:35:273
2770799170,1cyclictest0-21swapper/312:20:133
2770799170,1cyclictest0-21swapper/310:25:183
2770799170,1cyclictest0-21swapper/308:30:163
2770499178,6cyclictest15181-21cat10:05:002
2770499173,13cyclictest0-21swapper/209:30:132
2770499170,1cyclictest0-21swapper/211:25:182
2770499170,16cyclictest13390-21ls08:54:592
2770499170,14cyclictest25220-21apt-get09:20:002
2769799171,7cyclictest225-21systemd-journal08:15:000
2770799165,7cyclictest6404-21taskset08:40:123
2770799165,7cyclictest20746-21egrep10:15:143
2770799165,6cyclictest18950-21grep12:25:163
2770799163,12cyclictest20101-21perf10:15:003
2770799162,13cyclictest26011-21grep08:10:273
2770799161,14cyclictest22341-21unixbench_multi12:30:253
2770799161,14cyclictest0-21swapper/311:10:183
2770799160,1cyclictest14057-21grep12:15:163
2770799160,15cyclictest0-21swapper/309:50:153
2770799160,14cyclictest0-21swapper/310:45:193
2770499164,11cyclictest20084-21apt-get10:15:002
2770499162,13cyclictest5628-21grep08:35:262
2770499161,14cyclictest13768-21grep11:05:202
2770499160,1cyclictest0-21swapper/207:40:152
2770499160,15cyclictest24420-21mailstats12:35:232
2770499160,14cyclictest0-21swapper/212:15:172
2770499160,0cyclictest0-21swapper/212:05:182
2770299160,1cyclictest23100-21aten2_r7power_c12:35:141
2770299160,15cyclictest15707-21sleep08:59:591
2769799164,6cyclictest20465-21basename12:30:000
2770799158,6cyclictest31156-21cut07:15:173
2770799158,6cyclictest2544-21taskset10:44:343
2770799158,6cyclictest15830-21tr10:05:153
2770799158,6cyclictest12073-21taskset11:01:193
2770799157,7cyclictest22198-21tr08:05:133
2770799157,7cyclictest22198-21tr08:05:133
2770799152,12cyclictest1142-21runrttasks08:19:443
2770799150,1cyclictest0-21swapper/307:50:263
2770799150,1cyclictest0-21swapper/307:10:113
2770799150,14cyclictest0-21swapper/309:50:003
2770499159,5cyclictest21354-21wc11:20:262
2770499159,5cyclictest11292-21tail11:00:202
2770499158,6cyclictest7258-21chrt07:33:562
2770499158,6cyclictest22014-21basename08:05:142
2770499158,6cyclictest22014-21basename08:05:132
2770499157,7cyclictest7382-21apt-config09:50:002
2770499151,12cyclictest20728-21apache_volume09:10:132
2770499150,1cyclictest0-21swapper/211:30:242
2770499150,13cyclictest0-21swapper/207:15:012
2770499150,12cyclictest0-21swapper/210:40:202
2770499150,12cyclictest0-21swapper/207:25:172
27702991513,1cyclictest14322-21sed11:05:261
2769799156,7cyclictest0-21swapper/012:20:100
2769799151,8cyclictest30058-21awk07:15:000
2770799149,3cyclictest2326-21tr11:50:153
2770799147,6cyclictest7117-21chrt10:52:223
2770799147,6cyclictest19101-21taskset07:57:403
2770799147,6cyclictest13812-21hostname10:00:183
2770799147,6cyclictest1071-21taskset11:46:323
2770799146,7cyclictest6471-21head07:30:223
2770799146,7cyclictest19804-21cat09:05:263
2770799141,12cyclictest537-21snmpd11:44:263
2770799141,12cyclictest3964-21/usr/sbin/munin11:55:203
2770799141,12cyclictest3964-21/usr/sbin/munin11:55:203
2770799141,12cyclictest20694-21/usr/sbin/munin09:10:263
2770799141,11cyclictest20900-21munin-node09:30:433
2770499149,2cyclictest0-21swapper/210:50:142
2770499148,5cyclictest3106-21sort11:50:232
2770499147,6cyclictest26187-21cut10:25:162
27704991413,1cyclictest0-21swapper/211:45:132
2770499140,13cyclictest26012-21unixbench_singl08:10:262
2770499140,13cyclictest0-21swapper/212:20:122
2770499140,13cyclictest0-21swapper/211:10:172
2770299147,6cyclictest7343-21basename10:54:591
2769799144,9cyclictest31424-21perf11:45:000
27697991410,3cyclictest22096-21tr08:05:130
27697991410,3cyclictest22096-21tr08:05:130
2770499139,3cyclictest3079-21tr10:45:182
2770499138,4cyclictest29991-21grep11:40:182
2770499138,4cyclictest19098-21sed12:25:162
2770499137,5cyclictest3400-21chrt08:30:372
2770499137,5cyclictest28921-21cut10:30:172
2770499137,5cyclictest22301-21awk10:15:272
2770499136,6cyclictest17431-21taskset10:06:352
2770499132,10cyclictest30958-21seq08:21:542
2770499131,11cyclictest0-21swapper/211:55:122
2770499131,11cyclictest0-21swapper/211:55:122
2770499130,1cyclictest0-21swapper/211:35:132
2770499130,0cyclictest0-21swapper/208:45:252
2770299139,3cyclictest9992-21wc09:55:001
2770299139,3cyclictest4235-21tr11:55:141
2770299139,3cyclictest4235-21tr11:55:131
2770299139,3cyclictest24804-21tr11:30:131
2770299132,10cyclictest38150irq/131-snd_hda08:30:131
27702991310,2cyclictest6527-21cut12:00:141
27702991310,2cyclictest2700-21sort10:45:001
27702991310,2cyclictest22705-21grep11:25:161
2770299130,12cyclictest25807-21grep10:25:151
2769799139,3cyclictest25615-21grep11:30:180
2769799139,2cyclictest15390-21cat10:05:000
27697991310,2cyclictest2607-21basename10:44:590
27697991310,2cyclictest15699-21sleep12:20:000
2769799130,12cyclictest5709-21head11:55:260
2769799130,12cyclictest5709-21head11:55:250
2769799130,12cyclictest23802-21grep12:35:170
2769799130,12cyclictest17464-21tr07:55:150
2770499126,5cyclictest9804-21taskset09:54:202
2770499126,5cyclictest20992-21tr12:30:132
2770499121,8cyclictest225-21systemd-journal08:20:002
2770499121,10cyclictest0-21swapper/207:55:212
2770499120,11cyclictest0-21swapper/207:15:262
2770299129,2cyclictest9239-21sed07:35:261
2770299129,2cyclictest3733-21apt11:55:001
2770299129,2cyclictest31173-21sort07:15:171
2770299129,2cyclictest28797-21grep09:25:161
2770299129,2cyclictest17578-21sed12:20:321
2770299129,2cyclictest1724-21grep07:20:231
2770299129,2cyclictest13115-21wget10:00:141
2770299128,3cyclictest32317-21grep11:45:161
2770299128,3cyclictest2642-21date09:40:001
2770299128,2cyclictest18112-21grep10:10:131
2770299127,4cyclictest4366-21ls10:45:281
2770299127,3cyclictest17270-21ls11:15:001
2769799129,2cyclictest7354-21sleep10:54:590
2769799129,2cyclictest7172-21head08:40:170
2769799129,2cyclictest3643-21head10:45:190
2769799129,2cyclictest32114-21grep10:35:270
2769799129,2cyclictest31774-21cut11:45:130
2769799129,2cyclictest2290-21grep08:30:170
2769799129,2cyclictest20361-21grep08:00:150
2769799129,2cyclictest17014-21cat10:05:250
2769799129,2cyclictest11887-21grep11:00:260
2769799129,2cyclictest1149-21grep09:35:170
2769799129,2cyclictest11260-21tr08:50:150
2769799128,3cyclictest27735-21cat08:15:200
2769799121,5cyclictest7375-21perf07:35:000
2769799120,11cyclictest1090-21grep07:20:180
2770499114,6cyclictest19274-21grep09:05:202
2770299119,1cyclictest4242-21tr08:35:151
2770299118,2cyclictest9041-21tr12:05:151
2770299118,2cyclictest6204-21cat07:30:201
2770299118,2cyclictest28401-21grep10:30:151
2770299118,2cyclictest27640-21head11:35:161
2770299118,2cyclictest27347-21grep08:15:171
2770299118,2cyclictest21944-21head12:30:201
2770299118,2cyclictest19449-21grep09:05:211
2770299118,2cyclictest19289-21cron08:00:001
2770299118,2cyclictest17067-21grep09:00:201
2770299118,2cyclictest13288-21taskset12:14:581
2770299118,2cyclictest11145-21ntpq07:40:211
2770299118,2cyclictest10884-21sh08:49:591
2770299117,3cyclictest10637-21tr09:55:151
2770299117,3cyclictest10637-21tr09:55:141
2770299110,9cyclictest0-21swapper/109:30:111
2769799119,1cyclictest5315-21grep09:45:150
2769799119,1cyclictest3650-21head07:25:220
2769799118,2cyclictest9381-21grep12:05:170
2769799118,2cyclictest9004-21ls10:55:220
2769799118,2cyclictest3799-21cat11:55:000
2769799118,2cyclictest32410-21taskset08:25:190
2769799118,2cyclictest30166-21taskset09:29:480
2769799118,2cyclictest29438-21expr08:20:160
2769799118,2cyclictest27687-21head11:35:160
2769799118,2cyclictest22398-21grep11:25:150
2769799118,2cyclictest21019-21grep11:20:220
2769799118,2cyclictest20467-21munin-run09:09:590
2769799118,2cyclictest16206-21grep09:00:150
2769799118,2cyclictest14778-21wc11:10:000
2769799118,2cyclictest10551-21taskset08:45:490
2769799118,2cyclictest10301-21tr07:40:150
2769799117,3cyclictest27314-21grep09:20:290
2770299109,1cyclictest30889-21taskset08:21:001
2770299108,1cyclictest7766-21chrt10:55:011
2770299108,1cyclictest6617-21grep08:40:151
2770299108,1cyclictest3114-21expr07:25:181
2770299108,1cyclictest24750-21expr08:10:151
2770299108,1cyclictest23088-21bc09:15:111
2770299108,1cyclictest21026-21cat10:15:151
2770299108,1cyclictest19747-21taskset11:20:081
2770299108,1cyclictest14825-21grep12:15:231
2770299108,1cyclictest14805-21cat07:50:111
2770299108,1cyclictest13006-21taskset08:51:071
2770299108,1cyclictest12742-21cat07:45:131
2770299107,2cyclictest32203-21head08:25:181
2770299107,2cyclictest30695-21grep10:35:141
2770299107,2cyclictest29649-21expr11:40:161
2770299107,2cyclictest26199-21df09:20:171
2770299107,2cyclictest21587-21cat09:10:181
2770299107,2cyclictest21347-21chrt08:00:431
2770299107,2cyclictest17647-21taskset10:09:241
2770299106,3cyclictest21922-21taskset08:05:091
2770299106,3cyclictest21922-21taskset08:05:081
2770299105,2cyclictest24290-21ntp_kernel_err10:20:221
2770299101,6cyclictest7273-21cron09:50:011
2769799108,1cyclictest32576-21chrt07:19:270
2769799108,1cyclictest3110-21cat09:40:140
2769799108,1cyclictest23373-21grep10:20:170
2769799108,1cyclictest21082-21cat09:10:150
2769799108,1cyclictest18420-21expr10:10:140
2769799108,1cyclictest17364-21acpi11:15:130
2769799108,1cyclictest15427-21taskset08:56:460
2769799108,1cyclictest13590-21ntpq07:45:190
2769799107,2cyclictest6658-21grep12:00:140
2769799107,2cyclictest30910-21expr09:30:140
2769799107,2cyclictest25881-21expr10:25:150
2769799107,2cyclictest20830-21tr10:15:140
2769799107,2cyclictest14781-21chrt07:50:080
2769799107,2cyclictest12231-21seq09:56:310
2769799107,2cyclictest12231-21seq09:56:300
2769799107,2cyclictest11623-21df12:10:150
2769799100,9cyclictest28605-21expr10:30:150
277029997,1cyclictest18356-21grep12:25:131
276979997,1cyclictest8043-21expr09:50:140
276979997,1cyclictest7623-21apt07:35:130
276979997,1cyclictest4399-21expr08:35:150
276979997,1cyclictest23463-21expr09:15:150
276979997,1cyclictest21130-21expr12:30:140
276979996,2cyclictest15085-21grep11:10:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional