You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 01:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Nov 18, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
220672120109,6sleep00-21swapper/007:05:240
22476210490,10sleep10-21swapper/107:06:101
2252529987,8sleep30-21swapper/307:06:493
2244228465,15sleep20-21swapper/207:05:432
97752600,0sleep10-21swapper/109:31:311
2285999230,1cyclictest0-21swapper/212:19:252
22850992321,1cyclictest0-21swapper/010:30:150
2285999220,22cyclictest0-21swapper/210:16:132
2285999220,22cyclictest0-21swapper/209:42:592
2285999200,19cyclictest0-21swapper/212:34:472
2285099206,7cyclictest3532-21munin-run08:45:000
22859991817,1cyclictest0-21swapper/209:56:152
2285999180,18cyclictest0-21swapper/209:29:312
2285999180,16cyclictest0-21swapper/212:03:072
22855991817,1cyclictest0-21swapper/112:25:221
22850991817,1cyclictest0-21swapper/011:37:070
22850991815,2cyclictest0-21swapper/010:16:130
22850991815,1cyclictest5291-21smartctl12:30:220
309732170,0sleep30-21swapper/310:43:183
2285999175,4cyclictest26800-21cut08:25:172
2285999170,17cyclictest0-21swapper/209:32:312
2286499167,4cyclictest0-21swapper/309:30:163
22864991615,1cyclictest0-21swapper/311:58:563
22859991616,0cyclictest0-21swapper/210:24:512
2286499150,2cyclictest0-21swapper/308:10:163
2285999152,11cyclictest7247-21ps10:50:252
2285999151,1cyclictest15495-21diskmemload12:39:182
22859991510,4cyclictest6032-21cat07:40:192
22859991510,4cyclictest6032-21cat07:40:192
2285999150,1cyclictest0-21swapper/210:06:052
2285999150,15cyclictest0-21swapper/211:54:342
2285999150,14cyclictest0-21swapper/210:13:402
2285599153,7cyclictest1141-21systemctl08:39:591
22855991510,5cyclictest25-21ksoftirqd/111:42:551
2285099159,3cyclictest29574-21ssh12:25:180
2285099158,2cyclictest17462-21grep10:05:170
2285099151,8cyclictest28390-21munin-run12:24:590
2286499147,6cyclictest10632-21ssh11:46:203
22864991411,2cyclictest5807-21tr12:30:263
22864991411,2cyclictest4751-21wc08:45:203
22864991411,2cyclictest10965-21/usr/sbin/munin09:00:213
22859991410,3cyclictest9633-21wc08:55:232
2285999140,13cyclictest0-21swapper/211:07:002
2285599144,9cyclictest24386-21munin-node09:43:491
22855991411,2cyclictest29999-21sleep07:25:001
2285099143,9cyclictest22693-21df_inode09:15:170
2285099140,3cyclictest0-21swapper/010:02:330
2286499139,3cyclictest31692-21cut09:50:313
2286499139,3cyclictest26275-21cron10:40:003
2286499139,2cyclictest5927-21tr12:10:173
22864991312,1cyclictest22845-21cyclictest10:45:583
22864991310,2cyclictest30738-21sed12:25:273
22864991310,2cyclictest27927-21grep12:00:283
22864991310,2cyclictest16518-21cat10:30:273
22864991310,2cyclictest10319-21wc10:25:243
2286499130,1cyclictest141rcu_preempt09:45:103
2286499130,13cyclictest0-21swapper/309:39:253
2286499130,13cyclictest0-21swapper/309:39:243
2286499130,12cyclictest8326-21cron08:55:003
2286499130,12cyclictest0-21swapper/310:52:363
22859991310,2cyclictest21186-21grep08:10:282
22859991310,2cyclictest10832-21grep11:20:282
2285999130,13cyclictest0-21swapper/211:43:442
2285999130,13cyclictest0-21swapper/210:36:082
2285599139,2cyclictest30291-21head08:30:221
22855991310,2cyclictest9350-21grep08:55:201
22855991310,2cyclictest8846-21ls07:45:231
22855991310,2cyclictest29665-21grep11:35:261
22855991310,2cyclictest27462-21wc11:35:011
22855991310,2cyclictest24803-21df08:20:141
22855991310,2cyclictest24581-21grep11:05:211
22855991310,2cyclictest16193-21cat08:00:271
22855991310,2cyclictest14885-21cat09:05:281
22855991310,2cyclictest13159-21cut12:15:171
2285599130,12cyclictest4015-21grep11:15:191
2285599130,12cyclictest4015-21grep11:15:181
2285599130,12cyclictest32378-21awk07:30:001
2285099139,3cyclictest32295-21cut12:05:200
2285099139,3cyclictest31821-21cat07:25:270
2285099139,3cyclictest26084-21grep07:15:210
2285099137,5cyclictest7545-21apache212:37:050
22850991310,2cyclictest7329-21cat07:45:010
22850991310,2cyclictest7329-21cat07:45:000
22850991310,2cyclictest6695-21wc09:30:010
22850991310,2cyclictest32180-21grep08:35:190
22850991310,2cyclictest31655-21grep09:50:320
22850991310,2cyclictest29480-21fgrep09:20:250
22850991310,2cyclictest27466-21sort10:40:210
22850991310,2cyclictest26343-21chrt12:00:170
22850991310,2cyclictest24058-21tr08:20:010
22850991310,2cyclictest13343-21tr10:55:210
22850991310,2cyclictest12934-21tr09:35:200
22850991310,2cyclictest12934-21tr09:35:200
22850991310,2cyclictest1158-21sed10:45:200
2286499129,3cyclictest39-21ksoftirqd/311:24:203
2286499129,2cyclictest4153-21wc07:35:273
2286499129,2cyclictest3834-21cut11:15:183
2286499129,2cyclictest3834-21cut11:15:183
2286499129,2cyclictest31179-21dump-pmu-power08:34:593
2286499129,2cyclictest28944-21cut11:35:203
2286499129,2cyclictest27371-21cut08:25:203
2286499129,2cyclictest24773-21grep07:10:333
2286499129,2cyclictest22399-21grep08:15:183
2286499129,2cyclictest2177-21sort11:40:193
2286499129,2cyclictest18992-21grep11:00:183
2286499129,2cyclictest18810-21grep08:05:283
2286499129,2cyclictest17441-21sh11:25:303
2286499129,2cyclictest13869-21grep09:05:183
2286499129,2cyclictest10256-21tr07:50:133
2286499128,3cyclictest21784-21tail12:20:193
2286499126,2cyclictest8474-21latency_hist10:25:003
2286499123,8cyclictest24386-21munin-node12:38:433
2285999129,2cyclictest7201-21wc12:10:282
2285999129,2cyclictest31343-21cut07:25:212
2285999129,2cyclictest29843-21tail09:20:292
2285999129,2cyclictest28584-21grep07:20:172
2285999129,2cyclictest25743-21tr07:15:182
2285999129,2cyclictest20964-21cron11:30:012
2285999129,2cyclictest20290-21tr11:55:162
2285999129,2cyclictest15514-21cut10:30:182
2285999129,2cyclictest15510-21grep08:00:192
2285999129,2cyclictest12391-21tr10:55:162
2285999128,3cyclictest6954-21cut08:50:272
22859991211,1cyclictest0-21swapper/207:30:262
2285999120,4cyclictest0-21swapper/212:22:482
2285999120,12cyclictest0-21swapper/211:36:262
2285999120,12cyclictest0-21swapper/210:28:262
2285599129,2cyclictest6690-21sed10:50:201
2285599129,2cyclictest31551-21grep09:50:311
2285599129,2cyclictest29707-21wc09:20:281
2285599129,2cyclictest27399-21tr08:25:201
2285599129,2cyclictest27290-21tr12:00:231
2285599129,2cyclictest18475-21grep10:05:241
2285599129,2cyclictest14109-21ssh12:35:281
2285599129,2cyclictest10672-21grep10:25:271
2285599123,6cyclictest25201-21cron07:15:001
22855991210,1cyclictest20836-21basename10:35:151
2285099129,2cyclictest7601-21ssh09:30:170
2285099129,2cyclictest4279-21grep07:35:280
2285099129,2cyclictest30607-21wc08:30:260
2285099129,2cyclictest25683-21tail11:05:300
2285099129,2cyclictest20003-21grep11:00:280
2285099129,2cyclictest1915-21grep07:30:250
2285099129,2cyclictest17881-21grep08:05:200
2285099129,2cyclictest16190-21tr09:10:170
2285099129,2cyclictest15286-21ntpq11:50:250
2285099128,3cyclictest17231-21wc11:25:280
2285099128,3cyclictest15034-21sed09:05:280
2285099128,2cyclictest28754-21ls10:15:000
2285099125,1cyclictest0-21swapper/011:17:420
2285099125,1cyclictest0-21swapper/011:17:410
2285099123,3cyclictest224-21systemd-journal09:59:290
2285099122,8cyclictest27875-21apache_processe07:20:120
22850991210,1cyclictest10248-21grep07:50:140
2285099120,6cyclictest2779-21perf11:15:000
2286499119,2cyclictest15745-21sed08:00:213
2286499118,2cyclictest8823-21tr08:55:183
2286499118,2cyclictest4157-21ssh12:09:013
2286499118,2cyclictest3336-21ssh09:25:573
2286499118,2cyclictest31719-21ssh11:10:483
2286499118,2cyclictest28769-21cut09:20:193
2286499118,2cyclictest24931-21sort11:05:233
2286499118,2cyclictest23828-21wc10:10:163
2286499118,2cyclictest23794-21ssh11:31:003
2286499118,2cyclictest21817-21ssh09:41:493
2286499118,2cyclictest17775-21ssh10:05:183
2286499118,2cyclictest17102-21ssh12:17:103
2286499118,2cyclictest15875-21ssh09:10:123
2286499118,2cyclictest14978-21mii-tool11:50:233
2286499118,2cyclictest12361-21ssh10:55:163
2286499118,2cyclictest1109-21cron08:40:003
2286499117,3cyclictest5626-21wc09:55:213
2286499111,0cyclictest141rcu_preempt09:20:013
22864991110,1cyclictest61750irq/125-enp2s0-10:02:003
2285999118,2cyclictest9925-21wc07:50:012
2285999118,2cyclictest6020-21ssh11:16:402
2285999118,2cyclictest6020-21ssh11:16:402
2285999118,2cyclictest32239-21grep12:05:212
2285999118,2cyclictest30334-21grep11:10:172
2285999118,2cyclictest28609-21ssh12:25:012
2285999118,2cyclictest27607-21tail10:40:222
2285999118,2cyclictest24738-21grep07:10:322
2285999118,2cyclictest22355-21tr09:15:162
2285999118,2cyclictest18450-21tr11:00:162
2285999118,2cyclictest16771-21cut09:10:192
2285999118,2cyclictest1484-21ssh09:52:002
2285999118,2cyclictest14141-21sed09:05:192
2285999118,2cyclictest1289-21wc08:40:002
2285999118,2cyclictest12237-21grep10:00:252
2285999110,10cyclictest0-21swapper/211:34:472
2285999110,10cyclictest0-21swapper/207:50:122
2285599119,1cyclictest27653-21mktemp09:20:001
2285599119,1cyclictest25852-21cat09:45:221
2285599118,2cyclictest8153-21grep11:45:181
2285599118,2cyclictest6970-21head12:10:241
2285599118,2cyclictest4426-21cat09:55:121
2285599118,2cyclictest3108-21ssh09:25:401
2285599118,2cyclictest29469-21ssh11:10:101
2285599118,2cyclictest2888-21cut10:20:141
2285599118,2cyclictest22963-21sed12:20:291
2285599118,2cyclictest18840-21sed08:05:281
2285599118,2cyclictest16426-21ssh09:10:171
2285599118,2cyclictest15771-21aten2_r7power_v11:25:181
2285599118,2cyclictest14832-21expr11:50:211
2285599118,2cyclictest13798-21cat07:55:271
2285599118,2cyclictest12494-21ssh09:35:041
2285599118,2cyclictest12494-21ssh09:35:041
2285599116,4cyclictest2587-21apt07:35:141
2285599110,3cyclictest224-21systemd-journal10:19:371
2285599110,10cyclictest10254-21grep07:50:141
2285099118,2cyclictest8942-21sh10:51:460
2285099118,2cyclictest5477-21ssh12:10:050
2285099118,2cyclictest4537-21grep08:45:190
2285099118,2cyclictest2889-21tr10:20:140
2285099118,2cyclictest25633-21sed09:45:200
2285099118,2cyclictest25201-21ssh11:32:370
2285099118,2cyclictest24239-21ntpq07:10:270
2285099118,2cyclictest20480-21ssh11:55:160
2285099118,2cyclictest15955-21ssh12:16:140
2285099118,2cyclictest11259-21ssh10:26:040
2285099118,2cyclictest10994-21ssh11:46:440
2285099117,3cyclictest6959-21grep08:50:260
2285099117,3cyclictest21109-21tr10:35:160
2285099117,2cyclictest19800-21grep08:10:170
2286499108,1cyclictest30801-21ssh10:15:283
2286499108,1cyclictest296862sleep307:21:193
2286499108,1cyclictest14305-21chrt07:57:503
2286499107,2cyclictest7678-21cut07:45:153
2286499107,2cyclictest32073-21chrt07:26:283
2286499107,2cyclictest25570-21wc08:20:233
2286499107,2cyclictest25552-21cut07:15:183
2286499107,2cyclictest1674-21tr08:40:243
2285999108,1cyclictest24281-21taskset08:20:102
2285999108,1cyclictest22778-21cat08:15:212
2285999108,1cyclictest14293-21taskset07:57:392
2285999108,1cyclictest11706-21head09:00:182
2285999107,3cyclictest3535-21grep07:35:202
2285999107,2cyclictest9348-21cat11:45:262
2285999107,2cyclictest31282-21cat08:35:002
2285999107,2cyclictest1148-21grep10:45:192
2285999104,5cyclictest13947-21head09:35:272
2285999104,5cyclictest13947-21head09:35:262
2285999101,3cyclictest0-21swapper/209:45:152
2285999100,9cyclictest1780-21expr08:40:242
2285599108,2cyclictest15802-21sh10:57:371
2285599108,2cyclictest1457-21taskset12:06:111
2285599108,2cyclictest13191-21date09:04:591
2285599108,1cyclictest6458-21rm12:30:331
2285599108,1cyclictest1938-21cat10:45:271
2285599108,1cyclictest10880-21grep11:20:291
2285599107,2cyclictest945-21grep07:30:161
2285599107,2cyclictest8334-21turbostat.cron08:55:001
2285599107,2cyclictest29566-21ssh10:41:301
2285599107,2cyclictest28579-21ssh10:14:481
2285599107,2cyclictest27428-21chrt07:18:041
2285599107,2cyclictest1397-21basename08:40:141
2285599107,2cyclictest10663-21ssh10:00:011
2285599106,3cyclictest19851-21ssh11:55:061
2285599100,9cyclictest5580-21taskset08:46:051
2285599100,9cyclictest18107-21ssh11:00:121
2285099108,1cyclictest8921-21ssh11:20:100
2285099108,1cyclictest8589-21cat08:55:150
2285099108,1cyclictest7630-21apache_volume07:45:130
2285099108,1cyclictest28660-21chrt08:28:240
2285099108,1cyclictest2642-21sed11:40:220
2285099108,1cyclictest15224-21expr08:00:170
2285099107,2cyclictest20736-21ssh09:40:280
2285099107,2cyclictest11942-21cut09:00:210
228649994,2cyclictest4981-21apt-get07:40:123
228649994,2cyclictest4981-21apt-get07:40:113
228649991,4cyclictest32634-21apt-get07:30:133
228599997,2cyclictest18593-21sed08:05:272
228599996,2cyclictest4536-21tr08:45:182
228559998,1cyclictest6796-21taskset07:41:261
228559998,1cyclictest6796-21taskset07:41:251
228559998,1cyclictest19883-21cat08:10:171
228559997,1cyclictest22256-21expr08:15:171
228559997,1cyclictest14677-21ssh10:30:091
228509997,1cyclictest24317-21apt-get08:20:130
228509996,2cyclictest12769-21expr07:55:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional