You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 00:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Sun May 04, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1003301211659,53sleep30-21swapper/307:05:103
100330028754,9sleep20-21swapper/207:05:102
100496328654,9sleep10-21swapper/107:09:291
100473928458,20sleep00-21swapper/007:06:360
1005132993126,3cyclictest1208992-21sendmail12:00:002
1005140993026,1cyclictest0-21swapper/312:20:003
1005132993026,2cyclictest1187235-21perf11:40:012
1005132993025,3cyclictest1054372-21munin-run09:09:592
1005132992925,3cyclictest1005127-21cyclictest07:55:012
1005140992824,2cyclictest0-21swapper/312:05:153
100514099272,18cyclictest1110443-21chrt10:15:123
1005132992721,3cyclictest0-21swapper/212:20:012
1005132992719,5cyclictest1028820-21uname08:05:172
1005140992521,2cyclictest0-21swapper/311:55:243
1005140992520,2cyclictest0-21swapper/311:09:053
100513299258,14cyclictest1-21systemd11:15:002
1005132992521,2cyclictest1132012-21strings10:40:182
1005132992521,2cyclictest1009506-21sh07:20:002
1005132992521,2cyclictest0-21swapper/210:40:012
1005132992521,2cyclictest0-21swapper/209:10:212
1005132992520,3cyclictest716-21gdbus10:50:012
1005132992520,3cyclictest1247792-21cpuspeed_turbos12:35:132
1005132992520,3cyclictest1115646-21timerandwakeup10:20:252
1005132992520,3cyclictest1115646-21timerandwakeup10:20:252
1005132992516,6cyclictest1210299-21ntpq12:00:222
11722152240,12sleep01172264-21ntpq11:25:190
1005132992420,2cyclictest1220738-21fschecks_count12:10:152
1005132992420,2cyclictest1220738-21fschecks_count12:10:152
1005132992420,2cyclictest1059580-21pmu-power09:15:212
1005132992420,2cyclictest1046872-21cpuspeed_turbos08:50:142
1005132992420,2cyclictest0-21swapper/209:40:002
1005132992420,2cyclictest0-21swapper/208:55:122
1005132992412,9cyclictest1164607-21sshd11:18:342
100514099239,2cyclictest0-21swapper/309:27:293
1005140992319,2cyclictest0-21swapper/311:35:163
100514099231,16cyclictest1139779-21chrt10:50:123
100513299237,2cyclictest1054368-21diskmemload11:02:212
1005132992320,2cyclictest1005127-21cyclictest10:50:262
1005132992319,2cyclictest1231511-21cpuspeed_turbos12:20:152
1005132992319,2cyclictest0-21swapper/207:30:222
100513299231,8cyclictest1241408-21taskset12:29:282
1005132992318,3cyclictest1038673-21cpuspeed_turbos08:30:142
100513299231,7cyclictest1156163-21sh11:08:482
1005132992312,9cyclictest0-21swapper/211:47:272
1005132992312,7cyclictest1172801-21sed11:25:242
100513299231,19cyclictest1198634-21apt-get11:50:122
100513299231,18cyclictest1170814-21ssh11:24:452
100513299231,18cyclictest1170814-21ssh11:24:452
1005140992218,2cyclictest0-21swapper/309:20:163
1005140992217,3cyclictest0-21swapper/312:05:003
1005140992217,3cyclictest0-21swapper/311:30:133
1005140992217,2cyclictest0-21swapper/309:19:473
1005140992211,7cyclictest1054366-21turbostat09:09:593
100514099221,16cyclictest1147451-21chrt10:58:473
1005132992219,2cyclictest1005127-21cyclictest09:30:262
1005132992218,2cyclictest1181799-21cut11:35:012
1005132992218,2cyclictest0-21swapper/211:40:292
100513299221,7cyclictest1094528-21ldconfig09:55:232
1005132992217,3cyclictest1070427-21kworker/u16:1-rpciod09:50:172
1005132992217,3cyclictest1036089-21cut08:25:012
1005132992217,2cyclictest0-21swapper/210:05:002
100513299221,6cyclictest1119577-21diskmemload10:25:182
100513299221,6cyclictest1088071-21runrttasks09:48:222
100513299221,18cyclictest1103770-21sh10:05:362
100513299221,17cyclictest1108819-21sh10:12:082
1005132992211,7cyclictest1041963-21sshd08:36:342
1005132992210,8cyclictest1242800-21cat12:30:172
1005140992115,4cyclictest0-21swapper/307:45:153
100513299219,8cyclictest1044778-21apt-get08:45:122
100513299217,11cyclictest1145157-21tr10:55:222
1005132992117,2cyclictest700-21dbus-daemon07:39:352
1005132992117,2cyclictest1215340-21fschecks_count12:05:152
1005132992117,2cyclictest1030229-21cpuspeed_turbos08:10:142
1005132992117,2cyclictest0-21swapper/208:40:182
1005132992111,7cyclictest1027903-21cron08:05:012
100513299211,16cyclictest1011755-21taskset07:25:132
100513299210,18cyclictest1071664-21munin-run09:30:002
100514099201,7cyclictest1118836-21taskset10:25:113
1005140992017,1cyclictest0-21swapper/308:25:133
1005140992016,1cyclictest0-21swapper/307:25:193
1005140992015,2cyclictest0-21swapper/311:44:293
100513299209,7cyclictest1065477-21sh09:22:172
100513299208,8cyclictest1123634-21sh10:30:152
1005132992016,3cyclictest1025940-21/usr/sbin/munin08:00:002
1005132992015,3cyclictest1021876-21latency_hist07:50:002
1005132992010,7cyclictest944-21snmpd09:41:292
1005132992010,7cyclictest1110325-21ssh10:15:032
10123922191,5sleep01012449-21ntpq07:25:200
100514099192,12cyclictest1139423-21sleep10:50:013
1005140991916,1cyclictest0-21swapper/309:40:203
1005140991915,2cyclictest0-21swapper/312:10:263
1005140991915,2cyclictest0-21swapper/312:10:263
1005140991915,2cyclictest0-21swapper/310:20:313
1005140991915,2cyclictest0-21swapper/310:20:303
1005140991915,2cyclictest0-21swapper/309:55:013
1005140991915,2cyclictest0-21swapper/308:05:013
100514099191,15cyclictest82250irq/126-enp1s0-TxRx-309:55:323
100514099191,14cyclictest82250irq/126-enp1s0-TxRx-310:38:493
100514099191,14cyclictest0-21swapper/310:05:173
100514099191,14cyclictest0-21swapper/308:40:123
100513299199,6cyclictest264-21systemd-journal09:02:292
100513299199,6cyclictest1006816-21sshd07:12:352
1005132991915,2cyclictest1038169-21apt-key08:30:012
11713962181,12sleep11171431-21ssh11:25:121
11567702181,3sleep00-21swapper/011:09:540
100514099182,12cyclictest1249464-21taskset12:35:533
1005140991815,1cyclictest0-21swapper/308:45:193
1005140991815,1cyclictest0-21swapper/307:35:003
1005140991814,2cyclictest0-21swapper/312:26:283
1005140991814,2cyclictest0-21swapper/308:18:383
1005140991814,2cyclictest0-21swapper/307:58:093
1005140991814,2cyclictest0-21swapper/307:52:023
1005140991813,2cyclictest0-21swapper/309:30:063
1005140991812,3cyclictest0-21swapper/311:45:183
1005140991812,3cyclictest0-21swapper/311:19:243
1005140991811,4cyclictest0-21swapper/311:10:113
100514099181,14cyclictest0-21swapper/309:38:503
100514099181,13cyclictest0-21swapper/311:25:303
100514099181,13cyclictest0-21swapper/310:40:553
100514099181,13cyclictest0-21swapper/309:45:283
100514099180,12cyclictest0-21swapper/310:30:473
100513299180,16cyclictest1019676-21taskset07:44:072
100513299180,15cyclictest1010553-21ldconfig07:20:212
12080872171,12sleep01208097-21ssh11:58:500
11166812171,4sleep01054368-21diskmemload10:21:450
11166812171,4sleep01054368-21diskmemload10:21:440
11037912175,2sleep1191rcu_preempt10:05:381
10887902170,3sleep00-21swapper/009:49:540
10136992170,12sleep10-21swapper/107:30:141
100514099177,7cyclictest1168231-21taskset11:21:203
100514099177,7cyclictest1168231-21taskset11:21:193
1005140991714,1cyclictest0-21swapper/310:13:203
1005140991714,1cyclictest0-21swapper/309:04:443
1005140991714,1cyclictest0-21swapper/308:58:343
1005140991714,1cyclictest0-21swapper/308:09:253
1005140991714,1cyclictest0-21swapper/307:30:223
1005140991714,1cyclictest0-21swapper/307:24:233
1005140991714,1cyclictest0-21swapper/307:14:093
1005140991713,2cyclictest0-21swapper/308:30:153
1005140991712,3cyclictest82250irq/126-enp1s0-TxRx-312:21:103
100514099171,13cyclictest0-21swapper/312:30:193
100514099171,13cyclictest0-21swapper/311:00:173
100514099171,12cyclictest1101835-21chrt10:04:553
100514099170,14cyclictest0-21swapper/309:13:203
100514099170,13cyclictest0-21swapper/311:50:043
100513299171,12cyclictest1032395-21seq08:15:162
12388392161,11sleep01238852-21ssh12:26:080
11966462160,3sleep00-21swapper/011:47:560
11891442161,5sleep11189147-21ssh11:40:311
11218932160,4sleep01054368-21diskmemload10:28:170
10746072160,2sleep00-21swapper/009:32:570
100514099168,5cyclictest1031843-21turbostat08:15:003
1005140991614,1cyclictest0-21swapper/308:35:013
1005140991612,2cyclictest0-21swapper/308:50:173
1005140991610,3cyclictest0-21swapper/307:44:073
100514099160,13cyclictest0-21swapper/308:20:193
12133372150,1sleep10-21swapper/112:03:361
11655462151,9sleep01165561-21ssh11:19:490
100514099151,2cyclictest0-21swapper/307:16:173
12063252141,7sleep11054368-21diskmemload11:56:311
11895342140,3sleep00-21swapper/011:41:010
11595662140,9sleep01159579-21ssh11:12:000
11449692141,9sleep11145108-21ssh10:55:201
11440382140,2sleep00-21swapper/010:55:070
11304622140,2sleep00-21swapper/010:38:030
12426922131,8sleep01242824-21/usr/sbin/munin12:30:180
10835732130,2sleep00-21swapper/009:42:560
10432062131,7sleep01043252-21ntpq08:40:180
10360612131,8sleep11036280-21latency_hist08:25:011
10272192131,8sleep01027442-21grep08:00:260
10114712131,8sleep11011550-21sed07:25:001
12526152121,5sleep00-21swapper/012:40:000
12232672120,1sleep00-21swapper/012:12:060
12232672120,1sleep00-21swapper/012:12:050
12170982120,8sleep11054368-21diskmemload12:06:051
12153502121,7sleep01215554-21ls12:05:160
12017732120,2sleep01054368-21diskmemload11:52:360
11707462121,3sleep10-21swapper/111:24:401
11707462121,3sleep10-21swapper/111:24:391
10927722121,7sleep11092781-21sh09:54:041
10921072120,2sleep00-21swapper/009:52:530
10567732121,7sleep11056784-21ssh09:11:381
12336742111,4sleep11054368-21diskmemload12:21:291
12305552114,4sleep01005127-21cyclictest12:19:300
11111982110,1sleep10-21swapper/110:15:201
11099942111,2sleep01110001-21ssh10:14:540
10883822111,4sleep11054368-21diskmemload09:48:541
10757382110,1sleep10-21swapper/109:34:591
10659692111,4sleep11054368-21diskmemload09:23:091
10605062111,4sleep11054368-21diskmemload09:16:151
10177202111,6sleep1214-21jbd2/sda2-807:39:401
12471002101,3sleep11054368-21diskmemload12:34:521
12389472101,3sleep11054368-21diskmemload12:26:171
12289922101,2sleep10-21swapper/112:17:271
12113562100,1sleep00-21swapper/012:01:020
12024042101,3sleep11054368-21diskmemload11:53:251
11948072101,3sleep11054368-21diskmemload11:45:331
11867422101,4sleep11054368-21diskmemload11:39:271
11837032101,2sleep00-21swapper/011:35:290
11799982101,3sleep11054368-21diskmemload11:32:421
11677422101,2sleep00-21swapper/011:20:380
11677422101,2sleep00-21swapper/011:20:370
11634702101,3sleep11054368-21diskmemload11:16:251
11608002100,4sleep11054368-21diskmemload11:14:421
11512342101,3sleep00-21swapper/011:02:420
11388552101,3sleep11054368-21diskmemload10:48:591
11289882101,4sleep11054368-21diskmemload10:35:281
11187892101,3sleep11054368-21diskmemload10:25:061
11140032100,3sleep00-21swapper/010:19:520
11096922101,4sleep11054368-21diskmemload10:14:241
11050352101,5sleep01105046-21ssh10:08:220
10999922101,3sleep11054368-21diskmemload10:01:331
10988362100,3sleep00-21swapper/010:00:230
10944802101,4sleep11094536-21ntpq09:55:241
10827852100,1sleep10-21swapper/109:41:251
10699492101,4sleep11054368-21diskmemload09:27:061
10564902100,1sleep00-21swapper/009:11:090
10513752101,4sleep11051424-21ntpq09:00:201
10338792101,3sleep00-21swapper/008:17:340
10308862101,4sleep01030943-21ntp_states08:10:200
10248932101,5sleep11024934-21ntpq07:55:191
10162962101,4sleep01016400-21irqstats07:35:180
1250217290,3sleep11054368-21diskmemload12:36:511
1222514291,2sleep10-21swapper/112:11:061
1222514291,2sleep10-21swapper/112:11:051
1179325291,2sleep00-21swapper/011:31:470
1152826291,3sleep11054368-21diskmemload11:05:061
1149035290,4sleep11149164-21memory11:00:181
1139941291,4sleep01140048-21grep10:50:150
1137494291,2sleep00-21swapper/010:46:290
1133790291,3sleep11054368-21diskmemload10:42:381
1133593291,2sleep00-21swapper/010:42:170
1127089291,3sleep01054368-21diskmemload10:34:520
1123629291,4sleep11123709-21irqstats10:30:161
1114478291,4sleep11114629-21cut10:20:131
1114478291,4sleep11114629-21cut10:20:131
1096483290,4sleep01054368-21diskmemload09:58:080
1079061290,2sleep00-21swapper/009:38:030
1078047291,4sleep11078056-21ssh09:36:201
1066347291,3sleep01054368-21diskmemload09:23:470
1061106291,2sleep00-21swapper/009:17:180
1052260290,1sleep00-21swapper/009:03:360
1046743291,4sleep11046886-21df08:50:141
1039398291,4sleep11039542-21mailstats08:30:221
1033076291,3sleep11033125-21ntpq08:15:231
1028893291,4sleep01028942-21ntpq08:05:200
1026762291,5sleep11026892-21ld08:00:201
1012451291,4sleep11012497-21ntpq07:25:211
1010233291,4sleep01010412-21grep07:20:180
1233092280,2sleep00-21swapper/012:20:430
1141846281,3sleep11054368-21diskmemload10:51:401
1067447281,2sleep00-21swapper/009:25:080
1046721280,3sleep01046740-21apt-get08:50:120
1041227280,3sleep01041274-21ntpq08:35:180
1025792280,1sleep00-21swapper/007:58:300
1020012280,3sleep01020031-21apt-get07:45:160
1009289280,1sleep10-21swapper/107:17:351
1054062270,1sleep00-21swapper/009:06:320
1052635270,3sleep11052654-21apt-get09:05:131
1048466270,4sleep00-21swapper/008:55:020
1040092270,1sleep00-21swapper/008:32:440
1037901270,1sleep00-21swapper/008:28:280
1021542270,1sleep10-21swapper/107:47:531
1009220270,1sleep00-21swapper/007:16:420
1006735270,1sleep00-21swapper/007:11:320
1049397260,1sleep10-21swapper/108:55:191
1046396260,4sleep00-21swapper/008:49:130
1040872261,2sleep11040988-21irqstats08:35:151
1035970260,1sleep00-21swapper/008:23:500
1022248260,1sleep00-21swapper/007:50:120
1019718260,1sleep00-21swapper/007:44:390
1015056260,1sleep00-21swapper/007:30:360
1005706261,2sleep10-21swapper/107:10:171
1044220250,2sleep10-21swapper/108:45:011
1037335250,1sleep10-21swapper/108:25:251
1030588250,2sleep11030777-21strings08:10:181
1028161250,1sleep10-21swapper/108:05:131
1018213250,1sleep10-21swapper/107:40:161
1045101240,1sleep10-21swapper/108:45:171
1021749240,1sleep10-21swapper/107:50:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional