You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 03:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Tue Dec 02, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1173556212652,69sleep30-21swapper/319:05:433
154902421180,1sleep0221rcuc/023:39:450
117375728453,9sleep20-21swapper/219:08:152
117382127762,9sleep10-21swapper/119:09:061
117356827751,20sleep00-21swapper/019:05:510
13523262300,1sleep21352333-21latency_hist21:40:002
1174021992825,1cyclictest0-21swapper/321:40:003
13936802261,8sleep01393942-21df22:05:130
1174021992623,1cyclictest0-21swapper/321:15:023
1174021992516,5cyclictest13731002sleep321:50:213
117402199242,14cyclictest1529743-21taskset23:25:323
117402199241,19cyclictest1451025-21taskset22:40:113
117402199241,18cyclictest1303190-21turbostat21:10:003
11782912231,9sleep11178348-21ntp_states19:10:211
117402199232,15cyclictest1487897-21taskset23:00:243
1174021992313,5cyclictest14329152sleep322:28:003
117402199231,16cyclictest1524445-21turbostat23:25:003
117402199229,8cyclictest1594543-21taskset00:05:253
1174021992219,1cyclictest0-21swapper/323:35:163
1174021992219,1cyclictest0-21swapper/300:25:233
117402199221,7cyclictest1605974-21turbostat00:15:003
117402199221,17cyclictest1399842-21taskset22:07:313
117402199221,16cyclictest1367526-21chrt21:48:143
1174021992210,11cyclictest0-21swapper/323:40:503
16435152211,7sleep11303165-21diskmemload00:38:411
117402199213,14cyclictest0-21swapper/323:05:233
1174021992117,2cyclictest0-21swapper/321:43:353
1174021992116,3cyclictest0-21swapper/322:20:223
117402199211,13cyclictest1416611-21taskset22:18:093
14965862201,2sleep00-21swapper/023:05:310
14309452201,13sleep21426422-21/usr/sbin/munin22:25:272
117402199202,13cyclictest1587059-21taskset00:00:453
117402199202,13cyclictest1326881-21taskset21:23:303
1174021992017,1cyclictest0-21swapper/322:50:273
1174021992017,1cyclictest0-21swapper/321:00:233
1174021992011,7cyclictest1613027-21taskset00:18:283
117402199201,15cyclictest1392845-21chrt22:04:193
117402199201,15cyclictest1179264-21sleep19:15:003
117402199201,14cyclictest1342532-21chrt21:32:323
1174021992010,7cyclictest1581400-21taskset23:59:443
14634942191,9sleep21463591-21ssh22:45:232
13905982191,2sleep10-21swapper/122:00:521
13598342191,12sleep11303165-21diskmemload21:43:561
117402199195,12cyclictest0-21swapper/323:45:133
117402199192,12cyclictest1539110-21taskset23:32:233
117402199192,12cyclictest1464813-21taskset22:46:353
117402199192,12cyclictest1318279-21taskset21:17:483
1174021991916,1cyclictest0-21swapper/322:10:233
1174021991916,1cyclictest0-21swapper/320:30:193
1174021991916,1cyclictest0-21swapper/320:30:013
1174021991915,2cyclictest0-21swapper/300:22:333
1174021991913,3cyclictest0-21swapper/300:35:563
117402199191,14cyclictest1333529-21chrt21:26:093
117402199191,14cyclictest0-21swapper/323:10:213
117402199191,14cyclictest0-21swapper/322:35:063
117402199191,14cyclictest0-21swapper/300:30:073
117402199191,13cyclictest1565911-21chrt23:50:073
117402199191,13cyclictest1271651-21chrt20:40:133
117402199191,13cyclictest1271651-21chrt20:40:123
117402199191,12cyclictest1377247-21taskset21:55:083
117402199190,12cyclictest1440066-21taskset22:31:213
14898372181,3sleep01303165-21diskmemload23:02:290
13931672181,2sleep20-21swapper/222:04:502
13409492181,5sleep21341197-21users21:30:272
13181472181,3sleep20-21swapper/221:17:362
13092732181,3sleep10-21swapper/121:11:501
117402199189,6cyclictest1186602-21turbostat19:20:003
117402199186,8cyclictest1249043-21chrt20:15:243
1174021991815,1cyclictest0-21swapper/319:40:183
117402199181,14cyclictest0-21swapper/323:17:153
117402199181,13cyclictest0-21swapper/320:00:143
117402199181,10cyclictest1202716-21taskset19:35:043
16135572171,3sleep20-21swapper/200:19:252
16066362171,4sleep11303165-21diskmemload00:15:151
15808702171,2sleep10-21swapper/123:58:521
15565312171,10sleep11303165-21diskmemload23:43:431
15383212171,2sleep10-21swapper/123:31:121
15006092171,6sleep11501781-21cat23:10:161
14810562171,3sleep10-21swapper/122:56:301
14574702171,4sleep01303165-21diskmemload22:42:530
14442892171,11sleep01445876-21grep22:35:200
14432782171,10sleep21443437-21interrupts22:35:172
14386382172,7sleep21438759-21ntpq22:30:212
14338272171,4sleep01303165-21diskmemload22:29:230
14071522171,4sleep11303165-21diskmemload22:11:151
13908052171,4sleep01303165-21diskmemload22:01:110
13587612171,11sleep21358771-21ssh21:42:182
13353412171,3sleep00-21swapper/021:29:010
13272972171,2sleep20-21swapper/221:24:102
13270772171,4sleep11303165-21diskmemload21:23:481
13091112171,3sleep20-21swapper/221:11:352
117402199172,11cyclictest1479815-21taskset22:55:213
1174021991713,2cyclictest0-21swapper/320:05:233
117402199171,13cyclictest0-21swapper/319:50:263
16210862163,2sleep2171rcu_preempt00:23:572
15962882161,4sleep01303165-21diskmemload00:07:490
15814132161,2sleep01303165-21diskmemload23:59:470
15649642161,2sleep20-21swapper/223:49:012
15547182161,2sleep20-21swapper/223:40:532
15335822161,10sleep21534304-21cat23:30:182
15320102161,10sleep01532020-21ssh23:29:080
15231692161,3sleep01303165-21diskmemload23:23:060
14715992161,10sleep21471651-21ntpq22:50:202
14243622161,3sleep01303165-21diskmemload22:22:280
14149322161,3sleep00-21swapper/022:15:360
13500252161,2sleep00-21swapper/021:36:350
13147102161,10sleep01315900-21memory21:15:180
1174021991613,1cyclictest0-21swapper/319:45:163
1174021991613,1cyclictest0-21swapper/319:30:183
1174021991612,2cyclictest0-21swapper/320:45:183
117402199160,13cyclictest0-21swapper/320:35:133
16187802151,9sleep11618797-21/usr/sbin/munin00:20:241
16055962151,9sleep11303165-21diskmemload00:14:261
15732052151,2sleep00-21swapper/023:54:210
15642002151,2sleep10-21swapper/123:47:531
15633532151,9sleep01563355-21ssh23:46:360
15319722151,2sleep10-21swapper/123:29:041
14669252157,3sleep11303165-21diskmemload22:49:481
14235532151,2sleep21303165-21diskmemload22:21:152
14168102151,3sleep11416822-21ssh22:18:291
13684482151,8sleep21303165-21diskmemload21:49:432
13676632150,9sleep01303165-21diskmemload21:48:290
13076732151,3sleep01303165-21diskmemload21:10:200
12341602151,10sleep11234162-21cron20:05:001
11972122158,3sleep01197444-21cat19:30:010
117402199157,4cyclictest1186889-21taskset19:20:123
117402199151,9cyclictest1244060-21taskset20:10:263
117402199151,2cyclictest0-21swapper/319:55:193
1174021991512,1cyclictest0-21swapper/320:55:173
1174021991512,1cyclictest0-21swapper/319:25:173
117402199151,10cyclictest1282506-21chrt20:50:153
117402199150,12cyclictest0-21swapper/320:20:133
16277872140,2sleep10-21swapper/100:27:381
15868682141,8sleep21586880-21ssh00:00:282
15725722141,6sleep11303165-21diskmemload23:53:171
15484692141,8sleep21548481-21ssh23:38:552
15056002141,2sleep00-21swapper/023:11:350
14738642141,7sleep01303165-21diskmemload22:52:550
14585152141,8sleep11458525-21ssh22:44:301
14531202141,9sleep21454873-21cut22:40:192
13773592141,9sleep11377295-21/usr/sbin/munin21:55:161
13677142141,8sleep11303165-21diskmemload21:48:341
13516702141,8sleep11303165-21diskmemload21:39:041
13278452141,1sleep00-21swapper/021:24:590
13159012141,4sleep11303165-21diskmemload21:15:181
12594872141,8sleep01255407-21/usr/sbin/munin20:25:230
12344292141,2sleep10-21swapper/120:05:171
16271302130,3sleep00-21swapper/000:26:290
15793332131,7sleep21579347-21ssh23:56:192
15473392131,8sleep11547347-21ssh23:37:151
15159112131,6sleep11303165-21diskmemload23:19:541
15135902130,2sleep20-21swapper/223:16:172
14661142131,8sleep01303165-21diskmemload22:48:350
14306142132,5sleep11430667-21ntpq22:25:231
14248162131,7sleep11303165-21diskmemload22:23:091
14018152131,2sleep20-21swapper/222:10:112
12719052131,8sleep01271935-21fschecks_count20:40:150
12719052131,8sleep01271935-21fschecks_count20:40:140
12542702131,8sleep21254387-21/usr/sbin/munin20:20:242
12084332131,4sleep21208584-21/usr/sbin/munin19:40:162
16271452121,5sleep20-21swapper/200:26:312
15822842121,4sleep01582504-21smartctl00:00:140
15359202120,6sleep00-21swapper/023:30:190
15165382121,7sleep11516548-21ssh23:20:021
14798772121,5sleep01480004-21systemctl22:55:240
14492452121,5sleep10-21swapper/122:37:531
14419432121,7sleep01303165-21diskmemload22:34:150
13939432120,7sleep11393648-21/usr/sbin/munin22:05:141
13829042121,7sleep21303165-21diskmemload21:56:362
13741612121,4sleep01303165-21diskmemload21:50:570
13366522121,7sleep11336716-21fschecks_count21:30:131
13348392120,1sleep10-21swapper/121:28:131
12818242121,4sleep00-21swapper/020:48:190
11874322121,7sleep21187470-21kthreadcore19:20:162
11747652121,7sleep01175974-21kthreadcore19:10:160
16414422111,5sleep01303165-21diskmemload00:35:220
16357602111,3sleep10-21swapper/100:32:581
16302372110,7sleep20-21swapper/200:30:152
16196072111,4sleep01303165-21diskmemload00:21:190
16127412110,5sleep01303165-21diskmemload00:17:590
16024182111,6sleep21602467-21ntpq00:10:222
15968682111,3sleep20-21swapper/200:08:512
15713882111,3sleep20-21swapper/223:51:232
15549812110,1sleep00-21swapper/023:41:180
15302042111,3sleep20-21swapper/223:26:162
15225072111,4sleep20-21swapper/223:22:042
15136962110,4sleep01303165-21diskmemload23:16:290
15052362111,3sleep20-21swapper/223:11:022
14995842111,3sleep10-21swapper/123:09:571
14896342110,5sleep11303165-21diskmemload23:02:101
14878372111,3sleep20-21swapper/223:00:232
14797562111,4sleep21303165-21diskmemload22:55:212
14751032111,4sleep11303165-21diskmemload22:54:521
14177932111,3sleep20-21swapper/222:19:582
14071842111,4sleep01303165-21diskmemload22:11:180
13987802111,3sleep20-21swapper/222:05:542
13772122111,4sleep01303165-21diskmemload21:55:070
13753902111,4sleep21303165-21diskmemload21:52:472
13568762111,4sleep01356926-21ntpq21:40:230
13357052111,2sleep21303165-21diskmemload21:29:342
12721652111,5sleep21272528-21cut20:40:172
12721652111,5sleep21272528-21cut20:40:162
12185402112,4sleep01218562-21apt-get19:50:120
12138652111,6sleep11215053-21kthreadcore19:45:171
16425052101,3sleep20-21swapper/200:36:212
16353302101,2sleep00-21swapper/000:32:170
16034112101,2sleep00-21swapper/000:11:000
15882592101,3sleep11303165-21diskmemload00:02:441
14933162101,2sleep20-21swapper/223:05:202
14345262101,3sleep11303165-21diskmemload22:30:031
13692912100,6sleep11369336-21ssh21:50:141
13405182101,4sleep01303165-21diskmemload21:30:200
12808022101,4sleep11280942-21ntp_192.168.11420:45:201
12766852101,4sleep11276733-21munin-run20:45:011
12766852101,4sleep11276733-21munin-run20:45:001
12593462101,4sleep21259435-21ntpq20:25:222
12489682105,2sleep20-21swapper/220:15:242
12449092101,4sleep11245167-21df_inode20:15:151
12247092101,5sleep10-21swapper/119:55:191
12094512101,5sleep01211178-21kthreadcore19:40:180
12065822101,4sleep21206723-21ntpq19:35:202
11975022101,4sleep21197754-21df19:30:142
1595002291,2sleep11303165-21diskmemload00:05:451
1298124291,2sleep10-21swapper/121:05:121
1250179290,1sleep00-21swapper/020:20:140
1244888291,3sleep01244906-21apt-get20:15:130
1243932291,4sleep21244059-21sed20:10:262
1243585291,3sleep01243636-21ntpq20:10:200
1218131290,1sleep20-21swapper/219:48:112
1198026291,3sleep11198452-21cut19:30:171
1192498291,4sleep11192707-21hddtemp_smartct19:25:171
1186255290,1sleep20-21swapper/219:15:562
1288084281,3sleep21288122-21kthreadcore20:55:192
1287110280,1sleep20-21swapper/220:53:152
1276947280,3sleep20-21swapper/220:45:132
1271151280,2sleep10-21swapper/120:37:351
1270189280,1sleep20-21swapper/220:35:202
1264419281,3sleep01264977-21strings20:30:200
1261549281,3sleep21262626-21cut20:30:182
1234729281,2sleep00-21swapper/020:05:190
1232114280,5sleep21233009-21missed_timers20:00:202
1229156281,2sleep00-21swapper/020:00:140
1223839280,3sleep00-21swapper/019:55:140
1218565281,3sleep11218786-21cstates19:50:131
1212789280,1sleep10-21swapper/119:41:591
1186269281,4sleep1802-21snmpd19:16:071
1182946281,3sleep01183600-21kthreadcore19:15:170
1297309270,3sleep20-21swapper/221:00:292
1293444271,3sleep01295102-21kthreadcore21:00:180
1286961270,1sleep00-21swapper/020:51:200
1270988270,4sleep00-21swapper/020:35:300
1260834270,4sleep10-21swapper/120:30:011
1238421270,2sleep21238470-21ntpq20:05:262
1228780272,2sleep21174016-21cyclictest19:58:572
1223540270,1sleep20-21swapper/219:54:352
1213291271,2sleep01213310-21apt-get19:45:120
1203777271,3sleep00-21swapper/019:35:180
1203008271,3sleep11202727-21/usr/sbin/munin19:35:151
1192208270,2sleep20-21swapper/219:25:142
1302803260,1sleep00-21swapper/021:05:310
1298981261,2sleep20-21swapper/221:05:172
1297751260,1sleep10-21swapper/121:03:371
1292477260,1sleep10-21swapper/120:59:111
1287504260,1sleep00-21swapper/020:55:060
1282790260,2sleep10-21swapper/120:50:181
1255695261,2sleep10-21swapper/120:25:171
1254983260,4sleep10-21swapper/120:22:591
1202258260,1sleep00-21swapper/019:32:240
1177751260,2sleep21178182-21strings19:10:182
1243638250,1sleep10-21swapper/120:10:211
1191049250,2sleep00-21swapper/019:20:220
1187276250,1sleep10-21swapper/119:20:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional