You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 14:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Fri Nov 21, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
451401212462,58sleep30-21swapper/307:07:403
45132128568,10sleep10-21swapper/107:06:371
45146528362,10sleep20-21swapper/207:08:312
45153928055,19sleep00-21swapper/007:09:250
7341052270,1sleep30-21swapper/310:50:003
6575372261,8sleep0657803-21df_inode10:00:120
451710992622,2cyclictest0-21swapper/310:35:173
451710992622,2cyclictest0-21swapper/309:10:143
451710992522,1cyclictest0-21swapper/309:20:163
7396362241,3sleep00-21swapper/010:50:470
451710992421,1cyclictest0-21swapper/312:01:093
45171099232,16cyclictest661589-21chrt10:00:183
45171099232,16cyclictest618944-21taskset09:35:123
451710992320,1cyclictest0-21swapper/309:50:143
451710992320,1cyclictest0-21swapper/308:20:003
451710992319,1cyclictest0-21swapper/312:15:123
45171099230,21cyclictest0-21swapper/311:42:363
45171099230,19cyclictest904785-21taskset12:36:563
451710992218,2cyclictest0-21swapper/309:45:163
451710992216,3cyclictest0-21swapper/309:40:183
45171099221,17cyclictest808632-21taskset11:37:393
45171099221,16cyclictest894302-21chrt12:30:193
45171099220,17cyclictest685529-21taskset10:15:333
45171099220,16cyclictest6497242sleep309:55:103
451710992118,1cyclictest0-21swapper/307:38:453
451710992117,2cyclictest0-21swapper/311:15:523
451710992116,2cyclictest0-21swapper/310:55:003
451710992114,5cyclictest779440-21turbostat11:24:553
451710992113,4cyclictest8635492sleep312:12:083
45171099211,15cyclictest695213-21chrt10:24:233
8572242201,2sleep00-21swapper/012:09:560
45171099206,2cyclictest0-21swapper/310:56:303
45171099202,13cyclictest882135-21taskset12:25:023
45171099201,7cyclictest0-21swapper/311:07:193
451710992017,1cyclictest0-21swapper/312:25:013
451710992017,1cyclictest0-21swapper/310:30:183
45171099201,15cyclictest755636-21chrt11:03:283
451710992011,4cyclictest8232122sleep311:46:543
45171099200,18cyclictest0-21swapper/310:11:283
6634032191,5sleep2580972-21diskmemload10:02:302
6173842191,11sleep2580972-21diskmemload09:32:402
6109402191,3sleep00-21swapper/009:29:450
451710991916,1cyclictest0-21swapper/307:50:173
451710991912,4cyclictest6170732sleep309:32:043
45171099191,15cyclictest7013192chrt10:26:323
45171099191,15cyclictest0-21swapper/311:56:473
45171099191,14cyclictest0-21swapper/311:50:113
45171099191,14cyclictest0-21swapper/311:14:513
45171099191,11cyclictest548980-21taskset08:40:013
451710991910,4cyclictest5598782sleep308:50:123
7616282181,12sleep0761635-21ssh11:05:420
6569022181,5sleep1580972-21diskmemload09:59:371
6015632181,3sleep10-21swapper/109:20:521
45171099182,12cyclictest668242-21taskset10:05:193
45171099181,5cyclictest490804-21sleep07:44:593
45171099181,14cyclictest801079-21chrt11:32:543
45171099181,14cyclictest595154-21chrt09:18:043
45171099181,14cyclictest595154-21chrt09:18:043
45171099181,14cyclictest0-21swapper/311:25:533
45171099181,13cyclictest0-21swapper/312:05:103
45171099181,13cyclictest0-21swapper/307:20:103
8793152171,8sleep0879326-21ssh12:21:230
8720252172,8sleep134-21ksoftirqd/112:17:391
8655822170,3sleep00-21swapper/012:15:010
8645792171,11sleep2580972-21diskmemload12:13:432
8628162171,3sleep10-21swapper/112:11:021
8398892171,4sleep0580972-21diskmemload11:58:020
7558362171,9sleep1580972-21diskmemload11:03:551
7401842171,3sleep10-21swapper/110:51:441
7105662171,4sleep2580972-21diskmemload10:34:252
6246252171,11sleep2624635-21ssh09:37:092
45171099178,6cyclictest517064-21turbostat08:10:003
451710991715,1cyclictest0-21swapper/307:10:183
451710991714,1cyclictest0-21swapper/308:40:143
451710991714,1cyclictest0-21swapper/307:30:163
45171099171,13cyclictest0-21swapper/307:25:093
45171099170,9cyclictest527977-21chrt08:20:133
45171099170,9cyclictest527977-21chrt08:20:133
45171099170,14cyclictest0-21swapper/310:41:313
45171099170,14cyclictest0-21swapper/309:25:463
8871192161,10sleep2887129-21ssh12:25:342
8172812161,9sleep1580972-21diskmemload11:45:001
7944572161,3sleep2580972-21diskmemload11:29:502
7144622161,4sleep2715505-21memory10:35:182
6646592160,1sleep10-21swapper/110:04:381
5959642161,8sleep20-21swapper/209:19:372
5959642161,8sleep20-21swapper/209:19:372
5492462161,10sleep2549492-21expr08:40:132
4630312161,10sleep1463171-21ntp_192.168.11407:15:191
4630312161,10sleep1463171-21ntp_192.168.11407:15:191
45171099166,6cyclictest580991-21turbostat09:10:003
45171099162,9cyclictest491050-21taskset07:45:103
451710991614,1cyclictest0-21swapper/309:00:143
451710991614,1cyclictest0-21swapper/308:45:203
45171099161,13cyclictest0-21swapper/308:10:203
45171099161,12cyclictest0-21swapper/308:25:223
45171099161,10cyclictest501858-21taskset07:55:123
45171099160,13cyclictest0-21swapper/308:30:203
45171099160,10cyclictest511697-21taskset08:03:383
8952812150,3sleep0580972-21diskmemload12:30:370
7778622151,10sleep2777871-21sh11:17:152
6649252151,8sleep0580972-21diskmemload10:05:010
6486922150,2sleep10-21swapper/109:53:421
6240392151,9sleep0624050-21ssh09:36:020
5754242151,9sleep0575446-21munin-run09:05:000
45171099151,9cyclictest565188-21taskset08:55:123
45171099151,11cyclictest464212-21turbostat07:20:003
45171099151,11cyclictest464212-21turbostat07:20:003
8551362141,9sleep1855149-21sh12:06:291
8017642140,2sleep2801774-21ssh11:34:022
7563822141,6sleep2756440-21unin-run11:05:002
7164502140,3sleep0580972-21diskmemload10:35:360
7088942141,7sleep1580972-21diskmemload10:31:121
6796882141,9sleep2679699-21ssh10:13:572
5596122141,9sleep0559668-21latency_hist08:50:010
4594372141,9sleep0459539-21cut07:15:140
4594372141,9sleep0459539-21cut07:15:140
4524302141,8sleep0452926-21cat07:10:170
9070192131,8sleep2907209-21ssh12:40:012
8968372131,8sleep2896849-21ssh12:32:472
8891072131,8sleep0580972-21diskmemload12:28:370
8722432130,2sleep20-21swapper/212:18:002
8435852131,8sleep0845334-21cut12:00:180
8259632131,8sleep0826825-21cut11:50:160
7849232131,5sleep10-21swapper/111:21:031
7782582132,2sleep00-21swapper/011:17:530
7339802131,6sleep2734104-21apt-key10:50:012
7102472131,8sleep0710259-21ssh10:33:500
6864832131,6sleep0580972-21diskmemload10:17:200
6488022130,9sleep2580972-21diskmemload09:53:562
6382372133,5sleep2638328-21ntpq09:45:182
6332822131,8sleep0633286-21sh09:44:000
5214562131,7sleep0521598-21mailstats08:10:210
5178292133,5sleep1518106-21cut08:10:141
5021392131,7sleep1503211-21kthreadcore07:55:151
4853132131,6sleep0485255-31find07:38:590
9054742120,8sleep0905486-21sh12:37:550
8977472121,4sleep1580972-21diskmemload12:34:031
8617542121,5sleep0861807-21ntpq12:10:200
8406502121,4sleep2580972-21diskmemload11:59:122
8316912121,5sleep2580972-21diskmemload11:52:552
8227342120,8sleep0822744-21ssh11:45:590
8158182121,4sleep2580972-21diskmemload11:41:362
8080802121,3sleep2808087-21ssh11:36:422
8022112121,6sleep0802310-21grep11:35:010
8014132121,7sleep1580972-21diskmemload11:33:291
7932982121,4sleep0580972-21diskmemload11:27:390
7858002121,5sleep0580972-21diskmemload11:22:380
7851502121,4sleep2431ktimers/211:21:282
7623352121,4sleep1580972-21diskmemload11:06:551
7489592121,4sleep2580972-21diskmemload10:59:192
7487122121,2sleep1748718-21ssh10:58:421
7406552121,4sleep2580972-21diskmemload10:52:332
7328732121,4sleep0580972-21diskmemload10:48:060
7321422121,4sleep1580972-21diskmemload10:46:471
7190402121,5sleep0580972-21diskmemload10:40:090
7021092121,4sleep1580972-21diskmemload10:27:551
6950172121,7sleep1580972-21diskmemload10:24:021
6882302122,5sleep0688250-21apt-get10:20:120
6873212121,4sleep2580972-21diskmemload10:19:022
6797962121,4sleep0580972-21diskmemload10:14:070
6479102121,4sleep0580972-21diskmemload09:52:220
6391762121,5sleep1802-21snmpd09:45:421
6307122121,5sleep2630764-21ssh09:40:212
6175332121,4sleep1580972-21diskmemload09:32:551
6087822121,2sleep2608792-21ssh09:25:302
6031622121,6sleep0580972-21diskmemload09:24:190
5940752121,5sleep00-21swapper/009:16:140
5940752121,5sleep00-21swapper/009:16:140
5865982121,3sleep20-21swapper/209:11:152
5864762121,5sleep00-21swapper/009:11:010
5615142121,6sleep0563310-21kthreadcore08:50:170
5375472121,5sleep2537791-21users08:25:252
5226122121,7sleep0522698-21tr08:15:140
4871702120,7sleep1488944-21kthreadcore07:40:201
4856962122,5sleep0485714-21apt-get07:40:150
8987712111,4sleep1580972-21diskmemload12:35:071
8874612110,7sleep1887465-21diskmemload12:26:051
8738602111,3sleep10-21swapper/112:20:041
8546262111,4sleep2580972-21diskmemload12:05:372
8415312111,4sleep1580972-21diskmemload12:00:111
8402392111,4sleep1580972-21diskmemload11:58:351
8295172111,4sleep1580972-21diskmemload11:50:211
8224942111,3sleep10-21swapper/111:45:321
8088342111,4sleep1580972-21diskmemload11:38:021
7932222111,4sleep1580972-21diskmemload11:27:281
7783092111,3sleep10-21swapper/111:17:581
7694252111,3sleep00-21swapper/011:10:430
7646122111,3sleep10-21swapper/111:10:121
7482262111,3sleep00-21swapper/010:57:440
7245442111,3sleep20-21swapper/210:41:552
7190872111,4sleep10-21swapper/110:40:141
7179432111,4sleep1580972-21diskmemload10:38:411
7029862111,4sleep0580972-21diskmemload10:29:260
6934632111,3sleep2580972-21diskmemload10:21:182
6858372111,3sleep10-21swapper/110:16:081
6785752111,4sleep1580972-21diskmemload10:12:031
6723682111,4sleep2580972-21diskmemload10:09:322
6707862111,3sleep10-21swapper/110:06:461
6405812111,3sleep00-21swapper/009:48:090
6192902111,3sleep10-21swapper/109:35:151
6174402111,3sleep00-21swapper/009:32:450
6088422111,3sleep1580972-21diskmemload09:25:371
5948212111,3sleep1580972-21diskmemload09:17:311
5948212111,3sleep1580972-21diskmemload09:17:311
5424712111,6sleep1538427-21/usr/sbin/munin08:30:221
4789732111,4sleep1479014-21ntpq07:30:181
4529272111,6sleep1454644-21kthreadcore07:10:181
8791172101,3sleep20-21swapper/212:21:052
8471582101,3sleep2580972-21diskmemload12:01:372
8176112101,2sleep20-21swapper/211:45:092
8145552101,3sleep00-21swapper/011:40:220
8087532101,3sleep00-21swapper/011:37:530
7696332101,3sleep2580972-21diskmemload11:11:052
7622712101,3sleep2580972-21diskmemload11:06:482
7545252101,3sleep0580972-21diskmemload11:00:450
6497032101,3sleep0580972-21diskmemload09:55:070
6265692101,2sleep10-21swapper/109:40:081
5864182101,3sleep10-21swapper/109:10:541
5759342101,4sleep2575432-21apt-get09:05:022
5476172101,4sleep1547665-21ntpq08:35:191
5228792101,4sleep2523088-21hddtemp_smartct08:15:162
5223262101,5sleep2522356-21perf08:15:002
5158952101,4sleep0516036-21ntpq08:05:160
5120652101,4sleep1512084-21apt-get08:05:091
5068422101,4sleep2507094-21df08:00:132
5068202101,3sleep1506839-21apt-get08:00:121
4851882101,3sleep2485262-40gst-plugin-scan07:38:452
701150291,2sleep20-21swapper/210:26:122
654843290,3sleep20-21swapper/209:55:572
601599290,1sleep20-21swapper/209:20:572
576920291,4sleep0578618-21grep09:05:200
559270290,1sleep10-21swapper/108:45:401
503215291,4sleep2505010-21cut07:55:162
495004291,3sleep2495055-21ntpq07:45:182
479022291,3sleep2479071-21ntpq07:30:192
469765291,4sleep2469908-21cstates07:25:102
571007281,4sleep1572390-21cat09:00:161
567014283,2sleep00-21swapper/008:55:180
555129281,3sleep2555161-21kthreadcore08:45:182
554289280,1sleep00-21swapper/008:44:560
548375280,1sleep20-21swapper/208:36:232
538996281,3sleep0540140-21kthreadcore08:30:170
537472280,1sleep10-21swapper/108:25:251
511808283,2sleep0435648-21kworker/u16:2+events_power_efficient08:05:000
496302282,3sleep0496483-21cstates07:50:130
495597280,1sleep10-21swapper/107:45:251
459156280,2sleep20-21swapper/207:15:122
459156280,2sleep20-21swapper/207:15:122
564693270,1sleep20-21swapper/208:52:232
543056272,2sleep2451704-21cyclictest08:30:312
532901270,1sleep10-21swapper/108:24:491
532901270,1sleep10-21swapper/108:24:491
501252272,3sleep2431ktimers/207:54:162
490679270,1sleep20-21swapper/207:43:542
484968270,1sleep10-21swapper/107:35:551
478838271,3sleep0478963-21netstat07:30:170
469348270,1sleep00-21swapper/007:23:200
456008270,4sleep20-21swapper/207:10:232
580947260,4sleep10-21swapper/109:09:491
570427260,1sleep20-21swapper/209:00:052
569986260,1sleep20-21swapper/208:57:562
569804260,1sleep10-21swapper/108:55:341
564833260,4sleep10-21swapper/108:54:071
549224260,1sleep10-21swapper/108:40:121
547375260,2sleep0547611-21strings08:35:180
532655260,1sleep00-21swapper/008:21:430
532655260,1sleep00-21swapper/008:21:430
528477261,2sleep20-21swapper/208:20:172
528477261,2sleep20-21swapper/208:20:172
513144260,2sleep20-21swapper/208:05:142
500621261,2sleep10-21swapper/107:50:261
495788260,1sleep00-21swapper/007:47:080
469449260,4sleep20-21swapper/207:24:342
464747260,2sleep10-21swapper/107:20:121
533746250,2sleep00-21swapper/008:25:160
522700250,1sleep10-21swapper/108:15:151
506373250,4sleep00-21swapper/007:57:590
469743250,1sleep10-21swapper/107:25:091
470033240,1sleep00-21swapper/007:25:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional