You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 18:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Thu Nov 27, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
153263322080,3sleep21532626-31find07:39:242
1498596212656,65sleep30-21swapper/307:07:123
149850228555,9sleep20-21swapper/207:05:582
149850127962,9sleep10-21swapper/107:05:581
149873627455,13sleep00-21swapper/007:08:570
19074262520,2sleep00-21swapper/012:13:490
1498949993026,2cyclictest0-21swapper/312:15:183
1498949992824,2cyclictest0-21swapper/311:50:003
1498949992722,3cyclictest0-21swapper/309:25:263
1498949992622,2cyclictest0-21swapper/312:25:253
1498949992420,2cyclictest0-21swapper/312:37:593
149894999241,11cyclictest1852745-21chrt11:37:563
19399762231,2sleep00-21swapper/012:31:340
16799952231,8sleep21628160-21diskmemload09:43:342
1498949992320,2cyclictest0-21swapper/311:25:013
1498949992320,1cyclictest0-21swapper/307:39:493
1498949992320,1cyclictest0-21swapper/307:10:143
1498949992319,2cyclictest0-21swapper/309:55:163
149894999231,16cyclictest1628186-21sleep09:10:003
149894999222,13cyclictest1732151-21taskset10:17:583
1498949992218,2cyclictest0-21swapper/307:45:003
1498949992217,3cyclictest0-21swapper/310:35:123
149894999221,16cyclictest1777073-21taskset10:47:493
149894999221,14cyclictest1907342-21taskset12:13:403
18315782212,4sleep20-21swapper/211:25:172
149894999212,13cyclictest1688490-21taskset09:49:563
1498949992117,2cyclictest0-21swapper/311:30:013
1498949992117,2cyclictest0-21swapper/310:00:413
1498949992117,2cyclictest0-21swapper/308:35:003
18918682201,2sleep10-21swapper/112:04:451
18833122201,3sleep00-21swapper/011:58:460
149894999207,7cyclictest1813843-21taskset11:12:103
149894999202,12cyclictest1640180-21taskset09:15:203
1498949992017,1cyclictest0-21swapper/310:20:263
1498949992016,2cyclictest0-21swapper/311:40:163
149894999201,15cyclictest1679129-21chrt09:42:023
149894999201,15cyclictest0-21swapper/312:20:063
149894999201,15cyclictest0-21swapper/310:50:283
149894999201,14cyclictest1648359-21chrt09:20:493
149894999197,9cyclictest1791724-21chrt10:57:153
149894999192,13cyclictest1883132-21taskset11:58:283
149894999192,12cyclictest1633580-21taskset09:10:503
149894999192,11cyclictest1823019-21taskset11:19:563
1498949991915,2cyclictest0-21swapper/308:51:323
1498949991912,5cyclictest66550irq/131-enp1s0-TxRx-309:30:283
149894999191,14cyclictest1873672-21chrt11:50:223
149894999191,14cyclictest1873672-21chrt11:50:223
149894999191,14cyclictest1747310-21chrt10:28:443
149894999191,14cyclictest0-21swapper/312:05:093
149894999191,13cyclictest1616653-21chrt08:55:203
18751482181,3sleep00-21swapper/011:52:370
18751482181,3sleep00-21swapper/011:52:370
149894999188,6cyclictest1606754-21taskset08:45:423
149894999186,9cyclictest0-21swapper/310:10:583
1498949991815,1cyclictest0-21swapper/310:30:163
1498949991814,2cyclictest0-21swapper/309:35:103
149894999181,3cyclictest1800032-21taskset11:04:443
149894999181,14cyclictest1889672-21chrt12:01:133
149894999181,14cyclictest1716339-21chrt10:06:323
149894999181,13cyclictest1843877-21chrt11:30:503
149894999181,11cyclictest1768895-21taskset10:41:033
149894999180,16cyclictest0-21swapper/309:00:163
149894999180,15cyclictest18008042chrt11:05:123
149894999180,15cyclictest0-21swapper/312:30:373
149894999180,13cyclictest15760442sleep308:20:143
19153322171,3sleep00-21swapper/012:17:590
17774932171,4sleep21628160-21diskmemload10:48:332
17551612171,2sleep20-21swapper/210:34:102
16947352172,10sleep21628160-21diskmemload09:53:012
16497362171,3sleep01628160-21diskmemload09:23:400
16418162171,3sleep20-21swapper/209:17:332
149894999171,3cyclictest1511453-21turbostat07:24:553
1498949991713,2cyclictest0-21swapper/307:45:163
149894999171,12cyclictest1694741-21taskset09:53:013
149894999171,12cyclictest15042072sleep307:15:013
149894999171,11cyclictest1580746-21chrt08:25:103
149894999171,10cyclictest1522241-21taskset07:30:113
149894999170,13cyclictest0-21swapper/308:35:103
149894999170,13cyclictest0-21swapper/307:25:183
149894999170,11cyclictest1569633-21taskset08:11:043
19430022161,4sleep11628160-21diskmemload12:35:021
19335422161,4sleep21628160-21diskmemload12:29:272
19246402161,9sleep11628160-21diskmemload12:23:571
19158002161,3sleep21915811-21ssh12:18:372
18506952161,8sleep01850752-21ntpq11:35:210
18053402161,9sleep01628160-21diskmemload11:05:310
17921952160,3sleep00-21swapper/010:58:090
17328522161,8sleep01732927-21munin-run10:20:010
16709662161,10sleep21628160-21diskmemload09:36:172
16584532161,10sleep21658507-21forks09:30:162
16340532161,9sleep00-21swapper/009:11:540
1498949991614,1cyclictest0-21swapper/308:20:013
149894999161,10cyclictest1564330-21chrt08:05:463
149894999160,10cyclictest1554333-21chrt08:00:053
19244102151,2sleep20-21swapper/212:23:382
19158852151,2sleep10-21swapper/112:18:431
18519752151,2sleep10-21swapper/111:36:341
18298552150,2sleep20-21swapper/211:23:412
18077922151,9sleep11807867-21sed11:10:011
17999772150,3sleep11628160-21diskmemload11:04:391
17682082152,8sleep01628160-21diskmemload10:40:250
17153142151,7sleep11715371-21ntpq10:05:211
17021052151,3sleep00-21swapper/009:57:480
16622852151,9sleep11662340-21ntpq09:30:231
16334132151,8sleep20-21swapper/209:10:352
149894999151,10cyclictest1545918-21chrt07:50:193
149894999150,12cyclictest0-21swapper/308:40:113
149894999150,12cyclictest0-21swapper/307:55:203
19346492141,7sleep11934824-21/usr/sbin/munin12:30:141
19079082141,3sleep11907922-21ssh12:14:311
19065302141,9sleep21906543-21sh12:12:272
18660272141,8sleep01628160-21diskmemload11:45:230
18394422140,6sleep01840320-21cut11:30:160
17760062140,2sleep00-21swapper/010:45:460
17474842141,7sleep21628160-21diskmemload10:29:052
17469722141,8sleep11746978-21ssh10:28:031
17320582141,7sleep11628160-21diskmemload10:17:451
16659372141,8sleep11666094-21gltestperf09:35:121
16496002144,5sleep21634014-21kworker/u16:3+nfsiod09:23:262
19499572131,3sleep21949968-21ssh12:38:172
19496952131,7sleep01628160-21diskmemload12:37:550
19303002131,8sleep01930502-21sed12:25:240
18976202131,8sleep11897633-21ssh12:06:011
18836692131,8sleep21883679-21ssh11:59:202
18532812131,6sleep20-21swapper/211:38:522
18282642131,8sleep01828271-21sh11:20:300
18123202131,5sleep11812371-21ntpq11:10:211
18000612130,7sleep2431ktimers/211:04:482
17939052131,4sleep01795334-21ssh11:00:170
17256062133,6sleep21725616-21sh10:14:572
17243412131,7sleep01724350-21ssh10:12:390
16945252131,7sleep11694533-21ssh09:52:311
16920052131,7sleep01692816-21memory09:50:180
16856152131,8sleep11681146-21/usr/sbin/munin09:45:251
15504612131,7sleep21552250-21kthreadcore07:55:152
19222712121,4sleep01628160-21diskmemload12:20:390
18902992121,5sleep20-21swapper/212:02:132
18837682121,5sleep11628160-21diskmemload11:59:301
18617012121,5sleep10-21swapper/111:45:041
18603842120,3sleep21860393-21ssh11:42:452
18224002121,3sleep11822409-21ssh11:18:501
17826122121,5sleep01782668-21ntpq10:50:190
17688882121,4sleep21628160-21diskmemload10:41:032
17687572121,4sleep11628160-21diskmemload10:40:501
17001302121,5sleep11700186-21ntpq09:55:191
16734142121,5sleep01628160-21diskmemload09:40:120
16672102122,2sleep0171rcu_preempt09:35:140
16419802121,3sleep00-21swapper/009:17:510
16330612121,7sleep11628512-21/usr/sbin/munin09:10:261
19410642111,4sleep21628160-21diskmemload12:33:042
19257612111,3sleep10-21swapper/112:25:081
18985882111,4sleep21628160-21diskmemload12:07:382
18923262111,4sleep01628160-21diskmemload12:05:040
18894712111,3sleep00-21swapper/012:00:550
18669632111,2sleep20-21swapper/211:46:102
18596952111,4sleep11628160-21diskmemload11:41:311
18438682111,4sleep21628160-21diskmemload11:30:502
18384032111,3sleep10-21swapper/111:29:491
18369882111,4sleep01628160-21diskmemload11:27:110
18234282111,3sleep11628160-21diskmemload11:20:101
18216442111,4sleep21628160-21diskmemload11:17:312
18198212111,4sleep01819880-21ntpq11:15:210
18142282111,4sleep21628160-21diskmemload11:12:542
17929172111,3sleep10-21swapper/110:59:461
17912952111,4sleep21628160-21diskmemload10:56:272
17850382111,4sleep21628160-21diskmemload10:53:252
17835332111,4sleep11628160-21diskmemload10:50:481
17614932111,4sleep21628160-21diskmemload10:36:542
17578402111,6sleep11758378-21ssh10:35:171
17560072111,6sleep01756120-21cut10:35:120
17541302111,4sleep11628160-21diskmemload10:32:121
17540772110,7sleep01754090-21ssh10:32:070
17405292111,4sleep11628160-21diskmemload10:24:591
17405202111,4sleep01628160-21diskmemload10:24:580
17259642111,2sleep20-21swapper/210:15:092
17178092111,4sleep01628160-21diskmemload10:09:180
17085482111,4sleep11628160-21diskmemload10:00:291
17078452111,6sleep01707896-21ntpq10:00:200
17032092111,4sleep21628160-21diskmemload09:59:462
16872422111,4sleep21628160-21diskmemload09:47:352
16802492111,4sleep11628160-21diskmemload09:44:021
16572252111,4sleep11628160-21diskmemload09:28:381
16494732111,3sleep11628160-21diskmemload09:23:131
16240382111,6sleep21625744-21kthreadcore09:05:142
16218812111,4sleep01621931-21ntpq09:00:190
16181422110,7sleep11618168-21fschecks_count09:00:121
15918132111,5sleep21592023-21hddtemp_smartct08:35:122
15582082111,5sleep11558349-21ntpq08:00:161
15030612111,5sleep21503149-21ntpq07:10:202
18751722101,3sleep20-21swapper/211:52:392
18751722101,3sleep20-21swapper/211:52:392
18750342101,3sleep11628160-21diskmemload11:52:261
18750342101,3sleep11628160-21diskmemload11:52:251
18608032101,3sleep01628160-21diskmemload11:43:310
18454052101,3sleep10-21swapper/111:33:381
18087342101,5sleep01809956-21tr11:10:160
18056832101,3sleep21805697-21ssh11:06:092
17761752101,4sleep11628160-21diskmemload10:46:071
17457012101,3sleep00-21swapper/010:25:410
17382952101,3sleep20-21swapper/210:20:512
17238722101,3sleep10-21swapper/110:11:401
17153752101,3sleep20-21swapper/210:05:232
17088332101,4sleep21628160-21diskmemload10:00:582
16863432101,3sleep01628160-21diskmemload09:46:050
16566212101,3sleep20-21swapper/209:27:352
16562322101,3sleep00-21swapper/009:26:520
16408362101,3sleep10-21swapper/109:15:351
16021062101,3sleep21602125-21apt-get08:45:142
15973652101,5sleep01597832-21cut08:40:150
15865072101,5sleep244-21ksoftirqd/208:30:152
15795222101,4sleep01579579-21ntp_states08:20:190
15794192101,5sleep21579465-21ntp_kernel_err08:20:172
15583502101,4sleep21558396-21ntpq08:00:172
15475362101,4sleep11547577-21ntpq07:50:221
15371312101,5sleep01537193-21grep07:40:240
15268112101,4sleep11526903-21users07:30:271
15219452100,7sleep21516694-21turbostat.cron07:29:552
14991812101,4sleep01499401-21df_inode07:10:140
1658193291,3sleep01658220-21apt-get09:30:140
1616389291,4sleep01612599-21/usr/sbin/munin08:55:180
1591493291,3sleep01590996-21apt-get08:35:020
1586249291,4sleep01586274-21/usr/sbin/munin08:30:130
1579847291,4sleep11580092-21users08:20:241
1570204291,3sleep11570223-21apt-get08:15:111
1558680291,4sleep01558927-21awk08:00:230
1553780291,5sleep11553978-21cut08:00:001
1543569291,4sleep21543756-21expr07:50:152
1517230291,4sleep01517373-21gltestperf07:25:120
1515645291,4sleep01515687-21ntpq07:20:200
1511438291,4sleep1618-21gdbus07:20:001
1623649281,3sleep11624036-21kthreadcore09:05:131
1617586281,3sleep11617657-21unin-run09:00:001
1607353281,3sleep00-21swapper/008:50:130
1599565280,3sleep21600682-21memory08:40:172
1584791283,2sleep01584840-21open_inodes08:25:190
1570788281,3sleep20-21swapper/208:15:162
1569388280,1sleep00-21swapper/008:10:250
1564919281,2sleep11564938-21apt-get08:10:101
1549270281,3sleep01549386-21/usr/sbin/munin07:55:140
1537995280,1sleep20-21swapper/207:44:342
1527538281,2sleep01527557-21apt-get07:35:100
1606168270,3sleep01602107-21/usr/sbin/munin08:45:240
1580724270,2sleep20-21swapper/208:25:102
1564319270,1sleep20-21swapper/208:05:382
1563713271,2sleep00-21swapper/008:05:200
1559911271,3sleep10-21swapper/108:05:121
1543040270,1sleep10-21swapper/107:46:581
1539791271,2sleep00-21swapper/007:45:160
1532666270,2sleep10-21swapper/107:39:271
1526975270,1sleep00-21swapper/007:31:140
1516620270,1sleep20-21swapper/207:24:142
1516509271,3sleep1802-21snmpd07:22:491
1511359271,3sleep0802-21snmpd07:19:030
1621984260,3sleep20-21swapper/209:00:212
1616599260,2sleep21616645-21ntpq08:55:202
1612068260,1sleep10-21swapper/108:51:341
1607635261,2sleep20-21swapper/208:50:152
1597833261,2sleep10-21swapper/108:40:161
1591512260,3sleep10-21swapper/108:35:031
1590678260,1sleep10-21swapper/108:31:061
1585546260,3sleep10-21swapper/108:27:541
1569790260,4sleep20-21swapper/208:13:042
1543538260,3sleep00-21swapper/007:50:090
1522219260,2sleep20-21swapper/207:30:112
1511186260,1sleep20-21swapper/207:16:522
1499402261,2sleep10-21swapper/107:10:151
1627679250,2sleep00-21swapper/009:05:240
1573971250,1sleep00-21swapper/008:15:170
1542153250,1sleep20-21swapper/207:45:172
1536907251,1sleep10-21swapper/107:40:201
1602047240,1sleep10-21swapper/108:45:011
1516725240,1sleep10-21swapper/107:25:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional