You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-02 - 06:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot4s.osadl.org (updated Sun Nov 02, 2025 00:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18756142690,2chrt451irq_work/300:21:353
15924232690,2sleep3586-21kworker/3:2-events19:36:343
1563227996647,15cyclictest1595016-21latency_hist19:40:010
1563227996450,10cyclictest282-21systemd-journal22:35:010
1563227996444,17cyclictest1644731-21latency_hist20:30:000
1563227996349,10cyclictest282-21systemd-journal23:10:010
1563227996347,11cyclictest282-21systemd-journal21:40:000
1563231996253,6cyclictest1709250-21latency_hist21:35:011
16236592610,4sleep00-21swapper/020:06:360
1563231995852,4cyclictest1783625-21latency_hist22:50:011
1563231995751,4cyclictest1684201-21kworker/u8:2+flush-8:021:16:441
1563231995651,3cyclictest1783639-21kworker/u8:2+flush-8:023:51:441
1563231995647,6cyclictest1719183-21latency_hist21:45:011
1563231995645,4cyclictest1600243-21kworker/u8:1+flush-8:020:21:461
1563235995543,9cyclictest1729035-21awk21:55:002
1563231995449,4cyclictest1684201-21kworker/u8:2+events_unbound21:10:011
1563231995441,9cyclictest1614871-21apt-get20:00:001
1563231995348,4cyclictest1783639-21kworker/u8:2+flush-8:023:46:431
1563231995347,5cyclictest1664741-21kworker/u8:1+flush-8:021:11:381
1563231995246,4cyclictest1684209-21kworker/u8:4+flush-8:021:36:311
1563231995243,6cyclictest1585128-21latency_hist19:30:011
1563227995231,18cyclictest282-21systemd-journal20:40:000
1563235995145,4cyclictest1867878-21gpgv00:15:002
1563231995143,5cyclictest1858209-21kworker/u8:0+flush-8:000:11:441
1563231995045,4cyclictest1649429-21kworker/u8:0+events_unbound21:00:011
1563227995029,18cyclictest282-21systemd-journal22:55:000
1563231994942,5cyclictest1729255-21kworker/u8:1+events_unbound22:21:371
1563231994942,5cyclictest1729255-21kworker/u8:1+events_unbound22:21:371
1563231994842,4cyclictest1783639-21kworker/u8:2+flush-8:023:00:011
1563227994836,9cyclictest1675212-21diskstats21:01:280
1563235994737,7cyclictest1808412-21awk23:15:002
1563231994641,3cyclictest1887069-21ntpq00:31:431
1563231994641,3cyclictest1625212-21kworker/u8:2+events_unbound20:41:311
1563231994640,4cyclictest1696842-21kworker/u8:0+flush-8:021:26:281
1563227994626,17cyclictest1729026-21latency_hist21:55:010
1563235994540,4cyclictest1798263-21kworker/u8:1+flush-8:023:36:422
1563231994541,3cyclictest1708875-21kworker/u8:2+events_unbound21:46:261
1563231994540,3cyclictest1729255-21kworker/u8:1+flush-8:022:11:301
1563231994539,4cyclictest1558176-21kworker/u8:0+events_unbound19:51:341
1563231994539,4cyclictest1558176-21kworker/u8:0+events_unbound19:41:301
1563231994539,4cyclictest1558176-21kworker/u8:0+events_unbound19:31:351
1563227994535,8cyclictest1634055-21ntpq20:16:380
1563227994526,16cyclictest1867855-21latency_hist00:15:000
1563227994524,8cyclictest1877154-21ntpq00:21:390
16712972440,2sleep3586-21kworker/3:2+events20:56:323
1563235994440,3cyclictest1605222-21kworker/u8:3+flush-8:020:01:442
1563231994440,3cyclictest1798263-21kworker/u8:1+flush-8:023:41:311
1563231994440,3cyclictest1798263-21kworker/u8:1+events_unbound23:36:361
1563231994439,4cyclictest1696842-21kworker/u8:0+events_unbound23:16:261
1563231994439,4cyclictest1696842-21kworker/u8:0+events_unbound22:16:371
1563231994439,4cyclictest1511209-21kworker/u8:3+flush-8:019:21:311
1563231994438,5cyclictest1625212-21kworker/u8:2+flush-8:020:36:301
1563231994437,5cyclictest1625212-21kworker/u8:2+flush-8:020:46:371
1563231994431,10cyclictest282-21systemd-journal21:05:001
1563231994337,4cyclictest1600243-21kworker/u8:1+events_unbound20:26:391
1563231994331,9cyclictest1574191-21ntpq19:16:411
1563227994323,4cyclictest1862914-21awk00:10:000
1563231994237,4cyclictest1558176-21kworker/u8:0+events_unbound19:36:281
1563231994236,4cyclictest1696842-21kworker/u8:0+flush-8:021:51:321
1563231994236,4cyclictest1589794-21kworker/u8:2+events_unbound19:46:331
1563231994234,6cyclictest1696842-21kworker/u8:0+flush-8:023:31:351
1563227994234,5cyclictest1777849-21ntpq22:41:350
1563227994221,3cyclictest0-21swapper/020:35:000
1563227994221,3cyclictest0-21swapper/020:35:000
18737252410,4sleep2371rcuc/200:21:302
17646622410,2sleep0141ktimers/022:31:320
1563235994136,4cyclictest1625212-21kworker/u8:2+events_unbound20:31:342
1563235994136,4cyclictest1625212-21kworker/u8:2+events_unbound20:31:342
1563231994136,3cyclictest1783639-21kworker/u8:2+flush-8:023:21:371
1563231994135,4cyclictest1708875-21kworker/u8:2+events_unbound22:26:401
1563231994134,5cyclictest1783639-21kworker/u8:2+flush-8:000:26:411
1563227994124,3cyclictest1857856-21systemd-run00:05:000
1563227994121,2cyclictest1661067-21kthreadcore20:46:320
1563231994035,3cyclictest1605222-21kworker/u8:3+events_unbound20:51:311
1563231994034,4cyclictest1783639-21kworker/u8:2+events_unbound23:56:271
1563231994034,4cyclictest1729255-21kworker/u8:1+flush-8:022:36:451
1563231994033,5cyclictest1558176-21kworker/u8:0+events_unbound20:10:001
1563231994029,8cyclictest282-21systemd-journal00:01:511
1563227994030,7cyclictest1733965-21latency_hist22:00:010
1563231993929,4cyclictest1783639-21kworker/u8:2+events_unbound00:16:381
1563231993834,3cyclictest1729255-21kworker/u8:1+flush-8:022:50:021
1563231993833,3cyclictest1600243-21kworker/u8:1+flush-8:020:31:461
1563231993833,3cyclictest1600243-21kworker/u8:1+flush-8:020:31:461
1563231993832,4cyclictest1708875-21kworker/u8:2+events_unbound22:01:391
1563231993832,4cyclictest1696842-21kworker/u8:0+events_unbound22:31:401
1563231993828,8cyclictest1698557-21ntpq21:21:381
1563227993827,8cyclictest1813323-21latency_hist23:20:010
1563231993730,5cyclictest1696842-21kworker/u8:0+events_unbound21:56:301
1563231993726,8cyclictest282-21systemd-journal19:15:001
1563227993728,7cyclictest1579956-21latency_hist19:25:010
1563231993631,4cyclictest1696842-21kworker/u8:0+flush-8:022:41:431
1563227993630,4cyclictest1842627-21grep23:46:430
1563235993529,4cyclictest1600243-21kworker/u8:1+flush-8:020:06:352
1563231993531,3cyclictest1858209-21kworker/u8:0+events_unbound00:21:271
1563231993530,4cyclictest1783639-21kworker/u8:2+flush-8:023:06:291
1563235993430,3cyclictest1558176-21kworker/u8:0+flush-8:019:36:342
1563231993430,2cyclictest1892815-21systemd-run00:40:001
1563231993429,4cyclictest1625212-21kworker/u8:2+events_unbound20:15:061
1563231993429,3cyclictest1862096-21ntpq00:06:391
1563231993429,3cyclictest1783639-21kworker/u8:2+events_unbound23:01:411
1563235993323,8cyclictest1689462-21cat21:15:002
1563231993328,4cyclictest1696842-21kworker/u8:0+flush-8:023:26:431
1563231993328,3cyclictest1629954-21systemd-run20:15:001
1563227993328,3cyclictest1729255-21kworker/u8:1+events_unbound22:06:350
1563235993127,3cyclictest1625212-21kworker/u8:2+events_unbound20:11:472
1563231993127,3cyclictest1783639-21kworker/u8:2+flush-8:023:11:441
1563231993127,3cyclictest1708875-21kworker/u8:2+events_unbound22:06:521
1563227993122,6cyclictest1823231-21latency_hist23:30:010
18485302290,1sleep01848532-21cut23:56:270
156323599294,8cyclictest1808849-21cut23:16:252
1563227992925,2cyclictest1575044-21systemd-run19:20:000
1563227992921,6cyclictest1600635-21fschecks_count19:46:300
1563235992822,4cyclictest1748421-21unixbench_multi22:11:412
1563235992822,4cyclictest1664741-21kworker/u8:1+events_unbound20:56:432
1563235992821,3cyclictest1798263-21kworker/u8:1+events_unbound23:40:022
156322799289,17cyclictest1575041-21kworker/0:2+events19:57:310
1563227992823,3cyclictest1718825-21unixbench_multi21:41:430
156322799282,25cyclictest1575041-21kworker/0:2+events20:11:350
1563227992822,4cyclictest1808040-21unixbench_multi23:11:450
1563227992818,7cyclictest1590061-21latency_hist19:35:010
1563235992623,2cyclictest1838717-21fschecks_time23:46:272
1563235992622,3cyclictest1708875-21kworker/u8:2+events_unbound22:01:432
1563235992621,4cyclictest1798210-21unixbench_singl23:01:462
1563235992620,4cyclictest1852541-21unixbench_singl23:56:432
1563231992621,4cyclictest1600243-21kworker/u8:1+flush-8:020:01:551
156322799268,16cyclictest1828187-21kworker/0:0+events23:36:390
1563227992621,3cyclictest1748804-21systemd-run22:15:000
1563227992620,4cyclictest1659495-21unixbench_singl20:41:450
1563227992620,3cyclictest0-21swapper/022:20:000
156322799260,25cyclictest1649784-21kworker/0:0+events21:20:050
18084862250,2sleep3481ktimers/323:15:013
1563235992522,2cyclictest1768669-21dpkg22:35:012
1563235992522,2cyclictest1669311-21unixbench_multi20:51:432
1563235992521,3cyclictest1605222-21kworker/u8:3+events_unbound19:51:452
1563235992521,2cyclictest1570533-21aten2_r5power_v19:16:292
1563235992520,3cyclictest1778404-21unixbench_singl22:41:432
1563235992520,3cyclictest1558176-21kworker/u8:0+flush-8:019:40:012
1563235992515,8cyclictest1580374-21cat19:25:012
1563227992521,2cyclictest0-21swapper/021:20:000
156322799251,9cyclictest1883260-21apt-get00:31:270
1563235992421,2cyclictest1872681-21unixbench_singl00:16:452
1563235992419,4cyclictest1729255-21kworker/u8:1+flush-8:022:35:212
1563235992419,4cyclictest1708926-21unixbench_singl21:31:432
1563235992419,3cyclictest1634499-21unixbench_multi20:16:442
156322799247,16cyclictest1773814-21kworker/0:1+events23:31:310
1563227992418,4cyclictest587-21runrttasks21:11:260
1563227992418,4cyclictest1584756-21unixbench_multi19:26:450
1563227992416,5cyclictest1609932-21latency_hist19:55:010
15628392242,16sleep00-21swapper/019:07:260
1563235992320,2cyclictest1886927-21needreboot00:31:392
1563235992319,3cyclictest371rcuc/223:31:352
1563235992318,3cyclictest1818937-21fschecks_count23:26:272
1563235992316,5cyclictest1590069-21awk19:35:002
156323599230,9cyclictest1801208-21kthreadcore23:06:342
156322799238,13cyclictest1773814-21kworker/0:1+events22:46:270
156322799237,14cyclictest1575041-21kworker/0:2+events19:41:270
156322799234,17cyclictest1649784-21kworker/0:0+events22:24:430
156322799234,17cyclictest1649784-21kworker/0:0+events22:24:430
156322799233,18cyclictest1649784-21kworker/0:0+events21:46:250
1563227992318,3cyclictest1620031-21/usr/sbin/munin20:05:000
1563227992318,3cyclictest0-21swapper/022:01:260
1563227992317,5cyclictest1605222-21kworker/u8:3+flush-8:020:51:220
156322799231,18cyclictest584-21snmpd23:01:390
156323599222,14cyclictest1754091-21apt-get22:21:262
156323599222,14cyclictest1754091-21apt-get22:21:262
1563235992218,3cyclictest1783616-21turbostat.cron22:50:002
1563235992218,3cyclictest1739511-21cpuspeed_turbos22:06:282
1563235992217,4cyclictest1696842-21kworker/u8:0+flush-8:021:21:492
1563235992212,7cyclictest1818515-21cat23:25:012
156322799227,14cyclictest1575041-21kworker/0:2+events20:21:350
156322799226,14cyclictest1858177-21kworker/0:3+events00:26:350
156322799222,4cyclictest1670031-21apt-get20:56:250
156322799222,18cyclictest1858177-21kworker/0:3+events00:36:270
1563227992217,3cyclictest0-21swapper/022:30:000
156322799220,21cyclictest1649784-21kworker/0:0+events21:26:370
156323599212,2cyclictest0-21swapper/221:46:442
1563235992118,2cyclictest1789068-21aten2_r5power_v22:56:272
1563235992117,2cyclictest1690039-21cpuspeed_turbos21:16:272
1563235992117,2cyclictest1580599-21aten2_r5power_e19:26:272
156323599211,5cyclictest16799812wget21:06:282
156323599210,6cyclictest1893011-21latency_hist00:40:012
156322799214,15cyclictest1773814-21kworker/0:1+events23:21:390
156322799211,9cyclictest584-21snmpd21:31:240
1563227992112,6cyclictest1872242-21ntpq00:16:410
1563227992111,8cyclictest1792703-21ntpq22:56:380
156323599208,4cyclictest0-21swapper/221:36:412
156323599203,6cyclictest0-21swapper/220:36:332
156323599202,13cyclictest1878327-21apt-get00:26:262
1563235992017,2cyclictest1749165-21aten2_r5power_c22:16:262
1563235992016,3cyclictest0-21swapper/221:26:442
1563235992016,2cyclictest1640502-21fschecks_time20:26:292
1563235992016,2cyclictest0-21swapper/219:55:152
1563235992010,8cyclictest1857523-21awk00:01:432
1563235992010,8cyclictest1847976-21sed23:55:002
15584112204,10sleep10-21swapper/119:05:261
156323599192,2cyclictest0-21swapper/200:06:262
1563235991916,2cyclictest1719173-21cron21:45:002
1563235991915,3cyclictest1730309-21kthreadcore21:56:362
1563235991915,3cyclictest1604969-21turbostat.cron19:50:002
1563235991915,2cyclictest0-21swapper/220:41:332
1563235991914,3cyclictest0-21swapper/222:26:442
1563227991914,3cyclictest0-21swapper/023:41:350
1563227991912,5cyclictest0-21swapper/019:11:370
156322799191,17cyclictest1649784-21kworker/0:0+events21:06:410
156322799191,16cyclictest1828187-21kworker/0:0+events23:51:390
15627942194,10sleep20-21swapper/219:06:522
15587472193,8sleep30-21swapper/319:06:293
1563235991815,2cyclictest0-21swapper/220:48:322
1563235991810,6cyclictest1784747-21sed22:51:322
1563235991810,6cyclictest1675747-21sed21:01:322
1563235991810,6cyclictest1637301-21cut20:21:362
156323599162,2cyclictest0-21swapper/219:11:262
18901352143,7sleep30-21swapper/300:36:363
17790662142,8sleep30-21swapper/322:46:253
18570042131,8sleep30-21swapper/300:01:373
18488402133,7sleep30-21swapper/323:56:313
17340372132,6sleep30-21swapper/322:00:013
17204222132,7sleep30-21swapper/321:46:333
16844422133,5sleep30-21swapper/321:10:013
16349022132,6sleep30-21swapper/320:20:013
16241612132,7sleep30-21swapper/320:06:403
16099152131,7sleep30-21swapper/319:55:013
15968932133,6sleep30-21swapper/319:41:363
18871592122,6sleep30-21swapper/300:31:473
18787532122,6sleep30-21swapper/300:26:333
18582482122,6sleep30-21swapper/300:06:273
18335852122,6sleep30-21swapper/323:41:263
18323552122,6sleep30-21swapper/323:36:413
18271792122,6sleep30-21swapper/323:31:393
18182912122,5sleep30-21swapper/323:25:013
17988752122,6sleep30-21swapper/323:06:273
17938632122,6sleep30-21swapper/323:01:253
17921992122,6sleep30-21swapper/322:56:363
17781462122,6sleep30-21swapper/322:41:433
17685292121,7sleep30-21swapper/322:35:013
17528972122,7sleep30-21swapper/322:16:393
17438352122,5sleep30-21swapper/322:10:013
17335882122,6sleep30-21swapper/321:56:483
17040502122,6sleep30-21swapper/321:26:513
16902942122,6sleep30-21swapper/321:16:313
16753362122,6sleep30-21swapper/321:01:303
16307692122,6sleep30-21swapper/320:16:323
16138762122,6sleep30-21swapper/319:56:373
16000402122,6sleep30-21swapper/319:45:013
15890662122,6sleep30-21swapper/319:31:373
15843032122,7sleep30-21swapper/319:26:393
18683132113,5sleep30-21swapper/300:15:093
18633292112,5sleep30-21swapper/300:11:283
18429872112,5sleep30-21swapper/323:50:013
18387492112,5sleep30-21swapper/323:46:283
18144112112,5sleep30-21swapper/323:21:323
18127322112,5sleep30-21swapper/323:16:423
17851372112,5sleep30-21swapper/322:51:343
17690192112,5sleep30-21swapper/322:35:173
17593802112,5sleep30-21swapper/322:26:303
17543272112,5sleep30-21swapper/322:21:283
17543272112,5sleep30-21swapper/322:21:283
17244632112,6sleep30-21swapper/321:51:273
17145112112,6sleep30-21swapper/321:40:513
17092682112,5sleep30-21swapper/321:35:013
17056672112,5sleep30-21swapper/321:31:323
16947332113,5sleep30-21swapper/321:21:133
16847082112,6sleep30-21swapper/321:10:213
16655992112,5sleep30-21swapper/320:51:313
16601452112,5sleep30-21swapper/320:46:253
16557092112,6sleep30-21swapper/320:41:323
16504912112,5sleep30-21swapper/320:36:303
16450922112,5sleep30-21swapper/320:31:293
16450922112,5sleep30-21swapper/320:31:293
16400812112,5sleep30-21swapper/320:25:513
16275142110,6sleep30-21swapper/320:11:383
16154162112,6sleep30-21swapper/320:01:033
15750352112,5sleep30-21swapper/319:20:013
15695522112,5sleep30-21swapper/319:15:043
15673222112,5sleep30-21swapper/319:11:383
17391622102,5sleep30-21swapper/322:05:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional