You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 21:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sat Jun 13, 2026 12:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18054269911664,49cyclictest0-21swapper/010:05:010
1805426991023,64cyclictest0-21swapper/011:50:020
1805426991023,64cyclictest0-21swapper/011:50:020
1805426999692,2cyclictest0-21swapper/009:20:020
1805426999692,2cyclictest0-21swapper/009:20:020
180543199922,11cyclictest131rcu_preempt09:15:021
1805440998451,19cyclictest0-21swapper/311:05:023
1805440998451,19cyclictest0-21swapper/311:05:023
1805426998236,42cyclictest16950irq/224-800000011:30:000
180543199812,45cyclictest1870735-21rm11:21:571
1805426998177,2cyclictest0-21swapper/011:25:020
180543199801,51cyclictest1883551-21ssh11:50:371
180543199791,51cyclictest1850760-21ssh10:37:301
180543199791,45cyclictest1822451-21ssh09:34:031
1805426997920,53cyclictest16850irq/134-800000010:32:190
1805426997920,53cyclictest16850irq/134-800000010:32:190
180542699791,55cyclictest0-21swapper/008:00:020
180542699791,11cyclictest1894285-21latency_hist12:15:030
180543199783,68cyclictest1878556-21cron11:40:001
180543199782,49cyclictest1844173-21ssh10:22:431
180543199782,41cyclictest1892485-21ssh12:10:381
1805426997727,46cyclictest1854127-21ssh10:45:020
1805426997727,46cyclictest1854127-21ssh10:45:020
180543199761,49cyclictest1849291-21ssh10:34:221
180543199761,49cyclictest1849291-21ssh10:34:221
180543199741,36cyclictest1861191-21ssh11:00:201
180543199741,36cyclictest1861191-21ssh11:00:201
1805426997428,41cyclictest1857788-21ssh10:53:080
180543199731,43cyclictest1881026-21ssh11:45:021
180543199731,43cyclictest1881026-21ssh11:45:021
180543199731,42cyclictest1868553-21ssh11:17:061
1805431997218,30cyclictest1811608-21diskmemload11:13:511
180543199721,40cyclictest1887360-21ssh11:59:431
180542699722,64cyclictest1902125-21ssh12:32:320
180543199711,45cyclictest1895789-21ssh12:18:201
180542699711,53cyclictest1822113-21ssh09:33:170
1805426997113,53cyclictest16850irq/134-800000010:15:080
180543199701,36cyclictest1880049-21ssh11:43:081
180542699702,8cyclictest1-21systemd08:30:000
1805426997025,28cyclictest1837845-21ssh10:08:350
180542699702,18cyclictest1888590-21ssh12:02:040
180542699701,6cyclictest1818845-21ssh09:25:430
1805426997014,39cyclictest16850irq/134-800000009:42:200
1805426997014,39cyclictest16850irq/134-800000009:42:200
1805431996930,35cyclictest1864375-21sh11:07:511
1805426996928,24cyclictest1828885-21ssh09:48:290
1805426996921,31cyclictest1864158-21ssh11:07:090
180542699691,64cyclictest0-21swapper/011:17:570
1805426996814,49cyclictest16850irq/134-800000009:10:250
180543199671,40cyclictest1852765-21ssh10:41:481
180543199671,40cyclictest1852765-21ssh10:41:481
180542699677,40cyclictest16850irq/134-800000012:27:440
1805426996754,9cyclictest1905293-21cron12:40:000
1805426996729,32cyclictest1887490-21cron12:00:000
1805426996723,13cyclictest16850irq/134-800000011:36:010
1805431996632,31cyclictest1827270-21/usr/sbin/munin09:45:011
1805431996632,31cyclictest1827270-21/usr/sbin/munin09:45:011
180543199661,57cyclictest1806805-21cat07:35:021
180543199661,57cyclictest1806805-21cat07:35:021
180543199661,41cyclictest1835858-21ssh10:04:071
1805426996618,30cyclictest16850irq/134-800000009:59:560
1805426996510,49cyclictest16850irq/134-800000010:38:000
180542699649,39cyclictest16850irq/134-800000012:05:210
180542699642,6cyclictest1-21systemd10:23:330
1805426996418,42cyclictest1839973-21ssh10:13:230
1805426996418,42cyclictest1839973-21ssh10:13:230
1805426996417,43cyclictest0-21swapper/012:16:320
180542699641,4cyclictest1814796-21ssh09:16:550
180543199631,58cyclictest1846091-21ssh10:26:491
180543199631,39cyclictest1901446-21ssh12:30:511
180543199631,37cyclictest1860417-21ssh10:58:591
180542699634,53cyclictest0-21swapper/008:10:000
180542699631,56cyclictest1811645-21cat09:10:020
1805426996314,44cyclictest16850irq/134-800000011:50:120
1805431996227,29cyclictest1831138-21sshd09:53:321
180543199621,57cyclictest1891550-21ssh12:08:551
180543199621,56cyclictest1833820-21ssh09:59:441
180542699628,49cyclictest275-21dbus-daemon07:39:590
180542699622,4cyclictest1875302-21ssh11:32:140
180542699622,4cyclictest16850irq/134-800000009:35:280
1805440996142,14cyclictest1-21systemd07:30:013
1805435996151,6cyclictest1813806-21kworker/2:009:19:332
180543199612,55cyclictest1898152-21ssh12:23:441
180543199611,56cyclictest1855401-21ssh10:47:551
180543199611,55cyclictest1829413-21ssh09:49:511
180543199611,54cyclictest1875789-21ssh11:33:321
180543199611,54cyclictest1814209-21ssh09:15:281
180542699619,34cyclictest16850irq/134-800000010:59:360
1805426996136,22cyclictest16950irq/224-800000009:55:020
180542699613,5cyclictest1855855-21ssh10:48:560
180543199601,53cyclictest1857072-21sh10:51:241
180542699602,5cyclictest1879955-21ssh11:42:460
1805426996018,38cyclictest1806307-21cat07:25:020
1805435995946,6cyclictest1813806-21kworker/2:009:25:002
1805435995946,6cyclictest1813806-21kworker/2:009:25:002
180543199592,53cyclictest3467602-21systemd-journal08:00:401
180543199592,53cyclictest3467602-21systemd-journal08:00:401
180543199592,51cyclictest1841617-21rm10:16:531
180543199591,54cyclictest1872959-21ssh11:26:591
180543199591,54cyclictest1838204-21ssh10:09:331
180543199591,54cyclictest1816661-21ssh09:21:031
180543199591,54cyclictest1816661-21ssh09:21:031
180543199591,54cyclictest1811573-21sshd09:06:321
180543199591,52cyclictest1889114-21ssh12:03:271
1805431995823,31cyclictest1807200-21/usr/sbin/munin07:45:001
180543199581,53cyclictest1818723-21ssh09:25:271
1805431995815,38cyclictest1838965-21sshd10:10:461
1805431995815,38cyclictest1838965-21sshd10:10:461
180542699583,7cyclictest1809263-21sshd08:24:320
1805426995820,34cyclictest1866030-21rm11:11:270
1805435995747,6cyclictest1833981-21kworker/2:211:39:552
180543199572,50cyclictest1900110-21sh12:28:001
180543199571,51cyclictest1809986-21cat08:35:041
1805440995642,10cyclictest3467602-21systemd-journal11:29:333
180543199562,48cyclictest1808753-21unin-run08:15:011
1805431995620,32cyclictest1810072-21sh08:40:011
1805431995620,32cyclictest1810072-21sh08:40:011
180543199561,51cyclictest1903596-21ssh12:35:421
180543199561,51cyclictest1823343-21ssh09:35:421
180542699562,11cyclictest1806101-21cat07:20:030
180542699561,7cyclictest1808716-21sshd08:12:320
1805426995616,36cyclictest1809786-21sh08:35:010
180542699561,3cyclictest150-21pr/ttyS211:01:440
180542699561,3cyclictest150-21pr/ttyS211:01:440
1805435995544,6cyclictest1833981-21kworker/2:210:15:002
1805435995544,6cyclictest1833981-21kworker/2:210:15:002
180543199552,48cyclictest3467602-21systemd-journal07:12:121
180543199552,48cyclictest1806416-21munin-run07:30:001
1805426995528,21cyclictest48-21kcompactd010:25:090
180542699552,5cyclictest1807716-21munin-run07:55:010
180542699552,17cyclictest1809006-21munin-plugin-st08:20:010
180542699552,13cyclictest1811059-21cron09:00:000
180542699551,14cyclictest1897169-21ssh12:21:120
180543199542,47cyclictest48-21kcompactd007:55:021
180543199542,47cyclictest1811039-21modprobe08:55:371
1805426995414,36cyclictest150-21pr/ttyS207:48:330
1805435995344,6cyclictest1842919-21kworker/2:010:38:222
1805435995342,7cyclictest1833981-21kworker/2:211:16:142
180543199532,46cyclictest1809544-21munin-run08:30:011
180543199531,47cyclictest3467602-21systemd-journal08:10:011
180543199531,46cyclictest1810322-21true08:44:591
180542699532,6cyclictest1810567-21munin-run08:50:000
180542699532,16cyclictest1805623-21munin-plugin-st07:15:000
1805435995244,5cyclictest1793029-21kworker/2:309:59:262
1805435995241,6cyclictest1813806-21kworker/2:009:50:462
1805435995227,8cyclictest1833981-21kworker/2:211:41:592
180543199522,46cyclictest1807662-21latency_hist07:50:031
180543199521,47cyclictest0-21swapper/109:05:011
180542699521,9cyclictest1-21systemd07:45:010
1805440995129,15cyclictest275-21dbus-daemon08:45:013
1805435995143,5cyclictest1842919-21kworker/2:010:33:462
1805435995143,5cyclictest1842919-21kworker/2:010:33:462
1805435995143,5cyclictest1833981-21kworker/2:211:14:092
1805435995142,6cyclictest1833981-21kworker/2:210:20:012
1805435995141,6cyclictest1813806-21kworker/2:009:50:022
180543199512,44cyclictest1809339-21cat08:25:011
180543199511,46cyclictest1808987-21cron08:20:011
180543199511,44cyclictest48-21kcompactd008:52:451
180542699512,14cyclictest1810135-21latency_hist08:40:020
180542699512,14cyclictest1810135-21latency_hist08:40:010
180542699511,5cyclictest1811315-21sshd09:00:320
1805426995111,35cyclictest0-21swapper/008:51:320
1805435995043,4cyclictest1833981-21kworker/2:211:52:162
1805435995042,5cyclictest1833981-21kworker/2:211:23:192
1805435995042,5cyclictest1793029-21kworker/2:310:07:572
1805435995013,17cyclictest1887721-21ssh12:00:032
1805440994935,10cyclictest3467602-21systemd-journal12:05:003
1805435994941,5cyclictest1842919-21kworker/2:011:00:122
1805435994941,5cyclictest1842919-21kworker/2:011:00:122
1805435994941,5cyclictest1793029-21kworker/2:307:25:002
1805435994939,6cyclictest1793029-21kworker/2:310:22:592
1805435994930,14cyclictest3467602-21systemd-journal07:40:002
180543199495,39cyclictest1806135-21sshd07:21:321
180543199492,44cyclictest1806903-21sshd07:39:321
1805431994923,23cyclictest1807708-21systemd-run07:55:001
180543199491,43cyclictest1805926-21unin-run07:20:011
180542699492,5cyclictest1810329-21munin-run08:45:010
1805440994838,7cyclictest37-21ksoftirqd/310:31:473
1805440994838,7cyclictest37-21ksoftirqd/310:31:473
1805440994835,10cyclictest3467602-21systemd-journal10:20:473
1805440994834,11cyclictest1-21systemd10:50:003
1805440994832,12cyclictest275-21dbus-daemon10:55:013
1805440994830,11cyclictest275-21dbus-daemon09:20:013
1805440994828,15cyclictest1860782-21cron11:00:003
180543599484,4cyclictest131rcu_preempt10:51:232
1805435994840,5cyclictest1894212-21kworker/2:312:38:022
1805435994840,5cyclictest1842919-21kworker/2:010:58:232
1805435994840,5cyclictest1833981-21kworker/2:211:46:462
1805435994840,5cyclictest1833981-21kworker/2:211:46:462
1805435994840,5cyclictest1833981-21kworker/2:211:27:182
1805435994840,5cyclictest1793029-21kworker/2:310:03:562
1805435994839,6cyclictest1894212-21kworker/2:312:32:072
1805435994839,6cyclictest1813806-21kworker/2:009:40:262
1805435994839,6cyclictest1813806-21kworker/2:009:40:262
1805435994839,5cyclictest1793029-21kworker/2:307:20:022
1805435994838,7cyclictest1813806-21kworker/2:009:35:462
1805435994832,12cyclictest3467602-21systemd-journal12:25:022
1805435994826,15cyclictest275-21dbus-daemon12:20:002
1805440994736,6cyclictest309-21in:imuxsock10:10:003
1805440994732,10cyclictest275-21dbus-daemon10:40:003
1805435994740,5cyclictest1813806-21kworker/2:009:34:442
1805435994739,5cyclictest1842919-21kworker/2:011:06:332
1805435994739,5cyclictest1813806-21kworker/2:009:29:232
1805435994739,5cyclictest1793029-21kworker/2:309:10:022
1805435994738,6cyclictest1842919-21kworker/2:010:42:052
1805435994738,6cyclictest1842919-21kworker/2:010:42:052
1805435994724,16cyclictest275-21dbus-daemon07:14:592
1805426994726,17cyclictest12-21ksoftirqd/008:05:030
1805426994726,17cyclictest12-21ksoftirqd/008:05:020
180542699471,5cyclictest1806648-21sshd07:33:320
180542699471,5cyclictest1806648-21sshd07:33:320
180544099467,11cyclictest37-21ksoftirqd/309:36:323
180544099465,32cyclictest361rcuc/309:15:023
180544099463,39cyclictest1869723-21sh11:20:003
1805440994629,12cyclictest37-21ksoftirqd/308:05:023
1805440994629,12cyclictest37-21ksoftirqd/308:05:013
1805440994625,16cyclictest275-21dbus-daemon08:50:013
1805435994639,4cyclictest1793029-21kworker/2:308:00:562
1805435994639,4cyclictest1793029-21kworker/2:308:00:562
1805435994638,5cyclictest1833981-21kworker/2:212:10:022
180543599463,3cyclictest131rcu_preempt10:48:432
180542699462,10cyclictest3467602-21systemd-journal07:25:130
180544099453,5cyclictest131rcu_preempt12:11:073
1805440994530,12cyclictest275-21dbus-daemon12:35:003
1805440994530,11cyclictest37-21ksoftirqd/309:35:013
1805435994539,3cyclictest1833981-21kworker/2:211:33:012
1805435994536,5cyclictest1833981-21kworker/2:212:22:342
1805435994536,5cyclictest1793029-21kworker/2:308:55:022
180543199452,39cyclictest3467562-21sshd08:45:321
180544099444,30cyclictest3467602-21systemd-journal10:00:003
1805440994435,6cyclictest37-21ksoftirqd/310:20:003
1805440994431,10cyclictest37-21ksoftirqd/312:19:143
1805440994427,12cyclictest1806924-21cat07:40:003
180543599449,20cyclictest364-21snmpd10:30:022
1805435994436,5cyclictest1793029-21kworker/2:309:00:022
1805435994416,20cyclictest0-21swapper/212:13:242
180544099439,5cyclictest131rcu_preempt11:55:023
180544099435,5cyclictest131rcu_preempt10:41:383
180544099435,5cyclictest131rcu_preempt10:41:383
1805440994330,9cyclictest37-21ksoftirqd/307:25:003
1805440994330,10cyclictest37-21ksoftirqd/309:05:023
1805440994328,11cyclictest37-21ksoftirqd/310:30:003
1805440994324,14cyclictest275-21dbus-daemon12:10:013
180543199432,36cyclictest1805471-21grep07:10:021
180543199432,36cyclictest1805471-21grep07:10:011
180542699433,10cyclictest1805433-21latency_hist07:10:020
180542699433,10cyclictest1805433-21latency_hist07:10:010
1805440994230,8cyclictest1867479-21cron11:15:003
1805440994229,8cyclictest37-21ksoftirqd/307:19:593
1805440994229,10cyclictest37-21ksoftirqd/312:37:203
1805440994228,11cyclictest37-21ksoftirqd/308:18:323
1805440994226,12cyclictest37-21ksoftirqd/311:42:523
1805435994234,5cyclictest1793029-21kworker/2:308:46:182
1805435994225,12cyclictest275-21dbus-daemon07:30:002
1805440994132,7cyclictest37-21ksoftirqd/307:46:283
1805440994130,7cyclictest1829570-21latency_hist09:50:013
1805440994126,10cyclictest37-21ksoftirqd/312:23:593
1805440994125,12cyclictest37-21ksoftirqd/309:54:513
1805440994125,11cyclictest37-21ksoftirqd/311:09:383
1805440994123,13cyclictest37-21ksoftirqd/311:34:253
1805435994133,5cyclictest1833981-21kworker/2:212:04:312
1805435994133,5cyclictest1793029-21kworker/2:308:28:282
1805435994133,5cyclictest1793029-21kworker/2:308:15:012
1805435994132,5cyclictest1805839-21kworker/2:008:25:022
1805440994032,5cyclictest37-21ksoftirqd/307:43:053
1805440994030,8cyclictest37-21ksoftirqd/311:48:073
1805440994030,8cyclictest37-21ksoftirqd/311:48:073
1805440994028,8cyclictest37-21ksoftirqd/308:40:003
1805440994028,8cyclictest37-21ksoftirqd/308:40:003
1805440994027,9cyclictest37-21ksoftirqd/309:22:003
1805440994027,9cyclictest37-21ksoftirqd/309:22:003
1805440994022,12cyclictest275-21dbus-daemon08:15:013
1805440994021,14cyclictest275-21dbus-daemon09:30:013
1805435994032,5cyclictest1793029-21kworker/2:307:42:412
1805435994028,8cyclictest275-21dbus-daemon09:05:002
1805435994024,13cyclictest1806757-21sed07:35:022
1805435994024,13cyclictest1806757-21sed07:35:022
1805440993927,9cyclictest37-21ksoftirqd/310:15:023
1805440993927,9cyclictest37-21ksoftirqd/310:15:023
1805440993925,10cyclictest1827460-21latency_hist09:45:033
1805440993925,10cyclictest1827460-21latency_hist09:45:023
1805440993924,11cyclictest37-21ksoftirqd/310:03:373
180544099391,6cyclictest0-21swapper/311:20:403
1805435993929,4cyclictest309-21in:imuxsock08:10:002
1805435993919,15cyclictest3467602-21systemd-journal09:15:002
1805440993830,6cyclictest37-21ksoftirqd/309:08:073
1805440993826,9cyclictest37-21ksoftirqd/308:25:343
1805440993826,10cyclictest37-21ksoftirqd/311:35:303
1805440993825,9cyclictest1899232-21ssh12:25:503
1805435993823,11cyclictest30-21ksoftirqd/208:20:032
1805440993724,10cyclictest37-21ksoftirqd/307:35:023
1805440993724,10cyclictest37-21ksoftirqd/307:35:013
1805435993724,10cyclictest30-21ksoftirqd/208:35:002
1805435993722,11cyclictest309-21in:imuxsock08:44:322
180544099366,5cyclictest131rcu_preempt11:53:043
1805440993627,6cyclictest37-21ksoftirqd/307:53:113
1805440993618,16cyclictest361rcuc/309:00:023
180544099361,31cyclictest37-21ksoftirqd/307:56:323
180544099361,30cyclictest48-21kcompactd008:10:023
1805440993522,9cyclictest37-21ksoftirqd/307:15:013
1805440993513,16cyclictest275-21dbus-daemon08:55:013
1805435993525,7cyclictest275-21dbus-daemon08:00:012
1805440993424,7cyclictest37-21ksoftirqd/307:10:023
1805440993424,7cyclictest37-21ksoftirqd/307:10:013
1805440993410,5cyclictest131rcu_preempt08:25:003
180543599342,6cyclictest131rcu_preempt07:55:002
1805435993419,10cyclictest3466785-21systemd-udevd07:48:002
1805440993219,10cyclictest37-21ksoftirqd/308:35:003
1805435993123,5cyclictest1810113-21latency_hist08:40:022
1805435993123,5cyclictest1810113-21latency_hist08:40:022
1805435992819,5cyclictest1797774-21kworker/2:107:10:032
1805435992819,5cyclictest1797774-21kworker/2:107:10:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional