You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-27 - 21:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed May 27, 2026 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
181982422540,8sleep03056-21PK-Backend07:05:140
190396721300,4sleep13056-21PK-Backend11:00:181
187784421300,3sleep23056-21PK-Backend09:45:202
18201082790,9sleep20-21swapper/207:05:192
139099750,62rtkit-daemon0-21swapper/107:07:141
182089027439,10sleep30-21swapper/307:09:533
18293502520,4sleep23056-21PK-Backend07:30:152
19013312490,4sleep13056-21PK-Backend10:55:001
182107099360,17cyclictest1883750-21cat10:05:023
182107099352,18cyclictest0-21swapper/311:30:033
182107099321,18cyclictest1909674-21runrttasks11:17:523
182107099320,19cyclictest1823291-21sed07:15:033
182107099320,19cyclictest1823291-21sed07:15:033
182107099320,17cyclictest1908009-21taskset11:13:463
182107099310,17cyclictest1888703-21taskset10:19:173
139099310,4rtkit-daemon1389-21rtkit-daemon09:02:132
182107099301,16cyclictest42-21ksoftirqd/309:28:013
182107099300,18cyclictest0-21swapper/311:56:543
182107099300,18cyclictest0-21swapper/309:02:113
182107099300,17cyclictest1873118-21taskset09:34:313
139099300,4rtkit-daemon1389-21rtkit-daemon12:13:022
139099300,4rtkit-daemon1389-21rtkit-daemon08:25:392
182107099290,27cyclictest1932117-21needreboot12:20:183
182107099290,16cyclictest0-21swapper/309:56:443
182107099270,25cyclictest1938570-21cat12:40:003
182107099261,23cyclictest1848935-21df08:25:143
1821070992611,14cyclictest1859342-21df08:55:143
1821070992610,14cyclictest1897036-21cat10:40:203
182107099260,25cyclictest1925626-21seq12:00:363
182107099260,24cyclictest1885975-21hddtemp_smartct10:10:133
182107099260,17cyclictest1916800-21seq11:36:583
19014622250,3sleep0161rcuc/010:55:120
182107099259,14cyclictest1904256-21cat11:00:203
182107099259,14cyclictest1843151-21cut08:10:023
182107099252,21cyclictest1905567-21needreboot11:05:143
182107099251,22cyclictest1856364-21grep08:45:193
182107099251,20cyclictest1836183-21apt07:50:113
1821070992510,13cyclictest1894020-21ls10:35:013
182107099250,23cyclictest1885375-21cut10:10:013
182107099250,23cyclictest1844112-21ntp_states08:10:193
182107099250,23cyclictest1821413-21tr07:10:123
182107099250,15cyclictest3056-21PK-Backend12:29:583
182106599250,23cyclictest1389-21rtkit-daemon10:50:092
182107099249,14cyclictest1895241-21sed10:35:213
182107099249,13cyclictest1880092-21cron09:55:013
182107099246,16cyclictest1826368-21wc07:20:193
182107099245,17cyclictest1900302-21missed_timers10:50:163
182107099241,20cyclictest1862771-21apt09:05:103
182107099240,3cyclictest1895000-21snapd09:39:133
182107099240,22cyclictest1930076-21seq12:15:113
182107099240,22cyclictest1889782-21ntp_states10:20:193
182107099240,1cyclictest0-21swapper/308:35:233
182107099236,15cyclictest1936397-21grep12:34:593
182107099234,17cyclictest1868143-21tr09:20:123
182107099232,20cyclictest1846568-21cron08:20:003
182107099232,18cyclictest0-21swapper/308:40:113
1821070992312,9cyclictest1866209-21/usr/sbin/munin09:15:143
182107099231,19cyclictest1834189-21apt07:45:123
182107099231,19cyclictest0-21swapper/308:30:123
182107099231,18cyclictest3056-21PK-Backend07:40:113
182107099230,21cyclictest1926228-21/usr/sbin/munin12:05:113
182107099230,21cyclictest1898335-21missed_timers10:45:153
182107099230,11cyclictest1891137-21df10:25:133
182106599230,21cyclictest1868334-21df09:20:122
182106299230,20cyclictest1875174-21apt09:40:101
1821058992321,1cyclictest0-21swapper/008:51:160
182107099229,11cyclictest1919845-21ps11:45:193
182107099229,11cyclictest1901907-21if_err_enp2s010:55:163
182107099228,12cyclictest42-21ksoftirqd/307:55:143
182107099222,16cyclictest1910353-21apt-get11:20:043
1821070992220,1cyclictest0-21swapper/308:50:173
182107099221,19cyclictest1865577-21users09:10:223
182107099221,19cyclictest1829310-21/usr/sbin/munin07:30:153
182107099221,18cyclictest1927985-21apt12:10:103
182107099221,18cyclictest1846839-21apt08:20:103
182107099221,18cyclictest1846839-21apt08:20:103
1821070992210,10cyclictest1921396-21missed_timers11:50:163
182107099220,2cyclictest1912852-21ntpq11:25:203
1821065992217,3cyclictest1867708-21turbostat.cron09:19:592
1821062992219,2cyclictest3056-21PK-Backend09:28:271
182106299220,21cyclictest3056-21PK-Backend07:59:391
1821058992220,1cyclictest3056-21PK-Backend10:05:230
1821058992220,1cyclictest0-21swapper/012:34:140
1821058992220,1cyclictest0-21swapper/011:58:590
1821058992220,1cyclictest0-21swapper/010:04:010
1821058992220,1cyclictest0-21swapper/007:34:100
1821058992219,2cyclictest3056-21PK-Backend10:45:040
182105899220,21cyclictest0-21swapper/011:13:510
182105899220,21cyclictest0-21swapper/010:44:170
182107099211,17cyclictest0-21swapper/311:40:103
182107099210,19cyclictest1840493-21cat08:00:173
182106599212,17cyclictest1095-21runrttasks08:38:592
1821065992114,5cyclictest1858740-21systemctl08:55:002
182106599210,2cyclictest1887074-21unin-run10:15:012
182106599210,20cyclictest0-21swapper/211:44:102
182106599210,20cyclictest0-21swapper/207:53:202
182106599210,19cyclictest3056-21PK-Backend09:53:032
182106599210,19cyclictest0-21swapper/210:35:022
1821062992119,1cyclictest3056-21PK-Backend08:57:241
1821062992118,2cyclictest3056-21PK-Backend11:50:221
1821062992118,2cyclictest3056-21PK-Backend10:28:501
1821062992118,1cyclictest3056-21PK-Backend11:45:151
1821062992110,9cyclictest3056-21PK-Backend09:20:001
182106299210,20cyclictest3056-21PK-Backend08:35:001
182106299210,19cyclictest3056-21PK-Backend09:24:151
182106299210,19cyclictest3056-21PK-Backend08:41:081
1821058992119,1cyclictest0-21swapper/012:07:310
1821058992119,1cyclictest0-21swapper/010:38:350
1821058992119,1cyclictest0-21swapper/010:28:360
1821058992119,1cyclictest0-21swapper/010:13:550
1821058992119,1cyclictest0-21swapper/009:27:050
1821058992119,1cyclictest0-21swapper/008:32:500
1821058992119,1cyclictest0-21swapper/008:11:550
1821058992119,1cyclictest0-21swapper/008:04:240
182105899210,2cyclictest1860780-21munin-run09:00:010
182105899210,20cyclictest3056-21PK-Backend07:11:110
182105899210,20cyclictest0-21swapper/012:18:130
182105899210,20cyclictest0-21swapper/008:36:410
182107099209,10cyclictest3056-21PK-Backend09:44:373
182107099203,16cyclictest0-21swapper/307:27:553
182107099200,18cyclictest1877311-21idleruntime09:45:143
182107099200,18cyclictest1830885-21expr07:35:133
182106599203,16cyclictest1095-21runrttasks12:36:382
1821065992018,1cyclictest0-21swapper/212:20:192
1821065992018,1cyclictest0-21swapper/208:42:552
1821065992018,1cyclictest0-21swapper/208:17:332
1821065992013,5cyclictest1906402-21/usr/sbin/munin11:10:012
182106599200,1cyclictest0-21swapper/210:30:432
182106599200,1cyclictest0-21swapper/209:32:572
182106599200,1cyclictest0-21swapper/208:06:302
182106599200,19cyclictest0-21swapper/211:58:302
182106599200,19cyclictest0-21swapper/211:49:442
182106599200,19cyclictest0-21swapper/211:28:492
182106599200,19cyclictest0-21swapper/210:57:532
182106599200,19cyclictest0-21swapper/210:09:002
182106599200,19cyclictest0-21swapper/209:56:212
182106599200,19cyclictest0-21swapper/209:29:042
182106599200,19cyclictest0-21swapper/208:11:042
182106599200,19cyclictest0-21swapper/207:57:382
1821062992018,1cyclictest3056-21PK-Backend10:56:411
1821062992018,1cyclictest3056-21PK-Backend10:46:551
1821062992018,1cyclictest3056-21PK-Backend10:36:131
1821062992018,1cyclictest3056-21PK-Backend09:07:381
1821062992017,2cyclictest3056-21PK-Backend11:38:331
1821062992013,5cyclictest1856966-21cron08:50:011
1821062992013,5cyclictest1828051-21sensors_temp07:25:221
182106299200,1cyclictest0-21swapper/107:10:101
182106299200,19cyclictest3056-21PK-Backend12:01:581
182106299200,19cyclictest0-21swapper/112:32:261
182106299200,19cyclictest0-21swapper/109:51:111
182106299200,19cyclictest0-21swapper/108:15:401
182106299200,19cyclictest0-21swapper/107:35:041
182106299200,19cyclictest0-21swapper/107:20:181
182106299200,19cyclictest0-21swapper/107:19:301
182106299200,19cyclictest0-21swapper/107:19:291
182106299200,18cyclictest3056-21PK-Backend12:18:281
182106299200,18cyclictest3056-21PK-Backend07:47:351
1821058992018,1cyclictest0-21swapper/012:35:500
1821058992018,1cyclictest0-21swapper/009:51:390
1821058992018,1cyclictest0-21swapper/009:39:140
1821058992018,1cyclictest0-21swapper/009:30:560
1821058992018,1cyclictest0-21swapper/009:11:290
1821058992018,1cyclictest0-21swapper/007:53:350
1821058992018,1cyclictest0-21swapper/007:22:190
1821058992012,6cyclictest1892954-21cat10:30:130
182105899200,19cyclictest0-21swapper/012:25:090
182105899200,19cyclictest0-21swapper/012:23:340
182105899200,19cyclictest0-21swapper/010:24:170
182105899200,19cyclictest0-21swapper/009:17:370
182105899200,19cyclictest0-21swapper/008:48:310
182105899200,19cyclictest0-21swapper/008:29:320
182105899200,19cyclictest0-21swapper/008:20:390
182105899200,19cyclictest0-21swapper/008:20:390
182105899200,19cyclictest0-21swapper/007:55:450
182105899200,19cyclictest0-21swapper/007:43:490
182105899200,19cyclictest0-21swapper/007:15:160
182105899200,19cyclictest0-21swapper/007:15:160
19179022190,10sleep126-21ksoftirqd/111:40:171
1821065991917,1cyclictest3056-21PK-Backend09:40:062
1821065991917,1cyclictest3056-21PK-Backend08:47:342
1821065991917,1cyclictest0-21swapper/209:13:162
1821065991917,1cyclictest0-21swapper/208:31:042
1821065991912,5cyclictest34-21ksoftirqd/208:20:132
1821065991912,5cyclictest34-21ksoftirqd/208:20:122
1821065991912,5cyclictest1933897-21ntpq12:25:172
1821065991911,6cyclictest1873437-21awk09:35:102
182106599190,1cyclictest0-21swapper/212:08:372
182106599190,1cyclictest0-21swapper/212:00:532
182106599190,1cyclictest0-21swapper/211:54:302
182106599190,1cyclictest0-21swapper/210:45:492
182106599190,1cyclictest0-21swapper/210:41:302
182106599190,1cyclictest0-21swapper/210:29:542
182106599190,1cyclictest0-21swapper/210:01:022
182106599190,1cyclictest0-21swapper/209:07:282
1821062991917,1cyclictest3056-21PK-Backend12:08:281
1821062991917,1cyclictest3056-21PK-Backend09:32:391
1821062991917,1cyclictest0-21swapper/112:28:411
1821062991917,1cyclictest0-21swapper/109:35:431
182106299190,1cyclictest3056-21PK-Backend11:59:001
182106299190,1cyclictest3056-21PK-Backend10:24:441
182106299190,1cyclictest3056-21PK-Backend10:14:511
182106299190,1cyclictest0-21swapper/111:20:481
182106299190,1cyclictest0-21swapper/109:47:541
182106299190,1cyclictest0-21swapper/107:33:301
182105899195,3cyclictest18778432threads09:45:210
1821058991917,1cyclictest0-21swapper/011:50:290
1821058991917,1cyclictest0-21swapper/011:47:170
1821058991917,1cyclictest0-21swapper/011:44:530
1821058991917,1cyclictest0-21swapper/009:22:390
1821058991917,1cyclictest0-21swapper/009:07:370
1821058991917,1cyclictest0-21swapper/007:36:330
1821058991913,4cyclictest1875365-21expr09:40:110
182105899190,1cyclictest0-21swapper/010:17:130
182105899190,1cyclictest0-21swapper/009:02:450
182105899190,1cyclictest0-21swapper/008:18:430
182105899190,1cyclictest0-21swapper/008:07:220
182105899190,1cyclictest0-21swapper/007:26:390
182105899190,18cyclictest3056-21PK-Backend12:00:350
182106599187,9cyclictest1909860-21perf11:19:592
1821065991816,1cyclictest3056-21PK-Backend12:30:272
1821065991816,1cyclictest0-21swapper/207:14:472
1821065991812,4cyclictest2156-21gnome-shell11:35:092
1821065991812,4cyclictest1889819-21pluginstate10:20:192
1821065991812,4cyclictest1095-21runrttasks11:34:322
1821065991811,5cyclictest28650irq/123-ahci[0008:00:202
1821065991810,6cyclictest1930458-21ntpq12:15:172
182106599180,1cyclictest0-21swapper/207:46:172
1821062991816,1cyclictest3056-21PK-Backend08:38:041
1821062991816,1cyclictest3056-21PK-Backend08:07:151
1821062991815,2cyclictest3056-21PK-Backend07:52:151
1821062991811,5cyclictest1913304-21seq11:26:351
1821062991811,5cyclictest1832592-21expr07:40:111
1821062991810,6cyclictest1865967-21cron09:15:001
182106299180,1cyclictest3056-21PK-Backend12:10:371
182106299180,1cyclictest3056-21PK-Backend11:08:451
182106299180,1cyclictest3056-21PK-Backend10:40:541
182106299180,16cyclictest3056-21PK-Backend10:30:111
1821058991811,5cyclictest1900423-21ntp_kernel_pll_10:50:170
182106599179,6cyclictest1832449-21apt07:40:122
1821065991715,1cyclictest3056-21PK-Backend07:23:592
1821065991715,1cyclictest0-21swapper/211:24:022
1821065991715,1cyclictest0-21swapper/210:19:062
1821065991711,4cyclictest34-21ksoftirqd/207:15:162
1821065991711,4cyclictest34-21ksoftirqd/207:15:162
1821062991715,1cyclictest3056-21PK-Backend11:17:041
1821062991715,1cyclictest3056-21PK-Backend11:13:181
1821062991715,1cyclictest3056-21PK-Backend10:03:081
1821062991715,1cyclictest3056-21PK-Backend08:24:401
1821062991715,1cyclictest3056-21PK-Backend08:24:401
1821062991715,1cyclictest0-21swapper/108:00:271
1821062991711,5cyclictest1880858-21cut09:55:161
1821062991711,4cyclictest1858272-21cat08:50:221
1821058991715,1cyclictest0-21swapper/011:05:050
1821058991712,3cyclictest1834830-21needreboot07:45:160
1821058991711,4cyclictest1928827-21cut12:10:190
1821058991710,5cyclictest1908513-21grep11:15:120
1821065991614,1cyclictest3056-21PK-Backend07:26:562
1821065991612,3cyclictest1859560-21missed_timers08:55:182
1821065991611,4cyclictest1903590-21hddtemp_smartct11:00:142
1821065991610,4cyclictest1906806-21grep11:10:112
1821062991614,1cyclictest3056-21PK-Backend11:33:251
1821062991614,1cyclictest3056-21PK-Backend09:03:121
1821062991614,1cyclictest3056-21PK-Backend08:28:521
182106299160,2cyclictest141rcu_preempt10:05:111
182105899168,6cyclictest1913574-21cron11:29:590
182105899166,9cyclictest1910538-21cut11:20:130
1821058991610,4cyclictest1853903-21sort08:40:110
19323682150,4sleep13056-21PK-Backend12:20:221
1821065991513,1cyclictest0-21swapper/207:35:142
182106299158,5cyclictest1888936-21grep10:20:011
1821062991512,2cyclictest3056-21PK-Backend08:10:471
182105899159,4cyclictest1904043-21grep11:00:180
1821058991513,1cyclictest0-21swapper/009:58:480
182105899149,4cyclictest1914840-21sed11:30:200
182105899148,4cyclictest1915577-21sort11:35:100
1821062991311,1cyclictest3056-21PK-Backend12:37:331
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional