You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-13 - 04:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Sat Dec 13, 2025 00:44:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
192299621910,3sleep3575-21kworker/3:2-events23:19:003
17132092710,6sleep30-21swapper/319:54:073
17880052640,3sleep1281rcuc/121:04:101
17613982640,5sleep30-21swapper/320:39:093
18404132610,3sleep239-21ksoftirqd/221:54:182
19103902550,4chrt575-21kworker/3:2-events23:04:083
19313662520,2chrt471rcuc/323:24:183
1669041995246,4cyclictest1716829-21kworker/u8:1+flush-8:021:09:181
1669041995243,7cyclictest278-21systemd-journal23:10:011
1669038995039,9cyclictest278-21systemd-journal20:45:000
1669041994842,5cyclictest1876298-21kworker/u8:2+events_unbound23:54:131
1669038994636,7cyclictest2001084-21latency_hist00:35:010
1669041994539,4cyclictest1876298-21kworker/u8:2+flush-8:022:49:131
1669041994538,5cyclictest1976259-21kworker/u8:0+flush-8:000:19:131
1669041994436,5cyclictest1833491-21kworker/u8:2+flush-8:022:09:121
1669041994337,4cyclictest1876298-21kworker/u8:2+flush-8:022:54:141
1669041994237,4cyclictest1779148-21kworker/u8:0+flush-8:022:29:041
1669041994135,5cyclictest1937178-21kworker/u8:0+events_unbound23:58:581
1669038994133,5cyclictest1986113-21latency_hist00:20:000
1669041994035,3cyclictest1691316-21kworker/u8:3+events_unbound20:19:191
1669041994034,5cyclictest1912394-21kworker/u8:1+events_unbound23:19:081
1669041994034,5cyclictest1691316-21kworker/u8:3+events_unbound20:49:011
1669041994034,4cyclictest1891844-21kworker/u8:0+flush-8:023:14:101
1669041994034,4cyclictest1779148-21kworker/u8:0+events_unbound22:15:001
1669041994030,7cyclictest1941759-21latency_hist23:35:011
166903899401,36cyclictest1-21systemd23:15:000
1669043993927,10cyclictest1676701-21aten2_r5power_v19:19:022
1669041993934,3cyclictest1951848-21kworker/u8:3+events_unbound00:29:021
1669041993933,4cyclictest1716829-21kworker/u8:1+events_unbound20:09:171
1669041993932,6cyclictest1633631-21kworker/u8:2+events_unbound19:15:041
166903899392,31cyclictest1731906-21kworker/0:0+events21:20:010
1669043993822,14cyclictest1773809-21latency_hist20:50:002
1669041993834,3cyclictest1716829-21kworker/u8:1+flush-8:020:44:091
1669041993832,4cyclictest1876298-21kworker/u8:2+events_unbound22:44:141
1669041993827,5cyclictest1937178-21kworker/u8:0+events_unbound23:49:031
1669043993724,11cyclictest1789832-21aten2_r5power_v21:09:032
1669043993723,12cyclictest1707279-21aten2_r5power_v19:49:022
1669041993732,4cyclictest1716829-21kworker/u8:1+flush-8:022:34:161
1669041993732,4cyclictest1716829-21kworker/u8:1+flush-8:020:39:071
1669041993731,4cyclictest1716829-21kworker/u8:1+events_unbound20:04:201
1669041993631,3cyclictest1658188-21kworker/u8:1+flush-8:019:24:151
1669041993631,3cyclictest1658188-21kworker/u8:1+events_unbound19:29:061
1669041993630,4cyclictest1966392-21kworker/u8:1+flush-8:000:09:021
1669041993630,4cyclictest1716829-21kworker/u8:1+flush-8:021:49:081
1669041993630,4cyclictest1691316-21kworker/u8:3+flush-8:020:54:101
1669041993629,6cyclictest1691316-21kworker/u8:3+flush-8:019:34:151
1669041993629,5cyclictest1716829-21kworker/u8:1+events_unbound22:24:041
166904199360,34cyclictest1-21systemd21:20:001
1669038993631,4cyclictest1691316-21kworker/u8:3+events_unbound20:49:080
1669038993628,5cyclictest1955882-21ntpq23:49:080
1669038993626,7cyclictest278-21systemd-journal23:26:000
166903899361,31cyclictest1897413-21kworker/0:2+events22:56:510
1669043993523,10cyclictest1681988-21fschecks_time19:24:032
1669041993530,4cyclictest1794252-21kworker/u8:3+events_unbound21:34:131
1669041993529,5cyclictest1691316-21kworker/u8:3+flush-8:019:44:041
1669041993529,4cyclictest1716829-21kworker/u8:1+events_unbound21:59:081
1669041993528,5cyclictest1912394-21kworker/u8:1+events_unbound23:45:011
1669038993525,8cyclictest1892694-21diskstats22:48:590
1669038993525,7cyclictest1981176-21latency_hist00:15:000
1669038993525,7cyclictest1981176-21latency_hist00:15:000
1669043993417,15cyclictest1763073-21unixbench_singl20:39:192
166904399340,10cyclictest1830237-21unixbench_multi21:44:202
1669041993429,4cyclictest1951848-21kworker/u8:3+flush-8:000:04:031
1669041993429,3cyclictest1716829-21kworker/u8:1+flush-8:022:39:101
1669041993428,4cyclictest1716829-21kworker/u8:1+events_unbound22:03:591
1669041993422,3cyclictest1951848-21kworker/u8:3+flush-8:000:24:141
166903899346,23cyclictest1941807-21kworker/0:2+events00:25:030
166903899342,26cyclictest1731906-21kworker/0:0+events20:50:010
1669041993327,5cyclictest1779148-21kworker/u8:0+events_unbound21:50:371
166903899332,30cyclictest1731906-21kworker/0:0+events22:35:250
1669038993322,8cyclictest1835478-21latency_hist21:50:010
166903899332,27cyclictest1701542-21kworker/0:1+events19:50:010
1669038993321,10cyclictest278-21systemd-journal20:15:000
166903899331,31cyclictest1731906-21kworker/0:0+events22:42:050
1669043993225,5cyclictest1794252-21kworker/u8:3+flush-8:021:29:042
1669041993228,3cyclictest1927189-21kworker/u8:3+events_unbound23:24:091
1669041993226,5cyclictest1716829-21kworker/u8:1+events_unbound21:44:131
1669041993218,11cyclictest1721954-21apt-get20:00:001
1669041993218,11cyclictest1721954-21apt-get20:00:001
1669038993226,4cyclictest1976074-21grep00:09:160
1669038993225,5cyclictest1658188-21kworker/u8:1+flush-8:019:23:230
1669038993223,7cyclictest1804435-21latency_hist21:20:020
166903899322,28cyclictest1941807-21kworker/0:2+events23:39:110
166903899321,29cyclictest1923000-21sshd23:19:010
166904399312,11cyclictest1960805-21ntpq23:54:072
166904399312,10cyclictest1860278-21ntpq22:14:072
1669043993117,12cyclictest1697341-21fschecks_time19:39:042
1669041993126,4cyclictest1937178-21kworker/u8:0+events_unbound23:39:151
1669041993126,3cyclictest1633631-21kworker/u8:2+flush-8:019:14:131
1669041993124,5cyclictest1976259-21kworker/u8:0+flush-8:000:14:161
1669041993124,5cyclictest1976259-21kworker/u8:0+flush-8:000:14:151
1669041993124,5cyclictest1706628-21kworker/u8:0+events_unbound20:29:141
1669041993124,5cyclictest1691316-21kworker/u8:3+flush-8:019:49:161
1669038993123,6cyclictest1758102-21turbostat.cron20:35:010
1669038993123,6cyclictest1673900-21grep19:14:160
166903899312,28cyclictest1731906-21kworker/0:0+events22:08:290
1669038993121,8cyclictest278-21systemd-journal21:15:000
1669038993110,20cyclictest1897413-21kworker/0:2+events23:05:150
1669043993017,12cyclictest1981009-21unixbench_multi00:14:152
1669043993017,12cyclictest1981009-21unixbench_multi00:14:152
1669043993016,12cyclictest1922663-21aten2_r5power_p23:18:572
1669041993024,5cyclictest1716829-21kworker/u8:1+flush-8:021:24:181
1669041993024,4cyclictest1794252-21kworker/u8:3+events_unbound21:25:391
1669041993022,6cyclictest1885950-21kworker/u8:3+flush-8:023:04:191
1669038993027,2cyclictest1721936-21cron20:00:000
1669038993027,2cyclictest1721936-21cron20:00:000
166903899302,26cyclictest1917156-21kworker/0:1+events23:34:150
166904399293,21cyclictest1995363-21ntpq00:29:092
166904399293,21cyclictest1763999-21apt-get20:44:012
1669043992914,13cyclictest0-21swapper/200:40:002
166904399291,11cyclictest1747077-21ntpq20:24:132
1669043992911,16cyclictest0-21swapper/219:34:122
1669043992910,16cyclictest1945811-21tr23:39:132
1669041992925,3cyclictest1976259-21kworker/u8:0+events_unbound00:34:141
1669041992925,3cyclictest1691316-21kworker/u8:3+flush-8:019:39:171
1669041992923,4cyclictest1784026-21sendmail21:00:001
166903899299,18cyclictest1897413-21kworker/0:2+events23:04:070
166903899297,16cyclictest1731906-21kworker/0:0+events21:55:050
166903899292,25cyclictest1731906-21kworker/0:0+events20:21:330
1669038992922,5cyclictest1700869-21ntpq19:39:120
166903899292,22cyclictest1731906-21kworker/0:0+events22:01:050
79589797284,7rudpcli0-21swapper/319:09:033
79589797284,7rudpcli0-21swapper/319:09:033
18302422280,3sleep30-21swapper/321:44:203
166904399283,5cyclictest1996738-21apt-get00:33:572
166904399282,7cyclictest1855236-21ntpq22:09:112
166904399282,21cyclictest1880764-21ntpq22:34:142
166904399282,21cyclictest1865512-21ntpq22:19:102
166904399282,21cyclictest1819533-21ntpq21:34:112
166904399282,21cyclictest1741963-21ntpq20:19:272
166904399281,9cyclictest1962315-21/usr/sbin/munin23:59:002
166904399281,7cyclictest1901361-21/usr/sbin/munin22:54:062
166904399281,22cyclictest1794784-21apt-get21:14:012
1669041992824,3cyclictest1976259-21kworker/u8:0+flush-8:000:35:361
1669041992824,3cyclictest1691316-21kworker/u8:3+events_unbound19:54:091
1669038992818,7cyclictest1820194-21latency_hist21:35:020
16688582285,18sleep00-21swapper/019:10:010
16688582285,18sleep00-21swapper/019:10:000
18426597272,5rudpcli0-21swapper/321:24:073
166904399279,15cyclictest1928410-21cut23:24:102
166904399278,17cyclictest1810017-21awk21:25:022
166904399277,18cyclictest1954189-21cut23:49:042
166904399277,17cyclictest1973312-21cut00:09:052
166904399277,17cyclictest1820680-21seq21:38:272
166904399276,7cyclictest1948022-21sed23:44:082
166904399273,2cyclictest17546222cut20:34:072
166904399272,9cyclictest1702797-21ps19:44:062
166904399272,6cyclictest1845052-21ntpq21:59:092
166904399272,4cyclictest1981732-21apt-get00:18:562
166904399272,3cyclictest1774380-21apt-get20:53:592
166904399272,21cyclictest1731716-21ntpq20:09:252
166904399272,20cyclictest1726627-21ntpq20:04:262
1669043992714,11cyclictest0-21swapper/222:40:002
1669041992723,3cyclictest1716829-21kworker/u8:1+events_unbound21:38:591
166903899276,20cyclictest1731906-21kworker/0:0+events22:23:450
166903899275,21cyclictest1941807-21kworker/0:2+events23:50:390
1669038992722,3cyclictest1711745-21systemd-run19:50:000
166903899272,21cyclictest1941807-21kworker/0:2+events00:00:150
1669038992717,8cyclictest1839940-21ntpq21:54:110
17546192260,6sleep30-21swapper/320:34:073
166904399266,5cyclictest1890138-21cut22:44:052
166904399266,18cyclictest1688030-21cut19:29:072
166904399262,4cyclictest1941097-21ntpq23:34:082
166904399261,8cyclictest1971215-21grep00:04:152
166904399261,7cyclictest1717372-21/usr/sbin/munin19:59:002
166904399261,7cyclictest1717372-21/usr/sbin/munin19:59:002
1669043992612,12cyclictest0-21swapper/222:03:582
1669043992610,14cyclictest1779790-21fschecks_time20:59:042
1669043992610,14cyclictest0-21swapper/223:14:022
166904399260,8cyclictest1804318-21grep21:19:182
1669041992622,3cyclictest1716829-21kworker/u8:1+flush-8:022:19:101
1669041992621,4cyclictest1927189-21kworker/u8:3+flush-8:023:29:111
1669041992621,3cyclictest1716829-21kworker/u8:1+events_unbound20:24:001
166903899268,17cyclictest1731906-21kworker/0:0+events21:27:370
166903899268,17cyclictest1701542-21kworker/0:1+events20:08:210
166903899268,17cyclictest1701542-21kworker/0:1+events19:41:490
166903899267,18cyclictest1731906-21kworker/0:0+events20:35:010
166903899265,20cyclictest1917156-21kworker/0:1+events23:23:390
1669038992620,4cyclictest1876158-21unixbench_singl22:29:150
1669038992618,6cyclictest1691830-21/usr/sbin/munin19:34:120
1669038992614,2cyclictest0-21swapper/022:19:120
184265972518,4rudpcli0-21swapper/320:21:593
166904399255,5cyclictest1913612-21cut23:09:052
1669043992522,2cyclictest1733194-21fschecks_count20:14:192
166904399251,8cyclictest1875047-21cut22:29:062
166904399251,7cyclictest1892503-21cut22:48:572
166904399251,11cyclictest1671749-21kthreadcore19:14:072
166904399250,9cyclictest1870595-21ntpq22:24:152
166904399250,8cyclictest1907173-21/usr/sbin/munin23:01:302
166904399250,8cyclictest1712868-21tr19:54:062
166904399250,7cyclictest1752788-21rm20:30:002
166904399250,6cyclictest1834892-21awk21:49:142
166904399250,23cyclictest1988177-21cut00:24:062
1669041992520,2cyclictest0-21swapper/121:14:001
166903899256,18cyclictest1891932-21kworker/0:1+events22:54:270
166903899252,22cyclictest1941807-21kworker/0:2+events23:56:470
166903899252,22cyclictest1941807-21kworker/0:2+events23:40:510
166903899252,22cyclictest1941807-21kworker/0:2+events00:24:590
1669038992520,3cyclictest1753442-21kworker/u8:2+events_unbound21:00:000
166903899250,6cyclictest18566832grep22:13:590
79589797240,4rudpcli0-21swapper/319:48:533
18426597240,3rudpcli0-21swapper/319:24:423
166904399242,17cyclictest1936196-21ntpq23:29:142
166904399241,7cyclictest1785294-21cut21:04:072
166903899248,15cyclictest1941807-21kworker/0:2+events00:35:320
166903899247,16cyclictest1731906-21kworker/0:0+events21:02:020
166903899247,16cyclictest1701542-21kworker/0:1+events20:02:330
166903899246,17cyclictest1731906-21kworker/0:0+events20:15:410
166903899245,18cyclictest1731906-21kworker/0:0+events21:41:090
166903899244,14cyclictest1663897-21kworker/0:0+events19:25:010
166903899242,21cyclictest0-21swapper/021:09:180
166903899241,22cyclictest1663897-21kworker/0:0+events19:17:420
166903899240,23cyclictest1731906-21kworker/0:0+events22:32:050
79589797237,5rudpcli0-21swapper/321:30:013
21730497232,8rudpcli0-21swapper/321:14:003
166904399234,4cyclictest0-21swapper/222:59:192
1669041992319,3cyclictest0-21swapper/123:00:001
1669041992317,4cyclictest1716829-21kworker/u8:1+flush-8:020:34:091
1669041992315,5cyclictest0-21swapper/120:11:341
795897972214,5rudpcli0-21swapper/322:09:103
79589797220,4rudpcli0-21swapper/320:46:343
79589797220,3rudpcli0-21swapper/320:12:473
166904399220,7cyclictest1669092-21sort19:10:012
166904399220,7cyclictest1669092-21sort19:10:002
166903899224,17cyclictest1731906-21kworker/0:0+events21:39:570
166903899222,19cyclictest1731906-21kworker/0:0+events20:26:170
795897972115,4rudpcli0-21swapper/319:26:363
79589797210,7rudpcli0-21swapper/322:04:073
21730497210,9rudpcli1691654-21chrt19:31:443
21730497210,3rudpcli0-21swapper/319:13:113
18426597210,4rudpcli0-21swapper/320:40:023
18426597203,7rudpcli0-21swapper/321:40:003
18426597202,5rudpcli0-21swapper/321:09:033
18426597201,6rudpcli0-21swapper/319:19:103
79589797190,11rudpcli1861182-21chrt22:15:133
21730497190,8rudpcli1701761-21chrt19:40:163
184265971910,5rudpcli0-21swapper/320:59:193
18426597190,9rudpcli1856140-21chrt22:10:533
79589797180,7rudpcli1721371-21chrt19:59:143
79589797180,7rudpcli1721371-21chrt19:59:133
21730497180,8rudpcli1871145-21chrt22:24:363
18426597182,7rudpcli0-21swapper/320:09:253
18426597182,5rudpcli0-21swapper/320:04:233
18426597181,7rudpcli0-21swapper/319:39:163
18426597180,7rudpcli0-21swapper/322:54:063
79589797173,5rudpcli0-21swapper/321:19:023
21730497172,5rudpcli0-21swapper/320:17:113
18426597172,5rudpcli0-21swapper/322:48:003
18426597172,5rudpcli0-21swapper/322:35:013
18426597172,5rudpcli0-21swapper/321:55:013
18426597171,5rudpcli0-21swapper/321:04:093
18426597170,7rudpcli0-21swapper/321:48:133
18426597170,6rudpcli0-21swapper/321:34:183
18426597170,5rudpcli0-21swapper/322:34:083
18426597170,4rudpcli0-21swapper/321:54:083
21730497163,5rudpcli0-21swapper/320:28:343
18426597162,5rudpcli0-21swapper/322:40:033
18426597162,5rudpcli0-21swapper/322:29:083
18426597162,5rudpcli0-21swapper/320:50:013
16661912168,5sleep10-21swapper/119:09:041
16661912168,5sleep10-21swapper/119:09:041
19951262142,8sleep30-21swapper/300:29:063
19475002145,5sleep30-21swapper/323:44:053
19910342132,6sleep30-21swapper/300:25:013
19826202123,5sleep30-21swapper/300:19:043
19789202122,6sleep30-21swapper/300:14:063
19789202122,6sleep30-21swapper/300:14:063
19754542122,6sleep30-21swapper/300:09:083
19619082122,6sleep30-21swapper/323:58:013
19516272122,6sleep30-21swapper/323:45:013
19357442120,2sleep3471rcuc/323:29:113
19964192112,5sleep30-21swapper/300:30:073
19668282112,5sleep30-21swapper/300:02:433
19579902112,5sleep30-21swapper/323:54:043
19420792112,5sleep30-21swapper/323:36:153
19373802112,5sleep30-21swapper/323:33:583
19176392112,5sleep30-21swapper/323:13:563
19121882112,5sleep30-21swapper/323:05:013
19026712113,5sleep30-21swapper/322:56:393
20013512102,5sleep30-21swapper/300:35:403
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional