You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-29 - 06:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sun Mar 29, 2026 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
171266521160,4sleep20-21swapper/221:05:132
172070121150,4sleep21720697-21turbostat21:30:002
139099880,71rtkit-daemon0-21swapper/319:05:353
166976927744,22sleep10-21swapper/119:05:121
167098126739,22sleep00-21swapper/019:06:380
167099826341,14sleep20-21swapper/219:06:532
17581712490,2sleep13056-21PK-Backend23:15:121
17810252440,3sleep13056-21PK-Backend00:20:161
139099410,3rtkit-daemon1389-21rtkit-daemon21:59:083
139099400,3rtkit-daemon1389-21rtkit-daemon23:28:543
139099400,3rtkit-daemon1389-21rtkit-daemon20:30:033
167138699397,10cyclictest1770321-21expr23:50:112
17149712380,3sleep13056-21PK-Backend21:10:221
139099380,3rtkit-daemon1389-21rtkit-daemon22:49:103
139099380,3rtkit-daemon1389-21rtkit-daemon20:53:163
167138699355,15cyclictest0-21swapper/219:45:002
167138699348,10cyclictest1696111-21perf20:20:002
17825392320,5sleep00-21swapper/000:25:120
167139199321,6cyclictest1738455-21cut22:20:013
167138699324,14cyclictest0-21swapper/219:15:012
167138699324,13cyclictest0-21swapper/200:05:132
1671386993219,9cyclictest883-21dbus-daemon21:00:052
167138699317,10cyclictest1727653-21perf-exec21:50:002
167138699316,9cyclictest1754383-21cut23:05:012
167138699315,8cyclictest0-21swapper/222:40:222
1671386993111,6cyclictest1699597-21munin-run20:30:002
1671391993026,3cyclictest411rcuc/322:34:123
167138699308,9cyclictest1738339-21awk22:20:002
167138699307,10cyclictest1757712-21latency_hist23:15:012
1671391992913,9cyclictest42-21ksoftirqd/320:10:003
167138699299,2cyclictest1677111-21cut19:25:012
167138699297,8cyclictest1690895-21munin-run20:05:012
167138699296,9cyclictest1678751-21munin-run19:30:002
167138699294,15cyclictest0-21swapper/222:40:002
167138699294,14cyclictest0-21swapper/223:25:012
167138699286,9cyclictest1687616-21latency_hist19:55:002
167138699286,10cyclictest1726114-21sed21:45:002
167138699285,13cyclictest0-21swapper/220:50:002
167138699284,9cyclictest0-21swapper/220:40:002
167138699284,14cyclictest0-21swapper/223:40:212
1671386992812,10cyclictest0-21swapper/220:10:122
167138699277,7cyclictest1785789-21munin-run00:35:012
167138699276,5cyclictest1742279-21dpkg22:30:022
167138699274,9cyclictest0-21swapper/220:55:132
167138699274,9cyclictest0-21swapper/219:15:162
167138699274,9cyclictest0-21swapper/200:25:122
1671383992719,6cyclictest1710023-21cron21:00:001
167139199267,2cyclictest1727705-21sed21:50:013
167138699267,6cyclictest883-21dbus-daemon00:00:002
167138699267,6cyclictest1740169-21latency_hist22:25:022
167138699267,6cyclictest1715637-21cat21:15:012
167138699263,13cyclictest0-21swapper/221:50:142
1671386992616,8cyclictest1735977-21python322:10:202
1671386992616,6cyclictest1748216-21needreboot22:45:172
1671386992614,8cyclictest1732506-21python322:00:202
167139199257,16cyclictest1705387-21expr20:45:143
167139199255,2cyclictest1713804-21sed21:10:013
167138699259,14cyclictest1777313-21cstates00:10:132
167138699257,6cyclictest1782258-21cat00:25:002
167138699257,4cyclictest0-21swapper/219:55:192
167138699256,5cyclictest0-21swapper/220:25:012
167138699254,8cyclictest0-21swapper/220:30:122
167138699253,9cyclictest0-21swapper/223:30:202
1671386992516,7cyclictest1755373-21taskset23:05:212
1671386992515,8cyclictest1722829-21tr21:35:122
1671386992512,10cyclictest0-21swapper/220:40:122
1671386992512,10cyclictest0-21swapper/220:40:122
139099250,4rtkit-daemon1389-21rtkit-daemon21:31:252
167139199245,4cyclictest1692899-21apt20:10:133
167139199243,12cyclictest1760052-21expr23:20:163
167139199242,8cyclictest1742089-21apt-config22:30:013
1671391992420,3cyclictest411rcuc/323:46:083
1671386992416,6cyclictest1762936-21munin-run23:29:592
1671386992416,4cyclictest1773638-21apt00:00:122
1671386992415,6cyclictest1780326-21basename00:20:002
1671386992415,4cyclictest1733376-21apt22:05:122
1671386992414,7cyclictest1757950-21/usr/sbin/munin23:15:112
1671386992414,7cyclictest1749254-21/usr/sbin/munin22:50:172
1671386992413,8cyclictest1707689-21python320:50:202
139099240,3rtkit-daemon1389-21rtkit-daemon23:42:183
139099240,3rtkit-daemon1389-21rtkit-daemon00:24:553
17054682230,4sleep13056-21PK-Backend20:45:151
167139199233,7cyclictest1773813-21grep00:00:133
167139199232,5cyclictest1757809-21sed23:15:013
1671391992320,2cyclictest411rcuc/322:13:343
1671391992319,3cyclictest0-21swapper/319:15:203
167139199230,22cyclictest1389-21rtkit-daemon22:03:493
167139199230,21cyclictest1389-21rtkit-daemon20:40:333
167139199230,21cyclictest1389-21rtkit-daemon20:40:333
1671386992318,2cyclictest1685663-21cron19:50:002
1671386992316,3cyclictest1718976-21perf21:25:002
1671386992315,4cyclictest1716619-21python321:15:212
1671386992314,5cyclictest1787506-21cron00:40:002
1671386992313,8cyclictest1765880-21awk23:35:222
1671386992313,8cyclictest1741273-21find22:25:222
1671386992313,8cyclictest1730137-21uname21:55:172
1671383992320,2cyclictest3056-21PK-Backend00:13:171
1671383992320,1cyclictest3056-21PK-Backend19:26:431
16825482220,4sleep13056-21PK-Backend19:40:111
167139199224,7cyclictest42-21ksoftirqd/323:35:303
167139199222,6cyclictest1744214-21expr22:35:113
167139199221,9cyclictest1673078-21systemd-run19:15:003
1671391992219,2cyclictest411rcuc/321:44:063
167139199221,7cyclictest1715143-21taskset21:10:473
167139199221,5cyclictest1696550-21cpuspeed_turbos20:20:123
167139199220,8cyclictest1687347-21taskset19:54:213
167139199220,21cyclictest1389-21rtkit-daemon00:16:493
167139199220,20cyclictest1683971-21unin-run19:45:003
167138699229,11cyclictest1691366-21ls20:05:122
1671386992215,5cyclictest1751695-21grep22:55:192
1671386992212,6cyclictest1768540-21grep23:45:132
1671383992220,1cyclictest3056-21PK-Backend22:09:131
1671383992219,2cyclictest3056-21PK-Backend23:12:131
1671383992219,2cyclictest3056-21PK-Backend21:09:531
1671383992219,2cyclictest3056-21PK-Backend19:34:481
167138399220,20cyclictest3056-21PK-Backend22:35:291
1671381992219,2cyclictest3056-21PK-Backend19:15:310
1671381992216,5cyclictest1748997-21cron22:50:000
139099220,15rtkit-daemon0-21swapper/019:49:180
17795902210,3chrt3056-21PK-Backend00:15:191
167139199219,2cyclictest141rcu_preempt22:50:123
167139199213,7cyclictest42-21ksoftirqd/323:57:523
1671391992118,2cyclictest411rcuc/321:24:453
1671391992115,4cyclictest883-21dbus-daemon20:28:033
1671391992113,6cyclictest1710450-21mktemp21:00:013
1671391992112,7cyclictest957-21gdbus23:15:473
167139199210,6cyclictest1720698-21perf21:29:593
167139199210,20cyclictest1389-21rtkit-daemon22:41:183
167139199210,20cyclictest1389-21rtkit-daemon00:25:293
1671386992115,3cyclictest1679678-21strings19:30:172
1671386992114,5cyclictest1680869-21cut19:35:122
1671386992114,5cyclictest1680869-21cut19:35:122
167138399215,14cyclictest1761189-21cron23:25:001
1671383992119,1cyclictest3056-21PK-Backend23:56:141
1671383992119,1cyclictest3056-21PK-Backend23:47:491
1671383992119,1cyclictest3056-21PK-Backend23:25:251
1671383992119,1cyclictest3056-21PK-Backend22:27:451
1671383992119,1cyclictest3056-21PK-Backend21:29:051
1671383992119,1cyclictest3056-21PK-Backend20:35:041
1671383992119,1cyclictest3056-21PK-Backend20:16:041
1671383992119,1cyclictest3056-21PK-Backend20:14:161
1671383992119,1cyclictest3056-21PK-Backend19:23:191
1671383992118,1cyclictest3056-21PK-Backend21:40:131
1671383992111,7cyclictest1783786-21cron00:30:001
167138399210,20cyclictest3056-21PK-Backend21:37:241
167138399210,20cyclictest3056-21PK-Backend20:40:321
167138399210,20cyclictest3056-21PK-Backend20:40:311
167138399210,20cyclictest3056-21PK-Backend20:03:341
167138399210,19cyclictest3056-21PK-Backend23:03:351
167138399210,19cyclictest3056-21PK-Backend21:00:461
167138399210,19cyclictest3056-21PK-Backend19:58:061
1671381992119,1cyclictest0-21swapper/023:38:520
1671381992119,1cyclictest0-21swapper/023:00:470
1671381992119,1cyclictest0-21swapper/022:58:300
1671381992119,1cyclictest0-21swapper/022:51:070
1671381992119,1cyclictest0-21swapper/022:22:540
1671381992119,1cyclictest0-21swapper/021:20:350
1671381992119,1cyclictest0-21swapper/021:14:070
1671381992119,1cyclictest0-21swapper/020:57:440
1671381992119,1cyclictest0-21swapper/020:29:110
1671381992119,1cyclictest0-21swapper/020:23:450
1671381992119,1cyclictest0-21swapper/000:23:540
167138199210,20cyclictest0-21swapper/023:52:380
167138199210,20cyclictest0-21swapper/020:18:170
139099210,14rtkit-daemon0-21swapper/021:36:280
139099210,14rtkit-daemon0-21swapper/019:50:060
167139199205,13cyclictest42-21ksoftirqd/322:22:293
167139199201,9cyclictest1710033-21ls21:00:003
1671391992018,1cyclictest0-21swapper/323:33:543
1671391992018,1cyclictest0-21swapper/319:37:163
1671391992018,1cyclictest0-21swapper/319:37:163
167139199201,6cyclictest1696119-21munin-run20:20:003
1671391992014,4cyclictest1679869-21python319:30:193
1671391992014,3cyclictest1786911-21python300:35:223
167139199201,3cyclictest1785804-21basename00:35:003
1671391992013,5cyclictest1770497-21cstates23:50:123
1671391992012,6cyclictest1689970-21cat20:00:173
167139199200,1cyclictest0-21swapper/300:05:243
167139199200,19cyclictest0-21swapper/321:36:063
167139199200,19cyclictest0-21swapper/321:17:133
167139199200,19cyclictest0-21swapper/319:25:193
167139199200,19cyclictest0-21swapper/319:21:563
167139199200,19cyclictest0-21swapper/300:14:173
1671383992018,1cyclictest3056-21PK-Backend22:59:151
1671383992018,1cyclictest3056-21PK-Backend22:15:081
1671383992018,1cyclictest3056-21PK-Backend22:14:071
1671383992018,1cyclictest3056-21PK-Backend20:31:261
1671383992018,1cyclictest3056-21PK-Backend20:22:201
1671383992018,1cyclictest3056-21PK-Backend19:52:251
1671383992018,1cyclictest0-21swapper/123:05:521
1671383992018,1cyclictest0-21swapper/122:45:431
167138399200,1cyclictest3056-21PK-Backend22:51:521
167138399200,19cyclictest3056-21PK-Backend00:09:591
167138399200,18cyclictest3056-21PK-Backend21:48:311
1671381992018,1cyclictest0-21swapper/023:26:560
1671381992018,1cyclictest0-21swapper/023:23:380
1671381992018,1cyclictest0-21swapper/023:14:460
1671381992018,1cyclictest0-21swapper/022:27:480
1671381992018,1cyclictest0-21swapper/022:04:090
1671381992018,1cyclictest0-21swapper/021:29:410
1671381992018,1cyclictest0-21swapper/020:35:400
1671381992018,1cyclictest0-21swapper/019:58:290
1671381992018,1cyclictest0-21swapper/019:38:490
1671381992018,1cyclictest0-21swapper/019:38:490
1671381992018,1cyclictest0-21swapper/000:05:220
1671381992017,2cyclictest0-21swapper/021:30:430
1671381992013,5cyclictest883-21dbus-daemon22:08:040
167138199200,2cyclictest0-21swapper/021:15:570
167138199200,1cyclictest0-21swapper/022:14:570
167138199200,19cyclictest0-21swapper/023:30:140
167138199200,19cyclictest0-21swapper/022:31:400
167138199200,19cyclictest0-21swapper/021:03:120
167138199200,19cyclictest0-21swapper/020:45:490
167138199200,19cyclictest0-21swapper/020:01:200
167138199200,19cyclictest0-21swapper/000:35:510
1671391991917,1cyclictest0-21swapper/319:47:383
1671391991915,3cyclictest2156-21gnome-shell21:54:093
1671391991913,5cyclictest1702351-21open_inodes20:35:193
167139199190,18cyclictest0-21swapper/323:07:043
167139199190,18cyclictest0-21swapper/322:08:513
167139199190,17cyclictest1389-21rtkit-daemon22:55:473
1671383991917,1cyclictest3056-21PK-Backend21:21:481
1671383991917,1cyclictest3056-21PK-Backend20:25:581
1671383991917,1cyclictest3056-21PK-Backend19:49:351
1671383991912,5cyclictest1787510-21cron00:40:001
167138399190,1cyclictest3056-21PK-Backend22:22:521
167138399190,1cyclictest3056-21PK-Backend21:56:501
167138399190,1cyclictest3056-21PK-Backend20:51:401
167138399190,1cyclictest3056-21PK-Backend19:36:381
167138399190,1cyclictest3056-21PK-Backend19:36:381
167138399190,1cyclictest3056-21PK-Backend00:31:311
167138399190,1cyclictest0-21swapper/123:35:061
167138399190,18cyclictest3056-21PK-Backend22:01:301
1671381991917,1cyclictest0-21swapper/023:44:130
1671381991917,1cyclictest0-21swapper/020:12:010
1671381991917,1cyclictest0-21swapper/019:43:300
1671381991917,1cyclictest0-21swapper/019:11:320
1671381991917,1cyclictest0-21swapper/000:30:300
1671381991917,1cyclictest0-21swapper/000:11:580
167138199190,1cyclictest3056-21PK-Backend20:31:010
167138199190,1cyclictest0-21swapper/022:42:430
167138199190,1cyclictest0-21swapper/020:41:080
167138199190,1cyclictest0-21swapper/020:41:080
167138199190,1cyclictest0-21swapper/019:25:170
167138199190,18cyclictest0-21swapper/022:36:330
167138199190,18cyclictest0-21swapper/020:05:460
167139199183,13cyclictest0-21swapper/319:55:133
1671391991811,6cyclictest1721177-21expr21:30:123
167138399186,2cyclictest141rcu_preempt19:20:001
167138399184,2cyclictest141rcu_preempt00:00:131
167138399183,2cyclictest141rcu_preempt20:05:181
1671383991816,1cyclictest3056-21PK-Backend21:51:221
1671383991816,1cyclictest3056-21PK-Backend21:30:071
1671381991816,1cyclictest0-21swapper/023:16:020
1671381991816,1cyclictest0-21swapper/022:18:010
1671381991816,1cyclictest0-21swapper/019:33:210
1671381991810,6cyclictest436-20systemd-journal19:25:000
167139199171,3cyclictest1754231-21turbostat23:05:003
1671383991714,2cyclictest3056-21PK-Backend21:16:211
167138199176,5cyclictest0-21swapper/021:45:010
1671381991715,1cyclictest0-21swapper/021:55:510
1671381991715,1cyclictest0-21swapper/021:54:490
1671381991715,1cyclictest0-21swapper/020:51:290
167138199171,2cyclictest141rcu_preempt23:55:220
1671381991711,5cyclictest1755963-21cron23:10:000
167138399163,2cyclictest141rcu_preempt23:50:111
1671381991614,1cyclictest0-21swapper/021:48:200
1671381991614,1cyclictest0-21swapper/021:06:020
1671381991614,1cyclictest0-21swapper/000:19:360
1671381991611,3cyclictest1769129-21ntp_kernel_pll_23:45:200
1671383991510,4cyclictest1764883-21cut23:35:001
1671381991511,3cyclictest1774612-21timerandwakeup00:00:220
167138399141,3cyclictest141rcu_preempt22:30:211
1671383991410,3cyclictest1095-21runrttasks23:44:481
167138399131,2cyclictest141rcu_preempt22:40:171
167138399121,2cyclictest141rcu_preempt19:10:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional