You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-12 - 18:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue May 12, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
179010921340,3sleep30-21swapper/311:45:013
170521521290,4sleep13056-21PK-Backend07:40:181
17088892990,4sleep33056-21PK-Backend07:50:223
16925642770,6sleep1251rcuc/107:05:291
139099740,61rtkit-daemon0-21swapper/007:06:330
169290226343,14sleep30-21swapper/307:08:503
169286426343,14sleep20-21swapper/207:08:192
16939342540,2sleep03056-21PK-Backend07:10:180
17416492490,4sleep33056-21PK-Backend09:25:193
17473102460,3sleep20-21swapper/209:40:212
17160442460,4sleep013-21ksoftirqd/008:10:370
17253742330,1sleep00-21swapper/008:40:120
139099310,4rtkit-daemon1389-21rtkit-daemon11:32:452
139099310,4rtkit-daemon1389-21rtkit-daemon11:32:442
139099290,4rtkit-daemon1389-21rtkit-daemon11:45:242
169315899286,10cyclictest1709480-21munin-run07:55:012
169315899285,11cyclictest1740691-21systemctl09:25:002
169315899276,8cyclictest1707140-21python307:45:222
169315899275,9cyclictest1710605-21python307:55:212
169315899275,8cyclictest1760526-21apt10:20:102
1693158992711,14cyclictest1798695-21turbostat12:10:002
139099270,4rtkit-daemon1389-21rtkit-daemon12:04:273
169315899265,8cyclictest1736316-21memory09:10:192
169315899264,9cyclictest1717645-21timerandwakeup08:15:232
169315899263,13cyclictest0-21swapper/210:10:142
169315899255,11cyclictest0-21swapper/209:35:512
169315899254,8cyclictest0-21swapper/210:40:182
139099250,4rtkit-daemon1389-21rtkit-daemon08:40:172
169315899243,9cyclictest1793738-21/usr/sbin/munin11:55:212
139099240,16rtkit-daemon3056-21PK-Backend07:11:381
169315899239,12cyclictest436-20systemd-journal11:37:372
169315899237,14cyclictest1721682-21wc08:30:012
1693158992311,10cyclictest1765704-21find10:35:132
169315899231,10cyclictest0-21swapper/209:56:162
1693158992310,11cyclictest1755454-21cpuspeed_turbos10:05:142
169315399230,22cyclictest0-21swapper/008:15:140
139099230,4rtkit-daemon1389-21rtkit-daemon08:18:563
17564172220,4sleep00-21swapper/010:05:240
169316399222,18cyclictest1791404-21package-data-do11:45:363
1693163992220,1cyclictest0-21swapper/312:07:473
1693163992220,1cyclictest0-21swapper/311:33:403
1693163992220,1cyclictest0-21swapper/311:33:403
1693163992220,1cyclictest0-21swapper/310:33:573
1693163992219,2cyclictest3056-21PK-Backend09:14:163
1693163992216,4cyclictest436-20systemd-journal08:49:553
169315899228,6cyclictest1807371-21latency_hist12:35:002
169315899227,12cyclictest1748640-21kernelversion09:45:162
169315899223,8cyclictest0-21swapper/210:30:202
1693158992210,11cyclictest883-21dbus-daemon12:05:002
169315899220,21cyclictest0-21swapper/209:09:572
1693155992219,2cyclictest3056-21PK-Backend08:58:031
1693155992219,2cyclictest3056-21PK-Backend08:36:131
1693155992219,2cyclictest3056-21PK-Backend07:25:361
1693155992215,5cyclictest883-21dbus-daemon10:45:011
1693155992211,9cyclictest1747313-21switchtime09:40:211
169315599220,21cyclictest0-21swapper/112:28:191
169315599220,20cyclictest3056-21PK-Backend07:17:301
1693153992220,1cyclictest0-21swapper/009:20:570
1693153992219,2cyclictest0-21swapper/010:10:390
1693153992216,4cyclictest1737189-21cron09:15:000
139099220,15rtkit-daemon3056-21PK-Backend10:53:251
1693163992119,1cyclictest3056-21PK-Backend10:14:383
1693163992119,1cyclictest0-21swapper/312:24:233
1693163992119,1cyclictest0-21swapper/311:38:003
1693163992119,1cyclictest0-21swapper/311:28:193
1693163992119,1cyclictest0-21swapper/311:07:443
1693163992119,1cyclictest0-21swapper/310:47:583
1693163992119,1cyclictest0-21swapper/310:42:363
1693163992119,1cyclictest0-21swapper/307:12:083
1693163992118,2cyclictest3056-21PK-Backend08:06:023
1693163992114,5cyclictest1702685-21touch07:35:013
1693163992113,6cyclictest1697013-21cron07:20:003
1693163992112,7cyclictest1767192-21munin-plugin-st10:39:593
169316399210,20cyclictest1020-21ntpd12:33:233
169316399210,20cyclictest0-21swapper/310:52:173
169316399210,20cyclictest0-21swapper/308:20:553
169315899219,11cyclictest1694819-21awk07:15:002
1693158992119,1cyclictest0-21swapper/212:27:212
1693158992119,1cyclictest0-21swapper/212:24:172
1693158992119,1cyclictest0-21swapper/211:52:142
1693158992119,1cyclictest0-21swapper/211:18:402
1693158992119,1cyclictest0-21swapper/211:11:032
1693158992119,1cyclictest0-21swapper/211:03:132
1693158992119,1cyclictest0-21swapper/210:51:022
1693158992119,1cyclictest0-21swapper/208:37:462
1693158992119,1cyclictest0-21swapper/208:21:042
1693158992119,1cyclictest0-21swapper/208:06:572
1693158992119,1cyclictest0-21swapper/208:02:372
1693158992118,2cyclictest1706046-21latency_hist07:45:012
1693158992114,5cyclictest1696263-21ntpq07:15:182
1693158992113,6cyclictest1750582-21ntpq09:50:172
1693158992112,8cyclictest1777084-21ntpq11:05:192
1693158992110,9cyclictest1788938-21taskset11:40:152
169315899210,20cyclictest0-21swapper/209:30:192
169315899210,20cyclictest0-21swapper/209:30:182
1693155992119,1cyclictest3056-21PK-Backend12:01:411
1693155992119,1cyclictest3056-21PK-Backend11:18:491
1693155992119,1cyclictest3056-21PK-Backend11:04:021
1693155992119,1cyclictest3056-21PK-Backend09:54:131
1693155992119,1cyclictest0-21swapper/109:13:231
1693155992119,1cyclictest0-21swapper/107:51:591
1693155992119,1cyclictest0-21swapper/107:45:021
1693155992118,2cyclictest3056-21PK-Backend12:14:121
1693155992118,2cyclictest3056-21PK-Backend08:32:211
1693155992118,2cyclictest3056-21PK-Backend07:37:191
1693155992113,6cyclictest1734727-21ntpq09:05:181
169315599210,20cyclictest3056-21PK-Backend11:46:061
169315599210,20cyclictest0-21swapper/109:00:061
169315599210,19cyclictest3056-21PK-Backend12:33:111
169315599210,19cyclictest3056-21PK-Backend11:37:411
169315599210,19cyclictest3056-21PK-Backend11:25:111
169315599210,19cyclictest3056-21PK-Backend10:30:091
1693153992119,1cyclictest0-21swapper/011:44:350
1693153992119,1cyclictest0-21swapper/011:19:070
1693153992119,1cyclictest0-21swapper/008:24:330
169315399210,20cyclictest0-21swapper/011:35:090
169315399210,19cyclictest3056-21PK-Backend10:45:480
1693163992018,1cyclictest3056-21PK-Backend07:55:473
1693163992018,1cyclictest3056-21PK-Backend07:44:463
1693163992018,1cyclictest0-21swapper/311:22:593
1693163992018,1cyclictest0-21swapper/311:19:413
1693163992018,1cyclictest0-21swapper/310:57:513
1693163992017,2cyclictest3056-21PK-Backend09:31:533
1693163992017,2cyclictest3056-21PK-Backend09:31:533
169316399200,3cyclictest3056-21PK-Backend10:20:113
169316399200,1cyclictest3056-21PK-Backend09:19:223
169316399200,1cyclictest0-21swapper/309:48:423
169316399200,19cyclictest3056-21PK-Backend09:36:593
169316399200,19cyclictest0-21swapper/312:35:253
169316399200,19cyclictest0-21swapper/311:11:023
169316399200,19cyclictest0-21swapper/310:25:203
169316399200,19cyclictest0-21swapper/309:44:233
169316399200,19cyclictest0-21swapper/308:40:293
169316399200,19cyclictest0-21swapper/307:32:553
169315899207,11cyclictest0-21swapper/208:59:562
1693158992018,1cyclictest0-21swapper/212:14:522
1693158992018,1cyclictest0-21swapper/211:20:582
1693158992018,1cyclictest0-21swapper/210:26:572
1693158992018,1cyclictest0-21swapper/208:45:582
1693158992018,1cyclictest0-21swapper/208:30:502
1693158992018,1cyclictest0-21swapper/207:25:072
1693158992013,6cyclictest1774070-21sshd10:59:382
1693158992010,8cyclictest1753572-21/usr/sbin/munin10:00:222
169315899200,19cyclictest0-21swapper/212:39:252
169315899200,19cyclictest0-21swapper/207:39:532
1693155992018,1cyclictest3056-21PK-Backend12:23:251
1693155992018,1cyclictest3056-21PK-Backend11:24:091
1693155992018,1cyclictest3056-21PK-Backend10:58:411
1693155992018,1cyclictest3056-21PK-Backend10:22:451
1693155992018,1cyclictest0-21swapper/110:37:331
1693155992017,2cyclictest3056-21PK-Backend07:21:511
169315599200,19cyclictest3056-21PK-Backend10:00:351
169315599200,19cyclictest0-21swapper/110:15:231
1693153992018,1cyclictest0-21swapper/007:17:150
1693153992013,5cyclictest1706454-21expr07:45:130
169315399200,19cyclictest0-21swapper/011:53:000
169315399200,19cyclictest0-21swapper/009:39:360
169315399200,19cyclictest0-21swapper/009:02:190
169315399200,19cyclictest0-21swapper/008:57:110
169315399200,19cyclictest0-21swapper/007:56:480
139099200,2rtkit-daemon1389-21rtkit-daemon07:30:372
1693163991917,1cyclictest0-21swapper/312:15:103
1693163991914,3cyclictest1753066-21munin-run10:00:013
1693163991912,5cyclictest42-21ksoftirqd/309:50:213
169316399190,1cyclictest0-21swapper/311:55:023
169316399190,1cyclictest0-21swapper/308:31:293
169316399190,18cyclictest0-21swapper/312:26:253
169315899199,8cyclictest1729602-21ps08:50:212
1693158991917,1cyclictest0-21swapper/207:20:342
1693158991912,6cyclictest1738168-21awk09:15:182
1693158991911,6cyclictest883-21dbus-daemon11:28:042
1693158991910,7cyclictest1800779-21/usr/sbin/munin12:15:132
169315899190,1cyclictest0-21swapper/209:25:592
169315899190,1cyclictest0-21swapper/208:11:172
169315599194,13cyclictest1740649-21cron09:25:001
1693155991917,1cyclictest3056-21PK-Backend08:42:071
1693155991917,1cyclictest3056-21PK-Backend08:23:491
1693155991916,2cyclictest3056-21PK-Backend11:09:231
1693155991916,2cyclictest3056-21PK-Backend07:33:271
169315599190,1cyclictest3056-21PK-Backend11:10:251
169315599190,1cyclictest3056-21PK-Backend09:45:491
169315599190,1cyclictest3056-21PK-Backend08:54:591
169315599190,1cyclictest3056-21PK-Backend08:47:001
169315599190,1cyclictest3056-21PK-Backend08:28:291
169315599190,1cyclictest3056-21PK-Backend08:16:181
1693153991917,1cyclictest0-21swapper/012:06:460
1693153991917,1cyclictest0-21swapper/011:23:270
1693153991917,1cyclictest0-21swapper/011:12:450
1693153991917,1cyclictest0-21swapper/010:51:090
1693153991917,1cyclictest0-21swapper/010:17:020
1693153991917,1cyclictest0-21swapper/009:44:570
1693153991917,1cyclictest0-21swapper/009:08:280
1693153991915,3cyclictest1095-21runrttasks12:20:180
1693153991911,6cyclictest1783681-21df11:25:130
1693153991910,7cyclictest883-21dbus-daemon07:35:010
169315399190,5cyclictest1698770-21basename07:24:590
169315399190,1cyclictest0-21swapper/012:32:080
169315399190,1cyclictest0-21swapper/011:09:260
169315399190,1cyclictest0-21swapper/008:36:100
169315399190,1cyclictest0-21swapper/008:03:590
169315399190,1cyclictest0-21swapper/007:43:450
169315399190,18cyclictest3056-21PK-Backend12:27:140
169315399190,18cyclictest0-21swapper/012:15:100
169315399190,18cyclictest0-21swapper/010:55:280
169315399190,18cyclictest0-21swapper/008:45:080
169316399187,5cyclictest1730920-21cat08:55:143
1693163991816,1cyclictest0-21swapper/308:27:373
1693163991813,3cyclictest1723981-21diskstats08:35:133
1693163991812,4cyclictest42-21ksoftirqd/307:25:143
1693163991812,4cyclictest1792828-21processes11:50:193
1693163991811,5cyclictest1775136-21ntpq11:00:213
169315899189,7cyclictest1732590-21expr09:00:132
169315599182,12cyclictest1095-21runrttasks08:05:111
1693155991816,1cyclictest0-21swapper/108:00:101
1693155991811,5cyclictest1758515-21dump-pmu-power10:15:011
1693155991810,6cyclictest1770664-21munin-run10:50:011
1693153991816,1cyclictest0-21swapper/011:04:060
1693153991816,1cyclictest0-21swapper/010:29:460
1693153991816,1cyclictest0-21swapper/010:24:250
1693153991816,1cyclictest0-21swapper/007:53:440
1693153991816,1cyclictest0-21swapper/007:25:180
1693153991811,6cyclictest1754305-21awk10:00:190
169315399181,15cyclictest1737465-21apt09:15:110
169315399180,1cyclictest0-21swapper/010:40:270
169315399180,1cyclictest0-21swapper/009:29:100
1693163991715,1cyclictest3056-21PK-Backend09:05:033
1693163991715,1cyclictest3056-21PK-Backend09:00:573
1693163991715,1cyclictest3056-21PK-Backend08:00:553
1693163991714,2cyclictest3056-21PK-Backend10:19:583
169316399171,2cyclictest1753536-21cnf-update-db10:00:013
1693163991711,5cyclictest1698239-21grep07:20:223
1693163991711,4cyclictest42-21ksoftirqd/308:50:173
1693163991710,5cyclictest1715687-21ntpq08:10:203
169315899178,8cyclictest1769665-21irqstats10:45:152
1693158991713,3cyclictest1758939-21tr10:15:122
169315599179,6cyclictest1743072-21smartctl09:30:151
169315599179,6cyclictest1743072-21smartctl09:30:151
1693155991715,1cyclictest3056-21PK-Backend11:33:371
1693155991715,1cyclictest3056-21PK-Backend11:33:371
1693155991715,1cyclictest0-21swapper/109:35:201
1693155991715,1cyclictest0-21swapper/107:55:031
1693155991714,2cyclictest3056-21PK-Backend11:40:591
1693155991712,3cyclictest1807774-21cpuspeed_turbos12:35:131
169315399176,10cyclictest1730277-21grep08:55:000
1693153991715,1cyclictest0-21swapper/012:39:040
1693153991715,1cyclictest0-21swapper/008:32:180
1693153991715,1cyclictest0-21swapper/008:26:360
1693153991715,1cyclictest0-21swapper/007:33:100
1693153991712,4cyclictest1786506-21cron11:35:000
1693153991712,4cyclictest1786506-21cron11:34:590
1693153991711,5cyclictest1713727-21ls08:05:170
169316399169,5cyclictest1756078-21awk10:05:203
1693163991611,3cyclictest1798931-21/usr/sbin/munin12:10:123
1693155991612,3cyclictest1715817-21switchtime08:10:211
1693155991610,4cyclictest0-21swapper/109:30:001
169315599160,6cyclictest3056-21PK-Backend10:08:381
169315599160,6cyclictest3056-21PK-Backend10:00:001
169315399169,6cyclictest1765905-21cpuspeed_turbos10:35:130
169315399166,8cyclictest0-21swapper/010:34:590
1693153991614,1cyclictest0-21swapper/009:49:170
1693153991611,3cyclictest1795227-21munin-run12:00:010
1693153991611,3cyclictest1743674-21timerwakeupswit09:30:230
1693153991611,3cyclictest1743674-21timerwakeupswit09:30:230
1693163991513,1cyclictest3056-21PK-Backend07:46:493
169316399150,3cyclictest1740662-21ls09:24:593
169315599152,10cyclictest1737468-21apt09:15:121
1693155991513,1cyclictest3056-21PK-Backend10:29:071
169315599150,5cyclictest3056-21PK-Backend11:55:241
1693153991510,3cyclictest1790480-21cpuspeed_turbos11:45:120
169315599149,4cyclictest3056-21PK-Backend12:15:271
169315599140,4cyclictest1798703-21awk12:10:001
169315399148,5cyclictest1752119-21latency09:55:170
169315399147,5cyclictest1795486-21awk12:00:120
169315399146,7cyclictest1750519-21needreboot09:50:180
1693153991411,2cyclictest1798931-21/usr/sbin/munin12:10:110
169315599133,2cyclictest141rcu_preempt11:55:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional