You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 21:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Fri Nov 28, 2025 12:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991060,9rtkit-daemon1622-21rtkit-daemon08:00:303
16589482990,4chrt1658463-21/usr/sbin/munin10:05:210
162399970,9rtkit-daemon1622-21rtkit-daemon09:49:003
162399970,9rtkit-daemon1622-21rtkit-daemon08:39:473
162399970,8rtkit-daemon1622-21rtkit-daemon09:23:023
16723662960,7sleep00-21swapper/010:45:130
16520482950,4sleep20-21swapper/209:45:222
16472382890,8sleep30-21swapper/309:30:303
159568628041,15sleep00-21swapper/007:06:480
159579627858,14sleep30-21swapper/307:08:163
159565827139,24sleep20-21swapper/207:06:262
162399600,9rtkit-daemon1622-21rtkit-daemon09:03:223
159445426040,13sleep10-21swapper/107:05:041
162399590,9rtkit-daemon1622-21rtkit-daemon08:49:152
162399570,9rtkit-daemon1622-21rtkit-daemon09:13:202
1596110995724,11cyclictest1630062-21sshd08:44:251
162399560,9rtkit-daemon1622-21rtkit-daemon08:55:582
162399560,9rtkit-daemon1622-21rtkit-daemon08:55:582
1596106995528,10cyclictest13-21ksoftirqd/011:10:030
1596106995222,26cyclictest1705092-40lsb_release12:15:230
1596106995222,26cyclictest1705092-40lsb_release12:15:220
162399510,8rtkit-daemon1622-21rtkit-daemon08:36:542
1596106995124,9cyclictest13-21ksoftirqd/010:52:400
16854182500,14sleep3421ksoftirqd/311:20:133
159611099491,15cyclictest16922152cpuspeed_turbos11:40:151
162399470,33rtkit-daemon1598308-21cut07:15:061
1596114994728,14cyclictest34-21ksoftirqd/209:40:132
162399460,31rtkit-daemon1667524-21df10:30:161
159610699464,16cyclictest13-21ksoftirqd/010:34:590
162399450,32rtkit-daemon0-21swapper/110:14:361
159611899451,27cyclictest2197014-21PK-Backend12:22:093
159611499453,4cyclictest141rcu_preempt11:50:012
1596106994523,17cyclictest3242253-21nautilus11:10:030
159610699451,17cyclictest13-21ksoftirqd/010:40:020
162399440,30rtkit-daemon0-21swapper/112:25:101
162399440,30rtkit-daemon0-21swapper/010:58:180
159611899442,6cyclictest141rcu_preempt08:25:123
1596110994425,14cyclictest1905-21gdbus11:10:041
1596106994433,8cyclictest13-21ksoftirqd/008:15:170
1596106994424,15cyclictest855-21dbus-daemon11:20:010
1596106994424,15cyclictest855-21dbus-daemon11:20:000
1596106994413,6cyclictest141rcu_preempt08:31:270
1596106994413,6cyclictest141rcu_preempt08:31:270
1596106994412,15cyclictest854-21avahi-daemon09:45:030
16628172430,5sleep0161rcuc/010:15:270
16623512430,6chrt0-21swapper/310:15:183
16426162430,8sleep01642607-21cron09:20:010
16310322430,6sleep32197014-21PK-Backend08:45:223
1596110994327,12cyclictest1659939-21cron10:10:001
1596110994323,16cyclictest1615518-40apt-check08:00:261
1596110994317,22cyclictest1647824-21cron09:35:001
159610699438,12cyclictest0-21swapper/009:23:050
1596106994318,22cyclictest13-21ksoftirqd/007:15:210
1596106994311,13cyclictest0-21swapper/011:59:540
1596106994310,12cyclictest0-21swapper/011:39:130
16781562420,5sleep2331rcuc/211:00:242
16075142420,7sleep32197014-21PK-Backend07:40:143
159611899426,16cyclictest1753-21gnome-shell07:38:063
159611899422,6cyclictest141rcu_preempt07:22:283
159611499428,9cyclictest34-21ksoftirqd/212:00:112
1596114994225,6cyclictest34-21ksoftirqd/210:20:192
1596114994224,14cyclictest34-21ksoftirqd/211:42:232
1596114994224,14cyclictest34-21ksoftirqd/207:45:022
1596110994219,11cyclictest1609577-21expr07:45:151
1596110994214,24cyclictest1605901-21cat07:35:201
159610699429,18cyclictest13-21ksoftirqd/007:42:350
1596106994224,13cyclictest438-20systemd-journal11:55:000
1596106994220,7cyclictest13-21ksoftirqd/010:20:230
1596106994217,11cyclictest141rcu_preempt12:33:460
1596106994210,16cyclictest0-21swapper/008:35:090
17083672410,8chrt1753-21gnome-shell12:25:143
16870752410,7sleep22197014-21PK-Backend11:25:152
1596118994127,11cyclictest42-21ksoftirqd/311:15:023
1596118994122,14cyclictest42-21ksoftirqd/309:18:593
1596114994123,13cyclictest34-21ksoftirqd/212:10:252
1596114994123,13cyclictest1661086-21rt-features10:10:282
159610699417,17cyclictest13-21ksoftirqd/012:29:060
1596106994115,8cyclictest13-21ksoftirqd/007:36:060
1596106994114,8cyclictest13-21ksoftirqd/007:22:260
159610699411,18cyclictest13-21ksoftirqd/009:07:070
1596106994110,12cyclictest0-21swapper/010:10:070
162399400,28rtkit-daemon0-21swapper/110:41:101
162399400,28rtkit-daemon0-21swapper/110:41:101
159611899407,13cyclictest141rcu_preempt07:45:563
1596118994025,11cyclictest42-21ksoftirqd/310:10:273
1596118994024,13cyclictest42-21ksoftirqd/307:50:323
159611899402,33cyclictest1753-21gnome-shell08:16:593
159611899401,8cyclictest2197014-21PK-Backend08:06:093
159611899401,15cyclictest1655905-21/usr/sbin/munin09:55:303
159611899401,15cyclictest1655905-21/usr/sbin/munin09:55:303
1596114994028,7cyclictest34-21ksoftirqd/212:05:562
1596114994022,6cyclictest34-21ksoftirqd/211:20:002
1596114994022,6cyclictest34-21ksoftirqd/211:20:002
1596114994022,6cyclictest34-21ksoftirqd/208:35:012
1596114994022,6cyclictest34-21ksoftirqd/208:35:012
1596114994022,6cyclictest34-21ksoftirqd/207:15:062
159611499402,11cyclictest1753-21gnome-shell10:00:022
159611499402,11cyclictest1753-21gnome-shell10:00:012
1596110994027,9cyclictest1670390-21systemd-run10:40:011
1596110994015,20cyclictest1632-21dbus-daemon07:40:031
159611099401,15cyclictest16877552sensors_temp11:25:301
159610699409,13cyclictest0-21swapper/007:14:150
159610699409,12cyclictest0-21swapper/008:52:190
159610699403,18cyclictest13-21ksoftirqd/012:05:230
159610699403,17cyclictest0-21swapper/010:04:030
159610699403,17cyclictest0-21swapper/010:04:020
1596106994029,7cyclictest1616255-21grep08:05:000
1596106994021,14cyclictest1648461-21df09:35:170
1596106994014,8cyclictest13-21ksoftirqd/011:43:350
159610699401,21cyclictest13-21ksoftirqd/007:35:010
159610699401,21cyclictest13-21ksoftirqd/007:35:000
1596106994010,13cyclictest0-21swapper/009:40:170
1596106994010,13cyclictest0-21swapper/008:07:590
162399390,26rtkit-daemon0-21swapper/107:27:071
162399390,25rtkit-daemon0-21swapper/109:58:301
162399390,25rtkit-daemon0-21swapper/109:58:301
159611899396,19cyclictest141rcu_preempt10:00:363
159611899396,19cyclictest141rcu_preempt10:00:363
159611899391,8cyclictest2197014-21PK-Backend08:34:103
159611899391,8cyclictest2197014-21PK-Backend08:34:103
1596118993913,6cyclictest141rcu_preempt12:15:033
1596118993912,23cyclictest0-21swapper/311:56:153
1596118993911,6cyclictest141rcu_preempt08:14:583
1596118993911,11cyclictest141rcu_preempt09:45:003
1596114993925,10cyclictest1689299-21sshd11:30:232
159611499392,33cyclictest1753-21gnome-shell09:39:262
1596114993922,6cyclictest34-21ksoftirqd/209:15:242
1596114993922,5cyclictest34-21ksoftirqd/208:00:012
159611499392,12cyclictest1753-21gnome-shell10:00:282
159611499392,12cyclictest1753-21gnome-shell10:00:282
1596114993921,14cyclictest855-21dbus-daemon07:40:002
1596114993920,14cyclictest34-21ksoftirqd/210:37:472
159611099392,33cyclictest1753-21gnome-shell08:10:261
1596110993921,13cyclictest1638420-21awk09:05:281
1596110993920,14cyclictest855-21dbus-daemon12:35:001
159610699399,12cyclictest0-21swapper/009:29:290
159610699396,16cyclictest13-21ksoftirqd/008:26:000
159610699394,9cyclictest13-21ksoftirqd/008:00:020
159610699393,15cyclictest0-21swapper/009:32:240
1596106993925,10cyclictest1619424-21sshd08:11:240
1596106993923,12cyclictest1653701-21sh09:50:200
1596106993918,9cyclictest141rcu_preempt07:45:210
1596106993917,8cyclictest13-21ksoftirqd/009:01:190
159610699391,20cyclictest13-21ksoftirqd/009:00:010
159610699391,20cyclictest13-21ksoftirqd/009:00:010
162399380,26rtkit-daemon0-21swapper/110:28:481
162399380,25rtkit-daemon0-21swapper/111:51:421
16215112380,7sleep22197014-21PK-Backend08:20:022
159611899387,11cyclictest141rcu_preempt11:07:073
159611899385,12cyclictest141rcu_preempt11:20:003
159611899385,12cyclictest141rcu_preempt11:20:003
159611899382,32cyclictest1753-21gnome-shell07:26:003
1596118993818,5cyclictest141rcu_preempt11:51:453
159611899381,16cyclictest1638849-40lsb_release09:06:273
1596114993827,7cyclictest1753-21gnome-shell08:41:372
159611499382,32cyclictest1753-21gnome-shell12:32:042
1596114993821,5cyclictest34-21ksoftirqd/211:20:132
159611499382,12cyclictest1753-21gnome-shell10:40:102
159611499382,12cyclictest1753-21gnome-shell10:40:102
1596114993821,12cyclictest34-21ksoftirqd/208:10:312
1596114993820,6cyclictest34-21ksoftirqd/209:34:042
1596114993820,13cyclictest855-21dbus-daemon07:35:012
1596114993820,13cyclictest855-21dbus-daemon07:35:012
1596114993820,13cyclictest1704581-40apt-check12:15:192
1596114993820,13cyclictest1704581-40apt-check12:15:192
1596114993819,14cyclictest34-21ksoftirqd/208:07:562
159611499381,15cyclictest1753-21gnome-shell09:05:522
159611499381,10cyclictest141rcu_preempt09:50:262
159611099382,8cyclictest1753-21gnome-shell10:51:101
1596110993817,17cyclictest1625378-21grep08:30:131
1596110993817,17cyclictest1625378-21grep08:30:121
1596110993815,19cyclictest1650007-21grep09:40:121
1596110993812,22cyclictest1706643-21df12:20:171
1596110993812,22cyclictest1690963-21cat11:35:231
159611099381,10cyclictest438-20systemd-journal08:10:001
159610699387,6cyclictest141rcu_preempt12:24:100
159610699382,8cyclictest1753-21gnome-shell10:25:150
1596106993825,10cyclictest13-21ksoftirqd/012:05:010
1596106993824,10cyclictest13-21ksoftirqd/008:40:120
159610699382,32cyclictest1753-21gnome-shell10:42:460
159610699382,32cyclictest1753-21gnome-shell10:42:460
1596106993822,11cyclictest438-20systemd-journal11:29:240
159610699382,18cyclictest13-21ksoftirqd/007:25:130
159610699382,13cyclictest0-21swapper/008:22:480
159610699382,12cyclictest2197014-21PK-Backend11:45:340
1596106993818,16cyclictest1631088-21netstat08:45:230
159610699381,11cyclictest1678794-21chrt11:01:390
16727582370,18sleep342-21ksoftirqd/310:45:203
162399370,25rtkit-daemon0-21swapper/107:20:321
159611899378,15cyclictest42-21ksoftirqd/311:44:283
159611899373,20cyclictest1753-21gnome-shell10:30:283
159611899372,7cyclictest1596765-21irqstats07:10:223
159611899372,21cyclictest1753-21gnome-shell11:35:233
1596118993713,5cyclictest0-21swapper/312:34:213
1596118993712,7cyclictest1753-21gnome-shell12:01:053
159611899371,17cyclictest141rcu_preempt08:50:303
159611899371,10cyclictest141rcu_preempt09:14:563
1596118993710,6cyclictest1753-21gnome-shell09:38:323
159611499375,15cyclictest141rcu_preempt09:20:232
159611499373,16cyclictest1753-21gnome-shell10:45:292
159611499372,9cyclictest1645107-21/usr/sbin/munin09:25:202
1596114993725,8cyclictest1753-21gnome-shell12:37:422
159611499372,10cyclictest1753-21gnome-shell08:29:592
1596114993720,12cyclictest438-20systemd-journal11:55:002
1596114993719,15cyclictest1753-21gnome-shell11:35:152
1596114993716,13cyclictest34-21ksoftirqd/207:25:182
159611499371,4cyclictest141rcu_preempt07:25:012
159611499371,14cyclictest1613202-21chrt07:55:182
1596110993723,10cyclictest1621391-21sshd08:19:241
1596110993723,10cyclictest1-21systemd09:40:011
1596110993722,11cyclictest1673091-21ntp_kernel_pll_10:45:251
1596110993721,12cyclictest1690097-21/usr/sbin/munin11:35:011
1596110993721,12cyclictest1664862-21sshd10:21:231
1596110993720,12cyclictest438-20systemd-journal09:30:001
1596110993720,12cyclictest1632375-21cat08:50:141
1596110993712,10cyclictest1639700-21expr09:10:121
159610699378,12cyclictest0-21swapper/009:12:040
159610699374,8cyclictest13-21ksoftirqd/011:20:310
1596106993725,9cyclictest13-21ksoftirqd/012:15:020
1596106993724,9cyclictest13-21ksoftirqd/011:35:000
1596106993719,13cyclictest3242255-21gdbus12:40:000
162399360,24rtkit-daemon0-21swapper/111:16:351
162399360,24rtkit-daemon0-21swapper/111:16:341
162399360,23rtkit-daemon0-21swapper/111:04:201
159611899369,6cyclictest141rcu_preempt11:00:053
159611899369,6cyclictest141rcu_preempt10:05:273
159611899362,9cyclictest1753-21gnome-shell08:55:493
159611899362,9cyclictest1753-21gnome-shell08:55:493
1596118993625,7cyclictest1653462-21cpuspeed_turbos09:50:143
159611899362,30cyclictest1753-21gnome-shell11:34:263
1596118993620,11cyclictest1612808-21cat07:55:103
159611899361,31cyclictest2197014-21PK-Backend10:57:423
159611899361,15cyclictest2197014-21PK-Backend12:19:303
159611899361,15cyclictest2197014-21PK-Backend12:19:303
1596118993611,21cyclictest1712964-21unin-run12:40:013
159611499362,9cyclictest1753-21gnome-shell11:07:182
1596114993625,6cyclictest34-21ksoftirqd/212:25:072
1596114993625,6cyclictest34-21ksoftirqd/210:34:332
1596114993625,6cyclictest34-21ksoftirqd/207:50:152
1596114993624,8cyclictest1753-21gnome-shell10:26:592
1596114993623,7cyclictest34-21ksoftirqd/211:12:202
1596114993622,9cyclictest1753-21gnome-shell07:45:092
159611499362,10cyclictest1753-21gnome-shell12:20:172
1596114993618,6cyclictest34-21ksoftirqd/208:21:562
1596114993612,9cyclictest141rcu_preempt10:52:382
1596110993622,10cyclictest1-21systemd11:50:011
1596110993621,11cyclictest1622227-21sed08:20:221
1596110993620,12cyclictest1703813-21sshd12:13:241
1596110993620,12cyclictest1614222-21/usr/sbin/munin08:00:001
1596110993619,13cyclictest1635845-21expr09:00:151
1596110993619,13cyclictest1604323-21cat07:30:241
1596110993619,13cyclictest1604323-21cat07:30:241
159610699367,13cyclictest2197014-21PK-Backend09:55:140
159610699367,13cyclictest2197014-21PK-Backend09:55:140
1596106993610,6cyclictest141rcu_preempt07:51:270
159611899355,6cyclictest141rcu_preempt08:40:213
159611899353,8cyclictest1753-21gnome-shell11:47:413
1596118993524,7cyclictest1687024-21tr11:25:163
1596118993518,12cyclictest1664771-21uptime10:20:293
159611899351,30cyclictest2197014-21PK-Backend08:23:503
1596114993524,6cyclictest34-21ksoftirqd/211:55:132
1596114993519,11cyclictest34-21ksoftirqd/207:10:292
1596114993516,14cyclictest1637143-21pool-org.gnome.09:05:032
1596114993513,17cyclictest3242254-21gmain08:55:012
159611499351,18cyclictest1753-21gnome-shell10:06:402
159611099354,27cyclictest1654767-21apt09:55:001
159611099352,5cyclictest0-21swapper/108:55:281
159611099352,5cyclictest0-21swapper/108:55:271
1596110993521,10cyclictest1698537-21sshd11:57:241
1596110993520,11cyclictest1680839-21gpgv11:10:021
1596110993519,12cyclictest1700983-21expr12:05:151
1596110993519,12cyclictest1657068-21expr10:00:151
1596110993519,12cyclictest1657068-21expr10:00:141
1596110993518,12cyclictest468-21systemd-udevd12:38:501
159611899343,14cyclictest1753-21gnome-shell10:40:443
159611899343,14cyclictest1753-21gnome-shell10:40:443
1596118993422,8cyclictest1753-21gnome-shell10:28:353
159611899341,11cyclictest0-21swapper/309:27:413
1596118993410,11cyclictest141rcu_preempt10:50:023
159611499341,11cyclictest141rcu_preempt10:20:002
159611099349,21cyclictest1676038-21expr10:55:131
1596110993421,10cyclictest1685933-21/usr/sbin/munin11:20:281
1596110993419,11cyclictest1704795-21cat12:15:141
1596110993419,11cyclictest1704795-21cat12:15:141
1596110993419,11cyclictest1641544-21/usr/sbin/munin09:15:191
1596110993419,11cyclictest1611992-21cat07:50:321
1596110993418,12cyclictest1662595-21cat10:15:221
1596110993418,12cyclictest1624173-21grep08:25:231
1596110993418,11cyclictest1627096-21tr08:35:111
1596110993417,13cyclictest1597015-21ntpq07:10:261
1596118993318,11cyclictest42-21ksoftirqd/312:08:243
159611899331,4cyclictest2197014-21PK-Backend10:35:303
159611899326,21cyclictest1598074-21wc07:15:053
159611899323,25cyclictest16051312chrt07:34:353
159611899323,25cyclictest16051312chrt07:34:353
1596114993225,4cyclictest2197014-21PK-Backend10:57:002
1596110993216,12cyclictest1631026-21sed08:45:221
159611099321,4cyclictest0-21swapper/109:21:251
1596110993214,14cyclictest1824-21JS14:31:441
1596110993117,10cyclictest1651919-21expr09:45:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional