You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 09:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Dec 06, 2025 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16769552940,4sleep10-21swapper/120:55:151
17144702830,5chrt0-21swapper/022:40:310
17517402770,7sleep30-21swapper/300:25:343
163905427344,23sleep00-21swapper/019:07:240
163939699672,45cyclictest1753-21gnome-shell21:05:173
163891826739,22sleep10-21swapper/119:05:411
163820726640,20sleep30-21swapper/319:05:183
163939699652,42cyclictest1753-21gnome-shell22:15:053
163939699651,32cyclictest1673600-21cat20:45:123
163890726546,13sleep20-21swapper/219:05:342
1639396996429,31cyclictest1753-21gnome-shell21:36:453
163939699642,43cyclictest1753-21gnome-shell22:38:583
163939699631,37cyclictest1701884-21/usr/sbin/munin22:05:203
163939699631,36cyclictest1752492-21cat00:30:163
163939699631,35cyclictest860997-21snmpd20:16:153
162399630,9rtkit-daemon1622-21rtkit-daemon21:18:131
163939699622,44cyclictest1753-21gnome-shell21:50:553
163939699622,33cyclictest1753-21gnome-shell20:26:253
163939699621,37cyclictest969-21sshd20:44:403
163939699615,28cyclictest1617-10pipewire-media-23:05:123
163939699611,33cyclictest860997-21snmpd00:00:353
163939699602,33cyclictest1753-21gnome-shell21:20:223
163939699602,33cyclictest1753-21gnome-shell19:45:033
163939699602,31cyclictest1690473-21chrt21:34:493
163939699602,29cyclictest1753-21gnome-shell20:00:303
163939699601,33cyclictest1679835-21taskset21:02:203
163939699601,33cyclictest1679835-21taskset21:02:203
163939699601,32cyclictest854-21avahi-daemon21:15:023
163939699592,32cyclictest1753-21gnome-shell21:10:033
163939699592,30cyclictest1747785-21memory00:15:233
163939699592,30cyclictest1747785-21memory00:15:223
163939699592,29cyclictest1753-21gnome-shell19:15:003
163939699591,39cyclictest0-21swapper/319:55:393
163939699591,35cyclictest1728354-21sed23:20:243
163939699582,39cyclictest1753-21gnome-shell00:12:433
163939699582,33cyclictest854-21avahi-daemon00:00:033
163939699582,33cyclictest854-21avahi-daemon00:00:033
163939699582,30cyclictest1753-21gnome-shell23:10:103
163939699581,32cyclictest1730713-21taskset23:26:423
163939699581,32cyclictest1730713-21taskset23:26:413
163939699581,32cyclictest1688596-21taskset21:28:003
163939699581,31cyclictest860997-21snmpd00:07:323
1639396995811,42cyclictest1753-21gnome-shell23:01:163
163939699572,31cyclictest1753-21gnome-shell21:47:443
163939699572,30cyclictest1753-21gnome-shell23:46:173
163939699572,30cyclictest1753-21gnome-shell22:34:183
163939699572,25cyclictest16983682grep21:55:233
163939699571,34cyclictest1719875-21/usr/sbin/munin22:55:263
163939699571,34cyclictest1069-21runrttasks00:35:233
163939699571,34cyclictest1069-21runrttasks00:35:223
163939699571,30cyclictest1649986-21taskset19:37:073
1639388995720,12cyclictest1744744-21cat00:05:351
163939699562,29cyclictest1753-21gnome-shell22:10:363
163939699562,29cyclictest1753-21gnome-shell20:06:563
163939699552,29cyclictest1753-21gnome-shell20:10:063
163938899552,9cyclictest2181-21gdbus22:50:001
163939699542,27cyclictest1753-21gnome-shell20:52:573
163938899542,25cyclictest1668557-21/usr/sbin/munin20:30:181
1639388995417,11cyclictest1736971-21cat23:45:161
1639388995416,11cyclictest1726972-21sed23:15:311
1639388995416,11cyclictest1656343-21needreboot19:55:221
1639388995413,7cyclictest0-21swapper/100:15:011
163939699532,29cyclictest1753-21gnome-shell22:28:203
163939699532,28cyclictest1753-21gnome-shell19:43:173
163939699531,48cyclictest1665899-21sshd20:21:313
163938899532,15cyclictest1753288-21python300:30:301
162399530,8rtkit-daemon1622-21rtkit-daemon19:57:310
17487672520,7sleep22197014-21PK-Backend00:20:032
17487672520,7sleep22197014-21PK-Backend00:20:032
163939699522,27cyclictest1753-21gnome-shell00:22:343
1639396995221,25cyclictest1700894-21apt-get22:05:023
1639396995221,25cyclictest1700894-21apt-get22:05:013
163939699521,27cyclictest1678540-21latency_hist21:00:023
1639388995216,11cyclictest1693456-21grep21:40:291
1639388995216,10cyclictest1674662-21seq20:47:401
1639388995214,12cyclictest1646058-21awk19:25:251
163939699511,35cyclictest175350gnome-shell23:15:203
163938899511,11cyclictest1655432-21apt19:55:011
1639396995018,28cyclictest1734784-21basename23:40:013
1639396995018,28cyclictest1734784-21basename23:40:003
1639388995019,9cyclictest26-21ksoftirqd/100:17:401
1639388995019,9cyclictest26-21ksoftirqd/100:17:401
1639388995019,11cyclictest1707423-21ntpq22:20:271
1639388995016,11cyclictest1643772-21expr19:20:141
1639388995014,11cyclictest1710582-21seq22:30:211
1639388995014,11cyclictest1663941-21timerandwakeup20:15:301
163938899501,25cyclictest3242255-21gdbus23:45:011
1639396994920,24cyclictest1654740-21open_inodes19:50:273
163939699491,32cyclictest1708263-21cat22:25:023
1639388994917,9cyclictest1699794-21expr22:00:151
1639388994917,9cyclictest1699794-21expr22:00:141
1639388994913,12cyclictest1754152-21apt00:35:141
1639388994913,12cyclictest1754152-21apt00:35:131
162399490,40rtkit-daemon1506-21systemd19:10:260
16668582480,13sleep2331rcuc/220:25:222
163938899482,21cyclictest1652591-21df19:45:161
1639388994818,12cyclictest1690305-21sshd21:32:401
1639388994817,9cyclictest1739127-21missed_timers23:50:211
1639388994817,9cyclictest1739127-21missed_timers23:50:211
1639388994815,10cyclictest1672709-21grep20:40:321
1639388994813,8cyclictest0-21swapper/119:10:281
163938899481,28cyclictest0-21swapper/100:25:381
163938899481,24cyclictest1649630-21python319:35:291
1639385994823,21cyclictest1732067-40lsb_release23:30:230
17163342470,6sleep00-21swapper/022:45:290
163939699473,39cyclictest1753-21gnome-shell19:15:173
1639396994715,28cyclictest1739502-21cat23:50:303
1639396994715,28cyclictest1739502-21cat23:50:303
1639388994720,8cyclictest1753-21gnome-shell22:17:371
1639388994714,11cyclictest1681086-21open_files21:05:271
1639388994714,11cyclictest1671394-21/usr/sbin/munin20:40:021
163938899471,27cyclictest0-21swapper/120:53:171
163938899471,23cyclictest1697899-21cat21:55:131
1639396994617,24cyclictest1670214-21expr20:35:143
1639396994615,25cyclictest1646143-21ps19:25:273
1639396994613,28cyclictest438-20systemd-journal22:42:003
1639396994613,28cyclictest1716291-21cat22:45:293
1639392994619,22cyclictest438-20systemd-journal22:59:562
163938899464,22cyclictest1722726-21cpuspeed_turbos23:05:131
163938899461,21cyclictest1695454-21taskset21:46:591
163938899461,12cyclictest1753-21gnome-shell19:40:561
163939699452,39cyclictest1753-21gnome-shell21:44:083
163938899452,9cyclictest1732130-40lsb_release23:30:231
1639388994519,9cyclictest1753-21gnome-shell23:02:261
163938899451,22cyclictest1753-21gnome-shell23:55:141
163938899451,22cyclictest1753-21gnome-shell23:55:131
1639385994537,5cyclictest0-21swapper/021:34:350
1639385994518,23cyclictest1722351-21unin-run23:05:010
163939699442,38cyclictest1753-21gnome-shell19:20:283
163939699442,37cyclictest1753-21gnome-shell23:40:163
1639396994416,24cyclictest1647991-21sort19:30:303
163939699441,38cyclictest1717888-21sed22:50:223
163938899441,13cyclictest1687032-21ls21:25:011
162399440,30rtkit-daemon0-21swapper/223:30:112
163939699431,37cyclictest1668159-21seq20:30:053
1639392994329,9cyclictest34-21ksoftirqd/200:00:032
1639392994329,9cyclictest34-21ksoftirqd/200:00:032
163938899432,21cyclictest1658014-21memory20:00:241
163938899432,11cyclictest1753-21gnome-shell00:00:241
1639396994211,27cyclictest1753-21gnome-shell23:34:503
1639392994226,12cyclictest1753-21gnome-shell23:01:382
1639388994210,7cyclictest0-21swapper/122:35:221
1639385994213,12cyclictest141rcu_preempt22:30:140
162399420,33rtkit-daemon1506-21systemd21:17:180
16577232410,6sleep02197014-21PK-Backend20:00:150
1639392994122,14cyclictest3242253-21nautilus21:05:012
1639392994122,14cyclictest3242253-21nautilus21:05:012
1639388994124,12cyclictest1647974-21threads19:30:291
163938899411,11cyclictest1665931-21chrt20:21:531
162399410,29rtkit-daemon2197014-21PK-Backend21:40:262
16961652400,18sleep234-21ksoftirqd/221:50:132
1639388994022,13cyclictest438-20systemd-journal22:30:011
1639388994010,8cyclictest0-21swapper/121:39:591
1639385994031,5cyclictest17344042sleep023:35:360
1639385994031,5cyclictest17344042sleep023:35:360
1639385994023,13cyclictest1737137-21df_inode23:45:180
1639385994023,13cyclictest1691168-21fw_packets21:35:190
1639385994023,13cyclictest1686315-21ntp_states21:20:260
1639385994014,6cyclictest141rcu_preempt23:25:000
1639385994013,4cyclictest141rcu_preempt20:25:010
163938599401,13cyclictest1697448-21turbostat21:55:000
163938599401,11cyclictest141rcu_preempt21:55:140
1639385994011,12cyclictest141rcu_preempt00:00:210
16878042390,14sleep1261ksoftirqd/121:25:261
1639392993927,8cyclictest1753-21gnome-shell20:02:282
1639392993921,6cyclictest34-21ksoftirqd/220:33:182
163938599392,9cyclictest1753-21gnome-shell23:12:160
1639385993928,7cyclictest1753-21gnome-shell00:13:320
163938599392,20cyclictest1753-21gnome-shell22:00:590
163938599392,20cyclictest1753-21gnome-shell22:00:590
163938599392,12cyclictest0-21swapper/000:30:330
1639385993920,14cyclictest13-21ksoftirqd/019:47:360
1639385993912,5cyclictest141rcu_preempt20:10:170
162399390,28rtkit-daemon2197014-21PK-Backend22:46:352
16954702382,6sleep21753-21gnome-shell21:47:122
163939299383,8cyclictest1753-21gnome-shell23:15:432
1639392993828,7cyclictest1069-21runrttasks21:07:172
1639392993828,6cyclictest1655433-21idleruntime-cro19:55:002
163939299382,32cyclictest1753-21gnome-shell00:21:232
1639392993820,13cyclictest34-21ksoftirqd/200:35:362
1639392993820,13cyclictest34-21ksoftirqd/200:35:362
1639392993814,6cyclictest141rcu_preempt23:25:262
1639392993814,6cyclictest141rcu_preempt23:25:262
163939299381,12cyclictest141rcu_preempt22:05:262
163938599387,5cyclictest141rcu_preempt20:58:520
163938599384,12cyclictest141rcu_preempt20:35:210
163938599382,7cyclictest1753-21gnome-shell19:28:530
1639385993821,13cyclictest13-21ksoftirqd/021:14:210
1639385993811,5cyclictest141rcu_preempt20:05:210
163938599381,15cyclictest13581kworker/u9:120:30:270
162399380,27rtkit-daemon2197014-21PK-Backend20:14:502
163939299379,17cyclictest1753-21gnome-shell22:40:182
163939299377,9cyclictest34-21ksoftirqd/223:52:202
163939299377,9cyclictest34-21ksoftirqd/223:52:202
1639392993722,10cyclictest1692192-21sshd21:39:402
1639392993720,13cyclictest1689789-21ntp_states21:30:262
1639392993720,12cyclictest855-21dbus-daemon20:50:012
1639392993720,12cyclictest1686316-21sh21:20:252
163939299371,5cyclictest141rcu_preempt22:28:312
1639392993714,6cyclictest141rcu_preempt22:38:532
1639388993728,6cyclictest1753-21gnome-shell22:06:291
163938899372,7cyclictest1753-21gnome-shell23:39:031
163938899372,7cyclictest1753-21gnome-shell23:39:021
1639388993720,13cyclictest1718117-21open_inodes22:50:281
163938599376,9cyclictest1069-21runrttasks23:44:360
163938599376,13cyclictest141rcu_preempt22:13:430
163938599376,10cyclictest13-21ksoftirqd/019:44:090
163938599375,9cyclictest141rcu_preempt00:35:260
163938599375,9cyclictest141rcu_preempt00:35:260
163938599372,8cyclictest1753-21gnome-shell23:58:540
163938599372,8cyclictest1753-21gnome-shell23:58:540
163938599372,7cyclictest1753-21gnome-shell23:29:050
163938599372,7cyclictest1753-21gnome-shell23:29:040
1639385993726,7cyclictest1753-21gnome-shell19:24:130
1639385993726,6cyclictest16418262sleep019:15:120
1639385993722,11cyclictest1748652-21seq00:20:000
1639385993722,11cyclictest1748652-21seq00:20:000
1639385993721,12cyclictest1739215-21ntp_kernel_pll_23:50:240
1639385993721,12cyclictest1739215-21ntp_kernel_pll_23:50:240
1639385993720,12cyclictest1649509-21ntpq19:35:250
1639392993626,6cyclictest1069-21runrttasks23:46:552
1639392993625,7cyclictest1742210-21cstates00:00:162
1639392993620,12cyclictest1706675-21awk22:20:152
1639392993620,12cyclictest1705658-21ntpq22:15:272
1639392993620,12cyclictest1672386-21ntp_kernel_pll_20:40:252
1639392993618,14cyclictest1650911-21idleruntime19:40:212
1639392993618,13cyclictest1632-21dbus-daemon20:05:042
163939299361,32cyclictest1753-21gnome-shell00:12:192
1639388993625,7cyclictest1730985-21sh23:30:001
1639388993625,7cyclictest1730985-21sh23:30:001
163938599369,6cyclictest141rcu_preempt19:30:110
1639385993625,7cyclictest1671785-21cpuspeed_turbos20:40:150
1639385993619,13cyclictest1632-21dbus-daemon22:30:000
1639385993618,13cyclictest13-21ksoftirqd/020:53:280
1639385993617,14cyclictest1707690-21cat22:20:330
1639385993613,6cyclictest141rcu_preempt23:17:150
162399360,26rtkit-daemon0-21swapper/219:19:522
162399360,24rtkit-daemon0-21swapper/220:23:032
1639392993521,10cyclictest34-21ksoftirqd/223:20:192
1639392993521,10cyclictest34-21ksoftirqd/222:50:082
1639392993520,11cyclictest855-21dbus-daemon20:40:002
1639392993520,11cyclictest1735155-21sort23:40:132
1639392993520,11cyclictest1653810-21wc19:50:012
1639392993519,12cyclictest34-21ksoftirqd/200:09:402
163939299351,8cyclictest2197014-21PK-Backend22:00:132
163939299351,8cyclictest2197014-21PK-Backend22:00:132
1639392993518,13cyclictest34-21ksoftirqd/221:12:552
1639392993518,12cyclictest438-20systemd-journal21:26:032
1639392993517,13cyclictest438-20systemd-journal20:00:012
163939299351,30cyclictest1734273-21sed23:35:312
163939299351,30cyclictest1734273-21sed23:35:312
1639392993512,6cyclictest141rcu_preempt19:20:452
163938899355,4cyclictest141rcu_preempt23:20:201
1639388993520,11cyclictest1749503-21cat00:20:211
1639388993519,12cyclictest1660609-21seq20:06:241
1639388993516,14cyclictest1-21systemd19:20:011
163938599352,8cyclictest1753-21gnome-shell22:16:280
163938599352,8cyclictest1753-21gnome-shell20:19:160
1639385993524,7cyclictest1753-21gnome-shell19:53:510
1639385993522,9cyclictest13-21ksoftirqd/021:46:430
1639385993520,11cyclictest1743972-21cat00:05:150
1639385993519,12cyclictest1666768-21cat20:25:160
163938599351,8cyclictest1674260-21python320:45:260
163939299344,14cyclictest0-21swapper/219:14:402
1639392993426,5cyclictest2197014-21PK-Backend23:11:582
1639392993423,7cyclictest1674900-21kworker/2:222:35:012
1639392993417,13cyclictest1703956-21ntp_states22:10:262
1639392993417,13cyclictest1664132-21seq20:16:092
1639392993416,6cyclictest34-21ksoftirqd/221:55:102
1639392993416,13cyclictest2631682-21snapd19:30:182
1639388993418,11cyclictest1724719-21hddtemp_smartct23:10:181
1639388993418,11cyclictest1682581-21grep21:10:211
163938599344,26cyclictest1723264-21grep23:05:230
163938599342,6cyclictest141rcu_preempt21:00:130
163938599342,6cyclictest141rcu_preempt21:00:130
163938599342,27cyclictest1751078-21hddtemp_smartct00:25:230
163938599341,9cyclictest2197014-21PK-Backend00:21:340
1639385993419,11cyclictest1719833-21awk22:55:250
1639385993419,11cyclictest1712967-21awk22:35:320
1639385993418,12cyclictest1693585-21users21:40:290
163939299332,11cyclictest34-21ksoftirqd/220:55:482
1639392993320,9cyclictest1753-21gnome-shell20:52:192
1639392993314,14cyclictest438-20systemd-journal23:08:412
163939299331,29cyclictest2197014-21PK-Backend21:17:312
1639392993310,19cyclictest1753-21gnome-shell19:36:032
163938899338,6cyclictest17037622grep22:10:221
163938899338,21cyclictest1719461-21cat22:55:161
163938899336,10cyclictest26-21ksoftirqd/122:42:441
1639388993319,10cyclictest1662202-21sed20:10:351
1639388993318,11cyclictest1667615-21seq20:26:201
163938599331,13cyclictest2197014-21PK-Backend22:51:380
163938599331,10cyclictest0-21swapper/022:05:300
17509122320,7chrt2197014-21PK-Backend00:25:172
1639392993216,10cyclictest1752136-21cron00:30:022
163939299321,4cyclictest2197014-21PK-Backend19:26:302
163938599324,11cyclictest141rcu_preempt21:29:140
163938599322,9cyclictest13-21ksoftirqd/021:08:260
1639388993120,7cyclictest1696483-21cut21:50:211
1639388993115,11cyclictest1678918-21expr21:00:161
1639388993115,11cyclictest1678918-21expr21:00:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional