You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 11:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Sep 16, 2025 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991080,10rtkit-daemon1753-21gnome-shell22:17:151
16170552870,8sleep22197014-21PK-Backend19:25:162
162399720,63rtkit-daemon0-21swapper/219:07:192
161008127139,26sleep00-21swapper/019:08:410
161043799701,38cyclictest1711647-21cut23:55:020
161043799701,31cyclictest1664025-21latency_hist21:40:020
160870627043,19sleep30-21swapper/319:05:103
161044899692,15cyclictest1753-21gnome-shell22:55:352
161043799662,41cyclictest1753-21gnome-shell20:45:120
161043799652,45cyclictest1753-21gnome-shell00:39:000
161043799652,37cyclictest1753-21gnome-shell22:58:410
161043799652,35cyclictest1753-21gnome-shell22:10:150
1610437996414,29cyclictest1618308-21idleruntime-cro19:30:000
161043799632,37cyclictest1753-21gnome-shell20:32:080
1610437996322,36cyclictest1707202-21grep23:40:240
16805412620,7chrt2197014-21PK-Backend22:25:232
16805412620,7chrt2197014-21PK-Backend22:25:222
161044899622,10cyclictest1753-21gnome-shell23:20:382
161043799622,56cyclictest1753-21gnome-shell19:47:010
161043799622,35cyclictest875-21polkitd23:13:070
1610437996221,36cyclictest1699888-21sed23:20:230
162399610,47rtkit-daemon26-21ksoftirqd/121:28:141
161043799611,35cyclictest1614534-21/usr/sbin/munin19:20:000
16344182600,7sleep22197014-21PK-Backend20:15:152
17211032590,5sleep3411rcuc/300:20:233
17211032590,5sleep3411rcuc/300:20:233
161044899592,16cyclictest1753-21gnome-shell19:45:292
161043799592,33cyclictest1753-21gnome-shell22:07:370
161043799592,32cyclictest1753-21gnome-shell22:22:160
161043799592,30cyclictest1623510-21munin-run19:44:590
161043799591,35cyclictest1687071-21expr22:45:150
161043799591,33cyclictest1662292-21munin-run21:35:000
161043799591,32cyclictest854-21avahi-daemon21:26:310
161043799591,31cyclictest1671166-21latency_hist22:00:020
160992325939,13sleep10-21swapper/119:06:411
161045299589,30cyclictest1753-21gnome-shell22:28:173
161045299589,30cyclictest1753-21gnome-shell22:28:173
161043799582,33cyclictest1753-21gnome-shell00:10:340
161043799582,32cyclictest1753-21gnome-shell20:54:440
161043799572,32cyclictest1685486-21diskstats22:40:180
161043799572,31cyclictest1753-21gnome-shell23:29:180
161043799571,32cyclictest1699266-21latency_hist23:20:030
161043799562,31cyclictest1720444-21/usr/sbin/munin00:20:230
161043799562,31cyclictest1720444-21/usr/sbin/munin00:20:230
161043799561,31cyclictest1684344-21cat22:35:320
161045299555,22cyclictest1667936-21tr21:50:133
161043799553,29cyclictest1753-21gnome-shell21:04:170
161043799553,29cyclictest1753-21gnome-shell21:04:160
161043799552,48cyclictest1753-21gnome-shell21:10:010
1610437995522,28cyclictest1679873-21/usr/sbin/munin22:25:310
1610437995522,28cyclictest1679873-21/usr/sbin/munin22:25:310
1610437995520,31cyclictest1753-21gnome-shell19:33:430
1610437995520,31cyclictest1753-21gnome-shell19:33:430
161043799551,31cyclictest1714289-21ntp_states00:00:280
161043799551,31cyclictest1628967-21apt20:00:130
1610452995425,9cyclictest1753-21gnome-shell23:55:283
1610452995420,8cyclictest1648845-21sed20:55:213
1610452995419,11cyclictest1643247-21smartctl20:40:213
1610452995418,9cyclictest1753-21gnome-shell19:20:013
161043799542,46cyclictest1753-21gnome-shell20:40:310
1610437995424,26cyclictest1069-21runrttasks21:21:210
161043799542,30cyclictest1708682-21diskstats23:45:160
1610437995419,30cyclictest1665609-21sshd21:43:150
1610437995419,30cyclictest1611548-21ntpq19:10:260
1610437995417,32cyclictest1682076-21if_enp2s022:30:210
1610437995414,35cyclictest1688913-21expr22:50:160
1610437995414,35cyclictest1655595-21sshd21:15:160
16634072530,5chrt0-21swapper/121:35:281
1610452995321,8cyclictest1753-21gnome-shell23:37:393
1610452995319,10cyclictest1659779-21ntpq21:25:243
1610452995315,11cyclictest1672682-21cron22:05:013
1610452995314,12cyclictest908-21gdbus23:13:063
161043799532,26cyclictest0-21swapper/000:05:140
1610437995322,27cyclictest1692014-21/usr/sbin/munin23:00:220
1610437995315,33cyclictest1637788-21grep20:25:140
1610437995310,37cyclictest438-20systemd-journal19:24:590
161045299529,18cyclictest1753-21gnome-shell19:25:483
1610452995219,13cyclictest1753-21gnome-shell20:15:193
1610452995216,12cyclictest438-20systemd-journal20:04:593
1610452995215,11cyclictest1706427-21readlink23:40:043
1610452995210,16cyclictest1753-21gnome-shell23:05:473
1610452995210,16cyclictest1753-21gnome-shell21:19:303
161043799522,29cyclictest1753-21gnome-shell19:51:140
161043799522,29cyclictest1753-21gnome-shell19:51:140
161043799522,28cyclictest1753-21gnome-shell19:58:540
161043799522,28cyclictest1753-21gnome-shell19:58:540
161043799522,26cyclictest1706036-21munin-run23:40:020
1610437995219,28cyclictest1676621-21cpuspeed_turbos22:15:150
1610437995218,30cyclictest1069-21runrttasks20:40:020
1610437995218,30cyclictest1069-21runrttasks20:40:020
161043799521,47cyclictest1667336-21seq21:48:140
161043799521,47cyclictest1667336-21seq21:48:130
1610437995212,36cyclictest1722388-21cpuspeed_turbos00:25:140
1610437995212,35cyclictest1631119-21basename20:05:230
1610452995121,7cyclictest1725399-21dump-pmu-power00:35:003
161045299512,12cyclictest1666410-21diskstats21:45:173
161045299512,12cyclictest1666410-21diskstats21:45:173
1610452995113,14cyclictest1632-21dbus-daemon21:15:013
161045299511,24cyclictest1753-21gnome-shell23:48:563
1610437995119,28cyclictest1620741-21if_enp2s019:35:200
1610437995114,32cyclictest1634172-21cpu20:15:140
1610437995113,33cyclictest1719285-21wc00:15:230
161043799511,28cyclictest1635900-21apt20:20:110
161045299502,27cyclictest1753-21gnome-shell21:00:473
161045299502,27cyclictest1753-21gnome-shell21:00:463
1610452995014,14cyclictest1612017-40apt-check19:12:133
161043799502,27cyclictest1712135-21df23:55:140
161043799502,25cyclictest1668517-21ntp_states21:50:270
1610437995021,25cyclictest1069-21runrttasks21:14:230
1610437995019,27cyclictest1069-21runrttasks23:30:250
161045299493,23cyclictest1753-21gnome-shell22:43:153
161045299492,7cyclictest1753-21gnome-shell00:15:213
1610452994923,7cyclictest1716742-21munin-run00:10:003
1610452994920,7cyclictest1069-21runrttasks23:25:323
1610452994916,10cyclictest1703057-21ls23:30:113
161045299491,26cyclictest854-21avahi-daemon23:51:163
161045299491,25cyclictest1753-21gnome-shell22:18:173
161044899492,42cyclictest1753-21gnome-shell23:36:092
1610448994918,27cyclictest2197014-21PK-Backend20:56:322
1610448994917,27cyclictest1658882-21sed21:25:022
161044899491,43cyclictest3242255-21gdbus21:05:012
161044899491,43cyclictest3242255-21gdbus21:05:012
1610448994914,31cyclictest908-21gdbus21:25:362
1610437994916,29cyclictest1724454-21latency00:30:230
1610437994910,35cyclictest1753-21gnome-shell22:02:070
161045299482,25cyclictest1753-21gnome-shell19:40:213
161045299482,25cyclictest1753-21gnome-shell00:25:283
1610452994820,8cyclictest1753-21gnome-shell20:49:143
1610452994819,7cyclictest1069-21runrttasks20:38:273
1610452994819,7cyclictest1069-21runrttasks20:38:273
1610452994818,7cyclictest2197014-21PK-Backend20:06:343
1610452994815,12cyclictest42-21ksoftirqd/319:54:503
1610452994815,12cyclictest42-21ksoftirqd/319:54:503
161045299481,25cyclictest854-21avahi-daemon23:24:593
161045299481,24cyclictest1753-21gnome-shell20:53:563
161044899489,15cyclictest1753-21gnome-shell19:19:432
1610448994819,25cyclictest2197014-21PK-Backend20:50:572
161043799487,37cyclictest1753-21gnome-shell20:10:030
1610437994813,31cyclictest1694494-21awk23:05:260
161045299479,16cyclictest1753-21gnome-shell22:23:593
161045299473,16cyclictest0-21swapper/321:40:163
1610452994711,14cyclictest42-21ksoftirqd/321:35:133
1610452994710,17cyclictest1753-21gnome-shell00:03:263
161044899479,17cyclictest1753-21gnome-shell23:55:482
1610448994718,25cyclictest1611225-21if_enp2s019:10:202
1610448994716,27cyclictest2197014-21PK-Backend20:34:022
1610448994716,26cyclictest1753-21gnome-shell19:50:192
1610448994716,26cyclictest1753-21gnome-shell19:50:182
1610448994710,17cyclictest1753-21gnome-shell21:06:022
1610448994710,17cyclictest1753-21gnome-shell20:10:142
1610448994710,12cyclictest2181-21gdbus00:20:012
17087932460,13sleep21753-21gnome-shell23:45:212
161045299462,8cyclictest1753-21gnome-shell23:15:433
161045299462,24cyclictest1753-21gnome-shell21:20:403
161045299462,23cyclictest1753-21gnome-shell19:48:473
1610452994618,8cyclictest1627724-21kernelversion19:55:213
1610452994618,8cyclictest1627724-21kernelversion19:55:213
1610452994617,10cyclictest42-21ksoftirqd/322:54:553
161044899469,17cyclictest1753-21gnome-shell23:30:372
161044899468,18cyclictest1753-21gnome-shell00:34:262
161044899468,17cyclictest1753-21gnome-shell19:56:342
161044899468,17cyclictest1753-21gnome-shell19:56:342
161044899462,7cyclictest1753-21gnome-shell21:55:322
161044899462,40cyclictest3242253-21nautilus20:10:022
1610448994615,27cyclictest1707277-21ntp_states23:40:262
1610448994614,28cyclictest908-21gdbus23:13:062
1610448994610,17cyclictest1753-21gnome-shell20:04:042
1610448994610,17cyclictest1753-21gnome-shell19:41:092
1610448994610,16cyclictest1753-21gnome-shell22:42:532
1610448994610,16cyclictest1753-21gnome-shell21:42:292
1610437994615,27cyclictest1648789-21if_enp2s020:55:200
16885342450,8sleep2331rcuc/222:50:022
161045299451,23cyclictest1753-21gnome-shell22:05:193
161045299451,22cyclictest1753-21gnome-shell22:37:593
161044899459,16cyclictest1753-21gnome-shell00:07:152
1610448994518,22cyclictest28750irq/122-ahci[0023:19:552
1610448994517,24cyclictest2197014-21PK-Backend20:26:582
17266212440,7sleep22197014-21PK-Backend00:35:322
16504892440,7chrt1753-21gnome-shell21:00:221
16504892440,7chrt1753-21gnome-shell21:00:211
162399440,29rtkit-daemon0-21swapper/100:35:131
1610452994420,7cyclictest1681776-21cpuspeed_turbos22:30:153
161044899442,16cyclictest1753-21gnome-shell00:05:022
1610448994414,25cyclictest311irq_work/222:39:042
1610448994412,27cyclictest1666645-21missed_timers21:45:242
1610448994412,27cyclictest1666645-21missed_timers21:45:242
161045299431,8cyclictest1620950-21chrt19:35:233
1610452994312,8cyclictest0-21swapper/300:13:423
161044899438,13cyclictest1753-21gnome-shell23:00:142
161044899437,16cyclictest1753-21gnome-shell20:35:202
161044899437,16cyclictest1753-21gnome-shell20:35:192
161044899433,18cyclictest1753-21gnome-shell21:17:202
161044899432,7cyclictest1753-21gnome-shell22:35:022
161044899432,17cyclictest1753-21gnome-shell21:30:282
1610448994312,26cyclictest0-21swapper/221:35:432
161044899431,16cyclictest1674830-21tr22:10:152
161044899431,14cyclictest1702295-21chrt23:27:222
1610448994311,28cyclictest1672109-21python322:00:322
1610444994327,12cyclictest1656108-21cut21:15:251
161045299421,22cyclictest1670898-21chrt21:59:243
1610452994210,14cyclictest854-21avahi-daemon19:20:093
161044899425,13cyclictest1632-21dbus-daemon22:50:012
161044899422,14cyclictest860997-21snmpd22:06:282
161044899422,13cyclictest1669230-21munin-run21:55:002
161044899421,14cyclictest1694157-21taskset23:05:202
1610448994210,13cyclictest1753-21gnome-shell19:30:092
1610448994210,13cyclictest1753-21gnome-shell19:30:082
1610444994233,6cyclictest64850irq/126-i91523:29:501
1610444994226,13cyclictest1626231-21grep19:50:251
1610444994226,13cyclictest1626231-21grep19:50:251
1610444994222,15cyclictest3242253-21nautilus19:20:031
161044899412,12cyclictest1620831-21latency19:35:222
1610448994114,23cyclictest2197014-21PK-Backend19:24:112
1610448994114,23cyclictest1720639-21cut00:20:152
1610448994114,23cyclictest1720639-21cut00:20:152
1610448994114,23cyclictest1711631-21latency_hist23:55:022
1610448994112,25cyclictest0-21swapper/221:12:232
1610444994129,8cyclictest1713728-21cut00:00:161
1610444994128,9cyclictest1678373-21cstates22:20:141
161044499411,28cyclictest0-21swapper/120:12:301
1610448994016,21cyclictest17221682sleep200:25:062
161044899401,15cyclictest1753-21gnome-shell20:45:012
1610444994023,13cyclictest1705669-21fwupdmgr23:36:061
1610444994020,16cyclictest1687727-21/usr/sbin/munin22:45:311
1610444994019,17cyclictest1723051-21processes00:25:271
1610444994014,13cyclictest1665053-21ntp_states21:40:271
1610452993919,15cyclictest3242253-21nautilus20:15:023
1610448993911,25cyclictest2197014-21PK-Backend22:20:002
161044499392,33cyclictest1753-21gnome-shell21:53:491
161044499392,33cyclictest1753-21gnome-shell20:10:001
1610444993922,14cyclictest1622762-21/usr/sbin/munin19:40:251
1610444993922,12cyclictest1648336-21cat20:55:111
1610444993919,16cyclictest1825-21JS14:31:441
1610444993919,15cyclictest1632-21dbus-daemon23:50:001
1610444993912,11cyclictest1646545-21acpi20:50:151
162399380,24rtkit-daemon1753-21gnome-shell20:38:341
162399380,24rtkit-daemon1753-21gnome-shell20:38:341
161045299389,13cyclictest1827-21JS14:31:443
1610452993823,11cyclictest1637357-21cron20:25:003
161045299381,16cyclictest1687049-21cstates22:45:153
161044899382,14cyclictest854-21avahi-daemon20:46:122
161044899381,13cyclictest1637421-21sed20:25:012
1610444993824,10cyclictest26-21ksoftirqd/121:45:131
1610444993824,10cyclictest26-21ksoftirqd/121:45:121
161044499382,32cyclictest1753-21gnome-shell22:35:441
161044499382,32cyclictest1753-21gnome-shell19:13:041
1610444993823,11cyclictest26-21ksoftirqd/100:05:571
1610444993815,20cyclictest1717310-21expr00:10:171
161044499381,34cyclictest0-21swapper/119:55:371
161044499381,34cyclictest0-21swapper/119:55:361
162399370,25rtkit-daemon0-21swapper/119:37:521
161044499378,13cyclictest1639227-21sed20:30:021
1610444993722,11cyclictest1704240-21cron23:35:001
1610444993720,12cyclictest1617772-21cat19:25:321
1610444993719,14cyclictest1-21systemd23:05:001
1610444993710,12cyclictest1645332-21open_files20:45:281
162399360,26rtkit-daemon0-21swapper/121:57:591
1610452993616,16cyclictest3242253-21nautilus23:05:013
1610452993611,20cyclictest1638012-21sh20:25:163
1610444993621,11cyclictest1721188-21ntpq00:20:251
1610444993621,11cyclictest1721188-21ntpq00:20:251
1610444993620,12cyclictest855-21dbus-daemon22:15:001
1610444993620,12cyclictest1719812-21uptime00:15:311
1610444993619,13cyclictest1707141-21cat23:40:241
1610444993619,13cyclictest1660830-21apt21:30:131
1610444993619,13cyclictest1623764-21awk19:45:121
1610444993619,12cyclictest438-20systemd-journal22:10:001
161044499361,31cyclictest1657790-21cat21:20:221
161044499361,30cyclictest904-21systemd-logind23:13:041
162399350,25rtkit-daemon0-21swapper/122:34:121
161045299354,26cyclictest1639507-21tr20:30:113
161045299351,9cyclictest2197014-21PK-Backend21:05:263
1610448993512,7cyclictest1825-21JS14:31:442
161044899351,11cyclictest1718091-21who00:10:342
161044499358,12cyclictest1639455-21cat20:30:111
1610444993524,7cyclictest1629096-21cpuspeed_turbos20:00:151
1610444993523,8cyclictest64850irq/126-i91520:15:151
1610444993519,12cyclictest1698292-21awk23:15:271
1610444993519,11cyclictest438-20systemd-journal23:55:161
1610444993519,11cyclictest438-20systemd-journal21:12:281
1610444993519,11cyclictest1636628-21ntp_kernel_pll_20:20:231
1610444993518,13cyclictest1693946-21cut23:05:171
1610444993518,12cyclictest1685865-21ntpq22:40:281
161045299341,9cyclictest2197014-21PK-Backend21:30:293
1610444993423,7cyclictest1619528-21grep19:30:311
1610444993423,7cyclictest1619528-21grep19:30:301
1610444993420,11cyclictest1725420-21awk00:34:591
1610444993418,12cyclictest1672138-21sed22:00:331
1610444993418,12cyclictest1652362-21grep21:05:211
1610452993323,6cyclictest1618750-21tr19:30:153
1610452993323,6cyclictest1618750-21tr19:30:153
1610452993314,14cyclictest3242253-21nautilus22:15:013
1610444993320,9cyclictest26-21ksoftirqd/123:50:201
1610444993318,11cyclictest1688804-21expr22:50:161
1610444993318,11cyclictest1643203-21df20:40:161
1610444993316,13cyclictest1680872-21sed22:25:311
1610444993316,13cyclictest1680872-21sed22:25:301
1610444993220,9cyclictest1753-21gnome-shell22:59:101
1610444993216,12cyclictest1699516-21cut23:20:151
1610452993119,8cyclictest0-21swapper/322:59:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional