You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-05 - 21:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Sat Jul 05, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
221228521080,5sleep00-21swapper/010:05:150
226275721000,4sleep00-21swapper/012:25:300
162399960,78rtkit-daemon0-21swapper/307:05:473
214987828362,14sleep00-21swapper/007:06:460
22432692780,5sleep0161rcuc/011:30:230
214994227840,14sleep20-21swapper/207:07:372
22039802710,7chrt1824-21JS14:31:441
214866826945,15sleep10-21swapper/107:05:141
162399690,9rtkit-daemon1622-21rtkit-daemon08:00:260
2150284996524,12cyclictest2238264-21grep11:15:283
162399650,7rtkit-daemon1622-21rtkit-daemon08:45:211
22269392640,6sleep10-21swapper/110:45:251
2150284996422,12cyclictest2178863-21expr08:30:153
2150270996417,11cyclictest2237894-21df_inode11:15:200
162399640,10rtkit-daemon1622-21rtkit-daemon08:15:560
21901682630,8sleep21753-21gnome-shell09:00:252
21719582610,7sleep32197014-21PK-Backend08:10:153
162399610,9rtkit-daemon1622-21rtkit-daemon09:57:590
2150284995923,10cyclictest1753-21gnome-shell09:50:213
162399590,8rtkit-daemon1622-21rtkit-daemon08:39:370
2150284995822,10cyclictest2164241-21seq07:45:343
2150284995720,12cyclictest2206512-21systemd-udevd09:49:523
162399570,9rtkit-daemon1622-21rtkit-daemon07:47:521
162399570,8rtkit-daemon1622-21rtkit-daemon08:10:081
2150284995615,20cyclictest1753-21gnome-shell08:17:443
2150270995618,11cyclictest2193330-21smartctl09:10:200
162399560,9rtkit-daemon1622-21rtkit-daemon08:27:071
2150284995521,9cyclictest1753-21gnome-shell11:32:363
2150284995520,7cyclictest1069-21runrttasks09:30:463
2150284995515,14cyclictest2201986-21df09:35:173
2150284995514,14cyclictest2155055-21grep07:20:263
162399550,9rtkit-daemon1622-21rtkit-daemon08:33:591
162399550,9rtkit-daemon1622-21rtkit-daemon08:08:361
162399550,8rtkit-daemon1622-21rtkit-daemon10:00:131
215028499542,26cyclictest1753-21gnome-shell10:04:043
215028499542,26cyclictest1753-21gnome-shell09:55:293
215028499542,26cyclictest1753-21gnome-shell09:29:383
215028499542,26cyclictest1753-21gnome-shell07:40:243
215028499542,26cyclictest1753-21gnome-shell07:40:243
215028499542,25cyclictest1753-21gnome-shell10:40:083
2150270995418,13cyclictest2171227-21seq08:06:170
162399540,9rtkit-daemon1622-21rtkit-daemon07:52:261
21815582530,5sleep20-21swapper/208:35:332
215028499538,19cyclictest1753-21gnome-shell11:07:203
215028499538,19cyclictest1753-21gnome-shell11:07:203
215028499532,8cyclictest1753-21gnome-shell07:59:493
215028499532,26cyclictest1753-21gnome-shell10:34:513
215028499532,25cyclictest1753-21gnome-shell11:39:323
2150284995321,8cyclictest1753-21gnome-shell11:23:483
2150284995321,8cyclictest1753-21gnome-shell11:23:483
215028499532,12cyclictest2227113-21cat10:45:303
2150284995320,8cyclictest2159622-21apt07:35:133
2150284995310,18cyclictest1753-21gnome-shell07:16:343
2150284995310,17cyclictest1753-21gnome-shell09:41:423
2150270995321,28cyclictest2211755-21cron10:05:000
2150270995317,12cyclictest2221360-21entropy10:30:200
2150270995316,14cyclictest2231469-40apt-check10:57:190
2150270995313,12cyclictest875-21polkitd09:49:530
162399530,8rtkit-daemon1622-21rtkit-daemon10:09:241
22179962520,5sleep00-21swapper/010:20:280
215028499522,8cyclictest1753-21gnome-shell09:05:433
215028499522,8cyclictest1753-21gnome-shell09:05:423
215028499522,8cyclictest1753-21gnome-shell08:22:463
215028499522,8cyclictest1753-21gnome-shell08:22:463
215028499522,7cyclictest1753-21gnome-shell12:10:123
215028499522,7cyclictest1753-21gnome-shell12:10:123
215028499522,7cyclictest1753-21gnome-shell12:07:423
215028499522,25cyclictest2241395-21diskstats11:25:183
215028499522,25cyclictest1753-21gnome-shell08:09:563
215028499522,24cyclictest1753-21gnome-shell08:51:533
215028499522,24cyclictest1753-21gnome-shell07:51:203
215028499522,24cyclictest1753-21gnome-shell07:10:573
2150284995218,8cyclictest2189839-40lsb_release09:00:203
2150270995225,23cyclictest2162064-21needreboot07:40:240
2150270995225,23cyclictest2162064-21needreboot07:40:240
215028499512,27cyclictest1753-21gnome-shell08:27:513
215028499511,8cyclictest1753-21gnome-shell09:17:033
215028499511,24cyclictest1753-21gnome-shell07:33:383
2150270995118,7cyclictest2179692-21timerandwakeup08:30:310
2150270995118,12cyclictest2248672-21ntpq11:45:280
2150270995116,11cyclictest2246576-21expr11:40:150
2150270995115,13cyclictest2185923-21expr08:50:160
2150270995113,11cyclictest2235249-21ls11:10:020
2150270995113,11cyclictest2235249-21ls11:10:020
2150284995019,10cyclictest42-21ksoftirqd/308:59:113
215028499501,24cyclictest1753-21gnome-shell11:58:563
2150270995024,13cyclictest2266779-21apt-get12:40:000
2150270995015,8cyclictest2197897-21awk09:25:000
2150270995012,11cyclictest2239740-21if_err_enp2s011:20:200
2150270995012,11cyclictest2239740-21if_err_enp2s011:20:200
21630582490,14sleep234-21ksoftirqd/207:45:022
21630582490,14sleep234-21ksoftirqd/207:45:022
215028499492,8cyclictest1753-21gnome-shell11:44:093
2150284994916,10cyclictest42-21ksoftirqd/311:00:253
215028499491,24cyclictest1753-21gnome-shell12:27:493
215027099497,18cyclictest1753-21gnome-shell09:30:210
2150270994914,11cyclictest855-21dbus-daemon07:15:010
2150270994914,10cyclictest2214932-21sshd10:11:210
2150270994913,9cyclictest2231988-21/usr/sbin/munin11:00:310
215028499482,25cyclictest1753-21gnome-shell10:35:453
215028499482,24cyclictest1753-21gnome-shell10:08:333
215027099489,12cyclictest2156946-21proc_pri07:25:290
2150270994818,8cyclictest2197014-21PK-Backend09:36:170
2150270994813,13cyclictest908-21gdbus12:22:010
215028499472,8cyclictest1753-21gnome-shell12:21:153
2150284994718,20cyclictest42-21ksoftirqd/312:37:403
215028099472,16cyclictest860997-21snmpd09:15:542
215027099472,41cyclictest0-21swapper/007:51:050
2150270994716,12cyclictest2198742-21wc09:25:250
2150270994715,8cyclictest2255498-21missed_timers12:05:240
2150270994714,7cyclictest2216155-21needreboot10:15:250
2150270994714,11cyclictest2228591-21forks10:50:210
2150270994713,11cyclictest2152935-21expr07:15:160
215027099471,26cyclictest2259370-21sed12:15:320
162399470,41rtkit-daemon2287-21gmain09:17:531
22635212460,16sleep234-21ksoftirqd/212:30:022
22528492460,10sleep234-21ksoftirqd/211:59:562
21755482460,6chrt2197014-21PK-Backend08:20:172
21755482460,6chrt2197014-21PK-Backend08:20:162
215027099468,18cyclictest1753-21gnome-shell08:57:310
2150270994619,7cyclictest1069-21runrttasks09:08:130
2150270994619,7cyclictest1069-21runrttasks09:08:130
2150270994618,9cyclictest1753-21gnome-shell08:24:170
2150270994618,9cyclictest1753-21gnome-shell08:24:170
2150270994616,7cyclictest2219359-21forks10:25:210
2150270994614,8cyclictest2155031-21ls07:20:270
2150270994614,7cyclictest2167103-21kernelversion07:55:230
2150270994614,11cyclictest2171973-21tr08:10:150
2150270994613,9cyclictest2207922-21uptime09:50:310
2150270994613,7cyclictest2203456-21cpuspeed_turbos09:40:150
2150270994612,8cyclictest0-21swapper/011:25:300
2150270994611,12cyclictest2159334-21seq07:34:580
21971302450,7sleep22197014-21PK-Backend09:20:252
215027099459,16cyclictest1753-21gnome-shell10:44:310
215027099459,15cyclictest1753-21gnome-shell10:35:230
215027099459,10cyclictest2226325-21sort10:45:140
2150270994517,8cyclictest2197014-21PK-Backend12:12:420
2150270994517,8cyclictest2197014-21PK-Backend12:12:410
2150270994517,8cyclictest1753-21gnome-shell07:45:440
2150270994513,6cyclictest0-21swapper/011:50:300
215027099450,41cyclictest1622-21rtkit-daemon08:49:520
2150284994412,7cyclictest2236965-21cron11:15:003
2150275994420,19cyclictest2182956-21ntp_states08:40:311
215027099448,16cyclictest1753-21gnome-shell11:55:080
215027099442,17cyclictest1753-21gnome-shell07:35:150
215027099441,23cyclictest1753-21gnome-shell11:11:560
215027099441,16cyclictest1753-21gnome-shell09:00:110
22587092430,7chrt2197014-21PK-Backend12:15:162
22477302430,12sleep1231irq_work/111:45:021
2150280994332,8cyclictest1753-21gnome-shell11:44:312
2150275994325,13cyclictest438-20systemd-journal09:15:001
2150275994319,20cyclictest2189924-40lsb_release09:00:221
2150270994325,13cyclictest2178495-21pool-org.gnome.08:30:000
2150270994317,7cyclictest2197014-21PK-Backend12:34:430
2150270994317,7cyclictest2197014-21PK-Backend12:34:430
2150270994316,7cyclictest1753-21gnome-shell08:43:430
2150270994315,11cyclictest13-21ksoftirqd/009:15:140
162399430,30rtkit-daemon1753-21gnome-shell12:24:032
21619112420,17chrt26-21ksoftirqd/107:40:231
21619112420,17chrt26-21ksoftirqd/107:40:221
215028099421,15cyclictest1753-21gnome-shell07:39:392
2150275994229,9cyclictest2225965-21unin-run10:45:011
2150275994224,13cyclictest3242253-21nautilus09:40:011
2150275994223,15cyclictest855-21dbus-daemon11:50:011
2150275994217,21cyclictest2176281-21cat08:20:321
2150275994217,21cyclictest2176281-21cat08:20:321
2150275994216,13cyclictest907-21udisksd08:01:561
215028499411,10cyclictest2197014-21PK-Backend09:10:433
2150280994124,13cyclictest2221752-21gdbus10:30:282
2150280994123,13cyclictest2229849-21sed10:55:022
2150275994130,8cyclictest1069-21runrttasks09:58:061
2150275994117,20cyclictest855-21dbus-daemon10:25:011
215027599411,35cyclictest2197264-21gzip09:20:291
215027099411,24cyclictest2197014-21PK-Backend11:39:400
215028499401,24cyclictest2219827-21rt-features10:25:303
2150280994029,8cyclictest28750irq/122-ahci[0012:06:112
2150275994030,7cyclictest64850irq/126-i91509:49:521
2150275994028,9cyclictest26-21ksoftirqd/112:22:021
2150275994023,13cyclictest855-21dbus-daemon10:40:001
2150275994022,14cyclictest2241674-21grep11:25:241
2150275994016,17cyclictest2221110-21grep10:30:151
2150275994012,12cyclictest2214720-21threads10:10:321
215027099402,15cyclictest2254013-21python312:00:300
22652262390,5sleep30-21swapper/312:35:023
22652262390,5sleep30-21swapper/312:35:023
22506562390,13sleep1251rcuc/111:50:331
21689232390,6sleep2331rcuc/208:00:242
2150284993922,6cyclictest42-21ksoftirqd/310:17:293
215028099395,11cyclictest34-21ksoftirqd/208:43:512
2150280993928,7cyclictest2150445-21/usr/sbin/munin07:10:292
2150275993928,8cyclictest1069-21runrttasks11:12:451
2150275993924,11cyclictest2239941-21grep11:20:231
2150275993924,11cyclictest2239941-21grep11:20:231
2150275993921,14cyclictest2207601-21ntpq09:50:261
215027599391,34cyclictest1-21systemd08:40:011
215027599391,12cyclictest2159247-21taskset07:33:501
22505512380,14sleep3411rcuc/311:50:313
2150280993824,10cyclictest2218685-21cron10:25:002
2150280993822,12cyclictest2188298-21sensors_temp08:55:272
2150280993822,12cyclictest1753-21gnome-shell08:47:132
2150280993821,12cyclictest2159206-21sshd07:33:202
2150280993820,13cyclictest875-21polkitd11:11:152
2150275993825,9cyclictest26-21ksoftirqd/107:40:011
2150275993824,10cyclictest2259804-21/usr/sbin/munin12:20:011
2150275993822,12cyclictest1-21systemd11:20:011
21632562370,13sleep2341ksoftirqd/207:45:142
2150284993722,10cyclictest2230167-21touch10:55:033
2150284993720,13cyclictest2156900-21grep07:25:283
215028099376,10cyclictest34-21ksoftirqd/211:15:262
215028099372,32cyclictest1753-21gnome-shell10:59:292
215028099372,31cyclictest1753-21gnome-shell10:41:392
2150280993721,12cyclictest2204077-21awk09:40:262
2150280993720,13cyclictest1632-21dbus-daemon08:00:022
2150280993720,12cyclictest2249015-21cat11:45:342
2150275993717,17cyclictest2192695-21unin-run09:10:011
2150275993717,17cyclictest2192695-21unin-run09:10:011
162399370,31rtkit-daemon0-21swapper/107:59:091
162399370,25rtkit-daemon2197014-21PK-Backend07:15:082
162399370,24rtkit-daemon0-21swapper/212:31:072
162399370,24rtkit-daemon0-21swapper/212:31:072
21718102361,9sleep22171918-21cpuspeed_turbos08:10:152
2150284993620,12cyclictest2182966-21awk08:40:313
215028499361,31cyclictest2254534-21sshd12:04:213
215028499361,11cyclictest2181917-21cron08:40:003
2150280993626,6cyclictest2215664-21tr10:15:162
2150280993625,7cyclictest2244545-21ls11:35:162
2150280993622,10cyclictest34-21ksoftirqd/209:35:172
2150280993622,10cyclictest2241567-21latency11:25:232
2150280993621,11cyclictest34-21ksoftirqd/208:52:532
2150280993621,11cyclictest2233462-21cron11:05:002
2150280993621,11cyclictest2206509-21systemd-udevd09:49:542
2150280993620,12cyclictest2200632-21ntp_states09:30:282
2150280993619,13cyclictest2256601-21apt12:10:132
2150280993619,13cyclictest2256601-21apt12:10:132
2150275993622,9cyclictest3242253-21nautilus09:00:021
2150275993622,10cyclictest2200995-21sshd09:30:451
2150275993621,11cyclictest2185920-21ls08:50:161
2150275993621,11cyclictest2151213-21cat07:10:271
2150275993619,13cyclictest2266183-21python312:35:321
2150275993619,13cyclictest2228917-21ntpq10:50:271
2150275993616,16cyclictest26-21ksoftirqd/112:06:391
2150275993613,20cyclictest1826-21JS14:31:441
162399360,30rtkit-daemon0-21swapper/109:25:461
162399360,30rtkit-daemon0-21swapper/108:16:351
2150284993522,9cyclictest1753-21gnome-shell10:20:193
2150284993520,11cyclictest855-21dbus-daemon10:55:023
2150284993518,5cyclictest42-21ksoftirqd/311:48:343
2150284993518,12cyclictest438-20systemd-journal10:14:213
215028099358,16cyclictest1753-21gnome-shell11:25:012
215028099358,16cyclictest1753-21gnome-shell11:25:012
2150280993523,8cyclictest1753-21gnome-shell09:51:152
2150280993520,11cyclictest2226839-21sed10:45:232
2150280993519,12cyclictest2222744-21apt10:35:152
2150280993518,13cyclictest2215259-21pool-org.gnome.10:15:002
2150280993516,8cyclictest2191413-21cstates09:05:172
2150280993516,8cyclictest2191413-21cstates09:05:172
2150280993516,15cyclictest3242253-21nautilus12:40:002
2150275993524,7cyclictest2152284-21ntp_states07:15:051
2150275993521,10cyclictest2258051-21cron12:15:011
2150275993521,10cyclictest2258051-21cron12:15:011
2150275993520,11cyclictest2156732-21sed07:25:241
2150275993519,12cyclictest2235238-21pluginstate11:10:021
2150275993519,12cyclictest2235238-21pluginstate11:10:021
2150275993519,12cyclictest2231019-21pluginstate10:55:261
2150275993518,12cyclictest1-21systemd11:40:011
22196412340,6sleep22197014-21PK-Backend10:25:252
215028099344,9cyclictest0-21swapper/212:02:302
2150280993421,9cyclictest1753-21gnome-shell08:05:352
2150280993418,12cyclictest2173819-21cat08:15:152
215028099341,12cyclictest141rcu_preempt09:25:332
215027599342,9cyclictest2219825-21/usr/sbin/munin10:25:301
2150275993424,6cyclictest2243495-21ntpq11:30:271
2150275993418,12cyclictest2232237-21expr11:00:171
2150275993418,11cyclictest2262402-21cat12:25:211
2150275993417,13cyclictest1753-21gnome-shell11:55:091
2150284993318,11cyclictest2258524-21cpuspeed_turbos12:15:153
215028499331,10cyclictest1753-21gnome-shell09:20:243
2150280993323,6cyclictest2210031-21dump-pmu-power10:00:002
2150280993321,8cyclictest1753-21gnome-shell10:06:002
2150280993319,10cyclictest2165482-21sed07:50:252
2150280993318,11cyclictest2243729-21cat11:30:322
2150280993317,12cyclictest2211089-21awk10:00:292
2150280993317,12cyclictest2179461-21ntpq08:30:262
2150280993316,13cyclictest2250413-21ntpq11:50:272
215028099331,28cyclictest1753-21gnome-shell09:12:162
2150280993311,11cyclictest2177213-21cstates08:25:162
2150275993318,11cyclictest2264551-21cat12:30:351
2150275993318,11cyclictest2264551-21cat12:30:351
162399330,10rtkit-daemon141rcu_preempt07:21:432
2150284993210,19cyclictest21696302sleep308:03:033
2150284993210,18cyclictest1753-21gnome-shell08:45:043
2150280993210,18cyclictest1753-21gnome-shell11:08:542
2150280993210,18cyclictest1753-21gnome-shell11:08:542
2150280993210,18cyclictest1753-21gnome-shell07:28:322
215027599322,25cyclictest2216255-21ntp_states10:15:301
2150275993215,5cyclictest2254475-21taskset12:03:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional