You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 11:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Sep 17, 2025 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991010,8rtkit-daemon1622-21rtkit-daemon21:20:053
1623991000,8rtkit-daemon1622-21rtkit-daemon21:54:433
21773882890,5sleep00-21swapper/019:05:130
217873727964,9sleep30-21swapper/319:09:433
162399790,69rtkit-daemon0-21swapper/219:08:332
2178904997519,34cyclictest2197014-21PK-Backend22:38:393
217841027348,16sleep10-21swapper/119:05:311
217890499712,8cyclictest1753-21gnome-shell19:53:573
162399700,8rtkit-daemon860997-21snmpd20:34:533
162399690,10rtkit-daemon1785-21gmain23:02:083
217890499682,9cyclictest1753-21gnome-shell20:03:323
217890499682,7cyclictest1753-21gnome-shell21:55:153
162399680,9rtkit-daemon1622-21rtkit-daemon20:53:382
217890499673,8cyclictest1753-21gnome-shell00:33:293
217890499671,51cyclictest2197014-21PK-Backend00:22:513
217890499662,8cyclictest1753-21gnome-shell22:44:063
217890499662,8cyclictest1753-21gnome-shell19:28:043
217890499661,16cyclictest1753-21gnome-shell19:39:343
217890499652,9cyclictest1753-21gnome-shell19:17:113
217890499652,9cyclictest1753-21gnome-shell19:17:103
217890499642,8cyclictest1753-21gnome-shell22:26:333
217890499642,7cyclictest1753-21gnome-shell20:05:273
217890499641,8cyclictest1753-21gnome-shell19:11:003
2178892996414,29cyclictest0-21swapper/023:55:060
2178892996411,20cyclictest2247163-21sort22:25:170
217890499622,7cyclictest1753-21gnome-shell22:30:153
217890499622,7cyclictest1753-21gnome-shell21:06:043
217890499621,16cyclictest1753-21gnome-shell22:05:403
2178892996212,11cyclictest2218804-21ntp_states21:00:280
217890499612,9cyclictest1753-21gnome-shell20:26:173
2178904996118,38cyclictest2264725-40apt-check23:15:133
217890499611,15cyclictest1753-21gnome-shell21:25:133
2178892995915,25cyclictest1753-21gnome-shell19:33:000
2178892995911,20cyclictest2194637-21python319:50:270
162399590,8rtkit-daemon1622-21rtkit-daemon21:26:122
217890499583,50cyclictest2246162-21grep22:20:273
217890499582,50cyclictest1753-21gnome-shell20:35:283
217890499582,50cyclictest1753-21gnome-shell20:35:283
2178892995818,16cyclictest1753-21gnome-shell20:40:120
2178892995818,15cyclictest1753-21gnome-shell22:22:030
2178892995811,18cyclictest1753-21gnome-shell19:25:170
217890499572,10cyclictest1753-21gnome-shell00:13:173
2178892995715,17cyclictest2228515-40apt-check21:27:270
2178892995713,18cyclictest0-21swapper/023:25:050
217890499562,49cyclictest1753-21gnome-shell00:28:393
2178904995618,34cyclictest1753-21gnome-shell20:45:283
2178904995617,34cyclictest2222158-21cat21:10:243
2178892995610,14cyclictest2271395-21munin-run23:35:010
162399560,8rtkit-daemon1622-21rtkit-daemon23:00:072
22332932550,6sleep0161rcuc/021:40:280
217890499559,18cyclictest1753-21gnome-shell20:55:273
217890499559,18cyclictest1753-21gnome-shell20:55:273
217890499554,46cyclictest438-20systemd-journal00:40:003
217890499552,48cyclictest1753-21gnome-shell23:29:313
217890099550,50cyclictest1622-21rtkit-daemon21:54:432
2178892995512,16cyclictest1753-21gnome-shell21:35:310
2178892995511,26cyclictest1753-21gnome-shell00:10:160
2178892995511,16cyclictest2213832-21/usr/sbin/munin20:45:300
217890499542,8cyclictest1753-21gnome-shell21:00:343
217890499542,48cyclictest1753-21gnome-shell23:42:113
217890499542,48cyclictest1753-21gnome-shell23:42:103
217890499542,47cyclictest1753-21gnome-shell19:56:433
217890499542,47cyclictest1753-21gnome-shell19:56:433
2178904995417,33cyclictest1753-21gnome-shell21:43:223
2178904995416,33cyclictest2204540-21df20:20:153
2178892995418,12cyclictest1753-21gnome-shell22:33:200
2178892995413,17cyclictest1753-21gnome-shell20:31:560
2178892995411,19cyclictest2202605-21cpuspeed_turbos20:15:170
162399540,9rtkit-daemon1622-21rtkit-daemon23:06:102
162399540,8rtkit-daemon1622-21rtkit-daemon19:27:082
22675582531,26sleep22197014-21PK-Backend23:20:372
217890499532,8cyclictest1753-21gnome-shell20:43:043
217890499532,45cyclictest1753-21gnome-shell22:56:453
217890499531,46cyclictest854-21avahi-daemon22:45:063
217890499531,12cyclictest2266770-21expr23:20:163
162399530,8rtkit-daemon1622-21rtkit-daemon23:19:402
217890499529,38cyclictest1753-21gnome-shell22:13:513
217890499529,18cyclictest1753-21gnome-shell00:15:333
217890499529,18cyclictest1753-21gnome-shell00:15:323
217890499522,8cyclictest1753-21gnome-shell23:45:263
217890499522,8cyclictest1753-21gnome-shell00:09:233
2178904995221,27cyclictest1753-21gnome-shell23:36:523
217890499521,46cyclictest854-21avahi-daemon23:57:453
2178904995212,34cyclictest2280621-21gdbus00:00:033
2178904995212,34cyclictest2280621-21gdbus00:00:033
2178904995210,38cyclictest1753-21gnome-shell23:54:153
2178892995218,11cyclictest2275706-21memory23:45:200
2178892995212,17cyclictest1753-21gnome-shell21:50:300
2178892995210,19cyclictest2195893-21entropy19:55:220
2178892995210,19cyclictest2195893-21entropy19:55:210
162399520,8rtkit-daemon1622-21rtkit-daemon22:01:552
162399520,8rtkit-daemon1622-21rtkit-daemon20:23:512
22318822510,9sleep234-21ksoftirqd/221:35:312
217890499519,37cyclictest2202895-21cat20:15:233
217890499519,16cyclictest1753-21gnome-shell21:45:183
217890499512,7cyclictest1753-21gnome-shell22:50:403
217890499511,45cyclictest1753-21gnome-shell20:54:533
217890499511,45cyclictest1753-21gnome-shell19:42:153
2178904995110,37cyclictest1753-21gnome-shell19:47:563
217890099510,46cyclictest1622-21rtkit-daemon21:20:052
217889299519,16cyclictest2221492-21/usr/sbin/munin21:10:300
2178892995112,17cyclictest1753-21gnome-shell19:35:180
2178892995112,16cyclictest1753-21gnome-shell22:37:510
2178892995111,17cyclictest1753-21gnome-shell23:45:000
2178892995111,17cyclictest1753-21gnome-shell23:44:590
2178892995110,15cyclictest2257205-21munin-run22:55:020
22022142500,20sleep234-21ksoftirqd/220:15:022
217890499502,17cyclictest1753-21gnome-shell20:10:153
2178904995019,25cyclictest854-21avahi-daemon23:05:243
2178904995016,30cyclictest2270410-21kernelversion23:30:243
217890499501,44cyclictest2229069-21taskset21:30:133
217890499501,43cyclictest2197014-21PK-Backend21:35:203
2178892995017,11cyclictest1753-21gnome-shell20:05:130
2178892995010,15cyclictest854-21avahi-daemon00:05:270
217890499491,44cyclictest860997-21snmpd22:16:333
217889299499,14cyclictest2230538-21latency_hist21:35:010
2178892994918,23cyclictest2220456-21needreboot21:05:230
21934642480,5sleep1251rcuc/119:49:451
2178892994818,24cyclictest2239085-21sshd21:58:440
2178892994816,18cyclictest1753-21gnome-shell22:45:140
217889299481,42cyclictest1622-21rtkit-daemon23:19:580
22261162470,6chrt0-21swapper/121:20:281
2178904994717,26cyclictest1069-21runrttasks19:33:193
2178904994716,27cyclictest42-21ksoftirqd/323:12:283
2178904994710,33cyclictest2224947-21cut21:20:033
2178892994721,23cyclictest1358-1kworker/u9:120:15:010
2178892994717,25cyclictest2291788-21/usr/sbin/munin00:30:260
2178892994710,11cyclictest1753-21gnome-shell22:04:570
2178898994620,22cyclictest2246800-21unin-run22:25:011
2178892994616,25cyclictest1753-21gnome-shell00:35:450
2178892994611,14cyclictest2266756-21cstates23:20:160
2178892994611,11cyclictest2181723-21awk19:15:210
2178892994611,11cyclictest2181723-21awk19:15:210
2178892994610,15cyclictest1753-21gnome-shell19:41:090
22542782450,6sleep22197014-21PK-Backend22:45:182
217890499451,18cyclictest1753-21gnome-shell22:00:183
217890499451,12cyclictest2183181-21cpuspeed_turbos19:20:153
217889299457,33cyclictest0-21swapper/022:05:170
2178892994520,21cyclictest2234590-21expr21:45:160
22856552440,7sleep22197014-21PK-Backend00:15:022
22357512440,6sleep22197014-21PK-Backend21:50:002
2178900994426,14cyclictest855-21dbus-daemon20:45:002
2178898994421,19cyclictest2247913-21python322:25:321
217889299449,12cyclictest2209749-21df_inode20:35:150
217889299449,12cyclictest2209749-21df_inode20:35:150
217889299447,15cyclictest0-21swapper/023:52:590
2178892994432,9cyclictest0-21swapper/022:11:140
2178892994428,12cyclictest2286877-21timerwakeupswit00:15:310
2178892994428,12cyclictest2286877-21timerwakeupswit00:15:310
162399440,31rtkit-daemon1753-21gnome-shell23:29:582
21834182430,5sleep22197014-21PK-Backend19:20:162
2178898994318,10cyclictest2292216-21seq00:31:181
217889299438,18cyclictest141rcu_preempt19:22:120
217889299438,13cyclictest0-21swapper/019:49:040
2178892994323,15cyclictest3242253-21nautilus23:10:010
162399430,36rtkit-daemon1753-21gnome-shell23:14:132
217889299425,15cyclictest0-21swapper/022:57:510
217890099415,13cyclictest17531gnome-shell21:04:502
2178900994124,13cyclictest2180890-21python319:15:042
2178900994124,13cyclictest2180890-21python319:15:042
217890099411,9cyclictest2197014-21PK-Backend00:25:442
2178898994126,11cyclictest26-21ksoftirqd/121:45:371
217889899411,35cyclictest2180851-21vmstat19:15:031
217889899411,35cyclictest2180851-21vmstat19:15:031
2178898994110,6cyclictest141rcu_preempt21:45:001
2178892994124,12cyclictest1753-21gnome-shell20:25:200
2178892994118,18cyclictest2216321-21dpkg20:55:130
2178892994118,18cyclictest2216321-21dpkg20:55:130
2178892994118,18cyclictest1753-21gnome-shell20:20:270
2178892994116,20cyclictest2225459-21cstates21:20:150
2178892994115,21cyclictest1632-21dbus-daemon22:45:000
217889299410,15cyclictest0-21swapper/023:10:230
22879152400,5sleep22197014-21PK-Backend00:20:172
22068672400,7chrt2197014-21PK-Backend20:25:322
217890099402,34cyclictest1753-21gnome-shell20:56:282
217890099402,34cyclictest1753-21gnome-shell20:56:272
2178898994028,8cyclictest2210101-21tr20:35:231
2178898994028,8cyclictest2210101-21tr20:35:221
2178898994023,13cyclictest855-21dbus-daemon21:01:561
217889899401,35cyclictest2232276-21awk21:40:011
2178892994020,15cyclictest2180784-21sed19:15:020
2178892994018,17cyclictest2223264-21grep21:15:150
2178892994017,19cyclictest1753-21gnome-shell00:03:580
2178892994017,19cyclictest1753-21gnome-shell00:03:570
2178892994017,18cyclictest1753-21gnome-shell00:24:050
2178892994016,20cyclictest1069-21runrttasks23:35:390
162399400,26rtkit-daemon0-21swapper/200:34:322
22892812390,5sleep11753-21gnome-shell00:25:031
2178898993925,10cyclictest2283819-21sshd00:09:451
2178898993925,10cyclictest2267723-21sshd23:22:451
217889899392,33cyclictest1753-21gnome-shell20:23:101
217889899392,32cyclictest1753-21gnome-shell19:35:191
2178898993918,15cyclictest1-21systemd00:15:001
2178898993913,23cyclictest1713355-21cups-browsed23:01:101
2178898993912,22cyclictest2249026-21ls22:30:141
2178892993920,15cyclictest2243938-21df_inode22:15:170
2178892993918,16cyclictest2214990-21smartctl20:50:200
2178892993917,18cyclictest2290233-21cat00:25:320
2178892993916,19cyclictest2259543-21df_inode23:00:160
2178892993916,19cyclictest1938-21gsd-housekeepin20:01:200
2178900993824,10cyclictest2196626-21sshd19:55:452
2178900993824,10cyclictest2196626-21sshd19:55:452
217890099382,18cyclictest2222984-21turbostat.cron21:15:002
2178900993819,14cyclictest2280490-21cups-browsed00:00:062
2178900993819,14cyclictest2280490-21cups-browsed00:00:062
2178900993814,7cyclictest22566972grep22:50:312
2178898993827,8cyclictest26-21ksoftirqd/123:15:021
2178898993826,7cyclictest2195229-21sh19:55:001
2178898993824,10cyclictest2230496-21sshd21:34:461
217889899382,32cyclictest1753-21gnome-shell21:58:141
2178898993817,16cyclictest3242253-21nautilus23:00:021
217890099372,7cyclictest1753-21gnome-shell23:52:132
2178900993723,10cyclictest1753-21gnome-shell23:00:002
217890099371,6cyclictest2197014-21PK-Backend19:41:342
2178898993721,12cyclictest2242226-21fw_packets22:10:161
2178898993721,12cyclictest1-21systemd19:30:001
2178898993721,11cyclictest438-20systemd-journal22:42:441
2178898993720,13cyclictest2244153-21memory22:15:221
2178898993719,13cyclictest2239635-21tr22:00:151
2178898993714,20cyclictest2265737-21wc23:15:321
162399370,28rtkit-daemon1622-21rtkit-daemon22:26:172
162399370,28rtkit-daemon1622-21rtkit-daemon21:55:252
21797802360,8sleep12179779-21grep19:10:241
2178900993627,5cyclictest2197014-21PK-Backend22:44:032
2178900993625,7cyclictest2287327-21grep00:20:002
2178900993625,7cyclictest2287327-21grep00:20:002
217890099361,10cyclictest2197014-21PK-Backend19:39:202
217889899365,6cyclictest141rcu_preempt21:30:021
2178898993621,11cyclictest2215461-21swap20:50:301
2178898993620,12cyclictest855-21dbus-daemon00:39:491
2178898993619,13cyclictest2186894-21awk19:30:171
2178898993619,12cyclictest2270019-21cpuspeed_turbos23:30:151
2178898993619,12cyclictest2262002-21cat23:05:281
2178898993618,13cyclictest438-20systemd-journal21:54:551
2178898993616,16cyclictest2255713-21cpu22:50:131
2178898993611,13cyclictest1753-21gnome-shell20:19:161
217890099355,26cyclictest1069-21runrttasks19:49:442
2178900993524,8cyclictest1753-21gnome-shell23:48:462
2178900993524,7cyclictest1753-21gnome-shell22:14:552
2178900993523,8cyclictest1753-21gnome-shell21:33:072
2178900993520,11cyclictest2246864-21sed22:25:012
217890099351,9cyclictest2197014-21PK-Backend22:36:022
2178900993519,12cyclictest2194443-21grep19:50:232
217889899353,11cyclictest1824-21JS14:31:441
217889899353,11cyclictest1824-21JS14:31:441
2178898993524,7cyclictest2196249-21ntpq19:55:291
2178898993524,7cyclictest2196249-21ntpq19:55:281
217889899352,30cyclictest1753-21gnome-shell20:58:491
217889899352,30cyclictest1753-21gnome-shell20:58:491
217889899352,29cyclictest356-21jbd2/sdb2-820:09:561
2178898993521,10cyclictest2272848-21sshd23:36:451
2178898993520,11cyclictest2206714-21grep20:25:291
2178898993519,12cyclictest2190612-21cat19:40:151
2178898993519,12cyclictest1-21systemd00:00:021
2178898993518,13cyclictest2274408-21cat23:40:341
2178898993518,13cyclictest2274408-21cat23:40:341
217889899351,6cyclictest2281533-21python300:00:301
217889899351,6cyclictest2281533-21python300:00:301
2178898993511,19cyclictest1753-21gnome-shell21:10:141
217890099343,26cyclictest2180610-21munin-node19:15:022
2178900993421,9cyclictest1753-21gnome-shell20:03:472
2178900993416,13cyclictest2202827-21entropy20:15:212
2178900993413,7cyclictest2280487-21latency_hist00:00:032
2178898993423,7cyclictest2277738-21missed_timers23:50:241
2178898993419,11cyclictest2289671-21expr00:25:171
2178898993418,12cyclictest2251225-21ntpq22:35:261
2178898993418,12cyclictest2224750-21pool-org.gnome.21:20:011
2178898993418,12cyclictest2211851-21ntpq20:40:251
2178898993418,12cyclictest2200881-21grep20:10:141
2178898993417,13cyclictest2220070-21expr21:05:151
2178898993416,13cyclictest2276192-21cat23:45:301
2178898993412,7cyclictest2198699-21cron20:05:011
2178898993411,19cyclictest2212855-21cpu20:45:141
162399340,27rtkit-daemon0-21swapper/221:40:042
2178900993320,9cyclictest34-21ksoftirqd/223:30:292
217890099331,29cyclictest2197014-21PK-Backend21:08:022
217890099331,29cyclictest2197014-21PK-Backend00:07:322
217890099331,13cyclictest2197014-21PK-Backend20:46:342
2178898993318,11cyclictest2268930-21ntpq23:25:261
2178898993318,11cyclictest2254275-21expr22:45:181
2178898993318,11cyclictest2207975-21expr20:30:141
2178898993317,11cyclictest438-20systemd-journal22:05:461
2178898993316,12cyclictest2183207-21cut19:20:151
217890099329,19cyclictest1753-21gnome-shell21:17:012
2178900993224,5cyclictest2197014-21PK-Backend00:38:112
2178900993224,4cyclictest2197014-21PK-Backend22:19:202
2178900993217,11cyclictest2209362-21cat20:35:102
2178900993217,11cyclictest2209362-21cat20:35:092
217890099321,28cyclictest2197014-21PK-Backend20:31:082
217890099321,10cyclictest2197014-21PK-Backend23:42:112
217890099321,10cyclictest2197014-21PK-Backend23:42:112
2178900993210,18cyclictest1753-21gnome-shell22:05:372
2178900993210,18cyclictest1753-21gnome-shell19:31:302
217890099319,18cyclictest1753-21gnome-shell23:37:092
2178900993119,8cyclictest22488262sleep222:30:122
217890099308,18cyclictest1753-21gnome-shell20:07:422
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional