You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 13:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Nov 19, 2025 00:45:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36693824100,9sleep342-21ksoftirqd/319:05:033
162399950,9rtkit-daemon1622-21rtkit-daemon20:25:082
162399950,9rtkit-daemon1622-21rtkit-daemon20:25:082
162399920,8rtkit-daemon1622-21rtkit-daemon19:32:332
162399920,8rtkit-daemon1622-21rtkit-daemon19:32:332
162399910,8rtkit-daemon1622-21rtkit-daemon20:53:322
3869042890,8sleep21753-21gnome-shell20:00:152
162399890,9rtkit-daemon1622-21rtkit-daemon21:18:112
162399890,8rtkit-daemon1622-21rtkit-daemon20:41:402
162399840,9rtkit-daemon1622-21rtkit-daemon20:38:562
4546232750,4sleep20-21swapper/223:10:332
36819227540,29sleep00-21swapper/019:07:020
162399710,58rtkit-daemon0-21swapper/119:07:391
4493052690,4chrt0-21swapper/122:55:231
4493052690,4chrt0-21swapper/122:55:231
36835026747,13sleep20-21swapper/219:09:052
36856299633,55cyclictest1827-21JS14:31:441
36856299602,32cyclictest428483-21users21:55:321
368562996020,36cyclictest855-21dbus-daemon21:05:011
368562996017,38cyclictest392080-21/usr/sbin/munin20:15:311
36855799601,34cyclictest469282-21chrt23:53:380
368562995916,39cyclictest453838-21latency_hist23:10:021
368562995820,33cyclictest432632-21apt-get22:10:001
36856299581,38cyclictest0-21swapper/122:01:501
162399580,8rtkit-daemon1622-21rtkit-daemon21:27:053
162399580,8rtkit-daemon1622-21rtkit-daemon19:42:503
3691792570,5sleep1251rcuc/119:10:191
368562995718,34cyclictest411402-21date21:10:021
368562995718,34cyclictest411402-21date21:10:021
368557995724,28cyclictest428747-21sshd21:58:430
36855799571,30cyclictest479926-21chrt00:22:230
162399570,8rtkit-daemon1622-21rtkit-daemon19:21:203
162399570,8rtkit-daemon1622-21rtkit-daemon19:21:203
4153222560,7sleep32197014-21PK-Backend21:20:173
368562995626,27cyclictest377768-21apt-config19:35:021
368562995626,27cyclictest377768-21apt-config19:35:021
36856299562,49cyclictest380959-21sort19:40:311
368562995617,35cyclictest445577-21expr22:45:171
368562995617,34cyclictest855-21dbus-daemon20:25:011
36856299561,50cyclictest438-20systemd-journal22:23:441
36855799563,48cyclictest395768-21ls20:25:140
36855799563,48cyclictest395768-21ls20:25:140
36855799561,30cyclictest854-21avahi-daemon21:10:010
36855799561,30cyclictest854-21avahi-daemon21:10:010
162399560,9rtkit-daemon1622-21rtkit-daemon21:44:423
36856799550,50cyclictest1622-21rtkit-daemon20:07:562
368562995524,27cyclictest415086-21cpuspeed_turbos21:20:161
368562995518,33cyclictest469214-21sshd23:52:431
368562995517,34cyclictest444810-21seq22:41:451
368562995517,34cyclictest390161-21cut20:10:011
36855799552,47cyclictest428901-21pool-org.gnome.22:00:020
36855799551,48cyclictest438-20systemd-journal23:15:020
36855799551,48cyclictest438-20systemd-journal23:15:020
162399550,9rtkit-daemon1622-21rtkit-daemon19:16:193
162399550,8rtkit-daemon1622-21rtkit-daemon21:04:413
36856299542,47cyclictest406160-21tr20:55:151
368562995416,33cyclictest483716-21cut00:35:011
36856299541,48cyclictest480420-21seq00:25:131
36856299541,48cyclictest460996-21sort23:30:121
162399540,9rtkit-daemon1622-21rtkit-daemon20:46:053
162399540,9rtkit-daemon1622-21rtkit-daemon20:46:053
368567995320,13cyclictest451262-21awk23:00:272
368567995320,13cyclictest451262-21awk23:00:272
36856299532,47cyclictest1827-21JS14:31:441
368562995319,29cyclictest423712-21dpkg21:45:021
368562995316,33cyclictest376030-21cat19:30:021
36856299531,47cyclictest454320-21expr23:10:231
368562995313,35cyclictest483953-21cat00:35:161
36856299531,31cyclictest403970-21latency_hist20:50:021
36856299531,31cyclictest403970-21latency_hist20:50:021
368562995311,36cyclictest371734-21ntp_states19:15:271
36855799532,46cyclictest1753-21gnome-shell00:35:170
162399530,7rtkit-daemon1622-21rtkit-daemon19:12:002
4737322520,6chrt0-21swapper/200:05:232
4490292520,7sleep234-21ksoftirqd/222:55:162
4490292520,7sleep234-21ksoftirqd/222:55:162
36856799529,16cyclictest1753-21gnome-shell23:58:352
36856299522,46cyclictest1753-21gnome-shell00:00:141
368562995217,29cyclictest447486-21sh22:50:211
368562995216,31cyclictest434123-21timerandwakeup22:10:331
368562995215,33cyclictest374009-21seq19:22:541
368562995215,33cyclictest374009-21seq19:22:541
368562995214,34cyclictest390665-21cstates20:10:181
368562995212,35cyclictest466188-21cat23:45:121
368557995221,27cyclictest855-21dbus-daemon23:15:000
36856799519,17cyclictest1753-21gnome-shell00:14:172
36856299512,44cyclictest464211-21munin-run23:40:011
36856299512,44cyclictest456618-21sensors_temp23:15:311
36856299512,44cyclictest456618-21sensors_temp23:15:311
36856299512,44cyclictest1753-21gnome-shell00:24:001
36856299512,43cyclictest458021-21irqstats23:20:231
368562995118,29cyclictest421195-21timerwakeupswit21:35:321
368562995117,29cyclictest465493-21timerwakeupswit23:40:341
368562995116,30cyclictest436057-21sshd22:15:451
368562995116,30cyclictest436057-21sshd22:15:451
368562995115,32cyclictest419982-21date21:35:021
368562995115,32cyclictest419982-21date21:35:021
36856299511,45cyclictest417953-21seq21:27:431
36856299511,44cyclictest441199-21cat22:30:331
36856299511,27cyclictest0-21swapper/123:57:361
368562995112,34cyclictest438-20systemd-journal21:14:561
368562995111,35cyclictest414016-21proc_pri21:15:291
368557995121,24cyclictest470388-21ntp_kernel_err23:55:270
368557995114,32cyclictest480887-21cat00:25:210
162399510,36rtkit-daemon2197014-21PK-Backend23:15:393
162399510,36rtkit-daemon2197014-21PK-Backend23:15:393
36856799509,16cyclictest1753-21gnome-shell20:10:252
36856799507,16cyclictest1753-21gnome-shell21:51:132
368567995023,7cyclictest477131-21tr00:15:172
368567995020,7cyclictest393567-21sh20:20:002
368567995017,9cyclictest425538-21sed21:50:022
368567995015,11cyclictest406568-21grep20:55:242
36856799500,47cyclictest1622-21rtkit-daemon19:37:522
36856299502,44cyclictest3242253-21nautilus21:45:011
36856299502,43cyclictest378972-21/usr/sbin/munin19:35:291
36856299502,42cyclictest476047-21python300:10:321
36856299502,42cyclictest385167-21apt19:55:131
368562995024,21cyclictest426465-21proc_pri21:50:301
368562995020,26cyclictest439903-21idleruntime-cro22:30:001
36856299501,43cyclictest473943-21open_files00:05:281
368562995012,33cyclictest395796-21cat20:25:141
368562995012,33cyclictest395796-21cat20:25:141
368562995011,34cyclictest476953-21apt00:15:141
36855799507,38cyclictest1753-21gnome-shell19:40:230
36855799502,44cyclictest467678-21latency_hist23:50:030
368557995024,22cyclictest13-21ksoftirqd/019:20:010
368557995020,26cyclictest441018-21taskset22:30:300
368557995020,25cyclictest402043-21sshd20:42:430
36855799501,44cyclictest461219-21taskset23:30:140
368557995013,32cyclictest423857-40apt-check21:45:170
36856799499,16cyclictest1753-21gnome-shell23:09:322
368567994916,9cyclictest402675-21tr20:45:162
368567994916,9cyclictest402675-21tr20:45:162
368567994915,10cyclictest481316-21grep00:25:322
36856799491,22cyclictest1753-21gnome-shell00:31:462
368562994916,28cyclictest451170-21missed_timers23:00:261
368562994916,28cyclictest451170-21missed_timers23:00:251
368562994916,28cyclictest26-21ksoftirqd/119:45:151
368562994915,29cyclictest387821-21switchtime20:00:341
36856299491,44cyclictest401212-21if_enp2s020:40:251
36856299491,43cyclictest398033-21ntpq20:30:261
368562994913,32cyclictest441885-21awk22:35:131
368562994911,33cyclictest399144-21cat20:35:151
36855799492,42cyclictest1825-21JS14:31:440
36855799492,42cyclictest1753-21gnome-shell20:10:340
36855799492,41cyclictest389430-21/usr/sbin/munin20:05:280
368557994922,22cyclictest1753-21gnome-shell00:18:560
368557994920,24cyclictest482937-21awk00:30:290
368557994917,27cyclictest399237-21cstates20:35:160
368557994915,28cyclictest438-20systemd-journal19:58:430
368557994914,30cyclictest431815-21cat22:05:250
368557994913,31cyclictest464685-21cat23:40:170
36856799489,16cyclictest1753-21gnome-shell00:20:042
368567994822,7cyclictest1069-21runrttasks19:45:122
368567994820,7cyclictest1069-21runrttasks21:00:332
36856299482,41cyclictest1753-21gnome-shell20:55:021
36855799488,35cyclictest1632-21dbus-daemon21:45:010
36855799483,40cyclictest402621-21cpuspeed_turbos20:45:160
36855799483,40cyclictest402621-21cpuspeed_turbos20:45:160
36855799482,42cyclictest1753-21gnome-shell19:40:030
368557994816,27cyclictest1069-21runrttasks22:39:200
36855799481,43cyclictest4340232sleep022:10:310
36855799481,42cyclictest438414-21/usr/sbin/munin22:25:110
368557994814,29cyclictest476832-21date00:15:020
368557994813,31cyclictest393571-21cut20:20:010
368557994813,31cyclictest1753-21gnome-shell20:56:500
368557994812,32cyclictest1753-21gnome-shell20:54:460
368557994811,32cyclictest373090-21df19:20:190
368557994811,32cyclictest373090-21df19:20:180
368575994714,28cyclictest1617-10pipewire-media-00:10:043
36856799479,18cyclictest1753-21gnome-shell23:17:252
36856799479,18cyclictest1753-21gnome-shell23:17:242
368567994721,7cyclictest1069-21runrttasks21:38:092
368567994715,11cyclictest34-21ksoftirqd/221:55:402
36856799471,28cyclictest2197014-21PK-Backend20:20:392
368567994710,16cyclictest1753-21gnome-shell21:20:202
368557994719,24cyclictest384942-21latency_hist19:55:010
368557994719,23cyclictest452496-21cstates23:05:160
368557994718,25cyclictest416638-21latency_hist21:25:020
368557994717,25cyclictest443830-21cstates22:40:170
368557994716,26cyclictest1617-10pipewire-media-19:30:180
368557994716,26cyclictest1617-10pipewire-media-19:30:180
36855799471,40cyclictest414015-21ps21:15:290
368557994713,30cyclictest382845-21seq19:46:080
368557994711,31cyclictest408640-21cat21:00:310
162399470,35rtkit-daemon2197014-21PK-Backend22:07:573
162399470,34rtkit-daemon1753-21gnome-shell23:50:413
36856799469,17cyclictest1753-21gnome-shell22:36:232
36856799469,16cyclictest1753-21gnome-shell23:41:002
36856799469,16cyclictest1753-21gnome-shell23:25:282
36856799469,16cyclictest1753-21gnome-shell22:05:142
36856799469,16cyclictest1753-21gnome-shell21:41:482
36856799461,24cyclictest1753-21gnome-shell22:50:252
36856799461,18cyclictest461430-21sort23:30:212
368567994610,16cyclictest1753-21gnome-shell22:01:042
36855799468,33cyclictest418132-21cron21:30:000
368557994615,27cyclictest1069-21runrttasks21:37:580
368557994614,27cyclictest458396-21rt-features23:20:290
36855799461,41cyclictest1753-21gnome-shell20:21:540
36855799461,40cyclictest1753-21gnome-shell20:04:500
162399460,32rtkit-daemon1753-21gnome-shell22:39:413
36856799459,16cyclictest1753-21gnome-shell21:25:452
36856799459,16cyclictest1753-21gnome-shell19:25:462
368567994510,16cyclictest1753-21gnome-shell22:45:472
368567994510,16cyclictest1753-21gnome-shell19:50:452
368567994510,16cyclictest1753-21gnome-shell00:02:372
368567994510,15cyclictest1753-21gnome-shell20:31:322
368557994518,23cyclictest450765-21cstates23:00:170
368557994518,23cyclictest450765-21cstates23:00:170
368557994518,23cyclictest1753-21gnome-shell23:29:550
368557994516,25cyclictest1753-21gnome-shell21:14:330
36855799451,38cyclictest1753-21gnome-shell22:20:290
368557994511,30cyclictest1753-21gnome-shell21:32:590
368557994511,30cyclictest1753-21gnome-shell21:32:590
162399450,32rtkit-daemon1753-21gnome-shell23:06:363
162399450,32rtkit-daemon1753-21gnome-shell00:37:323
36856799449,14cyclictest1753-21gnome-shell22:30:312
36855799449,31cyclictest1753-21gnome-shell22:49:580
368557994417,23cyclictest426722-21sort21:50:340
368557994417,23cyclictest398842-21latency_hist20:35:010
368557994416,24cyclictest447297-21cstates22:50:170
368557994414,26cyclictest473334-21cpuspeed_turbos00:05:150
368557994412,28cyclictest368955-21cpuspeed_turbos19:10:140
368557994412,27cyclictest462956-21cstates23:35:180
4741332430,5sleep32197014-21PK-Backend00:05:323
36856799431,22cyclictest1753-21gnome-shell21:10:232
368557994313,25cyclictest472369-21switchtime00:00:300
368557994310,29cyclictest434962-21cpu22:15:150
368557994310,29cyclictest434962-21cpu22:15:150
36856799421,22cyclictest1753-21gnome-shell22:20:442
36855799421,36cyclictest448870-21/usr/sbin/munin22:55:150
36855799421,36cyclictest448870-21/usr/sbin/munin22:55:150
368575994127,9cyclictest438-20systemd-journal00:35:003
368575994122,14cyclictest378778-21memory19:35:243
36856799418,16cyclictest1753-21gnome-shell23:35:222
162399410,30rtkit-daemon881143-21gmain22:20:333
162399410,29rtkit-daemon2197014-21PK-Backend22:49:343
162399410,29rtkit-daemon2197014-21PK-Backend22:34:093
368575994022,14cyclictest375208-21ntp_states19:25:283
36856799409,8cyclictest1358-1kworker/u9:121:05:332
36856799409,8cyclictest1358-1kworker/u9:121:05:332
162399400,28rtkit-daemon2197014-21PK-Backend23:45:503
36857599397,14cyclictest1753-21gnome-shell20:53:443
36857599392,7cyclictest1753-21gnome-shell21:19:273
36857599392,33cyclictest1753-21gnome-shell22:19:103
36857599392,33cyclictest1753-21gnome-shell22:19:103
36857599392,33cyclictest1753-21gnome-shell22:12:553
368575993922,14cyclictest42-21ksoftirqd/320:35:263
368567993917,8cyclictest2197014-21PK-Backend19:22:322
368567993917,8cyclictest2197014-21PK-Backend19:22:322
162399390,29rtkit-daemon2197014-21PK-Backend23:58:083
162399390,27rtkit-daemon0-21swapper/322:56:523
162399390,27rtkit-daemon0-21swapper/322:56:513
162399390,27rtkit-daemon0-21swapper/300:29:063
162399390,27rtkit-daemon0-21swapper/300:03:103
368575993823,11cyclictest397022-21cron20:30:003
368575993823,11cyclictest397022-21cron20:30:003
36857599382,16cyclictest429992-40lsb_release22:00:263
36857599382,16cyclictest1753-21gnome-shell21:10:253
36857599382,14cyclictest2197014-21PK-Backend20:40:293
36857599382,12cyclictest859-21NetworkManager21:45:313
36857599381,33cyclictest1753-21gnome-shell21:07:153
36857599381,33cyclictest1753-21gnome-shell21:07:153
368575993813,21cyclictest0-21swapper/320:31:273
36857599381,15cyclictest2197014-21PK-Backend19:48:423
36857599381,10cyclictest2197014-21PK-Backend23:22:503
368567993818,7cyclictest2197014-21PK-Backend19:56:012
162399380,27rtkit-daemon2197014-21PK-Backend00:19:593
368575993726,7cyclictest859-21NetworkManager22:53:413
368575993722,9cyclictest1827-21JS14:31:443
36857599371,31cyclictest438-20systemd-journal20:00:003
36857599371,17cyclictest1753-21gnome-shell23:34:183
36856799372,17cyclictest1753-21gnome-shell23:45:252
162399370,26rtkit-daemon0-21swapper/323:37:523
162399370,26rtkit-daemon0-21swapper/300:24:263
36857599368,9cyclictest141rcu_preempt19:10:283
368575993625,7cyclictest391839-21basename20:15:003
36857599362,30cyclictest1753-21gnome-shell23:25:073
368575993622,9cyclictest1753-21gnome-shell23:40:243
368575993622,9cyclictest1753-21gnome-shell21:30:393
368575993622,9cyclictest1753-21gnome-shell21:30:393
36857599362,15cyclictest407466-21pool-org.gnome.21:00:013
368575993620,12cyclictest426394-21ntp_kernel_pll_21:50:273
36857599361,32cyclictest2197014-21PK-Backend20:00:403
36857599361,11cyclictest2197014-21PK-Backend19:31:113
36857599361,11cyclictest2197014-21PK-Backend19:31:113
36857599361,11cyclictest1358-1kworker/u9:120:20:003
368567993626,6cyclictest2197014-21PK-Backend19:42:302
368567993619,12cyclictest468310-21df_inode23:50:162
162399360,25rtkit-daemon0-21swapper/323:02:243
162399360,25rtkit-daemon0-21swapper/323:02:243
368575993517,14cyclictest1753-21gnome-shell22:28:143
368567993519,12cyclictest434395-40apt-check22:12:192
162399350,5rtkit-daemon0-21swapper/323:12:373
368575993424,7cyclictest1069-21runrttasks20:20:313
368575993422,8cyclictest1753-21gnome-shell21:57:003
36856799341,5cyclictest141rcu_preempt22:16:182
36856799341,5cyclictest141rcu_preempt22:16:172
36857599332,9cyclictest1753-21gnome-shell20:05:163
36857599331,9cyclictest2197014-21PK-Backend19:53:473
36857599331,30cyclictest0-21swapper/321:36:393
368567993321,8cyclictest484022-21grep00:35:162
368567993319,10cyclictest370880-21cut19:15:052
36856799331,29cyclictest2197014-21PK-Backend23:22:022
368567993225,4cyclictest2197014-21PK-Backend22:40:272
368567993218,10cyclictest34-21ksoftirqd/221:30:592
368567993218,10cyclictest34-21ksoftirqd/221:30:582
36856799301,5cyclictest141rcu_preempt22:25:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional