You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 00:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sun May 04, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991140,20rtkit-daemon0-21swapper/109:31:080
462237210782,16sleep30-21swapper/307:06:423
49762821000,4sleep10-21swapper/108:45:291
5751992940,6sleep10-21swapper/112:25:231
162399930,46rtkit-daemon0-21swapper/107:09:141
162399810,68rtkit-daemon0-21swapper/007:07:340
5773702780,5sleep32197014-21PK-Backend12:30:323
5553502760,4sleep30-21swapper/311:30:033
162399730,5rtkit-daemon2197014-21PK-Backend10:00:070
162399710,6rtkit-daemon2197014-21PK-Backend07:48:280
462616997016,48cyclictest556306-40lsb_release11:30:270
46228627040,24sleep20-21swapper/207:07:212
162399700,7rtkit-daemon2197014-21PK-Backend10:12:000
162399690,6rtkit-daemon2197014-21PK-Backend10:25:060
162399680,11rtkit-daemon1622-21rtkit-daemon12:08:391
162399660,51rtkit-daemon3242254-21gmain11:55:491
462622996012,28cyclictest1753-21gnome-shell08:45:212
162399600,43rtkit-daemon0-21swapper/112:02:131
46262299590,55cyclictest1622-21rtkit-daemon08:19:122
162399590,9rtkit-daemon1622-21rtkit-daemon12:09:300
162399590,8rtkit-daemon1622-21rtkit-daemon10:18:582
162399590,8rtkit-daemon1622-21rtkit-daemon10:18:582
462618995818,19cyclictest26-21ksoftirqd/112:15:001
46261899580,54cyclictest1622-21rtkit-daemon11:41:031
162399580,9rtkit-daemon1622-21rtkit-daemon11:51:411
162399580,8rtkit-daemon1622-21rtkit-daemon08:13:362
462622995712,25cyclictest1753-21gnome-shell12:26:082
46261699572,8cyclictest1753-21gnome-shell11:53:100
162399570,8rtkit-daemon1622-21rtkit-daemon09:07:162
162399570,10rtkit-daemon1622-21rtkit-daemon11:25:591
462622995613,23cyclictest1753-21gnome-shell11:32:352
462622995611,16cyclictest1753-21gnome-shell10:23:322
162399560,8rtkit-daemon1622-21rtkit-daemon09:04:273
462616995518,32cyclictest551072-21cat11:15:310
46262299544,19cyclictest518074-21gpgconf09:45:032
462622995411,22cyclictest1753-21gnome-shell08:07:312
462622995410,18cyclictest1753-21gnome-shell09:10:242
462616995416,33cyclictest573836-21expr12:20:330
462616995416,33cyclictest498800-21cstates08:50:150
162399540,8rtkit-daemon1622-21rtkit-daemon09:21:553
46262299530,48cyclictest1622-21rtkit-daemon09:24:142
462616995320,29cyclictest3242255-21gdbus08:45:010
462616995319,29cyclictest3242255-21gdbus08:10:010
462616995313,12cyclictest547553-21cat11:05:320
162399530,36rtkit-daemon0-21swapper/109:52:591
5004572520,14sleep2331rcuc/208:55:022
46262299526,21cyclictest473743-40lsb_release07:36:262
462616995218,29cyclictest479248-21sshd07:54:340
46262299516,26cyclictest1753-21gnome-shell09:40:162
46262299515,16cyclictest1753-21gnome-shell09:02:172
46262299514,28cyclictest1753-21gnome-shell11:47:132
462622995112,22cyclictest1753-21gnome-shell11:56:482
462616995120,26cyclictest521687-21/usr/sbin/munin09:55:230
462616995117,29cyclictest565100-21swap11:55:300
462616995117,29cyclictest529603-21netstat10:15:260
462616995117,29cyclictest529603-21netstat10:15:260
46262299507,26cyclictest1753-21gnome-shell10:33:502
46262299507,17cyclictest1753-21gnome-shell10:45:122
46262299506,26cyclictest1753-21gnome-shell10:39:482
46262299505,25cyclictest0-21swapper/211:15:002
46262299504,19cyclictest470295-21munin-run07:30:002
462622995011,15cyclictest1753-21gnome-shell07:55:212
462616995015,30cyclictest540785-21grep10:50:000
162399500,35rtkit-daemon1753-21gnome-shell08:53:391
46262299499,16cyclictest567895-21ls12:05:152
46262299496,15cyclictest520852-21python309:50:322
46262299495,19cyclictest1753-21gnome-shell09:25:302
46262299494,21cyclictest1753-21gnome-shell10:28:232
46262299494,19cyclictest527892-21cat10:10:322
46262299494,19cyclictest1753-21gnome-shell08:50:272
46262299492,27cyclictest0-21swapper/210:00:142
462622994911,21cyclictest1753-21gnome-shell07:15:562
46261699496,17cyclictest1753-21gnome-shell07:21:570
462616994921,23cyclictest1753-21gnome-shell09:35:130
462616994917,27cyclictest491656-21/usr/sbin/munin08:30:300
462616994915,11cyclictest505167-21cat09:05:320
46261699491,23cyclictest1753-21gnome-shell12:04:400
46262299486,16cyclictest1753-21gnome-shell08:42:012
46262299484,26cyclictest1753-21gnome-shell07:50:582
462616994819,25cyclictest1753-21gnome-shell10:55:010
462616994816,28cyclictest1632-21dbus-daemon07:20:000
462616994815,9cyclictest507327-21ls09:15:010
462616994812,12cyclictest2181-21gdbus09:45:020
162399480,34rtkit-daemon2197014-21PK-Backend11:06:012
162399480,34rtkit-daemon1753-21gnome-shell10:04:591
4994542470,6sleep32197014-21PK-Backend08:50:293
46262299475,25cyclictest1753-21gnome-shell09:56:132
46262299475,17cyclictest1753-21gnome-shell08:36:292
46262299475,16cyclictest570941-21latency_hist12:15:012
46262299474,18cyclictest1753-21gnome-shell07:40:172
46262299474,17cyclictest514806-21expr09:35:162
462616994720,22cyclictest548490-21cpuspeed_turbos11:10:140
462616994717,25cyclictest519058-21processes09:45:300
462616994715,9cyclictest1753-21gnome-shell11:38:440
46261699471,25cyclictest1753-21gnome-shell08:17:070
462616994711,12cyclictest488624-21which08:20:200
462616994711,10cyclictest552374-21sed11:20:220
162399470,38rtkit-daemon462611-21cyclictest10:26:073
162399470,33rtkit-daemon2197014-21PK-Backend08:25:420
162399470,33rtkit-daemon1753-21gnome-shell10:45:473
46262299468,19cyclictest1753-21gnome-shell07:34:342
46262299465,17cyclictest1753-21gnome-shell07:50:002
46262299464,18cyclictest546400-21sed11:05:022
46262299464,18cyclictest546400-21sed11:05:022
46261699469,17cyclictest1753-21gnome-shell12:19:470
46261699469,16cyclictest1753-21gnome-shell11:47:520
46261699469,16cyclictest1753-21gnome-shell07:35:050
46261699466,16cyclictest1753-21gnome-shell12:31:150
462616994614,10cyclictest554248-21ntp_kernel_err11:25:240
462616994613,12cyclictest497077-21cat08:45:150
46261699461,26cyclictest1753-21gnome-shell12:25:120
46261699461,22cyclictest1753-21gnome-shell10:35:160
162399460,32rtkit-daemon1753-21gnome-shell09:32:321
462624994521,20cyclictest1753-21gnome-shell08:19:103
462624994510,16cyclictest1753-21gnome-shell09:25:083
46262299456,16cyclictest1753-21gnome-shell10:43:032
46262299456,16cyclictest1753-21gnome-shell10:43:032
46262299456,16cyclictest1753-21gnome-shell08:24:422
462622994514,26cyclictest557491-21grep11:35:142
46261699459,16cyclictest1753-21gnome-shell08:03:240
46261699458,17cyclictest1753-21gnome-shell10:31:160
46261699458,16cyclictest1753-21gnome-shell09:16:040
46261699456,14cyclictest855-21dbus-daemon07:35:020
46261699452,17cyclictest1753-21gnome-shell09:29:010
46261699451,39cyclictest2197014-21PK-Backend08:10:510
462616994513,10cyclictest503837-21cut09:05:000
46261699451,24cyclictest1753-21gnome-shell07:11:340
462616994511,12cyclictest543963-21sshd10:56:340
462616994510,14cyclictest1-21systemd08:00:010
162399450,39rtkit-daemon881143-21gmain07:20:373
4921712440,13sleep126-21ksoftirqd/108:30:221
462624994413,27cyclictest501857-40lsb_release08:57:253
462624994413,26cyclictest536522-21grep10:35:253
46262299448,20cyclictest1753-21gnome-shell12:18:152
46262299446,11cyclictest1753-21gnome-shell11:18:332
46262299440,26cyclictest854-21avahi-daemon12:22:582
462618994416,5cyclictest141rcu_preempt11:30:401
46261899441,10cyclictest0-21swapper/109:13:571
46261699446,18cyclictest1753-21gnome-shell10:06:300
462616994418,22cyclictest2197014-21PK-Backend11:04:080
462616994418,22cyclictest2197014-21PK-Backend11:04:080
462616994418,21cyclictest1069-21runrttasks08:35:280
162399440,32rtkit-daemon2197014-21PK-Backend11:24:553
5102172430,5sleep02197014-21PK-Backend09:20:320
4962262430,9sleep126-21ksoftirqd/108:40:331
462624994325,13cyclictest438-20systemd-journal11:00:343
462624994325,13cyclictest438-20systemd-journal11:00:343
46262299435,16cyclictest1753-21gnome-shell10:54:222
46262299435,16cyclictest1753-21gnome-shell09:18:352
46262299435,16cyclictest1753-21gnome-shell08:27:112
46262299435,15cyclictest514410-21grep09:35:022
46262299435,15cyclictest481674-21df_inode08:00:192
46262299434,16cyclictest567399-21cron12:05:002
46262299434,15cyclictest553795-21expr11:25:142
46261899436,12cyclictest26-21ksoftirqd/110:30:151
462616994314,8cyclictest2197014-21PK-Backend07:44:140
46261699431,38cyclictest531867-21pool-udisksd10:21:560
46261699431,23cyclictest1753-21gnome-shell09:54:520
462616994311,8cyclictest438-20systemd-journal11:45:000
46262499429,15cyclictest1753-21gnome-shell10:22:353
46262499427,31cyclictest1753-21gnome-shell12:12:023
462624994223,14cyclictest555898-21df11:30:153
462624994217,7cyclictest1753-21gnome-shell10:00:253
462624994214,7cyclictest5576582expr11:35:153
46262299428,9cyclictest0-21swapper/207:21:162
46262299425,13cyclictest1753-21gnome-shell12:39:282
46261899429,30cyclictest231irq_work/112:15:581
46261899427,13cyclictest26-21ksoftirqd/107:30:001
462618994230,8cyclictest1753-21gnome-shell09:45:161
462618994224,15cyclictest251rcuc/107:19:051
46261699428,12cyclictest1753-21gnome-shell08:55:140
462616994215,7cyclictest1753-21gnome-shell12:37:090
462616994214,7cyclictest538575-21timerandwakeup10:40:470
462616994214,7cyclictest538575-21timerandwakeup10:40:470
462616994212,8cyclictest569417-21cpu12:10:150
5486782410,5sleep30-21swapper/311:10:153
46262299415,10cyclictest1753-21gnome-shell11:20:272
46262299414,13cyclictest576306-21/usr/sbin/munin12:30:322
46262299414,12cyclictest1753-21gnome-shell07:14:322
46262299413,15cyclictest544217-21apt-get10:59:342
46261899411,9cyclictest0-21swapper/111:20:241
46262299404,12cyclictest1753-21gnome-shell11:42:452
46261899403,17cyclictest542560-21kworker/1:211:18:211
46261899402,34cyclictest1753-21gnome-shell10:46:511
462618994019,17cyclictest511821-21processes09:25:271
162399400,33rtkit-daemon26-21ksoftirqd/111:45:131
46262499392,33cyclictest1753-21gnome-shell09:44:163
46262499392,33cyclictest1753-21gnome-shell07:48:343
462624993923,12cyclictest42-21ksoftirqd/307:25:153
462624993922,13cyclictest481021-21sh08:00:003
462624993916,20cyclictest391irq_work/310:12:563
462624993916,19cyclictest1753-21gnome-shell07:50:213
462624993914,9cyclictest518124-21apt-key09:45:043
462624993914,21cyclictest560486-21sh11:45:003
46262299394,11cyclictest1753-21gnome-shell08:32:572
46261899394,8cyclictest26-21ksoftirqd/109:05:001
46261899391,19cyclictest356-21jbd2/sdb2-811:40:001
46261899391,14cyclictest489699-21idleruntime-cro08:25:001
462624993821,13cyclictest563229-21ntpq11:50:253
462624993821,13cyclictest544532-21unattended-upgr10:59:513
462624993821,12cyclictest438-20systemd-journal08:04:343
462624993820,14cyclictest1753-21gnome-shell11:07:313
462624993820,13cyclictest485083-21cstates08:10:173
462624993814,7cyclictest490690-21ntp_offset08:25:263
462624993814,6cyclictest391irq_work/308:41:283
46261899384,29cyclictest510776-21unin-run09:25:011
462618993820,14cyclictest522075-21df09:55:201
462618993819,16cyclictest251rcuc/109:06:081
462618993816,18cyclictest470722-21cpuspeed_turbos07:30:151
46261899381,20cyclictest141rcu_preempt11:00:171
46261899381,20cyclictest141rcu_preempt11:00:171
462616993812,7cyclictest2197014-21PK-Backend07:30:000
162399380,25rtkit-daemon0-21swapper/112:35:511
46262499376,13cyclictest1753-21gnome-shell09:50:203
462624993725,9cyclictest462813-21/usr/sbin/munin07:10:343
462624993720,13cyclictest884-21gdbus07:35:013
462624993720,12cyclictest530263-21sshd10:18:343
462624993720,12cyclictest530263-21sshd10:18:343
462624993719,14cyclictest1753-21gnome-shell07:15:193
462624993719,13cyclictest534993-21python310:30:303
462624993716,18cyclictest1069-21runrttasks12:01:103
462622993722,11cyclictest563823-21sshd11:53:342
46261899374,19cyclictest26-21ksoftirqd/108:07:391
46261899374,18cyclictest26-21ksoftirqd/111:10:001
46261899373,17cyclictest26-21ksoftirqd/107:38:181
462618993726,7cyclictest1753-21gnome-shell10:51:321
462618993725,8cyclictest1753-21gnome-shell07:10:581
462618993720,13cyclictest549816-21/usr/sbin/munin11:15:001
462618993720,12cyclictest573931-21cat12:20:331
462618993719,13cyclictest26-21ksoftirqd/108:15:271
46261899371,6cyclictest141rcu_preempt10:15:231
46261899371,6cyclictest141rcu_preempt10:15:231
462618993715,8cyclictest26-21ksoftirqd/110:55:351
46261899371,19cyclictest1753-21gnome-shell08:00:341
46261899371,14cyclictest1753-21gnome-shell10:20:221
162399370,27rtkit-daemon0-21swapper/109:43:101
162399370,25rtkit-daemon0-21swapper/112:30:151
162399370,25rtkit-daemon0-21swapper/107:48:431
462624993624,8cyclictest1753-21gnome-shell11:56:153
46262499362,30cyclictest1753-21gnome-shell07:39:363
462624993620,12cyclictest494134-21ntpq08:35:243
462624993619,13cyclictest507312-21pool-org.gnome.09:15:013
462624993619,12cyclictest550308-21expr11:15:153
462624993619,12cyclictest468-21systemd-udevd10:06:503
462624993619,12cyclictest438-20systemd-journal08:35:003
462624993613,20cyclictest391irq_work/312:19:073
462624993612,10cyclictest42-21ksoftirqd/309:15:133
462622993622,10cyclictest34-21ksoftirqd/210:05:162
46261899369,8cyclictest26-21ksoftirqd/108:14:391
46261899364,16cyclictest473827-21kworker/1:007:50:121
462618993622,12cyclictest251rcuc/109:15:531
462618993618,15cyclictest251rcuc/110:06:221
46261899361,11cyclictest501758-21taskset08:56:251
46261899361,11cyclictest467938-21awk07:20:321
4834862350,8chrt2197014-21PK-Backend08:05:213
462624993522,9cyclictest42-21ksoftirqd/308:50:003
462624993521,10cyclictest855-21dbus-daemon09:31:063
462624993520,11cyclictest537935-21expr10:40:163
462624993520,11cyclictest537935-21expr10:40:163
462624993514,17cyclictest2197014-21PK-Backend09:55:253
462624993513,19cyclictest391irq_work/311:48:353
46262499351,30cyclictest578546-21irqstats12:35:223
462618993524,7cyclictest1753-21gnome-shell09:35:241
462618993520,11cyclictest480233-21cut07:55:241
462618993518,13cyclictest527160-21expr10:10:151
462618993513,6cyclictest141rcu_preempt10:43:171
462618993513,6cyclictest141rcu_preempt10:43:161
46262499342,9cyclictest1753-21gnome-shell09:39:333
462624993422,8cyclictest1753-21gnome-shell12:05:293
462624993422,8cyclictest1753-21gnome-shell10:52:313
462624993419,11cyclictest42-21ksoftirqd/307:40:133
462624993418,12cyclictest489400-21sshd08:21:343
46261899342,28cyclictest537288-21lsb_release10:39:421
46261899341,9cyclictest0-21swapper/108:26:481
462618993418,12cyclictest494179-21awk08:35:261
46262499331,29cyclictest2197014-21PK-Backend09:05:483
462618993322,7cyclictest532564-21cpuspeed_turbos10:25:141
462618993313,6cyclictest0-21swapper/107:40:261
462624993210,18cyclictest1753-21gnome-shell12:26:333
462624993019,7cyclictest572848-21/usr/sbin/munin12:20:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional