You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-03 - 06:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Mon Nov 03, 2025 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
380069322520,9sleep11753-21gnome-shell19:05:261
380725221130,8sleep22197014-21PK-Backend19:25:012
38079072880,5sleep00-21swapper/019:25:240
162399850,69rtkit-daemon0-21swapper/319:05:143
380123528465,13sleep20-21swapper/219:09:222
38092212830,8sleep32197014-21PK-Backend19:30:143
39080542760,5sleep21753-21gnome-shell00:10:262
162399740,10rtkit-daemon1622-21rtkit-daemon21:40:091
162399740,10rtkit-daemon1622-21rtkit-daemon21:40:091
380142699692,45cyclictest1753-21gnome-shell22:48:190
380105726945,16sleep00-21swapper/019:07:010
380142699672,36cyclictest3841205-40apt-check21:00:270
39048772660,13sleep126-21ksoftirqd/100:00:281
39048772660,13sleep126-21ksoftirqd/100:00:271
38398902660,8chrt2197014-21PK-Backend20:55:240
380142699662,46cyclictest3868019-40lsb_release22:15:210
380142699662,34cyclictest3811426-21df19:35:140
380142699661,38cyclictest3845343-21sed21:10:220
380142699653,43cyclictest3883730-21irqstats23:00:210
380142699652,37cyclictest1753-21gnome-shell23:54:180
380142699652,36cyclictest3822977-31apt-check20:06:300
38750042640,8chrt0-21swapper/322:35:213
380142699642,45cyclictest1753-21gnome-shell20:34:410
380142699642,39cyclictest1753-21gnome-shell22:08:490
380142699642,39cyclictest1753-21gnome-shell22:08:490
380142699642,39cyclictest1753-21gnome-shell21:39:430
380142699633,39cyclictest38200172cat20:00:140
380142699632,45cyclictest3907525-21/usr/sbin/munin00:10:280
380142699632,44cyclictest1753-21gnome-shell22:22:480
380142699632,36cyclictest1753-21gnome-shell00:36:270
380142699632,35cyclictest1753-21gnome-shell21:23:470
380142699632,35cyclictest1753-21gnome-shell21:23:470
380142699632,32cyclictest3912819-21munin-run00:25:010
380142699631,45cyclictest3876633-21expr22:40:160
380142699631,38cyclictest3880447-21grep22:50:240
380142699631,38cyclictest3871270-21cpuspeed_turbos22:25:180
162399630,7rtkit-daemon1622-21rtkit-daemon20:19:201
380142699624,52cyclictest1753-21gnome-shell20:35:150
380142699621,35cyclictest3855978-21/usr/sbin/munin21:40:270
380142699621,35cyclictest3855978-21/usr/sbin/munin21:40:270
380142699621,34cyclictest3913929-21python300:25:300
380142699621,34cyclictest3896034-21uname23:35:230
380142699621,31cyclictest3874825-21expr22:35:140
380142699612,35cyclictest1753-21gnome-shell22:11:330
380142699612,35cyclictest1753-21gnome-shell21:19:380
380142699612,33cyclictest0-21swapper/020:27:410
380142699611,34cyclictest3825629-21cstates20:15:150
162399610,9rtkit-daemon1622-21rtkit-daemon22:51:391
162399610,9rtkit-daemon1622-21rtkit-daemon21:05:221
380142699602,36cyclictest1753-21gnome-shell23:45:090
380142699601,35cyclictest3888817-21df_inode23:15:190
380142699601,35cyclictest3882639-21taskset22:55:510
162399600,9rtkit-daemon1622-21rtkit-daemon20:45:591
3801438995916,6cyclictest141rcu_preempt21:35:043
380142699592,34cyclictest1753-21gnome-shell21:53:000
380142699592,34cyclictest1753-21gnome-shell20:52:470
380142699592,34cyclictest1753-21gnome-shell20:42:400
380142699592,34cyclictest1753-21gnome-shell00:32:560
380142699592,33cyclictest1753-21gnome-shell19:42:510
380142699592,32cyclictest1753-21gnome-shell19:10:320
380142699592,30cyclictest3809457-21hddtemp_smartct19:30:190
380142699592,29cyclictest3857683-21ntp_states21:45:280
3801426995922,32cyclictest3891673-21pool-org.gnome.23:25:010
3801426995919,34cyclictest438-20systemd-journal23:07:390
380142699591,33cyclictest3835465-21perf20:49:540
380142699591,32cyclictest3905995-21expr00:05:140
162399590,9rtkit-daemon1622-21rtkit-daemon21:23:121
162399590,9rtkit-daemon1622-21rtkit-daemon21:23:121
380142699582,33cyclictest3818437-21hddtemp_smartct19:55:230
380142699582,33cyclictest1753-21gnome-shell23:42:280
380142699582,33cyclictest1753-21gnome-shell22:34:140
3801426995823,31cyclictest3242253-21nautilus00:00:020
380142699582,29cyclictest3860760-21df21:55:150
380142699572,33cyclictest1753-21gnome-shell19:46:410
380142699572,32cyclictest1753-21gnome-shell19:18:080
380142699572,30cyclictest3827421-21hddtemp_smartct20:20:220
380142699571,31cyclictest0-21swapper/000:15:220
380143099560,52cyclictest1622-21rtkit-daemon21:45:211
162399560,8rtkit-daemon1622-21rtkit-daemon21:31:011
39097962550,9chrt1753-21gnome-shell00:15:222
38978462550,9sleep234-21ksoftirqd/223:40:242
380142699552,29cyclictest0-21swapper/021:30:200
380142699551,31cyclictest0-21swapper/020:10:270
162399550,8rtkit-daemon1622-21rtkit-daemon20:55:561
3801426995421,29cyclictest3851146-21sshd21:27:130
3801426995416,33cyclictest3905555-21/usr/sbin/munin00:05:010
3801426995416,33cyclictest3905555-21/usr/sbin/munin00:05:010
162399540,8rtkit-daemon1622-21rtkit-daemon21:28:111
162399530,5rtkit-daemon26-21ksoftirqd/123:40:381
162399530,10rtkit-daemon1753-21gnome-shell22:47:532
3801426995220,28cyclictest1069-21runrttasks21:08:260
380143899503,23cyclictest141rcu_preempt00:20:023
3801434995037,9cyclictest34-21ksoftirqd/220:00:022
3801430995030,16cyclictest3862873-21ntp_states22:00:271
380142699502,43cyclictest1753-21gnome-shell19:20:280
380142699502,26cyclictest3887301-21memory23:10:250
3801426995010,35cyclictest3894308-21tr23:30:240
3801438994912,20cyclictest141rcu_preempt21:00:163
3801434994925,8cyclictest1753-21gnome-shell23:35:172
3801426994910,34cyclictest1753-21gnome-shell23:25:160
162399490,43rtkit-daemon1775-21gvfs-afc-volume20:42:171
38089732480,8chrt2197014-21PK-Backend19:30:023
380143499489,22cyclictest1753-21gnome-shell19:34:262
3801434994822,14cyclictest34-21ksoftirqd/220:00:172
3801430994826,17cyclictest3861865-21/usr/sbin/munin22:00:021
162399480,34rtkit-daemon2197014-21PK-Backend22:41:583
380143899471,16cyclictest860997-21snmpd22:11:303
3801438994711,18cyclictest42-21ksoftirqd/320:23:273
380143499479,21cyclictest1753-21gnome-shell19:15:252
380142699479,34cyclictest1753-21gnome-shell19:50:270
380142699479,34cyclictest1753-21gnome-shell19:50:270
3801438994624,18cyclictest1753-21gnome-shell22:05:343
3801438994624,18cyclictest1753-21gnome-shell22:05:343
3801438994621,10cyclictest42-21ksoftirqd/321:45:023
3801438994621,10cyclictest42-21ksoftirqd/321:45:023
380143899461,16cyclictest3870931-21latency_hist22:25:023
380143899461,16cyclictest3814424-21sort19:45:023
380143499468,21cyclictest1753-21gnome-shell21:34:222
380143499461,31cyclictest34-21ksoftirqd/200:00:282
380143499461,31cyclictest34-21ksoftirqd/200:00:282
3801434994611,17cyclictest34-21ksoftirqd/221:40:162
3801434994611,17cyclictest34-21ksoftirqd/221:40:162
162399460,39rtkit-daemon1903-21packagekitd23:01:001
3801438994532,9cyclictest1753-21gnome-shell23:11:503
380143899452,20cyclictest1753-21gnome-shell20:15:173
380143899452,19cyclictest1753-21gnome-shell00:00:143
380143899452,19cyclictest1753-21gnome-shell00:00:143
380143899452,15cyclictest3867941-21cstates22:15:163
380143899452,13cyclictest3242253-21nautilus21:00:023
3801438994518,13cyclictest42-21ksoftirqd/300:21:293
380143899451,15cyclictest2197014-21PK-Backend20:35:013
380143499457,20cyclictest1753-21gnome-shell23:20:292
380142699451,39cyclictest3862399-21cstates22:00:150
162399450,33rtkit-daemon2197014-21PK-Backend23:46:433
162399450,31rtkit-daemon3915464-21strings00:30:253
39030282440,7sleep32197014-21PK-Backend23:55:273
380143899444,17cyclictest3879524-21munin-run22:50:003
3801438994426,15cyclictest42-21ksoftirqd/320:50:023
3801438994424,15cyclictest42-21ksoftirqd/323:45:003
3801438994416,9cyclictest42-21ksoftirqd/320:10:013
380143899441,27cyclictest2197014-21PK-Backend20:10:203
380143899441,18cyclictest3812552-21cron19:39:593
3801438994411,10cyclictest42-21ksoftirqd/320:25:273
3801434994425,6cyclictest34-21ksoftirqd/223:50:262
3801434994425,6cyclictest34-21ksoftirqd/221:05:012
3801434994424,6cyclictest34-21ksoftirqd/222:15:232
3801434994422,17cyclictest34-21ksoftirqd/219:40:332
3801434994421,20cyclictest3832023-21awk20:35:002
3801434994414,18cyclictest34-21ksoftirqd/220:53:052
3801430994425,13cyclictest438-20systemd-journal23:30:171
3801438994321,16cyclictest42-21ksoftirqd/321:30:243
380143899431,26cyclictest141rcu_preempt22:30:303
380143899431,19cyclictest42-21ksoftirqd/321:45:153
380143499433,15cyclictest0-21swapper/222:55:182
3801434994325,14cyclictest34-21ksoftirqd/220:49:342
3801434994324,14cyclictest34-21ksoftirqd/221:10:192
380143499432,17cyclictest1753-21gnome-shell21:55:332
3801430994326,13cyclictest3810529-21apt-get19:35:001
3801430994326,12cyclictest438-20systemd-journal00:20:001
3801438994225,6cyclictest42-21ksoftirqd/323:26:543
3801438994224,14cyclictest42-21ksoftirqd/321:16:153
3801438994223,14cyclictest42-21ksoftirqd/320:39:083
3801438994222,14cyclictest42-21ksoftirqd/321:55:073
3801438994215,23cyclictest3821420-21cut20:05:013
3801438994213,6cyclictest141rcu_preempt19:50:213
3801438994213,6cyclictest141rcu_preempt19:50:213
3801434994224,6cyclictest34-21ksoftirqd/223:55:192
3801434994224,6cyclictest34-21ksoftirqd/221:20:322
3801434994224,6cyclictest34-21ksoftirqd/221:20:322
3801434994224,6cyclictest34-21ksoftirqd/221:05:152
3801434994224,6cyclictest34-21ksoftirqd/219:36:262
3801434994224,6cyclictest34-21ksoftirqd/200:20:262
3801434994224,14cyclictest34-21ksoftirqd/221:46:202
3801434994223,6cyclictest34-21ksoftirqd/222:40:192
3801434994223,15cyclictest34-21ksoftirqd/222:02:202
3801430994225,13cyclictest1-21systemd20:25:001
3801430994222,16cyclictest3807051-21dump-pmu-power19:25:001
3801430994222,15cyclictest3242253-21nautilus19:45:011
162399420,30rtkit-daemon2197014-21PK-Backend22:00:523
39176122410,7sleep32197014-21PK-Backend00:35:363
39174412410,5chrt0-21swapper/100:35:341
38877192410,8chrt1753-21gnome-shell23:10:342
3801438994124,13cyclictest42-21ksoftirqd/300:14:453
3801438994123,13cyclictest42-21ksoftirqd/321:52:353
3801438994122,6cyclictest42-21ksoftirqd/319:10:233
3801438994121,6cyclictest42-21ksoftirqd/323:30:043
380143899411,15cyclictest2197014-21PK-Backend19:15:063
3801434994124,13cyclictest34-21ksoftirqd/219:53:292
3801434994124,13cyclictest34-21ksoftirqd/219:53:292
3801434994123,6cyclictest34-21ksoftirqd/220:35:062
3801434994123,6cyclictest34-21ksoftirqd/200:40:002
3801434994123,14cyclictest34-21ksoftirqd/222:36:582
3801434994123,14cyclictest34-21ksoftirqd/220:28:072
3801434994122,6cyclictest34-21ksoftirqd/223:30:142
3801434994122,14cyclictest34-21ksoftirqd/221:50:322
3801434994115,10cyclictest34-21ksoftirqd/222:34:402
3801434994114,8cyclictest34-21ksoftirqd/219:25:492
380143499411,21cyclictest3828488-21turbostat20:25:002
380143499411,10cyclictest2197014-21PK-Backend22:25:012
3801430994121,6cyclictest26-21ksoftirqd/120:05:011
3801430994117,19cyclictest3841667-40lsb_release21:00:221
380143899402,34cyclictest3913189-21cpuspeed_turbos00:25:153
3801438994022,6cyclictest42-21ksoftirqd/322:25:193
3801438994020,14cyclictest42-21ksoftirqd/321:05:093
380143899401,9cyclictest2197014-21PK-Backend21:14:503
3801438994016,6cyclictest141rcu_preempt23:20:183
380143899401,18cyclictest3902049-21perf23:55:013
3801434994023,6cyclictest34-21ksoftirqd/220:55:142
3801434994022,14cyclictest34-21ksoftirqd/221:29:002
3801434994022,14cyclictest34-21ksoftirqd/220:10:242
3801434994022,14cyclictest34-21ksoftirqd/219:45:362
3801434994021,6cyclictest34-21ksoftirqd/223:00:142
3801434994021,6cyclictest34-21ksoftirqd/200:34:232
3801434994014,5cyclictest141rcu_preempt23:15:102
3801430994023,13cyclictest855-21dbus-daemon19:40:001
3801430994022,14cyclictest3867578-40apt-check22:15:141
3801430994022,13cyclictest438-20systemd-journal20:34:551
3801430994022,13cyclictest3822629-21sensors_temp20:05:291
162399400,28rtkit-daemon2197014-21PK-Backend23:37:083
3801438993928,7cyclictest1753-21gnome-shell20:53:053
3801438993928,7cyclictest1753-21gnome-shell19:22:273
3801438993924,11cyclictest42-21ksoftirqd/320:42:463
3801438993914,6cyclictest141rcu_preempt23:00:273
3801438993913,6cyclictest141rcu_preempt23:10:003
380143899391,17cyclictest1753-21gnome-shell00:05:153
3801438993911,6cyclictest141rcu_preempt22:50:303
3801434993922,13cyclictest34-21ksoftirqd/220:05:342
3801434993921,6cyclictest34-21ksoftirqd/219:14:512
3801434993920,6cyclictest34-21ksoftirqd/222:51:332
3801434993920,6cyclictest34-21ksoftirqd/222:25:182
3801434993920,6cyclictest34-21ksoftirqd/221:19:482
3801434993914,10cyclictest34-21ksoftirqd/200:28:452
3801434993911,11cyclictest141rcu_preempt23:10:012
380143099396,5cyclictest141rcu_preempt21:18:501
3801430993927,8cyclictest1753-21gnome-shell23:50:431
3801430993926,10cyclictest3817773-21awk19:55:001
3801430993926,10cyclictest3817773-21awk19:55:001
3801430993924,11cyclictest3912428-40apt-check00:21:161
3801430993922,13cyclictest855-21dbus-daemon22:15:011
3801430993922,13cyclictest3242255-21gdbus20:15:011
3801438993827,7cyclictest1753-21gnome-shell22:58:183
3801438993827,7cyclictest1753-21gnome-shell19:56:383
3801438993820,6cyclictest42-21ksoftirqd/319:45:253
380143899381,17cyclictest3850264-21cstates21:25:173
380143899381,16cyclictest2197014-21PK-Backend23:15:273
380143499389,10cyclictest141rcu_preempt00:07:542
3801434993820,6cyclictest34-21ksoftirqd/222:05:432
3801434993820,6cyclictest34-21ksoftirqd/222:05:422
3801434993816,5cyclictest141rcu_preempt22:11:192
3801434993812,5cyclictest141rcu_preempt20:20:022
3801434993810,6cyclictest141rcu_preempt20:40:212
3801430993827,7cyclictest3907741-21taskset00:10:181
3801430993821,13cyclictest26-21ksoftirqd/123:40:011
3801430993820,14cyclictest1632-21dbus-daemon21:40:011
3801430993820,13cyclictest438-20systemd-journal22:35:001
380143099381,4cyclictest0-21swapper/123:23:491
380143899371,15cyclictest2197014-21PK-Backend21:20:233
380143899371,15cyclictest2197014-21PK-Backend21:20:223
380143499378,5cyclictest141rcu_preempt21:35:052
3801430993727,7cyclictest3838143-21needreboot20:50:251
380143099372,30cyclictest1-21systemd23:50:011
3801430993722,11cyclictest3815990-21sshd19:49:411
380143099372,12cyclictest1753-21gnome-shell22:25:171
3801430993720,13cyclictest3903821-21systemd-run00:00:021
380143099371,10cyclictest0-21swapper/120:39:421
380143099366,17cyclictest3878025-21/usr/sbin/munin22:45:281
3801430993621,11cyclictest3916239-21cron00:35:001
3801430993619,13cyclictest3859684-21grep21:50:311
3801430993619,13cyclictest26-21ksoftirqd/119:15:251
3801430993615,17cyclictest3829476-21ntp_kernel_pll_20:25:251
38023402350,18sleep126-21ksoftirqd/119:10:241
380143499357,9cyclictest2197014-21PK-Backend23:45:232
380143499357,11cyclictest141rcu_preempt23:25:332
3801430993520,11cyclictest3819183-21seq19:56:121
380143099351,8cyclictest0-21swapper/122:35:411
380143099343,5cyclictest141rcu_preempt22:55:321
3801430993421,9cyclictest1753-21gnome-shell00:09:501
380143099342,15cyclictest3888826-21diskstats23:15:201
3801430993420,10cyclictest3877808-21systemctl22:45:011
3801430993420,10cyclictest3871048-21sed22:25:021
3801430993419,11cyclictest3914505-21cron00:30:001
3801430993418,12cyclictest3845080-21df21:10:161
3801430993411,10cyclictest3891924-21cpu23:25:161
3801430993411,10cyclictest141rcu_preempt19:25:291
380143099332,14cyclictest3885363-21diskstats23:05:211
3801430993319,10cyclictest3887206-21cut23:10:251
380143099331,10cyclictest3863849-21sort22:05:171
380143099331,10cyclictest3863849-21sort22:05:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional