You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 05:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Tue Jul 01, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
377014922580,8sleep20-21swapper/219:06:372
377418521110,5sleep00-21swapper/019:16:570
38556552990,8sleep22197014-21PK-Backend23:10:162
162399950,77rtkit-daemon0-21swapper/019:05:090
162399920,5rtkit-daemon0-21swapper/022:32:100
162399830,6rtkit-daemon0-21swapper/021:10:300
162399790,6rtkit-daemon0-21swapper/020:37:050
162399790,5rtkit-daemon0-21swapper/022:02:320
38701432750,9sleep30-21swapper/323:50:293
162399740,10rtkit-daemon970-21ntpd00:24:101
377032227341,26sleep10-21swapper/119:08:501
162399730,11rtkit-daemon970-21ntpd23:49:061
38810672720,5sleep10-21swapper/100:25:211
162399720,9rtkit-daemon881143-21gmain23:31:171
162399720,6rtkit-daemon970-21ntpd00:13:451
162399710,7rtkit-daemon970-21ntpd23:11:441
162399710,4rtkit-daemon0-21swapper/119:37:091
377026626954,9sleep30-21swapper/319:08:093
162399690,8rtkit-daemon970-21ntpd19:45:401
162399620,9rtkit-daemon1622-21rtkit-daemon23:50:271
377055599610,56cyclictest1622-21rtkit-daemon00:09:381
3770550996119,14cyclictest3882751-21df00:30:170
38621252600,14chrt1753-21gnome-shell23:30:013
3770550996029,26cyclictest3794237-21diskstats20:15:220
3770550996022,9cyclictest3810962-21mktemp21:05:020
3770550996022,9cyclictest3810962-21mktemp21:05:010
162399600,9rtkit-daemon1622-21rtkit-daemon23:55:270
3770550995819,12cyclictest3791272-21grep20:05:290
3770550995816,16cyclictest1753-21gnome-shell19:13:000
162399580,8rtkit-daemon1622-21rtkit-daemon20:12:580
377055099579,18cyclictest1753-21gnome-shell21:27:360
3770550995713,14cyclictest875-21polkitd19:50:370
3770550995710,16cyclictest1753-21gnome-shell20:34:540
38258712560,25sleep2331rcuc/221:45:272
3770550995612,14cyclictest3242255-21gdbus00:25:020
3770550995612,13cyclictest468-21systemd-udevd22:44:570
377055099559,16cyclictest1753-21gnome-shell21:47:070
377055099558,16cyclictest1753-21gnome-shell20:50:410
377055099558,16cyclictest1753-21gnome-shell20:21:060
377055099557,17cyclictest1753-21gnome-shell21:58:200
377055099553,24cyclictest3857963-21proc_pri23:15:270
3770550995519,7cyclictest3850218-21/usr/sbin/munin22:55:220
3770550995519,7cyclictest3850218-21/usr/sbin/munin22:55:220
3770550995514,11cyclictest3804397-21grep20:45:140
3770550995514,11cyclictest3804397-21grep20:45:140
3770550995510,16cyclictest1753-21gnome-shell21:10:000
162399550,8rtkit-daemon1622-21rtkit-daemon23:46:240
377055099549,17cyclictest1753-21gnome-shell23:10:110
377055099548,17cyclictest1753-21gnome-shell20:27:060
377055099548,16cyclictest1753-21gnome-shell00:13:460
377055099546,18cyclictest1753-21gnome-shell21:15:590
377055099541,8cyclictest1753-21gnome-shell21:39:060
3770550995418,7cyclictest3870050-21sh23:50:280
3770550995416,11cyclictest438-20systemd-journal21:54:560
377055099541,23cyclictest1753-21gnome-shell22:38:060
377055099541,22cyclictest3817359-21uname21:20:280
3770550995410,18cyclictest1753-21gnome-shell19:24:170
3770550995410,18cyclictest1753-21gnome-shell19:24:160
377055099539,16cyclictest1753-21gnome-shell23:04:310
377055099539,16cyclictest1753-21gnome-shell19:35:470
377055099539,16cyclictest1753-21gnome-shell19:28:130
377055099536,17cyclictest1753-21gnome-shell23:08:070
377055099531,21cyclictest3861203-21cat23:25:230
3770550995310,17cyclictest1753-21gnome-shell22:19:100
37948682520,6sleep10-21swapper/120:15:341
3770555995210,14cyclictest26-21ksoftirqd/120:45:451
3770555995210,14cyclictest26-21ksoftirqd/120:45:441
377055099529,17cyclictest1753-21gnome-shell22:11:370
377055099529,16cyclictest1753-21gnome-shell23:23:320
377055099529,16cyclictest1753-21gnome-shell19:43:470
377055099529,16cyclictest1753-21gnome-shell00:02:050
377055099528,18cyclictest1753-21gnome-shell22:06:060
377055099528,17cyclictest1753-21gnome-shell00:06:060
3770550995216,7cyclictest3840333-21grep22:25:250
3770550995216,7cyclictest1753-21gnome-shell23:41:050
377055099519,16cyclictest1753-21gnome-shell23:31:060
377055099519,16cyclictest1753-21gnome-shell21:40:310
377055099519,16cyclictest1753-21gnome-shell00:19:350
3770550995117,7cyclictest1069-21runrttasks00:25:370
3770550995115,7cyclictest1753-21gnome-shell00:35:320
377055099511,25cyclictest1753-21gnome-shell19:30:350
3770550995112,11cyclictest3864898-21proc_pri23:35:300
3770550995110,11cyclictest3788709-21cpuspeed_turbos20:00:130
377055099508,16cyclictest1753-21gnome-shell20:56:390
377055099508,16cyclictest1753-21gnome-shell19:47:460
3770550995019,7cyclictest1069-21runrttasks22:50:290
377055099501,23cyclictest1753-21gnome-shell19:59:260
3770556994927,18cyclictest3797060-21cut20:25:012
377055599499,24cyclictest3855023-21seq23:08:241
377055599497,8cyclictest26-21ksoftirqd/119:25:151
377055599484,12cyclictest3242253-21nautilus21:50:011
3770555994814,20cyclictest3791157-21processes20:05:281
38574482470,6sleep22197014-21PK-Backend23:15:162
377055599475,21cyclictest904-21systemd-logind22:44:571
3770555994712,13cyclictest26-21ksoftirqd/120:05:001
377055099472,20cyclictest1753-21gnome-shell20:40:340
3770559994624,19cyclictest0-21swapper/320:36:353
377055699465,19cyclictest1753-21gnome-shell00:15:282
377055699463,21cyclictest3866539-21netstat23:40:242
3770556994618,24cyclictest3772204-21cron19:15:002
377055599463,22cyclictest3831306-21cat22:00:201
377055099462,17cyclictest1753-21gnome-shell22:20:070
38604342450,7sleep32197014-21PK-Backend23:25:023
3770559994526,13cyclictest855-21dbus-daemon00:00:063
3770559994521,8cyclictest3812161-21ntp_kernel_pll_21:05:273
3770556994528,12cyclictest1800-21upowerd19:50:352
3770559994412,5cyclictest141rcu_preempt00:10:103
377055699442,13cyclictest1753-21gnome-shell00:12:062
377055599447,10cyclictest26-21ksoftirqd/120:20:211
377055599444,16cyclictest3788351-21turbostat20:00:001
377055599442,13cyclictest3807484-21latency_hist20:55:011
3770555994414,16cyclictest26-21ksoftirqd/122:55:021
377055599441,35cyclictest3873338-21grep00:00:211
377055599441,12cyclictest3242253-21nautilus21:55:021
3770555994410,18cyclictest26-21ksoftirqd/123:35:151
3770559994319,20cyclictest42-21ksoftirqd/322:35:143
3770556994326,13cyclictest34-21ksoftirqd/222:20:012
3770556994325,13cyclictest438-20systemd-journal20:00:452
377055699431,9cyclictest2197014-21PK-Backend19:24:462
377055699431,9cyclictest2197014-21PK-Backend19:24:462
3770556994312,12cyclictest141rcu_preempt20:35:142
3770555994322,16cyclictest3242253-21nautilus22:35:021
3770555994320,7cyclictest26-21ksoftirqd/122:48:251
377055599431,18cyclictest1753-21gnome-shell00:35:291
377055599431,13cyclictest26-21ksoftirqd/121:59:271
3770555994311,18cyclictest26-21ksoftirqd/121:30:261
3770555994311,12cyclictest26-21ksoftirqd/121:05:341
38643342420,7chrt2197014-21PK-Backend23:35:152
377055999421,37cyclictest1753-21gnome-shell23:10:243
3770556994227,10cyclictest3846448-21systemd-udevd22:44:572
3770556994213,6cyclictest141rcu_preempt23:54:082
377055599429,13cyclictest26-21ksoftirqd/122:35:161
377055599429,12cyclictest26-21ksoftirqd/123:58:311
377055599428,12cyclictest26-21ksoftirqd/120:25:031
377055599428,11cyclictest26-21ksoftirqd/121:25:011
377055599424,20cyclictest26-21ksoftirqd/100:30:081
377055599423,18cyclictest26-21ksoftirqd/123:20:411
3770555994216,22cyclictest3834040-21munin-run22:10:011
377055599421,17cyclictest26-21ksoftirqd/120:33:021
377055599421,12cyclictest3835060-21ntp_states22:10:311
3770555994210,12cyclictest26-21ksoftirqd/122:22:471
162399420,31rtkit-daemon2197014-21PK-Backend00:07:422
38773922410,6sleep32197014-21PK-Backend00:15:213
38454432410,9chrt1753-21gnome-shell22:40:223
377055999417,7cyclictest2197014-21PK-Backend21:50:153
3770559994129,8cyclictest42-21ksoftirqd/300:00:033
3770559994128,10cyclictest0-21swapper/321:11:533
377055999412,35cyclictest1753-21gnome-shell00:38:363
377055699418,6cyclictest141rcu_preempt22:23:052
3770556994124,13cyclictest3873651-21sensors_temp00:00:282
3770556994123,14cyclictest3872142-21sshd23:55:472
3770556994121,16cyclictest1069-21runrttasks00:28:062
3770556994120,16cyclictest2181-21gdbus22:50:002
3770556994116,12cyclictest438-20systemd-journal19:49:592
377055599418,18cyclictest26-21ksoftirqd/119:34:561
377055599415,10cyclictest3813952-21ntp_states21:10:271
377055599413,13cyclictest26-21ksoftirqd/121:04:401
377055599413,13cyclictest26-21ksoftirqd/121:04:401
3770555994130,8cyclictest37855072sleep119:50:201
3770555994123,13cyclictest3837505-21/usr/sbin/munin22:20:011
3770555994111,18cyclictest26-21ksoftirqd/119:44:561
377055099412,18cyclictest1753-21gnome-shell21:34:230
3770559994025,11cyclictest42-21ksoftirqd/321:57:263
3770559994024,11cyclictest3882130-21/usr/sbin/munin00:30:013
377055999402,12cyclictest2197014-21PK-Backend21:33:433
377055999401,16cyclictest3825097-21systemd-run21:45:003
377055699409,6cyclictest141rcu_preempt22:53:192
377055699408,6cyclictest141rcu_preempt21:31:452
3770556994023,13cyclictest34-21ksoftirqd/220:34:112
3770556994022,14cyclictest3798072-21ntp_states20:25:292
3770556994010,6cyclictest141rcu_preempt20:46:592
3770556994010,6cyclictest141rcu_preempt20:46:592
377055599408,16cyclictest26-21ksoftirqd/119:20:371
377055599408,16cyclictest26-21ksoftirqd/119:20:371
377055599407,11cyclictest26-21ksoftirqd/121:25:221
377055599407,10cyclictest26-21ksoftirqd/123:00:221
377055599407,10cyclictest26-21ksoftirqd/120:40:011
377055599405,14cyclictest26-21ksoftirqd/100:15:171
377055599405,10cyclictest26-21ksoftirqd/119:15:291
3770555994016,7cyclictest26-21ksoftirqd/120:14:051
377055599401,11cyclictest26-21ksoftirqd/119:10:211
377055999396,7cyclictest2197014-21PK-Backend19:21:563
377055999396,7cyclictest2197014-21PK-Backend19:21:553
377055999394,12cyclictest141rcu_preempt19:52:333
3770559993926,9cyclictest42-21ksoftirqd/320:11:013
3770559993924,11cyclictest42-21ksoftirqd/323:00:393
3770559993923,12cyclictest3826530-21sshd21:46:473
3770559993922,13cyclictest3863860-21pool-org.gnome.23:35:003
3770559993922,13cyclictest3787126-21cstates19:55:163
3770559993922,12cyclictest3838491-21if_err_enp2s022:20:253
3770559993916,6cyclictest3883072-21netstat00:30:263
3770559993915,20cyclictest1827-21JS14:31:443
3770559993913,14cyclictest42-21ksoftirqd/321:20:263
377055999391,17cyclictest38367872sh22:15:303
377055699397,6cyclictest141rcu_preempt22:34:242
3770556993929,6cyclictest3774483-21latency_hist19:20:022
377055699392,8cyclictest1753-21gnome-shell20:55:492
377055699392,13cyclictest1753-21gnome-shell21:00:172
377055699392,13cyclictest1753-21gnome-shell21:00:162
3770556993921,13cyclictest34-21ksoftirqd/219:41:102
3770556993911,5cyclictest141rcu_preempt23:45:162
377055699391,14cyclictest38607172/usr/sbin/munin23:25:152
377055699391,12cyclictest3827725-21tr21:50:242
377055699391,12cyclictest141rcu_preempt20:50:222
3770556993910,5cyclictest141rcu_preempt20:14:272
377055599398,10cyclictest26-21ksoftirqd/121:00:001
377055599397,10cyclictest26-21ksoftirqd/123:25:171
377055599396,13cyclictest26-21ksoftirqd/122:59:041
377055599396,13cyclictest26-21ksoftirqd/122:59:031
377055599394,17cyclictest1753-21gnome-shell20:41:001
377055599393,14cyclictest3823853-21expr21:40:161
377055999387,5cyclictest141rcu_preempt23:15:253
377055999382,32cyclictest1753-21gnome-shell19:18:203
377055999382,15cyclictest1753-21gnome-shell22:30:203
3770559993821,13cyclictest42-21ksoftirqd/319:31:543
377055999381,33cyclictest860997-21snmpd20:30:233
377055699389,6cyclictest141rcu_preempt19:26:212
377055699389,5cyclictest141rcu_preempt22:00:012
377055699385,7cyclictest2197014-21PK-Backend20:05:252
377055699385,5cyclictest141rcu_preempt22:26:542
377055699384,8cyclictest1753-21gnome-shell00:40:012
377055699383,12cyclictest141rcu_preempt21:10:102
3770556993822,12cyclictest34-21ksoftirqd/221:10:012
377055699382,20cyclictest3852339-21df23:00:162
3770556993811,5cyclictest141rcu_preempt20:43:462
377055699381,11cyclictest141rcu_preempt19:35:282
377055599384,12cyclictest3822522-21cat21:35:271
377055599381,13cyclictest1753-21gnome-shell23:40:291
162399380,26rtkit-daemon0-21swapper/121:15:581
377055999376,5cyclictest141rcu_preempt19:14:433
3770559993721,5cyclictest42-21ksoftirqd/323:35:333
3770559993721,12cyclictest3802817-21expr20:40:163
377055999371,17cyclictest1753-21gnome-shell22:58:313
377055999371,17cyclictest1753-21gnome-shell22:58:313
377055999371,13cyclictest2197014-21PK-Backend19:45:163
3770556993723,10cyclictest3862544-21cpuspeed_turbos23:30:152
377055699372,11cyclictest1753-21gnome-shell23:22:142
3770556993720,13cyclictest34-21ksoftirqd/222:10:352
377055699371,9cyclictest3823285-21chrt21:39:352
3770556993719,14cyclictest3242253-21nautilus21:30:022
3770556993712,5cyclictest141rcu_preempt21:24:152
162399370,25rtkit-daemon0-21swapper/122:27:191
377055999365,6cyclictest141rcu_preempt20:05:303
3770559993625,7cyclictest3867298-21perf23:45:013
3770559993621,11cyclictest42-21ksoftirqd/321:15:113
3770559993619,13cyclictest3805739-21pool-org.gnome.20:50:023
3770559993619,13cyclictest3805739-21pool-org.gnome.20:50:013
3770559993619,12cyclictest438-20systemd-journal22:06:473
377055999361,30cyclictest1753-21gnome-shell21:25:043
3770559993610,6cyclictest141rcu_preempt21:35:363
377055699367,5cyclictest141rcu_preempt22:37:442
377055699363,10cyclictest34-21ksoftirqd/222:01:262
377055699361,13cyclictest141rcu_preempt20:15:182
377055599366,10cyclictest26-21ksoftirqd/123:16:181
377055999356,5cyclictest141rcu_preempt20:54:483
3770559993523,8cyclictest1753-21gnome-shell20:02:053
3770559993523,8cyclictest1753-21gnome-shell19:26:053
3770559993518,13cyclictest1753-21gnome-shell22:54:523
3770559993517,13cyclictest42-21ksoftirqd/319:42:203
377055999351,31cyclictest2197014-21PK-Backend00:09:323
3770559993512,5cyclictest141rcu_preempt20:55:203
3770559993511,21cyclictest1753-21gnome-shell20:27:473
377055999351,10cyclictest3854152-21if_enp2s023:05:243
377055699356,9cyclictest141rcu_preempt00:34:392
377055699354,10cyclictest141rcu_preempt21:40:492
3770556993520,11cyclictest3833682-21seq22:05:422
3770556993520,11cyclictest3816460-21sed21:20:012
377055699351,8cyclictest1753-21gnome-shell19:34:582
377055099351,13cyclictest2197014-21PK-Backend22:46:130
3770559993421,9cyclictest1753-21gnome-shell22:01:353
3770559993419,11cyclictest42-21ksoftirqd/322:10:263
3770559993418,11cyclictest3796015-21cat20:20:233
3770559993417,12cyclictest438-20systemd-journal19:37:433
377055999341,6cyclictest141rcu_preempt21:05:013
377055999341,6cyclictest141rcu_preempt21:05:013
377055999341,10cyclictest2197014-21PK-Backend20:15:243
377055699347,5cyclictest141rcu_preempt00:23:122
3770556993419,11cyclictest3787763-21smartctl19:55:312
3770559993319,10cyclictest42-21ksoftirqd/322:25:273
377055999331,8cyclictest0-21swapper/323:46:503
3770559993311,5cyclictest141rcu_preempt00:20:503
377055699322,11cyclictest2197014-21PK-Backend22:55:132
377055699322,11cyclictest2197014-21PK-Backend22:55:122
3770556993216,11cyclictest3853665-21seq23:05:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional