You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 02:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Nov 04, 2025 12:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
47248921130,5sleep1251rcuc/107:30:251
46357329244,39sleep00-21swapper/007:05:510
162399890,72rtkit-daemon0-21swapper/207:07:282
4842392750,5sleep1251rcuc/108:05:031
4842392750,5sleep1251rcuc/108:05:021
46384427546,20sleep30-21swapper/307:09:243
46366227138,26sleep10-21swapper/107:06:591
4705482610,9sleep11753-21gnome-shell07:25:211
162399610,9rtkit-daemon1622-21rtkit-daemon11:20:211
162399590,9rtkit-daemon1622-21rtkit-daemon10:59:340
162399590,8rtkit-daemon1622-21rtkit-daemon07:57:231
162399580,9rtkit-daemon1622-21rtkit-daemon11:18:411
162399580,9rtkit-daemon1622-21rtkit-daemon08:33:250
162399580,9rtkit-daemon1622-21rtkit-daemon08:33:240
162399580,8rtkit-daemon1622-21rtkit-daemon11:49:171
162399580,8rtkit-daemon1622-21rtkit-daemon10:39:551
162399570,8rtkit-daemon1622-21rtkit-daemon12:39:011
162399570,8rtkit-daemon1622-21rtkit-daemon09:53:350
162399560,9rtkit-daemon1622-21rtkit-daemon11:37:270
162399560,9rtkit-daemon1622-21rtkit-daemon11:37:270
162399560,8rtkit-daemon1622-21rtkit-daemon12:26:360
162399560,8rtkit-daemon1622-21rtkit-daemon12:24:191
162399560,10rtkit-daemon1622-21rtkit-daemon12:20:570
464038995531,20cyclictest1753-21gnome-shell08:20:293
162399550,9rtkit-daemon1622-21rtkit-daemon08:13:220
162399550,8rtkit-daemon1622-21rtkit-daemon11:32:360
46403599541,10cyclictest2179-21gvfsd-metadata10:50:022
46403399540,50cyclictest1622-21rtkit-daemon12:29:591
162399540,9rtkit-daemon1622-21rtkit-daemon12:11:480
162399540,8rtkit-daemon1622-21rtkit-daemon11:08:240
162399540,8rtkit-daemon1622-21rtkit-daemon10:52:451
162399540,8rtkit-daemon1622-21rtkit-daemon10:52:451
46403399530,49cyclictest1622-21rtkit-daemon11:26:331
162399530,8rtkit-daemon1622-21rtkit-daemon11:56:351
162399530,8rtkit-daemon1622-21rtkit-daemon11:56:351
162399530,8rtkit-daemon1622-21rtkit-daemon08:25:240
5236672520,8sleep342-21ksoftirqd/309:55:323
464038995228,19cyclictest1753-21gnome-shell12:00:263
162399520,8rtkit-daemon1622-21rtkit-daemon08:15:080
464038995128,19cyclictest1753-21gnome-shell08:25:193
162399510,7rtkit-daemon1622-21rtkit-daemon08:28:151
4642462500,9sleep22197014-21PK-Backend07:10:142
46402899502,14cyclictest13-21ksoftirqd/010:15:010
46402899502,14cyclictest13-21ksoftirqd/010:15:010
46402899501,8cyclictest13-21ksoftirqd/012:00:010
46402899501,8cyclictest13-21ksoftirqd/012:00:010
46402899500,46cyclictest1622-21rtkit-daemon08:22:190
4898412490,3sleep1489847-21df_inode08:20:161
46402899493,9cyclictest0-21swapper/010:55:010
46402899493,9cyclictest0-21swapper/010:55:010
464028994921,9cyclictest13-21ksoftirqd/012:05:010
162399490,34rtkit-daemon2197014-21PK-Backend09:31:383
5120212480,6sleep32197014-21PK-Backend09:25:143
46403899482,14cyclictest5405642cat10:45:023
46402899482,15cyclictest515176-21chrt09:34:440
46402899481,25cyclictest498119-21chrt08:45:100
464035994728,7cyclictest34-21ksoftirqd/210:41:412
464035994715,15cyclictest141rcu_preempt11:20:402
464035994713,9cyclictest34-21ksoftirqd/207:51:072
46402899473,8cyclictest13-21ksoftirqd/011:55:020
5806212460,9sleep00-21swapper/012:36:250
46403599466,36cyclictest34-21ksoftirqd/209:10:012
464033994627,14cyclictest522098-40apt-check09:51:171
46402899468,16cyclictest13-21ksoftirqd/012:30:140
46402899464,19cyclictest13-21ksoftirqd/011:01:190
464028994620,23cyclictest512017-21apt09:25:140
464038994513,14cyclictest141rcu_preempt12:25:243
464035994533,7cyclictest34-21ksoftirqd/208:10:292
464035994525,15cyclictest34-21ksoftirqd/210:16:382
464035994515,14cyclictest141rcu_preempt10:39:512
464035994515,14cyclictest141rcu_preempt07:45:172
464035994515,14cyclictest141rcu_preempt07:45:172
46403599451,21cyclictest34-21ksoftirqd/207:36:092
46402899457,16cyclictest13-21ksoftirqd/010:00:030
464028994517,16cyclictest13-21ksoftirqd/011:50:010
464035994433,7cyclictest1753-21gnome-shell10:25:502
464035994412,12cyclictest34-21ksoftirqd/210:10:012
464035994412,12cyclictest34-21ksoftirqd/210:10:012
464033994413,14cyclictest0-21swapper/110:10:151
464033994413,14cyclictest0-21swapper/110:10:141
46402899447,16cyclictest13-21ksoftirqd/010:15:140
46402899447,16cyclictest13-21ksoftirqd/007:55:020
46402899446,15cyclictest13-21ksoftirqd/009:10:420
46402899446,15cyclictest13-21ksoftirqd/009:10:420
46402899445,15cyclictest13-21ksoftirqd/011:25:010
46402899442,22cyclictest560431-21chrt11:40:180
464028994420,6cyclictest1753-21gnome-shell08:36:580
46402899441,12cyclictest484105-21cut08:05:010
162399440,37rtkit-daemon1753-21gnome-shell11:39:191
162399440,37rtkit-daemon1753-21gnome-shell11:39:191
162399440,35rtkit-daemon13-21ksoftirqd/012:19:150
162399440,33rtkit-daemon2197014-21PK-Backend10:22:042
5158372430,6sleep3411rcuc/309:35:163
464035994326,7cyclictest1753-21gnome-shell08:47:242
464035994324,6cyclictest34-21ksoftirqd/211:40:002
464035994324,6cyclictest34-21ksoftirqd/211:40:002
464035994323,6cyclictest34-21ksoftirqd/209:35:302
46403599432,12cyclictest1753-21gnome-shell07:23:512
464035994315,10cyclictest34-21ksoftirqd/210:00:262
464035994313,14cyclictest141rcu_preempt11:54:532
46403599431,10cyclictest2197014-21PK-Backend09:28:152
464033994326,13cyclictest520419-21apt-get09:50:001
46402899439,12cyclictest2197014-21PK-Backend09:05:050
46402899438,17cyclictest13-21ksoftirqd/010:25:140
464028994334,5cyclictest161rcuc/009:45:500
46402899432,9cyclictest0-21swapper/007:35:260
464028994326,12cyclictest855-21dbus-daemon07:30:010
46402899431,38cyclictest1753-21gnome-shell09:38:010
46402899431,13cyclictest539178-21expr10:40:180
464028994310,15cyclictest13-21ksoftirqd/007:20:220
162399430,5rtkit-daemon0-21swapper/207:19:232
162399430,35rtkit-daemon464023-21cyclictest11:31:351
162399430,32rtkit-daemon2197014-21PK-Backend09:22:232
5642012421,14sleep31753-21gnome-shell11:50:283
464038994220,18cyclictest42-21ksoftirqd/308:50:293
46403899421,9cyclictest2197014-21PK-Backend08:07:533
46403899421,9cyclictest2197014-21PK-Backend08:07:533
464038994216,11cyclictest141rcu_preempt09:10:013
46403599425,9cyclictest34-21ksoftirqd/209:45:462
464035994224,7cyclictest1753-21gnome-shell11:11:242
464035994224,14cyclictest34-21ksoftirqd/208:36:132
464035994223,6cyclictest34-21ksoftirqd/211:55:232
464035994223,6cyclictest34-21ksoftirqd/211:55:232
464035994223,14cyclictest34-21ksoftirqd/211:00:112
464035994214,13cyclictest141rcu_preempt12:18:402
464035994213,14cyclictest141rcu_preempt09:19:452
464035994211,9cyclictest34-21ksoftirqd/208:25:282
464035994211,15cyclictest141rcu_preempt08:40:452
46402899429,19cyclictest13-21ksoftirqd/007:43:580
46402899427,16cyclictest13-21ksoftirqd/011:30:010
464028994233,5cyclictest161rcuc/007:59:090
464028994216,9cyclictest13-21ksoftirqd/009:43:230
46402899421,13cyclictest537857-21grep10:35:330
46402899421,13cyclictest484564-21cat08:05:170
46402899421,13cyclictest484564-21cat08:05:170
5267662410,6sleep1251rcuc/110:05:161
5267662410,6sleep1251rcuc/110:05:161
4807942410,7chrt0-21swapper/207:55:092
464038994128,9cyclictest1753-21gnome-shell12:15:433
46403899411,9cyclictest2197014-21PK-Backend10:57:423
46403599419,16cyclictest34-21ksoftirqd/207:27:212
46403599416,19cyclictest34-21ksoftirqd/211:15:152
464035994129,8cyclictest1753-21gnome-shell12:12:032
464035994129,8cyclictest1753-21gnome-shell11:34:452
464035994123,14cyclictest34-21ksoftirqd/212:35:092
464035994123,13cyclictest34-21ksoftirqd/209:58:452
464035994122,6cyclictest34-21ksoftirqd/211:47:162
464035994122,14cyclictest34-21ksoftirqd/209:04:032
464035994117,6cyclictest141rcu_preempt08:05:142
464035994117,6cyclictest141rcu_preempt08:05:142
464035994116,6cyclictest34-21ksoftirqd/208:15:402
464033994125,11cyclictest510021-21/usr/sbin/munin09:20:001
464033994119,7cyclictest533882-21if_enp2s010:25:221
464033994116,11cyclictest476948-21latency_hist07:45:021
464033994115,13cyclictest467935-21/usr/sbin/munin07:20:001
46402899418,16cyclictest13-21ksoftirqd/009:00:250
46402899417,18cyclictest13-21ksoftirqd/007:35:020
464028994132,5cyclictest161rcuc/012:05:520
46402899411,17cyclictest477779-21chrt07:45:220
46402899411,17cyclictest477779-21chrt07:45:210
46402899411,16cyclictest1753-21gnome-shell10:08:020
46402899411,16cyclictest1753-21gnome-shell10:08:010
46402899411,15cyclictest0-21swapper/008:55:170
46402899411,14cyclictest466519-21cpuspeed07:15:130
46403899407,29cyclictest1753-21gnome-shell11:40:593
46403899401,9cyclictest2197014-21PK-Backend09:48:203
46403899401,35cyclictest1753-21gnome-shell12:12:443
464035994029,7cyclictest1753-21gnome-shell12:04:172
464035994028,8cyclictest1753-21gnome-shell09:30:322
464035994023,13cyclictest34-21ksoftirqd/210:51:052
464035994023,13cyclictest34-21ksoftirqd/210:51:042
464035994023,12cyclictest34-21ksoftirqd/211:42:142
464035994022,14cyclictest34-21ksoftirqd/209:45:012
464035994022,13cyclictest34-21ksoftirqd/208:20:052
464035994018,14cyclictest34-21ksoftirqd/210:10:222
464035994018,14cyclictest34-21ksoftirqd/210:10:212
46403599401,19cyclictest1753-21gnome-shell08:55:022
464033994028,9cyclictest26-21ksoftirqd/109:15:011
464033994028,9cyclictest26-21ksoftirqd/109:15:001
46403399401,9cyclictest141rcu_preempt08:55:321
464033994018,16cyclictest529961-21sshd10:15:111
46402899408,18cyclictest13-21ksoftirqd/011:15:220
46402899408,16cyclictest13-21ksoftirqd/010:20:210
46402899407,15cyclictest13-21ksoftirqd/010:31:560
464028994011,15cyclictest1069-21runrttasks11:11:530
162399400,29rtkit-daemon2197014-21PK-Backend07:24:263
464038993927,8cyclictest1069-21runrttasks11:14:343
464038993922,13cyclictest498866-21ntp_kernel_pll_08:45:263
464038993920,15cyclictest1753-21gnome-shell08:17:093
464038993919,15cyclictest580635-21gdbus12:36:283
464035993928,7cyclictest1753-21gnome-shell12:21:452
464035993928,7cyclictest1753-21gnome-shell09:50:342
464035993926,9cyclictest1753-21gnome-shell11:28:532
464035993923,5cyclictest34-21ksoftirqd/210:30:292
464035993922,6cyclictest34-21ksoftirqd/211:05:272
464035993921,6cyclictest34-21ksoftirqd/212:32:552
464035993916,6cyclictest141rcu_preempt09:12:472
464035993916,6cyclictest141rcu_preempt09:12:462
46403599391,14cyclictest502987-21taskset08:59:062
464033993922,12cyclictest438-20systemd-journal07:14:551
464033993916,18cyclictest571798-21sshd12:13:421
46402899399,15cyclictest13-21ksoftirqd/010:45:060
46402899399,14cyclictest13-21ksoftirqd/009:19:360
46402899397,15cyclictest13-21ksoftirqd/008:50:130
464028993917,9cyclictest13-21ksoftirqd/007:12:320
46402899391,16cyclictest1753-21gnome-shell09:58:110
162399390,7rtkit-daemon2197014-21PK-Backend07:18:163
162399390,26rtkit-daemon2197014-21PK-Backend08:41:113
46403899387,18cyclictest141rcu_preempt07:29:273
46403899382,32cyclictest1753-21gnome-shell11:15:583
46403899382,17cyclictest1753-21gnome-shell09:20:173
464038993819,14cyclictest42-21ksoftirqd/312:34:193
46403599383,18cyclictest34-21ksoftirqd/207:30:202
464035993827,7cyclictest1753-21gnome-shell12:28:262
464035993827,7cyclictest1753-21gnome-shell10:57:252
464035993827,7cyclictest1753-21gnome-shell08:34:482
464035993827,7cyclictest1753-21gnome-shell08:34:482
464035993820,14cyclictest34-21ksoftirqd/207:40:172
464035993815,9cyclictest34-21ksoftirqd/208:01:092
464033993827,7cyclictest503942-21grep09:00:231
464033993824,10cyclictest573401-21cron12:17:001
464033993819,14cyclictest1-21systemd10:25:001
464033993818,15cyclictest539658-21open_files10:40:291
464033993814,20cyclictest26-21ksoftirqd/112:35:011
464033993813,5cyclictest141rcu_preempt10:55:541
46403399381,11cyclictest474092-21missed_timers07:35:241
464028993821,12cyclictest511059-21awk09:20:270
464038993724,9cyclictest1753-21gnome-shell11:56:523
464038993724,9cyclictest1753-21gnome-shell11:56:523
46403899372,31cyclictest517592-21df_inode09:40:173
46403899372,29cyclictest1753-21gnome-shell10:00:163
46403899372,10cyclictest1753-21gnome-shell08:35:183
464038993713,20cyclictest42-21ksoftirqd/310:11:473
464038993713,20cyclictest42-21ksoftirqd/310:11:473
464035993719,13cyclictest34-21ksoftirqd/212:05:092
464033993722,11cyclictest546358-21grep11:00:141
464033993722,11cyclictest542172-21systemd-run10:50:011
464033993721,12cyclictest500756-21smartctl08:50:321
464033993719,13cyclictest2179-21gvfsd-metadata11:10:011
46403399371,11cyclictest141rcu_preempt08:15:001
46402899375,16cyclictest13-21ksoftirqd/008:40:250
162399370,25rtkit-daemon0-21swapper/310:20:113
162399370,25rtkit-daemon0-21swapper/309:16:113
46403899363,15cyclictest141rcu_preempt07:41:183
46403899361,15cyclictest562957-21seq11:46:263
46403899361,15cyclictest1358-1kworker/u9:111:35:183
46403899361,15cyclictest1358-1kworker/u9:111:35:183
464033993621,11cyclictest855-21dbus-daemon07:22:021
464033993621,11cyclictest526206-21cron10:05:001
464033993617,14cyclictest1-21systemd11:15:011
464033993611,8cyclictest2284-21update-notifier08:15:121
464033993611,5cyclictest141rcu_preempt12:06:221
464033993611,5cyclictest141rcu_preempt11:41:361
464033993610,6cyclictest141rcu_preempt08:50:011
46403899352,9cyclictest464618-21/usr/sbin/munin07:10:173
464038993523,8cyclictest1753-21gnome-shell07:32:453
464038993520,10cyclictest544348-21date10:55:023
464038993520,10cyclictest544348-21date10:55:023
464038993520,10cyclictest42-21ksoftirqd/308:55:213
464038993519,12cyclictest546650-21interrupts11:00:213
464038993519,12cyclictest42-21ksoftirqd/309:13:073
464038993519,12cyclictest42-21ksoftirqd/309:13:073
464038993519,11cyclictest438-20systemd-journal07:45:103
464038993519,11cyclictest438-20systemd-journal07:45:103
464038993518,12cyclictest535885-21awk10:30:273
46403899351,31cyclictest42-21ksoftirqd/308:02:243
464038993511,21cyclictest1753-21gnome-shell07:38:423
464033993520,11cyclictest494171-21seq08:32:101
464033993520,11cyclictest494171-21seq08:32:101
464033993519,12cyclictest514510-21ntpq09:30:281
464033993519,12cyclictest478029-21ntpq07:45:261
464033993519,12cyclictest478029-21ntpq07:45:251
464033993518,13cyclictest1-21systemd08:40:001
464033993518,12cyclictest438-20systemd-journal07:50:021
464033993513,5cyclictest141rcu_preempt10:33:421
464033993512,6cyclictest141rcu_preempt09:05:241
464033993511,9cyclictest497037-21grep08:40:241
464033993511,6cyclictest141rcu_preempt09:40:271
464033993511,5cyclictest141rcu_preempt09:24:361
464033993510,6cyclictest141rcu_preempt12:01:351
162399350,24rtkit-daemon0-21swapper/309:01:473
46403899347,11cyclictest42-21ksoftirqd/312:20:303
464038993422,8cyclictest1753-21gnome-shell10:29:473
464038993418,12cyclictest557298-21awk11:30:243
464038993418,12cyclictest42-21ksoftirqd/308:12:473
464038993415,14cyclictest855-21dbus-daemon09:55:013
46403899341,15cyclictest1753-21gnome-shell10:45:453
46403399342,16cyclictest515852-21df09:35:181
46403399341,17cyclictest141rcu_preempt11:51:581
4799572330,7chrt2197014-21PK-Backend07:50:363
46403899332,27cyclictest1753-21gnome-shell11:06:223
464038993321,8cyclictest1753-21gnome-shell10:36:143
464038993321,8cyclictest1753-21gnome-shell10:18:263
464033993319,10cyclictest482638-21ls08:00:141
464033993316,6cyclictest26-21ksoftirqd/109:55:291
464033993310,5cyclictest141rcu_preempt09:29:281
464038993219,9cyclictest570266-21cat12:10:013
46403899321,4cyclictest0-21swapper/310:05:193
46403899321,4cyclictest0-21swapper/310:05:183
46403899321,29cyclictest0-21swapper/307:58:013
46403899312,8cyclictest1753-21gnome-shell08:33:483
46403899312,8cyclictest1753-21gnome-shell08:33:483
464038993124,5cyclictest0-21swapper/311:26:173
46403899312,25cyclictest1753-21gnome-shell11:21:593
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional