You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-16 - 20:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Mon Jun 16, 2025 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
39462692930,6sleep20-21swapper/208:45:012
39439942910,6sleep126-21ksoftirqd/108:38:531
162399910,9rtkit-daemon1622-21rtkit-daemon08:08:251
162399890,9rtkit-daemon1622-21rtkit-daemon12:00:291
162399890,9rtkit-daemon1622-21rtkit-daemon08:28:461
391081928658,17sleep10-21swapper/107:05:131
391081928658,17sleep10-21swapper/107:05:131
162399820,8rtkit-daemon1622-21rtkit-daemon07:36:571
162399820,8rtkit-daemon1622-21rtkit-daemon07:30:271
162399760,5rtkit-daemon0-21swapper/109:02:311
162399730,5rtkit-daemon0-21swapper/109:29:401
162399720,9rtkit-daemon1622-21rtkit-daemon12:05:221
391209327156,9sleep00-21swapper/007:07:290
391209327156,9sleep00-21swapper/007:07:290
391215027041,23sleep30-21swapper/307:08:123
391215027041,23sleep30-21swapper/307:08:123
162399690,11rtkit-daemon1622-21rtkit-daemon11:13:351
391227426645,15sleep20-21swapper/207:09:472
391227426645,15sleep20-21swapper/207:09:472
162399660,9rtkit-daemon1622-21rtkit-daemon07:25:271
391241199650,61cyclictest1622-21rtkit-daemon10:48:540
391241199630,59cyclictest1622-21rtkit-daemon07:12:270
39786312620,14sleep0131ksoftirqd/010:15:280
39786312620,14sleep0131ksoftirqd/010:15:280
162399620,9rtkit-daemon1622-21rtkit-daemon11:48:591
162399610,9rtkit-daemon1622-21rtkit-daemon08:45:241
391241199590,54cyclictest1622-21rtkit-daemon08:20:140
39168992580,5sleep10-21swapper/107:20:171
391241199580,54cyclictest1622-21rtkit-daemon11:44:340
162399580,9rtkit-daemon1622-21rtkit-daemon08:17:130
162399580,8rtkit-daemon1622-21rtkit-daemon11:32:191
40142362570,6chrt0-21swapper/111:55:351
162399570,8rtkit-daemon1622-21rtkit-daemon12:37:361
162399560,8rtkit-daemon1622-21rtkit-daemon10:35:340
162399560,8rtkit-daemon1622-21rtkit-daemon10:35:340
391241199550,51cyclictest1622-21rtkit-daemon11:24:310
162399550,9rtkit-daemon1622-21rtkit-daemon10:53:221
162399550,8rtkit-daemon1622-21rtkit-daemon12:18:000
3912413995421,10cyclictest26-21ksoftirqd/108:55:011
3912413995421,10cyclictest26-21ksoftirqd/108:55:001
162399540,8rtkit-daemon1622-21rtkit-daemon10:57:131
162399540,8rtkit-daemon1622-21rtkit-daemon08:34:561
162399520,8rtkit-daemon1622-21rtkit-daemon09:23:332
162399520,39rtkit-daemon1753-21gnome-shell09:52:450
3912411995022,19cyclictest13-21ksoftirqd/008:00:000
162399500,7rtkit-daemon1622-21rtkit-daemon11:05:050
391241399490,46cyclictest1622-21rtkit-daemon12:19:101
391241199490,45cyclictest1622-21rtkit-daemon11:00:120
391241199490,44cyclictest1622-21rtkit-daemon12:00:300
391241199480,43cyclictest1622-21rtkit-daemon07:36:560
40050142470,7chrt2197014-21PK-Backend11:30:242
3912420994727,6cyclictest0-21swapper/312:25:293
3912420994726,13cyclictest42-21ksoftirqd/308:50:313
3912420994726,13cyclictest42-21ksoftirqd/308:50:303
3912420994722,20cyclictest42-21ksoftirqd/309:00:223
391241199471,9cyclictest141rcu_preempt08:05:150
39940212460,13sleep1251rcuc/111:00:161
3912420994628,10cyclictest42-21ksoftirqd/312:06:493
3912417994619,10cyclictest34-21ksoftirqd/210:35:012
391241399462,40cyclictest1622-21rtkit-daemon07:58:431
3912411994611,15cyclictest141rcu_preempt09:10:160
391241199460,42cyclictest1622-21rtkit-daemon07:30:280
162399460,37rtkit-daemon2197014-21PK-Backend09:57:382
162399460,34rtkit-daemon869-21irqbalance12:09:152
3912420994534,7cyclictest42-21ksoftirqd/311:01:573
3912420994517,24cyclictest42-21ksoftirqd/309:37:033
3912420994511,6cyclictest141rcu_preempt07:38:353
3912417994522,14cyclictest34-21ksoftirqd/211:21:252
3912417994521,20cyclictest4022325-21irqstats12:20:212
3912417994518,11cyclictest34-21ksoftirqd/212:15:012
3912413994528,13cyclictest4026478-21sshd12:31:251
3912413994519,10cyclictest3928469-21seq07:54:451
3912413994517,12cyclictest26-21ksoftirqd/110:25:281
391241399451,40cyclictest1622-21rtkit-daemon12:24:011
3912411994525,7cyclictest13-21ksoftirqd/007:25:150
391241199450,41cyclictest1622-21rtkit-daemon11:35:400
162399450,34rtkit-daemon1938-21gsd-housekeepin08:41:113
162399450,32rtkit-daemon2197014-21PK-Backend10:59:112
162399450,32rtkit-daemon1753-21gnome-shell11:29:462
40247232440,9sleep013-21ksoftirqd/012:26:090
39413572440,5sleep32197014-21PK-Backend08:30:213
391242099446,11cyclictest42-21ksoftirqd/312:11:023
391242099443,16cyclictest141rcu_preempt08:00:213
3912420994426,6cyclictest42-21ksoftirqd/310:31:203
391242099442,12cyclictest1753-21gnome-shell07:20:313
3912420994419,13cyclictest42-21ksoftirqd/310:49:543
3912420994418,22cyclictest42-21ksoftirqd/310:05:123
3912420994418,22cyclictest42-21ksoftirqd/310:05:123
3912417994427,13cyclictest3974144-21/usr/sbin/munin10:05:012
3912417994423,13cyclictest34-21ksoftirqd/207:50:502
3912417994422,13cyclictest34-21ksoftirqd/210:06:212
3912417994422,13cyclictest34-21ksoftirqd/210:06:212
3912417994417,11cyclictest34-21ksoftirqd/211:40:022
3912413994426,13cyclictest438-20systemd-journal09:43:351
3912413994423,17cyclictest3946135-21latency_hist08:45:021
3912413994416,16cyclictest26-21ksoftirqd/111:25:031
3912413994415,14cyclictest26-21ksoftirqd/111:55:001
3912413994414,15cyclictest26-21ksoftirqd/109:25:011
162399440,35rtkit-daemon2197014-21PK-Backend09:18:092
162399440,32rtkit-daemon2197014-21PK-Backend12:27:572
162399440,32rtkit-daemon2197014-21PK-Backend10:38:232
162399440,32rtkit-daemon2197014-21PK-Backend10:38:232
39966622430,7chrt2197014-21PK-Backend11:06:182
39588072430,5sleep00-21swapper/009:20:170
3912420994321,14cyclictest42-21ksoftirqd/307:15:033
3912420994321,14cyclictest42-21ksoftirqd/307:15:023
3912420994319,19cyclictest42-21ksoftirqd/311:10:273
3912420994319,15cyclictest42-21ksoftirqd/309:21:453
3912420994317,23cyclictest42-21ksoftirqd/307:50:323
3912420994317,22cyclictest42-21ksoftirqd/310:04:293
391242099431,16cyclictest854-21avahi-daemon07:55:033
3912417994332,7cyclictest1753-21gnome-shell11:54:222
3912417994332,7cyclictest1753-21gnome-shell07:13:102
3912417994330,9cyclictest34-21ksoftirqd/209:46:582
3912417994324,6cyclictest34-21ksoftirqd/209:35:342
3912417994322,13cyclictest34-21ksoftirqd/208:35:022
3912417994321,13cyclictest34-21ksoftirqd/209:00:272
3912413994319,6cyclictest1069-21runrttasks09:32:241
3912413994317,12cyclictest26-21ksoftirqd/110:43:491
3912413994317,11cyclictest26-21ksoftirqd/109:35:211
3912413994316,12cyclictest26-21ksoftirqd/108:15:581
3912413994316,11cyclictest26-21ksoftirqd/111:40:161
3912413994314,16cyclictest26-21ksoftirqd/110:20:021
3912413994314,16cyclictest26-21ksoftirqd/110:20:021
3912413994313,17cyclictest26-21ksoftirqd/110:35:261
3912413994313,17cyclictest26-21ksoftirqd/110:35:251
3912413994312,12cyclictest0-21swapper/108:55:191
3912411994319,20cyclictest1753-21gnome-shell10:02:470
3912411994319,20cyclictest1753-21gnome-shell09:57:440
162399430,34rtkit-daemon1753-21gnome-shell08:54:032
162399430,34rtkit-daemon1753-21gnome-shell08:54:032
391242099423,18cyclictest42-21ksoftirqd/308:55:323
3912420994226,9cyclictest42-21ksoftirqd/307:10:323
3912420994225,6cyclictest42-21ksoftirqd/308:45:373
3912420994225,13cyclictest42-21ksoftirqd/312:36:223
3912420994224,7cyclictest42-21ksoftirqd/311:37:223
3912420994223,7cyclictest42-21ksoftirqd/309:56:263
3912420994223,6cyclictest42-21ksoftirqd/312:30:543
3912420994223,6cyclictest42-21ksoftirqd/310:53:393
3912420994220,14cyclictest42-21ksoftirqd/307:48:483
3912420994219,12cyclictest42-21ksoftirqd/311:05:073
3912420994211,7cyclictest141rcu_preempt07:40:243
391242099421,13cyclictest3274390-21snapd09:25:513
3912417994226,6cyclictest34-21ksoftirqd/210:50:022
3912417994224,6cyclictest34-21ksoftirqd/211:46:522
3912417994224,6cyclictest34-21ksoftirqd/208:10:152
3912417994224,6cyclictest34-21ksoftirqd/208:00:012
3912417994224,14cyclictest34-21ksoftirqd/208:45:022
3912417994224,14cyclictest34-21ksoftirqd/207:28:202
3912417994224,14cyclictest1753-21gnome-shell07:19:072
3912417994224,14cyclictest1753-21gnome-shell07:19:072
3912417994223,8cyclictest34-21ksoftirqd/207:25:002
3912417994223,6cyclictest34-21ksoftirqd/208:05:222
3912417994223,14cyclictest34-21ksoftirqd/208:25:162
3912417994221,13cyclictest34-21ksoftirqd/210:10:242
3912417994221,13cyclictest34-21ksoftirqd/208:40:002
3912413994217,7cyclictest3984642-21anacron10:32:361
3912413994215,15cyclictest26-21ksoftirqd/109:19:591
3912413994215,11cyclictest26-21ksoftirqd/110:00:351
391241199422,9cyclictest0-21swapper/008:55:590
3912411994225,13cyclictest855-21dbus-daemon08:30:000
3912411994225,13cyclictest3987652-21ps10:40:280
391241199422,10cyclictest0-21swapper/007:22:580
3912411994212,9cyclictest13-21ksoftirqd/009:25:320
3912420994130,7cyclictest1069-21runrttasks08:12:433
3912420994126,11cyclictest3968596-21sshd09:49:353
3912420994124,7cyclictest42-21ksoftirqd/309:50:293
3912420994124,6cyclictest42-21ksoftirqd/311:16:373
3912420994124,6cyclictest42-21ksoftirqd/310:26:533
3912420994124,6cyclictest42-21ksoftirqd/310:15:533
3912420994124,6cyclictest42-21ksoftirqd/310:15:533
3912420994124,6cyclictest42-21ksoftirqd/309:40:253
3912420994124,6cyclictest42-21ksoftirqd/307:30:273
3912420994123,6cyclictest42-21ksoftirqd/311:44:183
3912420994123,6cyclictest42-21ksoftirqd/310:20:593
3912420994123,6cyclictest42-21ksoftirqd/309:15:203
3912420994123,5cyclictest42-21ksoftirqd/311:21:253
3912420994123,4cyclictest141rcu_preempt11:53:353
3912420994122,15cyclictest42-21ksoftirqd/312:15:263
391242099411,9cyclictest0-21swapper/309:30:093
3912417994130,7cyclictest854-21avahi-daemon08:56:072
3912417994124,6cyclictest34-21ksoftirqd/207:39:222
3912417994124,5cyclictest34-21ksoftirqd/211:01:102
3912417994124,13cyclictest34-21ksoftirqd/210:15:452
3912417994124,13cyclictest34-21ksoftirqd/210:15:442
3912417994123,6cyclictest34-21ksoftirqd/207:45:142
3912417994123,13cyclictest34-21ksoftirqd/209:55:002
3912417994122,6cyclictest34-21ksoftirqd/212:30:192
3912417994122,15cyclictest34-21ksoftirqd/209:05:252
3912417994122,14cyclictest34-21ksoftirqd/209:34:052
3912417994121,15cyclictest34-21ksoftirqd/207:35:002
3912417994121,12cyclictest34-21ksoftirqd/212:16:032
391241399412,19cyclictest3242253-21nautilus08:25:011
3912413994118,6cyclictest26-21ksoftirqd/111:05:321
3912413994116,20cyclictest4024281-21netstat12:25:271
391241199416,5cyclictest141rcu_preempt09:00:360
391241199412,9cyclictest0-21swapper/010:24:240
3912411994124,12cyclictest438-20systemd-journal12:00:010
3912411994124,12cyclictest4004118-21sshd11:28:350
3912411994119,18cyclictest1753-21gnome-shell09:07:230
3912411994117,10cyclictest141rcu_preempt10:58:460
162399410,31rtkit-daemon2197014-21PK-Backend10:51:452
3912420994029,7cyclictest1753-21gnome-shell11:30:053
3912420994026,11cyclictest42-21ksoftirqd/308:18:323
3912420994023,6cyclictest42-21ksoftirqd/312:22:303
3912420994023,6cyclictest42-21ksoftirqd/311:57:393
3912420994023,6cyclictest42-21ksoftirqd/309:11:363
3912420994023,6cyclictest42-21ksoftirqd/308:25:313
3912420994023,6cyclictest42-21ksoftirqd/307:25:273
391242099402,34cyclictest1753-21gnome-shell09:05:213
3912420994023,13cyclictest42-21ksoftirqd/308:24:033
3912420994022,6cyclictest42-21ksoftirqd/311:26:583
3912420994022,6cyclictest42-21ksoftirqd/310:58:553
3912420994022,6cyclictest42-21ksoftirqd/308:35:163
3912420994021,6cyclictest42-21ksoftirqd/310:35:113
3912420994021,6cyclictest42-21ksoftirqd/310:35:113
3912420994018,6cyclictest141rcu_preempt11:49:593
3912417994029,7cyclictest1753-21gnome-shell07:42:252
3912417994024,12cyclictest34-21ksoftirqd/211:17:592
3912417994022,14cyclictest34-21ksoftirqd/209:10:182
3912417994019,13cyclictest34-21ksoftirqd/210:23:362
3912417994019,12cyclictest34-21ksoftirqd/212:35:162
3912417994018,13cyclictest34-21ksoftirqd/211:59:082
3912417994012,10cyclictest34-21ksoftirqd/211:15:012
3912413994015,10cyclictest141rcu_preempt10:00:001
391241199402,9cyclictest0-21swapper/008:39:500
391241199402,8cyclictest1753-21gnome-shell12:09:430
3912411994020,15cyclictest3242253-21nautilus11:35:020
391241199401,9cyclictest3949599-21turbostat08:55:010
391241199401,9cyclictest3949599-21turbostat08:55:000
3912411994017,8cyclictest141rcu_preempt10:30:160
3912411994015,9cyclictest141rcu_preempt10:13:180
3912411994012,12cyclictest141rcu_preempt10:05:310
3912411994012,12cyclictest141rcu_preempt10:05:310
391241199401,10cyclictest0-21swapper/011:46:440
3912420993923,6cyclictest42-21ksoftirqd/310:13:373
3912420993923,5cyclictest42-21ksoftirqd/312:00:343
3912420993923,5cyclictest42-21ksoftirqd/310:40:283
3912420993922,6cyclictest42-21ksoftirqd/308:08:073
3912417993928,7cyclictest1753-21gnome-shell08:16:232
3912417993922,6cyclictest34-21ksoftirqd/211:40:212
3912417993922,13cyclictest34-21ksoftirqd/212:01:362
3912417993921,14cyclictest34-21ksoftirqd/210:45:002
3912417993921,13cyclictest34-21ksoftirqd/208:20:362
391241399399,7cyclictest1753-21gnome-shell11:25:201
391241399397,6cyclictest3999789-21awk11:15:291
3912413993923,12cyclictest3968636-21apt-get09:50:001
391241399391,10cyclictest1753-21gnome-shell10:49:551
3912413993910,10cyclictest0-21swapper/111:36:261
391241199397,6cyclictest141rcu_preempt08:15:010
391241199392,9cyclictest0-21swapper/010:28:260
3912411993917,9cyclictest141rcu_preempt09:41:030
3912411993916,9cyclictest141rcu_preempt12:30:390
3912411993912,7cyclictest40185582grep12:10:170
3912417993821,6cyclictest34-21ksoftirqd/209:40:142
3912417993821,13cyclictest34-21ksoftirqd/210:29:032
3912417993820,13cyclictest34-21ksoftirqd/208:00:282
3912417993817,13cyclictest34-21ksoftirqd/209:27:252
3912413993816,5cyclictest26-21ksoftirqd/107:46:561
391241399381,5cyclictest0-21swapper/110:20:091
391241399381,5cyclictest0-21swapper/108:00:491
391241199385,6cyclictest3912406-21cyclictest11:14:590
391241199385,15cyclictest141rcu_preempt07:40:140
391241199382,9cyclictest0-21swapper/007:46:260
3912411993824,10cyclictest3957596-21wc09:15:300
3912411993815,10cyclictest141rcu_preempt10:51:090
3912411993811,10cyclictest141rcu_preempt07:16:550
3912411993811,10cyclictest141rcu_preempt07:16:540
391241399379,11cyclictest26-21ksoftirqd/112:14:501
3912413993721,11cyclictest1-21systemd07:45:011
3912413993719,14cyclictest3934695-21ntpq08:10:271
3912413993713,9cyclictest3969107-21touch09:50:031
391241199375,15cyclictest141rcu_preempt11:50:060
391241199374,17cyclictest3999054-21cpu11:15:140
3912411993717,15cyclictest13-21ksoftirqd/008:48:190
391241199371,5cyclictest2197014-21PK-Backend07:51:270
162399370,24rtkit-daemon0-21swapper/009:31:500
391241399366,11cyclictest3955141-21cpuspeed09:10:161
3912413993621,11cyclictest3915175-21cat07:15:171
3912413993621,11cyclictest3915175-21cat07:15:171
3912413993614,6cyclictest141rcu_preempt10:14:551
3912413993610,8cyclictest26-21ksoftirqd/110:05:111
3912413993610,8cyclictest26-21ksoftirqd/110:05:111
391241199369,12cyclictest141rcu_preempt08:31:320
391241199362,9cyclictest0-21swapper/009:45:570
3912411993612,10cyclictest141rcu_preempt12:35:220
162399360,29rtkit-daemon1753-21gnome-shell12:23:220
162399360,23rtkit-daemon0-21swapper/009:35:030
391241399358,7cyclictest26-21ksoftirqd/109:06:211
3912413993519,12cyclictest3913288-21cut07:10:221
3912411993517,13cyclictest2027-21gdbus08:41:560
391241199351,14cyclictest141rcu_preempt08:00:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional