You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 17:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Nov 30, 2025 12:45:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
272967821270,16sleep013-21ksoftirqd/007:40:160
279546221150,8sleep21753-21gnome-shell10:45:162
1623991080,60rtkit-daemon0-21swapper/207:05:292
27287022950,6sleep10-21swapper/107:35:301
27287022950,6sleep10-21swapper/107:35:301
162399930,25rtkit-daemon1622-21rtkit-daemon07:31:392
28298552900,5sleep0161rcuc/012:24:240
27786562850,5sleep1251rcuc/109:55:331
271683828340,37sleep30-21swapper/307:05:023
27803042760,5sleep00-21swapper/010:00:310
271724727350,15sleep10-21swapper/107:05:171
162399690,4rtkit-daemon0-21swapper/207:19:592
162399690,4rtkit-daemon0-21swapper/207:19:592
271799326844,15sleep00-21swapper/007:05:440
27799582600,16sleep126-21ksoftirqd/110:00:241
162399600,9rtkit-daemon1622-21rtkit-daemon08:02:322
2718470995823,19cyclictest42-21ksoftirqd/310:19:593
162399580,8rtkit-daemon1622-21rtkit-daemon08:20:590
162399580,8rtkit-daemon1622-21rtkit-daemon08:20:590
2718470995623,19cyclictest42-21ksoftirqd/309:15:353
28247222550,6sleep12824714-21awk12:10:001
28071532540,14chrt341ksoftirqd/211:20:022
28071532540,14chrt341ksoftirqd/211:20:022
271847099548,10cyclictest42-21ksoftirqd/310:55:023
271847099547,11cyclictest42-21ksoftirqd/309:20:133
2718470995420,18cyclictest42-21ksoftirqd/311:15:013
2718467995435,6cyclictest34-21ksoftirqd/207:25:262
27784182530,5sleep30-21swapper/309:55:273
27739772530,8sleep32197014-21PK-Backend09:45:003
271847099537,11cyclictest42-21ksoftirqd/312:10:013
162399530,9rtkit-daemon2804617-21ntp_states11:10:252
27425172520,12sleep126-21ksoftirqd/108:15:271
2718470995221,18cyclictest42-21ksoftirqd/308:35:013
2718470995217,19cyclictest42-21ksoftirqd/311:25:013
271847099521,11cyclictest42-21ksoftirqd/310:40:313
2718462995228,20cyclictest26-21ksoftirqd/107:25:141
162399520,8rtkit-daemon1622-21rtkit-daemon07:42:322
162399520,8rtkit-daemon1622-21rtkit-daemon07:27:133
27549542510,15chrt26-21ksoftirqd/108:50:241
2718470995117,19cyclictest42-21ksoftirqd/310:40:003
2718470995117,18cyclictest42-21ksoftirqd/312:05:013
2718470995112,22cyclictest1753-21gnome-shell10:46:423
162399510,8rtkit-daemon1622-21rtkit-daemon07:35:072
162399510,8rtkit-daemon1622-21rtkit-daemon07:35:072
271847099506,11cyclictest42-21ksoftirqd/312:15:013
2718470995016,19cyclictest42-21ksoftirqd/308:10:003
2718467995033,6cyclictest34-21ksoftirqd/209:47:532
162399500,6rtkit-daemon1622-21rtkit-daemon07:52:092
162399500,41rtkit-daemon2284-21update-notifier08:03:133
2718470994941,4cyclictest411rcuc/308:19:343
271847099492,26cyclictest2769086-21df09:30:143
2718470994919,10cyclictest42-21ksoftirqd/308:00:013
2718470994919,10cyclictest42-21ksoftirqd/308:00:013
2718470994917,18cyclictest42-21ksoftirqd/311:05:013
2718470994916,17cyclictest42-21ksoftirqd/307:50:003
2718470994915,18cyclictest42-21ksoftirqd/311:50:013
2718470994914,18cyclictest42-21ksoftirqd/311:30:013
2718462994923,22cyclictest26-21ksoftirqd/111:55:321
162399490,37rtkit-daemon2197014-21PK-Backend12:32:193
271847099488,13cyclictest42-21ksoftirqd/310:30:283
271847099488,13cyclictest42-21ksoftirqd/310:30:283
271847099486,25cyclictest42-21ksoftirqd/308:25:153
271847099482,11cyclictest1753-21gnome-shell08:10:313
2718467994826,17cyclictest34-21ksoftirqd/211:50:192
2718467994826,17cyclictest34-21ksoftirqd/211:50:182
27932942470,9sleep234-21ksoftirqd/210:40:012
27494182470,6sleep013-21ksoftirqd/008:35:180
271847099476,20cyclictest1753-21gnome-shell10:23:513
271847099475,18cyclictest42-21ksoftirqd/309:00:323
2718470994715,17cyclictest42-21ksoftirqd/311:00:023
2718470994713,18cyclictest42-21ksoftirqd/309:45:303
2718470994711,19cyclictest42-21ksoftirqd/309:15:003
271847099471,10cyclictest2197014-21PK-Backend09:55:003
2718467994729,14cyclictest34-21ksoftirqd/207:45:042
2718467994728,15cyclictest34-21ksoftirqd/208:47:592
2718467994726,16cyclictest34-21ksoftirqd/209:42:222
2718467994724,14cyclictest34-21ksoftirqd/212:15:222
2718467994724,14cyclictest34-21ksoftirqd/212:15:222
2718467994724,14cyclictest34-21ksoftirqd/210:53:402
2718467994724,14cyclictest34-21ksoftirqd/210:05:192
2718467994724,13cyclictest34-21ksoftirqd/212:10:022
2718467994724,12cyclictest0-21swapper/211:49:022
2718467994723,15cyclictest34-21ksoftirqd/209:25:322
2718467994723,15cyclictest34-21ksoftirqd/208:15:342
271846799471,6cyclictest141rcu_preempt10:55:222
2718462994720,11cyclictest2788902-21ntp_kernel_err10:25:251
2718462994718,25cyclictest26-21ksoftirqd/110:36:571
28001332460,11chrt2800123-21cron11:00:011
27335832460,9sleep3411rcuc/307:50:213
2718470994624,6cyclictest1753-21gnome-shell07:10:443
2718470994614,12cyclictest42-21ksoftirqd/310:15:013
2718470994613,13cyclictest42-21ksoftirqd/307:25:003
2718470994612,18cyclictest42-21ksoftirqd/312:15:183
2718470994612,18cyclictest42-21ksoftirqd/312:15:183
271847099461,15cyclictest2751532-21open_files08:40:283
2718467994635,8cyclictest1753-21gnome-shell10:44:102
2718467994625,18cyclictest34-21ksoftirqd/208:30:202
2718467994625,16cyclictest34-21ksoftirqd/211:25:222
2718467994625,16cyclictest34-21ksoftirqd/210:00:252
2718467994624,7cyclictest34-21ksoftirqd/209:05:262
2718467994624,14cyclictest34-21ksoftirqd/210:30:302
2718467994624,14cyclictest34-21ksoftirqd/210:30:302
2718467994624,14cyclictest34-21ksoftirqd/209:15:372
2718467994624,13cyclictest34-21ksoftirqd/208:21:232
2718467994624,13cyclictest34-21ksoftirqd/208:21:232
2718467994623,14cyclictest34-21ksoftirqd/211:40:192
2718467994623,14cyclictest34-21ksoftirqd/211:20:232
2718462994630,6cyclictest26-21ksoftirqd/111:50:241
2718462994630,6cyclictest26-21ksoftirqd/111:50:231
2718462994625,8cyclictest26-21ksoftirqd/107:55:291
2718462994625,8cyclictest26-21ksoftirqd/107:55:291
2718462994623,19cyclictest26-21ksoftirqd/112:20:291
27569322450,5sleep1251rcuc/108:55:311
271847099456,18cyclictest1753-21gnome-shell12:25:243
2718470994522,6cyclictest2752877-21df_inode08:45:173
2718470994521,7cyclictest1753-21gnome-shell11:40:003
2718470994521,7cyclictest1753-21gnome-shell11:40:003
2718470994521,7cyclictest1753-21gnome-shell08:55:193
2718470994512,17cyclictest42-21ksoftirqd/309:35:273
271847099451,16cyclictest1753-21gnome-shell10:25:173
2718470994511,18cyclictest42-21ksoftirqd/311:40:133
2718470994510,14cyclictest2197014-21PK-Backend12:35:193
2718467994532,9cyclictest2811403-21grep11:30:232
2718467994532,9cyclictest2811403-21grep11:30:222
2718467994526,14cyclictest34-21ksoftirqd/209:30:172
2718467994526,14cyclictest34-21ksoftirqd/207:20:172
2718467994525,16cyclictest34-21ksoftirqd/210:20:272
2718467994525,16cyclictest34-21ksoftirqd/208:50:172
2718467994525,15cyclictest34-21ksoftirqd/212:13:072
2718467994525,15cyclictest34-21ksoftirqd/208:10:312
2718467994525,15cyclictest34-21ksoftirqd/208:05:192
2718467994524,14cyclictest34-21ksoftirqd/212:00:532
2718467994524,12cyclictest34-21ksoftirqd/208:35:172
2718467994523,14cyclictest34-21ksoftirqd/209:00:172
2718467994523,13cyclictest34-21ksoftirqd/211:35:272
2718467994523,13cyclictest34-21ksoftirqd/211:35:272
2718467994522,16cyclictest34-21ksoftirqd/208:27:252
2718467994522,14cyclictest34-21ksoftirqd/212:30:192
2718467994522,14cyclictest34-21ksoftirqd/211:55:342
2718467994522,14cyclictest34-21ksoftirqd/209:39:492
2718462994532,9cyclictest875-21polkitd10:22:151
2718460994526,14cyclictest13-21ksoftirqd/009:42:160
2718460994514,15cyclictest141rcu_preempt10:20:220
271846099451,17cyclictest27919292cut10:35:150
271847099449,17cyclictest1753-21gnome-shell07:43:003
271847099443,20cyclictest1753-21gnome-shell09:29:363
2718470994421,6cyclictest1753-21gnome-shell07:31:403
2718470994413,17cyclictest42-21ksoftirqd/308:40:013
271847099441,26cyclictest1753-21gnome-shell11:31:573
271847099441,26cyclictest1753-21gnome-shell11:31:573
2718467994426,6cyclictest34-21ksoftirqd/209:55:022
2718467994426,13cyclictest34-21ksoftirqd/208:40:052
2718467994425,6cyclictest34-21ksoftirqd/211:05:212
2718467994425,6cyclictest34-21ksoftirqd/210:25:152
2718467994425,14cyclictest34-21ksoftirqd/212:35:332
2718467994425,14cyclictest34-21ksoftirqd/209:58:102
2718467994425,14cyclictest34-21ksoftirqd/209:11:372
2718467994425,14cyclictest34-21ksoftirqd/208:55:202
2718467994424,16cyclictest34-21ksoftirqd/212:25:212
2718467994423,16cyclictest34-21ksoftirqd/207:10:202
2718467994422,14cyclictest34-21ksoftirqd/207:57:122
2718467994422,14cyclictest34-21ksoftirqd/207:57:122
2718462994433,7cyclictest2804397-21grep11:10:231
2718462994427,13cyclictest26-21ksoftirqd/109:05:291
2718462994425,8cyclictest26-21ksoftirqd/111:20:301
2718462994425,7cyclictest26-21ksoftirqd/112:10:281
2718460994425,14cyclictest13-21ksoftirqd/011:15:240
2718460994425,14cyclictest13-21ksoftirqd/011:15:240
2718460994414,13cyclictest141rcu_preempt09:26:250
28277062430,6sleep12827710-21timerandwakeup12:15:321
28277062430,6sleep12827710-21timerandwakeup12:15:311
27217812430,5chrt2197014-21PK-Backend07:15:313
27217812430,5chrt2197014-21PK-Backend07:15:313
271847099438,12cyclictest42-21ksoftirqd/308:50:303
271847099436,18cyclictest42-21ksoftirqd/311:55:033
271847099436,12cyclictest42-21ksoftirqd/307:35:263
271847099436,12cyclictest42-21ksoftirqd/307:35:263
271847099435,20cyclictest42-21ksoftirqd/312:24:393
271847099434,19cyclictest1753-21gnome-shell11:06:253
271847099431,17cyclictest1753-21gnome-shell11:17:573
271847099431,17cyclictest1753-21gnome-shell11:17:573
2718467994325,14cyclictest34-21ksoftirqd/210:16:162
2718467994324,14cyclictest34-21ksoftirqd/209:23:542
2718467994322,16cyclictest34-21ksoftirqd/212:24:542
2718467994322,16cyclictest34-21ksoftirqd/211:00:332
2718467994322,16cyclictest34-21ksoftirqd/210:11:002
2718462994325,14cyclictest26-21ksoftirqd/109:15:301
2718462994324,14cyclictest26-21ksoftirqd/107:25:001
2718462994324,14cyclictest26-21ksoftirqd/107:15:021
2718462994320,19cyclictest26-21ksoftirqd/110:30:241
2718462994320,19cyclictest26-21ksoftirqd/110:30:231
2718460994326,13cyclictest13-21ksoftirqd/008:00:300
2718460994326,12cyclictest13-21ksoftirqd/009:06:180
2718460994325,14cyclictest13-21ksoftirqd/008:06:120
271847099428,18cyclictest42-21ksoftirqd/310:02:483
271847099428,17cyclictest42-21ksoftirqd/311:50:073
271847099428,17cyclictest42-21ksoftirqd/311:50:073
2718470994211,11cyclictest42-21ksoftirqd/308:20:223
2718470994211,11cyclictest42-21ksoftirqd/308:20:213
2718462994227,11cyclictest26-21ksoftirqd/108:30:011
2718462994225,6cyclictest26-21ksoftirqd/110:10:001
2718462994224,7cyclictest26-21ksoftirqd/111:30:261
2718462994224,7cyclictest26-21ksoftirqd/111:30:261
2718462994224,14cyclictest26-21ksoftirqd/109:55:021
2718462994223,15cyclictest2770582-21grep09:35:131
2718462994223,14cyclictest26-21ksoftirqd/111:36:251
2718462994223,14cyclictest26-21ksoftirqd/111:36:241
2718462994222,16cyclictest26-21ksoftirqd/111:15:241
2718462994222,16cyclictest26-21ksoftirqd/111:15:231
2718462994221,14cyclictest26-21ksoftirqd/111:40:211
2718460994225,13cyclictest13-21ksoftirqd/011:00:140
2718460994225,13cyclictest13-21ksoftirqd/008:49:590
2718460994224,14cyclictest13-21ksoftirqd/011:44:130
2718460994224,14cyclictest13-21ksoftirqd/010:30:150
2718460994224,14cyclictest13-21ksoftirqd/010:30:140
2718460994224,14cyclictest13-21ksoftirqd/010:10:590
2718460994224,13cyclictest13-21ksoftirqd/011:05:180
2718460994223,14cyclictest13-21ksoftirqd/011:36:280
2718460994223,14cyclictest13-21ksoftirqd/011:36:270
2718460994222,15cyclictest13-21ksoftirqd/010:05:270
2718460994222,15cyclictest13-21ksoftirqd/007:15:140
2718460994222,15cyclictest13-21ksoftirqd/007:15:140
27692192410,6sleep1251rcuc/109:30:211
271847099417,18cyclictest42-21ksoftirqd/310:06:413
271847099413,10cyclictest42-21ksoftirqd/309:06:213
2718462994129,9cyclictest26-21ksoftirqd/107:45:031
2718462994124,6cyclictest26-21ksoftirqd/111:45:191
2718462994124,6cyclictest26-21ksoftirqd/109:45:241
2718462994124,6cyclictest26-21ksoftirqd/108:05:161
2718462994124,6cyclictest26-21ksoftirqd/108:00:251
2718462994124,13cyclictest26-21ksoftirqd/108:20:151
2718462994124,13cyclictest26-21ksoftirqd/108:20:151
2718462994124,13cyclictest26-21ksoftirqd/108:13:091
2718462994124,13cyclictest26-21ksoftirqd/107:42:101
2718462994124,13cyclictest26-21ksoftirqd/107:30:251
2718462994123,6cyclictest26-21ksoftirqd/108:35:251
2718462994123,14cyclictest26-21ksoftirqd/108:45:161
2718462994123,13cyclictest26-21ksoftirqd/112:25:221
2718462994122,7cyclictest26-21ksoftirqd/109:42:221
271846299411,7cyclictest141rcu_preempt10:10:261
271846099418,13cyclictest141rcu_preempt09:50:280
2718460994126,11cyclictest13-21ksoftirqd/011:20:160
2718460994125,12cyclictest13-21ksoftirqd/009:40:000
2718460994124,13cyclictest13-21ksoftirqd/009:20:020
2718460994124,13cyclictest13-21ksoftirqd/009:15:060
2718460994124,13cyclictest13-21ksoftirqd/009:05:000
2718460994124,13cyclictest13-21ksoftirqd/007:10:150
2718460994123,14cyclictest13-21ksoftirqd/007:57:410
2718460994123,14cyclictest13-21ksoftirqd/007:57:410
2718460994123,13cyclictest13-21ksoftirqd/011:45:300
2718460994122,14cyclictest438-20systemd-journal07:34:140
2718460994122,14cyclictest13-21ksoftirqd/008:50:140
2718460994122,14cyclictest13-21ksoftirqd/008:30:160
2718460994113,12cyclictest141rcu_preempt11:53:450
2718460994113,12cyclictest141rcu_preempt11:53:450
271846099411,11cyclictest2741522-21turbostat08:15:000
2718462994024,12cyclictest1-21systemd10:50:011
2718462994023,6cyclictest26-21ksoftirqd/107:50:261
2718462994023,13cyclictest26-21ksoftirqd/112:00:251
2718462994023,13cyclictest26-21ksoftirqd/111:27:581
2718462994023,13cyclictest26-21ksoftirqd/111:05:121
2718462994023,13cyclictest26-21ksoftirqd/111:00:141
2718462994019,13cyclictest26-21ksoftirqd/109:25:131
271846099408,9cyclictest141rcu_preempt08:27:140
2718460994024,12cyclictest13-21ksoftirqd/007:35:140
2718460994024,12cyclictest13-21ksoftirqd/007:35:130
2718460994023,6cyclictest13-21ksoftirqd/010:50:320
2718460994023,13cyclictest13-21ksoftirqd/012:40:010
2718460994023,13cyclictest13-21ksoftirqd/009:00:010
2718460994022,13cyclictest13-21ksoftirqd/012:30:130
2718460994022,13cyclictest13-21ksoftirqd/011:35:010
2718460994022,13cyclictest13-21ksoftirqd/011:35:010
2718460994020,15cyclictest13-21ksoftirqd/008:15:340
2718460994020,15cyclictest13-21ksoftirqd/007:20:040
2718462993925,10cyclictest2762629-21sshd09:11:131
2718462993924,11cyclictest2766411-21sshd09:21:131
2718462993922,6cyclictest26-21ksoftirqd/112:30:141
2718462993922,6cyclictest26-21ksoftirqd/110:54:061
2718462993922,13cyclictest26-21ksoftirqd/110:16:251
2718462993921,6cyclictest26-21ksoftirqd/108:30:221
2718462993921,14cyclictest26-21ksoftirqd/110:40:331
2718462993920,15cyclictest26-21ksoftirqd/112:39:121
2718462993920,15cyclictest26-21ksoftirqd/109:00:141
271846099394,13cyclictest141rcu_preempt11:13:410
271846099393,6cyclictest141rcu_preempt07:50:310
2718460993926,9cyclictest1753-21gnome-shell12:12:450
2718460993923,12cyclictest2799455-21pluginstate10:55:280
2718460993922,13cyclictest13-21ksoftirqd/012:25:210
2718460993922,13cyclictest13-21ksoftirqd/012:00:260
2718460993922,13cyclictest13-21ksoftirqd/010:40:140
2718460993922,12cyclictest13-21ksoftirqd/010:25:040
2718460993921,13cyclictest13-21ksoftirqd/012:05:140
2718460993919,15cyclictest13-21ksoftirqd/007:25:250
2718460993918,16cyclictest3242255-21gdbus09:35:020
271846099384,5cyclictest141rcu_preempt09:11:260
2718460993822,12cyclictest13-21ksoftirqd/011:26:480
2718460993821,13cyclictest13-21ksoftirqd/010:00:010
2718460993820,13cyclictest13-21ksoftirqd/011:55:230
2718460993819,7cyclictest13-21ksoftirqd/010:15:090
271846099381,7cyclictest141rcu_preempt09:45:140
2718462993721,12cyclictest2750776-21sshd08:40:131
2718462993716,13cyclictest26-21ksoftirqd/107:15:031
2718462993716,13cyclictest26-21ksoftirqd/107:15:031
2718460993720,11cyclictest2731318-21systemd-udevd07:45:020
271846099369,12cyclictest141rcu_preempt10:46:480
2718460993617,15cyclictest13-21ksoftirqd/012:15:300
2718460993617,15cyclictest13-21ksoftirqd/012:15:300
271846099344,5cyclictest141rcu_preempt08:42:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional