You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 08:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Dec 01, 2025 00:44:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
308505121000,4sleep00-21swapper/022:55:150
30169452910,6sleep30-21swapper/319:40:253
162399880,9rtkit-daemon1622-21rtkit-daemon19:59:050
162399880,9rtkit-daemon1622-21rtkit-daemon00:23:450
300496728768,13sleep20-21swapper/219:09:502
162399870,9rtkit-daemon1622-21rtkit-daemon00:08:550
300476928459,16sleep10-21swapper/119:07:171
300473927963,9sleep00-21swapper/019:06:560
30869372760,4chrt3086938-21/usr/sbin/munin23:00:212
300489627440,28sleep30-21swapper/319:08:563
30268372710,8sleep11753-21gnome-shell20:10:021
300512499659,25cyclictest3097208-40apt-check23:30:212
3005125996433,9cyclictest42-21ksoftirqd/320:00:023
300512499645,17cyclictest1753-21gnome-shell21:37:382
300512499645,15cyclictest3033975-21unin-run20:30:022
300512499645,15cyclictest3033975-21unin-run20:30:022
300512499627,34cyclictest1753-21gnome-shell19:40:372
300512499624,13cyclictest1753-21gnome-shell19:25:272
300512499624,13cyclictest1753-21gnome-shell19:25:272
3005124996210,23cyclictest1753-21gnome-shell20:05:282
300512499615,15cyclictest1753-21gnome-shell21:25:322
300512499615,15cyclictest1753-21gnome-shell21:25:312
3005124996110,28cyclictest1753-21gnome-shell21:59:032
3005124996110,24cyclictest1753-21gnome-shell21:32:162
3005124996012,29cyclictest1753-21gnome-shell21:40:252
3005124996011,18cyclictest1753-21gnome-shell23:10:232
3005124996010,26cyclictest1753-21gnome-shell22:27:192
3005124996010,26cyclictest1753-21gnome-shell22:27:192
3005125995919,20cyclictest42-21ksoftirqd/323:55:013
300512499598,25cyclictest1753-21gnome-shell21:20:472
300512499598,25cyclictest1753-21gnome-shell21:20:472
300512499595,13cyclictest1753-21gnome-shell22:55:022
3005124995911,25cyclictest1753-21gnome-shell23:16:132
3005124995910,24cyclictest1753-21gnome-shell23:36:342
3005124995910,24cyclictest1753-21gnome-shell23:36:342
300512099598,15cyclictest26-21ksoftirqd/121:10:141
300512499589,25cyclictest1069-21runrttasks23:29:292
300512499589,25cyclictest1069-21runrttasks23:29:282
3005124995811,25cyclictest1753-21gnome-shell19:59:482
3005124995810,25cyclictest1753-21gnome-shell22:16:162
3005124995810,25cyclictest1753-21gnome-shell20:12:412
3005124995810,24cyclictest1753-21gnome-shell23:40:202
3005124995810,18cyclictest1753-21gnome-shell21:02:582
30206902570,6sleep00-21swapper/019:50:340
300512499579,26cyclictest1753-21gnome-shell00:39:312
300512499577,16cyclictest3023919-40lsb_release20:00:232
300512499573,16cyclictest3069586-21hddtemp_smartct22:10:192
3005124995712,25cyclictest1753-21gnome-shell00:24:592
3005124995712,22cyclictest1753-21gnome-shell20:17:442
3005124995711,25cyclictest1753-21gnome-shell20:34:492
3005124995711,24cyclictest1753-21gnome-shell21:53:252
3005124995710,28cyclictest1753-21gnome-shell21:11:042
3005124995710,25cyclictest1753-21gnome-shell23:58:102
3005124995710,25cyclictest1753-21gnome-shell21:46:092
3005124995710,25cyclictest1753-21gnome-shell21:46:092
3005124995710,25cyclictest1753-21gnome-shell00:05:402
3005120995716,38cyclictest1622-21rtkit-daemon00:23:501
300512499563,15cyclictest3117265-21date00:30:022
3005124995611,27cyclictest1753-21gnome-shell22:30:542
3005124995611,24cyclictest1753-21gnome-shell19:30:352
3005124995611,23cyclictest1753-21gnome-shell20:44:122
3005124995610,27cyclictest1753-21gnome-shell22:58:382
3005124995610,25cyclictest1753-21gnome-shell21:18:132
3005124995610,25cyclictest1753-21gnome-shell00:30:532
300512099567,10cyclictest26-21ksoftirqd/122:35:271
30882612550,6sleep00-21swapper/023:05:130
30158922550,5sleep3411rcuc/319:39:303
3005125995542,9cyclictest42-21ksoftirqd/320:00:163
300512499559,28cyclictest1753-21gnome-shell19:22:022
300512499559,27cyclictest1753-21gnome-shell20:45:052
300512499559,26cyclictest1753-21gnome-shell00:19:312
300512499559,24cyclictest1753-21gnome-shell23:45:552
300512499553,28cyclictest1753-21gnome-shell00:01:172
3005124995511,26cyclictest1753-21gnome-shell19:11:552
3005124995511,26cyclictest1753-21gnome-shell00:12:332
3005124995511,23cyclictest1753-21gnome-shell23:08:092
3005124995511,17cyclictest1753-21gnome-shell19:15:062
3005124995511,16cyclictest1753-21gnome-shell22:35:302
3005124995510,27cyclictest1753-21gnome-shell22:24:562
3005124995510,27cyclictest1753-21gnome-shell21:05:352
3005124995510,25cyclictest1753-21gnome-shell19:46:352
3005124995510,24cyclictest1753-21gnome-shell20:54:102
300512499549,25cyclictest1753-21gnome-shell23:54:192
300512499548,24cyclictest1753-21gnome-shell20:21:022
300512499548,24cyclictest1753-21gnome-shell20:21:022
300512499546,16cyclictest1753-21gnome-shell22:45:152
300512499544,26cyclictest3043255-21df_inode20:55:172
3005124995410,26cyclictest1753-21gnome-shell22:03:352
3005124995410,16cyclictest2631682-21snapd22:42:472
300512499539,27cyclictest1753-21gnome-shell19:54:292
300512499534,10cyclictest3093953-21irqstats23:20:222
3005124995310,26cyclictest1753-21gnome-shell20:39:072
3005124995310,19cyclictest3014457-21/usr/sbin/munin19:35:302
300512099537,8cyclictest26-21ksoftirqd/119:40:161
3005125995217,16cyclictest42-21ksoftirqd/322:10:193
300512099529,16cyclictest26-21ksoftirqd/122:05:321
300512099526,20cyclictest26-21ksoftirqd/120:35:001
3005120995242,7cyclictest251rcuc/121:20:321
3005120995242,7cyclictest251rcuc/121:20:311
3005120995217,9cyclictest26-21ksoftirqd/123:55:011
3005120995214,21cyclictest3041914-21open_files20:50:291
3005125995120,9cyclictest42-21ksoftirqd/321:25:033
3005125995120,9cyclictest42-21ksoftirqd/321:25:023
3005125995116,15cyclictest42-21ksoftirqd/320:45:283
3005125995110,22cyclictest42-21ksoftirqd/321:50:023
3005125995110,22cyclictest42-21ksoftirqd/321:50:013
300512099516,10cyclictest26-21ksoftirqd/122:00:251
300512099512,19cyclictest3242253-21nautilus19:40:011
3005120995117,10cyclictest26-21ksoftirqd/123:30:021
3005120995117,10cyclictest26-21ksoftirqd/123:30:011
300512099511,5cyclictest141rcu_preempt20:00:141
300512099511,22cyclictest1753-21gnome-shell19:45:161
30109492500,5chrt0-21swapper/119:25:031
3005125995014,9cyclictest42-21ksoftirqd/322:40:003
300512499505,15cyclictest3068212-21ntp_states22:05:282
3005120995021,8cyclictest26-21ksoftirqd/121:10:021
3005120995020,8cyclictest26-21ksoftirqd/100:30:331
3005120995018,8cyclictest26-21ksoftirqd/119:10:321
3005120995016,10cyclictest26-21ksoftirqd/121:20:011
3005120995010,6cyclictest141rcu_preempt00:35:341
300511699509,5cyclictest141rcu_preempt19:45:000
300512599496,6cyclictest141rcu_preempt00:15:013
3005125994919,9cyclictest42-21ksoftirqd/320:25:023
3005125994919,9cyclictest42-21ksoftirqd/320:25:023
300512099499,23cyclictest26-21ksoftirqd/122:25:251
300512099499,23cyclictest26-21ksoftirqd/122:25:251
300512099495,9cyclictest26-21ksoftirqd/119:50:531
300512099495,10cyclictest26-21ksoftirqd/122:40:181
300512099494,24cyclictest141rcu_preempt19:25:241
300512099494,24cyclictest141rcu_preempt19:25:241
300512099494,10cyclictest26-21ksoftirqd/122:49:041
3005120994914,10cyclictest26-21ksoftirqd/123:20:261
3005120994914,10cyclictest26-21ksoftirqd/100:27:031
3005120994910,13cyclictest26-21ksoftirqd/120:19:231
30724492480,7chrt0-21swapper/022:20:010
30087032480,7sleep32990212-21kworker/u8:219:16:193
300512599486,12cyclictest42-21ksoftirqd/323:30:023
300512599486,12cyclictest42-21ksoftirqd/323:30:023
3005125994818,10cyclictest42-21ksoftirqd/321:30:023
3005125994818,10cyclictest42-21ksoftirqd/321:30:013
3005125994817,15cyclictest141rcu_preempt22:00:173
3005125994816,10cyclictest42-21ksoftirqd/320:34:593
3005125994812,9cyclictest42-21ksoftirqd/321:10:033
300512099489,14cyclictest26-21ksoftirqd/119:15:291
300512099483,31cyclictest26-21ksoftirqd/121:42:461
3005120994820,8cyclictest26-21ksoftirqd/121:00:201
3005120994819,7cyclictest26-21ksoftirqd/100:17:111
3005120994817,6cyclictest141rcu_preempt23:35:201
3005120994817,6cyclictest141rcu_preempt23:35:201
3005120994816,9cyclictest26-21ksoftirqd/123:30:121
3005120994816,9cyclictest26-21ksoftirqd/122:10:271
3005120994816,10cyclictest26-21ksoftirqd/123:41:431
3005120994811,15cyclictest26-21ksoftirqd/122:15:171
3005116994822,8cyclictest13-21ksoftirqd/021:05:010
300512599477,18cyclictest42-21ksoftirqd/320:50:163
300512599475,27cyclictest42-21ksoftirqd/319:28:483
300512599475,27cyclictest42-21ksoftirqd/319:28:473
3005125994717,5cyclictest141rcu_preempt22:30:323
3005125994716,11cyclictest141rcu_preempt20:58:153
300512599471,28cyclictest2197014-21PK-Backend21:43:463
3005125994712,19cyclictest42-21ksoftirqd/300:00:033
3005125994712,10cyclictest42-21ksoftirqd/321:40:003
3005125994710,6cyclictest141rcu_preempt23:12:243
300512099477,20cyclictest26-21ksoftirqd/120:15:021
300512099473,9cyclictest26-21ksoftirqd/121:36:221
300512099472,19cyclictest26-21ksoftirqd/120:20:241
300512099472,19cyclictest26-21ksoftirqd/120:20:241
3005120994716,12cyclictest1753-21gnome-shell20:28:491
3005120994716,12cyclictest1753-21gnome-shell20:28:481
3005120994715,8cyclictest26-21ksoftirqd/121:30:251
3005120994714,10cyclictest26-21ksoftirqd/122:58:481
3005120994712,6cyclictest141rcu_preempt00:07:541
3005120994711,10cyclictest26-21ksoftirqd/123:45:171
300512599469,15cyclictest141rcu_preempt21:56:313
300512599466,5cyclictest141rcu_preempt22:55:033
3005125994616,12cyclictest42-21ksoftirqd/322:55:403
3005125994616,12cyclictest141rcu_preempt23:05:313
3005125994614,11cyclictest141rcu_preempt20:14:593
3005125994613,6cyclictest141rcu_preempt21:20:003
3005125994613,18cyclictest42-21ksoftirqd/323:35:023
300512099467,14cyclictest26-21ksoftirqd/123:15:021
300512099466,20cyclictest26-21ksoftirqd/122:25:031
300512099466,17cyclictest26-21ksoftirqd/123:10:031
300512099466,15cyclictest3053446-21touch21:25:031
300512099466,15cyclictest3053446-21touch21:25:031
3005120994618,8cyclictest26-21ksoftirqd/120:43:091
3005120994612,9cyclictest26-21ksoftirqd/123:00:301
3005120994612,13cyclictest26-21ksoftirqd/120:45:221
3005116994624,14cyclictest13-21ksoftirqd/021:40:030
3005116994623,13cyclictest13-21ksoftirqd/021:30:150
3005116994620,8cyclictest13-21ksoftirqd/023:35:190
3005116994620,8cyclictest13-21ksoftirqd/023:35:180
3005125994514,8cyclictest42-21ksoftirqd/300:25:263
3005125994514,5cyclictest141rcu_preempt00:25:013
3005125994512,9cyclictest42-21ksoftirqd/323:21:103
300512599451,29cyclictest42-21ksoftirqd/319:53:183
300512599451,20cyclictest42-21ksoftirqd/320:05:153
300512599451,14cyclictest3075122-21/usr/sbin/munin22:25:263
300512599451,14cyclictest3075122-21/usr/sbin/munin22:25:263
300512099457,13cyclictest26-21ksoftirqd/121:50:151
300512099456,5cyclictest141rcu_preempt23:55:031
3005120994510,10cyclictest26-21ksoftirqd/121:45:321
3005120994510,10cyclictest26-21ksoftirqd/121:45:321
3005116994518,8cyclictest13-21ksoftirqd/000:29:340
300512599444,15cyclictest0-21swapper/322:40:023
3005125994417,11cyclictest42-21ksoftirqd/300:36:293
3005125994416,11cyclictest141rcu_preempt23:48:433
300512099449,10cyclictest26-21ksoftirqd/121:55:161
300512099445,14cyclictest26-21ksoftirqd/120:55:261
300512099443,13cyclictest26-21ksoftirqd/120:36:531
300512099441,22cyclictest3084491-21chrt22:54:391
300512099441,13cyclictest2197014-21PK-Backend19:30:121
3005120994410,10cyclictest26-21ksoftirqd/120:00:001
3005116994433,8cyclictest13-21ksoftirqd/020:00:140
300511699443,11cyclictest141rcu_preempt00:00:030
3005116994425,14cyclictest3242253-21nautilus20:25:020
3005116994425,14cyclictest3242253-21nautilus20:25:020
3005116994423,12cyclictest13-21ksoftirqd/000:00:140
3005116994422,13cyclictest13-21ksoftirqd/000:10:210
3005125994316,6cyclictest141rcu_preempt23:45:003
3005125994314,6cyclictest141rcu_preempt21:30:233
3005125994314,12cyclictest141rcu_preempt21:53:253
3005125994313,5cyclictest141rcu_preempt00:33:593
3005125994312,9cyclictest42-21ksoftirqd/322:15:253
300512599431,22cyclictest3029558-21/usr/sbin/munin20:15:283
300512099439,9cyclictest26-21ksoftirqd/122:30:121
300512099437,11cyclictest26-21ksoftirqd/123:19:251
300512099432,20cyclictest3108352-21df00:00:161
3005116994321,17cyclictest3242255-21gdbus19:15:020
3005116994318,9cyclictest13-21ksoftirqd/021:27:250
3005116994318,9cyclictest13-21ksoftirqd/021:27:240
3005116994317,5cyclictest141rcu_preempt23:52:210
300512599428,19cyclictest3099193-21cat23:35:163
300512599428,19cyclictest3099193-21cat23:35:163
300512599424,9cyclictest42-21ksoftirqd/320:25:033
300512599424,9cyclictest42-21ksoftirqd/320:25:033
300512599423,20cyclictest42-21ksoftirqd/300:03:563
3005125994221,9cyclictest42-21ksoftirqd/319:35:013
300512599421,4cyclictest141rcu_preempt19:11:443
3005125994212,8cyclictest42-21ksoftirqd/320:40:233
3005125994212,8cyclictest42-21ksoftirqd/320:35:163
300512599421,25cyclictest2197014-21PK-Backend22:20:163
3005125994211,9cyclictest42-21ksoftirqd/322:05:283
300512599421,16cyclictest2197014-21PK-Backend00:15:283
300512099427,19cyclictest26-21ksoftirqd/100:13:401
300511699428,5cyclictest141rcu_preempt20:25:150
300511699428,5cyclictest141rcu_preempt20:25:150
3005116994224,7cyclictest13-21ksoftirqd/020:40:150
3005116994224,14cyclictest13-21ksoftirqd/022:42:320
3005116994221,12cyclictest13-21ksoftirqd/023:40:120
3005116994213,15cyclictest13-21ksoftirqd/019:25:220
3005116994213,15cyclictest13-21ksoftirqd/019:25:210
300512599419,15cyclictest42-21ksoftirqd/319:45:243
300512599412,21cyclictest42-21ksoftirqd/322:48:313
300512599412,20cyclictest42-21ksoftirqd/323:03:243
3005125994114,7cyclictest141rcu_preempt19:25:003
300512599411,21cyclictest42-21ksoftirqd/323:15:333
300511699418,16cyclictest13-21ksoftirqd/023:21:470
300511699417,5cyclictest141rcu_preempt21:41:300
300511699416,5cyclictest141rcu_preempt19:48:150
3005116994128,9cyclictest13-21ksoftirqd/019:35:000
3005116994123,7cyclictest13-21ksoftirqd/020:51:300
3005116994123,7cyclictest13-21ksoftirqd/019:15:160
3005116994123,6cyclictest13-21ksoftirqd/000:30:160
3005116994122,6cyclictest13-21ksoftirqd/022:10:310
3005116994122,5cyclictest13-21ksoftirqd/023:15:250
3005116994122,14cyclictest13-21ksoftirqd/000:15:270
300511699412,10cyclictest1753-21gnome-shell20:37:010
3005116994119,13cyclictest13-21ksoftirqd/023:05:010
300512599409,17cyclictest42-21ksoftirqd/321:10:263
3005116994028,8cyclictest3097746-21grep23:30:230
3005116994023,13cyclictest13-21ksoftirqd/023:10:160
3005116994023,13cyclictest13-21ksoftirqd/021:24:320
3005116994023,13cyclictest13-21ksoftirqd/021:24:310
3005116994023,13cyclictest13-21ksoftirqd/020:55:220
3005116994022,6cyclictest13-21ksoftirqd/022:50:300
3005116994022,14cyclictest13-21ksoftirqd/020:10:240
3005116994022,13cyclictest13-21ksoftirqd/022:40:010
3005116994022,13cyclictest13-21ksoftirqd/019:21:080
300511699402,16cyclictest0-21swapper/022:20:120
3005116994015,6cyclictest141rcu_preempt20:05:200
3005116994013,5cyclictest141rcu_preempt00:39:430
300511699401,28cyclictest141rcu_preempt22:45:160
3005116994011,9cyclictest13-21ksoftirqd/023:45:140
3005116994010,6cyclictest141rcu_preempt19:35:190
300512599392,7cyclictest1753-21gnome-shell21:00:343
3005116993922,13cyclictest13-21ksoftirqd/021:05:290
3005116993921,6cyclictest13-21ksoftirqd/022:00:210
3005116993921,13cyclictest438-20systemd-journal22:09:210
3005116993921,13cyclictest13-21ksoftirqd/020:16:010
3005116993920,7cyclictest13-21ksoftirqd/022:25:160
3005116993920,7cyclictest13-21ksoftirqd/022:25:160
3005116993919,6cyclictest13-21ksoftirqd/023:25:250
3005116993919,6cyclictest13-21ksoftirqd/023:25:250
3005116993916,6cyclictest141rcu_preempt22:34:100
3005116993916,12cyclictest3034409-21cstates20:30:140
300512599381,19cyclictest3109848-21runrttasks00:07:123
300511699384,9cyclictest2197014-21PK-Backend21:50:370
3005116993822,6cyclictest13-21ksoftirqd/021:50:020
3005116993822,6cyclictest13-21ksoftirqd/021:50:020
3005116993820,13cyclictest13-21ksoftirqd/021:10:210
300511699381,33cyclictest3039578-21grep20:45:150
300511699381,16cyclictest2197014-21PK-Backend21:55:310
3005116993619,13cyclictest13-21ksoftirqd/021:16:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional