You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 04:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Nov 18, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
399170822120,6sleep00-21swapper/019:05:230
405851721090,8sleep32197014-21PK-Backend22:15:183
1623991020,15rtkit-daemon0-21swapper/122:26:031
1623991020,15rtkit-daemon0-21swapper/122:26:031
399218829445,16sleep10-21swapper/119:06:411
162399920,15rtkit-daemon0-21swapper/120:09:151
40517962890,4sleep10-21swapper/121:55:241
162399830,9rtkit-daemon1753-21gnome-shell21:58:390
399259299752,31cyclictest1753-21gnome-shell00:20:150
162399720,5rtkit-daemon0-21swapper/020:14:180
162399710,5rtkit-daemon0-21swapper/020:24:580
399226126954,9sleep20-21swapper/219:07:392
399109226945,16sleep30-21swapper/319:05:153
162399690,5rtkit-daemon0-21swapper/020:45:470
162399640,53rtkit-daemon251rcuc/100:04:431
162399630,10rtkit-daemon1622-21rtkit-daemon19:49:501
40955762620,6sleep3411rcuc/300:00:183
399259299621,34cyclictest1753-21gnome-shell23:49:080
399259299612,34cyclictest1753-21gnome-shell23:10:190
399259299612,34cyclictest1753-21gnome-shell23:10:180
399259299612,33cyclictest1753-21gnome-shell20:59:590
399259299612,30cyclictest3997620-21python319:20:320
162399610,53rtkit-daemon34-21ksoftirqd/220:36:522
40367072600,7chrt0-21swapper/121:10:361
40367072600,7chrt0-21swapper/121:10:361
399259299602,40cyclictest1753-21gnome-shell23:24:390
399259299593,33cyclictest1753-21gnome-shell21:52:170
399259299582,33cyclictest1753-21gnome-shell22:02:160
399259299582,29cyclictest1753-21gnome-shell21:30:350
399259299582,29cyclictest1753-21gnome-shell20:05:340
399259299582,29cyclictest1753-21gnome-shell00:08:260
399259299582,28cyclictest1753-21gnome-shell22:39:140
399259299581,32cyclictest4042122-21taskset21:28:230
399259299581,31cyclictest4076006-21cat23:05:020
399259299572,32cyclictest1753-21gnome-shell22:14:030
399259299572,29cyclictest1753-21gnome-shell22:26:440
399259299572,29cyclictest1753-21gnome-shell22:26:430
399259299572,28cyclictest4017230-21ntp_states20:15:270
399259299572,28cyclictest1753-21gnome-shell23:35:010
399259299572,28cyclictest1753-21gnome-shell23:35:010
399259299562,32cyclictest1753-21gnome-shell19:26:380
399259299562,30cyclictest4072603-21/usr/sbin/munin22:55:350
399259299562,30cyclictest4035282-21seq21:09:350
399259299562,30cyclictest1753-21gnome-shell00:05:020
399259299562,26cyclictest4001993-21apt19:35:120
399259299561,30cyclictest4045636-21seq21:38:530
162399560,39rtkit-daemon0-21swapper/100:07:211
399259299552,31cyclictest1753-21gnome-shell22:20:140
399259299552,31cyclictest1753-21gnome-shell21:41:110
399259299552,28cyclictest4099402-21ntp_states00:10:250
399259299551,30cyclictest4077624-21cat23:10:020
162399550,5rtkit-daemon2197014-21PK-Backend20:18:072
40330122540,7chrt2197014-21PK-Backend21:00:312
399259299542,30cyclictest4095147-21latency_hist00:00:030
399259299542,28cyclictest3242253-21nautilus22:10:010
399259299541,28cyclictest4032346-21cstates21:00:160
162399540,35rtkit-daemon4079927-21df23:15:171
40219842530,6sleep00-21swapper/020:30:150
399259299532,28cyclictest4101873-21munin-run00:20:000
399259299532,27cyclictest4024220-21ntp_states20:35:300
3992592995321,27cyclictest438-20systemd-journal20:04:360
399259299522,26cyclictest1753-21gnome-shell19:34:290
399259299512,28cyclictest1753-21gnome-shell22:30:070
3992592995119,27cyclictest4069104-21sed22:45:020
399259299511,45cyclictest4083327-21expr23:25:180
399259299511,35cyclictest40361221latency21:10:250
399259299511,35cyclictest40361221latency21:10:250
399259299502,42cyclictest4088602-21/usr/sbin/munin23:40:190
399259299502,27cyclictest1753-21gnome-shell00:30:230
3992592995015,30cyclictest4007642-21grep19:50:160
399259299501,31cyclictest1069-21runrttasks20:29:020
399259299495,40cyclictest1069-21runrttasks19:46:310
399259299491,27cyclictest1069-21runrttasks21:23:330
162399490,36rtkit-daemon1753-21gnome-shell22:15:151
162399490,35rtkit-daemon4038055-21ntpq21:15:251
399259299481,42cyclictest4069568-21sh22:45:230
399259299481,42cyclictest4037518-21cpuspeed_turbos21:15:140
3992602994722,20cyclictest1753-21gnome-shell20:40:262
399259299472,40cyclictest1753-21gnome-shell20:50:280
3992592994721,22cyclictest1069-21runrttasks19:41:420
3992592994719,24cyclictest1753-21gnome-shell00:36:100
3992592994715,28cyclictest4080331-21open_inodes23:15:290
3992592994714,28cyclictest4048489-21ntpq21:45:250
162399470,35rtkit-daemon4025203-21apt20:40:131
40543982460,21chrt4054394-21cron22:05:003
399259299465,37cyclictest1069-21runrttasks19:14:240
399259299462,24cyclictest0-21swapper/020:44:300
399259299462,23cyclictest0-21swapper/019:18:330
3992592994615,26cyclictest4058419-40dpkg22:15:170
399259299461,40cyclictest1753-21gnome-shell22:52:170
162399460,32rtkit-daemon0-21swapper/100:39:551
3992607994520,11cyclictest4049688-21/usr/sbin/munin21:50:173
3992597994533,8cyclictest4032113-21find21:00:131
399259299452,39cyclictest1753-21gnome-shell23:54:530
399259299452,39cyclictest1753-21gnome-shell23:54:530
3992592994512,28cyclictest4009538-21expr19:55:180
162399450,31rtkit-daemon0-21swapper/123:02:411
399260799445,9cyclictest42-21ksoftirqd/300:00:003
3992602994425,15cyclictest4084523-21pool-org.gnome.23:30:002
3992602994410,7cyclictest141rcu_preempt22:45:352
3992597994422,18cyclictest1753-21gnome-shell21:25:421
3992597994420,20cyclictest4019976-21/usr/sbin/munin20:25:181
399259299449,31cyclictest1753-21gnome-shell00:27:300
3992592994410,29cyclictest4087433-21cat23:35:290
162399440,32rtkit-daemon0-21swapper/122:08:051
40898232430,7chrt2197014-21PK-Backend23:45:023
3992607994314,10cyclictest4065912-21grep22:35:243
399260299436,7cyclictest17531gnome-shell20:23:342
399260299431,24cyclictest4010624-21chrt19:59:172
399260299431,11cyclictest4058840-21smartctl22:15:252
3992597994314,11cyclictest4099024-21df00:10:151
399260799428,16cyclictest1753-21gnome-shell22:33:163
3992607994217,21cyclictest3992847-21cpuspeed19:10:153
3992607994216,10cyclictest1753-21gnome-shell20:38:393
162399420,31rtkit-daemon906-21gmain21:48:351
162399420,30rtkit-daemon0-21swapper/123:40:531
162399420,30rtkit-daemon0-21swapper/120:38:491
40197402410,7chrt4019739-21cut20:25:001
399260799419,29cyclictest0-21swapper/319:22:373
399260799412,9cyclictest1753-21gnome-shell22:24:183
3992607994123,6cyclictest42-21ksoftirqd/322:44:323
3992607994122,15cyclictest4056131-21cron22:10:003
3992607994115,8cyclictest4087311-21processes23:35:263
3992607994112,11cyclictest42-21ksoftirqd/300:38:473
3992597994123,13cyclictest438-20systemd-journal21:51:351
162399410,31rtkit-daemon2197014-21PK-Backend23:51:242
162399410,31rtkit-daemon2197014-21PK-Backend23:51:242
162399410,29rtkit-daemon0-21swapper/120:15:141
40706772400,8chrt0-21swapper/122:50:021
399260799408,12cyclictest141rcu_preempt19:15:143
399260799407,15cyclictest1753-21gnome-shell20:50:183
399260799401,34cyclictest4056500-21cat22:10:143
399260799401,10cyclictest4005204-21munin-plugin-st19:45:013
399260299409,6cyclictest141rcu_preempt22:12:382
399260299409,5cyclictest141rcu_preempt22:30:152
3992602994022,13cyclictest34-21ksoftirqd/221:10:012
399259799409,9cyclictest26-21ksoftirqd/119:21:301
3992597994029,7cyclictest1753-21gnome-shell23:39:301
3992597994024,11cyclictest438-20systemd-journal20:46:361
3992597994022,14cyclictest26-21ksoftirqd/123:15:011
3992597994022,14cyclictest26-21ksoftirqd/123:15:011
3992597994021,14cyclictest4072998-21df_inode22:55:181
3992597994018,6cyclictest26-21ksoftirqd/121:05:301
3992597994012,23cyclictest4014704-21apt20:10:151
3992597994012,12cyclictest4094934-40find00:00:001
39931662390,25sleep11753-21gnome-shell19:10:181
399260799398,6cyclictest141rcu_preempt21:35:153
3992607993929,7cyclictest1753-21gnome-shell21:07:413
3992607993921,13cyclictest3242253-21nautilus20:00:013
3992607993918,17cyclictest4038094-21/usr/sbin/munin21:15:273
3992607993914,7cyclictest1753-21gnome-shell00:10:263
399260799391,34cyclictest4016222-21awk20:15:003
3992607993912,23cyclictest0-21swapper/323:49:103
399260799391,15cyclictest1358-1kworker/u9:120:05:193
3992602993923,12cyclictest3996191-21sshd19:16:362
3992602993922,13cyclictest34-21ksoftirqd/219:11:482
3992602993913,5cyclictest141rcu_preempt22:55:152
3992597993922,13cyclictest4082378-21users23:20:331
3992597993913,22cyclictest4010727-21unin-run20:00:011
162399390,5rtkit-daemon2197014-21PK-Backend23:32:132
162399390,5rtkit-daemon2197014-21PK-Backend23:32:132
162399390,31rtkit-daemon1753-21gnome-shell00:32:571
162399390,27rtkit-daemon0-21swapper/123:34:251
162399390,27rtkit-daemon0-21swapper/123:34:251
162399390,27rtkit-daemon0-21swapper/123:29:091
162399390,26rtkit-daemon0-21swapper/120:57:271
162399390,26rtkit-daemon0-21swapper/100:24:241
162399390,25rtkit-daemon0-21swapper/100:28:401
3992607993823,11cyclictest4043277-21ntp_offset21:30:253
3992607993822,12cyclictest4097696-21open_inodes00:05:273
3992607993815,14cyclictest42-21ksoftirqd/320:00:253
3992607993811,23cyclictest1753-21gnome-shell21:13:493
3992607993811,23cyclictest1753-21gnome-shell21:13:483
3992602993824,10cyclictest34-21ksoftirqd/200:00:152
3992602993821,13cyclictest34-21ksoftirqd/220:45:502
3992602993821,13cyclictest34-21ksoftirqd/200:10:282
3992602993821,13cyclictest34-21ksoftirqd/200:07:102
3992602993819,15cyclictest4021494-21pool-org.gnome.20:30:002
399260299381,16cyclictest2197014-21PK-Backend22:55:012
399260299381,12cyclictest2197014-21PK-Backend20:55:022
3992597993827,7cyclictest1753-21gnome-shell00:15:311
3992597993825,9cyclictest26-21ksoftirqd/123:05:321
399259799382,32cyclictest1753-21gnome-shell19:42:581
3992597993820,13cyclictest26-21ksoftirqd/120:35:011
3992597993818,5cyclictest26-21ksoftirqd/123:50:341
3992597993818,5cyclictest26-21ksoftirqd/123:50:331
162399380,26rtkit-daemon0-21swapper/120:54:251
162399380,26rtkit-daemon0-21swapper/119:54:251
162399380,25rtkit-daemon0-21swapper/123:45:361
162399380,25rtkit-daemon0-21swapper/122:41:311
162399380,25rtkit-daemon0-21swapper/122:34:011
162399380,25rtkit-daemon0-21swapper/122:10:091
162399380,25rtkit-daemon0-21swapper/121:43:001
399260799376,5cyclictest141rcu_preempt23:25:333
399260799373,30cyclictest1753-21gnome-shell22:25:543
399260799373,30cyclictest1753-21gnome-shell22:25:543
3992607993726,7cyclictest4003456-21latency_hist19:40:003
3992607993723,11cyclictest42-21ksoftirqd/320:20:173
3992607993723,10cyclictest4073304-21grep22:55:253
399260799372,30cyclictest3242253-21nautilus19:50:023
3992607993712,21cyclictest0-21swapper/323:50:313
3992607993712,21cyclictest0-21swapper/323:50:313
399260799371,11cyclictest42-21ksoftirqd/323:02:323
399260299376,10cyclictest141rcu_preempt00:17:112
399260299373,9cyclictest34-21ksoftirqd/221:59:562
3992602993725,7cyclictest438-20systemd-journal22:45:002
3992602993724,9cyclictest4000425-21tr19:30:142
3992602993721,5cyclictest34-21ksoftirqd/200:25:282
3992602993712,21cyclictest1753-21gnome-shell19:45:022
399260299371,13cyclictest2197014-21PK-Backend22:05:152
399260299371,10cyclictest34-21ksoftirqd/221:52:312
3992597993726,7cyclictest1753-21gnome-shell19:34:321
162399370,30rtkit-daemon26-21ksoftirqd/119:26:451
162399370,26rtkit-daemon0-21swapper/122:01:401
162399370,25rtkit-daemon0-21swapper/122:24:591
162399370,25rtkit-daemon0-21swapper/120:02:251
162399370,24rtkit-daemon0-21swapper/122:35:591
399260799368,5cyclictest141rcu_preempt20:57:143
3992607993623,9cyclictest1753-21gnome-shell20:42:593
3992607993623,9cyclictest1753-21gnome-shell00:20:003
399260799362,16cyclictest1753-21gnome-shell20:17:503
399260799362,15cyclictest1753-21gnome-shell22:48:493
3992607993620,12cyclictest4080317-21ntpq23:15:273
399260799361,9cyclictest2197014-21PK-Backend00:20:323
399260299362,9cyclictest34-21ksoftirqd/220:57:402
399260299362,6cyclictest2197014-21PK-Backend19:50:022
3992602993619,13cyclictest34-21ksoftirqd/223:12:572
3992602993619,13cyclictest34-21ksoftirqd/223:12:572
3992602993618,14cyclictest1753-21gnome-shell23:57:512
399260299361,13cyclictest1358-1kworker/u9:123:15:312
399260299361,12cyclictest2197014-21PK-Backend21:20:202
3992597993621,11cyclictest4001964-21seq19:35:081
3992597993620,12cyclictest3994573-21taskset19:15:031
3992597993620,12cyclictest26-21ksoftirqd/122:50:151
162399360,24rtkit-daemon0-21swapper/121:35:441
399260799354,6cyclictest141rcu_preempt21:29:143
3992607993523,8cyclictest1753-21gnome-shell23:30:593
3992607993523,8cyclictest1753-21gnome-shell23:30:593
399260799352,29cyclictest1753-21gnome-shell21:20:293
3992607993522,8cyclictest1753-21gnome-shell22:50:053
399260799352,28cyclictest1753-21gnome-shell19:27:533
399260799352,15cyclictest1753-21gnome-shell21:03:113
3992607993521,10cyclictest4079355-21cut23:15:023
3992607993521,10cyclictest4079355-21cut23:15:023
3992607993520,11cyclictest4104322-21if_err_enp2s000:25:223
3992607993520,10cyclictest4076176-21cat23:05:173
399260799351,30cyclictest1753-21gnome-shell23:20:123
3992607993512,19cyclictest1753-21gnome-shell21:55:343
399260799351,11cyclictest2197014-21PK-Backend20:30:223
399260299358,6cyclictest141rcu_preempt22:30:012
399260299358,6cyclictest141rcu_preempt22:30:012
399260299357,7cyclictest4088159-21chrt23:40:022
3992602993523,8cyclictest4012682-21/usr/sbin/munin20:05:292
3992602993518,5cyclictest34-21ksoftirqd/200:34:562
3992602993518,12cyclictest4023187-21cron20:35:002
3992602993517,14cyclictest1787-21gdbus19:21:312
3992602993517,13cyclictest438-20systemd-journal23:45:482
3992602993511,5cyclictest141rcu_preempt19:39:282
399260299351,12cyclictest2197014-21PK-Backend22:20:202
399260299351,12cyclictest2197014-21PK-Backend19:53:482
399260299351,10cyclictest2197014-21PK-Backend21:15:142
3992597993513,18cyclictest4042547-21cpu21:30:141
162399350,24rtkit-daemon0-21swapper/121:24:001
399260799347,6cyclictest141rcu_preempt21:40:293
3992607993422,8cyclictest1753-21gnome-shell21:48:113
3992607993422,8cyclictest1753-21gnome-shell19:33:183
3992602993423,6cyclictest40477052chrt21:45:032
3992602993421,9cyclictest1826-21JS14:31:442
3992602993420,10cyclictest4081276-21basename23:20:122
3992602993419,11cyclictest4089692-21grep23:45:002
399260299341,6cyclictest40000932cat19:30:012
3992602993416,14cyclictest1753-21gnome-shell21:12:542
3992602993416,14cyclictest1753-21gnome-shell21:12:542
399260299341,14cyclictest2197014-21PK-Backend22:04:052
399260299341,12cyclictest2197014-21PK-Backend21:44:482
399260299341,12cyclictest2197014-21PK-Backend21:36:312
399260799336,5cyclictest141rcu_preempt19:46:433
399260799335,6cyclictest141rcu_preempt20:50:013
3992607993319,10cyclictest4105794-21cstates00:30:133
3992602993321,8cyclictest34-21ksoftirqd/222:35:182
399260299331,9cyclictest2197014-21PK-Backend00:35:132
399260299331,11cyclictest2197014-21PK-Backend20:01:092
399260299331,10cyclictest0-21swapper/221:33:002
402133223215,12sleep342-21ksoftirqd/320:28:333
399260299322,17cyclictest854-21avahi-daemon20:10:042
399260299321,12cyclictest2197014-21PK-Backend23:01:452
399260299321,12cyclictest2197014-21PK-Backend00:20:192
399260299321,10cyclictest2197014-21PK-Backend23:09:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional