You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 12:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sun Nov 23, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
264833722300,5sleep30-21swapper/319:05:313
272208121120,6sleep22722061-21perf22:35:002
272445721010,8sleep126-21ksoftirqd/122:40:181
272445721010,8sleep126-21ksoftirqd/122:40:181
1623991000,10rtkit-daemon1622-21rtkit-daemon23:09:071
1623991000,10rtkit-daemon1622-21rtkit-daemon23:09:071
26956722960,4sleep10-21swapper/121:20:001
26956722960,4sleep10-21swapper/121:20:001
27069902920,6sleep10-21swapper/121:50:231
162399900,9rtkit-daemon1622-21rtkit-daemon21:44:401
162399900,9rtkit-daemon1622-21rtkit-daemon20:32:091
162399900,8rtkit-daemon1622-21rtkit-daemon19:58:011
26909672880,7sleep1251rcuc/121:05:151
162399880,8rtkit-daemon1622-21rtkit-daemon20:20:511
162399860,8rtkit-daemon1622-21rtkit-daemon21:01:171
162399840,8rtkit-daemon1622-21rtkit-daemon20:49:281
162399840,15rtkit-daemon1753-21gnome-shell22:52:480
162399830,8rtkit-daemon1622-21rtkit-daemon20:39:511
162399830,8rtkit-daemon1622-21rtkit-daemon00:23:481
162399830,70rtkit-daemon0-21swapper/219:05:032
162399790,5rtkit-daemon2197014-21PK-Backend22:49:280
162399790,5rtkit-daemon2197014-21PK-Backend22:49:280
162399760,5rtkit-daemon2197014-21PK-Backend22:56:530
264868627247,16sleep10-21swapper/119:09:411
2648863996857,8cyclictest42-21ksoftirqd/322:05:243
264885599640,48cyclictest1622-21rtkit-daemon00:37:061
264721826447,10sleep00-21swapper/019:05:050
26600172630,8sleep30-21swapper/319:40:003
162399610,8rtkit-daemon1622-21rtkit-daemon22:08:251
162399610,8rtkit-daemon1622-21rtkit-daemon00:10:291
26662842600,6chrt0-21swapper/319:55:303
162399600,8rtkit-daemon1622-21rtkit-daemon00:00:311
162399590,8rtkit-daemon1622-21rtkit-daemon23:57:501
162399580,9rtkit-daemon1622-21rtkit-daemon21:04:190
162399580,9rtkit-daemon1622-21rtkit-daemon20:33:520
162399580,8rtkit-daemon1622-21rtkit-daemon21:45:070
162399580,8rtkit-daemon1622-21rtkit-daemon20:42:591
162399580,8rtkit-daemon1622-21rtkit-daemon20:09:520
162399580,8rtkit-daemon1622-21rtkit-daemon19:56:320
162399580,8rtkit-daemon1622-21rtkit-daemon19:44:461
162399570,9rtkit-daemon1622-21rtkit-daemon23:46:281
162399570,8rtkit-daemon1622-21rtkit-daemon21:56:521
162399560,9rtkit-daemon1622-21rtkit-daemon22:55:112
162399560,9rtkit-daemon1622-21rtkit-daemon20:04:141
162399560,9rtkit-daemon1622-21rtkit-daemon00:06:261
162399560,8rtkit-daemon1622-21rtkit-daemon23:13:251
162399560,8rtkit-daemon1622-21rtkit-daemon21:38:051
162399550,8rtkit-daemon1622-21rtkit-daemon00:12:470
27603812540,6sleep30-21swapper/300:20:263
2648863995432,18cyclictest860997-21snmpd21:25:263
2648863995426,24cyclictest1753-21gnome-shell00:29:353
2648858995425,26cyclictest34-21ksoftirqd/219:20:012
162399540,8rtkit-daemon1622-21rtkit-daemon23:26:441
162399540,8rtkit-daemon1622-21rtkit-daemon23:22:401
162399540,8rtkit-daemon1622-21rtkit-daemon20:18:251
162399540,8rtkit-daemon1622-21rtkit-daemon00:16:151
264885599530,49cyclictest1622-21rtkit-daemon21:33:051
162399530,8rtkit-daemon1622-21rtkit-daemon22:38:131
162399530,8rtkit-daemon1622-21rtkit-daemon19:13:201
162399520,8rtkit-daemon1622-21rtkit-daemon23:41:351
162399520,8rtkit-daemon1622-21rtkit-daemon22:25:072
162399520,8rtkit-daemon1622-21rtkit-daemon22:01:240
26876992510,7sleep22197014-21PK-Backend20:55:232
2648863995126,21cyclictest1753-21gnome-shell20:01:043
2648858995125,8cyclictest3242255-21gdbus20:55:012
2648858995039,8cyclictest34-21ksoftirqd/221:55:182
2648863994928,17cyclictest1753-21gnome-shell00:09:453
264885899492,19cyclictest34-21ksoftirqd/222:10:002
2648855994923,22cyclictest2657360-40lsb_release19:30:261
162399490,5rtkit-daemon0-21swapper/022:22:550
162399490,5rtkit-daemon0-21swapper/022:22:550
27256832480,7chrt1753-21gnome-shell22:45:022
27256832480,7chrt1753-21gnome-shell22:45:022
2648863994823,7cyclictest1753-21gnome-shell21:59:333
2648858994816,10cyclictest34-21ksoftirqd/220:10:322
264885599480,44cyclictest1622-21rtkit-daemon19:23:131
26774342470,6sleep1251rcuc/120:30:011
2648863994720,10cyclictest42-21ksoftirqd/323:50:023
2648858994714,6cyclictest141rcu_preempt19:35:002
2648855994715,15cyclictest26-21ksoftirqd/123:30:201
162399470,35rtkit-daemon1753-21gnome-shell22:27:490
264886399462,6cyclictest141rcu_preempt22:25:303
264885899469,9cyclictest34-21ksoftirqd/220:50:002
2648858994612,8cyclictest34-21ksoftirqd/220:29:032
264885899461,21cyclictest34-21ksoftirqd/221:06:192
264885899454,20cyclictest34-21ksoftirqd/200:37:472
2648858994514,6cyclictest141rcu_preempt21:50:022
2648858994512,8cyclictest34-21ksoftirqd/222:11:082
264885899451,19cyclictest34-21ksoftirqd/200:00:022
2648855994518,7cyclictest2764800-21date00:35:021
2648863994419,9cyclictest42-21ksoftirqd/321:04:463
2648858994433,8cyclictest34-21ksoftirqd/221:40:012
2648858994432,8cyclictest34-21ksoftirqd/222:05:022
2648858994411,28cyclictest2745248-21unin-run23:40:012
264885599442,38cyclictest1622-21rtkit-daemon20:09:351
162399440,34rtkit-daemon2197014-21PK-Backend20:33:302
162399440,30rtkit-daemon1753-21gnome-shell19:38:582
2648863994331,8cyclictest1753-21gnome-shell22:22:193
2648863994331,8cyclictest1753-21gnome-shell22:22:193
264885899434,5cyclictest141rcu_preempt21:04:442
2648858994316,6cyclictest141rcu_preempt21:11:242
2648858994316,5cyclictest141rcu_preempt22:15:232
2648858994314,6cyclictest141rcu_preempt20:25:002
2648858994312,7cyclictest141rcu_preempt21:40:152
2648858994311,6cyclictest141rcu_preempt23:30:092
2648858994311,6cyclictest141rcu_preempt00:25:252
2648855994315,15cyclictest2704616-21grep21:45:151
2648855994310,8cyclictest1753-21gnome-shell22:10:271
264885299432,11cyclictest0-21swapper/021:54:470
264885299431,10cyclictest0-21swapper/021:39:490
162399430,35rtkit-daemon2197014-21PK-Backend23:07:312
162399430,35rtkit-daemon2197014-21PK-Backend23:07:312
162399430,31rtkit-daemon2197014-21PK-Backend00:24:372
162399430,29rtkit-daemon2197014-21PK-Backend23:35:113
264886399428,30cyclictest1753-21gnome-shell19:21:573
2648863994232,7cyclictest1753-21gnome-shell22:41:533
2648863994232,7cyclictest1753-21gnome-shell22:41:533
2648863994231,7cyclictest1753-21gnome-shell23:27:133
2648863994231,7cyclictest1753-21gnome-shell21:24:303
2648863994230,8cyclictest2664214-21cut19:50:213
2648863994224,6cyclictest42-21ksoftirqd/322:45:293
2648863994224,6cyclictest42-21ksoftirqd/322:45:293
2648863994218,20cyclictest42-21ksoftirqd/323:15:473
2648863994216,10cyclictest42-21ksoftirqd/321:35:283
2648863994215,10cyclictest42-21ksoftirqd/320:30:243
2648863994215,10cyclictest42-21ksoftirqd/320:30:003
2648858994214,5cyclictest141rcu_preempt19:47:372
2648858994212,5cyclictest141rcu_preempt00:35:002
2648858994210,6cyclictest141rcu_preempt23:20:002
2648858994210,6cyclictest141rcu_preempt20:44:572
264885599429,14cyclictest26-21ksoftirqd/119:15:291
2648855994224,13cyclictest2685783-21df_inode20:50:171
2648855994215,13cyclictest2696091-21cpuspeed_turbos21:20:151
2648852994231,7cyclictest1753-21gnome-shell20:59:060
264885299422,12cyclictest0-21swapper/021:20:540
264885299421,10cyclictest2197014-21PK-Backend21:58:250
162399420,29rtkit-daemon0-21swapper/221:21:562
2648863994131,7cyclictest1753-21gnome-shell23:54:503
2648863994131,7cyclictest1753-21gnome-shell21:31:473
2648863994130,7cyclictest1753-21gnome-shell23:59:223
2648863994130,7cyclictest1753-21gnome-shell22:35:353
2648863994130,7cyclictest1753-21gnome-shell21:51:463
2648863994130,7cyclictest1753-21gnome-shell00:17:203
2648863994129,8cyclictest1753-21gnome-shell23:21:313
2648863994124,13cyclictest2661015-21ntpq19:40:283
2648863994123,14cyclictest2680013-21df_inode20:35:173
2648863994123,13cyclictest42-21ksoftirqd/320:43:373
2648863994122,15cyclictest3242255-21gdbus22:55:013
2648863994122,14cyclictest2765262-21df00:35:163
2648863994120,13cyclictest42-21ksoftirqd/321:07:053
2648863994115,10cyclictest42-21ksoftirqd/322:20:023
264885899419,5cyclictest141rcu_preempt23:41:522
264885899418,8cyclictest34-21ksoftirqd/223:00:312
2648858994131,6cyclictest34-21ksoftirqd/220:36:052
2648858994123,6cyclictest34-21ksoftirqd/222:23:092
2648858994123,6cyclictest34-21ksoftirqd/222:23:092
264885899412,15cyclictest0-21swapper/223:52:422
2648852994123,14cyclictest2735333-21sensors_temp23:10:290
2648852994123,14cyclictest2715449-21awk22:15:130
162399410,29rtkit-daemon2197014-21PK-Backend20:06:482
2648863994030,7cyclictest1753-21gnome-shell21:18:063
2648863994030,7cyclictest1753-21gnome-shell21:18:063
2648863994030,6cyclictest42-21ksoftirqd/300:10:223
2648863994029,7cyclictest1753-21gnome-shell22:03:253
2648863994029,7cyclictest1753-21gnome-shell20:46:453
2648863994025,11cyclictest2673743-21sshd20:18:413
2648863994023,5cyclictest42-21ksoftirqd/323:32:573
2648863994023,13cyclictest42-21ksoftirqd/322:55:563
2648863994022,13cyclictest42-21ksoftirqd/321:40:073
2648863994020,15cyclictest994-21lldpd00:03:193
2648863994017,19cyclictest42-21ksoftirqd/320:07:523
2648863994017,12cyclictest2655110-21expr19:25:163
2648863994012,10cyclictest42-21ksoftirqd/319:10:163
264886399401,10cyclictest2197014-21PK-Backend20:10:103
2648858994025,11cyclictest34-21ksoftirqd/219:56:102
264885899401,19cyclictest34-21ksoftirqd/223:20:032
2648858994010,8cyclictest34-21ksoftirqd/223:45:152
264885599409,12cyclictest141rcu_preempt22:28:351
2648852994029,7cyclictest2681457-21cp20:40:030
2648852994026,10cyclictest2743312-21sshd23:33:030
2648852994024,12cyclictest2758140-21grep00:15:140
264885299402,34cyclictest0-21swapper/019:42:420
2648852994021,14cyclictest2736945-21ntp_states23:15:260
162399400,31rtkit-daemon2197014-21PK-Backend23:27:352
162399400,28rtkit-daemon0-21swapper/219:26:152
27617712390,5sleep10-21swapper/100:25:151
264886399396,6cyclictest141rcu_preempt23:40:123
264886399392,9cyclictest1753-21gnome-shell22:31:553
2648863993927,9cyclictest42-21ksoftirqd/322:15:013
2648863993923,5cyclictest42-21ksoftirqd/320:54:203
2648863993922,13cyclictest42-21ksoftirqd/300:32:263
2648863993917,5cyclictest42-21ksoftirqd/323:00:163
2648863993911,13cyclictest141rcu_preempt20:55:303
2648858993927,8cyclictest34-21ksoftirqd/200:15:002
2648858993924,12cyclictest34-21ksoftirqd/222:52:192
2648858993923,6cyclictest34-21ksoftirqd/221:26:172
2648858993923,13cyclictest34-21ksoftirqd/219:23:312
2648858993922,6cyclictest34-21ksoftirqd/223:10:512
2648858993922,6cyclictest34-21ksoftirqd/220:16:252
2648858993922,6cyclictest34-21ksoftirqd/219:11:182
2648858993922,5cyclictest34-21ksoftirqd/221:50:122
2648858993922,14cyclictest34-21ksoftirqd/222:35:112
2648855993924,11cyclictest2729003-21/usr/sbin/munin22:55:001
2648855993922,13cyclictest2655472-21ntpq19:25:261
2648855993921,14cyclictest2688273-21users20:55:321
2648855993912,8cyclictest1753-21gnome-shell22:20:291
2648855993912,8cyclictest1753-21gnome-shell22:20:291
264885599391,13cyclictest2693212-21ntp_states21:10:251
264885299393,10cyclictest13-21ksoftirqd/022:40:010
2648852993924,9cyclictest2656874-21cpuspeed_turbos19:30:160
264885299391,6cyclictest2725550-21munin-run22:45:000
264885299391,6cyclictest2725550-21munin-run22:45:000
264885299391,34cyclictest2654442-21taskset19:24:440
2648863993827,7cyclictest1753-21gnome-shell23:05:173
2648863993827,7cyclictest1753-21gnome-shell23:05:173
2648863993826,8cyclictest42-21ksoftirqd/319:50:013
2648863993822,5cyclictest42-21ksoftirqd/323:10:313
2648863993821,13cyclictest42-21ksoftirqd/321:50:003
2648863993821,13cyclictest42-21ksoftirqd/319:15:363
2648863993814,5cyclictest42-21ksoftirqd/319:30:203
264885899382,7cyclictest1753-21gnome-shell20:02:442
2648858993826,9cyclictest2757937-21/usr/sbin/munin00:15:332
2648858993821,6cyclictest34-21ksoftirqd/221:30:262
2648858993821,5cyclictest34-21ksoftirqd/221:16:402
2648858993821,5cyclictest34-21ksoftirqd/221:16:392
2648858993821,13cyclictest1632-21dbus-daemon22:50:012
2648858993821,13cyclictest1632-21dbus-daemon22:50:012
264885899381,15cyclictest2197014-21PK-Backend19:50:312
2648855993823,11cyclictest855-21dbus-daemon23:01:561
2648855993821,13cyclictest2736937-21ntpq23:15:261
2648855993820,13cyclictest438-20systemd-journal19:40:001
264885599381,8cyclictest1753-21gnome-shell22:20:011
2648855993811,6cyclictest0-21swapper/122:00:291
2648855993810,8cyclictest26-21ksoftirqd/121:25:081
2648855993810,13cyclictest3242253-21nautilus23:55:021
264885299388,5cyclictest141rcu_preempt23:00:090
2648852993826,7cyclictest2676043-21tr20:25:170
2648852993826,7cyclictest1753-21gnome-shell20:46:050
2648852993820,13cyclictest2712439-21systemd-udevd22:05:240
2648852993819,14cyclictest438-20systemd-journal19:40:000
162399380,29rtkit-daemon2197014-21PK-Backend23:35:571
162399380,28rtkit-daemon2197014-21PK-Backend23:51:140
162399380,28rtkit-daemon2197014-21PK-Backend23:20:250
162399380,28rtkit-daemon2197014-21PK-Backend19:52:520
162399380,28rtkit-daemon2197014-21PK-Backend00:06:050
2648863993727,7cyclictest1753-21gnome-shell20:20:453
2648863993710,7cyclictest2693552-21users21:10:313
2648858993722,5cyclictest34-21ksoftirqd/200:01:262
2648858993720,5cyclictest34-21ksoftirqd/200:05:102
264885299374,11cyclictest1069-21runrttasks20:42:030
264885299372,32cyclictest0-21swapper/000:21:150
2648852993722,11cyclictest2656209-21apt-get19:30:000
2648852993720,12cyclictest2686364-21find20:50:300
264885299371,9cyclictest2197014-21PK-Backend23:35:230
2648852993719,13cyclictest438-20systemd-journal21:34:410
264885299371,4cyclictest2197014-21PK-Backend21:18:030
264885299371,4cyclictest2197014-21PK-Backend21:18:030
264885299371,30cyclictest2197014-21PK-Backend20:00:140
264885299371,15cyclictest2752594-21cat00:00:030
264885299371,12cyclictest141rcu_preempt20:10:210
2648858993620,5cyclictest34-21ksoftirqd/219:41:302
2648855993611,6cyclictest141rcu_preempt20:10:301
2648852993627,5cyclictest181irq_work/021:10:140
2648852993625,7cyclictest2745636-21cpuspeed_turbos23:40:140
2648852993625,7cyclictest1753-21gnome-shell23:07:420
2648852993625,7cyclictest1753-21gnome-shell23:07:420
2648852993623,10cyclictest2748694-21awk23:50:010
264885299362,11cyclictest1358-1kworker/u9:119:15:290
264885299361,9cyclictest0-21swapper/000:29:340
2648852993616,16cyclictest2766392-21systemctl00:40:010
264885299361,4cyclictest2197014-21PK-Backend19:45:250
2648852993612,11cyclictest141rcu_preempt00:31:430
264885299361,12cyclictest141rcu_preempt20:20:000
264885599352,10cyclictest2663467-21munin-run19:50:001
264885599351,12cyclictest2726152-21expr22:45:161
264885599351,12cyclictest2726152-21expr22:45:151
264885299354,12cyclictest0-21swapper/021:40:240
2648852993526,5cyclictest2197014-21PK-Backend00:00:410
264885299352,29cyclictest1753-21gnome-shell22:12:470
2648852993520,11cyclictest2649814-21open_files19:10:290
2648852993518,13cyclictest2691241-21sed21:05:230
264885299351,4cyclictest2197014-21PK-Backend22:31:120
264885599346,12cyclictest26-21ksoftirqd/119:52:141
264885599342,10cyclictest1753-21gnome-shell22:55:161
264885299344,7cyclictest141rcu_preempt20:22:200
2648852993423,7cyclictest1069-21runrttasks23:29:320
264885299341,10cyclictest2197014-21PK-Backend21:25:080
2648855993322,7cyclictest1753-21gnome-shell22:30:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional