You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-11 - 09:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sun May 11, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
418003821040,5sleep32197014-21PK-Backend23:15:243
413995121000,5sleep342-21ksoftirqd/321:20:193
409333929045,37sleep10-21swapper/119:08:541
409333929045,37sleep10-21swapper/119:08:531
162399830,26rtkit-daemon2197014-21PK-Backend22:55:590
162399820,24rtkit-daemon0-21swapper/023:12:400
162399810,6rtkit-daemon0-21swapper/000:31:060
162399810,68rtkit-daemon0-21swapper/319:09:153
162399810,68rtkit-daemon0-21swapper/319:09:153
409338827639,9sleep20-21swapper/219:09:312
409338827639,9sleep20-21swapper/219:09:312
41903772730,8sleep02197014-21PK-Backend23:45:180
409183127140,24sleep00-21swapper/019:05:030
409183127140,24sleep00-21swapper/019:05:020
162399710,7rtkit-daemon1622-21rtkit-daemon22:35:242
162399660,8rtkit-daemon1622-21rtkit-daemon00:10:463
162399650,8rtkit-daemon1622-21rtkit-daemon21:00:042
162399650,8rtkit-daemon1622-21rtkit-daemon20:38:503
162399650,8rtkit-daemon1622-21rtkit-daemon00:15:322
409356799641,8cyclictest4170424-21turbostat.cron22:50:002
409356799641,8cyclictest4170424-21turbostat.cron22:49:592
162399640,8rtkit-daemon1622-21rtkit-daemon20:19:373
162399630,9rtkit-daemon1622-21rtkit-daemon21:27:403
162399620,9rtkit-daemon1622-21rtkit-daemon21:59:303
162399620,9rtkit-daemon1622-21rtkit-daemon20:26:153
41722432610,7chrt1753-21gnome-shell22:52:300
162399610,9rtkit-daemon1622-21rtkit-daemon21:21:142
162399610,9rtkit-daemon1622-21rtkit-daemon20:50:113
4093567996018,14cyclictest855-21dbus-daemon22:50:042
162399600,8rtkit-daemon1622-21rtkit-daemon21:13:053
162399600,8rtkit-daemon1622-21rtkit-daemon21:13:053
162399590,9rtkit-daemon1622-21rtkit-daemon22:02:493
162399590,9rtkit-daemon1622-21rtkit-daemon00:38:413
162399580,9rtkit-daemon1622-21rtkit-daemon20:02:453
4093567995723,14cyclictest4152288-21grep21:55:252
162399570,10rtkit-daemon1622-21rtkit-daemon20:59:202
4093567995618,13cyclictest855-21dbus-daemon23:50:002
162399560,9rtkit-daemon1622-21rtkit-daemon22:08:123
162399550,8rtkit-daemon1622-21rtkit-daemon21:09:413
162399550,8rtkit-daemon1622-21rtkit-daemon20:30:143
409356799541,31cyclictest34-21ksoftirqd/223:32:242
409356499547,17cyclictest231irq_work/100:00:021
162399540,8rtkit-daemon1622-21rtkit-daemon21:15:293
41102512530,6sleep342-21ksoftirqd/319:55:193
162399530,8rtkit-daemon1622-21rtkit-daemon23:51:502
41672722520,8sleep32197014-21PK-Backend22:40:183
4093564995122,24cyclictest26-21ksoftirqd/119:15:121
162399510,7rtkit-daemon1622-21rtkit-daemon22:39:003
4093564995023,23cyclictest26-21ksoftirqd/122:04:161
41494372480,7sleep02197014-21PK-Backend21:47:170
4093564994822,22cyclictest26-21ksoftirqd/120:10:141
162399480,41rtkit-daemon1775-21gvfs-afc-volume20:24:263
4093564994721,21cyclictest4128163-40lsb_release20:45:261
4093560994727,16cyclictest15086-21cron00:40:000
162399470,6rtkit-daemon1622-21rtkit-daemon20:27:462
4093564994619,23cyclictest4183860-21open_files23:25:281
4093570994526,15cyclictest42-21ksoftirqd/300:25:173
4093570994523,14cyclictest42-21ksoftirqd/323:29:573
4093560994519,22cyclictest4116475-21cron20:14:590
162399450,37rtkit-daemon1693-40tracker-miner-f20:31:052
162399450,36rtkit-daemon4093555-21cyclictest00:04:323
162399450,30rtkit-daemon1753-21gnome-shell00:35:491
41465322440,6chrt4146535-21sort21:40:173
4093570994418,23cyclictest42-21ksoftirqd/322:20:163
4093570994417,23cyclictest42-21ksoftirqd/321:40:023
409357099441,6cyclictest141rcu_preempt00:00:033
4093564994424,15cyclictest26-21ksoftirqd/122:55:331
4093564994419,5cyclictest26-21ksoftirqd/123:16:121
409356099442,37cyclictest4128935-21munin-run20:50:020
4093560994410,9cyclictest13-21ksoftirqd/022:33:220
162399440,31rtkit-daemon0-21swapper/123:11:141
52412430,6sleep10-21swapper/100:10:161
4093564994324,14cyclictest26-21ksoftirqd/123:50:301
4093564994323,15cyclictest4106959-21memory19:45:211
4093564994322,5cyclictest26-21ksoftirqd/122:40:191
4093560994326,13cyclictest4104000-21users19:35:310
162399430,9rtkit-daemon2007-21evolution-alarm23:24:173
162399430,32rtkit-daemon2197014-21PK-Backend23:05:330
4093570994223,14cyclictest42-21ksoftirqd/320:55:333
4093567994223,14cyclictest1-21systemd20:10:012
4093564994224,6cyclictest26-21ksoftirqd/122:50:351
4093564994224,14cyclictest26-21ksoftirqd/123:30:561
4093564994224,14cyclictest26-21ksoftirqd/121:59:401
4093564994224,14cyclictest26-21ksoftirqd/121:54:071
4093564994224,14cyclictest26-21ksoftirqd/121:00:001
4093564994224,14cyclictest26-21ksoftirqd/120:29:391
4093564994224,14cyclictest26-21ksoftirqd/119:43:371
4093564994224,14cyclictest26-21ksoftirqd/100:01:051
4093564994223,6cyclictest26-21ksoftirqd/122:30:171
4093564994223,6cyclictest26-21ksoftirqd/121:10:291
4093564994223,6cyclictest26-21ksoftirqd/121:10:281
4093564994223,15cyclictest26-21ksoftirqd/123:00:321
4093564994222,7cyclictest26-21ksoftirqd/121:10:021
4093564994221,17cyclictest4142996-21/usr/sbin/munin21:30:391
409356499421,15cyclictest41120962latency20:00:241
409356099425,10cyclictest1358-1kworker/u9:121:10:440
409356099425,10cyclictest1358-1kworker/u9:121:10:440
409356099421,37cyclictest5365-21diskstats00:10:180
162399420,35rtkit-daemon970-21ntpd22:47:043
162399420,35rtkit-daemon970-21ntpd22:47:043
162399420,35rtkit-daemon970-21ntpd21:46:533
162399420,35rtkit-daemon2287-21gmain21:49:492
162399420,35rtkit-daemon2197014-21PK-Backend22:21:382
162399420,35rtkit-daemon2197014-21PK-Backend21:27:562
41223962410,7sleep126-21ksoftirqd/120:30:181
4093570994124,12cyclictest42-21ksoftirqd/323:49:593
4093570994122,16cyclictest42-21ksoftirqd/321:03:213
4093570994116,21cyclictest1753-21gnome-shell21:33:173
4093567994130,6cyclictest122332sleep200:30:172
4093567994128,9cyclictest4154879-21pool-org.gnome.22:05:002
4093567994123,13cyclictest3242253-21nautilus19:40:022
4093564994124,6cyclictest26-21ksoftirqd/123:20:321
4093564994124,6cyclictest26-21ksoftirqd/121:23:151
4093564994123,6cyclictest26-21ksoftirqd/122:48:551
4093564994123,6cyclictest26-21ksoftirqd/122:48:551
4093564994123,6cyclictest26-21ksoftirqd/122:35:151
4093564994123,6cyclictest26-21ksoftirqd/122:15:011
4093564994123,6cyclictest26-21ksoftirqd/119:39:271
4093564994123,6cyclictest26-21ksoftirqd/119:25:021
4093564994123,6cyclictest26-21ksoftirqd/119:25:011
4093564994123,6cyclictest26-21ksoftirqd/100:05:181
4093564994123,5cyclictest26-21ksoftirqd/121:25:291
4093564994123,14cyclictest26-21ksoftirqd/100:19:521
4093564994122,7cyclictest26-21ksoftirqd/123:40:271
4093564994122,6cyclictest26-21ksoftirqd/121:45:291
4093564994122,6cyclictest26-21ksoftirqd/120:25:011
4093564994122,6cyclictest26-21ksoftirqd/119:10:321
4093564994122,14cyclictest26-21ksoftirqd/119:55:011
4093564994119,13cyclictest26-21ksoftirqd/121:45:011
4093564994115,22cyclictest26-21ksoftirqd/122:15:271
4093564994115,22cyclictest26-21ksoftirqd/122:15:271
4093560994125,12cyclictest4154778-21sshd22:04:180
162399410,33rtkit-daemon707415-21gmain19:48:472
41895912400,5sleep2331rcuc/223:43:122
41512252402,7sleep31753-21gnome-shell21:53:053
41104172400,5chrt0-21swapper/119:55:221
41021462400,10sleep126-21ksoftirqd/119:30:341
4093570994022,14cyclictest42-21ksoftirqd/322:30:013
4093570994021,14cyclictest42-21ksoftirqd/323:35:143
4093570994021,14cyclictest42-21ksoftirqd/323:35:143
4093570994020,12cyclictest42-21ksoftirqd/319:50:513
4093570994013,23cyclictest4156913-21sort22:10:193
4093570994012,24cyclictest4126859-21cron20:45:013
4093567994013,9cyclictest1-21systemd21:40:002
4093564994023,6cyclictest26-21ksoftirqd/100:20:171
4093564994023,5cyclictest26-21ksoftirqd/120:50:231
4093564994023,5cyclictest26-21ksoftirqd/100:33:041
4093564994022,6cyclictest26-21ksoftirqd/123:45:371
4093564994022,6cyclictest26-21ksoftirqd/121:17:211
4093564994022,6cyclictest26-21ksoftirqd/121:00:451
4093564994022,14cyclictest26-21ksoftirqd/122:20:421
4093564994022,14cyclictest26-21ksoftirqd/122:07:521
409356499401,16cyclictest41002152open_files19:25:281
4093560994023,6cyclictest13-21ksoftirqd/022:40:190
4093560994022,6cyclictest13-21ksoftirqd/000:23:510
4093560994022,5cyclictest13-21ksoftirqd/019:31:410
409356099402,15cyclictest1753-21gnome-shell19:55:000
4093570993922,13cyclictest42-21ksoftirqd/300:30:163
4093570993921,14cyclictest42-21ksoftirqd/319:45:223
4093570993920,14cyclictest42-21ksoftirqd/319:35:253
4093570993915,20cyclictest42-21ksoftirqd/319:40:263
409357099391,34cyclictest1753-21gnome-shell23:12:473
409357099391,14cyclictest41707892apt-config22:50:023
4093567993930,6cyclictest4168719-21unin-run22:45:002
409356799392,15cyclictest1753-21gnome-shell21:32:412
4093567993912,7cyclictest141rcu_preempt19:10:252
4093564993922,6cyclictest26-21ksoftirqd/123:38:341
4093564993922,6cyclictest26-21ksoftirqd/123:38:341
4093564993921,6cyclictest26-21ksoftirqd/121:36:151
4093564993920,6cyclictest26-21ksoftirqd/122:25:241
4093564993917,17cyclictest4124086-21sh20:35:211
4093560993922,12cyclictest438-20systemd-journal23:59:560
4093560993922,12cyclictest4142024-21awk21:25:310
4093560993921,13cyclictest4163427-21sshd22:29:170
409356099391,34cyclictest4130640-21turbostat20:55:000
409356099391,15cyclictest41607992uname22:20:260
162399390,30rtkit-daemon34-21ksoftirqd/219:58:552
4093570993827,7cyclictest1753-21gnome-shell23:54:313
4093570993827,7cyclictest1753-21gnome-shell19:25:493
4093570993826,7cyclictest42-21ksoftirqd/323:03:543
409357099382,32cyclictest1753-21gnome-shell00:24:243
4093570993820,14cyclictest42-21ksoftirqd/319:30:333
4093570993820,13cyclictest4164515-21ntpq22:30:273
4093570993819,14cyclictest42-21ksoftirqd/323:30:193
4093567993822,12cyclictest4109492-21sshd19:54:182
4093564993823,11cyclictest4114760-21/usr/sbin/munin20:10:011
4093564993820,6cyclictest26-21ksoftirqd/123:08:041
4093564993817,7cyclictest4117801-21timerwakeupswit20:15:321
409356099387,5cyclictest141rcu_preempt20:06:000
409356099382,8cyclictest1753-21gnome-shell19:11:260
4093560993825,9cyclictest4099666-21cstates19:25:170
409356099382,32cyclictest1753-21gnome-shell20:28:360
4093560993819,6cyclictest13-21ksoftirqd/022:14:460
4093560993818,16cyclictest4133013-20journal-offline21:00:180
4093570993720,12cyclictest3980-21awk00:05:283
4093570993719,14cyclictest42-21ksoftirqd/323:41:113
4093570993719,14cyclictest42-21ksoftirqd/322:55:203
4093570993719,14cyclictest42-21ksoftirqd/320:45:143
4093570993719,13cyclictest42-21ksoftirqd/320:05:313
4093570993718,7cyclictest42-21ksoftirqd/319:25:013
4093570993718,7cyclictest42-21ksoftirqd/319:25:013
4093570993713,19cyclictest42-21ksoftirqd/319:10:233
409356799374,16cyclictest1753-21gnome-shell21:41:222
4093567993726,8cyclictest1753-21gnome-shell20:41:592
4093567993725,9cyclictest1753-21gnome-shell20:53:532
4093567993721,11cyclictest1198-21date00:00:032
4093567993715,6cyclictest141rcu_preempt19:25:222
409356799371,21cyclictest4163622-21wc22:30:022
409356799371,16cyclictest1069-21runrttasks00:38:262
4093564993721,5cyclictest26-21ksoftirqd/120:40:491
4093564993720,6cyclictest26-21ksoftirqd/100:25:211
409356099372,7cyclictest4192122-21diskstats23:50:180
409356099372,30cyclictest1753-21gnome-shell00:19:430
4093560993722,11cyclictest13-21ksoftirqd/000:00:060
4093560993721,11cyclictest4141002-21cron21:24:590
409356099371,9cyclictest0-21swapper/019:56:370
162399370,25rtkit-daemon0-21swapper/023:01:010
409357099369,5cyclictest141rcu_preempt20:13:343
4093570993619,13cyclictest42-21ksoftirqd/323:07:153
4093567993626,6cyclictest2197014-21PK-Backend00:05:162
409356799361,6cyclictest2197014-21PK-Backend22:59:102
409356799361,31cyclictest8800-21cstates00:20:162
4093567993613,10cyclictest141rcu_preempt20:45:102
409356099363,8cyclictest54-21kcompactd020:04:510
409356099363,28cyclictest1753-21gnome-shell23:21:270
4093560993626,6cyclictest1753-21gnome-shell22:18:400
4093560993626,6cyclictest1753-21gnome-shell22:18:400
4093560993625,7cyclictest1753-21gnome-shell19:22:560
4093560993625,7cyclictest1753-21gnome-shell19:22:560
4093560993621,11cyclictest4156126-21cat22:05:360
409356099362,10cyclictest356-21jbd2/sdb2-821:40:200
4093560993620,12cyclictest884-21gdbus23:30:020
4093560993619,13cyclictest4187720-21sshd23:36:200
4093560993619,13cyclictest4187720-21sshd23:36:190
409356099361,8cyclictest41862052chrt23:34:080
409356099361,29cyclictest41184522sleep020:20:040
162399360,26rtkit-daemon1622-21rtkit-daemon22:13:442
41463012350,7chrt2197014-21PK-Backend21:40:020
4093570993516,14cyclictest42-21ksoftirqd/300:17:093
4093567993522,9cyclictest1753-21gnome-shell20:21:062
409356799352,17cyclictest2197014-21PK-Backend20:03:082
409356799351,9cyclictest2197014-21PK-Backend00:02:332
4093567993519,12cyclictest4164001-21expr22:30:172
4093567993518,14cyclictest1753-21gnome-shell21:51:022
409356799351,13cyclictest4105239-21memory19:40:272
409356799351,12cyclictest2197014-21PK-Backend21:19:522
409356099356,11cyclictest1069-21runrttasks23:44:330
409356099352,8cyclictest1753-21gnome-shell21:50:170
409356099352,30cyclictest0-21swapper/020:57:400
409356099352,28cyclictest1753-21gnome-shell21:59:420
409356099352,10cyclictest1753-21gnome-shell21:16:360
409356099351,9cyclictest2197014-21PK-Backend22:35:120
4093560993519,12cyclictest4143262-21expr21:30:200
4093560993518,12cyclictest438-20systemd-journal23:19:190
4093560993518,12cyclictest4135828-21/usr/sbin/munin21:10:020
4093570993417,13cyclictest42-21ksoftirqd/319:15:073
4093567993424,6cyclictest1753-21gnome-shell22:05:182
4093567993422,9cyclictest40960782sleep219:15:142
4093567993422,9cyclictest1753-21gnome-shell23:05:312
4093567993422,8cyclictest1753-21gnome-shell23:21:462
4093567993416,14cyclictest5773-21which00:10:292
4093567993415,14cyclictest1-21systemd20:20:012
4093567993411,6cyclictest41795582cat23:15:032
4093560993426,5cyclictest0-21swapper/022:49:190
4093560993426,5cyclictest0-21swapper/022:49:190
409356099342,16cyclictest113-1kworker/0:1H19:15:200
409356099342,12cyclictest0-21swapper/019:40:330
4093560993420,10cyclictest10476-21ls00:25:180
4093560993419,11cyclictest4126873-21apt-get20:45:010
4093560993419,11cyclictest4106801-21cat19:45:200
4093560993418,12cyclictest4117000-21expr20:15:150
4093560993417,12cyclictest855-21dbus-daemon20:40:010
4093560993411,19cyclictest4710-21cron00:09:590
162399340,27rtkit-daemon4095276-21munin-node20:37:292
409357099331,28cyclictest2197014-21PK-Backend22:15:143
409357099331,28cyclictest2197014-21PK-Backend22:15:133
409356799332,9cyclictest1753-21gnome-shell22:15:332
409356799332,9cyclictest1753-21gnome-shell22:15:332
4093567993322,7cyclictest1753-21gnome-shell21:11:362
4093567993322,7cyclictest1753-21gnome-shell21:11:362
4093567993321,8cyclictest1753-21gnome-shell23:17:452
4093567993320,8cyclictest4098089-21sh19:20:162
4093567993320,8cyclictest4098089-21sh19:20:152
4093567993320,8cyclictest1753-21gnome-shell23:35:422
4093567993320,8cyclictest1753-21gnome-shell23:35:412
409356799331,7cyclictest2197014-21PK-Backend23:00:172
409356799331,6cyclictest2197014-21PK-Backend20:13:272
4093560993318,11cyclictest4122233-21expr20:30:140
409356799329,19cyclictest1753-21gnome-shell23:26:332
409356799327,10cyclictest141rcu_preempt00:25:172
409356799321,5cyclictest2197014-21PK-Backend21:05:502
162399310,25rtkit-daemon2197014-21PK-Backend19:31:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional