You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 11:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Thu Nov 06, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3324969992521,1cyclictest0-21swapper/300:40:003
3324969992514,1cyclictest0-21swapper/300:30:233
3324969992219,1cyclictest0-21swapper/322:25:003
157898220,8rtkit-daemon3331289-21munin-node23:33:050
332496999191,5cyclictest34387562sleep320:40:153
332496999191,5cyclictest34387562sleep320:40:143
3324969991815,1cyclictest0-21swapper/323:25:233
3324969991714,1cyclictest0-21swapper/321:40:013
3324969991614,1cyclictest0-21swapper/319:55:003
3324969991613,1cyclictest0-21swapper/322:26:403
3324969991613,1cyclictest0-21swapper/320:24:393
3324969991613,1cyclictest0-21swapper/319:22:113
3324969991513,1cyclictest0-21swapper/323:15:133
3324969991512,1cyclictest0-21swapper/323:57:263
3324969991512,1cyclictest0-21swapper/323:55:013
3324969991512,1cyclictest0-21swapper/323:48:403
3324969991512,1cyclictest0-21swapper/323:42:503
3324969991512,1cyclictest0-21swapper/323:07:133
3324969991512,1cyclictest0-21swapper/323:01:373
3324969991512,1cyclictest0-21swapper/323:01:373
3324969991512,1cyclictest0-21swapper/322:40:003
3324969991512,1cyclictest0-21swapper/322:19:453
3324969991512,1cyclictest0-21swapper/322:00:133
3324969991512,1cyclictest0-21swapper/321:52:223
3324969991512,1cyclictest0-21swapper/321:40:083
3324969991512,1cyclictest0-21swapper/321:32:063
3324969991512,1cyclictest0-21swapper/321:00:563
3324969991512,1cyclictest0-21swapper/320:51:123
3324969991512,1cyclictest0-21swapper/320:51:123
3324969991512,1cyclictest0-21swapper/320:45:103
3324969991512,1cyclictest0-21swapper/320:29:243
3324969991512,1cyclictest0-21swapper/320:17:553
3324969991512,1cyclictest0-21swapper/320:03:153
3324969991512,1cyclictest0-21swapper/319:40:173
3324969991512,1cyclictest0-21swapper/319:10:193
3324969991512,1cyclictest0-21swapper/300:17:053
3324969991512,1cyclictest0-21swapper/300:08:573
332496999147,4cyclictest63550irq/135-eno1:tx-300:00:333
332496999147,4cyclictest37058142sleep323:35:143
3324969991412,1cyclictest0-21swapper/320:06:443
3324969991412,1cyclictest0-21swapper/319:45:213
3324969991412,1cyclictest0-21swapper/300:24:543
3324969991411,1cyclictest0-21swapper/323:33:283
3324969991411,1cyclictest0-21swapper/322:46:583
3324969991411,1cyclictest0-21swapper/322:30:293
3324969991411,1cyclictest0-21swapper/322:11:433
3324969991411,1cyclictest0-21swapper/322:08:443
3324969991411,1cyclictest0-21swapper/321:57:293
3324969991411,1cyclictest0-21swapper/320:55:183
3324969991411,1cyclictest0-21swapper/320:37:463
3324969991411,1cyclictest0-21swapper/320:13:243
3324969991411,1cyclictest0-21swapper/319:19:523
3324969991411,1cyclictest0-21swapper/300:12:523
3324969991410,1cyclictest0-21swapper/322:55:003
3324969991311,1cyclictest0-21swapper/322:42:243
3324969991311,1cyclictest0-21swapper/319:28:203
3324969991310,1cyclictest0-21swapper/323:21:043
3324969991310,1cyclictest0-21swapper/321:45:153
3324969991310,1cyclictest0-21swapper/321:20:193
3324969991310,1cyclictest0-21swapper/321:10:303
3324969991310,1cyclictest0-21swapper/319:55:143
3324969991310,1cyclictest0-21swapper/300:25:353
157898130,4rtkit-daemon3331289-21munin-node23:14:071
36557022120,4sleep03655769-21ntp_states23:00:210
36557022120,4sleep03655769-21ntp_states23:00:210
332496999129,1cyclictest0-21swapper/321:26:593
332496999121,8cyclictest3432133-21turbostat20:35:003
3324969991210,1cyclictest0-21swapper/321:06:223
157898120,8rtkit-daemon3331289-21munin-node00:32:520
157898120,3rtkit-daemon724-21snmpd19:39:290
36109552111,7sleep03610988-21apt-get22:35:120
35694182110,4sleep03569426-21ssh22:07:070
332496999119,1cyclictest0-21swapper/323:10:043
332496999119,1cyclictest0-21swapper/319:35:173
332496999118,1cyclictest0-21swapper/322:55:153
332496999118,1cyclictest0-21swapper/321:15:593
332496999114,5cyclictest33564812sleep319:30:193
157898110,5rtkit-daemon3498891-21ssh21:23:571
157898110,3rtkit-daemon724-21snmpd22:03:301
38041552100,3sleep03806015-21ssh00:35:180
37363022100,3sleep03736310-21ssh23:53:260
35710162100,7sleep23571055-21ssh22:10:142
34696052100,2sleep00-21swapper/021:05:160
34264792100,2sleep20-21swapper/220:30:152
34015612100,3sleep03401670-21tr20:10:130
33954182101,6sleep03395538-21grep20:05:140
157898100,6rtkit-daemon331ktimers/123:15:111
157898100,5rtkit-daemon3564433-21irqrtprio22:05:171
157898100,5rtkit-daemon1610-21sendmail-mta19:25:030
157898100,4rtkit-daemon3376522-21apt-get19:50:010
157898100,3rtkit-daemon729-21lldpd19:41:540
157898100,3rtkit-daemon727-21ntpd19:53:551
157898100,3rtkit-daemon3331289-21munin-node19:34:410
3769416291,3sleep03769448-21apt-get00:15:150
3766577291,6sleep03766641-21ntp_states00:10:220
3642326291,5sleep03642365-21awk22:55:000
3531462290,7sleep03531621-21grep21:45:150
3319041291,4sleep10-21swapper/119:05:121
15789890,5rtkit-daemon899-21cron22:55:001
15789890,5rtkit-daemon724-21snmpd23:54:021
15789890,5rtkit-daemon3760852-21sh00:09:561
15789890,5rtkit-daemon1610-21sendmail-mta23:04:391
15789890,5rtkit-daemon1610-21sendmail-mta23:04:391
15789890,4rtkit-daemon3696801-21runrttasks23:27:340
15789890,4rtkit-daemon3630678-21cut22:45:221
15789890,4rtkit-daemon3595151-21ssh22:25:131
15789890,4rtkit-daemon3333541-21diskstats19:15:030
15789890,3rtkit-daemon3463110-21seq20:58:331
15789890,1rtkit-daemon0-21swapper/121:28:191
3800159280,1sleep20-21swapper/200:33:282
3753467280,5sleep03753508-21fschecks_count00:05:130
3745304281,3sleep03745338-21apt-get00:00:120
3712389280,1sleep20-21swapper/223:38:102
3666551281,4sleep03666706-21df_inode23:10:150
3603066280,2sleep03603080-21grep22:30:120
3541788280,6sleep23543867-21tr21:50:172
3445174281,4sleep13445323-21df_inode20:45:131
3414041281,3sleep03414063-21apt-get20:20:120
3406701280,4sleep23406896-21/usr/sbin/munin20:10:242
15789880,5rtkit-daemon0-21swapper/019:23:290
15789880,4rtkit-daemon98157-21packagekitd21:04:021
15789880,4rtkit-daemon899-21cron22:44:011
15789880,4rtkit-daemon899-21cron20:45:001
15789880,4rtkit-daemon899-21cron20:45:001
15789880,4rtkit-daemon729-21lldpd21:45:091
15789880,4rtkit-daemon727-21ntpd23:43:531
15789880,4rtkit-daemon727-21ntpd23:22:280
15789880,4rtkit-daemon727-21ntpd22:30:411
15789880,4rtkit-daemon724-21snmpd23:45:121
15789880,4rtkit-daemon724-21snmpd21:58:320
15789880,4rtkit-daemon724-21snmpd21:18:261
15789880,4rtkit-daemon724-21snmpd20:15:481
15789880,4rtkit-daemon724-21snmpd20:10:331
15789880,4rtkit-daemon724-21snmpd19:57:241
15789880,4rtkit-daemon724-21snmpd00:21:201
15789880,4rtkit-daemon724-21snmpd00:18:111
15789880,4rtkit-daemon387-21systemd-udevd19:06:060
15789880,4rtkit-daemon3744356-21sh23:58:391
15789880,4rtkit-daemon3649723-21diskmemload22:58:401
15789880,4rtkit-daemon3475474-21diskmemload22:13:531
15789880,4rtkit-daemon3475474-21diskmemload00:26:161
15789880,4rtkit-daemon3376414-21taskset19:48:550
15789880,4rtkit-daemon3331289-21munin-node23:40:540
15789880,4rtkit-daemon3331289-21munin-node22:39:201
15789880,4rtkit-daemon3331289-21munin-node22:16:581
15789880,4rtkit-daemon3331289-21munin-node21:11:001
15789880,4rtkit-daemon3331289-21munin-node20:09:121
15789880,4rtkit-daemon3331289-21munin-node20:02:131
15789880,4rtkit-daemon1610-21sendmail-mta21:31:091
15789880,4rtkit-daemon1610-21sendmail-mta20:37:341
15789880,3rtkit-daemon505-21wpa_supplicant22:21:181
15789880,3rtkit-daemon505-21wpa_supplicant20:50:231
15789880,3rtkit-daemon505-21wpa_supplicant20:50:231
15789880,3rtkit-daemon3706210-21smartctl23:35:170
15789880,3rtkit-daemon3527804-21kthreadcore21:40:181
15789880,1rtkit-daemon0-21swapper/123:08:581
15789880,1rtkit-daemon0-21swapper/019:11:230
3791269270,5sleep03791277-21ssh00:26:130
3767195270,4sleep23767203-21ssh00:10:312
3760780271,4sleep23760788-21ssh00:09:482
3745508271,3sleep23745634-21grep00:00:142
3688489270,4sleep23688491-21ssh23:21:432
3682316271,2sleep13682355-21apt-get23:20:101
3673933270,1sleep20-21swapper/223:14:382
3664739270,1sleep00-21swapper/023:06:410
3634402270,5sleep03634410-21ssh22:49:590
3626499271,3sleep23626550-21munin-run22:45:002
3587097270,4sleep00-21swapper/022:20:020
3585105270,1sleep00-21swapper/022:16:320
3571218270,4sleep03571358-21tail22:10:160
3529359270,4sleep03529367-21ssh21:41:180
3513786270,1sleep00-21swapper/021:32:040
3489744270,1sleep00-21swapper/021:16:140
3482026270,1sleep00-21swapper/021:11:490
3463501271,2sleep20-21swapper/221:00:132
3445138270,3sleep03445171-21apt-get20:45:120
3375974270,5sleep23376137-21/usr/sbin/munin19:45:232
3370356271,3sleep23370399-21munin-run19:45:002
15789870,3rtkit-daemon727-21ntpd22:00:100
15789870,3rtkit-daemon727-21ntpd20:34:361
15789870,3rtkit-daemon727-21ntpd20:25:161
15789870,3rtkit-daemon727-21ntpd00:10:161
15789870,3rtkit-daemon724-21snmpd21:55:391
15789870,3rtkit-daemon724-21snmpd21:38:461
15789870,3rtkit-daemon724-21snmpd21:06:011
15789870,3rtkit-daemon3539096-21ssh21:50:021
15789870,3rtkit-daemon1610-21sendmail-mta20:23:181
15789870,3rtkit-daemon1610-21sendmail-mta00:01:381
3783387260,4sleep20-21swapper/200:21:222
3783247260,5sleep03783254-21sh00:21:040
3743037260,1sleep00-21swapper/023:55:350
3737557260,3sleep23737681-21grep23:55:172
3727563260,1sleep00-21swapper/023:46:300
3699496261,3sleep23700538-21ssh23:30:172
3696836261,3sleep13696842-21ssh23:27:391
3657529260,1sleep20-21swapper/223:03:242
3657529260,1sleep20-21swapper/223:03:242
3632954260,4sleep23632962-21ssh22:46:322
3618942261,2sleep00-21swapper/022:40:130
3595524261,2sleep20-21swapper/222:25:172
3595155260,1sleep00-21swapper/022:25:140
3561685260,1sleep20-21swapper/222:02:322
3545456260,1sleep00-21swapper/021:51:560
3521147260,1sleep00-21swapper/021:35:450
3506342260,1sleep00-21swapper/021:28:040
3498619260,3sleep03498627-21ssh21:23:230
3492691260,4sleep23493532-21cut21:20:172
3456937260,1sleep00-21swapper/020:52:540
3456937260,1sleep00-21swapper/020:52:530
3440196260,3sleep03441363-21grep20:40:200
3440196260,3sleep03441363-21grep20:40:200
3434966260,4sleep03437075-21kthreadcore20:35:190
3427439260,4sleep00-21swapper/020:30:180
3425759260,1sleep00-21swapper/020:27:010
3413279260,1sleep00-21swapper/020:17:200
3394933260,1sleep00-21swapper/020:02:330
3389760261,3sleep23390728-21kernelversion20:00:192
3388832260,1sleep00-21swapper/019:57:480
3369470260,3sleep13369533-21awk19:40:181
3357272260,3sleep23357397-21python319:30:222
3353283260,4sleep13354402-21kthreadcore19:30:171
3331141260,3sleep13332680-21cut19:15:021
33249699964,1cyclictest0-21swapper/319:10:013
3808160251,2sleep10-21swapper/100:38:401
3806201250,3sleep23806272-21ntp_states00:35:212
3797046250,3sleep13798132-21memory00:30:191
3713488250,2sleep20-21swapper/223:40:112
3697511250,3sleep20-21swapper/223:29:122
3680190250,3sleep00-21swapper/023:15:520
3660135251,2sleep20-21swapper/223:05:202
3643975250,3sleep03645080-21cut22:55:200
3585723250,3sleep23585726-21sh22:18:032
3563077250,2sleep20-21swapper/222:05:132
3554086250,3sleep23554094-21ssh21:58:322
3531257250,2sleep20-21swapper/221:45:132
3507102250,3sleep2724-21snmpd21:29:502
3491353250,3sleep21-21systemd21:20:002
3465504250,3sleep03467570-21grep21:00:190
3462923250,1sleep00-21swapper/020:56:100
3432635250,2sleep20-21swapper/220:35:132
3420155250,2sleep20-21swapper/220:25:152
3399819250,3sleep23400330-21head20:05:202
3385738250,3sleep23387864-21cut19:55:182
3376148250,1sleep10-21swapper/119:45:311
3363957250,1sleep10-21swapper/119:36:451
3363558250,3sleep23363829-21awk19:35:222
3346281250,2sleep20-21swapper/219:25:172
3345481250,1sleep20-21swapper/219:23:042
3345014250,2sleep10-21swapper/119:20:231
3339325250,3sleep23339326-21sshd19:17:552
3326413250,3sleep13326615-21kthreadcore19:10:171
3320483250,2sleep20-21swapper/219:05:172
3785313240,2sleep20-21swapper/200:25:152
3775142240,1sleep20-21swapper/200:15:412
3730943240,2sleep20-21swapper/223:50:202
3721401240,1sleep20-21swapper/223:45:072
3698168240,1sleep10-21swapper/123:30:121
3675890240,1sleep20-21swapper/223:15:182
3649614240,2sleep23649622-21ssh22:58:272
3616720240,2sleep23616728-21ssh22:35:422
3608974240,2sleep23608982-21ssh22:31:102
3587115240,2sleep20-21swapper/222:20:042
3523723240,2sleep20-21swapper/221:40:162
3515381241,2sleep2431ktimers/221:35:112
3482857240,2sleep20-21swapper/221:13:432
3475246240,1sleep20-21swapper/221:07:132
3457358240,1sleep20-21swapper/220:55:122
3450741240,3sleep20-21swapper/220:46:592
3439008240,2sleep20-21swapper/220:40:182
3439008240,2sleep20-21swapper/220:40:182
3409119240,2sleep23410953-21grep20:15:172
3379487240,2sleep20-21swapper/219:50:212
3327579240,2sleep23329709-21kthreadcore19:10:182
3705801230,1sleep10-21swapper/123:35:051
3634459230,1sleep20-21swapper/222:50:012
3507437230,1sleep20-21swapper/221:30:022
3451422230,1sleep20-21swapper/220:50:152
3451422230,1sleep20-21swapper/220:50:152
3414223230,1sleep20-21swapper/220:20:142
3349342230,1sleep10-21swapper/119:25:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional