You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 04:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Thu Nov 20, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
35654649911691,1164cyclictest39924622sleep300:29:563
3565464992900,287cyclictest3571383-21turbostat19:44:563
3565464992720,269cyclictest3926852-21turbostat23:54:553
3565464992320,1cyclictest0-21swapper/323:00:003
3565464992219,1cyclictest0-21swapper/300:35:183
3565464992017,1cyclictest0-21swapper/322:10:003
38681292190,6sleep03868137-21ssh23:10:570
356546499186,9cyclictest3638206-21taskset20:45:003
3565464991816,1cyclictest0-21swapper/321:15:153
356546499175,8cyclictest3805023-21turbostat22:35:003
3565464991714,1cyclictest0-21swapper/321:50:013
3565464991714,1cyclictest0-21swapper/320:35:003
40045852160,2sleep03998645-21kworker/0:1+events00:35:190
356546499168,5cyclictest3656861-21turbostat21:00:003
3565464991613,1cyclictest0-21swapper/300:06:523
356546499161,11cyclictest3796951-21turbostat22:30:003
356546499151,9cyclictest36633672sleep321:05:133
356546499151,9cyclictest36633672sleep321:05:123
3565464991513,1cyclictest0-21swapper/320:09:083
3565464991512,1cyclictest0-21swapper/322:35:193
3565464991512,1cyclictest0-21swapper/321:25:013
3565464991512,1cyclictest0-21swapper/320:50:193
3565464991512,1cyclictest0-21swapper/319:55:323
3565464991512,1cyclictest0-21swapper/319:46:103
356546499151,11cyclictest3740291-21taskset21:50:243
356546499151,11cyclictest3566142-21turbostat19:15:013
356546499151,10cyclictest3984399-21turbostat00:25:013
190898150,2rtkit-daemon0-21swapper/121:48:461
3565464991413,0cyclictest0-21swapper/323:49:593
3565464991412,1cyclictest0-21swapper/323:07:133
3565464991412,1cyclictest0-21swapper/321:15:013
3565464991412,1cyclictest0-21swapper/320:12:003
3565464991412,1cyclictest0-21swapper/319:21:273
356546499141,11cyclictest3718417-21taskset21:40:013
3565464991411,1cyclictest0-21swapper/322:50:133
3565464991411,1cyclictest0-21swapper/322:03:453
356546499141,10cyclictest3961999-21taskset00:10:173
356546499141,10cyclictest3935797-21taskset23:55:153
356546499141,10cyclictest3903622-21taskset23:35:143
356546499141,10cyclictest3877140-21taskset23:17:563
356546499141,10cyclictest3821819-21chrt22:45:133
356546499141,10cyclictest3819501-21chrt22:40:533
356546499141,10cyclictest3745108-21taskset21:55:163
356546499141,10cyclictest3700538-21taskset21:26:483
356546499141,10cyclictest3662894-21taskset21:02:123
356546499141,10cyclictest3586201-21taskset20:00:363
356546499141,10cyclictest3571037-21taskset19:35:523
356546499141,10cyclictest3569964-21chrt19:28:083
190898140,3rtkit-daemon667-21snmpd19:38:362
38381092131,9sleep03838252-21sort22:55:180
356546499134,7cyclictest3943574-21taskset00:00:103
356546499134,7cyclictest3717577-21taskset21:38:423
356546499134,7cyclictest3644047-21taskset20:46:533
356546499131,9cyclictest39108152sleep323:40:093
356546499131,9cyclictest3900489-21taskset23:30:343
356546499131,9cyclictest3610282-21taskset20:20:213
356546499131,9cyclictest3570360-21chrt19:30:083
3565464991312,0cyclictest0-21swapper/323:21:223
3565464991312,0cyclictest0-21swapper/319:17:313
356546499131,10cyclictest3631397-21chrt20:39:003
356546499131,10cyclictest3612579-21chrt20:25:013
3565464991311,0cyclictest0-21swapper/323:26:243
3565464991311,0cyclictest0-21swapper/323:26:233
3565464991310,2cyclictest0-21swapper/322:20:133
3565464991310,1cyclictest0-21swapper/320:18:043
3565464991310,1cyclictest0-21swapper/319:52:523
3565464991310,1cyclictest0-21swapper/300:32:383
356546499130,10cyclictest3974523-21chrt00:16:053
356546499130,10cyclictest3851775-21taskset23:00:443
190898130,9rtkit-daemon3717696-21ssh21:38:582
356546499120,9cyclictest3862106-21taskset23:10:103
356546499120,9cyclictest3778414-21taskset22:15:253
356546499120,9cyclictest3766430-21chrt22:10:123
356546499120,9cyclictest3702191-21taskset21:30:053
35650532120,3sleep30-21swapper/319:08:143
35592782121,5sleep13559300-21apt-get19:05:151
39518632112,2sleep1171rcu_preempt00:05:151
38193812110,1sleep00-21swapper/022:40:390
37932922111,8sleep13794205-21missed_timers22:25:171
36397202110,3sleep23641045-21kthreadcore20:45:172
36120752111,6sleep13612268-21cat20:25:011
190898110,6rtkit-daemon3870263-21/usr/sbin/munin23:15:120
190898110,6rtkit-daemon3637808-21apt-config20:44:592
190898110,5rtkit-daemon3827729-21ssh22:46:170
190898110,4rtkit-daemon3566337-21munin-node21:06:562
190898110,4rtkit-daemon3566337-21munin-node21:06:562
39746712100,7sleep23974679-21ssh00:16:252
39602562101,5sleep03960459-21hddtemp_smartct00:10:150
38602532101,6sleep13860261-21ssh23:06:371
38055352101,6sleep03805617-21fschecks_time22:35:140
37415592100,7sleep23741565-21ssh21:53:132
36327292100,7sleep03633616-21cut20:40:180
190898100,6rtkit-daemon3977837-21cut00:20:160
190898100,5rtkit-daemon3794630-21sed22:25:200
190898100,5rtkit-daemon3630639-21ntpq20:35:212
190898100,4rtkit-daemon3684248-21ssh21:16:572
3992488290,7sleep23992498-21cron00:30:012
3934748290,6sleep03934756-21ssh23:54:030
3919020291,5sleep03919182-21diskstats23:45:190
3779092290,6sleep13779100-21ssh22:16:561
3766125291,4sleep03766158-21apt-get22:10:090
3758490291,5sleep03758504-21munin-run22:05:010
3676844291,4sleep03676852-21ssh21:13:280
3675833290,1sleep10-21swapper/121:11:121
3622678291,5sleep23623883-21memory20:30:222
19089890,5rtkit-daemon3836289-21ssh22:52:230
19089890,5rtkit-daemon3812240-21sh22:37:572
19089890,5rtkit-daemon3676163-21ssh21:11:542
19089890,4rtkit-daemon3893190-21sh23:27:182
19089890,4rtkit-daemon3893190-21sh23:27:182
19089890,4rtkit-daemon3860680-21ssh23:07:280
19089890,4rtkit-daemon3787181-21ssh22:21:530
19089890,4rtkit-daemon3744364-21cut21:55:161
19089890,4rtkit-daemon3595853-21grep20:10:202
19089890,3rtkit-daemon3852362-21ssh23:02:070
19089890,1rtkit-daemon0-21swapper/123:21:151
4006878281,4sleep24006886-21ssh00:36:022
3993610280,5sleep23994553-21cat00:30:212
3965655280,5sleep13965838-21vmstat00:10:241
3935673281,4sleep23935796-21perl23:55:152
3935517281,4sleep13935672-21df_inode23:55:131
3909548280,1sleep00-21swapper/023:37:530
3856467280,6sleep23858566-21cut23:05:182
3851040281,4sleep23851102-21ntp_states23:00:202
3826885281,4sleep23827017-21ssh22:45:222
3813125281,5sleep13813139-21cron22:39:591
3780808280,5sleep13780915-21cpuspeed_turbos22:20:121
3773079280,6sleep23773660-21sh22:15:172
3733526281,4sleep03733534-21ssh21:48:170
3701679280,1sleep00-21swapper/021:29:300
3638267281,3sleep13638300-21apt-get20:45:121
3625472280,4sleep13625626-21diskstats20:35:141
3574746280,4sleep13574768-21apt-get19:55:081
19089880,5rtkit-daemon432-21rsyslogd23:31:082
19089880,5rtkit-daemon1928-21sendmail-mta20:02:282
19089880,4rtkit-daemon81364-21packagekitd19:24:322
19089880,4rtkit-daemon674-21lldpd00:29:050
19089880,4rtkit-daemon667-21snmpd23:56:300
19089880,4rtkit-daemon438-21wpa_supplicant19:34:172
19089880,4rtkit-daemon407-21rpcbind19:55:082
19089880,4rtkit-daemon3916823-21taskset23:40:502
19089880,4rtkit-daemon3909634-21ssh23:38:022
19089880,4rtkit-daemon3797208-21ssh22:30:040
19089880,4rtkit-daemon3765472-21ssh22:07:232
19089880,4rtkit-daemon3589223-21kthreadcore20:05:172
19089880,4rtkit-daemon3566337-21munin-node23:42:500
19089880,4rtkit-daemon3566337-21munin-node21:51:541
19089880,4rtkit-daemon3566337-21munin-node21:02:302
19089880,4rtkit-daemon3566337-21munin-node20:53:402
19089880,3rtkit-daemon3973760-21ntp_states00:15:220
19089880,1rtkit-daemon0-21swapper/223:49:562
19089880,1rtkit-daemon0-21swapper/220:26:032
19089880,1rtkit-daemon0-21swapper/219:29:402
4001309271,3sleep14001614-21cut00:35:171
3958364271,3sleep03958380-21unattended-upgr00:06:540
3943772270,4sleep13943891-21fw_packets00:00:141
3888193270,5sleep13890147-21kthreadcore23:25:201
3888193270,5sleep13890147-21kthreadcore23:25:191
3887221270,4sleep03888192-21grep23:25:190
3887221270,4sleep03888192-21grep23:25:180
3884715270,1sleep00-21swapper/023:21:300
3872738271,3sleep13874831-21tr23:15:171
3843008270,5sleep23843086-21ssh22:55:242
3829030270,4sleep13829038-21ssh22:49:181
3819938270,4sleep13819946-21ssh22:41:561
3819553270,1sleep23819561-21ssh22:41:012
3802594271,3sleep244-21ksoftirqd/222:30:232
3786996270,4sleep23787004-21ssh22:21:212
3755814271,3sleep23755961-21ntp_kernel_pll_22:00:222
3724302270,1sleep10-21swapper/121:40:291
3717402270,5sleep10-21swapper/121:38:211
3710328270,3sleep03710361-21apt-get21:35:110
3694190272,2sleep13694229-21apt-get21:25:151
3692372273,2sleep1331ktimers/121:21:301
3684774270,4sleep03684782-21ssh21:18:120
3668548271,4sleep03668638-21readlink21:05:220
3668548271,4sleep03668638-21readlink21:05:210
3662295271,4sleep13662420-21python321:00:221
3654446271,3sleep03655799-21memory20:55:160
3587666270,4sleep03588659-21irqstats20:05:150
19089870,4rtkit-daemon81364-21packagekitd22:10:232
19089870,4rtkit-daemon671-21ntpd23:39:381
19089870,4rtkit-daemon671-21ntpd22:01:001
19089870,4rtkit-daemon667-21snmpd19:50:252
19089870,4rtkit-daemon667-21snmpd19:45:372
19089870,4rtkit-daemon667-21snmpd00:00:540
19089870,4rtkit-daemon473-21gmain21:25:092
19089870,4rtkit-daemon3566337-21munin-node23:24:462
19089870,4rtkit-daemon3566337-21munin-node21:32:442
19089870,4rtkit-daemon3566337-21munin-node21:20:062
19089870,4rtkit-daemon299-21systemd-journal19:06:382
19089870,4rtkit-daemon1928-21sendmail-mta19:40:192
19089870,3rtkit-daemon667-21snmpd23:40:151
19089870,3rtkit-daemon667-21snmpd19:18:152
19089870,3rtkit-daemon421-21avahi-daemon19:10:172
19089870,3rtkit-daemon3999442-21ssh00:32:240
19089870,3rtkit-daemon3933604-21sh23:51:242
19089870,3rtkit-daemon3669396-21diskmemload22:18:220
19089870,3rtkit-daemon3606093-21apt-get20:20:152
3998939260,1sleep10-21swapper/100:31:151
3984666260,4sleep10-21swapper/100:25:071
3983703260,1sleep20-21swapper/200:23:282
3976070260,1sleep10-21swapper/100:19:371
3958443260,1sleep20-21swapper/200:07:012
3949561260,1sleep20-21swapper/200:00:512
3927409260,4sleep10-21swapper/123:50:161
3925848260,1sleep10-21swapper/123:47:521
3868500260,1sleep10-21swapper/123:11:421
3862097260,4sleep20-21swapper/223:10:102
3846149260,1sleep10-21swapper/123:00:161
3830115261,2sleep20-21swapper/222:50:152
3794208261,3sleep23794369-21ntp_kernel_pll_22:25:182
3771304260,3sleep10-21swapper/122:10:181
3759779260,3sleep03760705-21grep22:05:170
3734665260,3sleep03734821-21df_inode21:50:140
3732511260,1sleep20-21swapper/221:45:552
3718522261,2sleep00-21swapper/021:40:120
3718408260,1sleep20-21swapper/221:40:012
3702990260,3sleep13703929-21cat21:30:151
3686120260,3sleep03686122-21ssh21:20:050
3679640261,2sleep10-21swapper/121:15:161
3645850260,3sleep03647009-21cut20:50:200
3618637261,3sleep01-21systemd20:30:000
3606464260,4sleep03607045-21sed20:20:190
3573178260,3sleep03573181-21apt-get19:50:210
3568568260,1sleep00-21swapper/019:16:240
19089860,3rtkit-daemon671-21ntpd21:31:130
19089860,2rtkit-daemon3654428-21cut20:55:152
19089860,2rtkit-daemon3600231-21/usr/sbin/munin20:15:162
3982376250,3sleep10-21swapper/100:20:261
3966362250,3sleep20-21swapper/200:11:332
3837741250,2sleep10-21swapper/122:55:151
3747209250,2sleep03747762-21needreboot21:55:190
3743940250,3sleep23745107-21cut21:55:162
3669312250,1sleep10-21swapper/121:09:041
3669312250,1sleep10-21swapper/121:09:041
3658096250,3sleep03659121-21cut21:00:170
3644106251,2sleep00-21swapper/020:47:390
3633618250,3sleep13635702-21tr20:40:191
3625464251,2sleep00-21swapper/020:35:130
3624661250,1sleep00-21swapper/020:31:440
3617604250,2sleep10-21swapper/120:25:211
3599757250,2sleep00-21swapper/020:15:140
3599164250,1sleep10-21swapper/120:10:421
3588660251,2sleep10-21swapper/120:05:171
3586658250,1sleep10-21swapper/120:01:171
3573353250,1sleep10-21swapper/119:50:421
3572852250,1sleep10-21swapper/119:49:391
3571914251,2sleep1667-21snmpd19:43:581
3571351250,3sleep13571415-21systemd-run19:40:001
3570353250,1sleep10-21swapper/119:30:031
3569148250,1sleep10-21swapper/119:20:341
3568784250,1sleep10-21swapper/119:19:141
3565848250,1sleep00-21swapper/019:11:180
3565765250,1sleep10-21swapper/119:10:151
3559639250,3sleep03559767-21irqcore19:05:180
3901350240,3sleep00-21swapper/023:32:280
3833538240,1sleep10-21swapper/122:50:171
3802506240,2sleep13669396-21diskmemload22:30:221
3760706240,2sleep13762733-21cut22:05:181
3656576240,3sleep10-21swapper/120:56:371
3647010240,1sleep10-21swapper/120:50:211
3624844240,2sleep10-21swapper/120:34:051
3601212240,2sleep10-21swapper/120:15:181
3586677240,3sleep00-21swapper/020:01:320
3575101240,1sleep00-21swapper/019:55:110
3571350240,2sleep00-21swapper/019:40:000
3570424240,3sleep00-21swapper/019:30:590
3569836240,3sleep00-21swapper/019:26:250
3894355230,1sleep10-21swapper/123:30:011
3870177230,1sleep20-21swapper/223:15:062
3593248230,1sleep00-21swapper/020:10:010
3571833230,1sleep00-21swapper/019:42:530
3570022230,1sleep10-21swapper/119:28:551
3572502220,1sleep00-21swapper/019:45:050
3569112220,1sleep00-21swapper/019:20:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional