You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 12:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Nov 18, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
688222780,6sleep121-21rcuc/119:05:261
72442242165,24sleep00-21swapper/019:06:590
73992236162,25sleep20-21swapper/219:08:582
71892236162,23sleep30-21swapper/319:06:193
7576995955,3cyclictest0-21swapper/221:05:132
7576995652,3cyclictest0-21swapper/223:30:102
69322560,3sleep2757699cyclictest00:00:142
210952520,3sleep10-21swapper/120:50:221
203912510,2sleep20-21swapper/220:50:152
34292480,2sleep00-21swapper/021:25:160
757299411,32cyclictest28639-21cat21:09:561
757299411,27cyclictest17646-21latency_hist20:45:011
757299392,4cyclictest848-21sssd_nss20:05:201
757299391,3cyclictest13073-21/usr/sbin/munin21:45:231
757299389,26cyclictest9262-21pluginstate22:50:241
757299381,25cyclictest768-21in:imjournal23:10:021
757299381,25cyclictest768-21in:imjournal23:10:021
7564993813,12cyclictest16094-21munin-run21:55:010
757299378,13cyclictest11689-21latency_hist21:45:011
757299374,3cyclictest29310-21/usr/sbin/munin19:55:251
757299374,3cyclictest29310-21/usr/sbin/munin19:55:251
757299371,29cyclictest0-21swapper/122:30:341
757299371,24cyclictest18516-21/usr/sbin/munin22:00:181
757299370,30cyclictest0-21swapper/122:27:141
757299365,3cyclictest6355-21latency_hist00:00:021
756499369,12cyclictest9755-21perf19:15:020
757299356,3cyclictest26945-21processes23:30:241
757299354,3cyclictest5951-21gltestperf21:30:171
757299351,6cyclictest32350-21/usr/sbin/munin23:45:221
757299351,6cyclictest23695-21sshd00:36:351
757299351,4cyclictest8080-21grep00:00:261
757299341,8cyclictest13219-21/usr/sbin/munin00:15:201
757299341,5cyclictest3760-21/usr/sbin/munin22:40:251
757299341,20cyclictest8642-21latency19:10:201
757299332,3cyclictest836-21runrttasks20:38:261
757299331,4cyclictest7599-21taskset22:45:561
757299331,3cyclictest9251-21cpuspeed_turbos20:25:151
757299331,28cyclictest3122-21sed22:35:291
7572993312,18cyclictest15474-21grep19:25:151
7572993310,20cyclictest8809-21/usr/sbin/munin00:05:211
757299330,21cyclictest0-21swapper/122:05:221
757299330,21cyclictest0-21swapper/122:05:221
757299329,20cyclictest15417-21/usr/sbin/munin00:20:181
757299327,2cyclictest0-21swapper/119:54:451
757299321,3cyclictest17146-21cstates23:10:141
757299321,3cyclictest16661-21df_inode21:55:161
757299321,3cyclictest11865-21/usr/sbin/munin00:10:171
757299321,3cyclictest11865-21/usr/sbin/munin00:10:171
757299321,2cyclictest1701-21/usr/sbin/munin21:20:191
7572993212,17cyclictest13254-21latency_hist20:35:021
7572993211,18cyclictest13805-21sendmail_mailtr23:00:251
757299311,28cyclictest0-21swapper/120:05:021
757299311,26cyclictest23970-21ntp_kernel_pll_22:10:221
7572993112,16cyclictest5089-21latency_hist21:30:021
757299311,20cyclictest14838-21awk19:25:021
7572993111,17cyclictest10826-21cstates19:15:131
7572993110,18cyclictest836-21runrttasks21:11:581
7572993110,18cyclictest836-21runrttasks20:58:571
756499318,11cyclictest2892-21idleruntime-cro21:25:010
102032310,2sleep010123-21latency_hist22:55:010
757299309,18cyclictest836-21runrttasks00:34:051
757299309,17cyclictest23557-21/usr/sbin/munin23:25:271
757299306,2cyclictest0-21swapper/119:45:411
757299302,5cyclictest3564-21ntp_states20:10:241
757299301,4cyclictest2194-21/usr/sbin/munin23:50:211
757299301,3cyclictest581-21systemd-journal21:55:011
757299301,3cyclictest20554-21/usr/sbin/munin19:35:241
757299301,26cyclictest30046-21cat23:40:021
757299301,25cyclictest19846-21latency_hist20:50:021
7572993011,16cyclictest20718-21unixbench_singl23:15:291
7572993010,17cyclictest836-21runrttasks20:18:591
757299297,2cyclictest7522-21/usr/sbin/munin21:35:271
757299295,3cyclictest22263-21head19:40:191
757299291,4cyclictest30499-21df_inode23:40:181
757299291,4cyclictest27339-21/usr/sbin/munin22:20:121
757299291,4cyclictest21710-21/usr/sbin/munin23:20:161
757299291,4cyclictest18404-21proc_pri19:30:241
757299291,4cyclictest11943-21users22:55:291
757299291,3cyclictest25958-21uname22:15:241
757299291,25cyclictest31328-21cstates21:15:131
7572992911,16cyclictest17719-21expr00:25:141
756499296,11cyclictest12643-21awk19:20:020
757299288,16cyclictest24843-21fschecks_count21:00:171
757299288,16cyclictest24843-21fschecks_count21:00:171
757299281,3cyclictest7931-21/usr/sbin/munin20:20:221
756499285,11cyclictest3537-21munin-run22:40:020
756499280,5cyclictest1542-21kworker/u8:022:49:520
756499280,5cyclictest11591-21kworker/u8:022:09:570
756499280,5cyclictest11591-21kworker/u8:022:09:570
758299271,12cyclictest0-21swapper/320:17:413
7576992724,2cyclictest0-21swapper/222:35:382
756499273,11cyclictest12341-21idleruntime-cro23:00:010
756499273,11cyclictest11709-21basename21:45:000
756499271,13cyclictest15455-21munin-run20:40:000
758299261,5cyclictest0-21swapper/300:00:153
758299260,12cyclictest0-21swapper/322:10:213
758299251,7cyclictest0-21swapper/323:11:173
7564992515,6cyclictest0-21swapper/023:25:270
756499251,21cyclictest25130-21/usr/sbin/munin22:15:200
756499251,21cyclictest15686-21/usr/sbin/munin20:40:150
756499251,11cyclictest814-21sshd00:11:360
756499251,11cyclictest814-21sshd00:11:360
97272241,19sleep29781-21munin-node19:15:012
758299241,5cyclictest0-21swapper/321:45:153
757699241,16cyclictest0-21swapper/222:01:272
756499242,11cyclictest28662-21diskmemload23:19:180
259462240,4sleep339-21rcuc/323:30:123
758299239,8cyclictest0-21swapper/320:55:213
757699236,9cyclictest0-21swapper/200:10:232
757699236,9cyclictest0-21swapper/200:10:232
757699231,5cyclictest0-21swapper/221:33:012
756499231,4cyclictest28662-21diskmemload23:30:220
758299221,7cyclictest0-21swapper/322:50:253
758299221,12cyclictest0-21swapper/300:36:183
757699222,11cyclictest29835-21users19:55:312
757699222,11cyclictest29835-21users19:55:312
756499228,3cyclictest14767-21/usr/sbin/munin23:05:260
756499228,3cyclictest14767-21/usr/sbin/munin23:05:260
756499228,3cyclictest12862-21/usr/sbin/munin19:20:220
756499226,6cyclictest0-21swapper/021:15:580
756499221,9cyclictest0-21swapper/020:14:410
756499221,6cyclictest0-21swapper/021:45:230
756499221,12cyclictest0-21swapper/023:57:270
756499221,11cyclictest0-21swapper/022:20:210
756499221,10cyclictest28662-21diskmemload23:20:450
756499221,10cyclictest16320-21/usr/sbin/munin21:55:190
205612220,4sleep3758299cyclictest23:15:273
758299214,4cyclictest28662-21diskmemload21:40:243
758299213,10cyclictest15417-21/usr/sbin/munin00:20:183
758299211,8cyclictest28025-21perf19:55:003
758299211,7cyclictest29241-21diskstats21:10:173
758299211,4cyclictest0-21swapper/322:06:343
758299211,4cyclictest0-21swapper/322:06:343
7582992113,5cyclictest848-21sssd_nss20:40:003
7582992111,2cyclictest86550irq/126-enp1s0-21:05:223
758299211,11cyclictest581-21systemd-journal20:05:003
758299211,11cyclictest32652-21/usr/sbin/munin20:05:203
758299211,11cyclictest27953-21/usr/sbin/munin23:35:153
758299211,11cyclictest0-21swapper/323:20:153
757699211,9cyclictest102062sed22:55:002
757699211,6cyclictest0-21swapper/222:49:022
757699211,6cyclictest0-21swapper/200:38:202
756499217,3cyclictest29524-21/usr/sbin/munin22:25:260
756499214,9cyclictest779-21ntpd22:14:390
756499213,3cyclictest3592-1kworker/u9:100:26:070
756499211,8cyclictest10679-21users20:25:280
756499211,11cyclictest13211-21sshd20:34:340
756499211,11cyclictest0-21swapper/021:11:300
756499211,10cyclictest0-21swapper/023:41:300
756499210,6cyclictest0-21swapper/021:30:360
758299205,7cyclictest0-21swapper/321:52:023
758299205,3cyclictest32214-21pmu-power21:15:223
758299203,5cyclictest3592-1kworker/u9:122:25:233
758299203,3cyclictest2900-21kworker/u8:020:14:413
758299203,3cyclictest12050irq/123-ahci[0021:35:013
758299202,5cyclictest0-21swapper/300:30:133
758299201,7cyclictest8584-21sshd20:21:363
758299201,6cyclictest7522-21/usr/sbin/munin21:35:233
758299201,6cyclictest14634-21snmpd19:15:013
7582992014,3cyclictest14634-21snmpd21:27:393
7582992013,3cyclictest14634-21snmpd20:25:143
758299201,14cyclictest86550irq/126-enp1s0-00:15:263
758299201,13cyclictest0-21swapper/323:02:133
758299201,12cyclictest0-21swapper/300:10:083
758299201,12cyclictest0-21swapper/300:10:083
758299201,11cyclictest1269-21fschecks_count21:20:153
757699205,7cyclictest29019-21ntp_states23:35:202
757699201,6cyclictest0-21swapper/222:07:092
757699201,6cyclictest0-21swapper/222:07:092
756499206,6cyclictest0-21swapper/000:30:380
756499205,3cyclictest6595-21/usr/sbin/munin00:00:200
756499204,14cyclictest21436-21idleruntime-cro19:40:000
756499202,8cyclictest14648-21/usr/sbin/munin00:15:270
756499202,5cyclictest22912-21smart_sda19:40:270
7564992014,3cyclictest28652-21idleruntime-cro21:10:000
758299199,3cyclictest86850irq/129-enp1s0-00:07:093
758299198,3cyclictest17880-21/usr/sbin/munin20:45:183
758299195,3cyclictest86550irq/126-enp1s0-22:45:303
758299194,9cyclictest0-21swapper/320:50:133
758299192,9cyclictest581-21systemd-journal22:18:073
758299191,6cyclictest2316-21awk22:35:193
758299191,5cyclictest11772-21fschecks_time20:30:153
758299191,4cyclictest814-21sshd23:59:343
7582991914,3cyclictest28248-21/usr/sbin/munin19:55:173
7582991914,3cyclictest28248-21/usr/sbin/munin19:55:173
758299191,12cyclictest0-21swapper/323:05:153
758299191,12cyclictest0-21swapper/323:05:153
758299191,12cyclictest0-21swapper/322:02:413
758299191,11cyclictest0-21swapper/323:40:203
758299191,11cyclictest0-21swapper/322:40:123
758299191,11cyclictest0-21swapper/319:38:403
758299191,11cyclictest0-21swapper/319:15:073
758299191,10cyclictest28662-21diskmemload23:27:213
758299191,10cyclictest0-21swapper/322:20:103
758299190,12cyclictest0-21swapper/323:53:543
757699197,3cyclictest86550irq/126-enp1s0-22:55:272
757699191,8cyclictest9721-21/usr/sbin/munin21:40:122
7576991913,3cyclictest1956-21dump-pmu-power23:50:012
757699191,11cyclictest0-21swapper/223:00:362
7576991910,2cyclictest30-21rcuc/221:50:282
756499197,3cyclictest4109-21diskstats22:40:180
756499196,3cyclictest86750irq/128-enp1s0-19:25:120
756499195,3cyclictest18642-21awk20:45:180
756499194,8cyclictest12168-21kworker/u8:123:00:520
756499191,5cyclictest0-21swapper/023:37:060
756499191,3cyclictest0-21swapper/023:53:060
7564991913,3cyclictest16956-21seq23:10:070
756499191,2cyclictest11591-21kworker/u8:022:01:330
7564991912,4cyclictest9441-21cut00:05:170
756499191,11cyclictest0-21swapper/021:39:300
157512191,14sleep215764-21diskstats00:20:142
758299185,3cyclictest86850irq/129-enp1s0-19:40:213
758299181,9cyclictest18682-21ntp_states00:25:243
758299181,7cyclictest0-21swapper/322:30:183
758299181,6cyclictest16320-21/usr/sbin/munin21:55:253
758299181,6cyclictest0-21swapper/323:45:153
7582991813,3cyclictest11149-21if_err_enp2s022:55:193
7582991812,3cyclictest14634-21snmpd19:27:393
757699189,2cyclictest30-21rcuc/221:55:282
757699184,4cyclictest3592-1kworker/u9:119:52:132
757699184,3cyclictest86550irq/126-enp1s0-22:40:312
757699182,6cyclictest8866-21munin-run20:25:022
757699181,7cyclictest17581-21gltestperf23:10:162
757699181,7cyclictest0-21swapper/221:25:342
757699181,7cyclictest0-21swapper/221:12:342
757699181,6cyclictest0-21swapper/221:15:222
7576991813,3cyclictest22328-21hddtemp_smartct19:40:202
7576991812,3cyclictest24238-21turbostat.cron21:00:012
757699181,15cyclictest0-21swapper/219:35:142
757699181,11cyclictest0-21swapper/221:47:082
757699181,10cyclictest0-21swapper/200:30:232
756499184,5cyclictest0-21swapper/023:45:500
756499181,9cyclictest14634-21snmpd20:15:120
756499181,5cyclictest22280-21/usr/sbin/munin20:55:290
756499181,5cyclictest1534-21smart_sda20:05:260
7564991813,3cyclictest24476-21/usr/sbin/munin21:00:260
7564991813,3cyclictest24476-21/usr/sbin/munin21:00:260
7564991812,3cyclictest31786-21awk22:30:140
7564991812,3cyclictest30012-21sshd19:57:360
7564991812,3cyclictest30012-21sshd19:57:360
46812181,2sleep20-21swapper/220:15:072
319752181,2sleep00-21swapper/020:00:270
758299171,9cyclictest0-21swapper/320:40:143
758299171,5cyclictest0-21swapper/319:48:083
7582991712,3cyclictest836-21runrttasks19:20:293
7582991711,3cyclictest25730-21sendmail_mailtr21:00:253
7582991711,3cyclictest25730-21sendmail_mailtr21:00:253
757699179,2cyclictest0-21swapper/223:50:162
757699174,3cyclictest86550irq/126-enp1s0-23:40:162
757699172,5cyclictest1244-21df_inode21:20:152
757699171,7cyclictest13567-21diskstats00:15:152
757699171,7cyclictest0-21swapper/221:35:302
757699171,6cyclictest9902-21irqstats20:25:202
757699171,6cyclictest31949-21grep20:00:272
757699171,6cyclictest28662-21diskmemload22:24:082
757699171,6cyclictest26392-21smart_sda22:15:292
757699171,6cyclictest0-21swapper/223:26:442
757699171,6cyclictest0-21swapper/223:21:182
757699171,5cyclictest0-21swapper/220:45:192
7576991712,3cyclictest31804-21cat22:30:132
7576991712,3cyclictest2491-21/usr/sbin/munin20:10:192
7576991712,3cyclictest15686-21/usr/sbin/munin20:40:272
7576991711,3cyclictest23276-21entropy22:10:152
756499171,6cyclictest8818-21sshd20:24:360
7564991712,3cyclictest15417-21/usr/sbin/munin00:20:210
7564991711,3cyclictest24120-21expr19:45:160
7564991711,3cyclictest17620-21grep19:30:150
166572171,2sleep20-21swapper/223:09:122
166572171,2sleep20-21swapper/223:09:122
124012171,2sleep20-21swapper/219:17:182
758299161,9cyclictest0-21swapper/319:30:143
757699165,3cyclictest12050irq/123-ahci[0021:05:022
757699165,3cyclictest12050irq/123-ahci[0021:05:022
757699164,3cyclictest15731-21hddtemp_smartct19:25:182
757699162,4cyclictest26450-21kworker/2:019:48:242
757699161,8cyclictest780-21rngd23:15:062
757699161,6cyclictest0-21swapper/222:26:252
757699161,5cyclictest12795-21grep20:30:272
757699161,4cyclictest19198-21users00:25:292
7576991611,3cyclictest4674-21cat23:55:132
757699161,12cyclictest8937-21tr00:05:132
756499161,9cyclictest0-21swapper/000:39:280
756499161,14cyclictest0-21swapper/019:51:230
7564991610,4cyclictest21341-21smart_sda20:50:250
20342160,2sleep20-21swapper/220:07:212
757699159,3cyclictest13490-21acpi20:35:172
757699151,6cyclictest0-21swapper/219:30:162
7576991510,3cyclictest12926-21cpuspeed_turbos19:20:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional