You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 12:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue May 13, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
139302239164,25sleep30-21swapper/319:07:063
139192239163,25sleep20-21swapper/219:06:572
141282237163,24sleep10-21swapper/119:09:381
139792203166,25sleep00-21swapper/019:07:440
2298421800,3sleep01424599cyclictest23:05:560
3005621620,5sleep01424599cyclictest20:55:220
95221550,4sleep01424599cyclictest22:15:340
95221550,4sleep01424599cyclictest22:15:340
50032610,1sleep011-21rcuc/020:00:140
288142610,3sleep339-21rcuc/300:35:153
7312580,1sleep20-21swapper/223:30:202
1425499411,30cyclictest21063-40dnf23:04:322
1425499411,25cyclictest16105-21latency_hist21:40:012
1424599399,14cyclictest7927-21munin-run22:35:010
1425499381,23cyclictest3761-21/usr/sbin/munin22:25:272
14245993812,12cyclictest31502-21munin-run22:15:010
14245993811,13cyclictest10726-21awk23:55:010
1425499361,21cyclictest25643-21cat23:15:022
1425499361,20cyclictest3161-21/usr/sbin/munin21:10:242
1425499361,20cyclictest3161-21/usr/sbin/munin21:10:242
1425499353,28cyclictest29624-21ntp_states00:35:252
1425499351,31cyclictest26780-21grep19:35:172
1425499351,24cyclictest2818-21sensors_temp22:20:252
1425499351,22cyclictest24498-21df00:25:162
1425499351,19cyclictest15366-21/usr/sbin/munin00:05:272
1425499341,24cyclictest5704-21diskstats21:15:182
1425499341,20cyclictest25044-21latency_hist22:00:012
1425499341,20cyclictest18535-21/usr/sbin/munin21:45:272
1425499341,17cyclictest8638-21latency_hist23:50:022
1425499340,26cyclictest0-21swapper/220:00:282
1425499335,24cyclictest26427-21/usr/sbin/munin00:30:162
1425499335,24cyclictest26427-21/usr/sbin/munin00:30:162
1425499335,21cyclictest86750irq/128-enp1s0-19:45:192
1425499333,20cyclictest12050irq/123-ahci[0020:40:022
1425499332,22cyclictest30129-21/usr/sbin/munin23:25:172
1425499331,19cyclictest18005-21sh22:55:232
1425499331,17cyclictest14329-21ps00:00:262
1425499322,27cyclictest32306-21/usr/sbin/munin22:15:182
1425499322,27cyclictest32306-21/usr/sbin/munin22:15:182
1425499322,25cyclictest24275-21diskstats19:30:162
1425499322,17cyclictest86750irq/128-enp1s0-23:35:282
1425499321,28cyclictest8783-21gltestperf22:35:192
1425499321,28cyclictest26378-21/usr/sbin/munin22:00:242
1425499321,20cyclictest20796-21netstat00:15:222
1425499321,19cyclictest6132-21taskset23:42:402
1425499321,18cyclictest27936-21/usr/sbin/munin23:20:282
14260993110,10cyclictest39-21rcuc/322:30:223
1425499314,17cyclictest86550irq/126-enp1s0-22:50:182
14254993113,15cyclictest31109-21/usr/sbin/munin21:00:212
1425499311,27cyclictest28909-21/usr/sbin/munin20:55:182
1425499311,27cyclictest28043-21idleruntime22:05:192
1425499311,27cyclictest22910-21head23:05:322
1425499311,27cyclictest20729-21/usr/sbin/munin21:50:132
1425499311,27cyclictest19537-21/usr/sbin/munin19:20:212
1425499311,27cyclictest12543-21/usr/sbin/munin22:45:242
1425499311,22cyclictest86550irq/126-enp1s0-20:30:322
1425499311,22cyclictest28385-21awk19:40:142
1425499311,18cyclictest29531-21/usr/sbin/munin22:10:152
1425499311,18cyclictest0-21swapper/221:30:192
14254993111,17cyclictest17833-21expr00:10:172
187242300,5sleep339-21rcuc/321:45:153
1425499303,17cyclictest5750irq/9-acpi19:50:242
1425499302,23cyclictest14797-21diskstats19:10:152
1425499301,26cyclictest9742-21/usr/sbin/munin21:25:242
1425499301,26cyclictest9147-21/usr/sbin/munin20:10:162
1425499301,26cyclictest8811-21/usr/sbin/munin21:20:262
1425499301,26cyclictest6949-21/usr/sbin/munin20:05:162
1425499301,26cyclictest6949-21/usr/sbin/munin20:05:162
1425499301,26cyclictest26720-21/usr/sbin/munin20:50:292
1425499301,26cyclictest25742-21/usr/sbin/munin23:15:232
1425499301,26cyclictest24523-21/usr/sbin/munin20:45:292
1425499301,26cyclictest21969-21/usr/sbin/munin00:20:232
1425499301,26cyclictest17341-21/usr/sbin/munin19:15:202
1425499301,26cyclictest15738-21/usr/sbin/munin20:25:172
1425499301,26cyclictest13535-21/usr/sbin/munin20:20:202
1425499301,26cyclictest10942-21/usr/sbin/munin23:55:222
1425499301,18cyclictest21728-21/usr/sbin/munin19:25:252
1425499300,27cyclictest936-21/usr/sbin/munin21:05:172
1425499300,27cyclictest10346-21/usr/sbin/munin22:40:242
1425499291,25cyclictest8751-21/usr/sbin/munin23:50:142
1425499291,25cyclictest2544-21munin-node19:55:122
1425499291,25cyclictest22328-21/usr/sbin/munin20:40:162
1425499291,25cyclictest16340-21/usr/sbin/munin21:40:152
1425499291,25cyclictest11335-21/usr/sbin/munin20:15:142
1425499290,26cyclictest6101-21cpuspeed_turbos22:30:142
293852271,2sleep329421-21/usr/sbin/munin19:40:233
167342270,6sleep3401ktimersoftd/320:25:223
1424599272,13cyclictest21120-21munin-run23:05:000
1426099261,2cyclictest86550irq/126-enp1s0-22:25:223
1425099269,8cyclictest0-21swapper/100:20:011
1425099261,5cyclictest14322-21proc_pri00:00:251
1425099261,17cyclictest9620-21cat23:50:191
1426099251,11cyclictest26811-21ntp_states23:15:253
1425099252,17cyclictest29636-21kworker/u8:123:24:481
1425099251,15cyclictest28403-21ntp_states22:05:231
1426099243,13cyclictest780-21rngd19:45:243
1426099242,5cyclictest86550irq/126-enp1s0-21:35:203
1425099245,9cyclictest0-21swapper/122:37:141
1425099244,2cyclictest3743-21taskset21:10:161
1425099244,2cyclictest3743-21taskset21:10:161
14250992418,3cyclictest16320-21latency_hist21:40:021
14250992414,9cyclictest0-21swapper/123:27:281
1425099241,17cyclictest0-21swapper/122:30:091
1425099241,16cyclictest86750irq/128-enp1s0-23:35:181
1425099241,12cyclictest0-21swapper/121:31:521
1425099240,6cyclictest0-21swapper/123:10:481
14245992419,3cyclictest10719-21awk20:10:290
1426099239,3cyclictest29855-21sshd23:24:343
1426099232,18cyclictest14593-21tr19:10:133
1426099231,11cyclictest784-21NetworkManager00:08:293
1426099231,11cyclictest31723-21/usr/sbin/munin22:15:253
1426099231,11cyclictest31723-21/usr/sbin/munin22:15:253
1426099231,11cyclictest23234-21smartctl00:20:273
1426099231,11cyclictest21377-21sshd19:23:333
1425099238,9cyclictest0-21swapper/122:27:441
1425099237,9cyclictest0-21swapper/122:19:221
1425099237,9cyclictest0-21swapper/122:19:221
1425099237,3cyclictest20589-21cat21:50:011
14250992314,2cyclictest2934-21diskmemload21:53:231
1425099231,19cyclictest18977-21tr20:30:231
1425099231,17cyclictest0-21swapper/122:55:381
1425099231,16cyclictest1067-21kworker/u8:223:05:441
1425099231,16cyclictest0-21swapper/122:41:501
1425099231,16cyclictest0-21swapper/122:00:581
1425099231,16cyclictest0-21swapper/100:05:331
1425099231,15cyclictest86750irq/128-enp1s0-22:45:161
1425099230,17cyclictest0-21swapper/121:55:071
1424599239,13cyclictest0-21swapper/020:55:000
1426099228,3cyclictest31109-21/usr/sbin/munin21:00:223
1426099228,3cyclictest17341-21/usr/sbin/munin19:15:233
1426099226,5cyclictest3265-21cpuspeed_turbos21:10:143
1426099226,5cyclictest3265-21cpuspeed_turbos21:10:143
1426099221,4cyclictest0-21swapper/322:49:543
1426099221,14cyclictest0-21swapper/323:34:303
1426099221,10cyclictest21728-21/usr/sbin/munin19:25:213
1426099221,10cyclictest0-21swapper/322:04:403
1425099227,4cyclictest16933-21df_inode19:15:021
1425099222,7cyclictest28556-21/usr/sbin/munin00:35:251
1425099222,3cyclictest0-21swapper/123:41:101
1425099222,11cyclictest25080-21memory00:25:211
1425099222,11cyclictest1907-21diskstats22:20:151
1425099221,9cyclictest0-21swapper/121:40:261
1425099221,19cyclictest0-21swapper/123:55:411
1425099221,18cyclictest27029-21switchtime23:15:291
1425099221,10cyclictest0-21swapper/123:03:071
1425099221,10cyclictest0-21swapper/121:25:531
14250992210,3cyclictest5690-21df21:15:161
1424599221,7cyclictest0-21swapper/023:33:490
1424599221,4cyclictest0-21swapper/022:03:370
323682210,2sleep00-21swapper/021:00:230
196202210,5sleep01424599cyclictest00:15:010
1426099217,3cyclictest24164-21/usr/sbin/munin00:25:213
1426099214,2cyclictest25244-21basename20:45:203
1426099211,5cyclictest12317-21munin-run22:45:013
1426099211,4cyclictest31195-21ntp_states23:25:233
1426099211,4cyclictest0-21swapper/323:12:553
14260992114,4cyclictest7152-21head23:45:173
14260992114,4cyclictest20683-21fschecks_time20:35:183
1426099211,10cyclictest27357-21cut00:30:223
1426099211,10cyclictest27357-21cut00:30:223
1425099218,11cyclictest23386-21kworker/u8:220:25:041
1425099217,3cyclictest18152-21grep00:10:181
1425099211,8cyclictest936-21/usr/sbin/munin21:05:161
1425099211,7cyclictest0-21swapper/122:50:071
14250992117,3cyclictest0-21swapper/120:09:101
14250992117,3cyclictest0-21swapper/120:09:101
1425099211,3cyclictest14634-21snmpd20:51:341
1425099211,18cyclictest0-21swapper/119:35:211
1425099211,18cyclictest0-21swapper/119:30:111
1425099211,18cyclictest0-21swapper/119:27:071
1425099211,15cyclictest0-21swapper/122:10:101
1425099211,15cyclictest0-21swapper/100:17:211
1425099211,14cyclictest19425-21kworker/u8:200:31:451
1425099211,14cyclictest19425-21kworker/u8:200:31:451
1425099211,14cyclictest0-21swapper/121:20:441
1425099211,11cyclictest7225-21hddtemp_smartct23:45:171
1425099211,11cyclictest31922-21irqstats21:00:181
1425099211,11cyclictest25588-21ntp_states20:45:241
1425099210,19cyclictest0-21swapper/120:00:281
1424599211,5cyclictest0-21swapper/021:55:590
261292201,15sleep326304-21cstates19:35:133
208992200,14sleep30-21swapper/300:15:243
1426099201,9cyclictest780-21rngd20:10:283
1426099201,7cyclictest23996-21ntp_states21:55:253
1426099201,4cyclictest2544-21/usr/sbin/munin19:55:163
1426099201,17cyclictest2930-21awk21:10:013
1425099202,6cyclictest28909-21/usr/sbin/munin20:55:161
14250992017,2cyclictest0-21swapper/120:41:081
14250992014,1cyclictest0-21swapper/123:31:581
1425099201,3cyclictest0-21swapper/120:13:151
1425099200,10cyclictest21-21rcuc/120:00:001
1424599201,5cyclictest0-21swapper/023:41:430
14245992010,6cyclictest0-21swapper/000:24:550
131442200,1sleep00-21swapper/022:45:180
182542191,2sleep00-21swapper/021:44:320
1426099194,8cyclictest0-21swapper/323:05:323
1426099192,14cyclictest12754-21irqstats21:30:193
1426099191,9cyclictest0-21swapper/322:05:343
1426099191,6cyclictest0-21swapper/321:21:213
1426099191,5cyclictest6721-21munin-run20:05:003
1426099191,4cyclictest4361-21/usr/sbin/munin23:40:233
1426099191,4cyclictest28909-21/usr/sbin/munin20:55:163
1426099191,10cyclictest3408-21sshd22:23:333
1426099191,10cyclictest13138-40updatedb00:00:103
14250991914,3cyclictest11818-21fschecks_time20:15:131
1425099191,2cyclictest0-21swapper/119:23:121
1425099191,17cyclictest0-21swapper/120:38:531
1425099191,17cyclictest0-21swapper/120:24:061
1425099191,17cyclictest0-21swapper/119:48:281
1425099191,17cyclictest0-21swapper/119:44:501
1425099191,17cyclictest0-21swapper/119:12:331
1424599197,11cyclictest0-21swapper/020:00:000
1424599191,15cyclictest20529-21strings19:20:240
1424599191,10cyclictest6554-21/usr/sbin/munin23:45:200
1424599191,10cyclictest28279-21sshd00:34:350
1424599191,10cyclictest28279-21sshd00:34:350
1424599191,10cyclictest20729-21/usr/sbin/munin21:50:260
1424599191,10cyclictest17937-21/usr/sbin/munin20:30:180
1424599191,10cyclictest16942-21/usr/sbin/munin22:55:150
1424599191,10cyclictest1545-21/usr/sbin/munin22:20:150
71982180,4sleep01424599cyclictest21:18:340
274412181,2sleep00-21swapper/023:16:450
1426099183,6cyclictest2934-21diskmemload22:55:243
1426099182,3cyclictest0-21swapper/323:35:233
1426099182,2cyclictest86550irq/126-enp1s0-22:10:193
1426099181,7cyclictest12408-21ntpq20:15:203
1426099181,4cyclictest0-21swapper/323:59:473
1426099181,2cyclictest86550irq/126-enp1s0-21:41:223
14260991812,3cyclictest14827-21cat22:50:133
1426099181,14cyclictest8909-21cpuspeed_turbos23:50:133
1426099181,14cyclictest225732sleep321:53:343
1426099181,10cyclictest0-21swapper/323:00:243
1426099180,6cyclictest23928-21/usr/sbin/munin19:30:253
1425099181,16cyclictest0-21swapper/119:53:151
1424599186,3cyclictest14634-21snmpd22:40:200
1424599182,9cyclictest14634-21snmpd23:21:210
1424599181,9cyclictest0-21swapper/023:58:490
1424599181,7cyclictest31827-1kworker/u9:221:25:230
1424599181,6cyclictest0-21swapper/000:15:540
1424599181,5cyclictest0-21swapper/023:11:110
1424599181,5cyclictest0-21swapper/022:27:090
1424599181,5cyclictest0-21swapper/020:35:230
1424599181,4cyclictest16760-21sed00:05:280
14245991811,4cyclictest14674-21cut21:35:170
50002171,2sleep00-21swapper/021:13:330
50002171,2sleep00-21swapper/021:13:330
173302170,2sleep00-21swapper/019:15:080
1426099171,7cyclictest0-21swapper/300:12:263
1426099171,4cyclictest0-21swapper/322:35:363
1426099171,15cyclictest0-21swapper/320:21:053
14260991711,4cyclictest842-21lldpd19:54:203
1426099171,13cyclictest5357-21/usr/sbin/munin21:15:223
1426099171,10cyclictest0-21swapper/321:25:523
14260991710,4cyclictest8375-21/usr/sbin/munin20:05:263
14260991710,4cyclictest8375-21/usr/sbin/munin20:05:263
14260991710,4cyclictest18596-21sed20:30:183
1424599177,3cyclictest86750irq/128-enp1s0-00:37:090
1424599175,4cyclictest14504-21wc00:00:280
1424599174,3cyclictest8620-21ntp_states21:20:240
1424599174,3cyclictest0-21swapper/019:50:240
1424599171,7cyclictest30514-21/usr/sbin/munin19:45:250
1424599171,7cyclictest30129-21/usr/sbin/munin23:25:230
1424599171,6cyclictest0-21swapper/022:37:120
1424599171,6cyclictest0-21swapper/021:30:460
14245991712,3cyclictest23124-21sed19:25:250
14245991712,3cyclictest14516-21cat20:20:210
14245991712,3cyclictest12360-21nfs4_client20:15:180
1424599171,15cyclictest0-21swapper/022:09:130
1424599171,15cyclictest0-21swapper/021:48:390
1424599171,13cyclictest28973-21grep19:40:170
14260991611,3cyclictest28148-21timerwakeupswit20:50:283
1426099161,12cyclictest22467-21tr20:40:143
1424599169,4cyclictest848-21sssd_nss19:35:200
1424599161,6cyclictest24164-21/usr/sbin/munin00:25:170
1424599161,6cyclictest0-21swapper/022:51:530
1424599161,5cyclictest2191-21cpu23:35:150
1424599161,2cyclictest0-21swapper/020:40:210
1424599161,2cyclictest0-21swapper/019:33:020
1424599161,14cyclictest0-21swapper/020:49:030
14245991611,3cyclictest7282-21df20:05:140
14245991611,3cyclictest7282-21df20:05:140
14245991611,3cyclictest16654-21memory20:25:210
14245991511,2cyclictest14447-21/usr/sbin/munin19:10:170
14245991510,3cyclictest1163-21cat21:05:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional