You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-22 - 09:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Oct 22, 2025 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
294662240163,24sleep10-21swapper/119:08:201
292742238163,24sleep30-21swapper/319:05:523
294032237164,60sleep00-21swapper/019:07:310
293742221192,20sleep20-21swapper/219:07:082
29695999476,15cyclictest2074-21munin-run21:50:012
29695998176,3cyclictest0-21swapper/221:50:572
29695997669,4cyclictest792-21avahi-daemon00:10:282
29695997067,2cyclictest0-21swapper/223:51:132
29695996965,3cyclictest0-21swapper/221:24:382
20442570,3sleep02968099cyclictest23:00:290
179312510,1sleep00-21swapper/023:37:590
2968099409,15cyclictest0-21swapper/020:20:010
323092370,2sleep30-21swapper/321:45:013
2968099371,12cyclictest20313-21perf23:45:010
136442350,3sleep32970199cyclictest22:15:163
2969199324,18cyclictest0-21swapper/120:26:421
2968099329,11cyclictest0-21swapper/020:10:020
178292320,5sleep10-21swapper/122:25:141
2968099313,13cyclictest26273-21awk22:45:010
29680993112,16cyclictest19666-21munin-run22:30:010
2969599309,2cyclictest86650irq/127-enp1s0-20:54:072
2969599300,15cyclictest25434-21kworker/2:219:17:442
29695992921,2cyclictest25434-21kworker/2:222:04:162
29695992913,3cyclictest25434-21kworker/2:200:22:402
2969599291,11cyclictest32-21ksoftirqd/221:08:352
44012280,4sleep011-21rcuc/023:05:270
2969599282,19cyclictest25434-21kworker/2:200:26:082
2969599282,14cyclictest25434-21kworker/2:200:05:442
2969599282,12cyclictest25434-21kworker/2:221:31:362
2968099281,13cyclictest22197-21perf20:05:010
29680992810,15cyclictest17482-21basename22:25:010
2968099280,13cyclictest10885-21awk22:10:010
2969599272,19cyclictest25434-21kworker/2:200:16:462
2969599271,7cyclictest0-21swapper/223:07:382
29695992717,2cyclictest25434-21kworker/2:221:35:062
29695992715,2cyclictest25434-21kworker/2:222:15:282
29695992712,3cyclictest25434-21kworker/2:223:16:522
54982260,1sleep10-21swapper/123:10:181
29695992619,4cyclictest22670-21/usr/sbin/munin22:35:162
29695992618,2cyclictest25434-21kworker/2:223:45:082
29695992618,2cyclictest25434-21kworker/2:223:45:082
29695992613,3cyclictest25434-21kworker/2:222:25:262
2969599261,11cyclictest25434-21kworker/2:221:29:232
29691992611,13cyclictest22805-21kworker/u8:220:15:141
2968099262,12cyclictest12159-21munin-run00:40:010
2969599259,3cyclictest25434-21kworker/2:223:33:522
29695992512,7cyclictest25434-21kworker/2:223:36:342
29695992511,3cyclictest25434-21kworker/2:223:03:122
29695992510,3cyclictest25434-21kworker/2:223:59:322
2969199259,14cyclictest22805-21kworker/u8:219:38:141
29691992511,12cyclictest30600-21kworker/u8:023:01:481
29701992413,6cyclictest223892sleep323:48:463
29701992413,6cyclictest223892sleep323:48:463
2969599245,9cyclictest25434-21kworker/2:219:55:482
29695992417,2cyclictest25434-21kworker/2:220:31:442
29695992414,2cyclictest25434-21kworker/2:222:11:142
29695992412,6cyclictest25434-21kworker/2:200:04:202
29695992412,3cyclictest25434-21kworker/2:220:40:092
29695992410,3cyclictest25434-21kworker/2:222:58:262
29695992410,3cyclictest25434-21kworker/2:221:17:312
2969199249,13cyclictest22083-21kworker/u8:122:45:451
2969199248,14cyclictest28999-21kworker/u8:120:35:141
2969199242,17cyclictest0-21swapper/123:07:401
2969199242,16cyclictest0-21swapper/121:40:481
29691992413,3cyclictest11000-21irqstats00:35:191
2969199240,15cyclictest86750irq/128-enp1s0-22:42:491
2970199231,5cyclictest0-21swapper/323:18:543
2969599235,7cyclictest86650irq/127-enp1s0-20:06:362
2969599234,9cyclictest25434-21kworker/2:219:46:582
29695992312,3cyclictest25434-21kworker/2:222:30:522
29695992312,2cyclictest25434-21kworker/2:223:29:242
2969599231,12cyclictest25434-21kworker/2:223:42:042
29695992311,2cyclictest25434-21kworker/2:222:52:482
29695992310,3cyclictest25434-21kworker/2:222:22:042
2969199231,9cyclictest31241-21diskstats22:55:161
29691992315,1cyclictest0-21swapper/121:35:181
2969199231,19cyclictest16472-21df_inode23:35:141
2969199231,16cyclictest0-21swapper/120:46:101
2969199231,16cyclictest0-21swapper/119:55:551
29691992311,10cyclictest23-21ksoftirqd/119:25:501
29691992311,10cyclictest23-21ksoftirqd/119:25:501
2968099233,2cyclictest0-21swapper/022:19:250
2968099231,11cyclictest9306-21munin-run23:20:020
2970199222,8cyclictest5570-21munin-run00:25:023
2970199222,17cyclictest16729-21grep23:35:163
2970199221,9cyclictest0-21swapper/323:59:583
2969599229,2cyclictest86650irq/127-enp1s0-20:25:582
2969599228,4cyclictest25434-21kworker/2:223:12:242
2969599228,3cyclictest25434-21kworker/2:222:43:302
2969599227,3cyclictest25434-21kworker/2:200:35:142
2969599226,9cyclictest32-21ksoftirqd/219:44:382
2969599226,3cyclictest25434-21kworker/2:221:42:282
2969599226,3cyclictest25434-21kworker/2:219:30:482
2969599222,7cyclictest25434-21kworker/2:219:11:482
29695992210,3cyclictest25434-21kworker/2:222:49:242
2969199222,18cyclictest0-21swapper/122:06:531
2969199222,15cyclictest0-21swapper/121:11:051
2969199222,11cyclictest24725-21ntp_states21:25:241
2969199221,9cyclictest0-21swapper/100:20:581
29691992214,3cyclictest0-21swapper/120:01:551
2969199221,3cyclictest0-21swapper/123:40:491
2969199221,19cyclictest0-21swapper/119:19:061
2969199221,16cyclictest0-21swapper/122:34:571
2969199221,16cyclictest0-21swapper/120:24:281
2969199221,16cyclictest0-21swapper/100:30:561
2969199221,10cyclictest780-21rngd21:20:241
29691992210,5cyclictest30013-21kworker/u8:021:46:481
2970199218,4cyclictest0-21swapper/323:25:133
29701992116,3cyclictest31324-21idleruntime-cro00:10:013
2969599218,2cyclictest25434-21kworker/2:221:55:402
2969599217,3cyclictest25434-21kworker/2:223:20:042
2969599216,3cyclictest25434-21kworker/2:219:39:222
2969599215,7cyclictest32-21ksoftirqd/221:02:432
2969599214,3cyclictest86650irq/127-enp1s0-20:59:182
2969599211,8cyclictest25434-21kworker/2:220:16:172
2969599211,13cyclictest25434-21kworker/2:220:10:182
2969199219,3cyclictest2654-21diskstats21:50:201
2969199216,4cyclictest0-21swapper/100:19:121
2969199212,17cyclictest0-21swapper/119:14:101
29691992117,3cyclictest0-21swapper/122:36:461
29691992114,3cyclictest0-21swapper/123:20:351
2969199211,18cyclictest0-21swapper/119:31:121
2969199211,15cyclictest0-21swapper/121:02:061
2969199211,15cyclictest0-21swapper/120:05:521
2969199211,14cyclictest0-21swapper/123:29:471
2969199211,14cyclictest0-21swapper/120:52:091
2969199211,14cyclictest0-21swapper/100:07:101
2969199210,17cyclictest86850irq/129-enp1s0-22:53:391
2969199210,16cyclictest0-21swapper/123:55:561
2968099212,11cyclictest13602-21/usr/sbin/munin19:45:150
2968099211,5cyclictest0-21swapper/021:14:120
2968099211,11cyclictest32458-21/usr/sbin/munin21:45:140
2968099211,11cyclictest28050-21memory00:00:220
2968099211,11cyclictest24925-21/usr/sbin/munin23:55:130
2968099211,11cyclictest14634-21snmpd21:09:500
86332201,2sleep30-21swapper/319:30:423
2970199207,3cyclictest12050irq/123-ahci[0020:25:013
2970199207,3cyclictest0-21swapper/322:41:103
2970199203,7cyclictest14634-21snmpd22:35:053
2970199202,15cyclictest32475-21grep22:55:283
2970199201,7cyclictest0-21swapper/300:30:163
2970199201,4cyclictest0-21swapper/300:00:123
2970199201,11cyclictest24610-21/usr/sbin/munin20:10:173
2969599208,2cyclictest25434-21kworker/2:220:47:372
2969599206,9cyclictest30-21rcuc/222:07:242
2969599206,3cyclictest25434-21kworker/2:220:36:242
2969599205,8cyclictest30-21rcuc/219:27:122
2969599205,8cyclictest30-21rcuc/219:27:122
2969599205,3cyclictest25434-21kworker/2:219:22:002
2969599204,5cyclictest32-21ksoftirqd/220:23:082
2969599203,3cyclictest25434-21kworker/2:221:14:472
2969199207,3cyclictest12229-21sshd20:57:431
2969199202,17cyclictest0-21swapper/122:13:001
29691992016,3cyclictest0-21swapper/100:13:211
2969199201,3cyclictest0-21swapper/119:51:461
29691992013,2cyclictest0-21swapper/121:30:451
2969199201,18cyclictest0-21swapper/123:33:461
2969199201,18cyclictest0-21swapper/122:03:171
2969199201,17cyclictest0-21swapper/122:17:141
2969199201,15cyclictest0-21swapper/121:17:221
2969199201,15cyclictest0-21swapper/121:07:231
2969199201,15cyclictest0-21swapper/119:46:061
2969199201,14cyclictest0-21swapper/123:49:101
2969199201,14cyclictest0-21swapper/123:49:101
2969199201,14cyclictest0-21swapper/120:42:521
2969199201,14cyclictest0-21swapper/100:26:201
2969199201,10cyclictest7441-21grep23:15:141
2969199201,10cyclictest17462-21latency_hist22:25:011
2969199201,10cyclictest0-21swapper/119:40:141
2969199200,15cyclictest0-21swapper/120:32:551
2968099209,8cyclictest11312-21sshd23:22:440
2968099208,2cyclictest3610-21strings19:20:200
2968099207,3cyclictest0-21swapper/000:17:280
2968099204,6cyclictest0-21swapper/023:53:180
2968099204,6cyclictest0-21swapper/022:38:150
2968099201,8cyclictest14634-21snmpd00:25:270
2968099201,6cyclictest0-21swapper/022:12:500
2968099201,6cyclictest0-21swapper/021:59:410
29680992015,3cyclictest7591-21grep20:45:240
29680992015,3cyclictest3785-21grep21:50:310
29680992013,4cyclictest8564-21grep19:30:260
29680992013,4cyclictest30975-21latency_hist20:25:010
2968099201,12cyclictest0-21swapper/019:56:150
2968099201,11cyclictest12655-21/usr/sbin/munin21:00:250
2968099201,11cyclictest0-21swapper/021:35:280
2968099201,10cyclictest7998-21/usr/sbin/munin00:30:160
195122201,3sleep30-21swapper/322:28:143
188232201,3sleep30-21swapper/321:12:313
57272191,2sleep30-21swapper/320:43:523
42762191,3sleep10-21swapper/119:21:091
2970199193,3cyclictest0-21swapper/323:08:443
2970199191,7cyclictest0-21swapper/321:37:443
2970199191,3cyclictest9315-21taskset20:50:233
2970199191,3cyclictest11778-21hddtemp_smartct22:10:193
2970199191,10cyclictest31565-21/usr/sbin/munin00:10:163
2970199191,10cyclictest21955-21sshd20:02:173
29701991910,1cyclictest0-21swapper/320:49:433
2969599195,3cyclictest86550irq/126-enp1s0-00:31:042
2969599195,3cyclictest32-21ksoftirqd/219:53:162
2969599193,10cyclictest32-21ksoftirqd/220:01:062
29691991915,3cyclictest0-21swapper/120:12:481
2969199191,16cyclictest0-21swapper/121:56:141
2968099199,3cyclictest5227-21awk20:40:010
2968099197,4cyclictest0-21swapper/021:21:320
2968099192,9cyclictest0-21swapper/023:10:300
2968099191,9cyclictest0-21swapper/000:05:330
2968099191,6cyclictest0-21swapper/023:45:470
2968099191,6cyclictest0-21swapper/023:45:470
2968099191,6cyclictest0-21swapper/022:34:190
2968099191,4cyclictest0-21swapper/021:15:260
29680991913,3cyclictest32236-21munin-run21:45:010
2968099191,10cyclictest3603-21/usr/sbin/munin00:20:160
2968099191,10cyclictest26490-21/usr/sbin/munin22:45:280
2968099191,10cyclictest25859-21/usr/sbin/munin21:30:300
2968099191,10cyclictest16829-21diskmemload22:50:450
2968099191,10cyclictest0-21swapper/022:57:010
2968099190,5cyclictest2051-1kworker/u9:023:31:240
2970199187,3cyclictest848-21sssd_nss19:18:433
2970199186,4cyclictest2051-1kworker/u9:000:15:193
2970199181,9cyclictest0-21swapper/319:53:303
2970199181,7cyclictest0-21swapper/300:38:273
2970199181,6cyclictest0-21swapper/323:52:053
2970199181,6cyclictest0-21swapper/323:24:113
2970199181,6cyclictest0-21swapper/322:52:363
2970199181,6cyclictest0-21swapper/321:51:343
29701991812,3cyclictest3328-21cat20:35:133
29701991812,3cyclictest14175-21grep21:00:293
29701991812,3cyclictest10448-21/usr/sbin/munin20:55:173
2970199181,15cyclictest0-21swapper/321:57:103
2970199181,15cyclictest0-21swapper/321:45:023
2970199181,14cyclictest16829-21diskmemload21:20:113
29701991811,4cyclictest10073-21ps22:05:263
2970199181,11cyclictest0-21swapper/320:30:183
2969199181,16cyclictest0-21swapper/100:02:511
2969199180,5cyclictest0-21swapper/123:50:121
2968099182,3cyclictest17661-21sshd19:53:430
2968099181,9cyclictest784-21NetworkManager20:51:360
2968099181,6cyclictest0-21swapper/021:27:330
2968099181,14cyclictest12962-21find23:25:240
2968099180,7cyclictest0-21swapper/019:10:150
263542181,2sleep00-21swapper/020:12:180
26102181,2sleep30-21swapper/319:20:073
165182181,2sleep316551-21ntpq22:20:233
313092170,8sleep3401ktimersoftd/319:10:263
2970199172,13cyclictest32764-21head20:25:283
2970199171,7cyclictest4815-21/usr/sbin/munin19:25:193
2970199171,7cyclictest4815-21/usr/sbin/munin19:25:193
2970199171,7cyclictest25859-21/usr/sbin/munin21:30:153
2970199171,6cyclictest28461-21munin-run22:50:013
2970199171,6cyclictest0-21swapper/323:30:043
2970199171,6cyclictest0-21swapper/322:30:073
2970199171,6cyclictest0-21swapper/319:35:203
2970199171,6cyclictest0-21swapper/300:26:543
29701991713,3cyclictest0-21swapper/321:25:043
29701991712,3cyclictest8211-21awk22:00:263
29701991712,3cyclictest14132-21fschecks_time19:45:143
2970199171,15cyclictest0-21swapper/323:11:333
2970199171,13cyclictest649-21/usr/sbin/munin23:00:193
29701991711,3cyclictest13375-21dump-pmu-power19:45:003
2968099171,4cyclictest1029-21/usr/sbin/munin20:30:230
2968099171,10cyclictest0-21swapper/022:00:560
2968099171,10cyclictest0-21swapper/000:10:530
29680991710,4cyclictest32675-21grep19:15:040
165842171,3sleep30-21swapper/321:07:053
2970199161,9cyclictest0-21swapper/319:55:233
2970199161,6cyclictest24491-21latency_hist20:10:023
2970199161,6cyclictest0-21swapper/323:42:133
29701991611,3cyclictest19342-21cpuspeed_turbos21:15:123
2970199160,14cyclictest0-21swapper/320:15:163
2968099161,8cyclictest3666-21kworker/0:119:41:120
2968099161,6cyclictest5062-21expr19:25:120
2968099161,6cyclictest5062-21expr19:25:120
2968099161,14cyclictest0-21swapper/020:56:220
29680991611,3cyclictest9512-21expr19:35:160
29680991611,3cyclictest32581-21timerandwakeup20:25:270
2968099152,5cyclictest836-21runrttasks20:43:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional