You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 11:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Nov 05, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
102002240163,24sleep10-21swapper/119:06:071
101762236162,25sleep20-21swapper/219:05:472
104602234162,58sleep30-21swapper/319:09:263
104362232162,56sleep00-21swapper/019:09:080
10607996763,3cyclictest0-21swapper/222:10:122
70242580,2sleep3401ktimersoftd/320:10:403
70242580,2sleep3401ktimersoftd/320:10:403
1059599384,7cyclictest17733-40dnf19:23:370
1059599383,5cyclictest17537-21munin-run21:55:020
1059599378,3cyclictest2370-21/usr/sbin/munin21:20:220
1059599378,3cyclictest2370-21/usr/sbin/munin21:20:220
10595993711,12cyclictest6540-21munin-run21:30:010
1059599370,10cyclictest0-21swapper/022:35:240
1059599368,3cyclictest5833-21/usr/sbin/munin23:55:200
10595993612,21cyclictest7401-21/usr/sbin/munin22:45:280
1059599357,13cyclictest16646-21awk00:20:010
10595993512,20cyclictest11478-21hwlatdetect19:10:170
10595993411,20cyclictest12466-21/usr/sbin/munin00:10:140
1059599341,10cyclictest3205-21sshd23:47:340
205172330,4sleep011-21rcuc/019:30:140
1059599339,20cyclictest28235-21cpuspeed_turbos21:05:090
1059599338,3cyclictest16516-21/usr/sbin/munin20:35:260
1059599338,2cyclictest17200-21df00:20:150
1059599331,4cyclictest13310-21grep22:55:270
10595993313,17cyclictest13688-21grep20:25:280
10595993312,18cyclictest23717-21fgrep19:35:270
10595993310,20cyclictest836-21runrttasks22:50:560
1059599327,5cyclictest24962-21grep00:35:270
1059599326,3cyclictest18085-21/usr/sbin/munin19:25:200
1059599322,5cyclictest27631-21ntp_states22:15:230
1059599322,27cyclictest18875-21grep20:40:170
1059599321,4cyclictest0-21swapper/021:30:270
10595993211,18cyclictest21531-21ls23:15:210
10595993211,18cyclictest20828-21seq00:27:360
10595993210,19cyclictest836-21runrttasks21:03:330
1060099311,22cyclictest25403-21fschecks_count23:25:161
1059599318,20cyclictest17916-21cpuspeed_turbos21:55:140
1059599317,3cyclictest25308-21/usr/sbin/munin20:55:250
1059599316,6cyclictest0-21swapper/019:15:130
1059599315,12cyclictest23232-21munin-run00:35:010
1059599313,8cyclictest0-21swapper/022:25:040
1059599313,8cyclictest0-21swapper/022:25:040
1059599312,6cyclictest25941-21smart_sda19:40:250
1059599311,26cyclictest9918-21/usr/sbin/munin20:20:280
10595993112,16cyclictest31111-21seq23:36:500
10595993112,16cyclictest29085-21seq23:34:050
10595993110,18cyclictest836-21runrttasks23:08:500
10595993110,18cyclictest26985-21awk23:30:010
10595993110,18cyclictest23483-21sort22:05:280
10595993110,18cyclictest15774-21seq23:02:300
10595993110,18cyclictest14610-21sendmail_mailst21:45:270
10595993110,17cyclictest10102-21grep21:35:240
1060099301,17cyclictest11885-21latency_hist20:25:021
1059599309,3cyclictest836-21runrttasks23:43:060
1059599309,18cyclictest19958-21/usr/sbin/munin22:00:120
1059599307,9cyclictest30443-21kworker/0:022:30:240
1059599307,2cyclictest30335-21/usr/sbin/munin21:10:240
1059599301,26cyclictest22755-21sshd20:48:340
1059599301,14cyclictest9691-21munin-run20:20:010
10595993010,17cyclictest9376-21timerandwakeup00:00:260
10595993010,17cyclictest836-21runrttasks22:22:020
10595993010,17cyclictest836-21runrttasks22:22:020
10595993010,17cyclictest836-21runrttasks22:11:420
10595993010,17cyclictest836-21runrttasks19:56:140
1059599299,3cyclictest961-21cut21:15:180
1059599299,3cyclictest961-21cut21:15:180
1059599298,4cyclictest0-21swapper/023:20:260
1059599298,3cyclictest24384-21switchtime20:50:250
1059599296,3cyclictest4165-21latency20:05:220
1059599292,6cyclictest0-21swapper/020:30:160
10595992911,15cyclictest11175-21/usr/sbin/munin21:40:220
10595992910,16cyclictest7244-21sed22:45:020
279392280,2sleep327951-21/usr/sbin/munin19:45:243
1060099281,5cyclictest15072-21ntp_states23:00:211
1060099281,22cyclictest0-21swapper/122:49:531
1059599287,4cyclictest5620-21cpuspeed_turbos20:10:270
1059599287,4cyclictest5620-21cpuspeed_turbos20:10:270
1059599284,3cyclictest0-21swapper/023:10:260
10611992716,8cyclictest9745-21systemd-journal00:01:393
1060099271,22cyclictest9476-21timerwakeupswit00:00:261
1059599279,3cyclictest0-21swapper/019:50:400
1059599277,3cyclictest1253-21cstates20:00:140
1059599276,4cyclictest0-21swapper/019:45:260
1059599276,3cyclictest11520-21swap00:05:250
1059599271,5cyclictest4402-21/usr/sbin/munin23:50:200
1060799264,11cyclictest0-21swapper/223:25:142
10600992620,3cyclictest25354-21hddtemp_smartct19:40:171
10600992612,9cyclictest0-21swapper/123:09:221
1060099261,19cyclictest1941-21kworker/u8:021:53:391
10600992611,9cyclictest0-21swapper/119:34:481
10600992611,3cyclictest6103-21unixbench_singl21:25:271
10607992519,4cyclictest25040-21users00:35:272
1060099259,3cyclictest0-21swapper/119:16:081
1060099258,4cyclictest12761-21latency_hist19:15:021
1060099251,6cyclictest0-21swapper/122:36:461
1060099251,17cyclictest0-21swapper/120:47:361
1060099251,13cyclictest0-21swapper/121:48:261
10600992511,2cyclictest0-21swapper/121:20:171
10600992511,2cyclictest0-21swapper/121:20:171
1060099251,11cyclictest11576-21munin-run22:55:011
1061199247,7cyclictest0-21swapper/323:57:123
1060099242,16cyclictest0-21swapper/119:22:581
1060099242,12cyclictest0-21swapper/123:51:441
10600992421,2cyclictest0-21swapper/100:22:191
1060099241,6cyclictest0-21swapper/123:41:431
1060099241,21cyclictest0-21swapper/120:35:141
10600992411,8cyclictest0-21swapper/119:26:091
1060099241,17cyclictest31014-21kworker/u8:200:33:491
1060099241,17cyclictest0-21swapper/100:05:221
1060099241,16cyclictest0-21swapper/100:12:171
1060099241,14cyclictest0-21swapper/123:38:131
1060099241,12cyclictest19058-21/usr/sbin/munin00:25:141
1060099241,11cyclictest1592-21if_err_enp2s022:30:201
1060099240,22cyclictest0-21swapper/120:55:141
1061199231,6cyclictest7985-21ps21:30:263
1060799237,5cyclictest11577-21awk22:55:012
1060799236,9cyclictest86550irq/126-enp1s0-21:40:272
1060799236,9cyclictest5853-21cpu23:55:122
1060799234,6cyclictest5031-21kworker/2:021:13:192
1060799234,5cyclictest21300-21cpu00:30:132
1060799232,6cyclictest28530-21sh22:20:012
1060099232,15cyclictest0-21swapper/123:31:021
10600992318,3cyclictest0-21swapper/123:47:111
10600992315,3cyclictest0-21swapper/119:58:291
1060099231,20cyclictest0-21swapper/119:35:191
1060099231,17cyclictest0-21swapper/119:49:581
1060099231,16cyclictest0-21swapper/120:13:391
1060099231,16cyclictest0-21swapper/120:13:391
10600992311,3cyclictest0-21swapper/120:09:551
1060099231,11cyclictest0-21swapper/122:40:061
1060099231,11cyclictest0-21swapper/122:16:081
1060099231,10cyclictest24718-21diskstats22:10:151
1060099230,21cyclictest0-21swapper/121:35:211
1061199227,4cyclictest2246-21irqstats23:45:183
1060799227,9cyclictest0-21swapper/220:47:272
1060799226,9cyclictest0-21swapper/222:35:272
1060799221,16cyclictest0-21swapper/219:16:552
1060799221,12cyclictest0-21swapper/200:00:102
1060099229,2cyclictest21697-21open_inodes23:15:231
1060099222,8cyclictest18831-21ntp_states21:55:231
1060099221,8cyclictest18411-21/usr/sbin/munin23:10:171
1060099221,19cyclictest0-21swapper/121:13:281
1060099221,16cyclictest0-21swapper/120:19:221
1060099221,15cyclictest0-21swapper/121:06:221
1060099221,15cyclictest0-21swapper/121:00:051
1060099221,15cyclictest0-21swapper/120:28:271
1060099221,10cyclictest12149-21diskstats22:55:151
1060099220,20cyclictest0-21swapper/121:30:231
1060099220,20cyclictest0-21swapper/121:15:131
1060099220,20cyclictest0-21swapper/121:15:131
1061199213,15cyclictest32738-21grep19:55:303
1061199212,15cyclictest842-21lldpd19:20:183
1061199212,11cyclictest14634-21snmpd19:30:413
10611992115,3cyclictest31039-21latency_hist19:55:013
1061199211,12cyclictest13025-21gzip22:55:253
1060799216,2cyclictest0-21swapper/219:40:232
1060799213,6cyclictest30-21rcuc/222:00:192
1060799212,6cyclictest7735-21df22:45:152
1060799211,7cyclictest28792-21users23:30:302
1060799211,7cyclictest2773-21latency_hist22:35:022
1060799211,7cyclictest12613-21irqstats22:55:182
1060799211,7cyclictest0-21swapper/221:17:322
1060799211,7cyclictest0-21swapper/221:17:322
1060799210,16cyclictest0-21swapper/221:34:122
1060799210,15cyclictest0-21swapper/219:30:382
1060099217,2cyclictest6432-21gltestperf23:55:161
1060099216,4cyclictest23238-21ntp_states22:05:241
1060099212,10cyclictest0-21swapper/120:02:231
1060099211,9cyclictest29829-21ntp_states22:20:211
1060099211,9cyclictest29829-21ntp_states22:20:211
1060099211,15cyclictest0-21swapper/122:02:041
1060099211,15cyclictest0-21swapper/121:44:061
1060099211,15cyclictest0-21swapper/120:53:531
1060099211,15cyclictest0-21swapper/120:40:521
1060099211,15cyclictest0-21swapper/119:51:431
1060099211,11cyclictest19697-21kworker/u8:122:27:461
1060099211,11cyclictest19697-21kworker/u8:122:27:461
1061199206,6cyclictest0-21swapper/300:19:513
1061199206,5cyclictest0-21swapper/322:51:143
1061199206,4cyclictest0-21swapper/323:30:323
1061199206,4cyclictest0-21swapper/300:28:263
1061199204,6cyclictest0-21swapper/322:16:513
1061199203,14cyclictest21205-21grep23:15:173
1061199202,10cyclictest7401-21/usr/sbin/munin22:45:263
1061199202,10cyclictest30335-21/usr/sbin/munin21:10:203
1061199201,16cyclictest25815-21irqstats23:25:203
1061199201,13cyclictest0-21swapper/322:40:193
1061199201,11cyclictest973-21kernelversion21:15:183
1061199201,11cyclictest973-21kernelversion21:15:183
1061199201,11cyclictest0-21swapper/322:35:033
1061199201,10cyclictest0-21swapper/322:30:073
1061199200,9cyclictest0-21swapper/300:20:333
1060799207,8cyclictest0-21swapper/219:46:082
1060799207,3cyclictest5031-21kworker/2:023:50:272
1060799205,4cyclictest86550irq/126-enp1s0-20:20:172
1060799205,4cyclictest18886-21pmu-power21:55:252
1060799205,3cyclictest21365-21if_enp2s023:15:182
1060799202,4cyclictest21030-21munin-run00:30:012
1060799201,7cyclictest6469-21smart_sda22:40:302
1060799201,7cyclictest27314-21cat21:00:002
1060799201,6cyclictest17072-21unixbench_multi21:50:272
1060799201,6cyclictest12466-21/usr/sbin/munin00:10:242
10607992012,5cyclictest15230-21sendmail_mailqu23:00:232
1060099202,3cyclictest0-21swapper/100:38:411
1060099202,15cyclictest15005-21df_inode00:15:151
10600992016,3cyclictest0-21swapper/123:22:111
1060099201,18cyclictest0-21swapper/120:30:201
1061199199,3cyclictest17762-21/usr/sbin/munin21:55:183
1061199198,3cyclictest31598-21/usr/sbin/munin23:40:223
1061199194,7cyclictest780-21rngd00:06:103
1061199192,14cyclictest21890-21head00:30:173
1061199191,7cyclictest0-21swapper/323:05:363
1061199191,7cyclictest0-21swapper/322:11:333
1061199191,6cyclictest0-21swapper/300:11:153
1061199191,4cyclictest0-21swapper/322:08:573
10611991912,4cyclictest19361-21grep23:10:213
1061199191,10cyclictest13370-21/usr/sbin/munin21:45:183
1061199191,10cyclictest1085-21/usr/sbin/munin20:00:223
1061199190,9cyclictest19697-21kworker/u8:122:25:503
1061199190,9cyclictest19697-21kworker/u8:122:25:503
1060799197,3cyclictest0-21swapper/221:49:362
1060799195,7cyclictest86550irq/126-enp1s0-23:35:242
1060799193,9cyclictest17733-40dnf19:23:372
1060799193,4cyclictest29348-21fw_packets22:20:172
1060799193,4cyclictest29348-21fw_packets22:20:172
1060799193,4cyclictest1425-21df20:00:132
1060799192,5cyclictest30137-21ntp_states19:50:202
1060799191,7cyclictest0-21swapper/223:14:362
1060799191,7cyclictest0-21swapper/200:21:152
1060799191,6cyclictest17120-21memory23:05:182
1060799191,6cyclictest0-21swapper/220:10:272
1060799191,6cyclictest0-21swapper/220:10:272
1060799191,5cyclictest6663-21date21:30:022
1060799191,5cyclictest32586-21strings23:40:192
1060799191,5cyclictest2348-21memory23:45:182
1060799191,5cyclictest0-21swapper/220:25:162
1060799191,11cyclictest5031-21kworker/2:021:00:552
1060799191,10cyclictest14634-21snmpd21:21:002
1060799191,10cyclictest14634-21snmpd21:20:592
1060799191,10cyclictest0-21swapper/222:07:492
1060799190,14cyclictest0-21swapper/200:09:162
1061199187,8cyclictest25087-21munin-run20:55:023
1061199185,5cyclictest0-21swapper/300:35:033
1061199183,5cyclictest0-21swapper/321:52:003
1061199181,8cyclictest19958-21/usr/sbin/munin22:00:253
1061199181,6cyclictest0-21swapper/323:50:103
10611991813,3cyclictest11783-21grep21:40:213
10611991812,3cyclictest23361-21wc19:35:223
10611991811,4cyclictest848-21sssd_nss20:45:253
10611991811,4cyclictest0-21swapper/321:05:143
1061199181,11cyclictest0-21swapper/322:21:303
1061199181,11cyclictest0-21swapper/322:21:303
1061199181,10cyclictest0-21swapper/323:24:393
1061199181,10cyclictest0-21swapper/321:20:163
1061199181,10cyclictest0-21swapper/321:20:163
1061199181,10cyclictest0-21swapper/319:10:163
1060799184,3cyclictest86550irq/126-enp1s0-19:25:212
1060799184,3cyclictest4874-21unixbench_singl20:05:292
1060799184,3cyclictest10576-21sshd21:35:342
1060799183,4cyclictest0-21swapper/200:15:032
1060799182,4cyclictest31780-21irqstats22:25:192
1060799182,4cyclictest31780-21irqstats22:25:192
1060799181,5cyclictest30104-21diskmemload21:10:002
1060799181,5cyclictest20304-21users20:40:292
10607991812,4cyclictest17495-21needreboot20:35:202
204762171,2sleep30-21swapper/320:42:323
1061199176,3cyclictest86550irq/126-enp1s0-20:05:193
1061199171,8cyclictest14634-21snmpd19:25:333
1061199171,7cyclictest8070-21diskstats20:15:193
1061199171,7cyclictest14104-21cut23:00:123
1061199171,6cyclictest0-21swapper/323:35:083
1061199171,5cyclictest0-21swapper/321:36:033
1061199171,2cyclictest27112-21chrt20:58:053
10611991712,3cyclictest4984-21fschecks_count21:25:163
10611991712,3cyclictest15185-21latency20:30:223
10611991712,3cyclictest12203-21cat20:25:143
10611991711,4cyclictest842-21lldpd20:20:183
1061199171,14cyclictest0-21swapper/320:39:033
1060799173,11cyclictest23713-21tail20:50:152
1060799171,8cyclictest5031-21kworker/2:019:10:062
1060799171,6cyclictest829-21sh20:00:002
1060799171,5cyclictest0-21swapper/223:20:162
1060799171,4cyclictest30-21rcuc/219:35:202
10607991712,3cyclictest7721-21/usr/sbin/munin20:15:212
10611991611,3cyclictest14030-21df19:15:153
10611991610,1cyclictest0-21swapper/319:40:153
1060799160,7cyclictest0-21swapper/220:30:182
275372151,2sleep30-21swapper/321:00:383
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional