You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 10:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Nov 22, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
62382251165,23sleep30-21swapper/319:07:273
61222237162,24sleep00-21swapper/019:05:580
62042236164,58sleep10-21swapper/119:07:001
63602233164,55sleep20-21swapper/219:09:032
3231921330,4sleep3653999cyclictest22:30:543
6532996258,3cyclictest0-21swapper/221:04:582
211632530,2sleep00-21swapper/020:55:020
102042530,2sleep20-21swapper/222:55:192
129302510,1sleep30-21swapper/320:35:183
129302510,1sleep30-21swapper/320:35:183
185932490,1sleep30-21swapper/323:15:143
168812490,1sleep316882-21cpu20:45:133
315362380,1sleep30-21swapper/323:45:123
652699343,16cyclictest27968-40dnf22:21:341
652699305,7cyclictest13425-40dnf19:20:341
6526993017,3cyclictest21016-21munin-run20:55:011
134302291,4sleep00-21swapper/019:20:330
651899285,15cyclictest10827-21sed22:55:260
279732281,4sleep00-21swapper/022:21:340
652699261,18cyclictest0-21swapper/119:59:251
6518992610,14cyclictest2502-21perf22:40:010
653299251,7cyclictest780-21rngd00:21:502
6526992514,3cyclictest0-21swapper/119:45:191
652699251,16cyclictest27618-21diskmemload00:26:431
652699251,16cyclictest0-21swapper/120:35:311
652699251,16cyclictest0-21swapper/120:35:311
6526992511,3cyclictest16345-21munin-run00:25:021
652699251,11cyclictest4679-21sh22:45:011
652699250,17cyclictest0-21swapper/119:25:191
653999243,12cyclictest429-21basename22:35:123
653999242,10cyclictest25008-21/usr/sbin/munin19:50:133
653299241,11cyclictest2490-21munin-run22:40:012
652699241,3cyclictest9202-21awk20:25:221
652699241,16cyclictest86850irq/129-enp1s0-22:15:181
652699241,16cyclictest0-21swapper/121:41:511
652699241,16cyclictest0-21swapper/121:41:511
653999239,3cyclictest23437-21/usr/sbin/munin21:00:123
653999232,11cyclictest24715-21/usr/sbin/munin23:30:153
653999231,11cyclictest9377-21sshd00:05:343
653999231,11cyclictest26278-21/usr/sbin/munin22:20:183
653999231,11cyclictest24210-21NetworkManager23:26:363
653999231,11cyclictest12795-21sshd19:20:203
653999231,10cyclictest19472-21ntp_states19:35:193
653299236,9cyclictest24789-21awk19:50:012
652699235,5cyclictest0-21swapper/119:15:261
652699232,16cyclictest0-21swapper/120:02:171
652699232,13cyclictest86850irq/129-enp1s0-22:48:121
652699232,13cyclictest86850irq/129-enp1s0-22:48:121
652699231,9cyclictest0-21swapper/100:35:021
652699231,8cyclictest0-21swapper/123:43:161
6526992312,4cyclictest0-21swapper/100:10:251
652699231,16cyclictest0-21swapper/122:51:141
652699231,16cyclictest0-21swapper/121:13:091
652699231,15cyclictest0-21swapper/123:25:061
652699231,11cyclictest768-21in:imjournal20:50:021
6526992310,3cyclictest14147-21munin-run00:20:011
652699230,17cyclictest0-21swapper/123:10:301
6518992310,3cyclictest30186-21ntp_states23:40:240
144942230,1sleep10-21swapper/121:50:261
653999224,3cyclictest86850irq/129-enp1s0-22:15:243
653999221,11cyclictest17476-21/usr/sbin/munin22:00:143
653299224,3cyclictest86550irq/126-enp1s0-00:05:252
6532992219,2cyclictest0-21swapper/200:31:552
652699223,7cyclictest20490-21seq23:20:121
6526992214,3cyclictest0-21swapper/123:32:431
6526992214,3cyclictest0-21swapper/120:10:401
652699221,15cyclictest0-21swapper/123:05:061
652699221,15cyclictest0-21swapper/120:55:371
652699221,15cyclictest0-21swapper/120:15:071
652699221,15cyclictest0-21swapper/119:37:081
652699221,15cyclictest0-21swapper/100:09:071
652699221,13cyclictest0-21swapper/122:26:241
652699221,11cyclictest22584-21dump-pmu-power19:45:011
652699220,6cyclictest0-21swapper/120:06:211
652699220,16cyclictest0-21swapper/121:04:561
652699220,16cyclictest0-21swapper/120:43:581
652699220,16cyclictest0-21swapper/100:01:201
651899229,3cyclictest14634-21snmpd21:25:170
651899223,13cyclictest18300-21kworker/u8:000:31:560
651899221,8cyclictest22033-21ntp_states00:35:230
653999219,2cyclictest0-21swapper/321:30:243
653999216,3cyclictest14634-21snmpd23:51:133
653999216,3cyclictest14634-21snmpd23:51:133
653999215,4cyclictest29761-21sshd21:14:263
653999213,4cyclictest30633-21gltestperf21:15:183
653999211,9cyclictest0-21swapper/322:28:033
653999211,6cyclictest12188-21hddtemp_smartct23:00:173
653999211,6cyclictest0-21swapper/323:39:563
653999211,5cyclictest0-21swapper/323:57:323
653999211,13cyclictest0-21swapper/322:55:513
653999211,10cyclictest22806-21/usr/sbin/munin19:45:213
653999211,10cyclictest0-21swapper/320:18:013
653999211,10cyclictest0-21swapper/320:04:303
653299211,7cyclictest0-21swapper/221:25:562
653299211,6cyclictest0-21swapper/221:41:182
653299211,6cyclictest0-21swapper/221:41:182
653299211,5cyclictest0-21swapper/223:45:152
653299211,5cyclictest0-21swapper/223:20:462
653299211,5cyclictest0-21swapper/200:37:342
653299211,13cyclictest0-21swapper/223:55:132
653299211,11cyclictest6834-21diskstats21:35:152
652699219,3cyclictest18468-21diskstats23:15:141
652699215,5cyclictest27618-21diskmemload21:30:221
652699212,3cyclictest0-21swapper/122:35:061
652699212,17cyclictest0-21swapper/122:08:151
652699212,10cyclictest8929-21munin-node19:50:291
652699211,9cyclictest0-21swapper/121:55:021
652699211,9cyclictest0-21swapper/121:26:241
652699211,8cyclictest0-21swapper/122:32:311
652699211,7cyclictest27254-21sh23:35:141
652699211,3cyclictest0-21swapper/121:35:141
6526992113,3cyclictest0-21swapper/121:15:061
6526992113,3cyclictest0-21swapper/100:38:001
652699211,16cyclictest0-21swapper/123:45:181
652699211,15cyclictest0-21swapper/123:51:491
652699211,15cyclictest0-21swapper/123:51:491
652699211,15cyclictest0-21swapper/121:07:171
652699211,15cyclictest0-21swapper/120:21:331
652699211,15cyclictest0-21swapper/119:34:001
652699211,15cyclictest0-21swapper/119:34:001
652699211,14cyclictest0-21swapper/122:14:101
652699211,11cyclictest10623-21grep22:55:251
652699211,10cyclictest581-21systemd-journal22:05:011
652699211,10cyclictest3941-21grep23:55:141
651899217,4cyclictest0-21swapper/021:37:280
651899214,9cyclictest27618-21diskmemload22:03:470
651899213,8cyclictest0-21swapper/021:30:060
651899211,5cyclictest0-21swapper/023:49:110
651899211,11cyclictest6724-21/usr/sbin/munin19:10:230
651899211,11cyclictest4913-21/usr/sbin/munin22:45:190
651899211,11cyclictest4913-21/usr/sbin/munin22:45:190
651899211,11cyclictest31611-21/usr/sbin/munin20:05:160
651899211,11cyclictest30049-21/usr/sbin/munin21:15:280
651899211,11cyclictest27618-21diskmemload00:15:020
651899211,11cyclictest0-21swapper/023:15:250
651899210,9cyclictest87950irq/132-enp2s0-00:21:500
280122210,4sleep2653299cyclictest21:10:122
653999208,2cyclictest0-21swapper/320:23:323
653999207,2cyclictest0-21swapper/323:23:243
653999203,4cyclictest893-21sshd20:06:213
653999203,4cyclictest565-21/usr/sbin/munin21:20:193
653999201,9cyclictest0-21swapper/323:41:133
653999201,7cyclictest0-21swapper/300:00:163
653999201,6cyclictest0-21swapper/322:13:573
653999201,6cyclictest0-21swapper/300:23:083
6539992016,2cyclictest15295-21switchtime19:25:263
653999201,5cyclictest13839-21cpuspeed_turbos23:05:113
653999201,18cyclictest0-21swapper/320:51:023
653999201,10cyclictest0-21swapper/320:11:013
653299205,8cyclictest17824-21sensors_temp00:25:232
653299204,9cyclictest3325-21gltestperf22:40:172
653299204,7cyclictest13743-21egrep23:05:112
653299204,4cyclictest11943-21ntp_states21:45:202
653299202,6cyclictest14634-21snmpd23:51:142
653299202,6cyclictest14634-21snmpd23:51:142
653299202,5cyclictest3980-21df20:15:132
653299201,8cyclictest0-21swapper/221:33:002
653299201,7cyclictest4913-21/usr/sbin/munin22:45:162
653299201,7cyclictest4913-21/usr/sbin/munin22:45:162
653299201,7cyclictest30049-21/usr/sbin/munin21:15:182
653299201,7cyclictest27204-21/usr/sbin/munin19:55:222
653299201,7cyclictest14647-21/usr/sbin/munin20:40:102
653299201,6cyclictest0-21swapper/223:05:012
653299201,5cyclictest0-21swapper/200:12:232
653299201,4cyclictest30293-21proc_pri23:40:262
653299201,4cyclictest0-21swapper/221:53:542
653299201,10cyclictest0-21swapper/200:15:152
6526992016,3cyclictest0-21swapper/119:11:001
652699200,15cyclictest0-21swapper/123:01:011
652699200,15cyclictest0-21swapper/121:21:181
652699200,15cyclictest0-21swapper/120:30:141
651899209,3cyclictest14634-21snmpd19:34:570
651899209,3cyclictest14634-21snmpd19:34:570
651899209,3cyclictest13722-21/usr/sbin/munin23:05:160
651899208,3cyclictest18410-21/usr/sbin/munin19:35:220
651899204,7cyclictest7683-21sort00:05:020
651899201,9cyclictest0-21swapper/023:26:390
651899201,8cyclictest0-21swapper/020:20:170
651899201,7cyclictest0-21swapper/022:15:230
651899201,5cyclictest1860-21awk21:25:010
651899201,4cyclictest0-21swapper/019:25:170
651899201,11cyclictest14647-21/usr/sbin/munin20:40:180
651899201,11cyclictest1121-21/usr/sbin/munin23:50:210
651899201,11cyclictest1121-21/usr/sbin/munin23:50:210
651899201,10cyclictest27204-21/usr/sbin/munin19:55:110
651899201,10cyclictest15279-21/usr/sbin/munin21:55:190
651899201,10cyclictest0-21swapper/023:34:580
651899201,10cyclictest0-21swapper/022:52:320
651899201,10cyclictest0-21swapper/022:40:030
651899201,10cyclictest0-21swapper/022:30:220
651899201,10cyclictest0-21swapper/022:06:290
651899201,10cyclictest0-21swapper/000:15:340
272822200,3sleep00-21swapper/023:35:150
653999193,2cyclictest86850irq/129-enp1s0-20:40:083
653999191,9cyclictest0-21swapper/300:35:203
653999191,7cyclictest0-21swapper/322:50:533
653999191,6cyclictest15279-21/usr/sbin/munin21:55:223
653999191,6cyclictest0-21swapper/323:14:233
653999191,6cyclictest0-21swapper/322:48:413
653999191,6cyclictest0-21swapper/322:48:413
653999191,5cyclictest0-21swapper/321:47:253
653999191,15cyclictest12520-21df00:15:143
653999190,9cyclictest0-21swapper/320:58:323
653999190,10cyclictest0-21swapper/321:50:173
653299195,4cyclictest26761-21sshd19:52:212
653299193,3cyclictest86850irq/129-enp1s0-23:25:262
653299192,3cyclictest86550irq/126-enp1s0-22:10:122
653299191,9cyclictest3111-21sshd20:11:102
653299191,5cyclictest24078-21/usr/sbin/munin22:15:132
653299191,5cyclictest19821-21sshd23:16:412
653299191,4cyclictest0-21swapper/221:24:242
653299191,10cyclictest0-21swapper/200:03:432
652699191,6cyclictest0-21swapper/121:46:051
651899199,2cyclictest27618-21diskmemload00:25:170
651899196,3cyclictest0-21swapper/023:01:090
651899195,3cyclictest31393-21awk20:05:020
651899194,3cyclictest12050irq/123-ahci[0020:49:560
651899193,4cyclictest25736-21cpuspeed_turbos21:05:120
651899191,10cyclictest27618-21diskmemload21:48:000
651899191,10cyclictest22996-21sshd20:57:250
651899191,10cyclictest14634-21snmpd20:35:070
651899191,10cyclictest14634-21snmpd20:35:070
651899191,10cyclictest0-21swapper/023:24:460
653999183,3cyclictest86850irq/129-enp1s0-22:40:053
653999183,2cyclictest86850irq/129-enp1s0-19:34:563
653999183,2cyclictest86850irq/129-enp1s0-19:34:563
653999181,6cyclictest0-21swapper/322:07:183
653999181,5cyclictest2997-21memory21:25:203
653999181,5cyclictest0-21swapper/321:40:083
653999181,5cyclictest0-21swapper/321:40:083
6539991813,2cyclictest17196-21gltestperf00:25:143
653999181,2cyclictest86850irq/129-enp1s0-00:10:183
6539991812,4cyclictest7111-21grep21:35:183
6539991812,3cyclictest86850irq/129-enp1s0-20:30:193
6539991812,3cyclictest27771-21cut19:55:143
653299184,3cyclictest86850irq/129-enp1s0-23:35:132
653299182,9cyclictest86850irq/129-enp1s0-22:50:162
653299181,8cyclictest0-21swapper/223:30:142
653299181,6cyclictest0-21swapper/222:06:032
653299181,6cyclictest0-21swapper/221:55:272
653299181,5cyclictest17476-21/usr/sbin/munin22:00:132
653299181,5cyclictest0-21swapper/222:28:002
6532991813,3cyclictest8147-21grep20:25:112
653299181,14cyclictest19965-21grep19:35:252
653299181,14cyclictest16562-21cut23:10:202
651899188,2cyclictest10252-21/usr/sbin/munin20:30:150
651899186,5cyclictest9367-21sed21:40:160
651899186,5cyclictest9367-21sed21:40:160
651899186,3cyclictest20615-21/usr/sbin/munin19:40:160
651899185,3cyclictest86750irq/128-enp1s0-21:50:230
651899185,3cyclictest28569-21basename21:10:170
651899182,4cyclictest11-21rcuc/019:15:260
651899181,8cyclictest14634-21snmpd22:25:290
651899181,6cyclictest25893-21latency19:50:190
651899181,6cyclictest0-21swapper/023:10:370
651899181,4cyclictest14634-21snmpd20:25:060
651899181,2cyclictest86550irq/126-enp1s0-00:05:480
651899181,10cyclictest0-21swapper/022:11:070
52152181,2sleep00-21swapper/023:58:560
115472181,2sleep30-21swapper/319:19:403
653999171,4cyclictest0-21swapper/300:31:463
6539991713,3cyclictest0-21swapper/319:40:023
6539991712,3cyclictest27616-21latency_hist21:10:023
6539991711,3cyclictest6890-21expr19:10:113
653299172,3cyclictest5750irq/9-acpi20:55:002
653299172,3cyclictest20615-21/usr/sbin/munin19:40:272
653299171,5cyclictest0-21swapper/222:33:432
653299171,4cyclictest0-21swapper/220:07:052
653299171,3cyclictest86550irq/126-enp1s0-21:08:252
653299171,3cyclictest27619-21awk22:20:242
653299171,3cyclictest14114-21sshd20:36:132
653299171,3cyclictest14114-21sshd20:36:132
653299171,15cyclictest0-21swapper/220:02:192
653299171,14cyclictest0-21swapper/219:20:012
6532991711,3cyclictest5897-21cpu20:20:152
6532991711,3cyclictest10402-21grep20:30:112
651899175,2cyclictest23797-21needreboot19:45:190
6518991711,3cyclictest23911-21sort21:00:160
6518991710,4cyclictest5554-21sshd20:19:230
6539991611,3cyclictest8127-21cat20:25:123
653299169,4cyclictest848-21sssd_nss19:15:032
653299163,3cyclictest168822cpu20:45:132
653299162,3cyclictest86550irq/126-enp1s0-19:10:122
653299161,3cyclictest0-21swapper/219:26:502
6532991611,3cyclictest16218-21/usr/sbin/munin19:30:142
6532991611,3cyclictest16218-21/usr/sbin/munin19:30:142
6518991610,3cyclictest836-21runrttasks20:10:060
6532991510,3cyclictest21305-21ls20:55:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional