You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-03 - 07:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Jul 03, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1687422650,1sleep339-21rcuc/319:05:163
182112242165,25sleep20-21swapper/219:09:102
179842231162,56sleep00-21swapper/019:06:140
182102202165,24sleep10-21swapper/119:09:091
141382510,4sleep011-21rcuc/021:25:120
217892480,1sleep10-21swapper/100:10:121
217892480,1sleep10-21swapper/100:10:121
45062400,1sleep30-21swapper/319:50:123
1835899359,12cyclictest5490-21munin-run22:20:010
1836699331,9cyclictest0-21swapper/121:21:071
1836699321,9cyclictest0-21swapper/120:19:031
1836699321,9cyclictest0-21swapper/119:26:151
1836699321,9cyclictest0-21swapper/119:26:151
1835899325,12cyclictest26241-21awk20:40:010
1835899321,15cyclictest32499-21idleruntime-cro00:35:010
1836699311,11cyclictest0-21swapper/122:50:471
1836699310,11cyclictest0-21swapper/121:56:421
1835899302,15cyclictest2663-21munin-run21:00:010
1836699291,10cyclictest0-21swapper/100:08:551
1836699290,11cyclictest0-21swapper/121:30:231
1836699280,12cyclictest0-21swapper/122:47:151
1836699280,11cyclictest0-21swapper/122:24:191
1836699280,10cyclictest0-21swapper/122:33:551
1836699280,10cyclictest0-21swapper/120:26:191
18358992811,14cyclictest30639-21perf20:50:010
1835899280,12cyclictest13669-21awk21:25:010
18378992714,3cyclictest18245-1kworker/u9:222:19:583
1836699271,9cyclictest0-21swapper/119:19:511
1836699271,21cyclictest0-21swapper/123:11:071
1836699271,11cyclictest808-21snmpd21:37:311
1836699271,10cyclictest26125-21/usr/sbin/munin21:50:231
1836699270,9cyclictest0-21swapper/119:55:421
1836699270,12cyclictest0-21swapper/123:40:351
1836699270,12cyclictest0-21swapper/123:34:591
1836699270,10cyclictest0-21swapper/121:15:251
1835899270,13cyclictest16484-21perf22:45:010
305202261,3sleep10-21swapper/100:30:021
1836699263,7cyclictest0-21swapper/120:45:291
1836699261,9cyclictest0-21swapper/120:54:271
1836699261,8cyclictest0-21swapper/122:57:231
1836699261,11cyclictest0-21swapper/122:14:551
1836699261,11cyclictest0-21swapper/120:57:511
1836699261,10cyclictest0-21swapper/123:01:471
1836699261,10cyclictest0-21swapper/123:01:471
1836699260,9cyclictest0-21swapper/119:40:151
1836699260,5cyclictest0-21swapper/121:25:431
1836699260,20cyclictest0-21swapper/100:25:471
1836699260,11cyclictest0-21swapper/122:43:231
1836699260,11cyclictest0-21swapper/100:36:071
1836699260,10cyclictest0-21swapper/100:00:231
1835899269,14cyclictest368-21perf20:55:010
18371992519,4cyclictest8728-21/usr/sbin/munin22:25:182
1837199251,6cyclictest0-21swapper/222:06:352
1837199251,6cyclictest0-21swapper/200:33:542
1836699251,9cyclictest0-21swapper/122:36:351
1836699251,12cyclictest8196-21sshd23:38:511
1836699251,12cyclictest0-21swapper/120:06:471
1836699251,10cyclictest0-21swapper/120:31:251
1836699250,9cyclictest0-21swapper/121:04:071
1836699250,9cyclictest0-21swapper/100:20:591
1836699250,6cyclictest0-21swapper/119:36:591
1836699250,16cyclictest0-21swapper/121:10:431
1836699250,11cyclictest0-21swapper/123:47:431
1836699250,10cyclictest0-21swapper/123:52:431
1836699250,10cyclictest0-21swapper/123:20:351
1836699250,10cyclictest0-21swapper/122:19:191
1835899257,15cyclictest20879-21munin-run22:55:010
225622240,3sleep00-21swapper/022:55:270
1837899242,12cyclictest26659-40dnf21:52:483
1837899241,16cyclictest0-21swapper/322:01:193
1837199241,6cyclictest0-21swapper/223:29:152
1836699248,3cyclictest0-21swapper/120:40:051
1836699248,3cyclictest0-21swapper/120:40:051
1836699242,12cyclictest10858-21munin-run20:05:011
1836699241,9cyclictest0-21swapper/123:56:071
1836699241,9cyclictest0-21swapper/119:31:491
1836699241,14cyclictest0-21swapper/123:15:031
1836699241,13cyclictest0-21swapper/122:00:071
1836699241,11cyclictest26599-21open_inodes23:05:211
1836699241,10cyclictest0-21swapper/120:36:471
1836699240,9cyclictest0-21swapper/121:41:111
94532231,2sleep30-21swapper/320:00:163
1837899231,7cyclictest0-21swapper/323:58:593
18378992315,5cyclictest28425-21dump-pmu-power20:45:013
18378992315,5cyclictest28425-21dump-pmu-power20:45:013
1837199235,6cyclictest0-21swapper/222:37:302
1837199231,7cyclictest0-21swapper/221:16:032
1837199231,6cyclictest0-21swapper/223:17:482
1837199231,6cyclictest0-21swapper/222:32:532
1836699236,3cyclictest0-21swapper/119:21:131
1836699231,9cyclictest0-21swapper/120:10:531
1836699231,9cyclictest0-21swapper/119:50:091
1836699231,9cyclictest0-21swapper/119:11:171
1836699231,11cyclictest0-21swapper/120:24:331
1836699230,8cyclictest0-21swapper/121:06:151
1836699230,8cyclictest0-21swapper/100:19:111
1836699230,15cyclictest0-21swapper/122:25:271
1836699230,10cyclictest0-21swapper/121:45:191
9852220,2sleep20-21swapper/220:55:122
196692221,3sleep30-21swapper/321:35:283
1837899221,7cyclictest0-21swapper/300:10:373
1837899221,7cyclictest0-21swapper/300:10:373
1837899221,6cyclictest0-21swapper/321:20:323
1837899221,12cyclictest0-21swapper/300:31:103
1837199223,4cyclictest0-21swapper/223:35:372
1837199221,6cyclictest0-21swapper/223:55:122
1837199221,6cyclictest0-21swapper/222:46:482
1837199221,6cyclictest0-21swapper/219:40:262
1837199221,2cyclictest0-21swapper/220:26:472
1836699220,10cyclictest0-21swapper/119:45:111
1835899224,10cyclictest0-21swapper/023:28:450
1835899221,5cyclictest0-21swapper/021:57:130
1835899221,5cyclictest0-21swapper/021:53:200
12922221,2sleep21342-21cpu22:10:102
1837899216,5cyclictest28241-21proc_pri21:55:233
18378992115,3cyclictest6239-21grep22:20:163
1837899211,4cyclictest0-21swapper/300:09:143
1837899211,10cyclictest0-21swapper/321:28:293
1837199215,4cyclictest0-21swapper/200:04:162
1837199212,4cyclictest0-21swapper/200:25:282
1837199211,6cyclictest0-21swapper/221:55:372
1837199211,5cyclictest0-21swapper/200:17:392
1835899211,9cyclictest0-21swapper/020:30:190
1835899211,14cyclictest0-21swapper/023:34:510
1835899211,12cyclictest26139-21/usr/sbin/munin00:20:160
1835899211,12cyclictest0-21swapper/000:15:170
1835899211,11cyclictest808-21snmpd19:15:310
1835899211,11cyclictest12301-21/usr/sbin/munin22:35:120
1835899211,10cyclictest827-21runrttasks23:10:210
1835899210,11cyclictest0-21swapper/023:56:300
1837899207,8cyclictest0-21swapper/319:39:143
1837899206,6cyclictest18245-1kworker/u9:222:31:543
1837899204,3cyclictest3894-21latency_hist23:30:023
1837899201,9cyclictest13308-21fschecks_count23:50:163
1837899201,6cyclictest0-21swapper/323:40:303
1837899201,4cyclictest23669-21grep21:45:193
1837899201,4cyclictest0-21swapper/300:38:353
1837899201,3cyclictest0-21swapper/323:23:313
1837899201,3cyclictest0-21swapper/323:09:113
1837899201,3cyclictest0-21swapper/300:21:363
1837199207,2cyclictest0-21swapper/220:02:192
1837199201,7cyclictest0-21swapper/222:58:062
1837199201,6cyclictest0-21swapper/223:53:182
1837199201,6cyclictest0-21swapper/223:10:122
1837199201,6cyclictest0-21swapper/221:28:562
1837199201,5cyclictest0-21swapper/219:20:262
1837199201,3cyclictest0-21swapper/222:16:092
18371992012,5cyclictest827-21runrttasks21:45:142
1836699200,15cyclictest0-21swapper/123:29:211
1835899203,3cyclictest0-21swapper/023:36:290
1835899202,2cyclictest2993-21kworker/u8:122:20:200
1835899202,10cyclictest17315-21/usr/sbin/munin00:00:120
1835899201,11cyclictest32094-21/usr/sbin/munin23:20:160
1835899201,11cyclictest19865-21/usr/sbin/munin20:25:160
1835899201,11cyclictest0-21swapper/023:17:420
1835899201,11cyclictest0-21swapper/023:05:160
1835899201,11cyclictest0-21swapper/023:03:260
1835899201,11cyclictest0-21swapper/023:03:260
1835899201,11cyclictest0-21swapper/022:05:260
1835899201,10cyclictest1293-21/usr/sbin/munin22:10:130
1835899200,3cyclictest0-21swapper/021:30:130
103552201,2sleep20-21swapper/223:43:212
297902191,2sleep20-21swapper/219:32:182
217062190,4sleep3401ktimersoftd/321:40:243
1837899199,3cyclictest808-21snmpd19:11:463
1837899196,3cyclictest9494-21/usr/sbin/munin21:15:283
1837899195,7cyclictest25204-21smart_sda00:15:253
1837899195,4cyclictest25415-21sshd19:22:293
1837899194,2cyclictest3996-21kworker/3:222:35:203
1837899192,3cyclictest87650irq/129-enp1s0-20:20:193
1837899191,9cyclictest16428-21diskstats21:30:153
1837899191,9cyclictest0-21swapper/320:58:503
1837899191,6cyclictest6693-21fschecks_count23:35:153
1837899191,6cyclictest32707-21/usr/sbin/munin22:05:223
1837899191,4cyclictest0-21swapper/322:44:263
1837899191,4cyclictest0-21swapper/321:13:003
18378991914,3cyclictest0-21swapper/320:10:243
1837899191,2cyclictest0-21swapper/323:15:133
18378991912,4cyclictest7093-21grep19:55:143
1837199194,4cyclictest25324-21ntp_states20:35:212
1837199194,3cyclictest87350irq/126-enp1s0-21:13:402
1837199193,7cyclictest30-21rcuc/219:15:122
1837199193,6cyclictest22064-21/usr/sbin/munin20:30:202
1837199193,4cyclictest0-21swapper/223:49:052
1837199193,13cyclictest26187-21wc21:50:242
1837199191,6cyclictest2275-21/usr/sbin/munin19:45:222
1837199191,6cyclictest0-21swapper/223:03:212
1837199191,6cyclictest0-21swapper/223:03:212
1837199191,6cyclictest0-21swapper/221:20:162
1837199191,5cyclictest0-21swapper/222:40:122
1837199191,4cyclictest0-21swapper/223:33:302
1837199191,4cyclictest0-21swapper/221:30:192
1837199191,3cyclictest19624-21ntpq19:10:222
18371991913,3cyclictest25977-21tr19:25:112
18371991913,3cyclictest25977-21tr19:25:112
1837199191,2cyclictest87350irq/126-enp1s0-00:35:042
18371991912,4cyclictest32220-21idleruntime-cro19:40:012
1837199191,10cyclictest0-21swapper/222:21:082
1836699194,3cyclictest0-21swapper/122:06:421
1835899198,3cyclictest87350irq/126-enp1s0-23:50:210
1835899198,2cyclictest4552-21kworker/0:121:05:250
1835899197,4cyclictest0-21swapper/000:05:570
1835899194,4cyclictest0-21swapper/022:27:510
1835899193,6cyclictest87350irq/126-enp1s0-20:05:120
1835899191,4cyclictest810-21sshd22:31:460
18358991913,3cyclictest8170-21unixbench_multi19:55:250
1835899191,11cyclictest0-21swapper/022:45:280
1835899191,10cyclictest28342-21/usr/sbin/munin00:25:230
1835899191,10cyclictest26454-21/usr/sbin/munin20:40:220
1835899191,10cyclictest26454-21/usr/sbin/munin20:40:220
1835899191,10cyclictest19985-21sshd21:36:460
1835899191,10cyclictest0-21swapper/022:01:590
1837899185,7cyclictest2993-21kworker/u8:122:52:223
1837899185,3cyclictest22221-21cut22:55:233
1837899184,8cyclictest7067-21diskmemload00:27:583
1837899182,5cyclictest8726-21irqstats22:25:183
1837899181,9cyclictest23540-21grep20:30:263
1837899181,6cyclictest0-21swapper/323:12:373
1837899181,5cyclictest4460-21df23:30:133
1837899181,5cyclictest1661-21diskstats22:10:123
1837899181,3cyclictest0-21swapper/323:46:443
1837899181,3cyclictest0-21swapper/322:47:273
1837899181,10cyclictest0-21swapper/321:05:143
1837199187,3cyclictest87650irq/129-enp1s0-20:05:152
1837199186,3cyclictest20130-21gzip22:50:242
1837199185,4cyclictest27625-21ps20:40:212
1837199185,4cyclictest27625-21ps20:40:212
1837199182,4cyclictest0-21swapper/223:05:142
1837199181,9cyclictest0-21swapper/223:21:262
1837199181,6cyclictest0-21swapper/222:01:302
1837199181,6cyclictest0-21swapper/221:37:032
1837199181,6cyclictest0-21swapper/200:05:452
1837199181,5cyclictest0-21swapper/221:40:162
1837199181,5cyclictest0-21swapper/200:25:012
1837199181,11cyclictest0-21swapper/200:10:142
1837199181,11cyclictest0-21swapper/200:10:142
1835899184,3cyclictest0-21swapper/000:35:160
1835899181,8cyclictest87350irq/126-enp1s0-00:10:470
1835899181,8cyclictest87350irq/126-enp1s0-00:10:470
1835899181,6cyclictest0-21swapper/023:40:270
1835899181,4cyclictest0-21swapper/019:45:200
18358991812,3cyclictest21155-21hddtemp_smartct21:40:150
1835899181,10cyclictest17664-21/usr/sbin/munin20:20:130
1835899181,10cyclictest0-21swapper/021:46:080
1835899180,8cyclictest5976-21kworker/0:023:47:530
1835899180,16cyclictest11467-21munin-run21:20:020
170872181,2sleep30-21swapper/320:15:423
61632171,2sleep00-21swapper/019:51:350
1837899175,3cyclictest12779-21sshd20:06:463
1837899172,12cyclictest808-21snmpd19:45:363
1837899171,8cyclictest24295-21strings23:00:263
1837899171,8cyclictest24295-21strings23:00:253
1837899171,5cyclictest28391-21diskstats19:30:143
1837899171,5cyclictest18918-21users00:00:273
1837899171,14cyclictest0-21swapper/320:28:483
1837899171,13cyclictest839-21lldpd19:25:143
1837899171,13cyclictest839-21lldpd19:25:143
18378991711,3cyclictest2884-21/usr/sbin/munin21:00:203
1837199172,12cyclictest19639-21turbostat20:25:012
18371991712,3cyclictest5686-21grep21:05:172
1837199170,7cyclictest15853-21fschecks_count20:15:132
1835899177,2cyclictest25849-21/usr/sbin/munin19:25:170
1835899177,2cyclictest25849-21/usr/sbin/munin19:25:160
1835899177,2cyclictest15467-21/usr/sbin/munin20:15:150
1835899176,2cyclictest0-21swapper/021:00:200
1835899171,9cyclictest808-21snmpd20:00:400
1835899171,4cyclictest808-21snmpd21:10:530
1835899171,4cyclictest32435-21/usr/sbin/munin19:40:220
18358991711,3cyclictest30585-21sh19:35:120
1835899171,13cyclictest28044-21/usr/sbin/munin19:30:170
18358991711,3cyclictest19232-21cut19:10:170
17652171,3sleep30-21swapper/319:41:343
245682160,3sleep011-21rcuc/019:20:190
1837899161,4cyclictest21088-21diskstats19:15:033
18378991611,3cyclictest30849-21/usr/sbin/munin20:50:143
18378991611,3cyclictest28654-21/usr/sbin/munin20:45:213
18378991610,3cyclictest24917-21gltestperf20:35:163
1837199162,7cyclictest0-21swapper/221:02:212
1837199162,6cyclictest13269-21/usr/sbin/munin20:10:192
1837199161,7cyclictest0-21swapper/220:46:202
18371991612,3cyclictest0-21swapper/219:50:162
18371991611,3cyclictest7520-21latency19:55:182
18371991611,3cyclictest31202-21entropy20:50:142
151892150,2sleep00-21swapper/020:14:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional