You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-06 - 12:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Aug 06, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
266302234163,58sleep10-21swapper/119:08:201
264762232163,56sleep20-21swapper/219:06:192
264202203165,25sleep30-21swapper/319:05:353
265162202165,25sleep00-21swapper/019:06:500
1158621350,1sleep30-21swapper/323:25:223
26884998582,2cyclictest0-21swapper/221:58:022
235272690,4sleep12687799cyclictest23:55:021
159032650,3sleep339-21rcuc/319:55:163
144562630,2sleep00-21swapper/022:20:130
19292520,1sleep20-21swapper/220:35:252
287882490,5sleep1579-21systemd-journal21:40:011
321092470,1sleep10-21swapper/121:45:191
26884994611,2cyclictest22821-21awk22:40:012
51872420,1sleep00-21swapper/020:45:130
2687299388,14cyclictest16838-21munin-run23:40:010
26884993613,20cyclictest7632-21grep00:30:202
26884993612,3cyclictest1681-21/usr/sbin/munin23:05:202
26884993511,3cyclictest25850-21/usr/sbin/munin00:00:252
2688499349,3cyclictest15830-21/usr/sbin/munin21:10:162
2688499341,5cyclictest827-21runrttasks20:44:262
26884993411,3cyclictest28077-21/usr/sbin/munin00:05:252
26884993410,3cyclictest27072-21/usr/sbin/munin19:10:212
2687299346,13cyclictest31612-21munin-run23:00:020
2688499338,3cyclictest0-21swapper/223:55:142
26884993330,2cyclictest0-21swapper/221:21:552
26884993330,2cyclictest0-21swapper/221:21:552
2688499328,3cyclictest7657-21/usr/sbin/munin22:05:252
2688499326,5cyclictest909-21sssd_nss23:25:192
2688499321,8cyclictest0-21swapper/222:10:162
2688499321,3cyclictest0-21swapper/223:10:262
2688499321,15cyclictest0-21swapper/223:35:502
26884993211,4cyclictest6476-21sa100:30:012
26884993210,3cyclictest28826-21cat21:40:012
2688499319,3cyclictest28954-21sed22:50:252
2688499318,4cyclictest0-21swapper/220:10:172
2688499318,4cyclictest0-21swapper/220:10:172
2688499318,2cyclictest4214-21/usr/sbin/munin19:30:222
2688499311,6cyclictest8995-21latency_hist20:55:012
2688499311,6cyclictest1525-21timerwakeupswit00:15:252
2688499311,6cyclictest12664-21/usr/sbin/munin23:30:142
2688499311,15cyclictest0-21swapper/222:49:082
71102300,2sleep00-21swapper/019:35:150
2688499308,4cyclictest24164-21ls22:40:222
2688499307,4cyclictest12310-21grep19:45:252
2688499306,3cyclictest12050irq/123-ahci[0021:05:012
2688499303,12cyclictest28352-21cat20:25:022
2688499301,6cyclictest9459-21perl00:35:162
2688499301,5cyclictest0-21swapper/222:15:142
2688499301,4cyclictest31671-21ntp_states20:30:212
26884993012,3cyclictest20853-21sendmail_mailst20:05:272
26884993011,3cyclictest10236-21idleruntime-cro23:25:012
2688499301,12cyclictest0-21swapper/221:45:562
26884993010,3cyclictest22600-21grep23:50:212
26884993010,3cyclictest18119-21if_enp1s020:00:172
2687299301,22cyclictest13347-21kworker/0:021:17:330
2688999291,16cyclictest21456-21/usr/sbin/munin23:50:153
2688499299,4cyclictest17372-21date20:00:022
2688499298,3cyclictest0-21swapper/219:40:182
2688499291,6cyclictest0-21swapper/223:19:572
26884992912,3cyclictest29215-21munin-node19:15:032
2688499291,13cyclictest0-21swapper/222:02:042
26884992910,2cyclictest827-21runrttasks19:39:522
2687299294,13cyclictest14034-21perf22:20:000
2688499289,3cyclictest17733-21sh23:40:152
2688499289,3cyclictest17733-21sh23:40:152
2688499288,3cyclictest20222-21uptime22:30:262
2688499288,3cyclictest20222-21uptime22:30:262
2688499288,3cyclictest14970-21latency_hist19:55:012
2688499284,3cyclictest18423-21awk22:30:012
2688499282,3cyclictest3442-21pmu-power00:20:232
2688499281,8cyclictest26862-21cyclictest22:20:152
2688499281,4cyclictest13633-21/usr/sbin/munin21:05:182
2688499281,13cyclictest0-21swapper/221:52:522
2688499281,13cyclictest0-21swapper/220:26:072
2687799282,18cyclictest350-21df_inode00:15:141
2687799282,17cyclictest32314-21tr19:20:111
2687799280,19cyclictest0-21swapper/122:55:191
2688499279,2cyclictest30195-21fschecks_time22:55:162
2688499272,6cyclictest31832-21/usr/sbin/munin23:00:212
2688499271,5cyclictest2080-21cpu19:25:112
2688499271,14cyclictest30878-21head00:10:142
2688499271,11cyclictest0-21swapper/220:56:032
2688499270,11cyclictest6353-21tail20:45:252
2687799271,16cyclictest4210-21unixbench_singl20:40:261
2687299272,12cyclictest4285-21munin-run00:25:020
2687299271,14cyclictest10871-21munin-run00:40:000
2687299270,20cyclictest666-21kworker/0:021:54:250
2688499268,3cyclictest741-21netstat19:20:192
2688499268,3cyclictest20631-21ls23:45:232
2688499262,3cyclictest87350irq/126-enp1s0-21:30:222
2688499261,11cyclictest0-21swapper/221:25:342
26877992615,3cyclictest11721-21memory19:45:191
2687299261,12cyclictest17814-21idleruntime-cro21:15:010
2688499259,3cyclictest18195-21cstates21:15:142
2688499258,3cyclictest0-21swapper/220:16:512
2687799251,17cyclictest0-21swapper/122:31:571
2687799251,17cyclictest0-21swapper/122:31:571
2687299252,7cyclictest3535-21kworker/0:123:11:430
2688999241,4cyclictest17397-21/usr/sbin/munin20:00:163
2688999241,17cyclictest0-21swapper/321:37:433
2688999241,15cyclictest0-21swapper/319:18:183
2688999241,10cyclictest0-21swapper/321:17:463
2688499241,10cyclictest0-21swapper/221:44:592
2687799248,4cyclictest0-21swapper/122:44:211
2687799245,14cyclictest2687-40dnf21:50:501
2687799241,17cyclictest0-21swapper/122:54:301
2687799241,17cyclictest0-21swapper/122:10:311
2687799241,17cyclictest0-21swapper/121:25:111
2687799241,17cyclictest0-21swapper/121:14:261
2687299241,9cyclictest25716-21kworker/0:120:17:090
59362231,2sleep10-21swapper/100:25:251
2688999233,10cyclictest39-21rcuc/323:15:253
2688999231,11cyclictest6413-21/usr/sbin/munin19:35:203
2687799236,4cyclictest12531-21latency_hist23:30:001
2687799231,3cyclictest21856-21chrt21:20:461
2687799231,3cyclictest21856-21chrt21:20:461
2687799231,16cyclictest0-21swapper/122:05:441
2687799231,16cyclictest0-21swapper/122:01:421
2687799231,14cyclictest20788-21unixbench_singl23:45:251
2687799231,12cyclictest7176-21tr20:50:111
2687799230,8cyclictest0-21swapper/121:31:261
2687299232,7cyclictest20255-21kworker/0:022:36:360
2687299231,9cyclictest25222-21kworker/0:123:01:350
2687299231,8cyclictest25222-21kworker/0:122:50:120
9632221,2sleep31052-21mailstats19:20:233
2688999225,4cyclictest18228-21ps23:40:223
2688999225,4cyclictest18228-21ps23:40:223
2688999221,9cyclictest222062fschecks_count20:10:143
2688999221,9cyclictest222062fschecks_count20:10:143
2688999221,6cyclictest0-21swapper/323:07:413
2688999221,5cyclictest0-21swapper/321:24:503
2688999221,5cyclictest0-21swapper/321:24:503
26889992215,4cyclictest6558-21ls22:00:223
2688999221,11cyclictest31776-21cut23:00:023
2687799229,4cyclictest0-21swapper/122:25:141
2687799229,3cyclictest26472-21head00:00:171
2687799223,11cyclictest7109-21fschecks_count00:30:161
26877992217,3cyclictest0-21swapper/123:35:501
26877992212,3cyclictest648-21smart_sda23:00:241
2687799221,16cyclictest87550irq/128-enp1s0-23:58:291
2687799221,16cyclictest87550irq/128-enp1s0-00:14:541
2687799221,16cyclictest0-21swapper/122:16:101
2687799221,15cyclictest0-21swapper/121:06:161
2687799221,11cyclictest0-21swapper/123:34:461
2687799221,10cyclictest0-21swapper/123:06:021
2687799220,16cyclictest0-21swapper/121:40:531
2687799220,16cyclictest0-21swapper/120:00:091
2687299222,8cyclictest10558-21kworker/0:319:51:170
2687299221,9cyclictest29109-21kworker/0:023:08:060
2687299221,9cyclictest13347-21kworker/0:021:36:530
2687299221,6cyclictest0-21swapper/000:25:230
2687299220,9cyclictest666-21kworker/0:021:57:450
90242211,16sleep09213-21sed22:05:250
33722211,16sleep03445-21irqstats20:40:170
2688999214,14cyclictest23637-21gltestperf22:40:153
2688999213,7cyclictest0-21swapper/321:34:363
2688999212,3cyclictest15598-21sh21:10:013
2688999211,7cyclictest0-21swapper/300:00:213
26889992114,4cyclictest2291-21find21:50:233
2688999211,11cyclictest14253-21users23:30:273
2688999211,11cyclictest0-21swapper/300:05:263
2688999210,6cyclictest0-21swapper/322:20:503
2687799218,10cyclictest8451-21cstates23:20:111
26877992114,2cyclictest0-21swapper/120:10:591
26877992114,2cyclictest0-21swapper/120:10:591
2687799211,15cyclictest0-21swapper/120:59:131
2687799211,15cyclictest0-21swapper/120:39:561
2687799211,15cyclictest0-21swapper/120:27:011
2687799211,15cyclictest0-21swapper/119:57:111
2687799211,11cyclictest29325-21smart_sda19:15:021
2687799211,11cyclictest24249-21expr20:15:131
2687799211,10cyclictest0-21swapper/122:25:001
2687799211,10cyclictest0-21swapper/100:36:501
26877992110,3cyclictest28888-21irqstats00:05:231
2687799210,15cyclictest87550irq/128-enp1s0-23:40:141
2687799210,15cyclictest87550irq/128-enp1s0-23:40:141
2687799210,15cyclictest0-21swapper/119:10:191
2687299213,10cyclictest17056-21/usr/sbin/munin23:40:230
2687299213,10cyclictest17056-21/usr/sbin/munin23:40:230
2687299212,11cyclictest27740-21kworker/0:100:11:500
2687299212,11cyclictest20227-21/usr/sbin/munin21:20:190
2687299212,11cyclictest20227-21/usr/sbin/munin21:20:190
2687299211,8cyclictest4572-21kworker/0:120:56:010
2687299211,6cyclictest30841-21kworker/0:121:43:570
26872992113,7cyclictest22325-21kworker/u8:119:10:270
2688999209,4cyclictest0-21swapper/321:45:153
2688999208,3cyclictest19610-21/usr/sbin/munin20:05:273
2688999206,4cyclictest0-21swapper/321:27:533
2688999205,6cyclictest0-21swapper/300:12:083
2688999205,4cyclictest0-21swapper/300:30:143
2688999204,3cyclictest87350irq/126-enp1s0-23:47:003
2688999201,3cyclictest10455-21gunzip20:55:253
2688999201,12cyclictest0-21swapper/323:55:253
2688999201,11cyclictest6091-21smart_sda20:45:243
2688999201,11cyclictest26944-21sshd22:46:463
2688999201,10cyclictest827-21runrttasks19:45:173
2688999201,10cyclictest39-21rcuc/320:25:253
2688999201,10cyclictest30530-21grep21:40:273
2688999201,10cyclictest1224-21irqstats20:35:183
2688999201,10cyclictest11374-21grep22:10:253
2688999201,10cyclictest0-21swapper/300:18:553
26889992010,2cyclictest1447-21awk23:05:023
2688999200,6cyclictest0-21swapper/322:30:293
2688999200,6cyclictest0-21swapper/322:30:293
2687799207,3cyclictest0-21swapper/100:24:111
2687799206,3cyclictest4724-21grep23:10:181
26877992014,3cyclictest21181-21df_inode22:35:141
26877992013,2cyclictest0-21swapper/119:25:411
2687799201,17cyclictest6703-21grep23:15:141
2687799201,17cyclictest0-21swapper/122:49:551
2687799201,14cyclictest0-21swapper/119:43:031
2687799201,13cyclictest0-21swapper/119:35:141
26877992011,2cyclictest0-21swapper/121:01:001
2687799201,11cyclictest30943-21diskstats20:30:141
2687799200,15cyclictest0-21swapper/119:33:281
2687799200,14cyclictest0-21swapper/120:21:361
2687799200,14cyclictest0-21swapper/119:50:101
2687299202,10cyclictest26192-21/usr/sbin/munin20:20:150
2687299201,6cyclictest7807-21kworker/0:323:28:220
2687299201,11cyclictest31215-21/usr/sbin/munin21:45:190
2687299201,11cyclictest25233-21/usr/sbin/munin22:45:270
2687299201,11cyclictest2019-21/usr/sbin/munin19:25:180
2687299200,11cyclictest20909-21kworker/0:123:49:100
2688999196,3cyclictest0-21swapper/323:20:533
2688999195,4cyclictest0-21swapper/300:25:593
2688999195,3cyclictest0-21swapper/323:38:053
2688999191,15cyclictest3285-21sed20:40:163
2688999191,11cyclictest0-21swapper/322:26:443
2688999191,10cyclictest7734-21cat22:05:113
2688999191,10cyclictest15599-21diskmemload21:57:183
2688999191,10cyclictest0-21swapper/322:17:593
2688999191,10cyclictest0-21swapper/300:37:023
2687799191,7cyclictest23-21ksoftirqd/121:55:221
2687799191,6cyclictest21464-21sshd20:08:471
2687799191,16cyclictest0-21swapper/120:45:091
2687799191,14cyclictest0-21swapper/121:17:131
2687299198,3cyclictest808-21snmpd22:14:310
2687299196,4cyclictest15599-21diskmemload00:15:140
2687299196,4cyclictest0-21swapper/000:05:280
2687299192,7cyclictest1201-21kworker/0:223:23:450
2687299190,8cyclictest3811-21kworker/0:219:30:130
2687299190,10cyclictest3998-21kworker/0:200:32:480
3132181,3sleep00-21swapper/020:35:030
2688999184,3cyclictest87650irq/129-enp1s0-20:15:123
2688999181,6cyclictest0-21swapper/321:00:173
2688999181,3cyclictest87350irq/126-enp1s0-23:10:033
26889991813,3cyclictest17420-21users21:10:273
2688999181,15cyclictest0-21swapper/320:52:333
2688999181,10cyclictest0-21swapper/322:54:213
2688999181,10cyclictest0-21swapper/322:35:033
2688999180,11cyclictest0-21swapper/300:20:183
2687299182,7cyclictest10181-21kworker/0:023:30:310
2687299181,4cyclictest24715-21ntp_states23:55:210
26872991813,3cyclictest29486-21grep19:15:020
249612181,2sleep00-21swapper/022:44:380
202592180,2sleep00-21swapper/022:30:370
202592180,2sleep00-21swapper/022:30:370
137212180,2sleep30-21swapper/319:50:183
2688999175,3cyclictest41-21ksoftirqd/320:20:153
26889991713,3cyclictest0-21swapper/319:28:393
26889991711,3cyclictest8614-21/usr/sbin/munin19:40:183
26889991710,4cyclictest302-21date20:35:023
2687299171,8cyclictest26451-21grep00:00:170
2687299171,7cyclictest23281-21sed20:10:250
2687299171,7cyclictest23281-21sed20:10:250
26872991712,3cyclictest20317-21sort20:05:200
26872991712,3cyclictest18228-21sshd22:27:460
26872991712,3cyclictest15203-21/usr/sbin/munin19:55:260
26872991712,3cyclictest13633-21/usr/sbin/munin21:05:220
262692170,2sleep00-21swapper/021:31:060
231522170,3sleep00-21swapper/023:51:410
14912171,2sleep00-21swapper/019:21:360
117512170,2sleep00-21swapper/019:45:200
26889991611,3cyclictest4230-21cpu19:30:113
26889991611,3cyclictest27722-21head19:10:153
2687299165,2cyclictest17051-21kworker/0:120:00:130
2687299164,3cyclictest7937-21cut20:50:180
26872991612,3cyclictest0-21swapper/021:00:210
26872991612,2cyclictest29467-21ntp_states20:25:210
26872991611,3cyclictest6191-21cat22:00:180
26872991610,4cyclictest22896-21uniq21:25:170
105372161,2sleep00-21swapper/019:44:400
2687299152,2cyclictest87650irq/129-enp1s0-20:30:140
26872991510,3cyclictest6075-21/usr/sbin/munin23:15:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional