You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 08:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed May 06, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
175162222191,21sleep00-21swapper/019:09:530
172162217188,19sleep30-21swapper/319:06:013
172732213163,37sleep10-21swapper/119:06:441
173942202162,27sleep20-21swapper/219:08:172
2622521280,1sleep20-21swapper/200:20:202
17624991060,2cyclictest0-21swapper/221:47:402
250902500,1sleep20-21swapper/223:05:152
250132500,3sleep121-21rcuc/119:25:021
231342500,3sleep1221ktimersoftd/100:15:011
303542480,2sleep00-21swapper/020:50:120
133412430,1sleep00-21swapper/021:25:120
1761299341,11cyclictest11796-21kworker/0:123:50:270
1761299332,11cyclictest20596-21kworker/0:000:18:350
1761299332,10cyclictest14778-21kworker/0:200:04:430
1761299321,23cyclictest0-21swapper/021:15:280
101342320,5sleep011-21rcuc/022:30:180
1761299311,24cyclictest0-21swapper/021:37:090
1761299311,10cyclictest18801-21kworker/0:320:25:450
1761299301,23cyclictest0-21swapper/022:13:160
1761299300,11cyclictest32438-21kworker/0:322:08:050
1761299292,11cyclictest20596-21kworker/0:000:34:070
131302290,4sleep339-21rcuc/323:50:193
1762999286,3cyclictest0-21swapper/300:10:553
1761999280,10cyclictest0-21swapper/122:25:191
1761299289,3cyclictest13411-21kworker/0:122:41:050
1761299287,3cyclictest17682-21kworker/0:322:47:450
1761299282,10cyclictest29533-21kworker/0:100:37:390
1761999276,7cyclictest32744-21wc23:20:241
1761999271,20cyclictest0-21swapper/122:50:041
1761999271,16cyclictest31258-21fschecks_time22:05:141
1761999271,10cyclictest0-21swapper/119:56:551
1761299272,8cyclictest8993-21kworker/0:022:54:050
1761299272,10cyclictest14131-21kworker/0:120:49:240
17612992714,3cyclictest5478-21kworker/0:020:11:450
1761299271,23cyclictest383-21unixbench_multi23:20:250
1761299271,12cyclictest24878-21munin-run19:25:010
1761299271,10cyclictest20596-21kworker/0:000:13:270
17612992710,14cyclictest6945-21perf22:25:010
1762999269,3cyclictest0-21swapper/319:14:053
1762999268,2cyclictest0-21swapper/323:10:013
17629992612,4cyclictest0-21swapper/300:34:593
17629992612,3cyclictest0-21swapper/322:36:333
17629992612,3cyclictest0-21swapper/321:06:213
17629992610,3cyclictest0-21swapper/321:37:573
1761999262,16cyclictest5816-21grep21:05:271
1761299262,9cyclictest21417-21kworker/0:221:43:010
1761299262,13cyclictest25109-21kworker/0:200:21:350
1761299261,11cyclictest4537-21kworker/0:122:18:290
1761299261,11cyclictest4537-21kworker/0:122:18:290
1761299261,11cyclictest3106-21kworker/0:223:41:350
1761299261,10cyclictest3106-21kworker/0:223:35:390
1761299260,12cyclictest0-21swapper/021:15:010
1762999258,3cyclictest0-21swapper/322:46:373
1762999258,3cyclictest0-21swapper/322:30:213
1762999258,3cyclictest0-21swapper/319:26:093
17629992512,3cyclictest0-21swapper/321:56:093
17629992511,3cyclictest0-21swapper/300:39:023
1761999250,9cyclictest0-21swapper/122:38:171
1761999250,9cyclictest0-21swapper/121:30:111
1761299252,12cyclictest5478-21kworker/0:020:00:250
1761299252,11cyclictest20596-21kworker/0:000:29:110
1761299251,12cyclictest32297-21/usr/sbin/munin20:55:180
1761299251,12cyclictest29813-21hddtemp_smartct23:15:160
1761299250,9cyclictest820-21kworker/0:023:30:030
1762999249,3cyclictest0-21swapper/323:40:033
1762999249,3cyclictest0-21swapper/319:52:293
1762999249,3cyclictest0-21swapper/319:52:293
1762999248,4cyclictest9188-21ntpq20:00:213
1762999248,3cyclictest6987-21ntp_states19:55:213
1762999248,3cyclictest0-21swapper/323:17:473
1762999248,3cyclictest0-21swapper/320:38:333
1762999247,3cyclictest0-21swapper/323:21:313
1762999247,3cyclictest0-21swapper/323:10:573
1762999246,3cyclictest9801-21ntp_states21:15:213
17629992420,3cyclictest0-21swapper/322:55:243
17629992412,3cyclictest0-21swapper/321:49:533
17629992411,3cyclictest32674-21grep19:40:173
1761999241,9cyclictest0-21swapper/120:13:471
17619992413,3cyclictest22399-21sort21:45:151
1761999240,9cyclictest0-21swapper/120:00:031
1761999240,10cyclictest0-21swapper/123:06:391
1761299249,3cyclictest17388-21kworker/0:119:30:010
1761299243,3cyclictest89050irq/133-enp2s0-19:17:090
1761299242,11cyclictest8993-21kworker/0:022:55:200
1761299241,8cyclictest19199-21kworker/0:121:53:010
1761299241,15cyclictest0-21swapper/023:28:490
1761299241,15cyclictest0-21swapper/023:28:490
1761299241,12cyclictest31684-21/usr/sbin/munin19:40:200
1761299241,12cyclictest24594-21kworker/0:219:25:170
1761299241,12cyclictest20779-21awk00:10:020
1761299241,11cyclictest27388-21ntp_states21:55:220
1761299241,10cyclictest26660-21kworker/0:023:11:050
1762999239,3cyclictest0-21swapper/322:22:053
1762999239,3cyclictest0-21swapper/321:53:093
1762999237,3cyclictest0-21swapper/321:30:533
1762999237,3cyclictest0-21swapper/321:20:493
1762999235,3cyclictest0-21swapper/319:49:173
1762999233,3cyclictest0-21swapper/320:23:373
17629992311,3cyclictest0-21swapper/322:18:133
17629992311,3cyclictest0-21swapper/322:18:133
17629992310,3cyclictest0-21swapper/322:54:493
17624992317,3cyclictest25076-21sed20:35:262
1761999236,9cyclictest0-21swapper/123:10:161
1761999232,15cyclictest87650irq/129-enp1s0-21:40:291
1761999231,3cyclictest26246-21cat21:55:011
1761999231,16cyclictest0-21swapper/120:46:491
1761999230,8cyclictest0-21swapper/123:58:291
1761999230,11cyclictest0-21swapper/100:32:371
1761999230,10cyclictest0-21swapper/122:57:071
1761299238,3cyclictest4022-21kworker/0:221:06:570
1761299237,2cyclictest19199-21kworker/0:122:04:210
1761299231,19cyclictest0-21swapper/023:55:290
1761299231,19cyclictest0-21swapper/021:45:390
1761299231,12cyclictest0-21swapper/021:30:520
1761299231,11cyclictest0-21swapper/023:06:230
1761299231,11cyclictest0-21swapper/019:55:190
1761299231,10cyclictest25712-21/usr/sbin/munin20:40:190
17612992310,2cyclictest19889-21kworker/0:123:00:170
1762999229,3cyclictest0-21swapper/321:11:293
1762999229,3cyclictest0-21swapper/321:04:493
1762999228,3cyclictest0-21swapper/323:25:473
1762999228,3cyclictest0-21swapper/323:25:473
1762999228,3cyclictest0-21swapper/319:36:453
1762999228,3cyclictest0-21swapper/300:16:313
1762999226,5cyclictest16605-21kworker/u8:020:52:163
1762999225,3cyclictest0-21swapper/300:04:153
1762999223,10cyclictest13743-21/usr/sbin/munin22:40:113
1762999222,6cyclictest7553-21awk23:40:013
17629992220,1cyclictest0-21swapper/319:30:123
17629992219,2cyclictest0-21swapper/322:00:253
1762999221,8cyclictest7504-21diskstats22:25:153
17629992218,3cyclictest0-21swapper/320:55:383
17629992213,3cyclictest0-21swapper/300:21:353
1762999221,15cyclictest87350irq/126-enp1s0-22:06:273
17629992211,3cyclictest0-21swapper/323:31:193
1762499225,3cyclictest87350irq/126-enp1s0-20:30:132
17624992218,3cyclictest0-21swapper/223:48:172
1762499221,6cyclictest0-21swapper/222:36:082
1761999221,5cyclictest0-21swapper/121:25:151
17619992215,2cyclictest0-21swapper/122:10:131
1761999221,18cyclictest24584-21ntp_states20:35:221
1761999221,17cyclictest11981-21ntp_kernel_pll_21:20:201
1761999221,16cyclictest0-21swapper/122:03:401
1761999221,15cyclictest0-21swapper/119:26:151
17619992210,3cyclictest23-21ksoftirqd/119:17:091
1761999220,8cyclictest0-21swapper/123:01:031
1761299228,3cyclictest1508-21/usr/sbin/munin19:45:260
1761299227,5cyclictest29040-21kworker/0:219:38:250
1761299227,3cyclictest3723-21/usr/sbin/munin19:50:200
1761299227,3cyclictest3723-21/usr/sbin/munin19:50:200
1761299225,3cyclictest14131-21kworker/0:120:35:130
1761299221,11cyclictest6267-21kworker/0:021:20:490
1761299221,11cyclictest11796-21kworker/0:123:49:550
1761299221,10cyclictest14131-21kworker/0:120:15:130
1761299221,10cyclictest0-21swapper/022:36:030
1761299220,20cyclictest0-21swapper/020:30:190
173172220,4sleep230-21rcuc/221:35:012
154932221,7sleep10-21swapper/122:42:201
1762999217,2cyclictest0-21swapper/320:06:383
1762999215,8cyclictest87350irq/126-enp1s0-22:10:133
1762999215,3cyclictest0-21swapper/323:58:593
1762999214,1cyclictest0-21swapper/319:18:253
1762999212,2cyclictest87450irq/127-enp1s0-20:17:553
1762999211,7cyclictest12864-21diskstats20:10:133
1762999211,3cyclictest0-21swapper/320:32:133
1762999211,11cyclictest27905-21/usr/sbin/munin20:45:243
1762999210,2cyclictest87450irq/127-enp1s0-20:42:233
1762499215,5cyclictest909-21sssd_nss23:55:132
1762499212,6cyclictest12517-21/usr/sbin/munin20:10:222
1762499211,8cyclictest0-21swapper/221:54:192
1761999219,3cyclictest27484-21wc19:30:131
1761999211,9cyclictest0-21swapper/121:56:001
17619992114,2cyclictest0-21swapper/120:57:061
17619992113,3cyclictest0-21swapper/121:36:281
1761999210,10cyclictest0-21swapper/123:25:491
1761999210,10cyclictest0-21swapper/123:25:491
1761299217,3cyclictest7157-21/usr/sbin/munin22:25:230
1761299211,9cyclictest1767-21kworker/0:021:01:010
1761299211,9cyclictest14131-21kworker/0:120:20:210
1761299210,2cyclictest0-21swapper/020:05:190
283092201,15sleep228397-21irqstats00:25:182
1762999208,3cyclictest0-21swapper/300:05:473
1762999204,3cyclictest12050irq/123-ahci[0019:20:223
17629992015,2cyclictest22543-21/usr/sbin/munin23:00:163
17629992010,3cyclictest0-21swapper/321:43:013
1762499205,8cyclictest16622-40dnf20:19:242
1762499201,10cyclictest32747-21ntp_states19:40:192
1761999209,3cyclictest1874-21diskstats19:45:141
1761999201,8cyclictest0-21swapper/100:17:191
17619992016,3cyclictest0-21swapper/100:35:221
17619992012,3cyclictest0-21swapper/122:17:461
17619992012,3cyclictest0-21swapper/122:17:461
1761999201,17cyclictest0-21swapper/121:13:191
1761999201,14cyclictest0-21swapper/122:30:071
1761999201,11cyclictest18157-21df_inode19:10:151
1761999201,10cyclictest1274-21sh19:45:011
1761999200,9cyclictest0-21swapper/123:37:241
1761999200,14cyclictest0-21swapper/120:41:441
1761299201,9cyclictest30576-21kworker/0:219:11:570
1762999198,2cyclictest0-21swapper/320:25:173
17629991913,3cyclictest9971-21/usr/sbin/munin23:45:193
17629991912,3cyclictest13942-21irqstats21:25:173
1762499196,3cyclictest13313-21head23:50:222
1762499193,10cyclictest460-21uptime23:20:252
1762499191,7cyclictest0-21swapper/222:45:192
1762499191,6cyclictest0-21swapper/223:27:532
1762499191,6cyclictest0-21swapper/223:27:532
1762499191,5cyclictest0-21swapper/221:22:222
17624991913,3cyclictest28054-21ls23:10:222
17624991913,3cyclictest10226-21cat22:30:192
1761999194,6cyclictest88350kworker/u8:123:40:251
1761999191,6cyclictest30945-21grep20:50:181
17619991915,3cyclictest0-21swapper/123:48:341
17619991915,3cyclictest0-21swapper/119:50:271
17619991915,3cyclictest0-21swapper/119:50:271
17619991915,3cyclictest0-21swapper/100:22:011
17619991915,2cyclictest22307-21strings20:30:191
17619991913,2cyclictest0-21swapper/121:17:051
1761999191,2cyclictest0-21swapper/120:24:361
1761999191,17cyclictest0-21swapper/120:15:051
1761999191,12cyclictest0-21swapper/123:31:391
1761999191,10cyclictest17915-21latency_hist22:50:021
1761999191,10cyclictest10088-21crond20:05:021
1761999190,17cyclictest0-21swapper/100:03:331
1761999190,14cyclictest0-21swapper/121:03:431
1762999182,9cyclictest0-21swapper/300:27:403
1762499188,2cyclictest0-21swapper/222:01:322
1762499185,4cyclictest893-21fschecks_count22:10:152
1762499183,6cyclictest6539-21/usr/sbin/munin21:10:162
1762499182,3cyclictest0-21swapper/222:25:202
1762499181,7cyclictest6037-21fschecks_count23:35:152
1762499181,7cyclictest22903-21/usr/sbin/munin19:20:162
1762499181,7cyclictest15252-21grep22:40:252
1762499181,7cyclictest0-21swapper/221:27:242
1762499181,6cyclictest1580-21crond00:40:012
1762499181,4cyclictest0-21swapper/223:42:252
1762499181,2cyclictest0-21swapper/219:30:132
17624991812,3cyclictest22543-21/usr/sbin/munin23:00:262
1762499181,10cyclictest0-21swapper/222:20:272
17619991815,2cyclictest0-21swapper/120:29:071
17619991815,2cyclictest0-21swapper/100:26:391
17619991815,2cyclictest0-21swapper/100:06:591
1761999181,2cyclictest87650irq/129-enp1s0-22:21:201
1761999181,16cyclictest0-21swapper/123:52:401
1761999181,16cyclictest0-21swapper/123:16:261
1761999181,10cyclictest0-21swapper/119:35:271
1762499175,3cyclictest87350irq/126-enp1s0-21:40:162
1762499175,3cyclictest3378-21sshd19:48:372
1762499175,2cyclictest17562-21kworker/u8:000:17:402
1762499174,4cyclictest4122-21cut19:50:152
1762499174,4cyclictest4122-21cut19:50:152
1762499172,3cyclictest87550irq/128-enp1s0-20:00:262
1762499172,10cyclictest0-21swapper/221:35:332
1762499171,7cyclictest2138-21/usr/sbin/munin21:00:142
1762499171,6cyclictest29870-21latency_hist20:50:012
1762499171,6cyclictest19209-21ntp_offset22:50:212
1762499171,5cyclictest0-21swapper/200:00:262
17624991713,3cyclictest0-21swapper/221:16:062
17624991713,3cyclictest0-21swapper/220:55:512
17624991712,3cyclictest30657-21sed23:15:262
17624991712,3cyclictest11475-21awk20:05:242
1762499171,15cyclictest0-21swapper/220:50:312
17624991711,3cyclictest3373-21/usr/sbin/munin23:30:242
17624991711,3cyclictest20765-21latency_hist00:10:022
17624991710,4cyclictest5053-21sort21:05:182
1762499168,2cyclictest0-21swapper/219:15:482
1762499167,6cyclictest0-21swapper/222:05:112
1762499166,2cyclictest87550irq/128-enp1s0-22:55:152
1762499163,4cyclictest7845-21sshd19:59:382
1762499162,5cyclictest26320-21/usr/sbin/munin21:55:142
1762499161,7cyclictest0-21swapper/200:09:542
1762499161,6cyclictest0-21swapper/200:34:552
1762499161,5cyclictest3202-21cut22:15:162
1762499161,5cyclictest3202-21cut22:15:162
1762499161,5cyclictest19720-21grep20:25:152
17624991612,3cyclictest0-21swapper/220:44:362
17624991611,3cyclictest16915-21/usr/sbin/munin20:20:132
1762499152,10cyclictest29648-21cpuspeed_turbos19:35:132
17624991510,3cyclictest827-21runrttasks19:12:052
17624991510,3cyclictest26436-21threads19:25:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional