You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 10:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sun Apr 05, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34652235158,24sleep20-21swapper/219:06:362
36562234161,25sleep00-21swapper/019:09:020
36792202165,25sleep30-21swapper/319:09:223
35062202165,25sleep10-21swapper/119:07:071
462721570,4sleep2383099cyclictest22:50:132
36321510,5sleep0381999cyclictest21:25:230
3115821200,2sleep20-21swapper/222:35:202
3830991131,2cyclictest0-21swapper/220:20:552
3830999491,2cyclictest0-21swapper/219:53:092
3830999390,2cyclictest0-21swapper/222:14:532
133672660,3sleep20-21swapper/200:22:352
57072560,4sleep121-21rcuc/121:40:011
15662520,2sleep339-21rcuc/320:15:153
220152500,1sleep10-21swapper/122:15:161
154032490,2sleep20-21swapper/222:00:162
46512410,5sleep04653-21gltestperf19:10:150
381999399,14cyclictest8528-21munin-run23:00:010
235062350,1sleep00-21swapper/021:05:160
381999331,11cyclictest8162-21kworker/0:123:15:550
265842330,4sleep230-21rcuc/223:40:122
52002310,4sleep339-21rcuc/319:10:223
3824992715,4cyclictest6061-21ntp_states00:05:221
382499271,14cyclictest0-21swapper/100:10:391
381999275,9cyclictest0-21swapper/023:34:470
381999271,13cyclictest8887-21awk19:20:010
382499260,18cyclictest0-21swapper/123:41:571
383599253,8cyclictest87350irq/126-enp1s0-22:58:513
383599250,18cyclictest0-21swapper/322:00:073
382499251,4cyclictest11907-21missed_timers23:05:211
381999254,10cyclictest0-21swapper/023:14:460
197732250,7sleep121-21rcuc/122:10:151
383599241,16cyclictest0-21swapper/323:55:213
383599240,17cyclictest0-21swapper/322:11:083
383099246,3cyclictest0-21swapper/223:05:222
383099241,11cyclictest12108-21irqstats19:25:162
383099241,11cyclictest12108-21irqstats19:25:162
382499244,16cyclictest25854-40dnf23:36:501
382499241,16cyclictest0-21swapper/121:16:291
381999246,2cyclictest29154-21kworker/0:221:20:110
381999242,12cyclictest3871-21kworker/0:000:20:220
381999241,15cyclictest8162-21kworker/0:123:26:350
381999241,12cyclictest26679-21/usr/sbin/munin20:00:250
382499231,19cyclictest26817-21grep20:00:131
382499231,17cyclictest0-21swapper/119:32:501
382499231,16cyclictest10318-21memory00:15:191
382499231,16cyclictest10318-21memory00:15:191
382499231,16cyclictest0-21swapper/122:56:501
382499231,16cyclictest0-21swapper/119:21:051
382499231,11cyclictest0-21swapper/121:29:551
382499230,8cyclictest0-21swapper/123:22:031
381999239,3cyclictest15995-21/usr/sbin/munin00:30:130
381999238,3cyclictest9887-21kworker/0:021:53:350
381999238,3cyclictest9887-21kworker/0:021:53:350
381999236,15cyclictest6992-21awk00:10:010
381999231,9cyclictest17006-21kworker/0:020:52:510
381999231,11cyclictest17553-21/usr/sbin/munin23:20:200
381999231,11cyclictest16446-21sshd22:01:450
381999231,10cyclictest3737-21/usr/sbin/munin21:35:160
381999231,10cyclictest25727-21/usr/sbin/munin22:25:150
381999231,10cyclictest10964-21/usr/sbin/munin23:05:220
3835992216,3cyclictest30853-21dump-pmu-power20:10:013
3835992211,3cyclictest0-21swapper/322:34:253
383599221,13cyclictest0-21swapper/322:20:143
383599221,12cyclictest21673-21df_inode22:15:143
383599221,11cyclictest10780-21ntp_states20:35:213
383599220,6cyclictest0-21swapper/321:49:073
383099225,5cyclictest0-21swapper/200:00:162
383099221,9cyclictest79432ntp_kernel_pll_19:15:192
383099221,9cyclictest155822grep20:45:252
382499229,5cyclictest0-21swapper/120:20:151
382499227,7cyclictest0-21swapper/100:25:131
382499227,4cyclictest0-21swapper/123:26:071
3824992217,3cyclictest24292-21if_err_enp1s022:20:161
382499221,19cyclictest0-21swapper/121:01:541
382499221,18cyclictest25803-21cat21:10:171
382499221,11cyclictest0-21swapper/121:51:391
382499221,11cyclictest0-21swapper/121:51:391
382499221,10cyclictest2350ksoftirqd/121:45:221
382499220,12cyclictest0-21swapper/100:24:221
381999228,3cyclictest25939-21kworker/0:123:40:150
381999227,3cyclictest6840-21kworker/0:300:17:230
381999227,3cyclictest6840-21kworker/0:300:17:230
381999221,10cyclictest819-21lldpd22:16:200
381999221,10cyclictest28542-21/usr/sbin/munin23:45:160
383599219,6cyclictest0-21swapper/322:47:013
383599219,4cyclictest0-21swapper/323:10:133
383599215,3cyclictest87350irq/126-enp1s0-23:00:033
383599214,9cyclictest26118-21turbostat23:40:013
383599213,3cyclictest0-21swapper/321:07:403
383599212,14cyclictest0-21swapper/300:09:393
383599211,9cyclictest0-21swapper/300:39:273
383599211,8cyclictest15695-21/usr/sbin/munin19:35:253
383599211,6cyclictest0-21swapper/323:08:253
3835992114,4cyclictest15962-21head23:15:153
3835992113,3cyclictest0-21swapper/323:30:253
383599211,14cyclictest0-21swapper/321:33:353
383599211,13cyclictest87350irq/126-enp1s0-22:35:243
383599211,11cyclictest808-21snmpd22:25:183
3835992110,6cyclictest0-21swapper/300:15:263
3835992110,6cyclictest0-21swapper/300:15:263
3835992110,5cyclictest0-21swapper/322:52:443
3835992110,3cyclictest0-21swapper/300:04:503
3830992111,6cyclictest11106-21cat20:35:252
382499219,4cyclictest0-21swapper/121:40:151
382499219,3cyclictest24832-21diskstats19:55:141
382499218,4cyclictest15329-21cat23:15:021
382499217,4cyclictest0-21swapper/123:51:201
3824992118,2cyclictest0-21swapper/123:15:131
382499211,14cyclictest0-21swapper/122:00:201
382499211,12cyclictest9336-21fw_packets23:00:161
382499211,11cyclictest24498-21users21:05:271
382499211,10cyclictest0-21swapper/122:05:211
382499211,10cyclictest0-21swapper/121:23:291
382499210,5cyclictest0-21swapper/123:46:481
382499210,15cyclictest0-21swapper/100:33:541
381999217,3cyclictest8137-21/usr/sbin/munin21:45:140
381999216,7cyclictest15365-21kworker/0:219:37:150
381999212,8cyclictest19859-21if_enp1s022:10:160
381999211,9cyclictest24649-21kworker/0:021:14:550
180492210,4sleep2383099cyclictest23:20:142
383599208,4cyclictest0-21swapper/300:10:183
383599207,7cyclictest0-21swapper/323:48:393
383599205,8cyclictest0-21swapper/320:26:473
383599203,8cyclictest0-21swapper/319:49:273
383599203,5cyclictest0-21swapper/319:57:593
383599203,2cyclictest87650irq/129-enp1s0-19:19:593
383599203,12cyclictest87350irq/126-enp1s0-21:25:303
383599202,9cyclictest808-21snmpd21:20:063
383599202,9cyclictest0-21swapper/321:02:463
383599201,6cyclictest808-21snmpd00:30:413
383599201,6cyclictest24880-21diskmemload22:40:123
3835992013,2cyclictest0-21swapper/323:40:203
383599201,14cyclictest0-21swapper/321:54:183
383599201,14cyclictest0-21swapper/321:54:183
383599201,12cyclictest0-21swapper/321:56:473
383599201,11cyclictest13189-21users00:20:283
3835992010,8cyclictest5528-21sshd21:37:453
383099206,3cyclictest87350irq/126-enp1s0-00:05:192
383099206,3cyclictest11457-21pmu-power21:50:232
383099206,3cyclictest11457-21pmu-power21:50:232
383099204,4cyclictest414-21cat23:55:012
3830992014,3cyclictest9365-21head23:00:162
3830992013,4cyclictest18524-21df00:35:122
382499208,3cyclictest31919-21sshd22:37:551
382499207,3cyclictest606-21perf20:15:021
382499203,14cyclictest0-21swapper/123:30:151
3824992012,3cyclictest0-21swapper/119:15:541
382499201,17cyclictest0-21swapper/120:50:391
382499201,14cyclictest0-21swapper/121:34:181
382499201,14cyclictest0-21swapper/119:13:331
382499201,13cyclictest0-21swapper/122:50:161
382499201,11cyclictest30851-21latency_hist20:10:021
382499201,11cyclictest17889-21/usr/sbin/munin19:40:231
382499201,11cyclictest0-21swapper/121:55:121
382499201,10cyclictest909-21sssd_nss00:00:151
382499201,10cyclictest19755-21awk20:55:231
382499200,14cyclictest0-21swapper/100:35:571
381999206,3cyclictest87550irq/128-enp1s0-20:15:160
381999204,2cyclictest87450irq/127-enp1s0-20:41:230
381999204,2cyclictest87450irq/127-enp1s0-20:41:230
381999201,4cyclictest3119-21/usr/sbin/munin20:20:250
381999201,4cyclictest28889-21/usr/sbin/munin20:05:160
24692201,2sleep00-21swapper/023:59:110
223332201,2sleep00-21swapper/021:00:480
131462200,2sleep10-21swapper/119:28:391
131462200,2sleep10-21swapper/119:28:391
383599195,4cyclictest7869-21diskstats20:30:153
383599192,5cyclictest808-21snmpd20:11:473
383599191,7cyclictest0-21swapper/319:25:103
383599191,7cyclictest0-21swapper/319:25:103
383599191,6cyclictest25111-21/usr/sbin/munin21:10:153
383599191,6cyclictest17553-21/usr/sbin/munin23:20:163
383599191,5cyclictest3119-21/usr/sbin/munin20:20:253
383599191,12cyclictest87350irq/126-enp1s0-21:40:133
383599191,12cyclictest0-21swapper/300:26:153
383599191,10cyclictest17889-21/usr/sbin/munin19:40:243
383599190,14cyclictest87350irq/126-enp1s0-22:08:143
383099195,3cyclictest87350irq/126-enp1s0-23:45:142
383099195,3cyclictest12050irq/123-ahci[0022:30:022
383099192,6cyclictest1294-21munin-run21:30:012
383099192,14cyclictest8792-21head21:45:152
383099192,11cyclictest0-21swapper/222:07:102
383099191,5cyclictest23129-21expr21:05:132
383099191,5cyclictest1519-21/usr/sbin/munin21:30:132
383099191,5cyclictest0-21swapper/200:33:202
383099191,4cyclictest25992-21tail19:55:252
3830991913,4cyclictest28188-21grep21:15:192
3830991913,3cyclictest8141-21gltestperf20:30:172
382499196,4cyclictest2440-21sshd23:58:461
3824991915,3cyclictest0-21swapper/120:19:151
382499191,3cyclictest28986-21ntp_states22:30:211
382499191,17cyclictest0-21swapper/119:51:181
382499191,12cyclictest0-21swapper/122:29:511
382499191,10cyclictest8690-21ps20:30:221
382499191,10cyclictest839-21lldpd22:40:451
382499191,10cyclictest3223-21ntp_states22:45:211
382499191,10cyclictest11288-21sed20:35:261
382499190,18cyclictest0-21swapper/120:25:171
382499190,17cyclictest0-21swapper/120:45:201
381999199,6cyclictest0-21swapper/019:29:560
381999199,6cyclictest0-21swapper/019:29:560
381999192,6cyclictest14503-21munin-run22:00:000
381999191,4cyclictest20090-21/usr/sbin/munin19:45:110
3819991912,5cyclictest3955-21kworker/u8:200:37:270
381999191,10cyclictest9718-21/usr/sbin/munin20:35:230
381999190,11cyclictest25150-21kworker/0:222:22:340
60802180,2sleep00-21swapper/022:51:560
383599186,6cyclictest0-21swapper/323:27:323
383599182,7cyclictest0-21swapper/320:01:273
383599182,4cyclictest808-21snmpd21:15:043
383599181,6cyclictest12891-21needreboot20:40:183
383599181,6cyclictest12891-21needreboot20:40:183
383599181,5cyclictest9100-21/usr/sbin/munin19:20:103
383599181,3cyclictest12050irq/123-ahci[0019:50:173
383599181,10cyclictest16310-21/usr/sbin/munin20:50:153
383599180,10cyclictest0-21swapper/320:55:243
383099184,3cyclictest21852-21awk21:00:232
383099182,3cyclictest0-21swapper/219:32:372
383099182,15cyclictest0-21swapper/223:33:572
383099182,10cyclictest0-21swapper/219:36:542
383099181,6cyclictest13881-21latency_hist20:45:002
383099181,6cyclictest13881-21latency_hist20:45:002
383099181,5cyclictest8523-21latency_hist23:00:012
383099181,5cyclictest5982-21cpu21:40:112
383099181,5cyclictest20302-21cat19:45:122
3830991813,3cyclictest827-21runrttasks21:20:252
3830991812,3cyclictest1990-21ntpq20:15:202
383099181,15cyclictest0-21swapper/220:53:292
383099181,14cyclictest31691-21gltestperf20:10:152
382499181,16cyclictest0-21swapper/120:41:561
382499181,16cyclictest0-21swapper/120:41:561
382499181,16cyclictest0-21swapper/119:47:161
382499180,16cyclictest0-21swapper/119:35:121
381999181,9cyclictest0-21swapper/021:41:460
381999181,5cyclictest8755-21/usr/sbin/munin23:00:160
3819991813,3cyclictest32451-21tr22:40:120
3819991813,3cyclictest24143-21/usr/sbin/munin23:35:120
3819991812,3cyclictest4767-21dump-pmu-power00:05:010
3819991812,3cyclictest30120-21/usr/sbin/munin22:35:170
3819991812,3cyclictest2159-21/usr/sbin/munin22:45:210
381999181,14cyclictest24484-21/usr/sbin/munin19:55:160
381999181,11cyclictest0-21swapper/021:15:210
155802180,3sleep015584-21unixbench-2d20:45:260
383599174,3cyclictest87650irq/129-enp1s0-19:30:153
383599172,12cyclictest14113-21/usr/sbin/munin20:45:193
3835991712,3cyclictest30741-21/usr/sbin/munin23:50:153
383099171,9cyclictest0-21swapper/223:35:302
383099171,8cyclictest13159-21/usr/sbin/munin23:10:162
383099171,6cyclictest21337-21/usr/sbin/munin22:15:122
383099171,4cyclictest15358-21/usr/sbin/munin23:15:212
383099171,3cyclictest0-21swapper/222:31:512
3830991712,3cyclictest3737-21/usr/sbin/munin21:35:222
3830991712,3cyclictest26333-21rt-features21:10:242
383099171,15cyclictest0-21swapper/223:29:222
383099171,15cyclictest0-21swapper/222:48:252
3830991711,3cyclictest9174-21turbostat00:15:012
3830991711,3cyclictest5487-21grep20:25:132
3830991711,3cyclictest13598-21ntpq21:55:212
383099170,11cyclictest29702-21/usr/sbin/munin20:05:192
381999173,2cyclictest87550irq/128-enp1s0-22:33:100
381999172,3cyclictest30765-21kworker/0:220:11:580
381999171,7cyclictest0-21swapper/000:26:050
381999171,5cyclictest16935-21/usr/sbin/munin22:05:230
381999171,4cyclictest7765-21cat20:30:140
381999171,3cyclictest8839-21sshd00:10:500
3819991712,3cyclictest23426-21pmu-power19:50:210
381999171,15cyclictest0-21swapper/023:52:150
381999171,13cyclictest20405-21taskset20:59:140
383099161,9cyclictest0-21swapper/222:40:552
383099161,6cyclictest24762-21find22:20:222
383099161,5cyclictest10091-21needreboot19:20:182
3830991612,2cyclictest9403-21/usr/sbin/munin00:15:262
3830991612,2cyclictest9403-21/usr/sbin/munin00:15:262
3830991611,3cyclictest27492-21kernelversion20:00:192
3830991611,3cyclictest18608-21grep20:55:132
3830991610,4cyclictest0-21swapper/200:25:232
3830991610,3cyclictest827-21runrttasks23:55:402
3830991610,3cyclictest4762-21if_enp2s019:10:162
3830991610,3cyclictest18042-21expr19:40:122
381999161,5cyclictest0-21swapper/020:25:240
381999161,4cyclictest13493-21/usr/sbin/munin19:30:250
3819991612,3cyclictest0-21swapper/019:44:140
3819991610,3cyclictest9753-21gltestperf19:20:150
381999151,6cyclictest0-21swapper/021:30:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional