You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 18:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 12, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
66432237162,25sleep30-21swapper/307:08:173
64132232169,51sleep10-21swapper/107:05:301
66452203165,25sleep00-21swapper/007:08:190
64312202165,24sleep20-21swapper/207:05:312
1277921530,3sleep1687499cyclictest09:45:271
212042720,1sleep10-21swapper/107:40:151
6879996864,3cyclictest0-21swapper/207:50:262
41852550,7sleep07-21ksoftirqd/008:15:140
244642520,2sleep121-21rcuc/110:15:141
6885993517,3cyclictest0-21swapper/308:57:223
688599321,22cyclictest0-21swapper/307:46:123
686899327,12cyclictest20394-21awk11:20:010
686899323,14cyclictest21001-21munin-run12:35:010
6868993111,16cyclictest21977-21idleruntime-cro10:10:020
687499291,19cyclictest5673-21ntp_states09:30:281
688599284,16cyclictest41-21ksoftirqd/310:05:163
687499281,8cyclictest0-21swapper/110:03:261
687499281,25cyclictest0-21swapper/109:42:081
6868992810,3cyclictest6852-21kworker/0:110:49:020
688599271,16cyclictest16258-21hddtemp_smartct09:55:173
688599271,13cyclictest22819-1kworker/3:2H12:18:103
688599271,12cyclictest0-21swapper/308:40:363
687499274,12cyclictest86750irq/128-enp1s0-11:50:001
6874992723,3cyclictest0-21swapper/108:45:311
6874992723,3cyclictest0-21swapper/108:38:061
687499271,21cyclictest0-21swapper/112:20:591
687499271,21cyclictest0-21swapper/110:14:461
687499271,20cyclictest0-21swapper/110:25:011
687499271,19cyclictest0-21swapper/111:24:081
687499271,18cyclictest14335-21cstates11:05:141
687499270,25cyclictest0-21swapper/110:55:211
686899278,16cyclictest3415-21munin-run11:55:010
686899275,10cyclictest16622-21perf12:25:010
273342270,5sleep2687999cyclictest11:35:142
273342270,5sleep2687999cyclictest11:35:142
688599264,12cyclictest780-21rngd09:06:423
688599261,8cyclictest0-21swapper/311:10:043
688599261,13cyclictest0-21swapper/311:39:003
688599261,13cyclictest0-21swapper/311:39:003
687499265,11cyclictest19027-21expr07:35:141
687499264,3cyclictest14360-21missed_timers09:50:221
6874992623,2cyclictest0-21swapper/108:27:071
6874992623,2cyclictest0-21swapper/108:21:171
6874992623,2cyclictest0-21swapper/108:21:171
6874992622,3cyclictest0-21swapper/108:11:021
687499262,11cyclictest27880-21df_inode07:55:141
6874992612,4cyclictest29970-21fschecks_time11:40:181
687499261,22cyclictest9788-21kworker/u8:112:09:081
687499261,21cyclictest21959-21sadc10:10:021
687499261,20cyclictest0-21swapper/112:36:351
687499261,20cyclictest0-21swapper/111:28:451
687499260,23cyclictest4991-21awk10:45:011
687499260,20cyclictest0-21swapper/110:39:331
687499260,20cyclictest0-21swapper/109:07:491
686899261,13cyclictest14414-21munin-run12:20:020
688599256,3cyclictest0-21swapper/308:24:583
688599256,3cyclictest0-21swapper/308:24:583
688599253,14cyclictest86550irq/126-enp1s0-10:40:143
688599252,11cyclictest0-21swapper/307:12:343
688599251,12cyclictest0-21swapper/308:27:363
688599250,17cyclictest0-21swapper/309:30:043
6874992522,2cyclictest0-21swapper/107:15:151
6874992522,2cyclictest0-21swapper/107:15:141
6874992521,3cyclictest0-21swapper/107:20:161
687499251,9cyclictest0-21swapper/111:55:291
687499251,22cyclictest0-21swapper/110:30:181
687499251,22cyclictest0-21swapper/110:30:181
687499251,21cyclictest31587-21gunzip09:15:251
687499251,16cyclictest0-21swapper/111:35:171
687499251,16cyclictest0-21swapper/111:35:171
6874992510,4cyclictest18624-21taskset12:27:551
687499250,20cyclictest0-21swapper/109:27:371
686899258,14cyclictest29523-21idleruntime-cro08:00:020
34142250,5sleep230-21rcuc/210:40:152
321362250,4sleep132139-21cstates08:05:191
688599245,8cyclictest18921-40dnf08:47:343
688599245,8cyclictest12050irq/123-ahci[0011:50:023
688599245,10cyclictest1424-21/usr/sbin/munin11:50:183
688599241,20cyclictest0-21swapper/309:02:043
688599241,12cyclictest0-21swapper/310:20:243
688599241,11cyclictest16529-21taskset12:24:123
688599241,10cyclictest0-21swapper/307:38:003
687999241,8cyclictest0-21swapper/209:25:232
687499249,3cyclictest19373-21df_inode12:30:151
6874992413,3cyclictest7044-21chrt10:48:221
687499241,12cyclictest29139-21diskstats10:25:191
687499241,11cyclictest8762-21latency_hist09:40:021
687499241,11cyclictest581-21systemd-journal11:35:011
687499241,11cyclictest28850-21hddtemp_smartct09:10:211
687499241,11cyclictest13400-21users11:00:311
687499240,19cyclictest0-21swapper/111:18:381
688599235,8cyclictest0-21swapper/312:05:443
688599231,8cyclictest0-21swapper/307:57:203
688599231,7cyclictest0-21swapper/307:18:123
688599231,7cyclictest0-21swapper/307:18:123
688599231,6cyclictest1893-21ntpq10:35:233
688599231,5cyclictest0-21swapper/312:36:263
688599231,4cyclictest0-21swapper/309:15:153
688599231,12cyclictest0-21swapper/311:59:123
688599231,11cyclictest0-21swapper/310:45:223
688599231,10cyclictest0-21swapper/310:10:193
687999237,4cyclictest4581-21gltestperf08:15:162
687999235,4cyclictest20387-21perf11:20:002
687999231,5cyclictest1555-21kworker/u8:209:43:102
6874992321,1cyclictest0-21swapper/107:10:581
6874992320,2cyclictest0-21swapper/108:17:351
6874992320,2cyclictest0-21swapper/107:48:041
687499231,9cyclictest16761-21ps09:55:241
687499231,3cyclictest7525-21cpuspeed_turbos10:50:141
6874992313,2cyclictest0-21swapper/109:00:531
687499231,20cyclictest0-21swapper/112:17:471
687499231,20cyclictest0-21swapper/111:15:021
687499231,19cyclictest22229-21hddtemp_smartct08:55:171
687499231,11cyclictest11124-21latency12:10:231
687499231,10cyclictest7264-21cat12:00:271
687499231,10cyclictest18722-21cat07:35:021
686899237,6cyclictest88050irq/133-enp2s0-07:21:560
686899235,15cyclictest18522-21munin-run07:35:010
686899234,2cyclictest19093-21kworker/0:008:55:160
686899231,12cyclictest32566-21/usr/sbin/munin09:20:270
686899230,11cyclictest7835-21perf12:05:020
313772231,3sleep231610-21tr10:30:162
313772231,3sleep231610-21tr10:30:162
124302230,2sleep232-21ksoftirqd/209:45:232
111192230,4sleep230-21rcuc/208:30:162
688599227,8cyclictest86550irq/126-enp1s0-12:00:163
688599227,3cyclictest0-21swapper/307:43:123
688599224,7cyclictest3467-21ntp_states09:25:223
688599224,2cyclictest0-21swapper/311:40:203
688599223,4cyclictest0-21swapper/310:26:283
688599221,9cyclictest0-21swapper/310:31:483
688599221,9cyclictest0-21swapper/310:31:483
688599221,9cyclictest0-21swapper/307:32:083
688599221,7cyclictest0-21swapper/308:11:003
688599221,6cyclictest0-21swapper/309:11:463
688599221,11cyclictest0-21swapper/311:24:363
688599221,10cyclictest0-21swapper/308:02:563
687999226,3cyclictest12050irq/123-ahci[0009:55:012
687999225,9cyclictest8277-21grep09:35:272
687999221,6cyclictest0-21swapper/211:40:222
6879992215,4cyclictest31952-21diskstats11:45:172
687999221,4cyclictest404-21kworker/u8:111:50:262
687499221,19cyclictest0-21swapper/108:50:111
687499221,19cyclictest0-21swapper/107:52:431
687499220,5cyclictest0-21swapper/111:53:551
686899222,11cyclictest3641-21/usr/sbin/munin11:55:180
686899222,11cyclictest23756-21/usr/sbin/munin09:00:240
688599217,3cyclictest0-21swapper/308:19:403
688599215,4cyclictest7764-21diskstats10:50:163
688599214,2cyclictest86550irq/126-enp1s0-09:49:413
688599212,17cyclictest41-21ksoftirqd/311:05:193
688599211,9cyclictest0-21swapper/307:23:523
688599211,6cyclictest0-21swapper/310:01:423
688599211,13cyclictest0-21swapper/312:30:203
688599210,9cyclictest0-21swapper/312:29:123
688599210,7cyclictest0-21swapper/308:39:163
688599210,10cyclictest0-21swapper/311:33:443
687999215,8cyclictest27935-21diskmemload11:58:522
687999211,8cyclictest6178-21/usr/sbin/munin08:20:232
687999211,8cyclictest6178-21/usr/sbin/munin08:20:232
687999211,7cyclictest29860-21ntp_states10:25:252
6874992118,2cyclictest0-21swapper/108:33:581
687499211,7cyclictest32566-21/usr/sbin/munin09:20:201
687499211,19cyclictest0-21swapper/108:01:051
687499211,17cyclictest16031-21ntp_states08:40:261
687499210,20cyclictest0-21swapper/107:25:531
686899216,6cyclictest29018-21kworker/0:211:40:160
686899215,9cyclictest17959-21taskset11:12:160
686899213,9cyclictest12768-21/usr/sbin/munin08:35:280
686899211,6cyclictest0-21swapper/010:38:280
686899211,11cyclictest21226-21/usr/sbin/munin12:35:200
688599207,2cyclictest0-21swapper/308:30:183
688599204,9cyclictest0-21swapper/311:03:223
688599202,12cyclictest41-21ksoftirqd/312:11:303
688599202,11cyclictest0-21swapper/307:52:363
688599201,3cyclictest9678-21cpuspeed_turbos10:55:143
688599200,13cyclictest0-21swapper/309:51:543
687999206,3cyclictest86550irq/126-enp1s0-09:15:202
687999203,3cyclictest86550irq/126-enp1s0-11:34:252
687999202,10cyclictest29738-21/usr/sbin/munin08:00:122
687999201,8cyclictest18169-21taskset11:14:562
687999201,7cyclictest281742/usr/sbin/munin09:10:202
687999201,7cyclictest15029-21grep08:40:152
687999201,6cyclictest22192-21/usr/sbin/munin10:10:232
687999201,6cyclictest16436-21cat07:30:012
687999201,6cyclictest12242-21ls07:20:132
687999201,6cyclictest0-21swapper/212:09:572
687999201,5cyclictest9-21rcu_preempt12:00:162
687999201,4cyclictest0-21swapper/207:45:182
6879992014,4cyclictest14405-21dump-pmu-power12:20:012
687999201,14cyclictest86850irq/129-enp1s0-08:45:152
687999201,12cyclictest0-21swapper/211:03:192
687999201,11cyclictest0-21swapper/209:30:242
687999200,13cyclictest0-21swapper/211:25:592
686899205,7cyclictest21620-21kworker/0:210:11:540
686899203,4cyclictest20617-21/usr/sbin/munin11:20:160
686899201,4cyclictest0-21swapper/011:02:320
686899201,13cyclictest0-21swapper/008:40:160
686899201,13cyclictest0-21swapper/008:05:330
686899201,12cyclictest0-21swapper/011:32:040
686899201,11cyclictest26591-21/usr/sbin/munin10:20:270
686899201,11cyclictest25345-21/usr/sbin/munin07:50:290
688599198,2cyclictest86850irq/129-enp1s0-09:35:233
688599197,3cyclictest21325-21latency_hist08:55:023
688599196,7cyclictest86550irq/126-enp1s0-10:15:173
688599196,3cyclictest0-21swapper/308:09:283
688599194,5cyclictest0-21swapper/309:23:463
688599194,3cyclictest0-21swapper/311:29:163
688599191,9cyclictest0-21swapper/309:40:223
688599191,10cyclictest0-21swapper/311:17:093
687999198,3cyclictest86850irq/129-enp1s0-10:35:232
687999195,3cyclictest16937-21grep07:30:182
687999193,3cyclictest19469-21fschecks_count12:30:152
687999191,6cyclictest27935-21diskmemload09:56:172
687999191,6cyclictest21052-21ntpq10:05:232
687999191,6cyclictest0-21swapper/210:55:162
6879991916,2cyclictest0-21swapper/207:35:562
687999191,5cyclictest8396-21needreboot10:50:202
687999191,5cyclictest10668-21cat07:15:172
687999191,5cyclictest10668-21cat07:15:162
687999191,5cyclictest0-21swapper/210:17:452
687999191,5cyclictest0-21swapper/209:20:252
6879991914,3cyclictest11661-21sort12:10:292
687999191,10cyclictest14017-21/usr/sbin/munin11:05:172
686899198,2cyclictest86750irq/128-enp1s0-09:41:310
686899197,3cyclictest14351-21/usr/sbin/munin07:25:180
686899195,7cyclictest18439-21kworker/0:307:38:000
686899195,4cyclictest7-21ksoftirqd/009:50:180
686899195,2cyclictest16757-21kworker/0:208:48:200
686899192,5cyclictest25963-21/usr/sbin/munin09:05:210
686899191,9cyclictest27761-21kworker/0:109:15:140
686899191,6cyclictest0-21swapper/011:37:250
686899191,6cyclictest0-21swapper/011:37:250
686899191,5cyclictest28174-21/usr/sbin/munin09:10:310
686899191,5cyclictest11452-21sshd10:58:330
686899191,5cyclictest0-21swapper/011:26:590
6868991914,3cyclictest11-21rcuc/009:40:010
6868991913,3cyclictest21573-21gltestperf07:40:180
6868991911,5cyclictest11-21rcuc/012:10:020
686899191,10cyclictest19362-21/usr/sbin/munin08:50:140
686899191,10cyclictest10574-21/usr/sbin/munin08:30:160
688599181,15cyclictest0-21swapper/307:27:073
687999185,3cyclictest86550irq/126-enp1s0-10:20:312
687999184,3cyclictest86750irq/128-enp1s0-09:05:142
687999184,3cyclictest28773-21gunzip07:55:282
687999184,3cyclictest27935-21diskmemload11:20:092
687999183,8cyclictest0-21swapper/210:04:342
687999183,4cyclictest23034-21sed08:55:262
687999183,3cyclictest86550irq/126-enp1s0-08:05:322
687999182,8cyclictest86550irq/126-enp1s0-08:35:172
687999182,5cyclictest6269-21ntpq10:45:242
687999181,6cyclictest0-21swapper/212:25:062
687999181,5cyclictest16228-21users12:20:282
6879991813,3cyclictest23756-21/usr/sbin/munin09:00:212
6879991812,4cyclictest21226-21/usr/sbin/munin12:35:142
686899189,7cyclictest1555-21kworker/u8:208:25:030
686899188,2cyclictest88050irq/133-enp2s0-07:46:040
686899187,2cyclictest5646-21kworker/0:208:20:280
686899187,2cyclictest5646-21kworker/0:208:20:280
686899184,5cyclictest3377-21needreboot09:25:200
686899183,9cyclictest7-21ksoftirqd/008:00:230
686899181,9cyclictest15587-21/usr/sbin/munin09:55:150
686899181,8cyclictest0-21swapper/012:14:070
686899181,7cyclictest0-21swapper/011:47:580
686899181,6cyclictest0-21swapper/009:30:090
686899181,5cyclictest32765-21taskset10:32:460
686899181,5cyclictest32765-21taskset10:32:460
686899181,5cyclictest11-21rcuc/010:25:190
686899181,5cyclictest0-21swapper/009:45:220
6868991812,3cyclictest25729-21threads10:15:270
686899181,10cyclictest0-21swapper/010:03:260
49302181,2sleep00-21swapper/010:44:320
210842181,3sleep20-21swapper/208:52:012
117342181,2sleep00-21swapper/007:17:430
117342181,2sleep00-21swapper/007:17:420
6879991712,3cyclictest8028-21cat07:10:232
6879991712,3cyclictest30-21rcuc/207:43:202
687999171,15cyclictest0-21swapper/208:29:252
6879991711,3cyclictest2489-21idleruntime08:10:192
686899174,3cyclictest87950irq/132-enp2s0-07:10:230
6868991713,3cyclictest7-21ksoftirqd/010:50:250
6868991711,3cyclictest14029-21basename11:05:130
686899170,7cyclictest14190-21kworker/0:312:25:400
686899162,6cyclictest780-21rngd08:12:310
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional