You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-13 - 19:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Oct 13, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
269512241164,25sleep00-21swapper/007:09:290
266942239163,24sleep30-21swapper/307:06:103
269302234162,58sleep10-21swapper/107:09:141
268122225177,38sleep20-21swapper/207:07:422
1582121510,5sleep32709999cyclictest07:55:163
2648021480,4sleep02708299cyclictest10:45:210
2139621420,3sleep12708799cyclictest10:35:151
26332680,7sleep339-21rcuc/309:50:293
27093996057,2cyclictest0-21swapper/207:30:422
2709999441,29cyclictest23653-21munin-run11:55:013
2709999381,8cyclictest1209-21unixbench_singl11:00:263
2709999381,27cyclictest17291-21/usr/sbin/munin11:40:193
2709999380,28cyclictest0-21swapper/310:10:183
2709999376,20cyclictest3458-21/usr/sbin/munin09:55:173
2709999372,25cyclictest26163-21latency_hist08:20:023
2709999371,25cyclictest27657-21/usr/sbin/munin10:50:143
2709999371,25cyclictest27657-21/usr/sbin/munin10:50:143
2709999368,17cyclictest17595-21/usr/sbin/munin08:00:173
2709999366,15cyclictest16033-21/usr/sbin/munin09:10:263
2709999361,25cyclictest9426-21/usr/sbin/munin08:55:163
2709999361,24cyclictest32457-21latency_hist12:15:013
27093993633,2cyclictest0-21swapper/209:46:392
27093993633,2cyclictest0-21swapper/209:46:392
2709999354,22cyclictest780-21rngd07:11:273
27099993516,16cyclictest2000-21idleruntime-cro07:25:003
2709999351,3cyclictest8568-21/usr/sbin/munin10:05:183
2709999351,18cyclictest984-21cut08:35:173
2709999344,17cyclictest88250irq/135-enp2s0-10:30:153
2709999343,24cyclictest86750irq/128-enp1s0-09:43:193
2709999343,18cyclictest86550irq/126-enp1s0-11:35:193
2709999342,23cyclictest23565-21missed_timers09:25:193
2709999341,30cyclictest24821-21/usr/sbin/munin09:30:233
2708799341,16cyclictest0-21swapper/110:20:151
2709999337,23cyclictest18577-21diskstats09:15:173
2709999335,15cyclictest0-21swapper/311:18:323
2709999331,7cyclictest0-21swapper/310:16:433
2709999331,5cyclictest0-21swapper/311:05:123
2709999331,23cyclictest0-21swapper/312:20:123
2709999331,20cyclictest848-21sssd_nss12:15:303
2709999325,13cyclictest86550irq/126-enp1s0-10:45:013
2709999324,15cyclictest86850irq/129-enp1s0-09:35:293
2709999323,2cyclictest86550irq/126-enp1s0-11:47:013
2709999323,24cyclictest3480-21smart_sda07:25:283
2709999321,2cyclictest0-21swapper/312:35:243
2709999321,27cyclictest15797-21diskmemload10:01:003
2709999321,27cyclictest14891-21ntp_states09:05:223
27099993212,17cyclictest31168-21grep08:30:143
2709999321,19cyclictest30235-21sshd12:09:513
2708799321,20cyclictest0-21swapper/111:34:571
2708799321,13cyclictest0-21swapper/107:48:311
2708799321,11cyclictest0-21swapper/111:45:271
2708799321,10cyclictest0-21swapper/110:26:091
2708299329,11cyclictest14856-21idleruntime-cro11:35:020
2709999313,2cyclictest86750irq/128-enp1s0-09:01:053
2709999312,4cyclictest29696-21smart_sda07:15:043
2709999311,2cyclictest0-21swapper/312:25:373
2709999311,27cyclictest11723-21nfs4_client11:25:243
2709999311,17cyclictest23873-21/usr/sbin/munin11:55:093
2709999310,4cyclictest0-21swapper/311:10:363
2708799311,15cyclictest0-21swapper/107:15:111
2708799311,13cyclictest0-21swapper/109:43:571
2708799310,13cyclictest0-21swapper/108:10:011
2708799310,12cyclictest0-21swapper/108:20:191
2709999304,16cyclictest780-21rngd07:32:073
2709999301,4cyclictest0-21swapper/309:45:203
2709999301,4cyclictest0-21swapper/309:45:203
2709999301,3cyclictest31126-21/usr/sbin/munin10:55:263
2709999301,3cyclictest26839-21awk12:00:183
2709999301,3cyclictest26839-21awk12:00:183
2709999301,3cyclictest0-21swapper/311:31:443
2709999301,3cyclictest0-21swapper/307:35:233
27099993013,14cyclictest14595-21grep10:20:123
2709999301,2cyclictest8595-21cut11:20:143
2709999301,2cyclictest0-21swapper/312:32:013
27099993010,17cyclictest23967-21latency_hist08:15:013
2708799302,11cyclictest22813-21irqstats08:10:191
2708799301,27cyclictest0-21swapper/110:10:271
2708799301,12cyclictest7736-21irqstats12:30:171
2708799301,11cyclictest0-21swapper/111:50:251
2708799300,14cyclictest0-21swapper/112:10:271
2708299306,11cyclictest23984-21idleruntime-cro08:15:010
2709999294,1cyclictest0-21swapper/310:45:253
2709999293,12cyclictest780-21rngd08:50:393
2709999292,23cyclictest29671-21ntp_states08:25:233
2709999291,6cyclictest22294-21chrt09:23:513
27099992912,14cyclictest10999-21/usr/sbin/munin07:45:153
2709999291,16cyclictest15797-21diskmemload10:28:313
27099992911,15cyclictest21778-21awk08:10:013
2708799291,25cyclictest0-21swapper/112:17:471
2708799291,12cyclictest0-21swapper/107:53:171
2708799291,10cyclictest0-21swapper/109:07:411
2709999287,18cyclictest0-21swapper/308:45:243
2709999287,18cyclictest0-21swapper/308:45:243
2709999284,12cyclictest780-21rngd07:44:033
2709999281,4cyclictest21414-21/usr/sbin/munin10:35:163
2709399281,6cyclictest0-21swapper/211:10:242
2708799282,15cyclictest1779-40dnf07:22:321
2708799281,9cyclictest0-21swapper/109:02:571
2708799281,3cyclictest0-21swapper/108:15:091
2708799281,21cyclictest0-21swapper/111:59:331
2708799281,13cyclictest0-21swapper/108:03:261
2708799281,12cyclictest27073-21cyclictest09:35:171
2708799281,12cyclictest0-21swapper/110:50:311
2708799281,12cyclictest0-21swapper/110:50:311
2708799281,12cyclictest0-21swapper/110:30:151
2708799281,10cyclictest0-21swapper/111:16:031
2709999272,4cyclictest3186-21/usr/sbin/munin08:40:153
2709999271,4cyclictest26734-21df08:20:173
2708799271,9cyclictest0-21swapper/107:29:411
2708799271,17cyclictest27565-21grep12:00:271
2708799271,17cyclictest27565-21grep12:00:271
27087992711,4cyclictest10196-21ntp_states12:35:271
2708799271,12cyclictest0-21swapper/109:15:071
2708799271,11cyclictest0-21swapper/112:27:011
2708799271,11cyclictest0-21swapper/107:41:251
2708799271,10cyclictest0-21swapper/109:27:191
2708799270,12cyclictest0-21swapper/109:23:431
27082992710,14cyclictest25854-21munin-run12:00:020
2709999261,3cyclictest13755-21perl07:50:163
27093992614,7cyclictest0-21swapper/208:02:362
2708799264,3cyclictest0-21swapper/111:05:031
2708799261,9cyclictest0-21swapper/110:43:131
2708799261,15cyclictest6184-21sed11:15:011
2708799261,12cyclictest0-21swapper/110:48:591
2708799261,11cyclictest0-21swapper/108:33:251
2708799261,11cyclictest0-21swapper/107:57:181
2708799261,10cyclictest0-21swapper/112:07:371
27087992610,4cyclictest26410-21users09:30:271
27082992622,2cyclictest6588-21unixbench_singl08:45:270
27082992622,2cyclictest6588-21unixbench_singl08:45:270
27082992616,7cyclictest28809-21awk10:50:260
27082992616,7cyclictest28809-21awk10:50:260
2708299261,5cyclictest0-21swapper/012:07:170
27082992610,8cyclictest24313-21kworker/u8:109:29:090
2708799254,3cyclictest0-21swapper/109:12:151
2708799251,14cyclictest395-21irqstats11:00:171
2708799251,13cyclictest9933-21latency_hist10:10:011
2708799251,10cyclictest0-21swapper/112:23:291
2708799250,13cyclictest86550irq/126-enp1s0-09:45:231
2708799250,13cyclictest86550irq/126-enp1s0-09:45:231
2708299251,6cyclictest0-21swapper/010:15:420
27093992418,3cyclictest23551-21grep08:10:272
2708799246,3cyclictest0-21swapper/111:38:391
2708799245,3cyclictest0-21swapper/108:27:591
2708799244,3cyclictest0-21swapper/110:19:551
2708799241,16cyclictest0-21swapper/111:41:411
2708799241,12cyclictest0-21swapper/111:25:571
2708799241,12cyclictest0-21swapper/110:04:531
2708799241,12cyclictest0-21swapper/108:45:021
2708799241,12cyclictest0-21swapper/108:45:011
2708799241,10cyclictest0-21swapper/111:21:331
2708299242,10cyclictest541-21/usr/sbin/munin08:35:170
27082992416,5cyclictest7762-21/usr/sbin/munin07:35:250
2708299241,5cyclictest0-21swapper/012:20:570
2708299241,10cyclictest1692-21timerandwakeup12:15:330
2708299241,10cyclictest13135-21sed09:00:280
2708299241,10cyclictest11601-21grep07:45:140
296112230,4sleep22709399cyclictest09:40:162
2709399235,9cyclictest2596-21cpuspeed_turbos12:20:122
27093992319,3cyclictest0-21swapper/209:51:462
27093992317,3cyclictest659-21cpuspeed_turbos08:35:152
2709399231,16cyclictest0-21swapper/208:05:112
2708799238,3cyclictest0-21swapper/109:51:551
2708799231,3cyclictest0-21swapper/108:42:331
2708799231,14cyclictest0-21swapper/108:53:341
2708799231,13cyclictest0-21swapper/107:37:171
2708799231,11cyclictest0-21swapper/110:55:251
2708799231,11cyclictest0-21swapper/108:37:491
2708299234,5cyclictest16314-40dnf10:23:420
27082992319,3cyclictest0-21swapper/012:10:260
2708299231,7cyclictest7989-21if_enp2s008:50:190
2708299231,16cyclictest0-21swapper/011:25:260
2708299231,11cyclictest32682-21smart_sda09:45:260
2708299231,11cyclictest32682-21smart_sda09:45:250
2708299231,11cyclictest27715-21sshd12:00:520
2708299231,11cyclictest27715-21sshd12:00:520
2708299231,10cyclictest9426-21/usr/sbin/munin08:55:210
2708299231,10cyclictest6483-21cstates11:15:180
2708299231,10cyclictest3070-21wc11:05:240
2708299231,10cyclictest24243-21needreboot10:40:220
2708299231,10cyclictest15857-21cut07:55:160
2709399225,4cyclictest0-21swapper/210:50:012
2709399221,4cyclictest0-21swapper/211:25:412
2709399221,16cyclictest0-21swapper/208:31:442
2708799227,3cyclictest0-21swapper/107:31:311
2708799225,3cyclictest0-21swapper/108:58:351
2708799221,18cyclictest29027-21sshd07:12:171
2708799221,14cyclictest0-21swapper/109:57:271
2708299228,2cyclictest32650-21gltestperf11:00:150
2708299227,3cyclictest2233-21cpu07:25:160
2708299227,3cyclictest11464-21timerwakeupswit10:10:270
2708299222,7cyclictest6329-21hddtemp_smartct10:00:180
27082992219,1cyclictest0-21swapper/009:35:200
2708299221,8cyclictest11-21rcuc/011:45:240
2708299221,7cyclictest27684-21switchtime08:20:290
2708299221,10cyclictest1240-21/usr/sbin/munin09:50:170
2709399216,9cyclictest0-21swapper/210:51:242
2709399216,9cyclictest0-21swapper/210:51:242
2709399216,3cyclictest86550irq/126-enp1s0-12:05:272
2708299218,2cyclictest0-21swapper/007:50:250
2708299217,3cyclictest29516-21memory08:25:200
2708299217,2cyclictest19457-21grep10:30:160
2708299211,9cyclictest0-21swapper/010:25:170
27082992119,1cyclictest0-21swapper/008:40:260
2708299211,8cyclictest16263-21ps11:35:260
2708299211,8cyclictest15182-21sed09:05:250
2708299211,7cyclictest5712-21needreboot12:25:200
2708299211,7cyclictest0-21swapper/012:30:110
27082992113,6cyclictest0-21swapper/007:22:330
2709399206,3cyclictest24098-21runrttasks10:40:192
2709399206,3cyclictest12050irq/123-ahci[0009:23:312
2709399205,4cyclictest17995-21idleruntime11:40:192
2709399205,3cyclictest86550irq/126-enp1s0-10:15:012
2709399203,4cyclictest1361-21ntp_kernel_pll_12:15:282
2709399201,7cyclictest3540-21cat07:25:282
2709399201,7cyclictest2842-21basename08:40:122
2709399201,7cyclictest26215-21cat08:20:022
2709399201,7cyclictest15151-21grep11:35:152
2709399201,7cyclictest0-21swapper/212:00:222
2709399201,7cyclictest0-21swapper/212:00:222
2709399201,6cyclictest14164-21df09:05:132
2709399201,16cyclictest404-21uname11:00:182
2709399201,10cyclictest32-21ksoftirqd/211:30:272
2709399200,15cyclictest0-21swapper/210:36:542
2708299208,3cyclictest0-21swapper/009:20:370
2708299206,3cyclictest86750irq/128-enp1s0-08:30:170
2708299203,4cyclictest0-21swapper/009:56:320
2708299201,9cyclictest0-21swapper/009:11:470
2708299201,9cyclictest0-21swapper/008:15:160
2708299201,8cyclictest0-21swapper/007:40:250
2708299201,7cyclictest15797-21diskmemload12:36:080
2708299201,7cyclictest0-21swapper/011:53:460
2708299201,6cyclictest30188-21df10:55:150
2708299201,6cyclictest0-21swapper/009:32:040
2708299201,3cyclictest11-21rcuc/011:20:180
2708299201,17cyclictest0-21swapper/010:35:220
2708299201,10cyclictest0-21swapper/009:15:150
2708299200,9cyclictest0-21swapper/009:40:190
2708299200,18cyclictest0-21swapper/007:15:230
82662191,3sleep20-21swapper/208:50:232
2709399196,3cyclictest86850irq/129-enp1s0-11:18:322
2709399195,3cyclictest86750irq/128-enp1s0-10:20:182
2709399195,3cyclictest629-21idleruntime07:20:182
2709399194,3cyclictest8516-21wc11:20:142
2709399193,4cyclictest14634-21snmpd07:47:502
2709399191,7cyclictest0-21swapper/210:00:172
2709399191,7cyclictest0-21swapper/209:30:172
2709399191,7cyclictest0-21swapper/209:18:192
2709399191,6cyclictest6988-21egrep12:30:122
2709399191,6cyclictest4728-21/usr/sbin/munin12:25:192
2709399191,6cyclictest23297-21hddtemp_smartct09:25:162
2709399191,6cyclictest15263-21cat07:55:012
2709399191,6cyclictest0-21swapper/211:49:222
2709399191,6cyclictest0-21swapper/210:15:052
2709399191,6cyclictest0-21swapper/209:56:182
2709399191,6cyclictest0-21swapper/209:12:102
2709399191,6cyclictest0-21swapper/207:10:152
2709399191,5cyclictest22714-21ntp_kernel_err11:50:232
2709399191,5cyclictest16745-21cpuspeed_turbos10:25:132
2709399191,5cyclictest15517-21grep07:55:132
2709399191,5cyclictest0-21swapper/212:38:072
2709399191,4cyclictest0-21swapper/212:11:342
27093991913,3cyclictest8674-21irqstats10:05:202
2709399191,16cyclictest0-21swapper/211:55:172
2709399191,12cyclictest0-21swapper/210:31:072
2709399191,10cyclictest0-21swapper/210:57:092
2709399191,10cyclictest0-21swapper/207:38:512
2708299197,3cyclictest17207-21kworker/0:108:08:130
2708299191,6cyclictest1522-21kworker/0:111:10:260
2708299191,16cyclictest0-21swapper/010:05:210
2708299191,15cyclictest81ktimersoftd/007:10:190
2708299190,2cyclictest18461-21proc_pri11:40:260
2709399187,2cyclictest0-21swapper/207:41:022
2709399183,3cyclictest86550irq/126-enp1s0-08:20:302
2709399182,8cyclictest22135-21kworker/2:008:27:312
2709399181,6cyclictest0-21swapper/211:07:392
2709399181,5cyclictest27672-21grep09:35:192
2709399181,14cyclictest5170-21cpuspeed_turbos08:45:132
2709399181,14cyclictest5170-21cpuspeed_turbos08:45:132
2708299181,5cyclictest18405-21irqstats08:00:190
133062181,3sleep20-21swapper/209:01:192
27093991712,3cyclictest30518-21sh07:15:152
27093991610,3cyclictest9927-21fschecks_time08:55:172
27082991611,3cyclictest4395-21date07:30:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional