You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-13 - 11:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Sat Dec 13, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2632617212752,70sleep30-21swapper/319:05:583
263286928873,9sleep20-21swapper/219:09:102
263266828153,21sleep00-21swapper/019:06:350
263278527762,9sleep10-21swapper/119:08:041
2633054992623,1cyclictest0-21swapper/323:10:003
2633054992418,3cyclictest2891251-21turbostat23:15:003
2633054992320,1cyclictest0-21swapper/300:15:073
263305499221,17cyclictest2760100-21taskset21:06:333
263305499211,14cyclictest0-21swapper/323:15:173
28114652201,4sleep22807450-21/usr/sbin/munin21:55:182
263305499202,12cyclictest2729119-21taskset20:40:113
2633054992017,1cyclictest0-21swapper/322:50:003
2633054992017,1cyclictest0-21swapper/319:50:113
2633054992017,1cyclictest0-21swapper/300:25:113
2633054991916,1cyclictest0-21swapper/323:30:013
2633054991916,1cyclictest0-21swapper/321:30:173
2633054991916,1cyclictest0-21swapper/320:55:153
2633054991916,1cyclictest0-21swapper/319:30:123
2633054991915,2cyclictest0-21swapper/323:45:133
2633054991915,2cyclictest0-21swapper/322:55:103
2633054991915,1cyclictest0-21swapper/323:25:003
263305499191,14cyclictest0-21swapper/321:00:123
28702842181,2sleep00-21swapper/022:55:080
263305499188,7cyclictest2683954-21chrt19:55:153
263305499182,11cyclictest2743485-21taskset20:50:173
2633054991816,1cyclictest0-21swapper/321:55:103
2633054991815,1cyclictest0-21swapper/322:35:193
2633054991815,1cyclictest0-21swapper/300:10:173
263305499181,14cyclictest0-21swapper/322:40:143
263305499181,12cyclictest2791570-21taskset21:39:553
263305499181,12cyclictest0-21swapper/319:35:113
263305499181,12cyclictest0-21swapper/300:30:083
263305499181,10cyclictest2931701-21taskset23:50:133
263305499181,10cyclictest2774844-21taskset21:20:153
263305499180,14cyclictest0-21swapper/322:05:093
28274202171,8sleep22823342-21/usr/sbin/munin22:10:172
263305499171,4cyclictest0-21swapper/321:15:103
2633054991714,1cyclictest0-21swapper/322:15:183
2633054991714,1cyclictest0-21swapper/320:10:153
263305499171,13cyclictest2849198-21turbostat22:34:593
263305499171,10cyclictest2776243-21taskset21:25:003
263305499171,10cyclictest2671428-21taskset19:44:533
29336132161,10sleep02933799-21cat23:55:080
263305499166,6cyclictest2723366-21taskset20:31:543
263305499161,9cyclictest2823334-21taskset22:10:013
2633054991613,1cyclictest0-21swapper/321:10:153
2633054991613,1cyclictest0-21swapper/319:10:153
263305499161,13cyclictest0-21swapper/300:05:153
263305499161,12cyclictest2817585-21turbostat22:05:003
263305499161,12cyclictest0-21swapper/319:20:113
263305499161,12cyclictest0-21swapper/300:00:143
263305499161,10cyclictest2703042-21chrt20:15:063
263305499161,10cyclictest2676496-21taskset19:47:533
263305499160,13cyclictest0-21swapper/323:35:203
263305499160,13cyclictest0-21swapper/322:50:093
263305499160,13cyclictest0-21swapper/322:25:113
263305499160,13cyclictest0-21swapper/321:50:143
263305499160,13cyclictest0-21swapper/321:40:263
263305499160,13cyclictest0-21swapper/320:20:183
263305499160,13cyclictest0-21swapper/320:00:253
263305499160,13cyclictest0-21swapper/319:15:113
263305499160,12cyclictest0-21swapper/320:05:093
263305499160,10cyclictest2718321-21taskset20:28:533
27438062151,10sleep02744058-21/usr/sbin/munin20:50:230
27290972153,7sleep22729116-21apt-get20:40:112
26963042151,11sleep20-21swapper/220:05:172
26566352151,9sleep02657318-21tr19:30:160
2633054991512,2cyclictest0-21swapper/322:20:103
2633054991512,1cyclictest0-21swapper/323:30:153
2633054991512,1cyclictest0-21swapper/320:35:133
263305499151,10cyclictest2875669-21chrt23:00:113
263305499151,10cyclictest2738260-21taskset20:45:163
29584702141,9sleep12958521-21/usr/sbin/munin00:15:161
263305499141,9cyclictest2959960-21taskset00:20:103
2633054991411,2cyclictest0-21swapper/323:55:073
2633054991411,1cyclictest0-21swapper/323:40:123
2633054991411,1cyclictest0-21swapper/321:45:203
263305499140,12cyclictest0-21swapper/319:25:143
263305499140,12cyclictest0-21swapper/300:35:013
29750162131,8sleep12975028-21turbostat.cron00:35:001
28676892132,7sleep22868870-21/usr/sbin/munin22:50:132
27716062131,8sleep02772054-21cat21:20:130
27612092131,7sleep02762662-21kthreadcore21:10:150
26573192131,8sleep12659066-21kthreadcore19:30:171
29489932121,7sleep22949158-21latency_hist00:10:002
29482992121,7sleep02948308-21switchtime00:05:220
29065552121,7sleep22906802-21awk23:25:242
28637652121,5sleep12863818-21ntpq22:45:151
28498862121,3sleep12849986-21if_enp1s022:35:131
28484942121,4sleep02844241-21/usr/sbin/munin22:30:230
28447732121,7sleep22844847-21kthreadcore22:30:132
28091562121,7sleep10-21swapper/121:55:131
27663942121,6sleep22767169-21kthreadcore21:15:142
27140382120,8sleep02714954-21cut20:25:140
26911622120,2sleep10-21swapper/120:00:181
26563612121,7sleep22656454-21fschecks_time19:30:142
28655952111,6sleep02866086-21kthreadcore22:50:110
28497312111,5sleep02849883-21grep22:35:120
28285712111,6sleep02828586-21cpu22:15:080
28221222111,5sleep02822176-21ntpq22:05:180
28080202111,6sleep02809153-21kthreadcore21:55:120
27904192111,6sleep12790556-21awk21:35:171
27832792111,6sleep12785121-21grep21:30:161
27603702111,5sleep12760392-21munin-run21:10:001
27277952111,4sleep22727828-21ntpq20:35:182
26927742111,6sleep12692806-21fschecks_count20:05:121
26913502110,1sleep00-21swapper/020:00:220
26509202111,5sleep12651162-21df_inode19:25:121
29531942101,4sleep22953249-21ntpq00:10:162
29492482101,4sleep12949267-21apt-get00:10:071
29443512101,4sleep12944454-21perl00:05:101
29388422101,4sleep02939101-21df_inode00:00:080
29231402101,5sleep22923275-21tr23:45:082
29174302100,1sleep20-21swapper/223:37:182
29085712100,2sleep2171rcu_preempt23:30:132
28337922101,3sleep00-21swapper/022:20:090
28325702101,4sleep12832622-21ntp_states22:15:171
28047872101,4sleep12806042-21memory21:50:161
27968212100,6sleep22796846-21/usr/sbin/munin21:45:002
27923142101,4sleep12792430-21irqstats21:40:141
27538672101,4sleep22753915-21ntpq21:00:192
27434302101,4sleep22743477-21ntpq20:50:172
27433412101,4sleep12743429-21ntpq20:50:161
27149552101,5sleep10-21swapper/120:25:151
26922362101,4sleep22692285-21munin-run20:05:012
26670702101,5sleep12668123-21cut19:40:151
26558152101,5sleep02655821-21cron19:30:000
2975008291,4sleep02969836-21perf00:34:550
2964861291,3sleep22964880-21apt-get00:25:072
2963592291,3sleep12963638-21ntpq00:20:171
2955003291,4sleep02956404-21cut00:15:110
2928369291,4sleep12928387-21apt-get23:50:081
2923118291,3sleep12923137-21apt-get23:45:071
2913198290,5sleep12912699-21/usr/sbin/munin23:35:131
2884542291,3sleep22884683-21ntpq23:05:162
2879463291,4sleep12879514-21ntpq23:00:181
2849462291,4sleep22849545-21expr22:35:102
2822400291,4sleep12822628-21sed22:05:231
2817221290,1sleep20-21swapper/222:00:312
2802579291,4sleep22802798-21/usr/sbin/munin21:50:132
2790557291,3sleep22790600-21ntpq21:35:182
2776559291,4sleep12776776-21basename21:25:151
2776257291,4sleep22776279-21sshd21:25:122
2775091291,4sleep22775142-21cut21:20:192
2775035291,4sleep12775089-21ntpq21:20:181
2722923291,4sleep12723129-21grep20:30:241
2718695291,4sleep02718948-21df20:30:110
2703618291,4sleep22705028-21cut20:15:152
2701811291,4sleep12701862-21ntpq20:10:211
2701764291,4sleep02701810-21ld20:10:200
2693457291,4sleep02695151-21cut20:05:150
2677202290,1sleep10-21swapper/119:50:131
2675642291,3sleep02675680-21ntpq19:45:190
2960643281,3sleep22962367-21tr00:20:142
2939102281,3sleep12939226-21grep00:00:091
2936023281,3sleep12937351-21memory23:55:131
2907496280,3sleep10-21swapper/123:30:081
2902250281,2sleep00-21swapper/023:25:100
2897586281,3sleep22897979-21kthreadcore23:20:152
2890371280,1sleep00-21swapper/023:10:190
2887067281,3sleep22888764-21cut23:10:132
2885016281,3sleep12885303-21unixbench_singl23:05:231
2875251280,1sleep00-21swapper/023:00:010
2875083280,1sleep10-21swapper/122:57:531
2866087281,3sleep12867684-21kthreadcore22:50:121
2859582281,3sleep22859609-21/usr/sbin/munin22:45:002
2859459280,1sleep10-21swapper/122:43:271
2842942281,3sleep12842993-21ntpq22:25:151
2839584281,3sleep22841012-21cut22:25:122
2834070281,3sleep22834123-21fschecks_count22:20:112
2827308281,2sleep02827358-21ntpq22:10:150
2803101281,3sleep02804785-21cut21:50:150
2801699280,1sleep10-21swapper/121:45:551
2786116280,1sleep00-21swapper/021:32:000
2762664280,1sleep10-21swapper/121:10:161
2759557280,1sleep00-21swapper/021:05:220
2754018280,1sleep10-21swapper/121:00:221
2744691281,2sleep20-21swapper/220:55:102
2678025281,3sleep00-21swapper/019:50:160
2672306281,3sleep12673751-21kthreadcore19:45:161
2645701281,2sleep12645720-21apt-get19:20:111
2640713281,4sleep12640818-21fschecks_time19:15:141
2638751281,3sleep22639659-21kthreadcore19:15:032
2637291281,3sleep02637381-21ps19:10:200
2975454270,1sleep20-21swapper/200:37:262
2929908271,2sleep22931700-21grep23:50:142
2923409271,3sleep00-21swapper/023:45:100
2922584270,1sleep10-21swapper/123:41:271
2916957271,3sleep00-21swapper/023:35:240
2892049273,1sleep20-21swapper/223:15:102
2886746270,3sleep12887066-21kthreadcore23:10:121
2876109271,3sleep20-21swapper/223:00:152
2864662270,1sleep20-21swapper/222:48:122
2859503270,1sleep00-21swapper/022:44:020
2844541271,3sleep12844772-21cat22:30:121
2834818271,2sleep10-21swapper/122:20:141
2832085271,2sleep22832456-21strings22:15:142
2818098270,1sleep20-21swapper/222:05:042
2817558270,1sleep10-21swapper/122:04:511
2802009270,5sleep20-21swapper/221:49:532
2786108270,1sleep20-21swapper/221:31:542
2781131270,1sleep00-21swapper/021:29:470
2770549270,1sleep00-21swapper/021:17:090
2760098270,1sleep20-21swapper/221:06:332
2749431270,1sleep10-21swapper/120:57:261
2739257271,3sleep12739399-21latency_hist20:50:001
2736370271,3sleep02738092-21latency20:45:140
2728678270,1sleep10-21swapper/120:38:181
2708876271,2sleep10-21swapper/120:20:141
2707838270,1sleep00-21swapper/020:17:490
2686670270,1sleep00-21swapper/019:55:370
2677273271,3sleep20-21swapper/219:50:142
2671881271,3sleep20-21swapper/219:45:132
2661279270,2sleep20-21swapper/219:35:112
2975559260,4sleep10-21swapper/100:38:461
2975330260,1sleep00-21swapper/000:35:510
2974854260,1sleep20-21swapper/200:33:052
2969611260,1sleep10-21swapper/100:27:311
2960010261,2sleep00-21swapper/000:20:120
2953902260,1sleep00-21swapper/000:11:350
2943780260,4sleep20-21swapper/200:04:352
2938216260,3sleep22938217-21sshd23:56:142
2919532261,2sleep02921335-21kthreadcore23:40:170
2903420261,2sleep10-21swapper/123:25:151
2901766260,1sleep00-21swapper/023:22:090
2901727260,1sleep10-21swapper/123:21:381
2885516260,4sleep00-21swapper/023:07:420
2873727260,2sleep22874199-21strings22:55:142
2812625260,4sleep00-21swapper/022:00:050
2795383260,2sleep02795776-21strings21:40:170
2765386260,4sleep20-21swapper/221:12:382
2729376261,2sleep00-21swapper/020:40:130
2728667260,1sleep00-21swapper/020:38:090
2716722261,2sleep20-21swapper/220:25:162
2707664260,1sleep10-21swapper/120:15:381
2686739260,1sleep10-21swapper/119:56:301
2682670261,2sleep20-21swapper/219:55:152
2670495260,1sleep20-21swapper/219:40:202
2646697261,2sleep20-21swapper/219:20:162
2644448260,2sleep00-21swapper/019:15:210
2637177260,2sleep20-21swapper/219:10:182
2633722261,2sleep12633800-21/usr/sbin/munin19:10:131
2964656250,2sleep00-21swapper/000:25:020
2922585250,4sleep20-21swapper/223:41:272
2912408250,4sleep00-21swapper/023:34:360
2892363250,2sleep00-21swapper/023:15:120
2839301251,1sleep00-21swapper/022:25:100
2769680250,1sleep10-21swapper/121:15:171
2744858250,2sleep00-21swapper/020:55:120
2734290250,1sleep20-21swapper/220:45:082
2723345250,3sleep20-21swapper/220:31:382
2708294250,1sleep00-21swapper/020:20:090
2670349250,1sleep00-21swapper/019:40:180
2665530250,1sleep00-21swapper/019:35:250
2661270250,1sleep10-21swapper/119:35:061
2655476250,1sleep20-21swapper/219:25:412
2954262240,1sleep20-21swapper/200:15:012
2928367240,1sleep00-21swapper/023:50:070
2891925240,1sleep10-21swapper/123:15:091
2859831240,2sleep00-21swapper/022:45:020
2823363240,1sleep10-21swapper/122:10:081
2797374240,1sleep00-21swapper/021:45:140
2786790240,1sleep00-21swapper/021:35:120
2749705240,1sleep00-21swapper/021:00:010
2729447240,1sleep10-21swapper/120:40:141
2708460240,1sleep20-21swapper/220:20:112
2697516240,1sleep20-21swapper/220:10:012
2645692240,1sleep00-21swapper/019:20:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional