You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 06:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Thu Nov 27, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
281139022240,2sleep02811392-21find23:40:210
2439809212860,64sleep30-21swapper/319:09:303
25947312880,9sleep00-21swapper/021:30:150
243951628065,9sleep20-21swapper/219:05:492
243948928060,13sleep00-21swapper/019:05:270
243961527661,10sleep10-21swapper/119:07:011
2439979993228,2cyclictest0-21swapper/300:15:263
2439979993127,1cyclictest0-21swapper/321:35:013
27980582300,2sleep12798057-21ssh23:34:281
243997999291,23cyclictest2589696-21chrt21:25:373
2439979992824,2cyclictest0-21swapper/323:25:193
2439979992724,1cyclictest0-21swapper/321:10:003
2439979992613,9cyclictest2765382-21sleep23:15:003
24847142251,10sleep02484902-21expr19:50:110
2439979992522,1cyclictest0-21swapper/300:25:143
243997999251,20cyclictest2580717-21taskset21:20:253
2439979992420,2cyclictest0-21swapper/321:55:113
2439979992419,3cyclictest0-21swapper/323:40:133
2439979992411,9cyclictest2824125-21taskset23:50:173
27087222230,2sleep30-21swapper/322:40:103
25809852230,3sleep02559572-21diskmemload21:20:250
243997999232,16cyclictest2609620-21chrt21:40:133
2439979992319,2cyclictest0-21swapper/323:55:183
243997999231,17cyclictest2819466-21chrt23:45:213
243997999231,12cyclictest2655508-21chrt22:05:353
28642152221,15sleep12864422-21expr00:15:151
28394402220,9sleep134-21ksoftirqd/123:59:461
26424042221,15sleep22642418-21apt-get22:00:142
243997999222,14cyclictest2886736-21taskset00:30:103
243997999222,14cyclictest2665613-21taskset22:13:283
243997999222,14cyclictest2633893-21taskset21:55:003
243997999222,12cyclictest2683645-21taskset22:25:073
2439979992219,1cyclictest0-21swapper/321:40:003
2439979992219,1cyclictest0-21swapper/319:20:243
2439979992218,2cyclictest0-21swapper/323:05:153
2439979992218,2cyclictest0-21swapper/323:05:153
2439979992218,2cyclictest0-21swapper/300:35:253
2439979992217,3cyclictest0-21swapper/322:22:113
243997999221,14cyclictest2648203-21taskset22:01:543
243997999212,13cyclictest2794108-21taskset23:30:183
243997999212,13cyclictest2697378-21taskset22:31:243
2439979992118,2cyclictest0-21swapper/322:55:003
2439979992118,1cyclictest0-21swapper/300:10:003
2439979992117,2cyclictest0-21swapper/322:39:563
243997999211,15cyclictest2721135-21chrt22:45:243
27828022201,8sleep02784110-21kthreadcore23:25:190
243997999202,6cyclictest26750252chrt22:19:543
2439979992016,2cyclictest0-21swapper/323:01:333
2439979992016,2cyclictest0-21swapper/321:15:133
2439979992016,2cyclictest0-21swapper/321:15:123
243997999201,3cyclictest0-21swapper/319:52:093
2439979992012,4cyclictest28739362sleep300:20:183
243997999201,15cyclictest2781079-21chrt23:23:563
243997999201,14cyclictest2617475-21chrt21:45:093
243997999201,14cyclictest2565402-21chrt21:11:303
28704342191,11sleep02559572-21diskmemload00:18:180
26547672191,5sleep12654823-21ssh22:05:211
24624422191,13sleep22462503-21/usr/sbin/munin19:25:212
243997999192,12cyclictest2770945-21taskset23:16:023
243997999192,11cyclictest2799385-21taskset23:35:153
2439979991916,1cyclictest0-21swapper/320:05:163
2439979991916,1cyclictest0-21swapper/319:55:183
2439979991915,2cyclictest0-21swapper/320:45:163
2439979991915,2cyclictest0-21swapper/320:34:593
243997999191,14cyclictest0-21swapper/322:55:063
243997999191,10cyclictest2511088-21taskset20:15:073
243997999191,10cyclictest2511088-21taskset20:15:063
28835152181,5sleep12883563-21ntpq00:25:211
28788432181,5sleep02878851-21ssh00:24:280
28217502180,2sleep20-21swapper/223:47:582
27380962181,2sleep00-21swapper/022:55:510
26804042181,3sleep00-21swapper/022:20:320
25762042182,11sleep12576284-21cpuspeed_turbos21:20:121
243997999188,7cyclictest2521232-21chrt20:23:083
243997999182,11cyclictest2839977-21taskset00:00:083
2439979991815,1cyclictest0-21swapper/300:10:243
243997999181,10cyclictest2483276-21taskset19:45:163
27328672171,4sleep12559572-21diskmemload22:55:041
27161842171,4sleep02559572-21diskmemload22:44:410
27156882171,5sleep22559572-21diskmemload22:43:552
26976612171,3sleep00-21swapper/022:31:510
25978462171,2sleep20-21swapper/221:30:392
25915012171,4sleep22559572-21diskmemload21:28:262
25906992170,2sleep12559572-21diskmemload21:27:091
2439979991714,2cyclictest0-21swapper/320:50:113
2439979991713,2cyclictest0-21swapper/319:15:213
2439979991711,3cyclictest2527282-21turbostat20:39:553
243997999171,13cyclictest0-21swapper/319:40:203
243997999171,10cyclictest2549083-21taskset21:00:003
28855892160,6sleep02559572-21diskmemload00:28:210
28507812161,10sleep22852205-21/usr/sbin/munin00:05:182
28220122161,3sleep10-21swapper/123:48:211
28111372162,7sleep12811190-21ntpq23:40:181
27802372160,8sleep00-21swapper/023:22:420
27711872161,10sleep22771195-21rm23:16:272
26656252161,9sleep02559572-21diskmemload22:13:300
261340321610,3sleep0221rcuc/021:40:180
26065332161,4sleep22559572-21diskmemload21:36:232
26049792161,3sleep00-21swapper/021:35:180
25803982161,10sleep22580458-21ntpq21:20:212
25665382161,2sleep20-21swapper/221:13:192
25664872161,9sleep12559572-21diskmemload21:13:141
24690122161,10sleep12469148-21diskstats19:35:141
243997999161,13cyclictest0-21swapper/319:10:213
243997999161,12cyclictest0-21swapper/319:25:193
243997999161,11cyclictest0-21swapper/321:00:113
243997999161,10cyclictest2521932-21taskset20:28:543
243997999160,13cyclictest0-21swapper/319:30:133
243997999160,10cyclictest2500158-21chrt20:03:533
28481932151,7sleep12848460-21df00:05:131
28395762151,9sleep21-21systemd00:00:002
28287602151,3sleep22559572-21diskmemload23:51:052
27710062151,2sleep10-21swapper/123:16:101
27580662159,2sleep0221rcuc/023:10:140
26408922151,10sleep22640906-21ssh21:58:202
25989602153,2sleep1171rcu_preempt21:32:221
25685882151,9sleep22569626-21grep21:15:152
25685882151,9sleep22569626-21grep21:15:152
25585512151,4sleep02558605-21ntpq21:05:180
243997999151,9cyclictest2473880-21taskset19:39:513
243997999151,3cyclictest0-21swapper/320:10:143
2439979991513,1cyclictest0-21swapper/320:40:103
28794702141,2sleep20-21swapper/200:25:142
28280392141,8sleep12828296-21unixbench_multi23:50:291
28255812141,8sleep02827350-21grep23:50:200
28151942142,6sleep02815218-21apt-get23:45:110
28125532141,7sleep22559572-21diskmemload23:41:312
27957402141,9sleep22795752-21rm23:31:012
27841132141,4sleep12785744-21tr23:25:201
27794282141,8sleep12559572-21diskmemload23:21:301
27240932141,8sleep22559572-21diskmemload22:49:252
26887372140,3sleep22559572-21diskmemload22:25:432
26767262141,8sleep12678345-21tr22:20:171
26173922140,2sleep00-21swapper/021:45:040
25231702141,8sleep02524854-21cut20:30:120
24574152141,8sleep22457632-21unixbench_multi19:20:282
24571402141,8sleep12457190-21ntpq19:20:221
24514062141,7sleep12451527-21ntpq19:15:181
24407262141,2sleep00-21swapper/019:10:150
28946752131,6sleep12894801-21perl00:35:151
28866962131,7sleep12559572-21diskmemload00:30:101
28639062132,5sleep22863928-21apt-get00:15:122
28473332130,3sleep10-21swapper/100:04:071
28374382131,6sleep00-21swapper/023:56:460
27872182131,6sleep22559572-21diskmemload23:25:412
26757822131,8sleep22675880-21tune2fs22:20:142
26570642131,5sleep00-21swapper/022:07:570
26500942131,6sleep12559572-21diskmemload22:04:501
26495562131,6sleep02559572-21diskmemload22:04:000
26172912131,8sleep22617380-21sh21:45:012
25734322131,7sleep12559572-21diskmemload21:16:231
25734322131,7sleep12559572-21diskmemload21:16:231
25447812131,7sleep22546422-21kthreadcore20:55:152
25399882130,2sleep00-21swapper/020:50:150
28867022121,5sleep22886731-21apt-get00:30:102
27954832120,2sleep0171rcu_preempt23:30:400
27817972121,5sleep22781831-21munin-run23:24:592
27636262121,7sleep12559572-21diskmemload23:12:241
27488652121,7sleep00-21swapper/023:04:500
27469972121,5sleep12559572-21diskmemload23:01:581
27316712121,7sleep22731682-21diskmemload22:53:372
27068302121,7sleep02706838-21sh22:38:230
26983742120,1sleep10-21swapper/122:32:561
26882282121,7sleep12688455-21/usr/sbin/munin22:25:251
26744352121,5sleep12559572-21diskmemload22:19:001
26081452121,7sleep12608156-21sh21:38:521
25900232121,7sleep02590026-21ssh21:26:080
25682902121,7sleep02568427-21gltestperf21:15:130
25682902121,7sleep02568427-21gltestperf21:15:120
25334442121,7sleep02533463-21apt-get20:45:100
25281982121,7sleep22528224-21cpu20:40:072
24995582121,7sleep12499753-21/usr/sbin/munin20:00:271
24960942121,7sleep02497752-21kthreadcore20:00:180
24869672121,7sleep12488537-21/usr/sbin/munin19:50:171
28983512111,5sleep22898492-21ntpq00:35:202
28791042111,5sleep12879160-21munin-run00:25:001
28616162111,4sleep12559572-21diskmemload00:11:421
28611872111,3sleep00-21swapper/000:10:570
28535142111,3sleep00-21swapper/000:06:160
28466162111,3sleep00-21swapper/000:02:590
28039562111,3sleep20-21swapper/223:35:592
28032432111,6sleep02803268-21grep23:35:250
27630562111,3sleep20-21swapper/223:11:322
27548632111,3sleep00-21swapper/023:06:340
27548632111,3sleep00-21swapper/023:06:330
27543112111,4sleep12559572-21diskmemload23:05:431
27543112111,4sleep12559572-21diskmemload23:05:421
27407792111,3sleep20-21swapper/222:59:592
27320472111,3sleep10-21swapper/122:54:131
27320342111,4sleep02559572-21diskmemload22:54:120
27239592111,4sleep12559572-21diskmemload22:49:101
27129332111,4sleep12559572-21diskmemload22:40:221
27049342111,4sleep22559572-21diskmemload22:35:292
26984672111,4sleep22559572-21diskmemload22:33:052
26639132111,4sleep12559572-21diskmemload22:10:551
26591482111,3sleep20-21swapper/222:10:162
26562122111,4sleep22559572-21diskmemload22:06:382
26331212111,4sleep22559572-21diskmemload21:54:002
26312092111,4sleep02559572-21diskmemload21:51:010
26225412111,3sleep10-21swapper/121:45:451
25663492111,4sleep02559572-21diskmemload21:13:010
24536552112,5sleep02454290-21tr19:20:170
29002842101,3sleep02559572-21diskmemload00:37:350
28618102101,2sleep20-21swapper/200:12:022
28458482100,4sleep22559572-21diskmemload00:01:482
28040292101,2sleep10-21swapper/123:36:061
27723762101,3sleep00-21swapper/023:18:170
27493662101,3sleep22559572-21diskmemload23:05:122
27493662101,3sleep22559572-21diskmemload23:05:122
27477832101,3sleep22559572-21diskmemload23:03:122
27216282101,2sleep00-21swapper/022:45:330
27062852100,1sleep10-21swapper/122:37:321
26837742101,4sleep02683999-21diskstats22:25:120
26725932101,3sleep22559572-21diskmemload22:16:132
26677942101,4sleep02668132-21cut22:15:140
26402572101,3sleep10-21swapper/121:57:201
26395872101,3sleep00-21swapper/021:56:190
26302712101,3sleep12559572-21diskmemload21:50:221
26093032101,3sleep12559572-21diskmemload21:40:121
25542922101,4sleep12554341-21munin-run21:05:001
25284782101,5sleep02528649-21sed20:40:100
25163872100,6sleep12516542-21cstates20:20:111
24797032101,4sleep22479806-21perl19:45:132
24794232101,4sleep02479442-21apt-get19:45:110
24679052101,5sleep02467910-21sensors_temp19:30:230
24474532101,4sleep22447472-21apt-get19:15:112
24403102101,4sleep12440433-21diskstats19:10:121
2891875290,2sleep00-21swapper/000:31:130
2609563291,2sleep20-21swapper/221:40:132
2558659291,4sleep22558721-21grep21:05:212
2544478291,4sleep12544780-21grep20:55:141
2533670291,4sleep22533738-21forks20:45:132
2520358291,4sleep02520404-21/usr/sbin/munin20:20:180
2506752290,1sleep20-21swapper/220:10:182
2483473291,3sleep12483523-21ntpq19:45:201
2474195291,4sleep22474367-21expr19:40:122
2473279291,4sleep22473523-21users19:35:272
2876997280,2sleep20-21swapper/200:21:042
2555759281,3sleep12557466-21grep21:05:161
2538993281,2sleep12539096-21perl20:50:131
2534174281,3sleep12535826-21kthreadcore20:45:161
2533025280,1sleep10-21swapper/120:43:001
2520686280,1sleep20-21swapper/220:20:232
2510583280,1sleep10-21swapper/120:11:501
2493981281,3sleep12494029-21ntpq19:55:221
2488538281,4sleep22488675-21netstat19:50:182
2478867280,1sleep10-21swapper/119:41:201
2468882281,4sleep02469008-21grep19:35:130
2553603271,2sleep00-21swapper/021:00:260
2549301271,3sleep22549412-21cut21:00:122
2527739270,1sleep20-21swapper/220:37:482
2527223270,1sleep20-21swapper/220:34:222
2521841270,1sleep00-21swapper/020:27:440
2516082270,3sleep22516115-21perf20:20:002
2516082270,3sleep22516115-21perf20:19:592
2515918270,1sleep10-21swapper/120:17:551
2515918270,1sleep10-21swapper/120:17:551
2511095271,2sleep02511114-21apt-get20:15:130
2511095271,2sleep02511114-21apt-get20:15:130
2510830270,1sleep00-21swapper/020:14:570
2501053271,2sleep10-21swapper/120:05:161
2494674270,1sleep00-21swapper/019:56:140
2459708271,2sleep10-21swapper/119:25:161
2458358270,2sleep00-21swapper/019:25:110
2445160270,1sleep20-21swapper/219:14:432
2543547260,1sleep20-21swapper/220:53:022
2527694260,1sleep00-21swapper/020:37:130
2521872260,1sleep10-21swapper/120:28:091
2500109260,4sleep20-21swapper/220:03:172
2478575260,2sleep00-21swapper/019:40:250
2445229261,2sleep00-21swapper/019:15:010
2548646250,1sleep00-21swapper/020:55:320
2521380250,1sleep20-21swapper/220:25:012
2501521251,1sleep20-21swapper/220:05:162
2490132250,1sleep20-21swapper/219:55:142
2468443250,1sleep20-21swapper/219:33:032
2466553250,1sleep10-21swapper/119:30:161
2527856240,1sleep10-21swapper/120:39:191
2522020240,1sleep10-21swapper/120:30:011
2500834240,1sleep00-21swapper/020:05:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional