You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-05 - 20:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Fri Dec 05, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2682420212755,66sleep30-21swapper/307:05:003
268295828762,20sleep00-21swapper/007:08:420
268298428554,9sleep10-21swapper/107:09:031
268290227759,10sleep20-21swapper/207:07:592
2683285994136,2cyclictest0-21swapper/310:20:003
2683285992825,1cyclictest0-21swapper/307:37:233
2683285992824,2cyclictest0-21swapper/311:36:543
268328599272,16cyclictest2873202-21taskset10:35:273
27993892261,20sleep02738803-21diskmemload09:49:190
2683285992623,1cyclictest0-21swapper/309:25:003
30560162250,8sleep02738803-21diskmemload12:35:280
268328599241,17cyclictest3018973-21chrt12:15:133
2683285992320,2cyclictest0-21swapper/309:55:183
2683285992319,2cyclictest0-21swapper/310:20:173
2683285992317,3cyclictest0-21swapper/312:35:133
268328599231,18cyclictest2692584-21taskset07:25:033
27974792221,10sleep12797489-21ssh09:45:431
268328599222,14cyclictest3027181-21taskset12:20:023
268328599222,14cyclictest2904813-21taskset10:57:593
268328599222,14cyclictest2883979-21taskset10:45:123
268328599222,13cyclictest2797539-21taskset09:45:473
2683285992219,1cyclictest0-21swapper/307:35:013
2683285992218,2cyclictest0-21swapper/312:35:013
2683285992218,2cyclictest0-21swapper/312:05:163
2683285992218,2cyclictest0-21swapper/309:40:233
268328599220,18cyclictest2920405-21taskset11:09:333
268328599214,14cyclictest28808452chrt10:40:243
268328599212,13cyclictest2827737-21taskset10:05:313
268328599212,12cyclictest2971169-21taskset11:41:093
2683285992118,1cyclictest0-21swapper/309:50:123
2683285992116,3cyclictest0-21swapper/312:00:153
268328599211,16cyclictest2835869-21taskset10:11:503
29046002201,2sleep00-21swapper/010:57:380
268328599202,13cyclictest2956181-21taskset11:32:153
268328599202,12cyclictest3042265-21taskset12:29:003
268328599202,12cyclictest2974035-21taskset11:45:143
2683285992016,2cyclictest0-21swapper/311:00:153
2683285992016,2cyclictest0-21swapper/310:35:013
2683285992015,2cyclictest0-21swapper/309:10:143
268328599201,16cyclictest2692196-21chrt07:23:193
268328599201,16cyclictest0-21swapper/309:35:553
2683285992010,7cyclictest2857616-21taskset10:25:253
28583912193,5sleep1171rcu_preempt10:26:121
2683285991915,2cyclictest0-21swapper/312:10:203
268328599191,14cyclictest2746749-21chrt09:15:053
268328599191,14cyclictest0-21swapper/311:19:013
268328599191,14cyclictest0-21swapper/311:10:103
268328599191,13cyclictest2761705-21chrt09:25:033
268328599191,13cyclictest2761705-21chrt09:25:033
30333702181,11sleep13033383-21ssh12:22:301
28697232181,13sleep12871383-21grep10:35:181
28288862181,2sleep10-21swapper/110:07:441
27134842181,8sleep22715155-21grep08:10:172
268328599182,12cyclictest2993924-21taskset11:56:013
2683285991815,1cyclictest0-21swapper/311:53:543
2683285991814,2cyclictest0-21swapper/310:50:163
2683285991814,2cyclictest0-21swapper/309:30:133
268328599181,14cyclictest0-21swapper/311:25:003
268328599181,14cyclictest0-21swapper/311:23:253
268328599181,13cyclictest2719383-21chrt08:25:103
268328599181,10cyclictest2712666-21taskset08:09:593
30092982171,11sleep03009307-21ssh12:07:040
29470532171,2sleep00-21swapper/011:25:200
28437242171,3sleep10-21swapper/110:17:101
28067682170,9sleep12806778-21ssh09:53:561
27753712171,2sleep20-21swapper/209:33:212
27693612172,9sleep12769526-21expr09:30:151
27609752171,10sleep22738803-21diskmemload09:24:112
2683285991714,1cyclictest0-21swapper/308:11:543
2683285991714,1cyclictest0-21swapper/307:10:153
268328599171,13cyclictest0-21swapper/310:00:473
268328599171,12cyclictest2691697-21turbostat07:19:593
268328599171,10cyclictest2705195-21chrt07:48:593
2683285991710,4cyclictest2726339-21chrt08:40:143
30416612161,2sleep00-21swapper/012:28:020
29359112161,11sleep12935992-21irqstats11:20:161
28361362161,4sleep22738803-21diskmemload10:12:182
28062802161,2sleep20-21swapper/209:53:062
27834562161,3sleep00-21swapper/009:38:570
27581532160,10sleep12758299-21ntpq09:20:171
27537442161,3sleep02738803-21diskmemload09:19:470
27299802161,10sleep12730038-21collect208:40:211
268328599161,9cyclictest2738406-21taskset09:05:063
268328599161,9cyclictest2711584-21taskset08:01:353
2683285991613,1cyclictest0-21swapper/307:57:533
268328599161,11cyclictest2699583-21chrt07:40:113
268328599161,10cyclictest2725364-21taskset08:36:333
30500342151,3sleep00-21swapper/012:33:390
30351352151,2sleep10-21swapper/112:25:011
30265252151,9sleep03026537-21ssh12:19:510
30189422153,6sleep23018962-21apt-get12:15:122
29866402150,3sleep22738803-21diskmemload11:50:562
29709442151,2sleep10-21swapper/111:40:481
29632862151,10sleep22963296-21ssh11:36:112
28882192159,3sleep12888300-21pmu-power10:45:241
28769612151,9sleep12878418-21kthreadcore10:40:141
28582542151,9sleep22738803-21diskmemload10:25:572
28447772150,3sleep22738803-21diskmemload10:19:072
27234432151,9sleep02723516-21grep08:25:210
268328599151,9cyclictest2737212-21taskset08:56:333
268328599151,9cyclictest2731950-21taskset08:51:413
268328599151,9cyclictest2724980-21taskset08:34:493
268328599151,9cyclictest2718118-21taskset08:16:013
2683285991513,1cyclictest0-21swapper/308:45:533
2683285991512,1cyclictest0-21swapper/307:51:543
268328599150,9cyclictest2719018-21taskset08:23:593
30405112140,1sleep20-21swapper/212:26:062
30272792143,5sleep03027303-21apt-get12:20:120
30246722141,9sleep13024683-21sh12:16:511
30161862141,8sleep12738803-21diskmemload12:11:031
30095162141,8sleep1802-21snmpd12:07:291
30027492140,1sleep10-21swapper/112:04:101
29945322141,7sleep22738803-21diskmemload11:57:162
29869342141,7sleep02986945-21sshd11:51:300
29715102141,6sleep00-21swapper/011:41:480
29630812140,2sleep00-21swapper/011:35:500
29491482141,9sleep12949152-21ssh11:28:051
29283032141,8sleep12928397-21tune2fs11:15:151
28620572141,8sleep22863760-21tr10:30:172
28288782141,9sleep02828885-21sh10:07:430
28225002140,1sleep20-21swapper/210:04:562
28206952141,9sleep02820700-21ssh10:01:290
28149382141,3sleep12814947-21ssh09:59:581
27912172141,9sleep12791222-21rm09:43:541
27885822141,6sleep02788639-21ntpq09:40:200
26928712141,8sleep02692939-21fschecks_count07:25:140
30575042131,2sleep10-21swapper/112:38:001
30517862131,6sleep23052260-21grep12:35:122
30018822131,3sleep23001896-21ssh12:01:542
29940042131,5sleep02738803-21diskmemload11:56:100
29404012131,6sleep00-21swapper/011:20:460
29191832131,6sleep02738803-21diskmemload11:07:160
28972052131,6sleep22738803-21diskmemload10:52:072
28818272130,9sleep02738803-21diskmemload10:41:410
28727042131,7sleep02872750-21ntpq10:35:210
28679342131,8sleep02868060-21cat10:35:000
28145832130,7sleep02738803-21diskmemload09:59:130
27809232131,4sleep12776685-21/usr/sbin/munin09:35:211
27528992131,6sleep12738803-21diskmemload09:18:161
268328599131,9cyclictest2737787-21chrt09:00:263
30270202121,5sleep23026631-21apt-get12:20:022
29935682121,4sleep12738803-21diskmemload11:55:271
29806832121,4sleep12738803-21diskmemload11:49:091
29621882121,4sleep12738803-21diskmemload11:35:181
29485762121,5sleep20-21swapper/211:27:012
29409522121,5sleep22738803-21diskmemload11:22:132
29328282120,3sleep22932838-21ssh11:15:312
29271542120,3sleep12927161-21ssh11:14:021
29269252121,7sleep22926930-21ssh11:13:272
29149942121,6sleep22916593-21kthreadcore11:05:172
29113562121,4sleep02738803-21diskmemload11:00:340
29064922122,5sleep22906510-21apt-get11:00:122
28955852121,5sleep02895638-21ntpq10:50:190
28820252121,6sleep22738803-21diskmemload10:42:032
28592792121,4sleep00-21swapper/010:27:470
28355532121,8sleep00-21swapper/010:11:110
28131692120,6sleep22738803-21diskmemload09:56:382
28050202121,4sleep02738803-21diskmemload09:50:420
27232192121,5sleep12723332-21ntpq08:25:181
27199152121,7sleep22719964-21kthreadcore08:25:142
30507992111,3sleep20-21swapper/212:34:532
30505532111,3sleep12738803-21diskmemload12:34:281
30179302110,5sleep22738803-21diskmemload12:14:012
30117452111,5sleep03013143-21cut12:10:180
30099302111,4sleep22738803-21diskmemload12:08:142
29778632111,5sleep20-21swapper/211:45:212
29711112111,4sleep22738803-21diskmemload11:41:052
29557482111,4sleep12738803-21diskmemload11:31:261
29503492111,4sleep22738803-21diskmemload11:30:032
29331432111,3sleep00-21swapper/011:16:100
29042622111,4sleep22738803-21diskmemload10:57:002
28956402110,1sleep10-21swapper/110:50:201
28908382111,3sleep02738803-21diskmemload10:49:240
28891172111,4sleep22738803-21diskmemload10:46:012
28660942111,3sleep12738803-21diskmemload10:31:281
28526422111,3sleep00-21swapper/010:24:300
28433562110,7sleep00-21swapper/010:16:290
28373572111,5sleep12738803-21diskmemload10:14:241
28228192110,1sleep20-21swapper/210:05:042
28213092111,3sleep10-21swapper/110:02:341
27973852111,3sleep20-21swapper/209:45:322
27893972111,4sleep22738803-21diskmemload09:40:432
27816182111,4sleep22738803-21diskmemload09:35:382
27706282111,6sleep02772344-21kthreadcore09:30:180
27677132111,3sleep00-21swapper/009:28:020
27677132111,3sleep00-21swapper/009:28:010
27669522111,6sleep12766963-21ssh09:26:071
27669522111,6sleep12766963-21ssh09:26:061
27669282110,6sleep22738803-21diskmemload09:26:042
27669282110,6sleep22738803-21diskmemload09:26:032
27590852111,4sleep02738803-21diskmemload09:20:370
27510582111,4sleep22738803-21diskmemload09:15:202
27442392110,4sleep22738803-21diskmemload09:10:562
27100912112,4sleep22710180-21ntpq07:55:172
26990052111,4sleep02699021-21munin-run07:40:010
26987542111,6sleep12698750-40gst-plugin-scan07:37:251
30013202100,4sleep02738803-21diskmemload12:00:520
29882992100,3sleep10-21swapper/111:54:431
29788182100,3sleep02738803-21diskmemload11:46:000
29555962101,3sleep00-21swapper/011:31:090
29256642102,4sleep02925675-21sh11:10:360
29195142100,1sleep10-21swapper/111:08:011
29122902101,3sleep10-21swapper/111:02:441
29038192101,2sleep10-21swapper/110:55:591
28740332101,3sleep22738803-21diskmemload10:36:512
28514482101,3sleep10-21swapper/110:22:221
28506172101,3sleep22738803-21diskmemload10:20:442
27451012101,3sleep10-21swapper/109:12:241
27127572100,4sleep12712776-21apt-get08:10:121
26887132101,4sleep12689093-21sed07:15:011
2740686291,4sleep02742389-21cut09:10:140
2724995291,4sleep12724999-21cron08:35:001
2711191290,1sleep10-21swapper/108:00:021
2698804290,4sleep22698741-31find07:37:372
2697603291,3sleep22697653-21munin-run07:30:002
2732471281,2sleep22732490-21apt-get08:55:082
2732207280,1sleep10-21swapper/108:54:561
2711728281,3sleep22711729-21sshd08:03:272
2699561281,2sleep20-21swapper/207:40:112
2692941281,4sleep12693139-21/usr/sbin/munin07:25:151
2691668281,3sleep00-21swapper/007:20:010
2738769270,1sleep20-21swapper/209:09:432
2738125270,1sleep00-21swapper/009:04:410
2718339270,1sleep10-21swapper/108:18:491
2718232270,1sleep00-21swapper/008:17:280
2712199270,1sleep20-21swapper/208:06:162
2711670270,1sleep00-21swapper/008:02:400
2705859270,1sleep00-21swapper/007:54:140
2704427270,1sleep10-21swapper/107:42:261
2698248271,2sleep12698296-21systemd-run07:35:011
2691423270,1sleep20-21swapper/207:16:572
2684212271,3sleep00-21swapper/007:10:140
2738065260,4sleep10-21swapper/109:03:581
2737928260,1sleep20-21swapper/209:02:152
2734532261,2sleep00-21swapper/008:55:150
2732173260,4sleep20-21swapper/208:54:292
2731221260,1sleep10-21swapper/108:45:341
2726486261,2sleep00-21swapper/008:40:150
2725367260,4sleep00-21swapper/008:36:360
2718142260,1sleep20-21swapper/208:16:212
2711005260,1sleep10-21swapper/107:57:411
2707698261,2sleep00-21swapper/007:55:160
2705046260,1sleep00-21swapper/007:47:060
2703941260,1sleep00-21swapper/007:40:240
2697992260,4sleep20-21swapper/207:31:492
2692014260,4sleep00-21swapper/007:21:000
2691338260,1sleep10-21swapper/107:15:531
2683623260,1sleep20-21swapper/207:10:052
2738414250,1sleep10-21swapper/109:05:131
2737458250,3sleep10-21swapper/108:59:401
2726203251,1sleep20-21swapper/208:40:142
2705833250,3sleep20-21swapper/207:53:572
2704911251,2sleep20-21swapper/207:45:232
2738741240,1sleep00-21swapper/009:09:260
2731824240,1sleep00-21swapper/008:50:050
2731444240,1sleep20-21swapper/208:48:272
2730932240,1sleep00-21swapper/008:45:010
2725294240,1sleep10-21swapper/108:35:411
2725005240,1sleep20-21swapper/208:35:002
2724918240,2sleep20-21swapper/208:34:052
2724600240,2sleep00-21swapper/008:30:010
2718961240,1sleep00-21swapper/008:23:160
2718815240,2sleep20-21swapper/208:21:272
2718508240,1sleep10-21swapper/108:20:011
2713055240,1sleep00-21swapper/008:10:140
2712489240,1sleep10-21swapper/108:09:551
2712241240,1sleep00-21swapper/008:06:460
2705556240,1sleep10-21swapper/107:50:281
2704890240,1sleep10-21swapper/107:45:071
2698054240,1sleep00-21swapper/007:32:340
2691943240,1sleep20-21swapper/207:20:072
2691830240,1sleep10-21swapper/107:20:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional