You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 15:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Thu Sep 18, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6253222185167,12sleep30-21swapper/307:07:223
6255202149102,16sleep10-21swapper/107:09:521
625458214398,19sleep20-21swapper/207:09:052
6252022131108,15sleep00-21swapper/007:05:500
85829321140,1sleep30-21swapper/310:20:203
625661993722,12cyclictest1026857-21tr12:00:223
625661993621,12cyclictest968198-21tr11:25:213
625661993428,4cyclictest657684-21gvfsd07:39:293
625661993415,3cyclictest849434-21gltestperf10:15:143
625661993315,16cyclictest1055400-21sendmail12:20:013
625661993223,6cyclictest868082-21cut10:25:163
62566199321,20cyclictest878873-21taskset10:31:233
625661993118,3cyclictest733-21gdbus09:25:003
625661993117,2cyclictest733651-21latency_hist08:55:003
625661993116,10cyclictest2050013-21systemd-journal09:14:223
625661993025,3cyclictest0-21swapper/312:20:163
625661993017,3cyclictest798722-21pmu-power09:40:243
625661993016,6cyclictest761575-21kworker/u16:1+rpciod09:16:503
625661992925,2cyclictest0-21swapper/307:45:183
625661992916,2cyclictest0-21swapper/312:05:163
625661992915,2cyclictest0-21swapper/310:40:203
62566199290,18cyclictest919672-21ssh10:55:423
62566199281,7cyclictest903013-21ssh10:45:263
625661992817,2cyclictest636493-21needreboot07:15:173
62566199281,6cyclictest629525-21missed_timers07:10:173
625661992816,2cyclictest840894-21sh10:10:003
625661992814,3cyclictest848721-21ssh10:15:013
62566199281,13cyclictest884648-21tr10:35:173
625661992810,7cyclictest832151-21ssh10:03:283
62566199280,17cyclictest928775-21sh11:01:513
625661992723,2cyclictest0-21swapper/308:20:193
625661992722,3cyclictest0-21swapper/311:20:263
625661992716,2cyclictest647948-21cpuspeed_turbos07:30:093
62566199271,5cyclictest748551-21diskmemload12:34:313
625661992715,3cyclictest932254-21kthreadcore11:05:163
625661992712,6cyclictest748110-21grep09:05:273
625661992710,6cyclictest945711-21ssh11:12:303
625661992710,6cyclictest1002483-21vmstat11:45:283
62566199270,17cyclictest1067999-21rm12:25:213
62566199270,17cyclictest1021441-21sh11:58:473
62566199270,17cyclictest1021441-21sh11:58:473
62566199268,9cyclictest791822-21seq09:36:073
62566199262,7cyclictest1009909-21latency11:50:183
62566199261,6cyclictest823278-21ssh09:56:473
625661992612,3cyclictest952393-21unixbench-2d11:15:273
625661992511,6cyclictest1088862-21sed12:40:013
625661992511,5cyclictest988879-21sh11:39:273
625661992410,6cyclictest809558-21ssh09:49:373
62566199235,15cyclictest1044272-21sh12:11:003
625661992216,4cyclictest511ktimers/308:09:463
625661992210,8cyclictest733-21gdbus11:30:063
10033962211,9sleep21003407-21ssh11:46:432
62566199209,7cyclictest2050013-21systemd-journal07:30:003
62566199201,6cyclictest995484-21runrttasks11:42:063
62566199201,6cyclictest697509-21runrttasks08:15:343
625661992016,2cyclictest728-21rs:main3
625661992016,2cyclictest0-21swapper/310:55:003
625661992013,5cyclictest706200-21cut08:25:133
625661992012,5cyclictest780836-21cut09:30:193
62566199201,17cyclictest815663-21runrttasks09:51:553
7643502191,11sleep2764563-21hddtemp_smartct09:20:142
625661991915,2cyclictest772218-21irqstats09:25:163
62566199191,16cyclictest738840-21/usr/sbin/munin09:00:193
62566199190,6cyclictest916-21lldpd08:57:063
62566199190,16cyclictest673396-21/usr/sbin/munin07:55:163
62566199190,15cyclictest658515-21/usr/sbin/munin07:40:143
8804892181,3sleep2748551-21diskmemload10:33:392
7772202181,1sleep10-21swapper/109:27:111
62566199189,6cyclictest712679-21unixbench_multi08:30:253
62566199185,8cyclictest681897-21ntpq08:00:203
625661991811,5cyclictest669497-21cut07:50:173
625661991713,3cyclictest723625-21cpuspeed_turbos08:45:143
62566199170,15cyclictest718103-21dpkg08:39:023
10780802171,2sleep10-21swapper/112:32:001
9396662161,9sleep081-21kswapd011:10:120
8392462161,10sleep1748551-21diskmemload10:07:231
7864182160,4sleep2411rcuc/209:34:532
7282852161,11sleep2728460-21latency_hist08:50:012
6984432161,3sleep10-21swapper/108:20:141
62566199166,7cyclictest736-21gdbus08:45:013
10849992161,2sleep00-21swapper/012:35:310
10563232161,3sleep01057211-21kthreadcore12:20:200
10169602161,8sleep11018169-21memory11:55:201
10169602161,8sleep11018169-21memory11:55:191
9314402150,2sleep10-21swapper/111:05:131
9210032150,1sleep00-21swapper/010:57:340
9125322150,2sleep20-21swapper/210:52:112
62566199156,6cyclictest691056-21cut08:10:183
10795812151,4sleep0748551-21diskmemload12:34:030
9558182141,8sleep2955901-21munin-run11:20:012
9279932141,9sleep0928007-21sh11:00:460
9213742140,2sleep10-21swapper/110:58:031
7549742140,7sleep10-21swapper/109:13:231
7133992141,8sleep2713555-21diskstats08:35:142
9967162130,1sleep00-21swapper/011:43:560
9813672132,6sleep1981393-21apt-get11:35:111
8548762131,8sleep0748551-21diskmemload10:17:010
10638402130,2sleep10-21swapper/112:25:021
10514302130,8sleep21047546-21/usr/sbin/munin12:15:212
9717212121,8sleep2971723-21ssh11:28:392
8882712121,7sleep1748551-21diskmemload10:37:591
8791242120,1sleep10-21swapper/110:31:451
8486192121,7sleep1848630-21ssh10:14:541
8241232120,1sleep10-21swapper/109:58:051
7868682121,6sleep2761575-21kworker/u16:1+rpciod09:35:142
7242842121,7sleep0725966-21cut08:45:180
10539972121,6sleep0748551-21diskmemload12:18:040
10296552121,7sleep11029663-21ssh12:03:361
10120342121,5sleep20-21swapper/211:52:122
9648242112,5sleep0964848-21apt-get11:25:160
9614482110,2sleep10-21swapper/111:21:191
9467482111,6sleep2748551-21diskmemload11:13:562
9363892111,2sleep00-21swapper/011:06:000
9047382111,3sleep00-21swapper/010:47:520
8973632110,1sleep00-21swapper/010:44:060
8481482111,3sleep20-21swapper/210:14:152
8381262111,3sleep00-21swapper/010:05:350
8173452111,3sleep00-21swapper/009:54:440
7872392111,5sleep1787360-21irqstats09:35:171
7840542111,3sleep10-21swapper/109:30:521
7601592111,5sleep0760220-21ntpq09:15:240
10704582111,3sleep20-21swapper/212:27:562
10558072111,3sleep20-21swapper/212:20:132
10549362111,3sleep10-21swapper/112:19:251
10456262111,3sleep00-21swapper/012:12:540
10443892111,3sleep10-21swapper/112:11:101
10309752110,1sleep10-21swapper/112:05:061
10306662111,7sleep0748551-21diskmemload12:05:000
10121022111,6sleep0748551-21diskmemload11:52:180
10049582110,6sleep0748551-21diskmemload11:48:520
9944592101,3sleep10-21swapper/111:40:401
9897192101,2sleep20-21swapper/211:40:122
9886352101,2sleep00-21swapper/011:39:070
9798162101,4sleep0748551-21diskmemload11:33:220
9695732101,2sleep10-21swapper/111:25:411
9562422101,4sleep2956266-21apt-get11:20:122
9548322101,5sleep10-21swapper/111:18:401
9518462101,4sleep0951911-21ntpq11:15:220
9456462101,2sleep10-21swapper/111:12:251
9373692101,6sleep2748551-21diskmemload11:07:222
9237062101,6sleep1925136-21kthreadcore11:00:151
9149492101,4sleep2914976-21apt-get10:55:152
9140422101,4sleep0748551-21diskmemload10:54:200
8961122101,3sleep10-21swapper/110:42:241
8635022100,1sleep10-21swapper/110:23:031
8462482101,4sleep0748551-21diskmemload10:11:120
8309172100,1sleep10-21swapper/110:01:311
8232212100,1sleep00-21swapper/009:56:400
8157162101,3sleep20-21swapper/209:52:022
8147622101,3sleep10-21swapper/109:50:281
8091262101,2sleep20-21swapper/209:48:562
8072602101,3sleep00-21swapper/009:46:010
8014342101,4sleep1748551-21diskmemload09:43:591
7991142101,2sleep00-21swapper/009:40:290
7945862101,3sleep20-21swapper/209:40:072
7856492101,2sleep00-21swapper/009:33:360
7763922101,2sleep20-21swapper/209:25:402
7562342101,3sleep20-21swapper/209:15:102
7425332101,4sleep1742584-21ntpq09:00:171
7121362101,4sleep1712183-21ntpq08:30:191
6931502101,4sleep0693169-21apt-get08:15:120
6831882101,4sleep1683344-21diskstats08:05:141
6719412101,5sleep2671992-21ntpq07:50:212
6533542101,5sleep0654785-21kthreadcore07:35:150
10736892101,5sleep21075406-21cut12:30:152
10640142101,4sleep0748551-21diskmemload12:25:090
10467112101,3sleep20-21swapper/212:14:242
10348772102,4sleep21034925-21ntpq12:05:232
10267512101,6sleep21026815-21sh12:00:212
10212722101,3sleep20-21swapper/211:58:332
10212722101,3sleep20-21swapper/211:58:332
10139582101,2sleep10-21swapper/111:54:511
10032832100,3sleep10-21swapper/111:46:341
981278291,2sleep20-21swapper/211:35:042
979110290,3sleep20-21swapper/211:32:232
978190291,3sleep10-21swapper/111:31:051
961205291,3sleep0748551-21diskmemload11:20:580
930036292,3sleep2748551-21diskmemload11:03:372
913852291,3sleep10-21swapper/110:54:061
904308290,3sleep10-21swapper/110:47:171
890580291,3sleep2748551-21diskmemload10:40:162
889559291,2sleep00-21swapper/010:39:460
887312291,4sleep2887324-21ssh10:36:392
878326291,3sleep0748551-21diskmemload10:30:390
865304291,3sleep1748551-21diskmemload10:25:121
865223291,2sleep20-21swapper/210:25:042
863907291,3sleep2748551-21diskmemload10:23:372
861947291,2sleep00-21swapper/010:20:540
854439290,2sleep10-21swapper/110:16:261
853381292,3sleep20-21swapper/210:15:262
838409290,2sleep20-21swapper/210:06:042
830884290,2sleep00-21swapper/010:01:260
825716291,3sleep2748551-21diskmemload10:00:112
808218291,3sleep1748551-21diskmemload09:47:311
777289291,5sleep0748551-21diskmemload09:27:170
768062291,4sleep1768120-21ssh09:20:221
760927291,3sleep1748551-21diskmemload09:15:371
754719291,4sleep2754724-21ssh09:12:522
753737290,4sleep0748551-21diskmemload09:10:550
710781291,4sleep2711970-21memory08:30:172
698824291,4sleep0698964-21cut08:20:160
688548291,4sleep1688619-21irqstats08:10:161
687895291,3sleep2687944-21munin-run08:10:002
658540291,4sleep2659919-21kthreadcore07:40:152
634185293,3sleep00-21swapper/007:15:150
1085126291,4sleep11085342-21users12:35:321
1080736291,4sleep21080765-21apt-get12:35:172
1060456291,3sleep10-21swapper/112:20:371
1031062291,3sleep0748551-21diskmemload12:05:130
1022266291,2sleep00-21swapper/011:59:560
1022266291,2sleep00-21swapper/011:59:550
898088281,3sleep2898296-21ssh10:45:012
870032280,5sleep00-21swapper/010:25:390
822719280,2sleep20-21swapper/209:55:542
792741281,2sleep00-21swapper/009:37:320
744295281,3sleep0745338-21grep09:05:180
733447280,2sleep20-21swapper/208:55:022
728937281,4sleep1729008-21irqstats08:50:151
723077280,2sleep10-21swapper/108:42:331
697242281,3sleep2697487-21users08:15:252
682823280,6sleep10-21swapper/108:04:031
682574280,1sleep00-21swapper/008:00:530
671894281,3sleep1671940-21ld07:50:201
661834281,3sleep1661882-21ntpq07:40:181
654811281,3sleep1656518-21sed07:35:161
648260281,3sleep2648598-21cut07:30:132
647516280,1sleep10-21swapper/107:29:151
647492280,1sleep20-21swapper/207:28:572
641869280,1sleep0191rcu_preempt07:20:250
637510280,2sleep20-21swapper/207:19:392
637212280,1sleep10-21swapper/107:15:521
629590281,3sleep0629619-21ntpq07:10:190
764119270,2sleep00-21swapper/009:20:090
748543271,3sleep2748546-21cron09:10:012
742586271,3sleep2742635-21ntpq09:00:182
738206270,1sleep10-21swapper/108:55:291
728549271,2sleep20-21swapper/208:50:122
717804270,1sleep10-21swapper/108:35:331
702458271,3sleep2702545-21timerandwakeup08:20:272
688620271,3sleep2689881-21kthreadcore08:10:172
667221271,3sleep2667247-21grep07:45:232
625914270,4sleep20-21swapper/207:10:122
745339261,2sleep10-21swapper/109:05:191
733080260,1sleep00-21swapper/008:52:470
721736261,2sleep20-21swapper/208:40:182
718492260,2sleep00-21swapper/008:40:120
708401260,2sleep00-21swapper/008:30:100
704049261,2sleep00-21swapper/008:25:140
682723260,1sleep20-21swapper/208:02:482
677869260,1sleep10-21swapper/107:59:331
673198261,2sleep00-21swapper/007:55:160
662585260,1sleep00-21swapper/007:41:170
647309260,1sleep00-21swapper/007:26:380
723698251,1sleep10-21swapper/108:45:151
705718250,1sleep10-21swapper/108:25:151
683627250,2sleep00-21swapper/008:05:180
668391250,1sleep00-21swapper/007:50:150
667593250,1sleep10-21swapper/107:45:341
653062250,1sleep20-21swapper/207:35:132
650307250,1sleep00-21swapper/007:30:150
648030251,2sleep1648246-21idleruntime07:30:121
641539250,1sleep20-21swapper/207:20:192
638011250,1sleep10-21swapper/107:20:141
630428250,1sleep10-21swapper/107:12:411
739023240,1sleep00-21swapper/009:00:130
733788240,1sleep00-21swapper/008:55:120
713362240,1sleep00-21swapper/008:35:040
703591241,1sleep20-21swapper/208:25:122
693172240,1sleep10-21swapper/108:15:131
691682240,1sleep00-21swapper/008:10:190
673360240,1sleep20-21swapper/207:55:172
663320240,1sleep00-21swapper/007:45:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional