You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-21 - 00:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue May 20, 2025 12:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
84932240164,25sleep30-21swapper/307:08:313
85322235162,25sleep20-21swapper/207:08:582
84232235162,60sleep00-21swapper/007:07:360
85082227154,59sleep10-21swapper/107:08:411
219721570,5sleep0869999cyclictest09:20:140
162221330,2sleep20-21swapper/211:50:012
8711997369,3cyclictest0-21swapper/211:57:062
8711996462,1cyclictest0-21swapper/210:17:322
304772530,5sleep121-21rcuc/107:55:251
276682400,3sleep121-21rcuc/109:05:021
869999393,27cyclictest22468-21kworker/0:107:53:370
138672390,3sleep313870-21processes12:15:233
869999384,5cyclictest22697-40dnf11:23:320
103382380,1sleep30-21swapper/312:10:013
8699993710,12cyclictest20985-21idleruntime-cro08:50:010
8699993612,21cyclictest30592-21grep09:10:160
8699993611,3cyclictest19283-21/usr/sbin/munin12:30:120
869999358,4cyclictest683-21/usr/sbin/munin11:45:280
8699993412,18cyclictest6543-21cut09:30:160
8699993410,4cyclictest14940-21/usr/sbin/munin07:20:200
8699993410,4cyclictest14940-21/usr/sbin/munin07:20:200
284322340,7sleep232-21ksoftirqd/207:50:362
8711993326,2cyclictest11515-21kworker/2:010:44:422
8711993323,2cyclictest11515-21kworker/2:012:11:232
8699993321,7cyclictest5220-21kworker/0:208:10:510
869999331,7cyclictest12873-21irqstats11:00:180
8699993310,2cyclictest23057-21/usr/sbin/munin11:25:140
93442320,4sleep230-21rcuc/208:20:282
869999329,4cyclictest848-21sssd_nss08:35:340
869999329,3cyclictest7765-21crond08:20:010
869999328,1cyclictest0-21swapper/008:20:270
869999327,7cyclictest86550irq/126-enp1s0-12:21:350
869999327,4cyclictest13024-21/usr/sbin/munin07:15:230
869999323,3cyclictest20240-21/usr/sbin/munin10:05:160
869999321,6cyclictest17876-21ntp_states08:40:230
8699993211,3cyclictest22889-21grep07:40:130
869999320,9cyclictest0-21swapper/009:40:270
871199316,8cyclictest86750irq/128-enp1s0-08:59:352
871199313,2cyclictest86750irq/128-enp1s0-08:36:492
8711993114,2cyclictest11515-21kworker/2:012:36:392
871199311,13cyclictest0-21swapper/209:59:332
871199311,12cyclictest0-21swapper/210:20:212
869999319,2cyclictest836-21runrttasks08:07:500
869999317,3cyclictest18036-21/usr/sbin/munin10:00:250
869999317,2cyclictest4145-21latency10:40:180
869999316,6cyclictest0-21swapper/009:55:590
869999316,12cyclictest28817-21awk10:25:010
869999311,6cyclictest30101-21ntp_states10:25:240
869999311,15cyclictest27231-21munin-run11:35:010
8699993111,3cyclictest836-21runrttasks07:36:230
8699993111,3cyclictest22029-21cut12:35:150
869999311,11cyclictest10947-21cat07:15:010
869999311,11cyclictest10947-21cat07:15:010
8711993019,3cyclictest11515-21kworker/2:009:38:252
8711993014,3cyclictest11515-21kworker/2:009:30:272
871199301,11cyclictest0-21swapper/210:32:372
871199301,11cyclictest0-21swapper/210:14:052
869999308,3cyclictest3975-21wc11:55:140
869999308,3cyclictest18424-21crond11:15:010
869999308,3cyclictest10994-21ls10:55:240
869999308,2cyclictest19050-21latency_hist12:30:010
869999305,3cyclictest86750irq/128-enp1s0-10:30:500
869999302,5cyclictest0-21swapper/009:15:140
869999302,3cyclictest14094-21ntp_states09:45:240
869999302,10cyclictest16242-21munin-run11:10:010
869999301,9cyclictest10760-21cat09:40:010
869999301,5cyclictest2554-21latency11:50:190
8699993012,15cyclictest26552-21seq10:19:310
869999301,13cyclictest15617-21munin-run09:55:010
869999300,9cyclictest0-21swapper/008:30:210
8717992916,4cyclictest6184-21systemd-journal09:29:593
8717992916,4cyclictest6184-21systemd-journal09:29:593
8711992918,2cyclictest11515-21kworker/2:011:21:082
8711992917,3cyclictest11515-21kworker/2:007:49:372
8711992916,3cyclictest11515-21kworker/2:011:50:252
869999299,3cyclictest23810-21timerandwakeup10:10:270
869999298,4cyclictest18775-21tr11:15:130
869999297,2cyclictest2148-21open_files10:35:250
869999297,2cyclictest2148-21open_files10:35:250
869999291,15cyclictest0-21swapper/007:25:250
8699992911,3cyclictest11092-21grep12:10:180
8699992910,4cyclictest2793-21kworker/0:110:49:050
8699992910,3cyclictest23641-21cat08:55:150
869999290,7cyclictest20848-21kworker/u8:011:38:440
871199285,3cyclictest30-21rcuc/208:44:232
871199282,12cyclictest32-21ksoftirqd/207:10:522
871199282,12cyclictest32-21ksoftirqd/207:10:512
8711992820,2cyclictest11515-21kworker/2:010:29:392
8711992818,2cyclictest11515-21kworker/2:012:24:172
8711992817,3cyclictest11515-21kworker/2:009:11:272
8711992815,2cyclictest11515-21kworker/2:008:15:202
871199281,19cyclictest11515-21kworker/2:010:38:582
871199281,19cyclictest11515-21kworker/2:010:38:572
871199281,13cyclictest0-21swapper/212:16:052
869999289,3cyclictest30824-21cat07:55:280
869999289,3cyclictest27436-21seq09:03:280
869999288,4cyclictest848-21sssd_nss09:05:270
869999288,3cyclictest356-21mailstats08:00:240
869999287,3cyclictest9108-21timerwakeupswit10:50:270
869999287,3cyclictest5259-21ntp_kernel_err09:25:210
869999287,3cyclictest5259-21ntp_kernel_err09:25:210
869999282,3cyclictest0-21swapper/012:02:050
869999281,12cyclictest8287-21/usr/sbin/munin12:05:250
8711992718,2cyclictest11515-21kworker/2:008:45:212
8711992716,3cyclictest11515-21kworker/2:011:11:212
8711992716,2cyclictest11515-21kworker/2:009:46:092
8711992715,3cyclictest11515-21kworker/2:009:20:412
8711992710,2cyclictest11515-21kworker/2:011:18:412
869999277,3cyclictest29741-21cpuspeed_turbos11:40:140
869999277,3cyclictest29741-21cpuspeed_turbos11:40:140
869999277,3cyclictest228112chrt08:50:360
869999276,4cyclictest11194-21needreboot08:25:210
869999276,4cyclictest11194-21needreboot08:25:210
869999274,3cyclictest86750irq/128-enp1s0-07:30:180
280932270,4sleep10-21swapper/107:50:191
871199269,2cyclictest86750irq/128-enp1s0-10:46:202
871199268,2cyclictest86650irq/127-enp1s0-08:11:302
871199261,8cyclictest11515-21kworker/2:007:42:172
8711992618,3cyclictest11515-21kworker/2:012:25:352
8711992614,6cyclictest11515-21kworker/2:011:36:352
8711992612,3cyclictest11515-21kworker/2:009:51:192
8711992612,2cyclictest11515-21kworker/2:009:25:372
8711992612,2cyclictest11515-21kworker/2:009:25:372
869999267,3cyclictest27068-21latency_hist07:50:020
869999261,6cyclictest0-21swapper/012:15:130
871199259,3cyclictest0-21swapper/208:04:462
871199254,10cyclictest11515-21kworker/2:011:41:042
871199254,10cyclictest11515-21kworker/2:011:41:042
8711992513,2cyclictest11515-21kworker/2:009:42:312
8711992511,3cyclictest11515-21kworker/2:011:31:562
8711992511,3cyclictest11515-21kworker/2:010:55:022
8711992511,2cyclictest11515-21kworker/2:007:34:272
871199251,10cyclictest11515-21kworker/2:007:37:592
870799251,18cyclictest0-21swapper/111:55:141
61102250,3sleep20-21swapper/212:00:132
871199247,3cyclictest11515-21kworker/2:010:00:152
871199242,10cyclictest11515-21kworker/2:008:10:002
8711992412,3cyclictest11515-21kworker/2:011:05:402
8711992412,3cyclictest11515-21kworker/2:009:07:452
8711992412,2cyclictest11515-21kworker/2:009:15:072
8707992410,11cyclictest19157-21latency_hist12:30:011
255412240,4sleep123-21ksoftirqd/111:30:131
871199238,3cyclictest11515-21kworker/2:008:25:042
871199238,3cyclictest11515-21kworker/2:008:25:032
871199233,10cyclictest32-21ksoftirqd/207:57:382
8707992311,5cyclictest16378-21systemd-journal12:20:271
871799229,3cyclictest21659-21interrupts11:20:193
871199229,4cyclictest11515-21kworker/2:007:21:532
871199229,4cyclictest11515-21kworker/2:007:21:532
871199229,2cyclictest11515-21kworker/2:011:26:522
8711992210,3cyclictest11515-21kworker/2:008:52:392
870799225,11cyclictest86750irq/128-enp1s0-09:15:161
870799221,3cyclictest5298-21ntp_states09:25:221
870799221,3cyclictest5298-21ntp_states09:25:221
870799221,19cyclictest0-21swapper/110:10:421
870799221,11cyclictest20380-21tr10:05:111
871799219,3cyclictest26069-21irqstats11:30:183
871799219,3cyclictest23695-21memory07:40:193
871799212,15cyclictest10866-21sort12:10:173
871799211,5cyclictest0-21swapper/311:40:573
871799211,5cyclictest0-21swapper/311:40:573
871799211,11cyclictest5463-21/usr/sbin/munin10:45:193
871199217,2cyclictest11515-21kworker/2:010:59:142
871199215,3cyclictest86850irq/129-enp1s0-12:08:252
871199215,3cyclictest11515-21kworker/2:011:00:082
871199215,3cyclictest11515-21kworker/2:007:15:152
871199212,4cyclictest32-21ksoftirqd/207:25:112
8711992110,2cyclictest11515-21kworker/2:012:33:512
870799219,3cyclictest15288-21hddtemp_smartct08:35:181
870799214,9cyclictest88150irq/134-enp2s0-08:02:071
870799212,10cyclictest9882-21users12:05:291
8707992117,3cyclictest0-21swapper/111:42:091
8707992117,3cyclictest0-21swapper/111:42:091
8707992117,3cyclictest0-21swapper/108:55:451
8707992117,3cyclictest0-21swapper/107:16:031
870799211,18cyclictest0-21swapper/112:10:151
870799211,18cyclictest0-21swapper/109:14:051
870799211,18cyclictest0-21swapper/107:32:151
870799211,11cyclictest17382-21memory11:10:211
870799211,11cyclictest14034-21latency_hist11:05:021
8707992110,3cyclictest15333-21ntp_states11:05:231
245472211,2sleep10-21swapper/107:42:371
871799208,3cyclictest21200-21/usr/sbin/munin08:50:173
871799206,3cyclictest0-21swapper/308:39:533
871799202,9cyclictest18036-21/usr/sbin/munin10:00:223
871799202,2cyclictest86850irq/129-enp1s0-09:43:593
871799201,7cyclictest0-21swapper/312:01:363
871799201,7cyclictest0-21swapper/311:59:053
8717992014,3cyclictest27472-21head10:20:173
8717992014,3cyclictest19047-21idleruntime-cro12:30:013
871799201,13cyclictest0-21swapper/311:51:153
871799201,12cyclictest0-21swapper/309:55:223
871799201,11cyclictest16192-21/usr/sbin/munin07:25:253
871799201,11cyclictest0-21swapper/307:50:183
871799201,10cyclictest8574-21memory10:50:213
871799201,10cyclictest0-21swapper/312:24:143
871199208,3cyclictest11515-21kworker/2:010:06:392
871199202,8cyclictest32-21ksoftirqd/208:33:292
870799209,3cyclictest3381-21latency_hist08:10:011
870799205,9cyclictest0-21swapper/111:18:301
8707992016,3cyclictest0-21swapper/110:47:361
8707992016,2cyclictest8309-21head10:50:181
8707992014,3cyclictest12374-21grep09:40:291
8707992014,3cyclictest10887-21hddtemp_smartct08:25:181
8707992014,3cyclictest10887-21hddtemp_smartct08:25:181
870799201,2cyclictest0-21swapper/110:35:241
870799201,2cyclictest0-21swapper/110:35:241
870799201,18cyclictest0-21swapper/111:53:441
870799201,18cyclictest0-21swapper/109:09:451
870799201,18cyclictest0-21swapper/108:23:101
870799201,17cyclictest0-21swapper/112:35:571
870799201,17cyclictest0-21swapper/112:15:501
870799201,17cyclictest0-21swapper/109:50:021
870799201,17cyclictest0-21swapper/108:45:401
870799201,17cyclictest0-21swapper/108:15:011
870799201,17cyclictest0-21swapper/107:20:061
870799201,17cyclictest0-21swapper/107:20:061
870799201,11cyclictest6778-21diskstats09:30:181
870799201,11cyclictest0-21swapper/111:20:161
870799201,10cyclictest6421-21df_inode12:00:151
870799201,10cyclictest4200-21cat09:25:011
870799201,10cyclictest26600-21latency_hist10:20:011
870799201,10cyclictest26095-21pmu-power07:45:241
870799201,10cyclictest18156-21latency_hist07:30:011
870799200,3cyclictest0-21swapper/109:35:271
870799200,18cyclictest0-21swapper/109:45:041
870799200,18cyclictest0-21swapper/108:30:171
226842200,4sleep339-21rcuc/310:10:153
201202201,2sleep30-21swapper/308:45:583
871799196,6cyclictest0-21swapper/311:36:373
871799191,6cyclictest0-21swapper/311:25:173
871799191,6cyclictest0-21swapper/311:10:243
871799191,6cyclictest0-21swapper/308:21:513
871799191,4cyclictest11665-21sed08:25:283
871799191,4cyclictest11665-21sed08:25:283
8717991914,3cyclictest0-21swapper/309:24:463
8717991912,4cyclictest3284-21cpu10:40:123
871799191,11cyclictest0-21swapper/310:58:163
871799191,11cyclictest0-21swapper/307:55:273
871799191,10cyclictest29035-21/usr/sbin/munin10:25:313
871799191,10cyclictest16808-21/usr/sbin/munin08:40:153
871199194,3cyclictest86650irq/127-enp1s0-09:03:172
870799198,3cyclictest9254-21diskstats07:10:161
870799198,3cyclictest9254-21diskstats07:10:151
870799198,3cyclictest17156-21diskstats08:40:151
870799196,4cyclictest10673-21irqstats10:55:191
870799191,9cyclictest19298-21sh10:00:261
8707991915,3cyclictest0-21swapper/112:33:091
8707991915,3cyclictest0-21swapper/108:52:011
870799191,2cyclictest0-21swapper/110:24:071
870799191,17cyclictest0-21swapper/111:26:431
870799191,17cyclictest0-21swapper/110:42:181
870799191,17cyclictest0-21swapper/110:31:581
870799191,17cyclictest0-21swapper/110:25:471
870799191,17cyclictest0-21swapper/108:15:471
870799190,4cyclictest0-21swapper/109:56:081
871799188,2cyclictest30019-21cpu09:10:133
871799184,6cyclictest0-21swapper/309:35:223
871799181,8cyclictest0-21swapper/308:05:193
871799181,7cyclictest24978-21/usr/sbin/munin07:45:223
871799181,7cyclictest0-21swapper/312:39:313
871799181,7cyclictest0-21swapper/310:33:133
871799181,7cyclictest0-21swapper/310:16:343
871799181,5cyclictest0-21swapper/311:07:513
871799181,3cyclictest25688-21cpuspeed_turbos09:00:133
8717991813,3cyclictest21514-21sed07:35:203
8717991812,3cyclictest20387-21cut10:05:113
8717991811,4cyclictest14280-21/usr/sbin/munin09:45:273
871799181,14cyclictest13648-21users11:00:273
871799181,11cyclictest0-21swapper/309:15:293
870799181,9cyclictest449-21ntpq11:45:231
8707991814,3cyclictest0-21swapper/111:36:421
870799181,16cyclictest0-21swapper/107:35:371
871799177,2cyclictest0-21swapper/309:51:393
871799173,7cyclictest88150irq/134-enp2s0-07:15:163
871799173,6cyclictest0-21swapper/311:45:243
871799171,9cyclictest0-21swapper/308:32:533
871799171,7cyclictest19181-21tune2fs11:15:163
871799171,6cyclictest32514-21/usr/sbin/munin08:00:203
871799171,6cyclictest0-21swapper/312:30:373
871799171,4cyclictest0-21swapper/309:30:103
871799171,3cyclictest0-21swapper/308:10:233
8717991712,3cyclictest1469-21cut10:35:173
8717991712,3cyclictest1469-21cut10:35:173
871799171,14cyclictest0-21swapper/309:05:153
8717991711,3cyclictest23552-21grep08:55:143
8717991711,3cyclictest18864-21uniq07:30:163
8717991710,4cyclictest6280-21/usr/sbin/munin08:15:173
8717991613,2cyclictest0-21swapper/307:21:003
8717991613,2cyclictest0-21swapper/307:20:593
106272161,2sleep30-21swapper/307:12:013
106272161,2sleep30-21swapper/307:12:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional