You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-22 - 12:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu May 22, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
90652240163,25sleep00-21swapper/019:07:080
90462237162,23sleep10-21swapper/119:06:531
92522231160,57sleep30-21swapper/319:09:313
89882201163,25sleep20-21swapper/219:06:082
1009521830,4sleep1938299cyclictest00:02:091
2439921520,4sleep1938299cyclictest22:09:161
1401921090,3sleep123-21ksoftirqd/119:15:281
9386991081,105cyclictest0-21swapper/222:39:292
314512550,2sleep00-21swapper/022:25:170
314512550,2sleep00-21swapper/022:25:170
303592550,4sleep339-21rcuc/323:35:183
68322430,1sleep00-21swapper/023:55:210
938699413,32cyclictest31626-40dnf23:39:342
938699413,32cyclictest17119-40dnf20:38:332
938699381,22cyclictest1494-21latency_hist23:45:022
938699371,25cyclictest21872-21/usr/sbin/munin20:50:152
938699370,20cyclictest12050irq/123-ahci[0000:00:022
938299370,15cyclictest0-21swapper/121:45:011
938699363,23cyclictest30451-21diskmemload00:00:322
938699361,27cyclictest86850irq/129-enp1s0-22:31:192
938699361,20cyclictest2839-21tr21:20:132
938699361,19cyclictest8827-21/usr/sbin/munin22:45:252
938699360,24cyclictest0-21swapper/220:10:162
938699360,24cyclictest0-21swapper/220:10:152
938299361,2cyclictest0-21swapper/123:14:501
937499367,13cyclictest19143-21awk00:25:000
938699351,26cyclictest17138-21sendmail_mailst21:50:232
938699351,26cyclictest16552-21cpu23:05:162
938699351,22cyclictest28484-21users22:15:292
938699351,22cyclictest14471-21runrttasks00:11:462
938699351,17cyclictest23636-21pmu-power22:05:252
938299359,3cyclictest3093-21latency_hist22:35:011
938299351,4cyclictest5257-21diskstats21:25:141
938699342,22cyclictest29443-21diskstats22:20:172
938699341,24cyclictest7455-21diskstats21:30:152
938699341,19cyclictest10884-21/usr/sbin/munin20:25:202
938299345,18cyclictest780-21rngd20:26:401
938699331,29cyclictest18444-21diskstats21:55:162
938699331,19cyclictest20686-21latency_hist23:15:002
938299332,22cyclictest780-21rngd20:12:401
938299332,22cyclictest780-21rngd20:12:401
938299331,11cyclictest26608-21df_inode21:00:141
938699326,18cyclictest86550irq/126-enp1s0-19:20:262
938699324,19cyclictest86550irq/126-enp1s0-19:55:182
9386993228,2cyclictest86650irq/127-enp1s0-23:49:232
938699321,24cyclictest0-21swapper/200:17:512
938699321,20cyclictest836-21runrttasks00:36:212
938299321,4cyclictest780-21rngd19:35:501
9374993212,17cyclictest31686-21munin-run23:40:010
938699319,11cyclictest86850irq/129-enp1s0-23:30:332
938699319,11cyclictest86850irq/129-enp1s0-23:30:332
938699315,17cyclictest86750irq/128-enp1s0-20:45:162
938699312,25cyclictest32363-21ntp_states22:25:272
938699312,25cyclictest32363-21ntp_states22:25:272
938699311,22cyclictest24064-21/usr/sbin/munin20:55:182
938699311,22cyclictest13996-21chrt20:30:192
938699311,21cyclictest27481-21cat23:30:022
938699311,21cyclictest12660-21cat00:10:022
938699311,18cyclictest11504-21/usr/sbin/munin21:40:202
938699310,27cyclictest16856-21/usr/sbin/munin19:25:152
938699310,22cyclictest0-21swapper/223:20:162
938699310,19cyclictest2047-21grep21:15:282
938699310,19cyclictest2047-21grep21:15:282
938299317,11cyclictest86650irq/127-enp1s0-19:28:381
938299314,10cyclictest780-21rngd19:43:561
938299312,13cyclictest0-21swapper/120:53:281
938299310,10cyclictest0-21swapper/121:09:341
6892310,5sleep121-21rcuc/120:00:191
938699304,18cyclictest14665-21diskstats23:00:162
938699303,19cyclictest86550irq/126-enp1s0-19:40:142
938699301,26cyclictest9929-21/usr/sbin/munin22:50:322
938699301,26cyclictest9300-21/usr/sbin/munin21:35:252
938699301,26cyclictest848-21sssd_nss19:30:142
938699301,26cyclictest6488-21/usr/sbin/munin21:25:262
938699301,26cyclictest6481-21/usr/sbin/munin20:15:152
938699301,26cyclictest3945-21/usr/sbin/munin23:50:272
938699301,26cyclictest30678-21/usr/sbin/munin21:10:202
938699301,26cyclictest26266-21/usr/sbin/munin21:00:162
938699301,26cyclictest25650-21/usr/sbin/munin19:45:142
938699301,26cyclictest2450-21sshd20:05:152
938699301,26cyclictest20292-21/usr/sbin/munin22:00:252
938699301,26cyclictest17510-21diskstats00:20:162
938699301,25cyclictest25045-21diskstats22:10:162
938699301,21cyclictest19358-21/usr/sbin/munin00:25:202
938699301,19cyclictest86750irq/128-enp1s0-21:05:272
938299304,14cyclictest780-21rngd21:38:261
938299304,12cyclictest780-21rngd19:47:041
9382993017,4cyclictest21592-21hddtemp_smartct23:15:171
938299301,15cyclictest0-21swapper/122:48:041
938299301,13cyclictest0-21swapper/123:07:401
938299301,13cyclictest0-21swapper/122:35:381
938299301,12cyclictest0-21swapper/121:53:281
938299300,7cyclictest0-21swapper/123:23:441
938299300,24cyclictest0-21swapper/122:11:381
938299300,10cyclictest0-21swapper/119:24:401
938699291,25cyclictest9579-21/usr/sbin/munin19:10:162
938699291,25cyclictest5525-21/usr/sbin/munin22:40:242
938699291,25cyclictest32245-21/usr/sbin/munin20:00:142
938699291,25cyclictest27853-21/usr/sbin/munin19:50:162
938699291,25cyclictest21907-21/usr/sbin/munin00:30:162
938699291,25cyclictest21255-21/usr/sbin/munin19:35:182
938699291,25cyclictest20917-21/usr/sbin/munin23:15:162
938699291,25cyclictest18588-21/usr/sbin/munin20:40:242
938699291,25cyclictest14392-21/usr/sbin/munin21:45:202
938699291,25cyclictest13142-21hddtemp_smartct19:15:182
938699291,25cyclictest12121-21/usr/sbin/munin22:55:232
9386992910,16cyclictest9031-21diskstats20:20:162
938299291,12cyclictest0-21swapper/122:50:161
938299291,11cyclictest0-21swapper/119:14:431
938299290,3cyclictest0-21swapper/120:37:481
938299290,14cyclictest0-21swapper/123:56:321
938299290,13cyclictest0-21swapper/122:16:421
207302290,4sleep0937499cyclictest22:00:160
938299285,11cyclictest780-21rngd20:47:301
938299281,13cyclictest0-21swapper/121:58:161
938299281,13cyclictest0-21swapper/120:24:021
938299280,13cyclictest0-21swapper/121:45:421
938299280,11cyclictest0-21swapper/121:20:561
938299273,12cyclictest780-21rngd20:09:141
938299271,11cyclictest0-21swapper/121:19:201
938299271,11cyclictest0-21swapper/121:19:201
938299270,8cyclictest0-21swapper/122:25:461
938299270,8cyclictest0-21swapper/122:25:461
938299270,13cyclictest0-21swapper/100:32:301
938299270,12cyclictest0-21swapper/123:44:101
937499274,11cyclictest9696-21munin-run22:50:010
937499271,9cyclictest0-21swapper/023:00:210
939299261,9cyclictest0-21swapper/321:25:233
938299266,3cyclictest0-21swapper/123:45:261
938299262,6cyclictest0-21swapper/123:00:021
938299261,12cyclictest0-21swapper/100:20:381
938299261,11cyclictest780-21rngd19:52:241
938299260,8cyclictest0-21swapper/121:30:261
938299260,13cyclictest0-21swapper/122:00:441
938299260,12cyclictest0-21swapper/122:41:401
938299260,11cyclictest0-21swapper/122:56:421
939299253,7cyclictest0-21swapper/319:22:033
938299253,10cyclictest780-21rngd20:18:241
9382992519,3cyclictest24742-21hddtemp_smartct20:55:161
938299251,12cyclictest780-21rngd23:31:021
938299251,12cyclictest780-21rngd23:31:021
938299250,2cyclictest0-21swapper/100:18:001
938299250,12cyclictest0-21swapper/100:38:561
938299250,11cyclictest0-21swapper/100:06:501
939299241,15cyclictest0-21swapper/300:26:293
938299243,10cyclictest780-21rngd19:33:261
938299241,13cyclictest0-21swapper/120:40:241
9374992417,4cyclictest29812-21sh19:55:000
937499241,4cyclictest8346-21/usr/sbin/munin00:00:180
160762240,5sleep011-21rcuc/020:35:180
121672240,1sleep00-21swapper/022:55:160
938299230,10cyclictest0-21swapper/121:10:141
937499237,4cyclictest0-21swapper/022:30:250
939299226,4cyclictest0-21swapper/323:01:173
9392992210,3cyclictest5017-21ntp_states23:50:253
938299221,18cyclictest31626-40dnf23:39:331
938299221,13cyclictest0-21swapper/100:11:421
937499227,5cyclictest0-21swapper/021:52:380
937499227,12cyclictest20692-21munin-run23:15:000
937499222,5cyclictest0-21swapper/023:30:240
937499222,5cyclictest0-21swapper/023:30:240
937499222,11cyclictest12469-21/usr/sbin/munin19:15:170
939299218,2cyclictest11504-21/usr/sbin/munin21:40:153
939299211,11cyclictest18726-21/usr/sbin/munin23:10:263
939299211,11cyclictest13705-21/usr/sbin/munin21:45:243
938299213,15cyclictest13750-21sed20:30:151
938299211,4cyclictest4550-21expr23:50:181
938299210,11cyclictest0-21swapper/119:55:521
937499218,3cyclictest24961-21grep19:40:260
937499211,8cyclictest11849-21df_inode21:40:150
937499211,14cyclictest0-21swapper/021:30:230
937499211,12cyclictest0-21swapper/019:55:180
937499211,11cyclictest4293-21/usr/sbin/munin20:10:290
937499211,11cyclictest4293-21/usr/sbin/munin20:10:290
937499211,11cyclictest31464-21if_err_enp2s021:10:200
937499211,10cyclictest3646-21grep20:05:260
937499211,10cyclictest24009-21tail22:05:280
937499210,7cyclictest0-21swapper/022:39:110
939299209,3cyclictest26893-21/usr/sbin/munin22:15:213
939299205,3cyclictest18336-21wc00:20:243
939299205,3cyclictest14325-21head00:10:273
939299203,4cyclictest30451-21diskmemload00:05:023
939299201,6cyclictest0-21swapper/300:31:513
939299201,12cyclictest0-21swapper/319:57:533
939299200,4cyclictest4569-21kworker/u8:123:48:233
938299205,3cyclictest0-21swapper/123:26:061
9382992013,2cyclictest0-21swapper/100:26:071
937499209,4cyclictest21336-21sshd20:46:160
937499207,4cyclictest0-21swapper/021:22:520
937499206,5cyclictest0-21swapper/000:10:240
937499202,6cyclictest9923-21df_inode19:10:140
937499201,16cyclictest30451-21diskmemload22:20:140
937499201,12cyclictest24506-21sshd00:35:180
937499201,10cyclictest581-21systemd-journal21:10:000
937499201,10cyclictest14634-21snmpd22:15:480
937499201,10cyclictest0-21swapper/023:25:190
34062200,4sleep339-21rcuc/321:20:183
207992200,4sleep339-21rcuc/319:32:163
205372200,2sleep081ktimersoftd/000:25:240
150272201,3sleep315296-21df00:15:133
109762201,3sleep010994-21ntp_states22:50:270
939299198,4cyclictest842-21lldpd22:51:073
939299196,3cyclictest0-21swapper/320:12:513
939299196,3cyclictest0-21swapper/320:12:513
939299195,3cyclictest39-21rcuc/321:15:153
939299195,3cyclictest39-21rcuc/321:15:153
939299193,3cyclictest17186-21grep23:05:203
939299192,9cyclictest12469-21/usr/sbin/munin19:15:273
939299192,8cyclictest0-21swapper/323:55:153
939299191,9cyclictest18093-21/usr/sbin/munin21:55:253
939299191,6cyclictest26545-21sendmail_mailqu23:25:243
939299191,6cyclictest23816-21cpuspeed_turbos00:35:153
939299191,10cyclictest9579-21/usr/sbin/munin19:10:173
939299191,10cyclictest30451-21diskmemload23:20:063
939299191,10cyclictest27853-21/usr/sbin/munin19:50:243
939299191,10cyclictest27511-21/usr/sbin/munin23:30:243
939299191,10cyclictest27511-21/usr/sbin/munin23:30:243
939299191,10cyclictest24368-21memory19:40:203
939299191,10cyclictest22976-21seq20:50:243
939299191,10cyclictest22791-21cat22:05:163
939299191,10cyclictest10884-21/usr/sbin/munin20:25:213
9382991916,2cyclictest0-21swapper/122:22:041
937499198,3cyclictest0-21swapper/000:30:190
937499195,12cyclictest11-21rcuc/023:15:260
937499193,6cyclictest17517-21strings23:05:230
937499192,6cyclictest26286-21users22:10:280
937499191,9cyclictest21275-21cpu19:35:140
937499191,6cyclictest14254-21ps20:30:230
9374991914,3cyclictest2669-21grep23:45:200
937499191,10cyclictest31910-21/usr/sbin/munin23:40:260
937499190,12cyclictest0-21swapper/021:45:220
247742191,2sleep00-21swapper/023:21:140
939299184,4cyclictest26218-21sort21:00:023
939299182,3cyclictest2698-21head20:05:173
939299181,9cyclictest27393-21sshd19:47:153
939299181,8cyclictest0-21swapper/322:48:193
939299181,5cyclictest17481-21/usr/sbin/munin20:40:263
939299181,5cyclictest12314-21sshd00:07:193
939299181,5cyclictest0-21swapper/322:55:223
939299181,3cyclictest0-21swapper/322:13:133
9392991813,3cyclictest22268-21dump-pmu-power22:05:013
9392991813,3cyclictest20917-21/usr/sbin/munin23:15:193
9392991813,3cyclictest20727-21ntp_offset20:45:213
9392991811,5cyclictest848-21sssd_nss22:35:253
9392991811,4cyclictest817-21grep23:40:263
9392991811,4cyclictest6205-21smartctl22:40:183
939299181,10cyclictest842-21lldpd21:54:073
937499188,2cyclictest0-21swapper/021:35:260
937499186,6cyclictest86550irq/126-enp1s0-23:51:040
937499186,3cyclictest0-21swapper/020:54:060
937499182,8cyclictest30451-21diskmemload00:15:080
937499181,7cyclictest10169-21unixbench-2d20:20:260
937499181,6cyclictest0-21swapper/021:27:490
937499181,6cyclictest0-21swapper/020:00:130
9374991812,3cyclictest17798-21ls20:40:150
937499181,11cyclictest0-21swapper/000:07:150
937499180,7cyclictest7082-21unixbench_singl22:40:270
208952181,2sleep00-21swapper/019:33:290
939299178,3cyclictest0-21swapper/320:30:253
939299176,4cyclictest10247-21head20:20:273
939299176,3cyclictest86550irq/126-enp1s0-21:30:223
939299173,3cyclictest86550irq/126-enp1s0-22:25:203
939299173,3cyclictest86550irq/126-enp1s0-22:25:203
939299172,3cyclictest29093-21/usr/sbin/munin22:20:293
939299172,12cyclictest26864-21grep21:00:173
939299171,9cyclictest0-21swapper/322:30:303
939299171,7cyclictest29541-21ntp_states21:05:263
939299171,2cyclictest0-21swapper/320:15:073
9392991712,3cyclictest18945-21cut19:30:023
9392991712,3cyclictest15865-21gltestperf20:35:173
9392991712,3cyclictest11274-21sadc21:40:023
9392991712,3cyclictest0-21swapper/321:10:183
939299171,14cyclictest0-21swapper/319:35:033
937499178,3cyclictest0-21swapper/020:15:160
937499177,2cyclictest26868-21gltestperf21:00:160
937499176,3cyclictest0-21swapper/019:25:150
937499172,5cyclictest14666-21/usr/sbin/munin19:20:200
937499172,12cyclictest11-21rcuc/020:25:200
937499171,6cyclictest0-21swapper/020:55:170
937499171,5cyclictest19547-21cat21:55:260
937499171,3cyclictest25650-21/usr/sbin/munin19:45:140
937499171,10cyclictest0-21swapper/021:15:270
937499171,10cyclictest0-21swapper/021:15:270
9392991611,3cyclictest32590-21df_inode20:00:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional