You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 02:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Nov 17, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
229202241165,25sleep30-21swapper/307:06:353
231102229158,58sleep10-21swapper/107:09:041
230112226194,21sleep00-21swapper/007:07:460
229912205167,25sleep20-21swapper/207:07:322
23278997875,2cyclictest0-21swapper/209:23:172
98512690,5sleep339-21rcuc/310:15:353
6242650,3sleep1636-21cpuspeed_turbos07:30:141
23278995350,2cyclictest0-21swapper/210:07:582
311562510,2sleep20-21swapper/212:20:152
2326499347,12cyclictest17679-21perf11:50:020
2326499292,19cyclictest0-21swapper/010:55:260
2327299287,11cyclictest16401-21idleruntime11:45:181
34502270,3sleep12327299cyclictest10:00:311
326642270,3sleep011-21rcuc/012:20:290
2326499271,18cyclictest0-21swapper/011:33:430
23264992710,8cyclictest2342-21wc11:15:010
2326499261,17cyclictest0-21swapper/010:22:400
2326499261,17cyclictest0-21swapper/010:22:400
23264992610,13cyclictest7600-21awk09:00:010
2326499260,17cyclictest0-21swapper/009:25:180
2327899256,6cyclictest0-21swapper/211:50:152
23272992519,3cyclictest12885-21users11:35:281
23272992511,12cyclictest21115-21kworker/u8:210:56:261
2326499259,3cyclictest298462uptime11:00:290
2326499257,10cyclictest0-21swapper/010:31:060
2326499251,17cyclictest0-21swapper/012:27:140
2326499251,17cyclictest0-21swapper/009:51:060
2326499251,16cyclictest0-21swapper/012:31:570
2326499251,16cyclictest0-21swapper/012:31:570
2326499250,17cyclictest0-21swapper/009:32:290
2328499246,3cyclictest32747-21awk07:30:013
2327899242,17cyclictest0-21swapper/210:17:492
2327299246,9cyclictest0-21swapper/112:15:111
2327299241,17cyclictest0-21swapper/112:37:381
23272992410,11cyclictest8313-21cut10:15:011
2327299240,17cyclictest0-21swapper/110:42:361
2326499249,3cyclictest2143-21memory08:45:210
2326499248,3cyclictest11897-21gltestperf11:35:170
2326499242,11cyclictest28571-21/usr/sbin/munin07:20:170
23264992419,2cyclictest24605-21/usr/sbin/munin07:10:240
2326499241,17cyclictest0-21swapper/011:50:410
2326499241,17cyclictest0-21swapper/011:18:590
2326499241,17cyclictest0-21swapper/011:05:180
2326499241,17cyclictest0-21swapper/010:29:190
2326499241,17cyclictest0-21swapper/010:08:040
2326499241,17cyclictest0-21swapper/009:13:450
2326499241,15cyclictest0-21swapper/009:38:030
2326499241,15cyclictest0-21swapper/009:18:420
2326499241,12cyclictest5408-21awk08:55:020
2326499241,10cyclictest23247-21users10:45:290
2327299238,3cyclictest28266-21/usr/sbin/munin11:00:191
2327299232,18cyclictest18743-21/usr/sbin/munin10:35:261
2327299231,17cyclictest0-21swapper/111:55:181
23272992311,4cyclictest9148-21kworker/u8:008:48:071
23272992310,10cyclictest836-21runrttasks12:01:541
23264992320,2cyclictest0-21swapper/012:05:280
23264992317,3cyclictest4706-21/usr/sbin/munin11:20:210
2326499231,16cyclictest0-21swapper/012:03:160
2326499231,16cyclictest0-21swapper/010:50:160
2326499231,16cyclictest0-21swapper/010:16:470
2326499231,15cyclictest0-21swapper/010:40:100
2326499231,15cyclictest0-21swapper/009:22:420
2326499230,14cyclictest20688-21kworker/u8:111:28:530
2326499230,14cyclictest20688-21kworker/u8:111:28:530
2328499221,7cyclictest0-21swapper/309:05:253
2328499221,11cyclictest20967-21grep10:40:273
2327899229,4cyclictest0-21swapper/210:55:192
2327899227,4cyclictest0-21swapper/212:00:242
2327899222,5cyclictest32739-21munin-run07:30:002
2327899222,11cyclictest0-21swapper/211:03:132
23278992216,3cyclictest911-21gltestperf11:10:162
23278992213,7cyclictest21471-21grep09:30:162
2327899221,13cyclictest0-21swapper/211:35:152
23272992215,2cyclictest0-21swapper/111:11:411
2327299221,3cyclictest32404-21cat12:20:271
2327299221,3cyclictest30748-21sed08:35:271
2327299221,3cyclictest19265-21ntp_states08:10:221
2327299221,3cyclictest19265-21ntp_states08:10:221
23272992212,4cyclictest0-21swapper/111:20:241
2327299221,18cyclictest30538-21munin-run07:25:001
2327299221,15cyclictest0-21swapper/112:32:231
2327299221,15cyclictest0-21swapper/112:32:231
2327299221,15cyclictest0-21swapper/110:29:001
2327299220,17cyclictest0-21swapper/111:50:251
2327299220,16cyclictest0-21swapper/110:05:101
2326499224,5cyclictest27062-21entropy12:10:170
2326499224,15cyclictest8675-21unixbench-2d07:45:280
2326499223,7cyclictest11-21rcuc/008:25:190
23264992219,2cyclictest0-21swapper/008:05:200
23264992218,2cyclictest0-21swapper/010:10:280
2326499221,4cyclictest0-21swapper/009:48:180
2326499221,16cyclictest0-21swapper/012:16:340
2326499221,16cyclictest0-21swapper/011:56:000
2326499221,15cyclictest0-21swapper/011:44:180
2326499221,15cyclictest0-21swapper/010:04:070
2326499221,15cyclictest0-21swapper/009:44:100
23264992210,2cyclictest32368-21vmstat07:25:290
2326499220,15cyclictest0-21swapper/010:35:110
17542221,3sleep11855-21memory12:25:201
2328499219,3cyclictest6897-21unixbench_singl12:35:313
2328499211,6cyclictest0-21swapper/307:45:203
2328499211,16cyclictest4671-21grep10:05:193
2328499211,11cyclictest21953-21unixbench_singl08:15:313
2328499211,11cyclictest18962-21ntp_states11:50:243
2328499211,11cyclictest15170-21sshd11:41:213
2328499211,10cyclictest31994-21grep08:40:163
2328499210,8cyclictest784-21NetworkManager12:16:283
2327899219,3cyclictest0-21swapper/210:50:242
2327899215,3cyclictest0-21swapper/210:04:472
2327899212,11cyclictest0-21swapper/211:41:412
2327899211,4cyclictest0-21swapper/209:59:402
2327899211,16cyclictest11535-21grep09:05:282
2327299212,10cyclictest24509-21/usr/sbin/munin12:05:311
23272992117,3cyclictest0-21swapper/107:40:191
23272992114,5cyclictest0-21swapper/109:35:331
23272992113,3cyclictest0-21swapper/110:48:031
23272992112,4cyclictest0-21swapper/111:15:401
2327299211,18cyclictest0-21swapper/111:26:231
2327299211,18cyclictest0-21swapper/111:26:231
2327299211,18cyclictest0-21swapper/110:33:331
2327299211,18cyclictest0-21swapper/107:16:131
2327299211,15cyclictest0-21swapper/111:40:241
2327299211,15cyclictest0-21swapper/111:06:351
2327299211,15cyclictest0-21swapper/110:20:031
2327299211,15cyclictest0-21swapper/110:20:031
2327299211,15cyclictest0-21swapper/109:48:511
2327299210,16cyclictest0-21swapper/110:16:311
2327299210,16cyclictest0-21swapper/109:41:001
2326499216,6cyclictest0-21swapper/012:35:160
2326499211,9cyclictest9997-21gltestperf07:50:180
2326499211,9cyclictest22979-21fschecks_count08:20:180
2326499211,9cyclictest0-21swapper/008:41:230
23264992117,1cyclictest81ktimersoftd/007:35:290
2326499211,11cyclictest0-21swapper/009:55:140
2326499211,10cyclictest0-21swapper/008:35:140
2326499211,10cyclictest0-21swapper/008:03:160
2328499205,3cyclictest12050irq/123-ahci[0009:25:013
2328499203,14cyclictest1-21systemd10:50:103
2328499201,6cyclictest4410-21runrttasks11:19:193
2328499201,11cyclictest2805-21/usr/sbin/munin07:35:273
2328499201,10cyclictest17257-21/usr/sbin/munin10:35:263
2327899208,2cyclictest12004-21diskmemload09:35:372
2327899206,3cyclictest28353-21cat08:30:272
2327899205,3cyclictest836-21runrttasks10:10:132
2327899203,7cyclictest0-21swapper/207:24:512
2327899203,4cyclictest29598-21munin-run09:50:012
2327899202,15cyclictest21276-21grep08:15:242
2327899201,9cyclictest4411-21taskset11:19:202
2327899201,9cyclictest0-21swapper/210:25:282
2327899201,7cyclictest27392-21latency_hist09:45:022
2327899201,7cyclictest11447-21cat07:55:012
2327899201,7cyclictest0-21swapper/212:08:522
2327899201,7cyclictest0-21swapper/210:44:132
2327899201,6cyclictest30820-21diskstats11:05:162
2327899201,18cyclictest0-21swapper/208:36:522
2327899201,10cyclictest0-21swapper/212:28:252
2327899201,10cyclictest0-21swapper/209:25:122
2327299209,2cyclictest2865-21egrep07:35:141
2327299201,18cyclictest0-21swapper/109:56:381
2327299201,17cyclictest0-21swapper/111:32:011
2327299201,17cyclictest0-21swapper/108:50:191
2327299201,15cyclictest0-21swapper/110:54:521
2327299201,10cyclictest28316-21chrt12:10:291
2327299201,10cyclictest20771-21cat09:29:551
2326499201,7cyclictest26371-21/usr/sbin/munin07:15:150
2326499201,7cyclictest12013-21basename09:10:010
2326499201,6cyclictest1500-21memory07:30:210
74182191,2sleep10-21swapper/108:57:581
27652191,3sleep30-21swapper/312:28:203
2328499191,7cyclictest0-21swapper/309:36:353
2328499191,6cyclictest0-21swapper/310:02:143
2328499191,5cyclictest0-21swapper/310:20:143
2328499191,5cyclictest0-21swapper/310:20:143
2328499191,3cyclictest39-21rcuc/308:55:083
23284991912,4cyclictest19851-21ntpq09:25:223
2328499191,10cyclictest29823-21unixbench_singl11:00:293
2328499191,10cyclictest26358-21sed08:25:283
2328499191,10cyclictest23478-21/usr/sbin/munin07:10:183
2328499190,8cyclictest14634-21snmpd12:32:093
2328499190,8cyclictest14634-21snmpd12:32:093
2327899197,3cyclictest0-21swapper/210:32:522
2327899195,3cyclictest86850irq/129-enp1s0-10:45:212
2327899195,3cyclictest12050irq/123-ahci[0008:05:012
2327899195,3cyclictest12004-21diskmemload11:30:282
2327899194,3cyclictest86550irq/126-enp1s0-08:45:182
2327899192,6cyclictest848-21sssd_nss12:35:262
2327899192,5cyclictest18998-21interrupts08:10:192
2327899192,5cyclictest18998-21interrupts08:10:192
2327899191,9cyclictest0-21swapper/212:13:332
2327899191,7cyclictest0-21swapper/211:25:412
2327899191,7cyclictest0-21swapper/211:25:412
2327899191,6cyclictest31084-21smart_sda09:50:252
2327899191,6cyclictest28908-21/usr/sbin/munin12:15:252
2327899191,6cyclictest0-21swapper/211:23:172
2327899191,5cyclictest22615-21wc08:20:152
23278991914,3cyclictest9168-21wc09:00:282
23278991913,3cyclictest13107-21unixbench_multi07:55:292
2327899191,16cyclictest0-21swapper/207:45:212
2327899191,10cyclictest0-21swapper/211:55:092
2327899191,10cyclictest0-21swapper/207:14:232
2327299198,2cyclictest12004-21diskmemload09:20:281
23272991916,2cyclictest0-21swapper/107:45:361
2327299191,2cyclictest12253-21cpu09:10:131
2327299191,2cyclictest0-21swapper/108:40:181
23272991912,4cyclictest9378-21head09:00:291
23272991912,2cyclictest0-21swapper/109:52:591
2327299191,17cyclictest0-21swapper/109:05:121
2327299191,17cyclictest0-21swapper/108:30:101
2327299191,17cyclictest0-21swapper/108:21:271
2327299191,17cyclictest0-21swapper/107:51:501
2327299191,17cyclictest0-21swapper/107:12:151
2327299191,10cyclictest14495-21cpuspeed_turbos09:15:131
2327299190,18cyclictest0-21swapper/108:16:331
2327299190,15cyclictest0-21swapper/109:31:091
2326499197,2cyclictest0-21swapper/008:30:280
2326499191,6cyclictest19006-21irqstats08:10:190
2326499191,6cyclictest19006-21irqstats08:10:190
155602191,3sleep10-21swapper/108:02:301
132402191,2sleep10-21swapper/107:56:041
2328499189,2cyclictest86550irq/126-enp1s0-09:42:463
2328499188,2cyclictest0-21swapper/307:30:143
2328499186,3cyclictest0-21swapper/312:10:183
2328499185,4cyclictest12004-21diskmemload09:57:483
2328499183,4cyclictest0-21swapper/309:46:023
2328499182,6cyclictest26715-21df_inode07:15:143
2328499181,8cyclictest768-21in:imjournal12:20:103
2328499181,8cyclictest0-21swapper/311:55:203
2328499181,7cyclictest12004-21diskmemload11:05:143
2328499181,7cyclictest0-21swapper/312:05:413
2328499181,6cyclictest22665-21netstat10:45:223
2328499181,6cyclictest15886-21cat08:05:013
23284991812,3cyclictest16647-21users10:30:283
2328499181,15cyclictest0-21swapper/307:55:483
23284991811,4cyclictest22394-21grep12:00:143
2328499181,10cyclictest28506-21unixbench_multi08:30:283
2328499181,10cyclictest0-21swapper/310:55:173
2328499181,10cyclictest0-21swapper/310:13:213
2328499180,5cyclictest0-21swapper/307:40:163
2327899187,2cyclictest0-21swapper/210:35:042
2327899185,3cyclictest12050irq/123-ahci[0010:25:012
2327899185,3cyclictest12050irq/123-ahci[0010:25:012
2327899181,9cyclictest0-21swapper/207:17:052
2327899181,8cyclictest0-21swapper/209:15:142
2327899181,7cyclictest3426-21/usr/sbin/munin08:50:172
2327899181,7cyclictest0-21swapper/208:28:382
2327899181,7cyclictest0-21swapper/207:43:162
2327899181,6cyclictest2805-21/usr/sbin/munin07:35:162
2327899181,6cyclictest0-21swapper/209:10:132
2327899181,5cyclictest17962-21latency_hist08:10:012
2327899181,5cyclictest0-21swapper/212:31:342
2327899181,5cyclictest0-21swapper/212:31:342
2327899181,5cyclictest0-21swapper/211:48:552
2327899181,4cyclictest25588-21munin-node07:30:292
23278991813,3cyclictest31809-21fschecks_count08:40:152
23272991814,2cyclictest31583-21kernelversion07:25:201
2327299180,1cyclictest0-21swapper/108:05:201
23264991812,3cyclictest6563-21sed07:40:300
2326499181,15cyclictest0-21swapper/008:15:410
140942181,3sleep30-21swapper/309:13:503
2328499171,7cyclictest23843-21sendmail_mailst08:20:273
2328499171,7cyclictest0-21swapper/311:45:193
2328499171,7cyclictest0-21swapper/311:30:483
2328499171,6cyclictest0-21swapper/310:25:143
2328499171,6cyclictest0-21swapper/309:50:033
2328499171,6cyclictest0-21swapper/309:15:263
23284991712,3cyclictest32671-21/usr/sbin/munin11:10:173
23284991711,3cyclictest4923-21expr11:20:133
23284991711,3cyclictest4381-21cat08:50:213
23284991710,4cyclictest8478-21gltestperf09:00:193
2327899171,4cyclictest6431-21irqstats08:55:182
2327299171,9cyclictest0-21swapper/108:25:131
2326499171,7cyclictest0-21swapper/007:55:240
2326499171,4cyclictest7821-21/usr/sbin/munin09:00:200
112812171,2sleep30-21swapper/311:35:053
2328499161,8cyclictest86850irq/129-enp1s0-08:05:283
2328499161,6cyclictest29202-21acpi08:35:143
2328499161,6cyclictest10384-21strings07:50:213
2328499161,14cyclictest0-21swapper/308:45:203
2328499161,13cyclictest6903-21/usr/sbin/munin11:25:193
2328499161,13cyclictest6903-21/usr/sbin/munin11:25:193
2328499161,13cyclictest21018-21/usr/sbin/munin09:30:263
23284991611,3cyclictest20259-21date08:15:013
23284991611,3cyclictest20259-21date08:15:013
2328499149,3cyclictest28693-21cut07:20:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional