You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 20:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Nov 05, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
276052241162,26sleep00-21swapper/007:08:170
277172238170,21sleep20-21swapper/207:09:442
274792237163,23sleep10-21swapper/107:06:411
274202233164,55sleep30-21swapper/307:05:553
1938421670,5sleep32784199cyclictest08:00:183
27836991181,115cyclictest0-21swapper/212:23:022
27836998480,3cyclictest0-21swapper/208:50:472
172552640,3sleep02782499cyclictest09:10:160
162642630,2sleep30-21swapper/311:35:153
157792560,4sleep339-21rcuc/310:20:163
107722560,2sleep20-21swapper/211:20:242
215972500,3sleep121601-21timerandwakeup11:45:291
27836993734,2cyclictest0-21swapper/207:18:512
27824993710,12cyclictest9043-21awk11:20:010
27841993617,3cyclictest14085-21runrttasks10:15:233
64262350,16sleep00-21swapper/012:25:230
2784199351,14cyclictest8996-21sshd11:19:443
2783699352,27cyclictest15430-21kworker/2:108:15:182
2782499357,13cyclictest25996-21awk10:45:000
2784199331,14cyclictest0-21swapper/309:45:233
2782499326,12cyclictest26932-21idleruntime-cro08:20:010
27841993115,7cyclictest0-21swapper/307:47:453
2782499311,15cyclictest11881-21awk12:40:020
27841993019,3cyclictest0-21swapper/308:40:153
2784199301,20cyclictest780-21rngd09:07:173
2784199301,12cyclictest0-21swapper/310:13:273
2783299305,3cyclictest0-21swapper/108:20:231
2782499305,12cyclictest3364-21munin-run08:40:000
123682300,3sleep20-21swapper/211:25:182
2784199298,4cyclictest0-21swapper/310:35:253
27841992911,4cyclictest5716-21cstates12:25:153
2782499299,17cyclictest2441-21munin-run11:05:000
2782499295,11cyclictest2744-21idleruntime-cro07:25:010
2784199289,7cyclictest0-21swapper/309:29:133
2784199286,8cyclictest0-21swapper/311:47:433
2784199286,6cyclictest0-21swapper/309:53:193
2784199281,10cyclictest0-21swapper/309:13:513
2784199280,8cyclictest0-21swapper/312:00:313
2784199280,12cyclictest27229-21netstat10:45:233
27841992711,3cyclictest0-21swapper/311:08:233
27841992711,3cyclictest0-21swapper/311:08:233
2784199271,13cyclictest0-21swapper/309:20:453
2784199271,12cyclictest0-21swapper/310:07:033
2784199271,11cyclictest0-21swapper/312:31:453
2784199270,11cyclictest0-21swapper/307:36:313
27832992711,4cyclictest780-21rngd07:59:461
2784199269,9cyclictest0-21swapper/307:42:433
2784199269,2cyclictest117172chrt12:38:173
2784199268,3cyclictest0-21swapper/309:31:433
2784199261,3cyclictest11800-21diskstats11:25:133
27841992613,3cyclictest0-21swapper/311:53:493
2784199261,15cyclictest0-21swapper/309:55:453
27841992611,4cyclictest14861-21memory07:50:193
27841992611,4cyclictest14861-21memory07:50:193
2784199261,10cyclictest25715-21ntp_states11:55:213
2784199260,13cyclictest0-21swapper/310:44:293
2783299268,8cyclictest0-21swapper/111:05:361
2783299268,8cyclictest0-21swapper/111:05:361
2783299262,3cyclictest2201-21irqstats08:35:201
2783299261,19cyclictest0-21swapper/112:15:111
2782499261,12cyclictest17828-21munin-run11:40:010
183332261,4sleep11-21systemd08:00:101
2784199259,3cyclictest0-21swapper/307:13:193
2784199253,2cyclictest86650irq/127-enp1s0-11:23:133
27841992520,4cyclictest0-21swapper/307:31:073
2783699252,8cyclictest15430-21kworker/2:109:45:262
2783699251,5cyclictest16782-21sort11:35:192
2783299251,17cyclictest0-21swapper/108:25:141
2782499257,15cyclictest28840-21munin-run12:05:010
2782499252,11cyclictest20041-21basename11:45:010
2784199249,3cyclictest27147-21/usr/sbin/munin08:20:213
27841992411,7cyclictest0-21swapper/309:35:113
2784199241,11cyclictest0-21swapper/308:50:533
2784199240,7cyclictest0-21swapper/312:19:593
2784199240,10cyclictest0-21swapper/310:28:593
2784199240,10cyclictest0-21swapper/309:19:133
2784199240,10cyclictest0-21swapper/308:09:393
27836992420,3cyclictest0-21swapper/207:48:152
2783299241,18cyclictest0-21swapper/111:30:581
2783299241,18cyclictest0-21swapper/109:26:591
2783299241,11cyclictest22350-21grep09:20:231
27832992410,12cyclictest27794-21kworker/u8:208:40:161
2783299240,10cyclictest0-21swapper/109:55:221
2782499241,11cyclictest30016-21awk07:15:020
176922240,4sleep02782499cyclictest07:55:290
304992231,2sleep00-21swapper/007:15:050
2784199238,3cyclictest0-21swapper/310:03:493
2784199238,3cyclictest0-21swapper/310:03:493
2784199231,14cyclictest0-21swapper/309:04:233
2784199231,11cyclictest16552-21diskmemload09:43:313
27841992310,3cyclictest0-21swapper/308:27:433
2784199230,9cyclictest0-21swapper/312:05:393
2784199230,9cyclictest0-21swapper/308:30:513
2784199230,3cyclictest0-21swapper/308:10:573
2784199230,10cyclictest0-21swapper/310:51:273
2783699232,13cyclictest15430-21kworker/2:110:31:332
2783699232,10cyclictest8812-21kworker/2:011:51:412
27836992316,4cyclictest4510-21grep07:25:282
2783299232,7cyclictest0-21swapper/110:15:281
2783299231,6cyclictest0-21swapper/111:13:441
2784199229,3cyclictest0-21swapper/312:21:273
2784199229,3cyclictest0-21swapper/310:33:473
27841992213,8cyclictest0-21swapper/308:17:113
2784199220,9cyclictest0-21swapper/312:12:573
2784199220,9cyclictest0-21swapper/312:12:573
2784199220,4cyclictest0-21swapper/310:57:313
2783699222,9cyclictest15430-21kworker/2:110:12:222
2783699221,9cyclictest8812-21kworker/2:011:17:412
2783699221,8cyclictest2671-21/usr/sbin/munin11:05:182
2783699221,8cyclictest2671-21/usr/sbin/munin11:05:182
2783699221,5cyclictest0-21swapper/209:41:262
2783699221,10cyclictest7301-21kworker/2:112:36:452
2783299229,11cyclictest6009-21kworker/u8:010:00:121
2783299229,11cyclictest6009-21kworker/u8:010:00:121
2783299224,17cyclictest0-21swapper/112:05:151
2783299224,14cyclictest86850irq/129-enp1s0-08:17:481
2783299222,18cyclictest0-21swapper/112:35:221
2783299222,11cyclictest1104-21cat09:45:281
2783299221,6cyclictest0-21swapper/110:20:411
2783299221,6cyclictest0-21swapper/109:37:051
27832992214,3cyclictest0-21swapper/111:18:181
2783299221,15cyclictest0-21swapper/108:14:401
27824992216,3cyclictest7034-21grep10:00:160
27824992216,3cyclictest7034-21grep10:00:160
2784199213,3cyclictest0-21swapper/307:55:533
2784199211,12cyclictest842-21lldpd11:40:313
2784199211,11cyclictest0-21swapper/311:00:393
2784199211,10cyclictest0-21swapper/311:31:393
27841992110,3cyclictest0-21swapper/308:38:433
2783299216,9cyclictest496-21jbd2/dm-0-810:50:261
2783299216,3cyclictest87950irq/132-enp2s0-08:30:221
2783299215,8cyclictest31952-21awk09:45:011
2783299212,10cyclictest0-21swapper/111:25:231
2783299211,9cyclictest0-21swapper/108:53:121
27832992117,3cyclictest0-21swapper/109:32:071
2783299211,5cyclictest0-21swapper/110:48:511
2783299211,2cyclictest19961-21df_inode10:30:181
2783299211,19cyclictest0-21swapper/112:32:321
2783299211,18cyclictest0-21swapper/112:11:491
2783299211,18cyclictest0-21swapper/112:11:491
2783299211,18cyclictest0-21swapper/111:55:221
2783299211,18cyclictest0-21swapper/110:35:201
2783299211,18cyclictest0-21swapper/109:17:001
2783299211,18cyclictest0-21swapper/107:11:011
2783299211,15cyclictest0-21swapper/108:47:421
2783299211,11cyclictest18372-21users09:10:281
2783299211,11cyclictest16533-21cat09:09:571
2783299211,10cyclictest0-21swapper/110:29:311
2783299210,15cyclictest86850irq/129-enp1s0-12:02:221
2782499211,11cyclictest21178-21/usr/sbin/munin09:20:150
2784199204,10cyclictest0-21swapper/307:15:153
2784199201,12cyclictest0-21swapper/308:55:233
2784199201,12cyclictest0-21swapper/308:55:233
2784199201,10cyclictest7312-21chrt08:45:263
2784199200,9cyclictest0-21swapper/311:10:573
2784199200,8cyclictest0-21swapper/307:28:593
2783699202,9cyclictest15430-21kworker/2:109:52:202
2783699201,5cyclictest0-21swapper/212:00:312
27836992014,3cyclictest5529-21sed11:10:142
2783699201,2cyclictest0-21swapper/209:37:342
2783699201,16cyclictest14058-21taskset09:01:242
2783299207,8cyclictest86850irq/129-enp1s0-10:57:451
2783299207,3cyclictest11523-21munin-run07:45:001
2783299205,3cyclictest0-21swapper/107:52:101
2783299205,3cyclictest0-21swapper/107:52:091
2783299201,8cyclictest0-21swapper/111:50:561
27832992017,2cyclictest0-21swapper/110:07:101
27832992016,3cyclictest0-21swapper/107:29:361
2783299201,5cyclictest3302-21/usr/sbin/munin12:20:191
2783299201,18cyclictest0-21swapper/111:38:171
2783299201,18cyclictest0-21swapper/110:42:011
2783299201,18cyclictest0-21swapper/108:06:181
2783299201,17cyclictest0-21swapper/107:32:211
2783299201,17cyclictest0-21swapper/107:21:021
2783299201,14cyclictest0-21swapper/111:24:101
2783299201,14cyclictest0-21swapper/108:55:461
2783299201,14cyclictest0-21swapper/108:55:461
2783299201,11cyclictest3100-21open_files09:50:211
2782499205,6cyclictest11531-21cat11:25:110
2782499205,4cyclictest0-21swapper/010:11:190
2782499202,10cyclictest0-21swapper/011:10:190
2782499201,17cyclictest29127-21munin-run08:25:010
2782499201,11cyclictest10174-21/usr/sbin/munin08:55:150
2782499201,11cyclictest10174-21/usr/sbin/munin08:55:150
2782499201,11cyclictest0-21swapper/012:23:020
2782499201,10cyclictest16552-21diskmemload11:53:220
2782499201,10cyclictest14634-21snmpd10:53:300
59812191,3sleep20-21swapper/209:57:372
2784199191,17cyclictest0-21swapper/307:23:573
2783699191,6cyclictest16826-21taskset10:20:322
27836991914,3cyclictest5796-21cut07:30:162
27836991913,3cyclictest18980-21/usr/sbin/munin09:15:122
27836991911,2cyclictest15430-21kworker/2:110:51:422
2783299195,3cyclictest0-21swapper/107:37:131
2783299192,3cyclictest1503-21ntp_states11:00:231
2783299191,3cyclictest0-21swapper/109:00:411
2783299191,16cyclictest0-21swapper/112:25:571
2783299191,10cyclictest11797-21cpu07:45:121
2782499193,2cyclictest86550irq/126-enp1s0-10:20:130
2782499193,2cyclictest0-21swapper/009:52:150
2782499191,12cyclictest0-21swapper/009:15:280
2782499191,10cyclictest32394-21sshd10:57:440
2782499191,10cyclictest2958-21/usr/sbin/munin07:25:160
2782499191,10cyclictest10843-21users11:20:250
2782499191,10cyclictest0-21swapper/012:34:590
2782499191,10cyclictest0-21swapper/011:58:010
2782499191,10cyclictest0-21swapper/009:55:350
2782499190,7cyclictest0-21swapper/012:09:310
236262190,2sleep20-21swapper/210:38:152
155042191,2sleep214634-21snmpd11:33:382
53312181,2sleep00-21swapper/008:42:190
281152181,3sleep20-21swapper/210:49:152
2783699186,4cyclictest16552-21diskmemload10:25:142
2783699182,3cyclictest0-21swapper/210:00:552
2783699182,3cyclictest0-21swapper/210:00:552
2783699182,10cyclictest15430-21kworker/2:108:03:182
2783699182,10cyclictest15430-21kworker/2:107:35:102
2783699181,9cyclictest7522-21sed12:30:012
2783699181,9cyclictest0-21swapper/208:25:132
2783699181,8cyclictest1152-21awk08:35:012
2783699181,7cyclictest27138-21chrt08:20:092
2783699181,3cyclictest0-21swapper/208:55:252
2783699181,3cyclictest0-21swapper/208:55:252
27836991813,3cyclictest22751-21/usr/sbin/munin08:10:182
27836991812,3cyclictest31096-21cut10:55:162
27836991812,3cyclictest22249-21ntpq09:20:232
2783699181,10cyclictest0-21swapper/212:11:402
2783699181,10cyclictest0-21swapper/212:11:402
2783299185,2cyclictest23-21ksoftirqd/107:15:181
2783299185,11cyclictest6009-21kworker/u8:010:10:151
2783299181,8cyclictest18055-21/usr/sbin/munin11:40:141
2782499187,3cyclictest27023-21irqstats10:45:210
2782499186,4cyclictest0-21swapper/012:14:160
2782499186,4cyclictest0-21swapper/012:14:160
2782499186,3cyclictest0-21swapper/009:48:100
2782499185,5cyclictest0-21swapper/009:35:140
2782499185,3cyclictest0-21swapper/012:17:110
2782499185,3cyclictest0-21swapper/010:05:030
2782499181,9cyclictest16549-21latency_hist09:10:010
2782499181,9cyclictest0-21swapper/011:09:430
2782499181,9cyclictest0-21swapper/011:09:430
2782499181,4cyclictest31551-21/usr/sbin/munin08:30:150
2782499181,4cyclictest11408-21sshd07:43:440
27824991813,3cyclictest9310-21seq07:40:000
27824991813,3cyclictest17551-21tr10:25:140
27824991811,4cyclictest848-21sssd_nss07:50:230
27824991811,4cyclictest848-21sssd_nss07:50:230
27824991811,4cyclictest20254-21/usr/sbin/munin11:45:150
2782499181,11cyclictest0-21swapper/010:17:480
2782499181,11cyclictest0-21swapper/009:40:160
2782499181,10cyclictest0-21swapper/011:33:540
249782181,3sleep00-21swapper/009:25:300
221932180,2sleep20-21swapper/211:49:522
51922171,2sleep20-21swapper/208:40:332
2783699176,3cyclictest86550irq/126-enp1s0-09:25:162
2783699171,7cyclictest5781-21/usr/sbin/munin08:45:152
2783699171,7cyclictest0-21swapper/210:15:122
27836991712,3cyclictest7702-21/usr/sbin/munin12:30:222
27836991712,3cyclictest24349-21sh10:40:142
27836991712,3cyclictest19394-21threads11:40:262
27836991711,3cyclictest8846-21expr10:05:152
27836991711,3cyclictest830-21grep11:00:162
27836991711,3cyclictest10080-21grep07:40:172
2783699171,10cyclictest15430-21kworker/2:107:58:202
2783699171,10cyclictest0-21swapper/212:17:362
2783699171,10cyclictest0-21swapper/209:10:382
2782499176,4cyclictest0-21swapper/010:37:120
2782499176,2cyclictest23060-21kworker/u8:209:31:350
2782499172,12cyclictest12892-21tr07:45:230
2782499171,6cyclictest0-21swapper/007:30:260
27824991712,3cyclictest8253-21expr08:50:120
27824991712,3cyclictest29943-21tail08:25:160
27824991711,3cyclictest23116-21awk08:10:160
2782499171,10cyclictest0-21swapper/010:33:570
2782499171,10cyclictest0-21swapper/009:00:160
246342171,2sleep20-21swapper/211:55:062
2783699164,3cyclictest15924-21perf07:55:012
2783699164,3cyclictest15924-21perf07:55:012
2783699163,3cyclictest20553-21/usr/sbin/munin08:05:212
2783699161,6cyclictest0-21swapper/207:23:402
27836991612,3cyclictest0-21swapper/208:36:512
27836991611,3cyclictest29061-21ntp_kernel_err07:10:202
27836991611,3cyclictest26212-21gltestperf09:30:202
2783699161,12cyclictest29198-21cpuspeed_turbos12:05:102
2782499163,4cyclictest20685-21grep08:05:150
2782499161,5cyclictest0-21swapper/008:45:160
27824991611,3cyclictest836-21runrttasks08:01:170
164042161,2sleep20-21swapper/209:08:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional