You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 09:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Aug 27, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
259422215182,21sleep30-21swapper/319:07:143
261512204166,25sleep20-21swapper/219:09:552
261072204167,24sleep00-21swapper/019:09:210
261502203165,25sleep10-21swapper/119:09:541
225521670,3sleep22626499cyclictest19:25:202
614921450,4sleep32626999cyclictest22:00:283
199922820,3sleep30-21swapper/320:05:233
257972610,1sleep325796-21proc_pri22:45:243
276532600,4sleep230-21rcuc/221:35:302
149972560,2sleep115001-21fschecks_count19:55:171
277542490,1sleep20-21swapper/222:50:202
2625899461,34cyclictest16944-21cstates20:00:131
2625899451,35cyclictest2961-21cat20:40:191
2626999434,33cyclictest2928-40dnf23:08:393
2626999431,28cyclictest31667-21latency_hist00:15:023
2625899430,9cyclictest22582-21/usr/sbin/munin21:25:191
2626999411,26cyclictest3334-21latency_hist19:30:023
2626999401,30cyclictest26200-21/usr/sbin/munin21:35:293
2625899407,4cyclictest2534-21if_err_enp2s000:20:181
2625899401,8cyclictest0-21swapper/122:26:071
2625899401,30cyclictest30941-21diskstats21:45:141
239032400,3sleep2311ktimersoftd/223:55:192
2626999398,21cyclictest16755-21/usr/sbin/munin20:00:133
2626999391,28cyclictest2541-21interrupts00:20:183
2625899391,7cyclictest8834-21cut00:35:161
2626999384,25cyclictest0-21swapper/319:35:383
2626999381,33cyclictest12601-21fschecks_time23:30:163
2626999381,28cyclictest7683-21/usr/sbin/munin23:20:233
2626999381,28cyclictest6132-21/usr/sbin/munin00:30:163
2626999381,24cyclictest12767-21latency_hist21:05:013
2626999381,22cyclictest0-21swapper/300:35:213
2625899383,6cyclictest2928-40dnf23:08:391
2625899382,9cyclictest19737-21ntp_states23:45:271
2625899381,9cyclictest0-21swapper/123:38:111
2626999371,27cyclictest836-21runrttasks23:55:333
2626999371,27cyclictest18673-21/usr/sbin/munin23:45:313
2626999371,24cyclictest30592-21/usr/sbin/munin21:45:113
2626999371,24cyclictest22439-21/usr/sbin/munin22:40:283
2625899371,9cyclictest0-21swapper/121:20:131
2625899371,5cyclictest29964-21/usr/sbin/munin20:30:141
2625899371,25cyclictest0-21swapper/123:24:001
2625899370,27cyclictest11497-21kworker/u8:200:05:361
2626999362,25cyclictest13639-21/usr/sbin/munin22:20:183
2626999361,28cyclictest6043-21sshd20:48:373
2625899366,3cyclictest24030-21cpu21:30:121
2625899366,3cyclictest24030-21cpu21:30:121
2625899361,32cyclictest10830-21cpu21:00:141
2625899361,23cyclictest12148-21basename22:15:181
26258993612,21cyclictest25949-21hddtemp_smartct00:00:171
2625899360,28cyclictest0-21swapper/122:11:161
2626999353,20cyclictest780-21rngd19:44:323
2626999351,26cyclictest2490-21cat21:55:023
2626999351,26cyclictest0-21swapper/322:11:523
2626999351,26cyclictest0-21swapper/321:32:503
2626999351,26cyclictest0-21swapper/321:32:503
2626999351,25cyclictest836-21runrttasks21:13:003
2626999351,21cyclictest12522-21tr19:50:133
26269993510,22cyclictest18665-21smart_sda21:15:273
2626999350,19cyclictest29339-21/usr/sbin/munin19:15:213
2626999350,19cyclictest29339-21/usr/sbin/munin19:15:213
2625899356,3cyclictest32709-21irqstats00:15:191
2625899351,9cyclictest0-21swapper/121:10:151
2625899351,8cyclictest32199-21kworker/u8:022:06:351
2625899351,7cyclictest5564-21chrt00:25:501
2625899351,3cyclictest22706-21/usr/sbin/munin20:10:301
2625899351,32cyclictest19382-21awk21:20:011
2625899351,31cyclictest28906-21grep22:55:021
2625899351,23cyclictest0-21swapper/119:19:211
2625899351,23cyclictest0-21swapper/119:19:211
2626999344,19cyclictest86550irq/126-enp1s0-21:55:193
2626999342,30cyclictest86650irq/127-enp1s0-20:34:503
2626999341,26cyclictest14268-21/usr/sbin/munin23:35:183
2625899347,18cyclictest0-21swapper/122:20:161
2625899341,7cyclictest4397-21sshd21:57:341
2625899341,3cyclictest6927-21cat22:05:021
2625899341,20cyclictest0-21swapper/123:40:471
2625899341,20cyclictest0-21swapper/123:40:471
2625899341,11cyclictest0-21swapper/123:14:201
2626999334,20cyclictest780-21rngd21:06:243
26269993321,4cyclictest780-21rngd19:58:193
2626999331,24cyclictest0-21swapper/300:15:273
2626999331,21cyclictest11498-21egrep22:15:143
2626999331,19cyclictest0-21swapper/321:20:273
2625899337,3cyclictest30996-21latency_hist23:00:011
2625899333,6cyclictest29028-21smart_sda20:25:251
2625899332,4cyclictest21495-21smart_sda22:35:261
2625899332,4cyclictest21495-21smart_sda22:35:261
2625899331,7cyclictest0-21swapper/122:49:441
2625899331,7cyclictest0-21swapper/122:42:001
2625899331,6cyclictest8364-21fschecks_count19:40:161
2625899331,5cyclictest1352-21/usr/sbin/munin19:25:171
26258993312,18cyclictest9876-21/usr/sbin/munin23:25:311
26258993312,18cyclictest32169-21/usr/sbin/munin20:35:241
26258993312,18cyclictest24107-21ntpq23:55:221
2625899330,9cyclictest0-21swapper/119:35:521
2626999323,24cyclictest22236-21ntp_states20:10:253
2626999322,25cyclictest86550irq/126-enp1s0-23:42:523
2626999322,25cyclictest86550irq/126-enp1s0-23:42:523
26269993219,4cyclictest780-21rngd00:25:043
2626999321,3cyclictest10675-21interrupts23:25:223
2626999321,22cyclictest86850irq/129-enp1s0-22:50:223
2626999321,22cyclictest581-21systemd-journal23:50:093
2626999321,20cyclictest12050irq/123-ahci[0022:30:003
2626999321,18cyclictest22891-21open_inodes21:25:243
2626999321,16cyclictest0-21swapper/319:32:243
2626999321,15cyclictest0-21swapper/319:11:193
2626999321,11cyclictest0-21swapper/320:28:333
2626999320,19cyclictest0-21swapper/322:05:243
2625899326,3cyclictest0-21swapper/121:38:221
2625899325,3cyclictest23480-21cpuspeed_turbos20:15:161
2625899325,17cyclictest780-21rngd20:50:081
2625899322,5cyclictest20726-40dnf20:07:311
2625899322,4cyclictest4635-21ntp_states19:30:231
2625899321,9cyclictest28414-21munin-plugin-st19:15:011
2625899321,9cyclictest0-21swapper/123:01:211
2625899321,6cyclictest12999-21/usr/sbin/munin21:05:251
26258993211,18cyclictest836-21runrttasks23:19:381
2625899321,10cyclictest0-21swapper/120:47:281
2625899320,10cyclictest0-21swapper/123:53:041
2626999313,19cyclictest0-21swapper/320:51:443
26269993113,15cyclictest27487-21/usr/sbin/munin00:05:153
2626999311,27cyclictest29380-21/usr/sbin/munin22:55:153
2626999311,27cyclictest14634-21snmpd20:24:363
2626999311,27cyclictest0-21swapper/319:23:213
2626999311,22cyclictest0-21swapper/321:44:003
2626999311,21cyclictest780-21rngd22:36:503
2626999311,21cyclictest780-21rngd22:36:503
2625899318,3cyclictest31901-21fschecks_count19:20:161
2625899316,3cyclictest27318-21sshd20:22:201
2625899312,5cyclictest13422-21ntp_states19:50:221
2625899311,5cyclictest29702-21/usr/sbin/munin00:10:261
2625299316,12cyclictest14043-21munin-run23:35:010
2626999302,3cyclictest86550irq/126-enp1s0-20:40:173
2626999301,3cyclictest86850irq/129-enp1s0-23:10:203
2626999301,26cyclictest18388-21diskstats22:30:183
2626999301,21cyclictest0-21swapper/323:01:353
26269993012,10cyclictest0-21swapper/323:15:223
2626999301,18cyclictest8926-21df20:55:143
2626999301,14cyclictest780-21rngd19:45:593
2626999300,1cyclictest0-21swapper/300:05:023
2625899301,9cyclictest0-21swapper/121:41:461
2625899301,9cyclictest0-21swapper/100:32:231
26258993010,17cyclictest1066-21head21:50:211
2625899300,9cyclictest0-21swapper/119:45:531
50192290,4sleep02625299cyclictest19:30:270
2626999292,24cyclictest377-21hddtemp_smartct20:35:213
2625899298,2cyclictest8591-21/usr/sbin/munin20:55:221
2625899296,4cyclictest18374-21df22:30:171
2625899290,15cyclictest0-21swapper/123:31:561
2625299292,12cyclictest28163-21idleruntime-cro21:40:010
2626999281,24cyclictest24437-21ntp_states20:15:253
2625299281,12cyclictest29489-21basename00:10:010
2625299271,14cyclictest21575-21munin-run21:25:010
2625299271,12cyclictest22218-21awk22:40:010
2625299271,12cyclictest22218-21awk22:40:010
2625299264,11cyclictest7-21ksoftirqd/021:15:010
2625299261,12cyclictest22841-21munin-run23:55:010
2626499247,9cyclictest0-21swapper/221:10:512
2625299241,12cyclictest27545-21munin-run20:25:010
2626499231,6cyclictest0-21swapper/221:32:362
2626499231,6cyclictest0-21swapper/221:32:362
2625299238,8cyclictest0-21swapper/023:15:220
2625299236,14cyclictest5913-21awk00:30:010
2626499211,5cyclictest0-21swapper/221:15:452
2625299216,3cyclictest86750irq/128-enp1s0-23:20:320
2625299215,5cyclictest11497-21kworker/u8:200:30:200
2625299212,16cyclictest19583-21grep20:05:180
2625299211,7cyclictest19649-21/usr/sbin/munin23:45:250
2625299211,11cyclictest15962-21sshd23:36:370
2626499206,4cyclictest848-21sssd_nss20:30:012
2626499201,10cyclictest0-21swapper/222:25:222
2625299209,6cyclictest0-21swapper/021:29:560
2625299209,3cyclictest26835-21/usr/sbin/munin22:50:180
2625299209,3cyclictest14634-21snmpd20:19:360
2625299205,9cyclictest0-21swapper/019:35:240
2625299202,10cyclictest10791-21/usr/sbin/munin21:00:220
2625299201,8cyclictest6386-21grep22:00:290
2625299201,7cyclictest0-21swapper/000:18:020
2625299201,11cyclictest8328-21/usr/sbin/munin00:35:180
2625299201,10cyclictest5709-21grep20:45:300
2625299201,10cyclictest3292-21grep00:20:270
2626499196,4cyclictest6189-21sort23:15:202
2626499191,9cyclictest0-21swapper/219:20:252
2626499191,6cyclictest257962proc_pri22:45:242
2626499191,5cyclictest0-21swapper/221:43:412
26264991913,5cyclictest11767-21kworker/u8:020:32:552
26264991913,3cyclictest17419-21missed_timers23:40:232
26264991913,3cyclictest17419-21missed_timers23:40:232
2626499191,15cyclictest18806-21grep23:45:182
2626499191,15cyclictest16248-21perf23:40:002
2625299194,5cyclictest0-21swapper/022:45:180
2625299191,9cyclictest24573-21grep23:55:270
2625299191,6cyclictest0-21swapper/022:34:120
2625299191,5cyclictest0-21swapper/022:15:070
26252991912,4cyclictest14965-21sh21:10:010
2625299191,10cyclictest30592-21/usr/sbin/munin21:45:270
2625299191,10cyclictest14634-21snmpd19:14:230
65102180,4sleep230-21rcuc/220:50:142
2626499185,5cyclictest7967-21/usr/sbin/munin19:40:192
2626499181,7cyclictest0-21swapper/200:19:072
2626499181,6cyclictest0-21swapper/222:35:312
2626499181,6cyclictest0-21swapper/222:35:312
2626499181,5cyclictest0-21swapper/223:51:272
2626499181,3cyclictest13422-21ntp_states19:50:222
26264991813,3cyclictest8975-21tail00:35:172
2626499180,12cyclictest32-21ksoftirqd/222:07:522
2625299188,3cyclictest86750irq/128-enp1s0-21:50:170
2625299187,4cyclictest17789-21dump-pmu-power22:29:560
2625299187,1cyclictest0-21swapper/020:26:370
2625299185,3cyclictest16908-21kworker/0:021:15:290
2625299183,9cyclictest0-21swapper/023:05:420
2625299181,9cyclictest14634-21snmpd23:40:110
2625299181,9cyclictest14634-21snmpd23:40:110
2625299181,7cyclictest7176-21tr22:05:140
2625299181,6cyclictest0-21swapper/000:12:300
2625299181,5cyclictest0-21swapper/022:44:380
26252991812,3cyclictest10617-21sort22:10:250
2625299181,14cyclictest13874-21tail19:50:270
154582180,2sleep00-21swapper/022:23:140
120932180,2sleep00-21swapper/019:49:430
306722171,2sleep00-21swapper/022:55:570
27092171,2sleep20-21swapper/223:05:512
2626499177,2cyclictest86550irq/126-enp1s0-20:55:572
2626499176,3cyclictest15097-21sed22:20:292
2626499175,3cyclictest32-21ksoftirqd/200:20:182
2626499174,4cyclictest1558-21sshd20:37:372
2626499171,7cyclictest29030-21/usr/sbin/munin22:55:162
2626499171,6cyclictest20000-21latency_hist22:35:012
26264991713,2cyclictest10309-21ntp_states22:10:222
26264991712,3cyclictest5420-21rt-features20:45:282
26264991712,3cyclictest32-21ksoftirqd/220:45:012
26264991712,3cyclictest29811-21cpuspeed_turbos00:10:132
26264991712,3cyclictest26238-21hddtemp_smartct20:20:172
2626499171,15cyclictest0-21swapper/223:33:592
26264991711,3cyclictest23205-21if_enp2s022:40:232
2626499171,11cyclictest0-21swapper/200:33:392
2626499171,10cyclictest0-21swapper/221:22:452
26264991710,4cyclictest581-21systemd-journal21:10:012
2625299177,3cyclictest21171-21/usr/sbin/munin20:10:160
2625299172,12cyclictest27283-21cat00:05:010
2625299171,9cyclictest0-21swapper/021:40:300
2625299171,8cyclictest0-21swapper/023:27:330
2625299171,6cyclictest3188-21perl21:55:180
26252991712,3cyclictest4081-21if_err_enp2s023:10:230
26252991712,3cyclictest3335-21idleruntime-cro19:30:010
26252991711,4cyclictest769-21sssd_be23:00:270
26252991711,3cyclictest15415-21sort19:55:200
26252991710,4cyclictest32612-21grep20:35:210
2626499167,7cyclictest11968-21fschecks_time22:15:172
2626499165,3cyclictest86550irq/126-enp1s0-23:10:242
2626499163,3cyclictest3259-21gltestperf21:55:182
2626499161,9cyclictest0-21swapper/219:55:182
2626499161,7cyclictest0-21swapper/221:45:472
2626499161,6cyclictest2400-21latency_hist21:55:022
2626499161,6cyclictest11776-21sshd19:45:372
2626499161,6cyclictest0-21swapper/219:10:252
2626499161,5cyclictest0-21swapper/222:04:302
2626499161,3cyclictest0-21swapper/200:28:072
26264991612,3cyclictest0-21swapper/219:15:462
26264991612,3cyclictest0-21swapper/219:15:462
26264991611,3cyclictest9050-21timerandwakeup23:20:312
26264991611,3cyclictest6841-21ntpq19:35:232
26264991611,3cyclictest4835-21swap19:30:262
26264991611,3cyclictest31247-21cpu23:00:132
26264991611,3cyclictest28809-21cat00:05:262
26264991611,3cyclictest23366-21/usr/sbin/munin20:15:202
26264991611,3cyclictest22476-21smartctl21:25:182
26264991611,3cyclictest11820-21ntp_kernel_err21:00:222
26264991610,3cyclictest21986-21uname20:10:222
26264991610,3cyclictest19048-21cpuspeed_turbos20:05:152
26264991610,3cyclictest17113-21cat20:00:152
26264991610,3cyclictest10036-21cpuspeed_turbos23:25:172
2626499160,6cyclictest0-21swapper/200:00:122
2625299161,9cyclictest0-21swapper/020:40:220
2625299161,6cyclictest9707-21sshd19:42:170
2625299161,6cyclictest0-21swapper/020:55:200
2625299161,5cyclictest0-21swapper/021:30:100
2625299161,5cyclictest0-21swapper/021:30:100
2625299161,4cyclictest29339-21/usr/sbin/munin19:15:150
2625299161,4cyclictest29339-21/usr/sbin/munin19:15:150
26252991612,3cyclictest0-21swapper/020:01:220
26252991611,3cyclictest7964-21grep20:50:270
2625299160,15cyclictest0-21swapper/020:34:030
2625299151,11cyclictest32199-21tail19:20:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional