You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 10:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Dec 05, 2025 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1991999736,53cyclictest1740-21ssh22:45:141
1991999693,48cyclictest8624-21ssh21:50:131
1994199524,32cyclictest29566irq/58-eth022:45:153
19929995125,21cyclictest27-21ksoftirqd/222:50:132
1990999504,5cyclictest131rcu_preempt20:50:150
19919994825,10cyclictest131rcu_preempt22:50:131
19919994715,19cyclictest21-21ksoftirqd/120:10:141
1992999452,19cyclictest27-21ksoftirqd/220:45:132
19919994517,18cyclictest21-21ksoftirqd/120:45:131
19941994428,9cyclictest29566irq/58-eth022:20:143
1990999442,34cyclictest17710-21ssh22:35:150
19909994413,4cyclictest131rcu_preempt22:20:130
1990999433,33cyclictest7452-21ssh21:30:180
1990999431,10cyclictest14649-21ssh21:15:140
1992999425,16cyclictest261rcuc/223:45:152
1990999423,28cyclictest15851-21diskmemload21:35:120
1990999412,33cyclictest17694-21ssh23:15:130
1990999411,8cyclictest20821-21awk21:55:300
1991999403,24cyclictest1412-21ssh23:45:141
1990999401,27cyclictest17480-21ssh23:35:140
1994199394,20cyclictest29566irq/58-eth023:45:143
1990999381,10cyclictest10817-21cut20:05:220
1994199373,22cyclictest0-21swapper/321:30:183
1990999371,7cyclictest6688-21munin-run19:30:010
1991999361,24cyclictest26051-21ssh22:40:161
1990999351,9cyclictest31296-21tune2fs19:55:180
1990999351,9cyclictest25560-21latency_hist00:00:020
1990999351,9cyclictest21169-21apt-get19:45:020
1990999351,9cyclictest13886-21latency_hist00:35:010
19909993512,20cyclictest29438-21cron19:20:000
1990999351,10cyclictest11868-21cut00:10:230
1994199346,16cyclictest29566irq/58-eth022:00:133
1994199343,17cyclictest29566irq/58-eth000:05:133
1992999342,26cyclictest20657-21ntp_states00:15:282
1990999346,3cyclictest29555-21cut21:00:240
1990999345,23cyclictest12-21ksoftirqd/000:35:170
1990999342,9cyclictest29660-21kthreadcore20:25:210
1990999341,9cyclictest3460-21kthreadcore22:05:220
1990999341,9cyclictest28865-21grep00:20:320
1990999341,9cyclictest26261-21interrupts23:00:180
1990999341,9cyclictest19761-21kthreadcore22:55:220
1990999341,7cyclictest29764-21unixbench_multi00:00:320
1990999341,7cyclictest25552-21kthreadcore20:55:210
1990999341,10cyclictest13810-21head23:10:330
1994199333,18cyclictest29566irq/58-eth023:35:143
1992999331,2cyclictest131rcu_preempt22:05:142
1990999331,9cyclictest9391-21tr19:30:210
1990999331,9cyclictest4778-21kthreadcore20:00:210
1990999331,9cyclictest14204-21cat22:30:350
1990999331,8cyclictest27636-21kthreadcore23:40:230
1990999331,8cyclictest27636-21kthreadcore23:40:230
1990999331,8cyclictest25522-21diskstats22:00:150
1990999331,8cyclictest16039-21kthreadcore20:45:210
1990999331,8cyclictest1007-21kthreadcore21:25:230
1990999331,8cyclictest10000-21/usr/sbin/munin20:40:190
1990999331,11cyclictest8846-21cut21:50:150
1990999331,10cyclictest29378-21awk23:20:270
1990999330,9cyclictest4900-21kthreadcore22:45:230
1990999330,9cyclictest27798-21tr22:40:230
19929993211,4cyclictest27-21ksoftirqd/220:55:152
1990999326,3cyclictest0-21swapper/023:26:430
1990999326,3cyclictest0-21swapper/022:15:190
1990999321,9cyclictest3745-21kthreadcore21:05:240
1990999321,9cyclictest24416-21kthreadcore20:20:220
1990999321,9cyclictest18087-21cut00:15:210
1990999321,9cyclictest10029-21irqrtprio23:50:180
1990999321,8cyclictest1526-21diskstats23:45:170
1990999321,7cyclictest32269-21cut00:25:220
1990999321,6cyclictest24450-21tune2fs21:40:150
1990999321,6cyclictest13761-21/usr/sbin/munin20:10:200
1990999321,6cyclictest10475-21tail21:10:290
1990999321,5cyclictest4025-21ssh21:45:300
1990999321,13cyclictest4764-21ntp_192.168.11400:05:250
1990999320,8cyclictest17149-21apt23:35:010
1990999320,21cyclictest23042-21ssh21:20:150
1992999312,19cyclictest932-21snmpd20:35:152
1991999313,22cyclictest25659-21ssh23:40:131
1991999313,22cyclictest25659-21ssh23:40:131
1990999311,9cyclictest6368-21kthreadcore20:35:210
1990999311,9cyclictest4574-21kthreadcore22:25:240
1990999311,9cyclictest3866-21timerwakeupswit20:30:360
1990999311,9cyclictest29673-21/usr/sbin/munin19:20:270
1990999311,9cyclictest20978-21cut19:10:200
1990999311,9cyclictest20978-21cut19:10:200
1990999311,9cyclictest17157-21irqrtprio19:40:180
1990999311,9cyclictest17157-21irqrtprio19:40:180
1990999311,7cyclictest14650-21/usr/sbin/munin22:50:320
1990999310,7cyclictest17921-21acpi20:15:120
1991999303,17cyclictest1173-21sh00:05:141
1991999302,21cyclictest22702-21users22:15:281
1990999301,9cyclictest27483-21cut19:50:200
1990999301,7cyclictest13567-21/usr/sbin/munin22:10:270
1990999301,6cyclictest15174-21ntp_kernel_pll_19:35:250
1994199293,19cyclictest29566irq/58-eth023:00:123
1992999292,18cyclictest31531-21kthreadcore19:20:192
1992999292,11cyclictest6050-21ssh00:29:292
1992999292,11cyclictest21555-21ntp_states22:55:282
1990999296,5cyclictest4573-21kthreadcore23:05:240
1992999283,20cyclictest0-21swapper/222:21:282
1992999283,17cyclictest26346irq/40-dwc2_hso21:10:122
1992999282,20cyclictest29316-21ssh22:00:282
1992999282,10cyclictest7828-21ssh22:47:282
1992999281,20cyclictest26048-21sh22:40:162
1991999289,11cyclictest21-21ksoftirqd/100:30:151
1991999283,15cyclictest32746-21ssh21:45:181
1994199273,15cyclictest0-21swapper/323:22:283
1994199272,18cyclictest33-21ksoftirqd/323:16:283
1994199272,18cyclictest0-21swapper/319:20:133
1992999274,14cyclictest26346irq/40-dwc2_hso19:55:142
1992999272,15cyclictest27-21ksoftirqd/200:05:132
1991999271,12cyclictest0-21swapper/119:10:171
1991999271,12cyclictest0-21swapper/119:10:171
19941992612,9cyclictest29566irq/58-eth023:05:163
19941992611,8cyclictest29566irq/58-eth021:25:163
1992999263,7cyclictest0-21swapper/200:30:142
1992999261,8cyclictest916366ssh00:10:162
19929992612,8cyclictest0-21swapper/220:15:132
1992999261,18cyclictest28342-21ssh21:40:282
1994199253,14cyclictest0-21swapper/321:52:283
1994199253,13cyclictest0-21swapper/322:50:143
19941992514,8cyclictest29566irq/58-eth022:35:153
19941992513,7cyclictest0-21swapper/323:55:163
1992999254,16cyclictest26346irq/40-dwc2_hso20:00:142
1992999253,6cyclictest0-21swapper/221:49:282
1992999253,20cyclictest0-21swapper/200:20:152
1992999253,17cyclictest0-21swapper/223:28:022
1992999253,17cyclictest0-21swapper/222:15:282
1992999252,18cyclictest14858-21ssh23:51:282
19929992511,11cyclictest17692-21apt-get22:35:142
1991999253,17cyclictest25419-21ssh22:00:141
1991999252,18cyclictest8078-21ssh23:07:591
1991999252,17cyclictest31924-21ssh21:44:281
19919992517,6cyclictest25271-21cron23:40:011
1990999250,6cyclictest19895-21cyclictest19:10:010
1994199247,12cyclictest0-21swapper/322:40:163
1994199244,9cyclictest0-21swapper/300:10:143
1994199244,11cyclictest29566irq/58-eth023:40:133
1994199244,11cyclictest29566irq/58-eth023:40:133
1994199242,14cyclictest29566irq/58-eth021:20:143
1992999246,14cyclictest22141-21cat23:35:312
1992999242,17cyclictest0-21swapper/223:07:282
19929992412,3cyclictest27-21ksoftirqd/221:20:022
1994199233,13cyclictest0-21swapper/322:57:283
1992999239,11cyclictest30118-21kthreadcore21:00:252
1992999238,11cyclictest27461-21ssh23:20:212
1992999234,14cyclictest26346irq/40-dwc2_hso22:10:162
1992999234,13cyclictest26346irq/40-dwc2_hso19:40:132
1992999234,13cyclictest26346irq/40-dwc2_hso19:40:132
1992999233,2cyclictest131rcu_preempt21:55:162
1992999232,6cyclictest0-21swapper/223:44:282
1992999232,6cyclictest0-21swapper/223:44:282
1992999232,16cyclictest0-21swapper/223:12:592
19929992311,3cyclictest27-21ksoftirqd/221:50:212
1991999231,9cyclictest21372-21ntp_states23:15:271
1994199224,12cyclictest0-21swapper/323:10:283
1994199221,10cyclictest29566irq/58-eth021:55:163
1992999228,5cyclictest32266-21sh20:30:012
1992999222,6cyclictest0-21swapper/223:15:282
1992999222,6cyclictest0-21swapper/221:35:292
1992999222,6cyclictest0-21swapper/221:31:282
1992999222,15cyclictest0-21swapper/223:03:282
1992999211,17cyclictest12902-21wc23:30:252
1992999211,10cyclictest31027-21latency_hist19:55:012
1992999211,10cyclictest2459-21latency21:25:252
1991999211,16cyclictest9407-21apt-get23:50:121
1991999211,16cyclictest1292-21gpgv22:25:011
1994199203,11cyclictest0-21swapper/322:05:283
1994199202,11cyclictest0-21swapper/319:45:153
1994199202,10cyclictest29566irq/58-eth023:25:143
19941992011,6cyclictest29566irq/58-eth000:30:153
19941992011,6cyclictest0-21swapper/321:40:283
19941992011,6cyclictest0-21swapper/321:20:023
1994199201,13cyclictest0-21swapper/322:30:183
19941992010,7cyclictest29566irq/58-eth000:00:163
19941992010,6cyclictest29566irq/58-eth022:10:153
19941992010,5cyclictest0-21swapper/300:15:323
1992999208,9cyclictest3460-21kthreadcore22:25:232
1992999206,11cyclictest2519-21kthreadcore21:05:232
1992999201,9cyclictest25032-21cut19:15:162
1992999201,9cyclictest14359-21fschecks_time00:35:222
1992999201,16cyclictest24466-21kthreadcore19:45:232
1992999201,15cyclictest22213-21kthreadcore20:50:242
1992999201,12cyclictest1842-21kthreadcore20:30:262
1992999201,11cyclictest19002-21cut23:55:242
1992999201,10cyclictest29263-21missed_timers00:00:262
1992999201,10cyclictest2398-21tail19:25:172
1992999201,10cyclictest13310-21ntp_states22:30:302
1992999201,10cyclictest10726-21tail19:30:282
1991999203,11cyclictest15725-21ssh23:12:281
1991999200,17cyclictest26129-21ls20:20:241
1994199193,11cyclictest0-21swapper/321:47:283
1994199193,11cyclictest0-21swapper/321:37:453
1994199193,11cyclictest0-21swapper/300:29:293
1994199192,10cyclictest0-21swapper/319:30:133
19941991912,4cyclictest0-21swapper/322:15:173
19941991912,3cyclictest0-21swapper/320:55:153
1994199191,13cyclictest0-21swapper/320:35:173
1992999192,9cyclictest24416-21kthreadcore21:20:212
1992999191,9cyclictest9903-21fschecks_time20:40:182
1992999191,9cyclictest23668-21kthreadcore20:20:212
1992999191,15cyclictest13230-21fschecks_count20:10:172
1992999191,14cyclictest13431-21kthreadcore19:35:202
1992999191,11cyclictest8504-21fschecks_time20:05:172
1992999191,11cyclictest20939-21kthreadcore19:10:202
1992999191,11cyclictest20939-21kthreadcore19:10:202
1991999191,3cyclictest131rcu_preempt20:00:161
19919991912,4cyclictest22997-21/usr/sbin/munin21:20:301
1991999191,15cyclictest4396-21cut22:05:231
1991999191,15cyclictest22765-21tr19:45:211
1991999191,13cyclictest25904-21ssh23:00:131
1991999191,11cyclictest28779-21munin-run21:00:011
19919991910,6cyclictest5035-21awk23:25:271
19919991910,6cyclictest0-21swapper/121:55:301
1994199189,6cyclictest0-21swapper/300:23:003
1994199182,10cyclictest0-21swapper/323:50:143
1994199182,10cyclictest0-21swapper/320:50:173
1994199182,10cyclictest0-21swapper/320:40:163
1994199182,10cyclictest0-21swapper/320:30:203
1994199182,10cyclictest0-21swapper/320:15:143
1994199182,10cyclictest0-21swapper/320:05:153
1994199181,11cyclictest0-21swapper/320:45:143
19941991810,6cyclictest0-21swapper/300:35:333
1991999189,7cyclictest8390-21kthreadcore21:30:251
1991999189,7cyclictest29072-21cat00:20:331
1991999189,6cyclictest31561-21unixbench_singl20:25:301
1991999188,7cyclictest31882-21apt-get00:25:171
1991999186,9cyclictest29441-21cron19:20:001
1991999186,9cyclictest26410-21grep19:50:161
1991999181,9cyclictest728-21kthreadcore20:30:251
1991999181,7cyclictest32187-21kthreadcore21:25:211
19919991811,4cyclictest0-21swapper/120:50:201
19919991810,6cyclictest16284-21sshd22:12:511
1991999180,15cyclictest11651-21cut20:40:221
1994199172,9cyclictest0-21swapper/319:10:143
1994199172,9cyclictest0-21swapper/319:10:143
1994199172,10cyclictest0-21swapper/323:30:163
1994199172,10cyclictest0-21swapper/320:20:153
1994199172,10cyclictest0-21swapper/320:10:153
1994199172,10cyclictest0-21swapper/319:55:153
1994199172,10cyclictest0-21swapper/319:40:143
1994199172,10cyclictest0-21swapper/319:40:143
1994199171,13cyclictest0-21swapper/321:05:313
1994199171,12cyclictest0-21swapper/322:25:193
1991999179,4cyclictest1438-21apt-get21:05:151
1991999178,7cyclictest2526-21cat22:25:211
1991999177,7cyclictest26692-21kthreadcore23:20:201
1991999174,10cyclictest31530-21kthreadcore19:20:191
1991999173,11cyclictest9401-21/usr/sbin/munin23:30:271
1991999173,11cyclictest8730-21sed20:05:191
1991999172,12cyclictest9939-21grep22:30:201
1991999172,12cyclictest22599-21ssh22:35:341
1991999172,11cyclictest17207-21perl00:15:181
1991999172,10cyclictest29021-21apt-get21:00:181
19919991712,3cyclictest0-21swapper/100:39:341
1991999171,14cyclictest19517-21kthreadcore19:40:211
1991999171,14cyclictest19517-21kthreadcore19:40:211
19919991711,4cyclictest0-21swapper/120:15:161
19919991711,4cyclictest0-21swapper/119:30:221
1991999171,13cyclictest9302-21hwlatdetect00:10:181
1991999171,13cyclictest4890-21/usr/sbin/munin20:35:181
1991999171,13cyclictest17023-21ssh21:35:191
19919991711,3cyclictest12392-21kthreadcore19:35:191
1991999171,12cyclictest14250-21idleruntime-cro21:15:011
19919991710,4cyclictest0-21swapper/119:55:251
1991999170,15cyclictest21625-21ps22:55:291
1994199162,9cyclictest0-21swapper/320:25:153
1994199162,10cyclictest0-21swapper/321:10:143
1994199161,12cyclictest0-21swapper/319:35:233
1991999162,11cyclictest17795-21diskstats23:55:181
1991999161,12cyclictest5240-21tr19:25:211
1991999161,12cyclictest25780-21apt-get00:00:161
1994199151,12cyclictest0-21swapper/320:05:023
1991999151,11cyclictest15154-21irqcore21:15:201
1994199149,3cyclictest0-21swapper/321:00:203
1994199142,9cyclictest0-21swapper/319:25:133
1991999141,7cyclictest0-21swapper/119:10:011
1994199131,6cyclictest0-21swapper/319:55:013
1992999131,9cyclictest19967-21latency_hist19:10:012
1994199127,3cyclictest0-21swapper/319:15:173
199419991,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional