You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-31 - 15:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Dec 31, 2025 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7365997847,22cyclictest28366irq/58-eth009:55:153
733399704,54cyclictest14807-21ssh12:10:160
734499645,46cyclictest13454-21ssh09:40:141
733399591,5cyclictest131rcu_preempt12:20:110
7355995441,10cyclictest0-21swapper/207:40:112
734499514,8cyclictest131rcu_preempt11:25:131
735599509,25cyclictest27-21ksoftirqd/212:25:152
735599509,25cyclictest27-21ksoftirqd/212:25:142
7365994930,10cyclictest28366irq/58-eth009:30:133
736599486,27cyclictest28366irq/58-eth010:10:143
733399474,11cyclictest131rcu_preempt12:05:110
736599463,23cyclictest28366irq/58-eth009:25:153
733399443,32cyclictest4679-21ssh09:35:140
736599414,24cyclictest28366irq/58-eth009:35:143
7355994017,16cyclictest20466-1kworker/u9:212:15:122
7344994035,3cyclictest22281-21ssh09:45:121
733399401,32cyclictest28412-21ssh09:30:140
734499390,5cyclictest131rcu_preempt10:00:131
736599381,22cyclictest28366irq/58-eth010:15:133
735599382,19cyclictest13711-21ssh09:57:402
734499381,23cyclictest26748-21ssh10:25:151
734499373,24cyclictest21-21ksoftirqd/112:25:161
734499373,24cyclictest21-21ksoftirqd/112:25:151
733399371,29cyclictest24551-21ssh10:05:130
736599363,16cyclictest28366irq/58-eth011:30:153
735599362,28cyclictest3266-21sh09:33:412
7355993610,17cyclictest27-21ksoftirqd/211:35:142
7355993610,17cyclictest27-21ksoftirqd/211:35:132
734499363,21cyclictest21-21ksoftirqd/111:05:121
733399365,21cyclictest21137-21ssh11:55:160
734499355,22cyclictest131rcu_preempt11:45:131
7355993419,9cyclictest27262-21ssh09:45:402
734499346,19cyclictest20224-21ssh10:40:141
734499345,21cyclictest9939-21ssh11:30:161
734499341,7cyclictest131rcu_preempt12:35:161
7344993412,13cyclictest21-21ksoftirqd/111:55:151
733399341,25cyclictest16154-21ssh11:15:130
734499331,24cyclictest10957-21ssh09:20:121
733399331,29cyclictest17280-21ssh12:30:130
736599323,16cyclictest28366irq/58-eth009:50:143
735599325,17cyclictest22346irq/40-dwc2_hso08:40:132
7344993221,6cyclictest21-21ksoftirqd/111:20:141
733399322,22cyclictest9484-21ssh10:15:130
735599312,14cyclictest15933-21diskmemload10:00:122
733399312,22cyclictest23464-21ssh12:15:130
736599302,16cyclictest28366irq/58-eth010:55:173
735599305,13cyclictest22346irq/40-dwc2_hso08:30:152
735599304,14cyclictest22346irq/40-dwc2_hso07:35:122
735599302,12cyclictest32748-21ssh11:24:292
735599301,22cyclictest11551-21rm10:35:132
734499304,23cyclictest25959-21ssh09:10:141
7344993023,3cyclictest21-21ksoftirqd/110:55:151
733399302,22cyclictest472-21ssh11:41:400
733399302,22cyclictest2281-21ssh09:15:140
735599296,20cyclictest27-21ksoftirqd/210:15:132
735599293,11cyclictest0-21swapper/209:41:402
735599292,21cyclictest0-21swapper/207:15:122
7355992914,11cyclictest25543-21diskmemload11:05:142
7344992917,7cyclictest21-21ksoftirqd/107:30:131
733399292,19cyclictest29027-21ssh10:45:130
736599282,15cyclictest28366irq/58-eth011:35:143
736599282,15cyclictest28366irq/58-eth011:35:133
735599282,21cyclictest23779-21ssh12:32:412
735599282,20cyclictest0-21swapper/207:50:152
7344992822,4cyclictest0-21swapper/112:30:121
7333992817,5cyclictest25543-21diskmemload11:45:150
733399281,22cyclictest73-21kswapd010:25:130
736599274,13cyclictest28366irq/58-eth010:35:133
735599275,12cyclictest22346irq/40-dwc2_hso08:55:152
735599272,14cyclictest22346irq/40-dwc2_hso09:00:152
7355992717,7cyclictest25404-21sshd09:26:392
7355992717,6cyclictest261rcuc/211:25:142
734499272,22cyclictest5145-21ssh10:50:141
7344992719,3cyclictest0-21swapper/107:40:111
733399274,17cyclictest31209-21ssh11:05:130
733399272,19cyclictest27512-21ssh10:43:400
736599262,17cyclictest0-21swapper/309:45:403
7365992612,10cyclictest28366irq/58-eth011:40:143
7365992611,10cyclictest28366irq/58-eth012:10:163
7365992610,10cyclictest28366irq/58-eth010:04:143
735599262,19cyclictest0-21swapper/208:10:122
735599262,15cyclictest7394-21ssh11:10:162
735599261,16cyclictest22346irq/40-dwc2_hso08:25:142
734499262,13cyclictest839-21munin-node08:40:131
7344992613,7cyclictest0-21swapper/111:54:401
733399263,15cyclictest0-21swapper/010:55:160
735599252,19cyclictest0-21swapper/210:09:152
735599252,18cyclictest11965-21ssh09:38:392
734499252,19cyclictest30969-21ssh09:50:131
733399253,16cyclictest633-21Async12:11:530
733399252,17cyclictest10986-21ssh09:20:150
736599243,13cyclictest0-21swapper/310:43:403
735599248,11cyclictest25543-21diskmemload12:07:402
735599243,16cyclictest0-21swapper/209:53:402
735599243,14cyclictest22346irq/40-dwc2_hso11:00:142
735599242,12cyclictest13891-21ssh10:55:162
735599242,10cyclictest0-21swapper/209:05:122
7355992412,9cyclictest0-21swapper/208:15:152
735599241,17cyclictest29026-21sh10:45:122
7344992417,4cyclictest0-21swapper/108:15:131
734499241,17cyclictest32222-21ssh12:20:131
734499241,17cyclictest29850-21apt-get12:00:121
733399242,3cyclictest24887-21apt-get11:20:130
733399241,18cyclictest29862-21ssh12:00:130
736599237,9cyclictest28366irq/58-eth012:00:113
7365992313,6cyclictest28366irq/58-eth012:35:173
735599239,5cyclictest15073-21unixbench_multi09:20:312
735599236,15cyclictest29029-21apt-config08:40:012
735599234,16cyclictest22346irq/40-dwc2_hso12:10:142
735599233,13cyclictest261rcuc/212:20:122
735599232,6cyclictest0-21swapper/211:43:412
735599232,6cyclictest0-21swapper/210:10:152
735599231,14cyclictest22346irq/40-dwc2_hso07:25:132
734499236,13cyclictest0-21swapper/110:20:121
7344992311,5cyclictest21-21ksoftirqd/108:30:131
733399231,18cyclictest16037-21ssh11:50:270
733399231,18cyclictest12283-21apt-get07:15:110
736599223,12cyclictest0-21swapper/312:30:153
7365992215,4cyclictest28366irq/58-eth009:40:133
735599228,11cyclictest31246-21cron07:35:012
735599221,18cyclictest28590-21apt-config10:45:022
7355992211,5cyclictest25188-21ssh10:23:352
735599221,14cyclictest0-21swapper/208:05:132
735599221,12cyclictest12908-21irqrtprio11:50:192
735599221,11cyclictest7127-21kthreadcore10:50:232
734499226,8cyclictest21-21ksoftirqd/107:25:141
733399224,11cyclictest22280-21sh09:45:130
733399221,7cyclictest4874-21munin-run08:15:000
7333992211,9cyclictest6113-21sshd11:25:410
733399221,17cyclictest24553-21cut08:35:150
7333992210,3cyclictest131rcu_preempt12:25:300
7333992210,3cyclictest131rcu_preempt12:25:290
736599214,9cyclictest28366irq/58-eth009:10:163
736599214,11cyclictest0-21swapper/311:55:153
735599212,12cyclictest31980-21kthreadcore12:00:192
735599212,10cyclictest29744-21kthreadcore10:25:232
735599211,8cyclictest906766ssh09:18:032
735599211,17cyclictest73-21kswapd011:30:152
735599211,13cyclictest22346irq/40-dwc2_hso08:45:152
734499218,4cyclictest131rcu_preempt11:35:331
734499218,4cyclictest131rcu_preempt11:35:321
734499214,15cyclictest0-21swapper/111:40:141
7344992115,4cyclictest0-21swapper/109:35:151
7344992111,7cyclictest18954-21cat12:10:311
734499211,14cyclictest4348-21ls10:10:251
734499210,18cyclictest10843-21latency11:10:251
733399213,13cyclictest18491-21ssh11:35:140
733399213,13cyclictest18491-21ssh11:35:140
733399212,16cyclictest800-21ssh10:10:150
7333992114,5cyclictest0-21swapper/007:20:190
733399211,16cyclictest10045-21cut08:20:150
736599209,4cyclictest0-21swapper/311:25:023
736599208,9cyclictest0-21swapper/312:16:233
736599205,10cyclictest0-21swapper/312:07:393
736599203,12cyclictest0-21swapper/311:06:413
736599202,13cyclictest0-21swapper/310:50:133
7365992011,6cyclictest0-21swapper/311:10:273
736599201,12cyclictest0-21swapper/308:05:233
735599209,8cyclictest16125-21/usr/sbin/munin11:15:182
735599206,12cyclictest14468-21cron08:25:012
735599203,8cyclictest12575-21head07:45:262
735599201,9cyclictest6021-21cut10:30:242
735599201,5cyclictest0-21swapper/207:10:122
735599201,16cyclictest29481-21munin-run12:00:022
735599201,16cyclictest26297-21kthreadcore08:00:222
735599201,15cyclictest29471-21awk12:35:262
735599201,15cyclictest18507-21fschecks_time07:55:142
735599201,10cyclictest3521-21/usr/sbin/munin11:45:152
735599201,10cyclictest26938-21kthreadcore09:10:232
734499209,8cyclictest26787-21cut12:15:231
734499209,8cyclictest13612-21df_inode07:50:141
734499202,5cyclictest19642-21memory11:15:231
734499202,12cyclictest839-21munin-node10:30:151
734499202,12cyclictest22567-21ssh11:00:141
734499201,5cyclictest11018-21/usr/sbin/munin09:55:281
7344992012,5cyclictest6138-21/usr/sbin/munin12:05:141
7344992010,7cyclictest13574-21unixbench_singl10:15:301
733399209,3cyclictest25543-21diskmemload09:53:040
733399202,7cyclictest22894-21cron08:00:020
733399202,12cyclictest25922-21ssh12:35:160
733399201,4cyclictest3035-21fschecks_time10:30:180
7333992013,4cyclictest23477-21sendmail_mailqu09:25:290
7333992013,4cyclictest22528-21/usr/sbin/munin11:00:290
733399201,16cyclictest13536-21ssh11:30:280
736599199,7cyclictest0-21swapper/309:19:503
736599193,11cyclictest0-21swapper/310:08:403
7365991913,4cyclictest28366irq/58-eth011:00:133
7365991911,6cyclictest0-21swapper/311:48:583
7365991911,5cyclictest0-21swapper/310:45:313
735599198,4cyclictest6908-21irqcore08:50:172
735599198,4cyclictest6908-21irqcore08:50:162
735599190,14cyclictest17162-21ls07:20:142
734499195,9cyclictest21-21ksoftirqd/108:20:121
734499192,9cyclictest19917-21fschecks_time09:25:181
734499192,15cyclictest13442-21kthreadcore08:55:231
734499192,10cyclictest0-21swapper/110:05:121
734499191,15cyclictest2254-21apt-get09:15:141
734499191,13cyclictest31305-21kthreadcore10:45:201
7344991910,5cyclictest27694-21kworker/1:008:05:141
733399199,7cyclictest25002-21tail08:00:200
733399198,8cyclictest1-21systemd08:43:590
733399195,10cyclictest18125-21ssh10:20:140
733399192,14cyclictest7062-21cut10:50:210
733399192,12cyclictest18446-21ssh09:40:390
7333991912,5cyclictest0-21swapper/007:40:170
7333991912,4cyclictest0-21swapper/007:50:170
733399191,11cyclictest5085-21find08:15:130
733399191,10cyclictest6674-21fschecks_time08:50:150
733399191,10cyclictest6674-21fschecks_time08:50:140
736599189,6cyclictest0-21swapper/311:29:313
736599187,6cyclictest0-21swapper/310:30:203
736599183,10cyclictest0-21swapper/310:24:413
736599182,10cyclictest0-21swapper/308:35:143
736599182,10cyclictest0-21swapper/308:25:053
736599182,10cyclictest0-21swapper/308:00:143
736599182,10cyclictest0-21swapper/307:45:123
736599182,10cyclictest0-21swapper/307:30:163
736599181,13cyclictest0-21swapper/308:30:243
7365991810,6cyclictest0-21swapper/310:27:133
7365991810,6cyclictest0-21swapper/309:20:213
735599181,9cyclictest7316-21cyclictest07:10:012
734499185,10cyclictest20995-21kthreadcore07:55:201
734499183,11cyclictest11336-21sed07:10:281
734499183,10cyclictest0-21swapper/107:35:141
734499181,9cyclictest26296-21kthreadcore08:00:231
734499181,7cyclictest18195-21ntp_kernel_pll_08:25:261
734499181,14cyclictest17142-21ls07:20:131
734499181,13cyclictest28395-21apt-get09:30:151
734499181,11cyclictest8796-21ls07:45:121
733399189,6cyclictest11959-21kthreadcore07:45:190
733399185,11cyclictest28736-21kthreadcore08:05:200
733399184,5cyclictest633-21Async12:11:530
733399184,11cyclictest15974-21cpu10:00:140
7333991812,4cyclictest15766-21idleruntime-cro11:15:010
7333991812,4cyclictest0-21swapper/007:30:180
7333991811,5cyclictest0-21swapper/007:35:220
733399181,12cyclictest19808-21cron10:40:020
736599179,6cyclictest0-21swapper/312:25:453
736599179,6cyclictest0-21swapper/312:25:443
736599179,6cyclictest0-21swapper/312:22:193
736599179,6cyclictest0-21swapper/311:51:463
736599179,6cyclictest0-21swapper/311:16:433
736599172,8cyclictest28366irq/58-eth007:10:143
736599172,10cyclictest0-21swapper/308:45:163
736599171,10cyclictest0-21swapper/307:55:153
734499177,7cyclictest338-21diskstats08:10:141
734499177,6cyclictest4717-21kthreadcore08:45:241
734499172,12cyclictest21071-21fschecks_count09:05:151
734499172,10cyclictest11637-21ssh10:35:141
734499171,12cyclictest0-21swapper/108:50:131
734499171,12cyclictest0-21swapper/108:50:121
734499171,11cyclictest20752-21latency_hist09:05:011
7344991710,5cyclictest29083-21latency_hist08:40:021
733399174,10cyclictest18589-21grep08:25:290
733399174,10cyclictest16705-21irqrtprio09:00:200
733399172,10cyclictest0-21swapper/009:05:120
733399172,10cyclictest0-21swapper/008:45:150
733399171,9cyclictest22873-21ls08:30:230
734499167,7cyclictest12615-21irqcore07:15:171
733399166,8cyclictest21979-21df07:25:150
733399162,11cyclictest8670-21kthreadcore07:10:200
733399161,6cyclictest11626-21irqcore08:55:190
736599157,6cyclictest0-21swapper/307:54:083
736599152,10cyclictest0-21swapper/307:20:143
736599151,6cyclictest0-21swapper/307:15:283
736599151,11cyclictest0-21swapper/309:00:173
734499152,10cyclictest7404-21latency_hist07:10:031
733399151,10cyclictest7416-21latency_hist07:10:020
736599141,9cyclictest0-21swapper/308:25:023
736599141,8cyclictest0-21swapper/307:25:163
736599141,10cyclictest0-21swapper/309:10:023
736599127,3cyclictest28366irq/58-eth008:15:153
736599123,7cyclictest0-21swapper/307:40:213
736599122,7cyclictest0-21swapper/308:10:243
736599121,8cyclictest0-21swapper/309:00:003
736599115,4cyclictest28366irq/58-eth008:40:113
736599111,7cyclictest0-21swapper/308:50:153
736599111,7cyclictest0-21swapper/308:50:143
736599111,7cyclictest0-21swapper/307:35:213
736599101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional