You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-03 - 06:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon Aug 03, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2981799601,8cyclictest31871-21ssh22:30:142
2981799592,6cyclictest0-21swapper/200:05:162
29825995848,5cyclictest35966irq/58-eth022:35:143
29825995848,5cyclictest35966irq/58-eth022:35:133
2981799581,31cyclictest0-21swapper/200:35:152
2980999575,44cyclictest13593-21ssh22:20:121
2981799541,5cyclictest0-21swapper/220:30:142
2981799521,36cyclictest29499-21ssh23:40:132
2981799501,7cyclictest27802-21ssh22:10:132
2981799483,10cyclictest0-21swapper/223:30:122
29817994824,16cyclictest0-21swapper/223:00:142
2980999489,28cyclictest844-21munin-node23:35:131
2980999481,40cyclictest16361-21ssh23:15:141
2980999473,6cyclictest131rcu_preempt19:30:141
29809994721,17cyclictest0-21swapper/123:30:131
2980199474,5cyclictest131rcu_preempt23:10:130
2981799442,7cyclictest22545-21ssh00:30:132
2981799432,7cyclictest13512-21ssh00:25:122
2980999422,32cyclictest15462-21diskmemload21:10:131
2980199424,31cyclictest21503-21ssh23:00:130
2981799411,13cyclictest2439-21cut00:00:212
2980199413,6cyclictest131rcu_preempt23:40:120
2981799401,9cyclictest21118-21kthreadcore00:10:212
2981799401,9cyclictest21118-21kthreadcore00:10:212
2980199403,32cyclictest27903-21ssh00:15:150
2981799391,5cyclictest20343-21ssh23:35:132
2980199391,34cyclictest13519-21sshd00:25:130
2981799382,9cyclictest0-21swapper/219:55:152
2981799380,8cyclictest23775-21ssh21:30:302
2980999381,28cyclictest4477-21ssh00:20:111
2981799375,5cyclictest26181-21cut23:20:192
2981799372,27cyclictest4465-21apt-get00:20:132
2981799371,9cyclictest8978-21kthreadcore23:10:212
2981799371,8cyclictest31819-21kthreadcore19:10:212
2980199372,22cyclictest12-21ksoftirqd/023:15:130
2980199371,33cyclictest0-21swapper/019:15:130
29825993623,9cyclictest35966irq/58-eth022:55:153
29825993619,9cyclictest35966irq/58-eth023:45:143
2981799361,9cyclictest25171-21ssh21:50:202
2981799361,9cyclictest25171-21ssh21:50:202
2981799361,9cyclictest24168-21ssh23:55:132
2981799361,8cyclictest24926-21/usr/sbin/munin21:15:162
2981799361,8cyclictest24577-21kthreadcore22:25:222
2981799361,7cyclictest23138-21unixbench_multi20:40:282
2981799361,13cyclictest25365-21grep19:35:262
2981799361,12cyclictest4613-21kthreadcore22:50:202
2981799361,12cyclictest11602-21kthreadcore21:05:202
2981799361,10cyclictest14418-21cut22:55:232
2981799360,9cyclictest31481-21kthreadcore23:05:192
2981799360,9cyclictest18424-21cat22:20:322
2981799360,10cyclictest26770-21/usr/sbin/munin22:45:152
2981799353,8cyclictest17848-21tr23:50:232
2981799351,13cyclictest10699-21ps19:20:262
2981799351,10cyclictest1301-21apt-get20:55:152
2980999356,19cyclictest24169-21ssh23:55:131
2980999352,3cyclictest131rcu_preempt23:05:131
2980199356,26cyclictest6109-21ssh23:45:140
2980199353,18cyclictest12-21ksoftirqd/020:35:140
2980199353,18cyclictest12-21ksoftirqd/020:35:140
29801993522,10cyclictest12-21ksoftirqd/021:40:130
2982599344,24cyclictest35966irq/58-eth023:55:123
29825993417,13cyclictest35966irq/58-eth022:00:133
2981799345,3cyclictest0-21swapper/220:05:022
2981799342,9cyclictest11280-21kthreadcore22:00:212
2981799342,7cyclictest0-21swapper/220:15:122
2981799341,8cyclictest2497-21fschecks_count19:15:152
2981799341,8cyclictest19797-21kthreadcore22:40:202
2981799341,8cyclictest19040-21kthreadcore19:30:222
2981799341,8cyclictest18962-21fschecks_time22:05:172
2981799341,12cyclictest4054-21ps19:50:182
2981799341,10cyclictest4458-21ssh22:15:112
2981799341,10cyclictest14444-21ntpq21:25:292
2981799340,9cyclictest310-21ssh21:55:032
2981799340,9cyclictest28908-21/usr/sbin/munin21:35:132
2981799340,8cyclictest29346-21cut19:40:202
2981799340,10cyclictest17095-21cut21:10:202
2980199342,22cyclictest12-21ksoftirqd/019:55:150
2981799332,10cyclictest21631-21proc_pri20:05:272
2981799331,9cyclictest9998-21unixbench_singl21:00:292
2981799331,9cyclictest9998-21unixbench_singl21:00:292
2981799331,9cyclictest0-21swapper/220:45:152
2981799331,9cyclictest0-21swapper/219:25:282
2981799331,8cyclictest6609-21cat23:25:312
2981799331,8cyclictest3228-21kthreadcore21:20:202
2981799331,8cyclictest32072-21usage_Anybus00:15:312
2981799331,13cyclictest943-21kthreadcore19:45:212
2981799331,13cyclictest15568-21cut21:45:202
2981799331,10cyclictest10080-21grep23:45:292
2980199334,20cyclictest30672-21ssh23:05:130
2980199332,24cyclictest12357-21ssh22:55:140
2982599329,15cyclictest35966irq/58-eth021:30:153
2981799321,9cyclictest8468-21/usr/sbin/munin22:35:322
2981799321,9cyclictest8468-21/usr/sbin/munin22:35:312
2981799321,12cyclictest135-21systemd-journal23:16:322
2980999325,20cyclictest14767-21ssh21:45:141
2980199323,20cyclictest822-21ssh00:00:130
2982599313,18cyclictest35966irq/58-eth023:10:133
2981799310,6cyclictest5672-21grep21:40:142
2980999313,19cyclictest1677-21sh21:20:131
29809993115,10cyclictest21-21ksoftirqd/123:50:151
2980199313,22cyclictest10782-21ssh21:25:170
2980199312,23cyclictest9705-21ssh22:00:140
2982599302,20cyclictest35966irq/58-eth020:25:133
2981799301,8cyclictest22909-21/usr/sbin/munin20:10:272
2981799301,8cyclictest0-21swapper/220:20:112
2981799301,5cyclictest930-21snmpd20:29:472
2980199302,20cyclictest8520-21ssh22:35:140
2980199302,20cyclictest8520-21ssh22:35:140
2982599297,11cyclictest35966irq/58-eth021:40:123
2982599295,20cyclictest35966irq/58-eth022:05:133
2981799291,8cyclictest29046-21fschecks_count20:50:162
2981799291,8cyclictest18338-21grep20:35:302
2981799291,8cyclictest18338-21grep20:35:302
2982599283,14cyclictest35966irq/58-eth023:05:133
2982599282,17cyclictest35966irq/58-eth000:15:143
2980999282,20cyclictest13508-21apt-get00:25:131
2982599274,14cyclictest35966irq/58-eth000:25:133
2980999272,20cyclictest27812-21ssh22:10:131
2982599262,18cyclictest0-21swapper/321:25:173
2982599261,15cyclictest35966irq/58-eth021:15:143
2980999263,16cyclictest22721-21ssh22:25:151
2980999262,19cyclictest6119-21ssh23:45:141
2980199264,18cyclictest19817-21ssh21:30:150
2980199261,20cyclictest32736-21kworker/u8:022:15:130
2980199261,19cyclictest2197-21rm23:25:130
2980999252,19cyclictest28908-21apt-get20:50:131
29809992514,9cyclictest9186-21cron22:00:011
2980199253,15cyclictest151rcuc/022:40:120
2980199251,5cyclictest131rcu_preempt20:50:140
29801992514,7cyclictest19070-21ssh21:45:310
2980199251,17cyclictest24976-21rm21:15:140
2980199251,14cyclictest12-21ksoftirqd/021:00:130
2980199251,14cyclictest12-21ksoftirqd/021:00:130
29801992510,10cyclictest15462-21diskmemload00:38:130
2980999243,15cyclictest4488-21ssh22:15:121
2980199247,8cyclictest131rcu_preempt22:10:120
2980199241,19cyclictest10232-21ls00:05:200
29801992411,6cyclictest0-21swapper/020:55:140
29825992315,5cyclictest33-21ksoftirqd/320:50:163
29825992314,6cyclictest0-21swapper/321:55:253
2982599231,13cyclictest35966irq/58-eth000:20:103
2980999232,5cyclictest131rcu_preempt23:40:131
2980999232,17cyclictest19826-21ssh21:30:151
29809992313,6cyclictest31570-21/usr/sbin/munin00:35:211
2980999231,17cyclictest23799-21ssh21:50:141
2980999231,17cyclictest23799-21ssh21:50:141
2980999231,16cyclictest18892-21ssh00:10:131
2980999231,16cyclictest18892-21ssh00:10:131
2982599229,7cyclictest35966irq/58-eth023:15:143
2982599224,11cyclictest35966irq/58-eth023:25:123
29825992214,5cyclictest0-21swapper/322:25:303
2980999229,4cyclictest131rcu_preempt22:35:271
2980999229,4cyclictest131rcu_preempt22:35:271
2980999224,11cyclictest13657-21irqcore20:00:191
2980999222,16cyclictest3282-21ssh22:50:141
29809992212,3cyclictest21-21ksoftirqd/121:10:011
2980199223,17cyclictest29293-21ps23:20:270
2980199222,4cyclictest131rcu_preempt21:55:210
29801992216,3cyclictest131rcu_preempt00:20:110
2980199221,10cyclictest10574-21kthreadcore20:30:210
2980999212,4cyclictest53646usb-storage19:35:041
2980199214,7cyclictest131rcu_preempt21:35:220
2980199211,3cyclictest131rcu_preempt20:40:140
29801992110,7cyclictest0-21swapper/019:50:130
2982599203,2cyclictest0-21swapper/320:05:133
29825992015,3cyclictest33-21ksoftirqd/323:00:293
29825992011,6cyclictest35966irq/58-eth000:05:323
29825992011,6cyclictest35966irq/58-eth000:00:123
2982599201,13cyclictest0-21swapper/320:35:303
2982599201,13cyclictest0-21swapper/320:35:303
2980999201,13cyclictest10821-21ssh23:10:231
2980999201,12cyclictest5811-21ntp_192.168.11423:25:251
2980199209,8cyclictest24735-21cat23:35:300
2980199209,8cyclictest12-21ksoftirqd/022:05:140
2980199202,14cyclictest16806-21apt-get19:30:130
2980199202,12cyclictest0-21swapper/022:30:130
2980199201,16cyclictest3439-21cut22:50:160
2980199201,10cyclictest2142-21cut19:15:010
29801992010,7cyclictest1769-21df21:20:150
2982599197,9cyclictest0-21swapper/322:10:243
2982599193,9cyclictest0-21swapper/323:50:153
2982599193,8cyclictest35966irq/58-eth019:45:133
2982599193,11cyclictest0-21swapper/300:38:133
29825991912,4cyclictest0-21swapper/323:40:133
2982599191,15cyclictest0-21swapper/321:20:183
2982599191,10cyclictest0-21swapper/321:40:013
2980999199,7cyclictest31867-21ssh22:30:141
2980999196,8cyclictest0-21swapper/120:10:131
2980999194,11cyclictest0-21swapper/121:40:121
29809991911,5cyclictest28637-21kthreadcore00:15:221
2980999191,15cyclictest14466-21ssh21:25:301
2980999191,13cyclictest0-21swapper/120:55:161
29809991910,3cyclictest0-21swapper/123:23:101
2980199196,7cyclictest0-21swapper/019:40:110
2980199196,10cyclictest26851-21diskmemload22:25:310
2980199194,10cyclictest151rcuc/020:05:130
2980199193,9cyclictest0-21swapper/023:50:150
2980199192,12cyclictest0-21swapper/020:20:120
2980199191,3cyclictest131rcu_preempt21:11:460
29801991913,4cyclictest31167-21cron00:35:010
2982599189,6cyclictest35966irq/58-eth021:45:143
2982599189,5cyclictest35966irq/58-eth021:10:143
2982599188,6cyclictest35966irq/58-eth022:40:133
2982599188,6cyclictest0-21swapper/323:23:163
2982599186,9cyclictest35966irq/58-eth021:00:123
2982599186,9cyclictest35966irq/58-eth021:00:123
2982599184,8cyclictest0-21swapper/321:05:133
2982599182,12cyclictest0-21swapper/300:30:193
2982599182,10cyclictest0-21swapper/320:55:163
2982599182,10cyclictest0-21swapper/320:20:123
29825991810,3cyclictest0-21swapper/322:17:203
2981799181,9cyclictest29869-21latency_hist19:10:002
2980999189,7cyclictest9716-21apt-get20:30:141
2980999189,7cyclictest10072-21diskstats00:05:181
2980999187,8cyclictest25362-21ps21:15:181
2980999187,3cyclictest131rcu_preempt22:45:131
2980999182,9cyclictest6188-21sort21:00:141
2980999182,9cyclictest6188-21sort21:00:141
2980999181,9cyclictest31860-21kthreadcore19:10:221
29809991813,3cyclictest3780-21ls19:50:141
29809991812,4cyclictest0-21swapper/120:40:011
29809991812,4cyclictest0-21swapper/120:40:011
2980999181,14cyclictest7781-21irqrtprio19:20:181
29809991811,4cyclictest21-21ksoftirqd/122:40:281
2980199188,5cyclictest11465-21cut23:30:160
2980199185,10cyclictest0-21swapper/000:10:160
2980199185,10cyclictest0-21swapper/000:10:160
2980199182,13cyclictest23803-21ssh21:50:140
2980199182,13cyclictest23803-21ssh21:50:140
2980199181,7cyclictest7869-21kthreadcore19:20:190
29801991812,4cyclictest42-21kauditd20:30:010
2980199181,15cyclictest27233-21kthreadcore20:45:240
2980199181,14cyclictest29517-21cut22:45:200
2980199181,14cyclictest24412-21fschecks_time23:55:160
2980199181,13cyclictest15462-21diskmemload22:20:140
2980199180,7cyclictest131rcu_preempt21:05:130
2982599179,6cyclictest0-21swapper/322:30:063
2982599179,6cyclictest0-21swapper/321:51:143
2982599179,6cyclictest0-21swapper/321:51:143
2982599179,3cyclictest0-21swapper/322:52:193
2982599178,6cyclictest0-21swapper/300:11:053
2982599178,6cyclictest0-21swapper/300:11:053
2982599172,10cyclictest0-21swapper/320:40:143
2982599172,10cyclictest0-21swapper/319:50:143
2982599172,10cyclictest0-21swapper/319:20:143
29825991711,4cyclictest0-21swapper/323:31:573
2982599171,12cyclictest0-21swapper/323:35:173
29825991710,4cyclictest0-21swapper/322:45:143
2980999179,6cyclictest9537-21sed20:30:021
2980999178,7cyclictest4424-21cat00:00:241
2980999178,6cyclictest22546-21sshd00:30:131
2980999175,4cyclictest131rcu_preempt22:55:321
2980999172,11cyclictest9761-21/usr/sbin/munin22:00:151
2980999171,9cyclictest22442-21kthreadcore23:00:191
2980999171,13cyclictest4692-21awk20:25:011
2980999171,13cyclictest10174-21kthreadcore19:55:221
29809991710,4cyclictest29323-21irqcore21:35:191
2980199178,6cyclictest0-21swapper/020:15:120
2980199175,10cyclictest22058-21irqcore19:35:160
2980199174,10cyclictest15254-21kthreadcore20:00:230
2980199172,3cyclictest131rcu_preempt19:45:130
2980199171,14cyclictest13608-21kthreadcore19:25:210
2982599162,9cyclictest0-21swapper/322:20:143
2982599161,12cyclictest0-21swapper/320:45:163
2980999167,7cyclictest27809-21fschecks_count20:15:151
2980999167,7cyclictest18609-21irqrtprio20:05:181
2980999163,2cyclictest131rcu_preempt22:05:131
2980999162,9cyclictest0-21swapper/119:25:151
2980999162,11cyclictest20850-21kthreadcore20:40:201
2980999162,10cyclictest0-21swapper/120:45:151
2980999161,13cyclictest5814-21ntp_kernel_pll_19:15:261
29809991610,4cyclictest0-21swapper/119:40:271
2980199162,11cyclictest22981-21cpuspeed20:10:140
2980199161,11cyclictest29919-21latency_hist19:10:020
2982599156,3cyclictest33-21ksoftirqd/319:40:203
29825991510,3cyclictest33-21ksoftirqd/320:00:173
2980999151,7cyclictest31395-21cut19:45:161
2982599149,3cyclictest0-21swapper/320:10:163
2982599149,3cyclictest0-21swapper/319:27:553
2982599143,6cyclictest0-21swapper/319:55:153
2982599141,10cyclictest0-21swapper/319:10:143
2980999142,9cyclictest29922-21latency_hist19:10:021
2982599139,3cyclictest33-21ksoftirqd/319:15:133
2982599131,10cyclictest0-21swapper/320:30:203
2982599122,7cyclictest0-21swapper/320:15:173
2982599121,9cyclictest0-21swapper/319:35:203
2982599121,8cyclictest0-21swapper/319:35:013
2982599111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional