You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-23 - 00:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon Jun 22, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
676699511,9cyclictest131rcu_preempt12:25:140
6801994931,13cyclictest0-21swapper/310:05:143
676699443,5cyclictest131rcu_preempt09:20:110
676699441,4cyclictest131rcu_preempt10:55:120
680199436,26cyclictest35966irq/58-eth011:00:133
6778994318,17cyclictest21-21ksoftirqd/109:35:141
6778994318,17cyclictest21-21ksoftirqd/109:35:141
676699424,35cyclictest12144-21ssh11:45:110
676699423,34cyclictest14003-21ssh11:10:140
676699413,35cyclictest14148-21ssh09:40:130
676699407,11cyclictest131rcu_preempt08:15:130
680199391,32cyclictest35966irq/58-eth010:35:133
676699397,24cyclictest8623-21ssh09:55:100
680199383,19cyclictest35966irq/58-eth011:40:123
679199372,26cyclictest19718-21ssh09:25:122
679199372,18cyclictest5262-21ssh11:58:402
677899372,24cyclictest24746-21ssh10:40:121
676699373,31cyclictest28104-21ssh12:30:130
680199363,19cyclictest35966irq/58-eth012:20:123
676699363,29cyclictest30082-21ssh10:25:120
6766993621,9cyclictest131rcu_preempt08:00:130
680199351,17cyclictest35966irq/58-eth011:45:133
679199351,29cyclictest12183-21ssh10:15:122
679199342,25cyclictest29863-21ssh10:40:482
6791993415,14cyclictest0-21swapper/208:00:132
679199335,16cyclictest27346irq/40-dwc2_hso11:45:112
679199332,5cyclictest0-21swapper/208:35:122
680199322,18cyclictest35966irq/58-eth011:10:153
6801993216,11cyclictest35966irq/58-eth010:45:123
6801993213,11cyclictest35966irq/58-eth009:45:123
6791993212,14cyclictest0-21swapper/212:10:122
679199321,19cyclictest21144-21ssh10:20:122
677899322,24cyclictest22969-21ssh11:15:131
677899321,22cyclictest28654-21ssh09:30:111
677899321,22cyclictest28654-21ssh09:30:101
676699321,25cyclictest1309-21ssh10:45:120
680199313,17cyclictest35966irq/58-eth008:35:133
680199312,17cyclictest35966irq/58-eth012:05:133
679199316,19cyclictest17685-21ssh10:00:142
679199313,17cyclictest23085-21diskmemload09:45:122
679199302,18cyclictest0-21swapper/211:10:152
676699302,22cyclictest21093-21ssh12:06:060
676699301,4cyclictest131rcu_preempt11:20:110
680199291,19cyclictest35966irq/58-eth008:30:133
679199293,16cyclictest27346irq/40-dwc2_hso11:00:132
679199293,16cyclictest27346irq/40-dwc2_hso07:20:132
679199292,11cyclictest930-21snmpd11:21:402
6791992911,12cyclictest27-21ksoftirqd/211:25:572
6791992910,11cyclictest131rcu_preempt09:19:112
677899293,16cyclictest6809-21ssh12:00:131
6766992910,4cyclictest12-21ksoftirqd/010:05:120
680199283,16cyclictest0-21swapper/310:40:493
6801992810,6cyclictest0-21swapper/310:31:103
679199282,20cyclictest0-21swapper/211:40:132
679199282,11cyclictest23218-21ssh12:08:402
679199282,10cyclictest12602-21ssh12:01:402
6791992810,11cyclictest27-21ksoftirqd/209:38:002
6791992810,11cyclictest27-21ksoftirqd/209:37:592
677899286,10cyclictest0-21swapper/110:15:121
677899282,16cyclictest6677-21ssh10:30:141
6778992811,6cyclictest21-21ksoftirqd/110:00:111
676699285,16cyclictest12-21ksoftirqd/009:45:110
676699282,21cyclictest12207-21ssh10:15:120
676699282,18cyclictest3151-21ssh11:40:120
676699282,17cyclictest8510-21ssh11:25:120
676699281,21cyclictest24755-21sh10:40:120
680199279,6cyclictest0-21swapper/311:35:043
680199272,6cyclictest0-21swapper/311:08:403
680199272,21cyclictest35966irq/58-eth010:10:133
680199272,21cyclictest35966irq/58-eth010:10:123
6801992712,10cyclictest0-21swapper/310:25:233
680199271,11cyclictest0-21swapper/311:55:203
680199271,10cyclictest0-21swapper/309:20:023
679199279,6cyclictest27-21ksoftirqd/210:32:012
679199272,11cyclictest10181-21ssh12:20:132
679199272,10cyclictest22767-21ssh11:30:582
679199271,8cyclictest25203-21ssh09:10:132
679199271,20cyclictest0-21swapper/208:45:142
679199271,11cyclictest7063-21ssh12:16:412
677899272,19cyclictest7024-21sh11:23:411
677899272,19cyclictest2462-21ssh11:39:411
677899272,19cyclictest17562-21ssh12:23:401
677899271,21cyclictest24800-21diskmemload12:35:111
676699272,19cyclictest13344-21ssh10:32:400
680199269,6cyclictest0-21swapper/311:17:513
680199269,6cyclictest0-21swapper/310:04:283
680199263,8cyclictest0-21swapper/312:10:153
680199261,9cyclictest0-21swapper/310:20:133
680199261,15cyclictest35966irq/58-eth009:20:123
679199269,4cyclictest27-21ksoftirqd/210:47:502
679199262,19cyclictest0-21swapper/211:35:142
6791992613,8cyclictest27-21ksoftirqd/210:36:402
679199261,14cyclictest30102-21ssh10:25:142
6791992610,10cyclictest131rcu_preempt09:52:172
677899265,13cyclictest0-21swapper/111:50:121
677899264,12cyclictest24800-21diskmemload09:25:121
677899262,9cyclictest0-21swapper/108:20:051
677899262,4cyclictest131rcu_preempt08:45:131
677899262,18cyclictest19183-21ssh11:10:531
676699262,18cyclictest28147-21sh11:15:540
680199253,7cyclictest0-21swapper/311:23:413
680199253,10cyclictest0-21swapper/311:30:583
680199252,17cyclictest0-21swapper/312:15:403
679199259,8cyclictest27-21ksoftirqd/208:15:222
679199257,5cyclictest27-21ksoftirqd/209:31:042
679199257,5cyclictest27-21ksoftirqd/209:31:032
679199255,3cyclictest131rcu_preempt11:05:142
679199254,14cyclictest27346irq/40-dwc2_hso10:50:122
679199253,15cyclictest4629-21ssh12:35:122
679199252,9cyclictest0-21swapper/209:05:142
679199251,4cyclictest27-21ksoftirqd/212:29:312
677899259,4cyclictest131rcu_preempt12:00:021
680199249,6cyclictest0-21swapper/312:37:313
680199242,6cyclictest0-21swapper/312:26:393
680199242,6cyclictest0-21swapper/311:28:393
680199242,11cyclictest35966irq/58-eth009:25:113
6801992416,4cyclictest0-21swapper/310:55:123
679199248,7cyclictest13205-21kthreadcore07:15:202
679199247,3cyclictest27-21ksoftirqd/210:59:142
679199245,6cyclictest27-21ksoftirqd/209:24:102
679199244,4cyclictest27-21ksoftirqd/212:33:512
679199242,9cyclictest27-21ksoftirqd/208:25:132
679199242,9cyclictest27-21ksoftirqd/208:25:122
679199242,16cyclictest0-21swapper/211:15:552
679199241,10cyclictest694566ssh10:10:272
679199241,10cyclictest694566ssh10:10:262
677899242,17cyclictest22393-21ssh12:07:391
677899241,19cyclictest26670-21ssh10:05:141
676699242,2cyclictest131rcu_preempt07:40:110
6766992415,5cyclictest24800-21diskmemload10:35:140
680199233,9cyclictest0-21swapper/312:01:043
680199233,14cyclictest0-21swapper/310:50:143
680199231,16cyclictest35966irq/58-eth009:55:113
679199236,8cyclictest15355-21latency_hist09:00:022
679199236,7cyclictest27-21ksoftirqd/209:58:062
679199236,11cyclictest27-21ksoftirqd/209:44:232
679199233,10cyclictest1485-21kthreadcore07:35:202
679199232,19cyclictest0-21swapper/207:30:142
6791992313,8cyclictest0-21swapper/210:05:232
679199231,18cyclictest3826-21cut07:40:162
679199231,15cyclictest20091-21ls09:05:012
679199231,12cyclictest23566-21ssh11:50:212
679199230,8cyclictest19258-21ls08:30:172
679199230,12cyclictest9433-21find08:20:102
677899231,17cyclictest14220-21apt-get08:25:141
677899231,17cyclictest14220-21apt-get08:25:131
676699236,9cyclictest0-21swapper/011:55:130
676699234,11cyclictest0-21swapper/008:10:140
676699233,15cyclictest12-21ksoftirqd/011:00:120
676699232,4cyclictest131rcu_preempt08:50:130
676699231,8cyclictest24800-21diskmemload10:52:400
6766992312,6cyclictest24800-21diskmemload11:05:120
676699231,17cyclictest24800-21diskmemload10:10:140
676699231,17cyclictest24800-21diskmemload10:10:130
680199224,12cyclictest35966irq/58-eth009:35:143
680199224,12cyclictest35966irq/58-eth009:35:133
680199223,6cyclictest0-21swapper/307:45:303
6801992210,9cyclictest0-21swapper/312:31:513
679199226,10cyclictest6913-21apt-get07:10:132
679199226,10cyclictest6913-21apt-get07:10:122
677899221,12cyclictest9812-21sendmail-msp12:20:011
676699227,4cyclictest131rcu_preempt08:35:010
676699222,14cyclictest29488-21ssh11:54:400
6766992211,8cyclictest21654-21grep11:30:300
680199213,8cyclictest0-21swapper/308:15:273
6801992111,6cyclictest0-21swapper/309:10:223
680199211,16cyclictest0-21swapper/308:10:233
6791992113,5cyclictest0-21swapper/207:55:012
679199211,10cyclictest12897-21latency_hist07:50:012
677899212,16cyclictest10401-21apt-get10:50:111
6778992113,5cyclictest12269-21open_inodes11:25:261
6778992112,5cyclictest31838-21ntp_states12:30:271
6778992111,8cyclictest10369-21cron08:55:011
677899211,16cyclictest30052-21ssh12:11:071
6778992111,6cyclictest21-21ksoftirqd/107:15:141
677899211,13cyclictest19151-21cut08:30:161
676699212,6cyclictest19715-21apt-get09:25:130
680199209,8cyclictest0-21swapper/309:42:333
679199203,9cyclictest27508-21fschecks_time08:05:162
679199201,9cyclictest7908-21kthreadcore08:50:202
679199201,11cyclictest32322-21fschecks_time08:10:162
677899209,3cyclictest131rcu_preempt09:40:201
677899204,10cyclictest0-21swapper/107:45:151
677899203,8cyclictest21-21ksoftirqd/110:20:281
677899202,10cyclictest29763-21switchtime07:30:271
677899201,8cyclictest24366-21run-parts10:40:011
677899201,14cyclictest32204-21ssh09:50:151
676699202,13cyclictest26586-21ssh11:35:140
676699202,13cyclictest17796-21ssh10:00:150
676699201,9cyclictest21374-21switchtime07:55:290
6766992014,4cyclictest0-21swapper/012:20:150
6766992013,3cyclictest0-21swapper/009:05:130
6766992011,6cyclictest6038-21cat09:15:310
676699201,15cyclictest6797-21ssh12:00:130
680199191,13cyclictest0-21swapper/311:50:133
679199192,10cyclictest28801-21fschecks_count08:40:162
679199191,8cyclictest21544-21fschecks_time07:25:172
679199191,14cyclictest17952-21wc07:55:182
677899199,7cyclictest23843-21usage_Anybus10:55:321
677899196,3cyclictest131rcu_preempt11:05:301
677899196,11cyclictest19281-21head09:00:301
6778991911,6cyclictest1166-21sshd11:00:511
677899191,12cyclictest6992-21timerandwakeup07:40:251
677899191,10cyclictest27854-21latency_hist12:30:011
6778991910,7cyclictest16744-1kworker/u9:009:11:411
676699199,7cyclictest4604-21apt-get12:35:120
676699192,11cyclictest0-21swapper/008:25:150
676699192,11cyclictest0-21swapper/008:25:150
6766991913,4cyclictest0-21swapper/008:45:250
6766991912,5cyclictest15183-21cron09:00:020
676699191,15cyclictest28400-21idleruntime-cro08:40:020
680199186,7cyclictest0-21swapper/310:15:113
680199182,10cyclictest0-21swapper/308:05:143
680199182,10cyclictest0-21swapper/307:35:133
680199181,10cyclictest0-21swapper/308:50:023
6801991810,6cyclictest35966irq/58-eth009:30:113
6801991810,6cyclictest35966irq/58-eth009:30:103
679199181,9cyclictest6826-21latency_hist07:10:012
677899189,5cyclictest3180-21cpu11:40:121
677899184,4cyclictest131rcu_preempt10:45:241
677899181,3cyclictest0-21swapper/110:10:141
677899181,3cyclictest0-21swapper/110:10:141
677899181,14cyclictest19173-21cut11:30:211
677899181,14cyclictest14138-21kthreadcore11:45:211
677899181,13cyclictest12139-21ssh09:55:211
676699189,6cyclictest0-21swapper/012:15:250
676699185,10cyclictest29394-21cut08:05:210
676699183,3cyclictest131rcu_preempt10:20:130
676699181,14cyclictest5172-21apt-get09:35:120
676699181,14cyclictest5172-21apt-get09:35:110
676699181,14cyclictest25883-21munin-run07:30:010
676699181,14cyclictest0-21swapper/009:50:120
6766991810,5cyclictest0-21swapper/007:10:160
6766991810,5cyclictest0-21swapper/007:10:150
676699180,12cyclictest12692-21latency_hist07:50:020
680199179,6cyclictest0-21swapper/309:50:043
680199172,10cyclictest0-21swapper/307:30:143
680199172,10cyclictest0-21swapper/307:20:143
680199171,10cyclictest0-21swapper/307:10:153
680199171,10cyclictest0-21swapper/307:10:143
677899179,5cyclictest31075-21cut07:35:151
677899178,7cyclictest25972-21ntpq08:00:251
677899176,9cyclictest25264-21tr09:45:211
677899176,8cyclictest1-21systemd08:05:031
677899173,11cyclictest29393-21cut08:40:191
6778991711,4cyclictest4742-21hwlatdetect08:15:151
6778991711,4cyclictest20081-21sendmail_mailqu07:20:271
677899171,12cyclictest30291-21fschecks_time10:25:161
677899171,10cyclictest24800-21diskmemload09:15:131
676699173,6cyclictest0-21swapper/008:20:110
676699172,10cyclictest0-21swapper/007:35:120
676699172,10cyclictest0-21swapper/007:30:140
6766991711,4cyclictest20009-21cat09:05:010
676699171,13cyclictest30647-21kthreadcore08:40:190
676699171,13cyclictest14640-21cut07:15:210
676699171,12cyclictest820-21usage_EthcTask09:30:290
676699171,12cyclictest820-21usage_EthcTask09:30:290
676699171,12cyclictest25314-21df_inode09:10:160
6766991710,5cyclictest24703-21apt-get12:10:110
6766991710,5cyclictest0-21swapper/007:50:220
680199161,8cyclictest0-21swapper/309:05:023
680199161,12cyclictest0-21swapper/308:00:153
677899166,7cyclictest23505-21latency09:05:221
677899163,10cyclictest18292-21irqrtprio07:55:181
677899162,9cyclictest0-21swapper/107:10:151
677899162,9cyclictest0-21swapper/107:10:151
677899162,11cyclictest2222-21cut08:10:221
677899161,12cyclictest24495-21kernelversion08:35:191
677899161,12cyclictest13284-21irqcore07:50:181
677899161,11cyclictest14256-21/usr/sbin/munin09:20:221
676699167,6cyclictest21108-21cron07:25:020
680199152,10cyclictest0-21swapper/307:25:153
677899152,10cyclictest10762-21fschecks_time08:55:151
677899151,11cyclictest21681-21cut07:25:191
676699152,9cyclictest6818-21latency_hist07:10:020
680199149,3cyclictest0-21swapper/307:55:243
680199141,7cyclictest0-21swapper/308:50:233
680199141,6cyclictest0-21swapper/309:10:013
680199141,10cyclictest0-21swapper/307:20:013
680199138,3cyclictest0-21swapper/307:50:143
680199131,9cyclictest0-21swapper/308:25:203
680199131,9cyclictest0-21swapper/308:25:193
680199131,5cyclictest0-21swapper/308:55:213
677899131,9cyclictest6779-21latency_hist07:10:011
680199122,6cyclictest0-21swapper/308:40:133
680199122,6cyclictest0-21swapper/308:20:053
680199121,9cyclictest0-21swapper/307:40:193
680199111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional