You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-29 - 19:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Jun 29, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
821299653,48cyclictest32406-21ssh11:00:120
8236996432,20cyclictest35966irq/58-eth012:20:133
822099565,41cyclictest9146-21ssh11:05:141
8236995514,28cyclictest35966irq/58-eth010:55:133
8228995530,19cyclictest15683-21ssh12:20:142
822099552,46cyclictest26805-21ssh09:10:131
8236995233,14cyclictest35966irq/58-eth009:15:143
8236995233,14cyclictest35966irq/58-eth009:15:133
822099524,37cyclictest24450-21ssh10:20:161
8236995036,7cyclictest35966irq/58-eth010:20:163
823699502,32cyclictest35966irq/58-eth010:40:153
822899501,26cyclictest15220-21ssh10:15:142
822899501,26cyclictest15220-21ssh10:15:132
822099502,38cyclictest26002-21ssh11:50:131
822099494,4cyclictest131rcu_preempt11:25:111
822899473,37cyclictest27346irq/40-dwc2_hso11:40:132
822099464,31cyclictest6521-21ssh12:15:121
822899452,33cyclictest0-21swapper/210:50:152
822099431,35cyclictest5227-21ssh10:45:131
821299434,25cyclictest7304-21ssh09:35:120
823699421,26cyclictest35966irq/58-eth007:55:123
823699411,26cyclictest35966irq/58-eth011:35:133
822899415,13cyclictest26390-21diskmemload09:55:122
821299412,34cyclictest24724-21ssh12:25:140
822899402,28cyclictest25543-21ssh09:45:112
8220994024,9cyclictest0-21swapper/111:30:141
8220994024,9cyclictest0-21swapper/111:30:141
821299395,31cyclictest7806-21ssh11:40:130
821299393,31cyclictest10390-21ssh12:35:160
821299392,32cyclictest20878-21ssh12:05:130
822899381,28cyclictest29481-21ssh10:05:132
8220993822,5cyclictest21-21ksoftirqd/112:00:131
821299384,23cyclictest12-21ksoftirqd/008:05:130
821299381,32cyclictest30666-21ssh09:30:150
822899372,29cyclictest26244-21ssh10:01:422
822099378,20cyclictest21-21ksoftirqd/111:55:131
821299373,28cyclictest31122-21rm11:35:140
823699362,17cyclictest35966irq/58-eth011:40:133
821299363,7cyclictest131rcu_preempt09:48:400
8212993617,17cyclictest12-21ksoftirqd/008:00:110
822099353,25cyclictest28533-21ssh10:40:141
821299346,22cyclictest151rcuc/010:30:130
821299343,21cyclictest18223-21ssh11:10:140
823699333,12cyclictest35966irq/58-eth009:55:123
823699332,16cyclictest35966irq/58-eth011:30:143
823699332,16cyclictest35966irq/58-eth011:30:133
822899335,18cyclictest27346irq/40-dwc2_hso08:40:102
8228993313,4cyclictest27-21ksoftirqd/211:20:422
823699324,18cyclictest35966irq/58-eth010:15:143
823699324,18cyclictest35966irq/58-eth010:15:133
822899323,21cyclictest2644-21ssh11:55:122
822099321,22cyclictest0-21swapper/111:15:141
823699313,15cyclictest35966irq/58-eth009:45:123
8236993114,11cyclictest35966irq/58-eth011:45:153
8220993110,11cyclictest21-21ksoftirqd/109:30:131
821299311,23cyclictest1298-21ssh12:30:110
821299311,22cyclictest20366-21ssh10:00:130
821299311,21cyclictest12529-21ssh09:20:140
822899302,21cyclictest0-21swapper/208:45:142
822899301,13cyclictest23385-21ssh10:55:132
822099301,18cyclictest14377-21ssh10:50:131
821299302,22cyclictest21557-21ssh09:25:130
8212993015,6cyclictest0-21swapper/008:20:110
822899298,12cyclictest27-21ksoftirqd/209:10:392
822899293,20cyclictest0-21swapper/209:51:412
822899293,20cyclictest0-21swapper/209:40:422
823699283,16cyclictest35966irq/58-eth011:55:133
822899287,11cyclictest27-21ksoftirqd/212:26:362
822899285,16cyclictest27-21ksoftirqd/212:02:372
822899283,19cyclictest27346irq/40-dwc2_hso09:05:132
822899283,18cyclictest27346irq/40-dwc2_hso07:45:102
822899283,17cyclictest27346irq/40-dwc2_hso08:35:142
822899282,19cyclictest0-21swapper/208:15:052
822899282,10cyclictest4941-21ssh09:32:412
822899281,18cyclictest27-21ksoftirqd/211:05:142
821299284,18cyclictest1130-21ssh10:25:130
821299282,20cyclictest19375-21ssh10:35:140
821299281,23cyclictest0-21swapper/009:10:150
823699273,11cyclictest0-21swapper/310:01:413
822899279,4cyclictest27-21ksoftirqd/210:44:232
822899277,12cyclictest27-21ksoftirqd/210:25:132
822899276,3cyclictest0-21swapper/211:45:132
822899274,17cyclictest27346irq/40-dwc2_hso08:30:102
822899274,16cyclictest27346irq/40-dwc2_hso08:05:142
821299278,7cyclictest131rcu_preempt07:15:130
823699264,18cyclictest35966irq/58-eth008:55:143
823699263,17cyclictest0-21swapper/309:54:413
822899269,11cyclictest27-21ksoftirqd/210:37:142
822899266,14cyclictest27-21ksoftirqd/207:20:242
822899264,4cyclictest27346irq/40-dwc2_hso09:25:132
822899262,14cyclictest27346irq/40-dwc2_hso08:20:112
822899261,21cyclictest7409-21find08:50:112
822899261,14cyclictest27-21ksoftirqd/210:34:372
821299263,7cyclictest131rcu_preempt09:54:420
821299261,20cyclictest3466-21ssh09:15:140
821299261,20cyclictest3466-21ssh09:15:140
823699254,15cyclictest35966irq/58-eth007:50:123
822899259,6cyclictest27-21ksoftirqd/212:06:292
822899259,4cyclictest27-21ksoftirqd/211:36:382
822899254,14cyclictest5253-21diskmemload10:45:142
822899251,10cyclictest27-21ksoftirqd/210:24:302
8228992510,9cyclictest27-21ksoftirqd/211:15:352
822099251,17cyclictest32376-21ssh11:00:091
8220992510,10cyclictest21-21ksoftirqd/110:30:141
821299251,17cyclictest27330-21ssh11:15:160
823699243,8cyclictest35966irq/58-eth012:25:133
823699243,12cyclictest0-21swapper/312:00:113
823699243,11cyclictest35966irq/58-eth010:45:133
823699241,11cyclictest35966irq/58-eth012:30:093
823699241,11cyclictest0-21swapper/308:30:213
822899248,8cyclictest27-21ksoftirqd/212:19:252
822899248,8cyclictest27-21ksoftirqd/211:52:332
822899248,8cyclictest27-21ksoftirqd/207:40:012
822899248,13cyclictest261rcuc/212:38:532
822899245,8cyclictest27-21ksoftirqd/211:29:562
822899244,14cyclictest27346irq/40-dwc2_hso08:00:112
822899242,4cyclictest271ksoftirqd/209:21:442
822899241,13cyclictest6145-21sh10:10:142
8228992410,8cyclictest27-21ksoftirqd/207:50:142
822899240,15cyclictest27-21ksoftirqd/211:00:432
822099242,17cyclictest6147-21ssh10:10:131
821299244,14cyclictest12-21ksoftirqd/008:55:140
821299242,16cyclictest5256-21ssh10:45:140
8212992410,5cyclictest0-21swapper/011:20:130
823699234,13cyclictest0-21swapper/312:05:133
822899239,8cyclictest0-21swapper/207:25:252
822899237,10cyclictest27-21ksoftirqd/209:18:172
822899237,10cyclictest27-21ksoftirqd/209:18:162
822899235,8cyclictest27-21ksoftirqd/212:13:302
822899234,11cyclictest27346irq/40-dwc2_hso07:10:132
822899231,8cyclictest7513-21fschecks_time09:35:152
822099230,18cyclictest28239-21ssh09:27:411
822099230,17cyclictest19434-21ls07:55:161
821299233,15cyclictest753-21sendmail-mta07:30:160
821299233,15cyclictest753-21sendmail-mta07:30:150
8212992315,5cyclictest0-21swapper/007:45:110
821299231,16cyclictest27065-21ssh10:55:270
8236992213,6cyclictest0-21swapper/309:35:203
823699221,18cyclictest0-21swapper/310:50:313
8236992210,8cyclictest35966irq/58-eth012:15:143
822899227,7cyclictest22776-21kthreadcore11:30:212
822899227,7cyclictest22776-21kthreadcore11:30:202
8228992212,8cyclictest0-21swapper/208:55:162
822899221,16cyclictest18545-21ssh11:10:182
822099223,15cyclictest15235-21ssh10:15:151
822099223,15cyclictest15235-21ssh10:15:141
822099222,15cyclictest16375-21ssh09:55:501
822099221,16cyclictest20259-21ls09:00:211
8220992210,3cyclictest131rcu_preempt10:00:231
823699217,8cyclictest35966irq/58-eth009:25:133
823699213,12cyclictest0-21swapper/309:20:143
823699211,14cyclictest35966irq/58-eth009:10:133
8236992110,8cyclictest0-21swapper/311:52:173
822899212,10cyclictest23765-21sendmail08:00:012
8228992115,4cyclictest0-21swapper/207:30:162
8228992115,4cyclictest0-21swapper/207:30:152
8228992111,4cyclictest27-21ksoftirqd/212:31:422
822899211,11cyclictest3168-21kthreadcore08:10:202
822899211,10cyclictest9568-21latency_hist07:45:012
822899211,10cyclictest16217-21kthreadcore07:15:232
822099213,7cyclictest131rcu_preempt12:10:121
822099212,7cyclictest24502-21ntp_192.168.11412:05:241
8220992113,5cyclictest16342-21/usr/sbin/munin09:40:121
822099211,17cyclictest26090-21ls07:25:191
822099211,16cyclictest7526-21gpgv11:40:021
822099211,14cyclictest16886-21ssh11:45:151
822099210,18cyclictest12415-21ls08:55:181
821299215,10cyclictest0-21swapper/011:25:120
821299212,15cyclictest26001-21sh11:50:140
821299211,15cyclictest26390-21diskmemload12:20:290
821299211,14cyclictest21998-21find11:30:130
821299211,14cyclictest21998-21find11:30:120
823699209,6cyclictest0-21swapper/309:44:233
823699203,11cyclictest35966irq/58-eth010:10:123
823699202,13cyclictest35966irq/58-eth010:05:153
8236992012,6cyclictest35966irq/58-eth010:25:133
8236992011,4cyclictest0-21swapper/311:27:363
822899202,10cyclictest19554-21grep08:25:292
822899201,9cyclictest17163-21cut09:00:152
822099205,3cyclictest131rcu_preempt09:15:331
822099205,3cyclictest131rcu_preempt09:15:321
822099202,4cyclictest20556-21kworker/1:212:23:161
822099201,8cyclictest26305-21kthreadcore10:55:231
8220992014,4cyclictest0-21swapper/112:35:151
822099201,16cyclictest27800-21fschecks_time07:30:181
822099201,16cyclictest27800-21fschecks_time07:30:181
822099201,14cyclictest3915-21ssh11:20:131
822099201,14cyclictest23095-21ssh10:35:281
822099201,11cyclictest0-21swapper/110:25:401
8220992010,7cyclictest17742-21cron07:20:011
821299202,15cyclictest7474-21kthreadcore07:40:220
821299202,13cyclictest19659-21ssh09:59:410
821299202,13cyclictest0-21swapper/010:05:150
821299202,12cyclictest26464-21kworker/u8:109:40:120
8212992013,4cyclictest0-21swapper/007:55:170
8212992012,5cyclictest9515-21interrupts11:05:170
821299201,10cyclictest14568-21fschecks_count07:50:140
8212992010,7cyclictest11359-21http12:00:010
821299200,14cyclictest28024-21ls10:20:260
823699198,8cyclictest0-21swapper/311:03:363
823699198,8cyclictest0-21swapper/310:36:193
823699191,15cyclictest0-21swapper/308:10:283
823699191,12cyclictest0-21swapper/308:15:273
8236991910,6cyclictest0-21swapper/310:34:053
8228991910,7cyclictest261rcuc/207:10:022
8228991910,7cyclictest261rcuc/207:10:012
822099196,10cyclictest614-21dbus-daemon11:41:071
822099192,14cyclictest17999-21apt-get07:20:141
822099192,13cyclictest5754-21apt-get08:15:031
8220991912,4cyclictest0-21swapper/108:45:181
822099191,15cyclictest5839-21latency_hist08:15:021
822099191,14cyclictest2215-21ssh09:50:131
822099190,17cyclictest14623-21ls08:20:281
821299199,7cyclictest12-21ksoftirqd/008:35:130
821299192,13cyclictest630-21Async09:36:310
821299191,8cyclictest7788-21tr12:15:200
821299191,8cyclictest6099-21/usr/sbin/munin10:10:110
821299191,15cyclictest2265-21cat12:10:300
821299191,15cyclictest18841-21ssh10:15:270
821299191,15cyclictest18841-21ssh10:15:260
8212991910,7cyclictest20557-21apt-get08:30:110
823699187,7cyclictest35966irq/58-eth012:35:163
823699182,9cyclictest0-21swapper/311:10:143
823699182,10cyclictest0-21swapper/311:05:063
823699182,10cyclictest0-21swapper/307:10:143
8236991810,5cyclictest35966irq/58-eth009:31:423
822099188,7cyclictest14619-21tune2fs07:50:151
822099188,7cyclictest14493-21cut09:20:211
822099186,9cyclictest21918-21grep09:05:171
822099183,12cyclictest1415-21kthreadcore07:35:201
822099182,11cyclictest32216-21ssh09:47:411
822099182,11cyclictest0-21swapper/109:37:421
822099181,11cyclictest20326-21latency_hist08:30:011
822099181,10cyclictest0-21swapper/107:40:151
8220991810,5cyclictest5744-21latency_hist10:10:021
822099180,7cyclictest23097-21usage_EthcTask11:10:321
822099180,4cyclictest24697-21apt-get12:25:131
821299188,7cyclictest18167-21grep10:50:300
821299181,2cyclictest131rcu_preempt11:49:350
8212991811,5cyclictest0-21swapper/012:00:270
821299181,13cyclictest21471-21ntp_states07:20:270
823699179,6cyclictest0-21swapper/312:11:283
823699179,6cyclictest0-21swapper/311:20:063
823699179,3cyclictest0-21swapper/311:17:093
823699172,9cyclictest0-21swapper/309:00:113
823699172,10cyclictest0-21swapper/308:20:143
823699172,10cyclictest0-21swapper/307:40:143
823699172,10cyclictest0-21swapper/307:35:153
823699172,10cyclictest0-21swapper/307:30:163
823699172,10cyclictest0-21swapper/307:30:153
823699171,9cyclictest0-21swapper/308:05:113
822099179,5cyclictest3274-21kthreadcore12:30:201
822099172,11cyclictest962-21munin-plugin-st08:10:011
821299178,6cyclictest2909-21kthreadcore08:10:190
821299178,6cyclictest17540-21irqrtprio09:00:180
821299177,8cyclictest6155-21df_inode08:15:150
821299175,9cyclictest32636-21grep10:40:310
821299173,9cyclictest0-21swapper/008:50:120
821299171,6cyclictest22621-21kthreadcore09:05:200
823699162,9cyclictest0-21swapper/307:25:113
823699162,11cyclictest0-21swapper/308:40:113
822099165,9cyclictest25660-21cut08:00:201
822099164,3cyclictest16766systemd-network08:54:331
822099163,10cyclictest27648-21kthreadcore08:35:221
822099162,9cyclictest0-21swapper/108:30:121
822099161,7cyclictest9961-21irqcore07:45:151
822099161,3cyclictest0-21swapper/108:40:101
822099161,12cyclictest8560-21cut07:10:161
821299162,11cyclictest844-21munin-node07:35:150
8212991611,3cyclictest12598-21usage_EthcTask07:10:300
821299161,12cyclictest17534-21kthreadcore08:25:200
821299161,12cyclictest1479-21awk08:40:250
821299161,11cyclictest2610-21apt-get08:45:140
821299160,13cyclictest8285-21latency_hist07:10:020
821299160,13cyclictest8285-21latency_hist07:10:010
823699151,8cyclictest0-21swapper/308:50:003
823699151,12cyclictest0-21swapper/308:25:133
823699151,10cyclictest0-21swapper/307:15:143
823699142,10cyclictest0-21swapper/308:35:143
823699142,10cyclictest0-21swapper/308:00:133
823699141,8cyclictest0-21swapper/307:45:263
823699141,7cyclictest0-21swapper/308:50:143
823699141,10cyclictest0-21swapper/309:10:023
822099141,9cyclictest8247-21latency_hist07:10:021
822099141,9cyclictest8247-21latency_hist07:10:011
823699121,9cyclictest0-21swapper/307:20:193
82369991,6cyclictest0-21swapper/307:10:023
82369991,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional