You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-03 - 18:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun May 03, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2594099621,50cyclictest15780-21ssh11:55:161
2593299584,11cyclictest12-21ksoftirqd/007:15:150
25948995045,2cyclictest27346irq/40-dwc2_hso12:30:122
2594899501,45cyclictest3464-21ssh11:30:132
2594899501,37cyclictest14610-21ssh11:00:132
2594899499,18cyclictest27-21ksoftirqd/210:30:112
2593299485,37cyclictest6851-21ssh11:50:140
25940994727,7cyclictest21-21ksoftirqd/110:45:131
2595699463,28cyclictest35966irq/58-eth011:00:133
2594099463,8cyclictest131rcu_preempt11:25:131
2595699443,24cyclictest35966irq/58-eth010:30:113
2594099434,28cyclictest27910-21ssh09:55:141
2593299433,37cyclictest20933-21ssh09:15:140
25948994219,17cyclictest27-21ksoftirqd/212:05:122
2593299423,4cyclictest12-21ksoftirqd/009:05:150
2593299411,35cyclictest2307-21ssh10:35:140
2595699394,25cyclictest35966irq/58-eth012:25:143
25940993923,3cyclictest21-21ksoftirqd/108:30:131
2594099391,6cyclictest131rcu_preempt09:35:131
2593299393,29cyclictest9080-21ssh11:15:120
25956993824,9cyclictest35966irq/58-eth012:20:113
2595699382,20cyclictest35966irq/58-eth009:40:133
2593299371,31cyclictest15417-21ssh09:30:130
2594099361,26cyclictest18927-21ssh09:50:141
2595699354,16cyclictest35966irq/58-eth009:35:123
25948993513,17cyclictest0-21swapper/210:20:132
2594099351,27cyclictest5711-21ssh10:55:131
25932993511,18cyclictest12-21ksoftirqd/012:35:140
25932993511,18cyclictest12-21ksoftirqd/012:35:130
2594899341,28cyclictest22208-21sh12:35:152
2594899341,28cyclictest22208-21sh12:35:142
2594899341,24cyclictest2321-21ssh10:35:152
2594099338,18cyclictest29452-21cat10:50:171
2593299331,23cyclictest9918-21ssh09:45:130
25956993224,5cyclictest35966irq/58-eth010:10:113
2595699322,18cyclictest35966irq/58-eth012:30:113
2593299323,20cyclictest12-21ksoftirqd/010:05:130
25956993121,5cyclictest33-21ksoftirqd/311:15:123
2594899311,22cyclictest9922-21ssh09:45:142
2594899303,20cyclictest0-21swapper/210:56:122
2594099303,18cyclictest17991-21ssh11:20:111
2594099302,21cyclictest25762-21ssh10:30:121
2594099302,21cyclictest24005-21ssh11:05:181
2594099302,16cyclictest27851-21ssh12:20:111
2593299302,21cyclictest23486-21ssh11:05:100
2593299302,19cyclictest4347-21ssh12:25:130
2595699293,17cyclictest0-21swapper/310:35:143
2595699292,17cyclictest35966irq/58-eth010:40:133
2594899293,10cyclictest0-21swapper/211:08:112
25948992914,12cyclictest261rcuc/212:10:132
2593299298,4cyclictest131rcu_preempt10:30:100
2594899283,10cyclictest0-21swapper/211:44:112
2594899282,14cyclictest27346irq/40-dwc2_hso10:50:122
2594899282,10cyclictest0-21swapper/207:55:132
2594899281,9cyclictest1950466ssh11:20:192
2594899281,9cyclictest1193566ssh09:10:112
25948992814,6cyclictest27-21ksoftirqd/209:38:282
2594899281,21cyclictest0-21swapper/207:30:152
2594899281,10cyclictest436366ssh10:00:142
2594099282,21cyclictest16805-21ssh10:25:161
2594099282,19cyclictest7131-21ssh11:30:231
2594099282,19cyclictest25135-21/usr/sbin/munin11:40:261
2594099282,18cyclictest22172-21apt10:10:101
2595699273,17cyclictest0-21swapper/310:59:113
2595699272,18cyclictest0-21swapper/311:05:103
25956992712,10cyclictest35966irq/58-eth009:45:133
2594899276,4cyclictest0-21swapper/210:05:142
2594899272,14cyclictest27346irq/40-dwc2_hso08:10:132
2594899271,9cyclictest2590666ssh10:10:252
2594099278,10cyclictest0-21swapper/110:15:131
2594099273,10cyclictest11274-21ssh10:40:141
2595699263,16cyclictest35966irq/58-eth010:45:133
2595699263,16cyclictest0-21swapper/311:48:123
2594899263,9cyclictest0-21swapper/210:40:142
2594899262,15cyclictest0-21swapper/211:50:132
25948992616,7cyclictest0-21swapper/212:30:012
25948992610,6cyclictest23787-21sh10:45:252
2594899260,16cyclictest31420-21missed_timers09:55:242
2594099265,16cyclictest958-21ssh09:40:141
2594099262,15cyclictest12474-21ssh11:35:151
2593299269,11cyclictest0-21swapper/010:46:580
2593299264,15cyclictest753-21sendmail-mta07:30:140
2593299262,18cyclictest4961-21ssh11:48:110
2593299262,18cyclictest1102-21ssh12:05:130
25932992614,3cyclictest12-21ksoftirqd/009:00:190
2593299261,19cyclictest2453-21ssh12:23:110
2595699253,15cyclictest0-21swapper/310:05:133
2595699251,8cyclictest0-21swapper/309:20:133
25956992515,7cyclictest0-21swapper/312:10:123
25956992512,9cyclictest35966irq/58-eth011:50:133
2595699251,16cyclictest0-21swapper/310:50:173
2594899259,13cyclictest18094-21kthreadcore10:25:202
25948992513,9cyclictest0-21swapper/211:46:142
25948992510,11cyclictest19746-21/usr/sbin/munin09:30:302
2593299257,4cyclictest131rcu_preempt11:10:140
25932992512,4cyclictest12-21ksoftirqd/010:28:450
2594899242,6cyclictest0-21swapper/211:29:112
2594899242,15cyclictest0-21swapper/208:25:042
2594899241,7cyclictest2176866ssh11:56:572
25948992412,9cyclictest17443-21ssh11:35:402
2594099242,17cyclictest19484-21ssh12:32:111
2594099242,16cyclictest6510-21ssh09:25:161
2594099241,19cyclictest24638-21sh12:00:131
2593299247,4cyclictest0-21swapper/009:11:490
2593299241,7cyclictest3261666ssh12:04:230
25932992412,5cyclictest12-21ksoftirqd/008:20:120
25932992411,3cyclictest12-21ksoftirqd/007:10:210
25932992410,7cyclictest0-21swapper/011:56:030
2595699236,10cyclictest35966irq/58-eth011:10:153
2594899235,13cyclictest9095-21ssh11:15:142
25948992312,9cyclictest2957-21ssh12:23:482
25948992312,9cyclictest20870-21ssh09:15:112
25948992312,8cyclictest0-21swapper/207:45:272
25948992312,4cyclictest0-21swapper/209:21:552
2594899231,15cyclictest0-21swapper/207:50:132
2594899230,10cyclictest27419-21ls08:15:282
2594099236,12cyclictest21-21ksoftirqd/111:02:111
2594099235,10cyclictest17406-21kworker/1:207:40:201
2594099232,15cyclictest1-21systemd12:25:391
2594099231,18cyclictest32652-21ls11:10:151
2593299239,3cyclictest12-21ksoftirqd/010:40:190
2593299233,14cyclictest7769-21ssh10:20:140
2593299232,16cyclictest32735-21ssh10:50:230
25932992312,3cyclictest12-21ksoftirqd/010:55:510
25932992311,3cyclictest12-21ksoftirqd/009:50:160
2593299231,11cyclictest10079-21tr07:25:220
25932992310,9cyclictest12-21ksoftirqd/008:35:210
25932992310,7cyclictest0-21swapper/012:12:520
25932992310,3cyclictest12-21ksoftirqd/007:50:160
25932992310,10cyclictest12-21ksoftirqd/012:16:100
25932992310,10cyclictest12-21ksoftirqd/012:16:090
2595699227,10cyclictest0-21swapper/311:32:123
2595699222,11cyclictest35966irq/58-eth012:00:123
2594899229,4cyclictest24855-21sh12:00:172
2594899225,7cyclictest131rcu_preempt07:25:292
2594899222,11cyclictest10895-21ps08:35:182
25948992211,8cyclictest0-21swapper/209:00:192
2594899221,15cyclictest27346irq/40-dwc2_hso07:20:132
25948992210,5cyclictest10183-21ssh09:25:292
2594899220,11cyclictest0-21swapper/209:40:202
2594099221,15cyclictest21395-21ls07:35:271
2594099221,15cyclictest21395-21ls07:35:261
2593299229,4cyclictest12-21ksoftirqd/010:14:480
2593299229,4cyclictest12-21ksoftirqd/007:55:220
2593299228,5cyclictest131rcu_preempt09:40:260
2593299227,3cyclictest12-21ksoftirqd/010:15:120
2593299227,11cyclictest12-21ksoftirqd/011:35:140
2593299225,5cyclictest0-21swapper/009:21:230
2593299224,3cyclictest131rcu_preempt07:35:100
2593299224,3cyclictest131rcu_preempt07:35:100
2593299223,4cyclictest131rcu_preempt08:40:200
2593299223,4cyclictest131rcu_preempt08:40:190
2593299223,11cyclictest0-21swapper/008:25:050
2593299222,16cyclictest14623-21sh11:00:140
25932992211,3cyclictest12-21ksoftirqd/009:25:300
25932992210,3cyclictest12-21ksoftirqd/009:39:120
2593299220,12cyclictest16994-21grep12:30:250
2595699213,7cyclictest0-21swapper/311:25:123
2595699211,13cyclictest0-21swapper/309:25:153
2594899218,10cyclictest26426-21grep07:10:172
2594899218,10cyclictest10426-21ntp_states09:05:272
2594899216,7cyclictest131rcu_preempt08:20:192
2594899216,5cyclictest16377-21kthreadcore08:05:212
2594899213,10cyclictest131rcu_preempt11:10:102
2594899212,9cyclictest2378-21ssh10:15:242
2594899211,9cyclictest22432-21grep09:50:222
25948992111,8cyclictest0-21swapper/208:55:262
25948992111,8cyclictest0-21swapper/208:50:312
25948992111,6cyclictest0-21swapper/208:45:142
2594899211,12cyclictest20189-21cut12:15:202
2594899211,12cyclictest20189-21cut12:15:192
2594899211,11cyclictest9399-21fschecks_count08:00:162
2594099214,10cyclictest0-21swapper/107:25:141
2594099212,15cyclictest73-21kswapd012:10:111
25940992114,5cyclictest0-21swapper/110:00:151
25940992112,5cyclictest24648-21proc_pri09:15:271
2593299219,3cyclictest12-21ksoftirqd/008:45:310
2593299218,3cyclictest12-21ksoftirqd/011:33:070
2593299218,10cyclictest27671-21ssh11:25:180
2593299218,10cyclictest12-21ksoftirqd/007:40:220
2593299212,10cyclictest12-21ksoftirqd/009:55:190
25932992113,5cyclictest0-21swapper/011:20:110
25932992110,3cyclictest12-21ksoftirqd/008:00:230
2595699208,9cyclictest0-21swapper/310:23:093
2595699202,13cyclictest35966irq/58-eth011:35:123
2594899206,6cyclictest27-21ksoftirqd/207:15:152
2594899201,10cyclictest15991-21irqrtprio08:40:192
2594899201,10cyclictest15991-21irqrtprio08:40:182
25948992010,8cyclictest0-21swapper/208:30:122
2594099208,6cyclictest0-21swapper/109:45:171
2594099207,10cyclictest823-21sed08:25:011
2594099204,5cyclictest131rcu_preempt07:20:021
25940992011,6cyclictest1-21systemd10:05:071
2594099201,13cyclictest0-21swapper/108:55:141
2594099201,12cyclictest17758-21ls11:20:021
2594099201,10cyclictest21256-21kthreadcore12:15:221
2594099201,10cyclictest21256-21kthreadcore12:15:211
2593299209,2cyclictest12-21ksoftirqd/008:05:220
2593299202,15cyclictest25052-21ntp_192.168.11411:40:250
2593299201,7cyclictest21532-21tr08:10:210
25932992013,4cyclictest12-21ksoftirqd/008:30:240
2595699199,8cyclictest0-21swapper/309:55:453
2595699193,11cyclictest0-21swapper/312:35:123
2595699193,11cyclictest0-21swapper/312:35:113
2595699192,11cyclictest0-21swapper/308:10:123
2595699192,11cyclictest0-21swapper/307:50:143
25956991911,6cyclictest0-21swapper/309:16:543
2595699191,13cyclictest0-21swapper/310:15:163
2595699191,12cyclictest0-21swapper/310:00:173
2595699191,12cyclictest0-21swapper/308:35:113
25956991910,6cyclictest0-21swapper/311:55:303
25956991910,6cyclictest0-21swapper/311:20:183
2594899192,14cyclictest17668-21cpu07:35:132
2594899192,14cyclictest17668-21cpu07:35:122
2594899191,9cyclictest22790-21if_eth007:40:172
2594099197,9cyclictest6104-21basename09:25:001
2594099191,8cyclictest15559-21/usr/sbin/munin09:30:161
2593299198,8cyclictest12-21ksoftirqd/008:50:120
2593299197,9cyclictest7113-21grep07:20:270
2593299197,3cyclictest12-21ksoftirqd/010:00:140
25932991912,4cyclictest12-21ksoftirqd/008:55:200
2595699187,7cyclictest0-21swapper/312:05:133
2595699183,9cyclictest0-21swapper/311:40:273
2595699182,10cyclictest0-21swapper/309:05:153
2595699182,10cyclictest0-21swapper/308:50:133
2595699182,10cyclictest0-21swapper/308:25:053
2595699182,10cyclictest0-21swapper/307:40:143
2595699181,9cyclictest35966irq/58-eth012:15:133
2595699181,9cyclictest35966irq/58-eth012:15:133
2595699181,12cyclictest0-21swapper/309:35:023
2595699181,11cyclictest0-21swapper/309:50:193
2595699181,11cyclictest0-21swapper/308:00:173
25956991810,6cyclictest0-21swapper/310:25:183
25956991810,6cyclictest0-21swapper/309:10:103
2594899180,7cyclictest25961-21latency_hist07:10:012
2594099183,4cyclictest21-21ksoftirqd/108:15:271
2594099182,14cyclictest11641-21latency_hist09:10:011
2594099181,9cyclictest20801-21tr08:45:191
2594099181,9cyclictest0-21swapper/110:35:191
2594099181,7cyclictest16029-21grep09:10:291
25940991811,4cyclictest7732-21/usr/sbin/munin10:20:261
2594099181,14cyclictest26095-21apt-get07:10:131
2594099181,13cyclictest15152-21idleruntime-cro08:40:001
2594099181,12cyclictest844-21munin-node11:45:151
2594099181,11cyclictest17384-21cut08:40:211
2594099181,11cyclictest17384-21cut08:40:201
2593299185,3cyclictest12-21ksoftirqd/008:15:160
25932991812,3cyclictest12-21ksoftirqd/007:45:210
2595699172,10cyclictest0-21swapper/308:20:133
2595699172,10cyclictest0-21swapper/307:55:133
2595699172,10cyclictest0-21swapper/307:10:133
2595699171,13cyclictest0-21swapper/308:15:243
2594099178,7cyclictest4224-21cron07:55:011
2594099175,3cyclictest131rcu_preempt08:50:171
2594099174,4cyclictest16672-21grep07:30:301
2594099173,4cyclictest21-21ksoftirqd/108:00:111
2594099173,12cyclictest6053-21kthreadcore07:55:211
2594099171,6cyclictest131rcu_preempt08:10:271
2594099171,5cyclictest25223-21kthreadcore12:35:231
2594099171,5cyclictest25223-21kthreadcore12:35:221
25940991711,4cyclictest0-21swapper/109:00:121
2594099171,13cyclictest1187-21df12:05:141
25940991710,4cyclictest6853-21cpu11:50:131
2595699161,9cyclictest0-21swapper/307:30:163
2595699161,12cyclictest0-21swapper/308:55:283
2594099166,2cyclictest131rcu_preempt08:05:191
2594099162,10cyclictest0-21swapper/107:45:141
2594099161,12cyclictest3848-21cut07:20:181
2595699152,9cyclictest0-21swapper/307:45:133
2595699151,8cyclictest0-21swapper/309:00:133
2594099156,7cyclictest607-21klogd07:10:011
2594099151,11cyclictest1374-21sed08:25:191
2595699149,3cyclictest0-21swapper/307:25:163
2595699141,7cyclictest0-21swapper/307:15:173
2593299142,9cyclictest25850-21latency_hist07:10:020
2595699131,7cyclictest0-21swapper/308:45:173
2595699131,10cyclictest0-21swapper/308:30:203
2595699128,3cyclictest0-21swapper/308:09:483
2595699127,3cyclictest0-21swapper/307:20:143
2595699121,8cyclictest0-21swapper/307:35:163
2595699121,8cyclictest0-21swapper/307:35:163
2595699112,5cyclictest0-21swapper/308:40:143
2595699112,5cyclictest0-21swapper/308:40:133
2595699101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional