You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-11 - 20:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat Jul 11, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6907997733,29cyclictest35966irq/58-eth009:50:153
6877996010,4cyclictest131rcu_preempt10:55:150
687799598,8cyclictest131rcu_preempt07:15:130
687799552,8cyclictest131rcu_preempt09:30:140
687799552,8cyclictest131rcu_preempt09:30:130
687799544,39cyclictest8853-21ssh12:00:140
6907995141,6cyclictest35966irq/58-eth011:15:143
688899501,41cyclictest9790-21ssh09:55:131
687799492,37cyclictest24957-21ssh11:15:130
690799443,26cyclictest35966irq/58-eth010:15:143
689799443,32cyclictest1504-21ssh11:20:122
689799443,32cyclictest1504-21ssh11:20:112
687799442,33cyclictest12192-21ssh10:50:140
689799431,15cyclictest28828-21latency_hist08:40:022
688899438,29cyclictest21-21ksoftirqd/109:50:141
687799434,32cyclictest30297-21ssh11:00:140
6877994215,18cyclictest12-21ksoftirqd/010:25:130
689799413,12cyclictest27346irq/40-dwc2_hso10:35:142
689799402,20cyclictest16034-21ntpq12:20:242
689799401,9cyclictest14344-21latency_hist08:25:022
6907993927,7cyclictest35966irq/58-eth009:40:143
6907993914,18cyclictest35966irq/58-eth010:30:153
688899392,22cyclictest27946-21ssh10:05:141
689799382,9cyclictest9554-21sendmail_mailqu08:50:272
689799381,10cyclictest13369-21tune2fs07:50:172
690799374,20cyclictest35966irq/58-eth010:45:143
690799374,20cyclictest35966irq/58-eth010:45:133
689799372,27cyclictest6938-21ssh11:05:152
689799372,26cyclictest27346irq/40-dwc2_hso09:45:162
6897993710,7cyclictest17574-21latency_hist12:05:012
687799373,21cyclictest12-21ksoftirqd/008:25:130
687799372,28cyclictest7070-21rm12:35:120
687799371,29cyclictest17965-21sh12:05:140
6877993710,5cyclictest131rcu_preempt09:15:130
690799362,24cyclictest35966irq/58-eth008:40:133
689799367,11cyclictest1652-21kthreadcore09:50:192
688899367,6cyclictest331-21unixbench_multi11:35:291
688899363,21cyclictest4573-21ssh10:10:141
688899362,10cyclictest22107-21kthreadcore10:55:211
687799361,21cyclictest12-21ksoftirqd/008:20:120
6877993612,16cyclictest151rcuc/012:20:140
689799355,11cyclictest23134-1kworker/u9:210:40:202
689799354,23cyclictest8492-21ssh10:30:152
687799351,28cyclictest18845-21ssh10:00:130
689799340,9cyclictest4727-21latency_hist08:15:022
689799340,9cyclictest4727-21latency_hist08:15:022
688899342,10cyclictest31387-21kthreadcore09:30:221
688899342,10cyclictest31387-21kthreadcore09:30:211
688899341,2cyclictest131rcu_preempt07:15:151
687799343,23cyclictest26937-21ssh12:10:140
687799342,16cyclictest25416-21ssh09:10:140
6907993315,12cyclictest35966irq/58-eth010:05:143
689799331,9cyclictest27553-21latency_hist08:05:022
689799331,25cyclictest10530-21ssh11:25:142
689799330,15cyclictest22271-21kthreadcore11:30:232
688899332,19cyclictest1523-21ssh11:20:131
688899332,19cyclictest1523-21ssh11:20:131
688899331,11cyclictest17049-21grep07:50:281
687799339,6cyclictest131rcu_preempt11:05:140
6907993213,14cyclictest35966irq/58-eth012:10:153
688899327,6cyclictest1-21systemd08:30:321
688899321,11cyclictest25164-21find07:25:271
690799316,12cyclictest35966irq/58-eth009:20:133
690799314,16cyclictest0-21swapper/310:40:133
6907993113,12cyclictest0-21swapper/311:51:183
689799312,6cyclictest0-21swapper/209:28:142
688899312,9cyclictest10753-21ntp_states12:35:271
688899312,6cyclictest0-21swapper/112:20:251
688899311,14cyclictest30616-21ssh12:30:171
6877993116,9cyclictest151rcuc/010:40:140
689799302,21cyclictest0-21swapper/207:55:152
688899301,10cyclictest23265-21/usr/sbin/munin11:50:191
688899301,10cyclictest17488-21ssh10:35:141
688899301,10cyclictest14163-21ntp_kernel_err11:25:241
687799302,22cyclictest4418-21ssh11:39:260
687799302,22cyclictest20111-21ssh10:17:240
690799297,14cyclictest35966irq/58-eth011:55:143
690799297,14cyclictest35966irq/58-eth011:55:143
690799292,18cyclictest0-21swapper/310:50:153
689799292,21cyclictest0-21swapper/212:15:142
688899297,5cyclictest19798-21sh11:30:181
688899292,6cyclictest0-21swapper/109:28:151
688899291,7cyclictest13658-21ssh10:15:161
688899291,24cyclictest6910-21ssh11:05:131
688899291,10cyclictest2401-21cut08:10:221
688899291,10cyclictest2401-21cut08:10:221
690799283,18cyclictest0-21swapper/311:29:253
690799283,16cyclictest0-21swapper/311:30:243
690799283,16cyclictest0-21swapper/311:22:263
690799283,16cyclictest0-21swapper/311:22:253
690799283,15cyclictest35966irq/58-eth011:05:133
689799284,17cyclictest536-21usb-storage09:30:052
689799284,17cyclictest536-21usb-storage09:30:042
688899283,18cyclictest14382-21ssh11:45:131
688899281,21cyclictest5962-21ssh09:35:131
6888992810,4cyclictest0-21swapper/109:14:051
6877992810,11cyclictest0-21swapper/009:40:130
690799273,18cyclictest0-21swapper/312:23:253
690799273,15cyclictest0-21swapper/310:20:253
6907992712,9cyclictest35966irq/58-eth009:30:153
6907992712,9cyclictest35966irq/58-eth009:30:153
690799271,10cyclictest35966irq/58-eth008:30:123
689799271,15cyclictest4582-21ssh10:10:162
688899278,3cyclictest22673-21usage_EthcTask12:05:341
688899271,10cyclictest7063-21apt-get07:10:151
688899271,10cyclictest31954-21kthreadcore11:00:211
687799272,20cyclictest0-21swapper/011:47:250
687799272,19cyclictest26400-21ssh10:20:250
690799263,17cyclictest0-21swapper/309:48:253
690799263,15cyclictest0-21swapper/311:39:263
689799263,17cyclictest27346irq/40-dwc2_hso07:35:142
689799261,9cyclictest30577-21ssh11:53:242
6897992618,3cyclictest27-21ksoftirqd/209:55:142
688899267,9cyclictest21-21ksoftirqd/109:45:151
688899262,5cyclictest0-21swapper/112:01:241
688899261,9cyclictest6881-21ntp_states10:45:271
688899261,9cyclictest6881-21ntp_states10:45:261
690799258,11cyclictest0-21swapper/309:57:243
690799251,19cyclictest35966irq/58-eth010:25:143
689799253,16cyclictest27346irq/40-dwc2_hso08:55:142
689799252,19cyclictest22407-21ssh11:49:252
6897992518,3cyclictest27-21ksoftirqd/209:40:142
689799251,6cyclictest1277266ssh11:43:362
688899251,2cyclictest131rcu_preempt09:15:141
688899250,10cyclictest21014-21cut10:55:021
687799254,14cyclictest151rcuc/012:30:140
687799251,19cyclictest3195-21ssh10:45:150
687799251,19cyclictest3195-21ssh10:45:140
689799246,14cyclictest8186-21cron07:45:012
689799244,14cyclictest27346irq/40-dwc2_hso07:30:152
689799242,17cyclictest0-21swapper/210:04:242
689799241,6cyclictest63866ssh11:19:292
689799241,6cyclictest2052766rm10:54:402
688899242,19cyclictest22711-21ssh10:20:101
688899240,11cyclictest16936-21cut09:00:191
687799249,12cyclictest12-21ksoftirqd/010:31:190
687799249,10cyclictest630-21Async09:36:310
687799242,18cyclictest32317-21ssh11:55:150
687799242,18cyclictest32317-21ssh11:55:150
6877992416,5cyclictest0-21swapper/009:20:140
689799233,13cyclictest27346irq/40-dwc2_hso09:35:142
689799232,16cyclictest0-21swapper/210:19:252
689799231,7cyclictest101266ssh12:12:312
689799231,6cyclictest305066ssh10:08:422
689799231,6cyclictest11090-21apt-get09:20:122
689799231,4cyclictest685066ssh10:45:272
689799231,4cyclictest685066ssh10:45:262
6897992312,9cyclictest2239-21ssh11:36:492
689799231,14cyclictest2124866ssh12:25:072
689799231,14cyclictest190266ssh12:30:292
688899231,4cyclictest1213566ssh11:42:511
6888992311,9cyclictest22567-21cron08:00:021
687799231,17cyclictest25002-21diskmemload11:30:140
6877992311,4cyclictest12-21ksoftirqd/009:58:530
690799222,13cyclictest0-21swapper/309:25:153
690799221,15cyclictest0-21swapper/310:10:153
689799227,3cyclictest131rcu_preempt10:20:162
689799221,9cyclictest12164-21irqcore07:15:192
689799221,6cyclictest482666ssh10:26:072
6897992211,9cyclictest23506-21sshd12:06:202
689799221,11cyclictest21571-21sshd07:25:122
6897992210,5cyclictest4894-21ssh11:03:032
6888992212,7cyclictest15902-21apt-get11:10:121
687799227,11cyclictest12-21ksoftirqd/009:25:150
687799223,11cyclictest0-21swapper/008:05:110
6877992214,3cyclictest0-21swapper/008:00:130
6877992212,3cyclictest12-21ksoftirqd/012:25:300
690799212,12cyclictest35966irq/58-eth008:25:123
6907992111,4cyclictest0-21swapper/310:55:143
689799214,9cyclictest7077-21kthreadcore08:15:212
689799212,11cyclictest3435-21kthreadcore08:45:232
689799211,6cyclictest2183766ssh11:11:452
689799211,13cyclictest26330-21kthreadcore09:10:212
688899216,3cyclictest131rcu_preempt11:15:221
688899214,9cyclictest0-21swapper/108:35:111
688899211,8cyclictest22789-21find10:00:281
688899211,7cyclictest29987-21tr12:10:231
688899211,15cyclictest31857-21sh10:25:151
687799212,15cyclictest5936-21apt-get09:35:130
687799212,14cyclictest0-21swapper/010:05:240
6877992111,7cyclictest0-21swapper/011:20:260
6877992111,7cyclictest0-21swapper/011:20:250
6877992111,7cyclictest0-21swapper/010:14:250
6877992110,2cyclictest12-21ksoftirqd/007:25:020
687799210,17cyclictest3518-21ls07:40:010
690799201,13cyclictest0-21swapper/307:25:133
690799201,12cyclictest35966irq/58-eth011:45:143
6907992010,8cyclictest0-21swapper/312:06:123
689799209,4cyclictest0-21swapper/212:35:122
689799202,9cyclictest14612-21grep08:25:162
689799202,15cyclictest27346irq/40-dwc2_hso11:55:132
689799202,15cyclictest27346irq/40-dwc2_hso11:55:122
689799201,9cyclictest27753-21fschecks_count08:05:162
689799201,9cyclictest11519-21cut07:45:252
689799201,7cyclictest24376-21kthreadcore10:55:242
689799201,5cyclictest773866ssh09:16:292
689799201,16cyclictest23873-21latency_hist08:35:012
689799200,11cyclictest29362-21irqrtprio08:40:172
688899207,10cyclictest26539-21grep07:30:171
688899202,2cyclictest131rcu_preempt08:50:161
688899201,6cyclictest14755-21ntp_192.168.11409:20:231
688899201,3cyclictest131rcu_preempt09:40:171
688899201,13cyclictest1237-21sshd08:45:151
688899201,12cyclictest15392-21latency_hist09:00:021
687799209,3cyclictest12-21ksoftirqd/009:54:250
687799207,7cyclictest12-21ksoftirqd/012:19:530
687799201,9cyclictest25043-21rm09:10:000
6877992010,8cyclictest1234-21apt-get08:45:140
690799197,9cyclictest35966irq/58-eth009:15:153
690799193,11cyclictest0-21swapper/312:01:243
690799193,11cyclictest0-21swapper/311:41:253
690799193,11cyclictest0-21swapper/311:11:253
690799193,11cyclictest0-21swapper/310:00:543
690799192,10cyclictest0-21swapper/307:45:163
690799192,10cyclictest0-21swapper/307:15:163
690799191,13cyclictest0-21swapper/312:25:133
690799191,12cyclictest0-21swapper/308:10:153
690799191,12cyclictest0-21swapper/308:10:143
6907991910,6cyclictest0-21swapper/311:02:403
689799195,11cyclictest22264-21kthreadcore09:05:202
689799192,9cyclictest9378-21kthreadcore07:10:222
689799191,15cyclictest15942-21awk09:00:182
689799190,11cyclictest17328-21cut07:20:192
688899198,9cyclictest31558-21grep07:35:191
688899192,13cyclictest21352-21apt-get12:25:131
688899191,9cyclictest0-21swapper/110:41:581
688899191,4cyclictest21-21ksoftirqd/109:05:181
6888991911,6cyclictest0-21swapper/112:15:171
688899191,12cyclictest8499-21sh10:30:141
6877991912,4cyclictest12-21ksoftirqd/011:40:220
6877991911,6cyclictest24790-21sshd10:38:240
687799191,10cyclictest24028-21kernelversion11:50:190
690799189,4cyclictest0-21swapper/309:10:153
690799182,9cyclictest35966irq/58-eth012:35:133
690799182,10cyclictest0-21swapper/308:35:133
690799182,10cyclictest0-21swapper/308:20:143
690799182,10cyclictest0-21swapper/307:55:143
690799182,10cyclictest0-21swapper/307:40:143
690799182,10cyclictest0-21swapper/307:35:143
690799181,12cyclictest0-21swapper/307:20:283
6907991810,6cyclictest35966irq/58-eth012:15:123
6888991810,6cyclictest26368-21sendmail_mailqu08:00:271
687799187,8cyclictest24537-21tr08:35:180
687799187,7cyclictest151rcuc/008:30:110
687799183,9cyclictest151rcuc/011:10:130
6877991812,3cyclictest12-21ksoftirqd/008:10:150
6877991812,3cyclictest12-21ksoftirqd/008:10:140
6877991811,4cyclictest12-21ksoftirqd/009:00:120
6877991811,4cyclictest12-21ksoftirqd/007:10:230
6877991811,4cyclictest10493-21/usr/sbin/munin11:25:230
6877991810,6cyclictest10570-21sshd08:54:150
690799179,3cyclictest0-21swapper/312:30:043
690799178,6cyclictest35966irq/58-eth009:35:143
690799175,9cyclictest0-21swapper/310:35:033
690799172,9cyclictest0-21swapper/308:55:153
690799172,10cyclictest0-21swapper/307:10:153
688899176,9cyclictest16359-21kthreadcore08:25:201
688899172,12cyclictest4040-21irqcore07:40:171
688899171,4cyclictest21-21ksoftirqd/107:45:331
6888991711,4cyclictest2473-21cut11:55:231
6888991711,4cyclictest2473-21cut11:55:221
687799177,2cyclictest12-21ksoftirqd/008:00:010
687799172,10cyclictest0-21swapper/008:40:130
6877991711,4cyclictest12-21ksoftirqd/007:25:260
6877991711,4cyclictest0-21swapper/008:55:160
6877991711,4cyclictest0-21swapper/007:40:150
687799171,13cyclictest7947-21cut08:15:220
690799162,9cyclictest0-21swapper/308:45:153
690799161,10cyclictest0-21swapper/307:55:023
688899163,10cyclictest7009-21ls07:10:021
688899163,10cyclictest30941-21tr08:40:201
688899162,10cyclictest4798-21apt-get08:15:121
688899161,11cyclictest9616-21find08:20:121
687799164,10cyclictest30320-21unixbench_singl07:30:310
687799163,10cyclictest17010-21grep07:50:290
687799161,6cyclictest9720-21kthreadcore07:45:220
690799155,7cyclictest0-21swapper/308:15:293
690799152,8cyclictest0-21swapper/308:00:163
690799151,8cyclictest0-21swapper/308:50:173
690799151,7cyclictest0-21swapper/307:35:023
688899151,11cyclictest28412-21cut08:05:201
688899151,11cyclictest17430-21kthreadcore07:20:181
690799141,9cyclictest0-21swapper/309:05:133
687799141,10cyclictest6980-21latency_hist07:10:000
690799134,7cyclictest0-21swapper/309:05:013
689799130,9cyclictest6817-21latency_hist07:10:012
690799126,4cyclictest35966irq/58-eth008:05:293
690799121,9cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional