You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 09:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Mon Dec 08, 2025 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7535997328,8cyclictest21-21ksoftirqd/120:55:151
7535997327,8cyclictest21-21ksoftirqd/123:50:141
752099626,25cyclictest12-21ksoftirqd/023:45:130
753599553,39cyclictest16184-21ssh23:00:141
754399523,25cyclictest2576-21ssh21:15:142
753599521,41cyclictest818-21ssh00:30:121
7543994925,12cyclictest19218-21ssh21:25:132
755199472,27cyclictest29566irq/58-eth022:40:143
753599466,32cyclictest25392-21ssh00:25:171
753599462,6cyclictest131rcu_preempt20:35:151
753599452,7cyclictest131rcu_preempt00:00:161
753599452,7cyclictest131rcu_preempt00:00:161
752099459,25cyclictest12-21ksoftirqd/023:40:140
752099452,36cyclictest25624-21ssh23:25:150
7535994427,10cyclictest26231-21diskmemload22:00:151
752099433,32cyclictest987-21sshd20:10:160
752099431,35cyclictest0-21swapper/022:05:140
7551994231,8cyclictest29566irq/58-eth000:15:163
754399421,24cyclictest31909-21ssh22:50:132
753599424,33cyclictest2803-21ssh00:10:121
7543994120,13cyclictest0-21swapper/223:40:142
753599402,27cyclictest0-21swapper/121:40:131
7535993923,5cyclictest21-21ksoftirqd/119:45:121
752099393,25cyclictest17871-21ssh00:20:120
7543993823,4cyclictest27-21ksoftirqd/220:10:162
754399372,28cyclictest10029-21ssh23:53:442
752099371,30cyclictest13977-21ssh22:20:140
7543993614,10cyclictest261rcuc/222:30:152
753599361,28cyclictest10903-21ssh21:20:131
752099362,21cyclictest12-21ksoftirqd/019:10:160
753599351,6cyclictest7149-21apt-get20:50:141
7535993514,16cyclictest21-21ksoftirqd/121:30:151
7520993515,15cyclictest12-21ksoftirqd/022:30:150
755199341,18cyclictest29566irq/58-eth021:10:153
753599345,20cyclictest17228-21ssh23:20:141
752099345,4cyclictest131rcu_preempt00:00:160
752099345,4cyclictest131rcu_preempt00:00:160
754399332,18cyclictest7702-21sshd19:10:162
752099333,21cyclictest15166-21ssh22:40:140
754399325,4cyclictest131rcu_preempt22:20:012
753599322,22cyclictest6469-21sh22:53:331
7535993218,6cyclictest21-21ksoftirqd/123:35:131
752099322,22cyclictest8827-21ssh23:15:140
7520993211,9cyclictest7511-21cyclictest23:05:190
7551993112,14cyclictest29566irq/58-eth023:55:153
754399312,13cyclictest32561-21ssh23:46:342
753599313,21cyclictest6734-21ssh22:35:141
753599313,21cyclictest6734-21ssh22:35:141
753599311,20cyclictest4568-21ssh21:55:131
752099311,7cyclictest12-21ksoftirqd/022:45:130
754399302,17cyclictest932-21snmpd19:55:162
7543993016,8cyclictest27-21ksoftirqd/220:50:142
7543993011,12cyclictest27-21ksoftirqd/223:36:222
7535993015,6cyclictest21-21ksoftirqd/119:15:141
752099302,14cyclictest0-21swapper/021:15:140
755199292,16cyclictest29566irq/58-eth000:00:163
755199292,16cyclictest29566irq/58-eth000:00:163
754399293,21cyclictest0-21swapper/222:07:332
754399293,19cyclictest0-21swapper/223:15:142
7543992914,9cyclictest0-21swapper/221:50:132
754399291,21cyclictest0-21swapper/222:55:162
753599298,5cyclictest131rcu_preempt21:50:191
752099298,7cyclictest0-21swapper/023:54:240
755199283,18cyclictest0-21swapper/322:54:333
7551992814,8cyclictest29566irq/58-eth021:40:143
754399288,8cyclictest26346irq/40-dwc2_hso00:35:132
754399288,6cyclictest27-21ksoftirqd/222:47:032
754399285,12cyclictest27-21ksoftirqd/221:55:202
754399282,10cyclictest24390-21ssh00:23:542
7543992817,8cyclictest261rcuc/200:15:142
753599282,19cyclictest14067-21ssh22:57:331
752099284,17cyclictest8095-21ssh00:35:130
752099282,19cyclictest9548-21ssh00:13:440
7520992814,5cyclictest12-21ksoftirqd/022:15:270
752099281,19cyclictest3551-21ssh21:35:140
755199274,10cyclictest29566irq/58-eth000:10:123
754399275,12cyclictest26346irq/40-dwc2_hso21:05:162
754399274,15cyclictest26346irq/40-dwc2_hso21:10:142
754399272,19cyclictest0-21swapper/220:15:162
752099279,13cyclictest30732-1kworker/u9:122:54:330
752099278,11cyclictest131rcu_preempt23:10:130
752099277,3cyclictest131rcu_preempt22:35:130
752099277,3cyclictest131rcu_preempt22:35:130
752099273,17cyclictest7894-21ssh22:55:160
7520992718,3cyclictest0-21swapper/000:05:140
752099271,20cyclictest25401-21ssh00:25:170
755199262,18cyclictest0-21swapper/322:02:363
755199262,12cyclictest29566irq/58-eth023:00:143
755199261,15cyclictest0-21swapper/321:30:153
754399268,11cyclictest27-21ksoftirqd/221:39:042
754399267,3cyclictest131rcu_preempt21:20:152
754399265,5cyclictest27-21ksoftirqd/222:25:152
754399264,15cyclictest27-21ksoftirqd/200:00:272
754399264,15cyclictest27-21ksoftirqd/200:00:272
754399264,14cyclictest27-21ksoftirqd/222:12:032
754399262,15cyclictest454-21ssh23:10:132
754399261,17cyclictest25455-21ls00:25:172
753599263,15cyclictest0-21swapper/120:00:141
753599262,18cyclictest26645-21ssh21:10:151
753599262,18cyclictest22415-21ssh22:25:151
753599261,21cyclictest19208-21ssh21:25:131
753599260,6cyclictest131rcu_preempt23:30:141
7520992614,7cyclictest151rcuc/019:40:130
7520992614,7cyclictest151rcuc/019:40:120
7520992613,5cyclictest131rcu_preempt21:05:170
7520992610,9cyclictest0-21swapper/023:55:520
755199253,13cyclictest0-21swapper/323:08:323
755199252,19cyclictest29566irq/58-eth022:10:133
755199252,18cyclictest0-21swapper/322:22:393
755199251,14cyclictest0-21swapper/319:25:143
754399257,11cyclictest27-21ksoftirqd/223:28:522
754399257,10cyclictest131rcu_preempt20:40:202
754399252,18cyclictest26346irq/40-dwc2_hso00:05:142
754399252,15cyclictest11932-21ssh21:40:142
7543992514,4cyclictest1526-21ssh23:30:142
7543992510,8cyclictest27-21ksoftirqd/222:38:492
7543992510,8cyclictest27-21ksoftirqd/222:38:492
7535992512,10cyclictest21-21ksoftirqd/119:35:151
752099258,9cyclictest0-21swapper/022:27:110
752099258,4cyclictest131rcu_preempt21:20:150
752099252,18cyclictest20321-21ssh21:45:160
755199243,18cyclictest29566irq/58-eth020:35:163
7551992412,10cyclictest0-21swapper/323:30:233
754399248,5cyclictest27-21ksoftirqd/223:22:502
754399247,10cyclictest0-21swapper/222:20:122
754399245,12cyclictest27-21ksoftirqd/219:15:132
754399244,15cyclictest26346irq/40-dwc2_hso20:30:132
754399241,10cyclictest31718-21latency_hist19:35:012
754399241,10cyclictest12593-21kthreadcore23:55:212
7543992410,11cyclictest4097-21cron19:40:012
753599248,8cyclictest7511-21cyclictest20:20:151
753599245,12cyclictest1402-21sshd23:48:221
753599243,16cyclictest201rcuc/121:45:151
753599241,19cyclictest10568-21ssh00:15:161
754399239,11cyclictest27-21ksoftirqd/220:55:342
754399238,4cyclictest27-21ksoftirqd/221:32:352
754399231,9cyclictest17201-21kthreadcore20:25:212
7543992312,8cyclictest27-21ksoftirqd/200:10:192
7543992311,9cyclictest1-21systemd20:37:222
7543992311,4cyclictest27-21ksoftirqd/222:00:212
7535992313,7cyclictest26226-21cron21:10:011
753599231,17cyclictest20267-21apt-get20:30:131
753599231,17cyclictest11312-21ssh23:55:161
752099239,7cyclictest0-21swapper/021:30:420
752099234,4cyclictest131rcu_preempt23:20:140
752099233,18cyclictest12-21ksoftirqd/023:30:140
752099233,16cyclictest26231-21diskmemload22:10:140
752099232,16cyclictest12-21ksoftirqd/019:55:150
7520992310,5cyclictest0-21swapper/021:27:470
754399222,11cyclictest131rcu_preempt22:44:042
754399221,9cyclictest0-21swapper/200:31:172
7543992218,2cyclictest0-21swapper/223:00:142
7543992212,8cyclictest0-21swapper/223:05:572
754399221,13cyclictest0-21swapper/220:45:142
753599225,5cyclictest21976-21kworker/1:023:15:021
753599221,10cyclictest1327-21latency_hist23:30:021
752099229,4cyclictest12-21ksoftirqd/022:00:010
752099228,4cyclictest12-21ksoftirqd/019:25:290
752099225,7cyclictest131rcu_preempt21:44:550
752099222,16cyclictest16821-21apt-get21:00:130
752099221,6cyclictest0-21swapper/019:31:070
7520992212,4cyclictest0-21swapper/020:45:190
7520992211,3cyclictest12-21ksoftirqd/019:50:010
7520992210,4cyclictest0-21swapper/021:52:220
755199212,13cyclictest0-21swapper/300:06:443
754399212,13cyclictest0-21swapper/221:00:142
7543992110,4cyclictest27-21ksoftirqd/221:45:022
753599218,3cyclictest21-21ksoftirqd/123:15:181
753599216,4cyclictest131rcu_preempt23:05:271
7535992114,4cyclictest11848-21sendmail_mailtr00:35:281
7535992112,6cyclictest7607-21latency_hist19:10:021
753599211,18cyclictest18055-21fschecks_time00:20:161
753599211,17cyclictest20140-21ls20:30:001
753599211,16cyclictest984-21apt-get20:10:161
7535992111,6cyclictest15895-21kthreadcore22:40:191
753599211,14cyclictest4343-21find19:40:131
753599211,14cyclictest4343-21find19:40:131
752099219,3cyclictest12-21ksoftirqd/000:15:150
752099217,7cyclictest12-21ksoftirqd/000:33:460
752099212,13cyclictest842-21munin-node20:50:160
752099211,8cyclictest0-21swapper/020:20:240
7520992110,3cyclictest12-21ksoftirqd/023:03:240
755199203,12cyclictest0-21swapper/321:35:143
755199203,11cyclictest0-21swapper/323:27:513
755199203,11cyclictest0-21swapper/323:22:523
755199201,9cyclictest29566irq/58-eth000:30:113
7551992011,6cyclictest29566irq/58-eth022:25:153
754399207,5cyclictest27-21ksoftirqd/219:25:152
754399205,7cyclictest131rcu_preempt20:25:012
754399205,7cyclictest131rcu_preempt20:00:272
754399203,6cyclictest0-21swapper/219:45:132
754399201,14cyclictest26346irq/40-dwc2_hso19:40:132
754399201,14cyclictest26346irq/40-dwc2_hso19:40:132
754399201,11cyclictest20822-21wc19:20:262
754399201,11cyclictest17852-21grep19:50:312
753599201,9cyclictest6576-21timerwakeupswit21:15:301
753599201,15cyclictest3663-21sh21:35:161
753599201,10cyclictest766-21ntp_192.168.11422:10:251
753599201,10cyclictest31947-21missed_timers20:05:241
752099209,2cyclictest12-21ksoftirqd/020:05:010
752099208,4cyclictest0-21swapper/019:15:130
752099201,3cyclictest131rcu_preempt22:00:220
752099201,16cyclictest26610-21find21:10:140
755199198,9cyclictest0-21swapper/321:59:113
755199193,9cyclictest0-21swapper/323:48:223
755199193,11cyclictest0-21swapper/323:35:343
755199193,11cyclictest0-21swapper/323:17:343
755199193,11cyclictest0-21swapper/322:55:333
755199193,11cyclictest0-21swapper/300:25:173
755199193,10cyclictest0-21swapper/323:50:343
755199193,10cyclictest0-21swapper/323:40:343
755199192,11cyclictest0-21swapper/323:12:483
755199192,11cyclictest0-21swapper/320:25:153
7551991911,5cyclictest29566irq/58-eth021:20:153
755199191,12cyclictest0-21swapper/321:50:133
755199191,11cyclictest0-21swapper/322:35:133
755199191,11cyclictest0-21swapper/322:35:133
754399191,9cyclictest29482-21kthreadcore20:05:202
753599193,9cyclictest0-21swapper/100:08:441
753599193,10cyclictest0-21swapper/119:25:141
753599192,11cyclictest20293-21ssh22:22:391
7535991912,5cyclictest15238-1kworker/u9:223:40:181
7535991912,4cyclictest6371-21latency_hist22:35:011
753599191,15cyclictest6624-21kthreadcore22:15:201
753599191,10cyclictest27292-21fschecks_count19:30:191
752099198,3cyclictest12-21ksoftirqd/020:05:320
752099198,2cyclictest12-21ksoftirqd/020:55:300
752099191,6cyclictest0-21swapper/020:18:320
7520991912,5cyclictest7511-21cyclictest19:10:010
7520991912,5cyclictest0-21swapper/023:35:140
752099191,14cyclictest12-21ksoftirqd/019:50:150
7520991911,2cyclictest12-21ksoftirqd/020:25:310
755199189,6cyclictest0-21swapper/322:49:053
755199189,5cyclictest29566irq/58-eth000:20:323
755199183,11cyclictest0-21swapper/322:16:333
755199182,10cyclictest0-21swapper/319:40:153
755199182,10cyclictest0-21swapper/319:40:153
755199181,13cyclictest0-21swapper/320:45:293
7551991810,6cyclictest0-21swapper/321:49:383
7551991810,5cyclictest29566irq/58-eth022:30:153
754399181,14cyclictest7627-21latency_hist19:10:022
753599188,7cyclictest22192-21missed_timers19:55:241
753599185,4cyclictest131rcu_preempt19:10:151
753599182,11cyclictest29920-21ssh22:47:431
753599182,11cyclictest0-21swapper/121:00:141
753599181,15cyclictest5362-21kthreadcore20:45:231
753599181,13cyclictest21517-21fschecks_time22:05:181
752099187,3cyclictest12-21ksoftirqd/019:20:250
755199179,6cyclictest0-21swapper/322:06:083
755199179,6cyclictest0-21swapper/321:26:303
755199179,6cyclictest0-21swapper/300:36:113
755199171,10cyclictest0-21swapper/321:15:293
753599179,5cyclictest29919-21acpi20:40:161
753599175,9cyclictest8529-21kthreadcore20:15:231
753599172,12cyclictest16299-21kthreadcore19:50:231
752099179,5cyclictest0-21swapper/019:35:290
752099178,2cyclictest12-21ksoftirqd/020:40:220
7520991710,4cyclictest12-21ksoftirqd/020:35:240
755199161,13cyclictest0-21swapper/320:20:193
753599161,12cyclictest18062-21cut19:20:191
7520991611,3cyclictest12-21ksoftirqd/020:30:240
755199153,6cyclictest0-21swapper/321:00:133
755199152,9cyclictest0-21swapper/320:05:153
755199151,11cyclictest0-21swapper/321:05:283
755199151,11cyclictest0-21swapper/319:20:253
755199146,6cyclictest0-21swapper/320:00:213
755199141,7cyclictest0-21swapper/319:15:293
755199141,10cyclictest0-21swapper/320:40:203
755199138,3cyclictest0-21swapper/319:51:523
755199138,3cyclictest0-21swapper/319:33:153
755199137,4cyclictest0-21swapper/320:50:193
755199133,8cyclictest0-21swapper/320:30:283
755199132,8cyclictest0-21swapper/320:00:013
755199131,9cyclictest0-21swapper/320:20:013
755199131,9cyclictest0-21swapper/319:10:323
755199123,7cyclictest0-21swapper/320:10:263
755199122,6cyclictest0-21swapper/319:35:153
755199122,5cyclictest0-21swapper/320:55:153
755199111,7cyclictest0-21swapper/319:45:183
755199101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional