You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 15:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu May 14, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7781996150,8cyclictest35966irq/58-eth012:30:113
777399566,32cyclictest25778-21ssh10:40:112
776499544,10cyclictest131rcu_preempt12:20:141
7773995243,5cyclictest27-21ksoftirqd/210:50:122
776499473,40cyclictest11240-21ssh10:50:121
775499464,35cyclictest4359-21ssh10:10:140
7773994524,13cyclictest27-21ksoftirqd/211:40:132
776499441,32cyclictest762-21ssh09:50:121
776499441,32cyclictest762-21ssh09:50:121
776499422,29cyclictest0-21swapper/110:40:131
7781994114,19cyclictest35966irq/58-eth010:30:133
778199405,22cyclictest35966irq/58-eth011:20:133
778199391,16cyclictest35966irq/58-eth011:50:123
778199381,24cyclictest35966irq/58-eth010:00:123
777399385,12cyclictest27-21ksoftirqd/211:05:132
777399381,30cyclictest0-21swapper/209:35:122
776499381,27cyclictest19435-21ssh12:25:111
776499381,16cyclictest14753-21ssh11:10:091
776499372,9cyclictest25098-21ssh12:10:141
775499375,18cyclictest3734-21ssh11:40:120
7781993613,19cyclictest321rcuc/309:05:123
777399364,19cyclictest25815-21kworker/2:109:30:102
776499361,10cyclictest17369-21kthreadcore09:00:201
777399356,17cyclictest27346irq/40-dwc2_hso09:05:132
777399353,18cyclictest27749-21ssh10:05:102
777399352,26cyclictest28352-21ssh12:30:122
776499351,10cyclictest17287-21tr09:40:211
776499350,12cyclictest12941-21grep10:15:021
775499358,20cyclictest12-21ksoftirqd/009:00:130
776499341,11cyclictest9908-21cpuspeed09:55:141
776499341,10cyclictest16787-21kthreadcore08:25:211
776499336,3cyclictest131rcu_preempt12:15:251
776499336,3cyclictest131rcu_preempt12:15:251
776499333,24cyclictest13273-21ssh10:15:131
776499332,9cyclictest1175-21wc09:30:271
776499332,23cyclictest5725-21ssh11:05:121
776499332,10cyclictest23776-21timerwakeupswit08:30:271
776499331,10cyclictest30548-21tr08:05:201
776499331,10cyclictest12957-21grep07:45:261
776499331,10cyclictest12957-21grep07:45:261
7754993321,6cyclictest536-21usb-storage11:10:110
777399322,19cyclictest27346irq/40-dwc2_hso09:20:132
777399322,18cyclictest9271-21sh11:25:122
777399322,13cyclictest23762-21ssh11:15:132
776499322,23cyclictest4936-21ssh12:35:141
776499321,9cyclictest26016-21kthreadcore08:35:201
7764993213,14cyclictest11828-21diskmemload09:20:131
776499321,11cyclictest6010-21kthreadcore11:40:211
776499321,10cyclictest434-21ssh09:50:011
776499320,12cyclictest3943-21awk11:20:231
775499322,23cyclictest16173-21ssh12:05:120
777399319,17cyclictest27-21ksoftirqd/211:35:122
777399312,15cyclictest4358-21sh10:10:142
776499312,8cyclictest0-21swapper/108:00:131
776499312,22cyclictest6316-21ssh09:35:131
776499311,9cyclictest29180-21kthreadcore11:35:201
776499311,10cyclictest32569-21ssh10:05:281
777399305,17cyclictest27346irq/40-dwc2_hso12:15:122
777399305,17cyclictest27346irq/40-dwc2_hso12:15:122
777399302,21cyclictest0-21swapper/208:40:142
776499304,15cyclictest20795-21ssh09:25:141
776499301,15cyclictest22320-21ssh10:20:141
776499301,11cyclictest6793-21cpu08:50:131
776499301,11cyclictest21700-21ssh11:50:131
775499303,18cyclictest27141-21ssh11:35:120
778199292,14cyclictest35966irq/58-eth009:35:123
777399294,8cyclictest2824-21ssh09:15:112
777399293,15cyclictest27346irq/40-dwc2_hso07:55:122
777399292,22cyclictest930-21snmpd08:00:112
7764992916,6cyclictest0-21swapper/111:25:121
776499291,11cyclictest20198-21latency10:35:221
776499291,10cyclictest20189-21rm10:55:151
776499291,10cyclictest18913-21df_inode10:00:131
775499292,19cyclictest5752-21ssh11:05:120
777399282,20cyclictest305-21ssh11:20:122
777399281,17cyclictest24244-21ssh09:45:142
777399281,16cyclictest745-21sh09:50:112
777399281,16cyclictest745-21sh09:50:112
776499287,6cyclictest16317-21cut12:05:151
776499282,7cyclictest29998-21ntp_states09:10:291
776499281,11cyclictest17587-21users11:45:321
7754992813,9cyclictest7741-21cyclictest09:45:130
7754992812,9cyclictest12-21ksoftirqd/012:28:190
777399278,11cyclictest27-21ksoftirqd/211:00:132
777399272,19cyclictest27346irq/40-dwc2_hso12:35:132
777399271,20cyclictest31295-21ssh10:25:132
777399271,20cyclictest30671-21ssh11:55:142
778199266,12cyclictest35966irq/58-eth010:10:143
777399262,9cyclictest0-21swapper/207:30:132
777399261,9cyclictest9829-21ssh09:55:132
776499260,10cyclictest18221-21ssh11:30:121
775499266,3cyclictest131rcu_preempt11:00:310
775499263,20cyclictest12697-21ssh11:45:120
777399254,16cyclictest27346irq/40-dwc2_hso12:00:142
777399254,10cyclictest13226-21acpi10:15:112
777399251,15cyclictest13652-21cron07:50:012
777399251,15cyclictest13652-21cron07:50:002
777399251,10cyclictest22476-21rm11:50:182
776499251,11cyclictest7222-21ssh12:00:131
775499252,16cyclictest28336-21ssh12:30:110
7754992512,7cyclictest0-21swapper/011:16:430
776499242,6cyclictest10155-21ls08:20:021
776499241,6cyclictest0-21swapper/112:30:121
775499249,4cyclictest12-21ksoftirqd/012:15:350
775499249,4cyclictest12-21ksoftirqd/012:15:340
775499246,10cyclictest12-21ksoftirqd/009:59:350
775499244,13cyclictest0-21swapper/008:10:110
7754992415,5cyclictest12-21ksoftirqd/010:00:270
7754992412,9cyclictest0-21swapper/007:15:120
775499241,18cyclictest26260-21ssh09:10:150
775499241,16cyclictest0-21swapper/011:25:130
7754992410,5cyclictest12-21ksoftirqd/012:21:010
778199236,9cyclictest321rcuc/308:40:133
778199233,14cyclictest321rcuc/308:25:133
7781992311,10cyclictest0-21swapper/309:15:123
777399238,7cyclictest23066-21ntp_kernel_pll_12:25:242
777399234,16cyclictest25845-21kthreadcore12:10:212
777399232,16cyclictest27346irq/40-dwc2_hso10:55:142
777399232,15cyclictest27346irq/40-dwc2_hso10:45:122
777399231,19cyclictest19964-21acpi08:30:112
777399231,14cyclictest21790-21ntp_192.168.11411:30:232
775499239,4cyclictest12-21ksoftirqd/011:52:510
775499238,9cyclictest12-21ksoftirqd/009:23:080
775499237,8cyclictest12-21ksoftirqd/010:25:130
775499233,7cyclictest12-21ksoftirqd/010:35:500
7754992314,5cyclictest0-21swapper/011:00:020
775499231,17cyclictest10867-21ssh12:00:250
778199229,9cyclictest35966irq/58-eth010:55:133
778199222,14cyclictest0-21swapper/311:45:133
7781992213,7cyclictest0-21swapper/311:55:133
778199221,14cyclictest0-21swapper/310:05:103
777399223,12cyclictest27346irq/40-dwc2_hso09:25:152
777399222,15cyclictest15286-21ssh09:40:122
777399222,11cyclictest15636-21cut11:10:172
777399221,17cyclictest14629-21kthreadcore08:55:222
777399221,12cyclictest22802-21irqrtprio07:25:182
775499229,4cyclictest12-21ksoftirqd/012:36:150
775499228,4cyclictest12-21ksoftirqd/010:54:410
775499227,8cyclictest12-21ksoftirqd/009:50:470
775499227,8cyclictest12-21ksoftirqd/009:50:470
775499227,7cyclictest131rcu_preempt07:55:120
775499227,2cyclictest131rcu_preempt11:31:040
775499224,5cyclictest12-21ksoftirqd/010:40:230
775499223,8cyclictest12-21ksoftirqd/009:25:100
775499221,7cyclictest12-21ksoftirqd/010:24:320
7754992212,4cyclictest12-21ksoftirqd/009:31:260
775499221,18cyclictest31169-21latency10:05:200
7754992210,10cyclictest151rcuc/008:25:130
778199219,9cyclictest0-21swapper/309:10:543
778199212,12cyclictest35966irq/58-eth011:25:123
7781992116,3cyclictest36066irq/59-eth009:45:133
7781992115,4cyclictest0-21swapper/310:15:243
7781992112,6cyclictest0-21swapper/307:15:163
7781992111,7cyclictest0-21swapper/312:25:113
778199211,11cyclictest35966irq/58-eth009:30:113
777399219,9cyclictest9013-21/usr/sbin/munin08:15:242
777399215,14cyclictest4131-21kthreadcore08:45:202
777399212,9cyclictest12457-21cron07:15:012
777399212,8cyclictest8033-21fschecks_time10:30:162
777399212,16cyclictest26670-21ps08:35:202
777399212,10cyclictest16776-21ssh10:35:132
777399211,9cyclictest23379-21cut10:20:212
777399211,10cyclictest8826-21kthreadcore08:50:202
777399211,10cyclictest26259-21sh09:10:152
777399211,10cyclictest1167-21kthreadcore07:35:182
776499219,3cyclictest131rcu_preempt10:45:141
775499216,3cyclictest131rcu_preempt12:10:130
775499215,4cyclictest131rcu_preempt08:45:170
775499214,3cyclictest131rcu_preempt10:31:270
775499211,17cyclictest7387-21ps08:15:190
775499211,17cyclictest22298-21find07:25:110
775499211,17cyclictest16817-21ls10:15:230
775499210,16cyclictest2819-21ssh09:15:120
778199209,8cyclictest0-21swapper/312:09:193
778199202,12cyclictest33-21ksoftirqd/310:40:133
7781992013,5cyclictest0-21swapper/312:38:413
7781992013,5cyclictest0-21swapper/311:30:113
7781992011,7cyclictest0-21swapper/311:44:533
7781992011,6cyclictest0-21swapper/309:40:153
7781992011,4cyclictest0-21swapper/311:18:533
7781992010,8cyclictest0-21swapper/311:00:583
777399201,9cyclictest5359-21kthreadcore07:40:192
777399201,9cyclictest3284-21kthreadcore08:10:202
777399201,9cyclictest30723-21kthreadcore08:05:202
777399201,9cyclictest167-21systemd-network08:20:172
777399201,9cyclictest16254-21awk12:05:132
777399201,9cyclictest12929-21idleruntime07:15:162
777399201,10cyclictest27-21ksoftirqd/207:50:182
777399201,10cyclictest19341-21irqcore10:00:172
777399201,10cyclictest11064-21grep12:20:182
7773992010,7cyclictest12691-21apt-get11:45:122
776499209,7cyclictest8115-21grep07:40:281
776499203,11cyclictest0-21swapper/107:10:141
776499202,16cyclictest17599-21timerandwakeup07:50:271
776499201,17cyclictest0-21swapper/108:45:121
775499209,8cyclictest1950-21ntpq11:55:260
775499208,9cyclictest28682-21grep08:35:290
775499208,4cyclictest12-21ksoftirqd/011:24:060
775499208,4cyclictest12-21ksoftirqd/007:40:140
775499207,3cyclictest131rcu_preempt09:35:120
775499207,3cyclictest12-21ksoftirqd/009:43:190
775499205,3cyclictest131rcu_preempt08:00:270
775499201,4cyclictest12-21ksoftirqd/010:47:310
7754992012,5cyclictest0-21swapper/008:44:440
778199199,8cyclictest0-21swapper/312:14:353
778199193,6cyclictest0-21swapper/310:45:133
778199192,11cyclictest0-21swapper/311:10:053
778199192,11cyclictest0-21swapper/307:20:123
7781991913,4cyclictest0-21swapper/311:08:393
7781991912,5cyclictest0-21swapper/310:25:143
7781991911,6cyclictest0-21swapper/309:55:143
777399192,9cyclictest19210-21kthreadcore07:20:192
777399192,8cyclictest16374-21kthreadcore08:25:202
777399191,9cyclictest17244-21kthreadcore09:00:202
776499199,7cyclictest22288-21sshd07:25:111
776499198,8cyclictest4297-21sshd11:03:441
776499194,4cyclictest131rcu_preempt07:40:011
776499192,11cyclictest0-21swapper/108:40:141
776499192,10cyclictest7847-21ssh10:30:131
776499191,13cyclictest23756-21ssh11:15:121
775499199,4cyclictest12-21ksoftirqd/007:45:300
775499199,4cyclictest12-21ksoftirqd/007:45:300
775499198,6cyclictest151rcuc/007:20:190
775499197,6cyclictest12-21ksoftirqd/007:53:570
775499195,11cyclictest7816-21latency_hist07:10:010
775499194,3cyclictest131rcu_preempt07:35:110
775499192,4cyclictest12-21ksoftirqd/008:55:200
775499192,3cyclictest0-21swapper/008:20:220
775499192,14cyclictest11523-21gpgv08:55:020
775499191,12cyclictest0-21swapper/008:30:110
7754991910,3cyclictest0-21swapper/007:30:220
778199189,6cyclictest35966irq/58-eth012:15:123
778199189,6cyclictest35966irq/58-eth012:15:113
778199189,6cyclictest35966irq/58-eth012:00:133
778199189,6cyclictest0-21swapper/311:39:243
778199189,4cyclictest0-21swapper/310:36:383
778199187,8cyclictest35966irq/58-eth010:50:123
778199182,11cyclictest35966irq/58-eth009:00:133
7781991811,5cyclictest35966irq/58-eth010:22:433
778199181,12cyclictest0-21swapper/309:20:223
778199181,11cyclictest0-21swapper/308:05:163
778199181,10cyclictest0-21swapper/307:40:023
7781991810,6cyclictest0-21swapper/312:21:333
776499181,13cyclictest31518-21tr11:55:191
7764991810,6cyclictest3574-21ps09:15:191
776499180,13cyclictest20584-21kthreadcore07:55:201
775499189,3cyclictest12-21ksoftirqd/008:05:160
775499183,3cyclictest131rcu_preempt09:05:280
778199172,10cyclictest0-21swapper/308:50:133
778199172,10cyclictest0-21swapper/307:45:123
778199172,10cyclictest0-21swapper/307:45:123
7781991711,4cyclictest0-21swapper/309:26:113
778199171,11cyclictest0-21swapper/309:50:143
778199171,11cyclictest0-21swapper/309:50:143
778199171,10cyclictest0-21swapper/308:00:263
778199171,10cyclictest0-21swapper/307:40:223
777399171,8cyclictest7641-21latency_hist07:10:022
776499179,5cyclictest13252-21kthreadcore08:20:201
776499176,8cyclictest12692-21sort07:15:121
776499175,10cyclictest25838-21cron09:10:021
776499172,12cyclictest870-21fschecks_count08:10:141
776499171,13cyclictest2587-21ssh10:25:271
7781991610,4cyclictest0-21swapper/307:50:293
776499164,10cyclictest12420-21kthreadcore08:55:201
776499162,10cyclictest0-21swapper/107:30:131
776499161,12cyclictest7837-21ls07:10:021
776499161,12cyclictest17504-21sshd07:20:101
775499162,11cyclictest10843-21cut07:10:220
778199151,9cyclictest0-21swapper/308:55:133
778199151,9cyclictest0-21swapper/307:30:133
778199151,11cyclictest0-21swapper/307:10:223
778199149,3cyclictest0-21swapper/308:45:273
778199149,3cyclictest0-21swapper/308:10:193
778199141,6cyclictest0-21swapper/308:20:023
778199138,3cyclictest0-21swapper/308:36:243
778199132,8cyclictest321rcuc/307:25:123
778199131,8cyclictest35966irq/58-eth008:20:113
778199131,10cyclictest0-21swapper/308:30:183
778199121,9cyclictest0-21swapper/307:55:133
778199101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional