You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 09:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Thu May 14, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
419399572,41cyclictest17529-21ssh21:25:112
417499543,38cyclictest19692-21ssh22:20:120
418299471,35cyclictest2125-21ssh23:40:131
4174994641,3cyclictest1501-21ssh22:10:130
417499433,31cyclictest22524-21diskmemload00:00:110
4193994112,23cyclictest8624-21rm23:42:342
417499411,3cyclictest131rcu_preempt21:20:140
418299388,25cyclictest16482-21ssh23:30:111
420199372,21cyclictest35966irq/58-eth021:55:123
4201993720,10cyclictest0-21swapper/322:30:133
418299373,31cyclictest24894-21ssh22:05:121
418299370,34cyclictest25513-21ssh23:35:131
417499371,31cyclictest6825-21ssh21:55:110
4174993622,6cyclictest12-21ksoftirqd/021:30:120
418299352,31cyclictest930-21snmpd21:15:131
418299351,30cyclictest0-21swapper/120:40:121
419399348,16cyclictest27346irq/40-dwc2_hso23:30:112
419399341,25cyclictest22934-21ssh21:10:152
417499342,20cyclictest30184-21ssh21:50:110
420199334,17cyclictest35966irq/58-eth022:35:123
420199333,18cyclictest35966irq/58-eth023:20:133
420199332,18cyclictest35966irq/58-eth000:15:103
419399339,11cyclictest29134-21diskmemload23:55:132
419399332,24cyclictest28825-21ssh00:10:412
419399332,24cyclictest28825-21ssh00:10:412
418299336,9cyclictest131rcu_preempt22:40:121
417499333,3cyclictest131rcu_preempt21:25:120
420199324,21cyclictest35966irq/58-eth022:00:123
4201993215,11cyclictest0-21swapper/300:06:453
4201993214,13cyclictest35966irq/58-eth023:50:133
420199321,21cyclictest35966irq/58-eth022:55:133
4193993215,3cyclictest0-21swapper/223:35:122
419399321,18cyclictest21711-21ssh23:15:142
419399321,14cyclictest27541-21ssh00:30:122
418299322,26cyclictest30208-21ssh21:50:131
418299321,24cyclictest0-21swapper/123:15:131
4174993218,9cyclictest536-21usb-storage22:50:120
418299316,4cyclictest131rcu_preempt00:25:121
4182993114,3cyclictest21-21ksoftirqd/123:55:131
417499311,21cyclictest2114-21ssh23:40:120
4201993015,12cyclictest0-21swapper/300:20:133
420199301,22cyclictest35966irq/58-eth020:20:123
420199301,17cyclictest35966irq/58-eth023:45:143
419399302,21cyclictest27-21ksoftirqd/223:45:442
419399293,21cyclictest18538-21apt-get00:25:132
419399292,11cyclictest14529-21rm00:20:452
419399291,18cyclictest27346irq/40-dwc2_hso23:05:112
4193992911,14cyclictest261rcuc/220:40:142
418299297,17cyclictest21-21ksoftirqd/121:05:111
4201992820,4cyclictest0-21swapper/320:00:123
419399282,10cyclictest8072-21ssh00:18:472
417499282,21cyclictest26037-21sshd00:28:420
417499282,20cyclictest2022-21ssh23:20:260
417499282,19cyclictest4932-21ssh22:45:440
4201992710,11cyclictest0-21swapper/300:38:583
419399278,4cyclictest27-21ksoftirqd/220:10:262
419399273,19cyclictest27346irq/40-dwc2_hso20:35:132
419399273,16cyclictest27346irq/40-dwc2_hso20:15:152
419399272,9cyclictest13311-21ssh00:03:442
419399272,16cyclictest27346irq/40-dwc2_hso21:50:122
419399271,17cyclictest30756-21ssh23:20:142
4193992710,8cyclictest27346irq/40-dwc2_hso21:15:142
418299271,4cyclictest0-21swapper/120:15:141
420199266,10cyclictest35966irq/58-eth020:40:123
420199264,14cyclictest0-21swapper/323:15:143
420199263,16cyclictest0-21swapper/300:30:133
420199262,16cyclictest0-21swapper/323:05:123
420199261,13cyclictest0-21swapper/322:05:123
419399266,14cyclictest8549-21apt-get21:20:132
419399263,4cyclictest131rcu_preempt20:50:122
4193992615,8cyclictest0-21swapper/223:29:122
418299263,19cyclictest21-21ksoftirqd/100:35:131
418299262,17cyclictest3650-21ssh23:05:131
417499263,16cyclictest15854-21ssh22:00:120
4174992615,7cyclictest121ksoftirqd/021:10:140
420199254,14cyclictest0-21swapper/321:25:133
420199253,15cyclictest0-21swapper/300:28:423
420199252,15cyclictest0-21swapper/323:55:133
419399253,17cyclictest27346irq/40-dwc2_hso23:50:132
419399253,11cyclictest131rcu_preempt00:05:172
419399252,8cyclictest3023-21ssh23:04:442
419399252,15cyclictest27346irq/40-dwc2_hso22:55:132
418299252,21cyclictest30536-21sh00:12:441
418299252,21cyclictest30536-21sh00:12:441
418299251,20cyclictest1475-21ssh22:10:111
417499252,5cyclictest131rcu_preempt21:15:130
417499252,18cyclictest26512-21ssh23:52:240
417499251,18cyclictest3073-21ssh21:35:120
420199243,14cyclictest0-21swapper/323:42:343
420199242,14cyclictest35966irq/58-eth023:10:133
420199242,14cyclictest35966irq/58-eth023:10:133
420199242,14cyclictest35966irq/58-eth022:25:123
419399249,6cyclictest13343-21grep21:00:162
419399246,4cyclictest27-21ksoftirqd/220:30:102
419399244,8cyclictest21175-21ssh21:45:132
419399242,12cyclictest27346irq/40-dwc2_hso19:20:122
4193992414,8cyclictest0-21swapper/222:35:132
4193992413,8cyclictest0-21swapper/222:52:552
4193992413,8cyclictest0-21swapper/221:55:122
419399241,16cyclictest0-21swapper/222:20:122
419399241,10cyclictest12051-21grep20:25:152
418299241,15cyclictest6850-21ssh21:55:121
417499247,3cyclictest131rcu_preempt22:30:270
417499244,15cyclictest15772-21ssh23:29:440
417499242,17cyclictest12171-21ssh21:40:130
420199235,11cyclictest0-21swapper/321:10:153
420199233,14cyclictest0-21swapper/300:14:443
420199233,14cyclictest0-21swapper/300:14:443
420199233,13cyclictest35966irq/58-eth021:45:123
420199232,14cyclictest0-21swapper/322:20:133
4201992313,8cyclictest0-21swapper/323:30:133
419399234,13cyclictest28667-21ssh22:25:122
419399234,11cyclictest26870-21kthreadcore22:05:222
419399232,6cyclictest0-21swapper/223:12:302
419399232,6cyclictest0-21swapper/223:12:302
419399232,14cyclictest27-21ksoftirqd/222:19:282
4193992312,9cyclictest0-21swapper/221:40:002
4193992312,8cyclictest0-21swapper/219:40:252
4193992312,8cyclictest0-21swapper/200:35:382
419399231,18cyclictest27346irq/40-dwc2_hso19:10:122
419399231,13cyclictest32701-21cut22:45:182
419399231,11cyclictest5253-21ssh22:30:122
418299239,5cyclictest131rcu_preempt21:36:181
418299238,5cyclictest21-21ksoftirqd/121:10:261
418299232,13cyclictest4173-21cyclictest20:45:141
4182992312,3cyclictest0-21swapper/121:21:001
418299231,19cyclictest5263-21ssh22:30:131
417499236,11cyclictest0-21swapper/021:00:140
4174992313,7cyclictest27470-21sshd21:47:140
4174992312,7cyclictest14792-21sh00:05:130
417499231,17cyclictest28188-21ssh23:17:300
417499231,16cyclictest27556-21ssh00:30:130
417499231,13cyclictest23262-21ssh22:40:120
420199223,13cyclictest0-21swapper/300:03:443
4201992214,5cyclictest0-21swapper/322:42:033
419399228,6cyclictest27-21ksoftirqd/221:07:082
419399228,6cyclictest17640-21cut22:00:192
419399224,15cyclictest27346irq/40-dwc2_hso19:50:112
419399222,10cyclictest11801-21latency_hist20:25:022
4193992216,4cyclictest25822-21ssh22:40:212
419399221,18cyclictest18792-21acpi19:25:122
419399221,17cyclictest26993-21irqcore21:30:182
419399221,17cyclictest12132-21/usr/sbin/munin21:40:152
419399221,15cyclictest27346irq/40-dwc2_hso19:45:132
419399221,15cyclictest27346irq/40-dwc2_hso19:15:132
4193992210,6cyclictest0-21swapper/219:30:132
4182992214,5cyclictest0-21swapper/119:25:261
4182992211,3cyclictest131rcu_preempt20:10:141
417499224,4cyclictest131rcu_preempt00:15:110
417499222,15cyclictest22524-21diskmemload00:39:440
417499222,15cyclictest12700-21ssh23:10:130
417499222,15cyclictest12700-21ssh23:10:130
4174992215,4cyclictest0-21swapper/022:15:120
417499221,16cyclictest8443-21cut20:55:150
417499221,16cyclictest18099-21cut21:05:150
420199214,12cyclictest0-21swapper/321:50:123
4201992115,4cyclictest321rcuc/321:15:163
4201992115,4cyclictest0-21swapper/323:40:013
4201992115,4cyclictest0-21swapper/323:26:413
4201992115,4cyclictest0-21swapper/322:52:173
4201992115,4cyclictest0-21swapper/319:35:223
4201992113,5cyclictest0-21swapper/322:11:183
420199211,11cyclictest0-21swapper/320:25:143
419399218,4cyclictest27-21ksoftirqd/220:00:192
419399218,11cyclictest27-21ksoftirqd/219:35:132
4193992112,7cyclictest0-21swapper/220:05:142
419399211,16cyclictest2491-21cut22:10:192
419399211,10cyclictest17448-21kthreadcore19:55:202
418299219,5cyclictest131rcu_preempt22:55:411
418299219,4cyclictest131rcu_preempt21:42:081
418299216,10cyclictest0-21swapper/100:34:441
418299212,15cyclictest32312-21ssh22:45:141
418299212,15cyclictest0-21swapper/123:51:441
418299212,13cyclictest21-21ksoftirqd/120:30:121
4182992113,4cyclictest14794-21ssh00:05:131
418299211,15cyclictest26553-21ssh21:30:141
4174992112,6cyclictest13093-21grep23:45:210
4174992111,7cyclictest12294-21cron23:10:010
417499211,12cyclictest3453-21ls23:05:020
420199203,12cyclictest0-21swapper/322:15:133
4201992014,4cyclictest0-21swapper/323:00:233
4201992014,4cyclictest0-21swapper/322:48:193
4201992014,4cyclictest0-21swapper/319:25:253
4201992013,5cyclictest0-21swapper/321:35:123
4201992013,5cyclictest0-21swapper/321:20:133
4201992013,4cyclictest0-21swapper/321:40:403
4201992010,8cyclictest0-21swapper/321:31:473
419399203,9cyclictest31265-21fschecks_count20:45:172
419399201,9cyclictest8521-21fschecks_time20:55:152
418299202,2cyclictest131rcu_preempt22:19:491
418299202,14cyclictest9198-21apt-get19:15:111
4182992012,6cyclictest15771-21sshd23:29:441
4182992012,5cyclictest13219-21proc_pri00:20:261
4182992012,4cyclictest11416-21if_err_eth123:45:181
418299201,10cyclictest8389-21cpuspeed20:55:141
418299200,17cyclictest19671-21find22:20:111
417499207,4cyclictest131rcu_preempt20:05:210
4174992013,3cyclictest11239-21cut00:20:190
417499201,2cyclictest131rcu_preempt19:25:120
4174992011,6cyclictest3310-21cron20:50:010
417499201,15cyclictest25113-21cut22:05:160
4201991914,3cyclictest0-21swapper/321:10:013
4201991914,3cyclictest0-21swapper/320:30:133
4201991913,4cyclictest0-21swapper/320:35:283
4201991913,4cyclictest0-21swapper/320:05:293
4201991913,4cyclictest0-21swapper/319:55:273
4201991913,4cyclictest0-21swapper/319:50:223
4201991913,4cyclictest0-21swapper/319:10:193
418299198,9cyclictest0-21swapper/123:12:441
418299198,9cyclictest0-21swapper/123:12:441
418299192,2cyclictest131rcu_preempt23:02:071
418299192,13cyclictest366-21ssh00:15:121
4182991912,5cyclictest0-21swapper/119:30:291
4182991911,5cyclictest6842-21kthreadcore00:00:181
4182991911,3cyclictest21-21ksoftirqd/120:25:241
417499193,5cyclictest14689-21grep22:35:180
417499193,10cyclictest0-21swapper/000:14:440
417499193,10cyclictest0-21swapper/000:14:440
417499191,2cyclictest131rcu_preempt19:35:120
4174991912,4cyclictest21915-21switchtime22:55:290
417499191,13cyclictest28672-21ssh22:25:130
420199182,9cyclictest0-21swapper/319:30:043
4201991812,4cyclictest0-21swapper/320:45:213
4201991812,4cyclictest0-21swapper/320:15:163
4201991812,4cyclictest0-21swapper/320:10:223
4201991812,4cyclictest0-21swapper/319:40:273
4201991812,4cyclictest0-21swapper/319:20:173
4201991812,4cyclictest0-21swapper/319:10:023
4201991811,4cyclictest0-21swapper/321:00:173
418299188,2cyclictest21-21ksoftirqd/119:50:251
418299184,11cyclictest16147-21/usr/sbin/munin22:00:161
418299181,8cyclictest9047-21wc19:45:231
418299181,14cyclictest30730-21apt-get23:20:131
4182991811,4cyclictest21-21ksoftirqd/122:35:221
418299181,14cyclictest0-21swapper/120:50:121
418299180,9cyclictest131rcu_preempt22:28:361
418299180,12cyclictest7716-21ls19:10:221
417499189,7cyclictest21246-21users20:30:290
417499185,3cyclictest131rcu_preempt23:30:210
417499183,13cyclictest12300-21irqcore20:25:170
417499182,14cyclictest10293-21kthreadcore19:15:180
417499181,9cyclictest0-21swapper/023:55:070
4174991811,5cyclictest25529-21cpu23:35:140
4174991811,4cyclictest0-21swapper/019:45:240
417499181,11cyclictest15508-21cut19:55:150
417499180,16cyclictest30001-21ls20:40:280
420199172,10cyclictest0-21swapper/319:45:143
4201991712,3cyclictest0-21swapper/320:55:163
4201991712,3cyclictest0-21swapper/320:50:133
4201991711,4cyclictest0-21swapper/319:15:243
418299179,6cyclictest17916-21irqcore21:25:161
418299174,11cyclictest0-21swapper/120:35:201
418299173,12cyclictest0-21swapper/119:40:131
4182991712,3cyclictest640-21sendmail19:40:011
4182991712,3cyclictest0-21swapper/120:00:231
418299171,13cyclictest9183-21fschecks_count22:50:141
418299171,13cyclictest21414-21fschecks_time21:45:161
4182991710,5cyclictest18561-21sshd19:21:501
4182991710,4cyclictest0-21swapper/119:55:041
417499176,8cyclictest0-21swapper/019:30:130
417499173,9cyclictest23437-21kthreadcore20:35:210
417499171,3cyclictest131rcu_preempt20:50:120
417499171,12cyclictest29594-21apt-get20:10:030
417499170,3cyclictest1422646fschecks_time19:20:150
418299164,10cyclictest13166-21grep21:00:141
418299161,7cyclictest26684-21kthreadcore20:05:191
418299161,6cyclictest7249-21tune2fs20:20:151
417499168,6cyclictest1195-21if_eth119:40:180
417499168,5cyclictest2800-21irqrtprio20:15:190
417499162,10cyclictest0-21swapper/020:00:140
417499161,12cyclictest6105-21kthreadcore19:10:200
419399151,10cyclictest4300-21grep19:10:022
418299157,5cyclictest4298-21latency_hist19:10:021
417499152,11cyclictest10660-21fschecks_count19:50:150
417499152,10cyclictest9222-21kthreadcore20:20:200
417499141,9cyclictest4130-21latency_hist19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional