You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 22:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Apr 28, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
444699641,4cyclictest0-21swapper/211:15:132
4446995420,24cyclictest27-21ksoftirqd/211:55:152
444699541,35cyclictest930-21snmpd09:15:142
444699533,3cyclictest0-21swapper/211:10:122
444699531,4cyclictest0-21swapper/209:10:122
444699531,3cyclictest4065-21ssh11:05:132
444699521,3cyclictest0-21swapper/212:35:132
444699518,7cyclictest131rcu_preempt10:55:142
444699513,44cyclictest31078-21ssh11:20:132
444699484,6cyclictest0-21swapper/212:20:132
444699473,28cyclictest28858-21ssh10:25:112
444699472,42cyclictest0-21swapper/211:45:122
444699472,3cyclictest0-21swapper/210:20:112
444699472,38cyclictest0-21swapper/210:14:482
444699471,34cyclictest9282-21ssh10:50:132
444699463,39cyclictest6080-21ssh12:00:142
444699451,5cyclictest0-21swapper/210:00:142
444699451,5cyclictest0-21swapper/208:25:122
444699441,6cyclictest0-21swapper/208:20:152
4424994412,20cyclictest131rcu_preempt08:50:140
444699434,2cyclictest0-21swapper/209:40:142
444699433,2cyclictest0-21swapper/212:25:122
444699424,35cyclictest27346irq/40-dwc2_hso09:55:112
444699413,34cyclictest27346irq/40-dwc2_hso10:30:142
444699413,34cyclictest27346irq/40-dwc2_hso10:30:132
444699411,35cyclictest25785-21ssh11:35:152
444699411,29cyclictest0-21swapper/207:40:132
444699402,34cyclictest31540-21irqcore08:45:182
444699398,28cyclictest20066-21sendmail08:00:012
444699394,2cyclictest0-21swapper/210:45:132
444699384,2cyclictest20437-21apt-get11:50:122
444699383,30cyclictest27346irq/40-dwc2_hso11:25:112
444699381,8cyclictest668-21grep12:15:132
444699381,12cyclictest16654-21ssh11:30:132
442499385,3cyclictest131rcu_preempt09:40:140
442499384,3cyclictest131rcu_preempt09:25:270
444699373,8cyclictest27346irq/40-dwc2_hso09:45:122
444699371,7cyclictest25304-21kthreadcore07:30:202
444699371,5cyclictest22524-21diskmemload12:30:162
4446993714,19cyclictest27-21ksoftirqd/212:10:112
444699371,10cyclictest29362-21kthreadcore07:35:192
444699370,8cyclictest29213-21kthreadcore09:30:242
442499377,4cyclictest131rcu_preempt10:40:130
444699361,8cyclictest14947-21grep10:15:282
444699361,5cyclictest23818-21grep10:40:152
444699361,30cyclictest14561-21apt-get10:35:152
443899361,32cyclictest14573-21ssh10:35:131
444699351,8cyclictest15179-21ssh12:05:162
444699351,8cyclictest13007-21switchtime07:15:252
444699351,7cyclictest22567-21sendmail_mailqu07:25:262
444699351,6cyclictest29360-21grep10:05:312
444699351,5cyclictest8972-21sort09:20:182
444699351,5cyclictest17979-21acpi09:05:132
444699351,30cyclictest27593-21grep11:00:162
444699343,26cyclictest0-21swapper/207:45:132
444699343,26cyclictest0-21swapper/207:45:132
444699341,8cyclictest8031-21ntp_states07:10:272
444699341,6cyclictest6974-21sed08:50:272
444699341,6cyclictest20353-21mailstats08:30:262
444699341,6cyclictest20353-21mailstats08:30:262
444699341,11cyclictest25267-21grep08:05:162
444699340,5cyclictest30929-21sshd08:44:092
4452993326,5cyclictest35966irq/58-eth012:15:103
444699331,8cyclictest24950-21ntp_192.168.11408:35:242
444699331,7cyclictest17741-21proc_pri07:20:262
444699331,7cyclictest1299-21unixbench_singl08:10:282
444699331,10cyclictest10846-21fschecks_count07:50:182
444699330,5cyclictest12908-21sshd08:57:472
443899331,25cyclictest25775-21ssh11:35:141
442499335,4cyclictest131rcu_preempt08:20:130
444699325,5cyclictest4416-21kthreadcore09:35:192
444699321,4cyclictest17512-21ssh09:25:022
444699320,7cyclictest5136-21ssh09:53:122
444699320,5cyclictest2573-21awk11:40:152
443899324,20cyclictest21-21ksoftirqd/110:15:121
442499327,3cyclictest131rcu_preempt10:20:120
442499324,6cyclictest12-21ksoftirqd/012:25:130
4452993118,10cyclictest35966irq/58-eth012:05:143
444699314,4cyclictest20502-21fschecks_time08:00:162
444699311,7cyclictest16652-21ntp_states09:00:252
444699310,6cyclictest6787-21sshd08:19:042
443899313,15cyclictest20448-21ssh11:50:131
4452992919,7cyclictest35966irq/58-eth011:05:143
443899297,4cyclictest131rcu_preempt10:20:151
443899292,21cyclictest5439-21ssh09:16:471
4438992913,6cyclictest30484-1kworker/u9:011:21:431
442499291,7cyclictest151rcuc/009:05:130
445299283,21cyclictest35966irq/58-eth010:20:123
4452992819,6cyclictest35966irq/58-eth009:35:133
442499285,17cyclictest17028-21diskmemload09:24:480
4424992814,8cyclictest0-21swapper/011:02:480
442499281,3cyclictest131rcu_preempt10:15:120
445299274,15cyclictest35966irq/58-eth012:00:143
443899278,12cyclictest21-21ksoftirqd/111:10:441
4424992713,5cyclictest26345-21/usr/sbin/munin08:40:150
445299263,15cyclictest35966irq/58-eth012:25:133
445299262,18cyclictest0-21swapper/309:16:473
445299262,13cyclictest35966irq/58-eth011:55:133
4452992611,11cyclictest0-21swapper/309:20:123
443899262,5cyclictest0-21swapper/109:40:261
4438992612,4cyclictest21-21ksoftirqd/109:35:251
442499262,4cyclictest131rcu_preempt11:45:150
442499262,21cyclictest25762-21ssh11:35:140
442499262,13cyclictest0-21swapper/007:30:150
4424992615,5cyclictest21181-21ssh10:37:360
4424992613,8cyclictest0-21swapper/009:59:150
4424992612,7cyclictest0-21swapper/009:53:410
445299259,11cyclictest35966irq/58-eth009:50:143
445299259,10cyclictest0-21swapper/310:44:283
445299252,18cyclictest0-21swapper/311:24:483
444699253,4cyclictest4411-21cyclictest07:10:012
444699253,4cyclictest4411-21cyclictest07:10:012
443899257,3cyclictest131rcu_preempt10:40:111
443899252,18cyclictest4044-21ssh10:28:481
443899251,8cyclictest16272-21cron11:30:021
4438992511,7cyclictest0-21swapper/109:29:071
4438992511,4cyclictest21-21ksoftirqd/109:25:011
442499259,7cyclictest0-21swapper/010:33:030
442499259,7cyclictest0-21swapper/010:33:030
442499257,4cyclictest131rcu_preempt09:45:210
442499256,9cyclictest0-21swapper/012:35:080
442499251,6cyclictest940-21ssh12:15:150
442499251,18cyclictest18315-21ssh10:55:140
4424992511,7cyclictest0-21swapper/011:27:280
445299243,17cyclictest35966irq/58-eth008:45:133
4452992413,8cyclictest35966irq/58-eth010:50:143
443899249,3cyclictest131rcu_preempt10:10:231
443899248,7cyclictest21-21ksoftirqd/110:01:211
443899247,9cyclictest21-21ksoftirqd/112:25:381
443899247,7cyclictest21-21ksoftirqd/109:45:041
443899244,16cyclictest2475-21ssh11:40:131
443899243,4cyclictest21-21ksoftirqd/112:30:161
443899242,4cyclictest21-21ksoftirqd/112:05:161
4438992415,6cyclictest26750-21ssh09:30:141
443899241,4cyclictest131rcu_preempt08:30:251
443899241,4cyclictest131rcu_preempt08:30:251
4438992410,3cyclictest21-21ksoftirqd/108:35:231
442499249,4cyclictest131rcu_preempt07:55:110
442499247,3cyclictest131rcu_preempt07:15:110
442499244,3cyclictest131rcu_preempt11:55:220
4424992411,7cyclictest0-21swapper/010:09:140
4424992411,7cyclictest0-21swapper/009:00:230
4424992411,3cyclictest12-21ksoftirqd/011:40:280
4424992410,7cyclictest0-21swapper/010:25:450
445299239,10cyclictest35966irq/58-eth010:00:143
445299238,11cyclictest35966irq/58-eth009:40:153
443899237,8cyclictest21-21ksoftirqd/109:58:131
443899236,9cyclictest21-21ksoftirqd/112:23:301
443899234,5cyclictest21-21ksoftirqd/112:00:091
443899234,11cyclictest21-21ksoftirqd/111:47:501
4438992311,4cyclictest21-21ksoftirqd/111:05:171
4438992310,4cyclictest21-21ksoftirqd/107:20:021
442499239,7cyclictest0-21swapper/012:22:340
442499239,4cyclictest0-21swapper/009:36:470
442499239,3cyclictest2370-21ssh12:33:170
442499236,5cyclictest12-21ksoftirqd/009:20:010
442499235,4cyclictest131rcu_preempt10:10:140
442499233,6cyclictest15470-21irqcore12:05:180
442499231,5cyclictest2354-21ntp_states11:20:260
4424992310,7cyclictest0-21swapper/012:12:440
4424992310,7cyclictest0-21swapper/009:13:180
445299227,10cyclictest0-21swapper/310:39:493
445299226,7cyclictest0-21swapper/311:15:133
445299223,13cyclictest0-21swapper/309:45:143
445299222,13cyclictest35966irq/58-eth008:40:113
4452992211,7cyclictest35966irq/58-eth010:55:143
4452992210,8cyclictest35966irq/58-eth010:25:123
4452992210,8cyclictest0-21swapper/310:10:143
443899229,8cyclictest21-21ksoftirqd/110:31:491
443899229,8cyclictest21-21ksoftirqd/110:31:491
443899229,3cyclictest21-21ksoftirqd/109:50:291
443899228,3cyclictest21-21ksoftirqd/111:18:301
443899225,3cyclictest131rcu_preempt11:31:131
443899222,5cyclictest21-21ksoftirqd/110:55:121
4438992212,3cyclictest21-21ksoftirqd/110:07:191
4438992211,8cyclictest13945-21ssh10:50:321
4438992211,8cyclictest1004-21ssh10:45:201
442499229,3cyclictest12-21ksoftirqd/011:05:300
442499222,4cyclictest131rcu_preempt08:00:180
4424992213,6cyclictest1945-21ssh10:45:210
4424992211,8cyclictest10084-21kthreadcore08:55:200
442499221,16cyclictest16665-21ssh11:30:120
442499221,15cyclictest14198-21find07:20:130
442499221,13cyclictest22524-21diskmemload11:50:110
4424992211,3cyclictest12-21ksoftirqd/011:15:140
442499221,10cyclictest12-21ksoftirqd/008:10:200
4424992210,5cyclictest0-21swapper/011:12:130
4424992210,3cyclictest12-21ksoftirqd/010:50:240
445299216,10cyclictest0-21swapper/309:30:143
445299212,12cyclictest0-21swapper/312:30:143
4452992110,8cyclictest0-21swapper/311:10:443
443899219,2cyclictest21-21ksoftirqd/112:35:081
443899216,2cyclictest131rcu_preempt12:17:361
443899215,3cyclictest131rcu_preempt08:10:201
443899213,15cyclictest630-21Async09:36:311
443899211,5cyclictest21-21ksoftirqd/107:25:181
4438992112,6cyclictest24010-21ssh12:10:111
4438992110,3cyclictest21-21ksoftirqd/111:55:431
442499219,9cyclictest22863-21grep07:25:280
442499219,4cyclictest12-21ksoftirqd/009:30:250
442499219,3cyclictest12-21ksoftirqd/008:45:120
442499218,3cyclictest12-21ksoftirqd/007:45:160
442499218,3cyclictest12-21ksoftirqd/007:45:160
442499213,10cyclictest12-21ksoftirqd/008:30:180
442499213,10cyclictest12-21ksoftirqd/008:30:180
4424992112,6cyclictest0-21swapper/008:15:130
4424992110,3cyclictest12-21ksoftirqd/010:00:190
445299209,8cyclictest0-21swapper/311:02:073
445299207,10cyclictest0-21swapper/309:05:133
445299203,11cyclictest0-21swapper/311:30:483
445299203,10cyclictest35966irq/58-eth011:35:153
445299203,10cyclictest0-21swapper/309:00:273
4452992014,4cyclictest321rcuc/312:24:423
4452992014,2cyclictest0-21swapper/312:35:133
4452992013,3cyclictest0-21swapper/309:10:113
445299201,11cyclictest0-21swapper/307:20:143
4452992010,6cyclictest35966irq/58-eth010:30:123
4452992010,6cyclictest35966irq/58-eth010:30:123
443899209,2cyclictest21-21ksoftirqd/109:10:071
443899206,5cyclictest13053-21latency_hist09:00:011
443899205,4cyclictest21-21ksoftirqd/111:00:181
443899204,8cyclictest21-21ksoftirqd/108:40:271
443899203,3cyclictest131rcu_preempt07:20:201
443899203,11cyclictest0-21swapper/108:50:151
443899202,5cyclictest21-21ksoftirqd/108:45:171
443899201,6cyclictest21-21ksoftirqd/107:50:161
443899201,16cyclictest5828-21apt-get07:45:121
443899201,16cyclictest5828-21apt-get07:45:121
4438992010,3cyclictest21-21ksoftirqd/108:00:301
442499209,2cyclictest131rcu_preempt07:50:330
442499208,4cyclictest12-21ksoftirqd/012:00:150
442499202,3cyclictest131rcu_preempt08:35:180
445299192,12cyclictest0-21swapper/309:59:213
4452991914,3cyclictest321rcuc/311:47:513
445299191,14cyclictest35966irq/58-eth011:25:133
445299191,12cyclictest0-21swapper/311:50:153
4452991910,6cyclictest35966irq/58-eth012:10:123
443899198,4cyclictest21-21ksoftirqd/108:15:211
443899194,3cyclictest131rcu_preempt07:10:301
443899192,2cyclictest131rcu_preempt07:55:211
443899191,14cyclictest25117-21apt-get08:05:131
442499197,3cyclictest12-21ksoftirqd/008:05:160
442499194,3cyclictest131rcu_preempt07:45:010
445299189,5cyclictest0-21swapper/308:55:143
445299182,9cyclictest0-21swapper/308:25:143
445299182,11cyclictest0-21swapper/307:56:483
445299182,11cyclictest0-21swapper/307:50:043
445299182,10cyclictest35966irq/58-eth008:05:123
445299182,10cyclictest0-21swapper/311:40:053
445299182,10cyclictest0-21swapper/310:45:153
445299182,10cyclictest0-21swapper/310:05:153
445299182,10cyclictest0-21swapper/308:30:133
445299182,10cyclictest0-21swapper/308:30:133
445299182,10cyclictest0-21swapper/308:15:133
445299182,10cyclictest0-21swapper/307:10:483
445299181,10cyclictest0-21swapper/308:00:173
445299181,10cyclictest0-21swapper/307:15:253
443899188,3cyclictest21-21ksoftirqd/109:02:481
443899182,13cyclictest11625-21apt-get08:25:041
442499189,2cyclictest12-21ksoftirqd/007:35:160
442499181,3cyclictest131rcu_preempt08:25:140
442499180,12cyclictest4823-21if_err_eth007:10:190
445299179,6cyclictest0-21swapper/309:26:043
445299179,3cyclictest0-21swapper/310:15:083
445299172,9cyclictest0-21swapper/307:45:133
445299172,9cyclictest0-21swapper/307:45:133
445299172,9cyclictest0-21swapper/307:25:143
445299172,10cyclictest0-21swapper/307:35:483
443899176,2cyclictest131rcu_preempt07:30:151
443899174,4cyclictest21-21ksoftirqd/108:20:181
443899173,3cyclictest131rcu_preempt07:40:171
443899171,4cyclictest18297-21if_eth109:05:181
445299162,9cyclictest0-21swapper/307:40:133
445299158,4cyclictest35966irq/58-eth008:10:133
445299149,3cyclictest0-21swapper/308:50:503
445299149,3cyclictest0-21swapper/308:35:433
445299145,3cyclictest0-21swapper/308:20:143
445299141,8cyclictest0-21swapper/307:30:173
443899147,4cyclictest4450-21latency_hist07:10:021
443899147,4cyclictest4450-21latency_hist07:10:011
442499141,10cyclictest4429-21latency_hist07:10:010
442499141,10cyclictest4429-21latency_hist07:10:010
445299111,7cyclictest0-21swapper/307:10:013
445299111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional