You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 17:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Jan 02, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1622299561,35cyclictest18578-21ssh09:55:132
1620399491,2cyclictest131rcu_preempt07:20:130
16212994511,9cyclictest21799-21apt-get12:05:121
1620399440,38cyclictest12756-21ssh10:10:120
1620399440,31cyclictest24735-21ssh10:35:140
16222993914,16cyclictest16195-21cyclictest10:10:132
1621299399,11cyclictest29420-21ssh11:13:521
16212993921,13cyclictest13103-21apt-get07:40:111
16212993911,10cyclictest22395-21ssh11:45:261
1620399391,28cyclictest4191-21ssh11:00:130
1622299382,19cyclictest3874-21ssh09:27:262
1621299388,12cyclictest25997-21ssh09:58:421
1621299388,11cyclictest7335-21ssh12:32:281
1621299388,11cyclictest7335-21ssh12:32:281
1621299381,34cyclictest6376-21ssh09:30:121
16212993810,11cyclictest15214-21rm11:24:301
1623299371,19cyclictest28366irq/58-eth008:25:123
16212993729,4cyclictest24710-21apt-get10:35:141
1622299362,27cyclictest26511-21sh09:21:252
1622299362,15cyclictest3847-21ssh10:05:122
1621299367,11cyclictest25018-21/usr/sbin/munin09:20:291
1621299367,11cyclictest11740-21ssh10:09:151
16212993624,9cyclictest25581-21apt-get09:00:131
16212993618,11cyclictest32449-21apt-get08:00:111
16212993610,9cyclictest9722-21ls08:10:121
16212993610,12cyclictest15159-21ssh10:45:411
16212993610,12cyclictest15159-21ssh10:45:401
1622299353,24cyclictest0-21swapper/207:15:262
1622299352,26cyclictest22642-21/usr/sbin/munin10:50:262
1621299359,8cyclictest2599-21diskmemload11:35:351
1621299359,4cyclictest4478-21ssh12:12:211
1621299359,12cyclictest10641-21ssh09:13:521
1621299359,10cyclictest22459-21ssh10:15:201
1621299358,9cyclictest32602-21ssh12:29:321
1621299358,10cyclictest11252-21cat10:25:291
1621299357,10cyclictest31586-21rm10:55:271
1621299357,10cyclictest19889-21ssh10:13:151
1621299356,11cyclictest24121-21ssh10:50:531
16212993530,3cyclictest24209-21apt-get08:25:131
1622299344,22cyclictest27-21ksoftirqd/211:00:122
1621299349,11cyclictest21609-21ssh11:26:211
1621299348,11cyclictest15663-21ssh12:20:051
1621299348,11cyclictest11905-21ssh11:04:051
1621299348,10cyclictest12125-21ssh11:59:261
16212993428,3cyclictest1428-21apt-get08:35:101
16212993419,11cyclictest10098-21ssh11:40:111
1622299333,18cyclictest22346irq/40-dwc2_hso09:10:112
1622299331,17cyclictest10119-21sh10:45:122
1622299331,17cyclictest10119-21sh10:45:112
1621299338,9cyclictest31883-21ssh11:33:181
1621299336,11cyclictest13233-21ssh12:17:131
16212993319,11cyclictest17954-21apt-get07:45:151
16212993317,12cyclictest13023-21apt-get11:05:121
16212993317,11cyclictest4854-21apt-get08:05:131
16212993310,6cyclictest20292-21ssh12:03:351
16212993310,10cyclictest2599-21diskmemload09:44:081
16212993310,10cyclictest2599-21diskmemload09:44:071
1622299327,16cyclictest27-21ksoftirqd/211:05:192
1621299328,9cyclictest16835-21ssh09:53:231
1621299328,8cyclictest3256-21ssh11:15:371
1621299327,9cyclictest0-21swapper/110:21:281
1621299327,8cyclictest30019-21diskstats09:25:171
1621299327,11cyclictest8993-21ssh09:49:411
1622299319,12cyclictest27-21ksoftirqd/211:39:502
1622299313,6cyclictest27-21ksoftirqd/211:22:492
1622299313,21cyclictest0-21swapper/209:17:252
1622299312,23cyclictest924-21snmpd09:44:342
1622299312,23cyclictest924-21snmpd09:44:332
1622299311,17cyclictest9718-21ssh12:35:122
1621299319,8cyclictest8780-21kworker/u8:209:37:051
1621299319,8cyclictest19536-21ssh09:18:521
1621299318,8cyclictest8208-21ssh10:43:071
1621299318,8cyclictest2018-21ntp_192.168.11407:25:241
1621299318,10cyclictest1288-21ssh11:51:511
16212993124,3cyclictest27600-21apt-get07:55:121
16212993124,3cyclictest27600-21apt-get07:55:121
1622299304,11cyclictest16104-21diskmemload11:25:142
1622299302,22cyclictest91-21pr/ttyS012:15:012
1622299301,15cyclictest27363-21cron07:55:012
1621299307,10cyclictest1639-21rm10:02:441
1621299306,8cyclictest0-21swapper/112:35:341
16212993021,3cyclictest15832-21apt-get10:30:141
1620399304,14cyclictest12-21ksoftirqd/008:05:120
1622299299,3cyclictest131rcu_preempt08:25:112
1622299298,9cyclictest27-21ksoftirqd/211:47:162
1622299295,18cyclictest27-21ksoftirqd/212:24:512
1622299295,14cyclictest27-21ksoftirqd/210:03:192
1622299292,10cyclictest14420-21sshd09:34:322
1621299297,9cyclictest21387-21/usr/sbin/munin07:15:141
1621299296,8cyclictest24167-21ntp_192.168.11408:55:251
16212992923,3cyclictest15911-21apt-get08:50:131
1620399297,3cyclictest131rcu_preempt07:30:000
1620399293,7cyclictest5853-21ssh12:14:060
1620399292,6cyclictest12978-21ssh09:15:190
16232992817,6cyclictest28366irq/58-eth011:50:123
1622299288,12cyclictest27-21ksoftirqd/210:44:072
1622299284,17cyclictest22346irq/40-dwc2_hso12:15:142
1622299283,17cyclictest22346irq/40-dwc2_hso07:40:122
16222992813,4cyclictest27-21ksoftirqd/210:27:052
16222992811,14cyclictest18195-21cron09:55:012
1621299285,8cyclictest19813-21ntp_192.168.11407:10:241
16212992818,7cyclictest26134-21apt-get07:20:121
1620399283,8cyclictest23921-21ssh11:45:420
16203992817,8cyclictest3837-21apt-get10:05:140
1620399281,15cyclictest24262-21ssh09:40:130
1620399281,15cyclictest24262-21ssh09:40:130
1622299279,8cyclictest27-21ksoftirqd/211:32:472
1622299278,12cyclictest27-21ksoftirqd/211:13:542
1622299277,13cyclictest131rcu_preempt08:40:012
1622299273,19cyclictest0-21swapper/209:49:252
1622299272,20cyclictest30267-21latency_hist12:10:012
1622299272,20cyclictest0-21swapper/207:45:162
1622299272,19cyclictest0-21swapper/208:20:142
16212992715,5cyclictest21-21ksoftirqd/108:40:131
1620399273,10cyclictest12-21ksoftirqd/009:32:080
1620399272,5cyclictest12-21ksoftirqd/011:10:270
16232992617,4cyclictest28366irq/58-eth010:25:113
1622299268,7cyclictest27-21ksoftirqd/211:40:542
1622299268,13cyclictest27-21ksoftirqd/211:52:322
1622299268,12cyclictest27-21ksoftirqd/210:31:342
1622299266,3cyclictest131rcu_preempt08:55:312
1622299266,12cyclictest27-21ksoftirqd/211:15:052
1622299262,9cyclictest0-21swapper/208:15:172
1622299262,9cyclictest0-21swapper/207:20:142
1622299262,19cyclictest0-21swapper/210:24:442
1622299261,16cyclictest0-21swapper/208:30:122
16212992620,3cyclictest29042-21apt-get08:30:121
16212992613,9cyclictest0-21swapper/107:35:111
1620399269,11cyclictest0-21swapper/011:15:480
1620399267,4cyclictest16038-21sshd09:52:250
1620399264,7cyclictest12-21ksoftirqd/011:25:120
1620399264,4cyclictest12-21ksoftirqd/012:30:300
1620399264,4cyclictest12-21ksoftirqd/012:30:290
1620399264,4cyclictest12-21ksoftirqd/012:05:170
1620399264,12cyclictest12-21ksoftirqd/010:27:250
1620399263,8cyclictest19954-21ssh11:08:040
1620399263,11cyclictest12-21ksoftirqd/012:15:240
1620399262,11cyclictest12-21ksoftirqd/010:41:040
16203992615,3cyclictest12-21ksoftirqd/012:21:570
16203992613,6cyclictest12-21ksoftirqd/010:19:130
1620399261,19cyclictest28995-21rm09:24:310
1623299255,12cyclictest0-21swapper/311:10:143
1622299256,12cyclictest27-21ksoftirqd/210:15:282
1622299254,16cyclictest27-21ksoftirqd/212:00:132
16222992515,4cyclictest0-21swapper/209:00:152
1622299251,18cyclictest0-21swapper/207:35:512
16222992511,4cyclictest27-21ksoftirqd/212:28:402
16222992510,4cyclictest27-21ksoftirqd/212:31:132
16222992510,4cyclictest27-21ksoftirqd/212:31:132
16222992510,4cyclictest27-21ksoftirqd/211:57:412
16212992515,6cyclictest7984-21users07:30:311
1620399257,4cyclictest12-21ksoftirqd/008:10:280
1620399255,4cyclictest12-21ksoftirqd/010:01:400
1620399254,9cyclictest12-21ksoftirqd/009:45:210
1620399254,5cyclictest12-21ksoftirqd/011:58:470
1620399254,5cyclictest12-21ksoftirqd/009:55:230
1620399254,5cyclictest12-21ksoftirqd/009:37:400
1620399254,5cyclictest12-21ksoftirqd/009:26:040
1620399254,4cyclictest12-21ksoftirqd/009:10:190
1620399253,8cyclictest18842-21ssh10:50:040
1620399253,5cyclictest12-21ksoftirqd/012:38:380
1620399253,5cyclictest12-21ksoftirqd/011:35:230
1620399253,5cyclictest12-21ksoftirqd/010:56:530
1620399253,5cyclictest12-21ksoftirqd/008:45:190
1620399253,13cyclictest12-21ksoftirqd/011:20:130
1620399252,12cyclictest12-21ksoftirqd/011:41:110
1620399252,11cyclictest12-21ksoftirqd/012:25:360
1620399251,6cyclictest12-21ksoftirqd/010:33:010
16203992510,8cyclictest0-21swapper/010:46:460
16203992510,8cyclictest0-21swapper/010:46:450
16232992419,3cyclictest28466irq/59-eth010:10:123
1622299247,8cyclictest27-21ksoftirqd/210:35:082
1622299247,4cyclictest27-21ksoftirqd/210:58:232
1622299246,10cyclictest131rcu_preempt08:50:152
1622299243,13cyclictest22346irq/40-dwc2_hso07:55:112
1622299243,13cyclictest22346irq/40-dwc2_hso07:55:102
1622299242,17cyclictest0-21swapper/209:35:242
16212992414,7cyclictest14504-21apt-get08:15:181
16212992411,9cyclictest24043-21cat07:50:191
16212992411,9cyclictest15777-21latency_hist08:50:011
1620399245,4cyclictest12-21ksoftirqd/012:01:120
1620399245,3cyclictest131rcu_preempt07:55:120
1620399245,3cyclictest131rcu_preempt07:55:120
1620399244,4cyclictest12-21ksoftirqd/008:55:240
1620399243,5cyclictest12-21ksoftirqd/011:30:260
1620399243,5cyclictest12-21ksoftirqd/009:05:190
1620399243,4cyclictest12-21ksoftirqd/007:30:270
1620399243,11cyclictest12-21ksoftirqd/011:50:030
1620399242,5cyclictest12-21ksoftirqd/008:30:300
1620399241,8cyclictest12-21ksoftirqd/010:20:300
16203992417,4cyclictest22815-21apt-get07:50:140
16232992311,7cyclictest28366irq/58-eth010:30:133
1623299231,11cyclictest0-21swapper/309:24:313
1622299235,11cyclictest131rcu_preempt08:00:272
1622299232,6cyclictest0-21swapper/207:25:242
1622299230,8cyclictest9905-21ls08:10:152
16212992314,7cyclictest2625-21http09:10:001
16212992312,8cyclictest23998-21latency_hist08:25:001
1620399238,8cyclictest12-21ksoftirqd/007:45:140
1620399236,5cyclictest12-21ksoftirqd/008:00:150
1620399234,9cyclictest12-21ksoftirqd/008:15:230
1620399233,5cyclictest12-21ksoftirqd/009:00:130
1620399233,4cyclictest12-21ksoftirqd/008:50:240
1620399233,4cyclictest12-21ksoftirqd/007:35:130
1620399232,5cyclictest12-21ksoftirqd/007:15:020
16232992217,3cyclictest321rcuc/311:15:153
16232992217,3cyclictest321rcuc/310:45:253
16232992217,3cyclictest321rcuc/310:45:253
16232992217,3cyclictest321rcuc/308:35:143
16232992216,4cyclictest321rcuc/312:01:333
16232992216,4cyclictest321rcuc/310:43:163
16232992216,4cyclictest321rcuc/308:40:273
1622299226,13cyclictest131rcu_preempt08:05:162
1622299225,4cyclictest27-21ksoftirqd/207:10:022
16222992212,8cyclictest0-21swapper/208:45:262
1620399224,4cyclictest12-21ksoftirqd/008:40:170
1620399222,5cyclictest12-21ksoftirqd/008:35:130
1620399222,5cyclictest12-21ksoftirqd/007:10:140
1620399221,5cyclictest12-21ksoftirqd/008:25:170
1620399221,4cyclictest12-21ksoftirqd/007:40:150
1623299217,10cyclictest28366irq/58-eth010:55:133
1623299212,13cyclictest0-21swapper/309:10:123
16232992116,3cyclictest321rcuc/308:55:223
16232992115,4cyclictest321rcuc/312:28:033
16232992115,4cyclictest321rcuc/311:26:453
16232992115,4cyclictest321rcuc/311:05:553
16232992115,4cyclictest321rcuc/307:30:333
16232992115,4cyclictest28466irq/59-eth009:19:043
16232992113,5cyclictest28366irq/58-eth011:00:113
1622299217,10cyclictest18604-21kthreadcore07:10:212
1622299212,12cyclictest27-21ksoftirqd/209:10:012
1620399214,4cyclictest151rcuc/008:25:010
1623299209,8cyclictest0-21swapper/310:03:303
1623299209,4cyclictest28366irq/58-eth012:35:113
1623299203,14cyclictest0-21swapper/310:05:123
16232992015,3cyclictest321rcuc/311:49:083
16232992015,3cyclictest321rcuc/309:50:343
16232992015,3cyclictest321rcuc/309:30:473
16232992014,4cyclictest321rcuc/312:30:133
16232992014,4cyclictest321rcuc/312:30:133
16232992014,4cyclictest321rcuc/309:25:493
16232992014,4cyclictest321rcuc/308:00:123
16232992014,4cyclictest321rcuc/307:40:303
16232992014,4cyclictest28466irq/59-eth010:15:193
16232992011,6cyclictest28366irq/58-eth010:35:143
16232992011,4cyclictest0-21swapper/311:30:303
1622299201,15cyclictest3743-21grep07:30:202
1620399201,7cyclictest12-21ksoftirqd/007:10:020
1623299191,9cyclictest0-21swapper/307:50:023
16232991914,3cyclictest321rcuc/307:20:133
16232991913,4cyclictest321rcuc/311:55:143
16232991913,4cyclictest0-21swapper/309:42:133
16232991913,4cyclictest0-21swapper/309:42:123
16232991910,6cyclictest0-21swapper/311:21:493
1622299191,9cyclictest6985-21kthreadcore08:40:192
16232991813,3cyclictest0-21swapper/307:50:293
16232991812,4cyclictest321rcuc/311:41:263
16232991812,4cyclictest321rcuc/308:10:113
16232991812,4cyclictest0-21swapper/312:17:213
16232991812,4cyclictest0-21swapper/310:21:293
16232991812,4cyclictest0-21swapper/309:45:163
16232991811,5cyclictest28366irq/58-eth009:35:123
16232991810,6cyclictest0-21swapper/311:37:413
1623299179,4cyclictest0-21swapper/312:10:153
16232991712,4cyclictest321rcuc/308:15:173
16232991712,3cyclictest321rcuc/308:30:183
16232991712,3cyclictest321rcuc/307:10:223
16232991711,4cyclictest321rcuc/309:00:133
16232991711,4cyclictest321rcuc/307:55:113
16232991711,4cyclictest321rcuc/307:55:103
16232991711,4cyclictest321rcuc/307:25:023
16232991711,4cyclictest321rcuc/307:15:133
16232991711,4cyclictest0-21swapper/312:20:113
16232991711,4cyclictest0-21swapper/309:55:223
16232991711,4cyclictest0-21swapper/308:45:123
1623299171,11cyclictest28366irq/58-eth012:05:123
1623299171,10cyclictest0-21swapper/310:50:263
1621299165,8cyclictest16266-21wc07:10:011
1623299159,4cyclictest0-21swapper/308:05:153
16232991510,3cyclictest0-21swapper/309:05:253
1623299148,4cyclictest321rcuc/307:35:123
1623299148,4cyclictest0-21swapper/308:50:233
1623299137,4cyclictest33-21ksoftirqd/308:20:133
1623299101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional