You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-18 - 11:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Jun 18, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26692997926,6cyclictest12-21ksoftirqd/000:25:120
2669299746,51cyclictest27458-21ssh23:05:130
2669299725,56cyclictest20884-21ssh21:50:120
26727996849,9cyclictest35966irq/58-eth023:20:133
2669299623,42cyclictest21870-21ssh21:15:120
2669299603,37cyclictest6627-21ssh22:00:110
2669299558,5cyclictest131rcu_preempt19:15:110
2670699538,36cyclictest25947-21ssh21:35:121
2670699538,36cyclictest25947-21ssh21:35:121
2669299534,36cyclictest131rcu_preempt19:30:120
2670699525,36cyclictest14534-21ssh22:40:141
26692994937,7cyclictest1140-21ssh00:20:130
2670699482,37cyclictest2700-21ssh23:45:121
2669299466,7cyclictest131rcu_preempt19:10:130
2669299465,35cyclictest4061-21rm23:10:130
26692994628,7cyclictest12-21ksoftirqd/023:55:130
2669299461,33cyclictest26067-21ssh23:40:120
2670699456,21cyclictest16811-21ssh21:30:131
2669299453,4cyclictest131rcu_preempt22:05:120
26727994323,16cyclictest35966irq/58-eth021:25:123
2669299432,6cyclictest131rcu_preempt23:50:130
2672799422,27cyclictest35966irq/58-eth000:35:113
26727994216,14cyclictest35966irq/58-eth023:30:123
26727994116,13cyclictest35966irq/58-eth021:05:133
2669299411,29cyclictest13113-21ssh23:15:120
2672799393,23cyclictest35966irq/58-eth023:05:143
2672799385,24cyclictest35966irq/58-eth000:20:133
2672799383,22cyclictest35966irq/58-eth000:25:133
2672799382,25cyclictest35966irq/58-eth000:10:133
2672799382,25cyclictest35966irq/58-eth000:10:133
2672799382,24cyclictest35966irq/58-eth023:00:123
26727993813,17cyclictest35966irq/58-eth021:20:123
2669299383,26cyclictest28764-21ssh22:30:130
2669299383,26cyclictest19685-21ssh22:25:110
2669299382,24cyclictest23590-21ssh22:45:130
2672799378,24cyclictest33-21ksoftirqd/322:45:113
2672799372,20cyclictest35966irq/58-eth022:10:133
2672799372,20cyclictest35966irq/58-eth022:10:133
2672799363,22cyclictest35966irq/58-eth021:30:133
2669299364,4cyclictest131rcu_preempt19:25:120
2669299363,3cyclictest131rcu_preempt21:20:120
2672799353,18cyclictest35966irq/58-eth023:55:123
2671899355,20cyclictest0-21swapper/222:40:142
2671899351,29cyclictest930-21snmpd22:15:132
2669299352,24cyclictest17026-21ssh23:35:130
2672799345,18cyclictest35966irq/58-eth023:50:133
2669299341,27cyclictest24912-21ssh22:10:130
2669299341,27cyclictest24912-21ssh22:10:120
2672799339,15cyclictest35966irq/58-eth023:35:133
2671899333,20cyclictest0-21swapper/221:35:122
2671899333,20cyclictest0-21swapper/221:35:122
26706993320,5cyclictest0-21swapper/119:40:151
2672799321,13cyclictest35966irq/58-eth022:20:133
2671899329,9cyclictest23575-21ssh22:45:122
2671899323,22cyclictest27346irq/40-dwc2_hso19:55:132
2669299327,18cyclictest3134-21diskmemload21:40:170
2669299320,23cyclictest15480-21rm00:10:120
2669299320,23cyclictest15480-21rm00:10:120
2671899317,6cyclictest27-21ksoftirqd/223:11:332
2671899313,21cyclictest21626-21kthreadcore21:50:172
2671899311,16cyclictest12319-21diskmemload23:00:122
2669299313,22cyclictest31237-21ssh23:25:110
26727993014,7cyclictest0-21swapper/319:10:143
2670699302,22cyclictest21841-21cut21:50:171
2672799293,15cyclictest0-21swapper/300:30:113
2671899296,13cyclictest27-21ksoftirqd/222:12:152
2671899296,13cyclictest27-21ksoftirqd/222:12:152
2671899294,15cyclictest27-21ksoftirqd/223:39:182
2671899291,13cyclictest73-21kswapd022:55:122
2671899291,13cyclictest73-21kswapd022:55:122
2669299291,19cyclictest32712-21ssh22:50:140
2669299291,19cyclictest32712-21ssh22:50:140
2671899287,9cyclictest27-21ksoftirqd/221:25:462
2671899286,9cyclictest27-21ksoftirqd/200:19:092
2671899286,9cyclictest27-21ksoftirqd/200:19:092
2671899284,15cyclictest27346irq/40-dwc2_hso19:50:132
2671899282,19cyclictest0-21swapper/220:05:132
2671899281,18cyclictest12044-21rm23:50:172
2669299287,12cyclictest6519-21ssh00:05:160
2669299282,14cyclictest7720-21ssh21:25:130
26692992821,3cyclictest12319-21diskmemload00:35:100
2672799272,15cyclictest35966irq/58-eth021:50:113
2671899279,12cyclictest0-21swapper/223:42:292
2671899278,8cyclictest27-21ksoftirqd/200:02:372
2671899275,8cyclictest27-21ksoftirqd/223:15:552
2671899272,21cyclictest4753-21cat00:40:022
2671899272,19cyclictest0-21swapper/221:00:132
2670699272,17cyclictest24457-21sh00:15:131
2670699272,17cyclictest24457-21sh00:15:131
26706992721,3cyclictest10200-21apt-get20:00:121
26706992721,3cyclictest10200-21apt-get20:00:121
2669299275,11cyclictest0-21swapper/022:35:120
2669299273,17cyclictest19280-21ssh00:30:120
2672799262,13cyclictest35966irq/58-eth022:15:123
2671899269,13cyclictest20801-21ssh23:55:122
2671899268,11cyclictest27-21ksoftirqd/221:46:472
2671899266,9cyclictest27-21ksoftirqd/223:48:052
2671899266,9cyclictest27-21ksoftirqd/222:01:392
2671899266,8cyclictest27-21ksoftirqd/222:23:152
2671899266,11cyclictest27-21ksoftirqd/223:26:392
2671899266,11cyclictest27-21ksoftirqd/200:34:332
2671899264,4cyclictest271ksoftirqd/220:35:202
2671899263,16cyclictest27346irq/40-dwc2_hso20:25:132
2671899262,21cyclictest23755-21kthreadcore21:15:202
2671899262,21cyclictest1327-21fschecks_time00:20:162
2671899262,18cyclictest0-21swapper/221:30:132
2671899262,16cyclictest10546-21kthreadcore21:05:212
2671899262,10cyclictest27-21ksoftirqd/200:10:182
2671899262,10cyclictest27-21ksoftirqd/200:10:182
2671899261,22cyclictest16561-21proc_pri21:10:272
2670699261,21cyclictest22213-21ssh23:20:141
2669299269,4cyclictest131rcu_preempt23:00:210
2669299262,16cyclictest10637-21ssh22:20:130
2669299261,18cyclictest29842-21ssh00:00:140
2671899253,4cyclictest271ksoftirqd/221:23:592
2671899253,13cyclictest27-21ksoftirqd/222:50:202
2671899253,13cyclictest27-21ksoftirqd/222:50:202
2671899253,11cyclictest27346irq/40-dwc2_hso20:20:122
2671899252,5cyclictest271ksoftirqd/222:25:132
2671899252,4cyclictest271ksoftirqd/200:06:292
2671899252,20cyclictest29148-21munin-plugin-st20:20:012
2671899252,14cyclictest31184-21ssh23:05:282
26718992514,3cyclictest26834-21apt-get19:10:132
2671899251,12cyclictest2921266ssh23:23:032
2671899250,19cyclictest1768-21ls20:55:272
2671899250,19cyclictest1768-21ls20:55:272
2670699259,10cyclictest0-21swapper/121:10:131
2670699258,9cyclictest21-21ksoftirqd/122:25:091
2670699258,9cyclictest21-21ksoftirqd/100:26:261
2670699257,9cyclictest21-21ksoftirqd/123:19:271
2670699257,9cyclictest21-21ksoftirqd/122:13:561
2670699257,9cyclictest21-21ksoftirqd/122:13:561
2670699257,9cyclictest21-21ksoftirqd/121:46:541
26706992510,6cyclictest0-21swapper/122:34:471
2672799241,14cyclictest35966irq/58-eth023:15:113
26727992410,8cyclictest0-21swapper/323:45:133
2671899244,8cyclictest27-21ksoftirqd/219:25:272
2671899244,2cyclictest131rcu_preempt22:05:222
2671899244,13cyclictest27-21ksoftirqd/221:59:122
2671899242,19cyclictest14288-21grep00:25:292
2671899242,18cyclictest28847-21irqrtprio19:45:172
2671899242,14cyclictest17008-21kthreadcore20:40:192
2671899241,4cyclictest27-21ksoftirqd/219:20:212
2671899241,19cyclictest22826-21kthreadcore20:10:212
2670699248,4cyclictest21-21ksoftirqd/123:58:421
2670699247,9cyclictest21-21ksoftirqd/100:10:401
2670699247,9cyclictest21-21ksoftirqd/100:10:401
2670699247,8cyclictest21-21ksoftirqd/122:38:321
2670699246,9cyclictest21-21ksoftirqd/122:00:401
2670699246,3cyclictest131rcu_preempt21:17:391
2670699245,6cyclictest21-21ksoftirqd/121:25:231
2670699245,5cyclictest21-21ksoftirqd/100:30:291
2670699244,4cyclictest21-21ksoftirqd/120:05:171
26706992411,4cyclictest21-21ksoftirqd/121:56:261
26706992410,8cyclictest0-21swapper/122:20:591
2669299244,18cyclictest151rcuc/022:55:120
2669299244,18cyclictest151rcuc/022:55:120
2669299242,16cyclictest844-21munin-node19:45:130
2671899234,6cyclictest131rcu_preempt21:40:032
2671899234,16cyclictest27151-21grep19:40:292
2671899234,13cyclictest27-21ksoftirqd/220:45:182
2671899232,8cyclictest27-21ksoftirqd/220:30:282
2671899232,17cyclictest32527-21grep22:30:262
2671899232,14cyclictest14328-21ssh22:40:022
2671899232,12cyclictest10874-21kthreadcore23:30:212
2670699239,4cyclictest21-21ksoftirqd/123:31:091
2670699237,4cyclictest21-21ksoftirqd/123:07:111
2670699236,9cyclictest21-21ksoftirqd/123:54:161
2670699236,9cyclictest21-21ksoftirqd/121:41:221
2670699236,8cyclictest21-21ksoftirqd/123:39:051
2670699236,8cyclictest21-21ksoftirqd/122:49:561
2670699236,4cyclictest21-21ksoftirqd/123:15:011
2670699235,8cyclictest21-21ksoftirqd/122:15:411
2670699234,9cyclictest21-21ksoftirqd/119:35:261
2670699234,3cyclictest131rcu_preempt00:08:061
2670699231,17cyclictest9311-21ssh22:55:121
2670699231,17cyclictest9311-21ssh22:55:121
2670699231,16cyclictest18382-21sh23:00:131
26692992312,8cyclictest2249-21cron21:40:010
26692992312,8cyclictest2249-21cron21:40:010
2672799222,13cyclictest0-21swapper/322:30:133
2671899223,12cyclictest27346irq/40-dwc2_hso20:50:142
2671899222,17cyclictest17202-21ps19:30:252
2670699226,8cyclictest21-21ksoftirqd/123:26:181
2670699226,3cyclictest131rcu_preempt22:51:271
2670699226,3cyclictest131rcu_preempt22:51:271
2670699225,8cyclictest21-21ksoftirqd/121:22:191
2670699225,8cyclictest21-21ksoftirqd/120:55:141
2670699225,8cyclictest21-21ksoftirqd/120:55:141
2670699225,3cyclictest131rcu_preempt20:50:221
2670699225,3cyclictest131rcu_preempt00:36:451
2670699221,15cyclictest23068-21kworker/1:219:55:141
2669299224,12cyclictest0-21swapper/019:20:130
2669299221,16cyclictest18246-21ssh22:40:270
2672799213,13cyclictest0-21swapper/300:15:143
2672799213,13cyclictest0-21swapper/300:15:143
2672799211,17cyclictest35966irq/58-eth019:15:123
26727992111,6cyclictest35966irq/58-eth022:55:253
26727992111,6cyclictest35966irq/58-eth022:55:253
2671899212,16cyclictest14042-21unixbench_multi20:00:272
2671899212,16cyclictest14042-21unixbench_multi20:00:262
2670699216,9cyclictest21-21ksoftirqd/100:21:131
2670699215,8cyclictest21-21ksoftirqd/100:02:581
2670699214,3cyclictest21-21ksoftirqd/119:45:231
2670699213,3cyclictest21-21ksoftirqd/119:25:151
2670699212,4cyclictest21-21ksoftirqd/119:10:281
2670699212,16cyclictest21141-21fschecks_count20:45:151
2670699212,15cyclictest16125-21tr22:05:171
2670699211,5cyclictest21-21ksoftirqd/123:40:281
2669299216,5cyclictest12-21ksoftirqd/020:40:010
26692992111,7cyclictest12327-21apt-get21:10:050
2669299211,16cyclictest555-21apt-get19:50:130
2669299211,15cyclictest28105-21ls19:45:020
2669299211,11cyclictest4405-21tr23:45:190
2672799205,10cyclictest35966irq/58-eth023:40:113
2672799202,12cyclictest35966irq/58-eth020:55:123
2672799202,12cyclictest35966irq/58-eth020:55:123
2672799202,11cyclictest0-21swapper/320:20:143
26727992011,7cyclictest0-21swapper/321:13:413
26727992010,6cyclictest0-21swapper/321:46:593
2671899203,12cyclictest27-21ksoftirqd/219:39:472
26718992010,7cyclictest31959-21irqcore19:15:162
2670699204,4cyclictest21-21ksoftirqd/119:20:151
2670699203,14cyclictest7566-21wc21:05:011
2670699203,14cyclictest10810-21cut21:05:211
2670699202,4cyclictest21-21ksoftirqd/120:20:181
2670699202,15cyclictest11276-21latency_hist20:35:011
2670699201,16cyclictest3680-21kthreadcore20:25:201
2670699201,16cyclictest26603-21kthreadcore20:15:191
2670699201,16cyclictest21463-21kthreadcore20:10:201
2669299207,10cyclictest24376-21latency_hist20:15:020
2669299202,14cyclictest5234-21ntp_states22:15:250
2669299202,13cyclictest844-21munin-node23:30:130
26692992011,6cyclictest11164-21cron20:35:010
2669299201,11cyclictest16776-21/usr/sbin/munin21:30:250
2672799199,8cyclictest0-21swapper/323:27:193
2672799193,12cyclictest0-21swapper/319:35:143
2672799191,15cyclictest0-21swapper/322:35:153
2672799191,12cyclictest0-21swapper/321:00:133
2672799191,12cyclictest0-21swapper/320:50:293
2670699191,15cyclictest14189-21sed19:30:161
2670699191,15cyclictest12213-21kthreadcore20:35:191
2669299193,12cyclictest6388-21rm22:00:010
2669299191,9cyclictest10955-21tr21:05:210
2669299190,15cyclictest24635-21cpuspeed20:15:120
2672799189,6cyclictest35966irq/58-eth021:35:113
2672799189,6cyclictest35966irq/58-eth021:35:113
2672799189,6cyclictest0-21swapper/322:41:093
2672799189,6cyclictest0-21swapper/322:06:123
2672799189,4cyclictest0-21swapper/300:08:433
2672799182,8cyclictest35966irq/58-eth019:45:113
2672799182,10cyclictest0-21swapper/319:40:153
2672799182,10cyclictest0-21swapper/319:25:053
2672799181,9cyclictest0-21swapper/319:20:143
2672799181,12cyclictest0-21swapper/321:55:133
26727991810,6cyclictest0-21swapper/322:27:593
26727991810,6cyclictest0-21swapper/322:02:223
26727991810,6cyclictest0-21swapper/321:15:373
26727991810,6cyclictest0-21swapper/300:04:153
26727991810,4cyclictest0-21swapper/323:10:133
26727991810,3cyclictest0-21swapper/322:53:563
26727991810,3cyclictest0-21swapper/322:53:563
2670699184,8cyclictest21-21ksoftirqd/119:50:151
2670699181,11cyclictest2491-21latency19:15:221
2669299189,6cyclictest30371-21cron20:55:010
2669299183,8cyclictest0-21swapper/020:00:120
2669299183,8cyclictest0-21swapper/020:00:120
2669299181,9cyclictest21145-21tr19:35:210
2669299181,13cyclictest1810-21apt-get20:25:120
2672799179,6cyclictest0-21swapper/321:43:543
2672799172,8cyclictest0-21swapper/320:00:123
2672799172,8cyclictest0-21swapper/320:00:123
2672799172,10cyclictest0-21swapper/320:35:143
26727991711,3cyclictest0-21swapper/320:40:123
2671899171,8cyclictest26743-21latency_hist19:10:012
2670699175,3cyclictest131rcu_preempt20:40:231
2669299179,6cyclictest20768-21sshd20:44:170
2669299177,6cyclictest0-21swapper/020:45:120
2669299175,9cyclictest23318-21sh23:20:200
2669299174,2cyclictest131rcu_preempt20:25:020
2669299173,11cyclictest704-21cron00:20:010
2669299173,11cyclictest704-21cron00:20:010
2669299172,2cyclictest0-21swapper/020:05:210
2672799161,9cyclictest0-21swapper/320:30:253
2669299167,6cyclictest6676-21cut19:55:200
2669299164,9cyclictest26742-21latency_hist19:10:010
2669299161,11cyclictest3408-21irqcore21:00:170
2669299161,11cyclictest32444-21kthreadcore20:55:200
2669299161,11cyclictest32444-21kthreadcore20:55:200
2669299161,10cyclictest151rcuc/021:45:110
2672799158,5cyclictest0-21swapper/319:50:253
2672799141,8cyclictest0-21swapper/320:15:143
2672799141,11cyclictest0-21swapper/319:30:183
2672799131,7cyclictest0-21swapper/320:45:153
2672799131,6cyclictest0-21swapper/319:55:163
2670699130,10cyclictest26735-21latency_hist19:10:011
2672799122,7cyclictest0-21swapper/320:30:013
2672799112,5cyclictest0-21swapper/320:10:133
2672799111,8cyclictest0-21swapper/320:05:163
2672799111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional