You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 10:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Wed Jan 07, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1070099552,9cyclictest131rcu_preempt21:00:130
1072599502,35cyclictest0-21swapper/200:10:132
1070099501,43cyclictest14893-21ssh21:20:140
1071199452,26cyclictest32382-21ssh21:30:141
1071199441,38cyclictest11418-21ssh21:55:141
1072599436,17cyclictest27-21ksoftirqd/219:10:122
1070099432,34cyclictest23620-21ssh21:25:120
1070099432,34cyclictest23620-21ssh21:25:120
1070099422,31cyclictest4573-21ssh23:05:110
1071199412,32cyclictest21008-21ssh00:10:141
1070099415,30cyclictest29850-21ssh21:10:140
1070099413,31cyclictest12300-21ssh00:05:130
10711994020,17cyclictest21-21ksoftirqd/100:20:131
1070099393,29cyclictest3577-21ssh00:00:110
1070099390,32cyclictest32226-21ssh00:35:140
10725993823,10cyclictest0-21swapper/222:40:132
10734993724,9cyclictest28366irq/58-eth022:45:143
1072599373,23cyclictest924-21snmpd22:20:152
1070099371,27cyclictest24782-21ssh23:35:140
1073499362,19cyclictest28366irq/58-eth022:30:133
1072599362,28cyclictest14223-21ssh22:32:082
1071199363,28cyclictest29448-21diskmemload22:10:141
10711993620,13cyclictest633-21Async12:11:531
1071199361,24cyclictest23538-21ssh00:30:131
1071199352,22cyclictest23008-21ssh22:20:141
1071199352,22cyclictest13348-21ssh23:10:141
1071199352,22cyclictest13348-21ssh23:10:141
1070099353,22cyclictest7208-21ssh23:25:130
10700993514,13cyclictest5318-1kworker/u9:122:30:130
10734993420,12cyclictest28366irq/58-eth021:45:103
10734993414,13cyclictest28366irq/58-eth021:20:163
1071199343,22cyclictest10770-21ssh22:50:131
1072599322,16cyclictest6057-21ssh00:20:132
10711993224,5cyclictest7195-21ssh23:25:121
1073499312,19cyclictest28366irq/58-eth021:40:133
1072599313,18cyclictest22346irq/40-dwc2_hso20:35:122
1071199311,24cyclictest29744-21ssh00:15:151
1071199311,24cyclictest16037-21ssh23:30:141
1070099313,17cyclictest5436-21kworker/u8:122:05:120
1070099312,22cyclictest6775-21ssh21:32:580
1073499303,20cyclictest28366irq/58-eth021:55:133
1073499303,18cyclictest28366irq/58-eth022:50:133
1072599302,13cyclictest633-21Async12:11:532
1072599302,13cyclictest633-21Async12:11:532
1071199302,21cyclictest6163-21ssh21:15:141
1070099301,21cyclictest27290-21ssh23:55:150
1070099301,18cyclictest31788-21sh22:25:130
10734992916,8cyclictest28366irq/58-eth021:50:143
1073499291,13cyclictest28366irq/58-eth023:15:133
1072599297,16cyclictest633-21Async12:11:532
1072599291,21cyclictest2596-21ssh21:50:142
1071199298,13cyclictest10691-21cyclictest19:40:121
1071199295,19cyclictest17476-21ssh21:40:141
1073499282,16cyclictest28366irq/58-eth023:30:133
1072599282,19cyclictest0-21swapper/220:15:142
1072599282,16cyclictest16036-21sh23:30:142
1073499272,18cyclictest28366irq/58-eth021:30:123
1072599272,17cyclictest22346irq/40-dwc2_hso19:45:132
1071199273,19cyclictest30919-21ssh23:20:151
1071199271,19cyclictest2595-21ssh21:50:141
1071199271,10cyclictest27280-21ssh23:55:141
1070099273,19cyclictest23557-21ssh00:30:150
1072599267,13cyclictest0-21swapper/221:30:122
1072599263,14cyclictest22499-21/usr/sbin/munin23:15:182
1072599262,19cyclictest2-21kthreadd21:36:582
1072599262,14cyclictest22346irq/40-dwc2_hso19:50:122
1072599262,10cyclictest0-21swapper/220:45:132
10700992611,11cyclictest12-21ksoftirqd/000:13:580
10700992610,11cyclictest0-21swapper/023:40:130
1073499257,13cyclictest28366irq/58-eth000:35:143
1072599254,16cyclictest22346irq/40-dwc2_hso21:05:142
1072599253,14cyclictest22346irq/40-dwc2_hso20:25:152
1072599253,14cyclictest22346irq/40-dwc2_hso20:25:152
1072599252,20cyclictest23637-21sh21:25:132
1072599252,20cyclictest23637-21sh21:25:132
1072599251,17cyclictest1996-21ssh22:45:142
1072599251,13cyclictest32015-21latency_hist00:35:022
1072599250,18cyclictest7731-21ls19:40:162
1070099252,15cyclictest10760-21ssh22:50:120
10734992414,6cyclictest28366irq/58-eth022:10:133
10734992414,6cyclictest28366irq/58-eth022:05:123
1072599248,12cyclictest5959-21irqrtprio22:10:182
1072599247,6cyclictest23322-21latency_hist00:30:022
1072599243,18cyclictest22346irq/40-dwc2_hso19:25:152
1072599243,16cyclictest0-21swapper/200:00:122
1072599243,12cyclictest967-21ls21:10:242
1072599242,18cyclictest0-21swapper/221:15:142
1072599242,17cyclictest73-21kswapd021:45:122
1072599242,14cyclictest18520-21ssh23:50:122
1072599242,12cyclictest19062-21kthreadcore21:40:212
10725992415,6cyclictest0-21swapper/222:55:132
10725992412,10cyclictest18319-21ssh22:53:542
1072599241,15cyclictest16877-21ssh22:35:122
10725992411,10cyclictest27541-21kthreadcore23:35:222
10725992410,6cyclictest6314-21sshd23:40:582
1071199245,8cyclictest0-21swapper/119:10:131
1071199242,17cyclictest29448-21diskmemload00:07:311
1071199242,16cyclictest26177-21ssh21:45:101
1071199242,14cyclictest21-21ksoftirqd/119:50:121
10711992414,7cyclictest4205-21cron23:05:021
10711992412,9cyclictest1476-21sed21:10:301
1073499232,13cyclictest0-21swapper/321:15:143
1073499232,12cyclictest28366irq/58-eth023:10:133
1073499232,12cyclictest28366irq/58-eth023:10:133
1073499232,10cyclictest0-21swapper/319:20:143
1073499231,12cyclictest28366irq/58-eth000:30:133
1072599231,12cyclictest5526-21cut23:05:182
1071199232,3cyclictest131rcu_preempt22:30:021
1071199231,9cyclictest8065-21/usr/sbin/munin22:30:251
1071199231,17cyclictest22132-21ssh23:15:141
1070099233,14cyclictest26364-21ssh22:40:180
1073499222,14cyclictest0-21swapper/323:50:113
1072599229,10cyclictest16589-21tr22:15:212
1072599227,11cyclictest14676-21kthreadcore00:05:212
1072599226,13cyclictest22038-21sort19:55:142
1072599223,15cyclictest22346irq/40-dwc2_hso19:35:112
1072599223,12cyclictest22346irq/40-dwc2_hso19:30:132
1072599221,14cyclictest13862-21ls23:45:292
1072599221,11cyclictest32686-21tr00:15:242
1072599221,11cyclictest30562-21cut23:00:222
1072599221,11cyclictest30429-21latency_hist23:20:022
1072599221,11cyclictest18534-21ntp_192.168.11421:20:252
1071199229,7cyclictest0-21swapper/121:05:131
1071199228,4cyclictest131rcu_preempt23:50:131
1071199224,5cyclictest21-21ksoftirqd/119:35:261
1071199222,14cyclictest9867-21ssh23:45:141
10711992213,3cyclictest29448-21diskmemload23:42:071
10711992212,7cyclictest5315-21cut00:00:211
1071199221,18cyclictest23430-21ls21:25:021
1071199221,18cyclictest12339-21ls20:15:231
1070099222,15cyclictest29448-21diskmemload23:33:580
10700992216,4cyclictest0-21swapper/022:55:140
10700992213,6cyclictest0-21swapper/021:35:260
1070099221,16cyclictest10853-21apt-get19:10:130
1073499219,8cyclictest28366irq/58-eth000:00:253
1072599218,6cyclictest4040-21date20:10:022
1072599217,11cyclictest21777-21cat22:00:222
1072599214,9cyclictest20523-21irqrtprio21:00:202
1072599213,9cyclictest5727-21kthreadcore20:10:202
1072599212,16cyclictest261rcuc/220:55:122
1072599212,16cyclictest22346irq/40-dwc2_hso20:30:142
1072599211,9cyclictest145566ssh00:35:232
1072599211,5cyclictest31770-21find22:25:122
1072599211,16cyclictest16363-21kthreadcore19:15:192
1072599211,12cyclictest29359-21/usr/sbin/munin22:05:172
1072599211,10cyclictest7604-21sort23:25:182
1072599211,10cyclictest4239-21sendmail_mailqu20:40:262
1072599211,10cyclictest28147-21kthreadcore23:55:202
1071199219,9cyclictest31392-21cron20:05:011
1071199214,9cyclictest0-21swapper/120:45:121
10711992111,7cyclictest24018-21sh22:00:281
1071199211,16cyclictest32659-21rm22:05:241
1071199211,11cyclictest0-21swapper/100:38:501
1070099217,9cyclictest0-21swapper/022:00:150
1070099211,18cyclictest29877-21uniq00:15:170
1070099211,16cyclictest8954-21apt-get20:15:120
1070099211,15cyclictest1986-21ssh22:45:130
1070099211,14cyclictest32297-21kthreadcore20:05:210
10700992110,8cyclictest16912-21cron19:50:010
10734992010,6cyclictest28366irq/58-eth022:55:283
1072599203,8cyclictest21349-21kthreadcore19:20:202
1072599202,10cyclictest16205-21kthreadcore20:20:202
1072599201,15cyclictest29110-21kthreadcore20:00:212
1071199207,10cyclictest28504-21sh23:35:261
1071199201,3cyclictest131rcu_preempt22:15:131
10711992013,4cyclictest0-21swapper/119:20:011
10711992012,5cyclictest0-21swapper/120:20:161
10711992011,7cyclictest134-21systemd-journal22:46:491
10711992011,6cyclictest0-21swapper/120:40:211
1071199201,15cyclictest21985-21apt-get19:55:121
1071199201,11cyclictest19347-21kthreadcore22:35:201
1070099205,12cyclictest22116-21ssh23:15:130
1070099203,12cyclictest0-21swapper/023:20:150
1070099202,12cyclictest21680-21ssh23:14:580
1070099202,12cyclictest21680-21ssh23:14:580
1070099201,15cyclictest18655-21apt-get20:25:140
1070099201,15cyclictest18655-21apt-get20:25:140
10700992010,3cyclictest131rcu_preempt19:25:210
1070099200,17cyclictest18254-21ls00:25:230
1070099200,14cyclictest23086-21ls22:20:150
1073499193,14cyclictest0-21swapper/323:20:153
1073499192,6cyclictest0-21swapper/300:07:323
1073499192,11cyclictest0-21swapper/322:20:143
1073499191,13cyclictest0-21swapper/320:45:223
1073499191,12cyclictest0-21swapper/323:55:243
1072599192,14cyclictest10402-21cpuspeed20:50:132
1071199199,7cyclictest23996-21irqcore21:25:171
1071199199,7cyclictest23996-21irqcore21:25:171
1071199191,4cyclictest21-21ksoftirqd/119:30:161
10711991913,4cyclictest0-21swapper/120:55:271
10711991913,4cyclictest0-21swapper/120:35:181
10711991912,4cyclictest0-21swapper/120:10:021
10711991911,4cyclictest9268-21cut21:35:171
1071199191,14cyclictest19757-21ssh22:55:181
1071199190,10cyclictest17127-21latency_hist19:50:031
1070099199,7cyclictest27799-21cut21:45:180
1070099198,3cyclictest131rcu_preempt21:40:230
1070099197,9cyclictest6266-21grep00:20:170
1070099192,15cyclictest20607-21apt-get19:20:130
1070099192,13cyclictest14167-21ssh22:15:130
1070099191,7cyclictest16347-21users21:55:330
1070099191,15cyclictest2571-21apt-get21:50:130
1070099191,14cyclictest73-21kswapd023:50:120
10700991910,7cyclictest15263-21df_inode20:55:160
1073499188,8cyclictest0-21swapper/322:41:153
1073499187,8cyclictest0-21swapper/323:40:313
1073499183,10cyclictest0-21swapper/321:39:583
1073499183,10cyclictest0-21swapper/321:28:583
1073499183,10cyclictest0-21swapper/321:28:583
1073499182,11cyclictest28366irq/58-eth022:00:153
1073499182,10cyclictest0-21swapper/320:40:133
1073499182,10cyclictest0-21swapper/320:35:133
1073499182,10cyclictest0-21swapper/320:15:143
1073499181,14cyclictest28366irq/58-eth000:25:133
1073499181,13cyclictest0-21swapper/322:25:163
10734991810,5cyclictest28366irq/58-eth000:10:133
10711991811,5cyclictest0-21swapper/120:30:261
1071199181,14cyclictest28686-21irqcore23:00:201
1071199181,13cyclictest14712-21apt-get00:25:121
1071199180,15cyclictest10327-21apt-get20:50:111
1070099189,7cyclictest12588-21cut23:45:220
1070099188,7cyclictest6867-21kernelversion21:15:190
1070099187,8cyclictest26926-21df_inode20:00:150
1070099183,12cyclictest2869-21grep19:35:160
1070099182,11cyclictest16866-21ssh22:35:120
1070099182,11cyclictest0-21swapper/020:10:040
10700991812,3cyclictest29220-21cut23:00:200
10700991810,6cyclictest17782-21irqstats19:50:180
1073499179,6cyclictest0-21swapper/323:36:473
1073499179,6cyclictest0-21swapper/323:03:483
1073499179,6cyclictest0-21swapper/322:35:303
1073499179,6cyclictest0-21swapper/300:15:233
1073499179,5cyclictest28366irq/58-eth023:45:253
1073499179,3cyclictest0-21swapper/323:09:093
1073499178,6cyclictest28366irq/58-eth022:15:133
1073499177,8cyclictest0-21swapper/321:12:023
1073499172,9cyclictest0-21swapper/319:45:133
1073499172,10cyclictest0-21swapper/320:50:123
1073499172,10cyclictest0-21swapper/319:50:133
1073499172,10cyclictest0-21swapper/319:10:143
1073499171,13cyclictest0-21swapper/323:25:303
1073499171,12cyclictest0-21swapper/320:25:253
1073499171,12cyclictest0-21swapper/320:25:243
1073499171,10cyclictest0-21swapper/319:30:163
1071199172,12cyclictest3831-21apt-get20:10:051
1071199172,12cyclictest30003-21apt19:30:011
1071199171,13cyclictest23099-21kthreadcore19:20:221
1071199171,12cyclictest23404-21cat21:00:251
1071199171,11cyclictest23233-21cron20:30:011
1071199171,11cyclictest23233-21cron20:30:011
1070099174,10cyclictest11016-21kthreadcore20:50:170
10700991711,4cyclictest0-21swapper/022:10:160
10700991711,4cyclictest0-21swapper/020:45:180
10700991711,4cyclictest0-21swapper/019:40:170
1070099171,11cyclictest4032-21missed_timers20:40:220
1070099171,11cyclictest14005-21fschecks_time20:20:160
1070099170,15cyclictest15797-21cut19:15:150
1073499163,7cyclictest0-21swapper/300:20:163
1073499161,12cyclictest0-21swapper/321:05:143
1071199161,10cyclictest10763-21latency_hist19:10:011
1070099169,4cyclictest12-21ksoftirqd/019:55:120
1070099168,5cyclictest26819-21cat20:30:230
1070099168,5cyclictest0-21swapper/020:35:140
1070099164,10cyclictest26683-21kthreadcore21:05:200
1070099161,13cyclictest32118-21sort19:30:200
1073499152,10cyclictest0-21swapper/319:40:143
1073499151,8cyclictest0-21swapper/320:10:123
1073499149,3cyclictest0-21swapper/321:00:153
1072599141,9cyclictest10759-21latency_hist19:10:022
1073499138,3cyclictest0-21swapper/320:59:583
1073499131,9cyclictest0-21swapper/320:05:193
1073499131,8cyclictest0-21swapper/320:00:203
1073499131,8cyclictest0-21swapper/319:25:313
1070099131,9cyclictest10691-21cyclictest19:10:010
1073499127,3cyclictest0-21swapper/319:35:583
1073499121,9cyclictest0-21swapper/319:55:303
1073499121,9cyclictest0-21swapper/319:15:153
1073499121,8cyclictest0-21swapper/320:20:213
1073499121,5cyclictest0-21swapper/320:30:283
1073499101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional