You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 13:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Apr 07, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17338998766,18cyclictest28366irq/58-eth023:50:133
1732199572,35cyclictest12511-21ssh21:15:141
1732199565,8cyclictest131rcu_preempt23:00:131
1733099531,35cyclictest21616-21ssh21:20:152
1732199532,9cyclictest21-21ksoftirqd/120:10:121
1732199504,36cyclictest24388-21ssh23:45:131
17321995020,23cyclictest22218-1kworker/u9:122:50:131
1733899495,26cyclictest28366irq/58-eth022:00:133
1732199496,6cyclictest131rcu_preempt22:40:121
17330994822,21cyclictest0-21swapper/222:35:132
1732199488,35cyclictest1929-21ssh21:45:101
1732199476,7cyclictest131rcu_preempt19:45:111
1732199462,33cyclictest17449-21ssh00:35:121
17330994524,9cyclictest9200-21apt-get19:35:132
17330994424,16cyclictest16222-21apt-get21:35:092
1732199443,33cyclictest20094-21ssh21:55:141
17321994316,21cyclictest131rcu_preempt22:15:111
1731399436,33cyclictest18517-21ssh22:30:130
1733899424,24cyclictest28366irq/58-eth020:50:113
1733899421,25cyclictest28366irq/58-eth019:20:113
1732199425,28cyclictest29685-21ssh23:30:141
1731399424,27cyclictest27544-21ssh22:35:130
1732199412,36cyclictest28238-21ssh00:05:141
1731399412,31cyclictest6283-21ssh23:35:140
1733899402,23cyclictest28366irq/58-eth022:10:133
1732199401,34cyclictest16238-21ssh21:35:111
1732199401,29cyclictest29124-21ssh22:00:131
1733899395,20cyclictest28366irq/58-eth021:25:133
1733899392,25cyclictest28366irq/58-eth000:35:113
17330993914,21cyclictest0-21swapper/221:15:142
1733899384,20cyclictest0-21swapper/300:10:133
1733899381,25cyclictest28366irq/58-eth023:25:143
1732199382,28cyclictest411-21ssh22:20:141
1731399383,30cyclictest25340-21ssh21:40:130
1733899375,23cyclictest28366irq/58-eth021:55:133
1732199373,28cyclictest25945-21ssh23:10:121
17321993723,8cyclictest21-21ksoftirqd/100:00:241
17321993721,9cyclictest21-21ksoftirqd/100:23:351
17330993627,6cyclictest261rcuc/222:45:142
17330993627,6cyclictest261rcuc/222:45:142
17330993617,13cyclictest0-21swapper/223:35:132
1732199362,4cyclictest131rcu_preempt21:30:121
1732199362,19cyclictest21-21ksoftirqd/121:10:131
1731399363,25cyclictest13836-21ssh00:15:120
1733099357,9cyclictest4297-21processes20:00:272
1732199351,31cyclictest11593-21ssh23:20:111
1733899342,22cyclictest28366irq/58-eth022:35:123
1733099346,19cyclictest8793-21rm23:53:582
1733099344,14cyclictest20328-21grep00:35:202
1733099342,16cyclictest25346-21ssh21:40:142
1733099334,10cyclictest24435-21usage_Anybus21:55:302
1733099333,14cyclictest30866-21latency_hist21:05:022
17330993327,3cyclictest22349-21apt-get19:15:122
17330993322,8cyclictest390-21apt-get22:20:142
1733099332,14cyclictest13672-21ntp_192.168.11423:55:242
17330993320,7cyclictest630-21tcf-agent00:25:132
17330993320,10cyclictest28208-21apt-get00:05:142
17330993320,10cyclictest22169-21apt-get22:50:142
1732199334,21cyclictest21-21ksoftirqd/100:10:141
1732199333,21cyclictest3105-21diskmemload23:40:121
1732199333,21cyclictest3105-21diskmemload23:40:121
17330993221,7cyclictest22346irq/40-dwc2_hso00:10:132
17330993218,10cyclictest11581-21apt-get23:20:112
17330993213,8cyclictest0-21swapper/223:25:132
1731399324,3cyclictest131rcu_preempt00:13:100
1731399322,22cyclictest22352-21sshd19:15:130
1733899312,15cyclictest28366irq/58-eth022:40:113
1733099316,21cyclictest0-21swapper/223:10:132
1733099314,8cyclictest28032-21cat19:50:292
1733099314,7cyclictest21635-21ssh22:12:562
17330993120,3cyclictest15326-21apt-get23:40:132
17330993120,3cyclictest15326-21apt-get23:40:132
17330993110,14cyclictest0-21swapper/220:10:122
1731399311,2cyclictest131rcu_preempt23:40:120
1731399311,2cyclictest131rcu_preempt23:40:110
1733899304,15cyclictest28366irq/58-eth023:05:143
1733899301,15cyclictest28366irq/58-eth022:20:143
1733099305,4cyclictest892-21ntp_states23:30:272
1733099305,11cyclictest9365-21ssh23:17:502
1733099304,14cyclictest10915-21cut22:25:202
1733099304,10cyclictest1191-21kthreadcore19:25:182
1733099302,10cyclictest16989-21ssh23:05:142
1733099301,15cyclictest23773-21ssh22:15:122
17330993010,9cyclictest0-21swapper/219:10:112
17313993014,11cyclictest25927-21apt-get23:10:120
1731399301,23cyclictest8078-21ssh23:53:060
1733899293,15cyclictest28366irq/58-eth023:55:133
1733899293,15cyclictest28366irq/58-eth022:30:133
17338992916,8cyclictest28366irq/58-eth021:15:153
1733099296,8cyclictest18873-21grep20:50:202
1733099294,13cyclictest18588-21ls19:45:012
1733099293,10cyclictest23282-21cat19:45:302
17330992922,4cyclictest3492-21apt-get21:10:122
1731399299,5cyclictest12-21ksoftirqd/023:30:260
1731399293,5cyclictest12-21ksoftirqd/021:20:020
1733899283,16cyclictest0-21swapper/323:36:243
1733899282,12cyclictest28366irq/58-eth022:15:113
17338992814,6cyclictest0-21swapper/322:50:133
17338992812,9cyclictest28366irq/58-eth023:30:143
1733099285,8cyclictest3105-21diskmemload21:53:542
1733099285,8cyclictest3105-21diskmemload21:53:542
1733099285,7cyclictest0-21swapper/223:49:162
1733099282,19cyclictest0-21swapper/222:05:152
1733099281,12cyclictest14537-21irqstats00:15:182
1732199285,16cyclictest1020-21ssh23:50:131
1732199284,16cyclictest10059-21ssh23:55:131
1732199282,21cyclictest6512-21rm00:28:131
1732199281,20cyclictest8406-21ssh00:30:121
1731399286,6cyclictest12-21ksoftirqd/022:40:280
1731399284,5cyclictest12-21ksoftirqd/000:35:250
1731399284,4cyclictest12-21ksoftirqd/000:30:020
1731399284,19cyclictest5776-21ssh22:05:150
1731399283,4cyclictest12-21ksoftirqd/021:25:170
1731399283,15cyclictest16974-21ssh23:05:130
1731399282,18cyclictest23604-21sshd19:50:130
17313992821,4cyclictest10352-21apt-get20:10:120
1733899271,16cyclictest28366irq/58-eth000:30:113
1733099276,15cyclictest0-21swapper/222:40:122
1733099275,14cyclictest3105-21diskmemload21:29:522
1733099273,12cyclictest18491-21/usr/sbin/munin22:30:262
17330992722,3cyclictest15244-21apt-get20:15:122
1733099272,13cyclictest11463-21rm23:00:242
1733099271,13cyclictest4541-21tr20:35:202
1732199274,3cyclictest131rcu_preempt23:35:191
1731399276,3cyclictest131rcu_preempt22:20:100
1731399275,5cyclictest151rcuc/021:35:050
1731399275,4cyclictest131rcu_preempt19:10:260
1731399275,4cyclictest12-21ksoftirqd/021:50:200
1731399275,4cyclictest12-21ksoftirqd/021:50:200
1731399274,6cyclictest12-21ksoftirqd/000:30:270
1731399274,5cyclictest12-21ksoftirqd/000:20:190
1731399273,6cyclictest12-21ksoftirqd/021:57:520
1731399273,5cyclictest12-21ksoftirqd/021:30:210
1731399273,5cyclictest12-21ksoftirqd/021:20:160
1731399272,19cyclictest31429-21ssh23:48:070
1731399272,19cyclictest24928-21sh00:01:350
1731399270,18cyclictest9439-21ssh22:25:130
1733899262,11cyclictest0-21swapper/300:20:143
1733099263,4cyclictest15123-21ntp_192.168.11420:45:252
1733099262,11cyclictest1919-21apt-get21:45:112
1733099262,10cyclictest2169-21ssh22:01:162
17330992610,8cyclictest0-21swapper/220:55:112
1732199266,15cyclictest21-21ksoftirqd/122:10:131
1732199266,11cyclictest17302-21cyclictest23:15:131
1731399266,7cyclictest12-21ksoftirqd/019:55:210
1731399265,5cyclictest12-21ksoftirqd/022:58:570
1731399264,6cyclictest12-21ksoftirqd/022:50:150
1731399264,5cyclictest12-21ksoftirqd/023:55:270
1731399264,4cyclictest12-21ksoftirqd/020:20:000
1731399263,19cyclictest0-21swapper/023:25:140
1731399262,5cyclictest12-21ksoftirqd/022:10:140
17313992619,5cyclictest151rcuc/022:00:060
17313992611,5cyclictest12-21ksoftirqd/021:00:180
1733899253,18cyclictest0-21swapper/300:18:343
1733899252,14cyclictest28366irq/58-eth019:45:113
1733099258,13cyclictest7655-21/usr/sbin/munin19:30:222
1733099254,9cyclictest1832-21cat20:30:292
1733099254,8cyclictest3105-21diskmemload00:21:092
1733099254,8cyclictest11484-21cat21:30:292
1733099253,9cyclictest28781-21kthreadcore19:20:192
1733099252,9cyclictest12782-21cat00:30:282
1733099252,6cyclictest9927-21usage_EthcTask20:05:312
1733099252,10cyclictest3883-21diskmemload22:55:492
17330992519,4cyclictest19448-21ssh00:00:172
17321992520,2cyclictest0-21swapper/122:25:121
1731399254,7cyclictest12-21ksoftirqd/020:30:140
1731399254,6cyclictest12-21ksoftirqd/021:45:170
1731399254,5cyclictest12-21ksoftirqd/023:20:140
1731399254,5cyclictest12-21ksoftirqd/021:10:150
1731399254,5cyclictest12-21ksoftirqd/020:00:250
1731399253,6cyclictest12-21ksoftirqd/023:03:010
1731399252,6cyclictest12-21ksoftirqd/000:05:130
1731399251,7cyclictest12-21ksoftirqd/020:25:230
1733099248,8cyclictest24723-21awk20:25:022
1733099246,15cyclictest518-21sendmail-msp20:00:012
1732199242,17cyclictest19575-21ssh00:16:351
1731399248,12cyclictest27147-21apt-get19:20:130
1731399246,4cyclictest12-21ksoftirqd/020:05:170
1731399246,3cyclictest131rcu_preempt23:15:130
1731399245,6cyclictest12-21ksoftirqd/019:25:220
1731399245,3cyclictest131rcu_preempt19:45:220
1731399244,3cyclictest131rcu_preempt22:15:120
1731399243,3cyclictest131rcu_preempt22:45:420
1731399243,3cyclictest131rcu_preempt22:45:410
17313992418,4cyclictest151rcuc/020:50:200
1733899231,14cyclictest0-21swapper/321:35:113
1733099239,12cyclictest11476-21sshd20:43:182
1731399234,5cyclictest12-21ksoftirqd/021:05:190
1731399234,5cyclictest12-21ksoftirqd/021:05:190
1731399234,5cyclictest12-21ksoftirqd/019:40:130
1731399234,3cyclictest131rcu_preempt20:40:140
1731399233,4cyclictest12-21ksoftirqd/020:35:190
1731399232,4cyclictest12-21ksoftirqd/019:35:250
1731399231,7cyclictest12-21ksoftirqd/020:20:150
1733899223,14cyclictest0-21swapper/323:48:073
17330992212,7cyclictest24940-21apt-get20:25:132
17321992211,9cyclictest5747-21apt-get22:05:141
17321992210,5cyclictest0-21swapper/120:05:281
1731399225,2cyclictest131rcu_preempt19:30:120
1731399224,3cyclictest131rcu_preempt21:00:020
1731399221,5cyclictest12-21ksoftirqd/020:45:130
1733899216,10cyclictest0-21swapper/300:03:083
1733899213,11cyclictest0-21swapper/323:15:143
1733099211,15cyclictest2405-21grep21:05:302
1733099211,15cyclictest2405-21grep21:05:302
1732199216,6cyclictest1059-21kthreadcore21:25:221
1732199212,3cyclictest131rcu_preempt19:20:131
1732199212,13cyclictest27638-21ssh22:35:141
1732199211,6cyclictest13511-21kthreadcore21:50:211
1732199211,6cyclictest13511-21kthreadcore21:50:211
1732199211,16cyclictest20188-21ls20:20:151
1732199211,14cyclictest24239-21ls23:25:241
1732199211,12cyclictest0-21swapper/121:42:031
1732199210,4cyclictest131rcu_preempt23:09:371
1731399213,6cyclictest12-21ksoftirqd/019:10:000
1733899203,12cyclictest0-21swapper/300:25:133
17338992012,6cyclictest28366irq/58-eth021:10:143
17338992012,5cyclictest0-21swapper/300:05:133
1733099208,9cyclictest17430-21latency_hist19:10:012
1732199205,6cyclictest743-21grep20:30:221
1732199200,10cyclictest131rcu_preempt22:58:211
1733899199,7cyclictest0-21swapper/321:46:073
1733899198,7cyclictest28366irq/58-eth023:10:123
1733899195,7cyclictest0-21swapper/322:05:153
1733899195,5cyclictest0-21swapper/321:05:133
1733899195,5cyclictest0-21swapper/321:05:133
1733899192,11cyclictest0-21swapper/321:00:153
17338991911,5cyclictest0-21swapper/322:55:233
1733899191,13cyclictest0-21swapper/320:15:153
1733899191,12cyclictest0-21swapper/321:20:163
1733899191,12cyclictest0-21swapper/320:05:273
17338991910,6cyclictest0-21swapper/323:40:293
17338991910,6cyclictest0-21swapper/323:40:293
1732199192,8cyclictest131rcu_preempt22:45:041
1732199192,8cyclictest131rcu_preempt22:45:041
1732199192,3cyclictest131rcu_preempt20:15:131
17321991911,5cyclictest24010-21cut21:20:231
1732199191,15cyclictest15043-21kthreadcore19:40:181
1732199191,12cyclictest0-21swapper/119:30:121
1732199190,3cyclictest131rcu_preempt20:35:231
1733899189,6cyclictest0-21swapper/323:00:303
1733899182,10cyclictest0-21swapper/320:30:123
1733899182,10cyclictest0-21swapper/320:00:143
1733899182,10cyclictest0-21swapper/319:40:133
1733899182,10cyclictest0-21swapper/319:35:043
1733899182,10cyclictest0-21swapper/319:25:133
1733899182,10cyclictest0-21swapper/319:10:123
1733899181,13cyclictest0-21swapper/321:50:283
1733899181,13cyclictest0-21swapper/321:50:283
1733899181,13cyclictest0-21swapper/320:45:013
17338991810,6cyclictest0-21swapper/323:23:423
17338991810,6cyclictest0-21swapper/321:42:453
1732199189,7cyclictest1-21systemd19:27:191
1732199189,2cyclictest21-21ksoftirqd/120:00:191
1732199187,9cyclictest13722-21cron19:40:011
1732199185,4cyclictest131rcu_preempt19:55:181
1732199182,3cyclictest131rcu_preempt20:45:231
1732199181,2cyclictest131rcu_preempt20:55:241
17321991812,4cyclictest0-21swapper/122:30:251
17321991811,4cyclictest21-21ksoftirqd/120:25:281
17321991811,3cyclictest21-21ksoftirqd/120:40:121
1733899179,6cyclictest0-21swapper/321:33:553
1733899172,9cyclictest28366irq/58-eth019:50:123
1733899172,10cyclictest0-21swapper/322:25:143
1733899172,10cyclictest0-21swapper/320:55:133
1733899172,10cyclictest0-21swapper/320:20:043
1733899172,10cyclictest0-21swapper/319:55:123
1732199175,9cyclictest18724-21kthreadcore20:50:201
1732199174,10cyclictest25465-21sed19:50:201
1732199171,14cyclictest29931-21grep21:00:301
17321991711,4cyclictest0-21swapper/121:05:281
17321991711,4cyclictest0-21swapper/121:05:281
1733899168,6cyclictest0-21swapper/322:47:343
1733899168,6cyclictest0-21swapper/322:47:343
1733899164,6cyclictest0-21swapper/319:30:113
1733899162,8cyclictest28366irq/58-eth019:15:113
1733899161,10cyclictest0-21swapper/320:45:143
1732199162,10cyclictest0-21swapper/119:15:131
1732199161,2cyclictest131rcu_preempt19:10:131
1733899149,3cyclictest0-21swapper/320:10:263
1732199141,9cyclictest17345-21latency_hist19:10:011
1733899131,7cyclictest0-21swapper/320:40:023
1733899131,6cyclictest0-21swapper/319:10:013
1733899112,5cyclictest0-21swapper/320:25:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional