You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 06:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Jul 19, 2025 00:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1054999644,46cyclictest24031-21ssh23:15:141
1054099583,44cyclictest31289-21ssh00:35:150
10540995833,21cyclictest328-21ssh22:25:110
1054099544,38cyclictest9288-21ssh23:25:130
1054099544,38cyclictest9288-21ssh23:25:130
1054099511,38cyclictest32231-21ssh21:30:130
1054099503,35cyclictest23392-21ssh21:25:120
10540994933,10cyclictest29835-21apt-get23:55:140
10540994817,20cyclictest0-21swapper/019:40:160
1056099474,35cyclictest23346irq/40-dwc2_hso23:05:142
1054099472,33cyclictest6402-21ssh23:05:140
10568994532,11cyclictest0-21swapper/323:10:113
10540994534,7cyclictest24023-21apt-get23:15:150
10540994430,9cyclictest21037-21apt-get23:50:170
1054099431,34cyclictest419-21rm23:20:130
10540994235,3cyclictest21670-21apt-get19:55:130
10560994112,17cyclictest27-21ksoftirqd/220:10:122
10540994116,8cyclictest20025-21ssh22:15:480
10540994012,11cyclictest13076-21ssh00:23:050
1056899398,20cyclictest0-21swapper/300:10:163
1056099392,21cyclictest32078-21ssh23:35:542
10540993913,10cyclictest6452-21diskstats00:00:160
10540993912,20cyclictest0-21swapper/023:40:140
10540993831,3cyclictest14449-21apt-get20:55:140
10540993830,4cyclictest30041-21apt-get19:30:140
10540993830,4cyclictest30041-21apt-get19:30:140
10540993816,8cyclictest17803-21users20:20:310
10540993816,8cyclictest17803-21users20:20:310
10540993813,10cyclictest32539-21ssh22:41:260
10540993813,10cyclictest14438-21ssh21:36:420
10540993812,8cyclictest3889-21ssh00:16:310
10540993715,11cyclictest27587-21ntp_192.168.11421:05:240
10540993713,10cyclictest3009-21ssh21:12:180
10540993713,10cyclictest17944-21ssh22:35:040
10540993712,10cyclictest16300-21sh22:33:170
10540993712,10cyclictest16300-21sh22:33:170
1056899363,17cyclictest32466irq/58-eth023:00:123
10540993629,4cyclictest3669-21apt-get20:10:120
10540993615,7cyclictest28748-21diskmemload00:31:330
10540993614,9cyclictest28495-21ssh22:58:410
10540993614,9cyclictest28495-21ssh22:58:410
10540993614,13cyclictest22468-21rm21:40:410
10540993613,5cyclictest12216-21apt-get22:50:130
10540993613,10cyclictest20895-21ssh00:07:520
10540993612,8cyclictest1659-21ssh21:49:000
10540993610,9cyclictest17480-21ssh21:56:280
1056899353,20cyclictest32466irq/58-eth021:10:143
1056099352,28cyclictest25667-21ssh23:33:542
1056099352,23cyclictest29981-21ssh23:00:132
10540993530,3cyclictest22876-21apt-get20:30:140
10540993519,11cyclictest3416-21apt-get22:45:150
10540993515,9cyclictest25775-21sh23:34:030
10540993514,7cyclictest20368-21ssh23:10:570
10540993512,7cyclictest0-21swapper/022:10:540
10540993512,20cyclictest27831-21grep20:35:200
10540993512,10cyclictest10744-21ssh21:54:130
10540993511,9cyclictest28748-21diskmemload22:21:580
10540993511,10cyclictest19697-21ssh21:20:520
10540993510,10cyclictest13834-21ssh21:19:390
1056899348,15cyclictest32466irq/58-eth000:25:143
1056899342,27cyclictest32466irq/58-eth021:15:133
1056099341,17cyclictest14571-21ssh21:20:152
10540993425,7cyclictest22792-21apt-get00:10:160
10540993421,10cyclictest29971-21apt-get23:00:120
10540993413,8cyclictest28748-21diskmemload00:27:050
10540993413,7cyclictest18965-21sed19:15:250
10540993412,9cyclictest17238-21sh23:45:480
10568993314,13cyclictest0-21swapper/323:20:143
1056099332,25cyclictest0-21swapper/222:39:542
1056099332,17cyclictest9340-21ssh22:30:172
1056099332,17cyclictest9340-21ssh22:30:172
1056099331,28cyclictest6194-21rm00:00:052
10540993317,12cyclictest866-21ntp_states22:05:250
10540993313,7cyclictest28748-21diskmemload22:02:150
10540993313,7cyclictest28748-21diskmemload22:02:150
10540993310,9cyclictest14654-21usage_Anybus19:10:300
10568993222,7cyclictest32466irq/58-eth022:40:163
1056899322,20cyclictest32466irq/58-eth023:55:143
1056099322,21cyclictest27-21ksoftirqd/219:30:132
1056099322,21cyclictest27-21ksoftirqd/219:30:132
1054099328,8cyclictest23836-21ntp_states19:20:290
1054099328,8cyclictest23836-21ntp_states19:20:290
10540993214,3cyclictest28748-21diskmemload23:37:190
10540993211,9cyclictest12169-21df_inode19:45:140
10540993210,8cyclictest3234-21cut19:35:210
1056899315,19cyclictest32466irq/58-eth021:20:153
1056099313,19cyclictest23346irq/40-dwc2_hso22:50:132
10549993115,13cyclictest21-21ksoftirqd/122:05:141
1054099311,22cyclictest0-21swapper/020:05:160
10540993110,9cyclictest5937-21kthreadcore20:45:200
10540993110,9cyclictest5937-21kthreadcore20:45:200
1056099303,20cyclictest23346irq/40-dwc2_hso00:10:162
1056099302,21cyclictest0-21swapper/220:30:152
1054999302,23cyclictest14577-21ssh21:20:161
10549993013,11cyclictest10536-21cyclictest19:55:121
10540993018,9cyclictest21384-21memory20:25:250
1056099292,20cyclictest0-21swapper/220:20:142
1056099292,20cyclictest0-21swapper/220:20:142
1056099292,18cyclictest23346irq/40-dwc2_hso22:15:142
1054999294,15cyclictest32215-21ssh21:30:121
1054999292,21cyclictest20102-21ssh22:15:541
10540992920,6cyclictest32579-21diskstats20:40:170
10540992917,9cyclictest8937-21sed20:15:170
10540992917,8cyclictest22700-21memory21:00:230
10540992916,10cyclictest16969-21df19:50:150
1056899283,16cyclictest0-21swapper/323:35:543
1056899282,18cyclictest32466irq/58-eth022:35:153
1056899281,19cyclictest0-21swapper/321:35:163
1056899281,16cyclictest32466irq/58-eth000:35:143
1056099285,13cyclictest23346irq/40-dwc2_hso20:35:162
1056099282,21cyclictest0-21swapper/200:22:042
1056099282,18cyclictest23346irq/40-dwc2_hso21:15:122
1056099281,22cyclictest1314-21ssh22:42:542
1054999285,14cyclictest28748-21diskmemload22:50:121
1054099286,9cyclictest0-21swapper/020:00:160
10540992820,5cyclictest24944-21apt-get19:25:040
1056899273,17cyclictest32466irq/58-eth019:10:143
1056899273,16cyclictest0-21swapper/321:51:543
1056099273,18cyclictest23346irq/40-dwc2_hso19:55:122
1056099271,18cyclictest19133-21ls21:00:022
1056099271,11cyclictest0-21swapper/219:35:162
1054999276,14cyclictest21-21ksoftirqd/121:55:121
10540992714,10cyclictest628-21Async09:12:570
1056899263,16cyclictest0-21swapper/322:15:543
1056099262,19cyclictest23346irq/40-dwc2_hso22:05:132
1054999269,12cyclictest21-21ksoftirqd/123:33:541
1054999265,12cyclictest131rcu_preempt20:20:131
1054999265,12cyclictest131rcu_preempt20:20:131
1056899252,17cyclictest0-21swapper/321:41:103
1056099257,8cyclictest131rcu_preempt23:25:182
1056099257,8cyclictest131rcu_preempt23:25:182
1056099251,20cyclictest20310-21latency_hist22:00:012
1054999253,15cyclictest23981-21ssh22:20:151
1056099241,7cyclictest30843-21ssh00:15:142
1056099241,11cyclictest11849-21latency_hist19:45:022
1054999243,18cyclictest626-21tcf-agent19:25:141
1054999241,17cyclictest26968-21rm23:35:151
10540992413,9cyclictest10548-21cat19:10:010
1056899235,10cyclictest321rcuc/323:40:143
1056899233,13cyclictest0-21swapper/321:46:543
1056899232,17cyclictest32466irq/58-eth000:05:183
1056899232,12cyclictest32466irq/58-eth023:30:143
10568992311,8cyclictest32466irq/58-eth023:50:153
1056099239,11cyclictest523-21df23:20:152
1056099233,13cyclictest29846-21ssh23:55:142
1056099232,6cyclictest0-21swapper/200:25:042
1056099232,6cyclictest0-21swapper/200:09:042
1056099232,17cyclictest23346irq/40-dwc2_hso22:20:152
10560992312,9cyclictest6673-21ssh21:33:032
1056099231,13cyclictest30043-21ssh21:45:272
1056099231,12cyclictest23408-21ssh21:25:132
1054999234,10cyclictest9330-21ssh22:30:161
1054999234,10cyclictest9330-21ssh22:30:161
1054999232,17cyclictest6482-21ssh21:50:241
1054999232,17cyclictest29985-21ssh23:00:141
10568992213,7cyclictest0-21swapper/323:25:213
10568992213,7cyclictest0-21swapper/323:25:213
1056099222,15cyclictest23346irq/40-dwc2_hso23:50:162
1056099222,15cyclictest0-21swapper/222:11:542
1056099222,15cyclictest0-21swapper/221:41:542
1056099222,15cyclictest0-21swapper/221:37:542
1056099222,12cyclictest0-21swapper/219:50:142
1056099221,8cyclictest916-21snmpd23:15:162
1056099221,18cyclictest21365-21cut22:55:192
1056099221,18cyclictest21365-21cut22:55:192
1056099221,15cyclictest0-21swapper/200:35:052
1054999228,11cyclictest18351-21ssh22:35:181
10549992213,3cyclictest21-21ksoftirqd/119:30:121
10549992213,3cyclictest21-21ksoftirqd/119:30:121
1054999221,12cyclictest0-21swapper/120:40:301
10549992210,9cyclictest5902-21sed00:00:011
1054999220,18cyclictest7418-21cut19:40:181
1056899219,9cyclictest0-21swapper/300:02:073
1056899212,12cyclictest32466irq/58-eth019:15:133
10568992111,7cyclictest32466irq/58-eth022:20:153
1056099213,9cyclictest22745-21cat20:30:012
10560992111,7cyclictest532-21sh22:25:142
1056099211,14cyclictest0-21swapper/222:05:012
1056099211,14cyclictest0-21swapper/222:05:012
1056099211,12cyclictest12061-21/usr/sbin/munin19:45:312
1056099211,11cyclictest20769-21latency_hist23:50:022
1056099210,17cyclictest23346irq/40-dwc2_hso23:40:142
1054999219,9cyclictest27015-21apt-get22:40:151
1054999217,7cyclictest211ksoftirqd/121:15:121
1054999216,12cyclictest20309-21ssh21:40:201
1054999215,3cyclictest131rcu_preempt22:45:141
1054999214,12cyclictest6535-21ssh00:20:141
1054999212,3cyclictest131rcu_preempt23:20:131
10549992114,5cyclictest18751-21ssh00:25:311
1054999211,3cyclictest131rcu_preempt20:35:171
1054999211,18cyclictest20735-21ssh22:00:141
1054999211,18cyclictest20735-21ssh22:00:141
10549992111,7cyclictest12154-21ssh23:45:111
1054999211,16cyclictest10754-21apt-get19:10:131
1054999211,11cyclictest7070-21awk19:40:011
10549992110,8cyclictest9736-21df20:50:191
1056899203,12cyclictest32466irq/58-eth022:45:133
1056899203,12cyclictest0-21swapper/300:21:043
10568992011,7cyclictest0-21swapper/322:01:493
10568992011,7cyclictest0-21swapper/322:01:493
10568992011,6cyclictest0-21swapper/300:15:163
1056099209,8cyclictest0-21swapper/220:40:202
1056099206,5cyclictest27-21ksoftirqd/223:10:192
1056099205,12cyclictest0-21swapper/220:00:232
1056099203,14cyclictest12380-21kthreadcore19:10:212
1056099202,9cyclictest0-21swapper/219:25:152
1056099201,9cyclictest10965-21kthreadcore20:50:242
10560992012,5cyclictest3151-21hwlatdetect21:50:172
1056099201,16cyclictest23873-21cut00:30:212
1056099200,5cyclictest5233-21latency_hist21:15:022
1054999202,12cyclictest21061-21sh23:50:171
1054999201,6cyclictest17093-21munin-run21:40:011
10549992014,3cyclictest0-21swapper/100:05:191
1054999201,15cyclictest3396-21ssh23:40:151
1056899199,7cyclictest0-21swapper/322:30:023
1056899195,10cyclictest32466irq/58-eth022:10:173
1056899193,11cyclictest0-21swapper/323:19:553
1056899192,11cyclictest0-21swapper/319:40:163
1056899192,10cyclictest0-21swapper/323:05:143
1056899191,12cyclictest0-21swapper/320:15:153
1056899191,11cyclictest0-21swapper/319:50:173
10568991910,6cyclictest32466irq/58-eth022:50:133
1056099193,13cyclictest16860-21kthreadcore19:15:202
1056099192,10cyclictest26650-21kthreadcore21:05:212
1056099191,9cyclictest8873-21irqcore20:15:172
1056099191,9cyclictest6082-21kthreadcore20:45:202
1056099191,9cyclictest6082-21kthreadcore20:45:202
1056099191,9cyclictest19544-21grep21:00:172
1056099191,11cyclictest0-21swapper/219:20:172
1056099191,11cyclictest0-21swapper/219:20:172
1056099191,10cyclictest4169-21kthreadcore22:45:212
1056099191,10cyclictest1007-21kthreadcore20:05:232
1054999199,7cyclictest9525-21fw_packets23:25:171
1054999199,7cyclictest9525-21fw_packets23:25:171
1054999199,7cyclictest14536-21df_inode20:55:171
1054999195,3cyclictest131rcu_preempt00:35:041
1054999192,7cyclictest31973-21kthreadcore00:15:201
1054999191,5cyclictest26995-21ntp_192.168.11421:25:241
10549991912,4cyclictest21-21ksoftirqd/100:00:221
1054999191,16cyclictest24284-21kthreadcore00:30:211
1054999191,15cyclictest21123-21apt-get22:55:151
1054999191,15cyclictest21123-21apt-get22:55:151
10549991911,5cyclictest0-21swapper/119:45:181
10549991911,4cyclictest32755-21/usr/sbin/munin22:25:121
1054999191,14cyclictest29187-21ssh21:10:161
1054999191,13cyclictest15335-21diskstats23:10:131
1054999191,12cyclictest26487-21ssh00:10:291
10549991910,6cyclictest7343-21ssh22:10:241
1056899189,6cyclictest32466irq/58-eth023:45:133
1056899188,7cyclictest32466irq/58-eth022:55:163
1056899188,7cyclictest32466irq/58-eth022:55:163
1056899187,8cyclictest0-21swapper/322:30:173
1056899187,8cyclictest0-21swapper/322:30:173
1056899184,10cyclictest32466irq/58-eth019:55:123
1056899182,11cyclictest0-21swapper/322:07:543
1056899182,10cyclictest0-21swapper/321:05:143
1056899182,10cyclictest0-21swapper/320:30:153
1056899182,10cyclictest0-21swapper/320:10:133
1056899182,10cyclictest0-21swapper/320:05:163
1056899182,10cyclictest0-21swapper/319:30:143
1056899182,10cyclictest0-21swapper/319:30:143
1056899182,10cyclictest0-21swapper/319:20:183
1056899182,10cyclictest0-21swapper/319:20:183
1056899182,10cyclictest0-21swapper/300:30:043
1056899181,13cyclictest0-21swapper/320:45:283
1056899181,13cyclictest0-21swapper/320:45:283
1056899181,10cyclictest0-21swapper/320:40:133
1056899181,10cyclictest0-21swapper/320:25:013
1056899181,10cyclictest0-21swapper/320:25:013
10568991810,6cyclictest0-21swapper/321:30:043
1054999189,6cyclictest21433-21cron19:55:011
1054999187,4cyclictest0-21swapper/120:30:141
1054999181,9cyclictest21725-21kthreadcore21:00:211
1054999181,13cyclictest9897-21memory23:05:241
1054999180,3cyclictest131rcu_preempt21:47:161
1054999180,11cyclictest25118-21gpgv19:25:011
1054999180,11cyclictest25118-21gpgv19:25:011
1056899179,6cyclictest0-21swapper/321:27:143
1056899178,6cyclictest32466irq/58-eth021:55:133
1056899172,10cyclictest0-21swapper/321:00:053
1056899172,10cyclictest0-21swapper/320:55:163
1056899172,10cyclictest0-21swapper/320:25:163
1056899172,10cyclictest0-21swapper/319:45:133
1056899172,10cyclictest0-21swapper/319:25:143
1056899171,13cyclictest0-21swapper/319:35:303
1056099171,13cyclictest10491-21latency_hist19:10:022
1054999175,9cyclictest31042-21cron20:05:011
1054999171,9cyclictest2271-21ntp_192.168.11420:05:261
10549991710,5cyclictest8232-21cron20:15:011
10549991710,4cyclictest18526-21cut19:15:221
1054999169,4cyclictest0-21swapper/120:15:171
1054999164,9cyclictest9422-21cron20:50:011
1054999164,9cyclictest9422-21cron20:50:011
1054999161,8cyclictest20469-21tr20:25:231
1056899157,6cyclictest33-21ksoftirqd/320:35:303
1056899151,8cyclictest0-21swapper/320:50:323
1054999151,11cyclictest10702-21grep19:10:021
1054999151,10cyclictest24749-21sed21:05:181
1056899141,7cyclictest0-21swapper/320:05:013
1056899111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional