You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 04:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Dec 04, 2025 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1060199481,24cyclictest932-21snmpd22:10:142
1061099451,25cyclictest29566irq/58-eth022:20:143
1061099451,25cyclictest29566irq/58-eth022:20:133
1061099442,24cyclictest29566irq/58-eth000:35:173
1058199430,34cyclictest22672-21rm00:20:140
1058199412,4cyclictest131rcu_preempt00:00:140
1061099404,22cyclictest29566irq/58-eth023:10:133
10593993921,10cyclictest21-21ksoftirqd/123:15:141
1061099385,20cyclictest29566irq/58-eth021:20:143
1061099379,25cyclictest29566irq/58-eth021:40:143
10610993714,17cyclictest29566irq/58-eth023:05:133
1061099371,2cyclictest0-21swapper/300:30:123
1060199372,28cyclictest932-21snmpd19:35:132
1058199371,29cyclictest5471-21ssh21:15:140
10601993610,6cyclictest27-21ksoftirqd/219:10:122
1058199361,29cyclictest12545-21ssh23:15:120
1058199361,19cyclictest26740-21ssh23:21:570
1061099357,18cyclictest0-21swapper/300:00:143
1060199352,27cyclictest20297-21sh23:19:402
1060199352,20cyclictest29243-21ssh23:25:122
1059399351,26cyclictest15029-21ssh21:40:141
1061099343,26cyclictest0-21swapper/300:15:153
1059399341,22cyclictest26041-21ssh22:25:131
1061099333,25cyclictest0-21swapper/322:55:133
1061099331,2cyclictest0-21swapper/320:50:143
1061099323,24cyclictest0-21swapper/323:55:123
1058199322,22cyclictest30574-21ssh00:25:160
1061099312,22cyclictest0-21swapper/300:10:153
1060199313,25cyclictest27-21ksoftirqd/222:55:132
1060199312,24cyclictest13722-21ssh23:53:542
10601993119,7cyclictest261rcuc/223:12:412
1060199311,25cyclictest23189-21ssh00:00:132
1059399312,24cyclictest9295-21ssh22:15:161
1058199315,19cyclictest31821-21ssh21:50:140
1058199312,22cyclictest6078-21/usr/sbin/munin00:30:160
1058199311,22cyclictest24777-21ssh22:05:170
1061099302,22cyclictest0-21swapper/321:15:133
10610993018,7cyclictest29566irq/58-eth022:35:273
1061099301,2cyclictest0-21swapper/322:05:173
1061099301,2cyclictest0-21swapper/321:55:113
1061099301,24cyclictest0-21swapper/323:20:153
1060199304,15cyclictest26346irq/40-dwc2_hso20:30:132
1060199304,11cyclictest131rcu_preempt23:30:112
1060199303,12cyclictest7045-21ssh00:10:172
1060199303,12cyclictest17631-21ssh22:20:122
1060199303,12cyclictest17631-21ssh22:20:122
10593993022,2cyclictest0-21swapper/122:30:141
1061099292,21cyclictest0-21swapper/319:25:133
1061099292,17cyclictest29566irq/58-eth023:50:143
1061099291,6cyclictest0-21swapper/323:00:133
1060199295,14cyclictest26346irq/40-dwc2_hso21:45:152
1060199295,14cyclictest26346irq/40-dwc2_hso21:20:142
1060199292,18cyclictest29646-21rm21:10:142
1060199291,15cyclictest30442-21ssh00:25:132
10581992921,3cyclictest12-21ksoftirqd/023:55:120
1058199291,20cyclictest17655-21ssh22:20:140
1058199291,20cyclictest17655-21ssh22:20:130
1061099281,8cyclictest0-21swapper/322:50:143
1061099281,8cyclictest0-21swapper/322:50:133
1061099281,6cyclictest0-21swapper/322:45:123
10610992815,10cyclictest29566irq/58-eth021:10:153
1061099281,3cyclictest0-21swapper/322:00:113
1061099281,3cyclictest0-21swapper/300:20:133
1061099281,22cyclictest0-21swapper/319:45:133
1060199285,17cyclictest27-21ksoftirqd/220:50:132
1060199283,20cyclictest0-21swapper/222:03:402
1060199282,16cyclictest26346irq/40-dwc2_hso23:45:132
1059399286,19cyclictest23442-21ssh21:45:151
1059399281,19cyclictest20950-21ssh23:20:151
1058199282,18cyclictest30667-21sh23:45:130
1061099274,17cyclictest0-21swapper/319:40:133
1061099272,7cyclictest0-21swapper/321:50:153
1061099272,2cyclictest0-21swapper/320:20:153
1061099271,7cyclictest0-21swapper/323:40:163
1061099271,4cyclictest0-21swapper/322:10:253
1061099271,2cyclictest0-21swapper/319:20:133
1061099271,23cyclictest0-21swapper/300:25:143
1061099270,2cyclictest0-21swapper/320:30:123
1060199275,7cyclictest27-21ksoftirqd/223:55:132
1060199273,18cyclictest5470-21sh21:15:152
1060199273,14cyclictest26346irq/40-dwc2_hso19:55:132
10601992720,4cyclictest14251-21tr20:20:222
1059399272,19cyclictest13459-21ssh21:56:401
1061099262,18cyclictest0-21swapper/322:40:133
1061099261,8cyclictest0-21swapper/320:25:123
1061099261,5cyclictest0-21swapper/320:40:133
1061099261,3cyclictest0-21swapper/323:25:183
1061099261,3cyclictest0-21swapper/321:26:343
1061099261,3cyclictest0-21swapper/300:05:163
1061099261,3cyclictest0-21swapper/300:05:153
1061099261,2cyclictest0-21swapper/321:05:123
1061099261,2cyclictest0-21swapper/319:30:123
1061099261,22cyclictest0-21swapper/321:35:133
1060199265,9cyclictest27-21ksoftirqd/221:05:132
1060199264,16cyclictest26346irq/40-dwc2_hso19:15:162
1060199262,16cyclictest26346irq/40-dwc2_hso21:25:142
10593992616,4cyclictest21-21ksoftirqd/123:40:151
1058199262,20cyclictest7807-21ssh21:55:140
10581992617,5cyclictest0-21swapper/022:15:170
1058199261,18cyclictest5327-21ssh23:30:140
1061099253,17cyclictest0-21swapper/319:35:143
1061099251,2cyclictest0-21swapper/323:48:503
1061099251,2cyclictest0-21swapper/322:30:123
1061099251,2cyclictest0-21swapper/322:19:533
1061099251,2cyclictest0-21swapper/321:30:303
1061099251,2cyclictest0-21swapper/319:55:153
1060199253,17cyclictest26346irq/40-dwc2_hso20:10:142
1060199253,17cyclictest0-21swapper/221:59:412
1060199253,14cyclictest26346irq/40-dwc2_hso23:00:152
1060199252,8cyclictest27273-21ssh23:22:402
1060199252,20cyclictest0-21swapper/222:05:052
1060199252,15cyclictest26346irq/40-dwc2_hso23:05:132
1060199251,20cyclictest932-21snmpd19:40:142
1060199250,15cyclictest386-21find20:40:142
1059399252,20cyclictest14979-21ssh00:15:161
1059399252,19cyclictest21-21ksoftirqd/120:20:151
1058199251,18cyclictest11499-21ssh22:55:140
1058199251,17cyclictest19813-21ssh23:00:140
1061099241,7cyclictest0-21swapper/321:00:153
1061099241,3cyclictest0-21swapper/322:28:023
1061099241,3cyclictest0-21swapper/321:46:373
1061099241,2cyclictest0-21swapper/323:31:543
1061099241,2cyclictest0-21swapper/323:15:133
1061099241,2cyclictest0-21swapper/319:50:123
1060199243,16cyclictest26346irq/40-dwc2_hso22:25:152
1060199243,14cyclictest26346irq/40-dwc2_hso19:25:112
1060199242,20cyclictest0-21swapper/222:44:392
1060199242,17cyclictest0-21swapper/200:33:172
1060199242,14cyclictest26346irq/40-dwc2_hso20:15:122
10601992414,8cyclictest27-21ksoftirqd/222:45:132
10601992412,4cyclictest0-21swapper/221:35:142
10601992411,10cyclictest16798-21cron19:50:002
1059399246,6cyclictest3704-21munin-plugin-st20:10:011
1059399241,17cyclictest854-21ssh22:10:171
1059399241,17cyclictest13836-21ssh21:20:141
1058199247,11cyclictest12-21ksoftirqd/022:35:140
1061099233,15cyclictest0-21swapper/320:55:143
1061099231,5cyclictest0-21swapper/323:35:163
1061099231,4cyclictest0-21swapper/320:35:293
1061099231,3cyclictest0-21swapper/320:15:123
1060199232,6cyclictest0-21swapper/222:54:412
1060199232,6cyclictest0-21swapper/222:54:402
10601992316,4cyclictest30605-21ssh21:30:152
10601992311,8cyclictest10351-21/usr/sbin/munin22:35:132
1060199230,19cyclictest22190-21ssh23:40:042
1060199230,16cyclictest20317-21ls19:50:232
1059399235,7cyclictest21-21ksoftirqd/121:25:321
1059399233,17cyclictest27241-21ssh22:45:131
1059399232,7cyclictest131rcu_preempt00:33:181
1059399232,17cyclictest6573-21ssh23:50:141
10593992314,4cyclictest21-21ksoftirqd/121:50:141
10593992312,8cyclictest2078-21sshd23:07:401
1059399231,17cyclictest21-21ksoftirqd/119:25:121
1058199231,17cyclictest31481-21ssh00:05:160
1058199231,17cyclictest31481-21ssh00:05:160
1061099221,3cyclictest0-21swapper/320:05:153
1061099221,18cyclictest0-21swapper/320:45:263
1061099221,18cyclictest0-21swapper/319:10:293
1060199227,7cyclictest27-21ksoftirqd/220:25:112
1060199222,15cyclictest0-21swapper/221:53:402
10601992216,4cyclictest0-21swapper/221:40:152
1060199221,11cyclictest30255-21grep00:25:022
1060199221,11cyclictest15716-21tr00:15:222
1059399222,15cyclictest10398-21ssh22:35:141
1059399221,18cyclictest23279-21ls00:00:151
1058199222,14cyclictest4070-21ssh23:10:150
10581992215,4cyclictest0-21swapper/023:05:140
1058199221,18cyclictest18780-21find22:40:110
1058199221,18cyclictest15642-21ls19:15:160
1058199221,17cyclictest22348-21sh23:40:170
1058199221,17cyclictest13805-21ssh23:35:150
1058199221,16cyclictest7253-21missed_timers20:10:230
1061099211,2cyclictest0-21swapper/320:10:243
1061099211,2cyclictest0-21swapper/319:15:153
1061099210,2cyclictest0-21swapper/320:00:133
1060199218,6cyclictest19832-21apt-get21:00:132
1060199215,9cyclictest10381-21ssh22:15:232
1060199212,11cyclictest2233-21fschecks_time22:30:162
10601992111,8cyclictest0-21swapper/219:20:162
1060199211,11cyclictest31725-21ssh00:05:202
1060199211,11cyclictest31725-21ssh00:05:192
1060199211,11cyclictest21944-21rm23:40:012
1060199210,18cyclictest31740-21timerandwakeup20:35:292
1059399218,10cyclictest15298-21cut23:55:181
1059399216,10cyclictest21-21ksoftirqd/119:55:141
1059399215,10cyclictest131rcu_preempt19:10:131
1059399211,17cyclictest1799-21ssh21:30:251
1059399211,14cyclictest7774-21ssh23:10:261
1059399211,14cyclictest10302-21uniq20:50:161
10593992110,8cyclictest9510-21cat23:30:291
1058199211,16cyclictest29644-21ssh21:10:150
1061099201,2cyclictest0-21swapper/319:10:013
1060199208,5cyclictest27-21ksoftirqd/220:45:122
1060199207,11cyclictest17777-21grep00:35:322
1060199201,9cyclictest28614-21kthreadcore20:00:222
10601992010,8cyclictest0-21swapper/220:55:162
1060199200,14cyclictest30178-21ls19:30:152
1059399204,14cyclictest201rcuc/120:25:111
1059399203,14cyclictest10105-21ssh21:35:221
10593992012,5cyclictest0-21swapper/100:10:281
10593992011,7cyclictest21-21ksoftirqd/123:00:331
1059399201,15cyclictest12198-21apt-get19:45:121
1059399201,12cyclictest20014-21grep21:00:171
1058199203,12cyclictest0-21swapper/019:30:140
1058199201,3cyclictest0-21swapper/021:47:030
10581992012,6cyclictest13818-21sshd00:13:490
1058199201,16cyclictest16262-21ssh22:00:130
1058199201,15cyclictest30604-21ssh21:30:150
1058199201,12cyclictest6828-21ls23:50:180
10581992010,7cyclictest2427-21sed22:30:180
1058199200,18cyclictest13903-21ls21:20:160
1060199191,7cyclictest0-21swapper/220:05:162
1059399195,10cyclictest0-21swapper/100:05:171
1059399195,10cyclictest0-21swapper/100:05:161
1059399194,12cyclictest13608-21latency_hist21:20:011
1059399192,15cyclictest5125-21kthreadcore22:50:221
1059399192,15cyclictest5125-21kthreadcore22:50:211
1059399192,10cyclictest11358-21kthreadcore20:15:211
1059399191,7cyclictest2717-21sshd21:11:311
10593991911,4cyclictest5615-21latency_hist00:30:021
1059399191,13cyclictest30626-21acpi23:45:111
1059399190,14cyclictest12032-21ls19:45:011
1058199191,10cyclictest0-21swapper/022:10:470
1058199190,16cyclictest24440-21ls21:05:020
1059399187,9cyclictest453-21dpkg22:10:011
1059399183,6cyclictest5623-21kthreadcore19:35:231
1059399182,11cyclictest21928-21ssh22:01:411
1059399182,11cyclictest0-21swapper/122:40:141
1059399181,9cyclictest31690-21kthreadcore23:25:201
10593991811,4cyclictest21-21ksoftirqd/120:45:121
10593991811,4cyclictest0-21swapper/120:30:151
1059399181,13cyclictest0-21swapper/119:15:131
10593991810,6cyclictest21624-21sshd19:51:361
10593991810,5cyclictest23399-21kthreadcore00:20:191
10593991810,5cyclictest11854-21irqcore22:55:191
1058199189,6cyclictest15757-21kthreadcore21:40:200
1058199186,10cyclictest1-21systemd20:36:240
1058199183,11cyclictest25108-21irqcore21:05:180
1058199183,10cyclictest13072-21kthreadcore19:45:180
1058199181,6cyclictest30851-21ntp_192.168.11422:45:240
1058199181,14cyclictest3867-21ssh22:50:180
1058199181,14cyclictest3867-21ssh22:50:180
1058199181,14cyclictest10420-21ssh21:35:270
10581991811,4cyclictest0-21swapper/020:55:140
1058199181,10cyclictest842-21munin-node19:40:130
10581991810,6cyclictest21633-21sshd19:54:360
1059399175,9cyclictest7106-21kthreadcore20:10:211
1059399173,10cyclictest662-21ntpd23:35:151
1059399171,8cyclictest5153-21latency_hist20:45:021
1059399171,8cyclictest31459-21kthreadcore19:30:201
1059399171,8cyclictest28222-21hwlatdetect20:35:181
10593991711,4cyclictest0-21swapper/120:00:231
1059399171,13cyclictest14211-21ssh00:35:211
10593991710,5cyclictest18774-21cat20:55:291
1058199177,8cyclictest3847-21ntp_192.168.11420:40:260
1058199177,7cyclictest16574-21cut00:35:250
1058199176,3cyclictest131rcu_preempt20:33:380
1058199173,11cyclictest7119-21sendmail19:40:020
1058199172,6cyclictest0-21swapper/023:26:070
1058199171,6cyclictest8785-21proc_pri20:45:250
1058199171,6cyclictest26369-21unixbench_singl21:25:320
1058199171,13cyclictest31521-21apt-get20:05:140
1058199171,12cyclictest26272-21uniq22:25:160
1059399165,8cyclictest24772-21tune2fs21:05:161
1059399162,10cyclictest20905-21irqcore19:20:191
1059399161,11cyclictest17633-21apt-get22:20:121
1059399161,11cyclictest17633-21apt-get22:20:121
1058199162,11cyclictest11155-21sed19:10:170
1058199162,10cyclictest9031-21irqcore20:15:160
1058199162,10cyclictest0-21swapper/020:50:150
1058199161,5cyclictest25613-21irqcore19:25:170
1058199161,13cyclictest0-21swapper/020:20:150
1058199161,11cyclictest21965-21diskstats19:55:160
1058199161,11cyclictest15711-21kthreadcore00:15:220
1058199160,11cyclictest21723-21missed_timers20:25:210
1059399159,4cyclictest10614-21latency_hist19:10:011
1058199152,10cyclictest26788-21fschecks_count20:00:170
1058199152,10cyclictest20825-21irqcore19:20:180
1060199140,10cyclictest10718-21latency_hist19:10:022
1058199141,10cyclictest10567-21cyclictest19:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional