You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-14 - 06:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Aug 14, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2449899601,4cyclictest131rcu_preempt20:15:130
2450899593,52cyclictest10059-21ssh23:15:141
2449899524,43cyclictest21028-21ssh00:15:130
2450899492,7cyclictest131rcu_preempt22:50:121
2449899482,42cyclictest6651-21ssh00:25:150
24519994525,15cyclictest24625-21apt-get19:10:142
2449899441,32cyclictest5426-21ssh21:25:160
24519994311,22cyclictest27-21ksoftirqd/223:40:132
2451999422,17cyclictest0-21swapper/220:15:132
24508994213,20cyclictest21-21ksoftirqd/122:55:121
2449899422,33cyclictest29687-21ssh22:50:130
24519994111,25cyclictest0-21swapper/220:00:122
2451999392,19cyclictest17390-21ssh23:55:112
24519993913,22cyclictest7763-21ssh22:20:142
2450899376,22cyclictest13005-21diskmemload22:05:141
24519993616,11cyclictest27-21ksoftirqd/200:05:162
2451999361,21cyclictest28812-21ssh21:20:142
2451999352,16cyclictest4890-21mailstats23:10:272
2451999346,13cyclictest0-21swapper/223:50:122
24519993421,3cyclictest4399-21apt-get20:30:142
24519993421,3cyclictest4399-21apt-get20:30:142
24519993413,13cyclictest930-21snmpd00:10:132
2452899333,16cyclictest35966irq/58-eth019:55:113
2452899333,16cyclictest35966irq/58-eth019:55:113
24528993315,13cyclictest35966irq/58-eth021:25:153
24519993319,9cyclictest29058-21ssh21:36:262
2449899332,21cyclictest20732-21ssh22:45:150
2449899331,4cyclictest131rcu_preempt22:55:130
24519993223,5cyclictest20702-21apt-get22:45:152
24519993221,7cyclictest3932-21apt-get22:00:132
24519993219,10cyclictest11650-21apt-get22:40:142
2450899324,22cyclictest24440-21ssh23:05:121
2450899323,16cyclictest13720-21ssh23:35:121
2451999313,17cyclictest27346irq/40-dwc2_hso22:50:152
2451999312,22cyclictest30363-21ssh00:36:332
2451999312,21cyclictest25779-21ntp_states22:10:272
24519993118,8cyclictest27346irq/40-dwc2_hso21:45:162
24519993118,8cyclictest27346irq/40-dwc2_hso00:15:142
2450899311,24cyclictest11662-21ssh22:40:131
2451999309,16cyclictest0-21swapper/221:32:242
2451999304,17cyclictest27346irq/40-dwc2_hso19:30:122
2449899302,21cyclictest13018-21ssh22:05:140
2452899291,17cyclictest35966irq/58-eth000:00:133
2451999295,12cyclictest0-21swapper/221:25:152
2451999294,20cyclictest0-21swapper/222:25:132
24519992923,3cyclictest6329-21apt-get22:55:132
2451999292,20cyclictest0-21swapper/221:00:112
2451999292,17cyclictest27346irq/40-dwc2_hso22:30:132
24508992910,14cyclictest21926-1kworker/u9:021:35:261
2452899283,22cyclictest35966irq/58-eth023:35:133
2452899282,17cyclictest0-21swapper/320:30:133
2452899282,17cyclictest0-21swapper/320:30:133
2451999287,17cyclictest0-21swapper/219:45:132
2451999285,18cyclictest0-21swapper/220:45:152
2451999283,21cyclictest0-21swapper/200:03:302
2451999283,20cyclictest0-21swapper/223:09:272
2451999281,11cyclictest6621-21ssh00:25:142
24508992812,4cyclictest2611-21ssh22:35:121
2449899282,21cyclictest32026-21ssh00:38:330
2449899282,16cyclictest31083-21ssh22:15:120
2449899281,22cyclictest3948-21ssh22:00:130
2449899281,17cyclictest3069-21ssh00:05:180
2451999277,12cyclictest0-21swapper/222:15:582
2451999273,15cyclictest27346irq/40-dwc2_hso21:55:142
2451999273,13cyclictest27346irq/40-dwc2_hso00:30:132
2451999272,19cyclictest0-21swapper/221:40:152
2451999272,10cyclictest0-21swapper/223:39:282
2450899274,18cyclictest28111-21ssh23:25:121
2451999265,12cyclictest0-21swapper/222:36:502
2451999264,13cyclictest27346irq/40-dwc2_hso20:55:132
2451999262,15cyclictest27346irq/40-dwc2_hso20:40:122
2451999261,14cyclictest15363-21ssh23:00:132
24519992610,14cyclictest27346irq/40-dwc2_hso21:10:142
2450899262,19cyclictest18028-21ssh21:30:261
2449899262,18cyclictest18023-21sh21:30:260
2452899251,16cyclictest35966irq/58-eth021:45:153
2452899251,12cyclictest0-21swapper/300:20:143
2451999259,10cyclictest0-21swapper/221:15:132
24519992514,8cyclictest31860-21apt-get23:45:122
24519992514,8cyclictest31860-21apt-get23:45:122
2451999251,21cyclictest614666ssh00:24:552
24519992510,12cyclictest0-21swapper/221:50:152
2451999250,15cyclictest2028-21grep19:20:162
24508992513,8cyclictest8972-21cron20:35:011
24508992513,8cyclictest8972-21cron20:35:011
2449899256,13cyclictest10327-21diskmemload23:30:130
2449899252,18cyclictest26404-21ssh00:00:140
2449899251,19cyclictest9179-21ssh21:45:160
2449899251,17cyclictest6618-21sshd19:25:140
2452899245,16cyclictest35966irq/58-eth023:45:123
2452899245,16cyclictest35966irq/58-eth023:45:123
2452899242,11cyclictest0-21swapper/321:24:263
24528992410,5cyclictest0-21swapper/322:00:283
2451999249,11cyclictest0-21swapper/223:20:122
2451999245,14cyclictest0-21swapper/220:10:132
2451999242,17cyclictest930-21snmpd20:20:132
2451999242,17cyclictest0-21swapper/220:50:042
2451999241,8cyclictest0-21swapper/222:06:032
2450899245,17cyclictest201rcuc/121:50:161
24498992418,4cyclictest13730-21rm23:35:120
2449899241,19cyclictest25789-21ntpq22:10:270
2452899233,14cyclictest35966irq/58-eth023:15:143
2452899233,13cyclictest0-21swapper/321:35:263
2452899233,13cyclictest0-21swapper/321:30:263
24528992313,5cyclictest0-21swapper/322:35:133
24528992310,8cyclictest35966irq/58-eth000:35:153
2451999235,10cyclictest6905-21if_eth119:25:192
2451999233,16cyclictest4061-21usage_EthcTask20:25:292
2451999233,14cyclictest27346irq/40-dwc2_hso19:15:132
2451999233,13cyclictest27346irq/40-dwc2_hso21:05:142
24519992312,8cyclictest4657-21/usr/sbin/munin23:30:162
24508992317,3cyclictest0-21swapper/123:00:121
2449899232,16cyclictest24429-21apt-get23:05:120
2452899229,9cyclictest35966irq/58-eth021:10:273
24528992213,4cyclictest0-21swapper/322:05:223
2451999226,13cyclictest19714-21open_inodes19:35:272
24519992211,9cyclictest974-21ssh23:26:022
2450899223,15cyclictest22040-21ssh22:10:141
2450899221,18cyclictest22805-21ssh23:40:121
2450899221,16cyclictest27272-21rm21:55:151
2450899221,15cyclictest9178-21sh21:45:161
2450899220,16cyclictest17542-21find20:10:121
2449899222,15cyclictest2639-21diskmemload22:35:140
2449899221,18cyclictest20593-21ssh22:25:270
2452899213,11cyclictest35966irq/58-eth022:50:133
2452899213,11cyclictest0-21swapper/319:25:143
24528992110,8cyclictest0-21swapper/322:10:453
2451999217,11cyclictest14892-21cat23:15:322
2451999214,13cyclictest0-21swapper/219:40:132
2451999212,8cyclictest0-21swapper/219:50:132
2451999212,14cyclictest0-21swapper/219:55:132
2451999212,14cyclictest0-21swapper/219:55:122
2451999211,11cyclictest11971-21kthreadcore20:35:212
2450899218,10cyclictest7638-21open_files22:00:261
2450899213,15cyclictest25993-21ssh22:30:141
2450899212,16cyclictest8427-21ssh23:50:131
24508992112,6cyclictest8038-21idleruntime22:20:171
2450899211,17cyclictest21294-21ls19:40:161
2450899211,17cyclictest15553-21ls00:10:241
2450899211,14cyclictest31890-21rm23:45:141
2450899211,14cyclictest31890-21rm23:45:141
2449899215,12cyclictest0-21swapper/021:20:120
2449899212,16cyclictest23454-21ssh21:35:150
24498992114,4cyclictest987-21ssh23:10:120
2449899210,18cyclictest8507-21ls23:50:130
2452899202,9cyclictest35966irq/58-eth022:45:133
2452899202,10cyclictest35966irq/58-eth022:40:133
24528992012,5cyclictest35966irq/58-eth023:40:123
2452899201,11cyclictest35966irq/58-eth020:10:123
2450899208,9cyclictest27741-21cron23:25:021
2450899205,12cyclictest19740-21ssh21:15:131
2450899203,7cyclictest21-21ksoftirqd/100:00:231
24508992012,5cyclictest0-21swapper/120:00:231
24508992012,4cyclictest0-21swapper/123:30:281
2450899201,15cyclictest32029-21find20:25:111
2450899201,15cyclictest30659-21apt-get19:50:121
24508992010,7cyclictest24374-21ntpq22:45:271
2449899202,11cyclictest12-21ksoftirqd/019:20:130
24498992014,4cyclictest0-21swapper/022:20:090
2452899199,8cyclictest0-21swapper/323:58:393
2452899198,7cyclictest35966irq/58-eth000:25:153
2452899193,11cyclictest0-21swapper/322:21:283
2452899192,10cyclictest0-21swapper/320:20:133
24528991910,6cyclictest0-21swapper/323:10:463
2451999191,15cyclictest12850-21fschecks_time20:05:132
2450899199,7cyclictest1244-21users21:20:341
2450899198,8cyclictest2790-21cut21:40:231
2450899198,8cyclictest21425-21cat22:25:311
2450899192,14cyclictest24716-21ssh00:35:151
2450899192,13cyclictest2746-21kthreadcore23:10:201
2450899192,13cyclictest0-21swapper/122:15:131
2450899191,7cyclictest3754-21/usr/sbin/munin00:05:221
24508991911,6cyclictest24680-21sshd23:58:271
2450899191,15cyclictest32076-21grep20:55:261
24508991911,5cyclictest0-21swapper/121:10:141
24508991911,5cyclictest0-21swapper/120:45:241
2450899191,15cyclictest0-21swapper/100:25:161
2450899191,14cyclictest9207-21apt-get20:35:121
24508991911,4cyclictest0-21swapper/119:25:121
2450899191,10cyclictest5787-21cpuspeed21:05:151
24508991910,6cyclictest0-21swapper/119:35:261
2449899199,8cyclictest20192-1kworker/u9:023:55:280
2449899199,7cyclictest18709-21munin-plugin-st23:20:020
2449899194,3cyclictest131rcu_preempt00:30:150
2449899193,9cyclictest10327-21diskmemload23:27:150
2449899192,6cyclictest23433-21usage_Anybus23:20:300
2449899191,9cyclictest16062-21kthreadcore20:40:200
2449899191,7cyclictest27237-21/usr/sbin/munin21:55:170
24498991911,5cyclictest32280-21cut19:15:220
2449899191,15cyclictest30029-21ssh00:20:150
2449899191,15cyclictest2381-21ls22:35:010
2449899191,11cyclictest8151-21ls20:30:280
2449899191,11cyclictest8151-21ls20:30:280
2452899189,6cyclictest35966irq/58-eth023:30:123
2452899188,6cyclictest0-21swapper/300:15:293
2452899182,10cyclictest0-21swapper/322:55:063
2452899182,10cyclictest0-21swapper/320:50:043
24528991812,4cyclictest0-21swapper/323:20:283
2452899181,12cyclictest36066irq/59-eth000:30:133
2452899181,10cyclictest0-21swapper/300:05:153
24528991810,6cyclictest35966irq/58-eth023:25:123
24528991810,6cyclictest35966irq/58-eth023:05:143
24528991810,6cyclictest35966irq/58-eth022:15:123
24528991810,6cyclictest0-21swapper/322:26:353
24528991810,6cyclictest0-21swapper/321:54:343
24528991810,6cyclictest0-21swapper/321:17:393
2450899189,6cyclictest21-21ksoftirqd/121:00:111
2450899186,8cyclictest0-21swapper/100:15:141
24508991811,5cyclictest9483-21tail21:25:311
2450899181,15cyclictest2346-21cut19:20:201
2450899181,13cyclictest29390-21/usr/sbin/munin19:15:331
2449899184,11cyclictest2900-21kthreadcore21:40:220
2449899183,3cyclictest131rcu_preempt19:40:010
2449899182,13cyclictest31888-21ssh23:45:140
2449899182,13cyclictest31888-21ssh23:45:140
2449899181,8cyclictest23512-21irqstats23:40:180
2449899181,7cyclictest19484-21kernelversion20:45:200
2449899181,5cyclictest0-21swapper/000:10:200
24498991811,5cyclictest20863-21load20:10:220
2449899181,14cyclictest0-21swapper/019:50:120
2449899181,13cyclictest18357-21df_inode21:50:170
2449899181,13cyclictest17243-21munin-run20:10:010
24498991810,6cyclictest9390-21fschecks_time20:35:160
24498991810,6cyclictest3678-21cut19:55:170
24498991810,6cyclictest3678-21cut19:55:170
24498991810,6cyclictest28292-21sshd20:51:210
2452899179,6cyclictest0-21swapper/321:58:543
2452899179,6cyclictest0-21swapper/321:41:153
2452899179,6cyclictest0-21swapper/300:10:503
2452899179,3cyclictest0-21swapper/323:54:113
2452899178,6cyclictest35966irq/58-eth023:00:123
2452899172,10cyclictest0-21swapper/319:20:133
24528991710,5cyclictest35966irq/58-eth022:30:133
2451999175,8cyclictest24588-21latency_hist19:10:012
2450899179,5cyclictest0-21swapper/120:40:261
2450899178,6cyclictest7638-21sshd19:57:151
2450899178,6cyclictest7638-21sshd19:57:141
2450899171,13cyclictest30437-21sshd20:20:211
2450899171,12cyclictest23385-21apt-get20:50:031
2449899179,6cyclictest25137-21cut19:10:180
2449899179,6cyclictest11618-21cut19:30:160
2449899178,7cyclictest24491-21ntp_192.168.11419:40:230
2449899175,9cyclictest10707-21sort21:10:130
2449899172,12cyclictest4814-21unixbench_singl21:00:260
2449899171,3cyclictest131rcu_preempt21:15:230
2449899171,14cyclictest3276-21timerandwakeup20:25:280
2449899171,12cyclictest30760-21kthreadcore20:55:200
2449899171,12cyclictest15484-21diskstats23:00:140
24498991710,5cyclictest29333-21open_files19:45:260
2449899170,6cyclictest13520-21kthreadcore22:40:210
2452899162,10cyclictest0-21swapper/320:35:143
2450899168,6cyclictest17307-21readlink20:10:031
2450899167,3cyclictest131rcu_preempt19:30:131
2450899164,9cyclictest30031-21cpu00:20:141
2450899163,10cyclictest28839-21kthreadcore19:45:221
2450899162,11cyclictest23674-21kthreadcore20:15:201
2450899161,12cyclictest25134-21cut19:10:181
2450899161,11cyclictest15750-21df00:30:151
2449899162,11cyclictest6638-21kthreadcore21:05:210
2449899161,12cyclictest8085-21cut20:00:160
2449899161,11cyclictest30973-21cat20:20:270
2452899151,11cyclictest0-21swapper/319:45:273
2452899149,3cyclictest0-21swapper/321:07:453
2452899149,3cyclictest0-21swapper/319:50:263
2452899143,8cyclictest0-21swapper/320:25:153
2452899143,6cyclictest35966irq/58-eth020:05:113
2452899142,6cyclictest0-21swapper/320:00:143
2452899141,8cyclictest35966irq/58-eth019:10:143
2452899141,11cyclictest0-21swapper/319:30:173
2452899141,10cyclictest0-21swapper/319:35:133
2449899144,7cyclictest24497-21latency_hist19:10:020
2452899131,9cyclictest0-21swapper/320:55:283
2450899130,10cyclictest24381-21latency_hist19:10:011
2452899127,3cyclictest0-21swapper/320:40:143
2452899127,3cyclictest0-21swapper/320:16:273
2452899121,7cyclictest0-21swapper/319:20:013
2452899121,5cyclictest0-21swapper/321:05:023
2452899121,4cyclictest0-21swapper/319:40:203
2452899111,7cyclictest0-21swapper/320:45:183
2452899101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional