You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-25 - 08:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat Jul 25, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20488998857,17cyclictest35966irq/58-eth000:00:143
20476995537,9cyclictest21-21ksoftirqd/119:55:131
2048499545,36cyclictest15863-21ssh00:15:112
2048899519,32cyclictest35966irq/58-eth023:45:133
2048899506,30cyclictest35966irq/58-eth022:05:123
20484994832,10cyclictest26598-21apt-get22:15:122
20484994830,10cyclictest27-21ksoftirqd/223:45:142
2047699483,7cyclictest131rcu_preempt20:45:131
20468994813,5cyclictest131rcu_preempt22:15:130
2048899466,23cyclictest35966irq/58-eth021:45:133
20484994615,26cyclictest0-21swapper/220:35:132
2048899451,2cyclictest0-21swapper/300:05:143
2046899454,37cyclictest19426-21ssh21:35:140
20484994325,15cyclictest27983-21apt-get23:10:122
20484994325,15cyclictest27983-21apt-get23:10:122
2047699433,13cyclictest21-21ksoftirqd/121:05:121
20484994231,8cyclictest30287-21apt-get19:20:122
20484994225,4cyclictest21092-21ssh22:30:122
20484994219,18cyclictest0-21swapper/221:45:122
2048499421,21cyclictest1591-21ssh00:25:132
2048499421,21cyclictest1591-21ssh00:25:132
20488994033,5cyclictest35966irq/58-eth023:05:133
2048499408,12cyclictest6987-21mailstats22:20:282
20484994021,15cyclictest27346irq/40-dwc2_hso23:25:132
20488993922,10cyclictest35966irq/58-eth023:00:113
2048899391,21cyclictest35966irq/58-eth022:40:153
20484993931,4cyclictest14847-21apt-get20:45:132
20484993914,15cyclictest6301-21diskmemload23:59:412
2048899385,19cyclictest35966irq/58-eth022:00:123
2048899385,19cyclictest35966irq/58-eth022:00:123
2048899381,3cyclictest0-21swapper/300:20:163
2048499388,14cyclictest1790-21wc00:05:312
20484993819,15cyclictest0-21swapper/200:30:132
20484993814,18cyclictest0-21swapper/219:40:112
20484993811,20cyclictest0-21swapper/221:40:132
2048899372,25cyclictest35966irq/58-eth021:00:143
2048899371,2cyclictest0-21swapper/323:10:133
2048899371,2cyclictest0-21swapper/323:10:123
2048499379,12cyclictest23167-21latency00:35:242
20484993731,3cyclictest24472-21apt-get22:50:142
20484993724,10cyclictest19016-21apt-get23:05:122
20484993718,14cyclictest0-21swapper/222:55:142
20484993712,6cyclictest28491-21ssh21:20:272
20484993711,18cyclictest14049-21processes21:30:282
2047699371,28cyclictest1265-21ssh21:25:141
2048899361,2cyclictest0-21swapper/300:30:123
2048499367,15cyclictest31507-21ssh00:22:282
20484993628,3cyclictest1722-21apt-get21:05:112
2048499362,18cyclictest14181-21ssh00:13:302
20484993621,11cyclictest27346irq/40-dwc2_hso21:50:142
20484993613,9cyclictest6301-21diskmemload21:10:502
20484993610,16cyclictest6301-21diskmemload22:09:352
2048899354,26cyclictest0-21swapper/322:55:133
2048899351,3cyclictest0-21swapper/323:50:123
20484993517,14cyclictest0-21swapper/222:10:122
20484993511,8cyclictest17458-21fschecks_count23:40:162
2048899345,17cyclictest35966irq/58-eth021:55:133
2048899343,2cyclictest0-21swapper/319:35:123
2048899341,3cyclictest0-21swapper/300:25:133
2048899341,3cyclictest0-21swapper/300:25:133
20484993415,12cyclictest0-21swapper/223:15:122
20484993414,9cyclictest6290-21switchtime19:25:272
20484993414,9cyclictest6290-21switchtime19:25:272
20484993414,6cyclictest7143-21ssh23:34:152
20484993410,11cyclictest6301-21diskmemload23:00:362
2046899342,5cyclictest131rcu_preempt21:45:110
2046899341,30cyclictest13584-21ssh23:20:130
2048899333,2cyclictest0-21swapper/321:15:113
2048899333,26cyclictest0-21swapper/323:15:133
2048899332,20cyclictest35966irq/58-eth020:40:113
20488993317,8cyclictest35966irq/58-eth021:20:133
20488993315,10cyclictest35966irq/58-eth021:30:153
2048499339,16cyclictest24127-21kthreadcore20:20:192
2048499334,18cyclictest0-21swapper/222:42:332
20484993317,10cyclictest0-21swapper/220:15:132
2047699332,20cyclictest2948-21ssh23:50:111
2048899324,19cyclictest35966irq/58-eth022:25:123
2048899321,4cyclictest0-21swapper/323:25:133
2048899321,2cyclictest0-21swapper/320:35:133
2048499329,19cyclictest31858-21tail20:25:252
2048499327,11cyclictest6301-21diskmemload00:04:292
2048499326,8cyclictest6137-21ssh21:25:312
2048499326,13cyclictest8786-21/usr/sbin/munin20:05:252
2048499326,13cyclictest17413-21kworker/u8:122:39:522
20484993216,9cyclictest13851-21unixbench_multi20:40:272
20484993212,16cyclictest0-21swapper/219:55:122
20484993211,12cyclictest12335-21/usr/sbin/munin19:35:242
20484993210,8cyclictest18184-21ssh22:26:592
2047699323,22cyclictest32069-21ssh22:00:121
2047699323,22cyclictest32069-21ssh22:00:121
2046899322,27cyclictest19024-21ssh23:05:130
2048899313,24cyclictest0-21swapper/320:25:103
2048899311,2cyclictest0-21swapper/320:30:123
2048899311,1cyclictest0-21swapper/322:20:133
2048499318,7cyclictest13546-21/usr/sbin/munin23:20:142
2048499316,17cyclictest10169-21ssh23:53:142
2048499316,11cyclictest0-21swapper/219:50:292
2048499315,10cyclictest0-21swapper/221:15:402
20484993114,13cyclictest3496-21kthreadcore20:30:212
20484993114,13cyclictest12220-21latency_hist19:35:012
2048899303,20cyclictest0-21swapper/300:10:133
2048899301,24cyclictest0-21swapper/319:40:123
20488993012,12cyclictest0-21swapper/322:54:533
2048499309,10cyclictest19502-21sh21:35:142
2048499308,8cyclictest16997-21ntp_192.168.11420:10:232
2048499307,12cyclictest26118-21usage_Anybus19:45:282
20484993014,14cyclictest16223-21ssh23:39:192
20484993012,15cyclictest29341-21/usr/sbin/munin21:00:312
20484993012,13cyclictest27346irq/40-dwc2_hso21:57:402
2047699302,22cyclictest922-21ssh23:48:031
2047699302,22cyclictest24410-21ssh21:35:401
2047699302,22cyclictest21630-21sshd21:16:541
20468993013,15cyclictest12-21ksoftirqd/023:25:120
2048899295,13cyclictest35966irq/58-eth000:15:113
2048899294,21cyclictest0-21swapper/319:25:133
2048899294,21cyclictest0-21swapper/319:25:133
2048899293,22cyclictest0-21swapper/321:40:113
2048899292,21cyclictest0-21swapper/322:30:123
2048899291,2cyclictest0-21swapper/323:30:143
2048899291,2cyclictest0-21swapper/320:20:123
20484992913,12cyclictest0-21swapper/220:50:152
20484992912,9cyclictest23524-21ssh22:49:202
20484992912,14cyclictest7637-21cat20:00:272
20484992910,16cyclictest29347-21head19:15:272
2046899294,7cyclictest131rcu_preempt23:00:130
2046899293,22cyclictest12-21ksoftirqd/020:10:130
2048899282,2cyclictest0-21swapper/321:05:123
2048899281,2cyclictest0-21swapper/322:15:133
2048899281,22cyclictest0-21swapper/322:10:133
20484992812,13cyclictest4619-21ssh22:00:382
20484992812,13cyclictest4619-21ssh22:00:382
2047699281,20cyclictest17674-21ssh22:10:131
2046899287,18cyclictest28011-21ssh23:10:130
2046899287,18cyclictest28011-21ssh23:10:130
2046899285,7cyclictest131rcu_preempt22:35:100
20468992821,4cyclictest630-21Async09:36:310
20468992818,7cyclictest6301-21diskmemload21:40:130
20468992814,7cyclictest131rcu_preempt21:00:150
2048899273,19cyclictest0-21swapper/319:55:133
2048899271,3cyclictest0-21swapper/319:10:263
2048899271,2cyclictest0-21swapper/322:45:593
2048899271,2cyclictest0-21swapper/321:10:123
2048899271,21cyclictest0-21swapper/321:50:143
2048899271,1cyclictest0-21swapper/323:55:133
2048499272,19cyclictest27346irq/40-dwc2_hso19:10:132
20484992711,13cyclictest29241-21latency_hist21:00:022
2047699272,21cyclictest26620-21ssh22:15:131
2047699272,19cyclictest19885-21ssh21:51:401
2046899279,9cyclictest131rcu_preempt19:10:120
2046899275,17cyclictest12-21ksoftirqd/020:35:130
2046899273,21cyclictest21248-21diskmemload00:00:150
2046899272,19cyclictest20155-21ssh21:52:000
20468992710,4cyclictest131rcu_preempt23:15:120
2048899266,12cyclictest0-21swapper/323:20:133
2048899262,2cyclictest0-21swapper/320:45:143
2048899261,2cyclictest0-21swapper/300:35:143
2047699262,17cyclictest12111-21ssh22:25:131
20476992613,7cyclictest0-21swapper/123:25:561
2047699260,21cyclictest25458-21apt-get19:15:121
2046899263,15cyclictest6301-21diskmemload22:50:130
20468992610,12cyclictest8178-21diskmemload23:35:130
2048899254,14cyclictest35966irq/58-eth020:00:113
2048899251,2cyclictest0-21swapper/323:44:103
2048899251,2cyclictest0-21swapper/322:35:113
2048899251,2cyclictest0-21swapper/320:10:123
2048899251,2cyclictest0-21swapper/319:45:123
2048899251,21cyclictest0-21swapper/320:55:293
2048899251,21cyclictest0-21swapper/319:50:153
2048899251,21cyclictest0-21swapper/319:20:203
2048899251,20cyclictest0-21swapper/323:35:293
2048899251,20cyclictest0-21swapper/320:05:113
2048899250,20cyclictest35966irq/58-eth021:35:123
2046899257,3cyclictest12-21ksoftirqd/023:50:110
2046899254,15cyclictest8665-21ssh22:05:120
2046899252,7cyclictest131rcu_preempt21:11:400
2046899251,18cyclictest10589-21sh00:30:130
2046899251,13cyclictest19713-21ssh00:35:160
2048899241,2cyclictest0-21swapper/321:26:183
2048899241,2cyclictest0-21swapper/319:30:133
2047699241,7cyclictest24566-21idleruntime-cro00:20:011
2046899246,5cyclictest131rcu_preempt22:55:130
2046899244,3cyclictest131rcu_preempt19:45:160
2046899242,17cyclictest30504-21kworker/u8:000:25:160
2046899242,17cyclictest30504-21kworker/u8:000:25:150
20468992418,3cyclictest0-21swapper/000:10:130
2048899231,5cyclictest0-21swapper/319:15:153
2048899231,2cyclictest0-21swapper/320:50:123
2048899231,2cyclictest0-21swapper/320:15:283
2047699233,15cyclictest30227-21ssh00:05:151
2047699232,17cyclictest5210-21apt-get20:35:121
2047699232,17cyclictest15544-21ssh22:45:141
20476992318,3cyclictest6806-21sh00:10:131
2047699231,17cyclictest31666-21ssh23:30:151
20476992311,3cyclictest131rcu_preempt00:00:291
2046899234,17cyclictest16087-21ssh21:31:400
2046899233,3cyclictest131rcu_preempt21:05:120
2046899232,18cyclictest20628-21rm21:15:400
2046899231,15cyclictest0-21swapper/019:20:110
2046899230,2cyclictest131rcu_preempt21:55:530
20484992210,9cyclictest20487-21latency_hist19:10:012
2047699222,3cyclictest131rcu_preempt21:30:151
2047699221,4cyclictest7943-21ssh21:45:211
2047699221,18cyclictest497-21ls21:00:281
2046899229,9cyclictest32058-21apt-get22:00:120
2046899229,9cyclictest32058-21apt-get22:00:120
2046899228,4cyclictest131rcu_preempt20:45:270
2046899226,12cyclictest12-21ksoftirqd/000:15:130
2046899223,15cyclictest1471-21ssh00:05:280
2046899222,14cyclictest24673-21ssh21:20:140
20468992214,5cyclictest20551-21sshd22:29:500
2046899221,3cyclictest131rcu_preempt19:40:170
2046899221,16cyclictest12090-21ssh23:55:140
2047699212,14cyclictest0-21swapper/120:26:401
2047699212,12cyclictest21-21ksoftirqd/122:55:141
2047699211,9cyclictest24843-21tr20:20:191
20476992113,5cyclictest12037-21/usr/sbin/munin23:55:201
2047699211,17cyclictest3164-21ssh22:20:151
2047699211,17cyclictest22035-21find19:45:121
2047699210,15cyclictest28472-21ls00:40:021
2046899213,13cyclictest12-21ksoftirqd/020:05:120
20468992115,4cyclictest0-21swapper/022:40:150
20468992113,5cyclictest121ksoftirqd/022:45:120
2046899211,14cyclictest29820-21ls23:45:220
2047699207,4cyclictest14906-21users23:00:301
2047699205,8cyclictest27409-1kworker/u9:223:10:131
2047699205,8cyclictest27409-1kworker/u9:223:10:131
2047699202,12cyclictest28831-21ssh21:56:391
20476992012,6cyclictest13365-21sshd20:06:461
2047699201,15cyclictest17479-21cut23:40:161
20476992011,4cyclictest21-21ksoftirqd/120:55:141
2047699201,10cyclictest16233-21kthreadcore20:10:201
2047699200,8cyclictest14416-21mailstats00:30:271
2046899202,14cyclictest3163-21diskmemload22:20:150
20468992013,5cyclictest0-21swapper/023:30:140
2046899201,17cyclictest7357-21latency_hist19:30:010
2046899201,17cyclictest7357-21latency_hist19:30:010
2046899201,10cyclictest1893-21irqrtprio21:25:180
2047699199,8cyclictest30374-21df_inode19:20:141
2047699199,7cyclictest19750-21diskstats20:50:151
2047699198,3cyclictest131rcu_preempt23:10:011
2047699192,11cyclictest30686-21ssh21:21:551
2047699191,3cyclictest0-21swapper/122:50:141
20476991913,4cyclictest28753-21ntp_states00:20:281
20476991913,4cyclictest0-21swapper/119:25:171
20476991913,4cyclictest0-21swapper/119:25:161
2047699191,2cyclictest131rcu_preempt19:50:121
2046899198,8cyclictest21904-21cat23:40:310
2046899197,9cyclictest1-21systemd20:21:480
2046899197,3cyclictest131rcu_preempt22:10:210
2046899196,9cyclictest12847-21grep19:35:170
2046899192,12cyclictest24716-21ntp_192.168.11422:30:240
2046899190,14cyclictest4085-21timerandwakeup20:30:270
2048899181,2cyclictest0-21swapper/319:10:013
2047699189,7cyclictest25222-21unixbench_singl22:30:301
2047699188,7cyclictest1588-21apt-get00:25:141
2047699188,7cyclictest1588-21apt-get00:25:141
2047699184,9cyclictest0-21swapper/121:10:131
2047699184,12cyclictest28804-21sh21:40:161
2047699182,12cyclictest1255-21ntp_states22:35:241
2047699181,9cyclictest0-21swapper/119:30:151
2047699181,14cyclictest9714-21kthreadcore22:05:191
2047699181,14cyclictest4603-21ssh23:15:131
20476991811,4cyclictest0-21swapper/123:35:111
2046899181,8cyclictest12261-21cat19:35:010
2046899181,13cyclictest29055-21sendmail-msp21:00:010
2047699179,5cyclictest16443-21cut23:20:211
2047699172,10cyclictest0-21swapper/122:40:151
20476991712,3cyclictest19065-21grep20:15:191
2046899176,3cyclictest131rcu_preempt20:00:160
2046899173,10cyclictest0-21swapper/020:25:100
2046899172,4cyclictest131rcu_preempt19:50:140
2046899172,2cyclictest131rcu_preempt19:15:160
2046899171,6cyclictest24235-21latency_hist20:55:010
20468991710,5cyclictest18718-21sed20:15:180
2047699167,6cyclictest21470-21kthreadcore19:10:181
2047699166,2cyclictest0-21swapper/119:40:171
2047699161,2cyclictest131rcu_preempt20:45:011
2047699161,13cyclictest0-21swapper/119:35:131
2047699161,12cyclictest4214-21fschecks_time20:00:151
2046899161,6cyclictest3096-21grep19:55:280
2046899161,12cyclictest11342-21kthreadcore20:40:180
2047699151,12cyclictest0-21swapper/120:30:131
2047699150,5cyclictest131rcu_preempt19:10:011
2046899142,9cyclictest20470-21latency_hist19:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional