You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 11:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Jul 07, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2613899609,6cyclictest131rcu_preempt21:05:120
26138995723,9cyclictest0-21swapper/022:10:110
2615599543,27cyclictest7404-21diskmemload21:45:122
2613899548,36cyclictest7810-21ssh21:10:140
2614799523,38cyclictest16814-21ssh21:15:131
2614799498,36cyclictest24683-21ssh00:20:131
2613899485,36cyclictest10285-21ssh00:30:140
2613899472,31cyclictest21228-21ssh00:00:100
26155994627,10cyclictest27-21ksoftirqd/223:20:142
2614799461,35cyclictest32378-21ssh23:30:131
2613899463,33cyclictest7404-21diskmemload00:10:150
2616399457,24cyclictest35966irq/58-eth020:15:133
2614799454,30cyclictest15651-21ssh00:15:151
2614799451,36cyclictest29500-21ssh21:40:141
26147994338,3cyclictest17825-21sh23:40:121
2616399423,25cyclictest35966irq/58-eth021:35:163
2615599422,20cyclictest29041-21ssh23:10:132
2616399414,23cyclictest35966irq/58-eth000:20:123
26163993918,13cyclictest0-21swapper/322:15:143
26163993918,13cyclictest0-21swapper/322:15:143
2613899391,2cyclictest0-21swapper/023:25:150
2616399383,24cyclictest35966irq/58-eth020:30:123
2616399382,20cyclictest35966irq/58-eth000:15:123
2616399374,20cyclictest35966irq/58-eth019:45:113
2616399371,25cyclictest35966irq/58-eth021:40:133
2615599372,20cyclictest530-21ssh22:00:142
2613899370,25cyclictest2438-21ssh21:25:120
2615599367,22cyclictest30995-21ssh22:35:122
2616399352,17cyclictest35966irq/58-eth021:10:143
2616399342,18cyclictest35966irq/58-eth023:25:143
26163993411,9cyclictest35966irq/58-eth021:15:133
2614799342,22cyclictest0-21swapper/122:45:141
26138993319,9cyclictest12-21ksoftirqd/021:45:120
2613899331,29cyclictest24021-21ssh21:55:150
2616399322,19cyclictest35966irq/58-eth023:55:133
2616399322,19cyclictest35966irq/58-eth023:55:133
2615599325,16cyclictest27346irq/40-dwc2_hso22:50:142
2615599321,21cyclictest9533-21ssh22:05:122
2615599321,21cyclictest9533-21ssh22:05:122
2616399311,19cyclictest35966irq/58-eth023:35:123
26147993112,9cyclictest21-21ksoftirqd/121:55:141
2613899311,24cyclictest13158-21ssh22:25:150
2616399303,13cyclictest35966irq/58-eth022:10:103
26147993019,5cyclictest7799-21apt-get21:10:141
2613899303,20cyclictest25513-21ssh22:50:130
2615599294,14cyclictest27346irq/40-dwc2_hso19:40:132
2615599291,16cyclictest23476-21ssh23:25:172
26147992922,4cyclictest11247-21apt-get20:40:121
26147992922,3cyclictest18226-21apt-get19:40:121
2614799292,21cyclictest26787-21ssh23:45:131
26147992914,11cyclictest23449-21apt-get23:25:161
2613899291,24cyclictest11497-21ssh21:30:150
2613899291,19cyclictest4111-21ssh22:20:130
26163992822,4cyclictest0-21swapper/322:05:113
26163992822,4cyclictest0-21swapper/322:05:113
26163992815,5cyclictest35966irq/58-eth022:20:113
2616399281,15cyclictest35966irq/58-eth023:10:133
2615599282,17cyclictest32387-21ssh23:30:142
2614799282,20cyclictest844-21munin-node19:20:161
2614799281,18cyclictest2420-21sh21:25:111
2613899282,2cyclictest131rcu_preempt19:35:120
2613899282,20cyclictest31004-21ssh22:35:130
2616399274,16cyclictest35966irq/58-eth023:00:143
26163992714,8cyclictest35966irq/58-eth022:50:133
26163992711,9cyclictest35966irq/58-eth000:30:153
2616399271,15cyclictest35966irq/58-eth022:55:173
2616399270,13cyclictest35966irq/58-eth020:50:133
2615599275,12cyclictest27346irq/40-dwc2_hso21:10:142
26147992720,4cyclictest536-21usb-storage22:20:111
26138992710,10cyclictest17845-21ssh23:40:130
2616399264,13cyclictest35966irq/58-eth020:25:143
2616399263,15cyclictest35966irq/58-eth000:10:123
2616399262,15cyclictest35966irq/58-eth023:50:133
2616399262,15cyclictest35966irq/58-eth023:50:133
2615599261,8cyclictest1936166ssh00:15:282
2614799268,12cyclictest11079-21apt-get23:00:131
26147992619,3cyclictest503-21apt-get22:00:131
26147992616,6cyclictest7404-21diskmemload23:15:271
2613899260,20cyclictest11095-21sh23:00:130
2616399252,15cyclictest35966irq/58-eth000:05:143
2615599253,14cyclictest27346irq/40-dwc2_hso20:40:122
2615599253,12cyclictest27346irq/40-dwc2_hso20:15:112
2615599251,17cyclictest23514-21latency23:05:222
2615599251,10cyclictest12249-21kthreadcore21:30:222
2614799256,15cyclictest0-21swapper/100:05:131
2614799254,6cyclictest21-21ksoftirqd/122:35:011
2614799252,18cyclictest844-21munin-node20:25:141
2614799251,13cyclictest9519-21ssh22:05:111
2614799251,13cyclictest9519-21ssh22:05:111
2613899259,6cyclictest131rcu_preempt22:00:120
2613899253,14cyclictest0-21swapper/020:45:150
2613899252,19cyclictest30224-21ssh00:05:150
2613899252,18cyclictest22097-21ssh22:30:150
2615599247,14cyclictest11097-21sh23:00:132
2615599247,13cyclictest10680-21apt-config23:00:012
2615599243,3cyclictest0-21swapper/221:05:132
2615599241,15cyclictest12290-21sh23:55:162
2615599241,15cyclictest12290-21sh23:55:152
26147992417,4cyclictest1241-21apt-get00:25:151
26147992415,5cyclictest2820-21apt-get21:05:141
2614799241,17cyclictest12291-21ssh23:55:151
2614799241,17cyclictest12291-21ssh23:55:151
2614799240,14cyclictest7404-21diskmemload22:59:291
2615599239,11cyclictest29600-21df_inode21:40:152
2615599236,8cyclictest17814-21apt-get23:40:132
2615599234,7cyclictest0-21swapper/222:10:102
2615599231,6cyclictest14304-21grep23:20:022
2615599231,19cyclictest23222-21grep19:45:142
2615599231,13cyclictest5894-21cut22:20:202
2615599231,13cyclictest29093-21apt21:40:012
2615599231,12cyclictest16603-21ls22:45:142
26155992310,11cyclictest0-21swapper/219:50:122
2614799238,4cyclictest0-21swapper/100:14:551
2614799237,4cyclictest0-21swapper/122:41:141
2614799235,9cyclictest0-21swapper/123:05:361
2614799235,7cyclictest201rcuc/123:12:001
26147992313,7cyclictest21-21ksoftirqd/119:45:111
2614799231,19cyclictest5007-21ls20:00:011
26147992310,10cyclictest16585-21cut23:20:231
2614799230,6cyclictest18829-21apt-get00:35:031
2616399225,10cyclictest35966irq/58-eth023:45:133
2616399223,13cyclictest0-21swapper/323:30:153
2615599222,17cyclictest353-21kthreadcore00:05:232
2615599222,17cyclictest0-21swapper/221:00:142
2615599221,6cyclictest6811-21sort00:10:142
2615599221,5cyclictest1938166ssh22:27:152
2615599221,18cyclictest1647-21grep00:25:192
2615599221,12cyclictest26736-21/usr/sbin/munin23:45:232
2615599221,12cyclictest2044-21cron21:25:002
2615599221,12cyclictest2044-21cron21:25:002
2615599221,11cyclictest23993-21apt-get21:55:152
2615599221,10cyclictest23269-21cut22:30:222
26155992210,9cyclictest23223-21ssh21:17:212
2614799227,9cyclictest0-21swapper/121:34:341
2614799227,4cyclictest0-21swapper/100:31:561
2614799226,4cyclictest0-21swapper/123:50:031
2614799226,4cyclictest0-21swapper/123:50:031
2614799226,4cyclictest0-21swapper/122:37:491
2614799226,3cyclictest0-21swapper/123:38:081
2614799224,9cyclictest21-21ksoftirqd/122:29:481
2614799223,5cyclictest21-21ksoftirqd/121:23:371
2614799223,5cyclictest21-21ksoftirqd/121:23:371
2614799223,4cyclictest21-21ksoftirqd/120:35:251
2614799222,5cyclictest21-21ksoftirqd/120:30:271
2614799222,4cyclictest21-21ksoftirqd/121:35:301
26147992215,4cyclictest30431-21apt-get21:00:131
2613899223,15cyclictest12-21ksoftirqd/021:50:130
2613899222,9cyclictest16523-21ssh22:45:140
26138992215,5cyclictest0-21swapper/023:30:140
2613899221,17cyclictest20806-21find20:50:130
2616399219,9cyclictest0-21swapper/323:40:393
2616399219,9cyclictest0-21swapper/322:36:503
2615599217,11cyclictest13288-21missed_timers20:05:222
2615599217,11cyclictest13288-21missed_timers20:05:222
2615599215,12cyclictest0-21swapper/222:40:292
2615599212,16cyclictest23495-21latency_hist00:40:022
2615599212,11cyclictest9097-21grep23:35:162
2615599212,10cyclictest24749-21grep00:00:212
2615599212,10cyclictest18388-21kthreadcore20:45:222
2615599211,9cyclictest740466diskmemload23:50:302
2615599211,9cyclictest740466diskmemload23:50:302
2615599211,12cyclictest6404-21kthreadcore20:00:192
2615599211,12cyclictest18694-21open_files21:50:262
2615599211,11cyclictest31428-21sort22:15:292
2615599211,11cyclictest31428-21sort22:15:292
2615599211,10cyclictest3687-21kthreadcore21:25:182
2615599211,10cyclictest10291-21ssh00:30:152
26155992110,8cyclictest0-21swapper/220:25:182
26155992110,8cyclictest0-21swapper/219:55:162
2614799216,5cyclictest21-21ksoftirqd/122:53:191
2614799215,4cyclictest0-21swapper/122:10:471
2614799214,4cyclictest12406-21ssh21:47:221
2614799213,6cyclictest21-21ksoftirqd/100:03:371
2613899211,6cyclictest23305-21munin-run00:40:020
2613899211,12cyclictest0-21swapper/019:40:130
2616399209,8cyclictest0-21swapper/321:21:233
2616399209,8cyclictest0-21swapper/321:21:223
2616399209,8cyclictest0-21swapper/300:27:433
2616399206,9cyclictest35966irq/58-eth021:55:153
2616399202,9cyclictest0-21swapper/321:25:133
26163992011,6cyclictest0-21swapper/322:45:163
2616399201,14cyclictest0-21swapper/319:10:013
2615599206,11cyclictest24772-21df00:20:152
2615599205,12cyclictest13161-21cron19:35:002
2615599204,13cyclictest6500-21cpuspeed20:35:142
2615599203,8cyclictest26705-21fschecks_time19:15:142
2615599202,14cyclictest1789-21fschecks_count20:30:152
2615599201,9cyclictest616-21systemd-logind20:12:152
2615599201,16cyclictest25383-21cron20:55:012
2615599201,11cyclictest26323-21munin-run19:15:012
2615599201,10cyclictest31561-21fschecks_time19:20:172
2615599200,16cyclictest3919-21grep19:25:172
2614799202,5cyclictest26343-21sort19:15:021
26147992012,6cyclictest0-21swapper/119:30:251
2614799201,14cyclictest23293-21timerwakeupswit20:15:261
2613899204,2cyclictest131rcu_preempt19:15:010
2613899203,2cyclictest131rcu_preempt23:50:130
2613899203,2cyclictest131rcu_preempt23:50:120
2613899202,14cyclictest8301-21apt-get19:30:040
2613899202,13cyclictest14520-21ssh23:20:140
26138992014,4cyclictest0-21swapper/022:20:020
26138992014,4cyclictest0-21swapper/022:20:020
2613899201,14cyclictest1269-21ssh00:25:150
2613899201,13cyclictest28006-21switchtime20:20:280
2616399192,11cyclictest0-21swapper/300:00:123
2616399192,10cyclictest0-21swapper/319:50:053
26163991911,5cyclictest35966irq/58-eth020:35:153
2616399191,13cyclictest0-21swapper/323:20:153
26163991911,3cyclictest0-21swapper/323:19:353
26163991910,6cyclictest0-21swapper/300:35:293
2615599191,14cyclictest26178-21cat19:10:022
2615599191,10cyclictest26199-21irqrtprio20:55:202
2615599191,10cyclictest14646-21kthreadcore19:35:212
2615599190,16cyclictest24620-21ls20:20:172
2614799197,9cyclictest31455-21ntp_states19:50:251
2614799193,4cyclictest21-21ksoftirqd/122:15:121
2614799193,4cyclictest21-21ksoftirqd/122:15:111
2614799192,13cyclictest16137-21kthreadcore19:35:211
2614799191,4cyclictest7036-21memory19:25:231
26147991914,3cyclictest16032-21apt-get20:45:141
26147991910,6cyclictest14983-21apt-get21:50:121
2613899197,9cyclictest12397-21missed_timers23:35:230
2613899191,9cyclictest25756-21cut19:45:190
2613899191,9cyclictest15601-21/usr/sbin/munin00:15:300
2613899191,3cyclictest0-21swapper/023:15:210
26138991912,5cyclictest0-21swapper/020:15:230
2613899191,13cyclictest29044-21sh23:10:130
2616399189,6cyclictest0-21swapper/321:50:513
2616399182,11cyclictest0-21swapper/320:45:153
2616399182,10cyclictest0-21swapper/320:05:133
2616399182,10cyclictest0-21swapper/320:05:133
26163991811,5cyclictest35966irq/58-eth022:40:143
26163991810,6cyclictest0-21swapper/322:25:393
2614799188,8cyclictest29022-21ntpq20:55:261
2614799186,9cyclictest5269-21diskstats20:00:151
2614799185,9cyclictest15084-21irqcore20:10:161
2614799185,3cyclictest0-21swapper/119:15:121
2614799183,4cyclictest21-21ksoftirqd/120:05:181
2614799183,4cyclictest21-21ksoftirqd/120:05:181
2614799182,10cyclictest0-21swapper/120:50:161
2613899187,8cyclictest21560-21cut21:35:230
2613899186,9cyclictest25653-21cut21:20:010
2613899184,10cyclictest12-21ksoftirqd/019:55:150
2613899182,12cyclictest10610-21munin-run23:00:010
2613899181,9cyclictest29641-21irqcore20:25:190
2613899181,7cyclictest31493-21kthreadcore21:40:210
2613899181,3cyclictest0-21swapper/023:45:490
2613899181,12cyclictest12499-21ssh23:55:170
2613899181,12cyclictest12499-21ssh23:55:170
2616399179,6cyclictest0-21swapper/321:45:053
2616399179,5cyclictest35966irq/58-eth022:30:193
2616399172,9cyclictest0-21swapper/320:55:153
2616399172,9cyclictest0-21swapper/319:55:153
2616399172,11cyclictest0-21swapper/321:30:153
2616399172,10cyclictest0-21swapper/320:40:143
2616399172,10cyclictest0-21swapper/319:30:133
26163991711,4cyclictest0-21swapper/322:01:363
26163991710,4cyclictest0-21swapper/323:05:153
2614799175,9cyclictest26160-21latency_hist19:10:011
2613899177,8cyclictest9986-21apt-get20:05:120
2613899177,8cyclictest9986-21apt-get20:05:120
2613899177,7cyclictest27218-21tr00:20:210
2613899172,10cyclictest0-21swapper/020:30:130
2613899171,9cyclictest18082-21memory20:10:210
2613899171,6cyclictest21397-21kthreadcore23:05:200
26138991711,4cyclictest0-21swapper/020:55:320
26138991710,5cyclictest27003-1kworker/u9:122:40:230
26138991710,4cyclictest0-21swapper/021:20:160
26138991710,4cyclictest0-21swapper/021:20:160
2616399168,5cyclictest35966irq/58-eth020:00:153
2616399163,11cyclictest321rcuc/319:35:133
2616399162,9cyclictest0-21swapper/319:15:123
2616399161,9cyclictest0-21swapper/320:20:163
2614799162,9cyclictest0-21swapper/120:20:141
2613899167,6cyclictest31176-21kthreadcore21:00:190
2613899166,7cyclictest4663-21kthreadcore19:25:200
2613899162,11cyclictest32702-21cat19:55:020
2613899162,11cyclictest11672-21kthreadcore22:05:200
2613899162,11cyclictest11672-21kthreadcore22:05:200
26138991611,3cyclictest14718-21open_inodes20:40:260
2613899161,12cyclictest31379-21apt-get19:20:140
2613899161,11cyclictest6338-21tr20:00:190
2616399159,4cyclictest0-21swapper/319:40:133
2616399151,8cyclictest0-21swapper/321:00:163
2613899155,8cyclictest6767-21irqcore20:35:180
2613899150,11cyclictest26599-21cpuspeed19:15:120
2616399141,7cyclictest0-21swapper/321:05:163
2613899141,10cyclictest26129-21cyclictest19:10:010
2616399126,4cyclictest0-21swapper/320:10:213
2616399121,8cyclictest0-21swapper/319:15:013
2616399121,5cyclictest0-21swapper/319:25:213
2616399113,6cyclictest0-21swapper/319:20:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional