You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 00:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon Sep 07, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2222099508,32cyclictest2934-21ssh09:25:121
2222099508,32cyclictest2934-21ssh09:25:121
2220999492,2cyclictest13914-21apt-get11:55:130
2223899484,32cyclictest35966irq/58-eth011:30:113
2222999462,39cyclictest31913-21ssh12:05:142
2220999462,3cyclictest0-21swapper/008:35:130
2222999446,24cyclictest24550-21ssh09:55:112
2220999442,2cyclictest0-21swapper/012:25:130
2220999442,2cyclictest0-21swapper/010:50:170
2220999441,3cyclictest0-21swapper/009:55:120
2220999441,2cyclictest8344-21apt-get09:10:120
2220999441,2cyclictest0-21swapper/008:15:130
2220999431,2cyclictest0-21swapper/012:30:120
2220999431,2cyclictest0-21swapper/012:30:120
22209994234,5cyclictest26526-21ssh12:20:130
2220999422,3cyclictest0-21swapper/008:40:130
2220999422,2cyclictest9219-21apt-get07:30:120
2223899415,21cyclictest35966irq/58-eth010:50:173
2223899411,20cyclictest35966irq/58-eth012:25:133
2220999412,4cyclictest0-21swapper/007:55:130
2220999412,3cyclictest0-21swapper/008:50:130
2222099401,32cyclictest17470-21ssh10:45:131
2220999402,2cyclictest0-21swapper/012:00:130
2220999401,4cyclictest31892-21ssh12:05:130
2220999401,4cyclictest0-21swapper/007:15:130
2220999401,2cyclictest0-21swapper/010:35:130
2222999391,8cyclictest29371-21cut07:50:192
2220999392,4cyclictest0-21swapper/007:50:140
2220999391,31cyclictest0-21swapper/010:10:180
2220999390,4cyclictest6553-21apt-get09:45:130
2223899386,18cyclictest35966irq/58-eth010:25:143
2223899384,22cyclictest35966irq/58-eth011:20:123
2222999381,9cyclictest21624-21kthreadcore11:40:212
2222999381,9cyclictest20640-21kthreadcore10:45:222
2222999381,13cyclictest25871-21ntpq07:10:252
2220999382,3cyclictest0-21swapper/008:00:130
2220999381,4cyclictest0-21swapper/007:40:130
2220999381,4cyclictest0-21swapper/007:40:130
2220999381,3cyclictest24968-21apt-get08:20:140
2220999381,3cyclictest0-21swapper/010:45:120
2222999372,19cyclictest14800-21ssh10:42:182
22229993716,18cyclictest24777-21ssh09:35:282
2222999371,18cyclictest0-21swapper/207:35:142
2222999371,18cyclictest0-21swapper/207:35:132
2222999371,13cyclictest13482-21kthreadcore11:00:202
2220999372,3cyclictest4411-21apt-get07:25:120
2220999371,3cyclictest0-21swapper/011:40:130
2220999370,5cyclictest2936-21apt-get09:25:140
2220999370,5cyclictest2936-21apt-get09:25:140
2223899360,22cyclictest35966irq/58-eth012:10:133
2222999367,7cyclictest0-21swapper/211:20:112
2222999362,28cyclictest23484-21sed10:30:182
2222999362,18cyclictest26650-21basename09:20:182
22220993617,10cyclictest630-21Async09:36:311
2220999363,29cyclictest11990-21ssh11:00:130
2220999362,31cyclictest28311-21sh10:15:120
2220999361,5cyclictest0-21swapper/012:10:140
2220999361,4cyclictest13900-21ssh10:25:120
2222999351,8cyclictest7192-21head12:25:292
2222099353,28cyclictest11921-21ssh09:30:121
2220999351,4cyclictest24744-21ssh11:25:130
2220999351,2cyclictest17510-21apt-get12:15:120
2220999351,2cyclictest1181-21apt-get10:00:140
2220999351,10cyclictest4373-21cut10:55:200
22238993417,11cyclictest35966irq/58-eth010:45:123
2222999341,9cyclictest28464-21fschecks_count10:15:152
2222999341,8cyclictest19959-21ntp_kernel_pll_08:45:252
2222999341,6cyclictest6239-21latency_hist11:15:012
2222999341,15cyclictest3185-21munin-run09:05:012
2222999341,13cyclictest8245-21kthreadcore10:20:222
2222999341,13cyclictest20150-21kthreadcore12:15:212
2222999340,9cyclictest8539-21/usr/sbin/munin12:10:302
2222999340,14cyclictest17790-21cut09:15:182
2222099343,16cyclictest6570-21ssh09:45:131
2220999343,3cyclictest15345-21apt-get08:10:130
2220999341,9cyclictest29969-21nfs_client09:20:250
2220999341,8cyclictest14205-21fw_packets07:35:150
2220999341,8cyclictest14205-21fw_packets07:35:150
2220999341,10cyclictest21196-21diskstats12:35:150
2222999332,9cyclictest0-21swapper/207:25:142
2222999332,10cyclictest31977-21kthreadcore08:25:212
2222999331,8cyclictest8328-21kthreadcore11:50:212
2222999331,8cyclictest1196-21ssh10:00:142
2222999331,11cyclictest12672-21ntp_states07:30:262
2222999330,13cyclictest10310-21latency_hist08:05:012
2220999335,25cyclictest27375-21sed07:45:270
2220999332,3cyclictest10538-21apt-get08:05:120
2220999332,2cyclictest0-21swapper/011:35:110
2220999331,9cyclictest5095-21ntp_states11:30:250
2220999331,11cyclictest29614-21awk11:10:010
2220999331,11cyclictest23289-21if_eth010:30:170
2220999331,11cyclictest1918-21memory09:00:220
2220999331,10cyclictest6182-21idleruntime-cro09:45:010
2220999331,10cyclictest18960-21tr09:50:230
2220999330,4cyclictest10170-21apt-get10:05:140
2220999330,3cyclictest26191-21apt-get08:55:110
2223899323,17cyclictest35966irq/58-eth009:00:123
2223899322,20cyclictest35966irq/58-eth010:05:123
2223899322,19cyclictest35966irq/58-eth010:10:143
2222999321,9cyclictest2129-21rm11:30:182
2222999321,8cyclictest12196-21fschecks_count12:30:162
2222999321,8cyclictest12196-21fschecks_count12:30:152
2222999321,6cyclictest6530-21/usr/sbin/munin09:45:262
2222999321,15cyclictest19186-21ssh10:10:142
2222099321,23cyclictest31903-21ssh12:05:141
2222099321,20cyclictest21-21ksoftirqd/111:55:131
2220999322,2cyclictest4858-21apt-get10:20:120
2220999321,9cyclictest18889-21cut09:15:200
2220999321,8cyclictest9173-21irqstats10:40:200
2220999321,8cyclictest15728-21diskstats11:20:130
2220999321,7cyclictest9794-21awk11:50:300
2220999321,7cyclictest0-21swapper/011:10:160
2220999321,4cyclictest29843-21diskstats08:25:150
2220999321,4cyclictest2747-21kernelversion08:30:190
2220999321,4cyclictest0-21swapper/009:05:120
2220999321,10cyclictest7366-21irqstats11:15:190
2220999321,10cyclictest21134-21/usr/sbin/munin09:35:160
2223899312,20cyclictest35966irq/58-eth009:15:133
2223899312,17cyclictest35966irq/58-eth010:15:113
2222999317,8cyclictest7951-21diskmemload12:02:482
2222999314,19cyclictest27346irq/40-dwc2_hso07:40:132
2222999314,19cyclictest27346irq/40-dwc2_hso07:40:132
2222999312,23cyclictest855-21kworker/u8:111:57:182
2222999312,11cyclictest15723-21ssh09:50:162
2222999311,8cyclictest23767-21switchtime08:15:272
2222999311,8cyclictest1534-21ssh09:40:272
2222999311,8cyclictest11927-21netstat09:10:232
22229993111,15cyclictest26492-21ssh10:50:182
2222099311,7cyclictest7951-21diskmemload11:10:161
2220999311,8cyclictest23541-21cut07:10:200
2220999311,8cyclictest11916-21apt09:30:120
2220999311,6cyclictest32049-21/usr/sbin/munin07:20:130
2220999311,11cyclictest32085-21ntp_states11:45:290
22238993011,6cyclictest35966irq/58-eth010:55:123
2222999301,8cyclictest930-21snmpd10:25:242
2222999300,13cyclictest1563766ssh09:30:262
2222099303,20cyclictest13906-21ssh10:25:131
2222099303,17cyclictest1378-21ssh11:30:121
2222099302,21cyclictest29638-21ssh09:55:431
2220999305,21cyclictest17031-21irqrtprio08:45:180
2222999296,9cyclictest17536-21ssh10:08:312
2222999294,15cyclictest27346irq/40-dwc2_hso11:45:142
2222999291,8cyclictest24503-21memory11:05:242
22229992912,3cyclictest930-21snmpd10:58:142
2222099292,21cyclictest20078-21ssh12:34:181
2222099292,21cyclictest20078-21ssh12:34:181
2223899283,18cyclictest0-21swapper/309:48:183
2223899283,16cyclictest0-21swapper/309:36:183
22238992814,8cyclictest35966irq/58-eth011:05:143
2222999285,13cyclictest27346irq/40-dwc2_hso11:25:132
2222099282,19cyclictest26495-21ssh09:36:181
2222099281,20cyclictest8356-21ssh09:10:131
2223899273,18cyclictest0-21swapper/309:50:423
2223899273,17cyclictest0-21swapper/311:50:183
2223899273,16cyclictest0-21swapper/309:20:183
2222999278,7cyclictest7951-21diskmemload10:37:582
2222999273,8cyclictest23723-21kworker/2:007:55:132
2222099273,19cyclictest17535-21ssh12:15:131
22220992715,7cyclictest0-21swapper/108:30:121
2223899265,18cyclictest35966irq/58-eth011:45:153
2223899263,17cyclictest0-21swapper/309:29:183
2223899263,17cyclictest0-21swapper/309:29:183
2223899263,15cyclictest35966irq/58-eth011:40:133
2223899263,15cyclictest0-21swapper/310:00:553
2223899263,15cyclictest0-21swapper/309:55:433
2222999263,15cyclictest27346irq/40-dwc2_hso08:35:132
22220992612,6cyclictest0-21swapper/110:00:131
22220992611,3cyclictest0-21swapper/107:45:301
2220999261,5cyclictest22282-21latency_hist07:10:010
2223899253,12cyclictest35966irq/58-eth012:30:123
2223899253,12cyclictest35966irq/58-eth012:30:123
2223899252,10cyclictest0-21swapper/309:10:113
2222999251,12cyclictest10538-21grep11:35:162
2222099259,8cyclictest21-21ksoftirqd/110:54:081
2222099253,5cyclictest21-21ksoftirqd/110:39:131
2222099253,3cyclictest131rcu_preempt10:20:451
2222099252,5cyclictest21-21ksoftirqd/111:07:361
2222099252,4cyclictest21-21ksoftirqd/108:50:021
22220992520,3cyclictest21-21ksoftirqd/109:50:211
2222999242,6cyclictest0-21swapper/209:25:182
2222999242,6cyclictest0-21swapper/209:25:182
2222999242,16cyclictest0-21swapper/207:20:152
22229992419,3cyclictest27-21ksoftirqd/208:20:122
22229992410,11cyclictest854-21sshd07:15:152
2222099248,6cyclictest21-21ksoftirqd/111:40:011
2222099245,7cyclictest21-21ksoftirqd/110:40:541
2222099244,3cyclictest131rcu_preempt12:20:251
22220992410,10cyclictest201rcuc/111:15:141
2223899232,14cyclictest0-21swapper/311:00:133
22238992314,6cyclictest35966irq/58-eth012:15:133
22238992313,6cyclictest0-21swapper/310:40:153
2222999234,3cyclictest0-21swapper/207:45:122
2222099236,8cyclictest25789-21kworker/1:011:43:231
2222099236,7cyclictest21-21ksoftirqd/109:20:561
2222099235,5cyclictest21-21ksoftirqd/108:40:011
2222099235,3cyclictest131rcu_preempt11:00:101
2222099234,9cyclictest21-21ksoftirqd/111:45:201
22220992317,4cyclictest201rcuc/112:36:071
22220992316,4cyclictest0-21swapper/108:05:131
2222099231,17cyclictest19201-21ssh10:10:151
22220992310,4cyclictest0-21swapper/110:06:421
2222099230,6cyclictest22019-21cat09:15:311
22238992213,7cyclictest0-21swapper/311:10:273
2223899221,13cyclictest35966irq/58-eth010:35:143
2222999229,4cyclictest22897-21kthreadcore12:35:202
2222999225,14cyclictest26798-21hwlatdetect12:20:182
2222999223,9cyclictest13971-21ntp_kernel_err08:05:242
2222999221,12cyclictest10279-21ssh11:15:262
2222099226,7cyclictest21-21ksoftirqd/110:55:351
2222099225,8cyclictest21-21ksoftirqd/109:44:421
2222099223,5cyclictest21-21ksoftirqd/111:50:171
2222099222,4cyclictest21-21ksoftirqd/109:10:031
22220992214,5cyclictest201rcuc/111:20:101
22220992210,4cyclictest21-21ksoftirqd/112:25:291
2223899214,9cyclictest0-21swapper/311:15:143
2223899213,9cyclictest35966irq/58-eth007:20:143
2223899213,12cyclictest0-21swapper/310:33:183
2223899213,11cyclictest35966irq/58-eth008:55:113
2223899213,10cyclictest35966irq/58-eth010:20:133
2223899211,14cyclictest35966irq/58-eth012:05:143
22238992111,4cyclictest0-21swapper/312:35:133
2222999212,4cyclictest0-21swapper/209:05:122
2222999212,16cyclictest11938-21fschecks_time08:40:162
2222999211,17cyclictest2347-21ls08:30:162
2222099216,7cyclictest21-21ksoftirqd/111:25:591
2222099216,7cyclictest21-21ksoftirqd/110:34:581
2222099215,5cyclictest21-21ksoftirqd/107:18:181
2222099214,3cyclictest131rcu_preempt07:35:191
2222099214,3cyclictest131rcu_preempt07:35:191
2222099213,3cyclictest131rcu_preempt12:02:081
2222099212,5cyclictest21-21ksoftirqd/107:50:241
2222099212,4cyclictest21-21ksoftirqd/108:55:131
22220992116,4cyclictest201rcuc/110:16:341
2223899209,6cyclictest0-21swapper/309:40:433
2223899204,12cyclictest0-21swapper/312:00:133
2223899203,11cyclictest0-21swapper/307:35:133
2223899203,11cyclictest0-21swapper/307:35:133
2223899202,11cyclictest0-21swapper/308:25:143
2222999201,9cyclictest25883-21munin-run08:55:012
2222999201,15cyclictest26685-21grep08:55:172
2222099207,5cyclictest21-21ksoftirqd/107:40:261
2222099207,5cyclictest21-21ksoftirqd/107:40:261
2222099207,10cyclictest201rcuc/108:00:221
2222099204,8cyclictest22196-21cyclictest09:00:121
2222099203,7cyclictest21-21ksoftirqd/108:15:241
2222099201,8cyclictest21-21ksoftirqd/107:20:141
2222099201,3cyclictest21-21ksoftirqd/108:50:251
22220992013,3cyclictest0-21swapper/107:25:131
2222099201,11cyclictest26038-21switchtime07:10:281
2222099200,14cyclictest24953-21find08:20:121
2223899193,11cyclictest0-21swapper/311:55:563
2223899192,10cyclictest0-21swapper/308:35:133
22238991913,4cyclictest33-21ksoftirqd/308:20:163
2223899191,11cyclictest35966irq/58-eth012:20:133
2223899191,11cyclictest0-21swapper/308:10:153
2223899191,11cyclictest0-21swapper/307:30:013
22238991910,6cyclictest35966irq/58-eth011:25:133
2222999191,15cyclictest17340-21kthreadcore08:10:202
2222099194,4cyclictest21-21ksoftirqd/107:55:201
2222099194,3cyclictest131rcu_preempt08:40:201
2222099193,4cyclictest21-21ksoftirqd/108:25:241
2222099190,6cyclictest21-21ksoftirqd/108:10:151
2223899185,6cyclictest0-21swapper/308:00:103
2223899182,11cyclictest0-21swapper/309:31:183
2223899182,10cyclictest0-21swapper/309:05:133
2223899182,10cyclictest0-21swapper/308:15:143
2223899182,10cyclictest0-21swapper/307:50:143
2223899182,10cyclictest0-21swapper/307:45:043
2223899182,10cyclictest0-21swapper/307:40:143
2223899182,10cyclictest0-21swapper/307:40:143
2223899182,10cyclictest0-21swapper/307:30:143
22238991812,4cyclictest0-21swapper/311:35:413
2223899181,12cyclictest0-21swapper/308:40:273
2223899181,10cyclictest0-21swapper/307:10:013
2222099186,3cyclictest131rcu_preempt07:30:141
2223899172,10cyclictest0-21swapper/308:50:153
2223899172,10cyclictest0-21swapper/307:55:143
2223899171,13cyclictest35966irq/58-eth007:15:133
2223899171,10cyclictest0-21swapper/308:30:163
2223899162,10cyclictest0-21swapper/307:10:053
2222999153,9cyclictest22128-21apt-get07:10:012
2222099151,10cyclictest22130-21latency_hist07:10:011
2223899141,10cyclictest0-21swapper/308:45:193
2223899122,6cyclictest0-21swapper/308:05:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional