You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-24 - 07:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Apr 24, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32260998444,27cyclictest35966irq/58-eth023:40:123
3223699654,42cyclictest31251-21ssh21:35:131
3223699581,43cyclictest17040-21ssh21:45:131
3222599582,37cyclictest20852-21ssh22:05:120
3222599582,37cyclictest20852-21ssh22:05:120
3223699573,9cyclictest131rcu_preempt23:30:121
3223699541,42cyclictest15154-21ssh00:25:141
3222599531,42cyclictest4039-21ssh23:25:130
3222599531,38cyclictest7729-21ssh23:45:110
3222599531,38cyclictest7729-21ssh23:45:100
32236995140,8cyclictest21-21ksoftirqd/121:20:121
3224999501,33cyclictest23626-21ssh23:00:122
3223699502,34cyclictest2775-21ssh21:55:151
32225995021,20cyclictest19676-21ssh22:40:130
32260994926,15cyclictest35966irq/58-eth022:50:113
3226099488,25cyclictest35966irq/58-eth022:40:133
3224999486,29cyclictest17757-21diskmemload23:15:122
32236994720,22cyclictest21-21ksoftirqd/121:30:131
32225994733,8cyclictest0-21swapper/000:10:130
3224999425,17cyclictest16788-21sh23:50:132
32225994235,4cyclictest9540-21apt-get19:20:100
3226099415,24cyclictest35966irq/58-eth023:00:123
3226099412,26cyclictest35966irq/58-eth022:15:153
3223699402,6cyclictest131rcu_preempt21:40:151
32236994016,20cyclictest21-21ksoftirqd/123:15:121
32225994032,4cyclictest19129-21apt-get19:30:110
32225994013,11cyclictest10980-21ssh21:23:010
3226099392,20cyclictest35966irq/58-eth021:45:123
3224999382,9cyclictest271ksoftirqd/222:35:142
32236993819,8cyclictest21-21ksoftirqd/119:20:111
32225993831,3cyclictest6710-21apt-get22:15:150
3226099379,19cyclictest0-21swapper/321:35:143
3226099377,20cyclictest35966irq/58-eth000:20:133
3226099377,20cyclictest35966irq/58-eth000:20:123
3222599379,12cyclictest1569-21ssh23:59:320
3222599378,11cyclictest30826-21ntp_192.168.11421:15:260
3222599378,10cyclictest29962-21/usr/sbin/munin21:50:290
32225993727,6cyclictest14320-21apt-get19:25:130
32225993712,8cyclictest10649-21ssh00:04:360
3222599371,26cyclictest5321-21ssh22:50:110
32225993711,8cyclictest25956-21ps23:35:270
32225993711,11cyclictest3843-21ssh22:12:150
32225993711,10cyclictest14508-21ssh21:42:280
32225993710,11cyclictest18811-21ssh22:39:310
3226099363,24cyclictest35966irq/58-eth022:25:123
3222599367,9cyclictest18988-21ssh23:31:410
32225993627,6cyclictest31245-21apt-get21:35:140
3222599361,24cyclictest13063-21rm21:25:120
32225993611,8cyclictest12772-21ssh00:22:440
32225993611,8cyclictest12772-21ssh00:22:430
32225993610,9cyclictest29455-21ssh23:01:430
32225993610,9cyclictest27355-21ssh21:30:550
32225993610,10cyclictest19395-21/usr/sbin/munin22:20:250
32225993610,10cyclictest19395-21/usr/sbin/munin22:20:250
3223699351,28cyclictest0-21swapper/123:20:121
3222599358,10cyclictest10983-21grep19:55:150
3222599357,8cyclictest32257-21ntp_states19:40:240
32225993527,4cyclictest8370-21apt-get21:00:130
32225993512,7cyclictest18129-21ssh00:07:430
32225993511,14cyclictest24053-21/usr/sbin/munin20:05:270
3224999342,27cyclictest27203-21ssh21:15:132
3224999341,5cyclictest131rcu_preempt20:20:142
32249993410,19cyclictest3368-21ssh00:17:232
32236993423,4cyclictest32729-21ssh23:05:111
3222599349,8cyclictest22772-21ssh21:46:320
3222599349,8cyclictest19246-21ssh22:03:320
32225993427,4cyclictest29993-21apt-get20:15:130
32225993426,4cyclictest18134-21apt-get21:10:140
32225993412,8cyclictest1116-21ssh23:22:020
32225993411,8cyclictest31353-21grep23:40:160
32225993411,8cyclictest23747-21cut20:40:200
3222599339,9cyclictest21350-21latency_hist20:40:000
32225993322,7cyclictest18365-21apt-get23:15:120
32225993312,7cyclictest0-21swapper/000:16:290
32225993311,8cyclictest30112-21ssh00:31:470
32225993311,7cyclictest6575-21ssh23:07:120
3226099322,18cyclictest35966irq/58-eth021:20:113
3223699322,22cyclictest27715-21ssh00:30:241
3223699322,21cyclictest6038-21ssh00:20:141
3223699322,21cyclictest6038-21ssh00:20:141
3222599328,10cyclictest19524-21cat00:25:300
3222599323,20cyclictest0-21swapper/022:25:130
32225993211,7cyclictest17757-21diskmemload23:53:060
3226099312,18cyclictest35966irq/58-eth019:25:133
3226099311,19cyclictest35966irq/58-eth023:15:113
3224999312,14cyclictest261rcuc/223:25:132
32249993112,15cyclictest261rcuc/223:40:132
3223699312,21cyclictest18148-21ssh21:10:131
3222599313,22cyclictest14454-21ssh22:55:150
32225993122,5cyclictest1172-21apt-get19:45:130
3226099302,17cyclictest35966irq/58-eth022:05:123
3226099302,17cyclictest35966irq/58-eth022:05:113
32260993012,11cyclictest35966irq/58-eth022:45:123
3224999304,15cyclictest27346irq/40-dwc2_hso22:00:122
3224999302,21cyclictest0-21swapper/219:10:132
3222599309,13cyclictest0-21swapper/000:38:430
32225993023,4cyclictest31143-21apt-get20:50:130
32225993019,6cyclictest25280-21diskstats20:10:160
3226099292,16cyclictest35966irq/58-eth020:20:143
3224999293,16cyclictest27346irq/40-dwc2_hso22:45:142
3223699293,17cyclictest630-21Async09:36:311
3223699293,15cyclictest6721-21ssh22:15:161
3222599296,7cyclictest0-21swapper/022:46:220
3224999284,14cyclictest27346irq/40-dwc2_hso23:35:142
3224999282,20cyclictest0-21swapper/221:30:152
3224999282,19cyclictest0-21swapper/219:45:132
3224999282,17cyclictest27346irq/40-dwc2_hso22:05:132
3224999282,17cyclictest27346irq/40-dwc2_hso22:05:122
32225992818,7cyclictest24063-21df_inode19:35:150
32225992815,10cyclictest0-21swapper/022:30:110
32225992814,11cyclictest32473-21entropy19:10:140
32225992814,10cyclictest0-21swapper/021:55:160
32225992811,12cyclictest4709-21apt-get19:15:120
3226099274,16cyclictest0-21swapper/323:10:143
3224999275,13cyclictest27346irq/40-dwc2_hso00:10:132
3224999273,17cyclictest27346irq/40-dwc2_hso23:10:132
3224999273,17cyclictest27346irq/40-dwc2_hso20:25:152
3224999273,17cyclictest0-21swapper/200:35:222
3224999272,19cyclictest0-21swapper/220:05:132
3224999272,18cyclictest27346irq/40-dwc2_hso19:50:112
32249992715,8cyclictest630-21Async09:36:312
3224999271,23cyclictest1563-21ssh22:30:132
3223699272,20cyclictest2421-21ssh00:00:151
32225992721,3cyclictest9344-21apt-get23:10:150
32225992717,6cyclictest15701-21df20:00:140
32225992715,9cyclictest0-21swapper/020:45:120
32225992713,9cyclictest4414-21kthreadcore20:55:180
3226099263,16cyclictest35966irq/58-eth023:20:123
3224999269,14cyclictest1-21systemd00:23:042
3224999269,14cyclictest1-21systemd00:23:032
3224999261,14cyclictest24915-21ssh22:25:142
32225992615,8cyclictest9295-21/usr/sbin/munin19:50:220
32225992615,8cyclictest11028-21head20:25:280
3226099259,9cyclictest35966irq/58-eth021:15:263
3226099259,10cyclictest35966irq/58-eth000:17:223
3226099254,14cyclictest35966irq/58-eth020:10:133
3224999258,10cyclictest27-21ksoftirqd/221:20:272
3224999255,13cyclictest27346irq/40-dwc2_hso21:10:132
3224999252,4cyclictest131rcu_preempt19:25:132
3224999252,17cyclictest27346irq/40-dwc2_hso21:35:122
3224999252,17cyclictest27346irq/40-dwc2_hso19:20:092
32249992513,9cyclictest27-21ksoftirqd/221:25:312
32249992513,9cyclictest27-21ksoftirqd/220:05:012
3222599258,13cyclictest14361-21tr21:05:190
3224999248,3cyclictest27-21ksoftirqd/223:20:562
3224999244,13cyclictest27346irq/40-dwc2_hso21:45:132
3224999243,17cyclictest27346irq/40-dwc2_hso20:55:122
3224999242,15cyclictest27346irq/40-dwc2_hso19:40:122
3223699242,7cyclictest131rcu_preempt00:38:221
3223699242,17cyclictest3540-21apt-get20:55:121
32225992411,9cyclictest16527-21idleruntime-cro20:35:010
32225992410,11cyclictest32213-21cyclictest19:10:020
3224999239,8cyclictest27-21ksoftirqd/200:30:172
3224999239,11cyclictest15688-21latency_hist22:20:022
3224999233,12cyclictest27346irq/40-dwc2_hso20:50:122
32249992313,8cyclictest0-21swapper/200:25:232
3224999231,11cyclictest13104-21sh23:30:142
32249992310,11cyclictest10707-21ssh21:59:132
3223699234,16cyclictest10605-21ssh22:35:141
32225992313,8cyclictest0-21swapper/020:20:130
32260992210,9cyclictest0-21swapper/323:57:323
3224999229,6cyclictest26416-21irqrtprio23:55:192
3224999229,10cyclictest3383-21kworker/u8:022:20:312
3224999229,10cyclictest3383-21kworker/u8:022:20:302
3224999228,7cyclictest27-21ksoftirqd/222:55:222
3224999223,12cyclictest27346irq/40-dwc2_hso20:45:132
3224999222,15cyclictest0-21swapper/222:50:112
3224999222,14cyclictest27346irq/40-dwc2_hso19:30:112
32249992211,9cyclictest19554-21ssh22:40:062
3223699222,16cyclictest15616-21apt-get20:00:111
3223699221,18cyclictest30879-21ls23:40:021
3223699221,18cyclictest11956-21cut22:00:161
3223699221,16cyclictest16795-21sh23:50:141
3223699221,10cyclictest0-21swapper/122:28:541
32236992210,3cyclictest131rcu_preempt19:45:021
3223699220,13cyclictest28996-21kthreadcore20:45:221
3226099219,9cyclictest0-21swapper/300:30:203
32260992115,4cyclictest0-21swapper/321:05:133
32260992112,6cyclictest0-21swapper/323:50:223
32260992111,7cyclictest0-21swapper/300:10:213
3224999218,10cyclictest6592-21usage_Anybus00:00:322
3224999217,11cyclictest23580-21kthreadcore20:40:202
3224999214,13cyclictest261rcuc/221:05:122
3224999212,15cyclictest885-21wc20:15:232
3224999212,13cyclictest630-21Async09:36:312
32249992111,8cyclictest14644-21ssh21:42:382
3224999211,11cyclictest15817-21head20:30:272
3224999211,11cyclictest13784-21tr00:05:222
3224999211,11cyclictest11676-21kthreadcore19:55:202
3224999210,18cyclictest26400-21hwlatdetect21:50:182
3224999210,17cyclictest27346irq/40-dwc2_hso23:45:112
3224999210,17cyclictest27346irq/40-dwc2_hso23:45:112
3223699212,3cyclictest131rcu_preempt21:50:161
3223699211,16cyclictest7578-21ssh23:25:241
3223699211,16cyclictest1761-21cut22:30:161
3223699211,15cyclictest11398-21sh00:05:131
3223699211,10cyclictest0-21swapper/120:40:191
3226099209,6cyclictest0-21swapper/321:59:103
3226099203,12cyclictest0-21swapper/300:35:233
3226099202,14cyclictest0-21swapper/323:05:133
32260992014,4cyclictest0-21swapper/323:38:573
32260992014,4cyclictest0-21swapper/321:50:183
32260992013,4cyclictest0-21swapper/322:56:403
32260992013,4cyclictest0-21swapper/319:55:283
32260992012,6cyclictest0-21swapper/321:30:143
32260992012,5cyclictest0-21swapper/322:35:133
32260992010,6cyclictest35966irq/58-eth022:21:133
32260992010,6cyclictest35966irq/58-eth022:21:123
3224999202,15cyclictest24080-21grep19:35:162
32249992014,4cyclictest0-21swapper/222:10:282
3224999201,11cyclictest8692-21grep21:00:182
3224999201,10cyclictest6339-21kthreadcore19:15:202
3223699208,10cyclictest30318-21cat23:55:301
3223699207,8cyclictest32213-21cyclictest20:50:121
3223699207,7cyclictest21-21ksoftirqd/123:40:111
3223699207,5cyclictest7647-21irqcore20:25:191
3223699206,8cyclictest211ksoftirqd/122:20:131
3223699206,8cyclictest211ksoftirqd/122:20:121
32236992013,5cyclictest0-21swapper/119:10:181
32236992011,6cyclictest0-21swapper/119:25:121
3223699200,17cyclictest19814-21cut22:40:161
3223699200,17cyclictest12075-21timerandwakeup21:00:271
3226099193,11cyclictest0-21swapper/300:28:053
3226099192,11cyclictest0-21swapper/320:00:133
32260991913,4cyclictest321rcuc/320:30:133
32260991913,4cyclictest0-21swapper/323:30:163
32260991913,4cyclictest0-21swapper/322:10:223
32260991913,4cyclictest0-21swapper/319:15:013
32260991913,4cyclictest0-21swapper/300:05:113
32260991911,6cyclictest0-21swapper/319:20:263
32260991911,5cyclictest0-21swapper/323:46:243
32260991911,5cyclictest0-21swapper/323:46:233
32260991910,7cyclictest0-21swapper/320:35:153
3224999191,15cyclictest16924-21fschecks_time20:35:152
3223699199,7cyclictest24542-21ntpq22:05:251
3223699199,7cyclictest24542-21ntpq22:05:241
3223699191,15cyclictest27866-21unixbench_singl19:35:281
32236991911,5cyclictest0-21swapper/119:15:171
3226099187,6cyclictest0-21swapper/322:01:023
3226099184,9cyclictest321rcuc/319:30:103
3226099183,7cyclictest321rcuc/319:45:113
3226099181,7cyclictest35966irq/58-eth021:00:123
32260991812,4cyclictest0-21swapper/323:25:163
32260991812,4cyclictest0-21swapper/322:30:023
32260991812,4cyclictest0-21swapper/321:41:203
32260991812,4cyclictest0-21swapper/321:10:183
32260991812,4cyclictest0-21swapper/320:05:153
32260991812,4cyclictest0-21swapper/300:00:163
3224999188,8cyclictest32311-21latency_hist19:10:012
3223699183,10cyclictest0-21swapper/100:14:241
3223699182,13cyclictest1237-21ntp_states22:10:271
3223699181,9cyclictest10559-21latency_hist19:55:031
3223699181,5cyclictest17753-21munin-run21:10:021
32236991811,4cyclictest29015-21if_err_eth022:45:171
32236991811,4cyclictest14399-21/usr/sbin/munin22:55:181
3223699181,13cyclictest7292-21apt-get23:45:051
3223699181,13cyclictest7292-21apt-get23:45:041
3223699181,13cyclictest13062-21sh21:25:131
3223699181,10cyclictest24340-21kernelversion23:00:191
3223699180,12cyclictest1400-21idleruntime19:45:151
3226099179,6cyclictest0-21swapper/321:25:093
3226099172,10cyclictest0-21swapper/320:50:143
3226099172,10cyclictest0-21swapper/320:45:133
32260991711,4cyclictest0-21swapper/319:45:023
32260991711,4cyclictest0-21swapper/319:35:183
3223699179,6cyclictest1-21systemd20:32:121
3223699178,6cyclictest31133-21cut00:15:181
3223699174,9cyclictest32213-21cyclictest20:15:131
3223699173,10cyclictest23658-21latency_hist19:35:021
3223699171,13cyclictest9361-21ssh23:10:141
3226099162,10cyclictest0-21swapper/320:55:133
32260991610,4cyclictest0-21swapper/320:40:163
32260991610,4cyclictest0-21swapper/320:25:263
32260991610,4cyclictest0-21swapper/320:15:283
32260991610,4cyclictest0-21swapper/319:15:153
3223699169,5cyclictest9000-21sed22:50:241
3223699167,6cyclictest21-21ksoftirqd/120:10:151
3223699164,10cyclictest13823-21kthreadcore19:55:231
3223699161,8cyclictest5677-21wc20:20:231
3223699161,12cyclictest22109-21kthreadcore20:05:201
3226099152,10cyclictest0-21swapper/319:50:143
3223699152,10cyclictest32282-21latency_hist19:10:021
3223699151,11cyclictest16854-21fschecks_count20:35:161
322609991,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional