You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 06:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat Jun 13, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5205997545,18cyclictest35966irq/58-eth019:55:133
5205997545,18cyclictest35966irq/58-eth019:55:123
518299583,39cyclictest8638-21ssh00:00:151
5195995431,17cyclictest0-21swapper/221:20:112
519599533,41cyclictest27564-21ssh21:30:122
518299526,36cyclictest29432-21ssh23:00:121
518299471,34cyclictest8388-21ssh21:55:131
518299422,31cyclictest17740-21ssh00:05:181
5195994126,12cyclictest631-21Async09:36:312
5195994126,12cyclictest631-21Async09:36:312
519599411,25cyclictest17478-21ssh22:00:122
518299412,35cyclictest12394-21ssh00:20:121
518299401,31cyclictest13879-21ssh23:45:111
5169994010,10cyclictest28316-21ssh21:46:400
519599391,2cyclictest0-21swapper/223:25:122
516999388,11cyclictest9174-21ssh22:11:460
5205993722,10cyclictest35966irq/58-eth022:15:113
518299372,30cyclictest22914-21ssh23:50:141
516999377,11cyclictest5126-21ssh21:51:440
516999377,10cyclictest18752-21ntp_192.168.11423:10:230
5169993720,12cyclictest11348-21ssh22:50:130
5169993719,13cyclictest4689-21ssh23:40:140
5169993710,10cyclictest14502-21ssh23:09:450
5169993710,10cyclictest14502-21ssh23:09:440
516999363,26cyclictest25509-21ssh22:40:130
5169993630,3cyclictest25793-21apt-get20:05:130
5169993627,5cyclictest7095-21apt-get00:35:130
520599354,21cyclictest35966irq/58-eth000:35:133
5182993517,13cyclictest5157-21cyclictest20:45:111
5169993520,11cyclictest9824-21apt-get23:25:120
5169993511,8cyclictest4212-21kworker/u8:022:47:050
5205993413,14cyclictest35966irq/58-eth021:35:113
5195993422,9cyclictest536-21usb-storage23:45:122
5182993416,11cyclictest26691-21apt-get00:10:121
516999349,10cyclictest3369-21ssh23:38:590
516999349,10cyclictest30061-21ssh23:16:440
516999348,11cyclictest30596-21ssh21:13:020
516999348,10cyclictest20778-21ssh22:19:340
5169993420,10cyclictest11411-21apt-get19:50:120
5169993417,12cyclictest26532-21ssh22:05:120
520599331,19cyclictest35966irq/58-eth022:30:103
520599331,19cyclictest35966irq/58-eth022:30:093
516999339,9cyclictest4289-21ssh23:03:200
516999339,14cyclictest6637-21sh23:58:110
516999339,10cyclictest26539-21ssh00:25:460
516999338,11cyclictest19894-21ssh21:42:270
516999338,11cyclictest19894-21ssh21:42:260
516999337,10cyclictest2020-21tr21:15:200
516999335,8cyclictest25194-21ntp_192.168.11422:20:240
5169993326,4cyclictest9369-21apt-get21:20:110
5169993326,3cyclictest16192-21apt-get19:55:140
5169993326,3cyclictest16192-21apt-get19:55:140
516999331,26cyclictest22904-21ssh23:50:130
5169993310,8cyclictest2089-21ssh21:32:590
5169993310,10cyclictest13481-21ssh21:55:490
519599324,17cyclictest27346irq/40-dwc2_hso20:55:132
5182993222,5cyclictest23255-21diskmemload00:25:141
518299321,28cyclictest3246-21ssh22:10:141
516999328,8cyclictest8931-21ssh00:16:240
516999328,11cyclictest13777-21ssh00:00:550
516999328,10cyclictest32624-21ssh00:11:440
516999325,11cyclictest24111-21ssh22:38:510
516999324,7cyclictest22212-21/usr/sbin/munin20:35:310
5169993210,9cyclictest19099-21ssh23:45:560
520599312,15cyclictest35966irq/58-eth023:30:123
519599313,21cyclictest0-21swapper/219:25:142
5182993121,3cyclictest15132-21ssh23:10:131
518299311,23cyclictest21586-21ssh22:20:141
516999316,9cyclictest22081-21ntp_192.168.11421:05:250
516999315,10cyclictest1805-21df_inode19:40:130
5169993124,3cyclictest5315-21apt-get19:10:120
5169993123,4cyclictest30603-21apt-get20:10:110
520599305,21cyclictest35966irq/58-eth021:55:153
5205993019,6cyclictest35966irq/58-eth022:45:133
519599303,18cyclictest27346irq/40-dwc2_hso20:30:132
5182993021,3cyclictest0-21swapper/122:50:111
518299301,25cyclictest0-21swapper/100:30:131
516999307,8cyclictest19983-21tr00:05:260
516999307,6cyclictest7260-21apt-get22:30:110
516999307,6cyclictest7260-21apt-get22:30:100
516999306,8cyclictest25691-21ssh23:32:440
516999306,7cyclictest30786-21diskstats22:25:160
516999306,10cyclictest17669-21idleruntime20:30:160
516999305,8cyclictest17277-21ntp_192.168.11421:00:230
5169993025,2cyclictest27038-21apt-get20:40:120
5169993010,10cyclictest16015-21sh00:20:250
519599293,18cyclictest27346irq/40-dwc2_hso00:10:122
519599291,19cyclictest22361-21cut20:35:152
5195992911,12cyclictest27-21ksoftirqd/200:35:112
516999298,7cyclictest23255-21diskmemload21:38:240
516999297,8cyclictest17407-21ssh22:00:080
516999296,8cyclictest22942-21cat21:25:280
5169992919,7cyclictest20413-21sh22:55:130
520599281,19cyclictest35966irq/58-eth000:15:143
519599285,15cyclictest27346irq/40-dwc2_hso19:10:102
519599283,19cyclictest27346irq/40-dwc2_hso22:05:112
519599282,20cyclictest0-21swapper/221:05:152
519599282,10cyclictest0-21swapper/220:40:132
5195992815,8cyclictest2746ksoftirqd/219:30:132
5182992814,9cyclictest16379-21ssh22:35:111
5182992810,10cyclictest20135-21kthreadcore21:25:171
516999286,9cyclictest30363-21ssh00:30:030
520599277,10cyclictest0-21swapper/322:00:113
520599272,13cyclictest35966irq/58-eth022:40:133
519599273,15cyclictest27346irq/40-dwc2_hso21:35:122
519599272,19cyclictest27346irq/40-dwc2_hso23:35:132
518299271,19cyclictest805-21ssh23:20:121
516999276,8cyclictest4364-21ssh23:20:220
520599265,16cyclictest0-21swapper/300:00:163
520599264,16cyclictest0-21swapper/322:05:123
520599262,18cyclictest35966irq/58-eth020:45:123
519599265,12cyclictest27346irq/40-dwc2_hso19:35:122
519599263,18cyclictest0-21swapper/222:15:042
519599263,17cyclictest0-21swapper/200:15:142
519599262,12cyclictest23012-21ssh23:50:142
519599261,16cyclictest13875-21find21:00:112
519599261,14cyclictest18960-21diskmemload23:30:122
518299268,12cyclictest31614-21apt-get21:50:141
518299264,3cyclictest131rcu_preempt23:29:501
518299263,5cyclictest21-21ksoftirqd/123:18:471
518299263,5cyclictest21-21ksoftirqd/121:43:511
518299263,5cyclictest21-21ksoftirqd/121:43:511
518299261,19cyclictest31263-21ssh21:30:261
5182992611,4cyclictest21-21ksoftirqd/122:29:541
516999267,14cyclictest0-21swapper/020:15:120
516999265,7cyclictest24605-21awk19:30:140
5169992615,8cyclictest12613-21apt-get20:25:110
520599254,17cyclictest35966irq/58-eth000:30:133
520599252,14cyclictest0-21swapper/322:25:143
519599251,15cyclictest20004-21ssh22:35:222
5195992510,10cyclictest0-21swapper/221:45:122
518299257,8cyclictest21-21ksoftirqd/120:30:251
518299254,4cyclictest21-21ksoftirqd/119:30:141
518299254,3cyclictest131rcu_preempt19:40:241
518299253,4cyclictest21-21ksoftirqd/122:03:401
518299253,4cyclictest21-21ksoftirqd/121:45:241
518299252,18cyclictest7286-21ssh22:30:111
518299252,18cyclictest7286-21ssh22:30:101
518299251,9cyclictest21-21ksoftirqd/123:58:491
518299251,20cyclictest27984-21apt-get23:35:111
518299251,18cyclictest25505-21ssh22:40:131
5169992517,4cyclictest4259-21apt-get20:50:100
520599247,13cyclictest0-21swapper/300:05:183
520599244,16cyclictest35966irq/58-eth022:10:123
520599243,17cyclictest33-21ksoftirqd/322:35:103
520599243,13cyclictest0-21swapper/321:50:133
5205992415,4cyclictest0-21swapper/322:25:013
5205992411,8cyclictest35966irq/58-eth021:45:133
5195992418,3cyclictest27-21ksoftirqd/220:05:132
518299245,5cyclictest21-21ksoftirqd/121:20:311
518299245,5cyclictest21-21ksoftirqd/121:15:191
518299244,5cyclictest21-21ksoftirqd/123:08:091
518299244,5cyclictest21-21ksoftirqd/123:08:081
518299244,4cyclictest21-21ksoftirqd/123:30:271
518299243,5cyclictest21-21ksoftirqd/121:37:211
518299243,5cyclictest21-21ksoftirqd/100:17:091
518299243,5cyclictest20392-21apt-get22:55:121
518299243,4cyclictest21-21ksoftirqd/123:40:591
518299243,4cyclictest21-21ksoftirqd/119:20:301
5182992418,4cyclictest201rcuc/100:36:171
5182992417,4cyclictest18672-21apt-get21:05:121
516999243,9cyclictest31828-21/usr/sbin/munin20:45:260
5169992411,10cyclictest0-21swapper/019:45:120
5169992411,10cyclictest0-21swapper/019:45:110
520599234,11cyclictest35966irq/58-eth022:55:133
5205992315,5cyclictest0-21swapper/323:05:143
5205992315,5cyclictest0-21swapper/323:05:133
5205992310,9cyclictest0-21swapper/323:15:273
519599233,16cyclictest27346irq/40-dwc2_hso22:25:122
519599233,15cyclictest2206-21ssh22:45:142
519599233,14cyclictest27346irq/40-dwc2_hso21:40:142
519599233,14cyclictest27346irq/40-dwc2_hso21:40:132
519599232,12cyclictest31632-21ssh21:50:142
519599231,5cyclictest2944-21cut23:55:232
519599231,11cyclictest19718-21kthreadcore00:05:252
518299235,6cyclictest21-21ksoftirqd/120:00:261
518299235,2cyclictest131rcu_preempt21:10:351
518299233,4cyclictest21-21ksoftirqd/122:15:591
518299232,4cyclictest21-21ksoftirqd/120:25:261
518299231,6cyclictest5801-21cut19:10:171
518299231,6cyclictest2555-21ssh22:45:171
516999239,10cyclictest0-21swapper/020:20:100
5169992311,9cyclictest13979-21unixbench_singl19:15:280
520599223,15cyclictest35966irq/58-eth023:20:113
520599221,14cyclictest0-21swapper/323:55:133
519599229,11cyclictest28134-21ssh21:10:302
519599229,10cyclictest415-21latency23:00:212
519599228,11cyclictest15102-21apt-get23:10:112
519599222,3cyclictest0-21swapper/219:50:112
519599222,11cyclictest6828-21ps19:45:182
519599222,11cyclictest6828-21ps19:45:172
519599222,11cyclictest10860-21rm00:00:232
5195992211,5cyclictest8413-21ssh23:23:472
519599221,12cyclictest7259-21ssh22:30:102
519599221,12cyclictest7259-21ssh22:30:092
519599221,11cyclictest20418-21sh22:55:142
5182992217,3cyclictest201rcuc/122:07:291
5182992216,3cyclictest2988-21apt-get20:15:111
518299221,4cyclictest12471-21ntp_192.168.11420:55:231
5169992216,3cyclictest14920-21apt-get19:20:130
5169992213,7cyclictest29389-21cpuspeed19:35:130
5169992212,7cyclictest23032-21memory19:25:220
5169992212,7cyclictest0-21swapper/020:55:130
5205992113,5cyclictest0-21swapper/323:10:143
5205992110,9cyclictest0-21swapper/300:20:133
519599213,10cyclictest1017-21kthreadcore21:15:212
519599212,9cyclictest8140-21grep20:50:262
519599212,17cyclictest16516-21sed00:20:302
5195992113,5cyclictest26231-21ssh22:40:172
519599211,12cyclictest10846-21ssh21:55:232
519599211,11cyclictest6532-21ssh23:05:192
519599211,11cyclictest6532-21ssh23:05:192
519599211,11cyclictest18788-21/usr/sbin/munin21:25:152
5195992110,9cyclictest22500-21ssh00:25:192
5195992110,9cyclictest19608-21ssh22:54:312
519599210,12cyclictest1704-21ntp_192.168.11400:30:252
518299214,4cyclictest21-21ksoftirqd/120:10:211
518299213,5cyclictest21-21ksoftirqd/119:50:251
518299213,4cyclictest21-21ksoftirqd/120:20:121
518299212,4cyclictest21-21ksoftirqd/120:55:011
518299212,4cyclictest21-21ksoftirqd/120:05:161
518299212,4cyclictest21-21ksoftirqd/119:25:321
518299211,5cyclictest21-21ksoftirqd/121:00:231
518299211,4cyclictest21-21ksoftirqd/119:15:131
5182992112,6cyclictest0-21swapper/119:55:241
5182992112,6cyclictest0-21swapper/119:55:231
516999218,9cyclictest21007-21acpi20:00:110
520599209,8cyclictest35966irq/58-eth000:10:133
520599203,10cyclictest0-21swapper/323:45:113
520599202,14cyclictest0-21swapper/321:30:133
5205992012,5cyclictest0-21swapper/323:00:143
519599207,10cyclictest15013-21diskstats19:20:132
519599204,13cyclictest4026-21cron20:50:012
519599202,9cyclictest3513-21hwlatdetect22:10:172
519599201,9cyclictest8319-21irqrtprio20:20:152
519599201,8cyclictest2168066ssh22:20:152
519599201,3cyclictest0-21swapper/220:25:112
519599201,16cyclictest30740-21fschecks_count20:10:142
519599201,11cyclictest4950-21cut23:40:162
519599201,10cyclictest3082-21diskstats20:15:142
519599201,10cyclictest2905-21kthreadcore19:40:182
519599201,10cyclictest11981-21kthreadcore19:15:202
518299203,5cyclictest21-21ksoftirqd/120:35:251
5182992010,5cyclictest6842-21grep19:45:181
5182992010,5cyclictest6842-21grep19:45:171
520599198,6cyclictest0-21swapper/321:15:503
520599192,13cyclictest35966irq/58-eth000:25:133
520599192,11cyclictest0-21swapper/321:40:143
520599192,11cyclictest0-21swapper/321:40:133
520599192,11cyclictest0-21swapper/320:30:143
520599192,10cyclictest0-21swapper/321:10:133
520599191,13cyclictest0-21swapper/319:15:143
520599191,12cyclictest0-21swapper/319:30:013
520599191,12cyclictest0-21swapper/319:10:153
520599191,11cyclictest0-21swapper/319:50:153
5205991910,6cyclictest0-21swapper/323:26:213
5205991910,6cyclictest0-21swapper/321:25:163
519599192,15cyclictest23471-21kthreadcore20:00:212
519599191,9cyclictest24468-21sed23:15:162
519599191,10cyclictest5263-21latency_hist19:10:022
518299194,5cyclictest21-21ksoftirqd/119:35:191
516999199,8cyclictest5217-21date19:10:010
520599189,6cyclictest0-21swapper/323:40:483
520599189,5cyclictest35966irq/58-eth022:50:123
520599182,11cyclictest0-21swapper/320:05:143
520599182,10cyclictest0-21swapper/321:20:123
520599182,10cyclictest0-21swapper/320:25:113
520599182,10cyclictest0-21swapper/319:20:133
5205991812,4cyclictest0-21swapper/323:35:233
5205991811,4cyclictest0-21swapper/323:53:073
518299186,9cyclictest29829-21kthreadcore20:40:211
520599172,10cyclictest0-21swapper/320:55:133
520599171,8cyclictest0-21swapper/321:05:123
520599171,12cyclictest0-21swapper/320:20:153
520599171,10cyclictest0-21swapper/320:10:133
520599171,10cyclictest0-21swapper/319:40:213
520599171,10cyclictest0-21swapper/319:35:273
520599161,10cyclictest0-21swapper/320:50:123
520599161,10cyclictest0-21swapper/319:30:133
520599152,9cyclictest0-21swapper/320:40:143
520599151,11cyclictest0-21swapper/319:45:273
520599151,11cyclictest0-21swapper/319:45:273
520599149,3cyclictest0-21swapper/320:18:263
520599141,9cyclictest0-21swapper/320:35:123
520599138,3cyclictest0-21swapper/321:00:423
518299131,9cyclictest5225-21latency_hist19:10:011
520599122,6cyclictest0-21swapper/320:00:123
520599101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional