You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-13 - 04:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Aug 13, 2026 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1860999675,7cyclictest12-21ksoftirqd/021:40:120
18629996139,12cyclictest27-21ksoftirqd/223:35:122
1862999552,34cyclictest6981-21ssh22:05:122
18642995332,16cyclictest35966irq/58-eth023:55:133
1864299531,23cyclictest35966irq/58-eth019:50:123
18642995238,10cyclictest35966irq/58-eth000:30:143
18629995224,17cyclictest0-21swapper/222:00:142
1864299515,29cyclictest35966irq/58-eth023:10:123
1862999491,25cyclictest32580-21ssh22:55:122
18629994818,6cyclictest131rcu_preempt20:15:142
1862999471,34cyclictest0-21swapper/219:30:122
1861899473,38cyclictest4502-21ssh21:10:121
1864299465,30cyclictest35966irq/58-eth021:10:133
1862999461,32cyclictest31208-21ssh23:30:142
18609994425,10cyclictest0-21swapper/021:55:140
1860999433,29cyclictest1728-21ssh22:20:130
1861899411,31cyclictest14529-21ssh22:45:111
1864299408,29cyclictest35966irq/58-eth022:55:113
1864299403,21cyclictest35966irq/58-eth021:50:113
18642994018,16cyclictest35966irq/58-eth021:30:143
1864299394,25cyclictest35966irq/58-eth023:40:123
1864299393,23cyclictest35966irq/58-eth022:45:143
1862999391,19cyclictest25826-21ssh23:45:122
1862999390,27cyclictest26404-21ssh21:40:132
1860999392,26cyclictest6155-21sh00:10:130
1864299384,21cyclictest35966irq/58-eth021:55:143
1864299383,19cyclictest35966irq/58-eth000:20:133
18629993829,4cyclictest3158-21kworker/u8:121:15:122
1862999382,29cyclictest27104-21rm00:39:342
1862999382,29cyclictest27104-21rm00:39:332
1862999382,29cyclictest18867-21ssh23:58:312
1862999382,21cyclictest22112-21ssh23:25:132
1860999383,27cyclictest8310-21ssh21:30:120
1862999376,17cyclictest27-21ksoftirqd/220:05:132
1864299362,19cyclictest35966irq/58-eth023:50:123
1862999363,23cyclictest6158-21ssh00:10:142
18629993617,13cyclictest0-21swapper/223:15:112
1862999351,27cyclictest9550-21ssh23:53:152
1864299342,17cyclictest35966irq/58-eth019:45:153
1864299332,18cyclictest35966irq/58-eth023:15:113
1860999332,23cyclictest16307-21users23:55:310
1862999325,17cyclictest27346irq/40-dwc2_hso20:30:152
1862999324,17cyclictest27-21ksoftirqd/200:15:142
1861899327,20cyclictest27312-21ssh23:10:111
1861899322,22cyclictest27822-21ntp_192.168.11400:20:251
18618993218,11cyclictest2405-21ssh23:50:101
18642993118,7cyclictest33-21ksoftirqd/322:30:113
1862999314,17cyclictest25067-21ssh22:15:102
1862999312,16cyclictest27346irq/40-dwc2_hso21:05:112
1860999319,11cyclictest12-21ksoftirqd/000:35:220
1860999319,11cyclictest12-21ksoftirqd/000:35:210
1860999314,17cyclictest19810-21ssh22:30:120
18609993112,9cyclictest12-21ksoftirqd/023:15:110
1864299308,13cyclictest35966irq/58-eth021:45:123
1864299304,15cyclictest35966irq/58-eth022:05:143
1861899303,11cyclictest5951-21ssh00:25:481
1861899301,11cyclictest25326-21ssh21:39:141
1860999302,19cyclictest6997-21sh22:05:140
1860999301,21cyclictest16041-21ssh22:10:110
18642992913,9cyclictest0-21swapper/320:15:123
1862999292,16cyclictest27346irq/40-dwc2_hso20:45:122
1862999291,18cyclictest9897-21ssh00:30:152
1861899294,4cyclictest21-21ksoftirqd/122:10:281
1861899293,6cyclictest32619-21kthreadcore00:05:271
1861899293,22cyclictest22097-21ssh23:25:131
18618992916,7cyclictest28838-21apt-get22:35:111
1860999299,8cyclictest12-21ksoftirqd/019:35:000
1860999291,3cyclictest131rcu_preempt19:15:120
18609992910,10cyclictest0-21swapper/023:05:020
1862999282,21cyclictest31725-21ssh00:23:422
1861899285,19cyclictest13010-21ssh23:20:131
1861899284,6cyclictest32647-21awk22:55:121
1861899283,22cyclictest12084-21ssh21:50:111
1861899282,7cyclictest21-21ksoftirqd/121:45:011
1861899281,8cyclictest23677-21diskstats22:50:151
1860999283,4cyclictest12-21ksoftirqd/023:35:120
18609992811,10cyclictest12-21ksoftirqd/021:10:010
1862999273,9cyclictest0-21swapper/220:55:142
1862999273,17cyclictest27346irq/40-dwc2_hso19:15:122
1862999272,21cyclictest0-21swapper/221:35:142
1862999272,12cyclictest8338-21diskmemload21:30:152
1862999272,10cyclictest630-21Async09:36:312
1861899277,5cyclictest21-21ksoftirqd/123:58:571
1861899276,5cyclictest21-21ksoftirqd/121:30:301
1861899274,6cyclictest25395-21cat00:00:321
1861899274,5cyclictest131rcu_preempt20:45:121
1861899272,5cyclictest21-21ksoftirqd/123:15:121
18618992719,5cyclictest1700-21apt-get22:20:131
1861899270,10cyclictest16709-21ssh23:40:041
1860999279,8cyclictest12-21ksoftirqd/000:26:040
1860999278,7cyclictest12-21ksoftirqd/023:25:470
18609992713,4cyclictest12-21ksoftirqd/023:50:010
18609992711,6cyclictest0-21swapper/000:09:400
1864299263,13cyclictest35966irq/58-eth000:15:143
1862999264,7cyclictest0-21swapper/223:40:142
1862999264,13cyclictest27346irq/40-dwc2_hso20:50:112
1862999263,18cyclictest27346irq/40-dwc2_hso21:10:132
1861899267,4cyclictest201rcuc/121:47:391
1861899267,13cyclictest7805-21ssh23:35:121
1861899264,6cyclictest26002-21cat21:55:321
1861899264,6cyclictest1513-21kthreadcore20:30:241
1861899263,9cyclictest11807-21ssh00:11:281
1861899263,6cyclictest21-21ksoftirqd/120:50:281
18618992619,5cyclictest5484-21apt-get22:40:131
1861899261,10cyclictest15019-21ssh00:30:511
1860999266,9cyclictest12-21ksoftirqd/000:04:260
1860999265,8cyclictest12-21ksoftirqd/023:13:050
1860999264,14cyclictest9250-21ssh23:00:110
1860999262,15cyclictest3064-21ssh21:45:140
18609992613,7cyclictest0-21swapper/021:20:150
18609992611,5cyclictest12-21ksoftirqd/000:24:470
1864299252,17cyclictest0-21swapper/300:39:033
1864299252,17cyclictest0-21swapper/300:39:023
18642992517,5cyclictest35966irq/58-eth020:00:123
1862999254,13cyclictest27346irq/40-dwc2_hso20:00:122
1862999251,10cyclictest93066snmpd21:25:192
1861899257,12cyclictest0-21swapper/119:55:111
1861899256,5cyclictest0-21swapper/120:15:171
1861899256,4cyclictest29717-21sed20:25:281
1861899256,3cyclictest131rcu_preempt21:05:191
1861899255,7cyclictest6564-21grep19:30:201
1861899255,6cyclictest2456-21cut20:00:181
1861899255,4cyclictest32390-21irqstats21:25:201
1861899255,4cyclictest2428-21ntp_192.168.11423:30:241
1861899254,7cyclictest21-21ksoftirqd/122:30:111
1861899254,6cyclictest31792-21awk19:20:241
1861899254,4cyclictest21-21ksoftirqd/120:55:161
1861899254,4cyclictest201rcuc/121:19:411
1861899254,4cyclictest10671-21ntp_states22:05:261
1861899254,18cyclictest15167-21ssh00:15:141
1861899251,8cyclictest18997-21diskstats00:35:161
1861899251,8cyclictest18997-21diskstats00:35:151
1861899251,10cyclictest18549-21ssh22:28:591
1860999259,9cyclictest12-21ksoftirqd/022:38:200
1860999253,3cyclictest131rcu_preempt22:55:280
1860999252,6cyclictest12-21ksoftirqd/022:50:270
1864299245,12cyclictest0-21swapper/322:40:123
1864299241,16cyclictest0-21swapper/323:45:123
1864299241,15cyclictest35966irq/58-eth023:35:123
1862999245,14cyclictest27-21ksoftirqd/222:20:132
1862999242,8cyclictest0-21swapper/222:35:122
18629992412,5cyclictest16000-21ssh23:02:452
1862999241,13cyclictest5509-21rm22:40:132
18629992411,10cyclictest7752-21latency_hist20:40:002
18629992410,11cyclictest32001-21cut00:05:252
1862999240,13cyclictest10743-21find22:25:112
1861899245,4cyclictest21-21ksoftirqd/123:02:121
1861899244,5cyclictest21-21ksoftirqd/120:35:211
1861899244,4cyclictest201rcuc/123:09:171
1861899243,5cyclictest21-21ksoftirqd/122:15:351
1861899242,7cyclictest4107-21diskmemload21:20:071
1861899242,10cyclictest31764-21ssh23:46:501
18618992417,4cyclictest1538-21sh22:00:261
1860999248,4cyclictest12-21ksoftirqd/023:33:330
1860999242,17cyclictest31699-21ssh21:25:150
1860999241,17cyclictest10770-21ssh22:25:140
18609992411,4cyclictest12-21ksoftirqd/022:16:230
1864299238,10cyclictest35966irq/58-eth023:00:143
1862999238,6cyclictest23116-21kthreadcore21:55:212
1862999232,16cyclictest27346irq/40-dwc2_hso20:25:132
1862999231,17cyclictest27346irq/40-dwc2_hso22:50:112
1862999231,11cyclictest31763-21munin-run21:05:002
1861899235,14cyclictest0-21swapper/119:45:131
1861899234,5cyclictest21-21ksoftirqd/120:45:011
1861899234,5cyclictest21-21ksoftirqd/119:50:251
1861899232,5cyclictest21-21ksoftirqd/119:40:121
1861899232,4cyclictest21-21ksoftirqd/119:15:261
1861899232,15cyclictest0-21swapper/119:10:121
1860999239,4cyclictest12-21ksoftirqd/022:40:490
1860999238,8cyclictest12-21ksoftirqd/023:43:540
1860999238,8cyclictest12-21ksoftirqd/000:19:200
1860999238,4cyclictest12-21ksoftirqd/023:52:180
1860999237,8cyclictest12-21ksoftirqd/022:03:560
1860999236,4cyclictest131rcu_preempt21:15:290
1860999236,3cyclictest131rcu_preempt19:35:270
1860999236,10cyclictest131rcu_preempt20:45:270
1860999235,4cyclictest12-21ksoftirqd/000:31:330
1860999234,6cyclictest12-21ksoftirqd/020:10:020
1860999234,5cyclictest12-21ksoftirqd/019:10:180
1860999232,4cyclictest131rcu_preempt22:45:140
18609992318,3cyclictest0-21swapper/019:50:220
1860999231,7cyclictest25832-21/usr/sbin/munin20:55:260
18609992311,6cyclictest0-21swapper/021:11:450
1864299223,13cyclictest0-21swapper/300:28:533
1864299222,15cyclictest35966irq/58-eth023:20:133
18642992214,6cyclictest0-21swapper/300:00:023
1862999222,6cyclictest0-21swapper/221:50:132
18629992215,4cyclictest27-21ksoftirqd/219:45:252
18629992212,8cyclictest0-21swapper/223:05:282
1862999221,12cyclictest14929-21sendmail-msp19:40:022
1862999221,12cyclictest0-21swapper/223:10:122
1861899225,4cyclictest0-21swapper/121:00:101
1861899222,4cyclictest14696-21cut20:10:211
1861899220,5cyclictest21-21ksoftirqd/120:05:111
1861899220,4cyclictest91-21pr/ttyS020:25:011
1860999227,8cyclictest12-21ksoftirqd/023:22:450
1860999227,8cyclictest12-21ksoftirqd/019:25:160
1860999223,5cyclictest12-21ksoftirqd/021:38:390
1860999223,5cyclictest12-21ksoftirqd/019:20:180
1860999221,3cyclictest0-21swapper/021:50:180
18609992210,4cyclictest12-21ksoftirqd/020:25:020
18609992210,4cyclictest12-21ksoftirqd/020:00:170
1860999220,6cyclictest844-21munin-node20:30:150
1864299213,13cyclictest0-21swapper/321:40:133
1864299212,15cyclictest0-21swapper/323:30:133
1862999213,10cyclictest18688-21kthreadcore22:10:212
1862999212,15cyclictest15617-21irqcore19:40:172
1862999212,10cyclictest3065-21kthreadcore19:25:212
18629992111,8cyclictest21823-21kthreadcore22:30:212
1862999211,16cyclictest31626-21latency19:20:202
1862999211,13cyclictest25366-21irqrtprio19:50:172
1862999211,11cyclictest22645-21ssh21:20:142
1862999211,11cyclictest20611-21ssh00:00:142
1861899219,9cyclictest711-21apt-get19:25:131
1861899211,6cyclictest21-21ksoftirqd/119:40:021
1860999217,5cyclictest12-21ksoftirqd/020:55:020
1860999215,3cyclictest131rcu_preempt19:55:290
1860999214,3cyclictest131rcu_preempt20:35:280
18609992110,4cyclictest12-21ksoftirqd/020:15:060
1864299209,8cyclictest0-21swapper/323:29:253
1864299208,9cyclictest0-21swapper/300:15:013
1862999208,9cyclictest12854-21ssh23:20:042
1862999202,9cyclictest18653-21latency_hist19:10:012
1862999201,9cyclictest31274-21kthreadcore19:55:202
1862999201,11cyclictest24636-21ntp_kernel_err20:20:242
1862999201,11cyclictest16639-21kthreadcore22:45:222
1862999201,10cyclictest8856-21kthreadcore20:40:172
1862999201,10cyclictest3223-21fschecks_count21:45:152
18629992010,8cyclictest0-21swapper/220:10:152
1860999209,4cyclictest12-21ksoftirqd/020:25:310
1860999204,11cyclictest131rcu_preempt21:00:280
1864299192,6cyclictest0-21swapper/322:25:143
1864299192,10cyclictest0-21swapper/320:40:133
1864299192,10cyclictest0-21swapper/320:30:173
1864299191,12cyclictest0-21swapper/319:55:283
1864299191,11cyclictest0-21swapper/319:40:133
18642991910,6cyclictest0-21swapper/322:35:153
1860999197,4cyclictest12-21ksoftirqd/020:45:020
1860999193,4cyclictest12-21ksoftirqd/019:40:200
1860999191,4cyclictest12-21ksoftirqd/019:50:020
1864299189,6cyclictest35966irq/58-eth021:25:153
1864299189,6cyclictest0-21swapper/300:05:433
1864299185,9cyclictest0-21swapper/323:05:253
1864299182,10cyclictest0-21swapper/320:25:143
18642991810,6cyclictest35966irq/58-eth022:50:103
18642991810,6cyclictest35966irq/58-eth021:35:153
1862999181,8cyclictest18875-21fschecks_count19:10:162
1860999185,5cyclictest131rcu_preempt19:10:020
1860999181,6cyclictest15354-21/usr/sbin/munin20:10:270
1864299179,6cyclictest0-21swapper/322:17:453
1864299179,6cyclictest0-21swapper/322:12:593
1864299178,6cyclictest0-21swapper/322:20:043
1864299178,6cyclictest0-21swapper/322:00:203
1864299178,6cyclictest0-21swapper/321:21:193
1864299178,6cyclictest0-21swapper/321:15:133
1864299174,5cyclictest0-21swapper/321:00:123
1864299172,9cyclictest0-21swapper/320:45:123
1864299171,10cyclictest0-21swapper/319:35:273
1864299162,9cyclictest0-21swapper/320:05:133
1864299161,7cyclictest0-21swapper/320:20:123
1864299157,6cyclictest0-21swapper/319:31:333
1864299152,10cyclictest0-21swapper/321:05:123
1864299151,7cyclictest0-21swapper/321:00:013
1864299151,11cyclictest0-21swapper/320:10:273
1864299151,11cyclictest0-21swapper/319:10:203
1864299141,9cyclictest35966irq/58-eth019:20:123
1864299141,7cyclictest0-21swapper/319:25:293
1861899141,10cyclictest18479-21latency_hist19:10:021
1864299136,5cyclictest0-21swapper/320:35:203
1864299131,10cyclictest0-21swapper/320:50:133
1864299121,4cyclictest0-21swapper/319:10:023
1864299112,5cyclictest0-21swapper/319:15:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional