You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-17 - 10:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Aug 17, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21437995640,12cyclictest35966irq/58-eth022:35:143
2142099544,9cyclictest23020-21apt-get00:00:141
21409995119,21cyclictest12-21ksoftirqd/022:55:150
21409994629,10cyclictest4938-21apt-get23:50:140
2142999453,35cyclictest930-21snmpd23:30:142
2143799447,22cyclictest35966irq/58-eth022:30:143
2143799426,23cyclictest35966irq/58-eth023:35:143
21437994221,11cyclictest35966irq/58-eth000:35:133
2142099422,7cyclictest0-21swapper/120:50:131
21429994133,5cyclictest0-21swapper/221:00:142
2142999411,29cyclictest16604-21sh21:15:132
2142099411,34cyclictest17295-21ssh22:45:131
2142099411,12cyclictest19304-21apt-get20:15:121
2143799401,21cyclictest35966irq/58-eth023:40:123
2142099407,31cyclictest22688-21ssh22:30:151
2142099402,6cyclictest0-21swapper/100:25:141
2142099373,10cyclictest10366-21ssh23:35:141
2142099371,7cyclictest1313-21apt-get20:30:131
2140999379,11cyclictest17744-21ssh22:09:100
2143799365,21cyclictest35966irq/58-eth021:50:133
2143799365,16cyclictest35966irq/58-eth023:00:153
2143799364,18cyclictest35966irq/58-eth000:00:153
2142099363,29cyclictest24105-21ssh21:55:121
2142099361,9cyclictest19654-21grep22:10:191
2142099361,12cyclictest11586-21sh21:10:301
21409993629,3cyclictest25416-21apt-get20:55:140
21409993618,11cyclictest13612-21ssh22:25:170
2143799353,21cyclictest35966irq/58-eth021:30:143
2143799351,22cyclictest35966irq/58-eth000:15:133
2142999352,27cyclictest20320-21diskmemload21:35:132
2142099352,6cyclictest0-21swapper/100:35:141
2142099351,9cyclictest25890-21irqcore20:55:191
2142099351,9cyclictest14617-21cut22:25:221
2142099351,8cyclictest29223-21sh23:45:181
21420993512,20cyclictest13970-21/usr/sbin/munin23:55:311
2142099351,15cyclictest12059-21rm23:00:151
2142099351,11cyclictest5160-21fschecks_time23:50:171
2142099351,10cyclictest31043-21kthreadcore21:00:201
21409993528,3cyclictest20599-21apt-get20:50:120
21409993524,8cyclictest0-21swapper/022:35:140
2143799344,18cyclictest35966irq/58-eth000:10:133
21437993412,15cyclictest35966irq/58-eth021:10:153
21429993419,12cyclictest0-21swapper/223:15:142
2142099343,5cyclictest0-21swapper/123:40:141
2142099342,9cyclictest4816-21timerandwakeup22:00:291
2142099341,9cyclictest9359-21kthreadcore00:10:201
2142099341,9cyclictest5144-21cut22:20:191
2142099341,8cyclictest25460-21latency_hist21:20:031
2142099341,8cyclictest14418-21ntp_192.168.11420:40:241
2142099341,10cyclictest32616-21ssh21:40:221
2142099340,9cyclictest31442-21tr23:10:221
2140999347,10cyclictest16560-21ssh22:44:380
2140999346,10cyclictest20829-21ssh21:35:180
2143799334,23cyclictest35966irq/58-eth021:15:133
2143799333,17cyclictest35966irq/58-eth022:25:163
2142099332,10cyclictest2800-21fschecks_time21:05:181
2142099331,9cyclictest28382-21cut00:20:231
2142099331,9cyclictest16176-21irqcore20:45:181
2142099331,9cyclictest12009-21kernelversion21:30:201
2142099331,8cyclictest27768-21cut19:15:191
2142099331,8cyclictest23577-21ssh23:05:191
2142099331,8cyclictest15867-21fschecks_time23:20:151
2142099331,8cyclictest11787-21cat23:00:011
2142099331,7cyclictest14162-21cut00:30:241
2142099331,11cyclictest25934-21hwlatdetect21:20:161
2142099331,10cyclictest14674-21ls21:50:011
2142099331,10cyclictest13370-21memory22:05:241
21409993322,8cyclictest19377-21ssh23:40:120
21409993320,8cyclictest6056-21ssh21:45:140
2143799323,18cyclictest35966irq/58-eth020:30:133
21437993215,10cyclictest35966irq/58-eth022:50:133
2142099329,20cyclictest21-21ksoftirqd/119:50:131
2142099329,20cyclictest21-21ksoftirqd/119:50:131
2142099322,7cyclictest2817-21irqcore21:25:221
21420993224,4cyclictest7124-21diskmemload21:50:141
2142099322,10cyclictest17416-21ssh00:15:141
2142099321,8cyclictest7128-21kthreadcore20:35:211
2142099321,8cyclictest24763-21tr19:45:231
2142099321,8cyclictest17881-21sendmail19:40:011
2142099321,10cyclictest8603-21wc20:00:311
2142099321,10cyclictest25404-21grep19:10:291
2140999328,14cyclictest7124-21diskmemload23:29:390
2140999327,10cyclictest27308-21ssh23:07:160
21409993222,8cyclictest15055-21apt-get21:50:140
2140999322,19cyclictest769-21ssh22:00:140
21409993221,9cyclictest10354-21apt-get23:35:150
2143799312,18cyclictest35966irq/58-eth020:40:133
21429993116,10cyclictest4919-21apt-get20:00:152
2142999311,11cyclictest4544-21ssh22:20:142
2142099314,5cyclictest15163-21kthreadcore20:10:211
2142099312,14cyclictest26283-21ssh22:50:131
2142099312,10cyclictest29029-21dpkg21:40:011
2142099311,10cyclictest2432-21wc19:20:281
2142099311,10cyclictest21811-21timerandwakeup19:40:281
2143799303,17cyclictest35966irq/58-eth021:20:133
21437993015,10cyclictest35966irq/58-eth021:25:173
2142999305,15cyclictest27346irq/40-dwc2_hso19:20:152
2142999304,13cyclictest27346irq/40-dwc2_hso22:55:142
2142999302,18cyclictest32006-21ssh00:05:162
2142099305,3cyclictest0-21swapper/122:41:161
2142099302,9cyclictest5997-21cat23:30:341
2142099301,9cyclictest32396-21ntp_states20:25:271
2142099301,8cyclictest27607-21ntp_states20:20:261
2142099300,10cyclictest27929-21ssh22:15:131
2142099300,10cyclictest27929-21ssh22:15:131
2140999307,8cyclictest7994-21ssh21:26:340
2140999307,7cyclictest12042-21apt-get23:00:150
2140999306,9cyclictest31087-21ssh00:04:250
21409993025,2cyclictest2859-21apt-get00:25:150
2143799292,18cyclictest35966irq/58-eth022:45:143
2142999297,9cyclictest27-21ksoftirqd/222:53:262
2142999297,8cyclictest27-21ksoftirqd/222:14:492
2142999297,8cyclictest27-21ksoftirqd/221:54:352
2142999295,14cyclictest27-21ksoftirqd/223:12:192
2142999294,14cyclictest27346irq/40-dwc2_hso19:35:112
2142999292,20cyclictest0-21swapper/222:35:142
2142099291,9cyclictest9971-21if_eth020:05:171
2142099291,9cyclictest28595-21grep23:25:321
2142099291,7cyclictest7350-21grep23:15:211
2142099291,7cyclictest31624-21/usr/sbin/munin22:35:261
2142099291,10cyclictest32098-21df00:05:161
2140999298,7cyclictest7124-21diskmemload21:44:160
21409992923,4cyclictest8481-21apt-get00:10:140
2140999292,23cyclictest16622-21ssh21:15:140
21409992921,4cyclictest11006-21apt-get20:40:130
21437992819,6cyclictest35966irq/58-eth022:05:133
2142999287,13cyclictest27-21ksoftirqd/223:49:132
2142999286,9cyclictest27-21ksoftirqd/223:00:072
2142999286,9cyclictest27-21ksoftirqd/221:28:362
2142999286,9cyclictest27-21ksoftirqd/200:13:172
2142999285,9cyclictest27-21ksoftirqd/221:33:182
2142999284,15cyclictest27346irq/40-dwc2_hso19:30:122
2142999284,14cyclictest27346irq/40-dwc2_hso21:20:132
21429992812,12cyclictest8463-21memory23:50:252
2142099281,8cyclictest7109-21sed19:25:261
2142099280,9cyclictest776-21cut19:55:201
2142099280,10cyclictest9961-21cut19:30:191
2140999286,10cyclictest28951-21apt20:25:140
2140999283,16cyclictest1341-21ssh23:30:150
21409992822,3cyclictest2601-21apt-get21:05:140
21409992822,3cyclictest22899-21apt-get19:45:140
21409992818,7cyclictest7305-21irqstats23:15:210
2142999277,8cyclictest27-21ksoftirqd/222:44:172
2142999277,8cyclictest27-21ksoftirqd/200:30:412
2142999276,14cyclictest27-21ksoftirqd/223:29:472
2142999275,3cyclictest131rcu_preempt22:31:242
2142999272,19cyclictest0-21swapper/223:05:122
2142099273,5cyclictest630-21Async09:36:311
2140999275,7cyclictest0-21swapper/023:48:040
2140999274,8cyclictest3507-21ntp_192.168.11419:55:250
2140999274,7cyclictest7389-21sed19:25:290
2140999272,11cyclictest23814-21ssh00:17:290
21409992717,7cyclictest0-21swapper/022:45:140
21409992715,9cyclictest0-21swapper/020:35:120
21409992715,8cyclictest25674-21apt-get21:20:130
2142999267,12cyclictest27-21ksoftirqd/200:35:472
2142999263,8cyclictest27-21ksoftirqd/221:44:292
2142999262,15cyclictest930-21snmpd23:20:122
2140999266,8cyclictest2770-21rm23:10:550
2140999265,8cyclictest31633-21usage_Anybus19:50:290
2140999265,8cyclictest31633-21usage_Anybus19:50:290
2140999265,8cyclictest26266-21apt-get22:50:130
2140999262,18cyclictest4535-21ssh22:20:140
21409992612,10cyclictest13827-21ssh23:55:020
21437992515,7cyclictest35966irq/58-eth000:20:273
2142999258,7cyclictest27-21ksoftirqd/221:58:112
2142999257,11cyclictest27-21ksoftirqd/220:40:272
2142999256,9cyclictest27-21ksoftirqd/221:48:282
2142999256,8cyclictest27-21ksoftirqd/220:30:012
2142999256,12cyclictest27-21ksoftirqd/222:03:032
2142999256,12cyclictest27-21ksoftirqd/200:17:052
2142999253,8cyclictest27-21ksoftirqd/221:11:352
2142999252,9cyclictest27-21ksoftirqd/200:00:222
2142999252,4cyclictest27-21ksoftirqd/223:35:232
2142999252,2cyclictest131rcu_preempt22:05:222
21429992514,7cyclictest27710-21apt-get19:50:132
21429992514,7cyclictest27710-21apt-get19:50:122
21429992511,4cyclictest27-21ksoftirqd/222:26:042
2140999256,8cyclictest22695-21cpu22:30:150
2140999255,10cyclictest18146-21ssh00:32:230
2140999254,12cyclictest16971-21ssh21:31:320
2140999253,8cyclictest25798-21ssh00:35:460
2140999253,17cyclictest7552-21ssh21:10:150
21409992512,10cyclictest24751-21irqstats20:20:200
2143799241,18cyclictest0-21swapper/323:30:163
2142999246,12cyclictest27346irq/40-dwc2_hso19:25:142
2142999243,10cyclictest27-21ksoftirqd/223:41:462
2142999242,5cyclictest271ksoftirqd/220:30:152
2142999241,6cyclictest27-21ksoftirqd/200:20:072
21429992414,8cyclictest261rcuc/223:57:522
2142999241,15cyclictest6318-21ls00:25:252
2142999241,12cyclictest23029-21grep19:45:172
2142999241,10cyclictest27938-21ssh22:15:142
2142999241,10cyclictest27938-21ssh22:15:132
2140999245,14cyclictest31080-21ssh21:57:580
21409992417,4cyclictest11910-21ntp_192.168.11419:30:230
21409992414,6cyclictest0-21swapper/020:30:140
21409992412,8cyclictest26155-21apt-get19:15:040
21409992411,10cyclictest25140-21ssh22:12:120
21409992410,9cyclictest15653-21ssh23:20:120
21409992410,11cyclictest3343-21processes00:05:290
21409992410,10cyclictest31666-21sed19:20:190
21409992410,10cyclictest19206-21cut19:40:200
2143799232,11cyclictest35966irq/58-eth000:30:163
2143799231,15cyclictest0-21swapper/323:45:113
21437992310,9cyclictest35966irq/58-eth022:40:133
2142999237,7cyclictest27-21ksoftirqd/219:10:272
2142999236,11cyclictest27-21ksoftirqd/222:46:112
2140999239,10cyclictest14833-21cut20:10:180
2140999238,12cyclictest0-21swapper/020:00:140
2140999234,6cyclictest0-21swapper/022:15:050
2140999234,6cyclictest0-21swapper/022:15:050
21409992316,3cyclictest15815-21apt-get20:45:130
21409992314,6cyclictest19391-21df_inode20:15:140
21409992312,8cyclictest21674-21/usr/sbin/munin19:10:160
21409992310,9cyclictest30161-21cut00:20:310
2142999223,15cyclictest0-21swapper/221:05:132
2142999221,5cyclictest27-21ksoftirqd/220:50:282
21429992211,8cyclictest0-21swapper/219:40:262
2140999222,11cyclictest12085-21kthreadcore20:05:210
2140999220,19cyclictest1078-21ls21:00:230
21437992112,7cyclictest0-21swapper/321:36:463
21437992112,6cyclictest35966irq/58-eth022:20:133
2143799211,13cyclictest35966irq/58-eth023:25:163
2142999212,15cyclictest9648-21ntp_states20:35:262
2142999211,4cyclictest27-21ksoftirqd/219:55:232
2142999210,9cyclictest13119-21/usr/sbin/munin20:05:252
2143799208,9cyclictest0-21swapper/322:15:233
2143799208,9cyclictest0-21swapper/322:15:223
21437992013,5cyclictest0-21swapper/321:55:153
21437992012,6cyclictest0-21swapper/323:17:363
2142999201,15cyclictest26693-21hwlatdetect19:15:162
2142999201,10cyclictest15924-21sort20:45:162
2142999200,17cyclictest25469-21sort20:55:152
2140999205,12cyclictest13288-21/usr/sbin/munin19:35:130
2143799191,12cyclictest0-21swapper/320:55:013
2143799191,12cyclictest0-21swapper/320:40:013
21437991910,6cyclictest0-21swapper/323:07:063
2142999191,9cyclictest21488-21cat19:10:022
21429991913,4cyclictest0-21swapper/220:10:162
2142999191,15cyclictest24242-21tune2fs20:20:162
2142999191,15cyclictest19635-21irqcore20:15:172
2143799189,6cyclictest0-21swapper/322:55:163
2143799189,6cyclictest0-21swapper/322:00:453
2143799188,5cyclictest0-21swapper/300:05:153
2143799187,6cyclictest0-21swapper/323:23:203
2143799182,10cyclictest0-21swapper/321:40:043
2143799182,10cyclictest0-21swapper/320:45:143
2143799182,10cyclictest0-21swapper/320:15:143
2143799182,10cyclictest0-21swapper/320:05:143
2143799182,10cyclictest0-21swapper/319:30:133
21437991811,5cyclictest0-21swapper/323:51:453
2143799181,13cyclictest0-21swapper/319:40:163
21437991810,5cyclictest0-21swapper/323:10:243
21437991810,3cyclictest0-21swapper/300:27:303
2140999186,9cyclictest21541-21latency_hist19:10:020
2143799179,6cyclictest0-21swapper/322:10:113
2143799179,4cyclictest0-21swapper/320:20:143
2143799178,6cyclictest35966irq/58-eth021:45:133
2143799172,9cyclictest0-21swapper/321:05:143
2143799172,10cyclictest0-21swapper/320:55:153
2143799172,10cyclictest0-21swapper/320:25:143
2143799172,10cyclictest0-21swapper/320:10:153
2143799172,10cyclictest0-21swapper/319:20:163
21437991711,4cyclictest0-21swapper/323:55:333
2143799171,12cyclictest0-21swapper/319:10:013
2143799157,6cyclictest0-21swapper/319:52:293
2143799157,6cyclictest0-21swapper/319:52:293
2143799151,8cyclictest0-21swapper/319:55:233
2143799142,7cyclictest0-21swapper/321:00:143
2143799142,10cyclictest0-21swapper/319:10:143
2143799141,10cyclictest0-21swapper/319:15:153
2143799138,3cyclictest0-21swapper/319:25:283
2143799131,7cyclictest0-21swapper/319:35:103
2143799121,9cyclictest0-21swapper/320:00:243
2143799111,8cyclictest0-21swapper/319:45:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional