You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-30 - 10:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Sat May 30, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15287996849,14cyclictest27346irq/40-dwc2_hso21:40:122
15276996130,6cyclictest27457-21apt-get20:30:121
1528799554,40cyclictest12572-21ssh22:10:122
15276995239,7cyclictest10173-21ssh21:15:121
1528799483,9cyclictest131rcu_preempt23:35:122
1528799482,25cyclictest20524-21fschecks_time22:50:152
1528799461,36cyclictest18701-21sh23:25:122
15276994640,3cyclictest26229-21apt-get19:55:131
1526499466,35cyclictest27746-21ssh23:30:110
1526499463,5cyclictest131rcu_preempt21:20:130
15287994512,17cyclictest15516-21diskmemload23:58:162
15276994536,4cyclictest29940-21apt-get19:25:121
15264994534,7cyclictest19072-21apt-get20:55:120
1526499452,35cyclictest17390-21ssh00:00:120
15264994523,18cyclictest12-21ksoftirqd/019:25:120
15276994414,12cyclictest12863-21kthreadcore00:15:191
15264994420,9cyclictest0-21swapper/022:40:120
15264994420,9cyclictest0-21swapper/022:40:120
15264994415,8cyclictest6932-21ssh23:17:100
15264994413,11cyclictest9464-21ssh22:06:450
15276994315,11cyclictest28508-21ssh00:23:261
15264994325,14cyclictest24168-21ssh23:10:110
15264994317,11cyclictest19924-21cat00:35:280
15296994212,19cyclictest35966irq/58-eth022:00:133
1528799421,22cyclictest27-21ksoftirqd/200:20:142
15276994215,11cyclictest3510-21ssh22:20:521
15276994213,11cyclictest22882-21ssh21:55:371
15264994216,11cyclictest20743-21ssh23:43:270
15264994214,10cyclictest12480-21ssh22:10:070
1526499421,29cyclictest6871-21ssh00:30:120
1529699415,24cyclictest35966irq/58-eth021:15:123
1528799412,31cyclictest16741-21if_eth022:30:152
15276994126,11cyclictest11240-21ssh22:45:111
15276994115,11cyclictest6933-21ssh23:17:101
15276994111,11cyclictest6924-21ssh21:48:201
15276994110,12cyclictest10947-21ssh22:25:221
15264994137,2cyclictest17867-21apt-get20:20:120
15264994126,10cyclictest5054-21ssh21:30:120
15264994115,10cyclictest18029-21ssh00:16:440
15264994114,9cyclictest1085-21ssh21:10:040
15264994113,8cyclictest26947-21ssh00:21:350
15276994032,5cyclictest25129-21apt-get19:20:121
15276994024,12cyclictest24158-21apt-get23:10:111
15276994015,9cyclictest26926-21sh22:00:091
15276994014,8cyclictest28964-21cut22:55:021
15276994014,10cyclictest4982-21ssh21:30:081
15264994032,6cyclictest25519-21apt-get22:35:140
15264994031,5cyclictest8215-21apt-get20:10:120
15264994014,7cyclictest31617-21ssh21:44:550
15264994014,15cyclictest28746-21/usr/sbin/munin19:20:260
15264994014,10cyclictest17644-21ssh23:24:130
15264994013,10cyclictest8867-21entropy21:50:150
1528799392,19cyclictest30847-21ssh22:20:152
1528799392,12cyclictest27-21ksoftirqd/200:30:132
15276993931,4cyclictest32226-21apt-get20:35:121
1527699392,31cyclictest22530-21ssh23:45:121
15276993914,9cyclictest29108-21sed21:05:181
15276993914,9cyclictest29108-21sed21:05:181
15276993913,11cyclictest28046-21ssh21:25:001
15264993929,8cyclictest16384-21apt-get22:30:110
15264993915,9cyclictest22535-21ssh23:08:380
15264993915,9cyclictest21357-21ssh21:38:250
15264993914,12cyclictest1-21systemd22:57:250
15264993914,11cyclictest7528-21ssh23:54:390
1527699389,11cyclictest2285-21/usr/sbin/munin19:30:221
15276993831,4cyclictest30254-21apt-get00:25:151
15276993830,5cyclictest31044-21apt-get20:00:111
15276993827,6cyclictest21494-21apt-get19:50:131
15276993821,13cyclictest23103-21ssh21:40:121
15276993815,8cyclictest17293-21ssh00:00:061
15276993811,10cyclictest31143-21ssh22:36:271
15264993813,10cyclictest26244-21ssh22:51:490
15264993812,8cyclictest5935-21ssh23:00:130
15264993812,7cyclictest819-21diskmemload23:45:460
15264993811,10cyclictest32194-21/usr/sbin/munin21:25:270
15264993811,10cyclictest14113-21ssh23:56:370
1528799374,26cyclictest27346irq/40-dwc2_hso19:50:122
1528799372,18cyclictest11459-21ssh23:01:152
15276993730,4cyclictest13027-21apt-get20:15:121
15276993719,12cyclictest8297-21apt-get23:55:111
15276993715,7cyclictest21383-21ssh22:30:381
15276993714,8cyclictest23740-21ssh23:25:391
15276993713,10cyclictest22833-21ssh00:37:551
1527699371,30cyclictest0-21swapper/121:10:121
1526499379,11cyclictest512-21ssh22:01:530
15264993731,3cyclictest17860-21apt-get21:55:110
15264993716,9cyclictest25904-21cat20:25:210
15264993714,8cyclictest14642-21ssh22:28:230
15264993714,10cyclictest11551-21ssh00:14:480
15264993714,10cyclictest11551-21ssh00:14:480
15264993713,8cyclictest16418-21ssh22:45:500
15264993713,8cyclictest12046-21ssh23:38:560
15264993712,8cyclictest25265-21ntp_192.168.11422:15:240
1529699365,22cyclictest35966irq/58-eth022:20:123
1528799362,28cyclictest23706-21ssh00:02:162
1527699368,11cyclictest12724-21/usr/sbin/munin20:45:221
15276993621,13cyclictest15049-21apt-get23:05:151
15276993614,8cyclictest393-21ssh23:30:501
15276993613,10cyclictest4910-21rm23:51:321
15276993612,10cyclictest12330-21ssh21:50:231
15276993611,8cyclictest0-21swapper/123:37:421
15264993623,9cyclictest0-21swapper/019:50:120
15264993618,15cyclictest0-21swapper/020:30:120
15264993614,10cyclictest28772-21cpuspeed21:05:140
15264993614,10cyclictest28772-21cpuspeed21:05:140
15264993612,9cyclictest32626-21cut20:35:170
15264993612,8cyclictest0-21swapper/020:50:240
15276993530,2cyclictest4633-21apt-get20:40:131
15276993530,2cyclictest4633-21apt-get20:40:131
15276993512,8cyclictest8827-21ssh00:11:351
15276993512,8cyclictest8827-21ssh00:11:351
15276993512,10cyclictest28548-21df21:25:151
15276993511,8cyclictest7329-21ssh22:40:511
15276993511,8cyclictest7329-21ssh22:40:511
15276993511,8cyclictest0-21swapper/121:37:581
15276993511,7cyclictest0-21swapper/122:19:381
1526499358,9cyclictest26995-21grep19:55:190
15264993523,8cyclictest15500-21diskstats19:10:140
15264993512,8cyclictest29973-21missed_timers00:05:240
15264993511,7cyclictest18814-21df_inode23:25:140
15264993511,7cyclictest15927-21ssh21:16:330
15264993511,10cyclictest20386-21/usr/sbin/munin19:15:130
1528799343,23cyclictest0-21swapper/221:52:122
15287993421,8cyclictest261rcuc/223:52:172
15287993410,18cyclictest3071-21ssh00:10:162
15287993410,18cyclictest3071-21ssh00:10:162
15276993429,2cyclictest5913-21apt-get23:00:131
15276993427,4cyclictest23901-21apt-get21:00:121
15276993424,5cyclictest7083-21apt-get19:35:121
15276993423,8cyclictest19983-21apt-get19:15:021
15276993413,7cyclictest16215-21rm23:22:301
1526499347,10cyclictest15370-21sed19:10:020
15264993414,7cyclictest5569-21ssh00:28:560
15264993412,8cyclictest12534-21irqstats19:40:170
1529699331,21cyclictest35966irq/58-eth000:25:153
1528799332,19cyclictest930-21snmpd21:35:152
15276993326,3cyclictest3405-21apt-get20:05:141
15276993311,8cyclictest0-21swapper/122:09:271
15276993311,7cyclictest19929-21ssh23:42:291
15276993311,12cyclictest0-21swapper/122:55:231
15264993321,9cyclictest0-21swapper/021:00:120
15264993314,6cyclictest31096-21cpuspeed20:00:120
15264993312,8cyclictest17835-21kthreadcore19:45:190
15264993311,8cyclictest5842-21run-parts19:30:260
15264993310,8cyclictest0-21swapper/020:45:180
15296993223,6cyclictest35966irq/58-eth022:05:133
1528799329,8cyclictest27-21ksoftirqd/223:43:162
1528799328,19cyclictest930-21snmpd22:57:152
15287993213,13cyclictest27-21ksoftirqd/221:10:122
1527699328,7cyclictest0-21swapper/100:08:351
15276993213,8cyclictest17544-21sh22:10:351
15264993220,8cyclictest13132-21/usr/sbin/munin20:15:150
15264993210,6cyclictest0-21swapper/021:47:500
15264993210,10cyclictest30845-21/usr/sbin/munin22:20:150
1529699312,19cyclictest35966irq/58-eth022:35:133
1529699312,19cyclictest0-21swapper/300:20:143
1528799312,21cyclictest27346irq/40-dwc2_hso20:35:132
15276993112,8cyclictest1465166ssh00:33:571
15264993119,9cyclictest6878-21ntp_states20:05:260
1529699305,11cyclictest35966irq/58-eth021:10:123
1529699303,23cyclictest35966irq/58-eth020:15:123
15296993025,3cyclictest0-21swapper/323:55:123
1528799303,19cyclictest27346irq/40-dwc2_hso20:50:122
15276993023,5cyclictest19983-21apt-get19:15:031
15276993018,9cyclictest14482-21sed20:50:151
15264993021,7cyclictest0-21swapper/019:35:130
15264993018,9cyclictest4839-21fw_packets20:40:160
15264993018,9cyclictest4839-21fw_packets20:40:160
1529699292,18cyclictest35966irq/58-eth020:25:123
1528799294,15cyclictest27346irq/40-dwc2_hso19:20:112
1528799293,17cyclictest27346irq/40-dwc2_hso19:30:112
1528799292,22cyclictest32118-21ssh21:45:142
1528799292,20cyclictest0-21swapper/220:00:112
15276992918,8cyclictest16759-21/usr/sbin/munin19:45:131
15276992917,9cyclictest21304-21ntpq20:20:241
15276992916,8cyclictest22679-21cpu20:25:131
1529699285,18cyclictest0-21swapper/321:50:123
1529699283,22cyclictest35966irq/58-eth023:35:133
1529699283,16cyclictest35966irq/58-eth023:50:113
1529699282,16cyclictest35966irq/58-eth020:40:123
1529699282,16cyclictest35966irq/58-eth020:40:123
15296992815,9cyclictest35966irq/58-eth023:15:103
15296992814,9cyclictest35966irq/58-eth023:05:163
1528799284,14cyclictest27-21ksoftirqd/221:30:122
1528799282,20cyclictest0-21swapper/220:15:132
1528799282,11cyclictest0-21swapper/220:10:132
1528799282,10cyclictest1083-21diskmemload22:39:152
1529699273,20cyclictest35966irq/58-eth021:25:133
1529699273,17cyclictest0-21swapper/322:48:163
1529699272,17cyclictest0-21swapper/321:45:143
1529699271,18cyclictest35966irq/58-eth023:00:113
1528799273,18cyclictest27346irq/40-dwc2_hso20:20:122
1528799273,18cyclictest0-21swapper/222:27:182
1528799272,18cyclictest27346irq/40-dwc2_hso20:40:132
1528799272,18cyclictest27346irq/40-dwc2_hso20:40:132
15276992722,2cyclictest11892-21apt-get19:40:121
15276992717,7cyclictest12061-21sed20:10:281
15276992716,9cyclictest0-21swapper/120:55:111
1529699265,12cyclictest35966irq/58-eth020:45:123
1529699263,9cyclictest35966irq/58-eth021:00:123
1529699263,17cyclictest0-21swapper/322:19:153
1529699263,15cyclictest0-21swapper/322:52:243
1529699263,15cyclictest0-21swapper/322:27:183
15296992615,8cyclictest0-21swapper/300:30:153
1529699261,15cyclictest35966irq/58-eth020:35:113
1528799264,17cyclictest271ksoftirqd/200:27:162
1528799263,15cyclictest27346irq/40-dwc2_hso21:00:132
1528799262,9cyclictest19279-21ssh22:49:152
1528799262,9cyclictest1610-21ssh23:32:152
1528799262,10cyclictest0-21swapper/219:45:132
1529699254,14cyclictest0-21swapper/300:10:163
1529699254,14cyclictest0-21swapper/300:10:163
1529699252,17cyclictest0-21swapper/321:35:153
1529699252,16cyclictest35966irq/58-eth021:40:133
1529699252,14cyclictest36066irq/59-eth000:05:133
1529699252,14cyclictest35966irq/58-eth000:35:133
1528799252,18cyclictest13321-1kworker/u9:000:36:162
1528799252,15cyclictest27346irq/40-dwc2_hso20:05:142
15287992513,5cyclictest1079-21ssh21:25:442
1528799251,14cyclictest734-21ssh23:15:112
15276992513,9cyclictest15252-21cyclictest19:10:021
1529699243,15cyclictest0-21swapper/323:33:153
15296992410,8cyclictest35966irq/58-eth023:20:133
1528799241,8cyclictest819-21diskmemload00:18:162
1528799241,14cyclictest3610-21ssh22:05:132
1528799241,12cyclictest2163866ssh22:15:122
1529699236,10cyclictest35966irq/58-eth022:55:113
1529699234,12cyclictest0-21swapper/322:40:133
1529699234,12cyclictest0-21swapper/322:40:133
1529699232,15cyclictest35966irq/58-eth000:15:123
1529699231,16cyclictest0-21swapper/322:10:133
15296992310,8cyclictest0-21swapper/321:55:213
1528799234,16cyclictest22486-21/usr/sbin/munin23:45:142
1528799232,16cyclictest0-21swapper/200:07:162
15287992313,5cyclictest27-21ksoftirqd/219:40:112
1528799231,17cyclictest27346irq/40-dwc2_hso23:10:112
1529699223,13cyclictest0-21swapper/323:25:133
1529699222,14cyclictest0-21swapper/321:30:123
1528799227,7cyclictest27-21ksoftirqd/221:05:192
1528799227,7cyclictest27-21ksoftirqd/221:05:192
1528799223,10cyclictest28332-21kthreadcore19:55:202
1528799222,15cyclictest0-21swapper/223:07:272
1528799221,4cyclictest0-21swapper/223:22:332
1529699213,7cyclictest35966irq/58-eth019:20:103
1529699213,12cyclictest0-21swapper/323:40:163
1528799215,7cyclictest9201-21kthreadcore19:35:202
1528799213,12cyclictest27346irq/40-dwc2_hso20:55:122
1528799212,10cyclictest27265-21hwlatdetect22:00:162
1528799211,17cyclictest27379-21latency_hist20:30:022
1528799211,16cyclictest19099-21timerandwakeup19:10:272
1528799211,13cyclictest18984-21latency_hist21:20:022
1528799211,11cyclictest19564-21kthreadcore21:55:192
1529699203,11cyclictest0-21swapper/300:00:163
1528799206,11cyclictest19367-21apt-get21:20:132
1528799205,12cyclictest27-21ksoftirqd/222:40:282
1528799205,12cyclictest27-21ksoftirqd/222:40:282
1528799201,8cyclictest32012-21kthreadcore19:25:192
1528799201,7cyclictest0-21swapper/220:35:012
1528799201,10cyclictest12150-21kthreadcore20:45:212
1529699193,8cyclictest0-21swapper/323:45:123
15296991912,4cyclictest0-21swapper/323:10:213
1529699191,11cyclictest0-21swapper/322:30:193
1529699191,11cyclictest0-21swapper/320:55:113
1528799191,9cyclictest21386-21kthreadcore19:15:182
1529699182,10cyclictest0-21swapper/321:05:143
1529699182,10cyclictest0-21swapper/321:05:143
1529699182,10cyclictest0-21swapper/320:20:133
1529699182,10cyclictest0-21swapper/320:10:133
1529699182,10cyclictest0-21swapper/319:15:043
1528799181,8cyclictest15311-21latency_hist19:10:022
1529699172,8cyclictest35966irq/58-eth021:20:143
1529699172,10cyclictest0-21swapper/319:55:133
1529699171,10cyclictest0-21swapper/319:25:213
1529699162,9cyclictest0-21swapper/319:45:133
1529699161,9cyclictest0-21swapper/320:00:263
1529699152,10cyclictest0-21swapper/320:05:153
1529699151,8cyclictest0-21swapper/320:30:273
1529699151,7cyclictest0-21swapper/319:55:013
1529699144,6cyclictest0-21swapper/319:40:113
1529699141,11cyclictest0-21swapper/319:10:273
1529699132,7cyclictest0-21swapper/319:35:123
1529699131,10cyclictest0-21swapper/319:30:283
1529699121,9cyclictest0-21swapper/320:50:163
1529699111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional