You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-02 - 16:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Sun Nov 02, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
898769916430,1638cyclictest3648032sleep310:44:413
898769916211,1615cyclictest1917732sleep308:40:193
898769915420,1537cyclictest1811462sleep308:30:263
898769915350,1530cyclictest5466002sleep312:37:333
898769915293,1521cyclictest2323862sleep309:15:473
898769915141,1508cyclictest1021902sleep307:17:313
898769915020,1497cyclictest1181432sleep307:32:463
898769914980,1493cyclictest2082752sleep308:56:473
898769914971,1491cyclictest4186462sleep311:19:573
898769914901,1484cyclictest1394632sleep307:53:473
898769914900,1485cyclictest2136282sleep309:02:323
898769914891,1483cyclictest5288112sleep312:27:073
898769914870,1482cyclictest4471142sleep311:36:323
898769914790,1463cyclictest4390222sleep311:30:393
898769914781,1472cyclictest948472sleep307:11:473
898769914611,1455cyclictest5378482sleep312:32:323
898769914601,1454cyclictest5210512sleep312:23:463
898769914200,1415cyclictest4693222sleep311:50:183
898769914081,1402cyclictest1234982sleep307:38:323
898769913831,1377cyclictest3203852sleep310:15:073
898769913721,1366cyclictest1708452sleep308:20:323
898769913691,1363cyclictest5040992sleep312:14:323
898769913531,1347cyclictest1606272sleep308:14:473
898769913321,1326cyclictest1871142sleep308:35:473
89876995400,536cyclictest304719-21turbostat10:09:553
24339221360,7sleep2243397-21gltestperf09:25:162
3492272271,2sleep20-21swapper/210:33:432
1985452241,18sleep1198570-21/usr/sbin/munin08:50:131
89876992219,1cyclictest0-21swapper/311:25:013
89876992215,4cyclictest486354-21taskset12:02:103
89876992212,7cyclictest473606-21chrt11:55:163
89876992117,2cyclictest0-21swapper/307:20:153
8987699211,16cyclictest240129-21taskset09:20:483
89876992110,7cyclictest3127962sleep310:10:063
8987699210,19cyclictest0-21swapper/309:10:033
8987699209,8cyclictest134319-21sleep07:50:003
8987699201,17cyclictest0-21swapper/309:41:133
8987699201,14cyclictest0-21swapper/310:55:133
8987699200,18cyclictest0-21swapper/311:47:403
8987699191,6cyclictest449539-21taskset11:40:133
8987699191,14cyclictest0-21swapper/311:26:583
8987699191,14cyclictest0-21swapper/309:35:143
89876991910,6cyclictest0-21swapper/309:09:593
4268152191,4sleep0427074-21df11:25:160
1236392190,7sleep2123529-31sort07:39:002
8987699183,13cyclictest257004-21taskset09:33:263
89876991815,2cyclictest0-21swapper/309:55:003
8987699181,14cyclictest0-21swapper/312:18:013
8987699181,14cyclictest0-21swapper/311:11:493
8987699181,14cyclictest0-21swapper/310:37:143
5308792181,3sleep10-21swapper/112:29:511
2972232180,4sleep2297252-21apt-get10:00:122
2267882181,3sleep00-21swapper/009:14:400
8987699171,13cyclictest0-21swapper/312:06:553
8987699171,13cyclictest0-21swapper/311:05:203
89876991711,3cyclictest0-21swapper/310:50:283
8987699171,13cyclictest0-21swapper/310:25:123
8987699171,13cyclictest0-21swapper/310:21:043
8987699171,13cyclictest0-21swapper/308:15:133
8987699171,12cyclictest0-21swapper/309:25:073
8987699170,14cyclictest0-21swapper/310:48:123
8987699170,14cyclictest0-21swapper/309:55:313
8987699170,14cyclictest0-21swapper/307:25:113
8987699170,13cyclictest0-21swapper/310:31:473
4881322171,11sleep0487958-21/usr/sbin/munin12:05:110
3182622171,3sleep10-21swapper/110:11:371
2879842171,4sleep1219194-21diskmemload09:53:221
2795872171,2sleep20-21swapper/209:47:252
8987699168,6cyclictest0-21swapper/309:45:163
8987699167,6cyclictest197702-21taskset08:46:053
8987699161,3cyclictest0-21swapper/310:02:433
89876991612,3cyclictest0-21swapper/311:00:293
89876991610,3cyclictest0-21swapper/308:50:173
5442462161,8sleep2544298-21ntpq12:35:212
4865742161,2sleep20-21swapper/212:02:462
4630952161,2sleep20-21swapper/211:46:502
4404482161,8sleep0219194-21diskmemload11:33:100
3866242161,3sleep2219194-21diskmemload10:58:042
3806422161,4sleep0380909-21df10:55:140
1874462161,4sleep2187469-21munin-run08:40:002
8987699151,11cyclictest1398122sleep307:55:033
8987699151,11cyclictest1398122sleep307:55:023
8987699151,11cyclictest0-21swapper/308:25:133
8987699150,12cyclictest0-21swapper/308:00:133
848112152,8sleep30-21swapper/307:05:103
5245262151,6sleep2526322-21grep12:25:192
3783832151,3sleep0219194-21diskmemload10:51:360
2974132152,6sleep0297542-21diskstats10:00:130
2654142151,9sleep1265425-21ssh09:39:341
89876991412,1cyclictest0-21swapper/308:06:193
8987699140,11cyclictest0-21swapper/307:40:133
5355822141,3sleep1535638-21ntpq12:30:221
4487652142,2sleep0171rcu_preempt11:39:230
3928022141,2sleep1219194-21diskmemload11:00:431
3325592141,8sleep2332809-21sed10:20:282
2794992141,7sleep1219194-21diskmemload09:47:161
1012322148,3sleep0101370-21nfs4_client07:15:220
5320242132,6sleep0532710-21kthreadcore12:30:170
4964562131,8sleep1496508-21fschecks_count12:10:141
4615412131,5sleep0461593-21ntpq11:45:240
4180942131,3sleep2418104-21ssh11:19:002
3641242131,5sleep10-21swapper/110:43:281
1880052131,8sleep0188030-21fschecks_count08:40:140
1388672133,5sleep0138883-21wc07:50:250
1243152131,8sleep2124404-21grep07:40:172
5186802121,5sleep0219194-21diskmemload12:20:440
4936682121,4sleep2219194-21diskmemload12:06:502
4800812121,6sleep1480087-21ssh11:59:481
4010072123,3sleep2219194-21diskmemload11:06:262
3935942121,7sleep2219194-21diskmemload11:02:042
3923112121,4sleep0219194-21diskmemload11:00:270
3708122121,7sleep0370814-21ssh10:46:440
3570412121,7sleep1357045-21sh10:39:181
3411102121,3sleep2341120-21ssh10:27:182
3410202121,7sleep10-21swapper/110:27:081
3207182121,3sleep1320758-21awk10:15:141
3050032121,7sleep0305017-21ssh10:05:070
2486882121,2sleep0248698-21ssh09:27:230
2418692121,2sleep20-21swapper/209:23:522
2343012121,7sleep1234307-21ssh09:19:071
1705472121,7sleep2170820-21grep08:20:282
1448542121,5sleep2144882-21munin-run08:00:002
1448542121,5sleep2144882-21munin-run07:59:592
1384382121,7sleep2138543-21ntpq07:50:182
848092110,2sleep10-21swapper/107:05:081
5484722110,1sleep00-21swapper/012:40:000
5191142111,3sleep20-21swapper/212:21:182
5177122111,4sleep1517772-21ntpq12:20:201
5131312111,4sleep1219194-21diskmemload12:19:561
5048602114,2sleep014-21ksoftirqd/012:15:140
5042112111,3sleep0219194-21diskmemload12:14:420
5015992111,3sleep20-21swapper/212:11:162
4940152111,4sleep1219194-21diskmemload12:07:211
4876162111,5sleep0487628-21apt12:05:000
4726352111,5sleep2472667-21munin-run11:54:592
4646962111,4sleep1219194-21diskmemload11:49:301
4554272111,3sleep10-21swapper/111:42:161
4550502111,4sleep0219194-21diskmemload11:41:380
4264092111,4sleep1426465-21munin-run11:25:011
4197812111,5sleep2421529-21tr11:20:172
4093312111,5sleep1219194-21diskmemload11:12:341
4086882111,4sleep0219194-21diskmemload11:11:250
4029342110,7sleep0403052-21date11:10:000
3866532110,1sleep10-21swapper/110:58:071
3781992111,3sleep10-21swapper/110:51:171
3637132111,3sleep00-21swapper/010:42:430
3555222111,4sleep0219194-21diskmemload10:36:100
3554192111,4sleep2219194-21diskmemload10:35:592
3471032111,6sleep0347257-21ssh10:30:250
3111082111,6sleep2311116-21sh10:07:342
3055642111,3sleep10-21swapper/110:05:161
2965752111,3sleep10-21swapper/109:59:311
2788362111,2sleep00-21swapper/009:45:420
2711552111,4sleep0219194-21diskmemload09:41:020
2660292111,3sleep10-21swapper/109:40:111
2634862111,4sleep0219194-21diskmemload09:36:160
2378882111,4sleep1239197-21memory09:20:181
2352682111,4sleep0235527-21df09:20:130
2326202111,4sleep2219194-21diskmemload09:16:112
1772982111,6sleep1177579-21cut08:30:161
1676542111,6sleep1169493-21kthreadcore08:20:191
1544112111,5sleep1154465-21ntpq08:05:191
1341192110,1sleep20-21swapper/207:47:542
1236772111,4sleep1123568-31find07:39:191
1172752111,5sleep0117323-21ntpq07:30:210
5477622101,2sleep10-21swapper/112:39:041
5226662101,3sleep00-21swapper/012:25:130
5114992101,3sleep2219194-21diskmemload12:17:482
4872982101,3sleep1219194-21diskmemload12:04:251
4788642100,2sleep20-21swapper/211:57:252
4771582101,4sleep0477217-21ntpq11:55:220
4717352100,4sleep0219194-21diskmemload11:53:140
4654032101,3sleep10-21swapper/111:50:131
4498602101,5sleep2450031-21sed11:40:172
4474162101,2sleep20-21swapper/211:37:042
4472702101,2sleep10-21swapper/111:36:511
4392532101,3sleep2219194-21diskmemload11:31:032
4313022101,3sleep2219194-21diskmemload11:25:292
4245792100,1sleep00-21swapper/011:21:430
4164462101,2sleep00-21swapper/011:16:010
4111972101,3sleep1410695-21apt-get11:15:021
4038682101,5sleep2403965-21kthreadcore11:10:192
3729822101,3sleep2219194-21diskmemload10:50:112
3710672100,3sleep20-21swapper/210:47:102
3645732101,2sleep20-21swapper/210:44:172
3484172101,3sleep1219194-21diskmemload10:32:021
3322052101,4sleep0332266-21ld10:20:220
3294842101,5sleep1331231-21kthreadcore10:20:191
3270602101,4sleep2327070-21ssh10:18:482
3212602101,5sleep0322926-21grep10:15:160
3183492101,3sleep0219194-21diskmemload10:11:490
2975452101,4sleep1297762-21ssh10:00:141
2965662100,1sleep00-21swapper/009:59:300
2958622101,2sleep20-21swapper/209:58:242
2871612101,2sleep00-21swapper/009:51:560
2861552101,3sleep2219194-21diskmemload09:50:262
2730012101,3sleep2219194-21diskmemload09:44:052
2574722100,3sleep20-21swapper/209:34:222
2569872101,2sleep10-21swapper/109:33:261
2559882101,2sleep00-21swapper/009:31:360
2477932101,2sleep10-21swapper/109:25:401
2262482101,3sleep2219194-21diskmemload09:13:472
2253512101,4sleep1219194-21diskmemload09:11:371
2189022101,4sleep1218945-21apt-config09:07:081
2126962101,3sleep1212809-21ntpq09:00:191
2023142101,4sleep0202368-21ntpq08:50:200
2021232101,4sleep2202264-21ntpq08:50:182
1941882100,6sleep1195986-21grep08:45:201
1935552100,6sleep0194186-21cut08:45:180
1877302101,3sleep20-21swapper/208:40:112
1725952101,5sleep20-21swapper/208:25:172
1503862101,4sleep0150405-21apt-get08:05:100
1491242101,4sleep2149173-21ntpq08:00:202
1438902101,4sleep1143944-21ntpq07:55:221
1438902101,4sleep1143944-21ntpq07:55:211
1280492101,4sleep0128088-21ntpq07:40:230
1235882100,1sleep00-21swapper/007:38:420
1086552101,4sleep1110411-21grep07:25:171
97890291,4sleep198211-21grep07:15:191
89592290,1sleep20-21swapper/207:07:492
537532291,2sleep20-21swapper/212:32:082
441420291,2sleep10-21swapper/111:34:511
431674290,1sleep10-21swapper/111:25:411
400721290,3sleep1219194-21diskmemload11:05:571
370930290,3sleep1219194-21diskmemload10:46:571
339925290,5sleep0340175-21/usr/sbin/munin10:25:270
318453291,2sleep2219194-21diskmemload10:11:582
264649291,4sleep2264657-21ssh09:38:162
231376291,4sleep0231481-21ld09:15:210
182457291,3sleep1182664-21diskstats08:35:141
180866291,3sleep2180921-21ld08:30:212
166205290,1sleep00-21swapper/008:20:130
161416291,4sleep0161516-21sed08:15:170
159952291,5sleep2160023-21find08:10:262
137589291,4sleep1138436-21grep07:50:171
128091291,4sleep1128145-21ntpq07:40:241
107968290,3sleep0107987-21apt-get07:25:120
102672291,2sleep20-21swapper/207:20:132
213819280,1sleep20-21swapper/209:04:532
204125281,3sleep00-21swapper/008:55:160
203537280,2sleep20-21swapper/208:55:062
159599280,4sleep0159648-21sed08:10:200
140370281,3sleep0140810-21kthreadcore07:55:170
140370281,3sleep0140810-21kthreadcore07:55:170
102989280,3sleep1103205-21if_enp1s007:20:151
90643271,3sleep00-21swapper/007:10:170
85610271,2sleep00-21swapper/007:05:150
219104270,2sleep20-21swapper/209:09:052
214542270,3sleep0214658-21irqstats09:05:150
211792270,3sleep00-21swapper/009:00:180
182395270,3sleep0182455-21tr08:35:130
171463270,2sleep10-21swapper/108:25:121
161519271,2sleep10-21swapper/108:15:181
130852270,3sleep1132687-21tr07:45:161
129890271,2sleep00-21swapper/007:45:150
118166270,1sleep10-21swapper/107:33:051
98212261,2sleep20-21swapper/207:15:192
94772260,4sleep20-21swapper/207:10:512
94173261,2sleep10-21swapper/107:10:221
192579260,1sleep10-21swapper/108:43:001
181499260,1sleep00-21swapper/008:30:430
176298260,1sleep00-21swapper/008:27:160
164976260,2sleep20-21swapper/208:15:222
156225260,2sleep10-21swapper/108:10:171
155065260,1sleep20-21swapper/208:05:482
149238261,2sleep00-21swapper/008:00:220
145589260,1sleep10-21swapper/108:00:151
102947261,2sleep00-21swapper/007:20:140
207839250,2sleep10-21swapper/108:55:251
192742251,1sleep20-21swapper/208:45:012
113371250,1sleep20-21swapper/207:30:142
107707250,2sleep20-21swapper/207:25:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional