You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 10:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Sat Jul 12, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
51802214398,36sleep10-21swapper/119:08:541
51663214198,34sleep30-21swapper/319:07:083
516152135106,21sleep00-21swapper/019:06:320
517682132109,15sleep20-21swapper/219:08:272
52012994425,17cyclictest471585-21awk00:15:013
52012993725,8cyclictest243688-21grep21:50:293
5201299362,23cyclictest318066-21ssh22:37:503
52012993228,3cyclictest703-21dbus-daemon00:20:003
52012992924,3cyclictest279878-21cpuspeed_turbos22:15:153
52012992823,3cyclictest231717-21kthreadcore21:45:163
52012992722,3cyclictest494341-21fschecks_time00:30:153
52012992622,2cyclictest83916-21sh19:39:593
52012992622,2cyclictest295489-21perf22:25:013
52012992622,2cyclictest0-21swapper/320:40:163
52012992515,8cyclictest0-21swapper/323:10:253
52012992420,2cyclictest104855-21gltestperf20:00:153
52012992411,9cyclictest305712-21cut22:30:173
52012992411,9cyclictest259312-21ntpq22:00:223
52012992320,2cyclictest129611-21turbostat.cron20:25:013
52012992319,2cyclictest943-21runrttasks23:26:193
52012992319,2cyclictest0-21swapper/323:30:183
52012992319,2cyclictest0-21swapper/320:55:253
52012992318,3cyclictest73967-21cron19:30:003
5201299231,7cyclictest186739-21/usr/sbin/munin21:15:223
52012992312,9cyclictest0-21swapper/300:00:213
5201299231,19cyclictest460691-21/usr/sbin/munin00:05:193
52012992310,7cyclictest381035-21ssh23:15:273
52012992219,2cyclictest367869-21apt-key23:10:003
52012992219,2cyclictest222800-21systemd-run21:40:003
52012992217,3cyclictest1056-21cron20:15:003
52012992214,6cyclictest174511-21kworker/u16:3+rpciod21:40:113
5201299221,18cyclictest263956-21tune2fs22:05:153
2945332220,2sleep00-21swapper/022:23:380
2137842221,7sleep2174462-21diskmemload21:33:392
5201299219,8cyclictest412207-21cut23:35:223
5201299213,15cyclictest352471-21/usr/sbin/munin23:00:153
52012992118,2cyclictest175142-21fschecks_time21:10:143
5201299211,7cyclictest507810-21systemd-run00:40:003
52012992117,3cyclictest384535-21/usr/sbin/munin23:20:223
52012992117,2cyclictest68993-21idleruntime-cro19:24:593
52012992117,2cyclictest0-21swapper/322:55:273
52012992112,6cyclictest150786-21cut20:45:163
52012992112,6cyclictest103075-21seq19:55:213
5201299211,17cyclictest334957-21ssh22:49:013
52012992111,6cyclictest336023-21apt-get22:50:153
5201299211,16cyclictest110260-21/usr/sbin/munin20:05:173
5201299211,15cyclictest437764-21kworker/u16:0+rpciod23:53:273
5201299211,15cyclictest247293-21chrt21:55:163
5201299209,7cyclictest442007-21cat23:55:173
5201299209,7cyclictest211033-21sshd21:30:223
5201299208,7cyclictest736-21gdbus00:25:233
5201299201,7cyclictest52574-21hddtemp_smartct19:10:133
52012992016,2cyclictest430967-21sh23:47:573
52012992016,2cyclictest0-21swapper/319:55:003
52012992012,5cyclictest133832-21find20:25:223
5201299200,18cyclictest326251-21taskset22:42:573
1485352201,5sleep0148579-21ntpq20:40:220
5201299199,7cyclictest143538-21ntpq20:35:203
5201299198,7cyclictest200804-21cut21:25:183
5201299192,13cyclictest945-21snmpd22:25:123
52012991916,2cyclictest275647-21ntp_kernel_pll_22:10:223
52012991914,3cyclictest169012-21unixbench_multi21:00:293
5201299191,15cyclictest422053-21taskset23:41:303
52012991910,7cyclictest84456-21grep19:40:183
4604412190,10sleep00-21swapper/000:05:160
3282332191,13sleep2328411-21tr22:45:142
5201299189,6cyclictest59167-21apt-get19:15:143
52012991814,3cyclictest138483-21needreboot20:30:183
52012991813,3cyclictest119927-21cpuspeed_turbos20:15:123
5201299181,13cyclictest483303-21ntpq00:20:193
5201299180,16cyclictest197483-21sh21:23:213
52012991710,5cyclictest173320-21sort21:05:173
5201299170,14cyclictest89368-21diskstats19:45:173
5201299170,14cyclictest89368-21diskstats19:45:163
3976462170,7sleep10-21swapper/123:26:071
3181312171,2sleep20-21swapper/222:37:562
4787182166,4sleep2421ktimers/200:19:262
4366552162,7sleep1436720-21ntp_states23:50:241
4045662161,8sleep2404626-21ntpq23:30:202
5201299156,6cyclictest154753-21awk20:50:123
5201299151,3cyclictest78298-21/usr/sbin/munin19:30:243
5001972151,2sleep20-21swapper/200:34:022
4315782151,2sleep00-21swapper/023:48:500
1961812151,2sleep10-21swapper/121:21:271
4919402141,7sleep0174462-21diskmemload00:26:110
4285302142,6sleep1428582-21ntpq23:45:181
4242222140,9sleep1424236-21ssh23:44:501
4137682141,7sleep0174462-21diskmemload23:36:070
3285342141,10sleep00-21swapper/022:45:170
3089492141,2sleep20-21swapper/222:31:162
2376262140,2sleep10-21swapper/121:48:181
2371712140,1sleep20-21swapper/221:47:392
2037472141,9sleep2203761-21ssh21:25:462
999482131,8sleep2101465-21cut19:55:202
4552522130,9sleep0455265-21ssh00:03:100
2553412130,2sleep20-21swapper/222:00:122
2449762131,3sleep00-21swapper/021:52:180
1797652131,7sleep0174462-21diskmemload21:11:080
1584982131,8sleep1158552-21ld20:50:211
4999382121,3sleep10-21swapper/100:33:281
4453092121,5sleep20-21swapper/223:55:302
4411652121,7sleep1441615-21grep23:55:161
3916412121,7sleep1391647-21ssh23:24:091
3911532120,8sleep2391167-21ssh23:23:262
3814532121,7sleep2381461-21ssh23:16:062
2781112121,7sleep1278119-21ssh22:13:111
2553872121,7sleep0174462-21diskmemload22:00:150
2472612121,4sleep20-21swapper/221:55:162
2112402121,7sleep1211274-21switchtime21:30:251
2059362126,3sleep01056-21cron21:29:010
1497682122,5sleep2149786-21apt-get20:45:132
1450052121,7sleep2145217-21smartctl20:40:172
841962111,4sleep184215-21apt-get19:40:161
732742111,6sleep273283-21switchtime19:25:262
5063122111,3sleep00-21swapper/000:36:330
5061382111,6sleep1174462-21diskmemload00:36:101
4917852110,7sleep1491795-21ssh00:25:551
4857472111,5sleep10-21swapper/100:23:351
4706252111,4sleep0174462-21diskmemload00:13:040
4694482111,3sleep10-21swapper/100:11:091
4572732111,6sleep2457309-21kthreadcore00:05:152
4382262111,2sleep00-21swapper/023:51:550
4160092111,3sleep20-21swapper/223:39:272
3819542111,6sleep0174462-21diskmemload23:16:480
3735852110,1sleep00-21swapper/023:11:220
3572902110,2sleep10-21swapper/123:01:281
3512582111,3sleep00-21swapper/022:59:280
3491332111,3sleep20-21swapper/222:56:162
3428782110,3sleep00-21swapper/022:53:540
3255542110,1sleep10-21swapper/122:41:561
3198712111,1sleep00-21swapper/022:40:070
3177942111,4sleep00-21swapper/022:37:270
3165042111,3sleep10-21swapper/122:35:351
2848782111,7sleep0284891-21diskmemload22:16:180
2836832110,1sleep20-21swapper/222:15:232
2305052110,7sleep1230515-21ssh21:44:361
2297132111,3sleep20-21swapper/221:43:252
1960942111,3sleep20-21swapper/221:21:202
1951882111,3sleep00-21swapper/021:20:300
1903942111,3sleep10-21swapper/121:19:561
1892802111,3sleep00-21swapper/021:18:140
1812062111,3sleep10-21swapper/121:13:171
979272101,4sleep197973-21ntpq19:50:211
4997472101,2sleep00-21swapper/000:33:030
4862742101,4sleep0174462-21diskmemload00:24:290
4721062101,3sleep0174462-21diskmemload00:15:150
4544762101,3sleep10-21swapper/100:02:031
4530552101,2sleep20-21swapper/200:00:272
4474592101,4sleep0174462-21diskmemload23:58:440
4385712100,7sleep20-21swapper/223:52:242
4230652101,4sleep0174462-21diskmemload23:43:050
4087332101,3sleep10-21swapper/123:35:131
4073432101,4sleep0174462-21diskmemload23:33:300
3979982101,6sleep0398006-21rm23:26:380
3883652101,4sleep0388415-21ntpq23:20:200
3732432101,3sleep20-21swapper/223:10:522
3648842101,3sleep20-21swapper/223:05:512
3648712101,3sleep10-21swapper/123:05:501
3592432101,4sleep2174462-21diskmemload23:04:222
3579962101,3sleep00-21swapper/023:02:320
3335432101,2sleep10-21swapper/122:46:541
3100012101,2sleep10-21swapper/122:32:501
3095792101,3sleep00-21swapper/022:32:130
3025932101,3sleep20-21swapper/222:28:422
2997072101,6sleep0299772-21modprobe22:25:210
2960102100,6sleep1296085-21fschecks_time22:25:131
2937882100,3sleep20-21swapper/222:22:342
2930342100,3sleep10-21swapper/122:21:291
2797182101,4sleep1174462-21diskmemload22:15:101
2783832101,3sleep00-21swapper/022:13:340
2780802101,2sleep20-21swapper/222:13:082
2695892101,2sleep20-21swapper/222:07:222
2675612101,5sleep1267622-21ntpq22:05:201
2531592101,4sleep0174462-21diskmemload21:57:260
2520462101,3sleep10-21swapper/121:55:461
2462272100,2sleep10-21swapper/121:54:071
2368712101,4sleep0174462-21diskmemload21:47:110
2210152101,4sleep2174462-21diskmemload21:37:282
2201702101,3sleep00-21swapper/021:36:140
2142622101,4sleep0174462-21diskmemload21:34:210
2050612101,3sleep10-21swapper/121:27:411
1747512100,3sleep20-21swapper/221:10:052
1384482102,3sleep2138546-21ntpq20:30:192
1234622101,4sleep1123568-21ntpq20:15:181
479919291,4sleep2480465-21ssh00:20:152
471124291,2sleep20-21swapper/200:14:062
461464291,2sleep10-21swapper/100:05:341
431095290,2sleep20-21swapper/223:48:092
420910291,3sleep20-21swapper/223:40:222
401113291,4sleep1401197-21irqstats23:30:151
398156291,2sleep20-21swapper/223:26:522
383382291,2sleep10-21swapper/123:18:531
368399290,3sleep10-21swapper/123:10:141
363998291,4sleep0364055-21ntp_states23:05:210
350154291,2sleep10-21swapper/122:57:491
341449290,3sleep20-21swapper/222:51:442
324723291,3sleep2174462-21diskmemload22:40:462
245269291,3sleep2174462-21diskmemload21:52:432
230455291,3sleep0174462-21diskmemload21:44:310
219913290,2sleep10-21swapper/121:35:511
189790291,2sleep20-21swapper/221:19:002
169741291,4sleep2169896-21diskstats21:05:132
169719290,3sleep10-21swapper/121:05:131
133623291,4sleep1133677-21ntpq20:25:201
109259291,4sleep1109371-21awk20:05:011
69231281,2sleep10-21swapper/119:25:121
55955281,3sleep256001-21ntpq19:10:192
501146281,3sleep2500648-21apt-get00:35:022
491631281,2sleep20-21swapper/200:25:402
471849280,2sleep10-21swapper/100:15:081
342106280,2sleep10-21swapper/122:52:431
269712281,4sleep0269722-21ssh22:07:320
260271280,2sleep10-21swapper/122:00:561
174397280,1sleep00-21swapper/021:09:200
140328283,2sleep0221rcuc/020:35:170
117054280,2sleep10-21swapper/120:10:171
103204281,3sleep1103240-21ntpq19:55:221
99640271,3sleep099851-21sh19:55:170
94180270,2sleep00-21swapper/019:50:130
93844271,3sleep0945-21snmpd19:49:040
93844271,3sleep0945-21snmpd19:49:040
88777270,1sleep20-21swapper/219:43:102
154737270,2sleep00-21swapper/020:50:120
154278270,5sleep00-21swapper/020:47:280
153431270,2sleep10-21swapper/120:45:201
139355270,3sleep1139356-21sshd20:32:221
126340271,3sleep10-21swapper/120:20:201
124867270,2sleep00-21swapper/020:20:150
124349270,1sleep00-21swapper/020:17:050
114567271,3sleep0114611-21systemd-run20:10:000
109797270,2sleep20-21swapper/220:05:132
78904260,5sleep10-21swapper/119:34:551
78875260,2sleep20-21swapper/219:34:312
72882261,2sleep00-21swapper/019:25:190
62748261,2sleep262890-21netstat19:15:212
57102261,2sleep00-21swapper/019:15:010
169134260,1sleep20-21swapper/221:00:562
168453260,2sleep00-21swapper/021:00:220
164392260,1sleep10-21swapper/120:59:071
163441260,2sleep20-21swapper/220:55:192
159174260,1sleep20-21swapper/220:51:192
134849260,1sleep00-21swapper/020:30:080
123569260,2sleep20-21swapper/220:15:192
118438260,2sleep2118531-21strings20:10:192
114849261,2sleep0114959-21cpuspeed_turbos20:10:130
104906260,3sleep2105003-21ps20:00:162
104500260,4sleep00-21swapper/020:00:070
92983250,1sleep10-21swapper/119:45:251
92983250,1sleep10-21swapper/119:45:241
89379251,2sleep20-21swapper/219:45:182
89379251,2sleep20-21swapper/219:45:172
83219251,1sleep00-21swapper/019:35:250
82886250,2sleep282929-21ldconfig19:35:192
79358250,1sleep10-21swapper/119:35:141
75762250,2sleep00-21swapper/019:30:170
68002250,1sleep00-21swapper/019:20:190
64001250,2sleep20-21swapper/219:20:002
58376250,2sleep10-21swapper/119:15:031
159814250,2sleep00-21swapper/020:55:140
149505250,1sleep10-21swapper/120:45:011
144535251,1sleep20-21swapper/220:40:002
132559250,1sleep20-21swapper/220:25:172
128128250,1sleep20-21swapper/220:20:212
96106240,1sleep20-21swapper/219:50:182
84551240,1sleep00-21swapper/019:40:190
64379240,1sleep10-21swapper/119:20:111
52354240,1sleep00-21swapper/019:10:130
52067240,1sleep10-21swapper/119:10:011
164820240,1sleep10-21swapper/121:00:151
139910240,1sleep10-21swapper/120:35:141
129857240,1sleep00-21swapper/020:25:110
109785240,1sleep10-21swapper/120:05:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional