You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 09:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Wed Aug 27, 2025 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911090,47ptp4l0-21swapper/119:06:401
1402327152,8sleep60-21swapper/619:05:238
1589126755,10sleep50-21swapper/519:08:347
1581926554,9sleep00-21swapper/019:07:300
164352630,0sleep60-21swapper/600:00:198
1581726352,9sleep110-21swapper/1119:07:293
1581326251,9sleep70-21swapper/719:07:259
296002610,0sleep757-21ksoftirqd/719:25:149
1597226049,9sleep40-21swapper/419:09:456
1589825949,8sleep100-21swapper/1019:08:402
72872560,0sleep30-21swapper/321:00:205
19542560,0sleep110-21swapper/1123:00:183
282692490,0sleep90-21swapper/920:45:2511
1639899400,3cyclictest0-21swapper/1021:49:322
1637099243,16cyclictest0-21swapper/420:40:016
1637099242,17cyclictest0-21swapper/421:40:006
1637099222,15cyclictest0-21swapper/421:50:006
1637099212,14cyclictest0-21swapper/422:05:006
1637099211,15cyclictest0-21swapper/423:20:006
1637099211,14cyclictest0-21swapper/421:00:016
1636599182,10cyclictest0-21swapper/323:54:595
1636199184,9cyclictest0-21swapper/221:15:014
275991170,0ptp4l0-21swapper/123:30:151
1638899174,9cyclictest0-21swapper/822:35:0010
1636599171,2cyclictest0-21swapper/323:25:215
1636199170,16cyclictest0-21swapper/200:25:194
286532160,0sleep60-21swapper/620:05:178
275991160,1ptp4l21-21ksoftirqd/120:40:361
275991160,0ptp4l0-21swapper/121:15:221
275991160,0ptp4l0-21swapper/120:55:211
275991160,0ptp4l0-21swapper/100:00:201
275792160,0sleep30-21swapper/300:15:185
1639399163,7cyclictest0-21swapper/900:40:0011
1638899164,7cyclictest0-21swapper/821:25:0010
1636599162,9cyclictest0-21swapper/322:05:005
1636599162,8cyclictest0-21swapper/319:25:015
1636199162,9cyclictest0-21swapper/220:40:244
156812167,7sleep80-21swapper/819:05:3710
321932150,0sleep50-21swapper/520:10:177
252902150,0sleep70-21swapper/721:25:149
1640299153,7cyclictest0-21swapper/1120:50:013
1639399152,7cyclictest0-21swapper/920:35:0111
1638899153,7cyclictest0-21swapper/821:40:0010
1638899153,7cyclictest0-21swapper/820:30:0110
1638899152,8cyclictest0-21swapper/822:50:0010
1638899152,8cyclictest0-21swapper/819:25:0010
1636599153,8cyclictest0-21swapper/319:55:005
1636199153,8cyclictest0-21swapper/200:15:004
1635199152,8cyclictest0-21swapper/020:15:010
275991140,1ptp4l21-21ksoftirqd/120:29:371
275991140,1ptp4l21-21ksoftirqd/119:20:201
275991140,1ptp4l21-21ksoftirqd/100:10:251
254212140,0sleep80-21swapper/820:00:1810
1640299142,7cyclictest0-21swapper/1120:10:273
1639399145,6cyclictest13630-21kworker/u24:121:00:0011
1639399142,7cyclictest0-21swapper/920:00:0011
1637099142,7cyclictest0-21swapper/422:40:306
1636599143,7cyclictest0-21swapper/300:25:255
1636599142,7cyclictest0-21swapper/323:05:285
1636599141,8cyclictest0-21swapper/323:14:595
1636199143,7cyclictest0-21swapper/220:10:014
1635199143,7cyclictest0-21swapper/023:29:590
1635199143,7cyclictest0-21swapper/000:20:000
1635199143,7cyclictest0-21swapper/000:00:000
1635199141,9cyclictest13630-21kworker/u24:100:30:010
275991130,1ptp4l21-21ksoftirqd/123:37:361
275991130,1ptp4l21-21ksoftirqd/120:30:401
275991130,1ptp4l21-21ksoftirqd/119:10:131
275991130,1ptp4l21-21ksoftirqd/100:07:281
275991130,0ptp4l0-21swapper/122:05:171
275991130,0ptp4l0-21swapper/119:55:231
1640299133,6cyclictest0-21swapper/1123:25:003
1640299132,7cyclictest0-21swapper/1123:55:003
1640299132,6cyclictest0-21swapper/1122:15:003
1640299132,6cyclictest0-21swapper/1120:40:003
1639399133,7cyclictest0-21swapper/921:55:0011
1639399133,6cyclictest0-21swapper/921:15:2711
1639399132,7cyclictest0-21swapper/922:50:2511
1639399131,7cyclictest0-21swapper/900:20:0011
1638899132,7cyclictest0-21swapper/822:20:0010
1636599132,7cyclictest0-21swapper/322:30:275
1636199133,7cyclictest0-21swapper/219:45:014
1636199133,6cyclictest0-21swapper/223:30:014
1635199134,6cyclictest0-21swapper/023:10:000
1635199132,7cyclictest0-21swapper/019:45:260
1635199132,6cyclictest0-21swapper/019:20:010
275991120,3ptp4l19827-21netstat22:40:241
275991120,1ptp4l21-21ksoftirqd/121:30:191
275991120,1ptp4l21-21ksoftirqd/120:38:161
275991120,0ptp4l29771-21strings20:05:221
275991120,0ptp4l0-21swapper/123:54:511
275991120,0ptp4l0-21swapper/123:47:261
275991120,0ptp4l0-21swapper/123:20:261
275991120,0ptp4l0-21swapper/119:35:151
275991120,0ptp4l0-21swapper/119:25:091
275991120,0ptp4l0-21swapper/100:20:161
1640299122,7cyclictest0-21swapper/1119:35:013
1639899120,3cyclictest4821-21grep21:40:182
1638399120,7cyclictest0-21swapper/721:49:599
1637999121,6cyclictest0-21swapper/619:25:018
1637499122,6cyclictest0-21swapper/522:25:007
1637499122,6cyclictest0-21swapper/520:50:007
1637099122,6cyclictest0-21swapper/422:10:236
1636599122,6cyclictest0-21swapper/319:10:005
1635199122,7cyclictest0-21swapper/000:10:010
275991110,4ptp4l21-21ksoftirqd/122:55:001
275991110,4ptp4l21-21ksoftirqd/120:50:221
275991110,2ptp4l20807-21df23:25:171
275991110,2ptp4l16426-21tune2fs21:55:171
275991110,2ptp4l0-21swapper/122:45:191
275991110,1ptp4l21-21ksoftirqd/120:14:381
275991110,1ptp4l21-21ksoftirqd/100:25:181
275991110,1ptp4l21-21ksoftirqd/100:15:121
275991110,0ptp4l0-21swapper/123:15:221
275991110,0ptp4l0-21swapper/122:36:011
275991110,0ptp4l0-21swapper/122:25:271
275991110,0ptp4l0-21swapper/121:10:161
275991110,0ptp4l0-21swapper/119:31:211
1639899111,6cyclictest0-21swapper/1019:40:002
1639399110,10cyclictest31385-21unixbench_singl21:30:2511
1638899118,2cyclictest0-21swapper/823:35:2010
1638899118,2cyclictest0-21swapper/819:25:2510
1637499112,6cyclictest0-21swapper/519:40:257
1636599112,7cyclictest0-21swapper/320:15:005
1636599110,10cyclictest0-21swapper/321:20:145
1635199118,2cyclictest0-21swapper/019:15:010
275991100,4ptp4l21-21ksoftirqd/120:00:351
275991100,1ptp4l9274-21/usr/sbin/munin00:35:191
275991100,1ptp4l31883-21fw_forwarded_lo22:15:171
275991100,1ptp4l27701-21df22:10:141
275991100,1ptp4l25428-21diskstats21:25:161
275991100,1ptp4l20060-21df_inode22:00:161
275991100,1ptp4l1020-21runrttasks21:05:161
275991100,0ptp4l0-21swapper/121:00:361
275991100,0ptp4l0-21swapper/119:45:201
1640299107,2cyclictest9254-21cat23:10:193
1640299102,5cyclictest0-21swapper/1123:40:293
1639399102,5cyclictest0-21swapper/921:25:0011
1639399101,5cyclictest0-21swapper/922:25:0011
1639399100,9cyclictest2232-21df00:25:1511
1638899100,5cyclictest0-21swapper/823:44:5910
1638399107,2cyclictest19927-21turbostat19:15:019
1638399100,9cyclictest17968-21latency_hist00:05:009
1638399100,5cyclictest0-21swapper/723:34:559
1637999108,1cyclictest0-21swapper/620:35:438
1637999106,3cyclictest12896-21netstat23:55:218
1637999102,5cyclictest0-21swapper/621:45:238
1637999100,9cyclictest0-21swapper/621:10:268
1637499102,5cyclictest0-21swapper/523:50:017
1637499101,6cyclictest13630-21kworker/u24:123:40:017
1636599107,2cyclictest1849-21fw_conntrack19:30:165
1636599103,6cyclictest0-21swapper/300:05:265
1636599100,9cyclictest0-21swapper/300:00:195
1636199100,3cyclictest8880-21latency20:20:204
1635699106,3cyclictest7382-21timerandwakeup123:05:281
1635199106,3cyclictest26137-21/usr/sbin/munin00:15:180
126922101,6sleep90-21swapper/919:05:1411
27599190,1ptp4l21-21ksoftirqd/122:30:161
27599190,1ptp4l21-21ksoftirqd/122:20:071
27599190,1ptp4l21-21ksoftirqd/121:45:121
27599190,1ptp4l21-21ksoftirqd/121:40:001
27599190,1ptp4l21-21ksoftirqd/120:45:191
27599190,1ptp4l21-21ksoftirqd/120:20:141
27599190,1ptp4l20368-21munin-node19:15:011
27599190,0ptp4l0-21swapper/123:55:011
27599190,0ptp4l0-21swapper/123:40:181
27599190,0ptp4l0-21swapper/123:10:211
27599190,0ptp4l0-21swapper/123:00:131
27599190,0ptp4l0-21swapper/122:55:131
27599190,0ptp4l0-21swapper/121:50:181
27599190,0ptp4l0-21swapper/121:35:181
27599190,0ptp4l0-21swapper/121:20:131
27599190,0ptp4l0-21swapper/120:15:151
27599190,0ptp4l0-21swapper/119:50:151
27599190,0ptp4l0-21swapper/119:40:141
27599190,0ptp4l0-21swapper/100:30:171
164029998,1cyclictest0-21swapper/1122:30:233
164029992,4cyclictest0-21swapper/1122:45:153
164029990,9cyclictest0-21swapper/1122:36:003
164029990,1cyclictest0-21swapper/1122:05:263
163989990,2cyclictest9325-21diskstats23:10:192
163939990,8cyclictest1020-21runrttasks19:28:3911
163939990,8cyclictest0-21swapper/921:15:0111
163939990,8cyclictest0-21swapper/920:50:1811
163889997,1cyclictest0-21swapper/820:35:1910
163889992,4cyclictest0-21swapper/819:35:3710
163839990,8cyclictest0-21swapper/723:54:559
163799998,1cyclictest0-21swapper/623:00:008
163799998,1cyclictest0-21swapper/622:50:208
163799992,4cyclictest0-21swapper/620:50:018
163799990,8cyclictest10275-21awk23:55:008
163799990,1cyclictest0-21swapper/620:15:178
163799990,1cyclictest0-21swapper/600:25:128
163749996,2cyclictest27642-21taskset21:26:107
163749996,2cyclictest21545-21rt-features22:00:247
163749991,5cyclictest13630-21kworker/u24:100:35:017
163709990,7cyclictest0-21swapper/419:35:006
163659998,1cyclictest0-21swapper/322:55:245
163659998,1cyclictest0-21swapper/320:35:255
163659990,8cyclictest0-21swapper/319:15:185
163619998,1cyclictest0-21swapper/222:55:144
163619990,3cyclictest1061-21/usr/sbin/munin20:10:224
163619990,2cyclictest8557-21df_inode21:45:134
163619990,2cyclictest19608-21sed22:40:244
163619990,2cyclictest14117-21sed21:50:294
163619990,1cyclictest0-21swapper/223:25:274
163619990,1cyclictest0-21swapper/200:35:244
163619990,1cyclictest0-21swapper/200:30:224
163519997,1cyclictest0-21swapper/021:40:000
164029987,1cyclictest0-21swapper/1120:00:003
164029980,8cyclictest12611-21diskmemload22:04:403
164029980,8cyclictest0-21swapper/1100:05:143
164029980,1cyclictest0-21swapper/1121:35:263
164029980,1cyclictest0-21swapper/1121:00:233
164029980,1cyclictest0-21swapper/1100:20:123
163989987,1cyclictest0-21swapper/1019:20:252
163989987,1cyclictest0-21swapper/1019:15:012
163989980,8cyclictest0-21swapper/1023:15:172
163989980,1cyclictest0-21swapper/1023:55:142
163989980,1cyclictest0-21swapper/1021:20:172
163989980,1cyclictest0-21swapper/1020:00:192
163939987,1cyclictest0-21swapper/923:55:1611
163939987,1cyclictest0-21swapper/921:05:1611
163939982,4cyclictest0-21swapper/922:35:2411
163939980,1cyclictest0-21swapper/923:15:2311
163939980,1cyclictest0-21swapper/920:00:1711
163889987,1cyclictest0-21swapper/823:15:2410
163889987,1cyclictest0-21swapper/821:15:2810
163889987,1cyclictest0-21swapper/800:00:1410
163889982,5cyclictest0-21swapper/821:05:2610
163889980,8cyclictest0-21swapper/823:58:3610
163889980,8cyclictest0-21swapper/821:40:1910
163889980,8cyclictest0-21swapper/800:10:0010
163889980,7cyclictest5704-21munin-run22:24:5910
163889980,1cyclictest0-21swapper/823:47:1310
163889980,1cyclictest0-21swapper/821:11:3610
163839987,1cyclictest0-21swapper/721:40:289
163839987,1cyclictest0-21swapper/719:15:239
163839986,1cyclictest0-21swapper/720:39:569
163839985,2cyclictest0-21swapper/720:30:159
163839980,7cyclictest0-21swapper/721:06:389
163839980,1cyclictest12611-21diskmemload23:39:289
163839980,1cyclictest0-21swapper/722:15:279
163839980,1cyclictest0-21swapper/720:40:189
163839980,1cyclictest0-21swapper/719:20:159
163799987,1cyclictest0-21swapper/619:35:278
163799987,1cyclictest0-21swapper/619:10:148
163799985,2cyclictest21028-21munin-run22:45:008
163799982,4cyclictest0-21swapper/600:30:018
163799982,3cyclictest0-21swapper/619:55:018
163799981,4cyclictest0-21swapper/623:20:008
163799980,1cyclictest0-21swapper/621:25:188
163749981,5cyclictest0-21swapper/523:25:007
163749980,8cyclictest0-21swapper/523:10:267
163749980,8cyclictest0-21swapper/500:10:187
163749980,1cyclictest0-21swapper/522:10:147
163749980,1cyclictest0-21swapper/500:15:207
163709987,1cyclictest381rcuc/423:05:196
163709986,1cyclictest12614-21turbostat21:10:016
163709986,1cyclictest0-21swapper/423:40:296
163709985,2cyclictest0-21swapper/423:30:156
163709980,8cyclictest381rcuc/423:27:116
163709980,8cyclictest381rcuc/422:17:376
163709980,8cyclictest0-21swapper/421:15:246
163709980,7cyclictest0-21swapper/419:20:156
163709980,3cyclictest19137-21sed23:20:316
163709980,1cyclictest0-21swapper/400:00:166
163659987,1cyclictest0-21swapper/321:05:205
163659980,8cyclictest0-21swapper/322:05:205
163659980,1cyclictest0-21swapper/323:00:145
163659980,1cyclictest0-21swapper/320:25:215
163659980,1cyclictest0-21swapper/319:55:225
163659980,1cyclictest0-21swapper/319:45:175
163659980,1cyclictest0-21swapper/319:25:185
163619987,1cyclictest0-21swapper/221:55:134
163619987,1cyclictest0-21swapper/221:20:164
163619987,1cyclictest0-21swapper/221:15:144
163619987,1cyclictest0-21swapper/220:35:324
163619987,1cyclictest0-21swapper/200:00:124
163619986,1cyclictest0-21swapper/223:55:004
163619980,8cyclictest0-21swapper/223:35:174
163619980,8cyclictest0-21swapper/223:10:254
163619980,8cyclictest0-21swapper/222:45:014
163619980,8cyclictest0-21swapper/221:25:154
163619980,8cyclictest0-21swapper/220:25:164
163619980,7cyclictest6525-21/usr/sbin/munin23:05:224
163619980,7cyclictest3669-21awk22:20:214
163619980,7cyclictest0-21swapper/221:00:214
163619980,7cyclictest0-21swapper/220:50:474
163619980,2cyclictest4016-21cut23:45:204
163619980,2cyclictest19153-21users23:20:314
163619980,2cyclictest15788-21/usr/sbin/munin22:35:204
163619980,2cyclictest10097-21df21:05:154
163619980,2cyclictest0-21swapper/223:40:234
163619980,1cyclictest0-21swapper/223:15:214
163619980,1cyclictest0-21swapper/222:05:234
163619980,1cyclictest0-21swapper/220:45:194
163619980,1cyclictest0-21swapper/220:15:274
163619980,1cyclictest0-21swapper/219:25:184
163519986,1cyclictest3969-21pool00:01:470
163519983,4cyclictest0-21swapper/023:20:230
163519980,8cyclictest0-21swapper/021:10:010
163519980,1cyclictest0-21swapper/022:25:200
163519980,1cyclictest0-21swapper/019:20:230
15676282,4sleep30-21swapper/319:05:325
27609170,6phc2sys0-21swapper/223:00:274
27609170,1phc2sys0-21swapper/219:06:444
164029976,1cyclictest0-21swapper/1123:45:523
164029976,1cyclictest0-21swapper/1123:30:253
164029976,1cyclictest0-21swapper/1122:50:183
164029976,1cyclictest0-21swapper/1122:44:183
164029976,1cyclictest0-21swapper/1122:20:433
164029976,1cyclictest0-21swapper/1121:45:443
164029976,1cyclictest0-21swapper/1121:44:033
164029976,1cyclictest0-21swapper/1121:25:023
164029976,1cyclictest0-21swapper/1120:07:153
164029976,1cyclictest0-21swapper/1120:00:243
164029976,1cyclictest0-21swapper/1119:35:273
164029976,1cyclictest0-21swapper/1119:17:513
164029976,1cyclictest0-21swapper/1119:11:383
164029974,2cyclictest5522-21sensors_fan20:15:253
164029974,2cyclictest1020-21runrttasks23:55:433
164029974,2cyclictest1020-21runrttasks23:17:043
164029974,2cyclictest1020-21runrttasks20:42:343
164029971,3cyclictest0-21swapper/1100:36:513
164029970,7cyclictest0-21swapper/1123:25:233
164029970,7cyclictest0-21swapper/1122:55:513
164029970,7cyclictest0-21swapper/1121:57:003
164029970,7cyclictest0-21swapper/1121:30:153
164029970,7cyclictest0-21swapper/1121:18:543
164029970,7cyclictest0-21swapper/1121:10:233
164029970,7cyclictest0-21swapper/1119:45:423
164029970,7cyclictest0-21swapper/1100:19:183
164029970,7cyclictest0-21swapper/1100:10:163
164029970,1cyclictest0-21swapper/1123:36:033
164029970,1cyclictest0-21swapper/1122:26:193
164029970,1cyclictest0-21swapper/1122:15:223
164029970,1cyclictest0-21swapper/1121:50:363
164029970,1cyclictest0-21swapper/1121:23:083
164029970,1cyclictest0-21swapper/1121:06:093
164029970,1cyclictest0-21swapper/1120:56:103
164029970,1cyclictest0-21swapper/1120:51:343
164029970,1cyclictest0-21swapper/1120:30:373
164029970,1cyclictest0-21swapper/1120:26:143
164029970,1cyclictest0-21swapper/1120:21:133
164029970,1cyclictest0-21swapper/1119:40:093
164029970,1cyclictest0-21swapper/1119:25:573
164029970,1cyclictest0-21swapper/1119:20:063
164029970,1cyclictest0-21swapper/1100:30:333
164029970,1cyclictest0-21swapper/1100:26:383
164029970,1cyclictest0-21swapper/1100:01:353
163989976,1cyclictest0-21swapper/1023:50:202
163989976,1cyclictest0-21swapper/1023:35:362
163989976,1cyclictest0-21swapper/1023:30:342
163989976,1cyclictest0-21swapper/1022:55:352
163989976,1cyclictest0-21swapper/1022:27:482
163989976,1cyclictest0-21swapper/1022:05:292
163989976,1cyclictest0-21swapper/1021:57:562
163989976,1cyclictest0-21swapper/1021:07:232
163989976,1cyclictest0-21swapper/1020:55:432
163989976,1cyclictest0-21swapper/1020:42:182
163989976,1cyclictest0-21swapper/1020:26:462
163989976,1cyclictest0-21swapper/1020:21:592
163989976,1cyclictest0-21swapper/1020:05:162
163989976,1cyclictest0-21swapper/1019:55:372
163989976,1cyclictest0-21swapper/1019:50:372
163989976,1cyclictest0-21swapper/1019:10:372
163989976,1cyclictest0-21swapper/1000:31:562
163989976,1cyclictest0-21swapper/1000:15:392
163989976,1cyclictest0-21swapper/1000:03:262
163989973,3cyclictest5051-21sed22:20:282
163989973,3cyclictest31974-21cat20:50:202
163989971,4cyclictest0-21swapper/1022:15:582
163989970,7cyclictest0-21swapper/1023:20:282
163989970,7cyclictest0-21swapper/1022:35:502
163989970,7cyclictest0-21swapper/1022:30:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional