You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 07:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Wed Nov 26, 2025 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
130932950,0sleep90-21swapper/900:22:2311
275991880,63ptp4l0-21swapper/119:06:201
2784226855,10sleep50-21swapper/519:06:197
2778726553,10sleep60-21swapper/619:05:328
2799226352,9sleep110-21swapper/1119:08:273
2798826354,7sleep70-21swapper/719:08:229
2791126352,9sleep100-21swapper/1019:07:142
2781126351,9sleep00-21swapper/019:05:510
2791826050,8sleep40-21swapper/419:07:196
174152540,0sleep90-21swapper/919:35:1511
2843099290,1cyclictest0-21swapper/1000:19:322
2840299252,17cyclictest0-21swapper/419:25:016
2843099240,1cyclictest0-21swapper/1000:30:282
2840299223,15cyclictest0-21swapper/420:35:006
2840299223,14cyclictest0-21swapper/420:25:016
2840299222,15cyclictest0-21swapper/423:39:596
2840299222,15cyclictest0-21swapper/421:05:016
28389992017,2cyclictest20158-21turbostat19:44:551
28398991917,1cyclictest26279-21turbostat20:30:015
2838499190,1cyclictest0-21swapper/000:30:270
2840299181,13cyclictest0-21swapper/422:10:006
69052170,0sleep20-21swapper/221:25:204
2843899173,9cyclictest0-21swapper/1121:05:003
2843099170,7cyclictest0-21swapper/1019:20:222
2840899172,10cyclictest0-21swapper/521:59:597
275991170,0ptp4l0-21swapper/121:50:151
28421991613,2cyclictest0-21swapper/821:00:1810
2839399160,15cyclictest0-21swapper/222:50:154
2838499161,3cyclictest0-21swapper/023:00:180
275991160,0ptp4l0-21swapper/123:00:221
275991160,0ptp4l0-21swapper/122:45:171
275991160,0ptp4l0-21swapper/121:55:001
275991160,0ptp4l0-21swapper/119:25:191
67722150,0sleep10-21swapper/121:25:181
2843899153,7cyclictest0-21swapper/1119:50:013
2843899152,8cyclictest0-21swapper/1123:49:593
2843899150,3cyclictest0-21swapper/1120:45:183
28408991511,3cyclictest0-21swapper/522:20:197
280302157,6sleep80-21swapper/819:09:0010
275991150,1ptp4l21-21ksoftirqd/121:50:001
275991150,1ptp4l21-21ksoftirqd/121:35:191
275991150,0ptp4l0-21swapper/122:10:201
2843899142,8cyclictest0-21swapper/1121:45:003
2843099142,5cyclictest0-21swapper/1019:55:002
2840899141,9cyclictest2188-21kworker/u24:022:59:597
28402991412,2cyclictest0-21swapper/423:05:196
2839899140,1cyclictest0-21swapper/320:45:185
2839399143,7cyclictest0-21swapper/220:30:014
275991140,1ptp4l21-21ksoftirqd/121:40:201
275991140,0ptp4l0-21swapper/123:55:151
275991140,0ptp4l0-21swapper/119:55:011
275991140,0ptp4l0-21swapper/119:15:141
2843899132,7cyclictest0-21swapper/1121:55:003
2843899132,7cyclictest0-21swapper/1119:15:013
2843899132,6cyclictest0-21swapper/1120:10:413
2842199130,12cyclictest0-21swapper/800:24:5510
2840899132,7cyclictest0-21swapper/522:35:007
2840899132,7cyclictest0-21swapper/520:35:017
2840899132,7cyclictest0-21swapper/519:45:017
2840899132,6cyclictest0-21swapper/500:00:297
2840899131,7cyclictest0-21swapper/523:19:597
2840299132,7cyclictest0-21swapper/419:40:246
2839399132,7cyclictest0-21swapper/220:55:014
2838499130,12cyclictest0-21swapper/020:10:000
275991130,4ptp4l0-21swapper/120:55:241
275991130,1ptp4l21-21ksoftirqd/100:07:371
275991130,0ptp4l0-21swapper/123:15:221
275991130,0ptp4l0-21swapper/123:10:241
275991130,0ptp4l0-21swapper/122:40:211
275991130,0ptp4l0-21swapper/122:20:251
318732120,0sleep50-21swapper/519:55:007
2843899122,6cyclictest0-21swapper/1122:50:223
2843899122,6cyclictest0-21swapper/1100:15:253
2843899121,7cyclictest0-21swapper/1122:20:003
2842699120,1cyclictest0-21swapper/919:55:1711
28417991210,1cyclictest0-21swapper/719:44:559
2841299120,7cyclictest0-21swapper/620:35:008
2841299120,7cyclictest0-21swapper/600:10:008
2840299122,6cyclictest0-21swapper/400:20:246
2839399123,6cyclictest0-21swapper/221:00:014
2839399122,7cyclictest0-21swapper/223:45:004
2839399121,7cyclictest0-21swapper/222:55:014
2838499120,7cyclictest0-21swapper/019:20:230
275991120,1ptp4l21-21ksoftirqd/121:15:251
275991120,1ptp4l21-21ksoftirqd/120:03:341
275991120,0ptp4l0-21swapper/123:26:211
275991120,0ptp4l0-21swapper/122:30:181
275991120,0ptp4l0-21swapper/122:25:121
275991120,0ptp4l0-21swapper/122:15:261
275991120,0ptp4l0-21swapper/120:50:191
275991120,0ptp4l0-21swapper/120:15:201
275991120,0ptp4l0-21swapper/120:05:121
94980110,0rtkit-daemon0-21swapper/823:04:5810
2843899112,5cyclictest2188-21kworker/u24:022:00:003
28421991110,1cyclictest0-21swapper/800:05:0010
2841799110,2cyclictest15593-21timerandwakeup19:30:259
2841299112,6cyclictest0-21swapper/623:25:258
2841299112,5cyclictest0-21swapper/620:05:018
2840899118,2cyclictest10000-21df_inode19:25:157
2840899110,10cyclictest19043-21users21:40:277
2840299110,10cyclictest4644-21df23:30:146
2839899110,10cyclictest0-21swapper/319:20:015
2839399112,6cyclictest0-21swapper/200:35:004
2838499114,6cyclictest0-21swapper/021:40:010
2838499112,7cyclictest0-21swapper/022:45:000
2838499112,7cyclictest0-21swapper/019:30:010
275991110,7ptp4l21-21ksoftirqd/120:26:111
275991110,2ptp4l0-21swapper/122:00:191
275991110,1ptp4l21-21ksoftirqd/121:08:051
275991110,1ptp4l0-21swapper/123:45:141
275991110,1ptp4l0-21swapper/123:40:191
275991110,1ptp4l0-21swapper/122:50:131
275991110,1ptp4l0-21swapper/120:40:121
275991110,1ptp4l0-21swapper/120:30:251
275991110,1ptp4l0-21swapper/120:20:231
275991110,1ptp4l0-21swapper/119:55:131
275991110,1ptp4l0-21swapper/119:10:141
275991110,1ptp4l0-21swapper/100:25:141
275991110,0ptp4l0-21swapper/123:35:121
275991110,0ptp4l0-21swapper/121:02:541
275991110,0ptp4l0-21swapper/119:30:151
2843899108,1cyclictest0-21swapper/1119:15:003
2843899101,6cyclictest0-21swapper/1120:35:253
2843099102,5cyclictest0-21swapper/1000:25:012
2843099100,9cyclictest5250-21if_err_enp4s023:30:182
2843099100,9cyclictest1302-21sed22:45:012
2843099100,9cyclictest0-21swapper/1019:50:012
2842199103,6cyclictest0-21swapper/822:40:2610
2842199102,7cyclictest0-21swapper/821:25:0110
2842199102,7cyclictest0-21swapper/821:15:0010
2842199100,9cyclictest0-21swapper/821:35:3010
2842199100,9cyclictest0-21swapper/820:40:1310
2841299102,5cyclictest0-21swapper/620:30:008
2841299102,5cyclictest0-21swapper/600:20:008
2840899102,4cyclictest0-21swapper/520:50:017
2840899100,9cyclictest9792-21open_inodes23:35:197
2840899100,9cyclictest9585-21memory20:05:187
2839899100,9cyclictest0-21swapper/323:15:235
2839899100,2cyclictest0-21swapper/320:50:195
2839399102,6cyclictest0-21swapper/223:20:254
2839399102,6cyclictest0-21swapper/200:24:594
2839399101,6cyclictest0-21swapper/223:05:004
2838499103,6cyclictest0-21swapper/023:11:170
2838499103,6cyclictest0-21swapper/022:00:000
2838499103,6cyclictest0-21swapper/019:30:000
2838499102,7cyclictest0-21swapper/023:10:010
2838499102,7cyclictest0-21swapper/022:09:590
2838499102,6cyclictest0-21swapper/021:50:000
2838499101,5cyclictest0-21swapper/022:30:000
275991100,5ptp4l21-21ksoftirqd/100:18:301
275991100,1ptp4l21-21ksoftirqd/122:55:151
275991100,1ptp4l0-21swapper/122:05:181
275991100,1ptp4l0-21swapper/120:35:141
275991100,1ptp4l0-21swapper/120:10:281
275991100,1ptp4l0-21swapper/100:34:591
275991100,0ptp4l0-21swapper/121:30:271
275991100,0ptp4l0-21swapper/121:20:231
275991100,0ptp4l0-21swapper/120:45:161
275991100,0ptp4l0-21swapper/119:20:181
275991100,0ptp4l0-21swapper/100:10:191
284389995,3cyclictest1986-21phc2sys-jitter19:55:193
284309992,6cyclictest0-21swapper/1022:29:592
284309992,5cyclictest0-21swapper/1021:55:002
284309992,5cyclictest0-21swapper/1021:15:012
284309991,5cyclictest0-21swapper/1022:20:002
284309990,9cyclictest0-21swapper/1022:05:182
284269996,2cyclictest2171-21diskstats19:15:1211
284219997,1cyclictest0-21swapper/822:35:2010
284219992,4cyclictest0-21swapper/800:40:0110
284219991,7cyclictest0-21swapper/820:30:2510
284179990,9cyclictest0-21swapper/721:15:239
284179990,8cyclictest8329-21if_err_p34p122:10:199
284179990,1cyclictest17388-21/usr/sbin/munin20:15:209
284129995,3cyclictest28292-21latency_hist19:50:008
284129992,5cyclictest2188-21kworker/u24:023:40:018
284129992,4cyclictest0-21swapper/623:50:018
284129992,4cyclictest0-21swapper/600:25:008
284129992,3cyclictest0-21swapper/622:25:008
284129991,5cyclictest0-21swapper/623:50:008
284129991,5cyclictest0-21swapper/621:20:018
284129991,5cyclictest0-21swapper/619:20:018
284129990,8cyclictest0-21swapper/621:20:188
284129990,1cyclictest51-21ksoftirqd/623:35:118
284089998,1cyclictest0-21swapper/520:20:017
284089991,5cyclictest0-21swapper/500:05:247
284089991,4cyclictest0-21swapper/522:05:007
284089990,1cyclictest0-21swapper/523:25:257
284029990,9cyclictest0-21swapper/422:20:186
284029990,8cyclictest23813-21fschecks_count20:25:156
284029990,1cyclictest0-21swapper/400:30:186
283989990,8cyclictest1020-21runrttasks19:20:155
283989990,8cyclictest1020-21runrttasks00:22:125
283989990,1cyclictest0-21swapper/323:20:205
283989990,1cyclictest0-21swapper/322:20:195
283939992,4cyclictest0-21swapper/219:50:014
283939991,4cyclictest0-21swapper/220:40:004
283939990,8cyclictest0-21swapper/220:45:164
283939990,8cyclictest0-21swapper/219:35:144
283939990,8cyclictest0-21swapper/200:15:154
283939990,2cyclictest20893-21cat23:10:134
283849998,1cyclictest0-21swapper/021:00:250
283849996,2cyclictest20158-21turbostat19:40:010
283849995,3cyclictest9173-21chrt00:16:010
283849992,6cyclictest0-21swapper/022:54:590
283849990,1cyclictest0-21swapper/021:30:170
27990291,6sleep90-21swapper/919:08:2511
27599190,1ptp4l21-21ksoftirqd/100:00:121
27599190,0ptp4l0-21swapper/123:50:111
27599190,0ptp4l0-21swapper/123:30:131
27599190,0ptp4l0-21swapper/123:20:151
27599190,0ptp4l0-21swapper/123:05:151
27599190,0ptp4l0-21swapper/122:35:121
27599190,0ptp4l0-21swapper/121:10:011
27599190,0ptp4l0-21swapper/119:45:131
27599190,0ptp4l0-21swapper/119:35:141
27599190,0ptp4l0-21swapper/100:35:011
27599190,0ptp4l0-21swapper/100:20:011
284389980,2cyclictest14796-21munin-run23:44:593
284389980,1cyclictest0-21swapper/1122:10:253
284309987,1cyclictest0-21swapper/1023:45:112
284309987,1cyclictest0-21swapper/1023:25:152
284309987,1cyclictest0-21swapper/1021:20:272
284309981,4cyclictest0-21swapper/1000:40:012
284309980,8cyclictest0-21swapper/1020:50:252
284309980,1cyclictest0-21swapper/1022:55:202
284309980,1cyclictest0-21swapper/1021:55:142
284269987,1cyclictest0-21swapper/920:30:0111
284269985,2cyclictest1651-21cut21:20:1711
284269980,8cyclictest0-21swapper/919:25:1911
284269980,8cyclictest0-21swapper/900:05:1811
284269980,1cyclictest0-21swapper/923:35:1511
284269980,1cyclictest0-21swapper/920:10:1611
284219982,5cyclictest0-21swapper/823:54:5910
284219980,1cyclictest0-21swapper/822:50:1110
284219980,1cyclictest0-21swapper/819:45:1510
284179987,1cyclictest0-21swapper/721:40:159
284179987,1cyclictest0-21swapper/720:05:189
284179980,8cyclictest0-21swapper/723:10:239
284179980,8cyclictest0-21swapper/719:55:159
284179980,1cyclictest0-21swapper/723:40:179
284179980,1cyclictest0-21swapper/723:20:209
284179980,1cyclictest0-21swapper/720:20:129
284179980,1cyclictest0-21swapper/719:35:159
284129987,1cyclictest0-21swapper/621:40:188
284129986,1cyclictest29381-21timerandwakeup00:00:298
284129982,4cyclictest0-21swapper/620:55:008
284129981,4cyclictest0-21swapper/600:30:008
284129981,3cyclictest0-21swapper/621:15:008
284129980,7cyclictest8690-21taskset19:25:018
284129980,7cyclictest31584-21chrt19:54:148
284129980,1cyclictest0-21swapper/622:00:258
284089987,1cyclictest0-21swapper/521:35:187
284089987,1cyclictest0-21swapper/521:20:217
284089987,1cyclictest0-21swapper/521:05:187
284089987,1cyclictest0-21swapper/520:40:227
284089981,5cyclictest0-21swapper/519:35:007
284089980,8cyclictest0-21swapper/523:10:197
284089980,8cyclictest0-21swapper/522:40:267
284089980,8cyclictest0-21swapper/522:07:237
284089980,8cyclictest0-21swapper/520:25:217
284089980,1cyclictest0-21swapper/522:10:237
284089980,1cyclictest0-21swapper/500:35:167
284029987,1cyclictest0-21swapper/420:55:126
284029984,4cyclictest14850irq/20-ata_piix00:10:016
284029982,3cyclictest14850irq/20-ata_piix20:15:016
284029981,4cyclictest14850irq/20-ata_piix22:25:016
284029980,8cyclictest0-21swapper/422:15:196
284029980,8cyclictest0-21swapper/421:27:186
284029980,3cyclictest10657-21interrupts19:25:206
284029980,1cyclictest0-21swapper/423:55:236
284029980,1cyclictest0-21swapper/422:50:166
284029980,1cyclictest0-21swapper/422:45:166
284029980,1cyclictest0-21swapper/421:35:186
284029980,1cyclictest0-21swapper/421:15:156
284029980,1cyclictest0-21swapper/421:05:246
284029980,1cyclictest0-21swapper/400:25:256
283989987,1cyclictest0-21swapper/323:55:125
283989987,1cyclictest0-21swapper/322:55:185
283989981,6cyclictest0-21swapper/322:50:145
283989980,8cyclictest0-21swapper/322:25:125
283989980,1cyclictest24750-21diskmemload22:40:185
283989980,1cyclictest0-21swapper/320:35:205
283989980,1cyclictest0-21swapper/320:20:255
283939980,8cyclictest0-21swapper/222:15:244
283939980,8cyclictest0-21swapper/222:10:184
283939980,8cyclictest0-21swapper/219:15:374
283939980,7cyclictest4820-21/usr/sbin/munin22:05:204
283939980,7cyclictest4608-21taskset22:46:404
283939980,7cyclictest31402-21sshd19:51:404
283939980,7cyclictest15980-21latency_hist21:40:014
283939980,7cyclictest0-21swapper/219:55:174
283939980,3cyclictest8439-21sshd19:21:404
283939980,3cyclictest19175-21/usr/sbin/munin22:25:124
283939980,1cyclictest0-21swapper/222:00:134
283939980,1cyclictest0-21swapper/219:10:154
283939980,1cyclictest0-21swapper/200:35:154
283849987,1cyclictest0-21swapper/019:45:170
283849986,1cyclictest29346-21turbostat22:40:000
283849985,2cyclictest22453-21turbostat20:25:010
283849982,5cyclictest0-21swapper/022:30:220
283849980,8cyclictest0-21swapper/000:00:170
283849980,1cyclictest91192sleep020:45:370
283849980,1cyclictest0-21swapper/023:40:260
27609180,1phc2sys0-21swapper/219:08:254
284389976,1cyclictest0-21swapper/1123:38:423
284389976,1cyclictest0-21swapper/1123:30:263
284389976,1cyclictest0-21swapper/1123:25:243
284389976,1cyclictest0-21swapper/1123:20:143
284389976,1cyclictest0-21swapper/1123:02:153
284389976,1cyclictest0-21swapper/1122:35:093
284389976,1cyclictest0-21swapper/1122:05:553
284389976,1cyclictest0-21swapper/1121:31:003
284389976,1cyclictest0-21swapper/1121:28:223
284389976,1cyclictest0-21swapper/1121:11:223
284389976,1cyclictest0-21swapper/1120:40:213
284389976,1cyclictest0-21swapper/1119:50:203
284389976,1cyclictest0-21swapper/1119:34:263
284389976,1cyclictest0-21swapper/1100:35:243
284389976,1cyclictest0-21swapper/1100:05:153
284389974,2cyclictest23013-21smicount23:10:243
284389974,2cyclictest11698-21timerandwakeup19:25:263
284389974,2cyclictest1020-21runrttasks22:26:233
284389974,2cyclictest1020-21runrttasks00:34:163
284389973,3cyclictest21048-21cpuspeed_turbos19:40:123
284389970,7cyclictest0-21swapper/1123:51:413
284389970,7cyclictest0-21swapper/1123:15:263
284389970,7cyclictest0-21swapper/1123:06:333
284389970,7cyclictest0-21swapper/1122:20:293
284389970,7cyclictest0-21swapper/1121:35:293
284389970,7cyclictest0-21swapper/1120:56:423
284389970,7cyclictest0-21swapper/1120:30:273
284389970,7cyclictest0-21swapper/1120:05:273
284389970,7cyclictest0-21swapper/1120:02:593
284389970,7cyclictest0-21swapper/1119:35:153
284389970,7cyclictest0-21swapper/1119:24:283
284389970,7cyclictest0-21swapper/1100:10:223
284389970,1cyclictest0-21swapper/1123:57:163
284389970,1cyclictest0-21swapper/1122:47:333
284389970,1cyclictest0-21swapper/1122:30:113
284389970,1cyclictest0-21swapper/1122:03:333
284389970,1cyclictest0-21swapper/1121:18:363
284389970,1cyclictest0-21swapper/1121:06:043
284389970,1cyclictest0-21swapper/1120:50:543
284389970,1cyclictest0-21swapper/1120:25:243
284389970,1cyclictest0-21swapper/1100:26:163
284389970,1cyclictest0-21swapper/1100:21:533
284389970,1cyclictest0-21swapper/1100:00:323
284309976,1cyclictest0-21swapper/1023:55:032
284309976,1cyclictest0-21swapper/1023:20:112
284309976,1cyclictest0-21swapper/1023:05:272
284309976,1cyclictest0-21swapper/1022:35:102
284309976,1cyclictest0-21swapper/1022:33:032
284309976,1cyclictest0-21swapper/1022:20:132
284309976,1cyclictest0-21swapper/1021:40:472
284309976,1cyclictest0-21swapper/1021:36:022
284309976,1cyclictest0-21swapper/1021:16:302
284309976,1cyclictest0-21swapper/1020:55:152
284309976,1cyclictest0-21swapper/1020:15:162
284309976,1cyclictest0-21swapper/1019:41:212
284309976,1cyclictest0-21swapper/1019:27:512
284309976,1cyclictest0-21swapper/1019:13:592
284309974,2cyclictest8379-21irqrtprio22:10:212
284309974,2cyclictest29738-21if_err_enp4s019:50:172
284309974,2cyclictest0-21swapper/1020:25:202
284309970,7cyclictest0-21swapper/1023:51:022
284309970,7cyclictest0-21swapper/1023:17:062
284309970,7cyclictest0-21swapper/1023:10:402
284309970,7cyclictest0-21swapper/1023:01:012
284309970,7cyclictest0-21swapper/1021:27:152
284309970,7cyclictest0-21swapper/1021:06:072
284309970,7cyclictest0-21swapper/1021:00:262
284309970,7cyclictest0-21swapper/1020:30:232
284309970,7cyclictest0-21swapper/1020:20:502
284309970,7cyclictest0-21swapper/1020:05:262
284309970,7cyclictest0-21swapper/1000:25:372
284309970,7cyclictest0-21swapper/1000:05:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional