You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 10:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Mon Nov 24, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1140921220,1sleep5441rcuc/519:05:227
2759911020,92ptp4l0-21swapper/119:09:061
1231327057,11sleep70-21swapper/719:08:019
1040426754,10sleep00-21swapper/019:05:180
1128426556,6sleep40-21swapper/419:05:216
1022726554,8sleep110-21swapper/1119:05:163
1232626355,6sleep60-21swapper/619:08:128
198202620,0sleep20-21swapper/223:30:004
1212926148,10sleep100-21swapper/1019:05:282
244722600,0sleep10-21swapper/123:35:111
75542520,0sleep50-21swapper/523:10:257
1281099233,16cyclictest0-21swapper/422:35:016
1281099222,15cyclictest0-21swapper/419:40:016
1281099212,14cyclictest0-21swapper/421:40:016
1280099213,12cyclictest0-21swapper/223:05:004
1281099191,17cyclictest0-21swapper/423:15:016
1280599191,16cyclictest0-21swapper/322:05:145
1283999180,1cyclictest0-21swapper/1022:50:222
275991160,0ptp4l0-21swapper/121:45:211
275991160,0ptp4l0-21swapper/119:55:181
275991160,0ptp4l0-21swapper/119:45:171
275991160,0ptp4l0-21swapper/119:30:251
1283499163,8cyclictest0-21swapper/922:15:0111
1283099162,8cyclictest0-21swapper/800:25:0110
1283099162,8cyclictest0-21swapper/800:25:0110
1283099160,3cyclictest0-21swapper/819:40:1810
1281499160,14cyclictest0-21swapper/523:30:187
1280599163,8cyclictest0-21swapper/300:40:015
1280599163,8cyclictest0-21swapper/300:40:005
1280599163,8cyclictest0-21swapper/300:00:015
1280599162,9cyclictest0-21swapper/320:20:015
1280099163,8cyclictest0-21swapper/222:05:004
1283999150,3cyclictest0-21swapper/1021:10:172
1283499150,10cyclictest0-21swapper/923:10:0111
1283099153,7cyclictest0-21swapper/822:40:0010
1283099153,7cyclictest0-21swapper/821:19:5910
1281999153,7cyclictest0-21swapper/620:05:008
1281499153,8cyclictest0-21swapper/520:49:597
1280599152,9cyclictest0-21swapper/322:50:005
1280599152,7cyclictest0-21swapper/323:55:005
1280099152,9cyclictest0-21swapper/221:34:594
1280099152,8cyclictest0-21swapper/221:25:274
1284399144,6cyclictest0-21swapper/1120:45:003
1284399143,7cyclictest0-21swapper/1119:15:003
1284399142,7cyclictest0-21swapper/1123:30:013
1283999140,2cyclictest0-21swapper/1019:50:272
1283499143,7cyclictest0-21swapper/920:54:5911
1283499142,7cyclictest13954-21kworker/u24:223:40:0011
1283499142,7cyclictest0-21swapper/922:40:0011
1283499141,9cyclictest13954-21kworker/u24:221:55:0011
1283499141,8cyclictest0-21swapper/900:00:0011
1281999145,8cyclictest0-21swapper/620:44:598
1281499143,7cyclictest0-21swapper/521:29:597
1281499143,7cyclictest0-21swapper/519:30:257
1281499143,6cyclictest0-21swapper/521:15:017
1281499142,7cyclictest0-21swapper/519:30:007
1281099142,7cyclictest0-21swapper/422:25:266
1280599141,8cyclictest0-21swapper/323:30:005
1280599141,8cyclictest0-21swapper/321:09:595
12796991411,2cyclictest0-21swapper/122:05:151
1279699140,8cyclictest0-21swapper/122:25:251
275991130,4ptp4l14645-21irqrtprio19:10:191
275991130,1ptp4l21-21ksoftirqd/120:09:501
275991130,0ptp4l0-21swapper/122:40:361
275991130,0ptp4l0-21swapper/120:03:141
1283999131,6cyclictest0-21swapper/1023:01:112
1283499133,6cyclictest0-21swapper/920:10:0111
1283499132,7cyclictest0-21swapper/920:20:0011
1283499132,7cyclictest0-21swapper/900:10:0111
1283499131,8cyclictest0-21swapper/923:25:0111
12834991311,1cyclictest0-21swapper/922:25:0011
1283499130,7cyclictest0-21swapper/900:20:0011
1283099132,7cyclictest0-21swapper/800:25:2610
1283099132,7cyclictest0-21swapper/800:25:2510
1283099132,6cyclictest0-21swapper/820:30:0210
1283099132,6cyclictest0-21swapper/819:20:0010
1283099131,8cyclictest0-21swapper/821:55:2410
1281999133,9cyclictest0-21swapper/623:55:018
1281999133,7cyclictest0-21swapper/619:25:008
1281999132,7cyclictest0-21swapper/623:00:018
1281999131,8cyclictest0-21swapper/619:30:018
1281499132,7cyclictest0-21swapper/523:20:017
1281499131,7cyclictest0-21swapper/522:05:017
1281499131,7cyclictest0-21swapper/520:15:017
1281499131,11cyclictest13954-21kworker/u24:222:09:597
1281099132,6cyclictest0-21swapper/400:35:246
1281099132,6cyclictest0-21swapper/400:35:246
1280599132,7cyclictest0-21swapper/320:50:245
1280599132,6cyclictest0-21swapper/319:50:005
1280099133,6cyclictest0-21swapper/219:55:004
1279699130,7cyclictest0-21swapper/122:55:201
275991120,0ptp4l0-21swapper/123:00:251
275991120,0ptp4l0-21swapper/120:51:191
275991120,0ptp4l0-21swapper/120:42:301
275991120,0ptp4l0-21swapper/120:24:281
1284399121,6cyclictest0-21swapper/1120:20:003
1283999120,7cyclictest0-21swapper/1019:25:012
1283999120,11cyclictest0-21swapper/1020:59:592
12834991210,1cyclictest0-21swapper/923:05:0011
1283499120,6cyclictest31299-21munin-run20:15:0111
1283099120,7cyclictest0-21swapper/800:20:0010
1281999121,7cyclictest0-21swapper/623:00:248
1281499122,7cyclictest0-21swapper/520:00:277
1281499122,7cyclictest0-21swapper/500:30:017
1281499122,7cyclictest0-21swapper/500:30:017
1280599122,6cyclictest0-21swapper/320:30:225
1279699124,2cyclictest1020-21runrttasks20:11:581
1279699121,2cyclictest0-21swapper/121:30:161
1279699120,7cyclictest0-21swapper/123:29:051
1279699120,11cyclictest0-21swapper/121:15:011
1279099122,5cyclictest0-21swapper/020:10:010
86212111,7sleep80-21swapper/819:05:0210
275991110,4ptp4l21-21ksoftirqd/119:20:001
275991110,1ptp4l0-21swapper/123:45:141
275991110,1ptp4l0-21swapper/123:40:131
275991110,1ptp4l0-21swapper/123:15:141
275991110,1ptp4l0-21swapper/120:45:251
275991110,1ptp4l0-21swapper/100:25:171
275991110,1ptp4l0-21swapper/100:25:161
275991110,1ptp4l0-21swapper/100:05:271
275991110,0ptp4l0-21swapper/122:39:061
275991110,0ptp4l0-21swapper/122:00:241
275991110,0ptp4l0-21swapper/120:31:151
275991110,0ptp4l0-21swapper/119:36:101
275991110,0ptp4l0-21swapper/119:25:241
1284399111,6cyclictest0-21swapper/1121:30:003
1284399110,2cyclictest27455-21perl20:50:143
1283999110,10cyclictest0-21swapper/1000:05:002
1283499112,5cyclictest0-21swapper/919:45:0111
1283499110,5cyclictest16774-21awk21:19:5911
1283099110,6cyclictest0-21swapper/822:04:5910
1283099110,10cyclictest0-21swapper/821:35:0010
12823991110,1cyclictest561rcuc/722:15:139
1282399110,11cyclictest0-21swapper/721:00:229
1281999112,8cyclictest0-21swapper/622:20:018
1281499110,2cyclictest30292-21df23:00:127
1281499110,10cyclictest0-21swapper/500:05:007
1281099110,10cyclictest7008-21cut21:05:166
1280599118,2cyclictest0-21swapper/323:00:135
1280599110,10cyclictest0-21swapper/319:55:155
1280099118,2cyclictest0-21swapper/219:30:254
1280099110,10cyclictest0-21swapper/223:35:234
1279699117,3cyclictest0-21swapper/122:15:121
1279699115,2cyclictest13499-21df_inode23:20:161
1279699114,2cyclictest5314-21cpuspeed_turbos19:40:151
1279699113,2cyclictest6083-21cpuspeed_turbos21:05:141
1279699113,2cyclictest16467-21irqrtprio20:35:181
1279699112,3cyclictest5230-21kernelversion23:50:191
1279699110,7cyclictest0-21swapper/100:30:561
1279699110,7cyclictest0-21swapper/100:30:551
1279699110,10cyclictest0-21swapper/121:40:181
275991100,2ptp4l19984-21perl00:10:191
275991100,1ptp4l0-21swapper/123:55:141
275991100,1ptp4l0-21swapper/123:10:231
275991100,1ptp4l0-21swapper/122:20:121
275991100,1ptp4l0-21swapper/121:50:181
275991100,1ptp4l0-21swapper/120:20:001
275991100,1ptp4l0-21swapper/100:20:141
275991100,1ptp4l0-21swapper/100:20:141
275991100,1ptp4l0-21swapper/100:00:131
275991100,0ptp4l0-21swapper/123:05:131
275991100,0ptp4l0-21swapper/122:10:161
275991100,0ptp4l0-21swapper/121:10:161
275991100,0ptp4l0-21swapper/121:00:181
275991100,0ptp4l0-21swapper/120:55:141
153232100,0sleep110-21swapper/1121:15:173
1283999102,5cyclictest0-21swapper/1022:10:012
1283999102,4cyclictest0-21swapper/1022:15:012
1283999101,5cyclictest0-21swapper/1021:20:002
1283499100,9cyclictest0-21swapper/919:35:1711
1283499100,1cyclictest0-21swapper/921:15:0011
1283099108,1cyclictest0-21swapper/820:09:5610
1283099100,9cyclictest0-21swapper/822:15:1810
1283099100,1cyclictest0-21swapper/822:20:2410
1282399109,1cyclictest1020-21runrttasks20:06:129
1281999100,9cyclictest0-21swapper/620:50:138
1281999100,9cyclictest0-21swapper/600:00:248
1281999100,6cyclictest0-21swapper/619:34:568
1281499108,1cyclictest0-21swapper/522:35:147
1280599100,1cyclictest0-21swapper/320:50:005
1280099102,5cyclictest0-21swapper/219:50:004
1280099100,9cyclictest0-21swapper/222:05:224
1279699103,2cyclictest19296-21idleruntime22:45:151
1279699101,3cyclictest0-21swapper/121:35:171
1279699100,6cyclictest0-21swapper/122:33:241
1279699100,1cyclictest0-21swapper/123:33:371
1279699100,1cyclictest0-21swapper/121:22:061
1279699100,1cyclictest0-21swapper/100:39:231
1279699100,1cyclictest0-21swapper/100:39:221
1279699100,10cyclictest0-21swapper/122:50:201
1279099102,5cyclictest0-21swapper/000:20:240
1279099102,5cyclictest0-21swapper/000:20:230
1279099101,5cyclictest0-21swapper/020:20:000
9498090,0rtkit-daemon0-21swapper/823:38:2810
27599190,0ptp4l0-21swapper/121:55:001
27599190,0ptp4l0-21swapper/121:25:011
27599190,0ptp4l0-21swapper/120:25:021
27599190,0ptp4l0-21swapper/119:50:131
27599190,0ptp4l0-21swapper/100:15:161
128439997,1cyclictest25178-21cpuspeed_turbos21:30:113
128439990,1cyclictest10661-21sed23:15:173
128399998,1cyclictest0-21swapper/1022:25:162
128399998,1cyclictest0-21swapper/1020:35:002
128399997,1cyclictest0-21swapper/1019:40:332
128399990,9cyclictest0-21swapper/1022:00:252
128399990,9cyclictest0-21swapper/1000:00:002
128399990,8cyclictest19481-21/usr/sbin/munin21:20:222
128399990,8cyclictest0-21swapper/1020:20:232
128399990,8cyclictest0-21swapper/1020:00:172
128349996,2cyclictest29520-21taskset00:21:3511
128349996,2cyclictest29520-21taskset00:21:3511
128309998,1cyclictest0-21swapper/820:55:1610
128309990,9cyclictest0-21swapper/822:30:1310
128309990,1cyclictest0-21swapper/823:25:2610
128239997,1cyclictest0-21swapper/700:18:299
128239991,8cyclictest561rcuc/700:25:169
128239991,8cyclictest561rcuc/700:25:159
128239991,8cyclictest0-21swapper/722:29:319
128239991,1cyclictest0-21swapper/722:20:139
128239990,8cyclictest30850-21awk23:40:249
128239990,8cyclictest24194-21smartctl22:50:239
128239990,2cyclictest25306-21fw_conntrack23:35:149
128199998,1cyclictest0-21swapper/620:20:258
128199992,6cyclictest0-21swapper/621:10:008
128149992,6cyclictest0-21swapper/522:50:017
128149990,9cyclictest0-21swapper/500:30:157
128149990,9cyclictest0-21swapper/500:30:157
128149990,1cyclictest0-21swapper/522:20:147
128109990,8cyclictest0-21swapper/422:10:196
128059998,1cyclictest0-21swapper/322:10:245
128059996,2cyclictest149102sleep321:55:395
128009998,1cyclictest0-21swapper/223:10:244
128009990,9cyclictest0-21swapper/219:55:164
128009990,8cyclictest23441-21chrt23:32:124
128009990,1cyclictest0-21swapper/200:15:174
127969993,2cyclictest1020-21runrttasks19:20:121
127909991,5cyclictest0-21swapper/021:19:590
127909990,9cyclictest0-21swapper/019:35:010
127909990,8cyclictest17863-21if_enp1s0.123:25:180
9137281,5sleep90-21swapper/919:05:0311
20447280,2sleep221127-21df20:00:144
128439987,1cyclictest0-21swapper/1123:00:223
128439982,5cyclictest0-21swapper/1121:50:003
128439980,7cyclictest20986-21sshd00:10:243
128439980,3cyclictest0-21swapper/1123:59:543
128439980,1cyclictest0-21swapper/1120:20:273
128399987,1cyclictest0-21swapper/1020:35:222
128399980,8cyclictest0-21swapper/1023:20:202
128399980,8cyclictest0-21swapper/1020:25:252
128399980,8cyclictest0-21swapper/1000:25:222
128399980,8cyclictest0-21swapper/1000:25:222
128399980,1cyclictest0-21swapper/1000:19:432
128349980,8cyclictest0-21swapper/920:48:4711
128349980,1cyclictest0-21swapper/922:25:2711
128349980,1cyclictest0-21swapper/921:35:1911
128349980,1cyclictest0-21swapper/919:15:2011
128349980,1cyclictest0-21swapper/900:00:2311
128309985,2cyclictest20585-21turbostat21:24:5910
128309980,8cyclictest0-21swapper/823:00:1610
128309980,1cyclictest0-21swapper/823:55:1610
128309980,1cyclictest0-21swapper/821:05:2310
128309980,1cyclictest0-21swapper/800:05:2410
128239987,1cyclictest57-21ksoftirqd/722:40:219
128239987,1cyclictest0-21swapper/723:12:009
128239987,1cyclictest0-21swapper/723:06:169
128239987,1cyclictest0-21swapper/722:38:089
128239987,1cyclictest0-21swapper/722:12:369
128239987,1cyclictest0-21swapper/721:45:299
128239987,1cyclictest0-21swapper/721:37:309
128239987,1cyclictest0-21swapper/721:29:289
128239987,1cyclictest0-21swapper/721:17:229
128239987,1cyclictest0-21swapper/720:39:599
128239987,1cyclictest0-21swapper/720:22:529
128239987,1cyclictest0-21swapper/720:00:529
128239987,1cyclictest0-21swapper/719:56:189
128239987,1cyclictest0-21swapper/719:38:509
128239987,1cyclictest0-21swapper/719:28:479
128239987,1cyclictest0-21swapper/719:16:389
128239987,1cyclictest0-21swapper/700:31:209
128239987,1cyclictest0-21swapper/700:31:199
128239987,1cyclictest0-21swapper/700:06:469
128239987,1cyclictest0-21swapper/700:00:539
128239984,3cyclictest12586-21if_p34p120:30:179
128239983,4cyclictest29466-21ps20:10:209
128239983,4cyclictest26917-21tail22:55:169
128239983,4cyclictest16955-21grep22:00:169
128239983,4cyclictest10181-21grep23:55:249
128239982,6cyclictest561rcuc/723:52:029
128239981,7cyclictest0-21swapper/723:17:379
128239981,7cyclictest0-21swapper/723:04:039
128239981,7cyclictest0-21swapper/721:53:509
128239981,7cyclictest0-21swapper/721:42:169
128239981,7cyclictest0-21swapper/721:20:069
128239981,7cyclictest0-21swapper/721:12:139
128239981,7cyclictest0-21swapper/700:21:389
128239981,7cyclictest0-21swapper/700:21:379
128239981,1cyclictest0-21swapper/723:30:289
128239981,1cyclictest0-21swapper/721:56:379
128239981,1cyclictest0-21swapper/721:08:169
128239981,1cyclictest0-21swapper/720:15:159
128239981,1cyclictest0-21swapper/700:36:439
128239981,1cyclictest0-21swapper/700:36:439
128239980,1cyclictest0-21swapper/721:30:199
128239980,1cyclictest0-21swapper/720:50:129
128199987,1cyclictest0-21swapper/623:55:248
128199987,1cyclictest0-21swapper/623:10:168
128199987,1cyclictest0-21swapper/619:15:118
128199981,4cyclictest0-21swapper/623:05:238
128199980,8cyclictest0-21swapper/623:30:198
128199980,8cyclictest0-21swapper/621:35:018
128199980,8cyclictest0-21swapper/620:45:168
128199980,8cyclictest0-21swapper/620:10:148
128199980,1cyclictest0-21swapper/621:45:258
128199980,1cyclictest0-21swapper/621:20:238
128199980,1cyclictest0-21swapper/600:05:258
128149987,1cyclictest0-21swapper/523:40:147
128149985,2cyclictest11833-21sshd19:45:357
128149980,8cyclictest0-21swapper/523:25:147
128149980,8cyclictest0-21swapper/521:45:217
128149980,8cyclictest0-21swapper/500:35:187
128149980,8cyclictest0-21swapper/500:35:187
128149980,7cyclictest9091-21diskmemload23:22:037
128149980,1cyclictest0-21swapper/523:57:207
128109986,1cyclictest22135-21turbostat00:19:556
128109985,2cyclictest3236-21runrttasks21:40:476
128109985,2cyclictest15227-21turbostat22:00:006
128109982,5cyclictest0-21swapper/419:10:266
128109981,6cyclictest0-21swapper/422:05:226
128109980,8cyclictest0-21swapper/423:20:176
128109980,8cyclictest0-21swapper/421:50:156
128109980,2cyclictest18590-21cstates22:45:116
128109980,1cyclictest0-21swapper/422:16:386
128109980,1cyclictest0-21swapper/420:47:416
128059987,1cyclictest0-21swapper/323:35:135
128059987,1cyclictest0-21swapper/322:20:165
128059987,1cyclictest0-21swapper/321:00:225
128059987,1cyclictest0-21swapper/320:00:175
128059982,4cyclictest0-21swapper/319:10:175
128059980,8cyclictest0-21swapper/320:05:185
128059980,7cyclictest9091-21diskmemload00:15:455
128059980,1cyclictest0-21swapper/322:20:015
128009987,1cyclictest0-21swapper/221:15:164
128009987,1cyclictest0-21swapper/220:25:014
128009981,6cyclictest20261-21processes19:15:204
128009980,8cyclictest0-21swapper/222:15:144
128009980,8cyclictest0-21swapper/221:55:244
128009980,8cyclictest0-21swapper/200:00:124
128009980,7cyclictest5277-21munin-run00:35:014
128009980,7cyclictest5277-21munin-run00:35:014
128009980,7cyclictest18852-21aten2_r3power_e20:40:134
128009980,7cyclictest12687-21/usr/sbin/munin23:20:314
128009980,7cyclictest10423-21sshd21:10:144
128009980,2cyclictest22666-21cut20:45:114
128009980,2cyclictest0-21swapper/219:35:204
128009980,1cyclictest0-21swapper/223:01:124
128009980,1cyclictest0-21swapper/219:40:334
128009980,1cyclictest0-21swapper/219:25:204
127909987,1cyclictest0-21swapper/000:35:160
127909987,1cyclictest0-21swapper/000:35:150
127909982,3cyclictest0-21swapper/022:35:000
127909981,4cyclictest0-21swapper/000:35:010
127909981,4cyclictest0-21swapper/000:35:000
127909980,8cyclictest0-21swapper/022:30:140
127909980,7cyclictest9091-21diskmemload23:55:500
127909980,1cyclictest2556-21taskset23:46:580
127909980,1cyclictest0-21swapper/021:35:160
9144271,5sleep30-21swapper/319:05:085
27609170,2phc2sys0-21swapper/219:07:284
27609170,1phc2sys0-21swapper/200:10:154
128439976,1cyclictest0-21swapper/1123:50:233
128439976,1cyclictest0-21swapper/1123:45:473
128439976,1cyclictest0-21swapper/1123:42:163
128439976,1cyclictest0-21swapper/1123:06:193
128439976,1cyclictest0-21swapper/1122:58:543
128439976,1cyclictest0-21swapper/1122:50:173
128439976,1cyclictest0-21swapper/1122:15:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional