You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 20:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Sun Nov 16, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
949801010,95rtkit-daemon0-21swapper/1007:09:522
275991970,85ptp4l0-21swapper/107:05:121
2511926552,10sleep50-21swapper/507:07:337
2509426546,9sleep60-21swapper/607:07:108
2499726554,9sleep40-21swapper/407:05:446
2496526553,9sleep110-21swapper/1107:05:263
2499326351,10sleep00-21swapper/007:05:400
289072620,0sleep70-21swapper/709:15:249
2430426252,7sleep70-21swapper/707:05:229
216442600,0sleep110-21swapper/1110:30:253
196452590,0sleep3321rcuc/310:30:165
2564599242,8cyclictest0-21swapper/412:00:016
2564599242,8cyclictest0-21swapper/412:00:016
2564599202,13cyclictest0-21swapper/408:25:006
2564599192,13cyclictest0-21swapper/412:00:006
2564599192,13cyclictest0-21swapper/411:59:596
2563999194,9cyclictest0-21swapper/310:30:005
2563099193,9cyclictest0-21swapper/109:40:001
2562299191,16cyclictest0-21swapper/009:45:160
2567699185,7cyclictest0-21swapper/1111:55:013
2567699185,7cyclictest0-21swapper/1111:55:013
2565899181,2cyclictest0-21swapper/708:35:199
2565899181,16cyclictest0-21swapper/708:45:189
2564899183,9cyclictest0-21swapper/511:10:007
275991170,0ptp4l0-21swapper/109:10:141
275991170,0ptp4l0-21swapper/108:10:171
2566799172,9cyclictest0-21swapper/910:15:0011
2565899170,3cyclictest0-21swapper/707:15:289
2563999172,9cyclictest0-21swapper/309:55:005
2562299173,8cyclictest0-21swapper/007:50:010
275991160,0ptp4l0-21swapper/111:25:231
275991160,0ptp4l0-21swapper/108:20:201
275991160,0ptp4l0-21swapper/107:50:251
275991160,0ptp4l0-21swapper/107:45:181
2565899164,8cyclictest0-21swapper/709:29:599
2565899161,9cyclictest0-21swapper/707:15:019
2565499163,8cyclictest0-21swapper/608:25:008
25654991611,5cyclictest0-21swapper/609:49:558
2565499160,8cyclictest0-21swapper/610:50:268
2564899163,8cyclictest0-21swapper/510:25:007
2564899161,3cyclictest0-21swapper/507:10:167
2563999160,15cyclictest0-21swapper/308:55:185
2563099163,8cyclictest0-21swapper/110:50:011
2563099162,9cyclictest0-21swapper/111:05:001
275991150,0ptp4l0-21swapper/111:10:001
2567699153,7cyclictest0-21swapper/1107:15:013
2567299151,3cyclictest0-21swapper/1011:30:182
2567299151,3cyclictest0-21swapper/1011:30:182
2566799151,3cyclictest0-21swapper/912:05:1911
2566799150,2cyclictest0-21swapper/909:55:1711
2565899153,7cyclictest0-21swapper/709:25:009
2564899152,8cyclictest0-21swapper/511:25:017
2564899151,13cyclictest0-21swapper/511:35:217
2564899151,13cyclictest0-21swapper/511:35:217
2563999152,8cyclictest0-21swapper/311:45:015
2563999152,8cyclictest0-21swapper/311:45:005
2563999152,8cyclictest0-21swapper/309:59:595
2563099156,7cyclictest25614-21cyclictest12:30:001
2563099153,6cyclictest0-21swapper/112:25:011
2563099152,7cyclictest0-21swapper/109:45:001
2563099151,2cyclictest0-21swapper/112:05:191
25630991512,2cyclictest6630-21turbostat08:50:001
2562299154,7cyclictest0-21swapper/008:04:590
2562299153,8cyclictest0-21swapper/012:30:010
2562299153,7cyclictest0-21swapper/008:10:000
2562299152,8cyclictest0-21swapper/011:35:010
2562299152,8cyclictest0-21swapper/011:35:010
2562299152,8cyclictest0-21swapper/011:35:010
2562299152,8cyclictest0-21swapper/011:35:010
250162157,6sleep80-21swapper/807:06:0010
137652150,0sleep70-21swapper/707:35:019
275991140,1ptp4l21-21ksoftirqd/110:05:211
275991140,0ptp4l0-21swapper/112:10:251
275991140,0ptp4l0-21swapper/109:55:181
275991140,0ptp4l0-21swapper/109:50:211
275991140,0ptp4l0-21swapper/108:50:191
275991140,0ptp4l0-21swapper/107:10:201
2567699142,7cyclictest0-21swapper/1109:15:003
2566799144,6cyclictest1386-21kworker/u24:111:45:0011
2566799144,6cyclictest1386-21kworker/u24:111:45:0011
2566799143,6cyclictest0-21swapper/909:00:0011
2564899143,7cyclictest0-21swapper/509:05:007
2564899143,6cyclictest0-21swapper/507:55:007
2564899142,8cyclictest0-21swapper/509:45:007
2564599142,7cyclictest0-21swapper/412:35:286
2564599142,6cyclictest0-21swapper/412:25:286
2563999142,7cyclictest0-21swapper/309:10:255
2563099142,6cyclictest0-21swapper/108:10:001
2562299142,7cyclictest0-21swapper/008:40:000
134822140,0sleep20-21swapper/212:25:284
275991130,3ptp4l0-21swapper/110:00:151
275991130,0ptp4l0-21swapper/111:45:101
275991130,0ptp4l0-21swapper/111:45:101
275991130,0ptp4l0-21swapper/108:37:131
2567699131,7cyclictest0-21swapper/1110:04:593
2566799132,7cyclictest0-21swapper/908:05:0011
2565899132,8cyclictest1386-21kworker/u24:109:40:009
2565899132,7cyclictest0-21swapper/711:35:009
2565899132,7cyclictest0-21swapper/711:35:009
2565899130,7cyclictest0-21swapper/710:40:009
2565499132,6cyclictest0-21swapper/610:30:008
2564899132,6cyclictest0-21swapper/508:20:257
2564899131,7cyclictest0-21swapper/508:45:007
2563999132,6cyclictest0-21swapper/312:25:015
2563999131,7cyclictest0-21swapper/308:20:005
2563099133,6cyclictest0-21swapper/107:24:271
275991120,1ptp4l2306-21/usr/sbin/munin08:00:211
275991120,0ptp4l0-21swapper/110:50:171
275991120,0ptp4l0-21swapper/107:55:581
2567699120,6cyclictest0-21swapper/1110:14:553
2567299120,2cyclictest0-21swapper/1008:10:242
2565899122,6cyclictest1386-21kworker/u24:107:40:019
2565899122,6cyclictest0-21swapper/710:55:259
2565899122,6cyclictest0-21swapper/710:50:009
2565899122,6cyclictest0-21swapper/710:05:009
2565899122,6cyclictest0-21swapper/710:00:009
2565899120,11cyclictest23473-21awk08:29:599
2565499122,6cyclictest0-21swapper/609:15:008
2564899122,6cyclictest0-21swapper/508:35:267
2564899122,6cyclictest0-21swapper/507:40:017
2564899120,7cyclictest0-21swapper/508:20:007
2564599123,5cyclictest0-21swapper/411:30:266
2564599123,5cyclictest0-21swapper/411:30:266
2564599123,5cyclictest0-21swapper/410:25:066
2564599122,6cyclictest0-21swapper/408:40:436
2563999121,7cyclictest0-21swapper/308:55:005
2563099122,6cyclictest0-21swapper/112:05:001
2563099122,6cyclictest0-21swapper/112:04:591
2563099120,8cyclictest0-21swapper/110:10:271
2562299121,7cyclictest0-21swapper/011:40:010
2562299121,7cyclictest0-21swapper/011:40:000
275991110,5ptp4l0-21swapper/111:20:271
275991110,2ptp4l8166-21kernelversion07:25:191
275991110,2ptp4l7648-21sed09:30:201
275991110,2ptp4l20389-21timerandwakeup07:40:271
275991110,0ptp4l0-21swapper/111:50:221
275991110,0ptp4l0-21swapper/111:50:221
275991110,0ptp4l0-21swapper/110:35:151
275991110,0ptp4l0-21swapper/109:09:471
275991110,0ptp4l0-21swapper/108:25:151
2567299117,3cyclictest18821-21/usr/sbin/munin09:45:192
2567299112,6cyclictest0-21swapper/1007:45:012
2567299112,5cyclictest0-21swapper/1011:40:012
2567299112,5cyclictest0-21swapper/1011:40:012
2567299110,10cyclictest0-21swapper/1011:50:252
2567299110,10cyclictest0-21swapper/1011:50:252
2566799110,10cyclictest17324-21/usr/sbin/munin11:10:0011
2565499112,6cyclictest0-21swapper/612:15:018
2565499112,6cyclictest0-21swapper/609:40:008
2565499112,5cyclictest0-21swapper/608:09:598
2565499111,6cyclictest0-21swapper/611:55:008
2565499111,6cyclictest0-21swapper/611:54:598
2564899112,6cyclictest0-21swapper/510:40:007
2564899112,6cyclictest0-21swapper/510:05:267
2564599110,10cyclictest10256-21/usr/sbin/munin10:15:266
2563999110,3cyclictest30904-21/usr/sbin/munin07:15:305
2563599110,10cyclictest9781-21perl11:40:174
2563599110,10cyclictest9781-21perl11:40:174
2562299118,2cyclictest29229-21turbostat07:15:010
275991100,5ptp4l21-21ksoftirqd/110:40:281
275991100,2ptp4l0-21swapper/112:30:221
275991100,1ptp4l4557-21smartctl12:15:171
275991100,1ptp4l22818-21fw_conntrack11:15:171
275991100,1ptp4l22034-21load11:55:201
275991100,1ptp4l22034-21load11:55:201
275991100,1ptp4l18045-21cpuspeed11:10:131
275991100,1ptp4l0-21swapper/111:40:171
275991100,1ptp4l0-21swapper/111:40:171
275991100,1ptp4l0-21swapper/109:00:201
275991100,0ptp4l0-21swapper/111:35:171
275991100,0ptp4l0-21swapper/111:35:171
275991100,0ptp4l0-21swapper/110:55:111
275991100,0ptp4l0-21swapper/110:15:161
275991100,0ptp4l0-21swapper/109:15:251
275991100,0ptp4l0-21swapper/108:55:161
275991100,0ptp4l0-21swapper/108:40:401
275991100,0ptp4l0-21swapper/108:30:181
275991100,0ptp4l0-21swapper/108:20:001
275991100,0ptp4l0-21swapper/107:30:161
275991100,0ptp4l0-21swapper/107:17:511
2567699106,3cyclictest31626-21/usr/sbin/munin12:10:223
2567699102,5cyclictest0-21swapper/1111:15:003
2567699101,5cyclictest0-21swapper/1111:00:003
2567699100,5cyclictest0-21swapper/1108:35:003
2567299102,6cyclictest0-21swapper/1011:00:002
2567299100,8cyclictest0-21swapper/1008:30:002
2566799102,5cyclictest0-21swapper/911:40:0111
2566799102,5cyclictest0-21swapper/911:40:0111
2566799102,5cyclictest0-21swapper/909:10:0011
2566799101,5cyclictest0-21swapper/907:35:2711
2566799100,9cyclictest19963-21awk09:45:2511
2566799100,9cyclictest11579-21expr12:25:1811
2566399100,9cyclictest0-21swapper/807:29:3010
2566399100,10cyclictest0-21swapper/809:05:0010
2565899102,5cyclictest0-21swapper/712:05:019
2565899102,5cyclictest0-21swapper/712:05:019
2565899100,2cyclictest0-21swapper/707:50:189
2565499106,3cyclictest9476-21wc12:20:238
2565499102,4cyclictest0-21swapper/610:00:008
2565499100,9cyclictest7016-21ls10:55:158
2564599100,9cyclictest25135-21cat09:10:256
2563999108,1cyclictest0-21swapper/309:40:185
2563999100,1cyclictest0-21swapper/311:30:205
2563999100,1cyclictest0-21swapper/311:30:205
2563599100,1cyclictest30170-21hddtemp_smartct07:15:024
2562299100,1cyclictest0-21swapper/012:00:140
2562299100,1cyclictest0-21swapper/012:00:140
220052101,6sleep90-21swapper/907:05:1211
27599190,0ptp4l0-21swapper/112:35:161
27599190,0ptp4l0-21swapper/111:30:151
27599190,0ptp4l0-21swapper/111:30:141
27599190,0ptp4l0-21swapper/110:30:171
27599190,0ptp4l0-21swapper/110:25:061
27599190,0ptp4l0-21swapper/110:20:141
27599190,0ptp4l0-21swapper/109:45:151
27599190,0ptp4l0-21swapper/109:25:171
27599190,0ptp4l0-21swapper/109:20:001
27599190,0ptp4l0-21swapper/107:35:161
256769990,8cyclictest0-21swapper/1110:05:163
256769990,1cyclictest0-21swapper/1110:30:003
256769990,1cyclictest0-21swapper/1107:15:023
256729998,1cyclictest0-21swapper/1010:00:172
256729992,4cyclictest0-21swapper/1011:50:012
256729992,4cyclictest0-21swapper/1011:50:002
256729992,4cyclictest0-21swapper/1011:25:002
256729990,9cyclictest0-21swapper/1011:15:132
256729990,8cyclictest8900-21sshd09:34:522
256729990,8cyclictest0-21swapper/1007:20:162
256729990,7cyclictest0-21swapper/1012:40:002
256729990,1cyclictest0-21swapper/1009:25:172
256679998,1cyclictest0-21swapper/910:05:0011
256679992,4cyclictest0-21swapper/911:35:0011
256679992,4cyclictest0-21swapper/911:34:5911
256679990,9cyclictest0-21swapper/909:20:2311
256679990,9cyclictest0-21swapper/908:35:2611
256679990,1cyclictest0-21swapper/912:10:2111
256639998,1cyclictest0-21swapper/811:40:1710
256639998,1cyclictest0-21swapper/811:40:1610
256639998,1cyclictest0-21swapper/811:20:1510
256639998,1cyclictest0-21swapper/808:55:1710
256639998,1cyclictest0-21swapper/808:25:1510
256639990,9cyclictest0-21swapper/812:15:2010
256639990,9cyclictest0-21swapper/809:50:2110
256639990,9cyclictest0-21swapper/808:45:2510
256639990,9cyclictest0-21swapper/808:40:3710
256639990,1cyclictest0-21swapper/808:35:2710
256589998,1cyclictest0-21swapper/709:30:259
256589991,4cyclictest0-21swapper/711:15:019
256589990,1cyclictest0-21swapper/708:20:259
256549998,1cyclictest0-21swapper/608:30:008
256549994,5cyclictest0-21swapper/611:04:558
256549993,6cyclictest0-21swapper/611:19:548
256549990,9cyclictest0-21swapper/609:00:208
256549990,8cyclictest21934-21diskmemload09:16:048
256549990,2cyclictest19990-21/usr/sbin/munin07:40:238
256489998,1cyclictest0-21swapper/511:10:157
256489997,1cyclictest0-21swapper/507:30:217
256489990,1cyclictest22676-21latency_hist10:35:017
256489990,1cyclictest0-21swapper/508:25:207
256459997,1cyclictest0-21swapper/408:25:146
256459996,2cyclictest20395-21turbostat09:54:556
256399997,1cyclictest0-21swapper/309:00:145
256399990,9cyclictest0-21swapper/309:35:145
256399990,9cyclictest0-21swapper/307:10:125
256359990,8cyclictest16008-21cut09:00:174
256359990,8cyclictest0-21swapper/208:55:154
256359990,3cyclictest32394-21sshd10:05:014
256359990,3cyclictest17751-21/usr/sbin/munin11:50:194
256359990,3cyclictest17751-21/usr/sbin/munin11:50:194
256359990,2cyclictest29606-21awk07:55:144
256359990,1cyclictest17089-21irqrtprio10:25:194
256359990,1cyclictest0-21swapper/208:40:344
256229998,1cyclictest0-21swapper/008:45:250
256229990,8cyclictest0-21swapper/011:55:140
256229990,8cyclictest0-21swapper/011:55:140
256229990,1cyclictest0-21swapper/012:40:010
256229990,1cyclictest0-21swapper/009:20:000
27609182,4phc2sys27-21ksoftirqd/207:05:374
256769987,1cyclictest0-21swapper/1110:35:143
256769981,5cyclictest0-21swapper/1111:05:003
256769980,8cyclictest0-21swapper/1109:30:143
256769980,1cyclictest0-21swapper/1109:35:173
256729987,1cyclictest0-21swapper/1010:05:192
256729987,1cyclictest0-21swapper/1007:35:142
256729980,1cyclictest0-21swapper/1011:55:242
256729980,1cyclictest0-21swapper/1011:55:242
256729980,1cyclictest0-21swapper/1011:05:132
256679987,1cyclictest0-21swapper/911:25:2311
256679987,1cyclictest0-21swapper/911:20:1511
256679982,5cyclictest0-21swapper/912:20:2411
256679980,8cyclictest0-21swapper/908:50:1711
256679980,1cyclictest0-21swapper/908:44:5911
256679980,1cyclictest0-21swapper/908:20:2211
256679980,1cyclictest0-21swapper/908:19:3811
256679980,1cyclictest0-21swapper/908:05:2211
256639987,1cyclictest0-21swapper/812:05:2110
256639987,1cyclictest0-21swapper/811:00:1510
256639987,1cyclictest0-21swapper/810:45:2510
256639987,1cyclictest0-21swapper/810:20:1610
256639987,1cyclictest0-21swapper/809:25:4410
256639987,1cyclictest0-21swapper/809:20:1710
256639987,1cyclictest0-21swapper/808:20:1310
256639987,1cyclictest0-21swapper/807:50:2010
256639987,1cyclictest0-21swapper/807:30:2110
256639980,8cyclictest0-21swapper/811:10:2010
256639980,8cyclictest0-21swapper/810:26:4710
256639980,8cyclictest0-21swapper/809:10:1910
256639980,8cyclictest0-21swapper/808:15:1910
256639980,8cyclictest0-21swapper/807:55:1710
256639980,7cyclictest0-21swapper/811:45:4010
256639980,7cyclictest0-21swapper/811:45:4010
256639980,1cyclictest0-21swapper/812:31:2710
256639980,1cyclictest0-21swapper/809:34:5210
256639980,1cyclictest0-21swapper/809:05:3010
256639980,1cyclictest0-21swapper/808:10:2810
256639980,1cyclictest0-21swapper/808:00:2010
256639980,1cyclictest0-21swapper/807:10:2110
256589987,1cyclictest0-21swapper/711:00:159
256589981,5cyclictest1386-21kworker/u24:111:30:019
256589980,8cyclictest0-21swapper/708:40:389
256589980,1cyclictest21934-21diskmemload10:16:299
256589980,1cyclictest0-21swapper/712:10:019
256549987,1cyclictest0-21swapper/609:30:148
256549987,1cyclictest0-21swapper/609:05:228
256549982,4cyclictest0-21swapper/610:10:008
256549981,4cyclictest0-21swapper/610:40:018
256549980,1cyclictest0-21swapper/607:10:138
256489987,1cyclictest0-21swapper/509:05:237
256489986,1cyclictest0-21swapper/511:55:257
256489986,1cyclictest0-21swapper/511:55:257
256489980,8cyclictest0-21swapper/509:34:597
256489980,1cyclictest0-21swapper/511:45:257
256489980,1cyclictest0-21swapper/511:45:257
256489980,1cyclictest0-21swapper/511:25:297
256459987,1cyclictest0-21swapper/407:20:016
256459986,1cyclictest11018-21/usr/sbin/munin08:10:226
256459980,8cyclictest0-21swapper/409:25:176
256459980,7cyclictest10452-21turbostat08:54:596
256459980,7cyclictest0-21swapper/409:30:146
256459980,1cyclictest0-21swapper/411:25:236
256459980,1cyclictest0-21swapper/408:55:186
256399987,1cyclictest0-21swapper/312:15:015
256399987,1cyclictest0-21swapper/310:10:215
256399984,3cyclictest8043-21taskset08:08:255
256399982,5cyclictest0-21swapper/308:45:255
256399981,6cyclictest0-21swapper/308:00:005
256399980,8cyclictest0-21swapper/311:55:205
256399980,8cyclictest0-21swapper/311:55:205
256399980,8cyclictest0-21swapper/310:05:265
256399980,1cyclictest0-21swapper/312:20:145
256399980,1cyclictest0-21swapper/311:50:205
256399980,1cyclictest0-21swapper/311:50:205
256399980,1cyclictest0-21swapper/311:20:215
256399980,1cyclictest0-21swapper/310:41:475
256399980,1cyclictest0-21swapper/307:40:225
256359987,1cyclictest0-21swapper/207:35:164
256359985,2cyclictest18690-21cut09:45:194
256359980,8cyclictest0-21swapper/211:55:144
256359980,8cyclictest0-21swapper/211:55:144
256359980,8cyclictest0-21swapper/211:35:284
256359980,8cyclictest0-21swapper/211:35:284
256359980,8cyclictest0-21swapper/211:23:584
256359980,8cyclictest0-21swapper/210:55:164
256359980,8cyclictest0-21swapper/210:45:164
256359980,8cyclictest0-21swapper/210:40:224
256359980,8cyclictest0-21swapper/210:35:144
256359980,8cyclictest0-21swapper/210:10:204
256359980,8cyclictest0-21swapper/209:10:164
256359980,8cyclictest0-21swapper/209:05:284
256359980,8cyclictest0-21swapper/207:50:264
256359980,8cyclictest0-21swapper/207:20:174
256359980,7cyclictest9064-21memory08:50:194
256359980,7cyclictest24959-21cat09:55:134
256359980,7cyclictest20146-21grep11:10:224
256359980,2cyclictest4044-21irqrtprio10:50:204
256359980,1cyclictest0-21swapper/212:20:194
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional