You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-15 - 15:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Mon Sep 15, 2025 12:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911140,57ptp4l0-21swapper/107:07:161
113392990,1sleep3321rcuc/307:05:195
1005427765,9sleep50-21swapper/507:05:107
1310926756,9sleep110-21swapper/1107:06:283
1149826552,10sleep40-21swapper/407:05:216
1323926452,10sleep70-21swapper/707:08:249
1085726452,9sleep00-21swapper/007:05:170
1311726352,9sleep60-21swapper/607:06:358
1309326150,9sleep100-21swapper/1007:06:152
32892600,0sleep30-21swapper/311:45:265
90902590,0sleep30-21swapper/311:55:155
24752560,0sleep80-21swapper/809:00:0110
1369499221,16cyclictest0-21swapper/411:30:006
1369499212,15cyclictest0-21swapper/409:00:016
1371699195,9cyclictest0-21swapper/911:49:5911
1367899180,16cyclictest0-21swapper/108:20:181
1367899180,16cyclictest0-21swapper/108:20:181
326852170,1sleep1201rcuc/108:15:011
1371699172,9cyclictest0-21swapper/909:00:0011
1371299173,9cyclictest0-21swapper/808:40:2510
1371299172,3cyclictest0-21swapper/808:50:1910
1370799171,15cyclictest0-21swapper/709:55:189
1370399173,8cyclictest0-21swapper/610:15:018
275991160,1ptp4l21-21ksoftirqd/112:10:221
275991160,1ptp4l21-21ksoftirqd/109:40:011
275991160,1ptp4l21-21ksoftirqd/109:05:211
275991160,0ptp4l0-21swapper/111:55:201
275991160,0ptp4l0-21swapper/107:35:251
1370799163,8cyclictest0-21swapper/708:40:019
1370799163,8cyclictest0-21swapper/707:40:019
1370399163,8cyclictest0-21swapper/611:15:018
1369499161,3cyclictest0-21swapper/410:20:246
1367899163,8cyclictest0-21swapper/112:25:001
1366999160,15cyclictest0-21swapper/008:15:260
1366999160,15cyclictest0-21swapper/008:15:260
94980150,12rtkit-daemon12140-21strings07:05:224
275991150,1ptp4l21-21ksoftirqd/112:05:111
275991150,1ptp4l21-21ksoftirqd/107:55:281
1371699152,7cyclictest0-21swapper/910:35:0111
1371299152,9cyclictest0-21swapper/811:10:0110
1371299152,8cyclictest0-21swapper/811:10:3210
13712991512,2cyclictest0-21swapper/811:35:1610
1370799153,8cyclictest0-21swapper/711:30:009
1370799153,8cyclictest0-21swapper/708:15:019
1370799153,8cyclictest0-21swapper/708:15:019
1369899153,7cyclictest0-21swapper/507:30:017
1369899152,8cyclictest0-21swapper/510:40:017
1368999153,7cyclictest0-21swapper/309:20:015
1368999153,7cyclictest0-21swapper/308:25:015
1368999153,7cyclictest0-21swapper/308:25:015
1368999152,8cyclictest0-21swapper/309:14:575
1368599153,7cyclictest0-21swapper/210:20:004
1367899153,7cyclictest0-21swapper/108:55:011
1367899151,10cyclictest18191-21kworker/u24:207:15:011
1366999152,7cyclictest0-21swapper/011:40:000
275991140,1ptp4l21-21ksoftirqd/108:35:321
275991140,0ptp4l0-21swapper/112:35:261
275991140,0ptp4l0-21swapper/111:43:181
1372199142,8cyclictest0-21swapper/1008:45:242
1371699143,7cyclictest0-21swapper/907:25:0111
1371699142,7cyclictest0-21swapper/910:55:0111
1371299142,7cyclictest0-21swapper/808:20:0110
1371299142,7cyclictest0-21swapper/808:20:0110
1370799143,7cyclictest0-21swapper/708:00:019
1370799143,6cyclictest0-21swapper/709:20:019
1370799142,8cyclictest0-21swapper/708:10:239
1370799142,8cyclictest0-21swapper/708:10:239
1370799142,7cyclictest0-21swapper/710:25:269
1370799142,7cyclictest0-21swapper/709:45:019
1370799142,7cyclictest0-21swapper/708:55:019
1370399142,7cyclictest0-21swapper/612:10:008
1369899142,7cyclictest0-21swapper/508:00:007
1369499143,7cyclictest0-21swapper/410:15:006
1369499142,7cyclictest0-21swapper/409:40:016
1368999142,7cyclictest0-21swapper/307:15:005
1368999140,13cyclictest0-21swapper/310:35:175
1368599143,8cyclictest0-21swapper/209:55:254
1368599142,7cyclictest0-21swapper/211:15:274
1367899144,5cyclictest0-21swapper/108:05:011
1367899143,10cyclictest0-21swapper/107:10:011
1367899142,7cyclictest0-21swapper/111:29:591
1366999143,7cyclictest0-21swapper/008:10:010
1366999143,7cyclictest0-21swapper/008:10:010
1366999142,7cyclictest0-21swapper/008:35:010
1366999140,14cyclictest0-21swapper/011:45:260
1366999140,14cyclictest0-21swapper/010:50:190
91042130,0sleep50-21swapper/508:25:427
59502130,0sleep90-21swapper/910:25:2311
40302130,0sleep00-21swapper/011:05:180
275991130,5ptp4l21-21ksoftirqd/111:45:161
275991130,1ptp4l21-21ksoftirqd/108:58:421
275991130,0ptp4l0-21swapper/111:53:261
1372599130,12cyclictest0-21swapper/1107:19:553
1372199132,7cyclictest0-21swapper/1008:20:252
1372199132,7cyclictest0-21swapper/1008:20:252
1372199132,7cyclictest0-21swapper/1007:30:262
1371699131,7cyclictest0-21swapper/910:05:0111
1371699131,7cyclictest0-21swapper/909:30:0111
1371699130,8cyclictest0-21swapper/908:35:0111
1371299133,6cyclictest0-21swapper/810:00:0110
1371299133,6cyclictest0-21swapper/807:45:0110
1370799132,7cyclictest0-21swapper/711:35:239
1370799131,8cyclictest0-21swapper/711:15:019
1369899132,7cyclictest0-21swapper/512:10:257
1369899132,7cyclictest0-21swapper/510:55:267
1369899131,7cyclictest0-21swapper/511:40:007
1368999133,6cyclictest0-21swapper/310:15:015
1368599132,7cyclictest0-21swapper/209:05:264
1367899133,6cyclictest0-21swapper/107:45:011
1367899132,6cyclictest0-21swapper/107:45:011
1366999133,6cyclictest0-21swapper/012:16:060
1366999132,6cyclictest0-21swapper/010:25:010
1366999130,13cyclictest0-21swapper/009:43:290
1366999130,13cyclictest0-21swapper/007:40:000
1366999130,12cyclictest0-21swapper/010:56:580
275991120,1ptp4l21-21ksoftirqd/108:15:231
275991120,1ptp4l21-21ksoftirqd/108:15:231
275991120,1ptp4l21-21ksoftirqd/107:20:181
275991120,0ptp4l0-21swapper/112:20:171
275991120,0ptp4l0-21swapper/112:04:271
275991120,0ptp4l0-21swapper/110:05:171
275991120,0ptp4l0-21swapper/109:40:161
275991120,0ptp4l0-21swapper/109:30:191
275991120,0ptp4l0-21swapper/109:27:581
275991120,0ptp4l0-21swapper/109:12:341
275991120,0ptp4l0-21swapper/108:41:411
1372599121,7cyclictest0-21swapper/1109:20:013
1371699122,7cyclictest0-21swapper/910:35:2411
1370799121,7cyclictest0-21swapper/710:45:019
1370799121,7cyclictest0-21swapper/708:30:019
1369899121,7cyclictest0-21swapper/510:20:017
1369499122,6cyclictest0-21swapper/410:40:246
1369499120,1cyclictest0-21swapper/409:20:006
1368999122,6cyclictest0-21swapper/310:15:015
1368999122,6cyclictest0-21swapper/307:35:005
1368599124,5cyclictest0-21swapper/207:10:254
1368599122,7cyclictest0-21swapper/208:10:014
1368599122,7cyclictest0-21swapper/208:10:004
1367899122,6cyclictest0-21swapper/109:25:011
1366999120,8cyclictest243611fschecks_count08:45:150
1366999120,8cyclictest185821missed_timers10:00:210
1366999120,12cyclictest0-21swapper/007:53:340
1366999120,11cyclictest30175-21grep08:50:260
1366999120,11cyclictest0-21swapper/008:55:170
93972110,0sleep40-21swapper/407:44:596
275991110,2ptp4l31662-21hddtemp_smartct11:00:181
275991110,2ptp4l14729-21fw_conntrack07:50:151
275991110,1ptp4l21-21ksoftirqd/111:15:151
275991110,1ptp4l21-21ksoftirqd/110:30:221
275991110,1ptp4l21-21ksoftirqd/108:45:241
275991110,1ptp4l0-21swapper/112:30:241
275991110,1ptp4l0-21swapper/111:35:121
275991110,0ptp4l0-21swapper/112:16:071
275991110,0ptp4l0-21swapper/111:34:481
275991110,0ptp4l0-21swapper/110:45:151
275991110,0ptp4l0-21swapper/109:00:121
275991110,0ptp4l0-21swapper/108:10:241
1372199112,6cyclictest0-21swapper/1009:40:252
1372199112,6cyclictest0-21swapper/1009:35:012
1371699112,5cyclictest0-21swapper/910:00:0111
1371699110,6cyclictest0-21swapper/912:40:0011
1370799118,2cyclictest466-21turbostat09:40:009
1370799118,2cyclictest4606-21turbostat09:45:009
1369899111,6cyclictest0-21swapper/511:05:017
1368999118,2cyclictest29884-21tune2fs08:10:145
1368999118,2cyclictest29884-21tune2fs08:10:145
1368999110,10cyclictest5661-21memory12:35:215
1368599112,6cyclictest0-21swapper/212:30:004
1368599112,6cyclictest0-21swapper/211:05:014
1368599112,6cyclictest0-21swapper/208:15:014
1368599112,6cyclictest0-21swapper/208:15:014
1366999116,4cyclictest10785-21cpuspeed_turbos09:10:140
1366999112,5cyclictest0-21swapper/010:30:010
1366999112,5cyclictest0-21swapper/009:15:240
1366999110,4cyclictest0-21swapper/011:45:000
1366999110,11cyclictest0-21swapper/011:14:080
1366999110,11cyclictest0-21swapper/009:58:260
1366999110,11cyclictest0-21swapper/009:47:450
1366999110,11cyclictest0-21swapper/009:36:290
1366999110,11cyclictest0-21swapper/007:55:270
1366999110,10cyclictest0-21swapper/012:05:180
123812111,7sleep80-21swapper/807:05:2510
275991100,2ptp4l21-21ksoftirqd/108:05:131
275991100,2ptp4l21-21ksoftirqd/108:05:131
275991100,2ptp4l1853-21processes10:20:281
275991100,2ptp4l16637-21aten2_r3power_c10:00:121
275991100,1ptp4l21-21ksoftirqd/110:10:291
275991100,1ptp4l12070-21df10:35:131
275991100,1ptp4l0-21swapper/110:50:161
275991100,0ptp4l0-21swapper/108:30:121
1372599108,1cyclictest0-21swapper/1110:15:013
1372199102,5cyclictest0-21swapper/1008:55:012
1372199102,5cyclictest0-21swapper/1007:20:012
1371699102,4cyclictest0-21swapper/908:25:5111
1371699100,9cyclictest9740-21df_inode09:50:1311
1371699100,9cyclictest31041-21expr07:30:1511
1371699100,1cyclictest0-21swapper/911:45:0011
1371299102,3cyclictest0-21swapper/809:50:2310
1370799107,2cyclictest0-21swapper/710:30:149
1370399100,9cyclictest0-21swapper/607:30:148
1369899100,9cyclictest0-21swapper/509:40:197
1369499100,9cyclictest1009-21snmpd08:34:576
1368999109,1cyclictest0-21swapper/310:20:235
1368999100,9cyclictest0-21swapper/311:30:145
1368599102,5cyclictest0-21swapper/209:00:014
1368599102,4cyclictest0-21swapper/211:00:004
1368599100,9cyclictest14407-21ps12:05:134
1368599100,9cyclictest10306-21needreboot08:25:474
1367899107,2cyclictest16136-21awk10:00:011
1367899107,2cyclictest0-21swapper/107:25:281
1367899100,9cyclictest2305-21grep11:05:131
1366999105,4cyclictest9908-21fschecks_time09:50:140
1366999104,5cyclictest8580-21cpuspeed_turbos08:25:400
1366999104,5cyclictest15645-21irqrtprio07:10:190
1366999103,6cyclictest19304-21cut09:20:160
1366999103,6cyclictest17359-21processes10:40:200
1366999102,7cyclictest21081-21netstat07:15:190
1366999101,5cyclictest0-21swapper/011:20:000
1366999100,9cyclictest5876-21irqrtprio11:50:170
1366999100,9cyclictest2602-21unixbench_multi12:30:230
1366999100,9cyclictest25898-21grep12:20:190
1366999100,9cyclictest17946-21needreboot08:35:210
1366999100,4cyclictest0-21swapper/008:20:590
1366999100,4cyclictest0-21swapper/008:20:580
1366999100,10cyclictest0-21swapper/011:00:280
1366999100,10cyclictest0-21swapper/010:32:160
130792105,4sleep90-21swapper/907:06:0111
27599190,1ptp4l21-21ksoftirqd/110:40:141
27599190,0ptp4l0-21swapper/111:20:131
27599190,0ptp4l0-21swapper/111:10:211
27599190,0ptp4l0-21swapper/110:55:011
27599190,0ptp4l0-21swapper/110:25:141
27599190,0ptp4l0-21swapper/110:15:011
27599190,0ptp4l0-21swapper/109:50:111
27599190,0ptp4l0-21swapper/109:45:131
27599190,0ptp4l0-21swapper/109:15:001
27599190,0ptp4l0-21swapper/108:25:411
27599190,0ptp4l0-21swapper/107:30:151
137259991,5cyclictest0-21swapper/1107:25:013
137259990,9cyclictest0-21swapper/1109:25:253
137219990,8cyclictest28245-21chrt11:38:322
137219990,8cyclictest16674-21smart_sda09:15:222
137219990,1cyclictest0-21swapper/1011:10:012
137219990,1cyclictest0-21swapper/1009:00:192
137169990,9cyclictest0-21swapper/911:10:0111
137129997,1cyclictest0-21swapper/812:30:1210
137129996,2cyclictest0-21swapper/812:10:2010
137129990,1cyclictest0-21swapper/811:25:0010
137129990,1cyclictest0-21swapper/807:45:1210
137079998,1cyclictest0-21swapper/710:10:259
137079996,2cyclictest26979-21memory08:05:189
137079996,2cyclictest26979-21memory08:05:189
137079996,2cyclictest11121-21if_err_enp1s007:45:159
137079991,7cyclictest18191-21kworker/u24:210:55:019
137039996,2cyclictest0-21swapper/610:30:018
137039995,3cyclictest12816-21cut12:05:008
137039990,9cyclictest0-21swapper/608:40:248
137039990,1cyclictest0-21swapper/608:20:168
137039990,1cyclictest0-21swapper/608:20:168
136989996,2cyclictest0-21swapper/508:20:237
136989996,2cyclictest0-21swapper/508:20:237
136989990,1cyclictest0-21swapper/512:30:147
136989990,1cyclictest0-21swapper/510:45:267
136949996,2cyclictest0-21swapper/407:10:196
136949990,9cyclictest0-21swapper/409:20:166
136949990,8cyclictest0-21swapper/412:25:156
136949990,8cyclictest0-21swapper/408:25:436
136899998,1cyclictest0-21swapper/309:50:235
136899997,1cyclictest0-21swapper/307:15:265
136899990,9cyclictest0-21swapper/312:20:125
136899990,9cyclictest0-21swapper/310:25:245
136899990,9cyclictest0-21swapper/308:35:165
136899990,1cyclictest0-21swapper/311:10:235
136899990,1cyclictest0-21swapper/309:55:155
136899990,1cyclictest0-21swapper/308:30:175
136859998,1cyclictest0-21swapper/211:25:154
136859990,8cyclictest18835-21expr09:20:144
136859990,8cyclictest11227-21irqrtprio07:45:154
136699992,6cyclictest26304-21cpuspeed_turbos09:30:120
136699990,9cyclictest0-21swapper/012:02:150
136699990,9cyclictest0-21swapper/011:30:080
136699990,9cyclictest0-21swapper/010:45:080
136699990,8cyclictest9962-21cpuspeed07:45:100
136699990,8cyclictest8545-21tr11:55:130
136699990,8cyclictest6991-21cpuspeed_turbos09:05:150
136699990,8cyclictest6065-21cpu07:40:120
136699990,8cyclictest3797-21cpu12:35:140
136699990,8cyclictest30778-21grep07:30:140
136699990,8cyclictest29224-21grep08:10:120
136699990,8cyclictest29224-21grep08:10:120
136699990,8cyclictest28288-21aten2_r3power_e12:25:140
136699990,8cyclictest2783-21cpu09:00:130
136699990,8cyclictest26934-21cpuspeed_turbos07:25:160
136699990,8cyclictest24417-21cpu10:10:130
136699990,8cyclictest23142-21grep07:20:120
136699990,8cyclictest21472-21cpuspeed08:00:140
136699990,8cyclictest20880-21cpuspeed_turbos10:05:160
136699990,8cyclictest20048-21cpuspeed_turbos08:40:140
136699990,8cyclictest17589-21entropy12:10:140
136699990,8cyclictest17408-21chrt11:25:090
136699990,8cyclictest13911-21cut11:20:130
136699990,8cyclictest11609-21cut10:35:130
136699990,8cyclictest1013-21lldpd09:25:200
136699990,3cyclictest0-21swapper/010:15:070
9498080,2rtkit-daemon0-21swapper/812:25:5310
137259987,1cyclictest0-21swapper/1108:30:003
137259987,1cyclictest0-21swapper/1108:05:233
137259987,1cyclictest0-21swapper/1108:05:233
137259985,2cyclictest29670-21df_inode11:40:143
137259982,4cyclictest0-21swapper/1111:35:003
137259982,4cyclictest0-21swapper/1110:10:013
137259982,3cyclictest0-21swapper/1108:25:013
137259982,3cyclictest0-21swapper/1108:25:013
137259981,5cyclictest0-21swapper/1111:20:003
137259981,5cyclictest0-21swapper/1111:10:013
137259980,8cyclictest0-21swapper/1111:00:163
137259980,8cyclictest0-21swapper/1109:55:153
137259980,8cyclictest0-21swapper/1107:30:003
137259980,1cyclictest7676-21munin-run11:55:003
137259980,1cyclictest0-21swapper/1111:35:193
137259980,1cyclictest0-21swapper/1110:30:153
137219987,1cyclictest0-21swapper/1012:25:242
137219987,1cyclictest0-21swapper/1008:20:002
137219987,1cyclictest0-21swapper/1008:20:002
137219985,2cyclictest19944-21awk10:05:012
137219980,1cyclictest0-21swapper/1008:35:182
137219980,1cyclictest0-21swapper/1008:10:122
137219980,1cyclictest0-21swapper/1008:10:122
137169987,1cyclictest0-21swapper/910:20:3111
137169987,1cyclictest0-21swapper/910:15:2511
137169987,1cyclictest0-21swapper/909:40:1911
137169987,1cyclictest0-21swapper/907:55:2011
137169985,2cyclictest26822-21awk08:50:0011
137169980,7cyclictest9967-21diskmemload11:00:4811
137169980,1cyclictest0-21swapper/912:15:1911
137169980,1cyclictest0-21swapper/912:05:2011
137169980,1cyclictest0-21swapper/911:55:2211
137169980,1cyclictest0-21swapper/911:30:2011
137169980,1cyclictest0-21swapper/909:35:2611
137129987,1cyclictest0-21swapper/811:00:1510
137129987,1cyclictest0-21swapper/808:25:4210
137129987,1cyclictest0-21swapper/807:15:2410
137129986,1cyclictest9967-21diskmemload09:27:4910
137129983,4cyclictest4117-21gdbus12:00:1310
137129980,8cyclictest0-21swapper/809:45:1410
137129980,2cyclictest9967-21diskmemload09:31:3510
137129980,1cyclictest0-21swapper/812:05:0410
137129980,1cyclictest0-21swapper/809:40:2010
137079987,1cyclictest12080-40dnf11:15:279
137079987,1cyclictest0-21swapper/711:45:009
137079987,1cyclictest0-21swapper/709:25:209
137079987,1cyclictest0-21swapper/707:40:189
137079987,1cyclictest0-21swapper/707:20:159
137079986,1cyclictest0-21swapper/711:30:589
137079980,8cyclictest0-21swapper/708:30:139
137079980,7cyclictest9967-21diskmemload12:17:129
137079980,1cyclictest0-21swapper/707:15:209
137039987,1cyclictest0-21swapper/611:05:178
137039987,1cyclictest0-21swapper/610:20:248
137039987,1cyclictest0-21swapper/608:45:188
137039986,1cyclictest1020-21runrttasks11:51:148
137039985,2cyclictest32330-21memory11:00:218
137039980,8cyclictest0-21swapper/610:50:188
137039980,8cyclictest0-21swapper/610:30:158
137039980,8cyclictest0-21swapper/608:35:268
137039980,1cyclictest0-21swapper/609:25:308
137039980,1cyclictest0-21swapper/607:45:238
136989987,1cyclictest0-21swapper/510:50:167
136989987,1cyclictest0-21swapper/509:30:177
136989987,1cyclictest0-21swapper/508:50:257
136989980,8cyclictest0-21swapper/512:15:247
136989980,8cyclictest0-21swapper/511:06:037
136989980,8cyclictest0-21swapper/509:55:177
136989980,8cyclictest0-21swapper/508:00:157
136989980,1cyclictest0-21swapper/509:10:267
136989980,1cyclictest0-21swapper/508:05:237
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional