You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 20:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot4s.osadl.org (updated Fri Nov 28, 2025 12:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
266806521170,0sleep1281rcuc/112:21:181
2354836995244,5cyclictest2550209-21latency_hist10:25:000
2354839995041,6cyclictest2634557-21latency_hist11:50:001
23736162490,3sleep20-21swapper/207:26:162
2354844994944,4cyclictest2381724-21kworker/u8:3+flush-8:008:36:232
2354844994735,9cyclictest282-21systemd-journal10:44:592
2354839994735,9cyclictest282-21systemd-journal10:56:331
2354836994737,8cyclictest2460079-21ntpq08:51:200
2354839994638,5cyclictest2490896-21/usr/sbin/munin09:26:181
2354836994635,8cyclictest2653486-21ntpq12:06:230
2354844994536,7cyclictest282-21systemd-journal07:55:002
2354839994528,6cyclictest282-21systemd-journal08:10:321
2354836994339,3cyclictest1519662-21kworker/u8:0+flush-8:007:16:280
2354836994336,5cyclictest2376501-21kworker/u8:2+flush-8:009:26:200
2354836994334,6cyclictest2669133-21latency_hist12:25:010
2354839994234,5cyclictest2579868-21latency_hist10:55:011
2354844994133,5cyclictest2679025-21apt-get12:35:002
2354844994133,5cyclictest2679025-21apt-get12:35:002
2354839994132,6cyclictest2678268-21ntpq12:31:281
2354839994132,6cyclictest2678268-21ntpq12:31:281
2354839994111,15cyclictest0-21swapper/108:51:071
235483699411,37cyclictest2684231-21true12:40:000
2354836994031,7cyclictest2405579-21ntpq07:56:180
2354839993931,6cyclictest2505446-21latency_hist09:40:011
2354839993928,8cyclictest2603915-21ntpq11:16:191
2354836993930,6cyclictest1519662-21kworker/u8:0+flush-8:007:31:110
2354836993928,8cyclictest2669712-21diskstats12:26:090
2354836993928,8cyclictest2669712-21diskstats12:26:090
2354844993833,4cyclictest2520488-21kworker/u8:0+flush-8:010:06:252
2354839993831,5cyclictest2396906-21apt-get07:51:081
2354836993833,4cyclictest2630505-21kworker/u8:0+events_unbound11:51:160
2354836993830,6cyclictest2395744-21ntpq07:46:220
2354836993827,9cyclictest2509535-21ntpq09:41:200
23500032383,9sleep20-21swapper/207:05:092
2354844993731,4cyclictest2359726-21turbostat.cron07:14:592
2354839993726,8cyclictest282-21systemd-journal07:23:331
2354836993730,5cyclictest2623754-21ntpq11:36:230
2354836993730,5cyclictest2386430-21apt-get07:40:010
2354839993627,6cyclictest282-21systemd-journal09:48:341
235483999357,9cyclictest0-21swapper/112:26:061
235483999357,9cyclictest0-21swapper/112:26:061
2354839993524,8cyclictest2480696-21latency_hist09:15:011
2354839993524,8cyclictest2480696-21latency_hist09:15:011
2354836993525,7cyclictest2629530-21latency_hist11:45:010
2354839993429,3cyclictest2531098-21irqcore10:06:151
2354839993425,7cyclictest2609746-21latency_hist11:25:011
2354836993428,4cyclictest2376501-21kworker/u8:2+flush-8:009:51:120
23543812343,9sleep10-21swapper/107:06:451
2354844993327,4cyclictest2376501-21kworker/u8:2+events_unbound10:01:212
2354839993325,5cyclictest2684201-21latency_hist12:40:011
2354836993326,5cyclictest2381724-21kworker/u8:3+flush-8:008:56:150
2354836993323,8cyclictest2381724-21kworker/u8:3+events_unbound09:01:280
235483699332,26cyclictest2340027-21kworker/0:1+events08:05:400
2354839993228,2cyclictest2443657-21kthreadcore08:36:141
2354839993223,7cyclictest2471132-21/usr/sbin/munin09:06:201
2354839993223,7cyclictest2471132-21/usr/sbin/munin09:06:201
235483699322,25cyclictest2490606-21kworker/0:0+events09:30:190
235483999313,15cyclictest2618734-21ntpq11:31:201
2354839993126,3cyclictest2536382-21kthreadcore10:11:161
2354839993124,5cyclictest2441153-21latency_hist08:35:011
2354839993124,5cyclictest2406414-21latency_hist08:00:011
2354839993122,7cyclictest2564207-21ntpq10:36:181
2354839993116,5cyclictest2579605-21kworker/u8:3+flush-8:011:50:041
2354836993124,5cyclictest282-21systemd-journal10:11:100
235483699312,28cyclictest2594946-21kworker/0:2+events11:59:400
2354836993119,9cyclictest2618750-21ntpq11:31:200
2354844993027,2cyclictest2381445-21cron07:35:002
2354844993025,4cyclictest2520488-21kworker/u8:0+flush-8:011:16:212
235483999309,10cyclictest0-21swapper/107:26:081
235483999306,12cyclictest2395803-21ntpq07:46:241
235483999303,16cyclictest2623799-21ntpq11:36:241
235483999303,15cyclictest2663382-21ntpq12:16:201
235483999303,15cyclictest2509603-21ntpq09:41:221
235483999302,17cyclictest2545372-21latency_hist10:20:011
2354839993020,8cyclictest2390759-21ntpq07:41:201
2354839993019,8cyclictest2426749-21diskstats08:21:121
2354836993022,6cyclictest2520488-21kworker/u8:0+events_unbound10:35:010
2354836993021,7cyclictest2504560-21ntpq09:36:210
235483699300,25cyclictest2490606-21kworker/0:0+events09:55:520
235483699300,25cyclictest2490606-21kworker/0:0+events09:45:160
2354844992924,4cyclictest2579605-21kworker/u8:3+flush-8:011:01:292
2354844992924,4cyclictest2376501-21kworker/u8:2+events_unbound10:51:212
235483999298,3cyclictest0-21swapper/109:51:151
235483999296,10cyclictest2590277-21sshd11:05:341
235483999293,15cyclictest2653460-21ntpq12:06:221
235483999293,14cyclictest2574109-21ntpq10:46:221
2354839992925,2cyclictest2481752-21irqrtprio09:16:161
2354839992925,2cyclictest2481752-21irqrtprio09:16:151
2354839992924,3cyclictest2362210-21cpuspeed_turbos07:16:111
2354839992923,4cyclictest2520596-21apt-get09:56:081
235483999292,15cyclictest2489721-21ntpq09:21:191
235483999292,15cyclictest0-21swapper/111:11:071
235483999292,10cyclictest2416445-21latency_hist08:10:011
235483999291,18cyclictest2381412-21runrttasks07:34:361
235483699293,25cyclictest2340027-21kworker/0:1+events07:11:400
2354836992925,2cyclictest2654927-21fschecks_count12:11:110
2354836992924,3cyclictest2578957-21needreboot10:51:200
2354836992923,4cyclictest2520488-21kworker/u8:0+flush-8:010:36:170
2354836992922,5cyclictest2583915-21journal-offline10:56:190
235483699292,22cyclictest2451061-21kworker/0:1+events08:45:320
235483699292,22cyclictest2451061-21kworker/0:1+events08:45:310
2354836992921,5cyclictest282-21systemd-journal08:35:000
2354836992920,7cyclictest2431079-21apt-get08:25:010
235483699291,26cyclictest2376200-21sshd07:26:330
2354844992824,3cyclictest2376501-21kworker/u8:2+flush-8:010:11:212
2354844992823,4cyclictest2630505-21kworker/u8:0+flush-8:011:46:232
2354844992823,4cyclictest2520488-21kworker/u8:0+events_unbound10:26:212
2354844992820,6cyclictest282-21systemd-journal07:20:002
235483999285,12cyclictest2455198-21ntpq08:46:221
235483999285,12cyclictest2455198-21ntpq08:46:221
235483999284,11cyclictest2465091-21ntpq08:56:221
235483999283,12cyclictest2610084-21apt-get11:26:111
2354839992823,3cyclictest2640566-21irqrtprio11:56:131
235483999282,15cyclictest2496476-21irqcore09:31:151
235483999282,14cyclictest2559339-21ntpq10:31:241
235483999282,12cyclictest2525675-21aten2_r5power_v10:01:101
2354839992817,8cyclictest282-21systemd-journal10:20:011
2354839992812,11cyclictest0-21swapper/108:16:071
2354836992825,2cyclictest2386433-21sendmail07:40:000
2354836992823,3cyclictest2639533-21kworker/u8:1+flush-8:012:16:150
2354836992823,3cyclictest2594917-21cron11:10:000
235483699282,25cyclictest2594946-21kworker/0:2+events11:22:360
235483699282,25cyclictest2530204-21kworker/0:2+events10:44:360
235483699282,21cyclictest2594946-21kworker/0:2+events12:00:520
235483699282,21cyclictest2340027-21kworker/0:1+events08:15:320
2354836992821,5cyclictest2629887-21/usr/sbin/munin11:46:220
2354836992820,6cyclictest2589779-21apt-get11:05:010
235483699281,26cyclictest2490606-21kworker/0:0+events10:19:440
2354844992724,2cyclictest2584473-21unixbench_singl10:56:272
2354844992724,2cyclictest2391263-21unixbench_multi07:41:292
2354844992723,2cyclictest2610437-21fschecks_time11:26:152
2354844992723,2cyclictest2595370-21aten2_r5power_v11:11:112
2354844992723,2cyclictest2431860-21cpuspeed_turbos08:26:092
2354844992723,2cyclictest2406825-21aten2_r5power_e08:01:092
2354844992722,4cyclictest2376501-21kworker/u8:2+events_unbound09:46:202
2354844992722,3cyclictest2515877-21cpuspeed_turbos09:51:112
2354844992722,3cyclictest2386056-21unixbench_multi07:36:272
235483999274,12cyclictest2658509-21ntpq12:11:231
235483999274,11cyclictest2410512-21ntpq08:01:211
235483999273,8cyclictest2645542-21sed12:01:141
235483999273,12cyclictest2628777-21ntpq11:41:251
235483999272,9cyclictest2554398-21ntpq10:26:251
235483999272,11cyclictest2586311-21kthreadcore11:01:181
235483999272,11cyclictest2568849-21kthreadcore10:41:161
235483999272,11cyclictest2449809-21kthreadcore08:41:171
235483999272,11cyclictest2382086-21fschecks_count07:36:111
2354839992719,5cyclictest2359719-21latency_hist07:15:011
235483999271,17cyclictest2470559-21taskset09:01:451
235483699272,24cyclictest2594946-21kworker/0:2+events11:12:440
2354836992721,4cyclictest2381724-21kworker/u8:3+flush-8:008:36:090
2354836992721,4cyclictest2376501-21kworker/u8:2+flush-8:008:21:250
2354836992718,7cyclictest2479853-21ntpq09:11:190
2354836992718,7cyclictest2479853-21ntpq09:11:190
235483699271,25cyclictest2490606-21kworker/0:0+events10:08:440
235483699271,25cyclictest2451061-21kworker/0:1+events09:18:000
235483699271,25cyclictest2451061-21kworker/0:1+events09:18:000
235483699271,25cyclictest2340027-21kworker/0:1+events08:04:040
235483699270,22cyclictest2451061-21kworker/0:1+events09:21:000
2354844992621,4cyclictest2520488-21kworker/u8:0+events_unbound10:16:242
2354844992621,3cyclictest2452300-21kthreadcore08:46:152
2354844992621,3cyclictest2452300-21kthreadcore08:46:142
2354844992619,5cyclictest282-21systemd-journal11:55:002
235483999262,14cyclictest2435464-21ntpq08:26:221
2354836992619,5cyclictest2579605-21kworker/u8:3+flush-8:012:31:250
2354836992619,5cyclictest2579605-21kworker/u8:3+flush-8:012:31:250
235483699260,21cyclictest2594946-21kworker/0:2+events11:15:080
2354844992519,4cyclictest2480308-21unixbench_multi09:11:252
2354844992519,4cyclictest2480308-21unixbench_multi09:11:252
2354844992517,5cyclictest282-21systemd-journal09:25:002
235483699252,22cyclictest2451061-21kworker/0:1+events09:08:320
235483699252,22cyclictest2451061-21kworker/0:1+events09:08:320
235483699252,22cyclictest2340027-21kworker/0:1+events07:54:120
2354836992521,3cyclictest2312954-21kworker/u8:1+flush-8:008:11:310
235483699250,20cyclictest2594946-21kworker/0:2+events11:25:270
2354844992421,2cyclictest2644908-21aten2_r5power_p12:01:092
2354844992420,3cyclictest2376501-21kworker/u8:2+flush-8:012:36:242
2354844992420,2cyclictest0-21swapper/212:00:002
2354844992415,7cyclictest2485629-21awk09:20:002
2354844992415,7cyclictest2485629-21awk09:20:002
235483699242,21cyclictest2530204-21kworker/0:2+events10:47:040
235483699242,21cyclictest2490606-21kworker/0:0+events10:00:400
235483699242,21cyclictest2431114-21kworker/0:0+events08:42:280
235483699242,21cyclictest2340027-21kworker/0:1+events07:21:320
235483699241,22cyclictest2530204-21kworker/0:2+events10:25:240
25496772230,5sleep20-21swapper/210:21:242
2354844992320,2cyclictest2593985-21needreboot11:06:192
2354844992318,3cyclictest2510085-21unixbench_singl09:41:282
2354844992218,2cyclictest2605460-21fschecks_time11:21:112
2354844992218,2cyclictest2570583-21cpuspeed_turbos10:46:112
2354844992218,2cyclictest2505069-21unixbench_multi09:36:282
2354844992218,2cyclictest0-21swapper/208:21:092
2354844992217,4cyclictest2376501-21kworker/u8:2+flush-8:011:36:182
235456922210,6sleep0831-21mta-sts-daemon07:09:130
2354844992112,7cyclictest282-21systemd-journal11:41:292
25829662200,2sleep3481ktimers/310:56:173
2354844992016,2cyclictest2446522-21aten2_r5power_e08:41:092
2354844992016,2cyclictest0-21swapper/211:31:082
2354844992015,3cyclictest2499852-21mailstats09:31:262
235484499199,7cyclictest2421769-21seq08:16:102
2354844991916,2cyclictest0-21swapper/208:00:002
2354844991915,3cyclictest0-21swapper/209:00:012
2354844991915,2cyclictest0-21swapper/207:46:252
2354844991914,3cyclictest2495190-21unixbench_multi09:26:272
23544452194,8sleep30-21swapper/307:07:353
235484499189,7cyclictest2672547-21tr12:26:152
235484499189,7cyclictest2672547-21tr12:26:152
235484499188,2cyclictest0-21swapper/208:51:082
2354844991814,3cyclictest0-21swapper/212:10:002
2354844991814,2cyclictest0-21swapper/212:11:242
2354844991814,2cyclictest0-21swapper/209:56:272
2354844991814,2cyclictest0-21swapper/209:06:132
2354844991814,2cyclictest0-21swapper/209:06:132
2354844991814,2cyclictest0-21swapper/209:01:192
2354844991814,2cyclictest0-21swapper/207:21:262
2354844991812,4cyclictest2661841-21tr12:16:152
235484499178,7cyclictest2439019-21tr08:31:162
235484499172,14cyclictest0-21swapper/210:31:272
2354844991713,2cyclictest0-21swapper/212:21:262
2354844991713,2cyclictest0-21swapper/208:11:142
24513782162,9sleep30-21swapper/308:46:073
24513782162,9sleep30-21swapper/308:46:073
235484499167,7cyclictest2411701-21seq08:06:062
2354844991612,2cyclictest0-21swapper/210:36:092
25754702132,7sleep30-21swapper/310:51:123
24816242130,9sleep30-21swapper/309:16:173
24816242130,9sleep30-21swapper/309:16:173
24682672132,6sleep30-21swapper/309:01:243
26694662122,6sleep30-21swapper/312:26:073
26694662122,6sleep30-21swapper/312:26:073
26544482122,5sleep30-21swapper/312:10:023
26501912122,6sleep30-21swapper/312:06:163
26449852122,6sleep30-21swapper/312:01:113
26438422122,6sleep30-21swapper/311:56:253
26356092122,6sleep30-21swapper/311:51:143
26242812122,6sleep30-21swapper/311:36:313
26163212123,5sleep30-21swapper/311:31:173
26103332123,6sleep30-21swapper/311:26:153
25892322121,7sleep30-21swapper/311:01:303
25692062122,6sleep30-21swapper/310:41:213
25608222122,6sleep30-21swapper/310:36:113
25592282122,6sleep30-21swapper/310:31:223
25551422123,6sleep30-21swapper/310:30:013
25409832122,6sleep30-21swapper/310:16:133
25194962122,6sleep30-21swapper/309:51:253
25134402122,6sleep30-21swapper/309:46:193
24976532122,6sleep30-21swapper/309:31:183
24559712122,6sleep30-21swapper/308:50:013
24344172122,6sleep30-21swapper/308:26:173
24269702122,6sleep30-21swapper/308:21:143
24216242122,6sleep30-21swapper/308:16:103
24070162122,6sleep30-21swapper/308:01:133
23965932122,6sleep30-21swapper/307:50:013
23924982122,6sleep30-21swapper/307:46:163
23770662122,6sleep30-21swapper/307:31:103
23715592122,6sleep30-21swapper/307:25:013
23583082122,6sleep30-21swapper/307:11:213
26834302112,5sleep30-21swapper/312:36:263
26782642112,5sleep30-21swapper/312:31:293
26782642112,5sleep30-21swapper/312:31:293
26642182112,5sleep30-21swapper/312:20:013
26595512113,5sleep30-21swapper/312:15:513
26344992112,6sleep30-21swapper/311:49:273
26249032112,5sleep30-21swapper/311:41:083
26048302112,5sleep30-21swapper/311:20:013
26005612112,5sleep30-21swapper/311:16:123
25972552112,5sleep30-21swapper/311:11:183
25940482112,5sleep30-21swapper/311:06:213
25703632112,5sleep30-21swapper/310:46:103
25465002112,5sleep30-21swapper/310:21:143
25358822112,5sleep30-21swapper/310:11:153
25308802112,6sleep30-21swapper/310:06:133
25291632112,5sleep30-21swapper/310:01:203
25214832110,7sleep30-21swapper/309:56:173
25096852112,5sleep30-21swapper/309:41:253
25032432113,5sleep30-21swapper/309:36:203
24912982112,5sleep30-21swapper/309:26:123
24859602112,5sleep30-21swapper/309:21:073
24759672112,5sleep30-21swapper/309:10:073
24759672112,5sleep30-21swapper/309:10:063
24715032112,5sleep30-21swapper/309:06:143
24715032112,5sleep30-21swapper/309:06:143
24612422112,5sleep30-21swapper/308:56:073
24462232112,5sleep30-21swapper/308:40:013
24414592112,5sleep30-21swapper/308:36:043
24405602112,5sleep30-21swapper/308:31:273
24169822112,5sleep30-21swapper/308:11:163
24116722112,5sleep30-21swapper/308:06:063
24017202112,6sleep30-21swapper/307:55:193
23869562112,5sleep30-21swapper/307:40:423
23834192111,6sleep30-21swapper/307:36:173
23669222112,5sleep30-21swapper/307:21:083
23615762112,5sleep30-21swapper/307:15:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional