You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-17 - 23:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Dec 17, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
33612240164,25sleep10-21swapper/107:08:221
32512236163,24sleep20-21swapper/207:06:572
33632203166,24sleep30-21swapper/307:08:233
32242199162,25sleep00-21swapper/007:06:360
429621600,5sleep0356599cyclictest08:20:230
2511121540,4sleep1357499cyclictest09:10:121
586421460,5sleep3358699cyclictest07:15:023
586421460,5sleep3358699cyclictest07:15:013
1176821430,4sleep2357999cyclictest11:05:192
1770321370,4sleep1357499cyclictest10:05:191
1714121320,3sleep0356599cyclictest08:50:220
3574997466,3cyclictest0-21swapper/109:09:471
158192650,2sleep00-21swapper/011:15:170
120872650,4sleep212091-21ntp_kernel_err07:25:192
3772550,5sleep2357999cyclictest11:55:112
3772550,5sleep2357999cyclictest11:55:112
34342550,2sleep20-21swapper/212:00:172
80452520,1sleep10-21swapper/107:15:241
80452520,1sleep10-21swapper/107:15:231
3574994946,2cyclictest0-21swapper/109:47:301
245022440,1sleep20-21swapper/207:55:132
3574994238,3cyclictest0-21swapper/110:28:271
357999411,8cyclictest7233-21cstates12:10:112
3579993612,21cyclictest28302-21/usr/sbin/munin11:45:112
357999354,4cyclictest13143-21unixbench-2d08:40:242
3579993511,3cyclictest13548-21sensors_temp09:55:222
357999349,2cyclictest13597-21hddtemp_smartct11:10:142
356599348,12cyclictest0-21swapper/011:45:010
357999339,3cyclictest4120-21/usr/sbin/munin10:50:202
357999338,3cyclictest6666-21/usr/sbin/munin07:15:162
357999338,3cyclictest6666-21/usr/sbin/munin07:15:162
357999329,4cyclictest25469-21unixbench_singl11:35:252
357999329,4cyclictest0-21swapper/210:35:172
357999329,3cyclictest3496-21/usr/sbin/munin09:35:212
357999329,3cyclictest3496-21/usr/sbin/munin09:35:212
357999328,2cyclictest831-21runrttasks08:35:132
357999322,7cyclictest2903-21taskset09:30:262
357999322,11cyclictest10163-21cpuspeed_turbos09:50:102
357999321,6cyclictest831-21runrttasks08:56:192
3579993211,3cyclictest26551-21ntp_states10:25:192
3579993211,3cyclictest21093-21/usr/sbin/munin10:15:202
3579993211,17cyclictest27785-21tr10:30:122
3579993210,3cyclictest28768-21expr08:05:122
356599326,12cyclictest30283-21awk11:50:010
357999318,3cyclictest3773-21/usr/sbin/munin07:10:292
357999318,3cyclictest3773-21/usr/sbin/munin07:10:282
3579993111,3cyclictest26365-21expr11:40:112
357999309,3cyclictest15120-21/usr/sbin/munin11:15:222
357999308,4cyclictest23680-21ls11:35:002
357999308,3cyclictest14990-21sshd07:32:102
357999308,2cyclictest831-21runrttasks12:26:372
357999303,4cyclictest23208-21fschecks_time09:05:152
357999301,9cyclictest18669-21munin-run10:10:012
357999301,5cyclictest6322-21/usr/sbin/munin10:55:192
357999301,5cyclictest14488-21/usr/sbin/munin10:00:192
3579993010,3cyclictest30840-21/usr/sbin/munin08:10:132
3579993010,3cyclictest30494-21/usr/sbin/munin11:50:212
358699294,22cyclictest15841-21fschecks_count07:35:153
357999299,3cyclictest831-21runrttasks08:51:492
357999299,3cyclictest23225-21processes07:50:222
357999299,3cyclictest18564-21load07:40:182
357999299,2cyclictest831-21runrttasks12:39:572
357999298,4cyclictest16715-21smart_sda07:35:242
357999298,3cyclictest831-21runrttasks12:08:382
357999298,3cyclictest11934-21grep12:20:142
357999298,2cyclictest831-21runrttasks07:46:202
357999291,3cyclictest12050irq/123-ahci[0008:05:022
3579992911,3cyclictest3962-21sh08:20:212
357999289,3cyclictest9877-21netstat07:20:192
357999289,3cyclictest25788-21load09:10:182
357999288,3cyclictest0-21swapper/210:20:202
357999287,3cyclictest17444-21cpuspeed_turbos11:20:122
357999281,9cyclictest1681-21latency_hist10:45:022
357999281,11cyclictest0-21swapper/210:13:142
3579992810,3cyclictest13909-21cpu08:45:122
358699273,22cyclictest41-21ksoftirqd/307:45:223
357999279,3cyclictest17787-21sed12:35:022
357999278,3cyclictest28152-21open_files09:15:202
3579992715,7cyclictest0-21swapper/211:26:532
3574992710,6cyclictest0-21swapper/109:54:001
358699263,21cyclictest41-21ksoftirqd/309:20:183
357999269,4cyclictest8079-21cstates09:45:112
357999269,3cyclictest87150irq/126-enp1s0-09:20:122
357999268,3cyclictest20800-21expr09:00:122
357999265,3cyclictest87450irq/129-enp1s0-08:15:122
357999261,4cyclictest5085-21/usr/sbin/munin08:25:122
357999261,4cyclictest5085-21/usr/sbin/munin08:25:122
357999261,13cyclictest0-21swapper/212:19:342
357499261,16cyclictest0-21swapper/108:10:121
356599261,13cyclictest8292-21munin-run11:00:010
358699252,21cyclictest39-21rcuc/310:30:253
358699251,21cyclictest10722-21/usr/sbin/munin11:05:193
357999250,14cyclictest0-21swapper/210:46:402
357999250,14cyclictest0-21swapper/210:46:402
3574992510,3cyclictest0-21swapper/107:10:321
3574992510,3cyclictest0-21swapper/107:10:311
357499250,19cyclictest0-21swapper/109:35:101
357499250,19cyclictest0-21swapper/109:35:101
356599256,16cyclictest15508-21munin-run12:30:010
358699242,20cyclictest39-21rcuc/311:40:223
358699241,20cyclictest4261-21if_enp2s009:35:163
358699241,20cyclictest4261-21if_enp2s009:35:163
358699241,11cyclictest2876-21/usr/sbin/munin08:20:243
357999247,3cyclictest32628-21proc_pri09:25:232
357999240,22cyclictest0-21swapper/211:02:552
357499241,13cyclictest0-21swapper/108:55:071
357499240,6cyclictest0-21swapper/112:03:121
356599249,3cyclictest15427-21unixbench_singl08:45:250
356599241,12cyclictest26858-21idleruntime-cro09:15:010
356599241,11cyclictest23075-21awk10:20:010
358699231,20cyclictest401ktimersoftd/310:50:123
358699231,11cyclictest775-21ntpd09:14:453
358699231,11cyclictest25107-21sshd10:23:103
358699231,11cyclictest16696-21/usr/sbin/munin10:05:253
358699231,11cyclictest16071-21/usr/sbin/munin08:50:153
358699231,10cyclictest11602-21sed09:50:233
357999237,3cyclictest0-21swapper/209:41:052
3574992317,3cyclictest13428-21pmu-power09:55:221
357499231,4cyclictest15797-21hddtemp_smartct11:15:161
357499231,16cyclictest0-21swapper/108:15:121
358699221,7cyclictest26400-21memory10:25:163
358699221,7cyclictest22923-21sshd10:18:233
358699221,5cyclictest0-21swapper/310:35:123
358699221,3cyclictest0-21swapper/307:50:103
358699221,10cyclictest821-21lldpd07:26:373
358699221,10cyclictest800-21snmpd09:48:593
358699221,10cyclictest27799-21timerandwakeup08:00:243
358699221,10cyclictest20270-21ethtool11:25:163
358699221,10cyclictest14488-21/usr/sbin/munin10:00:163
358699220,9cyclictest0-21swapper/307:40:163
358699220,10cyclictest0-21swapper/311:22:083
357999225,3cyclictest0-21swapper/208:30:292
3574992218,2cyclictest24026-21awk10:20:161
3574992215,4cyclictest13312-21munin-run12:25:021
3574992214,4cyclictest8868-21diskstats11:00:141
357499221,3cyclictest7379-21cat10:55:201
357499221,19cyclictest0-21swapper/111:29:271
357499221,18cyclictest14792-21runrttasks11:13:541
357499221,15cyclictest0-21swapper/108:25:191
357499221,15cyclictest0-21swapper/108:25:191
357499221,11cyclictest9234-21grep12:15:111
3574992210,6cyclictest3023-21kworker/u8:112:11:461
3574992210,3cyclictest0-21swapper/108:53:071
358699218,2cyclictest5109-21cpu08:25:133
358699218,2cyclictest5109-21cpu08:25:133
358699214,4cyclictest7613-21cat10:55:253
358699212,16cyclictest6464-21users07:15:033
358699212,16cyclictest6464-21users07:15:033
358699211,10cyclictest0-21swapper/311:50:073
357499219,4cyclictest9419-21perl07:20:161
357499212,4cyclictest22201-21sshd07:50:121
357499211,3cyclictest550-21sed09:25:261
357499211,18cyclictest0-21swapper/107:44:331
357499211,17cyclictest0-21swapper/108:35:191
357499211,11cyclictest24117-21cstates11:35:121
357499211,11cyclictest17335-21cpu11:20:111
356599213,14cyclictest0-21swapper/010:25:160
356599212,11cyclictest12292-21/usr/sbin/munin09:55:180
276192211,16sleep127701-21sensors_temp08:00:241
358699206,3cyclictest811-21tune2fs11:55:143
358699206,3cyclictest811-21tune2fs11:55:143
358699206,2cyclictest19706-21irqstats10:10:173
358699205,4cyclictest10420-21sshd11:04:123
358699204,4cyclictest0-21swapper/309:15:173
3586992013,4cyclictest15790-21tail11:15:173
358699201,10cyclictest0-21swapper/308:55:053
357499201,7cyclictest25240-21head07:55:201
357499201,3cyclictest0-21swapper/109:18:131
357499201,17cyclictest9293-21sed08:35:011
357499201,17cyclictest0-21swapper/112:28:191
357499201,17cyclictest0-21swapper/111:45:121
357499201,17cyclictest0-21swapper/109:02:071
357499201,10cyclictest5665-21memory12:05:191
357499201,10cyclictest1692-21perf10:45:001
3574992010,4cyclictest3023-21kworker/u8:110:45:261
3574992010,4cyclictest3023-21kworker/u8:110:45:261
3574992010,3cyclictest12933-21smart_sda08:40:221
357499200,18cyclictest0-21swapper/108:08:451
357499200,12cyclictest87150irq/126-enp1s0-11:30:181
356599207,3cyclictest24246-21chrt07:55:110
356599206,3cyclictest87350irq/128-enp1s0-07:35:160
356599202,10cyclictest587-21/usr/sbin/munin08:15:180
3565992014,4cyclictest12311-21sed08:40:140
356599201,16cyclictest14823-21df10:00:130
358699191,5cyclictest11273-21cat12:20:013
358699191,11cyclictest0-21swapper/308:05:103
358699191,10cyclictest2674-21if_enp2s010:45:163
358699191,10cyclictest2674-21if_enp2s010:45:163
358699191,10cyclictest0-21swapper/309:09:013
357499199,2cyclictest2466-21latency_hist12:00:021
357499199,2cyclictest2466-21latency_hist12:00:021
357499196,3cyclictest87350irq/128-enp1s0-07:35:141
357499196,2cyclictest0-21swapper/110:52:051
357499195,9cyclictest0-21swapper/108:20:121
357499192,9cyclictest0-21swapper/111:50:181
3574991915,3cyclictest0-21swapper/108:46:461
3574991913,3cyclictest16400-21grep12:30:151
357499191,17cyclictest0-21swapper/110:35:181
357499191,17cyclictest0-21swapper/109:21:391
357499191,17cyclictest0-21swapper/107:33:061
357499191,16cyclictest0-21swapper/110:30:441
357499191,10cyclictest19886-21cpu07:45:121
357499191,10cyclictest1360-21egrep09:30:111
357499190,17cyclictest0-21swapper/111:05:231
356599198,3cyclictest10544-21ntp_states08:35:210
3565991913,4cyclictest18781-21grep11:20:260
356599191,15cyclictest27068-21/usr/sbin/munin09:15:190
356599191,10cyclictest7894-21/usr/sbin/munin09:45:180
295222191,3sleep00-21swapper/010:33:280
36812180,2sleep00-21swapper/010:47:280
36812180,2sleep00-21swapper/010:47:280
358699185,3cyclictest12050irq/123-ahci[0012:05:013
358699185,3cyclictest12050irq/123-ahci[0009:45:013
358699181,9cyclictest0-21swapper/310:40:223
358699181,5cyclictest19514-21users12:35:253
358699181,5cyclictest17699-21latency_hist12:35:013
358699181,5cyclictest12628-21df09:55:123
358699181,5cyclictest11332-21/usr/sbin/munin12:20:193
358699181,11cyclictest0-21swapper/311:30:163
358699181,10cyclictest0-21swapper/309:25:253
357499181,7cyclictest0-21swapper/110:15:141
3574991814,3cyclictest0-21swapper/110:14:541
3574991814,3cyclictest0-21swapper/110:01:521
357499181,16cyclictest0-21swapper/111:42:401
357499181,16cyclictest0-21swapper/109:42:301
357499180,9cyclictest0-21swapper/112:35:111
357499180,16cyclictest0-21swapper/107:25:141
356599188,3cyclictest13252-21/usr/sbin/munin07:30:160
356599187,3cyclictest30840-21/usr/sbin/munin08:10:200
356599184,4cyclictest8585-21cpuspeed_turbos11:00:130
356599184,3cyclictest87350irq/128-enp1s0-10:50:140
356599183,7cyclictest20104-21expr11:25:150
356599182,13cyclictest9139-21/usr/sbin/munin12:15:140
356599181,9cyclictest5392-21tail12:05:150
356599181,4cyclictest15733-21/usr/sbin/munin12:30:210
356599181,3cyclictest12312-21vmstat11:05:260
3565991813,3cyclictest17767-21ntp_states10:05:200
356599181,2cyclictest0-21swapper/009:05:220
3565991812,4cyclictest10715-21head09:50:150
356599181,10cyclictest800-21snmpd08:06:500
356599181,10cyclictest23285-21/usr/sbin/munin10:20:120
30362181,2sleep30-21swapper/309:32:113
358699174,3cyclictest87450irq/129-enp1s0-08:15:153
358699173,3cyclictest29374-21ntpq11:45:203
358699171,6cyclictest8871-21acpi07:20:113
358699171,4cyclictest22116-21bash09:01:013
358699171,15cyclictest0-21swapper/308:31:583
358699171,14cyclictest39-21rcuc/312:10:103
358699171,14cyclictest0-21swapper/311:10:123
358699171,13cyclictest12812-21pmu-power08:40:223
358699170,8cyclictest0-21swapper/311:35:123
358699170,15cyclictest0-21swapper/308:35:053
356599176,3cyclictest2535-21/usr/sbin/munin12:00:220
356599174,6cyclictest87350irq/128-enp1s0-07:50:150
356599171,9cyclictest800-21snmpd08:31:540
3565991712,3cyclictest4404-21gltestperf07:10:160
3565991712,3cyclictest4404-21gltestperf07:10:160
3565991712,3cyclictest31463-21/usr/sbin/munin09:25:240
3565991711,4cyclictest851-21grep11:55:150
3565991711,4cyclictest851-21grep11:55:150
3565991711,3cyclictest21057-21cat10:15:010
258932171,3sleep30-21swapper/307:56:033
233632171,2sleep00-21swapper/011:31:070
106062170,2sleep00-21swapper/007:22:180
358699161,6cyclictest0-21swapper/312:25:163
358699161,14cyclictest0-21swapper/312:05:313
3586991610,4cyclictest0-21swapper/307:30:213
3586991610,3cyclictest30840-21/usr/sbin/munin08:10:163
358699160,6cyclictest0-21swapper/308:45:123
356599162,12cyclictest6666-21/usr/sbin/munin07:15:120
356599162,12cyclictest6666-21/usr/sbin/munin07:15:120
356599161,8cyclictest23905-21taskset11:35:120
356599161,5cyclictest0-21swapper/009:00:220
3565991612,3cyclictest0-21swapper/007:47:410
3565991611,3cyclictest831-21runrttasks12:21:320
3565991611,3cyclictest831-21runrttasks07:40:520
3565991611,3cyclictest7967-21netstat12:10:170
3565991611,3cyclictest17929-21/usr/sbin/munin12:35:130
3565991611,3cyclictest11057-21/usr/sbin/munin07:25:170
356599161,12cyclictest5697-21/usr/sbin/munin09:40:210
356599161,12cyclictest32081-21/usr/sbin/munin10:40:130
356599161,12cyclictest30494-21/usr/sbin/munin11:50:130
356599161,12cyclictest29947-21sshd10:35:110
356599160,7cyclictest0-21swapper/009:35:230
356599160,7cyclictest0-21swapper/009:35:230
356599160,3cyclictest0-21swapper/009:30:120
356599160,3cyclictest0-21swapper/008:55:260
310742161,2sleep00-21swapper/009:23:100
281942161,2sleep00-21swapper/008:02:140
145452161,2sleep00-21swapper/011:10:450
356599151,4cyclictest5434-21diskstats08:25:140
356599151,4cyclictest5434-21diskstats08:25:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional