You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 03:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Dec 11, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
189442239163,24sleep20-21swapper/219:09:572
169882239162,24sleep00-21swapper/019:05:070
186402204175,19sleep10-21swapper/119:06:021
169912203166,25sleep30-21swapper/319:05:113
465021610,4sleep11905299cyclictest21:00:231
1231321410,5sleep01904699cyclictest22:30:250
1536421290,4sleep01904699cyclictest23:51:510
1536421290,4sleep01904699cyclictest23:51:510
1996121180,2sleep20-21swapper/222:50:152
19056996562,2cyclictest0-21swapper/219:25:262
191982590,3sleep119203-21pmu-power00:00:231
1905299441,35cyclictest25267-21gltestperf00:15:171
19056994340,2cyclictest0-21swapper/219:55:102
1905299431,27cyclictest16829-21gltestperf20:15:171
19052994310,24cyclictest28571-21if_err_enp2s021:55:181
1905299411,28cyclictest25062-40dnf21:46:251
1905299411,27cyclictest13745-21awk20:10:021
1905299410,29cyclictest20719-21runrttasks21:37:031
1905299402,26cyclictest14589-21/usr/sbin/munin21:25:191
1905299401,32cyclictest17454-21kworker/u8:023:59:071
1905299401,30cyclictest8639-21acpi22:25:131
1905299401,24cyclictest29093-21cpu00:25:131
1905299401,23cyclictest836-21runrttasks21:52:211
1905699391,29cyclictest0-21swapper/220:54:282
1905299391,31cyclictest15267-21kworker/u8:122:18:131
1905299391,25cyclictest23154-21latency_hist21:45:021
1905299391,25cyclictest23154-21latency_hist21:45:021
1905299391,18cyclictest0-21swapper/122:22:001
19056993816,13cyclictest19109-40dnf22:46:332
19056993816,13cyclictest19109-40dnf22:46:332
1905299382,32cyclictest9026-21munin-run23:40:011
1905299381,30cyclictest17454-21kworker/u8:000:31:221
1905299381,28cyclictest22928-21/usr/sbin/munin22:55:241
1905299381,19cyclictest15006-21awk22:40:011
1905299380,26cyclictest2129-21runrttasks23:21:201
19056993724,7cyclictest3953-40dnf19:45:252
1905299371,33cyclictest29121-21sh20:45:011
1905299371,28cyclictest848-21sssd_nss23:30:151
1905299371,28cyclictest27142-21memory23:05:211
1905299371,25cyclictest16926-21sshd22:41:451
1905299371,22cyclictest26944-21awk00:20:141
1905299370,29cyclictest0-21swapper/119:35:121
1905299370,21cyclictest0-21swapper/121:30:161
1905299364,18cyclictest21205-21grep22:50:281
1905299363,19cyclictest0-21swapper/123:41:051
1905299362,30cyclictest19109-40dnf22:46:331
1905299362,30cyclictest19109-40dnf22:46:331
1905299361,26cyclictest0-21swapper/119:32:481
1905299361,26cyclictest0-21swapper/119:32:481
1905299361,25cyclictest9260-21smartctl21:10:241
1905299361,25cyclictest22762-21/usr/sbin/munin20:30:241
1905299361,22cyclictest2722-21sh19:45:011
1905299361,22cyclictest15918-21cat20:14:571
19052993610,23cyclictest31609-21cpu20:50:141
52472350,2sleep20-21swapper/219:50:112
1905699353,23cyclictest29263-21wc00:25:132
1905299353,22cyclictest86550irq/126-enp1s0-20:05:021
1905299352,27cyclictest86750irq/128-enp1s0-21:22:251
1905299351,31cyclictest19132-21/usr/sbin/munin20:20:171
1905299351,31cyclictest15627-21perf23:55:011
1905299351,31cyclictest15627-21perf23:55:011
1905299351,30cyclictest24499-21cat00:15:021
1905299351,23cyclictest7368-21cpu19:55:131
1905299351,22cyclictest0-21swapper/100:05:221
1905299351,21cyclictest22129-21/usr/sbin/munin19:15:221
1905299350,22cyclictest0-21swapper/123:10:161
1905699341,26cyclictest0-21swapper/220:02:562
19056993410,11cyclictest12677-21memory20:05:192
1905299344,25cyclictest4140-21cut19:45:261
1905299343,27cyclictest25403-21ntp_states19:20:231
1905299341,25cyclictest0-21swapper/122:07:481
1905299341,24cyclictest27450-21kworker/u8:022:04:531
1905299341,20cyclictest7764-21diskmemload23:49:451
282972330,2sleep30-21swapper/321:55:163
1905699339,15cyclictest20309-21cpu00:05:162
1905699336,16cyclictest0-21swapper/223:30:012
19052993314,16cyclictest20567-21acpi20:25:151
1905299331,29cyclictest3082-21munin-run00:40:011
23392320,2sleep30-21swapper/300:35:243
1905699328,8cyclictest13937-21unixbench_singl21:20:282
1905299323,20cyclictest3893-21fgrep23:25:251
1905299322,27cyclictest20314-21ntpq19:10:211
1905299321,28cyclictest3355-21latency_hist21:00:021
1905299321,27cyclictest5792-21acpi21:05:111
1905299317,14cyclictest0-21swapper/122:31:471
1905299311,27cyclictest30626-21/usr/sbin/munin23:15:251
1905299311,26cyclictest24374-21entropy23:00:141
1904699318,11cyclictest21612-21perf22:55:000
1905699308,13cyclictest21192-21/usr/sbin/munin21:40:222
1905699308,13cyclictest21192-21/usr/sbin/munin21:40:222
1905699307,13cyclictest23384-21/usr/sbin/munin21:45:162
1905699306,16cyclictest31341-21ls00:30:122
1905699306,13cyclictest28335-21cat23:10:022
1905699301,13cyclictest0-21swapper/222:32:022
19056993011,11cyclictest0-21swapper/222:40:112
19056993010,7cyclictest24849-21cat00:15:132
19056993010,13cyclictest18389-21sh00:00:142
1905299307,20cyclictest25950-21needreboot20:35:201
1905299307,20cyclictest25950-21needreboot20:35:201
1905299301,26cyclictest27788-21smart_sda19:25:261
1905299301,26cyclictest10194-21/usr/sbin/munin21:15:191
1905299301,20cyclictest0-21swapper/119:50:051
19052993011,16cyclictest2032-21/usr/sbin/munin22:10:261
1905699299,11cyclictest27691-21irqstats00:20:192
1905699299,11cyclictest15848-21/usr/sbin/munin23:55:152
1905699298,8cyclictest0-21swapper/219:39:282
1905699298,12cyclictest24884-21cat23:00:182
1905699297,14cyclictest22129-21/usr/sbin/munin19:15:142
1905699295,15cyclictest0-21swapper/221:15:262
1905699294,15cyclictest30626-21/usr/sbin/munin23:15:272
1905699292,19cyclictest0-21swapper/220:45:152
1905699291,21cyclictest0-21swapper/220:41:202
19056992912,10cyclictest12232-21if_err_enp1s023:45:172
19056992910,7cyclictest9112-21cat23:40:012
1905699290,11cyclictest2614-21/usr/sbin/munin22:10:182
1905299291,25cyclictest29354-21/usr/sbin/munin20:45:181
1904699291,7cyclictest0-21swapper/023:46:400
1905699288,8cyclictest0-21swapper/220:19:282
1905699288,13cyclictest0-21swapper/200:35:182
1905699286,12cyclictest7601-21processes22:20:232
1905699286,12cyclictest28962-21processes21:55:242
1905699279,10cyclictest14090-21ntp_states22:35:222
1905699279,10cyclictest0-21swapper/222:16:322
1905699278,16cyclictest18762-21perf21:35:012
1905699277,12cyclictest13656-21/usr/sbin/munin23:50:292
1905699277,12cyclictest13656-21/usr/sbin/munin23:50:292
1905699277,10cyclictest86550irq/126-enp1s0-21:10:132
1905699276,9cyclictest6274-21kworker/2:320:10:222
1905699276,12cyclictest0-21swapper/222:05:172
1905699274,13cyclictest0-21swapper/221:05:202
1905699271,16cyclictest0-21swapper/220:36:172
1905699271,16cyclictest0-21swapper/220:36:172
19056992710,8cyclictest0-21swapper/220:26:242
1904699274,11cyclictest15931-21idleruntime-cro20:15:020
291932260,4sleep230-21rcuc/223:10:182
1905699269,14cyclictest29716-21cut19:30:212
1905699269,14cyclictest29716-21cut19:30:212
1905699267,12cyclictest0-21swapper/221:35:172
1905699265,6cyclictest30-21rcuc/219:13:542
1905699263,7cyclictest0-21swapper/223:30:252
1905699261,8cyclictest0-21swapper/219:24:282
19056992617,6cyclictest23056-21find22:55:252
1905699261,12cyclictest0-21swapper/221:54:502
1905699256,5cyclictest0-21swapper/222:25:212
1905699255,3cyclictest0-21swapper/220:59:002
1905699251,7cyclictest0-21swapper/221:03:082
1905699251,11cyclictest0-21swapper/220:24:202
1904699251,11cyclictest0-21swapper/019:45:540
1905699247,10cyclictest86850irq/129-enp1s0-00:10:262
19056992416,5cyclictest1268-21fschecks_time19:40:172
1906399230,7cyclictest0-21swapper/323:48:333
1905699231,7cyclictest0-21swapper/221:25:282
19056992316,4cyclictest31205-21/usr/sbin/munin22:00:252
19056992310,10cyclictest24377-21runrttasks20:30:382
1904699232,4cyclictest0-21swapper/021:21:250
1906399225,4cyclictest0-21swapper/321:15:153
1906399223,4cyclictest0-21swapper/322:32:093
1906399221,4cyclictest20056-21ntp_states21:35:223
1906399220,7cyclictest0-21swapper/323:41:293
19056992215,4cyclictest1205-21/usr/sbin/munin23:20:192
1904699223,11cyclictest27157-21/usr/sbin/munin20:40:210
1904699221,6cyclictest0-21swapper/023:56:290
1904699221,6cyclictest0-21swapper/021:25:240
1906399215,7cyclictest0-21swapper/322:47:293
1906399215,7cyclictest0-21swapper/322:47:293
1906399214,15cyclictest22253-21sadc00:10:003
1906399211,11cyclictest19559-21grep00:00:253
1906399211,11cyclictest14568-21memory23:50:213
1906399211,11cyclictest14568-21memory23:50:213
1906399211,10cyclictest29958-21tail20:45:153
19063992110,3cyclictest4238-21switchtime19:45:263
19063992110,3cyclictest3622-21users22:10:283
1906399210,12cyclictest0-21swapper/322:15:143
19056992112,6cyclictest9421-21/usr/sbin/munin23:40:142
1904699212,5cyclictest10596-21munin-run22:30:000
1904699211,7cyclictest0-21swapper/021:11:500
1904699211,5cyclictest7-21ksoftirqd/022:55:180
1904699211,4cyclictest0-21swapper/023:12:050
1904699211,12cyclictest7052-21/usr/sbin/munin23:35:220
22842201,3sleep00-21swapper/023:23:170
1906399209,3cyclictest0-21swapper/300:25:163
1906399206,3cyclictest0-21swapper/300:34:573
1906399205,5cyclictest0-21swapper/300:22:013
1906399205,4cyclictest0-21swapper/300:12:363
1906399204,2cyclictest86550irq/126-enp1s0-23:20:153
1906399202,9cyclictest39-21rcuc/320:30:153
1906399202,10cyclictest6425-21/usr/sbin/munin22:20:183
1906399201,9cyclictest0-21swapper/320:05:243
1906399201,4cyclictest0-21swapper/322:25:493
1906399201,17cyclictest0-21swapper/323:05:243
1906399201,12cyclictest0-21swapper/321:25:253
1906399201,11cyclictest86550irq/126-enp1s0-22:55:133
1906399201,11cyclictest3144-21sshd20:57:233
1906399201,11cyclictest25958-21fschecks_count21:50:173
1906399201,11cyclictest0-21swapper/323:34:553
1906399201,11cyclictest0-21swapper/300:15:213
1906399201,10cyclictest39-21rcuc/322:05:263
1906399201,10cyclictest0-21swapper/320:21:303
1904699209,3cyclictest20566-21/usr/sbin/munin20:25:270
1904699202,10cyclictest22483-21/usr/sbin/munin00:10:230
1904699201,6cyclictest0-21swapper/021:36:230
1904699201,5cyclictest9418-21tr23:40:140
1904699201,11cyclictest22129-21/usr/sbin/munin19:15:200
1904699201,11cyclictest1368-21/usr/sbin/munin20:55:240
1904699201,10cyclictest0-21swapper/000:35:290
261922190,4sleep01904699cyclictest23:05:010
1906399198,3cyclictest9182-21sshd19:58:213
1906399194,9cyclictest0-21swapper/323:02:043
1906399191,9cyclictest31741-21kernelversion19:35:163
1906399191,9cyclictest0-21swapper/320:26:203
1906399191,7cyclictest26927-21munin-run20:40:003
1906399191,7cyclictest26927-21munin-run20:40:003
1906399191,6cyclictest0-21swapper/321:24:043
1906399191,6cyclictest0-21swapper/319:30:113
1906399191,6cyclictest0-21swapper/319:30:113
1906399191,5cyclictest0-21swapper/319:26:053
1906399191,3cyclictest0-21swapper/323:13:533
1906399191,11cyclictest0-21swapper/322:52:323
1906399191,10cyclictest7764-21diskmemload21:40:313
1906399191,10cyclictest7764-21diskmemload21:40:313
1906399191,10cyclictest2657-21/usr/sbin/munin23:25:143
1906399191,10cyclictest0-21swapper/321:45:123
1906399191,10cyclictest0-21swapper/320:41:233
1906399191,10cyclictest0-21swapper/319:50:233
1906399190,7cyclictest0-21swapper/319:21:073
1906399190,6cyclictest0-21swapper/322:43:553
1904699199,3cyclictest86750irq/128-enp1s0-00:04:130
1904699191,7cyclictest0-21swapper/023:31:390
1904699191,10cyclictest10194-21/usr/sbin/munin21:15:220
1904699190,7cyclictest29785-21ntp_offset19:30:220
1904699190,7cyclictest29785-21ntp_offset19:30:220
170792191,3sleep00-21swapper/022:43:420
230752181,3sleep00-21swapper/021:44:050
230752181,3sleep00-21swapper/021:44:050
1906399188,3cyclictest39-21rcuc/323:35:223
1906399187,3cyclictest14634-21snmpd20:50:153
1906399185,3cyclictest5807-21cpu21:05:113
1906399184,5cyclictest4411-21grep21:00:193
1906399181,7cyclictest9736-21cstates20:00:143
19063991813,3cyclictest22800-21hddtemp_smartct19:15:153
19063991812,3cyclictest7996-21/usr/sbin/munin21:10:263
1906399181,11cyclictest0-21swapper/322:01:353
1906399181,11cyclictest0-21swapper/319:10:213
1906399181,10cyclictest18081-21switchtime21:30:273
1906399180,6cyclictest10341-21kworker/u8:022:36:153
1906399180,3cyclictest86550irq/126-enp1s0-23:15:123
1904699185,4cyclictest27500-21sshd21:54:230
1904699182,6cyclictest20614-21df_inode00:05:170
1904699181,9cyclictest18359-21/usr/sbin/munin20:20:190
1904699181,9cyclictest14634-21snmpd20:09:240
1904699181,9cyclictest14634-21snmpd19:14:590
1904699181,8cyclictest2379-21diskstats22:10:150
1904699181,8cyclictest0-21swapper/021:59:070
1904699181,6cyclictest0-21swapper/022:18:070
1904699181,6cyclictest0-21swapper/022:00:560
1904699181,5cyclictest0-21swapper/023:19:520
19046991813,3cyclictest18260-21grep21:30:270
1904699181,15cyclictest856-21perf00:35:020
1904699181,10cyclictest671-21/usr/sbin/munin19:40:140
1904699180,16cyclictest31329-21munin-run20:50:010
19063991713,2cyclictest2021-21smart_sda19:40:253
19063991712,3cyclictest16656-21interrupts23:55:163
1904699177,8cyclictest9328-21awk20:00:010
1904699172,12cyclictest25060-21sshd21:46:240
1904699171,7cyclictest0-21swapper/022:45:030
1904699171,7cyclictest0-21swapper/022:45:030
1904699171,7cyclictest0-21swapper/019:25:260
1904699171,6cyclictest0-21swapper/023:05:050
1904699171,5cyclictest0-21swapper/022:38:230
1904699171,4cyclictest2657-21/usr/sbin/munin23:25:190
1904699171,3cyclictest32364-21irqstats20:50:180
1904699171,3cyclictest23830-21ntp_states20:30:210
1904699171,3cyclictest0-21swapper/020:00:160
19046991712,3cyclictest5834-21hwlatdetect19:50:150
1904699171,10cyclictest0-21swapper/000:15:210
19046991710,4cyclictest7041-21/usr/sbin/munin21:05:220
1904699170,6cyclictest0-21swapper/000:20:580
1904699170,14cyclictest29222-21tr00:25:140
179732171,2sleep00-21swapper/020:18:080
1906399161,5cyclictest18289-21cat20:20:003
19063991611,3cyclictest14879-21memory20:10:183
1904699162,11cyclictest4183-21tail21:00:160
1904699161,8cyclictest0-21swapper/022:09:370
1904699161,7cyclictest0-21swapper/022:20:230
19046991612,3cyclictest0-21swapper/020:39:530
19046991612,3cyclictest0-21swapper/020:39:530
1904699161,14cyclictest0-21swapper/019:38:310
19046991510,3cyclictest25410-21open_files19:20:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional