You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 06:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 19, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
203172256163,41sleep20-21swapper/219:08:002
204322254165,22sleep10-21swapper/119:09:281
202432240164,25sleep00-21swapper/019:07:020
203862234164,57sleep30-21swapper/319:08:543
289752710,1sleep20-21swapper/221:54:122
141172610,2sleep20-21swapper/220:05:212
9732530,4sleep12056599cyclictest19:35:211
2056599452,31cyclictest23658-21users20:25:281
87452430,1sleep20-21swapper/219:55:022
82902420,1sleep20-21swapper/222:20:172
2056599411,27cyclictest9065-21taskset21:07:331
2056599411,27cyclictest9065-21taskset21:07:331
2056599401,4cyclictest20256-21munin-run21:35:011
2056599376,3cyclictest0-21swapper/123:15:201
2056599374,7cyclictest3065-40dnf22:07:121
20565993714,20cyclictest30282-21fschecks_count23:10:121
2056599371,15cyclictest0-21swapper/122:20:471
2056599371,11cyclictest15580-21ntp_states22:35:231
2055899377,13cyclictest12101-21idleruntime-cro22:30:010
2056599361,13cyclictest4863-21ls21:00:001
2055899368,13cyclictest6449-21munin-run19:50:000
2056599359,9cyclictest86750irq/128-enp1s0-20:18:431
2056599351,6cyclictest0-21swapper/100:19:171
2056599351,15cyclictest0-21swapper/119:10:421
2056599341,9cyclictest2156-21expr23:20:131
2056599341,9cyclictest2156-21expr23:20:131
2056599341,6cyclictest0-21swapper/121:52:581
20565993414,2cyclictest0-21swapper/122:04:271
20565993412,2cyclictest19525-21/usr/sbin/munin00:00:271
2056599341,15cyclictest0-21swapper/121:37:121
20565993411,4cyclictest31430-21cat00:25:201
20565993411,4cyclictest31430-21cat00:25:201
2056599341,11cyclictest28435-21munin-run20:40:011
2056599341,11cyclictest13744-21irqstats23:45:171
2056599341,10cyclictest28670-21acpi20:40:131
2056599340,9cyclictest0-21swapper/100:25:011
2055899349,12cyclictest2667-21munin-run20:55:000
20565993313,3cyclictest0-21swapper/119:59:151
20565993313,2cyclictest0-21swapper/119:16:091
2056599331,15cyclictest0-21swapper/120:05:441
20565993311,3cyclictest30247-21/usr/sbin/munin19:30:231
2056599324,4cyclictest86750irq/128-enp1s0-21:10:221
2056599324,4cyclictest86750irq/128-enp1s0-21:10:221
2056599324,18cyclictest780-21rngd20:47:191
2056599323,14cyclictest780-21rngd19:28:231
2056599322,10cyclictest86750irq/128-enp1s0-22:57:231
2056599321,7cyclictest21745-21/usr/sbin/munin00:05:271
20565993214,3cyclictest12116-21grep21:15:141
20565993213,3cyclictest4814-21df19:45:131
20565993213,2cyclictest0-21swapper/121:49:071
20565993212,3cyclictest18366-21seq22:41:101
2056599321,14cyclictest0-21swapper/121:24:431
2056599321,14cyclictest0-21swapper/119:52:281
20565993211,3cyclictest836-21runrttasks22:45:171
2056599321,10cyclictest20080-21cat20:20:121
2056599320,15cyclictest0-21swapper/119:43:011
2057599317,10cyclictest0-21swapper/319:38:023
2057599310,24cyclictest0-21swapper/300:36:473
2056599319,3cyclictest10833-21sort23:40:131
2056599317,4cyclictest0-21swapper/123:33:371
2056599311,9cyclictest11185-21sh22:25:191
2056599311,9cyclictest0-21swapper/121:56:571
2056599311,6cyclictest0-21swapper/120:50:051
2056599311,13cyclictest3646-21ntp_states00:35:221
20565993111,3cyclictest24549-21cat20:30:141
2056599311,11cyclictest2404-21cat00:35:011
2056599311,10cyclictest0-21swapper/121:25:121
2055899318,10cyclictest12699-21idleruntime-cro23:45:010
228522300,3sleep30-21swapper/321:40:113
2057599306,12cyclictest780-21rngd19:40:233
2056599307,4cyclictest27473-21crond23:05:001
2056599301,11cyclictest24017-21chrt00:10:141
20565993010,3cyclictest22678-21seq21:40:041
2055899301,9cyclictest5198-21kworker/0:122:11:430
2055899301,14cyclictest25278-21munin-run23:00:010
2056599291,7cyclictest28125-21tune2fs23:05:161
2056599291,4cyclictest4207-21hddtemp_smartct22:10:201
2056599291,2cyclictest0-21swapper/123:39:081
20565992911,4cyclictest27356-21sed19:20:301
2056599291,13cyclictest6304-21perl22:15:151
2056599291,13cyclictest6304-21perl22:15:151
2056599291,11cyclictest14281-21latency_hist22:35:011
2056599291,10cyclictest0-21swapper/121:02:531
2056599290,14cyclictest86750irq/128-enp1s0-20:14:351
2057599284,15cyclictest8839-21memory22:20:203
2057599282,5cyclictest20809-21kworker/u8:123:09:243
2057599281,19cyclictest0-21swapper/321:59:573
2056599281,3cyclictest22281-21ps22:50:221
2056599281,12cyclictest15190-21cpu23:50:151
20565992810,3cyclictest4755-21head23:25:161
20565992810,3cyclictest12333-21smartctl20:00:251
2057599274,13cyclictest86550irq/126-enp1s0-00:30:133
2057599272,15cyclictest86550irq/126-enp1s0-22:15:193
2057599272,15cyclictest86550irq/126-enp1s0-22:15:193
20575992718,5cyclictest3065-40dnf22:07:143
2057599271,15cyclictest0-21swapper/323:53:163
2057599270,19cyclictest0-21swapper/323:27:273
2057599270,19cyclictest0-21swapper/300:20:183
2057599270,16cyclictest27088-21/usr/sbin/munin21:50:253
2057099271,13cyclictest0-21swapper/219:36:342
2056599279,3cyclictest19452-21cut00:00:021
324022260,1sleep20-21swapper/219:35:032
2057599267,9cyclictest0-21swapper/320:13:363
2057599265,12cyclictest27536-21ntp_states20:35:253
2057599261,18cyclictest0-21swapper/321:47:173
2057599261,17cyclictest0-21swapper/320:44:473
2057599261,16cyclictest0-21swapper/300:16:483
2057599261,14cyclictest780-21rngd20:24:333
2057599261,14cyclictest20635-21tr21:35:153
2057599261,13cyclictest780-21rngd22:41:533
2057599261,11cyclictest30899-21strings23:10:173
20575992610,4cyclictest20440-21memory00:00:203
2057599260,12cyclictest0-21swapper/320:25:513
2057599257,9cyclictest32630-21tune2fs23:15:173
2057599255,9cyclictest780-21rngd21:31:133
2057599253,14cyclictest780-21rngd20:56:553
2057599251,8cyclictest0-21swapper/321:00:193
2057599251,13cyclictest780-21rngd20:16:083
2057599251,11cyclictest0-21swapper/323:33:113
20575992510,8cyclictest0-21swapper/322:37:353
2057599250,11cyclictest0-21swapper/319:25:063
2057099259,6cyclictest6921-21processes22:15:242
2057099259,6cyclictest6921-21processes22:15:242
2055899258,5cyclictest0-21swapper/023:53:030
2055899252,4cyclictest4859-21munin-run21:00:000
2055899251,11cyclictest6126-21idleruntime-cro23:30:010
2055899251,11cyclictest2132-21chrt00:32:210
2057599248,6cyclictest22416-21hddtemp_smartct00:05:213
2057599244,13cyclictest0-21swapper/319:18:273
2057599241,9cyclictest0-21swapper/322:51:133
2057599241,10cyclictest19291-21munin-run00:00:013
2057599241,10cyclictest0-21swapper/319:49:223
2057599241,10cyclictest0-21swapper/319:14:073
2057599240,10cyclictest0-21swapper/300:28:113
2057599240,10cyclictest0-21swapper/300:28:113
2057099243,4cyclictest14634-21snmpd22:50:162
2055899241,11cyclictest7299-21/usr/sbin/munin21:05:150
2055899241,11cyclictest7299-21/usr/sbin/munin21:05:150
2057599236,2cyclictest0-21swapper/322:10:273
2057599233,11cyclictest0-21swapper/320:48:333
2057599231,9cyclictest0-21swapper/320:00:013
2057599231,7cyclictest0-21swapper/319:58:153
2057599231,19cyclictest0-21swapper/320:08:333
2057599231,13cyclictest30247-21/usr/sbin/munin19:30:233
2057599231,12cyclictest10740-21/usr/sbin/munin23:40:243
2057599230,6cyclictest86850irq/129-enp1s0-22:59:353
2057599230,3cyclictest0-21swapper/321:07:473
2057599230,3cyclictest0-21swapper/321:07:473
2057599230,11cyclictest1950-21/usr/sbin/munin23:20:163
2057599230,11cyclictest1950-21/usr/sbin/munin23:20:163
2057099231,13cyclictest0-21swapper/222:30:222
2055899231,11cyclictest24269-21/usr/sbin/munin20:30:190
2057599223,11cyclictest11076-21unixbench_singl21:10:273
2057599223,11cyclictest11076-21unixbench_singl21:10:273
20575992212,6cyclictest27119-21smart_sda19:20:293
2057599221,13cyclictest0-21swapper/322:00:473
2057599221,11cyclictest0-21swapper/320:50:333
2057599221,10cyclictest0-21swapper/320:32:253
2057599220,17cyclictest86550irq/126-enp1s0-22:47:313
2057099225,8cyclictest821-21sshd22:02:112
2055899227,12cyclictest369-21awk20:50:010
2055899223,16cyclictest32217-21munin-run19:35:010
2055899221,7cyclictest0-21swapper/021:37:510
2055899221,10cyclictest0-21swapper/023:15:280
20558992210,3cyclictest29159-21irqstats00:20:190
2057599215,6cyclictest26181-21hddtemp_smartct23:00:163
2057599215,3cyclictest0-21swapper/322:31:193
2057599214,2cyclictest86750irq/128-enp1s0-22:25:183
2057599213,8cyclictest0-21swapper/321:20:233
2057599211,9cyclictest780-21rngd19:51:233
2057599211,11cyclictest0-21swapper/300:10:093
2057099219,2cyclictest27199-21if_enp1s020:35:212
2057099215,7cyclictest2327-21netstat22:05:202
2057099211,13cyclictest0-21swapper/200:25:302
2057099211,13cyclictest0-21swapper/200:25:302
2057099211,12cyclictest0-21swapper/223:37:082
2057099211,12cyclictest0-21swapper/200:32:172
2055899214,15cyclictest9887-21idleruntime-cro22:25:010
2055899211,11cyclictest13901-21/usr/sbin/munin21:20:240
2055899211,10cyclictest1946-21sshd23:20:100
2055899211,10cyclictest1946-21sshd23:20:100
2057599201,13cyclictest0-21swapper/321:26:093
2057599201,10cyclictest9268-21diskmemload21:15:193
2057599200,6cyclictest8536-21/usr/sbin/munin23:35:193
2057099204,9cyclictest3641-21kworker/u8:023:31:022
2057099204,8cyclictest17294-21fw_packets22:40:182
2057099202,9cyclictest6006-21memory21:00:202
2057099201,7cyclictest21843-21latency_hist20:25:022
2057099201,4cyclictest0-21swapper/222:56:322
2055899208,4cyclictest0-21swapper/000:25:190
2055899208,4cyclictest0-21swapper/000:25:190
2055899208,3cyclictest24581-21sshd21:44:110
2055899205,4cyclictest9268-21diskmemload23:35:130
2055899201,9cyclictest14634-21snmpd21:55:050
2055899201,7cyclictest0-21swapper/019:17:160
2055899201,12cyclictest0-21swapper/019:50:240
2055899201,10cyclictest8871-21/usr/sbin/munin19:55:190
2055899201,10cyclictest12932-21/usr/sbin/munin23:45:250
273712191,3sleep20-21swapper/223:03:462
2057599191,7cyclictest0-21swapper/323:45:473
2057099199,4cyclictest0-21swapper/223:20:182
2057099199,4cyclictest0-21swapper/223:20:182
2057099196,3cyclictest86850irq/129-enp1s0-19:40:262
2057099193,8cyclictest0-21swapper/200:22:232
2057099192,4cyclictest3842-21smart_sda00:35:262
2057099192,10cyclictest0-21swapper/223:55:202
2057099191,6cyclictest23656-21/usr/sbin/munin19:15:262
2057099191,6cyclictest14634-21snmpd23:10:202
2057099191,6cyclictest0-21swapper/223:19:082
2057099191,6cyclictest0-21swapper/221:57:082
2057099191,11cyclictest0-21swapper/222:29:062
2057099191,11cyclictest0-21swapper/221:27:212
2057099191,10cyclictest14634-21snmpd20:04:452
2057099191,10cyclictest12509-21irqstats21:15:182
2057099191,10cyclictest0-21swapper/223:25:142
2055899198,3cyclictest0-21swapper/022:50:210
2055899197,3cyclictest26469-21/usr/sbin/munin20:35:270
2055899194,8cyclictest0-21swapper/022:33:160
2055899194,3cyclictest0-21swapper/022:09:060
2055899191,6cyclictest0-21swapper/023:06:290
2055899191,6cyclictest0-21swapper/022:16:220
2055899191,6cyclictest0-21swapper/022:16:220
2055899191,5cyclictest0-21swapper/000:01:000
20558991913,4cyclictest19645-21perf20:24:560
2055899191,13cyclictest0-21swapper/021:29:060
2055899191,10cyclictest7397-21ntpq23:30:210
2055899191,10cyclictest31480-21/usr/sbin/munin22:00:240
2055899191,10cyclictest24883-21sshd21:45:110
2055899190,9cyclictest7-21ksoftirqd/000:05:550
2055899190,8cyclictest0-21swapper/020:05:280
2057099185,7cyclictest0-21swapper/219:49:122
2057099185,3cyclictest86850irq/129-enp1s0-23:40:172
2057099182,8cyclictest0-21swapper/200:08:242
2057099182,4cyclictest24765-21irqstats00:10:212
2057099181,6cyclictest2677-21awk20:55:012
2057099181,6cyclictest22792-21cat19:15:012
2057099181,6cyclictest0-21swapper/221:22:562
2057099181,6cyclictest0-21swapper/220:55:242
2057099181,5cyclictest23369-21smartctl21:40:162
2057099181,5cyclictest0-21swapper/222:37:572
20570991810,2cyclictest86750irq/128-enp1s0-21:45:242
2055899188,2cyclictest23728-21sshd20:26:110
2055899186,3cyclictest0-21swapper/020:01:590
2055899185,5cyclictest9268-21diskmemload22:40:400
2055899185,10cyclictest7-21ksoftirqd/021:35:010
2055899184,7cyclictest0-21swapper/020:12:160
2055899182,10cyclictest0-21swapper/022:37:260
2055899181,5cyclictest0-21swapper/021:17:060
2055899181,5cyclictest0-21swapper/021:10:170
2055899181,5cyclictest0-21swapper/021:10:170
20558991813,3cyclictest28773-21cut20:40:140
20558991812,3cyclictest30597-21hwlatdetect23:10:140
20558991812,3cyclictest27085-21sshd21:50:110
2055899181,14cyclictest32445-21/usr/sbin/munin19:35:150
20558991811,4cyclictest18909-21/usr/sbin/munin20:15:220
2055899181,12cyclictest0-21swapper/023:58:000
2055899181,11cyclictest0-21swapper/000:35:520
20558991810,5cyclictest25190-21sendmail_mailqu00:10:260
2055899180,16cyclictest20894-21idleruntime-cro22:50:010
170322181,3sleep20-21swapper/223:54:242
295542170,3sleep20-21swapper/223:08:352
2057099173,3cyclictest86850irq/129-enp1s0-21:05:232
2057099173,3cyclictest86850irq/129-enp1s0-21:05:232
2057099171,9cyclictest0-21swapper/221:10:252
2057099171,9cyclictest0-21swapper/221:10:252
2057099171,8cyclictest0-21swapper/200:15:192
2057099171,6cyclictest27978-21cat19:25:022
2057099171,5cyclictest0-21swapper/223:47:512
2057099171,4cyclictest0-21swapper/222:45:092
20570991712,3cyclictest4905-21wc22:10:302
20570991712,3cyclictest28859-21irqstats19:25:192
20570991712,3cyclictest24512-21ls20:30:152
2057099171,10cyclictest0-21swapper/221:35:042
2057099170,9cyclictest836-21runrttasks20:44:092
2055899175,3cyclictest21438-21hddtemp_smartct19:10:180
2055899171,7cyclictest0-21swapper/000:15:190
2055899171,6cyclictest0-21swapper/023:01:130
20558991712,3cyclictest28343-21expr19:25:140
199552170,2sleep20-21swapper/221:31:172
2057099169,4cyclictest18139-21fschecks_time20:15:152
2057099161,6cyclictest0-21swapper/200:01:152
2057099161,4cyclictest32-21ksoftirqd/220:10:152
2057099161,4cyclictest22986-21memory20:25:222
20570991611,3cyclictest8868-21sshd19:55:112
2055899162,11cyclictest25860-21/usr/sbin/munin19:20:190
2055899161,14cyclictest0-21swapper/021:03:160
2055899161,12cyclictest2283-21/usr/sbin/munin19:40:190
2057099159,3cyclictest30885-21cpu20:45:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional