You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 13:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Nov 20, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
170002240166,22sleep10-21swapper/119:09:191
169912236164,21sleep30-21swapper/319:09:143
170192229158,22sleep00-21swapper/019:09:350
168522204165,26sleep20-21swapper/219:07:232
2798921480,4sleep11714199cyclictest23:41:251
17147999989,9cyclictest0-21swapper/223:24:302
17147996460,3cyclictest0-21swapper/223:50:522
17147995350,2cyclictest0-21swapper/221:15:082
116802450,3sleep3401ktimersoftd/300:20:123
27462380,1sleep30-21swapper/321:30:243
17152993626,3cyclictest0-21swapper/300:29:013
1715299353,20cyclictest24808-21diskmemload23:02:043
1715299353,20cyclictest24808-21diskmemload23:02:043
17152993521,8cyclictest0-21swapper/300:36:433
1715299341,16cyclictest0-21swapper/322:09:243
1715299341,13cyclictest0-21swapper/322:25:163
1715299341,13cyclictest0-21swapper/300:18:203
1715299341,12cyclictest0-21swapper/323:11:073
1713599348,12cyclictest12233-21munin-run21:55:000
1713599346,13cyclictest29214-21munin-run21:20:010
17152993213,7cyclictest0-21swapper/321:26:163
1715299321,22cyclictest0-21swapper/323:32:313
1715299321,19cyclictest0-21swapper/323:38:453
1715299321,14cyclictest0-21swapper/322:33:463
1715299321,13cyclictest0-21swapper/319:29:443
1715299321,13cyclictest0-21swapper/300:03:313
17152993118,3cyclictest0-21swapper/322:38:303
1715299311,20cyclictest0-21swapper/300:07:113
1715299311,20cyclictest0-21swapper/300:07:113
1715299311,16cyclictest0-21swapper/321:11:293
1715299311,14cyclictest0-21swapper/322:01:593
1715299311,13cyclictest0-21swapper/322:24:583
1715299301,20cyclictest0-21swapper/321:06:583
1715299301,20cyclictest0-21swapper/320:08:303
1715299301,19cyclictest0-21swapper/322:55:193
1715299301,19cyclictest0-21swapper/322:48:503
1715299301,14cyclictest0-21swapper/322:15:103
1715299301,13cyclictest29572-21ntpq23:45:263
1715299301,13cyclictest0-21swapper/323:59:583
1715299301,13cyclictest0-21swapper/322:50:523
1715299301,13cyclictest0-21swapper/321:46:243
1715299301,13cyclictest0-21swapper/300:11:533
1715299301,12cyclictest0-21swapper/321:50:403
1715299301,11cyclictest0-21swapper/322:11:193
17152992918,3cyclictest0-21swapper/323:23:233
1715299291,20cyclictest0-21swapper/320:50:533
1715299291,18cyclictest0-21swapper/321:43:583
1715299291,14cyclictest0-21swapper/321:56:403
1715299291,14cyclictest0-21swapper/320:30:323
1715299291,12cyclictest0-21swapper/323:26:193
1715299291,12cyclictest0-21swapper/323:05:333
1715299291,12cyclictest0-21swapper/321:22:203
1713599291,13cyclictest21677-21idleruntime-cro23:30:010
1715299281,12cyclictest0-21swapper/322:43:563
1715299281,12cyclictest0-21swapper/320:28:583
1715299281,12cyclictest0-21swapper/319:30:043
1715299281,11cyclictest0-21swapper/323:52:263
1715299281,10cyclictest0-21swapper/319:55:323
1715299280,11cyclictest0-21swapper/320:16:403
1714199281,18cyclictest27787-21ps20:00:361
1713599284,22cyclictest87950irq/132-enp2s0-23:07:470
1713599281,10cyclictest30676-21kworker/0:020:10:250
1715299278,9cyclictest0-21swapper/320:24:383
17152992715,3cyclictest0-21swapper/319:42:063
1715299271,19cyclictest0-21swapper/320:58:013
1715299271,19cyclictest0-21swapper/320:38:093
1715299271,18cyclictest0-21swapper/320:48:173
1715299271,11cyclictest0-21swapper/321:35:043
1715299271,11cyclictest0-21swapper/319:38:343
1715299270,15cyclictest0-21swapper/323:15:353
1714199271,17cyclictest0-21swapper/123:30:151
1715299269,3cyclictest0-21swapper/319:24:163
17152992618,3cyclictest0-21swapper/319:51:103
17152992618,3cyclictest0-21swapper/319:51:103
17152992613,7cyclictest0-21swapper/319:47:103
1715299261,18cyclictest0-21swapper/321:18:003
1715299261,17cyclictest0-21swapper/323:42:373
1715299261,10cyclictest0-21swapper/319:13:213
17147992622,2cyclictest20502-21timerwakeupswit22:10:232
1714199261,13cyclictest28942-21ntp_states22:30:221
1715299251,16cyclictest0-21swapper/320:10:183
1715299251,13cyclictest0-21swapper/320:40:273
1715299251,11cyclictest0-21swapper/320:04:063
1715299251,11cyclictest0-21swapper/300:30:553
17141992519,3cyclictest2106-21unixbench_singl23:55:271
17141992518,4cyclictest17722-21munin-run19:15:021
1714199251,6cyclictest15559-21latency22:00:211
1714199251,16cyclictest15989-21smartctl23:15:171
1714199251,10cyclictest18846-21sh22:10:011
1713599250,13cyclictest991-21kworker/0:221:30:250
62192240,4sleep11714199cyclictest00:05:251
62192240,4sleep11714199cyclictest00:05:251
1715299240,12cyclictest0-21swapper/321:01:033
1714799248,6cyclictest0-21swapper/221:10:282
1714799244,7cyclictest0-21swapper/222:30:272
1714199248,3cyclictest939-21tr20:15:141
17141992417,4cyclictest2738-40updatedb00:00:111
17141992415,3cyclictest0-21swapper/122:11:311
1714199241,16cyclictest0-21swapper/123:28:471
1714199241,16cyclictest0-21swapper/123:05:201
1714199240,22cyclictest0-21swapper/120:10:011
1713599246,15cyclictest19010-21idleruntime-cro19:20:020
1715299237,3cyclictest0-21swapper/319:19:373
17147992311,2cyclictest17955-21awk00:35:022
17141992315,3cyclictest0-21swapper/119:41:311
17141992314,2cyclictest0-21swapper/119:35:001
1714199231,3cyclictest29217-21perf21:20:021
1714199231,19cyclictest8272-21pluginstate00:10:231
1714199231,19cyclictest3974-21sed20:20:221
1714199231,16cyclictest0-21swapper/100:32:351
1714199231,15cyclictest0-21swapper/122:45:141
1714199231,14cyclictest0-21swapper/121:56:031
1714199230,6cyclictest0-21swapper/100:25:341
1714199230,18cyclictest0-21swapper/100:35:291
1714199230,17cyclictest0-21swapper/123:37:491
1713599231,9cyclictest15239-21kworker/0:020:53:050
1713599231,16cyclictest0-21swapper/021:40:240
1714799226,4cyclictest1421-21grep20:15:182
1714799222,6cyclictest14634-21snmpd19:55:102
1714199229,4cyclictest6048-21irqstats20:25:181
1714199227,9cyclictest0-21swapper/122:40:331
1714199225,9cyclictest2642-21grep21:30:221
1714199222,8cyclictest0-21swapper/122:15:141
17141992220,1cyclictest0-21swapper/121:45:011
17141992216,3cyclictest7174-21hddtemp_smartct22:55:191
17141992216,3cyclictest4480-21irqstats21:35:181
17141992214,3cyclictest0-21swapper/120:51:351
1714199221,2cyclictest0-21swapper/119:50:321
1714199221,2cyclictest0-21swapper/119:50:321
1714199221,19cyclictest0-21swapper/123:23:511
1714199221,16cyclictest0-21swapper/121:48:031
1714199221,16cyclictest0-21swapper/119:16:281
1714199221,15cyclictest23834-21kworker/u8:100:23:311
1714199221,15cyclictest0-21swapper/122:26:071
1714199221,15cyclictest0-21swapper/122:23:461
1714199221,15cyclictest0-21swapper/121:20:231
1714199221,15cyclictest0-21swapper/121:00:201
1714199221,15cyclictest0-21swapper/120:32:501
1714199221,15cyclictest0-21swapper/120:13:241
17141992210,4cyclictest30569-21cat23:50:011
1714199220,16cyclictest0-21swapper/122:39:021
1713599221,8cyclictest0-21swapper/000:31:370
1713599221,7cyclictest23351-21kworker/0:219:55:110
1713599221,12cyclictest3815-21ntp_states00:00:210
1713599221,12cyclictest11-21rcuc/022:00:190
231522210,4sleep01713599cyclictest19:50:480
231522210,4sleep01713599cyclictest19:50:480
1714799212,6cyclictest31021-21users21:20:252
1714799212,5cyclictest24171-21munin-run19:55:012
1714799212,5cyclictest24171-21munin-run19:55:012
1714799212,15cyclictest27749-21cat22:30:022
1714799211,7cyclictest2903-21cat22:45:182
17147992115,3cyclictest400-21find21:25:262
1714799211,10cyclictest1400-21unixbench_multi22:40:292
1714199216,9cyclictest0-21swapper/121:28:561
1714199216,3cyclictest10935-21hddtemp_smartct21:50:171
1714199211,18cyclictest0-21swapper/119:23:211
1714199211,15cyclictest0-21swapper/123:12:301
1714199211,15cyclictest0-21swapper/123:02:251
1714199211,15cyclictest0-21swapper/123:02:251
1714199211,15cyclictest0-21swapper/122:51:241
1714199211,15cyclictest0-21swapper/121:07:231
1714199211,15cyclictest0-21swapper/120:40:441
1714199211,15cyclictest0-21swapper/120:35:321
1714199211,15cyclictest0-21swapper/119:29:091
1714199211,15cyclictest0-21swapper/100:18:271
1714199211,12cyclictest0-21swapper/121:10:551
17141992110,1cyclictest0-21swapper/120:55:141
1714199210,10cyclictest0-21swapper/123:50:221
1713599216,2cyclictest20151-21kworker/0:121:00:230
1713599215,6cyclictest0-21swapper/022:24:520
1713599214,10cyclictest11-21rcuc/021:10:220
1713599211,12cyclictest8232-21ntp_states00:10:210
93212201,2sleep20-21swapper/220:34:082
1714799204,4cyclictest9015-21sshd00:13:352
1714799204,3cyclictest15087-21perf23:15:012
1714799203,4cyclictest10622-21needreboot20:35:202
1714799202,8cyclictest5642-21munin-run21:40:022
1714799202,15cyclictest19392-21grep00:35:252
1714799201,5cyclictest1470-21cpu21:30:112
17147992015,3cyclictest31665-21grep20:10:182
17147992013,4cyclictest23887-21perf23:35:022
1714199201,11cyclictest24387-21/usr/sbin/munin19:55:091
1714199201,10cyclictest0-21swapper/120:45:161
1714199200,15cyclictest0-21swapper/119:47:281
1713599206,5cyclictest0-21swapper/000:07:270
1713599206,5cyclictest0-21swapper/000:07:270
1713599201,9cyclictest8452-21kworker/0:023:02:090
1713599201,9cyclictest8452-21kworker/0:023:02:090
17135992011,7cyclictest0-21swapper/023:45:010
1713599201,11cyclictest3668-21/usr/sbin/munin21:35:240
1713599201,11cyclictest22838-21/usr/sbin/munin21:05:180
1713599201,10cyclictest0-21swapper/022:34:330
1713599200,9cyclictest0-21swapper/000:20:430
323722191,3sleep00-21swapper/023:51:190
1714799195,3cyclictest86850irq/129-enp1s0-23:45:272
1714799194,3cyclictest86750irq/128-enp1s0-20:55:142
1714799194,3cyclictest86550irq/126-enp1s0-00:15:152
1714799193,3cyclictest13343-21tail20:40:272
1714799191,7cyclictest10264-21/usr/sbin/munin21:50:192
1714799191,6cyclictest27408-21/usr/sbin/munin23:40:192
1714799191,6cyclictest12069-21/usr/sbin/munin23:05:222
1714799191,6cyclictest10140-21timerwakeupswit23:00:262
1714799191,6cyclictest10140-21timerwakeupswit23:00:262
17147991915,2cyclictest30072-21/usr/sbin/munin22:35:212
1714799191,4cyclictest0-21swapper/223:39:312
17147991912,4cyclictest20367-21grep23:25:192
1714799191,11cyclictest0-21swapper/200:28:022
17141991912,2cyclictest0-21swapper/119:36:291
1713599196,4cyclictest0-21swapper/022:52:240
1713599196,3cyclictest0-21swapper/000:35:090
1713599192,9cyclictest24808-21diskmemload23:30:260
1713599191,8cyclictest0-21swapper/020:30:230
1713599191,10cyclictest32274-21/usr/sbin/munin22:40:250
1713599191,10cyclictest26600-21/usr/sbin/munin20:00:160
1713599191,10cyclictest19751-21hddtemp_smartct22:10:130
1713599191,10cyclictest13760-21/usr/sbin/munin00:25:220
1713599191,10cyclictest12458-21/usr/sbin/munin21:55:250
1713599191,10cyclictest0-21swapper/021:45:110
1713599191,10cyclictest0-21swapper/000:18:410
46632181,3sleep20-21swapper/200:04:192
252292181,2sleep20-21swapper/222:22:182
223292181,2sleep00-21swapper/019:46:010
175702181,3sleep20-21swapper/219:13:142
1714799185,3cyclictest86550irq/126-enp1s0-22:05:232
1714799184,3cyclictest12050irq/123-ahci[0020:05:022
1714799183,3cyclictest86850irq/129-enp1s0-21:55:152
1714799182,6cyclictest0-21swapper/222:04:172
1714799181,8cyclictest850-21/usr/sbin/munin23:55:152
1714799181,5cyclictest6503-21/usr/sbin/munin22:55:212
1714799181,5cyclictest4959-21/usr/sbin/munin00:05:142
1714799181,5cyclictest4959-21/usr/sbin/munin00:05:142
1714799181,5cyclictest13199-21taskset00:20:522
1714799181,5cyclictest0-21swapper/221:46:052
1714799181,4cyclictest17596-21sshd20:50:282
17147991811,4cyclictest29906-21pluginstate20:05:222
1714799180,17cyclictest0-21swapper/222:51:192
1713599187,3cyclictest14027-21/usr/sbin/munin20:45:110
1713599187,2cyclictest4996-21kworker/0:120:36:490
1713599185,2cyclictest18922-21kworker/0:019:22:150
1713599182,3cyclictest88050irq/133-enp2s0-19:40:570
1713599181,7cyclictest0-21swapper/019:10:210
1713599181,6cyclictest0-21swapper/021:28:450
17135991814,3cyclictest7-21ksoftirqd/023:20:370
17135991812,3cyclictest21886-21grep22:15:160
1713599181,10cyclictest0-21swapper/023:35:110
1713599180,9cyclictest0-21swapper/023:48:200
1713599180,6cyclictest0-21swapper/022:36:570
1713599180,11cyclictest0-21swapper/023:10:170
200332171,2sleep00-21swapper/020:55:280
1714799173,3cyclictest86550irq/126-enp1s0-19:24:102
1714799172,3cyclictest19187-21cat19:20:022
1714799171,7cyclictest23577-21if_enp1s021:05:172
1714799171,5cyclictest0-21swapper/223:17:572
1714799171,15cyclictest0-21swapper/221:00:462
1714799171,13cyclictest14520-21tune2fs20:45:152
1713599178,2cyclictest30426-21strings21:20:170
1713599177,7cyclictest17247-21fschecks_count22:05:130
1713599175,6cyclictest0-21swapper/020:05:010
17135991713,3cyclictest0-21swapper/023:15:180
17135991713,3cyclictest0-21swapper/019:30:030
17135991712,3cyclictest4919-21seq20:24:000
17135991711,4cyclictest12490-21grep20:40:170
1713599171,13cyclictest0-21swapper/023:56:140
1713599171,11cyclictest0-21swapper/022:25:290
1713599171,10cyclictest0-21swapper/019:35:050
17135991710,4cyclictest0-21swapper/022:45:320
1714799161,6cyclictest22731-21cat19:50:012
1714799161,6cyclictest0-21swapper/222:18:282
1714799161,3cyclictest0-21swapper/219:28:122
1714799161,2cyclictest0-21swapper/219:40:322
1714799161,14cyclictest0-21swapper/219:30:422
17147991611,3cyclictest6305-21ntp_states20:25:222
17147991611,3cyclictest5865-21/usr/sbin/munin21:40:162
1714799160,3cyclictest3750-21idleruntime20:20:202
1713599168,2cyclictest0-21swapper/022:55:190
1713599165,3cyclictest86750irq/128-enp1s0-20:15:120
1713599162,7cyclictest19564-21kworker/0:119:29:310
17135991611,2cyclictest5236-21/usr/sbin/munin20:25:160
1714799127,3cyclictest210642chrt19:35:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional