You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 13:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Nov 19, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
177192234165,55sleep10-21swapper/119:06:021
176682225194,21sleep00-21swapper/019:05:310
178262221145,24sleep20-21swapper/219:07:222
177872217165,39sleep30-21swapper/319:06:513
181359910797,9cyclictest0-21swapper/221:10:552
18135999693,2cyclictest0-21swapper/219:39:322
18135996965,3cyclictest0-21swapper/221:15:262
41032640,4sleep011-21rcuc/022:15:190
23492550,4sleep20-21swapper/200:40:012
157152510,1sleep10-21swapper/123:55:201
301512500,2sleep10-21swapper/123:15:181
18125993910,13cyclictest23816-21munin-run20:35:010
18135993835,2cyclictest0-21swapper/223:27:592
18125993710,12cyclictest22891-21awk23:00:010
91872310,4sleep011-21rcuc/021:15:020
1813099311,24cyclictest0-21swapper/122:54:031
1813099300,24cyclictest0-21swapper/121:16:441
1812599301,13cyclictest30409-21munin-run20:50:010
1812599301,13cyclictest11253-21munin-run21:20:010
1813099295,17cyclictest7545-21timerandwakeup23:35:291
18130992721,3cyclictest18763-21hwlatdetect21:35:181
1812599278,16cyclictest23195-21basename19:20:010
158342270,5sleep230-21rcuc/221:30:012
1813599261,13cyclictest86550irq/126-enp1s0-20:23:352
1813099264,9cyclictest0-21swapper/121:52:131
1813099261,17cyclictest29332-21proc_pri00:25:261
1813099261,17cyclictest17322-21irqstats22:45:221
1813099261,17cyclictest17322-21irqstats22:45:221
18130992610,3cyclictest17842-21munin-run21:35:011
1814299251,4cyclictest0-21swapper/300:12:433
1813599251,11cyclictest0-21swapper/219:21:112
18130992519,3cyclictest1342-21cat20:55:201
1813099251,3cyclictest26906-21users23:05:271
1813099251,2cyclictest0-21swapper/100:12:431
18130992511,9cyclictest0-21swapper/123:46:121
1813099251,18cyclictest0-21swapper/123:31:201
1813099251,18cyclictest0-21swapper/100:22:411
1813099251,16cyclictest0-21swapper/119:15:191
1812599252,12cyclictest26411-40dnf21:52:130
1812599250,12cyclictest4022-21perf19:50:010
18142992418,3cyclictest2353-21find22:10:253
18142992418,3cyclictest23413-21/usr/sbin/munin19:20:273
18142992418,3cyclictest15246-21/usr/sbin/munin20:15:283
1814299241,18cyclictest86550irq/126-enp1s0-19:49:333
1813599241,12cyclictest0-21swapper/222:56:262
1813099249,3cyclictest27060-21irqstats20:40:221
18130992417,4cyclictest382-21cut19:40:161
1813099241,17cyclictest0-21swapper/123:54:071
1813099241,17cyclictest0-21swapper/123:54:071
1813099241,15cyclictest86750irq/128-enp1s0-22:12:171
1813099241,12cyclictest0-21swapper/122:09:111
1813099241,11cyclictest12825-21perf20:10:011
1813099241,10cyclictest22194-21diskstats20:30:151
1812599248,14cyclictest10632-21perf20:05:010
1812599248,12cyclictest9170-21kernelversion23:40:210
18125992414,7cyclictest32141-21systemd-journal00:32:400
1812599240,12cyclictest0-21swapper/022:40:450
18142992321,1cyclictest0-21swapper/320:05:143
1814299231,6cyclictest22847-21fschecks_count21:45:173
1814299231,3cyclictest3277-21grep21:00:183
1813099238,3cyclictest1497-21sh23:25:011
18130992320,2cyclictest0-21swapper/121:03:571
18130992320,2cyclictest0-21swapper/120:10:471
1813099231,5cyclictest0-21swapper/122:28:011
1813099231,3cyclictest287582chrt23:10:321
1813099231,3cyclictest16509-21swap20:15:281
18130992313,3cyclictest0-21swapper/122:20:211
1813099231,10cyclictest8399-21idleruntime-cro20:00:011
1813099230,5cyclictest0-21swapper/121:57:361
1813099230,17cyclictest0-21swapper/122:18:231
1812599231,9cyclictest14426-21kworker/0:223:52:390
1812599231,9cyclictest14426-21kworker/0:223:52:390
1814299226,8cyclictest12050irq/123-ahci[0019:20:013
1814299225,4cyclictest398-21cat00:35:163
1814299221,8cyclictest25896-21chrt21:50:253
18142992218,3cyclictest0-21swapper/323:26:253
1814299221,7cyclictest0-21swapper/323:47:043
1814299221,7cyclictest0-21swapper/322:46:243
1814299221,7cyclictest0-21swapper/322:46:243
1814299221,2cyclictest0-21swapper/322:23:333
1814299221,12cyclictest842-21lldpd21:30:223
1814299221,10cyclictest6257-21cpuspeed_turbos23:35:153
1814299221,10cyclictest14634-21snmpd21:15:523
1813599223,16cyclictest17029-21tr21:30:262
18135992216,3cyclictest10619-21grep20:05:012
1813599221,11cyclictest0-21swapper/221:05:312
1813099228,2cyclictest30538-21cstates00:30:141
1813099226,6cyclictest25967-21sshd20:39:361
1813099226,6cyclictest25967-21sshd20:39:361
1813099226,4cyclictest2821-21ntp_states23:25:241
1813099221,9cyclictest17731-21taskset00:00:171
18130992219,2cyclictest0-21swapper/121:22:361
18130992217,3cyclictest0-21swapper/122:00:171
18130992213,6cyclictest26873-21smart_sda19:25:261
1813099221,20cyclictest0-21swapper/121:05:221
1813099221,15cyclictest0-21swapper/123:40:091
1813099221,15cyclictest0-21swapper/123:00:331
1813099221,15cyclictest0-21swapper/122:31:201
1813099221,10cyclictest32582-21cpu00:35:151
1813099221,10cyclictest22541-21cpuspeed_turbos21:45:151
1813099221,10cyclictest21331-21ntpq21:40:221
1813099221,10cyclictest21331-21ntpq21:40:221
1812599228,3cyclictest276532if_err_enp1s021:55:180
1812599225,9cyclictest16511-21/usr/sbin/munin22:45:200
1812599225,9cyclictest16511-21/usr/sbin/munin22:45:200
1812599225,10cyclictest0-21swapper/019:25:200
18125992210,3cyclictest5152-21memory19:50:210
1814299217,3cyclictest12743-21/usr/sbin/munin23:50:273
1814299217,3cyclictest12743-21/usr/sbin/munin23:50:273
1814299212,10cyclictest0-21swapper/323:56:413
1814299211,9cyclictest41052if_err_enp2s022:15:193
1814299211,9cyclictest14816-21sshd20:12:343
18142992118,2cyclictest0-21swapper/300:16:253
1814299211,7cyclictest0-21swapper/321:56:353
18142992117,2cyclictest8638-21/usr/sbin/munin20:00:253
1814299211,5cyclictest19078-21/usr/sbin/munin21:35:223
18142992114,4cyclictest31698-21/usr/sbin/munin22:05:173
1814299211,12cyclictest86550irq/126-enp1s0-22:40:273
1814299211,11cyclictest31914-21/usr/sbin/munin23:20:253
1814299211,11cyclictest30640-21/usr/sbin/munin20:50:213
1814299211,10cyclictest16944-21kworker/u8:120:37:273
1814299211,10cyclictest16944-21kworker/u8:120:37:273
1813599215,2cyclictest0-21swapper/220:28:332
1813599215,10cyclictest0-21swapper/220:33:532
1813599212,9cyclictest30-21rcuc/220:50:152
1813599212,7cyclictest12050irq/123-ahci[0000:05:022
1813599211,9cyclictest0-21swapper/219:15:212
1813599211,12cyclictest0-21swapper/223:40:272
1813599211,11cyclictest0-21swapper/223:20:142
18135992110,3cyclictest86550irq/126-enp1s0-19:50:272
1813599210,11cyclictest1011-21kworker/dying19:25:152
1813099218,2cyclictest0-21swapper/119:14:071
1813099217,3cyclictest31380-21if_enp1s020:50:191
1813099217,11cyclictest20269-21load00:05:201
1813099215,6cyclictest0-21swapper/121:13:131
1813099213,9cyclictest0-21swapper/100:15:271
1813099213,11cyclictest86750irq/128-enp1s0-19:36:341
1813099211,17cyclictest28163-21/usr/sbin/munin19:30:181
1813099211,17cyclictest28163-21/usr/sbin/munin19:30:181
1813099211,17cyclictest12281-21cstates22:35:151
1813099211,11cyclictest5053-21kworker/u8:122:42:251
1813099211,11cyclictest25380-21latency_hist19:25:011
1813099211,11cyclictest10612-21latency_hist20:05:021
1813099210,19cyclictest0-21swapper/119:50:211
1813099210,10cyclictest0-21swapper/120:20:151
1812599217,5cyclictest0-21swapper/023:06:160
1812599212,11cyclictest366-21/usr/sbin/munin20:55:280
1812599211,11cyclictest32566-21/usr/sbin/munin00:35:230
1812599211,11cyclictest20432-21ntp_states00:05:230
265302201,3sleep20-21swapper/221:53:392
1814299208,3cyclictest0-21swapper/319:30:253
1814299208,3cyclictest0-21swapper/319:30:253
1814299205,5cyclictest768-21in:imjournal22:58:493
1814299201,7cyclictest7429-21diskstats21:10:163
1814299201,7cyclictest20276-21/usr/sbin/munin21:40:263
1814299201,7cyclictest20276-21/usr/sbin/munin21:40:263
1814299201,6cyclictest29116-21users23:10:353
1814299201,6cyclictest20287-21latency_hist19:15:013
1814299201,5cyclictest27945-21munin-run00:25:013
18142992014,4cyclictest842-21lldpd22:25:243
1814299201,16cyclictest19844-21grep22:50:243
1814299201,10cyclictest17149-21/usr/sbin/munin00:00:153
1814299201,10cyclictest0-21swapper/319:35:163
1813599206,3cyclictest0-21swapper/220:45:232
1813599204,9cyclictest0-21swapper/219:41:032
1813599204,6cyclictest11163-21grep23:45:172
1813599202,9cyclictest0-21swapper/220:59:512
1813599201,8cyclictest0-21swapper/220:14:392
1813599201,4cyclictest87950irq/132-enp2s0-22:54:412
18135992011,2cyclictest86550irq/126-enp1s0-22:10:282
1813099201,3cyclictest0-21swapper/120:27:511
1813099201,17cyclictest0-21swapper/121:25:031
1813099200,11cyclictest0-21swapper/122:56:301
1812599205,9cyclictest32704-21if_err_enp2s023:20:190
1812599205,3cyclictest81212grep23:40:010
1812599204,6cyclictest0-21swapper/023:17:310
1812599202,9cyclictest6859-21munin-run21:10:010
1812599202,8cyclictest29078-21/usr/sbin/munin22:00:240
1812599201,11cyclictest18713-21/usr/sbin/munin22:50:130
1812599201,10cyclictest0-21swapper/020:05:280
1812599200,6cyclictest0-21swapper/023:26:260
252162191,2sleep00-21swapper/019:22:590
212482191,2sleep00-21swapper/020:25:290
1814299198,3cyclictest29718-21/usr/sbin/munin23:15:183
1814299195,3cyclictest12050irq/123-ahci[0023:05:013
1814299194,3cyclictest86850irq/129-enp1s0-00:25:183
1814299194,3cyclictest12113-21/usr/sbin/munin22:35:273
1814299192,4cyclictest30541-21sed22:00:293
1814299191,9cyclictest0-21swapper/322:31:363
18142991917,1cyclictest0-21swapper/321:25:163
1814299191,6cyclictest30369-21/usr/sbin/munin00:30:193
1814299191,6cyclictest28441-21/usr/sbin/munin20:45:143
1814299191,6cyclictest0-21swapper/300:05:143
18142991913,3cyclictest32204-21/usr/sbin/munin19:40:123
1814299191,10cyclictest0-21swapper/323:42:163
1813599198,2cyclictest86850irq/129-enp1s0-22:24:122
1813599198,2cyclictest0-21swapper/223:10:342
1813599195,4cyclictest5283-21latency_hist22:20:012
1813599195,4cyclictest24832-21ntp_offset00:15:242
1813599195,3cyclictest0-21swapper/220:38:392
1813599195,3cyclictest0-21swapper/220:38:392
1813599193,3cyclictest21919-21kworker/2:200:21:052
1813599191,6cyclictest0-21swapper/223:16:112
1813599191,12cyclictest0-21swapper/223:37:582
1813099190,17cyclictest0-21swapper/120:45:281
1813099190,12cyclictest0-21swapper/119:45:301
1812599199,2cyclictest7695-21smart_sda19:55:250
1812599195,6cyclictest0-21swapper/021:41:080
1812599195,6cyclictest0-21swapper/021:41:080
1812599195,5cyclictest0-21swapper/023:55:510
1812599191,15cyclictest13848-21taskset22:37:110
1812599191,10cyclictest9468-21runrttasks22:27:250
1812599191,10cyclictest18322-21/usr/sbin/munin19:10:200
1812599191,10cyclictest0-21swapper/023:47:190
77502180,4sleep21813599cyclictest22:25:132
1814299184,3cyclictest86550irq/126-enp1s0-20:25:123
1814299184,3cyclictest4174-21cat23:30:143
1814299182,4cyclictest17774-21df20:20:143
1814299182,13cyclictest25320-21/usr/sbin/munin23:05:133
18142991813,3cyclictest13034-21sed21:20:283
18142991812,3cyclictest366-21/usr/sbin/munin20:55:213
1814299181,14cyclictest14634-21snmpd19:55:373
1813599188,3cyclictest86550irq/126-enp1s0-20:07:392
1813599187,3cyclictest86550irq/126-enp1s0-23:32:222
1813599187,2cyclictest86550irq/126-enp1s0-19:55:232
1813599186,3cyclictest0-21swapper/222:30:272
1813599185,6cyclictest86850irq/129-enp1s0-00:30:142
1813599184,4cyclictest0-21swapper/222:00:302
1813599183,8cyclictest86550irq/126-enp1s0-23:58:512
1813599182,7cyclictest20171-21sed21:40:012
1813599182,3cyclictest88250irq/135-enp2s0-21:00:252
1813599181,9cyclictest0-21swapper/221:20:182
1813599181,7cyclictest0-21swapper/222:45:592
1813599181,7cyclictest0-21swapper/222:45:592
1813599181,6cyclictest0-21swapper/222:05:202
1813599181,6cyclictest0-21swapper/221:48:312
1813599181,4cyclictest0-21swapper/222:42:312
18135991812,4cyclictest27393-21head20:40:272
1813599181,15cyclictest0-21swapper/219:10:222
1812599188,3cyclictest0-21swapper/021:25:220
1812599188,2cyclictest30640-21/usr/sbin/munin20:50:280
1812599187,4cyclictest0-21swapper/000:12:340
1812599187,2cyclictest18782-21idleruntime21:35:180
1812599184,7cyclictest0-21swapper/000:15:040
1812599184,4cyclictest0-21swapper/022:30:050
1812599182,5cyclictest19238-21kworker/0:320:23:350
1812599181,9cyclictest14634-21snmpd21:45:580
1812599181,9cyclictest12060-21perl21:20:160
1812599181,9cyclictest0-21swapper/023:03:000
1812599181,9cyclictest0-21swapper/000:29:420
1812599181,5cyclictest0-21swapper/022:23:190
18125991813,2cyclictest17464-21users21:30:300
18125991812,3cyclictest28174-21gltestperf23:10:240
18125991812,3cyclictest27980-21seq20:42:100
18125991810,5cyclictest0-21swapper/022:05:560
1812599180,5cyclictest0-21swapper/000:03:530
56822170,2sleep00-21swapper/023:32:120
255982171,3sleep30-21swapper/319:25:033
1814299172,3cyclictest86550irq/126-enp1s0-21:05:213
1814299171,6cyclictest0-21swapper/320:30:013
18142991713,2cyclictest26248-21/usr/sbin/munin20:40:243
18142991712,3cyclictest4307-21wc19:50:143
1813599177,2cyclictest30-21rcuc/221:57:062
1813599176,3cyclictest0-21swapper/222:35:132
1813599175,4cyclictest311ktimersoftd/219:31:342
1813599175,4cyclictest311ktimersoftd/219:31:342
1813599171,8cyclictest0-21swapper/200:07:032
18135991713,3cyclictest0-21swapper/223:51:582
18135991713,3cyclictest0-21swapper/223:51:572
18135991713,3cyclictest0-21swapper/223:06:512
1813599171,11cyclictest0-21swapper/219:46:432
1813599171,10cyclictest32-21ksoftirqd/200:25:222
1812599177,3cyclictest0-21swapper/000:20:410
1812599171,7cyclictest0-21swapper/019:32:020
1812599171,7cyclictest0-21swapper/019:32:020
1812599171,6cyclictest13579-21/usr/sbin/munin20:10:190
1812599171,6cyclictest0-21swapper/019:40:210
1812599171,3cyclictest1267-21cat22:10:140
18125991711,3cyclictest30009-21acpi19:35:130
168772171,2sleep00-21swapper/020:15:540
1813599161,8cyclictest0-21swapper/223:00:172
1813599161,5cyclictest0-21swapper/200:10:202
18135991612,3cyclictest0-21swapper/221:40:022
18135991612,3cyclictest0-21swapper/221:40:022
18125991612,3cyclictest0-21swapper/020:35:190
18125991612,3cyclictest0-21swapper/020:35:190
18125991611,2cyclictest2682-21acpi21:00:130
1813599153,4cyclictest16851-21sshd20:15:332
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional