You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-13 - 11:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Nov 13, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
221172235162,59sleep00-21swapper/019:07:020
221172235162,59sleep00-21swapper/019:07:020
223152205168,24sleep10-21swapper/119:09:371
223152205168,24sleep10-21swapper/119:09:371
220672204168,23sleep30-21swapper/319:06:233
220672204168,23sleep30-21swapper/319:06:233
223012203166,25sleep20-21swapper/219:09:252
223012203166,25sleep20-21swapper/219:09:242
1592721640,4sleep12244399cyclictest23:45:221
154742740,1sleep10-21swapper/122:30:261
22448996859,8cyclictest0-21swapper/223:46:162
175052560,1sleep20-21swapper/220:10:142
175052560,1sleep20-21swapper/220:10:142
186642480,2sleep30-21swapper/321:25:183
2243799441,29cyclictest12033-21/usr/sbin/munin22:25:250
2243799428,23cyclictest32751-21/usr/sbin/munin20:45:250
281222410,1sleep10-21swapper/123:00:171
2243799411,26cyclictest30556-21/usr/sbin/munin20:40:170
2243799389,14cyclictest22165-21munin-run21:35:020
2243799371,4cyclictest1-21systemd23:25:430
2243799371,4cyclictest1-21systemd23:25:430
2243799371,24cyclictest3869-21/usr/sbin/munin23:20:240
22437993611,22cyclictest8267-21/usr/sbin/munin23:30:270
2243799360,4cyclictest14597-21kworker/0:120:15:210
2243799352,3cyclictest31120-21grep23:05:260
2243799351,4cyclictest32252-21latency_hist00:25:000
2243799351,2cyclictest0-21swapper/000:31:440
2243799351,2cyclictest0-21swapper/000:31:440
2243799351,20cyclictest31903-21munin-run19:30:010
2243799348,12cyclictest6509-21perf00:40:000
2243799344,14cyclictest1737-21munin-run19:35:010
2243799343,3cyclictest21289-21latency_hist00:00:010
2243799341,5cyclictest86550irq/126-enp1s0-21:35:170
2243799341,10cyclictest0-21swapper/022:38:310
22437993410,21cyclictest375-21cat00:25:160
2243799340,22cyclictest0-21swapper/021:20:250
2243799337,3cyclictest27086-21sshd22:58:430
2243799337,3cyclictest21460-21/usr/sbin/munin00:00:210
2243799334,3cyclictest6360-21/usr/sbin/munin19:45:260
2243799331,5cyclictest28996-21/usr/sbin/munin21:50:270
2243799331,4cyclictest26725-21cut00:10:180
2243799331,3cyclictest384-21chrt21:56:140
2243799331,28cyclictest14634-21snmpd20:05:200
2243799321,6cyclictest0-21swapper/021:42:510
2243799321,4cyclictest29016-21users23:00:270
2243799312,5cyclictest29426-21ntp_states20:35:250
2243799311,4cyclictest14634-21snmpd20:50:290
2243799311,4cyclictest14573-21df22:30:160
22437993110,18cyclictest9893-21cat19:50:250
22437993110,18cyclictest6992-21grep22:10:280
22437993110,18cyclictest14175-21/usr/sbin/munin21:15:140
22437993110,18cyclictest14175-21/usr/sbin/munin21:15:140
22437993110,18cyclictest11091-21sh19:55:110
2243799310,5cyclictest22383-21kworker/0:219:10:230
2243799310,24cyclictest26158-21kworker/0:020:32:070
2243799308,19cyclictest1-21systemd22:50:090
2243799306,21cyclictest12954-21/usr/sbin/munin20:00:170
2243799301,6cyclictest3115-21pmu-power19:35:230
2243799301,4cyclictest6138-21perf19:45:010
2243799301,4cyclictest24416-21/usr/sbin/munin00:05:170
2243799301,4cyclictest21590-21/usr/sbin/munin22:45:210
2243799301,4cyclictest19440-21/usr/sbin/munin22:40:180
2243799301,4cyclictest11400-21/usr/sbin/munin21:10:270
2243799301,27cyclictest87950irq/132-enp2s0-20:58:510
2243799298,18cyclictest1866-21kernelversion22:00:210
2243799294,5cyclictest27494-21crond19:20:000
2243799293,4cyclictest0-21swapper/022:05:140
2243799292,4cyclictest18730-21/usr/sbin/munin21:25:190
2243799292,3cyclictest848-21sssd_nss20:20:240
2243799292,3cyclictest10265-21/usr/sbin/munin21:05:210
2243799291,4cyclictest32190-21/usr/sbin/munin23:10:140
2243799291,4cyclictest28163-21/usr/sbin/munin21:45:270
2243799291,4cyclictest28106-21cpu00:15:110
2243799291,4cyclictest18575-21/usr/sbin/munin20:10:240
2243799291,4cyclictest18575-21/usr/sbin/munin20:10:240
2243799291,4cyclictest15557-21/usr/sbin/munin23:45:160
2243799291,3cyclictest17755-21/usr/sbin/munin23:50:160
2243799291,3cyclictest13380-21/usr/sbin/munin23:40:140
2243799291,3cyclictest11721-21/usr/sbin/munin23:35:250
2243799291,19cyclictest9396-21kworker/0:122:20:120
2243799291,19cyclictest9396-21kworker/0:122:20:110
22437992910,16cyclictest7073-21cpuspeed_turbos21:00:120
2243799288,17cyclictest24492-21fschecks_time20:25:200
2243799287,3cyclictest28215-21fschecks_time19:20:160
2243799281,4cyclictest2847-21/usr/sbin/munin23:15:230
2243799281,3cyclictest8445-21irqstats22:15:190
2244399271,18cyclictest28577-40dnf21:47:441
2244399261,17cyclictest29636-21/usr/sbin/munin23:05:211
2244399261,15cyclictest15560-21latency_hist21:20:011
2244399261,15cyclictest15560-21latency_hist21:20:011
22443992518,4cyclictest22816-21/usr/sbin/munin20:20:211
2244399251,3cyclictest8976-21perf21:05:001
2244399250,6cyclictest0-21swapper/123:54:321
2244399250,10cyclictest0-21swapper/122:05:591
8172240,2sleep232-21ksoftirqd/223:10:262
229272240,10sleep123-21ksoftirqd/122:50:011
2245499248,3cyclictest11818-21fschecks_count21:10:173
2244899242,11cyclictest0-21swapper/223:50:172
2244399241,17cyclictest86750irq/128-enp1s0-19:46:171
2245499236,4cyclictest0-21swapper/322:30:283
2245499232,7cyclictest86550irq/126-enp1s0-00:35:493
2245499231,11cyclictest14860-21/usr/sbin/munin23:45:113
2244399231,3cyclictest21596-21latency_hist20:20:011
2244399231,3cyclictest11526-21ntp_states23:35:221
22443992311,4cyclictest0-21swapper/122:22:101
22443992311,4cyclictest0-21swapper/122:22:101
2244399230,17cyclictest0-21swapper/123:29:471
2244399230,17cyclictest0-21swapper/123:29:471
2245499227,8cyclictest2-21kthreadd22:46:463
2245499222,17cyclictest13608-21tail20:00:173
2245499221,12cyclictest23027-21/usr/sbin/munin22:50:173
2245499221,11cyclictest8561-21/usr/sbin/munin19:50:153
2244899220,10cyclictest0-21swapper/221:44:572
22443992215,2cyclictest0-21swapper/122:39:561
2244399221,3cyclictest26163-21/usr/sbin/munin20:30:181
2244399221,18cyclictest0-21swapper/119:10:161
2244399221,16cyclictest0-21swapper/121:55:071
2244399221,16cyclictest0-21swapper/100:12:051
2244399221,15cyclictest0-21swapper/123:57:361
2244399221,15cyclictest0-21swapper/122:19:231
2244399221,12cyclictest24741-21ntp_states00:05:231
2244399220,15cyclictest0-21swapper/120:40:031
21992220,3sleep230-21rcuc/223:15:162
2245499217,5cyclictest10199-21fschecks_count22:20:163
2245499217,5cyclictest10199-21fschecks_count22:20:163
2245499211,9cyclictest0-21swapper/322:25:243
2245499211,6cyclictest27263-21wc23:00:023
2245499211,6cyclictest0-21swapper/300:11:383
2245499211,13cyclictest0-21swapper/320:21:503
2245499210,2cyclictest0-21swapper/323:09:043
2244899214,5cyclictest0-21swapper/222:47:162
2244899214,4cyclictest0-21swapper/221:56:182
2244899211,5cyclictest12371-21df22:25:172
2244399217,6cyclictest0-21swapper/121:25:161
2244399215,4cyclictest0-21swapper/123:40:111
2244399211,18cyclictest0-21swapper/119:35:021
2244399211,17cyclictest130962sleep121:11:381
2244399211,15cyclictest0-21swapper/123:30:181
2244399211,15cyclictest0-21swapper/123:21:571
2244399211,15cyclictest0-21swapper/122:28:341
2244399211,15cyclictest0-21swapper/121:52:311
2244399211,15cyclictest0-21swapper/120:56:061
2244399211,15cyclictest0-21swapper/119:16:481
2244399211,15cyclictest0-21swapper/100:33:221
2244399211,15cyclictest0-21swapper/100:33:221
2244399211,15cyclictest0-21swapper/100:19:521
2244399211,14cyclictest0-21swapper/122:41:421
2244399211,14cyclictest0-21swapper/122:14:051
2244399211,14cyclictest0-21swapper/120:46:221
2244399211,11cyclictest4511-21diskstats19:40:161
2244399211,11cyclictest3148-21perl20:50:151
2244399211,11cyclictest28080-21diskstats19:20:141
2244399211,11cyclictest26687-21cat21:45:011
2244399210,19cyclictest0-21swapper/100:03:441
2245499207,3cyclictest6686-21cstates19:45:163
2245499204,4cyclictest2374-21perf20:50:023
2245499202,10cyclictest26163-21/usr/sbin/munin20:30:243
2245499201,7cyclictest16778-21diskstats22:35:173
2245499201,6cyclictest0-21swapper/323:35:173
2245499201,12cyclictest0-21swapper/323:21:393
2244899206,4cyclictest21873-21tune2fs00:00:152
2244899204,7cyclictest780-21rngd21:30:352
2244899202,7cyclictest0-21swapper/221:27:432
2244899202,3cyclictest8037-21perf23:30:012
2244899202,3cyclictest8037-21perf23:30:012
2244899201,9cyclictest5588-21ntp_states00:35:232
2244899201,7cyclictest0-21swapper/200:05:142
2244899201,6cyclictest0-21swapper/222:50:082
22448992014,3cyclictest29777-21cut23:05:142
22448992013,4cyclictest28577-21dnf21:47:442
22448992011,2cyclictest30334-21awk20:40:022
22443992013,4cyclictest32174-21df_inode23:10:131
22443992012,3cyclictest0-21swapper/122:50:231
2244399201,16cyclictest16129-21df21:20:131
2244399201,14cyclictest0-21swapper/100:36:341
2244399201,10cyclictest32476-21seq00:25:071
2244399201,10cyclictest29100-21if_enp1s020:35:201
22443992010,5cyclictest0-21swapper/121:31:141
22443992010,2cyclictest11168-21diskmemload21:35:171
2244399200,15cyclictest0-21swapper/122:55:371
2244399200,15cyclictest0-21swapper/121:06:421
2244399200,14cyclictest0-21swapper/122:00:231
300202191,2sleep20-21swapper/221:50:222
2245499191,7cyclictest31721-21cat00:20:263
2245499191,7cyclictest0-21swapper/323:57:173
2245499191,6cyclictest8309-21hddtemp_smartct22:15:183
2245499191,4cyclictest0-21swapper/322:11:503
22454991913,3cyclictest15920-21cpuspeed_turbos21:20:133
22454991913,3cyclictest11170-21grep19:55:133
22454991912,4cyclictest28703-21df_inode20:35:183
2245499191,10cyclictest27469-21sshd19:19:453
2244899196,2cyclictest6064-21kworker/u8:021:15:262
2244899196,2cyclictest6064-21kworker/u8:021:15:262
2244899195,4cyclictest17908-21unixbench-2d22:35:282
2244899194,7cyclictest780-21rngd00:27:482
2244899194,3cyclictest0-21swapper/222:34:412
2244899193,5cyclictest0-21swapper/222:05:322
2244899192,9cyclictest22985-21df_inode19:10:152
2244899192,5cyclictest14634-21snmpd20:00:202
2244899192,5cyclictest0-21swapper/222:59:352
2244899191,8cyclictest4171-21acpi19:40:152
2244899191,4cyclictest0-21swapper/223:34:222
22448991914,3cyclictest10606-21tr23:35:142
2244899190,4cyclictest17726-21kworker/u8:121:39:462
22443991914,3cyclictest9241-21hwlatdetect19:50:171
22443991912,2cyclictest0-21swapper/123:16:561
2244399191,16cyclictest0-21swapper/119:25:071
22443991911,3cyclictest0-21swapper/120:25:551
2244399190,17cyclictest0-21swapper/120:00:221
2244399190,15cyclictest0-21swapper/100:21:061
231722180,2sleep30-21swapper/300:01:383
2245499186,3cyclictest86550irq/126-enp1s0-23:12:453
2245499182,13cyclictest23677-21/usr/sbin/munin00:05:243
2245499181,9cyclictest41-21ksoftirqd/321:44:333
2245499181,9cyclictest0-21swapper/323:16:183
2245499181,8cyclictest768-21in:imjournal23:41:583
2245499181,8cyclictest0-21swapper/322:40:133
2245499181,7cyclictest4338-21cat00:35:013
2245499181,7cyclictest4338-21cat00:35:013
2245499181,7cyclictest23882-21unixbench_multi21:35:283
2245499181,7cyclictest0-21swapper/323:31:313
2245499181,7cyclictest0-21swapper/300:25:243
2245499181,6cyclictest15182-21users21:15:263
2245499181,6cyclictest15182-21users21:15:263
2245499181,4cyclictest21522-21latency_hist20:20:023
22454991813,3cyclictest2595-21grep19:35:183
22454991812,3cyclictest4045-21interrupts22:05:193
22454991811,4cyclictest4169-21grep20:50:273
22454991810,4cyclictest29655-21gltestperf21:50:183
2244899185,6cyclictest0-21swapper/223:04:152
2244899185,3cyclictest21791-21cpu20:20:132
2244899184,5cyclictest780-21rngd00:10:082
2244899184,3cyclictest8141-21sshd19:47:452
2244899184,3cyclictest28090-21/usr/sbin/munin00:15:192
2244899183,10cyclictest86550irq/126-enp1s0-19:36:182
2244899182,8cyclictest12658-21/usr/sbin/munin23:40:242
2244899181,7cyclictest21503-21chrt20:19:542
2244899181,7cyclictest15802-21/usr/sbin/munin21:20:132
2244899181,5cyclictest7962-21cat22:15:142
2244899181,4cyclictest3980-21cut23:20:132
22448991811,4cyclictest20806-21grep23:55:292
2244399182,3cyclictest0-21swapper/120:05:151
2244399181,16cyclictest0-21swapper/119:56:541
2244399181,16cyclictest0-21swapper/119:31:331
2244399180,16cyclictest0-21swapper/120:10:081
2244399180,16cyclictest0-21swapper/120:10:081
86322171,2sleep20-21swapper/221:01:012
2245499179,5cyclictest10679-21grep21:05:263
2245499171,6cyclictest6207-21tr23:25:173
2245499171,6cyclictest6207-21tr23:25:173
2245499171,6cyclictest32056-21cat19:30:023
2245499171,6cyclictest31793-21tail21:55:173
2245499171,6cyclictest29697-21latency_hist19:25:023
2245499171,6cyclictest27144-21diskstats21:45:173
22454991712,3cyclictest4790-21/usr/sbin/munin20:55:173
22454991712,3cyclictest4451-21expr19:40:163
22454991711,3cyclictest17363-21/usr/sbin/munin20:10:253
22454991711,3cyclictest17363-21/usr/sbin/munin20:10:253
22454991711,3cyclictest1082-21timerandwakeup19:30:253
2245499171,11cyclictest17059-21/usr/sbin/munin23:50:193
2245499171,11cyclictest0-21swapper/300:17:323
2245499170,11cyclictest0-21swapper/320:05:283
2244899174,3cyclictest86550irq/126-enp1s0-22:02:432
2244899172,13cyclictest10253-21fschecks_count22:20:162
2244899172,13cyclictest10253-21fschecks_count22:20:162
2244899171,6cyclictest30843-21perl00:20:172
2244899171,5cyclictest5786-21diskstats22:10:162
2244899171,5cyclictest2659-21df00:30:152
2244899171,5cyclictest2659-21df00:30:152
2244899171,5cyclictest0-21swapper/220:05:212
2244899171,3cyclictest9638-21fschecks_count21:05:162
22448991712,3cyclictest29934-21/usr/sbin/munin19:25:222
22448991712,3cyclictest28557-21kernelversion19:20:192
22448991712,3cyclictest1408-21/usr/sbin/munin20:45:212
22448991712,3cyclictest11506-21cut21:10:142
2244899171,15cyclictest0-21swapper/220:59:082
2244899171,15cyclictest0-21swapper/220:41:142
2244899171,14cyclictest19270-21grep22:40:162
22448991711,3cyclictest8561-21/usr/sbin/munin19:50:182
2244899170,15cyclictest0-21swapper/219:15:192
2245499163,3cyclictest86550irq/126-enp1s0-21:31:273
2245499161,4cyclictest0-21swapper/322:01:193
22454991612,3cyclictest0-21swapper/319:11:203
2245499161,14cyclictest0-21swapper/320:27:483
22454991611,3cyclictest7594-21gltestperf21:00:163
22454991611,3cyclictest31483-21sort20:40:203
22454991610,3cyclictest27578-21cpuspeed_turbos23:00:143
2244899164,4cyclictest86550irq/126-enp1s0-20:25:192
2244899161,2cyclictest0-21swapper/220:55:022
22448991611,3cyclictest27655-21unixbench_multi20:30:272
22448991611,3cyclictest1343-21vmstat19:30:272
22448991510,3cyclictest11411-21gltestperf19:55:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional