You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 12:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Nov 30, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
46622204166,25sleep00-21swapper/019:09:520
46622204166,25sleep00-21swapper/019:09:520
44622204166,25sleep10-21swapper/119:07:191
44622204166,25sleep10-21swapper/119:07:181
45292203166,25sleep30-21swapper/319:08:093
45292203166,25sleep30-21swapper/319:08:093
43552202165,25sleep20-21swapper/219:05:552
43552202165,25sleep20-21swapper/219:05:552
790321690,4sleep1476399cyclictest00:09:011
3254521540,5sleep2477099cyclictest20:10:192
477099104101,2cyclictest0-21swapper/222:31:172
102232730,3sleep2311ktimersoftd/221:45:242
251512540,4sleep339-21rcuc/322:20:173
263112470,2sleep20-21swapper/221:10:162
263112470,2sleep20-21swapper/221:10:162
4757994212,14cyclictest640-21munin-run22:40:010
477099375,4cyclictest12050irq/123-ahci[0021:25:022
477099374,27cyclictest31231-40dnf23:46:352
477099371,21cyclictest0-21swapper/219:22:272
475799356,13cyclictest18982-21munin-run00:35:010
4757993511,11cyclictest25873-21idleruntime-cro21:10:000
67462340,4sleep3401ktimersoftd/300:05:183
477099345,3cyclictest12728-21date20:40:022
475799347,12cyclictest6053-21idleruntime-cro20:25:010
4775993317,8cyclictest0-21swapper/319:44:453
477099335,3cyclictest25852-21diskmemload23:15:012
477099332,28cyclictest21138-21cut19:45:152
4770993312,18cyclictest26081-21grep19:55:182
4770993312,18cyclictest13677-21grep21:55:142
477099331,14cyclictest0-21swapper/222:39:252
475799339,11cyclictest8906-21awk21:45:010
477099329,5cyclictest0-21swapper/200:00:282
477099328,4cyclictest7317-21switchtime00:05:262
477099325,3cyclictest16653-21sshd00:28:332
477099325,3cyclictest16653-21sshd00:28:332
477099321,8cyclictest17853-21kernelversion00:30:202
477099321,5cyclictest0-21swapper/221:26:442
4770993212,17cyclictest18244-21cat20:50:212
4770993210,18cyclictest8885-21idleruntime-cro21:45:022
475799322,14cyclictest9528-21munin-run23:00:010
477099317,2cyclictest17062-21awk20:50:022
477099317,20cyclictest0-21swapper/220:00:102
477099315,5cyclictest0-21swapper/222:45:202
477099312,5cyclictest5983-21smart_sda21:35:252
477099311,6cyclictest13680-21ntp_states00:20:232
477099311,5cyclictest5361-21/usr/sbin/munin22:50:312
477099311,27cyclictest848-21sssd_nss22:40:152
477099311,13cyclictest0-21swapper/222:26:372
477099311,12cyclictest0-21swapper/220:06:132
477099311,11cyclictest0-21swapper/223:51:132
4770993110,3cyclictest4485-21latency_hist21:35:022
4770993110,2cyclictest23646-21latency_hist21:05:022
4775993023,4cyclictest5063-21/usr/sbin/munin21:35:143
477099309,3cyclictest2801-21processes23:55:242
477099305,3cyclictest86850irq/129-enp1s0-21:05:152
477099302,7cyclictest0-21swapper/221:15:262
477099302,5cyclictest6938-21munin-run19:15:022
477099302,10cyclictest17690-21latency_hist22:05:022
477099301,7cyclictest8363-21interrupts22:55:202
4770993013,14cyclictest22329-21sed23:25:292
4770993011,3cyclictest8669-21cat20:30:152
477099301,12cyclictest0-21swapper/220:18:452
4770993011,16cyclictest8044-21ls19:15:162
4770993011,16cyclictest26275-21pmu-power23:35:232
4770993010,3cyclictest8231-21acpi00:10:142
4770993010,3cyclictest19713-21cat20:55:152
475799301,13cyclictest24292-21munin-run22:20:010
475799301,13cyclictest24292-21munin-run22:20:010
136162300,7sleep121-21rcuc/120:40:191
477599297,4cyclictest13502-21grep20:40:193
477099299,3cyclictest836-21runrttasks19:53:232
477099299,17cyclictest19831-21needreboot19:40:222
477099298,2cyclictest16358-21/usr/sbin/munin23:15:222
477099296,4cyclictest5352-21/usr/sbin/munin20:20:272
477099295,4cyclictest22902-21/usr/sbin/munin22:15:192
477099295,4cyclictest22902-21/usr/sbin/munin22:15:192
477099293,6cyclictest24207-21smart_sda23:30:242
477099292,6cyclictest0-21swapper/219:25:252
477099292,5cyclictest26495-21munin-run22:25:022
477099292,5cyclictest10750-21df00:15:142
477099291,6cyclictest0-21swapper/222:10:182
477099291,3cyclictest12050irq/123-ahci[0023:45:022
4770992910,3cyclictest10823-21ntpq23:00:242
477099290,17cyclictest0-21swapper/200:39:032
4757992910,16cyclictest29620-21munin-run20:05:000
477599285,9cyclictest0-21swapper/320:09:133
4775992811,4cyclictest18960-21ntpq22:05:223
477599281,10cyclictest31231-40dnf23:46:323
477099289,3cyclictest836-21runrttasks21:50:242
477099288,3cyclictest18925-21netstat22:05:232
477099288,3cyclictest15562-21pmu-power19:30:252
477099287,3cyclictest7321-21ntpq20:25:222
150432280,4sleep3477599cyclictest23:10:183
477599271,15cyclictest759-21dbus-daemon22:25:323
477099279,3cyclictest15017-21sed20:45:022
477099277,3cyclictest12358-21cut23:05:162
475799271,13cyclictest17699-21munin-run22:05:000
477599269,4cyclictest13315-21ntp_states19:25:233
477599269,3cyclictest0-21swapper/319:49:293
4775992610,4cyclictest10860-21irqstats19:20:193
477599260,12cyclictest20120-21/usr/sbin/munin22:10:173
477099269,3cyclictest18206-21sed19:35:282
477099267,3cyclictest18686-21cpuspeed_turbos23:20:142
477099267,3cyclictest18686-21cpuspeed_turbos23:20:142
475799261,13cyclictest14859-21munin-run20:45:010
105722260,4sleep121-21rcuc/121:45:261
477599257,8cyclictest3785-21swap21:30:253
477599256,3cyclictest11728-21sh23:05:003
477599254,10cyclictest25852-21diskmemload22:45:163
477599252,14cyclictest86550irq/126-enp1s0-22:03:223
4775992511,3cyclictest11677-21smart_sda00:15:243
4763992519,3cyclictest16526-21cat20:45:271
475799256,16cyclictest15487-21munin-run22:00:000
477599246,3cyclictest0-21swapper/321:52:083
477599242,15cyclictest7052-21kworker/u8:122:53:013
477599242,13cyclictest8221-21/usr/sbin/munin00:10:163
477599242,11cyclictest86850irq/129-enp1s0-21:20:273
477599240,18cyclictest0-21swapper/323:55:213
477599240,13cyclictest17854-21kernelversion00:30:193
476399243,9cyclictest0-21swapper/123:06:551
476399241,7cyclictest0-21swapper/123:38:391
476399241,6cyclictest0-21swapper/123:50:291
476399241,6cyclictest0-21swapper/123:16:461
476399241,17cyclictest0-21swapper/123:34:461
475799241,11cyclictest31831-21perf20:10:000
477599237,2cyclictest0-21swapper/321:15:153
477599235,9cyclictest21672-21/usr/sbin/munin21:00:143
477599234,9cyclictest27539-21sed21:10:283
477599234,9cyclictest27539-21sed21:10:283
477599234,3cyclictest0-21swapper/320:59:493
477599234,13cyclictest88050irq/133-enp2s0-19:50:273
477599234,11cyclictest32681-21/usr/sbin/munin21:25:253
4775992318,3cyclictest6063-21/usr/sbin/munin20:25:023
4775992313,3cyclictest0-21swapper/323:54:263
477599231,12cyclictest0-21swapper/319:10:123
476399232,12cyclictest0-21swapper/123:55:151
4763992317,3cyclictest7836-21expr22:55:151
476399231,17cyclictest0-21swapper/121:53:551
476399231,17cyclictest0-21swapper/100:32:171
476399231,16cyclictest0-21swapper/123:22:391
476399231,16cyclictest0-21swapper/123:22:391
476399231,13cyclictest0-21swapper/121:27:221
476399231,12cyclictest0-21swapper/121:24:251
476399231,11cyclictest0-21swapper/121:32:401
475799232,10cyclictest0-21swapper/000:25:000
475799231,5cyclictest0-21swapper/000:15:270
475799231,4cyclictest0-21swapper/022:41:230
475799231,11cyclictest11106-21perf21:50:010
167712231,2sleep016796-21basename00:30:010
167712231,2sleep016796-21basename00:30:010
477599225,3cyclictest0-21swapper/320:54:493
477599225,12cyclictest88150irq/134-enp2s0-23:05:183
477599224,9cyclictest0-21swapper/300:00:433
477599224,7cyclictest0-21swapper/320:35:313
477599224,11cyclictest86750irq/128-enp1s0-21:42:223
477599223,3cyclictest0-21swapper/323:41:423
477599223,13cyclictest86550irq/126-enp1s0-00:20:183
477599222,15cyclictest86550irq/126-enp1s0-00:28:483
477599222,15cyclictest86550irq/126-enp1s0-00:28:483
477599222,11cyclictest86850irq/129-enp1s0-19:31:283
477599221,17cyclictest86550irq/126-enp1s0-22:35:223
477599221,14cyclictest0-21swapper/322:55:263
4775992210,3cyclictest0-21swapper/323:25:263
477599220,8cyclictest29834-21/usr/sbin/munin22:30:193
477599220,11cyclictest19272-21cpuspeed_turbos00:35:133
476399228,9cyclictest0-21swapper/123:01:191
476399227,4cyclictest0-21swapper/100:11:081
476399222,11cyclictest6634-21diskstats20:25:151
4763992215,2cyclictest0-21swapper/122:34:441
4763992215,2cyclictest0-21swapper/119:11:091
4763992213,3cyclictest0-21swapper/100:20:261
476399221,16cyclictest0-21swapper/121:01:541
476399221,16cyclictest0-21swapper/119:42:531
476399221,15cyclictest0-21swapper/120:54:071
476399221,10cyclictest0-21swapper/121:41:151
476399221,10cyclictest0-21swapper/100:15:191
476399220,16cyclictest0-21swapper/120:35:231
475799221,5cyclictest0-21swapper/023:40:070
475799221,5cyclictest0-21swapper/022:21:040
475799221,11cyclictest86750irq/128-enp1s0-21:20:010
477599218,3cyclictest0-21swapper/322:41:243
477599217,8cyclictest88150irq/134-enp2s0-20:45:473
477599214,9cyclictest27002-21kworker/u8:223:39:343
477599214,3cyclictest0-21swapper/320:14:213
477599213,4cyclictest0-21swapper/321:45:133
477599212,10cyclictest18556-21/usr/sbin/munin23:20:123
477599212,10cyclictest18556-21/usr/sbin/munin23:20:123
4775992112,6cyclictest9561-21ntpq20:30:253
477599211,12cyclictest22946-21/usr/sbin/munin23:30:123
477599210,8cyclictest86550irq/126-enp1s0-21:59:443
476399218,3cyclictest31824-21munin-run20:10:001
476399217,3cyclictest0-21swapper/122:15:211
476399217,3cyclictest0-21swapper/122:15:201
476399217,2cyclictest25852-21diskmemload22:05:151
476399212,2cyclictest26899-21irqstats21:10:201
476399212,2cyclictest26899-21irqstats21:10:201
476399212,10cyclictest32375-21smart_sda22:35:241
4763992117,3cyclictest0-21swapper/122:43:441
4763992114,2cyclictest0-21swapper/119:56:071
476399211,17cyclictest21184-21perf00:40:021
476399211,15cyclictest0-21swapper/122:22:351
476399211,15cyclictest0-21swapper/122:01:581
476399211,15cyclictest0-21swapper/121:58:461
476399211,15cyclictest0-21swapper/121:37:021
476399211,15cyclictest0-21swapper/121:05:321
476399211,15cyclictest0-21swapper/120:33:591
476399211,15cyclictest0-21swapper/120:20:111
476399211,15cyclictest0-21swapper/120:15:371
476399211,15cyclictest0-21swapper/119:46:041
476399211,15cyclictest0-21swapper/119:21:211
476399211,11cyclictest23767-21fschecks_time19:50:161
476399211,11cyclictest1649-21latency_hist20:15:011
476399211,11cyclictest0-21swapper/123:10:421
476399211,11cyclictest0-21swapper/122:26:351
476399211,10cyclictest0-21swapper/122:11:471
476399210,19cyclictest0-21swapper/119:31:501
476399210,15cyclictest0-21swapper/123:45:151
476399210,15cyclictest0-21swapper/100:02:151
475799212,9cyclictest7065-21munin-node20:10:200
475799211,7cyclictest0-21swapper/022:49:050
475799211,4cyclictest0-21swapper/000:10:230
477599209,8cyclictest8316-21cut19:15:173
477599203,3cyclictest0-21swapper/320:25:243
477599200,9cyclictest23915-21vmstat22:15:293
477599200,9cyclictest23915-21vmstat22:15:293
476399206,4cyclictest0-21swapper/122:54:021
4763992017,2cyclictest0-21swapper/119:35:361
476399201,14cyclictest0-21swapper/123:26:021
476399201,14cyclictest0-21swapper/121:16:421
4763992011,2cyclictest4765-40dnf22:45:321
4763992011,2cyclictest25852-21diskmemload23:40:331
476399201,11cyclictest20736-21smart_sda20:55:261
476399201,11cyclictest0-21swapper/119:25:191
476399201,10cyclictest29783-21cat20:05:011
476399200,15cyclictest0-21swapper/100:29:291
476399200,15cyclictest0-21swapper/100:29:291
475799205,14cyclictest0-21swapper/023:05:000
475799202,4cyclictest14219-21munin-run19:30:010
475799201,5cyclictest0-21swapper/000:07:010
475799201,4cyclictest0-21swapper/023:30:250
475799201,4cyclictest0-21swapper/023:28:250
475799201,11cyclictest18556-21/usr/sbin/munin23:20:160
475799201,11cyclictest18556-21/usr/sbin/munin23:20:160
4775991912,4cyclictest17353-21basename19:35:193
477599191,15cyclictest16454-21cpuspeed_turbos23:15:133
477599190,5cyclictest25437-21/usr/sbin/munin19:55:133
476399191,10cyclictest8008-21cut19:15:151
475799199,3cyclictest86750irq/128-enp1s0-22:32:280
475799198,3cyclictest21672-21/usr/sbin/munin21:00:250
475799192,5cyclictest27529-21irqstats22:25:170
475799191,9cyclictest2259-21hddtemp_smartct23:55:160
4757991912,4cyclictest848-21sssd_nss20:50:010
475799191,10cyclictest8488-21/usr/sbin/munin20:30:170
475799191,10cyclictest10050-21/usr/sbin/munin19:20:250
4775991811,5cyclictest836-21runrttasks21:09:003
4775991811,5cyclictest27984-21df_inode20:00:183
477599180,9cyclictest86550irq/126-enp1s0-20:15:233
475799189,2cyclictest13735-1kworker/u9:021:38:150
475799187,3cyclictest30487-21/usr/sbin/munin21:20:280
475799185,1cyclictest0-21swapper/019:15:180
475799183,12cyclictest2503-21grep20:15:160
475799182,7cyclictest6169-21irqstats22:50:190
475799182,6cyclictest13735-1kworker/u9:022:12:240
475799181,8cyclictest0-21swapper/021:32:430
475799181,7cyclictest0-21swapper/023:54:540
475799181,7cyclictest0-21swapper/021:28:230
475799181,6cyclictest13735-1kworker/u9:000:02:210
475799181,4cyclictest14634-21snmpd21:50:130
475799181,4cyclictest0-21swapper/023:16:430
475799181,3cyclictest30464-21memory23:45:200
475799181,3cyclictest20667-21processes20:55:250
4757991813,3cyclictest10681-21/usr/sbin/munin20:35:170
475799181,11cyclictest0-21swapper/023:10:230
475799174,8cyclictest0-21swapper/019:35:170
475799173,7cyclictest86550irq/126-enp1s0-23:38:540
475799173,4cyclictest0-21swapper/000:36:330
475799172,12cyclictest6928-21latency_hist19:15:010
475799171,9cyclictest0-21swapper/022:05:020
475799171,6cyclictest26040-21tail19:55:160
475799171,2cyclictest86550irq/126-enp1s0-19:30:200
4757991711,4cyclictest0-21swapper/019:45:240
279932171,2sleep00-21swapper/021:14:230
279932171,2sleep00-21swapper/021:14:230
136122171,2sleep00-21swapper/023:06:070
475799161,4cyclictest0-21swapper/020:54:350
475799161,14cyclictest0-21swapper/020:28:030
475799161,12cyclictest18843-21/usr/sbin/munin19:40:150
4757991610,4cyclictest24631-21cat19:50:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional