You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-12 - 07:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jan 12, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
124582236163,24sleep20-21swapper/219:08:272
123692202164,25sleep10-21swapper/119:07:181
123762201164,24sleep30-21swapper/319:07:253
123682200163,25sleep00-21swapper/019:07:170
165821500,3sleep01266899cyclictest23:35:200
2797421180,4sleep227971-21find19:40:212
183052940,5sleep21267999cyclictest19:20:162
70612510,4sleep01266899cyclictest23:50:010
172272430,3sleep230-21rcuc/220:30:182
12668994010,14cyclictest24638-21basename00:30:020
1266899367,13cyclictest7354-21munin-run20:10:020
1266899338,11cyclictest0-21swapper/019:55:010
1266899335,13cyclictest15847-21awk00:10:010
1266899334,13cyclictest22752-21idleruntime-cro20:45:010
12679993220,2cyclictest14890-21kworker/2:021:40:172
277782290,4sleep339-21rcuc/319:40:193
1267599271,6cyclictest0-21swapper/100:16:011
1266899279,15cyclictest16162-21idleruntime-cro20:30:020
1267999269,3cyclictest14890-21kworker/2:020:45:012
12679992612,2cyclictest14890-21kworker/2:022:45:132
12679992612,2cyclictest14890-21kworker/2:020:15:132
1267599261,9cyclictest0-21swapper/119:45:251
1266899260,13cyclictest0-21swapper/023:44:550
1266899260,12cyclictest19604-21perf23:05:010
1268499251,7cyclictest0-21swapper/322:05:243
1266899256,16cyclictest28720-21munin-run19:45:000
1266899251,10cyclictest0-21swapper/022:51:410
1267599249,12cyclictest8211-21needreboot23:50:181
1267599241,9cyclictest0-21swapper/122:30:201
1266899241,10cyclictest0-21swapper/023:09:410
1266899241,10cyclictest0-21swapper/000:23:060
1268499239,3cyclictest87150irq/126-enp1s0-00:22:553
1268499237,9cyclictest0-21swapper/322:35:203
1268499231,5cyclictest27832-21if_err_enp2s000:35:173
12679992313,2cyclictest32160-21awk22:20:002
1267599231,19cyclictest5157-21munin-run20:05:001
1266899236,15cyclictest23358-21munin-run22:00:020
1266899231,12cyclictest12996-21perf22:50:010
1266899230,11cyclictest0-21swapper/023:28:330
1268499221,7cyclictest0-21swapper/322:52:213
1268499221,5cyclictest0-21swapper/323:00:193
1268499221,3cyclictest21886-21fschecks_time21:55:133
1268499221,12cyclictest8194-21/usr/sbin/munin21:25:123
1268499221,11cyclictest0-21swapper/322:15:263
1267599223,10cyclictest0-21swapper/100:03:471
1267599221,3cyclictest13270-21hddtemp_smartct21:35:161
1267599221,18cyclictest24394-21irqstats22:00:181
1267599221,11cyclictest29695-21cpuspeed_turbos21:00:111
1267599220,5cyclictest5666-21kworker/u8:121:45:271
1266899221,9cyclictest0-21swapper/019:47:590
12668992216,4cyclictest22128-21awk19:30:010
1266899220,8cyclictest0-21swapper/021:35:310
1268499213,9cyclictest12444-21kworker/3:021:45:233
1268499211,8cyclictest16569-21irqstats19:15:183
1268499211,6cyclictest0-21swapper/323:25:403
1268499211,6cyclictest0-21swapper/321:52:433
1268499211,6cyclictest0-21swapper/321:32:433
12684992111,2cyclictest312-21awk23:35:003
1268499211,10cyclictest0-21swapper/319:26:143
1267999219,2cyclictest14890-21kworker/2:023:32:382
1267999214,4cyclictest14890-21kworker/2:023:27:202
1267999212,10cyclictest14890-21kworker/2:022:00:012
1267999211,6cyclictest0-21swapper/221:13:002
12679992116,3cyclictest22243-40dnf00:22:522
1267999211,4cyclictest0-21swapper/221:47:472
1267599219,3cyclictest28913-21cat23:25:131
1267599211,18cyclictest0-21swapper/123:01:231
1267599211,17cyclictest0-21swapper/123:30:171
12675992111,3cyclictest29246-21smart_sda22:10:251
1267599211,12cyclictest30398-21cstates22:15:131
1267599211,11cyclictest8730-21cat22:40:021
1267599211,11cyclictest4864-21ntp_states21:15:221
1267599211,11cyclictest4864-21ntp_states21:15:221
1267599211,11cyclictest22969-21users21:55:251
1267599211,11cyclictest22807-21grep00:25:121
1267599211,11cyclictest19859-21/usr/sbin/munin21:50:131
1267599210,15cyclictest87450irq/129-enp1s0-22:50:561
1266899216,6cyclictest0-21swapper/022:30:150
1266899215,2cyclictest87150irq/126-enp1s0-23:13:120
1266899213,15cyclictest17397-21idleruntime-cro23:00:010
1266899212,16cyclictest11404-21munin-run00:00:010
1266899211,4cyclictest4191-21munin-run22:30:020
1266899211,4cyclictest4191-21munin-run22:30:020
12668992114,4cyclictest26134-21/usr/sbin/munin22:05:140
1266899211,10cyclictest12379-21perf21:35:010
1268499209,3cyclictest14791-21/usr/sbin/munin21:40:143
1268499208,3cyclictest4204-21awk22:30:013
1268499208,3cyclictest4204-21awk22:30:013
1268499201,6cyclictest0-21swapper/322:59:043
1268499201,4cyclictest0-21swapper/323:21:523
1268499201,11cyclictest24591-21NetworkManager20:45:463
1268499201,11cyclictest21058-21cstates20:40:143
1268499201,11cyclictest17900-21sshd00:13:323
1268499201,11cyclictest0-21swapper/320:50:113
1268499201,10cyclictest31944-21irqstats19:50:183
1267999206,3cyclictest2356-21kworker/2:223:45:542
1267999205,8cyclictest87450irq/129-enp1s0-19:30:182
1267999205,5cyclictest725-21cat22:20:172
1267999202,6cyclictest0-21swapper/221:21:552
1267999201,6cyclictest0-21swapper/221:29:242
1267999201,6cyclictest0-21swapper/200:25:242
1267999201,4cyclictest0-21swapper/223:11:592
1267999201,4cyclictest0-21swapper/223:07:382
1267999201,11cyclictest14890-21kworker/2:019:15:182
1267599209,3cyclictest23704-21cat20:45:161
1267599209,3cyclictest1989-21latency_hist22:25:021
1267599209,3cyclictest15751-21df_inode22:55:121
1267599208,5cyclictest87250irq/127-enp1s0-23:12:101
1267599204,5cyclictest28112-21sshd23:21:461
12675992017,2cyclictest0-21swapper/122:48:261
1267599201,18cyclictest0-21swapper/121:20:401
1267599201,17cyclictest0-21swapper/120:40:191
1267599201,17cyclictest0-21swapper/120:37:031
1267599201,17cyclictest0-21swapper/119:10:221
1267599201,14cyclictest17389-21taskset19:15:481
1267599201,11cyclictest5880-21kernelversion23:45:181
1267599201,11cyclictest5008-21cut23:45:011
1267599201,11cyclictest29320-21latency_hist21:00:011
1267599201,11cyclictest28709-21latency_hist19:45:001
1267599201,11cyclictest21413-21smart_sda19:25:241
1267599201,10cyclictest3706-21cat21:15:011
1267599201,10cyclictest1450-21kernelversion23:35:181
1267599201,10cyclictest11396-21latency_hist00:00:011
1266899205,6cyclictest0-21swapper/022:15:030
1266899202,7cyclictest0-21swapper/000:16:250
1266899201,6cyclictest0-21swapper/022:41:550
1266899201,11cyclictest4799-21sshd20:00:460
1266899201,11cyclictest29550-21/usr/sbin/munin21:00:150
1266899201,10cyclictest0-21swapper/000:01:180
1266899200,9cyclictest0-21swapper/021:10:360
1266899200,11cyclictest0-21swapper/023:16:490
1266899200,11cyclictest0-21swapper/021:23:260
1266899200,10cyclictest0-21swapper/021:40:120
101682200,2sleep20-21swapper/223:55:172
246692191,2sleep00-21swapper/020:46:460
239842191,3sleep00-21swapper/019:30:490
1268499198,3cyclictest8810-21/usr/sbin/munin22:40:113
1268499191,9cyclictest25838-21strings00:30:203
1268499191,7cyclictest5764-21sh21:20:013
1268499191,7cyclictest5764-21sh21:20:013
1268499191,6cyclictest0-21swapper/300:10:023
1268499191,15cyclictest32093-21df21:05:123
1268499191,10cyclictest5038-21/usr/sbin/munin23:45:213
12684991910,2cyclictest19854-21vmstat00:15:273
1267999193,7cyclictest20153-21/usr/sbin/munin19:25:212
1267999193,2cyclictest14890-21kworker/2:020:00:132
1267999191,7cyclictest0-21swapper/200:31:122
1267999191,6cyclictest0-21swapper/222:50:382
1267999191,5cyclictest0-21swapper/223:15:502
1267999191,5cyclictest0-21swapper/222:56:312
1267999191,4cyclictest0-21swapper/222:44:502
12679991912,4cyclictest12680-21cpuspeed_turbos21:35:122
1267599191,8cyclictest0-21swapper/123:17:191
1267599191,8cyclictest0-21swapper/122:09:051
12675991915,3cyclictest0-21swapper/120:22:101
12675991915,3cyclictest0-21swapper/120:11:091
12675991915,2cyclictest17723-21chrt00:11:131
12675991913,4cyclictest14837-21strings00:05:191
1267599191,17cyclictest0-21swapper/121:41:271
1267599191,17cyclictest0-21swapper/120:54:211
1267599191,17cyclictest0-21swapper/120:28:021
1267599191,17cyclictest0-21swapper/120:05:021
1267599191,17cyclictest0-21swapper/100:22:261
1267599191,16cyclictest0-21swapper/119:35:031
1267599191,10cyclictest9785-21vmstat21:25:261
1267599191,10cyclictest25801-21ls00:30:201
1266899197,4cyclictest0-21swapper/023:32:310
1266899196,4cyclictest0-21swapper/022:04:050
12668991914,3cyclictest1103-21ls22:20:230
1266899191,10cyclictest21124-21sshd21:54:470
1266899190,10cyclictest0-21swapper/000:11:440
120742191,2sleep10-21swapper/121:31:301
1268499187,3cyclictest17883-21sed20:30:253
1268499184,12cyclictest23967-21fschecks_count22:00:153
1268499181,9cyclictest814-21snmpd00:01:123
1268499181,9cyclictest0-21swapper/323:11:453
1268499181,8cyclictest0-21swapper/323:06:013
1268499181,7cyclictest18339-21fschecks_count19:20:163
1268499181,6cyclictest29524-21unixbench_singl22:10:263
12684991814,3cyclictest0-21swapper/320:27:113
12684991812,3cyclictest7933-21entropy20:10:163
1268499181,15cyclictest0-21swapper/322:23:173
1268499181,14cyclictest401ktimersoftd/320:05:263
12684991811,4cyclictest3715-21/usr/sbin/munin23:40:193
1268499181,12cyclictest0-21swapper/320:01:263
1268499181,11cyclictest0-21swapper/322:45:193
1268499181,11cyclictest0-21swapper/321:10:503
1268499181,10cyclictest20079-21tail20:35:273
1267999187,2cyclictest14890-21kworker/2:021:32:252
1267999186,3cyclictest12050irq/123-ahci[0020:00:002
1267999186,2cyclictest14890-21kworker/2:021:53:372
1267999184,2cyclictest87350irq/128-enp1s0-19:51:092
1267999183,3cyclictest14890-21kworker/2:020:35:212
1267999181,6cyclictest87450irq/129-enp1s0-19:10:092
1267999181,6cyclictest24938-21munin-run20:50:022
1267999181,6cyclictest0-21swapper/222:13:262
1267999181,6cyclictest0-21swapper/221:16:142
1267999181,6cyclictest0-21swapper/221:16:142
1267999181,3cyclictest6460-21latency_hist22:35:012
12679991812,3cyclictest980-21vmstat21:05:272
12679991812,3cyclictest27170-21cut20:55:012
12679991811,4cyclictest27708-21dnf20:55:162
1267999181,10cyclictest0-21swapper/200:17:412
12679991810,5cyclictest885-21sssd_nss22:00:242
1267599187,3cyclictest18302-21diskstats19:20:151
12675991814,3cyclictest0-21swapper/123:05:211
1267599181,16cyclictest0-21swapper/122:42:391
1267599181,16cyclictest0-21swapper/122:25:121
1267599181,16cyclictest0-21swapper/122:25:121
1267599181,16cyclictest0-21swapper/121:07:591
1267599181,16cyclictest0-21swapper/120:32:021
1267599181,16cyclictest0-21swapper/120:15:461
1267599181,16cyclictest0-21swapper/119:56:311
1267599181,10cyclictest32565-21timerwakeupswit19:50:251
1267599180,3cyclictest766-21sssd_be19:30:531
1266899187,4cyclictest15759-21/usr/sbin/munin19:15:220
1266899186,3cyclictest0-21swapper/023:53:030
1266899185,8cyclictest0-21swapper/020:15:230
1266899183,3cyclictest0-21swapper/000:36:560
1266899181,2cyclictest0-21swapper/021:16:320
1266899181,2cyclictest0-21swapper/021:16:320
1266899181,11cyclictest0-21swapper/021:27:250
1266899181,10cyclictest0-21swapper/000:32:120
1266899180,8cyclictest0-21swapper/023:21:390
72352171,2sleep20-21swapper/220:08:442
271082171,2sleep00-21swapper/020:54:530
177962170,3sleep20-21swapper/200:12:112
1268499175,4cyclictest87150irq/126-enp1s0-20:20:113
1268499171,9cyclictest0-21swapper/319:10:043
1268499171,7cyclictest29827-21latency19:45:183
1268499171,6cyclictest13266-21hddtemp_smartct21:35:173
1268499171,6cyclictest10986-21unixbench_singl23:55:273
1268499171,6cyclictest0-21swapper/323:50:153
12684991713,3cyclictest0-21swapper/321:20:453
12684991712,3cyclictest24619-21dump-pmu-power00:30:023
1268499171,13cyclictest27437-21cpuspeed_turbos20:55:133
12684991711,3cyclictest22627-21ls19:30:133
1268499171,10cyclictest0-21swapper/319:57:113
1268499170,14cyclictest5409-21cut22:30:203
1267999173,6cyclictest14890-21kworker/2:023:35:142
1267999172,2cyclictest87450irq/129-enp1s0-23:41:212
1267999171,7cyclictest0-21swapper/200:35:142
1267999171,5cyclictest0-21swapper/222:25:162
1267999171,5cyclictest0-21swapper/222:25:162
12679991713,3cyclictest0-21swapper/223:51:562
1267999171,2cyclictest87250irq/127-enp1s0-00:03:452
12679991712,3cyclictest0-21swapper/220:10:052
1267999171,14cyclictest0-21swapper/223:20:442
1267999171,14cyclictest0-21swapper/221:01:522
12679991711,3cyclictest26622-21cut22:05:172
1267599171,8cyclictest0-21swapper/100:35:331
1266899175,3cyclictest0-21swapper/022:35:590
1266899173,5cyclictest16991-21/usr/sbin/munin21:45:200
1266899171,7cyclictest768-21rngd21:05:250
1266899171,5cyclictest0-21swapper/020:35:210
12668991711,3cyclictest8163-21grep20:10:180
1266899171,13cyclictest17954-21/usr/sbin/munin19:20:120
1266899171,13cyclictest11964-21/usr/sbin/munin20:20:110
1266899171,10cyclictest0-21swapper/022:11:330
12668991710,4cyclictest2957-21awk20:00:010
1268499161,7cyclictest539-21/usr/sbin/munin23:35:133
1268499161,14cyclictest0-21swapper/319:35:273
12684991611,3cyclictest9540-21latency_hist20:15:013
1268499161,13cyclictest401ktimersoftd/321:00:253
1268499161,12cyclictest24596-21fschecks_count23:15:143
1267999163,3cyclictest87450irq/129-enp1s0-22:35:212
1267999162,3cyclictest14890-21kworker/2:020:25:132
1267999161,5cyclictest24546-21/usr/sbin/munin19:35:182
12679991612,2cyclictest14912-21ntp_states00:05:212
12679991611,3cyclictest29511-21cut19:45:142
12679991610,3cyclictest17951-21diskstats23:00:142
12679991610,3cyclictest12296-21expr20:20:112
1266899164,3cyclictest28982-21sshd20:55:450
1266899160,14cyclictest0-21swapper/019:35:100
1266899160,13cyclictest0-21swapper/020:30:380
12668991511,3cyclictest0-21swapper/019:10:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional