You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 18:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Dec 08, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
182542241163,26sleep00-21swapper/007:06:140
184682238162,24sleep10-21swapper/107:09:001
184692237163,25sleep20-21swapper/207:09:002
183502236162,23sleep30-21swapper/307:07:293
1340421610,4sleep31865399cyclictest10:35:193
138821420,5sleep31865399cyclictest07:40:213
2710121050,1sleep30-21swapper/308:40:143
293012600,1sleep10-21swapper/107:30:191
65312550,2sleep30-21swapper/309:05:233
207642530,4sleep21864999cyclictest09:40:022
223772420,4sleep222383-21gltestperf07:15:152
18637993916,12cyclictest20984-21/usr/sbin/munin12:05:270
18637993713,16cyclictest30649-21ntp_states10:00:220
18637993713,16cyclictest30649-21ntp_states10:00:220
1863799359,15cyclictest17659-21/usr/sbin/munin12:00:250
18637993410,11cyclictest8882-21sed10:25:170
1863799334,14cyclictest4239-21munin-run11:30:010
1863799334,14cyclictest28456-21perf12:25:010
1863799332,15cyclictest22134-21munin-run08:30:020
18637993314,6cyclictest32130-21smart_sda12:30:250
18637993311,6cyclictest86750irq/128-enp1s0-11:15:170
18637993311,13cyclictest22071-21/usr/sbin/munin12:10:260
18637993310,13cyclictest22796-21ntp_states07:15:200
1863799329,13cyclictest966-21/usr/sbin/munin08:55:150
1863799329,10cyclictest4958-21grep11:30:160
1863799328,16cyclictest7598-21/usr/sbin/munin09:10:260
1863799328,10cyclictest848-21sssd_nss09:25:160
1863799327,13cyclictest23932-21/usr/sbin/munin07:20:190
1863799326,17cyclictest0-21swapper/009:40:170
18637993219,4cyclictest6036-21/usr/sbin/munin10:20:110
18637993213,10cyclictest24578-21users09:45:280
18637993212,11cyclictest6955-21/usr/sbin/munin07:55:270
18637993210,8cyclictest581-21systemd-journal10:35:020
1863799318,12cyclictest0-21swapper/009:05:250
1863799313,11cyclictest30847-21kworker/0:208:50:370
18637993122,5cyclictest1965-21smart_sda12:35:250
18637993112,7cyclictest2332-21awk07:45:010
18637993111,16cyclictest24661-21tune2fs12:15:160
18637993111,12cyclictest17187-21cat09:30:170
18637993110,18cyclictest24325-21latency_hist08:35:020
18637993110,12cyclictest26132-21/usr/sbin/munin07:25:250
1863799309,9cyclictest0-21swapper/010:15:300
1863799308,9cyclictest3101-21grep10:10:260
1863799308,19cyclictest4758-21/usr/sbin/munin07:50:270
1863799303,7cyclictest20826-21idleruntime-cro07:15:010
18637993014,9cyclictest15159-21sshd11:54:130
18637993013,7cyclictest1122-21taskset10:06:490
18637993010,8cyclictest15758-21/usr/sbin/munin08:15:140
18637993010,12cyclictest12014-21egrep09:20:120
18637993010,10cyclictest86750irq/128-enp1s0-11:20:130
1864999291,10cyclictest24966-21ntp_kernel_err07:20:202
1863799299,12cyclictest30334-21chrt12:26:150
1863799297,13cyclictest7363-21diskmemload10:50:250
1863799295,12cyclictest3192-21/usr/sbin/munin09:00:220
18637992920,5cyclictest10412-21smart_sda08:00:250
18637992917,9cyclictest15179-21diskstats10:40:160
18637992915,11cyclictest13463-21cat08:10:020
18637992913,10cyclictest10596-21interrupts09:15:200
18637992911,4cyclictest4149-21users07:45:270
18637992910,7cyclictest28744-21idleruntime11:10:170
1863799289,16cyclictest24556-21/usr/sbin/munin08:35:190
1863799286,9cyclictest19668-21sh09:35:220
18637992821,4cyclictest19090-21/usr/sbin/munin08:20:230
18637992817,8cyclictest7004-21df11:35:150
18637992815,6cyclictest14634-21snmpd11:58:460
18637992813,7cyclictest12979-21diskstats10:35:160
197222270,3sleep341-21ksoftirqd/310:50:163
193122270,5sleep3581-21systemd-journal12:01:013
1864999271,11cyclictest0-21swapper/212:30:182
1863799279,15cyclictest19014-21idleruntime-cro10:50:010
1863799278,10cyclictest86550irq/126-enp1s0-11:45:180
18637992720,4cyclictest9219-21diskstats11:40:170
18637992718,6cyclictest30524-21/usr/sbin/munin07:35:150
18637992713,6cyclictest28416-21/usr/sbin/munin09:55:210
18637992710,5cyclictest86850irq/129-enp1s0-10:55:140
1863799267,11cyclictest86750irq/128-enp1s0-08:40:380
1863799267,11cyclictest86750irq/128-enp1s0-07:30:140
18637992619,4cyclictest14369-21irqstats08:10:160
18637992618,5cyclictest24685-21/usr/sbin/munin11:00:240
18637992617,6cyclictest25896-21/usr/sbin/munin09:50:170
18637992615,5cyclictest86750irq/128-enp1s0-11:05:260
18637992610,5cyclictest28955-21/usr/sbin/munin08:45:150
228322250,4sleep339-21rcuc/309:45:013
1865399250,9cyclictest0-21swapper/311:12:123
1864299250,17cyclictest0-21swapper/110:57:111
1864299250,16cyclictest0-21swapper/108:24:091
1865399241,10cyclictest11348-21memory10:30:213
1864999241,11cyclictest20552-21hddtemp_smartct12:05:202
1864299243,3cyclictest14907-21sshd11:50:571
18642992414,3cyclictest27851-21timerandwakeup12:20:271
1864299241,15cyclictest30254-21users12:25:261
1865399231,13cyclictest0-21swapper/310:04:383
1865399231,13cyclictest0-21swapper/310:04:383
1864299231,4cyclictest24549-21missed_timers11:00:221
1864299231,19cyclictest30023-21ntp_states08:45:241
18642992310,4cyclictest26241-21munin-run12:20:001
1864299230,18cyclictest0-21swapper/109:34:561
1864299230,17cyclictest14383-21kworker/u8:109:28:371
1864299230,17cyclictest0-21swapper/110:30:321
1865399224,10cyclictest0-21swapper/311:30:243
1865399223,4cyclictest0-21swapper/309:15:183
1865399222,7cyclictest30921-21munin-run08:50:013
1865399221,15cyclictest0-21swapper/311:48:473
1864999228,3cyclictest0-21swapper/210:31:122
1864999222,3cyclictest0-21swapper/211:45:202
1864299229,3cyclictest23-21ksoftirqd/107:20:231
1864299228,11cyclictest4461-21head10:15:191
1864299222,15cyclictest86750irq/128-enp1s0-11:48:371
1864299222,15cyclictest86550irq/126-enp1s0-10:25:081
1864299221,12cyclictest31212-21df_inode12:30:141
1864299221,11cyclictest0-21swapper/111:38:351
1865399219,3cyclictest20542-21grep12:05:193
1865399218,4cyclictest23628-21/usr/sbin/munin11:00:193
1865399217,4cyclictest22204-21date08:30:023
1865399215,6cyclictest0-21swapper/311:26:253
1865399211,14cyclictest0-21swapper/312:30:243
1865399211,11cyclictest29736-21ntp_states12:25:223
1864299218,4cyclictest8410-21irqstats09:10:181
1864299214,8cyclictest0-21swapper/111:14:361
18642992115,3cyclictest21134-21users07:15:021
1864299211,3cyclictest16236-21fschecks_time08:15:141
1864299211,2cyclictest31298-21ntpq11:15:201
1864299211,19cyclictest0-21swapper/107:13:451
1864299211,18cyclictest0-21swapper/111:57:141
1864299211,18cyclictest0-21swapper/109:55:251
1864299211,18cyclictest0-21swapper/108:27:141
1864299211,18cyclictest0-21swapper/107:35:221
1864299211,17cyclictest7363-21diskmemload12:01:021
1864299211,15cyclictest0-21swapper/112:38:531
1864299211,12cyclictest0-21swapper/110:25:011
1864299211,12cyclictest0-21swapper/109:48:301
1864299211,11cyclictest24332-21munin-run08:35:011
18642992110,3cyclictest9208-21df11:40:151
18642992110,3cyclictest20557-21latency_hist09:40:021
33412200,4sleep230-21rcuc/207:45:182
3122200,4sleep121-21rcuc/110:05:221
1865399209,3cyclictest151412uptime08:10:243
1865399207,4cyclictest0-21swapper/311:05:243
1865399205,2cyclictest86550irq/126-enp1s0-09:53:023
1865399204,6cyclictest0-21swapper/311:59:293
1865399201,5cyclictest32295-1kworker/u9:010:58:453
18653992015,3cyclictest15752-21grep09:25:283
1865399201,11cyclictest27403-21cpu09:55:133
1865399201,10cyclictest0-21swapper/310:45:463
1865399200,13cyclictest0-21swapper/311:36:123
1864999207,4cyclictest780-21rngd08:16:082
1864999205,9cyclictest5480-40dnf10:16:032
1864999205,4cyclictest0-21swapper/211:44:402
1864999205,3cyclictest0-21swapper/207:52:082
1864999204,3cyclictest0-21swapper/207:33:202
1864999203,9cyclictest0-21swapper/209:10:012
1864999201,7cyclictest0-21swapper/207:11:262
18649992015,3cyclictest2097-21ls08:55:242
1864999201,11cyclictest0-21swapper/210:20:242
1864999201,10cyclictest7922-21mailstats11:35:252
1864299203,2cyclictest86750irq/128-enp1s0-11:20:561
18642992016,3cyclictest0-21swapper/110:14:371
18642992014,3cyclictest7297-21df_inode07:55:161
18642992013,4cyclictest22006-21processes09:40:231
1864299201,18cyclictest0-21swapper/107:45:001
1864299201,17cyclictest0-21swapper/109:04:031
1864299201,17cyclictest0-21swapper/107:50:141
1864299201,15cyclictest0-21swapper/111:27:451
1864299201,14cyclictest0-21swapper/111:30:051
18642992011,3cyclictest0-21swapper/112:05:241
1864299201,12cyclictest0-21swapper/112:14:391
1864299201,10cyclictest0-21swapper/111:07:281
41772191,2sleep20-21swapper/211:29:222
25992191,2sleep10-21swapper/108:55:341
1865399199,2cyclictest25340-21ntp_states12:15:233
1865399196,5cyclictest0-21swapper/312:39:563
1865399196,4cyclictest20348-21grep07:10:253
1865399195,6cyclictest0-21swapper/311:20:323
1865399192,9cyclictest7363-21diskmemload11:50:273
1865399191,9cyclictest3733-21wc07:45:233
1865399191,9cyclictest22722-21needreboot07:15:183
1865399191,5cyclictest9939-21ntp_states11:40:243
1865399191,10cyclictest14634-21snmpd07:22:503
1865399191,10cyclictest0-21swapper/310:27:223
1864999196,3cyclictest0-21swapper/209:01:382
1864999195,5cyclictest28114-21awk07:30:022
1864999194,6cyclictest4472-21/usr/sbin/munin11:30:222
1864999194,3cyclictest0-21swapper/210:10:002
1864999193,3cyclictest0-21swapper/208:00:162
1864999192,14cyclictest29707-21ntpq12:25:202
1864999191,8cyclictest581-21systemd-journal08:35:022
1864999191,8cyclictest13958-21crond09:25:012
18649991914,3cyclictest31322-21/usr/sbin/munin07:35:182
1864999191,3cyclictest86550irq/126-enp1s0-08:55:012
1864999191,15cyclictest1280-21grep12:35:172
18649991910,6cyclictest22707-21head12:10:152
1864299192,3cyclictest25371-21irqstats08:35:171
1864299191,7cyclictest0-21swapper/109:16:341
18642991916,2cyclictest0-21swapper/108:42:181
1864299191,3cyclictest0-21swapper/110:39:351
1864299191,17cyclictest0-21swapper/110:01:431
1864299191,17cyclictest0-21swapper/110:01:431
1864299191,17cyclictest0-21swapper/109:51:091
1864299191,17cyclictest0-21swapper/108:07:161
1864299191,16cyclictest0-21swapper/110:45:211
1864299191,12cyclictest0-21swapper/110:40:021
1864299190,17cyclictest0-21swapper/107:45:121
29082181,2sleep30-21swapper/308:59:343
283222181,2sleep20-21swapper/209:55:212
259112180,1sleep20-21swapper/211:05:142
1865399187,4cyclictest0-21swapper/309:10:173
1865399186,4cyclictest0-21swapper/309:20:193
1865399185,3cyclictest11131-21munin-run08:05:013
1865399183,9cyclictest0-21swapper/312:14:053
1865399183,9cyclictest0-21swapper/309:39:253
1865399183,8cyclictest0-21swapper/310:10:273
1865399182,13cyclictest28328-21/usr/sbin/munin07:30:243
1865399181,5cyclictest0-21swapper/310:17:203
18653991812,3cyclictest7102-21ntpq10:20:223
1865399181,15cyclictest0-21swapper/307:35:143
1865399181,10cyclictest0-21swapper/310:40:283
1865399181,10cyclictest0-21swapper/309:49:593
1865399180,2cyclictest0-21swapper/311:16:323
1864999186,3cyclictest0-21swapper/212:24:342
1864999184,6cyclictest32-21ksoftirqd/210:45:222
1864999181,8cyclictest19167-21kworker/2:009:25:282
1864999181,8cyclictest0-21swapper/209:54:222
1864999181,6cyclictest0-21swapper/211:20:172
1864999181,6cyclictest0-21swapper/210:38:442
1864999181,3cyclictest19230-21/usr/sbin/munin10:50:132
1864999181,3cyclictest0-21swapper/210:03:382
1864999181,3cyclictest0-21swapper/210:03:382
18649991812,3cyclictest19928-21dump-pmu-power08:25:012
1864299182,13cyclictest31494-21df_inode08:50:151
18642991815,2cyclictest0-21swapper/109:05:061
18642991815,2cyclictest0-21swapper/108:11:381
1864299181,2cyclictest0-21swapper/110:51:371
1864299181,2cyclictest0-21swapper/107:25:521
1864299181,16cyclictest0-21swapper/109:20:151
1864299181,16cyclictest0-21swapper/108:02:591
1865399176,3cyclictest86550irq/126-enp1s0-08:05:263
1865399172,12cyclictest31797-21/usr/sbin/munin10:05:183
1865399171,6cyclictest22943-21perl08:30:183
1865399171,4cyclictest0-21swapper/309:31:153
1865399171,3cyclictest0-21swapper/307:50:033
18653991712,3cyclictest8096-21grep07:55:263
18653991712,3cyclictest1-21systemd08:50:103
1865399171,14cyclictest0-21swapper/308:35:543
1865399171,13cyclictest17957-21/usr/sbin/munin08:20:263
1865399171,10cyclictest0-21swapper/312:20:453
1864999178,7cyclictest8768-21fschecks_time10:25:172
1864999174,4cyclictest768-21in:imjournal08:25:102
1864999172,5cyclictest17424-21latency_hist12:00:012
1864999172,12cyclictest17782-21cut12:00:142
1864999171,7cyclictest1983-21df_inode10:10:152
1864999171,7cyclictest0-21swapper/209:31:062
1864999171,6cyclictest21709-21memory09:40:202
18649991712,3cyclictest31222-21needreboot11:15:192
18649991712,3cyclictest15909-21awk10:40:242
1864999171,15cyclictest0-21swapper/211:04:382
1864999171,14cyclictest0-21swapper/211:10:062
18649991711,3cyclictest1600-21grep07:40:242
1864999171,11cyclictest0-21swapper/209:47:412
1864999171,10cyclictest0-21swapper/209:10:092
1864999170,14cyclictest22936-21grep10:55:272
155162171,2sleep30-21swapper/308:15:023
130562171,2sleep20-21swapper/208:06:422
18653991611,3cyclictest3805-21gltestperf09:00:193
1865399161,13cyclictest28092-21latency_hist07:30:023
1864999165,3cyclictest86550irq/126-enp1s0-07:55:192
1864999164,3cyclictest0-21swapper/208:35:222
18649991613,2cyclictest0-21swapper/212:19:012
18649991611,3cyclictest28955-21/usr/sbin/munin08:45:172
18649991611,3cyclictest26757-21/usr/sbin/munin08:40:182
1864999161,13cyclictest13940-21hddtemp_smartct11:50:172
18649991611,3cyclictest13585-21cpu08:10:102
18649991611,3cyclictest11010-21cut09:15:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional