You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 14:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat May 02, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
277522235161,24sleep20-21swapper/207:09:002
276912235164,58sleep30-21swapper/307:08:123
277182234161,60sleep00-21swapper/007:08:340
277762203165,25sleep10-21swapper/107:09:201
2261321740,4sleep02791499cyclictest10:35:160
40442680,1sleep10-21swapper/112:20:161
321322590,2sleep2311ktimersoftd/212:10:182
156962550,2sleep20-21swapper/209:05:212
302342510,4sleep1221ktimersoftd/110:52:351
302342510,4sleep1221ktimersoftd/110:52:351
300532510,3sleep1221ktimersoftd/112:05:201
27932994210,2cyclictest25417-21awk09:30:013
2793299391,10cyclictest10875-21/usr/sbin/munin10:10:173
27914993710,13cyclictest5634-21munin-run08:45:010
2793299365,4cyclictest12976-21fschecks_time09:00:163
2793299364,4cyclictest23888-21wc11:50:243
2793299361,5cyclictest28333-21timerwakeupswit12:00:253
2793299347,4cyclictest5553-21/usr/sbin/munin09:55:233
27932993412,2cyclictest29432-21/usr/sbin/munin08:25:253
27932993411,2cyclictest22830-21/usr/sbin/munin08:10:183
2791499345,13cyclictest2834-21basename07:25:010
2793299339,3cyclictest24059-21/usr/sbin/munin10:40:233
2793299336,3cyclictest0-21swapper/311:04:423
27932993315,16cyclictest909-21sssd_nss11:45:153
2793299331,4cyclictest87350irq/126-enp1s0-07:33:573
2793299331,12cyclictest11836-21/usr/sbin/munin07:45:243
2791499333,14cyclictest7831-21munin-run08:50:020
2793299329,3cyclictest6472-21/usr/sbin/munin10:00:143
2793299329,3cyclictest21246-21/usr/sbin/munin09:20:163
2793299325,3cyclictest12050irq/123-ahci[0009:15:013
2793299322,4cyclictest5454-21sshd08:42:553
27932993212,17cyclictest20973-21df08:05:163
27932993211,3cyclictest16229-21/usr/sbin/munin07:55:113
2793299319,3cyclictest3692-21fschecks_count12:20:153
2793299315,5cyclictest0-21swapper/307:40:153
2793299311,6cyclictest27646-21timerandwakeup10:45:273
2793299311,5cyclictest11315-21ntp_states08:55:213
27932993112,16cyclictest29118-21cpu12:05:133
2793299311,11cyclictest10326-21chrt10:05:583
27932993110,4cyclictest0-21swapper/312:25:263
27932993110,3cyclictest827-21runrttasks08:22:293
27932993110,3cyclictest827-21runrttasks08:22:293
27932993110,3cyclictest17136-21sendmail_mailtr11:35:223
100042310,2sleep10-21swapper/111:20:171
2793299309,3cyclictest827-21runrttasks11:30:463
2793299309,3cyclictest788-21/usr/sbin/munin07:20:233
2793299309,3cyclictest14975-21missed_timers07:50:183
2793299308,3cyclictest827-21runrttasks11:07:073
2793299308,3cyclictest1106-21/usr/sbin/munin12:15:243
2793299308,2cyclictest827-21runrttasks08:32:043
2793299307,3cyclictest0-21swapper/307:35:183
2793299302,6cyclictest30361-21irqstats07:15:023
2793299301,4cyclictest87350irq/126-enp1s0-08:45:123
2793299301,4cyclictest27843-21/usr/sbin/munin09:35:223
2793299301,4cyclictest27843-21/usr/sbin/munin09:35:213
2793299301,4cyclictest0-21swapper/310:30:253
27932993010,3cyclictest12536-21ntp_kernel_pll_11:25:203
2791499301,14cyclictest23215-21munin-run09:25:010
2793299299,3cyclictest30724-21smartctl09:40:153
2793299299,3cyclictest22713-21cat10:35:193
2793299299,3cyclictest0-21swapper/311:55:243
2793299298,3cyclictest827-21runrttasks08:04:383
2793299298,3cyclictest5877-21cat11:10:193
2793299297,3cyclictest19117-21awk11:40:203
2793299296,4cyclictest26005-21fschecks_count09:30:163
2793299293,6cyclictest18729-21smart_sda10:25:253
2793299293,3cyclictest87650irq/129-enp1s0-12:40:023
2793299291,4cyclictest9066-21latency_hist11:20:023
27932992911,3cyclictest32244-21uptime10:55:253
27932992910,3cyclictest2697-21sendmail_mailqu08:35:233
2793299290,14cyclictest0-21swapper/310:50:223
2793299290,14cyclictest0-21swapper/310:50:223
2793299290,14cyclictest0-21swapper/309:19:583
2793299289,3cyclictest827-21runrttasks09:45:433
2793299289,2cyclictest827-21runrttasks10:18:103
2793299281,5cyclictest808-21snmpd07:27:313
2793299281,4cyclictest0-21swapper/310:20:193
27932992811,3cyclictest7832-21expr12:30:133
2793299271,7cyclictest0-21swapper/312:10:083
2793299271,3cyclictest87350irq/126-enp1s0-09:55:013
2793299271,13cyclictest0-21swapper/311:23:453
2793299270,12cyclictest0-21swapper/307:10:463
2793299268,3cyclictest25092-21cpuspeed_turbos08:15:123
2793299261,13cyclictest0-21swapper/309:06:583
2793299260,14cyclictest0-21swapper/308:54:563
2792299261,8cyclictest17805-21df10:25:151
27922992613,11cyclictest20811-21kworker/u8:109:35:071
27922992613,11cyclictest20811-21kworker/u8:109:35:071
2792299260,20cyclictest0-21swapper/111:16:481
2791499267,2cyclictest6048-21chrt09:57:370
2791499266,11cyclictest391-21/usr/sbin/munin11:00:220
2792299253,7cyclictest30364-21kworker/u8:111:04:421
27922992514,6cyclictest23027-40dnf10:35:261
27922992511,6cyclictest3897-21kworker/u8:010:33:261
27922992511,12cyclictest20811-21kworker/u8:110:07:261
2792299250,19cyclictest0-21swapper/112:04:251
2791499255,18cyclictest0-21swapper/012:28:490
2791499254,12cyclictest24681-21/usr/sbin/munin11:55:150
2791499251,12cyclictest18208-21awk08:00:000
2792299243,5cyclictest23-21ksoftirqd/110:00:191
2792299241,10cyclictest0-21swapper/110:16:441
2792299240,3cyclictest0-21swapper/111:30:121
2791499248,4cyclictest0-21swapper/011:15:500
27914992412,3cyclictest26258-21/usr/sbin/munin10:45:180
2792299239,12cyclictest0-21swapper/107:55:131
2792299234,11cyclictest23-21ksoftirqd/110:10:141
2792299232,16cyclictest87650irq/129-enp1s0-12:39:111
27922992312,3cyclictest15915-21smart_sda09:05:241
2792299231,17cyclictest0-21swapper/109:31:171
27922992311,3cyclictest27071-21irqstats10:45:191
2791499235,7cyclictest0-21swapper/012:12:030
2791499235,4cyclictest14479-21/usr/sbin/munin11:30:170
27914992314,2cyclictest26634-21kworker/u8:212:04:580
2791499231,12cyclictest9914-21/usr/sbin/munin12:35:130
2792699221,4cyclictest30364-21kworker/u8:111:00:242
2792299229,6cyclictest7292-21kworker/u8:207:33:261
2792299222,3cyclictest23-21ksoftirqd/111:00:011
2792299221,3cyclictest0-21swapper/108:15:241
2792299220,6cyclictest0-21swapper/109:45:121
2792299220,4cyclictest0-21swapper/111:58:551
2791499227,7cyclictest11208-21df10:10:110
2791499227,6cyclictest0-21swapper/009:10:150
2791499225,5cyclictest0-21swapper/011:35:050
2791499224,5cyclictest5811-21load11:10:180
2791499224,3cyclictest87350irq/126-enp1s0-12:20:340
2791499223,8cyclictest29821-21munin-run09:40:010
2791499223,8cyclictest29821-21munin-run09:40:010
2791499223,13cyclictest81ktimersoftd/010:58:060
2791499222,10cyclictest22830-21/usr/sbin/munin08:10:120
27914992213,2cyclictest20811-21kworker/u8:110:17:120
2791499221,12cyclictest7443-21/usr/sbin/munin07:35:230
2791499220,12cyclictest11836-21/usr/sbin/munin07:45:130
12922220,3sleep00-21swapper/009:45:250
2792699211,13cyclictest0-21swapper/211:57:512
2792699211,11cyclictest22383-21fschecks_time10:35:162
2792699211,10cyclictest0-21swapper/211:06:472
2792299218,6cyclictest7292-21kworker/u8:208:23:001
2792299218,6cyclictest7292-21kworker/u8:208:23:001
27922992117,3cyclictest0-21swapper/108:02:341
2792299211,3cyclictest20065-21munin-run11:45:011
2792299211,18cyclictest0-21swapper/109:50:081
2792299211,11cyclictest5248-21diskstats11:10:131
27922992110,2cyclictest8062-21diskstats12:30:131
2791499217,5cyclictest28800-21df10:50:140
2791499217,5cyclictest28800-21df10:50:130
2791499217,3cyclictest0-21swapper/011:40:210
2791499214,7cyclictest87550irq/128-enp1s0-12:05:250
2791499214,3cyclictest29432-21/usr/sbin/munin08:25:150
2791499213,11cyclictest0-21swapper/009:50:110
2791499212,16cyclictest19046-21/usr/sbin/munin09:15:190
2791499211,8cyclictest0-21swapper/010:42:540
27914992111,6cyclictest6892-21grep10:00:140
2791499211,14cyclictest0-21swapper/009:28:110
2791499211,12cyclictest15267-21/usr/sbin/munin10:20:160
2791499211,12cyclictest0-21swapper/010:07:110
2791499211,10cyclictest0-21swapper/011:21:390
2792699209,4cyclictest2417-21df09:50:132
2792699206,4cyclictest0-21swapper/211:35:122
2792699201,9cyclictest18815-21sh09:15:002
2792699201,8cyclictest30095-21munin-run07:15:012
2792699201,6cyclictest0-21swapper/209:41:112
2792299207,6cyclictest7292-21kworker/u8:208:53:251
2792299207,11cyclictest7292-21kworker/u8:207:24:561
2792299201,8cyclictest23-21ksoftirqd/111:35:121
27922992016,3cyclictest0-21swapper/109:24:031
2792299201,18cyclictest0-21swapper/111:29:561
2792299201,16cyclictest31999-21strings07:15:191
2792299201,14cyclictest0-21swapper/110:22:271
2792299200,6cyclictest24874-21kernelversion10:40:161
2792299200,18cyclictest0-21swapper/111:48:341
2792299200,14cyclictest0-21swapper/112:26:571
2792299200,13cyclictest23-21ksoftirqd/112:15:021
2792299200,12cyclictest0-21swapper/109:17:241
2791499209,3cyclictest1106-21/usr/sbin/munin12:15:150
2791499208,9cyclictest21546-21smart_sda11:45:240
2791499207,4cyclictest0-21swapper/011:25:240
2791499205,6cyclictest0-21swapper/009:43:510
2791499203,8cyclictest22861-21fschecks_count11:50:140
2791499203,3cyclictest19665-21/usr/sbin/munin10:30:240
2791499202,11cyclictest0-21swapper/011:05:090
2791499201,9cyclictest0-21swapper/012:33:160
27914992012,5cyclictest1153-40dnf08:34:220
2791499201,11cyclictest87550irq/128-enp1s0-08:05:260
2791499201,10cyclictest9771-21sshd08:51:550
2791499201,10cyclictest27436-21sshd09:32:560
2791499201,10cyclictest25028-21/usr/sbin/munin08:15:200
2791499200,13cyclictest5242-21/usr/sbin/munin07:30:200
29072191,2sleep20-21swapper/212:17:462
2792699196,5cyclictest7919-1kworker/u9:009:25:272
2792699194,5cyclictest0-21swapper/212:00:142
2792699193,6cyclictest0-21swapper/208:00:042
2792699192,9cyclictest0-21swapper/211:48:032
2792699192,6cyclictest7114-40dnf07:33:562
2792699191,8cyclictest0-21swapper/211:20:062
2792699191,7cyclictest1849-21latency_hist09:50:022
2792699191,7cyclictest0-21swapper/210:53:032
2792699191,7cyclictest0-21swapper/210:53:032
2792699191,6cyclictest27521-21smart_sda10:45:262
2792699191,6cyclictest0-21swapper/212:35:052
27926991913,3cyclictest5291-21latency_hist12:25:022
2792699191,11cyclictest0-21swapper/212:28:302
2792699191,11cyclictest0-21swapper/209:16:022
2792699191,10cyclictest0-21swapper/210:18:032
2792299199,2cyclictest21-21rcuc/107:10:201
2792299198,3cyclictest87550irq/128-enp1s0-08:14:561
2792299196,12cyclictest21-21rcuc/111:09:561
2792299191,6cyclictest16618-21diskmemload09:10:271
27922991915,3cyclictest0-21swapper/108:39:531
2792299191,4cyclictest31288-21sendmail_mailst09:40:231
2792299191,16cyclictest0-21swapper/108:28:291
2792299190,18cyclictest0-21swapper/107:44:041
2791499199,6cyclictest10902-21smart_sda07:40:250
27914991912,4cyclictest3718-21hddtemp_smartct07:25:170
2791499191,11cyclictest0-21swapper/010:29:160
27914991910,5cyclictest31358-21diskstats07:15:130
2792699186,4cyclictest87350irq/126-enp1s0-07:25:252
2792699186,3cyclictest0-21swapper/209:33:312
2792699185,4cyclictest13489-21chrt07:46:082
2792699183,7cyclictest0-21swapper/210:20:242
2792699181,8cyclictest579-21systemd-journal10:15:022
2792699181,8cyclictest16670-21tune2fs07:55:152
2792699181,7cyclictest23291-21irqstats11:50:182
2792699181,7cyclictest0-21swapper/212:31:072
2792699181,7cyclictest0-21swapper/211:18:392
2792699181,6cyclictest25748-21idleruntime08:15:172
2792699181,6cyclictest0-21swapper/209:22:152
2792699181,6cyclictest0-21swapper/207:20:082
2792699181,3cyclictest0-21swapper/207:35:222
27926991813,3cyclictest11060-21interrupts08:55:172
2792699181,14cyclictest24217-21tr10:40:112
2792299187,9cyclictest0-21swapper/108:45:061
2792299185,3cyclictest0-21swapper/109:25:251
2792299183,3cyclictest23-21ksoftirqd/112:15:211
27922991815,2cyclictest0-21swapper/109:56:321
27922991815,2cyclictest0-21swapper/108:42:101
27922991815,2cyclictest0-21swapper/108:32:371
2792299181,16cyclictest0-21swapper/109:00:581
2792299181,16cyclictest0-21swapper/107:50:391
2792299181,15cyclictest0-21swapper/107:45:181
2792299180,14cyclictest0-21swapper/111:50:421
2791499182,13cyclictest28120-21/usr/sbin/munin07:10:240
27914991812,4cyclictest25126-21kworker/u8:008:00:120
2791499181,15cyclictest0-21swapper/008:37:400
295322171,2sleep20-21swapper/209:36:322
295322171,2sleep20-21swapper/209:36:322
2792699175,3cyclictest4639-21seq08:40:202
2792699174,3cyclictest0-21swapper/207:17:362
2792699171,7cyclictest5191-21memory09:55:182
2792699171,7cyclictest31240-21/usr/sbin/munin10:55:152
2792699171,7cyclictest1212-21latency_hist08:35:022
2792699171,6cyclictest21629-21sh10:35:012
2792699171,6cyclictest14422-21turbostat.cron09:05:012
2792699171,6cyclictest0-21swapper/207:53:372
2792699171,3cyclictest0-21swapper/208:10:462
27926991712,3cyclictest9311-21mailstats08:50:242
27926991712,3cyclictest28052-21cut08:20:192
27926991712,3cyclictest28052-21cut08:20:192
2792699171,15cyclictest0-21swapper/211:27:182
27926991711,3cyclictest18085-21/usr/sbin/munin11:40:152
27926991711,3cyclictest13688-21/usr/sbin/munin11:30:242
27922991714,2cyclictest0-21swapper/107:25:111
27922991713,3cyclictest0-21swapper/108:05:471
2792299171,2cyclictest10599-21diskstats08:55:141
2792299171,13cyclictest9030-21users07:35:271
2791499178,6cyclictest28485-21/usr/sbin/munin08:20:230
2791499178,6cyclictest28485-21/usr/sbin/munin08:20:230
2791499176,8cyclictest14405-21sshd09:04:560
2791499171,14cyclictest0-21swapper/007:50:290
27914991710,4cyclictest11062-21irqstats08:55:180
2792699164,4cyclictest16618-21diskmemload11:10:522
2792699161,7cyclictest0-21swapper/210:04:552
2792699161,3cyclictest0-21swapper/210:08:392
27926991613,2cyclictest0-21swapper/207:44:092
27926991612,3cyclictest0-21swapper/208:26:002
27926991611,3cyclictest29095-21acpi12:05:132
27926991611,3cyclictest20638-21/usr/sbin/munin08:05:262
27926991611,3cyclictest18415-21wc10:25:202
2791499160,14cyclictest0-21swapper/009:05:110
27926991510,3cyclictest6554-21idleruntime08:45:162
2792699150,14cyclictest0-21swapper/208:38:432
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional