You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-18 - 11:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Aug 18, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
124142238163,24sleep20-21swapper/219:05:052
141972237162,24sleep00-21swapper/019:07:410
141832236162,23sleep30-21swapper/319:07:323
141412203166,24sleep10-21swapper/119:06:571
1447599108104,3cyclictest0-21swapper/221:54:542
1447599107103,3cyclictest0-21swapper/222:23:012
144759910198,2cyclictest0-21swapper/219:35:372
14475999591,3cyclictest0-21swapper/220:18:112
14475997168,2cyclictest0-21swapper/222:05:502
14475997168,2cyclictest0-21swapper/219:41:062
14475996664,1cyclictest0-21swapper/221:48:422
260582600,4sleep07-21ksoftirqd/023:15:100
1447999429,17cyclictest3174-21diskmemload21:12:273
1447599405,6cyclictest10131-40dnf23:50:242
1447599362,7cyclictest26666-40dnf20:49:252
14475993513,3cyclictest17996-21tr00:10:122
14475993512,3cyclictest26406-21idleruntime-cro00:30:012
18042340,4sleep31447999cyclictest23:30:263
1447999348,18cyclictest87350irq/126-enp1s0-23:05:213
14479993410,14cyclictest28621-21awk00:35:013
1447599342,2cyclictest0-21swapper/223:45:162
1447999338,10cyclictest12325-21cpuspeed_turbos21:30:113
1446299335,13cyclictest13554-21idleruntime-cro20:20:010
1447999329,19cyclictest8409-21/usr/sbin/munin22:35:173
1447999327,8cyclictest2563-21awk19:55:013
1447999326,19cyclictest87350irq/126-enp1s0-21:30:023
14479993210,7cyclictest4521-21cat23:40:013
14479993210,11cyclictest3786-21latency_hist22:25:013
1447599323,10cyclictest14450-21cyclictest00:20:402
293502310,1sleep30-21swapper/319:40:183
14479993111,12cyclictest19170-21latency_hist23:00:023
1447599315,3cyclictest27974-21idleruntime-cro23:20:012
1447599313,5cyclictest763-21dbus-daemon20:40:202
1447599311,8cyclictest22319-21grep19:25:162
1447599311,7cyclictest0-21swapper/222:33:042
1447599311,6cyclictest2212-21awk23:35:012
1447599311,3cyclictest87350irq/126-enp1s0-21:05:162
1447599311,3cyclictest87350irq/126-enp1s0-21:05:162
14475993113,2cyclictest28838-21/usr/sbin/munin00:35:172
14475993113,2cyclictest28838-21/usr/sbin/munin00:35:162
14475993112,3cyclictest30135-21needreboot20:55:212
14475993112,3cyclictest18338-21expr20:30:132
206392300,5sleep11446899cyclictest19:20:181
1447999309,14cyclictest5848-21awk23:40:223
1447999309,11cyclictest29779-21/usr/sbin/munin22:10:163
1447999307,5cyclictest87350irq/126-enp1s0-21:45:153
1447999306,15cyclictest0-21swapper/300:37:053
1447999306,15cyclictest0-21swapper/300:37:053
1447999306,13cyclictest6825-21/usr/sbin/munin23:45:183
14479993010,9cyclictest808-21snmpd22:33:253
14479993010,6cyclictest8759-21uptime20:05:253
14479993010,5cyclictest2765-21grep21:05:283
14479993010,5cyclictest2765-21grep21:05:283
14475993012,3cyclictest9105-21idleruntime22:35:172
27892290,2sleep20-21swapper/223:35:152
1447999299,7cyclictest11226-21/usr/sbin/munin23:55:273
1447999297,6cyclictest24146-21/usr/sbin/munin19:30:143
1447999296,11cyclictest0-21swapper/321:35:143
14479992911,9cyclictest18868-21ls19:15:243
14479992911,10cyclictest87350irq/126-enp1s0-22:25:283
14479992910,4cyclictest808-21snmpd21:40:223
14479992910,10cyclictest2775-21/usr/sbin/munin19:55:153
14479992910,10cyclictest2775-21/usr/sbin/munin19:55:143
1447599291,4cyclictest0-21swapper/200:05:492
14475992912,3cyclictest15384-21taskset00:04:442
14475992911,3cyclictest17923-21if_enp1s022:55:162
1446299294,12cyclictest4744-21munin-run20:00:010
1446299294,12cyclictest4744-21munin-run20:00:010
1447999289,6cyclictest26408-21latency_hist00:30:013
1447999289,5cyclictest87350irq/126-enp1s0-23:50:253
1447999289,15cyclictest26369-21needreboot22:00:213
1447999287,7cyclictest11788-21expr20:15:143
1447999287,13cyclictest13769-21/usr/sbin/munin20:20:253
1447999287,13cyclictest10109-21if_enp1s020:10:183
14479992817,3cyclictest694-21sshd21:02:393
14479992813,7cyclictest17850-21/usr/sbin/munin00:10:283
14479992813,6cyclictest682-21/usr/sbin/munin22:15:223
14479992813,6cyclictest682-21/usr/sbin/munin22:15:223
1447999281,19cyclictest0-21swapper/300:05:463
14479992811,5cyclictest12050irq/123-ahci[0023:30:013
14479992811,5cyclictest12050irq/123-ahci[0023:30:013
1447999281,12cyclictest0-21swapper/320:55:133
1447599285,12cyclictest495-21jbd2/dm-0-800:33:042
1447599282,4cyclictest808-21snmpd19:50:002
1447599281,5cyclictest3845-21ntp_states19:55:222
1447599281,5cyclictest3845-21ntp_states19:55:222
1447599281,4cyclictest8564-1kworker/u9:123:05:512
1447599281,12cyclictest0-21swapper/221:18:142
14475992810,9cyclictest0-21swapper/221:21:202
1446899286,13cyclictest0-21swapper/122:15:121
1446899286,13cyclictest0-21swapper/122:15:121
1446299284,11cyclictest32361-21sadc23:30:010
1446299284,11cyclictest32361-21sadc23:30:010
1447999278,8cyclictest12801-21/usr/sbin/munin22:45:143
1447999278,12cyclictest28291-21sort23:20:133
1447999277,7cyclictest22225-21/usr/sbin/munin21:50:233
1447999276,12cyclictest13427-21/usr/sbin/munin00:00:253
1447999273,17cyclictest0-21swapper/323:03:003
1447999273,11cyclictest0-21swapper/322:50:193
1447999271,7cyclictest0-21swapper/300:20:373
1447599279,3cyclictest0-21swapper/222:12:162
1447599279,3cyclictest0-21swapper/220:24:162
1447599271,13cyclictest0-21swapper/221:30:122
14475992711,3cyclictest0-21swapper/221:00:162
14475992711,2cyclictest3174-21diskmemload23:10:202
14475992710,3cyclictest26304-21ls22:00:202
14475992710,3cyclictest21037-21hddtemp_smartct20:35:162
1447999268,12cyclictest87350irq/126-enp1s0-21:16:433
1447999267,11cyclictest20051-21/usr/sbin/munin00:15:233
1447999266,16cyclictest8607-21irqstats21:20:183
1447999265,13cyclictest87350irq/126-enp1s0-23:15:233
1447999263,13cyclictest24758-21/usr/sbin/munin20:45:273
14479992611,6cyclictest30751-21grep19:45:113
1447999260,19cyclictest87350irq/126-enp1s0-22:42:033
1447599269,3cyclictest0-21swapper/222:26:402
1447599269,11cyclictest0-21swapper/222:45:202
1447599268,10cyclictest0-21swapper/221:55:442
1447599261,9cyclictest0-21swapper/221:41:212
1447599261,4cyclictest11957-21latency_hist21:30:022
14475992611,3cyclictest0-21swapper/219:20:202
14475992610,3cyclictest0-21swapper/223:23:442
14468992610,8cyclictest0-21swapper/123:20:561
1446899260,10cyclictest0-21swapper/122:09:401
1446299261,13cyclictest24544-21awk20:45:010
1446299261,12cyclictest22340-21munin-run20:40:000
1446299261,11cyclictest25822-21ntp_states20:45:220
1446299261,11cyclictest23535-21sshd23:09:390
1447999256,11cyclictest808-21snmpd20:00:013
1447999255,12cyclictest23240-21cpu21:55:143
1447999254,9cyclictest21972-21egrep19:25:133
1447999254,11cyclictest87350irq/126-enp1s0-22:05:133
14479992516,6cyclictest22622-21/usr/sbin/munin20:40:133
14479992516,5cyclictest24153-21diskstats23:10:153
14479992516,5cyclictest21625-21smart_sda20:35:243
14479992511,6cyclictest19153-21sed20:30:203
1447599259,4cyclictest0-21swapper/220:11:242
1447599259,3cyclictest576-21ntpq22:15:202
1447599259,3cyclictest576-21ntpq22:15:202
1447599259,3cyclictest0-21swapper/222:41:402
1447599259,3cyclictest0-21swapper/220:09:362
1447599259,3cyclictest0-21swapper/200:18:522
1447599258,3cyclictest0-21swapper/223:44:242
1447599258,3cyclictest0-21swapper/221:35:242
1447599258,3cyclictest0-21swapper/220:27:122
14475992511,2cyclictest3174-21diskmemload21:11:002
1446299251,11cyclictest18786-21/usr/sbin/munin21:45:260
1446299251,11cyclictest14664-21/usr/sbin/munin19:10:150
1447999248,5cyclictest26959-21/usr/sbin/munin20:50:273
1447999240,15cyclictest0-21swapper/319:10:143
1447599249,3cyclictest0-21swapper/223:02:442
1447599247,3cyclictest0-21swapper/223:26:322
1447599247,3cyclictest0-21swapper/223:26:322
1447599241,4cyclictest14918-21expr19:10:142
14475992410,3cyclictest0-21swapper/219:50:082
1446899242,3cyclictest23579-21munin-run23:10:011
1446899241,4cyclictest12979-21sshd23:57:261
14468992410,6cyclictest8564-1kworker/u9:122:59:291
1446299243,2cyclictest87550irq/128-enp1s0-00:12:530
1446299241,11cyclictest11226-21/usr/sbin/munin23:55:130
1446299241,10cyclictest7797-21/usr/sbin/munin21:20:130
1446299241,10cyclictest24146-21/usr/sbin/munin19:30:200
14462992410,9cyclictest909-21sssd_nss23:50:250
185342230,9sleep081ktimersoftd/019:15:190
14479992314,6cyclictest20108-21/usr/sbin/munin19:20:143
14479992314,6cyclictest15972-21/usr/sbin/munin20:25:173
1447599239,3cyclictest0-21swapper/223:55:282
1447599238,3cyclictest0-21swapper/222:51:002
1447599237,3cyclictest0-21swapper/220:52:162
1446899232,5cyclictest32510-21sshd19:47:391
14468992311,3cyclictest0-21swapper/121:40:221
1446299231,6cyclictest0-21swapper/023:42:430
1446299231,5cyclictest0-21swapper/023:01:110
1446299231,4cyclictest0-21swapper/022:32:130
1446299231,11cyclictest0-21swapper/023:10:230
1446299231,10cyclictest0-21swapper/022:09:160
1446299230,4cyclictest0-21swapper/022:59:290
93782221,3sleep00-21swapper/020:10:140
14479992213,6cyclictest26339-21/usr/sbin/munin19:35:133
1447599227,3cyclictest0-21swapper/219:18:522
1447599225,3cyclictest0-21swapper/220:03:082
1446899229,3cyclictest15994-21diskstats00:05:171
1446899229,11cyclictest31778-21kworker/u8:220:30:451
14468992217,2cyclictest26508-21cat00:30:011
14468992213,7cyclictest23-21ksoftirqd/120:02:101
14468992212,3cyclictest2426-21rt-features21:05:251
14468992212,3cyclictest2426-21rt-features21:05:251
1446899221,18cyclictest6558-21df_inode22:30:141
1446899221,15cyclictest0-21swapper/121:19:151
1446899221,13cyclictest0-21swapper/121:51:011
1446899220,3cyclictest0-21swapper/122:46:431
1446299221,6cyclictest0-21swapper/000:08:520
1446299221,4cyclictest0-21swapper/022:20:180
1446299221,4cyclictest0-21swapper/000:27:310
1446299221,10cyclictest6677-21sshd20:01:390
177762211,2sleep00-21swapper/020:28:040
1446899219,6cyclictest11920-21sshd21:29:391
1446899219,11cyclictest27513-21kworker/u8:020:15:101
1446899218,4cyclictest8564-1kworker/u9:123:03:111
1446899214,3cyclictest495-21jbd2/dm-0-823:35:061
1446899211,5cyclictest8564-1kworker/u9:100:10:421
1446899211,3cyclictest25731-21users19:30:251
1446899211,2cyclictest5091-21ntpq22:25:231
1446899211,17cyclictest3766-21strings19:55:201
1446899211,17cyclictest3766-21strings19:55:201
1446899211,17cyclictest20762-21munin-run21:50:021
1446899211,14cyclictest0-21swapper/123:30:201
1446899211,13cyclictest7636-21irqstats23:45:171
1446899211,13cyclictest24477-21hddtemp_smartct23:10:171
1446899210,16cyclictest0-21swapper/122:22:001
1446299217,3cyclictest10605-21/usr/sbin/munin22:40:150
1446299215,4cyclictest3174-21diskmemload22:45:220
1446299211,9cyclictest0-21swapper/000:32:510
1446299211,6cyclictest0-21swapper/022:29:570
1446299211,6cyclictest0-21swapper/000:04:370
1446299211,5cyclictest28838-21/usr/sbin/munin00:35:240
1446299211,5cyclictest28838-21/usr/sbin/munin00:35:240
1446299211,5cyclictest0-21swapper/021:39:360
1446299211,4cyclictest0-21swapper/021:26:150
1446299211,4cyclictest0-21swapper/021:12:410
145812200,3sleep114585-21pmu-power00:00:221
1446899207,4cyclictest8564-1kworker/u9:100:36:561
1446899207,4cyclictest8564-1kworker/u9:100:36:561
1446899202,11cyclictest0-21swapper/123:42:321
14468992017,2cyclictest0-21swapper/119:29:141
14468992016,3cyclictest0-21swapper/119:35:231
1446899201,5cyclictest22562-21/usr/sbin/munin20:40:201
14468992011,3cyclictest0-21swapper/100:20:261
14468992011,2cyclictest23313-21cpuspeed_turbos21:55:141
1446899201,12cyclictest0-21swapper/122:35:081
1446899201,11cyclictest0-21swapper/123:53:001
1446899200,9cyclictest0-21swapper/122:11:581
1446899200,8cyclictest0-21swapper/122:53:341
1446899200,2cyclictest0-21swapper/122:00:221
1446899200,18cyclictest0-21swapper/122:40:171
1446899200,18cyclictest0-21swapper/121:35:211
1446899200,13cyclictest0-21swapper/100:18:131
1446299201,8cyclictest0-21swapper/023:35:410
14462992013,4cyclictest29923-21tr22:10:110
1446299201,11cyclictest0-21swapper/022:39:150
1446299200,10cyclictest8564-1kworker/u9:122:53:070
1447599198,3cyclictest31778-21kworker/u8:219:34:442
1446899196,4cyclictest0-21swapper/123:16:011
1446899191,4cyclictest32427-21ntp_states21:00:241
14468991914,3cyclictest0-21swapper/121:20:171
1446899191,17cyclictest0-21swapper/120:25:491
1446899191,16cyclictest0-21swapper/120:52:001
1446899191,16cyclictest0-21swapper/119:41:301
1446899191,10cyclictest22317-21cat20:39:561
1446899190,17cyclictest0-21swapper/121:30:121
1446899190,17cyclictest0-21swapper/120:55:241
1446899190,17cyclictest0-21swapper/119:15:011
1446899190,13cyclictest0-21swapper/123:25:261
1446899190,13cyclictest0-21swapper/123:25:261
1446899190,13cyclictest0-21swapper/100:32:341
1446299194,7cyclictest87550irq/128-enp1s0-21:30:110
1446299191,7cyclictest3177-21munin-run21:10:010
1446299191,7cyclictest3177-21munin-run21:10:010
1446299191,6cyclictest0-21swapper/023:23:210
1446299191,5cyclictest22249-21/usr/sbin/munin00:20:150
1446299191,4cyclictest5602-21/usr/sbin/munin21:15:160
87672181,2sleep00-21swapper/023:49:460
14468991812,3cyclictest8435-21runrttasks20:05:231
14468991812,3cyclictest4464-21ntp_states21:10:181
1446899181,16cyclictest0-21swapper/120:49:541
1446899181,14cyclictest19143-21users19:15:261
1446299185,11cyclictest31368-21cat21:00:140
1446299182,13cyclictest25379-21/usr/sbin/munin22:00:240
1446299181,7cyclictest0-21swapper/021:54:130
1446299181,6cyclictest0-21swapper/023:31:330
14462991813,3cyclictest20723-21hddtemp_smartct00:15:150
1446299181,14cyclictest21942-21taskset19:25:060
112642181,3sleep10-21swapper/120:14:081
1446899175,3cyclictest14098-21cstates20:20:141
1446899171,13cyclictest1225-21hddtemp_smartct19:50:141
1446299173,2cyclictest87550irq/128-enp1s0-22:15:110
1446299173,2cyclictest87550irq/128-enp1s0-22:15:110
1446299171,7cyclictest16585-21/usr/sbin/munin21:40:200
1446299171,6cyclictest29158-21/usr/sbin/munin20:55:280
1446299171,6cyclictest0-21swapper/021:56:250
1446299171,3cyclictest0-21swapper/019:45:150
1446299171,15cyclictest0-21swapper/020:54:310
1446299171,14cyclictest0-21swapper/019:20:550
1446299171,13cyclictest473-21/usr/sbin/munin19:50:120
1446299171,13cyclictest28536-21/usr/sbin/munin19:40:170
1446299171,13cyclictest18172-21/usr/sbin/munin20:30:240
14462991711,3cyclictest15740-21latency_hist20:25:020
14462991611,3cyclictest7884-21idleruntime20:05:160
1446299161,12cyclictest26339-21/usr/sbin/munin19:35:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional