You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-26 - 23:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Dec 26, 2025 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
41812236162,23sleep10-21swapper/107:06:231
44412232164,56sleep00-21swapper/007:09:450
41672204167,25sleep20-21swapper/207:06:122
25042202166,24sleep30-21swapper/307:05:083
1071221340,4sleep2455999cyclictest08:35:142
277192530,3sleep1455499cyclictest08:00:141
151532530,1sleep20-21swapper/207:30:162
140162490,1sleep30-21swapper/307:30:023
251892450,1sleep10-21swapper/109:05:261
4547993910,14cyclictest27200-21awk08:00:010
4547993511,11cyclictest12104-21perf12:20:020
454799344,22cyclictest19605-21kworker/u8:010:21:280
454799333,22cyclictest858-21kworker/u8:011:58:240
454799331,24cyclictest0-21swapper/012:07:460
454799306,12cyclictest11809-21munin-run07:25:010
310272300,8sleep339-21rcuc/308:05:253
456499292,4cyclictest88750irq/134-enp2s0-08:55:263
456499291,11cyclictest0-21swapper/307:51:253
455499281,17cyclictest6977-21latency_hist07:15:021
456499273,9cyclictest3873-21kworker/u8:209:57:443
455499271,18cyclictest12216-21cstates07:25:121
454799273,21cyclictest20177-21unixbench_singl07:40:260
454799272,6cyclictest11081-21cpu09:50:120
454799271,22cyclictest3583-21cpuspeed_turbos12:00:120
223932270,4sleep341-21ksoftirqd/309:00:193
454799264,20cyclictest11-21rcuc/007:45:190
454799261,22cyclictest24883-21/usr/sbin/munin11:35:240
208132260,5sleep121-21rcuc/110:10:201
454799251,2cyclictest0-21swapper/010:59:090
454799251,21cyclictest1607-21head09:25:260
454799251,21cyclictest0-21swapper/011:49:030
454799251,21cyclictest0-21swapper/011:42:100
454799251,21cyclictest0-21swapper/011:42:100
454799251,21cyclictest0-21swapper/011:25:010
454799251,21cyclictest0-21swapper/010:35:230
454799251,21cyclictest0-21swapper/010:06:550
454799251,11cyclictest17646-21grep11:15:270
4559992420,3cyclictest0-21swapper/207:45:592
455499245,10cyclictest9882-21crond12:15:021
454799241,21cyclictest0-21swapper/009:15:170
454799241,21cyclictest0-21swapper/009:15:170
454799241,20cyclictest18929-21cpu12:35:100
454799241,20cyclictest0-21swapper/012:13:170
454799241,20cyclictest0-21swapper/011:53:510
454799241,20cyclictest0-21swapper/011:02:390
454799241,20cyclictest0-21swapper/010:50:570
454799241,20cyclictest0-21swapper/010:44:040
454799241,20cyclictest0-21swapper/010:16:340
454799241,20cyclictest0-21swapper/010:00:470
454799241,20cyclictest0-21swapper/009:56:370
454799241,20cyclictest0-21swapper/009:36:310
454799241,20cyclictest0-21swapper/009:22:150
454799241,11cyclictest16708-21/usr/sbin/munin12:30:180
454799241,11cyclictest0-21swapper/011:05:180
455999234,4cyclictest30339-21runrttasks10:31:292
455999231,13cyclictest0-21swapper/212:35:412
455999231,13cyclictest0-21swapper/210:35:062
455499231,16cyclictest0-21swapper/108:39:171
455499231,11cyclictest28831-21cat10:30:131
455499230,17cyclictest0-21swapper/111:13:261
454799239,3cyclictest10450-21/usr/sbin/munin08:35:160
454799238,3cyclictest12841-21irqstats07:25:180
454799231,20cyclictest7-21ksoftirqd/011:10:180
454799231,20cyclictest0-21swapper/012:28:210
454799231,20cyclictest0-21swapper/011:30:130
454799231,20cyclictest0-21swapper/010:12:400
454799231,20cyclictest0-21swapper/009:42:560
454799231,19cyclictest25840-21/usr/sbin/munin09:10:110
454799231,19cyclictest0-21swapper/011:20:590
454799231,11cyclictest16151-21sshd07:34:340
454799231,11cyclictest14839-21/usr/sbin/munin08:45:100
454799231,10cyclictest2572-21memory08:15:200
456499221,6cyclictest28378-21df09:15:143
456499221,6cyclictest28378-21df09:15:143
456499221,16cyclictest0-21swapper/309:47:543
456499221,15cyclictest0-21swapper/309:32:173
455999221,9cyclictest23589-21fschecks_time07:50:152
455999221,7cyclictest380-21grep09:25:142
455999221,10cyclictest9293-21/usr/sbin/munin08:30:202
455499228,4cyclictest31591-21munin-run08:10:011
455499227,13cyclictest19637-21kworker/u8:009:35:011
455499226,4cyclictest0-21swapper/109:10:191
4554992214,2cyclictest0-21swapper/110:19:271
4554992213,3cyclictest0-21swapper/112:00:141
455499221,16cyclictest0-21swapper/108:27:301
455499220,16cyclictest0-21swapper/107:43:571
454799221,19cyclictest11827-21kworker/u8:112:22:490
454799221,19cyclictest0-21swapper/010:29:130
454799221,11cyclictest826-21lldpd08:24:240
454799221,11cyclictest0-21swapper/007:35:140
454799221,10cyclictest801-21snmpd09:07:250
454799221,10cyclictest14502-21kworker/0:308:50:250
456499217,3cyclictest0-21swapper/308:53:443
456499216,3cyclictest0-21swapper/308:11:493
456499211,9cyclictest0-21swapper/308:31:053
455999216,7cyclictest0-21swapper/211:59:392
455999211,7cyclictest0-21swapper/209:19:582
455999211,7cyclictest0-21swapper/209:19:582
455999211,6cyclictest14952-21grep08:45:112
455999211,3cyclictest0-21swapper/210:26:062
455999211,12cyclictest0-21swapper/209:30:072
455999211,11cyclictest24625-21sed09:05:212
455499218,4cyclictest28048-21users10:25:261
455499212,11cyclictest32625-21irqstats08:10:181
455499211,3cyclictest29670-21grep11:45:121
4554992113,3cyclictest0-21swapper/110:05:021
455499211,2cyclictest21272-21fschecks_count07:45:151
455499211,15cyclictest0-21swapper/111:08:461
455499211,15cyclictest0-21swapper/109:52:351
455499211,15cyclictest0-21swapper/109:45:471
455499211,14cyclictest0-21swapper/110:35:021
455499211,14cyclictest0-21swapper/107:22:121
455499211,12cyclictest9524-21cpu11:00:141
455499210,15cyclictest0-21swapper/108:18:351
454799211,9cyclictest9023-21cpuspeed_turbos09:45:120
454799211,9cyclictest3746-21grep09:30:260
454799211,11cyclictest0-21swapper/010:46:580
167962210,5sleep121-21rcuc/110:00:251
456499206,3cyclictest15904-21ntp_states08:45:203
456499206,2cyclictest26574-21grep10:25:113
456499205,3cyclictest0-21swapper/307:59:533
456499201,7cyclictest0-21swapper/310:12:323
456499201,13cyclictest0-21swapper/312:22:553
456499201,13cyclictest0-21swapper/310:52:383
456499201,10cyclictest0-21swapper/312:35:163
456499201,10cyclictest0-21swapper/311:35:273
455999209,2cyclictest87150irq/128-enp1s0-11:22:192
455999206,3cyclictest87250irq/129-enp1s0-12:30:132
455999206,3cyclictest87250irq/129-enp1s0-08:00:242
455999205,4cyclictest6537-21NetworkManager07:12:492
455999205,4cyclictest25613-21diskmemload09:35:132
455999205,3cyclictest22604-21ps09:00:232
455999204,8cyclictest86950irq/126-enp1s0-10:15:172
455999203,8cyclictest27759-21hddtemp_smartct11:40:192
455999203,8cyclictest27759-21hddtemp_smartct11:40:192
455999202,7cyclictest15463-21/usr/sbin/munin10:00:142
455999201,7cyclictest6400-21hddtemp_smartct12:05:152
455999201,7cyclictest19949-21kworker/u8:011:35:172
455999201,7cyclictest0-21swapper/212:16:382
455999201,6cyclictest0-21swapper/211:00:072
455999201,6cyclictest0-21swapper/210:42:122
455999201,6cyclictest0-21swapper/209:48:122
455999201,10cyclictest11095-21smart_sda07:20:242
4554992016,3cyclictest0-21swapper/109:16:171
4554992016,3cyclictest0-21swapper/109:16:171
4554992012,3cyclictest0-21swapper/109:36:101
4554992012,3cyclictest0-21swapper/109:21:031
4554992012,3cyclictest0-21swapper/107:17:501
4554992012,2cyclictest0-21swapper/110:21:151
455499201,17cyclictest0-21swapper/112:15:121
455499201,15cyclictest0-21swapper/108:56:451
455499201,14cyclictest0-21swapper/110:42:291
455499201,14cyclictest0-21swapper/109:02:421
455499201,14cyclictest0-21swapper/108:48:101
455499201,14cyclictest0-21swapper/108:43:481
455499201,14cyclictest0-21swapper/108:31:041
455499201,14cyclictest0-21swapper/108:21:031
455499201,11cyclictest6872-21cstates09:40:141
455499201,11cyclictest20835-21diskstats11:25:151
455499201,11cyclictest14843-21head07:30:131
455499201,11cyclictest0-21swapper/110:45:201
455499201,10cyclictest1068-21perf11:55:011
4554992010,3cyclictest18321-21cpu11:20:121
454799204,14cyclictest7-21ksoftirqd/008:05:240
454799204,14cyclictest0-21swapper/010:31:360
454799201,6cyclictest8364-21tr08:30:110
4547992015,3cyclictest14631-21awk08:45:010
456499199,2cyclictest18072-21awk11:20:013
456499199,2cyclictest10118-21/usr/sbin/munin12:15:243
456499198,3cyclictest13891-21/usr/sbin/munin11:10:243
456499196,7cyclictest18639-40dnf12:34:413
456499196,4cyclictest8012-21threads09:40:263
456499194,3cyclictest41-21ksoftirqd/311:40:173
456499194,3cyclictest41-21ksoftirqd/311:40:173
456499192,9cyclictest0-21swapper/307:39:403
456499191,9cyclictest0-21swapper/310:32:243
456499191,8cyclictest5338-21sed08:20:263
456499191,8cyclictest0-21swapper/310:55:253
456499191,6cyclictest32291-21kernelversion11:50:213
456499191,6cyclictest0-21swapper/309:25:583
4564991913,3cyclictest10644-21irqstats07:20:183
456499191,12cyclictest0-21swapper/309:10:253
456499191,11cyclictest0-21swapper/311:55:223
456499191,10cyclictest9490-21/usr/sbin/munin11:00:153
456499191,10cyclictest7921-21/usr/sbin/munin12:10:143
456499191,10cyclictest3504-21sshd08:18:383
456499191,10cyclictest24263-21/usr/sbin/munin10:20:153
456499191,10cyclictest17672-21/usr/sbin/munin10:05:193
456499190,5cyclictest0-21swapper/309:20:123
455999195,3cyclictest87250irq/129-enp1s0-11:15:152
455999194,3cyclictest32322-21kernelversion11:50:202
455999193,8cyclictest86950irq/126-enp1s0-09:50:212
455999193,4cyclictest32084-21sshd09:23:392
455999191,7cyclictest21562-21sshd10:11:382
455999191,6cyclictest7047-21needreboot08:25:212
455999191,6cyclictest30-21rcuc/212:10:152
455999191,6cyclictest24263-21/usr/sbin/munin10:20:232
455999191,6cyclictest22070-21uptime11:25:282
455999191,6cyclictest0-21swapper/212:00:242
4559991913,3cyclictest13365-21cpuspeed_turbos09:55:122
455999191,11cyclictest0-21swapper/210:59:512
455999191,10cyclictest801-21snmpd07:57:132
455999191,10cyclictest0-21swapper/212:23:242
455499198,3cyclictest25002-21awk07:55:011
455499197,3cyclictest23593-21memory11:30:171
455499191,9cyclictest0-21swapper/111:35:201
4554991913,4cyclictest16766-21df07:35:151
4554991912,2cyclictest0-21swapper/108:51:051
455499191,14cyclictest0-21swapper/110:53:111
455499191,14cyclictest0-21swapper/109:55:211
455499191,14cyclictest0-21swapper/107:56:291
455499191,12cyclictest0-21swapper/111:16:391
455499191,11cyclictest0-21swapper/112:25:281
455499191,10cyclictest0-21swapper/112:35:101
455499190,14cyclictest0-21swapper/110:55:321
455499190,14cyclictest0-21swapper/109:26:241
454799193,15cyclictest7-21ksoftirqd/008:00:250
454799191,2cyclictest0-21swapper/009:00:140
456499187,3cyclictest4474-21/usr/sbin/munin09:35:183
456499184,11cyclictest41-21ksoftirqd/311:25:183
456499182,7cyclictest1763-21ntp_states10:40:223
456499181,6cyclictest401ktimersoftd/307:32:553
456499181,6cyclictest0-21swapper/307:15:133
456499181,3cyclictest15576-21ntp_states12:25:223
456499181,11cyclictest0-21swapper/311:05:183
456499181,10cyclictest0-21swapper/311:32:373
456499180,10cyclictest0-21swapper/308:25:153
455999185,3cyclictest87250irq/129-enp1s0-08:40:202
455999185,3cyclictest87250irq/129-enp1s0-07:40:242
455999185,2cyclictest87150irq/128-enp1s0-11:30:252
455999184,3cyclictest87250irq/129-enp1s0-11:45:132
455999184,3cyclictest87250irq/129-enp1s0-09:40:162
455999184,3cyclictest30339-21idleruntime08:05:162
455999181,9cyclictest801-21snmpd10:52:442
455999181,9cyclictest21205-21latency_hist09:00:022
455999181,8cyclictest7246-21idleruntime07:15:032
455999181,8cyclictest0-21swapper/211:05:242
455999181,6cyclictest2893-21/usr/sbin/munin10:45:212
455999181,5cyclictest26515-21hddtemp_smartct09:10:152
455999181,5cyclictest17672-21/usr/sbin/munin10:05:152
455999181,5cyclictest15369-21unixbench-2d11:10:252
4559991813,3cyclictest31838-21cpu08:10:122
455499187,9cyclictest826-21lldpd11:40:201
455499187,9cyclictest826-21lldpd11:40:201
4554991812,3cyclictest1624-21df11:55:131
455499181,16cyclictest0-21swapper/112:21:341
455499181,11cyclictest0-21swapper/112:30:271
455499180,16cyclictest0-21swapper/112:05:091
454799181,8cyclictest0-21swapper/007:10:030
454799181,5cyclictest24083-21ntp_states07:50:220
454799181,3cyclictest8272-21gltestperf07:15:160
4547991813,3cyclictest7119-21ntp_states08:25:220
4547991812,3cyclictest32459-21gltestperf08:10:160
456499174,5cyclictest39-21rcuc/311:20:533
456499173,3cyclictest0-21swapper/307:41:103
456499172,3cyclictest87250irq/129-enp1s0-11:45:113
456499171,6cyclictest0-21swapper/312:02:163
456499171,5cyclictest751-21rngd10:03:383
4564991713,3cyclictest0-21swapper/310:37:153
456499171,15cyclictest0-21swapper/308:04:383
456499171,13cyclictest4747-21/usr/sbin/munin07:10:173
4564991711,3cyclictest3159-21cat10:45:123
456499171,10cyclictest0-21swapper/309:50:223
455999177,2cyclictest87250irq/129-enp1s0-08:18:522
455999172,5cyclictest0-21swapper/207:25:532
455999171,8cyclictest14536-21cpu12:25:122
455999171,6cyclictest0-21swapper/208:50:122
4559991712,3cyclictest5118-21smart_sda08:20:252
4559991712,3cyclictest17485-21ntpq07:35:222
454799172,8cyclictest18102-21kworker/0:108:55:250
456499168,2cyclictest39-21rcuc/309:06:253
456499166,3cyclictest87250irq/129-enp1s0-10:15:223
456499162,2cyclictest88850irq/135-enp2s0-07:48:113
456499161,9cyclictest0-21swapper/312:05:223
456499161,5cyclictest0-21swapper/308:40:183
4564991510,3cyclictest12590-21date08:40:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional