You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-11 - 23:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Mon May 11, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
284912205167,26sleep20-21swapper/207:09:012
284992203165,26sleep00-21swapper/007:09:060
285152202166,24sleep10-21swapper/107:09:201
266152201164,24sleep30-21swapper/307:05:103
3110821250,4sleep22866399cyclictest07:15:012
28663999895,2cyclictest0-21swapper/211:12:442
28663998683,2cyclictest0-21swapper/211:52:212
28663998582,2cyclictest0-21swapper/212:16:372
28663998582,2cyclictest0-21swapper/212:16:372
28663997572,2cyclictest0-21swapper/209:30:172
196892750,2sleep339-21rcuc/309:15:013
317312620,4sleep339-21rcuc/310:55:153
90402560,2sleep20-21swapper/212:30:172
79592530,4sleep00-21swapper/008:45:250
79592530,4sleep00-21swapper/008:45:250
28663994339,3cyclictest0-21swapper/211:00:122
2866999427,18cyclictest25547-21cpuspeed_turbos11:55:123
2866999421,27cyclictest22234-21chrt10:33:203
2866999402,27cyclictest14430-21/usr/sbin/munin11:30:123
2866999383,29cyclictest87350irq/126-enp1s0-12:10:193
2866999381,28cyclictest23220-21/usr/sbin/munin11:50:213
2866999381,28cyclictest1861-21/usr/sbin/munin12:15:233
2866999381,28cyclictest1861-21/usr/sbin/munin12:15:233
2866999381,25cyclictest31060-21memory07:15:023
2866999381,24cyclictest31128-21sshd10:54:373
2866999381,23cyclictest12345-21latency_hist07:45:013
2865199389,13cyclictest29958-21idleruntime-cro08:25:000
2865199389,13cyclictest18613-21awk11:40:020
2866999372,26cyclictest16625-21/usr/sbin/munin11:35:133
2866999371,25cyclictest22987-21latency_hist11:50:013
2866999371,24cyclictest28356-21munin-run09:35:003
2866999363,22cyclictest87550irq/128-enp1s0-12:25:233
2866999352,28cyclictest2659-21ntp_states07:20:223
2866999351,25cyclictest31051-21gzip12:05:233
2866999351,20cyclictest6423-21cat08:45:023
2866999351,19cyclictest9716-21smart_sda12:30:253
2866999342,23cyclictest10651-21/usr/sbin/munin12:35:173
2866999341,21cyclictest808-21snmpd07:55:373
2866999331,3cyclictest87350irq/126-enp1s0-09:40:213
2866999331,20cyclictest808-21snmpd11:28:333
2866999330,19cyclictest0-21swapper/311:15:163
2866399331,12cyclictest5172-21tr09:55:142
2866999322,25cyclictest15741-40dnf10:19:403
2866999321,4cyclictest25766-21/usr/sbin/munin08:15:223
2866999321,28cyclictest7620-21grep10:00:163
2866999321,28cyclictest28850-21taskset10:48:373
28669993212,17cyclictest19167-21/usr/sbin/munin08:00:133
28669993212,17cyclictest18118-21cut10:25:023
28669993211,17cyclictest26774-21perf10:45:013
2866999314,17cyclictest0-21swapper/309:25:303
2866999314,13cyclictest0-21swapper/311:21:273
2866999313,25cyclictest5642-21/usr/sbin/munin11:10:153
2866999311,16cyclictest0-21swapper/309:55:433
2866999311,16cyclictest0-21swapper/309:35:123
28669993111,17cyclictest3214-21idleruntime-cro11:05:013
2866999306,21cyclictest763-21dbus-daemon08:10:213
2866999301,2cyclictest87550irq/128-enp1s0-09:15:163
2866999301,2cyclictest27621-21acpi12:00:123
2866999301,19cyclictest808-21snmpd12:23:533
2866999301,16cyclictest323-21fschecks_count08:30:163
28669993010,17cyclictest15194-21fschecks_count07:50:153
2866999300,23cyclictest0-21swapper/307:15:113
28651993011,16cyclictest15172-21munin-run09:05:020
2866999297,19cyclictest4337-21cat11:05:203
2866999297,19cyclictest4337-21cat11:05:193
2866999296,14cyclictest16565-21ps09:05:233
2866999292,24cyclictest23276-21hddtemp_smartct10:35:173
2866999292,23cyclictest22337-21diskstats09:20:143
2866999291,3cyclictest19745-21/usr/sbin/munin10:25:273
2866999291,3cyclictest1882-21smart_sda09:45:223
2866999291,25cyclictest28313-21diskstats08:20:143
2866999291,25cyclictest18993-21/usr/sbin/munin11:40:123
2866999291,25cyclictest13189-21/usr/sbin/munin09:00:203
2866999291,25cyclictest12201-21/usr/sbin/munin10:10:173
28669992910,16cyclictest10961-21cut08:55:023
2866999290,2cyclictest87550irq/128-enp1s0-09:50:213
2866999290,1cyclictest0-21swapper/308:05:223
28663992913,8cyclictest0-21swapper/212:12:542
2865199291,13cyclictest3847-21basename12:20:010
2865199291,13cyclictest3847-21basename12:20:010
118552290,4sleep230-21rcuc/207:40:262
2866999286,19cyclictest6723-21cpuspeed_turbos08:45:133
2866999286,19cyclictest6723-21cpuspeed_turbos08:45:133
2866999285,20cyclictest9170-21needreboot07:35:183
2866999285,20cyclictest9170-21needreboot07:35:183
2866999282,22cyclictest6931-21missed_timers07:30:183
2866999281,3cyclictest10206-21/usr/sbin/munin10:05:183
2866999281,24cyclictest909-21sssd_nss08:25:193
2866999281,24cyclictest4403-21tail07:25:153
2866999281,24cyclictest2202-21/usr/sbin/munin08:35:153
2865999283,11cyclictest0-21swapper/110:22:281
28659992810,11cyclictest909-21sssd_nss12:20:251
2866999271,4cyclictest13639-21ntp_states07:45:213
2866999270,3cyclictest10993-21/usr/sbin/munin08:55:123
2865999277,11cyclictest0-21swapper/111:35:191
2865999275,16cyclictest22611-21kworker/u8:110:41:031
2865999261,17cyclictest0-21swapper/112:35:481
2865999261,17cyclictest0-21swapper/110:17:491
2865999261,15cyclictest4868-21timerwakeupswit11:05:261
2865999261,15cyclictest4868-21timerwakeupswit11:05:261
2865999261,11cyclictest0-21swapper/112:14:511
2865999261,11cyclictest0-21swapper/109:58:551
2865999261,11cyclictest0-21swapper/109:50:391
2865199261,10cyclictest24998-21kworker/0:111:58:130
2866399251,10cyclictest0-21swapper/209:17:092
2865999251,17cyclictest0-21swapper/109:31:101
2865999251,16cyclictest0-21swapper/111:15:471
2865999251,16cyclictest0-21swapper/110:56:431
2865999251,11cyclictest0-21swapper/111:14:111
2865999251,11cyclictest0-21swapper/110:25:091
28659992511,12cyclictest7462-21kworker/u8:111:30:291
2865199251,12cyclictest32131-21idleruntime-cro08:30:000
28651992510,4cyclictest5134-40dnf12:20:240
28663992411,8cyclictest0-21swapper/212:00:212
2866399241,11cyclictest0-21swapper/210:47:492
2865999248,5cyclictest28719-21tr09:35:121
2865999248,4cyclictest0-21swapper/110:09:301
2865999241,16cyclictest0-21swapper/109:27:331
2865999241,10cyclictest0-21swapper/112:09:531
2866399237,7cyclictest31791-21latency_hist12:10:012
2865999238,3cyclictest0-21swapper/109:16:351
2865999238,12cyclictest23597-21systemd-journal09:20:301
28659992317,3cyclictest16152-21sort07:50:241
2865999231,12cyclictest11484-21timerwakeupswit11:20:251
2865199231,11cyclictest15800-21perf10:20:010
111722231,3sleep111333-21df_inode08:55:141
2866399228,6cyclictest6512-21cstates12:25:142
2866399228,3cyclictest16806-21cstates11:35:132
2866399228,3cyclictest0-21swapper/207:46:302
2866399227,3cyclictest0-21swapper/208:02:302
2866399226,8cyclictest0-21swapper/207:51:502
2866399224,3cyclictest87350irq/126-enp1s0-09:50:142
2866399221,9cyclictest0-21swapper/210:35:172
28663992215,4cyclictest31542-21cat08:25:252
2866399221,13cyclictest0-21swapper/210:15:182
2866399221,13cyclictest0-21swapper/207:55:142
2866399221,11cyclictest0-21swapper/209:39:542
2865999228,3cyclictest27234-21taskset11:58:141
2865999226,3cyclictest0-21swapper/108:33:471
2865999225,9cyclictest0-21swapper/110:46:011
2865999225,4cyclictest0-21swapper/112:00:071
2865999224,8cyclictest32643-21kworker/u8:209:46:561
2865999222,11cyclictest17368-21munin-run09:10:011
2865999222,10cyclictest10155-21munin-run07:40:011
2865999222,10cyclictest10155-21munin-run07:40:011
2865999221,9cyclictest4036-21date12:20:021
2865999221,15cyclictest0-21swapper/109:14:081
2865999221,12cyclictest0-21swapper/112:27:161
2865999221,10cyclictest11789-21wc10:10:131
2865999221,10cyclictest0-21swapper/112:31:261
2865999221,10cyclictest0-21swapper/111:03:441
2865199221,3cyclictest0-21swapper/010:30:380
322082210,2sleep1221ktimersoftd/108:30:001
2866399219,3cyclictest0-21swapper/210:42:292
2866399218,3cyclictest0-21swapper/209:05:532
2866399216,9cyclictest0-21swapper/210:02:102
2866399211,12cyclictest1746-21cpuspeed_turbos07:20:132
2866399211,11cyclictest29668-21irqstats07:10:182
28663992110,2cyclictest28196-21kworker/u8:010:10:302
2865999211,18cyclictest0-21swapper/108:01:391
2865999211,17cyclictest24249-21sh08:10:161
2865999211,17cyclictest17374-21diskmemload10:50:111
2865999211,11cyclictest21084-21hddtemp_smartct10:30:171
2865999211,11cyclictest0-21swapper/110:04:011
2865999210,18cyclictest0-21swapper/111:25:221
2865199211,7cyclictest0-21swapper/011:08:300
2865199211,7cyclictest0-21swapper/011:08:300
177592210,3sleep011-21rcuc/009:10:140
11992211,3sleep11200-40dnf07:18:221
2866399207,3cyclictest0-21swapper/210:20:242
2866399204,8cyclictest0-21swapper/208:23:422
2866399203,8cyclictest0-21swapper/208:51:212
2866399201,1cyclictest87450irq/127-enp1s0-10:50:412
2866399201,11cyclictest0-21swapper/207:35:142
2866399201,11cyclictest0-21swapper/207:35:142
2866399201,10cyclictest0-21swapper/208:55:132
2866399200,1cyclictest0-21swapper/211:30:252
2866399200,14cyclictest0-21swapper/208:15:102
2866399200,12cyclictest0-21swapper/211:45:292
2865999208,3cyclictest4167-21sh08:40:001
2865999202,10cyclictest2642-21ntp_kernel_pll_07:20:221
2865999201,9cyclictest10818-21sed08:55:011
2865999201,9cyclictest0-21swapper/111:51:431
2865999201,9cyclictest0-21swapper/111:46:221
28659992017,2cyclictest0-21swapper/107:40:201
28659992016,3cyclictest0-21swapper/108:09:541
2865999201,13cyclictest17638-1kworker/u9:008:40:281
2865999201,10cyclictest27735-21latency_hist08:20:021
2865999201,10cyclictest17304-21df07:55:131
2865999201,10cyclictest0-21swapper/112:20:021
28659992010,6cyclictest7931-21systemd-journal07:34:531
2865999200,18cyclictest0-21swapper/107:10:551
2865199205,4cyclictest0-21swapper/010:35:140
2865199202,10cyclictest1793-21perl11:00:160
2865199202,10cyclictest11619-21/usr/sbin/munin10:10:180
2865199201,10cyclictest5021-21/usr/sbin/munin09:55:140
28651992010,6cyclictest0-21swapper/011:44:570
74002191,2sleep00-21swapper/011:12:270
2866399196,8cyclictest0-21swapper/208:48:132
2866399196,8cyclictest0-21swapper/208:48:132
2866399191,7cyclictest1-21systemd11:30:012
2866399191,3cyclictest0-21swapper/211:09:572
2866399191,3cyclictest0-21swapper/211:09:572
2866399191,11cyclictest0-21swapper/211:55:172
2866399190,12cyclictest0-21swapper/211:15:292
2865999195,3cyclictest0-21swapper/110:37:271
2865999193,9cyclictest0-21swapper/111:40:031
28659991915,3cyclictest0-21swapper/108:48:441
28659991915,3cyclictest0-21swapper/108:48:441
2865999191,10cyclictest28196-21kworker/u8:009:40:561
2865999191,10cyclictest0-21swapper/107:25:141
2865999190,17cyclictest0-21swapper/109:03:031
2865999190,17cyclictest0-21swapper/107:45:031
2865199196,3cyclictest26695-21kworker/u8:010:45:220
2865199195,4cyclictest766-21ntpd12:30:150
2865199192,6cyclictest0-21swapper/008:34:380
2865199192,5cyclictest17501-21/usr/sbin/munin10:20:250
28651991912,4cyclictest9395-21tail08:50:150
28651991912,4cyclictest18804-21gltestperf10:25:160
2865199191,11cyclictest8183-21sshd12:29:370
2865199191,11cyclictest14003-21sshd11:27:370
2865199191,10cyclictest808-21snmpd07:37:530
2865199191,10cyclictest808-21snmpd07:37:530
2865199191,10cyclictest0-21swapper/010:05:170
2865199191,10cyclictest0-21swapper/009:26:300
89002180,2sleep081ktimersoftd/011:15:230
2866399185,6cyclictest30782-21/usr/sbin/munin09:40:182
2866399185,4cyclictest24329-21taskset09:25:132
2866399185,3cyclictest0-21swapper/210:33:532
2866399183,3cyclictest0-21swapper/209:49:142
2866399181,4cyclictest1039-21tr08:30:222
2866399181,4cyclictest0-21swapper/210:25:272
2866399181,3cyclictest22123-21grep09:20:122
28663991812,3cyclictest4693-21head12:20:172
2866399180,12cyclictest0-21swapper/208:41:052
2865999181,16cyclictest0-21swapper/108:20:271
2865199186,2cyclictest8441-21kworker/0:208:55:260
2865199185,3cyclictest30259-21grep10:50:220
2865199184,3cyclictest18937-21crond08:00:010
2865199183,9cyclictest21698-40dnf09:19:260
2865199181,9cyclictest3311-21NetworkManager07:21:580
2865199181,7cyclictest0-21swapper/011:50:420
2865199181,6cyclictest0-21swapper/010:55:370
2865199181,5cyclictest541-21grep09:45:110
2865199181,4cyclictest0-21swapper/012:12:060
2865199181,4cyclictest0-21swapper/010:00:210
2865199181,4cyclictest0-21swapper/008:05:130
2865199181,2cyclictest0-21swapper/008:00:250
113432181,3sleep20-21swapper/210:09:362
2866399171,8cyclictest2547-21df_inode08:35:132
2866399171,6cyclictest0-21swapper/212:35:132
2866399171,6cyclictest0-21swapper/210:55:572
2866399171,3cyclictest0-21swapper/208:07:142
28663991712,3cyclictest20794-21dump-pmu-power11:45:012
2865199177,3cyclictest808-21snmpd12:37:370
2865199174,3cyclictest26017-21rt-features10:40:220
2865199173,7cyclictest30948-21cut09:40:110
2865199172,8cyclictest17374-21diskmemload12:06:110
2865199171,8cyclictest0-21swapper/009:05:290
2865199171,7cyclictest23571-21/usr/sbin/munin08:10:140
2865199171,4cyclictest0-21swapper/012:00:140
2865199171,4cyclictest0-21swapper/007:12:180
2865199171,3cyclictest7968-21awk07:35:000
2865199171,3cyclictest0-21swapper/011:20:130
2865199171,2cyclictest0-21swapper/009:35:110
28651991711,4cyclictest4372-21awk09:50:260
2865199170,5cyclictest0-21swapper/009:30:170
2865199170,15cyclictest0-21swapper/011:45:500
116562170,2sleep20-21swapper/211:20:382
2866399163,3cyclictest87350irq/126-enp1s0-07:30:192
2866399162,4cyclictest23919-21diskstats08:10:152
28663991611,3cyclictest18938-21timerandwakeup09:10:252
28663991611,3cyclictest13189-21/usr/sbin/munin09:00:182
2866399161,11cyclictest0-21swapper/207:25:092
2865199169,5cyclictest0-21swapper/007:50:120
2865199162,2cyclictest87350irq/126-enp1s0-08:15:120
2865199161,5cyclictest0-21swapper/007:40:160
2865199161,4cyclictest13385-21irqstats07:45:170
2865199161,4cyclictest0-21swapper/011:30:070
2865199161,3cyclictest0-21swapper/007:25:150
28651991611,3cyclictest2202-21/usr/sbin/munin08:35:130
2865199161,13cyclictest21990-21/usr/sbin/munin09:20:240
2865199161,12cyclictest4398-21/usr/sbin/munin08:40:150
28651991510,3cyclictest32490-21if_enp1s007:15:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional