You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-26 - 17:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue May 26, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
75172238161,24sleep00-21swapper/007:09:380
73692237163,23sleep30-21swapper/307:07:433
72852236161,25sleep20-21swapper/207:06:382
73192203166,24sleep10-21swapper/107:07:061
2494221730,5sleep2764599cyclictest09:00:162
1962121580,4sleep0763399cyclictest10:00:210
1474921520,4sleep0763399cyclictest12:15:260
76459910198,2cyclictest0-21swapper/211:05:012
7645999187,3cyclictest0-21swapper/208:09:512
7645997875,2cyclictest0-21swapper/210:14:202
67922520,1sleep20-21swapper/210:45:182
67922520,1sleep20-21swapper/210:45:182
232612520,4sleep011-21rcuc/012:35:230
295082510,1sleep00-21swapper/010:25:020
250572400,10sleep011-21rcuc/010:15:020
763899390,12cyclictest2842-21/usr/sbin/munin10:35:261
7633993913,12cyclictest18012-21grep11:10:200
765099381,31cyclictest0-21swapper/310:11:053
765099371,27cyclictest0-21swapper/308:32:163
763899371,12cyclictest0-21swapper/110:17:221
763899371,11cyclictest0-21swapper/112:37:501
763399378,14cyclictest27120-21munin-run10:20:020
765099360,28cyclictest0-21swapper/310:27:133
763899361,6cyclictest0-21swapper/112:02:061
163672360,1sleep10-21swapper/107:25:241
765099350,25cyclictest0-21swapper/312:21:413
763899351,9cyclictest0-21swapper/109:35:531
763899351,9cyclictest0-21swapper/109:35:531
763899351,13cyclictest0-21swapper/111:59:541
763899351,13cyclictest0-21swapper/111:59:541
7633993511,9cyclictest19102-21unixbench_multi12:25:230
7650993420,6cyclictest0-21swapper/311:30:143
765099341,26cyclictest0-21swapper/308:41:503
765099341,14cyclictest0-21swapper/307:59:333
763899341,9cyclictest0-21swapper/109:47:501
763899341,14cyclictest0-21swapper/111:03:421
7633993418,8cyclictest2042-21/usr/sbin/munin09:20:220
7633993414,17cyclictest9446-21grep12:05:170
7633993411,14cyclictest17925-21/usr/sbin/munin08:45:220
7650993320,8cyclictest0-21swapper/309:39:143
7650993320,8cyclictest0-21swapper/309:39:143
765099331,30cyclictest0-21swapper/308:50:163
763899331,11cyclictest0-21swapper/111:21:531
7633993311,13cyclictest6937-21/usr/sbin/munin08:20:140
7633993310,11cyclictest87550irq/128-enp1s0-11:05:200
765099321,24cyclictest0-21swapper/310:59:343
765099321,18cyclictest0-21swapper/311:04:383
765099321,17cyclictest0-21swapper/307:49:093
763899321,9cyclictest0-21swapper/110:33:501
763899321,9cyclictest0-21swapper/109:12:371
763899321,9cyclictest0-21swapper/107:45:301
763899321,14cyclictest0-21swapper/109:40:061
763899321,12cyclictest0-21swapper/111:35:091
763899321,12cyclictest0-21swapper/109:56:221
763899321,11cyclictest0-21swapper/110:02:221
763899321,11cyclictest0-21swapper/108:58:451
763899320,9cyclictest0-21swapper/110:05:281
763399327,10cyclictest26148-21kworker/0:109:12:210
7633993223,6cyclictest15770-21sed07:25:160
7633993214,5cyclictest87550irq/128-enp1s0-10:55:240
7633993214,5cyclictest2896-21basename11:50:170
7633993214,11cyclictest31738-21/usr/sbin/munin10:30:150
765099311,16cyclictest0-21swapper/309:22:023
765099311,15cyclictest31937-21grep08:00:253
765099311,10cyclictest0-21swapper/311:20:503
765099310,22cyclictest0-21swapper/309:59:573
763899311,13cyclictest0-21swapper/110:46:181
763899311,13cyclictest0-21swapper/110:46:181
763899311,10cyclictest0-21swapper/110:24:101
763399318,9cyclictest0-21swapper/009:45:230
763399310,16cyclictest8236-21hddtemp_smartct09:35:170
763399310,16cyclictest8236-21hddtemp_smartct09:35:170
152242310,5sleep230-21rcuc/207:25:122
765099308,3cyclictest0-21swapper/307:39:453
765099307,3cyclictest0-21swapper/308:11:253
765099301,14cyclictest0-21swapper/312:17:533
765099301,11cyclictest7755-21irqstats08:20:153
765099300,18cyclictest0-21swapper/309:07:183
765099300,12cyclictest0-21swapper/309:00:223
763899301,13cyclictest0-21swapper/109:54:581
763899301,12cyclictest0-21swapper/110:11:381
763899301,11cyclictest0-21swapper/111:42:371
763399308,12cyclictest0-21swapper/008:50:110
7633993021,6cyclictest25170-21smart_sda07:45:250
7633993020,7cyclictest2649-21ntp_states10:35:220
7633993013,4cyclictest87550irq/128-enp1s0-11:55:250
7633993013,4cyclictest87550irq/128-enp1s0-11:55:250
765099298,3cyclictest0-21swapper/308:09:013
7650992913,10cyclictest0-21swapper/310:08:413
765099291,21cyclictest0-21swapper/311:25:063
765099291,16cyclictest0-21swapper/309:40:263
765099291,10cyclictest0-21swapper/309:30:203
763899291,9cyclictest0-21swapper/112:29:581
763899291,12cyclictest0-21swapper/109:32:451
763899291,11cyclictest14270-21ntp_states12:15:221
763899291,11cyclictest0-21swapper/109:05:131
763899291,10cyclictest0-21swapper/112:22:581
763899291,10cyclictest0-21swapper/111:33:091
763399298,8cyclictest6757-21sed09:30:260
7633992920,5cyclictest3614-21ntp_states08:10:220
7633992914,12cyclictest10123-21cut09:40:140
7633992911,15cyclictest808-21snmpd07:52:060
7633992911,11cyclictest18245-1kworker/u9:211:22:130
765099287,3cyclictest0-21swapper/307:17:063
765099287,3cyclictest0-21swapper/307:17:053
7650992813,3cyclictest0-21swapper/308:27:063
7650992813,3cyclictest0-21swapper/308:27:053
765099281,13cyclictest0-21swapper/309:48:063
765099281,11cyclictest0-21swapper/312:01:213
765099280,18cyclictest0-21swapper/308:47:183
763899281,9cyclictest0-21swapper/111:07:441
763899281,9cyclictest0-21swapper/108:35:511
763899281,7cyclictest0-21swapper/111:47:371
763899281,7cyclictest0-21swapper/109:29:171
763899281,18cyclictest0-21swapper/111:26:171
763899281,12cyclictest0-21swapper/112:12:341
763899281,11cyclictest0-21swapper/111:19:251
763899281,11cyclictest0-21swapper/110:53:301
763399287,14cyclictest0-21swapper/008:29:180
763399287,14cyclictest0-21swapper/008:29:180
7633992820,6cyclictest32711-21/usr/sbin/munin08:05:280
7633992819,6cyclictest31753-21/usr/sbin/munin11:40:270
7633992819,6cyclictest3161-21/usr/sbin/munin09:25:230
7633992819,6cyclictest29543-21/usr/sbin/munin10:25:140
7633992819,6cyclictest26102-21/usr/sbin/munin11:30:130
7633992819,6cyclictest21705-21/usr/sbin/munin07:40:160
7633992819,6cyclictest19175-21/usr/sbin/munin11:15:160
7633992819,6cyclictest17101-21if_enp2s009:55:160
7633992819,6cyclictest11338-21/usr/sbin/munin08:30:210
7633992818,7cyclictest19790-21/usr/sbin/munin12:30:130
7633992816,9cyclictest19860-21/usr/sbin/munin07:35:140
7633992815,10cyclictest808-21snmpd07:22:000
7633992814,11cyclictest7934-21tr07:10:110
7633992812,6cyclictest29761-21taskset11:37:580
7633992812,5cyclictest0-21swapper/009:15:280
7633992811,7cyclictest8276-21cpuspeed_turbos10:50:120
7633992811,15cyclictest8582-21munin-run12:05:020
7633992810,15cyclictest10724-21/usr/sbin/munin07:15:120
7633992810,15cyclictest10724-21/usr/sbin/munin07:15:110
765099279,3cyclictest0-21swapper/312:32:373
765099271,11cyclictest0-21swapper/312:11:093
765099271,11cyclictest0-21swapper/312:08:573
765099271,11cyclictest0-21swapper/309:29:263
7650992711,10cyclictest0-21swapper/307:44:533
7650992710,3cyclictest29267-21df09:10:143
765099270,13cyclictest0-21swapper/308:35:103
763899271,8cyclictest0-21swapper/109:24:411
763399277,11cyclictest0-21swapper/008:55:120
7633992720,5cyclictest5971-21/usr/sbin/munin10:45:110
7633992720,5cyclictest5971-21/usr/sbin/munin10:45:110
7633992720,4cyclictest5429-21/usr/sbin/munin08:15:150
7633992720,4cyclictest18129-21/usr/sbin/munin07:30:180
7633992719,6cyclictest3774-21/usr/sbin/munin10:40:150
7633992719,5cyclictest32355-21/usr/sbin/munin11:45:140
7633992719,5cyclictest20756-21/usr/sbin/munin10:05:170
7633992719,5cyclictest14853-21/usr/sbin/munin09:50:160
7633992718,6cyclictest28300-21/usr/sbin/munin07:55:200
7633992718,6cyclictest24526-21/usr/sbin/munin09:00:120
7633992718,6cyclictest15727-21/usr/sbin/munin08:40:170
7633992718,6cyclictest15394-21/usr/sbin/munin12:20:260
7633992718,6cyclictest13535-21/usr/sbin/munin08:35:200
7633992718,6cyclictest12571-21/usr/sbin/munin11:00:170
7633992712,6cyclictest23892-21cat11:25:130
765099269,2cyclictest0-21swapper/310:48:173
765099269,2cyclictest0-21swapper/310:48:173
765099268,3cyclictest0-21swapper/311:18:343
765099268,3cyclictest0-21swapper/308:56:503
765099261,10cyclictest28702-21diskmemload10:44:533
765099260,18cyclictest4107-21kworker/u8:011:55:053
765099260,18cyclictest4107-21kworker/u8:011:55:053
765099260,13cyclictest0-21swapper/311:05:423
763899266,3cyclictest0-21swapper/109:15:211
763899262,3cyclictest0-21swapper/107:16:111
763899262,3cyclictest0-21swapper/107:16:111
7638992623,2cyclictest0-21swapper/107:13:531
7638992612,3cyclictest9036-21cat08:25:021
763899261,22cyclictest32487-21munin-run08:05:011
763899261,12cyclictest0-21swapper/110:42:181
763899261,10cyclictest26836-21kworker/1:007:24:261
763899261,10cyclictest26355-21taskset09:03:151
7633992619,4cyclictest11347-21/usr/sbin/munin12:10:150
7633992618,6cyclictest30501-21/usr/sbin/munin08:00:160
7633992618,6cyclictest27782-21ntpq09:05:180
765099259,3cyclictest0-21swapper/307:31:013
765099258,3cyclictest0-21swapper/307:28:253
7650992512,5cyclictest0-21swapper/310:51:063
7650992512,2cyclictest195512sleep312:30:013
765099251,13cyclictest0-21swapper/310:36:373
765099251,11cyclictest0-21swapper/309:16:423
765099251,10cyclictest0-21swapper/310:15:093
765099250,9cyclictest0-21swapper/311:44:383
765099250,11cyclictest0-21swapper/307:22:133
763899258,3cyclictest9479-21df08:25:141
763899258,3cyclictest9479-21df08:25:141
763899254,3cyclictest0-21swapper/108:15:401
763899251,21cyclictest9336-21/usr/sbin/munin12:05:171
763899251,21cyclictest13312-21munin-run08:35:011
763899251,18cyclictest0-21swapper/111:14:231
763899251,11cyclictest0-21swapper/110:26:221
7638992510,3cyclictest19848-21df07:35:141
765099248,3cyclictest0-21swapper/307:12:263
765099247,3cyclictest0-21swapper/310:23:413
7650992411,5cyclictest0-21swapper/311:47:273
7650992411,4cyclictest0-21swapper/307:51:413
763899247,3cyclictest0-21swapper/108:48:231
763899246,3cyclictest0-21swapper/107:32:221
763899245,3cyclictest0-21swapper/108:10:061
763899241,13cyclictest0-21swapper/112:31:141
763899241,11cyclictest28641-21df_inode07:55:131
763899241,10cyclictest0-21swapper/107:52:561
765099239,3cyclictest0-21swapper/308:19:173
765099238,9cyclictest0-21swapper/309:52:573
7650992313,3cyclictest0-21swapper/311:37:023
765099231,15cyclictest0-21swapper/310:03:333
765099231,13cyclictest0-21swapper/312:38:013
765099231,11cyclictest0-21swapper/311:51:383
764599231,11cyclictest32076-21runrttasks11:44:292
763899231,14cyclictest0-21swapper/110:58:421
763899231,11cyclictest0-21swapper/111:53:091
763899231,10cyclictest0-21swapper/107:42:161
763899230,15cyclictest0-21swapper/108:42:301
765099221,8cyclictest18510-21tail11:10:263
765099220,13cyclictest0-21swapper/310:32:223
7645992115,3cyclictest4986-21grep11:55:142
7645992115,3cyclictest4986-21grep11:55:142
764599207,4cyclictest18271-21cat11:10:252
764599202,14cyclictest24046-21tr07:45:132
764599201,4cyclictest0-21swapper/212:32:432
764599201,10cyclictest8787-21sort07:10:192
763899208,3cyclictest0-21swapper/108:50:071
763899201,10cyclictest0-21swapper/108:05:291
764599199,3cyclictest30-21rcuc/210:00:212
764599195,5cyclictest19486-21sed08:45:262
764599194,3cyclictest4419-21sensors_temp09:25:232
764599191,8cyclictest1-21systemd08:00:022
764599191,6cyclictest0-21swapper/209:44:092
764599191,5cyclictest18245-1kworker/u9:210:41:392
7645991914,3cyclictest20642-21grep11:15:262
764599191,10cyclictest18245-1kworker/u9:209:10:262
764599191,10cyclictest12922-21/usr/sbin/munin07:20:202
120472191,2sleep20-21swapper/212:10:222
93492181,2sleep20-21swapper/209:37:522
93492181,2sleep20-21swapper/209:37:522
764599188,3cyclictest87350irq/126-enp1s0-09:50:102
764599184,5cyclictest32-21ksoftirqd/211:05:472
764599184,5cyclictest30-21rcuc/212:00:252
764599183,6cyclictest30-21rcuc/207:35:232
764599182,6cyclictest29555-21users11:35:262
764599182,6cyclictest14206-21hddtemp_smartct08:35:182
764599181,9cyclictest12159-21ls09:45:112
764599181,7cyclictest25534-21latency_hist11:30:012
764599181,7cyclictest0-21swapper/212:36:582
764599181,6cyclictest0-21swapper/212:09:122
764599181,5cyclictest0-21swapper/209:59:592
764599181,4cyclictest8534-21users08:20:242
7645991812,3cyclictest2991-21sort10:35:272
764599181,11cyclictest0-21swapper/210:55:542
764599181,10cyclictest0-21swapper/210:25:052
275342181,2sleep20-21swapper/211:32:342
764599177,3cyclictest87550irq/128-enp1s0-09:16:322
764599173,2cyclictest87550irq/128-enp1s0-07:54:462
764599172,5cyclictest774-21avahi-daemon07:35:012
764599172,2cyclictest87550irq/128-enp1s0-10:31:462
764599171,7cyclictest17708-21awk08:45:022
764599171,7cyclictest0-21swapper/208:10:132
764599171,6cyclictest0-21swapper/211:49:092
764599171,6cyclictest0-21swapper/210:08:162
764599171,5cyclictest0-21swapper/210:24:072
764599171,4cyclictest18719-21grep12:25:202
764599171,4cyclictest0-21swapper/209:20:352
764599171,3cyclictest0-21swapper/210:53:392
7645991712,3cyclictest26729-21/usr/sbin/munin09:05:232
764599171,14cyclictest10611-21grep08:25:262
764599171,14cyclictest10611-21grep08:25:262
267122170,5sleep20-21swapper/210:15:232
764599162,5cyclictest22320-21/usr/sbin/munin08:55:192
764599161,9cyclictest0-21swapper/212:17:002
764599161,3cyclictest0-21swapper/211:21:592
764599161,2cyclictest0-21swapper/208:16:212
7645991612,3cyclictest0-21swapper/209:31:012
7645991611,3cyclictest827-21runrttasks08:02:072
7645991611,3cyclictest21769-21cpuspeed_turbos07:40:102
7645991611,3cyclictest21192-21ntp_states08:50:212
7645991610,3cyclictest827-21runrttasks08:31:402
764599154,3cyclictest87350irq/126-enp1s0-07:15:032
764599154,3cyclictest87350irq/126-enp1s0-07:15:022
7645991510,3cyclictest2989-21cat11:50:172
7645991510,3cyclictest15459-21cpuspeed_turbos12:20:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional