You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-26 - 06:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue May 26, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21002233157,24sleep00-21swapper/019:05:470
24002223193,20sleep30-21swapper/319:09:393
21532202165,25sleep10-21swapper/119:06:281
21072200164,24sleep20-21swapper/219:05:522
206821490,4sleep1251399cyclictest21:33:191
2519999794,2cyclictest0-21swapper/222:00:522
2519999288,3cyclictest0-21swapper/222:39:472
2519997977,1cyclictest0-21swapper/222:57:022
2519997168,2cyclictest0-21swapper/222:27:502
2519996461,2cyclictest0-21swapper/221:13:392
2519995344,8cyclictest0-21swapper/220:24:522
305852490,1sleep20-21swapper/221:25:142
252699432,9cyclictest10006-21diskstats23:05:133
252699381,10cyclictest18631-21sshd00:37:173
2504993810,13cyclictest27024-21awk23:45:000
2504993711,12cyclictest28593-21munin-run22:35:010
252699361,11cyclictest4639-21/usr/sbin/munin21:40:263
2526993513,19cyclictest3693-21/usr/sbin/munin00:05:183
251999351,26cyclictest0-21swapper/223:26:452
252699348,3cyclictest20646-21/usr/sbin/munin23:30:223
252699348,3cyclictest20646-21/usr/sbin/munin23:30:223
252699347,3cyclictest808-21snmpd00:15:373
2526993410,3cyclictest18798-21users20:55:263
70402330,5sleep2251999cyclictest00:10:212
252699336,3cyclictest17833-21/usr/sbin/munin22:10:193
252699331,14cyclictest0-21swapper/320:40:213
252699331,14cyclictest0-21swapper/319:52:563
2526993310,3cyclictest23807-21/usr/sbin/munin21:10:143
252699329,2cyclictest827-21runrttasks00:32:013
252699328,3cyclictest18450-21/usr/sbin/munin23:25:233
252699328,3cyclictest1459-21/usr/sbin/munin00:00:253
252699327,4cyclictest17654-21/usr/sbin/munin23:20:253
252699325,3cyclictest87350irq/126-enp1s0-22:30:113
252699321,6cyclictest11972-21munin-run19:30:023
252699321,5cyclictest0-21swapper/322:27:333
252699321,19cyclictest0-21swapper/322:19:453
2526993211,3cyclictest3983-21unixbench_singl21:35:263
252699321,12cyclictest27294-21sshd20:04:163
2526993210,3cyclictest27455-21expr23:45:133
252699319,3cyclictest9034-21/usr/sbin/munin21:50:173
252699319,3cyclictest827-21runrttasks00:23:103
252699318,3cyclictest12849-21sed19:30:153
252699316,3cyclictest87350irq/126-enp1s0-23:00:013
252699313,3cyclictest87650irq/129-enp1s0-22:20:133
252699313,3cyclictest87650irq/129-enp1s0-22:20:133
252699311,5cyclictest344-21processes23:55:243
2526993111,3cyclictest7271-21timerandwakeup00:10:233
2526993111,3cyclictest6533-21expr20:30:133
252699311,13cyclictest29946-21pmu-power22:35:203
252699309,3cyclictest827-21runrttasks00:29:083
252699308,4cyclictest0-21swapper/319:15:163
252699308,3cyclictest827-21runrttasks21:33:183
252699302,10cyclictest3973-21memory22:50:183
252699301,6cyclictest27973-21latency_hist21:20:023
252699301,5cyclictest0-21swapper/320:25:263
252699301,14cyclictest0-21swapper/321:20:213
2526993011,3cyclictest8761-21cat19:20:193
252699301,10cyclictest0-21swapper/322:00:003
2526993010,3cyclictest827-21runrttasks23:38:553
2526993010,3cyclictest3039-21date22:50:013
2526993010,3cyclictest21753-21cut21:05:133
252699300,10cyclictest0-21swapper/320:54:253
251999301,12cyclictest0-21swapper/220:14:432
251999300,2cyclictest0-21swapper/219:52:522
251999300,23cyclictest0-21swapper/221:19:072
251999300,23cyclictest0-21swapper/200:36:442
251999300,18cyclictest0-21swapper/223:53:392
250499301,14cyclictest6615-21munin-run21:45:020
252699299,3cyclictest31196-21cstates22:40:123
252699299,2cyclictest827-21runrttasks20:39:043
252699298,3cyclictest827-21runrttasks23:17:323
252699298,3cyclictest2758-21missed_timers20:20:193
252699293,6cyclictest0-21swapper/323:02:403
252699291,4cyclictest0-21swapper/323:44:183
2526992910,3cyclictest827-21runrttasks19:11:473
2526992910,3cyclictest7752-21load21:45:193
2526992910,3cyclictest13526-21sort22:00:123
2526992910,2cyclictest827-21runrttasks19:46:073
251999291,19cyclictest0-21swapper/223:19:272
251999291,16cyclictest0-21swapper/220:36:192
251999290,24cyclictest0-21swapper/219:40:282
252699289,3cyclictest1708-21cut20:20:013
252699284,8cyclictest87350irq/126-enp1s0-23:13:173
252699281,7cyclictest19407-21/usr/sbin/munin21:00:163
252699281,5cyclictest27589-21/usr/sbin/munin20:05:153
252699281,5cyclictest27589-21/usr/sbin/munin20:05:153
252699281,3cyclictest0-21swapper/322:06:373
251999281,18cyclictest0-21swapper/219:24:372
251999281,11cyclictest0-21swapper/222:42:452
251999281,10cyclictest0-21swapper/221:34:312
251999280,13cyclictest0-21swapper/222:20:172
251999280,13cyclictest0-21swapper/222:20:172
251999280,10cyclictest0-21swapper/223:24:372
251399281,8cyclictest1118-21kworker/u8:000:05:011
251399281,17cyclictest0-21swapper/100:22:171
250499280,13cyclictest31418-21awk23:55:020
252699278,3cyclictest25307-21date20:00:003
252699271,10cyclictest0-21swapper/323:53:393
2526992710,2cyclictest401ktimersoftd/321:25:203
251999271,9cyclictest27390-21sshd22:30:172
251999271,7cyclictest0-21swapper/219:58:452
251999271,13cyclictest0-21swapper/219:49:582
251999271,12cyclictest0-21swapper/222:10:122
251999271,12cyclictest0-21swapper/221:05:092
251999271,11cyclictest21551-21cat21:05:012
251999270,12cyclictest0-21swapper/220:27:312
2513992713,7cyclictest827-21runrttasks20:52:201
251399271,20cyclictest0-21swapper/122:48:141
251399271,17cyclictest0-21swapper/123:38:151
251399271,16cyclictest0-21swapper/123:24:371
252699261,6cyclictest16587-21/usr/sbin/munin19:40:213
252699261,5cyclictest14175-21perf19:39:573
251999267,8cyclictest0-21swapper/200:02:512
251999261,9cyclictest0-21swapper/200:27:342
251999261,15cyclictest24423-21uptime23:35:252
251999261,10cyclictest0-21swapper/220:57:152
251999261,10cyclictest0-21swapper/219:14:232
251999260,10cyclictest0-21swapper/223:55:132
2513992615,3cyclictest13490-21needreboot00:25:211
250499261,12cyclictest25795-21awk21:15:010
251999251,9cyclictest0-21swapper/220:30:432
251999251,17cyclictest0-21swapper/200:32:042
251999251,11cyclictest0-21swapper/220:52:432
251999251,10cyclictest0-21swapper/220:08:352
251999251,10cyclictest0-21swapper/220:08:352
251999250,9cyclictest0-21swapper/223:12:312
251999250,9cyclictest0-21swapper/223:05:172
251999250,13cyclictest0-21swapper/220:05:012
251999250,12cyclictest0-21swapper/220:41:592
251999250,11cyclictest0-21swapper/220:16:512
251999250,10cyclictest0-21swapper/222:45:272
251999250,10cyclictest0-21swapper/221:41:352
251999250,10cyclictest0-21swapper/221:22:172
251399258,10cyclictest0-21swapper/122:42:321
251399254,11cyclictest0-21swapper/121:45:011
251399253,3cyclictest4398-21sshd00:05:161
251399251,16cyclictest4395-21kworker/u8:021:46:221
250499251,12cyclictest5035-21munin-run22:55:020
208842250,4sleep011-21rcuc/019:50:010
251999241,9cyclictest0-21swapper/219:35:262
251999241,10cyclictest0-21swapper/219:27:382
251999240,9cyclictest0-21swapper/222:16:282
251999240,12cyclictest0-21swapper/200:08:052
251999240,11cyclictest0-21swapper/221:45:202
251399247,10cyclictest0-21swapper/122:06:351
2513992412,10cyclictest23-21ksoftirqd/122:00:011
251399241,16cyclictest0-21swapper/122:10:041
251399241,11cyclictest20375-21df_inode22:15:131
250499245,16cyclictest24817-21munin-run23:40:000
250499241,10cyclictest0-21swapper/020:27:480
140752240,4sleep011-21rcuc/023:15:120
251999238,9cyclictest0-21swapper/221:52:482
251999235,3cyclictest0-21swapper/222:53:112
251999231,9cyclictest0-21swapper/219:19:312
251999231,11cyclictest0-21swapper/221:57:062
251999230,9cyclictest0-21swapper/223:44:232
251399232,18cyclictest4690-21munin-run19:15:011
2513992317,3cyclictest26619-21find20:00:241
251399231,16cyclictest0-21swapper/121:19:381
251399231,16cyclictest0-21swapper/120:10:141
251399231,16cyclictest0-21swapper/119:32:101
251399231,16cyclictest0-21swapper/100:18:541
251399231,16cyclictest0-21swapper/100:12:401
251399231,15cyclictest0-21swapper/123:55:241
2513992310,3cyclictest0-21swapper/120:41:571
251399230,17cyclictest0-21swapper/122:35:041
250499235,5cyclictest3588-21latency_hist00:05:020
251999228,7cyclictest0-21swapper/219:31:482
251999220,9cyclictest0-21swapper/222:09:202
251999220,9cyclictest0-21swapper/220:45:052
251399221,14cyclictest0-21swapper/119:50:201
251399221,11cyclictest0-21swapper/122:29:501
251399220,16cyclictest0-21swapper/119:37:351
252699216,3cyclictest0-21swapper/320:11:003
251999214,3cyclictest0-21swapper/223:46:412
251399218,11cyclictest1118-21kworker/u8:000:34:481
2513992117,3cyclictest0-21swapper/123:00:261
251399211,3cyclictest0-21swapper/123:05:051
251399211,3cyclictest0-21swapper/119:27:001
2513992113,3cyclictest0-21swapper/120:15:591
251399211,17cyclictest14788-21sleep20:50:011
251399211,15cyclictest0-21swapper/119:41:471
251399211,11cyclictest30616-21processes23:50:211
251399211,10cyclictest0-21swapper/123:25:511
251399211,10cyclictest0-21swapper/121:21:491
2513992110,3cyclictest19575-21if_err_enp2s019:45:171
251399210,13cyclictest5603-21diskstats22:55:131
250499216,4cyclictest26122-21tr21:15:130
250499216,3cyclictest0-21swapper/021:46:570
250499211,6cyclictest0-21swapper/022:27:200
250499211,6cyclictest0-21swapper/021:24:580
250499211,6cyclictest0-21swapper/019:50:250
250499211,11cyclictest0-21swapper/023:14:530
250499211,10cyclictest18472-21smart_sda20:55:230
164702210,1sleep00-21swapper/022:05:180
251999208,3cyclictest0-21swapper/221:36:572
251999206,2cyclictest0-21swapper/223:02:372
251999200,13cyclictest0-21swapper/223:32:492
251999200,13cyclictest0-21swapper/223:32:492
251399202,8cyclictest31913-21unixbench_multi21:25:261
251399201,9cyclictest0-21swapper/122:20:501
251399201,9cyclictest0-21swapper/122:20:501
251399201,6cyclictest0-21swapper/123:41:301
2513992016,3cyclictest0-21swapper/123:31:571
2513992016,3cyclictest0-21swapper/123:31:571
2513992014,3cyclictest10101-21ntp_states21:50:211
251399201,2cyclictest0-21swapper/120:30:251
251399201,17cyclictest0-21swapper/121:00:111
251399201,10cyclictest4840-21irqstats20:25:201
251399201,10cyclictest23579-21diskmemload00:37:501
251399201,10cyclictest0-21swapper/122:33:371
2513992010,3cyclictest25149-21latency_hist20:00:011
251399200,10cyclictest0-21swapper/122:50:071
250499201,11cyclictest808-21snmpd22:10:130
53472191,3sleep00-21swapper/000:06:060
251999191,2cyclictest99602sleep200:18:462
251399196,4cyclictest0-21swapper/121:36:361
251399192,11cyclictest0-21swapper/123:15:171
2513991913,2cyclictest0-21swapper/120:06:081
2513991913,2cyclictest0-21swapper/120:06:081
2513991912,6cyclictest8442-21kworker/u8:122:04:481
251399191,17cyclictest0-21swapper/123:46:381
251399191,17cyclictest0-21swapper/123:13:331
251399191,14cyclictest0-21swapper/119:15:361
251399191,10cyclictest9198-21cat20:35:181
251399191,10cyclictest18056-21latency20:55:181
251399190,6cyclictest0-21swapper/121:11:511
250499199,3cyclictest87550irq/128-enp1s0-00:20:260
250499198,3cyclictest13058-21sshd21:58:170
250499195,3cyclictest827-21runrttasks00:37:500
250499192,5cyclictest2211-21munin-run21:35:010
2504991912,4cyclictest29880-21grep20:10:130
250499191,10cyclictest5609-21/usr/sbin/munin19:15:200
250499191,10cyclictest21931-21sshd22:19:170
250499191,10cyclictest16250-21/usr/sbin/munin23:20:250
250499190,6cyclictest0-21swapper/023:45:230
206572191,15sleep020713-21switchtime21:00:240
93512181,3sleep00-21swapper/000:15:250
80712180,1sleep00-21swapper/023:00:160
4182181,2sleep00-21swapper/022:43:510
2519991812,3cyclictest10293-21/usr/sbin/munin00:20:252
251399188,7cyclictest22032-21fschecks_count21:05:151
251399180,16cyclictest0-21swapper/120:23:431
250499185,3cyclictest11169-21grep23:05:260
250499182,5cyclictest0-21swapper/022:59:490
250499181,9cyclictest0-21swapper/020:35:510
250499181,6cyclictest0-21swapper/021:25:360
250499181,3cyclictest4691-21awk19:15:010
250499181,14cyclictest6894-21grep00:10:180
250499181,11cyclictest0-21swapper/020:45:170
250499181,10cyclictest829-21cpu22:45:130
237392181,2sleep00-21swapper/019:55:160
153742181,2sleep00-21swapper/022:04:420
252699171,12cyclictest0-21swapper/320:47:173
251399171,3cyclictest9392-21users19:20:241
250499178,2cyclictest808-21snmpd20:34:550
250499178,2cyclictest10000-21/usr/sbin/munin19:25:270
250499171,8cyclictest0-21swapper/023:30:130
250499171,8cyclictest0-21swapper/023:30:130
250499171,6cyclictest26453-21ntp_states20:00:230
250499171,6cyclictest23661-21cat21:10:010
250499171,6cyclictest0-21swapper/000:31:060
2504991713,2cyclictest14467-21awk00:30:010
250499171,15cyclictest0-21swapper/020:50:550
2504991711,4cyclictest8884-21awk19:20:210
250499171,14cyclictest0-21swapper/019:41:080
2504991710,5cyclictest12371-21systemd-journal20:42:240
250499170,7cyclictest87550irq/128-enp1s0-22:35:120
250499165,2cyclictest3155-21idleruntime21:35:170
250499163,3cyclictest87550irq/128-enp1s0-23:55:170
250499161,7cyclictest808-21snmpd20:09:500
250499161,7cyclictest808-21snmpd20:09:500
250499161,6cyclictest22969-21taskset22:20:180
250499161,6cyclictest22969-21taskset22:20:180
250499161,14cyclictest0-21swapper/021:50:500
2504991611,3cyclictest2130-21expr20:20:130
250499161,11cyclictest0-21swapper/019:39:570
250499155,7cyclictest18601-21cpuspeed_turbos23:25:120
2504991510,3cyclictest353-21cut20:15:190
250499149,3cyclictest13176-21needreboot19:30:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional