You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-18 - 15:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Apr 18, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
243862237161,24sleep30-21swapper/307:09:133
242712236162,24sleep20-21swapper/207:07:432
2363122230,2sleep1221ktimersoftd/107:05:231
243112200162,26sleep00-21swapper/007:08:130
1047521670,5sleep02453699cyclictest10:15:170
2198521330,2sleep081ktimersoftd/008:15:140
332921280,4sleep02453699cyclictest10:00:120
579021220,1sleep10-21swapper/107:37:081
24549998380,2cyclictest0-21swapper/208:31:052
27652520,2sleep00-21swapper/007:30:190
24549995149,1cyclictest0-21swapper/209:35:262
24549994744,2cyclictest0-21swapper/210:46:152
26382460,1sleep20-21swapper/211:10:212
295972440,1sleep20-21swapper/209:45:272
2454299431,29cyclictest0-21swapper/110:03:011
216972420,2sleep10-21swapper/110:40:181
2454299362,13cyclictest0-21swapper/110:14:011
2454299362,13cyclictest0-21swapper/110:14:011
24536993621,2cyclictest32474-21kworker/0:111:23:440
24536993610,12cyclictest29256-21perf11:00:010
2454999330,12cyclictest0-21swapper/209:16:352
2454299331,8cyclictest0-21swapper/108:41:151
2454299331,27cyclictest0-21swapper/110:56:211
24536993323,2cyclictest8038-21kworker/0:007:52:290
2454299321,13cyclictest18837-21/usr/sbin/munin10:35:151
2454999311,23cyclictest0-21swapper/211:59:392
2454999311,22cyclictest0-21swapper/210:58:572
2454999310,23cyclictest0-21swapper/211:08:532
2454299311,14cyclictest0-21swapper/112:38:021
2454299311,14cyclictest0-21swapper/112:38:021
2454299311,13cyclictest0-21swapper/112:31:181
2454299311,13cyclictest0-21swapper/108:35:511
2454299311,12cyclictest0-21swapper/111:02:091
2454299311,12cyclictest0-21swapper/110:46:291
2454299311,12cyclictest0-21swapper/110:20:171
24536993122,2cyclictest3862-21kworker/0:012:30:590
2454999301,21cyclictest0-21swapper/208:57:032
2454999301,21cyclictest0-21swapper/208:57:032
2454999301,12cyclictest18257-21perf10:35:012
2454999300,22cyclictest0-21swapper/212:20:462
2454999300,22cyclictest0-21swapper/211:35:212
2454999300,12cyclictest0-21swapper/210:06:212
2454299301,19cyclictest0-21swapper/108:21:511
2454299301,14cyclictest0-21swapper/111:21:251
2454299301,13cyclictest0-21swapper/109:16:081
2454299301,13cyclictest0-21swapper/108:46:231
2454299301,12cyclictest0-21swapper/108:53:391
2454299301,12cyclictest0-21swapper/108:28:311
2454299301,11cyclictest0-21swapper/112:27:061
2454299301,11cyclictest0-21swapper/112:09:061
2454299301,11cyclictest0-21swapper/109:13:361
2454299301,10cyclictest10924-21chrt09:03:361
2454299300,13cyclictest0-21swapper/108:10:231
24536993020,2cyclictest23263-21kworker/0:208:25:250
24536993019,2cyclictest24503-21kworker/0:107:14:410
2454999291,19cyclictest0-21swapper/209:56:122
2454999291,11cyclictest22980-21threads08:15:252
2454999291,10cyclictest0-21swapper/209:27:522
2454999291,10cyclictest0-21swapper/207:54:222
2454999290,23cyclictest0-21swapper/210:53:212
2454999290,11cyclictest0-21swapper/207:55:162
2454299291,13cyclictest0-21swapper/110:17:371
2454299291,13cyclictest0-21swapper/109:09:121
2454299291,12cyclictest0-21swapper/110:53:131
2454299291,12cyclictest0-21swapper/107:55:551
2454299291,12cyclictest0-21swapper/107:48:351
2454299291,12cyclictest0-21swapper/107:48:351
2454299291,11cyclictest0-21swapper/112:16:021
2454299291,11cyclictest0-21swapper/112:00:181
2454299291,11cyclictest0-21swapper/111:38:411
2454299291,11cyclictest0-21swapper/109:46:091
2454299291,11cyclictest0-21swapper/108:00:231
2454299291,10cyclictest0-21swapper/111:47:421
2453699296,11cyclictest29887-21perf12:15:000
2453699292,20cyclictest8038-21kworker/0:008:13:230
2454999281,9cyclictest0-21swapper/209:11:072
2454999281,2cyclictest0-21swapper/207:38:302
2454999281,12cyclictest0-21swapper/207:42:322
2454299281,14cyclictest0-21swapper/110:08:211
2454299281,13cyclictest0-21swapper/110:29:291
2454299281,12cyclictest0-21swapper/111:10:411
2454299281,11cyclictest30738-21memory07:20:201
2454299281,11cyclictest0-21swapper/111:08:571
2454299281,11cyclictest0-21swapper/109:51:371
2454299281,10cyclictest0-21swapper/111:58:501
2454299281,10cyclictest0-21swapper/109:36:021
2454299281,10cyclictest0-21swapper/109:28:581
2454299281,10cyclictest0-21swapper/107:14:151
2454299280,12cyclictest0-21swapper/109:57:451
2454299280,11cyclictest0-21swapper/109:41:051
24536992817,6cyclictest10865-21kworker/0:109:03:420
24536992817,5cyclictest13514-21kworker/0:110:21:230
2454999271,8cyclictest0-21swapper/211:32:112
2454999271,13cyclictest8799-21diskstats07:45:162
2454999271,13cyclictest8799-21diskstats07:45:162
2454999271,10cyclictest0-21swapper/209:32:082
2454999271,10cyclictest0-21swapper/208:52:052
2454999270,18cyclictest0-21swapper/212:31:162
2454999270,18cyclictest0-21swapper/209:00:592
2454299271,13cyclictest0-21swapper/111:33:331
2454299271,11cyclictest0-21swapper/112:23:301
2454299271,11cyclictest0-21swapper/110:34:171
2454299270,13cyclictest0-21swapper/109:33:301
2453699272,9cyclictest13076-21kworker/0:009:33:390
2453699272,11cyclictest28290-21kworker/0:009:55:470
2454999261,9cyclictest0-21swapper/212:00:092
2454999261,9cyclictest0-21swapper/208:20:132
2454999261,8cyclictest0-21swapper/210:10:042
2454999261,8cyclictest0-21swapper/210:10:042
2454999261,18cyclictest0-21swapper/207:21:552
2454999261,16cyclictest0-21swapper/211:42:552
2454999261,12cyclictest0-21swapper/212:16:492
2454999261,10cyclictest0-21swapper/209:50:182
2454999261,10cyclictest0-21swapper/208:03:462
24549992610,3cyclictest0-21swapper/208:26:432
2454999260,21cyclictest0-21swapper/212:37:022
2454999260,21cyclictest0-21swapper/212:37:022
2454999260,21cyclictest0-21swapper/211:19:212
2454999260,19cyclictest0-21swapper/210:03:262
2454999260,18cyclictest0-21swapper/210:36:112
2454999260,11cyclictest0-21swapper/211:49:572
2454999260,10cyclictest25705-21/usr/sbin/munin12:05:132
2454999260,10cyclictest0-21swapper/211:20:032
2454999260,10cyclictest0-21swapper/210:19:202
2454299261,11cyclictest0-21swapper/111:50:101
2454299261,11cyclictest0-21swapper/111:16:091
2454299261,10cyclictest0-21swapper/107:52:511
2453699267,13cyclictest89150irq/134-enp2s0-09:27:130
2453699266,10cyclictest89150irq/134-enp2s0-08:06:410
24536992617,2cyclictest5632-21kworker/0:110:08:470
24536992616,2cyclictest25567-21kworker/0:108:43:100
24536992615,2cyclictest25567-21kworker/0:108:30:220
24536992611,5cyclictest28290-21kworker/0:009:45:530
2454999251,9cyclictest0-21swapper/210:25:192
2454999251,8cyclictest21019-21diskstats10:40:132
2454999251,12cyclictest0-21swapper/210:22:092
2454999251,11cyclictest0-21swapper/207:32:422
2454999251,11cyclictest0-21swapper/207:12:232
2454999250,12cyclictest0-21swapper/212:29:422
2453699252,9cyclictest25567-21kworker/0:108:22:490
2453699251,9cyclictest6328-21kworker/0:008:51:200
24536992516,2cyclictest32474-21kworker/0:111:09:440
2453699251,10cyclictest10865-21kworker/0:109:12:170
2454999241,9cyclictest0-21swapper/208:45:472
2454999241,17cyclictest0-21swapper/211:53:452
2454999241,11cyclictest0-21swapper/208:15:002
2454999240,9cyclictest0-21swapper/212:10:412
2454999240,8cyclictest0-21swapper/209:41:482
2454999240,11cyclictest0-21swapper/207:25:502
2454299246,4cyclictest0-21swapper/109:21:441
2454299241,9cyclictest0-21swapper/107:18:511
2454299241,13cyclictest0-21swapper/111:26:251
2454299241,10cyclictest0-21swapper/107:42:281
2453699246,12cyclictest87650irq/129-enp1s0-10:13:090
2453699246,12cyclictest87650irq/129-enp1s0-10:13:090
2453699241,8cyclictest24716-21kworker/0:110:52:040
2453699241,8cyclictest18794-21kworker/0:112:23:110
2453699241,8cyclictest17967-21kworker/0:110:41:130
24536992414,2cyclictest5507-21kworker/0:211:18:240
24536992414,2cyclictest15749-21kworker/0:010:26:430
2453699241,18cyclictest0-21swapper/012:37:220
2453699241,18cyclictest0-21swapper/012:37:220
2454999237,9cyclictest0-21swapper/211:25:172
2454999237,3cyclictest0-21swapper/209:08:292
2454999231,6cyclictest0-21swapper/208:39:552
2454999231,12cyclictest0-21swapper/208:05:102
2454999231,10cyclictest0-21swapper/207:16:112
2454299237,3cyclictest0-21swapper/112:12:101
2453699239,6cyclictest1491-21kworker/0:311:14:540
2453699239,2cyclictest5681-21kworker/0:307:41:410
2453699237,9cyclictest29198-21kworker/0:011:03:560
2453699237,12cyclictest89050irq/133-enp2s0-09:17:450
2453699234,12cyclictest89150irq/134-enp2s0-11:27:000
2453699233,3cyclictest89050irq/133-enp2s0-09:40:470
2453699233,3cyclictest14337-21kworker/0:311:45:180
2453699232,9cyclictest5681-21kworker/0:307:39:330
2453699232,8cyclictest17553-21kworker/0:209:23:050
2453699231,9cyclictest7225-21kworker/0:210:30:230
2453699231,9cyclictest29426-21kworker/0:307:20:520
24536992313,3cyclictest0-21swapper/011:34:020
24536992313,2cyclictest1602-21kworker/0:212:27:350
2453699231,12cyclictest8038-21kworker/0:007:58:210
2453699230,9cyclictest13076-21kworker/0:009:08:540
2454999229,3cyclictest0-21swapper/211:00:192
2454999221,14cyclictest0-21swapper/208:44:212
2454999220,11cyclictest0-21swapper/209:22:422
2454299226,3cyclictest15772-21ntp_states11:40:221
2454299226,3cyclictest0-21swapper/108:16:591
2454299221,13cyclictest0-21swapper/107:28:161
2454299221,12cyclictest0-21swapper/108:05:111
2454299221,11cyclictest0-21swapper/108:32:111
2453699228,2cyclictest8038-21kworker/0:008:02:170
2453699226,7cyclictest327-21kworker/0:109:53:390
2453699225,10cyclictest87650irq/129-enp1s0-12:15:370
2453699222,9cyclictest29426-21kworker/0:307:18:120
2453699222,8cyclictest21016-21kworker/0:011:55:070
2453699221,12cyclictest30079-21kworker/0:208:37:000
2453699220,9cyclictest25467-21kworker/0:012:08:030
2453699220,10cyclictest25567-21kworker/0:108:46:180
2455399217,3cyclictest0-21swapper/308:44:273
24553992111,2cyclictest18263-21awk10:35:023
2455399211,11cyclictest0-21swapper/309:45:003
2454299211,3cyclictest7936-21ntp_states08:55:231
2454299211,3cyclictest7936-21ntp_states08:55:231
2453699216,5cyclictest14337-21kworker/0:311:41:020
2453699216,5cyclictest14337-21kworker/0:311:38:380
2453699216,11cyclictest89150irq/134-enp2s0-07:25:450
2453699215,2cyclictest18794-21kworker/0:111:53:440
2453699214,10cyclictest89150irq/134-enp2s0-09:37:390
2453699212,9cyclictest3772-21kworker/0:207:49:010
2453699212,9cyclictest3772-21kworker/0:207:49:010
2453699212,11cyclictest89150irq/134-enp2s0-12:03:550
2453699211,7cyclictest22477-21kworker/0:010:45:110
2455399208,3cyclictest12050irq/123-ahci[0009:25:013
2455399206,3cyclictest41-21ksoftirqd/309:15:193
2455399203,8cyclictest87650irq/129-enp1s0-11:23:083
2455399201,7cyclictest0-21swapper/312:02:153
2455399201,7cyclictest0-21swapper/310:18:063
2455399201,3cyclictest39-21rcuc/310:39:273
2455399201,13cyclictest0-21swapper/310:40:183
2455399201,12cyclictest808-21snmpd12:18:573
2455399201,11cyclictest14703-21/usr/sbin/munin11:40:113
2455399201,11cyclictest0-21swapper/312:20:233
2455399200,14cyclictest0-21swapper/312:25:233
2454299201,11cyclictest0-21swapper/107:30:481
204512200,4sleep339-21rcuc/308:10:203
2455399198,3cyclictest12050irq/123-ahci[0007:15:033
2455399196,4cyclictest401ktimersoftd/311:05:223
2455399193,7cyclictest29416-21proc_pri08:30:243
2455399193,4cyclictest0-21swapper/312:30:213
2455399192,14cyclictest22915-21sed09:30:153
2455399192,10cyclictest808-21snmpd11:33:363
24553991914,3cyclictest15625-21grep10:25:263
2455399191,2cyclictest808-21snmpd08:17:573
2455399191,15cyclictest49892chrt10:04:103
2455399191,15cyclictest13475-21users10:20:263
2455399191,11cyclictest839-21lldpd12:08:273
2455399191,11cyclictest839-21lldpd07:20:213
2455399191,11cyclictest0-21swapper/310:51:553
2455399191,10cyclictest808-21snmpd08:07:553
2455399191,10cyclictest0-21swapper/310:05:253
24553991910,5cyclictest41-21ksoftirqd/308:45:153
2453699193,2cyclictest25567-21kworker/0:108:56:160
2453699193,2cyclictest25567-21kworker/0:108:56:160
2453699193,12cyclictest89150irq/134-enp2s0-10:37:430
2455399188,8cyclictest10094-21awk11:30:013
2455399186,4cyclictest87350irq/126-enp1s0-09:45:333
2455399185,3cyclictest0-21swapper/311:45:113
2455399184,6cyclictest12050irq/123-ahci[0007:25:043
2455399183,9cyclictest808-21snmpd08:03:003
2455399183,7cyclictest0-21swapper/311:35:203
2455399181,8cyclictest0-21swapper/311:53:013
2455399181,8cyclictest0-21swapper/309:55:263
2455399181,8cyclictest0-21swapper/307:47:553
2455399181,8cyclictest0-21swapper/307:47:553
2455399181,6cyclictest2316-21cat11:10:173
2455399181,5cyclictest21293-21/usr/sbin/munin11:55:173
24553991814,3cyclictest0-21swapper/307:31:493
24553991811,4cyclictest5790-21wc11:20:013
2455399181,10cyclictest9133-21sshd10:10:573
2455399181,10cyclictest9133-21sshd10:10:573
323332170,2sleep30-21swapper/308:39:193
2455399175,3cyclictest87350irq/126-enp1s0-12:40:013
2455399175,3cyclictest87350irq/126-enp1s0-12:40:013
2455399172,6cyclictest28004-21latency_hist08:30:013
2455399171,8cyclictest0-21swapper/310:55:103
2455399171,7cyclictest0-21swapper/310:46:213
2455399171,4cyclictest0-21swapper/312:10:263
2455399171,15cyclictest0-21swapper/309:50:283
24553991711,3cyclictest7276-21netstat07:40:203
2455399171,13cyclictest4058-21/usr/sbin/munin07:35:123
24553991710,4cyclictest10748-21grep07:50:123
2455399170,15cyclictest0-21swapper/307:19:593
2455399168,2cyclictest87350irq/126-enp1s0-11:00:123
2455399166,3cyclictest808-21snmpd08:22:583
2455399161,6cyclictest4667-21/usr/sbin/munin08:50:223
2455399161,6cyclictest14560-21sshd07:56:573
2455399161,6cyclictest0-21swapper/309:35:203
2455399161,6cyclictest0-21swapper/309:25:163
2455399161,5cyclictest0-21swapper/309:00:143
24553991611,3cyclictest7540-21hddtemp_smartct08:55:173
24553991611,3cyclictest7540-21hddtemp_smartct08:55:173
24553991611,3cyclictest14980-21sed09:10:273
2455399161,12cyclictest11274-21/usr/sbin/munin09:05:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional