You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-18 - 15:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat Apr 18, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
68873222780,17sleep00-21swapper/007:05:200
70555321270,3sleep20-21swapper/207:50:202
68863221260,15sleep3421ksoftirqd/307:05:203
74085121250,2sleep2331rcuc/209:30:192
73273121220,3sleep00-21swapper/009:10:010
77837921100,4sleep13056-21PK-Backend11:20:021
68947726741,20sleep20-21swapper/207:09:182
68926826241,14sleep10-21swapper/107:06:361
8045592530,2sleep13056-21PK-Backend12:35:011
8045592530,2sleep13056-21PK-Backend12:35:001
8052192470,10sleep234-21ksoftirqd/212:35:132
8016862470,2sleep30-21swapper/312:25:163
7702962440,3sleep20-21swapper/210:55:202
7165422340,4sleep03056-21PK-Backend08:20:210
68967299345,9cyclictest709736-21grep08:04:593
68967299331,9cyclictest0-21swapper/310:25:003
689670993312,5cyclictest34-21ksoftirqd/207:20:002
68967299321,14cyclictest0-21swapper/307:20:013
68967099322,2cyclictest141rcu_preempt09:15:002
68967299315,15cyclictest753847-21chrt10:10:033
689672993110,10cyclictest764269-21/usr/sbin/munin10:40:213
68967299304,9cyclictest704563-21cut07:50:013
68967299301,14cyclictest0-21swapper/309:40:003
68967099304,4cyclictest34-21ksoftirqd/209:00:212
689670993011,10cyclictest34-21ksoftirqd/209:20:012
68967299295,11cyclictest734548-21latency_hist09:15:013
68967299292,14cyclictest701833-21ntpq07:40:193
68967299291,6cyclictest0-21swapper/310:30:013
139099290,3rtkit-daemon1389-21rtkit-daemon12:35:393
68967299284,13cyclictest0-21swapper/307:30:593
68967299284,10cyclictest729102-21unin-run08:59:593
68967299273,14cyclictest0-21swapper/311:50:543
68967299272,9cyclictest763094-21sed10:35:183
68967299272,15cyclictest697481-21cat07:30:013
68967299272,14cyclictest795750-21munin-run12:10:003
68967299272,14cyclictest786874-21cut11:45:013
68967299272,14cyclictest786874-21cut11:45:003
68967299272,10cyclictest705236-21idleruntime07:50:163
68967299271,16cyclictest718895-21cut08:30:023
68967099279,9cyclictest34-21ksoftirqd/212:10:012
68967099277,11cyclictest34-21ksoftirqd/210:35:142
689670992710,5cyclictest34-21ksoftirqd/212:05:012
68967299267,6cyclictest711478-21latency_hist08:10:003
68967299265,12cyclictest0-21swapper/308:52:153
68967299262,9cyclictest719548-21sed08:30:183
68967299262,8cyclictest750096-21taskset09:59:303
68967299262,14cyclictest797484-21munin-run12:15:003
68967299262,14cyclictest730337-21cat09:00:223
68967299262,14cyclictest0-21swapper/310:50:163
68967299261,14cyclictest721774-21taskset08:35:313
68967299260,8cyclictest0-21swapper/310:20:003
68967299260,14cyclictest0-21swapper/312:00:153
68967099266,2cyclictest141rcu_preempt07:35:012
68967099265,4cyclictest34-21ksoftirqd/210:20:012
689670992613,4cyclictest34-21ksoftirqd/210:30:012
68967299258,15cyclictest799632-21grep12:20:133
68967299257,16cyclictest716312-21awk08:20:193
68967299254,12cyclictest0-21swapper/311:09:013
68967299253,7cyclictest736262-21munin-run09:20:013
68967299253,13cyclictest0-21swapper/311:57:573
68967299253,12cyclictest0-21swapper/312:15:333
68967299252,13cyclictest0-21swapper/311:29:503
689672992514,8cyclictest42-21ksoftirqd/308:45:223
68967299250,8cyclictest0-21swapper/309:55:003
68967299250,14cyclictest0-21swapper/309:20:143
68967099258,4cyclictest34-21ksoftirqd/210:25:022
689670992511,12cyclictest765789-21unin-run10:45:002
68967299242,9cyclictest775535-21taskset11:10:203
68967299242,8cyclictest740489-21irqstats09:30:163
689672992413,9cyclictest731938-21runrttasks09:05:223
68967299241,15cyclictest776627-21cat11:15:123
68967299241,12cyclictest0-21swapper/311:35:223
68967299240,14cyclictest0-21swapper/307:35:123
68967099248,5cyclictest34-21ksoftirqd/209:39:592
68967099246,10cyclictest34-21ksoftirqd/207:15:002
68967099243,4cyclictest34-21ksoftirqd/209:55:112
689670992411,11cyclictest772720-21unin-run11:05:002
68967099240,6cyclictest755372-21munin-run10:15:012
68967099240,23cyclictest1389-21rtkit-daemon10:45:382
689661992416,2cyclictest13-21ksoftirqd/009:35:140
139099240,16rtkit-daemon3056-21PK-Backend08:05:370
68967299232,13cyclictest722602-21cstates08:40:113
68967299231,9cyclictest751162-21ntp_states10:00:203
68967299231,7cyclictest0-21swapper/310:45:113
68967299231,10cyclictest714871-21taskset08:19:193
68967299230,9cyclictest691518-21sed07:15:003
68967099239,6cyclictest34-21ksoftirqd/207:50:002
68967099238,4cyclictest34-21ksoftirqd/208:30:142
68967099233,11cyclictest34-21ksoftirqd/211:35:122
68967099232,4cyclictest34-21ksoftirqd/208:50:012
68967099232,10cyclictest34-21ksoftirqd/208:05:142
8049162220,3sleep00-21swapper/012:35:040
68967299228,4cyclictest753833-21latency_hist10:10:013
689672992219,2cyclictest0-21swapper/310:56:583
68967299221,8cyclictest712275-21sed08:10:173
68967299221,7cyclictest782076-21cstates11:30:103
68967299220,21cyclictest42-21ksoftirqd/309:41:223
68967299220,13cyclictest0-21swapper/311:00:173
68967099225,8cyclictest34-21ksoftirqd/209:25:182
68967099223,5cyclictest34-21ksoftirqd/211:16:092
689670992215,6cyclictest714935-21apt-get08:20:012
68967099221,3cyclictest34-21ksoftirqd/207:30:012
689670992212,3cyclictest34-21ksoftirqd/210:06:522
689670992210,2cyclictest141rcu_preempt08:00:132
68967099220,3cyclictest34-21ksoftirqd/208:30:022
68967099220,21cyclictest0-21swapper/211:53:012
68966199228,3cyclictest7702992processes10:55:210
689661992220,1cyclictest0-21swapper/012:21:110
689661992220,1cyclictest0-21swapper/009:03:130
68966199220,21cyclictest0-21swapper/009:46:310
68966199220,21cyclictest0-21swapper/008:49:270
689672992119,1cyclictest0-21swapper/309:26:343
689672992119,1cyclictest0-21swapper/307:57:403
68967299211,8cyclictest694100-21apt07:20:123
68967299210,20cyclictest0-21swapper/311:48:433
68967099213,2cyclictest141rcu_preempt12:30:232
68967099213,2cyclictest141rcu_preempt12:30:232
689670992115,4cyclictest2156-21gnome-shell11:07:092
68967099211,4cyclictest34-21ksoftirqd/209:00:002
689670992113,6cyclictest799687-21cat12:20:122
68967099210,20cyclictest0-21swapper/211:45:322
689666992118,2cyclictest3056-21PK-Backend10:25:131
68966699210,20cyclictest3056-21PK-Backend07:21:001
68966699210,1cyclictest3056-21PK-Backend12:07:271
68966699210,19cyclictest7922292sleep111:59:351
68966699210,19cyclictest3056-21PK-Backend12:16:391
68966699210,19cyclictest3056-21PK-Backend09:55:461
68966699210,19cyclictest3056-21PK-Backend08:40:151
68966199216,4cyclictest917-21in:imuxsock12:29:590
689661992119,1cyclictest3056-21PK-Backend09:15:570
689661992119,1cyclictest0-21swapper/011:05:410
689661992119,1cyclictest0-21swapper/010:00:040
689661992119,1cyclictest0-21swapper/008:16:490
689661992119,1cyclictest0-21swapper/008:10:540
68966199210,20cyclictest0-21swapper/012:33:410
68966199210,20cyclictest0-21swapper/012:33:410
68966199210,20cyclictest0-21swapper/011:34:140
68966199210,20cyclictest0-21swapper/011:25:550
68966199210,20cyclictest0-21swapper/011:11:070
68966199210,20cyclictest0-21swapper/011:04:400
68966199210,20cyclictest0-21swapper/010:46:130
68966199210,20cyclictest0-21swapper/010:26:340
68966199210,20cyclictest0-21swapper/008:03:570
68967299208,9cyclictest0-21swapper/310:30:123
689672992015,3cyclictest746098-21python309:45:313
68967099204,4cyclictest34-21ksoftirqd/211:40:192
68967099204,4cyclictest34-21ksoftirqd/211:40:192
68967099202,5cyclictest34-21ksoftirqd/209:10:002
68967099202,4cyclictest34-21ksoftirqd/207:40:142
68967099202,3cyclictest34-21ksoftirqd/211:55:212
689670992018,1cyclictest0-21swapper/212:19:112
689670992018,1cyclictest0-21swapper/208:10:022
68967099201,4cyclictest34-21ksoftirqd/209:50:152
689670992014,5cyclictest707401-21python307:55:202
68967099200,4cyclictest34-21ksoftirqd/207:35:132
68967099200,4cyclictest34-21ksoftirqd/207:20:112
68967099200,19cyclictest0-21swapper/212:10:532
68967099200,19cyclictest0-21swapper/211:30:302
68967099200,19cyclictest0-21swapper/210:51:112
68967099200,19cyclictest0-21swapper/210:03:512
68967099200,19cyclictest0-21swapper/208:42:062
68967099200,19cyclictest0-21swapper/208:36:592
68967099200,19cyclictest0-21swapper/208:23:552
689666992018,1cyclictest3056-21PK-Backend10:32:501
689666992018,1cyclictest3056-21PK-Backend08:38:261
689666992017,2cyclictest3056-21PK-Backend07:53:101
68966699200,1cyclictest3056-21PK-Backend09:08:001
68966699200,19cyclictest3056-21PK-Backend12:12:271
68966699200,19cyclictest3056-21PK-Backend12:01:241
68966699200,19cyclictest3056-21PK-Backend08:46:431
68966699200,19cyclictest3056-21PK-Backend07:48:311
68966699200,18cyclictest3056-21PK-Backend11:31:161
68966699200,18cyclictest3056-21PK-Backend11:15:491
68966699200,18cyclictest3056-21PK-Backend10:22:361
689661992019,1cyclictest0-21swapper/009:56:260
689661992019,1cyclictest0-21swapper/008:59:330
689661992018,1cyclictest0-21swapper/012:08:260
689661992018,1cyclictest0-21swapper/010:43:230
689661992018,1cyclictest0-21swapper/010:09:580
689661992018,1cyclictest0-21swapper/009:12:180
689661992018,1cyclictest0-21swapper/008:31:280
689661992018,1cyclictest0-21swapper/008:29:250
689661992018,1cyclictest0-21swapper/007:30:050
689661992018,1cyclictest0-21swapper/007:10:300
689661992012,6cyclictest754955-21users10:10:210
68966199200,1cyclictest0-21swapper/011:49:000
68966199200,19cyclictest0-21swapper/012:10:160
68966199200,19cyclictest0-21swapper/010:20:040
68966199200,19cyclictest0-21swapper/008:53:180
7390012190,4sleep13056-21PK-Backend09:25:181
7367502190,11sleep234-21ksoftirqd/209:20:142
689672991917,1cyclictest0-21swapper/311:23:143
68967299190,1cyclictest0-21swapper/312:32:433
68967299190,1cyclictest0-21swapper/312:32:433
689670991913,5cyclictest779046-21cat11:20:162
68967099190,9cyclictest34-21ksoftirqd/210:30:172
68967099190,1cyclictest0-21swapper/211:29:292
68967099190,19cyclictest0-21swapper/212:25:412
689666991917,1cyclictest3056-21PK-Backend12:20:031
689666991917,1cyclictest3056-21PK-Backend11:52:471
689666991917,1cyclictest3056-21PK-Backend11:10:481
689666991917,1cyclictest0-21swapper/111:40:431
689666991917,1cyclictest0-21swapper/111:40:421
689666991916,2cyclictest3056-21PK-Backend08:30:221
68966699190,1cyclictest3056-21PK-Backend11:46:451
68966699190,1cyclictest3056-21PK-Backend11:00:351
68966699190,1cyclictest3056-21PK-Backend10:49:191
68966699190,1cyclictest3056-21PK-Backend10:40:401
68966699190,1cyclictest3056-21PK-Backend09:03:001
68966699190,1cyclictest3056-21PK-Backend08:16:491
68966699190,1cyclictest3056-21PK-Backend07:44:531
68966699190,1cyclictest3056-21PK-Backend07:38:241
68966699190,18cyclictest3056-21PK-Backend11:38:531
68966699190,18cyclictest3056-21PK-Backend10:19:111
68966699190,18cyclictest3056-21PK-Backend10:11:331
68966699190,18cyclictest3056-21PK-Backend07:17:211
68966699190,17cyclictest3056-21PK-Backend08:03:041
689661991918,1cyclictest0-21swapper/010:53:440
689661991918,1cyclictest0-21swapper/009:50:100
689661991917,1cyclictest0-21swapper/012:17:330
689661991917,1cyclictest0-21swapper/012:03:000
689661991917,1cyclictest0-21swapper/011:35:150
689661991917,1cyclictest0-21swapper/009:27:520
689661991917,1cyclictest0-21swapper/008:40:280
689661991916,2cyclictest3056-21PK-Backend11:51:510
689661991912,5cyclictest776692-21ls11:15:130
68966199190,1cyclictest0-21swapper/009:30:440
68966199190,1cyclictest0-21swapper/007:16:120
68966199190,18cyclictest0-21swapper/007:51:070
68967099186,2cyclictest141rcu_preempt09:45:002
68967099184,4cyclictest34-21ksoftirqd/208:50:342
68967099181,5cyclictest34-21ksoftirqd/209:45:222
689666991816,1cyclictest3056-21PK-Backend11:08:111
689666991816,1cyclictest3056-21PK-Backend10:38:031
689666991816,1cyclictest3056-21PK-Backend08:27:311
689666991816,1cyclictest3056-21PK-Backend08:15:011
689666991816,1cyclictest3056-21PK-Backend07:56:481
68966699180,1cyclictest3056-21PK-Backend10:00:461
68966699180,16cyclictest3056-21PK-Backend12:25:511
689661991817,1cyclictest0-21swapper/009:42:540
689661991816,1cyclictest0-21swapper/010:16:260
689661991816,1cyclictest0-21swapper/008:38:250
689661991815,2cyclictest701785-21awk07:40:190
689661991813,3cyclictest737073-21latency09:20:170
689661991812,4cyclictest762540-21acpi10:35:120
689666991712,3cyclictest770953-21sh11:00:001
689666991711,5cyclictest883-21dbus-daemon09:36:391
689661991716,1cyclictest0-21swapper/011:20:270
689661991715,1cyclictest0-21swapper/007:22:070
689661991711,5cyclictest791184-21df11:55:140
68967099160,5cyclictest3056-21PK-Backend11:10:242
68966699169,5cyclictest710930-21sed08:05:231
689666991610,4cyclictest746683-21latency_hist09:50:001
68966699160,15cyclictest3056-21PK-Backend10:51:091
68966199169,5cyclictest761555-21awk10:30:190
68966199166,2cyclictest141rcu_preempt11:40:120
68966199166,2cyclictest141rcu_preempt11:40:120
689661991614,1cyclictest0-21swapper/007:55:130
689661991610,5cyclictest696683-21gzip07:25:220
689666991513,1cyclictest3056-21PK-Backend11:27:051
689666991512,2cyclictest734948-21tr09:15:131
689666991512,2cyclictest3056-21PK-Backend07:31:061
689666991511,3cyclictest3056-21PK-Backend08:54:131
68966699150,5cyclictest3056-21PK-Backend08:55:181
68966699150,4cyclictest716261-21ps08:20:171
68966699150,14cyclictest3056-21PK-Backend09:10:361
68966199158,5cyclictest703527-21awk07:45:200
689661991514,1cyclictest0-21swapper/007:35:110
68966699143,2cyclictest141rcu_preempt12:35:181
68966699142,2cyclictest141rcu_preempt09:40:121
689666991412,1cyclictest3056-21PK-Backend10:06:331
689666991410,3cyclictest3056-21PK-Backend09:53:131
68966699140,6cyclictest3056-21PK-Backend09:30:121
68966699140,13cyclictest3056-21PK-Backend07:12:551
68966699133,2cyclictest141rcu_preempt07:25:201
68966699130,5cyclictest3056-21PK-Backend09:20:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional