You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 22:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue May 19, 2026 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
146899921200,17sleep2341ksoftirqd/207:05:202
152248321170,3sleep13056-21PK-Backend09:35:201
157479821000,4sleep13056-21PK-Backend12:05:141
139099910,56rtkit-daemon0-21swapper/307:07:153
146953927840,13sleep00-21swapper/007:05:380
146955527346,17sleep10-21swapper/107:05:521
15644392520,3sleep13056-21PK-Backend11:35:141
15308202470,2sleep30-21swapper/310:00:133
15516872370,12sleep126-21ksoftirqd/111:00:131
14766812300,11sleep126-21ksoftirqd/107:25:201
147002799280,26cyclictest1509203-21cron09:00:013
15089132270,4sleep13056-21PK-Backend08:56:391
15444842250,4chrt3056-21PK-Backend10:40:001
1470027992512,11cyclictest42-21ksoftirqd/310:10:023
15642192240,5sleep2331rcuc/211:35:112
1470027992414,8cyclictest436-20systemd-journal10:30:003
147001799241,3cyclictest0-21swapper/007:20:210
147001799240,7cyclictest1479104-21turbostat07:34:590
15708012230,8sleep342-21ksoftirqd/311:55:013
147002799230,21cyclictest1561712-21taskset11:26:353
147002799230,21cyclictest1561712-21taskset11:26:353
147002799230,21cyclictest0-21swapper/309:40:013
1470021992320,1cyclictest3056-21PK-Backend10:40:201
1470017992315,6cyclictest1536865-21python310:15:210
1470027992220,1cyclictest0-21swapper/311:37:223
1470027992220,1cyclictest0-21swapper/310:32:003
1470027992220,1cyclictest0-21swapper/310:32:003
1470027992220,1cyclictest0-21swapper/307:14:363
1470027992219,2cyclictest1520824-21find09:30:223
1470027992218,3cyclictest0-21swapper/311:10:193
1470027992213,7cyclictest883-21dbus-daemon08:25:003
147002799221,18cyclictest82332-21update-manager11:30:113
147002799220,20cyclictest1529203-21hddtemp_smartct09:55:163
1470022992215,5cyclictest1488723-21df08:00:132
1470022992213,7cyclictest1497494-21/usr/sbin/munin08:25:172
147002299220,20cyclictest1479100-21apt-get07:35:022
1470021992220,1cyclictest3056-21PK-Backend09:06:571
1470021992219,2cyclictest3056-21PK-Backend11:11:501
1470021992219,2cyclictest3056-21PK-Backend09:30:501
147002199220,20cyclictest3056-21PK-Backend11:07:571
15713652210,4sleep00-21swapper/011:55:180
15661402210,4sleep13056-21PK-Backend11:40:161
1470027992120,1cyclictest0-21swapper/308:01:333
1470027992119,1cyclictest0-21swapper/311:45:203
1470027992119,1cyclictest0-21swapper/309:26:033
1470027992119,1cyclictest0-21swapper/309:22:443
1470027992119,1cyclictest0-21swapper/308:43:233
1470027992119,1cyclictest0-21swapper/308:38:313
1470027992119,1cyclictest0-21swapper/307:57:353
147002799211,7cyclictest1515548-21python309:15:213
1470027992113,6cyclictest1506435-21ntp_states08:50:183
147002799210,20cyclictest0-21swapper/311:56:093
147002799210,20cyclictest0-21swapper/309:45:223
147002799210,20cyclictest0-21swapper/308:26:283
147002799210,20cyclictest0-21swapper/308:05:313
147002799210,20cyclictest0-21swapper/307:53:363
147002799210,19cyclictest42-21ksoftirqd/310:35:123
1470022992119,1cyclictest0-21swapper/212:31:362
1470022992119,1cyclictest0-21swapper/212:26:572
1470022992119,1cyclictest0-21swapper/209:39:062
1470022992117,3cyclictest1504702-21ntp_states08:45:192
1470022992115,4cyclictest1526935-21latency_hist09:50:012
1470022992114,5cyclictest1523502-21mktemp09:40:012
147002299210,1cyclictest0-21swapper/210:45:502
1470021992119,1cyclictest3056-21PK-Backend11:45:361
1470021992119,1cyclictest3056-21PK-Backend10:33:091
1470021992119,1cyclictest3056-21PK-Backend10:33:091
1470021992119,1cyclictest3056-21PK-Backend10:19:041
1470021992119,1cyclictest3056-21PK-Backend09:21:241
1470021992119,1cyclictest3056-21PK-Backend08:45:491
1470021992118,2cyclictest3056-21PK-Backend08:44:471
147002199210,20cyclictest3056-21PK-Backend08:29:321
147002199210,19cyclictest3056-21PK-Backend10:50:541
147002199210,19cyclictest3056-21PK-Backend10:29:041
147002199210,19cyclictest3056-21PK-Backend09:40:171
1470017992119,1cyclictest3056-21PK-Backend07:10:340
1470017992119,1cyclictest0-21swapper/007:42:180
1470017992119,1cyclictest0-21swapper/007:15:210
1470017992118,2cyclictest3056-21PK-Backend12:04:100
147001799210,20cyclictest0-21swapper/012:13:160
147001799210,20cyclictest0-21swapper/012:07:490
147001799210,20cyclictest0-21swapper/011:24:300
139099210,15rtkit-daemon3056-21PK-Backend10:53:293
139099210,14rtkit-daemon0-21swapper/310:42:103
1470027992018,1cyclictest3056-21PK-Backend12:01:373
1470027992018,1cyclictest3056-21PK-Backend10:57:423
1470027992018,1cyclictest0-21swapper/312:29:563
1470027992018,1cyclictest0-21swapper/312:16:093
1470027992018,1cyclictest0-21swapper/312:05:143
1470027992018,1cyclictest0-21swapper/311:18:033
1470027992018,1cyclictest0-21swapper/310:20:443
1470027992018,1cyclictest0-21swapper/309:51:593
1470027992018,1cyclictest0-21swapper/308:45:263
1470027992018,1cyclictest0-21swapper/307:47:213
1470027992018,1cyclictest0-21swapper/307:44:503
1470027992018,1cyclictest0-21swapper/307:38:483
147002799200,8cyclictest1566137-21/usr/sbin/munin11:40:153
147002799200,1cyclictest3056-21PK-Backend08:13:293
147002799200,1cyclictest0-21swapper/311:00:333
147002799200,19cyclictest3056-21PK-Backend11:23:443
147002799200,19cyclictest0-21swapper/309:03:323
1470022992018,1cyclictest0-21swapper/208:33:432
1470022992018,1cyclictest0-21swapper/207:16:492
1470022992014,5cyclictest1501988-21systemd-run08:40:002
1470022992014,4cyclictest1534912-21grep10:10:162
1470022992012,6cyclictest1508132-21ntpq08:55:192
147002299200,1cyclictest0-21swapper/207:31:032
147002299200,19cyclictest0-21swapper/212:09:192
147002299200,19cyclictest0-21swapper/211:42:422
147002299200,19cyclictest0-21swapper/211:19:022
147002299200,19cyclictest0-21swapper/211:07:142
147002299200,19cyclictest0-21swapper/210:52:332
147002299200,19cyclictest0-21swapper/210:25:082
147002299200,19cyclictest0-21swapper/209:57:372
147002299200,19cyclictest0-21swapper/209:26:072
147002299200,19cyclictest0-21swapper/209:22:492
1470021992018,1cyclictest3056-21PK-Backend12:12:191
1470021992018,1cyclictest3056-21PK-Backend11:30:071
1470021992018,1cyclictest3056-21PK-Backend10:02:401
1470021992018,1cyclictest3056-21PK-Backend09:13:471
1470021992018,1cyclictest3056-21PK-Backend08:35:271
1470021992018,1cyclictest3056-21PK-Backend08:31:491
1470021992017,2cyclictest3056-21PK-Backend11:51:301
1470021992017,2cyclictest3056-21PK-Backend09:58:211
147002199200,1cyclictest3056-21PK-Backend07:45:191
147002199200,19cyclictest3056-21PK-Backend08:00:191
147002199200,19cyclictest3056-21PK-Backend07:53:511
147002199200,19cyclictest0-21swapper/112:35:591
147002199200,19cyclictest0-21swapper/111:21:231
147002199200,19cyclictest0-21swapper/110:58:231
147002199200,19cyclictest0-21swapper/110:46:141
147002199200,19cyclictest0-21swapper/107:20:031
1470017992018,1cyclictest0-21swapper/012:16:080
1470017992018,1cyclictest0-21swapper/011:31:250
1470017992018,1cyclictest0-21swapper/009:58:110
1470017992018,1cyclictest0-21swapper/008:55:250
1470017992018,1cyclictest0-21swapper/008:20:030
1470017992018,1cyclictest0-21swapper/008:00:020
1470017992018,1cyclictest0-21swapper/007:56:040
1470017992018,1cyclictest0-21swapper/007:49:350
1470017992018,1cyclictest0-21swapper/007:25:350
1470017992013,5cyclictest1493897-21cat08:15:140
147001799200,1cyclictest0-21swapper/011:54:040
147001799200,1cyclictest0-21swapper/008:32:470
147001799200,19cyclictest3056-21PK-Backend11:17:470
147001799200,19cyclictest0-21swapper/011:29:240
147001799200,19cyclictest0-21swapper/011:29:240
147001799200,19cyclictest0-21swapper/011:13:550
147001799200,19cyclictest0-21swapper/009:35:460
147001799200,19cyclictest0-21swapper/009:06:190
147001799200,19cyclictest0-21swapper/008:13:270
15712332190,3sleep13056-21PK-Backend11:55:141
1470027991917,1cyclictest0-21swapper/310:17:403
147002799191,16cyclictest1499624-21python308:30:203
147002799191,13cyclictest1553564-21apt-get11:05:033
147002799190,1cyclictest0-21swapper/310:13:343
147002799190,1cyclictest0-21swapper/309:13:523
1470022991917,1cyclictest0-21swapper/212:12:102
1470022991917,1cyclictest0-21swapper/211:56:342
1470022991917,1cyclictest0-21swapper/211:21:052
1470022991917,1cyclictest0-21swapper/210:06:442
1470022991917,1cyclictest0-21swapper/209:16:282
1470022991917,1cyclictest0-21swapper/209:06:552
1470022991917,1cyclictest0-21swapper/208:14:232
1470022991917,1cyclictest0-21swapper/207:14:192
1470022991913,4cyclictest1486095-21sshd07:51:522
1470022991912,6cyclictest1562945-21grep11:30:172
1470022991912,5cyclictest1542668-21cron10:35:002
1470022991912,5cyclictest1542668-21cron10:34:592
1470022991912,5cyclictest1490226-21expr08:05:112
1470022991912,5cyclictest1484778-21wc07:50:002
1470022991911,6cyclictest1513340-21fw_packets09:10:162
147002299190,1cyclictest0-21swapper/212:36:152
147002299190,1cyclictest0-21swapper/211:52:422
147002299190,1cyclictest0-21swapper/211:49:382
147002299190,1cyclictest0-21swapper/211:28:502
147002299190,1cyclictest0-21swapper/211:28:492
147002299190,1cyclictest0-21swapper/210:44:502
147002299190,1cyclictest0-21swapper/210:21:032
147002299190,1cyclictest0-21swapper/209:34:472
147002299190,1cyclictest0-21swapper/208:42:152
147002299190,18cyclictest0-21swapper/210:19:012
147002199195,12cyclictest1528551-21idleruntime-cro09:55:001
1470021991917,1cyclictest3056-21PK-Backend12:25:501
1470021991917,1cyclictest3056-21PK-Backend12:15:581
1470021991917,1cyclictest3056-21PK-Backend10:10:181
1470021991917,1cyclictest0-21swapper/112:02:251
1470021991917,1cyclictest0-21swapper/111:26:151
1470021991917,1cyclictest0-21swapper/111:26:151
1470021991917,1cyclictest0-21swapper/108:09:321
1470021991916,2cyclictest3056-21PK-Backend10:22:081
147002199190,1cyclictest3056-21PK-Backend12:22:121
147002199190,1cyclictest3056-21PK-Backend12:22:121
147002199190,1cyclictest3056-21PK-Backend07:35:321
147002199190,1cyclictest3056-21PK-Backend07:16:191
147002199190,1cyclictest3056-21PK-Backend07:12:481
147002199190,1cyclictest0-21swapper/111:17:311
147002199190,1cyclictest0-21swapper/107:31:331
147002199190,18cyclictest3056-21PK-Backend12:31:181
1470017991917,1cyclictest0-21swapper/011:42:140
1470017991917,1cyclictest0-21swapper/011:05:230
1470017991917,1cyclictest0-21swapper/010:48:200
1470017991917,1cyclictest0-21swapper/010:00:280
1470017991917,1cyclictest0-21swapper/009:53:520
1470017991917,1cyclictest0-21swapper/009:16:400
1470017991917,1cyclictest0-21swapper/007:38:320
1470017991914,3cyclictest436-20systemd-journal10:15:000
1470017991914,3cyclictest1510947-21idleruntime-cro09:05:010
147001799191,2cyclictest141rcu_preempt09:40:170
1470017991912,5cyclictest1550332-21interrupts10:55:170
147001799190,1cyclictest3056-21PK-Backend09:10:520
147001799190,1cyclictest0-21swapper/012:36:210
147001799190,1cyclictest0-21swapper/012:33:310
147001799190,1cyclictest0-21swapper/011:01:310
147001799190,1cyclictest0-21swapper/008:48:010
147001799190,1cyclictest0-21swapper/008:36:250
147001799190,1cyclictest0-21swapper/007:54:350
147001799190,18cyclictest0-21swapper/012:21:350
147001799190,18cyclictest0-21swapper/012:21:350
147001799190,18cyclictest0-21swapper/010:08:040
15511412180,8sleep20-21swapper/210:57:502
15339662180,5sleep1251rcuc/110:10:001
1470027991816,1cyclictest0-21swapper/309:07:043
1470027991816,1cyclictest0-21swapper/307:17:063
1470027991813,3cyclictest1546765-21df10:45:133
147002799180,5cyclictest1584120-21timerwakeupswit12:30:233
1470022991816,1cyclictest0-21swapper/212:02:022
1470022991816,1cyclictest0-21swapper/209:01:212
1470022991813,4cyclictest34-21ksoftirqd/211:10:122
1470022991811,5cyclictest1530935-21fw_packets10:00:152
1470022991810,6cyclictest883-21dbus-daemon10:40:012
147002299180,3cyclictest1496767-21perf08:24:592
147002199184,2cyclictest141rcu_preempt07:40:151
1470021991816,1cyclictest3056-21PK-Backend09:49:431
1470021991816,1cyclictest3056-21PK-Backend09:15:021
1470021991816,1cyclictest0-21swapper/108:16:011
1470017991816,1cyclictest0-21swapper/012:26:140
1470017991816,1cyclictest0-21swapper/010:22:110
1470017991816,1cyclictest0-21swapper/008:25:240
1470017991813,3cyclictest1095-21runrttasks10:54:510
1470017991812,5cyclictest886-21NetworkManager11:45:130
1470017991812,5cyclictest1564976-21sed11:35:200
147001799180,1cyclictest0-21swapper/010:34:140
147001799180,1cyclictest0-21swapper/010:34:140
147002799171,4cyclictest3056-21PK-Backend12:10:123
1470027991711,5cyclictest1580471-21gzip12:20:203
1470027991711,5cyclictest1580471-21gzip12:20:203
147002799171,12cyclictest1523421-21apt-key09:40:013
1470027991710,5cyclictest1494840-21sshd08:17:523
147002299179,6cyclictest1527216-21expr09:50:112
1470022991715,1cyclictest0-21swapper/208:18:422
1470022991715,1cyclictest0-21swapper/207:22:032
1470022991712,3cyclictest1487110-21kernelversion07:55:162
1470022991710,5cyclictest436-20systemd-journal07:29:562
1470022991710,5cyclictest1505690-21apt08:50:102
1470021991715,1cyclictest3056-21PK-Backend08:24:121
147001799172,13cyclictest1490259-21cpuspeed_turbos08:05:100
1470017991715,1cyclictest0-21swapper/010:25:150
147001799170,4cyclictest1517879-21perf09:25:010
147001799170,10cyclictest15028332cat08:40:160
139099170,4rtkit-daemon3056-21PK-Backend07:33:113
147002799161,11cyclictest1477371-21sh07:30:003
147002299169,6cyclictest1551759-21expr11:00:142
1470022991610,5cyclictest1580852-21seq12:22:122
1470022991610,5cyclictest1580852-21seq12:22:122
1470022991610,4cyclictest1481560-21grep07:40:112
147002199169,5cyclictest1492453-21cat08:10:191
147002199164,2cyclictest141rcu_preempt08:50:171
147001799169,5cyclictest1546142-21perf10:44:590
1470017991612,3cyclictest1543942-21timerwakeupswit10:35:230
147002799150,14cyclictest0-21swapper/312:35:133
147002299159,4cyclictest1578323-21if_enp2s012:15:162
1470021991513,1cyclictest3056-21PK-Backend09:03:391
147001799151,1cyclictest141rcu_preempt09:30:130
147001799150,2cyclictest141rcu_preempt09:46:340
147001799150,2cyclictest141rcu_preempt09:28:340
147001799150,2cyclictest141rcu_preempt08:50:150
147002799149,3cyclictest1474387-21cstates07:20:143
147002199144,2cyclictest141rcu_preempt09:25:161
1470021991210,1cyclictest3056-21PK-Backend07:59:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional