You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-28 - 20:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sat Mar 28, 2026 12:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
148035621280,2sleep13056-21PK-Backend11:30:161
138624421100,6sleep10-21swapper/107:05:221
138655228262,13sleep00-21swapper/007:05:460
138574626844,16sleep30-21swapper/307:05:133
138611126242,14sleep20-21swapper/207:05:202
15032572510,2sleep30-21swapper/312:35:133
14622912460,2sleep30-21swapper/310:40:143
14622912460,2sleep30-21swapper/310:40:133
14841732430,2sleep20-21swapper/211:40:192
14588082350,2sleep11458810-21expr10:30:141
14305762320,2sleep30-21swapper/309:10:113
138702399283,2cyclictest141rcu_preempt11:10:002
14245372270,3sleep20-21swapper/208:50:212
1387023992711,4cyclictest34-21ksoftirqd/207:10:182
138702999262,3cyclictest141rcu_preempt11:00:003
139099250,4rtkit-daemon1389-21rtkit-daemon12:27:400
1387023992510,3cyclictest34-21ksoftirqd/211:00:012
1387021992513,10cyclictest1389502-21/usr/sbin/munin07:15:121
1387029992410,12cyclictest1490263-21unin-run12:00:013
138702399244,5cyclictest34-21ksoftirqd/211:25:132
1387023992416,6cyclictest1412285-21sshd08:19:162
1387023992410,4cyclictest34-21ksoftirqd/210:26:012
1387029992318,3cyclictest1413000-21diskstats08:20:143
1387029992311,3cyclictest42-21ksoftirqd/310:00:013
138702399238,2cyclictest141rcu_preempt10:49:182
138702399237,4cyclictest34-21ksoftirqd/212:27:512
1387023992316,5cyclictest1442596-21cron09:45:002
1387023992312,3cyclictest34-21ksoftirqd/210:30:062
1387023992312,2cyclictest141rcu_preempt07:25:292
1387023992311,11cyclictest34-21ksoftirqd/209:25:012
1387023992310,4cyclictest34-21ksoftirqd/211:46:552
138702399230,3cyclictest1452995-21cron10:15:002
138701899230,8cyclictest14093462expr08:10:150
139099220,19rtkit-daemon1788643-21eadedCompositor11:43:590
139099220,15rtkit-daemon0-21swapper/007:54:240
138702999228,13cyclictest1412362-21latency_hist08:20:013
1387029992219,2cyclictest3056-21PK-Backend10:55:003
1387029992218,3cyclictest411rcuc/308:10:063
1387029992217,4cyclictest42-21ksoftirqd/309:05:013
1387029992214,6cyclictest1490889-21fw_packets12:00:123
138702399229,3cyclictest34-21ksoftirqd/209:09:342
138702399229,3cyclictest34-21ksoftirqd/208:45:512
138702399225,2cyclictest141rcu_preempt10:55:002
1387023992220,1cyclictest0-21swapper/211:32:412
1387023992220,1cyclictest0-21swapper/209:54:592
1387023992220,1cyclictest0-21swapper/209:02:002
1387023992220,1cyclictest0-21swapper/207:17:502
1387023992219,2cyclictest0-21swapper/208:36:322
1387023992218,2cyclictest0-21swapper/210:24:132
1387023992210,3cyclictest34-21ksoftirqd/212:11:422
1387023992210,3cyclictest34-21ksoftirqd/211:55:552
1387023992210,3cyclictest34-21ksoftirqd/208:22:412
138702399220,10cyclictest1395302-21tr07:30:142
1387021992220,1cyclictest3056-21PK-Backend10:27:031
1387021992220,1cyclictest3056-21PK-Backend10:21:081
1387021992220,1cyclictest3056-21PK-Backend08:41:181
1387021992220,1cyclictest3056-21PK-Backend07:55:001
1387021992220,1cyclictest0-21swapper/111:05:301
1387021992219,2cyclictest3056-21PK-Backend09:30:331
1387021992219,2cyclictest0-21swapper/107:35:071
138702199220,20cyclictest3056-21PK-Backend08:49:501
1387018992220,1cyclictest0-21swapper/009:14:190
1387018992220,1cyclictest0-21swapper/008:52:490
1387018992216,4cyclictest13-21ksoftirqd/011:20:030
138701899220,21cyclictest0-21swapper/009:03:570
138701899220,20cyclictest3056-21PK-Backend08:05:300
14791192210,3chrt0-21swapper/011:25:210
139099210,4rtkit-daemon1389-21rtkit-daemon11:55:281
1387029992119,1cyclictest0-21swapper/307:53:453
1387029992117,3cyclictest42-21ksoftirqd/311:55:013
1387029992114,5cyclictest436-20systemd-journal11:30:183
138702399218,3cyclictest34-21ksoftirqd/210:40:472
138702399218,3cyclictest34-21ksoftirqd/210:40:472
1387023992119,1cyclictest0-21swapper/212:31:142
1387023992119,1cyclictest0-21swapper/212:04:242
1387023992119,1cyclictest0-21swapper/211:39:222
1387023992119,1cyclictest0-21swapper/211:23:082
1387023992119,1cyclictest0-21swapper/207:55:492
1387023992119,1cyclictest0-21swapper/207:53:452
1387023992119,1cyclictest0-21swapper/207:47:512
1387023992119,1cyclictest0-21swapper/207:22:432
1387023992116,4cyclictest34-21ksoftirqd/209:35:192
1387023992114,5cyclictest1503571-21strings12:35:192
1387023992113,5cyclictest1493687-21cron12:10:012
138702399210,20cyclictest0-21swapper/208:59:572
1387021992119,1cyclictest3056-21PK-Backend11:42:141
1387021992119,1cyclictest3056-21PK-Backend10:38:401
1387021992119,1cyclictest3056-21PK-Backend09:07:071
1387021992119,1cyclictest3056-21PK-Backend08:29:411
1387021992119,1cyclictest0-21swapper/111:23:201
1387021992119,1cyclictest0-21swapper/110:59:351
1387021992119,1cyclictest0-21swapper/107:11:551
1387021992118,2cyclictest3056-21PK-Backend10:01:351
1387021992118,2cyclictest3056-21PK-Backend08:56:321
1387021992118,2cyclictest3056-21PK-Backend08:38:281
1387021992114,5cyclictest1475822-21latency_hist11:20:001
1387021992114,5cyclictest1428482-21grep09:05:001
138702199210,2cyclictest1465154-21apt-get10:50:011
138702199210,20cyclictest3056-21PK-Backend08:50:511
138702199210,19cyclictest3056-21PK-Backend12:00:531
138702199210,19cyclictest3056-21PK-Backend10:47:101
138702199210,19cyclictest3056-21PK-Backend10:41:301
138702199210,19cyclictest3056-21PK-Backend10:41:301
138702199210,19cyclictest3056-21PK-Backend09:10:591
1387018992119,1cyclictest0-21swapper/012:02:090
1387018992119,1cyclictest0-21swapper/010:56:250
1387018992119,1cyclictest0-21swapper/010:44:500
1387018992119,1cyclictest0-21swapper/010:44:500
1387018992119,1cyclictest0-21swapper/010:18:130
1387018992119,1cyclictest0-21swapper/009:22:030
1387018992119,1cyclictest0-21swapper/008:38:010
1387018992119,1cyclictest0-21swapper/007:31:450
138701899210,20cyclictest0-21swapper/010:24:560
138701899210,20cyclictest0-21swapper/009:44:130
138701899210,20cyclictest0-21swapper/008:20:040
138701899210,20cyclictest0-21swapper/008:02:270
138701899210,20cyclictest0-21swapper/007:39:070
139099200,14rtkit-daemon3056-21PK-Backend08:08:051
139099200,14rtkit-daemon0-21swapper/011:16:240
138702999209,10cyclictest1422559-21ntp_states08:45:193
138702999206,13cyclictest42-21ksoftirqd/308:57:213
1387029992018,1cyclictest3056-21PK-Backend07:23:233
1387029992018,1cyclictest0-21swapper/312:26:123
1387029992018,1cyclictest0-21swapper/311:49:013
1387029992017,2cyclictest0-21swapper/309:35:103
1387029992012,6cyclictest1492285-21awk12:05:113
1387029992012,6cyclictest1420582-21awk08:40:183
138702999201,17cyclictest978-21snmpd10:08:083
138702999201,17cyclictest1482436-21python311:35:203
138702999200,1cyclictest3056-21PK-Backend11:00:413
138702999200,1cyclictest0-21swapper/312:17:413
138702999200,1cyclictest0-21swapper/310:33:503
138702999200,1cyclictest0-21swapper/310:01:463
138702999200,1cyclictest0-21swapper/308:02:573
138702999200,19cyclictest3056-21PK-Backend07:19:313
138702999200,19cyclictest0-21swapper/311:43:203
138702999200,19cyclictest0-21swapper/311:29:543
138702999200,19cyclictest0-21swapper/310:10:453
138702999200,19cyclictest0-21swapper/309:50:323
138702999200,19cyclictest0-21swapper/308:51:313
138702999200,19cyclictest0-21swapper/307:47:503
138702999200,19cyclictest0-21swapper/307:10:043
138702399208,3cyclictest34-21ksoftirqd/208:27:202
138702399207,3cyclictest34-21ksoftirqd/210:15:412
138702399207,3cyclictest0-21swapper/211:18:512
138702399206,4cyclictest34-21ksoftirqd/208:05:502
1387023992018,1cyclictest0-21swapper/211:53:502
1387023992018,1cyclictest0-21swapper/211:13:332
1387023992018,1cyclictest0-21swapper/210:38:192
1387023992018,1cyclictest0-21swapper/210:09:062
1387023992018,1cyclictest0-21swapper/209:25:052
1387023992018,1cyclictest0-21swapper/207:42:042
1387023992010,3cyclictest34-21ksoftirqd/208:13:162
138702399200,1cyclictest0-21swapper/209:17:542
138702399200,19cyclictest0-21swapper/208:40:242
138702399200,19cyclictest0-21swapper/208:30:372
1387021992018,1cyclictest3056-21PK-Backend09:46:281
1387021992018,1cyclictest3056-21PK-Backend09:44:251
1387021992018,1cyclictest3056-21PK-Backend09:28:291
1387021992018,1cyclictest3056-21PK-Backend09:18:431
1387021992018,1cyclictest0-21swapper/110:10:211
1387021992018,1cyclictest0-21swapper/107:21:491
1387021992017,2cyclictest3056-21PK-Backend12:37:091
1387021992017,2cyclictest3056-21PK-Backend11:39:231
1387021992017,2cyclictest3056-21PK-Backend10:06:291
1387021992017,2cyclictest3056-21PK-Backend07:54:001
1387021992012,6cyclictest1495037-21users12:10:211
138702199200,19cyclictest3056-21PK-Backend11:46:521
138702199200,19cyclictest0-21swapper/111:11:111
138702199200,19cyclictest0-21swapper/111:03:271
138702199200,19cyclictest0-21swapper/110:15:141
138702199200,19cyclictest0-21swapper/107:40:281
138702199200,18cyclictest3056-21PK-Backend12:28:501
1387018992018,1cyclictest0-21swapper/012:21:360
1387018992018,1cyclictest0-21swapper/010:52:340
1387018992018,1cyclictest0-21swapper/010:48:420
1387018992018,1cyclictest0-21swapper/010:02:580
1387018992014,4cyclictest1425405-21cpuspeed_turbos08:55:140
138701899200,4cyclictest1458227-21perf10:30:010
138701899200,19cyclictest0-21swapper/010:30:500
138701899200,19cyclictest0-21swapper/009:46:160
138701899200,19cyclictest0-21swapper/008:41:410
138701899200,19cyclictest0-21swapper/008:30:310
138701899200,19cyclictest0-21swapper/007:26:380
138701899200,19cyclictest0-21swapper/007:24:350
139099190,13rtkit-daemon3056-21PK-Backend09:20:521
138702999199,8cyclictest42-21ksoftirqd/308:25:113
138702999197,11cyclictest42-21ksoftirqd/312:21:053
138702999195,13cyclictest42-21ksoftirqd/309:05:173
1387029991917,1cyclictest0-21swapper/312:30:513
1387029991917,1cyclictest0-21swapper/310:35:533
1387029991917,1cyclictest0-21swapper/309:41:183
1387029991917,1cyclictest0-21swapper/309:20:033
1387029991917,1cyclictest0-21swapper/308:36:023
1387029991917,1cyclictest0-21swapper/308:09:193
1387029991914,4cyclictest42-21ksoftirqd/309:30:013
1387029991914,3cyclictest1416723-21df_inode08:30:153
1387029991913,5cyclictest1474639-21expr11:15:143
1387029991912,6cyclictest1438649-21sed09:30:213
138702999191,16cyclictest1404269-21df07:55:143
138702999190,1cyclictest3056-21PK-Backend11:20:363
138702999190,1cyclictest0-21swapper/310:21:483
138702999190,18cyclictest3056-21PK-Backend11:07:233
138702999190,18cyclictest0-21swapper/310:45:273
138702399192,13cyclictest1448079-21apt10:00:122
1387023991917,1cyclictest0-21swapper/209:30:122
1387023991917,1cyclictest0-21swapper/207:36:292
1387023991911,6cyclictest1498750-21sshd12:23:182
1387023991911,6cyclictest1496073-21df12:15:142
138702399190,7cyclictest0-21swapper/209:55:172
1387021991917,1cyclictest3056-21PK-Backend12:31:411
1387021991917,1cyclictest3056-21PK-Backend11:29:021
1387021991917,1cyclictest3056-21PK-Backend09:55:401
1387021991917,1cyclictest3056-21PK-Backend07:46:021
1387021991917,1cyclictest0-21swapper/107:27:431
138702199190,1cyclictest3056-21PK-Backend11:51:321
138702199190,1cyclictest0-21swapper/107:30:471
1387018991917,1cyclictest3056-21PK-Backend12:35:070
1387018991917,1cyclictest0-21swapper/011:58:300
1387018991917,1cyclictest0-21swapper/011:07:000
1387018991917,1cyclictest0-21swapper/010:36:310
1387018991917,1cyclictest0-21swapper/010:06:510
1387018991917,1cyclictest0-21swapper/009:53:120
1387018991917,1cyclictest0-21swapper/009:18:110
1387018991914,3cyclictest1401879-21cron07:50:000
1387018991913,4cyclictest13-21ksoftirqd/007:10:140
138701899190,1cyclictest0-21swapper/011:50:130
138701899190,1cyclictest0-21swapper/011:11:400
138701899190,1cyclictest0-21swapper/011:04:570
138701899190,1cyclictest0-21swapper/009:31:350
138701899190,1cyclictest0-21swapper/007:59:230
138701899190,18cyclictest0-21swapper/011:30:320
138701899190,18cyclictest0-21swapper/010:14:210
1387029991816,1cyclictest3056-21PK-Backend11:13:053
1387029991816,1cyclictest3056-21PK-Backend07:43:163
138702399186,3cyclictest34-21ksoftirqd/208:00:052
138702399185,2cyclictest141rcu_preempt11:03:352
1387023991816,1cyclictest0-21swapper/209:12:012
1387021991816,1cyclictest3056-21PK-Backend12:23:231
1387021991816,1cyclictest3056-21PK-Backend12:16:541
1387021991816,1cyclictest3056-21PK-Backend09:50:331
1387021991816,1cyclictest3056-21PK-Backend09:37:291
138702199180,1cyclictest3056-21PK-Backend12:08:231
138701899182,14cyclictest1446490-21cpuspeed_turbos09:55:110
1387018991816,1cyclictest0-21swapper/012:06:480
1387018991816,1cyclictest0-21swapper/011:39:520
1387018991812,5cyclictest1495643-21sed12:15:010
1387018991811,5cyclictest1391125-21cat07:20:010
138701899180,1cyclictest0-21swapper/012:32:170
1387029991715,1cyclictest0-21swapper/310:15:533
1387029991715,1cyclictest0-21swapper/307:25:263
1387029991712,4cyclictest42-21ksoftirqd/310:25:113
1387029991712,3cyclictest1443680-21/usr/sbin/munin09:45:213
1387029991711,5cyclictest1432217-21apt09:15:143
1387029991710,5cyclictest1395908-21ntpq07:30:203
138702999170,15cyclictest1494357-21hddtemp_smartct12:10:143
1387021991715,1cyclictest3056-21PK-Backend08:30:571
1387018991715,1cyclictest3056-21PK-Backend08:45:330
1387018991715,1cyclictest0-21swapper/012:17:580
1387018991712,4cyclictest1398794-21sort07:40:120
14363442160,7chrt13-21ksoftirqd/009:25:130
138702399167,3cyclictest34-21ksoftirqd/209:47:142
138702199169,5cyclictest1406057-21basename08:00:151
1387018991613,2cyclictest0-21swapper/009:07:490
138702999151,3cyclictest141rcu_preempt07:35:213
1387018991513,1cyclictest3056-21PK-Backend08:15:590
1387018991512,2cyclictest1440866-21cron09:39:590
138702199148,5cyclictest1411775-21cat08:15:211
1387021991412,1cyclictest3056-21PK-Backend08:24:221
1387021991412,1cyclictest3056-21PK-Backend08:14:421
138701899147,5cyclictest1415073-21awk08:25:150
1387018991412,1cyclictest0-21swapper/011:48:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional