You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-27 - 15:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Apr 27, 2026 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
160972721160,15sleep2341ksoftirqd/207:05:252
139099790,61rtkit-daemon0-21swapper/007:05:150
161014427757,14sleep10-21swapper/107:06:331
161015126344,14sleep30-21swapper/307:06:383
16313072520,3sleep20-21swapper/208:05:202
16347022400,2sleep00-21swapper/008:15:210
161056999381,23cyclictest1626051-21python307:50:200
161056999381,22cyclictest1673690-21python310:05:220
161056999380,24cyclictest1689551-21python310:50:210
161056999371,20cyclictest1715557-21python312:05:200
161056999370,25cyclictest0-21swapper/010:15:110
161056999370,16cyclictest0-21swapper/010:00:010
161056999370,16cyclictest0-21swapper/009:25:010
161056999365,17cyclictest1651467-21taskset09:05:100
161056999361,24cyclictest1895067-21snapd10:34:130
161056999361,19cyclictest1629514-21python308:00:200
161056999360,27cyclictest1717175-21sh12:10:180
161056999360,26cyclictest1725371-21apt12:35:100
161056999360,25cyclictest0-21swapper/012:20:110
161056999360,25cyclictest0-21swapper/009:50:110
161056999360,23cyclictest1653481-21cat09:10:130
161056999351,22cyclictest1724501-21python312:30:200
161056999351,18cyclictest1613387-21hddtemp_smartct07:15:160
161056999350,26cyclictest1656221-21sed09:15:220
161056999350,23cyclictest1713128-21expr12:00:130
161056999350,23cyclictest1647744-21/usr/sbin/munin08:55:240
161056999350,20cyclictest1719824-21date12:20:000
161056999350,20cyclictest0-21swapper/010:50:010
161056999350,17cyclictest1667523-21latency_hist09:50:000
161056999350,16cyclictest0-21swapper/010:45:000
161056999341,19cyclictest1696367-21ntp_states11:10:180
161056999340,24cyclictest0-21swapper/011:50:110
161056999340,23cyclictest1650124-21cat09:00:180
161056999340,20cyclictest1660896-21irqstats09:30:160
161056999340,20cyclictest1659547-21python309:25:210
161056999340,20cyclictest1631122-21ntp_states08:05:180
161056999340,20cyclictest1628376-21sh07:59:590
161056999340,20cyclictest0-21swapper/007:45:110
161056999340,18cyclictest1611503-21ntp_states07:10:190
161056999340,18cyclictest0-21swapper/007:20:110
161056999333,16cyclictest0-21swapper/011:01:270
161056999332,17cyclictest1617323-21python307:25:230
161056999331,23cyclictest1699208-21cpuspeed_turbos11:20:120
161056999331,21cyclictest1665556-21nm-dispatcher09:44:300
161056999330,24cyclictest1674525-21chrt10:10:040
161056999330,23cyclictest1680682-21needreboot10:25:190
161056999330,18cyclictest1678958-21needreboot10:20:170
161056999320,22cyclictest1646236-21expr08:50:150
161056999320,21cyclictest0-21swapper/010:00:120
161056999320,19cyclictest1694640-21ntp_states11:05:190
161056999320,19cyclictest1638397-21users08:25:240
161056999320,19cyclictest1636431-21ps08:20:190
161056999320,18cyclictest1722963-21users12:25:210
161056999320,18cyclictest1722963-21users12:25:210
161056999320,18cyclictest1640786-21apt08:35:110
161056999320,18cyclictest1640786-21apt08:35:110
161056999310,18cyclictest1691287-21python310:55:210
161056999301,20cyclictest0-21swapper/007:30:140
161056999290,18cyclictest1684798-21taskset10:38:080
161056999290,15cyclictest0-21swapper/008:30:170
161056999280,19cyclictest1662819-21smartctl09:35:170
161056999280,16cyclictest0-21swapper/008:15:010
1610580992710,4cyclictest42-21ksoftirqd/310:03:463
1610575992711,4cyclictest34-21ksoftirqd/211:20:012
161056999270,15cyclictest1698663-21pool-udisksd11:18:040
16886512250,9sleep234-21ksoftirqd/210:50:012
16349262250,4sleep20-21swapper/208:15:232
161058099258,3cyclictest141rcu_preempt09:25:113
161058099253,2cyclictest141rcu_preempt10:10:013
161058099250,5cyclictest3056-21PK-Backend10:55:013
1610575992514,4cyclictest34-21ksoftirqd/212:35:012
1610575992512,4cyclictest34-21ksoftirqd/208:00:012
1610575992511,3cyclictest34-21ksoftirqd/208:15:012
161056999258,15cyclictest1095-21runrttasks11:48:370
161058099246,10cyclictest42-21ksoftirqd/307:50:013
161058099244,10cyclictest42-21ksoftirqd/309:40:223
1610580992414,6cyclictest1683494-21apt10:35:113
161056999245,17cyclictest1704581-21expr11:35:140
161056999240,9cyclictest16443762tr08:45:120
161056999240,23cyclictest0-21swapper/011:32:270
161056999240,22cyclictest1712857-21ls12:00:010
161056999240,22cyclictest1643605-21vmstat08:40:230
161058099236,3cyclictest42-21ksoftirqd/309:15:133
1610580992315,6cyclictest1698071-21ntpq11:15:183
161058099230,3cyclictest1709032-21taskset11:47:403
161057599239,2cyclictest141rcu_preempt11:04:322
161057599237,4cyclictest34-21ksoftirqd/207:25:232
1610575992312,5cyclictest34-21ksoftirqd/207:30:202
161057599230,22cyclictest1389-21rtkit-daemon11:36:102
161056999238,13cyclictest1621454-21sh07:40:000
161056999238,13cyclictest1621454-21sh07:39:590
161056999230,22cyclictest0-21swapper/011:41:260
161056999230,21cyclictest1622572-21grep07:40:200
139099230,2rtkit-daemon3056-21PK-Backend10:57:011
16821232220,2sleep20-21swapper/210:30:122
161058099227,4cyclictest42-21ksoftirqd/309:23:563
1610580992220,1cyclictest3056-21PK-Backend12:21:343
1610580992220,1cyclictest3056-21PK-Backend12:07:153
1610580992220,1cyclictest3056-21PK-Backend09:07:073
1610580992220,1cyclictest3056-21PK-Backend08:26:173
1610580992220,1cyclictest0-21swapper/311:28:073
1610580992220,1cyclictest0-21swapper/308:47:403
161058099220,7cyclictest1611722-21sort07:10:213
161058099220,21cyclictest3056-21PK-Backend11:23:413
161058099220,20cyclictest3056-21PK-Backend07:32:313
161058099220,20cyclictest3056-21PK-Backend07:20:413
1610575992210,4cyclictest34-21ksoftirqd/210:45:002
1610575992210,4cyclictest34-21ksoftirqd/207:20:002
1610571992220,1cyclictest0-21swapper/110:53:521
1610571992214,6cyclictest1724498-21python312:30:191
161057199220,20cyclictest3056-21PK-Backend08:50:441
161056999221,19cyclictest1701316-21kernelversion11:25:160
139099220,14rtkit-daemon0-21swapper/108:33:061
16537262210,2sleep33056-21PK-Backend09:10:183
161058099218,3cyclictest42-21ksoftirqd/311:36:123
1610580992119,1cyclictest3056-21PK-Backend12:31:283
1610580992119,1cyclictest3056-21PK-Backend10:20:493
1610580992119,1cyclictest3056-21PK-Backend07:55:423
1610580992119,1cyclictest0-21swapper/308:11:453
1610580992118,2cyclictest3056-21PK-Backend12:19:453
1610580992118,2cyclictest3056-21PK-Backend10:18:113
1610580992116,4cyclictest42-21ksoftirqd/311:05:003
1610580992114,5cyclictest1711696-21interrupts11:55:153
1610580992113,3cyclictest42-21ksoftirqd/312:36:273
161058099210,6cyclictest3056-21PK-Backend08:00:163
161058099210,6cyclictest1663306-21sort09:35:233
161058099210,5cyclictest3056-21PK-Backend08:35:123
161058099210,5cyclictest3056-21PK-Backend08:35:123
161058099210,20cyclictest3056-21PK-Backend08:05:293
161058099210,20cyclictest3056-21PK-Backend07:25:213
161058099210,20cyclictest0-21swapper/307:51:573
161058099210,20cyclictest0-21swapper/307:41:433
161057599218,4cyclictest34-21ksoftirqd/209:29:592
161057599218,3cyclictest141rcu_preempt10:15:022
161057599217,5cyclictest34-21ksoftirqd/208:35:202
161057599217,5cyclictest34-21ksoftirqd/208:35:192
161057599217,3cyclictest34-21ksoftirqd/212:20:002
1610575992119,1cyclictest0-21swapper/210:48:102
1610575992115,2cyclictest34-21ksoftirqd/207:45:202
161057599211,19cyclictest0-21swapper/208:04:262
161057599210,20cyclictest0-21swapper/210:12:422
161057599210,20cyclictest0-21swapper/209:44:302
161057599210,20cyclictest0-21swapper/208:41:372
1610571992119,1cyclictest3056-21PK-Backend11:35:161
1610571992119,1cyclictest3056-21PK-Backend08:45:361
1610571992119,1cyclictest3056-21PK-Backend07:41:421
1610571992119,1cyclictest3056-21PK-Backend07:35:131
1610571992119,1cyclictest3056-21PK-Backend07:35:131
1610571992119,1cyclictest0-21swapper/112:05:511
1610571992119,1cyclictest0-21swapper/111:06:301
1610571992119,1cyclictest0-21swapper/110:28:101
1610571992119,1cyclictest0-21swapper/109:45:251
1610571992119,1cyclictest0-21swapper/109:41:471
1610571992119,1cyclictest0-21swapper/108:27:181
1610571992119,1cyclictest0-21swapper/108:15:271
1610571992119,1cyclictest0-21swapper/107:27:081
1610571992119,1cyclictest0-21swapper/107:17:491
1610571992118,2cyclictest3056-21PK-Backend11:03:051
1610571992113,6cyclictest436-20systemd-journal10:00:011
1610571992113,6cyclictest1687537-21memory10:45:171
1610571992113,6cyclictest1685589-21df10:40:141
161057199210,20cyclictest3056-21PK-Backend08:40:291
161057199210,20cyclictest0-21swapper/111:12:321
161057199210,20cyclictest0-21swapper/109:25:181
139099210,13rtkit-daemon0-21swapper/112:20:181
139099210,13rtkit-daemon0-21swapper/107:55:301
161058099207,3cyclictest141rcu_preempt11:05:153
161058099207,2cyclictest141rcu_preempt10:11:213
1610580992018,1cyclictest3056-21PK-Backend08:30:373
1610580992018,1cyclictest3056-21PK-Backend08:15:303
1610580992018,1cyclictest0-21swapper/310:26:033
1610580992018,1cyclictest0-21swapper/307:37:443
1610580992018,1cyclictest0-21swapper/307:37:443
161058099201,6cyclictest1614452-21grep07:20:013
1610580992010,3cyclictest42-21ksoftirqd/311:10:303
161058099200,8cyclictest1667798-21tr09:50:123
161058099200,7cyclictest1647984-21seq08:55:153
161058099200,1cyclictest3056-21PK-Backend09:55:483
161058099200,1cyclictest0-21swapper/310:59:223
161058099200,18cyclictest3056-21PK-Backend10:33:533
161057599208,2cyclictest141rcu_preempt08:34:592
161057599206,3cyclictest34-21ksoftirqd/207:24:552
1610575992018,1cyclictest0-21swapper/212:02:322
1610575992018,1cyclictest0-21swapper/211:29:412
1610575992018,1cyclictest0-21swapper/209:33:212
1610575992018,1cyclictest0-21swapper/207:43:562
1610575992017,2cyclictest3056-21PK-Backend07:10:182
1610575992014,4cyclictest34-21ksoftirqd/211:55:202
161057599200,1cyclictest3056-21PK-Backend10:03:222
161057599200,1cyclictest0-21swapper/212:11:392
161057599200,1cyclictest0-21swapper/208:29:132
161057599200,19cyclictest0-21swapper/212:26:252
161057599200,19cyclictest0-21swapper/212:26:252
161057599200,19cyclictest0-21swapper/212:06:102
161057599200,19cyclictest0-21swapper/211:33:532
161057599200,19cyclictest0-21swapper/210:29:252
161057599200,19cyclictest0-21swapper/210:22:212
161057599200,19cyclictest0-21swapper/209:51:592
161057599200,19cyclictest0-21swapper/208:55:212
1610571992018,1cyclictest3056-21PK-Backend11:57:001
1610571992018,1cyclictest3056-21PK-Backend11:40:291
1610571992018,1cyclictest0-21swapper/112:01:261
1610571992018,1cyclictest0-21swapper/111:51:461
1610571992018,1cyclictest0-21swapper/110:19:321
1610571992018,1cyclictest0-21swapper/108:20:291
1610571992018,1cyclictest0-21swapper/108:05:281
1610571992013,6cyclictest1633387-21sshd08:12:421
1610571992012,6cyclictest1716790-21df12:10:121
1610571992012,6cyclictest1674558-21apt10:10:111
161057199201,11cyclictest1653475-21cat09:10:141
161057199200,19cyclictest0-21swapper/112:17:211
161057199200,19cyclictest0-21swapper/111:49:561
161057199200,19cyclictest0-21swapper/110:38:391
161057199200,19cyclictest0-21swapper/110:33:241
161057199200,19cyclictest0-21swapper/107:31:141
161057199200,19cyclictest0-21swapper/107:12:221
161058099195,4cyclictest42-21ksoftirqd/308:22:323
1610580991917,1cyclictest3056-21PK-Backend11:51:193
1610580991917,1cyclictest0-21swapper/310:40:083
1610580991915,3cyclictest42-21ksoftirqd/311:45:013
161058099190,5cyclictest3056-21PK-Backend12:00:213
161058099190,1cyclictest3056-21PK-Backend12:14:183
161058099190,1cyclictest3056-21PK-Backend10:45:223
161058099190,1cyclictest0-21swapper/311:32:333
161057599197,5cyclictest34-21ksoftirqd/210:05:222
161057599197,4cyclictest34-21ksoftirqd/212:40:002
161057599197,3cyclictest34-21ksoftirqd/212:20:192
1610575991917,1cyclictest3056-21PK-Backend09:35:242
1610575991917,1cyclictest0-21swapper/209:13:002
161057599191,4cyclictest34-21ksoftirqd/209:56:042
1610575991911,6cyclictest1710158-21tail11:50:182
161057599190,1cyclictest0-21swapper/211:46:242
161057599190,1cyclictest0-21swapper/208:50:022
161057599190,18cyclictest0-21swapper/211:44:352
161057599190,18cyclictest0-21swapper/211:24:002
1610571991917,1cyclictest0-21swapper/112:29:041
1610571991917,1cyclictest0-21swapper/112:29:031
1610571991917,1cyclictest0-21swapper/111:17:451
1610571991917,1cyclictest0-21swapper/109:01:591
1610571991916,2cyclictest3056-21PK-Backend11:28:131
1610571991914,3cyclictest1630116-21idleruntime-cro08:05:011
1610571991911,6cyclictest1667889-21cat09:50:131
161057199190,1cyclictest3056-21PK-Backend11:30:501
161057199190,1cyclictest0-21swapper/112:37:071
161057199190,1cyclictest0-21swapper/109:39:571
161057199190,18cyclictest0-21swapper/108:59:561
161058099188,3cyclictest42-21ksoftirqd/308:41:263
1610580991816,1cyclictest3056-21PK-Backend09:49:183
1610580991813,3cyclictest1646360-21diskstats08:50:153
1610580991811,2cyclictest42-21ksoftirqd/309:34:233
1610580991811,2cyclictest42-21ksoftirqd/309:00:393
161058099180,7cyclictest3056-21PK-Backend12:28:043
161058099180,7cyclictest3056-21PK-Backend12:28:043
161057599185,4cyclictest34-21ksoftirqd/209:08:522
161057599184,4cyclictest34-21ksoftirqd/209:00:182
161057599183,4cyclictest0-21swapper/209:46:092
161057599182,8cyclictest34-21ksoftirqd/209:20:032
1610575991816,1cyclictest0-21swapper/211:10:412
1610575991812,5cyclictest1683729-21cat10:35:132
161057599180,5cyclictest0-21swapper/210:55:162
161057599180,1cyclictest0-21swapper/207:37:152
161057599180,1cyclictest0-21swapper/207:37:152
1610571991816,1cyclictest0-21swapper/110:05:391
1610571991816,1cyclictest0-21swapper/110:00:591
1610571991816,1cyclictest0-21swapper/109:08:541
1610571991816,1cyclictest0-21swapper/107:54:121
1610571991811,6cyclictest1679338-21sed10:20:211
1610571991811,6cyclictest1657704-21open_files09:20:201
1610575991710,5cyclictest1644453-21cat08:45:142
1610571991715,1cyclictest0-21swapper/108:37:111
1610571991715,1cyclictest0-21swapper/108:37:111
1610571991712,3cyclictest1655887-21strings09:15:191
1610571991710,5cyclictest1615446-21ntpq07:20:171
161057599164,3cyclictest34-21ksoftirqd/209:15:102
1610575991614,1cyclictest0-21swapper/207:50:402
1610575991610,5cyclictest1693994-21wc11:05:112
161057199167,7cyclictest1623822-21df_inode07:45:121
161057599158,2cyclictest34-21ksoftirqd/208:20:172
161057199159,4cyclictest1660507-21cpuspeed09:30:111
1610571991511,3cyclictest1700174-21users11:20:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional