You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 11:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Dec 30, 2025 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
155029021130,17sleep234-21ksoftirqd/219:15:302
154647828442,15sleep00-21swapper/019:05:430
197399810,71rtkit-daemon0-21swapper/119:09:351
15634222790,4sleep21563252-21latency_hist19:55:012
197399750,6rtkit-daemon0-21swapper/322:56:193
16517032710,4chrt1651702-21sshd00:04:303
15598082670,6sleep20-21swapper/219:44:592
15748112660,14sleep0131ksoftirqd/020:25:300
1546976996618,20cyclictest1566909-21latency_hist20:05:013
1546976996618,20cyclictest1566909-21latency_hist20:05:013
154677726546,13sleep20-21swapper/219:09:362
154664826343,14sleep30-21swapper/319:07:563
1546976996121,35cyclictest2080-21gdbus21:56:083
1546976996115,20cyclictest1630405-21grep23:05:023
1546976996113,31cyclictest1597265-21cstates21:30:153
1546976996113,31cyclictest1597265-21cstates21:30:143
197399600,8rtkit-daemon1972-21rtkit-daemon00:25:450
1546976996015,22cyclictest1624873-21perf22:49:593
1546976995916,16cyclictest1560932-21python319:45:263
1546976995816,17cyclictest1611650-21df22:10:163
1546976995813,19cyclictest1602879-21cat21:45:213
1546976995716,16cyclictest1564004-21irqstats19:55:213
1546976995716,15cyclictest1608627-21awk22:00:293
1546976995715,37cyclictest1661851-21taskset00:31:243
154697199578,6cyclictest141rcu_preempt21:30:012
1546976995616,18cyclictest1650016-21sh23:59:593
1546976995616,18cyclictest1610699-21taskset22:06:323
1546976995613,16cyclictest1604718-21ntp_states21:50:243
1546976995613,16cyclictest1604718-21ntp_states21:50:233
1546976995516,16cyclictest1592305-21ntp_states21:15:253
1546976995515,24cyclictest1556172-21taskset19:33:083
1546976995515,19cyclictest1639155-21taskset23:28:033
1546976995515,16cyclictest1576385-21ntp_states20:30:233
1546976995514,14cyclictest1549135-21hddtemp_smartct19:15:053
1546976995419,30cyclictest1577761-21forks20:35:203
1546976995417,14cyclictest1571026-21memory20:15:243
1546976995416,16cyclictest1640345-21ntp_states23:30:273
1546976995416,16cyclictest1640345-21ntp_states23:30:273
1546976995415,24cyclictest1637479-21taskset23:23:453
1546976995415,17cyclictest1593980-21cat21:20:183
1546976995415,16cyclictest1647952-21taskset23:51:323
1546976995415,16cyclictest1632138-21ls23:10:013
1546976995415,16cyclictest1632138-21ls23:10:013
1546976995317,14cyclictest1572759-21memory20:20:223
1546976995315,14cyclictest1589711-21wc21:10:033
1546976995315,14cyclictest1589711-21wc21:10:023
1546976995315,14cyclictest0-21swapper/320:45:253
1546976995314,18cyclictest1551502-21cpuspeed_turbos19:20:153
1546976995212,15cyclictest1569086-21df_inode20:10:183
1546976995124,23cyclictest1599653-21python321:35:283
1546976995120,27cyclictest1562530-21ntpq19:50:243
1546976995116,14cyclictest1554742-21latency_hist19:30:023
1546976995115,14cyclictest1643459-21smartctl23:40:203
15793232500,5sleep0161rcuc/020:40:160
1546976995021,25cyclictest1626173-21sed22:50:303
1546976995021,24cyclictest1634229-21apt-config23:15:023
1546976995020,25cyclictest1574627-21tr20:25:253
1546976995015,15cyclictest1587513-40lsb_release21:02:023
1546976995015,13cyclictest1614735-21/usr/sbin/munin22:20:283
1546976995015,12cyclictest1596481-21taskset21:26:213
1546976995014,14cyclictest1580075-21cat20:40:313
1546976995013,14cyclictest1660421-21turbostat00:30:003
1546976995013,14cyclictest1657973-21ntpq00:20:263
1546976995010,23cyclictest1618541-21cat22:30:153
1546976995010,17cyclictest0-21swapper/319:37:563
1546976995010,17cyclictest0-21swapper/319:37:563
1546976994924,21cyclictest1591487-21sort21:15:013
1546976994922,22cyclictest1621655-21/usr/sbin/munin22:40:213
1546976994922,22cyclictest1621655-21/usr/sbin/munin22:40:203
1546976994919,26cyclictest1616927-21if_enp2s022:25:203
1546976994919,26cyclictest1559436-21seq19:40:353
1546976994821,23cyclictest1656667-21taskset00:16:303
1546976994821,22cyclictest1600847-21cat21:40:153
1546976994820,23cyclictest1620275-21expr22:35:163
1546976994819,24cyclictest1652805-21/usr/sbin/munin00:05:273
1546976994815,16cyclictest1568491-21sed20:10:013
1546976994719,23cyclictest1584666-21diskstats20:55:173
1546976994714,29cyclictest0-21swapper/323:35:133
1546976994622,20cyclictest1654645-21switchtime00:10:313
1546976994617,25cyclictest1645623-21processes23:45:263
1546976994615,26cyclictest1662863-21if_enp2s000:35:243
1546976994519,21cyclictest438-20systemd-journal22:18:303
1546976994518,22cyclictest1632366-21apt23:10:113
16470372440,6sleep00-21swapper/023:50:140
1546976994422,17cyclictest1548521-21seq19:14:033
1546966994424,7cyclictest1568851-21cut20:10:160
1546968994319,10cyclictest1588027-21systemd-udevd21:05:101
1546968994319,10cyclictest1588027-21systemd-udevd21:05:101
1546976994219,19cyclictest1583808-21sshd20:52:293
1546971994228,10cyclictest1564988-21turbostat.cron20:00:002
1546968994226,12cyclictest1633817-21apt-get23:15:021
1546971994113,7cyclictest16350442netstat23:15:242
154696899415,11cyclictest26-21ksoftirqd/122:16:141
154696699411,36cyclictest1561523-21turbostat19:50:010
16147602400,5sleep20-21swapper/222:20:142
15788292400,17sleep126-21ksoftirqd/120:40:001
1546971994016,20cyclictest1624901-21unin-run22:50:002
154696899405,10cyclictest26-21ksoftirqd/121:04:111
1546966994012,6cyclictest141rcu_preempt22:45:250
197399390,27rtkit-daemon4497-21PK-Backend19:36:062
197399390,27rtkit-daemon4497-21PK-Backend19:36:062
197399390,25rtkit-daemon0-21swapper/221:00:552
1546971993920,6cyclictest34-21ksoftirqd/200:40:002
1546971993919,6cyclictest34-21ksoftirqd/222:10:152
1546971993918,16cyclictest1658093-21/usr/sbin/munin00:20:282
1546971993916,14cyclictest34-21ksoftirqd/219:45:282
1546968993923,12cyclictest923-21dbus-daemon22:10:131
154696899391,33cyclictest15647902sleep119:57:431
154696699396,6cyclictest141rcu_preempt21:20:000
1546966993921,13cyclictest1565957-21awk20:00:220
1546966993921,13cyclictest1565957-21awk20:00:220
1546966993912,8cyclictest13-21ksoftirqd/021:32:140
1546966993912,8cyclictest13-21ksoftirqd/021:32:140
1546966993912,7cyclictest13-21ksoftirqd/019:16:000
197399380,25rtkit-daemon0-21swapper/220:41:152
1546971993827,7cyclictest1547130-21sort19:10:132
1546971993821,12cyclictest1602638-21fw_packets21:45:162
1546968993824,10cyclictest1598503-21cron21:34:591
1546968993824,10cyclictest1598503-21cron21:34:591
1546966993824,10cyclictest1036-21gdbus19:29:570
1546966993821,13cyclictest1625864-21ntpq22:50:240
1546966993821,13cyclictest1567853-21python320:05:290
1546966993814,7cyclictest1644515-21cron23:45:000
1546966993811,8cyclictest13-21ksoftirqd/023:02:300
1546966993811,8cyclictest13-21ksoftirqd/022:13:400
16586712374,10sleep00-21swapper/000:25:000
16326082370,5sleep00-21swapper/023:10:140
154697199372,7cyclictest1620535-21memory22:35:222
1546971993722,11cyclictest1585743-21apt-get21:00:012
1546971993720,13cyclictest1552656-40dpkg19:22:562
1546971993718,14cyclictest1601383-21python321:40:272
1546971993712,5cyclictest15737092latency_hist20:25:032
154697199371,12cyclictest1588033-21systemd-udevd21:05:112
154697199371,12cyclictest1588033-21systemd-udevd21:05:112
154696899374,13cyclictest141rcu_preempt19:45:181
1546968993722,11cyclictest1648285-21munin-run23:55:011
1546968993722,11cyclictest1551354-21sort19:20:141
1546968993721,11cyclictest1643834-21pluginstate23:40:261
1546968993720,13cyclictest26-21ksoftirqd/120:46:401
1546966993721,12cyclictest1605050-21uptime21:50:300
1546966993721,12cyclictest1605050-21uptime21:50:290
1546966993721,12cyclictest1556331-21sh19:35:010
1546966993720,13cyclictest1639957-21df23:30:170
1546966993720,13cyclictest1639957-21df23:30:170
1546966993720,13cyclictest1594831-40apt-check21:22:560
1546966993719,13cyclictest1572924-21ntp_states20:20:250
197399360,23rtkit-daemon0-21swapper/222:05:062
154697199366,26cyclictest1651790-21systemd00:05:012
154697199364,11cyclictest34-21ksoftirqd/223:50:162
1546971993621,11cyclictest1556319-21cron19:35:012
1546971993620,11cyclictest438-20systemd-journal20:17:002
1546971993620,11cyclictest1590699-21swap21:10:292
1546971993619,12cyclictest1597500-21iwlist21:30:202
1546971993619,12cyclictest1597500-21iwlist21:30:192
154696899366,6cyclictest15939972cat21:20:191
154696899365,10cyclictest141rcu_preempt22:50:321
1546968993622,10cyclictest1591272-21cron21:14:591
1546968993620,12cyclictest1585236-21sshd20:55:301
154696899361,9cyclictest26-21ksoftirqd/121:46:241
1546968993617,14cyclictest26-21ksoftirqd/120:50:101
154696699368,10cyclictest13-21ksoftirqd/022:58:440
154696699366,8cyclictest16558692entropy00:15:190
1546966993625,7cyclictest161rcuc/021:45:110
1546966993624,8cyclictest1639316-21dump-pmu-power23:30:000
1546966993621,11cyclictest1556861-21cat19:35:150
1546966993621,11cyclictest1556861-21cat19:35:150
1546966993618,14cyclictest1622043-21df22:40:150
1546966993618,14cyclictest1622043-21df22:40:150
197399350,24rtkit-daemon0-21swapper/222:56:492
16307142350,13sleep2331rcuc/223:05:142
16307142350,13sleep2331rcuc/223:05:142
16111072350,7sleep04497-21PK-Backend22:10:020
1546971993520,11cyclictest1592934-21seq21:19:072
1546971993519,12cyclictest923-21dbus-daemon23:15:002
1546971993519,12cyclictest1-21systemd22:00:012
1546971993518,12cyclictest466-21systemd-udevd22:04:402
1546971993517,14cyclictest1643423-21df_inode23:40:162
1546968993526,6cyclictest1918-21kerneloops21:19:031
1546968993525,7cyclictest1614946-21cpuspeed_turbos22:20:161
1546968993525,6cyclictest1570162-21cron20:15:001
1546968993524,7cyclictest1636316-21tr23:20:141
1546968993522,8cyclictest466-21systemd-udevd23:16:401
1546968993520,10cyclictest1-21systemd19:45:011
154696899351,9cyclictest4497-21PK-Backend19:26:091
1546968993519,12cyclictest1662980-21cat00:35:241
1546968993519,11cyclictest1616956-21interrupts22:25:201
1546968993518,13cyclictest1661410-21ntpq00:30:251
154696899351,15cyclictest4497-21PK-Backend00:05:301
154696899351,11cyclictest26-21ksoftirqd/119:30:261
154696699358,10cyclictest13-21ksoftirqd/021:42:570
1546966993525,7cyclictest161rcuc/021:05:430
1546966993525,7cyclictest161rcuc/021:05:430
1546966993525,6cyclictest161rcuc/022:29:370
1546966993525,6cyclictest161rcuc/019:10:330
1546966993525,6cyclictest1614491-21turbostat22:20:000
1546966993524,8cyclictest1606636-21tail21:55:240
1546966993524,7cyclictest161rcuc/023:21:460
1546966993523,4cyclictest0-21swapper/021:01:590
1546966993518,12cyclictest1589916-21cpuspeed_turbos21:10:130
1546971993424,6cyclictest1630354-21cron23:05:002
1546971993424,6cyclictest1625333-21cpuspeed_turbos22:50:142
1546971993422,9cyclictest1614497-21munin-plugin-st22:20:002
1546971993422,8cyclictest1641064-21grep23:35:002
1546971993422,8cyclictest1641064-21grep23:34:592
1546971993421,9cyclictest1553263-21tr19:25:152
1546971993420,10cyclictest1654865-21seq00:10:382
1546971993419,11cyclictest1652319-21expr00:05:142
1546971993418,12cyclictest2193-21gdbus23:39:562
1546971993418,12cyclictest1583006-21cat20:50:202
1546971993418,12cyclictest1574311-21which20:25:212
1546971993418,11cyclictest1616966-21awk22:25:212
1546971993416,6cyclictest34-21ksoftirqd/223:21:512
1546971993411,6cyclictest141rcu_preempt20:03:562
1546971993411,6cyclictest141rcu_preempt20:03:562
154696899344,9cyclictest26-21ksoftirqd/122:42:501
154696899344,9cyclictest26-21ksoftirqd/122:42:501
1546968993420,9cyclictest1561976-21tr19:50:131
1546968993420,10cyclictest26-21ksoftirqd/120:05:181
1546968993419,11cyclictest26-21ksoftirqd/119:35:181
1546968993419,11cyclictest26-21ksoftirqd/119:35:171
1546968993418,12cyclictest923-21dbus-daemon20:25:011
1546968993418,11cyclictest466-21systemd-udevd22:36:401
154696899341,5cyclictest141rcu_preempt23:05:591
154696899341,5cyclictest141rcu_preempt23:05:591
154696899341,5cyclictest141rcu_preempt23:00:361
154696899341,5cyclictest141rcu_preempt21:42:011
154696899341,29cyclictest4497-21PK-Backend21:50:371
154696899341,29cyclictest4497-21PK-Backend21:50:371
154696699343,6cyclictest141rcu_preempt19:45:000
1546966993425,5cyclictest1578839-21turbostat20:40:010
1546966993424,7cyclictest1-21systemd00:05:010
1546966993418,12cyclictest1620900-21threads22:35:300
1546966993418,11cyclictest1615390-21cat22:20:240
154696699341,29cyclictest4497-21PK-Backend23:58:030
154696699341,29cyclictest4497-21PK-Backend23:19:360
154697199339,5cyclictest141rcu_preempt20:35:232
1546971993320,9cyclictest2069-21gnome-shell21:51:592
1546971993320,9cyclictest2069-21gnome-shell21:51:582
1546971993318,11cyclictest1599647-21gunzip21:35:282
1546971993318,10cyclictest1576720-21users20:30:282
1546971993317,12cyclictest1638742-21awk23:25:272
1546971993317,12cyclictest1623131-21cron22:45:002
1546971993317,12cyclictest1623131-21cron22:44:592
1546971993317,11cyclictest1649132-21cut23:55:232
1546971993316,12cyclictest438-20systemd-journal00:19:562
1546971993315,13cyclictest438-20systemd-journal21:25:002
1546971993315,13cyclictest1661786-21awk00:30:332
154697199331,29cyclictest0-21swapper/220:45:372
1546971993311,11cyclictest1568208-21sshd20:07:122
154696899338,6cyclictest141rcu_preempt19:15:161
154696899337,6cyclictest141rcu_preempt22:07:591
154696899337,5cyclictest141rcu_preempt20:00:201
154696899337,5cyclictest141rcu_preempt20:00:191
1546968993322,7cyclictest1547312-21tr19:10:141
1546968993319,10cyclictest1595824-21kernelversion21:25:231
1546968993318,11cyclictest1579988-21awk20:40:291
1546968993317,11cyclictest438-20systemd-journal22:56:291
1546968993317,11cyclictest1606608-21cat21:55:231
1546968993316,12cyclictest1648642-21cpuspeed_turbos23:55:171
154696899331,4cyclictest141rcu_preempt23:30:111
154696899331,4cyclictest141rcu_preempt23:30:101
154696899331,29cyclictest4497-21PK-Backend21:37:111
154696899331,28cyclictest4497-21PK-Backend20:31:491
154696699337,11cyclictest141rcu_preempt21:25:180
1546966993326,5cyclictest0-21swapper/023:09:160
1546966993326,5cyclictest0-21swapper/023:09:160
1546966993320,9cyclictest1564391-21smartctl19:55:280
154696699331,7cyclictest1563263-21turbostat19:55:010
1546966993316,13cyclictest1662401-21cat00:35:160
16189712320,7chrt0-21swapper/022:30:260
1546971993217,11cyclictest1569128-21idleruntime20:10:202
1546971993215,13cyclictest2193-21gdbus23:49:572
1546971993215,12cyclictest1659327-21df00:25:162
154696899328,5cyclictest141rcu_preempt00:00:141
1546968993226,4cyclictest0-21swapper/120:27:271
1546968993217,10cyclictest1645283-21sed23:45:201
154696899321,4cyclictest4497-21PK-Backend20:15:511
154696899321,4cyclictest0-21swapper/122:46:361
154696899321,29cyclictest0-21swapper/122:33:091
154696699326,11cyclictest141rcu_preempt20:59:120
1546966993225,5cyclictest0-21swapper/023:47:440
1546966993218,10cyclictest1661308-21cat00:30:240
1546966993218,10cyclictest1654668-21sed00:10:300
1546966993218,10cyclictest1581014-21grep20:45:160
1546966993217,11cyclictest1599785-21cat21:35:300
1546966993217,11cyclictest1576574-21sed20:30:280
1546966993216,11cyclictest956-21systemd-logind00:06:410
154696699321,4cyclictest0-21swapper/020:17:550
154696699321,28cyclictest4497-21PK-Backend23:37:260
1546968993124,4cyclictest4497-21PK-Backend00:20:061
1546968993124,4cyclictest0-21swapper/123:38:301
1546968993118,9cyclictest1653962-21chrt00:10:151
1546968993116,11cyclictest1659505-21grep00:25:221
154696899311,4cyclictest0-21swapper/122:04:231
154696699313,12cyclictest141rcu_preempt20:50:550
1546966993112,6cyclictest141rcu_preempt19:20:030
154696899301,4cyclictest4497-21PK-Backend00:16:581
1546966993015,11cyclictest1608518-21pluginstate22:00:260
1546971992915,10cyclictest1619221-21cat22:30:312
154696899291,6cyclictest4497-21PK-Backend23:25:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional