You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-03 - 16:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Mon Aug 03, 2026 12:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1620664999992,3cyclictest0-21swapper/007:45:010
1620664999126,12cyclictest1665516-21ssh10:30:310
1620664998755,27cyclictest1-21systemd09:05:010
1620664998725,45cyclictest12-21ksoftirqd/010:58:470
1620664998528,16cyclictest1676385-21ssh10:53:410
1620664998336,4cyclictest16950irq/224-800000009:52:550
1620664998326,13cyclictest1718032-21ssh12:20:200
1620664998226,12cyclictest1690187-21ssh11:22:180
1620664998128,8cyclictest1681977-21modprobe11:05:030
1620664997927,12cyclictest1663080-21ssh10:25:210
1620664997829,14cyclictest1659461-21ssh10:18:040
1620664997828,11cyclictest1692484-21ssh11:27:090
1620664997827,12cyclictest1680040-21ssh11:00:580
1620664997725,15cyclictest1637532-21ssh09:32:090
162066499771,36cyclictest1656197-21ssh10:11:090
1620664997618,13cyclictest1716601-21ssh12:17:380
1620664997618,13cyclictest1716601-21ssh12:17:380
1620664997528,11cyclictest1713430-21ssh12:10:480
1620664997527,11cyclictest1653610-21ssh10:05:320
1620664997527,11cyclictest1631889-21ssh09:20:080
1620664997527,11cyclictest1631889-21ssh09:20:080
1620675997331,37cyclictest1643342-21sh09:44:302
1620669997344,26cyclictest23-21ksoftirqd/107:55:011
1620664997328,11cyclictest1642232-21ssh09:41:520
162066499731,38cyclictest0-21swapper/012:38:520
1620664997039,27cyclictest16850irq/134-800000009:15:010
1620664997026,13cyclictest1700932-21munin-plugin-st11:45:010
1620679996747,15cyclictest275-21dbus-daemon08:00:003
1620679996714,12cyclictest37-21ksoftirqd/312:10:013
1620664996729,9cyclictest16950irq/224-800000012:05:010
162066499672,10cyclictest1640413-21ssh09:38:150
1620675996624,37cyclictest1635879-21sh09:28:532
162067599662,16cyclictest1681300-21sshd11:03:542
1620664996624,14cyclictest1694251-21ssh11:30:420
162067599652,16cyclictest1665407-21ssh10:30:192
162067599652,16cyclictest1625814-21munin-run08:50:002
162067599652,13cyclictest1631378-21ssh09:19:362
162067599651,14cyclictest1655518-21munin-run10:10:002
162067999647,9cyclictest37-21ksoftirqd/311:10:013
162067599642,14cyclictest364-21snmpd11:30:132
1620675996419,41cyclictest1624987-21sshd08:31:552
162067599641,15cyclictest1693720-21munin-run11:30:022
162067599638,9cyclictest0-21swapper/207:44:592
1620675996323,35cyclictest30-21ksoftirqd/212:36:402
162067599632,14cyclictest364-21snmpd09:38:472
162067599632,14cyclictest1626594-21munin-run09:05:012
162067599632,13cyclictest1625560-21munin-run08:45:012
1620675996321,39cyclictest30-21ksoftirqd/212:32:082
1620675996317,40cyclictest1712990-21latency_hist12:10:022
162067599631,13cyclictest1620878-21munin-run07:15:002
162067999624,9cyclictest37-21ksoftirqd/310:45:003
162067599621,16cyclictest1652306-21ssh10:03:082
1620664996229,11cyclictest1671388-21ssh10:43:060
1620679996121,12cyclictest37-21ksoftirqd/312:25:003
1620679996118,13cyclictest37-21ksoftirqd/309:55:033
162067599612,50cyclictest1715274-21rm12:15:012
162067599612,14cyclictest1672411-21latency_hist10:45:032
162067599611,14cyclictest1622687-21munin-run07:50:012
162067599611,14cyclictest1621405-21sshd07:25:012
162066499616,34cyclictest19350irq/34-mmc111:55:020
162067999608,9cyclictest37-21ksoftirqd/312:00:213
1620679996017,11cyclictest37-21ksoftirqd/311:32:553
162067999600,8cyclictest1653133-21ssh10:05:013
162067599604,6cyclictest1656972-21sshd10:12:552
162067599602,14cyclictest1626343-21munin-run09:00:012
1620675996018,36cyclictest1688932-21latency_hist11:20:002
162067599601,13cyclictest1627995-21ssh09:12:162
162066999606,36cyclictest1626835-21diskmemload10:55:021
162067999596,11cyclictest37-21ksoftirqd/312:35:073
1620675995918,37cyclictest1672583-21rm10:45:162
1620675995916,37cyclictest3467602-21systemd-journal10:17:542
162067599591,15cyclictest1623176-21cron08:00:002
162067599591,13cyclictest364-21snmpd12:01:052
1620664995937,18cyclictest16950irq/224-800000011:40:000
162067999587,12cyclictest37-21ksoftirqd/309:35:013
1620679995820,9cyclictest37-21ksoftirqd/311:22:543
1620675995822,31cyclictest30-21ksoftirqd/211:12:282
1620675995822,31cyclictest1699287-21sh11:41:172
162067599582,13cyclictest1690568-21ssh11:23:172
162067599582,12cyclictest1649012-21ssh09:55:562
1620675995819,35cyclictest1717028-21rm12:18:362
1620675995819,35cyclictest1717028-21rm12:18:362
1620675995816,37cyclictest1624942-21cat08:30:022
1620675995815,5cyclictest1646773-21sh09:51:172
162067599581,13cyclictest1645865-21ssh09:49:442
162066999581,42cyclictest1620010-21kworker/u8:107:15:021
1620669995812,8cyclictest23-21ksoftirqd/112:30:021
1620669995812,8cyclictest23-21ksoftirqd/112:30:021
1620664995848,5cyclictest16850irq/134-800000009:19:590
1620664995847,8cyclictest12-21ksoftirqd/009:50:020
1620664995828,14cyclictest1669312-21ssh10:38:490
1620664995828,14cyclictest1669312-21ssh10:38:490
162067999576,9cyclictest37-21ksoftirqd/309:30:013
1620675995722,31cyclictest30-21ksoftirqd/210:21:112
162067599572,16cyclictest1701765-21sh11:46:292
162067599572,16cyclictest1632044-21ssh09:20:282
162067599572,16cyclictest1632044-21ssh09:20:272
162067599572,15cyclictest1668905-21ssh10:37:572
162067599572,15cyclictest1668905-21ssh10:37:572
162067599571,16cyclictest1626841-21modprobe09:10:012
162067599571,14cyclictest1676953-21munin-plugin-st10:55:002
162067599571,13cyclictest1664605-21ssh10:28:552
1620664995739,14cyclictest1-21systemd08:00:000
1620664995727,12cyclictest1673903-21ssh10:48:200
162066499572,6cyclictest1626847-21munin-run09:10:010
162067999566,10cyclictest37-21ksoftirqd/310:36:223
162067999566,10cyclictest37-21ksoftirqd/310:36:213
162067999563,10cyclictest37-21ksoftirqd/310:09:403
162067599567,9cyclictest1703777-21sh11:50:402
162067599564,15cyclictest1624402-21latency_hist08:20:042
162067599562,14cyclictest1696786-21sshd11:35:542
162067599562,14cyclictest1683908-21ssh11:09:282
162067599562,13cyclictest1678149-21sh10:57:142
1620675995620,32cyclictest1721464-21sh12:27:512
1620675995620,32cyclictest1721464-21sh12:27:512
1620675995618,34cyclictest30-21ksoftirqd/212:20:482
162067599561,19cyclictest1637037-21ssh09:30:572
1620664995624,16cyclictest1649037-21ssh09:55:590
162067999555,9cyclictest37-21ksoftirqd/311:11:323
162067999555,8cyclictest37-21ksoftirqd/307:45:013
1620679995543,7cyclictest361rcuc/311:45:533
1620675995544,6cyclictest1603893-21kworker/2:207:35:022
162067599552,9cyclictest1708108-21unin-run12:00:012
162067599551,15cyclictest1621166-21munin-run07:20:002
1620669995537,13cyclictest275-21dbus-daemon09:40:001
1620669995525,8cyclictest23-21ksoftirqd/110:10:021
1620664995527,13cyclictest1636122-21ssh09:29:210
1620664995527,12cyclictest1723075-21ssh12:31:030
162067999547,10cyclictest37-21ksoftirqd/311:30:003
1620679995438,12cyclictest1667419-21cron10:35:013
162067999542,9cyclictest37-21ksoftirqd/311:52:433
1620675995422,28cyclictest1623950-21sshd08:12:532
162067599542,15cyclictest1626056-21systemd-run08:55:012
1620669995434,12cyclictest275-21dbus-daemon12:10:011
162066499541,38cyclictest1712248-21ssh12:08:360
162067999538,14cyclictest37-21ksoftirqd/309:00:003
162067999535,11cyclictest37-21ksoftirqd/309:19:513
162067999531,7cyclictest37-21ksoftirqd/311:40:013
1620679995315,11cyclictest37-21ksoftirqd/310:11:273
162067599532,12cyclictest1622163-21munin-run07:40:012
1620675995311,4cyclictest1623595-21latency_hist08:05:032
1620669995320,8cyclictest23-21ksoftirqd/108:40:011
1620669995320,8cyclictest23-21ksoftirqd/108:40:011
162066499534,7cyclictest1-21systemd08:50:010
1620664995345,5cyclictest16950irq/224-800000007:15:010
1620664995336,12cyclictest3467602-21systemd-journal08:55:010
1620664995327,11cyclictest1651857-21ssh10:02:010
1620679995243,6cyclictest37-21ksoftirqd/310:51:163
162067999523,8cyclictest37-21ksoftirqd/311:40:083
1620679995213,11cyclictest37-21ksoftirqd/307:35:013
162067599521,14cyclictest364-21snmpd08:22:582
162067999519,11cyclictest37-21ksoftirqd/311:17:553
162067999518,11cyclictest37-21ksoftirqd/312:28:173
162067999518,11cyclictest37-21ksoftirqd/312:28:173
162067999517,10cyclictest37-21ksoftirqd/309:23:443
162067999517,10cyclictest37-21ksoftirqd/309:23:443
162067999516,12cyclictest37-21ksoftirqd/308:05:023
162067999516,11cyclictest37-21ksoftirqd/309:36:063
162067999515,10cyclictest37-21ksoftirqd/309:43:103
1620679995134,13cyclictest275-21dbus-daemon07:15:013
1620664995117,3cyclictest131rcu_preempt11:57:450
1620679995041,6cyclictest361rcuc/310:27:523
162067999504,13cyclictest37-21ksoftirqd/310:25:013
1620679995040,7cyclictest361rcuc/310:45:413
162067999501,10cyclictest0-21swapper/310:58:513
162067599501,15cyclictest1622945-21systemd-run07:55:012
1620669995031,13cyclictest309-21in:imuxsock08:44:591
1620664995040,7cyclictest12-21ksoftirqd/007:20:010
1620664995029,18cyclictest12-21ksoftirqd/011:15:020
1620664995024,11cyclictest1687276-21ssh11:16:090
162067999497,9cyclictest37-21ksoftirqd/309:05:023
162067999497,7cyclictest37-21ksoftirqd/312:15:063
162067999497,7cyclictest37-21ksoftirqd/312:15:053
162067999497,11cyclictest37-21ksoftirqd/309:47:513
162067999497,10cyclictest37-21ksoftirqd/311:58:513
162067999496,8cyclictest37-21ksoftirqd/311:05:023
162067999496,7cyclictest37-21ksoftirqd/309:10:553
162067999493,34cyclictest3467602-21systemd-journal07:50:013
162067599492,13cyclictest1623742-21cat08:10:012
162066999498,33cyclictest1646076-21ssh09:50:011
1620669994932,14cyclictest1-21systemd08:30:011
1620669994914,12cyclictest23-21ksoftirqd/111:11:151
1620669994914,11cyclictest131rcu_preempt09:30:001
162066499498,18cyclictest151rcuc/007:30:020
1620679994839,6cyclictest361rcuc/310:16:403
162067999481,10cyclictest37-21ksoftirqd/307:40:003
1620669994838,8cyclictest23-21ksoftirqd/111:00:161
162066999482,6cyclictest131rcu_preempt12:20:021
162066999482,6cyclictest131rcu_preempt12:20:021
162067999478,7cyclictest131rcu_preempt07:20:013
162067999476,11cyclictest37-21ksoftirqd/312:30:083
162067599472,9cyclictest1620674-21latency_hist07:10:022
162067599472,9cyclictest1620674-21latency_hist07:10:012
162067599471,14cyclictest1625310-21munin-run08:40:022
162067599471,14cyclictest1625310-21munin-run08:40:022
1620669994733,9cyclictest1-21systemd11:25:011
1620669994717,11cyclictest1696239-21ssh11:35:021
1620664994737,7cyclictest16850irq/134-800000012:30:010
1620664994737,7cyclictest16850irq/134-800000012:30:000
1620664994728,15cyclictest1622449-21unin-run07:45:010
1620664994717,7cyclictest12-21ksoftirqd/011:46:230
162067999469,6cyclictest131rcu_preempt12:12:163
162067999467,11cyclictest37-21ksoftirqd/309:58:513
162067999462,8cyclictest37-21ksoftirqd/308:40:023
162067999462,8cyclictest37-21ksoftirqd/308:40:023
1620669994636,7cyclictest23-21ksoftirqd/111:40:031
1620669994630,11cyclictest1-21systemd07:35:011
1620669994627,13cyclictest23-21ksoftirqd/111:20:001
162066999461,5cyclictest0-21swapper/109:42:051
1620669994611,20cyclictest1639000-21ssh09:35:011
162066499462,13cyclictest1625279-21munin-run08:40:010
162066499462,13cyclictest1625279-21munin-run08:40:000
162067999458,7cyclictest37-21ksoftirqd/308:35:003
1620679994531,9cyclictest309-21in:imuxsock08:05:083
162067999453,10cyclictest37-21ksoftirqd/308:25:003
1620669994528,12cyclictest1-21systemd10:00:011
1620669994519,22cyclictest23-21ksoftirqd/111:42:501
1620669994518,22cyclictest23-21ksoftirqd/110:51:311
162066999451,6cyclictest23-21ksoftirqd/112:38:061
1620669994510,10cyclictest23-21ksoftirqd/112:30:351
1620664994524,7cyclictest1661922-21ssh10:23:070
162067999442,9cyclictest37-21ksoftirqd/308:55:013
162066999449,5cyclictest131rcu_preempt12:02:081
1620669994435,6cyclictest23-21ksoftirqd/109:14:061
1620669994434,7cyclictest23-21ksoftirqd/110:21:271
162066999442,5cyclictest131rcu_preempt09:20:261
162066999442,5cyclictest131rcu_preempt09:20:251
162066999441,9cyclictest0-21swapper/111:57:261
1620669994411,5cyclictest131rcu_preempt10:32:011
1620664994433,7cyclictest1622143-21cron07:40:010
162067999438,5cyclictest131rcu_preempt09:10:003
162067999436,7cyclictest37-21ksoftirqd/308:30:033
162067999431,7cyclictest0-21swapper/307:27:363
162066999437,4cyclictest131rcu_preempt10:16:591
162066999437,4cyclictest131rcu_preempt10:05:011
1620669994329,10cyclictest3467602-21systemd-journal10:49:591
1620669994329,10cyclictest309-21in:imuxsock07:18:551
1620669994320,19cyclictest23-21ksoftirqd/111:05:051
1620669994317,22cyclictest23-21ksoftirqd/111:45:561
1620669994317,16cyclictest23-21ksoftirqd/110:10:201
1620664994336,4cyclictest16950irq/224-800000008:25:010
1620679994211,5cyclictest131rcu_preempt07:10:033
1620679994211,5cyclictest131rcu_preempt07:10:023
162067999420,6cyclictest150-21pr/ttyS208:12:543
1620669994219,19cyclictest23-21ksoftirqd/112:20:491
1620669994217,15cyclictest131rcu_preempt11:51:291
1620669994214,5cyclictest131rcu_preempt09:50:161
1620664994234,5cyclictest16950irq/224-800000008:30:010
1620664994224,14cyclictest3467602-21systemd-journal08:35:010
162067999413,6cyclictest37-21ksoftirqd/307:53:553
1620679994114,4cyclictest131rcu_preempt08:20:043
162066999411,9cyclictest0-21swapper/111:28:091
162066999411,9cyclictest0-21swapper/110:41:541
1620669994116,19cyclictest131rcu_preempt08:10:011
1620669994110,9cyclictest131rcu_preempt07:30:001
1620664994134,4cyclictest16950irq/224-800000008:03:530
1620664994130,8cyclictest1625546-21munin-plugin-st08:45:000
1620664994127,8cyclictest1626290-21cron09:00:000
162067999407,8cyclictest37-21ksoftirqd/307:20:133
162067999404,10cyclictest37-21ksoftirqd/308:50:013
1620675994032,5cyclictest1611130-21kworker/2:307:27:332
162066999404,4cyclictest131rcu_preempt10:27:341
1620669994029,8cyclictest23-21ksoftirqd/112:13:151
1620669994029,7cyclictest221rcuc/107:45:011
162066999401,7cyclictest0-21swapper/110:36:151
162066999401,7cyclictest0-21swapper/110:36:151
1620669994012,6cyclictest131rcu_preempt09:10:021
1620664994033,4cyclictest16950irq/224-800000007:34:590
1620664994027,10cyclictest364-21snmpd08:17:440
162067999391,13cyclictest0-21swapper/308:41:573
162066999399,5cyclictest131rcu_preempt09:17:171
1620669993927,8cyclictest3467602-21systemd-journal08:54:541
1620669993925,9cyclictest309-21in:imuxsock09:00:021
162066499396,4cyclictest131rcu_preempt07:10:020
162066499396,4cyclictest131rcu_preempt07:10:010
1620669993727,6cyclictest23-21ksoftirqd/109:00:011
1620669993727,5cyclictest1625796-21sh08:50:001
1620669993726,8cyclictest23-21ksoftirqd/107:40:001
1620669993718,11cyclictest23-21ksoftirqd/108:17:391
1620669993716,16cyclictest0-21swapper/107:10:021
1620669993716,16cyclictest0-21swapper/107:10:011
1620664993728,6cyclictest1623963-21cron08:15:000
162066999368,6cyclictest131rcu_preempt07:50:011
1620669993626,6cyclictest23-21ksoftirqd/108:05:031
1620669993623,11cyclictest23-21ksoftirqd/108:15:011
1620669993619,12cyclictest275-21dbus-daemon08:35:001
1620664993623,9cyclictest309-21in:imuxsock07:21:540
162066499362,31cyclictest0-21swapper/008:08:000
1620669993527,5cyclictest1621399-21munin-plugin-st07:25:001
1620664993528,4cyclictest16950irq/224-800000007:55:020
1620669993423,8cyclictest23-21ksoftirqd/108:00:031
1620669993214,8cyclictest1620660-21cyclictest08:25:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional