You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-30 - 22:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat May 30, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
159644999836,6cyclictest0-21swapper/008:00:010
1596449997868,8cyclictest0-21swapper/011:15:020
159644999779,46cyclictest1600581-21kworker/u8:008:30:020
1596449997526,46cyclictest12-21ksoftirqd/010:55:010
1596462997435,34cyclictest3467602-21systemd-journal09:15:022
1596462997132,11cyclictest3467602-21systemd-journal11:39:322
1596454997018,19cyclictest1688614-21ssh12:22:251
1596454997014,18cyclictest1605950-21ssh09:16:581
1596454996917,14cyclictest1662172-21sh11:23:091
1596454996816,11cyclictest1630730-21ssh10:12:211
1596454996712,16cyclictest1650255-21ssh10:56:201
1596454996622,13cyclictest1647294-21munin-run10:50:011
1596454996610,22cyclictest1639561-21ssh10:32:251
1596462996522,14cyclictest275-21dbus-daemon10:45:002
1596449996513,25cyclictest131rcu_preempt10:25:010
1596454996419,9cyclictest23-21ksoftirqd/108:35:021
1596454996415,10cyclictest23-21ksoftirqd/108:15:011
1596454996411,14cyclictest1673055-21ssh11:47:241
1596462996324,9cyclictest30-21ksoftirqd/212:25:232
1596454996314,9cyclictest23-21ksoftirqd/111:10:031
1596454996314,9cyclictest23-21ksoftirqd/111:10:021
1596454996312,15cyclictest1633411-21ssh10:18:381
1596462996227,11cyclictest3467718-21lldpd11:01:102
1596462996226,10cyclictest30-21ksoftirqd/209:50:012
1596462996224,9cyclictest30-21ksoftirqd/210:05:012
159645499629,15cyclictest1635783-21ssh10:24:031
159645499626,25cyclictest1618206-21ssh09:44:321
159645499626,19cyclictest1648274-21ssh10:51:551
1596454996211,14cyclictest1686254-21ssh12:17:061
159644999623,40cyclictest131rcu_preempt12:26:150
1596454996117,17cyclictest1693175-21ssh12:32:421
1596454996116,9cyclictest23-21ksoftirqd/107:10:011
1596454996115,7cyclictest23-21ksoftirqd/107:20:021
1596449996129,15cyclictest1687702-21ssh12:20:130
159645499608,16cyclictest1658587-21sed11:15:011
1596462995925,13cyclictest30-21ksoftirqd/212:21:012
1596454995916,8cyclictest23-21ksoftirqd/110:37:201
1596454995912,16cyclictest1659976-21rm11:18:201
1596462995847,7cyclictest1598005-21kworker/2:310:07:012
1596462995847,7cyclictest1598005-21kworker/2:310:07:002
1596454995818,8cyclictest23-21ksoftirqd/109:31:051
1596454995816,8cyclictest23-21ksoftirqd/112:06:501
1596454995811,10cyclictest23-21ksoftirqd/110:28:411
1596454995811,10cyclictest23-21ksoftirqd/110:28:411
1596449995829,13cyclictest1610238-21ssh09:26:350
159645499577,19cyclictest1603692-21ssh09:11:561
1596454995710,14cyclictest1683384-21ssh12:10:361
1596449995722,29cyclictest12-21ksoftirqd/008:05:030
1596449995722,29cyclictest12-21ksoftirqd/008:05:020
1596449995719,12cyclictest12-21ksoftirqd/012:32:260
159645499565,14cyclictest1625568-21ssh10:00:391
1596454995616,11cyclictest1671547-21ssh11:44:191
1596454995616,10cyclictest23-21ksoftirqd/109:30:011
159645499559,9cyclictest0-21swapper/109:58:331
159645499557,15cyclictest1621155-21ssh09:50:411
1596454995514,8cyclictest23-21ksoftirqd/111:58:101
1596449995514,32cyclictest12-21ksoftirqd/012:00:460
159645499548,14cyclictest1597772-21latency_hist07:35:021
159645499548,14cyclictest1597772-21latency_hist07:35:011
159645499547,14cyclictest1678935-21ssh12:00:301
159645499547,14cyclictest1644855-21ssh10:44:281
159645499543,20cyclictest1608942-21sh09:23:521
1596454995414,8cyclictest23-21ksoftirqd/111:01:201
1596449995410,38cyclictest12-21ksoftirqd/011:35:020
1596464995339,10cyclictest1-21systemd09:50:003
1596464995328,22cyclictest37-21ksoftirqd/310:25:023
1596454995333,16cyclictest1602161-21systemctl09:00:001
159644999533,14cyclictest1625129-21modprobe10:00:010
1596449995317,10cyclictest12-21ksoftirqd/011:37:360
1596449995314,6cyclictest131rcu_preempt11:10:010
1596449995314,6cyclictest131rcu_preempt11:10:000
1596449995312,34cyclictest131rcu_preempt10:43:100
1596462995244,5cyclictest1588634-21kworker/2:009:29:282
159645499528,14cyclictest1629471-21ssh10:09:521
159645499528,14cyclictest1629471-21ssh10:09:511
1596449995244,4cyclictest16950irq/224-800000007:15:000
1596449995234,12cyclictest12-21ksoftirqd/009:25:010
1596449995233,5cyclictest16850irq/134-800000009:20:020
1596449995224,12cyclictest1622162-21ssh09:53:190
1596449995212,36cyclictest12-21ksoftirqd/010:10:090
1596449995212,36cyclictest12-21ksoftirqd/010:10:090
1596464995134,12cyclictest3467602-21systemd-journal11:40:013
1596462995141,6cyclictest1618436-21kworker/2:212:32:212
1596462995140,7cyclictest1598005-21kworker/2:309:37:162
159645499516,14cyclictest1668578-21ssh11:37:301
1596454995114,8cyclictest23-21ksoftirqd/109:35:501
1596454995112,13cyclictest364-21snmpd11:33:351
159644999511,44cyclictest1665132-21cat11:30:020
1596462995043,5cyclictest1640654-21kworker/2:012:17:322
1596462995043,5cyclictest1618436-21kworker/2:210:38:072
1596462995042,5cyclictest1618436-21kworker/2:211:22:502
1596462995042,5cyclictest1598005-21kworker/2:307:50:022
1596462995042,5cyclictest1598005-21kworker/2:307:50:012
1596462995025,8cyclictest1618436-21kworker/2:210:27:432
1596462995025,8cyclictest1618436-21kworker/2:210:27:432
159645499506,14cyclictest1691283-21ssh12:28:351
1596454995035,12cyclictest1663179-21ssh11:25:081
1596454995031,13cyclictest3467602-21systemd-journal08:20:021
1596449995023,12cyclictest1633622-21ssh10:19:170
1596449995019,26cyclictest12-21ksoftirqd/012:17:560
1596449995016,31cyclictest12-21ksoftirqd/012:14:170
1596449995012,34cyclictest151rcuc/010:40:020
1596449995012,11cyclictest12-21ksoftirqd/010:30:280
159646499497,39cyclictest1589419-21kworker/3:007:50:033
159646499497,39cyclictest1589419-21kworker/3:007:50:023
1596462994941,5cyclictest1640654-21kworker/2:012:05:032
1596462994941,5cyclictest1640654-21kworker/2:011:13:522
1596462994941,5cyclictest1618436-21kworker/2:211:50:232
1596462994941,5cyclictest1598005-21kworker/2:309:35:012
1596462994939,6cyclictest1598005-21kworker/2:309:23:042
1596454994929,16cyclictest1695391-21ssh12:37:431
1596454994928,16cyclictest1620274-21ssh09:49:141
159644999499,20cyclictest1666657-21kworker/u8:011:45:010
1596449994940,6cyclictest16950irq/224-800000011:05:020
159644999493,9cyclictest12-21ksoftirqd/012:05:370
1596449994925,21cyclictest12-21ksoftirqd/010:57:040
1596449994922,23cyclictest19350irq/34-mmc110:15:010
1596449994916,30cyclictest12-21ksoftirqd/009:10:050
1596449994915,9cyclictest12-21ksoftirqd/011:17:310
1596462994839,6cyclictest1596220-21kworker/2:207:30:022
1596462994832,12cyclictest1-21systemd08:35:012
1596462994832,12cyclictest1-21systemd08:35:002
1596462994832,11cyclictest275-21dbus-daemon11:35:012
1596462994831,12cyclictest3467602-21systemd-journal08:10:002
1596462994829,15cyclictest1-21systemd07:45:012
1596454994827,17cyclictest1674645-21ssh11:50:481
1596449994839,3cyclictest0-21swapper/007:45:030
1596449994835,8cyclictest1618569-21latency_hist09:45:020
1596449994832,13cyclictest1674313-21ssh11:50:030
1596449994816,5cyclictest131rcu_preempt08:25:000
1596464994730,14cyclictest37-21ksoftirqd/310:50:023
1596462994739,5cyclictest1618436-21kworker/2:210:52:092
1596462994739,5cyclictest1598005-21kworker/2:310:13:152
1596462994739,5cyclictest1588634-21kworker/2:009:43:252
1596462994739,5cyclictest1588634-21kworker/2:009:16:042
1596462994738,6cyclictest1640654-21kworker/2:011:41:442
159644999473,7cyclictest12-21ksoftirqd/007:39:590
1596449994730,14cyclictest1661448-21ssh11:21:180
1596449994730,13cyclictest1625787-21ssh10:01:070
1596449994711,9cyclictest12-21ksoftirqd/011:59:530
1596464994632,10cyclictest1-21systemd07:30:023
1596464994631,11cyclictest275-21dbus-daemon10:40:023
1596464994612,7cyclictest131rcu_preempt10:55:013
1596464994611,6cyclictest131rcu_preempt12:15:253
159646499461,10cyclictest1638163-21rm10:29:233
159646499461,10cyclictest1638163-21rm10:29:223
1596462994638,5cyclictest1640654-21kworker/2:012:39:582
1596462994637,6cyclictest1618436-21kworker/2:210:21:112
1596462994631,10cyclictest30-21ksoftirqd/211:00:002
1596454994632,10cyclictest1-21systemd08:00:011
1596449994628,14cyclictest1613201-21ssh09:33:230
159644999462,6cyclictest1616091-21modprobe09:40:010
159644999462,6cyclictest1602707-21munin-run09:10:000
1596449994620,6cyclictest12-21ksoftirqd/007:55:010
1596449994618,13cyclictest1694769-21ssh12:36:050
159646499455,7cyclictest37-21ksoftirqd/310:10:093
159646499455,7cyclictest37-21ksoftirqd/310:10:083
1596464994529,12cyclictest1-21systemd11:35:003
1596462994538,5cyclictest1640654-21kworker/2:011:26:232
1596462994537,5cyclictest1618436-21kworker/2:209:52:342
1596462994532,9cyclictest30-21ksoftirqd/207:25:012
1596462994512,7cyclictest131rcu_preempt11:08:012
1596462994512,7cyclictest131rcu_preempt11:08:002
1596454994533,9cyclictest23-21ksoftirqd/109:10:001
1596454994531,11cyclictest1597914-21sshd07:35:181
1596454994527,12cyclictest1-21systemd08:45:011
1596454994527,12cyclictest1-21systemd08:45:011
1596449994537,6cyclictest12-21ksoftirqd/009:47:060
1596449994530,12cyclictest1636384-21ssh10:25:030
1596449994530,12cyclictest1636384-21ssh10:25:020
1596449994526,16cyclictest1646744-21ssh10:48:330
1596449994521,21cyclictest1674130-21latency_hist11:50:020
1596464994431,9cyclictest309-21in:imuxsock10:04:593
1596464994430,11cyclictest1657048-21sshd11:11:313
1596464994428,12cyclictest3467602-21systemd-journal08:45:033
1596464994428,12cyclictest3467602-21systemd-journal08:45:013
1596464994428,12cyclictest3467602-21systemd-journal08:45:003
1596464994411,5cyclictest131rcu_preempt11:55:573
1596462994439,3cyclictest1618436-21kworker/2:210:48:342
1596462994437,5cyclictest1618436-21kworker/2:210:32:462
1596462994436,5cyclictest1618436-21kworker/2:212:10:312
1596462994436,5cyclictest1618436-21kworker/2:209:56:082
1596462994435,6cyclictest1598005-21kworker/2:308:36:552
1596462994428,13cyclictest1-21systemd12:00:012
1596462994425,13cyclictest30-21ksoftirqd/212:00:172
1596454994431,10cyclictest1602396-21sshd09:00:461
1596454994431,10cyclictest1598708-21sshd07:52:311
1596454994429,10cyclictest1598482-21munin-run07:50:011
1596454994429,10cyclictest1598482-21munin-run07:50:011
159644999447,6cyclictest12-21ksoftirqd/009:00:020
1596449994428,13cyclictest1600846-21sshd08:34:330
1596449994428,13cyclictest1600846-21sshd08:34:320
1596449994411,7cyclictest12-21ksoftirqd/008:50:010
1596464994327,13cyclictest1616153-21munin-run09:40:013
1596464994324,5cyclictest37-21ksoftirqd/308:20:013
1596464994310,6cyclictest131rcu_preempt11:17:403
1596449994330,8cyclictest12-21ksoftirqd/008:20:010
1596449994328,9cyclictest12-21ksoftirqd/008:15:000
159646499427,5cyclictest131rcu_preempt09:45:023
1596464994228,9cyclictest1654020-21date11:05:013
1596464994223,13cyclictest275-21dbus-daemon08:30:013
1596462994229,9cyclictest275-21dbus-daemon07:40:012
1596462994227,11cyclictest3467602-21systemd-journal07:57:322
1596462994225,12cyclictest1602706-21systemd-run09:10:012
1596454994230,9cyclictest1601872-21sshd08:52:321
1596454994229,9cyclictest1600359-21munin-run08:25:001
1596454994228,10cyclictest364-21snmpd07:27:371
1596454994227,12cyclictest3467602-21systemd-journal07:11:021
1596449994233,6cyclictest12-21ksoftirqd/007:30:000
159646499415,30cyclictest1694139-21grep12:35:023
1596464994132,7cyclictest37-21ksoftirqd/311:53:323
1596464994129,9cyclictest37-21ksoftirqd/309:52:443
1596464994129,10cyclictest37-21ksoftirqd/307:52:323
1596454994127,9cyclictest48-21kcompactd008:32:051
1596454994127,9cyclictest48-21kcompactd008:32:041
159646499406,6cyclictest37-21ksoftirqd/312:04:033
1596464994030,7cyclictest1597176-21sshd07:24:333
1596464994028,10cyclictest37-21ksoftirqd/310:44:263
1596464994027,9cyclictest1640673-21grep10:35:013
1596464994027,10cyclictest37-21ksoftirqd/312:14:593
159646499402,5cyclictest1612297-21ssh09:31:073
1596464994022,14cyclictest1-21systemd11:30:013
159646499401,9cyclictest1633708-21ssh10:19:303
1596464994019,16cyclictest275-21dbus-daemon08:55:013
1596464994018,17cyclictest275-21dbus-daemon08:25:003
159646499401,4cyclictest131rcu_preempt11:40:083
159646299406,5cyclictest131rcu_preempt11:15:022
1596462994033,4cyclictest1640654-21kworker/2:011:46:122
1596462994027,10cyclictest30-21ksoftirqd/208:13:162
1596462994011,10cyclictest30-21ksoftirqd/210:19:562
1596454994031,7cyclictest23-21ksoftirqd/108:10:021
1596454994029,8cyclictest1601608-21sshd08:45:321
1596454994026,10cyclictest1597206-21munin-run07:25:001
1596454994024,10cyclictest275-21dbus-daemon08:30:001
159644999403,32cyclictest0-21swapper/007:20:000
1596449994028,9cyclictest1601362-21sshd08:40:450
1596449994028,9cyclictest1601362-21sshd08:40:440
159646499396,3cyclictest131rcu_preempt09:00:003
1596464993929,8cyclictest37-21ksoftirqd/309:10:023
1596464993928,9cyclictest37-21ksoftirqd/309:26:403
1596464993928,7cyclictest1601085-21sshd08:36:333
1596464993926,10cyclictest1696470-21sed12:40:033
1596464993922,14cyclictest275-21dbus-daemon11:00:013
159646499391,4cyclictest0-21swapper/311:22:493
1596462993931,5cyclictest1596220-21kworker/2:207:15:022
1596462993928,8cyclictest3467602-21systemd-journal07:18:322
1596454993927,9cyclictest3467602-21systemd-journal07:40:291
1596454993918,16cyclictest1599233-21modprobe08:00:341
1596454993918,16cyclictest1599233-21modprobe08:00:341
159646499384,4cyclictest131rcu_preempt09:16:493
1596464993824,11cyclictest37-21ksoftirqd/309:59:273
1596462993823,11cyclictest30-21ksoftirqd/208:30:012
1596449993833,3cyclictest16950irq/224-800000007:23:330
159644999382,6cyclictest1598482-21munin-run07:50:000
159644999382,6cyclictest1598482-21munin-run07:50:000
1596449993825,9cyclictest12-21ksoftirqd/008:40:010
1596464993725,9cyclictest1689790-21sort12:25:023
1596464993724,9cyclictest3467602-21systemd-journal11:46:103
1596464993720,13cyclictest275-21dbus-daemon09:15:013
1596464993720,13cyclictest1-21systemd08:00:013
1596462993724,10cyclictest30-21ksoftirqd/209:05:002
1596462993722,11cyclictest30-21ksoftirqd/209:00:022
1596462993722,11cyclictest1-21systemd08:55:002
1596449993723,10cyclictest12-21ksoftirqd/008:54:160
159644999372,12cyclictest1602391-21modprobe09:00:020
159646499363,5cyclictest131rcu_preempt09:22:253
1596464993626,8cyclictest37-21ksoftirqd/310:11:563
1596462993628,5cyclictest1600357-21munin-plugin-st08:25:002
159646299361,5cyclictest131rcu_preempt07:55:022
1596449993625,9cyclictest12-21ksoftirqd/007:35:010
1596449993625,9cyclictest12-21ksoftirqd/007:35:010
1596449993624,10cyclictest12-21ksoftirqd/008:10:000
159646499359,5cyclictest131rcu_preempt08:02:443
159646499359,5cyclictest131rcu_preempt08:02:433
1596464993523,8cyclictest37-21ksoftirqd/311:05:203
1596464993523,8cyclictest37-21ksoftirqd/311:05:203
1596464993522,10cyclictest0-21swapper/312:09:213
159646499351,7cyclictest0-21swapper/312:26:463
1596464993516,14cyclictest275-21dbus-daemon07:20:013
159646299354,3cyclictest131rcu_preempt08:45:012
159646299354,3cyclictest131rcu_preempt08:45:002
1596462993522,10cyclictest30-21ksoftirqd/208:05:022
1596462993522,10cyclictest30-21ksoftirqd/208:05:012
1596449993526,7cyclictest12-21ksoftirqd/007:10:020
1596464993426,5cyclictest1598226-21latency_hist07:45:033
1596464993422,8cyclictest309-21in:imuxsock07:10:333
159646299342,3cyclictest131rcu_preempt07:10:022
159646499334,3cyclictest131rcu_preempt09:00:463
159646499332,4cyclictest131rcu_preempt07:39:593
159646499332,27cyclictest1596536-21latency_hist07:10:013
1596464993318,11cyclictest309-21in:imuxsock08:09:593
159646299334,3cyclictest131rcu_preempt07:35:012
159646299334,3cyclictest131rcu_preempt07:35:002
1596462993321,9cyclictest1601846-21sed08:50:022
159646499321,5cyclictest0-21swapper/308:11:413
1596464993121,7cyclictest37-21ksoftirqd/307:35:023
1596464993121,7cyclictest37-21ksoftirqd/307:35:013
1596462993121,7cyclictest1600062-21sshd08:19:322
1596464993016,11cyclictest1600873-21systemd-run08:35:013
1596464993016,11cyclictest1600873-21systemd-run08:35:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional