You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-31 - 11:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun May 31, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1754162999615,40cyclictest0-21swapper/023:59:320
1754162999615,40cyclictest0-21swapper/023:59:310
1754162999416,38cyclictest0-21swapper/000:35:010
175416299933,86cyclictest0-21swapper/023:40:010
1754162999327,32cyclictest150-21pr/ttyS223:11:070
1754162999327,32cyclictest150-21pr/ttyS223:11:070
1754162998910,47cyclictest0-21swapper/023:46:060
1754162998532,26cyclictest0-21swapper/022:10:490
1754162998526,33cyclictest150-21pr/ttyS223:08:280
1754162998421,33cyclictest0-21swapper/022:23:590
1754162998229,15cyclictest1827452-21ssh23:40:140
1754162998226,23cyclictest1794534-21ssh22:26:330
175416299811,77cyclictest0-21swapper/020:00:020
1754162997928,27cyclictest150-21pr/ttyS221:49:170
1754162997928,27cyclictest150-21pr/ttyS221:49:170
1754162997927,15cyclictest1848254-21ssh00:27:150
1754162997927,15cyclictest1848254-21ssh00:27:140
175416299788,48cyclictest0-21swapper/023:03:260
175416299788,48cyclictest0-21swapper/023:03:260
1754162997813,34cyclictest0-21swapper/000:37:490
1754162997630,18cyclictest3467602-21systemd-journal21:28:160
1754162997625,18cyclictest1838232-21systemd-run00:05:000
1754162997570,3cyclictest0-21swapper/000:05:030
1754162997426,15cyclictest1765427-21ssh21:20:550
1754162997326,13cyclictest1821608-21ssh23:27:190
1754162997316,23cyclictest0-21swapper/022:33:390
175416299728,61cyclictest0-21swapper/020:55:010
1754162997221,16cyclictest1760627-21ssh21:10:050
1754162997126,14cyclictest1824931-21sed23:35:020
1754162997126,14cyclictest1824931-21sed23:35:010
1754162997120,21cyclictest1778577-21sh21:50:360
1754162997112,30cyclictest1844724-21sh00:19:320
1754162997112,30cyclictest1844724-21sh00:19:320
1754171997047,7cyclictest1753367-21kworker/2:321:16:092
1754162997031,12cyclictest1782609-21ssh22:00:000
1754162997021,9cyclictest0-21swapper/023:23:300
1754162996921,14cyclictest1785622-21ssh22:06:320
1754162996915,24cyclictest0-21swapper/021:05:020
1754162996828,14cyclictest364-21snmpd22:59:460
1754162996823,17cyclictest1845454-21ssh00:20:470
1754162996823,13cyclictest1805999-21ssh22:52:300
1754162996739,22cyclictest1756474-21munin-plugin-st19:55:010
1754162996730,11cyclictest1803326-21sh22:46:200
1754162996725,38cyclictest1790394-21sshd22:17:310
1754162996721,16cyclictest1817633-21ssh23:18:300
1754162996629,4cyclictest1784813-21munin-run22:05:010
1754175996419,11cyclictest37-21ksoftirqd/300:10:013
1754162996424,15cyclictest1759095-21systemd-run20:45:010
1754175996320,12cyclictest37-21ksoftirqd/321:20:003
1754162996328,31cyclictest1832537-21sh23:52:060
1754175996217,9cyclictest37-21ksoftirqd/322:50:033
1754171996244,7cyclictest1822748-21kworker/2:200:18:112
1754171996244,7cyclictest1822748-21kworker/2:200:18:102
1754162996245,14cyclictest1775099-21sshd21:42:570
175417599616,7cyclictest37-21ksoftirqd/322:45:013
175417599616,7cyclictest37-21ksoftirqd/321:25:013
1754162996111,21cyclictest0-21swapper/021:39:240
175417599604,14cyclictest37-21ksoftirqd/323:30:013
175417599593,8cyclictest37-21ksoftirqd/321:45:023
1754162995946,9cyclictest1764207-21sshd21:18:320
1754162995923,5cyclictest1800544-21latency_hist22:40:030
1754162995921,35cyclictest0-21swapper/021:30:360
175417599586,10cyclictest37-21ksoftirqd/321:32:123
175417599585,6cyclictest37-21ksoftirqd/322:00:003
175417599584,16cyclictest37-21ksoftirqd/300:00:153
1754175995817,8cyclictest37-21ksoftirqd/321:39:413
1754175995812,12cyclictest37-21ksoftirqd/323:55:003
1754171995849,6cyclictest1835966-21modprobe23:59:592
1754171995849,6cyclictest1835966-21modprobe00:00:002
1754171995849,5cyclictest1834031-21kworker/2:000:00:122
1754162995843,11cyclictest1755167-21sshd19:29:320
1754162995825,15cyclictest150-21pr/ttyS222:41:350
1754162995819,35cyclictest1-21systemd20:55:330
175417599579,10cyclictest37-21ksoftirqd/320:55:013
175417599576,9cyclictest37-21ksoftirqd/300:27:413
175417599576,9cyclictest37-21ksoftirqd/300:27:403
175417599575,8cyclictest37-21ksoftirqd/300:14:093
1754171995747,5cyclictest1753367-21kworker/2:321:46:082
1754171995747,5cyclictest1753367-21kworker/2:321:46:082
1754162995722,32cyclictest150-21pr/ttyS220:07:420
175417599567,11cyclictest37-21ksoftirqd/323:41:503
1754175995610,10cyclictest37-21ksoftirqd/300:00:023
1754175995610,10cyclictest37-21ksoftirqd/300:00:013
175417199566,14cyclictest30-21ksoftirqd/219:10:022
175417199566,14cyclictest30-21ksoftirqd/219:10:022
1754171995648,5cyclictest1787015-21kworker/2:323:40:212
1754171995646,6cyclictest1822748-21kworker/2:200:15:012
1754162995643,10cyclictest1756199-21sshd19:48:320
1754162995643,10cyclictest1756199-21sshd19:48:320
1754162995643,10cyclictest1754374-21sshd19:14:320
1754162995641,12cyclictest150-21pr/ttyS219:34:500
1754162995641,12cyclictest150-21pr/ttyS219:34:500
1754162995638,14cyclictest1758778-21sshd20:37:450
1754162995636,16cyclictest364-21snmpd19:41:460
1754162995621,31cyclictest0-21swapper/020:26:310
175417599557,9cyclictest37-21ksoftirqd/322:58:483
175417599557,7cyclictest37-21ksoftirqd/322:36:353
175417599552,8cyclictest1-21systemd20:25:003
1754171995546,6cyclictest1759350-21kworker/2:022:54:552
1754162995541,10cyclictest1758565-21munin-run20:35:010
1754162995541,10cyclictest1758565-21munin-run20:35:010
1754162995534,16cyclictest1755716-21sendmail-msp19:40:000
1754162995533,18cyclictest1754666-21munin-plugin-st19:20:000
1754162995533,18cyclictest1754666-21munin-plugin-st19:20:000
1754162995523,29cyclictest150-21pr/ttyS220:23:220
1754162995521,30cyclictest150-21pr/ttyS220:48:580
175417599547,8cyclictest37-21ksoftirqd/322:10:023
175417599546,9cyclictest37-21ksoftirqd/322:30:003
175417599545,7cyclictest37-21ksoftirqd/323:05:013
175417599545,7cyclictest37-21ksoftirqd/323:05:013
1754175995417,10cyclictest37-21ksoftirqd/321:55:013
1754171995446,5cyclictest1834031-21kworker/2:000:08:552
1754171995446,5cyclictest1822748-21kworker/2:200:34:532
1754171995446,5cyclictest1759350-21kworker/2:021:55:562
1754171995445,6cyclictest1753367-21kworker/2:321:52:392
1754171995445,5cyclictest1753367-21kworker/2:320:05:022
1754171995444,6cyclictest1753367-21kworker/2:321:12:392
1754162995440,11cyclictest1756987-21sshd20:04:320
1754162995410,12cyclictest1760372-21diskmemload00:12:520
175417599537,10cyclictest37-21ksoftirqd/322:04:273
175417599536,8cyclictest37-21ksoftirqd/323:35:033
175417599535,7cyclictest37-21ksoftirqd/322:16:153
175417599535,11cyclictest37-21ksoftirqd/323:11:473
175417599535,11cyclictest37-21ksoftirqd/323:11:473
175417599534,7cyclictest37-21ksoftirqd/300:37:243
1754175995343,8cyclictest361rcuc/300:34:573
1754175995333,15cyclictest275-21dbus-daemon22:55:013
1754175995327,11cyclictest37-21ksoftirqd/322:23:383
1754175995310,13cyclictest37-21ksoftirqd/321:12:313
1754171995345,5cyclictest1822748-21kworker/2:223:50:272
1754171995345,5cyclictest1787015-21kworker/2:323:20:142
1754171995345,5cyclictest1787015-21kworker/2:322:42:162
1754171995345,5cyclictest1759350-21kworker/2:023:11:352
1754171995345,5cyclictest1759350-21kworker/2:023:11:342
1754171995345,5cyclictest1753367-21kworker/2:319:35:022
1754171995345,5cyclictest1753367-21kworker/2:319:35:012
1754171995344,6cyclictest1759350-21kworker/2:022:02:522
1754171995344,5cyclictest1787015-21kworker/2:323:16:412
1754166995333,7cyclictest23-21ksoftirqd/120:50:011
1754162995339,9cyclictest364-21snmpd20:12:340
175417599527,8cyclictest37-21ksoftirqd/300:20:023
175417599527,8cyclictest37-21ksoftirqd/300:20:013
175417599527,11cyclictest37-21ksoftirqd/322:32:193
175417599526,9cyclictest37-21ksoftirqd/323:33:293
175417599526,9cyclictest37-21ksoftirqd/323:33:283
175417599526,7cyclictest37-21ksoftirqd/321:00:023
175417599524,9cyclictest37-21ksoftirqd/319:30:023
1754175995213,9cyclictest37-21ksoftirqd/321:28:353
175417599521,12cyclictest1757930-21date20:20:023
1754175995210,5cyclictest131rcu_preempt23:24:113
1754171995244,5cyclictest1787015-21kworker/2:322:20:302
1754171995244,5cyclictest1759350-21kworker/2:022:56:112
1754171995243,6cyclictest1787015-21kworker/2:322:26:382
1754171995243,6cyclictest1759350-21kworker/2:021:26:182
1754171995243,5cyclictest1753367-21kworker/2:321:05:022
1754162995237,11cyclictest364-21snmpd20:19:290
1754162995229,20cyclictest1754154-21cyclictest19:10:020
1754162995229,20cyclictest1754154-21cyclictest19:10:020
175417599517,7cyclictest37-21ksoftirqd/323:49:063
175417599517,10cyclictest37-21ksoftirqd/300:20:163
175417599511,7cyclictest1755694-21munin-run19:40:003
1754171995144,4cyclictest1787015-21kworker/2:322:35:102
1754171995144,4cyclictest1787015-21kworker/2:322:31:242
1754171995143,5cyclictest1822748-21kworker/2:223:48:362
1754171995143,5cyclictest1787015-21kworker/2:323:27:032
1754171995143,5cyclictest1753367-21kworker/2:321:36:402
1754171995143,5cyclictest1753367-21kworker/2:321:21:352
1754171995143,5cyclictest1753367-21kworker/2:320:31:152
1754171995143,5cyclictest1753367-21kworker/2:320:31:142
1754171995142,6cyclictest1787015-21kworker/2:323:33:412
1754171995142,6cyclictest1787015-21kworker/2:323:33:402
1754171995142,5cyclictest1759350-21kworker/2:022:12:552
1754171995142,5cyclictest1753367-21kworker/2:321:33:402
1754171995141,6cyclictest1753367-21kworker/2:320:45:022
1754162995132,15cyclictest364-21snmpd21:07:450
175417599504,5cyclictest131rcu_preempt21:46:243
175417599504,5cyclictest131rcu_preempt21:46:243
175417599504,10cyclictest37-21ksoftirqd/322:13:473
1754175995014,10cyclictest37-21ksoftirqd/320:35:013
1754175995014,10cyclictest37-21ksoftirqd/320:35:013
1754175995012,14cyclictest37-21ksoftirqd/319:50:023
1754175995012,14cyclictest37-21ksoftirqd/319:50:013
1754171995042,5cyclictest1787015-21kworker/2:323:38:482
1754171995042,5cyclictest1759350-21kworker/2:022:06:222
1754171995042,5cyclictest1753367-21kworker/2:321:40:282
1754171995042,5cyclictest1753367-21kworker/2:319:15:012
1754171995041,5cyclictest1753367-21kworker/2:320:06:252
175417599498,5cyclictest131rcu_preempt23:18:323
175417599498,5cyclictest131rcu_preempt23:05:553
175417599497,8cyclictest37-21ksoftirqd/321:10:023
175417599497,8cyclictest37-21ksoftirqd/319:55:003
1754171994942,4cyclictest1759350-21kworker/2:022:15:102
1754171994941,5cyclictest1840471-21kworker/2:100:39:302
1754171994941,5cyclictest1840471-21kworker/2:100:27:092
1754171994941,5cyclictest1840471-21kworker/2:100:27:092
1754171994941,5cyclictest1787015-21kworker/2:322:45:582
1754171994940,5cyclictest1840471-21kworker/2:100:22:422
1754166994937,9cyclictest23-21ksoftirqd/100:00:021
1754166994937,9cyclictest23-21ksoftirqd/100:00:011
1754166994935,10cyclictest3467602-21systemd-journal22:05:001
175417599484,6cyclictest37-21ksoftirqd/319:15:033
175417599484,6cyclictest37-21ksoftirqd/319:15:023
175417599484,11cyclictest37-21ksoftirqd/320:00:013
1754175994834,11cyclictest37-21ksoftirqd/320:15:003
175417599482,5cyclictest0-21swapper/319:45:003
1754171994840,5cyclictest1759350-21kworker/2:023:04:372
1754171994840,5cyclictest1759350-21kworker/2:023:04:362
1754171994840,5cyclictest1753367-21kworker/2:319:36:022
1754171994840,5cyclictest1753367-21kworker/2:319:19:322
1754171994840,5cyclictest1753367-21kworker/2:319:19:322
1754166994831,12cyclictest3467602-21systemd-journal00:35:001
1754171994739,5cyclictest1753367-21kworker/2:320:45:322
1754171994739,5cyclictest1753367-21kworker/2:320:15:462
1754171994738,6cyclictest1759350-21kworker/2:023:06:522
1754171994738,5cyclictest1753367-21kworker/2:321:10:002
175416699478,5cyclictest131rcu_preempt00:20:431
1754166994729,14cyclictest3467602-21systemd-journal21:15:001
175416699472,4cyclictest131rcu_preempt19:10:021
175416699472,4cyclictest131rcu_preempt19:10:021
1754162994734,10cyclictest364-21snmpd19:21:520
1754171994639,4cyclictest1753367-21kworker/2:320:36:132
1754171994635,7cyclictest1753367-21kworker/2:320:29:322
1754171994628,13cyclictest3467602-21systemd-journal21:00:012
175417599454,7cyclictest37-21ksoftirqd/319:25:003
1754171994537,5cyclictest1753367-21kworker/2:319:55:022
1754166994533,8cyclictest23-21ksoftirqd/121:20:081
1754166994532,10cyclictest1-21systemd00:30:011
1754166994532,10cyclictest1-21systemd00:30:011
175416699452,3cyclictest131rcu_preempt22:35:021
175416699452,3cyclictest131rcu_preempt21:45:491
175416699452,3cyclictest131rcu_preempt21:45:491
1754175994436,6cyclictest37-21ksoftirqd/320:30:003
1754171994436,5cyclictest1753367-21kworker/2:319:21:022
1754171994435,6cyclictest1753367-21kworker/2:319:30:002
1754171994431,9cyclictest1-21systemd20:15:002
1754166994427,13cyclictest1759971-21latency_hist21:00:021
175416699442,3cyclictest131rcu_preempt23:10:101
175416699442,3cyclictest131rcu_preempt23:10:101
1754166994423,15cyclictest275-21dbus-daemon20:20:011
1754166994415,10cyclictest0-21swapper/123:44:081
175417599437,8cyclictest37-21ksoftirqd/320:05:003
175417599431,9cyclictest1755426-21modprobe19:35:003
175417599431,9cyclictest1755426-21modprobe19:35:003
175417599431,8cyclictest1759059-21sshd20:42:463
1754171994335,5cyclictest1753367-21kworker/2:320:51:182
1754171994335,5cyclictest1753367-21kworker/2:319:45:012
1754166994329,11cyclictest1764838-21latency_hist21:20:021
1754166994329,10cyclictest23-21ksoftirqd/122:52:271
1754166994325,12cyclictest275-21dbus-daemon21:10:001
1754166994315,3cyclictest150-21pr/ttyS223:07:251
175416699431,3cyclictest131rcu_preempt00:10:461
175417599427,9cyclictest37-21ksoftirqd/319:10:023
175417599427,9cyclictest37-21ksoftirqd/319:10:023
175417599422,11cyclictest37-21ksoftirqd/321:05:023
175417599421,5cyclictest0-21swapper/320:08:323
1754171994235,4cyclictest1753367-21kworker/2:319:48:482
1754171994235,4cyclictest1753367-21kworker/2:319:48:482
1754166994230,8cyclictest1800559-21latency_hist22:40:031
1754166994228,9cyclictest1775953-21wc21:45:011
1754166994227,11cyclictest3467602-21systemd-journal19:40:001
1754166994226,12cyclictest23-21ksoftirqd/121:58:071
175416699421,3cyclictest131rcu_preempt00:40:021
1754166994211,6cyclictest131rcu_preempt21:34:441
175417599417,8cyclictest37-21ksoftirqd/320:50:013
1754175994124,10cyclictest275-21dbus-daemon19:15:003
1754171994133,5cyclictest1753367-21kworker/2:320:24:332
1754171994133,5cyclictest1753367-21kworker/2:319:53:502
175416699417,7cyclictest23-21ksoftirqd/122:17:021
175416699415,3cyclictest131rcu_preempt00:08:141
1754166994123,13cyclictest1786995-21systemd-run22:10:021
175416699411,5cyclictest131rcu_preempt23:46:541
175416699411,3cyclictest131rcu_preempt23:00:011
1754166994031,5cyclictest1818267-21sh23:20:001
1754166994030,7cyclictest1755160-21sshd19:27:391
1754166994028,7cyclictest275-21dbus-daemon23:25:001
1754166994027,10cyclictest23-21ksoftirqd/122:45:211
1754166994026,11cyclictest1759772-21sed20:55:021
175416699402,35cyclictest1789205-21cut22:15:001
175416699401,3cyclictest131rcu_preempt22:22:301
175416699399,5cyclictest131rcu_preempt00:00:081
1754166993928,8cyclictest1-21systemd23:55:001
1754166993926,11cyclictest23-21ksoftirqd/123:02:411
1754166993926,11cyclictest23-21ksoftirqd/123:02:401
1754166993924,12cyclictest275-21dbus-daemon19:25:001
175416699383,3cyclictest131rcu_preempt19:50:001
175416699383,3cyclictest131rcu_preempt19:50:001
1754166993828,6cyclictest1754640-21cron19:20:001
1754166993828,6cyclictest1754640-21cron19:20:001
1754166993827,9cyclictest23-21ksoftirqd/122:27:451
1754166993824,12cyclictest23-21ksoftirqd/121:30:001
1754166993824,11cyclictest23-21ksoftirqd/123:39:001
1754166993823,12cyclictest23-21ksoftirqd/121:51:101
175416699382,10cyclictest1843969-21ssh00:17:351
175416699382,10cyclictest1843969-21ssh00:17:351
175416699381,10cyclictest1755943-21sshd19:42:391
1754166993723,11cyclictest23-21ksoftirqd/123:29:301
175416699369,5cyclictest131rcu_preempt21:35:361
1754166993625,9cyclictest23-21ksoftirqd/120:41:461
1754166993625,8cyclictest275-21dbus-daemon20:05:011
1754166993624,9cyclictest23-21ksoftirqd/123:33:161
1754166993624,9cyclictest23-21ksoftirqd/123:33:161
1754166993623,11cyclictest23-21ksoftirqd/122:44:441
1754166993621,10cyclictest23-21ksoftirqd/119:50:371
175417599351,9cyclictest37-21ksoftirqd/320:40:013
1754166993527,6cyclictest23-21ksoftirqd/120:30:031
1754166993523,9cyclictest23-21ksoftirqd/119:15:011
1754166993523,8cyclictest23-21ksoftirqd/120:07:161
1754166993518,12cyclictest3467602-21systemd-journal20:40:011
175416699332,5cyclictest131rcu_preempt20:25:001
1754166993324,6cyclictest23-21ksoftirqd/121:05:021
1754166993319,11cyclictest23-21ksoftirqd/119:56:261
1754166993222,8cyclictest1758641-21latency_hist20:35:021
1754166993222,8cyclictest1758641-21latency_hist20:35:021
175416699313,5cyclictest131rcu_preempt19:35:001
175416699313,5cyclictest131rcu_preempt19:35:001
1754166992920,5cyclictest309-21in:imuxsock20:11:331
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional