You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-27 - 14:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Aug 27, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15254819910841,13cyclictest131rcu_preempt19:30:012
1525472999639,52cyclictest1526978-21run-parts19:40:000
1525472998410,6cyclictest131rcu_preempt00:35:000
1525472998011,5cyclictest131rcu_preempt23:13:560
1525472997813,41cyclictest131rcu_preempt23:25:130
152548199774,4cyclictest131rcu_preempt19:50:022
152547299768,5cyclictest131rcu_preempt00:17:170
1525472997611,6cyclictest131rcu_preempt23:55:120
1525472997611,6cyclictest131rcu_preempt23:55:120
1525486997013,18cyclictest1589371-21cut23:10:023
1525472997025,5cyclictest131rcu_preempt21:56:480
1525481996755,8cyclictest1529595-21kworker/2:122:10:022
152547299672,8cyclictest121ksoftirqd/022:09:570
152547299672,45cyclictest12-21ksoftirqd/021:11:040
152547299667,40cyclictest131rcu_preempt23:47:030
152547299667,40cyclictest131rcu_preempt23:47:030
152547299667,38cyclictest131rcu_preempt22:20:170
152547299667,38cyclictest131rcu_preempt22:20:170
1525472996630,5cyclictest131rcu_preempt00:10:010
1525472996630,5cyclictest131rcu_preempt00:10:010
1525472996611,10cyclictest12-21ksoftirqd/022:25:460
1525472996525,11cyclictest12-21ksoftirqd/023:25:010
1525486996411,16cyclictest1606053-21munin-run23:45:013
1525472996414,6cyclictest131rcu_preempt21:16:210
1525472996414,6cyclictest131rcu_preempt21:16:200
152548699636,21cyclictest1606993-21ssh23:46:443
152548699636,21cyclictest1606993-21ssh23:46:433
1525486996311,14cyclictest1630731-21ssh00:36:103
1525486996311,14cyclictest1630731-21ssh00:36:093
152547299636,28cyclictest131rcu_preempt22:18:210
1525486996216,9cyclictest37-21ksoftirqd/321:53:543
1525486996210,15cyclictest1572149-21ssh22:34:213
1525476996238,14cyclictest23-21ksoftirqd/121:53:461
152547299625,10cyclictest12-21ksoftirqd/000:25:030
152548699616,17cyclictest1575384-21ssh22:40:473
1525486996118,10cyclictest37-21ksoftirqd/323:33:503
1525486996115,11cyclictest37-21ksoftirqd/322:13:113
1525472996132,12cyclictest1557015-21ssh22:02:330
1525486996017,9cyclictest37-21ksoftirqd/300:31:103
1525486996014,10cyclictest37-21ksoftirqd/321:27:133
152547299603,6cyclictest121ksoftirqd/000:10:120
1525472996022,7cyclictest12-21ksoftirqd/021:39:400
152548699599,16cyclictest1580422-21ssh22:51:193
152548699599,14cyclictest1557493-21ssh22:03:343
152548699598,14cyclictest1566470-21ssh22:22:123
152548699598,14cyclictest1566470-21ssh22:22:123
152548699598,13cyclictest1573214-21ssh22:36:203
1525486995910,19cyclictest1620530-21cut00:15:023
152547699596,5cyclictest131rcu_preempt23:50:031
152547699596,5cyclictest131rcu_preempt23:50:021
1525472995932,11cyclictest1582065-21ssh22:55:020
1525472995929,25cyclictest131rcu_preempt23:15:580
1525472995913,5cyclictest131rcu_preempt00:00:490
152547299591,10cyclictest12-21ksoftirqd/021:40:560
1525472995910,7cyclictest131rcu_preempt20:15:000
152548699589,14cyclictest1615636-21systemd-run00:05:013
1525486995819,7cyclictest37-21ksoftirqd/321:14:293
1525486995816,7cyclictest37-21ksoftirqd/323:27:463
152547699584,6cyclictest131rcu_preempt19:20:011
152547299586,7cyclictest12-21ksoftirqd/020:50:010
152547299586,10cyclictest12-21ksoftirqd/022:15:010
152547299581,4cyclictest131rcu_preempt23:05:030
1525472995812,9cyclictest12-21ksoftirqd/023:07:260
1525472995811,9cyclictest0-21swapper/021:10:030
152548699579,9cyclictest37-21ksoftirqd/322:10:013
152548699579,13cyclictest1545388-21ssh21:38:363
1525486995715,8cyclictest37-21ksoftirqd/321:21:383
1525486995713,9cyclictest37-21ksoftirqd/323:13:243
1525486995712,14cyclictest1529423-21cut20:25:013
1525486995712,13cyclictest0-21swapper/300:27:083
1525481995745,8cyclictest1518207-21kworker/2:319:55:002
1525481995745,8cyclictest1518207-21kworker/2:319:55:002
152547299579,5cyclictest131rcu_preempt23:50:490
1525472995721,31cyclictest12-21ksoftirqd/019:46:220
1525472995720,31cyclictest16850irq/134-800000022:34:250
1525472995710,11cyclictest12-21ksoftirqd/021:30:010
152548699569,14cyclictest1542443-21ssh21:32:163
1525486995635,17cyclictest1621439-21cron00:17:003
1525486995615,9cyclictest37-21ksoftirqd/323:20:283
1525481995634,14cyclictest3467602-21systemd-journal20:10:002
152547699563,5cyclictest131rcu_preempt21:40:011
152547299567,5cyclictest131rcu_preempt21:45:280
1525472995613,5cyclictest131rcu_preempt22:45:190
1525486995514,10cyclictest37-21ksoftirqd/322:16:263
152547299556,5cyclictest131rcu_preempt00:20:120
1525472995520,29cyclictest12-21ksoftirqd/021:32:300
1525472995518,32cyclictest16850irq/134-800000022:59:040
1525472995512,6cyclictest131rcu_preempt23:30:580
152548699545,13cyclictest1531654-21diskmemload22:45:063
152547699544,6cyclictest131rcu_preempt00:00:021
152547699544,6cyclictest131rcu_preempt00:00:011
152547299549,7cyclictest131rcu_preempt23:35:290
152547299549,7cyclictest131rcu_preempt23:35:290
152547299549,5cyclictest131rcu_preempt19:20:020
1525472995419,6cyclictest16850irq/134-800000000:37:140
1525472995419,6cyclictest16850irq/134-800000000:37:130
1525472995418,32cyclictest16850irq/134-800000021:21:060
1525472995418,31cyclictest16850irq/134-800000021:50:100
1525472995417,6cyclictest16850irq/134-800000022:42:030
1525472995412,6cyclictest131rcu_preempt19:35:000
1525486995317,9cyclictest37-21ksoftirqd/319:25:023
1525486995314,9cyclictest37-21ksoftirqd/321:40:393
1525486995314,10cyclictest37-21ksoftirqd/320:35:023
1525486995313,8cyclictest37-21ksoftirqd/323:56:393
1525486995313,8cyclictest37-21ksoftirqd/323:56:383
1525481995345,5cyclictest1577299-21kworker/2:323:05:012
1525481995344,6cyclictest1529595-21kworker/2:120:45:022
1525481995343,6cyclictest1577299-21kworker/2:323:22:122
1525472995343,7cyclictest12-21ksoftirqd/023:41:220
152547299533,7cyclictest12-21ksoftirqd/020:00:010
1525472995317,32cyclictest16850irq/134-800000022:37:230
1525486995215,8cyclictest37-21ksoftirqd/323:52:163
1525481995244,5cyclictest1577299-21kworker/2:323:31:332
1525481995244,5cyclictest1529595-21kworker/2:121:21:052
1525481995244,5cyclictest1529595-21kworker/2:121:10:562
1525481995243,6cyclictest1529595-21kworker/2:122:02:172
1525481995243,6cyclictest1529595-21kworker/2:121:52:352
1525481995243,5cyclictest1529595-21kworker/2:122:45:502
1525481995240,8cyclictest1577299-21kworker/2:323:29:432
1525476995235,13cyclictest1-21systemd22:25:021
1525476995235,13cyclictest1-21systemd22:25:011
152547299527,9cyclictest12-21ksoftirqd/020:40:000
1525472995212,5cyclictest131rcu_preempt21:05:000
1525481995143,5cyclictest1529595-21kworker/2:123:11:062
1525481995143,5cyclictest1529595-21kworker/2:122:23:272
1525481995143,5cyclictest1529595-21kworker/2:122:23:272
1525481995142,5cyclictest1529595-21kworker/2:100:03:472
1525481995139,6cyclictest1577299-21kworker/2:322:58:172
1525476995113,6cyclictest131rcu_preempt00:15:301
152547299519,6cyclictest131rcu_preempt20:10:000
152547299511,46cyclictest0-21swapper/020:44:150
152548699505,40cyclictest1603603-21munin-run23:40:003
152548699505,40cyclictest1603603-21munin-run23:40:003
1525486995032,14cyclictest1569640-21ssh22:29:033
1525481995044,3cyclictest1577299-21kworker/2:323:06:372
1525481995043,4cyclictest1626020-21kworker/2:000:34:582
1525481995043,4cyclictest1529595-21kworker/2:121:57:332
1525481995042,5cyclictest1577299-21kworker/2:322:53:192
1525481995042,5cyclictest1567693-21kworker/2:022:31:542
1525481995041,5cyclictest1529595-21kworker/2:122:19:382
1525481995032,12cyclictest309-21in:imuxsock20:20:002
1525481995031,13cyclictest275-21dbus-daemon22:15:012
152547699509,6cyclictest131rcu_preempt23:24:121
152547699507,6cyclictest131rcu_preempt23:40:001
152547699507,6cyclictest131rcu_preempt23:40:001
1525476995041,6cyclictest23-21ksoftirqd/100:35:011
1525476995032,15cyclictest23-21ksoftirqd/123:05:021
1525476995021,11cyclictest23-21ksoftirqd/121:20:011
1525476995021,11cyclictest23-21ksoftirqd/121:20:001
152547699501,8cyclictest0-21swapper/122:49:211
1525476995017,25cyclictest131rcu_preempt22:00:011
152547699501,6cyclictest0-21swapper/122:51:361
152547299509,5cyclictest131rcu_preempt20:30:020
152547299509,5cyclictest131rcu_preempt19:40:580
152548699498,7cyclictest1585781-21ssh23:02:293
1525486994931,14cyclictest1534768-21ssh21:16:143
1525486994931,14cyclictest1534768-21ssh21:16:143
1525481994941,5cyclictest1567693-21kworker/2:022:41:332
1525481994941,5cyclictest1529595-21kworker/2:123:55:542
1525481994941,5cyclictest1529595-21kworker/2:123:55:532
1525481994941,5cyclictest1529595-21kworker/2:100:20:592
1525476994937,7cyclictest221rcuc/123:40:131
152547699492,9cyclictest23-21ksoftirqd/122:30:001
1525476994919,18cyclictest0-21swapper/100:26:171
152547699491,7cyclictest1601225-21modprobe23:35:001
1525476994911,6cyclictest131rcu_preempt20:35:031
152547299498,7cyclictest12-21ksoftirqd/019:10:020
152547299498,6cyclictest131rcu_preempt20:20:020
152547299497,7cyclictest12-21ksoftirqd/020:35:030
1525472994912,6cyclictest131rcu_preempt19:25:000
1525486994833,12cyclictest1584070-21ssh22:59:063
1525486994833,12cyclictest1528507-21sshd20:05:583
1525486994831,13cyclictest1592255-21ssh23:15:543
1525486994810,13cyclictest1554565-21ssh21:57:313
1525481994841,5cyclictest1529595-21kworker/2:100:12:312
1525481994840,5cyclictest1626020-21kworker/2:000:37:512
1525481994840,5cyclictest1626020-21kworker/2:000:37:502
1525481994840,5cyclictest1529595-21kworker/2:121:48:132
1525481994840,5cyclictest1529595-21kworker/2:100:18:282
1525481994838,6cyclictest1541265-21kworker/2:021:40:292
1525476994839,6cyclictest221rcuc/122:15:311
1525476994837,7cyclictest23-21ksoftirqd/119:30:011
152547699483,10cyclictest23-21ksoftirqd/122:00:551
152547699481,8cyclictest0-21swapper/121:47:171
1525476994818,16cyclictest23-21ksoftirqd/121:34:411
1525476994814,7cyclictest131rcu_preempt23:00:001
152547299481,5cyclictest131rcu_preempt19:30:000
1525472994813,6cyclictest131rcu_preempt21:00:020
1525472994813,6cyclictest131rcu_preempt20:55:020
1525472994813,6cyclictest131rcu_preempt20:55:020
1525472994813,6cyclictest131rcu_preempt20:05:010
1525472994813,6cyclictest131rcu_preempt20:05:010
1525486994734,7cyclictest309-21in:imuxsock19:20:013
1525486994731,13cyclictest1623287-21ssh00:20:373
1525481994739,5cyclictest1577299-21kworker/2:323:42:502
1525481994739,5cyclictest1577299-21kworker/2:323:17:442
1525481994739,5cyclictest1541265-21kworker/2:021:39:002
1525481994739,5cyclictest1529595-21kworker/2:121:00:022
1525481994738,6cyclictest1529595-21kworker/2:121:15:182
1525481994738,6cyclictest1529595-21kworker/2:121:15:182
1525481994736,7cyclictest1529595-21kworker/2:122:29:122
1525476994739,6cyclictest221rcuc/100:36:281
1525476994739,6cyclictest221rcuc/100:36:271
1525476994737,7cyclictest23-21ksoftirqd/121:40:321
1525476994730,11cyclictest3467602-21systemd-journal21:30:001
152547699472,5cyclictest131rcu_preempt23:16:161
152547699471,7cyclictest0-21swapper/123:10:471
152547699471,6cyclictest0-21swapper/100:22:171
1525486994634,7cyclictest309-21in:imuxsock19:50:013
1525486994631,12cyclictest1549634-21ssh21:47:123
1525486994631,10cyclictest0-21swapper/319:55:023
1525486994631,10cyclictest0-21swapper/319:55:023
1525486994629,12cyclictest1529037-21cron20:20:003
1525481994639,4cyclictest1529595-21kworker/2:100:27:582
1525481994627,12cyclictest3467602-21systemd-journal20:40:002
152547699469,5cyclictest131rcu_preempt00:12:371
1525476994619,19cyclictest0-21swapper/122:39:301
1525476994618,19cyclictest0-21swapper/122:11:121
152547699461,4cyclictest131rcu_preempt22:45:001
152547299462,7cyclictest3467602-21systemd-journal20:25:000
1525472994610,5cyclictest131rcu_preempt19:55:020
1525472994610,5cyclictest131rcu_preempt19:55:020
1525486994534,8cyclictest1531635-21sshd21:09:583
1525486994523,18cyclictest364-21snmpd19:26:053
1525481994539,4cyclictest1567693-21kworker/2:022:39:362
1525481994537,5cyclictest1529595-21kworker/2:123:48:392
1525481994537,5cyclictest1529595-21kworker/2:123:48:392
1525481994537,5cyclictest1529595-21kworker/2:120:50:012
1525481994535,7cyclictest1529595-21kworker/2:123:53:242
1525481994535,6cyclictest1529595-21kworker/2:121:30:002
152547699458,5cyclictest131rcu_preempt22:33:371
152547699455,4cyclictest131rcu_preempt20:00:001
1525476994528,13cyclictest3467602-21systemd-journal19:35:001
1525476994528,11cyclictest275-21dbus-daemon20:10:011
1525476994513,5cyclictest131rcu_preempt23:50:291
1525476994513,5cyclictest131rcu_preempt20:50:011
1525476994513,14cyclictest23-21ksoftirqd/121:13:071
1525486994432,9cyclictest1525652-21sshd19:13:583
1525486994431,10cyclictest1530838-21sshd20:50:583
1525486994431,10cyclictest1530838-21sshd20:50:583
1525486994429,13cyclictest1616638-21ssh00:06:583
1525486994429,13cyclictest1616638-21ssh00:06:583
1525481994436,5cyclictest1529595-21kworker/2:121:32:202
152547699446,6cyclictest131rcu_preempt19:25:001
152547699441,9cyclictest0-21swapper/121:22:441
152547699441,7cyclictest0-21swapper/100:04:381
1525476994415,24cyclictest131rcu_preempt23:30:011
1525476994413,9cyclictest131rcu_preempt22:09:011
1525476994411,6cyclictest131rcu_preempt21:00:021
152547299448,6cyclictest131rcu_preempt19:15:000
1525486994336,4cyclictest1526979-21munin-run19:40:013
1525486994331,8cyclictest1528015-21sendmail20:00:003
1525486994324,15cyclictest275-21dbus-daemon20:45:013
1525486994310,17cyclictest1531116-21modprobe21:00:003
1525481994333,6cyclictest1531664-21sh21:10:002
152547699431,7cyclictest0-21swapper/100:05:511
152547699431,7cyclictest0-21swapper/100:05:501
1525476994315,15cyclictest23-21ksoftirqd/123:07:331
1525486994229,10cyclictest1530597-21sshd20:48:583
1525486994228,10cyclictest48-21kcompactd019:40:293
1525486994227,11cyclictest1528775-21sshd20:14:583
1525486994225,13cyclictest1-21systemd20:05:013
1525486994225,13cyclictest1-21systemd20:05:013
1525486994223,15cyclictest364-21snmpd20:36:163
1525481994234,5cyclictest1529595-21kworker/2:100:09:262
1525481994234,5cyclictest1529595-21kworker/2:100:09:252
1525481994232,6cyclictest1528065-21kworker/2:020:25:002
1525481994230,9cyclictest1601825-21sshd23:36:002
1525481994230,9cyclictest1601825-21sshd23:36:002
1525476994215,11cyclictest1528978-21cut20:15:021
1525486994129,8cyclictest1526701-21sshd19:34:583
1525486994126,11cyclictest364-21snmpd20:29:183
1525486994117,20cyclictest1531401-21/usr/sbin/munin21:05:013
1525481994129,8cyclictest3467602-21systemd-journal19:15:002
152547699419,5cyclictest131rcu_preempt20:00:581
152547699419,5cyclictest131rcu_preempt20:00:581
1525476994116,7cyclictest221rcuc/119:55:011
1525476994116,7cyclictest221rcuc/119:55:011
1525476994110,5cyclictest131rcu_preempt21:10:021
1525481994032,5cyclictest1529595-21kworker/2:120:55:172
152547699401,5cyclictest0-21swapper/120:18:581
1525481993931,5cyclictest1529595-21kworker/2:120:51:282
1525481993931,5cyclictest1529595-21kworker/2:120:51:282
1525481993927,8cyclictest30-21ksoftirqd/219:33:582
1525481993927,6cyclictest1518207-21kworker/2:320:05:002
1525481993927,6cyclictest1518207-21kworker/2:320:05:002
152547699398,21cyclictest131rcu_preempt20:30:021
152547699396,5cyclictest131rcu_preempt20:40:351
1525476993924,10cyclictest1-21systemd21:05:011
152547699391,4cyclictest0-21swapper/120:22:091
1525481993831,4cyclictest1518207-21kworker/2:319:58:442
1525481993824,10cyclictest0-21swapper/220:30:012
1525481993810,5cyclictest131rcu_preempt20:13:582
152547699386,5cyclictest131rcu_preempt20:36:411
152547699384,5cyclictest131rcu_preempt19:45:261
1525476993828,6cyclictest1526974-21sendmail19:40:001
1525481993727,7cyclictest1526937-21sshd19:35:582
1525481993720,13cyclictest1-21systemd19:25:012
152547699376,6cyclictest131rcu_preempt19:45:001
152547699373,11cyclictest23-21ksoftirqd/119:10:581
1525476993728,5cyclictest1525482-21latency_hist19:10:021
1525486993618,14cyclictest1525557-21latency_hist19:10:023
1525481993526,7cyclictest30-21ksoftirqd/219:45:002
152547699358,3cyclictest131rcu_preempt20:50:581
152547699358,3cyclictest131rcu_preempt20:50:581
152548199334,4cyclictest131rcu_preempt19:10:022
152548199331,29cyclictest48-21kcompactd019:15:312
1525481993218,11cyclictest1-21systemd20:35:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional