You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 00:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Mar 25, 2026 12:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1772437991084,7cyclictest131rcu_preempt12:25:022
1772439998824,6cyclictest131rcu_preempt10:10:033
1772437996820,13cyclictest275-21dbus-daemon09:40:012
1772432996847,13cyclictest1772424-21cyclictest10:30:021
1772429996533,24cyclictest1869134-21munin-run12:40:010
177242999652,38cyclictest131rcu_preempt11:10:050
1772429996341,17cyclictest12-21ksoftirqd/009:27:520
177242999623,37cyclictest131rcu_preempt12:35:010
1772437996154,5cyclictest30-21ksoftirqd/211:50:022
1772437996154,5cyclictest30-21ksoftirqd/211:50:012
1772437996128,14cyclictest1-21systemd09:25:012
1772429996121,20cyclictest131rcu_preempt11:08:220
1772429996121,20cyclictest131rcu_preempt11:08:210
1772429996020,35cyclictest16850irq/134-800000011:36:110
1772437995926,11cyclictest30-21ksoftirqd/207:25:022
1772437995926,11cyclictest30-21ksoftirqd/207:25:012
1772432995941,13cyclictest3467602-21systemd-journal12:30:011
1772432995941,13cyclictest3467602-21systemd-journal12:30:001
1772429995955,2cyclictest0-21swapper/007:35:030
177242999594,34cyclictest131rcu_preempt10:18:560
1772429995929,14cyclictest1794008-21ssh09:45:270
1772429995918,35cyclictest16850irq/134-800000009:45:020
1772429995915,13cyclictest131rcu_preempt09:19:190
1772429995819,33cyclictest16850irq/134-800000011:35:010
1772429995818,20cyclictest16850irq/134-800000012:20:010
1772429995729,12cyclictest1846490-21ssh11:47:290
1772429995729,12cyclictest1846490-21ssh11:47:280
1772429995718,32cyclictest16850irq/134-800000010:40:420
177242999571,39cyclictest131rcu_preempt11:28:380
1772437995636,16cyclictest3467602-21systemd-journal08:40:012
1772429995627,14cyclictest1835800-21ssh11:22:380
1772429995626,27cyclictest12-21ksoftirqd/010:25:020
1772429995625,26cyclictest1853991-21ssh12:05:000
1772429995619,33cyclictest16850irq/134-800000010:48:320
1772429995619,32cyclictest16850irq/134-800000009:36:030
1772429995616,35cyclictest16850irq/134-800000011:52:150
1772437995547,5cyclictest1774702-21kworker/2:309:49:212
1772437995544,6cyclictest1774702-21kworker/2:311:39:592
1772432995510,11cyclictest131rcu_preempt09:15:031
177242999554,18cyclictest1789418-21latency_hist09:35:010
1772429995523,15cyclictest1804429-21ssh10:10:050
1772429995521,30cyclictest12-21ksoftirqd/009:56:330
1772429995519,32cyclictest16850irq/134-800000011:55:530
1772429995519,32cyclictest16850irq/134-800000011:01:190
1772429995518,33cyclictest16850irq/134-800000012:10:220
1772429995518,33cyclictest16850irq/134-800000010:57:310
1772429995518,33cyclictest16850irq/134-800000009:14:060
1772429995517,34cyclictest16850irq/134-800000012:25:320
1772429995517,34cyclictest16850irq/134-800000012:25:320
1772429995517,34cyclictest16850irq/134-800000011:16:320
1772429995517,34cyclictest16850irq/134-800000010:32:110
1772429995517,34cyclictest16850irq/134-800000010:01:370
1772429995516,35cyclictest16850irq/134-800000009:54:110
1772429995516,34cyclictest16850irq/134-800000011:44:090
1772439995438,11cyclictest3467602-21systemd-journal10:14:593
177243999541,9cyclictest131rcu_preempt12:40:003
1772439995410,6cyclictest131rcu_preempt08:55:013
1772437995446,5cyclictest1774702-21kworker/2:311:56:042
177243799544,3cyclictest131rcu_preempt10:56:522
177242999544,5cyclictest131rcu_preempt09:21:440
1772429995429,21cyclictest1856332-21ssh12:10:020
1772429995418,31cyclictest16850irq/134-800000010:35:310
1772429995416,12cyclictest131rcu_preempt08:39:390
1772439995338,10cyclictest1-21systemd09:40:003
1772429995337,11cyclictest275-21dbus-daemon07:20:010
1772429995317,33cyclictest16850irq/134-800000010:27:520
1772429995317,32cyclictest16850irq/134-800000012:21:500
1772429995317,21cyclictest16850irq/134-800000010:06:430
1772437995245,3cyclictest1772317-21kworker/2:007:15:022
1772437995243,6cyclictest1808822-21kworker/2:010:52:402
1772437995243,5cyclictest1774702-21kworker/2:311:33:492
1772437995242,6cyclictest1774702-21kworker/2:309:00:022
1772429995217,31cyclictest16850irq/134-800000010:50:510
1772437995144,4cyclictest1774702-21kworker/2:310:35:402
177243999507,8cyclictest37-21ksoftirqd/311:37:413
1772439995036,10cyclictest3467602-21systemd-journal11:20:013
1772437995042,5cyclictest1774702-21kworker/2:311:18:162
1772437995042,5cyclictest1774702-21kworker/2:310:05:582
1772437995041,6cyclictest1774702-21kworker/2:310:32:282
1772437995041,6cyclictest1774702-21kworker/2:309:57:312
1772437995040,6cyclictest1761932-21kworker/2:110:17:502
177243299504,3cyclictest131rcu_preempt11:25:021
177242999502,5cyclictest131rcu_preempt08:00:020
177242999502,42cyclictest309-21in:imuxsock08:20:000
177243999494,39cyclictest0-21swapper/312:10:033
1772437994942,4cyclictest1774702-21kworker/2:309:43:222
1772437994941,5cyclictest1843200-21kworker/2:112:33:362
1772437994941,5cyclictest1808822-21kworker/2:010:42:472
1772437994941,5cyclictest1808822-21kworker/2:010:21:582
1772439994834,10cyclictest275-21dbus-daemon09:50:003
1772439994832,13cyclictest37-21ksoftirqd/311:20:073
1772437994841,4cyclictest1774702-21kworker/2:308:19:212
1772437994840,5cyclictest1843200-21kworker/2:112:04:052
1772437994840,5cyclictest1808822-21kworker/2:011:14:192
1772437994839,6cyclictest1774702-21kworker/2:308:55:012
177242999481,5cyclictest131rcu_preempt07:50:020
177243999475,7cyclictest131rcu_preempt08:05:013
177243999473,4cyclictest131rcu_preempt09:50:473
1772439994733,11cyclictest275-21dbus-daemon09:45:013
1772439994723,5cyclictest37-21ksoftirqd/309:59:163
1772439994721,6cyclictest37-21ksoftirqd/311:54:563
1772437994740,4cyclictest1774702-21kworker/2:311:44:482
1772437994739,5cyclictest1774702-21kworker/2:310:30:032
1772437994739,5cyclictest1774702-21kworker/2:310:11:072
1772437994739,5cyclictest1774702-21kworker/2:309:30:272
1772437994739,5cyclictest1761932-21kworker/2:110:02:042
1772437994716,7cyclictest131rcu_preempt08:35:002
177243999463,38cyclictest275-21dbus-daemon12:25:003
1772439994633,7cyclictest309-21in:imuxsock10:45:003
1772439994630,12cyclictest3467602-21systemd-journal12:00:003
1772437994638,5cyclictest1843200-21kworker/2:112:39:572
1772437994638,5cyclictest1808822-21kworker/2:010:45:262
1772437994637,6cyclictest1772317-21kworker/2:007:45:012
1772437994615,7cyclictest30-21ksoftirqd/211:30:022
1772432994628,13cyclictest275-21dbus-daemon11:35:001
1772432994625,16cyclictest275-21dbus-daemon08:15:001
1772432994620,16cyclictest23-21ksoftirqd/109:39:011
177243999459,6cyclictest131rcu_preempt10:40:023
177243999456,9cyclictest37-21ksoftirqd/312:26:253
177243999456,9cyclictest37-21ksoftirqd/312:26:243
177243999455,35cyclictest1778061-21modprobe08:59:593
1772439994528,13cyclictest275-21dbus-daemon08:20:023
1772439994522,5cyclictest37-21ksoftirqd/310:25:013
1772439994520,5cyclictest37-21ksoftirqd/312:00:243
1772439994519,5cyclictest37-21ksoftirqd/309:18:533
1772437994537,5cyclictest1774702-21kworker/2:311:21:522
1772437994537,5cyclictest1774702-21kworker/2:309:55:022
1772437994535,7cyclictest1843200-21kworker/2:112:10:142
177243799453,3cyclictest131rcu_preempt12:18:082
1772437994524,13cyclictest3467602-21systemd-journal08:50:002
177243999445,7cyclictest37-21ksoftirqd/310:53:453
177243999445,3cyclictest131rcu_preempt11:30:083
1772439994422,5cyclictest37-21ksoftirqd/311:48:293
1772439994422,5cyclictest37-21ksoftirqd/311:48:293
1772439994419,6cyclictest37-21ksoftirqd/309:28:523
1772439994419,4cyclictest37-21ksoftirqd/308:35:033
1772439994418,5cyclictest37-21ksoftirqd/312:18:263
1772439994418,4cyclictest37-21ksoftirqd/311:10:223
1772439994417,6cyclictest37-21ksoftirqd/310:15:173
177243999441,4cyclictest131rcu_preempt09:24:163
1772437994437,4cyclictest1808822-21kworker/2:011:05:132
1772437994437,4cyclictest1808822-21kworker/2:011:05:132
1772437994437,4cyclictest1808822-21kworker/2:011:03:002
1772437994431,10cyclictest1774913-21sshd07:57:222
1772437994430,10cyclictest3467602-21systemd-journal08:29:222
1772432994435,5cyclictest1815006-21sh10:34:591
1772432994430,10cyclictest3467602-21systemd-journal12:00:021
1772432994418,17cyclictest1788676-21ssh09:33:171
1772429994430,11cyclictest3467602-21systemd-journal07:42:310
1772429994427,12cyclictest12-21ksoftirqd/009:00:010
1772439994334,6cyclictest1825822-21cron11:00:003
1772439994334,5cyclictest309-21in:imuxsock07:40:013
177243999432,8cyclictest37-21ksoftirqd/311:41:433
1772439994317,6cyclictest37-21ksoftirqd/309:34:153
1772439994317,5cyclictest37-21ksoftirqd/311:07:473
1772439994317,5cyclictest37-21ksoftirqd/311:07:463
177243999431,4cyclictest131rcu_preempt12:10:183
1772439994314,5cyclictest37-21ksoftirqd/309:11:043
1772437994333,7cyclictest1-21systemd08:10:012
1772432994330,9cyclictest1-21systemd07:40:021
1772432994317,17cyclictest23-21ksoftirqd/112:20:021
1772429994318,21cyclictest16850irq/134-800000008:30:210
177243999424,3cyclictest131rcu_preempt08:00:013
1772439994230,8cyclictest1-21systemd07:35:003
1772439994220,5cyclictest37-21ksoftirqd/310:34:303
1772439994219,5cyclictest37-21ksoftirqd/310:01:293
1772439994218,4cyclictest37-21ksoftirqd/311:28:143
1772439994218,4cyclictest37-21ksoftirqd/311:02:313
1772437994234,5cyclictest1761932-21kworker/2:109:10:012
1772437994234,5cyclictest1761932-21kworker/2:108:15:012
1772437994233,6cyclictest1843200-21kworker/2:112:29:332
1772437994233,6cyclictest1843200-21kworker/2:112:29:332
1772437994229,10cyclictest1-21systemd11:55:012
1772437994228,11cyclictest30-21ksoftirqd/207:35:002
1772437994222,14cyclictest3467602-21systemd-journal08:45:002
177243299429,23cyclictest0-21swapper/111:56:201
177243299428,4cyclictest0-21swapper/111:27:431
177243299423,3cyclictest131rcu_preempt12:06:361
1772432994228,11cyclictest1845389-21wc11:45:011
1772432994228,10cyclictest3467602-21systemd-journal07:50:011
1772432994228,10cyclictest275-21dbus-daemon07:55:021
1772432994226,12cyclictest275-21dbus-daemon11:00:011
1772432994225,13cyclictest1777033-21systemctl08:40:001
177243299421,3cyclictest131rcu_preempt10:40:011
1772429994236,4cyclictest16950irq/224-800000007:40:010
1772429994235,4cyclictest16950irq/224-800000007:51:200
1772429994229,10cyclictest1-21systemd08:55:010
177242999422,36cyclictest1778792-21latency_hist09:10:020
1772429994223,15cyclictest16950irq/224-800000007:23:530
1772429994223,15cyclictest16950irq/224-800000007:23:530
177242999421,7cyclictest1772483-21latency_hist07:10:010
177243999419,5cyclictest131rcu_preempt07:51:253
177243999416,6cyclictest131rcu_preempt10:26:383
1772439994121,4cyclictest37-21ksoftirqd/307:45:003
1772439994120,5cyclictest37-21ksoftirqd/308:45:003
1772439994119,5cyclictest37-21ksoftirqd/312:32:013
177243999411,8cyclictest37-21ksoftirqd/310:48:353
1772439994117,5cyclictest37-21ksoftirqd/308:50:013
177243799418,3cyclictest131rcu_preempt09:15:432
1772437994133,5cyclictest1843200-21kworker/2:112:07:512
1772437994131,6cyclictest1774702-21kworker/2:309:12:172
1772432994130,8cyclictest23-21ksoftirqd/111:40:011
1772432994130,7cyclictest1776238-21sshd08:22:211
1772432994130,6cyclictest1785098-21latency_hist09:25:001
177243299412,6cyclictest131rcu_preempt08:20:011
177242999414,33cyclictest0-21swapper/007:13:520
177242999411,6cyclictest1775474-21modprobe08:10:000
1772439994024,12cyclictest3467602-21systemd-journal07:15:213
1772439994018,5cyclictest37-21ksoftirqd/308:40:013
1772437994029,7cyclictest309-21in:imuxsock09:30:002
1772432994028,9cyclictest23-21ksoftirqd/112:21:101
1772432994026,10cyclictest3467602-21systemd-journal10:20:001
177243299402,3cyclictest131rcu_preempt12:15:011
177243299402,3cyclictest131rcu_preempt09:45:521
177243299401,3cyclictest131rcu_preempt12:30:071
177243299401,3cyclictest131rcu_preempt11:45:311
177243299401,3cyclictest131rcu_preempt11:45:301
177243299401,3cyclictest131rcu_preempt10:02:381
1772429994030,7cyclictest12-21ksoftirqd/008:05:010
1772429994026,10cyclictest275-21dbus-daemon08:15:010
1772429994015,6cyclictest12-21ksoftirqd/008:30:010
1772439993928,7cyclictest37-21ksoftirqd/309:05:023
1772439993922,13cyclictest1776254-21cron08:24:593
1772439993915,4cyclictest37-21ksoftirqd/308:15:023
1772439993913,5cyclictest37-21ksoftirqd/309:10:003
1772437993929,5cyclictest309-21in:imuxsock07:30:002
1772432993928,7cyclictest309-21in:imuxsock10:15:001
1772432993927,9cyclictest3467602-21systemd-journal11:12:321
1772432993927,9cyclictest23-21ksoftirqd/111:01:471
1772432993926,10cyclictest23-21ksoftirqd/110:00:011
1772432993924,11cyclictest3467602-21systemd-journal09:29:001
1772432993924,11cyclictest275-21dbus-daemon12:40:001
1772432993923,12cyclictest1803891-21sshd10:09:201
177243299391,3cyclictest131rcu_preempt10:24:361
1772429993924,10cyclictest275-21dbus-daemon09:05:010
1772429993913,5cyclictest12-21ksoftirqd/008:45:010
1772439993828,6cyclictest1775469-21sh08:10:003
1772439993817,12cyclictest37-21ksoftirqd/307:10:023
1772439993811,5cyclictest37-21ksoftirqd/307:30:003
177243299384,3cyclictest131rcu_preempt11:07:531
177243299384,3cyclictest131rcu_preempt11:07:521
177243299382,3cyclictest131rcu_preempt09:10:001
1772432993817,18cyclictest23-21ksoftirqd/109:53:431
1772432993816,15cyclictest275-21dbus-daemon10:50:011
1772429993827,8cyclictest1777756-21date08:50:020
1772439993718,5cyclictest37-21ksoftirqd/308:30:003
1772439993714,5cyclictest37-21ksoftirqd/307:25:013
1772439993714,5cyclictest37-21ksoftirqd/307:25:003
1772437993729,5cyclictest1774702-21kworker/2:308:04:002
1772437993723,11cyclictest3467602-21systemd-journal07:17:002
1772437993721,12cyclictest1778506-21sed09:05:022
1772432993728,7cyclictest1823637-21sh10:55:011
1772432993727,8cyclictest23-21ksoftirqd/109:00:011
1772432993725,9cyclictest23-21ksoftirqd/108:45:021
177243299371,3cyclictest131rcu_preempt11:54:541
177242999373,3cyclictest131rcu_preempt08:25:000
177243999361,3cyclictest131rcu_preempt07:15:023
1772437993622,10cyclictest275-21dbus-daemon08:25:002
1772432993629,4cyclictest1780798-21cron09:15:001
1772432993628,4cyclictest1772835-21cron07:17:011
1772432993625,8cyclictest23-21ksoftirqd/107:45:021
177243299362,3cyclictest131rcu_preempt11:20:021
1772432993623,10cyclictest1773371-21sshd07:26:211
1772432993618,12cyclictest275-21dbus-daemon08:05:011
177242999364,3cyclictest131rcu_preempt07:30:000
1772437993525,7cyclictest30-21ksoftirqd/207:45:282
1772432993525,8cyclictest23-21ksoftirqd/110:44:101
177243299352,3cyclictest131rcu_preempt08:00:001
1772432993523,9cyclictest1793654-21cron09:45:011
1772432993522,10cyclictest23-21ksoftirqd/107:15:011
1772432993521,11cyclictest1776969-21sed08:35:031
177243999343,3cyclictest131rcu_preempt07:48:583
1772432993422,8cyclictest3467602-21systemd-journal07:23:431
1772432993422,8cyclictest3467602-21systemd-journal07:23:421
177243299342,18cyclictest23-21ksoftirqd/109:05:021
1772432993420,11cyclictest23-21ksoftirqd/108:46:261
177243299341,3cyclictest131rcu_preempt07:35:001
1772437993323,8cyclictest30-21ksoftirqd/207:37:592
1772432993323,8cyclictest23-21ksoftirqd/107:10:011
1772432993323,6cyclictest1775462-21cron08:10:001
177243299331,5cyclictest0-21swapper/108:55:001
1772432993223,7cyclictest23-21ksoftirqd/108:30:011
1772437993120,9cyclictest30-21ksoftirqd/207:52:272
1772437992818,7cyclictest30-21ksoftirqd/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional