You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-22 - 09:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Jul 22, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1731502996634,6cyclictest1735099-21kworker/2:121:30:012
1731502996542,8cyclictest1763916-21kworker/2:223:13:482
1731493996421,21cyclictest1788162-21ssh22:55:300
1731493996420,23cyclictest1751040-21ssh21:37:520
1731504996211,6cyclictest131rcu_preempt21:40:013
1731502996238,15cyclictest1724396-21kworker/2:300:28:472
1731493996219,21cyclictest1823467-21ssh00:09:390
1731493996120,19cyclictest1784878-21ssh22:48:440
173149399611,40cyclictest0-21swapper/022:26:360
1731502996052,5cyclictest1724396-21kworker/2:323:54:182
1731502996043,7cyclictest1724396-21kworker/2:300:24:182
1731493996017,22cyclictest1770213-21ssh22:18:020
1731493995932,12cyclictest1820083-21ssh00:02:230
1731493995925,30cyclictest12-21ksoftirqd/023:20:020
1731493995917,21cyclictest1825564-21ssh00:14:070
1731502995836,13cyclictest1724396-21kworker/2:323:27:092
1731502995816,9cyclictest0-21swapper/223:42:252
1731502995810,25cyclictest1802347-21kworker/u8:300:00:022
173149799589,5cyclictest131rcu_preempt23:55:021
1731502995734,14cyclictest1724396-21kworker/2:322:10:022
1731502995734,14cyclictest1724396-21kworker/2:322:10:022
173150299573,40cyclictest1766415-21ssh22:10:022
1731493995727,15cyclictest1738368-21ssh21:11:080
1731493995724,8cyclictest12-21ksoftirqd/023:55:280
173149399572,39cyclictest1752411-21ssh21:40:280
173149399572,39cyclictest1752411-21ssh21:40:280
173149399571,39cyclictest1802460-21ssh23:25:310
1731493995643,7cyclictest16950irq/224-800000023:45:000
1731493995629,12cyclictest1812780-21ssh23:47:180
1731493995628,24cyclictest1773305-21ssh22:24:440
1731493995628,24cyclictest1773305-21ssh22:24:440
1731493995627,13cyclictest12-21ksoftirqd/022:45:010
1731493995626,14cyclictest1766654-21ssh22:10:210
1731493995626,14cyclictest1766654-21ssh22:10:210
173149399562,44cyclictest0-21swapper/000:37:160
1731502995541,5cyclictest1724396-21kworker/2:321:58:432
173149799553,5cyclictest131rcu_preempt00:00:001
1731493995528,11cyclictest1756648-21ssh21:49:510
1731493995524,10cyclictest1808572-21ssh23:38:310
173149399551,38cyclictest1747212-21ssh21:29:580
1731502995446,5cyclictest1735099-21kworker/2:121:17:082
1731502995445,6cyclictest1724396-21kworker/2:322:44:032
1731497995430,6cyclictest131rcu_preempt22:55:031
1731493995427,12cyclictest1795459-21ssh23:10:500
1731493995426,13cyclictest1763024-21ssh22:02:560
1731493995426,13cyclictest1763024-21ssh22:02:560
1731493995426,13cyclictest1744496-21ssh21:24:110
173150499537,8cyclictest37-21ksoftirqd/323:50:013
173150499537,6cyclictest131rcu_preempt22:10:023
173150499536,43cyclictest1754544-21ssh21:45:033
173150499536,43cyclictest1754544-21ssh21:45:023
1731504995314,9cyclictest131rcu_preempt22:45:003
1731502995345,5cyclictest1763916-21kworker/2:223:09:262
1731502995345,5cyclictest1735099-21kworker/2:121:32:462
1731502995345,5cyclictest1724396-21kworker/2:322:33:162
1731502995344,6cyclictest1724396-21kworker/2:322:03:542
1731502995344,6cyclictest1724396-21kworker/2:322:03:532
1731502995344,5cyclictest1724396-21kworker/2:320:45:002
1731493995328,10cyclictest1742080-21ssh21:19:120
173150499528,6cyclictest131rcu_preempt00:35:013
173150499528,6cyclictest131rcu_preempt00:35:013
1731502995244,5cyclictest1763916-21kworker/2:222:27:002
1731502995244,5cyclictest1724396-21kworker/2:322:50:442
1731502995244,5cyclictest1724396-21kworker/2:322:50:442
1731502995244,5cyclictest1724396-21kworker/2:322:47:142
1731502995243,6cyclictest1724396-21kworker/2:320:55:022
1731502995242,5cyclictest1724396-21kworker/2:323:18:562
1731502995231,5cyclictest1724396-21kworker/2:323:37:482
173149799526,9cyclictest1833212-21ssh00:30:021
1731493995226,10cyclictest12-21ksoftirqd/000:35:000
1731493995226,10cyclictest12-21ksoftirqd/000:35:000
173149399522,43cyclictest0-21swapper/021:33:120
1731493995224,13cyclictest1805376-21ssh23:31:410
1731493995224,13cyclictest1792172-21ssh23:04:160
173149399522,29cyclictest131rcu_preempt00:22:110
173149399521,42cyclictest0-21swapper/022:39:050
173150499519,5cyclictest131rcu_preempt21:58:133
1731504995130,17cyclictest275-21dbus-daemon22:15:013
1731504995130,17cyclictest275-21dbus-daemon22:15:003
173150499512,4cyclictest131rcu_preempt23:40:033
1731502995143,5cyclictest1735099-21kworker/2:121:43:222
1731502995143,5cyclictest1735099-21kworker/2:121:43:222
1731502995143,5cyclictest1724396-21kworker/2:323:46:002
1731502995143,5cyclictest1724396-21kworker/2:322:55:312
1731502995143,5cyclictest1724396-21kworker/2:322:35:162
1731502995143,5cyclictest1724396-21kworker/2:300:30:412
1731502995143,5cyclictest1724396-21kworker/2:300:30:412
1731502995142,6cyclictest1735099-21kworker/2:121:37:512
1731502995135,13cyclictest3467602-21systemd-journal19:32:182
1731493995136,12cyclictest275-21dbus-daemon21:10:010
1731493995128,9cyclictest1776945-21ssh22:32:000
173150499506,7cyclictest131rcu_preempt00:00:023
173150499504,6cyclictest37-21ksoftirqd/320:55:023
1731504995027,9cyclictest37-21ksoftirqd/321:14:553
173150499502,43cyclictest37-21ksoftirqd/322:25:003
173150499502,43cyclictest37-21ksoftirqd/322:24:593
173150499501,5cyclictest131rcu_preempt22:33:163
1731504995010,6cyclictest131rcu_preempt21:30:183
1731502995042,5cyclictest1724396-21kworker/2:323:20:552
1731502995042,5cyclictest1724396-21kworker/2:323:00:412
1731502995042,5cyclictest1724396-21kworker/2:321:10:052
1731502995042,5cyclictest1724396-21kworker/2:300:07:222
1731502995030,5cyclictest1724396-21kworker/2:300:01:522
1731502995015,6cyclictest131rcu_preempt21:22:352
1731497995021,21cyclictest23-21ksoftirqd/100:02:191
1731493995031,14cyclictest1-21systemd23:10:010
1731493995026,10cyclictest1737666-21diskmemload23:53:110
1731493995023,19cyclictest0-21swapper/000:25:400
1731493995014,31cyclictest16850irq/134-800000023:23:270
173149399501,3cyclictest131rcu_preempt22:09:320
173150499496,6cyclictest131rcu_preempt21:16:243
173150499496,6cyclictest131rcu_preempt00:15:003
1731504994934,11cyclictest1-21systemd20:40:013
1731504994934,11cyclictest1-21systemd20:40:013
173150499492,5cyclictest131rcu_preempt20:45:003
1731502994941,5cyclictest1724396-21kworker/2:323:31:572
1731502994941,5cyclictest1724396-21kworker/2:321:48:502
1731502994941,5cyclictest1724396-21kworker/2:319:16:032
1731502994941,5cyclictest1724396-21kworker/2:300:39:172
173149799499,6cyclictest131rcu_preempt22:50:021
173149799499,5cyclictest131rcu_preempt21:20:001
1731497994932,13cyclictest1-21systemd22:30:011
173149799491,5cyclictest131rcu_preempt23:10:011
1731497994915,8cyclictest131rcu_preempt22:00:021
173149399493,3cyclictest131rcu_preempt00:15:480
1731493994917,28cyclictest275-21dbus-daemon21:55:020
1731493994917,28cyclictest275-21dbus-daemon21:55:020
173150499484,5cyclictest131rcu_preempt22:28:193
1731504994832,13cyclictest3467602-21systemd-journal20:15:013
1731504994829,14cyclictest275-21dbus-daemon21:25:013
1731504994819,19cyclictest37-21ksoftirqd/300:17:223
1731504994813,6cyclictest131rcu_preempt21:05:003
1731502994840,5cyclictest1763916-21kworker/2:222:21:242
1731502994840,5cyclictest1763916-21kworker/2:222:21:242
1731502994840,5cyclictest1763916-21kworker/2:222:19:322
173149799486,9cyclictest131rcu_preempt19:45:021
173149799486,9cyclictest131rcu_preempt19:45:011
173149799483,6cyclictest131rcu_preempt22:15:001
173149799483,6cyclictest131rcu_preempt22:15:001
173149799483,6cyclictest131rcu_preempt19:20:001
1731497994833,11cyclictest1756739-21systemd-run21:50:011
1731497994819,19cyclictest23-21ksoftirqd/121:21:091
173149799481,5cyclictest0-21swapper/100:35:021
173149799481,5cyclictest0-21swapper/100:35:021
1731497994811,6cyclictest131rcu_preempt23:45:001
1731493994824,9cyclictest1759768-21ssh21:55:540
173150499478,6cyclictest131rcu_preempt23:05:003
173150499478,5cyclictest131rcu_preempt23:28:133
173150499476,7cyclictest1819039-21kworker/u8:000:29:333
173150499476,6cyclictest131rcu_preempt23:40:533
173150499473,6cyclictest131rcu_preempt19:45:013
173150499473,6cyclictest131rcu_preempt19:45:003
173150499471,6cyclictest131rcu_preempt21:30:003
1731504994710,7cyclictest131rcu_preempt23:20:013
1731502994739,5cyclictest1724396-21kworker/2:319:20:022
1731502994738,5cyclictest1724396-21kworker/2:300:10:582
1731502994729,14cyclictest275-21dbus-daemon20:15:012
1731502994727,5cyclictest1724396-21kworker/2:320:30:022
1731502994727,5cyclictest1724396-21kworker/2:320:30:012
173149799472,10cyclictest1771082-21cron22:20:001
1731497994718,6cyclictest131rcu_preempt22:35:291
1731497994718,24cyclictest23-21ksoftirqd/119:40:011
1731493994712,31cyclictest0-21swapper/020:51:170
173150499469,6cyclictest131rcu_preempt21:10:023
173150499466,8cyclictest37-21ksoftirqd/322:19:023
173150499464,5cyclictest131rcu_preempt23:25:023
173150499464,3cyclictest131rcu_preempt22:01:013
173150499464,3cyclictest131rcu_preempt22:01:013
1731504994632,10cyclictest1-21systemd19:15:013
1731504994631,11cyclictest37-21ksoftirqd/320:20:013
1731504994615,22cyclictest1731934-21modprobe19:19:593
1731504994612,8cyclictest131rcu_preempt00:05:223
1731504994610,5cyclictest131rcu_preempt19:35:013
1731502994638,5cyclictest1724396-21kworker/2:321:54:432
1731502994638,5cyclictest1724396-21kworker/2:321:54:422
1731502994634,8cyclictest1-21systemd19:40:002
173149799469,6cyclictest131rcu_preempt20:00:011
173149799465,6cyclictest23-21ksoftirqd/122:03:091
173149799465,6cyclictest23-21ksoftirqd/122:03:081
173149799463,8cyclictest23-21ksoftirqd/123:35:481
173149799463,7cyclictest131rcu_preempt22:45:021
173149799463,5cyclictest131rcu_preempt19:30:011
173149799463,5cyclictest131rcu_preempt19:30:011
1731497994629,13cyclictest275-21dbus-daemon19:35:001
1731497994623,5cyclictest23-21ksoftirqd/100:15:011
1731497994616,4cyclictest23-21ksoftirqd/119:15:021
173149799461,5cyclictest131rcu_preempt21:15:011
1731497994610,6cyclictest131rcu_preempt00:07:561
1731493994628,15cyclictest1787529-21ssh22:54:270
1731493994628,15cyclictest1787529-21ssh22:54:260
173150499458,5cyclictest131rcu_preempt21:46:033
173150499457,8cyclictest1778971-21ssh22:36:133
173150499457,6cyclictest131rcu_preempt19:40:013
173150499455,5cyclictest131rcu_preempt22:52:513
173150499455,5cyclictest131rcu_preempt22:52:513
173150499454,4cyclictest131rcu_preempt23:08:293
173150499453,7cyclictest131rcu_preempt23:10:253
173150499452,12cyclictest1819966-21sh00:02:093
173150499451,6cyclictest131rcu_preempt23:55:023
1731502994538,4cyclictest1724396-21kworker/2:300:19:062
1731502994537,5cyclictest1724396-21kworker/2:321:08:152
173149799457,6cyclictest131rcu_preempt20:40:011
173149799457,6cyclictest131rcu_preempt20:40:011
1731497994518,5cyclictest23-21ksoftirqd/121:44:331
1731497994518,5cyclictest23-21ksoftirqd/121:44:331
173149799451,7cyclictest131rcu_preempt20:55:011
173149799451,5cyclictest131rcu_preempt23:31:111
1731493994528,14cyclictest1735569-21sshd20:28:180
1731493994528,14cyclictest1735569-21sshd20:28:170
1731493994520,20cyclictest1732723-21latency_hist19:35:020
173150499443,7cyclictest131rcu_preempt21:00:023
173150499443,7cyclictest131rcu_preempt21:00:023
173150499441,6cyclictest131rcu_preempt20:34:593
1731504994412,6cyclictest131rcu_preempt23:35:003
173149799448,6cyclictest131rcu_preempt21:40:021
173149799446,6cyclictest131rcu_preempt00:40:001
173149799444,5cyclictest131rcu_preempt21:30:021
173149799443,6cyclictest131rcu_preempt22:10:001
1731497994435,6cyclictest23-21ksoftirqd/120:20:011
1731497994432,9cyclictest3467602-21systemd-journal20:27:191
1731497994432,9cyclictest3467602-21systemd-journal20:27:191
1731497994428,11cyclictest1-21systemd00:20:011
173149799442,12cyclictest23-21ksoftirqd/120:35:001
1731497994418,6cyclictest23-21ksoftirqd/123:14:581
1731497994417,21cyclictest131rcu_preempt21:55:031
1731497994410,6cyclictest131rcu_preempt22:35:001
1731493994429,11cyclictest3467602-21systemd-journal19:50:000
1731493994427,14cyclictest1732943-21sshd19:35:180
1731493994414,7cyclictest131rcu_preempt19:20:010
1731493994414,28cyclictest12-21ksoftirqd/019:15:010
173149399441,21cyclictest0-21swapper/019:55:030
1731493994411,29cyclictest12-21ksoftirqd/020:36:170
1731493994411,29cyclictest12-21ksoftirqd/020:36:160
173150499439,4cyclictest131rcu_preempt00:24:213
173150499436,5cyclictest131rcu_preempt22:57:193
1731504994319,19cyclictest37-21ksoftirqd/321:51:243
1731504994319,19cyclictest37-21ksoftirqd/321:51:243
173150499431,5cyclictest131rcu_preempt22:45:423
173150499431,5cyclictest131rcu_preempt00:36:443
1731502994335,5cyclictest1724396-21kworker/2:321:04:012
1731502994335,5cyclictest1724396-21kworker/2:319:54:182
1731502994335,5cyclictest1724396-21kworker/2:319:10:022
1731502994334,5cyclictest1724396-21kworker/2:320:00:022
173149799437,7cyclictest23-21ksoftirqd/121:10:021
173149799437,6cyclictest131rcu_preempt21:05:011
173149799434,3cyclictest131rcu_preempt23:02:341
1731497994330,10cyclictest1829717-21sshd00:22:341
1731497994318,12cyclictest0-21swapper/123:22:531
1731497994317,5cyclictest23-21ksoftirqd/121:28:241
173150499428,6cyclictest131rcu_preempt20:10:013
173150299424,3cyclictest131rcu_preempt20:10:032
173150299424,3cyclictest131rcu_preempt19:15:022
173149799427,4cyclictest131rcu_preempt22:51:001
173149799427,4cyclictest131rcu_preempt22:50:591
173149799425,5cyclictest131rcu_preempt22:25:011
173149799425,5cyclictest131rcu_preempt22:25:001
173149799423,5cyclictest131rcu_preempt20:15:001
173149799423,3cyclictest131rcu_preempt23:28:521
1731497994218,5cyclictest23-21ksoftirqd/121:00:001
1731497994218,5cyclictest23-21ksoftirqd/120:59:591
173149799421,4cyclictest131rcu_preempt23:15:191
173149799421,4cyclictest131rcu_preempt20:05:011
1731497994210,6cyclictest131rcu_preempt23:47:161
1731493994235,4cyclictest16950irq/224-800000019:25:010
173150499413,5cyclictest131rcu_preempt19:30:023
173150499413,5cyclictest131rcu_preempt19:30:013
1731502994133,5cyclictest1724396-21kworker/2:320:04:542
173149799416,7cyclictest23-21ksoftirqd/119:49:191
173149799414,3cyclictest131rcu_preempt21:53:401
173149799412,5cyclictest131rcu_preempt20:25:021
1731493994134,4cyclictest16950irq/224-800000021:00:010
1731493994134,4cyclictest16950irq/224-800000021:00:010
1731493994134,4cyclictest16950irq/224-800000019:30:000
1731493994134,4cyclictest16950irq/224-800000019:30:000
173150499409,5cyclictest131rcu_preempt20:20:113
1731502994033,4cyclictest1724396-21kworker/2:319:47:172
1731502994032,5cyclictest1724396-21kworker/2:320:35:012
1731497994015,6cyclictest131rcu_preempt20:44:591
1731493994028,9cyclictest0-21swapper/019:10:020
1731493994027,10cyclictest12-21ksoftirqd/021:05:000
173149399401,35cyclictest12-21ksoftirqd/020:05:500
173150499395,4cyclictest131rcu_preempt20:45:183
173150499394,5cyclictest131rcu_preempt20:30:033
173150499394,5cyclictest131rcu_preempt20:30:033
173150499391,5cyclictest131rcu_preempt20:00:033
1731504993910,4cyclictest131rcu_preempt19:25:013
1731502993927,8cyclictest1-21systemd20:25:012
173149399395,5cyclictest131rcu_preempt20:50:000
1731493993924,11cyclictest12-21ksoftirqd/020:05:000
173150499384,4cyclictest131rcu_preempt19:55:003
1731502993831,4cyclictest1724396-21kworker/2:319:30:022
1731502993831,4cyclictest1724396-21kworker/2:319:30:012
1731497993826,9cyclictest23-21ksoftirqd/120:48:231
173149799381,5cyclictest131rcu_preempt19:55:021
1731497993811,10cyclictest23-21ksoftirqd/119:21:381
1731493993829,6cyclictest1735088-21sendmail-msp20:20:010
173149399382,31cyclictest1734812-21munin-run20:15:010
1731493993822,12cyclictest3467602-21systemd-journal20:41:020
173150499371,6cyclictest131rcu_preempt19:46:443
1731502993729,5cyclictest1724396-21kworker/2:319:45:022
1731502993729,5cyclictest1724396-21kworker/2:319:45:022
173149799371,6cyclictest131rcu_preempt20:10:031
1731493993725,8cyclictest3467602-21systemd-journal19:55:180
173150499361,5cyclictest131rcu_preempt20:05:023
173150299362,3cyclictest131rcu_preempt20:40:022
173150299362,3cyclictest131rcu_preempt20:40:012
173150299351,3cyclictest131rcu_preempt21:00:022
173150299351,3cyclictest131rcu_preempt21:00:012
173149399352,5cyclictest1735855-21systemd-run20:35:010
1731502993423,8cyclictest1-21systemd20:50:002
173150299341,3cyclictest131rcu_preempt20:20:022
173149399341,30cyclictest0-21swapper/019:42:070
173149399341,30cyclictest0-21swapper/019:42:070
173149399341,29cyclictest0-21swapper/020:21:150
173150499332,5cyclictest131rcu_preempt19:10:023
173149799331,5cyclictest0-21swapper/119:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional