You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-04 - 20:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon May 04, 2026 12:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1440864999624,44cyclictest1452704-21ssh09:21:340
1440864998817,61cyclictest16850irq/134-800000010:12:200
1440864998731,52cyclictest1495913-21ssh10:51:030
1440864998715,62cyclictest16850irq/134-800000011:55:130
144086499852,65cyclictest364-21snmpd12:23:410
1440864998218,10cyclictest1451153-21ssh09:18:330
1440864998028,34cyclictest1460986-21ssh09:39:050
1440864998028,34cyclictest1460986-21ssh09:39:040
1440864998025,37cyclictest1530871-21ssh12:03:540
1440864998025,37cyclictest1516074-21ssh11:33:030
1440864998022,54cyclictest3467602-21systemd-journal09:11:200
1440864998017,42cyclictest16850irq/134-800000012:15:220
1440864998013,6cyclictest16850irq/134-800000010:28:020
1440864997921,53cyclictest1483179-21cat10:25:020
1440864997916,17cyclictest1546499-21ssh12:35:570
1440864997827,33cyclictest1457395-21ssh09:31:130
1440864997818,42cyclictest16850irq/134-800000011:35:570
1440864997818,42cyclictest16850irq/134-800000011:35:560
1440864997721,52cyclictest1462677-21sshd09:42:190
1440864997630,30cyclictest1494703-21ssh10:48:500
1440864997328,27cyclictest1470950-21ssh09:59:500
1440864997325,32cyclictest1468814-21ssh09:55:020
1440864997221,34cyclictest1473145-21ssh10:04:130
1440864997129,24cyclictest1504616-21ssh11:09:270
1440864997117,12cyclictest1521938-21ssh11:45:020
1440864997029,25cyclictest1497886-21ssh10:55:090
1440864997026,27cyclictest1501573-21ssh11:02:550
1440864997023,30cyclictest1492746-21ssh10:44:520
1440864996923,40cyclictest1507301-21ssh11:15:010
144086499691,4cyclictest0-21swapper/007:25:010
1440864996912,52cyclictest16850irq/134-800000011:18:310
1440864996821,29cyclictest1487199-21ssh10:33:120
1440864996717,2cyclictest0-21swapper/009:50:020
144087899668,11cyclictest37-21ksoftirqd/312:40:023
1440878996610,9cyclictest37-21ksoftirqd/312:35:023
1440864996620,41cyclictest1523545-21ssh11:48:380
144086499655,22cyclictest1440867-21latency_hist07:10:010
1440864996522,39cyclictest1514143-21ssh11:29:090
1440864996522,26cyclictest1473794-21ssh10:05:130
144086499652,15cyclictest1444477-21munin-run08:20:020
1440864996521,40cyclictest1478758-21rm10:15:260
1440864996521,28cyclictest1488764-21ssh10:36:150
1440864996517,2cyclictest0-21swapper/012:28:090
1440864996426,34cyclictest1511203-21sh11:22:570
1440864996424,23cyclictest1526322-21ssh11:54:320
144086499641,59cyclictest0-21swapper/012:15:010
144086499641,21cyclictest0-21swapper/012:30:240
144086499641,15cyclictest1443650-21sshd08:02:210
144086499632,14cyclictest1446514-21munin-run09:00:000
144086499631,57cyclictest1447265-21sed09:10:020
144086499622,15cyclictest1445757-21true08:45:010
144086499621,14cyclictest1442873-21sshd07:49:220
1440864996014,9cyclictest1532093-21ssh12:06:090
1440864996012,6cyclictest718586-1kworker/0:2H08:30:020
1440864996012,6cyclictest718586-1kworker/0:2H08:30:020
144087899596,11cyclictest37-21ksoftirqd/310:12:373
144087899592,9cyclictest37-21ksoftirqd/309:45:023
1440878995920,8cyclictest37-21ksoftirqd/310:48:383
144086499593,9cyclictest1445464-21modprobe08:40:000
1440864995926,29cyclictest1456414-21sh09:29:310
1440864995922,31cyclictest309-21in:imuxsock07:35:010
144086499591,21cyclictest3467602-21systemd-journal07:20:000
144087899588,8cyclictest37-21ksoftirqd/311:18:523
144087899586,9cyclictest37-21ksoftirqd/309:10:493
144087899584,13cyclictest37-21ksoftirqd/310:09:073
144087899584,12cyclictest37-21ksoftirqd/311:46:243
144086499582,14cyclictest1444729-21munin-run08:25:000
1440864995820,34cyclictest12-21ksoftirqd/008:00:010
144087899577,5cyclictest131rcu_preempt09:36:243
144087899577,5cyclictest131rcu_preempt09:36:233
144087899575,7cyclictest37-21ksoftirqd/309:59:183
144086499571,52cyclictest0-21swapper/007:42:210
144087899564,9cyclictest37-21ksoftirqd/312:25:293
1440873995646,6cyclictest1543497-21kworker/2:012:35:012
1440864995616,3cyclictest0-21swapper/008:06:130
144087899557,11cyclictest37-21ksoftirqd/311:07:393
144087899556,8cyclictest37-21ksoftirqd/311:50:123
144087899555,9cyclictest37-21ksoftirqd/311:00:333
144087899553,8cyclictest37-21ksoftirqd/309:51:533
1440878995513,11cyclictest37-21ksoftirqd/310:20:013
1440878995511,12cyclictest37-21ksoftirqd/309:00:013
144086499552,20cyclictest1443161-21grep07:55:010
1440864995515,5cyclictest1446176-21latency_hist08:50:020
144087899547,13cyclictest37-21ksoftirqd/310:34:083
144087899546,12cyclictest37-21ksoftirqd/309:22:193
144087899546,11cyclictest37-21ksoftirqd/308:30:033
144087899546,11cyclictest37-21ksoftirqd/308:30:023
144087899545,10cyclictest37-21ksoftirqd/311:29:593
1440878995432,16cyclictest275-21dbus-daemon08:45:013
1440878995419,10cyclictest37-21ksoftirqd/312:13:043
1440878995411,11cyclictest37-21ksoftirqd/308:25:023
1440878995411,10cyclictest37-21ksoftirqd/311:38:453
1440878995411,10cyclictest37-21ksoftirqd/311:38:443
1440873995445,6cyclictest1443367-21kworker/2:010:09:322
144087899537,10cyclictest37-21ksoftirqd/310:26:143
144087899535,6cyclictest37-21ksoftirqd/310:24:213
144087899535,11cyclictest37-21ksoftirqd/310:55:023
144087899533,15cyclictest37-21ksoftirqd/311:58:363
144087899532,13cyclictest37-21ksoftirqd/312:01:213
144087899532,13cyclictest37-21ksoftirqd/309:48:363
144087899531,9cyclictest1443836-21date08:05:033
1440873995345,5cyclictest1504898-21kworker/2:312:07:012
1440873995338,7cyclictest1443367-21kworker/2:010:35:422
1440873995334,15cyclictest1-21systemd07:20:002
1440869995333,6cyclictest131rcu_preempt08:40:031
144086499536,41cyclictest0-21swapper/008:55:010
144086499536,41cyclictest0-21swapper/008:55:000
144087899528,9cyclictest37-21ksoftirqd/312:20:013
144087899528,7cyclictest37-21ksoftirqd/310:36:373
144087899527,7cyclictest37-21ksoftirqd/310:05:013
144087899524,10cyclictest37-21ksoftirqd/311:00:003
1440878995218,10cyclictest37-21ksoftirqd/309:10:023
1440873995243,5cyclictest1504898-21kworker/2:312:12:442
144087899518,10cyclictest37-21ksoftirqd/311:30:123
144087899517,11cyclictest37-21ksoftirqd/310:43:433
144087899515,8cyclictest37-21ksoftirqd/311:22:403
144087899515,12cyclictest37-21ksoftirqd/312:23:183
144087899513,10cyclictest37-21ksoftirqd/311:15:003
144087899512,17cyclictest1456295-21ssh09:29:163
1440873995143,5cyclictest1443367-21kworker/2:011:04:352
1440873995142,6cyclictest1504898-21kworker/2:311:57:112
1440873995141,6cyclictest1443432-21kworker/2:310:45:132
1440873995141,6cyclictest1443432-21kworker/2:309:35:482
1440873995141,6cyclictest1443432-21kworker/2:309:35:482
144086999513,4cyclictest131rcu_preempt10:27:181
1440869995133,14cyclictest1-21systemd10:50:001
144086499512,14cyclictest1442366-21cron07:40:000
144087899508,12cyclictest37-21ksoftirqd/307:50:023
144087899505,9cyclictest37-21ksoftirqd/307:20:003
144087899505,7cyclictest37-21ksoftirqd/309:33:273
144087899504,8cyclictest37-21ksoftirqd/311:44:363
144087899501,11cyclictest1442105-21munin-run07:35:013
1440873995044,3cyclictest1500044-21kworker/2:111:22:112
1440873995042,5cyclictest1504898-21kworker/2:311:46:122
1440873995042,5cyclictest1500044-21kworker/2:112:23:372
1440873995042,5cyclictest1500044-21kworker/2:111:25:232
1440873995042,5cyclictest1443367-21kworker/2:010:40:072
1440873995039,6cyclictest1500044-21kworker/2:111:19:482
1440873995033,13cyclictest3467602-21systemd-journal08:05:002
1440873995033,13cyclictest1447036-21cron09:10:002
144087399502,29cyclictest1483299-21sh10:25:022
144086499501,45cyclictest0-21swapper/007:15:030
144086499501,45cyclictest0-21swapper/007:15:020
144087899496,12cyclictest37-21ksoftirqd/307:55:013
1440873994942,4cyclictest1504898-21kworker/2:311:53:482
1440873994941,5cyclictest1500044-21kworker/2:111:33:012
1440873994941,5cyclictest1443367-21kworker/2:009:50:592
1440873994941,5cyclictest1443367-21kworker/2:009:45:102
1440873994940,5cyclictest1443367-21kworker/2:010:56:082
1440873994934,11cyclictest3467602-21systemd-journal09:19:202
1440869994933,12cyclictest275-21dbus-daemon09:10:021
144087899481,8cyclictest1445478-21munin-run08:40:013
1440878994814,5cyclictest131rcu_preempt09:17:273
1440873994840,5cyclictest1443367-21kworker/2:010:10:522
1440873994839,6cyclictest1504898-21kworker/2:312:03:502
1440873994839,5cyclictest1443367-21kworker/2:010:55:002
1440873994838,6cyclictest1443367-21kworker/2:010:26:392
1440869994843,3cyclictest231ksoftirqd/110:25:011
144086499481,4cyclictest0-21swapper/009:03:020
144087899476,9cyclictest37-21ksoftirqd/312:05:493
144087899472,12cyclictest37-21ksoftirqd/307:25:013
144087899471,6cyclictest0-21swapper/308:20:033
1440873994740,5cyclictest1443367-21kworker/2:009:24:052
1440873994739,5cyclictest1504898-21kworker/2:312:28:562
1440873994739,5cyclictest1443432-21kworker/2:309:25:042
1440873994739,5cyclictest1443367-21kworker/2:008:12:582
1440869994731,12cyclictest1-21systemd09:35:011
1440869994725,17cyclictest1451773-21cat09:20:011
144087899461,5cyclictest0-21swapper/308:14:593
1440873994638,5cyclictest1443432-21kworker/2:310:00:002
1440873994638,5cyclictest1443367-21kworker/2:011:05:542
1440873994638,5cyclictest1443367-21kworker/2:009:41:252
1440873994638,5cyclictest1443367-21kworker/2:009:11:052
1440873994637,6cyclictest1504898-21kworker/2:311:37:592
1440873994637,6cyclictest1504898-21kworker/2:311:37:592
1440873994636,6cyclictest1500044-21kworker/2:111:13:312
1440873994633,9cyclictest1-21systemd08:35:012
144086999466,35cyclictest1447038-21diskmemload12:35:021
1440869994632,10cyclictest1-21systemd12:40:011
1440864994627,14cyclictest275-21dbus-daemon07:30:000
1440864994612,6cyclictest12-21ksoftirqd/008:12:550
1440873994537,5cyclictest1443367-21kworker/2:008:24:262
1440873994536,5cyclictest1443367-21kworker/2:010:18:082
1440873994535,6cyclictest30-21ksoftirqd/210:32:552
1440873994525,15cyclictest3467602-21systemd-journal07:25:032
144086999453,3cyclictest131rcu_preempt11:05:011
1440869994530,11cyclictest1-21systemd08:05:001
144086999452,9cyclictest0-21swapper/111:18:381
144087899444,13cyclictest37-21ksoftirqd/309:00:373
1440878994434,6cyclictest37-21ksoftirqd/308:35:013
144087899442,8cyclictest37-21ksoftirqd/308:51:223
144087899442,8cyclictest37-21ksoftirqd/308:51:223
144087899441,11cyclictest1442363-21/usr/sbin/munin07:40:023
144087399449,6cyclictest131rcu_preempt10:04:252
1440873994436,5cyclictest1500044-21kworker/2:112:16:332
1440873994419,19cyclictest275-21dbus-daemon08:50:012
1440869994432,9cyclictest23-21ksoftirqd/110:53:291
1440869994430,8cyclictest309-21in:imuxsock08:15:001
1440869994428,12cyclictest275-21dbus-daemon12:05:011
1440869994428,11cyclictest3467602-21systemd-journal09:00:001
1440869994426,14cyclictest275-21dbus-daemon09:05:001
1440869994420,5cyclictest23-21ksoftirqd/111:54:481
1440869994420,15cyclictest1461440-21cron09:40:001
1440869994420,15cyclictest1461440-21cron09:39:591
144086999441,38cyclictest23-21ksoftirqd/108:35:011
144086499441,4cyclictest0-21swapper/008:31:590
1440873994334,6cyclictest1443367-21kworker/2:009:31:542
1440873994333,7cyclictest1504898-21kworker/2:311:44:232
1440873994330,10cyclictest275-21dbus-daemon07:45:012
1440873994323,12cyclictest309-21in:imuxsock09:00:002
1440869994328,11cyclictest309-21in:imuxsock08:00:001
144086999432,3cyclictest131rcu_preempt12:09:591
1440869994320,6cyclictest23-21ksoftirqd/110:15:441
144087899424,7cyclictest37-21ksoftirqd/307:10:013
144087899421,16cyclictest37-21ksoftirqd/308:50:003
1440873994235,4cyclictest1504898-21kworker/2:312:39:272
1440873994224,13cyclictest3467602-21systemd-journal08:50:032
1440873994224,13cyclictest3467602-21systemd-journal08:50:022
144086999423,3cyclictest131rcu_preempt11:37:131
144086999423,3cyclictest131rcu_preempt11:37:121
1440869994230,9cyclictest1492076-21sshd10:43:211
1440869994228,10cyclictest23-21ksoftirqd/111:56:541
1440869994226,13cyclictest3467602-21systemd-journal10:00:011
1440869994215,6cyclictest23-21ksoftirqd/112:18:581
144087899414,9cyclictest37-21ksoftirqd/308:10:003
144087899414,7cyclictest37-21ksoftirqd/307:45:013
144087899413,8cyclictest0-21swapper/307:27:403
1440869994128,10cyclictest1473500-21systemctl10:05:001
1440869994126,10cyclictest309-21in:imuxsock09:45:021
1440869994123,13cyclictest275-21dbus-daemon07:20:001
1440869994121,15cyclictest275-21dbus-daemon08:20:011
144086999411,3cyclictest131rcu_preempt11:45:001
144087899401,8cyclictest37-21ksoftirqd/307:13:223
144087899401,8cyclictest37-21ksoftirqd/307:13:213
1440873994025,11cyclictest3467602-21systemd-journal08:30:012
1440873994025,11cyclictest3467602-21systemd-journal08:30:002
144086999403,8cyclictest1442361-21sendmail07:40:011
1440869994032,6cyclictest23-21ksoftirqd/111:32:171
1440869994028,6cyclictest1536173-21munin-plugin-st12:15:001
1440869994026,10cyclictest1-21systemd12:30:011
1440869994023,12cyclictest3467602-21systemd-journal08:55:011
1440869994023,12cyclictest3467602-21systemd-journal08:55:001
144086999401,4cyclictest131rcu_preempt11:06:291
144087899391,10cyclictest37-21ksoftirqd/308:00:003
1440873993924,12cyclictest275-21dbus-daemon07:25:002
144086999394,3cyclictest131rcu_preempt09:28:251
144086999393,3cyclictest131rcu_preempt11:28:241
1440869993929,7cyclictest23-21ksoftirqd/111:11:191
1440869993922,12cyclictest23-21ksoftirqd/107:55:011
1440869993916,6cyclictest23-21ksoftirqd/109:11:311
144086999391,5cyclictest131rcu_preempt10:40:021
1440869993910,6cyclictest131rcu_preempt08:25:021
1440873993810,4cyclictest131rcu_preempt07:47:372
1440869993827,9cyclictest1462677-21sshd09:42:201
1440869993826,10cyclictest23-21ksoftirqd/111:45:211
1440869993824,10cyclictest3467602-21systemd-journal08:50:001
144086999381,4cyclictest131rcu_preempt12:20:021
144086999381,3cyclictest131rcu_preempt10:12:271
1440873993725,9cyclictest1445763-21pluginstate08:45:012
1440869993725,10cyclictest23-21ksoftirqd/110:57:341
1440869993724,10cyclictest23-21ksoftirqd/107:10:011
144086999372,3cyclictest131rcu_preempt11:24:061
1440869993723,10cyclictest0-21swapper/107:30:021
1440873993623,10cyclictest30-21ksoftirqd/207:35:022
1440873993620,12cyclictest3467602-21systemd-journal08:20:002
1440869993627,7cyclictest23-21ksoftirqd/110:05:271
144086999362,5cyclictest131rcu_preempt10:32:141
1440869993625,8cyclictest1442612-21sshd07:42:201
1440869993624,9cyclictest23-21ksoftirqd/109:25:001
1440869993623,9cyclictest3467602-21systemd-journal08:32:221
1440869993622,10cyclictest3467602-21systemd-journal07:35:001
144086999361,3cyclictest131rcu_preempt09:50:551
1440873993527,5cyclictest1443367-21kworker/2:008:10:002
1440873993525,7cyclictest1443116-21sshd07:52:202
1440873993521,9cyclictest30-21ksoftirqd/208:40:022
1440873993520,12cyclictest30-21ksoftirqd/208:00:022
144087399341,3cyclictest131rcu_preempt09:05:012
1440869993426,5cyclictest1441056-21latency_hist07:15:021
1440869993426,5cyclictest1441056-21latency_hist07:15:011
1440869993421,9cyclictest1441577-21sshd07:22:221
1440869993318,10cyclictest309-21in:imuxsock08:05:211
144087399323,5cyclictest131rcu_preempt07:15:012
144087399323,5cyclictest131rcu_preempt07:15:012
1440873993220,8cyclictest30-21ksoftirqd/207:40:012
1440869993124,4cyclictest1443055-21latency_hist07:50:021
1440869993121,8cyclictest23-21ksoftirqd/108:27:551
1440869993121,8cyclictest23-21ksoftirqd/108:27:541
144087399261,3cyclictest131rcu_preempt07:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional