You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-06 - 21:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sun Sep 06, 2026 12:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13090149911388,23cyclictest0-21swapper/008:10:010
1309014991041,3cyclictest131rcu_preempt11:05:020
1309021998027,14cyclictest1403170-21sshd12:20:072
130901899795,57cyclictest1409892-21ssh12:35:021
1309021997829,18cyclictest1324938-21ssh09:30:432
1309021997727,16cyclictest1387100-21rm11:45:332
1309021997725,19cyclictest364-21snmpd11:39:582
1309021997724,15cyclictest1318522-21ssh09:17:022
1309021997723,19cyclictest1334755-21ssh09:52:092
1309021997723,19cyclictest1334755-21ssh09:52:082
1309021997531,14cyclictest1320052-21sh09:20:062
1309021997530,13cyclictest1371731-21ssh11:12:142
1309021997524,18cyclictest1376961-21ssh11:23:332
1309021997524,17cyclictest1411565-21ssh12:38:302
1309014997513,42cyclictest12-21ksoftirqd/009:15:400
1309021997424,15cyclictest1332591-21ssh09:47:402
1309021997422,18cyclictest1399128-21ssh12:11:342
1309021997327,14cyclictest1330379-21ssh09:42:352
1309021997325,16cyclictest1353228-21ssh10:32:102
1309021997322,16cyclictest1352070-21ssh10:29:552
1309021997222,13cyclictest1367718-21ssh11:03:492
1309024997162,7cyclictest0-21swapper/307:45:013
1309021997127,13cyclictest0-21swapper/211:54:352
1309021997127,11cyclictest1385803-21ssh11:42:452
130901499715,45cyclictest12-21ksoftirqd/010:12:160
1309021997026,13cyclictest1369035-21ssh11:06:212
1309021997024,16cyclictest1409241-21ssh12:33:302
1309014997064,3cyclictest0-21swapper/011:10:020
1309014997036,30cyclictest12-21ksoftirqd/009:43:220
1309021996926,14cyclictest1337223-21ssh09:57:432
1309021996923,14cyclictest1315792-21ssh09:10:482
1309014996965,2cyclictest0-21swapper/007:55:010
1309021996855,10cyclictest1401693-21ssh12:17:142
1309021996830,13cyclictest1391651-21ssh11:55:192
1309021996823,17cyclictest1348056-21ssh10:20:362
1309021996725,13cyclictest1323877-21diskmemload09:28:372
1309021996718,16cyclictest1315176-21diskmemload11:29:122
1309021996646,16cyclictest1347159-21ssh10:19:052
1309021996620,14cyclictest1379481-21kworker/u8:012:01:512
130901499661,18cyclictest1396078-21latency_hist12:05:020
130902499652,9cyclictest37-21ksoftirqd/311:25:013
1309021996553,9cyclictest1327657-21ssh09:36:342
1309021996551,11cyclictest1374990-21ssh11:19:292
1309021996546,15cyclictest1355787-21ssh10:38:022
1309021996525,12cyclictest1341152-21ssh10:05:512
1309021996518,13cyclictest1360257-21ssh10:47:292
1309014996513,33cyclictest131rcu_preempt09:12:510
1309021996450,11cyclictest1380690-21ssh11:31:292
1309021996450,11cyclictest1380690-21ssh11:31:292
1309021996446,14cyclictest1357909-21ssh10:42:362
1309021996412,13cyclictest1338844-21ssh10:00:592
1309021996412,13cyclictest1338844-21ssh10:00:582
130901499642,5cyclictest131rcu_preempt11:15:120
130902499637,11cyclictest37-21ksoftirqd/312:20:033
1309021996347,13cyclictest1406387-21ssh12:27:102
1309021996346,14cyclictest1397988-21ssh12:09:292
1309021996346,14cyclictest1397988-21ssh12:09:282
1309021996346,14cyclictest1311243-21sshd07:51:082
1309021996346,13cyclictest1344083-21ssh10:12:252
1309021996244,15cyclictest1361679-21ssh10:50:292
1309021996227,11cyclictest0-21swapper/208:32:002
130901499624,5cyclictest131rcu_preempt12:14:480
1309021996146,12cyclictest1365169-21ssh10:58:232
1309021996144,14cyclictest1312548-21sshd08:18:072
1309021996044,13cyclictest1314899-21sshd09:02:082
1309021996044,13cyclictest1312819-21sshd08:21:082
1309014996015,36cyclictest12-21ksoftirqd/011:13:100
1309021995941,14cyclictest364-21snmpd07:27:452
1309021995940,14cyclictest1309163-21sshd07:11:062
130901499599,42cyclictest12-21ksoftirqd/012:31:380
1309014995824,17cyclictest1407793-21ssh12:30:030
130901499581,37cyclictest131rcu_preempt10:19:430
130902499576,8cyclictest37-21ksoftirqd/309:33:273
1309024995712,4cyclictest131rcu_preempt11:35:143
1309021995743,11cyclictest1314360-21sshd08:51:072
1309021995734,19cyclictest1313872-21sshd08:44:062
1309021995734,18cyclictest3467602-21systemd-journal08:05:002
1309014995747,7cyclictest1391486-21latency_hist11:55:020
1309014995730,12cyclictest1381174-21ssh11:32:450
1309014995730,12cyclictest1381174-21ssh11:32:440
1309014995729,12cyclictest3467602-21systemd-journal10:46:300
1309014995720,9cyclictest12-21ksoftirqd/011:37:360
1309024995636,15cyclictest3467602-21systemd-journal09:20:013
130902499561,14cyclictest37-21ksoftirqd/310:45:013
1309024995610,5cyclictest131rcu_preempt09:56:473
1309021995644,9cyclictest1310750-21sshd07:43:072
1309021995641,11cyclictest364-21snmpd09:05:482
1309021995634,17cyclictest1310214-21sshd07:32:072
1309021995632,19cyclictest1313656-21latency_hist08:40:022
130901899567,35cyclictest364-21snmpd12:00:021
1309018995636,14cyclictest275-21dbus-daemon09:55:011
1309018995636,14cyclictest275-21dbus-daemon09:55:001
1309014995627,14cyclictest1404844-21ssh12:24:020
130901499561,39cyclictest1328599-21ssh09:38:570
130902499557,11cyclictest37-21ksoftirqd/309:41:223
1309024995546,6cyclictest361rcuc/312:28:073
1309021995543,9cyclictest1311500-21sshd07:57:072
1309021995537,14cyclictest364-21snmpd07:18:092
1309021995533,18cyclictest1314787-21sed09:00:012
1309018995535,15cyclictest1-21systemd12:20:011
130902499541,11cyclictest37-21ksoftirqd/310:13:073
1309021995446,4cyclictest1311024-21munin-run07:50:002
1309021995446,4cyclictest1311024-21munin-run07:50:002
1309021995445,5cyclictest1313073-21sshd08:26:062
130901499543,31cyclictest19350irq/34-mmc110:05:030
130901499543,31cyclictest19350irq/34-mmc110:05:020
130901499542,4cyclictest131rcu_preempt10:56:510
1309014995424,13cyclictest1362544-21ssh10:52:300
130901499541,2cyclictest0-21swapper/009:55:030
130902499534,11cyclictest37-21ksoftirqd/311:16:543
130902499533,7cyclictest37-21ksoftirqd/310:35:033
130902499533,10cyclictest37-21ksoftirqd/310:40:023
130902499532,13cyclictest37-21ksoftirqd/309:14:593
1309024995312,5cyclictest131rcu_preempt12:36:363
1309024995311,6cyclictest131rcu_preempt11:47:113
1309021995336,13cyclictest364-21snmpd07:24:372
1309021995336,13cyclictest364-21snmpd07:24:372
130901499533,33cyclictest1367235-21kworker/u8:111:30:020
1309014995332,15cyclictest1-21systemd12:20:010
1309014995325,12cyclictest1324263-21ssh09:29:360
1309014995311,38cyclictest12-21ksoftirqd/009:47:080
130902499526,13cyclictest37-21ksoftirqd/308:00:013
130902499524,12cyclictest37-21ksoftirqd/310:25:013
130902499521,6cyclictest1309197-21modprobe07:15:003
130902499521,10cyclictest37-21ksoftirqd/308:45:013
1309021995245,4cyclictest1310469-21sshd07:38:072
1309021995239,10cyclictest364-21snmpd08:11:042
1309014995224,13cyclictest1386202-21ssh11:43:470
1309014995211,37cyclictest131rcu_preempt12:07:590
1309014995211,37cyclictest131rcu_preempt12:07:590
130902499519,7cyclictest131rcu_preempt09:40:033
130902499517,9cyclictest37-21ksoftirqd/308:10:083
130902499515,13cyclictest37-21ksoftirqd/308:35:003
1309024995113,9cyclictest37-21ksoftirqd/309:24:173
1309024995111,5cyclictest131rcu_preempt11:58:133
1309021995131,15cyclictest1314104-21sshd08:45:072
1309018995135,11cyclictest309-21in:imuxsock08:20:031
1309014995143,4cyclictest16950irq/224-800000010:39:590
130901499511,41cyclictest13575151ssh10:41:340
130902499508,5cyclictest131rcu_preempt11:13:463
130902499507,7cyclictest37-21ksoftirqd/312:25:023
130902499507,4cyclictest131rcu_preempt10:29:193
130902499506,9cyclictest37-21ksoftirqd/311:03:103
1309024995016,5cyclictest131rcu_preempt10:56:113
1309024995010,9cyclictest37-21ksoftirqd/312:30:333
1309021995037,9cyclictest364-21snmpd08:09:502
130901499508,34cyclictest12-21ksoftirqd/012:39:360
1309014995041,6cyclictest12-21ksoftirqd/011:46:060
1309014995032,13cyclictest275-21dbus-daemon07:30:000
1309014995022,13cyclictest1391784-21ssh11:55:450
130901499502,16cyclictest1313615-21systemctl08:40:010
130902499497,11cyclictest37-21ksoftirqd/308:50:023
130902499494,8cyclictest37-21ksoftirqd/308:20:003
1309024994938,8cyclictest361rcuc/311:42:393
1309024994933,11cyclictest1347512-21cron10:20:003
130902499493,12cyclictest37-21ksoftirqd/309:46:563
1309024994914,5cyclictest131rcu_preempt11:33:153
1309024994914,5cyclictest131rcu_preempt11:33:153
1309024994913,6cyclictest131rcu_preempt09:27:333
1309024994912,6cyclictest131rcu_preempt12:10:013
1309024994912,6cyclictest131rcu_preempt12:10:013
1309018994941,5cyclictest23-21ksoftirqd/109:34:071
1309014994928,17cyclictest1349864-21ssh10:25:010
1309014994916,29cyclictest1376203-21ssh11:21:450
130901499491,33cyclictest1334826-21ssh09:52:280
130901499491,33cyclictest1334826-21ssh09:52:270
130902499486,7cyclictest37-21ksoftirqd/310:50:023
130902499486,6cyclictest37-21ksoftirqd/309:52:553
130902499486,6cyclictest37-21ksoftirqd/309:52:553
130902499485,9cyclictest37-21ksoftirqd/311:05:273
130902499485,12cyclictest37-21ksoftirqd/307:40:003
1309024994831,13cyclictest1-21systemd08:30:023
130902499482,10cyclictest37-21ksoftirqd/312:12:283
1309024994815,5cyclictest131rcu_preempt12:04:573
1309024994815,5cyclictest131rcu_preempt11:55:003
1309024994811,5cyclictest131rcu_preempt11:29:153
1309018994838,8cyclictest23-21ksoftirqd/110:50:341
1309018994829,14cyclictest3467602-21systemd-journal11:15:001
130901899481,4cyclictest131rcu_preempt10:35:321
130901499488,5cyclictest131rcu_preempt10:08:440
130901499486,26cyclictest19350irq/34-mmc108:05:020
130901499483,3cyclictest131rcu_preempt10:34:440
130902499475,9cyclictest37-21ksoftirqd/309:10:023
130902499473,12cyclictest37-21ksoftirqd/310:52:293
1309024994719,6cyclictest131rcu_preempt10:06:223
1309018994713,5cyclictest131rcu_preempt12:05:021
130901499473,7cyclictest1326774-21modprobe09:35:000
1309014994737,7cyclictest12-21ksoftirqd/008:50:010
1309014994726,15cyclictest275-21dbus-daemon08:45:000
1309024994615,5cyclictest131rcu_preempt10:02:563
1309024994615,5cyclictest131rcu_preempt10:02:563
1309024994612,5cyclictest131rcu_preempt07:45:073
1309024994612,5cyclictest131rcu_preempt07:45:063
1309018994632,11cyclictest23-21ksoftirqd/111:40:021
1309024994533,7cyclictest37-21ksoftirqd/307:20:013
130902499451,6cyclictest0-21swapper/308:55:023
130901899454,3cyclictest131rcu_preempt09:28:221
1309018994532,10cyclictest1311507-21sshd07:59:071
130901899452,3cyclictest131rcu_preempt11:21:011
130901899452,3cyclictest131rcu_preempt09:44:351
1309014994538,4cyclictest16950irq/224-800000009:21:410
1309014994527,12cyclictest275-21dbus-daemon08:55:020
130901499452,6cyclictest3467602-21systemd-journal07:40:000
130901499452,39cyclictest1351221-21ssh10:27:520
1309014994437,4cyclictest16950irq/224-800000007:15:020
130901499441,39cyclictest0-21swapper/007:44:400
1309024994325,13cyclictest275-21dbus-daemon08:40:013
1309024994315,5cyclictest131rcu_preempt09:00:013
1309018994329,12cyclictest23-21ksoftirqd/112:38:471
130901899432,7cyclictest0-21swapper/109:10:291
130901899432,3cyclictest131rcu_preempt11:07:241
130901899431,21cyclictest0-21swapper/111:48:511
1309014994336,4cyclictest16950irq/224-800000009:10:000
1309014994327,12cyclictest12-21ksoftirqd/007:15:020
130901499432,36cyclictest1312325-21unin-run08:15:010
130902499425,7cyclictest37-21ksoftirqd/307:10:023
130902499425,7cyclictest37-21ksoftirqd/307:10:013
1309024994228,10cyclictest1-21systemd08:25:003
130902499421,8cyclictest150-21pr/ttyS207:21:483
130902499421,8cyclictest150-21pr/ttyS207:21:473
130902499421,6cyclictest150-21pr/ttyS207:29:503
1309018994228,11cyclictest23-21ksoftirqd/110:20:361
1309018994228,11cyclictest1379447-21sshd11:29:071
1309018994223,13cyclictest3467602-21systemd-journal08:09:591
1309014994226,12cyclictest1-21systemd07:50:020
1309014994226,12cyclictest1-21systemd07:50:010
1309024994132,6cyclictest37-21ksoftirqd/307:35:003
130902499413,10cyclictest37-21ksoftirqd/309:05:023
1309024994112,5cyclictest131rcu_preempt08:10:013
130901899418,7cyclictest131rcu_preempt09:23:091
1309018994131,7cyclictest1338906-21sshd10:01:081
1309018994131,7cyclictest1338906-21sshd10:01:071
1309018994115,6cyclictest131rcu_preempt09:36:241
1309014994131,6cyclictest1312852-21modprobe08:25:010
130901899404,3cyclictest131rcu_preempt10:13:241
1309018994029,8cyclictest23-21ksoftirqd/109:45:341
1309018994027,8cyclictest1313593-21cron08:40:001
1309018994027,11cyclictest23-21ksoftirqd/110:45:451
1309018994027,10cyclictest23-21ksoftirqd/110:28:461
1309018994027,10cyclictest23-21ksoftirqd/109:57:541
1309018994027,10cyclictest23-21ksoftirqd/107:38:071
130901899402,27cyclictest221rcuc/107:45:011
130902499391,4cyclictest0-21swapper/307:55:013
1309021993928,8cyclictest0-21swapper/207:10:022
1309021993928,8cyclictest0-21swapper/207:10:012
1309018993927,10cyclictest23-21ksoftirqd/110:58:341
1309018993924,12cyclictest275-21dbus-daemon12:25:481
1309018993924,12cyclictest1311789-21sshd08:04:071
1309014993918,5cyclictest131rcu_preempt07:10:010
1309014993918,5cyclictest131rcu_preempt07:10:010
130902499381,6cyclictest150-21pr/ttyS208:00:313
1309018993828,7cyclictest23-21ksoftirqd/111:30:301
1309018993828,7cyclictest23-21ksoftirqd/111:30:291
1309018993826,9cyclictest1359140-21sed10:45:021
1309018993819,15cyclictest275-21dbus-daemon12:10:011
1309018993819,15cyclictest275-21dbus-daemon12:10:001
1309018993811,5cyclictest131rcu_preempt12:25:021
1309014993830,4cyclictest16950irq/224-800000008:20:000
1309014993824,11cyclictest12-21ksoftirqd/008:55:170
130901499382,31cyclictest0-21swapper/007:25:020
130901499382,31cyclictest0-21swapper/007:25:010
1309018993728,6cyclictest1314645-21sendmail09:00:001
1309018993727,7cyclictest1309703-21sshd07:20:071
1309018993727,7cyclictest1309703-21sshd07:20:071
130901899372,5cyclictest131rcu_preempt09:17:001
1309018993725,9cyclictest23-21ksoftirqd/110:18:341
1309018993725,8cyclictest23-21ksoftirqd/112:12:501
1309018993724,11cyclictest23-21ksoftirqd/110:34:251
130901499372,4cyclictest0-21swapper/008:00:020
130901499371,32cyclictest1313370-21unin-run08:35:000
130901899369,7cyclictest23-21ksoftirqd/107:20:021
1309018993628,5cyclictest1315317-21sh09:10:031
1309018993627,7cyclictest23-21ksoftirqd/111:52:171
130901899362,5cyclictest1309078-21latency_hist07:10:021
130901899362,5cyclictest1309078-21latency_hist07:10:011
130901899361,3cyclictest131rcu_preempt11:16:361
1309014993627,6cyclictest1313268-21latency_hist08:30:020
130901499361,6cyclictest131rcu_preempt07:35:000
130901899356,7cyclictest23-21ksoftirqd/110:06:471
130901899355,3cyclictest131rcu_preempt11:05:031
1309018993526,7cyclictest23-21ksoftirqd/111:42:261
1309018993524,9cyclictest23-21ksoftirqd/107:30:031
1309018993522,10cyclictest275-21dbus-daemon07:55:001
1309014993511,4cyclictest2383687-1kworker/0:2H09:01:080
130901899348,5cyclictest131rcu_preempt08:50:261
1309018993426,5cyclictest1312457-21latency_hist08:15:021
1309018993425,6cyclictest23-21ksoftirqd/107:50:021
1309018993425,6cyclictest23-21ksoftirqd/107:50:011
1309018993425,5cyclictest23-21ksoftirqd/107:14:381
1309018993418,11cyclictest275-21dbus-daemon09:05:001
1309018993325,5cyclictest1310344-21latency_hist07:35:021
1309018993318,11cyclictest3467602-21systemd-journal08:20:011
1309018993223,6cyclictest1313343-21cron08:35:001
1309018993221,8cyclictest23-21ksoftirqd/108:45:021
1309018993116,10cyclictest309-21in:imuxsock08:25:031
130901899303,4cyclictest131rcu_preempt08:42:081
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional