You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-20 - 23:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Apr 20, 2026 12:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1302046999476,12cyclictest1302048-21cat07:10:020
1302051998024,48cyclictest1-21systemd10:15:011
130205899793,41cyclictest1333019-21ssh10:07:223
1302051997334,34cyclictest1-21systemd08:10:011
130205899711,66cyclictest48-21kcompactd010:27:133
1302058996938,28cyclictest3467602-21systemd-journal10:10:423
130205599694,60cyclictest1312724-21ssh09:20:042
1302051996919,46cyclictest23-21ksoftirqd/111:19:081
1302046996929,36cyclictest1-21systemd09:25:010
1302046996929,36cyclictest1-21systemd09:25:000
1302058996833,31cyclictest1392816-21sh12:25:543
1302058996833,31cyclictest1392816-21sh12:25:543
130205899671,62cyclictest1380164-21ssh11:56:423
130205199673,18cyclictest275-21dbus-daemon11:05:001
1302051996721,41cyclictest23-21ksoftirqd/111:41:131
1302046996719,33cyclictest12-21ksoftirqd/009:58:360
130205899661,61cyclictest1356145-21ssh11:00:393
130205199664,12cyclictest284-21systemd-logind07:55:011
130205899652,58cyclictest1313466-21ssh09:21:503
130205899652,58cyclictest1313466-21ssh09:21:503
130205899651,60cyclictest1391067-21ssh12:21:583
130205899651,60cyclictest1387204-21ssh12:13:073
130205899651,60cyclictest1378692-21ssh11:53:223
130205899651,60cyclictest1371214-21ssh11:35:463
130205899651,60cyclictest1361955-21ssh11:14:453
130205899651,60cyclictest1347588-21ssh10:40:593
130205899651,60cyclictest1347588-21ssh10:40:583
130205899651,60cyclictest1325043-21ssh09:49:033
1302051996526,35cyclictest1379035-21sh11:54:191
1302051996526,34cyclictest1366388-21sh11:24:541
130205199652,13cyclictest1336873-21ssh10:16:151
130205199651,14cyclictest1368463-21ssh11:29:481
130205199651,13cyclictest1354589-21ssh10:57:241
1302058996433,28cyclictest1-21systemd10:20:003
130205899642,58cyclictest1385010-21ssh12:08:043
130205899642,56cyclictest1302765-21sshd07:24:523
130205899641,59cyclictest1397241-21ssh12:36:193
130205899641,59cyclictest1397241-21ssh12:36:183
130205899641,59cyclictest1390025-21ssh12:19:533
130205899641,59cyclictest1383581-21ssh12:04:523
130205899641,59cyclictest1376925-21ssh11:49:203
130205899641,59cyclictest1369337-21ssh11:31:273
130205899641,59cyclictest1367959-21ssh11:28:253
130205899641,59cyclictest1358930-21ssh11:07:273
130205899641,59cyclictest1345872-21ssh10:37:133
130205899641,59cyclictest1343804-21ssh10:32:213
130205899641,59cyclictest1339305-21ssh10:21:553
130205899641,59cyclictest1331849-21ssh10:05:003
130205899641,59cyclictest1331849-21ssh10:05:003
130205899641,59cyclictest1322268-21ssh09:42:183
130205899641,59cyclictest1317994-21ssh09:32:293
130205899641,59cyclictest1312212-21ssh09:19:133
1302055996454,8cyclictest30-21ksoftirqd/208:15:022
1302051996414,45cyclictest284-21systemd-logind07:35:001
1302046996435,20cyclictest12-21ksoftirqd/010:35:030
1302046996426,34cyclictest1349175-21latency_hist10:45:030
1302046996426,34cyclictest1349175-21latency_hist10:45:020
130205899632,57cyclictest1365093-21ssh11:21:323
130205899632,57cyclictest1350516-21ssh10:48:013
130205899631,58cyclictest1395394-21ssh12:32:073
130205899631,58cyclictest1363567-21ssh11:18:183
130205899631,58cyclictest1352363-21sshd10:52:133
130205899631,58cyclictest1352363-21sshd10:52:123
130205899631,58cyclictest1329004-21ssh09:58:123
130205899631,58cyclictest1326884-21ssh09:53:123
130205899631,58cyclictest1320271-21ssh09:37:513
130205899631,58cyclictest1316082-21ssh09:28:093
130205899631,58cyclictest1308894-21ssh09:11:143
130205199635,13cyclictest1306919-21modprobe08:44:591
1302051996323,36cyclictest23-21ksoftirqd/109:56:331
1302051996323,36cyclictest1348296-21sh10:42:481
1302051996323,36cyclictest1348296-21sh10:42:471
1302051996314,44cyclictest311-21rs:main1
130205199631,13cyclictest364-21snmpd12:03:541
1302046996326,11cyclictest275-21dbus-daemon08:35:000
1302046996312,47cyclictest1385938-21ssh12:10:030
1302058996221,37cyclictest1304659-21latency_hist08:00:013
130205899621,57cyclictest1374535-21ssh11:43:403
130205899621,57cyclictest1354291-21ssh10:56:323
130205899621,57cyclictest1306111-21sshd08:25:133
130205899621,57cyclictest1305075-21sshd08:07:113
1302055996213,7cyclictest30-21ksoftirqd/211:25:002
130205199625,11cyclictest1306033-21latency_hist08:25:031
130205199623,8cyclictest1377195-21cron11:50:001
1302051996215,43cyclictest1353520-21munin-run10:55:021
1302051996215,43cyclictest1353520-21munin-run10:55:011
1302046996223,6cyclictest0-21swapper/010:45:050
130205899611,56cyclictest1305574-21sshd08:16:133
130205899611,56cyclictest1304825-21sshd08:02:133
130205899611,54cyclictest48-21kcompactd008:36:313
1302051996144,13cyclictest1-21systemd11:35:001
1302051996121,36cyclictest23-21ksoftirqd/109:22:431
1302051996121,36cyclictest23-21ksoftirqd/109:22:421
1302051996114,5cyclictest1304944-21latency_hist08:05:021
1302046996124,32cyclictest16850irq/134-800000012:15:360
130205899603,53cyclictest1308183-21sshd09:06:113
130205899602,53cyclictest1307409-21sshd08:53:113
130205899601,54cyclictest48-21kcompactd008:23:463
130205199603,47cyclictest1302503-21sh07:20:001
1302051996024,32cyclictest23-21ksoftirqd/110:50:011
130205199602,12cyclictest1396532-21munin-run12:35:001
130205199602,12cyclictest1345659-21ssh10:36:411
130205199601,14cyclictest1331340-21sh10:03:371
130205199601,14cyclictest1331340-21sh10:03:371
130205199601,13cyclictest48-21kcompactd011:55:171
130205199601,12cyclictest1312063-21ssh09:18:531
130205899591,54cyclictest1303512-21sshd07:38:133
130205899591,54cyclictest1303265-21sshd07:34:113
130205599596,9cyclictest30-21ksoftirqd/210:25:012
1302055995949,7cyclictest291rcuc/209:37:522
130205199593,12cyclictest1394384-21modprobe12:30:011
130205199593,12cyclictest1394384-21modprobe12:30:001
130205199592,16cyclictest1398368-21ssh12:39:101
130205199592,16cyclictest1398368-21ssh12:39:091
130205199592,14cyclictest1333865-21sh10:09:331
130205199592,12cyclictest364-21snmpd11:35:541
130205199592,12cyclictest1302795-21munin-run07:25:011
1302051995919,36cyclictest1384310-21sh12:06:141
1302051995916,39cyclictest23-21ksoftirqd/109:36:411
1302046995919,8cyclictest12-21ksoftirqd/012:32:010
130205899582,52cyclictest364-21snmpd07:53:393
130205899581,53cyclictest1307928-21sshd09:00:243
130205199583,9cyclictest1303017-21sshd07:29:541
130205199583,16cyclictest1327130-21sh09:53:551
130205199583,14cyclictest309-21in:imuxsock08:20:011
130205199583,12cyclictest284-21systemd-logind08:00:001
130205199583,10cyclictest1307431-21munin-plugin-st08:55:001
1302051995825,29cyclictest23-21ksoftirqd/109:34:431
130205199582,17cyclictest1344069-21ssh10:33:111
130205199582,13cyclictest1316604-21ssh09:29:241
1302051995819,34cyclictest1324975-21sh09:48:531
1302046995829,14cyclictest1353745-21ssh10:55:030
1302046995822,7cyclictest1327778-21date09:55:020
1302046995814,10cyclictest12-21ksoftirqd/011:09:250
1302055995713,10cyclictest30-21ksoftirqd/211:50:012
130205199572,16cyclictest1388029-21wc12:15:021
130205199572,15cyclictest1361516-21ssh11:13:351
130205199572,15cyclictest1342775-21latency_hist10:30:021
130205199572,13cyclictest1339873-21sh10:23:161
130205199572,11cyclictest364-21snmpd08:34:331
130205199572,10cyclictest1390090-21systemctl12:20:001
1302051995720,34cyclictest23-21ksoftirqd/111:08:391
1302051995720,33cyclictest1308215-21diskmemload09:10:071
130205199571,47cyclictest1306638-21modprobe08:40:001
1302046995720,6cyclictest16850irq/134-800000012:00:000
1302046995720,32cyclictest16850irq/134-800000011:49:200
1302046995719,34cyclictest16850irq/134-800000010:25:310
1302046995715,7cyclictest12-21ksoftirqd/012:36:340
1302046995715,7cyclictest12-21ksoftirqd/012:36:330
130205899562,50cyclictest1305318-21sshd08:10:123
130205899561,50cyclictest48-21kcompactd007:43:493
130205899561,50cyclictest48-21kcompactd007:43:493
130205899561,50cyclictest1304084-21/usr/sbin/munin07:50:013
130205899561,50cyclictest1302457-21sshd07:15:123
1302051995617,34cyclictest23-21ksoftirqd/109:10:011
1302046995629,12cyclictest1363882-21ssh11:19:050
1302046995628,13cyclictest1366033-21ssh11:24:030
130205899551,49cyclictest1306396-21munin-run08:35:013
130205599555,8cyclictest30-21ksoftirqd/209:25:012
130205599555,8cyclictest30-21ksoftirqd/209:25:002
1302055995547,6cyclictest291rcuc/209:13:242
130205199553,13cyclictest1-21systemd07:40:001
130205199551,14cyclictest1322746-21ssh09:43:371
1302046995528,12cyclictest1391761-21ssh12:23:440
1302046995528,12cyclictest1310357-21ssh09:14:580
1302046995526,14cyclictest1373440-21ssh11:40:460
1302046995526,14cyclictest1345919-21ssh10:37:190
130204699552,36cyclictest1317800-21ssh09:32:010
1302046995518,19cyclictest16850irq/134-800000011:39:530
130205899541,49cyclictest48-21kcompactd007:29:243
130205899541,48cyclictest364-21snmpd07:10:453
130205899541,48cyclictest1307142-21sshd08:45:123
130205599549,10cyclictest30-21ksoftirqd/210:55:072
1302055995428,8cyclictest30-21ksoftirqd/211:42:312
1302055995415,10cyclictest30-21ksoftirqd/210:42:102
1302055995415,10cyclictest30-21ksoftirqd/210:42:102
130205199544,44cyclictest1305457-21sort08:15:011
130205199543,44cyclictest1302236-21modprobe07:15:011
130205199543,14cyclictest1304145-21latency_hist07:50:011
130205199542,14cyclictest1390610-21ssh12:20:431
1302046995429,10cyclictest1322174-21ssh09:42:050
1302046995420,30cyclictest16850irq/134-800000009:29:130
1302046995419,18cyclictest16850irq/134-800000009:45:160
1302046995419,17cyclictest16850irq/134-800000012:26:220
1302046995419,17cyclictest16850irq/134-800000012:26:210
1302046995419,17cyclictest16850irq/134-800000010:50:100
1302046995419,17cyclictest16850irq/134-800000010:50:090
1302046995418,6cyclictest16850irq/134-800000010:11:380
1302046995418,19cyclictest16850irq/134-800000011:00:480
1302046995418,18cyclictest16850irq/134-800000012:11:460
130205899531,47cyclictest1307653-21sshd08:56:123
1302046995320,17cyclictest16850irq/134-800000010:03:540
1302046995320,17cyclictest16850irq/134-800000010:03:540
1302046995318,17cyclictest16850irq/134-800000012:01:060
1302046995317,6cyclictest16850irq/134-800000011:14:150
1302046995317,18cyclictest16850irq/134-800000009:18:290
130205899522,45cyclictest364-21snmpd08:43:183
130205599525,8cyclictest30-21ksoftirqd/210:13:492
130205599524,7cyclictest30-21ksoftirqd/210:46:172
130205599522,6cyclictest30-21ksoftirqd/212:18:082
130205199524,6cyclictest0-21swapper/107:10:011
130205199523,8cyclictest1307166-21cron08:50:011
130205199522,12cyclictest1303824-21true07:45:011
130205199522,12cyclictest1303824-21true07:45:001
1302046995232,17cyclictest16850irq/134-800000007:54:590
1302046995223,14cyclictest1332568-21ssh10:06:110
1302046995218,17cyclictest16850irq/134-800000011:54:300
1302046995218,17cyclictest16850irq/134-800000011:26:110
1302046995216,6cyclictest16850irq/134-800000011:33:390
1302055995142,6cyclictest291rcuc/210:18:462
130205199513,8cyclictest1306167-21wc08:30:001
1302046995146,2cyclictest0-21swapper/008:05:010
1302046995116,6cyclictest16850irq/134-800000010:19:020
1302046995116,6cyclictest16850irq/134-800000009:38:060
130205599508,8cyclictest30-21ksoftirqd/209:59:312
130205599508,5cyclictest131rcu_preempt09:40:112
130205599507,7cyclictest30-21ksoftirqd/209:54:382
130205599504,8cyclictest30-21ksoftirqd/210:27:512
1302055995041,7cyclictest30-21ksoftirqd/211:57:052
1302055995040,7cyclictest30-21ksoftirqd/212:12:492
130205199502,11cyclictest1308020-21latency_hist09:05:021
1302046995015,18cyclictest16850irq/134-800000010:22:180
1302046995011,35cyclictest1-21systemd09:10:020
130205599496,8cyclictest30-21ksoftirqd/210:31:452
1302055994940,6cyclictest291rcuc/211:38:282
130204699492,40cyclictest1305354-21munin-plugin-st08:15:000
130205599487,7cyclictest30-21ksoftirqd/212:00:442
130205599486,7cyclictest30-21ksoftirqd/212:25:492
130205599486,7cyclictest30-21ksoftirqd/212:25:492
130205599484,8cyclictest30-21ksoftirqd/209:33:172
130205599484,11cyclictest30-21ksoftirqd/211:29:542
1302055994840,5cyclictest1305397-21kworker/2:111:02:122
1302055994840,5cyclictest1305397-21kworker/2:108:55:442
1302055994840,5cyclictest1301426-21kworker/2:307:16:512
1302055994840,5cyclictest1294733-21kworker/2:207:10:082
130205599483,8cyclictest30-21ksoftirqd/210:05:212
1302055994813,6cyclictest131rcu_preempt10:35:432
1302055994810,7cyclictest30-21ksoftirqd/211:30:282
130205599475,9cyclictest30-21ksoftirqd/212:06:162
130205599475,7cyclictest30-21ksoftirqd/210:51:012
130205599475,7cyclictest30-21ksoftirqd/210:51:012
130205599474,9cyclictest30-21ksoftirqd/212:23:202
1302055994739,5cyclictest1303588-21kworker/2:008:53:292
1302055994739,5cyclictest1303588-21kworker/2:008:49:362
1302055994739,5cyclictest1303588-21kworker/2:007:59:272
1302055994738,6cyclictest291rcuc/211:12:362
1302055994738,5cyclictest1303588-21kworker/2:008:34:252
1302055994732,11cyclictest275-21dbus-daemon10:05:002
1302055994732,11cyclictest275-21dbus-daemon10:05:002
1302055994730,13cyclictest3467602-21systemd-journal07:25:022
130205899461,41cyclictest0-21swapper/307:10:023
130205599466,7cyclictest30-21ksoftirqd/211:15:142
130205599464,10cyclictest30-21ksoftirqd/209:29:382
1302055994638,5cyclictest1305397-21kworker/2:109:08:132
1302055994638,5cyclictest1305397-21kworker/2:108:37:542
1302055994638,5cyclictest1303588-21kworker/2:008:22:382
1302055994638,5cyclictest1303588-21kworker/2:008:06:322
1302055994638,5cyclictest1303588-21kworker/2:007:50:502
1302055994638,5cyclictest1301426-21kworker/2:308:01:282
1302055994638,5cyclictest1301426-21kworker/2:307:39:042
1302055994638,4cyclictest1303588-21kworker/2:008:16:572
1302055994625,17cyclictest30-21ksoftirqd/211:51:092
130205599456,9cyclictest30-21ksoftirqd/212:35:072
130205599456,9cyclictest30-21ksoftirqd/212:35:072
130205599455,8cyclictest30-21ksoftirqd/212:30:582
130205599455,8cyclictest30-21ksoftirqd/211:06:182
1302055994538,4cyclictest1301426-21kworker/2:307:43:532
1302055994538,4cyclictest1301426-21kworker/2:307:43:522
1302055994538,4cyclictest1301426-21kworker/2:307:21:352
1302055994537,5cyclictest1303588-21kworker/2:008:25:162
1302055994510,9cyclictest30-21ksoftirqd/209:46:422
1302046994527,14cyclictest1302206-21sshd07:10:110
130204699452,5cyclictest1306656-21munin-run08:40:000
1302055994438,4cyclictest1301426-21kworker/2:307:34:272
1302055994437,4cyclictest1303588-21kworker/2:008:43:362
1302055994437,4cyclictest1301426-21kworker/2:307:46:202
1302055994436,5cyclictest1305397-21kworker/2:109:00:242
1302046994435,5cyclictest16850irq/134-800000008:00:030
1302046994427,14cyclictest1305082-21sshd08:09:110
130204699436,30cyclictest1307690-21run-parts09:00:000
1302046994336,4cyclictest16950irq/224-800000007:40:010
130204699433,36cyclictest0-21swapper/008:50:000
130204699433,35cyclictest1303015-21sshd07:29:120
1302046994326,13cyclictest1302474-21sshd07:17:120
1302046994227,11cyclictest0-21swapper/007:35:010
1302046994212,25cyclictest1-21systemd08:55:010
1302046994129,8cyclictest3467602-21systemd-journal08:20:030
1302046994027,10cyclictest1302752-21sshd07:21:120
1302046993924,11cyclictest1303817-21munin-run07:45:010
1302046993924,11cyclictest1303817-21munin-run07:45:000
1302046993922,4cyclictest12-21ksoftirqd/009:00:420
1302046993826,8cyclictest1306126-21sshd08:29:120
1302046993723,10cyclictest1304043-21sshd07:48:120
1302046993721,13cyclictest364-21snmpd08:40:270
130204699361,31cyclictest1305677-21latency_hist08:20:010
1302055993321,9cyclictest1302060-21sed07:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional