You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-17 - 01:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Sep 16, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
67301399901,86cyclictest0-21swapper/011:55:020
673013998879,7cyclictest151rcuc/012:10:020
673013998581,2cyclictest0-21swapper/009:30:020
673013998529,39cyclictest766225-21ssh12:16:500
673013998427,52cyclictest721528-21ssh10:40:270
673013998319,60cyclictest690617-21ssh09:34:430
673013998031,45cyclictest719357-21ssh10:35:500
67301399802,71cyclictest673475-21munin-plugin-st07:20:000
673013998024,39cyclictest772094-21ssh12:29:300
673013998024,39cyclictest772094-21ssh12:29:290
673013998013,62cyclictest16850irq/134-800000010:26:030
673013997924,37cyclictest751186-21ssh11:44:400
673013997819,38cyclictest16850irq/134-800000011:10:040
673013997733,27cyclictest709686-21ssh10:15:030
673013997632,28cyclictest772754-21ssh12:30:350
67301399762,5cyclictest682236-21ssh09:16:260
673013997623,49cyclictest703931-21ssh10:02:560
67301399761,66cyclictest0-21swapper/009:36:010
67301399751,6cyclictest749052-21munin-run11:40:010
673013997516,41cyclictest16850irq/134-800000009:20:250
67301399741,69cyclictest717911-21ssh10:32:590
67301399741,56cyclictest680684-21ssh09:13:130
67301399739,53cyclictest16850irq/134-800000009:50:030
67301399731,55cyclictest0-21swapper/010:51:380
673028997265,3cyclictest37-21ksoftirqd/308:45:033
67301399722,12cyclictest731789-21ssh11:02:430
673013997126,28cyclictest776303-21ssh12:38:280
673013997126,28cyclictest776303-21ssh12:38:270
673013997027,39cyclictest769712-21ssh12:24:300
673013997022,44cyclictest679201-21diskmemload11:28:350
673028996924,11cyclictest1-21systemd12:30:023
673028996924,11cyclictest1-21systemd12:30:013
67301399695,60cyclictest70775050ssh10:10:580
673013996926,27cyclictest758077-21ssh11:59:240
673013996924,41cyclictest706990-21ssh10:09:500
673013996923,41cyclictest16850irq/134-800000011:50:020
67301399692,15cyclictest675043-21munin-run07:50:010
673013996914,11cyclictest740905-21ssh11:22:100
673013996914,11cyclictest740905-21ssh11:22:090
67301399682,15cyclictest693978-21sshd09:41:330
673028996727,14cyclictest275-21dbus-daemon11:50:013
673013996734,16cyclictest765240-21ssh12:15:020
673013996725,25cyclictest698245-21ssh09:50:310
673013996717,45cyclictest16950irq/224-800000008:30:020
673013996717,45cyclictest16950irq/224-800000008:30:010
673013996636,27cyclictest16950irq/224-800000011:15:220
673013996624,25cyclictest736391-21ssh11:12:370
67301399661,50cyclictest0-21swapper/009:59:100
673013996524,37cyclictest19350irq/34-mmc111:35:000
673013996524,37cyclictest19350irq/34-mmc111:35:000
673013996524,25cyclictest0-21swapper/010:20:280
673013996418,42cyclictest758984-21ssh12:01:000
673028996125,9cyclictest37-21ksoftirqd/309:42:313
673026996151,6cyclictest678956-21kworker/2:311:40:592
67301399612,15cyclictest676081-21munin-run08:10:000
67301399611,11cyclictest678640-21sshd08:57:330
673026996050,6cyclictest695449-21kworker/2:110:33:092
67301399602,16cyclictest724293-21ssh10:46:340
673028995924,7cyclictest681542-21sh09:15:023
67301399592,12cyclictest364-21snmpd08:52:220
673028995823,8cyclictest0-21swapper/309:30:033
673026995831,19cyclictest30-21ksoftirqd/211:10:022
67301399581,4cyclictest0-21swapper/008:45:270
67301399581,14cyclictest364-21snmpd08:22:210
673028995727,5cyclictest725762-21latency_hist10:50:033
673028995723,11cyclictest37-21ksoftirqd/310:38:323
67302699579,5cyclictest131rcu_preempt08:40:022
67301399572,13cyclictest677925-21unin-run08:45:010
673028995630,9cyclictest37-21ksoftirqd/310:11:193
67302899562,6cyclictest751355-21munin-run11:45:013
67301399562,44cyclictest16850irq/134-800000010:55:230
67301399562,15cyclictest674552-21munin-run07:40:020
67301399562,15cyclictest48-21kcompactd007:30:320
67301399561,13cyclictest364-21snmpd07:50:510
673026995547,5cyclictest749048-21kworker/2:011:56:102
673026995536,14cyclictest730368-21tr11:00:002
673021995539,11cyclictest275-21dbus-daemon07:20:021
67301399551,13cyclictest364-21snmpd08:10:240
673028995426,7cyclictest37-21ksoftirqd/311:57:263
673026995446,5cyclictest749048-21kworker/2:011:46:382
673026995446,5cyclictest678956-21kworker/2:312:32:512
673026995446,5cyclictest678956-21kworker/2:309:34:552
673026995435,14cyclictest1-21systemd10:55:012
67301399541,8cyclictest284-21systemd-logind09:10:020
67301399541,11cyclictest673007-21cyclictest07:10:020
67301399541,11cyclictest673007-21cyclictest07:10:020
673026995345,5cyclictest695449-21kworker/2:110:46:202
673026995345,5cyclictest695449-21kworker/2:110:16:182
673026995345,5cyclictest432067-21kworker/2:109:16:562
673026995344,6cyclictest749048-21kworker/2:012:25:552
673026995344,6cyclictest749048-21kworker/2:012:25:552
673013995336,14cyclictest12-21ksoftirqd/008:05:020
673013995325,23cyclictest675756-21latency_hist08:00:020
67301399532,12cyclictest673972-21sshd07:25:330
673028995232,15cyclictest1-21systemd12:15:013
673026995244,5cyclictest749048-21kworker/2:012:03:492
673026995244,5cyclictest695449-21kworker/2:111:38:472
673026995244,5cyclictest678956-21kworker/2:311:15:222
673026995244,5cyclictest678956-21kworker/2:310:21:182
673026995244,5cyclictest678956-21kworker/2:309:41:272
673026995244,5cyclictest678956-21kworker/2:309:39:142
673026995244,5cyclictest432067-21kworker/2:109:13:572
673026995243,5cyclictest695449-21kworker/2:111:21:392
673026995243,5cyclictest695449-21kworker/2:111:21:392
673026995243,5cyclictest678956-21kworker/2:310:35:272
673026995243,5cyclictest678956-21kworker/2:310:01:482
673026995242,7cyclictest695449-21kworker/2:110:29:252
673026995242,6cyclictest432067-21kworker/2:108:20:022
673021995232,15cyclictest275-21dbus-daemon12:30:021
673021995232,15cyclictest275-21dbus-daemon12:30:011
673013995221,26cyclictest0-21swapper/008:19:330
673026995144,4cyclictest678956-21kworker/2:309:49:062
673026995143,5cyclictest749048-21kworker/2:012:19:502
673026995143,5cyclictest695449-21kworker/2:111:12:442
673026995143,5cyclictest695449-21kworker/2:111:01:092
673026995143,5cyclictest695449-21kworker/2:110:09:142
673026995143,5cyclictest678956-21kworker/2:311:26:082
673026995143,5cyclictest678956-21kworker/2:309:58:372
673026995143,5cyclictest678956-21kworker/2:309:25:122
673026995142,6cyclictest695449-21kworker/2:111:30:502
673026995142,6cyclictest695449-21kworker/2:111:30:502
673026995141,6cyclictest432067-21kworker/2:107:50:012
673026995043,4cyclictest749048-21kworker/2:012:05:572
673026995042,5cyclictest678956-21kworker/2:312:38:352
673026995042,5cyclictest678956-21kworker/2:312:38:352
673026995042,5cyclictest678956-21kworker/2:312:10:042
673026995042,5cyclictest678956-21kworker/2:309:53:042
673026995041,6cyclictest695449-21kworker/2:110:10:592
673026995041,5cyclictest749048-21kworker/2:011:54:092
673021995037,8cyclictest309-21in:imuxsock07:40:001
673028994916,8cyclictest37-21ksoftirqd/309:36:293
673026994941,5cyclictest749048-21kworker/2:012:22:392
673026994941,5cyclictest695449-21kworker/2:110:44:192
673026994941,5cyclictest432067-21kworker/2:109:01:332
673026994941,5cyclictest432067-21kworker/2:108:13:512
673021994931,14cyclictest731239-21sshd11:01:331
67301399492,14cyclictest673222-21systemd-run07:15:010
673028994832,13cyclictest3467602-21systemd-journal07:35:033
67302899482,13cyclictest766257-21rm12:16:553
673026994840,5cyclictest678956-21kworker/2:309:23:292
673026994840,5cyclictest432067-21kworker/2:107:29:332
673026994840,5cyclictest432067-21kworker/2:107:22:352
673021994840,6cyclictest23-21ksoftirqd/110:28:371
673013994829,14cyclictest677658-21grep08:40:000
673026994740,5cyclictest432067-21kworker/2:108:29:332
673026994740,5cyclictest432067-21kworker/2:108:29:332
673021994733,9cyclictest767592-21systemctl12:20:011
673013994716,27cyclictest677376-21latency_hist08:35:010
67302899469,6cyclictest131rcu_preempt12:01:173
673028994635,7cyclictest37-21ksoftirqd/311:25:023
673028994635,7cyclictest37-21ksoftirqd/311:25:013
673028994631,11cyclictest1-21systemd10:05:003
673026994638,5cyclictest432067-21kworker/2:109:10:022
673026994638,5cyclictest432067-21kworker/2:108:56:312
673026994638,5cyclictest432067-21kworker/2:107:20:022
673021994632,10cyclictest275-21dbus-daemon07:15:011
673021994620,22cyclictest23-21ksoftirqd/111:35:001
673021994620,22cyclictest23-21ksoftirqd/111:35:001
67301399469,6cyclictest0-21swapper/007:25:000
67302899453,3cyclictest131rcu_preempt11:39:123
673028994532,9cyclictest1-21systemd10:30:013
673028994532,10cyclictest275-21dbus-daemon07:25:003
673028994511,5cyclictest131rcu_preempt10:43:063
673026994537,5cyclictest432067-21kworker/2:108:22:062
673026994537,5cyclictest432067-21kworker/2:108:04:592
673026994537,5cyclictest432067-21kworker/2:107:53:212
67302199453,3cyclictest131rcu_preempt09:55:021
673021994531,11cyclictest23-21ksoftirqd/111:20:031
673021994531,11cyclictest23-21ksoftirqd/111:20:021
673021994531,10cyclictest1-21systemd10:25:011
673021994530,11cyclictest3467602-21systemd-journal10:35:011
673021994521,20cyclictest23-21ksoftirqd/109:11:161
673021994517,17cyclictest690591-21ssh09:34:391
67302199451,3cyclictest131rcu_preempt11:42:001
673013994533,9cyclictest674751-21date07:40:020
673028994426,14cyclictest275-21dbus-daemon09:00:003
67302899441,30cyclictest732912-21rm11:05:023
673026994438,3cyclictest432067-21kworker/2:108:08:422
673026994437,4cyclictest432067-21kworker/2:107:56:372
673026994436,5cyclictest432067-21kworker/2:107:10:272
67302699442,22cyclictest291rcuc/207:35:032
67302199449,7cyclictest131rcu_preempt09:19:591
673021994427,12cyclictest309-21in:imuxsock08:40:001
673021994424,15cyclictest275-21dbus-daemon07:45:011
67302199442,3cyclictest131rcu_preempt09:20:481
673021994420,17cyclictest275-21dbus-daemon07:25:001
673028994323,15cyclictest275-21dbus-daemon09:55:003
67302899432,13cyclictest683149-21ssh09:18:443
673026994335,5cyclictest432067-21kworker/2:108:48:432
673026994335,5cyclictest432067-21kworker/2:108:30:172
673026994334,5cyclictest432067-21kworker/2:108:42:002
67302199436,24cyclictest0-21swapper/111:10:021
673021994333,8cyclictest23-21ksoftirqd/109:42:451
673021994333,6cyclictest721084-21sendmail10:40:001
673021994332,7cyclictest274-21cron12:00:001
673021994330,10cyclictest23-21ksoftirqd/110:13:121
673028994224,14cyclictest1-21systemd07:55:003
673026994234,5cyclictest432067-21kworker/2:107:40:492
67302199424,4cyclictest131rcu_preempt11:46:311
67302199423,3cyclictest131rcu_preempt10:03:061
673021994229,10cyclictest676093-21systemd-run08:10:001
673021994228,10cyclictest1-21systemd08:05:011
673021994213,6cyclictest131rcu_preempt11:19:591
673021994210,6cyclictest131rcu_preempt10:10:011
673028994132,5cyclictest711780-21sendmail10:20:013
673028994128,10cyclictest275-21dbus-daemon08:50:003
673028994111,6cyclictest131rcu_preempt07:30:023
673026994133,5cyclictest432067-21kworker/2:108:51:582
673026994110,6cyclictest131rcu_preempt07:30:332
673021994128,10cyclictest23-21ksoftirqd/111:40:001
673021994127,11cyclictest23-21ksoftirqd/110:19:541
673021994126,11cyclictest23-21ksoftirqd/109:28:451
673021994125,12cyclictest723443-21sh10:45:001
673021994125,12cyclictest1-21systemd12:35:001
673013994127,10cyclictest309-21in:imuxsock09:05:010
673028994031,7cyclictest37-21ksoftirqd/308:30:013
673028994031,7cyclictest37-21ksoftirqd/308:30:013
673028994029,9cyclictest37-21ksoftirqd/312:08:293
673028994026,10cyclictest37-21ksoftirqd/312:35:003
673028994025,12cyclictest37-21ksoftirqd/307:20:013
67302899402,4cyclictest131rcu_preempt12:40:023
67302899402,4cyclictest131rcu_preempt12:40:013
673028994022,14cyclictest275-21dbus-daemon11:10:013
67302899401,20cyclictest0-21swapper/311:20:023
67302199408,5cyclictest131rcu_preempt10:45:271
673021994026,11cyclictest765399-21sed12:15:021
673021994022,15cyclictest1-21systemd08:35:011
673021994017,19cyclictest23-21ksoftirqd/111:10:151
67302199401,4cyclictest131rcu_preempt12:22:541
673028993929,8cyclictest37-21ksoftirqd/311:13:413
673028993928,9cyclictest37-21ksoftirqd/309:05:013
673028993923,12cyclictest37-21ksoftirqd/311:28:543
67302199397,7cyclictest23-21ksoftirqd/108:56:331
67302199395,31cyclictest150-21pr/ttyS209:04:441
673021993928,9cyclictest23-21ksoftirqd/111:25:151
673021993927,9cyclictest23-21ksoftirqd/108:20:011
673021993925,11cyclictest23-21ksoftirqd/111:51:561
673021993925,10cyclictest275-21dbus-daemon08:50:011
673021993914,6cyclictest23-21ksoftirqd/109:48:541
67302899385,5cyclictest131rcu_preempt12:25:003
673028993830,5cyclictest37-21ksoftirqd/307:15:013
673028993828,6cyclictest730514-21sh11:00:033
673028993827,9cyclictest37-21ksoftirqd/309:45:043
673028993826,9cyclictest37-21ksoftirqd/309:30:143
673028993826,9cyclictest37-21ksoftirqd/309:24:533
673028993811,4cyclictest131rcu_preempt08:07:523
67302199384,6cyclictest131rcu_preempt07:10:021
67302199384,6cyclictest131rcu_preempt07:10:021
673021993829,6cyclictest23-21ksoftirqd/107:35:021
673021993825,10cyclictest679368-21latency_hist09:10:021
673021993813,6cyclictest23-21ksoftirqd/110:50:511
67302199381,33cyclictest48-21kcompactd008:20:281
67302899373,4cyclictest131rcu_preempt10:25:013
673028993726,8cyclictest37-21ksoftirqd/311:51:043
673028993722,10cyclictest37-21ksoftirqd/311:30:583
673028993722,10cyclictest37-21ksoftirqd/311:30:573
673028993721,10cyclictest309-21in:imuxsock08:05:003
673021993730,4cyclictest762780-21sh12:09:301
673021993728,7cyclictest23-21ksoftirqd/110:56:531
673021993727,8cyclictest23-21ksoftirqd/112:35:301
673021993727,8cyclictest23-21ksoftirqd/112:35:291
673021993725,9cyclictest675112-21sed07:50:021
673021993724,10cyclictest23-21ksoftirqd/109:39:141
67302199372,3cyclictest131rcu_preempt09:55:421
673021993723,11cyclictest23-21ksoftirqd/112:01:551
673021993723,11cyclictest23-21ksoftirqd/108:15:021
673021993722,11cyclictest23-21ksoftirqd/107:27:331
673028993628,6cyclictest707065-21ssh10:10:003
673028993626,7cyclictest675007-21sshd07:45:333
673028993623,9cyclictest37-21ksoftirqd/307:31:313
673028993623,10cyclictest675584-21systemd-run08:00:013
673028993622,11cyclictest37-21ksoftirqd/310:30:453
673028993622,11cyclictest37-21ksoftirqd/309:56:423
67302199364,3cyclictest131rcu_preempt07:55:011
673028993524,8cyclictest37-21ksoftirqd/310:52:353
673028993523,9cyclictest37-21ksoftirqd/309:06:083
673028993522,10cyclictest37-21ksoftirqd/308:17:003
67302899351,7cyclictest0-21swapper/308:55:013
673028993514,16cyclictest275-21dbus-daemon08:25:013
67302199351,4cyclictest131rcu_preempt08:00:011
673026993427,4cyclictest673142-21latency_hist07:10:022
673026993427,4cyclictest673142-21latency_hist07:10:022
673021993427,4cyclictest1-21systemd08:55:001
67302899336,3cyclictest131rcu_preempt08:40:013
67302899331,11cyclictest131rcu_preempt08:35:003
673021993324,5cyclictest23-21ksoftirqd/108:30:011
673021993324,5cyclictest23-21ksoftirqd/108:30:011
673028993223,6cyclictest37-21ksoftirqd/307:45:013
67302899302,5cyclictest131rcu_preempt07:10:033
67302899302,5cyclictest131rcu_preempt07:10:023
673028993018,8cyclictest37-21ksoftirqd/308:10:523
67302199301,9cyclictest0-21swapper/108:45:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional