You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-29 - 21:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Mar 29, 2026 12:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
298674099841,46cyclictest3467602-21systemd-journal12:17:120
298674099766,8cyclictest12-21ksoftirqd/012:08:530
2986740997520,32cyclictest3047446-21rm11:17:150
2986740997520,32cyclictest3047446-21rm11:17:140
298674699723,56cyclictest3014399-21ssh10:00:032
2986740997066,2cyclictest0-21swapper/011:25:020
2986740997066,2cyclictest0-21swapper/011:25:020
2986740997048,18cyclictest16850irq/134-800000009:45:000
2986740996931,18cyclictest3467602-21systemd-journal09:30:000
298674099686,24cyclictest3031396-21unin-run10:40:000
298674999671,41cyclictest361rcuc/311:15:023
298674099656,9cyclictest3027249-21ssh10:30:020
298674099655,9cyclictest12-21ksoftirqd/008:00:000
2986740996457,5cyclictest0-21swapper/010:00:020
2986740996411,30cyclictest131rcu_preempt11:40:360
2986740996411,30cyclictest131rcu_preempt11:40:350
298674099639,30cyclictest131rcu_preempt11:25:010
2986740996341,17cyclictest2992910-21diskmemload10:20:010
2986740996327,5cyclictest131rcu_preempt09:33:510
2986740996327,28cyclictest12-21ksoftirqd/012:23:230
298674099632,58cyclictest0-21swapper/009:15:010
298674099628,5cyclictest131rcu_preempt11:55:220
298674099622,8cyclictest12-21ksoftirqd/010:55:010
2986740996227,30cyclictest3015073-21ssh10:01:370
2986740996227,30cyclictest3015073-21ssh10:01:370
2986740996125,32cyclictest12-21ksoftirqd/012:33:530
2986740996125,32cyclictest12-21ksoftirqd/012:33:520
298674099608,11cyclictest12-21ksoftirqd/009:50:000
298674099608,11cyclictest12-21ksoftirqd/009:50:000
2986740996046,9cyclictest12-21ksoftirqd/010:10:010
2986740996020,8cyclictest12-21ksoftirqd/011:53:350
2986746995943,7cyclictest3022848-21kworker/2:211:25:102
2986746995943,7cyclictest3022848-21kworker/2:211:25:102
298674099594,8cyclictest121ksoftirqd/011:35:040
2986740995917,34cyclictest2992910-21diskmemload11:35:450
2986742995817,9cyclictest23-21ksoftirqd/111:40:021
298674099588,5cyclictest131rcu_preempt10:13:370
298674099588,5cyclictest131rcu_preempt10:13:370
298674099587,20cyclictest0-21swapper/010:57:450
2986740995854,2cyclictest0-21swapper/008:45:020
2986740995834,18cyclictest275-21dbus-daemon11:15:000
2986746995748,5cyclictest3022848-21kworker/2:211:20:102
298674099577,7cyclictest12-21ksoftirqd/012:30:000
298674099576,6cyclictest12-21ksoftirqd/010:44:580
298674099576,4cyclictest131rcu_preempt08:15:020
298674099571,10cyclictest12-21ksoftirqd/012:05:010
298674099564,14cyclictest12-21ksoftirqd/009:55:000
2986740995627,13cyclictest3060285-21ssh11:47:080
298674099562,12cyclictest12-21ksoftirqd/009:15:430
298674099562,12cyclictest12-21ksoftirqd/009:15:430
2986746995545,7cyclictest2966963-21kworker/2:309:21:262
2986746995537,7cyclictest3007764-21kworker/2:009:48:272
2986746995537,7cyclictest3007764-21kworker/2:009:48:272
2986742995513,5cyclictest131rcu_preempt09:20:011
2986742995513,5cyclictest131rcu_preempt09:20:011
298674099555,8cyclictest12-21ksoftirqd/008:35:000
298674099555,6cyclictest121ksoftirqd/010:46:450
2986740995519,18cyclictest12-21ksoftirqd/012:38:360
298674099551,51cyclictest0-21swapper/007:50:020
2986740995511,6cyclictest131rcu_preempt11:07:210
298674099551,13cyclictest12-21ksoftirqd/009:25:000
298674099551,11cyclictest12-21ksoftirqd/007:35:000
2986740995510,6cyclictest131rcu_preempt08:05:010
2986740995510,6cyclictest131rcu_preempt08:05:010
2986749995426,16cyclictest37-21ksoftirqd/312:37:433
2986746995446,5cyclictest2966963-21kworker/2:309:13:092
2986746995445,5cyclictest3022848-21kworker/2:211:19:102
2986746995445,5cyclictest3022848-21kworker/2:211:19:092
2986746995444,5cyclictest2966963-21kworker/2:308:00:002
2986746995439,7cyclictest2966963-21kworker/2:311:09:592
2986746995418,8cyclictest3057127-21kworker/2:012:21:102
298674099548,25cyclictest131rcu_preempt10:20:440
2986740995424,20cyclictest3042098-21latency_hist11:05:020
2986749995337,13cyclictest1-21systemd07:40:003
2986746995345,5cyclictest3057127-21kworker/2:012:35:002
2986746995345,5cyclictest3057127-21kworker/2:012:35:002
2986746995345,5cyclictest2966963-21kworker/2:309:33:282
2986746995344,6cyclictest3057127-21kworker/2:012:03:402
2986746995344,5cyclictest3022848-21kworker/2:211:40:002
2986746995314,6cyclictest131rcu_preempt08:15:002
2986742995334,15cyclictest1-21systemd12:35:011
2986742995334,15cyclictest1-21systemd12:35:001
2986742995313,5cyclictest131rcu_preempt09:56:571
298674099539,5cyclictest131rcu_preempt07:55:020
2986740995345,5cyclictest16850irq/134-800000009:10:020
2986740995345,5cyclictest16850irq/134-800000009:10:010
2986746995245,4cyclictest2966963-21kworker/2:309:25:312
2986746995244,5cyclictest2966963-21kworker/2:310:00:342
2986746995244,5cyclictest2966963-21kworker/2:310:00:342
2986746995243,6cyclictest2966963-21kworker/2:311:13:222
2986746995243,5cyclictest3057127-21kworker/2:012:39:322
2986746995243,5cyclictest2966963-21kworker/2:309:44:432
2986746995242,6cyclictest2966963-21kworker/2:311:35:012
2986742995210,5cyclictest131rcu_preempt12:15:021
298674099523,7cyclictest12-21ksoftirqd/008:10:010
2986740995225,12cyclictest3071990-21ssh12:14:590
2986740995225,12cyclictest3071990-21ssh12:14:590
2986746995144,4cyclictest2966963-21kworker/2:310:16:162
2986746995144,4cyclictest2966963-21kworker/2:307:40:002
2986746995143,5cyclictest3022848-21kworker/2:210:53:512
2986746995143,5cyclictest2966963-21kworker/2:310:06:512
2986746995143,5cyclictest2966963-21kworker/2:309:17:412
2986746995143,5cyclictest2966963-21kworker/2:309:17:412
2986746995143,4cyclictest2966963-21kworker/2:307:30:012
2986746995142,6cyclictest3022848-21kworker/2:210:30:072
2986746995142,5cyclictest2966963-21kworker/2:310:27:512
2986746995133,7cyclictest3057127-21kworker/2:012:10:132
2986746995133,7cyclictest3057127-21kworker/2:012:10:132
298674299517,7cyclictest2557645-1kworker/1:2H09:55:011
2986742995133,13cyclictest3074634-21tmpfiles12:20:361
2986742995118,21cyclictest131rcu_preempt09:15:001
298674099517,9cyclictest12-21ksoftirqd/009:40:010
298674099513,41cyclictest0-21swapper/007:20:000
298674099513,41cyclictest0-21swapper/007:20:000
298674099512,8cyclictest12-21ksoftirqd/007:40:010
2986746995042,5cyclictest3057127-21kworker/2:012:25:342
2986746995042,5cyclictest3057127-21kworker/2:012:07:082
2986746995042,5cyclictest3057127-21kworker/2:011:47:012
2986746995042,5cyclictest3022848-21kworker/2:210:42:592
2986746995042,5cyclictest3022848-21kworker/2:210:39:102
2986746995042,5cyclictest2966963-21kworker/2:309:36:322
2986746995036,6cyclictest2966963-21kworker/2:310:58:512
2986746995035,11cyclictest275-21dbus-daemon11:05:012
298674299506,6cyclictest131rcu_preempt11:10:061
298674299502,8cyclictest3065673-21latency_hist12:00:021
2986740995018,11cyclictest12-21ksoftirqd/008:20:010
298674999495,5cyclictest131rcu_preempt09:20:013
298674999495,5cyclictest131rcu_preempt09:20:013
2986749994932,12cyclictest3467602-21systemd-journal09:00:003
2986746994941,5cyclictest3057127-21kworker/2:011:50:542
2986746994941,5cyclictest3057127-21kworker/2:011:44:402
2986746994941,5cyclictest3057127-21kworker/2:011:44:402
2986746994941,5cyclictest3022848-21kworker/2:210:49:222
2986746994941,5cyclictest2966963-21kworker/2:309:51:222
2986746994940,6cyclictest3057127-21kworker/2:012:19:092
298674299498,7cyclictest131rcu_preempt09:40:011
298674299496,4cyclictest131rcu_preempt07:20:011
298674299496,4cyclictest131rcu_preempt07:20:011
2986742994912,9cyclictest131rcu_preempt11:45:501
2986742994910,5cyclictest131rcu_preempt11:30:271
298674099498,5cyclictest131rcu_preempt08:40:000
298674099492,9cyclictest12-21ksoftirqd/010:30:060
2986749994831,13cyclictest3467602-21systemd-journal11:10:013
2986746994840,5cyclictest3057127-21kworker/2:011:59:142
2986746994840,5cyclictest2966963-21kworker/2:310:10:522
2986746994840,5cyclictest2966963-21kworker/2:310:10:522
298674299487,6cyclictest131rcu_preempt11:17:351
298674299487,6cyclictest131rcu_preempt11:17:351
2986742994816,8cyclictest23-21ksoftirqd/109:50:011
2986742994816,8cyclictest23-21ksoftirqd/109:50:011
298674299481,5cyclictest131rcu_preempt10:06:081
2986740994829,10cyclictest309-21in:imuxsock09:00:010
298674999478,6cyclictest131rcu_preempt10:56:353
2986749994732,11cyclictest3467602-21systemd-journal11:50:023
2986746994740,4cyclictest2966963-21kworker/2:307:33:282
2986746994739,5cyclictest2966963-21kworker/2:307:20:012
2986746994739,5cyclictest2966963-21kworker/2:307:20:012
298674299479,6cyclictest131rcu_preempt10:26:221
298674299478,6cyclictest131rcu_preempt10:55:001
298674299476,5cyclictest131rcu_preempt12:38:361
298674299474,10cyclictest23-21ksoftirqd/112:20:001
298674299473,4cyclictest131rcu_preempt10:02:441
298674299473,4cyclictest131rcu_preempt10:02:441
2986742994733,11cyclictest3050505-21sshd11:24:351
2986742994729,6cyclictest23-21ksoftirqd/109:05:021
2986742994729,4cyclictest23-21ksoftirqd/111:00:011
2986742994723,6cyclictest23-21ksoftirqd/109:45:021
2986749994630,13cyclictest37-21ksoftirqd/311:40:033
2986749994628,7cyclictest2986912-21sshd07:14:353
2986746994639,4cyclictest2966963-21kworker/2:310:23:212
2986746994639,4cyclictest2966963-21kworker/2:308:20:012
2986746994637,6cyclictest2966963-21kworker/2:307:12:352
2986746994630,11cyclictest3467602-21systemd-journal08:10:002
298674299468,4cyclictest131rcu_preempt11:09:321
298674299466,7cyclictest0-21swapper/110:48:091
298674299466,5cyclictest131rcu_preempt09:20:041
298674299462,7cyclictest0-21swapper/111:04:091
2986742994625,5cyclictest23-21ksoftirqd/108:25:011
2986742994621,5cyclictest23-21ksoftirqd/112:08:561
2986742994618,22cyclictest23-21ksoftirqd/110:20:001
2986742994611,6cyclictest131rcu_preempt12:10:411
2986740994628,14cyclictest2987436-21sshd07:21:350
2986749994531,11cyclictest3078924-21sshd12:30:363
2986749994531,11cyclictest3078924-21sshd12:30:363
2986749994531,11cyclictest1-21systemd08:00:013
2986749994531,10cyclictest1-21systemd11:35:003
2986749994531,10cyclictest1-21systemd10:25:023
2986746994538,4cyclictest2966963-21kworker/2:308:46:012
2986746994529,13cyclictest3467602-21systemd-journal09:05:022
2986746994529,13cyclictest3467602-21systemd-journal09:05:022
298674299457,29cyclictest131rcu_preempt12:29:551
298674299454,6cyclictest131rcu_preempt10:15:001
298674299454,6cyclictest131rcu_preempt10:15:001
2986742994528,12cyclictest23-21ksoftirqd/112:05:011
2986742994521,4cyclictest23-21ksoftirqd/109:34:431
2986742994520,5cyclictest23-21ksoftirqd/110:44:421
2986742994520,5cyclictest23-21ksoftirqd/108:15:021
2986742994520,4cyclictest23-21ksoftirqd/110:33:281
298674299451,5cyclictest131rcu_preempt10:40:001
2986740994533,7cyclictest12-21ksoftirqd/007:15:010
2986740994510,5cyclictest131rcu_preempt07:45:000
298674999446,9cyclictest37-21ksoftirqd/311:29:473
298674999446,9cyclictest37-21ksoftirqd/311:29:463
298674999443,3cyclictest131rcu_preempt09:40:003
2986749994430,10cyclictest1-21systemd08:40:013
298674999441,3cyclictest131rcu_preempt09:28:083
2986746994437,4cyclictest2966963-21kworker/2:307:42:132
2986746994435,5cyclictest2966963-21kworker/2:309:00:012
298674299449,5cyclictest131rcu_preempt07:51:351
2986742994427,13cyclictest3467602-21systemd-journal07:14:181
298674299442,5cyclictest131rcu_preempt09:00:001
2986742994424,4cyclictest23-21ksoftirqd/108:50:001
2986742994422,5cyclictest23-21ksoftirqd/108:55:001
2986742994422,5cyclictest23-21ksoftirqd/108:05:011
2986742994422,5cyclictest23-21ksoftirqd/108:05:011
2986742994421,5cyclictest23-21ksoftirqd/111:42:041
2986742994421,5cyclictest23-21ksoftirqd/111:42:041
2986742994411,6cyclictest131rcu_preempt07:35:001
2986742994410,6cyclictest131rcu_preempt08:38:351
298674099446,8cyclictest12-21ksoftirqd/007:30:000
2986740994410,6cyclictest131rcu_preempt07:10:010
2986740994410,6cyclictest131rcu_preempt07:10:010
2986749994330,10cyclictest37-21ksoftirqd/311:02:353
2986749994329,9cyclictest275-21dbus-daemon08:50:013
2986749994326,10cyclictest37-21ksoftirqd/309:50:003
2986749994326,10cyclictest37-21ksoftirqd/309:50:003
2986749994315,20cyclictest0-21swapper/310:16:033
2986749994314,21cyclictest0-21swapper/312:13:483
2986749994314,21cyclictest0-21swapper/312:13:473
298674999431,3cyclictest131rcu_preempt12:27:463
298674699436,6cyclictest131rcu_preempt08:30:022
2986742994331,7cyclictest2990305-21sendmail08:20:001
298674299432,4cyclictest131rcu_preempt11:26:411
298674299432,4cyclictest131rcu_preempt11:26:401
2986742994321,5cyclictest23-21ksoftirqd/110:20:141
2986742994321,4cyclictest23-21ksoftirqd/109:10:011
2986742994321,4cyclictest23-21ksoftirqd/109:10:011
2986742994321,4cyclictest23-21ksoftirqd/108:33:491
2986742994320,6cyclictest23-21ksoftirqd/108:10:001
2986742994320,5cyclictest23-21ksoftirqd/108:00:021
298674099438,13cyclictest12-21ksoftirqd/008:30:010
298674099433,8cyclictest12-21ksoftirqd/009:05:010
2986740994326,11cyclictest2992137-21systemd-run08:55:010
2986749994229,10cyclictest37-21ksoftirqd/310:55:023
2986749994227,11cyclictest275-21dbus-daemon07:35:013
298674999422,5cyclictest131rcu_preempt10:40:003
2986749994225,13cyclictest3467602-21systemd-journal10:15:013
2986749994225,13cyclictest3467602-21systemd-journal10:15:003
2986749994215,20cyclictest0-21swapper/312:20:553
2986749994211,22cyclictest0-21swapper/310:25:363
2986746994234,5cyclictest2966963-21kworker/2:308:24:492
2986746994227,12cyclictest30-21ksoftirqd/207:55:022
298674299424,6cyclictest3062011-21ssh11:51:041
2986742994220,4cyclictest23-21ksoftirqd/109:26:281
2986742994218,6cyclictest23-21ksoftirqd/107:30:001
2986740994232,7cyclictest12-21ksoftirqd/008:25:010
298674999414,3cyclictest131rcu_preempt11:41:273
298674999414,3cyclictest131rcu_preempt11:41:273
2986749994130,7cyclictest309-21in:imuxsock07:45:013
2986749994129,9cyclictest3050880-21sed11:25:023
2986749994125,13cyclictest1-21systemd09:15:003
2986749994117,9cyclictest0-21swapper/311:15:463
2986749994117,9cyclictest0-21swapper/311:15:463
2986749994117,20cyclictest37-21ksoftirqd/312:00:453
2986749994115,18cyclictest37-21ksoftirqd/309:43:163
298674999411,3cyclictest131rcu_preempt09:52:183
2986746994127,10cyclictest3467602-21systemd-journal08:40:002
2986742994132,6cyclictest2987466-21latency_hist07:25:001
2986742994125,11cyclictest275-21dbus-daemon07:45:011
2986742994124,4cyclictest23-21ksoftirqd/107:50:021
2986742994119,5cyclictest23-21ksoftirqd/108:28:471
298674999403,3cyclictest131rcu_preempt10:30:283
2986749994031,5cyclictest37-21ksoftirqd/308:15:003
2986749994025,12cyclictest37-21ksoftirqd/311:45:223
298674999401,8cyclictest3065641-21grep12:00:003
2986749994014,16cyclictest37-21ksoftirqd/310:09:513
2986749994010,22cyclictest0-21swapper/310:00:303
2986749994010,22cyclictest0-21swapper/310:00:293
2986746994032,5cyclictest2966963-21kworker/2:309:02:182
2986746994017,5cyclictest30-21ksoftirqd/208:55:002
2986746994015,5cyclictest30-21ksoftirqd/208:05:002
2986746994015,5cyclictest30-21ksoftirqd/208:05:002
298674999396,6cyclictest0-21swapper/309:21:063
298674999395,6cyclictest131rcu_preempt09:34:053
298674999393,6cyclictest131rcu_preempt12:20:003
298674999392,5cyclictest131rcu_preempt12:07:523
2986749993913,14cyclictest37-21ksoftirqd/308:20:013
2986746993932,4cyclictest2966963-21kworker/2:307:50:022
2986746993931,5cyclictest2966963-21kworker/2:308:40:102
298674299394,3cyclictest0-21swapper/108:45:021
2986742993927,9cyclictest2986744-21latency_hist07:10:011
2986742993927,9cyclictest2986744-21latency_hist07:10:011
2986740993926,9cyclictest2991886-21systemd-run08:50:000
298674999384,3cyclictest131rcu_preempt07:20:013
298674999384,3cyclictest131rcu_preempt07:20:003
2986749993827,8cyclictest1-21systemd08:30:013
2986749993826,10cyclictest37-21ksoftirqd/309:56:073
2986749993824,10cyclictest37-21ksoftirqd/310:40:103
298674999381,3cyclictest131rcu_preempt09:10:003
298674999381,3cyclictest131rcu_preempt09:10:003
2986749993812,16cyclictest37-21ksoftirqd/310:49:573
2986746993830,5cyclictest2966963-21kworker/2:307:23:232
298674299383,4cyclictest131rcu_preempt07:40:001
2986749993727,7cyclictest2990566-21sshd08:24:353
298674999372,18cyclictest0-21swapper/308:44:313
2986749993713,16cyclictest310-21in:imklog08:05:023
2986749993713,16cyclictest310-21in:imklog08:05:023
298674699375,3cyclictest131rcu_preempt08:35:002
2986749993621,11cyclictest37-21ksoftirqd/307:30:003
298674999361,11cyclictest131rcu_preempt09:02:163
298674999354,3cyclictest131rcu_preempt08:55:013
298674999354,3cyclictest131rcu_preempt08:10:003
298674999353,3cyclictest131rcu_preempt07:55:013
2986749993518,13cyclictest1-21systemd07:50:013
298674999351,4cyclictest2987482-21systemd-run07:25:003
298674999341,3cyclictest131rcu_preempt08:31:353
2986749993224,6cyclictest37-21ksoftirqd/307:10:013
2986749993224,6cyclictest37-21ksoftirqd/307:10:013
298674699321,3cyclictest131rcu_preempt07:10:012
298674699321,3cyclictest131rcu_preempt07:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional