You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-04 - 20:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Jun 04, 2026 12:45:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
317636099881,82cyclictest3224238-21ssh10:42:200
317636099881,82cyclictest3224238-21ssh10:42:200
3176360998229,49cyclictest3188324-21ssh09:22:290
317636899766,22cyclictest3270256-21cut12:25:002
317636099761,58cyclictest3252552-21ssh11:45:050
3176373997422,18cyclictest3215149-21ssh10:22:143
3176373997422,18cyclictest3215149-21ssh10:22:143
317636099748,15cyclictest3209460-21ssh10:09:470
3176360997419,51cyclictest16850irq/134-800000010:47:440
3176368997321,17cyclictest275-21dbus-daemon11:55:012
317636899724,22cyclictest3205009-21modprobe09:59:592
3176360997223,44cyclictest0-21swapper/009:32:570
3176360997127,40cyclictest3212275-21ssh10:15:450
3176373997032,13cyclictest3208827-21ssh10:08:133
3176373997024,17cyclictest3272048-21ssh12:28:583
3176360997016,50cyclictest3268525-21ssh12:20:450
3176373996938,27cyclictest3222455-21ssh10:38:353
3176360996934,31cyclictest16850irq/134-800000012:05:020
3176360996934,31cyclictest16850irq/134-800000012:05:020
3176360996930,22cyclictest3185630-21ssh09:16:360
317636099692,6cyclictest3198288-21munin-run09:45:000
3176360996916,35cyclictest16850irq/134-800000010:04:430
3176360996910,42cyclictest16850irq/134-800000012:36:190
3176360996910,42cyclictest16850irq/134-800000012:36:190
3176373996829,13cyclictest3242083-21ssh11:22:083
317636099686,53cyclictest16850irq/134-800000012:29:510
3176360996829,35cyclictest3228760-21ssh10:52:270
3176360996829,35cyclictest3228760-21ssh10:52:270
3176360996828,36cyclictest3248043-21sh11:35:100
317636099682,62cyclictest3211943-21latency_hist10:15:010
3176360996815,22cyclictest16850irq/134-800000012:02:080
3176373996747,16cyclictest3187721-21ssh09:21:003
317636099678,6cyclictest16850irq/134-800000012:10:340
3176360996716,34cyclictest16850irq/134-800000009:46:010
3176360996716,34cyclictest16850irq/134-800000009:46:010
3176360996712,49cyclictest3241898-21ssh11:21:320
3176360996711,30cyclictest16850irq/134-800000011:02:570
3176360996711,30cyclictest16850irq/134-800000011:02:560
3176373996627,16cyclictest364-21snmpd11:38:483
3176363996616,8cyclictest23-21ksoftirqd/109:35:271
3176363996616,8cyclictest23-21ksoftirqd/109:35:271
3176363996616,10cyclictest23-21ksoftirqd/112:22:001
3176360996625,37cyclictest309-21in:imuxsock11:15:000
3176373996546,15cyclictest3244906-21ssh11:28:293
3176373996544,17cyclictest3193476-21ssh09:34:073
3176373996544,16cyclictest3266766-21ssh12:16:553
3176373996544,16cyclictest3266766-21ssh12:16:553
3176360996522,38cyclictest3181480-21sshd08:46:290
3176373996444,16cyclictest3253853-21ssh11:48:183
3176373996444,16cyclictest3235680-21ssh11:08:013
3176373996444,16cyclictest3235680-21ssh11:08:013
3176373996444,16cyclictest3217204-21ssh10:26:493
3176373996444,15cyclictest3234080-21ssh11:04:343
3176373996444,15cyclictest3234080-21ssh11:04:343
3176373996443,17cyclictest3250698-21ssh11:41:073
3176373996443,17cyclictest3219009-21ssh10:30:413
3176373996443,17cyclictest3219009-21ssh10:30:413
3176373996443,16cyclictest3190735-21ssh09:27:503
3176368996453,6cyclictest3182318-21kworker/2:011:40:002
3176363996417,10cyclictest23-21ksoftirqd/109:25:401
3176360996453,7cyclictest3274967-21sed12:35:040
3176360996453,7cyclictest3274967-21sed12:35:040
317636099643,20cyclictest3182710-21wc09:10:030
317636099642,11cyclictest3266970-21ssh12:17:280
317636099642,11cyclictest3266970-21ssh12:17:280
3176373996347,13cyclictest3275510-21ssh12:36:183
3176373996347,13cyclictest3275510-21ssh12:36:183
3176373996347,13cyclictest3265315-21ssh12:13:583
3176373996347,12cyclictest3224306-21ssh10:42:403
3176373996347,12cyclictest3224306-21ssh10:42:393
3176373996346,13cyclictest3257110-21ssh11:55:163
3176373996346,13cyclictest3246344-21ssh11:31:303
3176373996345,14cyclictest3212324-21ssh10:15:513
3176373996345,14cyclictest3205293-21ssh10:00:073
3176373996345,14cyclictest3202359-21ssh09:53:583
3176373996344,16cyclictest3198105-21ssh09:44:283
3176373996344,15cyclictest3274097-21ssh12:33:233
3176373996344,15cyclictest3274097-21ssh12:33:233
3176373996344,15cyclictest3228368-21ssh10:51:343
3176373996344,15cyclictest3228368-21ssh10:51:333
3176373996343,16cyclictest3268605-21ssh12:20:573
3176373996343,16cyclictest3263155-21ssh12:09:083
3176373996343,16cyclictest3263155-21ssh12:09:083
3176373996343,16cyclictest3239236-21ssh11:15:333
3176373996343,16cyclictest3239236-21ssh11:15:323
3176363996319,10cyclictest23-21ksoftirqd/109:34:311
3176360996334,24cyclictest309-21in:imuxsock08:05:010
317636099632,5cyclictest3218498-21munin-run10:30:000
3176360996322,37cyclictest3215006-21rm10:21:550
3176360996322,37cyclictest3215006-21rm10:21:540
317636099632,18cyclictest3258971-21ssh11:59:550
3176373996248,11cyclictest3203066-21ssh09:55:143
3176373996248,11cyclictest3181510-21munin-run08:50:003
3176373996247,12cyclictest3260275-21ssh12:02:343
3176373996247,12cyclictest3225829-21ssh10:45:423
3176373996246,13cyclictest3210642-21ssh10:12:143
3176373996246,12cyclictest3255385-21ssh11:51:333
3176373996246,12cyclictest3231158-21ssh10:57:523
3176373996241,17cyclictest3184070-21ssh09:13:173
3176373996239,18cyclictest3238016-21ssh11:13:113
3176363996225,12cyclictest23-21ksoftirqd/109:11:411
3176363996217,10cyclictest23-21ksoftirqd/111:30:151
3176363996214,9cyclictest23-21ksoftirqd/110:35:011
3176363996214,9cyclictest23-21ksoftirqd/110:35:011
317636099622,8cyclictest3230357-21sshd10:55:490
3176373996146,12cyclictest3194641-21ssh09:36:303
3176373996146,12cyclictest3194641-21ssh09:36:293
3176373996145,12cyclictest3198727-21ssh09:45:303
3176373996145,12cyclictest3198727-21ssh09:45:293
3176363996115,17cyclictest3265050-21ssh12:13:141
3176360996138,20cyclictest16950irq/224-800000011:29:490
3176360996130,27cyclictest3235487-21ssh11:07:220
3176360996130,27cyclictest3235487-21ssh11:07:220
3176360996126,31cyclictest3219308-21ssh10:31:270
3176360996126,31cyclictest3219308-21ssh10:31:270
317636099612,4cyclictest364-21snmpd10:35:140
317636099611,57cyclictest3202016-21ssh09:53:030
3176373996046,11cyclictest3180955-21sshd08:38:283
3176373996046,11cyclictest3180955-21sshd08:38:273
3176373996045,12cyclictest3185040-21ssh09:15:053
3176373996044,13cyclictest3182259-21sshd09:00:343
3176373996044,13cyclictest3180700-21sshd08:32:283
317636099603,6cyclictest3180943-21sshd08:35:290
317636099603,6cyclictest3180943-21sshd08:35:290
317636099603,15cyclictest3191561-21munin-run09:30:000
3176360996023,33cyclictest0-21swapper/009:56:340
3176360996013,43cyclictest3183576-21ssh09:11:590
317636099601,2cyclictest0-21swapper/011:43:110
3176360996012,43cyclictest16850irq/134-800000011:51:290
317637399597,36cyclictest3175885-21kworker/u8:107:40:023
3176373995946,10cyclictest3177583-21sshd07:33:283
3176373995946,10cyclictest3177583-21sshd07:33:273
3176373995945,11cyclictest3181734-21sshd08:52:283
317636099592,8cyclictest3178102-21sshd07:40:070
3176360995927,29cyclictest12-21ksoftirqd/007:55:000
317636099591,54cyclictest0-21swapper/009:40:020
317636099591,54cyclictest0-21swapper/009:40:010
3176373995847,8cyclictest3180199-21sshd08:23:283
3176373995845,10cyclictest3180435-21sshd08:25:243
3176373995844,10cyclictest3181240-21sshd08:44:283
3176363995814,9cyclictest23-21ksoftirqd/110:04:081
3176373995745,9cyclictest3182519-21sshd09:07:283
3176373995745,9cyclictest3178610-21sshd07:50:283
3176373995743,11cyclictest3177081-21sshd07:24:283
3176373995743,10cyclictest364-21snmpd07:11:063
3176373995742,12cyclictest3179679-21munin-run08:15:013
3176368995748,5cyclictest3182318-21kworker/2:012:19:592
3176368995748,5cyclictest3182318-21kworker/2:012:19:592
3176368995739,13cyclictest3467602-21systemd-journal09:09:592
317636399574,21cyclictest3242063-21ssh11:21:581
3176363995715,9cyclictest23-21ksoftirqd/112:32:111
3176363995715,9cyclictest23-21ksoftirqd/112:32:111
3176363995715,10cyclictest23-21ksoftirqd/110:08:041
3176363995715,10cyclictest23-21ksoftirqd/108:50:011
3176363995713,18cyclictest3239842-21ssh11:17:051
3176363995713,18cyclictest3239842-21ssh11:17:051
3176363995711,9cyclictest23-21ksoftirqd/110:39:011
317636099577,14cyclictest3247084-21ssh11:33:200
3176360995739,13cyclictest275-21dbus-daemon07:40:000
3176360995714,31cyclictest3239434-21ssh11:16:100
3176360995714,31cyclictest3239434-21ssh11:16:100
3176373995643,10cyclictest3179400-21sshd08:07:273
3176373995643,10cyclictest3176803-21sshd07:19:283
3176373995642,11cyclictest3178105-21sshd07:40:293
3176368995642,10cyclictest1-21systemd10:45:012
3176368995642,10cyclictest1-21systemd10:45:002
3176363995612,9cyclictest23-21ksoftirqd/109:17:311
317636099563,9cyclictest3177782-21cat07:35:040
317636099563,9cyclictest3177782-21cat07:35:040
317636099562,15cyclictest3181770-21systemd-run08:55:010
3176373995545,6cyclictest3179186-21munin-run08:05:023
3176373995536,14cyclictest364-21snmpd08:56:263
3176368995545,5cyclictest3179979-21kworker/2:209:10:362
3176363995514,10cyclictest23-21ksoftirqd/111:10:151
317636099552,19cyclictest3177100-21munin-run07:25:000
317636099551,7cyclictest3179651-21sshd08:11:280
3176373995446,4cyclictest3177320-21sshd07:25:283
3176368995445,5cyclictest3179979-21kworker/2:210:11:422
3176368995445,5cyclictest3168760-21kworker/2:008:15:002
3176368995432,17cyclictest275-21dbus-daemon08:50:012
3176368995430,18cyclictest275-21dbus-daemon11:50:012
3176363995415,9cyclictest23-21ksoftirqd/111:03:221
3176363995415,9cyclictest23-21ksoftirqd/111:03:211
3176363995415,8cyclictest23-21ksoftirqd/111:28:051
3176368995335,13cyclictest1-21systemd08:10:002
3176363995320,9cyclictest23-21ksoftirqd/110:48:061
3176363995317,6cyclictest23-21ksoftirqd/111:38:211
317636099533,7cyclictest275-21dbus-daemon08:00:010
317636099533,6cyclictest3180738-21unin-run08:35:010
317636099532,15cyclictest3181258-21/usr/sbin/munin08:44:590
3176373995245,4cyclictest3178360-21sshd07:46:283
3176373995245,4cyclictest3178360-21sshd07:46:273
3176373995240,10cyclictest364-21snmpd07:57:393
317636899525,3cyclictest131rcu_preempt07:25:012
3176368995244,5cyclictest3179979-21kworker/2:209:15:162
3176368995241,6cyclictest3179979-21kworker/2:210:59:042
3176363995213,7cyclictest23-21ksoftirqd/112:05:581
3176363995213,7cyclictest23-21ksoftirqd/112:05:581
317636099523,2cyclictest0-21swapper/007:15:020
317636099521,13cyclictest3176880-21latency_hist07:20:020
3176373995144,4cyclictest3179904-21cron08:17:013
317636899514,3cyclictest131rcu_preempt12:08:442
317636899514,3cyclictest131rcu_preempt12:08:442
3176368995143,5cyclictest3179979-21kworker/2:210:51:582
3176368995143,5cyclictest3179979-21kworker/2:210:51:572
3176368995142,6cyclictest3182318-21kworker/2:009:36:092
3176368995142,6cyclictest3182318-21kworker/2:009:36:092
317636399512,6cyclictest3277242-21latency_hist12:40:021
317636399512,6cyclictest3277242-21latency_hist12:40:021
3176363995110,6cyclictest3257636-21ssh11:56:371
3176360995122,25cyclictest12-21ksoftirqd/007:30:010
317636099511,46cyclictest0-21swapper/008:55:060
3176368995042,5cyclictest3179979-21kworker/2:210:04:412
3176368995041,6cyclictest3179979-21kworker/2:212:12:302
317636399507,12cyclictest0-21swapper/112:04:191
317636099503,8cyclictest3179418-21modprobe08:10:000
317636099501,14cyclictest3182284-21cron09:05:000
3176368994939,6cyclictest3182318-21kworker/2:011:04:422
3176368994939,6cyclictest3182318-21kworker/2:011:04:422
317636899492,42cyclictest3467602-21systemd-journal08:50:272
317636399498,7cyclictest3234788-21ssh11:05:381
317636399498,7cyclictest3234788-21ssh11:05:381
317636099497,4cyclictest0-21swapper/007:48:070
317636099497,4cyclictest0-21swapper/007:48:070
317636099492,18cyclictest3176370-21latency_hist07:10:010
317636099492,12cyclictest3179889-21date08:15:030
3176360994910,33cyclictest0-21swapper/008:30:020
3176368994840,5cyclictest3182318-21kworker/2:012:39:072
3176368994840,5cyclictest3182318-21kworker/2:012:39:072
3176368994840,5cyclictest3182318-21kworker/2:011:07:122
3176368994840,5cyclictest3182318-21kworker/2:011:07:112
3176368994840,5cyclictest3179979-21kworker/2:210:21:392
3176368994840,5cyclictest3179979-21kworker/2:210:21:392
317636899483,3cyclictest131rcu_preempt09:50:012
317636899483,3cyclictest131rcu_preempt09:50:002
3176368994829,14cyclictest1-21systemd10:50:012
3176368994816,6cyclictest131rcu_preempt08:20:002
317636399489,13cyclictest3182544-21diskmemload12:26:551
317636399487,12cyclictest0-21swapper/110:57:111
317636399481,15cyclictest0-21swapper/112:18:131
317636399481,15cyclictest0-21swapper/112:18:131
317636399481,11cyclictest23-21ksoftirqd/111:40:121
317636099482,8cyclictest309-21in:imuxsock08:25:000
3176368994739,5cyclictest3182318-21kworker/2:011:22:122
3176368994739,5cyclictest3179979-21kworker/2:208:27:112
3176368994738,6cyclictest3182318-21kworker/2:009:41:472
3176368994737,6cyclictest3182318-21kworker/2:011:26:572
317636399479,7cyclictest3255589-21ssh11:52:121
317636399476,10cyclictest0-21swapper/110:26:161
3176363994713,7cyclictest23-21ksoftirqd/110:16:131
3176363994711,7cyclictest23-21ksoftirqd/110:54:111
3176363994711,7cyclictest23-21ksoftirqd/110:54:111
3176368994638,5cyclictest3179979-21kworker/2:211:17:322
3176368994638,5cyclictest3179979-21kworker/2:211:17:322
3176368994638,5cyclictest3179979-21kworker/2:210:37:102
3176368994637,6cyclictest3182318-21kworker/2:010:16:592
3176368994637,5cyclictest3179979-21kworker/2:209:22:252
3176368994632,10cyclictest3272760-21sh12:30:042
3176368994632,10cyclictest3272760-21sh12:30:042
317636399469,32cyclictest364-21snmpd07:14:111
317636399464,7cyclictest23-21ksoftirqd/109:20:351
3176363994632,11cyclictest1-21systemd11:50:011
3176363994629,13cyclictest3467602-21systemd-journal09:45:011
3176363994625,17cyclictest3201253-21ssh09:51:101
317636899456,30cyclictest3467602-21systemd-journal11:35:002
3176368994537,5cyclictest3179979-21kworker/2:211:14:022
3176368994528,13cyclictest3467602-21systemd-journal08:25:002
3176368994528,12cyclictest3467602-21systemd-journal07:39:592
3176363994528,13cyclictest275-21dbus-daemon07:50:011
3176363994528,13cyclictest275-21dbus-daemon07:50:001
3176363994523,17cyclictest3200161-21ssh09:49:061
3176363994523,17cyclictest3200161-21ssh09:49:061
3176363994515,10cyclictest23-21ksoftirqd/110:22:181
3176363994515,10cyclictest23-21ksoftirqd/110:22:181
3176368994437,4cyclictest3182318-21kworker/2:009:51:582
3176368994436,5cyclictest3179979-21kworker/2:209:25:212
3176368994427,13cyclictest3467602-21systemd-journal10:25:032
317636899441,3cyclictest0-21swapper/210:07:302
317636399444,11cyclictest0-21swapper/109:59:501
3176363994425,16cyclictest3210204-21ssh10:11:061
317636399442,21cyclictest3182554-21munin-run09:10:001
3176368994335,5cyclictest3182318-21kworker/2:012:29:402
3176368994335,5cyclictest3182318-21kworker/2:012:00:412
3176368994310,19cyclictest3180967-21modprobe08:40:002
3176368994310,19cyclictest3180967-21modprobe08:39:592
317636399431,14cyclictest0-21swapper/110:43:071
317636399431,14cyclictest0-21swapper/110:43:071
3176368994234,5cyclictest3169780-21kworker/2:307:55:012
3176368994231,6cyclictest3169780-21kworker/2:307:16:352
3176368994219,17cyclictest275-21dbus-daemon09:00:012
3176363994231,7cyclictest3176825-21sendmail07:20:001
3176363994226,12cyclictest275-21dbus-daemon07:25:011
317636899415,3cyclictest131rcu_preempt10:35:012
317636899415,3cyclictest131rcu_preempt10:35:002
3176368994133,5cyclictest3182318-21kworker/2:011:59:502
3176368994132,5cyclictest3182318-21kworker/2:009:30:202
3176363994121,16cyclictest3179065-21ls08:00:011
3176373994023,12cyclictest3176477-21latency_hist07:10:023
3176368994028,10cyclictest30-21ksoftirqd/211:43:002
3176363994021,14cyclictest275-21dbus-daemon09:05:001
3176368993928,9cyclictest30-21ksoftirqd/207:15:032
3176368993928,8cyclictest30-21ksoftirqd/208:35:012
3176368993926,9cyclictest3178390-21munin-plugin-st07:50:012
3176368993926,9cyclictest3178390-21munin-plugin-st07:50:002
3176368993924,12cyclictest30-21ksoftirqd/208:05:002
3176363993926,9cyclictest48-21kcompactd008:09:031
3176363993926,10cyclictest3181994-21sshd08:59:291
3176363993925,10cyclictest3180438-21sshd08:25:281
3176363993921,14cyclictest364-21snmpd08:20:111
3176363993916,18cyclictest3177357-21modprobe07:29:591
317636899387,7cyclictest30-21ksoftirqd/207:28:302
317636899386,22cyclictest3181259-21munin-plugin-st08:45:002
3176363993825,9cyclictest3467602-21systemd-journal07:42:061
3176363993825,9cyclictest3180692-21sshd08:30:281
3176363993825,10cyclictest364-21snmpd08:02:401
3176363993825,10cyclictest3181237-21sshd08:43:281
3176363993824,9cyclictest48-21kcompactd007:36:531
3176363993824,10cyclictest23-21ksoftirqd/108:15:011
317636899376,7cyclictest131rcu_preempt09:05:002
3176368993721,11cyclictest30-21ksoftirqd/207:40:132
3176363993725,9cyclictest48-21kcompactd008:50:401
3176363993725,9cyclictest48-21kcompactd007:50:451
3176363993724,9cyclictest48-21kcompactd008:15:091
3176368993618,14cyclictest275-21dbus-daemon08:00:012
3176363993625,8cyclictest48-21kcompactd007:31:201
3176363993625,8cyclictest48-21kcompactd007:31:201
3176363993622,11cyclictest3176367-21wc07:10:021
3176363993613,19cyclictest3181012-21true08:40:011
3176363993613,19cyclictest3181012-21true08:40:011
317636899341,29cyclictest48-21kcompactd007:33:002
317636899341,29cyclictest48-21kcompactd007:32:592
3176368993324,6cyclictest3176516-21latency_hist07:10:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional