You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 09:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat May 23, 2026 00:45:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3356315991281,4cyclictest0-21swapper/021:25:020
3356315998984,3cyclictest1250ksoftirqd/021:40:020
335631599823,13cyclictest3436301-21sh23:39:590
3356323998121,19cyclictest3445514-21ssh23:58:522
3356323998020,20cyclictest3425288-21ssh23:16:582
335632399787,55cyclictest3417051-21ssh23:00:012
3356323997720,19cyclictest3373222-21ssh21:31:482
3356323997719,21cyclictest3397426-21rm22:19:572
3356323997620,18cyclictest3438628-21ssh23:44:432
3356323997620,18cyclictest3438628-21ssh23:44:432
3356323997619,19cyclictest3391373-21ssh22:07:082
3356323997518,16cyclictest3410966-21ssh22:47:212
3356315997515,34cyclictest16850irq/134-800000000:00:000
3356323997416,16cyclictest3453950-21ssh00:15:562
3356315997428,12cyclictest3399399-21ssh22:23:540
3356315997427,13cyclictest3461437-21ssh00:31:330
3356323997320,15cyclictest3446373-21ssh00:00:142
3356323997320,15cyclictest3446373-21ssh00:00:132
3356323997319,17cyclictest3402245-21ssh22:29:422
3356315997369,2cyclictest0-21swapper/023:55:020
3356323997212,21cyclictest3460598-21latency_hist00:30:012
3356323997120,14cyclictest0-21swapper/200:10:102
3356323997120,14cyclictest0-21swapper/200:10:102
3356323997113,20cyclictest3389394-21ssh22:03:012
3356323996916,16cyclictest3443307-21ssh23:54:172
3356323996916,15cyclictest3419324-21ssh23:04:492
3356323996916,15cyclictest3419324-21ssh23:04:492
3356323996916,15cyclictest3405630-21ssh22:36:232
3356323996913,18cyclictest3371298-21ssh21:27:562
3356323996822,10cyclictest0-21swapper/221:51:452
335631899688,44cyclictest3423701-21kworker/u8:023:20:021
3356323996725,8cyclictest3404070-21ssh22:33:172
3356323996723,11cyclictest3365457-21ssh21:15:442
3356323996718,14cyclictest3464568-21ssh00:38:142
3356323996717,15cyclictest3435136-21ssh23:37:112
3356323996717,15cyclictest3430410-21ssh23:27:352
3356323996717,15cyclictest3419913-21ssh23:05:432
3356323996616,15cyclictest3368937-21ssh21:23:002
3356323996616,14cyclictest3376266-21ssh21:38:182
3356323996518,13cyclictest3440319-21ssh23:48:042
3356323996517,14cyclictest3378977-21ssh21:43:542
3356323996517,13cyclictest3428354-21ssh23:23:222
3356323996516,13cyclictest3380007-21ssh21:45:432
335631599651,61cyclictest0-21swapper/020:40:010
3356323996219,15cyclictest3362497-21diskmemload22:12:292
3356323996218,14cyclictest3433132-21ssh23:33:142
3356323996217,14cyclictest3357855-21munin-run19:40:012
3356323996217,13cyclictest3456807-21rm00:22:002
3356323996215,14cyclictest3363498-21ssh21:11:452
3356323996215,14cyclictest3363498-21ssh21:11:452
3356323996214,12cyclictest3385754-21sshd21:56:422
3356315996157,2cyclictest0-21swapper/023:10:020
3356323995919,10cyclictest0-21swapper/223:14:372
3356323995919,10cyclictest0-21swapper/223:14:362
3356315995938,15cyclictest1-21systemd20:20:010
3356323995844,11cyclictest3408768-21ssh22:42:472
3356323995844,11cyclictest3408768-21ssh22:42:472
3356318995841,13cyclictest1-21systemd00:00:021
3356315995824,30cyclictest12-21ksoftirqd/023:16:190
3356323995740,14cyclictest3461800-21ssh00:32:192
3356323995740,13cyclictest3413796-21ssh22:53:162
3356323995740,13cyclictest3413796-21ssh22:53:162
3356323995740,13cyclictest3398691-21ssh22:22:132
3356315995728,14cyclictest3458865-21ssh00:26:120
3356323995640,13cyclictest3448793-21ssh00:05:152
335631599562,3cyclictest131rcu_preempt22:29:120
335631599562,38cyclictest3389891-21ssh22:04:140
335632399554,40cyclictest291rcuc/221:10:022
3356323995540,10cyclictest3358358-21munin-run19:50:002
3356323995525,24cyclictest311-21rs:main2
3356323995525,24cyclictest311-21rs:main2
3356315995528,12cyclictest3457782-21ssh00:24:160
3356315995527,12cyclictest3372575-21ssh21:30:190
3356327995427,15cyclictest3436306-21munin-plugin-st23:39:593
3356323995441,10cyclictest3360911-21sshd20:39:072
3356323995439,12cyclictest3357648-21cat19:35:022
3356323995438,12cyclictest3360145-21sshd20:21:192
3356315995435,15cyclictest3362497-21diskmemload21:10:000
3356315995434,15cyclictest3397496-21tr22:20:000
3356315995417,32cyclictest16850irq/134-800000021:46:320
3356327995337,12cyclictest1-21systemd00:10:003
3356323995339,11cyclictest3361691-21sshd20:53:072
3356323995339,10cyclictest3356753-21sshd19:18:072
3356315995326,12cyclictest3463757-21ssh00:36:260
3356315995326,11cyclictest3391659-21ssh22:07:450
335631599531,34cyclictest12-21ksoftirqd/021:57:040
3356323995240,9cyclictest3360640-21sshd20:30:072
3356323995240,9cyclictest3360640-21sshd20:30:072
3356323995238,9cyclictest48-21kcompactd020:08:302
3356323995238,9cyclictest48-21kcompactd020:08:302
3356315995236,12cyclictest12-21ksoftirqd/021:40:040
3356315995228,9cyclictest3422466-21ssh23:11:040
3356315995228,9cyclictest3422466-21ssh23:11:040
3356315995226,11cyclictest3426879-21ssh23:20:030
3356315995225,12cyclictest3453141-21ssh00:14:360
3356315995225,12cyclictest3453141-21ssh00:14:360
3356315995224,13cyclictest3450490-21ssh00:09:060
3356315995222,15cyclictest3432059-21ssh23:30:500
3356327995135,12cyclictest1-21systemd20:15:003
3356327995130,16cyclictest275-21dbus-daemon00:20:003
3356323995139,9cyclictest3359143-21munin-run20:05:012
3356323995138,10cyclictest364-21snmpd19:51:092
335631599513,31cyclictest19350irq/34-mmc120:50:020
335631599513,31cyclictest19350irq/34-mmc120:50:020
335631599512,45cyclictest3407263-21munin-run22:40:010
335631599511,4cyclictest131rcu_preempt00:04:030
335631599511,4cyclictest131rcu_preempt00:04:030
3356323995038,9cyclictest364-21snmpd19:11:012
3356323995035,12cyclictest3358893-21pluginstate20:00:022
3356323995029,16cyclictest275-21dbus-daemon21:00:002
3356315995027,8cyclictest12-21ksoftirqd/021:15:020
3356315995027,8cyclictest12-21ksoftirqd/021:15:020
3356315995026,9cyclictest3409614-21ssh22:44:480
3356315995026,9cyclictest3409614-21ssh22:44:480
3356315995022,12cyclictest3370485-21ssh21:26:040
3356315995013,3cyclictest131rcu_preempt23:28:590
3356327994935,10cyclictest1-21systemd19:55:013
3356327994912,28cyclictest3429221-21ssh23:25:033
3356323994935,10cyclictest3467602-21systemd-journal20:14:482
3356318994936,10cyclictest23-21ksoftirqd/122:45:131
3356318994935,11cyclictest3398242-21sshd22:21:071
3356318994932,13cyclictest284-21systemd-logind21:57:051
3356318994918,6cyclictest23-21ksoftirqd/123:48:591
3356315994940,5cyclictest3360424-21munin-plugin-st20:30:000
3356315994929,15cyclictest3412526-21ssh22:50:260
3356315994929,15cyclictest3412526-21ssh22:50:250
3356327994836,9cyclictest3467602-21systemd-journal00:35:013
3356327994831,12cyclictest3402395-21cron22:30:003
3356323994835,10cyclictest364-21snmpd20:27:412
3356323994830,14cyclictest364-21snmpd19:21:302
335631899484,3cyclictest131rcu_preempt00:07:491
3356318994836,6cyclictest0-21swapper/119:10:011
3356315994832,13cyclictest3365154-21ssh21:15:060
3356315994824,10cyclictest3393213-21ssh22:10:490
3356315994821,23cyclictest2186142-1kworker/0:1H20:10:020
3356315994821,23cyclictest2186142-1kworker/0:1H20:10:010
3356327994735,9cyclictest275-21dbus-daemon21:30:003
3356327994729,13cyclictest3446019-21sh00:00:003
3356327994715,7cyclictest37-21ksoftirqd/322:33:213
335632799471,3cyclictest131rcu_preempt21:56:483
3356327994712,6cyclictest131rcu_preempt22:06:193
3356323994735,9cyclictest364-21snmpd20:48:452
3356323994735,9cyclictest364-21snmpd20:48:452
3356323994728,15cyclictest364-21snmpd20:17:382
3356323994723,20cyclictest3361221-21munin-run20:45:022
3356318994737,6cyclictest23-21ksoftirqd/122:25:041
3356315994730,12cyclictest3358093-21munin-run19:45:000
3356315994730,12cyclictest3358093-21munin-run19:45:000
335631599472,14cyclictest3438508-21ssh23:44:210
335631599472,14cyclictest3438508-21ssh23:44:210
3356327994631,10cyclictest309-21in:imuxsock20:35:033
3356327994621,20cyclictest37-21ksoftirqd/323:30:183
335631899465,4cyclictest131rcu_preempt21:29:071
335631599464,4cyclictest131rcu_preempt20:45:010
3356315994631,12cyclictest3418019-21ssh23:01:490
3356315994631,12cyclictest3418019-21ssh23:01:480
3356315994631,10cyclictest12-21ksoftirqd/022:30:470
3356315994619,12cyclictest3416115-21ssh22:58:040
335631599461,3cyclictest131rcu_preempt21:00:000
3356315994610,32cyclictest16850irq/134-800000000:18:160
3356327994533,8cyclictest1-21systemd20:35:013
3356327994533,8cyclictest1-21systemd20:35:013
335632799452,3cyclictest131rcu_preempt23:53:123
3356323994526,16cyclictest3357295-21sendmail-mta19:29:442
3356318994535,7cyclictest3382740-21sshd21:51:331
3356318994532,9cyclictest3467602-21systemd-journal20:31:201
3356318994532,9cyclictest3467602-21systemd-journal20:31:201
3356318994529,11cyclictest23-21ksoftirqd/121:31:301
3356318994516,6cyclictest131rcu_preempt22:35:021
3356315994527,14cyclictest3439731-21ssh23:46:430
3356315994526,15cyclictest3357788-21sshd19:37:070
335631599451,3cyclictest0-21swapper/019:20:000
3356327994428,13cyclictest3407420-21sed22:40:023
3356323994430,11cyclictest364-21snmpd21:01:242
3356323994430,11cyclictest364-21snmpd21:01:242
3356318994429,11cyclictest3467602-21systemd-journal19:30:001
3356318994427,13cyclictest23-21ksoftirqd/100:25:011
3356318994426,14cyclictest3467602-21systemd-journal21:10:011
335631899441,3cyclictest131rcu_preempt23:23:361
335631899441,3cyclictest131rcu_preempt00:17:131
335631599444,32cyclictest0-21swapper/021:52:170
3356315994425,15cyclictest3410872-21ssh22:47:090
335632799433,3cyclictest131rcu_preempt22:52:213
335632799433,3cyclictest131rcu_preempt22:52:213
3356327994330,8cyclictest37-21ksoftirqd/320:20:003
3356327994327,12cyclictest275-21dbus-daemon23:15:023
3356327994327,12cyclictest275-21dbus-daemon23:15:013
3356318994331,9cyclictest3467602-21systemd-journal21:41:321
3356318994328,10cyclictest3467602-21systemd-journal19:16:141
3356318994327,12cyclictest275-21dbus-daemon19:45:011
3356318994327,12cyclictest275-21dbus-daemon19:45:011
335631899432,3cyclictest131rcu_preempt21:35:191
3356318994318,6cyclictest23-21ksoftirqd/123:33:221
3356318994311,22cyclictest3436297-21modprobe23:39:591
3356315994311,4cyclictest3360800-21sed20:35:020
3356315994311,4cyclictest3360800-21sed20:35:020
335632799424,27cyclictest3377190-21ssh21:40:033
3356327994230,9cyclictest3421821-21/usr/sbin/munin23:10:013
3356327994225,12cyclictest3441412-21sort23:50:023
3356327994223,13cyclictest275-21dbus-daemon20:05:013
335631899425,5cyclictest131rcu_preempt23:05:001
335631899425,5cyclictest131rcu_preempt23:05:001
3356318994233,6cyclictest23-21ksoftirqd/100:28:561
3356318994231,8cyclictest3360897-21sshd20:36:071
3356318994229,10cyclictest23-21ksoftirqd/119:55:021
3356318994228,8cyclictest309-21in:imuxsock23:30:001
3356318994226,10cyclictest3446293-21date00:00:041
3356318994226,10cyclictest3446293-21date00:00:041
335631899422,3cyclictest131rcu_preempt21:18:141
335631899421,3cyclictest131rcu_preempt00:12:211
335631899421,3cyclictest131rcu_preempt00:12:211
3356318994211,6cyclictest131rcu_preempt22:41:421
3356318994211,6cyclictest131rcu_preempt22:41:411
335631599421,37cyclictest3361693-21sshd20:53:070
3356327994128,10cyclictest3357957-21sed19:40:023
3356327994125,12cyclictest3467602-21systemd-journal22:44:073
3356327994125,12cyclictest3467602-21systemd-journal22:44:073
3356327994125,12cyclictest275-21dbus-daemon21:35:013
3356318994129,9cyclictest3404863-21sed22:35:011
3356318994129,10cyclictest23-21ksoftirqd/122:51:071
3356318994129,10cyclictest23-21ksoftirqd/122:51:071
3356318994126,12cyclictest3467602-21systemd-journal21:00:021
3356318994126,12cyclictest3467602-21systemd-journal21:00:021
3356318994125,11cyclictest23-21ksoftirqd/120:00:011
3356318994124,13cyclictest3467602-21systemd-journal20:50:011
3356318994124,13cyclictest3467602-21systemd-journal20:50:011
3356318994122,14cyclictest1-21systemd22:10:011
3356315994129,8cyclictest3356487-21sshd19:11:130
3356315994124,13cyclictest3357544-21sshd19:34:070
335632799404,3cyclictest131rcu_preempt22:45:533
335632799404,3cyclictest131rcu_preempt20:45:023
335632799402,3cyclictest131rcu_preempt21:24:593
3356327994023,13cyclictest3467602-21systemd-journal21:10:013
3356327994016,10cyclictest0-21swapper/323:27:083
335632799401,3cyclictest131rcu_preempt00:02:393
335632799401,3cyclictest131rcu_preempt00:02:383
3356323994016,5cyclictest3348999-21kworker/2:119:10:012
335631899404,3cyclictest0-21swapper/123:06:121
3356318994028,9cyclictest23-21ksoftirqd/100:33:341
3356318994028,8cyclictest23-21ksoftirqd/122:20:001
3356318994027,4cyclictest23-21ksoftirqd/123:13:071
3356318994027,4cyclictest23-21ksoftirqd/123:13:071
3356318994026,10cyclictest23-21ksoftirqd/122:55:571
3356318994013,5cyclictest131rcu_preempt00:35:031
3356327993924,12cyclictest37-21ksoftirqd/322:16:313
3356327993915,20cyclictest37-21ksoftirqd/321:12:273
3356327993915,20cyclictest37-21ksoftirqd/321:12:273
335632799391,3cyclictest131rcu_preempt21:45:363
335631899393,3cyclictest131rcu_preempt22:14:051
3356318993927,9cyclictest23-21ksoftirqd/120:40:021
3356318993926,10cyclictest23-21ksoftirqd/119:50:001
3356318993925,9cyclictest3364308-1kworker/u9:121:15:001
3356318993925,9cyclictest3364308-1kworker/u9:121:15:001
335631599399,6cyclictest131rcu_preempt20:00:020
3356315993932,4cyclictest16950irq/224-800000019:22:060
3356315993928,8cyclictest12-21ksoftirqd/021:05:020
3356315993928,8cyclictest12-21ksoftirqd/021:05:020
335632799382,8cyclictest3419088-21ssh23:04:203
335632799382,8cyclictest3419088-21ssh23:04:203
3356327993827,9cyclictest37-21ksoftirqd/321:17:093
3356327993826,7cyclictest3358836-21cron20:00:003
3356327993826,10cyclictest37-21ksoftirqd/319:45:113
335632799381,4cyclictest131rcu_preempt00:26:043
335632799381,3cyclictest131rcu_preempt00:35:103
3356318993826,7cyclictest309-21in:imuxsock21:25:001
3356318993817,14cyclictest275-21dbus-daemon23:45:011
3356318993817,14cyclictest275-21dbus-daemon23:45:011
335631899381,33cyclictest0-21swapper/120:55:011
335632799377,20cyclictest0-21swapper/321:41:343
3356327993729,4cyclictest309-21in:imuxsock22:10:043
3356327993727,6cyclictest3361951-21cron21:00:003
3356327993726,8cyclictest37-21ksoftirqd/322:01:183
3356327993725,9cyclictest3467602-21systemd-journal20:50:003
3356327993725,9cyclictest3467602-21systemd-journal20:50:003
3356327993721,12cyclictest3467602-21systemd-journal20:09:073
3356327993721,12cyclictest3467602-21systemd-journal20:09:073
3356327993711,16cyclictest37-21ksoftirqd/300:24:543
335631899379,6cyclictest131rcu_preempt21:50:001
335631899373,3cyclictest131rcu_preempt22:01:281
3356318993728,5cyclictest3357690-21latency_hist19:35:021
3356318993725,9cyclictest23-21ksoftirqd/123:55:021
3356318993723,10cyclictest275-21dbus-daemon20:10:011
3356318993723,10cyclictest275-21dbus-daemon20:10:011
3356315993729,6cyclictest12-21ksoftirqd/020:15:020
3356315993724,9cyclictest12-21ksoftirqd/019:55:020
3356315993719,14cyclictest275-21dbus-daemon19:30:010
335632799363,22cyclictest3362497-21diskmemload23:15:263
3356327993627,5cyclictest309-21in:imuxsock20:25:013
3356327993623,10cyclictest37-21ksoftirqd/323:41:503
3356327993623,10cyclictest37-21ksoftirqd/323:41:503
3356327993620,12cyclictest37-21ksoftirqd/321:05:023
3356327993620,12cyclictest37-21ksoftirqd/321:05:023
335632799361,29cyclictest3357868-21kworker/3:119:45:023
335632799361,29cyclictest3357868-21kworker/3:119:45:023
335631899362,8cyclictest131rcu_preempt20:25:001
3356318993620,12cyclictest3359297-21sed20:05:021
3356318993620,12cyclictest1-21systemd20:30:011
335631599361,31cyclictest3356407-21ls19:10:000
335632799353,3cyclictest131rcu_preempt22:24:403
335632799352,6cyclictest3414831-21ssh22:55:073
3356327993523,10cyclictest37-21ksoftirqd/300:15:023
3356327993523,10cyclictest37-21ksoftirqd/300:15:013
3356327993522,10cyclictest309-21in:imuxsock21:55:013
3356327993521,11cyclictest3356344-21date19:10:013
3356327993519,12cyclictest37-21ksoftirqd/319:15:013
335631899355,20cyclictest3359889-21modprobe20:20:001
335631899351,8cyclictest131rcu_preempt19:25:011
335631599352,3cyclictest131rcu_preempt20:05:010
3356315993520,11cyclictest1-21systemd19:50:000
3356315993519,12cyclictest1-21systemd20:25:010
3356327993422,10cyclictest37-21ksoftirqd/319:20:013
3356327993419,11cyclictest364-21snmpd19:31:453
3356318993420,10cyclictest275-21dbus-daemon19:14:591
335632799331,6cyclictest3357055-21munin-run19:25:013
335631899335,5cyclictest131rcu_preempt19:40:021
3356327993222,6cyclictest3361711-21munin-plugin-st20:55:003
3356327993220,10cyclictest37-21ksoftirqd/320:30:013
335632799315,3cyclictest131rcu_preempt19:26:153
335631899315,5cyclictest131rcu_preempt20:56:071
335631899314,3cyclictest131rcu_preempt20:15:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional