You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-13 - 06:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Jul 13, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29708449912692,22cyclictest3056780-21sh00:00:032
2970831998130,20cyclictest2991867-21ssh21:41:130
2970831998129,16cyclictest48-21kcompactd023:25:110
2970831998023,51cyclictest12-21ksoftirqd/021:36:330
2970831998011,38cyclictest0-21swapper/022:00:000
2970844997859,7cyclictest30-21ksoftirqd/221:40:012
2970831997828,9cyclictest3016479-21ssh22:34:030
2970831997741,31cyclictest48-21kcompactd022:28:530
2970831997730,14cyclictest3049749-21munin-run23:45:000
2970831997723,17cyclictest3029376-21ssh23:01:250
297084499765,5cyclictest131rcu_preempt22:45:032
297084499765,5cyclictest131rcu_preempt22:45:022
2970831997629,19cyclictest3061576-21cat00:10:020
2970831997626,18cyclictest3054526-21cat23:55:010
2970831997613,33cyclictest3059221-21cat00:05:020
2970831997512,59cyclictest0-21swapper/020:25:020
2970831997512,59cyclictest0-21swapper/020:25:020
2970831997442,4cyclictest3046529-21sshd23:38:100
2970831997442,4cyclictest3046529-21sshd23:38:100
2970831997429,12cyclictest3028728-21sed23:00:020
2970831997429,12cyclictest3028728-21sed23:00:020
2970831997345,4cyclictest131rcu_preempt00:20:000
2970831997330,39cyclictest48-21kcompactd021:30:560
2970831997327,42cyclictest0-21swapper/023:47:580
2970831997327,42cyclictest0-21swapper/023:47:570
2970831997326,10cyclictest2972333-21sendmail19:40:000
2970831997325,15cyclictest3009628-21ssh22:19:240
2970831997321,47cyclictest2994514-21ssh21:46:560
2970831997218,19cyclictest275-21dbus-daemon22:55:000
2970831997127,14cyclictest3011930-21ssh22:24:190
2970831997123,14cyclictest3005801-21ssh22:10:560
2970844997030,11cyclictest1-21systemd23:35:012
2970831997011,32cyclictest0-21swapper/000:20:100
297084499696,33cyclictest2977012-21diskmemload21:45:022
2970831996932,11cyclictest3001675-21ssh22:02:100
2970831996930,14cyclictest2986675-21latency_hist21:30:020
2970831996926,15cyclictest3033000-21ssh23:09:320
2970831996920,17cyclictest3039342-21ssh23:22:430
2970844996828,10cyclictest275-21dbus-daemon21:05:002
297083699685,14cyclictest23-21ksoftirqd/100:35:011
2970831996826,11cyclictest0-21swapper/023:13:450
2970831996825,15cyclictest2981726-21modprobe21:20:000
2970831996720,11cyclictest3037866-21ssh23:20:000
2970831996630,33cyclictest2996725-21ssh21:51:370
2970831996620,13cyclictest3062948-21ssh00:13:150
2970831996429,4cyclictest2982002-21ssh21:20:060
2970831996426,5cyclictest2973305-21latency_hist19:55:020
2970831996425,12cyclictest3043168-21ssh23:30:410
2970831996422,17cyclictest0-21swapper/021:12:200
2970831996412,23cyclictest0-21swapper/022:50:000
2970831996412,23cyclictest0-21swapper/022:50:000
297083699636,6cyclictest23-21ksoftirqd/121:25:011
2970836996315,12cyclictest23-21ksoftirqd/121:35:011
2970831996329,4cyclictest3069059-21ssh00:26:010
2970831996316,14cyclictest3072310-21ssh00:33:030
2970831996231,4cyclictest2976489-21munin-run21:00:000
2970831996229,5cyclictest3075584-21munin-run00:40:010
2970836996129,10cyclictest23-21ksoftirqd/100:20:011
2970831996140,16cyclictest2976994-21sshd21:09:090
2970831996128,4cyclictest3004398-21sshd22:08:080
2970836996022,8cyclictest23-21ksoftirqd/121:35:311
2970836996010,10cyclictest23-21ksoftirqd/123:10:011
2970831996041,16cyclictest2971494-21cat19:20:020
2970831996041,16cyclictest2971494-21cat19:20:020
2970831996020,36cyclictest2976755-21sh21:05:000
2970836995947,9cyclictest23-21ksoftirqd/123:20:021
2970831995944,11cyclictest2972059-21sshd19:33:080
2970831995916,39cyclictest0-21swapper/000:00:030
2970836995810,10cyclictest23-21ksoftirqd/100:08:481
2970831995843,12cyclictest2973610-21sshd20:00:080
2970831995843,12cyclictest2973337-21sshd19:56:310
2970831995843,11cyclictest2973879-21sshd20:09:080
2970831995842,13cyclictest2975709-21sshd20:44:080
2970831995841,14cyclictest2975417-21sshd20:36:350
2970831995841,14cyclictest2975417-21sshd20:36:340
2970831995744,10cyclictest2974368-21sshd20:16:080
2970831995743,10cyclictest3019961-21sshd22:41:090
2970831995740,14cyclictest2972831-21sshd19:46:080
2970831995728,14cyclictest3017426-21ssh22:35:460
2970831995728,14cyclictest3017426-21ssh22:35:450
2970844995648,5cyclictest2959945-21kworker/2:122:44:022
2970836995637,14cyclictest1-21systemd22:55:011
2970831995642,11cyclictest2971590-21munin-run19:25:010
2970831995621,30cyclictest2971126-21date19:15:020
2970846995530,7cyclictest37-21ksoftirqd/300:00:003
2970844995546,5cyclictest2959945-21kworker/2:123:20:432
297083699557,9cyclictest23-21ksoftirqd/123:13:061
297083699556,7cyclictest23-21ksoftirqd/123:50:511
2970836995511,13cyclictest23-21ksoftirqd/121:15:361
2970831995523,28cyclictest0-21swapper/019:45:020
2970831995519,32cyclictest0-21swapper/020:45:130
2970846995431,16cyclictest275-21dbus-daemon21:50:003
2970844995443,8cyclictest2959945-21kworker/2:100:02:362
297083699546,10cyclictest23-21ksoftirqd/100:00:021
297083699545,11cyclictest23-21ksoftirqd/122:16:511
297083699545,10cyclictest3003344-21ssh22:05:321
297083699544,45cyclictest3006092-1kworker/u9:122:15:021
2970831995439,11cyclictest2971807-21sshd19:28:080
2970831995435,16cyclictest364-21snmpd20:29:320
2970844995345,5cyclictest2964780-21kworker/2:021:23:022
2970844995344,6cyclictest3063748-21kworker/2:000:30:582
2970844995343,7cyclictest2959945-21kworker/2:123:11:252
2970844995343,7cyclictest2959945-21kworker/2:100:20:092
297083699539,11cyclictest23-21ksoftirqd/100:35:101
297083699536,8cyclictest23-21ksoftirqd/122:44:161
2970831995340,9cyclictest2974131-21sshd20:14:080
2970846995235,12cyclictest37-21ksoftirqd/321:19:263
2970844995243,6cyclictest2959945-21kworker/2:122:36:062
2970844995243,6cyclictest2959945-21kworker/2:122:36:062
2970844995243,6cyclictest2959945-21kworker/2:122:10:012
2970844995243,5cyclictest2959945-21kworker/2:122:25:242
297083699526,9cyclictest23-21ksoftirqd/123:48:491
297083699526,9cyclictest23-21ksoftirqd/123:48:491
297083699526,8cyclictest23-21ksoftirqd/100:25:161
297083699525,6cyclictest23-21ksoftirqd/121:11:161
297083699524,7cyclictest23-21ksoftirqd/122:27:271
2970831995239,10cyclictest2976191-21sshd20:50:090
2970844995143,5cyclictest2964780-21kworker/2:022:53:062
2970844995142,6cyclictest3040282-21kworker/2:200:10:092
2970844995142,6cyclictest2964780-21kworker/2:022:23:092
2970844995142,6cyclictest2959945-21kworker/2:123:02:472
2970844995130,15cyclictest3036549-21cron23:17:012
297083699517,10cyclictest23-21ksoftirqd/121:43:341
297083699517,10cyclictest23-21ksoftirqd/100:00:401
297083699515,9cyclictest23-21ksoftirqd/100:13:301
297083699514,6cyclictest23-21ksoftirqd/122:37:301
297083699514,6cyclictest23-21ksoftirqd/122:37:301
2970836995113,9cyclictest23-21ksoftirqd/122:23:091
2970846995033,10cyclictest1-21systemd00:20:013
2970844995042,5cyclictest2959945-21kworker/2:122:00:032
297083699507,13cyclictest23-21ksoftirqd/121:55:191
297083699506,8cyclictest23-21ksoftirqd/123:25:001
297083699505,6cyclictest23-21ksoftirqd/121:00:011
297083699504,15cyclictest23-21ksoftirqd/123:26:421
297083699501,5cyclictest0-21swapper/123:32:021
2970836995010,6cyclictest23-21ksoftirqd/122:57:231
2970836995010,6cyclictest23-21ksoftirqd/122:57:231
2970831995043,4cyclictest0-21swapper/019:10:010
2970831995034,11cyclictest364-21snmpd20:31:250
2970844994942,5cyclictest2959945-21kworker/2:122:03:182
2970844994941,5cyclictest2959945-21kworker/2:100:15:482
2970844994940,5cyclictest2959945-21kworker/2:122:30:222
2970844994938,7cyclictest2964780-21kworker/2:022:20:022
2970844994931,12cyclictest3467602-21systemd-journal21:51:502
2970836994937,10cyclictest3029244-21sshd23:01:081
297083699492,16cyclictest23-21ksoftirqd/120:00:001
297083699491,9cyclictest2984848-21ssh21:26:191
297083699491,7cyclictest2995811-21munin-run21:50:001
297084699485,39cyclictest150-21pr/ttyS222:46:133
297084699485,39cyclictest150-21pr/ttyS222:46:133
2970846994832,12cyclictest37-21ksoftirqd/321:52:063
2970846994832,12cyclictest3467602-21systemd-journal20:20:003
2970846994830,14cyclictest37-21ksoftirqd/322:40:003
2970846994830,14cyclictest37-21ksoftirqd/322:40:003
2970846994830,13cyclictest0-21swapper/323:05:003
2970846994824,17cyclictest3467602-21systemd-journal23:34:593
2970844994841,5cyclictest3040282-21kworker/2:223:54:092
2970844994840,5cyclictest2964780-21kworker/2:021:13:242
2970844994840,5cyclictest2959945-21kworker/2:122:58:282
2970844994840,5cyclictest2959945-21kworker/2:122:58:272
2970844994840,5cyclictest2959945-21kworker/2:100:25:452
2970844994839,6cyclictest2959945-21kworker/2:123:06:382
2970844994839,5cyclictest2964780-21kworker/2:022:13:582
297083699487,10cyclictest23-21ksoftirqd/123:40:171
297083699486,6cyclictest23-21ksoftirqd/122:01:551
297083699485,12cyclictest23-21ksoftirqd/100:20:021
2970836994839,6cyclictest221rcuc/122:47:161
2970836994839,6cyclictest221rcuc/122:47:161
297083699481,5cyclictest0-21swapper/121:51:581
297083699481,11cyclictest23-21ksoftirqd/122:30:091
2970846994729,13cyclictest275-21dbus-daemon21:15:013
2970846994712,7cyclictest0-21swapper/322:17:413
2970844994739,5cyclictest2964780-21kworker/2:019:50:022
2970844994738,6cyclictest2959945-21kworker/2:121:32:322
2970844994737,6cyclictest2964780-21kworker/2:021:16:142
2970844994732,11cyclictest3467602-21systemd-journal20:21:332
2970844994732,11cyclictest3467602-21systemd-journal20:21:332
2970844994729,13cyclictest3467602-21systemd-journal20:45:002
297083699474,13cyclictest23-21ksoftirqd/120:40:021
297083699474,13cyclictest23-21ksoftirqd/120:40:021
2970836994731,11cyclictest275-21dbus-daemon20:30:001
2970846994631,9cyclictest3028541-21cron23:00:013
2970846994631,9cyclictest3028541-21cron23:00:003
2970846994630,11cyclictest37-21ksoftirqd/321:57:223
2970844994639,5cyclictest2959945-21kworker/2:100:06:432
2970844994638,5cyclictest2964780-21kworker/2:020:35:082
2970844994638,5cyclictest2964780-21kworker/2:020:35:082
2970844994637,6cyclictest2959945-21kworker/2:123:48:302
2970844994637,6cyclictest2959945-21kworker/2:123:48:292
2970844994637,6cyclictest2959945-21kworker/2:123:44:302
2970844994637,5cyclictest3040282-21kworker/2:223:29:122
297083699466,8cyclictest23-21ksoftirqd/123:35:221
297083699466,8cyclictest23-21ksoftirqd/123:35:221
297083699461,10cyclictest23-21ksoftirqd/119:25:001
2970846994526,13cyclictest275-21dbus-daemon21:35:013
297083699453,8cyclictest23-21ksoftirqd/120:10:001
297083699453,10cyclictest23-21ksoftirqd/119:20:021
297083699453,10cyclictest23-21ksoftirqd/119:20:021
2970836994527,13cyclictest3467602-21systemd-journal19:30:021
2970846994431,10cyclictest3041033-21sshd23:26:093
2970846994429,8cyclictest0-21swapper/319:45:023
2970846994429,12cyclictest37-21ksoftirqd/300:14:563
2970846994426,13cyclictest3037859-21cron23:20:003
2970846994426,13cyclictest275-21dbus-daemon20:40:003
2970846994426,13cyclictest275-21dbus-daemon20:40:003
297084699442,10cyclictest3059289-21latency_hist00:05:033
2970844994436,5cyclictest2964780-21kworker/2:021:25:082
2970844994436,5cyclictest2964780-21kworker/2:020:30:022
297083699446,8cyclictest23-21ksoftirqd/119:35:001
297083699445,10cyclictest23-21ksoftirqd/120:05:001
2970836994433,8cyclictest23-21ksoftirqd/119:50:001
2970836994429,10cyclictest1-21systemd19:45:021
2970846994328,12cyclictest37-21ksoftirqd/322:00:493
2970846994324,13cyclictest275-21dbus-daemon19:50:003
2970846994320,9cyclictest37-21ksoftirqd/321:20:493
2970844994335,5cyclictest2964780-21kworker/2:020:07:332
2970844994335,5cyclictest2959945-21kworker/2:120:50:022
2970844994335,5cyclictest2959945-21kworker/2:100:39:292
2970844994324,14cyclictest3467602-21systemd-journal20:00:012
2970844994322,14cyclictest275-21dbus-daemon21:00:002
297083699436,7cyclictest23-21ksoftirqd/120:55:021
297083699431,7cyclictest2975977-21munin-run20:50:001
2970846994229,9cyclictest37-21ksoftirqd/320:50:013
2970846994229,10cyclictest37-21ksoftirqd/321:36:483
2970846994227,12cyclictest37-21ksoftirqd/323:51:063
2970846994227,11cyclictest3061556-21sed00:10:023
2970844994234,5cyclictest2964780-21kworker/2:021:45:252
2970844994234,5cyclictest2964780-21kworker/2:021:10:032
297083699424,7cyclictest23-21ksoftirqd/120:13:081
2970836994213,7cyclictest0-21swapper/121:10:011
297084699419,7cyclictest131rcu_preempt21:00:013
2970846994129,9cyclictest2973879-21sshd20:09:093
2970846994128,10cyclictest37-21ksoftirqd/322:44:493
2970846994127,11cyclictest37-21ksoftirqd/322:22:383
2970844994133,5cyclictest3040282-21kworker/2:223:35:082
2970844994133,5cyclictest3040282-21kworker/2:223:35:072
2970844994124,12cyclictest30-21ksoftirqd/220:55:012
2970836994132,5cyclictest2972333-21sendmail19:40:001
297083699412,8cyclictest23-21ksoftirqd/119:50:021
297084699407,7cyclictest37-21ksoftirqd/322:30:013
297084699403,6cyclictest131rcu_preempt21:10:003
2970846994029,8cyclictest37-21ksoftirqd/323:40:023
2970846994029,8cyclictest37-21ksoftirqd/323:40:013
2970846994028,9cyclictest37-21ksoftirqd/320:05:013
2970846994028,9cyclictest3467602-21systemd-journal19:31:273
2970846994028,10cyclictest37-21ksoftirqd/300:35:203
2970846994026,11cyclictest37-21ksoftirqd/321:05:003
2970846994026,11cyclictest37-21ksoftirqd/320:00:023
2970846994025,12cyclictest2993636-21sed21:45:023
297083699409,5cyclictest0-21swapper/119:10:581
297083699406,7cyclictest23-21ksoftirqd/120:31:341
297083699401,7cyclictest2974680-21modprobe20:25:001
297083699401,7cyclictest2974680-21modprobe20:25:001
2970846993928,9cyclictest37-21ksoftirqd/322:33:093
2970846993928,8cyclictest37-21ksoftirqd/323:50:013
2970846993928,8cyclictest37-21ksoftirqd/323:50:003
2970846993927,10cyclictest37-21ksoftirqd/321:27:003
297084699392,5cyclictest131rcu_preempt23:06:023
2970846993923,11cyclictest37-21ksoftirqd/300:20:343
2970836993932,5cyclictest23-21ksoftirqd/121:04:081
2970846993824,11cyclictest37-21ksoftirqd/323:44:513
297084699381,5cyclictest131rcu_preempt00:26:553
2970844993830,5cyclictest2959945-21kworker/2:119:12:092
2970844993820,13cyclictest1-21systemd20:05:012
2970836993831,5cyclictest2970842-21ls19:10:011
297084699374,4cyclictest131rcu_preempt00:30:153
2970846993727,7cyclictest37-21ksoftirqd/320:15:023
2970846993725,10cyclictest37-21ksoftirqd/323:11:143
2970846993724,9cyclictest3007695-21latency_hist22:15:023
2970846993724,10cyclictest3004714-21ssh22:08:473
2970846993723,11cyclictest37-21ksoftirqd/320:25:023
2970846993723,11cyclictest37-21ksoftirqd/320:25:013
2970846993722,12cyclictest37-21ksoftirqd/323:21:593
2970844993729,5cyclictest2964780-21kworker/2:019:53:422
297084499371,10cyclictest2971786-21latency_hist19:25:022
297083699373,9cyclictest23-21ksoftirqd/120:44:091
297084699361,5cyclictest131rcu_preempt22:52:363
297084499365,5cyclictest131rcu_preempt19:18:382
297084499365,5cyclictest131rcu_preempt19:18:382
297084499363,4cyclictest131rcu_preempt19:40:012
2970844993625,8cyclictest2970987-21sed19:10:022
297083699362,7cyclictest23-21ksoftirqd/120:16:321
2970846993519,12cyclictest37-21ksoftirqd/319:40:023
2970846993422,9cyclictest37-21ksoftirqd/320:50:023
297084699341,8cyclictest2975115-21latency_hist20:30:023
2970844993425,7cyclictest30-21ksoftirqd/219:45:022
2970844993421,9cyclictest3467602-21systemd-journal20:30:002
2970844993420,11cyclictest30-21ksoftirqd/220:13:292
2970846993326,5cyclictest2971704-21latency_hist19:25:023
2970846993324,5cyclictest2972036-21latency_hist19:30:023
2970846993321,9cyclictest37-21ksoftirqd/319:15:023
2970846993320,10cyclictest275-21dbus-daemon19:20:013
2970846993320,10cyclictest275-21dbus-daemon19:20:003
2970844993324,7cyclictest30-21ksoftirqd/220:20:022
2970846993220,8cyclictest37-21ksoftirqd/319:55:023
2970846993119,10cyclictest37-21ksoftirqd/320:45:023
2970844993118,11cyclictest30-21ksoftirqd/219:35:002
2970846993020,8cyclictest37-21ksoftirqd/320:35:013
2970844993015,11cyclictest275-21dbus-daemon19:30:012
2970846992817,7cyclictest2970959-21wc19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional