You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-12 - 04:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sun Apr 12, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2940266999326,15cyclictest2947773-21ssh21:12:500
2940266999326,15cyclictest2947773-21ssh21:12:490
2940266998928,13cyclictest3019202-21ssh23:58:190
2940266998728,12cyclictest3014472-21ssh23:47:140
2940266998413,67cyclictest2998466-21ssh23:10:020
2940266998328,12cyclictest3021759-21ssh00:04:180
2940266998325,6cyclictest3000769-21ssh23:15:160
2940266998225,14cyclictest3037016-21rm00:39:430
2940266998127,12cyclictest3023199-21ssh00:07:350
2940266997924,13cyclictest3008993-21ssh23:34:460
2940266997824,8cyclictest3016535-21ssh23:51:540
294026699761,38cyclictest3033406-21ssh00:31:050
2940266997522,8cyclictest2977326-21ssh22:21:010
2940266997426,11cyclictest2995349-21ssh23:02:520
2940266997424,7cyclictest3005443-21ssh23:26:110
294026699733,60cyclictest2968049-21modprobe22:00:000
2940266997335,34cyclictest16950irq/224-800000000:13:210
2940266997227,15cyclictest2974181-21ssh22:14:120
2940266997227,12cyclictest3030601-21ssh00:24:550
294026699722,5cyclictest3467602-21systemd-journal22:36:370
294026699722,36cyclictest2975014-21ssh22:15:410
2940276997122,26cyclictest37-21ksoftirqd/322:00:023
2940266997129,13cyclictest3032231-21ssh00:28:320
2940266997127,12cyclictest2964028-21ssh21:50:110
2940266997027,15cyclictest2963348-21ssh21:48:570
2940266997011,23cyclictest3010026-21ssh23:36:490
2940266996821,12cyclictest3012729-21ssh23:43:130
2940266996820,13cyclictest2993507-21ssh22:58:460
2940266996721,10cyclictest0-21swapper/021:19:250
2940266996626,35cyclictest16850irq/134-800000021:30:010
2940266996624,9cyclictest3027567-21ssh00:17:400
2940266996617,12cyclictest2960561-21ssh21:42:250
2940266996525,7cyclictest2989053-21ssh22:48:240
2940266996520,39cyclictest16850irq/134-800000022:06:070
2940266996519,13cyclictest3004552-21ssh23:24:250
2940266996424,5cyclictest2945912-21sendmail21:00:000
2940266996424,10cyclictest2980739-21ssh22:29:190
2940266996418,17cyclictest16850irq/134-800000022:30:150
294027099635,45cyclictest3009128-21ssh23:35:021
2940266996321,10cyclictest2953024-21latency_hist21:25:020
2940266996321,10cyclictest2953024-21latency_hist21:25:010
294026699622,8cyclictest1-21systemd19:51:500
2940266996227,8cyclictest2958099-21ssh21:36:400
2940266996226,9cyclictest2999077-21ssh23:11:300
294026699613,6cyclictest364-21snmpd22:03:320
2940266996135,23cyclictest16950irq/224-800000022:41:210
2940272995841,12cyclictest3467602-21systemd-journal23:05:002
294026699581,28cyclictest131rcu_preempt21:33:550
294027299574,7cyclictest131rcu_preempt23:25:012
2940266995735,18cyclictest16950irq/224-800000022:50:350
2940266995727,6cyclictest2940767-21latency_hist19:20:010
2940266995727,6cyclictest2940767-21latency_hist19:20:010
2940272995647,6cyclictest2942253-21kworker/2:021:32:452
2940276995538,12cyclictest1-21systemd21:00:003
294027699552,4cyclictest131rcu_preempt23:12:023
2940272995523,22cyclictest275-21dbus-daemon20:35:002
2940276995437,12cyclictest1-21systemd22:05:013
2940272995445,6cyclictest2942253-21kworker/2:022:32:412
2940272995445,6cyclictest2939643-21kworker/2:321:20:082
2940272995445,6cyclictest2939643-21kworker/2:321:20:082
2940266995423,7cyclictest12-21ksoftirqd/021:10:010
2940266995420,10cyclictest2941218-21sshd19:25:500
2940272995345,5cyclictest2942253-21kworker/2:022:53:372
2940272995345,5cyclictest2939643-21kworker/2:323:53:282
2940272995344,6cyclictest2942253-21kworker/2:022:49:362
2940272995344,6cyclictest2939643-21kworker/2:300:24:262
294027299533,7cyclictest30-21ksoftirqd/200:06:352
2940272995333,14cyclictest3467602-21systemd-journal19:15:002
294027299533,17cyclictest3032904-21grep00:30:022
2940272995330,17cyclictest275-21dbus-daemon00:00:062
2940266995335,14cyclictest1-21systemd19:45:010
294027699522,3cyclictest131rcu_preempt00:31:303
2940272995244,5cyclictest2942253-21kworker/2:022:55:462
2940272995244,5cyclictest2942253-21kworker/2:021:37:212
2940272995244,5cyclictest2939643-21kworker/2:323:43:392
2940272995244,5cyclictest2939643-21kworker/2:323:33:342
2940272995243,6cyclictest2942253-21kworker/2:021:11:342
2940272995243,6cyclictest2942253-21kworker/2:021:11:332
2940272995243,5cyclictest2942253-21kworker/2:023:08:572
2940272995242,6cyclictest2942253-21kworker/2:022:44:302
2940270995234,13cyclictest275-21dbus-daemon21:25:011
2940270995234,13cyclictest275-21dbus-daemon21:25:011
2940272995143,5cyclictest2942253-21kworker/2:023:15:062
2940272995143,5cyclictest2942253-21kworker/2:023:14:282
2940272995143,5cyclictest2942253-21kworker/2:021:55:152
2940272995143,5cyclictest2939643-21kworker/2:323:55:192
2940272995143,5cyclictest2939643-21kworker/2:323:46:382
2940272995143,5cyclictest2939643-21kworker/2:300:34:072
2940272995142,5cyclictest2942253-21kworker/2:021:25:492
2940272995140,7cyclictest2942253-21kworker/2:022:29:282
2940272995121,5cyclictest131rcu_preempt00:40:002
2940276995019,12cyclictest37-21ksoftirqd/320:17:533
2940272995042,5cyclictest2942253-21kworker/2:022:00:162
2940272995042,5cyclictest2939643-21kworker/2:321:18:262
2940272995042,5cyclictest2939643-21kworker/2:319:30:022
2940272995041,6cyclictest2942253-21kworker/2:021:53:332
2940272995041,5cyclictest2942253-21kworker/2:021:41:042
2940272995040,6cyclictest2942253-21kworker/2:022:10:412
2940272995040,6cyclictest2942253-21kworker/2:021:50:002
2940270995035,11cyclictest1-21systemd22:30:011
2940270995015,23cyclictest0-21swapper/121:32:371
2940276994925,6cyclictest37-21ksoftirqd/323:05:013
294027299496,6cyclictest131rcu_preempt19:50:002
2940272994941,5cyclictest2942253-21kworker/2:022:37:192
2940272994941,5cyclictest2942253-21kworker/2:022:18:202
2940272994941,5cyclictest2942253-21kworker/2:022:07:232
2940272994941,5cyclictest2942253-21kworker/2:021:03:522
2940272994941,5cyclictest2942253-21kworker/2:020:48:532
2940272994941,5cyclictest2939643-21kworker/2:323:28:162
2940272994941,5cyclictest2939643-21kworker/2:319:57:072
2940272994941,5cyclictest2939643-21kworker/2:300:17:092
2940272994940,6cyclictest2939643-21kworker/2:319:44:322
2940272994939,6cyclictest2942253-21kworker/2:022:22:052
2940272994931,13cyclictest1-21systemd20:10:022
294027099495,4cyclictest131rcu_preempt23:50:011
2940276994834,10cyclictest0-21swapper/322:55:023
2940276994832,13cyclictest1-21systemd21:55:013
2940272994840,5cyclictest2942253-21kworker/2:021:09:532
2940272994840,5cyclictest2939643-21kworker/2:323:35:432
2940272994840,5cyclictest2939643-21kworker/2:320:00:072
2940272994840,5cyclictest2939643-21kworker/2:319:20:162
2940272994840,5cyclictest2939643-21kworker/2:300:10:232
2940276994733,10cyclictest275-21dbus-daemon00:00:013
294027699471,3cyclictest131rcu_preempt22:25:013
2940272994739,5cyclictest2939643-21kworker/2:320:37:352
2940272994739,5cyclictest2939643-21kworker/2:320:25:412
2940272994738,6cyclictest2939643-21kworker/2:319:54:012
294026699472,7cyclictest2943133-21latency_hist20:05:010
294027699463,3cyclictest131rcu_preempt00:30:023
294027699463,3cyclictest131rcu_preempt00:22:413
2940276994630,13cyclictest37-21ksoftirqd/321:25:013
2940276994630,13cyclictest37-21ksoftirqd/321:25:013
2940276994630,13cyclictest1-21systemd22:15:003
2940272994639,4cyclictest2939643-21kworker/2:319:25:552
2940272994638,5cyclictest2942253-21kworker/2:020:55:262
2940272994638,5cyclictest2942253-21kworker/2:020:52:262
2940272994638,5cyclictest2942253-21kworker/2:020:41:312
2940272994638,5cyclictest2939643-21kworker/2:319:37:122
2940272994638,5cyclictest2939643-21kworker/2:319:17:172
2940272994638,5cyclictest2939643-21kworker/2:319:17:172
2940270994632,10cyclictest23-21ksoftirqd/123:30:001
2940270994628,12cyclictest275-21dbus-daemon19:20:011
2940270994628,12cyclictest275-21dbus-daemon19:20:011
2940266994637,5cyclictest16850irq/134-800000019:35:000
2940266994632,9cyclictest0-21swapper/020:15:020
2940266994629,13cyclictest2944875-21cat20:40:010
294027699454,3cyclictest131rcu_preempt22:05:403
294027699453,5cyclictest131rcu_preempt21:36:553
2940276994531,10cyclictest3467602-21systemd-journal19:25:003
294027699451,3cyclictest131rcu_preempt21:11:323
294027699451,3cyclictest131rcu_preempt21:11:323
2940272994538,4cyclictest2939643-21kworker/2:320:17:562
2940272994537,5cyclictest2939643-21kworker/2:320:23:552
294027099457,6cyclictest3019903-21systemd-run00:00:011
2940270994532,10cyclictest2941469-21sshd19:30:501
2940270994527,13cyclictest1-21systemd22:00:011
2940266994530,12cyclictest12-21ksoftirqd/020:00:010
2940266994530,11cyclictest275-21dbus-daemon20:55:010
294027699443,3cyclictest131rcu_preempt23:20:163
2940276994427,13cyclictest275-21dbus-daemon21:20:013
294027699442,3cyclictest131rcu_preempt21:33:143
2940276994423,5cyclictest37-21ksoftirqd/322:35:043
2940270994429,11cyclictest3467602-21systemd-journal20:57:391
294026699441,38cyclictest2946171-21sshd21:04:530
294026699441,38cyclictest12-21ksoftirqd/019:20:030
294027699433,3cyclictest131rcu_preempt23:44:083
2940276994331,9cyclictest37-21ksoftirqd/320:25:013
2940276994330,9cyclictest2943549-21sshd20:11:543
2940276994330,8cyclictest309-21in:imuxsock19:35:013
294027699432,4cyclictest131rcu_preempt21:46:353
294027099433,3cyclictest131rcu_preempt21:50:021
2940270994329,11cyclictest23-21ksoftirqd/100:00:171
2940270994329,10cyclictest3467602-21systemd-journal19:26:501
2940270994327,12cyclictest23-21ksoftirqd/123:12:071
294026699439,6cyclictest131rcu_preempt20:25:010
294027699424,3cyclictest131rcu_preempt23:16:193
294027699424,3cyclictest131rcu_preempt00:38:203
294027699424,3cyclictest131rcu_preempt00:00:333
2940276994234,5cyclictest30056101kworker/u8:223:29:053
294027699423,3cyclictest131rcu_preempt20:45:033
2940276994215,19cyclictest37-21ksoftirqd/300:10:533
294027699421,3cyclictest131rcu_preempt22:49:533
2940272994235,4cyclictest2939643-21kworker/2:320:12:232
294027099426,3cyclictest131rcu_preempt22:20:001
2940270994228,11cyclictest23-21ksoftirqd/100:31:481
2940270994228,10cyclictest309-21in:imuxsock00:30:001
2940270994222,15cyclictest3467602-21systemd-journal19:40:001
2940266994230,7cyclictest2941747-21kworker/u8:019:40:000
294027699413,3cyclictest131rcu_preempt19:15:003
2940276994131,7cyclictest274-21cron00:20:013
2940276994127,11cyclictest37-21ksoftirqd/323:00:023
2940276994127,11cyclictest2943526-21sed20:10:023
2940276994123,5cyclictest37-21ksoftirqd/323:30:193
2940276994123,13cyclictest275-21dbus-daemon19:40:003
294027699411,3cyclictest131rcu_preempt22:31:193
294027699411,3cyclictest131rcu_preempt21:26:003
294027099415,3cyclictest150-21pr/ttyS220:50:021
294027099414,3cyclictest131rcu_preempt22:39:571
294027099413,3cyclictest131rcu_preempt00:40:001
2940270994129,9cyclictest23-21ksoftirqd/123:24:531
2940270994129,8cyclictest309-21in:imuxsock20:15:031
2940270994126,11cyclictest3467602-21systemd-journal22:15:001
2940270994125,12cyclictest275-21dbus-daemon21:45:011
2940270994123,14cyclictest1-21systemd20:45:021
294027099411,3cyclictest131rcu_preempt00:25:021
2940266994128,10cyclictest3467602-21systemd-journal20:42:350
294026699411,12cyclictest2943813-21sshd20:17:530
294027699404,3cyclictest131rcu_preempt19:10:013
2940276994029,9cyclictest37-21ksoftirqd/322:15:433
2940276994022,5cyclictest37-21ksoftirqd/321:42:263
2940272994031,7cyclictest30-21ksoftirqd/219:10:022
294027099408,24cyclictest0-21swapper/123:01:141
2940270994030,7cyclictest2942057-21latency_hist19:45:011
294027099402,3cyclictest131rcu_preempt23:20:031
294027099402,3cyclictest131rcu_preempt22:42:411
294027099402,3cyclictest131rcu_preempt21:19:371
294027099401,3cyclictest131rcu_preempt23:43:531
294026699403,29cyclictest2940265-21wc19:10:020
2940266994028,9cyclictest12-21ksoftirqd/020:49:530
294027699393,3cyclictest131rcu_preempt23:51:023
294027699393,3cyclictest131rcu_preempt23:07:453
2940276993923,13cyclictest275-21dbus-daemon23:50:013
2940276993922,5cyclictest37-21ksoftirqd/323:36:483
2940276993919,5cyclictest131rcu_preempt22:25:413
294027699391,3cyclictest131rcu_preempt00:05:413
294027099394,4cyclictest131rcu_preempt22:05:001
294027099394,3cyclictest131rcu_preempt22:50:001
294027099393,3cyclictest131rcu_preempt22:20:311
2940270993926,9cyclictest23-21ksoftirqd/100:14:021
2940270993926,10cyclictest23-21ksoftirqd/122:05:121
2940270993924,9cyclictest309-21in:imuxsock21:40:001
294027099392,27cyclictest0-21swapper/121:30:021
2940270993922,14cyclictest3017727-21munin-run23:55:001
2940270993916,6cyclictest131rcu_preempt20:35:011
2940270993914,6cyclictest23-21ksoftirqd/122:51:381
294026699391,31cyclictest12-21ksoftirqd/019:45:020
2940276993825,9cyclictest2940778-21sed19:20:013
2940276993825,9cyclictest2940778-21sed19:20:013
294027699382,4cyclictest131rcu_preempt22:42:073
2940276993821,5cyclictest37-21ksoftirqd/320:50:003
2940276993821,4cyclictest37-21ksoftirqd/320:40:003
2940276993820,12cyclictest275-21dbus-daemon21:10:013
294027099385,4cyclictest131rcu_preempt21:50:091
294027099383,3cyclictest131rcu_preempt20:25:001
2940270993827,8cyclictest23-21ksoftirqd/123:38:581
2940270993825,10cyclictest23-21ksoftirqd/121:10:361
2940270993825,10cyclictest23-21ksoftirqd/121:10:361
2940270993822,12cyclictest2944411-21sed20:30:021
294027099381,3cyclictest131rcu_preempt00:05:111
2940270993813,7cyclictest23-21ksoftirqd/100:18:241
294027099381,12cyclictest3467718-21lldpd21:09:541
294026699381,33cyclictest0-21swapper/020:33:090
294027699377,8cyclictest37-21ksoftirqd/320:35:023
2940276993727,5cyclictest37-21ksoftirqd/320:00:013
2940276993725,9cyclictest37-21ksoftirqd/319:30:003
2940276993721,4cyclictest37-21ksoftirqd/320:05:003
2940276993719,5cyclictest37-21ksoftirqd/319:55:003
2940270993724,10cyclictest23-21ksoftirqd/120:40:011
2940270993723,11cyclictest3467602-21systemd-journal19:53:021
2940276993627,7cyclictest37-21ksoftirqd/319:50:023
2940276993620,12cyclictest37-21ksoftirqd/320:55:023
2940270993625,8cyclictest3467602-21systemd-journal22:55:521
2940270993625,7cyclictest2945639-21sshd20:54:531
2940270993624,8cyclictest2998306-21cron23:10:001
294027099361,3cyclictest131rcu_preempt22:35:011
2940266993630,4cyclictest16950irq/224-800000019:15:000
2940276993520,11cyclictest0-21swapper/319:45:013
2940270993527,5cyclictest2943168-21latency_hist20:05:021
2940270993523,8cyclictest23-21ksoftirqd/120:07:311
294026699351,5cyclictest2943339-21munin-run20:10:010
294027099345,3cyclictest131rcu_preempt19:25:001
2940270993421,10cyclictest23-21ksoftirqd/121:05:011
2940270993420,10cyclictest275-21dbus-daemon19:15:011
294027099341,3cyclictest131rcu_preempt19:50:011
2940270993317,12cyclictest2943783-21sed20:15:021
294026699332,29cyclictest0-21swapper/020:28:290
2940276993121,5cyclictest37-21ksoftirqd/321:01:533
2940276993116,5cyclictest131rcu_preempt20:30:013
2940270993119,8cyclictest23-21ksoftirqd/120:00:021
294027099292,5cyclictest2940307-21grep19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional