You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-01 - 01:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Jun 30, 2026 12:45:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30651039911487,2cyclictest0-21swapper/008:25:020
30651039911487,2cyclictest0-21swapper/008:25:020
3065103999086,2cyclictest0-21swapper/008:30:020
3065110997336,9cyclictest3080813-21kworker/2:009:45:312
3065103997227,15cyclictest131rcu_preempt09:53:500
3065103997227,15cyclictest131rcu_preempt09:53:500
3065110997028,6cyclictest3146258-21sh11:46:522
3065110996827,6cyclictest3083181-21latency_hist09:35:012
3065106996843,4cyclictest23-21ksoftirqd/108:35:021
306510699663,20cyclictest3094708-21ssh09:59:091
306510399667,4cyclictest131rcu_preempt10:50:020
3065106996525,36cyclictest3163606-21sh12:23:191
3065106996525,36cyclictest3163606-21sh12:23:191
3065103996561,2cyclictest0-21swapper/010:05:020
3065110996427,11cyclictest1-21systemd10:55:002
3065110996425,11cyclictest3467602-21systemd-journal11:15:002
306510699645,51cyclictest3161915-21cron12:20:011
306510699642,21cyclictest3125258-21ssh11:03:041
306510699642,19cyclictest3142168-21ssh11:38:321
306510699642,15cyclictest3149968-21munin-run11:55:001
306510699641,13cyclictest3127886-21sh11:08:441
306510699641,13cyclictest3127886-21sh11:08:431
3065110996325,10cyclictest30-21ksoftirqd/208:15:012
306510699636,27cyclictest131rcu_preempt08:23:501
306510699636,27cyclictest131rcu_preempt08:23:501
306510699632,17cyclictest3136157-21ssh11:25:401
306510699632,15cyclictest3130840-21munin-run11:15:001
306510699631,6cyclictest3115009-21ssh10:41:251
306510699631,6cyclictest3089783-21ssh09:48:451
306510699631,16cyclictest3082350-21ssh09:33:081
306510699631,15cyclictest3165381-21ssh12:26:561
306510699631,10cyclictest3117930-21ssh10:47:381
3065103996316,8cyclictest16850irq/134-800000009:40:000
3065103996316,8cyclictest16850irq/134-800000009:40:000
3065110996229,4cyclictest3164042-21sh12:24:232
3065110996229,4cyclictest3164042-21sh12:24:222
306510699622,8cyclictest3170051-21ssh12:36:451
306510699622,55cyclictest3091066-21ssh09:51:161
306510699622,55cyclictest3091066-21ssh09:51:161
306510699622,16cyclictest3098518-21ssh10:06:541
306510699621,7cyclictest3079749-21ssh09:27:461
306510699621,7cyclictest3079749-21ssh09:27:461
306510699621,12cyclictest3119318-21ssh10:50:201
306510699621,10cyclictest3139950-21ssh11:33:551
306510699621,10cyclictest3139950-21ssh11:33:551
3065103996257,3cyclictest0-21swapper/007:40:020
306510699612,8cyclictest3110777-21ssh10:32:361
306510699612,37cyclictest3070202-21kworker/u8:008:55:021
306510699612,37cyclictest3070202-21kworker/u8:008:55:021
306510699612,16cyclictest3070251-21munin-run08:50:011
306510699612,15cyclictest3069984-21munin-run08:45:001
306510699611,9cyclictest3158207-21ssh12:11:551
306510699611,9cyclictest3151880-21ssh11:58:521
306510699611,9cyclictest3144894-21ssh11:44:201
306510699611,9cyclictest3103684-21ssh10:17:501
306510699611,8cyclictest3167713-21ssh12:31:441
306510699611,8cyclictest3105031-21ssh10:20:161
306510699611,7cyclictest3155236-21ssh12:05:381
306510699611,7cyclictest3108880-21ssh10:28:481
306510699611,7cyclictest3101416-21ssh10:13:081
306510699611,7cyclictest3101416-21ssh10:13:081
306510699611,7cyclictest3083742-21ssh09:35:531
306510699611,7cyclictest3083742-21ssh09:35:531
306510699611,6cyclictest3134224-21ssh11:21:371
306510699611,6cyclictest3134224-21ssh11:21:371
306510699611,6cyclictest3072506-21ssh09:12:321
306510699611,5cyclictest3075063-21sshd09:17:521
306510699611,5cyclictest3075063-21sshd09:17:521
3065106996112,45cyclictest364-21snmpd07:39:451
306510699611,17cyclictest3096352-21ssh10:02:261
3065103996157,2cyclictest0-21swapper/008:20:020
3065110996027,8cyclictest30-21ksoftirqd/212:05:162
3065110996014,11cyclictest0-21swapper/210:25:012
306510699602,7cyclictest3087647-21ssh09:44:221
306510699602,7cyclictest3087647-21ssh09:44:211
306510699601,9cyclictest3112286-21ssh10:35:341
306510699601,8cyclictest3153645-21ssh12:02:331
306510699601,7cyclictest3145641-21ssh11:45:361
306510699601,7cyclictest3131300-21ssh11:15:311
306510699601,7cyclictest3121887-21ssh10:55:491
306510699601,7cyclictest3121887-21ssh10:55:491
306510699601,7cyclictest3076238-21ssh09:20:041
306510699601,5cyclictest3065848-21munin-run07:25:021
3065103996029,16cyclictest3102748-21ssh10:15:340
3065103996028,16cyclictest3098099-21ssh10:05:550
3065112995952,3cyclictest0-21swapper/311:35:023
3065112995952,3cyclictest0-21swapper/311:35:023
3065110995926,5cyclictest3066759-21latency_hist07:40:022
306510699591,9cyclictest3066846-21sshd07:43:501
3065110995839,15cyclictest1-21systemd10:30:012
3065110995826,11cyclictest30-21ksoftirqd/212:35:542
3065103995828,12cyclictest3146344-21ssh11:47:110
3065103995827,15cyclictest3107003-21ssh10:24:530
3065103995820,21cyclictest16850irq/134-800000010:40:120
306510399581,41cyclictest3134338-21ssh11:21:510
306510399581,41cyclictest3134338-21ssh11:21:510
3065110995724,11cyclictest30-21ksoftirqd/209:44:252
3065110995724,11cyclictest30-21ksoftirqd/209:44:242
3065103995722,31cyclictest12-21ksoftirqd/011:35:520
3065103995720,33cyclictest16850irq/134-800000012:35:020
3065103995719,33cyclictest16850irq/134-800000010:14:110
3065103995719,33cyclictest16850irq/134-800000010:14:110
3065110995626,9cyclictest30-21ksoftirqd/207:35:012
306510699562,6cyclictest3067384-21munin-run07:55:011
3065103995627,13cyclictest3089237-21ssh09:47:280
3065103995626,12cyclictest3133211-21ssh11:19:550
306510399562,37cyclictest3151337-21ssh11:57:450
3065103995620,32cyclictest16850irq/134-800000009:40:390
3065103995620,32cyclictest16850irq/134-800000009:40:390
3065103995619,33cyclictest16850irq/134-800000012:22:250
3065103995619,33cyclictest16850irq/134-800000012:22:250
3065103995619,32cyclictest16850irq/134-800000010:39:100
3065103995618,34cyclictest16850irq/134-800000011:27:500
3065110995527,9cyclictest30-21ksoftirqd/212:15:402
306510699551,9cyclictest3071249-21sshd09:07:521
306510699551,9cyclictest3071249-21sshd09:07:521
306510699551,5cyclictest364-21snmpd08:08:461
3065103995528,11cyclictest3161631-21ssh12:19:160
3065103995520,31cyclictest16850irq/134-800000011:06:020
3065103995520,31cyclictest16850irq/134-800000011:06:020
3065103995519,32cyclictest16850irq/134-800000012:01:050
3065103995519,31cyclictest16850irq/134-800000011:52:200
3065103995519,31cyclictest16850irq/134-800000010:30:530
3065103995518,32cyclictest16850irq/134-800000009:29:480
3065103995518,32cyclictest16850irq/134-800000009:29:480
306510699541,4cyclictest364-21snmpd07:28:381
306510699541,11cyclictest3067597-21sshd07:57:511
3065103995418,32cyclictest16850irq/134-800000012:26:330
3065103995418,32cyclictest16850irq/134-800000011:32:270
3065103995418,32cyclictest16850irq/134-800000011:32:270
3065103995418,32cyclictest16850irq/134-800000011:01:060
3065103995418,32cyclictest16850irq/134-800000010:58:460
3065103995418,32cyclictest16850irq/134-800000010:58:460
3065103995418,32cyclictest16850irq/134-800000010:29:190
3065103995418,32cyclictest16850irq/134-800000009:56:130
3065103995418,31cyclictest16850irq/134-800000012:12:160
3065103995417,7cyclictest3130995-21wc11:15:030
3065103995417,32cyclictest16850irq/134-800000012:07:360
306511299531,37cyclictest0-21swapper/309:40:023
306511299531,37cyclictest0-21swapper/309:40:023
306510699533,43cyclictest3069694-21modprobe08:40:001
306510699531,9cyclictest3070990-21sendmail-mta09:00:151
306510699531,7cyclictest3468525-21sendmail-mta07:10:151
306510699531,12cyclictest3065594-21systemd-run07:20:011
306510399533,16cyclictest3070768-21grep09:00:000
3065103995325,13cyclictest3169495-21ssh12:35:250
3065103995317,33cyclictest16850irq/134-800000009:17:180
3065103995317,33cyclictest16850irq/134-800000009:17:170
3065103995317,32cyclictest16850irq/134-800000010:47:540
3065103995317,31cyclictest16850irq/134-800000009:10:520
3065103995315,33cyclictest16850irq/134-800000009:10:020
3065103995315,33cyclictest16850irq/134-800000009:10:020
3065112995234,13cyclictest3114182-21cron10:40:003
3065110995244,5cyclictest3123758-21kworker/2:111:03:402
306510699522,4cyclictest364-21snmpd08:03:461
306510699522,4cyclictest364-21snmpd08:03:461
3065103995217,31cyclictest16850irq/134-800000009:20:490
3065103995217,20cyclictest16850irq/134-800000011:40:590
3065110995143,5cyclictest3140429-21kworker/2:311:59:372
3065110995143,5cyclictest3090323-21kworker/2:110:16:472
3065110995143,5cyclictest3080813-21kworker/2:009:35:042
3065110995143,5cyclictest3080813-21kworker/2:009:35:042
3065110995143,5cyclictest3064480-21kworker/2:212:01:482
3065110995142,6cyclictest3064480-21kworker/2:210:04:222
306510699513,5cyclictest3069203-21sh08:30:001
306510699512,4cyclictest3066311-21sshd07:32:501
3065103995141,7cyclictest19350irq/34-mmc107:15:000
3065110995042,5cyclictest3064480-21kworker/2:209:27:342
3065110995042,5cyclictest3064480-21kworker/2:209:27:342
3065110995042,5cyclictest3056087-21kworker/2:307:29:502
3065110995041,6cyclictest3109457-21kworker/2:310:41:302
3065110995041,5cyclictest3140429-21kworker/2:311:40:102
306511099503,4cyclictest131rcu_preempt11:35:002
306511099503,4cyclictest131rcu_preempt11:35:002
306510699502,5cyclictest3070720-21sshd08:58:521
3065103995031,15cyclictest3467602-21systemd-journal07:20:000
3065110994941,5cyclictest3140429-21kworker/2:312:31:062
3065110994941,5cyclictest3140429-21kworker/2:312:27:182
3065110994941,5cyclictest3109457-21kworker/2:310:31:512
3065110994941,5cyclictest3080813-21kworker/2:009:50:152
3065110994941,5cyclictest3080813-21kworker/2:009:50:142
3065110994941,5cyclictest3064480-21kworker/2:211:22:292
3065110994941,5cyclictest3064480-21kworker/2:211:22:282
306510699492,5cyclictest364-21snmpd08:19:551
306510699491,5cyclictest3068370-21sendmail-mta08:10:151
3065103994930,13cyclictest1-21systemd07:40:000
306510399492,42cyclictest3069705-21latency_hist08:40:000
3065103994913,32cyclictest16850irq/134-800000009:30:410
3065112994836,9cyclictest3115622-21sshd10:42:523
306511299483,27cyclictest3126278-21ssh11:05:023
3065112994831,13cyclictest3152349-21cron12:00:003
3065112994816,17cyclictest37-21ksoftirqd/309:55:023
3065112994816,17cyclictest37-21ksoftirqd/309:55:023
3065110994840,5cyclictest3109457-21kworker/2:310:45:122
3065110994840,5cyclictest3064480-21kworker/2:211:29:322
306511099482,7cyclictest0-21swapper/207:20:002
3065106994818,26cyclictest3467562-21sshd07:48:501
3065106994818,26cyclictest3467562-21sshd07:48:501
306511299474,9cyclictest37-21ksoftirqd/312:38:003
3065112994725,5cyclictest37-21ksoftirqd/310:10:013
3065110994730,12cyclictest275-21dbus-daemon11:55:002
306510399475,36cyclictest3067611-21cron08:00:000
306510399472,12cyclictest3067905-21modprobe08:05:000
306510399472,12cyclictest3067905-21modprobe08:05:000
306511299462,38cyclictest3467602-21systemd-journal08:05:013
306511299462,38cyclictest3467602-21systemd-journal08:05:013
3065112994619,6cyclictest37-21ksoftirqd/309:25:003
306511299461,8cyclictest37-21ksoftirqd/311:25:003
306511299461,8cyclictest37-21ksoftirqd/311:25:003
3065110994639,4cyclictest3064480-21kworker/2:208:05:022
3065110994639,4cyclictest3064480-21kworker/2:208:05:022
3065110994637,6cyclictest3109457-21kworker/2:310:58:552
3065110994637,6cyclictest3109457-21kworker/2:310:58:542
3065110994637,6cyclictest3064480-21kworker/2:209:14:592
3065110994632,11cyclictest30-21ksoftirqd/208:10:012
3065110994631,12cyclictest30-21ksoftirqd/208:20:002
3065112994520,5cyclictest37-21ksoftirqd/309:56:393
3065112994516,7cyclictest37-21ksoftirqd/309:15:013
306511099452,38cyclictest30-21ksoftirqd/210:35:132
306510699452,11cyclictest3065036-21latency_hist07:10:031
306510399452,3cyclictest131rcu_preempt07:55:020
3065112994429,11cyclictest275-21dbus-daemon07:50:013
3065112994429,11cyclictest275-21dbus-daemon07:50:013
3065112994420,6cyclictest37-21ksoftirqd/310:29:483
3065112994420,4cyclictest37-21ksoftirqd/311:51:043
3065112994420,4cyclictest37-21ksoftirqd/309:17:343
3065112994420,4cyclictest37-21ksoftirqd/309:17:343
306511299441,6cyclictest131rcu_preempt11:00:003
306511299441,6cyclictest131rcu_preempt11:00:003
3065110994432,8cyclictest3095137-21systemctl10:00:002
3065103994431,10cyclictest1-21systemd07:30:000
3065103994426,13cyclictest275-21dbus-daemon08:50:010
306510399441,39cyclictest12-21ksoftirqd/008:50:450
306510399441,39cyclictest12-21ksoftirqd/008:50:450
306511299438,5cyclictest131rcu_preempt10:00:103
3065112994333,5cyclictest3104660-21sendmail10:20:003
3065112994328,11cyclictest3467602-21systemd-journal07:40:003
3065112994327,12cyclictest275-21dbus-daemon11:45:013
3065112994320,6cyclictest37-21ksoftirqd/312:00:523
3065112994320,5cyclictest37-21ksoftirqd/310:10:493
3065112994320,5cyclictest37-21ksoftirqd/310:10:493
3065112994320,5cyclictest37-21ksoftirqd/308:00:013
306511299431,3cyclictest131rcu_preempt12:11:533
3065110994335,5cyclictest3123758-21kworker/2:111:06:402
3065110994335,5cyclictest3123758-21kworker/2:111:06:402
3065110994334,6cyclictest3064480-21kworker/2:211:35:412
3065110994333,7cyclictest3067341-21sshd07:51:502
3065110994331,9cyclictest3071244-21sshd09:06:522
3065110994331,9cyclictest3071244-21sshd09:06:522
3065110994330,10cyclictest30-21ksoftirqd/210:11:072
3065110994330,10cyclictest30-21ksoftirqd/210:11:072
3065110994329,10cyclictest3467602-21systemd-journal09:20:012
3065110994329,10cyclictest3467602-21systemd-journal09:20:012
306511299424,6cyclictest131rcu_preempt08:45:003
3065112994230,9cyclictest1-21systemd09:45:003
3065112994230,9cyclictest1-21systemd09:45:003
3065112994227,10cyclictest37-21ksoftirqd/311:35:373
3065112994222,4cyclictest37-21ksoftirqd/310:23:353
3065112994221,5cyclictest37-21ksoftirqd/309:46:133
3065112994220,5cyclictest37-21ksoftirqd/312:25:023
3065112994220,5cyclictest37-21ksoftirqd/312:25:013
3065110994236,3cyclictest3064480-21kworker/2:212:10:312
3065110994234,5cyclictest3064480-21kworker/2:210:09:442
3065110994234,5cyclictest3064480-21kworker/2:208:50:022
3065110994234,5cyclictest3064480-21kworker/2:208:50:022
3065110994228,10cyclictest3077876-21ssh09:23:502
3065110994227,12cyclictest30-21ksoftirqd/208:00:002
306510699421,11cyclictest3065124-21latency_hist07:10:021
3065103994218,4cyclictest12-21ksoftirqd/007:10:020
3065103994218,4cyclictest12-21ksoftirqd/007:10:020
306511299416,5cyclictest131rcu_preempt09:28:593
306511299416,5cyclictest131rcu_preempt09:28:593
306511299413,3cyclictest131rcu_preempt12:28:113
3065112994124,12cyclictest275-21dbus-daemon07:30:003
3065112994118,5cyclictest37-21ksoftirqd/312:33:353
3065112994118,5cyclictest37-21ksoftirqd/310:32:373
3065112994117,5cyclictest37-21ksoftirqd/309:34:493
3065112994117,4cyclictest37-21ksoftirqd/308:15:003
3065112994116,5cyclictest37-21ksoftirqd/311:06:193
3065112994116,5cyclictest37-21ksoftirqd/311:06:193
3065110994132,6cyclictest30-21ksoftirqd/209:05:002
3065110994125,12cyclictest30-21ksoftirqd/211:15:212
306510399411,3cyclictest131rcu_preempt08:45:000
3065112994024,12cyclictest37-21ksoftirqd/311:11:143
3065112994019,5cyclictest37-21ksoftirqd/311:28:193
3065112994019,5cyclictest37-21ksoftirqd/310:53:403
306511299401,5cyclictest131rcu_preempt10:50:013
306511299396,6cyclictest131rcu_preempt11:47:093
3065112993927,8cyclictest3070250-21systemd-run08:50:013
3065112993926,9cyclictest275-21dbus-daemon08:25:013
3065112993926,9cyclictest275-21dbus-daemon08:25:013
306511299392,5cyclictest131rcu_preempt09:00:023
3065112993916,5cyclictest37-21ksoftirqd/312:19:333
3065110993926,10cyclictest30-21ksoftirqd/207:25:022
3065110993923,11cyclictest3069708-21cat08:40:002
3065103993932,4cyclictest16950irq/224-800000008:10:000
306511299384,3cyclictest131rcu_preempt12:05:303
3065112993825,8cyclictest3467602-21systemd-journal08:55:003
3065112993825,8cyclictest3467602-21systemd-journal08:55:003
3065112993824,11cyclictest37-21ksoftirqd/309:10:013
3065112993824,11cyclictest37-21ksoftirqd/309:10:013
3065103993825,9cyclictest12-21ksoftirqd/008:14:250
3065112993724,10cyclictest37-21ksoftirqd/311:18:463
3065110993725,9cyclictest30-21ksoftirqd/208:47:402
3065110993723,11cyclictest30-21ksoftirqd/209:00:012
3065110993722,11cyclictest30-21ksoftirqd/207:11:152
306510399372,11cyclictest1-21systemd07:32:500
306510399371,32cyclictest3071000-21sshd09:02:520
306511299364,3cyclictest131rcu_preempt07:55:013
3065112993625,8cyclictest3066316-21sshd07:33:503
3065112993624,9cyclictest275-21dbus-daemon08:30:013
3065112993624,8cyclictest37-21ksoftirqd/308:17:003
3065112993620,11cyclictest275-21dbus-daemon07:20:013
3065110993623,10cyclictest30-21ksoftirqd/208:20:302
3065110993623,10cyclictest30-21ksoftirqd/208:20:292
3065110993622,10cyclictest30-21ksoftirqd/207:40:442
306511099361,31cyclictest48-21kcompactd008:40:542
306510399362,4cyclictest3065843-21systemd-run07:25:010
306511299354,3cyclictest131rcu_preempt08:33:523
3065112993525,8cyclictest37-21ksoftirqd/308:40:023
3065112993525,5cyclictest37-21ksoftirqd/307:45:013
3065112993518,4cyclictest37-21ksoftirqd/308:06:503
306510399351,7cyclictest3467602-21systemd-journal08:31:350
3065112993424,8cyclictest37-21ksoftirqd/307:25:013
306511099342,3cyclictest131rcu_preempt07:50:022
306511099342,3cyclictest131rcu_preempt07:50:022
3065110993423,8cyclictest30-21ksoftirqd/208:32:342
3065110993421,9cyclictest30-21ksoftirqd/208:27:052
306510399348,22cyclictest3067108-21cron07:50:000
306510399348,22cyclictest3067108-21cron07:50:000
306511299331,4cyclictest131rcu_preempt09:05:003
3065110993325,5cyclictest3065036-21latency_hist07:10:022
3065110993325,5cyclictest3065036-21latency_hist07:10:022
306511299325,20cyclictest1-21systemd07:15:003
3065112993015,11cyclictest3065193-21cat07:10:023
3065112993015,11cyclictest3065193-21cat07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional