You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 06:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri May 08, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2597079998030,13cyclictest2679269-21ssh23:47:110
2597079997941,34cyclictest16950irq/224-800000023:43:450
259707999793,7cyclictest2685317-21cat00:00:000
2597079997634,17cyclictest2599714-21tr20:00:000
2597079997334,14cyclictest2704648-21systemctl00:40:000
2597079997329,11cyclictest3467602-21systemd-journal21:25:000
2597079997231,9cyclictest16950irq/224-800000023:55:020
2597079997013,53cyclictest2630051-21ssh22:05:020
2597079997011,48cyclictest2702243-21munin-plugin-st00:35:000
2597079997011,48cyclictest2702243-21munin-plugin-st00:35:000
259707999692,10cyclictest2656676-21cat23:00:020
259709099689,16cyclictest2603340-21diskmemload23:47:233
2597079996825,12cyclictest275-21dbus-daemon23:35:010
2597079996721,13cyclictest2690654-21ssh00:10:430
2597079996720,11cyclictest2604555-21ssh21:12:120
2597079996719,11cyclictest2606879-21ssh21:17:060
2597079996719,11cyclictest2606879-21ssh21:17:060
2597080996616,11cyclictest23-21ksoftirqd/122:50:121
2597090996513,15cyclictest2650279-21ssh22:46:533
2597080996515,11cyclictest23-21ksoftirqd/121:41:361
2597080996418,9cyclictest23-21ksoftirqd/121:57:141
2597090996317,9cyclictest37-21ksoftirqd/322:05:203
2597090996220,14cyclictest2659828-21ssh23:06:353
2597080996218,10cyclictest23-21ksoftirqd/121:32:061
259709099619,15cyclictest2628096-21ssh22:00:593
259709099616,14cyclictest2642221-21ssh22:30:063
2597080996117,9cyclictest23-21ksoftirqd/121:25:021
2597080996113,15cyclictest2697197-21rm00:24:401
2597079996127,30cyclictest12-21ksoftirqd/000:02:360
2597090996014,10cyclictest37-21ksoftirqd/321:14:283
2597080996021,9cyclictest2600476-21latency_hist20:15:011
2597080996015,8cyclictest23-21ksoftirqd/119:50:021
2597079996036,18cyclictest2644361-21munin-plugin-st22:35:000
2597080995917,9cyclictest23-21ksoftirqd/119:45:001
2597090995815,14cyclictest2691968-21ssh00:13:443
2597090995813,8cyclictest37-21ksoftirqd/323:34:123
2597084995839,13cyclictest1-21systemd20:25:012
2597080995817,10cyclictest23-21ksoftirqd/121:28:421
2597080995815,9cyclictest23-21ksoftirqd/120:45:021
2597080995813,9cyclictest23-21ksoftirqd/122:49:221
259709099579,14cyclictest2639754-21sed22:25:023
2597090995715,8cyclictest37-21ksoftirqd/320:55:023
2597090995713,11cyclictest37-21ksoftirqd/322:25:583
2597080995716,8cyclictest23-21ksoftirqd/123:40:001
2597080995713,6cyclictest23-21ksoftirqd/122:21:081
2597079995711,4cyclictest12-21ksoftirqd/021:10:020
2597079995711,4cyclictest12-21ksoftirqd/021:10:020
259709099567,14cyclictest2653442-21ssh22:53:373
2597090995615,7cyclictest37-21ksoftirqd/321:27:173
2597090995614,9cyclictest37-21ksoftirqd/321:50:593
2597090995610,9cyclictest37-21ksoftirqd/323:24:063
2597090995610,9cyclictest37-21ksoftirqd/323:24:063
2597084995647,5cyclictest2656818-21kworker/2:023:31:392
259708099566,14cyclictest2656253-21ssh22:59:301
2597080995617,9cyclictest23-21ksoftirqd/123:51:271
2597079995625,27cyclictest2603340-21diskmemload00:25:000
2597090995513,8cyclictest37-21ksoftirqd/321:34:403
2597084995544,6cyclictest2644382-21kworker/2:122:57:292
2597084995528,8cyclictest2590305-21kworker/2:321:30:172
259708099558,15cyclictest2698401-21ssh00:26:511
2597080995516,7cyclictest23-21ksoftirqd/121:12:281
2597080995514,7cyclictest23-21ksoftirqd/123:27:231
2597080995514,11cyclictest23-21ksoftirqd/100:17:181
2597084995446,5cyclictest2590305-21kworker/2:322:33:532
2597084995446,5cyclictest2590305-21kworker/2:321:05:012
2597084995430,5cyclictest2644382-21kworker/2:122:54:182
259708099548,15cyclictest2633030-21ssh22:11:171
2597080995412,10cyclictest23-21ksoftirqd/123:07:361
2597079995426,13cyclictest2663219-21ssh23:13:580
2597079995425,13cyclictest2652857-21ssh22:52:130
259707999542,37cyclictest2615451-21ssh21:34:580
2597079995413,37cyclictest0-21swapper/022:30:020
259709099538,11cyclictest0-21swapper/322:19:463
2597084995345,5cyclictest2644382-21kworker/2:123:15:562
2597084995345,5cyclictest2644382-21kworker/2:123:00:242
2597084995345,5cyclictest2590305-21kworker/2:321:21:102
2597084995345,5cyclictest2590305-21kworker/2:321:13:112
2597084995344,5cyclictest2656818-21kworker/2:000:03:462
2597084995344,5cyclictest2644382-21kworker/2:123:14:352
2597084995342,8cyclictest2590305-21kworker/2:321:17:092
2597084995342,8cyclictest2590305-21kworker/2:321:17:082
2597084995336,12cyclictest1-21systemd23:40:002
2597084995335,13cyclictest1-21systemd23:45:012
2597080995335,12cyclictest3467602-21systemd-journal23:05:011
2597080995315,9cyclictest23-21ksoftirqd/122:30:121
2597080995313,8cyclictest23-21ksoftirqd/122:35:021
2597079995329,8cyclictest12-21ksoftirqd/022:20:020
2597079995327,11cyclictest2693363-21ssh00:16:180
2597079995327,11cyclictest2668383-21ssh23:24:450
2597079995327,11cyclictest2668383-21ssh23:24:450
2597079995324,14cyclictest2644847-21ssh22:35:370
259707999532,36cyclictest2647661-21ssh22:41:250
259709099527,14cyclictest2662816-21ssh23:12:583
2597090995213,9cyclictest37-21ksoftirqd/321:43:473
2597090995213,8cyclictest37-21ksoftirqd/321:17:103
2597090995213,8cyclictest37-21ksoftirqd/321:17:093
2597090995212,9cyclictest37-21ksoftirqd/300:19:583
2597084995244,5cyclictest2678218-21kworker/2:300:12:332
2597084995244,5cyclictest2656818-21kworker/2:000:05:422
2597084995244,5cyclictest2590305-21kworker/2:321:58:152
2597084995243,6cyclictest2602575-21kworker/2:222:22:182
2597084995243,6cyclictest2590305-21kworker/2:322:39:552
2597084995243,5cyclictest2590305-21kworker/2:322:02:082
2597084995242,5cyclictest2590305-21kworker/2:319:10:462
2597080995216,6cyclictest23-21ksoftirqd/122:00:251
2597079995225,12cyclictest2639525-21ssh22:24:570
2597079995222,14cyclictest2618643-21ssh21:41:170
2597090995134,14cyclictest1-21systemd00:35:013
2597090995134,14cyclictest1-21systemd00:35:013
2597090995116,9cyclictest37-21ksoftirqd/323:52:153
2597084995143,5cyclictest2644382-21kworker/2:122:48:422
2597084995143,5cyclictest2602575-21kworker/2:222:06:592
2597084995143,5cyclictest2590305-21kworker/2:321:49:072
2597084995143,5cyclictest2590305-21kworker/2:319:25:022
2597084995143,5cyclictest2590305-21kworker/2:319:25:012
2597084995142,5cyclictest2678218-21kworker/2:300:16:232
2597084995142,5cyclictest2656818-21kworker/2:023:47:102
2597084995141,6cyclictest2590305-21kworker/2:322:41:152
2597084995141,6cyclictest2590305-21kworker/2:322:11:472
259708099516,15cyclictest2700626-21ssh00:31:201
259708099516,15cyclictest2700626-21ssh00:31:201
259709099502,6cyclictest2678170-21ssh23:45:013
2597084995042,5cyclictest2697410-21kworker/2:200:30:082
2597084995042,5cyclictest2697410-21kworker/2:200:30:082
2597084995042,5cyclictest2697410-21kworker/2:200:29:362
2597084995042,5cyclictest2590305-21kworker/2:321:53:452
2597084995042,5cyclictest2590305-21kworker/2:321:25:032
2597084995040,6cyclictest2656818-21kworker/2:000:22:532
259708099506,12cyclictest2686630-21sh00:02:231
2597080995029,16cyclictest275-21dbus-daemon21:10:021
2597080995029,16cyclictest275-21dbus-daemon21:10:021
2597080995014,8cyclictest23-21ksoftirqd/121:38:061
259707999505,41cyclictest2617883-21ssh21:40:010
259707999502,4cyclictest131rcu_preempt21:48:580
2597079995024,6cyclictest0-21swapper/022:06:190
259709099495,6cyclictest131rcu_preempt23:00:033
2597090994931,14cyclictest2684127-21ssh23:57:143
2597084994942,4cyclictest2656818-21kworker/2:023:56:042
2597084994941,5cyclictest2656818-21kworker/2:023:07:522
2597084994941,5cyclictest2656818-21kworker/2:000:35:342
2597084994941,5cyclictest2644382-21kworker/2:123:24:092
2597084994941,5cyclictest2644382-21kworker/2:123:24:092
2597084994941,5cyclictest2602575-21kworker/2:222:18:122
2597084994941,5cyclictest2602575-21kworker/2:221:40:112
2597084994941,5cyclictest2602575-21kworker/2:221:39:252
2597084994941,5cyclictest2590305-21kworker/2:322:25:142
2597084994934,10cyclictest3467602-21systemd-journal21:00:002
2597084994934,10cyclictest3467602-21systemd-journal20:59:592
259708099496,12cyclictest0-21swapper/121:47:341
2597080994935,11cyclictest2677449-21sshd23:43:251
259707999493,3cyclictest131rcu_preempt22:13:090
259709099489,6cyclictest131rcu_preempt21:25:003
259709099481,7cyclictest0-21swapper/323:05:023
2597090994813,9cyclictest37-21ksoftirqd/323:17:383
2597084994840,5cyclictest2656818-21kworker/2:023:50:152
2597080994828,14cyclictest2702905-21ssh00:36:061
2597080994820,7cyclictest23-21ksoftirqd/123:47:081
2597079994835,7cyclictest309-21in:imuxsock23:10:010
259707999483,40cyclictest2675710-21modprobe23:40:000
2597079994825,5cyclictest0-21swapper/000:28:440
2597090994721,19cyclictest2601760-21cron20:40:003
2597084994740,4cyclictest2590305-21kworker/2:320:45:122
2597084994739,5cyclictest2656818-21kworker/2:023:28:402
2597084994738,5cyclictest2590305-21kworker/2:320:33:072
2597084994734,7cyclictest309-21in:imuxsock20:55:002
259708099475,7cyclictest23-21ksoftirqd/123:14:051
2597080994735,8cyclictest275-21dbus-daemon00:00:001
2597080994731,13cyclictest48-21kcompactd022:15:071
2597080994726,15cyclictest275-21dbus-daemon20:40:001
2597080994711,14cyclictest2671856-21ssh23:31:431
259707999475,9cyclictest0-21swapper/023:05:000
2597079994736,7cyclictest19350irq/34-mmc120:50:000
2597079994728,13cyclictest3467602-21systemd-journal22:00:020
259707999472,15cyclictest364-21snmpd22:46:400
2597079994720,12cyclictest2689877-21ssh00:09:230
2597090994633,10cyclictest2626926-21ssh21:58:543
2597090994632,11cyclictest2647952-21ssh22:42:013
2597090994630,13cyclictest2702480-21ssh00:35:123
2597090994630,13cyclictest2695356-21ssh00:20:203
2597090994630,11cyclictest1-21systemd19:35:013
2597090994629,14cyclictest2698241-21ssh00:26:313
2597090994629,14cyclictest2685949-21ssh00:00:493
2597090994629,13cyclictest2644897-21ssh22:35:483
2597090994629,13cyclictest2620745-21ssh21:45:313
259709099461,13cyclictest2600040-21sort20:05:023
2597090994610,6cyclictest131rcu_preempt21:40:003
2597084994638,5cyclictest2590305-21kworker/2:320:41:482
2597084994638,5cyclictest2590305-21kworker/2:319:55:552
2597084994638,5cyclictest2590305-21kworker/2:319:45:052
2597080994632,11cyclictest2640807-21ssh22:27:271
2597080994632,10cyclictest2602744-21cat20:55:011
2597080994631,12cyclictest2666594-21ssh23:20:311
2597080994631,12cyclictest2666594-21ssh23:20:311
2597080994630,11cyclictest275-21dbus-daemon20:25:011
2597080994629,13cyclictest3467602-21systemd-journal22:05:591
2597080994629,13cyclictest2692228-21ssh00:14:161
2597080994629,13cyclictest2623159-21ssh21:50:391
2597079994639,4cyclictest16850irq/134-800000023:20:020
259709099458,8cyclictest37-21ksoftirqd/300:05:493
259709099454,24cyclictest0-21swapper/319:15:053
2597090994535,7cyclictest210-21jbd2/mmcblk1p2-19:55:053
2597090994530,11cyclictest2670570-21sshd23:29:103
2597090994529,14cyclictest2633346-21ssh22:11:573
2597090994529,11cyclictest2603100-21munin-run21:05:013
259709099452,5cyclictest37-21ksoftirqd/320:40:073
2597084994538,4cyclictest2590305-21kworker/2:321:07:392
2597084994538,4cyclictest2590305-21kworker/2:321:07:382
2597084994538,4cyclictest2590305-21kworker/2:320:15:362
2597084994538,4cyclictest2590305-21kworker/2:319:41:392
2597084994537,5cyclictest2590305-21kworker/2:320:39:582
2597084994537,5cyclictest2590305-21kworker/2:320:13:482
2597084994537,5cyclictest2590305-21kworker/2:320:04:402
2597084994537,5cyclictest2590305-21kworker/2:319:38:232
2597084994537,5cyclictest2590305-21kworker/2:319:34:322
2597084994537,5cyclictest2590305-21kworker/2:319:17:222
2597080994528,11cyclictest2597304-21cron19:15:011
2597079994532,8cyclictest274-21cron20:20:010
2597079994527,14cyclictest2669926-21ssh23:27:490
2597090994430,10cyclictest2603326-21cron21:10:003
2597090994430,10cyclictest2603326-21cron21:09:593
2597090994427,13cyclictest2673576-21ssh23:35:103
2597090994425,15cyclictest2599384-21apt.systemd.dai19:54:003
2597084994437,4cyclictest2590305-21kworker/2:319:51:112
2597084994436,5cyclictest2590305-21kworker/2:319:25:032
259708099442,9cyclictest2600742-21latency_hist20:20:011
2597080994429,13cyclictest2688653-21ssh00:06:411
2597080994428,13cyclictest2664551-21ssh23:16:311
2597080994428,12cyclictest2606071-21ssh21:15:141
2597080994428,12cyclictest2606071-21ssh21:15:141
2597080994426,13cyclictest275-21dbus-daemon20:30:011
259708099441,6cyclictest23-21ksoftirqd/122:40:061
259707999442,13cyclictest2597330-21munin-run19:15:010
2597079994419,12cyclictest2623678-21ssh21:51:510
259707999441,39cyclictest2613030-21ssh21:29:570
2597090994327,12cyclictest2600994-21sshd20:22:363
2597090994325,13cyclictest275-21dbus-daemon20:20:013
2597084994336,4cyclictest2590305-21kworker/2:320:28:062
2597080994328,11cyclictest2597858-21munin-run19:25:011
2597080994328,11cyclictest2597858-21munin-run19:25:001
2597080994325,14cyclictest2602297-21cron20:50:011
259707999432,36cyclictest2597619-21/usr/sbin/munin19:20:000
259709099425,6cyclictest37-21ksoftirqd/320:11:443
2597090994230,8cyclictest2597285-21sshd19:11:063
2597090994229,9cyclictest2598637-21basename19:40:013
2597090994229,10cyclictest2601484-21sshd20:30:063
2597080994223,14cyclictest364-21snmpd21:00:291
2597080994222,16cyclictest364-21snmpd19:19:511
2597079994231,6cyclictest2602794-21cron21:00:000
2597079994231,6cyclictest2602794-21cron21:00:000
2597079994226,11cyclictest1-21systemd20:45:010
2597090994132,6cyclictest2600242-21munin-run20:10:023
2597090994129,8cyclictest2602266-21sshd20:45:063
2597090994127,9cyclictest48-21kcompactd019:29:313
2597084994133,5cyclictest2590305-21kworker/2:320:05:352
259708099417,6cyclictest131rcu_preempt19:30:011
2597080994127,11cyclictest2598336-21sshd19:32:271
2597079994134,4cyclictest16950irq/224-800000019:42:060
2597079994131,6cyclictest2599148-21grep19:50:000
2597079994130,6cyclictest274-21cron20:40:000
259707999412,8cyclictest309-21in:imuxsock20:05:000
2597090994028,9cyclictest2601242-21sshd20:27:383
2597090994026,11cyclictest1-21systemd19:25:013
2597090994026,11cyclictest1-21systemd19:25:013
2597080994027,9cyclictest2598655-21munin-run19:40:021
2597080994026,11cyclictest3467602-21systemd-journal20:01:001
2597080994026,10cyclictest2599667-21sshd19:57:061
2597079994028,9cyclictest2598343-21sshd19:34:060
2597079994025,12cyclictest2601338-21sed20:30:020
2597079994019,15cyclictest275-21dbus-daemon21:05:000
259709099399,7cyclictest0-21swapper/319:50:013
2597090993929,7cyclictest2602873-21sed21:00:023
2597090993929,7cyclictest2602873-21sed21:00:013
2597080993927,9cyclictest2600191-21sshd20:05:071
2597080993927,9cyclictest2599368-21sshd19:52:071
259707999391,3cyclictest131rcu_preempt20:25:000
2597080993825,8cyclictest48-21kcompactd020:57:451
2597080993825,8cyclictest48-21kcompactd020:57:451
2597079993826,9cyclictest2598092-21sshd19:29:070
2597079993817,16cyclictest3467602-21systemd-journal20:34:590
2597090993724,10cyclictest364-21snmpd19:41:273
2597080993725,9cyclictest364-21snmpd20:31:501
2597079993725,9cyclictest2597986-21sed19:25:020
2597079993725,9cyclictest2597986-21sed19:25:020
259707999372,31cyclictest3467602-21systemd-journal20:15:020
2597079993627,5cyclictest2598614-21sendmail19:40:000
259707999361,5cyclictest2597203-21latency_hist19:10:020
259707999361,3cyclictest150-21pr/ttyS220:09:050
2597090993510,21cyclictest2597036-21latency_hist19:10:023
2597079993524,8cyclictest12-21ksoftirqd/020:54:060
2597079993523,10cyclictest2599364-21sshd19:51:070
259708099348,21cyclictest2597207-21wc19:10:021
2597084992314,7cyclictest0-21swapper/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional