You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-26 - 19:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Apr 26, 2026 12:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31470809910288,7cyclictest0-21swapper/010:40:030
3147080998532,13cyclictest3177512-21ssh10:06:070
3147080998030,47cyclictest16850irq/134-800000010:28:100
3147080997517,18cyclictest16850irq/134-800000012:20:190
3147080997225,12cyclictest3169556-21ssh09:47:400
3147080997225,12cyclictest3169556-21ssh09:47:390
3147080997219,18cyclictest16850irq/134-800000010:58:370
3147080996922,15cyclictest3223855-21ssh11:55:010
3147080996819,13cyclictest3208066-21ssh11:17:400
3147080996819,13cyclictest3208066-21ssh11:17:400
3147092996714,11cyclictest275-21dbus-daemon11:55:013
3147085996728,15cyclictest23-21ksoftirqd/107:19:591
3147085996723,12cyclictest23-21ksoftirqd/111:10:011
3147080996721,13cyclictest3182694-21ssh10:18:360
3147092996621,10cyclictest275-21dbus-daemon12:25:013
3147080996616,6cyclictest16850irq/134-800000009:37:190
3147092996518,9cyclictest3161723-21sshd09:29:433
3147080996517,17cyclictest16850irq/134-800000011:13:170
3147080996512,32cyclictest12-21ksoftirqd/009:24:500
3147092996421,4cyclictest3211195-21latency_hist11:25:023
3147092996420,5cyclictest3242981-21latency_hist12:40:033
3147089996421,40cyclictest301ksoftirqd/209:20:012
3147092996320,4cyclictest3182352-21sh10:17:403
3147092996228,30cyclictest3228932-21sh12:06:313
3147092996226,32cyclictest3219498-21sh11:44:403
3147080996257,3cyclictest0-21swapper/011:25:020
3147080996242,7cyclictest12-21ksoftirqd/009:10:580
3147092996121,6cyclictest3232475-21sed12:15:013
3147092996118,4cyclictest3175303-21sh10:00:523
3147092996117,6cyclictest309-21in:imuxsock08:05:003
314708099612,50cyclictest3153269-21grep09:10:000
314708099612,50cyclictest3153269-21grep09:10:000
3147092996024,31cyclictest3178084-21sh10:07:413
3147085996020,11cyclictest23-21ksoftirqd/109:45:011
3147092995927,28cyclictest3154320-21sh09:12:093
3147092995922,4cyclictest3238791-21latency_hist12:30:023
3147085995939,15cyclictest3208928-21systemctl11:20:001
3147085995939,15cyclictest3208928-21systemctl11:20:001
3147092995823,4cyclictest3200577-21latency_hist11:00:023
314709299582,19cyclictest3209087-21latency_hist11:20:023
314709299582,19cyclictest3209087-21latency_hist11:20:023
3147092995820,12cyclictest3152745-21sendmail09:00:013
3147092995813,10cyclictest1-21systemd10:25:013
3147089995848,5cyclictest3151468-21kworker/2:311:23:342
3147089995846,8cyclictest3166210-21kworker/2:210:31:282
314709299574,17cyclictest3147580-21unin-run07:20:003
3147092995718,7cyclictest3196258-21wc10:50:013
3147092995715,10cyclictest37-21ksoftirqd/310:54:523
3147089995749,5cyclictest3134147-21kworker/2:007:35:002
3147085995711,6cyclictest131rcu_preempt09:11:571
3147080995720,33cyclictest12-21ksoftirqd/012:12:520
3147092995623,9cyclictest3206653-21sshd11:14:433
3147092995620,4cyclictest3153261-21diskmemload11:48:023
3147092995619,8cyclictest37-21ksoftirqd/310:12:423
3147092995617,5cyclictest3241025-21latency_hist12:35:023
3147092995615,11cyclictest37-21ksoftirqd/312:01:343
3147089995649,4cyclictest3234537-21kworker/2:212:36:462
3147089995648,5cyclictest3166210-21kworker/2:210:09:322
3147089995648,5cyclictest3151468-21kworker/2:310:52:362
3147089995647,5cyclictest3134147-21kworker/2:009:33:122
314708599565,8cyclictest23-21ksoftirqd/110:10:021
3147080995621,31cyclictest12-21ksoftirqd/010:50:140
3147080995619,18cyclictest16850irq/134-800000012:05:220
314708099561,39cyclictest3170983-21ssh09:50:500
3147089995547,5cyclictest3166210-21kworker/2:210:17:142
3147089995546,6cyclictest3151468-21kworker/2:309:55:432
3147089995546,6cyclictest3151468-21kworker/2:309:55:422
3147089995546,5cyclictest3151468-21kworker/2:311:10:012
314708599555,4cyclictest131rcu_preempt09:20:011
3147085995516,10cyclictest23-21ksoftirqd/110:35:011
3147080995548,4cyclictest3238913-21latency_hist12:30:030
3147080995528,11cyclictest3184170-21ssh10:21:440
314709299548,13cyclictest3148868-21systemd-run07:45:003
314709299542,19cyclictest3204700-21wc11:10:013
3147092995421,5cyclictest3157538-21sendmail09:20:003
3147092995419,8cyclictest37-21ksoftirqd/310:30:013
3147092995418,7cyclictest0-21swapper/309:53:443
3147092995417,7cyclictest3193921-21sshd10:44:523
3147089995446,5cyclictest3196121-21kworker/2:112:15:052
3147089995446,5cyclictest3166210-21kworker/2:210:24:152
3147089995446,5cyclictest3166210-21kworker/2:210:11:352
3147089995445,6cyclictest3151468-21kworker/2:311:00:002
3147089995445,5cyclictest3151468-21kworker/2:309:41:302
314708599544,9cyclictest23-21ksoftirqd/112:13:541
314708099545,43cyclictest19350irq/34-mmc108:55:010
3147080995428,12cyclictest3201171-21ssh11:01:320
3147080995426,12cyclictest3224343-21ssh11:55:410
3147080995419,6cyclictest16850irq/134-800000010:10:240
3147080995419,18cyclictest16850irq/134-800000009:27:380
3147080995419,17cyclictest16850irq/134-800000012:15:110
3147080995418,7cyclictest16850irq/134-800000010:36:490
3147080995418,7cyclictest16850irq/134-800000010:36:490
3147080995418,6cyclictest16850irq/134-800000009:15:070
3147080995417,7cyclictest3168481-21latency_hist09:45:020
3147092995324,25cyclictest37-21ksoftirqd/309:35:523
3147092995320,7cyclictest37-21ksoftirqd/308:21:443
3147092995318,4cyclictest3151521-21latency_hist08:35:023
3147089995345,5cyclictest3196121-21kworker/2:112:30:112
3147089995345,5cyclictest3151468-21kworker/2:311:39:362
3147089995345,5cyclictest3151468-21kworker/2:311:39:362
3147089995345,5cyclictest3151468-21kworker/2:311:15:432
3147089995345,5cyclictest3151468-21kworker/2:311:15:432
3147089995345,5cyclictest3151468-21kworker/2:309:49:312
3147089995345,5cyclictest3151468-21kworker/2:309:49:312
3147089995345,5cyclictest3134147-21kworker/2:008:29:432
3147089995344,6cyclictest3196121-21kworker/2:112:10:102
3147089995344,6cyclictest3151468-21kworker/2:311:12:222
3147089995344,6cyclictest3151468-21kworker/2:309:38:522
3147089995344,6cyclictest3134147-21kworker/2:009:26:322
3147089995337,11cyclictest3467602-21systemd-journal07:39:592
314708599537,12cyclictest23-21ksoftirqd/111:52:031
314708599536,6cyclictest131rcu_preempt11:28:081
314708599536,6cyclictest131rcu_preempt11:28:081
314708599534,6cyclictest131rcu_preempt11:40:031
314708599534,6cyclictest131rcu_preempt11:40:031
3147085995316,13cyclictest23-21ksoftirqd/111:25:001
3147080995318,6cyclictest16850irq/134-800000011:47:030
3147080995318,18cyclictest16850irq/134-800000012:01:590
3147080995317,6cyclictest16850irq/134-800000012:30:340
3147080995317,6cyclictest16850irq/134-800000011:42:000
3147092995222,7cyclictest37-21ksoftirqd/312:16:053
3147092995220,5cyclictest3151888-21latency_hist08:40:023
3147092995218,7cyclictest37-21ksoftirqd/309:55:423
3147092995218,7cyclictest37-21ksoftirqd/309:55:413
3147092995216,10cyclictest3186941-1kworker/u9:110:30:503
3147089995244,5cyclictest3196121-21kworker/2:112:02:332
3147089995244,5cyclictest3196121-21kworker/2:111:48:092
3147089995244,5cyclictest3134147-21kworker/2:009:15:282
3147089995244,5cyclictest3134147-21kworker/2:009:12:232
3147089995242,6cyclictest3134147-21kworker/2:007:15:032
314708599525,9cyclictest23-21ksoftirqd/112:34:461
3147085995235,13cyclictest1-21systemd07:49:591
3147080995226,22cyclictest3204049-21ssh11:08:330
3147080995225,5cyclictest3189648-21latency_hist10:35:020
3147080995217,6cyclictest16850irq/134-800000009:34:050
3147080995217,17cyclictest16850irq/134-800000011:34:070
3147080995216,18cyclictest16850irq/134-800000010:49:420
3147092995119,6cyclictest3170416-21latency_hist09:50:023
3147092995119,6cyclictest3170416-21latency_hist09:50:023
3147089995143,5cyclictest3196121-21kworker/2:112:20:192
3147089995143,5cyclictest3196121-21kworker/2:111:26:362
3147089995143,5cyclictest3196121-21kworker/2:111:26:362
3147089995143,5cyclictest3166210-21kworker/2:210:28:102
3147089995142,5cyclictest3151468-21kworker/2:309:51:202
3147085995131,14cyclictest309-21in:imuxsock10:25:031
3147080995124,12cyclictest3216022-21ssh11:36:090
3147080995124,12cyclictest3216022-21ssh11:36:090
3147092995038,8cyclictest3467602-21systemd-journal09:10:003
3147092995038,8cyclictest3467602-21systemd-journal09:09:593
3147092995020,7cyclictest37-21ksoftirqd/311:33:553
3147092995018,6cyclictest3159011-21rm09:23:083
3147089995043,5cyclictest3151468-21kworker/2:310:03:382
3147089995042,5cyclictest3196121-21kworker/2:112:05:512
3147089995042,5cyclictest3196121-21kworker/2:111:30:512
3147089995042,5cyclictest3134147-21kworker/2:007:10:022
3147089995041,6cyclictest3151468-21kworker/2:310:37:452
3147089995041,6cyclictest3151468-21kworker/2:310:37:452
3147085995035,11cyclictest3467602-21systemd-journal12:38:431
314708599502,6cyclictest23-21ksoftirqd/110:25:011
3147085995025,18cyclictest1-21systemd10:40:011
3147085995025,18cyclictest1-21systemd10:40:011
3147085995021,10cyclictest23-21ksoftirqd/112:23:071
314708099503,37cyclictest364-21snmpd11:30:020
314708099503,37cyclictest364-21snmpd11:30:020
3147092994938,9cyclictest37-21ksoftirqd/311:05:023
3147092994915,8cyclictest37-21ksoftirqd/309:40:173
3147089994941,5cyclictest3196121-21kworker/2:112:26:012
3147089994941,5cyclictest3196121-21kworker/2:111:54:002
3147089994941,5cyclictest3151468-21kworker/2:310:45:582
3147089994940,6cyclictest3134147-21kworker/2:008:35:012
3147089994939,6cyclictest3134147-21kworker/2:008:55:012
314708599498,9cyclictest23-21ksoftirqd/112:27:441
3147085994929,14cyclictest3467602-21systemd-journal08:35:001
314708599491,9cyclictest0-21swapper/111:45:021
3147085994914,6cyclictest131rcu_preempt08:20:011
314708599491,10cyclictest23-21ksoftirqd/111:55:081
3147080994941,5cyclictest16950irq/224-800000008:20:000
3147080994935,9cyclictest3467602-21systemd-journal08:00:010
314708099492,7cyclictest3153001-21modprobe09:05:000
3147092994822,10cyclictest37-21ksoftirqd/309:30:173
3147089994840,5cyclictest3196121-21kworker/2:111:56:432
3147089994840,5cyclictest3151468-21kworker/2:311:43:212
3147089994840,4cyclictest3134147-21kworker/2:008:25:012
3147089994839,6cyclictest3166210-21kworker/2:210:41:202
314708599489,7cyclictest23-21ksoftirqd/108:55:021
314708599488,6cyclictest131rcu_preempt12:05:021
314708599487,9cyclictest23-21ksoftirqd/110:00:001
314708599487,9cyclictest23-21ksoftirqd/110:00:001
314708599487,8cyclictest23-21ksoftirqd/110:12:431
314708599487,11cyclictest23-21ksoftirqd/112:18:201
314708599487,11cyclictest23-21ksoftirqd/107:30:011
3147085994837,8cyclictest221rcuc/111:15:021
3147085994837,7cyclictest23-21ksoftirqd/109:50:001
3147085994837,7cyclictest23-21ksoftirqd/109:50:001
3147085994825,16cyclictest275-21dbus-daemon09:40:011
3147085994811,8cyclictest23-21ksoftirqd/111:04:031
314708099488,4cyclictest131rcu_preempt12:36:100
3147080994812,5cyclictest131rcu_preempt09:56:460
3147080994812,5cyclictest131rcu_preempt09:56:460
314709299473,15cyclictest3467602-21systemd-journal11:40:003
314709299473,15cyclictest3467602-21systemd-journal11:40:003
3147089994740,5cyclictest3151468-21kworker/2:311:01:402
3147089994739,5cyclictest3134147-21kworker/2:009:10:022
3147089994739,5cyclictest3134147-21kworker/2:009:10:022
3147089994739,5cyclictest3134147-21kworker/2:008:47:292
3147089994738,6cyclictest3134147-21kworker/2:008:43:432
3147089994738,5cyclictest3134147-21kworker/2:008:10:002
3147089994738,5cyclictest3134147-21kworker/2:007:30:012
314708599476,8cyclictest23-21ksoftirqd/109:52:431
3147085994737,7cyclictest23-21ksoftirqd/110:19:211
3147085994734,9cyclictest23-21ksoftirqd/107:34:591
3147085994719,5cyclictest131rcu_preempt08:10:021
3147085994712,8cyclictest23-21ksoftirqd/110:48:231
3147085994712,5cyclictest131rcu_preempt10:41:341
3147089994636,6cyclictest3134147-21kworker/2:007:25:002
314708599466,5cyclictest131rcu_preempt11:31:381
314708599464,10cyclictest23-21ksoftirqd/109:21:561
3147085994637,6cyclictest23-21ksoftirqd/111:47:591
3147085994634,9cyclictest23-21ksoftirqd/109:26:081
3147085994633,10cyclictest23-21ksoftirqd/110:51:121
3147085994611,9cyclictest23-21ksoftirqd/110:04:371
3147080994622,20cyclictest3152953-21latency_hist09:00:030
3147080994621,22cyclictest12-21ksoftirqd/007:15:010
3147080994619,11cyclictest3176372-21ssh10:03:490
3147089994537,5cyclictest3134147-21kworker/2:009:01:502
3147089994537,5cyclictest3134147-21kworker/2:007:54:592
3147089994536,5cyclictest3134147-21kworker/2:007:45:022
3147089994521,20cyclictest30-21ksoftirqd/208:20:022
3147085994533,6cyclictest309-21in:imuxsock09:05:011
3147085994520,15cyclictest23-21ksoftirqd/112:07:381
3147085994516,5cyclictest131rcu_preempt09:32:071
3147092994430,9cyclictest275-21dbus-daemon08:50:013
314708999444,34cyclictest3149368-21kworker/u8:409:00:022
3147089994435,5cyclictest3134147-21kworker/2:007:45:032
314708599449,5cyclictest131rcu_preempt08:45:001
314708599441,3cyclictest131rcu_preempt07:45:011
3147092994326,12cyclictest309-21in:imuxsock07:55:013
3147089994330,8cyclictest30-21ksoftirqd/208:15:022
3147089994330,5cyclictest3134147-21kworker/2:007:15:002
314708599437,7cyclictest23-21ksoftirqd/109:10:021
314708599437,7cyclictest23-21ksoftirqd/109:10:021
3147085994330,10cyclictest23-21ksoftirqd/107:40:011
3147085994327,11cyclictest23-21ksoftirqd/111:00:001
3147085994312,11cyclictest23-21ksoftirqd/107:10:021
3147085994311,5cyclictest131rcu_preempt09:00:011
314708099431,13cyclictest3149255-21cut07:50:020
3147085994224,12cyclictest23-21ksoftirqd/108:00:011
314708599421,9cyclictest23-21ksoftirqd/108:30:001
3147080994232,7cyclictest12-21ksoftirqd/007:35:010
3147080994231,8cyclictest12-21ksoftirqd/008:15:020
3147080994230,8cyclictest3151427-21cron08:35:000
3147092994127,10cyclictest3467602-21systemd-journal11:59:593
3147092994125,11cyclictest1-21systemd08:55:003
3147089994134,5cyclictest3134147-21kworker/2:008:05:002
3147089994133,5cyclictest3134147-21kworker/2:007:55:102
3147085994112,6cyclictest131rcu_preempt07:25:011
314708099413,28cyclictest3147084-21latency_hist07:10:020
3147080994132,5cyclictest3151960-21cron08:45:010
3147092994026,11cyclictest275-21dbus-daemon08:15:003
3147085994027,9cyclictest23-21ksoftirqd/108:40:001
3147080994028,9cyclictest1-21systemd07:20:000
3147080994027,9cyclictest275-21dbus-daemon07:45:010
314708099402,30cyclictest16950irq/224-800000008:25:010
3147085993922,12cyclictest275-21dbus-daemon08:05:001
3147080993932,4cyclictest16950irq/224-800000007:25:000
3147080993922,11cyclictest0-21swapper/007:30:030
314709299388,6cyclictest131rcu_preempt08:45:023
314709299383,3cyclictest131rcu_preempt07:10:023
3147092993829,5cyclictest3191856-21sendmail10:40:003
3147092993829,5cyclictest3191856-21sendmail10:40:003
3147085993823,11cyclictest23-21ksoftirqd/108:46:591
3147080993828,7cyclictest3148805-21sed07:40:020
314708099382,7cyclictest3151735-21munin-run08:40:000
3147080993823,12cyclictest3467602-21systemd-journal08:30:010
3147080993817,15cyclictest275-21dbus-daemon07:55:010
314708099381,33cyclictest3152286-21wc08:50:020
314709299373,7cyclictest3148173-21latency_hist07:30:033
3147092993727,8cyclictest37-21ksoftirqd/311:25:053
3147092993727,8cyclictest37-21ksoftirqd/311:25:053
3147092993727,7cyclictest3148704-21cat07:40:023
3147092993722,11cyclictest37-21ksoftirqd/307:33:423
314708599374,24cyclictest0-21swapper/108:15:021
314709299365,4cyclictest131rcu_preempt07:25:013
3147092993622,11cyclictest37-21ksoftirqd/308:17:103
3147092993622,10cyclictest37-21ksoftirqd/307:15:013
314708599366,7cyclictest23-21ksoftirqd/107:15:001
3147085993627,6cyclictest23-21ksoftirqd/107:55:021
3147080993624,9cyclictest12-21ksoftirqd/008:05:010
314708099362,10cyclictest3467602-21systemd-journal08:09:550
3147092993519,12cyclictest3153205-21cat09:05:023
314708599356,5cyclictest131rcu_preempt08:25:001
3147092993421,9cyclictest37-21ksoftirqd/307:47:313
3147089993426,5cyclictest3134147-21kworker/2:008:39:042
314709299333,10cyclictest131rcu_preempt07:55:023
314709299316,3cyclictest131rcu_preempt08:30:003
3147092993124,4cyclictest3149889-21latency_hist08:05:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional