You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-20 - 13:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Sep 20, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3386101999971,25cyclictest23-21ksoftirqd/100:00:031
3386101999126,33cyclictest3438640-21sh22:49:501
3386101999126,33cyclictest3438640-21sh22:49:491
338610199903,11cyclictest131rcu_preempt21:15:031
3386101999025,34cyclictest3426863-21sh22:24:251
3386097998938,46cyclictest12-21ksoftirqd/019:40:000
3386101998819,34cyclictest3448431-21sh23:11:021
3386101998628,24cyclictest3447988-21latency_hist23:10:031
3386101998628,24cyclictest3447988-21latency_hist23:10:031
338609799841,81cyclictest0-21swapper/019:25:020
3386101998322,33cyclictest23-21ksoftirqd/123:52:151
338610199811,48cyclictest3398325-21ssh21:23:051
3386097997942,33cyclictest16950irq/224-800000000:00:010
338609799722,13cyclictest3450264-21cut23:15:020
3386097997125,13cyclictest3411506-21ssh21:51:130
3386101997029,37cyclictest3402454-21sh21:31:371
3386097997028,14cyclictest3447460-21ssh23:09:190
3386097997028,14cyclictest3447460-21ssh23:09:180
338609799701,37cyclictest3422340-21ssh22:14:560
338610199682,60cyclictest3484144-21ssh00:28:391
3386097996824,9cyclictest3482075-21ssh00:24:220
3386097996824,9cyclictest3482075-21ssh00:24:220
338610199672,60cyclictest3387624-21/usr/sbin/munin19:40:021
3386101996624,38cyclictest3485351-21sh00:30:541
3386101996624,38cyclictest3485351-21sh00:30:531
338610199661,13cyclictest3428912-21ssh22:29:001
338610199652,16cyclictest3409779-21ssh21:47:321
338610199651,18cyclictest3414869-21ssh21:58:421
338610199651,16cyclictest3441527-21ssh22:56:061
338610199651,16cyclictest3441527-21ssh22:56:051
3386097996416,14cyclictest3479341-21ssh00:18:120
3386101996329,30cyclictest3461591-21sh23:39:341
3386101996329,30cyclictest3461591-21sh23:39:341
338610199632,56cyclictest3475446-21munin-run00:10:001
338610199632,19cyclictest3423391-21ssh22:16:521
338610199632,16cyclictest3472650-21ssh00:03:461
338610199631,9cyclictest3479266-21ssh00:18:031
338610199631,9cyclictest3453498-21ssh23:22:021
3386101996318,40cyclictest23-21ksoftirqd/123:00:491
3386103996254,5cyclictest3443172-21kworker/2:200:05:032
338610199623,55cyclictest3395622-21ssh21:17:011
338610199623,55cyclictest3395622-21ssh21:17:001
3386101996227,31cyclictest3407577-21sh21:42:551
3386101996227,31cyclictest3407577-21sh21:42:541
338610199622,14cyclictest3419008-21ssh22:07:221
338610199621,9cyclictest3457408-21ssh23:30:151
338610199621,9cyclictest3432503-21ssh22:36:291
338610199621,7cyclictest3465780-21ssh23:48:191
338610199621,57cyclictest3429956-21ssh22:30:531
338610199621,19cyclictest3416057-21ssh22:00:591
338610199621,16cyclictest3451868-21ssh23:18:391
3386097996229,16cyclictest3406198-21ssh21:40:000
3386101996128,29cyclictest23-21ksoftirqd/100:23:241
3386101996128,29cyclictest23-21ksoftirqd/100:23:241
338610199612,7cyclictest3388661-21munin-run20:00:021
338610199611,8cyclictest3456535-21ssh23:28:431
338610199611,8cyclictest3456535-21ssh23:28:421
338610199611,8cyclictest3421448-21ssh22:12:391
338610199611,7cyclictest3439839-21ssh22:52:251
338610199611,20cyclictest3386135-21wc19:10:011
338610199611,14cyclictest3389163-21munin-run20:10:011
338610199602,6cyclictest3434600-21ssh22:40:421
3386101996025,31cyclictest3467562-21sshd20:20:241
338610199602,4cyclictest3462307-21ssh23:40:311
338610199601,7cyclictest3399695-21ssh21:25:371
338610199601,3cyclictest0-21swapper/120:35:021
338610199601,3cyclictest0-21swapper/120:35:011
338610199601,20cyclictest3467602-21systemd-journal19:30:011
338610199601,20cyclictest3467602-21systemd-journal19:30:001
3386103995951,5cyclictest3392287-21kworker/2:221:57:062
338610199592,18cyclictest3388368-21modprobe19:55:001
338610199591,9cyclictest3412560-21ssh21:53:471
338610199591,6cyclictest3488014-21ssh00:36:411
338610199591,53cyclictest3389366-21sshd20:10:241
3386103995848,7cyclictest3387088-21kworker/2:322:50:422
338610199582,50cyclictest48-21kcompactd019:19:341
338610199581,5cyclictest3405344-21ssh21:38:041
338610199581,4cyclictest3387831-21sshd19:41:251
338610199581,4cyclictest3387831-21sshd19:41:241
3386097995828,14cyclictest3483817-21ssh00:27:480
3386097995822,8cyclictest12-21ksoftirqd/000:08:260
3386097995817,26cyclictest16950irq/224-800000021:45:040
3386097995817,26cyclictest16950irq/224-800000021:45:040
338609799581,53cyclictest3413626-21ssh21:55:450
338609799581,40cyclictest3451663-21ssh23:18:140
338610199571,8cyclictest3386288-21munin-run19:15:011
338609799574,15cyclictest3443123-21modprobe23:00:000
338609799574,15cyclictest3443123-21modprobe23:00:000
3386097995726,14cyclictest3392985-21ssh21:11:180
3386103995647,6cyclictest3429432-21kworker/2:023:56:522
338610199562,49cyclictest0-21swapper/120:16:531
338610199562,19cyclictest3476627-21ssh00:12:201
338609799564,3cyclictest131rcu_preempt00:00:080
3386097995626,14cyclictest3432324-21ssh22:36:070
338609799561,39cyclictest3488948-21ssh00:38:490
3386103995547,5cyclictest3387088-21kworker/2:321:18:192
3386103995547,5cyclictest3387088-21kworker/2:321:18:192
3386103995546,6cyclictest3392287-21kworker/2:221:50:392
3386103995545,6cyclictest3387088-21kworker/2:321:21:242
338610199552,4cyclictest3392028-21munin-run21:05:021
338610199552,49cyclictest3391484-21munin-run20:54:591
338610199552,11cyclictest3391231-21munin-run20:50:011
3386103995446,5cyclictest3443172-21kworker/2:200:05:332
3386103995446,5cyclictest3392287-21kworker/2:222:16:042
3386103995445,6cyclictest3443172-21kworker/2:200:33:112
3386103995445,6cyclictest3443172-21kworker/2:200:33:102
338610199543,45cyclictest364-21snmpd19:46:201
338610199542,47cyclictest3391785-21latency_hist21:00:011
338609799545,6cyclictest131rcu_preempt23:33:400
3386097995429,10cyclictest3477449-21ssh00:14:190
3386103995345,5cyclictest3443172-21kworker/2:223:44:022
3386103995345,5cyclictest3387088-21kworker/2:322:24:442
3386103995344,6cyclictest3443172-21kworker/2:223:22:442
3386103995344,6cyclictest3392287-21kworker/2:222:11:002
3386103995344,6cyclictest3392287-21kworker/2:221:32:462
3386103995344,5cyclictest3429432-21kworker/2:022:44:152
3386103995344,5cyclictest3387088-21kworker/2:322:47:202
3386103995344,5cyclictest3387088-21kworker/2:322:47:192
3386103995343,6cyclictest3443172-21kworker/2:223:18:292
338610199532,7cyclictest364-21snmpd20:41:571
338610199531,4cyclictest3387294-21sshd19:30:241
3386097995327,11cyclictest3398275-21ssh21:22:580
3386097995326,12cyclictest3425927-21ssh22:22:180
3386097995324,13cyclictest3429653-21ssh22:30:040
338609799531,36cyclictest3409706-21ssh21:47:230
3386106995235,13cyclictest3406148-21cron21:40:003
3386106995235,12cyclictest3467602-21systemd-journal19:40:003
3386106995218,27cyclictest3417838-21grep22:05:013
3386103995244,5cyclictest3443172-21kworker/2:223:54:352
3386103995244,5cyclictest3443172-21kworker/2:223:29:042
3386103995244,5cyclictest3443172-21kworker/2:223:29:042
3386103995244,5cyclictest3387088-21kworker/2:322:58:472
3386103995244,5cyclictest3387088-21kworker/2:322:58:462
3386103995243,5cyclictest3443172-21kworker/2:200:15:442
3386103995243,5cyclictest3429432-21kworker/2:022:32:222
3386103995243,5cyclictest3392287-21kworker/2:221:26:492
3386103995242,6cyclictest3443172-21kworker/2:223:35:242
3386103995242,6cyclictest3443172-21kworker/2:223:35:232
3386103995242,6cyclictest3443172-21kworker/2:223:30:342
3386103995241,6cyclictest3392287-21kworker/2:222:01:162
3386103995241,6cyclictest3377979-21kworker/2:019:33:232
3386101995226,23cyclictest3392265-21latency_hist21:10:021
338610199521,3cyclictest150-21pr/ttyS219:22:471
3386097995234,13cyclictest1-21systemd21:00:010
3386097995228,9cyclictest3468587-21ssh23:54:380
3386097995227,9cyclictest3453691-21ssh23:22:240
3386097995224,13cyclictest3440570-21ssh22:54:140
3386097995217,31cyclictest3461823-21latency_hist23:40:020
3386097995217,31cyclictest3461823-21latency_hist23:40:010
3386103995143,5cyclictest3443172-21kworker/2:200:27:342
3386103995143,5cyclictest3443172-21kworker/2:200:21:522
3386103995143,5cyclictest3443172-21kworker/2:200:21:512
3386103995143,5cyclictest3443172-21kworker/2:200:10:212
3386103995143,5cyclictest3429432-21kworker/2:022:36:322
3386103995143,5cyclictest3387088-21kworker/2:321:14:122
3386103995142,6cyclictest3443172-21kworker/2:223:48:562
3386103995142,6cyclictest3443172-21kworker/2:200:35:542
3386103995142,6cyclictest3429432-21kworker/2:023:01:122
3386103995142,5cyclictest3387088-21kworker/2:321:10:002
338610199511,4cyclictest3390673-21sshd20:37:371
338610199511,3cyclictest48-21kcompactd020:00:361
3386097995133,3cyclictest16850irq/134-800000023:45:010
3386097995112,34cyclictest0-21swapper/022:10:020
3386103995042,5cyclictest3392287-21kworker/2:221:41:422
3386103995042,5cyclictest3392287-21kworker/2:221:41:412
3386103995042,5cyclictest3387088-21kworker/2:322:26:412
3386103995042,5cyclictest3387088-21kworker/2:320:10:022
3386103995042,5cyclictest3377979-21kworker/2:019:40:012
3386103995041,6cyclictest3443172-21kworker/2:223:11:562
3386103995041,6cyclictest3443172-21kworker/2:223:07:552
3386103995041,6cyclictest3443172-21kworker/2:223:07:542
338610199501,11cyclictest364-21snmpd20:26:001
3386097995042,4cyclictest16950irq/224-800000021:10:010
3386097995020,15cyclictest3466048-21ssh23:49:000
3386103994941,5cyclictest3392287-21kworker/2:221:36:272
3386103994941,5cyclictest3387088-21kworker/2:320:52:402
3386097994923,6cyclictest0-21swapper/022:15:560
3386097994922,5cyclictest3400072-21sh21:26:380
3386097994916,29cyclictest12-21ksoftirqd/022:02:020
3386097994916,28cyclictest0-21swapper/000:30:490
3386097994916,28cyclictest0-21swapper/000:30:490
3386103994840,5cyclictest3392287-21kworker/2:222:06:222
3386103994840,5cyclictest3392287-21kworker/2:221:45:502
3386103994840,5cyclictest3387088-21kworker/2:320:35:032
3386103994840,5cyclictest3387088-21kworker/2:320:35:022
3386097994824,9cyclictest3402207-21ssh21:31:070
3386106994731,11cyclictest3467602-21systemd-journal20:29:593
3386106994728,16cyclictest1-21systemd22:15:013
3386103994739,5cyclictest3387088-21kworker/2:319:50:022
3386103994730,13cyclictest3467602-21systemd-journal20:10:022
3386103994730,12cyclictest3467602-21systemd-journal19:25:032
3386103994730,12cyclictest3467602-21systemd-journal19:25:022
338609799472,7cyclictest3445475-21munin-run23:05:000
338610699464,4cyclictest131rcu_preempt22:22:273
338610699462,6cyclictest37-21ksoftirqd/323:13:593
338610699461,3cyclictest131rcu_preempt00:00:013
3386103994632,10cyclictest3388631-21systemctl20:00:002
3386097994639,4cyclictest16950irq/224-800000022:45:000
3386097994631,11cyclictest3390111-21latency_hist20:25:020
3386097994628,14cyclictest3455941-21ssh23:27:150
3386097994628,14cyclictest3455941-21ssh23:27:150
3386097994627,15cyclictest3427768-21ssh22:26:110
3386097994616,10cyclictest12-21ksoftirqd/021:15:400
3386097994616,10cyclictest12-21ksoftirqd/021:15:390
338610699452,7cyclictest131rcu_preempt22:30:013
3386106994525,14cyclictest275-21dbus-daemon22:00:003
338610399459,32cyclictest3390303-21sed20:30:012
3386097994536,5cyclictest151rcuc/020:15:030
3386106994432,7cyclictest3389651-21cron20:20:003
3386106994429,11cyclictest3424677-21cron22:20:003
3386106994429,10cyclictest275-21dbus-daemon22:50:013
3386106994429,10cyclictest275-21dbus-daemon22:50:013
3386106994428,13cyclictest1-21systemd00:05:013
338610399442,38cyclictest1-21systemd19:45:022
338610399442,38cyclictest1-21systemd19:45:012
3386097994437,4cyclictest16950irq/224-800000020:45:010
3386097994428,13cyclictest3437784-21ssh22:47:450
3386097994428,13cyclictest3437784-21ssh22:47:450
338610699434,4cyclictest131rcu_preempt21:47:313
338610699433,3cyclictest131rcu_preempt23:18:553
3386106994333,6cyclictest3479254-21sshd00:18:013
3386106994332,6cyclictest309-21in:imuxsock23:40:003
3386106994332,6cyclictest309-21in:imuxsock23:39:593
3386106994331,9cyclictest3467602-21systemd-journal00:25:013
3386106994331,9cyclictest3467602-21systemd-journal00:25:013
3386106994325,12cyclictest275-21dbus-daemon22:10:013
338610699431,4cyclictest131rcu_preempt23:04:233
3386106994310,28cyclictest131rcu_preempt22:45:023
3386103994335,5cyclictest3387088-21kworker/2:320:40:022
3386103994329,10cyclictest1-21systemd19:25:012
338610399432,37cyclictest1-21systemd20:50:002
3386097994331,9cyclictest3467602-21systemd-journal20:12:270
3386097994329,10cyclictest3386560-21systemctl19:20:000
3386097994328,11cyclictest3387092-21munin-run19:30:010
3386097994328,11cyclictest3387092-21munin-run19:30:010
3386097994327,12cyclictest3386098-21latency_hist19:10:020
3386106994230,9cyclictest37-21ksoftirqd/323:35:013
3386106994227,12cyclictest275-21dbus-daemon19:25:493
3386106994227,12cyclictest275-21dbus-daemon19:25:483
338610699421,4cyclictest131rcu_preempt23:41:093
338610699421,3cyclictest131rcu_preempt21:22:353
338610699421,3cyclictest131rcu_preempt00:29:203
3386103994234,5cyclictest3387088-21kworker/2:321:05:022
3386103994234,5cyclictest3387088-21kworker/2:320:16:402
3386103994230,7cyclictest3386557-21modprobe19:20:012
3386097994231,6cyclictest309-21in:imuxsock20:50:010
338610699414,4cyclictest131rcu_preempt22:34:333
338610699413,3cyclictest131rcu_preempt22:37:303
3386106994131,8cyclictest37-21ksoftirqd/300:15:033
3386106994128,9cyclictest1-21systemd00:10:013
3386106994127,10cyclictest37-21ksoftirqd/322:53:573
338610699412,4cyclictest131rcu_preempt21:41:043
338610699412,4cyclictest131rcu_preempt21:41:033
3386106994124,13cyclictest37-21ksoftirqd/319:20:003
3386106994120,15cyclictest275-21dbus-daemon19:15:013
3386103994133,5cyclictest3387088-21kworker/2:319:55:022
3386097994123,15cyclictest3467602-21systemd-journal20:54:590
338609799411,35cyclictest3467718-21lldpd20:29:590
3386106994027,8cyclictest1-21systemd19:55:013
3386106994027,10cyclictest37-21ksoftirqd/321:50:023
3386106994022,7cyclictest3403877-21sh21:34:573
338610699401,4cyclictest131rcu_preempt20:05:003
3386106994012,5cyclictest131rcu_preempt21:16:433
3386106994012,5cyclictest131rcu_preempt21:16:423
3386097994027,8cyclictest309-21in:imuxsock19:15:010
3386097994026,10cyclictest275-21dbus-daemon20:05:000
338610699395,3cyclictest131rcu_preempt00:35:053
3386106993927,9cyclictest37-21ksoftirqd/323:47:233
3386106993925,10cyclictest1-21systemd20:25:013
3386106993924,11cyclictest37-21ksoftirqd/300:31:053
3386106993924,11cyclictest37-21ksoftirqd/300:31:053
3386103993932,4cyclictest3387088-21kworker/2:320:55:392
3386103993926,9cyclictest3467602-21systemd-journal20:05:002
3386097993933,3cyclictest16950irq/224-800000020:06:240
3386097993931,5cyclictest19350irq/34-mmc120:34:370
3386097993931,5cyclictest19350irq/34-mmc120:34:360
3386097993923,10cyclictest3467602-21systemd-journal19:45:020
3386097993923,10cyclictest3467602-21systemd-journal19:45:010
3386097993917,18cyclictest16950irq/224-800000021:01:180
338609799391,35cyclictest150-21pr/ttyS220:39:590
338610699384,6cyclictest131rcu_preempt20:15:023
338610699384,3cyclictest131rcu_preempt23:28:463
338610699384,3cyclictest131rcu_preempt23:28:453
338610699383,5cyclictest131rcu_preempt23:00:003
338610699383,5cyclictest131rcu_preempt23:00:003
3386106993828,8cyclictest37-21ksoftirqd/319:20:023
3386106993826,9cyclictest3392251-21cron21:10:003
3386106993826,10cyclictest37-21ksoftirqd/323:51:293
338610699382,4cyclictest131rcu_preempt21:25:033
338610699382,3cyclictest131rcu_preempt23:21:483
338609799386,10cyclictest12-21ksoftirqd/019:55:010
338609799381,33cyclictest3467602-21systemd-journal19:49:590
338610699374,4cyclictest131rcu_preempt20:40:003
3386106993725,8cyclictest309-21in:imuxsock20:08:243
3386106993724,9cyclictest3388758-21date20:00:033
3386106993723,10cyclictest37-21ksoftirqd/323:05:043
3386106993723,10cyclictest37-21ksoftirqd/323:05:043
3386106993723,10cyclictest37-21ksoftirqd/321:10:223
3386106993723,10cyclictest275-21dbus-daemon19:45:023
3386106993723,10cyclictest275-21dbus-daemon19:45:013
3386106993722,10cyclictest275-21dbus-daemon19:50:003
3386103993726,8cyclictest3390136-21sed20:25:012
338610699363,3cyclictest131rcu_preempt21:00:023
338610699361,3cyclictest131rcu_preempt20:50:023
338609799362,31cyclictest0-21swapper/019:58:260
3386097993620,12cyclictest3387303-21sshd19:32:200
3386106993522,9cyclictest37-21ksoftirqd/320:50:243
3386106993518,13cyclictest275-21dbus-daemon20:35:013
3386106993518,13cyclictest275-21dbus-daemon20:35:003
338610699343,3cyclictest131rcu_preempt21:00:023
338610699343,3cyclictest131rcu_preempt20:45:013
3386103993427,5cyclictest3387088-21kworker/2:320:44:282
338610699334,3cyclictest131rcu_preempt19:35:003
3386103993320,10cyclictest30-21ksoftirqd/219:11:342
3386103993118,9cyclictest30-21ksoftirqd/219:10:022
3386106992417,4cyclictest3386132-21latency_hist19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional