You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-25 - 14:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Apr 25, 2026 00:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
268627499841,53cyclictest2703661-21ssh21:35:242
2686274998120,32cyclictest2692485-21diskmemload00:08:062
2686274997433,37cyclictest3467602-21systemd-journal19:20:002
2686274997330,38cyclictest1-21systemd00:00:002
268627499681,21cyclictest2718153-21ssh22:09:262
2686262996858,7cyclictest1033191-1kworker/0:1H21:00:020
2686262996849,16cyclictest12-21ksoftirqd/021:23:430
268626799674,10cyclictest23-21ksoftirqd/122:00:001
268627499663,56cyclictest2703271-21cron21:35:002
268627499663,56cyclictest2703271-21cron21:35:002
268627499662,16cyclictest2741518-21ssh23:03:272
268627499661,18cyclictest2761268-21ssh23:49:312
268627499661,18cyclictest2761268-21ssh23:49:312
268627499661,18cyclictest2747724-21sh23:17:522
268627499651,14cyclictest2723030-21ssh22:20:242
268627499651,14cyclictest2694619-21rm21:14:482
268627499651,14cyclictest2694619-21rm21:14:482
268627499642,4cyclictest2738750-21ssh22:57:012
268627499642,14cyclictest2767516-21ssh00:03:472
268627499641,13cyclictest2773192-21ssh00:16:512
268627499638,51cyclictest2776984-21ssh00:25:302
268627499632,20cyclictest2726964-21munin-run22:30:002
268627499632,18cyclictest2744814-21ssh23:10:512
268627499632,15cyclictest2721510-21ssh22:17:042
268627499632,13cyclictest2695238-21ssh21:15:572
2686274996319,40cyclictest30-21ksoftirqd/223:53:112
268627499631,18cyclictest2754423-21ssh23:33:272
268627499622,18cyclictest2751603-21ssh23:26:492
268627499621,9cyclictest2749844-21ssh23:22:462
268627499621,9cyclictest2734546-21ssh22:47:202
268627499621,9cyclictest2734546-21ssh22:47:202
268627499621,9cyclictest2713632-21ssh21:58:482
268627499621,8cyclictest2781181-21ssh00:35:102
268627499621,8cyclictest2728554-21ssh22:33:412
268627499621,8cyclictest2710069-21ssh21:50:142
268627499621,7cyclictest2714397-21ssh22:00:142
268627499621,5cyclictest2774670-21ssh00:20:062
268627499621,19cyclictest2755569-21ssh23:36:002
268627499621,18cyclictest2720220-21ssh22:14:112
268627499621,10cyclictest2733185-21ssh22:44:262
268627499621,10cyclictest2706915-21ssh21:43:112
268627499621,10cyclictest2706915-21ssh21:43:112
2686262996222,36cyclictest12-21ksoftirqd/021:12:100
2686262996222,36cyclictest12-21ksoftirqd/021:12:100
268627499611,9cyclictest2779166-21ssh00:30:372
268627499611,9cyclictest2770351-21ssh00:10:042
268627499611,8cyclictest2708312-21ssh21:46:172
268627499611,8cyclictest2697175-21ssh21:20:172
268627499611,7cyclictest2736196-21ssh22:51:012
268627499611,6cyclictest2757601-21ssh23:40:312
268627499611,19cyclictest2729776-21ssh22:36:102
268627499611,16cyclictest2688086-21munin-run19:45:002
268627499601,8cyclictest2700347-21ssh21:27:582
268627499601,4cyclictest2743010-21ssh23:06:482
268627499601,4cyclictest2743010-21ssh23:06:482
2686262996041,4cyclictest16850irq/134-800000022:00:010
2686262996040,6cyclictest12-21ksoftirqd/023:15:270
2686262996029,13cyclictest2774394-21ssh00:20:000
268627499594,50cyclictest2688891-21wc20:00:012
268626299583,40cyclictest2709981-21ssh21:50:020
268627499572,5cyclictest2689133-21munin-run20:05:012
268627499572,5cyclictest2687790-21sshd19:39:232
268627499572,50cyclictest275-21dbus-daemon20:15:002
268627499572,12cyclictest364-21snmpd20:33:092
268627499572,12cyclictest2687554-21/usr/sbin/munin19:35:002
268626799578,5cyclictest131rcu_preempt00:18:541
2686262995728,13cyclictest2767090-21ssh00:02:450
268626299571,29cyclictest131rcu_preempt23:29:550
268627499561,5cyclictest364-21snmpd20:56:242
268627499561,4cyclictest2686336-21cat19:10:022
268627499561,12cyclictest364-21snmpd19:54:262
268627499561,12cyclictest364-21snmpd19:54:252
2686274995610,41cyclictest0-21swapper/219:28:402
2686262995634,19cyclictest16850irq/134-800000023:30:040
2686262995629,11cyclictest2771261-21ssh00:12:270
2686262995627,13cyclictest2765680-21ssh23:59:430
2686262995627,13cyclictest2725696-21ssh22:26:410
268627499553,44cyclictest2692471-21cron21:10:002
268627499552,5cyclictest364-21snmpd19:21:012
268627499552,5cyclictest364-21snmpd19:21:012
268627499552,5cyclictest2689385-21munin-run20:10:012
268627499551,9cyclictest2690397-21sshd20:29:232
268627499551,11cyclictest2692200-21sshd21:01:232
2686262995529,10cyclictest2702587-21ssh21:33:150
2686262995529,10cyclictest2702587-21ssh21:33:150
2686262995527,13cyclictest2757955-21ssh23:41:320
268627499543,45cyclictest2689872-21cron20:20:002
268627499543,45cyclictest2689872-21cron20:20:002
268627499542,6cyclictest2686462-21munin-run19:15:012
268627499542,6cyclictest2686462-21munin-run19:15:012
268627499542,46cyclictest2691453-21munin-plugin-st20:50:012
268627499541,12cyclictest364-21snmpd20:41:512
2686262995429,10cyclictest2742758-21ssh23:06:100
2686262995429,10cyclictest2742758-21ssh23:06:100
2686262995425,14cyclictest2762326-21ssh23:51:390
2686267995312,6cyclictest131rcu_preempt21:00:001
268626299533,3cyclictest131rcu_preempt23:22:300
268626299531,3cyclictest131rcu_preempt00:09:310
268627499522,5cyclictest364-21snmpd20:21:172
268627499522,4cyclictest0-21swapper/220:52:272
2686262995226,12cyclictest2742087-21ssh23:04:590
268627899517,32cyclictest2765794-40mandb00:00:043
2686278995137,10cyclictest1-21systemd19:55:013
2686278995137,10cyclictest1-21systemd19:55:013
2686278995113,9cyclictest131rcu_preempt19:20:003
2686267995139,7cyclictest23-21ksoftirqd/119:25:021
2686267995139,7cyclictest23-21ksoftirqd/119:25:021
2686262995142,5cyclictest16950irq/224-800000020:30:000
2686262995133,14cyclictest275-21dbus-daemon19:40:000
2686262995127,20cyclictest19350irq/34-mmc100:35:470
2686262995126,10cyclictest2731011-21ssh22:39:260
2686262995122,25cyclictest2717898-21ssh22:08:470
2686278995033,13cyclictest2774404-21systemctl00:20:003
268627499501,5cyclictest364-21snmpd19:47:492
268627499501,12cyclictest364-21snmpd20:36:572
2686267995040,6cyclictest23-21ksoftirqd/100:00:031
2686267995030,14cyclictest275-21dbus-daemon23:05:001
2686267995014,9cyclictest131rcu_preempt22:50:021
2686267995014,9cyclictest131rcu_preempt22:50:021
2686267995012,6cyclictest131rcu_preempt22:11:261
2686262995042,5cyclictest16850irq/134-800000021:30:000
2686262995041,5cyclictest2715426-21sh22:02:510
2686262995031,15cyclictest16850irq/134-800000022:35:000
2686262995025,10cyclictest2706810-21ssh21:42:560
2686262995025,10cyclictest2706810-21ssh21:42:560
268626299501,34cyclictest12-21ksoftirqd/023:00:010
2686278994935,10cyclictest2689894-21systemctl20:20:003
2686278994935,10cyclictest2689894-21systemctl20:20:003
2686278994929,15cyclictest275-21dbus-daemon22:40:013
2686267994936,9cyclictest23-21ksoftirqd/121:45:021
2686267994936,9cyclictest23-21ksoftirqd/121:45:021
2686267994933,11cyclictest275-21dbus-daemon19:40:011
2686267994930,14cyclictest3467602-21systemd-journal00:14:021
2686262994929,13cyclictest275-21dbus-daemon19:45:000
2686262994922,12cyclictest2724665-21ssh22:24:350
2686262994919,26cyclictest2760279-21sh23:46:510
2686262994919,26cyclictest2760279-21sh23:46:500
268626299491,3cyclictest131rcu_preempt00:23:330
2686267994831,12cyclictest23-21ksoftirqd/120:10:031
2686267994817,10cyclictest23-21ksoftirqd/100:33:511
268626799481,5cyclictest131rcu_preempt21:20:591
268626799481,5cyclictest131rcu_preempt00:21:151
2686267994813,5cyclictest131rcu_preempt22:34:011
2686267994813,10cyclictest23-21ksoftirqd/121:53:031
2686262994841,4cyclictest16850irq/134-800000000:35:000
2686262994840,4cyclictest16950irq/224-800000000:30:020
268626299483,25cyclictest131rcu_preempt21:15:110
2686262994828,15cyclictest3467602-21systemd-journal19:35:010
2686262994817,8cyclictest16850irq/134-800000021:10:000
2686278994722,15cyclictest37-21ksoftirqd/300:32:333
2686267994736,8cyclictest23-21ksoftirqd/100:35:361
2686267994729,13cyclictest3467602-21systemd-journal20:41:231
268626799471,7cyclictest2705483-21ssh21:40:001
2686262994733,10cyclictest12-21ksoftirqd/021:50:000
2686262994731,13cyclictest2704211-21ssh21:36:550
268626299472,8cyclictest2757152-21modprobe23:40:010
2686262994720,12cyclictest2744993-21ssh23:11:250
2686267994635,7cyclictest23-21ksoftirqd/123:50:011
2686267994635,7cyclictest23-21ksoftirqd/123:50:011
2686267994629,13cyclictest2757148-21systemctl23:40:001
2686267994624,11cyclictest2728929-1kworker/u9:222:40:481
268626299462,15cyclictest3467602-21systemd-journal22:11:450
2686278994531,11cyclictest2744238-21cron23:10:003
2686278994531,11cyclictest2744238-21cron23:10:003
2686278994525,15cyclictest275-21dbus-daemon22:25:013
2686278994522,17cyclictest275-21dbus-daemon21:40:003
268626799458,7cyclictest23-21ksoftirqd/100:00:451
268626799453,7cyclictest131rcu_preempt21:20:021
2686267994531,10cyclictest23-21ksoftirqd/122:20:261
2686267994531,10cyclictest23-21ksoftirqd/121:14:421
2686267994531,10cyclictest23-21ksoftirqd/121:14:421
2686267994528,13cyclictest1-21systemd23:00:011
2686267994527,14cyclictest2692276-21cut21:05:021
2686267994526,14cyclictest309-21in:imuxsock19:45:001
268626799452,10cyclictest23-21ksoftirqd/123:25:001
2686267994520,20cyclictest221rcuc/123:31:301
2686267994512,6cyclictest131rcu_preempt22:50:231
2686267994510,5cyclictest131rcu_preempt23:16:031
2686262994538,4cyclictest16950irq/224-800000020:35:030
2686262994528,13cyclictest2732146-21ssh22:41:410
268626299452,14cyclictest3467602-21systemd-journal20:51:230
268626299452,10cyclictest2721988-21sshd22:18:230
268626299451,16cyclictest2737941-21latency_hist22:55:020
268627899445,5cyclictest131rcu_preempt22:46:333
268627899445,5cyclictest131rcu_preempt22:46:333
2686278994429,11cyclictest1-21systemd00:10:013
2686278994425,13cyclictest275-21dbus-daemon21:45:023
2686278994425,13cyclictest275-21dbus-daemon21:45:013
2686278994425,12cyclictest309-21in:imuxsock00:30:003
2686267994433,7cyclictest2690917-21sh20:40:001
268626799441,5cyclictest131rcu_preempt22:17:091
2686267994411,6cyclictest131rcu_preempt23:11:101
2686262994437,4cyclictest16950irq/224-800000020:00:000
268627899431,6cyclictest131rcu_preempt20:40:003
268627899431,3cyclictest131rcu_preempt21:23:293
268626799439,6cyclictest131rcu_preempt21:45:141
268626799433,5cyclictest131rcu_preempt21:32:001
268626799433,5cyclictest131rcu_preempt21:32:001
2686267994333,7cyclictest23-21ksoftirqd/123:44:181
2686267994332,7cyclictest23-21ksoftirqd/122:35:151
2686267994331,9cyclictest23-21ksoftirqd/100:30:001
2686267994330,9cyclictest23-21ksoftirqd/119:55:041
2686267994323,15cyclictest275-21dbus-daemon20:00:001
2686267994312,6cyclictest131rcu_preempt23:09:521
2686267994312,6cyclictest131rcu_preempt23:09:511
2686262994336,4cyclictest16950irq/224-800000020:25:000
268626299431,12cyclictest2689368-21latency_hist20:10:030
2686278994231,8cyclictest1-21systemd22:20:013
2686278994229,9cyclictest2757147-21modprobe23:40:003
2686278994228,11cyclictest275-21dbus-daemon23:45:003
2686278994224,13cyclictest275-21dbus-daemon22:10:003
2686278994216,7cyclictest131rcu_preempt22:42:123
2686278994216,22cyclictest361rcuc/322:56:443
268627899421,3cyclictest131rcu_preempt21:53:013
268626799428,6cyclictest131rcu_preempt22:26:121
268626799427,6cyclictest131rcu_preempt19:20:001
268626799425,6cyclictest131rcu_preempt22:05:011
2686267994233,6cyclictest23-21ksoftirqd/123:25:361
2686267994219,15cyclictest275-21dbus-daemon21:10:011
268626799421,6cyclictest2770101-21cron00:10:001
2686267994215,6cyclictest131rcu_preempt22:09:231
2686262994235,4cyclictest16950irq/224-800000020:20:020
2686262994235,4cyclictest16950irq/224-800000020:20:010
2686262994225,14cyclictest2735463-21ssh22:49:420
2686262994225,14cyclictest2735463-21ssh22:49:420
2686262994223,14cyclictest275-21dbus-daemon19:25:010
2686262994223,14cyclictest275-21dbus-daemon19:25:010
2686262994223,13cyclictest275-21dbus-daemon19:55:010
2686262994223,13cyclictest275-21dbus-daemon19:55:010
2686278994130,8cyclictest37-21ksoftirqd/322:31:303
2686278994128,10cyclictest37-21ksoftirqd/300:05:023
2686278994127,10cyclictest309-21in:imuxsock21:45:033
2686278994126,10cyclictest2690429-21unin-run20:30:013
268627899411,3cyclictest131rcu_preempt21:29:343
2686278994112,6cyclictest131rcu_preempt23:28:243
268626799417,8cyclictest23-21ksoftirqd/123:55:011
2686262994132,6cyclictest19350irq/34-mmc120:45:000
2686262994119,17cyclictest3467602-21systemd-journal19:20:000
268627899407,3cyclictest131rcu_preempt21:15:573
268626799409,6cyclictest131rcu_preempt20:25:011
268626799405,5cyclictest131rcu_preempt20:50:021
2686267994031,7cyclictest23-21ksoftirqd/121:27:471
2686262994026,10cyclictest3467602-21systemd-journal19:48:230
2686262994023,13cyclictest2686445-21sshd19:14:230
2686262994023,13cyclictest2686445-21sshd19:14:230
268626299401,35cyclictest2690918-21sh20:40:000
268627899394,26cyclictest2687556-21cut19:35:003
2686278993924,11cyclictest37-21ksoftirqd/322:26:033
2686278993923,12cyclictest275-21dbus-daemon00:25:013
2686278993921,12cyclictest275-21dbus-daemon20:35:013
268627899391,9cyclictest2712470-1kworker/u9:123:19:193
268626799393,13cyclictest23-21ksoftirqd/119:50:011
2686267993910,5cyclictest131rcu_preempt19:55:011
2686278993821,11cyclictest309-21in:imuxsock19:25:013
2686278993821,11cyclictest309-21in:imuxsock19:25:003
2686278993819,6cyclictest131rcu_preempt23:00:133
2686278993819,14cyclictest3467602-21systemd-journal21:35:003
2686278993819,14cyclictest3467602-21systemd-journal21:35:003
2686278993817,15cyclictest275-21dbus-daemon23:55:003
268627899381,6cyclictest131rcu_preempt21:10:003
2686267993827,8cyclictest2687469-21sshd19:30:231
2686267993812,7cyclictest131rcu_preempt20:30:031
2686267993812,5cyclictest131rcu_preempt20:02:231
268626299382,32cyclictest3467602-21systemd-journal20:13:040
2686278993729,5cyclictest2760813-21sh23:48:133
2686278993729,5cyclictest2760813-21sh23:48:133
2686278993727,7cyclictest3467602-21systemd-journal19:15:003
2686278993727,7cyclictest3467602-21systemd-journal19:15:003
2686278993727,7cyclictest2687229-21sshd19:27:233
2686278993717,15cyclictest3467602-21systemd-journal20:25:003
2686278993717,14cyclictest2750864-21date23:25:023
2686267993726,8cyclictest2687415-21sed19:30:021
2686267993725,7cyclictest23-21ksoftirqd/120:15:011
2686262993725,9cyclictest2692218-21cron21:05:010
268627899363,3cyclictest131rcu_preempt23:10:263
2686278993628,5cyclictest2694041-21sh21:13:193
2686278993628,5cyclictest2694041-21sh21:13:193
2686278993627,6cyclictest2782382-21sh00:38:243
2686278993626,8cyclictest37-21ksoftirqd/322:11:123
2686278993625,8cyclictest2688224-21cat19:45:023
2686278993624,9cyclictest2771234-21sshd00:12:233
2686278993623,10cyclictest37-21ksoftirqd/320:05:273
2686267993620,12cyclictest275-21dbus-daemon20:35:011
268627899352,8cyclictest131rcu_preempt19:40:003
2686278993524,8cyclictest37-21ksoftirqd/320:05:003
2686278993519,12cyclictest37-21ksoftirqd/322:04:223
2686278993517,13cyclictest275-21dbus-daemon20:55:013
2686278993515,6cyclictest131rcu_preempt21:59:393
2686278993515,14cyclictest275-21dbus-daemon20:15:013
2686267993525,7cyclictest2686448-21cron19:15:001
2686267993525,7cyclictest2686448-21cron19:15:001
268626299352,10cyclictest3467602-21systemd-journal20:46:160
2686278993426,5cyclictest2754686-21sh23:34:133
2686278993423,8cyclictest37-21ksoftirqd/319:50:013
2686278993419,10cyclictest37-21ksoftirqd/320:44:183
2686262993426,5cyclictest3467602-21systemd-journal19:27:370
268626299341,30cyclictest2686324-21cat19:10:020
268627899335,8cyclictest37-21ksoftirqd/322:51:243
2686278993324,5cyclictest2691455-21sh20:50:013
268626799333,5cyclictest131rcu_preempt20:20:001
268626799333,5cyclictest131rcu_preempt20:20:001
2686262993323,8cyclictest12-21ksoftirqd/020:05:020
2686278993222,6cyclictest309-21in:imuxsock19:58:243
268626799323,6cyclictest23-21ksoftirqd/119:10:021
2686267993221,8cyclictest23-21ksoftirqd/120:55:011
268627899313,3cyclictest131rcu_preempt19:10:013
2686278993120,9cyclictest37-21ksoftirqd/321:00:013
2686278993016,10cyclictest37-21ksoftirqd/321:05:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional