You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-21 - 06:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Aug 21, 2026 00:45:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2863660998810,33cyclictest2946130-21rm23:51:100
2863660998126,50cyclictest2877222-21ssh21:25:090
2863672997810,16cyclictest2947709-21modprobe23:55:003
286367299776,29cyclictest2893409-21munin-plugin-st21:59:593
286367299758,21cyclictest2867246-21systemctl20:19:593
2863672997324,43cyclictest275-21dbus-daemon21:10:013
2863672997225,42cyclictest1-21systemd22:45:013
2863660997115,39cyclictest2961447-21ssh00:23:510
286367299706,22cyclictest3467602-21systemd-journal23:45:003
286367299705,16cyclictest2966701-21grep00:35:003
2863660997027,37cyclictest2919368-21grep22:55:020
286367299694,19cyclictest2928875-21grep23:15:003
2863665996918,9cyclictest23-21ksoftirqd/121:59:271
286367299686,14cyclictest2954896-21modprobe00:10:003
2863672996826,34cyclictest1-21systemd23:59:593
2863672996822,8cyclictest2905312-21latency_hist22:25:023
2863672996822,8cyclictest2905312-21latency_hist22:25:013
2863660996828,36cyclictest364-21snmpd21:52:020
2863660996828,36cyclictest364-21snmpd21:52:020
286367299677,18cyclictest309-21in:imuxsock21:55:013
286367299677,18cyclictest309-21in:imuxsock21:55:013
286367299676,17cyclictest2865941-21munin-plugin-st19:55:013
286367299674,19cyclictest2916994-21modprobe22:50:003
286367299674,15cyclictest2959564-21systemctl00:20:013
286367299674,15cyclictest2959564-21systemctl00:20:013
286367299673,19cyclictest275-21dbus-daemon22:40:003
2863660996723,40cyclictest2921851-21latency_hist23:00:020
2863660996723,40cyclictest2921851-21latency_hist23:00:010
286367299665,20cyclictest1-21systemd20:10:013
286367299663,19cyclictest2879374-21wc21:30:013
2863672996627,35cyclictest2921997-21rm23:00:093
2863665996611,35cyclictest2907744-21ssh22:30:021
2863660996627,4cyclictest2903918-21ssh22:21:570
2863660996627,4cyclictest2903918-21ssh22:21:570
286367299659,9cyclictest0-21swapper/323:00:003
286367299659,9cyclictest0-21swapper/322:59:593
286367299655,15cyclictest3467602-21systemd-journal19:19:593
2863672996525,5cyclictest2955469-21sh00:10:563
2863660996515,45cyclictest2958165-21ssh00:16:490
2863660996515,45cyclictest2958165-21ssh00:16:480
286367299643,20cyclictest2868081-21grep20:35:023
286367299643,20cyclictest2868081-21grep20:35:013
2863665996413,15cyclictest2961720-21ssh00:24:261
2863665996411,15cyclictest2869836-21diskmemload21:15:561
2863660996424,35cyclictest2952672-21latency_hist00:05:030
2863660996420,40cyclictest0-21swapper/022:45:010
286367299636,11cyclictest2865156-21latency_hist19:40:003
286367299635,13cyclictest2945356-21munin-plugin-st23:50:013
286367299635,13cyclictest2945356-21munin-plugin-st23:50:003
286367299634,26cyclictest2969087-21munin-run00:40:013
286367299634,11cyclictest2961928-21modprobe00:24:593
286367299633,16cyclictest2940637-21cron23:40:003
286367299633,16cyclictest2940637-21cron23:40:003
2863672996320,8cyclictest2926713-21latency_hist23:10:033
2863672996312,18cyclictest275-21dbus-daemon19:25:003
2863672996310,13cyclictest0-21swapper/322:09:593
286367299624,21cyclictest1-21systemd23:35:013
286367299622,19cyclictest2936020-21latency_hist23:30:013
2863672996221,4cyclictest2888028-21sh21:48:393
2863660996222,16cyclictest2886501-21sed21:45:030
2863660996215,42cyclictest2966897-21latency_hist00:35:020
286367299615,13cyclictest284-21systemd-logind22:05:013
286367299615,13cyclictest284-21systemd-logind22:05:013
286367299613,16cyclictest2866486-21unin-run20:05:003
2863672996121,7cyclictest37-21ksoftirqd/322:55:013
2863672996112,6cyclictest0-21swapper/319:35:013
2863665996118,10cyclictest23-21ksoftirqd/123:16:021
2863665996118,10cyclictest23-21ksoftirqd/123:16:011
2863660996129,29cyclictest150-21pr/ttyS221:19:470
2863660996118,9cyclictest2868249-21sshd20:38:560
2863660996116,41cyclictest2881546-21ssh21:34:550
2863660996116,41cyclictest2881546-21ssh21:34:540
2863660996115,12cyclictest2871617-21ssh21:13:370
286367299605,18cyclictest2869295-21modprobe21:00:003
286367299604,13cyclictest2874708-21latency_hist21:20:023
286367299603,14cyclictest2952497-21ssh00:05:013
2863665996016,9cyclictest23-21ksoftirqd/121:26:211
2863665996011,14cyclictest3467602-21systemd-journal22:31:251
2863665996011,14cyclictest3467602-21systemd-journal22:31:241
2863660996024,4cyclictest2969124-21munin-run00:40:020
2863660996017,10cyclictest2944448-21ssh23:47:520
2863660996017,10cyclictest2944448-21ssh23:47:520
286367299594,16cyclictest2883970-21cat21:39:593
286367299594,15cyclictest275-21dbus-daemon19:45:023
286367299593,15cyclictest2868274-21modprobe20:40:003
2863672995920,8cyclictest37-21ksoftirqd/321:40:563
2863660995911,44cyclictest0-21swapper/023:15:390
2863660995911,44cyclictest0-21swapper/023:15:380
2863660995823,4cyclictest2898999-21ssh22:11:290
2863660995823,31cyclictest2954047-21ssh00:08:050
2863660995820,34cyclictest1-21systemd22:40:010
2863660995817,10cyclictest2902115-21ssh22:18:230
286367299576,12cyclictest2867767-21modprobe20:29:593
286367299573,16cyclictest2865692-21true19:50:003
2863672995718,10cyclictest37-21ksoftirqd/320:25:013
2863672995718,10cyclictest37-21ksoftirqd/320:25:003
2863672995715,6cyclictest0-21swapper/321:05:013
2863665995713,35cyclictest23-21ksoftirqd/121:00:021
2863660995728,25cyclictest2943049-21wc23:45:010
2863660995728,13cyclictest2933035-21ssh23:23:480
2863660995724,29cyclictest364-21snmpd21:55:400
2863660995721,28cyclictest16850irq/134-800000021:39:400
2863660995719,35cyclictest0-21swapper/023:10:010
2863660995715,39cyclictest2876291-21ssh21:23:330
286367299563,12cyclictest2909872-21modprobe22:35:013
286367299563,12cyclictest2909872-21modprobe22:35:013
2863672995617,17cyclictest275-21dbus-daemon20:55:013
2863672995615,37cyclictest0-21swapper/323:25:003
2863660995632,21cyclictest0-21swapper/022:29:110
2863660995631,8cyclictest2937620-21ssh23:33:300
2863660995627,13cyclictest2928542-21ssh23:14:210
2863660995625,13cyclictest286983650diskmemload22:46:250
2863660995619,33cyclictest16850irq/134-800000000:29:490
2863660995619,33cyclictest16850irq/134-800000000:29:480
2863660995616,36cyclictest0-21swapper/020:10:010
2863660995615,11cyclictest0-21swapper/023:40:010
2863660995615,11cyclictest0-21swapper/023:40:000
286367299553,13cyclictest1-21systemd20:15:023
2863672995523,11cyclictest37-21ksoftirqd/300:27:023
2863672995523,11cyclictest37-21ksoftirqd/300:27:023
2863669995547,5cyclictest2924530-21kworker/2:023:44:332
2863660995529,11cyclictest2888149-21ssh21:48:540
2863660995518,33cyclictest16850irq/134-800000022:02:030
2863660995518,33cyclictest16850irq/134-800000022:02:020
2863660995517,34cyclictest2866643-21latency_hist20:05:020
2863660995515,36cyclictest0-21swapper/023:58:110
286367299544,9cyclictest2902940-21latency_hist22:20:023
286367299544,18cyclictest2900444-21ssh22:15:013
286367299544,17cyclictest2872438-21latency_hist21:15:023
286367299543,12cyclictest2907625-21latency_hist22:30:003
2863669995445,5cyclictest2909889-21kworker/2:222:50:522
2863665995421,6cyclictest23-21ksoftirqd/123:58:551
2863660995418,33cyclictest16850irq/134-800000022:10:010
2863672995320,9cyclictest37-21ksoftirqd/321:23:583
2863669995344,5cyclictest2851688-21kworker/2:022:29:112
2863660995314,35cyclictest0-21swapper/019:17:550
2863672995219,11cyclictest3467718-21lldpd23:18:313
2863672995219,11cyclictest3467718-21lldpd23:18:313
2863669995234,14cyclictest275-21dbus-daemon20:35:012
2863669995234,14cyclictest275-21dbus-daemon20:35:002
2863665995212,7cyclictest2899023-21ssh22:11:311
2863660995234,13cyclictest2863664-21wc19:10:020
2863660995226,23cyclictest0-21swapper/023:03:450
2863660995226,20cyclictest3467602-21systemd-journal19:45:010
2863660995223,24cyclictest16850irq/134-800000021:10:000
2863660995218,31cyclictest16850irq/134-800000023:27:140
2863660995217,6cyclictest16850irq/134-800000000:12:000
2863669995143,5cyclictest2909889-21kworker/2:222:49:292
2863665995123,23cyclictest275-21dbus-daemon22:20:011
2863660995125,22cyclictest2866076-21latency_hist19:55:030
2863660995114,5cyclictest2865699-21munin-run19:50:000
2863669995041,5cyclictest2851688-21kworker/2:020:45:002
2863669995031,14cyclictest275-21dbus-daemon19:55:002
286366599505,14cyclictest2875498-21sh21:21:361
2863660995025,4cyclictest364-21snmpd20:40:240
2863660995017,29cyclictest2908478-21sh22:31:370
2863660995017,29cyclictest2908478-21sh22:31:370
2863669994941,5cyclictest2874916-21kworker/2:122:18:392
2863669994941,5cyclictest2851688-21kworker/2:021:55:052
2863669994940,6cyclictest2924530-21kworker/2:023:52:292
286366999492,12cyclictest2950051-40mandb00:00:012
286366099499,36cyclictest0-21swapper/019:35:010
286366099492,16cyclictest2869292-21cron21:00:000
286367299484,13cyclictest2868802-21grep20:49:593
2863669994840,5cyclictest2874916-21kworker/2:122:31:382
2863669994840,5cyclictest2874916-21kworker/2:122:31:372
2863669994839,6cyclictest2851688-21kworker/2:022:05:012
2863669994839,6cyclictest2851688-21kworker/2:022:05:002
2863669994839,5cyclictest2851688-21kworker/2:019:45:012
2863669994812,26cyclictest2940767-21ssh23:40:022
2863669994812,26cyclictest2940767-21ssh23:40:022
2863665994828,15cyclictest1-21systemd23:15:011
2863665994824,21cyclictest275-21dbus-daemon23:05:001
2863660994828,16cyclictest2868131-21latency_hist20:35:020
2863660994828,16cyclictest2868131-21latency_hist20:35:020
2863660994824,19cyclictest1-21systemd19:40:010
2863660994823,21cyclictest0-21swapper/020:20:200
2863660994823,21cyclictest0-21swapper/020:20:190
2863660994821,23cyclictest2864656-21true19:30:010
2863669994739,5cyclictest2950314-21kworker/2:100:10:012
2863669994739,5cyclictest2924530-21kworker/2:023:29:232
2863669994738,5cyclictest2950314-21kworker/2:100:30:032
286366999472,10cyclictest364-21snmpd21:44:452
2863665994728,12cyclictest3467602-21systemd-journal19:25:001
2863665994710,8cyclictest23-21ksoftirqd/121:44:501
2863660994729,14cyclictest2867277-21systemd20:20:010
2863660994725,17cyclictest2863822-21sshd19:10:540
286366099472,39cyclictest1-21systemd20:55:010
2863669994639,5cyclictest2867273-21kworker/2:321:13:082
2863669994638,5cyclictest2874916-21kworker/2:122:12:312
286366599464,7cyclictest23-21ksoftirqd/123:34:571
2863665994637,6cyclictest23-21ksoftirqd/100:08:351
2863665994634,8cyclictest3467602-21systemd-journal22:55:011
2863665994625,16cyclictest309-21in:imuxsock20:39:591
2863665994611,9cyclictest0-21swapper/123:36:541
2863665994611,9cyclictest0-21swapper/123:36:531
2863665994611,7cyclictest131rcu_preempt00:40:011
2863665994610,8cyclictest0-21swapper/121:33:541
2863665994610,8cyclictest0-21swapper/121:33:541
2863660994639,4cyclictest16950irq/224-800000019:25:020
2863660994629,13cyclictest275-21dbus-daemon20:50:010
2863669994537,5cyclictest2924530-21kworker/2:023:24:152
2863669994537,5cyclictest2851688-21kworker/2:021:19:532
2863669994536,5cyclictest2917016-21kworker/2:323:50:012
2863669994536,5cyclictest2917016-21kworker/2:323:50:012
2863669994531,11cyclictest275-21dbus-daemon22:45:012
2863669994511,5cyclictest131rcu_preempt00:25:242
2863665994531,11cyclictest1-21systemd21:15:001
2863665994520,22cyclictest2958613-21ssh00:17:451
2863665994520,22cyclictest2958613-21ssh00:17:441
2863665994520,21cyclictest2883965-21latency_hist21:39:591
2863665994511,9cyclictest0-21swapper/100:01:571
2863660994530,11cyclictest2866154-21sshd19:55:540
286366999448,5cyclictest131rcu_preempt19:40:002
286366999445,9cyclictest0-21swapper/223:05:022
2863669994436,5cyclictest2950314-21kworker/2:100:37:342
2863669994435,6cyclictest2924530-21kworker/2:023:18:502
2863669994435,6cyclictest2924530-21kworker/2:023:18:492
2863669994432,9cyclictest2928814-21sshd23:14:552
2863669994431,9cyclictest1-21systemd19:25:012
2863669994428,12cyclictest275-21dbus-daemon19:30:012
286366999442,12cyclictest2878894-21ssh21:29:032
2863665994430,11cyclictest3467602-21systemd-journal22:38:561
2863665994411,7cyclictest0-21swapper/123:54:071
2863660994426,13cyclictest2867768-21munin-run20:30:000
2863660994423,18cyclictest150-21pr/ttyS220:11:520
286366999437,7cyclictest0-21swapper/223:34:402
286366999436,6cyclictest131rcu_preempt23:00:022
286366999436,6cyclictest131rcu_preempt23:00:012
2863669994335,5cyclictest2874916-21kworker/2:121:31:182
2863669994335,5cyclictest2874916-21kworker/2:121:31:172
2863669994335,5cyclictest2851688-21kworker/2:021:22:532
2863669994334,6cyclictest2874916-21kworker/2:121:52:102
2863669994334,6cyclictest2874916-21kworker/2:121:52:092
2863669994333,6cyclictest2868787-21cron20:50:002
2863669994329,10cyclictest2955355-21sh00:10:432
286366999432,8cyclictest2952706-21sed00:05:032
286366999432,10cyclictest2867131-21date20:15:022
2863665994328,12cyclictest2886906-21sshd21:45:551
2863665994323,15cyclictest275-21dbus-daemon23:45:011
2863665994317,22cyclictest2916751-21ssh22:49:301
2863665994314,25cyclictest2912913-21sh22:41:061
2863665994311,7cyclictest0-21swapper/100:25:281
2863665994311,7cyclictest0-21swapper/100:25:281
286366599431,12cyclictest0-21swapper/100:30:211
2863672994228,11cyclictest2868562-21systemd-run20:45:013
286366999428,6cyclictest131rcu_preempt22:09:342
286366999428,5cyclictest131rcu_preempt21:50:022
286366999428,4cyclictest131rcu_preempt00:22:352
286366999423,5cyclictest131rcu_preempt23:05:052
2863669994229,9cyclictest3467602-21systemd-journal00:30:032
2863669994227,11cyclictest1-21systemd19:20:012
286366599429,8cyclictest0-21swapper/122:56:561
286366599429,8cyclictest0-21swapper/122:56:561
286366599429,14cyclictest0-21swapper/123:25:011
2863665994233,6cyclictest2905222-21latency_hist22:25:021
2863665994233,6cyclictest2905222-21latency_hist22:25:011
2863665994226,12cyclictest23-21ksoftirqd/121:52:341
2863665994226,12cyclictest23-21ksoftirqd/121:52:331
2863665994216,21cyclictest2925570-21ssh23:07:581
2863660994228,10cyclictest2869584-21munin-run21:05:010
286366999414,3cyclictest131rcu_preempt20:54:592
2863669994134,5cyclictest2851688-21kworker/2:022:39:192
2863665994128,9cyclictest23-21ksoftirqd/120:00:011
2863665994127,10cyclictest23-21ksoftirqd/119:20:001
2863665994121,16cyclictest2894065-21sendmail-mta22:00:571
2863665994121,16cyclictest2894065-21sendmail-mta22:00:561
2863665994116,21cyclictest2955383-21ssh00:10:461
2863665994115,21cyclictest2896614-21ssh22:06:251
2863672994025,12cyclictest37-21ksoftirqd/321:33:583
2863672994025,12cyclictest37-21ksoftirqd/321:33:573
2863669994032,5cyclictest2856708-21kworker/2:119:50:022
2863669994030,6cyclictest2866178-21cron20:00:002
2863669994023,13cyclictest275-21dbus-daemon22:25:012
2863669994023,13cyclictest275-21dbus-daemon22:25:002
2863669994022,14cyclictest1-21systemd00:20:012
2863669994022,14cyclictest1-21systemd00:20:002
2863665994028,9cyclictest2865662-21sshd19:47:551
2863665994025,10cyclictest2869570-21cron21:04:591
2863665994016,19cyclictest2943341-21ssh23:45:241
2863665994016,19cyclictest2943341-21ssh23:45:231
2863665994015,21cyclictest2935026-21ssh23:27:531
2863672993918,15cyclictest275-21dbus-daemon19:15:013
2863669993925,10cyclictest3467602-21systemd-journal21:10:002
2863665993927,9cyclictest23-21ksoftirqd/120:10:021
2863669993826,9cyclictest2882861-21sh21:37:242
2863669993824,11cyclictest309-21in:imuxsock21:00:032
2863669993824,11cyclictest1-21systemd20:20:012
2863669993824,10cyclictest30-21ksoftirqd/220:40:002
2863669993819,14cyclictest3467602-21systemd-journal21:00:012
286366999381,8cyclictest0-21swapper/220:10:012
2863665993828,8cyclictest23-21ksoftirqd/121:10:031
2863665993824,10cyclictest2866954-21sshd20:13:561
2863665993823,11cyclictest2866460-21cron20:04:591
2863665993820,14cyclictest2867277-21munin-run20:20:001
2863665993813,21cyclictest2863953-21wc19:15:031
286367299373,5cyclictest131rcu_preempt19:30:013
2863672993724,10cyclictest37-21ksoftirqd/320:00:013
2863665993728,5cyclictest2865202-21latency_hist19:40:011
2863665993725,9cyclictest23-21ksoftirqd/120:20:261
2863665993725,9cyclictest23-21ksoftirqd/120:20:261
2863669993628,5cyclictest2864900-21latency_hist19:35:022
2863669993622,10cyclictest30-21ksoftirqd/219:15:002
286366599369,21cyclictest2867943-21latency_hist20:30:011
2863665993625,8cyclictest23-21ksoftirqd/120:54:151
2863665993623,9cyclictest23-21ksoftirqd/120:49:301
2863665993623,10cyclictest23-21ksoftirqd/120:41:591
2863669993524,8cyclictest30-21ksoftirqd/220:05:022
2863665993523,9cyclictest23-21ksoftirqd/119:32:341
2863665993521,10cyclictest3467602-21systemd-journal19:41:031
2863665993422,9cyclictest23-21ksoftirqd/120:30:071
2863665993422,9cyclictest23-21ksoftirqd/120:30:061
2863665993419,11cyclictest2865898-21modprobe19:50:021
2863665993416,14cyclictest284-21systemd-logind19:30:021
2863669993224,5cyclictest2863757-21latency_hist19:10:022
2863665993215,14cyclictest2863697-21cat19:10:011
2863672993016,11cyclictest2863685-21cat19:10:013
2863669992817,8cyclictest30-21ksoftirqd/220:28:552
2863669992814,11cyclictest0-21swapper/220:23:122
2863669992814,11cyclictest0-21swapper/220:23:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional