You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-27 - 10:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Wed May 27, 2026 00:42:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2730577995537,15cyclictest261-21systemd-journal23:55:002
2730577995344,6cyclictest2847378-21idleruntime-cro22:50:012
2730577995330,19cyclictest0-21swapper/223:20:012
2730577995127,20cyclictest2753060-21sh20:25:012
2730577995038,8cyclictest2917001-21latency00:15:552
2730577995038,8cyclictest2917001-21latency00:15:552
2730577994828,17cyclictest0-21swapper/200:15:012
2730575994826,19cyclictest0-21swapper/022:25:000
2730575994824,21cyclictest2896326-21/usr/sbin/munin23:51:140
2730575994818,27cyclictest926102-21cups-browsed23:46:030
2730577994729,15cyclictest2875593-21cron23:25:012
2730577994722,3cyclictest0-21swapper/200:36:062
2730577994627,16cyclictest0-21swapper/200:05:332
2730577994624,18cyclictest0-21swapper/221:55:542
2730577994622,2cyclictest0-21swapper/223:05:542
2730577994622,21cyclictest439-21systemd-logind20:35:012
2730576994625,18cyclictest0-21swapper/123:55:011
2730575994626,17cyclictest2776704-21unixbench_multi21:21:190
2730577994538,4cyclictest2774651-21munin-run21:20:012
2730577994537,4cyclictest2819179-21sh22:15:012
2730577994534,8cyclictest2851416-21ssh22:55:022
2730577994524,18cyclictest2893578-21timerwakeupswit23:46:112
2730577994521,21cyclictest2767854-21needreboot21:10:572
2730577994520,22cyclictest695-21cron22:35:012
2730576994536,5cyclictest2799070-21systemd-run21:50:011
2730577994420,3cyclictest0-21swapper/222:35:362
2730577994418,3cyclictest0-21swapper/219:16:042
2730577994417,3cyclictest0-21swapper/219:45:552
273057699443,7cyclictest2804373-21ntpq21:55:591
2730576994422,19cyclictest0-21swapper/123:40:011
2730576994422,19cyclictest0-21swapper/123:40:011
273057699442,17cyclictest2772088-21ntpq21:16:031
2730576994418,22cyclictest0-21swapper/121:31:071
2730576994418,22cyclictest0-21swapper/121:31:071
2730575994428,14cyclictest0-21swapper/000:12:060
2730575994419,22cyclictest2903711-21cups-browsed00:26:290
2730575994419,22cyclictest2903711-21cups-browsed00:26:290
2730575994412,2cyclictest0-21swapper/021:06:020
2730575994411,2cyclictest0-21swapper/019:35:560
2730577994324,16cyclictest2749481-21ntp_offset20:11:012
2730577994320,3cyclictest0-21swapper/223:36:062
2730577994320,3cyclictest0-21swapper/223:36:062
2730577994319,3cyclictest0-21swapper/223:10:582
2730577994318,3cyclictest0-21swapper/223:40:492
2730577994318,3cyclictest0-21swapper/223:26:112
273057699436,14cyclictest2757219-21sed20:36:091
2730576994335,5cyclictest2903725-21cron00:00:011
2730576994335,5cyclictest2855420-21munin-plugin-st23:00:011
2730576994335,5cyclictest2855420-21munin-plugin-st23:00:011
273057699432,7cyclictest2753954-21ntpq20:26:001
273057699431,14cyclictest2879638-21/usr/sbin/munin23:30:021
273057699431,11cyclictest2750994-21ntpq20:16:011
273057599431,21cyclictest2844722-21ntpq22:45:590
273057599431,11cyclictest2752530-21ntpq20:21:040
273057799422,15cyclictest2812446-21sh22:05:562
2730577994220,2cyclictest0-21swapper/222:25:582
2730577994219,6cyclictest421-21dbus-daemon00:23:332
2730577994219,6cyclictest421-21dbus-daemon00:23:332
2730577994215,8cyclictest2901026-21ntpq23:56:002
273057699429,3cyclictest2776382-21ssh21:21:111
273057699427,9cyclictest926102-21cups-browsed21:10:361
273057699427,6cyclictest1038537-21munin-node19:50:521
273057699423,5cyclictest2732918-21ntpq19:15:571
273057699422,5cyclictest2744930-21ntpq19:55:571
2730576994224,15cyclictest2762565-21fschecks_count20:55:521
2730576994219,20cyclictest0-21swapper/123:05:511
2730576994212,27cyclictest2910489-21sh00:07:281
273057699421,11cyclictest2847029-21ssh22:49:231
273057699421,11cyclictest2828590-21ntpq22:25:591
273057699421,10cyclictest2881061-21ntpq23:31:001
273057699421,10cyclictest2869872-21ssh23:16:461
2730575994231,7cyclictest420-21avahi-daemon22:26:020
273057599422,8cyclictest2743409-21ntpq19:50:580
273057599421,11cyclictest2881897-21ssh23:31:360
273057799416,9cyclictest2776110-21ssh21:21:022
2730577994121,2cyclictest0-21swapper/222:20:022
2730577994119,2cyclictest0-21swapper/223:32:062
2730577994119,2cyclictest0-21swapper/221:46:122
273057699419,4cyclictest2835054-21ssh22:34:331
273057699413,5cyclictest2740534-21ntpq19:41:021
273057699411,9cyclictest2905148-21ntpq00:01:021
2730576994119,18cyclictest0-21swapper/123:02:551
2730576994117,20cyclictest0-21swapper/123:15:021
273057699411,14cyclictest2741563-21diskstats19:45:511
273057699411,11cyclictest2801792-21ssh21:52:351
273057699411,10cyclictest2825976-21ssh22:22:341
273057699411,10cyclictest2809737-21ssh22:02:201
273057599419,4cyclictest2781604-21ssh21:27:490
273057599414,14cyclictest2804466-21ssh21:56:020
2730575994130,9cyclictest2856711-21needreboot23:00:560
2730575994126,12cyclictest2883689-21kworker/u8:11-xprtiod23:40:010
2730575994126,12cyclictest2883689-21kworker/u8:11-xprtiod23:40:010
273057599412,15cyclictest2753926-21ntpq20:25:580
2730575994117,22cyclictest0-21swapper/000:15:490
2730575994117,22cyclictest0-21swapper/000:15:490
273057599411,4cyclictest2766972-21ssh21:10:230
2730575994114,24cyclictest2811766-21basename22:05:460
273057599411,13cyclictest2742009-21ntpq19:46:020
2730575994110,7cyclictest2860784-21ntpq23:05:580
2730577994031,6cyclictest439-21systemd-logind00:30:012
2730577994031,6cyclictest439-21systemd-logind00:30:012
2730577994030,7cyclictest2766558-21cut21:10:012
273057799402,16cyclictest2824663-21ssh22:21:032
2730576994032,5cyclictest2921928-21unixbench_singl00:21:111
2730576994032,5cyclictest2921928-21unixbench_singl00:21:111
273057699402,6cyclictest2755458-21ntpq20:31:001
2730576994024,13cyclictest0-21swapper/100:35:021
273057699401,9cyclictest2915343-21ssh00:14:021
2730576994011,5cyclictest2766552-21diskmemload22:53:141
273057699401,11cyclictest2929343-21ntpq00:31:011
273057699401,11cyclictest2924400-21ssh00:25:311
273057699401,11cyclictest2924400-21ssh00:25:311
273057699401,11cyclictest2874130-21ssh23:22:111
273057699401,11cyclictest2787303-21ssh21:35:351
273057699401,10cyclictest2891528-21ssh23:44:431
273057699401,10cyclictest2819057-21ssh22:14:481
273057699401,10cyclictest2791475-21ssh21:40:421
273057699400,10cyclictest2813166-21ssh22:06:111
2730575994034,4cyclictest1-21systemd00:35:010
2730575994031,6cyclictest926102-21cups-browsed22:44:240
273057599402,7cyclictest2832741-21ntpq22:31:020
2730575994026,11cyclictest1055-21mta-sts-daemon21:53:280
2730575994023,14cyclictest0-21swapper/023:20:560
273057599402,16cyclictest2877054-21ntpq23:26:020
2730575994018,19cyclictest2851420-21pluginstate22:55:020
273057599401,11cyclictest2905187-21ntpq00:01:040
2730575994011,13cyclictest2901050-21ssh23:56:010
2730577993920,3cyclictest0-21swapper/222:00:532
273057699399,8cyclictest2895900-21cut23:50:021
273057699391,9cyclictest2746522-21ntp_states20:01:031
2730576993915,22cyclictest2733881-21tail19:20:481
273057699391,12cyclictest2779673-21fschecks_time21:25:521
273057699391,11cyclictest2835714-21ssh22:35:231
273057699391,10cyclictest2839770-21ssh22:40:271
273057699391,10cyclictest2821416-21ssh22:16:331
273057699390,11cyclictest2731445-21ntpq19:10:591
273057599393,7cyclictest2740432-21ntpq19:40:570
2730575993920,16cyclictest926102-21cups-browsed23:44:230
2730575993912,2cyclictest0-21swapper/019:11:040
273057599391,12cyclictest2783761-21awk21:30:540
273057599391,12cyclictest2783761-21awk21:30:540
273057599391,11cyclictest2869833-21ssh23:16:420
273057599391,11cyclictest2758454-21ntpq20:40:590
273057599391,10cyclictest2770858-21ssh21:15:040
2730577993831,4cyclictest2744049-21idleruntime-cro19:55:012
2730577993831,3cyclictest0-21swapper/221:26:002
273057699383,7cyclictest2758528-21ntpq20:41:031
2730576993828,7cyclictest2916728-21cpuspeed00:15:501
2730576993828,7cyclictest2916728-21cpuspeed00:15:501
273057699381,9cyclictest2766514-21users21:06:111
273057699381,9cyclictest2749155-21if_wlan020:10:531
273057699381,8cyclictest2735933-21ntpq19:25:561
273057699381,12cyclictest2760021-21ntpq20:46:041
273057699381,11cyclictest2748535-21sshd20:07:231
273057699381,11cyclictest2748535-21sshd20:07:231
273057599385,11cyclictest421-21dbus-daemon21:44:230
273057599383,6cyclictest2816069-21diskstats22:10:510
273057599382,8cyclictest2755502-21ntpq20:31:020
273057599382,7cyclictest2747944-21ntpq20:05:590
273057599382,7cyclictest2747944-21ntpq20:05:590
273057599382,14cyclictest2808449-21ntpq22:01:000
2730575993819,16cyclictest2751525-21kworker/u8:5-xprtiod21:00:530
2730575993818,17cyclictest2728033-21kworker/u8:9-xprtiod20:00:520
2730575993818,17cyclictest12350irq/32-eth000:06:030
2730575993816,20cyclictest2736962-21aten2.4-expect19:30:480
2730575993815,21cyclictest2744379-21aten_r2power_fr19:55:470
2730575993815,20cyclictest0-21swapper/020:35:560
2730575993814,2cyclictest0-21swapper/020:55:510
273057599381,14cyclictest2732688-21irqstats19:15:530
273057599381,11cyclictest2922257-21ssh00:21:480
273057599381,11cyclictest2922257-21ssh00:21:480
273057599381,10cyclictest2935356-21ssh00:38:490
273057599381,10cyclictest2822747-21ssh22:19:090
2730577993729,5cyclictest2751590-21systemd-run20:20:012
273057799372,16cyclictest2852717-21ntpq22:55:572
273057799372,16cyclictest2852717-21ntpq22:55:572
2730576993729,5cyclictest421-21dbus-daemon20:25:021
2730576993711,23cyclictest0-21swapper/119:31:081
273057599377,27cyclictest0-21swapper/020:45:480
273057599372,11cyclictest2766552-21diskmemload22:37:530
273057599371,13cyclictest2788188-21ntpq21:36:000
273057599371,12cyclictest2749442-21ntpq20:10:590
273057599371,11cyclictest2864421-21ssh23:10:520
273057599371,10cyclictest2750931-21ntpq20:15:580
2730577993629,2cyclictest0-21swapper/220:10:002
2730577993629,2cyclictest0-21swapper/220:10:002
2730577993628,5cyclictest421-21dbus-daemon21:35:022
2730577993628,5cyclictest421-21dbus-daemon21:35:022
2730577993627,6cyclictest926102-21cups-browsed21:50:592
2730577993625,8cyclictest2734389-21needreboot19:20:562
273057699368,8cyclictest2739328-21timerwakeupswit19:36:081
273057599361,12cyclictest2799040-21ssh21:50:000
273057599361,12cyclictest2761422-21ntpq20:50:590
2730577993528,4cyclictest0-21swapper/220:40:012
2730577993527,5cyclictest2841313-21unixbench_singl22:41:102
273057699358,8cyclictest261-21systemd-journal20:50:591
273057699350,9cyclictest2765036-21sshd21:02:321
273057599352,6cyclictest2734496-21ntpq19:21:010
2730575993513,19cyclictest0-21swapper/019:10:010
2730575993513,19cyclictest0-21swapper/019:10:010
273057599351,11cyclictest2854811-21ssh22:58:510
273057599351,11cyclictest2854811-21ssh22:58:510
2730577993427,4cyclictest2859590-21latency_hist23:05:022
273057599347,6cyclictest2735914-21ntp_kernel_err19:25:560
273057799334,12cyclictest428-21mta-sts-daemon21:43:572
2730577993328,2cyclictest0-21swapper/200:30:532
2730577993223,6cyclictest548-21gdbus00:00:072
2730576993123,5cyclictest2730580-21systemd-run19:10:011
2730576993123,5cyclictest2730580-21systemd-run19:10:011
2730577992924,2cyclictest0-21swapper/220:50:012
2730577992923,2cyclictest0-21swapper/221:36:012
2730577992922,4cyclictest1056-21mta-sts-daemon21:01:022
2730577992921,5cyclictest1051-21mta-sts-daemon19:25:522
2730577992820,5cyclictest421-21dbus-daemon20:26:002
2730577992722,2cyclictest0-21swapper/219:40:012
2730577992719,5cyclictest1055-21mta-sts-daemon19:31:012
2730577992620,4cyclictest2730574-21cyclictest20:41:022
2730577992520,2cyclictest0-21swapper/220:00:002
2730577992519,3cyclictest0-21swapper/219:45:012
2730577992518,4cyclictest1054-21mta-sts-daemon19:11:052
273057799242,17cyclictest2746443-21ntpq20:01:002
2730577992413,8cyclictest2762722-21latency20:55:552
2730577992317,4cyclictest1056-21mta-sts-daemon20:50:152
273057899174,7cyclictest0-21swapper/300:11:033
2730577991711,3cyclictest0-21swapper/219:10:002
2730577991711,3cyclictest0-21swapper/219:10:002
273057899163,9cyclictest0-21swapper/323:35:583
273057899163,9cyclictest0-21swapper/323:35:583
273057899162,10cyclictest0-21swapper/322:51:003
273057899154,7cyclictest0-21swapper/322:25:593
273057899153,9cyclictest0-21swapper/323:50:593
273057899153,8cyclictest0-21swapper/323:55:583
273057899153,8cyclictest0-21swapper/323:05:493
273057899153,8cyclictest0-21swapper/300:25:563
273057899153,8cyclictest0-21swapper/300:25:563
273057899153,8cyclictest0-21swapper/300:16:013
273057899153,8cyclictest0-21swapper/300:16:013
273057899153,8cyclictest0-21swapper/300:01:063
273057899153,7cyclictest0-21swapper/323:00:593
273057899152,9cyclictest0-21swapper/323:30:573
273057899152,9cyclictest0-21swapper/323:20:583
273057899152,9cyclictest0-21swapper/323:15:573
273057899152,9cyclictest0-21swapper/322:56:043
273057899152,9cyclictest0-21swapper/322:56:043
273057899152,9cyclictest0-21swapper/321:15:593
273057899152,9cyclictest0-21swapper/300:06:013
273057899152,8cyclictest0-21swapper/319:55:563
273057899152,8cyclictest0-21swapper/300:35:583
273057899152,6cyclictest0-21swapper/319:20:583
273057899151,9cyclictest0-21swapper/321:51:103
273057899151,9cyclictest0-21swapper/321:20:583
273057899143,8cyclictest0-21swapper/323:46:063
273057899143,7cyclictest0-21swapper/323:41:053
273057899143,7cyclictest0-21swapper/323:25:573
273057899143,7cyclictest0-21swapper/322:41:043
273057899143,7cyclictest0-21swapper/322:00:583
273057899143,7cyclictest0-21swapper/321:11:013
273057899142,9cyclictest0-21swapper/321:31:063
273057899142,9cyclictest0-21swapper/321:31:063
273057899142,8cyclictest0-21swapper/322:30:563
273057899142,8cyclictest0-21swapper/322:10:563
273057899142,8cyclictest0-21swapper/321:55:553
273057899142,8cyclictest0-21swapper/321:40:573
273057899142,8cyclictest0-21swapper/321:35:583
273057899142,8cyclictest0-21swapper/321:25:573
273057899142,6cyclictest0-21swapper/319:40:563
273057899141,9cyclictest0-21swapper/322:46:053
273057899141,9cyclictest0-21swapper/322:35:503
273057899141,9cyclictest0-21swapper/322:06:003
273057899141,9cyclictest0-21swapper/320:51:003
273057899141,9cyclictest0-21swapper/300:20:593
273057899141,9cyclictest0-21swapper/300:20:593
273057899141,10cyclictest0-21swapper/321:45:573
273057899141,10cyclictest0-21swapper/319:30:593
273057899133,7cyclictest0-21swapper/320:46:023
273057899133,7cyclictest0-21swapper/320:35:583
273057899133,7cyclictest0-21swapper/319:46:023
273057899133,6cyclictest0-21swapper/322:20:553
273057899132,8cyclictest0-21swapper/322:15:543
273057899132,8cyclictest0-21swapper/321:05:593
273057899132,8cyclictest0-21swapper/321:00:583
273057899132,8cyclictest0-21swapper/320:15:573
273057899132,8cyclictest0-21swapper/320:11:043
273057899132,8cyclictest0-21swapper/320:05:583
273057899132,8cyclictest0-21swapper/320:05:583
273057899132,7cyclictest0-21swapper/323:10:583
273057899132,7cyclictest0-21swapper/320:20:593
273057899131,9cyclictest0-21swapper/320:30:583
273057899131,9cyclictest0-21swapper/320:01:013
273057899131,8cyclictest0-21swapper/320:26:043
273057899131,8cyclictest0-21swapper/319:51:013
273057899131,8cyclictest0-21swapper/319:25:583
273057899131,8cyclictest0-21swapper/300:30:583
273057899131,7cyclictest0-21swapper/319:35:563
273057899122,7cyclictest0-21swapper/319:15:583
273057899121,8cyclictest0-21swapper/320:55:583
273057899121,8cyclictest0-21swapper/320:40:573
273057899121,8cyclictest0-21swapper/319:10:583
27305789992,5cyclictest0-21swapper/319:10:013
27305789992,5cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional