You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-26 - 00:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon May 25, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
41820839911182,26cyclictest0-21swapper/011:35:020
4182083998581,2cyclictest0-21swapper/007:25:020
4182092997124,10cyclictest30-21ksoftirqd/212:35:012
4182092996915,8cyclictest30-21ksoftirqd/212:05:012
4182092996915,8cyclictest30-21ksoftirqd/212:05:002
418208999680,48cyclictest0-21swapper/107:40:021
418208399661,34cyclictest131rcu_preempt10:55:000
4182083996526,18cyclictest59559-21ssh11:25:010
4182092996421,8cyclictest30-21ksoftirqd/212:08:362
4182092996414,9cyclictest30-21ksoftirqd/209:35:022
4182083996460,2cyclictest0-21swapper/009:40:020
4182092996316,9cyclictest30-21ksoftirqd/210:58:522
418209299626,7cyclictest38247-21ssh10:40:292
418209299626,7cyclictest38247-21ssh10:40:292
4182092996223,7cyclictest30-21ksoftirqd/210:32:462
4182092996220,8cyclictest30-21ksoftirqd/212:24:532
4182092996220,8cyclictest131rcu_preempt10:17:452
4182092996213,8cyclictest30-21ksoftirqd/210:15:022
4182089996251,6cyclictest0-21swapper/107:10:011
4182092996117,8cyclictest30-21ksoftirqd/210:35:292
4182083996153,6cyclictest19350irq/34-mmc108:15:020
4182083996125,33cyclictest12-21ksoftirqd/011:16:380
4182083996123,3cyclictest131rcu_preempt10:55:290
418208399611,58cyclictest0-21swapper/009:10:010
4182092996024,8cyclictest131rcu_preempt10:24:342
4182092996018,9cyclictest30-21ksoftirqd/210:45:052
4182092996015,6cyclictest30-21ksoftirqd/209:38:442
418209699599,28cyclictest361rcuc/308:55:013
418209699597,12cyclictest37-21ksoftirqd/310:54:333
418209299597,9cyclictest30-21ksoftirqd/209:50:002
4182092995915,9cyclictest30-21ksoftirqd/211:25:022
4182083995955,2cyclictest0-21swapper/012:30:030
4182083995929,15cyclictest80934-21ssh12:09:180
4182083995925,3cyclictest131rcu_preempt11:53:430
4182096995823,27cyclictest0-21swapper/311:50:023
4182092995819,7cyclictest131rcu_preempt12:20:012
4182092995816,6cyclictest30-21ksoftirqd/211:52:192
4182092995814,7cyclictest30-21ksoftirqd/211:30:432
4182092995813,9cyclictest30-21ksoftirqd/212:39:002
4182092995813,8cyclictest30-21ksoftirqd/209:20:002
418209699577,10cyclictest37-21ksoftirqd/310:14:343
4182096995711,7cyclictest131rcu_preempt11:33:163
4182092995735,15cyclictest66814-21systemctl11:40:002
4182092995717,8cyclictest30-21ksoftirqd/209:58:082
4182092995710,9cyclictest30-21ksoftirqd/211:04:112
4182092995710,10cyclictest30-21ksoftirqd/211:17:562
4182083995727,15cyclictest25127-21ssh10:13:460
4182083995721,21cyclictest19350irq/34-mmc109:50:020
418209699567,7cyclictest37-21ksoftirqd/307:30:023
4182096995640,11cyclictest1-21systemd08:50:013
418209699561,9cyclictest311-21rs:main3
418209299563,15cyclictest83315-21ssh12:14:082
4182092995617,9cyclictest131rcu_preempt10:03:112
4182092995613,8cyclictest30-21ksoftirqd/210:26:282
4182083995627,13cyclictest5076-21ssh09:32:060
418208399562,47cyclictest37753-21ssh10:39:590
4182083995622,3cyclictest131rcu_preempt09:57:070
418209699559,11cyclictest37-21ksoftirqd/310:20:013
4182096995517,9cyclictest37-21ksoftirqd/311:40:013
4182096995513,5cyclictest131rcu_preempt11:53:483
4182096995510,6cyclictest131rcu_preempt09:36:163
418209299556,10cyclictest30-21ksoftirqd/209:45:012
4182092995538,12cyclictest3467602-21systemd-journal09:10:002
4182092995535,15cyclictest3467602-21systemd-journal08:40:002
4182092995535,15cyclictest3467602-21systemd-journal08:40:002
4182092995515,7cyclictest30-21ksoftirqd/211:47:482
4182092995514,12cyclictest30-21ksoftirqd/209:13:242
4182092995512,10cyclictest30-21ksoftirqd/207:25:002
4182083995526,14cyclictest35160-21ssh10:34:330
418208399551,3cyclictest131rcu_preempt12:40:000
418209699549,5cyclictest131rcu_preempt10:27:083
418209699549,11cyclictest37-21ksoftirqd/311:00:013
418209699542,9cyclictest37-21ksoftirqd/310:49:593
4182096995417,8cyclictest37-21ksoftirqd/312:24:433
4182096995411,5cyclictest131rcu_preempt08:35:023
4182096995411,5cyclictest131rcu_preempt08:35:013
418209299548,11cyclictest30-21ksoftirqd/208:25:022
418209299545,10cyclictest30-21ksoftirqd/209:54:592
4182092995415,9cyclictest30-21ksoftirqd/209:21:472
4182089995416,8cyclictest131rcu_preempt10:25:001
4182083995447,5cyclictest19350irq/34-mmc111:40:020
418209699538,7cyclictest95859-21munin-run12:40:013
418209699538,6cyclictest131rcu_preempt10:24:573
418209699533,10cyclictest37-21ksoftirqd/309:55:093
418209699531,7cyclictest93380-21unin-run12:35:013
4182096995317,10cyclictest37-21ksoftirqd/311:02:053
4182096995315,15cyclictest37-21ksoftirqd/312:30:003
4182096995312,6cyclictest131rcu_preempt09:55:003
418209299538,10cyclictest30-21ksoftirqd/211:56:432
418209299537,10cyclictest30-21ksoftirqd/207:40:012
418209299532,10cyclictest30-21ksoftirqd/210:06:112
4182092995316,7cyclictest30-21ksoftirqd/211:10:322
4182092995311,10cyclictest88979-21sh12:25:332
4182092995310,9cyclictest30-21ksoftirqd/209:27:592
4182083995326,12cyclictest86406-21ssh12:20:120
418208399532,46cyclictest69359-21cat11:45:020
418209699527,12cyclictest37-21ksoftirqd/311:11:283
418209699526,7cyclictest131rcu_preempt09:47:043
418209699525,10cyclictest37-21ksoftirqd/309:43:163
418209699524,8cyclictest37-21ksoftirqd/310:45:013
418209699524,8cyclictest37-21ksoftirqd/310:45:003
4182096995241,8cyclictest37-21ksoftirqd/311:20:003
418209299526,11cyclictest30-21ksoftirqd/208:15:022
418209299526,11cyclictest30-21ksoftirqd/208:15:022
418209299525,8cyclictest30-21ksoftirqd/211:09:142
418209299524,22cyclictest0-21swapper/207:50:012
4182092995211,10cyclictest30-21ksoftirqd/211:41:572
418208999522,31cyclictest4184873-21kworker/u8:408:25:021
418209699517,10cyclictest37-21ksoftirqd/309:30:003
418209699512,9cyclictest37-21ksoftirqd/311:44:203
418209699511,9cyclictest79308-21kworker/u8:212:20:003
4182096995117,5cyclictest131rcu_preempt11:09:593
418209299516,8cyclictest30-21ksoftirqd/210:50:322
4182092995133,13cyclictest275-21dbus-daemon09:05:022
4182089995127,8cyclictest23-21ksoftirqd/112:24:431
4182089995122,20cyclictest23-21ksoftirqd/111:15:111
4182089995119,19cyclictest23-21ksoftirqd/107:15:021
418208399513,26cyclictest131rcu_preempt09:48:440
4182083995124,23cyclictest48346-21ssh11:01:260
4182083995124,12cyclictest53238-21ssh11:11:380
418208399512,3cyclictest131rcu_preempt10:45:330
418208399512,3cyclictest131rcu_preempt09:17:340
418208399511,44cyclictest4186697-21systemctl08:40:000
418208399511,44cyclictest4186697-21systemctl08:39:590
418209699509,5cyclictest131rcu_preempt09:12:373
418209699508,13cyclictest37-21ksoftirqd/312:15:023
418209699506,9cyclictest37-21ksoftirqd/310:34:283
4182096995014,5cyclictest131rcu_preempt09:19:553
4182096995012,6cyclictest131rcu_preempt10:04:363
418209299506,12cyclictest30-21ksoftirqd/208:30:022
418208399503,3cyclictest131rcu_preempt09:25:110
4182083995030,16cyclictest275-21dbus-daemon10:10:010
4182083995024,11cyclictest82422-21ssh12:12:050
4182083995020,14cyclictest19899-21ssh10:02:520
4182083995019,27cyclictest0-21swapper/010:44:380
4182083995019,27cyclictest0-21swapper/010:44:380
418208399501,3cyclictest131rcu_preempt12:02:390
418208399501,3cyclictest131rcu_preempt12:02:380
4182083995012,33cyclictest12-21ksoftirqd/008:51:160
418209699498,8cyclictest37-21ksoftirqd/310:39:593
418209699498,10cyclictest37-21ksoftirqd/312:02:073
418209699498,10cyclictest37-21ksoftirqd/312:02:073
418209699498,10cyclictest37-21ksoftirqd/311:25:223
418209699497,9cyclictest37-21ksoftirqd/312:05:413
418209699497,16cyclictest37-21ksoftirqd/311:20:063
418209699497,10cyclictest37-21ksoftirqd/310:06:513
418209699493,9cyclictest37-21ksoftirqd/311:55:183
4182096994933,12cyclictest3467602-21systemd-journal07:40:003
4182096994933,12cyclictest309-21in:imuxsock08:55:023
418209299498,9cyclictest61528-21rm11:28:582
4182089994925,19cyclictest88805-21kworker/u8:112:30:011
4182083994924,10cyclictest1022-21ssh09:23:500
4182083994921,13cyclictest27144-21ssh10:17:450
418208399491,3cyclictest131rcu_preempt10:29:220
4182096994810,5cyclictest131rcu_preempt08:40:003
4182096994810,5cyclictest131rcu_preempt08:40:003
418209299487,8cyclictest30-21ksoftirqd/207:20:012
418209299486,10cyclictest30-21ksoftirqd/208:55:012
4182092994825,17cyclictest275-21dbus-daemon08:00:012
4182092994825,17cyclictest275-21dbus-daemon08:00:002
4182092994812,10cyclictest30-21ksoftirqd/207:45:012
4182089994831,12cyclictest4184117-21systemd-run07:50:011
4182089994828,16cyclictest3467602-21systemd-journal08:15:021
4182089994828,16cyclictest3467602-21systemd-journal08:15:011
4182089994819,20cyclictest23-21ksoftirqd/112:30:051
4182089994814,6cyclictest131rcu_preempt10:20:011
418208399483,4cyclictest131rcu_preempt09:10:070
4182083994826,7cyclictest69596-21ssh11:45:170
418208399482,42cyclictest74620-21ssh11:55:500
418208399482,28cyclictest131rcu_preempt07:15:010
4182083994819,15cyclictest12-21ksoftirqd/008:40:020
4182083994814,17cyclictest16850irq/134-800000011:25:160
418209699475,11cyclictest37-21ksoftirqd/309:23:463
418209699475,10cyclictest37-21ksoftirqd/308:00:023
418209699475,10cyclictest37-21ksoftirqd/308:00:013
4182096994739,6cyclictest37-21ksoftirqd/309:10:023
418209699472,12cyclictest37-21ksoftirqd/308:30:003
418209699471,12cyclictest37-21ksoftirqd/308:45:003
418209699471,10cyclictest37-21ksoftirqd/308:15:023
418209699471,10cyclictest37-21ksoftirqd/308:15:023
4182096994710,7cyclictest131rcu_preempt07:35:013
418209299473,11cyclictest30-21ksoftirqd/208:50:032
418209299471,8cyclictest30-21ksoftirqd/207:55:002
4182089994732,10cyclictest23-21ksoftirqd/111:45:091
418208999471,5cyclictest131rcu_preempt11:10:011
4182083994731,11cyclictest12-21ksoftirqd/007:40:000
4182083994722,12cyclictest89374-21ssh12:26:210
418208399471,3cyclictest131rcu_preempt10:21:300
418209699461,5cyclictest0-21swapper/307:15:013
4182092994627,12cyclictest275-21dbus-daemon07:30:002
4182089994637,6cyclictest23-21ksoftirqd/110:43:061
4182089994637,6cyclictest23-21ksoftirqd/110:43:051
4182089994633,10cyclictest3467602-21systemd-journal08:39:591
4182089994633,10cyclictest3467602-21systemd-journal08:39:591
4182089994631,11cyclictest4185655-21cron08:20:001
4182083994639,4cyclictest16950irq/224-800000009:40:470
4182083994632,11cyclictest52109-21ssh11:09:370
4182083994620,11cyclictest84859-21ssh12:17:090
4182083994610,29cyclictest4182085-21latency_hist07:10:010
4182096994513,5cyclictest131rcu_preempt07:20:003
418209299455,7cyclictest30-21ksoftirqd/208:35:012
418209299455,7cyclictest30-21ksoftirqd/208:35:012
4182092994534,7cyclictest274-21cron09:00:002
418208999453,5cyclictest131rcu_preempt09:19:241
4182089994529,7cyclictest23-21ksoftirqd/107:51:501
4182089994529,12cyclictest1-21systemd09:50:011
4182089994510,8cyclictest131rcu_preempt11:41:071
4182089994510,6cyclictest131rcu_preempt11:02:411
4182092994428,11cyclictest309-21in:imuxsock08:00:022
4182089994431,10cyclictest3467602-21systemd-journal12:00:031
4182089994431,10cyclictest3467602-21systemd-journal12:00:021
4182089994430,10cyclictest23-21ksoftirqd/110:39:591
4182089994429,11cyclictest309-21in:imuxsock10:10:011
4182089994425,11cyclictest275-21dbus-daemon09:55:001
4182089994412,6cyclictest131rcu_preempt11:40:001
418208399444,35cyclictest4184600-21modprobe08:00:000
418208399444,35cyclictest4184600-21modprobe08:00:000
418208399441,10cyclictest4185318-21latency_hist08:10:020
418209699436,5cyclictest131rcu_preempt07:45:003
4182096994310,7cyclictest131rcu_preempt08:10:003
4182089994331,8cyclictest275-21dbus-daemon07:45:001
4182089994330,10cyclictest23-21ksoftirqd/110:50:151
418208999432,5cyclictest131rcu_preempt12:35:321
418208999432,5cyclictest131rcu_preempt08:35:011
418208999432,5cyclictest131rcu_preempt08:35:011
4182089994323,15cyclictest0-21swapper/111:00:011
4182089994320,16cyclictest0-21swapper/109:30:011
4182083994336,4cyclictest16950irq/224-800000007:15:350
418209699428,9cyclictest37-21ksoftirqd/307:55:003
418208999428,6cyclictest131rcu_preempt11:25:001
4182089994231,7cyclictest23-21ksoftirqd/112:10:011
4182089994230,7cyclictest131rcu_preempt11:15:021
4182089994229,9cyclictest1-21systemd10:50:001
418208999421,5cyclictest131rcu_preempt09:42:301
4182083994235,4cyclictest16950irq/224-800000007:35:010
4182083994234,5cyclictest16950irq/224-800000009:00:010
418208399421,36cyclictest4184499-21ls07:55:020
4182096994132,5cyclictest4184105-21modprobe07:50:003
4182096994128,10cyclictest37-21ksoftirqd/309:05:003
418209699412,11cyclictest37-21ksoftirqd/308:05:003
418209699411,9cyclictest0-21swapper/308:24:113
418209299413,8cyclictest30-21ksoftirqd/207:15:002
4182092994130,8cyclictest4183316-21sshd07:35:002
418208999413,4cyclictest131rcu_preempt09:55:321
4182089994131,7cyclictest23-21ksoftirqd/109:25:011
4182089994130,8cyclictest23-21ksoftirqd/110:26:371
4182089994129,9cyclictest23752-21sshd10:10:351
4182089994129,9cyclictest23-21ksoftirqd/109:35:181
4182089994129,8cyclictest231ksoftirqd/110:05:021
4182089994129,10cyclictest23-21ksoftirqd/112:15:451
4182089994125,11cyclictest23-21ksoftirqd/112:00:011
4182089994118,19cyclictest23-21ksoftirqd/111:27:551
4182089994115,20cyclictest131rcu_preempt09:30:041
4182083994130,6cyclictest309-21in:imuxsock07:50:010
418209699401,14cyclictest0-21swapper/307:23:013
4182092994026,9cyclictest4185652-21sh08:20:012
4182089994029,8cyclictest4184598-21cron08:00:001
4182089994029,8cyclictest4184598-21cron08:00:001
4182089994022,4cyclictest23-21ksoftirqd/110:34:281
4182089994016,19cyclictest221rcuc/111:50:351
4182083994025,11cyclictest3467602-21systemd-journal08:33:410
4182083994025,11cyclictest3467602-21systemd-journal08:33:410
4182083994024,13cyclictest4185369-21sshd08:13:170
4182083994024,13cyclictest4185369-21sshd08:13:160
4182083994019,15cyclictest275-21dbus-daemon07:45:000
418209699391,6cyclictest150-21pr/ttyS208:15:213
418209299398,4cyclictest131rcu_preempt08:44:162
418208999392,5cyclictest131rcu_preempt09:05:021
418208999391,6cyclictest0-21swapper/112:12:501
4182089993916,9cyclictest23-21ksoftirqd/107:35:011
4182089993913,5cyclictest131rcu_preempt07:30:021
4182089993913,17cyclictest23-21ksoftirqd/109:15:021
418208399393,4cyclictest0-21swapper/009:05:010
418209699382,7cyclictest37-21ksoftirqd/307:10:013
418208999384,3cyclictest131rcu_preempt08:25:001
4182089993828,7cyclictest23-21ksoftirqd/108:07:351
4182089993827,8cyclictest4187308-21sed08:50:021
4182089993826,8cyclictest4187438-21sshd08:50:241
4182083993825,10cyclictest4186159-21sshd08:26:160
418208399381,8cyclictest131rcu_preempt07:30:000
418208399381,33cyclictest0-21swapper/008:50:020
418209299371,8cyclictest30-21ksoftirqd/208:05:162
4182089993723,11cyclictest23-21ksoftirqd/107:19:551
4182089993717,17cyclictest23-21ksoftirqd/111:32:151
418208999371,5cyclictest131rcu_preempt08:05:001
4182083993724,10cyclictest4185950-21wc08:25:010
418208999368,5cyclictest131rcu_preempt08:58:201
418208399362,30cyclictest48-21kcompactd008:03:400
4182089993513,6cyclictest131rcu_preempt09:08:011
418208999339,16cyclictest23-21ksoftirqd/108:42:531
418208999325,5cyclictest131rcu_preempt07:22:341
4182092992819,7cyclictest30-21ksoftirqd/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional