You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 06:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Apr 16, 2026 00:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4124425991024,66cyclictest4110639-21munin-node20:40:023
4124414998856,23cyclictest4160275-21kworker/u8:422:45:020
4124414998314,63cyclictest16850irq/134-800000022:29:440
4124414998216,61cyclictest16850irq/134-800000021:56:430
4124423997830,43cyclictest275-21dbus-daemon00:05:012
4124414997817,51cyclictest16850irq/134-800000023:33:240
4124414997722,51cyclictest4192648-21ssh23:37:430
4124414997722,51cyclictest4192648-21ssh23:37:430
4124414997714,51cyclictest16850irq/134-800000023:27:560
4124414997714,51cyclictest16850irq/134-800000023:27:560
412441499768,63cyclictest16850irq/134-800000023:17:310
412441499768,63cyclictest16850irq/134-800000023:17:310
412441499763,9cyclictest1-21systemd21:45:010
4124414997627,30cyclictest4180062-21ssh23:07:550
4124414997627,30cyclictest4180062-21ssh23:07:550
4124414997615,51cyclictest16850irq/134-800000022:50:020
4124414997615,51cyclictest16850irq/134-800000022:50:020
4124414997615,38cyclictest16850irq/134-800000021:46:240
4124414997515,5cyclictest19350irq/34-mmc120:00:020
4124414997514,52cyclictest16850irq/134-800000000:29:150
4124414997424,34cyclictest4167416-21ssh22:37:440
4124418997335,10cyclictest275-21dbus-daemon00:30:001
4124414997329,39cyclictest4160927-21ssh22:22:040
4124414997326,43cyclictest4186739-21ssh23:23:370
4124414997319,15cyclictest5373-21ssh23:53:420
412441499728,6cyclictest16850irq/134-800000023:01:450
412441499726,6cyclictest16850irq/134-800000023:59:330
4124414997235,33cyclictest16850irq/134-800000021:15:010
4124414997228,27cyclictest4153781-21ssh22:05:020
4124414997212,43cyclictest16850irq/134-800000000:05:080
4124418997122,17cyclictest275-21dbus-daemon21:55:011
4124414997122,30cyclictest9372-21ssh00:03:120
412441499711,5cyclictest4140995-21munin-run21:35:020
4124425997062,6cyclictest37-21ksoftirqd/300:10:033
4124425997062,6cyclictest37-21ksoftirqd/300:10:023
4124423996923,41cyclictest4155766-21latency_hist22:10:012
412441499696,5cyclictest16850irq/134-800000023:10:220
4124414996910,6cyclictest16850irq/134-800000021:23:340
412442599681,36cyclictest4192714-21ssh23:37:513
412442599681,36cyclictest4192714-21ssh23:37:513
412441899685,23cyclictest1-21systemd21:00:011
412441899685,21cyclictest4147374-21grep21:50:011
412441499688,5cyclictest16850irq/134-800000022:30:260
412441499682,15cyclictest48-21kcompactd022:08:380
4124414996821,41cyclictest275-21dbus-daemon00:20:010
4124414996810,6cyclictest16850irq/134-800000023:46:030
412442599672,59cyclictest4132871-21sshd21:15:123
4124423996726,36cyclictest4181428-21sh23:10:542
4124423996722,7cyclictest309-21in:imuxsock21:35:002
412441899676,21cyclictest7930-21latency_hist00:00:021
412441899676,19cyclictest284-21systemd-logind19:30:011
412441499679,48cyclictest16850irq/134-800000021:36:350
412441499676,6cyclictest16850irq/134-800000000:12:310
412441499676,6cyclictest16850irq/134-800000000:12:300
4124414996729,33cyclictest4132832-21ssh21:15:080
4124414996721,40cyclictest4176631-21munin-plugin-st23:00:000
4124414996712,50cyclictest16850irq/134-800000021:27:570
412442599662,59cyclictest4152506-21ssh22:02:003
412442599661,61cyclictest4165315-21ssh22:32:383
412442599661,61cyclictest4156064-21ssh22:10:223
412442599661,61cyclictest4148247-21ssh21:51:533
412441899665,19cyclictest4185226-21latency_hist23:20:011
412441899665,19cyclictest4185226-21latency_hist23:20:011
4124414996612,44cyclictest16850irq/134-800000023:44:450
412441499661,15cyclictest4126404-21sshd19:47:130
4124425996518,43cyclictest37-21ksoftirqd/323:05:063
4124425996518,43cyclictest37-21ksoftirqd/323:05:053
412442599651,60cyclictest4153870-21ssh22:05:063
412442599651,60cyclictest4141680-21ssh21:36:123
412442599651,60cyclictest22556-21ssh00:34:463
412442399653,20cyclictest4143301-21latency_hist21:40:022
4124414996525,36cyclictest4159973-21ssh22:20:010
4124414996519,28cyclictest16850irq/134-800000021:52:560
4124414996510,13cyclictest20995-21ssh00:30:420
412442599642,57cyclictest4126974-21munin-run20:00:013
4124425996424,35cyclictest4162796-21sh22:26:283
412442599641,59cyclictest687-21ssh23:42:253
412442599641,59cyclictest5217-21ssh23:53:203
412442599641,59cyclictest4188813-21ssh23:28:303
412442599641,59cyclictest4188813-21ssh23:28:303
412442599641,59cyclictest4174962-21ssh22:55:283
412442599641,59cyclictest4169173-21ssh22:41:453
412442599641,59cyclictest4146633-21ssh21:48:143
412442599641,59cyclictest4143719-21ssh21:41:023
412442599641,59cyclictest4135636-21ssh21:21:523
412442599641,59cyclictest4128985-21sshd20:37:123
412442599641,59cyclictest18989-21ssh00:25:553
412442599641,59cyclictest17570-21ssh00:22:413
4124425996410,50cyclictest2443-21ssh23:46:393
412442399643,19cyclictest4127964-21cron20:20:002
4124423996422,37cyclictest4162198-21latency_hist22:25:022
4124423996421,5cyclictest5840-21latency_hist23:55:012
412441899645,16cyclictest4176636-21cut23:00:011
412441899644,17cyclictest1644-21unin-run23:45:001
4124418996423,12cyclictest275-21dbus-daemon22:10:011
4124418996414,10cyclictest4129541-21munin-plugin-st20:50:001
412442599632,57cyclictest4137325-21ssh21:25:473
412442599632,55cyclictest364-21snmpd19:30:273
412442599631,58cyclictest8794-21ssh00:01:353
412442599631,58cyclictest4183995-21ssh23:17:053
412442599631,58cyclictest4183995-21ssh23:17:043
412442599631,58cyclictest4181239-21ssh23:10:273
412442599631,58cyclictest4178401-21ssh23:04:073
412442599631,58cyclictest4166547-21ssh22:35:243
412442599631,58cyclictest4160311-21ssh22:20:213
412442599631,58cyclictest4159126-21ssh22:17:553
412442599631,58cyclictest4149798-21ssh21:55:273
412442599631,58cyclictest4139403-21ssh21:30:533
412442599631,58cyclictest24007-21ssh00:38:043
412442599631,58cyclictest14808-21ssh00:16:033
4124423996313,14cyclictest275-21dbus-daemon20:55:012
412441899633,18cyclictest4125894-21modprobe19:40:001
4124414996319,39cyclictest4157898-21latency_hist22:15:010
412442599621,57cyclictest6102-21ssh23:55:053
412442599621,57cyclictest4186525-21ssh23:23:053
412442599621,57cyclictest4173576-21ssh22:52:253
412442599621,57cyclictest4172487-21ssh22:49:593
412442599621,57cyclictest4172487-21ssh22:49:583
412442599621,57cyclictest11281-21ssh00:07:373
412442599621,56cyclictest4189738-21ssh23:30:233
4124423996227,31cyclictest13173-21sshd00:12:142
4124423996227,31cyclictest13173-21sshd00:12:142
412441499628,4cyclictest16850irq/134-800000022:52:010
412441499627,51cyclictest16850irq/134-800000000:38:190
412441499623,19cyclictest4130583-21munin-run21:10:000
412442599611,56cyclictest4130867-21ssh21:10:093
412442599611,56cyclictest4130541-21sshd21:05:123
412442599611,55cyclictest48-21kcompactd019:51:253
412442599611,55cyclictest4126160-21sshd19:44:133
412442399614,16cyclictest3790-21cat23:50:012
412442399612,20cyclictest4153647-21cat22:05:012
412442399612,16cyclictest1645-21rm23:45:012
4124423996119,8cyclictest4185325-21cat23:20:022
4124423996119,8cyclictest4185325-21cat23:20:022
412441499617,6cyclictest16850irq/134-800000000:21:210
4124414996152,6cyclictest4129552-21kworker/0:220:50:010
412441499612,4cyclictest131rcu_preempt20:05:010
412442599601,54cyclictest48-21kcompactd020:34:083
412442599601,54cyclictest48-21kcompactd020:07:303
412442599601,54cyclictest48-21kcompactd019:26:273
412442599601,54cyclictest364-21snmpd20:49:103
4124423996026,30cyclictest1-21systemd20:00:012
412442399602,11cyclictest4125411-21unin-run19:30:012
412441499602,9cyclictest275-21dbus-daemon19:30:010
4124414996019,37cyclictest0-21swapper/020:53:020
412442599592,52cyclictest4128017-21munin-run20:20:013
412442599592,52cyclictest4125159-21munin-run19:25:013
412442599591,54cyclictest4130022-21sshd20:58:123
412442599591,53cyclictest48-21kcompactd020:52:273
412442399594,12cyclictest4191410-21cron23:34:592
412442399592,12cyclictest4147375-21wc21:50:012
4124423995920,4cyclictest4150144-21sh21:56:242
4124423995920,4cyclictest18847-21sh00:25:342
412441899596,20cyclictest3467602-21systemd-journal20:25:001
4124418995916,11cyclictest4143121-21sh21:40:011
4124414995917,6cyclictest4125809-21sed19:35:020
412441499591,4cyclictest0-21swapper/019:50:130
412442599581,52cyclictest48-21kcompactd019:10:553
4124423995829,24cyclictest4130588-21sh21:10:002
4124423995822,5cyclictest4162859-21sh22:26:392
4124423995818,5cyclictest3467562-21sshd20:31:122
4124423995816,7cyclictest30-21ksoftirqd/223:22:582
4124423995816,5cyclictest4193170-21sh23:39:102
4124423995816,5cyclictest4193170-21sh23:39:102
412441899585,15cyclictest22617-21modprobe00:35:001
412441499583,19cyclictest4129019-21cut20:40:000
4124423995723,8cyclictest30-21ksoftirqd/221:22:392
412442399572,17cyclictest4129540-21modprobe20:50:002
412442399572,15cyclictest4127900-21latency_hist20:15:022
412442399572,12cyclictest22620-21grep00:35:012
4124423995715,9cyclictest30-21ksoftirqd/200:16:522
4124423995715,10cyclictest30-21ksoftirqd/200:07:252
4124414995729,25cyclictest16950irq/224-800000019:22:130
4124414995726,26cyclictest275-21dbus-daemon20:35:000
412441499571,18cyclictest4128490-21sshd20:29:120
412442599562,49cyclictest364-21snmpd21:00:443
412442599562,49cyclictest364-21snmpd20:04:153
4124423995625,27cyclictest4176041-21sh22:58:242
4124423995624,29cyclictest4124869-21latency_hist19:20:022
412442399562,20cyclictest7940-21systemctl00:00:012
4124423995621,10cyclictest30-21ksoftirqd/222:36:452
4124423995619,8cyclictest30-21ksoftirqd/222:43:392
4124423995615,7cyclictest0-21swapper/222:53:332
412441499563,16cyclictest4130073-21modprobe21:00:010
412441499562,15cyclictest4126192-21munin-run19:45:010
412442599553,47cyclictest4127683-21sshd20:10:133
412442599551,48cyclictest48-21kcompactd019:38:393
4124423995531,20cyclictest4189383-21systemctl23:30:022
4124423995531,20cyclictest4189383-21systemctl23:30:012
4124423995526,25cyclictest30-21ksoftirqd/222:31:062
4124423995517,7cyclictest4169422-1kworker/u9:023:05:572
4124423995517,7cyclictest4169422-1kworker/u9:023:05:572
4124423995514,5cyclictest4125286-21latency_hist19:25:012
412442599542,47cyclictest364-21snmpd19:49:473
412442599541,49cyclictest4124441-21latency_hist19:10:023
412442599541,48cyclictest48-21kcompactd019:17:013
412442599541,48cyclictest48-21kcompactd019:17:013
4124423995423,9cyclictest4127262-21sed20:05:012
4124423995422,4cyclictest4130577-21diskmemload00:21:192
412442399542,20cyclictest4134707-21cat21:20:002
4124423995417,5cyclictest4126677-21latency_hist19:55:032
4124423995416,7cyclictest30-21ksoftirqd/221:10:092
4124423995411,11cyclictest275-21dbus-daemon21:25:482
4124414995421,29cyclictest12-21ksoftirqd/020:09:120
4124414995416,34cyclictest16950irq/224-800000020:11:120
412442599531,47cyclictest364-21snmpd20:29:103
412442399533,21cyclictest4126201-21grep19:45:022
4124423995325,24cyclictest30-21ksoftirqd/222:13:412
4124423995318,4cyclictest4159928-21latency_hist22:20:022
4124423995317,4cyclictest4130577-21diskmemload21:51:092
4124423995313,9cyclictest30-21ksoftirqd/223:01:132
4124423995313,14cyclictest275-21dbus-daemon21:45:012
4124414995319,6cyclictest12-21ksoftirqd/020:20:000
4124423995218,6cyclictest30-21ksoftirqd/222:45:142
4124423995218,6cyclictest30-21ksoftirqd/222:45:142
4124423995215,10cyclictest1-21systemd21:05:012
412442599512,44cyclictest364-21snmpd20:20:183
4124423995115,9cyclictest30-21ksoftirqd/219:20:022
4124423995115,7cyclictest4124547-21latency_hist19:10:022
4124414995110,10cyclictest12-21ksoftirqd/019:19:140
4124414995110,10cyclictest12-21ksoftirqd/019:19:130
412442399502,16cyclictest4129009-21cron20:40:002
4124414995015,9cyclictest12-21ksoftirqd/020:25:020
4124423994915,4cyclictest0-21swapper/220:30:022
412441899499,5cyclictest131rcu_preempt00:00:061
412441499491,12cyclictest4130319-21sh21:05:000
4124418994813,6cyclictest131rcu_preempt22:02:021
4124414994822,23cyclictest4125904-21latency_hist19:40:040
4124423994739,5cyclictest4187297-21kworker/2:300:36:402
412442399472,11cyclictest4124604-21munin-plugin-st19:15:012
4124418994735,8cyclictest1-21systemd19:50:011
412442399463,12cyclictest4125895-21modprobe19:40:002
412441499461,41cyclictest4124611-21systemd-run19:15:010
412442399452,8cyclictest0-21swapper/220:10:012
4124418994535,6cyclictest4180956-21latency_hist23:10:021
4124418994535,6cyclictest4180956-21latency_hist23:10:021
4124418994530,4cyclictest4161715-21sshd22:24:121
4124418994528,13cyclictest275-21dbus-daemon19:20:011
4124418994528,13cyclictest275-21dbus-daemon19:20:011
4124418994525,14cyclictest309-21in:imuxsock21:45:011
4124418994432,8cyclictest3467602-21systemd-journal00:36:151
4124418994410,5cyclictest131rcu_preempt00:07:171
4124414994414,4cyclictest0-21swapper/020:40:410
4124423994326,13cyclictest3467602-21systemd-journal20:45:002
412441899438,6cyclictest131rcu_preempt23:26:261
412441899438,6cyclictest131rcu_preempt23:26:251
412441899437,6cyclictest131rcu_preempt21:13:271
412441899436,27cyclictest4166173-21munin-plugin-st22:35:001
4124418994327,12cyclictest275-21dbus-daemon21:35:021
412441899431,12cyclictest4186240-21ssh23:22:121
412441899427,5cyclictest131rcu_preempt00:15:551
412441899427,26cyclictest4151679-21ssh22:00:011
412441899425,23cyclictest4126241-21wc19:45:021
4124418994230,7cyclictest309-21in:imuxsock23:35:001
4124418994228,11cyclictest23-21ksoftirqd/122:11:431
4124418994227,11cyclictest23-21ksoftirqd/122:54:591
412441899422,3cyclictest131rcu_preempt22:45:001
412441899422,11cyclictest12597-21sh00:10:361
412441899422,11cyclictest12597-21sh00:10:351
4124418994221,15cyclictest275-21dbus-daemon23:40:021
4124418994221,15cyclictest275-21dbus-daemon23:40:011
412441899413,10cyclictest4135045-21ssh21:20:141
4124418994129,8cyclictest23-21ksoftirqd/122:25:231
4124418994123,13cyclictest275-21dbus-daemon20:45:011
412441899406,6cyclictest131rcu_preempt21:16:361
412441899404,29cyclictest4126683-21munin-plugin-st19:55:001
4124418994029,8cyclictest23-21ksoftirqd/123:03:571
4124418994026,10cyclictest3467602-21systemd-journal20:20:001
4124418994024,12cyclictest4159910-21cron22:20:011
412441899395,22cyclictest4129050-21unin-run20:40:011
4124423993822,12cyclictest4130067-21basename21:00:002
412441899388,26cyclictest221rcuc/120:10:021
412441899385,23cyclictest3467602-21systemd-journal21:10:001
4124418993827,8cyclictest4128490-21sshd20:29:121
4124418993826,10cyclictest23-21ksoftirqd/119:11:131
4124418993825,9cyclictest1-21systemd22:50:011
4124418993825,9cyclictest1-21systemd22:50:011
4124418993823,11cyclictest0-21swapper/122:40:001
4124423993729,5cyclictest4116848-21kworker/2:319:50:012
412441899374,6cyclictest131rcu_preempt19:30:141
4124418993720,13cyclictest4130477-21sed21:05:021
4124418993719,10cyclictest309-21in:imuxsock23:54:591
412441899365,27cyclictest4126939-21munin-plugin-st20:00:001
412441899365,21cyclictest18609-21ssh00:25:031
412441899362,7cyclictest0-21swapper/123:49:001
4124418993625,8cyclictest23-21ksoftirqd/121:25:491
4124418993624,8cyclictest4128817-21cat20:35:011
412441499362,31cyclictest4124488-21latency_hist19:10:020
4124423993511,20cyclictest4125647-21munin-plugin-st19:35:002
412441899357,17cyclictest3467602-21systemd-journal19:25:001
412441899354,4cyclictest131rcu_preempt23:10:301
4124418993524,7cyclictest309-21in:imuxsock20:13:131
4124423993427,4cyclictest1-21systemd20:25:022
4124418993222,7cyclictest3467602-21systemd-journal20:05:001
4124418993217,12cyclictest3467602-21systemd-journal20:50:121
412441899313,3cyclictest131rcu_preempt19:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional