You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-09 - 09:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Jun 09, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
38086799911,85cyclictest384389-21cron20:17:000
380867998426,12cyclictest422059-21ssh22:27:520
38086799812,10cyclictest467862-21munin-run00:10:010
380867998040,37cyclictest16850irq/134-800000000:25:010
380867997968,8cyclictest12-21ksoftirqd/021:10:010
380867997926,13cyclictest418398-21ssh22:19:560
380867997828,10cyclictest275-21dbus-daemon21:45:000
380867997619,13cyclictest390337-21ssh21:17:040
380867997619,13cyclictest390337-21ssh21:17:030
380867997528,5cyclictest432008-21latency_hist22:50:030
380867997528,14cyclictest456608-21ssh23:44:570
38086799751,15cyclictest0-21swapper/023:25:010
38086799748,59cyclictest16850irq/134-800000000:19:590
380867997434,35cyclictest16850irq/134-800000000:37:550
380867997434,35cyclictest16850irq/134-800000000:37:540
380867997431,35cyclictest1-21systemd19:15:020
380867997431,35cyclictest1-21systemd19:15:010
380867997232,36cyclictest389269-21true21:15:010
380867997230,12cyclictest419060-21ssh22:20:590
380867997225,11cyclictest410959-21ssh22:03:020
380871997118,12cyclictest23-21ksoftirqd/100:05:021
380867997123,13cyclictest275-21dbus-daemon19:20:000
380867997123,12cyclictest408804-21ssh21:58:270
380867997123,11cyclictest427888-21ssh22:40:300
38086799702,5cyclictest463435-21munin-run00:00:020
380867997022,13cyclictest477660-21ssh00:31:280
380877996926,27cyclictest418575-21ssh22:20:023
380867996932,32cyclictest16950irq/224-800000000:30:020
380867996927,3cyclictest0-21swapper/023:45:140
38086799683,6cyclictest443135-21modprobe23:14:590
380867996830,34cyclictest16950irq/224-800000000:03:130
38086799682,8cyclictest412894-21ssh22:07:200
38086799682,8cyclictest412894-21ssh22:07:190
380867996827,6cyclictest309-21in:imuxsock22:35:020
380867996824,5cyclictest445667-21sh23:20:030
380867996717,12cyclictest399931-21ssh21:38:340
380867996526,10cyclictest438029-21ssh23:03:290
380867996522,8cyclictest3467602-21systemd-journal19:45:000
38086799651,5cyclictest435306-21ssh22:57:180
38086799651,30cyclictest395923-21ssh21:29:510
380867996325,3cyclictest16850irq/134-800000022:15:000
380867996312,46cyclictest3467602-21systemd-journal20:50:000
380873996252,5cyclictest364659-21kworker/2:100:40:012
380873996252,5cyclictest364659-21kworker/2:100:40:002
38087199624,13cyclictest23-21ksoftirqd/123:30:011
380867996227,4cyclictest433735-21sh22:54:000
38087199616,10cyclictest23-21ksoftirqd/122:07:371
38087199616,10cyclictest23-21ksoftirqd/122:07:361
380867996129,11cyclictest12-21ksoftirqd/021:20:300
380867996128,13cyclictest407156-21ssh21:54:560
380867996125,5cyclictest450939-21sh23:32:090
380867996122,15cyclictest3467602-21systemd-journal20:25:000
38086799611,39cyclictest439073-21ssh23:05:280
38086799611,28cyclictest429619-21kworker/u8:200:11:040
380873996038,17cyclictest275-21dbus-daemon23:59:592
380867996026,12cyclictest396435-21ssh21:30:380
380867996024,4cyclictest426187-21sh22:37:000
380867996021,10cyclictest448331-21ssh23:26:040
380877995910,11cyclictest37-21ksoftirqd/300:16:213
380873995936,16cyclictest275-21dbus-daemon22:40:002
380877995818,7cyclictest37-21ksoftirqd/322:45:003
38087199589,9cyclictest23-21ksoftirqd/122:50:011
38087199589,12cyclictest23-21ksoftirqd/100:32:111
380867995824,8cyclictest12-21ksoftirqd/023:50:090
380877995714,10cyclictest37-21ksoftirqd/323:20:003
380877995712,7cyclictest37-21ksoftirqd/300:37:223
380877995712,7cyclictest37-21ksoftirqd/300:37:213
380877995711,9cyclictest37-21ksoftirqd/322:50:313
380877995710,9cyclictest37-21ksoftirqd/321:39:433
380871995722,6cyclictest419103-21sh22:21:031
380871995717,9cyclictest23-21ksoftirqd/123:46:231
38086799572,31cyclictest403851-21rm21:47:090
38087799563,11cyclictest37-21ksoftirqd/320:10:013
380877995612,10cyclictest37-21ksoftirqd/321:34:043
380877995611,8cyclictest37-21ksoftirqd/322:09:083
380877995611,8cyclictest37-21ksoftirqd/322:09:083
380877995611,8cyclictest37-21ksoftirqd/321:11:443
380871995626,12cyclictest23-21ksoftirqd/122:54:011
380867995624,5cyclictest1-21systemd20:05:010
38087799558,8cyclictest37-21ksoftirqd/320:15:003
38087799558,11cyclictest37-21ksoftirqd/323:45:023
380877995517,8cyclictest37-21ksoftirqd/300:05:003
380877995511,8cyclictest37-21ksoftirqd/323:23:493
38087199554,8cyclictest23-21ksoftirqd/122:15:391
380871995515,8cyclictest23-21ksoftirqd/121:15:011
380871995514,8cyclictest23-21ksoftirqd/100:40:011
380871995514,8cyclictest23-21ksoftirqd/100:40:011
38086799559,25cyclictest19350irq/34-mmc120:10:020
380867995527,4cyclictest383386-21latency_hist20:00:010
380877995411,7cyclictest37-21ksoftirqd/321:55:173
380877995411,10cyclictest37-21ksoftirqd/321:52:003
380871995414,8cyclictest23-21ksoftirqd/123:05:011
380877995314,11cyclictest37-21ksoftirqd/319:55:003
380877995312,8cyclictest37-21ksoftirqd/323:07:133
380873995344,6cyclictest364659-21kworker/2:122:40:072
380873995344,6cyclictest364659-21kworker/2:121:30:022
380873995339,10cyclictest1-21systemd20:00:002
38087199539,7cyclictest23-21ksoftirqd/122:38:421
38087199535,7cyclictest23-21ksoftirqd/121:40:421
380871995321,11cyclictest23-21ksoftirqd/122:10:251
380871995317,10cyclictest23-21ksoftirqd/120:50:031
380871995316,7cyclictest131rcu_preempt22:28:551
380871995315,9cyclictest23-21ksoftirqd/100:00:001
380871995310,8cyclictest23-21ksoftirqd/122:00:011
38087799529,9cyclictest37-21ksoftirqd/322:11:323
38087799528,12cyclictest449684-21ssh23:29:293
38087799526,5cyclictest131rcu_preempt00:00:023
380877995217,7cyclictest37-21ksoftirqd/322:24:053
380877995214,7cyclictest37-21ksoftirqd/321:45:003
380877995214,7cyclictest37-21ksoftirqd/321:27:203
38087799520,15cyclictest474626-21munin-run00:25:023
380873995243,6cyclictest364659-21kworker/2:123:02:242
380873995243,5cyclictest411739-21kworker/2:223:48:272
38087199527,9cyclictest23-21ksoftirqd/121:51:201
38087199526,9cyclictest23-21ksoftirqd/122:30:091
38087199525,11cyclictest23-21ksoftirqd/122:45:031
38087199523,9cyclictest23-21ksoftirqd/121:17:501
38087199523,9cyclictest23-21ksoftirqd/121:17:491
38087199522,8cyclictest452353-21ssh23:35:021
380871995210,11cyclictest444032-21ssh23:16:401
38087799519,7cyclictest37-21ksoftirqd/322:57:423
38087799519,6cyclictest0-21swapper/322:30:033
380877995122,9cyclictest131rcu_preempt00:30:013
380877995112,6cyclictest0-21swapper/322:35:423
380877995110,9cyclictest37-21ksoftirqd/322:30:003
38087399514,3cyclictest131rcu_preempt23:14:402
380873995143,5cyclictest364659-21kworker/2:123:35:012
380873995143,5cyclictest364659-21kworker/2:123:16:442
380873995143,5cyclictest364659-21kworker/2:122:12:172
380873995143,4cyclictest364659-21kworker/2:121:36:012
380873995141,7cyclictest30-21ksoftirqd/221:20:012
380873995141,7cyclictest30-21ksoftirqd/221:20:002
380873995139,7cyclictest364659-21kworker/2:122:08:352
380873995139,7cyclictest364659-21kworker/2:122:08:352
380871995115,9cyclictest23-21ksoftirqd/100:19:551
38087199511,14cyclictest435163-21ssh22:57:001
380867995113,6cyclictest16850irq/134-800000023:37:590
38087799507,4cyclictest131rcu_preempt19:11:153
38087799507,4cyclictest131rcu_preempt19:11:143
380877995013,9cyclictest0-21swapper/300:15:013
380877995012,8cyclictest0-21swapper/321:16:163
380877995012,8cyclictest0-21swapper/321:16:163
380873995043,5cyclictest364659-21kworker/2:122:34:472
380873995042,5cyclictest364659-21kworker/2:121:44:262
380873995041,6cyclictest364659-21kworker/2:122:47:002
380873995041,5cyclictest411739-21kworker/2:200:06:572
380873995033,12cyclictest1-21systemd00:30:002
38087199508,7cyclictest23-21ksoftirqd/121:50:001
38087199506,10cyclictest23-21ksoftirqd/100:23:271
38087199504,13cyclictest23-21ksoftirqd/120:30:011
380871995040,7cyclictest221rcuc/123:42:221
38087199503,9cyclictest23-21ksoftirqd/100:05:371
38087199501,8cyclictest23-21ksoftirqd/121:36:541
380871995010,5cyclictest131rcu_preempt22:02:221
38087799498,7cyclictest37-21ksoftirqd/323:37:383
380877994918,9cyclictest37-21ksoftirqd/323:01:153
380877994911,8cyclictest0-21swapper/300:06:173
38087399494,3cyclictest131rcu_preempt00:19:462
380873994940,6cyclictest380034-21kworker/2:219:40:032
380873994940,5cyclictest411739-21kworker/2:200:13:312
38087199496,9cyclictest23-21ksoftirqd/100:15:011
38087199495,7cyclictest23-21ksoftirqd/121:26:291
38087199494,6cyclictest23-21ksoftirqd/121:30:561
38087199492,8cyclictest23-21ksoftirqd/119:30:021
380871994912,8cyclictest0-21swapper/120:52:261
38087199491,11cyclictest454423-21latency_hist23:40:021
38086799492,43cyclictest386525-21sendmail-msp21:00:010
38087799488,8cyclictest37-21ksoftirqd/322:45:423
38087799488,7cyclictest37-21ksoftirqd/320:30:013
38087799485,9cyclictest37-21ksoftirqd/320:41:363
38087799484,7cyclictest37-21ksoftirqd/321:22:123
38087799482,10cyclictest37-21ksoftirqd/323:34:223
380877994810,7cyclictest0-21swapper/300:35:003
380873994841,5cyclictest403039-21kworker/2:021:58:042
380873994841,5cyclictest364659-21kworker/2:123:24:512
380873994840,5cyclictest411739-21kworker/2:200:21:482
380873994840,5cyclictest380034-21kworker/2:221:10:292
380873994840,5cyclictest364659-21kworker/2:123:39:202
380873994840,5cyclictest364659-21kworker/2:121:45:132
380873994839,6cyclictest364659-21kworker/2:122:19:222
380871994812,9cyclictest23-21ksoftirqd/119:55:001
38087799478,8cyclictest37-21ksoftirqd/320:00:023
38087799474,5cyclictest37-21ksoftirqd/323:11:303
380877994736,6cyclictest309-21in:imuxsock20:20:003
38087799471,9cyclictest3590952-1kworker/3:2H19:10:023
380877994710,7cyclictest0-21swapper/323:48:563
380873994742,3cyclictest364659-21kworker/2:122:04:262
380873994741,3cyclictest364659-21kworker/2:123:05:162
380873994740,4cyclictest364659-21kworker/2:123:42:192
380873994739,5cyclictest411739-21kworker/2:200:00:122
380873994739,5cyclictest380034-21kworker/2:221:21:072
380873994739,5cyclictest364659-21kworker/2:122:51:182
380873994738,5cyclictest380034-21kworker/2:220:44:422
380873994738,5cyclictest364659-21kworker/2:122:25:112
38087199475,7cyclictest23-21ksoftirqd/100:28:251
38087199475,5cyclictest387032-21diskmemload23:05:181
38087199474,6cyclictest385607-21latency_hist20:40:021
38087199471,7cyclictest442578-21ssh23:13:361
38087199471,11cyclictest459654-21ssh23:51:191
380867994717,19cyclictest12-21ksoftirqd/020:35:030
380867994717,19cyclictest12-21ksoftirqd/020:35:030
38087799467,7cyclictest37-21ksoftirqd/321:48:193
38087799463,7cyclictest386870-21date21:05:013
380877994613,6cyclictest0-21swapper/323:50:463
380873994638,5cyclictest411739-21kworker/2:200:31:372
380873994638,5cyclictest380034-21kworker/2:220:56:152
380873994638,5cyclictest380034-21kworker/2:220:49:442
380873994630,12cyclictest30-21ksoftirqd/223:25:232
380873994626,15cyclictest275-21dbus-daemon20:35:022
380873994626,15cyclictest275-21dbus-daemon20:35:012
38087199469,9cyclictest23-21ksoftirqd/121:23:501
380871994632,10cyclictest1-21systemd21:00:001
38087199462,12cyclictest23-21ksoftirqd/120:25:021
380877994529,12cyclictest275-21dbus-daemon21:00:003
380873994537,5cyclictest364659-21kworker/2:121:33:042
380873994530,12cyclictest3467602-21systemd-journal22:57:152
380873994526,15cyclictest3467602-21systemd-journal19:16:592
380873994525,15cyclictest275-21dbus-daemon20:25:002
38087199457,7cyclictest383609-21sed20:00:031
38087199453,9cyclictest23-21ksoftirqd/123:25:031
380877994412,7cyclictest0-21swapper/322:03:433
380873994437,5cyclictest364659-21kworker/2:122:22:402
380873994436,5cyclictest411739-21kworker/2:223:50:432
380873994436,5cyclictest380034-21kworker/2:220:39:402
380873994436,5cyclictest364659-21kworker/2:120:29:072
380873994434,7cyclictest30-21ksoftirqd/219:25:032
380873994421,15cyclictest3467602-21systemd-journal19:30:002
380873994411,8cyclictest131rcu_preempt19:11:142
380873994411,8cyclictest131rcu_preempt19:11:142
38087199447,8cyclictest23-21ksoftirqd/120:20:021
38087199442,5cyclictest0-21swapper/119:50:001
380871994411,7cyclictest0-21swapper/120:34:091
380871994411,7cyclictest0-21swapper/120:34:091
38086799443,7cyclictest385733-21modprobe20:45:010
38087799438,5cyclictest131rcu_preempt21:07:293
380877994310,8cyclictest0-21swapper/319:30:013
380873994319,17cyclictest275-21dbus-daemon20:55:002
38087199434,9cyclictest23-21ksoftirqd/119:25:001
38087199433,8cyclictest23-21ksoftirqd/119:45:011
380871994328,11cyclictest381312-21cron19:20:011
380867994333,7cyclictest19350irq/34-mmc119:25:020
380867994332,8cyclictest385469-21modprobe20:39:590
38087799424,4cyclictest131rcu_preempt19:20:023
380877994224,14cyclictest3467602-21systemd-journal20:39:593
380873994224,13cyclictest3467602-21systemd-journal20:15:002
38087199429,12cyclictest23-21ksoftirqd/119:40:011
38087199427,6cyclictest23-21ksoftirqd/120:45:011
380871994231,6cyclictest309-21in:imuxsock21:05:011
380867994226,13cyclictest386226-21sshd20:53:150
380877994127,10cyclictest1-21systemd19:50:003
380873994133,5cyclictest380034-21kworker/2:219:47:532
380873994129,8cyclictest382598-21sshd19:42:122
38086799411,36cyclictest0-21swapper/019:30:020
38087799408,4cyclictest131rcu_preempt20:31:353
38087799408,4cyclictest131rcu_preempt20:31:343
38087399405,5cyclictest131rcu_preempt20:20:012
38087199409,5cyclictest131rcu_preempt20:05:011
38087199407,6cyclictest23-21ksoftirqd/119:13:161
38087199407,6cyclictest23-21ksoftirqd/119:13:151
38087199406,3cyclictest131rcu_preempt20:15:011
380867994018,16cyclictest275-21dbus-daemon20:30:000
380873993927,9cyclictest383102-21sshd19:52:152
380873993919,15cyclictest275-21dbus-daemon21:10:012
38087199393,7cyclictest23-21ksoftirqd/119:35:021
38087799389,5cyclictest131rcu_preempt19:20:153
380877993824,11cyclictest3467602-21systemd-journal20:45:033
380877993811,5cyclictest131rcu_preempt19:35:153
380873993826,9cyclictest407377-21wc21:55:022
380873993824,10cyclictest309-21in:imuxsock20:07:172
38087199386,5cyclictest131rcu_preempt21:09:141
38087199383,6cyclictest23-21ksoftirqd/119:10:021
380867993817,16cyclictest386790-21unin-run21:05:000
38086799381,6cyclictest382357-21munin-run19:40:010
38086799381,33cyclictest0-21swapper/020:13:150
380877993724,9cyclictest384701-21cron20:25:003
380877993722,11cyclictest37-21ksoftirqd/319:43:153
380873993727,5cyclictest383657-21latency_hist20:05:002
380873993726,7cyclictest380034-21kworker/2:219:35:012
38087399368,5cyclictest131rcu_preempt21:00:352
38087199362,7cyclictest23-21ksoftirqd/120:05:141
38086799364,3cyclictest0-21swapper/019:55:000
380867993619,13cyclictest12-21ksoftirqd/019:35:020
38087799354,6cyclictest37-21ksoftirqd/320:54:153
380877993524,8cyclictest309-21in:imuxsock19:35:003
380877993420,11cyclictest37-21ksoftirqd/320:03:533
38086799341,10cyclictest380905-21sort19:10:010
380867993318,11cyclictest12-21ksoftirqd/019:48:150
38087399322,4cyclictest131rcu_preempt19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional