You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-20 - 14:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3s.osadl.org (updated Mon Jul 20, 2026 00:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1072727998526,12cyclictest1161436-21ssh00:05:420
107272799841,80cyclictest0-21swapper/020:50:030
1072727998330,13cyclictest1122542-21ssh22:43:050
1072727998330,13cyclictest1122542-21ssh22:43:050
1072727998329,14cyclictest1129880-21ssh22:58:550
1072727998327,13cyclictest1174566-21ssh00:34:030
1072727998327,12cyclictest1158304-21ssh23:59:260
1072727998228,13cyclictest1159484-21ssh00:01:300
1072727998132,12cyclictest275-21dbus-daemon20:10:010
1072727998016,4cyclictest131rcu_preempt20:05:010
1072727998014,6cyclictest16850irq/134-800000023:24:290
1072727997826,15cyclictest1080535-21ssh21:13:170
1072727997826,13cyclictest1098723-21ssh21:52:070
1072727997817,56cyclictest16850irq/134-800000000:25:350
1072727997726,12cyclictest1124673-21ssh22:47:450
107272799764,35cyclictest3467602-21systemd-journal21:20:000
1072727997539,31cyclictest16850irq/134-800000022:40:010
1072727997531,4cyclictest16950irq/224-800000023:11:150
1072727997527,12cyclictest1108199-21ssh22:12:260
107272799752,6cyclictest1149193-21sendmail23:40:010
1072727997434,5cyclictest1167954-21cron00:20:000
1072727997427,13cyclictest1086370-21ssh21:25:310
1072727997419,50cyclictest16850irq/134-800000000:36:260
1072727997337,4cyclictest16950irq/224-800000021:25:010
1072727997327,11cyclictest1138215-21ssh23:16:240
107272799732,36cyclictest1151856-21ssh23:45:100
1072727997214,30cyclictest16850irq/134-800000022:52:540
1072727997116,6cyclictest16850irq/134-800000022:20:340
1072727997116,6cyclictest16850irq/134-800000022:20:340
107272799711,27cyclictest0-21swapper/021:40:260
1072727997028,13cyclictest1145709-21ssh23:32:160
1072727997027,11cyclictest1143212-21ssh23:27:050
1072727997016,6cyclictest16850irq/134-800000023:08:080
107272799692,6cyclictest1102303-21munin-run22:00:000
1072727996922,12cyclictest1118237-21ssh22:34:120
1072727996922,12cyclictest1118237-21ssh22:34:120
1072727996915,6cyclictest16850irq/134-800000021:47:070
1072727996915,49cyclictest16850irq/134-800000021:30:230
1072732996825,10cyclictest1-21systemd00:10:002
1072730996822,8cyclictest23-21ksoftirqd/100:10:011
107272799685,17cyclictest1072726-21latency_hist19:10:010
1072727996818,45cyclictest16850irq/134-800000021:35:570
1072727996816,47cyclictest16850irq/134-800000022:28:260
1072732996727,17cyclictest1-21systemd00:25:012
1072732996726,7cyclictest30-21ksoftirqd/222:24:572
1072732996726,7cyclictest30-21ksoftirqd/222:24:572
1072727996714,18cyclictest16850irq/134-800000023:44:460
1072727996713,19cyclictest16850irq/134-800000023:53:490
1072732996626,7cyclictest30-21ksoftirqd/221:41:512
1072732996618,10cyclictest1-21systemd00:35:012
1072727996612,6cyclictest16850irq/134-800000023:01:070
1072732996522,9cyclictest30-21ksoftirqd/223:20:002
1072727996535,4cyclictest16950irq/224-800000019:47:290
1072727996522,12cyclictest1103625-21ssh22:02:400
1072732996429,30cyclictest1159332-21sh00:01:122
1072730996418,15cyclictest23-21ksoftirqd/123:25:021
1072737996339,22cyclictest37-21ksoftirqd/320:10:013
1072732996326,9cyclictest3467602-21systemd-journal19:20:002
1072730996319,10cyclictest23-21ksoftirqd/122:20:011
1072727996328,13cyclictest3467602-21systemd-journal20:55:000
1072732996227,8cyclictest30-21ksoftirqd/223:06:512
1072730996244,14cyclictest3467602-21systemd-journal23:50:001
1072732996126,7cyclictest30-21ksoftirqd/221:16:402
1072732996120,7cyclictest30-21ksoftirqd/221:59:512
1072730996110,8cyclictest23-21ksoftirqd/121:20:021
1072732996024,8cyclictest30-21ksoftirqd/222:12:302
1072732996024,12cyclictest30-21ksoftirqd/223:26:212
1072732996023,6cyclictest1166110-21sh00:15:412
1072730996015,8cyclictest23-21ksoftirqd/123:15:011
1072732995930,25cyclictest30-21ksoftirqd/223:45:562
1072732995923,6cyclictest48-21kcompactd021:24:532
1072732995923,4cyclictest1137451-21latency_hist23:15:032
1072732995820,6cyclictest1-21systemd23:25:012
107273299581,5cyclictest131rcu_preempt22:08:502
107273099585,11cyclictest23-21ksoftirqd/121:54:121
1072730995813,8cyclictest23-21ksoftirqd/123:05:001
107273799573,42cyclictest1170323-21ssh00:25:033
1072730995737,14cyclictest3467602-21systemd-journal19:40:021
1072727995719,33cyclictest16850irq/134-800000022:05:060
1072737995637,12cyclictest309-21in:imuxsock20:00:013
1072732995622,7cyclictest30-21ksoftirqd/219:25:002
107273099565,11cyclictest23-21ksoftirqd/123:30:571
1072730995618,8cyclictest23-21ksoftirqd/100:02:471
1072730995618,7cyclictest23-21ksoftirqd/121:28:561
1072737995532,17cyclictest275-21dbus-daemon21:30:013
1072732995546,5cyclictest1078924-21kworker/2:323:55:172
1072732995523,6cyclictest30-21ksoftirqd/222:30:482
1072732995523,6cyclictest30-21ksoftirqd/222:30:472
1072732995512,12cyclictest275-21dbus-daemon19:15:002
1072732995445,5cyclictest1078924-21kworker/2:323:03:242
1072732995445,5cyclictest1078924-21kworker/2:322:36:412
1072732995425,5cyclictest30-21ksoftirqd/223:53:502
1072732995425,4cyclictest1078470-21latency_hist21:00:022
1072732995424,8cyclictest30-21ksoftirqd/223:45:012
1072732995420,10cyclictest30-21ksoftirqd/222:50:522
107273099548,8cyclictest23-21ksoftirqd/120:40:031
107273099547,10cyclictest23-21ksoftirqd/100:00:011
107273099546,11cyclictest23-21ksoftirqd/123:15:561
107273099545,12cyclictest23-21ksoftirqd/123:42:591
107273099545,12cyclictest23-21ksoftirqd/122:06:031
107273099545,12cyclictest23-21ksoftirqd/121:57:251
1072730995439,10cyclictest3467602-21systemd-journal22:46:301
1072730995412,11cyclictest23-21ksoftirqd/123:06:171
1072730995410,10cyclictest23-21ksoftirqd/122:30:041
1072730995410,10cyclictest23-21ksoftirqd/122:30:031
1072737995337,13cyclictest361rcuc/323:00:043
1072732995344,6cyclictest1078924-21kworker/2:323:33:232
107273099537,12cyclictest23-21ksoftirqd/100:21:401
107273099535,10cyclictest23-21ksoftirqd/121:13:021
1072730995316,7cyclictest23-21ksoftirqd/100:39:201
1072727995317,32cyclictest16850irq/134-800000022:15:430
1072727995317,32cyclictest16850irq/134-800000000:22:570
107273799525,34cyclictest37-21ksoftirqd/322:55:023
1072732995243,6cyclictest1172676-21kworker/2:200:39:582
1072732995223,7cyclictest30-21ksoftirqd/221:50:072
1072732995222,7cyclictest30-21ksoftirqd/221:28:062
107273299522,14cyclictest1125619-21munin-plugin-st22:49:592
107273099526,10cyclictest23-21ksoftirqd/123:50:431
107273099524,12cyclictest23-21ksoftirqd/122:43:571
107273099524,12cyclictest23-21ksoftirqd/122:43:561
107273099524,11cyclictest23-21ksoftirqd/100:15:031
107273099522,13cyclictest23-21ksoftirqd/119:55:011
1072727995224,13cyclictest1163676-21ssh00:10:250
1072737995137,10cyclictest275-21dbus-daemon23:20:013
1072732995142,5cyclictest1078138-21kworker/2:022:40:272
1072732995142,5cyclictest1078138-21kworker/2:022:40:262
107273099517,8cyclictest23-21ksoftirqd/123:25:291
107273099516,6cyclictest23-21ksoftirqd/119:15:021
107273099515,9cyclictest23-21ksoftirqd/121:20:151
1072730995131,16cyclictest275-21dbus-daemon22:55:011
107273099512,14cyclictest23-21ksoftirqd/100:33:171
107273099512,10cyclictest23-21ksoftirqd/100:29:091
1072730995111,10cyclictest23-21ksoftirqd/122:26:111
1072737995030,15cyclictest1-21systemd22:35:023
1072737995030,15cyclictest1-21systemd22:35:013
1072732995043,4cyclictest1078138-21kworker/2:021:30:302
1072732995042,5cyclictest1078924-21kworker/2:322:16:302
1072732995042,5cyclictest1078138-21kworker/2:022:01:562
1072732995042,5cyclictest1078138-21kworker/2:021:14:592
1072732995041,5cyclictest1074253-21kworker/2:319:45:032
1072732995041,5cyclictest1074253-21kworker/2:319:45:022
107273299501,4cyclictest131rcu_preempt21:37:022
107273099507,8cyclictest23-21ksoftirqd/122:25:021
107273099507,8cyclictest23-21ksoftirqd/122:25:011
1072730995017,13cyclictest23-21ksoftirqd/121:41:381
107273099501,12cyclictest23-21ksoftirqd/120:05:021
1072732994941,5cyclictest1078924-21kworker/2:321:48:102
107273299492,4cyclictest131rcu_preempt20:00:002
107273099498,12cyclictest23-21ksoftirqd/121:37:591
1072730994940,7cyclictest23-21ksoftirqd/122:01:581
1072730994912,6cyclictest23-21ksoftirqd/121:31:071
1072730994912,10cyclictest23-21ksoftirqd/122:39:431
1072730994911,11cyclictest23-21ksoftirqd/122:59:021
1072727994928,17cyclictest3467718-21lldpd19:20:010
1072727994926,20cyclictest12-21ksoftirqd/020:16:520
107273799486,38cyclictest1102485-21ssh22:00:023
1072737994829,15cyclictest1-21systemd19:40:013
1072737994823,20cyclictest37-21ksoftirqd/321:15:013
1072732994840,5cyclictest1078924-21kworker/2:323:35:362
1072732994839,6cyclictest1078924-21kworker/2:322:55:532
1072732994835,8cyclictest1172558-21sshd00:29:502
107273099485,10cyclictest23-21ksoftirqd/121:50:011
1072730994838,7cyclictest23-21ksoftirqd/123:35:101
107273099483,11cyclictest23-21ksoftirqd/122:13:291
1072730994815,11cyclictest23-21ksoftirqd/121:10:011
107272799484,4cyclictest131rcu_preempt19:35:010
1072727994841,4cyclictest16950irq/224-800000021:10:010
1072732994739,5cyclictest1078924-21kworker/2:322:25:112
1072732994738,5cyclictest1074175-21kworker/2:120:35:012
107273099476,5cyclictest23-21ksoftirqd/100:10:411
107273099475,7cyclictest23-21ksoftirqd/120:45:011
107273099472,11cyclictest23-21ksoftirqd/119:50:011
107273099470,16cyclictest1078504-21sed21:00:011
1072732994622,5cyclictest1074993-21latency_hist19:55:032
107273099464,11cyclictest23-21ksoftirqd/119:20:031
107273099461,14cyclictest1075891-21date20:10:021
1072737994529,11cyclictest309-21in:imuxsock21:30:023
1072732994536,6cyclictest1078924-21kworker/2:300:13:232
1072732994522,16cyclictest3467602-21systemd-journal19:34:592
1072732994516,10cyclictest1075981-21cut20:10:042
107273299451,4cyclictest131rcu_preempt20:30:012
107273099455,8cyclictest23-21ksoftirqd/120:16:591
1072737994433,7cyclictest274-21cron20:19:593
1072737994432,7cyclictest274-21cron19:20:003
1072737994430,8cyclictest1177347-21ssh00:40:013
107273799442,6cyclictest131rcu_preempt00:35:003
1072730994413,8cyclictest23-21ksoftirqd/119:25:191
1072730994413,8cyclictest23-21ksoftirqd/119:25:191
1072727994437,4cyclictest16950irq/224-800000020:00:020
1072727994430,10cyclictest3467602-21systemd-journal21:02:070
1072727994430,10cyclictest3467602-21systemd-journal21:02:060
1072727994425,14cyclictest1-21systemd20:15:010
1072727994425,14cyclictest1-21systemd20:15:010
1072737994332,9cyclictest37-21ksoftirqd/322:24:113
1072737994332,9cyclictest37-21ksoftirqd/322:24:103
1072737994325,14cyclictest1-21systemd23:55:013
107273299436,6cyclictest131rcu_preempt20:40:002
1072732994329,10cyclictest3467602-21systemd-journal19:30:002
1072732994329,10cyclictest3467602-21systemd-journal19:30:002
1072732994329,10cyclictest309-21in:imuxsock20:17:002
107273299431,4cyclictest131rcu_preempt19:40:012
107273099437,7cyclictest23-21ksoftirqd/120:25:021
107273099434,8cyclictest23-21ksoftirqd/121:05:011
107273099434,8cyclictest23-21ksoftirqd/121:05:001
107273099433,10cyclictest23-21ksoftirqd/119:56:401
1072727994327,12cyclictest1074966-21sshd19:51:570
107272799431,3cyclictest131rcu_preempt19:30:020
107272799431,3cyclictest131rcu_preempt19:30:020
107273799428,6cyclictest131rcu_preempt00:10:553
107273799425,6cyclictest131rcu_preempt22:05:523
1072737994230,9cyclictest1128394-21sshd22:55:293
1072737994229,11cyclictest37-21ksoftirqd/323:12:543
1072737994229,11cyclictest37-21ksoftirqd/322:36:563
1072737994228,11cyclictest37-21ksoftirqd/300:27:423
1072737994228,10cyclictest275-21dbus-daemon21:10:003
1072737994225,12cyclictest275-21dbus-daemon19:30:023
1072737994225,12cyclictest275-21dbus-daemon19:30:013
1072737994225,12cyclictest0-21swapper/323:25:003
1072737994223,15cyclictest275-21dbus-daemon20:35:003
1072737994221,5cyclictest37-21ksoftirqd/323:45:443
1072737994211,5cyclictest131rcu_preempt00:00:013
1072732994233,5cyclictest1074253-21kworker/2:319:50:002
1072732994229,10cyclictest3467602-21systemd-journal21:03:322
1072732994229,10cyclictest3467602-21systemd-journal21:03:322
1072732994228,11cyclictest1-21systemd20:50:002
107273299422,3cyclictest131rcu_preempt20:21:292
107273099425,7cyclictest23-21ksoftirqd/120:30:011
1072730994230,7cyclictest309-21in:imuxsock20:55:021
107272799421,7cyclictest1077315-21cron20:40:000
107273799414,5cyclictest131rcu_preempt22:50:003
1072737994130,9cyclictest37-21ksoftirqd/322:25:553
1072737994130,9cyclictest37-21ksoftirqd/322:11:273
1072737994128,10cyclictest37-21ksoftirqd/321:54:023
1072737994128,10cyclictest37-21ksoftirqd/321:44:533
1072737994128,10cyclictest37-21ksoftirqd/321:24:143
1072737994127,12cyclictest37-21ksoftirqd/322:05:003
1072737994125,11cyclictest309-21in:imuxsock00:20:003
1072737994123,12cyclictest275-21dbus-daemon19:45:013
1072737994123,12cyclictest275-21dbus-daemon19:45:013
1072737994120,15cyclictest3467602-21systemd-journal20:55:013
107273299416,8cyclictest30-21ksoftirqd/221:08:302
107273099414,9cyclictest23-21ksoftirqd/120:35:001
107273099412,9cyclictest23-21ksoftirqd/119:10:021
107272799412,8cyclictest275-21dbus-daemon21:00:010
1072727994123,14cyclictest1073432-21sshd19:23:290
107273799403,4cyclictest131rcu_preempt23:27:093
107273799403,32cyclictest361rcuc/323:40:023
107273299404,6cyclictest131rcu_preempt20:05:012
107273099401,9cyclictest1077996-21wc20:50:021
107272799404,5cyclictest131rcu_preempt20:25:010
1072727994031,6cyclictest19350irq/34-mmc119:41:250
1072727994031,6cyclictest19350irq/34-mmc119:41:240
1072727994029,7cyclictest1077594-21cron20:44:590
1072737993928,9cyclictest37-21ksoftirqd/319:25:013
1072737993927,9cyclictest37-21ksoftirqd/323:32:203
1072737993926,10cyclictest275-21dbus-daemon23:10:003
1072737993925,10cyclictest37-21ksoftirqd/322:18:033
1072737993924,10cyclictest0-21swapper/321:45:223
1072737993922,13cyclictest1-21systemd19:55:013
1072737993920,14cyclictest1078372-21systemctl21:00:003
1072737993919,14cyclictest37-21ksoftirqd/320:44:593
1072732993931,5cyclictest1076054-21kworker/2:220:41:502
107273099396,9cyclictest23-21ksoftirqd/119:30:321
107273099394,6cyclictest23-21ksoftirqd/119:45:021
107273099394,6cyclictest23-21ksoftirqd/119:45:021
107273099391,8cyclictest23-21ksoftirqd/119:25:011
1072727993925,10cyclictest3467602-21systemd-journal19:15:000
107273799385,5cyclictest131rcu_preempt23:45:033
1072737993828,7cyclictest37-21ksoftirqd/320:50:003
1072737993827,6cyclictest1074737-21sh19:49:593
1072737993825,9cyclictest310-21in:imklog21:39:103
1072737993825,10cyclictest37-21ksoftirqd/322:40:033
1072737993825,10cyclictest37-21ksoftirqd/322:40:023
107273799381,6cyclictest131rcu_preempt19:35:003
1072727993830,4cyclictest16950irq/224-800000019:40:000
1072737993726,9cyclictest37-21ksoftirqd/320:15:033
1072737993726,9cyclictest37-21ksoftirqd/320:15:023
1072737993721,12cyclictest1076810-21sshd20:29:293
1072737993721,11cyclictest37-21ksoftirqd/321:20:023
107273099377,7cyclictest23-21ksoftirqd/120:11:441
107273099377,7cyclictest23-21ksoftirqd/120:11:441
107273799366,6cyclictest131rcu_preempt00:00:203
1072737993626,6cyclictest1161679-21sh00:06:133
1072737993623,10cyclictest275-21dbus-daemon19:15:013
1072737993622,11cyclictest37-21ksoftirqd/321:05:023
1072737993622,11cyclictest37-21ksoftirqd/321:05:013
1072737993610,5cyclictest131rcu_preempt20:05:003
107272799362,4cyclictest131rcu_preempt20:35:010
1072737993526,6cyclictest0-21swapper/320:25:023
1072732993524,7cyclictest30-21ksoftirqd/220:55:022
1072727993521,10cyclictest12-21ksoftirqd/020:26:050
107273799346,5cyclictest131rcu_preempt20:37:023
107273299347,7cyclictest30-21ksoftirqd/219:10:022
1072732993326,4cyclictest1076029-21latency_hist20:15:022
1072732993326,4cyclictest1076029-21latency_hist20:15:022
1072737993021,5cyclictest1072797-21latency_hist19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional