You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 02:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Mar 12, 2026 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
108410399470452,15cyclictest1123001-21turbostat09:04:553
108410399458450,5cyclictest1114357-21turbostat08:39:563
108410399456449,4cyclictest1170722-21turbostat11:15:003
108410399456447,6cyclictest1165527-21turbostat11:04:563
108410399456447,6cyclictest1135431-21turbostat09:35:003
108410399454432,19cyclictest1139201-21turbostat09:45:013
108408799451444,4cyclictest1154843-21turbostat10:34:550
108408799451444,4cyclictest1154843-21turbostat10:34:550
108410399448441,4cyclictest1121277-21turbostat08:55:003
108410399448440,5cyclictest1107435-21turbostat08:19:553
108409599448441,4cyclictest1193534-21turbostat12:24:552
108410399444436,5cyclictest1140917-21turbostat09:50:003
108410399444436,5cyclictest1102244-21turbostat08:00:003
108410399442435,4cyclictest1126467-21turbostat09:14:553
108410399442434,5cyclictest1198741-21turbostat12:39:563
108410399442434,5cyclictest1156578-21turbostat10:35:013
108410399442434,5cyclictest1156578-21turbostat10:35:013
108410399441434,4cyclictest1197008-21turbostat12:34:553
108410399440433,4cyclictest1144467-21turbostat10:00:003
108409099440432,4cyclictest1109158-21turbostat08:24:551
108410399438430,5cyclictest1112630-21turbostat08:34:553
108410399437430,4cyclictest1087989-21turbostat07:20:003
108409599436427,6cyclictest1156578-21turbostat10:39:562
108409599435428,4cyclictest1183154-21turbostat11:54:562
108409599435428,4cyclictest1146191-21turbostat10:09:552
108410399434426,5cyclictest1102244-21turbostat08:04:553
108410399434425,6cyclictest1095269-21turbostat07:40:003
108410399432424,5cyclictest1183154-21turbostat11:50:013
108410399432423,6cyclictest1131965-21turbostat09:29:553
108410399430422,5cyclictest1097052-21turbostat07:49:553
108410399430422,5cyclictest1097052-21turbostat07:49:553
108409599426419,4cyclictest1163801-21turbostat10:59:552
108408799426417,6cyclictest1184875-21turbostat11:59:550
108410399425418,4cyclictest1105703-21turbostat08:14:553
108410399425418,4cyclictest1105703-21turbostat08:14:553
108410399424417,4cyclictest1193534-21turbostat12:20:003
108410399423415,5cyclictest1167265-21turbostat11:09:553
108409099423416,4cyclictest1147925-21turbostat10:14:551
108410399422400,19cyclictest1179684-21turbostat11:44:553
108410399421413,5cyclictest1160343-21turbostat10:49:543
108409599421413,5cyclictest1124739-21turbostat09:09:552
108410399420413,4cyclictest1140917-21turbostat09:54:553
108410399419411,5cyclictest1149652-21turbostat10:19:553
108410399417410,4cyclictest1186613-21turbostat12:00:003
108409599416408,5cyclictest1095269-21turbostat07:44:552
108410399415394,18cyclictest1117817-21turbostat08:45:003
108410399413406,4cyclictest1112630-21turbostat08:30:003
108410399407400,4cyclictest1191813-21turbostat12:15:013
108410399406398,5cyclictest1131965-21turbostat09:25:003
108410399406398,5cyclictest1131965-21turbostat09:24:593
108410399402395,4cyclictest1158607-21turbostat10:44:553
108410399401380,18cyclictest1110895-21turbostat08:25:003
108410399400392,5cyclictest1153120-21turbostat10:25:003
108409099400393,4cyclictest1190076-21turbostat12:14:561
108410399399376,20cyclictest1123001-21turbostat09:00:013
108410399398391,4cyclictest1105703-21turbostat08:10:003
108409599398391,4cyclictest1093483-21turbostat07:39:552
108410399397390,4cyclictest1129934-21turbostat09:20:003
108410399396389,4cyclictest1126467-21turbostat09:10:003
108410399396388,5cyclictest1179684-21turbostat11:40:003
108408799395387,5cyclictest1142654-21turbostat09:59:550
108410399393385,5cyclictest1137449-21turbostat09:40:003
108410399393370,20cyclictest1174492-21turbostat11:25:013
108410399393370,20cyclictest1174492-21turbostat11:25:013
108410399390383,4cyclictest1089723-21turbostat07:29:553
108409599389366,20cyclictest1168996-21turbostat11:14:552
108408799388381,4cyclictest1188347-21turbostat12:09:550
108409599384376,5cyclictest1151387-21turbostat10:24:552
108408799383372,8cyclictest1121277-21turbostat08:59:550
108410399382374,5cyclictest1172755-21turbostat11:19:593
108410399381374,4cyclictest1147925-21turbostat10:10:013
108410399379371,5cyclictest1089723-21turbostat07:25:003
108409099379372,4cyclictest1195270-21turbostat12:29:551
108409599377370,4cyclictest1172755-21turbostat11:24:552
108409599377370,4cyclictest1172755-21turbostat11:24:542
108408799376357,16cyclictest1098775-21turbostat07:54:550
108409599375368,4cyclictest1119554-21turbostat08:54:552
108409099371364,4cyclictest1110895-21turbostat08:29:551
108409599370363,4cyclictest1186613-21turbostat12:04:552
108410399369361,5cyclictest1184875-21turbostat11:55:003
108410399369347,3cyclictest1162069-21turbostat10:54:553
108410399366358,5cyclictest1144467-21turbostat10:04:553
108410399363356,4cyclictest1083983-21turbostat07:14:553
108409599362339,20cyclictest1100517-21turbostat07:59:552
108409599361355,3cyclictest1133697-21turbostat09:34:552
108408799359353,3cyclictest1153120-21turbostat10:29:550
108410399356350,3cyclictest1197008-21turbostat12:30:003
108410399354347,4cyclictest1100517-21turbostat07:54:593
108410399354346,5cyclictest1174492-21turbostat11:29:563
108410399353341,9cyclictest1165527-21turbostat11:00:013
108410399352345,4cyclictest1097052-21turbostat07:45:003
108410399348341,4cyclictest1177948-21turbostat11:35:003
108409599348319,15cyclictest1170722-21turbostat11:19:552
108410399339332,4cyclictest1117817-21turbostat08:49:553
108408799339332,4cyclictest1128196-21turbostat09:19:550
108409599335327,5cyclictest1129934-21turbostat09:24:552
108409599335327,5cyclictest1129934-21turbostat09:24:552
108410399328321,4cyclictest1188347-21turbostat12:05:003
108410399326319,4cyclictest1091450-21turbostat07:34:553
108410399323316,4cyclictest1195270-21turbostat12:25:003
108301822240,10sleep20-21swapper/207:05:192
108410399168153,11cyclictest1158607-21turbostat10:40:003
1083767212272,40sleep00-21swapper/007:07:390
1973991210,101rtkit-daemon0-21swapper/307:09:253
10841039910899,6cyclictest1190076-21turbostat12:10:003
1083654210673,23sleep10-21swapper/107:06:121
10841039910089,8cyclictest1154843-21turbostat10:30:003
1084095999790,4cyclictest1177948-21turbostat11:39:552
1084090999790,4cyclictest1191813-21turbostat12:19:561
1084103999689,4cyclictest1149652-21turbostat10:15:003
11584602870,5sleep10-21swapper/110:38:201
11115732610,11sleep2141rcu_preempt08:25:202
11978402580,6sleep10-21swapper/112:30:211
11554842510,7chrt0-21swapper/110:30:151
11554842510,7chrt0-21swapper/110:30:141
11378512480,6sleep1251rcuc/109:40:111
11366802480,5sleep2331rcuc/209:35:222
108409099461,32cyclictest1102478-21acpi08:00:101
11943882440,8chrt0-21swapper/012:20:210
11843522430,5sleep1251rcuc/111:50:281
11184252430,14sleep2341ksoftirqd/208:45:162
11062822430,8chrt0-21swapper/108:10:131
11062822430,8chrt0-21swapper/108:10:131
108408799438,6cyclictest141rcu_preempt07:40:000
1084095994226,12cyclictest1085736-21cron07:15:002
108408799425,7cyclictest141rcu_preempt10:19:590
197399410,29rtkit-daemon2094-21JS10:49:102
1084095994126,11cyclictest923-21dbus-daemon09:55:012
108409099416,9cyclictest26-21ksoftirqd/110:00:331
11676382400,4sleep21167878-21expr11:05:142
108409099406,9cyclictest26-21ksoftirqd/111:20:221
108409099406,9cyclictest26-21ksoftirqd/111:20:221
108409099405,9cyclictest26-21ksoftirqd/108:07:361
1084087994019,13cyclictest2069-21gnome-shell07:42:040
197399390,26rtkit-daemon0-21swapper/209:42:292
197399390,25rtkit-daemon0-21swapper/208:01:092
11749592390,6sleep20-21swapper/211:25:142
108409099396,12cyclictest26-21ksoftirqd/109:35:381
1084090993925,10cyclictest1154847-21ls10:30:001
1084090993921,6cyclictest26-21ksoftirqd/109:45:071
1084090993921,13cyclictest1108174-21sshd08:15:201
1084090993920,14cyclictest1127274-21sed09:10:211
108408799396,6cyclictest141rcu_preempt07:45:210
108408799396,6cyclictest141rcu_preempt07:45:210
1084087993921,13cyclictest1139847-21fw_packets09:45:170
1084087993915,7cyclictest11871242expr12:00:110
1084087993912,8cyclictest13-21ksoftirqd/011:41:340
1084087993911,8cyclictest13-21ksoftirqd/010:56:300
1084087993911,11cyclictest13-21ksoftirqd/010:04:220
11842842380,5chrt0-21swapper/011:50:270
1084095993818,14cyclictest956-21systemd-logind09:57:052
1084095993815,6cyclictest141rcu_preempt11:55:142
1084090993823,10cyclictest26-21ksoftirqd/110:20:181
1084090993821,6cyclictest26-21ksoftirqd/108:33:401
1084090993821,5cyclictest26-21ksoftirqd/109:16:141
1084090993820,6cyclictest26-21ksoftirqd/108:55:011
108408799387,11cyclictest13-21ksoftirqd/010:51:550
1084087993822,12cyclictest1107040-21uptime08:10:290
1084087993822,12cyclictest1107040-21uptime08:10:290
1084087993821,12cyclictest1171243-21apt11:15:100
1084087993813,21cyclictest1158064-40lsb_release10:35:280
197399370,24rtkit-daemon0-21swapper/212:31:352
1084095993722,11cyclictest1160730-21cut10:45:142
1084095993721,12cyclictest1089741-21munin-run07:25:012
1084095993719,14cyclictest1176847-21df11:30:142
1084095993717,16cyclictest1145399-21head10:00:202
1084090993721,5cyclictest26-21ksoftirqd/109:53:291
1084090993721,5cyclictest26-21ksoftirqd/108:49:541
108409099371,15cyclictest1114730-21chrt08:35:131
108408799379,11cyclictest13-21ksoftirqd/008:01:550
108408799378,11cyclictest13-21ksoftirqd/009:05:020
1084087993722,11cyclictest1135123-21sshd09:31:200
1084087993722,11cyclictest1110876-21cron08:25:000
108408799372,12cyclictest1177962-21munin-run11:35:000
1084087993719,14cyclictest1-21systemd07:18:200
1084087993719,14cyclictest1137079-40lsb_release09:35:300
1084087993719,13cyclictest438-20systemd-journal09:45:000
197399360,8rtkit-daemon34-21ksoftirqd/209:29:012
197399360,25rtkit-daemon0-21swapper/211:48:532
1084095993620,12cyclictest1155819-21awk10:30:232
1084095993620,12cyclictest1155819-21awk10:30:232
1084090993621,11cyclictest1036-21gdbus10:40:211
1084090993619,13cyclictest923-21dbus-daemon09:55:141
1084090993618,13cyclictest976-21rs:main1
108408799367,11cyclictest13-21ksoftirqd/011:35:490
108408799366,5cyclictest141rcu_preempt09:50:230
1084087993626,7cyclictest161rcuc/007:27:380
1084087993621,11cyclictest1148903-21ntp_kernel_pll_10:10:220
1084087993621,10cyclictest1-21systemd08:10:010
1084087993620,11cyclictest1161010-21idleruntime10:45:190
1084087993619,12cyclictest438-20systemd-journal11:08:200
1084087993619,12cyclictest1190731-21forks12:10:170
1084087993618,14cyclictest1198353-21users12:30:290
1084087993617,16cyclictest1131435-40lsb_release09:20:290
1084087993617,16cyclictest1131435-40lsb_release09:20:290
197399350,24rtkit-daemon0-21swapper/210:54:002
1084095993520,11cyclictest1091585-21cat07:30:012
1084095993519,11cyclictest1107790-21wc08:15:142
108409099355,10cyclictest26-21ksoftirqd/111:30:561
108409099353,11cyclictest26-21ksoftirqd/112:22:511
1084090993522,9cyclictest1160661-21ls10:45:131
1084090993520,11cyclictest1116505-21cpuspeed_turbos08:40:131
1084090993519,11cyclictest1168604-21awk11:05:301
1084090993518,5cyclictest26-21ksoftirqd/107:41:121
1084090993518,13cyclictest1171961-21ntpq11:15:221
1084090993518,12cyclictest1163393-21cat10:50:301
108409099351,30cyclictest4497-21PK-Backend11:39:041
108409099351,30cyclictest4497-21PK-Backend07:16:231
108408799356,10cyclictest13-21ksoftirqd/008:32:410
1084087993520,11cyclictest438-20systemd-journal11:27:200
1084087993520,11cyclictest1111544-21forks08:25:180
1084087993520,11cyclictest1101724-21threads07:55:280
1084087993520,11cyclictest1089697-21cron07:25:000
1084087993519,12cyclictest1152296-21needreboot10:20:220
1084087993519,12cyclictest1109012-21sshd08:18:200
1084087993518,12cyclictest1158876-21cpu10:40:100
197399340,23rtkit-daemon0-21swapper/212:28:112
1084095993423,6cyclictest34-21ksoftirqd/208:30:512
1084095993420,10cyclictest34-21ksoftirqd/211:40:182
1084095993419,11cyclictest1129625-21sshd09:16:202
1084095993418,12cyclictest976-21rs:main2
1084095993418,11cyclictest466-21systemd-udevd10:16:052
1084095993417,13cyclictest1116716-21df08:40:142
1084095993416,13cyclictest956-21systemd-logind12:12:042
108409599341,11cyclictest0-21swapper/211:04:502
1084090993427,5cyclictest0-21swapper/111:27:101
108409099341,9cyclictest0-21swapper/107:57:031
1084090993419,11cyclictest1091442-21ls07:30:001
1084090993418,12cyclictest1170010-21ntpq11:10:241
1084090993417,6cyclictest1188355-21awk12:05:001
108409099341,30cyclictest4497-21PK-Backend07:24:121
108409099341,29cyclictest4497-21PK-Backend10:56:511
108409099341,12cyclictest4497-21PK-Backend11:03:491
1084087993425,6cyclictest161rcuc/011:23:230
1084087993425,6cyclictest161rcuc/011:23:230
1084087993418,12cyclictest1-21systemd12:29:050
1084087993418,11cyclictest1120661-21cut08:50:260
11247532330,6sleep21124736-21perf09:05:002
108409599336,10cyclictest34-21ksoftirqd/208:05:232
1084095993323,5cyclictest34-21ksoftirqd/212:08:482
1084095993321,7cyclictest1092344-21sh07:30:142
1084095993319,10cyclictest34-21ksoftirqd/207:15:052
1084095993319,10cyclictest1140850-21seq09:49:212
1084095993318,11cyclictest1106485-21grep08:10:202
1084095993318,11cyclictest1106485-21grep08:10:202
1084095993317,12cyclictest1148593-21ls10:10:182
1084095993316,12cyclictest956-21systemd-logind08:59:052
108409099334,5cyclictest141rcu_preempt09:27:231
1084090993320,8cyclictest1134454-21uname09:30:191
1084090993316,5cyclictest26-21ksoftirqd/108:56:461
1084090993316,5cyclictest26-21ksoftirqd/107:10:571
108409099331,4cyclictest4497-21PK-Backend09:20:321
108409099331,4cyclictest4497-21PK-Backend09:20:321
108409099331,4cyclictest4497-21PK-Backend07:32:011
108409099331,29cyclictest4497-21PK-Backend11:55:391
108409099331,12cyclictest4497-21PK-Backend11:44:551
108409099331,12cyclictest4497-21PK-Backend10:17:211
1084087993323,7cyclictest161rcuc/007:32:250
1084087993322,7cyclictest1124739-21turbostat09:05:000
1084087993320,9cyclictest1147496-21sed10:05:290
1084087993319,10cyclictest13-21ksoftirqd/008:45:010
1084087993319,10cyclictest1166501-21awk11:00:210
1084087993318,11cyclictest1132535-21expr09:25:140
1084087993318,11cyclictest1115124-21cut08:35:200
1084087993317,11cyclictest1193085-21timerwakeupswit12:15:290
1084087993317,11cyclictest1169588-21expr11:10:120
1084087993316,12cyclictest1081-21ntpd07:10:240
1084095993221,6cyclictest34-21ksoftirqd/207:54:252
1084090993218,10cyclictest1189003-21awk12:05:181
1084090993217,11cyclictest1100346-21seq07:53:101
108409099321,4cyclictest4497-21PK-Backend11:46:341
1084087993218,9cyclictest1118168-21ls08:45:150
1084087993218,10cyclictest1182716-21cat11:45:280
1084087993214,13cyclictest1126768-21cpu09:10:130
1084087993211,6cyclictest141rcu_preempt12:35:120
1084095993125,4cyclictest0-21swapper/208:38:152
108409599312,15cyclictest4497-21PK-Backend07:45:082
108409599312,15cyclictest4497-21PK-Backend07:45:082
1084095993117,10cyclictest1154397-21grep10:25:292
1084095993115,11cyclictest438-20systemd-journal10:42:202
108409599311,28cyclictest0-21swapper/212:19:022
1084090993114,13cyclictest1097686-21df07:45:151
1084090993114,13cyclictest1097686-21df07:45:151
108409099311,27cyclictest0-21swapper/109:05:501
108409099308,5cyclictest141rcu_preempt10:05:171
108409099301,27cyclictest0-21swapper/112:37:141
1084095992917,8cyclictest1109397-21/usr/sbin/munin08:20:242
108409099262,5cyclictest141rcu_preempt09:00:461
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional