You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-27 - 00:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Aug 26, 2026 12:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11357279911329,63cyclictest1211845-21ssh11:37:140
1135727998928,56cyclictest1179247-21ssh10:28:170
1135727998731,51cyclictest1239647-21ssh12:36:070
1135727998731,51cyclictest1239647-21ssh12:36:070
1135727998624,58cyclictest1236176-21ssh12:29:020
1135727998528,38cyclictest1184061-21ssh10:38:330
1135727998037,39cyclictest275-21dbus-daemon07:50:000
1135727998021,55cyclictest1200381-21ssh11:13:070
1135727997922,38cyclictest1197166-21ssh11:06:020
113572799783,7cyclictest1228878-21ssh12:13:280
1135727997827,45cyclictest1-21systemd12:25:000
1135727997822,51cyclictest1229867-21ssh12:15:130
113572799772,7cyclictest1196466-21grep11:05:010
113572799772,16cyclictest1237302-21ssh12:31:120
1135727997719,48cyclictest16850irq/134-800000012:06:050
1135727997716,56cyclictest16850irq/134-800000011:59:240
113572799751,70cyclictest1171126-21ssh10:11:010
1135727997427,43cyclictest1182315-21munin-run10:35:010
1135727997427,30cyclictest1166758-21ssh10:01:550
1135727997412,57cyclictest16850irq/134-800000010:19:330
1135731997361,7cyclictest0-21swapper/107:10:021
113572799735,58cyclictest121467250ssh11:43:230
1135727997332,37cyclictest48-21kcompactd011:51:070
113572799731,7cyclictest1177069-21ssh10:23:490
1135727997310,58cyclictest16850irq/134-800000009:56:030
113572799729,13cyclictest1203964-21ssh11:20:240
1135727997215,52cyclictest16850irq/134-800000011:27:390
1135727997214,53cyclictest16850irq/134-800000009:16:430
1135737997119,11cyclictest1224990-21cut12:05:033
1135727997128,25cyclictest1147326-21ssh09:20:570
113572799711,66cyclictest0-21swapper/009:43:110
1135727997111,50cyclictest16850irq/134-800000009:48:180
113572799704,61cyclictest114282750ssh09:11:330
1135727997032,34cyclictest16950irq/224-800000007:40:000
1135727997031,35cyclictest1152203-21ssh09:31:130
1135727997031,35cyclictest1152203-21ssh09:31:120
113572799692,6cyclictest1169988-21ssh10:08:580
1135727996915,49cyclictest16850irq/134-800000010:45:300
1135727996829,35cyclictest1192425-21ssh10:56:030
1135727996828,36cyclictest1210243-21ssh11:33:590
1135727996816,47cyclictest16850irq/134-800000011:17:340
1135727996711,51cyclictest16850irq/134-800000009:36:300
113573399662,17cyclictest1173228-21ssh10:15:282
113572799668,7cyclictest16850irq/134-800000010:50:160
113572799661,50cyclictest0-21swapper/009:50:570
1135737996426,12cyclictest275-21dbus-daemon10:40:023
1135737996423,5cyclictest1172847-21latency_hist10:15:033
113573799642,21cyclictest3467602-21systemd-journal11:20:003
1135737996421,5cyclictest309-21in:imuxsock10:25:003
1135733996425,35cyclictest1240418-21sh12:37:542
1135733996425,35cyclictest1240418-21sh12:37:532
113572799642,58cyclictest122461450ssh12:04:260
113572799642,16cyclictest1185670-21ssh10:41:430
1135727996421,38cyclictest1138020-21sshd07:54:000
1135737996324,4cyclictest1136749-21latency_hist07:30:023
1135733996319,40cyclictest1181600-1kworker/u9:110:42:302
1135733996318,40cyclictest1230443-1kworker/u9:112:21:242
1135737996228,7cyclictest1139841-21sshd08:28:173
1135737996228,7cyclictest1139841-21sshd08:28:173
113573399622,17cyclictest1218184-21ssh11:50:312
113573399622,13cyclictest1235478-21ssh12:27:222
113572799622,17cyclictest1216356-21ssh11:46:470
1135737996125,5cyclictest1234737-21sh12:25:383
113573399613,17cyclictest1170517-21ssh10:10:022
1135733996123,32cyclictest30-21ksoftirqd/209:58:292
113573399612,17cyclictest1168085-21ssh10:04:592
1135733996119,38cyclictest1184762-21latency_hist10:40:012
113573399611,21cyclictest1209562-21ssh11:32:252
1135727996115,41cyclictest16850irq/134-800000009:29:470
113573399608,48cyclictest1-21systemd09:45:012
113573399602,16cyclictest1201234-21munin-run11:15:022
113573399601,7cyclictest1237980-21ssh12:32:462
113573399601,7cyclictest1220835-21ssh11:56:152
113573399601,7cyclictest1194008-21ssh10:59:452
113573399601,6cyclictest1150842-21ssh09:28:422
1135733996016,39cyclictest1195765-21diskmemload11:03:232
113573399601,5cyclictest1140340-21sshd08:36:582
113573399601,17cyclictest1171740-21ssh10:12:322
1135727996022,33cyclictest1138548-21sshd08:03:590
113572799601,12cyclictest1139073-21munin-run08:15:000
1135737995931,25cyclictest37-21ksoftirqd/310:55:513
1135733995926,29cyclictest30-21ksoftirqd/211:43:352
113573399592,17cyclictest1203511-21ssh11:19:532
113573399592,16cyclictest1215797-21ssh11:45:282
113573399591,9cyclictest1205892-21ssh11:25:002
113573399591,6cyclictest1190285-21ssh10:51:362
113573399591,6cyclictest1159595-21ssh09:46:482
1135727995913,41cyclictest16850irq/134-800000008:23:440
1135737995828,10cyclictest37-21ksoftirqd/312:40:013
1135737995828,10cyclictest37-21ksoftirqd/312:40:003
1135737995821,10cyclictest37-21ksoftirqd/309:16:473
1135737995820,12cyclictest275-21dbus-daemon11:50:003
1135737995819,8cyclictest37-21ksoftirqd/310:20:023
113573399582,6cyclictest1187996-21ssh10:46:462
1135733995822,32cyclictest30-21ksoftirqd/211:09:052
113573399582,17cyclictest1152571-21ssh09:32:092
113573399582,17cyclictest1152571-21ssh09:32:082
113573399582,16cyclictest1137004-21munin-run07:35:002
113573399581,7cyclictest1182132-21ssh10:34:402
113573399581,7cyclictest1145671-21ssh09:17:442
113573399581,6cyclictest1212899-21ssh11:39:462
113573399581,6cyclictest1176611-21ssh10:22:482
113573399581,6cyclictest1142560-21ssh09:10:472
113573399581,5cyclictest1228644-21ssh12:12:592
1135737995726,6cyclictest1154534-21sh09:36:143
1135737995725,8cyclictest37-21ksoftirqd/309:11:063
1135737995724,8cyclictest37-21ksoftirqd/309:43:393
1135737995724,11cyclictest37-21ksoftirqd/309:21:043
1135737995722,8cyclictest37-21ksoftirqd/309:56:313
113573399572,16cyclictest1179959-21munin-run10:30:012
113573399572,16cyclictest1139085-21munin-run08:15:002
113573399571,5cyclictest1223095-21ssh12:00:562
113573399571,5cyclictest1161188-21ssh09:50:032
113573399571,4cyclictest1146993-21ssh09:20:042
113572799577,41cyclictest1135726-21wc07:10:020
113572799572,16cyclictest1137640-21sort07:45:020
113573799561,33cyclictest37-21ksoftirqd/311:04:063
1135733995618,32cyclictest275-21dbus-daemon07:25:022
113573399561,7cyclictest1226978-21ssh12:09:362
113573399561,6cyclictest1206438-21ssh11:25:412
113573399561,6cyclictest1156302-21ssh09:40:032
113573399561,6cyclictest1141378-21sshd08:57:582
1135731995641,12cyclictest1-21systemd08:45:001
1135731995634,17cyclictest3467602-21systemd-journal09:05:011
1135731995634,17cyclictest3467602-21systemd-journal09:05:001
113572799562,7cyclictest1138331-21munin-run08:00:010
1135727995621,31cyclictest3467602-21systemd-journal08:43:490
113572799561,15cyclictest1136216-21modprobe07:20:000
1135727995610,39cyclictest309-21in:imuxsock08:38:590
113573799555,7cyclictest131rcu_preempt10:10:013
1135737995528,12cyclictest37-21ksoftirqd/312:24:213
113573399551,16cyclictest1229899-21rm12:15:182
113572799552,14cyclictest1140045-21wc08:30:030
113572799552,14cyclictest1140045-21wc08:30:020
113572799552,13cyclictest1140118-21munin-run08:35:010
1135737995424,5cyclictest1141941-21diskmemload11:34:193
1135737995420,10cyclictest1152095-21sshd09:30:593
1135737995420,10cyclictest1152095-21sshd09:30:593
113573399542,5cyclictest364-21snmpd07:58:382
113573399541,6cyclictest1-21systemd07:45:002
1135733995414,36cyclictest1-21systemd07:30:002
1135733995413,37cyclictest30-21ksoftirqd/207:20:012
1135731995436,14cyclictest23-21ksoftirqd/109:45:141
113572799542,14cyclictest1136513-21munin-run07:25:010
1135727995421,28cyclictest275-21dbus-daemon08:10:020
1135737995323,7cyclictest37-21ksoftirqd/312:08:293
1135737995320,8cyclictest37-21ksoftirqd/311:56:403
1135733995337,13cyclictest1-21systemd08:10:012
1135733995336,13cyclictest1137262-21cat07:40:012
113573399532,6cyclictest364-21snmpd08:22:302
113573399532,5cyclictest1-21systemd08:55:022
113573399532,15cyclictest1140652-21munin-run08:45:012
113573399532,15cyclictest1140131-21munin-run08:35:002
113573399531,6cyclictest364-21snmpd08:28:462
113573399531,6cyclictest364-21snmpd08:28:452
1135731995333,15cyclictest3467602-21systemd-journal11:55:011
1135727995311,36cyclictest2129045-1kworker/0:2H08:55:020
1135737995220,9cyclictest37-21ksoftirqd/310:35:023
1135737995219,11cyclictest37-21ksoftirqd/312:10:353
1135733995224,23cyclictest3467602-21systemd-journal08:20:012
113573399522,12cyclictest1141876-21latency_hist09:05:032
113573399522,12cyclictest1141876-21latency_hist09:05:022
1135731995226,17cyclictest275-21dbus-daemon10:45:011
113572799523,6cyclictest1141412-21/usr/sbin/munin09:00:010
113572799521,12cyclictest1135960-21munin-run07:15:010
1135737995111,5cyclictest131rcu_preempt11:11:293
113573399518,37cyclictest0-21swapper/208:48:582
113573399512,6cyclictest1137908-21sed07:50:012
113573399512,5cyclictest0-21swapper/208:04:072
113573199515,36cyclictest1213014-21modprobe11:40:001
1135731995141,7cyclictest23-21ksoftirqd/110:36:361
1135731995136,11cyclictest3467602-21systemd-journal11:30:051
113572799512,9cyclictest275-21dbus-daemon09:05:010
113572799512,9cyclictest275-21dbus-daemon09:05:010
1135737995018,12cyclictest37-21ksoftirqd/311:53:323
1135737995013,6cyclictest131rcu_preempt10:02:503
113573399502,6cyclictest1135957-21modprobe07:15:002
113573199505,23cyclictest48-21kcompactd012:05:021
1135731995030,15cyclictest0-21swapper/112:40:011
1135731995030,15cyclictest0-21swapper/112:40:001
1135727995012,33cyclictest1136730-21sshd07:28:110
113573399492,4cyclictest0-21swapper/209:08:532
113573399491,13cyclictest1138018-21sshd07:54:002
113573199494,3cyclictest131rcu_preempt11:00:181
1135731994927,16cyclictest1-21systemd10:05:021
1135731994926,13cyclictest309-21in:imuxsock12:15:031
113573199492,4cyclictest131rcu_preempt11:22:371
113572799492,13cyclictest1140961-21latency_hist08:50:010
1135737994832,13cyclictest37-21ksoftirqd/308:00:013
1135731994828,12cyclictest3467602-21systemd-journal08:00:011
113572799487,5cyclictest1139338-21sendmail08:20:010
113572799482,7cyclictest1-21systemd09:10:010
1135737994734,9cyclictest309-21in:imuxsock08:15:023
113573799472,14cyclictest1150525-21ssh09:27:523
113573199477,28cyclictest1146706-21modprobe09:19:591
113573199476,29cyclictest1138821-21modprobe08:10:001
1135731994732,11cyclictest1239015-21ssh12:35:011
113572799473,6cyclictest1136997-21modprobe07:35:000
1135737994611,6cyclictest131rcu_preempt10:28:473
113573199467,29cyclictest1165689-21modprobe09:59:591
113573199467,15cyclictest1137266-21systemctl07:40:001
113573199463,3cyclictest131rcu_preempt09:35:001
113573199463,3cyclictest131rcu_preempt09:34:591
113573199462,3cyclictest131rcu_preempt11:18:151
1135737994535,6cyclictest1136221-21sh07:20:013
1135737994530,11cyclictest275-21dbus-daemon09:10:013
1135737994526,14cyclictest275-21dbus-daemon12:35:013
113573199454,3cyclictest131rcu_preempt12:26:131
113573199454,3cyclictest131rcu_preempt10:57:241
113573199454,28cyclictest1139344-21grep08:20:001
1135731994533,7cyclictest1139071-21munin-plugin-st08:15:001
1135731994523,16cyclictest3467602-21systemd-journal07:20:011
1135731994519,17cyclictest275-21dbus-daemon08:05:001
1135737994429,11cyclictest1198813-21sshd11:10:003
113573399442,5cyclictest1135887-21latency_hist07:10:032
113573199444,25cyclictest3467602-21systemd-journal09:09:591
1135731994424,13cyclictest3467602-21systemd-journal09:55:001
113573199442,3cyclictest131rcu_preempt10:52:071
113573799434,4cyclictest150-21pr/ttyS209:00:003
113573199434,3cyclictest131rcu_preempt11:05:151
1135731994331,10cyclictest23-21ksoftirqd/109:24:041
1135731994330,10cyclictest3467602-21systemd-journal07:54:591
1135731994328,9cyclictest309-21in:imuxsock10:30:021
1135731994323,12cyclictest1222455-21munin-plugin-st12:00:001
1135731994229,10cyclictest23-21ksoftirqd/109:40:011
1135731994229,10cyclictest1-21systemd10:10:001
1135731994226,12cyclictest23-21ksoftirqd/112:09:081
1135737994130,7cyclictest3467602-21systemd-journal08:35:023
1135737994129,9cyclictest3467602-21systemd-journal07:40:003
113573199416,24cyclictest1141398-21modprobe08:59:591
1135731994129,10cyclictest23-21ksoftirqd/112:25:021
1135731994128,10cyclictest23-21ksoftirqd/110:30:141
1135731994125,12cyclictest310-21in:imklog10:48:461
113573199411,4cyclictest131rcu_preempt11:44:461
1135731994110,10cyclictest131rcu_preempt11:15:031
1135737994030,4cyclictest309-21in:imuxsock08:50:013
1135737994018,16cyclictest275-21dbus-daemon11:45:013
113573799401,6cyclictest131rcu_preempt10:54:173
1135731994030,8cyclictest23-21ksoftirqd/109:25:101
1135731994027,8cyclictest1141143-21cron08:55:001
1135731994027,10cyclictest1217918-21wc11:50:021
113573199402,4cyclictest131rcu_preempt10:15:001
113573199402,3cyclictest131rcu_preempt09:10:301
1135737993930,6cyclictest1163307-21munin-plugin-st09:55:003
1135737993929,8cyclictest37-21ksoftirqd/312:18:263
1135737993927,7cyclictest1187085-21unin-run10:45:013
113573199395,25cyclictest1177807-21latency_hist10:25:021
1135731993924,11cyclictest23-21ksoftirqd/108:40:011
1135737993826,8cyclictest1139597-21sshd08:24:593
113573199384,21cyclictest3467602-21systemd-journal07:14:591
1135731993826,9cyclictest23-21ksoftirqd/109:40:201
1135731993826,7cyclictest23-21ksoftirqd/107:45:021
1135731993826,7cyclictest1141941-21diskmemload12:10:031
1135731993825,10cyclictest23-21ksoftirqd/110:16:441
113573799373,5cyclictest1206066-21latency_hist11:25:033
1135737993728,7cyclictest37-21ksoftirqd/310:50:013
1135737993727,8cyclictest37-21ksoftirqd/311:35:143
1135737993722,11cyclictest37-21ksoftirqd/311:29:443
1135737993711,5cyclictest131rcu_preempt09:46:423
113573199374,21cyclictest1136891-21grep07:30:011
113573199372,3cyclictest131rcu_preempt11:26:171
113573199371,32cyclictest23-21ksoftirqd/107:20:481
1135737993626,7cyclictest1141655-21sshd09:01:593
1135737993626,7cyclictest1141655-21sshd09:01:593
1135737993618,11cyclictest275-21dbus-daemon08:10:023
1135731993620,12cyclictest275-21dbus-daemon08:35:001
1135737993526,5cyclictest309-21in:imuxsock07:22:593
1135737993525,5cyclictest309-21in:imuxsock07:45:013
1135737993524,9cyclictest37-21ksoftirqd/308:55:013
1135737993520,4cyclictest1135478-1kworker/3:0H07:10:033
1135737993423,6cyclictest1138217-21grep07:55:023
113573199335,21cyclictest1139885-21latency_hist08:30:021
113573199335,21cyclictest1139885-21latency_hist08:30:011
113573199334,4cyclictest131rcu_preempt07:35:021
113573199333,5cyclictest131rcu_preempt07:50:001
1135731993321,7cyclictest1139749-21latency_hist08:25:011
113573799323,3cyclictest131rcu_preempt07:14:593
1135737993221,9cyclictest37-21ksoftirqd/308:35:013
1135737993218,10cyclictest309-21in:imuxsock08:40:023
113573199314,3cyclictest131rcu_preempt08:45:591
113573799301,3cyclictest131rcu_preempt08:15:023
1135737992923,4cyclictest1138567-21latency_hist08:05:013
1135737992921,5cyclictest309-21in:imuxsock07:30:593
113573799291,4cyclictest131rcu_preempt07:48:583
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional