You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-19 - 20:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Apr 19, 2026 12:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
995922991122,107cyclictest0-21swapper/009:55:010
99592299109105,2cyclictest0-21swapper/007:50:020
995922998840,43cyclictest1055912-21munin-run11:15:000
995922998825,20cyclictest1081592-21latency_hist12:15:010
995922998636,33cyclictest1035230-21ssh10:26:510
995922998333,15cyclictest1090207-21latency_hist12:34:590
99592299821,2cyclictest0-21swapper/010:20:030
995922998032,10cyclictest1058025-21modprobe11:20:010
995922997731,41cyclictest1008115-21ssh09:24:040
995922997723,17cyclictest1041084-21ssh10:40:110
995922997722,49cyclictest1051112-21ssh11:03:510
995922997540,31cyclictest1070154-21sshd11:48:030
995922997525,16cyclictest1029463-21ssh10:13:390
995925997424,45cyclictest1-21systemd09:50:011
995925997424,45cyclictest1-21systemd09:50:001
995922997320,17cyclictest996034-21sed07:10:020
995922997312,30cyclictest1000132-21latency_hist08:30:030
99592599724,64cyclictest1047386-21sh10:55:021
995925997226,41cyclictest275-21dbus-daemon10:05:021
995922997128,15cyclictest1004721-21ssh09:15:500
995922997021,42cyclictest1061268-21ssh11:27:130
995925996924,40cyclictest3467602-21systemd-journal10:30:011
995922996927,38cyclictest1086148-21date12:25:020
995922996921,15cyclictest1077963-21ssh12:06:090
995922996864,2cyclictest0-21swapper/011:50:020
995922996831,34cyclictest1040808-21ssh10:40:010
995922996829,34cyclictest1-21systemd09:29:020
995922996820,15cyclictest1083186-21ssh12:18:420
995925996624,38cyclictest1049126-21sh10:59:261
995922996616,14cyclictest1-21systemd08:00:010
995925996424,36cyclictest23-21ksoftirqd/109:12:361
995922996420,11cyclictest364-21snmpd10:50:020
995922996413,14cyclictest1049028-21ssh10:59:120
995925996323,35cyclictest1084978-21sh12:22:391
995922996325,34cyclictest150-21pr/ttyS208:33:560
995922996320,39cyclictest0-21swapper/010:35:010
99593099627,25cyclictest131rcu_preempt08:15:022
99592599622,19cyclictest1008531-21grep09:25:021
995922996245,12cyclictest1000757-21sshd08:40:040
995922996241,4cyclictest998724-21munin-run08:05:010
995922996219,38cyclictest0-21swapper/012:00:280
99592599612,18cyclictest1030147-21grep10:15:021
99592599611,19cyclictest1062359-21latency_hist11:30:011
995922996143,14cyclictest1062975-21sshd11:31:020
995922996126,4cyclictest1031882-21ssh10:19:220
995922996122,33cyclictest16850irq/134-800000009:46:040
995922996122,33cyclictest16850irq/134-800000009:46:030
995922996122,33cyclictest1002397-21ssh09:10:310
995922996119,17cyclictest1090437-21latency_hist12:35:020
995922996119,17cyclictest1090437-21latency_hist12:35:010
995922996117,14cyclictest1045480-21ssh10:50:230
995925996030,26cyclictest274-21cron11:04:591
99592599602,17cyclictest1089448-21ssh12:33:021
995925996018,5cyclictest1057595-21sh11:18:581
99592299607,29cyclictest0-21swapper/009:31:040
99592599597,43cyclictest0-21swapper/108:10:021
995925995919,10cyclictest23-21ksoftirqd/109:27:191
995922995941,14cyclictest996073-21sshd07:13:030
995922995913,42cyclictest1058413-21ssh11:20:190
995930995850,5cyclictest1008518-21kworker/2:010:15:112
995925995829,25cyclictest3467602-21systemd-journal12:39:031
995925995829,25cyclictest3467602-21systemd-journal12:39:021
995925995829,25cyclictest23-21ksoftirqd/111:45:001
995925995826,28cyclictest23-21ksoftirqd/108:55:011
995925995822,8cyclictest1043563-21sshd10:46:021
995925995822,32cyclictest23-21ksoftirqd/109:53:191
99592599582,18cyclictest1032157-21modprobe10:20:011
995925995818,11cyclictest23-21ksoftirqd/111:14:291
995925995813,6cyclictest1042928-21modprobe10:45:001
99592599581,16cyclictest1086400-21systemd-tmpfile12:25:481
99592599581,16cyclictest1086400-21systemd-tmpfile12:25:471
995922995846,9cyclictest1066828-21sshd11:40:030
995922995844,11cyclictest1075220-21munin-run12:00:010
995922995842,13cyclictest150-21pr/ttyS209:35:390
99592599579,12cyclictest1053699-21munin-plugin-st11:10:011
99592599578,8cyclictest0-21swapper/108:45:001
995925995724,29cyclictest1039197-21sh10:36:021
995925995717,6cyclictest1073682-21sh11:56:101
995925995714,9cyclictest23-21ksoftirqd/112:04:111
995925995713,7cyclictest23-21ksoftirqd/109:58:521
995922995743,10cyclictest997439-21munin-run07:40:010
995922995742,11cyclictest1016293-21sshd09:43:030
995922995731,22cyclictest1-21systemd10:05:000
995930995648,5cyclictest1000291-21kworker/2:309:28:132
995930995647,6cyclictest1040804-21kworker/2:211:04:342
995925995625,28cyclictest23-21ksoftirqd/109:31:011
995925995624,29cyclictest1014949-21readlink09:40:011
99592599562,19cyclictest1026387-21ssh10:06:101
99592599562,15cyclictest1006344-21ssh09:20:011
995925995620,5cyclictest1066632-21latency_hist11:40:011
995925995618,9cyclictest1022769-1kworker/u9:110:30:431
995930995546,5cyclictest1032201-21kworker/2:110:49:562
995930995545,6cyclictest1000291-21kworker/2:309:21:022
995925995525,26cyclictest23-21ksoftirqd/111:33:221
995925995523,27cyclictest23-21ksoftirqd/108:36:031
995922995541,9cyclictest48-21kcompactd007:32:000
995922995535,16cyclictest1027270-21sh10:08:310
995922995530,20cyclictest275-21dbus-daemon09:00:000
995922995530,19cyclictest999641-21cat08:20:010
995922995529,22cyclictest996626-21sshd07:24:460
995922995526,13cyclictest1052594-21ssh11:07:070
995922995510,18cyclictest1066334-21ssh11:39:190
995930995446,5cyclictest983103-21kworker/2:107:15:012
995930995445,6cyclictest1060217-21kworker/2:011:38:362
995930995444,6cyclictest1000291-21kworker/2:310:22:432
995930995444,6cyclictest1000291-21kworker/2:309:34:052
99592599543,15cyclictest998438-21sendmail07:59:591
99592599542,14cyclictest3467602-21systemd-journal07:55:021
995925995415,8cyclictest1-21systemd07:15:021
99592599541,14cyclictest999729-21sshd08:23:031
995922995442,9cyclictest998145-21sshd07:50:030
995922995442,9cyclictest1086926-21sshd12:27:030
995922995442,9cyclictest1086926-21sshd12:27:030
995922995428,18cyclictest1002078-21modprobe09:10:010
995930995345,5cyclictest999780-21kworker/2:208:40:012
995930995345,5cyclictest1060217-21kworker/2:011:44:252
995930995345,5cyclictest1032201-21kworker/2:112:08:482
995930995345,5cyclictest1032201-21kworker/2:111:21:422
995930995345,5cyclictest1008518-21kworker/2:010:01:382
995930995344,6cyclictest1008518-21kworker/2:010:27:092
995930995344,5cyclictest1032201-21kworker/2:110:38:012
995930995343,6cyclictest1032201-21kworker/2:110:55:012
99592599534,19cyclictest998703-21modprobe08:05:011
995925995326,23cyclictest1016924-21sh09:44:321
995925995325,25cyclictest23-21ksoftirqd/111:52:011
99592599532,15cyclictest1034106-21ssh10:24:211
995925995318,4cyclictest1083185-21sh12:18:411
995932995236,12cyclictest3467602-21systemd-journal11:25:023
995930995244,5cyclictest999780-21kworker/2:209:16:582
995930995244,5cyclictest1060217-21kworker/2:011:32:042
995930995244,5cyclictest1032201-21kworker/2:112:12:342
995930995244,5cyclictest1032201-21kworker/2:111:27:332
995930995244,5cyclictest1008518-21kworker/2:010:06:582
995930995242,7cyclictest1032201-21kworker/2:112:15:282
995930995242,6cyclictest1032201-21kworker/2:111:59:422
995930995242,6cyclictest1000291-21kworker/2:309:45:342
995930995242,6cyclictest1000291-21kworker/2:309:45:342
995930995234,14cyclictest3467602-21systemd-journal10:35:002
995930995227,19cyclictest275-21dbus-daemon08:00:012
99592599523,11cyclictest996567-21cat07:20:021
99592599523,11cyclictest996567-21cat07:20:021
99592599522,16cyclictest1001825-21latency_hist09:05:001
995922995244,4cyclictest999754-21munin-run08:25:000
995922995243,5cyclictest1001013-21sshd08:46:030
995922995243,5cyclictest1001013-21sshd08:46:030
995922995236,11cyclictest999228-21munin-run08:15:020
995922995235,12cyclictest364-21snmpd08:37:480
995922995221,27cyclictest0-21swapper/009:55:180
995922995218,30cyclictest0-21swapper/007:43:030
995930995143,5cyclictest999780-21kworker/2:209:13:072
995930995143,5cyclictest983103-21kworker/2:108:10:022
995930995143,5cyclictest1060217-21kworker/2:012:36:352
995930995143,5cyclictest1060217-21kworker/2:012:36:342
995930995143,5cyclictest1032201-21kworker/2:110:57:192
995930995142,6cyclictest1060217-21kworker/2:012:28:302
995930995142,6cyclictest1060217-21kworker/2:012:28:292
995930995142,6cyclictest1008518-21kworker/2:010:11:532
995930995142,6cyclictest1000291-21kworker/2:309:50:402
995930995142,5cyclictest1040804-21kworker/2:211:13:532
995930995142,5cyclictest1040804-21kworker/2:211:06:052
995930995141,6cyclictest1060217-21kworker/2:012:34:352
995930995140,7cyclictest1032201-21kworker/2:112:03:142
995930995140,7cyclictest1032201-21kworker/2:110:42:492
99592599514,22cyclictest996702-21grep07:25:021
995925995118,8cyclictest23-21ksoftirqd/111:20:241
995922995144,4cyclictest996404-21munin-run07:20:020
995922995144,4cyclictest996404-21munin-run07:20:010
995922995137,10cyclictest364-21snmpd07:27:400
995932995046,2cyclictest0-21swapper/308:00:013
995932995024,12cyclictest275-21dbus-daemon10:20:013
995930995042,5cyclictest1040804-21kworker/2:211:16:272
995930995042,5cyclictest1032201-21kworker/2:112:20:082
995930995042,5cyclictest1032201-21kworker/2:111:50:492
995930995042,5cyclictest1000291-21kworker/2:309:57:112
995930995041,5cyclictest983103-21kworker/2:108:25:022
99592599503,15cyclictest1071010-21cat11:50:031
99592599503,13cyclictest996894-21munin-run07:30:001
995925995024,22cyclictest23-21ksoftirqd/108:59:281
995922995036,10cyclictest364-21snmpd08:07:380
995930994941,5cyclictest1000291-21kworker/2:309:36:532
995930994940,5cyclictest983103-21kworker/2:108:13:042
995925994922,24cyclictest23-21ksoftirqd/112:13:391
995932994831,11cyclictest1-21systemd11:50:023
99593299481,3cyclictest131rcu_preempt09:35:093
995930994840,5cyclictest1008518-21kworker/2:009:42:452
995930994839,6cyclictest1032201-21kworker/2:111:46:332
995930994839,6cyclictest1000291-21kworker/2:309:04:592
995930994833,10cyclictest3467602-21systemd-journal07:24:032
99592599483,16cyclictest999503-21sendmail-msp08:20:011
995925994815,8cyclictest23-21ksoftirqd/109:05:211
995925994815,7cyclictest999178-21sshd08:10:031
995922994835,10cyclictest364-21snmpd08:50:300
995932994734,10cyclictest1075158-21cron12:00:003
99593299471,3cyclictest131rcu_preempt12:40:013
99593299471,3cyclictest131rcu_preempt12:40:013
995930994740,4cyclictest983103-21kworker/2:107:40:072
995930994739,5cyclictest983103-21kworker/2:107:26:142
995930994738,6cyclictest1000291-21kworker/2:309:06:182
995930994731,11cyclictest1-21systemd07:40:012
99592599472,17cyclictest1078054-21ssh12:06:231
99592599472,16cyclictest1001059-21munin-run08:50:021
99592599472,16cyclictest1001059-21munin-run08:50:011
99592599471,17cyclictest997525-21cat07:40:031
995932994631,12cyclictest37-21ksoftirqd/310:22:043
99593299461,4cyclictest131rcu_preempt10:39:383
99593099469,26cyclictest995641-21kworker/u8:107:10:012
995930994639,4cyclictest999780-21kworker/2:208:40:042
995930994638,5cyclictest999780-21kworker/2:208:46:202
995930994638,5cyclictest999780-21kworker/2:208:46:192
995930994638,5cyclictest999780-21kworker/2:208:28:012
995930994638,5cyclictest983103-21kworker/2:107:16:542
995930994638,5cyclictest983103-21kworker/2:107:16:542
995930994638,5cyclictest1000291-21kworker/2:308:57:152
995930994638,5cyclictest1000291-21kworker/2:308:50:132
995930994633,9cyclictest3467602-21systemd-journal07:50:022
995922994632,11cyclictest3476162-21ntpd09:00:110
995932994530,11cyclictest3467602-21systemd-journal12:34:593
995932994529,12cyclictest37-21ksoftirqd/310:30:033
99593299452,3cyclictest131rcu_preempt09:32:133
995930994539,3cyclictest983103-21kworker/2:108:00:452
995930994538,4cyclictest983103-21kworker/2:107:30:302
995930994537,5cyclictest999780-21kworker/2:208:30:402
99593299448,6cyclictest131rcu_preempt11:17:173
99593299444,3cyclictest131rcu_preempt12:07:003
995932994435,6cyclictest37-21ksoftirqd/308:24:563
995932994434,7cyclictest37-21ksoftirqd/307:35:013
995932994425,13cyclictest275-21dbus-daemon07:55:013
995925994416,9cyclictest23-21ksoftirqd/107:50:011
995932994332,7cyclictest37-21ksoftirqd/309:55:003
995932994327,12cyclictest275-21dbus-daemon12:25:003
995932994324,13cyclictest37-21ksoftirqd/307:30:013
995932994317,6cyclictest0-21swapper/311:39:043
995930994336,4cyclictest983103-21kworker/2:107:49:132
995932994229,8cyclictest1086401-21sh12:25:483
995932994229,8cyclictest1086401-21sh12:25:473
995932994226,12cyclictest275-21dbus-daemon09:45:003
995932994226,12cyclictest275-21dbus-daemon09:05:003
99593299414,3cyclictest131rcu_preempt11:50:543
995932994130,8cyclictest1059061-21sshd11:22:033
995932994128,9cyclictest1064458-21cron11:35:003
995932994128,10cyclictest37-21ksoftirqd/311:09:033
995932994126,12cyclictest1000531-21tr08:40:003
99593299411,3cyclictest131rcu_preempt10:10:303
99593299411,3cyclictest131rcu_preempt10:04:433
99593299404,4cyclictest131rcu_preempt10:09:073
99593299404,3cyclictest131rcu_preempt08:50:023
99593299404,3cyclictest131rcu_preempt08:50:023
995932994032,5cyclictest1002268-21latency_hist09:10:033
995932994030,5cyclictest309-21in:imuxsock08:35:003
995932994028,9cyclictest37-21ksoftirqd/309:16:443
995932994024,11cyclictest37-21ksoftirqd/310:56:563
99593299402,3cyclictest131rcu_preempt12:10:463
995932994023,13cyclictest1051571-21ssh11:05:013
995932994017,5cyclictest37-21ksoftirqd/312:00:513
99593299401,3cyclictest131rcu_preempt09:14:383
995932994011,6cyclictest131rcu_preempt10:53:213
995925994016,7cyclictest23-21ksoftirqd/108:29:541
995925994012,8cyclictest23-21ksoftirqd/107:45:011
995925994011,4cyclictest0-21swapper/107:35:021
995932993929,8cyclictest37-21ksoftirqd/312:15:083
99593299391,3cyclictest131rcu_preempt09:20:213
99593299391,13cyclictest1054505-21rm11:11:343
99593299386,8cyclictest37-21ksoftirqd/310:40:213
995932993829,6cyclictest37-21ksoftirqd/311:40:043
995932993827,8cyclictest1018491-21sshd09:48:043
995932993827,8cyclictest1018491-21sshd09:48:033
995932993824,11cyclictest37-21ksoftirqd/310:31:463
995932993824,11cyclictest37-21ksoftirqd/309:56:373
995932993823,12cyclictest37-21ksoftirqd/310:48:263
995932993724,11cyclictest37-21ksoftirqd/309:29:513
995932993724,10cyclictest37-21ksoftirqd/308:56:123
995932993720,13cyclictest275-21dbus-daemon08:15:013
995932993627,7cyclictest37-21ksoftirqd/308:55:023
995932993624,9cyclictest37-21ksoftirqd/307:24:023
99593299362,3cyclictest131rcu_preempt08:10:003
995932993612,4cyclictest131rcu_preempt07:19:273
995932993612,4cyclictest131rcu_preempt07:19:273
99592599365,26cyclictest1000400-21grep08:35:021
995932993523,9cyclictest37-21ksoftirqd/308:45:033
995932993424,8cyclictest37-21ksoftirqd/308:30:023
995932993423,9cyclictest37-21ksoftirqd/307:10:013
995932993323,7cyclictest37-21ksoftirqd/307:45:003
995932993321,10cyclictest37-21ksoftirqd/307:15:013
995932993221,9cyclictest37-21ksoftirqd/308:04:513
995932993213,14cyclictest275-21dbus-daemon07:50:013
995932993213,13cyclictest3467602-21systemd-journal07:40:003
995932993119,10cyclictest37-21ksoftirqd/308:20:013
99592599242,3cyclictest131rcu_preempt07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional