You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 21:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Mon Mar 23, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
115743999111105,3cyclictest0-21swapper/008:50:020
1157449997216,9cyclictest37-21ksoftirqd/310:30:013
1157444997126,14cyclictest1235490-21sh11:56:571
1157439997163,6cyclictest16950irq/224-800000011:05:020
1157444997019,18cyclictest1233695-21ssh11:52:551
1157444997018,15cyclictest1186381-21ssh10:02:381
1157444997018,15cyclictest1186381-21ssh10:02:381
1157444996923,16cyclictest1195216-21sh10:23:101
1157444996919,14cyclictest1175704-21ssh09:37:501
1157444996918,19cyclictest1230264-21rm11:44:551
1157444996918,15cyclictest1243570-21ssh12:15:261
1157444996828,15cyclictest1241348-21sshd12:10:141
1157444996816,16cyclictest1191125-21ssh10:13:501
1157444996725,7cyclictest1207395-21ssh10:51:191
1157444996622,6cyclictest1247119-21ssh12:24:071
1157444996612,21cyclictest1216345-21ssh11:12:181
115743999669,40cyclictest12-21ksoftirqd/012:03:570
115743999669,40cyclictest12-21ksoftirqd/012:03:570
1157439996631,30cyclictest12-21ksoftirqd/009:54:490
1157439996631,30cyclictest12-21ksoftirqd/009:54:490
1157444996518,16cyclictest1250121-21ssh12:30:411
1157444996516,15cyclictest1225718-21ssh11:34:231
1157444996515,14cyclictest1197285-21ssh10:27:501
1157439996525,36cyclictest12-21ksoftirqd/010:47:430
1157439996525,36cyclictest12-21ksoftirqd/010:47:420
1157449996422,8cyclictest37-21ksoftirqd/311:24:273
1157444996417,14cyclictest1226329-21ssh11:35:181
1157439996431,30cyclictest12-21ksoftirqd/012:05:590
1157444996323,14cyclictest1214472-21ssh11:07:571
1157444996316,15cyclictest1199998-21rm10:34:291
1157444996316,12cyclictest1240376-21ssh12:08:201
1157444996315,16cyclictest1159202-21latency_hist07:45:021
1157444996215,13cyclictest1167336-21ssh09:18:321
1157439996227,9cyclictest12-21ksoftirqd/012:28:180
1157444996122,13cyclictest1188164-21ssh10:06:411
1157444996120,9cyclictest1203573-21ssh10:42:391
1157444996117,14cyclictest1221189-21ssh11:23:421
1157444996116,15cyclictest1253579-21ssh12:39:121
1157439996116,41cyclictest12-21ksoftirqd/010:20:250
1157444996017,8cyclictest1230938-21ssh11:46:071
1157444996016,13cyclictest1183883-21ssh09:56:451
1157444996014,12cyclictest1238261-21ssh12:03:351
1157444996014,12cyclictest1238261-21ssh12:03:341
1157439996032,25cyclictest12-21ksoftirqd/010:11:150
1157439996015,10cyclictest12-21ksoftirqd/010:27:300
1157449995914,10cyclictest37-21ksoftirqd/311:34:503
115744699596,8cyclictest30-21ksoftirqd/212:16:422
115744699594,9cyclictest30-21ksoftirqd/210:06:072
1157439995929,15cyclictest1200754-21ssh10:35:550
1157439995911,21cyclictest12-21ksoftirqd/009:46:240
115744999585,15cyclictest1164227-21ssh09:11:103
115744699586,10cyclictest30-21ksoftirqd/212:22:242
1157449995713,10cyclictest37-21ksoftirqd/309:43:273
115744699575,8cyclictest30-21ksoftirqd/210:25:022
1157446995717,10cyclictest30-21ksoftirqd/211:00:022
1157446995717,10cyclictest30-21ksoftirqd/211:00:022
1157439995725,11cyclictest12-21ksoftirqd/011:35:080
115743999571,5cyclictest131rcu_preempt11:42:160
1157449995616,8cyclictest37-21ksoftirqd/311:52:473
1157449995614,9cyclictest37-21ksoftirqd/311:09:223
1157449995614,9cyclictest37-21ksoftirqd/310:15:263
1157449995612,9cyclictest37-21ksoftirqd/311:03:413
1157449995612,16cyclictest1183689-21ssh09:56:173
1157444995643,10cyclictest1201529-21ssh10:37:541
1157444995639,14cyclictest1210464-21ssh10:58:421
1157444995639,14cyclictest1210464-21ssh10:58:421
1157444995639,13cyclictest1212348-21ssh11:03:051
1157444995639,13cyclictest1165514-21ssh09:14:181
1157444995638,14cyclictest1173938-21ssh09:33:521
1157444995638,14cyclictest1169096-21ssh09:22:281
1157444995637,15cyclictest1217921-21ssh11:15:541
1157444995637,15cyclictest1217921-21ssh11:15:531
1157444995637,15cyclictest1163582-21sshd09:07:151
1157444995628,24cyclictest1206628-21modprobe10:50:001
1157444995628,24cyclictest1206628-21modprobe10:50:001
1157444995620,11cyclictest0-21swapper/108:21:481
1157439995631,8cyclictest1178870-21ssh09:45:020
1157439995627,14cyclictest1204032-21ssh10:43:450
115744999559,9cyclictest37-21ksoftirqd/309:17:353
115744999559,10cyclictest37-21ksoftirqd/312:30:593
115744699559,6cyclictest291rcuc/210:15:002
115744699556,6cyclictest30-21ksoftirqd/211:33:532
115744499553,6cyclictest131rcu_preempt09:30:021
1157444995536,15cyclictest3467602-21systemd-journal10:16:191
1157444995514,14cyclictest1157431-21cyclictest09:44:231
115743999551,39cyclictest1243100-21ssh12:14:510
115743999551,38cyclictest1183953-21ssh09:57:040
115743999551,25cyclictest12-21ksoftirqd/011:26:130
1157439995512,39cyclictest131rcu_preempt10:17:150
1157449995413,7cyclictest37-21ksoftirqd/310:56:563
1157449995413,7cyclictest37-21ksoftirqd/310:56:563
115744699547,9cyclictest30-21ksoftirqd/211:06:122
115744699547,10cyclictest30-21ksoftirqd/212:12:142
115744699545,8cyclictest30-21ksoftirqd/210:18:162
115744699544,9cyclictest30-21ksoftirqd/212:38:152
1157446995437,13cyclictest3467602-21systemd-journal11:30:002
1157444995439,11cyclictest1180087-21ssh09:48:101
1157444995437,13cyclictest1247952-21ssh12:25:401
1157444995437,13cyclictest1222514-21ssh11:26:361
1157444995436,16cyclictest1181433-21ssh09:50:561
1157444995436,16cyclictest1181433-21ssh09:50:561
115743999543,10cyclictest12-21ksoftirqd/009:22:370
115743999541,4cyclictest131rcu_preempt07:30:140
1157449995315,8cyclictest37-21ksoftirqd/310:50:413
115744699537,12cyclictest30-21ksoftirqd/211:04:502
115744699537,11cyclictest30-21ksoftirqd/211:48:262
1157444995339,10cyclictest1160475-21sshd08:09:151
1157444995316,13cyclictest1162411-21cat08:45:021
115743999536,6cyclictest12-21ksoftirqd/007:12:010
1157439995346,4cyclictest16950irq/224-800000009:35:010
1157439995342,8cyclictest12-21ksoftirqd/010:08:570
1157439995326,12cyclictest1199751-21ssh10:33:540
1157439995325,12cyclictest1217355-21ssh11:14:540
115743999531,43cyclictest12463331ssh12:22:070
1157449995213,7cyclictest37-21ksoftirqd/309:35:333
1157449995212,9cyclictest37-21ksoftirqd/311:39:473
115744699526,8cyclictest30-21ksoftirqd/209:44:012
115744699525,8cyclictest30-21ksoftirqd/209:56:032
115744699525,13cyclictest30-21ksoftirqd/209:34:492
1157446995215,8cyclictest30-21ksoftirqd/210:46:562
1157446995215,8cyclictest30-21ksoftirqd/210:46:562
1157444995241,8cyclictest1158385-21sshd07:25:151
1157444995239,10cyclictest1162807-21sshd08:54:151
1157444995239,10cyclictest1159423-21sshd07:45:271
1157444995237,11cyclictest1162155-21cat08:40:021
1157444995237,11cyclictest1157603-21sshd07:12:141
1157444995236,13cyclictest1161509-21sshd08:27:141
1157444995236,11cyclictest1161755-21sshd08:30:361
1157444995236,11cyclictest1161755-21sshd08:30:361
1157444995234,11cyclictest1163456-21grep09:05:011
115743999525,24cyclictest1163949-21kworker/u8:410:55:340
115743999525,24cyclictest1163949-21kworker/u8:410:55:340
1157439995225,23cyclictest151rcuc/010:53:150
1157439995215,17cyclictest16850irq/134-800000011:45:140
1157439995214,6cyclictest131rcu_preempt12:40:000
115743999521,42cyclictest12208111ssh11:22:360
1157449995122,7cyclictest37-21ksoftirqd/312:22:213
1157449995118,6cyclictest37-21ksoftirqd/309:22:503
1157449995111,9cyclictest37-21ksoftirqd/310:11:113
115744699517,6cyclictest131rcu_preempt11:50:022
115744699516,15cyclictest30-21ksoftirqd/211:55:102
115744699516,10cyclictest30-21ksoftirqd/209:10:082
115744699515,12cyclictest30-21ksoftirqd/212:00:102
115744699515,12cyclictest30-21ksoftirqd/212:00:092
115744699515,10cyclictest30-21ksoftirqd/209:24:022
115744699514,12cyclictest30-21ksoftirqd/212:32:222
115744699513,6cyclictest30-21ksoftirqd/212:05:222
115744699513,6cyclictest30-21ksoftirqd/211:14:102
115744699512,13cyclictest30-21ksoftirqd/211:20:492
1157446995111,8cyclictest30-21ksoftirqd/208:58:112
1157446995111,8cyclictest30-21ksoftirqd/208:58:112
1157446995110,5cyclictest131rcu_preempt11:40:522
1157444995138,10cyclictest1162545-21sshd08:47:131
1157439995122,25cyclictest1233475-21sh11:52:110
1157439995118,15cyclictest12-21ksoftirqd/012:17:010
1157439995116,8cyclictest12-21ksoftirqd/012:31:320
1157449995031,14cyclictest275-21dbus-daemon11:50:003
1157449995024,7cyclictest37-21ksoftirqd/310:34:423
1157449995021,22cyclictest1163615-21munin-run09:10:003
1157449995012,6cyclictest37-21ksoftirqd/312:05:223
1157449995010,9cyclictest37-21ksoftirqd/310:41:253
115744699507,14cyclictest30-21ksoftirqd/210:30:492
115744699506,8cyclictest30-21ksoftirqd/212:26:382
115744699505,8cyclictest30-21ksoftirqd/209:28:362
1157446995042,5cyclictest1144545-21kworker/2:308:31:502
1157446995042,5cyclictest1144545-21kworker/2:308:31:502
1157446995036,10cyclictest1-21systemd10:04:142
1157446995036,10cyclictest1-21systemd10:04:142
1157446995033,13cyclictest275-21dbus-daemon08:15:012
115744699503,11cyclictest30-21ksoftirqd/209:52:362
115744699503,11cyclictest30-21ksoftirqd/209:52:362
1157446995015,5cyclictest131rcu_preempt09:37:462
1157444995038,9cyclictest1158655-21sshd07:34:141
1157439995040,7cyclictest12-21ksoftirqd/009:10:010
1157439995030,16cyclictest275-21dbus-daemon09:30:010
1157439995023,12cyclictest1167352-21ssh09:18:340
1157439995017,30cyclictest16850irq/134-800000011:04:450
1157439995016,10cyclictest12-21ksoftirqd/008:02:140
115744999495,6cyclictest37-21ksoftirqd/312:38:063
115744999495,14cyclictest1162330-21sh08:45:013
115744999495,10cyclictest37-21ksoftirqd/310:08:013
115744999494,8cyclictest37-21ksoftirqd/311:10:233
115744699499,8cyclictest30-21ksoftirqd/210:29:032
115744699497,5cyclictest30-21ksoftirqd/211:36:072
115744699497,11cyclictest30-21ksoftirqd/210:44:272
115744699494,8cyclictest30-21ksoftirqd/207:49:292
1157446994942,4cyclictest1144545-21kworker/2:308:09:482
1157446994941,5cyclictest1161050-21kworker/2:209:19:222
115744699494,13cyclictest30-21ksoftirqd/210:38:072
1157444994922,22cyclictest1158351-21ls07:25:021
1157439994934,11cyclictest1163342-21cron09:05:000
115743999492,44cyclictest0-21swapper/008:25:010
1157449994812,7cyclictest37-21ksoftirqd/310:45:303
1157449994812,7cyclictest37-21ksoftirqd/310:45:303
1157446994841,4cyclictest1144545-21kworker/2:307:54:332
1157446994840,5cyclictest1144545-21kworker/2:308:40:072
1157446994840,5cyclictest1144545-21kworker/2:307:16:512
1157446994839,5cyclictest1162842-21kworker/2:109:07:062
115744699481,6cyclictest0-21swapper/211:18:242
115744699481,6cyclictest0-21swapper/211:18:232
1157446994811,5cyclictest131rcu_preempt10:51:182
1157444994834,11cyclictest1159931-21sshd07:56:141
1157444994828,16cyclictest1159672-21sshd07:50:141
1157439994819,15cyclictest12-21ksoftirqd/012:00:020
1157439994813,10cyclictest12-21ksoftirqd/009:10:150
115743999481,18cyclictest0-21swapper/011:30:200
1157449994713,7cyclictest0-21swapper/311:58:323
115744999471,15cyclictest1223245-21ssh11:28:303
1157449994711,5cyclictest0-21swapper/312:13:293
1157446994740,4cyclictest1161050-21kworker/2:208:45:162
1157446994740,4cyclictest1157397-21kworker/2:007:26:072
1157446994740,4cyclictest1144545-21kworker/2:308:01:062
1157446994739,5cyclictest1162842-21kworker/2:109:02:092
1157446994739,5cyclictest1157397-21kworker/2:007:20:522
1157446994739,5cyclictest1144545-21kworker/2:308:24:492
1157446994739,5cyclictest1144545-21kworker/2:307:56:492
1157446994739,5cyclictest1144545-21kworker/2:307:11:512
1157444994734,10cyclictest364-21snmpd07:19:441
1157444994723,17cyclictest1160992-21cron08:20:011
1157439994735,8cyclictest12-21ksoftirqd/007:45:000
1157439994728,16cyclictest12-21ksoftirqd/010:04:550
1157439994728,16cyclictest12-21ksoftirqd/010:04:550
1157449994631,11cyclictest1157851-21sshd07:16:143
1157449994612,8cyclictest0-21swapper/309:53:353
1157449994612,8cyclictest0-21swapper/309:53:353
1157449994610,7cyclictest37-21ksoftirqd/310:38:253
1157446994639,4cyclictest1144545-21kworker/2:308:53:542
1157446994639,4cyclictest1144545-21kworker/2:308:35:152
1157446994638,5cyclictest1157397-21kworker/2:007:40:302
1157446994638,5cyclictest1157397-21kworker/2:007:36:272
1157446994638,5cyclictest1144545-21kworker/2:308:29:292
1157446994636,6cyclictest291rcuc/209:46:092
1157444994639,4cyclictest364-21snmpd07:37:461
115743999466,7cyclictest12-21ksoftirqd/007:30:010
1157439994639,4cyclictest16950irq/224-800000011:16:160
1157439994639,4cyclictest16950irq/224-800000011:16:160
1157439994637,6cyclictest12-21ksoftirqd/009:38:380
115743999463,6cyclictest1162038-21modprobe08:40:000
115743999462,38cyclictest0-21swapper/007:20:010
1157439994619,19cyclictest12-21ksoftirqd/007:50:080
115744999459,7cyclictest37-21ksoftirqd/310:23:363
115744999456,7cyclictest1171266-21ssh09:27:273
115744999455,7cyclictest37-21ksoftirqd/311:19:223
115744999455,7cyclictest37-21ksoftirqd/311:19:223
1157449994530,12cyclictest37-21ksoftirqd/307:45:023
1157446994538,4cyclictest1160501-21kworker/2:108:15:422
1157446994537,5cyclictest1157397-21kworker/2:007:30:562
1157444994524,17cyclictest1-21systemd09:00:011
1157444994524,17cyclictest1-21systemd09:00:011
115743999451,11cyclictest1158961-21true07:40:000
115744999449,7cyclictest0-21swapper/311:44:133
115744999449,5cyclictest0-21swapper/312:18:353
1157449994417,11cyclictest0-21swapper/310:05:013
1157449994417,11cyclictest0-21swapper/310:05:013
1157449994413,10cyclictest1179010-1kworker/u9:009:47:133
1157444994432,9cyclictest364-21snmpd08:14:001
1157444994430,10cyclictest364-21snmpd08:01:361
1157439994429,12cyclictest1162295-21sshd08:42:140
1157449994326,14cyclictest1-21systemd08:05:013
1157439994324,13cyclictest12-21ksoftirqd/008:15:000
115744999426,7cyclictest37-21ksoftirqd/312:25:233
115744999426,7cyclictest37-21ksoftirqd/309:33:063
115744999423,7cyclictest37-21ksoftirqd/312:01:203
115744999423,7cyclictest37-21ksoftirqd/312:01:203
1157449994225,13cyclictest1-21systemd08:40:013
1157449994224,14cyclictest1163050-21sshd08:57:153
1157449994224,14cyclictest1163050-21sshd08:57:153
115743999428,5cyclictest131rcu_preempt08:52:440
1157449994130,6cyclictest309-21in:imuxsock08:25:003
1157449994126,11cyclictest275-21dbus-daemon08:10:013
1157449994126,11cyclictest1160958-21sshd08:15:143
115744999411,4cyclictest0-21swapper/309:04:103
1157439994125,12cyclictest12-21ksoftirqd/008:30:020
1157439994125,12cyclictest12-21ksoftirqd/008:30:020
1157449994026,11cyclictest1159427-21sshd07:46:143
1157449994025,12cyclictest1161499-21sshd08:25:143
1157449994024,12cyclictest1158397-21sshd07:28:143
1157439994013,7cyclictest131rcu_preempt07:23:130
1157449993924,11cyclictest1158147-21sshd07:23:143
115744999391,11cyclictest37-21ksoftirqd/307:55:003
1157439993933,3cyclictest16950irq/224-800000007:55:140
1157439993910,7cyclictest131rcu_preempt07:45:130
1157449993825,10cyclictest1158641-21sshd07:30:233
1157449993819,16cyclictest1160744-21/usr/sbin/munin08:15:013
115743999382,6cyclictest1161015-21munin-run08:20:010
1157439993822,12cyclictest12-21ksoftirqd/009:00:000
1157439993822,12cyclictest12-21ksoftirqd/009:00:000
1157449993724,9cyclictest1158908-21sshd07:39:143
1157449993722,11cyclictest3467602-21systemd-journal08:50:013
115744999372,13cyclictest1160078-21latency_hist08:00:023
1157449993720,13cyclictest275-21dbus-daemon08:55:013
115743999372,6cyclictest1161551-21munin-run08:30:010
115743999371,18cyclictest12-21ksoftirqd/008:09:260
1157449993623,9cyclictest48-21kcompactd008:33:163
1157449993623,9cyclictest48-21kcompactd008:33:153
1157444993518,14cyclictest1157492-21sed07:10:021
1157449993322,8cyclictest37-21ksoftirqd/307:10:023
1157449993118,10cyclictest37-21ksoftirqd/307:10:023
1157446993019,7cyclictest1157515-21cat07:10:022
1157439992919,7cyclictest12-21ksoftirqd/007:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional