You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 06:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu May 28, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
80517499118113,3cyclictest0-21swapper/023:15:030
80518499834,25cyclictest131rcu_preempt20:10:013
805184996814,9cyclictest37-21ksoftirqd/319:55:003
805177996820,44cyclictest873710-21ssh23:29:061
805177996739,24cyclictest856050-21munin-plugin-st22:49:591
805177996722,41cyclictest3467562-21sshd21:32:201
80517799672,15cyclictest843975-21ssh22:22:501
80517799672,15cyclictest843975-21ssh22:22:491
80517799672,13cyclictest902167-21ssh00:32:121
80517799672,13cyclictest851457-21ssh22:39:471
80517799671,13cyclictest807270-21cat19:50:021
80517799671,13cyclictest807270-21cat19:50:011
805177996623,38cyclictest309-21in:imuxsock21:20:001
80517799662,18cyclictest888224-21ssh00:01:071
805177996621,40cyclictest23-21ksoftirqd/100:18:051
805177996619,43cyclictest23-21ksoftirqd/122:33:301
80517799661,15cyclictest856981-21ssh22:51:451
80517799661,15cyclictest856981-21ssh22:51:441
80517799661,14cyclictest839045-21sshd22:11:421
80517799661,14cyclictest835218-21ssh22:03:201
80518199659,35cyclictest30-21ksoftirqd/222:40:022
805177996532,29cyclictest23-21ksoftirqd/122:17:001
80517799652,15cyclictest838176-21munin-run22:10:021
80517799652,14cyclictest868481-21ssh23:17:051
80517799652,14cyclictest845364-21ssh22:25:391
80517799651,16cyclictest828145-21sshd21:47:221
80517799651,15cyclictest887071-21ssh23:58:461
805181996457,5cyclictest30-21ksoftirqd/221:30:032
805177996426,34cyclictest869978-21sh23:20:141
80517799642,15cyclictest900645-21ssh00:28:521
80517799642,15cyclictest806934-21munin-run19:45:011
80517799642,13cyclictest879846-21ssh23:42:371
80517799642,13cyclictest863587-21ssh23:06:081
80517799642,13cyclictest852124-21ssh22:40:461
80517799642,13cyclictest819872-21ssh21:28:581
80517799641,16cyclictest858537-21ssh22:55:031
80517799641,15cyclictest897003-21ssh00:20:321
80517799641,15cyclictest897003-21ssh00:20:311
80517799641,15cyclictest830490-21ssh21:52:451
80517799641,14cyclictest884776-21ssh23:53:471
80517799641,14cyclictest884776-21ssh23:53:471
80517799641,13cyclictest904237-21ssh00:36:461
80517499644,3cyclictest131rcu_preempt21:30:330
805184996324,11cyclictest37-21ksoftirqd/323:27:523
805177996327,31cyclictest811806-21sh21:10:461
80517799632,14cyclictest861295-21ssh23:01:111
80517799632,14cyclictest823773-21ssh21:37:401
80517799632,13cyclictest891434-21ssh00:08:261
80517799632,13cyclictest876845-21ssh23:35:431
80517799632,13cyclictest364-21snmpd23:45:031
80517799631,15cyclictest816699-21ssh21:21:381
80517799631,14cyclictest867221-21ssh23:14:391
80517799631,14cyclictest825892-21ssh21:42:211
80517799623,17cyclictest807576-21grep19:55:021
80517799622,14cyclictest809517-21munin-run20:35:001
80517799622,13cyclictest874818-21ssh23:31:191
80517799622,13cyclictest874818-21ssh23:31:191
80517799622,13cyclictest805663-21munin-run19:20:011
80517799613,13cyclictest807982-21munin-run20:05:021
80517799613,13cyclictest807982-21munin-run20:05:011
80517799612,13cyclictest831795-21ssh21:55:261
80517499613,51cyclictest887536-40mandb00:00:000
805174996114,14cyclictest275-21dbus-daemon00:20:010
805181996010,6cyclictest131rcu_preempt00:10:582
80517799602,17cyclictest806696-21munin-run19:40:011
80517799602,13cyclictest811096-21/usr/sbin/munin21:05:011
805177996015,39cyclictest0-21swapper/120:20:011
80517799601,14cyclictest810766-21sshd20:55:221
80517799601,13cyclictest893605-21ssh00:13:181
805177996010,4cyclictest0-21swapper/119:29:221
80517499606,46cyclictest892134-21ssh00:10:010
805174996030,26cyclictest16950irq/224-800000022:40:020
80518199593,10cyclictest30-21ksoftirqd/221:40:002
80517499593,46cyclictest3467602-21systemd-journal23:05:000
805174995851,4cyclictest16950irq/224-800000022:15:000
80518499578,8cyclictest37-21ksoftirqd/322:49:423
80518499574,13cyclictest37-21ksoftirqd/322:15:163
805184995717,11cyclictest37-21ksoftirqd/322:24:553
805184995717,11cyclictest37-21ksoftirqd/322:24:543
80518199571,9cyclictest30-21ksoftirqd/221:20:022
80517799575,16cyclictest809155-21sort20:25:021
80517799571,15cyclictest805461-21wc19:15:021
805174995729,13cyclictest860184-21ssh22:59:040
805174995728,13cyclictest855368-21ssh22:48:140
805174995711,41cyclictest1-21systemd00:05:010
805184995610,9cyclictest37-21ksoftirqd/323:46:583
80518199563,12cyclictest30-21ksoftirqd/223:39:452
805181995611,5cyclictest131rcu_preempt23:06:252
80517799562,14cyclictest364-21snmpd20:28:071
80517799562,14cyclictest364-21snmpd20:28:061
805174995649,4cyclictest16950irq/224-800000023:40:010
805174995625,15cyclictest878918-21ssh23:40:180
805184995524,10cyclictest37-21ksoftirqd/321:14:423
805184995519,9cyclictest131rcu_preempt20:15:033
805184995510,12cyclictest37-21ksoftirqd/322:40:023
80517799551,13cyclictest811317-21sshd21:09:201
80517799551,12cyclictest364-21snmpd20:47:161
805174995524,15cyclictest836169-21ssh22:05:070
805174995517,32cyclictest0-21swapper/021:50:010
80518499545,8cyclictest37-21ksoftirqd/300:01:133
80518499543,9cyclictest37-21ksoftirqd/323:02:363
80518199548,5cyclictest131rcu_preempt00:20:002
805181995412,6cyclictest131rcu_preempt23:20:522
805181995412,5cyclictest131rcu_preempt22:29:412
805181995412,5cyclictest131rcu_preempt00:03:052
805181995411,5cyclictest131rcu_preempt21:58:172
805177995417,33cyclictest808199-21sshd20:08:201
80517499543,46cyclictest833640-21grep22:00:010
805174995430,9cyclictest905367-21ssh00:39:310
805174995428,11cyclictest848727-21ssh22:33:280
80517499542,37cyclictest869567-21ssh23:19:530
805184995312,11cyclictest37-21ksoftirqd/323:45:023
80518199538,12cyclictest30-21ksoftirqd/221:35:022
805181995346,4cyclictest811109-21kworker/2:321:11:372
805181995343,7cyclictest291rcuc/220:40:012
805181995335,13cyclictest3467602-21systemd-journal22:20:012
80518199532,12cyclictest30-21ksoftirqd/219:40:012
805181995316,6cyclictest131rcu_preempt21:42:092
805181995311,6cyclictest131rcu_preempt23:52:492
805181995311,6cyclictest131rcu_preempt23:52:482
805181995311,5cyclictest131rcu_preempt21:51:392
805181995311,5cyclictest131rcu_preempt00:30:272
805181995310,5cyclictest131rcu_preempt22:31:552
805181995310,5cyclictest131rcu_preempt21:20:292
80517799532,11cyclictest3467602-21systemd-journal20:00:001
805177995317,10cyclictest23-21ksoftirqd/120:55:021
805174995310,5cyclictest131rcu_preempt23:09:240
80518499529,11cyclictest37-21ksoftirqd/322:51:483
80518499529,11cyclictest37-21ksoftirqd/322:51:473
80518499527,6cyclictest37-21ksoftirqd/300:08:523
80518499526,6cyclictest37-21ksoftirqd/320:20:033
80518499526,6cyclictest37-21ksoftirqd/319:21:163
80518499526,6cyclictest37-21ksoftirqd/319:21:153
80518499525,12cyclictest37-21ksoftirqd/323:56:023
805184995214,6cyclictest37-21ksoftirqd/300:20:353
805184995214,6cyclictest37-21ksoftirqd/300:20:343
80518199527,10cyclictest30-21ksoftirqd/200:39:512
80518199526,5cyclictest131rcu_preempt21:48:552
80518199525,8cyclictest30-21ksoftirqd/222:00:032
805181995244,5cyclictest865144-21kworker/2:023:46:432
805181995213,6cyclictest131rcu_preempt23:30:502
805181995213,6cyclictest131rcu_preempt23:30:502
805181995213,5cyclictest131rcu_preempt23:04:282
805181995213,5cyclictest131rcu_preempt22:13:552
805181995211,6cyclictest131rcu_preempt23:12:272
80517499524,44cyclictest805921-21latency_hist19:25:010
80517499524,44cyclictest805921-21latency_hist19:25:010
80518499516,9cyclictest37-21ksoftirqd/322:34:043
80518499515,8cyclictest37-21ksoftirqd/323:30:163
80518499515,8cyclictest37-21ksoftirqd/323:30:163
80518499515,7cyclictest37-21ksoftirqd/322:05:233
80518499513,12cyclictest37-21ksoftirqd/322:12:403
80518499512,11cyclictest37-21ksoftirqd/322:02:453
805184995115,7cyclictest37-21ksoftirqd/323:25:013
80518199518,7cyclictest30-21ksoftirqd/222:40:372
805181995143,5cyclictest840379-21kworker/2:100:06:232
805181995143,5cyclictest824751-21kworker/2:222:07:522
805181995143,5cyclictest808773-21kworker/2:120:50:012
80518199514,10cyclictest30-21ksoftirqd/223:26:042
80517799514,9cyclictest810070-21wc20:45:011
805174995136,12cyclictest16950irq/224-800000022:20:000
805174995127,18cyclictest3467602-21systemd-journal00:25:010
805174995127,18cyclictest3467602-21systemd-journal00:25:000
805174995112,5cyclictest131rcu_preempt20:15:210
80518499508,9cyclictest37-21ksoftirqd/323:10:343
80518499507,9cyclictest37-21ksoftirqd/321:44:443
80518499507,6cyclictest37-21ksoftirqd/321:20:103
80518499506,7cyclictest37-21ksoftirqd/323:35:033
80518499506,7cyclictest37-21ksoftirqd/321:31:573
80518499506,13cyclictest37-21ksoftirqd/323:17:573
80518499506,10cyclictest37-21ksoftirqd/323:07:103
80518499505,8cyclictest37-21ksoftirqd/321:18:213
80518499504,9cyclictest37-21ksoftirqd/321:54:303
805184995012,7cyclictest37-21ksoftirqd/319:45:033
80518499501,11cyclictest37-21ksoftirqd/321:39:033
80518199509,5cyclictest131rcu_preempt22:23:552
80518199509,5cyclictest131rcu_preempt22:23:542
80518199504,8cyclictest30-21ksoftirqd/222:57:262
805181995041,5cyclictest797879-21kworker/2:119:27:042
805181995040,6cyclictest865144-21kworker/2:000:27:082
805181995040,6cyclictest30-21ksoftirqd/223:16:012
805181995016,6cyclictest131rcu_preempt22:46:272
805181995016,5cyclictest131rcu_preempt23:41:112
805181995010,6cyclictest131rcu_preempt23:57:082
80517799502,13cyclictest805901-21munin-run19:25:011
80517799502,13cyclictest805901-21munin-run19:25:001
805174995011,6cyclictest131rcu_preempt21:00:000
80518499494,8cyclictest37-21ksoftirqd/321:25:583
80518499493,11cyclictest37-21ksoftirqd/319:50:013
80518499493,11cyclictest37-21ksoftirqd/319:50:013
805184994915,7cyclictest37-21ksoftirqd/300:36:423
805184994915,10cyclictest37-21ksoftirqd/300:26:223
805184994910,8cyclictest37-21ksoftirqd/323:51:283
805184994910,8cyclictest37-21ksoftirqd/323:51:273
80518199499,6cyclictest131rcu_preempt22:52:222
80518199499,6cyclictest131rcu_preempt22:52:212
80518199496,7cyclictest30-21ksoftirqd/200:21:532
80518199496,7cyclictest30-21ksoftirqd/200:21:522
805181994941,5cyclictest804557-21kworker/2:319:53:462
805181994941,5cyclictest797879-21kworker/2:119:23:032
805181994941,5cyclictest797879-21kworker/2:119:23:022
80517499496,6cyclictest131rcu_preempt21:23:590
80517499493,5cyclictest131rcu_preempt23:27:150
805174994912,34cyclictest12-21ksoftirqd/022:04:560
80518499486,6cyclictest37-21ksoftirqd/320:00:013
805184994841,5cyclictest37-21ksoftirqd/321:45:213
80518499484,13cyclictest37-21ksoftirqd/322:40:563
805184994837,5cyclictest860609-21cron23:00:003
805184994812,7cyclictest37-21ksoftirqd/322:29:593
805181994841,4cyclictest808773-21kworker/2:120:34:322
805181994841,4cyclictest807994-21kworker/2:020:15:482
805181994840,5cyclictest808773-21kworker/2:120:29:162
805181994840,5cyclictest808773-21kworker/2:120:29:162
805181994840,5cyclictest804557-21kworker/2:319:57:162
805181994840,5cyclictest804557-21kworker/2:319:40:532
805181994839,6cyclictest811109-21kworker/2:321:06:542
805181994838,6cyclictest807994-21kworker/2:020:55:562
80517799482,12cyclictest806404-21munin-run19:35:011
80517499483,7cyclictest826968-21cat21:45:020
805174994828,17cyclictest3467602-21systemd-journal22:55:000
805174994828,17cyclictest3467602-21systemd-journal22:55:000
805174994826,7cyclictest3467602-21systemd-journal21:38:040
805174994826,18cyclictest813523-21unin-run21:15:000
805174994822,11cyclictest819176-21ssh21:27:130
80518499475,6cyclictest37-21ksoftirqd/300:19:333
80518499474,5cyclictest37-21ksoftirqd/300:11:263
80518499473,12cyclictest37-21ksoftirqd/320:45:013
80518499473,10cyclictest37-21ksoftirqd/320:25:013
805181994740,4cyclictest804557-21kworker/2:320:00:222
805181994740,4cyclictest804557-21kworker/2:320:00:222
805181994739,5cyclictest808773-21kworker/2:120:51:172
805181994739,5cyclictest808773-21kworker/2:120:41:282
805181994739,5cyclictest807994-21kworker/2:021:03:012
805181994739,5cyclictest807994-21kworker/2:020:22:452
805181994739,5cyclictest804557-21kworker/2:320:10:032
805181994739,5cyclictest804557-21kworker/2:320:06:132
805181994739,5cyclictest804557-21kworker/2:319:33:082
805181994739,5cyclictest804557-21kworker/2:319:10:222
805181994738,6cyclictest804557-21kworker/2:319:49:082
805181994738,6cyclictest804557-21kworker/2:319:49:072
80517799472,13cyclictest808625-21latency_hist20:15:021
805174994736,8cyclictest16950irq/224-800000022:45:010
805174994729,15cyclictest893106-21ssh00:12:020
805174994728,5cyclictest12-21ksoftirqd/023:48:490
80517499472,41cyclictest903313-21ssh00:35:010
80518499464,6cyclictest37-21ksoftirqd/320:30:023
80518499464,6cyclictest37-21ksoftirqd/320:30:013
805184994636,8cyclictest37-21ksoftirqd/300:33:183
805181994638,5cyclictest797879-21kworker/2:119:18:472
805174994638,4cyclictest807237-21latency_hist19:50:010
805174994638,4cyclictest807237-21latency_hist19:50:010
805174994528,14cyclictest814318-21rm21:16:280
805174994517,24cyclictest12-21ksoftirqd/022:21:210
805174994517,24cyclictest12-21ksoftirqd/022:21:200
80518499449,9cyclictest37-21ksoftirqd/321:56:433
80518499447,8cyclictest37-21ksoftirqd/320:35:003
80517499445,5cyclictest12-21ksoftirqd/000:28:170
805174994437,4cyclictest16950irq/224-800000021:51:140
805174994434,7cyclictest12-21ksoftirqd/019:40:020
805174994430,10cyclictest275-21dbus-daemon19:55:000
80517499442,14cyclictest884677-21ssh23:53:260
80517499442,14cyclictest884677-21ssh23:53:250
80517499442,13cyclictest810552-21munin-run20:55:010
805174994420,5cyclictest131rcu_preempt22:28:190
80518499435,10cyclictest37-21ksoftirqd/319:35:003
80518499434,9cyclictest37-21ksoftirqd/319:10:033
80518499432,10cyclictest809774-21latency_hist20:40:023
80518499431,10cyclictest37-21ksoftirqd/319:40:003
805181994334,7cyclictest0-21swapper/219:10:012
80517499433,5cyclictest0-21swapper/020:34:590
805174994329,11cyclictest12-21ksoftirqd/019:35:000
80517499432,6cyclictest807698-21grep20:00:000
80517499432,13cyclictest364-21snmpd23:31:280
80517499432,13cyclictest364-21snmpd23:31:280
80517799423,8cyclictest809755-21cron20:40:001
805174994235,4cyclictest16950irq/224-800000020:50:000
805174994224,14cyclictest1-21systemd21:10:010
80518499418,9cyclictest0-21swapper/321:07:303
80518499416,8cyclictest37-21ksoftirqd/319:20:003
805184994133,5cyclictest37-21ksoftirqd/321:00:003
805174994128,9cyclictest275-21dbus-daemon20:40:020
805174994126,11cyclictest807945-21sshd20:02:220
805174994126,11cyclictest807945-21sshd20:02:210
80517499411,36cyclictest870019-21ssh23:20:200
80518499405,8cyclictest37-21ksoftirqd/320:55:023
805174994033,4cyclictest16950irq/224-800000019:16:150
805174994028,9cyclictest12-21ksoftirqd/019:30:030
805174994011,6cyclictest131rcu_preempt19:45:010
805184993921,12cyclictest3467602-21systemd-journal21:05:003
805174993926,8cyclictest309-21in:imuxsock20:30:020
805174993926,8cyclictest309-21in:imuxsock20:30:010
805174993917,16cyclictest275-21dbus-daemon21:05:020
805174993826,9cyclictest12-21ksoftirqd/020:25:010
805174993825,8cyclictest805363-21cut19:15:000
805174993724,9cyclictest12-21ksoftirqd/020:10:010
80518499365,5cyclictest0-21swapper/319:26:173
80518499361,7cyclictest0-21swapper/320:01:093
80518499361,7cyclictest0-21swapper/320:01:093
805174993623,10cyclictest364-21snmpd20:40:590
805184993520,11cyclictest3467602-21systemd-journal19:13:213
805174993519,14cyclictest12-21ksoftirqd/020:15:010
80517499321,27cyclictest0-21swapper/019:10:020
805184993016,10cyclictest810280-21sshd20:48:203
805177992721,4cyclictest805301-21latency_hist19:10:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional