You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 09:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Mar 22, 2026 00:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6970899912380,22cyclictest131rcu_preempt20:55:020
697089991201,3cyclictest131rcu_preempt19:45:010
69708999941,3cyclictest0-21swapper/019:15:020
697089998713,33cyclictest729306-21ssh22:10:020
69710499777,10cyclictest37-21ksoftirqd/322:20:013
69710199737,11cyclictest30-21ksoftirqd/200:00:032
69708999708,32cyclictest131rcu_preempt22:30:020
69710499696,8cyclictest37-21ksoftirqd/321:25:013
69710499696,11cyclictest37-21ksoftirqd/300:00:023
697101996712,6cyclictest131rcu_preempt23:10:012
697089996516,40cyclictest12-21ksoftirqd/021:55:180
697089996514,47cyclictest12-21ksoftirqd/022:03:010
69710499649,5cyclictest131rcu_preempt22:35:023
69710499649,5cyclictest131rcu_preempt22:35:013
697104996415,5cyclictest131rcu_preempt22:00:003
697101996410,6cyclictest131rcu_preempt19:10:022
697101996410,6cyclictest131rcu_preempt19:10:022
697089996320,34cyclictest131rcu_preempt00:16:260
69710499624,9cyclictest37-21ksoftirqd/322:39:593
697104996138,8cyclictest37-21ksoftirqd/320:05:013
697104996138,8cyclictest37-21ksoftirqd/320:05:013
69710199615,9cyclictest30-21ksoftirqd/200:20:012
697104996011,5cyclictest131rcu_preempt23:55:023
69710199606,5cyclictest131rcu_preempt22:35:012
69710199606,5cyclictest131rcu_preempt22:35:012
69708999602,45cyclictest735548-21kworker/0:122:59:440
697089996020,5cyclictest131rcu_preempt00:04:030
69710499599,6cyclictest131rcu_preempt21:15:023
69710499596,9cyclictest37-21ksoftirqd/321:05:013
69710499594,14cyclictest37-21ksoftirqd/300:35:003
69710499592,8cyclictest37-21ksoftirqd/300:40:003
697104995913,6cyclictest131rcu_preempt22:00:453
697104995911,5cyclictest131rcu_preempt23:30:003
69710499591,11cyclictest37-21ksoftirqd/300:25:003
697101995930,7cyclictest30-21ksoftirqd/222:20:432
697101995924,10cyclictest30-21ksoftirqd/200:00:062
697089995923,10cyclictest12-21ksoftirqd/023:51:550
697089995918,5cyclictest131rcu_preempt23:20:300
697104995813,6cyclictest131rcu_preempt23:15:013
69710199587,6cyclictest131rcu_preempt23:50:022
69710199586,11cyclictest30-21ksoftirqd/223:25:012
69710199584,11cyclictest30-21ksoftirqd/221:42:572
69710199583,7cyclictest30-21ksoftirqd/223:00:012
697101995814,5cyclictest131rcu_preempt23:30:022
697089995827,15cyclictest753731-21ssh23:07:110
697089995823,9cyclictest12-21ksoftirqd/023:26:540
69708999582,38cyclictest791285-21ssh00:34:590
69710499578,8cyclictest37-21ksoftirqd/320:00:003
69710499575,7cyclictest37-21ksoftirqd/319:30:013
697104995710,6cyclictest131rcu_preempt23:00:013
69710199579,9cyclictest30-21ksoftirqd/221:20:002
69710199577,9cyclictest30-21ksoftirqd/222:00:002
69710199576,9cyclictest30-21ksoftirqd/222:40:082
69710199576,9cyclictest30-21ksoftirqd/222:40:072
69710199576,5cyclictest131rcu_preempt23:10:132
69710199575,9cyclictest30-21ksoftirqd/222:30:012
697101995712,6cyclictest131rcu_preempt19:40:002
697101995711,6cyclictest131rcu_preempt21:10:002
697101995710,9cyclictest131rcu_preempt19:50:032
697089995730,12cyclictest751989-21ssh23:03:060
697089995723,31cyclictest12-21ksoftirqd/000:07:220
697089995720,31cyclictest12-21ksoftirqd/000:00:030
69710499569,6cyclictest131rcu_preempt23:24:173
69710499569,12cyclictest37-21ksoftirqd/321:35:013
697104995646,7cyclictest37-21ksoftirqd/323:45:023
697104995610,6cyclictest131rcu_preempt22:10:003
697104995610,6cyclictest131rcu_preempt21:20:003
69710199569,9cyclictest131rcu_preempt20:45:002
69710199569,7cyclictest131rcu_preempt21:40:002
69710199569,5cyclictest131rcu_preempt21:25:442
69710199562,9cyclictest30-21ksoftirqd/200:30:012
697101995627,7cyclictest30-21ksoftirqd/223:39:352
697101995613,10cyclictest30-21ksoftirqd/223:00:302
697101995612,6cyclictest131rcu_preempt22:19:402
69708999569,35cyclictest131rcu_preempt22:25:020
697089995620,32cyclictest12-21ksoftirqd/021:46:070
697089995613,5cyclictest131rcu_preempt00:25:490
69710499555,7cyclictest37-21ksoftirqd/300:05:003
697104995512,12cyclictest37-21ksoftirqd/300:25:203
697104995511,6cyclictest131rcu_preempt23:40:013
697104995511,6cyclictest131rcu_preempt21:43:533
69710199558,5cyclictest131rcu_preempt21:52:012
69710199556,9cyclictest30-21ksoftirqd/200:40:012
69710199555,8cyclictest30-21ksoftirqd/222:04:322
69710199555,11cyclictest30-21ksoftirqd/220:20:012
69710199554,7cyclictest30-21ksoftirqd/222:10:002
697101995513,5cyclictest131rcu_preempt21:11:502
697101995510,5cyclictest131rcu_preempt21:47:572
697089995520,31cyclictest707721-21cut21:20:020
697089995512,35cyclictest131rcu_preempt21:38:330
69710499549,5cyclictest131rcu_preempt22:46:373
69710499544,12cyclictest37-21ksoftirqd/322:40:033
69710499544,12cyclictest37-21ksoftirqd/322:40:023
69710499544,10cyclictest37-21ksoftirqd/319:40:003
697104995415,10cyclictest37-21ksoftirqd/320:10:003
697104995414,6cyclictest131rcu_preempt23:35:003
697104995414,6cyclictest131rcu_preempt23:09:003
697104995414,6cyclictest131rcu_preempt21:00:023
697104995412,6cyclictest131rcu_preempt21:55:033
697104995411,5cyclictest131rcu_preempt22:22:343
697104995410,6cyclictest131rcu_preempt22:55:003
69710199549,6cyclictest131rcu_preempt22:39:492
69710199546,11cyclictest30-21ksoftirqd/223:20:012
69710199545,9cyclictest30-21ksoftirqd/221:23:532
69710199545,6cyclictest131rcu_preempt00:10:012
697101995446,5cyclictest784945-21kworker/2:300:30:112
697101995413,6cyclictest131rcu_preempt23:35:012
697101995411,7cyclictest131rcu_preempt21:34:432
697101995410,5cyclictest131rcu_preempt20:55:022
697101995410,5cyclictest131rcu_preempt19:35:012
697089995413,36cyclictest0-21swapper/019:25:000
69710499537,10cyclictest37-21ksoftirqd/322:15:013
69710499536,12cyclictest37-21ksoftirqd/323:20:013
697104995327,10cyclictest37-21ksoftirqd/320:20:013
697104995311,6cyclictest131rcu_preempt21:40:033
697104995310,9cyclictest131rcu_preempt00:10:003
69710199537,8cyclictest30-21ksoftirqd/219:30:012
69710199534,11cyclictest30-21ksoftirqd/222:15:012
69710199532,12cyclictest30-21ksoftirqd/219:15:002
69710199532,12cyclictest30-21ksoftirqd/200:13:452
697101995315,6cyclictest131rcu_preempt20:00:002
697101995311,6cyclictest131rcu_preempt23:41:412
69710199531,12cyclictest30-21ksoftirqd/221:00:002
697101995310,5cyclictest131rcu_preempt19:45:012
69708999532,36cyclictest709087-21ssh21:23:210
697089995322,9cyclictest3467602-21systemd-journal20:35:010
697089995316,7cyclictest12-21ksoftirqd/021:42:080
69710499529,5cyclictest131rcu_preempt19:10:033
69710499529,5cyclictest131rcu_preempt19:10:023
69710499528,9cyclictest37-21ksoftirqd/300:20:013
69710499523,11cyclictest37-21ksoftirqd/322:30:003
697104995225,19cyclictest37-21ksoftirqd/320:49:263
69710499522,12cyclictest37-21ksoftirqd/321:46:503
69710499521,7cyclictest37-21ksoftirqd/323:05:023
697104995214,10cyclictest37-21ksoftirqd/319:25:023
697104995212,6cyclictest131rcu_preempt00:12:583
69710199526,8cyclictest30-21ksoftirqd/222:46:572
69710199524,7cyclictest30-21ksoftirqd/200:21:152
69710199523,9cyclictest30-21ksoftirqd/222:51:232
697101995213,9cyclictest30-21ksoftirqd/219:55:022
697094995214,8cyclictest131rcu_preempt19:50:001
697089995233,16cyclictest1-21systemd21:10:020
697089995225,12cyclictest769548-21ssh23:44:140
697089995215,28cyclictest131rcu_preempt22:09:540
69708999521,45cyclictest711849-21cut21:30:000
69708999521,36cyclictest792695-21ssh00:38:080
69710499517,9cyclictest37-21ksoftirqd/320:40:003
69710499517,9cyclictest37-21ksoftirqd/319:15:003
69710499517,5cyclictest131rcu_preempt21:05:033
697104995113,5cyclictest346771850lldpd19:31:463
697104995111,5cyclictest131rcu_preempt23:46:213
69710199518,5cyclictest131rcu_preempt23:50:342
697094995135,12cyclictest275-21dbus-daemon20:15:001
69708999516,5cyclictest131rcu_preempt23:36:090
69708999515,5cyclictest131rcu_preempt23:14:030
697089995125,11cyclictest781997-21ssh00:13:080
69710499507,7cyclictest37-21ksoftirqd/321:30:003
69710499506,7cyclictest37-21ksoftirqd/319:50:013
69710199508,5cyclictest131rcu_preempt20:09:592
697101995012,6cyclictest131rcu_preempt20:50:022
697101995010,5cyclictest131rcu_preempt20:30:022
697089995027,8cyclictest739628-21ssh22:34:320
697089995027,8cyclictest739628-21ssh22:34:320
697089995025,6cyclictest131rcu_preempt22:50:450
69708999502,27cyclictest131rcu_preempt00:21:470
69708999501,34cyclictest131rcu_preempt22:49:000
697089995010,34cyclictest733343-21latency_hist22:20:020
69710499499,6cyclictest131rcu_preempt19:17:003
69710499494,8cyclictest37-21ksoftirqd/320:15:033
697104994919,19cyclictest37-21ksoftirqd/320:25:293
697101994940,5cyclictest690366-21kworker/2:120:32:342
697101994934,11cyclictest690366-21kworker/2:120:02:262
697101994934,11cyclictest690366-21kworker/2:120:02:262
697101994913,5cyclictest131rcu_preempt19:20:032
69708999499,22cyclictest131rcu_preempt23:15:040
697089994945,2cyclictest0-21swapper/021:30:020
69708999493,6cyclictest131rcu_preempt23:34:010
697089994927,3cyclictest131rcu_preempt22:36:230
697089994910,10cyclictest131rcu_preempt19:40:010
697104994810,6cyclictest131rcu_preempt20:45:023
697101994840,5cyclictest702503-21kworker/2:221:04:442
697101994840,5cyclictest690366-21kworker/2:120:20:282
697101994839,6cyclictest699379-21kworker/2:220:11:242
69710199483,5cyclictest131rcu_preempt19:20:012
697101994812,6cyclictest131rcu_preempt20:35:442
69709499487,37cyclictest729601-21sshd22:10:441
697094994837,8cyclictest732450-21sshd22:17:431
697094994830,14cyclictest776317-21systemd-run00:00:011
697094994827,14cyclictest0-21swapper/100:00:281
697094994823,20cyclictest23-21ksoftirqd/120:20:001
697089994844,2cyclictest0-21swapper/021:00:020
697089994836,7cyclictest12-21ksoftirqd/021:15:020
69710499478,6cyclictest131rcu_preempt19:55:013
69710499476,6cyclictest131rcu_preempt20:55:023
697104994714,5cyclictest131rcu_preempt20:25:013
697104994711,5cyclictest131rcu_preempt20:35:023
69709499476,33cyclictest23-21ksoftirqd/120:50:021
697089994727,16cyclictest771375-21ssh23:48:270
69709499467,5cyclictest131rcu_preempt21:05:021
697094994635,7cyclictest716136-21sendmail-msp21:40:001
697089994627,7cyclictest12-21ksoftirqd/021:55:010
697089994618,23cyclictest12-21ksoftirqd/021:05:020
697089994615,18cyclictest12-21ksoftirqd/022:45:020
697089994615,18cyclictest12-21ksoftirqd/022:45:020
697089994611,6cyclictest131rcu_preempt19:55:010
697104994536,7cyclictest37-21ksoftirqd/319:45:023
69709499453,3cyclictest131rcu_preempt00:30:001
697094994530,11cyclictest0-21swapper/100:06:321
697089994511,5cyclictest12-21ksoftirqd/019:32:350
697094994431,10cyclictest699885-21sshd20:04:441
697094994431,10cyclictest699885-21sshd20:04:441
697094994428,13cyclictest275-21dbus-daemon20:40:011
69709499442,3cyclictest131rcu_preempt23:40:271
69709499441,27cyclictest690050-21kworker/u8:319:25:021
697089994413,22cyclictest12-21ksoftirqd/020:50:010
697094994331,8cyclictest700964-21latency_hist20:25:021
697094994330,9cyclictest1-21systemd23:20:021
697094994328,11cyclictest23-21ksoftirqd/123:30:001
69709499431,3cyclictest131rcu_preempt22:29:201
69708999432,37cyclictest700118-21sshd20:05:440
697094994229,10cyclictest23-21ksoftirqd/100:35:241
69709499422,3cyclictest131rcu_preempt22:25:031
697089994235,4cyclictest16950irq/224-800000019:30:000
697089994234,4cyclictest16950irq/224-800000020:20:000
697089994216,17cyclictest0-21swapper/019:50:010
69709499414,3cyclictest131rcu_preempt22:00:241
697094994130,8cyclictest23-21ksoftirqd/122:10:021
697094994128,10cyclictest1-21systemd21:10:001
697094994126,12cyclictest23-21ksoftirqd/121:45:521
697094994119,18cyclictest23-21ksoftirqd/119:15:001
69709499411,4cyclictest131rcu_preempt22:39:591
697094994111,6cyclictest131rcu_preempt21:58:101
69708999412,8cyclictest3467602-21systemd-journal20:41:540
697089994125,10cyclictest701718-21systemctl20:40:010
697089994123,15cyclictest12-21ksoftirqd/019:20:010
69709499404,32cyclictest275-21dbus-daemon00:25:021
697094994030,7cyclictest717789-21sshd21:43:451
697094994029,8cyclictest23-21ksoftirqd/123:45:181
697094994027,10cyclictest753967-21sshd23:07:441
697094994027,10cyclictest23-21ksoftirqd/100:10:131
697094994026,11cyclictest275-21dbus-daemon20:55:471
697094994025,11cyclictest3467602-21systemd-journal19:17:001
697094994023,12cyclictest23-21ksoftirqd/121:14:221
69709499401,3cyclictest131rcu_preempt23:04:511
697089994012,24cyclictest12-21ksoftirqd/019:55:160
69709499399,5cyclictest131rcu_preempt21:26:141
697094993931,6cyclictest23-21ksoftirqd/120:45:021
697094993931,5cyclictest707610-21sh21:20:001
697094993926,9cyclictest23-21ksoftirqd/122:40:081
697094993926,9cyclictest23-21ksoftirqd/122:40:071
697094993925,9cyclictest0-21swapper/120:10:021
697094993924,12cyclictest1-21systemd23:35:011
697094993922,13cyclictest275-21dbus-daemon21:55:011
697094993922,13cyclictest275-21dbus-daemon21:25:011
69709499385,8cyclictest23-21ksoftirqd/123:55:021
69709499385,8cyclictest23-21ksoftirqd/100:30:161
697094993830,5cyclictest698590-21sendmail19:40:001
697094993828,8cyclictest23-21ksoftirqd/123:22:531
69709499381,3cyclictest131rcu_preempt22:45:231
69709499381,3cyclictest131rcu_preempt20:35:011
69709499378,6cyclictest131rcu_preempt22:50:191
697094993729,5cyclictest714007-21cron21:35:001
697094993729,4cyclictest698105-21latency_hist19:30:011
697094993728,7cyclictest23-21ksoftirqd/100:19:121
697094993726,8cyclictest702467-21sshd20:53:441
69709499372,4cyclictest131rcu_preempt19:45:011
69709499372,3cyclictest131rcu_preempt22:33:111
69709499372,3cyclictest131rcu_preempt22:33:101
697089993717,17cyclictest0-21swapper/020:03:210
697089993717,17cyclictest0-21swapper/020:03:200
69708999371,32cyclictest0-21swapper/020:25:220
697089993711,17cyclictest697091-21latency_hist19:10:020
697089993711,17cyclictest697091-21latency_hist19:10:020
697094993623,11cyclictest23-21ksoftirqd/123:38:241
69709499361,3cyclictest131rcu_preempt22:55:201
697094993611,5cyclictest131rcu_preempt23:10:561
69708999362,30cyclictest0-21swapper/020:20:440
697094993524,8cyclictest23-21ksoftirqd/119:35:021
697094993521,10cyclictest701376-21cat20:30:021
697094993416,14cyclictest699633-21systemctl20:00:001
697089993425,7cyclictest12-21ksoftirqd/020:15:020
697094993317,12cyclictest3467602-21systemd-journal19:54:441
697094993023,4cyclictest697182-21latency_hist19:10:021
697094993023,4cyclictest697182-21latency_hist19:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional