You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-11 - 10:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Aug 11, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3772960997565,7cyclictest12-21ksoftirqd/021:15:010
3772960997212,5cyclictest131rcu_preempt22:00:000
3772974996925,11cyclictest37-21ksoftirqd/321:15:023
3772970996927,11cyclictest30-21ksoftirqd/223:25:012
3772965996812,6cyclictest131rcu_preempt00:40:011
377297499646,12cyclictest37-21ksoftirqd/323:21:183
3772974996326,11cyclictest37-21ksoftirqd/300:00:023
377297499627,7cyclictest37-21ksoftirqd/322:45:003
377297099627,5cyclictest131rcu_preempt22:20:012
377297099626,8cyclictest30-21ksoftirqd/200:40:022
377297099624,9cyclictest30-21ksoftirqd/200:35:012
377296099624,6cyclictest121ksoftirqd/021:29:010
377296099624,6cyclictest121ksoftirqd/021:29:000
377296099615,9cyclictest12-21ksoftirqd/021:40:010
377296099614,9cyclictest12-21ksoftirqd/023:40:010
3772960996139,15cyclictest12-21ksoftirqd/022:58:320
377297499605,14cyclictest37-21ksoftirqd/319:45:023
377297099605,9cyclictest30-21ksoftirqd/221:20:022
377297099605,9cyclictest30-21ksoftirqd/221:20:012
377297099604,8cyclictest30-21ksoftirqd/223:00:022
3772960996031,7cyclictest12-21ksoftirqd/022:17:130
3772960996014,8cyclictest12-21ksoftirqd/000:14:150
377297499598,10cyclictest37-21ksoftirqd/300:36:443
377297499597,9cyclictest37-21ksoftirqd/322:16:433
377297499597,11cyclictest37-21ksoftirqd/321:52:463
377297499595,9cyclictest37-21ksoftirqd/323:35:003
3772974995933,10cyclictest37-21ksoftirqd/322:39:473
3772974995916,9cyclictest131rcu_preempt23:10:573
377297099594,9cyclictest30-21ksoftirqd/223:14:492
3772970995910,5cyclictest131rcu_preempt22:29:522
377296099598,6cyclictest131rcu_preempt23:30:020
377296099598,6cyclictest131rcu_preempt23:30:010
377296099591,7cyclictest12-21ksoftirqd/021:35:000
3772960995917,37cyclictest16850irq/134-800000023:31:410
377296099591,42cyclictest3822892-21ssh22:42:370
377297499588,10cyclictest37-21ksoftirqd/321:25:463
377297499588,10cyclictest37-21ksoftirqd/321:25:453
377297499587,9cyclictest37-21ksoftirqd/322:50:053
377297499586,9cyclictest37-21ksoftirqd/323:42:323
377297499583,12cyclictest37-21ksoftirqd/300:17:003
377297499583,10cyclictest37-21ksoftirqd/320:20:013
377297099589,5cyclictest131rcu_preempt20:40:022
377297099584,7cyclictest30-21ksoftirqd/221:55:012
3772970995819,12cyclictest30-21ksoftirqd/223:05:022
3772970995817,10cyclictest30-21ksoftirqd/200:24:252
3772970995817,10cyclictest30-21ksoftirqd/200:24:242
3772970995810,5cyclictest131rcu_preempt21:15:012
3772970995810,5cyclictest131rcu_preempt00:04:092
377296099589,6cyclictest131rcu_preempt23:50:020
3772960995819,34cyclictest16850irq/134-800000023:55:030
377297499578,9cyclictest37-21ksoftirqd/321:40:023
377297499577,9cyclictest37-21ksoftirqd/320:35:003
377297499576,12cyclictest37-21ksoftirqd/323:18:423
377297499576,10cyclictest37-21ksoftirqd/323:47:113
3772974995714,9cyclictest37-21ksoftirqd/300:25:033
3772974995714,9cyclictest37-21ksoftirqd/300:25:023
377297099579,6cyclictest131rcu_preempt23:35:262
3772970995740,12cyclictest291rcuc/221:10:012
3772970995740,12cyclictest291rcuc/221:10:012
3772970995717,10cyclictest30-21ksoftirqd/223:06:312
3772970995714,7cyclictest30-21ksoftirqd/223:58:542
3772970995711,7cyclictest131rcu_preempt00:27:502
377296099579,6cyclictest131rcu_preempt22:05:010
377296099575,9cyclictest12-21ksoftirqd/023:17:200
3772960995728,14cyclictest3812432-21ssh22:20:240
3772960995727,15cyclictest3797110-21ssh21:47:530
3772960995715,5cyclictest131rcu_preempt23:05:010
377297499569,9cyclictest37-21ksoftirqd/322:25:533
377297499568,9cyclictest37-21ksoftirqd/322:56:453
377297499567,7cyclictest37-21ksoftirqd/321:10:023
377297499567,7cyclictest37-21ksoftirqd/321:10:013
377297499567,10cyclictest37-21ksoftirqd/300:25:303
377297499563,12cyclictest37-21ksoftirqd/320:00:013
377297499563,12cyclictest37-21ksoftirqd/320:00:013
377297499563,11cyclictest37-21ksoftirqd/300:05:013
3772974995616,21cyclictest37-21ksoftirqd/323:40:033
3772974995615,11cyclictest37-21ksoftirqd/321:16:353
3772974995615,11cyclictest37-21ksoftirqd/321:16:343
3772974995614,11cyclictest37-21ksoftirqd/322:34:213
377297099569,6cyclictest131rcu_preempt23:51:442
377296099567,32cyclictest3772964-21latency_hist19:10:020
377296099565,9cyclictest12-21ksoftirqd/000:20:020
377296099564,10cyclictest12-21ksoftirqd/000:05:020
377296099563,41cyclictest12-21ksoftirqd/021:20:020
3772960995630,8cyclictest12-21ksoftirqd/022:30:220
3772960995628,13cyclictest3782766-21ssh21:17:390
3772960995628,13cyclictest3782766-21ssh21:17:390
377297499558,7cyclictest37-21ksoftirqd/322:03:203
377297499558,13cyclictest37-21ksoftirqd/323:54:533
377297499557,9cyclictest37-21ksoftirqd/323:04:593
377297499557,9cyclictest37-21ksoftirqd/322:23:283
377297499557,10cyclictest37-21ksoftirqd/322:06:403
377297499557,10cyclictest37-21ksoftirqd/321:42:083
377297499556,10cyclictest37-21ksoftirqd/322:50:013
377297499556,10cyclictest37-21ksoftirqd/322:50:003
377297499555,9cyclictest37-21ksoftirqd/300:06:353
377297499555,11cyclictest37-21ksoftirqd/319:15:013
377297499554,9cyclictest37-21ksoftirqd/323:10:003
377297499554,10cyclictest37-21ksoftirqd/321:33:203
3772974995514,5cyclictest131rcu_preempt19:25:023
3772974995514,10cyclictest131rcu_preempt00:33:203
3772974995512,6cyclictest131rcu_preempt21:56:573
377297099559,5cyclictest131rcu_preempt22:40:002
377297099556,8cyclictest30-21ksoftirqd/221:56:522
377297099555,11cyclictest30-21ksoftirqd/222:09:482
377297099554,5cyclictest131rcu_preempt22:35:032
377297099554,10cyclictest30-21ksoftirqd/220:29:592
3772970995512,5cyclictest131rcu_preempt22:03:562
3772970995510,5cyclictest131rcu_preempt21:40:022
377296099558,6cyclictest131rcu_preempt22:11:260
377296099554,7cyclictest121ksoftirqd/022:29:500
3772960995546,5cyclictest16850irq/134-800000021:10:010
3772960995546,5cyclictest16850irq/134-800000021:10:000
3772960995528,8cyclictest12-21ksoftirqd/022:54:110
3772960995528,24cyclictest151rcuc/023:55:090
3772960995517,29cyclictest16850irq/134-800000000:25:440
377296099551,39cyclictest3842176-21ssh23:23:230
377297499547,7cyclictest37-21ksoftirqd/321:46:183
377297499547,10cyclictest37-21ksoftirqd/319:55:033
377297499546,7cyclictest37-21ksoftirqd/323:26:473
377297499546,7cyclictest37-21ksoftirqd/323:26:473
377297499541,7cyclictest37-21ksoftirqd/320:45:023
377297499541,7cyclictest37-21ksoftirqd/320:45:013
3772974995415,5cyclictest131rcu_preempt21:05:003
3772974995411,5cyclictest131rcu_preempt22:10:393
377297099545,9cyclictest30-21ksoftirqd/223:20:022
3772970995446,5cyclictest3775006-21kworker/2:321:44:232
377297099543,9cyclictest30-21ksoftirqd/223:33:152
3772970995418,6cyclictest131rcu_preempt21:34:502
3772970995412,5cyclictest131rcu_preempt23:47:582
3772970995411,6cyclictest131rcu_preempt19:50:032
3772970995410,5cyclictest131rcu_preempt00:15:022
377296599544,42cyclictest3802911-21ssh22:00:031
3772965995437,13cyclictest275-21dbus-daemon20:15:011
3772965995433,17cyclictest3859496-21sendmail-msp00:00:011
377296099545,6cyclictest121ksoftirqd/021:44:110
3772960995426,13cyclictest3837988-21ssh23:14:380
3772960995410,6cyclictest131rcu_preempt21:55:030
377297499536,10cyclictest37-21ksoftirqd/319:40:033
377297499535,9cyclictest37-21ksoftirqd/321:22:513
377297499531,10cyclictest37-21ksoftirqd/321:00:013
377297499531,10cyclictest37-21ksoftirqd/321:00:013
377297099539,5cyclictest131rcu_preempt21:23:022
377297099537,9cyclictest30-21ksoftirqd/222:41:582
3772970995340,5cyclictest3796142-21kworker/2:122:45:412
3772970995340,5cyclictest3796142-21kworker/2:122:45:412
3772970995314,6cyclictest131rcu_preempt21:50:012
3772970995312,7cyclictest131rcu_preempt22:11:042
3772970995312,10cyclictest30-21ksoftirqd/200:20:032
3772970995311,5cyclictest131rcu_preempt23:42:242
3772970995310,6cyclictest131rcu_preempt20:15:012
3772960995333,12cyclictest0-21swapper/023:40:040
3772960995325,13cyclictest3834497-21ssh23:07:080
3772960995324,14cyclictest3806094-21ssh22:06:410
3772960995318,31cyclictest16850irq/134-800000000:32:590
3772960995318,31cyclictest16850irq/134-800000000:06:590
3772960995317,30cyclictest16850irq/134-800000000:25:010
3772960995317,30cyclictest16850irq/134-800000000:25:000
3772960995314,21cyclictest16850irq/134-800000022:35:020
377296099531,14cyclictest12-21ksoftirqd/022:50:020
377296099531,14cyclictest12-21ksoftirqd/022:50:010
3772974995210,9cyclictest37-21ksoftirqd/319:45:013
377297099529,5cyclictest131rcu_preempt23:25:592
377297099529,5cyclictest131rcu_preempt23:25:592
377297099529,5cyclictest131rcu_preempt21:27:062
377297099529,5cyclictest131rcu_preempt21:27:052
3772970995244,5cyclictest3796142-21kworker/2:122:52:052
3772970995224,11cyclictest30-21ksoftirqd/220:00:022
3772970995224,11cyclictest30-21ksoftirqd/220:00:022
3772970995211,6cyclictest131rcu_preempt19:45:012
377296099523,7cyclictest12-21ksoftirqd/019:30:000
377297499516,7cyclictest37-21ksoftirqd/320:30:023
377297499516,7cyclictest37-21ksoftirqd/300:10:113
377297099515,5cyclictest131rcu_preempt20:35:012
3772970995135,11cyclictest1-21systemd19:15:002
3772970995112,6cyclictest131rcu_preempt22:24:442
3772970995112,5cyclictest131rcu_preempt00:08:192
3772970995110,4cyclictest131rcu_preempt20:20:002
3772965995143,4cyclictest3776541-21modprobe20:20:011
377296099517,5cyclictest131rcu_preempt19:25:020
377297499508,7cyclictest37-21ksoftirqd/320:55:023
377297499507,8cyclictest37-21ksoftirqd/320:05:013
377297499503,10cyclictest37-21ksoftirqd/319:10:013
377297499501,19cyclictest0-21swapper/320:40:003
3772970995043,4cyclictest3776284-21kworker/2:120:55:302
3772970995043,4cyclictest3776284-21kworker/2:120:55:302
3772970995041,6cyclictest3769055-21kworker/2:119:37:282
3772970995012,6cyclictest131rcu_preempt19:25:012
3772960995013,32cyclictest16850irq/134-800000000:35:180
377296099501,21cyclictest12-21ksoftirqd/020:45:020
377297499497,8cyclictest37-21ksoftirqd/320:25:023
377297499496,7cyclictest37-21ksoftirqd/320:15:023
377297499491,14cyclictest37-21ksoftirqd/320:10:013
377297099497,10cyclictest30-21ksoftirqd/220:04:592
3772970994941,5cyclictest3776284-21kworker/2:120:41:332
3772970994941,5cyclictest3776284-21kworker/2:120:41:332
3772970994941,5cyclictest3769055-21kworker/2:119:31:452
3772965994929,16cyclictest1-21systemd21:00:011
3772965994929,16cyclictest1-21systemd21:00:001
3772960994930,15cyclictest275-21dbus-daemon20:10:000
377297499482,9cyclictest37-21ksoftirqd/319:20:023
3772970994841,4cyclictest3775006-21kworker/2:320:45:322
3772970994840,5cyclictest3776284-21kworker/2:120:22:002
3772970994840,5cyclictest3774746-21kworker/2:219:50:072
3772970994839,5cyclictest3775006-21kworker/2:320:08:102
3772970994833,11cyclictest3467602-21systemd-journal19:27:452
3772965994830,13cyclictest3467602-21systemd-journal22:11:461
3772960994832,12cyclictest1-21systemd19:50:010
3772960994812,7cyclictest131rcu_preempt20:40:010
3772960994810,6cyclictest131rcu_preempt20:20:010
377297499476,8cyclictest37-21ksoftirqd/320:50:013
3772970994739,5cyclictest3776284-21kworker/2:121:02:012
3772970994739,5cyclictest3776284-21kworker/2:120:51:262
3772970994739,5cyclictest3772685-21kworker/2:019:16:462
377296099477,5cyclictest131rcu_preempt19:20:010
3772960994733,10cyclictest3467602-21systemd-journal20:45:320
377296099471,43cyclictest0-21swapper/020:40:020
3772960994712,11cyclictest12-21ksoftirqd/019:35:010
3772960994711,6cyclictest131rcu_preempt21:00:020
3772960994711,6cyclictest131rcu_preempt21:00:010
3772965994618,4cyclictest23-21ksoftirqd/122:30:021
377296099469,6cyclictest131rcu_preempt19:58:450
377296099469,6cyclictest131rcu_preempt19:58:440
3772960994613,5cyclictest131rcu_preempt19:15:010
377297499451,9cyclictest37-21ksoftirqd/319:26:183
377296599457,6cyclictest131rcu_preempt22:05:001
377296599457,4cyclictest23-21ksoftirqd/119:50:021
3772965994530,12cyclictest1-21systemd19:20:011
3772965994529,12cyclictest3467602-21systemd-journal00:26:321
377296599452,3cyclictest131rcu_preempt00:23:191
377296599452,3cyclictest131rcu_preempt00:23:181
377296599451,5cyclictest131rcu_preempt00:10:031
3772960994538,4cyclictest16950irq/224-800000019:40:010
3772960994511,5cyclictest131rcu_preempt19:55:010
377297499446,8cyclictest37-21ksoftirqd/319:35:023
377296599447,6cyclictest131rcu_preempt21:10:011
377296599447,6cyclictest131rcu_preempt21:10:001
3772965994430,11cyclictest23-21ksoftirqd/121:43:321
377296599441,3cyclictest131rcu_preempt21:50:021
377296099445,7cyclictest12-21ksoftirqd/019:42:450
377296099445,6cyclictest131rcu_preempt20:15:030
3772960994429,11cyclictest1-21systemd20:35:000
3772960994420,7cyclictest12-21ksoftirqd/021:02:370
3772960994416,11cyclictest12-21ksoftirqd/020:25:020
3772960994410,5cyclictest131rcu_preempt20:55:020
3772965994329,11cyclictest23-21ksoftirqd/122:40:011
3772965994329,10cyclictest23-21ksoftirqd/123:15:021
3772965994328,11cyclictest1-21systemd22:55:001
3772965994326,12cyclictest1-21systemd20:25:011
3772965994322,15cyclictest3467602-21systemd-journal23:10:001
377296099437,7cyclictest12-21ksoftirqd/020:30:020
377297099429,5cyclictest131rcu_preempt19:10:012
377296599429,8cyclictest131rcu_preempt23:50:001
377296599428,5cyclictest131rcu_preempt22:25:021
3772965994228,9cyclictest3467602-21systemd-journal20:50:001
3772965994228,11cyclictest23-21ksoftirqd/122:20:001
3772965994228,10cyclictest3467602-21systemd-journal00:00:031
3772965994225,14cyclictest23-21ksoftirqd/119:15:021
377296599422,4cyclictest131rcu_preempt23:28:281
377296599422,4cyclictest131rcu_preempt23:28:281
3772965994224,14cyclictest3467602-21systemd-journal22:08:451
3772965994219,17cyclictest3467602-21systemd-journal21:25:001
377296599421,4cyclictest131rcu_preempt23:02:211
377296599421,3cyclictest131rcu_preempt23:15:541
377296599413,3cyclictest131rcu_preempt23:23:531
377296599413,3cyclictest131rcu_preempt23:00:011
377296599413,3cyclictest131rcu_preempt19:45:011
3772965994128,10cyclictest23-21ksoftirqd/100:07:371
377296599411,3cyclictest131rcu_preempt21:32:321
3772960994124,12cyclictest275-21dbus-daemon20:05:000
3772965994032,5cyclictest3849996-21latency_hist23:40:011
377296599402,3cyclictest131rcu_preempt23:31:101
377296599401,3cyclictest131rcu_preempt21:39:061
3772965993929,8cyclictest23-21ksoftirqd/122:30:151
3772965993928,8cyclictest23-21ksoftirqd/100:20:001
3772965993924,11cyclictest23-21ksoftirqd/121:11:101
377296599392,3cyclictest131rcu_preempt22:46:581
377296599392,3cyclictest131rcu_preempt22:46:581
377296599384,3cyclictest131rcu_preempt21:17:481
377296599384,3cyclictest131rcu_preempt21:17:481
377296599383,3cyclictest131rcu_preempt21:05:021
3772965993826,9cyclictest23-21ksoftirqd/121:53:321
3772965993825,8cyclictest23-21ksoftirqd/119:40:011
3772965993825,10cyclictest23-21ksoftirqd/120:39:591
3772965993824,11cyclictest23-21ksoftirqd/122:44:591
3772965993823,11cyclictest3467602-21systemd-journal20:27:181
3772965993822,12cyclictest3467602-21systemd-journal00:30:021
377296599381,4cyclictest131rcu_preempt20:00:011
377296599381,4cyclictest131rcu_preempt20:00:001
377296599381,3cyclictest131rcu_preempt23:53:351
3772965993810,5cyclictest131rcu_preempt19:25:011
3772965993729,5cyclictest3467562-21sshd21:25:461
3772965993729,5cyclictest3467562-21sshd21:25:451
3772965993718,3cyclictest23-21ksoftirqd/119:30:021
3772965993718,14cyclictest275-21dbus-daemon20:55:011
3772965993627,7cyclictest23-21ksoftirqd/123:42:571
3772965993621,11cyclictest1-21systemd20:05:011
3772965993620,11cyclictest0-21swapper/119:35:001
3772965993522,10cyclictest1-21systemd20:35:021
3772965993521,11cyclictest23-21ksoftirqd/120:10:001
377296599343,3cyclictest131rcu_preempt19:55:031
377296599332,5cyclictest131rcu_preempt20:45:011
377296599332,5cyclictest131rcu_preempt20:45:011
377296599272,3cyclictest131rcu_preempt19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional