You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-28 - 04:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Jul 28, 2026 00:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
368804399981,94cyclictest0-21swapper/000:00:030
3688043999226,22cyclictest131rcu_preempt20:25:020
3688043998726,14cyclictest3784648-21ssh00:18:580
368804399833,3cyclictest131rcu_preempt19:45:020
3688043998324,14cyclictest3771161-21ssh23:50:290
3688043998127,13cyclictest3776302-21ssh00:01:100
3688043998027,14cyclictest3746605-21ssh22:59:150
3688043997729,12cyclictest3747814-21ssh23:01:520
3688043997310,41cyclictest12-21ksoftirqd/021:40:030
3688052996914,13cyclictest30-21ksoftirqd/223:15:012
3688052996912,5cyclictest131rcu_preempt00:25:022
3688052996526,8cyclictest30-21ksoftirqd/223:44:132
368805799649,6cyclictest131rcu_preempt21:35:023
368805799649,6cyclictest131rcu_preempt21:35:013
3688057996414,32cyclictest37-21ksoftirqd/300:25:033
368805299636,8cyclictest30-21ksoftirqd/200:30:002
368805299634,9cyclictest30-21ksoftirqd/223:55:022
3688057996221,9cyclictest37-21ksoftirqd/323:23:463
368805299625,8cyclictest30-21ksoftirqd/221:20:022
3688052996222,9cyclictest30-21ksoftirqd/223:20:022
3688052996218,10cyclictest30-21ksoftirqd/222:05:022
3688052996217,11cyclictest30-21ksoftirqd/221:40:002
3688043996238,16cyclictest1-21systemd00:30:000
3688043996221,36cyclictest275-21dbus-daemon22:45:010
368805799616,6cyclictest131rcu_preempt20:25:013
368805299617,9cyclictest30-21ksoftirqd/222:50:012
368805299617,9cyclictest30-21ksoftirqd/222:50:012
368805299616,12cyclictest30-21ksoftirqd/200:00:482
368805299611,10cyclictest30-21ksoftirqd/221:55:012
368804899612,14cyclictest23-21ksoftirqd/119:25:021
3688043996118,9cyclictest3787219-21ssh00:24:180
368805799602,8cyclictest37-21ksoftirqd/323:40:013
368805299603,8cyclictest30-21ksoftirqd/222:00:012
368804899604,11cyclictest23-21ksoftirqd/123:20:021
3688048996037,10cyclictest23-21ksoftirqd/122:00:131
368804399601,52cyclictest3693691-21latency_hist21:00:000
368805299597,8cyclictest30-21ksoftirqd/221:15:032
368805299594,13cyclictest30-21ksoftirqd/222:25:022
3688052995939,7cyclictest3708702-21kworker/2:322:54:532
368805299592,11cyclictest30-21ksoftirqd/220:40:002
3688052995913,5cyclictest131rcu_preempt21:26:522
3688052995911,7cyclictest131rcu_preempt23:06:542
368804399593,30cyclictest0-21swapper/022:00:020
3688043995914,39cyclictest3715708-21latency_hist21:55:040
368805799584,8cyclictest37-21ksoftirqd/322:15:013
368805799584,8cyclictest37-21ksoftirqd/322:15:013
368805299586,10cyclictest30-21ksoftirqd/221:50:012
368805299586,10cyclictest30-21ksoftirqd/221:40:182
368804899587,8cyclictest23-21ksoftirqd/123:40:361
3688043995837,5cyclictest16850irq/134-800000022:40:010
3688043995836,14cyclictest3467602-21systemd-journal23:20:000
368804399582,18cyclictest3696622-21munin-run21:14:590
368805799576,8cyclictest37-21ksoftirqd/321:55:043
3688057995713,6cyclictest131rcu_preempt22:05:003
3688057995710,5cyclictest131rcu_preempt23:20:023
368805299577,8cyclictest30-21ksoftirqd/220:25:032
368805299577,8cyclictest30-21ksoftirqd/200:35:032
368805299576,8cyclictest30-21ksoftirqd/223:30:522
368805299573,9cyclictest30-21ksoftirqd/219:55:012
3688052995718,7cyclictest30-21ksoftirqd/222:17:072
3688052995715,11cyclictest30-21ksoftirqd/219:35:012
3688052995715,10cyclictest30-21ksoftirqd/223:40:012
3688052995713,11cyclictest30-21ksoftirqd/223:50:012
3688052995712,10cyclictest30-21ksoftirqd/223:56:532
368804899579,9cyclictest23-21ksoftirqd/121:50:241
368804399571,4cyclictest131rcu_preempt00:40:000
368805799568,9cyclictest37-21ksoftirqd/323:50:023
368805799564,9cyclictest37-21ksoftirqd/323:15:013
368805299567,12cyclictest30-21ksoftirqd/221:10:012
368805299567,12cyclictest30-21ksoftirqd/221:10:012
368805299566,9cyclictest30-21ksoftirqd/222:40:352
3688052995642,6cyclictest3722968-21kworker/2:222:59:032
3688052995616,11cyclictest30-21ksoftirqd/222:32:512
3688052995615,12cyclictest30-21ksoftirqd/222:15:022
3688052995615,12cyclictest30-21ksoftirqd/222:15:012
3688052995614,11cyclictest30-21ksoftirqd/221:00:012
3688052995612,6cyclictest131rcu_preempt22:29:572
3688052995612,6cyclictest131rcu_preempt21:30:022
3688052995612,6cyclictest131rcu_preempt21:30:022
3688052995611,6cyclictest131rcu_preempt20:05:022
368804899567,7cyclictest23-21ksoftirqd/121:50:021
368804899567,10cyclictest23-21ksoftirqd/123:55:091
368804899566,8cyclictest23-21ksoftirqd/122:07:051
368804899565,8cyclictest23-21ksoftirqd/100:10:011
3688048995632,9cyclictest23-21ksoftirqd/121:34:281
3688048995632,9cyclictest23-21ksoftirqd/121:34:281
3688048995611,6cyclictest131rcu_preempt22:44:001
368804399564,4cyclictest131rcu_preempt23:10:030
3688043995642,8cyclictest16950irq/224-800000021:10:010
3688043995642,8cyclictest16950irq/224-800000021:10:000
368805799553,8cyclictest37-21ksoftirqd/322:26:463
3688057995538,12cyclictest1-21systemd23:55:003
368805299558,8cyclictest30-21ksoftirqd/221:22:482
368805299556,7cyclictest30-21ksoftirqd/220:35:022
3688052995517,10cyclictest30-21ksoftirqd/200:13:222
3688052995514,10cyclictest30-21ksoftirqd/220:15:022
368804899557,9cyclictest23-21ksoftirqd/123:47:371
368804899557,10cyclictest23-21ksoftirqd/100:20:091
368804899556,9cyclictest23-21ksoftirqd/123:15:011
3688048995517,10cyclictest23-21ksoftirqd/100:13:481
3688048995516,11cyclictest23-21ksoftirqd/123:28:081
3688048995510,10cyclictest23-21ksoftirqd/121:19:461
3688043995527,12cyclictest3725638-21ssh22:15:090
3688043995519,33cyclictest12-21ksoftirqd/021:17:050
3688043995519,13cyclictest3699664-21ssh21:20:460
368804399551,38cyclictest3742751-21ssh22:50:590
3688043995510,39cyclictest0-21swapper/023:40:030
368805799549,6cyclictest131rcu_preempt23:32:473
368805799546,5cyclictest131rcu_preempt21:30:013
368805799545,5cyclictest131rcu_preempt22:50:013
368805799545,5cyclictest131rcu_preempt22:50:003
368805799543,11cyclictest37-21ksoftirqd/300:15:013
368805799542,6cyclictest131rcu_preempt22:20:033
3688057995423,21cyclictest37-21ksoftirqd/319:40:093
368805299549,5cyclictest131rcu_preempt22:07:092
368805299548,5cyclictest131rcu_preempt23:25:022
368805299547,6cyclictest131rcu_preempt22:39:442
368805299544,10cyclictest30-21ksoftirqd/219:40:002
368805299543,11cyclictest30-21ksoftirqd/223:04:262
368805299541,6cyclictest0-21swapper/200:40:002
368804899548,10cyclictest23-21ksoftirqd/121:56:251
368804899547,9cyclictest23-21ksoftirqd/122:19:541
368804899547,8cyclictest23-21ksoftirqd/100:25:461
368804899546,7cyclictest23-21ksoftirqd/122:48:581
368804899546,7cyclictest23-21ksoftirqd/122:48:571
368804899546,7cyclictest23-21ksoftirqd/120:35:021
368804899546,10cyclictest23-21ksoftirqd/123:20:381
368804899545,10cyclictest23-21ksoftirqd/122:50:111
368804899544,9cyclictest23-21ksoftirqd/123:50:261
368804899541,6cyclictest0-21swapper/121:45:021
3688048995414,12cyclictest23-21ksoftirqd/123:40:001
3688048995412,11cyclictest23-21ksoftirqd/100:04:391
368804399545,3cyclictest131rcu_preempt22:49:420
368804399545,3cyclictest131rcu_preempt22:49:420
3688043995437,13cyclictest1-21systemd22:30:010
368804399543,3cyclictest131rcu_preempt21:25:150
3688043995432,19cyclictest12-21ksoftirqd/022:03:390
3688043995419,8cyclictest131rcu_preempt22:24:080
3688043995412,7cyclictest12-21ksoftirqd/022:10:020
368805799538,5cyclictest131rcu_preempt21:44:123
368805799537,6cyclictest131rcu_preempt00:30:033
368805799533,9cyclictest37-21ksoftirqd/321:25:013
3688057995323,19cyclictest37-21ksoftirqd/322:44:273
3688057995312,6cyclictest131rcu_preempt21:55:163
368805299539,6cyclictest131rcu_preempt20:50:012
368805299536,8cyclictest30-21ksoftirqd/223:25:032
368805299533,9cyclictest30-21ksoftirqd/200:18:342
3688052995315,11cyclictest30-21ksoftirqd/219:20:002
3688052995315,11cyclictest30-21ksoftirqd/219:20:002
3688052995313,6cyclictest131rcu_preempt00:08:182
368804899537,8cyclictest23-21ksoftirqd/120:45:021
368804899537,8cyclictest23-21ksoftirqd/100:39:301
368804899536,8cyclictest23-21ksoftirqd/100:33:561
368804899535,9cyclictest23-21ksoftirqd/123:07:191
368804899535,8cyclictest23-21ksoftirqd/122:22:271
368804899535,10cyclictest23-21ksoftirqd/122:28:571
368804899534,14cyclictest23-21ksoftirqd/121:11:331
368804899534,11cyclictest23-21ksoftirqd/119:55:011
368804899531,5cyclictest3725205-21ssh22:14:451
368804899531,5cyclictest3725205-21ssh22:14:441
3688048995315,9cyclictest23-21ksoftirqd/122:32:361
3688048995313,6cyclictest131rcu_preempt00:18:311
3688048995311,10cyclictest23-21ksoftirqd/120:20:031
368804399534,4cyclictest131rcu_preempt21:43:360
3688043995325,12cyclictest3792080-21ssh00:34:280
368805799529,6cyclictest131rcu_preempt22:10:013
3688057995234,14cyclictest37-21ksoftirqd/300:10:003
3688057995225,11cyclictest37-21ksoftirqd/322:57:513
3688057995215,6cyclictest131rcu_preempt20:00:003
3688057995210,5cyclictest131rcu_preempt00:20:013
368805299528,9cyclictest30-21ksoftirqd/219:15:012
368805299527,9cyclictest30-21ksoftirqd/220:30:022
3688052995212,11cyclictest30-21ksoftirqd/219:45:032
368804899526,8cyclictest23-21ksoftirqd/122:40:021
368804899525,7cyclictest23-21ksoftirqd/121:26:041
368804899523,14cyclictest23-21ksoftirqd/121:20:181
3688048995213,7cyclictest23-21ksoftirqd/123:01:211
3688048995210,8cyclictest23-21ksoftirqd/123:31:361
3688043995231,7cyclictest12-21ksoftirqd/023:31:360
3688043995229,8cyclictest3705054-21ssh21:32:160
3688043995229,8cyclictest3705054-21ssh21:32:150
3688043995219,3cyclictest131rcu_preempt23:24:070
368805799517,6cyclictest131rcu_preempt20:30:023
368805799516,6cyclictest37-21ksoftirqd/300:00:533
3688057995115,12cyclictest37-21ksoftirqd/320:35:003
3688057995113,6cyclictest131rcu_preempt21:12:513
3688057995112,6cyclictest131rcu_preempt23:45:023
368805799511,11cyclictest37-21ksoftirqd/320:55:013
3688057995110,6cyclictest131rcu_preempt23:28:293
3688057995110,6cyclictest131rcu_preempt00:39:583
368805299517,8cyclictest30-21ksoftirqd/219:30:012
3688052995143,5cyclictest3691381-21kworker/2:221:05:022
3688052995141,6cyclictest3674994-21kworker/2:119:56:492
3688052995136,7cyclictest3688252-21kworker/2:219:23:542
3688052995113,5cyclictest131rcu_preempt20:10:012
368804899517,6cyclictest23-21ksoftirqd/121:36:241
368804899515,8cyclictest23-21ksoftirqd/119:35:021
368804899515,10cyclictest23-21ksoftirqd/122:57:391
3688048995110,6cyclictest131rcu_preempt19:48:461
368804399514,27cyclictest131rcu_preempt21:48:460
3688043995125,11cyclictest3760276-21ssh23:27:570
368805799506,8cyclictest37-21ksoftirqd/320:40:003
368805799504,6cyclictest131rcu_preempt21:36:583
3688057995025,17cyclictest37-21ksoftirqd/321:19:533
3688057995021,20cyclictest37-21ksoftirqd/321:05:013
3688057995017,8cyclictest37-21ksoftirqd/300:35:013
3688057995013,27cyclictest37-21ksoftirqd/319:21:433
3688057995011,6cyclictest131rcu_preempt22:20:093
3688057995011,5cyclictest131rcu_preempt23:02:003
3688052995011,5cyclictest131rcu_preempt20:20:022
368804899504,11cyclictest23-21ksoftirqd/121:10:021
368804899504,11cyclictest23-21ksoftirqd/121:10:011
368804399508,37cyclictest3751585-21kworker/0:223:10:150
368804399502,3cyclictest131rcu_preempt22:14:490
368804399502,3cyclictest131rcu_preempt22:14:480
368805799498,6cyclictest131rcu_preempt22:51:003
368805799493,6cyclictest131rcu_preempt21:45:053
3688057994920,17cyclictest37-21ksoftirqd/320:05:013
3688057994911,6cyclictest131rcu_preempt20:50:013
3688057994910,6cyclictest131rcu_preempt22:36:573
3688057994910,6cyclictest131rcu_preempt20:10:013
3688057994910,6cyclictest131rcu_preempt19:15:023
3688057994910,6cyclictest131rcu_preempt19:15:023
3688057994910,5cyclictest131rcu_preempt00:00:013
368804899499,11cyclictest23-21ksoftirqd/120:00:021
368804899496,9cyclictest23-21ksoftirqd/121:00:021
368804899496,9cyclictest23-21ksoftirqd/120:25:021
368804899496,6cyclictest23-21ksoftirqd/120:30:011
368804899492,10cyclictest23-21ksoftirqd/120:10:461
3688043994939,5cyclictest3690572-21kworker/u8:120:00:010
3688043994936,9cyclictest16950irq/224-800000000:15:010
368804399493,39cyclictest3779037-21ssh00:07:020
368805799482,5cyclictest131rcu_preempt23:10:013
3688057994818,9cyclictest37-21ksoftirqd/322:35:033
3688052994840,5cyclictest3691381-21kworker/2:220:44:272
3688052994839,6cyclictest3691381-21kworker/2:220:50:382
368804899489,7cyclictest23-21ksoftirqd/119:40:031
368804899487,10cyclictest23-21ksoftirqd/119:30:021
3688043994821,12cyclictest3767815-21ssh23:43:460
3688057994727,10cyclictest37-21ksoftirqd/320:15:523
3688052994739,5cyclictest3690034-21kworker/2:019:48:592
368804899477,8cyclictest23-21ksoftirqd/120:50:001
368804899472,13cyclictest23-21ksoftirqd/119:20:021
368804899472,13cyclictest23-21ksoftirqd/119:20:021
368804899471,9cyclictest0-21swapper/120:51:021
368804899471,8cyclictest0-21swapper/120:08:441
368804399473,3cyclictest131rcu_preempt19:20:010
368804399473,3cyclictest131rcu_preempt19:20:000
368805799464,8cyclictest37-21ksoftirqd/319:40:003
368805799464,6cyclictest3690433-21cut19:55:023
368805799462,13cyclictest37-21ksoftirqd/321:00:033
368805299467,10cyclictest30-21ksoftirqd/219:10:012
368804899462,9cyclictest3692690-21systemd-run20:40:001
368804399465,28cyclictest151rcuc/019:55:020
3688043994629,13cyclictest1-21systemd20:30:010
368804399462,12cyclictest3689546-21modprobe19:40:000
368804399461,4cyclictest131rcu_preempt23:45:480
3688048994536,6cyclictest23-21ksoftirqd/119:43:471
3688043994533,8cyclictest12-21ksoftirqd/022:35:010
3688043994533,6cyclictest309-21in:imuxsock20:55:010
3688057994426,6cyclictest37-21ksoftirqd/320:11:093
3688057994411,5cyclictest131rcu_preempt21:10:023
3688057994411,5cyclictest131rcu_preempt21:10:023
368804899448,6cyclictest23-21ksoftirqd/119:10:011
368804899445,9cyclictest23-21ksoftirqd/119:15:011
368804899441,10cyclictest3694000-21latency_hist21:05:011
368805799439,5cyclictest131rcu_preempt20:45:013
368804899437,7cyclictest23-21ksoftirqd/120:05:011
368804399433,7cyclictest3692415-21unin-run20:35:010
3688043994327,11cyclictest12-21ksoftirqd/020:20:010
368804399431,4cyclictest131rcu_preempt19:25:010
368804399431,3cyclictest131rcu_preempt19:15:020
368805799425,6cyclictest37-21ksoftirqd/319:15:023
368805799424,5cyclictest131rcu_preempt19:50:023
3688057994223,8cyclictest37-21ksoftirqd/319:30:013
3688043994233,5cyclictest16950irq/224-800000020:15:010
3688043994225,12cyclictest3467602-21systemd-journal20:40:000
368804399413,6cyclictest131rcu_preempt19:10:020
3688043994128,7cyclictest309-21in:imuxsock21:04:590
3688043994110,9cyclictest12-21ksoftirqd/019:34:270
368805799408,20cyclictest37-21ksoftirqd/319:30:413
368805799404,9cyclictest37-21ksoftirqd/319:10:013
368804399402,9cyclictest3693262-21latency_hist20:50:010
368804399392,14cyclictest3690086-21munin-run19:50:010
368804399391,3cyclictest131rcu_preempt20:06:080
3688043993824,9cyclictest309-21in:imuxsock20:05:000
368804399353,3cyclictest131rcu_preempt19:30:000
368804399331,29cyclictest0-21swapper/020:40:400
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional