You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-06 - 11:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Apr 06, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
109308499882,81cyclictest1151922-21ssh23:13:030
109308499882,81cyclictest1151922-21ssh23:13:030
1093084998424,12cyclictest1134616-21ssh22:32:230
1093084998421,14cyclictest1110618-21ssh21:36:020
1093084998126,12cyclictest1118950-21ssh21:55:450
1093084998126,12cyclictest1118950-21ssh21:55:450
1093093998032,43cyclictest3467602-21systemd-journal00:35:002
1093084998030,46cyclictest16850irq/134-800000023:27:200
109308499791,72cyclictest1146120-21ssh22:59:320
1093084997915,59cyclictest16850irq/134-800000000:15:540
1093084997912,57cyclictest16850irq/134-800000000:37:080
1093084997912,57cyclictest16850irq/134-800000000:37:070
109308499781,6cyclictest1139954-21munin-run22:45:010
109308499781,36cyclictest1108341-21ssh21:30:350
1093084997811,61cyclictest16850irq/134-800000022:12:000
109308499779,5cyclictest16850irq/134-800000021:41:260
1093084997725,2cyclictest0-21swapper/022:28:440
1093084997711,61cyclictest16850irq/134-800000023:16:110
1093084997670,4cyclictest0-21swapper/019:15:020
1093084997519,15cyclictest1172870-21ssh00:02:170
1093084997411,53cyclictest16850irq/134-800000000:10:410
1093084997321,48cyclictest1159765-21rm23:31:280
1093084997224,14cyclictest1141916-21ssh22:49:430
109308499721,53cyclictest1185204-21ssh00:31:390
1093093997124,13cyclictest275-21dbus-daemon23:20:002
1093084997120,13cyclictest1149522-21ssh23:07:150
1093084997116,49cyclictest16850irq/134-800000022:05:390
1093084997030,35cyclictest16950irq/224-800000023:54:230
109308499702,7cyclictest1171744-21munin-run00:00:000
109308499702,7cyclictest1171744-21munin-run00:00:000
1093084997024,10cyclictest1137576-21ssh22:39:260
1093084997023,13cyclictest1143435-21ssh22:53:110
1093084997020,12cyclictest1164178-21ssh23:41:450
109308499701,9cyclictest3467602-21systemd-journal22:01:470
1093084997019,8cyclictest1128014-21ssh22:16:560
1093084996927,38cyclictest1106069-21sh21:25:230
1093084996927,26cyclictest1116714-21ssh21:50:290
1093084996919,12cyclictest1147508-21ssh23:02:350
1093084996916,48cyclictest16850irq/134-800000023:24:270
1093084996916,48cyclictest16850irq/134-800000023:24:270
1093084996914,11cyclictest1181950-21ssh00:24:180
1093084996826,5cyclictest0-21swapper/023:35:570
1093093996718,12cyclictest1-21systemd21:30:012
109308499671,57cyclictest16850irq/134-800000021:19:510
1093084996714,43cyclictest16850irq/134-800000021:24:030
1093093996626,11cyclictest3467602-21systemd-journal21:50:032
1093084996632,30cyclictest16850irq/134-800000022:25:000
1093084996530,32cyclictest16950irq/224-800000020:40:010
1093084996527,14cyclictest1175853-21ssh00:09:470
109308499652,14cyclictest1183633-21sh00:28:060
1093084996326,6cyclictest1166497-21sh23:47:180
1093084996214,12cyclictest1116021-21ssh21:49:080
1093093996115,5cyclictest0-21swapper/222:55:482
109308499612,29cyclictest1099582-21ssh21:10:130
1093093995927,5cyclictest1137753-21sendmail22:40:002
1093093995923,5cyclictest1098213-21cron20:50:002
1093084995913,15cyclictest275-21dbus-daemon19:50:010
1093084995913,15cyclictest275-21dbus-daemon19:50:010
1093093995825,8cyclictest30-21ksoftirqd/223:25:512
109308499583,50cyclictest0-21swapper/019:55:020
1093093995723,10cyclictest30-21ksoftirqd/200:09:542
1093093995647,5cyclictest1095648-21kworker/2:222:15:032
1093084995627,5cyclictest0-21swapper/019:33:340
1093093995523,6cyclictest1133667-21latency_hist22:30:022
1093093995521,11cyclictest30-21ksoftirqd/221:05:012
1093084995515,10cyclictest275-21dbus-daemon20:35:000
1093098995434,14cyclictest3467602-21systemd-journal22:40:003
109309399545,4cyclictest131rcu_preempt00:20:022
1093093995446,5cyclictest1078174-21kworker/2:320:25:012
1093093995445,5cyclictest1095648-21kworker/2:223:31:282
1093089995434,15cyclictest275-21dbus-daemon23:40:011
1093084995425,4cyclictest1099267-21munin-plugin-st21:10:000
1093084995425,4cyclictest1099267-21munin-plugin-st21:10:000
109308499542,46cyclictest1093085-21latency_hist19:10:020
109308499542,46cyclictest1093085-21latency_hist19:10:020
1093098995331,17cyclictest275-21dbus-daemon19:40:013
1093093995344,6cyclictest1098783-21kworker/2:122:31:292
1093093995328,6cyclictest30-21ksoftirqd/222:24:282
1093093995324,5cyclictest1164574-21sh23:42:542
1093089995311,6cyclictest131rcu_preempt00:00:001
1093089995311,6cyclictest131rcu_preempt00:00:001
1093098995228,17cyclictest275-21dbus-daemon22:20:013
1093098995226,6cyclictest37-21ksoftirqd/322:54:593
1093093995244,5cyclictest1095648-21kworker/2:221:30:382
1093093995242,7cyclictest1098783-21kworker/2:122:02:472
1093093995240,7cyclictest1098783-21kworker/2:122:49:342
1093093995224,12cyclictest1-21systemd22:10:012
1093084995215,8cyclictest12-21ksoftirqd/020:12:250
109309899515,3cyclictest131rcu_preempt23:35:023
109309899513,3cyclictest131rcu_preempt23:40:003
109309399516,3cyclictest131rcu_preempt21:15:022
1093093995142,6cyclictest1159050-21kworker/2:023:52:392
1093093995129,5cyclictest1176781-21sh00:11:422
109308999515,4cyclictest131rcu_preempt00:40:021
109308999515,4cyclictest131rcu_preempt00:40:011
1093084995125,2cyclictest0-21swapper/020:08:250
109308499511,4cyclictest0-21swapper/019:37:430
109308499511,4cyclictest0-21swapper/019:24:290
109309899504,3cyclictest131rcu_preempt23:53:363
1093098995035,11cyclictest275-21dbus-daemon23:10:003
1093098995013,26cyclictest1099254-21modprobe21:10:003
1093098995013,26cyclictest1099254-21modprobe21:10:003
1093093995042,5cyclictest1159050-21kworker/2:000:00:192
109308999507,4cyclictest131rcu_preempt23:22:461
109308999507,4cyclictest131rcu_preempt23:22:451
109308999504,5cyclictest131rcu_preempt22:26:561
109308999504,4cyclictest131rcu_preempt22:09:281
1093089995035,11cyclictest1-21systemd20:55:011
109308999501,3cyclictest131rcu_preempt22:50:561
1093084995035,11cyclictest275-21dbus-daemon20:20:000
1093084995017,10cyclictest12-21ksoftirqd/020:00:450
109309899495,4cyclictest131rcu_preempt23:14:063
109309899495,4cyclictest131rcu_preempt23:14:063
109309899495,3cyclictest131rcu_preempt21:24:213
109309899494,4cyclictest131rcu_preempt22:21:453
109309899494,3cyclictest131rcu_preempt23:23:233
109309899494,3cyclictest131rcu_preempt23:23:233
109309899494,3cyclictest131rcu_preempt23:03:063
109309899494,3cyclictest131rcu_preempt22:29:043
109309899494,3cyclictest131rcu_preempt22:05:013
1093098994927,15cyclictest275-21dbus-daemon00:35:013
1093098994920,6cyclictest37-21ksoftirqd/319:20:003
109309899491,5cyclictest131rcu_preempt21:10:413
109309899491,3cyclictest131rcu_preempt22:50:013
1093093994941,5cyclictest1095648-21kworker/2:223:45:242
1093093994940,6cyclictest1182310-21kworker/2:300:35:282
1093093994940,6cyclictest1182310-21kworker/2:300:35:282
1093093994934,11cyclictest1171736-21cron00:00:002
1093093994934,11cyclictest1171736-21cron00:00:002
109309399492,6cyclictest131rcu_preempt20:35:012
109308999497,31cyclictest1095604-21modprobe20:00:001
109308999497,31cyclictest1095604-21modprobe20:00:001
109308999495,3cyclictest131rcu_preempt23:16:501
109308999494,3cyclictest131rcu_preempt21:55:001
1093089994933,12cyclictest1-21systemd19:15:001
1093089994920,6cyclictest1106563-21sh21:26:321
109309899484,3cyclictest131rcu_preempt23:58:513
109309899484,3cyclictest131rcu_preempt23:58:503
109309899484,3cyclictest131rcu_preempt23:45:443
109309899484,3cyclictest131rcu_preempt23:20:023
109309899484,3cyclictest131rcu_preempt22:31:263
109309899483,3cyclictest131rcu_preempt21:27:023
1093098994833,12cyclictest1-21systemd23:30:003
1093098994825,16cyclictest275-21dbus-daemon21:55:003
109309899482,4cyclictest131rcu_preempt21:42:403
109309899482,3cyclictest131rcu_preempt21:40:003
1093093994840,5cyclictest1098783-21kworker/2:123:02:502
1093093994839,6cyclictest1095648-21kworker/2:223:13:272
1093093994839,6cyclictest1095648-21kworker/2:223:13:272
1093093994821,8cyclictest30-21ksoftirqd/221:39:042
109308999484,4cyclictest131rcu_preempt21:20:141
109308999482,4cyclictest131rcu_preempt23:31:321
109308999482,3cyclictest131rcu_preempt19:35:011
109308999482,3cyclictest131rcu_preempt00:25:021
1093089994820,5cyclictest23-21ksoftirqd/123:45:001
109308999481,3cyclictest131rcu_preempt21:40:451
109308499481,4cyclictest0-21swapper/020:47:350
109309899475,8cyclictest37-21ksoftirqd/322:11:083
109309899474,4cyclictest131rcu_preempt00:38:043
109309899474,4cyclictest131rcu_preempt00:38:043
109309899474,3cyclictest131rcu_preempt21:55:083
109309899474,3cyclictest131rcu_preempt21:55:083
109309899474,3cyclictest131rcu_preempt19:55:003
109309899474,3cyclictest131rcu_preempt00:22:413
109309899474,3cyclictest131rcu_preempt00:17:233
109309899472,4cyclictest131rcu_preempt00:00:083
109309899472,3cyclictest131rcu_preempt21:31:453
1093093994739,5cyclictest1095648-21kworker/2:223:07:162
1093093994738,6cyclictest1095648-21kworker/2:200:25:302
1093093994710,6cyclictest131rcu_preempt20:41:282
109308999473,3cyclictest131rcu_preempt21:58:171
109308999473,3cyclictest131rcu_preempt21:58:171
109308999473,3cyclictest131rcu_preempt00:12:031
1093089994726,16cyclictest275-21dbus-daemon19:40:011
109308999472,3cyclictest131rcu_preempt23:47:111
109308999472,3cyclictest131rcu_preempt22:58:061
109308999472,3cyclictest131rcu_preempt22:10:471
1093089994723,5cyclictest23-21ksoftirqd/121:20:001
109308999471,3cyclictest131rcu_preempt21:50:021
1093084994729,14cyclictest1-21systemd19:20:010
109309899464,4cyclictest131rcu_preempt19:30:003
109309899464,3cyclictest131rcu_preempt23:41:343
109309899464,3cyclictest131rcu_preempt22:40:123
1093098994626,15cyclictest275-21dbus-daemon20:40:013
109309899462,4cyclictest131rcu_preempt00:26:563
109309899462,3cyclictest131rcu_preempt23:00:033
1093093994638,5cyclictest1098783-21kworker/2:121:57:512
1093093994638,5cyclictest1098783-21kworker/2:121:57:502
1093093994637,5cyclictest1095648-21kworker/2:223:37:352
1093093994611,6cyclictest131rcu_preempt20:20:002
109308999466,3cyclictest131rcu_preempt00:20:011
109308999465,3cyclictest131rcu_preempt23:05:111
109308999464,3cyclictest131rcu_preempt22:48:541
109308999464,3cyclictest131rcu_preempt22:45:001
109308999464,3cyclictest131rcu_preempt00:06:081
109308999463,3cyclictest131rcu_preempt22:24:491
1093089994631,11cyclictest23-21ksoftirqd/121:35:331
1093089994628,15cyclictest23-21ksoftirqd/100:32:291
109308999462,5cyclictest131rcu_preempt20:50:001
109308999461,4cyclictest131rcu_preempt22:16:351
109308999461,4cyclictest131rcu_preempt00:01:121
109308999461,3cyclictest131rcu_preempt23:53:401
109308999461,3cyclictest131rcu_preempt22:32:021
1093089994611,23cyclictest1099256-21modprobe21:10:001
1093089994611,23cyclictest1099256-21modprobe21:10:001
109309899454,4cyclictest131rcu_preempt20:15:013
109309899454,3cyclictest131rcu_preempt21:45:313
109309899454,3cyclictest131rcu_preempt20:30:033
109309899454,3cyclictest131rcu_preempt19:45:013
109309899454,3cyclictest131rcu_preempt00:11:553
109309899453,3cyclictest131rcu_preempt22:08:433
109309899453,3cyclictest131rcu_preempt00:05:523
1093098994522,5cyclictest37-21ksoftirqd/321:00:003
1093098994510,5cyclictest131rcu_preempt20:35:013
1093093994538,5cyclictest1095648-21kworker/2:221:45:292
1093093994537,5cyclictest1078174-21kworker/2:319:50:022
1093093994537,5cyclictest1078174-21kworker/2:319:50:022
1093093994536,6cyclictest1083473-21kworker/2:019:40:012
109309399453,4cyclictest131rcu_preempt22:44:242
109308999454,4cyclictest131rcu_preempt23:11:041
109308999454,4cyclictest131rcu_preempt23:11:041
109308999454,3cyclictest131rcu_preempt22:38:111
1093089994530,12cyclictest1-21systemd19:25:011
109308999452,3cyclictest131rcu_preempt22:01:141
109308999452,3cyclictest131rcu_preempt21:10:241
109308999451,3cyclictest131rcu_preempt00:25:341
109309899443,3cyclictest131rcu_preempt21:16:113
109309899443,3cyclictest131rcu_preempt20:51:003
109309899443,3cyclictest131rcu_preempt20:10:023
109309899443,3cyclictest131rcu_preempt19:50:003
109309899443,3cyclictest131rcu_preempt19:50:003
1093098994427,11cyclictest309-21in:imuxsock20:45:033
109309899442,4cyclictest131rcu_preempt19:55:493
109309899442,4cyclictest131rcu_preempt19:55:493
109309899442,3cyclictest131rcu_preempt19:35:013
109309399444,4cyclictest131rcu_preempt22:55:012
1093093994436,5cyclictest1098783-21kworker/2:121:22:242
1093093994434,6cyclictest1095648-21kworker/2:222:18:372
109309399442,11cyclictest1094974-21sed19:45:022
1093093994411,5cyclictest131rcu_preempt21:00:012
109308999444,4cyclictest131rcu_preempt21:30:551
109308999443,3cyclictest131rcu_preempt20:30:011
109308499441,4cyclictest0-21swapper/019:43:080
109308499441,26cyclictest0-21swapper/020:25:060
1093098994332,8cyclictest3467602-21systemd-journal21:03:273
1093098994327,11cyclictest3467602-21systemd-journal20:05:003
109309399433,3cyclictest131rcu_preempt19:10:022
109309399433,3cyclictest131rcu_preempt19:10:022
1093093994332,8cyclictest3467602-21systemd-journal23:25:022
1093093994332,8cyclictest3467602-21systemd-journal23:25:012
1093093994331,9cyclictest30-21ksoftirqd/219:30:202
1093093994329,10cyclictest3467602-21systemd-journal19:14:592
1093093994328,11cyclictest3467602-21systemd-journal19:28:202
1093093994311,5cyclictest131rcu_preempt20:15:012
109308999434,3cyclictest131rcu_preempt23:03:371
109308999433,4cyclictest131rcu_preempt19:20:001
109308999433,3cyclictest131rcu_preempt21:05:011
109308999431,3cyclictest131rcu_preempt23:30:001
109308999431,3cyclictest131rcu_preempt19:55:011
109308999431,3cyclictest131rcu_preempt19:30:001
109309899423,3cyclictest131rcu_preempt20:25:023
1093093994233,5cyclictest1098783-21kworker/2:121:41:102
109309399421,24cyclictest0-21swapper/221:20:022
109308999424,3cyclictest131rcu_preempt20:25:001
109308999423,3cyclictest131rcu_preempt20:35:011
109308999421,4cyclictest131rcu_preempt19:45:011
109308999421,3cyclictest131rcu_preempt20:45:011
1093084994231,7cyclictest12-21ksoftirqd/020:00:030
1093084994231,7cyclictest12-21ksoftirqd/020:00:030
1093084994223,4cyclictest0-21swapper/021:01:050
109308499421,38cyclictest1098456-21sshd20:53:280
109309899414,3cyclictest131rcu_preempt20:15:253
109309399414,3cyclictest131rcu_preempt00:21:462
1093093994129,6cyclictest1097679-21cron20:40:002
109308999412,4cyclictest131rcu_preempt20:00:501
109308499411,4cyclictest0-21swapper/020:24:530
109309399404,3cyclictest131rcu_preempt19:50:032
1093093994026,11cyclictest1099247-21cron21:10:002
1093093994026,11cyclictest1099247-21cron21:10:002
1093093994021,10cyclictest3467602-21systemd-journal20:00:002
1093093994021,10cyclictest3467602-21systemd-journal20:00:002
109308999404,3cyclictest131rcu_preempt20:15:011
1093089994032,5cyclictest1096660-21latency_hist20:20:001
1093089994030,7cyclictest1098715-21cron21:00:001
109308999401,3cyclictest131rcu_preempt20:05:531
109309399392,5cyclictest131rcu_preempt19:25:022
109308999391,4cyclictest131rcu_preempt19:50:001
109308999391,4cyclictest131rcu_preempt19:50:001
109309899381,3cyclictest131rcu_preempt20:40:573
109308999381,3cyclictest131rcu_preempt20:40:011
109308499383,2cyclictest0-21swapper/021:00:010
1093084993823,11cyclictest1094081-21systemd-run19:30:000
1093098993725,9cyclictest37-21ksoftirqd/319:25:023
1093093993725,9cyclictest1098456-21sshd20:53:282
1093093993725,10cyclictest30-21ksoftirqd/220:30:022
1093093993723,11cyclictest30-21ksoftirqd/220:10:012
109309899364,3cyclictest131rcu_preempt19:10:223
1093093993624,8cyclictest1093746-21cat19:20:022
109309399352,4cyclictest131rcu_preempt20:05:022
1093084993423,7cyclictest12-21ksoftirqd/020:43:270
109309899321,3cyclictest131rcu_preempt19:10:023
109309899321,3cyclictest131rcu_preempt19:10:023
109308999321,3cyclictest131rcu_preempt19:10:021
109308999321,3cyclictest131rcu_preempt19:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional