You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-05 - 12:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3s.osadl.org (updated Fri Jun 05, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
333720899902,73cyclictest3437635-21ssh00:40:021
333720899735,9cyclictest131rcu_preempt20:15:011
3337203996953,13cyclictest12-21ksoftirqd/022:00:020
3337203996935,30cyclictest12-21ksoftirqd/023:16:440
3337203996919,41cyclictest12-21ksoftirqd/023:08:080
333720399679,37cyclictest131rcu_preempt22:15:020
3337203996618,29cyclictest12-21ksoftirqd/022:37:380
3337203996618,29cyclictest12-21ksoftirqd/022:37:370
3337203996518,43cyclictest12-21ksoftirqd/022:28:350
3337203996412,29cyclictest12-21ksoftirqd/023:05:020
3337203996313,46cyclictest12-21ksoftirqd/021:41:090
3337203996313,46cyclictest12-21ksoftirqd/021:41:090
333721399627,25cyclictest30-21ksoftirqd/220:45:032
333721399626,9cyclictest30-21ksoftirqd/221:43:532
333721399626,9cyclictest30-21ksoftirqd/221:43:532
3337213996241,17cyclictest3467602-21systemd-journal22:47:372
3337213996219,9cyclictest30-21ksoftirqd/223:19:102
3337203996218,4cyclictest12-21ksoftirqd/023:39:260
3337203996218,4cyclictest12-21ksoftirqd/023:39:260
333721399613,11cyclictest30-21ksoftirqd/221:30:012
333721399611,8cyclictest3419615-40ogrotate23:59:592
333721399607,14cyclictest30-21ksoftirqd/223:25:022
3337213996020,13cyclictest30-21ksoftirqd/222:30:002
3337213996014,10cyclictest30-21ksoftirqd/200:17:432
3337213996014,10cyclictest30-21ksoftirqd/200:17:422
3337203996040,15cyclictest275-21dbus-daemon20:40:000
3337203996019,25cyclictest12-21ksoftirqd/023:22:390
3337203996018,38cyclictest12-21ksoftirqd/022:22:150
3337203996016,26cyclictest12-21ksoftirqd/021:12:510
3337203996015,40cyclictest12-21ksoftirqd/000:30:020
3337203996014,38cyclictest12-21ksoftirqd/023:59:140
333720399599,6cyclictest131rcu_preempt21:49:040
333720399599,6cyclictest131rcu_preempt21:49:030
3337203995925,30cyclictest309-21in:imuxsock21:25:010
3337213995810,6cyclictest131rcu_preempt23:51:062
3337203995819,30cyclictest16850irq/134-800000023:32:500
3337203995819,30cyclictest16850irq/134-800000021:33:550
3337203995812,41cyclictest12-21ksoftirqd/023:43:590
3337213995745,8cyclictest291rcuc/223:27:092
333721399571,16cyclictest30-21ksoftirqd/220:20:002
3337203995751,3cyclictest16950irq/224-800000023:15:030
3337203995751,3cyclictest16950irq/224-800000023:15:020
333721399567,7cyclictest30-21ksoftirqd/200:13:202
333721399565,10cyclictest30-21ksoftirqd/222:40:292
3337213995647,5cyclictest30-21ksoftirqd/220:30:012
3337213995616,12cyclictest30-21ksoftirqd/220:05:012
3337213995615,9cyclictest30-21ksoftirqd/221:30:172
333720399564,26cyclictest131rcu_preempt00:15:120
333720399564,26cyclictest131rcu_preempt00:15:120
333720399563,33cyclictest131rcu_preempt00:01:150
3337203995619,6cyclictest16850irq/134-800000022:31:220
3337203995618,6cyclictest16850irq/134-800000022:57:150
3337203995616,21cyclictest12-21ksoftirqd/020:15:020
3337203995614,19cyclictest12-21ksoftirqd/019:20:020
333720399561,31cyclictest131rcu_preempt21:40:020
3337215995539,12cyclictest3467602-21systemd-journal23:36:373
3337215995539,12cyclictest3467602-21systemd-journal23:36:363
3337215995536,15cyclictest1-21systemd23:00:003
333721399558,6cyclictest30-21ksoftirqd/223:45:552
333721399557,11cyclictest30-21ksoftirqd/200:05:032
333721399556,8cyclictest30-21ksoftirqd/200:23:262
3337213995546,5cyclictest3330870-21kworker/2:019:30:022
3337213995526,10cyclictest30-21ksoftirqd/200:33:252
3337213995520,8cyclictest30-21ksoftirqd/223:37:292
3337213995520,8cyclictest30-21ksoftirqd/223:37:292
3337203995539,13cyclictest12-21ksoftirqd/000:08:530
3337203995518,18cyclictest16850irq/134-800000022:03:190
3337203995518,17cyclictest16850irq/134-800000023:54:040
3337203995518,17cyclictest16850irq/134-800000000:30:200
3337203995517,6cyclictest16850irq/134-800000000:12:100
3337203995517,18cyclictest16850irq/134-800000022:44:270
333721599542,6cyclictest131rcu_preempt23:25:013
333721399548,11cyclictest30-21ksoftirqd/221:59:062
333721399548,10cyclictest30-21ksoftirqd/221:47:032
333721399548,10cyclictest30-21ksoftirqd/221:47:022
333721399547,9cyclictest30-21ksoftirqd/223:03:462
333721399547,9cyclictest30-21ksoftirqd/223:03:462
333721399547,11cyclictest30-21ksoftirqd/221:25:012
333721399547,10cyclictest30-21ksoftirqd/221:15:052
333721399545,8cyclictest30-21ksoftirqd/223:42:522
333721399545,11cyclictest30-21ksoftirqd/222:15:282
333721399544,11cyclictest30-21ksoftirqd/221:38:202
333721399543,9cyclictest30-21ksoftirqd/222:50:042
3337203995418,17cyclictest16850irq/134-800000021:28:000
3337203995418,17cyclictest16850irq/134-800000000:23:260
3337203995415,6cyclictest16850irq/134-800000021:52:280
333721399537,7cyclictest30-21ksoftirqd/219:45:022
333721399537,11cyclictest30-21ksoftirqd/200:26:212
333721399536,14cyclictest30-21ksoftirqd/222:58:162
3337213995316,10cyclictest30-21ksoftirqd/222:05:532
333721399531,12cyclictest3467602-21systemd-journal22:20:182
3337208995331,15cyclictest3467602-21systemd-journal22:44:591
3337203995332,14cyclictest275-21dbus-daemon22:10:010
3337203995317,17cyclictest16850irq/134-800000022:46:180
3337203995317,16cyclictest16850irq/134-800000023:28:010
3337203995310,38cyclictest3374789-21sh22:19:590
333721399527,9cyclictest30-21ksoftirqd/223:31:522
333721399527,9cyclictest30-21ksoftirqd/200:39:212
333721399525,7cyclictest30-21ksoftirqd/220:00:012
333721399524,9cyclictest30-21ksoftirqd/221:51:212
3337213995214,9cyclictest30-21ksoftirqd/222:36:102
3337213995214,9cyclictest30-21ksoftirqd/222:36:102
333721399521,14cyclictest30-21ksoftirqd/222:10:502
3337213995210,11cyclictest30-21ksoftirqd/223:10:022
3337203995214,18cyclictest16850irq/134-800000000:39:340
3337215995137,10cyclictest1-21systemd00:00:013
333721399516,11cyclictest30-21ksoftirqd/219:40:002
3337213995143,5cyclictest3338202-21kworker/2:120:31:382
333721399514,14cyclictest30-21ksoftirqd/200:10:002
333721399514,10cyclictest30-21ksoftirqd/222:02:052
3337213995140,6cyclictest3363584-21kworker/2:023:15:012
3337213995140,6cyclictest3363584-21kworker/2:023:15:012
333721399513,13cyclictest30-21ksoftirqd/220:10:012
3337213995124,17cyclictest275-21dbus-daemon21:15:002
3337213995112,10cyclictest30-21ksoftirqd/219:15:022
3337213995112,10cyclictest30-21ksoftirqd/219:15:012
3337213995110,10cyclictest30-21ksoftirqd/220:25:022
3337203995132,5cyclictest16850irq/134-800000023:02:280
3337203995131,16cyclictest3347274-21ssh21:18:350
3337203995122,5cyclictest131rcu_preempt23:47:450
3337203995114,17cyclictest16850irq/134-800000022:54:100
3337215995023,13cyclictest275-21dbus-daemon19:25:003
3337213995042,5cyclictest3338202-21kworker/2:120:55:172
333721399501,7cyclictest3381507-21ssh22:35:012
3337215994931,13cyclictest1-21systemd23:55:013
333721399496,11cyclictest30-21ksoftirqd/220:50:002
3337213994941,5cyclictest3338202-21kworker/2:119:52:032
333721399494,11cyclictest30-21ksoftirqd/219:25:032
333720899493,4cyclictest131rcu_preempt23:10:001
3337208994915,25cyclictest0-21swapper/120:04:571
333720399499,5cyclictest131rcu_preempt20:30:030
3337203994913,7cyclictest131rcu_preempt19:15:010
3337203994913,7cyclictest131rcu_preempt19:15:000
3337215994835,9cyclictest1-21systemd21:00:013
3337215994834,11cyclictest1-21systemd23:10:003
3337213994840,5cyclictest3338202-21kworker/2:119:49:432
333720399486,6cyclictest131rcu_preempt20:45:030
3337215994727,15cyclictest3467602-21systemd-journal19:55:013
333721599471,5cyclictest131rcu_preempt22:04:063
333721399474,6cyclictest30-21ksoftirqd/221:10:032
3337213994739,5cyclictest3338202-21kworker/2:120:36:162
3337213994739,5cyclictest3336577-21kworker/2:319:17:332
3337213994739,5cyclictest3330870-21kworker/2:021:01:302
3337213994739,5cyclictest3330870-21kworker/2:020:11:002
3337213994739,5cyclictest3330870-21kworker/2:019:30:232
333721399473,6cyclictest30-21ksoftirqd/219:10:012
333720399471,12cyclictest3467602-21systemd-journal20:00:000
3337215994634,10cyclictest37-21ksoftirqd/320:35:033
333721599463,3cyclictest131rcu_preempt21:40:593
333721599463,3cyclictest131rcu_preempt21:40:583
3337213994639,4cyclictest3338202-21kworker/2:120:53:172
333721599455,3cyclictest131rcu_preempt21:22:133
333721599452,4cyclictest131rcu_preempt22:10:203
333721599452,3cyclictest131rcu_preempt22:49:433
3337208994526,13cyclictest3467602-21systemd-journal23:34:591
3337203994530,9cyclictest16950irq/224-800000021:10:000
3337215994428,12cyclictest37-21ksoftirqd/323:40:153
3337215994422,17cyclictest275-21dbus-daemon20:50:013
333720899447,20cyclictest131rcu_preempt23:55:021
333720899444,3cyclictest131rcu_preempt22:52:051
3337208994431,10cyclictest23-21ksoftirqd/122:50:001
333720899441,3cyclictest131rcu_preempt22:39:231
333720899441,3cyclictest131rcu_preempt22:39:221
3337203994436,5cyclictest16950irq/224-800000020:02:280
3337203994417,22cyclictest12-21ksoftirqd/020:20:000
333721599435,7cyclictest37-21ksoftirqd/300:07:593
3337215994330,9cyclictest3356834-21cron21:39:593
3337215994328,11cyclictest37-21ksoftirqd/321:16:233
3337215994318,4cyclictest37-21ksoftirqd/322:20:023
333721599431,3cyclictest131rcu_preempt22:06:013
333721599431,3cyclictest131rcu_preempt21:25:383
3337215994312,8cyclictest131rcu_preempt20:00:003
333720899436,7cyclictest150-21pr/ttyS223:45:011
333720899433,3cyclictest131rcu_preempt23:16:491
333720899432,6cyclictest131rcu_preempt21:54:571
3337215994232,7cyclictest3337352-21sed19:10:023
3337215994230,8cyclictest37-21ksoftirqd/321:15:023
3337215994227,11cyclictest37-21ksoftirqd/300:38:083
333721599422,11cyclictest1-21systemd00:35:013
333721599421,7cyclictest131rcu_preempt22:40:013
333721599421,7cyclictest131rcu_preempt22:40:013
333721599421,3cyclictest131rcu_preempt00:15:033
3337215994212,5cyclictest131rcu_preempt22:25:003
333720899422,4cyclictest131rcu_preempt00:06:181
333720899421,3cyclictest131rcu_preempt00:26:501
333720399425,6cyclictest131rcu_preempt19:40:010
333721599416,6cyclictest131rcu_preempt22:40:283
3337215994125,12cyclictest3467602-21systemd-journal19:19:593
333721599412,3cyclictest131rcu_preempt21:52:373
3337215994120,15cyclictest275-21dbus-daemon20:45:013
3337215994110,5cyclictest131rcu_preempt23:26:283
3337208994131,7cyclictest23-21ksoftirqd/122:15:021
3337208994128,10cyclictest23-21ksoftirqd/121:15:151
3337208994128,10cyclictest23-21ksoftirqd/120:35:021
3337208994127,11cyclictest3467602-21systemd-journal23:40:001
3337208994127,11cyclictest3467602-21systemd-journal23:40:001
333720899411,3cyclictest131rcu_preempt23:01:041
333720899411,3cyclictest131rcu_preempt23:01:041
333720399417,28cyclictest0-21swapper/020:49:590
3337203994126,10cyclictest3342857-21systemctl21:00:000
333720399411,3cyclictest131rcu_preempt20:33:280
333721599405,29cyclictest3341312-21true20:30:003
333721599403,3cyclictest131rcu_preempt22:25:313
3337215994028,8cyclictest275-21dbus-daemon21:10:013
3337215994024,12cyclictest1-21systemd22:00:023
333721599402,3cyclictest131rcu_preempt23:17:393
3337215994019,15cyclictest275-21dbus-daemon20:15:003
333721599401,6cyclictest131rcu_preempt23:50:003
333721599401,4cyclictest131rcu_preempt00:29:293
3337215994014,3cyclictest37-21ksoftirqd/323:02:093
3337215994014,3cyclictest37-21ksoftirqd/323:02:093
3337208994030,8cyclictest23-21ksoftirqd/123:50:011
3337208994027,10cyclictest23-21ksoftirqd/100:20:031
333720899401,4cyclictest131rcu_preempt21:32:081
333720399402,5cyclictest275-21dbus-daemon19:30:010
3337203994021,15cyclictest0-21swapper/019:20:280
333721599396,7cyclictest37-21ksoftirqd/322:34:473
3337215993927,10cyclictest37-21ksoftirqd/321:33:053
3337215993926,9cyclictest3341783-21sshd20:37:253
3337215993924,12cyclictest3467602-21systemd-journal00:20:013
3337215993924,12cyclictest3467602-21systemd-journal00:20:003
3337215993924,11cyclictest3340250-21cron20:09:593
3337215993923,11cyclictest275-21dbus-daemon21:05:013
3337215993920,13cyclictest275-21dbus-daemon20:20:023
333721599391,3cyclictest131rcu_preempt21:45:233
333721599391,3cyclictest131rcu_preempt21:45:223
333720899396,7cyclictest23-21ksoftirqd/100:19:521
333720899396,7cyclictest23-21ksoftirqd/100:19:521
333720899394,3cyclictest131rcu_preempt19:35:011
333720899394,24cyclictest221rcuc/120:55:011
3337208993925,10cyclictest275-21dbus-daemon21:00:001
333720899392,3cyclictest131rcu_preempt21:40:241
333720899392,3cyclictest131rcu_preempt21:40:231
333720899392,3cyclictest131rcu_preempt19:30:031
333720899391,4cyclictest131rcu_preempt22:30:021
333720899391,3cyclictest131rcu_preempt23:12:051
333720899391,3cyclictest131rcu_preempt23:12:041
333720899391,3cyclictest131rcu_preempt22:10:021
333720399392,7cyclictest3339477-21munin-run19:55:000
3337203993926,9cyclictest3467602-21systemd-journal19:31:280
3337203993926,10cyclictest3467602-21systemd-journal20:05:280
3337215993828,7cyclictest37-21ksoftirqd/319:15:033
3337215993828,7cyclictest37-21ksoftirqd/319:15:023
3337215993827,9cyclictest37-21ksoftirqd/323:32:223
3337215993825,9cyclictest37-21ksoftirqd/300:03:173
333721599382,3cyclictest131rcu_preempt22:52:343
3337215993822,11cyclictest275-21dbus-daemon19:30:023
333720899384,3cyclictest131rcu_preempt22:57:491
3337208993830,5cyclictest3341523-21kworker/u8:221:15:001
333720899382,8cyclictest0-21swapper/100:32:461
3337208993828,6cyclictest23-21ksoftirqd/100:00:191
3337208993828,5cyclictest3356820-21cron21:39:591
3337208993827,9cyclictest23-21ksoftirqd/123:24:141
3337208993827,8cyclictest23-21ksoftirqd/122:02:091
333720899381,3cyclictest131rcu_preempt22:18:551
3337215993728,5cyclictest3397575-21sh23:10:093
3337215993728,5cyclictest3397575-21sh23:10:083
3337215993724,11cyclictest37-21ksoftirqd/300:20:533
3337215993722,12cyclictest275-21dbus-daemon19:35:013
333720899379,6cyclictest131rcu_preempt20:00:001
333720899379,5cyclictest131rcu_preempt21:27:551
3337208993728,6cyclictest3381715-21cat22:35:021
3337208993728,5cyclictest309-21in:imuxsock21:10:001
3337208993726,8cyclictest3404850-21sshd23:26:291
3337208993726,8cyclictest23-21ksoftirqd/121:58:471
3337208993726,8cyclictest23-21ksoftirqd/100:11:191
3337208993725,8cyclictest3361501-21sed21:50:041
3337208993725,8cyclictest3361501-21sed21:50:031
3337208993720,13cyclictest275-21dbus-daemon22:25:011
333720399372,6cyclictest3467602-21systemd-journal21:01:280
3337203993710,6cyclictest131rcu_preempt19:50:040
3337215993624,9cyclictest3340107-21wc20:05:033
3337215993623,10cyclictest3338682-21cron19:40:023
3337215993622,10cyclictest37-21ksoftirqd/319:35:033
333720899368,5cyclictest131rcu_preempt20:07:211
333720899363,3cyclictest131rcu_preempt19:40:021
3337208993613,15cyclictest23-21ksoftirqd/121:21:241
3337203993626,7cyclictest19350irq/34-mmc120:55:000
3337203993618,5cyclictest131rcu_preempt20:25:010
3337215993525,7cyclictest3341083-21sed20:25:013
3337208993527,5cyclictest3340778-21sendmail20:20:011
3337208993521,10cyclictest23-21ksoftirqd/119:44:541
3337208993514,16cyclictest275-21dbus-daemon23:55:001
3337208993427,4cyclictest3337646-21cron19:20:001
3337208993426,5cyclictest3341304-21munin-plugin-st20:30:001
3337208993422,9cyclictest23-21ksoftirqd/120:45:021
3337208993416,14cyclictest275-21dbus-daemon19:55:001
333720399341,5cyclictest131rcu_preempt19:45:010
333721599332,5cyclictest131rcu_preempt19:50:003
3337208993324,7cyclictest23-21ksoftirqd/120:50:011
333720899331,3cyclictest131rcu_preempt19:15:031
333720899331,3cyclictest131rcu_preempt19:15:021
3337215993220,10cyclictest37-21ksoftirqd/320:55:003
333720899323,3cyclictest131rcu_preempt19:50:031
3337208993220,9cyclictest3343193-21sed21:05:021
3337208993217,11cyclictest3341566-21sh20:35:021
3337208993120,7cyclictest309-21in:imuxsock19:23:281
3337208993022,5cyclictest3341151-21latency_hist20:25:021
3337208993018,9cyclictest3337271-21cat19:10:021
333720399306,15cyclictest3337195-21cyclictest19:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional