You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-06 - 06:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Jul 06, 2025 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3289211991083,4cyclictest131rcu_preempt20:55:030
3289211991083,4cyclictest131rcu_preempt20:55:020
3289216999910,5cyclictest131rcu_preempt00:30:011
3289211998557,25cyclictest0-21swapper/022:25:030
328921899771,55cyclictest0-21swapper/220:30:022
328921899751,4cyclictest131rcu_preempt00:10:012
3289211997028,8cyclictest3305544-21cron00:25:010
3289216996526,13cyclictest23-21ksoftirqd/100:00:021
3289216996526,13cyclictest23-21ksoftirqd/100:00:011
3289221996412,8cyclictest131rcu_preempt21:50:003
3289211996424,11cyclictest272-21dbus-daemon23:25:020
3289211996424,11cyclictest272-21dbus-daemon23:25:010
3289216996221,27cyclictest23-21ksoftirqd/122:10:021
328922199618,9cyclictest37-21ksoftirqd/320:00:003
328921699617,5cyclictest131rcu_preempt23:55:021
328922199604,7cyclictest37-21ksoftirqd/300:00:033
328922199604,7cyclictest37-21ksoftirqd/300:00:023
3289221996014,6cyclictest131rcu_preempt22:15:023
3289218996043,13cyclictest3302152-21cron23:20:002
3289216996013,6cyclictest131rcu_preempt19:15:001
3289216996013,6cyclictest131rcu_preempt19:15:001
3289216996012,9cyclictest23-21ksoftirqd/121:40:031
3289216996012,9cyclictest23-21ksoftirqd/121:40:021
3289221995916,7cyclictest131rcu_preempt20:30:003
3289216995924,10cyclictest23-21ksoftirqd/120:20:001
3289216995915,9cyclictest131rcu_preempt20:39:591
3289221995814,6cyclictest131rcu_preempt22:20:023
328922199575,11cyclictest37-21ksoftirqd/320:03:203
3289221995612,5cyclictest131rcu_preempt22:00:013
3289221995611,5cyclictest131rcu_preempt23:15:003
328922199552,10cyclictest37-21ksoftirqd/320:25:013
328921699554,5cyclictest131rcu_preempt22:50:011
3289216995539,12cyclictest235-21systemd-journal21:15:011
3289216995512,9cyclictest23-21ksoftirqd/123:10:011
3289211995539,12cyclictest1-21systemd22:55:000
328922199549,8cyclictest131rcu_preempt21:25:033
328922199549,8cyclictest131rcu_preempt21:25:023
3289221995414,5cyclictest131rcu_preempt19:20:003
328921699542,8cyclictest23-21ksoftirqd/119:25:001
328921699541,21cyclictest0-21swapper/119:35:001
328922199535,7cyclictest37-21ksoftirqd/300:05:213
328922199535,11cyclictest37-21ksoftirqd/321:10:013
3289221995311,5cyclictest131rcu_preempt20:40:003
328921699539,6cyclictest131rcu_preempt20:45:011
328921699536,6cyclictest131rcu_preempt00:15:031
328921699536,6cyclictest131rcu_preempt00:15:021
328921699534,5cyclictest131rcu_preempt23:35:001
3289221995212,5cyclictest131rcu_preempt23:49:213
3289221995210,5cyclictest131rcu_preempt19:15:023
3289221995210,5cyclictest131rcu_preempt19:15:013
3289218995210,32cyclictest291rcuc/200:30:022
328922199518,9cyclictest37-21ksoftirqd/321:30:023
328922199517,9cyclictest37-21ksoftirqd/320:50:053
328922199515,8cyclictest37-21ksoftirqd/319:40:003
328922199515,8cyclictest37-21ksoftirqd/319:25:013
328922199515,5cyclictest131rcu_preempt22:25:023
3289221995133,12cyclictest235-21systemd-journal23:20:003
3289221995115,5cyclictest131rcu_preempt19:55:013
328921699518,6cyclictest131rcu_preempt22:30:011
328921699517,9cyclictest23-21ksoftirqd/122:00:011
328921699515,7cyclictest131rcu_preempt00:00:031
328921699512,43cyclictest272-21dbus-daemon22:55:011
3289211995139,6cyclictest3304513-21munin-plugin-st00:04:590
328922199509,5cyclictest131rcu_preempt21:05:003
328922199507,8cyclictest37-21ksoftirqd/300:15:003
328922199507,8cyclictest37-21ksoftirqd/300:15:003
328922199503,7cyclictest37-21ksoftirqd/300:30:003
328922199502,9cyclictest37-21ksoftirqd/300:35:033
328922199502,9cyclictest37-21ksoftirqd/300:35:023
3289221995021,20cyclictest361rcuc/322:03:203
3289221995012,6cyclictest131rcu_preempt23:10:033
3289221995012,5cyclictest131rcu_preempt21:15:013
3289221995012,5cyclictest131rcu_preempt00:20:003
3289221995010,6cyclictest131rcu_preempt20:55:013
3289216995040,7cyclictest23-21ksoftirqd/119:20:011
3289216995040,6cyclictest23-21ksoftirqd/119:50:001
3289211995035,11cyclictest3293822-21run-parts20:39:590
328922199498,5cyclictest131rcu_preempt20:06:203
328922199496,9cyclictest37-21ksoftirqd/300:00:033
3289221994914,6cyclictest131rcu_preempt00:40:003
3289218994933,13cyclictest1-21systemd20:55:012
3289218994931,14cyclictest272-21dbus-daemon23:05:002
328921699494,6cyclictest131rcu_preempt22:35:011
3289216994912,6cyclictest131rcu_preempt21:20:011
3289216994910,5cyclictest131rcu_preempt20:00:021
3289211994919,11cyclictest3291526-21cat19:55:010
328922199488,6cyclictest131rcu_preempt22:30:023
328922199486,8cyclictest37-21ksoftirqd/321:30:193
328922199484,8cyclictest37-21ksoftirqd/322:35:013
3289221994810,9cyclictest131rcu_preempt22:55:013
3289221994810,5cyclictest131rcu_preempt21:00:013
3289221994810,5cyclictest131rcu_preempt21:00:013
3289218994832,13cyclictest30-21ksoftirqd/223:40:002
3289218994832,13cyclictest30-21ksoftirqd/223:40:002
3289216994813,10cyclictest131rcu_preempt19:30:021
3289211994830,9cyclictest12-21ksoftirqd/021:10:010
328921199482,5cyclictest131rcu_preempt23:55:030
3289211994816,29cyclictest12-21ksoftirqd/000:37:400
328922199478,4cyclictest131rcu_preempt23:32:013
328922199476,11cyclictest37-21ksoftirqd/321:20:003
328922199475,8cyclictest37-21ksoftirqd/323:40:023
328922199475,8cyclictest37-21ksoftirqd/323:40:013
328922199474,10cyclictest37-21ksoftirqd/321:40:023
328922199474,10cyclictest37-21ksoftirqd/321:40:013
3289221994712,6cyclictest131rcu_preempt20:20:023
3289218994730,12cyclictest235-21systemd-journal21:40:012
3289218994730,12cyclictest235-21systemd-journal21:40:012
328921899471,3cyclictest131rcu_preempt23:45:022
328921699479,5cyclictest131rcu_preempt00:40:021
328921699477,8cyclictest23-21ksoftirqd/120:50:011
3289216994737,7cyclictest23-21ksoftirqd/121:30:001
328921699473,5cyclictest131rcu_preempt20:10:031
3289216994715,6cyclictest131rcu_preempt23:20:011
3289216994712,6cyclictest131rcu_preempt00:19:591
3289211994736,8cyclictest1-21systemd20:55:000
3289211994732,9cyclictest3304217-21cron00:00:010
3289211994732,9cyclictest3304217-21cron00:00:000
3289211994732,9cyclictest3302157-21munin-plugin-st23:20:000
328922199464,10cyclictest37-21ksoftirqd/300:25:013
328922199463,7cyclictest37-21ksoftirqd/322:10:003
328922199463,11cyclictest37-21ksoftirqd/323:43:343
3289221994612,4cyclictest131rcu_preempt22:42:213
3289218994630,11cyclictest3295924-21systemctl21:20:002
328921699468,7cyclictest131rcu_preempt21:35:011
328921699467,6cyclictest131rcu_preempt22:40:011
3289216994614,5cyclictest131rcu_preempt19:40:001
3289216994612,12cyclictest23-21ksoftirqd/123:37:211
3289216994612,12cyclictest23-21ksoftirqd/123:37:201
3289216994611,5cyclictest131rcu_preempt23:00:011
3289211994632,11cyclictest235-21systemd-journal23:13:190
3289211994628,14cyclictest3301603-21sshd23:06:210
328922199459,7cyclictest131rcu_preempt19:50:023
328922199451,6cyclictest0-21swapper/320:45:013
3289221994514,5cyclictest131rcu_preempt19:45:023
328921899452,3cyclictest131rcu_preempt21:45:022
328921699457,6cyclictest131rcu_preempt22:25:021
328921699456,8cyclictest23-21ksoftirqd/119:55:021
3289216994531,10cyclictest272-21dbus-daemon22:20:001
3289216994531,10cyclictest235-21systemd-journal20:15:011
3289216994514,7cyclictest131rcu_preempt22:05:011
3289216994513,7cyclictest23-21ksoftirqd/123:05:031
328922199447,5cyclictest131rcu_preempt23:30:013
3289218994424,16cyclictest1-21systemd00:00:002
3289218994424,16cyclictest1-21systemd00:00:002
328921699443,5cyclictest131rcu_preempt21:10:001
3289216994435,7cyclictest23-21ksoftirqd/119:40:201
3289216994434,7cyclictest23-21ksoftirqd/100:25:021
3289216994431,9cyclictest3295108-21sshd21:03:071
328922199439,5cyclictest131rcu_preempt21:54:193
328922199436,9cyclictest37-21ksoftirqd/323:00:193
328922199436,7cyclictest37-21ksoftirqd/319:35:013
3289221994330,10cyclictest37-21ksoftirqd/321:45:003
328922199431,5cyclictest0-21swapper/322:40:013
3289218994332,7cyclictest306-21in:imuxsock00:35:002
3289218994332,7cyclictest306-21in:imuxsock00:35:002
3289218994329,8cyclictest306-21in:imuxsock22:00:012
328921899432,5cyclictest131rcu_preempt22:15:002
328921699437,6cyclictest131rcu_preempt19:10:021
328921699434,6cyclictest131rcu_preempt00:35:011
328921699434,6cyclictest131rcu_preempt00:35:001
328921699433,6cyclictest131rcu_preempt23:15:011
328921699432,8cyclictest23-21ksoftirqd/120:25:031
328921699432,11cyclictest23-21ksoftirqd/123:45:011
3289216994313,5cyclictest131rcu_preempt22:45:001
3289211994329,10cyclictest235-21systemd-journal21:50:210
3289211994327,8cyclictest0-21swapper/021:20:000
3289211994326,13cyclictest3298744-21sshd22:13:180
3289211994324,14cyclictest272-21dbus-daemon23:00:020
328922199427,5cyclictest131rcu_preempt20:35:023
328922199427,5cyclictest131rcu_preempt20:35:023
328922199425,8cyclictest37-21ksoftirqd/319:27:203
3289221994212,5cyclictest131rcu_preempt20:10:213
328921899421,3cyclictest131rcu_preempt20:40:022
328921699429,5cyclictest131rcu_preempt00:10:011
328921699422,6cyclictest131rcu_preempt20:55:011
3289216994214,22cyclictest131rcu_preempt21:25:021
3289216994214,22cyclictest131rcu_preempt21:25:021
328921199423,11cyclictest3298533-21unin-run22:10:010
3289211994230,8cyclictest16950irq/224-800000000:25:040
3289211994223,9cyclictest12-21ksoftirqd/020:25:020
328921199421,38cyclictest0-21swapper/020:20:000
328922199417,6cyclictest131rcu_preempt22:58:263
3289221994132,6cyclictest361rcuc/323:25:023
3289221994132,6cyclictest361rcuc/323:25:013
328922199412,9cyclictest37-21ksoftirqd/323:53:203
3289218994128,9cyclictest1-21systemd21:05:012
328921699416,5cyclictest131rcu_preempt21:55:001
328921699414,13cyclictest23-21ksoftirqd/123:30:001
328921699413,6cyclictest131rcu_preempt20:30:021
328921699411,5cyclictest131rcu_preempt21:00:011
328921699411,5cyclictest131rcu_preempt21:00:001
3289216994114,11cyclictest23-21ksoftirqd/120:05:011
328921199418,25cyclictest19050irq/34-mmc122:30:010
3289211994135,4cyclictest16950irq/224-800000000:10:010
3289211994134,4cyclictest16950irq/224-800000023:40:020
3289211994134,4cyclictest16950irq/224-800000023:40:010
3289211994134,4cyclictest16850irq/134-800000022:25:010
3289211994122,12cyclictest235-21systemd-journal21:15:000
328921199411,9cyclictest3296876-21latency_hist21:35:020
328921199411,3cyclictest131rcu_preempt21:25:030
328921199411,3cyclictest131rcu_preempt21:25:020
328921199411,32cyclictest0-21swapper/020:10:010
328921899404,3cyclictest131rcu_preempt22:25:022
3289218994029,7cyclictest235-21systemd-journal19:51:202
3289218994024,11cyclictest3302861-21sort23:30:012
328921899402,3cyclictest131rcu_preempt21:10:002
328921699404,5cyclictest131rcu_preempt22:15:031
328921699402,8cyclictest23-21ksoftirqd/120:35:001
328921699402,8cyclictest23-21ksoftirqd/120:35:001
3289216994027,10cyclictest3303843-21sed23:50:021
328921199407,6cyclictest131rcu_preempt22:05:020
328921199403,6cyclictest3291750-21sendmail20:00:000
3289211994032,5cyclictest12-21ksoftirqd/020:50:020
3289211994031,7cyclictest12-21ksoftirqd/022:50:010
3289211994025,11cyclictest235-21systemd-journal22:00:000
3289211994022,12cyclictest272-21dbus-daemon20:45:000
328921899394,3cyclictest131rcu_preempt23:35:022
3289218993931,5cyclictest3294084-21cron20:45:002
3289218993929,7cyclictest3293028-21sshd20:21:202
3289218993926,10cyclictest3306336-21systemctl00:40:002
3289218993926,10cyclictest3289849-21sed19:20:022
3289218993925,10cyclictest30-21ksoftirqd/222:20:022
328921699397,7cyclictest23-21ksoftirqd/121:50:021
328921699397,6cyclictest131rcu_preempt21:43:201
3289216993913,6cyclictest131rcu_preempt23:20:031
3289216993913,6cyclictest131rcu_preempt23:20:021
328921199392,3cyclictest131rcu_preempt00:35:030
328921199392,3cyclictest131rcu_preempt00:35:020
3289211993919,14cyclictest272-21dbus-daemon19:15:010
3289211993919,14cyclictest272-21dbus-daemon19:15:010
328922199381,5cyclictest0-21swapper/322:48:453
3289218993828,8cyclictest30-21ksoftirqd/221:50:012
3289218993828,8cyclictest30-21ksoftirqd/220:12:202
3289218993827,6cyclictest3302587-21latency_hist23:25:032
3289218993827,6cyclictest3302587-21latency_hist23:25:022
3289218993825,9cyclictest3305536-21sshd00:23:412
3289218993822,12cyclictest272-21dbus-daemon00:20:012
3289218993818,13cyclictest272-21dbus-daemon19:15:022
3289218993818,13cyclictest272-21dbus-daemon19:15:012
328921199389,10cyclictest3289827-21grep19:20:020
3289211993829,5cyclictest306-21in:imuxsock19:45:010
328921199382,31cyclictest1-21systemd00:15:210
328921199381,4cyclictest3295122-21modprobe21:04:590
328921199381,34cyclictest0-21swapper/022:45:000
3289211993813,13cyclictest0-21swapper/019:10:010
328921899375,4cyclictest131rcu_preempt20:35:002
328921899375,4cyclictest131rcu_preempt20:35:002
3289218993726,7cyclictest306-21in:imuxsock19:50:002
3289218993725,8cyclictest30-21ksoftirqd/222:45:002
3289218993722,11cyclictest235-21systemd-journal23:10:002
328921899371,4cyclictest131rcu_preempt23:15:012
328921899371,4cyclictest131rcu_preempt22:48:252
3289211993728,6cyclictest19050irq/34-mmc120:02:400
3289211993727,7cyclictest0-21swapper/023:32:010
3289211993726,8cyclictest3290426-21sshd19:34:200
328921199372,13cyclictest235-21systemd-journal23:50:010
3289211993721,12cyclictest408-21lldpd21:45:000
328922199369,6cyclictest131rcu_preempt19:10:013
328921899363,4cyclictest131rcu_preempt19:10:022
328921899363,3cyclictest131rcu_preempt22:10:032
3289218993621,11cyclictest235-21systemd-journal22:40:002
3289218993620,12cyclictest30-21ksoftirqd/223:55:012
328921899361,4cyclictest131rcu_preempt19:25:012
3289211993626,6cyclictest3305004-21cron00:15:010
3289211993626,6cyclictest3305004-21cron00:15:000
3289211993626,6cyclictest3293564-21munin-plugin-st20:35:000
3289211993626,6cyclictest3293564-21munin-plugin-st20:35:000
328921199362,31cyclictest1-21systemd21:37:210
328921199362,31cyclictest1-21systemd21:37:200
3289211993622,11cyclictest12-21ksoftirqd/019:50:010
3289211993620,12cyclictest272-21dbus-daemon19:25:020
328921199361,31cyclictest3293290-21sshd20:28:030
3289211993611,7cyclictest131rcu_preempt19:25:020
3289218993526,5cyclictest3299597-21latency_hist22:30:012
3289218993525,8cyclictest30-21ksoftirqd/222:55:012
3289218993525,8cyclictest30-21ksoftirqd/200:05:022
328921899352,4cyclictest131rcu_preempt23:50:002
3289218993524,9cyclictest30-21ksoftirqd/221:55:022
328921899352,3cyclictest131rcu_preempt20:10:022
3289218993523,8cyclictest30-21ksoftirqd/223:00:022
3289218993522,7cyclictest306-21in:imuxsock22:05:002
3289211993528,4cyclictest1-21systemd23:30:010
3289211993525,7cyclictest3296568-21latency_hist21:30:010
3289211993524,7cyclictest235-21systemd-journal22:15:200
3289211993519,12cyclictest235-21systemd-journal23:02:200
328921199351,31cyclictest0-21swapper/020:15:010
328921899345,5cyclictest131rcu_preempt22:35:012
328921899345,3cyclictest131rcu_preempt20:50:002
3289218993425,6cyclictest3294839-21cron21:00:002
3289218993425,6cyclictest3294839-21cron21:00:002
328921899342,4cyclictest131rcu_preempt21:35:032
3289218993424,8cyclictest30-21ksoftirqd/221:25:002
3289218993424,8cyclictest30-21ksoftirqd/221:25:002
3289218993424,7cyclictest30-21ksoftirqd/221:15:032
3289218993422,8cyclictest3305020-21modprobe00:15:012
3289218993422,8cyclictest3305020-21modprobe00:15:002
3289218993422,8cyclictest3296425-21sshd21:28:112
3289218993420,11cyclictest3290705-21systemctl19:40:002
3289211993428,4cyclictest0-21swapper/021:47:070
3289211993427,4cyclictest3300072-21latency_hist22:40:010
328921199341,30cyclictest0-21swapper/023:44:530
3289218993325,5cyclictest3290499-21latency_hist19:35:012
328921899331,3cyclictest131rcu_preempt19:45:022
328921199331,4cyclictest0-21swapper/019:39:060
328921899327,3cyclictest131rcu_preempt20:00:012
328921899324,4cyclictest131rcu_preempt20:17:002
3289218993223,6cyclictest3292003-21cron20:05:002
328921899312,4cyclictest131rcu_preempt19:30:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional