You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 20:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Nov 30, 2025 12:45:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
623790999423,40cyclictest632852-21sort10:05:022
623790999423,40cyclictest632852-21sort10:05:012
62378299793,69cyclictest235-21systemd-journal10:00:000
623786997769,6cyclictest23-21ksoftirqd/107:30:011
62378299742,41cyclictest638838-21munin-run12:00:010
62378299741,41cyclictest634484-21cat10:35:020
623790996923,13cyclictest638032-21modprobe11:40:012
62378299692,36cyclictest272-21dbus-daemon12:30:000
62378299681,62cyclictest272-21dbus-daemon11:00:000
62378299671,61cyclictest0-21swapper/010:44:590
623782996632,30cyclictest19050irq/34-mmc109:10:020
62378299642,58cyclictest635969-21sshd11:03:150
62378299642,57cyclictest637266-21sshd11:28:150
623782996410,36cyclictest0-21swapper/011:36:470
623790996322,15cyclictest639121-21munin-run12:05:022
62378299632,56cyclictest630193-21sshd09:14:150
62378299631,58cyclictest628878-21sshd08:45:330
62378299623,55cyclictest640164-21munin-run12:25:010
62378299622,56cyclictest631240-21sshd09:34:160
62378299622,55cyclictest633079-21munin-run10:09:590
62378299622,53cyclictest632297-21grep09:55:000
623782996225,33cyclictest12-21ksoftirqd/011:10:030
62378299621,37cyclictest624962-21latency_hist07:30:020
62378299612,55cyclictest633840-21sshd10:20:540
62378299612,55cyclictest633840-21sshd10:20:530
62378299612,55cyclictest625544-21munin-run07:45:000
62378299612,54cyclictest627625-21munin-run08:25:000
62378299612,53cyclictest637552-21munin-run11:35:020
623782996123,34cyclictest629506-21latency_hist09:00:010
62378299611,56cyclictest627853-21sshd08:27:160
62378299611,55cyclictest639595-21sshd12:14:160
62378299602,53cyclictest638538-21sshd11:50:150
62378299602,53cyclictest632056-21munin-run09:50:010
62378299602,53cyclictest632056-21munin-run09:50:010
62378299601,54cyclictest635143-21apt-get10:45:280
62378299601,54cyclictest630507-21true09:20:020
62378299601,53cyclictest631874-21latency_hist09:45:010
623782995928,28cyclictest1-21systemd12:36:150
62378299592,53cyclictest633633-21munin-run10:20:000
62378299592,52cyclictest623801-21latency_hist07:10:010
62378299591,54cyclictest635461-21munin-run10:55:010
623792995836,16cyclictest235-21systemd-journal09:10:013
623790995837,17cyclictest626015-21sshd07:50:252
623786995841,13cyclictest23-21ksoftirqd/110:45:301
62378299582,52cyclictest363-21snmpd12:15:040
62378299582,52cyclictest363-21snmpd12:15:030
62378299581,53cyclictest629152-21cron08:55:010
62378299581,52cyclictest636710-21sshd11:15:160
62378299581,52cyclictest626061-21munin-run07:55:010
62378299581,52cyclictest0-21swapper/009:40:000
623790995732,20cyclictest639741-21latency_hist12:15:012
623790995729,23cyclictest624230-21sendmail07:20:002
62378299572,50cyclictest637017-21sshd11:24:150
62378299571,52cyclictest633315-21sshd10:14:150
62378299571,51cyclictest625307-21munin-run07:40:020
62378299571,51cyclictest624240-21munin-run07:20:000
623790995633,17cyclictest281-21systemd-logind09:40:012
623790995632,19cyclictest634335-21modprobe10:30:022
623790995629,23cyclictest638002-21latency_hist11:40:022
62378299562,49cyclictest640658-21munin-run12:35:010
62378299562,49cyclictest627388-21munin-run08:20:010
62378299562,49cyclictest363-21snmpd10:37:070
62378299562,49cyclictest363-21snmpd10:37:060
62378299561,51cyclictest639342-21sshd12:09:150
62378299561,50cyclictest363-21snmpd08:43:390
62378299561,50cyclictest363-21snmpd07:45:460
623792995537,13cyclictest635715-21cron10:59:593
623790995530,20cyclictest630520-21cat09:20:012
623790995530,20cyclictest625468-21latency_hist07:40:022
623790995529,22cyclictest624790-21latency_hist07:30:022
62378299552,49cyclictest638039-21sshd11:41:080
62378299552,49cyclictest628111-21sshd08:34:140
62378299552,48cyclictest630752-21munin-run09:25:000
62378299551,51cyclictest626276-21sshd07:57:150
62378299551,50cyclictest632791-21sshd10:00:160
62378299551,50cyclictest632791-21sshd10:00:150
62378299551,49cyclictest624976-21modprobe07:30:140
623790995432,17cyclictest640643-21cron12:35:002
623790995430,19cyclictest641146-21date12:40:022
623790995428,20cyclictest631166-21latency_hist09:30:022
623790995424,25cyclictest627864-21cron08:30:002
62378299542,47cyclictest631104-21latency_hist09:30:010
62378299541,49cyclictest639075-21sshd12:01:120
62378299541,49cyclictest626548-21sshd08:00:270
62378299541,49cyclictest624480-21sshd07:21:140
62378299541,49cyclictest0-21swapper/008:40:010
62378299541,48cyclictest363-21snmpd10:26:230
62378299541,47cyclictest623944-21sshd07:12:000
623792995320,27cyclictest37-21ksoftirqd/310:20:013
623790995331,18cyclictest639835-21sshd12:16:162
623790995331,18cyclictest639835-21sshd12:16:152
623790995331,18cyclictest633092-21munin-run10:10:002
623790995331,17cyclictest626516-21sed08:00:022
623790995330,17cyclictest640499-21latency_hist12:30:012
623790995328,21cyclictest635465-21sh10:55:002
62378299531,49cyclictest629672-21sshd09:03:150
623790995237,11cyclictest638286-21sshd11:45:152
623790995233,16cyclictest628349-21modprobe08:35:322
623790995232,16cyclictest624985-21sshd07:32:142
623790995231,17cyclictest632027-21modprobe09:49:162
623790995231,17cyclictest632027-21modprobe09:49:152
623790995230,17cyclictest623973-21true07:15:002
623790995229,19cyclictest636618-21cut11:15:022
623782995223,27cyclictest12-21ksoftirqd/011:10:020
623792995135,13cyclictest235-21systemd-journal08:35:013
623790995137,10cyclictest630726-21sshd09:22:162
623790995136,11cyclictest634908-21munin-run10:45:012
623790995134,11cyclictest632557-21/usr/sbin/munin10:00:022
623790995131,16cyclictest635126-21sendmail-mta10:45:212
623790995130,15cyclictest628972-21cat08:50:012
623790995129,19cyclictest627086-21/usr/sbin/munin08:15:002
623790995128,18cyclictest637547-21systemd-run11:35:002
623790995123,23cyclictest629714-21latency_hist09:05:012
623790995035,11cyclictest640180-21sed12:25:012
623790995034,11cyclictest631773-21sshd09:44:152
623790995031,15cyclictest363-21snmpd10:13:382
623790995028,18cyclictest634094-21sshd10:26:142
623786995032,13cyclictest235-21systemd-journal10:18:151
623792994933,12cyclictest235-21systemd-journal11:39:593
62379299493,12cyclictest235-21systemd-journal10:00:003
623792994929,13cyclictest272-21dbus-daemon12:00:003
623790994935,11cyclictest412-21sshd09:33:152
623790994935,10cyclictest638588-21munin-run11:55:012
623790994933,12cyclictest363-21snmpd10:37:252
623790994933,12cyclictest363-21snmpd10:37:242
623790994931,14cyclictest635813-21cat11:00:022
623790994929,16cyclictest632301-21systemd-run09:55:002
623790994928,17cyclictest637301-21unin-run11:30:002
623790994927,17cyclictest637005-21sshd11:21:062
623790994923,22cyclictest629290-21latency_hist08:55:022
62378299492,42cyclictest272-21dbus-daemon08:15:000
62378299491,42cyclictest363-21snmpd08:05:490
623782994913,32cyclictest638325-21grep11:50:010
623792994833,11cyclictest235-21systemd-journal09:30:003
623790994836,9cyclictest635978-21cron11:05:002
623790994834,11cyclictest625768-21sshd07:46:152
623790994833,10cyclictest363-21snmpd07:21:242
623790994831,13cyclictest633698-21cat10:20:022
623790994831,12cyclictest627637-21systemd-run08:25:012
623790994826,18cyclictest634050-21latency_hist10:25:032
623790994826,18cyclictest634050-21latency_hist10:25:022
623790994822,22cyclictest626931-21latency_hist08:10:022
623786994829,14cyclictest1-21systemd07:45:001
62379299473,3cyclictest131rcu_preempt08:25:013
623792994727,16cyclictest272-21dbus-daemon10:15:013
623790994735,9cyclictest363-21snmpd11:59:022
623790994734,10cyclictest636719-21sshd11:16:152
623790994733,11cyclictest363-21snmpd12:08:472
623790994732,10cyclictest363-21snmpd08:03:132
623790994728,15cyclictest363-21snmpd08:43:462
623790994727,16cyclictest629405-21cron09:00:002
623790994725,18cyclictest630207-21latency_hist09:15:012
623792994635,6cyclictest630459-21cron09:20:013
623792994631,10cyclictest306-21in:imuxsock07:40:013
623790994634,9cyclictest627329-21sshd08:19:162
623790994632,10cyclictest363-21snmpd11:07:102
623786994632,11cyclictest23-21ksoftirqd/108:25:011
623786994628,13cyclictest235-21systemd-journal10:10:001
62378699462,23cyclictest235-21systemd-journal07:50:021
62379299451,3cyclictest131rcu_preempt08:20:023
623790994532,10cyclictest625521-21sshd07:43:152
623790994532,10cyclictest363-21snmpd08:34:062
62379099452,3cyclictest131rcu_preempt09:10:002
623786994533,6cyclictest306-21in:imuxsock10:35:001
62379299442,3cyclictest131rcu_preempt08:30:033
623786994428,10cyclictest235-21systemd-journal07:20:011
62378699441,3cyclictest131rcu_preempt12:24:591
62378699441,3cyclictest131rcu_preempt08:40:031
62379299438,12cyclictest37-21ksoftirqd/308:50:003
623786994330,10cyclictest272-21dbus-daemon12:15:011
623786994329,9cyclictest235-21systemd-journal11:30:011
62379299425,7cyclictest131rcu_preempt12:39:593
62379299422,8cyclictest624804-21latency_hist07:30:013
623792994225,11cyclictest272-21dbus-daemon11:10:013
623792994224,13cyclictest272-21dbus-daemon07:55:013
623786994233,6cyclictest23-21ksoftirqd/112:10:021
62378699422,35cyclictest638246-21wc11:45:031
623786994211,13cyclictest23-21ksoftirqd/108:00:021
62379299413,4cyclictest131rcu_preempt11:45:013
623792994129,8cyclictest1-21systemd10:05:023
623792994129,8cyclictest1-21systemd10:05:013
623790994121,16cyclictest623825-21sort07:10:012
623792994032,5cyclictest625011-21munin-plugin-st07:35:003
623792994029,8cyclictest637514-21sshd11:32:153
623792994027,9cyclictest235-21systemd-journal12:30:033
623792994023,10cyclictest272-21dbus-daemon11:05:003
623786994027,10cyclictest235-21systemd-journal09:10:001
62378699402,3cyclictest131rcu_preempt11:39:591
623786994023,11cyclictest235-21systemd-journal08:59:591
623786994022,14cyclictest272-21dbus-daemon10:30:011
623792993929,8cyclictest37-21ksoftirqd/308:54:153
623792993924,11cyclictest37-21ksoftirqd/309:00:373
623792993923,11cyclictest631497-21cron09:39:593
623792993922,13cyclictest1-21systemd08:45:003
623792993919,13cyclictest631783-21modprobe09:45:013
62378699394,3cyclictest131rcu_preempt11:59:591
623786993930,4cyclictest626580-21munin-plugin-st08:05:001
623786993929,6cyclictest306-21in:imuxsock12:05:001
623786993928,9cyclictest23-21ksoftirqd/108:35:011
623786993926,8cyclictest636486-21modprobe11:15:001
623786993923,13cyclictest632980-21sed10:05:021
623786993923,13cyclictest632980-21sed10:05:021
623786993921,14cyclictest235-21systemd-journal10:55:151
62378699391,3cyclictest131rcu_preempt11:05:011
62379299385,23cyclictest624225-21modprobe07:20:003
623792993829,5cyclictest271-21cron07:59:593
623786993830,5cyclictest23-21ksoftirqd/109:40:011
623786993828,6cyclictest306-21in:imuxsock11:10:001
623786993823,11cyclictest235-21systemd-journal10:53:151
623786993823,10cyclictest272-21dbus-daemon11:50:001
623786993822,11cyclictest272-21dbus-daemon12:30:011
62378699381,3cyclictest131rcu_preempt09:30:011
62378699381,33cyclictest23-21ksoftirqd/110:00:021
623792993724,5cyclictest306-21in:imuxsock10:25:013
623792993724,5cyclictest306-21in:imuxsock10:25:003
623792993721,12cyclictest1-21systemd10:35:013
62378699375,3cyclictest131rcu_preempt07:40:021
62378699373,26cyclictest627334-21cron08:20:001
623786993728,7cyclictest23-21ksoftirqd/107:10:011
623786993726,9cyclictest23-21ksoftirqd/109:35:011
623786993726,6cyclictest306-21in:imuxsock11:35:001
62379299366,18cyclictest306-21in:imuxsock08:10:003
62379299364,4cyclictest131rcu_preempt08:40:003
623792993627,5cyclictest306-21in:imuxsock09:00:003
623792993627,5cyclictest306-21in:imuxsock07:15:003
623792993626,6cyclictest306-21in:imuxsock12:10:013
623792993624,5cyclictest37-21ksoftirqd/312:15:003
623792993623,9cyclictest1-21systemd12:30:003
623792993623,10cyclictest235-21systemd-journal11:55:023
623792993622,10cyclictest1-21systemd11:20:013
62379299361,4cyclictest131rcu_preempt07:45:003
623792993613,16cyclictest272-21dbus-daemon09:55:013
623786993625,9cyclictest23-21ksoftirqd/109:55:021
623786993623,9cyclictest23-21ksoftirqd/110:40:011
623786993623,9cyclictest23-21ksoftirqd/110:40:001
623786993623,10cyclictest634025-21sed10:25:021
623786993623,10cyclictest634025-21sed10:25:021
623786993623,10cyclictest23-21ksoftirqd/112:35:001
623786993622,10cyclictest23-21ksoftirqd/110:15:001
623786993618,14cyclictest1-21systemd07:55:011
62378699361,32cyclictest23-21ksoftirqd/111:19:151
62379299355,3cyclictest131rcu_preempt11:30:003
623792993524,8cyclictest632079-21sed09:50:013
623792993524,8cyclictest632079-21sed09:50:013
623792993522,9cyclictest1-21systemd10:45:003
623792993521,10cyclictest235-21systemd-journal11:20:023
623792993510,18cyclictest634638-21sendmail10:40:013
623792993510,18cyclictest634638-21sendmail10:40:003
62378699354,3cyclictest131rcu_preempt11:25:011
62378699353,3cyclictest131rcu_preempt10:43:151
62378699352,3cyclictest131rcu_preempt08:45:001
623786993523,9cyclictest629034-21sed08:50:031
623786993523,9cyclictest23-21ksoftirqd/111:54:151
623786993523,9cyclictest23-21ksoftirqd/109:20:011
623786993521,9cyclictest235-21systemd-journal07:15:001
623786993518,13cyclictest272-21dbus-daemon07:35:011
62378699351,3cyclictest131rcu_preempt09:05:011
62378699351,3cyclictest131rcu_preempt08:28:151
62379299346,3cyclictest131rcu_preempt10:50:013
623792993419,12cyclictest1-21systemd10:10:003
62378699344,5cyclictest131rcu_preempt12:40:001
62378699344,10cyclictest131rcu_preempt08:55:001
62378699343,3cyclictest131rcu_preempt09:11:151
623786993424,6cyclictest306-21in:imuxsock09:50:021
623786993424,6cyclictest306-21in:imuxsock09:50:011
62379299333,3cyclictest131rcu_preempt12:24:593
623792993326,4cyclictest1-21systemd09:35:023
623792993324,7cyclictest1-21systemd11:15:003
623792993318,11cyclictest235-21systemd-journal11:50:013
623786993321,9cyclictest626991-21cat08:10:021
623786993320,11cyclictest23-21ksoftirqd/107:25:001
62378699331,3cyclictest131rcu_preempt12:20:011
62378699331,3cyclictest131rcu_preempt12:20:001
62378699331,3cyclictest131rcu_preempt08:11:141
62379299327,14cyclictest639866-21modprobe12:20:013
62379299327,14cyclictest639866-21modprobe12:20:003
62379299325,20cyclictest635454-21modprobe10:55:013
623792993218,11cyclictest37-21ksoftirqd/310:30:003
623786993222,7cyclictest630729-21sshd09:23:151
623786993221,8cyclictest23-21ksoftirqd/109:45:021
62379299315,14cyclictest625789-21modprobe07:50:013
62379299312,5cyclictest131rcu_preempt08:15:013
623792993119,8cyclictest639077-21sshd12:01:123
623792993119,10cyclictest37-21ksoftirqd/307:10:013
623792993117,7cyclictest0-21swapper/308:05:033
62379299311,3cyclictest131rcu_preempt07:25:013
623792993023,5cyclictest630207-21latency_hist09:15:013
623792992913,10cyclictest306-21in:imuxsock09:20:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional