You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 15:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sat Apr 27, 2024 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
716641991231,118cyclictest0-21swapper/020:40:020
716651998881,4cyclictest707749-21kworker/2:023:25:012
716655997221,45cyclictest235-21systemd-journal22:25:013
71665599714,18cyclictest308-21systemd-logind20:10:003
71665199702,63cyclictest730610-21munin-run23:40:002
71665599684,22cyclictest719225-21grep20:00:013
716655996827,38cyclictest729166-21latency_hist23:10:023
71665599676,31cyclictest731644-21systemctl00:00:013
71665599676,31cyclictest731644-21systemctl00:00:003
71665599676,14cyclictest731920-21munin-run00:04:593
716641996718,10cyclictest12-21ksoftirqd/023:05:020
71665599664,12cyclictest718415-21grep19:45:003
71665199663,58cyclictest723857-21cron21:30:002
71665199662,59cyclictest729278-21sshd23:12:482
71665599653,19cyclictest717381-21munin-run19:25:003
71665599653,16cyclictest725451-21unin-run22:00:013
716651996526,35cyclictest274-21dbus-daemon19:15:012
716651996526,35cyclictest274-21dbus-daemon19:15:012
71665199652,58cyclictest719719-21munin-run20:10:012
71664199654,49cyclictest235-21systemd-journal22:10:020
71665199642,58cyclictest732727-21systemd-run00:20:002
71665199641,16cyclictest717394-21munin-run19:25:012
71665599632,23cyclictest235-21systemd-journal22:15:003
71665599632,19cyclictest717104-21cron19:20:003
71665199633,56cyclictest722785-21sshd21:06:482
71665199633,54cyclictest728006-21cron22:50:002
71665199632,56cyclictest724904-21munin-run21:50:012
71665199632,56cyclictest722294-21munin-run21:00:012
71665199632,56cyclictest720500-21munin-run20:25:002
71665199631,16cyclictest718178-21munin-run19:40:012
71665599626,10cyclictest718661-21modprobe19:49:593
71665599625,10cyclictest722825-21modprobe21:10:003
71665599623,13cyclictest721250-21modprobe20:40:003
71665599622,17cyclictest274-21dbus-daemon22:20:013
71665199622,55cyclictest727764-21munin-run22:45:002
71665199622,55cyclictest724415-21munin-run21:40:022
71665199622,55cyclictest721294-21munin-run20:40:012
71665199622,53cyclictest732411-21cron00:15:012
71665199622,53cyclictest732411-21cron00:15:002
71665199622,14cyclictest727200-21sshd22:31:472
71665199622,12cyclictest728562-21munin-run23:00:012
71665599615,19cyclictest722554-21modprobe21:05:003
71665599613,21cyclictest716840-21munin-plugin-st19:15:003
71665599613,21cyclictest716840-21munin-plugin-st19:15:003
71665599613,13cyclictest718916-21sh19:55:003
71665599613,11cyclictest726983-21munin-plugin-st22:30:003
71665599612,15cyclictest725748-21grep22:05:023
71665599612,15cyclictest725748-21grep22:05:023
71665199613,53cyclictest731933-21munin-run00:05:012
71665199612,54cyclictest725688-21munin-run22:05:022
71665199612,54cyclictest725688-21munin-run22:05:012
71665199611,56cyclictest1917536-21snmpd19:28:212
71665199611,56cyclictest1917536-21snmpd19:28:212
71665199611,11cyclictest1917536-21snmpd23:27:112
716641996119,38cyclictest720479-21sshd20:23:490
71665599603,19cyclictest724926-21wc21:50:013
716655996027,29cyclictest274-21dbus-daemon23:35:013
716655996027,29cyclictest274-21dbus-daemon23:35:013
71665199603,51cyclictest731650-40logrotate00:00:002
71665199603,51cyclictest731650-40logrotate00:00:002
71665199602,53cyclictest726994-21munin-run22:30:012
71665199602,53cyclictest724092-21sshd21:32:352
71665199602,4cyclictest1917536-21snmpd20:41:372
71665199601,5cyclictest730359-21munin-run23:35:022
71665199601,5cyclictest730359-21munin-run23:35:012
71664899605,9cyclictest23-21ksoftirqd/100:20:011
71665599594,16cyclictest304-21in:imuxsock20:30:013
71665599594,10cyclictest729550-21cron23:20:013
71665599594,10cyclictest307-21rs:main3
71665599593,18cyclictest719473-21cat20:05:023
71665599593,16cyclictest721002-21cut20:35:003
716655995922,8cyclictest37-21ksoftirqd/300:15:023
716655995922,8cyclictest37-21ksoftirqd/300:15:013
71665599592,15cyclictest728977-21latency_hist23:05:033
71665599592,15cyclictest723455-21latency_hist21:20:023
71665199593,51cyclictest1917536-21snmpd19:51:282
71665199592,53cyclictest733344-21cat00:30:022
71665199592,48cyclictest722550-21cron21:05:002
71665599587,13cyclictest0-21swapper/319:40:003
71665599584,22cyclictest731123-21modprobe23:50:003
71665599583,16cyclictest721773-21cron20:49:593
71665599583,16cyclictest717884-21munin-plugin-st19:35:013
71665599582,20cyclictest733708-21sshd00:37:473
71665599582,14cyclictest724361-21cron21:40:003
71665199583,50cyclictest728788-21cron23:05:002
71665199582,52cyclictest719971-21munin-run20:15:012
71665199582,51cyclictest729020-21sshd23:06:492
71665199582,51cyclictest1917536-21snmpd20:01:382
71665199582,51cyclictest1917536-21snmpd19:56:292
71665199581,4cyclictest732136-21sshd00:05:482
71665599575,22cyclictest727826-21grep22:45:013
71665599573,9cyclictest722297-21systemctl21:00:003
716655995717,10cyclictest37-21ksoftirqd/322:35:023
71665199572,50cyclictest1917536-21snmpd22:18:112
71665199572,50cyclictest1917536-21snmpd21:11:442
71665199571,7cyclictest730881-21munin-run23:45:012
71665199571,5cyclictest1917536-21snmpd20:16:342
71665199571,52cyclictest1917536-21snmpd19:15:382
71665199571,14cyclictest1917536-21snmpd19:44:002
71665599563,21cyclictest724655-21modprobe21:45:003
71665599563,14cyclictest729306-21grep23:15:003
716655995622,4cyclictest723073-21latency_hist21:15:023
716655995614,11cyclictest274-21dbus-daemon20:55:013
71665599561,15cyclictest732963-21modprobe00:25:013
71665199562,50cyclictest1917536-21snmpd19:46:272
71665199562,49cyclictest1917536-21snmpd22:06:122
71665199562,13cyclictest1917536-21snmpd21:18:492
71665199561,50cyclictest1917536-21snmpd22:52:472
71665599558,9cyclictest0-21swapper/300:20:013
71665599554,19cyclictest733282-21latency_hist00:30:013
71665199559,42cyclictest0-21swapper/222:24:032
71665199553,47cyclictest1917536-21snmpd20:25:322
71665199552,49cyclictest1917536-21snmpd23:16:022
71665199551,17cyclictest718085-21date19:35:022
71665199551,12cyclictest1917536-21snmpd23:53:332
71664899557,6cyclictest23-21ksoftirqd/120:15:011
716641995547,5cyclictest151rcuc/020:05:010
716641995538,12cyclictest235-21systemd-journal19:30:000
716641995538,12cyclictest235-21systemd-journal19:30:000
71665599544,12cyclictest725158-21cut21:55:003
71665599543,9cyclictest720230-21systemctl20:20:003
71665599543,13cyclictest716725-21grep19:10:023
716655995417,6cyclictest0-21swapper/321:30:503
71665199542,6cyclictest1917536-21snmpd00:31:432
71665199542,47cyclictest1917536-21snmpd21:24:292
71665199542,46cyclictest733721-21cron00:40:002
71665199542,46cyclictest725335-21wc21:55:022
71665199541,4cyclictest1917536-21snmpd20:46:392
716648995414,9cyclictest23-21ksoftirqd/120:30:021
716648995414,11cyclictest23-21ksoftirqd/120:20:011
71665599534,15cyclictest719954-21latency_hist20:15:033
71665599534,11cyclictest1-21systemd21:25:003
71665199532,9cyclictest724629-21modprobe21:42:502
716648995317,8cyclictest23-21ksoftirqd/120:45:011
71665599522,15cyclictest731433-21latency_hist23:55:023
716655995217,9cyclictest730192-21cat23:30:023
716655995215,8cyclictest717707-21sed19:30:023
716655995215,8cyclictest717707-21sed19:30:023
71665199522,46cyclictest722007-21sshd20:52:272
716651995221,28cyclictest30-21ksoftirqd/222:00:022
71665199521,46cyclictest1917536-21snmpd00:23:162
71665199521,17cyclictest716759-21latency_hist19:10:022
71664199521,48cyclictest0-21swapper/023:10:020
71665199512,44cyclictest1917536-21snmpd20:30:122
71665199511,5cyclictest1917536-21snmpd22:10:032
71665199511,4cyclictest1917536-21snmpd23:50:002
716648995113,9cyclictest23-21ksoftirqd/122:20:011
71665599503,13cyclictest732174-21munin-run00:10:003
71665199501,44cyclictest1917536-21snmpd22:36:522
71664899506,7cyclictest23-21ksoftirqd/122:40:021
71664899505,13cyclictest23-21ksoftirqd/120:55:001
71664899501,12cyclictest23-21ksoftirqd/123:30:011
71664199509,4cyclictest721012-21munin-run20:35:010
716641995043,4cyclictest0-21swapper/023:35:030
71664199502,41cyclictest731124-21modprobe23:50:000
716641995010,36cyclictest721798-21unin-run20:50:000
71664899493,8cyclictest23-21ksoftirqd/119:15:001
71664899493,8cyclictest23-21ksoftirqd/119:15:001
716648994934,12cyclictest731644-21systemctl00:00:001
716648994934,12cyclictest731644-21systemctl00:00:001
71664899485,9cyclictest23-21ksoftirqd/121:30:021
71664899485,11cyclictest23-21ksoftirqd/119:45:021
71664199485,38cyclictest0-21swapper/020:06:490
71664199483,41cyclictest0-21swapper/023:35:010
71664199483,41cyclictest0-21swapper/023:35:010
716641994831,13cyclictest274-21dbus-daemon20:45:010
71664199482,11cyclictest308-21systemd-logind23:55:010
71664899474,9cyclictest23-21ksoftirqd/100:05:001
716648994712,7cyclictest23-21ksoftirqd/123:10:011
716641994740,4cyclictest16850irq/224-800000022:00:000
71664899469,8cyclictest723295-21kworker/u8:021:45:031
716648994633,10cyclictest723585-21sshd21:21:491
716641994632,10cyclictest1-21systemd21:00:010
716641994628,14cyclictest1-21systemd21:10:000
716641994627,13cyclictest274-21dbus-daemon00:35:000
716641994626,16cyclictest1-21systemd21:30:010
716641994622,8cyclictest12-21ksoftirqd/019:12:010
716641994622,8cyclictest12-21ksoftirqd/019:12:010
716655994530,12cyclictest37-21ksoftirqd/323:00:003
71664899459,5cyclictest131rcu_preempt23:05:011
71664899455,8cyclictest23-21ksoftirqd/122:50:021
71664899455,7cyclictest23-21ksoftirqd/120:25:021
716648994528,14cyclictest722284-21cron21:00:001
716648994524,18cyclictest23-21ksoftirqd/120:10:021
716641994529,12cyclictest729524-21sshd23:16:490
716641994524,15cyclictest274-21dbus-daemon00:25:010
71664899446,6cyclictest23-21ksoftirqd/121:40:001
71664899445,9cyclictest23-21ksoftirqd/120:40:001
71664899441,7cyclictest0-21swapper/123:20:011
71664199446,9cyclictest12-21ksoftirqd/000:20:000
71664199444,35cyclictest0-21swapper/020:30:010
716641994440,2cyclictest0-21swapper/022:35:020
716641994430,11cyclictest12-21ksoftirqd/000:00:020
716641994430,11cyclictest12-21ksoftirqd/000:00:010
71664199441,14cyclictest727184-21date22:30:020
716655994324,14cyclictest274-21dbus-daemon20:25:003
71664899436,8cyclictest23-21ksoftirqd/123:50:021
71664899434,9cyclictest23-21ksoftirqd/119:55:001
716648994328,11cyclictest1-21systemd22:00:001
71664899431,23cyclictest0-21swapper/119:25:011
716641994330,10cyclictest235-21systemd-journal19:47:130
716641994329,10cyclictest235-21systemd-journal00:09:480
716641994326,13cyclictest728245-21sshd22:52:490
71664199432,37cyclictest717852-21sshd19:31:500
716641994317,23cyclictest0-21swapper/000:28:340
71664899427,6cyclictest23-21ksoftirqd/122:15:011
71664899426,6cyclictest23-21ksoftirqd/120:05:021
71664899425,7cyclictest23-21ksoftirqd/122:10:011
71664899425,6cyclictest23-21ksoftirqd/119:40:011
716641994213,25cyclictest716703-21latency_hist19:10:020
71664899416,8cyclictest23-21ksoftirqd/123:41:481
71664899416,8cyclictest23-21ksoftirqd/100:07:491
71664899413,8cyclictest23-21ksoftirqd/122:45:011
71664899412,7cyclictest23-21ksoftirqd/119:19:491
716648994126,12cyclictest728615-21cat23:00:021
716648994125,12cyclictest721772-21cron20:49:591
71664899411,8cyclictest0-21swapper/121:10:011
71664899411,7cyclictest722753-21date21:05:031
71664199415,31cyclictest0-21swapper/021:15:020
71664199412,36cyclictest235-21systemd-journal19:59:430
71664199411,35cyclictest401-21lldpd21:03:090
71665599405,28cyclictest733474-21munin-plugin-st00:35:003
716655994025,11cyclictest304-21in:imuxsock22:10:013
716655994019,13cyclictest304-21in:imuxsock23:25:003
716655994018,16cyclictest1-21systemd23:40:013
71664899406,6cyclictest131rcu_preempt00:30:011
71664899403,7cyclictest23-21ksoftirqd/122:35:011
716648994024,10cyclictest23-21ksoftirqd/123:25:011
716648994023,12cyclictest274-21dbus-daemon22:25:001
716648994021,13cyclictest274-21dbus-daemon21:15:011
71664199406,8cyclictest12-21ksoftirqd/021:43:520
716641994031,5cyclictest18650irq/34-mmc123:15:010
716641994029,7cyclictest235-21systemd-journal21:37:490
716641994028,8cyclictest725142-21cron21:55:000
716641994026,10cyclictest235-21systemd-journal22:50:010
71664199401,5cyclictest726204-21munin-run22:15:010
71665599395,28cyclictest721533-21munin-run20:45:003
71664899395,7cyclictest23-21ksoftirqd/100:25:011
71664899394,6cyclictest23-21ksoftirqd/100:15:021
71664899394,6cyclictest23-21ksoftirqd/100:15:021
716641993931,5cyclictest730889-21systemd-run23:45:000
716641993931,4cyclictest16850irq/224-800000000:40:000
716641993925,11cyclictest723403-21sed21:20:020
716641993923,13cyclictest727453-21sshd22:36:480
71664199392,11cyclictest12-21ksoftirqd/019:20:490
71664199391,34cyclictest0-21swapper/020:51:500
71664899384,6cyclictest23-21ksoftirqd/121:20:011
71664899383,10cyclictest23-21ksoftirqd/123:35:021
71664899383,10cyclictest23-21ksoftirqd/123:35:011
71664899382,10cyclictest23-21ksoftirqd/122:05:021
71664899382,10cyclictest23-21ksoftirqd/122:05:011
716648993819,15cyclictest274-21dbus-daemon21:55:001
71664899381,6cyclictest149-21pr/ttyS200:35:511
716641993827,8cyclictest304-21in:imuxsock23:21:470
71664199382,6cyclictest720231-21munin-run20:20:000
71664199381,34cyclictest0-21swapper/000:05:000
716648993729,6cyclictest23-21ksoftirqd/123:10:031
71664899370,5cyclictest149-21pr/ttyS223:51:241
716641993724,10cyclictest718400-21cron19:45:000
716655993622,10cyclictest1-21systemd22:50:023
71665599361,7cyclictest723883-21munin-run21:30:013
716648993624,8cyclictest304-21in:imuxsock23:39:591
71664899360,6cyclictest149-21pr/ttyS219:33:511
716641993630,4cyclictest16850irq/224-800000022:43:480
716641993628,5cyclictest18650irq/34-mmc122:55:180
71664199362,30cyclictest0-21swapper/022:03:050
71664199362,30cyclictest0-21swapper/022:03:050
71664199361,5cyclictest730094-21/usr/sbin/munin23:30:010
71664199361,5cyclictest724122-21munin-run21:35:010
71664199361,31cyclictest0-21swapper/021:25:010
71664199361,31cyclictest0-21swapper/019:18:490
71664199361,15cyclictest718968-21latency_hist19:55:010
71664899354,7cyclictest23-21ksoftirqd/100:34:481
716648993525,8cyclictest23-21ksoftirqd/122:30:021
716648993522,10cyclictest23-21ksoftirqd/120:00:001
716641993517,14cyclictest724880-21sshd21:47:490
716655993427,4cyclictest730848-21sshd23:41:483
716655993419,10cyclictest304-21in:imuxsock22:54:593
716648993422,9cyclictest23-21ksoftirqd/119:50:011
716648993419,11cyclictest274-21dbus-daemon21:35:011
716641993428,4cyclictest0-21swapper/022:16:190
71664199341,30cyclictest0-21swapper/019:37:110
716648993310,18cyclictest131rcu_preempt22:53:481
716641993327,4cyclictest0-21swapper/000:13:020
716641993327,4cyclictest0-21swapper/000:13:010
716641993322,8cyclictest12-21ksoftirqd/022:25:020
716648993222,7cyclictest717614-21sshd19:29:501
716648993222,7cyclictest717614-21sshd19:29:491
71664899321,7cyclictest0-21swapper/120:34:481
71664199321,4cyclictest0-21swapper/020:10:090
716648992919,7cyclictest23-21ksoftirqd/121:48:561
716648992821,4cyclictest716580-21latency_hist19:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional