You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri May 09, 2025 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
273203998516,27cyclictest285944-21modprobe23:20:002
27319699756,37cyclictest0-21swapper/021:20:020
27319699756,37cyclictest0-21swapper/021:20:020
27319899746,16cyclictest23-21ksoftirqd/122:35:001
27320699726,17cyclictest276709-21modprobe20:20:003
273203996921,14cyclictest274405-21latency_hist19:35:012
273206996829,12cyclictest37-21ksoftirqd/323:55:003
27320699674,26cyclictest283648-21munin-run22:35:003
273206996618,14cyclictest1-21systemd20:45:013
273196996636,26cyclictest284403-21sshd22:48:250
273196996419,11cyclictest275196-21unin-run19:50:010
273196996338,19cyclictest12-21ksoftirqd/000:05:020
273198995937,16cyclictest235-21systemd-journal20:00:011
273196995935,4cyclictest288045-21munin-run00:00:020
273203995814,31cyclictest273192-21cyclictest20:00:022
273196995826,27cyclictest277643-21latency_hist20:35:020
273196995739,14cyclictest286961-21cron23:40:000
273196995714,38cyclictest274932-21modprobe19:45:010
273196995641,11cyclictest274126-21sshd19:25:250
273196995641,11cyclictest274126-21sshd19:25:250
273198995512,8cyclictest23-21ksoftirqd/121:25:001
273196995548,4cyclictest16950irq/224-800000023:00:000
273196995518,33cyclictest616-21sendmail-mta21:24:250
27319899544,13cyclictest23-21ksoftirqd/123:40:001
27319899542,16cyclictest23-21ksoftirqd/100:40:001
27319699543,8cyclictest283897-21sh22:40:010
273196995413,13cyclictest1-21systemd21:47:250
273206995339,9cyclictest235-21systemd-journal20:15:003
273206995339,9cyclictest235-21systemd-journal20:14:593
273198995311,6cyclictest131rcu_preempt21:10:011
27319699535,8cyclictest131rcu_preempt23:50:030
273196995340,8cyclictest274373-21sshd19:30:250
273196995338,11cyclictest274640-21sshd19:39:250
273196995337,12cyclictest273358-21sshd19:14:250
273203995237,11cyclictest288296-21munin-run00:05:012
273203995237,11cyclictest273390-21munin-run19:15:012
273203995235,13cyclictest290040-21sshd00:38:252
273203995232,16cyclictest283869-21sshd22:36:252
27319899522,5cyclictest131rcu_preempt20:50:011
273196995238,10cyclictest279046-21munin-run21:05:010
273196995238,10cyclictest278013-21munin-run20:45:000
273196995212,36cyclictest289029-21modprobe00:19:590
273206995131,14cyclictest235-21systemd-journal20:00:013
273203995138,10cyclictest273922-21munin-run19:25:002
273203995137,10cyclictest287501-21munin-run23:50:002
273203995136,12cyclictest282084-21sshd22:00:262
273203995134,13cyclictest288753-21sshd00:11:252
273203995122,23cyclictest276667-21modprobe20:15:022
27320399511,9cyclictest283646-21munin-plugin-st22:35:002
273196995137,10cyclictest279256-21sshd21:07:250
273206995034,12cyclictest272-21dbus-daemon22:40:013
273203995038,9cyclictest274139-21sshd19:28:252
273203995038,9cyclictest274139-21sshd19:28:252
273203995037,9cyclictest48-21kcompactd023:30:382
273203995037,10cyclictest288504-21sshd00:06:252
273203995037,10cyclictest280785-21sshd21:35:252
273203995037,10cyclictest277718-21sshd20:39:252
273203995037,10cyclictest276184-21sshd20:08:262
273203995036,10cyclictest287720-21sshd23:51:262
273203995036,10cyclictest285212-21munin-run23:05:012
273203995036,10cyclictest284438-21munin-run22:50:012
273203995035,12cyclictest282830-21sshd22:15:252
273203995035,11cyclictest282592-21sshd22:13:252
273203995035,11cyclictest280072-21munin-run21:25:002
273203995033,13cyclictest277461-21sshd20:32:252
273203995032,13cyclictest286231-21munin-run23:25:012
27319899505,10cyclictest23-21ksoftirqd/123:50:021
273198995038,9cyclictest23-21ksoftirqd/121:40:011
273198995015,6cyclictest131rcu_preempt22:00:001
273196995036,11cyclictest276969-21sshd20:23:250
273196995036,10cyclictest276426-21sshd20:11:250
273196995036,10cyclictest276426-21sshd20:11:250
273196995036,10cyclictest275957-21munin-run20:05:000
273196995033,14cyclictest287218-21sshd23:40:250
273196995033,14cyclictest287218-21sshd23:40:250
273196995033,13cyclictest285165-21sshd23:00:250
273206994936,6cyclictest306-21in:imuxsock21:00:013
273203994940,6cyclictest281314-21sshd21:46:252
273203994936,8cyclictest281107-21munin-run21:45:012
273203994936,10cyclictest284176-21munin-run22:45:012
273203994936,10cyclictest283374-21sshd22:27:252
273203994936,10cyclictest283125-21sshd22:22:262
273203994936,10cyclictest281813-21sshd21:57:252
273203994936,10cyclictest281592-21munin-run21:55:002
273203994936,10cyclictest278227-21sshd20:45:252
273203994935,10cyclictest287223-21sshd23:41:252
273203994935,10cyclictest287223-21sshd23:41:252
273203994935,10cyclictest278765-21munin-run20:59:592
273203994935,10cyclictest275433-21munin-run19:55:002
273203994934,12cyclictest279521-21sshd21:14:252
273203994933,12cyclictest273676-21munin-run19:20:022
273203994933,11cyclictest48-21kcompactd023:11:462
273198994921,8cyclictest23-21ksoftirqd/123:55:011
273198994919,15cyclictest23-21ksoftirqd/123:20:301
273196994936,10cyclictest287716-21sshd23:50:250
273196994936,10cyclictest281574-21sshd21:54:250
273196994935,11cyclictest280791-21sshd21:36:250
273196994934,10cyclictest278260-21munin-run20:50:010
273196994933,12cyclictest278477-21sshd20:50:250
273196994931,14cyclictest363-21snmpd22:04:410
273196994927,16cyclictest272-21dbus-daemon22:30:010
273206994832,12cyclictest272-21dbus-daemon20:40:013
273206994830,12cyclictest235-21systemd-journal19:35:003
27320399486,4cyclictest131rcu_preempt23:40:002
273203994835,9cyclictest48-21kcompactd021:03:372
273203994835,9cyclictest48-21kcompactd019:49:502
273203994834,11cyclictest274914-21sshd19:43:242
273203994834,10cyclictest48-21kcompactd021:26:552
273203994833,10cyclictest235-21systemd-journal00:25:012
273198994830,14cyclictest272-21dbus-daemon22:40:011
27319899482,12cyclictest23-21ksoftirqd/100:00:001
273198994813,9cyclictest131rcu_preempt19:10:011
273198994812,6cyclictest131rcu_preempt22:55:001
273196994836,9cyclictest289779-21sshd00:30:250
273196994836,9cyclictest289548-21sshd00:29:250
273196994835,9cyclictest290044-21sshd00:39:250
273196994835,10cyclictest282857-21cron22:20:000
273196994831,12cyclictest285446-21munin-run23:10:000
273196994829,14cyclictest363-21snmpd19:51:360
27319699482,4cyclictest131rcu_preempt22:35:020
273196994822,21cyclictest282534-21latency_hist22:10:020
273196994822,21cyclictest282534-21latency_hist22:10:020
273206994735,7cyclictest282877-21munin-plugin-st22:20:003
273203994734,9cyclictest48-21kcompactd020:25:542
273203994734,10cyclictest276429-21sshd20:12:252
273203994734,10cyclictest276429-21sshd20:12:252
273203994733,11cyclictest284640-21sshd22:50:252
273203994732,12cyclictest277978-21sshd20:40:252
273203994731,13cyclictest48-21kcompactd023:55:032
273198994733,11cyclictest23-21ksoftirqd/120:10:001
27319899471,8cyclictest23-21ksoftirqd/121:10:251
273196994739,5cyclictest282582-21sshd22:10:250
273196994736,5cyclictest273192-21cyclictest19:10:010
273196994735,9cyclictest280293-21sshd21:26:260
273196994735,8cyclictest273608-21sshd19:17:240
273196994734,10cyclictest273877-21sshd19:20:250
273196994733,10cyclictest280571-21munin-run21:35:010
273196994727,15cyclictest363-21snmpd22:40:300
273203994634,9cyclictest278481-21sshd20:51:252
273203994633,9cyclictest48-21kcompactd023:27:522
273203994633,9cyclictest48-21kcompactd023:27:522
273203994633,10cyclictest279771-21sshd21:17:252
273203994633,10cyclictest279771-21sshd21:17:252
273203994632,11cyclictest280544-21sshd21:32:242
273203994628,14cyclictest289027-21cron00:19:592
273203994614,6cyclictest30-21ksoftirqd/219:35:252
273198994633,7cyclictest285999-21latency_hist23:20:011
273198994629,12cyclictest221rcuc/120:30:011
27319899462,8cyclictest23-21ksoftirqd/123:10:001
273198994611,5cyclictest131rcu_preempt19:20:031
273196994639,4cyclictest275691-21munin-run20:00:010
273196994637,5cyclictest281850-21munin-run22:00:010
273196994633,9cyclictest363-21snmpd00:23:420
273196994633,9cyclictest235-21systemd-journal21:00:000
273196994633,10cyclictest288545-21munin-run00:10:010
273196994632,11cyclictest281065-21sshd21:41:250
273196994616,26cyclictest277763-21modprobe20:40:000
273206994524,15cyclictest272-21dbus-daemon21:05:013
273203994537,5cyclictest276962-21sshd20:21:252
273203994534,8cyclictest48-21kcompactd000:33:222
273203994530,12cyclictest48-21kcompactd023:05:072
273203994522,19cyclictest273295-21sed19:10:022
27319899458,9cyclictest23-21ksoftirqd/100:35:001
273198994512,6cyclictest131rcu_preempt23:15:001
273198994512,6cyclictest131rcu_preempt00:03:251
273196994534,7cyclictest285707-21munin-run23:15:010
273196994533,9cyclictest363-21snmpd23:26:590
273196994533,9cyclictest363-21snmpd23:26:590
273196994533,9cyclictest285911-21sshd23:15:250
273206994435,7cyclictest37-21ksoftirqd/323:30:023
273206994435,7cyclictest37-21ksoftirqd/323:30:023
27319899443,7cyclictest23-21ksoftirqd/122:20:001
273198994433,8cyclictest23-21ksoftirqd/123:45:031
273198994433,8cyclictest23-21ksoftirqd/123:45:021
273198994433,8cyclictest23-21ksoftirqd/121:34:251
27319899441,8cyclictest23-21ksoftirqd/120:25:011
273196994437,4cyclictest276722-21munin-run20:20:000
273196994435,6cyclictest279510-21sshd21:11:250
273196994432,9cyclictest286706-21sshd23:33:250
273196994431,9cyclictest363-21snmpd22:22:120
27319699441,22cyclictest0-21swapper/020:10:020
273206994330,7cyclictest306-21in:imuxsock00:15:013
273206994326,11cyclictest1-21systemd19:55:013
27320699431,8cyclictest131rcu_preempt23:20:003
27319899436,8cyclictest23-21ksoftirqd/119:40:001
27319899434,8cyclictest23-21ksoftirqd/122:05:001
27319899432,10cyclictest23-21ksoftirqd/120:40:011
273196994335,4cyclictest277213-21sshd20:27:250
273196994330,9cyclictest363-21snmpd22:50:310
27320699429,16cyclictest290057-21modprobe00:40:003
273206994227,10cyclictest306-21in:imuxsock20:05:023
273206994225,9cyclictest279782-21cron21:20:003
273206994225,9cyclictest279782-21cron21:20:003
273206994223,13cyclictest272-21dbus-daemon23:25:013
273206994222,15cyclictest272-21dbus-daemon23:50:013
273206994221,15cyclictest235-21systemd-journal21:35:003
273206994220,15cyclictest272-21dbus-daemon22:15:003
273203994232,8cyclictest48-21kcompactd021:05:172
273203994224,15cyclictest48-21kcompactd022:55:082
27319899429,13cyclictest23-21ksoftirqd/122:45:011
273198994233,7cyclictest23-21ksoftirqd/100:05:041
273198994230,9cyclictest23-21ksoftirqd/122:30:001
273198994225,11cyclictest23-21ksoftirqd/123:00:001
273198994210,5cyclictest131rcu_preempt00:20:001
273206994129,10cyclictest37-21ksoftirqd/319:25:023
273206994127,8cyclictest306-21in:imuxsock23:00:013
273206994127,8cyclictest306-21in:imuxsock00:20:003
273206994123,13cyclictest0-21swapper/323:15:013
273203994135,3cyclictest48-21kcompactd020:01:292
273203994128,9cyclictest363-21snmpd00:25:272
273203994117,19cyclictest235-21systemd-journal22:05:252
273203994117,19cyclictest235-21systemd-journal22:05:252
27319899413,5cyclictest131rcu_preempt22:45:251
273198994133,6cyclictest23-21ksoftirqd/119:45:011
273198994129,9cyclictest235-21systemd-journal21:05:001
273198994127,10cyclictest23-21ksoftirqd/121:00:001
273198994114,18cyclictest221rcuc/121:55:011
273198994112,8cyclictest131rcu_preempt23:30:011
273198994112,8cyclictest131rcu_preempt23:30:011
273196994129,9cyclictest363-21snmpd00:14:270
273196994128,9cyclictest0-21swapper/023:22:550
273206994028,9cyclictest288258-21sshd00:02:253
273206994027,10cyclictest37-21ksoftirqd/323:05:403
273206994026,11cyclictest37-21ksoftirqd/300:25:003
273198994030,6cyclictest23-21ksoftirqd/122:10:011
273198994030,6cyclictest23-21ksoftirqd/122:10:011
273198994028,9cyclictest276698-21cron20:20:001
27319899402,7cyclictest23-21ksoftirqd/122:25:011
27319899401,5cyclictest131rcu_preempt19:11:261
27320699395,25cyclictest288004-21munin-run00:00:003
27320699393,4cyclictest131rcu_preempt21:40:003
273206993928,8cyclictest37-21ksoftirqd/322:50:013
273206993926,10cyclictest235-21systemd-journal20:27:253
273206993923,11cyclictest37-21ksoftirqd/321:55:013
273206993923,11cyclictest306-21in:imuxsock21:22:253
273206993916,16cyclictest1-21systemd19:20:003
27319899399,5cyclictest131rcu_preempt19:55:021
27319899395,6cyclictest131rcu_preempt23:33:251
273198993928,7cyclictest23-21ksoftirqd/119:49:591
273198993927,9cyclictest276437-21sshd20:14:251
273198993927,9cyclictest276437-21sshd20:14:251
273198993924,11cyclictest221rcuc/119:25:001
27319899392,11cyclictest23-21ksoftirqd/122:15:001
27319899392,10cyclictest23-21ksoftirqd/119:35:001
273198993910,6cyclictest131rcu_preempt20:35:001
273206993828,8cyclictest37-21ksoftirqd/323:35:003
273206993827,6cyclictest279314-21kworker/3:121:30:023
273206993825,10cyclictest37-21ksoftirqd/300:10:003
273206993822,12cyclictest281848-21systemctl22:00:003
273198993828,7cyclictest23-21ksoftirqd/119:30:021
273198993828,7cyclictest23-21ksoftirqd/119:30:021
273198993825,9cyclictest23-21ksoftirqd/123:05:011
27320699374,5cyclictest0-21swapper/320:35:003
273206993730,4cyclictest286977-21latency_hist23:40:013
273206993726,8cyclictest287234-21sshd23:44:253
273206993726,8cyclictest287234-21sshd23:44:253
273206993725,9cyclictest37-21ksoftirqd/322:45:013
273206993723,12cyclictest37-21ksoftirqd/319:10:013
27319899376,8cyclictest23-21ksoftirqd/120:40:251
273198993729,6cyclictest23-21ksoftirqd/121:50:011
273198993729,6cyclictest23-21ksoftirqd/121:20:021
273198993729,6cyclictest23-21ksoftirqd/121:20:021
273198993726,8cyclictest281088-21cron21:45:001
27319899372,5cyclictest131rcu_preempt20:55:021
27320699362,5cyclictest131rcu_preempt22:55:003
273206993624,8cyclictest37-21ksoftirqd/322:05:033
273206993623,11cyclictest37-21ksoftirqd/320:05:003
273206993623,10cyclictest37-21ksoftirqd/319:50:013
273206993621,11cyclictest1-21systemd00:35:003
273198993629,4cyclictest289291-21sshd00:22:251
27319899361,9cyclictest23-21ksoftirqd/121:30:011
273198993618,14cyclictest272-21dbus-daemon00:30:011
273206993528,4cyclictest1-21systemd20:25:013
273206993522,9cyclictest37-21ksoftirqd/321:45:013
273198993526,7cyclictest23-21ksoftirqd/120:05:011
27320699345,6cyclictest131rcu_preempt21:15:003
27320699345,3cyclictest131rcu_preempt20:55:003
273206993423,8cyclictest37-21ksoftirqd/319:45:023
273206993419,9cyclictest279270-21modprobe21:10:013
273206993416,14cyclictest272-21dbus-daemon19:15:013
273198993421,10cyclictest288757-21sshd00:12:251
273206993323,7cyclictest37-21ksoftirqd/323:05:003
273206993323,6cyclictest282354-21cron22:10:003
273206993323,6cyclictest282354-21cron22:10:003
273206993321,9cyclictest37-21ksoftirqd/321:50:023
273206993319,11cyclictest37-21ksoftirqd/300:30:013
273206993221,9cyclictest37-21ksoftirqd/319:30:013
273206993221,9cyclictest37-21ksoftirqd/319:30:013
27320699316,5cyclictest131rcu_preempt22:20:313
27320699313,4cyclictest131rcu_preempt19:37:253
273206993120,9cyclictest37-21ksoftirqd/322:25:033
273206993110,15cyclictest272-21dbus-daemon20:50:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional