You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 22:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Sep 03, 2026 12:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3171132999086,2cyclictest0-21swapper/010:35:020
3171132998825,33cyclictest0-21swapper/012:26:170
317113299871,83cyclictest0-21swapper/010:30:030
3171132998518,57cyclictest16850irq/134-800000011:12:100
3171132998427,52cyclictest3210965-21ssh10:20:310
3171137998316,34cyclictest3232439-21sh11:05:551
3171132998320,32cyclictest3266900-21rm12:19:280
317113299822,48cyclictest3188543-21ssh09:33:310
3171132998132,45cyclictest3181776-21ssh09:19:210
3171132998028,35cyclictest3209242-21ssh10:17:110
3171132998022,48cyclictest16850irq/134-800000011:34:020
3171132998017,34cyclictest3239363-21ssh11:20:400
317113299792,72cyclictest48-21kcompactd010:40:170
317113299792,42cyclictest3189470-21ssh09:35:070
317113299782,72cyclictest3196932-21ssh09:50:580
317113299782,45cyclictest3467602-21systemd-journal09:24:390
317113299771,74cyclictest0-21swapper/010:05:020
3171132997628,44cyclictest3256751-21ssh11:57:540
317113299762,33cyclictest3174490-21modprobe08:14:590
317113299761,69cyclictest3223042-21ssh10:45:580
317113299761,68cyclictest3201100-21latency_hist10:00:020
3171132997531,27cyclictest3227081-21ssh10:54:540
3171132997524,18cyclictest0-21swapper/010:35:220
317113799733,61cyclictest3262797-21ssh12:10:311
317113799733,61cyclictest3262797-21ssh12:10:311
3171132997322,46cyclictest16850irq/134-800000010:06:410
317113299721,55cyclictest3195357-21ssh09:47:490
317114599713,55cyclictest3241355-21ssh11:25:023
3171132997159,3cyclictest0-21swapper/008:10:020
3171132997112,55cyclictest16850irq/134-800000010:59:020
3171132997030,24cyclictest3259418-21ssh12:03:310
317113299702,63cyclictest3186851-21latency_hist09:30:010
3171132997010,55cyclictest16850irq/134-800000012:12:240
3171132997010,55cyclictest16850irq/134-800000012:12:240
3171137996930,35cyclictest3208627-21sh10:15:451
3171132996930,22cyclictest3238758-21ssh11:19:540
3171132996919,31cyclictest3272282-21ssh12:30:380
3171132996916,37cyclictest3247055-21ssh11:37:180
317113299691,2cyclictest0-21swapper/011:00:100
3171137996817,34cyclictest23-21ksoftirqd/109:35:011
317113299685,58cyclictest16850irq/134-800000011:27:440
3171132996821,42cyclictest16850irq/134-800000009:14:370
3171132996818,41cyclictest16850irq/134-800000009:42:210
317113299681,6cyclictest3276518-21/usr/sbin/munin12:40:000
317113799676,3cyclictest131rcu_preempt08:20:031
317113799676,3cyclictest131rcu_preempt08:20:021
3171137996728,35cyclictest3213511-21sh10:25:511
3171137996631,31cyclictest3467602-21systemd-journal12:31:221
3171137996622,39cyclictest1-21systemd10:55:011
3171137996619,41cyclictest275-21dbus-daemon11:55:011
3171137996619,41cyclictest275-21dbus-daemon11:55:001
3171132996532,28cyclictest275-21dbus-daemon07:40:000
317113299651,60cyclictest3252958-21ssh11:50:010
3171137996427,33cyclictest3217248-21sh10:34:041
3171132996422,26cyclictest3206461-21ssh10:11:120
3171137996324,35cyclictest3201469-21sh10:00:321
317113799632,20cyclictest3198468-21ssh09:54:321
3171132996319,25cyclictest16850irq/134-800000011:08:370
317113799622,54cyclictest3266101-21ssh12:17:361
317113799622,18cyclictest3221039-21ssh10:41:421
3171132996227,31cyclictest3268925-21ssh12:23:370
317113299622,55cyclictest3254379-21ssh11:52:520
317113299622,55cyclictest3254379-21ssh11:52:520
317113299621,5cyclictest0-21swapper/011:43:160
3171137996122,35cyclictest3191186-21sh09:39:061
317113299615,40cyclictest3171135-21latency_hist07:10:010
3171137996029,27cyclictest3175016-21cron08:25:001
3171137996022,34cyclictest3177358-21diskmemload09:26:181
3171137996017,37cyclictest3467602-21systemd-journal09:00:001
3171137996016,39cyclictest3276518-21sh12:40:011
317113299602,26cyclictest3177517-21latency_hist09:10:020
317113799592,16cyclictest3193999-21ssh09:45:011
317113799592,14cyclictest3259635-21ssh12:04:071
3171137995916,39cyclictest3238101-1kworker/u9:111:31:161
3171141995834,16cyclictest275-21dbus-daemon07:20:022
3171137995824,30cyclictest3177358-21diskmemload12:27:461
3171137995822,32cyclictest3229951-1kworker/u9:011:13:491
3171137995819,35cyclictest3230795-21sh11:02:331
317113799581,4cyclictest3249649-21ssh11:42:511
3171132995819,35cyclictest3175281-21/usr/sbin/munin08:30:020
317113299581,2cyclictest0-21swapper/008:45:010
3171141995727,10cyclictest30-21ksoftirqd/211:10:492
3171137995730,23cyclictest3217977-21sh10:35:181
317113799572,49cyclictest3175772-21cron08:40:001
3171137995718,35cyclictest3177358-21diskmemload09:59:331
317113299575,35cyclictest326132050ssh12:07:290
317113299572,52cyclictest0-21swapper/008:15:020
317113299572,52cyclictest0-21swapper/008:15:020
317113299572,51cyclictest0-21swapper/008:05:020
317113299572,50cyclictest3172149-21cron07:30:000
3171141995646,7cyclictest3173755-21kworker/2:109:10:002
317114199562,4cyclictest131rcu_preempt10:20:002
3171137995630,22cyclictest3252927-21sh11:50:001
317113799561,8cyclictest3182380-21ssh09:20:121
3171137995618,34cyclictest3238101-1kworker/u9:111:24:331
317113799561,7cyclictest3212194-21ssh10:23:181
317113799561,5cyclictest3257165-21ssh11:58:571
317113799561,16cyclictest3243134-21sh11:29:091
317113299562,49cyclictest3176570-21/usr/sbin/munin08:55:000
317113299561,51cyclictest3173446-21latency_hist07:55:020
317113799553,45cyclictest3171369-21munin-run07:14:591
3171137995516,35cyclictest3229405-21rm10:59:531
317113799551,5cyclictest3206583-21ssh10:11:281
317113799551,4cyclictest3268320-21ssh12:22:131
317113799551,3cyclictest3223829-21ssh10:47:491
317113799551,17cyclictest3180021-21ssh09:15:101
317113299552,48cyclictest3175771-21cron08:39:590
3171132995524,27cyclictest3172945-21/usr/sbin/munin07:45:010
3171132995524,27cyclictest3172945-21/usr/sbin/munin07:45:010
317113299551,5cyclictest0-21swapper/007:25:000
317113799542,5cyclictest3172654-21cron07:40:001
317113799542,18cyclictest3171643-21cut07:20:001
317113799541,9cyclictest3194353-21ssh09:45:211
3171137995419,31cyclictest23-21ksoftirqd/108:10:031
317113799541,8cyclictest3247842-21ssh11:39:061
317113799541,15cyclictest3204360-21ssh10:06:481
3171145995334,14cyclictest3467602-21systemd-journal12:20:003
317114599532,24cyclictest3184565-21ssh09:25:023
3171145995316,26cyclictest3173694-21modprobe08:00:003
3171141995344,6cyclictest3173755-21kworker/2:110:21:432
317113799532,17cyclictest3236853-21ssh11:15:231
3171137995318,29cyclictest3172434-21cat07:35:011
317113799531,14cyclictest3261145-21sh12:07:061
3171145995237,10cyclictest37-21ksoftirqd/309:30:013
3171141995243,6cyclictest3173755-21kworker/2:111:26:102
3171141995241,7cyclictest3174263-21kworker/2:310:57:132
3171141995236,12cyclictest1-21systemd07:55:012
3171137995222,26cyclictest3178773-21sh09:12:511
317113799522,16cyclictest3177476-21grep09:10:021
317113799521,45cyclictest3174586-21latency_hist08:15:021
317113299529,37cyclictest3467602-21systemd-journal07:50:000
3171145995117,8cyclictest131rcu_preempt10:59:433
3171141995143,5cyclictest3174263-21kworker/2:310:54:012
3171141995142,5cyclictest3173755-21kworker/2:109:39:322
317113799513,7cyclictest3176376-21latency_hist08:50:021
317113799512,7cyclictest3172148-21cron07:30:001
3171145995029,13cyclictest3172653-21cron07:40:003
3171141995042,5cyclictest3173755-21kworker/2:111:38:062
3171141995042,5cyclictest3173755-21kworker/2:110:37:242
3171141995041,6cyclictest3173755-21kworker/2:110:44:152
3171141995041,6cyclictest3173755-21kworker/2:109:42:482
317113799508,35cyclictest275-21dbus-daemon07:50:011
317113799502,6cyclictest3176530-21modprobe08:50:031
317113299502,43cyclictest3171684-21latency_hist07:20:020
317113299502,41cyclictest3175213-21latency_hist08:25:010
317113299501,2cyclictest0-21swapper/009:00:000
317114599491,5cyclictest131rcu_preempt09:41:323
3171141994941,5cyclictest3174263-21kworker/2:310:06:422
3171141994940,6cyclictest3174263-21kworker/2:311:07:562
3171141994940,5cyclictest3173755-21kworker/2:111:23:072
3171141994938,6cyclictest3249018-21kworker/2:211:50:002
3171141994929,15cyclictest3467602-21systemd-journal10:46:222
317113799493,6cyclictest3176052-21cron08:44:591
317113799493,40cyclictest1-21systemd07:25:011
317113799491,14cyclictest3173979-21modprobe08:05:011
317113299492,42cyclictest1-21systemd07:35:020
3171132994917,27cyclictest3176343-21sort08:50:010
3171145994831,12cyclictest3467602-21systemd-journal11:40:013
3171145994830,14cyclictest3467602-21systemd-journal11:32:213
3171145994817,21cyclictest361rcuc/309:56:373
3171141994840,5cyclictest3272141-21kworker/2:012:37:112
3171141994840,5cyclictest3174263-21kworker/2:309:23:012
3171141994840,5cyclictest3173755-21kworker/2:112:30:052
3171141994840,5cyclictest3173755-21kworker/2:111:56:492
3171141994840,5cyclictest3173755-21kworker/2:111:54:492
3171141994840,5cyclictest3173755-21kworker/2:111:54:492
317113799485,37cyclictest3171139-21ls07:10:021
317113799482,3cyclictest0-21swapper/107:45:011
317113799482,3cyclictest0-21swapper/107:45:001
317113799482,39cyclictest3173711-21munin-run08:00:001
317113299481,42cyclictest3177098-21latency_hist09:05:020
317113299481,42cyclictest3177098-21latency_hist09:05:010
317113299481,42cyclictest1-21systemd07:10:040
3171145994711,5cyclictest3195825-21ssh09:48:493
3171141994739,5cyclictest3174263-21kworker/2:309:53:222
3171141994739,5cyclictest3173755-21kworker/2:112:23:362
3171141994739,5cyclictest3173755-21kworker/2:112:19:562
3171141994736,8cyclictest1-21systemd08:35:002
3171141994729,12cyclictest30-21ksoftirqd/212:05:132
317113799471,10cyclictest3177302-21cut09:05:021
317113299474,38cyclictest3173927-21latency_hist08:00:010
317113299472,41cyclictest3175504-21sshd08:31:220
317114599469,6cyclictest131rcu_preempt12:08:173
3171141994638,5cyclictest3173755-21kworker/2:109:28:112
3171141994637,6cyclictest3174263-21kworker/2:310:03:432
3171141994637,6cyclictest3173755-21kworker/2:109:12:522
3171141994637,5cyclictest3173755-21kworker/2:111:19:552
3171141994611,31cyclictest3172637-21kworker/u8:007:50:032
317113799463,38cyclictest3173549-21grep07:55:021
317113799462,4cyclictest3175480-21latency_hist08:30:011
317113799462,10cyclictest3177280-21wc09:05:031
317114599458,6cyclictest131rcu_preempt11:05:033
3171145994536,5cyclictest37-21ksoftirqd/311:55:013
3171145994536,5cyclictest37-21ksoftirqd/311:55:003
3171145994532,10cyclictest37-21ksoftirqd/307:30:013
3171145994528,13cyclictest3205740-21cron10:10:013
3171145994520,17cyclictest0-21swapper/310:20:003
3171145994514,8cyclictest131rcu_preempt10:43:203
3171141994537,5cyclictest3249018-21kworker/2:211:44:212
3171141994537,5cyclictest3161967-21kworker/2:207:28:032
3171145994432,7cyclictest309-21in:imuxsock07:25:013
3171145994431,9cyclictest3206127-21sshd10:10:213
3171145994430,9cyclictest3191582-21systemctl09:40:013
3171145994429,11cyclictest3173209-21systemd-run07:50:003
3171145994426,13cyclictest275-21dbus-daemon12:25:013
3171145994425,15cyclictest309-21in:imuxsock10:01:233
3171145994422,16cyclictest3467602-21systemd-journal11:49:593
317114199446,5cyclictest131rcu_preempt12:01:282
3171141994437,4cyclictest3174263-21kworker/2:309:48:492
3171141994436,5cyclictest3174263-21kworker/2:310:10:242
3171141994436,5cyclictest3174263-21kworker/2:309:55:422
3171141994436,5cyclictest3173755-21kworker/2:108:49:472
3171141994436,5cyclictest3173755-21kworker/2:108:44:222
3171141994435,6cyclictest3173755-21kworker/2:112:27:222
3171141994435,5cyclictest3161967-21kworker/2:207:45:012
3171141994435,5cyclictest3161967-21kworker/2:207:45:002
3171141994426,13cyclictest1-21systemd11:35:012
3171141994422,17cyclictest275-21dbus-daemon08:55:012
317113799441,16cyclictest3175566-21date08:35:021
317114599436,14cyclictest37-21ksoftirqd/312:40:013
3171145994334,6cyclictest37-21ksoftirqd/308:05:003
3171145994333,8cyclictest37-21ksoftirqd/311:26:343
3171145994332,8cyclictest1-21systemd12:00:013
3171145994329,11cyclictest37-21ksoftirqd/309:30:193
3171145994329,10cyclictest37-21ksoftirqd/309:14:393
3171145994328,12cyclictest37-21ksoftirqd/308:35:013
3171145994327,12cyclictest37-21ksoftirqd/308:55:003
3171145994327,11cyclictest37-21ksoftirqd/310:46:563
317114199432,10cyclictest3467718-21lldpd07:32:562
317114599428,28cyclictest3234228-21sh11:10:013
3171145994232,5cyclictest3180511-21sh09:16:223
3171145994230,9cyclictest37-21ksoftirqd/308:47:233
3171145994230,8cyclictest3467602-21systemd-journal12:03:223
3171145994226,13cyclictest37-21ksoftirqd/307:14:593
3171141994234,5cyclictest3173755-21kworker/2:110:26:502
317114199422,13cyclictest346771850lldpd09:18:582
3171145994131,5cyclictest3177358-21diskmemload10:29:213
3171145994130,7cyclictest3177357-21latency_hist09:10:003
3171145994128,10cyclictest37-21ksoftirqd/311:45:013
3171145994128,10cyclictest3212908-21systemd-run10:25:013
3171145994128,10cyclictest1-21systemd09:05:013
3171145994128,10cyclictest1-21systemd09:05:003
3171145994114,16cyclictest37-21ksoftirqd/311:11:413
317114199415,6cyclictest0-21swapper/210:33:292
3171141994135,3cyclictest3174263-21kworker/2:311:01:562
3171141994132,6cyclictest3173667-21sshd07:55:222
317114599403,5cyclictest131rcu_preempt10:55:003
3171145994027,10cyclictest37-21ksoftirqd/310:31:433
3171141994029,8cyclictest30-21ksoftirqd/207:21:222
3171141994024,12cyclictest3264738-21munin-run12:15:002
3171141994024,12cyclictest3264738-21munin-run12:14:592
3171145993929,7cyclictest37-21ksoftirqd/312:27:483
3171145993928,8cyclictest37-21ksoftirqd/312:10:273
3171145993928,8cyclictest37-21ksoftirqd/312:10:263
3171145993928,8cyclictest37-21ksoftirqd/307:53:563
3171145993924,12cyclictest37-21ksoftirqd/311:15:223
3171141993932,4cyclictest3175027-21latency_hist08:24:592
3171141993929,6cyclictest3171356-21cron07:15:002
3171145993826,9cyclictest37-21ksoftirqd/312:32:303
3171145993824,11cyclictest37-21ksoftirqd/310:35:173
3171141993829,6cyclictest3174263-21kworker/2:308:59:132
3171141993823,11cyclictest3175770-21cron08:40:002
3171141993810,6cyclictest131rcu_preempt08:09:232
317114599377,5cyclictest131rcu_preempt07:45:013
317114599377,5cyclictest131rcu_preempt07:45:003
3171145993729,5cyclictest3176825-21latency_hist09:00:003
317114599371,5cyclictest131rcu_preempt09:51:053
3171145993712,5cyclictest131rcu_preempt08:24:593
3171141993725,9cyclictest30-21ksoftirqd/209:34:032
3171141993720,12cyclictest30-21ksoftirqd/208:30:022
317114599366,4cyclictest131rcu_preempt08:20:033
317114599366,4cyclictest131rcu_preempt08:20:023
3171145993624,9cyclictest275-21dbus-daemon08:45:013
3171145993623,8cyclictest309-21in:imuxsock07:30:033
3171145993622,11cyclictest37-21ksoftirqd/308:38:223
3171145993621,12cyclictest3174472-21sshd08:14:223
3171145993524,7cyclictest3175300-21latency_hist08:30:013
3171141993524,8cyclictest30-21ksoftirqd/208:11:412
3171141993524,7cyclictest3177080-21sshd09:03:232
3171141993524,7cyclictest3177080-21sshd09:03:222
3171145993422,9cyclictest3171605-21sshd07:17:223
317114199348,6cyclictest131rcu_preempt08:15:222
317114199348,6cyclictest131rcu_preempt08:15:212
3171141993426,5cyclictest3172800-21latency_hist07:40:012
3171145993324,5cyclictest3171253-21latency_hist07:10:033
3171145993321,10cyclictest37-21ksoftirqd/308:09:003
3171141993218,10cyclictest30-21ksoftirqd/208:05:032
3171141993018,9cyclictest3171170-21date07:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional