You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 23:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Fri Apr 03, 2026 12:43:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
184373995634,17cyclictest330191-40mandb11:22:520
184374995446,5cyclictest380175-21ls12:25:001
184373995243,7cyclictest151ktimers/007:50:280
184375995132,12cyclictest0-21swapper/209:45:292
184374995041,6cyclictest249447-21aten_r2power_cu09:45:191
184375994839,4cyclictest0-21swapper/210:35:352
184374994838,7cyclictest3496356-21cups-browsed10:20:021
184375994739,5cyclictest327283-21systemd-run11:20:012
184375994738,6cyclictest421-21dbus-daemon09:25:322
184375994737,6cyclictest1-21systemd09:55:012
184375994736,7cyclictest3496356-21cups-browsed11:31:002
18437399474,31cyclictest365153-21ssh12:05:360
18437599462,38cyclictest320141-21ssh11:10:302
18437599461,33cyclictest356827-21ssh11:55:302
18437499462,37cyclictest352823-21ssh11:50:321
184373994537,6cyclictest151ktimers/007:55:330
184373994435,7cyclictest187365-21systemctl07:20:000
184373994435,7cyclictest187365-21systemctl07:20:000
184373994432,7cyclictest0-21swapper/007:30:280
18437599433,7cyclictest171rcu_preempt10:05:302
184375994335,5cyclictest380171-21sh12:25:002
184374994333,6cyclictest3496356-21cups-browsed12:33:571
184374994333,6cyclictest303709-21ntp_kernel_pll_10:50:311
184373994336,4cyclictest290268-21latency_hist10:35:020
184373994330,5cyclictest0-21swapper/009:20:310
184375994233,5cyclictest1-21systemd09:25:012
184374994214,25cyclictest324296-21sh11:15:311
184373994229,8cyclictest220348-21diskmemload11:15:340
184373994225,11cyclictest12350irq/32-eth009:55:310
184375994133,5cyclictest428-21mta-sts-daemon09:35:262
184375994133,5cyclictest314927-21ssh11:05:012
184374994133,5cyclictest206855-21cron08:25:001
184374994129,9cyclictest261785-21acpi10:00:181
18437399413,30cyclictest220348-21diskmemload09:25:280
184373994129,6cyclictest191rcuc/009:40:300
184373994129,6cyclictest191rcuc/009:40:300
184373994128,10cyclictest244912-21munin-run09:40:010
184373994119,16cyclictest12350irq/32-eth009:45:320
184374994031,6cyclictest232637-21munin-run09:25:011
184374994031,5cyclictest357038-21proc_pri11:55:361
184373994033,4cyclictest214322-21kworker/u8:3-xprtiod12:30:010
184373994029,7cyclictest3496356-21cups-browsed09:31:020
184373994027,6cyclictest0-21swapper/011:46:250
184373994026,4cyclictest0-21swapper/009:10:000
18437399402,31cyclictest385507-21ssh12:30:370
184375993932,4cyclictest191856-21munin-plugin-st07:35:012
18437499396,28cyclictest336490-21ssh11:30:291
184374993931,5cyclictest240883-21latency_hist09:35:021
184374993930,6cyclictest421-21dbus-daemon11:00:341
184374993929,7cyclictest197861-21sh07:55:011
184374993929,6cyclictest245950-21needreboot09:40:261
184374993929,6cyclictest245950-21needreboot09:40:261
184374993928,7cyclictest254501-21proc_pri09:50:401
18437499391,6cyclictest171rcu_preempt10:35:371
184373993930,5cyclictest275540-21vmstat10:15:430
184373993929,6cyclictest392377-21systemd-run12:40:010
184373993929,6cyclictest3496356-21cups-browsed10:12:530
184373993911,11cyclictest14-21ksoftirqd/011:35:360
184373993911,11cyclictest14-21ksoftirqd/011:35:360
184375993831,4cyclictest284390-21kworker/u8:4-nfsiod11:22:552
184375993829,5cyclictest316466-21timerandwakeup11:05:402
184375993828,6cyclictest351669-21munin-run11:50:012
184374993830,5cyclictest335463-21latency_hist11:30:021
184374993830,4cyclictest281976-21munin-plugin-st10:25:001
184374993828,6cyclictest220510-21latency_hist09:10:021
184374993827,7cyclictest267082-21timerandwakeup10:05:431
184374993825,10cyclictest241389-21cpu09:35:211
18437499382,28cyclictest316039-21ssh11:05:291
184373993820,14cyclictest0-21swapper/010:25:010
184375993728,6cyclictest439-21systemd-logind09:20:012
184375993727,6cyclictest333145-21unixbench_singl11:25:442
18437599371,28cyclictest381497-21ps12:25:392
184374993732,2cyclictest0-21swapper/111:22:521
184374993730,4cyclictest0-21swapper/109:10:341
184374993729,5cyclictest384238-21sh12:30:011
184374993728,5cyclictest302625-21latency_hist10:50:021
184374993727,7cyclictest212857-21grep08:45:011
184374993727,7cyclictest1-21systemd09:20:011
184374993725,9cyclictest270847-21ntp_offset10:10:351
184373993733,2cyclictest0-21swapper/011:00:020
184373993733,2cyclictest0-21swapper/011:00:020
18437399373,25cyclictest356808-21ssh11:55:290
184373993727,7cyclictest254558-21mailstats09:50:400
184373993723,6cyclictest311926-21ssh11:00:310
184375993631,2cyclictest0-21swapper/212:40:012
184375993628,4cyclictest224425-21cron09:15:012
184375993628,3cyclictest0-21swapper/212:35:002
184375993627,5cyclictest277829-21cron10:20:012
184375993626,6cyclictest3496356-21cups-browsed10:55:262
184375993626,6cyclictest3496356-21cups-browsed10:55:262
184374993628,5cyclictest373781-21unixbench_singl12:15:421
184374993628,5cyclictest373781-21unixbench_singl12:15:421
184374993628,4cyclictest271ktimers/112:10:341
18437399367,9cyclictest3496356-21cups-browsed07:20:340
18437399367,9cyclictest3496356-21cups-browsed07:20:340
18437399363,25cyclictest659-21snmpd10:50:340
184373993627,6cyclictest3496356-21cups-browsed11:51:110
18437399362,10cyclictest14-21ksoftirqd/012:00:280
184375993529,3cyclictest0-21swapper/210:55:022
184375993528,3cyclictest0-21swapper/210:45:352
184375993526,6cyclictest420-21avahi-daemon10:05:022
184374993526,6cyclictest1-21systemd11:40:021
184374993526,6cyclictest1-21systemd11:40:021
184374993526,5cyclictest220348-21diskmemload10:55:191
184374993526,5cyclictest220348-21diskmemload10:55:191
184374993525,7cyclictest209158-21needreboot08:30:271
184373993528,4cyclictest151ktimers/011:40:330
184373993527,5cyclictest1-21systemd07:40:010
184373993526,6cyclictest3496356-21cups-browsed10:37:500
184373993526,4cyclictest0-21swapper/008:05:340
18437399352,27cyclictest295457-21ssh10:40:300
184373993510,4cyclictest171rcu_preempt12:18:530
184373993510,4cyclictest171rcu_preempt12:18:530
184375993423,8cyclictest286992-21latency10:30:242
18437499343,21cyclictest365048-21ssh12:05:321
184374993426,5cyclictest200732-21unixbench_multi08:00:401
184374993424,7cyclictest0-21swapper/112:35:191
184374993424,6cyclictest1051-21mta-sts-daemon09:59:001
184374993422,9cyclictest185027-21latency07:10:261
184374993422,9cyclictest185027-21latency07:10:261
184374993420,11cyclictest347984-21/usr/sbin/munin11:45:181
184373993428,3cyclictest0-21swapper/007:15:000
184373993428,3cyclictest0-21swapper/007:15:000
184373993426,6cyclictest217335-21sendmail09:00:010
184373993425,6cyclictest220732-21/usr/sbin/munin09:10:260
184373993425,4cyclictest300167-21tail10:45:430
18437399342,26cyclictest332442-21ssh11:25:300
18437399342,24cyclictest225583-21ssh09:15:310
184373993421,4cyclictest171rcu_preempt10:05:250
184375993326,4cyclictest206866-21cut08:25:002
184375993326,3cyclictest0-21swapper/209:35:012
184375993325,5cyclictest286156-21latency_hist10:30:022
184375993325,4cyclictest0-21swapper/210:40:332
18437599332,20cyclictest220348-21diskmemload12:00:362
184374993328,2cyclictest0-21swapper/108:40:011
184374993320,9cyclictest0-21swapper/111:40:321
18437399333,20cyclictest206215-21ntpq08:20:280
184373993325,4cyclictest374962-21kworker/u8:13-rpciod12:24:560
184373993320,10cyclictest1-21systemd12:10:540
184373993319,12cyclictest151ktimers/011:10:280
184373993310,9cyclictest14-21ksoftirqd/010:30:010
184375993226,3cyclictest0-21swapper/207:45:012
184375993222,4cyclictest0-21swapper/210:10:382
18437599321,24cyclictest344660-21ssh11:40:322
18437499325,21cyclictest320239-21ssh11:10:331
18437499322,8cyclictest233788-21ssh09:25:301
184374993227,2cyclictest0-21swapper/107:50:021
184373993225,5cyclictest1051-21mta-sts-daemon11:06:210
184373993223,5cyclictest336701-21wc11:30:360
184375993123,6cyclictest353294-21timerwakeupswit11:50:412
184375993121,7cyclictest753-21ntpd10:20:292
18437499313,22cyclictest287252-21ssh10:30:311
184374993125,3cyclictest0-21swapper/108:50:011
184374993125,3cyclictest0-21swapper/108:00:011
184374993123,5cyclictest295045-21fschecks_time10:40:221
184374993122,7cyclictest215589-21timerandwakeup08:50:401
184374993114,12cyclictest271ktimers/107:35:311
18437399315,8cyclictest14-21ksoftirqd/008:50:330
18437399312,9cyclictest14-21ksoftirqd/008:40:330
184373993124,4cyclictest191rcuc/008:17:170
184373993124,4cyclictest191rcuc/008:17:170
184373993117,9cyclictest12350irq/32-eth008:30:300
184373993114,6cyclictest171rcu_preempt10:00:310
184375993025,2cyclictest0-21swapper/212:20:012
184375993025,2cyclictest0-21swapper/212:20:012
184375993025,2cyclictest0-21swapper/209:00:012
184375993024,3cyclictest0-21swapper/211:35:362
184375993024,3cyclictest0-21swapper/211:35:362
184375993023,4cyclictest261377-21basename10:00:012
184375993022,5cyclictest212858-21munin-plugin-st08:45:012
184375993022,5cyclictest187354-21sh07:20:002
184375993022,5cyclictest187354-21sh07:20:002
184375993022,3cyclictest0-21swapper/209:40:352
184375993022,3cyclictest0-21swapper/209:40:352
184375993021,7cyclictest369104-21ntp_kernel_pll_12:10:322
18437399306,9cyclictest14-21ksoftirqd/009:00:310
184373993020,3cyclictest171rcu_preempt07:30:010
184373993011,16cyclictest191rcuc/008:38:010
184375992922,3cyclictest0-21swapper/208:15:012
184375992922,3cyclictest0-21swapper/208:15:012
184374992923,3cyclictest0-21swapper/109:00:011
18437399293,7cyclictest14-21ksoftirqd/008:25:300
184373992916,3cyclictest171rcu_preempt08:50:020
184373992916,11cyclictest0-21swapper/007:40:220
184375992820,5cyclictest202247-21unixbench_multi08:05:412
184375992820,5cyclictest196782-21cpu07:50:202
184375992819,6cyclictest516-21in:imuxsock08:35:012
184375992814,11cyclictest1056-21mta-sts-daemon07:10:332
184375992814,11cyclictest1056-21mta-sts-daemon07:10:332
18437599281,22cyclictest365194-21ssh12:05:372
18437499283,21cyclictest624-21lldpd07:20:271
18437499283,21cyclictest624-21lldpd07:20:271
184374992822,3cyclictest0-21swapper/112:00:191
184374992822,3cyclictest0-21swapper/107:20:001
184374992822,3cyclictest0-21swapper/107:20:001
184374992816,9cyclictest203155-21needreboot08:10:281
184374992816,9cyclictest203155-21needreboot08:10:281
184373992822,4cyclictest191rcuc/008:14:420
184373992822,4cyclictest191rcuc/008:14:420
184373992821,5cyclictest151ktimers/007:45:280
18437399281,18cyclictest200254-21ntpq08:00:300
184375992721,3cyclictest0-21swapper/207:25:012
184375992721,3cyclictest0-21swapper/207:25:012
184375992720,4cyclictest3496356-21cups-browsed08:35:052
184375992720,4cyclictest1051-21mta-sts-daemon09:00:232
184375992720,4cyclictest0-21swapper/208:20:002
184375992720,4cyclictest0-21swapper/208:20:002
184375992719,3cyclictest0-21swapper/207:25:292
184374992717,7cyclictest218373-21proc_pri09:00:361
184374992717,4cyclictest0-21swapper/110:25:401
184373992719,5cyclictest184434-21latency_hist07:10:020
184375992621,3cyclictest1054-21mta-sts-daemon08:50:182
18437499268,4cyclictest202205-21unixbench-2d08:05:411
184375992518,4cyclictest0-21swapper/208:05:012
184375992517,6cyclictest196351-21latency_hist07:50:022
18437599251,19cyclictest213818-21sendmail-mta08:45:332
18437499253,3cyclictest207731-21ntpq08:25:291
184374992516,5cyclictest184378-21systemd-run07:10:011
18437499244,14cyclictest3496356-21cups-browsed07:30:281
184375992316,5cyclictest0-21swapper/209:10:012
184374992317,3cyclictest0-21swapper/107:25:231
184374992316,4cyclictest1056-21mta-sts-daemon07:40:321
18437599223,14cyclictest441-21gpm08:25:282
184375992216,3cyclictest0-21swapper/207:35:412
184374992213,6cyclictest548-21gdbus08:19:521
184374992213,6cyclictest548-21gdbus08:19:521
184375992115,3cyclictest1054-21mta-sts-daemon07:58:232
18437699192,8cyclictest0-21swapper/310:10:343
18437699182,9cyclictest0-21swapper/309:40:293
18437699182,9cyclictest0-21swapper/309:40:293
18437699182,8cyclictest0-21swapper/310:30:343
18437699181,11cyclictest0-21swapper/310:15:303
18437699162,7cyclictest0-21swapper/312:20:363
18437699162,7cyclictest0-21swapper/311:50:293
18437699162,7cyclictest0-21swapper/309:55:313
18437599161,11cyclictest184380-21true07:10:012
18437699154,7cyclictest0-21swapper/311:22:563
18437699153,7cyclictest0-21swapper/309:20:313
18437699152,8cyclictest0-21swapper/312:05:313
18437699152,8cyclictest0-21swapper/310:40:013
18437699152,8cyclictest0-21swapper/309:25:283
18437699151,8cyclictest0-21swapper/311:00:323
18437699151,3cyclictest0-21swapper/312:25:313
18437699143,8cyclictest0-21swapper/310:05:333
18437699143,7cyclictest0-21swapper/311:35:283
18437699143,7cyclictest0-21swapper/311:35:283
18437699143,6cyclictest0-21swapper/309:50:333
18437699142,7cyclictest0-21swapper/312:00:293
18437699142,7cyclictest0-21swapper/312:00:013
18437699142,7cyclictest0-21swapper/311:40:313
18437699142,7cyclictest0-21swapper/311:25:313
18437699142,7cyclictest0-21swapper/310:40:293
18437699142,7cyclictest0-21swapper/310:20:353
18437699142,7cyclictest0-21swapper/309:30:313
18437699142,6cyclictest0-21swapper/308:45:343
18437699141,9cyclictest0-21swapper/311:30:283
18437699141,7cyclictest0-21swapper/309:15:233
18437699141,6cyclictest0-21swapper/312:35:263
18437699133,6cyclictest0-21swapper/311:05:333
18437699133,5cyclictest0-21swapper/308:05:013
18437699132,7cyclictest0-21swapper/310:50:253
18437699132,6cyclictest0-21swapper/307:20:003
18437699132,6cyclictest0-21swapper/307:20:003
18437699131,8cyclictest0-21swapper/312:30:313
18437699131,8cyclictest0-21swapper/312:10:353
18437699131,8cyclictest0-21swapper/309:10:253
18437699131,7cyclictest0-21swapper/312:15:293
18437699131,7cyclictest0-21swapper/312:15:293
18437699131,7cyclictest0-21swapper/311:45:203
18437699131,7cyclictest0-21swapper/311:10:323
18437699131,7cyclictest0-21swapper/309:45:193
18437699131,7cyclictest0-21swapper/309:35:303
18437699131,6cyclictest0-21swapper/309:05:323
18437699131,6cyclictest0-21swapper/308:50:313
18437699131,6cyclictest0-21swapper/308:25:003
18437699131,6cyclictest0-21swapper/307:35:303
18437699123,6cyclictest0-21swapper/310:55:193
18437699123,6cyclictest0-21swapper/310:55:193
18437699122,6cyclictest0-21swapper/310:25:343
18437699122,6cyclictest0-21swapper/310:00:283
18437699121,7cyclictest0-21swapper/310:45:293
18437699121,7cyclictest0-21swapper/308:35:303
18437699121,7cyclictest0-21swapper/307:45:353
18437699121,6cyclictest0-21swapper/308:55:273
18437699121,6cyclictest0-21swapper/308:40:283
18437699121,6cyclictest0-21swapper/307:30:003
18437699121,6cyclictest0-21swapper/307:10:413
18437699121,6cyclictest0-21swapper/307:10:413
18437699121,5cyclictest0-21swapper/307:30:233
18437699113,4cyclictest0-21swapper/311:15:313
18437699111,6cyclictest0-21swapper/308:30:353
18437699111,6cyclictest0-21swapper/308:25:323
18437699111,6cyclictest0-21swapper/308:00:013
18437699111,6cyclictest0-21swapper/307:50:353
18437699111,6cyclictest0-21swapper/307:40:293
18437699111,6cyclictest0-21swapper/307:20:313
18437699111,6cyclictest0-21swapper/307:20:313
18437699111,5cyclictest0-21swapper/308:05:193
18437699103,5cyclictest0-21swapper/308:15:393
18437699103,5cyclictest0-21swapper/308:15:393
18437699101,5cyclictest0-21swapper/308:10:203
18437699101,5cyclictest0-21swapper/308:10:203
1843769992,4cyclictest0-21swapper/309:00:363
1843769981,4cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional