You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 15:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Tue Apr 07, 2026 12:43:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
189131995643,10cyclictest303928-21/usr/sbin/munin10:45:280
189133995445,5cyclictest0-21swapper/210:05:352
18913399482,35cyclictest328956-21ssh11:15:402
18913399477,33cyclictest242543-21ssh09:30:302
18913399477,33cyclictest242543-21ssh09:30:302
189132994729,12cyclictest0-21swapper/110:35:371
18913399465,33cyclictest250905-21sh09:40:352
189133994638,4cyclictest0-21swapper/211:20:302
18913299462,38cyclictest283718-21ssh10:20:321
189131994620,22cyclictest12450irq/33-eth009:25:330
189133994538,3cyclictest0-21swapper/212:35:282
189133994537,4cyclictest0-21swapper/207:55:252
189132994535,6cyclictest327842-21latency_hist11:15:021
189131994532,6cyclictest3521708-21cups-browsed10:05:400
189133994231,6cyclictest247146-21grep09:35:442
18913399422,33cyclictest337118-21ssh11:25:372
189132994227,11cyclictest261-21systemd-journal10:10:481
189132994215,8cyclictest421-21dbus-daemon11:55:021
189131994236,4cyclictest347977-21ssh11:40:010
189131994225,10cyclictest12350irq/32-eth010:50:300
189133994131,6cyclictest388701-21latency_hist12:30:032
189132994133,5cyclictest421-21dbus-daemon09:20:011
189132994133,5cyclictest210161-21sendmail-msp08:20:011
189132994131,6cyclictest420-21avahi-daemon12:23:581
189131994133,5cyclictest1-21systemd12:25:010
189131994127,11cyclictest202957-40mandb07:55:230
189131994125,13cyclictest221497-21ntpq08:55:310
189133994029,7cyclictest420-21avahi-daemon10:48:352
189133994027,8cyclictest325107-21timerandwakeup11:10:422
18913399402,27cyclictest316632-21ssh11:00:372
18913399402,27cyclictest316632-21ssh11:00:372
18913299408,27cyclictest353391-21ssh11:45:401
189132994032,5cyclictest189142-21systemd-run07:10:011
189132994032,5cyclictest189142-21systemd-run07:10:011
189132994031,6cyclictest421-21dbus-daemon11:15:431
189132994031,6cyclictest348011-21tr11:40:011
189132994031,6cyclictest335872-21latency_hist11:25:011
189132994031,4cyclictest0-21swapper/110:40:361
189131994034,4cyclictest380428-21systemd-run12:20:010
189131994034,3cyclictest0-21swapper/009:15:450
189133993933,3cyclictest0-21swapper/212:00:012
189133993931,5cyclictest695-21cron09:25:012
189133993931,5cyclictest368269-21ssh12:05:002
189133993931,5cyclictest211659-21systemd-run08:25:012
18913399392,28cyclictest353149-21ssh11:45:322
189132993932,4cyclictest0-21swapper/109:25:011
189132993931,5cyclictest376347-21munin-run12:15:011
189132993931,5cyclictest282607-21ssh10:20:021
189131993933,4cyclictest360151-21grep11:55:010
189131993932,4cyclictest225136-21cut09:10:000
189131993930,6cyclictest420-21avahi-daemon09:32:590
189131993930,6cyclictest420-21avahi-daemon09:32:590
189131993930,6cyclictest257221-21ssh09:48:490
189131993928,8cyclictest278450-21cron10:15:010
189131993925,11cyclictest261269-21ssh09:53:400
18913199392,31cyclictest250954-21ssh09:40:370
189132993831,3cyclictest0-21swapper/111:45:011
189132993827,9cyclictest3521708-21cups-browsed11:35:011
189131993830,5cyclictest352067-21ssh11:45:010
189131993825,11cyclictest228526-21ssh09:13:440
189131993825,10cyclictest268140-21ssh10:01:030
189131993822,14cyclictest319148-21ssh11:04:300
189131993822,14cyclictest319148-21ssh11:04:300
189131993822,12cyclictest297374-21ssh10:37:010
189133993731,3cyclictest0-21swapper/211:35:212
189133993729,5cyclictest270460-21latency_hist10:05:022
189133993729,4cyclictest266163-21kworker/u8:0+xprtiod10:00:012
189133993727,7cyclictest296670-21unixbench_singl10:35:472
18913399372,30cyclictest275563-21ssh10:10:342
189132993729,5cyclictest223652-21systemd-run09:05:011
189132993726,8cyclictest1055-21mta-sts-daemon11:56:071
189132993711,8cyclictest253519-21sh09:44:331
189132993710,10cyclictest246640-21ld09:35:321
189131993729,6cyclictest335225-21ssh11:23:510
189131993729,4cyclictest0-21swapper/010:55:370
189131993727,8cyclictest375692-21ssh12:13:460
189131993727,8cyclictest352290-21ssh11:45:030
189131993726,8cyclictest421-21dbus-daemon11:35:010
189131993726,8cyclictest388325-21ssh12:29:420
189131993725,10cyclictest247912-21ssh09:36:470
18913199371,30cyclictest283792-21ssh10:20:340
189133993626,6cyclictest284390-21unixbench_singl10:20:442
189133993623,9cyclictest389504-21missed_timers12:30:282
189132993628,5cyclictest274340-21cron10:10:011
189132993627,6cyclictest659-21snmpd08:35:321
189132993627,6cyclictest376952-21aten2.4-expect12:15:241
189132993611,4cyclictest1055-21mta-sts-daemon07:55:321
189132993610,10cyclictest311125-21sh10:54:471
189131993627,5cyclictest198979-21ntpq07:40:310
189131993626,8cyclictest329921-21ssh11:16:360
189131993626,8cyclictest292785-21ssh10:31:030
189131993626,8cyclictest264374-21ssh09:56:450
189131993626,7cyclictest189143-21/usr/sbin/munin07:10:010
189131993626,7cyclictest189143-21/usr/sbin/munin07:10:010
189131993623,10cyclictest369303-21ssh12:05:290
189131993623,10cyclictest322301-21ssh11:07:410
189133993527,5cyclictest373748-21systemctl12:10:432
189132993511,7cyclictest391705-21sh12:33:271
189132993510,7cyclictest199790-21latency_hist07:45:021
189131993527,6cyclictest364213-21sendmail12:00:010
189131993527,5cyclictest365411-21ntpq12:00:340
189131993527,5cyclictest200508-21ntpq07:45:330
189131993526,7cyclictest396377-21ssh12:39:440
189131993526,7cyclictest281844-21ssh10:18:410
189131993526,6cyclictest301568-21ssh10:42:100
189131993525,8cyclictest323811-21ssh11:10:060
189131993525,7cyclictest235357-21ssh09:21:080
189131993522,10cyclictest0-21swapper/012:35:020
189131993514,14cyclictest12350irq/32-eth011:25:370
189133993428,3cyclictest0-21swapper/212:05:352
189133993424,6cyclictest421-21dbus-daemon11:30:462
189133993424,6cyclictest421-21dbus-daemon10:56:392
18913299349,7cyclictest200544-21ntpq07:45:351
18913299348,7cyclictest292679-21ssh10:30:521
189132993427,3cyclictest0-21swapper/109:10:251
189132993425,4cyclictest0-21swapper/110:55:361
189132993424,6cyclictest365170-21latency12:00:281
189132993422,8cyclictest659-21snmpd10:45:241
189132993418,10cyclictest255150-21ps09:45:401
189132993414,9cyclictest225131-21diskmemload11:06:521
189132993410,8cyclictest395196-21sh12:37:261
189132993410,8cyclictest384851-21ssh12:25:211
189131993427,4cyclictest223029-21ntpq09:00:330
189131993425,6cyclictest289249-21ssh10:27:080
189131993425,6cyclictest289249-21ssh10:27:080
189131993425,6cyclictest1-21systemd07:55:010
189131993424,7cyclictest217032-21/usr/sbin/munin08:40:320
189133993327,3cyclictest0-21swapper/209:15:022
189133993326,3cyclictest0-21swapper/209:45:382
189133993325,5cyclictest280030-21threads10:15:442
189133993325,5cyclictest202618-21munin-run07:55:012
189133993324,5cyclictest377616-21ntpq12:15:382
189133993324,5cyclictest217655-21systemd-run08:45:012
18913399331,25cyclictest212603-21sendmail-mta08:25:352
18913299337,11cyclictest225131-21diskmemload11:25:441
189132993328,2cyclictest0-21swapper/109:35:021
189132993328,2cyclictest0-21swapper/109:35:021
189132993326,4cyclictest1051-21mta-sts-daemon10:03:071
189132993325,5cyclictest221016-21aten_r2power_vo08:55:221
189132993324,6cyclictest420-21avahi-daemon11:03:491
189132993324,6cyclictest420-21avahi-daemon11:03:491
189132993324,6cyclictest213661-21sort08:30:241
189131993326,5cyclictest195992-21ntpq07:30:320
189131993326,4cyclictest212538-21ntpq08:25:330
189131993325,5cyclictest193049-21ntpq07:20:330
189131993322,9cyclictest211013-21/usr/sbin/munin08:20:320
189131993322,8cyclictest220520-21/usr/sbin/munin08:50:430
189131993322,8cyclictest197179-21/usr/sbin/munin07:35:250
18913399322,21cyclictest600-21gmain09:05:312
189133993221,6cyclictest0-21swapper/211:40:252
189132993224,5cyclictest213136-21grep08:30:011
189132993223,6cyclictest3521708-21cups-browsed08:45:381
189132993223,5cyclictest204657-21fschecks_time08:00:261
189132993223,5cyclictest204657-21fschecks_time08:00:261
18913299322,24cyclictest263216-21sh09:55:341
189131993226,4cyclictest214010-21ntpq08:30:310
189131993224,5cyclictest191188-21/usr/sbin/munin07:15:260
189131993223,6cyclictest194469-21/usr/sbin/munin07:25:300
189131993223,6cyclictest190069-21ntpq07:10:340
189131993222,8cyclictest215244-21/usr/sbin/munin08:35:270
189131993222,8cyclictest205976-21/usr/sbin/munin08:05:240
189131993221,9cyclictest209564-21ntpq08:15:360
189133993123,5cyclictest199614-21cron07:45:012
189133993123,3cyclictest1055-21mta-sts-daemon11:50:252
189133993123,3cyclictest0-21swapper/209:20:012
189133993122,6cyclictest320679-21sh11:05:352
189133993122,5cyclictest219147-21systemd-run08:50:012
189133993121,7cyclictest210141-21idleruntime-cro08:20:012
189133993116,10cyclictest290692-21sshd10:29:512
189133993116,10cyclictest290692-21sshd10:29:512
189132993124,3cyclictest0-21swapper/108:05:341
189132993123,5cyclictest192120-21basename07:20:001
189132993122,6cyclictest197710-21mailstats07:35:391
189132993121,7cyclictest3521708-21cups-browsed12:05:371
189132993120,8cyclictest287027-21sh10:25:151
189132993120,8cyclictest287027-21sh10:25:151
189131993122,6cyclictest204617-21/usr/sbin/munin08:00:240
189131993122,6cyclictest204617-21/usr/sbin/munin08:00:240
189131993121,8cyclictest217947-21/usr/sbin/munin08:45:230
189133993025,2cyclictest0-21swapper/208:55:012
189133993023,4cyclictest308303-21ntp_kernel_pll_10:50:322
189133993023,3cyclictest0-21swapper/212:20:452
189133993023,3cyclictest0-21swapper/210:40:382
189133993023,3cyclictest0-21swapper/210:30:262
189133993022,5cyclictest192122-21munin-plugin-st07:20:002
189133993019,8cyclictest221963-21unixbench-2d08:55:422
18913299306,7cyclictest225131-21diskmemload09:54:301
189132993021,6cyclictest238788-21systemctl09:25:411
189132993019,8cyclictest193634-21cut07:25:011
189131993021,7cyclictest207679-21/usr/sbin/munin08:10:260
189133992922,4cyclictest1055-21mta-sts-daemon08:00:222
189133992922,4cyclictest1055-21mta-sts-daemon08:00:222
189133992920,6cyclictest222591-21cpuspeed09:00:232
189132992916,8cyclictest217062-21ntpq08:40:331
189132992914,11cyclictest196083-21ntpq07:30:361
189133992821,4cyclictest206663-21processes08:05:392
189133992819,5cyclictest0-21swapper/209:30:012
189133992817,8cyclictest194498-21ntp_kernel_pll_07:25:322
189133992813,9cyclictest421-21dbus-daemon08:35:332
18913399281,22cyclictest259172-21ssh09:50:382
189132992815,9cyclictest224531-21ntpq09:05:301
189132992815,10cyclictest219979-21ntpq08:50:311
18913399274,5cyclictest171rcu_preempt07:20:312
189132992717,5cyclictest208098-21ntpq08:10:371
189132992716,7cyclictest190003-21ntpq07:10:311
189132992715,7cyclictest201961-21ntpq07:50:291
189132992714,10cyclictest195146-21munin-run07:30:021
189133992618,5cyclictest198023-21unixbench_singl07:35:422
189133992617,7cyclictest1054-21mta-sts-daemon07:34:382
189132992613,9cyclictest211044-21ntpq08:20:341
189133992516,6cyclictest420-21avahi-daemon08:10:402
189133992511,6cyclictest214092-21ntpq08:30:352
189133992416,4cyclictest189642-21fschecks_time07:10:252
189133992415,6cyclictest201275-21sort07:50:022
18913499192,9cyclictest0-21swapper/310:05:343
18913499182,9cyclictest0-21swapper/312:15:353
18913499182,10cyclictest0-21swapper/311:30:383
18913499174,8cyclictest0-21swapper/310:35:393
18913499172,9cyclictest0-21swapper/310:40:313
18913499172,8cyclictest0-21swapper/310:15:313
18913499164,8cyclictest0-21swapper/307:55:263
18913499163,8cyclictest0-21swapper/312:05:363
18913499163,7cyclictest0-21swapper/312:30:423
18913499163,7cyclictest0-21swapper/311:35:373
18913499162,8cyclictest0-21swapper/312:20:403
18913499162,8cyclictest0-21swapper/312:10:323
18913499162,8cyclictest0-21swapper/311:55:333
18913499162,7cyclictest0-21swapper/309:30:033
18913499161,9cyclictest0-21swapper/309:55:023
18913499161,8cyclictest0-21swapper/311:10:323
18913499161,7cyclictest0-21swapper/308:35:313
18913499153,7cyclictest0-21swapper/309:45:323
18913499152,9cyclictest0-21swapper/311:00:033
18913499152,8cyclictest0-21swapper/310:00:303
18913499152,8cyclictest0-21swapper/309:15:323
18913499152,8cyclictest0-21swapper/309:10:323
18913499152,7cyclictest0-21swapper/309:40:313
18913499152,6cyclictest0-21swapper/311:05:253
18913499151,8cyclictest0-21swapper/311:25:383
18913499151,8cyclictest0-21swapper/310:50:253
18913499151,8cyclictest0-21swapper/309:30:343
18913499151,8cyclictest0-21swapper/309:30:343
18913499142,8cyclictest0-21swapper/312:25:253
18913499142,8cyclictest0-21swapper/309:55:313
18913499142,7cyclictest0-21swapper/312:00:343
18913499142,7cyclictest0-21swapper/309:20:353
18913499141,8cyclictest0-21swapper/311:15:333
18913499141,8cyclictest0-21swapper/310:25:313
18913499141,8cyclictest0-21swapper/310:25:313
18913499141,6cyclictest0-21swapper/308:40:333
18913499141,6cyclictest0-21swapper/307:40:343
189133991410,2cyclictest0-21swapper/207:10:012
189133991410,2cyclictest0-21swapper/207:10:012
18913499134,6cyclictest0-21swapper/311:40:333
18913499134,6cyclictest0-21swapper/311:20:323
18913499133,6cyclictest0-21swapper/312:35:233
18913499132,7cyclictest0-21swapper/311:50:353
18913499132,7cyclictest0-21swapper/311:00:353
18913499132,7cyclictest0-21swapper/311:00:353
18913499132,7cyclictest0-21swapper/310:45:223
18913499132,6cyclictest0-21swapper/307:50:013
18913499131,9cyclictest0-21swapper/311:45:233
18913499131,8cyclictest0-21swapper/310:10:213
18913499131,7cyclictest0-21swapper/310:30:313
18913499131,6cyclictest0-21swapper/307:50:313
18913499123,6cyclictest0-21swapper/308:25:303
18913499122,7cyclictest0-21swapper/308:15:323
18913499121,8cyclictest0-21swapper/309:35:303
18913499121,7cyclictest0-21swapper/307:15:333
18913499121,6cyclictest0-21swapper/309:10:003
18913499121,6cyclictest0-21swapper/308:45:383
18913499121,6cyclictest0-21swapper/308:30:323
18913499121,6cyclictest0-21swapper/308:10:323
18913499121,6cyclictest0-21swapper/307:35:333
18913499120,6cyclictest0-21swapper/308:55:293
18913499114,5cyclictest0-21swapper/308:00:363
18913499114,5cyclictest0-21swapper/308:00:363
18913499112,6cyclictest0-21swapper/307:30:263
18913499112,5cyclictest0-21swapper/310:20:323
18913499112,5cyclictest0-21swapper/309:05:003
18913499111,6cyclictest0-21swapper/308:20:373
18913499111,5cyclictest0-21swapper/307:20:213
18913499102,5cyclictest0-21swapper/307:10:273
18913499102,4cyclictest0-21swapper/307:25:253
18913499101,6cyclictest0-21swapper/308:10:013
1891349991,4cyclictest0-21swapper/308:55:023
1891349991,4cyclictest0-21swapper/307:10:023
1891349991,4cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional