You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-21 - 08:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot4s.osadl.org (updated Tue Oct 21, 2025 00:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
385646022450,3sleep13854202-21kworker/1:0+events19:06:391
414397821850,2sleep30-21swapper/323:56:403
414397821850,2sleep30-21swapper/323:56:403
39745242720,5sleep20-21swapper/221:01:402
3859396996656,7cyclictest4130328-21latency_hist23:45:011
40847742610,3sleep0201rcuc/022:56:460
39954522600,1sleep03995454-21grep21:21:490
39954522600,1sleep03995454-21grep21:21:490
3859396995949,8cyclictest282-21systemd-journal23:35:001
3859392995847,8cyclictest282-21systemd-journal00:20:010
3859396995748,6cyclictest4006267-21latency_hist21:40:011
3859392995749,5cyclictest3990817-21latency_hist21:20:000
3859392995347,4cyclictest3951166-21kworker/u8:0+flush-8:023:10:010
3859392995243,7cyclictest3866651-21apt-get19:16:310
3859396995040,8cyclictest282-21systemd-journal00:15:011
3859392995034,8cyclictest4005635-21latency_hist21:35:010
3859398994936,10cyclictest3906092-21awk19:55:002
3859396994837,8cyclictest4000704-21latency_hist21:30:001
3859396994835,10cyclictest4049954-21ntpq22:21:451
3859396994737,7cyclictest282-21systemd-journal22:55:011
3859392994738,6cyclictest3876184-21latency_hist19:25:010
3859396994640,4cyclictest4021100-21kworker/u8:1+flush-8:022:56:511
3859396994638,5cyclictest4115476-21latency_hist23:30:011
3859398994434,8cyclictest3950987-21apt-get20:40:002
3859396994438,4cyclictest2493853-21kworker/u8:4+flush-8:020:26:501
3859396994337,4cyclictest3951166-21kworker/u8:0+events_unbound21:41:471
3859396994331,8cyclictest282-21systemd-journal19:32:531
3859392994336,5cyclictest4085984-21apt-get23:01:320
3859392994232,8cyclictest282-21systemd-journal20:05:020
3859396994136,4cyclictest2493853-21kworker/u8:4+flush-8:020:21:511
3859396994136,3cyclictest4140399-21kworker/u8:3+events_unbound00:05:001
3859396994136,3cyclictest3951166-21kworker/u8:0+flush-8:023:01:501
3859396994133,6cyclictest4145109-21latency_hist00:00:001
3859396994133,6cyclictest4145109-21latency_hist00:00:001
3859392994131,8cyclictest3900893-21apt-get19:50:000
3859392994130,8cyclictest3863529-21ntpq19:11:430
3859392994128,10cyclictest282-21systemd-journal23:20:000
3859396994034,4cyclictest3859631-21kworker/u8:0+flush-8:019:16:541
3859396994031,7cyclictest282-21systemd-journal00:37:031
3859392994030,7cyclictest4075723-21latency_hist22:50:000
3859396993935,3cyclictest3951166-21kworker/u8:0+flush-8:022:01:371
3859396993832,4cyclictest4130688-21apt-get23:46:301
3859396993832,4cyclictest3859631-21kworker/u8:0+flush-8:019:51:451
3859396993830,6cyclictest282-21systemd-journal20:35:001
3859396993825,10cyclictest4011825-21diskstats21:46:351
3859392993830,6cyclictest4124521-21ntpq23:36:440
3859396993728,6cyclictest3930960-21latency_hist20:20:011
3859396993726,8cyclictest282-21systemd-journal23:52:011
3859392993729,6cyclictest3906083-21latency_hist19:55:010
3859396993632,3cyclictest2493853-21kworker/u8:4+flush-8:022:16:341
3859396993631,4cyclictest3951166-21kworker/u8:0+flush-8:021:11:361
40047512350,5sleep3451irq_work/321:31:433
3859398993523,9cyclictest3890984-21awk19:40:002
3859396993529,4cyclictest3951166-21kworker/u8:0+flush-8:021:56:521
3859396993529,4cyclictest2493853-21kworker/u8:4+events_unbound19:41:421
3859396993527,5cyclictest4070539-21latency_hist22:45:001
3859396993527,5cyclictest4070539-21latency_hist22:45:001
3859396993526,7cyclictest3881137-21latency_hist19:30:011
3859396993525,8cyclictest4056021-21/usr/sbin/munin22:31:441
3859392993524,8cyclictest4075954-21sshd22:50:010
3859396993424,8cyclictest3911017-21latency_hist20:00:001
385939299346,9cyclictest4125942-21ls23:41:340
3859392993424,7cyclictest282-21systemd-journal23:15:000
3859392993423,8cyclictest3960363-21ntpq20:46:440
3859398993323,8cyclictest4070542-21apt-get22:45:002
3859398993323,8cyclictest4070542-21apt-get22:45:002
3859396993328,3cyclictest3951166-21kworker/u8:0+events_unbound20:51:521
3859396993327,4cyclictest4001064-21apt-get21:31:311
3859396993326,5cyclictest282-21systemd-journal19:38:571
3859396993323,8cyclictest3875389-21ntpq19:21:481
3859398993223,7cyclictest4095130-21grep23:06:522
3859398993220,9cyclictest4100880-21cp23:15:012
3859396993226,5cyclictest4151203-21irqrtprio00:06:371
3859396993226,5cyclictest3951166-21kworker/u8:0+events_unbound23:16:351
3859396993226,4cyclictest4178897-21ntpq00:31:431
3859396993226,4cyclictest3859631-21kworker/u8:0+flush-8:020:11:521
3859396993225,4cyclictest4140399-21kworker/u8:3+flush-8:000:16:411
3859396993221,4cyclictest3951166-21kworker/u8:0+events_unbound21:16:451
385939299323,16cyclictest3884722-21kthreadcore19:31:430
3859398993121,7cyclictest282-21systemd-journal19:45:002
3859396993128,2cyclictest3921118-21munin-plugin-st20:10:001
3859396993126,4cyclictest2493853-21kworker/u8:4+flush-8:022:46:511
3859396993126,4cyclictest2493853-21kworker/u8:4+flush-8:021:21:461
3859396993126,4cyclictest2493853-21kworker/u8:4+flush-8:021:21:451
3859396993126,3cyclictest2493853-21kworker/u8:4+flush-8:020:41:411
3859396993125,4cyclictest2464914-21kworker/u8:3+flush-8:020:36:541
385939299318,13cyclictest4040302-21python322:11:550
385939299314,13cyclictest4146085-21ps00:01:390
3859392993122,7cyclictest3980111-21ntpq21:06:450
385939299312,18cyclictest4060836-21latency_hist22:35:010
3859392993120,8cyclictest282-21systemd-journal21:29:590
39939762300,2sleep20-21swapper/221:21:392
39939762300,2sleep20-21swapper/221:21:392
3859396993026,3cyclictest2464914-21kworker/u8:3+flush-8:019:46:401
385939299307,13cyclictest3906535-21/usr/sbin/munin19:56:360
385939299306,14cyclictest4049918-21/usr/sbin/munin22:21:440
385939299306,13cyclictest3925319-21ntpq20:11:460
385939299305,15cyclictest4173941-21ntpq00:26:500
385939299305,15cyclictest3942182-21irqrtprio20:31:400
385939299305,13cyclictest4054910-21ntpq22:26:490
385939299303,17cyclictest3951049-21apt-config20:40:010
3859392993026,3cyclictest4169839-21cron00:25:000
3859392993026,2cyclictest4175651-21irqcore00:31:370
3859392993026,2cyclictest4115473-21cron23:30:000
3859392993026,2cyclictest3931004-21systemd-run20:20:010
385939299302,17cyclictest3932228-21cut20:21:410
385939299302,16cyclictest3895009-21wc19:41:400
3859396992926,2cyclictest2493853-21kworker/u8:4+flush-8:023:06:521
3859396992923,4cyclictest3970296-21kworker/u8:1+flush-8:021:03:591
3859396992923,4cyclictest2464914-21kworker/u8:3+flush-8:020:01:381
3859396992922,3cyclictest0-21swapper/122:36:431
3859396992922,3cyclictest0-21swapper/122:36:431
385939299297,4cyclictest3936325-21apt-get20:26:340
385939299297,13cyclictest4022033-21/usr/sbin/munin21:56:420
385939299297,10cyclictest4130922-21/usr/sbin/munin23:46:330
385939299296,10cyclictest587-21runrttasks21:01:300
385939299295,14cyclictest4144367-21ntp_states23:56:460
385939299295,14cyclictest4144367-21ntp_states23:56:460
385939299295,14cyclictest4109795-21ntpq23:21:480
385939299294,15cyclictest4016743-21df21:51:340
385939299294,14cyclictest4183904-21ntpq00:36:480
385939299294,14cyclictest3961893-21hddtemp_smartct20:51:370
385939299293,15cyclictest4155985-21irqcore00:11:400
385939299293,15cyclictest4135834-21df23:51:350
385939299293,15cyclictest3970314-21ntp_states20:56:470
385939299293,14cyclictest4041461-21sh22:16:330
385939299293,14cyclictest0-21swapper/019:26:310
385939299291,15cyclictest4069791-21uname22:41:470
385939299291,15cyclictest4069791-21uname22:41:470
3859396992824,3cyclictest4140399-21kworker/u8:3+events_unbound00:26:571
385939299288,10cyclictest4119726-21ntp_states23:31:480
385939299286,12cyclictest4064862-21/usr/sbin/munin22:36:490
385939299286,12cyclictest4064862-21/usr/sbin/munin22:36:490
385939299283,15cyclictest4010448-21ntpq21:41:420
385939299283,14cyclictest4030015-21memory22:01:440
3859398992723,2cyclictest3926522-21aten2_r5power_v20:16:352
3859396992724,2cyclictest4106312-21cut23:21:371
3859396992723,3cyclictest2493853-21kworker/u8:4+flush-8:023:11:481
3859396992723,2cyclictest4032082-21irqrtprio22:06:391
3859396992722,4cyclictest3951166-21kworker/u8:0+flush-8:021:51:281
3859396992721,4cyclictest4140399-21kworker/u8:3+events_unbound00:21:501
3859392992722,3cyclictest3952130-21irqcore20:41:380
385939299272,15cyclictest4031418-21aten2_r5power_p22:06:340
385939299272,14cyclictest3981550-21diskstats21:11:340
385939299272,12cyclictest4150819-21fschecks_time00:06:350
385939299272,10cyclictest3912386-21cat20:01:380
385939299271,16cyclictest4011507-21sshd21:45:560
38870482260,6sleep3451irq_work/319:36:393
3859398992620,4cyclictest3875898-21unixbench_singl19:21:532
3859396992621,4cyclictest2493853-21kworker/u8:4+flush-8:022:14:021
3859396992620,4cyclictest3980613-21unixbench_multi21:06:521
3859396992617,6cyclictest3971055-21latency_hist21:00:011
385939299264,11cyclictest3886682-21diskstats19:36:350
385939299263,7cyclictest0-21swapper/021:36:000
3859392992612,8cyclictest3859377-21gpgv19:10:010
3859398992522,2cyclictest3966672-21cpuspeed_turbos20:56:342
3859398992521,3cyclictest3859762-21aten2_r5power_c19:11:322
3859398992520,3cyclictest3980587-21unixbench_multi21:06:522
3859398992518,5cyclictest282-21systemd-journal23:27:012
3859398992516,6cyclictest4075730-21turbostat.cron22:49:592
3859396992517,6cyclictest2493853-21kworker/u8:4+events_unbound22:26:491
3859398992421,2cyclictest4086068-21aten2_r5power_p23:01:352
3859398992421,2cyclictest4031387-21aten2_r5power_c22:06:342
3859398992420,2cyclictest4165312-21aten2_r5power_c00:21:352
3859398992317,4cyclictest3915935-21cron20:05:002
3859398992313,8cyclictest4080666-21awk22:55:002
3859396992316,5cyclictest2493853-21kworker/u8:4+events_unbound20:46:331
385939899229,8cyclictest3921473-21apt-get20:11:322
385939899222,14cyclictest3941460-21apt-get20:31:342
3859398992219,2cyclictest4135864-21fschecks_count23:51:372
3859398992219,2cyclictest4061081-21aten2_r5power_p22:36:362
3859398992219,2cyclictest4061081-21aten2_r5power_p22:36:362
3859396992217,4cyclictest2464914-21kworker/u8:3+events_unbound19:11:481
3859396992217,3cyclictest0-21swapper/123:36:471
385939899212,13cyclictest3896328-21apt-get19:46:372
3859398992117,3cyclictest3940857-21turbostat.cron20:30:002
3859398992115,3cyclictest0-21swapper/220:05:022
3859398992017,2cyclictest3906712-21fschecks_count19:56:372
3859398992016,2cyclictest0-21swapper/222:16:432
385939899192,12cyclictest4036289-21apt-get22:11:352
385939899192,12cyclictest4026423-21apt-get22:01:342
385939899192,11cyclictest3931347-21apt-get20:21:352
3859398991916,2cyclictest4180110-21aten2_r5power_c00:36:332
3859398991915,3cyclictest0-21swapper/223:00:002
3859398991915,2cyclictest4115902-21aten2_r5power_e23:31:332
3859398991911,6cyclictest4006249-21cron21:40:002
385939899191,12cyclictest3951604-21apt-get20:41:332
38544462196,9sleep20-21swapper/219:05:372
38541662192,8sleep30-21swapper/319:05:013
385939899182,11cyclictest4149324-21ntpq00:01:462
385939899182,11cyclictest4120829-21taskset23:36:342
3859398991814,2cyclictest4006860-21cpuspeed_turbos21:41:302
3859398991814,2cyclictest0-21swapper/223:21:512
3859398991814,2cyclictest0-21swapper/221:11:322
3859398991814,2cyclictest0-21swapper/219:30:012
3859398991813,3cyclictest0-21swapper/221:51:512
3859398991812,2cyclictest0-21swapper/200:07:022
3859398991811,5cyclictest3957380-21seq20:46:382
385939899181,13cyclictest3990246-21python321:16:512
41773332178,5sleep30-21swapper/300:31:403
385939899179,6cyclictest4133197-21tr23:46:402
385939899174,10cyclictest4050748-21cat22:25:002
3859398991713,3cyclictest4000718-21systemd-run21:30:002
3859398991712,3cyclictest4056390-21grep22:31:372
3859398991711,4cyclictest0-21swapper/200:16:442
385939899169,5cyclictest4144678-21wc23:56:492
385939899169,5cyclictest4144678-21wc23:56:492
385939899169,5cyclictest4127943-21cut23:41:402
385939899168,2cyclictest421rcuog/220:51:342
385939899167,7cyclictest4175309-21ls00:31:332
3859398991612,2cyclictest0-21swapper/222:26:392
3859398991612,2cyclictest0-21swapper/221:56:442
3859398991612,2cyclictest0-21swapper/219:16:512
3859398991612,2cyclictest0-21swapper/200:12:522
39516072152,8sleep30-21swapper/320:41:323
385939899156,7cyclictest4014226-21cut21:46:412
385939899156,7cyclictest3881703-21sort19:31:362
3859398991510,3cyclictest0-21swapper/223:16:392
3859398991510,3cyclictest0-21swapper/200:26:452
41550122143,6sleep30-21swapper/300:10:013
40746862142,8sleep30-21swapper/322:46:433
39159322142,7sleep30-21swapper/320:05:013
39008792141,8sleep30-21swapper/319:50:003
3859398991410,2cyclictest0-21swapper/221:31:402
41648812132,7sleep30-21swapper/300:20:003
41109372132,7sleep30-21swapper/323:26:363
41047932132,7sleep30-21swapper/323:16:453
40781672132,7sleep30-21swapper/322:51:423
40582182132,7sleep30-21swapper/322:31:433
38684862133,6sleep30-21swapper/319:16:413
41693632122,6sleep30-21swapper/300:21:543
41556902123,5sleep30-21swapper/300:11:383
41500992123,5sleep30-21swapper/300:05:003
40998892122,6sleep30-21swapper/323:11:513
40905372120,7sleep30-21swapper/323:05:013
40511182122,6sleep30-21swapper/322:26:373
40408852122,6sleep30-21swapper/322:15:013
40263502122,6sleep30-21swapper/322:00:573
40169782123,6sleep30-21swapper/321:51:383
39917082122,6sleep30-21swapper/321:21:373
39917082122,6sleep30-21swapper/321:21:373
39817582122,6sleep30-21swapper/321:11:373
39619972122,6sleep30-21swapper/320:51:393
39561902122,6sleep30-21swapper/320:45:013
39368022122,6sleep30-21swapper/320:26:393
39307772122,6sleep30-21swapper/320:17:533
39226782122,6sleep30-21swapper/320:11:393
39169202122,6sleep30-21swapper/320:06:373
39021672122,6sleep30-21swapper/319:51:363
38803312123,5sleep30-21swapper/319:26:463
41797312112,5sleep30-21swapper/300:35:013
41740502112,6sleep30-21swapper/300:26:553
41354652112,6sleep30-21swapper/323:50:073
41351032113,5sleep30-21swapper/323:48:313
41300572112,5sleep30-21swapper/323:41:533
41207672112,6sleep30-21swapper/323:36:073
41157952112,5sleep30-21swapper/323:31:133
41069172112,5sleep30-21swapper/323:21:423
40819172112,5sleep30-21swapper/322:56:423
40656532112,5sleep30-21swapper/322:40:013
40656532112,5sleep30-21swapper/322:40:013
40610012112,5sleep30-21swapper/322:36:363
40610012112,5sleep30-21swapper/322:36:363
40460972112,6sleep30-21swapper/322:20:513
40362632112,5sleep30-21swapper/322:11:323
40312932112,6sleep30-21swapper/322:06:023
40118362112,5sleep30-21swapper/321:46:363
40101902112,5sleep30-21swapper/321:41:393
40059142112,6sleep30-21swapper/321:35:453
39992952112,5sleep30-21swapper/321:26:463
39861312112,6sleep30-21swapper/321:15:483
39762322112,6sleep30-21swapper/321:05:073
39714032112,5sleep30-21swapper/321:01:303
39663672112,6sleep30-21swapper/320:55:443
39497982112,6sleep30-21swapper/320:36:423
39422512111,7sleep30-21swapper/320:31:413
39312542113,5sleep30-21swapper/320:20:413
39061802112,5sleep30-21swapper/319:55:013
38912792112,6sleep30-21swapper/319:40:453
38817602112,5sleep30-21swapper/319:31:383
38718992112,6sleep30-21swapper/319:21:393
38642322112,6sleep30-21swapper/319:13:363
40907802102,5sleep30-21swapper/323:05:053
40213112102,5sleep30-21swapper/321:55:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional