You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-14 - 05:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot4s.osadl.org (updated Tue Oct 14, 2025 00:44:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
92157523070,3sleep30-21swapper/319:26:383
11442412780,5chrt481ktimers/323:21:463
904403996550,11cyclictest970482-21latency_hist20:15:011
904403996248,10cyclictest1121135-21latency_hist23:00:011
904398996254,5cyclictest1175493-21latency_hist23:55:010
904398995950,7cyclictest282-21systemd-journal23:45:000
904403995848,7cyclictest916143-21latency_hist19:20:011
904398995852,4cyclictest930781-21kworker/u8:0+events_unbound21:00:000
904403995648,5cyclictest1111262-21latency_hist22:50:011
904398995444,7cyclictest1071412-21latency_hist22:10:010
904398995345,5cyclictest1031951-21latency_hist21:30:010
11038482530,3sleep3586-21kworker/3:2+events22:41:463
904398995045,3cyclictest4183748-21kworker/u8:2+events_unbound20:26:460
904398994840,6cyclictest904233-21latency_hist19:10:020
12138592480,4chrt0-21swapper/100:31:451
904403994738,6cyclictest979531-21ntpq20:21:521
904398994741,4cyclictest4183748-21kworker/u8:2+events_unbound22:31:540
904403994627,16cyclictest945020-21ntpq19:46:521
904398994641,3cyclictest4183748-21kworker/u8:2+events_unbound21:06:380
904403994536,7cyclictest1175838-21/usr/sbin/munin23:56:471
904403994536,6cyclictest1000980-21latency_hist20:50:011
904403994534,8cyclictest1045924-21ntpq21:41:471
904403994523,9cyclictest282-21systemd-journal21:06:211
904398994536,6cyclictest995420-21kworker/u8:3+flush-8:022:46:500
904403994438,4cyclictest43823-21kworker/u8:4+flush-8:019:31:571
904403994434,7cyclictest282-21systemd-journal21:50:001
904403994423,3cyclictest1005932-21true20:55:001
904403994334,6cyclictest995816-21latency_hist20:45:011
904403994326,15cyclictest1-21systemd23:40:001
904403994326,14cyclictest282-21systemd-journal00:05:001
904398994338,3cyclictest1101573-21kworker/u8:4+flush-8:023:01:500
904403994216,8cyclictest282-21systemd-journal20:00:491
904398994237,3cyclictest995420-21kworker/u8:3+events_unbound22:45:010
904398994234,6cyclictest1046752-21latency_hist21:45:010
904398994230,9cyclictest282-21systemd-journal22:22:530
904403994133,6cyclictest1174658-21ntpq23:51:491
904398994136,3cyclictest1145592-21kworker/u8:2+events_unbound23:36:440
904398994135,5cyclictest1158255-21kworker/u8:1+flush-8:000:06:380
904408994030,7cyclictest1116566-21apt-get22:56:392
904403994028,10cyclictest1101573-21kworker/u8:4+flush-8:023:10:111
904403994021,8cyclictest1056983-21apt-get21:56:351
904403994021,4cyclictest18912-21cron22:05:001
904398994036,3cyclictest4183748-21kworker/u8:2+flush-8:019:36:540
904398994034,4cyclictest930781-21kworker/u8:0+flush-8:020:01:510
904398994032,6cyclictest4183748-21kworker/u8:2+flush-8:022:26:490
904398994028,9cyclictest915293-21ntpq19:16:480
904403993931,5cyclictest1056616-21latency_hist21:55:001
904403993920,7cyclictest1150772-21latency_hist23:30:011
904398993934,4cyclictest1158255-21kworker/u8:1+flush-8:023:46:500
904398993930,7cyclictest1191035-21diskstats00:11:390
904403993831,5cyclictest1136315-21apt-get23:16:341
904403993831,5cyclictest1136315-21apt-get23:16:341
904398993833,3cyclictest4183748-21kworker/u8:2+events_unbound19:46:520
9040262384,9sleep10-21swapper/119:07:351
904408993728,7cyclictest1116189-21turbostat.cron22:55:002
904408993728,7cyclictest1096210-21gpgv22:35:002
904403993721,2cyclictest0-21swapper/120:31:531
904398993733,3cyclictest43823-21kworker/u8:4+flush-8:020:21:530
904398993732,4cyclictest4183748-21kworker/u8:2+events_unbound22:36:510
904398993731,4cyclictest1158255-21kworker/u8:1+flush-8:000:26:400
904398993730,5cyclictest282-21systemd-journal21:55:000
904398993729,6cyclictest1145839-21latency_hist23:25:010
904408993621,13cyclictest0-21swapper/223:20:012
904408993621,13cyclictest0-21swapper/223:20:002
904408993620,2cyclictest1091254-21munin-plugin-st22:30:002
904403993631,3cyclictest940907-21munin-plugin-st19:45:001
904403993631,3cyclictest4183748-21kworker/u8:2+flush-8:020:26:571
904398993631,3cyclictest1006786-21kworker/u8:1+events_unbound21:31:530
904398993627,6cyclictest1215395-21latency_hist00:35:010
904408993525,8cyclictest1135943-21turbostat.cron23:15:002
904408993522,11cyclictest1151366-21cpuspeed_turbos23:31:392
904403993527,6cyclictest282-21systemd-journal20:36:531
904403993525,8cyclictest1080589-21ntpq22:16:531
904398993531,3cyclictest1158255-21kworker/u8:1+events_unbound00:21:540
904398993531,3cyclictest1101573-21kworker/u8:4+flush-8:000:01:540
904398993529,4cyclictest1101573-21kworker/u8:4+events_unbound00:36:390
90439899350,33cyclictest1041861-21munin-run21:40:000
904408993420,7cyclictest1180524-21gzip00:00:002
904408993420,2cyclictest1081010-21unixbench_singl22:16:572
904408993414,7cyclictest1185473-21cron00:05:002
904403993424,7cyclictest1091249-21latency_hist22:30:011
904398993427,5cyclictest4183748-21kworker/u8:2+flush-8:022:11:540
904408993322,8cyclictest1141247-21munin-node23:21:352
904408993319,2cyclictest1211001-21cpuspeed_turbos00:31:382
904408993316,4cyclictest985291-21cron20:30:002
90440899331,3cyclictest11607162sh23:40:002
904403993324,7cyclictest1070625-21ntpq22:06:491
904403993324,6cyclictest1126423-21/usr/sbin/munin23:06:471
904398993328,3cyclictest904256-21kworker/u8:0+events_unbound19:21:470
904398993328,3cyclictest4183748-21kworker/u8:2+events_unbound19:41:390
904398993228,3cyclictest995420-21kworker/u8:3+flush-8:022:51:470
904398993227,4cyclictest1101573-21kworker/u8:4+events_unbound23:16:390
904398993227,4cyclictest1101573-21kworker/u8:4+events_unbound23:16:390
904398993227,3cyclictest1145592-21kworker/u8:2+flush-8:023:26:450
904398993226,4cyclictest43823-21kworker/u8:4+events_unbound19:31:530
904398993226,4cyclictest4183748-21kworker/u8:2+events_unbound21:01:450
904398993226,4cyclictest1101573-21kworker/u8:4+events_unbound22:56:490
904398993223,7cyclictest1017150-21latency_hist21:15:010
904398993222,8cyclictest994333-21ntpq20:36:490
904403993127,2cyclictest931574-21tr19:36:381
904398993127,3cyclictest1006786-21kworker/u8:1+events_unbound21:21:560
904398993126,4cyclictest4183748-21kworker/u8:2+flush-8:022:04:330
904398993125,4cyclictest1158255-21kworker/u8:1+flush-8:023:56:490
11877482310,3sleep10-21swapper/100:06:461
904408993027,2cyclictest1051404-21unixbench_singl21:46:572
904398993026,3cyclictest1116496-21kworker/u8:0+events_unbound23:31:380
904398993025,4cyclictest1158255-21kworker/u8:1+flush-8:000:15:060
904398993019,8cyclictest1000155-21ntpq20:46:440
904398992925,3cyclictest4183748-21kworker/u8:2+events_unbound22:16:400
904398992925,2cyclictest921679-21cpuspeed_turbos19:26:380
904398992924,3cyclictest995420-21kworker/u8:3+events_unbound21:50:000
904398992924,3cyclictest930781-21kworker/u8:0+events_unbound20:45:000
904398992924,3cyclictest43823-21kworker/u8:4+events_unbound20:16:420
904398992924,3cyclictest1086043-21kworker/u8:1+events_unbound23:06:530
904403992824,2cyclictest1072370-21irqcore22:11:431
904403992821,4cyclictest1175514-21kworker/u8:2+flush-8:000:26:471
904403992819,7cyclictest1200653-21/usr/sbin/munin00:21:541
904403992819,6cyclictest926012-21latency_hist19:30:011
904398992824,3cyclictest43823-21kworker/u8:4+events_unbound20:10:540
904398992823,3cyclictest995420-21kworker/u8:3+flush-8:021:16:420
904398992822,4cyclictest995420-21kworker/u8:3+flush-8:020:51:500
904398992822,4cyclictest1135631-21unixbench_multi23:11:560
904408992724,2cyclictest915787-21unixbench_multi19:16:562
904403992722,3cyclictest975412-21awk20:20:001
904403992721,4cyclictest4183748-21kworker/u8:2+flush-8:022:21:481
904403992720,5cyclictest1100969-21grep22:36:541
904403992719,6cyclictest1155729-21latency_hist23:35:011
904403992717,8cyclictest1169723-21ntpq23:46:451
904398992723,3cyclictest930781-21kworker/u8:0+events_unbound20:06:420
904398992723,3cyclictest43823-21kworker/u8:4+flush-8:020:31:550
904398992721,4cyclictest904882-21cpuspeed_turbos19:11:390
904403992621,4cyclictest1011228-21unixbench_singl21:01:501
904403992620,4cyclictest1116496-21kworker/u8:0+flush-8:023:20:121
904398992622,3cyclictest930781-21kworker/u8:0+events_unbound19:56:460
904398992622,3cyclictest930781-21kworker/u8:0+events_unbound19:56:460
10404502260,3sleep20-21swapper/221:36:452
904408992521,3cyclictest1101938-21cpuspeed_turbos22:41:392
904408992521,3cyclictest1032373-21aten2_r5power_e21:31:352
904408992521,2cyclictest1171249-21fschecks_time23:51:412
904403992521,2cyclictest0-21swapper/123:01:441
904403992520,3cyclictest1161072-21aten2_r5power_c23:41:371
904403992515,8cyclictest1105520-21ntpq22:41:501
904398992522,2cyclictest995420-21kworker/u8:3+events_unbound21:56:580
904398992518,5cyclictest4183748-21kworker/u8:2+events_unbound19:51:530
11166862250,2sleep3481ktimers/322:56:423
904408992420,2cyclictest990718-21aten2_r5power_v20:36:392
904408992420,2cyclictest971109-21fschecks_count20:16:402
904408992420,2cyclictest945814-21sh19:50:002
904408992420,2cyclictest1215817-21aten2_r5power_e00:36:382
904403992419,4cyclictest405-21dbus-daemon20:00:001
904403992419,4cyclictest405-21dbus-daemon20:00:001
904403992419,3cyclictest905383-21irqrtprio19:11:431
904403992418,4cyclictest833-21mta-sts-daemon21:27:451
904408992320,2cyclictest916843-21fschecks_time19:21:412
904408992320,2cyclictest1022508-21aten2_r5power_e21:21:392
904408992320,2cyclictest1011987-21aten2_r5power_e21:06:372
904408992318,3cyclictest1066235-21unixbench_singl22:01:562
904408992318,3cyclictest1042316-21aten2_r5power_v21:41:372
904408992316,5cyclictest996385-21munin-node20:46:302
904403992319,3cyclictest1158255-21kworker/u8:1+flush-8:000:11:441
904403992319,2cyclictest0-21swapper/121:35:001
904403992319,2cyclictest0-21swapper/121:16:471
904403992319,2cyclictest0-21swapper/100:36:431
904403992318,3cyclictest0-21swapper/121:00:001
904403992318,3cyclictest0-21swapper/120:06:391
904408992218,2cyclictest1200902-21cpuspeed_turbos00:21:392
904408992218,2cyclictest1125810-21unixbench_singl23:01:572
904408992218,2cyclictest10212842chrt21:16:472
90440399226,2cyclictest10245622kthreadcore21:21:461
904403992218,2cyclictest0-21swapper/121:10:531
904403992218,2cyclictest0-21swapper/119:25:001
904403992217,3cyclictest0-21swapper/122:35:001
904403992215,4cyclictest1-21systemd21:40:001
904403992212,6cyclictest1200297-21latency_hist00:20:001
10572822220,2sleep3471rcuc/321:56:403
904408992118,2cyclictest921812-21fschecks_time19:26:412
904408992117,2cyclictest996090-21latency_hist20:45:012
904408992117,2cyclictest0-21swapper/222:50:002
90440899210,19cyclictest1199279-21grep00:16:462
904403992116,2cyclictest0-21swapper/122:51:571
90440899203,4cyclictest9072992cut19:11:462
90440899203,3cyclictest9762742if_enp4s0.320:21:452
904408992017,2cyclictest940903-21sh19:45:012
904408992016,2cyclictest0-21swapper/219:36:482
904408992011,7cyclictest1161480-21awk23:41:412
90440899200,18cyclictest1100370-21missed_timers22:36:452
904403992014,4cyclictest0-21swapper/119:51:421
904408991914,4cyclictest0-21swapper/222:00:002
904408991914,3cyclictest0-21swapper/221:51:412
904408991911,6cyclictest1007354-21awk21:01:362
90440899191,16cyclictest1005136-21awk20:51:492
9039692193,9sleep30-21swapper/319:06:573
904408991814,2cyclictest0-21swapper/223:26:392
904408991813,3cyclictest964772-21ntpq20:06:512
90440899181,15cyclictest1194521-21awk00:11:472
90440899181,12cyclictest969666-21ntpq20:11:522
8996302184,10sleep20-21swapper/219:05:592
90440899177,8cyclictest1068954-21cut22:06:442
90440899172,14cyclictest0-21swapper/220:59:062
904408991714,2cyclictest0-21swapper/221:15:022
904408991713,2cyclictest0-21swapper/200:06:572
904408991712,3cyclictest0-21swapper/222:21:462
90440899171,12cyclictest1129354-21kthreadcore23:06:452
904408991710,5cyclictest956177-21wget20:01:382
90440899170,14cyclictest989381-21awk20:31:472
90440899165,5cyclictest949871-21seq19:51:512
90440899164,8cyclictest1071782-21apt-get22:11:352
90440899162,13cyclictest0-21swapper/219:55:352
90440899162,13cyclictest0-21swapper/219:55:352
904408991612,2cyclictest0-21swapper/219:31:402
904408991611,3cyclictest0-21swapper/221:26:472
904408991610,4cyclictest1207113-21cut00:26:442
904408991510,3cyclictest0-21swapper/223:46:422
10679142144,5sleep30-21swapper/322:06:443
9946122133,7sleep30-21swapper/320:36:563
9546382132,7sleep30-21swapper/319:56:483
9546382132,7sleep30-21swapper/319:56:483
9408862132,6sleep30-21swapper/319:45:003
9072962130,2sleep3481ktimers/319:11:453
11908292132,7sleep30-21swapper/300:11:383
11507592132,6sleep30-21swapper/323:30:013
9952162122,6sleep30-21swapper/320:40:013
9899862122,6sleep30-21swapper/320:31:593
9843272122,6sleep30-21swapper/320:26:503
9613732122,6sleep30-21swapper/320:06:433
9557232122,6sleep30-21swapper/320:00:013
9417272122,6sleep30-21swapper/319:46:453
9335002122,6sleep30-21swapper/319:36:463
9268552122,6sleep30-21swapper/319:31:443
12003212122,5sleep30-21swapper/300:20:013
11755102122,5sleep30-21swapper/323:55:013
11709942123,6sleep30-21swapper/323:51:403
11697552123,6sleep30-21swapper/323:46:463
11653722122,6sleep30-21swapper/323:41:573
11359622122,6sleep30-21swapper/323:15:013
11359622122,6sleep30-21swapper/323:15:013
11265342122,6sleep30-21swapper/323:06:383
11253542122,6sleep30-21swapper/323:01:533
11112732122,6sleep30-21swapper/322:50:013
10972022122,6sleep30-21swapper/322:36:413
10961652120,8sleep30-21swapper/322:35:003
10820902122,6sleep30-21swapper/322:21:443
10662682122,6sleep30-21swapper/322:02:113
10513992122,6sleep30-21swapper/321:46:583
10211402122,6sleep30-21swapper/321:16:453
10115392122,6sleep30-21swapper/321:05:013
10013522122,6sleep30-21swapper/320:51:373
9963842112,5sleep30-21swapper/320:46:313
9767102112,5sleep30-21swapper/320:21:483
9708672112,5sleep30-21swapper/320:16:383
9697872112,5sleep30-21swapper/320:11:583
9466772112,5sleep30-21swapper/319:51:473
9164272112,6sleep30-21swapper/319:20:373
9097422112,5sleep30-21swapper/319:15:033
12156392113,5sleep30-21swapper/300:35:213
12152762113,5sleep30-21swapper/300:33:463
12096112112,5sleep30-21swapper/300:26:553
11955972112,6sleep30-21swapper/300:15:203
11857912112,5sleep30-21swapper/300:06:083
11854682112,5sleep30-21swapper/300:05:013
11559772112,6sleep30-21swapper/323:35:233
11470912112,5sleep30-21swapper/323:26:443
11069452112,5sleep30-21swapper/322:46:403
10911202112,6sleep30-21swapper/322:28:393
10717012112,6sleep30-21swapper/322:10:533
10519582112,6sleep30-21swapper/321:50:273
10460332112,5sleep30-21swapper/321:41:523
10372092112,6sleep30-21swapper/321:36:233
10322882112,5sleep30-21swapper/321:31:333
10309732112,5sleep30-21swapper/321:26:483
10235022112,5sleep30-21swapper/321:21:463
10168802112,6sleep30-21swapper/321:11:473
10063792112,6sleep30-21swapper/320:58:113
11312482102,5sleep30-21swapper/323:10:113
10766322102,5sleep30-21swapper/322:15:153
10068802102,5sleep30-21swapper/321:01:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional