You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-21 - 01:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Tue May 20, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
157363993533,1cyclictest260881-21fwupd08:49:461
157357993528,5cyclictest490507-21latency_hist12:35:010
157369993428,4cyclictest475050-21latency_hist12:20:023
157366993431,2cyclictest464796-21kworker/u8:2+flush-259:012:25:292
157363993428,4cyclictest307361-21latency_hist09:35:001
157366993229,2cyclictest203644-21kworker/u8:1+flush-259:007:55:262
157366993229,2cyclictest203644-21kworker/u8:1+flush-259:007:55:262
157366993128,2cyclictest355510-21kworker/u8:0+flush-259:011:05:262
157366993128,2cyclictest146812-21kworker/u8:4+flush-259:007:14:592
157363993027,2cyclictest438370-21kworker/u8:4+flush-259:011:45:191
157357993024,4cyclictest453164-21ntpq11:55:230
157363992923,4cyclictest437689-21ntpq11:40:231
157366992825,2cyclictest322858-21kworker/u8:3+flush-259:010:35:182
157366992825,2cyclictest229947-21kworker/u8:2+flush-259:009:40:002
157366992823,3cyclictest297074-21latency_hist09:25:002
157366992821,3cyclictest229947-21kworker/u8:2+events_unbound08:40:212
157366992724,2cyclictest355510-21kworker/u8:0+events_unbound10:55:182
157366992724,2cyclictest229947-21kworker/u8:2+flush-259:009:40:272
157366992723,3cyclictest146812-21kworker/u8:4+flush-259:007:45:212
157363992723,3cyclictest203644-21kworker/u8:1+flush-259:008:10:251
157357992723,3cyclictest402144-21kworker/u8:1+flush-259:011:10:190
157357992723,3cyclictest312755-21apt-get09:40:150
157369992622,3cyclictest365635-21kworker/u8:1+flush-259:010:45:213
157366992623,2cyclictest203644-21kworker/u8:1+flush-259:008:05:262
157366992623,2cyclictest146812-21kworker/u8:4+flush-259:007:20:262
157366992622,3cyclictest229947-21kworker/u8:2+events_unbound09:55:172
157363992623,2cyclictest479716-21kworker/u8:4+flush-259:012:30:241
157363992623,2cyclictest454268-21kworker/u8:3+flush-259:012:07:561
157363992623,2cyclictest146812-21kworker/u8:4+flush-259:007:30:191
157363992622,3cyclictest260891-21kworker/u8:0+flush-259:009:15:151
157366992522,2cyclictest402144-21kworker/u8:1+flush-259:011:10:202
157366992520,4cyclictest417104-21ntpq11:20:232
157363992521,3cyclictest291923-21kworker/u8:1+flush-259:009:25:191
157363992521,3cyclictest250694-21kworker/u8:1+flush-259:009:00:011
157357992522,2cyclictest235122-21apt-get08:25:140
157369992419,4cyclictest3000860-21systemd-journal12:25:003
157366992421,2cyclictest291923-21kworker/u8:1+events_unbound09:30:582
157363992421,2cyclictest291923-21kworker/u8:1+flush-259:010:15:001
157363992421,2cyclictest146812-21kworker/u8:4+events_unbound07:10:221
157363992420,3cyclictest402144-21kworker/u8:1+flush-259:012:15:161
157363992420,3cyclictest260891-21kworker/u8:0+events_unbound09:14:521
157357992418,4cyclictest301296-21ntpq09:25:210
15735799240,23cyclictest1-21systemd10:28:500
157366992320,2cyclictest443762-21kworker/u8:0+flush-259:011:50:212
157366992320,2cyclictest428472-21kworker/u8:2+flush-259:011:39:562
157366992320,2cyclictest229947-21kworker/u8:2+events_unbound08:45:182
157369992219,2cyclictest167875-21kworker/u8:3+flush-259:007:40:003
157366992219,2cyclictest250694-21kworker/u8:1+flush-259:008:40:012
157366992219,2cyclictest229947-21kworker/u8:2+events_unbound08:20:142
157363992219,2cyclictest229947-21kworker/u8:2+events_unbound09:00:211
157363992219,2cyclictest203644-21kworker/u8:1+events_unbound08:00:011
157363992219,2cyclictest203644-21kworker/u8:1+events_unbound08:00:001
157357992219,1cyclictest333819-21unin-run10:05:000
157357992218,3cyclictest381119-21apt-get10:50:000
157357992218,3cyclictest162937-21kworker/u8:1+flush-259:007:15:160
157366992118,2cyclictest355510-21kworker/u8:0+flush-259:010:50:202
157366992118,2cyclictest355510-21kworker/u8:0+flush-259:010:50:202
157366992118,2cyclictest229947-21kworker/u8:2+events_unbound09:15:292
157366992117,3cyclictest146812-21kworker/u8:4+events_unbound07:50:192
157366992115,4cyclictest3000860-21systemd-journal11:35:002
157363992118,2cyclictest464796-21kworker/u8:2+flush-259:012:35:331
157363992118,2cyclictest402144-21kworker/u8:1+flush-259:011:25:201
157363992118,2cyclictest381148-21gpgv10:50:001
157363992118,2cyclictest162937-21kworker/u8:1+events_unbound07:15:161
157363992117,3cyclictest219436-21latency_hist08:10:011
157363992116,3cyclictest3000860-21systemd-journal10:21:521
157363992115,4cyclictest234852-21latency_hist08:25:011
157357992117,3cyclictest250694-21kworker/u8:1+flush-259:008:55:190
157357992116,4cyclictest386732-21kworker/u8:3+flush-259:011:00:160
157369992017,2cyclictest3000860-21systemd-journal07:25:003
157369992016,3cyclictest188740-21apt-get07:40:123
157366992018,2cyclictest344730-21kworker/u8:2+events_unbound10:22:512
157366992018,2cyclictest229947-21kworker/u8:2+events_unbound09:05:272
157366992017,2cyclictest479716-21kworker/u8:4+events_unbound12:30:152
157366992017,2cyclictest402144-21kworker/u8:1+flush-259:011:25:142
157366992017,2cyclictest328640-21kworker/u8:0+events_unbound10:05:202
157366992016,3cyclictest250694-21kworker/u8:1+events_unbound08:53:172
157366992016,3cyclictest146812-21kworker/u8:4+events_unbound07:40:122
157363992017,2cyclictest381082-21kworker/u8:2+flush-259:010:50:051
157363992017,2cyclictest381082-21kworker/u8:2+flush-259:010:50:051
157363992017,2cyclictest355510-21kworker/u8:0+events_unbound10:40:161
157363992017,2cyclictest229947-21kworker/u8:2+flush-259:009:05:261
157363992015,3cyclictest479716-21kworker/u8:4+events_unbound12:25:211
157363992015,3cyclictest3000860-21systemd-journal07:45:001
157357992018,1cyclictest281606-21latency_hist09:10:010
157357992014,4cyclictest291891-21latency_hist09:20:000
157357992014,4cyclictest255806-21latency_hist08:45:010
157369991916,2cyclictest188496-21kworker/u8:5+flush-259:008:35:283
157366991916,2cyclictest355510-21kworker/u8:0+flush-259:010:40:172
157366991916,2cyclictest344730-21kworker/u8:2+flush-259:010:30:212
157366991915,3cyclictest443762-21kworker/u8:0+events_unbound11:59:532
157366991915,3cyclictest328640-21kworker/u8:0+events_unbound10:13:022
157363991916,2cyclictest443762-21kworker/u8:0+events_unbound11:50:221
157363991916,2cyclictest402144-21kworker/u8:1+events_unbound11:20:261
157363991916,2cyclictest291923-21kworker/u8:1+events_unbound09:58:531
157363991916,2cyclictest229947-21kworker/u8:2+events_unbound09:22:481
157363991916,2cyclictest198852-21latency_hist07:50:001
157363991916,2cyclictest188496-21kworker/u8:5+events_unbound08:35:291
157363991916,2cyclictest188496-21kworker/u8:5+events_unbound08:19:261
157363991915,3cyclictest464796-21kworker/u8:2+events_unbound12:24:521
157357991915,3cyclictest291923-21kworker/u8:1+flush-259:010:00:000
157369991814,3cyclictest454278-21/usr/sbin/munin12:00:213
157369991814,3cyclictest438370-21kworker/u8:4+events_unbound11:45:203
157369991813,4cyclictest3000860-21systemd-journal08:55:013
157369991813,4cyclictest254941-21ntpq08:40:223
157366991816,2cyclictest322858-21kworker/u8:3+events_unbound10:29:242
157366991816,2cyclictest229947-21kworker/u8:2+events_unbound09:02:472
157366991816,1cyclictest157340-21kworker/u8:0+events_unbound07:19:562
157366991815,2cyclictest386732-21kworker/u8:3+events_unbound11:02:512
157366991815,2cyclictest328640-21kworker/u8:0+events_unbound10:00:232
157366991815,2cyclictest291923-21kworker/u8:1+events_unbound09:25:192
157366991815,2cyclictest260891-21kworker/u8:0+events_unbound09:10:522
157366991815,2cyclictest167875-21kworker/u8:3+events_unbound07:25:212
157366991812,4cyclictest459304-21cp12:05:012
15736699180,2cyclictest214298-21sh08:04:592
157363991815,2cyclictest454268-21kworker/u8:3+events_unbound12:11:531
157363991815,2cyclictest386732-21kworker/u8:3+events_unbound10:57:511
157363991814,3cyclictest443762-21kworker/u8:0+events_unbound12:00:031
157363991814,3cyclictest417011-21kworker/u8:3+events_unbound11:35:211
157363991814,3cyclictest344730-21kworker/u8:2+events_unbound10:25:521
157357991815,2cyclictest0-21swapper/007:55:000
157357991814,3cyclictest391835-21cron11:00:000
157357991814,3cyclictest317910-21apt-get09:45:160
157357991812,4cyclictest381792-21/usr/sbin/munin10:50:160
157357991812,4cyclictest381792-21/usr/sbin/munin10:50:160
157357991812,4cyclictest240844-21diskstats08:30:170
157369991714,2cyclictest301344-21ntp_kernel_pll_09:25:223
157369991712,4cyclictest459834-21cp12:05:013
157369991712,4cyclictest355510-21kworker/u8:0+flush-259:011:30:173
157369991712,4cyclictest328611-21latency_hist10:00:003
157369991711,5cyclictest3000860-21systemd-journal08:25:013
157369991711,4cyclictest490937-21diskstats12:35:143
157366991715,2cyclictest229947-21kworker/u8:2+flush-259:008:55:512
157366991714,2cyclictest464796-21kworker/u8:2+events_unbound12:20:222
157366991714,2cyclictest229947-21kworker/u8:2+events_unbound08:34:532
157366991713,3cyclictest381149-21apt-key10:50:002
157366991713,3cyclictest188496-21kworker/u8:5+events_unbound08:29:462
157363991715,2cyclictest146812-21kworker/u8:4+flush-259:007:50:251
157363991714,2cyclictest397580-21base6411:05:011
157363991714,2cyclictest146812-21kworker/u8:4+events_unbound07:28:371
157363991713,3cyclictest250694-21kworker/u8:1+events_unbound08:50:021
157363991713,3cyclictest209430-21apt-get08:00:131
157363991712,3cyclictest328640-21kworker/u8:0+flush-259:010:05:171
157357991715,1cyclictest459192-21sh12:05:000
157357991714,2cyclictest61450irq/134-enp0s29f1:tx-011:40:270
157357991714,2cyclictest276405-21cron09:05:000
157369991613,2cyclictest3002-21cron07:35:003
157369991613,2cyclictest198843-21idleruntime-cro07:50:003
157369991610,4cyclictest214298-21latency_hist08:05:003
157366991613,2cyclictest417011-21kworker/u8:3+events_unbound11:40:172
157366991613,2cyclictest348809-21ntp_kernel_pll_10:15:212
157366991613,2cyclictest291923-21kworker/u8:1+events_unbound09:51:522
157366991613,2cyclictest146812-21kworker/u8:4+events_unbound07:30:292
157363991614,2cyclictest402144-21kworker/u8:1+flush-259:011:55:161
157363991613,2cyclictest355510-21kworker/u8:0+events_unbound11:30:181
157363991613,2cyclictest355510-21kworker/u8:0+events_unbound11:04:521
157363991613,2cyclictest229947-21kworker/u8:2+flush-259:009:45:221
157363991613,2cyclictest229947-21kworker/u8:2+flush-259:009:40:281
157363991613,2cyclictest229947-21kworker/u8:2+events_unbound08:34:551
157363991613,2cyclictest178199-21kworker/u8:2+flush-259:007:35:251
157363991612,3cyclictest229947-21kworker/u8:2+flush-259:009:35:261
157357991614,1cyclictest60650irq/133-enp0s29f1:rx-012:20:200
157357991614,1cyclictest60650irq/133-enp0s29f1:rx-011:25:160
157357991613,2cyclictest60650irq/133-enp0s29f1:rx-011:05:040
157357991613,2cyclictest60650irq/133-enp0s29f1:rx-010:35:130
157357991613,2cyclictest484556-21ntp_offset12:25:260
157357991612,3cyclictest219702-21apt-get08:10:120
157357991610,4cyclictest3000860-21systemd-journal10:05:000
15735799160,15cyclictest1-21systemd10:19:520
157369991512,2cyclictest365151-21unixbench_multi10:30:253
157369991511,3cyclictest428472-21kworker/u8:2+flush-259:011:35:573
157369991511,3cyclictest271232-21sh09:00:003
157369991510,3cyclictest407924-21/usr/sbin/munin11:15:203
157366991512,2cyclictest490489-21kworker/u8:0+events_unbound12:35:192
157366991512,2cyclictest438370-21kworker/u8:4+flush-259:011:45:262
157363991513,2cyclictest344730-21kworker/u8:2+flush-259:010:15:201
157363991512,2cyclictest408364-21irqcore11:15:171
157363991512,2cyclictest322858-21kworker/u8:3+flush-259:010:35:161
157357991513,2cyclictest60650irq/133-enp0s29f1:rx-012:40:000
157357991513,2cyclictest60650irq/133-enp0s29f1:rx-011:51:370
157357991513,1cyclictest60650irq/133-enp0s29f1:rx-012:15:160
157357991513,1cyclictest60650irq/133-enp0s29f1:rx-011:30:200
157357991513,1cyclictest60650irq/133-enp0s29f1:rx-010:40:130
157357991513,1cyclictest60650irq/133-enp0s29f1:rx-008:45:110
157357991513,1cyclictest60650irq/133-enp0s29f1:rx-007:25:250
157357991512,2cyclictest168527-21irqcore07:20:170
157357991512,2cyclictest0-21swapper/009:25:010
15736999149,4cyclictest381817-21apt-get10:50:143
15736999149,4cyclictest381817-21apt-get10:50:143
15736999149,4cyclictest344496-21latency_hist10:15:003
15736999149,4cyclictest260891-21kworker/u8:0+flush-259:009:15:023
157369991411,2cyclictest366292-21irqcore10:35:183
157369991411,2cyclictest229947-21kworker/u8:2+flush-259:009:45:013
157369991411,2cyclictest178197-21kworker/u8:0+flush-259:007:29:593
157369991411,2cyclictest0-21swapper/311:05:003
157369991410,3cyclictest365635-21kworker/u8:1+flush-259:010:40:253
157366991413,1cyclictest203644-21kworker/u8:1+events_unbound08:16:082
157366991412,2cyclictest402144-21kworker/u8:1+events_unbound12:13:092
157366991411,2cyclictest291923-21kworker/u8:1+events_unbound09:46:112
157366991411,2cyclictest203644-21kworker/u8:1+events_unbound08:10:592
157366991410,3cyclictest178199-21kworker/u8:2+events_unbound07:38:172
157363991411,2cyclictest328640-21kworker/u8:0+events_unbound10:00:191
157363991411,2cyclictest229947-21kworker/u8:2+events_unbound08:44:521
157363991410,3cyclictest344730-21kworker/u8:2+events_unbound10:30:021
157363991410,3cyclictest291923-21kworker/u8:1+events_unbound09:51:501
157363991410,3cyclictest0-21swapper/108:29:591
15735799149,4cyclictest3000860-21systemd-journal08:00:010
157357991413,1cyclictest60650irq/133-enp0s29f1:rx-008:15:160
157357991413,1cyclictest60650irq/133-enp0s29f1:rx-007:55:150
157357991413,1cyclictest60650irq/133-enp0s29f1:rx-007:55:140
157357991412,1cyclictest60650irq/133-enp0s29f1:rx-011:45:190
157357991412,1cyclictest60650irq/133-enp0s29f1:rx-011:15:170
157357991412,1cyclictest60650irq/133-enp0s29f1:rx-010:30:200
157357991412,1cyclictest60650irq/133-enp0s29f1:rx-008:51:300
157357991412,1cyclictest60650irq/133-enp0s29f1:rx-007:30:160
157357991412,1cyclictest198842-21sh07:49:590
157357991411,2cyclictest60650irq/133-enp0s29f1:rx-007:40:180
157357991411,2cyclictest469929-21sh12:15:000
157357991411,2cyclictest250253-21unixbench_singl08:35:270
157357991410,1cyclictest3085522cut09:35:180
15736999138,4cyclictest334059-21apt-get10:05:113
15736999138,3cyclictest318071-21diskstats09:45:183
157369991311,1cyclictest0-21swapper/309:30:183
157369991310,2cyclictest433830-21fschecks_count11:40:163
157369991310,2cyclictest297081-21idleruntime-cro09:25:003
157369991310,2cyclictest0-21swapper/312:30:003
157366991310,2cyclictest402144-21kworker/u8:1+flush-259:012:15:052
15735799131,9cyclictest0-21swapper/012:10:000
157357991312,1cyclictest60650irq/133-enp0s29f1:rx-011:21:420
157357991312,1cyclictest60650irq/133-enp0s29f1:rx-007:35:230
157357991311,1cyclictest60650irq/133-enp0s29f1:rx-011:35:160
157357991311,1cyclictest60650irq/133-enp0s29f1:rx-010:20:240
157357991311,1cyclictest60650irq/133-enp0s29f1:rx-010:10:000
157357991311,1cyclictest60650irq/133-enp0s29f1:rx-008:05:140
157357991311,1cyclictest60650irq/133-enp0s29f1:rx-007:10:250
15736999129,2cyclictest239096-21ntp_kernel_pll_08:25:233
15736999129,2cyclictest157877-21cpuspeed_turbos07:10:133
15736999129,1cyclictest0-21swapper/312:00:003
15736999128,3cyclictest271508-21apt-get09:00:133
15736999127,4cyclictest240683-21apt-get08:30:143
15736999127,4cyclictest167632-21sshd07:17:373
15736999126,4cyclictest3000860-21systemd-journal08:12:513
157369991210,1cyclictest0-21swapper/307:50:263
15735799129,2cyclictest0-21swapper/009:15:010
15736999119,1cyclictest0-21swapper/310:55:273
15736999119,1cyclictest0-21swapper/309:05:183
15736999118,2cyclictest402051-21unixbench_multi11:05:263
15736999118,2cyclictest214990-21irqcore08:05:203
15736999118,2cyclictest209173-21latency_hist08:00:003
15736999118,2cyclictest209173-21latency_hist07:59:593
15736999117,2cyclictest331552-21kthreadcore10:00:213
15736399118,2cyclictest167875-21kworker/u8:3+events_unbound07:20:251
15736999108,1cyclictest0-21swapper/312:10:273
15736999106,3cyclictest291923-21kworker/u8:1+flush-259:009:52:213
15735799108,1cyclictest60650irq/133-enp0s29f1:rx-009:50:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional