You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 14:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack8slot6s.osadl.org (updated Thu Sep 18, 2025 00:43:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11555991918,1cyclictest66850irq/125-lan23:10:190
11555991918,1cyclictest66850irq/125-lan22:20:150
11555991918,1cyclictest66850irq/125-lan22:14:170
11555991918,1cyclictest66850irq/125-lan21:15:130
11555991918,1cyclictest66850irq/125-lan00:25:160
11555991818,0cyclictest66850irq/125-lan23:33:350
11555991818,0cyclictest66850irq/125-lan23:00:040
11555991818,0cyclictest66850irq/125-lan21:35:540
11555991818,0cyclictest66850irq/125-lan21:20:420
11555991818,0cyclictest66850irq/125-lan00:17:080
11555991818,0cyclictest66850irq/125-lan00:00:190
11555991817,1cyclictest66850irq/125-lan23:55:140
11555991817,1cyclictest66850irq/125-lan23:55:000
11555991817,1cyclictest66850irq/125-lan23:25:170
11555991817,1cyclictest66850irq/125-lan23:20:190
11555991817,1cyclictest66850irq/125-lan23:15:190
11555991817,1cyclictest66850irq/125-lan23:07:260
11555991817,1cyclictest66850irq/125-lan22:59:150
11555991817,1cyclictest66850irq/125-lan22:30:140
11555991817,1cyclictest66850irq/125-lan22:29:560
11555991817,1cyclictest66850irq/125-lan21:10:130
11555991817,1cyclictest66850irq/125-lan19:20:130
11555991817,1cyclictest66850irq/125-lan00:20:000
11555991817,1cyclictest66850irq/125-lan00:10:140
11555991717,0cyclictest66850irq/125-lan22:45:140
11555991717,0cyclictest66850irq/125-lan22:37:130
11555991717,0cyclictest66850irq/125-lan22:15:200
11555991717,0cyclictest66850irq/125-lan22:06:380
11555991717,0cyclictest66850irq/125-lan21:59:530
11555991717,0cyclictest66850irq/125-lan20:45:130
11555991717,0cyclictest66850irq/125-lan20:10:140
11555991717,0cyclictest66850irq/125-lan19:55:200
11555991717,0cyclictest66850irq/125-lan19:51:520
11555991717,0cyclictest66850irq/125-lan00:38:280
11555991717,0cyclictest66850irq/125-lan00:34:140
11555991716,1cyclictest66850irq/125-lan23:45:190
11555991716,1cyclictest66850irq/125-lan23:44:410
11555991716,1cyclictest66850irq/125-lan23:35:170
11555991716,1cyclictest66850irq/125-lan22:50:050
11555991716,1cyclictest66850irq/125-lan22:40:000
11555991716,1cyclictest66850irq/125-lan22:00:020
11555991716,1cyclictest66850irq/125-lan21:53:550
11555991716,1cyclictest66850irq/125-lan21:40:180
11555991616,0cyclictest66850irq/125-lan21:30:230
11555991616,0cyclictest66850irq/125-lan20:30:200
11555991616,0cyclictest66850irq/125-lan20:20:160
11555991615,1cyclictest66850irq/125-lan21:45:110
11555991615,1cyclictest66850irq/125-lan21:25:450
11555991515,0cyclictest66850irq/125-lan21:00:160
11555991515,0cyclictest66850irq/125-lan20:05:130
11555991515,0cyclictest66850irq/125-lan19:45:150
11555991515,0cyclictest66850irq/125-lan19:25:530
11555991514,1cyclictest66850irq/125-lan20:35:160
11555991514,1cyclictest66850irq/125-lan00:05:100
11555991414,0cyclictest66850irq/125-lan20:55:010
11555991414,0cyclictest66850irq/125-lan20:50:200
11555991414,0cyclictest66850irq/125-lan20:15:130
11555991414,0cyclictest66850irq/125-lan19:30:170
11555991413,1cyclictest66850irq/125-lan20:40:140
11555991413,1cyclictest66850irq/125-lan19:15:160
11555991413,1cyclictest66850irq/125-lan19:10:010
11555991313,0cyclictest66850irq/125-lan20:00:140
11555991211,1cyclictest66850irq/125-lan19:35:160
11555991111,0cyclictest771-21snmpd20:27:240
11555991111,0cyclictest66850irq/125-lan19:40:210
115679998,1cyclictest771-21snmpd00:20:583
115639998,1cyclictest771-21snmpd23:01:442
115639994,4cyclictest586-21rs:main2
115599998,1cyclictest771-21snmpd21:44:391
115679988,0cyclictest771-21snmpd21:57:063
115679988,0cyclictest771-21snmpd20:50:423
115679988,0cyclictest771-21snmpd20:15:383
115679988,0cyclictest771-21snmpd00:05:123
115679988,0cyclictest769-21lldpd23:57:123
115679987,1cyclictest771-21snmpd21:14:043
115679986,2cyclictest771-21snmpd23:54:423
115679986,2cyclictest771-21snmpd21:04:583
115679986,2cyclictest771-21snmpd20:47:063
115679986,2cyclictest771-21snmpd20:25:513
115679986,2cyclictest771-21snmpd19:35:433
115679986,2cyclictest771-21snmpd19:30:253
115639988,0cyclictest771-21snmpd23:20:112
115639988,0cyclictest771-21snmpd21:07:042
115639988,0cyclictest769-21lldpd21:48:112
115639988,0cyclictest769-21lldpd19:25:252
115639987,1cyclictest771-21snmpd00:16:222
115639987,1cyclictest769-21lldpd23:42:402
115639987,1cyclictest524-21avahi-daemon22:34:402
115639987,1cyclictest517-21systemd-resolve21:30:362
115639986,2cyclictest771-21snmpd22:37:042
115639986,2cyclictest771-21snmpd20:44:332
115599988,0cyclictest771-21snmpd21:49:021
115599988,0cyclictest771-21snmpd20:07:201
115599988,0cyclictest771-21snmpd00:31:231
115599988,0cyclictest769-21lldpd23:41:061
115599988,0cyclictest769-21lldpd20:55:181
115599988,0cyclictest769-21lldpd19:58:451
115599987,1cyclictest771-21snmpd21:23:351
115599987,1cyclictest771-21snmpd21:01:161
115599987,1cyclictest771-21snmpd20:01:021
115599987,1cyclictest771-21snmpd00:15:431
115599987,1cyclictest7105-21ntp_kernel_err23:25:171
115599987,1cyclictest524-21avahi-daemon23:07:271
115599987,1cyclictest517-21systemd-resolve22:33:421
115599986,2cyclictest771-21snmpd21:10:581
115599986,2cyclictest771-21snmpd21:07:161
115599986,2cyclictest771-21snmpd20:33:151
115599986,2cyclictest771-21snmpd20:25:481
115599986,2cyclictest771-21snmpd20:16:201
115599986,2cyclictest771-21snmpd20:12:501
115599986,2cyclictest771-21snmpd19:23:001
115559987,1cyclictest66850irq/125-lan21:10:000
115679977,0cyclictest771-21snmpd23:12:123
115679977,0cyclictest771-21snmpd22:11:543
115679977,0cyclictest769-21lldpd23:32:203
115679977,0cyclictest769-21lldpd23:06:163
115679977,0cyclictest769-21lldpd22:05:413
115679977,0cyclictest769-21lldpd21:53:433
115679977,0cyclictest769-21lldpd20:00:453
115679977,0cyclictest769-21lldpd19:17:213
115679977,0cyclictest769-21lldpd00:31:173
115679976,1cyclictest771-21snmpd21:08:163
115679976,1cyclictest771-21snmpd00:10:073
115679976,1cyclictest769-21lldpd21:19:503
115679975,2cyclictest771-21snmpd23:47:333
115679975,2cyclictest771-21snmpd23:29:423
115679975,2cyclictest771-21snmpd19:29:103
115679973,3cyclictest29848-21fw_conntrack22:45:123
115679973,3cyclictest24217-21ssh21:36:153
115679973,3cyclictest15007-21ssh22:31:073
115679973,3cyclictest13515-21ssh21:26:413
115639977,0cyclictest771-21snmpd21:20:522
115639977,0cyclictest771-21snmpd20:14:262
115639977,0cyclictest771-21snmpd00:22:162
115639977,0cyclictest769-21lldpd19:14:302
115639977,0cyclictest517-21systemd-resolve00:30:162
115639976,1cyclictest769-21lldpd23:17:342
115639976,1cyclictest524-21avahi-daemon23:56:472
115639976,1cyclictest524-21avahi-daemon23:53:342
115639976,1cyclictest524-21avahi-daemon22:52:432
115639976,1cyclictest524-21avahi-daemon19:51:242
115639976,1cyclictest517-21systemd-resolve22:05:202
115639976,1cyclictest517-21systemd-resolve21:26:572
115639975,2cyclictest771-21snmpd21:10:282
115639975,2cyclictest771-21snmpd20:59:402
115639975,2cyclictest771-21snmpd20:29:332
115639975,2cyclictest6458-21cat20:15:152
115639975,2cyclictest22681-21cat19:35:142
115639974,3cyclictest255-20systemd-journal22:10:152
115639974,3cyclictest255-20systemd-journal21:38:562
115639972,4cyclictest255-20systemd-journal22:04:002
115639972,4cyclictest22654-21cat20:55:002
115599977,0cyclictest771-21snmpd22:13:131
115599977,0cyclictest771-21snmpd00:23:581
115599977,0cyclictest769-21lldpd23:22:301
115599977,0cyclictest769-21lldpd23:14:461
115599977,0cyclictest769-21lldpd22:50:261
115599977,0cyclictest769-21lldpd22:27:251
115599976,1cyclictest771-21snmpd00:09:121
115599975,2cyclictest771-21snmpd23:49:091
115599975,2cyclictest25190-21ls19:40:181
115599974,2cyclictest29869-21grep22:45:131
115599972,5cyclictest550-21NetworkManager21:18:031
115679966,0cyclictest771-21snmpd20:23:423
115679966,0cyclictest769-21lldpd22:19:393
115679966,0cyclictest769-21lldpd20:40:143
115679966,0cyclictest524-21avahi-daemon23:39:553
115679966,0cyclictest524-21avahi-daemon20:39:533
115679966,0cyclictest517-21systemd-resolve22:42:083
115679965,1cyclictest771-21snmpd20:12:323
115679965,1cyclictest2316-21gltestperf20:05:153
115679965,1cyclictest19527-21nvmesmart_nvme022:35:193
115679964,2cyclictest771-21snmpd23:02:353
115679964,2cyclictest771-21snmpd22:21:543
115679964,2cyclictest771-21snmpd00:28:053
115679963,3cyclictest749-21ntpd22:54:123
115679963,3cyclictest550-21NetworkManager19:20:143
115679963,2cyclictest28041-21awk21:40:173
115639966,0cyclictest771-21snmpd21:18:132
115639966,0cyclictest771-21snmpd20:21:022
115639966,0cyclictest771-21snmpd00:04:422
115639966,0cyclictest769-21lldpd22:25:382
115639966,0cyclictest769-21lldpd20:37:402
115639966,0cyclictest769-21lldpd20:09:472
115639966,0cyclictest769-21lldpd19:45:212
115639965,1cyclictest771-21snmpd20:03:532
115639965,1cyclictest771-21snmpd19:59:592
115639965,1cyclictest524-21avahi-daemon00:12:592
115639965,1cyclictest18095-21cpuspeed_turbos23:05:132
115639964,2cyclictest771-21snmpd22:45:522
115639964,2cyclictest771-21snmpd19:42:462
115639964,2cyclictest771-21snmpd19:18:272
115639963,3cyclictest550-21NetworkManager23:12:302
115639963,3cyclictest550-21NetworkManager21:50:042
115639962,4cyclictest517-21systemd-resolve22:21:082
115639961,3cyclictest29222-21cat22:45:002
115599966,0cyclictest771-21snmpd20:53:011
115599966,0cyclictest771-21snmpd19:39:551
115599966,0cyclictest771-21snmpd19:19:481
115599966,0cyclictest769-21lldpd23:35:081
115599966,0cyclictest769-21lldpd00:37:071
115599966,0cyclictest769-21lldpd00:03:491
115599966,0cyclictest524-21avahi-daemon19:28:001
115599965,1cyclictest8250-21fschecks_count20:20:131
115599965,1cyclictest584-21in:imuxsock00:29:141
115599964,2cyclictest771-21snmpd22:20:331
115599964,2cyclictest771-21snmpd22:19:061
115599964,2cyclictest771-21snmpd22:02:451
115599964,2cyclictest771-21snmpd21:36:471
115599964,2cyclictest771-21snmpd19:52:101
115599964,2cyclictest771-21snmpd19:48:041
115599964,2cyclictest550-21NetworkManager23:51:181
115599962,3cyclictest20249-21sh00:13:131
115599962,3cyclictest20246-21ntpq22:05:181
115599962,3cyclictest11432-21ssh21:56:441
115679953,2cyclictest771-21snmpd21:45:113
115679952,3cyclictest771-21snmpd00:16:463
115679952,3cyclictest550-21NetworkManager00:35:463
115679952,2cyclictest9746-21ssh21:23:343
115679952,2cyclictest6559-21cups-browsed00:00:023
115679952,2cyclictest586-21rs:main3
115679952,2cyclictest28353-21grep23:15:123
115679952,2cyclictest27266-21awk19:45:183
115679952,2cyclictest23407-21ssh23:40:393
115679952,2cyclictest14870-21missed_timers22:00:163
115679952,2cyclictest12475-21grep19:10:173
115679952,2cyclictest10325-21ssh22:57:133
115679952,1cyclictest191rcu_preempt22:25:163
115679951,3cyclictest28513-21diskstats19:50:143
115679951,3cyclictest24433-21grep19:40:123
115679951,3cyclictest16497-21ssh21:30:043
115679950,4cyclictest23570-21sed20:55:143
115639955,0cyclictest524-21avahi-daemon21:56:342
115639955,0cyclictest12928-21ntpq20:30:162
115639954,1cyclictest771-21snmpd22:56:592
115639952,2cyclictest586-21rs:main2
115639952,2cyclictest31256-21unixbench_multi00:25:222
115639952,2cyclictest30768-21ssh21:43:212
115639952,2cyclictest21437-21sed19:30:222
115639952,2cyclictest16938-21wc19:20:212
115639952,2cyclictest14289-21awk00:05:182
115639952,2cyclictest12296-21ntpq23:30:152
115639951,3cyclictest688-21ssh22:18:042
115599954,1cyclictest11549-21cyclictest20:45:161
115599953,2cyclictest771-21snmpd20:40:481
115599953,2cyclictest586-21rs:main1
115599953,2cyclictest550-21NetworkManager23:00:071
115599952,2cyclictest5788-21ssh21:51:091
115599952,2cyclictest24908-21load22:40:151
115599952,2cyclictest21056-21ld19:30:181
115599952,2cyclictest12575-21wc23:30:181
115599952,2cyclictest11869-21nfs4_client21:25:171
115599952,2cyclictest11106-21ssh22:58:171
115599950,5cyclictest0-21swapper/123:55:191
115599950,4cyclictest16188-21base6421:30:011
115679942,2cyclictest550-21NetworkManager19:59:183
115639941,2cyclictest19365-21ntp_kernel_pll_20:45:172
115639940,3cyclictest6709-21expr23:25:142
115639940,3cyclictest17298-21ntp_192.168.11423:35:172
115599942,2cyclictest28928-21ntpq23:15:161
115599941,2cyclictest11881-21cpuspeed_turbos19:10:131
115599940,4cyclictest14822-21ntpq20:35:161
115639932,1cyclictest771-21snmpd21:00:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional