You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-17 - 13:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot6s.osadl.org (updated Wed Sep 17, 2025 00:43:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13171991919,0cyclictest66850irq/125-lan22:48:350
13171991918,1cyclictest66850irq/125-lan21:20:150
13171991918,1cyclictest66850irq/125-lan21:10:220
13171991818,0cyclictest66850irq/125-lan23:35:570
13171991818,0cyclictest66850irq/125-lan22:40:130
13171991818,0cyclictest66850irq/125-lan22:36:440
13171991818,0cyclictest66850irq/125-lan21:55:180
13171991818,0cyclictest66850irq/125-lan21:30:130
13171991818,0cyclictest66850irq/125-lan21:27:480
13171991818,0cyclictest66850irq/125-lan20:48:560
13171991818,0cyclictest66850irq/125-lan19:35:220
13171991817,1cyclictest66850irq/125-lan23:56:380
13171991817,1cyclictest66850irq/125-lan23:50:030
13171991817,1cyclictest66850irq/125-lan23:45:110
13171991817,1cyclictest66850irq/125-lan23:42:220
13171991817,1cyclictest66850irq/125-lan23:30:200
13171991817,1cyclictest66850irq/125-lan23:15:170
13171991817,1cyclictest66850irq/125-lan23:05:150
13171991817,1cyclictest66850irq/125-lan23:00:180
13171991817,1cyclictest66850irq/125-lan22:56:040
13171991817,1cyclictest66850irq/125-lan22:33:150
13171991817,1cyclictest66850irq/125-lan21:45:170
13171991817,1cyclictest66850irq/125-lan21:42:470
13171991817,1cyclictest66850irq/125-lan20:20:140
13171991817,1cyclictest66850irq/125-lan00:39:210
13171991817,1cyclictest66850irq/125-lan00:27:020
13171991817,1cyclictest66850irq/125-lan00:05:150
13171991717,0cyclictest66850irq/125-lan22:01:260
13171991717,0cyclictest66850irq/125-lan21:15:150
13171991717,0cyclictest66850irq/125-lan19:49:340
13171991717,0cyclictest66850irq/125-lan19:25:300
13171991717,0cyclictest66850irq/125-lan00:33:030
13171991717,0cyclictest66850irq/125-lan00:16:360
13171991716,1cyclictest66850irq/125-lan23:28:250
13171991716,1cyclictest66850irq/125-lan23:20:120
13171991716,1cyclictest66850irq/125-lan23:12:510
13171991716,1cyclictest66850irq/125-lan22:25:120
13171991716,1cyclictest66850irq/125-lan22:16:070
13171991716,1cyclictest66850irq/125-lan22:12:250
13171991716,1cyclictest66850irq/125-lan21:51:310
13171991716,1cyclictest66850irq/125-lan21:35:210
13171991716,1cyclictest66850irq/125-lan20:50:210
13171991716,1cyclictest66850irq/125-lan00:14:300
13171991616,0cyclictest66850irq/125-lan21:09:220
13171991616,0cyclictest66850irq/125-lan21:01:460
13171991616,0cyclictest66850irq/125-lan20:41:180
13171991616,0cyclictest66850irq/125-lan20:41:170
13171991616,0cyclictest66850irq/125-lan20:25:230
13171991616,0cyclictest66850irq/125-lan20:10:270
13171991616,0cyclictest66850irq/125-lan19:55:160
13171991616,0cyclictest66850irq/125-lan19:50:020
13171991616,0cyclictest66850irq/125-lan19:15:210
13171991615,1cyclictest66850irq/125-lan22:50:150
13171991615,1cyclictest66850irq/125-lan22:20:170
13171991615,1cyclictest66850irq/125-lan00:21:580
13171991615,1cyclictest66850irq/125-lan00:00:230
13171991515,0cyclictest66850irq/125-lan22:05:160
13171991514,1cyclictest66850irq/125-lan20:35:110
13171991514,1cyclictest66850irq/125-lan19:40:250
13171991514,1cyclictest66850irq/125-lan19:10:160
13171991414,0cyclictest66850irq/125-lan20:55:160
13171991414,0cyclictest66850irq/125-lan19:20:130
13171991212,0cyclictest66850irq/125-lan20:16:450
13171991212,0cyclictest66850irq/125-lan19:32:450
13171991211,1cyclictest66850irq/125-lan20:05:150
1317999117,3cyclictest14193-21open_inodes21:25:232
13171991111,0cyclictest66850irq/125-lan20:00:210
13171991110,1cyclictest66850irq/125-lan20:30:460
131829998,1cyclictest771-21snmpd21:59:503
131829998,1cyclictest771-21snmpd21:05:433
131829998,1cyclictest771-21snmpd20:56:153
131799998,1cyclictest771-21snmpd23:17:342
131759998,1cyclictest769-21lldpd20:52:251
131759998,1cyclictest524-21avahi-daemon21:41:261
131759998,1cyclictest28005-21cut21:40:001
131829988,0cyclictest771-21snmpd22:59:283
131829988,0cyclictest771-21snmpd00:17:423
131829988,0cyclictest771-21snmpd00:14:273
131829988,0cyclictest769-21lldpd22:12:443
131829988,0cyclictest769-21lldpd19:53:243
131829988,0cyclictest769-21lldpd19:11:553
131829987,1cyclictest771-21snmpd21:29:523
131829987,1cyclictest771-21snmpd20:02:113
131829987,1cyclictest524-21avahi-daemon22:34:393
131829986,2cyclictest771-21snmpd20:46:293
131829986,2cyclictest771-21snmpd20:08:463
131829986,2cyclictest771-21snmpd19:49:183
131799988,0cyclictest771-21snmpd22:53:022
131799988,0cyclictest771-21snmpd21:46:532
131799988,0cyclictest769-21lldpd23:47:052
131799988,0cyclictest769-21lldpd23:22:502
131799988,0cyclictest769-21lldpd22:57:452
131799987,1cyclictest25596-21fschecks_time22:10:102
131799987,0cyclictest191rcu_preempt20:45:162
131799986,2cyclictest8052-21cat20:15:172
131799986,2cyclictest771-21snmpd22:29:302
131799986,2cyclictest771-21snmpd19:48:102
131799986,2cyclictest771-21snmpd19:42:162
131799986,2cyclictest749-21ntpd23:41:022
131799986,2cyclictest749-21ntpd00:18:132
131799986,0cyclictest191rcu_preempt22:34:372
131799985,2cyclictest0-21swapper/219:35:212
131799984,2cyclictest0-21swapper/222:21:332
131799983,3cyclictest31039-21sh21:42:032
131799982,3cyclictest14266-21cut19:10:202
131799980,4cyclictest28510-21tr00:25:142
131759988,0cyclictest771-21snmpd23:38:221
131759988,0cyclictest771-21snmpd19:18:481
131759988,0cyclictest769-21lldpd19:29:551
131759988,0cyclictest769-21lldpd00:31:201
131759987,1cyclictest771-21snmpd19:21:151
131759987,1cyclictest590-21gmain23:46:421
131759987,1cyclictest517-21systemd-resolve20:00:171
131759986,2cyclictest771-21snmpd23:32:171
131759986,2cyclictest771-21snmpd22:52:021
131759986,2cyclictest24559-21tail20:55:151
131759986,2cyclictest16888-21cat22:00:151
131759986,2cyclictest11783-21cat00:05:141
131829977,0cyclictest771-21snmpd23:15:493
131829977,0cyclictest771-21snmpd22:35:033
131829977,0cyclictest771-21snmpd21:19:333
131829977,0cyclictest771-21snmpd19:23:583
131829977,0cyclictest771-21snmpd00:31:133
131829977,0cyclictest771-21snmpd00:02:273
131829977,0cyclictest769-21lldpd23:34:553
131829977,0cyclictest769-21lldpd22:25:183
131829977,0cyclictest769-21lldpd20:44:253
131829977,0cyclictest769-21lldpd20:44:253
131829976,1cyclictest771-21snmpd19:56:233
131829976,1cyclictest771-21snmpd19:36:223
131829976,1cyclictest524-21avahi-daemon00:37:153
131829976,1cyclictest517-21systemd-resolve21:36:083
131829976,1cyclictest255-20systemd-journal22:41:373
131829975,2cyclictest771-21snmpd23:50:263
131829975,2cyclictest771-21snmpd23:46:503
131829975,2cyclictest771-21snmpd22:18:083
131829975,2cyclictest771-21snmpd22:03:203
131829975,2cyclictest771-21snmpd21:41:403
131829975,2cyclictest771-21snmpd21:24:313
131829975,2cyclictest771-21snmpd19:34:123
131829975,2cyclictest19180-21cat23:40:183
131829975,2cyclictest11761-21cat00:05:153
131829974,3cyclictest1358-21diskstats22:50:123
131799977,0cyclictest769-21lldpd23:05:122
131799977,0cyclictest25123-21gcc00:20:182
131799976,1cyclictest771-21snmpd20:32:452
131799976,1cyclictest749-21ntpd20:35:142
131799976,1cyclictest584-21in:imuxsock22:00:212
131799976,0cyclictest191rcu_preempt23:31:092
131799976,0cyclictest191rcu_preempt23:28:312
131799976,0cyclictest191rcu_preempt23:11:152
131799976,0cyclictest191rcu_preempt22:35:152
131799976,0cyclictest191rcu_preempt21:22:452
131799976,0cyclictest191rcu_preempt00:35:212
131799976,0cyclictest191rcu_preempt00:02:562
131799975,2cyclictest771-21snmpd21:11:372
131799975,2cyclictest749-21ntpd00:10:032
131799974,2cyclictest21155-21kworker/u16:2+xprtiod22:07:372
131799974,1cyclictest191rcu_preempt23:35:182
131799974,1cyclictest191rcu_preempt19:20:172
131799973,3cyclictest43-21ksoftirqd/223:50:172
131799973,3cyclictest11444-21diskstats21:55:132
131799973,2cyclictest0-21swapper/221:30:152
131799972,2cyclictest43-21ksoftirqd/223:58:182
131799971,4cyclictest43-21ksoftirqd/219:30:442
131799971,3cyclictest43-21ksoftirqd/222:40:132
131759977,0cyclictest771-21snmpd22:44:451
131759977,0cyclictest771-21snmpd22:31:241
131759977,0cyclictest769-21lldpd22:46:561
131759977,0cyclictest769-21lldpd21:53:351
131759977,0cyclictest769-21lldpd21:10:491
131759977,0cyclictest769-21lldpd20:40:581
131759977,0cyclictest769-21lldpd20:40:581
131759977,0cyclictest769-21lldpd19:32:331
131759977,0cyclictest524-21avahi-daemon23:51:231
131759976,1cyclictest771-21snmpd21:34:191
131759976,1cyclictest771-21snmpd21:01:391
131759976,1cyclictest771-21snmpd20:37:291
131759976,1cyclictest771-21snmpd20:10:351
131759976,1cyclictest771-21snmpd19:38:421
131759976,1cyclictest517-21systemd-resolve23:56:321
131759975,2cyclictest771-21snmpd23:27:311
131759975,2cyclictest771-21snmpd22:55:311
131759975,2cyclictest771-21snmpd20:30:571
131759975,2cyclictest771-21snmpd20:22:101
131759973,3cyclictest19612-21ntp_states23:40:211
131829966,0cyclictest771-21snmpd22:09:413
131829966,0cyclictest769-21lldpd23:27:313
131829966,0cyclictest769-21lldpd22:48:203
131829966,0cyclictest769-21lldpd19:43:343
131829965,1cyclictest11721-21ntp_kernel_err23:00:163
131829964,2cyclictest771-21snmpd20:25:353
131829963,3cyclictest550-21NetworkManager21:52:093
131829963,2cyclictest20811-21ssh23:10:023
131829963,2cyclictest16105-21ssh23:36:453
131829962,3cyclictest3217-21ssh23:55:353
131829962,3cyclictest30592-21diskstats21:10:133
131799966,0cyclictest771-21snmpd20:53:032
131799965,1cyclictest517-21systemd-resolve23:00:082
131799965,1cyclictest411rcuc/220:20:122
131799965,1cyclictest255-20systemd-journal22:45:032
131799965,0cyclictest191rcu_preempt21:52:042
131799965,0cyclictest191rcu_preempt00:05:082
131799964,2cyclictest749-21ntpd20:40:032
131799964,2cyclictest749-21ntpd20:40:032
131799964,0cyclictest191rcu_preempt20:05:202
131799963,2cyclictest550-21NetworkManager21:35:162
131799963,2cyclictest13232-21sshd20:27:462
131799963,1cyclictest43-21ksoftirqd/219:50:002
131799963,0cyclictest191rcu_preempt21:05:222
131799962,3cyclictest31782-21sed22:15:222
131799962,2cyclictest43-21ksoftirqd/220:10:182
131799962,1cyclictest191rcu_preempt20:55:192
131799962,1cyclictest0-21swapper/221:18:562
131799962,1cyclictest0-21swapper/220:02:032
131799961,3cyclictest43-21ksoftirqd/219:55:212
131799960,4cyclictest32269-21apt-get00:30:122
131799960,3cyclictest26436-21diskstats21:00:142
131759966,0cyclictest769-21lldpd00:20:021
131759965,1cyclictest771-21snmpd20:26:561
131759965,1cyclictest771-21snmpd20:07:311
131759965,1cyclictest771-21snmpd00:11:571
131759965,1cyclictest524-21avahi-daemon21:06:161
131759965,1cyclictest517-21systemd-resolve22:05:461
131759965,1cyclictest21214-21fschecks_time23:10:161
131759965,1cyclictest0-21swapper/100:00:021
131759964,2cyclictest771-21snmpd21:20:191
131759964,2cyclictest771-21snmpd19:51:281
131759964,2cyclictest20521-21cat00:15:151
131759963,3cyclictest550-21NetworkManager23:05:291
131759963,2cyclictest5028-21ntpq00:35:191
131829955,0cyclictest771-21snmpd21:02:033
131829955,0cyclictest769-21lldpd23:20:533
131829954,1cyclictest1798-21needreboot21:45:173
131829954,1cyclictest15996-21gltestperf20:35:123
131829953,2cyclictest6341-21ntpq20:10:163
131829953,1cyclictest12738-21kworker/u16:0+nfsiod23:07:123
131829952,3cyclictest28735-21fw_packets00:25:153
131829952,2cyclictest9484-21sed20:20:013
131829952,2cyclictest6488-21ssh22:22:543
131829952,2cyclictest20433-21ssh21:31:273
131829952,2cyclictest10356-21ntp_192.168.11420:20:153
131829951,3cyclictest255-20systemd-journal20:32:393
131829951,2cyclictest22886-21head20:50:173
131799951,1cyclictest0-21swapper/219:15:182
131799950,1cyclictest19413-21apt-get19:25:102
131759955,0cyclictest13557-21gltestperf21:25:181
131759954,1cyclictest32023-21cpuspeed_turbos19:55:161
131759954,1cyclictest27793-21gltestperf19:45:141
131759953,2cyclictest771-21snmpd21:55:201
131759953,2cyclictest771-21snmpd20:16:351
131759953,2cyclictest771-21snmpd19:40:251
131759953,2cyclictest550-21NetworkManager22:15:371
131759952,2cyclictest5608-21ssh22:21:391
131759952,2cyclictest4609-21ssh21:15:361
131759952,2cyclictest30160-21kworker/u16:2+nfsiod21:45:151
131759952,2cyclictest28029-21awk00:25:001
131759952,2cyclictest26197-21grep22:10:141
131759952,2cyclictest25942-21sh23:15:121
131759952,2cyclictest21290-21ssh22:37:091
131759952,2cyclictest13028-21ssh23:01:171
131759951,3cyclictest32674-21ssh23:20:421
131759951,3cyclictest13866-21sed19:10:161
131759950,4cyclictest771-21snmpd22:25:151
131829942,2cyclictest749-21ntpd19:25:023
131829940,3cyclictest16176-21ntpq19:15:193
131759941,2cyclictest19923-21dpkg20:45:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional