You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 16:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot6s.osadl.org (updated Fri Nov 28, 2025 00:43:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28548991918,1cyclictest66850irq/125-lan23:25:191
28548991918,1cyclictest66850irq/125-lan23:20:211
28548991918,1cyclictest66850irq/125-lan23:10:181
28548991918,1cyclictest66850irq/125-lan22:25:131
28548991918,1cyclictest66850irq/125-lan22:04:411
28548991918,1cyclictest66850irq/125-lan21:30:171
28548991918,1cyclictest66850irq/125-lan00:07:361
28548991918,1cyclictest66850irq/125-lan00:02:481
28555991817,1cyclictest4655-21kworker/2:1+events23:25:272
28548991818,0cyclictest66850irq/125-lan23:40:141
28548991818,0cyclictest66850irq/125-lan23:30:581
28548991818,0cyclictest66850irq/125-lan22:11:231
28548991818,0cyclictest66850irq/125-lan22:06:201
28548991818,0cyclictest66850irq/125-lan21:45:171
28548991818,0cyclictest66850irq/125-lan21:28:031
28548991818,0cyclictest66850irq/125-lan21:05:301
28548991818,0cyclictest66850irq/125-lan00:28:121
28548991818,0cyclictest66850irq/125-lan00:21:031
28548991817,1cyclictest66850irq/125-lan23:50:231
28548991817,1cyclictest66850irq/125-lan23:19:001
28548991817,1cyclictest66850irq/125-lan22:50:171
28548991817,1cyclictest66850irq/125-lan22:35:181
28548991817,1cyclictest66850irq/125-lan21:14:111
28548991817,1cyclictest66850irq/125-lan20:55:191
28548991817,1cyclictest66850irq/125-lan20:20:171
28548991817,1cyclictest66850irq/125-lan00:32:221
2856099171,16cyclictest15305-21turbostat22:19:553
28555991716,1cyclictest4655-21kworker/2:1+events22:37:522
28555991715,2cyclictest749-21ntpd22:09:112
28548991717,0cyclictest66850irq/125-lan23:55:131
28548991717,0cyclictest66850irq/125-lan23:47:221
28548991717,0cyclictest66850irq/125-lan21:52:441
28548991717,0cyclictest66850irq/125-lan21:24:021
28548991717,0cyclictest66850irq/125-lan21:15:071
28548991717,0cyclictest66850irq/125-lan00:15:221
28548991716,1cyclictest66850irq/125-lan23:05:041
28548991716,1cyclictest66850irq/125-lan23:00:181
28548991716,1cyclictest66850irq/125-lan22:55:151
28548991716,1cyclictest66850irq/125-lan22:34:101
28548991716,1cyclictest66850irq/125-lan22:22:571
28548991716,1cyclictest66850irq/125-lan21:40:171
28548991716,1cyclictest66850irq/125-lan21:38:411
28548991716,1cyclictest66850irq/125-lan21:00:231
28548991716,1cyclictest66850irq/125-lan20:25:161
28548991716,1cyclictest66850irq/125-lan20:10:141
28548991716,1cyclictest66850irq/125-lan00:37:341
28548991616,0cyclictest66850irq/125-lan23:38:151
28548991616,0cyclictest66850irq/125-lan22:15:541
28548991616,0cyclictest66850irq/125-lan21:55:161
28548991616,0cyclictest66850irq/125-lan19:53:151
28548991616,0cyclictest66850irq/125-lan19:46:251
28548991615,1cyclictest66850irq/125-lan22:45:131
28548991615,1cyclictest66850irq/125-lan22:41:261
28548991615,1cyclictest66850irq/125-lan20:00:141
28548991615,1cyclictest66850irq/125-lan00:13:251
28555991515,0cyclictest771-21snmpd23:09:112
28555991515,0cyclictest771-21snmpd22:54:282
28555991515,0cyclictest771-21snmpd00:04:402
28555991515,0cyclictest31049-21cat00:05:132
28555991515,0cyclictest1914-21cat21:30:152
28555991515,0cyclictest17683-21cat23:20:152
28548991515,0cyclictest66850irq/125-lan20:30:211
28548991514,1cyclictest66850irq/125-lan19:41:171
28548991514,1cyclictest66850irq/125-lan19:25:151
28548991513,2cyclictest66850irq/125-lan20:35:111
28555991414,0cyclictest771-21snmpd23:39:332
28555991414,0cyclictest771-21snmpd23:15:592
28555991414,0cyclictest771-21snmpd19:11:032
28555991414,0cyclictest23015-21cat20:15:152
28555991414,0cyclictest14439-21cat22:45:162
28555991413,1cyclictest4655-21kworker/2:1+events23:57:572
28548991414,0cyclictest66850irq/125-lan19:12:161
28548991413,1cyclictest66850irq/125-lan20:40:131
28548991413,1cyclictest66850irq/125-lan19:35:141
28555991313,0cyclictest771-21snmpd22:44:192
28555991313,0cyclictest771-21snmpd21:17:532
28555991313,0cyclictest771-21snmpd00:21:372
28555991312,1cyclictest749-21ntpd00:18:092
28555991312,1cyclictest4655-21kworker/2:1+events21:12:212
28555991312,1cyclictest4655-21kworker/2:1+events00:32:192
28555991312,1cyclictest4655-21kworker/2:1+events00:29:372
28548991312,1cyclictest66850irq/125-lan20:07:311
28548991312,1cyclictest66850irq/125-lan19:55:151
28548991312,1cyclictest66850irq/125-lan19:15:221
28555991212,0cyclictest771-21snmpd22:20:302
28548991212,0cyclictest66850irq/125-lan19:20:151
28548991211,1cyclictest66850irq/125-lan20:15:101
28555991111,0cyclictest771-21snmpd00:37:412
28548991111,0cyclictest66850irq/125-lan19:30:151
2856099109,1cyclictest524-21avahi-daemon00:29:543
285609998,1cyclictest771-21snmpd20:16:573
285609998,1cyclictest771-21snmpd20:00:023
285609998,1cyclictest771-21snmpd19:56:083
285609998,1cyclictest771-21snmpd19:42:523
285559998,1cyclictest769-21lldpd19:29:132
285559998,1cyclictest4655-21kworker/2:1+events22:55:192
285559998,1cyclictest4655-21kworker/2:1+events22:26:492
285559996,3cyclictest4869-21cat19:30:142
285489999,0cyclictest66850irq/125-lan20:50:151
285489999,0cyclictest66850irq/125-lan20:46:441
285429998,1cyclictest771-21snmpd21:19:200
285429998,1cyclictest769-21lldpd22:55:420
285429998,1cyclictest769-21lldpd19:20:240
285429998,1cyclictest524-21avahi-daemon22:44:350
285609988,0cyclictest771-21snmpd23:18:173
285609988,0cyclictest771-21snmpd23:13:353
285609988,0cyclictest771-21snmpd22:40:283
285609988,0cyclictest771-21snmpd21:24:473
285609988,0cyclictest771-21snmpd19:31:403
285609988,0cyclictest771-21snmpd00:16:133
285609988,0cyclictest769-21lldpd20:20:373
285609988,0cyclictest769-21lldpd19:24:513
285609987,1cyclictest771-21snmpd19:29:553
285609987,1cyclictest769-21lldpd23:56:373
285609987,1cyclictest524-21avahi-daemon21:19:113
285609987,1cyclictest524-21avahi-daemon21:11:453
285609987,1cyclictest15801-21cut23:50:003
285609986,2cyclictest771-21snmpd21:38:413
285609986,2cyclictest771-21snmpd21:05:373
285559988,0cyclictest771-21snmpd20:08:252
285559988,0cyclictest769-21lldpd00:12:572
285559987,1cyclictest771-21snmpd20:33:092
285559987,1cyclictest771-21snmpd20:27:182
285559987,1cyclictest4655-21kworker/2:1+events23:01:372
285559986,2cyclictest771-21snmpd23:45:572
285559986,2cyclictest771-21snmpd23:33:062
285559986,2cyclictest771-21snmpd19:59:472
285559986,2cyclictest749-21ntpd22:15:402
285559986,2cyclictest749-21ntpd21:45:342
285559985,3cyclictest749-21ntpd21:42:022
285429988,0cyclictest771-21snmpd23:26:570
285429988,0cyclictest771-21snmpd21:20:590
285429988,0cyclictest771-21snmpd19:35:480
285429988,0cyclictest771-21snmpd00:03:250
285429988,0cyclictest769-21lldpd20:25:390
285429988,0cyclictest769-21lldpd20:02:270
285429988,0cyclictest769-21lldpd19:44:570
285429987,1cyclictest771-21snmpd22:51:230
285429987,1cyclictest771-21snmpd22:45:140
285429987,1cyclictest771-21snmpd22:08:180
285429987,1cyclictest769-21lldpd23:47:390
285429987,1cyclictest524-21avahi-daemon21:36:540
285429984,3cyclictest255-20systemd-journal21:43:300
285609977,0cyclictest771-21snmpd22:03:153
285609977,0cyclictest771-21snmpd21:00:193
285609977,0cyclictest769-21lldpd23:00:083
285609976,1cyclictest875-21cpuspeed_turbos23:35:113
285609976,1cyclictest771-21snmpd19:37:583
285609976,1cyclictest769-21lldpd22:59:203
285609976,1cyclictest769-21lldpd22:06:513
285609976,1cyclictest584-21in:imuxsock20:29:473
285609976,1cyclictest255-20systemd-journal23:06:183
285609975,2cyclictest771-21snmpd23:51:133
285609975,2cyclictest771-21snmpd23:31:003
285609975,2cyclictest771-21snmpd23:22:413
285609975,2cyclictest771-21snmpd21:58:423
285609975,2cyclictest771-21snmpd19:54:473
285609975,2cyclictest771-21snmpd00:09:103
285609973,3cyclictest32596-21ssh21:28:513
285609973,3cyclictest22669-21fw_packets23:25:153
285609973,3cyclictest10431-21ssh22:10:063
285559977,0cyclictest771-21snmpd21:00:162
285559977,0cyclictest771-21snmpd20:46:552
285559977,0cyclictest771-21snmpd20:23:082
285559977,0cyclictest769-21lldpd23:41:362
285559976,1cyclictest749-21ntpd21:55:102
285559976,1cyclictest749-21ntpd21:25:552
285559976,1cyclictest749-21ntpd21:05:142
285559976,1cyclictest749-21ntpd19:15:152
285559976,1cyclictest4655-21kworker/2:1+events22:10:152
285559976,1cyclictest4655-21kworker/2:1+events21:50:202
285559976,1cyclictest4655-21kworker/2:1+events20:00:112
285559975,2cyclictest749-21ntpd23:10:182
285559975,2cyclictest749-21ntpd22:32:512
285559975,2cyclictest749-21ntpd22:00:382
285559975,2cyclictest749-21ntpd21:21:272
285429977,0cyclictest771-21snmpd22:02:330
285429977,0cyclictest769-21lldpd23:51:530
285429977,0cyclictest769-21lldpd22:25:090
285429977,0cyclictest769-21lldpd21:08:340
285429977,0cyclictest769-21lldpd20:58:400
285429977,0cyclictest769-21lldpd19:29:390
285429976,1cyclictest524-21avahi-daemon23:42:310
285429976,1cyclictest524-21avahi-daemon21:47:000
285429976,1cyclictest21907-21nvmesmart_nvme000:30:220
285429975,2cyclictest771-21snmpd23:38:570
285429975,2cyclictest771-21snmpd19:10:120
285429974,0cyclictest1800-21ssh22:32:400
285429973,4cyclictest550-21NetworkManager19:55:120
285429973,3cyclictest6159-21ssh21:34:530
285609966,0cyclictest771-21snmpd21:50:333
285609966,0cyclictest771-21snmpd19:46:203
285609966,0cyclictest771-21snmpd00:11:493
285609966,0cyclictest769-21lldpd20:44:063
285609965,1cyclictest771-21snmpd20:33:393
285609965,1cyclictest771-21snmpd00:21:253
285609965,1cyclictest769-21lldpd22:32:493
285609965,1cyclictest6690-21grep23:40:153
285609965,1cyclictest524-21avahi-daemon19:17:233
285609964,2cyclictest21320-21cat22:20:143
285609963,2cyclictest586-21rs:main3
285609963,2cyclictest255-20systemd-journal00:03:283
285609963,2cyclictest22499-21sshd22:53:193
285609963,2cyclictest18406-21ssh22:49:383
285609962,3cyclictest27061-21ssh00:36:243
285609962,3cyclictest11794-21ssh21:40:093
285559965,1cyclictest749-21ntpd19:20:202
285559965,1cyclictest4655-21kworker/2:1+events23:52:552
285559965,1cyclictest4655-21kworker/2:1+events21:35:182
285559965,1cyclictest4655-21kworker/2:1+events19:41:332
285559964,2cyclictest771-21snmpd20:54:272
285559964,2cyclictest771-21snmpd20:44:372
285559964,2cyclictest749-21ntpd20:55:192
285559964,2cyclictest749-21ntpd19:45:182
285429966,0cyclictest771-21snmpd23:10:380
285429966,0cyclictest771-21snmpd21:55:180
285429966,0cyclictest771-21snmpd20:12:320
285429966,0cyclictest769-21lldpd00:21:490
285429965,1cyclictest771-21snmpd22:11:090
285429965,1cyclictest771-21snmpd21:26:230
285429965,1cyclictest771-21snmpd19:16:220
285429965,1cyclictest771-21snmpd00:29:050
285429964,2cyclictest771-21snmpd23:22:540
285429964,2cyclictest771-21snmpd20:48:130
285429964,2cyclictest771-21snmpd20:06:460
285429963,2cyclictest25250-21ssh22:24:280
285429963,2cyclictest23012-21ntpq21:50:190
285429962,3cyclictest5218-21ld19:30:160
285429962,3cyclictest32450-21ssh23:34:290
285429962,3cyclictest29344-21ntp_192.168.11420:30:170
285429962,3cyclictest24975-21sed20:20:160
285429962,3cyclictest1854-21grep23:05:130
285429962,3cyclictest14599-21ssh21:10:390
285429962,3cyclictest10916-21ssh00:18:280
285609955,0cyclictest769-21lldpd21:46:373
285609955,0cyclictest517-21systemd-resolve20:07:023
285609955,0cyclictest24733-21sh00:34:573
285609954,1cyclictest31260-21fschecks_count20:35:133
285609952,2cyclictest3611-21cut20:45:183
285609952,2cyclictest29630-21awk19:10:193
285609952,2cyclictest255-20systemd-journal22:35:083
285609952,2cyclictest20557-21apt-get20:10:123
285609952,2cyclictest1324-21sh21:30:043
285609950,4cyclictest5522-21ntpq20:50:183
285559954,1cyclictest749-21ntpd19:35:232
285559953,2cyclictest749-21ntpd19:50:212
285559952,2cyclictest586-21rs:main2
285429955,0cyclictest16954-21timerwakeupswit22:15:210
285429953,2cyclictest771-21snmpd21:04:070
285429953,2cyclictest550-21NetworkManager23:55:220
285429953,2cyclictest32362-21tail20:35:220
285429952,2cyclictest5152-21cat22:35:220
285429952,2cyclictest32756-21ssh00:06:300
285429952,2cyclictest27793-21ssh00:37:470
285429952,2cyclictest23559-21rt-features20:15:200
285429952,2cyclictest22643-21kworker/u16:3+nfsiod23:01:010
285429952,2cyclictest12074-21apt-get23:15:120
285429951,3cyclictest3031-21ssh00:10:090
285609942,2cyclictest771-21snmpd20:55:103
285559943,1cyclictest4655-21kworker/2:1+events20:35:002
285429942,2cyclictest749-21ntpd20:50:190
285429940,4cyclictest13360-21ntpq19:50:190
285429940,3cyclictest889-21/usr/sbin/munin20:40:130
285429940,3cyclictest10645-21diskstats19:45:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional