You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 00:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot6s.osadl.org (updated Sun Nov 23, 2025 12:43:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18814994040,0cyclictest24494-21fwupd07:21:191
18821991918,1cyclictest66850irq/125-lan11:50:233
18821991918,1cyclictest66850irq/125-lan09:50:193
18821991918,1cyclictest66850irq/125-lan09:20:173
18821991918,1cyclictest66850irq/125-lan07:24:373
18821991818,0cyclictest66850irq/125-lan11:39:123
18821991818,0cyclictest66850irq/125-lan10:35:193
18821991818,0cyclictest66850irq/125-lan09:40:163
18821991817,1cyclictest66850irq/125-lan12:32:093
18821991817,1cyclictest66850irq/125-lan12:22:583
18821991817,1cyclictest66850irq/125-lan12:11:323
18821991817,1cyclictest66850irq/125-lan12:06:493
18821991817,1cyclictest66850irq/125-lan11:44:223
18821991817,1cyclictest66850irq/125-lan11:25:123
18821991817,1cyclictest66850irq/125-lan11:00:043
18821991817,1cyclictest66850irq/125-lan10:45:263
18821991817,1cyclictest66850irq/125-lan10:43:343
18821991817,1cyclictest66850irq/125-lan10:31:013
18821991817,1cyclictest66850irq/125-lan10:21:263
18821991817,1cyclictest66850irq/125-lan10:15:203
18821991817,1cyclictest66850irq/125-lan10:13:053
18821991817,1cyclictest66850irq/125-lan09:57:573
18821991817,1cyclictest66850irq/125-lan09:17:333
18821991817,1cyclictest66850irq/125-lan07:40:003
18821991717,0cyclictest66850irq/125-lan11:06:213
18821991717,0cyclictest66850irq/125-lan10:25:203
18821991717,0cyclictest66850irq/125-lan10:05:133
18821991717,0cyclictest66850irq/125-lan09:25:213
18821991717,0cyclictest66850irq/125-lan08:55:143
18821991716,1cyclictest66850irq/125-lan12:25:243
18821991716,1cyclictest66850irq/125-lan12:18:463
18821991716,1cyclictest66850irq/125-lan11:55:473
18821991716,1cyclictest66850irq/125-lan11:10:313
18821991716,1cyclictest66850irq/125-lan09:46:213
18821991716,1cyclictest66850irq/125-lan09:35:323
18821991716,1cyclictest66850irq/125-lan09:10:343
18821991616,0cyclictest66850irq/125-lan12:38:353
18821991616,0cyclictest66850irq/125-lan11:34:553
18821991616,0cyclictest66850irq/125-lan10:51:193
18821991615,1cyclictest66850irq/125-lan11:45:123
18821991615,1cyclictest66850irq/125-lan11:21:293
18821991615,1cyclictest66850irq/125-lan10:55:143
18821991615,1cyclictest66850irq/125-lan10:01:393
18821991615,1cyclictest66850irq/125-lan09:34:183
18821991615,1cyclictest66850irq/125-lan08:20:173
18821991615,1cyclictest66850irq/125-lan08:00:173
18821991615,1cyclictest66850irq/125-lan07:50:163
18821991615,1cyclictest66850irq/125-lan07:45:193
18821991615,1cyclictest66850irq/125-lan07:25:153
18821991515,0cyclictest66850irq/125-lan07:30:303
18821991515,0cyclictest66850irq/125-lan07:19:003
18821991514,1cyclictest66850irq/125-lan12:00:173
18821991514,1cyclictest66850irq/125-lan11:17:113
18821991514,1cyclictest66850irq/125-lan08:50:173
18821991414,0cyclictest66850irq/125-lan08:25:163
18821991414,0cyclictest66850irq/125-lan07:10:143
18821991413,1cyclictest66850irq/125-lan07:40:243
18821991313,0cyclictest66850irq/125-lan09:05:173
18821991313,0cyclictest66850irq/125-lan08:35:123
18821991313,0cyclictest66850irq/125-lan08:05:173
18821991211,1cyclictest66850irq/125-lan08:59:593
18821991211,1cyclictest66850irq/125-lan08:18:373
18821991211,1cyclictest66850irq/125-lan07:55:173
18821991111,0cyclictest66850irq/125-lan08:10:213
18821991010,0cyclictest66850irq/125-lan08:30:173
188219999,0cyclictest66850irq/125-lan08:49:173
188189994,5cyclictest550-21NetworkManager08:55:182
188149998,1cyclictest771-21snmpd09:38:001
188149998,1cyclictest771-21snmpd09:01:381
188099998,1cyclictest771-21snmpd12:09:310
188099998,1cyclictest517-21systemd-resolve09:22:030
188099998,1cyclictest255-20systemd-journal10:53:410
188219987,1cyclictest66850irq/125-lan08:40:173
188189988,0cyclictest771-21snmpd10:44:182
188189988,0cyclictest771-21snmpd09:14:542
188189988,0cyclictest771-21snmpd07:55:212
188189988,0cyclictest769-21lldpd10:48:212
188189988,0cyclictest769-21lldpd07:53:482
188189988,0cyclictest769-21lldpd07:26:582
188189987,1cyclictest771-21snmpd10:22:532
188189987,1cyclictest524-21avahi-daemon09:16:072
188189986,2cyclictest771-21snmpd12:15:032
188189986,2cyclictest771-21snmpd11:25:252
188189986,2cyclictest771-21snmpd08:45:522
188189986,2cyclictest771-21snmpd07:20:382
188149988,0cyclictest771-21snmpd12:06:351
188149988,0cyclictest771-21snmpd10:25:081
188149988,0cyclictest771-21snmpd08:59:561
188149988,0cyclictest771-21snmpd08:52:441
188149988,0cyclictest771-21snmpd08:30:091
188149988,0cyclictest771-21snmpd08:27:521
188149988,0cyclictest769-21lldpd11:36:591
188149988,0cyclictest769-21lldpd08:39:551
188149988,0cyclictest517-21systemd-resolve11:28:121
188149987,1cyclictest771-21snmpd09:17:301
188149987,1cyclictest771-21snmpd08:42:051
188149987,1cyclictest769-21lldpd12:17:481
188149987,1cyclictest524-21avahi-daemon11:12:101
188149987,1cyclictest524-21avahi-daemon08:45:201
188149986,2cyclictest771-21snmpd12:35:211
188149986,2cyclictest771-21snmpd07:50:241
188099988,0cyclictest771-21snmpd12:30:330
188099988,0cyclictest771-21snmpd11:50:520
188099988,0cyclictest771-21snmpd09:51:490
188099988,0cyclictest771-21snmpd09:35:050
188099987,1cyclictest769-21lldpd12:15:520
188099987,1cyclictest524-21avahi-daemon10:41:450
188099986,2cyclictest771-21snmpd09:49:160
188099985,2cyclictest12966-21grep10:30:140
188189977,0cyclictest771-21snmpd10:10:462
188189977,0cyclictest771-21snmpd09:37:572
188189977,0cyclictest771-21snmpd09:31:472
188189977,0cyclictest771-21snmpd09:09:502
188189977,0cyclictest771-21snmpd07:40:302
188189977,0cyclictest771-21snmpd07:30:432
188189977,0cyclictest769-21lldpd10:33:362
188189977,0cyclictest769-21lldpd09:00:222
188189976,1cyclictest771-21snmpd12:20:572
188189976,1cyclictest769-21lldpd08:07:342
188189976,1cyclictest517-21systemd-resolve11:36:462
188189976,1cyclictest421ktimers/208:50:002
188189976,1cyclictest25458-21cut12:25:152
188189975,2cyclictest771-21snmpd11:33:012
188189975,2cyclictest771-21snmpd07:38:542
188189973,3cyclictest29651-21apt-get12:30:142
188189973,3cyclictest26191-21ssh11:54:512
188189973,3cyclictest255-20systemd-journal09:42:092
188189973,3cyclictest10942-21ssh09:53:422
188189972,4cyclictest255-20systemd-journal10:17:552
188149977,0cyclictest9476-21turbostat.cron08:04:591
188149977,0cyclictest771-21snmpd12:13:391
188149977,0cyclictest771-21snmpd11:00:151
188149977,0cyclictest771-21snmpd10:00:521
188149977,0cyclictest771-21snmpd09:43:041
188149977,0cyclictest771-21snmpd07:25:201
188149977,0cyclictest769-21lldpd12:00:481
188149977,0cyclictest769-21lldpd10:32:061
188149977,0cyclictest769-21lldpd10:14:361
188149977,0cyclictest769-21lldpd10:06:021
188149977,0cyclictest769-21lldpd07:18:301
188149976,1cyclictest771-21snmpd10:22:491
188149976,1cyclictest771-21snmpd09:25:421
188149976,1cyclictest771-21snmpd09:06:111
188149976,1cyclictest771-21snmpd08:19:361
188149976,1cyclictest771-21snmpd07:13:561
188149976,1cyclictest769-21lldpd09:20:411
188149976,1cyclictest769-21lldpd08:23:521
188149976,1cyclictest769-21lldpd07:40:141
188149976,1cyclictest25796-21sh10:43:461
188149975,2cyclictest771-21snmpd10:16:291
188149975,2cyclictest771-21snmpd09:51:341
188149975,2cyclictest13533-21cat11:05:141
188149975,2cyclictest12170-21cat08:10:161
188149974,2cyclictest255-20systemd-journal09:30:001
188099977,0cyclictest8756-21needreboot11:35:150
188099977,0cyclictest771-21snmpd12:37:240
188099977,0cyclictest771-21snmpd10:18:170
188099977,0cyclictest771-21snmpd09:09:260
188099977,0cyclictest771-21snmpd07:37:390
188099977,0cyclictest769-21lldpd11:30:210
188099977,0cyclictest769-21lldpd11:14:570
188099977,0cyclictest769-21lldpd10:20:030
188099977,0cyclictest769-21lldpd08:43:290
188099976,1cyclictest769-21lldpd11:41:270
188099976,1cyclictest769-21lldpd09:43:400
188099976,1cyclictest524-21avahi-daemon09:26:220
188099976,1cyclictest517-21systemd-resolve09:32:240
188099975,2cyclictest771-21snmpd11:55:080
188099975,2cyclictest771-21snmpd10:26:050
188099975,2cyclictest771-21snmpd09:57:550
188099975,2cyclictest771-21snmpd09:15:150
188099975,2cyclictest771-21snmpd09:03:320
188099975,2cyclictest771-21snmpd07:58:270
188099974,2cyclictest586-21rs:main0
188099974,2cyclictest255-20systemd-journal12:11:090
188099973,3cyclictest22209-21diskstats11:15:130
188189966,0cyclictest769-21lldpd08:44:212
188189966,0cyclictest769-21lldpd08:01:192
188189965,1cyclictest771-21snmpd10:07:552
188189965,1cyclictest769-21lldpd11:19:112
188189965,1cyclictest769-21lldpd10:03:082
188189965,1cyclictest769-21lldpd08:20:322
188189965,1cyclictest524-21avahi-daemon11:07:442
188189964,2cyclictest771-21snmpd09:46:152
188189964,2cyclictest771-21snmpd08:34:332
188189963,3cyclictest550-21NetworkManager12:39:052
188189963,2cyclictest586-21rs:main2
188189963,2cyclictest19914-21ssh11:12:052
188189963,2cyclictest19189-21ssh10:36:102
188189962,3cyclictest255-20systemd-journal11:20:382
188189962,3cyclictest21221-21ssh09:28:542
188189962,3cyclictest17805-21df_inode11:45:132
188189961,5cyclictest550-21NetworkManager10:57:202
188149966,0cyclictest771-21snmpd11:53:411
188149966,0cyclictest771-21snmpd11:45:171
188149966,0cyclictest771-21snmpd11:24:011
188149965,1cyclictest771-21snmpd09:55:251
188149965,1cyclictest769-21lldpd11:55:011
188149965,1cyclictest255-20systemd-journal10:36:071
188149965,1cyclictest22224-21fschecks_count11:15:141
188149965,1cyclictest13212-21gltestperf11:40:141
188149963,2cyclictest7011-21ssh09:12:181
188149963,2cyclictest255-20systemd-journal12:25:511
188099966,0cyclictest771-21snmpd11:45:050
188099966,0cyclictest771-21snmpd11:06:470
188099966,0cyclictest771-21snmpd08:27:400
188099966,0cyclictest769-21lldpd12:25:000
188099965,1cyclictest771-21snmpd11:20:450
188099965,1cyclictest771-21snmpd09:10:530
188099964,2cyclictest771-21snmpd11:02:420
188099964,2cyclictest771-21snmpd07:30:250
188099963,2cyclictest255-20systemd-journal10:46:510
188099962,3cyclictest32106-21grep12:00:130
188099962,3cyclictest17951-21grep10:00:180
188099962,3cyclictest12931-21sed08:10:220
188189955,0cyclictest771-21snmpd12:03:292
188189955,0cyclictest524-21avahi-daemon07:11:022
188189954,1cyclictest421ktimers/209:59:072
188189954,1cyclictest31852-21fschecks_count10:50:142
188189954,1cyclictest11817-21cpuspeed_turbos08:10:132
188189953,2cyclictest771-21snmpd08:26:042
188189953,2cyclictest771-21snmpd07:18:252
188189952,3cyclictest771-21snmpd08:15:062
188189952,3cyclictest550-21NetworkManager09:22:542
188189952,3cyclictest11102-21ssh10:28:102
188189952,2cyclictest9940-21ssh11:00:252
188189952,2cyclictest2421-21cat07:45:212
188189952,2cyclictest12858-21ssh11:40:112
188189952,2cyclictest10498-21ntpq12:10:192
188189951,3cyclictest6661-21ssh12:06:532
188149955,0cyclictest771-21snmpd12:23:091
188149954,1cyclictest0-21swapper/107:30:111
188149953,2cyclictest550-21NetworkManager12:30:071
188149952,2cyclictest3592-21df11:30:131
188149952,2cyclictest3277-21ssh10:54:411
188149952,2cyclictest27617-21grep10:45:141
188149950,2cyclictest386050/usr/sbin/munin10:55:141
188099955,0cyclictest771-21snmpd07:16:130
188099955,0cyclictest769-21lldpd08:06:560
188099954,1cyclictest524-21avahi-daemon07:40:250
188099954,1cyclictest25219-21grep12:25:130
188099953,2cyclictest771-21snmpd08:22:460
188099953,2cyclictest31264-21grep11:25:110
188099952,3cyclictest8159-21cat08:00:130
188099952,3cyclictest5410-21ssh10:55:300
188099952,2cyclictest255-20systemd-journal10:06:440
188099952,2cyclictest255-20systemd-journal08:19:200
188099952,2cyclictest23818-21awk07:20:160
188099952,2cyclictest21481-21grep08:30:210
188099952,2cyclictest1861-21sed07:45:160
188099952,2cyclictest17269-21ssh10:35:060
188099951,3cyclictest3370-21gpg-connect-age07:50:000
188099950,5cyclictest27135-21ntpq08:45:190
188099950,4cyclictest31619-21ntpq08:55:250
188189941,2cyclictest749-21ntpd08:35:162
188149942,2cyclictest3424-21cat09:45:161
188149942,2cyclictest10177-21cat08:05:141
188149941,2cyclictest5663-21cat07:55:141
188149941,2cyclictest1319-21grep07:45:121
188099943,1cyclictest25075-21apt07:25:110
188099942,2cyclictest19097-21grep07:10:130
188099941,2cyclictest28781-21grep08:50:100
188099941,2cyclictest23461-21grep08:35:210
188149930,3cyclictest29067-21taskset07:35:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional