You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-21 - 16:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack8slot6s.osadl.org (updated Thu Aug 21, 2025 12:43:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29175994545,0cyclictest4409-21fwupd07:25:262
29170991917,2cyclictest749-21ntpd11:49:581
29164991918,1cyclictest66850irq/125-lan12:08:190
29164991918,1cyclictest66850irq/125-lan11:37:450
29164991918,1cyclictest66850irq/125-lan11:15:190
29164991918,1cyclictest66850irq/125-lan10:27:390
29164991818,0cyclictest66850irq/125-lan12:30:180
29164991818,0cyclictest66850irq/125-lan12:25:200
29164991818,0cyclictest66850irq/125-lan10:50:160
29164991818,0cyclictest66850irq/125-lan10:18:180
29164991818,0cyclictest66850irq/125-lan09:15:210
29164991817,1cyclictest66850irq/125-lan12:38:480
29164991817,1cyclictest66850irq/125-lan11:40:010
29164991817,1cyclictest66850irq/125-lan11:21:450
29164991817,1cyclictest66850irq/125-lan11:06:500
29164991817,1cyclictest66850irq/125-lan10:48:190
29164991817,1cyclictest66850irq/125-lan10:35:470
29164991717,0cyclictest66850irq/125-lan12:20:180
29164991717,0cyclictest66850irq/125-lan12:11:180
29164991717,0cyclictest66850irq/125-lan11:47:190
29164991717,0cyclictest66850irq/125-lan11:27:430
29164991717,0cyclictest66850irq/125-lan10:40:180
29164991717,0cyclictest66850irq/125-lan10:30:180
29164991717,0cyclictest66850irq/125-lan10:20:150
29164991717,0cyclictest66850irq/125-lan10:00:200
29164991717,0cyclictest66850irq/125-lan09:52:160
29164991717,0cyclictest66850irq/125-lan09:30:210
29164991717,0cyclictest66850irq/125-lan09:25:130
29164991717,0cyclictest66850irq/125-lan07:40:140
29164991717,0cyclictest66850irq/125-lan07:20:130
29164991716,1cyclictest66850irq/125-lan11:00:490
29164991716,1cyclictest66850irq/125-lan10:55:260
29164991716,1cyclictest66850irq/125-lan10:05:540
29164991716,1cyclictest66850irq/125-lan09:12:280
29164991715,2cyclictest20045-21kworker/0:3+events11:58:430
29164991616,0cyclictest66850irq/125-lan11:50:130
29164991616,0cyclictest66850irq/125-lan11:30:210
29164991616,0cyclictest66850irq/125-lan10:12:110
29164991616,0cyclictest66850irq/125-lan09:00:210
29164991616,0cyclictest66850irq/125-lan08:00:200
29164991616,0cyclictest66850irq/125-lan07:45:180
29164991616,0cyclictest66850irq/125-lan07:32:510
29164991616,0cyclictest66850irq/125-lan07:15:160
29164991615,1cyclictest66850irq/125-lan12:15:130
29164991615,1cyclictest66850irq/125-lan12:02:110
29164991615,1cyclictest66850irq/125-lan11:11:040
29164991615,1cyclictest66850irq/125-lan09:21:140
29164991615,1cyclictest66850irq/125-lan09:05:120
29164991615,1cyclictest66850irq/125-lan08:55:140
29164991615,1cyclictest66850irq/125-lan08:25:190
29164991615,1cyclictest66850irq/125-lan07:10:140
29164991515,0cyclictest771-21snmpd08:50:120
29164991515,0cyclictest66850irq/125-lan09:45:160
29164991515,0cyclictest66850irq/125-lan09:43:030
29164991515,0cyclictest66850irq/125-lan09:38:120
29164991515,0cyclictest66850irq/125-lan08:40:230
29164991515,0cyclictest66850irq/125-lan08:23:200
29164991515,0cyclictest66850irq/125-lan08:15:220
29164991514,1cyclictest66850irq/125-lan09:56:420
29164991514,1cyclictest66850irq/125-lan07:28:580
29164991414,0cyclictest66850irq/125-lan08:36:210
29164991414,0cyclictest66850irq/125-lan08:05:200
29164991414,0cyclictest66850irq/125-lan07:50:210
29164991414,0cyclictest66850irq/125-lan07:35:150
2917999134,7cyclictest255-20systemd-journal09:26:133
2917999130,9cyclictest30219-21/usr/sbin/munin10:45:113
29164991313,0cyclictest771-21snmpd07:57:040
29164991312,1cyclictest66850irq/125-lan08:10:150
2917999123,7cyclictest255-20systemd-journal12:38:233
29179991210,2cyclictest771-21snmpd12:15:223
2917999120,10cyclictest2891-21ssh12:00:473
29164991212,0cyclictest66850irq/125-lan08:30:140
2917999112,5cyclictest3252-21grep07:25:143
2917999110,9cyclictest16172-21ssh09:12:223
2917999110,9cyclictest13245-21diskmemload12:12:443
2917999110,8cyclictest13940-21cut11:04:593
2917999110,8cyclictest10987-21chrt07:40:423
29164991110,1cyclictest66850irq/125-lan08:45:150
2917999102,6cyclictest30722-21sh11:24:033
2917999102,5cyclictest586-21rs:main3
2917999101,5cyclictest19562-21kworker/u16:2+xprtiod12:28:193
2917999100,9cyclictest19414-21turbostat08:09:553
2917999100,8cyclictest31454-21chrt10:08:253
2917999100,8cyclictest29200-21/usr/sbin/munin11:55:193
2917999100,8cyclictest2552-21ssh11:29:113
2917999100,8cyclictest24683-21ssh11:16:143
2917999100,8cyclictest23250-21ssh09:58:353
2917999100,8cyclictest13245-21diskmemload11:30:393
2917999100,7cyclictest24033-21timerandwakeup11:50:233
2917999100,7cyclictest21966-21ssh11:14:123
2917999100,7cyclictest16273-21ssh11:43:043
2917999100,7cyclictest15960-21ssh10:28:333
2917999100,7cyclictest15192-21grep11:05:233
2917999100,7cyclictest13245-21diskmemload10:15:223
2917999100,6cyclictest9880-21taskset11:35:403
2917999100,6cyclictest8166-21sh10:57:313
2917999100,6cyclictest6502-21chrt07:31:173
2917999100,6cyclictest27205-21ssh12:24:293
2917999100,6cyclictest20628-21ssh10:33:183
2917999100,6cyclictest17344-21chrt07:59:513
2917999100,6cyclictest13245-21diskmemload12:06:563
2917999100,6cyclictest13245-21diskmemload11:44:593
2917999100,6cyclictest11568-21chrt07:45:063
2917599109,1cyclictest20044-21cupsd09:35:252
291799991,6cyclictest2123-21kworker/u16:1-nfsiod10:11:363
291799991,5cyclictest2292-21cut08:40:203
291799990,8cyclictest9181-21chrt07:38:423
291799990,8cyclictest19352-21chrt08:04:313
291799990,7cyclictest9336-21sh09:41:363
291799990,7cyclictest7417-21chrt08:55:033
291799990,7cyclictest4902-21taskset08:46:373
291799990,7cyclictest26822-21sed08:20:203
291799990,7cyclictest25335-21taskset08:18:463
291799990,7cyclictest24272-21sh09:20:573
291799990,7cyclictest13245-21diskmemload10:50:393
291799990,6cyclictest775-21runrttasks08:35:003
291799990,6cyclictest588-21seq07:18:493
291799990,6cyclictest4944-21ssh09:37:003
291799990,6cyclictest28747-21chrt10:42:473
291799990,6cyclictest11890-21seq10:23:593
291799990,5cyclictest29295-21chrt08:27:373
291799990,5cyclictest25341-21smartctl10:00:223
291799990,5cyclictest23937-21chrt10:36:553
291799990,5cyclictest23293-21taskset08:13:273
291799990,5cyclictest2143-21sed07:20:203
291799990,5cyclictest17918-21ssh09:15:043
291799990,5cyclictest13245-21diskmemload12:30:123
291799990,5cyclictest13245-21diskmemload09:30:503
291799990,5cyclictest11769-21fschecks_count09:45:133
291799990,5cyclictest10968-21chrt09:01:473
291759998,1cyclictest771-21snmpd12:30:502
291759998,1cyclictest771-21snmpd09:06:012
291759998,1cyclictest771-21snmpd08:06:382
291709998,1cyclictest771-21snmpd11:50:001
291709998,1cyclictest771-21snmpd09:52:321
291709998,1cyclictest771-21snmpd09:44:091
291799986,2cyclictest749-21ntpd09:07:233
291799980,5cyclictest5410-21chrt08:50:063
291799980,5cyclictest30477-21ld08:30:173
291799980,5cyclictest15063-21seq07:51:233
291759988,0cyclictest771-21snmpd10:10:212
291759988,0cyclictest771-21snmpd10:00:572
291759988,0cyclictest771-21snmpd08:59:012
291759988,0cyclictest771-21snmpd07:15:392
291759988,0cyclictest771-21snmpd07:11:122
291759988,0cyclictest769-21lldpd12:27:192
291759988,0cyclictest769-21lldpd12:17:222
291759987,1cyclictest771-21snmpd11:48:122
291759987,1cyclictest771-21snmpd07:21:542
291759987,1cyclictest517-21systemd-resolve09:33:292
291759986,2cyclictest771-21snmpd11:21:292
291759986,2cyclictest771-21snmpd08:53:512
291709988,0cyclictest771-21snmpd12:18:521
291709988,0cyclictest771-21snmpd10:48:341
291709988,0cyclictest771-21snmpd09:15:011
291709988,0cyclictest771-21snmpd09:13:381
291709988,0cyclictest771-21snmpd07:37:191
291709988,0cyclictest769-21lldpd10:58:081
291709988,0cyclictest769-21lldpd09:22:061
291709988,0cyclictest769-21lldpd07:59:121
291709988,0cyclictest769-21lldpd07:11:391
291709987,1cyclictest771-21snmpd10:51:341
291709987,1cyclictest524-21avahi-daemon10:32:301
291709986,2cyclictest771-21snmpd08:15:231
291709986,2cyclictest771-21snmpd07:52:101
291799970,4cyclictest30372-21awk07:10:193
291759977,0cyclictest771-21snmpd10:18:152
291759977,0cyclictest769-21lldpd12:37:072
291759976,1cyclictest771-21snmpd11:11:542
291759976,1cyclictest771-21snmpd09:01:222
291759976,1cyclictest771-21snmpd07:31:512
291759976,1cyclictest769-21lldpd11:38:062
291759976,1cyclictest769-21lldpd08:31:332
291759976,1cyclictest584-21in:imuxsock07:54:312
291759975,2cyclictest771-21snmpd10:37:052
291759975,2cyclictest771-21snmpd09:25:172
291759975,2cyclictest771-21snmpd08:35:452
291709977,0cyclictest771-21snmpd07:44:171
291709977,0cyclictest769-21lldpd11:27:131
291709977,0cyclictest769-21lldpd10:28:341
291709977,0cyclictest769-21lldpd09:45:281
291709977,0cyclictest0-21swapper/112:33:181
291709976,1cyclictest771-21snmpd08:10:241
291709976,1cyclictest524-21avahi-daemon11:21:581
291709976,1cyclictest524-21avahi-daemon09:25:281
291709975,2cyclictest771-21snmpd12:08:431
291709975,2cyclictest771-21snmpd11:33:001
291709975,2cyclictest771-21snmpd10:17:001
291709975,2cyclictest3059-21cat09:35:151
291709974,3cyclictest550-21NetworkManager12:29:551
291759966,0cyclictest771-21snmpd12:04:342
291759966,0cyclictest771-21snmpd10:41:262
291759966,0cyclictest771-21snmpd07:41:142
291759966,0cyclictest769-21lldpd11:05:352
291759966,0cyclictest769-21lldpd10:52:592
291759966,0cyclictest769-21lldpd09:50:482
291759966,0cyclictest769-21lldpd08:26:162
291759966,0cyclictest769-21lldpd08:01:232
291759966,0cyclictest524-21avahi-daemon09:56:182
291759965,1cyclictest771-21snmpd09:44:052
291759965,1cyclictest769-21lldpd09:11:552
291759965,1cyclictest769-21lldpd08:18:552
291759965,1cyclictest524-21avahi-daemon11:53:052
291759964,2cyclictest771-21snmpd10:33:252
291759964,2cyclictest771-21snmpd08:21:232
291759963,3cyclictest550-21NetworkManager11:55:422
291759962,3cyclictest495-21sed11:25:252
291759962,3cyclictest255-20systemd-journal10:20:292
291759962,3cyclictest15292-21ssh11:41:332
291759962,3cyclictest12415-21memory12:10:152
291709966,0cyclictest771-21snmpd10:43:351
291709966,0cyclictest769-21lldpd10:06:481
291709966,0cyclictest769-21lldpd08:33:471
291709966,0cyclictest524-21avahi-daemon08:29:431
291709965,1cyclictest5719-21needreboot07:30:171
291709965,1cyclictest524-21avahi-daemon12:20:031
291709965,1cyclictest524-21avahi-daemon09:56:321
291709965,1cyclictest517-21systemd-resolve11:07:441
291709965,1cyclictest331ktimers/109:05:121
291709965,1cyclictest255-20systemd-journal11:02:381
291709965,1cyclictest12270-21cut07:45:171
291709964,2cyclictest771-21snmpd09:04:401
291709963,2cyclictest255-20systemd-journal12:04:071
291709962,3cyclictest945-21sed10:10:171
291709962,3cyclictest9425-21ntpq10:20:171
291709962,3cyclictest22133-21fw_forwarded_lo10:35:131
291759954,1cyclictest23192-21cut09:20:162
291759954,1cyclictest12249-21grep09:45:152
291759953,2cyclictest771-21snmpd10:49:342
291759953,2cyclictest4746-21sshd11:30:212
291759953,2cyclictest0-21swapper/208:10:152
291759952,2cyclictest8451-21pluginstate07:35:202
291759952,2cyclictest586-21rs:main2
291759952,2cyclictest586-21rs:main2
291759952,2cyclictest31831-21sh10:09:122
291759952,2cyclictest24396-21ssh11:15:412
291759952,2cyclictest23356-21netstat12:20:172
291759952,2cyclictest19523-21ssh09:15:282
291759952,2cyclictest14944-21ssh10:26:342
291759952,2cyclictest10547-21wc11:00:162
291759951,3cyclictest8413-21ssh12:06:012
291709955,0cyclictest771-21snmpd08:46:001
291709954,1cyclictest771-21snmpd08:04:461
291709954,1cyclictest769-21lldpd08:57:281
291709954,1cyclictest769-21lldpd08:05:111
291709954,1cyclictest31448-21cpuspeed_turbos07:15:131
291709953,2cyclictest771-21snmpd11:40:401
291709953,2cyclictest749-21ntpd08:40:241
291709952,3cyclictest749-21ntpd08:50:121
291709952,3cyclictest550-21NetworkManager11:36:371
291709952,3cyclictest550-21NetworkManager10:00:071
291709952,3cyclictest32037-21perl08:35:131
291709952,3cyclictest13090-21expr12:10:211
291709952,2cyclictest8789-21ssh12:36:071
291709952,2cyclictest586-21rs:main1
291709952,2cyclictest31636-21ssh11:58:061
291709951,3cyclictest31614-21ntp_states09:30:171
291759942,2cyclictest4194-21ld08:45:172
291759942,2cyclictest1889-21cut08:40:172
291759941,2cyclictest12055-21interrupts07:45:162
291709942,2cyclictest18764-21expr11:10:151
291709942,2cyclictest1446-21cat07:20:141
291709940,3cyclictest25992-21perl08:20:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional