You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-30 - 11:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Sep 30, 2023 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991060,8rtkit-daemon1622-21rtkit-daemon00:21:451
1623991020,9rtkit-daemon1622-21rtkit-daemon19:33:131
1623991020,9rtkit-daemon1622-21rtkit-daemon19:33:131
1623991000,9rtkit-daemon1622-21rtkit-daemon19:56:171
1623991000,9rtkit-daemon1622-21rtkit-daemon19:56:171
61321599963,7cyclictest0-21swapper/300:31:593
162399960,8rtkit-daemon1622-21rtkit-daemon21:14:501
61321299953,88cyclictest0-21swapper/220:30:592
61321599923,85cyclictest0-21swapper/321:45:593
61200029165,17sleep00-21swapper/019:05:240
61321299902,8cyclictest0-21swapper/222:31:592
61275928867,14sleep30-21swapper/319:06:043
613205998779,5cyclictest0-21swapper/000:31:590
61274328760,17sleep20-21swapper/219:05:502
162399870,9rtkit-daemon1622-21rtkit-daemon20:48:211
61303828363,14sleep10-21swapper/119:09:351
6188632800,4sleep20-21swapper/219:25:002
613205997971,5cyclictest0-21swapper/021:45:590
7129402760,7sleep0708125-21kworker/0:023:50:420
61320599732,6cyclictest0-21swapper/022:30:260
613215997114,50cyclictest663461-21entropy21:30:263
162399710,9rtkit-daemon1622-21rtkit-daemon19:45:170
162399710,8rtkit-daemon1622-21rtkit-daemon23:02:401
162399700,7rtkit-daemon1622-21rtkit-daemon20:34:411
61320599660,61cyclictest1622-21rtkit-daemon22:24:400
61320999643,53cyclictest1753-21gnome-shell19:15:261
61320599640,60cyclictest1622-21rtkit-daemon22:47:320
6316262630,5sleep1251rcuc/120:00:211
61320599632,16cyclictest1753-21gnome-shell00:16:590
613205996245,12cyclictest13-21ksoftirqd/023:45:260
613205996245,12cyclictest13-21ksoftirqd/023:45:250
162399620,9rtkit-daemon1622-21rtkit-daemon23:43:510
61321599614,44cyclictest0-21swapper/323:45:263
61321599614,44cyclictest0-21swapper/323:45:263
61320599612,53cyclictest1753-21gnome-shell20:30:260
162399610,9rtkit-daemon1622-21rtkit-daemon23:06:270
162399610,9rtkit-daemon1622-21rtkit-daemon22:00:170
61321299603,49cyclictest1753-21gnome-shell23:45:262
61321299603,49cyclictest1753-21gnome-shell23:45:262
61321299602,51cyclictest1753-21gnome-shell23:00:262
162399600,9rtkit-daemon1622-21rtkit-daemon00:05:310
61321599594,13cyclictest880726-21firefox00:15:583
613212995946,8cyclictest880726-21firefox21:45:262
61321299591,16cyclictest1753-21gnome-shell22:27:592
61320999593,50cyclictest881331-21Privileged14:31:441
61320599593,14cyclictest1753-21gnome-shell20:15:260
61320599593,12cyclictest881335-21WRScene~ilder#119:15:260
61320599590,54cyclictest1622-21rtkit-daemon19:24:530
162399590,9rtkit-daemon1622-21rtkit-daemon19:54:130
162399590,8rtkit-daemon1622-21rtkit-daemon20:22:440
162399590,8rtkit-daemon1622-21rtkit-daemon20:22:440
613212995823,29cyclictest677335-21df22:10:242
61320999582,5cyclictest0-21swapper/122:31:591
61320999581,51cyclictest881253-21Renderer22:00:261
613205995818,35cyclictest673382-21grep22:00:020
61321599573,12cyclictest880726-21firefox23:00:263
61321299576,47cyclictest880898-21Xwayland20:15:262
61321299571,52cyclictest855-21dbus-daemon23:45:012
61320599572,50cyclictest631679-21hddtemp_smartct20:00:260
613205995721,31cyclictest713760-21cut23:55:210
613205995715,36cyclictest881251-21Softwar~cThread20:30:020
162399570,8rtkit-daemon1622-21rtkit-daemon21:54:430
61321599561,50cyclictest1753-21gnome-shell20:37:593
61321299562,49cyclictest880898-21Xwayland22:45:252
61320999563,48cyclictest0-21swapper/122:45:261
61320999562,51cyclictest0-21swapper/100:16:591
61320999560,51cyclictest1622-21rtkit-daemon22:40:461
61320599561,50cyclictest1753-21gnome-shell20:08:580
162399560,9rtkit-daemon1622-21rtkit-daemon23:12:310
162399560,8rtkit-daemon1622-21rtkit-daemon23:50:461
162399560,8rtkit-daemon1622-21rtkit-daemon00:12:061
162399560,8rtkit-daemon1622-21rtkit-daemon00:12:061
61321599553,10cyclictest881251-21Softwar~cThread22:08:593
613215995522,11cyclictest646760-21sh20:45:003
61321599552,14cyclictest669806-21kworker/3:222:04:593
613212995538,12cyclictest1753-21gnome-shell21:06:592
613205995517,32cyclictest881251-21Softwar~cThread21:25:320
613205995516,34cyclictest624506-21tr19:40:190
162399550,8rtkit-daemon1622-21rtkit-daemon00:01:230
6948472540,18sleep013-21ksoftirqd/023:00:220
61321599542,10cyclictest881253-21Renderer22:45:263
61321299543,44cyclictest631679-21hddtemp_smartct20:00:272
61321299542,47cyclictest880726-21firefox19:32:582
61321299542,47cyclictest880726-21firefox19:32:582
61321299542,47cyclictest1753-21gnome-shell21:34:592
61321299542,11cyclictest1753-21gnome-shell21:53:592
61321299542,10cyclictest880726-21firefox22:56:592
61321299541,50cyclictest572337-21kworker/2:019:54:582
61321299541,42cyclictest1753-21gnome-shell20:51:592
61320999548,40cyclictest1753-21gnome-shell23:22:591
61320999548,40cyclictest1753-21gnome-shell23:22:581
61320599542,47cyclictest685761-21nm-dispatcher22:35:120
61320599542,10cyclictest1753-21gnome-shell19:25:580
61321599534,44cyclictest0-21swapper/319:41:583
61321599532,7cyclictest0-21swapper/323:50:593
61321599532,46cyclictest1753-21gnome-shell22:55:593
61321599532,46cyclictest1753-21gnome-shell21:15:583
61321299532,46cyclictest1753-21gnome-shell00:36:592
61321299532,45cyclictest616026-21diskstats19:15:272
613212995319,11cyclictest721159-21expr00:15:242
61320999534,46cyclictest231irq_work/122:25:591
613209995344,6cyclictest0-21swapper/100:31:581
61320999534,19cyclictest633285-21expr20:05:191
61320999531,11cyclictest1753-21gnome-shell23:15:591
61320999530,49cyclictest1622-21rtkit-daemon22:18:031
61320599539,11cyclictest881251-21Softwar~cThread20:10:340
613205995313,34cyclictest677571-21grep22:10:330
61321599522,47cyclictest669806-21kworker/3:222:54:593
613212995239,8cyclictest880726-21firefox19:41:582
61321299522,46cyclictest1753-21gnome-shell21:43:592
61321299522,23cyclictest438-20systemd-journal19:14:552
61321299522,10cyclictest1753-21gnome-shell23:51:592
613212995218,11cyclictest715754-21expr00:00:212
613212995218,10cyclictest713768-21seq23:55:212
613212995217,13cyclictest881251-21Softwar~cThread21:18:592
613212995216,12cyclictest881251-21Softwar~cThread22:05:262
613212995216,10cyclictest653149-21ls21:00:382
61320999522,46cyclictest1753-21gnome-shell00:07:581
61320599523,45cyclictest0-21swapper/021:30:590
162399520,6rtkit-daemon1622-21rtkit-daemon00:26:081
61321599512,45cyclictest1753-21gnome-shell21:02:593
61321299512,45cyclictest1753-21gnome-shell22:40:592
61321299512,44cyclictest1753-21gnome-shell22:51:592
613212995117,10cyclictest706696-21cat23:35:202
613212995116,11cyclictest720039-21who00:10:402
613212995116,11cyclictest720039-21who00:10:402
61321299511,47cyclictest572337-21kworker/2:019:47:582
61320999514,7cyclictest231irq_work/122:58:591
61320999513,45cyclictest231irq_work/121:27:591
61320999513,45cyclictest231irq_work/120:27:591
61320999512,45cyclictest231irq_work/120:17:591
61320999512,45cyclictest1753-21gnome-shell23:26:591
61320999512,45cyclictest1753-21gnome-shell19:27:581
61320599514,6cyclictest613199-21cyclictest22:27:590
61320599512,8cyclictest881251-21Softwar~cThread00:28:390
61320599512,8cyclictest680201-21munin-run22:20:010
61320599512,45cyclictest0-21swapper/021:16:590
613205995110,18cyclictest141rcu_preempt00:23:100
61321599502,7cyclictest0-21swapper/321:25:593
61321599501,46cyclictest42-21ksoftirqd/321:09:593
613212995037,9cyclictest1753-21gnome-shell22:18:592
61321299502,7cyclictest0-21swapper/223:09:592
613212995017,9cyclictest620732-21date19:30:022
613212995016,9cyclictest706505-21wc23:35:022
613212995015,11cyclictest881251-21Softwar~cThread23:20:332
613212995015,11cyclictest881251-21Softwar~cThread23:20:332
613212995014,12cyclictest659444-21ls21:20:192
613212995012,11cyclictest643798-21cat20:35:212
61320999502,7cyclictest231irq_work/122:39:591
61320999502,45cyclictest231irq_work/121:50:591
61320999502,45cyclictest231irq_work/120:57:591
61320999502,39cyclictest231irq_work/121:58:591
61320999501,46cyclictest231irq_work/122:12:591
61320999501,45cyclictest231irq_work/122:20:591
61320999500,46cyclictest1622-21rtkit-daemon23:57:521
613205995031,14cyclictest1753-21gnome-shell23:25:590
61320599502,45cyclictest0-21swapper/022:43:590
613205995016,30cyclictest675757-21grep22:05:280
613205995014,32cyclictest720441-21systemd-run00:15:010
613205995014,32cyclictest720441-21systemd-run00:15:000
61320599500,45cyclictest1622-21rtkit-daemon20:48:210
613215994937,8cyclictest1753-21gnome-shell21:13:593
61321599492,7cyclictest0-21swapper/322:35:593
61321599492,7cyclictest0-21swapper/322:11:583
61321599492,7cyclictest0-21swapper/321:55:593
61321599492,7cyclictest0-21swapper/321:35:593
61321599492,7cyclictest0-21swapper/320:18:593
61321599492,6cyclictest0-21swapper/322:19:593
61321599492,44cyclictest669806-21kworker/3:222:32:593
61321599492,44cyclictest0-21swapper/321:54:593
61321599492,44cyclictest0-21swapper/320:54:593
61321599492,39cyclictest0-21swapper/319:59:583
61321599492,39cyclictest0-21swapper/319:59:583
613215994921,11cyclictest42-21ksoftirqd/319:15:583
613212994938,8cyclictest572337-21kworker/2:020:13:592
61321299492,7cyclictest0-21swapper/221:58:592
61321299492,7cyclictest0-21swapper/221:28:592
61321299492,6cyclictest0-21swapper/221:10:592
61321299492,44cyclictest0-21swapper/223:12:592
613212994918,12cyclictest630195-21needreboot19:55:312
613212994918,12cyclictest630195-21needreboot19:55:312
613212994916,10cyclictest681079-21cat22:20:312
613209994941,6cyclictest231irq_work/100:01:591
61320999493,7cyclictest231irq_work/121:32:581
61320999492,45cyclictest231irq_work/121:02:591
61320999492,45cyclictest231irq_work/120:35:591
61320999492,45cyclictest231irq_work/120:23:591
61320999492,45cyclictest231irq_work/120:23:591
61320999492,44cyclictest231irq_work/122:09:591
61320999492,44cyclictest231irq_work/121:16:591
61320999492,44cyclictest231irq_work/120:50:591
61320599499,35cyclictest661042-21latency_hist21:25:020
61320599494,40cyclictest700083-21Indexed~#11150923:15:380
61320599492,44cyclictest0-21swapper/023:35:590
61320599492,43cyclictest0-21swapper/022:56:590
613205994918,26cyclictest646806-21wc20:45:020
613205994910,33cyclictest652251-21cpu21:00:210
7059302480,5sleep00-21swapper/023:30:380
61321599482,7cyclictest0-21swapper/323:56:593
61321599482,7cyclictest0-21swapper/323:11:593
61321599482,7cyclictest0-21swapper/322:41:593
61321599482,7cyclictest0-21swapper/321:20:593
61321599482,7cyclictest0-21swapper/320:32:593
61321599482,7cyclictest0-21swapper/300:39:593
61321599482,6cyclictest0-21swapper/323:35:593
61321599482,6cyclictest0-21swapper/320:10:593
61321599482,6cyclictest0-21swapper/319:46:583
61321599482,6cyclictest0-21swapper/319:35:583
61321599482,6cyclictest0-21swapper/319:23:583
61321599482,6cyclictest0-21swapper/300:02:593
61321599482,43cyclictest0-21swapper/320:48:593
61321599482,37cyclictest0-21swapper/323:28:583
61321299483,18cyclictest665001-21tr21:35:192
61321299482,7cyclictest0-21swapper/220:45:592
61321299482,43cyclictest0-21swapper/220:09:592
61321299482,42cyclictest0-21swapper/223:25:592
61321299482,42cyclictest0-21swapper/220:23:592
61321299482,42cyclictest0-21swapper/220:23:592
61321299482,41cyclictest0-21swapper/200:30:262
613212994817,9cyclictest722961-21fw_packets00:20:272
613212994811,11cyclictest673786-21cat22:00:222
613209994840,6cyclictest231irq_work/120:11:591
61320999483,43cyclictest231irq_work/121:24:591
61320599482,6cyclictest0-21swapper/019:38:580
61320599482,5cyclictest0-21swapper/022:53:590
613205994817,26cyclictest728277-21cut00:35:280
613215994730,13cyclictest1753-21gnome-shell21:42:593
61321599472,6cyclictest0-21swapper/320:58:593
61321599472,5cyclictest0-21swapper/323:32:593
61321599472,5cyclictest0-21swapper/300:11:593
61321599472,5cyclictest0-21swapper/300:11:583
61321599472,42cyclictest0-21swapper/323:44:593
61321599472,41cyclictest0-21swapper/319:53:583
61321299472,6cyclictest0-21swapper/223:16:592
613212994716,12cyclictest881251-21Softwar~cThread20:29:142
613212994714,10cyclictest686155-21expr22:35:242
61320999472,43cyclictest231irq_work/123:48:591
61320999472,43cyclictest231irq_work/123:48:591
61320999472,43cyclictest231irq_work/123:33:591
61320999472,43cyclictest231irq_work/121:06:591
61320999472,43cyclictest231irq_work/100:38:591
61320999471,29cyclictest691208-21cat22:50:181
61320599472,42cyclictest0-21swapper/021:41:590
61320599472,42cyclictest0-21swapper/021:08:590
613205994715,27cyclictest665248-21idleruntime21:35:270
613205994712,30cyclictest657531-21cat21:15:020
613215994619,6cyclictest42-21ksoftirqd/323:17:003
61321299462,41cyclictest0-21swapper/220:40:592
613212994612,12cyclictest881251-21Softwar~cThread19:36:452
613209994638,6cyclictest231irq_work/119:22:581
613209994638,6cyclictest231irq_work/119:11:581
613209994637,6cyclictest231irq_work/123:43:581
61320999462,42cyclictest0-21swapper/123:38:591
613205994638,5cyclictest0-21swapper/019:30:580
613205994638,5cyclictest0-21swapper/019:30:580
6399392450,21sleep342-21ksoftirqd/320:25:023
6399392450,21sleep342-21ksoftirqd/320:25:023
61321599452,41cyclictest0-21swapper/320:00:583
61321599452,41cyclictest0-21swapper/319:28:583
613212994512,12cyclictest724693-21smartctl00:25:272
613209994537,6cyclictest231irq_work/119:54:531
613209994510,15cyclictest667031-21interrupts21:40:271
61320599459,30cyclictest651273-21awk20:55:350
61320599452,9cyclictest881331-21Privileged14:31:440
613205994514,27cyclictest630591-21timerwakeupswit19:55:390
613205994514,27cyclictest630591-21timerwakeupswit19:55:390
613205994512,28cyclictest438-20systemd-journal23:22:290
613205994512,28cyclictest438-20systemd-journal23:22:290
613209994436,6cyclictest231irq_work/123:07:591
7174612430,5sleep30-21swapper/300:05:223
613212994313,11cyclictest881251-21Softwar~cThread20:59:192
613212994313,11cyclictest881251-21Softwar~cThread00:05:582
61320999428,11cyclictest664942-21cpuspeed_turbos21:35:191
613205994225,14cyclictest0-21swapper/020:39:390
613215994034,4cyclictest0-21swapper/319:34:593
613215994034,4cyclictest0-21swapper/319:34:583
613215994023,12cyclictest438-20systemd-journal23:09:593
613215994013,22cyclictest881343-21Timer20:05:253
61321599401,13cyclictest0-21swapper/320:25:203
61320999407,9cyclictest0-21swapper/119:35:351
613209994034,4cyclictest0-21swapper/123:12:591
61320999401,15cyclictest880726-21firefox19:43:581
61320599402,13cyclictest648950-21tr20:50:220
6831912390,5sleep3411rcuc/322:25:373
613215993817,16cyclictest42-21ksoftirqd/300:28:343
61320999385,11cyclictest626924-21ps19:45:371
613215993724,8cyclictest880726-21firefox00:20:263
162399370,25rtkit-daemon0-21swapper/323:21:263
162399370,25rtkit-daemon0-21swapper/323:21:263
613215993518,13cyclictest613698-21cat19:10:233
61321599341,9cyclictest0-21swapper/322:20:303
613209993416,14cyclictest645949-21cat20:40:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional