You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 21:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Thu Sep 18, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1373099251,23cyclictest0-21swapper/110:35:021
1373999240,23cyclictest0-21swapper/307:21:323
1373499240,23cyclictest7543-21dump-pmu-power09:09:592
13730992421,2cyclictest0-21swapper/108:05:011
1373099240,2cyclictest313-21systemd-journal09:50:001
1372899240,23cyclictest313-21systemd-journal09:40:000
1372899240,1cyclictest28064-21apt-get10:56:320
13739992321,2cyclictest34-21ksoftirqd/308:34:593
13739992321,1cyclictest10903-21kworker/u8:010:10:003
1373999231,21cyclictest593-21dbus-daemon09:19:593
1373999230,22cyclictest20734-21latency_hist08:29:593
1373999230,22cyclictest18451-21apt-get10:36:283
13734992321,1cyclictest18306-21apt-get10:35:022
1373499231,21cyclictest271rcuc/208:10:002
1373499231,1cyclictest28-21ksoftirqd/207:15:012
1373499230,22cyclictest313-21systemd-journal10:55:002
1373499230,22cyclictest14245-21idleruntime-cro12:40:002
1373499230,1cyclictest2709-21cron09:00:012
13728992323,0cyclictest13200-21apt-get10:26:320
1372899230,22cyclictest29625-21dump-pmu-power09:55:000
1372899230,1cyclictest16329-21apt-get08:21:340
13739992221,1cyclictest0-21swapper/308:46:323
13739992221,1cyclictest0-21swapper/308:46:323
13739992220,1cyclictest593-21dbus-daemon07:34:593
13739992220,1cyclictest27925-21cron08:45:003
1373999220,21cyclictest419-21apt-get11:06:323
1373999220,21cyclictest32155-21apt-get12:11:323
1373999220,21cyclictest18348-21turbostat08:24:593
1373999220,21cyclictest1032-21apt-get07:51:353
1373999220,21cyclictest0-21swapper/309:21:313
1373999220,21cyclictest0-21swapper/309:00:013
1373999220,21cyclictest0-21swapper/308:01:313
13734992220,1cyclictest8230-21dpkg10:15:012
1373499220,21cyclictest593-21dbus-daemon11:50:002
1373499220,21cyclictest31383-21turbostat12:10:002
1373499220,21cyclictest2866-21apt-get11:11:322
1373499220,21cyclictest0-21swapper/209:21:312
1373499220,1cyclictest26611-21basename12:00:002
13730992222,0cyclictest0-21swapper/107:31:321
13730992220,1cyclictest0-21swapper/107:55:001
1373099220,21cyclictest30403-21apt-get09:56:341
1373099220,21cyclictest0-21swapper/112:16:331
1373099220,21cyclictest0-21swapper/110:21:321
1373099220,21cyclictest0-21swapper/109:15:011
1373099220,21cyclictest0-21swapper/108:51:271
1373099220,21cyclictest0-21swapper/108:26:321
1372899220,21cyclictest7354-21apt-get12:26:320
1372899220,21cyclictest679-21apt-get08:56:300
1372899220,21cyclictest5858-21apt-get08:01:300
1372899220,21cyclictest4885-21awk11:15:010
1372899220,21cyclictest18947-21apt-get07:21:320
1372899220,21cyclictest12475-21apt-get11:31:310
1372899220,21cyclictest0-21swapper/012:36:310
1372899220,1cyclictest2144-21apt-get12:16:330
1372899220,1cyclictest10801-21apt-get10:21:320
13739992120,1cyclictest20873-21apt-get10:41:323
13739992120,1cyclictest12841-21perf10:25:003
13739992120,1cyclictest0-21swapper/310:15:013
1373999210,21cyclictest22817-21apt-get09:41:353
1373999210,21cyclictest0-21swapper/309:36:303
1373999210,20cyclictest7927-21apt-get09:11:333
1373999210,0cyclictest0-21swapper/308:36:313
13734992121,0cyclictest0-21swapper/210:56:312
13734992121,0cyclictest0-21swapper/208:31:322
13734992119,1cyclictest0-21swapper/210:35:012
13734992119,1cyclictest0-21swapper/209:45:012
1373499210,20cyclictest9755-21apt-get12:31:352
1373499210,20cyclictest0-21swapper/209:56:352
1373499210,20cyclictest0-21swapper/208:41:312
1373099213,1cyclictest24151-21cron11:55:001
13730992121,0cyclictest8381-21apt-get10:16:321
13730992121,0cyclictest0-21swapper/112:11:321
13730992121,0cyclictest0-21swapper/111:36:331
13730992121,0cyclictest0-21swapper/107:16:311
13730992119,1cyclictest0-21swapper/108:11:301
13730992113,0cyclictest0-21swapper/108:45:511
13730992113,0cyclictest0-21swapper/108:45:511
1373099210,21cyclictest0-21swapper/110:56:311
1373099210,17cyclictest0-21swapper/109:41:341
1373099210,0cyclictest0-21swapper/110:11:281
1373099210,0cyclictest0-21swapper/109:21:321
13728992121,0cyclictest30482-21apt-get11:01:320
13728992121,0cyclictest0-21swapper/009:55:020
13728992120,1cyclictest18001-21apt-get09:31:320
1372899210,20cyclictest26133-21apt-get07:36:310
1372899210,0cyclictest3131-21apt-get09:01:310
1372899210,0cyclictest19756-21apt-get11:46:320
13739992020,0cyclictest0-21swapper/310:56:323
13739992020,0cyclictest0-21swapper/310:01:333
13739992020,0cyclictest0-21swapper/308:11:313
13739992017,2cyclictest14513-21dump-pmu-power11:34:593
1373999200,20cyclictest0-21swapper/312:21:333
1373999200,0cyclictest5529-21apt-get09:06:323
1373999200,0cyclictest0-21swapper/312:31:343
1373999200,0cyclictest0-21swapper/311:46:333
1373999200,0cyclictest0-21swapper/311:01:333
1373999200,0cyclictest0-21swapper/310:46:363
1373999200,0cyclictest0-21swapper/309:46:323
1373999200,0cyclictest0-21swapper/307:36:303
13734992020,0cyclictest0-21swapper/211:31:322
13734992020,0cyclictest0-21swapper/211:06:322
13734992020,0cyclictest0-21swapper/209:36:322
13734992020,0cyclictest0-21swapper/207:41:302
13734992019,0cyclictest0-21swapper/210:11:282
13734992018,2cyclictest1752-21sh12:15:002
13734992018,1cyclictest313-21systemd-journal09:19:592
13734992018,1cyclictest0-21swapper/208:26:312
1373499200,20cyclictest0-21swapper/211:51:362
1373499200,20cyclictest0-21swapper/211:26:302
1373499200,0cyclictest15543-21apt-get09:26:292
13730992020,0cyclictest0-21swapper/112:21:341
13730992020,0cyclictest0-21swapper/111:41:331
13730992020,0cyclictest0-21swapper/110:46:351
13730992020,0cyclictest0-21swapper/107:41:301
13730992019,1cyclictest25569-21perf08:40:001
13730992018,1cyclictest22457-21lockfile-remove09:40:011
13730992018,1cyclictest0-21swapper/109:11:321
1373099200,20cyclictest0-21swapper/108:56:301
1373099200,19cyclictest0-21swapper/108:41:311
1373099200,0cyclictest0-21swapper/111:21:361
13728992020,0cyclictest0-21swapper/011:51:350
13728992020,0cyclictest0-21swapper/011:44:590
13728992019,1cyclictest23740-21apt-get07:31:330
13728992019,1cyclictest21338-21apt-get07:26:300
13728992019,1cyclictest13139-21apt-get09:21:320
13728992019,1cyclictest11547-21apt-get12:32:160
13728992014,3cyclictest0-21swapper/010:10:010
1372899200,20cyclictest0-21swapper/010:16:320
1372899200,20cyclictest0-21swapper/008:06:300
1372899200,19cyclictest16528-21apt-get07:16:310
13739991919,0cyclictest17357-21apt-get11:41:323
13739991919,0cyclictest0-21swapper/311:56:323
13739991919,0cyclictest0-21swapper/309:26:283
13739991918,1cyclictest12088-21awk11:29:593
13739991917,1cyclictest593-21dbus-daemon11:25:003
1373999190,1cyclictest31372-21awk12:10:013
1373999190,1cyclictest14264-21sendmail-msp12:39:593
13734991919,0cyclictest25210-21apt-get09:46:322
13734991919,0cyclictest0-21swapper/212:26:322
13734991919,0cyclictest0-21swapper/212:21:332
13734991919,0cyclictest0-21swapper/212:16:322
13734991919,0cyclictest0-21swapper/211:21:352
13734991919,0cyclictest0-21swapper/210:21:312
13734991919,0cyclictest0-21swapper/208:51:282
13734991917,1cyclictest3072-21awk07:55:012
1373499190,17cyclictest28-21ksoftirqd/211:17:112
1373499190,0cyclictest0-21swapper/212:01:302
1373499190,0cyclictest0-21swapper/207:36:312
1373499190,0cyclictest0-21swapper/207:21:312
13730991919,0cyclictest0-21swapper/110:26:321
13728991919,0cyclictest23516-21apt-get08:36:310
13728991919,0cyclictest23141-21perf08:35:000
13728991919,0cyclictest10075-21apt-get11:26:310
13728991919,0cyclictest0-21swapper/011:16:310
13728991919,0cyclictest0-21swapper/010:51:300
13728991919,0cyclictest0-21swapper/009:26:290
13728991919,0cyclictest0-21swapper/009:15:020
13728991919,0cyclictest0-21swapper/008:41:300
13728991918,1cyclictest4564-21apt-get12:21:340
13728991918,1cyclictest25210-21apt-get09:46:310
13728991917,1cyclictest25296-21latency_hist10:49:590
13728991917,1cyclictest0-21swapper/011:10:010
1372899190,19cyclictest0-21swapper/010:30:540
13739991818,0cyclictest0-21swapper/309:51:333
13739991818,0cyclictest0-21swapper/307:11:313
13739991817,1cyclictest14886-21apt-get11:36:323
13739991816,1cyclictest30684-21perf07:44:593
13734991818,0cyclictest0-21swapper/211:41:332
13734991818,0cyclictest0-21swapper/211:36:332
13734991818,0cyclictest0-21swapper/207:31:312
13734991817,1cyclictest0-21swapper/207:16:312
1373499181,16cyclictest28-21ksoftirqd/208:46:332
1373499181,16cyclictest28-21ksoftirqd/208:46:332
13730991818,0cyclictest0-21swapper/110:41:311
13730991818,0cyclictest0-21swapper/107:46:301
13730991818,0cyclictest0-21swapper/107:36:301
13730991817,0cyclictest0-21swapper/112:26:321
13730991815,2cyclictest28999-21perf12:05:001
13730991815,2cyclictest0-21swapper/110:51:331
1373099180,0cyclictest0-21swapper/107:11:301
13728991818,0cyclictest28330-21apt-get08:46:320
13728991818,0cyclictest28330-21apt-get08:46:320
13728991818,0cyclictest0-21swapper/012:11:330
13728991818,0cyclictest0-21swapper/007:51:340
13728991818,0cyclictest0-21swapper/007:46:300
13728991816,1cyclictest24551-21apt-get11:56:310
13728991816,1cyclictest20478-21latency_hist10:40:000
1373999176,10cyclictest31051-21apt-get07:46:313
13739991717,0cyclictest8646-21apt-get08:06:303
13739991717,0cyclictest30719-21apt-get08:51:283
13739991717,0cyclictest30403-21apt-get09:56:353
13739991717,0cyclictest0-21swapper/312:26:313
13739991717,0cyclictest0-21swapper/311:11:323
13739991717,0cyclictest0-21swapper/309:01:323
13739991717,0cyclictest0-21swapper/308:15:013
13739991717,0cyclictest0-21swapper/307:26:293
13734991717,0cyclictest0-21swapper/209:51:332
13734991717,0cyclictest0-21swapper/209:11:322
13734991715,1cyclictest650-21munin-run07:50:002
13730991717,0cyclictest0-21swapper/112:31:351
13730991717,0cyclictest0-21swapper/109:31:331
13730991717,0cyclictest0-21swapper/109:26:301
13730991715,1cyclictest21853-21kworker/u8:208:31:361
13730991714,2cyclictest596-21in:imuxsock08:20:001
13730991714,2cyclictest0-21swapper/109:01:391
1373099171,1cyclictest0-21swapper/111:01:371
1373099170,16cyclictest0-21swapper/109:51:321
13728991717,0cyclictest5591-21apt-get10:11:290
13728991717,0cyclictest0-21swapper/012:06:330
13728991716,1cyclictest0-21swapper/009:11:330
13728991715,1cyclictest61550irq/126-eno110:44:590
1372899170,0cyclictest3435-21apt-get07:56:280
1372899170,0cyclictest0-21swapper/008:11:340
13739991616,0cyclictest0-21swapper/312:16:323
13739991615,0cyclictest5259-21apt-get11:16:303
1373999160,3cyclictest0-21swapper/309:31:323
13734991616,0cyclictest0-21swapper/210:46:362
13734991616,0cyclictest0-21swapper/208:11:302
13734991613,2cyclictest0-21swapper/209:01:402
1373499160,16cyclictest583-21avahi-daemon07:28:592
1373499160,16cyclictest28-21ksoftirqd/209:31:332
13730991616,0cyclictest0-21swapper/112:36:321
13730991616,0cyclictest0-21swapper/111:16:301
13730991616,0cyclictest0-21swapper/111:11:331
13730991616,0cyclictest0-21swapper/110:06:331
13730991616,0cyclictest0-21swapper/110:01:321
13730991616,0cyclictest0-21swapper/107:56:281
13730991614,1cyclictest0-21swapper/111:46:351
13730991614,1cyclictest0-21swapper/111:31:361
13730991614,1cyclictest0-21swapper/110:31:351
13730991614,1cyclictest0-21swapper/109:06:391
13728991616,0cyclictest354-21apt-get10:01:340
13728991616,0cyclictest0-21swapper/008:51:280
13739991515,0cyclictest0-21swapper/310:11:293
13739991515,0cyclictest0-21swapper/307:16:323
13739991513,1cyclictest14389-21tr10:26:373
13734991515,0cyclictest0-21swapper/210:41:322
13734991515,0cyclictest0-21swapper/210:26:322
13734991515,0cyclictest0-21swapper/208:40:002
13734991512,2cyclictest0-21swapper/210:06:422
1373499150,14cyclictest30482-21apt-get11:01:312
1373099158,6cyclictest0-21swapper/111:56:351
13730991515,0cyclictest29363-21apt-get12:06:321
13730991515,0cyclictest0-21swapper/108:01:301
13730991514,0cyclictest0-21swapper/111:26:391
1373099150,1cyclictest1361-21memory11:06:361
13728991515,0cyclictest0-21swapper/012:02:450
13728991513,1cyclictest20753-21perf08:30:000
13728991512,2cyclictest612-21gdbus09:45:000
13739991414,0cyclictest22155-21apt-get11:51:363
13739991414,0cyclictest0-21swapper/312:01:313
1373999140,14cyclictest0-21swapper/307:55:363
13734991411,2cyclictest7895-21munin-run08:05:002
13734991411,2cyclictest0-21swapper/207:56:342
1373499140,14cyclictest0-21swapper/210:01:032
1373499140,0cyclictest0-21swapper/208:23:002
1373099140,14cyclictest0-21swapper/108:20:351
13728991414,0cyclictest0-21swapper/009:06:320
13728991414,0cyclictest0-21swapper/007:41:310
13728991414,0cyclictest0-21swapper/007:10:460
1372899140,0cyclictest0-21swapper/008:15:300
13739991313,0cyclictest0-21swapper/310:50:553
13728991313,0cyclictest0-21swapper/011:35:450
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional