You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 00:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Thu Nov 13, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3239699426,0cyclictest0-21swapper/311:08:343
32388993332,1cyclictest20016-21lspci11:08:341
32384992619,6cyclictest28213-21turbostat.cron09:15:010
32384992619,6cyclictest1757-21idleruntime-cro08:20:000
3239699240,23cyclictest27897-21awk12:30:013
3239699240,1cyclictest5126-21perf10:40:003
3238899240,23cyclictest28502-21apt-get11:26:261
32384992418,5cyclictest11006-21apt-get11:56:270
3238499240,2cyclictest0-21swapper/010:51:260
3239699230,22cyclictest1759-21sendmail-msp08:19:593
3239399230,1cyclictest593-21dbus-daemon09:25:012
3238899230,22cyclictest20314-21perf12:15:001
32384992321,1cyclictest0-21swapper/012:11:230
32384992321,1cyclictest0-21swapper/011:55:000
32384992321,1cyclictest0-21swapper/007:20:000
32384992318,4cyclictest0-21swapper/011:21:240
32396992220,1cyclictest3001-21awk11:40:003
32396992220,1cyclictest0-21swapper/308:50:003
3239699220,1cyclictest12323-21apt-get07:36:253
32393992220,1cyclictest30675-21apt-get12:36:262
32393992220,1cyclictest18266-21apt-get12:11:232
32393992220,1cyclictest11968-21perf07:35:002
3239399220,21cyclictest6905-21apt-get08:31:242
3239399220,21cyclictest2996-21sendmail-msp11:40:002
3239399220,21cyclictest23455-21apt-get10:11:242
3239399220,21cyclictest16161-21perf08:49:592
3239399220,21cyclictest0-21swapper/211:21:252
3239399220,21cyclictest0-21swapper/210:10:012
3239399220,21cyclictest0-21swapper/207:21:232
32388992220,1cyclictest3423-21perf09:30:001
32388992220,1cyclictest29348-21munin-run08:10:001
32388992220,1cyclictest26940-21latency_hist08:05:001
32388992219,2cyclictest7858-21perf11:50:011
3238899220,21cyclictest31394-21apt-get09:21:231
3238899220,21cyclictest30901-21apt-get11:31:251
3238899220,21cyclictest16191-21apt-get09:56:251
3238899220,21cyclictest0-21swapper/111:41:251
3238899220,1cyclictest28205-21cron09:15:011
3238899220,1cyclictest17120-21perf11:04:591
32384992220,1cyclictest18602-21ls08:55:000
32384992220,1cyclictest10655-21munin-run09:45:000
3238499220,21cyclictest25525-21munin-run12:25:000
3238499220,21cyclictest25525-21munin-run12:25:000
3238499220,21cyclictest0-21swapper/010:11:240
3238499220,21cyclictest0-21swapper/009:51:250
3238499220,21cyclictest0-21swapper/009:26:230
3238499220,21cyclictest0-21swapper/008:41:230
3238499220,1cyclictest17140-21munin-run11:05:000
3238499220,16cyclictest313-21systemd-journal07:30:000
32396992121,0cyclictest0-21swapper/312:21:233
32396992121,0cyclictest0-21swapper/312:21:233
32396992121,0cyclictest0-21swapper/312:16:253
32396992120,1cyclictest23109-21perf10:10:013
32396992120,0cyclictest639-21apt-get10:31:233
3239699210,21cyclictest0-21swapper/311:21:243
3239699210,20cyclictest0-21swapper/310:01:243
3239699210,20cyclictest0-21swapper/309:26:233
3239699210,1cyclictest27299-21apt-get08:06:243
3239699210,1cyclictest13067-21munin-run12:00:003
3239699210,1cyclictest0-21swapper/310:55:013
3239699210,0cyclictest0-21swapper/308:56:243
3239699210,0cyclictest0-21swapper/308:01:223
32393992120,1cyclictest30617-21apt-get09:20:012
32393992120,1cyclictest17865-21munin-plugin-st12:09:592
32393992120,1cyclictest12674-21apt-get10:56:242
32393992119,1cyclictest481-21apt11:35:002
32393992119,1cyclictest0-21swapper/207:45:012
3239399210,21cyclictest0-21swapper/211:56:272
3239399210,20cyclictest0-21swapper/207:16:232
3239399210,0cyclictest3793-21apt-get09:31:242
3239399210,0cyclictest23480-21apt-get12:21:242
3239399210,0cyclictest23480-21apt-get12:21:242
3239399210,0cyclictest0-21swapper/212:16:242
3239399210,0cyclictest0-21swapper/211:06:232
32388992121,0cyclictest0-21swapper/112:21:241
32388992121,0cyclictest0-21swapper/112:21:241
32388992119,1cyclictest612-21gdbus07:25:001
32388992119,1cyclictest24518-21latency_hist08:00:001
32388992119,1cyclictest15465-21idleruntime-cro12:04:591
3238899211,19cyclictest0-21swapper/107:15:001
3238899210,21cyclictest0-21swapper/108:16:231
3238899210,21cyclictest0-21swapper/107:31:231
3238899210,20cyclictest0-21swapper/109:16:251
3238899210,1cyclictest7886-21apt-get10:46:271
3238899210,0cyclictest18968-21apt-get08:56:251
3238899210,0cyclictest0-21swapper/112:15:021
3238499213,4cyclictest30839-21gpg-connect-age09:20:010
32384992121,0cyclictest0-21swapper/009:56:250
32384992121,0cyclictest0-21swapper/008:31:230
32384992120,1cyclictest0-21swapper/012:26:260
32384992120,0cyclictest3793-21apt-get09:31:250
3238499210,6cyclictest0-21swapper/010:21:230
3238499210,3cyclictest313-21systemd-journal09:50:000
3238499210,20cyclictest925-21apt-get11:36:250
3238499210,0cyclictest21058-21apt-get12:16:250
3238499210,0cyclictest0-21swapper/012:06:240
3238499210,0cyclictest0-21swapper/010:26:240
32396992020,0cyclictest0-21swapper/311:16:253
32396992020,0cyclictest0-21swapper/311:11:243
32396992020,0cyclictest0-21swapper/310:41:243
32396992020,0cyclictest0-21swapper/309:31:243
32396992020,0cyclictest0-21swapper/309:11:253
32396992020,0cyclictest0-21swapper/307:55:013
32396992020,0cyclictest0-21swapper/307:21:233
32396992018,1cyclictest25808-21latency_hist09:09:593
32396992018,1cyclictest0-21swapper/312:40:013
32396992017,2cyclictest16878-21apt07:45:013
3239699200,20cyclictest0-21swapper/309:36:253
3239699200,0cyclictest0-21swapper/312:11:243
3239699200,0cyclictest0-21swapper/308:41:233
3239699200,0cyclictest0-21swapper/307:46:243
32393992020,0cyclictest0-21swapper/211:26:262
32393992020,0cyclictest0-21swapper/210:36:262
32393992020,0cyclictest0-21swapper/209:56:242
32393992020,0cyclictest0-21swapper/208:56:242
32393992020,0cyclictest0-21swapper/208:36:232
32393992020,0cyclictest0-21swapper/208:11:242
32393992019,1cyclictest0-21swapper/209:11:262
3239399200,0cyclictest32091-21apt-get08:16:232
3239399200,0cyclictest21708-21apt-get07:55:002
3239399200,0cyclictest0-21swapper/207:51:242
32388992020,0cyclictest0-21swapper/111:56:261
32388992020,0cyclictest0-21swapper/111:51:241
32388992020,0cyclictest0-21swapper/110:36:261
32388992020,0cyclictest0-21swapper/110:06:261
32388992020,0cyclictest0-21swapper/108:36:221
32388992019,0cyclictest0-21swapper/107:41:251
32388992018,1cyclictest593-21dbus-daemon09:44:591
3238899200,20cyclictest0-21swapper/110:56:231
3238899200,20cyclictest0-21swapper/107:26:231
3238899200,0cyclictest0-21swapper/110:11:231
3238899200,0cyclictest0-21swapper/110:01:241
3238899200,0cyclictest0-21swapper/107:51:241
32384992020,0cyclictest0-21swapper/012:36:240
32384992020,0cyclictest0-21swapper/011:26:260
32384992020,0cyclictest0-21swapper/009:21:220
32384992016,4cyclictest0-21swapper/011:06:240
3238499200,0cyclictest25852-21apt-get10:16:240
3238499200,0cyclictest0-21swapper/012:31:230
3238499200,0cyclictest0-21swapper/008:56:250
32396991919,0cyclictest0-21swapper/310:11:233
32396991919,0cyclictest0-21swapper/309:21:233
32396991919,0cyclictest0-21swapper/308:36:233
32396991918,1cyclictest19692-21apt-get07:51:243
32396991918,1cyclictest0-21swapper/309:50:003
32396991918,1cyclictest0-21swapper/308:31:233
32396991918,0cyclictest0-21swapper/311:26:253
32396991917,1cyclictest0-21swapper/310:56:243
3239699190,1cyclictest15819-21apt-get12:06:253
3239699190,0cyclictest0-21swapper/308:26:233
32393991919,0cyclictest15094-21apt-get11:01:252
32393991919,0cyclictest13421-21apt-get12:01:242
32393991919,0cyclictest0-21swapper/210:26:242
32393991919,0cyclictest0-21swapper/209:41:232
32393991919,0cyclictest0-21swapper/207:11:242
32388991919,0cyclictest2708-21apt-get07:16:231
32388991919,0cyclictest0-21swapper/112:26:261
32388991919,0cyclictest0-21swapper/111:20:011
32388991919,0cyclictest0-21swapper/109:46:241
32388991919,0cyclictest0-21swapper/109:31:231
32388991919,0cyclictest0-21swapper/109:06:251
32388991919,0cyclictest0-21swapper/108:51:231
32388991919,0cyclictest0-21swapper/107:46:231
32388991918,0cyclictest0-21swapper/111:36:261
32384991919,0cyclictest0-21swapper/011:46:230
32384991919,0cyclictest0-21swapper/011:11:240
32384991919,0cyclictest0-21swapper/008:46:240
32384991919,0cyclictest0-21swapper/008:06:240
32384991917,1cyclictest0-21swapper/007:51:240
3238499190,0cyclictest0-21swapper/008:21:250
32396991818,0cyclictest0-21swapper/312:01:243
32396991818,0cyclictest0-21swapper/309:20:013
32396991818,0cyclictest0-21swapper/309:01:253
32396991818,0cyclictest0-21swapper/307:26:243
32396991818,0cyclictest0-21swapper/307:16:233
32396991817,1cyclictest29697-21apt-get08:11:233
32396991817,0cyclictest0-21swapper/307:35:003
32396991816,1cyclictest477-21sh11:35:003
32393991818,0cyclictest6189-21apt-get09:36:252
32393991818,0cyclictest21387-21apt-get09:01:242
32393991818,0cyclictest17243-21apt-get07:46:242
32393991818,0cyclictest0-21swapper/212:35:012
32393991818,0cyclictest0-21swapper/211:41:252
32393991818,0cyclictest0-21swapper/209:06:252
32393991817,0cyclictest11722-21apt-get08:41:242
32393991816,1cyclictest18607-21latency_hist08:54:592
32388991818,0cyclictest28275-21apt-get10:21:231
32388991818,0cyclictest0-21swapper/109:01:251
32388991818,0cyclictest0-21swapper/108:31:241
32388991818,0cyclictest0-21swapper/108:26:231
32384991818,0cyclictest9303-21apt-get08:36:230
32384991818,0cyclictest5131-21apt-get07:21:220
32384991818,0cyclictest0-21swapper/009:06:250
32384991818,0cyclictest0-21swapper/007:41:260
32384991818,0cyclictest0-21swapper/007:36:250
32384991813,4cyclictest26946-21perf08:05:000
32396991717,0cyclictest0-21swapper/311:51:243
32396991717,0cyclictest0-21swapper/310:26:243
32396991717,0cyclictest0-21swapper/309:56:253
32396991716,0cyclictest8606-21apt-get09:41:233
3239699170,16cyclictest18577-21cron08:55:003
32393991717,0cyclictest0-21swapper/212:26:262
32393991717,0cyclictest0-21swapper/211:11:232
32393991717,0cyclictest0-21swapper/207:36:252
32393991716,1cyclictest13397-21apt-get09:51:242
32393991715,1cyclictest7526-21apt-get07:26:232
32393991715,1cyclictest25355-21perf11:19:592
3239399170,1cyclictest612-21gdbus10:20:002
3239399170,16cyclictest11001-21apt-get09:46:232
32388991717,0cyclictest0-21swapper/110:16:241
32388991717,0cyclictest0-21swapper/108:11:241
3238499174,12cyclictest20756-21munin-run10:05:010
32384991716,0cyclictest0-21swapper/010:56:240
3238499170,2cyclictest313-21systemd-journal11:43:000
32396991616,0cyclictest0-21swapper/311:01:253
32396991616,0cyclictest0-21swapper/307:11:233
32396991613,2cyclictest583-21avahi-daemon09:55:003
32393991616,0cyclictest0-21swapper/211:51:242
32393991616,0cyclictest0-21swapper/210:31:242
32393991616,0cyclictest0-21swapper/209:26:232
32393991614,1cyclictest7534-21munin-run10:45:002
32393991614,1cyclictest0-21swapper/211:50:012
32388991616,0cyclictest0-21swapper/109:51:241
32388991616,0cyclictest0-21swapper/108:41:231
32388991615,1cyclictest5487-21apt-get10:41:241
3238899160,15cyclictest29398-21phc2sys-jitter12:31:301
3238899160,15cyclictest0-21swapper/107:36:241
32384991616,0cyclictest0-21swapper/010:41:230
32384991615,1cyclictest9923-21apt-get07:31:210
32384991614,1cyclictest3872-21irqstats10:36:290
32384991614,1cyclictest0-21swapper/008:26:260
3238499160,9cyclictest0-21swapper/010:46:280
3238499160,15cyclictest593-21dbus-daemon12:05:000
3238499160,0cyclictest0-21swapper/009:03:230
32396991515,0cyclictest0-21swapper/312:31:243
32396991515,0cyclictest0-21swapper/311:41:253
32396991513,1cyclictest0-21swapper/310:46:283
3239399153,11cyclictest313-21systemd-journal10:04:552
32393991515,0cyclictest0-21swapper/210:46:272
32393991513,1cyclictest28798-21perl10:21:252
32393991513,1cyclictest0-21swapper/208:01:292
32388991515,0cyclictest639-21apt-get10:31:241
32388991515,0cyclictest2113-21apt-get08:21:241
32388991514,1cyclictest30671-21apt-get10:26:241
3238499153,0cyclictest0-21swapper/010:06:260
32384991515,0cyclictest0-21swapper/011:31:260
32384991514,1cyclictest21708-21apt-get07:55:010
32384991514,0cyclictest0-21swapper/008:11:240
3239699140,14cyclictest0-21swapper/310:24:383
3239699140,14cyclictest0-21swapper/310:15:333
3239699140,0cyclictest0-21swapper/311:45:093
3239699140,0cyclictest0-21swapper/308:21:203
32393991414,0cyclictest0-21swapper/210:50:172
3239399140,14cyclictest0-21swapper/208:25:202
3239399140,14cyclictest0-21swapper/208:20:522
3239399140,0cyclictest0-21swapper/208:07:012
32388991414,0cyclictest0-21swapper/112:06:241
32388991414,0cyclictest0-21swapper/109:37:251
3238899140,14cyclictest0-21swapper/110:52:221
3238899140,0cyclictest0-21swapper/111:10:241
32384991414,0cyclictest0-21swapper/009:35:020
3238499140,14cyclictest0-21swapper/007:14:270
3238499140,0cyclictest0-21swapper/011:16:200
32384991313,0cyclictest0-21swapper/010:30:570
32384991313,0cyclictest0-21swapper/007:45:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional