You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 00:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Nov 27, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
33562242166,24sleep30-21swapper/307:07:283
32772228163,52sleep20-21swapper/207:06:262
35432210136,24sleep10-21swapper/107:09:511
34682210181,19sleep00-21swapper/007:08:530
2158921580,5sleep3366499cyclictest11:29:563
1053721240,3sleep3366499cyclictest12:15:243
294322990,3sleep2365799cyclictest11:45:202
128852770,4sleep1365199cyclictest12:20:251
261242620,9sleep341-21ksoftirqd/310:25:133
124312610,2sleep00-21swapper/009:55:110
152422570,3sleep20-21swapper/210:00:162
95142540,2sleep00-21swapper/012:15:140
3664994427,10cyclictest0-21swapper/309:34:013
113702380,2sleep10-21swapper/107:25:121
366499371,25cyclictest0-21swapper/311:49:193
365799364,4cyclictest31439-21cat10:35:252
366499351,27cyclictest0-21swapper/309:28:413
366499351,23cyclictest0-21swapper/311:23:273
366499351,15cyclictest0-21swapper/311:35:013
366499351,14cyclictest0-21swapper/312:38:253
365799352,5cyclictest2453-40dnf08:16:332
3657993514,18cyclictest24045-21gltestperf10:20:162
120822350,5sleep0364499cyclictest09:53:590
366499341,14cyclictest0-21swapper/311:09:133
366499341,14cyclictest0-21swapper/307:49:433
366499341,11cyclictest0-21swapper/307:54:133
365799341,8cyclictest0-21swapper/209:35:212
3657993412,19cyclictest12737-21smart_sda12:20:232
366499331,19cyclictest0-21swapper/312:34:353
366499331,19cyclictest0-21swapper/310:49:173
366499331,19cyclictest0-21swapper/310:49:173
366499331,16cyclictest0-21swapper/309:15:553
366499331,14cyclictest0-21swapper/309:51:013
366499331,13cyclictest0-21swapper/308:16:013
365799339,3cyclictest5171-21/usr/sbin/munin08:25:182
365799337,2cyclictest6185-21cat12:05:232
365799334,3cyclictest20549-21kworker/2:009:49:082
365799333,26cyclictest26289-21turbostat.cron08:00:012
365799332,5cyclictest5109-21smart_sda07:10:242
366499321,15cyclictest0-21swapper/310:44:213
366499321,14cyclictest0-21swapper/312:22:233
366499321,11cyclictest0-21swapper/308:00:153
365799329,3cyclictest28378-21fschecks_time10:30:152
365799328,3cyclictest15873-21/usr/sbin/munin12:30:222
365799325,3cyclictest12050irq/123-ahci[0008:05:132
365799324,5cyclictest30629-21smart_sda09:20:262
365799324,25cyclictest7360-21head09:40:272
365799323,26cyclictest24089-21systemd-journal07:55:022
365799321,8cyclictest25888-21cat10:25:122
365799321,22cyclictest0-21swapper/211:26:102
3657993211,3cyclictest3541-21fschecks_time08:20:132
364499326,12cyclictest4028-21perf10:50:000
364499326,12cyclictest4028-21perf10:50:000
366499311,15cyclictest0-21swapper/310:19:593
366499311,13cyclictest0-21swapper/307:30:213
365799319,3cyclictest848-21sssd_nss08:55:022
365799319,3cyclictest25667-21idleruntime09:10:172
365799319,2cyclictest12399-21/usr/sbin/munin09:55:132
365799318,4cyclictest1552-21/usr/sbin/munin11:55:222
365799312,6cyclictest836-21runrttasks11:18:322
365799312,5cyclictest9896-21smart_sda11:00:242
365799312,4cyclictest11124-40dnf12:18:342
365799311,4cyclictest4573-21df10:50:132
365799311,12cyclictest2982-21grep09:30:282
3657993110,3cyclictest836-21runrttasks11:42:012
3657993110,3cyclictest14297-21gltestperf12:25:142
3651993110,10cyclictest23-21ksoftirqd/111:10:221
366499301,19cyclictest0-21swapper/312:07:093
366499301,19cyclictest0-21swapper/309:11:253
366499301,19cyclictest0-21swapper/308:58:133
366499301,18cyclictest0-21swapper/309:36:533
366499301,15cyclictest0-21swapper/311:43:533
366499301,13cyclictest0-21swapper/312:04:513
366499301,13cyclictest0-21swapper/310:55:153
366499301,13cyclictest0-21swapper/310:14:173
366499301,12cyclictest0-21swapper/310:31:013
366499301,12cyclictest0-21swapper/309:22:053
366499301,12cyclictest0-21swapper/307:15:313
366499301,11cyclictest0-21swapper/312:29:573
366499301,11cyclictest0-21swapper/307:21:333
366499301,10cyclictest0-21swapper/308:38:073
3664993010,3cyclictest0-21swapper/310:05:153
365799309,3cyclictest836-21runrttasks12:02:412
365799309,3cyclictest836-21runrttasks11:34:552
365799308,4cyclictest18975-21/usr/sbin/munin07:40:262
365799307,3cyclictest32444-21grep08:10:242
365799304,3cyclictest86550irq/126-enp1s0-10:45:272
365799304,3cyclictest86550irq/126-enp1s0-10:45:272
365799304,3cyclictest18323-21grep11:20:212
365799304,11cyclictest780-21rngd07:24:012
365799302,3cyclictest15294-21crond07:35:012
365799301,4cyclictest569-21/usr/sbin/munin09:25:262
3657993011,16cyclictest814-21sshd10:58:332
3657993011,16cyclictest21028-21pluginstate07:45:222
365799301,10cyclictest0-21swapper/209:17:442
366499291,9cyclictest0-21swapper/310:24:533
366499291,18cyclictest0-21swapper/307:57:493
366499291,12cyclictest24735-21diskmemload09:55:113
366499291,12cyclictest0-21swapper/309:40:093
366499291,12cyclictest0-21swapper/307:41:193
366499291,11cyclictest0-21swapper/311:10:113
366499291,11cyclictest0-21swapper/307:29:573
365799299,3cyclictest8469-21open_inodes08:30:212
365799298,3cyclictest86550irq/126-enp1s0-11:10:002
365799298,3cyclictest30640-21cat11:50:112
365799298,3cyclictest30640-21cat11:50:112
365799298,3cyclictest19512-21proc_pri08:55:222
365799298,3cyclictest13042-21/usr/sbin/munin11:10:152
365799298,3cyclictest12016-21sort07:25:172
365799293,5cyclictest25107-21ntp_states11:35:222
365799290,22cyclictest0-21swapper/209:50:362
366499281,19cyclictest0-21swapper/308:43:333
366499281,19cyclictest0-21swapper/308:43:333
366499281,13cyclictest0-21swapper/311:02:173
366499281,12cyclictest0-21swapper/311:32:253
366499281,12cyclictest0-21swapper/309:48:153
366499281,11cyclictest7714-21df_inode08:30:133
366499281,11cyclictest0-21swapper/311:17:033
365799282,6cyclictest13961-21/usr/sbin/munin08:45:182
365799282,3cyclictest17876-21ntp_states10:05:242
3657992815,4cyclictest780-21rngd07:38:522
3657992815,4cyclictest780-21rngd07:38:522
365799281,4cyclictest1402-21/usr/sbin/munin10:40:262
365199281,14cyclictest0-21swapper/109:55:121
366499271,12cyclictest0-21swapper/310:38:493
366499271,12cyclictest0-21swapper/310:03:193
366499271,12cyclictest0-21swapper/308:53:353
366499271,11cyclictest0-21swapper/311:53:393
366499271,11cyclictest0-21swapper/311:53:393
366499271,11cyclictest0-21swapper/310:51:193
365799278,3cyclictest21554-21netstat09:00:192
365799277,3cyclictest9562-21/usr/sbin/munin08:35:232
365799271,5cyclictest0-21swapper/210:19:152
365799271,4cyclictest18405-21df12:35:122
365799271,3cyclictest11762-21/usr/sbin/munin08:40:142
365799271,3cyclictest11762-21/usr/sbin/munin08:40:142
365799271,2cyclictest0-21swapper/207:16:222
3657992712,12cyclictest23835-21open_files09:05:202
3657992710,3cyclictest20850-21sshd10:13:342
365199271,10cyclictest18513-21ntp_states11:20:241
365199270,20cyclictest0-21swapper/111:49:111
364499270,12cyclictest2754-21awk08:20:010
3664992612,3cyclictest0-21swapper/307:13:023
366499261,10cyclictest0-21swapper/312:12:113
366499261,10cyclictest0-21swapper/309:01:413
366499261,10cyclictest0-21swapper/308:20:173
365799268,3cyclictest27179-21cut08:00:132
365799261,5cyclictest0-21swapper/212:11:032
365199261,17cyclictest0-21swapper/111:05:211
364499261,13cyclictest9351-21awk08:35:000
216142260,4sleep0364499cyclictest11:30:010
366499251,17cyclictest0-21swapper/308:10:053
366499251,16cyclictest0-21swapper/308:47:193
366499251,13cyclictest0-21swapper/308:06:493
3651992519,3cyclictest25780-21irqstats09:10:181
365199251,5cyclictest2453-40dnf08:16:341
365199251,15cyclictest780-21rngd10:15:261
365199250,15cyclictest0-21swapper/112:37:521
366499241,14cyclictest0-21swapper/308:26:073
366499241,12cyclictest0-21swapper/309:05:033
366499241,12cyclictest0-21swapper/307:37:393
366499241,12cyclictest0-21swapper/307:37:393
366499240,10cyclictest0-21swapper/311:55:233
365199241,19cyclictest15258-21users12:25:251
365199241,12cyclictest0-21swapper/109:45:371
365199239,5cyclictest0-21swapper/112:00:571
365199236,7cyclictest23456-21switchtime07:50:281
365199235,8cyclictest0-21swapper/108:38:321
365199232,11cyclictest0-21swapper/112:15:151
365199231,15cyclictest9-21rcu_preempt10:45:241
365199231,15cyclictest9-21rcu_preempt10:45:241
365199231,11cyclictest0-21swapper/111:18:261
365199231,11cyclictest0-21swapper/110:52:091
364499238,3cyclictest26807-21kworker/0:109:20:270
364499236,15cyclictest10620-21munin-run11:05:010
365199228,12cyclictest31509-21kworker/u8:108:10:321
365199222,18cyclictest0-21swapper/110:30:111
3651992218,3cyclictest0-21swapper/111:34:411
3651992216,3cyclictest31372-21if_enp2s011:50:171
3651992216,3cyclictest31372-21if_enp2s011:50:171
365199221,14cyclictest0-21swapper/111:25:261
365199221,12cyclictest27172-21/usr/sbin/munin09:15:121
365199221,12cyclictest18043-21expr07:40:151
365199221,11cyclictest31246-21taskset09:24:001
3644992215,5cyclictest848-21sssd_nss08:20:220
3644992210,3cyclictest12363-21grep08:40:150
3644992210,3cyclictest12363-21grep08:40:150
365199217,3cyclictest0-21swapper/110:57:161
365199214,9cyclictest21-21rcuc/108:20:111
365199214,8cyclictest848-21sssd_nss07:30:211
365199212,3cyclictest0-21swapper/109:34:291
365199212,18cyclictest0-21swapper/110:08:241
365199211,8cyclictest11985-21chrt09:52:441
3651992117,3cyclictest0-21swapper/111:42:301
365199211,2cyclictest8853-21expr11:00:141
365199211,2cyclictest24735-21diskmemload10:10:501
365199211,18cyclictest0-21swapper/110:40:051
365199211,15cyclictest0-21swapper/112:33:491
365199211,14cyclictest0-21swapper/111:38:141
365199211,13cyclictest0-21swapper/109:39:191
365199211,13cyclictest0-21swapper/109:26:571
365199211,11cyclictest7390-21users09:40:271
365199211,11cyclictest10466-21awk07:20:261
364499217,4cyclictest1119-21kworker/0:109:31:270
364499213,4cyclictest19423-21awk11:25:010
364499211,6cyclictest0-21swapper/009:49:440
365199208,3cyclictest8144-21ntp_states12:10:231
365199204,10cyclictest0-21swapper/110:21:421
365199202,17cyclictest0-21swapper/109:09:241
365199202,17cyclictest0-21swapper/107:59:301
3651992017,2cyclictest0-21swapper/107:35:531
3651992017,2cyclictest0-21swapper/107:35:531
3651992016,3cyclictest0-21swapper/112:05:411
3651992016,3cyclictest0-21swapper/111:59:131
365199201,18cyclictest0-21swapper/110:35:471
365199201,18cyclictest0-21swapper/108:06:451
365199201,17cyclictest0-21swapper/110:25:071
365199201,11cyclictest7085-21diskstats07:15:141
365199201,11cyclictest14657-21grep08:45:201
365199201,10cyclictest0-21swapper/110:00:151
364499209,3cyclictest6436-21/usr/sbin/munin10:55:150
364499209,3cyclictest3598-21/usr/sbin/munin09:35:230
364499207,3cyclictest152462tail10:00:150
364499206,4cyclictest15022-21perf11:15:010
364499203,9cyclictest11-21rcuc/008:10:100
364499201,9cyclictest0-21swapper/008:50:240
364499201,6cyclictest0-21swapper/011:45:350
3644992015,3cyclictest585-21grep09:25:250
3644992014,3cyclictest19635-21grep10:10:170
364499201,16cyclictest29361-21head10:30:250
364499201,11cyclictest17522-21sshd12:31:030
364499201,11cyclictest0-21swapper/011:57:050
364499201,11cyclictest0-21swapper/009:00:230
364499201,10cyclictest23639-21grep07:50:280
364499201,10cyclictest22711-21unixbench_multi10:15:270
188742200,4sleep011-21rcuc/012:35:170
365199192,8cyclictest26516-21/usr/sbin/munin08:00:171
3651991915,3cyclictest0-21swapper/108:30:301
365199191,17cyclictest0-21swapper/108:57:511
365199191,16cyclictest0-21swapper/109:00:171
365199191,16cyclictest0-21swapper/108:40:051
365199191,16cyclictest0-21swapper/108:40:051
365199191,10cyclictest16173-21bc08:50:141
365199190,18cyclictest0-21swapper/107:46:481
364499199,3cyclictest86750irq/128-enp1s0-11:43:190
364499199,2cyclictest6868-21ntp_states09:40:210
364499197,4cyclictest24244-21kernelversion10:20:180
364499196,4cyclictest12619-21grep07:25:310
364499196,3cyclictest27611-21fschecks_count09:15:140
364499193,7cyclictest16747-21grep11:15:260
364499193,12cyclictest86750irq/128-enp1s0-12:23:370
364499192,2cyclictest5357-21grep07:10:260
364499191,10cyclictest22826-21needreboot11:30:190
364499191,10cyclictest17408-21grep10:05:170
364499191,10cyclictest14634-21snmpd07:17:560
364499191,10cyclictest0-21swapper/012:11:430
364499191,10cyclictest0-21swapper/010:41:030
364499190,10cyclictest14662-21needreboot12:25:180
365199188,2cyclictest5686-21taskset07:13:301
3651991813,3cyclictest6714-21grep08:25:251
364499182,4cyclictest12163-21sed11:05:240
364499181,8cyclictest0-21swapper/011:50:090
364499181,8cyclictest0-21swapper/011:50:090
364499181,7cyclictest14952-21needreboot08:45:220
364499181,6cyclictest0-21swapper/010:51:470
364499181,6cyclictest0-21swapper/007:41:290
364499181,2cyclictest0-21swapper/009:09:090
364499181,1cyclictest0-21swapper/012:01:550
364499181,10cyclictest4878-21/usr/sbin/munin12:05:260
364499181,10cyclictest11037-21grep08:35:230
364499174,3cyclictest86750irq/128-enp1s0-10:25:230
364499173,4cyclictest9147-21ls07:20:130
364499171,9cyclictest0-21swapper/011:35:210
364499171,6cyclictest0-21swapper/010:35:110
364499171,3cyclictest0-21swapper/007:45:180
3644991712,3cyclictest29428-21sort08:05:150
364499171,15cyclictest0-21swapper/007:37:000
364499171,15cyclictest0-21swapper/007:37:000
3644991711,3cyclictest28496-21idleruntime-cro08:05:010
150012170,3sleep00-21swapper/007:31:170
364499163,4cyclictest0-21swapper/008:25:050
364499163,11cyclictest26141-21grep09:10:230
364499163,11cyclictest11-21rcuc/008:55:160
364499161,5cyclictest0-21swapper/007:55:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional