You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 23:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Jun 28, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
236492240164,23sleep10-21swapper/107:06:161
238642236163,23sleep30-21swapper/307:09:043
238122236163,59sleep00-21swapper/007:08:240
235912222191,20sleep20-21swapper/207:05:332
965021680,4sleep32404199cyclictest07:46:483
24036998075,3cyclictest0-21swapper/208:43:332
24036997674,1cyclictest0-21swapper/212:05:222
24036996359,3cyclictest0-21swapper/210:11:162
226792580,3sleep20-21swapper/209:30:252
313492550,3sleep32404199cyclictest09:50:213
313442480,1sleep30-21swapper/307:25:003
24036994625,3cyclictest4777-21kworker/2:011:42:322
24029994510,4cyclictest20569-21munin-run11:55:011
24036994028,3cyclictest4777-21kworker/2:010:40:262
2402499399,14cyclictest14944-21awk09:15:000
24036993826,3cyclictest4777-21kworker/2:012:37:522
2402999389,3cyclictest30593-21cpu09:50:151
24029993715,3cyclictest0-21swapper/108:35:301
24036993621,9cyclictest4777-21kworker/2:010:00:382
24029993613,3cyclictest12670-21df07:55:151
24029993611,3cyclictest29371-21latency_hist12:15:021
24036993526,2cyclictest4777-21kworker/2:009:36:442
24036993526,2cyclictest4777-21kworker/2:009:36:442
24036993524,3cyclictest4777-21kworker/2:007:13:122
2403699352,22cyclictest4777-21kworker/2:011:20:242
2402999352,14cyclictest712-21munin-run11:10:021
2402999352,14cyclictest0-21swapper/107:35:151
2402999351,13cyclictest8430-21sort09:00:021
24029993511,3cyclictest3836-21/usr/sbin/munin12:30:181
2402999350,17cyclictest0-21swapper/110:30:431
2402999348,3cyclictest0-21swapper/111:35:291
2402999341,3cyclictest1239-21sed08:40:371
2403699330,27cyclictest4777-21kworker/2:011:48:322
2403699330,27cyclictest4777-21kworker/2:011:48:322
2402999339,3cyclictest1096-21latency_hist07:30:011
2402999334,10cyclictest1-21systemd11:00:101
2402999332,12cyclictest19430-21grep09:25:011
2402999331,3cyclictest15255-21/usr/sbin/munin08:00:191
24029993312,3cyclictest1638-21seq12:25:131
24029993312,3cyclictest15969-21seq11:42:341
2402999331,10cyclictest13608-21cpu10:25:171
24029993310,3cyclictest1882-21grep09:55:281
178442330,5sleep117843-21pmu-power08:05:251
2403699322,21cyclictest4777-21kworker/2:011:06:482
24036993218,3cyclictest4777-21kworker/2:011:16:242
2402999322,9cyclictest5860-21ntp_states10:05:241
2402999321,7cyclictest18771-21if_err_enp2s010:35:211
2402999321,7cyclictest18771-21if_err_enp2s010:35:211
2402999321,3cyclictest26237-21munin-node07:15:001
24029993212,3cyclictest22469-21basename09:30:221
2402999321,16cyclictest0-21swapper/110:42:201
2402999321,15cyclictest12050irq/123-ahci[0009:03:201
2402999321,13cyclictest30328-21awk09:50:011
24029993211,3cyclictest26737-21sort07:15:021
24029993211,2cyclictest3220-21/usr/sbin/munin11:15:331
2402999321,11cyclictest2935-21diskstats10:00:201
2402999321,10cyclictest27922-21pmu-power10:55:261
2403699312,18cyclictest4777-21kworker/2:007:23:462
24036993118,7cyclictest4777-21kworker/2:012:13:122
2402999316,2cyclictest28204-21cat09:45:011
2402999315,3cyclictest19757-21expr09:25:141
2402999315,2cyclictest16402-21kworker/u8:011:59:081
24029993112,3cyclictest3936-21perf08:54:561
2402999311,13cyclictest12726-21latency_hist09:10:021
24029993110,4cyclictest28153-21cut08:30:171
24029993110,3cyclictest11837-21seq07:51:441
2402499313,13cyclictest12736-21munin-run09:10:010
2402999301,3cyclictest0-21swapper/108:21:081
2402999301,17cyclictest0-21swapper/108:46:551
2402999301,16cyclictest0-21swapper/111:31:031
2402999301,16cyclictest0-21swapper/111:23:021
24029993011,3cyclictest32427-21grep12:20:191
2402999301,12cyclictest25109-21awk09:35:261
2402999301,12cyclictest25109-21awk09:35:261
2402999301,12cyclictest1342-21df_inode11:10:181
2402999301,10cyclictest0-21swapper/110:10:541
24029993010,3cyclictest13284-21expr09:10:151
2402499307,11cyclictest19346-21perf09:25:000
24036992916,3cyclictest4777-21kworker/2:009:15:202
24036992914,3cyclictest4777-21kworker/2:009:28:542
24036992911,3cyclictest86750irq/128-enp1s0-10:30:062
2403699291,11cyclictest4777-21kworker/2:011:27:522
2403699291,11cyclictest4777-21kworker/2:010:18:242
2402999291,4cyclictest24384-1kworker/u9:012:19:241
24029992913,2cyclictest0-21swapper/110:24:431
2402999291,2cyclictest0-21swapper/112:39:331
2402999291,16cyclictest0-21swapper/110:45:061
24029992911,3cyclictest8571-21cat11:25:241
2402999290,16cyclictest0-21swapper/107:31:041
24024992910,16cyclictest712-21munin-run11:10:020
24036992815,3cyclictest4777-21kworker/2:007:49:242
24036992813,3cyclictest4777-21kworker/2:012:19:082
2403699281,10cyclictest4777-21kworker/2:010:51:082
2402999286,3cyclictest8280-21df07:45:171
2402999281,7cyclictest31239-21runrttasks07:24:331
24029992815,2cyclictest0-21swapper/110:52:001
2402999281,2cyclictest0-21swapper/112:05:111
2402999281,2cyclictest0-21swapper/109:19:271
2402999281,2cyclictest0-21swapper/108:12:581
2402999281,16cyclictest0-21swapper/111:48:411
2402999281,16cyclictest0-21swapper/111:48:411
24029992811,3cyclictest26312-21if_err_enp1s008:25:211
2402999280,2cyclictest0-21swapper/108:16:241
2403699272,12cyclictest4777-21kworker/2:009:21:142
24036992718,2cyclictest4777-21kworker/2:012:00:462
24036992716,3cyclictest4777-21kworker/2:012:31:002
24036992712,2cyclictest4777-21kworker/2:009:58:182
2403699271,19cyclictest4777-21kworker/2:009:46:282
2403699271,14cyclictest4777-21kworker/2:010:46:562
2402999271,17cyclictest0-21swapper/107:43:451
24029992710,2cyclictest9207-21cpu10:15:151
2402499275,10cyclictest12120-21perf07:55:010
110312270,5sleep00-21swapper/011:30:240
2403699262,12cyclictest32-21ksoftirqd/208:07:402
24036992615,3cyclictest4777-21kworker/2:009:04:162
24036992614,3cyclictest4777-21kworker/2:011:56:262
24036992613,3cyclictest4777-21kworker/2:007:58:322
2403699261,13cyclictest4777-21kworker/2:011:35:382
24036992610,3cyclictest4777-21kworker/2:010:26:502
24029992610,3cyclictest25188-21cat12:05:021
2403699252,9cyclictest4777-21kworker/2:007:25:162
24036992516,2cyclictest4777-21kworker/2:011:11:242
24036992512,2cyclictest4777-21kworker/2:007:50:222
2403699251,15cyclictest4777-21kworker/2:008:26:182
24036992510,3cyclictest4777-21kworker/2:010:23:482
2402499257,15cyclictest31571-21munin-run12:20:020
231442250,2sleep00-21swapper/009:30:300
2404199247,3cyclictest0-21swapper/309:28:203
2404199241,11cyclictest0-21swapper/308:33:203
2403699249,3cyclictest4777-21kworker/2:009:42:082
2403699247,5cyclictest4777-21kworker/2:008:35:522
24036992412,3cyclictest4777-21kworker/2:009:14:022
24036992412,2cyclictest4777-21kworker/2:007:16:472
2403699241,12cyclictest4777-21kworker/2:008:30:122
2403699241,10cyclictest4777-21kworker/2:011:32:582
24036992410,3cyclictest4777-21kworker/2:011:03:222
24036992410,3cyclictest4777-21kworker/2:010:09:322
2402499247,9cyclictest0-21swapper/010:20:160
2402499241,10cyclictest0-21swapper/011:10:350
24041992316,4cyclictest4259-21uniq12:30:163
2403699239,4cyclictest4777-21kworker/2:012:20:342
2403699239,3cyclictest4777-21kworker/2:012:26:582
2403699238,3cyclictest4777-21kworker/2:010:35:182
2403699238,3cyclictest4777-21kworker/2:010:35:182
2403699235,9cyclictest4777-21kworker/2:008:49:422
2402499237,6cyclictest0-21swapper/010:37:040
2402499237,6cyclictest0-21swapper/010:37:040
2404199228,6cyclictest27530-21tr12:10:143
2404199227,12cyclictest401ktimersoftd/311:00:213
24041992216,3cyclictest11415-21gltestperf09:05:193
2404199221,17cyclictest4534-21fschecks_count08:50:173
2403699229,2cyclictest4777-21kworker/2:007:35:082
2403699228,4cyclictest4777-21kworker/2:009:53:302
2403699225,8cyclictest4777-21kworker/2:008:51:142
2403699223,16cyclictest27363-21grep10:55:182
2403699221,12cyclictest4777-21kworker/2:008:15:362
2402499227,6cyclictest0-21swapper/009:51:280
2402499226,14cyclictest14313-21awk08:00:010
2402499225,15cyclictest20917-21perf08:15:010
2402499220,5cyclictest0-21swapper/011:55:510
202142220,4sleep339-21rcuc/308:10:293
318002211,2sleep30-21swapper/312:20:143
2404199217,3cyclictest0-21swapper/307:32:353
2404199212,11cyclictest4373-21kernelversion07:35:203
2404199211,11cyclictest22994-21/usr/sbin/munin12:00:143
2403699218,4cyclictest4777-21kworker/2:009:05:082
2403699216,3cyclictest4777-21kworker/2:008:12:122
2403699215,8cyclictest4777-21kworker/2:008:02:222
2403699215,3cyclictest86850irq/129-enp1s0-07:45:022
2403699211,8cyclictest4777-21kworker/2:008:21:062
24036992110,3cyclictest4777-21kworker/2:011:54:462
24036992110,2cyclictest4777-21kworker/2:007:32:082
2402499218,3cyclictest28504-21grep09:45:140
2402499218,2cyclictest14634-21snmpd11:00:290
2402499212,11cyclictest32745-21/usr/sbin/munin09:55:180
2402499212,10cyclictest16739-21/usr/sbin/munin08:05:190
2402499211,9cyclictest10329-21grep10:15:260
24024992115,3cyclictest4146-21unixbench_singl10:00:320
2402499211,4cyclictest27078-21memory09:40:250
238662210,4sleep339-21rcuc/310:45:273
2404199202,10cyclictest17357-21/usr/sbin/munin09:20:253
2404199201,6cyclictest0-21swapper/311:30:143
2404199201,11cyclictest19411-21irqstats11:50:223
2404199201,11cyclictest16403-21/usr/sbin/munin11:45:163
2404199201,11cyclictest16403-21/usr/sbin/munin11:45:163
2404199201,11cyclictest13775-21irqstats09:10:203
2404199201,10cyclictest0-21swapper/310:37:133
2404199201,10cyclictest0-21swapper/310:37:133
2403699205,2cyclictest4777-21kworker/2:008:55:182
2402499208,3cyclictest25249-21sort09:35:270
2402499208,3cyclictest25249-21sort09:35:270
2402499204,6cyclictest0-21swapper/010:13:230
2402499202,15cyclictest25224-21sed10:50:210
2402499202,10cyclictest19557-21/usr/sbin/munin09:25:270
2402499201,6cyclictest0-21swapper/011:16:280
2402499201,11cyclictest7943-21/usr/sbin/munin07:45:160
2402499201,11cyclictest6035-21/usr/sbin/munin12:35:220
2402499201,11cyclictest0-21swapper/011:37:250
54882191,2sleep00-21swapper/007:39:480
2404199194,6cyclictest86550irq/126-enp1s0-10:20:163
2404199191,8cyclictest814-21sshd11:21:423
2404199191,7cyclictest8136-21fschecks_time11:25:203
2404199191,5cyclictest0-21swapper/309:33:253
24041991914,3cyclictest32112-21grep07:25:173
2404199191,15cyclictest28284-21cat09:45:023
2404199191,12cyclictest0-21swapper/310:03:203
2404199191,11cyclictest41-21ksoftirqd/308:00:193
2404199191,10cyclictest7296-21smart_sda12:35:263
2404199191,10cyclictest7181-21kernelversion08:55:203
2404199191,10cyclictest26567-21chrt12:05:283
2404199190,17cyclictest8962-21perf10:15:023
2402499198,3cyclictest1333-21/usr/sbin/munin07:30:160
2402499197,3cyclictest0-21swapper/012:20:100
2402499197,3cyclictest0-21swapper/010:29:190
2402499194,5cyclictest0-21swapper/012:33:050
2402499191,9cyclictest28653-21taskset10:59:020
2402499191,9cyclictest0-21swapper/011:21:220
2402499191,9cyclictest0-21swapper/010:06:300
2402499191,7cyclictest0-21swapper/010:33:400
2402499191,3cyclictest0-21swapper/012:04:430
2402499191,11cyclictest16058-21sshd11:43:430
2402499191,10cyclictest7615-21/usr/sbin/munin11:25:230
2402499191,10cyclictest31517-21/usr/sbin/munin07:25:150
2402499191,10cyclictest29630-21NetworkManager08:34:170
2402499191,10cyclictest0-21swapper/012:05:320
2402499191,10cyclictest0-21swapper/009:16:040
2402499190,8cyclictest0-21swapper/012:25:290
2404199189,2cyclictest14634-21snmpd10:50:233
2404199188,3cyclictest21132-21/usr/sbin/munin08:15:303
2404199187,3cyclictest86850irq/129-enp1s0-10:55:263
2404199181,8cyclictest0-21swapper/312:00:013
2404199181,7cyclictest2496-21uniq08:45:183
2404199181,7cyclictest0-21swapper/310:08:033
2404199181,6cyclictest0-21swapper/309:49:353
2404199181,4cyclictest15356-21sshd10:27:423
24041991813,3cyclictest32409-21gunzip11:05:303
24041991812,4cyclictest1-21systemd09:40:013
24041991812,4cyclictest1-21systemd09:40:013
24041991812,3cyclictest32128-21/usr/sbin/munin08:40:233
2404199181,11cyclictest15826-21sshd11:40:423
2404199181,11cyclictest0-21swapper/311:17:353
2404199181,10cyclictest842-21lldpd12:15:233
2404199181,10cyclictest24955-21idleruntime07:10:223
2402499187,3cyclictest27409-21/usr/sbin/munin12:10:290
2402499187,3cyclictest0-21swapper/011:51:520
2402499181,7cyclictest25410-21grep07:10:280
2402499181,5cyclictest0-21swapper/010:46:130
24024991812,3cyclictest14668-21expr08:00:150
2402499181,10cyclictest5747-21/usr/sbin/munin07:40:220
210522180,4sleep339-21rcuc/310:40:223
81842171,2sleep00-21swapper/008:58:230
302762170,1sleep00-21swapper/008:35:180
2404199172,6cyclictest0-21swapper/307:56:433
2404199172,14cyclictest0-21swapper/307:50:593
2404199171,6cyclictest0-21swapper/309:56:503
2404199171,6cyclictest0-21swapper/309:18:573
2404199171,5cyclictest842-21lldpd10:18:533
24041991712,3cyclictest5812-21cpuspeed_turbos07:40:133
24041991712,3cyclictest28720-21vmstat07:15:293
24041991711,3cyclictest17268-21cut08:05:183
2404199171,13cyclictest15781-21/usr/sbin/munin10:30:163
2404199171,10cyclictest0-21swapper/311:35:293
2404199171,10cyclictest0-21swapper/308:25:273
2402499176,3cyclictest14634-21snmpd07:19:220
2402499171,9cyclictest0-21swapper/008:40:130
24024991712,3cyclictest29964-21expr07:20:200
24024991712,3cyclictest17196-21if_err_enp2s011:45:200
24024991712,3cyclictest17196-21if_err_enp2s011:45:200
2402499171,11cyclictest0-21swapper/008:45:210
105052171,2sleep00-21swapper/009:04:500
2404199169,4cyclictest848-21sssd_nss08:35:203
2404199169,4cyclictest2137-21tune2fs12:25:193
2404199161,6cyclictest0-21swapper/311:10:223
2404199161,2cyclictest0-21swapper/308:20:203
2404199161,13cyclictest9621-21ntp_states09:00:253
2402499161,6cyclictest0-21swapper/008:50:130
24024991612,3cyclictest0-21swapper/008:25:050
24024991611,3cyclictest23418-21ls08:20:150
24024991611,3cyclictest21759-21gltestperf08:15:190
2402499161,13cyclictest0-21swapper/010:40:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional