You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 08:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Nov 24, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
165682236162,23sleep20-21swapper/219:08:572
163992230160,57sleep30-21swapper/319:06:443
163372216184,22sleep00-21swapper/019:05:570
166312203164,25sleep10-21swapper/119:09:441
16757999895,2cyclictest0-21swapper/220:21:402
16757998783,3cyclictest0-21swapper/222:51:192
11942560,4sleep212847-21kworker/2:223:25:192
275212530,1sleep00-21swapper/000:25:190
1675399410,32cyclictest0-21swapper/123:42:041
16753993610,23cyclictest19043-21cpuspeed_turbos21:40:111
16753993610,14cyclictest14877-21fschecks_count21:30:141
16746993611,12cyclictest17761-21awk00:05:000
1675399358,14cyclictest0-21swapper/122:05:221
16753993513,19cyclictest19157-21swap19:15:011
16753993513,18cyclictest26560-21munin-run00:25:011
16753993412,19cyclictest8298-21munin-run22:30:011
16753993412,13cyclictest11217-21/usr/sbin/munin21:20:241
16753993411,13cyclictest836-21runrttasks00:12:231
16753993411,13cyclictest16685-21/usr/sbin/munin21:35:131
16753993410,16cyclictest6759-21ntp_states21:10:211
1674699348,12cyclictest21510-21perf23:00:010
1675399338,14cyclictest12162-21interrupts23:50:191
16753993313,11cyclictest10720-21/usr/sbin/munin22:35:181
16753993312,9cyclictest17948-21latency_hist00:05:021
16763993221,3cyclictest0-21swapper/320:55:363
1675399329,11cyclictest28747-21latency_hist00:30:011
1675399326,13cyclictest30616-21idleruntime-cro19:40:011
16753993216,4cyclictest2995-21ntp_states22:15:221
16753993216,13cyclictest5758-21/usr/sbin/munin23:35:191
16753993216,13cyclictest5758-21/usr/sbin/munin23:35:191
16753993212,12cyclictest14634-21snmpd22:34:531
16753993212,12cyclictest14634-21snmpd22:34:531
16753993211,18cyclictest31540-21fschecks_count00:35:151
16753993210,13cyclictest27053-21/usr/sbin/munin20:45:231
16753993210,12cyclictest848-21sssd_nss22:41:111
16753993210,12cyclictest19524-21/usr/sbin/munin22:55:181
16753993210,11cyclictest792-21avahi-daemon00:15:191
1675399319,19cyclictest25856-21sshd19:25:431
1675399318,5cyclictest86550irq/126-enp1s0-23:22:171
1675399317,10cyclictest12050irq/123-ahci[0023:35:011
1675399317,10cyclictest0-21swapper/123:00:111
16753993112,4cyclictest8882-21needreboot21:15:191
16753993112,16cyclictest15274-21grep22:45:111
16753993110,9cyclictest5462-21diskmemload21:53:371
1675399308,19cyclictest30290-21latency_hist23:20:011
1675399304,13cyclictest0-21swapper/123:10:271
1675399303,17cyclictest5462-21diskmemload22:51:311
16753993021,5cyclictest15122-21smart_sda20:15:261
16753993015,12cyclictest19146-21/usr/sbin/munin00:05:261
16753993011,16cyclictest28980-21/usr/sbin/munin00:30:121
16753993010,17cyclictest29377-21cpuspeed_turbos20:50:131
16757992915,7cyclictest14291-21timerandwakeup22:40:242
1675399298,17cyclictest814-21sshd19:55:451
1675399295,10cyclictest0-21swapper/121:55:121
1675399292,17cyclictest780-21rngd22:12:231
16753992920,5cyclictest22161-21ntp_states21:45:221
16753992914,12cyclictest1456-21irqstats19:45:191
1675399291,15cyclictest496-21jbd2/dm-0-821:08:241
16753992910,16cyclictest21799-21sh19:20:011
16763992820,5cyclictest14291-21timerandwakeup22:40:243
1675399289,5cyclictest9460-21/usr/sbin/munin20:05:161
1675399288,17cyclictest16587-21fschecks_time20:20:171
1675399287,18cyclictest5508-21sort22:20:261
1675399283,12cyclictest0-21swapper/123:50:011
16753992819,5cyclictest2548-21smart_sda21:00:261
16753992811,13cyclictest23349-21sshd20:35:181
16753992810,7cyclictest665-21tune2fs23:25:161
1675399278,4cyclictest12440-21if_err_enp1s020:10:191
1675399274,13cyclictest0-21swapper/121:00:021
16753992719,5cyclictest14614-21ntp_states23:55:221
16753992714,10cyclictest20284-21date20:30:011
16753992714,10cyclictest20284-21date20:30:011
16753992713,11cyclictest12287-21/usr/sbin/munin21:25:251
16753992713,11cyclictest12287-21/usr/sbin/munin21:25:251
16753992711,6cyclictest29274-21users22:00:271
16753992710,15cyclictest0-21swapper/120:05:021
1674699271,12cyclictest7039-21perf20:00:010
1675399268,5cyclictest24855-21/usr/sbin/munin20:40:171
16753992618,5cyclictest3652-21/usr/sbin/munin19:50:181
16753992617,6cyclictest24604-21hddtemp_smartct23:05:161
1674699261,12cyclictest30298-21munin-run23:20:010
1675799251,4cyclictest0-21swapper/221:45:122
16753992516,6cyclictest30830-21/usr/sbin/munin19:40:141
16753992516,6cyclictest27013-21/usr/sbin/munin19:30:161
16753992516,6cyclictest20457-21/usr/sbin/munin20:30:141
16753992515,7cyclictest22033-21/usr/sbin/munin19:20:251
16753992515,7cyclictest22033-21/usr/sbin/munin19:20:251
1674699247,14cyclictest11136-21idleruntime-cro23:50:010
1674699241,12cyclictest1046-21idleruntime-cro21:00:010
272372230,1sleep011-21rcuc/020:45:120
1676399236,3cyclictest26458-21sshd00:23:563
1675799231,6cyclictest0-21swapper/223:43:262
16746992317,3cyclictest19200-21gunzip00:05:270
1674699231,11cyclictest31233-21awk20:55:010
1674699231,11cyclictest11992grep19:45:170
1674699230,11cyclictest9249-21idleruntime-cro20:05:020
1676399226,8cyclictest16787-21sshd22:46:123
1676399226,4cyclictest0-21swapper/300:36:483
1676399222,10cyclictest30949-21munin-run00:35:013
1676399222,10cyclictest27460-21munin-run22:00:013
1676399221,3cyclictest0-21swapper/321:18:083
1676399221,18cyclictest11869-21pmu-power22:35:243
1675799221,7cyclictest0-21swapper/222:38:152
1675799221,7cyclictest0-21swapper/222:24:062
1674699223,10cyclictest28153-21sed23:15:020
1674699220,11cyclictest22174-21basename00:15:010
1676399218,7cyclictest8303-21awk22:30:013
1676399215,8cyclictest24403-21sshd20:37:183
1676399215,8cyclictest18257-21/usr/sbin/munin20:25:213
1676399215,8cyclictest18257-21/usr/sbin/munin20:25:213
1676399212,10cyclictest21806-21munin-run19:20:013
16763992114,4cyclictest28027-21df_inode22:00:153
1676399211,3cyclictest0-21swapper/323:05:183
1676399211,11cyclictest21016-21/usr/sbin/munin00:10:223
1676399211,10cyclictest0-21swapper/300:08:593
1675799211,7cyclictest0-21swapper/221:54:242
1675799211,7cyclictest0-21swapper/221:21:552
1675799211,6cyclictest0-21swapper/223:52:422
1675799211,6cyclictest0-21swapper/221:43:492
1674699214,15cyclictest26216-21perf19:30:010
1674699213,9cyclictest0-21swapper/022:00:140
1674699211,8cyclictest9824-21runrttasks21:19:410
1674699211,5cyclictest0-21swapper/022:23:220
1676399206,4cyclictest3492-21/usr/sbin/munin21:05:133
1676399201,5cyclictest0-21swapper/300:25:213
1676399201,3cyclictest0-21swapper/323:40:443
1676399201,11cyclictest5695-21/usr/sbin/munin21:10:173
1676399201,11cyclictest24855-21/usr/sbin/munin20:40:233
1676399201,11cyclictest14634-21snmpd21:34:423
1676399201,11cyclictest14634-21snmpd20:06:503
1676399201,11cyclictest10901-21irqstats21:20:193
1676399201,10cyclictest16835-21ntp_states00:00:223
1676399201,10cyclictest0-21swapper/323:45:443
1675799201,7cyclictest30256-21switchtime00:30:262
1675799201,7cyclictest0-21swapper/222:59:042
1675799201,7cyclictest0-21swapper/221:05:182
1675799201,6cyclictest28107-21perf23:15:022
1675799201,6cyclictest25499-21memory00:20:182
1675799201,6cyclictest0-21swapper/223:59:152
1675799201,6cyclictest0-21swapper/223:15:212
1675799201,5cyclictest0-21swapper/222:25:032
16757992013,4cyclictest15416-21unixbench_singl20:15:282
16757992012,4cyclictest24670-21latency_hist20:40:012
1674699209,3cyclictest22389-21/usr/sbin/munin00:15:290
1674699204,6cyclictest0-21swapper/021:51:320
1674699201,3cyclictest0-21swapper/023:55:210
1674699201,13cyclictest0-21swapper/021:25:130
1674699201,13cyclictest0-21swapper/021:25:130
1674699201,11cyclictest21901-21irqstats21:45:180
1674699201,11cyclictest14634-21snmpd20:39:320
1674699201,11cyclictest0-21swapper/022:25:080
1674699201,10cyclictest5462-21diskmemload21:35:250
1674699201,10cyclictest21721-21/usr/sbin/munin23:00:140
1674699201,10cyclictest0-21swapper/023:08:470
1674699200,10cyclictest1413-21kworker/u8:023:36:150
1674699200,10cyclictest1413-21kworker/u8:023:36:140
263942191,2sleep20-21swapper/221:55:202
21912190,4sleep339-21rcuc/321:00:213
1676399199,3cyclictest16685-21/usr/sbin/munin21:35:163
1676399199,2cyclictest1928-21/usr/sbin/munin22:15:213
1676399196,3cyclictest27448-21sort23:10:263
1676399194,4cyclictest14211-21sed23:55:163
1676399192,9cyclictest14634-21snmpd19:44:223
1676399192,7cyclictest30564-21hddtemp_smartct22:05:153
1676399191,9cyclictest12726-21chrt20:10:233
1676399191,5cyclictest5040-21memory22:20:203
1676399191,5cyclictest30528-21/usr/sbin/munin23:20:163
16763991914,3cyclictest28550-21grep20:45:263
16763991913,3cyclictest18959-21munin-node19:15:013
1676399191,11cyclictest14634-21snmpd23:04:223
1676399191,11cyclictest0-21swapper/320:05:013
1676399191,10cyclictest984-21gltestperf23:25:173
1676399191,10cyclictest21088-21/usr/sbin/munin21:45:163
1676399191,10cyclictest17326-21/usr/sbin/munin22:50:173
1676399191,10cyclictest12515-21expr21:25:143
1676399191,10cyclictest12515-21expr21:25:143
1675799191,8cyclictest5396-21gltestperf23:35:152
1675799191,8cyclictest5396-21gltestperf23:35:152
1675799191,7cyclictest0-21swapper/219:15:042
1675799191,6cyclictest22044-21kworker/u8:000:35:152
1675799191,6cyclictest0-21swapper/222:05:272
1675799191,5cyclictest24447-21cat00:20:012
1675799191,5cyclictest22638-21tail19:20:172
1675799191,5cyclictest22638-21tail19:20:162
1675799191,5cyclictest0-21swapper/223:45:572
1675799191,4cyclictest32214-21cpuspeed_turbos22:10:132
1675799191,4cyclictest0-21swapper/222:01:232
16757991913,3cyclictest18322-21df_inode00:05:172
16757991912,4cyclictest25697-21unixbench-2d19:25:272
1674699198,3cyclictest5462-21diskmemload22:49:590
1674699196,5cyclictest0-21swapper/022:05:280
1674699196,5cyclictest0-21swapper/021:43:420
1674699196,4cyclictest0-21swapper/022:54:190
1674699192,9cyclictest0-21swapper/022:36:050
1674699191,9cyclictest11442-21kworker/u8:121:10:210
1674699191,6cyclictest0-21swapper/023:24:160
1674699191,3cyclictest9821-21kworker/u8:221:30:150
1674699191,3cyclictest0-21swapper/021:24:200
1674699191,3cyclictest0-21swapper/020:10:170
1674699191,11cyclictest0-21swapper/022:32:250
1674699191,11cyclictest0-21swapper/022:32:250
1674699191,10cyclictest5462-21diskmemload00:39:240
1674699191,10cyclictest30605-21sshd00:30:450
1674699191,10cyclictest14634-21snmpd20:29:300
1674699191,10cyclictest14634-21snmpd20:29:300
1674699191,10cyclictest14634-21snmpd19:54:240
1674699191,10cyclictest0-21swapper/023:53:220
1674699191,10cyclictest0-21swapper/022:12:350
16746991910,3cyclictest25922-21awk00:20:240
160462191,2sleep00-21swapper/020:20:080
11962190,3sleep29-21rcu_preempt19:45:172
9762181,2sleep20-21swapper/220:59:272
75602181,2sleep20-21swapper/221:13:482
1676399184,4cyclictest23289-21/usr/sbin/munin21:50:143
1676399182,8cyclictest19559-21hddtemp_smartct21:40:163
1676399181,9cyclictest15822-21sh20:20:013
1676399181,8cyclictest14634-21snmpd23:35:053
1676399181,8cyclictest14634-21snmpd23:35:053
1676399181,7cyclictest24903-21hddtemp_smartct19:25:183
1676399181,6cyclictest0-21swapper/323:34:083
1676399181,2cyclictest0-21swapper/323:15:363
16763991812,3cyclictest22994-21head00:15:203
1675799185,8cyclictest30544-40dnf19:39:322
1675799183,6cyclictest5462-21diskmemload23:23:062
1675799181,7cyclictest0-21swapper/222:47:452
1675799181,7cyclictest0-21swapper/221:29:292
1675799181,7cyclictest0-21swapper/221:29:292
1675799181,6cyclictest0-21swapper/223:31:132
1675799181,6cyclictest0-21swapper/223:08:072
1675799181,6cyclictest0-21swapper/223:00:192
1675799181,6cyclictest0-21swapper/222:34:022
1675799181,6cyclictest0-21swapper/222:34:022
1675799181,6cyclictest0-21swapper/221:39:402
1675799181,6cyclictest0-21swapper/221:32:282
1675799181,6cyclictest0-21swapper/220:50:162
1675799181,6cyclictest0-21swapper/200:14:372
16757991812,3cyclictest9035-21grep21:15:222
16757991812,3cyclictest327-21munin-plugin-st19:45:012
1675799180,6cyclictest0-21swapper/222:19:352
1674699185,5cyclictest0-21swapper/021:58:270
1674699182,6cyclictest0-21swapper/022:15:120
16746991812,3cyclictest3384-21kernelversion23:30:190
16746991811,4cyclictest2437-21cat23:30:000
16746991811,4cyclictest1-21systemd21:00:090
1674699181,11cyclictest0-21swapper/019:40:160
101622181,2sleep30-21swapper/322:30:583
101622181,2sleep30-21swapper/322:30:583
1676399174,3cyclictest311-21hwlatdetect22:10:183
1676399172,5cyclictest20115-21tail22:55:173
1676399171,6cyclictest22033-21/usr/sbin/munin19:20:213
1676399171,6cyclictest22033-21/usr/sbin/munin19:20:213
1676399171,14cyclictest0-21swapper/319:31:183
16763991711,3cyclictest5056-21/usr/sbin/munin19:55:173
16763991711,3cyclictest3516-21gltestperf19:50:163
16763991711,3cyclictest30735-21date19:40:023
16763991711,3cyclictest11896-21cut23:50:163
16763991710,4cyclictest16738-21sh20:20:183
1675799179,5cyclictest2393-21pmu-power21:00:242
1675799177,8cyclictest7028-21munin-plugin-st20:00:012
1675799176,3cyclictest12050irq/123-ahci[0020:05:462
1675799172,6cyclictest14634-21snmpd20:31:262
1675799172,4cyclictest0-21swapper/220:40:012
1675799171,5cyclictest0-21swapper/200:28:162
1675799171,5cyclictest0-21swapper/200:02:462
16757991711,3cyclictest18257-21/usr/sbin/munin20:25:222
16757991711,3cyclictest18257-21/usr/sbin/munin20:25:212
1674699178,2cyclictest26432-21/usr/sbin/munin19:30:250
1674699172,9cyclictest0-21swapper/023:40:340
1674699171,7cyclictest0-21swapper/022:40:150
1674699171,6cyclictest24964-21cpuspeed_turbos20:40:130
16746991711,3cyclictest16946-21/usr/sbin/munin19:10:170
1674699170,5cyclictest0-21swapper/019:35:190
89972161,2sleep20-21swapper/220:02:082
221732160,2sleep00-21swapper/020:31:520
1676399162,11cyclictest566-21/usr/sbin/munin19:45:173
1676399161,6cyclictest0-21swapper/320:50:173
16763991611,3cyclictest20639-21cstates20:30:123
1675799163,3cyclictest86850irq/129-enp1s0-19:10:192
16757991611,3cyclictest3888-21ntp_kernel_err19:50:202
16757991611,3cyclictest27608-21perl20:45:142
16757991611,3cyclictest12562-21cat20:10:202
1675799161,12cyclictest26589-21tr19:30:132
1674699161,14cyclictest0-21swapper/021:06:250
16746991611,3cyclictest21277-21cat19:15:240
1674699160,4cyclictest14462-21grep20:15:180
1674699151,6cyclictest22033-21/usr/sbin/munin19:20:130
1674699151,6cyclictest22033-21/usr/sbin/munin19:20:130
16746991510,3cyclictest10554-21open_inodes20:05:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional