You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 11:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Dec 01, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
275662234160,60sleep00-21swapper/019:05:400
275892233164,57sleep30-21swapper/319:05:593
276492203164,25sleep10-21swapper/119:06:461
259402203166,24sleep20-21swapper/219:05:062
445921300,3sleep02799599cyclictest20:40:180
110002560,1sleep10-21swapper/119:40:261
235892550,4sleep341-21ksoftirqd/323:50:213
57952520,4sleep339-21rcuc/321:55:283
61012490,1sleep10-21swapper/119:30:181
28006994744,2cyclictest0-21swapper/220:24:292
2800299371,24cyclictest12256-21diskstats19:45:131
2801199321,23cyclictest0-21swapper/323:15:143
2800299321,6cyclictest0-21swapper/121:59:011
2801199302,14cyclictest26587-40dnf23:57:343
28011992923,4cyclictest0-21swapper/322:25:223
28011992923,4cyclictest0-21swapper/322:25:223
28002992910,4cyclictest28882-21systemd-journal00:03:221
2799599291,13cyclictest1320-21awk20:35:010
144442290,4sleep121-21rcuc/123:30:161
144442290,4sleep121-21rcuc/123:30:161
28011992821,4cyclictest15183-21ntp_states19:50:243
2801199281,14cyclictest26626-21grep20:15:263
2801199281,14cyclictest26626-21grep20:15:263
267272281,23sleep026738-21/usr/sbin/munin22:45:160
28002992711,4cyclictest4317-21sed23:05:281
2799599272,13cyclictest8285-21munin-run00:35:010
27995992710,14cyclictest1977-21idleruntime-cro21:50:010
96262260,2chrt0-21swapper/119:40:021
28011992611,3cyclictest22327-21df22:35:163
2800699266,10cyclictest0-21swapper/220:06:162
2800699263,16cyclictest29708-21kworker/2:121:50:272
28002992620,3cyclictest12016-40dnf20:56:231
2800299261,15cyclictest0-21swapper/123:50:051
2800299261,11cyclictest0-21swapper/123:38:311
2800299261,11cyclictest0-21swapper/100:34:261
2800299260,17cyclictest0-21swapper/122:35:271
2801199251,17cyclictest0-21swapper/321:45:183
2801199251,17cyclictest0-21swapper/321:27:023
2801199251,15cyclictest0-21swapper/322:00:203
2801199251,13cyclictest1462-21runrttasks00:17:363
2801199251,12cyclictest0-21swapper/322:30:153
2800699250,5cyclictest0-21swapper/221:25:012
2800699250,5cyclictest0-21swapper/221:25:012
28002992519,3cyclictest6938-21diskstats22:00:141
2801199249,3cyclictest32053-21smart_sda22:55:253
2801199242,11cyclictest39-21rcuc/300:10:223
28011992419,3cyclictest14634-21snmpd19:49:273
28011992418,3cyclictest28595-21/usr/sbin/munin22:50:263
2801199241,12cyclictest0-21swapper/300:35:063
2801199241,11cyclictest21942-21gunzip20:05:233
2801199241,11cyclictest15739-21diskstats22:20:143
2801199241,10cyclictest21282-21taskset21:20:013
2800699243,12cyclictest0-21swapper/219:17:432
2800299242,12cyclictest30163-21munin-run19:15:011
28002992418,3cyclictest28228-21latency20:20:211
28002992415,5cyclictest0-21swapper/121:26:341
2800299241,20cyclictest19142-21memory23:40:181
2800299241,17cyclictest0-21swapper/123:15:141
2800299241,17cyclictest0-21swapper/122:57:361
2800299241,17cyclictest0-21swapper/121:21:211
2800299241,17cyclictest0-21swapper/121:21:211
2800299241,14cyclictest0-21swapper/100:11:241
2800299241,11cyclictest10763-21latency_hist22:10:001
2800299241,10cyclictest0-21swapper/121:15:401
2800299240,16cyclictest86750irq/128-enp1s0-22:26:361
2800299240,16cyclictest86750irq/128-enp1s0-22:26:361
2799599241,5cyclictest0-21swapper/022:55:200
2799599241,5cyclictest0-21swapper/022:05:260
2801199239,3cyclictest6648-21df_inode00:30:163
2801199239,3cyclictest27197-21irqstats22:45:193
2801199239,3cyclictest10630-21memory19:40:203
2801199238,3cyclictest9608-21irqstats22:05:193
2801199231,9cyclictest0-21swapper/319:18:093
28011992319,3cyclictest0-21swapper/323:35:213
28011992319,2cyclictest28198-21/usr/sbin/munin19:10:203
28011992319,2cyclictest16729-21diskmemload23:30:173
28011992319,2cyclictest16729-21diskmemload23:30:173
2801199231,5cyclictest30298-21cpuspeed_turbos21:40:133
2801199231,13cyclictest0-21swapper/322:18:193
2801199231,12cyclictest0-21swapper/300:02:353
2801199231,11cyclictest11963-21df23:25:143
2801199231,10cyclictest3649-21irqstats23:05:203
2801199231,10cyclictest2514-21gltestperf00:20:153
2801199231,10cyclictest0-21swapper/323:20:163
2801199231,10cyclictest0-21swapper/321:54:133
2800699237,9cyclictest0-21swapper/221:45:462
2800299238,3cyclictest26723-21df22:45:151
28002992320,2cyclictest0-21swapper/123:46:431
28002992320,2cyclictest0-21swapper/120:45:551
28002992319,3cyclictest0-21swapper/122:41:501
28002992319,3cyclictest0-21swapper/120:04:271
28002992318,2cyclictest5185-21cat19:30:011
28002992317,3cyclictest2543-21/usr/sbin/munin21:50:151
2800299231,16cyclictest0-21swapper/121:30:231
2800299231,12cyclictest13111-21unixbench-2d23:25:261
2800299231,11cyclictest0-21swapper/121:39:111
2800299231,10cyclictest603-21strings00:15:161
2800299231,10cyclictest15821-21ntp_states21:05:231
2800299231,10cyclictest15554-21cpuspeed_turbos22:20:121
2800299230,10cyclictest12961-21sh22:15:001
2799599235,15cyclictest26809-21awk00:00:010
2801199227,3cyclictest24189-21/usr/sbin/munin22:40:133
2801199227,3cyclictest15565-21irqstats21:05:203
2801199224,5cyclictest39-21rcuc/320:30:243
28011992218,2cyclictest39-21rcuc/321:35:183
2801199221,12cyclictest0-21swapper/323:10:523
2801199221,12cyclictest0-21swapper/321:22:233
2801199221,12cyclictest0-21swapper/321:22:233
2801199221,10cyclictest20328-21sshd20:03:093
2801199221,10cyclictest18706-21sshd21:12:223
2801199221,10cyclictest12158-21proc_pri22:10:233
2800699223,7cyclictest27079-21latency_hist20:20:012
2800699223,7cyclictest27079-21latency_hist20:20:002
2800699221,7cyclictest12050irq/123-ahci[0023:55:022
2800299227,10cyclictest0-21swapper/123:56:251
2800299222,10cyclictest0-21swapper/123:20:201
28002992219,2cyclictest0-21swapper/120:41:131
28002992218,3cyclictest0-21swapper/121:11:061
28002992216,3cyclictest16978-21hddtemp_smartct19:55:161
2800299221,18cyclictest0-21swapper/122:30:231
2800299221,15cyclictest0-21swapper/122:50:141
2800299221,15cyclictest0-21swapper/100:21:191
2800299221,10cyclictest4923-21uname00:25:181
2800299221,10cyclictest22995-21runrttasks20:10:141
2800299221,10cyclictest0-21swapper/122:15:521
2800299221,10cyclictest0-21swapper/121:48:281
2800299221,10cyclictest0-21swapper/121:41:221
2800299221,10cyclictest0-21swapper/100:36:241
2799599221,4cyclictest0-21swapper/000:23:090
157842220,3sleep10-21swapper/119:51:231
2801199217,4cyclictest14634-21snmpd19:34:243
2801199217,3cyclictest842-21lldpd19:24:193
2801199217,2cyclictest19077-21latency23:40:193
2801199216,3cyclictest8431-21memory19:35:193
2801199213,9cyclictest41-21ksoftirqd/320:37:403
2801199211,11cyclictest0-21swapper/320:27:343
2801199211,10cyclictest16729-21diskmemload23:49:293
2801199211,10cyclictest0-21swapper/300:05:483
2800699216,9cyclictest0-21swapper/222:45:282
2800699214,5cyclictest780-21rngd23:35:592
2800699214,10cyclictest29708-21kworker/2:122:40:202
2800699211,3cyclictest0-21swapper/222:19:452
2800699211,13cyclictest0-21swapper/200:15:162
2800299217,3cyclictest26577-21cat20:15:261
2800299217,3cyclictest26577-21cat20:15:261
28002992117,3cyclictest0-21swapper/120:33:121
2800299211,15cyclictest0-21swapper/123:12:281
2800299211,10cyclictest0-21swapper/123:01:261
2799599213,15cyclictest31483-21idleruntime-cro20:30:010
2799599212,11cyclictest27319-21/usr/sbin/munin20:20:200
2801199202,5cyclictest2614-21munin-run23:05:013
2801199202,16cyclictest41-21ksoftirqd/321:00:193
2801199202,16cyclictest28573-21mailstats20:20:263
2801199201,8cyclictest0-21swapper/300:25:223
2801199201,7cyclictest10345-21/usr/sbin/munin20:55:143
2800699209,3cyclictest86850irq/129-enp1s0-21:25:182
2800699207,3cyclictest0-21swapper/221:02:342
2800699206,5cyclictest14634-21snmpd19:25:222
2800699205,6cyclictest87950irq/132-enp2s0-23:25:182
2800699205,4cyclictest0-21swapper/220:37:282
2800699204,8cyclictest12050irq/123-ahci[0019:45:012
2800699202,7cyclictest0-21swapper/223:57:342
2800699201,8cyclictest4805-21sa123:10:022
28006992014,3cyclictest17426-21grep21:10:162
28006992012,5cyclictest11633-21sed22:10:162
2800299208,2cyclictest0-21swapper/119:15:201
2800299201,9cyclictest0-21swapper/100:06:571
2800299201,2cyclictest0-21swapper/121:02:211
2800299201,2cyclictest0-21swapper/120:37:421
2800299201,17cyclictest0-21swapper/119:24:141
2800299201,10cyclictest0-21swapper/120:26:151
2800299200,18cyclictest0-21swapper/120:05:151
2799599203,8cyclictest0-21swapper/023:01:010
2799599202,15cyclictest6072-40dnf21:56:310
2799599201,11cyclictest899-21/usr/sbin/munin19:20:280
2799599201,11cyclictest20423-21/usr/sbin/munin23:45:140
2799599201,10cyclictest5175-21ntp_states00:25:220
2801199192,4cyclictest8822-21hddtemp_smartct20:50:183
2801199191,9cyclictest16729-21diskmemload21:33:053
2801199191,6cyclictest4626-21unixbench_multi19:25:243
2801199191,6cyclictest3754-21/usr/sbin/munin20:40:193
2800699197,2cyclictest0-21swapper/200:10:162
2800699195,3cyclictest0-21swapper/220:04:162
2800699192,5cyclictest23121-21ls22:35:252
2800699191,7cyclictest0-21swapper/200:02:312
28006991913,3cyclictest8351-21pmu-power23:15:242
2800699191,12cyclictest0-21swapper/221:55:582
2800699191,10cyclictest6030-21ls19:30:172
2800299191,6cyclictest9063-21memory20:50:211
2799599194,7cyclictest0-21swapper/000:15:120
2799599193,13cyclictest25919-21tr21:30:140
2799599191,8cyclictest0-21swapper/019:15:080
2799599191,6cyclictest0-21swapper/023:20:010
2799599191,6cyclictest0-21swapper/021:51:500
27995991913,3cyclictest31744-21users21:40:270
27995991913,3cyclictest24892-21perf20:15:010
2799599191,10cyclictest8509-21/usr/sbin/munin00:35:260
2799599191,10cyclictest8149-21/usr/sbin/munin20:50:140
2799599191,10cyclictest3120-21/usr/sbin/munin19:25:110
2799599191,10cyclictest30266-21sshd22:51:230
2799599191,10cyclictest1542-21/usr/sbin/munin20:35:210
2799599191,10cyclictest14634-21snmpd22:32:490
2799599191,10cyclictest0-21swapper/023:29:020
2799599190,16cyclictest16082-21munin-run19:55:010
254172191,2sleep00-21swapper/021:28:480
109332190,4sleep232-21ksoftirqd/220:55:182
2801199181,4cyclictest0-21swapper/319:55:183
28011991812,3cyclictest22921-21/usr/sbin/munin20:10:223
2800699187,6cyclictest86750irq/128-enp1s0-23:22:272
2800699182,10cyclictest0-21swapper/222:53:032
2800699181,9cyclictest31818-21chrt21:41:142
2800699181,8cyclictest0-21swapper/222:28:212
2800699181,8cyclictest0-21swapper/222:28:212
2800699181,7cyclictest0-21swapper/223:30:192
2800699181,7cyclictest0-21swapper/223:30:192
2800699181,7cyclictest0-21swapper/219:35:232
2800699181,6cyclictest0-21swapper/220:52:292
28006991814,3cyclictest32-21ksoftirqd/223:49:322
28006991814,3cyclictest32-21ksoftirqd/219:46:102
28006991814,2cyclictest86750irq/128-enp1s0-22:04:302
28006991812,4cyclictest26864-21ls21:30:242
28006991811,4cyclictest18674-21fschecks_count23:40:152
2800699181,11cyclictest0-21swapper/219:12:092
2800699181,11cyclictest0-21swapper/200:08:452
2800699181,10cyclictest0-21swapper/222:21:202
2799599188,8cyclictest86550irq/126-enp1s0-23:15:130
2799599182,7cyclictest24189-21/usr/sbin/munin22:40:200
2799599181,9cyclictest23740-21sshd22:37:230
2799599181,7cyclictest14247-21sort23:30:150
2799599181,7cyclictest14247-21sort23:30:150
2799599181,7cyclictest0-21swapper/000:04:470
2799599181,6cyclictest0-21swapper/021:38:360
2799599181,4cyclictest11369-21netstat20:55:220
27995991812,3cyclictest20002-21grep20:00:260
27995991812,3cyclictest12937-21nfs4_client19:45:190
2799599181,14cyclictest3441-21grep23:05:180
231102181,2sleep00-21swapper/021:22:260
231102181,2sleep00-21swapper/021:22:260
182032181,2sleep00-21swapper/019:59:130
177832181,2sleep00-21swapper/023:37:260
77842171,2sleep00-21swapper/020:48:260
315192170,2sleep00-21swapper/000:10:580
29932170,4sleep22800699cyclictest19:25:002
2801199171,3cyclictest0-21swapper/320:45:363
2800699178,2cyclictest86750irq/128-enp1s0-19:50:222
2800699174,5cyclictest19314-21cpuspeed_turbos21:15:142
2800699172,7cyclictest86750irq/128-enp1s0-22:55:222
2800699171,8cyclictest9578-21ntpq00:35:242
2800699171,7cyclictest4879-21pmu-power20:40:242
2800699171,6cyclictest0-21swapper/200:23:162
2800699171,3cyclictest0-21swapper/223:13:392
2800699171,13cyclictest30137-21gltestperf20:25:172
2800699171,11cyclictest0-21swapper/200:32:022
2800699171,11cyclictest0-21swapper/200:29:592
2800699171,10cyclictest0-21swapper/221:37:172
2800699171,10cyclictest0-21swapper/220:45:322
2799599178,8cyclictest86550irq/126-enp1s0-22:15:140
2799599173,5cyclictest0-21swapper/023:10:020
2799599171,8cyclictest784-21NetworkManager22:22:520
2799599171,7cyclictest18660-21ntp_states22:25:230
2799599171,7cyclictest18660-21ntp_states22:25:230
2799599171,6cyclictest0-21swapper/023:40:440
2799599171,6cyclictest0-21swapper/022:00:520
27995991712,3cyclictest22887-21ls23:50:140
2799599171,15cyclictest0-21swapper/021:16:530
2799599171,14cyclictest0-21swapper/019:35:080
27995991711,3cyclictest16956-21/usr/sbin/munin21:10:250
27995991711,3cyclictest13486-21cat21:00:190
16942171,13sleep20-21swapper/223:00:232
161212171,12sleep016309-21unixbench_singl21:05:280
292842160,2sleep00-21swapper/000:05:400
2800699167,8cyclictest24898-21awk20:15:002
2800699165,3cyclictest86850irq/129-enp1s0-22:30:262
2800699164,3cyclictest0-21swapper/222:05:282
2800699164,3cyclictest0-21swapper/219:55:142
2800699164,10cyclictest21477-21kworker/u8:020:30:162
28006991611,3cyclictest15063-21cat21:05:152
2799599165,3cyclictest7-21ksoftirqd/022:10:030
2799599161,7cyclictest842-21lldpd19:11:470
27995991612,3cyclictest0-21swapper/019:43:430
2799599161,14cyclictest0-21swapper/020:18:010
2799599161,14cyclictest0-21swapper/020:18:010
27995991510,3cyclictest5304-21date19:30:020
27995991510,3cyclictest22040-21switchtime20:05:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional