You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 14:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Dec 05, 2025 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
275302237164,24sleep20-21swapper/207:07:572
276512227195,21sleep10-21swapper/107:09:291
274862206168,25sleep00-21swapper/007:07:230
273512200163,24sleep30-21swapper/307:05:363
303621740,4sleep12778599cyclictest12:00:241
962821540,4sleep12778599cyclictest11:05:041
1218121480,3sleep32779599cyclictest08:15:013
27789995148,2cyclictest0-21swapper/208:47:342
2777799337,12cyclictest0-21swapper/010:10:010
301282310,5sleep12778599cyclictest10:35:251
2777799301,13cyclictest9817-21munin-run08:10:010
2777799301,13cyclictest9817-21munin-run08:10:010
2777799284,11cyclictest2824-21awk10:50:010
2777799270,13cyclictest6389-21perf07:55:010
2777799263,11cyclictest27011-21awk11:45:010
2777799261,13cyclictest9412-21munin-run11:05:010
2778999253,5cyclictest0-21swapper/211:45:082
2778999252,7cyclictest0-21swapper/212:30:282
2778599252,12cyclictest19772-21munin-run10:15:011
2778599252,12cyclictest10971-21munin-run09:55:011
2777799255,17cyclictest8775-21munin-run09:50:010
2778999241,9cyclictest31663-21kworker/2:109:58:452
2778599242,19cyclictest4015-21df12:05:151
2778599242,19cyclictest13746-21diskstats10:00:171
272332240,4sleep341-21ksoftirqd/309:15:233
2779599239,6cyclictest8076-21/usr/sbin/munin12:15:263
2778999232,10cyclictest31663-21kworker/2:109:30:432
2778599234,9cyclictest86850irq/129-enp1s0-12:10:231
2778599231,2cyclictest29958-21munin-run07:15:011
2778599231,19cyclictest11536-21df_inode09:55:151
2778599231,15cyclictest0-21swapper/111:46:001
2778599230,5cyclictest0-21swapper/111:02:161
2778599230,5cyclictest0-21swapper/110:56:081
2778599230,18cyclictest0-21swapper/109:19:271
2778599230,17cyclictest0-21swapper/111:31:161
2777799234,3cyclictest9555-21fschecks_time09:50:150
2777799232,13cyclictest0-21swapper/010:55:140
27789992215,4cyclictest2949-21idleruntime-cro07:40:022
2778599228,12cyclictest22493-21kworker/u8:211:40:111
2778599222,11cyclictest23264-21ntp_states10:20:211
2778599221,9cyclictest0-21swapper/110:48:371
2778599221,2cyclictest0-21swapper/112:35:101
2778599221,19cyclictest0-21swapper/112:25:211
2778599221,19cyclictest0-21swapper/110:30:271
2778599221,19cyclictest0-21swapper/108:26:401
2778599221,11cyclictest29754-21users09:20:251
2778599221,10cyclictest935-21chrt11:56:301
2778599220,16cyclictest0-21swapper/110:41:421
2777799226,4cyclictest836-21runrttasks08:35:580
2777799224,12cyclictest86750irq/128-enp1s0-09:41:370
2777799222,12cyclictest27733-21kworker/u8:209:33:420
2777799220,12cyclictest0-21swapper/012:20:180
2777799220,11cyclictest0-21swapper/009:38:510
2778999211,5cyclictest0-21swapper/211:43:072
27789992114,4cyclictest29633-21tr10:35:192
2778999211,13cyclictest0-21swapper/212:21:012
2778599219,2cyclictest19372-21missed_timers11:25:191
2778599215,3cyclictest14706-21hddtemp_smartct11:15:171
2778599211,8cyclictest0-21swapper/109:48:171
2778599211,8cyclictest0-21swapper/109:26:331
27785992117,3cyclictest0-21swapper/108:55:151
2778599211,18cyclictest0-21swapper/111:37:401
2778599211,18cyclictest0-21swapper/108:02:471
2778599211,10cyclictest16582-21diskstats11:20:151
2778599211,10cyclictest0-21swapper/110:28:191
2778599210,16cyclictest0-21swapper/110:06:341
2777799217,3cyclictest25293-21sed09:10:220
2777799217,3cyclictest25293-21sed09:10:220
2777799216,6cyclictest0-21swapper/009:27:570
2777799214,7cyclictest0-21swapper/010:38:190
2777799212,11cyclictest22196-21/usr/sbin/munin10:20:240
2777799212,11cyclictest19418-21kworker/u8:009:22:430
2777799212,10cyclictest3014-21sed12:00:240
2777799211,12cyclictest32107-21/usr/sbin/munin07:25:240
2777799211,11cyclictest31044-21sshd11:50:300
2777799211,11cyclictest20316-21/usr/sbin/munin08:55:110
2779599202,10cyclictest16233-21/usr/sbin/munin11:20:233
2779599201,6cyclictest10622-21/usr/sbin/munin12:20:143
27795992015,3cyclictest9197-21idleruntime-cro08:05:013
2779599201,12cyclictest0-21swapper/312:27:083
2779599201,11cyclictest6596-21/usr/sbin/munin07:55:163
2779599201,11cyclictest32570-21/usr/sbin/munin09:30:203
2779599201,11cyclictest18869-21ntp_states10:10:223
2779599201,11cyclictest15602-21/usr/sbin/munin10:05:163
2779599201,10cyclictest3039-21/usr/sbin/munin10:50:123
2778599208,5cyclictest23-21ksoftirqd/107:33:461
2778599205,9cyclictest0-21swapper/111:10:521
2778599201,7cyclictest0-21swapper/112:31:221
27785992016,3cyclictest0-21swapper/107:21:371
27785992015,3cyclictest0-21swapper/107:40:131
27785992012,6cyclictest0-21swapper/108:23:511
2778599201,18cyclictest0-21swapper/108:10:241
2778599201,17cyclictest0-21swapper/111:52:511
2778599201,17cyclictest0-21swapper/108:16:431
2778599201,17cyclictest0-21swapper/107:25:191
2778599201,16cyclictest0-21swapper/108:36:371
2778599200,18cyclictest0-21swapper/107:38:181
2777799208,3cyclictest27733-21kworker/u8:209:55:260
2777799207,4cyclictest0-21swapper/011:27:320
2777799206,3cyclictest27396-21interrupts10:30:180
2777799203,11cyclictest17257-21kworker/u8:210:53:320
27777992013,4cyclictest4670-21strings12:05:210
2777799201,11cyclictest5670-21sshd07:49:050
27777992010,3cyclictest13718-21awk12:25:250
131482201,3sleep00-21swapper/008:24:030
31582190,4sleep011-21rcuc/007:40:020
2779599196,4cyclictest842-21lldpd07:40:253
2779599195,3cyclictest86550irq/126-enp1s0-12:30:183
2779599191,9cyclictest0-21swapper/310:00:273
2779599191,8cyclictest16105-21cat08:35:013
2779599191,7cyclictest22724-21latency_hist11:35:013
2779599191,7cyclictest0-21swapper/311:58:073
2779599191,6cyclictest0-21swapper/309:42:383
2779599191,5cyclictest0-21swapper/311:05:143
2779599191,11cyclictest0-21swapper/309:55:043
2779599191,10cyclictest5234-21/usr/sbin/munin10:55:173
2779599191,10cyclictest26597-21/usr/sbin/munin10:30:293
2779599191,10cyclictest0-21swapper/310:28:573
2778999195,9cyclictest6441-40dnf09:43:322
2778999192,14cyclictest3912-21tail09:35:232
2778999191,7cyclictest23542-21awk09:10:012
2778999191,6cyclictest0-21swapper/208:41:352
2778999191,12cyclictest0-21swapper/211:26:342
2778599197,4cyclictest0-21swapper/110:50:211
2778599192,9cyclictest24500-21if_enp2s009:10:141
2778599192,9cyclictest24500-21if_enp2s009:10:141
27785991916,2cyclictest0-21swapper/107:53:201
27785991915,3cyclictest0-21swapper/109:05:101
27785991915,3cyclictest0-21swapper/109:04:221
27785991915,3cyclictest0-21swapper/108:47:001
27785991915,3cyclictest0-21swapper/107:18:361
27785991913,3cyclictest21058-21ntp_states10:15:211
2778599191,17cyclictest0-21swapper/112:16:041
2778599191,16cyclictest0-21swapper/112:20:551
2778599191,16cyclictest0-21swapper/108:52:381
2778599191,16cyclictest0-21swapper/108:32:021
2778599191,10cyclictest6612-21cpu07:55:101
2778599190,17cyclictest0-21swapper/109:35:111
2778599190,17cyclictest0-21swapper/107:49:001
2778599190,10cyclictest0-21swapper/109:42:551
2777799198,3cyclictest2199-21sshd07:33:300
2777799196,3cyclictest11545-21grep08:10:250
2777799195,7cyclictest86550irq/126-enp1s0-10:25:230
2777799193,2cyclictest0-21swapper/011:15:280
2777799192,5cyclictest18203-21munin-run11:25:010
2777799191,11cyclictest0-21swapper/010:18:420
2777799191,10cyclictest9639-21/usr/sbin/munin11:05:230
2777799191,10cyclictest31634-21/usr/sbin/munin11:55:170
2777799191,10cyclictest22311-21sshd11:31:300
82852180,3sleep00-21swapper/007:56:330
2779599187,3cyclictest28803-21/usr/sbin/munin10:35:273
2779599186,3cyclictest12050irq/123-ahci[0007:40:013
2779599184,4cyclictest23160-21df11:35:113
2779599182,8cyclictest27980-21/usr/sbin/munin07:10:223
2779599182,8cyclictest0-21swapper/309:20:193
2779599182,6cyclictest16862-21/usr/sbin/munin12:35:203
2779599181,9cyclictest0-21swapper/312:05:163
2779599181,9cyclictest0-21swapper/309:28:353
2779599181,7cyclictest8368-21grep09:45:263
2779599181,7cyclictest0-21swapper/312:10:263
2779599181,7cyclictest0-21swapper/311:47:163
27795991814,3cyclictest0-21swapper/311:40:103
27795991812,3cyclictest13627-21cut08:25:123
2779599181,15cyclictest0-21swapper/307:20:593
27795991811,4cyclictest24106-21/usr/sbin/munin09:10:113
27795991811,4cyclictest24106-21/usr/sbin/munin09:10:113
2779599181,10cyclictest23398-21sshd09:08:303
2778999188,3cyclictest32347-21awk09:30:012
2778999185,4cyclictest18447-21users12:35:232
2778999185,10cyclictest3681-21/usr/sbin/munin12:05:252
2778999184,3cyclictest28319-21/usr/sbin/munin07:10:112
2778999183,3cyclictest12050irq/123-ahci[0009:50:142
2778999182,15cyclictest0-21swapper/208:17:252
2778999182,14cyclictest17574-21munin-run10:10:012
2778999181,6cyclictest0-21swapper/210:15:432
2778999181,6cyclictest0-21swapper/209:24:082
2778999181,6cyclictest0-21swapper/208:35:582
2778999181,6cyclictest0-21swapper/207:51:522
27789991813,3cyclictest9539-21grep12:15:262
27789991813,3cyclictest14974-21tail08:25:252
27789991812,4cyclictest25727-21crond09:15:012
27789991812,4cyclictest25727-21crond09:15:012
27785991815,2cyclictest0-21swapper/108:05:331
27785991815,2cyclictest0-21swapper/108:05:331
2778599181,2cyclictest0-21swapper/109:30:021
2778599181,16cyclictest0-21swapper/108:41:031
2777799186,2cyclictest23530-21diskmemload12:36:230
2777799181,9cyclictest8986-21sshd08:02:300
2777799181,9cyclictest0-21swapper/007:23:510
2777799181,6cyclictest0-21swapper/012:33:570
2777799181,2cyclictest8103-21cpu12:15:130
2777799181,11cyclictest0-21swapper/008:25:130
2777799181,10cyclictest12395-21fschecks_time11:10:140
27777991810,5cyclictest18242-21tune2fs10:10:150
2777799180,10cyclictest0-21swapper/012:11:380
267372181,2sleep00-21swapper/009:15:170
240542180,3sleep30-21swapper/310:23:443
199732181,2sleep30-21swapper/308:53:403
134522181,2sleep30-21swapper/311:10:373
83752171,3sleep20-21swapper/207:57:432
54442170,2sleep20-21swapper/207:46:112
2779599176,3cyclictest86550irq/126-enp1s0-08:42:303
2779599171,7cyclictest19996-21/usr/sbin/munin10:15:233
2779599171,6cyclictest20574-21expr08:55:113
2779599171,6cyclictest0-21swapper/309:50:483
2779599171,4cyclictest29437-21/usr/sbin/munin11:50:143
2779599171,2cyclictest23530-21diskmemload10:48:553
27795991712,3cyclictest2403-21/usr/sbin/munin09:35:223
27795991712,3cyclictest15093-21ntp_offset11:15:223
2779599171,15cyclictest0-21swapper/311:25:453
2779599171,15cyclictest0-21swapper/309:04:573
2779599171,15cyclictest0-21swapper/308:49:083
2779599171,13cyclictest7431-21/usr/sbin/munin11:00:173
27795991711,3cyclictest31333-21expr10:40:113
2778999178,3cyclictest0-21swapper/208:30:472
2778999172,12cyclictest9550-21cat11:05:012
2778999171,8cyclictest848-21sssd_nss10:26:302
2778999171,7cyclictest22822-21/usr/sbin/munin11:35:142
2778999171,7cyclictest0-21swapper/210:20:242
2778999171,7cyclictest0-21swapper/209:46:282
2778999171,6cyclictest3386-21diskstats10:50:142
2778999171,6cyclictest0-21swapper/212:02:542
2778999171,6cyclictest0-21swapper/211:55:332
2778999171,6cyclictest0-21swapper/211:10:082
2778999171,6cyclictest0-21swapper/210:55:042
2778999171,6cyclictest0-21swapper/210:44:152
2778999171,6cyclictest0-21swapper/209:17:512
2778999171,3cyclictest88050irq/133-enp2s0-10:10:152
27789991713,3cyclictest0-21swapper/212:28:502
2778999171,2cyclictest86750irq/128-enp1s0-07:33:322
27789991712,3cyclictest9724-21cpuspeed_turbos11:05:132
27789991712,3cyclictest32107-21/usr/sbin/munin07:25:132
27789991712,3cyclictest2428-21users10:45:282
27789991712,3cyclictest1-21systemd10:00:092
27789991712,3cyclictest10714-21hddtemp_smartct08:10:162
2778999171,14cyclictest0-21swapper/211:50:052
2778999171,14cyclictest0-21swapper/207:15:182
27789991711,3cyclictest5880-21/usr/sbin/munin12:10:272
2777799176,3cyclictest14634-21snmpd10:42:440
2777799175,3cyclictest0-21swapper/009:00:350
2777799172,4cyclictest3774-21grep07:40:140
2777799171,5cyclictest29019-21ntpq07:10:180
27777991712,3cyclictest28489-21mailstats11:45:250
2777799171,14cyclictest0-21swapper/010:00:020
2777799171,14cyclictest0-21swapper/009:05:190
2777799171,14cyclictest0-21swapper/008:15:550
27777991711,3cyclictest30089-21munin-node07:15:030
27777991711,3cyclictest22822-21/usr/sbin/munin11:35:130
2777799171,10cyclictest0-21swapper/008:40:220
15062170,3sleep30-21swapper/307:27:463
150602171,2sleep20-21swapper/211:15:222
95222161,2sleep30-21swapper/308:06:313
95222161,2sleep30-21swapper/308:06:313
94322160,2sleep20-21swapper/208:05:222
94322160,2sleep20-21swapper/208:05:222
2779599169,4cyclictest2427-21date07:35:023
2779599161,6cyclictest30468-21munin-node07:15:033
2779599161,2cyclictest0-21swapper/308:36:163
2779599161,14cyclictest0-21swapper/307:53:503
27795991611,3cyclictest1888-21fschecks_count12:00:133
27795991611,3cyclictest13349-21date08:25:013
27795991610,3cyclictest12615-21latency_hist08:20:013
2778999162,4cyclictest22552-1kworker/u9:110:30:212
2778999161,5cyclictest30-21rcuc/211:31:232
2778999161,5cyclictest0-21swapper/211:20:152
27789991612,3cyclictest0-21swapper/208:24:392
2778999161,14cyclictest0-21swapper/207:20:372
27789991611,3cyclictest836-21runrttasks08:50:312
27789991611,3cyclictest3363-21cat07:40:112
27789991611,3cyclictest20316-21/usr/sbin/munin08:55:152
2777799161,4cyclictest14634-21snmpd08:47:220
27777991612,3cyclictest0-21swapper/008:50:500
27777991612,3cyclictest0-21swapper/008:32:290
2779599157,2cyclictest0-21swapper/307:49:553
27789991511,3cyclictest0-21swapper/209:01:472
2778999141,5cyclictest0-21swapper/208:00:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional