You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-11 - 20:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Aug 11, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
52502224149,24sleep20-21swapper/207:08:482
51522224152,59sleep00-21swapper/007:07:340
45502203175,18sleep10-21swapper/107:05:211
50412202165,25sleep30-21swapper/307:06:073
543999105101,3cyclictest0-21swapper/211:19:512
5439996359,3cyclictest0-21swapper/212:08:172
87712550,1sleep10-21swapper/107:15:131
5439995450,3cyclictest0-21swapper/209:45:432
179112540,2sleep121-21rcuc/111:15:201
89202520,1sleep30-21swapper/307:15:153
212352510,1sleep30-21swapper/311:25:013
71382470,2sleep10-21swapper/110:50:241
23642470,3sleep20-21swapper/209:25:252
543499447,28cyclictest4832-40dnf10:45:231
5434994313,22cyclictest21773-21ntp_kernel_err10:10:201
5434994221,16cyclictest21414-40dnf07:44:251
5434994010,14cyclictest32668-21expr11:50:111
543499396,22cyclictest15109-21df11:10:141
543499396,16cyclictest28254-21memory10:25:201
543499396,16cyclictest28254-21memory10:25:201
5434993910,20cyclictest19187-21tail10:05:131
543499389,15cyclictest18052-21fschecks_count12:30:161
5434993817,12cyclictest31006-21irqstats11:45:191
5434993813,12cyclictest11330-21acpi08:35:121
5434993811,14cyclictest1303-21fschecks_count09:25:141
542999387,14cyclictest11727-21munin-run09:50:010
543499379,19cyclictest8477-21expr10:55:141
543499376,20cyclictest13396-21grep12:20:121
5429993711,12cyclictest2040-21awk11:55:010
543999364,8cyclictest3762-21/usr/sbin/munin10:45:262
543499369,15cyclictest11884-21/usr/sbin/munin12:15:181
543499368,15cyclictest16730-21smart_sda12:25:241
543499359,12cyclictest14158-21sendmail_mailst07:25:241
543499358,9cyclictest19270-21cpuspeed_turbos11:20:121
543499358,24cyclictest24472-21memory11:30:181
543499358,24cyclictest0-21swapper/112:00:161
543499357,18cyclictest14824-21hddtemp_smartct09:55:181
8602340,3sleep3401ktimersoftd/311:50:173
543999341,12cyclictest7154-21unixbench_multi07:10:252
543499348,18cyclictest0-21swapper/109:15:151
543499344,22cyclictest0-21swapper/109:24:261
543499344,22cyclictest0-21swapper/109:24:261
543999335,3cyclictest26495-21diskmemload10:30:262
5439993313,3cyclictest28670-21hddtemp_smartct11:40:162
5434993316,14cyclictest12299-21diskstats09:50:151
5434993315,16cyclictest11732-21awk09:50:011
5434993315,15cyclictest26449-21diskstats07:55:161
5434993314,5cyclictest7323-21munin-run09:40:011
543499325,17cyclictest0-21swapper/111:08:311
5434993214,15cyclictest30850-21diskstats08:05:141
5434993214,15cyclictest30850-21diskstats08:05:141
5434993212,6cyclictest26113-21df11:35:141
5434993212,17cyclictest27117-21perf10:25:011
543999311,6cyclictest14596-21ntp_states08:40:212
543499318,11cyclictest4316-21proc_pri09:30:231
5434993114,14cyclictest24584-21/usr/sbin/munin07:50:171
5434993114,14cyclictest15453-40dnf08:44:361
5434993113,15cyclictest8652-21munin-run12:10:011
5434993110,10cyclictest26990-21expr09:10:131
543999301,5cyclictest30193-21/usr/sbin/munin11:45:212
543499309,8cyclictest325-21ntp_states10:35:221
543499308,14cyclictest3793-21smart_sda08:15:241
543499308,11cyclictest9742-21cat12:10:201
543499307,20cyclictest9520-21sh09:45:011
5434993013,14cyclictest2927-21hddtemp_smartct11:55:181
5434993011,4cyclictest6325-21users08:20:261
543999291,3cyclictest0-21swapper/212:37:212
543999291,13cyclictest27138-21tune2fs09:10:152
543499298,18cyclictest19989-21tr12:35:121
5434992910,16cyclictest28107-21cpuspeed_turbos11:40:121
543999281,14cyclictest0-21swapper/211:34:032
543999281,13cyclictest0-21swapper/208:26:112
543999281,12cyclictest0-21swapper/210:50:172
543999281,12cyclictest0-21swapper/209:39:592
543999281,12cyclictest0-21swapper/208:33:312
543999281,11cyclictest26127-21diskstats11:35:152
543499289,16cyclictest8899-21latency_hist08:30:021
543499287,9cyclictest17388-21ntp_kernel_pll_10:00:221
543499284,9cyclictest21715-21entropy11:25:161
5434992820,5cyclictest1912-21diskstats10:40:141
5434992820,5cyclictest1594-21smart_sda08:10:241
5434992810,9cyclictest10367-21/usr/sbin/munin11:00:161
5429992811,14cyclictest81ktimersoftd/011:25:010
543999271,9cyclictest0-21swapper/210:36:032
5439992712,3cyclictest0-21swapper/209:30:292
543999271,13cyclictest0-21swapper/212:12:232
543999271,13cyclictest0-21swapper/209:06:232
543499277,7cyclictest9135-21/usr/sbin/munin08:30:181
543499275,17cyclictest87550irq/128-enp1s0-10:17:371
5434992718,6cyclictest25787-21smart_sda09:05:241
5434992711,8cyclictest8941-1kworker/1:1H07:49:561
5434992711,4cyclictest15938-21expr08:45:121
5434992710,14cyclictest20298-21cstates08:55:141
543999261,5cyclictest16526-21cstates10:00:142
543999261,13cyclictest0-21swapper/209:56:512
543999261,12cyclictest0-21swapper/207:59:152
543999261,11cyclictest0-21swapper/209:22:112
543999261,11cyclictest0-21swapper/209:22:112
543499267,11cyclictest30115-21/usr/sbin/munin10:30:161
5434992618,5cyclictest23235-21/usr/sbin/munin09:00:191
5434992618,5cyclictest18371-21ntp_states07:35:211
5434992617,6cyclictest5966-21/usr/sbin/munin07:10:131
5434992613,10cyclictest763-21dbus-daemon08:50:201
5434992611,6cyclictest28298-21/usr/sbin/munin08:00:211
233342260,4sleep2543999cyclictest09:00:202
543999259,3cyclictest0-21swapper/207:30:592
543999257,3cyclictest0-21swapper/211:04:472
543999251,16cyclictest0-21swapper/208:35:152
543999251,13cyclictest0-21swapper/211:54:432
543999251,10cyclictest0-21swapper/207:22:232
5439992510,3cyclictest9758-21users10:55:272
5439992510,3cyclictest0-21swapper/210:08:552
5434992516,6cyclictest10716-21/usr/sbin/munin07:20:171
5434992510,12cyclictest15447-21df07:30:141
544499241,10cyclictest0-21swapper/311:58:253
543999249,3cyclictest0-21swapper/212:04:032
543999248,3cyclictest19163-21/usr/sbin/munin11:20:232
543999247,3cyclictest0-21swapper/211:58:072
543999241,11cyclictest0-21swapper/212:29:232
543999241,11cyclictest0-21swapper/210:40:192
543999241,10cyclictest0-21swapper/212:30:152
543999241,10cyclictest0-21swapper/210:12:272
544499231,9cyclictest0-21swapper/308:08:323
544499231,9cyclictest0-21swapper/308:08:323
544499231,6cyclictest0-21swapper/312:09:123
544499231,6cyclictest0-21swapper/310:41:443
543999239,3cyclictest0-21swapper/208:13:192
543999239,3cyclictest0-21swapper/207:26:472
543999237,3cyclictest0-21swapper/209:15:152
543999237,3cyclictest0-21swapper/207:52:432
5439992311,2cyclictest0-21swapper/209:50:152
543999231,10cyclictest0-21swapper/211:13:392
542999233,5cyclictest19008-21threads12:30:270
544499222,11cyclictest0-21swapper/309:30:173
544499221,15cyclictest0-21swapper/310:48:283
543999228,3cyclictest0-21swapper/207:17:092
543999227,3cyclictest0-21swapper/208:57:352
5439992218,3cyclictest0-21swapper/208:24:482
543999221,11cyclictest0-21swapper/210:29:192
543999221,11cyclictest0-21swapper/210:29:192
542999227,6cyclictest7-21ksoftirqd/011:17:470
542999221,9cyclictest0-21swapper/010:45:210
544499214,2cyclictest0-21swapper/311:45:133
544499210,6cyclictest0-21swapper/310:12:353
543999218,3cyclictest0-21swapper/207:35:142
543999217,3cyclictest0-21swapper/212:20:552
543999217,3cyclictest0-21swapper/210:24:072
543999217,3cyclictest0-21swapper/210:17:472
543999217,3cyclictest0-21swapper/208:00:152
543999211,11cyclictest0-21swapper/209:40:192
543999211,10cyclictest3320-21if_err_enp2s008:15:172
542999215,5cyclictest87350irq/126-enp1s0-08:30:200
542999214,3cyclictest81ktimersoftd/011:40:230
114462210,4sleep339-21rcuc/307:20:173
544499206,5cyclictest0-21swapper/311:18:473
544499206,4cyclictest0-21swapper/312:00:123
544499201,4cyclictest0-21swapper/309:37:363
5444992011,8cyclictest87350irq/126-enp1s0-09:45:133
544499201,12cyclictest0-21swapper/312:32:393
544499201,12cyclictest0-21swapper/312:16:083
544499201,12cyclictest0-21swapper/310:05:233
544499201,11cyclictest29533-21/usr/sbin/munin10:30:153
544499201,11cyclictest19502-21/usr/sbin/munin07:40:173
544499200,5cyclictest18679-21kworker/u8:011:28:553
543999208,3cyclictest0-21swapper/208:45:032
543999205,6cyclictest774-21avahi-daemon07:45:012
542999209,3cyclictest1550-21/usr/sbin/munin10:40:190
542999207,5cyclictest87550irq/128-enp1s0-10:20:120
542999205,5cyclictest81ktimersoftd/012:00:220
542999204,8cyclictest7-21ksoftirqd/010:17:370
542999202,10cyclictest8519-21/usr/sbin/munin07:15:200
542999202,10cyclictest14771-21/usr/sbin/munin11:10:240
542999201,11cyclictest18546-21/usr/sbin/munin10:05:190
542999201,11cyclictest17307-21/usr/sbin/munin07:35:160
542999201,11cyclictest16343-21/usr/sbin/munin10:00:280
77862191,2sleep00-21swapper/010:53:130
544499197,3cyclictest0-21swapper/312:14:043
544499196,4cyclictest0-21swapper/309:59:473
544499194,8cyclictest5408-21kworker/u8:210:56:383
544499194,6cyclictest17575-21sendmail_mailqu10:00:253
544499194,5cyclictest0-21swapper/309:41:073
544499193,9cyclictest0-21swapper/309:10:133
544499192,3cyclictest0-21swapper/312:26:323
544499192,14cyclictest28475-21tr10:25:243
544499192,14cyclictest28475-21tr10:25:243
544499191,6cyclictest0-21swapper/309:30:003
5444991911,5cyclictest14869-21cpuspeed_turbos11:10:133
544499191,12cyclictest0-21swapper/307:50:193
544499191,11cyclictest839-21lldpd10:35:413
544499191,10cyclictest4729-21/usr/sbin/munin08:20:143
544499191,10cyclictest0-21swapper/311:04:103
544499191,10cyclictest0-21swapper/310:54:423
544499191,10cyclictest0-21swapper/310:23:473
544499190,13cyclictest0-21swapper/311:40:243
543999195,2cyclictest0-21swapper/211:25:272
543999194,3cyclictest0-21swapper/211:05:192
542999196,5cyclictest87550irq/128-enp1s0-12:35:120
542999196,3cyclictest31331-21tr08:05:180
542999196,3cyclictest31331-21tr08:05:180
542999195,5cyclictest11-21rcuc/009:25:190
542999194,8cyclictest0-21swapper/009:05:160
542999192,7cyclictest87550irq/128-enp1s0-12:12:590
542999192,5cyclictest11-21rcuc/008:10:180
542999191,6cyclictest0-21swapper/010:35:150
542999191,6cyclictest0-21swapper/009:40:180
542999191,3cyclictest7-21ksoftirqd/009:57:260
542999191,16cyclictest0-21swapper/012:25:180
542999191,10cyclictest808-21snmpd10:57:410
542999190,7cyclictest87550irq/128-enp1s0-11:35:160
3552190,2sleep30-21swapper/309:21:493
3552190,2sleep30-21swapper/309:21:493
544499187,3cyclictest12915-21/usr/sbin/munin07:25:253
544499184,4cyclictest1307-21head08:10:203
544499183,3cyclictest20455-21tail12:35:153
544499182,2cyclictest87350irq/126-enp1s0-11:30:263
544499181,9cyclictest808-21snmpd08:17:133
544499181,4cyclictest0-21swapper/312:24:133
5444991813,3cyclictest302362sleep308:04:393
5444991813,3cyclictest22739-21grep09:00:153
5444991811,4cyclictest26722-21grep07:55:173
544499181,14cyclictest13426-21sed09:50:273
5444991811,4cyclictest0-21swapper/311:38:403
544499181,11cyclictest0-21swapper/311:08:223
544499181,10cyclictest5630-21/usr/sbin/munin07:10:203
5439991812,4cyclictest19484-21unixbench_singl08:50:262
543999181,14cyclictest0-21swapper/208:09:472
543999181,14cyclictest0-21swapper/208:09:472
542999186,3cyclictest0-21swapper/008:15:230
542999185,5cyclictest11389-21cpu08:35:120
542999184,4cyclictest7100-21unixbench-2d07:10:250
542999182,9cyclictest3488-21df09:30:130
542999182,7cyclictest28922-21/usr/sbin/munin09:15:140
542999182,7cyclictest10716-21/usr/sbin/munin07:20:220
542999181,6cyclictest0-21swapper/012:15:200
542999181,5cyclictest0-21swapper/011:09:260
542999181,5cyclictest0-21swapper/009:13:040
5429991813,3cyclictest2674-21grep11:55:170
5429991811,4cyclictest31257-21ntp_states11:45:220
301482180,2sleep00-21swapper/008:03:330
544499175,3cyclictest0-21swapper/310:16:223
544499172,6cyclictest24957-21fschecks_count09:05:153
544499172,12cyclictest15725-21/usr/sbin/munin08:45:163
544499172,12cyclictest11329-21/usr/sbin/munin08:35:153
544499171,7cyclictest29833-21cut09:15:193
544499171,7cyclictest22286-21perl07:45:173
544499171,6cyclictest17206-21cat07:35:023
544499171,13cyclictest839-21lldpd08:30:393
544499171,11cyclictest0-21swapper/308:55:283
5444991710,4cyclictest0-21swapper/308:50:213
543999174,4cyclictest11943-21cat12:15:192
542999176,10cyclictest0-21swapper/012:25:020
542999174,5cyclictest87550irq/128-enp1s0-09:50:180
542999173,3cyclictest87550irq/128-enp1s0-08:20:230
542999171,7cyclictest6685-21/usr/sbin/munin12:05:240
542999171,7cyclictest0-21swapper/010:30:140
542999171,6cyclictest0-21swapper/010:25:200
542999171,6cyclictest0-21swapper/010:25:200
542999171,3cyclictest16339-21kworker/0:210:10:150
5429991713,2cyclictest81ktimersoftd/008:45:110
5429991713,2cyclictest26495-21diskmemload09:37:240
5429991712,3cyclictest31464-21diskstats09:20:140
5429991712,3cyclictest31464-21diskstats09:20:140
542999171,14cyclictest0-21swapper/007:27:500
542999171,13cyclictest20116-21grep07:40:130
5429991710,4cyclictest7564-21grep08:25:190
5429991710,4cyclictest23328-21idleruntime-cro11:30:010
255742171,3sleep00-21swapper/007:51:210
235752170,2sleep00-21swapper/007:48:520
190332171,2sleep30-21swapper/307:37:033
120512170,2sleep00-21swapper/011:01:230
544499161,12cyclictest6937-21/usr/sbin/munin08:25:293
5444991610,3cyclictest13716-21expr08:40:123
543999161,2cyclictest0-21swapper/207:45:032
542999164,5cyclictest22321-21/usr/sbin/munin09:00:220
542999162,5cyclictest23900-21df_inode11:30:130
5429991612,3cyclictest0-21swapper/007:34:210
5429991611,3cyclictest18584-21sed08:50:160
135202160,2sleep00-21swapper/008:40:090
542999159,3cyclictest22239-21cat09:00:010
5429991510,3cyclictest26337-21expr07:55:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional