You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-21 - 02:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Apr 21, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
289562240164,25sleep20-21swapper/219:06:402
291052223192,21sleep00-21swapper/019:08:350
291032203166,24sleep30-21swapper/319:08:343
270242203167,24sleep10-21swapper/119:05:021
609321850,3sleep12930699cyclictest00:20:251
2382721760,4sleep32931699cyclictest23:50:013
350321420,4sleep32931699cyclictest23:00:333
1921321200,3sleep12930699cyclictest22:25:161
318912750,2sleep10-21swapper/122:55:011
71512600,2sleep1221ktimersoftd/123:10:181
61022560,2sleep232-21ksoftirqd/221:55:162
252062490,1sleep30-21swapper/323:50:253
56732440,1sleep30-21swapper/321:55:123
2930699422,18cyclictest23-21ksoftirqd/122:33:131
2930699422,18cyclictest23-21ksoftirqd/122:33:131
29300994110,14cyclictest10798-21munin-run19:40:010
2931699374,7cyclictest2301-40dnf20:31:063
2930099379,13cyclictest6723-21awk00:25:010
2930699361,26cyclictest0-21swapper/121:10:541
71752350,4sleep011-21rcuc/020:45:030
29316993512,3cyclictest2232-21/usr/sbin/munin19:20:203
29316993510,3cyclictest8001-21ntp_states00:25:223
2930699351,25cyclictest0-21swapper/123:41:021
2931699341,3cyclictest11950-21smart_sda23:20:243
2930699341,23cyclictest0-21swapper/122:38:291
2930099349,11cyclictest0-21swapper/000:05:010
2931699331,6cyclictest8259-21sh23:15:013
29316993312,3cyclictest29605-21needreboot20:20:203
29316993310,3cyclictest6627-21/usr/sbin/munin19:30:113
29316993310,3cyclictest16665-21/usr/sbin/munin22:20:203
2930099334,13cyclictest25849-21munin-run23:55:010
29316993211,3cyclictest827-21runrttasks22:02:063
2931699321,12cyclictest4476-21chrt00:19:433
2931699321,12cyclictest32147-21grep19:15:043
2931699321,11cyclictest808-21snmpd20:46:053
29316993210,2cyclictest31435-21/usr/sbin/munin21:40:203
2930699321,14cyclictest0-21swapper/123:38:291
2930699320,23cyclictest0-21swapper/100:33:011
2931699312,5cyclictest0-21swapper/321:50:203
2931699312,11cyclictest8828-21/usr/sbin/munin19:35:223
29316993111,3cyclictest22927-21expr21:20:133
29316993111,3cyclictest22927-21expr21:20:133
29316993111,3cyclictest19088-21expr22:25:143
29316993110,3cyclictest16600-21cat22:20:013
2930699311,2cyclictest0-21swapper/119:34:051
2930699310,25cyclictest0-21swapper/100:02:051
2931699309,4cyclictest1919-21tail21:45:163
2931699301,7cyclictest0-21swapper/300:34:343
2931699301,6cyclictest23980-21latency_hist20:10:023
2931699301,6cyclictest23980-21latency_hist20:10:013
2931699301,5cyclictest0-21swapper/322:47:413
2931699301,5cyclictest0-21swapper/321:10:243
2931699301,3cyclictest19811-21/usr/sbin/munin20:00:213
29316993012,4cyclictest24806-21grep20:10:153
29316993012,3cyclictest32611-21date00:10:023
29316993011,3cyclictest827-21runrttasks22:36:343
29316993011,3cyclictest11804-21cstates20:55:123
2931699301,12cyclictest0-21swapper/323:16:593
29316993010,3cyclictest29731-21cat19:10:133
29316993010,3cyclictest23196-21cat22:35:023
29316993010,3cyclictest23196-21cat22:35:013
2931699300,4cyclictest18020-21awk21:10:013
2931299300,20cyclictest0-21swapper/220:15:432
2931299300,13cyclictest0-21swapper/221:50:522
2930699301,21cyclictest0-21swapper/121:09:431
2930699301,13cyclictest0-21swapper/119:14:531
2930699301,11cyclictest0-21swapper/122:45:451
2930699300,23cyclictest0-21swapper/123:04:331
2930699300,23cyclictest0-21swapper/122:59:351
2931699297,4cyclictest839-21lldpd22:59:013
2931699291,6cyclictest16375-21cat19:50:213
2931699291,5cyclictest30530-21hddtemp_smartct22:50:173
2931699291,3cyclictest27786-21iwlist21:30:173
29316992911,3cyclictest9442-21/usr/sbin/munin20:50:103
2931699291,12cyclictest26077-21/usr/sbin/munin23:55:163
29316992910,3cyclictest25506-21hddtemp_smartct21:25:163
29316992910,3cyclictest13185-21cat23:25:143
2931699290,13cyclictest0-21swapper/321:39:433
2931299291,18cyclictest0-21swapper/220:11:032
2931299291,17cyclictest0-21swapper/221:43:592
2931299290,21cyclictest0-21swapper/223:33:472
2931299290,11cyclictest0-21swapper/223:28:432
2930699291,12cyclictest0-21swapper/120:04:051
2930699291,11cyclictest0-21swapper/120:10:551
2930699290,2cyclictest0-21swapper/100:18:571
2930699290,13cyclictest0-21swapper/119:19:531
2930699290,12cyclictest0-21swapper/121:53:371
2931699285,3cyclictest87350irq/126-enp1s0-22:10:123
2931699282,11cyclictest30809-21/usr/sbin/munin20:25:173
2931699281,5cyclictest1126-21interrupts00:10:183
2931699280,5cyclictest0-21swapper/320:15:143
29312992815,3cyclictest0-21swapper/220:49:332
2931299281,11cyclictest0-21swapper/222:27:192
2931299280,12cyclictest0-21swapper/223:11:232
2930699281,3cyclictest0-21swapper/120:43:491
2930699281,2cyclictest0-21swapper/120:18:531
2930699281,12cyclictest0-21swapper/121:04:171
2930699281,12cyclictest0-21swapper/119:56:171
2930699281,12cyclictest0-21swapper/119:24:451
2930699281,11cyclictest0-21swapper/120:37:171
2930699281,10cyclictest0-21swapper/121:38:571
2930699280,9cyclictest0-21swapper/121:48:111
2930699280,2cyclictest0-21swapper/123:25:451
2930699280,11cyclictest0-21swapper/123:20:091
2930699280,11cyclictest0-21swapper/122:20:211
2930699280,11cyclictest0-21swapper/122:01:531
2930699280,10cyclictest0-21swapper/122:15:571
2931699279,3cyclictest5668-21gltestperf20:40:153
2931699279,11cyclictest0-21swapper/300:20:373
2931699273,3cyclictest0-21swapper/300:36:083
2931699271,8cyclictest0-21swapper/300:02:063
2931699271,6cyclictest0-21swapper/323:39:183
2931699271,12cyclictest4779-21dnf19:25:153
2931299271,13cyclictest0-21swapper/222:31:512
2931299271,13cyclictest0-21swapper/222:31:512
2931299270,17cyclictest0-21swapper/220:08:552
2931299270,17cyclictest0-21swapper/220:08:552
2931299270,13cyclictest0-21swapper/222:42:432
2931299270,11cyclictest0-21swapper/219:29:042
2930699271,13cyclictest0-21swapper/123:45:301
2930699271,11cyclictest0-21swapper/120:52:491
2930699271,11cyclictest0-21swapper/119:44:111
2930699271,10cyclictest0-21swapper/120:28:371
2930699271,10cyclictest0-21swapper/120:09:371
2930699271,10cyclictest0-21swapper/120:09:371
2930699270,12cyclictest0-21swapper/123:34:451
2930699270,11cyclictest0-21swapper/122:41:051
2931699261,11cyclictest0-21swapper/319:43:513
29312992615,3cyclictest0-21swapper/221:03:292
29312992612,3cyclictest0-21swapper/223:08:332
2931299261,18cyclictest0-21swapper/220:42:352
2931299261,17cyclictest0-21swapper/219:14:232
29312992610,10cyclictest0-21swapper/220:51:412
2931299260,17cyclictest0-21swapper/220:56:232
2930699261,9cyclictest0-21swapper/123:17:231
2930699261,9cyclictest0-21swapper/121:40:111
2930699261,9cyclictest0-21swapper/120:47:531
2930699261,11cyclictest0-21swapper/122:13:531
2930699261,11cyclictest0-21swapper/121:17:341
2930699261,10cyclictest2201-21runrttasks00:13:411
2930699261,10cyclictest0-21swapper/123:55:361
2930699260,2cyclictest0-21swapper/100:37:251
2930699260,20cyclictest0-21swapper/121:32:291
2930699260,11cyclictest0-21swapper/119:39:531
29300992611,7cyclictest0-21swapper/023:45:170
2931699258,3cyclictest0-21swapper/323:08:073
2931699251,6cyclictest0-21swapper/322:05:353
29316992510,3cyclictest0-21swapper/323:41:433
2931299259,9cyclictest0-21swapper/219:50:052
2931299259,11cyclictest0-21swapper/222:16:052
2931299256,8cyclictest0-21swapper/222:45:192
29312992511,8cyclictest0-21swapper/220:27:052
2931299251,12cyclictest0-21swapper/220:21:392
2931299251,11cyclictest0-21swapper/219:30:592
2931299250,15cyclictest0-21swapper/221:30:352
2931299250,12cyclictest0-21swapper/222:08:082
2931299250,11cyclictest0-21swapper/223:46:512
2931299250,11cyclictest0-21swapper/222:58:512
2931299250,10cyclictest0-21swapper/221:07:192
2931299250,10cyclictest0-21swapper/200:28:442
2930699251,7cyclictest0-21swapper/120:56:351
2930699251,12cyclictest0-21swapper/100:09:251
2930699251,10cyclictest0-21swapper/123:50:061
2930699250,11cyclictest0-21swapper/120:34:251
2930699250,10cyclictest0-21swapper/123:06:571
2931699241,11cyclictest827-21runrttasks23:30:593
2931299244,6cyclictest9021-21fschecks_time23:15:152
2931299241,11cyclictest0-21swapper/223:52:512
2931299241,10cyclictest0-21swapper/219:44:242
29312992410,5cyclictest0-21swapper/221:37:192
2931299240,12cyclictest0-21swapper/200:04:472
2931299240,11cyclictest0-21swapper/222:50:312
2931299240,11cyclictest0-21swapper/222:13:282
2931299240,11cyclictest0-21swapper/221:48:352
2931299240,10cyclictest0-21swapper/220:30:592
2930699241,12cyclictest0-21swapper/119:25:051
2930699241,11cyclictest0-21swapper/121:20:341
2930699241,11cyclictest0-21swapper/121:20:341
2930699241,10cyclictest0-21swapper/120:22:051
2930699240,11cyclictest0-21swapper/119:45:331
2930699240,10cyclictest0-21swapper/122:05:191
2930099248,14cyclictest0-21swapper/020:15:010
2930099245,16cyclictest19579-21munin-run20:00:010
2931699237,3cyclictest0-21swapper/321:01:353
2931699231,17cyclictest21947-21grep21:15:273
2931299239,4cyclictest18546-21smart_sda23:35:232
2931299232,16cyclictest88950irq/132-enp2s0-19:22:262
2931299231,9cyclictest0-21swapper/200:13:272
2931299230,9cyclictest0-21swapper/223:20:332
2931299230,9cyclictest0-21swapper/223:00:032
2931299230,17cyclictest0-21swapper/219:45:392
2931299230,11cyclictest0-21swapper/200:34:082
2930699231,12cyclictest0-21swapper/119:52:131
2930699230,10cyclictest0-21swapper/100:28:251
2930099236,15cyclictest12672-21perf23:25:010
120212232,17sleep012074-21ntpq19:40:220
2931299229,3cyclictest0-21swapper/219:57:272
2931299228,3cyclictest0-21swapper/200:37:352
2931299227,7cyclictest2725-21/usr/sbin/munin00:15:112
2931299222,12cyclictest0-21swapper/200:09:252
2931299221,11cyclictest0-21swapper/221:24:472
2931299221,11cyclictest0-21swapper/221:24:472
2931299220,11cyclictest0-21swapper/221:15:512
2930699223,3cyclictest0-21swapper/121:59:451
2930699220,11cyclictest0-21swapper/121:25:381
29300992216,3cyclictest2799-21grep21:45:250
2930099221,5cyclictest0-21swapper/022:56:470
2930099221,10cyclictest0-21swapper/019:32:430
75922210,7sleep011-21rcuc/020:45:150
2931699213,3cyclictest0-21swapper/322:44:203
2931699211,10cyclictest48012chrt20:39:593
2931299215,8cyclictest0-21swapper/219:16:152
2931299213,10cyclictest0-21swapper/200:20:192
2931299211,10cyclictest0-21swapper/223:56:152
2931299211,10cyclictest0-21swapper/222:03:522
2931299211,10cyclictest0-21swapper/220:38:112
2931299211,10cyclictest0-21swapper/220:02:272
2931299210,10cyclictest0-21swapper/222:39:172
2930099218,7cyclictest87350irq/126-enp1s0-21:14:530
2930099214,6cyclictest0-21swapper/023:29:290
2930099214,10cyclictest808-21snmpd23:32:260
2930099212,5cyclictest27426-21munin-run22:45:010
2930099210,9cyclictest0-21swapper/021:29:520
2931299206,3cyclictest0-21swapper/219:36:442
2931299201,11cyclictest0-21swapper/222:24:082
2931299201,10cyclictest0-21swapper/223:43:512
2930099209,3cyclictest18010-21diskmemload22:00:150
2930099205,3cyclictest0-21swapper/023:15:570
2930099204,3cyclictest0-21swapper/021:22:150
2930099204,3cyclictest0-21swapper/021:22:150
2930099201,9cyclictest0-21swapper/021:43:320
2930099201,16cyclictest6399-21munin-run19:30:010
2930099201,11cyclictest28271-21/usr/sbin/munin21:30:240
2930099201,11cyclictest11152-21open_files22:05:220
2930099201,11cyclictest0-21swapper/023:40:350
2930099201,11cyclictest0-21swapper/021:16:550
2930099201,10cyclictest8493-21unixbench_singl00:25:260
2930099201,10cyclictest18010-21diskmemload23:10:130
2930099201,10cyclictest10124-21strings00:30:190
2930099200,10cyclictest0-21swapper/022:19:500
59772191,2sleep00-21swapper/023:09:020
2931699191,3cyclictest89250irq/135-enp2s0-19:59:483
2931299196,4cyclictest25509-21which21:25:162
2930099198,4cyclictest0-21swapper/021:36:500
2930099194,2cyclictest87350irq/126-enp1s0-23:57:310
2930099194,12cyclictest11-21rcuc/023:00:180
2930099191,9cyclictest13260-21needreboot22:10:190
2930099191,6cyclictest0-21swapper/022:39:210
2930099191,13cyclictest0-21swapper/000:36:380
2930099191,10cyclictest21060-21/usr/sbin/munin22:30:250
2930099191,10cyclictest21060-21/usr/sbin/munin22:30:250
2930099191,10cyclictest18056-21sed21:10:010
2930099191,10cyclictest15497-21sshd21:01:060
2930099191,10cyclictest0-21swapper/022:49:110
2930099191,10cyclictest0-21swapper/000:10:250
2931699182,3cyclictest0-21swapper/319:45:393
2931299181,9cyclictest0-21swapper/221:13:232
2930099188,9cyclictest7-21ksoftirqd/022:50:230
2930099188,3cyclictest5277-21sshd21:53:050
2930099187,3cyclictest7013-21timerandwakeup21:55:250
2930099186,3cyclictest0-21swapper/022:21:280
2930099183,6cyclictest3453-21gltestperf20:35:170
2930099181,9cyclictest0-21swapper/023:36:350
2930099181,6cyclictest0-21swapper/022:29:170
2930099181,5cyclictest0-21swapper/000:06:430
2930099181,14cyclictest3203-21gltestperf00:15:150
2930099181,10cyclictest14691-21grep19:45:240
2930099175,3cyclictest28766-21cpuspeed_turbos20:20:130
2930099173,12cyclictest11-21rcuc/020:55:230
2930099171,7cyclictest20195-21fschecks_count20:00:150
2930099171,3cyclictest0-21swapper/019:21:510
29300991713,3cyclictest0-21swapper/020:34:290
2930099171,14cyclictest0-21swapper/020:50:190
29300991711,3cyclictest30145-21gltestperf19:10:160
2930099170,10cyclictest11-21rcuc/019:15:240
2930099161,6cyclictest31194-21fschecks_count20:25:140
2930099161,2cyclictest27402-21head20:15:190
2930099161,14cyclictest0-21swapper/019:51:170
29300991611,3cyclictest23041-21nfs4_client20:05:200
29300991611,3cyclictest23041-21nfs4_client20:05:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional