You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 22:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Mar 19, 2026 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
74852232162,56sleep00-21swapper/007:09:210
72592230163,55sleep10-21swapper/107:06:251
74882202165,25sleep30-21swapper/307:09:263
55772201165,24sleep20-21swapper/207:05:102
7645999896,1cyclictest0-21swapper/307:22:583
7645998683,2cyclictest0-21swapper/307:53:123
7645997976,2cyclictest0-21swapper/309:36:443
7645997773,3cyclictest0-21swapper/308:03:183
79212620,4sleep011-21rcuc/012:00:240
7645995955,3cyclictest0-21swapper/311:33:453
237242560,1sleep30-21swapper/310:10:163
118872560,4sleep232-21ksoftirqd/212:10:182
219382480,1sleep20-21swapper/212:35:012
764599425,30cyclictest19270-40dnf07:35:043
764599425,30cyclictest19270-40dnf07:35:043
764599411,30cyclictest17568-21/usr/sbin/munin12:25:173
764599411,30cyclictest17568-21/usr/sbin/munin12:25:163
219242410,2sleep30-21swapper/310:05:223
7625993910,13cyclictest15153-21munin-run12:20:010
764599358,14cyclictest9126-21/usr/sbin/munin08:25:153
764599351,6cyclictest0-21swapper/311:51:323
764599345,3cyclictest86850irq/129-enp1s0-07:40:153
764599343,5cyclictest7045-21open_files10:45:223
764599341,30cyclictest23534-21grep12:35:233
764599341,22cyclictest30213-21switchtime09:10:223
764599335,17cyclictest0-21swapper/312:19:153
7645993317,13cyclictest28043-21grep11:35:103
764599331,4cyclictest8163-21/usr/sbin/munin10:50:293
762599339,11cyclictest0-21swapper/007:45:000
762599334,13cyclictest13912-21munin-run09:50:000
764599328,21cyclictest6316-21users08:15:253
764599324,3cyclictest15928-21perl12:20:143
7645993229,2cyclictest0-21swapper/307:31:293
7645993229,2cyclictest0-21swapper/307:31:293
764599322,6cyclictest0-21swapper/310:36:493
764599322,4cyclictest15153-40dnf08:35:483
764599321,5cyclictest15109-21/usr/sbin/munin07:25:203
764599321,4cyclictest809-21sshd08:46:473
764599321,4cyclictest809-21sshd08:46:463
764599321,4cyclictest7886-21/usr/sbin/munin07:10:133
764599321,3cyclictest2283-21grep09:20:243
7645993213,16cyclictest4055-21grep08:10:263
7645993210,19cyclictest8926-21sort08:25:013
762599327,12cyclictest24907-21munin-run10:15:000
762599324,13cyclictest23942-21munin-run12:40:020
764599319,19cyclictest20238-21/usr/sbin/munin12:30:143
764599318,20cyclictest30266-21idleruntime-cro08:00:023
764599313,5cyclictest13201-21smart_sda09:45:253
764599312,6cyclictest5817-21cat10:45:023
764599312,3cyclictest12586-21smart_sda08:30:233
764599312,26cyclictest28262-21head09:05:263
764599312,26cyclictest1093-21grep11:45:223
764599311,6cyclictest26316-21processes10:15:243
764599311,6cyclictest0-21swapper/310:00:263
764599311,6cyclictest0-21swapper/310:00:263
764599311,3cyclictest20325-21ps11:15:223
764599311,3cyclictest14262-21grep09:50:113
764599311,3cyclictest12050irq/123-ahci[0010:00:013
764599311,2cyclictest8275-21taskset12:01:323
764599311,27cyclictest21353-21/usr/sbin/munin11:20:133
7645993111,17cyclictest7307-21latency_hist09:35:013
7632993110,9cyclictest0-21swapper/110:28:571
140382310,1sleep30-21swapper/311:00:233
764599309,18cyclictest812-21snmpd08:07:443
764599309,18cyclictest15721-21/usr/sbin/munin08:40:153
764599308,19cyclictest16134-21grep11:05:233
764599306,21cyclictest11375-21sshd09:40:473
764599305,3cyclictest86550irq/126-enp1s0-12:05:193
764599305,13cyclictest86550irq/126-enp1s0-08:55:213
764599303,4cyclictest12243-21smart_sda12:10:233
764599302,25cyclictest4372-21/usr/sbin/munin11:55:193
764599301,3cyclictest32020-21memory09:15:183
764599301,3cyclictest25632-21pmu-power09:00:233
764599301,26cyclictest3139-21/usr/sbin/munin09:25:273
764599301,26cyclictest20113-21/usr/sbin/munin08:50:163
764599301,25cyclictest11324-21sed10:55:193
764599298,18cyclictest10924-21expr07:15:123
764599298,18cyclictest10025-21munin-node11:40:093
764599292,23cyclictest849-21grep10:30:253
764599291,4cyclictest17309-21/usr/sbin/munin11:10:123
764599291,3cyclictest24527-21needreboot11:25:183
764599288,16cyclictest24739-21cut07:45:193
764599281,6cyclictest28693-21diskmemload10:25:173
764599281,3cyclictest27908-21/usr/sbin/munin10:20:153
32602280,5sleep07-21ksoftirqd/011:50:200
39922270,3sleep1763299cyclictest10:40:101
314742270,1sleep00-21swapper/011:40:210
763299261,6cyclictest27171-21timerwakeupswit11:30:241
7632992611,12cyclictest19215-21kworker/u8:012:35:241
763299260,19cyclictest0-21swapper/109:09:471
763299259,3cyclictest0-21swapper/111:15:501
763299256,4cyclictest3267-40dnf10:36:491
763299251,4cyclictest20452-21hwlatdetect12:30:151
763299251,2cyclictest0-21swapper/109:48:201
763299251,13cyclictest28693-21diskmemload11:05:261
762599251,12cyclictest14539-21munin-run11:05:010
763299241,19cyclictest0-21swapper/112:05:471
7632992410,11cyclictest12910-21systemd-journal07:20:111
762599241,10cyclictest0-21swapper/010:58:580
763299238,4cyclictest0-21swapper/107:45:131
763299236,2cyclictest0-21swapper/108:10:011
763299235,11cyclictest0-21swapper/111:02:181
763299234,3cyclictest10025-21munin-node07:30:101
763299234,3cyclictest10025-21munin-node07:30:091
763299231,4cyclictest16058-21df08:40:131
763299231,16cyclictest0-21swapper/111:50:321
763299231,16cyclictest0-21swapper/111:23:381
763299231,15cyclictest0-21swapper/109:26:451
763299227,2cyclictest28693-21diskmemload09:41:541
763299226,3cyclictest0-21swapper/110:55:341
763299225,3cyclictest0-21swapper/111:41:101
763299225,3cyclictest0-21swapper/108:49:421
763299225,3cyclictest0-21swapper/108:49:421
763299225,3cyclictest0-21swapper/107:38:171
763299225,3cyclictest0-21swapper/107:38:171
763299224,4cyclictest0-21swapper/108:34:451
763299222,11cyclictest1024-21ntp_states11:45:211
763299221,9cyclictest0-21swapper/110:24:451
7632992215,4cyclictest9336-21tail07:10:251
763299221,3cyclictest1259-21df09:20:141
763299221,18cyclictest26101-21missed_timers10:15:191
763299221,15cyclictest0-21swapper/111:10:161
763299220,20cyclictest0-21swapper/108:15:181
762599223,16cyclictest1966-21perf11:50:010
762599221,5cyclictest0-21swapper/010:52:330
762599221,4cyclictest21747-21munin-run12:35:010
57322220,3sleep230-21rcuc/209:30:142
100052221,16sleep210079-21iwlist07:15:022
763899211,7cyclictest0-21swapper/211:12:452
763899211,6cyclictest0-21swapper/211:38:002
7638992115,4cyclictest32305-21/usr/sbin/munin10:30:152
7638992110,3cyclictest24873-21sshd10:14:482
763299219,3cyclictest7269-21diskstats08:20:121
763299218,3cyclictest0-21swapper/111:29:261
763299217,2cyclictest0-21swapper/111:55:211
763299216,3cyclictest0-21swapper/112:01:291
763299216,3cyclictest0-21swapper/110:01:501
763299216,3cyclictest0-21swapper/110:01:501
763299215,3cyclictest0-21swapper/108:53:141
763299215,3cyclictest0-21swapper/108:10:451
763299215,3cyclictest0-21swapper/107:42:531
7632992113,3cyclictest0-21swapper/110:46:591
763299211,17cyclictest28693-21diskmemload09:37:481
763299211,15cyclictest0-21swapper/111:36:031
763299211,15cyclictest0-21swapper/107:16:331
7632992111,4cyclictest10654-21kworker/u8:212:12:171
763299211,11cyclictest318-21ntp_states10:30:201
763299211,11cyclictest29170-21cat07:55:161
763299211,11cyclictest23741-21irqstats10:10:161
763299211,10cyclictest10125-21latency_hist10:55:001
763299211,10cyclictest0-21swapper/112:25:291
763299211,10cyclictest0-21swapper/112:25:281
763299210,19cyclictest0-21swapper/107:25:191
763299210,14cyclictest0-21swapper/108:59:471
762599211,11cyclictest26737-21cpu09:05:130
7625992110,7cyclictest833-21runrttasks10:36:500
142332210,3sleep20-21swapper/212:15:202
763899201,6cyclictest0-21swapper/209:25:062
763899200,13cyclictest0-21swapper/210:55:232
763299209,3cyclictest16668-21df09:55:121
763299209,3cyclictest14817-21switchtime08:35:241
763299204,3cyclictest0-21swapper/107:52:411
7632992015,3cyclictest14764-21grep09:50:151
763299201,17cyclictest0-21swapper/112:20:031
763299201,11cyclictest619-21latency_hist09:20:021
763299201,11cyclictest28924-21acpi09:10:111
763299201,11cyclictest0-21swapper/109:00:161
763299201,10cyclictest10537-21ls08:25:241
763299201,10cyclictest0-21swapper/110:09:011
762599203,14cyclictest20391-21find11:15:230
762599202,6cyclictest0-21swapper/010:45:210
762599201,6cyclictest0-21swapper/009:58:220
762599201,4cyclictest23331-21perf11:25:010
762599201,10cyclictest809-21sshd09:42:470
762599201,10cyclictest30486-21/usr/sbin/munin08:00:190
762599201,10cyclictest21727-21needreboot10:05:180
762599201,10cyclictest1458-21head08:05:220
763899192,10cyclictest19503-21missed_timers10:00:202
763899192,10cyclictest19503-21missed_timers10:00:202
763899191,6cyclictest0-21swapper/212:29:422
763899191,6cyclictest0-21swapper/212:29:422
763899191,5cyclictest0-21swapper/211:04:392
763899191,5cyclictest0-21swapper/210:15:192
763899191,5cyclictest0-21swapper/209:22:292
7638991913,3cyclictest3514-21needreboot08:10:182
763899191,10cyclictest984-21taskset11:45:202
763899191,10cyclictest8469-21gltestperf07:10:152
763299198,3cyclictest6403-21ntp_states09:30:201
763299191,17cyclictest0-21swapper/108:02:021
763299190,18cyclictest0-21swapper/112:15:571
762599199,3cyclictest0-21swapper/012:20:160
762599198,3cyclictest5719-1kworker/u9:211:35:290
762599198,2cyclictest0-21swapper/011:05:150
762599193,3cyclictest0-21swapper/007:29:230
762599191,3cyclictest9593-21irqstats12:05:160
762599191,10cyclictest5441-21ntp_states11:55:210
762599191,10cyclictest5343-21users10:40:240
762599191,10cyclictest5337-21/usr/sbin/munin09:30:230
762599191,10cyclictest4122-21users08:10:260
762599191,10cyclictest29524-21/usr/sbin/munin10:25:120
762599191,10cyclictest24952-21ntpq07:45:200
763899185,3cyclictest86550irq/126-enp1s0-10:45:182
763899181,7cyclictest21478-21munin-run07:40:012
763899181,7cyclictest21478-21munin-run07:40:012
763899181,6cyclictest0-21swapper/211:20:452
763899181,5cyclictest0-21swapper/210:05:112
7638991814,2cyclictest9369-21gltestperf12:05:142
763899181,3cyclictest0-21swapper/211:25:522
7638991812,3cyclictest8164-21users12:00:262
7638991812,3cyclictest13704-21if_err_enp2s007:20:172
763899181,15cyclictest0-21swapper/210:38:482
7638991811,4cyclictest28997-21sort07:55:162
763899181,14cyclictest28680-21latency_hist09:10:012
763899181,11cyclictest0-21swapper/212:22:082
763899181,11cyclictest0-21swapper/210:29:492
762599189,3cyclictest0-21swapper/011:28:560
762599188,3cyclictest7827-21/usr/sbin/munin07:10:150
762599187,3cyclictest11-21rcuc/011:30:110
762599182,6cyclictest15498-21munin-run08:40:010
762599182,6cyclictest11-21rcuc/010:15:190
762599181,9cyclictest812-21snmpd10:03:020
762599181,9cyclictest812-21snmpd10:03:020
762599181,7cyclictest15126-21strings09:50:180
762599181,5cyclictest0-21swapper/009:21:350
762599181,4cyclictest0-21swapper/012:27:270
762599181,4cyclictest0-21swapper/012:27:270
7625991812,3cyclictest32355-21sed10:30:160
7625991812,3cyclictest18158-21ls08:45:130
7625991812,3cyclictest18158-21ls08:45:130
762599181,15cyclictest0-21swapper/008:59:400
5692180,2sleep00-21swapper/009:19:330
277662180,2sleep00-21swapper/007:51:200
256502181,2sleep20-21swapper/207:47:252
93632171,2sleep20-21swapper/209:38:132
763899176,3cyclictest86850irq/129-enp1s0-08:55:142
763899176,3cyclictest86850irq/129-enp1s0-08:15:142
763899171,8cyclictest0-21swapper/211:42:082
763899171,6cyclictest29093-21cstates09:10:112
763899171,6cyclictest0-21swapper/210:20:192
763899171,3cyclictest0-21swapper/212:37:152
763899171,3cyclictest0-21swapper/210:52:342
763899171,3cyclictest0-21swapper/210:43:052
7638991712,3cyclictest32381-21switchtime09:15:232
7638991712,3cyclictest25654-21grep09:00:232
7638991712,3cyclictest19313-21sed07:35:012
7638991712,3cyclictest19313-21sed07:35:012
7638991712,3cyclictest0-21swapper/211:54:412
7638991711,3cyclictest31743-21sensors_temp08:00:232
763899171,10cyclictest0-21swapper/211:30:402
763899171,10cyclictest0-21swapper/208:32:512
762599176,3cyclictest0-21swapper/007:21:560
762599174,4cyclictest20874-21if_enp2s008:50:160
762599174,4cyclictest17348-21sshd08:40:470
762599171,8cyclictest0-21swapper/007:18:500
762599171,7cyclictest0-21swapper/012:11:440
762599171,6cyclictest9715-21gltestperf08:25:150
762599171,5cyclictest0-21swapper/011:10:020
762599171,14cyclictest0-21swapper/010:20:500
762599171,14cyclictest0-21swapper/009:25:060
762599171,14cyclictest0-21swapper/008:16:020
762599171,10cyclictest0-21swapper/008:30:150
235132171,2sleep20-21swapper/207:43:122
21942171,2sleep20-21swapper/208:08:472
763899165,3cyclictest86750irq/128-enp1s0-09:55:222
763899163,4cyclictest20978-21sed08:50:172
763899162,8cyclictest32-21ksoftirqd/211:05:182
763899162,6cyclictest6923-21/usr/sbin/munin08:20:152
763899161,9cyclictest0-21swapper/207:25:282
763899161,6cyclictest26095-21/usr/sbin/munin07:50:142
763899161,5cyclictest0-21swapper/208:45:522
763899161,5cyclictest0-21swapper/208:45:522
763899161,2cyclictest0-21swapper/211:56:092
7638991612,3cyclictest0-21swapper/208:26:352
7638991611,3cyclictest9911-21cstates09:40:122
7638991611,3cyclictest15009-21sed09:50:182
7638991611,3cyclictest14336-21kernelversion08:35:172
7638991610,4cyclictest13176-21fgrep09:45:242
7638991610,4cyclictest0-21swapper/211:15:242
762599166,8cyclictest7025-21grep08:20:110
762599161,7cyclictest0-21swapper/009:10:090
762599161,2cyclictest0-21swapper/009:00:420
762599161,14cyclictest0-21swapper/007:35:470
762599161,14cyclictest0-21swapper/007:35:470
7625991611,3cyclictest8391-21latency09:35:160
7625991611,3cyclictest28290-21/usr/sbin/munin07:55:220
7625991611,3cyclictest17933-21grep07:30:130
7625991611,3cyclictest17933-21grep07:30:130
7638991510,3cyclictest15943-21expr08:40:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional