You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-11-12 - 05:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Nov 12, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113782234165,22sleep30-21swapper/319:05:483
113602234161,25sleep00-21swapper/019:05:330
115672214162,39sleep10-21swapper/119:08:171
116002203132,58sleep20-21swapper/219:08:412
1261221810,4sleep01179299cyclictest19:10:150
1179299607,10cyclictest0-21swapper/021:20:010
11802995350,2cyclictest0-21swapper/222:20:452
303602520,2sleep339-21rcuc/319:50:113
8112440,2sleep00-21swapper/022:21:500
195102440,1sleep10-21swapper/100:20:021
1179299431,13cyclictest23748-21awk00:30:010
1179299387,16cyclictest25936-21munin-run00:35:010
11802993734,2cyclictest0-21swapper/223:09:072
1179299361,6cyclictest0-21swapper/023:54:330
1179299352,4cyclictest0-21swapper/021:13:530
11792993410,11cyclictest15899-21awk21:45:020
1179299321,16cyclictest29082-21munin-run22:15:010
1179799311,20cyclictest0-21swapper/123:28:471
1179799311,20cyclictest0-21swapper/122:55:471
1179799311,17cyclictest0-21swapper/121:18:521
1179299314,12cyclictest27853-21idleruntime-cro19:45:010
11942300,4sleep31197-21pmu-power19:55:203
1179799306,12cyclictest0-21swapper/121:33:001
1179799301,22cyclictest0-21swapper/121:25:161
1179799301,21cyclictest0-21swapper/121:51:301
1179799301,21cyclictest0-21swapper/121:51:301
1179799301,19cyclictest0-21swapper/121:43:331
1179799301,18cyclictest0-21swapper/121:48:091
1179299304,12cyclictest0-21swapper/019:30:010
1179299301,14cyclictest28142-21awk00:40:010
1179799296,11cyclictest16484-21grep20:30:181
1179799291,9cyclictest0-21swapper/122:36:091
1179799291,22cyclictest0-21swapper/122:07:451
1179799291,22cyclictest0-21swapper/120:10:171
1179799291,13cyclictest0-21swapper/123:13:431
1179799281,20cyclictest0-21swapper/123:38:321
1179799281,11cyclictest840-21cpuspeed_turbos21:10:111
1179799281,10cyclictest0-21swapper/119:25:121
1179299283,12cyclictest3935-21munin-run23:45:020
11802992716,5cyclictest5647-21unixbench_multi23:45:252
1179799271,9cyclictest0-21swapper/122:51:571
1179799271,3cyclictest0-21swapper/121:23:401
1179799271,20cyclictest0-21swapper/123:03:161
1179799271,20cyclictest0-21swapper/123:03:161
1179799271,17cyclictest0-21swapper/100:08:241
1179799271,12cyclictest0-21swapper/122:32:141
1179799271,12cyclictest0-21swapper/100:39:171
1179799271,11cyclictest19711-21cpuspeed_turbos00:20:131
1179799271,11cyclictest0-21swapper/123:30:191
11809992622,3cyclictest0-21swapper/321:49:063
11809992616,3cyclictest5495-21strings20:05:203
1180999261,4cyclictest32570-21ntp_states22:20:213
1179799267,3cyclictest3331-21cat23:40:221
11797992615,3cyclictest21766-21sensors21:55:221
11797992614,8cyclictest0-21swapper/119:10:201
1179799261,19cyclictest0-21swapper/120:50:191
1179799261,18cyclictest0-21swapper/123:20:381
1179799261,18cyclictest0-21swapper/122:42:321
1179799261,18cyclictest0-21swapper/122:18:281
1179799261,17cyclictest3480-21processes20:00:251
1179799261,17cyclictest0-21swapper/122:25:231
1179799261,16cyclictest23722-21kworker/u8:000:30:061
1179799261,16cyclictest0-21swapper/121:38:131
11797992610,3cyclictest10727-21grep22:45:141
1179799260,16cyclictest0-21swapper/123:15:131
1179299264,10cyclictest19677-21perf20:40:010
1179299261,12cyclictest7097-21munin-run21:25:010
218042250,4sleep10-21swapper/120:44:281
218042250,4sleep10-21swapper/120:44:281
11809992512,8cyclictest0-21swapper/323:58:303
1180999251,16cyclictest0-21swapper/319:18:173
1180999250,22cyclictest0-21swapper/321:19:013
1179799253,13cyclictest344-21sshd21:09:371
1179799252,10cyclictest17271-21cat00:15:021
1179799252,10cyclictest17271-21cat00:15:011
11797992511,9cyclictest0-21swapper/122:24:591
1179799251,14cyclictest0-21swapper/123:06:151
1179299251,12cyclictest14310-21munin-run22:55:010
11792992510,3cyclictest13627-21kworker/0:021:46:280
81162240,2sleep20-21swapper/221:25:172
11809992420,2cyclictest12445-21seq00:01:183
11809992417,4cyclictest20484-21memory00:20:193
1180999241,20cyclictest0-21swapper/323:50:373
1180999241,17cyclictest0-21swapper/300:30:113
1180999241,16cyclictest0-21swapper/323:34:383
1180999241,16cyclictest0-21swapper/321:35:373
1180999240,22cyclictest0-21swapper/321:40:543
1179799243,3cyclictest0-21swapper/123:50:371
1179799243,15cyclictest20013-21kworker/u8:022:02:591
1179799241,3cyclictest0-21swapper/123:59:181
1179799241,16cyclictest0-21swapper/123:47:091
1179799241,16cyclictest0-21swapper/122:10:431
1179799241,11cyclictest0-21swapper/100:25:161
1179799241,10cyclictest2072-21munin-run20:00:021
11797992410,3cyclictest29944-21smart_sda21:00:241
1179299249,3cyclictest24348-21kworker/0:122:02:230
1179299249,3cyclictest14966-21kworker/0:219:21:560
1179299248,3cyclictest21607-21kworker/0:020:46:110
1179299247,4cyclictest26586-21kworker/0:222:25:550
1179299242,10cyclictest8496-21kworker/0:320:20:030
1179299241,11cyclictest2081-21awk20:00:010
1180999234,9cyclictest24353-21runrttasks23:15:203
11809992319,2cyclictest2020-21/usr/sbin/munin23:40:103
11809992318,3cyclictest0-21swapper/323:37:383
1180999231,2cyclictest87550irq/128-enp1s0-22:12:203
1180999231,16cyclictest0-21swapper/321:57:413
1180999231,15cyclictest0-21swapper/322:09:493
1180999231,11cyclictest0-21swapper/322:33:213
1180999230,21cyclictest0-21swapper/322:57:173
1180999230,20cyclictest0-21swapper/300:39:573
1180999230,18cyclictest0-21swapper/323:04:143
1180999230,18cyclictest0-21swapper/323:04:143
1180299231,4cyclictest0-21swapper/223:15:292
1179799239,3cyclictest12025-21smart_sda00:00:231
1179799239,2cyclictest579-21systemd-journal20:10:021
1179799238,3cyclictest19042-21latency_hist19:25:011
1179799231,16cyclictest0-21swapper/119:30:421
1179799231,12cyclictest27772-40dnf19:44:221
1179799231,11cyclictest27551-21ntp_states20:55:211
11797992310,8cyclictest0-21swapper/120:26:101
1179299237,4cyclictest7564-21kworker/0:122:55:470
1179299237,4cyclictest12956-21kworker/0:020:28:470
1179299231,6cyclictest0-21swapper/023:37:230
1179299231,6cyclictest0-21swapper/000:19:480
11809992218,2cyclictest0-21swapper/323:49:063
11809992217,3cyclictest0-21swapper/322:04:223
1180999221,15cyclictest0-21swapper/323:07:423
1180999221,13cyclictest0-21swapper/323:26:373
1180299225,4cyclictest26567-21ls19:40:152
1180299222,7cyclictest32117-21/usr/sbin/munin23:35:192
1179799229,3cyclictest0-21swapper/120:23:321
1179799226,3cyclictest21-21rcuc/119:50:251
1179799221,8cyclictest22085-21/usr/sbin/munin20:45:121
1179799221,15cyclictest0-21swapper/119:18:371
1179799221,10cyclictest24734-21ntpq19:35:231
1179299229,3cyclictest29381-21kworker/0:123:35:000
1179299229,3cyclictest28609-21kworker/0:119:17:310
1179299228,3cyclictest21607-21kworker/0:021:03:190
1179299226,4cyclictest19218-21kworker/0:100:24:110
1179299223,3cyclictest87350irq/126-enp1s0-20:30:130
1179299222,11cyclictest13627-21kworker/0:022:05:130
1179299222,10cyclictest8496-21kworker/0:320:14:350
1179299221,5cyclictest0-21swapper/023:08:250
1179299221,5cyclictest0-21swapper/023:01:160
1179299221,5cyclictest0-21swapper/023:01:160
1179299221,5cyclictest0-21swapper/022:38:430
1179299221,5cyclictest0-21swapper/021:28:440
1179299221,4cyclictest0-21swapper/022:44:470
1179299221,10cyclictest10545-21kworker/0:100:04:040
1180999219,2cyclictest0-21swapper/322:40:213
1180999218,2cyclictest0-21swapper/320:15:243
1180999212,9cyclictest0-21swapper/319:38:223
11809992118,2cyclictest0-21swapper/300:05:213
11809992116,3cyclictest9515-21/usr/sbin/munin21:30:203
11809992116,3cyclictest20086-21sshd21:52:373
11809992116,3cyclictest20086-21sshd21:52:373
1180999211,5cyclictest0-21swapper/300:27:103
1180999211,17cyclictest0-21swapper/300:15:213
1180299213,6cyclictest17375-21/usr/sbin/munin00:15:252
1180299213,4cyclictest25763-21grep20:50:242
11802992115,4cyclictest20979-21grep21:55:142
1180299211,4cyclictest0-21swapper/221:41:322
1180299211,10cyclictest0-21swapper/200:00:372
1179799217,2cyclictest9478-21/usr/sbin/munin20:15:151
1179799211,14cyclictest0-21swapper/119:47:201
1179299218,3cyclictest26189-21kworker/0:221:05:590
1179299218,2cyclictest18943-21kworker/0:319:31:030
1179299216,4cyclictest18943-21kworker/0:319:37:190
1179299215,4cyclictest19352-21kworker/0:220:40:590
1179299215,4cyclictest19352-21kworker/0:220:40:590
1179299212,3cyclictest0-21swapper/000:14:340
1179299212,3cyclictest0-21swapper/000:14:330
1179299212,15cyclictest773-21NetworkManager22:45:010
1179299212,10cyclictest1912-21kworker/0:220:08:350
1179299211,8cyclictest1573-21kworker/0:223:57:400
1179299211,5cyclictest0-21swapper/023:17:210
1179299211,5cyclictest0-21swapper/023:13:210
1179299211,4cyclictest0-21swapper/023:28:130
1179299211,4cyclictest0-21swapper/021:30:580
1180999209,3cyclictest28684-21/usr/sbin/munin21:00:143
1180999201,6cyclictest0-21swapper/322:26:003
11809992016,3cyclictest0-21swapper/321:20:163
11809992015,3cyclictest15991-21irqstats00:10:193
11809992015,3cyclictest15991-21irqstats00:10:193
1180999201,17cyclictest0-21swapper/323:20:203
1180999201,11cyclictest21476-21/usr/sbin/munin19:30:253
1180999201,11cyclictest21133-21/usr/sbin/munin23:10:113
1180999200,19cyclictest0-21swapper/322:35:203
1180299203,6cyclictest0-21swapper/222:47:352
1180299202,6cyclictest0-21swapper/223:27:222
1180299202,4cyclictest22202-21ntp_states23:10:182
1180299202,16cyclictest0-21swapper/219:13:532
1180299201,9cyclictest0-21swapper/200:34:552
1180299201,7cyclictest22112-21diskstats00:25:132
1180299201,7cyclictest0-21swapper/223:30:332
1180299201,6cyclictest32219-21NetworkManager19:55:002
1180299201,6cyclictest23786-21awk22:00:232
1180299201,6cyclictest0-21swapper/221:10:152
1180299201,5cyclictest19061-21if_enp2s021:50:162
1180299201,5cyclictest19061-21if_enp2s021:50:162
1180299201,5cyclictest17679-21unixbench_singl21:45:242
11802992015,3cyclictest10517-21dump-pmu-power00:00:002
1180299201,4cyclictest8846-21memory22:40:182
1180299201,3cyclictest0-21swapper/223:02:242
1180299201,3cyclictest0-21swapper/223:02:242
11802992012,5cyclictest2387-21/usr/sbin/munin22:25:222
1179799201,14cyclictest0-21swapper/120:35:271
1179299202,15cyclictest18918-21grep21:50:160
1179299202,15cyclictest18918-21grep21:50:160
1179299202,10cyclictest24283-21/usr/sbin/munin20:50:180
1179299201,6cyclictest0-21swapper/023:46:110
11792992011,2cyclictest87550irq/128-enp1s0-22:33:070
1179299201,11cyclictest20508-21/usr/sbin/munin21:55:220
1179299200,8cyclictest21607-21kworker/0:020:55:150
1180999199,1cyclictest0-21swapper/320:20:173
1180999198,7cyclictest0-21swapper/322:47:583
1180999195,3cyclictest0-21swapper/319:11:233
1180999191,7cyclictest0-21swapper/321:25:223
11809991917,1cyclictest0-21swapper/319:45:113
1180999191,6cyclictest0-21swapper/322:15:233
11809991916,2cyclictest0-21swapper/320:00:213
11809991913,3cyclictest20920-21ntpq20:40:203
11809991913,3cyclictest20920-21ntpq20:40:193
1180999190,17cyclictest0-21swapper/321:10:053
1180999190,17cyclictest0-21swapper/320:25:203
1180299195,3cyclictest87550irq/128-enp1s0-20:25:252
1180299194,8cyclictest0-21swapper/200:20:552
1180299194,3cyclictest87550irq/128-enp1s0-19:15:182
1180299193,4cyclictest15521-21df_inode00:10:152
1180299193,4cyclictest15521-21df_inode00:10:152
1180299192,3cyclictest87650irq/129-enp1s0-21:10:012
1180299191,7cyclictest0-21swapper/223:23:032
1180299191,6cyclictest8897-21/usr/sbin/munin20:15:132
1180299191,6cyclictest19601-21sshd20:39:272
1180299191,6cyclictest0-21swapper/222:11:162
1180299191,5cyclictest10490-21cat21:30:172
1180299191,4cyclictest3918-21strings21:15:202
1180299191,4cyclictest0-21swapper/220:30:112
1180299191,3cyclictest0-21swapper/222:56:572
1180299191,15cyclictest12775-21ntp_offset21:35:192
1179299195,3cyclictest6428-21kworker/0:020:16:510
1179299191,9cyclictest0-21swapper/019:49:220
1179299191,5cyclictest0-21swapper/023:23:180
1180999186,2cyclictest22085-21/usr/sbin/munin20:45:133
11809991816,1cyclictest0-21swapper/319:20:243
1180999180,9cyclictest0-21swapper/322:50:223
1180299185,2cyclictest87550irq/128-enp1s0-21:20:172
1180299183,3cyclictest18141-21sshd19:20:212
1180299182,6cyclictest0-21swapper/223:42:142
1180299182,4cyclictest0-21swapper/200:05:262
1180299182,3cyclictest28069-21users20:55:252
1180299182,14cyclictest21121-21find20:40:232
1180299182,14cyclictest21121-21find20:40:222
1180299181,5cyclictest32457-21/usr/sbin/munin19:55:232
1180299181,5cyclictest23845-21sshd20:47:262
11802991812,3cyclictest7321-21users22:35:252
11802991812,3cyclictest29940-21gltestperf22:15:132
11802991812,3cyclictest14343-21cat22:55:022
1180299181,14cyclictest21619-21cpuspeed_turbos19:30:152
1179299185,4cyclictest1912-21kworker/0:220:01:390
1179299181,9cyclictest808-21snmpd21:39:350
1179299181,9cyclictest808-21snmpd19:54:130
1179299181,5cyclictest0-21swapper/000:08:160
284432171,2sleep20-21swapper/221:00:022
1180999177,7cyclictest7357-21gltestperf20:10:133
1180999175,2cyclictest28272-21kworker/3:220:30:143
1180999172,13cyclictest28317-21sshd20:58:273
1180999171,3cyclictest0-21swapper/319:26:263
11809991712,3cyclictest25715-21timerwakeupswit20:50:253
1180999171,14cyclictest41-21ksoftirqd/319:40:103
1180999171,14cyclictest0-21swapper/321:05:243
1180299172,14cyclictest0-21swapper/219:28:272
1180299171,6cyclictest0-21swapper/223:52:402
1180299171,6cyclictest0-21swapper/222:34:062
1180299171,6cyclictest0-21swapper/200:37:472
1180299171,5cyclictest0-21swapper/222:05:082
11802991713,2cyclictest13061-21crond20:25:012
1180299171,2cyclictest0-21swapper/220:05:142
11802991710,4cyclictest7863-21ps20:10:192
1179299171,7cyclictest29301-21/usr/sbin/munin22:15:090
1180999160,5cyclictest0-21swapper/320:36:053
1180299164,2cyclictest87550irq/128-enp1s0-20:00:162
1180299161,3cyclictest0-21swapper/219:37:082
11802991612,2cyclictest29289-21gzip19:45:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional