You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-14 - 04:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Jun 14, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
121462246160,72sleep10-21swapper/119:07:441
122972236161,24sleep20-21swapper/219:09:432
121392207134,25sleep00-21swapper/019:07:390
119922200165,23sleep30-21swapper/319:05:463
124129910197,3cyclictest0-21swapper/221:27:442
12412998178,2cyclictest0-21swapper/223:00:252
12412998072,3cyclictest0-21swapper/200:25:462
12412996764,2cyclictest0-21swapper/222:19:302
30112490,4sleep121-21rcuc/123:40:131
12416994012,17cyclictest10126-21/usr/sbin/munin21:30:213
1241699378,3cyclictest22695-21/usr/sbin/munin20:45:233
1241699362,7cyclictest9900-21munin-run21:30:013
1241699355,3cyclictest22143-21fschecks_count23:10:133
1241699354,3cyclictest20848-21diskstats20:40:153
12416993513,19cyclictest13283-21df22:50:153
1241699349,3cyclictest28072-21users19:40:273
1241699345,3cyclictest5750irq/9-acpi23:30:133
1241699341,4cyclictest26049-21latency_hist23:20:023
1241299340,24cyclictest0-21swapper/223:05:552
1239599347,13cyclictest7-21ksoftirqd/020:10:010
1241699333,5cyclictest13933-21diskstats00:05:153
1241699332,5cyclictest32459-21diskstats22:20:143
1241699331,29cyclictest27287-21sshd22:07:313
12416993312,18cyclictest6471-21sort20:05:263
12416993310,3cyclictest31496-21/usr/sbin/munin21:05:173
12416993310,20cyclictest827-21runrttasks22:55:303
12416993310,20cyclictest810-21sshd00:20:313
12416993310,20cyclictest17692-21/usr/sbin/munin19:20:253
1241299331,24cyclictest0-21swapper/220:42:542
1241299331,24cyclictest0-21swapper/220:37:022
1241299330,23cyclictest0-21swapper/219:53:152
1241699327,3cyclictest0-21swapper/323:53:363
1241699323,4cyclictest28038-21smart_sda00:35:243
1241699322,5cyclictest26328-40dnf00:32:253
1241699322,27cyclictest10015-21grep22:40:273
1241699321,4cyclictest0-21swapper/300:04:463
12416993210,2cyclictest11084-21df_inode22:45:143
1241299321,24cyclictest0-21swapper/222:51:192
1241299321,24cyclictest0-21swapper/222:20:352
1241299321,23cyclictest0-21swapper/221:50:032
1241699319,3cyclictest16108-21/usr/sbin/munin20:30:173
1241699316,4cyclictest20255-21/usr/sbin/munin19:25:133
1241699316,3cyclictest87350irq/126-enp1s0-23:55:123
1241699311,3cyclictest12050irq/123-ahci[0020:00:023
12416993113,15cyclictest23932-21grep00:25:253
1241699311,27cyclictest7901-21grep22:35:253
12416993111,17cyclictest23320-21/usr/sbin/munin19:30:233
12416993111,17cyclictest19646-21if_enp1s021:50:173
12416993110,3cyclictest2912-21/usr/sbin/munin20:00:233
12416993110,18cyclictest25007-21grep20:50:133
12416993110,18cyclictest1681-21df_inode21:10:143
1241299311,24cyclictest0-21swapper/222:40:192
1241299311,12cyclictest0-21swapper/200:22:002
1241299310,23cyclictest0-21swapper/222:07:472
1241699307,4cyclictest6118-21crond22:35:013
1241699301,5cyclictest0-21swapper/321:35:203
12416993012,15cyclictest24939-21cut19:35:163
12416993011,16cyclictest3107-21proc_pri22:25:223
12416993010,3cyclictest17757-21latency_hist00:15:013
1241699300,11cyclictest0-21swapper/319:45:213
1241299301,9cyclictest0-21swapper/219:58:112
1241299301,22cyclictest0-21swapper/223:14:462
1241299301,18cyclictest0-21swapper/221:57:192
1241299301,18cyclictest0-21swapper/219:14:062
1241299301,14cyclictest0-21swapper/219:35:352
1241299301,13cyclictest0-21swapper/222:46:552
1241299300,22cyclictest0-21swapper/221:40:422
1241299300,22cyclictest0-21swapper/220:01:032
1241699299,2cyclictest827-21runrttasks23:23:433
1241699298,3cyclictest16634-21grep19:15:213
1241699297,2cyclictest13911-21/usr/sbin/munin20:25:183
1241699295,4cyclictest0-21swapper/321:15:163
1241699295,3cyclictest0-21swapper/321:40:143
1241699295,3cyclictest0-21swapper/321:20:143
1241699294,3cyclictest28037-21/usr/sbin/munin20:55:193
1241699293,5cyclictest24579-21smart_sda22:00:253
1241699293,5cyclictest24579-21smart_sda22:00:253
1241699292,3cyclictest28067-21diskstats22:10:113
1241699292,3cyclictest20390-21cat20:40:023
1241699291,3cyclictest87350irq/126-enp1s0-00:15:133
12416992911,15cyclictest17997-21head23:00:163
12416992910,16cyclictest8143-21sed20:10:183
12412992920,8cyclictest0-21swapper/223:28:132
1241299291,9cyclictest0-21swapper/223:35:102
1241299291,13cyclictest0-21swapper/220:53:022
1241299291,13cyclictest0-21swapper/220:06:432
1241299291,13cyclictest0-21swapper/219:32:012
1241299291,12cyclictest0-21swapper/220:25:422
1241299291,12cyclictest0-21swapper/219:24:432
1241299290,13cyclictest0-21swapper/223:50:062
1240699291,13cyclictest0-21swapper/120:27:561
1241699289,2cyclictest827-21runrttasks20:22:553
1241699288,3cyclictest13480-21latency19:10:183
1241699286,3cyclictest0-21swapper/321:00:223
1241699285,7cyclictest17363-21gltestperf21:45:153
1241699284,4cyclictest909-21sssd_nss20:15:153
1241699282,4cyclictest6029-21smart_sda23:45:243
12416992811,14cyclictest29918-21/usr/sbin/munin22:15:213
1241299281,9cyclictest0-21swapper/220:15:072
1241299281,12cyclictest0-21swapper/220:13:302
1241299281,11cyclictest17021-21memory20:30:182
1241299281,11cyclictest0-21swapper/220:20:062
1241299281,10cyclictest0-21swapper/219:19:142
12412992810,3cyclictest0-21swapper/219:40:272
1241299280,13cyclictest0-21swapper/221:10:102
1241299280,12cyclictest0-21swapper/221:20:462
12395992812,13cyclictest7707-21munin-run21:25:020
1241699279,3cyclictest827-21runrttasks23:40:043
1241699278,3cyclictest28367-21cpu23:25:123
1241699271,12cyclictest0-21swapper/323:39:573
1241299271,13cyclictest0-21swapper/223:21:262
1241299271,11cyclictest8134-21chrt22:37:472
1241299271,10cyclictest0-21swapper/219:47:592
1241299270,13cyclictest0-21swapper/221:38:322
1241299270,11cyclictest0-21swapper/222:33:192
1241299270,11cyclictest0-21swapper/220:56:382
1240699274,11cyclictest11737-40dnf21:30:351
92922261,21sleep09483-21grep21:25:260
1241699263,2cyclictest87650irq/129-enp1s0-23:05:243
1241299261,11cyclictest0-21swapper/221:17:182
1241299261,11cyclictest0-21swapper/220:47:502
1241299261,10cyclictest0-21swapper/223:41:222
1241299261,10cyclictest0-21swapper/221:48:312
1241299260,9cyclictest0-21swapper/223:33:222
1241299260,17cyclictest0-21swapper/223:45:382
1241299260,13cyclictest0-21swapper/200:05:102
1241299260,12cyclictest0-21swapper/200:31:432
1241299260,11cyclictest0-21swapper/200:16:262
1241299260,10cyclictest0-21swapper/222:01:312
1241299260,10cyclictest0-21swapper/222:01:312
1241299260,10cyclictest0-21swapper/221:04:582
1239599267,16cyclictest13686-21munin-run20:25:010
1239599263,3cyclictest0-21swapper/000:07:000
12395992613,3cyclictest31272-21awk21:05:020
12395992610,13cyclictest18080-21munin-run20:35:000
1241699257,3cyclictest32320-21cat19:50:283
1241699251,4cyclictest0-21swapper/321:55:203
1241299251,6cyclictest0-21swapper/222:10:152
1241299250,9cyclictest0-21swapper/222:58:352
1241299250,12cyclictest0-21swapper/223:16:102
1241299250,10cyclictest0-21swapper/200:13:342
1239599259,13cyclictest12734-21awk22:50:000
1239599251,12cyclictest22473-21munin-run20:45:010
1241299248,3cyclictest0-21swapper/221:08:382
1241299247,3cyclictest0-21swapper/200:35:232
1241299240,12cyclictest0-21swapper/221:32:302
1241299240,11cyclictest0-21swapper/222:27:452
1241299240,11cyclictest0-21swapper/200:03:362
1241299240,10cyclictest0-21swapper/223:57:542
1240699242,12cyclictest20276-21munin-run20:40:001
12406992418,3cyclictest23448-21ntp_states00:25:221
1240699241,17cyclictest0-21swapper/123:01:541
1240699240,17cyclictest0-21swapper/121:00:191
270912231,3sleep127243-21tr20:55:131
1240699234,4cyclictest16986-21sshd22:58:311
1240699232,11cyclictest1731-21munin-run22:25:021
12406992314,3cyclictest0-21swapper/120:05:581
12406992312,3cyclictest29749-21ntp_states19:45:211
1240699231,16cyclictest0-21swapper/121:27:061
1240699231,16cyclictest0-21swapper/119:40:391
1239599232,4cyclictest0-21swapper/021:30:350
1240699221,9cyclictest5832-21ntp_states23:45:211
1240699221,3cyclictest14805-21irqstats19:15:031
1240699221,14cyclictest0-21swapper/122:42:151
1240699221,13cyclictest0-21swapper/122:05:321
1240699220,16cyclictest0-21swapper/121:35:061
1240699220,15cyclictest0-21swapper/122:10:181
1240699220,15cyclictest0-21swapper/120:00:101
1239599224,15cyclictest5506-21munin-run21:20:020
88712210,2sleep10-21swapper/123:54:191
1241299210,9cyclictest0-21swapper/219:25:072
1240699212,15cyclictest7952-21grep20:10:151
1240699212,14cyclictest0-21swapper/119:51:301
1240699211,9cyclictest0-21swapper/122:50:471
12406992117,3cyclictest0-21swapper/120:30:501
1240699211,2cyclictest0-21swapper/123:23:331
1240699211,15cyclictest0-21swapper/121:24:341
1240699211,14cyclictest0-21swapper/100:22:501
1240699211,11cyclictest18846-21latency00:15:181
1240699211,11cyclictest13314-21idleruntime19:10:161
12406992110,3cyclictest12779-21ntp_states20:20:221
1239599216,6cyclictest0-21swapper/023:20:270
1239599215,2cyclictest87350irq/126-enp1s0-22:07:380
1239599213,3cyclictest0-21swapper/022:25:500
12395992116,3cyclictest20689-21head23:05:210
85102200,2sleep10-21swapper/122:40:011
280342200,2sleep1221ktimersoftd/100:35:241
179782201,3sleep00-21swapper/000:15:060
1240699209,3cyclictest32219-40dnf23:31:321
1240699205,8cyclictest12437-21sshd22:46:321
1240699202,16cyclictest0-21swapper/119:30:071
12406992016,3cyclictest0-21swapper/120:52:281
12406992014,4cyclictest30062-21tr22:15:131
1240699201,3cyclictest0-21swapper/122:00:251
1240699201,3cyclictest0-21swapper/122:00:251
1240699201,14cyclictest0-21swapper/123:28:561
1240699201,14cyclictest0-21swapper/120:15:311
1240699201,11cyclictest2302-21users19:55:261
1240699201,11cyclictest12258-21latency00:00:181
1239599209,3cyclictest25905-21latency_hist23:20:010
1239599209,3cyclictest17342-21/usr/sbin/munin23:00:230
1239599207,4cyclictest0-21swapper/021:44:280
1239599206,5cyclictest0-21swapper/000:14:420
1239599205,2cyclictest21999-21kworker/0:100:38:140
1239599203,7cyclictest0-21swapper/023:28:090
1239599203,3cyclictest0-21swapper/022:19:220
1239599202,2cyclictest0-21swapper/023:54:140
1239599201,6cyclictest0-21swapper/022:23:060
1239599201,5cyclictest0-21swapper/000:04:170
1239599201,4cyclictest0-21swapper/022:38:410
12395992012,5cyclictest29911-21fgrep19:45:230
1239599201,11cyclictest30880-21/usr/sbin/munin19:50:150
1239599201,11cyclictest23662-21df_inode22:00:140
1239599201,11cyclictest23662-21df_inode22:00:140
1239599201,11cyclictest16724-21/usr/sbin/munin21:45:150
1239599200,6cyclictest0-21swapper/000:33:390
1240699194,8cyclictest0-21swapper/100:10:211
1240699193,6cyclictest2908050kworker/u8:121:15:191
1240699192,3cyclictest0-21swapper/122:28:151
1240699192,16cyclictest0-21swapper/123:56:011
1240699192,16cyclictest0-21swapper/119:27:271
1240699192,16cyclictest0-21swapper/100:08:381
1240699191,8cyclictest0-21swapper/100:31:361
12406991916,2cyclictest0-21swapper/121:10:081
12406991915,3cyclictest0-21swapper/121:51:561
12406991915,3cyclictest0-21swapper/121:06:181
1240699191,16cyclictest0-21swapper/123:05:261
1240699191,16cyclictest0-21swapper/120:40:451
1240699191,10cyclictest23963-21egrep23:15:111
1240699191,10cyclictest23281-21unixbench_singl23:10:231
1240699191,10cyclictest23043-21diskstats20:45:131
1240699190,14cyclictest0-21swapper/121:44:021
1240699190,14cyclictest0-21swapper/119:39:551
1239599196,5cyclictest16625-21pmu-power19:15:220
1239599196,5cyclictest0-21swapper/021:53:050
1239599196,4cyclictest0-21swapper/000:21:510
1239599194,3cyclictest4201-21cpu22:30:120
1239599192,10cyclictest19889-21/usr/sbin/munin19:25:190
1239599191,15cyclictest18364-21hddtemp_smartct19:20:160
1239599191,12cyclictest0-21swapper/023:55:250
1239599191,10cyclictest2229-21sshd23:38:420
1239599191,10cyclictest13187-21sort21:35:180
1239599191,10cyclictest11197-21sshd20:16:310
1239599191,10cyclictest0-21swapper/022:59:150
1239599190,8cyclictest0-21swapper/023:12:270
1239599190,6cyclictest0-21swapper/023:45:510
1240699187,2cyclictest1484-21ls23:35:231
1240699182,15cyclictest0-21swapper/122:31:111
12406991814,3cyclictest0-21swapper/121:45:341
1240699181,16cyclictest0-21swapper/121:56:201
1240699180,16cyclictest0-21swapper/119:20:211
1239599188,6cyclictest0-21swapper/020:29:560
1239599184,4cyclictest2931-21cpu20:00:100
1239599182,15cyclictest23815-21kworker/0:219:35:380
1239599181,6cyclictest0-21swapper/020:35:220
1239599181,6cyclictest0-21swapper/000:28:040
1239599181,5cyclictest29378-21runrttasks22:11:070
1239599181,11cyclictest0-21swapper/023:40:560
1239599181,10cyclictest0-21swapper/022:50:190
1239599180,11cyclictest0-21swapper/022:43:110
1239599172,12cyclictest24892-21/usr/sbin/munin20:50:150
1239599171,6cyclictest14102-21unixbench_multi19:10:250
1239599171,5cyclictest0-21swapper/021:57:340
12395991711,3cyclictest9471-21sort20:15:010
12395991711,3cyclictest30532-21/usr/sbin/munin23:30:130
1239599171,11cyclictest0-21swapper/021:12:400
1239599161,3cyclictest0-21swapper/021:00:020
1239599161,14cyclictest0-21swapper/019:30:090
12395991611,3cyclictest23294-21expr20:45:150
12395991611,3cyclictest1979-21smartctl19:55:240
1239599159,3cyclictest31496-21/usr/sbin/munin21:05:220
12395991510,3cyclictest26482-21/usr/sbin/munin19:40:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional