You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 12:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Dec 09, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
125592236163,24sleep20-21swapper/219:07:232
108002236164,59sleep30-21swapper/319:05:093
126342204165,25sleep00-21swapper/019:08:200
107982203164,25sleep10-21swapper/119:05:081
2543021500,4sleep01285599cyclictest20:50:130
12867999693,2cyclictest0-21swapper/221:42:222
198282580,3sleep20-21swapper/220:35:212
72622570,1sleep30-21swapper/322:35:183
72622570,1sleep30-21swapper/322:35:183
54112530,4sleep15410-21cat20:05:021
172662530,1sleep10-21swapper/100:10:211
226372510,1sleep00-21swapper/021:55:200
1286799401,27cyclictest18702-21cstates00:15:162
1286799365,20cyclictest17419-21expr21:45:132
1286799361,23cyclictest14097-21/usr/sbin/munin00:05:172
1286799352,3cyclictest86550irq/126-enp1s0-23:30:162
1285599355,14cyclictest7551-21munin-run20:10:020
1286799342,3cyclictest12050irq/123-ahci[0023:45:012
1286799342,3cyclictest12050irq/123-ahci[0023:45:012
1286799342,27cyclictest1857-40dnf22:20:342
1286799341,5cyclictest2449-21/usr/sbin/munin22:25:212
1286799341,5cyclictest19882-21fschecks_time21:50:152
1286799341,19cyclictest86550irq/126-enp1s0-23:25:162
1286799341,19cyclictest10606-21/usr/sbin/munin21:30:182
1286799331,7cyclictest13059-21expr21:35:152
1286799331,7cyclictest13059-21expr21:35:142
1286799331,6cyclictest25796-21idleruntime-cro22:05:022
1286799331,6cyclictest21531-21grep20:40:152
1286799331,6cyclictest11238-21/usr/sbin/munin22:45:172
1286799331,3cyclictest11638-21latency_hist00:00:022
1286799328,20cyclictest15778-21cpuspeed_turbos22:55:162
1286799322,26cyclictest26432-21ntp_states20:50:212
1286799321,2cyclictest0-21swapper/200:40:022
12867993212,17cyclictest25106-21/usr/sbin/munin00:30:302
1286799321,20cyclictest32-21ksoftirqd/222:05:132
1286799321,18cyclictest23838-21users23:10:282
1287299316,17cyclictest22464-40dnf00:22:293
1286799313,24cyclictest28332-40dnf23:21:322
1286799313,23cyclictest14380-21/usr/sbin/munin20:25:292
1286799312,3cyclictest12050irq/123-ahci[0022:55:022
1286799312,25cyclictest5924-21diskstats20:05:162
1286799311,5cyclictest0-21swapper/200:00:172
1286799311,4cyclictest0-21swapper/221:20:142
1286799311,26cyclictest19380-21grep23:00:262
12867993112,16cyclictest2975-21cat23:40:022
1286799307,20cyclictest28381-21irqstats20:55:222
1286799306,20cyclictest16949-21gltestperf00:10:182
1286799305,22cyclictest8740-21df21:25:132
1286799304,22cyclictest5121-21fschecks_time22:30:172
1286799303,15cyclictest86750irq/128-enp1s0-19:30:162
1286799302,23cyclictest2234-21ntp_states19:55:222
1286799302,23cyclictest15066-21munin-node19:15:012
1286799301,3cyclictest27931-21grep19:40:202
1286799301,3cyclictest17863-40dnf19:19:342
1286799301,3cyclictest17276-21idleruntime20:30:162
1286799301,3cyclictest17276-21idleruntime20:30:162
1286799301,3cyclictest12174-21/usr/sbin/munin20:20:192
1286799301,2cyclictest86750irq/128-enp1s0-19:35:192
1286799301,2cyclictest7822-21df_inode23:50:162
1286799301,26cyclictest9744-21latency_hist20:15:002
1286799301,24cyclictest2885-21ntp_states21:10:252
12867993012,15cyclictest21317-21expr00:20:172
1286799301,20cyclictest5279-21/usr/sbin/munin23:45:282
1286799300,5cyclictest0-21swapper/222:10:142
12855993011,16cyclictest18283-21munin-run00:15:010
1287299299,9cyclictest10606-21/usr/sbin/munin21:30:193
1286799296,20cyclictest10045-21netstat22:40:202
1286799292,4cyclictest31747-21munin-run21:05:012
1286799292,23cyclictest4360-21diskstats21:15:152
1286799292,23cyclictest3741-21uniq20:00:162
1286799292,23cyclictest21310-21smart_sda23:05:262
1286799292,22cyclictest23550-21sed00:25:182
1286799291,6cyclictest0-21swapper/223:18:372
1286799291,4cyclictest31128-21/usr/sbin/munin22:15:182
1286799291,4cyclictest11311-21/usr/sbin/munin20:15:242
12867992913,14cyclictest21843-21grep19:25:252
1286799291,2cyclictest31971-21/usr/sbin/munin21:05:132
12867992911,15cyclictest836-21runrttasks19:21:252
1286299297,8cyclictest3509-21fschecks_count23:40:151
1286299297,8cyclictest3509-21fschecks_count23:40:151
1287299280,21cyclictest12132-21cstates00:00:143
1286799284,21cyclictest29197-21cpuspeed_turbos19:45:122
1286799281,4cyclictest24076-21/usr/sbin/munin20:45:202
1286799281,3cyclictest7937-21/usr/sbin/munin22:35:252
1286799281,3cyclictest7937-21/usr/sbin/munin22:35:252
1286799281,3cyclictest21948-21chrt21:55:132
1286799281,2cyclictest31331-21/usr/sbin/munin19:50:172
1286299281,10cyclictest1590-21diskmemload22:30:231
211752270,3sleep081ktimersoftd/000:20:150
1287299278,3cyclictest15868-21sshd00:07:283
1287299276,3cyclictest28849-21/usr/sbin/munin23:25:203
1287299275,14cyclictest2589-21runrttasks23:36:483
1287299271,10cyclictest20489-21idleruntime-cro00:20:023
1286299271,16cyclictest1590-21diskmemload21:49:241
1286299271,10cyclictest0-21swapper/123:30:491
1287299264,7cyclictest0-21swapper/321:50:423
1287299260,18cyclictest17380-21ntp_states00:10:243
1287299260,16cyclictest26040-21users23:15:293
1287299260,16cyclictest0-21swapper/321:25:223
1287299260,15cyclictest0-21swapper/319:37:433
1286299265,8cyclictest1857-40dnf22:20:331
1286299261,9cyclictest0-21swapper/100:03:171
1285599268,6cyclictest3508-21fschecks_count23:40:150
1285599268,6cyclictest3508-21fschecks_count23:40:150
1285599262,11cyclictest28908-21munin-run19:45:020
1285599261,13cyclictest12589-21basename21:35:010
12872992513,2cyclictest22251-21/usr/sbin/munin23:10:143
12872992512,5cyclictest6257-21cpu21:20:153
1287299251,11cyclictest0-21swapper/323:32:103
1287299250,18cyclictest6430-21awk23:45:243
1287299247,8cyclictest1816-21/usr/sbin/munin21:10:153
1287299247,7cyclictest18613-21grep21:45:273
1287299245,8cyclictest7748-21cat20:10:013
1287299245,7cyclictest29766-21/usr/sbin/munin21:00:203
1287299245,6cyclictest28405-21/usr/sbin/munin00:35:233
1287299244,5cyclictest0-21swapper/321:05:293
1287299242,7cyclictest24146-21diskstats22:00:153
1287299242,13cyclictest20108-21latency_hist19:25:013
1287299242,10cyclictest0-21swapper/323:05:143
1287299241,9cyclictest0-21swapper/320:57:283
1287299241,11cyclictest780-21rngd20:20:163
12872992410,5cyclictest86550irq/126-enp1s0-19:30:133
1286299242,17cyclictest0-21swapper/100:19:021
12862992412,6cyclictest0-21swapper/100:24:371
12862992411,3cyclictest17642-21ntp_states20:30:221
12862992411,3cyclictest17642-21ntp_states20:30:221
1286299241,10cyclictest0-21swapper/123:45:171
1286299241,10cyclictest0-21swapper/122:09:141
1286299240,17cyclictest0-21swapper/123:01:331
1285599246,16cyclictest22024-21munin-run23:10:010
1285599242,7cyclictest20522-21cat00:20:010
193542230,4sleep121-21rcuc/120:35:171
1287299235,7cyclictest15132-21taskset22:51:323
1287299232,13cyclictest3645-21processes22:25:253
1287299231,6cyclictest0-21swapper/322:56:413
1287299231,15cyclictest0-21swapper/321:58:443
1287299231,13cyclictest0-21swapper/323:03:013
12872992310,5cyclictest86550irq/126-enp1s0-23:55:193
12872992310,4cyclictest9041-21/usr/sbin/munin22:40:253
1286299239,3cyclictest3141-21munin-run20:00:021
1286299232,5cyclictest0-21swapper/122:20:011
1286299231,9cyclictest0-21swapper/123:36:121
1286299231,9cyclictest0-21swapper/100:05:011
1286299231,3cyclictest0-21swapper/122:55:161
1286299231,2cyclictest21110-21ntpq23:05:221
1286299231,19cyclictest32396-21ntp_states19:50:211
1286299231,16cyclictest0-21swapper/122:11:021
1286299231,15cyclictest0-21swapper/122:37:181
1286299231,15cyclictest0-21swapper/122:37:181
1286299231,12cyclictest0-21swapper/121:40:011
1286299231,12cyclictest0-21swapper/121:40:011
1286299231,11cyclictest29832-21idleruntime19:45:171
1286299231,11cyclictest13730-21hddtemp_smartct19:10:181
1286299231,10cyclictest0-21swapper/122:01:201
1286299231,10cyclictest0-21swapper/121:19:131
1286299230,17cyclictest0-21swapper/121:30:161
1285599235,13cyclictest88150irq/134-enp2s0-20:44:080
1285599232,8cyclictest521-21kworker/0:023:45:330
1285599231,2cyclictest0-21swapper/000:01:320
83162220,3sleep081ktimersoftd/021:25:010
1287299229,2cyclictest26168-21ntp_states00:30:233
1287299221,9cyclictest30407-21/usr/sbin/munin22:15:253
1287299221,7cyclictest18545-21latency_hist20:35:023
1287299221,7cyclictest18545-21latency_hist20:35:013
1287299221,5cyclictest86750irq/128-enp1s0-19:15:203
12872992211,2cyclictest32708-21cpuspeed_turbos22:20:133
1287299221,10cyclictest22305-21latency_hist19:30:013
1286299229,3cyclictest12404-21ps22:45:231
1286299228,4cyclictest581-21systemd-journal20:55:011
1286299228,3cyclictest0-21swapper/122:26:351
1286299227,4cyclictest28510-21sed19:40:281
1286299227,3cyclictest0-21swapper/121:56:031
1286299225,3cyclictest1592-21munin-run21:10:011
1286299222,11cyclictest29763-21memory23:25:201
1286299221,19cyclictest0-21swapper/123:53:411
1286299221,16cyclictest0-21swapper/121:02:521
1286299221,15cyclictest0-21swapper/119:30:341
1286299221,15cyclictest0-21swapper/119:26:351
1286299221,15cyclictest0-21swapper/100:39:181
12862992211,3cyclictest86750irq/128-enp1s0-21:21:461
1286299221,11cyclictest0-21swapper/121:29:001
1286299221,10cyclictest0-21swapper/123:55:181
1286299221,10cyclictest0-21swapper/100:29:091
12862992210,3cyclictest86750irq/128-enp1s0-21:10:201
1285599226,3cyclictest28813-21grep22:10:170
1285599225,13cyclictest88150irq/134-enp2s0-23:12:270
1285599225,13cyclictest86750irq/128-enp1s0-22:59:090
1285599223,3cyclictest32217-21ps23:30:230
1285599221,5cyclictest0-21swapper/021:44:460
1287299215,8cyclictest780-21rngd21:35:043
1287299215,8cyclictest780-21rngd21:35:043
1287299213,2cyclictest20738-21latency_hist20:40:013
1287299211,7cyclictest0-21swapper/322:45:243
1287299211,17cyclictest20972-21/usr/sbin/munin20:40:203
1287299211,12cyclictest0-21swapper/323:42:473
1287299211,12cyclictest0-21swapper/323:42:473
1287299210,7cyclictest5987-21perf21:20:013
1287299210,15cyclictest0-21swapper/321:40:533
1286299212,4cyclictest26421-21munin-run23:20:011
1286299211,19cyclictest0-21swapper/119:23:581
1286299211,18cyclictest0-21swapper/122:50:261
1286299211,18cyclictest0-21swapper/121:52:371
1286299211,11cyclictest21310-21df20:40:141
1286299211,11cyclictest15283-21latency20:25:201
1286299211,11cyclictest10980-21taskset22:44:461
1286299211,10cyclictest7906-21cpuspeed_turbos20:10:131
1286299211,10cyclictest0-21swapper/123:11:091
1286299211,10cyclictest0-21swapper/100:33:021
1286299210,14cyclictest0-21swapper/121:42:361
1285599216,4cyclictest0-21swapper/022:29:350
1285599214,10cyclictest780-21rngd21:26:070
1285599213,9cyclictest12804-21kworker/u8:121:37:010
1285599213,9cyclictest12804-21kworker/u8:121:37:010
1285599211,6cyclictest0-21swapper/022:50:160
1285599211,5cyclictest13209-21idleruntime-cro22:50:000
1285599211,4cyclictest5345-21sh20:05:020
235692201,2sleep00-21swapper/000:25:180
1287299206,7cyclictest86550irq/126-enp1s0-22:09:573
1287299206,5cyclictest86850irq/129-enp1s0-23:20:163
1287299206,5cyclictest780-21rngd19:58:413
1287299206,3cyclictest86550irq/126-enp1s0-23:50:183
12872992012,5cyclictest31108-21idleruntime-cro19:50:013
1287299201,12cyclictest0-21swapper/322:34:283
1287299201,11cyclictest0-21swapper/320:25:223
1287299200,17cyclictest0-21swapper/322:10:233
1287299200,12cyclictest26939-21/usr/sbin/munin19:40:123
1286299206,3cyclictest12992-21irqstats20:20:171
1286299201,9cyclictest26670-21cpu23:20:121
1286299201,3cyclictest0-21swapper/120:55:271
12862992012,2cyclictest0-21swapper/119:15:231
1286299200,18cyclictest0-21swapper/120:45:061
1285599205,4cyclictest0-21swapper/022:00:150
1285599204,8cyclictest24529-21cat23:15:140
1285599202,10cyclictest7657-21irqstats22:35:200
1285599202,10cyclictest7657-21irqstats22:35:200
1285599202,10cyclictest4649-21/usr/sbin/munin22:30:280
1285599201,11cyclictest9674-21/usr/sbin/munin23:55:250
12855992010,3cyclictest31396-21strings22:15:200
12855992010,3cyclictest28849-21/usr/sbin/munin23:25:230
1287299199,2cyclictest86750irq/128-enp1s0-20:10:253
1287299196,7cyclictest0-21swapper/300:28:353
1287299193,2cyclictest23170-21/usr/sbin/munin20:45:263
12872991913,3cyclictest4001-21gltestperf20:00:183
1287299191,10cyclictest31331-21/usr/sbin/munin19:50:213
1287299190,7cyclictest12050irq/123-ahci[0020:55:023
1286299191,17cyclictest0-21swapper/120:16:591
1286299191,16cyclictest0-21swapper/120:05:051
1285599196,3cyclictest11-21rcuc/023:00:250
1285599194,8cyclictest839-21/usr/sbin/munin23:35:140
1285599194,5cyclictest28803-21sendmail_mailqu20:55:280
1285599193,9cyclictest12050irq/123-ahci[0020:50:020
1285599192,9cyclictest28428-21sshd23:22:410
1285599192,5cyclictest25453-21diskstats00:30:150
1285599191,8cyclictest4778-21if_enp2s021:15:190
1285599191,6cyclictest0-21swapper/023:50:260
1285599191,5cyclictest16046-21tr19:15:140
1285599191,4cyclictest0-21swapper/022:20:160
1285599191,10cyclictest14380-21/usr/sbin/munin20:25:180
1285599191,10cyclictest12174-21/usr/sbin/munin20:20:250
242792181,2sleep00-21swapper/019:32:140
1287299184,2cyclictest13057-21/usr/sbin/munin19:10:163
1286299185,2cyclictest23-21ksoftirqd/119:35:301
1285599187,3cyclictest1145-21/usr/sbin/munin19:55:260
1285599186,4cyclictest30138-21kworker/u8:122:42:490
1285599185,3cyclictest1097-21grep21:05:280
1285599183,3cyclictest25726-21needreboot19:35:200
1285599181,5cyclictest0-21swapper/021:46:400
1285599181,4cyclictest29133-21/usr/sbin/munin19:45:230
1285599181,4cyclictest0-21swapper/021:10:140
1285599181,3cyclictest19119-21df_inode20:35:150
1285599181,3cyclictest0-21swapper/022:05:300
1285599181,3cyclictest0-21swapper/019:21:500
1285599181,2cyclictest0-21swapper/000:05:220
12855991812,3cyclictest9964-21seq20:15:050
12855991812,3cyclictest13044-21seq19:10:050
1285599181,15cyclictest27947-21sed00:35:160
1287299178,2cyclictest86550irq/126-enp1s0-20:15:463
1285599176,3cyclictest86750irq/128-enp1s0-19:25:160
1285599175,3cyclictest86750irq/128-enp1s0-20:10:170
1285599175,3cyclictest18088-21grep20:30:260
1285599175,3cyclictest18088-21grep20:30:260
1285599172,5cyclictest31331-21/usr/sbin/munin19:50:250
1285599171,8cyclictest14634-21snmpd21:00:540
1285599171,4cyclictest0-21swapper/021:50:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional