You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-21 - 13:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Sun Dec 21, 2025 00:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4408998521,42cyclictest16219-21runrttasks19:40:300
4408998318,50cyclictest26090-21sed20:10:050
4408998221,50cyclictest0-21swapper/022:25:050
4408998220,42cyclictest0-21swapper/021:15:290
4408998218,50cyclictest27838-21cron20:15:050
440899817,70cyclictest0-21swapper/023:35:100
4408998011,26cyclictest0-21swapper/019:30:200
4408997921,45cyclictest14713-21ls19:35:340
4408997919,47cyclictest12611-21/usr/sbin/munin00:00:230
4408997919,44cyclictest3586-21/usr/sbin/munin22:05:290
4408997919,44cyclictest3586-21/usr/sbin/munin22:05:290
440999787,34cyclictest0-21swapper/123:35:101
4408997817,42cyclictest30135-21forks20:20:240
4408997816,46cyclictest0-21swapper/023:40:250
4408997719,45cyclictest0-21swapper/023:15:150
4408997719,45cyclictest0-21swapper/023:15:150
4408997719,33cyclictest0-21swapper/000:36:480
4408997719,13cyclictest15431-21grep00:05:350
4408997718,47cyclictest0-21swapper/021:59:530
4408997715,49cyclictest22607-21grep00:25:360
4408997715,49cyclictest22607-21grep00:25:360
4408997619,41cyclictest0-21swapper/021:40:310
4408997615,49cyclictest0-21swapper/023:25:050
4408997612,54cyclictest0-21swapper/022:50:300
4408997519,43cyclictest10663-21fschecks_time22:25:230
4408997519,41cyclictest8285-21tr20:50:160
4408997518,46cyclictest19444-21chrt00:19:580
4408997420,45cyclictest28321-21/usr/sbin/munin20:15:250
4408997420,41cyclictest6067-21taskset22:11:070
4408997419,44cyclictest10056-21chrt23:51:370
4408997322,7cyclictest11424-21munin-run19:30:050
4408997322,41cyclictest9756-21latency_hist19:25:060
4408997322,41cyclictest9756-21latency_hist19:25:060
4408997319,40cyclictest0-21swapper/020:36:410
4408997316,49cyclictest0-21swapper/023:01:170
4408997314,48cyclictest0-21swapper/022:40:270
4408997222,35cyclictest0-21swapper/019:10:080
4408997219,42cyclictest0-21swapper/023:10:050
4408997216,42cyclictest0-21swapper/023:48:500
298712720,4sleep029873-21grep21:50:060
4409997130,20cyclictest12270-21latency_hist00:00:051
4409997130,20cyclictest12270-21latency_hist00:00:051
4409997130,11cyclictest27564-21awk21:40:341
440899713,51cyclictest0-21swapper/022:00:170
4408997127,24cyclictest12240-21/usr/sbin/munin22:30:360
4408997116,40cyclictest0-21swapper/022:36:140
4408997115,49cyclictest0-21swapper/000:30:520
4408997111,14cyclictest0-21swapper/020:30:180
4409997032,26cyclictest20165-21/usr/sbin/munin22:50:301
4408997016,20cyclictest0-21swapper/021:15:060
4408996919,47cyclictest128-21systemd-journal22:47:260
4408996916,41cyclictest20338-21taskset19:51:320
4408996913,44cyclictest0-21swapper/020:00:250
4409996833,13cyclictest18009-21/usr/sbin/munin21:15:301
4408996821,36cyclictest0-21swapper/022:58:290
4408996818,29cyclictest1125-21runrttasks21:29:210
4408996817,40cyclictest7530-21taskset19:16:530
4408996815,38cyclictest0-21swapper/021:20:420
4408996812,48cyclictest0-21swapper/019:45:500
4408996812,48cyclictest0-21swapper/019:45:500
4409996728,11cyclictest28483-21apt-get23:15:181
4409996728,11cyclictest28483-21apt-get23:15:181
4408996716,44cyclictest0-21swapper/021:31:470
4409996629,12cyclictest15729-21seq00:08:481
4408996617,36cyclictest934-21chrt23:26:500
4408996616,39cyclictest0-21swapper/023:32:240
4408996614,43cyclictest0-21swapper/021:09:360
4408996612,45cyclictest0-21swapper/021:50:170
4408996518,35cyclictest11322-21chrt20:56:040
4408996517,45cyclictest0-21swapper/020:44:530
4408996517,45cyclictest0-21swapper/020:44:530
4408996516,35cyclictest13382-21chrt21:04:210
4409996427,12cyclictest16521-21unixbench-2d19:40:331
4409996426,24cyclictest8437-21/usr/sbin/munin22:20:201
4409996426,12cyclictest1244-21wget22:00:181
4409996424,14cyclictest26114-21latency_hist20:10:051
4408996416,37cyclictest0-21swapper/020:28:340
4408996415,36cyclictest0-21swapper/023:57:140
4408996415,36cyclictest0-21swapper/023:57:140
206662640,2sleep10-21swapper/100:20:321
4409996334,21cyclictest0-21swapper/119:10:081
4409996331,9cyclictest10012-21apt-get22:25:071
4409996323,20cyclictest22163-21runrttasks19:56:471
4408996318,32cyclictest6478-21acpi20:45:170
4409996226,22cyclictest31329-21runrttasks20:23:281
4408996220,28cyclictest0-21swapper/000:22:300
4408996218,34cyclictest0-21swapper/000:11:330
4408996215,35cyclictest0-21swapper/022:19:290
4408996214,37cyclictest0-21swapper/023:12:280
4409996135,10cyclictest5635-21irqstats23:40:251
4409996111,11cyclictest7950irq/56-dwc_otg_22:55:191
4409996031,11cyclictest13757-21sshd00:02:491
4409996029,12cyclictest14730-21sed19:35:351
4409996028,16cyclictest17160-21latency_hist21:15:061
4409995925,14cyclictest21531-21basename00:25:211
4409995925,14cyclictest21531-21basename00:25:211
4408995925,23cyclictest22358-21taskset19:59:110
440999584,9cyclictest7950irq/56-dwc_otg_22:45:181
4409995828,10cyclictest26604-21taskset00:39:171
4409995729,11cyclictest11752-21/usr/sbin/munin19:30:231
4409995725,13cyclictest11475-21latency_hist19:30:051
4409995721,10cyclictest0-21swapper/122:05:371
4409995721,10cyclictest0-21swapper/122:05:371
4408995715,23cyclictest0-21swapper/021:37:230
4409995627,13cyclictest17144-21taskset22:41:471
4409995625,15cyclictest4149-21seq20:36:401
4409995625,13cyclictest8491-21awk20:50:201
4409995624,19cyclictest20555-21chrt21:23:341
13652560,2sleep20-21swapper/223:30:062
4409995528,11cyclictest8463-21sshd23:49:151
4409995527,18cyclictest6033-21chrt20:42:331
4409995527,18cyclictest6033-21chrt20:42:331
4409995527,12cyclictest2316-21taskset20:31:171
4409995525,12cyclictest6283-21chrt22:13:531
4409995524,20cyclictest24001-21taskset20:02:121
4409995524,11cyclictest8148-21taskset22:19:311
4409995519,12cyclictest28486-21fschecks_count21:45:221
229842550,2sleep20-21swapper/221:30:232
440999542,9cyclictest0-21swapper/121:05:161
4409995425,16cyclictest9629-21awk19:25:061
4409995425,16cyclictest9629-21awk19:25:061
4409995423,14cyclictest0-21swapper/121:37:281
4409995422,19cyclictest7950irq/56-dwc_otg_23:00:341
4409995420,11cyclictest19644-21latency_hist00:20:061
4409995321,12cyclictest13356-21chrt21:04:021
39202533,42sleep30-21swapper/319:07:233
4409995223,20cyclictest15339-21taskset22:36:411
4409995220,10cyclictest7850irq/56-dwc_otg_23:22:421
4409995141,7cyclictest17270-21apt-get19:45:171
4409995141,7cyclictest17270-21apt-get19:45:171
4409995034,14cyclictest30254-21/usr/sbin/munin21:50:211
4409995031,15cyclictest29860-21sed20:20:061
4409995026,11cyclictest24047-21runrttasks21:32:131
4409995024,15cyclictest2993-21chrt23:34:561
4409995023,16cyclictest1157-21chrt23:29:361
4409995017,21cyclictest6475-21/usr/sbin/munin20:45:171
4409995017,12cyclictest0-21swapper/119:17:181
40582503,11sleep00-21swapper/019:09:060
4409994922,12cyclictest11303-21taskset20:55:491
4409994920,12cyclictest13673-21runrttasks22:33:271
4409994918,19cyclictest0-21swapper/120:26:121
441199482,8cyclictest0-21swapper/321:00:183
4409994832,13cyclictest23974-21/usr/sbin/munin00:30:281
4409994823,11cyclictest1124-21snmpd23:12:311
4409994819,12cyclictest26201-21sshd23:07:161
4409994819,10cyclictest0-21swapper/123:51:411
4409994732,12cyclictest16302-21df00:10:211
4409994730,13cyclictest21586-21/usr/sbin/munin21:25:301
4409994717,17cyclictest20547-21chrt19:54:071
441199467,23cyclictest0-21swapper/321:50:193
441199462,8cyclictest7950irq/56-dwc_otg_21:25:223
441099467,24cyclictest0-21swapper/219:40:202
441199457,24cyclictest0-21swapper/319:35:103
441199447,23cyclictest0-21swapper/319:20:193
441199447,23cyclictest0-21swapper/319:20:193
441199445,24cyclictest0-21swapper/323:20:213
441099447,15cyclictest0-21swapper/223:35:102
441099441,13cyclictest0-21swapper/219:55:192
4409994419,10cyclictest0-21swapper/121:55:231
441099437,5cyclictest0-21swapper/220:35:202
441099433,3cyclictest0-21swapper/219:20:172
441099433,3cyclictest0-21swapper/219:20:172
441199427,24cyclictest0-21swapper/322:20:193
441199427,22cyclictest0-21swapper/300:05:103
441099428,8cyclictest7950irq/56-dwc_otg_23:30:192
441099426,22cyclictest0-21swapper/200:10:192
4409994221,11cyclictest26265-21runrttasks20:10:071
266022420,2sleep30-21swapper/323:10:063
4410994120,5cyclictest31-21ksoftirqd/222:00:062
4410994118,8cyclictest18392-21/usr/sbin/munin21:15:352
4410994013,17cyclictest301ktimersoftd/200:08:242
26642400,5sleep20-21swapper/219:05:192
441199389,4cyclictest0-21swapper/320:25:383
441199381,25cyclictest28761-21latency21:45:263
441199377,22cyclictest0-21swapper/322:25:103
441199377,20cyclictest0-21swapper/320:45:193
441099375,18cyclictest7950irq/56-dwc_otg_21:25:062
441099374,5cyclictest7850irq/56-dwc_otg_22:15:302
441199367,22cyclictest0-21swapper/321:20:193
441199366,19cyclictest0-21swapper/319:40:203
441199366,15cyclictest7850irq/56-dwc_otg_21:45:063
441099363,17cyclictest7850irq/56-dwc_otg_23:45:252
4410993630,4cyclictest29732-21ls20:20:062
441099362,2cyclictest0-21swapper/200:35:172
441099361,9cyclictest11248-21grep20:55:392
4410993614,6cyclictest23825-21grep20:00:362
441099361,23cyclictest11960-21ls19:30:222
441199356,14cyclictest7850irq/56-dwc_otg_23:55:233
441199356,14cyclictest7850irq/56-dwc_otg_23:55:233
441199355,5cyclictest13526-21latency_hist19:35:063
441199351,28cyclictest9242-21ls20:50:313
441199351,22cyclictest24908-21uniq21:35:233
441099357,19cyclictest0-21swapper/222:25:102
441099355,7cyclictest7950irq/56-dwc_otg_22:20:332
441099355,21cyclictest7950irq/56-dwc_otg_20:20:182
4410993518,6cyclictest9171-21if_err_wlan023:50:222
4410993517,6cyclictest26008-21unixbench_singl23:05:352
441099351,3cyclictest7950irq/56-dwc_otg_19:10:192
4410993512,14cyclictest301ktimersoftd/222:05:292
4410993512,14cyclictest301ktimersoftd/222:05:292
441199347,7cyclictest7850irq/56-dwc_otg_21:05:313
441199347,15cyclictest0-21swapper/322:30:113
441199342,22cyclictest21144-21cut19:55:243
441199341,18cyclictest1216-21latency_hist23:30:063
4411993410,11cyclictest7950irq/56-dwc_otg_00:25:063
441099346,3cyclictest0-21swapper/219:35:112
441099342,2cyclictest0-21swapper/222:10:172
4410993419,5cyclictest14704-21timerwakeupswit21:05:322
4410993413,6cyclictest635650wget19:15:212
441099341,22cyclictest19432-21ls19:50:242
101872340,2sleep30-21swapper/320:55:233
441199337,23cyclictest0-21swapper/322:35:203
4411993322,8cyclictest14125-21apt-key00:05:063
441199332,12cyclictest7850irq/56-dwc_otg_23:15:063
441199331,6cyclictest7950irq/56-dwc_otg_19:50:183
441199331,5cyclictest7750irq/56-dwc_otg20:30:353
441199331,21cyclictest6827-21ls22:15:203
441099338,5cyclictest7750irq/56-dwc_otg22:35:352
441099334,15cyclictest7950irq/56-dwc_otg_23:00:062
441099333,7cyclictest7850irq/56-dwc_otg_21:15:062
441099333,6cyclictest0-21swapper/223:55:172
441099333,6cyclictest0-21swapper/223:55:172
441099332,19cyclictest0-21swapper/221:47:262
33232335,15sleep13403-21/usr/sbin/munin19:05:311
441199328,14cyclictest7950irq/56-dwc_otg_20:00:343
441199325,6cyclictest7850irq/56-dwc_otg_22:45:323
441199324,6cyclictest10997-21grep19:25:373
441199324,15cyclictest0-21swapper/320:15:263
441199323,7cyclictest24609-21wget20:05:203
4411993224,6cyclictest6275-21grep19:15:203
441199321,8cyclictest22076-21/usr/sbin/munin22:55:323
441199321,21cyclictest2227-21timerwakeupswit22:00:323
441099325,12cyclictest7850irq/56-dwc_otg_23:15:232
441099325,12cyclictest7850irq/56-dwc_otg_23:15:232
441099323,21cyclictest0-21swapper/222:00:172
4410993224,6cyclictest301ktimersoftd/222:30:182
441099321,22cyclictest27541-21ls23:10:322
4410993210,11cyclictest1125-21runrttasks20:27:432
441199318,5cyclictest7850irq/56-dwc_otg_20:25:053
441199316,5cyclictest7850irq/56-dwc_otg_00:10:303
441199314,15cyclictest16386-21switchtime21:10:323
441199313,5cyclictest7850irq/56-dwc_otg_23:50:213
4411993120,7cyclictest5438-21fschecks_count23:40:233
441199311,6cyclictest7750irq/56-dwc_otg23:15:273
441199311,6cyclictest7750irq/56-dwc_otg23:15:273
441199311,6cyclictest7750irq/56-dwc_otg00:30:283
4411993112,4cyclictest3959-21missed_timers23:35:293
441099318,5cyclictest7750irq/56-dwc_otg21:25:362
441099317,5cyclictest21419-21ls00:25:062
441099315,6cyclictest31-21ksoftirqd/222:50:052
441099315,5cyclictest31-21ksoftirqd/223:40:232
441099313,15cyclictest1125-21runrttasks21:53:032
441099312,7cyclictest26729-21grep21:40:232
441099312,21cyclictest31-21ksoftirqd/222:44:522
441099311,6cyclictest13142-21/usr/sbin/munin00:00:312
441199307,5cyclictest7950irq/56-dwc_otg_00:27:473
441199307,5cyclictest7950irq/56-dwc_otg_00:27:473
441199305,6cyclictest7850irq/56-dwc_otg_00:15:333
441199305,5cyclictest7850irq/56-dwc_otg_22:05:193
441199305,5cyclictest7850irq/56-dwc_otg_22:05:193
441199303,14cyclictest7850irq/56-dwc_otg_22:50:333
441199302,8cyclictest0-21swapper/322:10:213
441199302,19cyclictest0-21swapper/319:49:413
441199302,19cyclictest0-21swapper/319:49:413
441199302,14cyclictest0-21swapper/321:33:503
441199301,8cyclictest3397-21latency20:35:273
4411993014,11cyclictest0-21swapper/319:10:083
441099307,18cyclictest0-21swapper/220:40:192
441099307,18cyclictest0-21swapper/220:40:192
4410993023,5cyclictest31-21ksoftirqd/220:30:302
3342300,2sleep30-21swapper/321:55:323
441199297,20cyclictest0-21swapper/322:40:193
441199296,12cyclictest7850irq/56-dwc_otg_23:00:333
441199292,13cyclictest0-21swapper/300:38:193
441099295,14cyclictest7950irq/56-dwc_otg_19:49:592
441099295,14cyclictest7950irq/56-dwc_otg_19:49:592
441099292,5cyclictest21038-21munin-run22:55:052
441099291,6cyclictest0-21swapper/223:05:062
441099291,24cyclictest0-21swapper/220:10:222
441099291,21cyclictest0-21swapper/220:54:212
441099291,11cyclictest7750irq/56-dwc_otg19:30:052
441099291,10cyclictest0-21swapper/200:17:502
20402290,2sleep30-21swapper/323:30:273
441199285,5cyclictest7850irq/56-dwc_otg_21:15:323
441199282,12cyclictest7950irq/56-dwc_otg_20:40:213
441199282,12cyclictest7950irq/56-dwc_otg_20:40:213
441099287,4cyclictest7950irq/56-dwc_otg_23:20:352
441099282,13cyclictest7950irq/56-dwc_otg_21:03:242
441099281,4cyclictest7750irq/56-dwc_otg20:05:262
441099281,17cyclictest21753-21cut00:25:252
441099281,17cyclictest21753-21cut00:25:252
441099281,16cyclictest25744-21sed21:35:342
441099281,13cyclictest7750irq/56-dwc_otg20:48:352
441099271,17cyclictest0-21swapper/200:33:552
441199253,4cyclictest27597-21sshd20:12:043
441199251,6cyclictest8708-21latency_hist23:50:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional