You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 08:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Dec 10, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
40322102920,8sleep22197014-21PK-Backend23:50:212
162399910,9rtkit-daemon1622-21rtkit-daemon19:27:211
162399910,8rtkit-daemon1622-21rtkit-daemon19:19:521
162399910,12rtkit-daemon0-21swapper/120:34:511
162399850,9rtkit-daemon1753-21gnome-shell21:54:321
162399850,12rtkit-daemon26-21ksoftirqd/123:44:461
162399830,6rtkit-daemon0-21swapper/122:35:341
162399800,6rtkit-daemon0-21swapper/122:54:231
162399800,5rtkit-daemon0-21swapper/100:26:241
162399790,5rtkit-daemon0-21swapper/121:27:321
162399790,5rtkit-daemon0-21swapper/100:13:301
162399780,5rtkit-daemon0-21swapper/123:34:561
162399780,5rtkit-daemon0-21swapper/122:42:151
162399770,5rtkit-daemon0-21swapper/123:36:171
162399770,5rtkit-daemon0-21swapper/121:15:091
162399770,5rtkit-daemon0-21swapper/120:37:261
393172227662,9sleep20-21swapper/219:06:472
393185927541,9sleep10-21swapper/119:08:351
162399750,5rtkit-daemon0-21swapper/100:19:311
162399750,4rtkit-daemon0-21swapper/122:57:101
162399740,5rtkit-daemon0-21swapper/122:06:441
162399740,5rtkit-daemon0-21swapper/121:08:421
162399740,5rtkit-daemon0-21swapper/121:08:421
162399740,5rtkit-daemon0-21swapper/120:52:161
162399720,5rtkit-daemon0-21swapper/120:08:511
162399720,10rtkit-daemon1622-21rtkit-daemon19:14:151
393174527040,23sleep00-21swapper/019:07:050
393194026747,14sleep30-21swapper/319:09:373
39801752660,12sleep1251rcuc/121:20:301
39543142660,6chrt0-21swapper/320:10:213
162399600,9rtkit-daemon1622-21rtkit-daemon22:28:423
162399600,10rtkit-daemon1622-21rtkit-daemon00:36:072
162399580,9rtkit-daemon1622-21rtkit-daemon20:59:402
3932119995723,29cyclictest3242253-21nautilus19:20:023
162399560,9rtkit-daemon1622-21rtkit-daemon00:27:502
162399560,10rtkit-daemon1622-21rtkit-daemon22:34:582
3932119995522,10cyclictest42-21ksoftirqd/300:00:013
3932117995522,6cyclictest141rcu_preempt20:20:002
162399550,8rtkit-daemon1622-21rtkit-daemon22:23:032
162399550,8rtkit-daemon1622-21rtkit-daemon19:22:491
393211399530,49cyclictest1622-21rtkit-daemon22:48:251
3932117995228,20cyclictest4041480-40lsb_release00:15:262
162399520,9rtkit-daemon1622-21rtkit-daemon20:46:372
3932119995124,9cyclictest42-21ksoftirqd/319:56:123
3932119995122,11cyclictest42-21ksoftirqd/323:50:003
3932117995029,14cyclictest34-21ksoftirqd/221:25:162
3932117995015,7cyclictest141rcu_preempt21:36:382
393210799500,46cyclictest1622-21rtkit-daemon19:47:280
40220412490,4sleep00-21swapper/023:20:320
40158162490,12sleep1251rcuc/123:05:111
3932117994917,6cyclictest141rcu_preempt23:45:162
3932117994827,14cyclictest34-21ksoftirqd/200:34:082
3932113994818,18cyclictest26-21ksoftirqd/121:00:201
162399480,40rtkit-daemon1506-21systemd23:22:142
162399480,35rtkit-daemon2197014-21PK-Backend00:10:150
3932117994735,8cyclictest1753-21gnome-shell21:12:162
3932117994720,5cyclictest141rcu_preempt00:15:022
3932117994717,25cyclictest3962027-40lsb_release20:30:242
3932113994718,6cyclictest3971713-21latency_hist21:00:011
3932113994716,12cyclictest26-21ksoftirqd/100:25:011
3932113994716,12cyclictest26-21ksoftirqd/100:25:011
3932107994724,14cyclictest13-21ksoftirqd/021:40:020
393211999469,6cyclictest141rcu_preempt20:30:213
3932119994618,10cyclictest42-21ksoftirqd/323:40:013
3932119994618,10cyclictest42-21ksoftirqd/322:40:013
3932117994610,14cyclictest141rcu_preempt21:30:342
3932113994615,17cyclictest26-21ksoftirqd/119:35:021
3932113994614,19cyclictest26-21ksoftirqd/120:20:221
162399460,39rtkit-daemon1506-21systemd21:51:542
162399460,38rtkit-daemon1506-21systemd23:44:062
162399460,38rtkit-daemon1506-21systemd22:10:582
162399460,37rtkit-daemon1506-21systemd21:18:292
39840792450,6sleep3411rcuc/321:35:013
393211999455,35cyclictest4039114-21hddtemp_smartct00:10:203
3932119994524,13cyclictest42-21ksoftirqd/320:51:313
3932119994520,21cyclictest42-21ksoftirqd/321:13:393
3932119994518,10cyclictest42-21ksoftirqd/323:45:033
3932119994517,10cyclictest42-21ksoftirqd/300:30:003
3932117994534,7cyclictest1753-21gnome-shell00:06:112
3932113994516,16cyclictest3960310-21ls20:25:321
3932113994516,12cyclictest26-21ksoftirqd/123:10:321
3932113994515,7cyclictest3955825-21cpuspeed_turbos20:15:191
3932113994514,18cyclictest26-21ksoftirqd/120:45:021
3932113994513,18cyclictest26-21ksoftirqd/123:20:271
162399450,38rtkit-daemon1506-21systemd23:31:562
162399450,37rtkit-daemon1506-21systemd23:09:262
162399450,37rtkit-daemon1506-21systemd20:13:422
162399450,37rtkit-daemon1506-21systemd00:23:302
162399450,37rtkit-daemon1506-21systemd00:23:302
162399450,34rtkit-daemon2197014-21PK-Backend22:55:252
162399450,31rtkit-daemon4029418-21apt-get23:45:080
3932119994424,17cyclictest860997-21snmpd20:05:423
3932119994416,9cyclictest42-21ksoftirqd/300:22:413
3932119994416,9cyclictest42-21ksoftirqd/300:22:403
3932119994416,10cyclictest42-21ksoftirqd/321:55:023
3932119994414,26cyclictest1753-21gnome-shell00:15:263
3932117994423,13cyclictest34-21ksoftirqd/219:57:162
3932113994416,14cyclictest3999442-21cat22:15:301
3932113994415,7cyclictest3985890-21cmp21:40:021
3932113994415,17cyclictest26-21ksoftirqd/100:30:131
3932113994415,15cyclictest26-21ksoftirqd/121:12:461
3932107994417,17cyclictest141rcu_preempt20:00:140
3932107994417,17cyclictest141rcu_preempt20:00:140
162399440,36rtkit-daemon906-21gmain22:47:072
162399440,35rtkit-daemon1506-21systemd19:54:113
162399440,34rtkit-daemon2197014-21PK-Backend22:30:283
393211999439,30cyclictest1753-21gnome-shell21:40:373
393211999439,16cyclictest1753-21gnome-shell23:10:353
3932119994331,8cyclictest1753-21gnome-shell19:20:183
3932119994325,14cyclictest1753-21gnome-shell19:13:153
3932119994324,14cyclictest3939043-21ntpq19:25:253
3932119994317,9cyclictest42-21ksoftirqd/319:35:013
3932119994315,11cyclictest42-21ksoftirqd/320:30:023
3932119994314,13cyclictest141rcu_preempt20:55:183
3932119994311,13cyclictest855-21dbus-daemon00:04:233
393211799439,7cyclictest141rcu_preempt21:00:032
3932117994332,7cyclictest1753-21gnome-shell22:41:082
3932117994332,7cyclictest1753-21gnome-shell19:26:012
393211799432,5cyclictest141rcu_preempt20:25:012
3932117994314,5cyclictest141rcu_preempt19:11:562
3932117994312,8cyclictest34-21ksoftirqd/221:20:322
3932113994327,12cyclictest3989266-21sshd21:46:391
3932113994323,11cyclictest26-21ksoftirqd/120:00:241
3932113994323,11cyclictest26-21ksoftirqd/120:00:231
3932113994319,6cyclictest26-21ksoftirqd/121:30:181
3932113994316,9cyclictest4030378-21basename23:45:211
3932113994315,14cyclictest26-21ksoftirqd/119:35:141
393211399431,28cyclictest3997620-21python322:10:281
3932113994312,15cyclictest26-21ksoftirqd/119:40:141
3932113994312,15cyclictest26-21ksoftirqd/119:40:131
3932113994311,14cyclictest1632-21dbus-daemon23:59:591
393210799432,5cyclictest141rcu_preempt22:20:140
3932107994324,15cyclictest13-21ksoftirqd/019:45:010
3932107994324,15cyclictest13-21ksoftirqd/019:45:000
3932107994316,17cyclictest141rcu_preempt21:00:120
3932107994315,18cyclictest141rcu_preempt20:20:010
162399430,36rtkit-daemon2197014-21PK-Backend20:24:533
393211999428,30cyclictest1753-21gnome-shell21:29:043
3932119994231,7cyclictest1753-21gnome-shell22:13:473
3932119994225,12cyclictest42-21ksoftirqd/322:49:223
3932119994217,8cyclictest1753-21gnome-shell23:19:303
3932119994214,9cyclictest42-21ksoftirqd/300:39:593
3932119994212,7cyclictest42-21ksoftirqd/319:40:263
3932119994212,7cyclictest42-21ksoftirqd/319:40:263
393211799428,6cyclictest141rcu_preempt20:05:152
393211799428,10cyclictest34-21ksoftirqd/221:57:152
3932117994229,9cyclictest1753-21gnome-shell19:31:542
3932117994225,12cyclictest3960883-21apt-get20:30:012
3932117994224,7cyclictest1753-21gnome-shell22:16:252
3932117994224,6cyclictest34-21ksoftirqd/223:38:472
3932117994224,14cyclictest34-21ksoftirqd/222:36:462
3932117994223,14cyclictest34-21ksoftirqd/222:05:142
3932117994221,15cyclictest34-21ksoftirqd/223:55:242
3932117994216,8cyclictest34-21ksoftirqd/221:45:012
3932117994213,6cyclictest141rcu_preempt20:44:482
3932113994225,12cyclictest438-20systemd-journal22:25:001
3932113994224,14cyclictest3987843-21pool-org.gnome.21:45:021
3932113994217,9cyclictest3968350-21sed20:50:031
3932113994214,12cyclictest26-21ksoftirqd/121:57:531
3932107994219,19cyclictest1753-21gnome-shell20:05:340
3932107994214,18cyclictest141rcu_preempt22:30:220
3932107994210,11cyclictest1069-21runrttasks20:54:010
40482672410,14sleep1251rcuc/100:35:161
3932119994124,7cyclictest1753-21gnome-shell23:20:403
3932119994124,13cyclictest42-21ksoftirqd/322:59:233
3932119994123,6cyclictest42-21ksoftirqd/322:51:563
3932119994123,6cyclictest42-21ksoftirqd/320:15:183
3932119994123,6cyclictest42-21ksoftirqd/319:40:023
3932119994123,13cyclictest42-21ksoftirqd/300:34:183
3932119994121,16cyclictest860997-21snmpd21:46:023
3932119994121,15cyclictest42-21ksoftirqd/321:40:013
3932119994121,14cyclictest42-21ksoftirqd/321:00:153
3932119994117,20cyclictest42-21ksoftirqd/321:20:173
3932119994113,10cyclictest42-21ksoftirqd/323:10:013
3932119994112,11cyclictest42-21ksoftirqd/300:05:033
3932119994112,10cyclictest42-21ksoftirqd/322:06:063
3932119994111,11cyclictest42-21ksoftirqd/320:00:243
3932119994111,11cyclictest42-21ksoftirqd/320:00:243
3932117994131,6cyclictest1753-21gnome-shell19:36:132
3932117994130,7cyclictest1753-21gnome-shell21:47:192
3932117994123,5cyclictest34-21ksoftirqd/220:00:142
3932117994123,5cyclictest34-21ksoftirqd/220:00:142
3932117994123,14cyclictest34-21ksoftirqd/222:00:332
3932117994121,12cyclictest34-21ksoftirqd/200:00:302
3932117994120,12cyclictest34-21ksoftirqd/219:41:592
3932117994120,12cyclictest34-21ksoftirqd/219:41:592
3932117994110,10cyclictest34-21ksoftirqd/219:25:002
393211399412,9cyclictest1753-21gnome-shell00:05:271
393211399412,25cyclictest3949093-21diskstats19:55:171
3932113994113,15cyclictest26-21ksoftirqd/123:25:311
393211399411,19cyclictest4003414-21apt22:30:001
3932113994110,16cyclictest4019846-21cat23:15:231
393210799412,35cyclictest0-21swapper/023:19:550
3932107994117,8cyclictest141rcu_preempt19:25:290
3932107994116,9cyclictest141rcu_preempt23:00:010
3932107994114,12cyclictest141rcu_preempt23:13:280
3932107994114,12cyclictest141rcu_preempt21:50:190
3932107994114,12cyclictest141rcu_preempt00:08:410
393210799411,11cyclictest13-21ksoftirqd/023:00:150
162399410,33rtkit-daemon2197014-21PK-Backend19:55:002
3932119994029,7cyclictest1753-21gnome-shell23:34:373
3932119994023,6cyclictest42-21ksoftirqd/319:49:593
3932119994022,14cyclictest3994028-21ntp_states22:00:263
3932119994019,13cyclictest42-21ksoftirqd/323:01:583
393211999401,35cyclictest1753-21gnome-shell23:50:223
3932119994012,11cyclictest42-21ksoftirqd/323:28:553
3932119994011,11cyclictest42-21ksoftirqd/322:40:223
3932117994024,6cyclictest34-21ksoftirqd/222:26:262
3932117994023,7cyclictest34-21ksoftirqd/223:10:482
3932117994023,6cyclictest34-21ksoftirqd/221:10:022
3932117994023,6cyclictest34-21ksoftirqd/221:10:012
3932117994023,13cyclictest34-21ksoftirqd/223:28:342
3932117994022,6cyclictest34-21ksoftirqd/222:50:092
3932117994022,5cyclictest34-21ksoftirqd/220:50:172
3932117994013,5cyclictest141rcu_preempt19:50:012
393211399409,11cyclictest3948450-21/usr/sbin/munin19:55:021
3932113994023,13cyclictest1-21systemd19:50:001
3932113994015,8cyclictest26-21ksoftirqd/123:50:161
3932107994024,12cyclictest4047446-21sshd00:31:390
3932107994016,8cyclictest141rcu_preempt21:45:270
3932107994014,11cyclictest141rcu_preempt23:05:270
3932107994013,12cyclictest141rcu_preempt23:31:050
162399400,34rtkit-daemon0-21swapper/220:38:162
162399400,29rtkit-daemon4043432-21chrt00:20:260
162399400,29rtkit-daemon4043432-21chrt00:20:260
3932119993928,7cyclictest1753-21gnome-shell22:21:303
3932119993928,7cyclictest1753-21gnome-shell20:40:353
3932119993922,13cyclictest42-21ksoftirqd/321:58:313
3932119993918,13cyclictest42-21ksoftirqd/320:37:033
3932119993918,12cyclictest42-21ksoftirqd/320:45:323
3932117993922,6cyclictest34-21ksoftirqd/223:16:272
3932117993922,6cyclictest34-21ksoftirqd/223:02:082
393211399392,18cyclictest1753-21gnome-shell23:00:161
393211399392,12cyclictest1753-21gnome-shell22:00:401
3932107993927,8cyclictest1753-21gnome-shell23:55:150
3932107993925,10cyclictest4049338-21sshd00:38:400
393210799392,20cyclictest1753-21gnome-shell22:50:060
3932107993914,9cyclictest141rcu_preempt21:25:170
3932107993914,9cyclictest141rcu_preempt21:15:180
393210799391,33cyclictest4027696-21turbostat23:40:010
3932107993913,13cyclictest141rcu_preempt20:29:210
3932107993913,10cyclictest141rcu_preempt00:15:230
3932107993912,12cyclictest141rcu_preempt20:31:200
162399390,5rtkit-daemon0-21swapper/021:24:350
162399390,28rtkit-daemon2197014-21PK-Backend22:36:410
3932119993827,7cyclictest1753-21gnome-shell21:06:243
3932119993827,7cyclictest1753-21gnome-shell21:06:233
3932119993821,13cyclictest42-21ksoftirqd/322:17:053
3932119993821,12cyclictest42-21ksoftirqd/321:15:163
3932117993818,5cyclictest34-21ksoftirqd/219:15:312
3932107993829,5cyclictest2197014-21PK-Backend21:30:450
393210799382,8cyclictest1753-21gnome-shell20:12:530
3932107993828,6cyclictest4032813-21users23:50:320
393210799382,7cyclictest1753-21gnome-shell00:01:480
393210799382,33cyclictest0-21swapper/021:09:400
393210799382,33cyclictest0-21swapper/021:09:390
393210799382,32cyclictest1753-21gnome-shell20:59:520
393210799382,10cyclictest13-21ksoftirqd/021:12:130
3932107993819,14cyclictest4002314-21df22:25:170
3932107993817,16cyclictest855-21dbus-daemon19:55:010
393210799381,34cyclictest0-21swapper/020:22:450
3932107993812,9cyclictest141rcu_preempt20:40:290
162399380,9rtkit-daemon881143-21gmain00:03:281
162399380,26rtkit-daemon2197014-21PK-Backend00:28:170
393211399371,8cyclictest4004445-21awk22:30:291
393211399371,12cyclictest3954080-21tr20:10:171
393210799374,12cyclictest141rcu_preempt20:00:020
3932107993724,9cyclictest1753-21gnome-shell20:45:050
3932107993723,10cyclictest3935637-21awk19:15:340
3932107993721,12cyclictest4008939-21awk22:45:120
3932107993714,9cyclictest141rcu_preempt19:20:250
393210799371,33cyclictest0-21swapper/022:09:240
393210799371,31cyclictest4008562-21taskset22:44:300
3932107993713,19cyclictest141rcu_preempt21:55:270
3932107993712,11cyclictest141rcu_preempt19:14:550
162399370,25rtkit-daemon2197014-21PK-Backend21:42:220
393210799366,4cyclictest141rcu_preempt23:40:200
393210799365,4cyclictest141rcu_preempt19:35:000
3932107993625,7cyclictest3999597-21chrt22:15:440
393210799362,11cyclictest2197014-21PK-Backend22:03:390
3932107993613,10cyclictest141rcu_preempt20:38:420
3932107993520,11cyclictest3942726-21cat19:35:320
3932107993519,12cyclictest3997932-21sshd22:11:400
393210799341,10cyclictest2197014-21PK-Backend23:26:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional