You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 03:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Thu Dec 11, 2025 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991430,10rtkit-daemon1622-21rtkit-daemon19:35:353
162399960,9rtkit-daemon1622-21rtkit-daemon20:31:382
3419532950,6sleep21358-1kworker/u9:120:45:202
307214999222,53cyclictest407659-21/usr/sbin/munin23:50:010
30721499882,21cyclictest1753-21gnome-shell23:42:030
30721499862,20cyclictest1753-21gnome-shell23:39:590
30701428569,10sleep10-21swapper/119:09:171
307218998313,17cyclictest26-21ksoftirqd/123:48:021
30722099827,13cyclictest1753-21gnome-shell23:56:292
307218998210,19cyclictest26-21ksoftirqd/100:11:591
307218998114,17cyclictest26-21ksoftirqd/123:39:591
307220997969,6cyclictest0-21swapper/223:52:232
307220997969,6cyclictest0-21swapper/223:52:232
162399790,5rtkit-daemon0-21swapper/100:21:031
30722499785,13cyclictest42-21ksoftirqd/300:03:473
162399780,5rtkit-daemon0-21swapper/123:09:451
162399780,5rtkit-daemon0-21swapper/123:09:441
30721499762,48cyclictest1753-21gnome-shell00:11:340
162399760,8rtkit-daemon1622-21rtkit-daemon19:15:412
162399760,5rtkit-daemon0-21swapper/123:14:251
30721499743,12cyclictest1753-21gnome-shell23:59:460
30721499736,12cyclictest141rcu_preempt00:08:200
30680627354,13sleep30-21swapper/319:06:323
162399730,5rtkit-daemon0-21swapper/123:20:561
30722499726,48cyclictest42-21ksoftirqd/323:50:203
30722499726,48cyclictest42-21ksoftirqd/323:50:193
307224997250,16cyclictest42-21ksoftirqd/323:57:423
30722499723,64cyclictest1753-21gnome-shell23:45:153
30721499722,46cyclictest408965-21timerwakeupswit23:50:330
30721499722,46cyclictest408965-21timerwakeupswit23:50:330
30687727240,26sleep20-21swapper/219:07:292
162399720,5rtkit-daemon0-21swapper/123:02:031
162399720,5rtkit-daemon0-21swapper/120:40:431
30721499702,10cyclictest1069-21runrttasks00:00:150
162399700,5rtkit-daemon0-21swapper/121:10:341
307214996826,38cyclictest362730-21pool-org.gnome.21:44:500
307224996736,10cyclictest1753-21gnome-shell00:06:093
307220996743,18cyclictest413923-21ntpq00:05:302
30722099668,16cyclictest34-21ksoftirqd/200:00:252
30685326638,22sleep00-21swapper/019:07:110
307224996556,5cyclictest42-21ksoftirqd/323:45:013
30722099652,19cyclictest1753-21gnome-shell23:43:142
307224996417,43cyclictest42-21ksoftirqd/323:39:563
30721899642,19cyclictest1753-21gnome-shell23:40:271
162399640,4rtkit-daemon0-21swapper/322:34:523
307224996322,5cyclictest42-21ksoftirqd/300:10:163
30721899632,57cyclictest1753-21gnome-shell23:57:281
30721899632,51cyclictest408387-21irqstats23:50:241
30721899632,51cyclictest408387-21irqstats23:50:241
30722099622,19cyclictest1753-21gnome-shell00:10:562
30721899622,55cyclictest1753-21gnome-shell00:00:041
162399610,7rtkit-daemon2197014-21PK-Backend23:48:462
307214995922,12cyclictest382130-21cron22:40:000
30722099589,44cyclictest1753-21gnome-shell23:40:002
162399580,9rtkit-daemon1622-21rtkit-daemon22:39:533
162399580,9rtkit-daemon1622-21rtkit-daemon19:12:332
162399580,8rtkit-daemon1622-21rtkit-daemon22:12:442
30721899572,51cyclictest413654-21/usr/sbin/munin00:05:241
162399570,8rtkit-daemon1622-21rtkit-daemon22:26:112
30722099561,10cyclictest358727-21chrt21:32:092
162399560,8rtkit-daemon1622-21rtkit-daemon20:15:542
162399560,8rtkit-daemon1622-21rtkit-daemon20:01:363
30721499551,32cyclictest1753-21gnome-shell22:58:360
30721499542,26cyclictest1753-21gnome-shell22:05:440
307214995422,10cyclictest13-21ksoftirqd/019:53:480
30722499531,49cyclictest1-21systemd19:27:573
307220995318,13cyclictest34-21ksoftirqd/220:05:002
307214995322,6cyclictest1069-21runrttasks20:24:280
307214995320,11cyclictest13-21ksoftirqd/020:19:320
162399530,9rtkit-daemon1622-21rtkit-daemon22:22:013
162399530,8rtkit-daemon1622-21rtkit-daemon20:21:442
307220995216,17cyclictest34-21ksoftirqd/221:25:002
307214995220,7cyclictest1069-21runrttasks21:33:030
307214995216,13cyclictest395423-40apt-check23:15:190
162399520,9rtkit-daemon1622-21rtkit-daemon22:41:033
162399520,7rtkit-daemon1622-21rtkit-daemon20:24:283
30722099512,11cyclictest1753-21gnome-shell19:35:122
307220995112,19cyclictest34-21ksoftirqd/219:35:012
307214995119,7cyclictest326755-21latency_hist20:05:020
307214995111,16cyclictest0-21swapper/021:15:160
307214995111,16cyclictest0-21swapper/021:15:160
162399510,8rtkit-daemon1622-21rtkit-daemon19:52:462
162399510,8rtkit-daemon1622-21rtkit-daemon19:48:432
307220995010,12cyclictest34-21ksoftirqd/221:18:242
307220995010,12cyclictest34-21ksoftirqd/221:18:232
307214995019,7cyclictest1069-21runrttasks19:59:290
307214995017,10cyclictest13-21ksoftirqd/000:27:170
162399500,37rtkit-daemon1753-21gnome-shell20:45:171
30721499492,24cyclictest399799-21sensors_temp23:25:290
307214994919,8cyclictest1753-21gnome-shell19:14:490
30721499491,25cyclictest1753-21gnome-shell22:02:260
307214994910,18cyclictest1753-21gnome-shell21:09:380
307214994910,18cyclictest1753-21gnome-shell00:32:030
162399490,6rtkit-daemon1622-21rtkit-daemon20:14:372
307220994810,19cyclictest34-21ksoftirqd/200:23:172
30721499489,17cyclictest1753-21gnome-shell21:21:140
307214994816,10cyclictest13-21ksoftirqd/022:40:500
30721499481,25cyclictest1753-21gnome-shell00:15:190
30721499481,25cyclictest1753-21gnome-shell00:15:180
3722402470,7sleep10-21swapper/122:10:221
307224994730,13cyclictest395801-21diskstats23:15:193
307220994713,18cyclictest34-21ksoftirqd/222:45:042
307220994623,18cyclictest366870-21df21:55:162
307220994615,17cyclictest34-21ksoftirqd/223:30:012
307220994615,11cyclictest34-21ksoftirqd/221:45:172
307220994612,18cyclictest34-21ksoftirqd/220:05:202
307218994628,14cyclictest1-21systemd21:45:011
30721499461,24cyclictest1753-21gnome-shell21:29:070
30721499461,24cyclictest1753-21gnome-shell19:38:090
30721499461,23cyclictest1753-21gnome-shell22:50:020
307214994611,6cyclictest141rcu_preempt21:15:020
30722499458,20cyclictest42-21ksoftirqd/320:50:133
307224994530,5cyclictest42-21ksoftirqd/300:35:553
307224994530,5cyclictest42-21ksoftirqd/300:35:543
307224994520,13cyclictest42-21ksoftirqd/323:00:243
307224994510,16cyclictest1753-21gnome-shell21:40:433
30722099457,14cyclictest1753-21gnome-shell22:57:112
307220994537,4cyclictest331rcuc/222:30:482
307220994520,7cyclictest1753-21gnome-shell21:04:022
307220994512,13cyclictest34-21ksoftirqd/220:50:322
30722099451,12cyclictest34-21ksoftirqd/223:00:242
307220994511,13cyclictest34-21ksoftirqd/223:20:242
30722099451,10cyclictest1825-21JS14:31:442
30721499451,23cyclictest1753-21gnome-shell22:23:570
30721499451,23cyclictest1753-21gnome-shell21:50:420
30721499451,23cyclictest1753-21gnome-shell21:01:400
30721499451,23cyclictest1753-21gnome-shell00:35:510
30721499451,23cyclictest1753-21gnome-shell00:35:510
30721499451,22cyclictest1753-21gnome-shell00:22:010
162399450,38rtkit-daemon860997-21snmpd19:20:192
30722099446,19cyclictest1753-21gnome-shell23:11:542
307220994420,8cyclictest1753-21gnome-shell22:04:432
307220994420,7cyclictest1753-21gnome-shell00:39:332
307220994420,7cyclictest1753-21gnome-shell00:39:332
307220994420,20cyclictest339564-21awk20:40:012
307220994420,20cyclictest339564-21awk20:40:012
307220994410,19cyclictest34-21ksoftirqd/221:10:212
307218994422,17cyclictest396923-21sh23:20:001
307214994433,7cyclictest1069-21runrttasks19:29:020
30721499441,4cyclictest2197014-21PK-Backend20:40:220
30721499441,22cyclictest1753-21gnome-shell22:29:510
30721499441,22cyclictest1753-21gnome-shell20:45:570
307224994326,14cyclictest1753-21gnome-shell21:27:273
307224994324,15cyclictest3242255-21gdbus23:30:013
307224994323,13cyclictest42-21ksoftirqd/300:30:193
307224994321,6cyclictest42-21ksoftirqd/319:40:213
30722099439,13cyclictest34-21ksoftirqd/220:55:152
30722099439,13cyclictest34-21ksoftirqd/220:55:142
30722099437,12cyclictest34-21ksoftirqd/223:35:032
30722099437,12cyclictest34-21ksoftirqd/223:35:032
30722099436,19cyclictest34-21ksoftirqd/222:45:012
30722099435,17cyclictest1753-21gnome-shell22:09:572
307220994320,7cyclictest1753-21gnome-shell20:40:052
307220994315,15cyclictest34-21ksoftirqd/221:44:272
30722499429,15cyclictest1753-21gnome-shell20:33:403
307224994232,6cyclictest419836-21dump-pmu-power00:25:003
307224994223,14cyclictest42-21ksoftirqd/321:15:313
307224994223,14cyclictest42-21ksoftirqd/321:15:313
30722099429,18cyclictest34-21ksoftirqd/222:15:272
30722099427,19cyclictest34-21ksoftirqd/200:26:282
30722099427,16cyclictest34-21ksoftirqd/222:25:012
30722099426,11cyclictest34-21ksoftirqd/221:35:372
307220994211,16cyclictest34-21ksoftirqd/200:16:232
307220994211,16cyclictest34-21ksoftirqd/200:16:222
307220994211,15cyclictest1753-21gnome-shell21:54:492
307218994223,15cyclictest365502-21/usr/sbin/munin21:50:261
307224994130,7cyclictest1753-21gnome-shell20:15:163
307224994128,8cyclictest42-21ksoftirqd/320:13:213
307224994123,14cyclictest42-21ksoftirqd/321:45:163
307224994122,14cyclictest42-21ksoftirqd/320:49:203
307224994122,14cyclictest42-21ksoftirqd/319:50:143
307224994122,14cyclictest42-21ksoftirqd/300:15:303
307224994122,14cyclictest42-21ksoftirqd/300:15:293
307224994120,6cyclictest42-21ksoftirqd/319:10:133
307224994120,17cyclictest860997-21snmpd21:30:373
30722099417,12cyclictest34-21ksoftirqd/223:05:572
30722099417,12cyclictest34-21ksoftirqd/223:05:572
30722099417,11cyclictest34-21ksoftirqd/221:10:002
30722099415,9cyclictest34-21ksoftirqd/223:15:202
30722099415,13cyclictest34-21ksoftirqd/200:32:052
30722099411,18cyclictest324979-21chrt19:59:442
307218994119,18cyclictest354585-21latency21:20:221
30721899411,36cyclictest343408-21cut20:50:151
307214994125,12cyclictest393139-21cron23:10:010
307214994125,12cyclictest393139-21cron23:10:000
30721499411,8cyclictest0-21swapper/023:10:210
30722499406,29cyclictest346484-21munin-run21:00:013
30722499406,29cyclictest346484-21munin-run21:00:013
307224994030,7cyclictest1753-21gnome-shell22:29:023
307224994029,7cyclictest1753-21gnome-shell21:14:553
307224994029,7cyclictest1753-21gnome-shell20:41:233
307224994021,14cyclictest42-21ksoftirqd/321:20:433
30722499401,35cyclictest1753-21gnome-shell20:28:103
307224994012,10cyclictest420263-21grep00:25:153
30722499401,13cyclictest2197014-21PK-Backend21:03:403
30722099408,10cyclictest34-21ksoftirqd/222:55:022
30722099406,17cyclictest1753-21gnome-shell20:29:452
30722099402,18cyclictest34-21ksoftirqd/222:40:022
307218994012,13cyclictest1632-21dbus-daemon20:30:011
30721899400,36cyclictest1622-21rtkit-daemon21:58:281
3979102390,7sleep01753-21gnome-shell23:20:260
3098832390,7sleep3411rcuc/319:15:133
307224993929,5cyclictest42-21ksoftirqd/319:34:253
307224993928,7cyclictest1753-21gnome-shell22:15:133
307224993928,7cyclictest1753-21gnome-shell22:10:273
307224993927,8cyclictest1753-21gnome-shell22:57:263
307224993922,13cyclictest42-21ksoftirqd/322:05:153
30722499392,11cyclictest311754-21diskstats19:20:193
307224993920,6cyclictest42-21ksoftirqd/321:08:103
30722499391,9cyclictest42-21ksoftirqd/322:01:593
30722099396,10cyclictest34-21ksoftirqd/219:40:252
30721899392,8cyclictest1753-21gnome-shell22:07:481
307218993927,9cyclictest26-21ksoftirqd/120:13:221
307218993927,7cyclictest1800-21upowerd21:15:161
307218993927,7cyclictest1800-21upowerd21:15:161
30721899392,33cyclictest1753-21gnome-shell22:01:331
307218993922,12cyclictest438-20systemd-journal21:27:231
307218993920,15cyclictest331993-21perf20:20:001
307214993926,9cyclictest328739-21find20:10:120
307214993922,13cyclictest334852-21memory20:25:250
307214993921,14cyclictest390695-21ntpq23:00:250
162399390,29rtkit-daemon2197014-21PK-Backend22:50:190
162399390,27rtkit-daemon0-21swapper/119:20:461
3367582380,6sleep02197014-21PK-Backend20:30:250
307224993828,7cyclictest1753-21gnome-shell21:58:193
307224993827,7cyclictest1753-21gnome-shell21:35:243
307224993827,7cyclictest1069-21runrttasks20:36:123
307224993827,7cyclictest1069-21runrttasks20:36:123
307224993824,11cyclictest42-21ksoftirqd/322:45:303
307224993823,11cyclictest394522-21awk23:10:323
307224993820,6cyclictest42-21ksoftirqd/323:30:193
307224993820,6cyclictest42-21ksoftirqd/323:30:193
30722499381,21cyclictest391694-21apt23:05:133
30722499381,21cyclictest391694-21apt23:05:123
30722099385,13cyclictest34-21ksoftirqd/221:27:432
307218993821,13cyclictest325685-21fw_packets20:00:211
307218993821,13cyclictest313695-21memory19:25:241
307218993820,14cyclictest332989-21ntpq20:20:241
307214993824,10cyclictest13-21ksoftirqd/021:59:400
307214993823,10cyclictest359607-21sh21:35:170
307214993822,11cyclictest312837-21/usr/sbin/munin19:25:020
162399380,23rtkit-daemon2197014-21PK-Backend23:33:180
162399380,23rtkit-daemon2197014-21PK-Backend23:33:180
30722499373,10cyclictest141rcu_preempt19:59:243
307224993722,10cyclictest42-21ksoftirqd/321:52:023
307224993721,6cyclictest42-21ksoftirqd/323:21:063
307224993720,13cyclictest42-21ksoftirqd/320:06:333
30722499371,14cyclictest386560-21diskstats22:50:173
307218993724,9cyclictest359227-21awk21:35:121
307218993722,11cyclictest323173-21seq19:53:461
307218993720,12cyclictest438-20systemd-journal19:40:001
307218993710,23cyclictest375285-21cpuspeed_turbos22:20:141
307224993626,7cyclictest1753-21gnome-shell19:46:143
307218993622,10cyclictest378783-21apt-config22:30:031
307218993621,11cyclictest350063-21seq21:07:481
307218993620,12cyclictest387655-21apt-get22:55:001
307218993620,12cyclictest363645-21cat21:45:201
307218993618,13cyclictest388172-21awk22:55:131
307214993625,7cyclictest320276-21cpuspeed_turbos19:45:170
307214993620,12cyclictest1632-21dbus-daemon21:50:010
162399360,5rtkit-daemon0-21swapper/123:30:351
162399360,5rtkit-daemon0-21swapper/123:30:351
307218993525,6cyclictest316354-21turbostat.cron19:35:001
307218993524,7cyclictest358311-21grep21:30:301
307218993520,11cyclictest346025-21sed20:55:331
307218993520,11cyclictest346025-21sed20:55:321
307218993520,11cyclictest336915-21python320:30:301
307218993519,12cyclictest310806-21sshd19:16:221
307218993517,13cyclictest438-20systemd-journal19:49:591
307218993516,15cyclictest855-21dbus-daemon20:40:011
307218993516,15cyclictest855-21dbus-daemon20:40:001
307214993523,8cyclictest1753-21gnome-shell19:31:120
30721499351,9cyclictest2197014-21PK-Backend19:16:440
307214993519,12cyclictest318918-21sed19:40:240
30721899346,24cyclictest1824-21JS14:31:441
30721899342,15cyclictest347618-21python321:00:281
307218993418,12cyclictest398988-21ls23:25:131
307218993416,14cyclictest3242253-21nautilus22:45:011
307218993415,15cyclictest422449-21sed00:30:221
307214993421,9cyclictest373178-21turbostat22:15:000
30721499341,9cyclictest2197014-21PK-Backend22:19:080
307214993418,12cyclictest339323-21sshd20:37:210
307214993418,12cyclictest339323-21sshd20:37:210
307218993322,7cyclictest1753-21gnome-shell00:25:211
30721899332,15cyclictest318754-21hddtemp_smartct19:40:211
307218993319,10cyclictest378400-21gpgv22:30:011
307218993318,11cyclictest328072-21grep20:05:311
307218993317,12cyclictest374160-21awk22:15:241
307218993313,16cyclictest325205-21sed20:00:021
30721899331,10cyclictest3080682needreboot19:10:231
307214993318,11cyclictest327075-21cpuspeed20:05:140
30721499331,29cyclictest0-21swapper/020:54:080
30721899322,14cyclictest1753-21gnome-shell22:39:411
307218993217,11cyclictest423715-21cpuspeed_turbos00:35:161
307218993217,11cyclictest423715-21cpuspeed_turbos00:35:151
307218993119,8cyclictest416999-21df00:15:171
307218993119,8cyclictest416999-21df00:15:171
30721499311,14cyclictest2197014-21PK-Backend20:55:270
30721499311,14cyclictest2197014-21PK-Backend20:55:270
307214993110,17cyclictest1753-21gnome-shell22:34:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional