You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 06:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Dec 20, 2025 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8163021340,6sleep1251rcuc/123:50:131
41783312930,6sleep24497-21PK-Backend19:15:312
591852880,5sleep10-21swapper/122:45:251
4174823998880,5cyclictest4497-21PK-Backend20:12:200
197399880,8rtkit-daemon1972-21rtkit-daemon22:23:411
60972850,5sleep30-21swapper/320:15:293
197399850,8rtkit-daemon1972-21rtkit-daemon22:25:311
197399710,60rtkit-daemon0-21swapper/019:08:030
4174833997062,5cyclictest0-21swapper/320:12:203
4174828996838,7cyclictest141rcu_preempt00:05:592
4174826996860,5cyclictest0-21swapper/120:12:201
4174823996817,17cyclictest54400-40apt-check22:32:030
4174828996637,8cyclictest141rcu_preempt21:28:592
417448026647,13sleep20-21swapper/219:07:162
417436226544,14sleep10-21swapper/119:05:461
4174828996434,7cyclictest141rcu_preempt20:44:582
417442926445,13sleep30-21swapper/319:06:363
4174823996112,30cyclictest0-21swapper/022:52:030
4174833996021,12cyclictest42-21ksoftirqd/319:31:293
417482399609,29cyclictest79180-40lsb_release23:40:590
417482399602,53cyclictest2069-21gnome-shell22:35:590
4174823996015,21cyclictest35354-21perf21:44:550
4174823996013,25cyclictest14993-21ntp_states20:40:230
4174823996010,30cyclictest13777-40lsb_release20:38:010
4174823996010,30cyclictest13777-40lsb_release20:38:010
417482899592,52cyclictest2069-21gnome-shell20:58:592
417482899592,52cyclictest2069-21gnome-shell20:58:592
417482699592,15cyclictest2069-21gnome-shell22:00:581
417482399593,34cyclictest0-21swapper/021:28:100
4174823995915,23cyclictest42190-21seq21:59:080
638842580,8chrt42-21ksoftirqd/323:00:123
417483399582,47cyclictest2069-21gnome-shell21:30:593
417482899582,52cyclictest2069-21gnome-shell20:50:592
417482699582,52cyclictest2069-21gnome-shell23:34:581
4174823995814,19cyclictest4189030-21taskset19:49:170
4174823995812,26cyclictest75186-21python323:30:280
4174823995812,25cyclictest4193885-21ntp_states20:00:260
4174823995811,26cyclictest73334-21ntp_states23:25:270
4174823995811,26cyclictest73334-21ntp_states23:25:270
4174823995811,26cyclictest4189749-21diskstats19:50:160
4174823995811,15cyclictest62636-21ntp_states22:55:260
4174823995810,27cyclictest19293-40lsb_release20:53:050
417482699572,51cyclictest2069-21gnome-shell21:21:591
4174823995717,11cyclictest45050-21/usr/sbin/munin22:05:250
4174823995711,26cyclictest16847-21python320:45:280
4174823995711,25cyclictest90576-21systemctl00:15:000
4174823995711,17cyclictest21296-21apt-key21:00:020
4174823995711,17cyclictest21296-21apt-key21:00:010
985942560,4sleep00-21swapper/000:35:260
41780062560,12sleep126-21ksoftirqd/119:15:261
417483399561,51cyclictest2069-21gnome-shell20:40:593
417482399564,16cyclictest4497-21PK-Backend20:15:300
4174823995617,17cyclictest49440-21/usr/sbin/munin22:20:190
4174823995615,18cyclictest4185432-21seq19:37:310
4174823995610,17cyclictest84669-40lsb_release23:55:590
417482399552,33cyclictest0-21swapper/023:05:430
417482399552,10cyclictest2069-21gnome-shell22:01:590
4174823995518,14cyclictest57938-21taskset22:42:190
4174823995513,18cyclictest1137-21runrttasks21:37:140
4174823995510,22cyclictest30092-21taskset21:24:220
4174833995427,8cyclictest42-21ksoftirqd/322:42:333
4174833995427,15cyclictest42-21ksoftirqd/300:29:293
417483399542,47cyclictest2069-21gnome-shell20:33:593
4174828995431,7cyclictest141rcu_preempt00:10:592
417482699542,48cyclictest2069-21gnome-shell23:05:581
417482399543,30cyclictest4497-21PK-Backend21:45:240
417482399542,10cyclictest2069-21gnome-shell00:26:590
4174823995417,15cyclictest68800-21seq23:14:380
4174823995413,19cyclictest1137-21runrttasks19:42:290
4174823995413,17cyclictest8833-21/usr/sbin/munin20:25:130
4174823995411,17cyclictest82289-21ntp_states23:50:240
417482399541,10cyclictest2069-21gnome-shell00:22:590
4174828995337,12cyclictest2069-21gnome-shell22:03:592
417482899532,46cyclictest2069-21gnome-shell22:24:592
417482699532,47cyclictest2069-21gnome-shell00:27:581
417482399539,17cyclictest25135-21cat21:10:020
4174823995310,17cyclictest86449-21chrt00:00:540
4174823995310,17cyclictest4183200-21ntp_states19:30:250
4174823995310,16cyclictest4182190-21/usr/sbin/munin19:30:010
4174833995226,7cyclictest42-21ksoftirqd/322:53:303
417482699522,46cyclictest2069-21gnome-shell23:20:591
417482399528,16cyclictest4177136-21irqstats19:15:050
417482399523,23cyclictest0-21swapper/021:10:300
417482399521,47cyclictest2069-21gnome-shell21:19:580
4174823995211,15cyclictest0-21swapper/021:54:530
4174823995210,17cyclictest2069-21gnome-shell23:45:090
417483399512,46cyclictest4497-21PK-Backend20:54:593
417483399512,45cyclictest2069-21gnome-shell21:37:593
417482899512,45cyclictest2069-21gnome-shell22:18:592
417482899512,45cyclictest2069-21gnome-shell20:32:592
417482699512,47cyclictest231irq_work/120:54:591
417482699512,45cyclictest2069-21gnome-shell23:37:591
417482699512,39cyclictest231irq_work/121:13:581
417482399512,7cyclictest0-21swapper/022:16:590
417482399512,45cyclictest0-21swapper/023:01:590
4174823995115,14cyclictest70217-21taskset23:15:350
4174823995113,22cyclictest21203-21turbostat21:04:560
4174823995110,19cyclictest88418-21chrt00:05:360
4174823995110,17cyclictest51177-21/usr/sbin/munin22:25:270
4174823995110,16cyclictest7893-21smartctl20:20:290
560142500,15sleep1251rcuc/122:35:301
4174833995042,5cyclictest411rcuc/322:36:423
4174833995038,8cyclictest2069-21gnome-shell22:17:583
4174833995030,7cyclictest42-21ksoftirqd/323:35:453
4174833995027,7cyclictest42-21ksoftirqd/323:25:353
4174833995027,7cyclictest42-21ksoftirqd/323:25:353
4174833995026,7cyclictest42-21ksoftirqd/300:35:453
417482899502,9cyclictest2069-21gnome-shell22:12:592
4174826995041,6cyclictest231irq_work/120:45:581
4174826995040,7cyclictest251rcuc/120:01:501
417482699503,44cyclictest231irq_work/122:55:591
417482699502,7cyclictest231irq_work/100:20:591
417482699502,45cyclictest231irq_work/123:41:591
4174826995016,17cyclictest26-21ksoftirqd/121:30:311
417482699501,45cyclictest1972-21rtkit-daemon19:13:571
4174826995014,14cyclictest956-21systemd-logind00:19:481
417482399508,19cyclictest4179934-21threads19:20:270
417482399502,44cyclictest0-21swapper/021:32:590
417482399502,44cyclictest0-21swapper/020:33:590
4174823995015,14cyclictest4175601-21cat19:10:190
4174823995010,16cyclictest76048-21apt23:35:110
417483399496,28cyclictest42-21ksoftirqd/321:05:153
4174833994929,7cyclictest42-21ksoftirqd/322:24:263
4174833994929,7cyclictest42-21ksoftirqd/320:27:593
4174833994929,7cyclictest42-21ksoftirqd/300:17:223
4174833994926,7cyclictest42-21ksoftirqd/320:57:253
4174833994926,7cyclictest42-21ksoftirqd/320:57:253
4174833994926,7cyclictest42-21ksoftirqd/319:17:243
4174833994925,8cyclictest42-21ksoftirqd/321:29:533
4174828994937,8cyclictest2069-21gnome-shell20:46:592
417482699492,6cyclictest0-21swapper/100:32:581
417482699492,45cyclictest231irq_work/121:48:591
417482699492,44cyclictest231irq_work/123:12:591
417482699492,44cyclictest231irq_work/121:57:591
417482699492,44cyclictest231irq_work/120:42:591
417482699492,44cyclictest231irq_work/100:37:591
417482699492,44cyclictest231irq_work/100:05:591
417482699492,43cyclictest231irq_work/122:06:581
417482699491,45cyclictest231irq_work/121:51:591
4174826994913,18cyclictest26-21ksoftirqd/119:49:101
417482399499,16cyclictest59721-21chrt22:48:010
417482399499,16cyclictest46738-21awk22:10:230
72732480,3sleep30-21swapper/320:20:153
4174833994837,7cyclictest2069-21gnome-shell00:11:583
4174833994829,7cyclictest42-21ksoftirqd/323:08:443
4174833994828,7cyclictest42-21ksoftirqd/323:18:423
417483399482,7cyclictest0-21swapper/322:00:593
417483399482,6cyclictest0-21swapper/323:20:593
417483399482,6cyclictest0-21swapper/321:13:593
4174833994826,7cyclictest42-21ksoftirqd/322:57:583
4174833994825,7cyclictest42-21ksoftirqd/321:01:363
4174833994825,7cyclictest42-21ksoftirqd/319:13:573
4174833994824,7cyclictest42-21ksoftirqd/320:08:523
417483399482,43cyclictest4497-21PK-Backend22:28:593
417482699485,16cyclictest26-21ksoftirqd/121:40:211
4174826994840,6cyclictest231irq_work/121:38:581
4174826994836,9cyclictest26-21ksoftirqd/121:02:591
417482699483,43cyclictest231irq_work/123:57:581
417482699482,6cyclictest0-21swapper/120:18:591
417482699482,5cyclictest0-21swapper/122:51:591
417482699482,44cyclictest231irq_work/120:28:581
417482699482,43cyclictest231irq_work/122:18:591
417482699482,43cyclictest231irq_work/122:11:591
417482699481,6cyclictest4497-21PK-Backend20:33:591
417482699481,45cyclictest231irq_work/123:28:591
417482699481,45cyclictest231irq_work/123:28:591
417482699481,44cyclictest231irq_work/121:05:591
4174823994839,6cyclictest0-21swapper/000:31:590
4174823994839,6cyclictest0-21swapper/000:18:590
4174833994739,5cyclictest0-21swapper/300:06:593
417483399472,6cyclictest0-21swapper/323:57:593
417483399472,6cyclictest0-21swapper/323:12:593
417483399472,6cyclictest0-21swapper/321:57:593
417483399472,6cyclictest0-21swapper/321:22:593
4174833994724,7cyclictest42-21ksoftirqd/319:54:393
417483399472,43cyclictest0-21swapper/323:41:583
417483399472,42cyclictest4497-21PK-Backend22:11:593
417483399472,42cyclictest0-21swapper/322:06:593
417483399472,42cyclictest0-21swapper/321:40:593
4174833994723,8cyclictest42-21ksoftirqd/320:37:493
4174833994723,8cyclictest42-21ksoftirqd/320:37:493
417483399471,38cyclictest4497-21PK-Backend23:31:593
4174828994738,5cyclictest595602sleep222:45:592
417482899472,41cyclictest0-21swapper/200:20:582
417482899471,6cyclictest4497-21PK-Backend23:56:592
417482899471,6cyclictest4497-21PK-Backend23:51:582
417482899471,43cyclictest4497-21PK-Backend21:48:592
417482899471,42cyclictest4497-21PK-Backend21:33:592
4174826994739,5cyclictest231irq_work/122:31:591
417482699472,43cyclictest0-21swapper/122:44:591
4174826994724,19cyclictest4191766-21/usr/sbin/munin19:55:191
4174826994716,15cyclictest26-21ksoftirqd/119:30:311
417482699471,44cyclictest231irq_work/120:20:591
4174826994710,19cyclictest26-21ksoftirqd/100:01:581
417482699470,44cyclictest1972-21rtkit-daemon19:26:101
4174833994638,6cyclictest411rcuc/319:20:243
4174833994625,7cyclictest42-21ksoftirqd/319:43:153
417483399462,41cyclictest4497-21PK-Backend20:47:593
4174833994613,11cyclictest42-21ksoftirqd/323:53:483
4174828994640,4cyclictest0-21swapper/200:30:592
417482899462,5cyclictest4497-21PK-Backend23:06:592
4174828994625,7cyclictest141rcu_preempt23:10:592
417482899462,42cyclictest0-21swapper/222:55:592
417482899461,42cyclictest4497-21PK-Backend22:31:592
417482899461,42cyclictest4497-21PK-Backend00:02:592
417482899461,41cyclictest4497-21PK-Backend22:05:592
417482899461,37cyclictest4497-21PK-Backend20:23:592
417482699462,42cyclictest0-21swapper/123:15:591
4174826994616,15cyclictest26-21ksoftirqd/119:23:411
417482399467,14cyclictest71254-21smartctl23:20:190
4174833994537,6cyclictest411rcuc/319:35:043
4174833994537,5cyclictest0-21swapper/322:30:593
4174833994536,6cyclictest411rcuc/321:54:413
417483399452,41cyclictest0-21swapper/321:49:593
417483399451,41cyclictest4497-21PK-Backend00:22:593
4174833994513,11cyclictest42-21ksoftirqd/319:57:293
4174828994538,4cyclictest4497-21PK-Backend23:22:592
4174828994538,4cyclictest4497-21PK-Backend00:16:592
4174828994524,17cyclictest51697-21/usr/sbin/munin22:25:202
417482899451,41cyclictest4497-21PK-Backend21:55:582
417482899451,41cyclictest4497-21PK-Backend20:12:402
417482899451,41cyclictest4497-21PK-Backend00:29:592
417482699456,10cyclictest2069-21gnome-shell21:25:591
4174826994537,6cyclictest231irq_work/123:48:591
4174826994537,5cyclictest251rcuc/120:37:571
4174826994537,5cyclictest251rcuc/120:37:571
417482399459,18cyclictest4191340-21/usr/sbin/munin19:55:120
4174823994513,17cyclictest1137-21runrttasks20:06:590
417483399444,35cyclictest2069-21gnome-shell00:04:593
4174833994437,4cyclictest4497-21PK-Backend23:48:583
4174833994435,6cyclictest411rcuc/320:01:293
4174833994412,19cyclictest42-21ksoftirqd/319:25:333
4174828994438,4cyclictest0-21swapper/223:00:582
4174828994438,4cyclictest0-21swapper/221:16:592
4174828994438,4cyclictest0-21swapper/221:02:592
4174828994437,4cyclictest4497-21PK-Backend22:53:592
4174828994423,7cyclictest141rcu_preempt23:15:592
197399440,4rtkit-daemon4497-21PK-Backend21:41:272
4174833994337,4cyclictest0-21swapper/321:19:593
417482699438,19cyclictest26-21ksoftirqd/100:14:111
417483399427,12cyclictest42-21ksoftirqd/300:30:353
4174833994233,6cyclictest411rcuc/322:46:293
4174828994236,4cyclictest0-21swapper/223:32:592
417482899422,6cyclictest0-21swapper/220:37:592
417482899422,6cyclictest0-21swapper/220:37:592
4174826994236,4cyclictest0-21swapper/121:19:591
4174826994235,4cyclictest0-21swapper/120:59:591
4174826994235,4cyclictest0-21swapper/120:59:591
4174833994133,5cyclictest411rcuc/319:47:533
4174828994135,4cyclictest0-21swapper/200:36:592
417482699417,10cyclictest26-21ksoftirqd/119:40:261
417482699415,22cyclictest26-21ksoftirqd/119:37:091
4174826994132,6cyclictest231irq_work/123:01:591
41917652400,7chrt4497-21PK-Backend19:55:192
4174828994021,15cyclictest4181080-21df19:25:152
4174828993932,4cyclictest4497-21PK-Backend21:20:592
4174828993911,5cyclictest141rcu_preempt19:45:022
417482699391,15cyclictest4190639-21taskset19:52:431
4174828993831,4cyclictest4497-21PK-Backend20:29:592
4174828993827,7cyclictest2069-21gnome-shell21:36:592
4174828993814,5cyclictest141rcu_preempt23:40:532
417482899381,32cyclictest4497-21PK-Backend22:36:592
417482899375,11cyclictest141rcu_preempt19:52:102
4174828993720,13cyclictest6501-21pkexec20:17:062
4174828993716,17cyclictest4182900-21/usr/sbin/munin19:30:192
13652370,4chrt0-21swapper/120:05:201
4174828993613,6cyclictest141rcu_preempt20:05:282
4174828993611,5cyclictest141rcu_preempt21:05:122
4174828993525,6cyclictest34-21ksoftirqd/221:55:012
4174828993517,13cyclictest4179112-21tr19:20:122
4174828993512,5cyclictest141rcu_preempt23:35:232
417482899351,13cyclictest4497-21PK-Backend19:38:062
4174828993510,6cyclictest141rcu_preempt19:10:242
417482899348,6cyclictest141rcu_preempt20:00:152
417482899335,7cyclictest924-21NetworkManager21:10:282
4174828993324,6cyclictest0-21swapper/222:40:592
417482899328,7cyclictest141rcu_preempt23:45:242
417482899328,6cyclictest141rcu_preempt19:45:212
417482899321,11cyclictest4497-21PK-Backend23:25:482
417482899321,11cyclictest4497-21PK-Backend23:25:472
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional