You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-13 - 07:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Dec 13, 2025 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13984322990,9sleep00-21swapper/019:05:150
1973991050,5rtkit-daemon0-21swapper/300:04:312
1973991050,5rtkit-daemon0-21swapper/300:04:312
1973991000,5rtkit-daemon0-21swapper/322:37:432
197399980,6rtkit-daemon4497-21PK-Backend20:25:092
197399980,5rtkit-daemon0-21swapper/322:55:262
197399960,5rtkit-daemon0-21swapper/319:10:082
1832802910,6sleep00-21swapper/021:10:010
140882998713,14cyclictest2069-21gnome-shell21:55:580
1756912840,4sleep30-21swapper/320:45:253
2367102760,4chrt236189-21/usr/sbin/munin23:40:200
197399760,5rtkit-daemon4497-21PK-Backend00:30:553
197399750,62rtkit-daemon0-21swapper/319:06:053
2341872730,5sleep14497-21PK-Backend23:35:021
14089699714,5cyclictest141rcu_preempt22:07:183
140896997114,53cyclictest2069-21gnome-shell21:51:583
14049526747,14sleep10-21swapper/119:06:451
140882996614,47cyclictest2069-21gnome-shell23:21:580
140882996449,11cyclictest2069-21gnome-shell22:01:580
140882996448,12cyclictest2069-21gnome-shell21:12:580
14049626344,13sleep20-21swapper/219:06:462
197399610,9rtkit-daemon1972-21rtkit-daemon21:55:301
197399600,9rtkit-daemon1972-21rtkit-daemon21:49:061
14089199602,17cyclictest2069-21gnome-shell19:32:582
14088799603,42cyclictest4497-21PK-Backend00:20:591
197399590,9rtkit-daemon1972-21rtkit-daemon21:48:580
14089699592,53cyclictest2069-21gnome-shell19:41:593
14089699592,53cyclictest2069-21gnome-shell19:41:583
14089699592,16cyclictest2069-21gnome-shell22:37:583
140896995811,44cyclictest2069-21gnome-shell20:50:583
14089199582,52cyclictest2069-21gnome-shell20:33:582
14089199581,7cyclictest141rcu_preempt23:20:592
140882995848,6cyclictest2069-21gnome-shell23:38:590
140882995843,11cyclictest2069-21gnome-shell23:32:590
14089699572,51cyclictest2069-21gnome-shell22:57:583
14089699571,52cyclictest2069-21gnome-shell21:27:583
14089199571,15cyclictest2069-21gnome-shell20:55:582
140882995747,6cyclictest2069-21gnome-shell23:46:580
2216162560,5sleep1251rcuc/122:57:281
14089699562,50cyclictest2069-21gnome-shell21:04:593
14089199562,12cyclictest2069-21gnome-shell19:38:582
1949772550,7sleep14497-21PK-Backend21:40:221
1949772550,7sleep14497-21PK-Backend21:40:211
14089699553,11cyclictest193892-40lsb_release21:37:583
14089199551,50cyclictest2069-21gnome-shell20:46:582
140887995536,15cyclictest2069-21gnome-shell23:13:581
140887995536,15cyclictest2069-21gnome-shell23:13:581
14088299552,10cyclictest2069-21gnome-shell22:05:580
140896995444,7cyclictest411rcuc/319:56:583
140896995439,11cyclictest2069-21gnome-shell19:34:583
14089699542,10cyclictest2069-21gnome-shell20:13:583
14089699542,10cyclictest2069-21gnome-shell00:17:583
14089199546,8cyclictest20691gnome-shell22:10:582
14089199542,48cyclictest2069-21gnome-shell22:41:582
14089199542,10cyclictest2069-21gnome-shell21:39:582
140882995445,5cyclictest2069-21gnome-shell00:04:590
140882995445,5cyclictest2069-21gnome-shell00:04:590
14088299541,10cyclictest2069-21gnome-shell23:10:590
14088299541,10cyclictest2069-21gnome-shell23:10:590
14089699532,10cyclictest2069-21gnome-shell21:20:593
14089699532,10cyclictest2069-21gnome-shell20:16:583
140891995337,12cyclictest2069-21gnome-shell20:23:582
14089199532,47cyclictest2069-21gnome-shell21:02:592
140891995319,13cyclictest141rcu_preempt21:26:592
14088299533,30cyclictest217288-21df_inode22:45:120
14088299531,10cyclictest2069-21gnome-shell23:15:580
14089699521,18cyclictest923-21dbus-daemon19:15:033
14089699521,11cyclictest2069-21gnome-shell22:24:593
14089699521,10cyclictest2069-21gnome-shell22:49:593
140891995230,17cyclictest2069-21gnome-shell19:57:582
14089199522,10cyclictest2069-21gnome-shell20:42:592
14089199522,10cyclictest2069-21gnome-shell19:18:592
14088299524,29cyclictest180525-21memory21:00:210
14088299522,8cyclictest923-21dbus-daemon22:57:160
140896995128,8cyclictest42-21ksoftirqd/319:15:033
14089699512,5cyclictest0-21swapper/320:31:593
14089699512,45cyclictest2069-21gnome-shell20:03:593
14089699511,46cyclictest2069-21gnome-shell20:25:593
14089699511,46cyclictest2069-21gnome-shell00:13:583
14089199511,46cyclictest331rcuc/200:20:582
140896995038,8cyclictest2069-21gnome-shell21:14:583
14089699502,44cyclictest2069-21gnome-shell22:17:593
14089699502,44cyclictest2069-21gnome-shell20:56:593
14089699502,44cyclictest2069-21gnome-shell20:21:583
14089699501,45cyclictest2069-21gnome-shell00:24:583
140891995014,20cyclictest250506-21unin-run00:20:002
140887995031,14cyclictest26-21ksoftirqd/119:30:001
14088799502,7cyclictest231irq_work/120:50:591
14088799501,6cyclictest4497-21PK-Backend00:02:581
14088799501,6cyclictest4497-21PK-Backend00:02:581
14089699496,38cyclictest2069-21gnome-shell22:31:583
14089199495,39cyclictest2069-21gnome-shell20:06:592
14089199492,43cyclictest0-21swapper/222:46:582
14089199492,11cyclictest2069-21gnome-shell21:46:582
14089199491,44cyclictest2069-21gnome-shell21:06:592
14088799492,6cyclictest4497-21PK-Backend20:32:581
14088799492,45cyclictest231irq_work/122:00:591
14088799492,44cyclictest231irq_work/119:30:581
14088799491,7cyclictest4497-21PK-Backend22:41:581
14088799491,7cyclictest4497-21PK-Backend00:12:581
14088799491,44cyclictest4497-21PK-Backend21:06:581
14088799491,43cyclictest4497-21PK-Backend21:54:581
14088299495,26cyclictest158942-21munin-run20:00:010
14088299494,26cyclictest216664-21munin-run22:45:010
14088299493,28cyclictest219024-21diskstats22:50:120
14088299493,28cyclictest219024-21diskstats22:50:120
140896994837,7cyclictest2069-21gnome-shell19:39:593
14089699482,7cyclictest0-21swapper/300:25:593
14089699482,6cyclictest0-21swapper/322:44:583
140887994839,5cyclictest4497-21PK-Backend23:55:591
140887994839,5cyclictest4497-21PK-Backend20:04:581
14088799482,43cyclictest0-21swapper/121:26:581
14088799482,42cyclictest0-21swapper/122:29:581
14088799481,6cyclictest4497-21PK-Backend00:29:581
14088799481,43cyclictest4497-21PK-Backend23:52:591
14088799481,43cyclictest4497-21PK-Backend22:32:591
14088299487,16cyclictest210866-21grep22:25:220
14088299485,25cyclictest250484-21munin-run00:20:000
14088299485,21cyclictest166983-21python320:20:270
14088299484,16cyclictest255689-40lsb_release00:32:000
14088299483,28cyclictest175238-21irqstats20:45:170
14088299483,27cyclictest189221-21diskstats21:25:170
14088299483,27cyclictest179293-21users20:55:300
14088299483,26cyclictest171045-21munin-run20:35:000
14088299482,43cyclictest0-21swapper/019:21:580
14088299482,43cyclictest0-21swapper/019:21:580
14088299482,28cyclictest173401-21df_inode20:40:150
14089699478,34cyclictest2069-21gnome-shell23:02:583
14089699476,35cyclictest2069-21gnome-shell00:36:593
14089699472,6cyclictest0-21swapper/321:05:583
14089699472,42cyclictest0-21swapper/323:57:583
14089699471,43cyclictest2069-21gnome-shell19:49:583
14089199472,42cyclictest4497-21PK-Backend22:29:592
140887994739,6cyclictest231irq_work/121:30:591
14088799471,5cyclictest4497-21PK-Backend22:38:591
140882994740,5cyclictest0-21swapper/022:13:590
140882994739,5cyclictest0-21swapper/020:51:590
140882994739,5cyclictest0-21swapper/000:26:590
14088299473,26cyclictest242328-21memory23:55:170
14088299473,26cyclictest150929-21diskstats19:35:170
14088299473,20cyclictest214932-21latency_hist22:40:020
14088299472,28cyclictest156473-21ntp_offset19:50:220
14088299472,28cyclictest152655-21diskstats19:40:150
14088299472,28cyclictest152655-21diskstats19:40:150
14088299472,27cyclictest223599-21munin-run23:05:000
14089699462,5cyclictest0-21swapper/323:50:583
14089699462,5cyclictest0-21swapper/323:49:583
14089699462,5cyclictest0-21swapper/300:09:583
14089699462,41cyclictest0-21swapper/323:22:583
14089199462,5cyclictest0-21swapper/221:54:592
14089199462,41cyclictest0-21swapper/223:52:582
14089199462,41cyclictest0-21swapper/222:32:582
14089199462,41cyclictest0-21swapper/220:52:582
140887994639,4cyclictest0-21swapper/121:35:591
14088799461,5cyclictest4497-21PK-Backend22:46:581
14088799461,5cyclictest141rcu_preempt23:44:581
14088799461,42cyclictest4497-21PK-Backend00:09:581
14088299469,14cyclictest937-21polkitd19:15:020
14088299469,14cyclictest193462-21grep21:35:250
14088299466,17cyclictest141839-21open_inodes19:10:210
140882994639,5cyclictest0-21swapper/022:21:580
140882994639,4cyclictest0-21swapper/022:33:580
140882994639,4cyclictest0-21swapper/020:36:580
140882994639,4cyclictest0-21swapper/019:29:580
14088299463,26cyclictest194743-21df_inode21:40:160
14088299463,26cyclictest194743-21df_inode21:40:160
197399450,31rtkit-daemon4497-21PK-Backend23:15:123
14089699455,35cyclictest2069-21gnome-shell20:35:583
140891994539,4cyclictest0-21swapper/219:27:582
140891994538,4cyclictest0-21swapper/223:55:582
140891994524,7cyclictest141rcu_preempt19:47:592
14089199452,41cyclictest0-21swapper/221:41:582
14089199452,41cyclictest0-21swapper/221:41:582
14089199452,41cyclictest0-21swapper/200:09:592
14089199451,5cyclictest4497-21PK-Backend22:00:582
14088799451,41cyclictest4497-21PK-Backend20:26:581
14088299456,18cyclictest251468-21awk00:20:210
14088299454,15cyclictest198696-21python321:50:260
140882994537,5cyclictest0-21swapper/023:26:580
14088299453,25cyclictest186590-21munin-run21:20:000
14088299453,25cyclictest186590-21munin-run21:20:000
14088299452,26cyclictest163023-21df20:10:140
14089699444,20cyclictest42-21ksoftirqd/300:00:073
14089699444,20cyclictest42-21ksoftirqd/300:00:073
140896994437,4cyclictest4497-21PK-Backend23:12:593
140896994437,4cyclictest4497-21PK-Backend23:12:593
140896994434,6cyclictest42-21ksoftirqd/319:25:133
140896994424,16cyclictest186630-21sort21:20:013
140896994424,16cyclictest186630-21sort21:20:003
140891994438,4cyclictest0-21swapper/223:31:582
140891994438,4cyclictest0-21swapper/220:04:582
140891994437,4cyclictest0-21swapper/200:28:582
140891994436,5cyclictest0-21swapper/200:39:592
140891994425,6cyclictest20691gnome-shell19:44:592
140891994425,6cyclictest20691gnome-shell19:44:592
140891994423,13cyclictest34-21ksoftirqd/221:10:282
140891994418,22cyclictest34-21ksoftirqd/222:09:552
140887994437,4cyclictest4497-21PK-Backend23:37:581
140887994437,4cyclictest4497-21PK-Backend22:14:581
140887994437,4cyclictest0-21swapper/120:23:591
140887994436,6cyclictest231irq_work/120:59:581
140887994436,5cyclictest231irq_work/120:47:591
14088799441,4cyclictest141rcu_preempt23:47:581
140882994437,4cyclictest0-21swapper/023:05:590
14088299443,19cyclictest167815-21/usr/sbin/munin20:25:250
140882994426,13cyclictest438-20systemd-journal21:30:120
140882994412,15cyclictest241124-21python323:50:400
140896994336,4cyclictest0-21swapper/323:26:583
14089199435,11cyclictest2069-21gnome-shell19:24:592
14089199435,11cyclictest2069-21gnome-shell19:24:582
14089199434,4cyclictest141rcu_preempt22:20:592
140891994337,4cyclictest0-21swapper/223:37:582
140891994337,4cyclictest0-21swapper/223:13:582
140891994337,4cyclictest0-21swapper/223:13:582
140887994336,4cyclictest4497-21PK-Backend00:39:581
14088299435,15cyclictest246720-21munin-run00:10:000
14088299434,16cyclictest0-21swapper/000:10:250
140882994336,4cyclictest0-21swapper/020:06:580
140882994335,5cyclictest0-21swapper/000:35:580
14088299433,15cyclictest207162-21sed22:15:180
14088299432,19cyclictest186906-21cpuspeed21:20:100
14089699423,17cyclictest0-21swapper/323:45:013
140891994234,5cyclictest0-21swapper/220:16:582
140891994231,7cyclictest2069-21gnome-shell20:39:592
140891994224,14cyclictest34-21ksoftirqd/223:00:232
140891994224,14cyclictest34-21ksoftirqd/200:10:112
14089199421,4cyclictest141rcu_preempt23:45:592
140891994214,14cyclictest141rcu_preempt00:30:312
140887994235,5cyclictest231irq_work/100:30:581
140887994234,5cyclictest4497-21PK-Backend00:17:591
140887994232,7cyclictest231irq_work/123:24:591
14088299423,17cyclictest164468-21ls20:15:120
14088299423,16cyclictest155546-21ls19:50:010
14088299423,15cyclictest148877-21grep19:30:140
1443372410,9sleep1251rcuc/119:15:291
14089699416,29cyclictest42-21ksoftirqd/322:25:583
14089699411,35cyclictest438-20systemd-journal22:50:123
14089699411,35cyclictest438-20systemd-journal22:50:123
140896994112,8cyclictest42-21ksoftirqd/323:10:013
14089699411,22cyclictest202550-21chrt22:02:403
140891994135,4cyclictest0-21swapper/221:56:592
140887994134,4cyclictest4497-21PK-Backend19:41:581
140887994134,4cyclictest4497-21PK-Backend19:41:581
140887994134,4cyclictest4497-21PK-Backend19:12:581
140887994123,13cyclictest26-21ksoftirqd/122:05:411
140896994010,25cyclictest42-21ksoftirqd/321:55:093
140891994029,7cyclictest2069-21gnome-shell20:11:592
140891994024,12cyclictest34-21ksoftirqd/221:35:012
140887994033,4cyclictest4497-21PK-Backend20:16:591
14088299401,16cyclictest159458-21expr20:00:140
140891993925,8cyclictest34-21ksoftirqd/223:08:582
140891993923,12cyclictest34-21ksoftirqd/221:20:002
140891993923,12cyclictest34-21ksoftirqd/221:20:002
140891993921,14cyclictest34-21ksoftirqd/223:45:012
140891993921,14cyclictest34-21ksoftirqd/223:29:562
140891993921,13cyclictest34-21ksoftirqd/219:50:132
140887993927,7cyclictest231irq_work/121:12:591
140887993922,13cyclictest26-21ksoftirqd/122:22:061
140887993921,14cyclictest26-21ksoftirqd/123:10:011
140887993921,14cyclictest26-21ksoftirqd/120:09:121
14089699381,17cyclictest233957-21sh23:35:013
140887993831,4cyclictest4497-21PK-Backend20:39:581
140887993828,7cyclictest0-21swapper/123:17:581
140887993827,6cyclictest231irq_work/120:40:581
140887993826,8cyclictest231irq_work/121:02:581
14089699371,31cyclictest2069-21gnome-shell21:45:593
14089699371,16cyclictest235880-21latency_hist23:40:023
140891993720,13cyclictest34-21ksoftirqd/221:20:132
140887993730,4cyclictest4497-21PK-Backend21:20:581
140887993730,4cyclictest4497-21PK-Backend20:13:581
140887993721,11cyclictest231477-21ntp_kernel_err23:25:231
140887993720,12cyclictest26-21ksoftirqd/122:50:511
140887993720,12cyclictest26-21ksoftirqd/122:50:511
140887993712,5cyclictest141rcu_preempt19:59:481
14089699365,26cyclictest2069-21gnome-shell19:20:583
14089699365,26cyclictest2069-21gnome-shell19:20:583
140896993619,12cyclictest161885-21wc20:07:273
140896993615,6cyclictest141rcu_preempt20:40:293
140891993618,6cyclictest34-21ksoftirqd/223:15:172
140891993618,14cyclictest34-21ksoftirqd/222:54:072
140891993618,14cyclictest34-21ksoftirqd/222:54:072
140887993623,10cyclictest220002-21kworker/u8:223:03:591
14088799361,15cyclictest4497-21PK-Backend22:18:511
140896993520,11cyclictest157252-21latency_hist19:55:013
140891993517,13cyclictest34-21ksoftirqd/222:15:152
140887993525,6cyclictest4497-21PK-Backend19:54:591
140887993511,4cyclictest141rcu_preempt19:20:321
140887993511,4cyclictest141rcu_preempt19:20:311
140896993424,6cyclictest195520-21sshd21:41:123
140896993424,6cyclictest195520-21sshd21:41:113
140896993419,11cyclictest42-21ksoftirqd/321:30:213
140887993425,5cyclictest4497-21PK-Backend19:47:591
140896993323,6cyclictest42-21ksoftirqd/322:10:193
14088799337,6cyclictest141rcu_preempt19:38:581
140887993316,13cyclictest186219-21awk21:15:301
140887993316,13cyclictest186219-21awk21:15:301
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional