You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-14 - 03:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sun Dec 14, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
71858625240,19sleep342-21ksoftirqd/319:05:183
82746021100,3sleep2827459-21acpi00:15:122
7796952980,4sleep30-21swapper/322:00:193
8337312790,3sleep00-21swapper/000:30:310
8313022790,7sleep34497-21PK-Backend00:25:143
7814582760,5sleep20-21swapper/222:05:212
71942299752,16cyclictest2069-21gnome-shell00:13:592
71856127544,22sleep20-21swapper/219:05:172
197399750,5rtkit-daemon0-21swapper/122:24:001
71781527444,20sleep00-21swapper/019:05:080
197399740,5rtkit-daemon0-21swapper/123:00:411
197399740,5rtkit-daemon0-21swapper/123:00:401
71942099731,11cyclictest2069-21gnome-shell22:36:591
8310752720,6sleep20-21swapper/200:25:132
71942299722,12cyclictest2069-21gnome-shell21:35:582
197399720,5rtkit-daemon4497-21PK-Backend19:48:491
71942099719,38cyclictest738702-21seq20:03:021
197399710,5rtkit-daemon4497-21PK-Backend20:43:491
197399710,5rtkit-daemon0-21swapper/120:21:451
197399710,4rtkit-daemon0-21swapper/122:08:521
71942099702,63cyclictest2069-21gnome-shell23:12:581
719420996839,24cyclictest2069-21gnome-shell19:17:581
71942099681,47cyclictest2069-21gnome-shell19:28:591
71784326739,22sleep10-21swapper/119:05:091
71942599662,22cyclictest2069-21gnome-shell20:28:593
71942099662,5cyclictest0-21swapper/119:20:581
71942099652,7cyclictest719411-21cyclictest22:52:581
71942099652,6cyclictest0-21swapper/100:08:581
71941699655,7cyclictest141rcu_preempt22:16:580
71942099632,5cyclictest0-21swapper/119:59:591
8002522610,5chrt0-21swapper/022:55:290
71942299612,54cyclictest2069-21gnome-shell00:03:582
719420996152,6cyclictest719411-21cyclictest19:41:591
71942099614,6cyclictest0-21swapper/120:52:581
71942099612,5cyclictest0-21swapper/123:27:591
71942099612,5cyclictest0-21swapper/123:27:581
719420996037,20cyclictest0-21swapper/119:11:581
71942099602,8cyclictest785896-40lsb_release22:16:581
71941699602,16cyclictest2069-21gnome-shell21:11:590
71942099592,6cyclictest0-21swapper/121:49:581
71942099592,5cyclictest0-21swapper/121:33:591
71942099591,7cyclictest2069-21gnome-shell21:20:581
71942099591,16cyclictest2069-21gnome-shell23:05:581
719420995840,15cyclictest0-21swapper/120:16:591
719420995840,15cyclictest0-21swapper/120:16:581
719420995829,25cyclictest2069-21gnome-shell23:19:581
71942099581,6cyclictest4497-21PK-Backend20:55:581
71942599572,50cyclictest2069-21gnome-shell00:11:583
71942299572,50cyclictest2069-21gnome-shell20:25:582
71942099572,42cyclictest0-21swapper/121:43:581
7275252560,7sleep34497-21PK-Backend19:30:203
71942099567,46cyclictest26-21ksoftirqd/119:33:591
71942599557,43cyclictest141rcu_preempt20:24:583
719420995526,25cyclictest828973-21systemd00:20:021
71942099552,48cyclictest2069-21gnome-shell22:27:591
71942099552,48cyclictest2069-21gnome-shell00:38:581
719416995539,12cyclictest2069-21gnome-shell22:49:580
8144782540,9sleep0814462-21cron23:40:000
71942599543,46cyclictest796884-40lsb_release22:46:593
71942299542,9cyclictest2069-21gnome-shell21:57:592
71942299542,48cyclictest2069-21gnome-shell20:40:582
71942299542,11cyclictest2069-21gnome-shell22:01:582
71941699542,10cyclictest2069-21gnome-shell19:21:580
71942099525,7cyclictest231irq_work/100:13:581
71942099525,45cyclictest231irq_work/120:11:581
719420995225,22cyclictest2069-21gnome-shell00:23:591
719420995217,30cyclictest760613-21which21:05:201
71941699526,43cyclictest0-21swapper/021:20:580
71941699523,45cyclictest826843-40lsb_release00:10:580
71941699521,10cyclictest2069-21gnome-shell19:34:580
719422995139,7cyclictest2069-21gnome-shell20:05:582
71942299512,45cyclictest2069-21gnome-shell23:34:582
71942299512,45cyclictest2069-21gnome-shell21:05:582
71942099514,7cyclictest231irq_work/123:52:591
719420995142,6cyclictest231irq_work/120:26:581
71942099512,45cyclictest231irq_work/122:12:581
719420995116,30cyclictest812335-21cat23:30:271
719420995115,31cyclictest833429-21seq00:30:261
719425995033,13cyclictest2069-21gnome-shell20:42:583
71942299502,44cyclictest0-21swapper/222:46:592
71942099506,39cyclictest770782-21cstates21:35:131
719420995043,4cyclictest231irq_work/121:50:581
71942099502,44cyclictest2069-21gnome-shell22:42:581
719420995018,27cyclictest821582-21date00:00:031
719420995012,34cyclictest1-21systemd19:40:021
719416995038,8cyclictest2069-21gnome-shell00:28:580
197399500,36rtkit-daemon821614-21date00:00:030
197399500,35rtkit-daemon4497-21PK-Backend23:45:180
7783882490,6chrt4497-21PK-Backend21:55:271
71942599492,6cyclictest4497-21PK-Backend19:40:593
719425994922,23cyclictest42-21ksoftirqd/321:20:233
719425994918,26cyclictest2069-21gnome-shell21:44:593
71942299498,35cyclictest2069-21gnome-shell21:52:592
719420994938,7cyclictest823158-40lsb_release00:01:581
719420994937,8cyclictest2069-21gnome-shell22:30:591
71942099492,44cyclictest231irq_work/123:24:591
71942099492,44cyclictest231irq_work/120:39:591
71942099492,44cyclictest231irq_work/120:39:581
71942099492,44cyclictest231irq_work/100:29:591
719420994917,28cyclictest759348-21python321:00:281
719420994917,28cyclictest759348-21python321:00:271
71941699493,43cyclictest4497-21PK-Backend20:32:590
71942599482,42cyclictest4497-21PK-Backend22:50:583
71942599482,42cyclictest4497-21PK-Backend22:35:593
71942299485,36cyclictest2069-21gnome-shell20:11:592
71942299482,6cyclictest0-21swapper/220:21:582
71942299482,42cyclictest0-21swapper/222:25:592
719420994840,6cyclictest231irq_work/121:28:591
71942099481,6cyclictest4497-21PK-Backend20:46:591
719416994839,6cyclictest2069-21gnome-shell23:08:590
719416994836,8cyclictest2069-21gnome-shell19:42:590
71941699482,6cyclictest0-21swapper/022:41:590
197399480,34rtkit-daemon751977-21awk20:40:210
197399480,32rtkit-daemon771171-21memory21:35:210
7582582470,18sleep234-21ksoftirqd/221:00:022
71942599472,42cyclictest0-21swapper/323:13:593
71942599472,41cyclictest4497-21PK-Backend20:05:593
71942599471,6cyclictest4497-21PK-Backend19:18:593
71942599471,43cyclictest4497-21PK-Backend22:23:583
71942599471,42cyclictest4497-21PK-Backend21:02:583
71942599471,42cyclictest4497-21PK-Backend21:02:583
71942299472,41cyclictest4497-21PK-Backend23:15:592
71942299472,41cyclictest0-21swapper/200:35:582
71942099479,33cyclictest739365-21cat20:05:141
719420994739,4cyclictest4497-21PK-Backend23:45:591
719420994738,6cyclictest231irq_work/122:04:591
71942099472,42cyclictest0-21swapper/123:40:581
719420994716,27cyclictest762821-21switchtime21:10:271
719420994711,31cyclictest795979-21expr22:45:131
71941699472,5cyclictest0-21swapper/020:57:590
719416994725,6cyclictest141rcu_preempt20:53:590
71941699472,42cyclictest0-21swapper/023:02:590
71941699472,42cyclictest0-21swapper/023:02:580
197399470,33rtkit-daemon0-21swapper/023:10:130
71942599465,36cyclictest2069-21gnome-shell23:57:583
719425994638,5cyclictest4497-21PK-Backend19:56:583
71942599461,6cyclictest4497-21PK-Backend19:10:593
71942599461,5cyclictest4497-21PK-Backend22:12:593
71942299466,34cyclictest2069-21gnome-shell23:48:582
719422994639,4cyclictest0-21swapper/222:50:592
719422994634,8cyclictest2069-21gnome-shell21:25:592
71942299462,5cyclictest0-21swapper/221:16:582
71942299462,5cyclictest0-21swapper/221:13:582
71942299462,41cyclictest0-21swapper/222:13:582
71942299462,41cyclictest0-21swapper/219:40:582
71942299462,41cyclictest0-21swapper/200:32:582
71942099468,34cyclictest763986-21cat21:15:151
71942099465,37cyclictest813178-21tr23:35:131
719420994616,26cyclictest800714-21ls23:00:021
197399460,6rtkit-daemon0-21swapper/023:30:020
197399460,6rtkit-daemon0-21swapper/023:30:020
197399460,31rtkit-daemon750003-21irqstats20:35:190
197399460,31rtkit-daemon750003-21irqstats20:35:190
7378172450,17sleep234-21ksoftirqd/220:00:172
719425994537,4cyclictest4497-21PK-Backend22:05:593
719422994533,8cyclictest2069-21gnome-shell21:48:592
71942299452,41cyclictest0-21swapper/219:18:582
719420994536,6cyclictest231irq_work/120:31:581
719420994511,29cyclictest734602-21sed19:50:221
719420994511,29cyclictest734602-21sed19:50:221
71941699451,4cyclictest141rcu_preempt22:31:580
197399450,30rtkit-daemon797668-21expr22:50:120
197399450,30rtkit-daemon738283-21python320:00:270
71942599441,37cyclictest1941-21systemd20:51:083
71942599441,36cyclictest2069-21gnome-shell20:10:593
719422994437,4cyclictest0-21swapper/222:43:582
719422994433,7cyclictest2069-21gnome-shell20:30:582
719416994438,4cyclictest0-21swapper/021:15:590
719416994436,5cyclictest0-21swapper/022:06:590
197399440,30rtkit-daemon1-21systemd22:30:000
8348982430,8sleep013-21ksoftirqd/000:35:220
719425994315,8cyclictest4497-21PK-Backend21:10:293
71942299431,36cyclictest2069-21gnome-shell23:56:582
197399430,29rtkit-daemon0-21swapper/021:10:020
197399430,29rtkit-daemon0-21swapper/000:20:140
71942599423,34cyclictest2069-21gnome-shell23:29:593
71942599423,34cyclictest2069-21gnome-shell23:29:583
71942599422,35cyclictest2069-21gnome-shell20:00:583
71942299423,34cyclictest2069-21gnome-shell20:35:592
71942299423,34cyclictest2069-21gnome-shell20:35:592
719422994223,14cyclictest787081-21df22:20:162
197399420,29rtkit-daemon0-21swapper/022:10:150
8034852410,7sleep34497-21PK-Backend23:05:273
719425994129,8cyclictest2069-21gnome-shell23:49:583
719425994121,17cyclictest0-21swapper/319:48:593
71942599411,5cyclictest4497-21PK-Backend00:32:583
719422994134,4cyclictest4497-21PK-Backend22:34:582
719422994129,8cyclictest700800-21kworker/2:220:19:592
719422994129,8cyclictest700800-21kworker/2:220:19:592
719422994125,10cyclictest2069-21gnome-shell23:20:582
71942299412,33cyclictest2069-21gnome-shell21:41:582
71941699411,36cyclictest768631-21kworker/0:021:50:580
197399410,26rtkit-daemon4497-21PK-Backend19:55:270
719425994017,11cyclictest42-21ksoftirqd/321:08:113
71942599401,33cyclictest2069-21gnome-shell20:49:583
719422994031,6cyclictest2069-21gnome-shell19:56:592
719422994023,12cyclictest752714-21apt-get20:45:032
71941699401,35cyclictest768631-21kworker/0:023:17:580
197399400,26rtkit-daemon0-21swapper/022:20:160
719422993924,11cyclictest816188-21cron23:45:012
719422993915,7cyclictest7292962wc19:35:212
71941699394,32cyclictest13-21ksoftirqd/020:06:590
719416993925,10cyclictest13-21ksoftirqd/019:50:230
719416993925,10cyclictest13-21ksoftirqd/019:50:230
71941699391,34cyclictest821434-21kworker/0:200:03:590
71941699391,34cyclictest768631-21kworker/0:022:36:590
71941699391,34cyclictest742337-21turbostat20:15:000
197399390,25rtkit-daemon0-21swapper/021:42:100
7888532380,7chrt4497-21PK-Backend22:25:143
719425993832,4cyclictest0-21swapper/323:50:583
719425993821,5cyclictest42-21ksoftirqd/319:27:423
719422993815,7cyclictest141rcu_preempt21:30:212
71942299381,32cyclictest2069-21gnome-shell23:35:592
71942299381,21cyclictest800251-21users22:55:292
719416993827,7cyclictest0-21swapper/021:34:580
719416993826,8cyclictest0-21swapper/023:30:590
719416993826,8cyclictest0-21swapper/000:09:580
197399380,24rtkit-daemon0-21swapper/021:57:470
197399380,24rtkit-daemon0-21swapper/021:47:450
719425993727,6cyclictest734100-21cpuspeed_turbos19:50:143
719425993727,6cyclictest734100-21cpuspeed_turbos19:50:133
719425993720,5cyclictest42-21ksoftirqd/320:59:313
719425993720,13cyclictest835133-21ntpq00:35:253
719422993721,12cyclictest825176-21apt-get00:10:012
719422993721,11cyclictest976-21rs:main2
719422993721,11cyclictest976-21rs:main2
719416993727,6cyclictest2069-21gnome-shell19:45:590
719416993727,6cyclictest181irq_work/020:20:590
71941699372,31cyclictest0-21swapper/023:40:580
719416993722,11cyclictest1-21systemd20:20:000
719416993722,11cyclictest1-21systemd20:20:000
71941699371,32cyclictest699072-21kworker/0:119:35:590
197399370,23rtkit-daemon0-21swapper/021:29:030
7640882360,7sleep34497-21PK-Backend21:15:203
719425993629,4cyclictest4497-21PK-Backend21:26:583
719425993622,10cyclictest778952-21cron21:59:593
719425993621,5cyclictest42-21ksoftirqd/300:19:553
719425993621,11cyclictest823433-21cron00:05:013
719425993612,20cyclictest823924-21cstates00:05:153
719422993628,5cyclictest0-21swapper/219:13:582
719422993620,12cyclictest784850-21tr22:15:142
719416993626,6cyclictest4497-21PK-Backend23:20:580
719416993626,6cyclictest1081-21ntpd21:00:230
719416993626,6cyclictest1081-21ntpd21:00:230
197399360,25rtkit-daemon0-21swapper/023:53:460
719425993519,5cyclictest42-21ksoftirqd/321:33:593
71942599351,29cyclictest4497-21PK-Backend22:15:523
719422993524,7cyclictest723741-21tr19:20:132
719422993524,7cyclictest2069-21gnome-shell23:50:582
719416993522,9cyclictest778953-21cron22:04:550
719416993519,12cyclictest722806-21find19:15:370
719416993519,12cyclictest720620-21cat19:10:250
71941699351,30cyclictest740655-21kworker/0:020:28:590
719425993427,4cyclictest4497-21PK-Backend19:24:593
719425993424,6cyclictest4497-21PK-Backend23:36:593
719425993424,6cyclictest0-21swapper/322:31:583
719425993419,11cyclictest748247-21cat20:30:223
71942599341,5cyclictest4497-21PK-Backend21:51:583
71942599341,29cyclictest4497-21PK-Backend22:41:123
719425993410,20cyclictest42-21ksoftirqd/300:21:243
719422993425,6cyclictest0-21swapper/222:39:582
719422993420,10cyclictest805852-21systemd23:15:022
719422993419,11cyclictest726015-21awk19:25:232
719422993418,11cyclictest727139-21sort19:30:142
71942299341,7cyclictest4497-21PK-Backend21:00:222
71942299341,7cyclictest4497-21PK-Backend21:00:222
719416993424,6cyclictest4497-21PK-Backend20:47:590
719416993419,11cyclictest827528-21awk00:15:130
719416993418,11cyclictest438-20systemd-journal19:25:150
71942599338,13cyclictest42-21ksoftirqd/319:36:323
71942599332,9cyclictest774917-21python321:45:283
719425993323,6cyclictest4497-21PK-Backend23:30:593
71942599332,26cyclictest806484-21hddtemp_smartct23:15:153
719425993318,11cyclictest815760-21grep23:40:283
719425993317,5cyclictest42-21ksoftirqd/321:37:103
71942599331,29cyclictest4497-21PK-Backend22:59:383
71942599331,29cyclictest0-21swapper/323:22:583
71942599331,28cyclictest802035-21taskset23:01:023
71942599331,28cyclictest802035-21taskset23:01:013
719422993325,5cyclictest0-21swapper/221:21:592
71942599321,4cyclictest0-21swapper/320:18:353
71942599321,4cyclictest0-21swapper/320:18:353
719422993221,7cyclictest809598-21cpu23:25:132
719422993221,7cyclictest809598-21cpu23:25:132
71942299321,27cyclictest733095-21runrttasks19:45:272
719425993118,10cyclictest42-21ksoftirqd/320:39:213
719425993118,10cyclictest42-21ksoftirqd/320:39:203
719422993125,4cyclictest0-21swapper/220:53:502
719422993121,5cyclictest34-21ksoftirqd/200:23:162
719422993116,11cyclictest802628-21apt23:05:122
71942299311,10cyclictest0-21swapper/223:00:132
71942299311,10cyclictest0-21swapper/223:00:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional