You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 07:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Thu Mar 19, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
287622212262,56sleep30-21swapper/33
28752128962,10sleep20-21swapper/22
28768928051,12sleep10-21swapper/11
28748127654,16sleep00-21swapper/00
28785899232,16cyclictest0-21swapper/33
28785899221,18cyclictest0-21swapper/33
28785899221,18cyclictest0-21swapper/33
28785899211,8cyclictest0-21swapper/33
287858992112,5cyclictest6474352sleep33
28785899211,16cyclictest66950irq/132-enp1s0-TxRx-33
7263632201,4sleep20-21swapper/22
4882162200,5sleep2171rcu_preempt2
4441162202,9sleep2424687-21kworker/2:1+mm_percpu_wq2
4441162202,9sleep2424687-21kworker/2:1+mm_percpu_wq2
28785899204,12cyclictest434892-21taskset3
28785899202,14cyclictest412766-21turbostat3
287858992017,1cyclictest0-21swapper/33
28785899201,15cyclictest0-21swapper/33
28785899201,15cyclictest0-21swapper/33
28785899200,16cyclictest628335-21taskset3
4586412190,2sleep20-21swapper/22
28785899195,12cyclictest0-21swapper/33
287858991916,1cyclictest0-21swapper/33
287858991912,5cyclictest0-21swapper/33
28785899191,15cyclictest0-21swapper/33
28785899191,15cyclictest0-21swapper/33
28785899191,15cyclictest0-21swapper/33
28785899191,15cyclictest0-21swapper/33
28785899191,15cyclictest0-21swapper/33
28785899191,14cyclictest0-21swapper/33
28785899191,14cyclictest0-21swapper/33
28785899191,14cyclictest0-21swapper/33
28785899191,14cyclictest0-21swapper/33
28785899191,14cyclictest0-21swapper/33
6699322181,2sleep00-21swapper/00
5505522181,3sleep20-21swapper/22
28785899181,14cyclictest0-21swapper/33
28785899181,14cyclictest0-21swapper/33
28785899181,14cyclictest0-21swapper/33
28785899181,14cyclictest0-21swapper/33
28785899181,14cyclictest0-21swapper/33
28785899181,14cyclictest0-21swapper/33
28785899181,14cyclictest0-21swapper/33
28785899181,14cyclictest0-21swapper/33
28785899181,14cyclictest0-21swapper/33
28785899181,13cyclictest664909-21taskset3
28785899181,13cyclictest0-21swapper/33
28785899181,13cyclictest0-21swapper/33
28785899181,13cyclictest0-21swapper/33
28785899181,13cyclictest0-21swapper/33
28785899180,5cyclictest5046352sleep33
28785899180,16cyclictest0-21swapper/33
6901762171,3sleep10-21swapper/11
6562292171,2sleep00-21swapper/00
6554342172,2sleep10-21swapper/11
6449172171,3sleep20-21swapper/22
6367582171,2sleep20-21swapper/22
4840462171,8sleep1484491-21/usr/sbin/munin1
4421222171,2sleep10-21swapper/11
4421222171,2sleep10-21swapper/11
28785899177,6cyclictest369858-21turbostat3
28785899171,14cyclictest0-21swapper/33
28785899171,14cyclictest0-21swapper/33
28785899171,13cyclictest0-21swapper/33
28785899171,13cyclictest0-21swapper/33
28785899171,13cyclictest0-21swapper/33
28785899170,14cyclictest0-21swapper/33
28785899170,13cyclictest0-21swapper/33
7600922161,4sleep2418093-21diskmemload2
7091132161,4sleep1709334-21hddtemp_smartct1
6759812161,4sleep1418093-21diskmemload1
6726432161,3sleep20-21swapper/22
5233142160,1sleep20-21swapper/22
4785152161,3sleep00-21swapper/00
4347602161,3sleep1418093-21diskmemload1
28785899162,10cyclictest532834-21taskset3
28785899160,13cyclictest0-21swapper/33
28785899160,13cyclictest0-21swapper/33
28785899160,13cyclictest0-21swapper/33
28785899160,13cyclictest0-21swapper/33
28785899160,12cyclictest0-21swapper/33
28785899160,12cyclictest0-21swapper/33
7344112156,3sleep1418093-21diskmemload1
5917322150,3sleep20-21swapper/22
5287712151,10sleep2529041-21grep2
5159602151,4sleep1516048-21irqstats1
4805202151,9sleep0480735-21diskstats0
28785899152,8cyclictest306064-21taskset3
28785899152,8cyclictest306064-21taskset3
28785899151,11cyclictest0-21swapper/33
28785899150,13cyclictest0-21swapper/33
28785899150,12cyclictest0-21swapper/33
28785899150,12cyclictest0-21swapper/33
28785899150,12cyclictest0-21swapper/33
5861652141,9sleep0586386-21hddtemp_smartct0
5857322141,7sleep1418093-21diskmemload1
5596142140,2sleep20-21swapper/22
5318542141,8sleep1418093-21diskmemload1
5047572140,2sleep00-21swapper/00
4936762140,4sleep1493720-21ssh1
4419532140,2sleep00-21swapper/00
4419532140,2sleep00-21swapper/00
3249072140,2sleep1301irq_work/11
28785899149,3cyclictest0-21swapper/33
28785899143,6cyclictest0-21swapper/33
287858991411,2cyclictest0-21swapper/33
28785899140,12cyclictest0-21swapper/33
7660332131,7sleep1766291-21users1
7062592130,2sleep20-21swapper/22
6996482131,8sleep0699659-21ssh0
6578942131,4sleep2659564-21cut2
6358812130,2sleep1418093-21diskmemload1
6180132131,8sleep2618026-21ssh2
6162032131,7sleep1616301-21ntpq1
5501582131,8sleep2550211-21/usr/sbin/munin2
4671842131,8sleep2467244-21ssh2
4624622131,6sleep1462489-21munin-run1
4453282131,8sleep1445362-21ssh1
3262022131,6sleep0326251-21ntpq0
2994582132,5sleep1299512-21ntpq1
7865052121,3sleep1786518-21ssh1
7576322120,5sleep10-21swapper/11
7505122121,6sleep0418093-21diskmemload0
7087862120,2sleep20-21swapper/22
7005492121,7sleep1700634-21ssh1
6218642121,7sleep2623487-21cut2
5740412120,6sleep0418093-21diskmemload0
5418012120,1sleep00-21swapper/00
5033802121,5sleep10-21swapper/11
4337282121,2sleep00-21swapper/00
4180832121,6sleep2418109-21munin-run2
3921452121,6sleep1392261-21irqstats1
28785899129,1cyclictest0-21swapper/33
7568302111,5sleep0756888-21ntpq0
7350122111,4sleep1418093-21diskmemload1
7315542111,3sleep00-21swapper/00
7256522110,1sleep20-21swapper/22
7242672112,4sleep0418093-21diskmemload0
6985032110,2sleep20-21swapper/22
6720222111,4sleep1418093-21diskmemload1
6626612111,4sleep1418093-21diskmemload1
6446032111,4sleep1418093-21diskmemload1
6264702111,7sleep0418093-21diskmemload0
6036472111,5sleep1603821-21grep1
5992262111,4sleep20-21swapper/22
5945452111,4sleep1418093-21diskmemload1
5845772111,4sleep1418093-21diskmemload1
5829952111,5sleep00-21swapper/00
5283722111,4sleep00-21swapper/00
5110232111,4sleep1418093-21diskmemload1
4794992110,1sleep10-21swapper/11
4600212111,4sleep0418093-21diskmemload0
4505442110,3sleep2450555-21ssh2
4241512111,5sleep1424164-21ssh1
3650902111,6sleep1365216-21gltestperf1
3648352111,5sleep0365089-21df_inode0
3421662111,5sleep0342213-21ntpq0
3154732111,5sleep1315526-21ntpq1
3063182111,2sleep00-21swapper/00
3063182111,2sleep00-21swapper/00
3007122111,4sleep2300731-21apt-get2
2921212111,4sleep0292167-21ntpq0
7840402101,3sleep20-21swapper/22
7787152101,3sleep072-21kcompactd00
7762932101,2sleep20-21swapper/22
7760022101,4sleep0418093-21diskmemload0
7741102101,4sleep1774262-21ntpq1
7495512101,3sleep20-21swapper/22
7490392101,4sleep1418093-21diskmemload1
7176032101,5sleep1717631-21acpi1
7075072101,4sleep0418093-21diskmemload0
6992042100,3sleep10-21swapper/11
6879572101,3sleep20-21swapper/22
6868502101,2sleep00-21swapper/00
6791612101,2sleep20-21swapper/22
6536462100,3sleep0653659-21rm0
6457092100,1sleep00-21swapper/00
6258842101,3sleep10-21swapper/11
6003222101,4sleep0418093-21diskmemload0
5812612101,4sleep2581321-21ntpq2
5634812101,4sleep1563542-21ntpq1
5634172101,4sleep0563473-21ntpq0
5558662101,3sleep00-21swapper/00
5504822101,2sleep10-21swapper/11
5357522101,4sleep0536953-21memory0
5328202101,3sleep20-21swapper/22
5124542101,3sleep20-21swapper/22
4946982101,3sleep00-21swapper/00
4894432101,2sleep20-21swapper/22
4670542101,5sleep0467114-21ntpq0
4515422100,2sleep00-21swapper/00
4336992101,5sleep2433714-21ssh2
4254492101,2sleep00-21swapper/00
4245682101,2sleep20-21swapper/22
4117722101,4sleep0411823-21ntpq0
3762152101,5sleep0377908-21tr0
3637972100,1sleep10-21swapper/11
3591932101,4sleep1359241-21munin-run1
3488062101,5sleep2348839-21grep2
3381392101,4sleep1338397-21df_inode1
2956482101,4sleep2295751-21perl2
765652291,4sleep0765720-21ssh0
765532291,4sleep2765590-21ntpq2
741145290,3sleep20-21swapper/22
740516291,3sleep0740526-21ssh0
715218290,1sleep00-21swapper/00
679900291,4sleep0679914-21ssh0
648431291,4sleep2650053-21grep2
634995291,2sleep00-21swapper/00
615420290,5sleep0616202-21sed0
609629291,2sleep20-21swapper/22
604978291,4sleep0606666-21tr0
574053291,4sleep1574068-21ssh1
573290290,1sleep20-21swapper/22
541958291,3sleep1542089-21diskstats1
539591290,1sleep10-21swapper/11
511869290,2sleep00-21swapper/00
506049291,2sleep20-21swapper/22
476330291,4sleep2476556-21users2
468319290,5sleep1418093-21diskmemload1
411591291,4sleep2411727-21netstat2
403512291,4sleep2405243-21kthreadcore2
402911291,4sleep1403509-21kthreadcore1
352980291,5sleep0353067-21processes0
332799291,4sleep2332871-21cpuspeed_turbos2
515748281,3sleep0515956-21gltestperf0
397036280,2sleep10-21swapper/11
384796281,3sleep1384842-21ntpq1
381909281,3sleep2383641-21cut2
379583281,4sleep1379634-21open_inodes1
359972281,3sleep0360125-21kthreadcore0
354701281,3sleep2356119-21cut2
348426280,1sleep00-21swapper/00
347817281,3sleep1347847-21mailstats1
337268281,3sleep0337485-21users0
328002281,3sleep20-21swapper/22
327434280,1sleep10-21swapper/11
316751281,3sleep0316830-21tr0
306042280,2sleep20-21swapper/22
306042280,2sleep20-21swapper/22
295923281,3sleep0296663-21grep0
414410271,3sleep00-21swapper/00
401957270,1sleep00-21swapper/00
397058271,3sleep20-21swapper/22
396527270,1sleep00-21swapper/00
396439270,1sleep20-21swapper/22
391161271,3sleep00-21swapper/00
390954270,1sleep10-21swapper/11
390465271,3sleep2390528-21cat2
381115270,1sleep00-21swapper/00
374260270,1sleep00-21swapper/00
370107270,2sleep20-21swapper/22
364229270,1sleep20-21swapper/22
354124270,1sleep00-21swapper/00
344938271,3sleep2346729-21cut2
336811271,2sleep10-21swapper/11
321582270,1sleep20-21swapper/22
316718270,1sleep10-21swapper/11
311471270,3sleep20-21swapper/22
305644270,1sleep10-21swapper/11
292822270,1sleep20-21swapper/22
379636260,2sleep20-21swapper/22
370407260,3sleep1370634-21idleruntime1
353600260,1sleep10-21swapper/11
327065260,1sleep20-21swapper/22
316429260,4sleep00-21swapper/00
310873260,1sleep10-21swapper/11
310873260,1sleep10-21swapper/11
293061260,1sleep10-21swapper/11
412805250,1sleep10-21swapper/11
412065250,1sleep10-21swapper/11
327720250,2sleep0327850-21gltestperf0
303344250,1sleep00-21swapper/00
402712240,1sleep00-21swapper/00
364804240,1sleep20-21swapper/22
337855240,1sleep20-21swapper/22
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional