You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 19:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot5s.osadl.org (updated Sat Nov 15, 2025 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2638543995445,4cyclictest2659768-21munin-plugin-st08:10:012
2638540994840,4cyclictest2661565-21latency_hist08:15:011
2638540994824,6cyclictest131rcu_preempt10:15:121
2638535994841,5cyclictest2711650-21kworker/u8:2+events_unbound10:56:080
2638535994841,4cyclictest2732465-21cut11:40:000
2638548994738,5cyclictest2711998-21munin-plugin-st10:40:013
2638543994638,4cyclictest2753163-21sendmail12:40:002
263854399462,38cyclictest2668729-21apt-get08:35:132
2638540994636,7cyclictest2702793-21grep10:10:331
2638540994634,8cyclictest2677052-21sendmail-msp09:00:011
263854399452,38cyclictest2677483-21/usr/sbin/munin09:00:132
263854399442,36cyclictest2694538-21apt-get09:50:132
2638543994419,21cyclictest2690475-21grep09:35:272
2638543994415,25cyclictest2731224-21perl11:35:192
263854399441,38cyclictest2724347-21/usr/sbin/munin11:15:172
263854399432,36cyclictest2684769-21/usr/sbin/munin09:20:222
2638543994319,20cyclictest2645506-21grep07:25:272
263854399431,39cyclictest2714550-21/usr/sbin/munin10:45:252
263854399431,37cyclictest2750278-21/usr/sbin/munin12:30:202
263854399431,37cyclictest2741442-21apt-get12:05:132
263854399431,36cyclictest2654656-21apt-get07:55:142
2638540994332,7cyclictest2647198-21grep07:30:271
2638535994331,7cyclictest0-21swapper/008:00:010
263854399428,29cyclictest2734799-21sed11:45:192
263854399421,38cyclictest2747132-21/usr/sbin/munin12:20:222
263854399421,38cyclictest2729029-21sshd11:29:482
263854399421,38cyclictest2705684-21/usr/sbin/munin10:20:192
263854399421,38cyclictest2641466-21/usr/sbin/munin07:15:212
263854399421,36cyclictest2698175-21apt-get10:00:152
263854399421,36cyclictest2683149-21/usr/sbin/munin09:15:252
263854399421,36cyclictest2665572-21/usr/sbin/munin08:25:172
263854399421,34cyclictest2699874-21apt-get10:05:152
2638548994133,5cyclictest299-21systemd-logind11:01:063
263854399412,36cyclictest2736656-21/usr/sbin/munin11:50:242
2638543994114,24cyclictest2675840-21if_eth008:55:192
263854399411,37cyclictest2681463-21ntp_states09:10:252
263854399411,35cyclictest2715643-21apt-get10:50:152
263854399411,35cyclictest2674407-21/usr/sbin/munin08:50:222
263854399411,34cyclictest2719066-21apt-get11:00:162
2638540994111,6cyclictest131rcu_preempt12:20:131
2638535994128,9cyclictest121ktimers/007:35:130
263854399403,34cyclictest2708497-21cron10:29:002
263854399401,36cyclictest2749460-21seq12:26:442
263854399401,36cyclictest2738663-21/usr/sbin/munin11:55:302
263854399401,36cyclictest2738663-21/usr/sbin/munin11:55:292
263854399401,36cyclictest2726335-21/usr/sbin/munin11:20:192
263854399401,36cyclictest2704170-21/usr/sbin/munin10:15:242
263854399401,36cyclictest2697523-21/usr/sbin/munin09:55:272
263854399401,36cyclictest2644108-21cron07:24:012
263854399401,35cyclictest2692966-21/usr/sbin/munin09:45:162
263854399401,35cyclictest2679289-21df09:05:162
263854399401,34cyclictest2729291-21munin-node11:30:162
263854399401,34cyclictest2729291-21munin-node11:30:162
263854399400,34cyclictest0-21swapper/208:30:142
263854399400,34cyclictest0-21swapper/208:30:142
2638540994034,4cyclictest2682585-21kworker/u8:1+events_unbound09:46:001
2638540994019,9cyclictest131rcu_preempt09:50:121
263854399398,28cyclictest2722840-21wc11:10:192
263854399397,27cyclictest2703468-21date10:15:012
263854399393,33cyclictest2664076-21memory08:20:222
263854399391,35cyclictest2717398-21/usr/sbin/munin10:55:152
263854399391,35cyclictest2662211-21/usr/sbin/munin08:15:182
263854399391,35cyclictest2649999-21/usr/sbin/munin07:40:212
263854399391,32cyclictest2638706-21apt-get07:10:122
2638540993930,6cyclictest2721822-21grep11:05:271
2638540993911,5cyclictest131rcu_preempt08:00:121
2638535993928,7cyclictest2735881-21apt11:50:010
2638535993922,10cyclictest131rcu_preempt11:19:490
2638548993831,4cyclictest2684075-21grep09:20:013
263854399383,32cyclictest2686466-21memory09:25:202
263854399382,33cyclictest2733692-21/usr/sbin/munin11:40:272
263854399382,33cyclictest2691230-21/usr/sbin/munin09:40:162
263854399381,34cyclictest2743243-21/usr/sbin/munin12:10:172
263854399381,34cyclictest2712542-21if_eth010:40:192
263854399381,34cyclictest2709843-21unixbench_singl10:30:272
263854399381,34cyclictest2688165-21/usr/sbin/munin09:30:212
263854399381,34cyclictest2659990-21/usr/sbin/munin08:10:122
263854399381,34cyclictest2648320-21/usr/sbin/munin07:35:212
263854399381,34cyclictest2646134-21/usr/sbin/munin07:30:132
263854399381,30cyclictest2710523-21apt-get10:35:142
2638540993831,5cyclictest2654320-21kworker/u8:1+events_unbound07:55:301
2638535993832,4cyclictest2687181-21kworker/u8:2+events_unbound09:30:570
2638535993830,4cyclictest2751445-21idleruntime-cro12:35:000
2638535993822,4cyclictest131rcu_preempt10:01:070
2638535993817,5cyclictest2697768-21kworker/u8:2-events_unbound09:59:080
2638548993728,4cyclictest0-21swapper/309:30:013
263854399371,34cyclictest2721336-21ntp_states11:05:232
263854399371,33cyclictest53-21kcompactd012:01:182
263854399371,33cyclictest2672262-21df_inode08:45:172
263854399371,33cyclictest2652952-21/usr/sbin/munin07:50:132
263854399371,30cyclictest2651254-21apt-get07:45:142
2638540993715,17cyclictest231ktimers/111:10:131
2638535993731,4cyclictest2665024-21kworker/u8:1+events_unbound08:30:410
2638535993731,4cyclictest2665024-21kworker/u8:1+events_unbound08:30:400
2638535993731,3cyclictest2704982-21sendmail10:20:010
2638535993728,6cyclictest2702804-21unixbench_multi10:10:330
2638535993722,4cyclictest131rcu_preempt10:48:590
2638535993719,10cyclictest131rcu_preempt09:44:070
2638535993718,5cyclictest131rcu_preempt09:15:590
2638535993715,5cyclictest131rcu_preempt11:35:000
2638535993715,5cyclictest131rcu_preempt11:35:000
2638548993627,5cyclictest0-21swapper/312:00:003
2638548993627,5cyclictest0-21swapper/312:00:003
2638548993627,4cyclictest0-21swapper/308:35:003
2638548993627,4cyclictest0-21swapper/308:34:593
2638548993626,6cyclictest0-21swapper/307:55:263
263854399362,31cyclictest2745290-21/usr/sbin/munin12:15:212
2638543993619,13cyclictest0-21swapper/208:40:212
263854399360,2cyclictest0-21swapper/208:00:112
2638540993629,4cyclictest2718815-21cron11:00:001
2638535993627,6cyclictest2661450-21cron08:15:010
2638535993617,9cyclictest131rcu_preempt10:05:050
2638548993527,4cyclictest0-21swapper/310:05:273
263854099352,4cyclictest0-21swapper/109:20:151
2638540993511,7cyclictest131rcu_preempt11:30:181
2638540993511,7cyclictest131rcu_preempt11:30:181
2638535993522,4cyclictest131rcu_preempt12:24:260
2638535993522,4cyclictest131rcu_preempt08:46:230
2638535993518,4cyclictest131rcu_preempt10:30:210
2638535993518,10cyclictest131rcu_preempt11:55:310
2638535993518,10cyclictest131rcu_preempt11:55:310
2638535993517,4cyclictest131rcu_preempt11:07:410
2638535993517,4cyclictest0-21swapper/009:39:410
2638548993425,4cyclictest0-21swapper/311:15:263
2638540993427,4cyclictest0-21swapper/108:25:261
2638540993427,3cyclictest0-21swapper/112:00:001
2638540993427,3cyclictest0-21swapper/112:00:001
2638540993426,4cyclictest0-21swapper/112:35:291
2638540993424,7cyclictest2680661-21ls09:10:011
2638540993424,6cyclictest0-21swapper/108:20:001
2638540993418,10cyclictest2742653-21kworker/u8:0+events_unbound12:09:491
263853599348,5cyclictest131rcu_preempt11:50:210
263853599347,6cyclictest11-21ksoftirqd/007:10:010
2638535993428,4cyclictest2730406-21kworker/u8:1+events_unbound12:00:320
2638535993426,4cyclictest0-21swapper/010:20:280
2638535993419,4cyclictest131rcu_preempt12:39:590
2638535993418,4cyclictest131rcu_preempt10:52:590
2638535993418,4cyclictest131rcu_preempt08:18:190
2638535993416,4cyclictest131rcu_preempt08:25:000
2638535993416,11cyclictest131rcu_preempt08:55:170
2638535993414,9cyclictest131rcu_preempt09:00:350
2638535993414,11cyclictest131rcu_preempt07:25:170
2638535993413,11cyclictest131rcu_preempt08:10:010
2638548993325,4cyclictest0-21swapper/308:45:193
2638548993324,5cyclictest0-21swapper/309:10:013
2638548993323,7cyclictest2658045-21apt-key08:05:013
263854099337,4cyclictest131rcu_preempt12:10:161
2638540993325,4cyclictest0-21swapper/109:15:431
2638540993322,7cyclictest2690893-21run-parts09:40:001
263853599336,16cyclictest11-21ksoftirqd/009:25:160
2638535993319,6cyclictest131rcu_preempt07:15:010
2638535993319,5cyclictest131rcu_preempt07:43:100
2638535993319,4cyclictest131rcu_preempt11:40:440
2638535993319,4cyclictest131rcu_preempt11:03:440
2638535993318,4cyclictest131rcu_preempt12:10:120
2638535993316,9cyclictest131rcu_preempt08:28:350
2638535993315,9cyclictest131rcu_preempt09:20:240
2638535993315,4cyclictest131rcu_preempt10:36:320
2638535993315,4cyclictest131rcu_preempt09:06:110
2638535993313,9cyclictest131rcu_preempt09:10:590
2638535993313,5cyclictest131rcu_preempt07:15:020
2638535993313,11cyclictest131rcu_preempt07:30:250
2638535993312,4cyclictest131rcu_preempt12:16:500
2638535993311,4cyclictest131rcu_preempt08:04:450
2638548993225,5cyclictest2673351-21kworker/u8:0+events_unbound08:50:493
2638548993222,6cyclictest2661457-21cron08:15:013
2638540993225,4cyclictest2733860-21kworker/u8:2+events_unbound12:01:181
2638540993218,6cyclictest2680685-21kworker/u8:0+events_unbound09:25:371
2638540993218,4cyclictest2658280-21kworker/u8:4+events_unbound08:08:431
263854099321,27cyclictest1-21systemd09:00:491
263853599325,14cyclictest2713703-21cron10:45:010
2638535993220,7cyclictest121ktimers/008:40:000
2638535993219,4cyclictest131rcu_preempt11:11:450
2638535993215,4cyclictest131rcu_preempt12:26:450
263853599321,15cyclictest2727914-21/usr/sbin/munin11:25:190
2638535993211,13cyclictest11-21ksoftirqd/009:55:010
2638548993125,4cyclictest2682585-21kworker/u8:1+events_unbound09:20:563
2638548993124,4cyclictest2753152-21sendmail12:40:003
2638548993123,4cyclictest0-21swapper/312:16:183
2638548993123,4cyclictest0-21swapper/311:05:263
2638548993123,4cyclictest0-21swapper/308:25:003
2638548993122,6cyclictest0-21swapper/310:20:013
2638548993122,4cyclictest0-21swapper/311:25:283
2638548993120,7cyclictest0-21swapper/309:50:263
263854099312,5cyclictest131rcu_preempt09:10:131
2638540993123,3cyclictest0-21swapper/109:45:001
2638540993118,9cyclictest1-21systemd11:36:131
263854099311,5cyclictest131rcu_preempt10:50:141
2638540993112,10cyclictest2751281-21kworker/u8:0+events_unbound12:33:591
2638535993118,4cyclictest131rcu_preempt09:45:500
2638535993115,3cyclictest131rcu_preempt08:42:110
2638535993114,9cyclictest131rcu_preempt10:25:190
2638535993113,10cyclictest131rcu_preempt07:20:230
2638535993111,9cyclictest131rcu_preempt12:06:080
2638548993023,5cyclictest2643852-21kworker/u8:0+events_unbound07:25:173
263854099302,3cyclictest0-21swapper/107:15:131
2638540993023,4cyclictest2746321-21cron12:20:011
2638540993021,5cyclictest2727037-21kworker/u8:3+events_unbound11:30:001
2638540993020,6cyclictest0-21swapper/107:35:171
2638540993012,6cyclictest2732124-21kworker/u8:0+events_unbound11:45:211
263853599303,4cyclictest11-21ksoftirqd/007:50:010
2638535993012,4cyclictest131rcu_preempt11:24:190
2638548992921,4cyclictest0-21swapper/311:40:273
2638548992918,7cyclictest0-21swapper/311:35:003
2638548992918,7cyclictest0-21swapper/311:34:593
263854099292,2cyclictest131rcu_preempt10:05:131
2638540992922,5cyclictest2654320-21kworker/u8:1+events_unbound07:54:211
2638540992922,4cyclictest0-21swapper/107:45:291
2638548992820,4cyclictest0-21swapper/307:50:263
2638548992819,6cyclictest2642099-21kworker/u8:2+events_unbound07:40:193
2638540992823,3cyclictest637-21runrttasks12:25:171
2638540992822,4cyclictest2733860-21kworker/u8:2+events_unbound11:53:431
2638540992822,4cyclictest2709931-21kworker/u8:0+events_unbound10:31:051
2638540992822,4cyclictest2642099-21kworker/u8:2+events_unbound07:27:101
2638540992822,3cyclictest2668499-21cron08:35:001
2638540992822,3cyclictest2668499-21cron08:35:001
2638540992821,5cyclictest2718492-21kworker/u8:1+events_unbound11:01:081
263853599289,4cyclictest131rcu_preempt08:50:260
2638548992720,3cyclictest0-21swapper/308:35:173
2638548992719,4cyclictest0-21swapper/311:40:003
2638548992717,6cyclictest0-21swapper/307:35:263
263854099274,4cyclictest131rcu_preempt08:40:141
263854099272,4cyclictest131rcu_preempt08:50:131
2638540992720,4cyclictest0-21swapper/109:55:271
2638540992719,4cyclictest0-21swapper/111:20:001
2638540992718,6cyclictest2726946-21unixbench_singl11:20:241
2638540992717,6cyclictest0-21swapper/110:45:011
2638540992711,6cyclictest2713686-21kworker/u8:0+events_unbound10:45:311
263854099271,14cyclictest0-21swapper/108:37:091
2638548992618,4cyclictest0-21swapper/311:10:273
2638548992617,6cyclictest2639768-21unixbench_singl07:10:253
2638548992616,6cyclictest2744609-21apt12:15:003
2638548992615,7cyclictest0-21swapper/308:55:273
2638548992614,8cyclictest1-21systemd09:56:003
2638548992613,8cyclictest0-21swapper/307:30:173
263854099269,5cyclictest2730406-21kworker/u8:1+events_unbound11:43:501
2638540992619,5cyclictest2709931-21kworker/u8:0+events_unbound10:35:281
2638540992619,3cyclictest0-21swapper/110:20:181
2638540992618,6cyclictest2636943-21kworker/u8:0+events_unbound07:11:211
2638540992617,6cyclictest2689181-21sh09:35:001
263854099261,4cyclictest131rcu_preempt07:20:151
2638548992518,4cyclictest2687856-21fschecks_count09:30:163
2638548992516,4cyclictest0-21swapper/310:25:193
2638540992511,6cyclictest24-21ksoftirqd/110:25:161
263854899249,11cyclictest391ktimers/312:30:153
263854899242,18cyclictest2658454-21fschecks_count08:05:163
2638548992417,5cyclictest2746024-21kworker/u8:3+events_unbound12:21:213
2638548992417,3cyclictest0-21swapper/309:35:173
2638548992416,4cyclictest0-21swapper/312:05:263
2638548992416,4cyclictest0-21swapper/310:00:183
2638548992414,5cyclictest0-21swapper/309:00:263
263854899240,7cyclictest2642488-21munin-plugin-st07:20:013
2638540992414,6cyclictest0-21swapper/110:05:001
2638540992411,6cyclictest24-21ksoftirqd/108:50:021
263854099240,12cyclictest0-21swapper/107:40:191
2638548992317,4cyclictest2661302-21kworker/u8:1+events_unbound08:18:153
2638548992317,3cyclictest0-21swapper/312:00:223
2638548992317,3cyclictest0-21swapper/310:55:223
2638548992315,5cyclictest2644191-21cron07:25:013
263854099237,7cyclictest131rcu_preempt08:20:201
263854899222,3cyclictest26515152cat07:45:193
2638548992218,2cyclictest0-21swapper/308:40:213
263854899221,7cyclictest637-21runrttasks09:45:133
2638548992216,4cyclictest2701238-21kworker/u8:1+events_unbound10:41:063
2638548992216,4cyclictest146-21jbd2/mmcblk1p2-811:21:053
2638548992215,3cyclictest637-21runrttasks10:50:013
2638548992215,3cyclictest0-21swapper/311:50:193
2638548992215,3cyclictest0-21swapper/309:15:013
2638548992213,5cyclictest0-21swapper/310:20:283
2638548992212,6cyclictest1-21systemd09:40:593
2638548992115,3cyclictest0-21swapper/311:50:013
2638548992113,6cyclictest391ktimers/312:30:013
2638548992112,7cyclictest2701998-21fschecks_time10:10:163
2638548992015,3cyclictest2638526-21cyclictest10:53:183
2638548992014,3cyclictest2665845-21nfs4_client08:25:213
2638548992012,4cyclictest0-21swapper/310:35:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional