You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-30 - 02:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Wed Oct 29, 2025 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3265723995425,25cyclictest0-21swapper/307:15:293
3265715995334,15cyclictest3281609-21munin-plugin-st07:55:011
3265723994918,26cyclictest0-21swapper/312:30:293
3265720994921,10cyclictest3274691-21taskset07:34:052
3265720994921,10cyclictest3274691-21taskset07:34:042
3265720994823,17cyclictest3267988-21memory07:15:042
3265715994735,8cyclictest3326593-21debian-sa110:05:011
3265715994724,19cyclictest0-21swapper/110:50:161
326572399463,12cyclictest3280268-21cut07:50:193
3265711994632,10cyclictest121ktimers/009:45:130
3265723994517,4cyclictest0-21swapper/312:00:273
3265720994536,6cyclictest3313157-21fschecks_time09:25:182
3265711994535,7cyclictest3270916-21unixbench_multi07:20:270
326572399442,16cyclictest3380106-21awk12:39:593
326572399442,16cyclictest3380106-21awk12:39:593
3265711994439,3cyclictest3290358-21sendmail08:20:010
326572399435,10cyclictest3310378-21grep09:15:303
3265723994316,5cyclictest0-21swapper/311:25:003
3265720994334,6cyclictest3348291-21unixbench_multi11:05:312
3265720994322,13cyclictest0-21swapper/207:40:152
3265715994323,17cyclictest0-21swapper/110:25:271
3265723994233,6cyclictest391ktimers/308:15:143
3265723994232,7cyclictest391ktimers/312:15:143
3265723994215,5cyclictest0-21swapper/310:30:003
3265720994236,3cyclictest3359664-21apt-get11:40:152
3265720994220,4cyclictest3286719-21apt-get08:10:042
326572399412,13cyclictest3371100-21grep12:10:273
3265723994114,3cyclictest0-21swapper/312:25:263
3265720994134,4cyclictest3359227-21idleruntime-cro11:40:002
3265720994125,11cyclictest3291127-21/usr/sbin/munin08:20:222
3265720994125,11cyclictest3291127-21/usr/sbin/munin08:20:212
326571599415,8cyclictest3337024-21date10:35:021
3265715994128,10cyclictest231ktimers/108:30:151
326571599411,9cyclictest3327739-21/usr/sbin/munin10:05:271
326571599411,9cyclictest3269900-21/usr/sbin/munin07:20:181
326571599411,7cyclictest3288232-21/usr/sbin/munin08:10:281
326571599411,7cyclictest3268941-21/usr/sbin/munin07:15:271
326571599411,10cyclictest3313542-21/usr/sbin/munin09:25:231
3265711994133,4cyclictest3274768-21cron07:35:000
3265711994133,4cyclictest3274768-21cron07:35:000
3265711994131,7cyclictest3323547-21fschecks_count09:55:170
3265711994131,6cyclictest3327897-21unixbench_singl10:05:280
3265720994034,3cyclictest3356031-21apt-get11:30:122
3265720994031,6cyclictest3271364-21cron07:25:002
3265720994030,7cyclictest3339082-21/usr/sbin/munin10:40:192
3265720994026,11cyclictest3301786-21/usr/sbin/munin08:50:272
3265720994025,12cyclictest3343874-21/usr/sbin/munin10:55:262
3265720994023,9cyclictest0-21swapper/207:25:132
326571599403,5cyclictest3349336-21irqstats11:10:181
3265715994029,6cyclictest3344874-21grep10:55:261
326571599402,11cyclictest3374582-21head12:20:341
326571599402,10cyclictest3274299-21unixbench_multi07:30:251
326571599402,10cyclictest3274299-21unixbench_multi07:30:251
326571599401,13cyclictest3357498-21debian-sa111:35:001
326571599401,12cyclictest3360710-21sed11:40:271
3265715994011,27cyclictest3303281-21wc08:55:251
326571599401,11cyclictest3316706-21diskstats09:35:201
326571599401,11cyclictest3310833-21sendmail09:20:001
3265723993930,5cyclictest3300577-21latency_hist08:50:013
326572399392,13cyclictest3338524-21apt10:40:013
326572099399,13cyclictest3307375-21seq09:09:522
3265720993931,5cyclictest0-21swapper/211:20:002
3265720993930,5cyclictest3330234-21apt-get10:15:142
3265720993929,6cyclictest3366486-21apt-get12:00:142
3265720993925,12cyclictest3298528-21cron08:41:002
3265720993923,10cyclictest3280657-21tr07:50:222
3265720993922,12cyclictest3281912-21/usr/sbin/munin07:55:162
3265720993920,10cyclictest0-21swapper/211:50:142
326571599397,7cyclictest3311467-21cut09:20:171
326571599392,7cyclictest3331952-21/usr/sbin/munin10:20:281
326571599392,6cyclictest3285370-21diskstats08:05:171
326571599392,6cyclictest3277336-21/usr/sbin/munin07:40:271
326571599392,5cyclictest3272235-21/usr/sbin/munin07:25:241
326571599391,9cyclictest3354118-21/usr/sbin/munin11:25:241
3265715993918,17cyclictest0-21swapper/111:20:261
326571599391,6cyclictest3307760-21/usr/sbin/munin09:10:171
326571599391,5cyclictest3376239-21grep12:25:261
3265715993915,21cyclictest0-21swapper/111:39:121
3265715993911,24cyclictest0-21swapper/108:35:271
326571599391,11cyclictest3372837-21grep12:15:271
326571599391,11cyclictest3276511-21awk07:40:011
326571599391,11cyclictest3276511-21awk07:40:001
326571599391,10cyclictest3337595-21tr10:35:231
3265711993930,6cyclictest3269167-21unixbench_multi07:15:280
3265711993928,6cyclictest0-21swapper/009:45:010
3265723993818,3cyclictest0-21swapper/310:40:163
3265720993832,3cyclictest3278429-21apt-get07:45:142
3265720993831,4cyclictest3340477-21apt-get10:45:142
3265720993828,7cyclictest3376297-21/usr/sbin/munin12:25:262
3265720993828,7cyclictest3325210-21/usr/sbin/munin10:00:152
3265720993828,7cyclictest3302847-21/usr/sbin/munin08:55:202
3265720993827,8cyclictest3365215-21memory11:55:202
3265720993827,8cyclictest3328647-21/usr/sbin/munin10:10:162
3265720993826,9cyclictest3317219-21/usr/sbin/munin09:35:282
3265720993826,8cyclictest3311546-21/usr/sbin/munin09:20:192
3265720993825,10cyclictest3373525-21apt-get12:20:142
3265720993825,10cyclictest3362733-21cron11:49:002
3265720993824,11cyclictest3327898-21/usr/sbin/munin10:05:282
3265720993821,12cyclictest3321467-21cron09:50:002
3265720993820,14cyclictest3338555-21awk10:40:002
3265720993820,13cyclictest3299283-21fschecks_count08:45:162
326571599387,8cyclictest3316153-21seq09:32:511
326571599381,8cyclictest3330231-21/usr/sbin/munin10:15:121
326571599381,8cyclictest3322724-21grep09:50:251
326571599381,8cyclictest3297565-21/usr/sbin/munin08:40:191
326571599381,8cyclictest3289615-21/usr/sbin/munin08:15:261
326571599381,7cyclictest3351026-21/usr/sbin/munin11:15:221
326571599381,7cyclictest3283760-21/usr/sbin/munin08:00:171
326571599381,6cyclictest3377372-21meminfo12:30:221
3265715993814,21cyclictest0-21swapper/109:40:251
3265715993813,21cyclictest0-21swapper/107:55:271
326571599380,7cyclictest3300268-21sshd08:46:291
3265720993731,3cyclictest3314881-21apt-get09:30:142
3265720993730,4cyclictest3371799-21apt-get12:15:142
3265720993730,3cyclictest3265874-21apt-get07:10:162
3265720993727,7cyclictest3323173-21auditctl09:55:012
3265720993726,8cyclictest3352866-21meminfo11:20:202
3265720993726,8cyclictest3285462-21fschecks_time08:05:172
3265720993725,7cyclictest3309908-21ntp_states09:15:252
3265720993724,8cyclictest3275123-21/usr/sbin/munin07:35:192
3265720993724,8cyclictest3275123-21/usr/sbin/munin07:35:182
3265720993723,8cyclictest0-21swapper/211:00:122
3265720993723,8cyclictest0-21swapper/211:00:122
3265720993722,9cyclictest3284143-21/usr/sbin/munin08:00:242
3265720993722,12cyclictest3368844-21/usr/sbin/munin12:05:202
3265720993719,15cyclictest3305291-21grep09:00:272
3265720993719,13cyclictest3378778-21fschecks_count12:35:182
3265720993719,13cyclictest3378778-21fschecks_count12:35:182
326571599373,5cyclictest3305920-21/usr/sbin/munin09:05:141
326571599372,4cyclictest3345593-21/usr/sbin/munin11:00:221
326571599372,4cyclictest3345593-21/usr/sbin/munin11:00:221
326571599371,7cyclictest3369089-21fgrep12:05:241
326571599371,6cyclictest3328526-21/usr/sbin/munin10:10:151
326571599371,5cyclictest3339322-21grep10:40:221
3265715993714,20cyclictest3379353-21mailstats12:35:251
3265715993714,20cyclictest3379353-21mailstats12:35:251
326571599371,13cyclictest3267330-21cron07:15:001
3265715993711,22cyclictest0-21swapper/107:50:001
326571599371,11cyclictest3340813-21irqstats10:45:181
3265711993730,4cyclictest3292045-21cron08:25:010
3265711993730,4cyclictest3292045-21cron08:25:000
3265711993729,4cyclictest3307401-21grep09:10:000
3265723993628,4cyclictest0-21swapper/311:35:193
3265723993628,4cyclictest0-21swapper/310:05:003
3265720993630,3cyclictest3331955-21apt-get10:20:142
3265720993630,3cyclictest3307635-21apt-get09:10:132
3265720993626,7cyclictest3342288-21df_inode10:50:152
3265720993626,7cyclictest3294328-21kernelversion08:30:202
3265720993625,8cyclictest3336763-21sshd10:34:292
3265720993625,7cyclictest3334218-21/usr/sbin/munin10:25:212
3265720993621,11cyclictest3355187-21head11:25:252
3265720993620,11cyclictest3297187-21awk08:40:012
326571599363,4cyclictest3361365-21/usr/sbin/munin11:45:301
326571599362,5cyclictest3290583-21/usr/sbin/munin08:20:171
326571599362,5cyclictest3290583-21/usr/sbin/munin08:20:171
326571599361,8cyclictest3364875-21/usr/sbin/munin11:55:151
326571599361,8cyclictest3364543-21grep11:55:011
326571599361,6cyclictest3370555-21taskset12:10:201
326571599361,5cyclictest3323405-21/usr/sbin/munin09:55:161
326571599361,10cyclictest3304852-21ps09:00:261
326571599360,9cyclictest3320006-21apt-get09:45:131
3265711993628,4cyclictest0-21swapper/009:00:000
3265711993626,6cyclictest0-21swapper/009:50:160
3265711993626,6cyclictest0-21swapper/009:35:300
3265711993625,6cyclictest0-21swapper/007:35:190
3265711993625,6cyclictest0-21swapper/007:35:190
3265720993528,4cyclictest3318308-21apt-get09:40:152
3265720993524,7cyclictest3370851-21/usr/sbin/munin12:10:262
3265720993522,10cyclictest3349604-21/usr/sbin/munin11:10:232
3265720993521,10cyclictest3323540-21/usr/sbin/munin09:55:162
3265720993519,13cyclictest3289775-21timerwakeupswit08:15:282
326571599351,6cyclictest3348561-21taskset11:07:401
326571599351,5cyclictest3300802-21/usr/sbin/munin08:50:201
3265711993528,4cyclictest3283291-21cron08:00:010
3265711993526,6cyclictest0-21swapper/011:15:300
3265711993526,4cyclictest0-21swapper/008:50:170
3265711993525,6cyclictest0-21swapper/011:10:250
3265711993522,7cyclictest0-21swapper/008:10:270
3265723993424,6cyclictest0-21swapper/311:00:003
3265723993424,5cyclictest3286310-21grep08:05:273
326572399342,11cyclictest3297184-21sendmail-msp08:40:013
3265720993422,9cyclictest3292399-21/usr/sbin/munin08:25:172
3265720993419,12cyclictest3377403-21missed_timers12:30:232
326571599341,7cyclictest3265784-21wc07:10:011
3265711993424,6cyclictest0-21swapper/009:00:270
3265711993424,6cyclictest0-21swapper/008:10:010
3265711993423,7cyclictest0-21swapper/012:10:270
3265711993423,7cyclictest0-21swapper/011:00:260
3265711993423,7cyclictest0-21swapper/011:00:260
3265723993327,3cyclictest3293757-21kthreadd08:30:003
3265723993325,4cyclictest0-21swapper/308:55:263
3265723993322,7cyclictest0-21swapper/307:15:003
326571199337,6cyclictest0-21swapper/011:40:220
3265711993323,6cyclictest0-21swapper/010:45:260
3265711993317,13cyclictest0-21swapper/008:35:000
3265723993226,3cyclictest0-21swapper/308:50:183
3265723993225,3cyclictest0-21swapper/309:45:013
3265711993224,4cyclictest0-21swapper/011:00:000
3265711993223,5cyclictest0-21swapper/011:40:000
3265723993123,4cyclictest0-21swapper/308:00:173
3265723993122,5cyclictest0-21swapper/310:15:273
326571199317,6cyclictest0-21swapper/009:10:180
3265711993121,6cyclictest0-21swapper/011:45:290
3265711993121,6cyclictest0-21swapper/010:25:000
3265723993022,4cyclictest0-21swapper/309:40:003
3265711993022,4cyclictest0-21swapper/009:30:240
3265711993021,5cyclictest0-21swapper/008:40:010
3265711993020,7cyclictest3379026-21latency12:35:200
3265711993020,7cyclictest3379026-21latency12:35:200
3265711993020,6cyclictest0-21swapper/010:10:170
3265711993019,4cyclictest131rcu_preempt10:00:540
326572399292,18cyclictest215-21dbus-daemon12:24:503
3265723992919,6cyclictest3365814-21grep11:55:263
3265723992917,8cyclictest3270895-21grep07:20:273
3265711992922,4cyclictest3314453-21idleruntime-cro09:30:010
3265711992922,4cyclictest3298883-21munin-plugin-st08:45:020
3265711992922,3cyclictest0-21swapper/009:20:270
3265711992919,6cyclictest0-21swapper/008:00:260
3265711992917,5cyclictest131rcu_preempt08:25:020
3265723992821,3cyclictest0-21swapper/311:45:273
3265723992818,6cyclictest0-21swapper/311:15:313
326571199288,6cyclictest0-21swapper/012:00:170
326571199287,4cyclictest0-21swapper/010:25:140
326571199285,8cyclictest0-21swapper/007:45:190
3265711992818,7cyclictest3365821-21unixbench_singl11:55:270
326572399273,17cyclictest3347063-21debian-sa111:05:013
326572399273,17cyclictest3347063-21debian-sa111:05:003
3265723992720,4cyclictest0-21swapper/311:10:233
3265723992719,4cyclictest0-21swapper/307:30:003
3265723992717,5cyclictest0-21swapper/310:45:203
326571199277,6cyclictest0-21swapper/007:10:160
326571199276,6cyclictest0-21swapper/011:25:190
326571199272,18cyclictest11-21ksoftirqd/007:50:220
3265711992719,5cyclictest0-21swapper/010:15:260
3265711992718,6cyclictest0-21swapper/010:40:010
3265723992620,3cyclictest0-21swapper/308:35:003
3265723992617,5cyclictest3305228-21grep09:00:273
3265723992616,6cyclictest3283296-21apt08:00:013
326571199268,5cyclictest0-21swapper/012:20:290
326571199268,5cyclictest0-21swapper/011:20:230
326571199265,5cyclictest0-21swapper/008:45:160
326571199262,14cyclictest11-21ksoftirqd/012:25:460
3265711992618,5cyclictest0-21swapper/011:05:300
3265711992617,5cyclictest0-21swapper/010:50:230
3265723992516,6cyclictest0-21swapper/309:30:013
3265723992516,5cyclictest440-21cron10:25:003
3265723992515,7cyclictest3324404-21grep09:55:273
3265723992515,6cyclictest0-21swapper/310:35:003
3265723992515,5cyclictest3287322-21fschecks_count08:10:183
3265723992514,7cyclictest0-21swapper/311:30:203
326571199258,7cyclictest131rcu_preempt10:31:210
326571199257,11cyclictest131rcu_preempt11:51:350
326571199256,12cyclictest0-21swapper/007:44:070
326571199255,5cyclictest0-21swapper/007:25:230
326571199255,11cyclictest0-21swapper/010:43:180
326571199254,6cyclictest0-21swapper/012:15:110
326571199252,7cyclictest0-21swapper/012:30:230
3265711992519,3cyclictest0-21swapper/009:15:200
326571199251,16cyclictest0-21swapper/012:05:200
326572399243,14cyclictest3321464-21cron09:50:013
3265723992418,3cyclictest0-21swapper/309:50:233
3265723992414,6cyclictest0-21swapper/311:50:263
326571199249,4cyclictest131rcu_preempt11:32:200
3265723992317,3cyclictest0-21swapper/311:25:263
3265723992315,5cyclictest0-21swapper/312:05:273
3265723992315,4cyclictest0-21swapper/310:50:253
3265723992314,5cyclictest3274289-21grep07:30:263
3265723992314,5cyclictest3274289-21grep07:30:253
3265723992313,5cyclictest3361122-21cron11:45:003
326572399222,14cyclictest3276943-21fschecks_time07:40:203
326572399222,13cyclictest3329521-21grep10:10:283
3265723992216,3cyclictest0-21swapper/309:35:003
3265723992213,6cyclictest0-21swapper/309:25:003
3265723992115,3cyclictest0-21swapper/309:05:093
3265723992114,5cyclictest391ktimers/308:40:203
3265723992111,6cyclictest0-21swapper/307:45:273
326572399211,15cyclictest3275149-21grep07:35:193
326572399211,15cyclictest3275149-21grep07:35:193
326572399203,10cyclictest3308653-21tail09:10:273
326572399192,12cyclictest3327849-21unixbench_multi10:05:283
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional