You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-26 - 15:43
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot4.osadl.org (updated Tue Sep 26, 2023 12:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29996852270182,57sleep10-21swapper/107:08:591
29996752261218,28sleep00-21swapper/007:08:520
159199970,39rtkit-daemon0-21swapper/111:21:440
159199940,3rtkit-daemon0-21swapper/012:25:400
159199910,3rtkit-daemon0-21swapper/012:34:260
159199850,14rtkit-daemon1590-21rtkit-daemon08:07:430
159199770,4rtkit-daemon1590-21rtkit-daemon09:53:480
159199740,4rtkit-daemon1590-21rtkit-daemon08:01:301
159199670,3rtkit-daemon0-21swapper/111:04:581
159199670,3rtkit-daemon0-21swapper/011:44:191
159199670,3rtkit-daemon0-21swapper/010:04:261
30807462650,4chrt3080743-21munin-run09:55:011
159199650,3rtkit-daemon0-21swapper/010:46:431
159199640,3rtkit-daemon0-21swapper/009:02:061
159199630,3rtkit-daemon0-21swapper/012:12:141
159199630,3rtkit-daemon0-21swapper/009:58:021
159199620,4rtkit-daemon1590-21rtkit-daemon09:37:300
159199620,3rtkit-daemon0-21swapper/107:55:340
159199620,3rtkit-daemon0-21swapper/011:47:061
159199620,3rtkit-daemon0-21swapper/008:34:011
159199620,3rtkit-daemon0-21swapper/008:34:011
159199620,3rtkit-daemon0-21swapper/008:15:451
159199620,3rtkit-daemon0-21swapper/007:11:281
159199610,4rtkit-daemon0-21swapper/009:39:071
159199610,3rtkit-daemon0-21swapper/012:32:591
159199610,3rtkit-daemon0-21swapper/012:25:551
159199610,3rtkit-daemon0-21swapper/009:07:421
159199610,3rtkit-daemon0-21swapper/008:23:271
159199610,3rtkit-daemon0-21swapper/008:11:261
159199600,5rtkit-daemon0-21swapper/010:41:381
159199600,3rtkit-daemon0-21swapper/111:50:540
159199600,3rtkit-daemon0-21swapper/012:24:551
159199600,3rtkit-daemon0-21swapper/009:19:591
299992499598,16cyclictest3195315-21fschecks_time12:35:250
2999924995935,7cyclictest3028945-21/usr/sbin/munin08:20:360
159199590,4rtkit-daemon1590-21rtkit-daemon11:25:120
2999924995818,18cyclictest3130524-21latency_hist11:05:010
159199580,3rtkit-daemon0-21swapper/010:09:081
159199580,3rtkit-daemon0-21swapper/009:27:501
159199580,3rtkit-daemon0-21swapper/008:41:251
2999924995718,12cyclictest3099999-21needreboot10:20:310
159199570,3rtkit-daemon0-21swapper/011:24:251
159199570,3rtkit-daemon0-21swapper/011:12:281
159199570,3rtkit-daemon0-21swapper/009:46:311
159199570,3rtkit-daemon0-21swapper/009:12:071
159199560,3rtkit-daemon0-21swapper/010:57:371
159199560,3rtkit-daemon0-21swapper/010:10:061
159199560,3rtkit-daemon0-21swapper/008:29:221
2999924995527,10cyclictest3123578-21dump-pmu-power10:54:590
2999924995510,18cyclictest3131941-21switchtime11:05:360
2999932995413,3cyclictest0-21swapper/110:38:391
2999924995431,8cyclictest3063059-21/usr/sbin/munin09:30:210
2999924995431,8cyclictest3063059-21/usr/sbin/munin09:30:210
2999924995422,17cyclictest3156203-21kernelversion11:40:290
299992499542,11cyclictest3012172-21apt07:40:010
2999924995416,23cyclictest3186107-21sh12:22:040
159199540,4rtkit-daemon1590-21rtkit-daemon08:26:020
159199540,3rtkit-daemon0-21swapper/012:39:501
2999924995319,13cyclictest3046808-21fschecks_count09:05:250
159199530,3rtkit-daemon1590-21rtkit-daemon10:48:390
159199530,3rtkit-daemon1590-21rtkit-daemon10:28:511
2999924995227,9cyclictest3166249-21cpu11:55:210
2999924995227,9cyclictest3166249-21cpu11:55:210
2999924995226,9cyclictest3052968-21missed_timers09:15:300
2999924995225,8cyclictest3182650-21sh12:16:530
299992499518,4cyclictest3151802-21ls11:35:010
299992499515,12cyclictest3137940-21apt-get11:15:180
2999924995131,9cyclictest3085143-21/usr/sbin/munin10:00:380
2999924995126,9cyclictest3036870-21cpuspeed_turbos08:40:200
2999924995126,8cyclictest3072794-21sh09:42:560
2999924995125,11cyclictest3057069-21/usr/sbin/munin09:20:320
2999924995119,9cyclictest3109282-21sh10:34:340
2999924995118,11cyclictest3177096-21/usr/sbin/munin12:10:340
2999924995115,11cyclictest3028117-21timerandwakeup08:15:390
2999924995113,10cyclictest3110037-21fschecks_count10:35:220
159199510,4rtkit-daemon1590-21rtkit-daemon09:46:170
159199510,3rtkit-daemon1590-21rtkit-daemon07:11:400
159199510,3rtkit-daemon0-21swapper/110:19:500
30981392500,10sleep10-21swapper/110:18:061
299993299508,10cyclictest3141226-21basename11:19:591
2999932995010,6cyclictest3174108-21sh12:05:271
2999924995027,7cyclictest3124250-21gltestperf10:55:250
2999924995024,11cyclictest3082453-21/usr/sbin/munin09:55:380
2999924995023,13cyclictest3025126-21/usr/sbin/munin08:10:240
2999924995023,11cyclictest3089565-21/usr/sbin/munin10:05:360
2999924995023,10cyclictest3059945-21sh09:25:270
2999924995016,9cyclictest3051284-21sh09:13:320
2999924995011,9cyclictest3103626-21processes10:25:340
159199500,5rtkit-daemon3166551-21fschecks_time11:55:271
159199500,5rtkit-daemon3166551-21fschecks_time11:55:271
159199500,3rtkit-daemon0-21swapper/107:34:510
159199500,3rtkit-daemon0-21swapper/008:45:431
299993299490,2cyclictest141rcu_preempt09:30:241
299993299490,2cyclictest141rcu_preempt09:30:231
299992499493,6cyclictest3011654-21sshd11:10:110
2999924994924,9cyclictest3113498-21cstates10:40:200
2999924994921,8cyclictest3038781-21cpu08:45:240
2999924994921,10cyclictest3005719-21unixbench-2d07:20:350
2999924994917,10cyclictest3170934-21ntp_kernel_err12:00:350
159199490,4rtkit-daemon1590-21rtkit-daemon09:03:280
159199490,3rtkit-daemon0-21swapper/112:02:091
159199490,3rtkit-daemon0-21swapper/008:08:401
299993299488,3cyclictest141rcu_preempt07:35:001
299993299482,2cyclictest141rcu_preempt10:22:261
2999924994826,6cyclictest3002557-21/usr/sbin/munin07:15:230
2999924994822,10cyclictest3152941-21/usr/sbin/munin11:35:290
2999924994822,10cyclictest3035621-21pmu-power08:35:330
2999932994726,9cyclictest3120962-21ssh10:50:291
299993299472,2cyclictest141rcu_preempt09:40:251
2999932994719,7cyclictest3162889-21forks11:50:241
2999932994719,3cyclictest3004299-21cron07:20:011
2999932994714,3cyclictest0-21swapper/109:24:031
2999932994712,3cyclictest0-21swapper/112:18:411
299992499475,2cyclictest13-21ksoftirqd/007:25:320
2999924994720,11cyclictest3033251-21gltestperf08:30:300
2999924994720,11cyclictest3033251-21gltestperf08:30:290
2999924994718,7cyclictest3042979-21gltestperf08:55:270
2999924994717,14cyclictest3041029-21gltestperf08:50:250
2999924994714,11cyclictest3173781-21df_inode12:05:210
2999924994710,2cyclictest0-21swapper/011:45:220
159199470,6rtkit-daemon3015036-21diskstats07:45:240
159199470,6rtkit-daemon3015036-21diskstats07:45:230
159199470,1rtkit-daemon1590-21rtkit-daemon08:50:181
2999932994610,3cyclictest0-21swapper/110:30:261
159199460,3rtkit-daemon0-21swapper/107:46:551
159199460,3rtkit-daemon0-21swapper/107:46:541
2999932994523,2cyclictest3034589-21cron08:35:021
2999932994518,3cyclictest3152181-21cpuspeed_turbos11:35:191
299992499458,4cyclictest3092458-21fschecks_count10:10:250
159199450,3rtkit-daemon1590-21rtkit-daemon11:05:131
299993299442,3cyclictest0-21swapper/108:55:261
159199440,3rtkit-daemon0-21swapper/007:41:271
159199440,3rtkit-daemon0-21swapper/007:26:491
299993299432,2cyclictest0-21swapper/111:33:281
2999932994322,3cyclictest3010797-21cut07:35:271
2999932994313,3cyclictest3146271-21grep11:25:371
299992499430,6cyclictest13-21ksoftirqd/008:05:010
2999932994219,4cyclictest3004653-21grep07:20:211
30133122410,2sleep00-21swapper/007:40:260
299992499354,8cyclictest3016928-21cstates07:50:220
159199310,3rtkit-daemon0-21swapper/107:55:131
159199300,3rtkit-daemon0-21swapper/107:51:151
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional