You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 22:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot5s.osadl.org (updated Sat Nov 22, 2025 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
402032995847,7cyclictest433047-21grep08:35:412
402023995142,6cyclictest442381-21grep09:05:160
402032994735,8cyclictest488727-21sendmail11:20:002
402023994432,8cyclictest447141-21sendmail09:20:000
402036994331,8cyclictest0-21swapper/312:40:013
402032994330,9cyclictest311ktimers/208:20:142
402023994331,6cyclictest0-21swapper/011:00:010
402036994229,8cyclictest0-21swapper/309:00:003
402023994231,7cyclictest463770-21unixbench_singl10:05:270
402036994132,6cyclictest508749-21unixbench_singl12:15:283
402023994133,4cyclictest435198-21munin-plugin-st08:45:010
402023994133,4cyclictest435198-21munin-plugin-st08:45:010
402032994033,3cyclictest407927-21apt-get07:25:132
402032994031,6cyclictest0-21swapper/209:35:002
402032994031,6cyclictest0-21swapper/209:34:592
402036993929,6cyclictest403655-21auditctl07:15:003
402036993928,7cyclictest503636-21grep12:00:253
402036993926,8cyclictest507473-21apt12:15:013
402032993932,4cyclictest428594-21apt-get08:25:142
402023993927,5cyclictest0-21swapper/010:20:000
402036993824,9cyclictest489295-21/usr/sbin/munin11:20:193
402036993812,11cyclictest405409-21/usr/sbin/munin07:15:263
402036993810,13cyclictest0-21swapper/309:32:233
402036993810,13cyclictest0-21swapper/309:32:233
402032993832,3cyclictest413044-21apt-get07:40:152
402032993831,4cyclictest488954-21apt-get11:20:122
402032993831,4cyclictest445656-21apt-get09:15:162
402032993828,7cyclictest454198-21apt-get09:40:142
402032993828,7cyclictest454198-21apt-get09:40:142
402032993824,10cyclictest491099-21/usr/sbin/munin11:25:192
402032993817,12cyclictest0-21swapper/212:35:132
402027993830,4cyclictest0-21swapper/108:55:001
402036993730,4cyclictest423025-21auditctl08:10:003
402036993726,7cyclictest457655-21/usr/sbin/munin09:50:163
402036993725,8cyclictest409621-21/usr/sbin/munin07:30:133
402036993723,10cyclictest430967-21/usr/sbin/munin08:30:233
402036993723,10cyclictest391ktimers/308:40:143
402036993723,10cyclictest391ktimers/308:40:143
402032993731,3cyclictest442245-21apt-get09:05:122
402032993730,4cyclictest497489-21apt-get11:45:172
402032993724,10cyclictest425989-21/usr/sbin/munin08:15:252
402032993723,11cyclictest494782-21/usr/sbin/munin11:35:242
402032993723,11cyclictest493032-21/usr/sbin/munin11:30:242
402032993723,11cyclictest459356-21/usr/sbin/munin09:55:162
402036993630,3cyclictest432001-21apt-get08:35:153
402036993629,4cyclictest426433-21latency_hist08:20:013
402036993624,9cyclictest493483-21sshd11:31:003
402036993624,7cyclictest488823-21cat11:20:003
402036993622,10cyclictest391ktimers/310:15:143
40203299366,22cyclictest0-21swapper/211:13:032
402032993630,3cyclictest507709-21apt-get12:15:142
402032993630,3cyclictest457601-21apt-get09:50:152
402032993629,4cyclictest476589-21apt-get10:45:152
402032993625,8cyclictest411429-21df07:35:162
402032993624,9cyclictest509557-21/usr/sbin/munin12:20:162
402032993624,7cyclictest0-21swapper/211:05:012
402032993623,10cyclictest449835-21/usr/sbin/munin09:25:312
402032993623,10cyclictest449835-21/usr/sbin/munin09:25:312
402032993623,10cyclictest439261-21/usr/sbin/munin08:55:192
402032993623,10cyclictest431204-21/usr/sbin/munin08:30:262
402032993623,10cyclictest410909-21cron07:33:012
402032993622,11cyclictest478532-21/usr/sbin/munin10:50:172
402032993622,11cyclictest462836-21df_inode10:05:162
402032993622,11cyclictest402633-21/usr/sbin/munin07:10:212
402032993621,11cyclictest405371-21/usr/sbin/munin07:15:252
402032993620,10cyclictest471580-21diskstats10:30:172
402032993619,12cyclictest437644-21grep08:50:212
402032993618,14cyclictest500169-21grep11:50:272
402032993618,14cyclictest500169-21grep11:50:262
402027993627,6cyclictest0-21swapper/112:25:011
402027993625,6cyclictest0-21swapper/110:05:261
402023993625,6cyclictest0-21swapper/011:10:250
402036993525,6cyclictest474353-21sshd10:36:433
402036993525,6cyclictest474353-21sshd10:36:433
402036993522,9cyclictest419854-21/usr/sbin/munin08:00:263
402036993522,10cyclictest415418-21/usr/sbin/munin07:45:283
402036993521,11cyclictest441293-21/usr/sbin/munin09:00:253
402032993529,3cyclictest506008-21apt-get12:10:142
402032993525,7cyclictest475455-21ntp_states10:40:232
402032993524,8cyclictest502975-21/usr/sbin/munin12:00:172
402032993524,7cyclictest512834-21apt-get12:30:152
402032993523,9cyclictest505217-21/usr/sbin/munin12:05:252
402032993522,9cyclictest435865-21meminfo08:45:202
402032993522,10cyclictest461584-21ntp_states10:00:222
402032993519,13cyclictest415269-21/usr/sbin/munin07:45:252
402027993526,4cyclictest0-21swapper/107:55:181
402027993525,6cyclictest0-21swapper/111:40:011
402023993529,3cyclictest0-21swapper/012:20:250
402036993427,3cyclictest0-21swapper/308:45:363
402036993426,4cyclictest0-21swapper/311:35:253
402036993425,6cyclictest485082-21sh11:10:003
402036993424,7cyclictest498327-21/usr/sbin/munin11:45:303
402036993423,8cyclictest464872-21/usr/sbin/munin10:10:213
402036993423,8cyclictest453769-21cron09:37:593
402036993423,8cyclictest453769-21cron09:37:593
402036993423,7cyclictest479638-21sshd10:53:433
402036993423,7cyclictest438472-21sshd08:53:413
402036993420,11cyclictest482223-21/usr/sbin/munin11:00:213
402032993428,3cyclictest440545-21apt-get09:00:142
402032993428,3cyclictest416443-21apt-get07:50:162
402032993428,3cyclictest416443-21apt-get07:50:162
402032993427,3cyclictest452480-21apt-get09:35:172
402032993427,3cyclictest452480-21apt-get09:35:172
402032993425,7cyclictest464729-21/usr/sbin/munin10:10:192
402032993424,7cyclictest511505-21/usr/sbin/munin12:25:192
402032993424,7cyclictest484243-21/usr/sbin/munin11:05:272
402032993424,7cyclictest474540-21cron10:39:002
402032993424,7cyclictest474540-21cron10:38:592
402032993424,7cyclictest468758-21/usr/sbin/munin10:20:262
402032993424,7cyclictest444513-21ntp_states09:10:212
402032993424,7cyclictest434120-21/usr/sbin/munin08:40:202
402032993424,7cyclictest434120-21/usr/sbin/munin08:40:202
402032993422,9cyclictest419975-21/usr/sbin/munin08:00:162
402032993422,9cyclictest418136-21/usr/sbin/munin07:55:132
402032993422,8cyclictest456012-21df09:45:192
402032993419,10cyclictest407677-21cron07:25:002
402027993423,5cyclictest0-21swapper/108:35:191
402023993424,5cyclictest0-21swapper/010:45:510
40203699336,4cyclictest4249102grep08:15:013
402036993324,6cyclictest450107-21/usr/sbin/munin09:25:343
402036993324,6cyclictest450107-21/usr/sbin/munin09:25:333
402036993323,7cyclictest510617-21sshd12:21:423
402036993323,7cyclictest468916-21/usr/sbin/munin10:20:263
402036993323,7cyclictest455492-21cron09:43:003
402036993323,7cyclictest455492-21cron09:43:003
402036993322,8cyclictest481103-21sshd10:55:553
402036993322,7cyclictest447895-21/usr/sbin/munin09:20:203
402036993321,9cyclictest407484-21sshd07:22:433
402036993321,8cyclictest427673-21grep08:20:273
402036993320,8cyclictest464186-21cron10:10:013
402036993320,8cyclictest444082-21/usr/sbin/munin09:10:163
402036993320,10cyclictest456245-21/usr/sbin/munin09:45:233
402036993320,10cyclictest416439-21munin-node07:50:163
402036993320,10cyclictest416439-21munin-node07:50:163
402036993316,11cyclictest513854-21tail12:30:273
402032993323,7cyclictest481062-21users10:55:262
402032993323,7cyclictest424320-21/usr/sbin/munin08:10:442
402032993322,7cyclictest421672-21df_inode08:05:172
402032993321,9cyclictest501220-21/usr/sbin/munin11:55:192
402032993321,9cyclictest501220-21/usr/sbin/munin11:55:192
402032993320,9cyclictest495863-21/usr/sbin/munin11:40:142
402032993317,12cyclictest469704-21fschecks_count10:25:142
402027993325,4cyclictest0-21swapper/109:25:181
402027993325,4cyclictest0-21swapper/109:25:181
402027993324,4cyclictest0-21swapper/112:35:171
402027993323,6cyclictest0-21swapper/111:35:001
402027993322,6cyclictest0-21swapper/111:50:261
402027993322,6cyclictest0-21swapper/111:50:261
402036993222,7cyclictest476686-21/usr/sbin/munin10:45:163
402036993222,7cyclictest446745-21sshd09:15:393
402036993222,7cyclictest413844-21/usr/sbin/munin07:40:263
402036993222,6cyclictest475220-21irqstats10:40:183
402036993221,8cyclictest499907-21/usr/sbin/munin11:50:263
402036993221,8cyclictest499907-21/usr/sbin/munin11:50:263
402036993221,8cyclictest461326-21/usr/sbin/munin10:00:183
402036993217,12cyclictest471581-21apt-get10:30:183
402036993216,13cyclictest442326-21/usr/sbin/munin09:05:143
402036993216,12cyclictest486522-21sed11:10:253
402032993222,7cyclictest448136-21/usr/sbin/munin09:20:242
402032993219,9cyclictest466337-21cut10:15:182
402027993224,4cyclictest0-21swapper/110:10:361
402027993224,4cyclictest0-21swapper/110:00:271
402027993224,3cyclictest0-21swapper/111:20:171
402027993223,5cyclictest0-21swapper/110:25:151
402027993223,4cyclictest0-21swapper/107:35:001
402023993225,4cyclictest424906-21latency_hist08:15:020
402023993224,4cyclictest431303-21unixbench_multi08:30:260
402036993122,4cyclictest0-21swapper/312:05:173
402036993121,7cyclictest512481-21sshd12:28:413
402036993121,7cyclictest501192-21/usr/sbin/munin11:55:193
402036993121,7cyclictest501192-21/usr/sbin/munin11:55:183
402036993121,7cyclictest490661-21apt-get11:25:143
402036993121,7cyclictest428591-21munin-node08:25:123
402036993121,7cyclictest418718-21ntp_states07:55:243
402036993121,7cyclictest411627-21if_eth007:35:183
402036993117,9cyclictest408919-21tail07:25:263
402036993117,10cyclictest471010-21cron10:30:003
402027993123,3cyclictest0-21swapper/108:00:181
402027993122,6cyclictest487370-21fschecks_count11:15:181
402027993122,5cyclictest0-21swapper/110:00:011
402027993121,7cyclictest467614-21idleruntime-cro10:20:001
402023993123,4cyclictest0-21swapper/008:35:420
402023993122,5cyclictest0-21swapper/008:50:010
402023993120,7cyclictest502740-21grep12:00:160
40202399311,9cyclictest474624-21cron10:40:000
40202399311,9cyclictest474624-21cron10:39:590
402036993021,6cyclictest459741-21/usr/sbin/munin09:55:203
40202799308,4cyclictest0-21swapper/112:05:161
402027993022,5cyclictest423036-21idleruntime-cro08:10:011
402027993022,5cyclictest0-21swapper/111:05:191
402027993022,5cyclictest0-21swapper/107:40:281
402027993022,3cyclictest0-21swapper/107:20:011
402027993019,7cyclictest425950-21grep08:15:251
402027993018,3cyclictest131rcu_preempt09:21:461
402023993020,6cyclictest0-21swapper/012:25:160
402023993020,6cyclictest0-21swapper/011:35:250
402023993020,6cyclictest0-21swapper/007:25:270
402036992916,10cyclictest496018-21/usr/sbin/munin11:40:173
40202799297,5cyclictest24-21ksoftirqd/107:10:141
40202799297,5cyclictest0-21swapper/108:40:171
40202799297,5cyclictest0-21swapper/108:40:171
402027992923,3cyclictest0-21swapper/108:30:001
402027992922,3cyclictest0-21swapper/109:15:281
402027992921,5cyclictest0-21swapper/108:35:011
402023992922,4cyclictest465480-21grep10:10:370
402023992921,5cyclictest486620-21kworker/u8:3+events_unbound11:21:010
40202799286,7cyclictest0-21swapper/111:25:211
402027992821,4cyclictest0-21swapper/111:00:261
402027992821,4cyclictest0-21swapper/108:55:261
402027992821,3cyclictest0-21swapper/112:20:001
402027992820,4cyclictest0-21swapper/109:50:281
402027992818,7cyclictest231ktimers/110:24:511
402027992818,6cyclictest504218-21grep12:05:011
402027992817,5cyclictest131rcu_preempt07:36:201
40202399281,9cyclictest469549-21apt-get10:25:120
402023992819,5cyclictest488708-21sh11:20:010
402023992817,7cyclictest215-21dbus-daemon07:45:180
402027992720,5cyclictest402014-21cyclictest08:46:441
402027992720,4cyclictest0-21swapper/107:30:001
402027992719,6cyclictest231ktimers/112:12:161
402027992719,5cyclictest450776-21apt-get09:30:121
402027992719,5cyclictest450776-21apt-get09:30:121
402027992718,6cyclictest476347-21sh10:45:001
402027992718,5cyclictest0-21swapper/108:20:171
402027992717,6cyclictest0-21swapper/110:50:001
402027992714,4cyclictest131rcu_preempt07:20:041
402027992710,10cyclictest131rcu_preempt10:54:231
40202399272,6cyclictest453950-21munin-run09:40:000
40202399272,6cyclictest453950-21munin-run09:40:000
402023992721,4cyclictest498735-21kworker/u8:0+events_unbound11:51:040
402023992721,4cyclictest498735-21kworker/u8:0+events_unbound11:51:040
402023992720,3cyclictest0-21swapper/012:35:270
402023992718,4cyclictest0-21swapper/008:20:270
402023992717,5cyclictest439854-21grep08:55:260
40202399271,23cyclictest476160-21chrt10:42:440
40202799262,16cyclictest24-21ksoftirqd/107:45:121
402027992619,4cyclictest485528-21apt-get11:10:131
402027992619,3cyclictest0-21swapper/109:05:291
402027992615,9cyclictest231ktimers/110:35:281
402027992615,9cyclictest231ktimers/110:35:281
402027992614,3cyclictest131rcu_preempt09:00:251
402027992613,3cyclictest131rcu_preempt09:36:261
402027992613,3cyclictest131rcu_preempt09:36:261
402027992611,8cyclictest497024-21kworker/u8:1-events_freezable_power_11:43:021
402023992619,3cyclictest0-21swapper/007:10:270
402023992618,5cyclictest0-21swapper/008:00:000
402023992618,4cyclictest0-21swapper/009:10:160
402023992616,7cyclictest417914-21basename07:55:010
402023992616,7cyclictest417914-21basename07:55:010
40202799257,5cyclictest24-21ksoftirqd/111:58:341
40202799257,5cyclictest24-21ksoftirqd/111:58:331
40202799257,13cyclictest471676-21fschecks_time10:30:191
402027992518,4cyclictest512172-21unixbench_singl12:25:261
402027992518,3cyclictest402014-21cyclictest12:32:581
402027992516,6cyclictest231ktimers/109:49:551
402027992514,8cyclictest454647-21memory09:40:221
402027992514,8cyclictest454647-21memory09:40:211
402027992514,2cyclictest415862-21kworker/u8:2-events_freezable_power_07:53:581
402027992514,2cyclictest415862-21kworker/u8:2-events_freezable_power_07:53:571
402027992512,10cyclictest497828-21/usr/sbin/munin11:45:241
40202799251,17cyclictest0-21swapper/109:14:391
402027992511,10cyclictest481537-21wc11:00:001
40202399252,19cyclictest485086-21apt-get11:09:590
402023992517,5cyclictest492124-21sh11:30:010
402023992517,4cyclictest13950irq/36-mmc109:55:000
402023992516,5cyclictest0-21swapper/012:13:080
402023992515,6cyclictest0-21swapper/009:02:160
40202399251,21cyclictest4107832sleep007:31:280
40202399251,20cyclictest502382-21sendmail-msp11:59:590
40202399251,20cyclictest502382-21sendmail-msp11:59:590
40202399251,20cyclictest0-21swapper/007:23:280
402023992417,4cyclictest13950irq/36-mmc109:30:010
402023992417,4cyclictest13950irq/36-mmc109:30:010
40202399241,19cyclictest0-21swapper/010:04:240
40202399241,19cyclictest0-21swapper/007:19:530
402027992313,8cyclictest423451-21cut08:10:181
402023992317,2cyclictest0-21swapper/011:31:000
402023992317,2cyclictest0-21swapper/010:50:230
40202399231,5cyclictest448384-21grep09:20:250
402023992314,6cyclictest0-21swapper/012:05:170
40202399231,3cyclictest0-21swapper/011:00:130
40202399231,18cyclictest0-21swapper/008:09:070
40202399230,19cyclictest0-21swapper/011:46:410
402023992216,2cyclictest0-21swapper/009:44:280
402023992216,2cyclictest0-21swapper/009:44:280
40202399221,5cyclictest469306-21cron10:25:000
402023992215,4cyclictest0-21swapper/010:30:320
402023992215,3cyclictest0-21swapper/012:15:220
40202399220,18cyclictest0-21swapper/012:31:540
40202399220,18cyclictest0-21swapper/009:59:350
40202399220,18cyclictest0-21swapper/008:51:150
40202399211,18cyclictest0-21swapper/007:38:440
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional