You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 04:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Mon Nov 24, 2025 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
221123995041,6cyclictest251065-21grep20:30:240
221128994834,9cyclictest325696-21idleruntime-cro00:10:001
221123994842,4cyclictest328957-21kworker/u8:2+events_unbound00:23:560
221128994741,4cyclictest325442-21kworker/u8:0+events_unbound00:13:421
221132994635,7cyclictest287478-21grep22:15:272
221137994413,10cyclictest301055-21mailstats22:55:253
221132994435,5cyclictest225075-21idleruntime-cro19:20:002
221137994317,9cyclictest278265-21if_eth121:50:173
221137994317,9cyclictest278265-21if_eth121:50:173
221123994337,4cyclictest229866-21kworker/u8:1+events_unbound20:06:370
221123994336,5cyclictest235235-21kworker/u8:0+events_unbound19:51:350
221123994333,6cyclictest230169-21cron19:35:000
22113799429,11cyclictest222026-21grep19:10:263
221137994213,12cyclictest296841-21/usr/sbin/munin22:45:143
221137994213,11cyclictest295923-21cat22:40:243
221132994233,6cyclictest230187-21munin-plugin-st19:35:012
221123994234,4cyclictest253427-21sendmail20:40:010
221123994230,9cyclictest121ktimers/000:05:160
221137994116,10cyclictest227738-21/usr/sbin/munin19:25:263
221137994111,11cyclictest241428-21/usr/sbin/munin20:05:183
221123994133,6cyclictest305106-21kworker/u8:0+events_unbound23:21:580
221137994015,7cyclictest327009-21users00:10:313
221137994011,9cyclictest288951-21/usr/sbin/munin22:20:253
221137994010,10cyclictest261470-21ntp_states21:00:233
221128994029,7cyclictest287524-21grep22:15:271
221123994034,4cyclictest282659-21kworker/u8:0+events_unbound22:11:500
221123994033,5cyclictest315348-21kworker/u8:1+events_unbound23:42:010
221123994021,11cyclictest131rcu_preempt22:05:010
221132993931,4cyclictest0-21swapper/220:50:022
221123993932,5cyclictest315348-21kworker/u8:1+events_unbound23:52:490
221123993921,9cyclictest131rcu_preempt21:20:100
221137993826,7cyclictest0-21swapper/300:20:253
221137993810,11cyclictest324294-21cat00:05:173
221132993827,7cyclictest0-21swapper/223:40:002
221123993829,7cyclictest256690-21kworker/u8:1+events_unbound21:06:430
221137993724,10cyclictest271478-21/usr/sbin/munin21:30:173
221137993720,3cyclictest334567-21diskstats00:35:163
221137993712,9cyclictest229427-21mailstats19:30:243
221128993731,4cyclictest256690-21kworker/u8:1+events_unbound21:01:431
221128993728,6cyclictest293517-21fschecks_time22:35:171
221123993731,4cyclictest309456-21kworker/u8:2+events_unbound23:31:570
221123993730,5cyclictest282659-21kworker/u8:0+events_unbound22:36:540
221123993722,9cyclictest131rcu_preempt21:10:180
221123993720,4cyclictest221118-21cyclictest20:55:330
221123993718,4cyclictest131rcu_preempt23:25:170
221123993718,4cyclictest131rcu_preempt20:02:180
221123993717,5cyclictest131rcu_preempt23:45:160
221123993717,5cyclictest131rcu_preempt23:45:160
221123993712,16cyclictest131rcu_preempt22:45:260
221137993625,7cyclictest286473-21munin-node22:15:113
221137993624,8cyclictest293069-21auditctl22:35:013
221137993623,9cyclictest322864-21/usr/sbin/munin00:00:173
221137993623,10cyclictest266736-21/usr/sbin/munin21:15:243
221137993614,5cyclictest319551-21memory23:50:223
221132993629,4cyclictest0-21swapper/200:20:162
221128993629,5cyclictest248058-21kworker/u8:0+events_unbound20:26:401
221128993623,9cyclictest440-21cron22:04:591
221128993623,9cyclictest231ktimers/122:30:111
221128993613,4cyclictest221118-21cyclictest23:05:131
221123993630,4cyclictest305106-21kworker/u8:0+events_unbound23:16:590
221123993621,4cyclictest131rcu_preempt22:52:260
221123993618,5cyclictest282659-21kworker/u8:0-events_freezable_power_22:21:490
221123993617,11cyclictest131rcu_preempt00:35:280
221123993617,10cyclictest131rcu_preempt21:29:140
221123993615,4cyclictest131rcu_preempt19:38:100
221123993613,5cyclictest131rcu_preempt21:45:010
221137993525,5cyclictest0-21swapper/322:50:213
221137993524,8cyclictest332526-21munin-plugin-st00:30:003
221137993524,7cyclictest302968-21/usr/sbin/munin23:00:263
221137993523,9cyclictest255012-21sshd20:41:153
221137993523,8cyclictest308645-21sendmail23:20:013
221137993522,10cyclictest251341-21sshd20:33:143
221128993528,4cyclictest225050-21cron19:20:011
221123993526,4cyclictest0-21swapper/019:11:280
221123993524,7cyclictest296078-21grep22:40:240
221123993524,7cyclictest0-21swapper/020:54:590
221123993519,5cyclictest131rcu_preempt21:02:590
221123993518,4cyclictest131rcu_preempt19:58:060
221123993517,9cyclictest131rcu_preempt22:25:410
221123993515,9cyclictest131rcu_preempt19:16:230
221123993515,5cyclictest131rcu_preempt21:55:100
221123993515,5cyclictest131rcu_preempt20:26:390
221123993515,12cyclictest131rcu_preempt23:00:250
221137993427,3cyclictest262603-21apt-get21:05:143
221137993427,3cyclictest232342-21apt-get19:40:143
221137993426,4cyclictest0-21swapper/323:39:593
221137993425,6cyclictest294547-21sshd22:37:233
221137993424,7cyclictest245345-21/usr/sbin/munin20:15:223
221137993423,7cyclictest276600-21/usr/sbin/munin21:45:183
221137993422,9cyclictest53-21kcompactd023:22:413
221137993422,9cyclictest329006-21cron00:19:013
221137993421,10cyclictest282101-21/usr/sbin/munin22:00:263
221137993420,10cyclictest283415-21/usr/sbin/munin22:05:173
221137993420,10cyclictest259296-21/usr/sbin/munin20:55:133
221137993418,5cyclictest312615-21irqstats23:30:183
221137993418,13cyclictest285381-21/usr/sbin/munin22:10:233
221137993418,12cyclictest258025-21/usr/sbin/munin20:50:213
221132993425,5cyclictest0-21swapper/220:00:012
221132993419,11cyclictest311ktimers/219:35:112
22112899347,5cyclictest131rcu_preempt22:05:101
221128993426,4cyclictest0-21swapper/100:35:161
221128993425,5cyclictest264066-21cron21:10:011
221128993425,5cyclictest0-21swapper/123:55:171
221128993425,4cyclictest0-21swapper/120:06:361
221123993420,10cyclictest131rcu_preempt21:37:090
221123993416,5cyclictest237123-21kworker/u8:2-events_freezable_power_20:14:120
221123993416,4cyclictest131rcu_preempt21:31:050
221123993416,4cyclictest131rcu_preempt19:44:130
221123993416,4cyclictest0-21swapper/021:48:460
221123993416,10cyclictest131rcu_preempt19:23:040
221123993414,4cyclictest0-21swapper/022:30:340
221123993413,4cyclictest131rcu_preempt00:25:330
221137993327,3cyclictest269436-21apt-get21:25:133
221137993325,4cyclictest0-21swapper/323:15:003
221137993324,5cyclictest239591-21fschecks_time20:00:163
221137993323,7cyclictest333329-21/usr/sbin/munin00:30:223
221137993323,7cyclictest235276-21sshd19:47:153
221137993321,9cyclictest224906-21sshd19:18:153
221137993320,10cyclictest231142-21/usr/sbin/munin19:35:193
221137993319,11cyclictest236818-21/usr/sbin/munin19:50:273
221137993317,11cyclictest310694-21/usr/sbin/munin23:25:163
221137993316,12cyclictest269193-21cron21:25:003
221132993324,6cyclictest248089-21sh20:25:012
221132993324,5cyclictest0-21swapper/221:40:252
221128993325,4cyclictest0-21swapper/121:45:251
221128993325,4cyclictest0-21swapper/120:40:011
221128993323,6cyclictest0-21swapper/120:05:011
22112899330,5cyclictest0-21swapper/122:45:121
221123993323,6cyclictest246379-21cron20:20:010
221123993319,4cyclictest131rcu_preempt23:37:430
221123993319,4cyclictest131rcu_preempt23:15:000
221123993317,5cyclictest131rcu_preempt22:05:220
221123993316,3cyclictest0-21swapper/023:07:360
221123993314,4cyclictest131rcu_preempt20:40:300
221123993311,4cyclictest131rcu_preempt00:03:150
221137993222,7cyclictest290354-21meminfo22:25:233
221137993222,7cyclictest275127-21ntp_states21:40:223
221137993222,7cyclictest264305-21/usr/sbin/munin21:10:273
221137993222,7cyclictest252062-21/usr/sbin/munin20:35:163
221137993222,7cyclictest246623-21munin-node20:20:143
221137993221,6cyclictest53-21kcompactd019:20:433
221137993220,9cyclictest318525-21cron23:46:003
221137993220,9cyclictest318525-21cron23:46:003
221137993219,10cyclictest303937-21/usr/sbin/munin23:05:153
221137993216,12cyclictest243740-21/usr/sbin/munin20:10:263
221137993214,5cyclictest255689-21diskstats20:45:163
221132993224,4cyclictest0-21swapper/221:10:252
221132993223,5cyclictest0-21swapper/220:40:172
221128993225,4cyclictest0-21swapper/120:45:261
221128993222,6cyclictest0-21swapper/123:54:591
221128993221,7cyclictest0-21swapper/120:20:001
22112899321,6cyclictest131rcu_preempt23:00:111
22112899320,6cyclictest0-21swapper/120:55:121
22112399329,11cyclictest0-21swapper/023:56:020
221123993221,7cyclictest121ktimers/020:50:010
221123993215,9cyclictest131rcu_preempt00:30:150
221123993215,10cyclictest131rcu_preempt19:45:250
221123993211,9cyclictest131rcu_preempt21:15:230
221123993211,12cyclictest11-21ksoftirqd/000:15:230
221137993121,7cyclictest322008-21cron23:57:003
221137993121,7cyclictest317076-21cron23:43:593
221137993118,10cyclictest273563-21/usr/sbin/munin21:35:253
221132993124,4cyclictest0-21swapper/222:45:012
221132993123,5cyclictest333816-21unixbench_singl00:30:252
22112899314,4cyclictest131rcu_preempt21:10:121
221128993124,5cyclictest315348-21kworker/u8:1+events_unbound23:47:381
221128993124,5cyclictest315348-21kworker/u8:1+events_unbound23:47:371
221128993123,4cyclictest0-21swapper/121:30:261
22112399319,14cyclictest11-21ksoftirqd/021:50:160
22112399319,14cyclictest11-21ksoftirqd/021:50:160
221123993121,6cyclictest0-21swapper/020:20:270
221123993113,4cyclictest131rcu_preempt22:58:130
221123993113,4cyclictest131rcu_preempt19:25:220
221137993019,7cyclictest237573-21/usr/sbin/munin19:55:163
221132993022,4cyclictest0-21swapper/223:50:272
22112899303,4cyclictest131rcu_preempt20:10:141
221128993023,5cyclictest328957-21kworker/u8:2+events_unbound00:20:231
221128993022,6cyclictest256690-21kworker/u8:1+events_unbound21:26:441
221128993022,5cyclictest275876-21kworker/u8:1+events_unbound22:15:001
22112899301,5cyclictest0-21swapper/122:55:131
221123993012,4cyclictest131rcu_preempt00:12:470
221137992920,6cyclictest280223-21ntp_states21:55:233
221132992923,3cyclictest251697-21apt-key20:35:012
221132992922,4cyclictest0-21swapper/200:15:302
221132992921,4cyclictest0-21swapper/222:00:172
221132992920,5cyclictest0-21swapper/200:10:202
221128992922,5cyclictest235235-21kworker/u8:0+events_unbound19:51:441
221128992921,5cyclictest306916-21cron23:15:011
221128992921,4cyclictest0-21swapper/122:20:171
221128992921,4cyclictest0-21swapper/119:55:171
221128992920,5cyclictest0-21swapper/119:20:281
221128992919,6cyclictest0-21swapper/121:55:271
221128992918,7cyclictest0-21swapper/120:50:161
22112899291,3cyclictest0-21swapper/123:30:131
221128992910,5cyclictest256690-21kworker/u8:1+events_unbound21:19:101
221123992913,3cyclictest131rcu_preempt22:16:220
221132992822,3cyclictest253409-21munin-plugin-st20:40:002
221132992820,6cyclictest221118-21cyclictest00:35:192
22112899288,12cyclictest0-21swapper/120:20:261
22112899282,4cyclictest131rcu_preempt21:20:141
221128992820,4cyclictest291377-21munin-plugin-st22:30:011
221128992820,4cyclictest0-21swapper/119:10:191
221128992813,5cyclictest250018-21kworker/u8:2+events_unbound20:30:281
22112899280,6cyclictest0-21swapper/119:40:151
221132992721,3cyclictest0-21swapper/223:40:152
221132992720,4cyclictest0-21swapper/222:55:272
221132992720,4cyclictest0-21swapper/221:30:242
221132992719,4cyclictest0-21swapper/221:10:002
221132992718,6cyclictest267524-21cat21:20:012
221132992717,5cyclictest274336-21sendmail-msp21:40:002
221128992720,4cyclictest0-21swapper/121:40:241
221128992711,6cyclictest131rcu_preempt00:30:181
22112899271,14cyclictest0-21swapper/122:53:141
22112899270,6cyclictest277956-21apt-get21:50:131
22112899270,6cyclictest277956-21apt-get21:50:131
221132992618,4cyclictest0-21swapper/222:10:012
221132992616,6cyclictest0-21swapper/220:15:182
221132992614,7cyclictest0-21swapper/221:25:272
221128992617,6cyclictest1-21systemd22:43:421
221128992616,7cyclictest312049-21idleruntime-cro23:30:001
221128992612,6cyclictest24-21ksoftirqd/100:15:251
22112899260,5cyclictest0-21swapper/123:15:121
221132992519,4cyclictest221118-21cyclictest00:10:012
221132992519,3cyclictest0-21swapper/221:20:172
221132992519,3cyclictest0-21swapper/220:10:002
221132992518,5cyclictest275876-21kworker/u8:1+events_unbound22:31:532
221132992518,3cyclictest0-21swapper/221:00:002
221132992518,3cyclictest0-21swapper/220:05:012
221132992517,4cyclictest0-21swapper/221:50:172
221132992517,4cyclictest0-21swapper/221:50:162
221132992516,4cyclictest0-21swapper/223:00:242
221132992514,8cyclictest440-21cron21:50:012
221128992514,8cyclictest382-21ntpd23:20:231
221128992512,11cyclictest24-21ksoftirqd/123:35:141
22112899251,14cyclictest0-21swapper/100:29:541
221132992418,3cyclictest0-21swapper/223:55:262
221132992417,3cyclictest0-21swapper/223:10:012
221132992416,4cyclictest0-21swapper/221:55:162
22112899249,3cyclictest0-21swapper/119:45:201
221128992418,4cyclictest267162-21kworker/u8:0+events_unbound21:35:171
221128992417,4cyclictest0-21swapper/123:40:171
221128992416,4cyclictest0-21swapper/119:36:331
221128992411,3cyclictest0-21swapper/119:25:231
22113299239,10cyclictest301rcuc/223:20:302
221132992317,3cyclictest0-21swapper/222:35:202
221132992316,4cyclictest0-21swapper/223:35:012
221132992315,4cyclictest0-21swapper/221:00:182
221132992314,6cyclictest0-21swapper/219:15:012
22112899239,3cyclictest0-21swapper/100:00:141
221132992215,4cyclictest318867-21cron23:49:592
221132992215,4cyclictest318867-21cron23:49:592
221132992215,4cyclictest307911-21sensors_temp23:15:232
221132992213,6cyclictest299328-21cat22:50:252
221132992212,7cyclictest331164-21df_inode00:25:162
221128992216,4cyclictest226732-21kworker/u8:0+events_unbound19:30:531
22113299211,5cyclictest310727-21fschecks_count23:25:172
22113299211,5cyclictest284923-21grep22:10:182
221132992115,4cyclictest288969-21swap22:20:252
221132992115,3cyclictest0-21swapper/219:50:262
221132992111,6cyclictest290963-21grep22:25:282
221132992110,8cyclictest257459-21gdbus20:50:032
221132992110,6cyclictest0-21swapper/200:00:152
221132992015,3cyclictest311ktimers/219:45:142
221132992015,3cyclictest248547-21fschecks_time20:25:172
22113299201,4cyclictest0-21swapper/222:45:142
221132992014,4cyclictest311ktimers/219:25:252
221132992014,4cyclictest233969-21latency_hist19:45:012
221132991914,2cyclictest0-21swapper/219:20:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional