You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 13:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot5s.osadl.org (updated Fri Aug 07, 2026 00:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
662437995543,5cyclictest678200-21kworker/u8:1+events_unbound20:47:222
662436995137,10cyclictest231ktimers/119:20:141
662437995043,4cyclictest752772-21idleruntime-cro23:30:002
662437995043,4cyclictest752772-21idleruntime-cro23:30:002
662437994737,4cyclictest755863-21kworker/u8:1+events_unbound23:57:522
662436994738,5cyclictest775152-21auditctl00:35:001
662437994534,7cyclictest739277-21fschecks_count22:50:172
662435994537,5cyclictest731546-21kworker/u8:2+events_unbound22:42:360
662435994435,6cyclictest740976-21fschecks_count22:55:160
662437994333,7cyclictest380-21ntpd23:20:002
662436994331,7cyclictest0-21swapper/119:55:161
662436994234,5cyclictest728473-21sendmail22:20:011
662438994133,4cyclictest0-21swapper/321:00:003
662438994111,11cyclictest773607-21sed00:30:003
662438994111,11cyclictest773607-21sed00:30:003
662437994131,4cyclictest658608-21kworker/u8:0+events_unbound19:22:142
662437994130,8cyclictest735503-21sendmail22:40:002
662438994016,5cyclictest732664-21ntp_states22:30:223
662435994026,11cyclictest121ktimers/019:20:130
662438993929,5cyclictest0-21swapper/300:39:593
662437993929,7cyclictest756157-21cron23:39:592
662437993927,7cyclictest0-21swapper/223:40:262
662437993926,9cyclictest311ktimers/221:40:132
662436993927,8cyclictest685387-21cut20:15:011
662435993933,4cyclictest655131-21kworker/u8:2+events_unbound19:12:140
662435993931,4cyclictest725054-21idleruntime-cro22:10:000
66243899389,10cyclictest667210-21ntp_states19:20:223
662435993828,5cyclictest0-21swapper/000:40:000
662438993730,4cyclictest735489-21sh22:39:593
662438993730,4cyclictest0-21swapper/322:10:163
662438993711,10cyclictest718219-21cat21:50:013
662437993730,4cyclictest680261-21munin-run20:00:012
662437993729,4cyclictest0-21swapper/221:10:002
662435993731,4cyclictest744180-21kworker/u8:2+events_unbound23:12:420
66243899368,10cyclictest682094-21sed20:05:003
662438993623,10cyclictest391ktimers/319:55:143
662437993626,7cyclictest311ktimers/220:20:122
662436993627,6cyclictest754465-21auditctl23:35:001
66243599369,8cyclictest11-21ksoftirqd/023:55:000
662435993627,4cyclictest0-21swapper/022:05:010
662435993617,10cyclictest131rcu_preempt21:40:010
662435993616,11cyclictest131rcu_preempt19:40:250
662438993527,5cyclictest662433-21cyclictest21:25:003
662438993527,4cyclictest0-21swapper/321:55:173
662438993520,11cyclictest760379-21/usr/sbin/munin23:50:243
662438993517,10cyclictest0-21swapper/320:50:103
662437993528,4cyclictest718018-21latency_hist21:50:012
662437993525,6cyclictest0-21swapper/200:35:272
662436993522,9cyclictest231ktimers/121:45:141
662435993529,3cyclictest759588-21sh23:50:010
662435993525,6cyclictest693483-21unixbench_singl20:35:290
66243799346,4cyclictest131rcu_preempt21:20:182
662436993426,4cyclictest0-21swapper/119:10:251
662436993424,5cyclictest0-21swapper/120:05:181
662435993427,4cyclictest707782-21sendmail21:20:010
662435993427,4cyclictest707782-21sendmail21:20:000
662435993426,4cyclictest0-21swapper/023:55:260
662435993418,5cyclictest131rcu_preempt23:00:240
662435993417,13cyclictest0-21swapper/020:45:010
662435993413,12cyclictest131rcu_preempt21:40:250
662435993411,14cyclictest11-21ksoftirqd/023:35:240
662438993324,5cyclictest769844-21idleruntime-cro00:20:013
662438993324,5cyclictest0-21swapper/321:00:163
662437993326,4cyclictest0-21swapper/200:20:172
662437993319,7cyclictest131rcu_preempt00:30:212
662436993325,5cyclictest714168-21unixbench_singl21:35:251
662436993325,4cyclictest0-21swapper/123:00:261
662436993325,4cyclictest0-21swapper/119:20:011
662436993324,4cyclictest0-21swapper/121:50:281
662436993320,9cyclictest745468-21grep23:05:271
662435993326,5cyclictest667733-21kworker/u8:1+events_unbound19:37:150
662435993326,5cyclictest667733-21kworker/u8:1+events_unbound19:37:150
662435993318,9cyclictest121ktimers/021:05:120
66243899329,6cyclictest40-21ksoftirqd/319:40:013
662438993223,6cyclictest686628-21unixbench_multi20:15:263
662438993223,6cyclictest686628-21unixbench_multi20:15:263
662438993216,3cyclictest687893-21ntp_states20:20:213
662438993215,13cyclictest0-21swapper/323:20:143
662437993225,4cyclictest0-21swapper/221:10:152
662437993225,3cyclictest0-21swapper/223:50:292
662437993224,4cyclictest0-21swapper/222:17:342
662437993223,6cyclictest695624-21sh20:45:002
662437993222,6cyclictest0-21swapper/219:40:012
662437993222,6cyclictest0-21swapper/219:40:002
662437993220,5cyclictest721088-21kworker/u8:2+events_unbound22:10:062
662436993221,8cyclictest720940-21grep21:55:261
662436993221,7cyclictest0-21swapper/121:20:311
66243599329,13cyclictest11-21ksoftirqd/022:15:170
662435993226,4cyclictest707621-21kworker/u8:0+events_unbound21:22:250
662435993224,6cyclictest736977-21kworker/u8:0+events_unbound22:52:400
662435993224,6cyclictest692140-21kworker/u8:2+events_unbound20:52:230
662435993221,7cyclictest733600-21latency_hist22:35:000
662435993213,5cyclictest131rcu_preempt21:30:000
662435993213,12cyclictest712826-21kworker/u8:1-events_freezable_power_21:49:180
662435993212,8cyclictest11-21ksoftirqd/000:20:140
662435993212,12cyclictest131rcu_preempt23:05:190
662435993212,12cyclictest11-21ksoftirqd/022:25:010
662435993212,12cyclictest11-21ksoftirqd/000:15:010
662435993210,12cyclictest11-21ksoftirqd/000:30:210
662438993122,5cyclictest0-21swapper/320:45:273
662438993121,7cyclictest753903-21/usr/sbin/munin23:30:243
662438993120,8cyclictest663154-21grep19:10:213
662438993118,9cyclictest0-21swapper/320:25:143
662438993117,11cyclictest766777-21/usr/sbin/munin00:10:153
662438993117,11cyclictest711762-21/usr/sbin/munin21:30:193
662438993112,15cyclictest742764-21/usr/sbin/munin23:00:253
662437993117,8cyclictest678200-21kworker/u8:1+events_unbound20:12:202
662437993115,6cyclictest767745-21kworker/u8:2+events_unbound00:13:292
662436993126,3cyclictest715669-21switchtime21:40:251
662436993124,4cyclictest0-21swapper/122:30:251
66243599319,3cyclictest0-21swapper/022:45:590
662435993125,3cyclictest0-21swapper/020:00:280
662435993121,6cyclictest0-21swapper/020:25:240
662435993118,5cyclictest131rcu_preempt00:25:160
662435993118,5cyclictest131rcu_preempt00:25:160
662435993111,12cyclictest131rcu_preempt23:25:170
662435993111,12cyclictest131rcu_preempt23:25:170
66243899308,6cyclictest40-21ksoftirqd/322:55:013
662438993023,4cyclictest763243-21apt-get00:00:123
662438993022,4cyclictest0-21swapper/300:25:013
662438993021,4cyclictest0-21swapper/321:35:163
662438993021,4cyclictest0-21swapper/321:30:003
662438993020,7cyclictest729042-21/usr/sbin/munin22:20:183
662438993019,8cyclictest706292-21apt-get21:15:123
662438993019,8cyclictest706292-21apt-get21:15:113
662438993018,9cyclictest746940-21/usr/sbin/munin23:10:233
662438993017,10cyclictest715474-21/usr/sbin/munin21:40:233
662438993017,10cyclictest692553-21/usr/sbin/munin20:35:173
662437993023,5cyclictest700385-21kworker/u8:0+events_unbound20:57:252
662437993022,4cyclictest0-21swapper/219:30:172
662437993022,4cyclictest0-21swapper/219:30:162
662437993021,6cyclictest0-21swapper/220:15:252
662437993021,6cyclictest0-21swapper/220:15:252
662436993023,4cyclictest752766-21kthreadd23:29:591
662436993023,4cyclictest752766-21kthreadd23:29:591
662436993023,4cyclictest0-21swapper/122:25:171
662436993023,3cyclictest0-21swapper/100:05:261
662436993022,5cyclictest771542-21latency_hist00:25:001
662436993022,4cyclictest0-21swapper/120:00:171
662436993022,4cyclictest0-21swapper/100:00:001
662436993021,4cyclictest0-21swapper/100:35:171
66243599309,11cyclictest0-21swapper/020:21:160
66243599308,12cyclictest11-21ksoftirqd/023:20:250
662435993024,4cyclictest721440-21sendmail22:00:010
662435993024,4cyclictest0-21swapper/023:30:110
662435993019,8cyclictest151rcuc/022:25:140
662435993015,4cyclictest131rcu_preempt00:15:250
662435993014,3cyclictest131rcu_preempt21:53:130
66243599301,3cyclictest0-21swapper/020:45:140
662435993013,5cyclictest131rcu_preempt00:10:010
662435993012,5cyclictest131rcu_preempt21:30:170
662435993011,11cyclictest11-21ksoftirqd/021:00:010
662438992920,6cyclictest764946-21apt-get00:05:123
662438992920,6cyclictest0-21swapper/319:15:273
662438992919,7cyclictest739091-21latency_hist22:50:013
662438992919,7cyclictest719622-21cron21:54:003
662438992919,6cyclictest0-21swapper/323:10:003
662438992918,8cyclictest677647-21/usr/sbin/munin19:50:253
662438992917,9cyclictest762361-21/usr/sbin/munin23:55:263
662438992916,9cyclictest773683-21apt-get00:30:133
662438992916,10cyclictest669098-21/usr/sbin/munin19:25:263
662438992914,11cyclictest671751-21/usr/sbin/munin19:35:203
662437992923,3cyclictest0-21swapper/222:00:002
662437992922,4cyclictest0-21swapper/223:05:172
662437992921,4cyclictest0-21swapper/223:30:272
662437992921,4cyclictest0-21swapper/223:20:272
662437992920,6cyclictest0-21swapper/222:35:012
662437992917,6cyclictest721088-21kworker/u8:2+events_unbound22:05:192
662437992915,8cyclictest131rcu_preempt20:25:132
662436992921,4cyclictest0-21swapper/122:45:181
66243599299,11cyclictest11-21ksoftirqd/023:45:010
662435992913,5cyclictest131rcu_preempt19:55:020
662435992910,12cyclictest11-21ksoftirqd/022:10:170
662438992822,3cyclictest756418-21apt-get23:40:123
662438992818,7cyclictest749134-21sshd23:17:403
662438992818,7cyclictest737197-21sh22:44:593
662438992818,7cyclictest705770-21sshd21:11:403
662438992818,7cyclictest691067-21irqstats20:30:173
662438992818,7cyclictest673473-21apt-get19:40:113
662438992816,9cyclictest703554-21processes21:05:223
662438992816,9cyclictest682452-21/usr/sbin/munin20:05:173
662438992815,10cyclictest751429-21fschecks_count23:25:163
662438992815,10cyclictest751429-21fschecks_count23:25:153
66243799286,15cyclictest746848-21gzip23:10:222
662437992822,4cyclictest709391-21kworker/u8:1+events_unbound21:27:262
662437992821,5cyclictest740415-21kworker/u8:1+events_unbound22:55:232
662437992821,4cyclictest0-21swapper/221:15:262
662437992821,4cyclictest0-21swapper/221:15:262
662437992819,5cyclictest0-21swapper/200:25:272
662437992819,5cyclictest0-21swapper/200:25:272
662437992818,4cyclictest678200-21kworker/u8:1+events_unbound20:05:202
662437992815,4cyclictest731546-21kworker/u8:2+events_unbound22:28:502
662436992819,5cyclictest0-21swapper/119:45:011
66243599289,11cyclictest11-21ksoftirqd/020:30:240
662435992821,3cyclictest0-21swapper/020:00:000
662435992820,4cyclictest0-21swapper/020:15:000
66243599281,9cyclictest11-21ksoftirqd/021:00:120
662435992818,6cyclictest0-21swapper/021:15:010
66243599281,6cyclictest664400-21munin-node19:15:040
662435992815,5cyclictest131rcu_preempt19:30:210
662435992815,5cyclictest131rcu_preempt19:30:200
662435992810,6cyclictest11-21ksoftirqd/000:05:000
662438992717,7cyclictest758230-21/usr/sbin/munin23:45:153
662438992717,7cyclictest727381-21/usr/sbin/munin22:15:203
662438992717,7cyclictest676158-21/usr/sbin/munin19:45:253
662438992717,7cyclictest670145-21/usr/sbin/munin19:30:153
662438992717,7cyclictest670145-21/usr/sbin/munin19:30:153
662438992717,7cyclictest0-21swapper/322:25:133
662438992716,8cyclictest754748-21/usr/sbin/munin23:35:123
662438992715,9cyclictest722774-21cat22:00:253
662438992715,9cyclictest694147-21munin-node20:40:103
662438992710,13cyclictest684215-21if_eth120:10:163
662437992718,7cyclictest663483-21timerandwakeup19:10:252
662437992718,7cyclictest311ktimers/200:00:132
662437992714,7cyclictest131rcu_preempt21:35:002
662437992714,5cyclictest658608-21kworker/u8:0+events_unbound19:19:392
66243799271,3cyclictest131rcu_preempt20:50:112
662436992719,4cyclictest0-21swapper/121:20:001
662436992719,4cyclictest0-21swapper/121:20:001
66243599275,6cyclictest0-21swapper/020:15:170
66243599275,6cyclictest0-21swapper/020:15:170
662435992714,3cyclictest131rcu_preempt22:35:160
662435992711,6cyclictest11-21ksoftirqd/019:25:170
662435992710,10cyclictest0-21swapper/020:06:240
662438992612,12cyclictest724013-21ls22:05:183
662437992620,4cyclictest729818-21kworker/u8:0+events_unbound22:22:022
662437992619,5cyclictest757616-21kworker/u8:2+events_unbound23:45:422
662437992617,7cyclictest690274-21kworker/u8:0+events_unbound20:30:202
662437992616,6cyclictest0-21swapper/221:50:262
662437992613,7cyclictest131rcu_preempt20:40:002
662436992619,5cyclictest721088-21kworker/u8:2+events_unbound22:07:331
662436992619,5cyclictest676228-21unixbench_singl19:45:261
662436992618,5cyclictest0-21swapper/100:00:281
662436992617,5cyclictest0-21swapper/100:30:001
662436992617,5cyclictest0-21swapper/100:29:591
662436992616,6cyclictest0-21swapper/123:15:291
66243699261,13cyclictest693503-21head20:35:291
662435992614,9cyclictest131rcu_preempt19:45:250
662435992610,5cyclictest131rcu_preempt23:15:190
662437992519,4cyclictest731546-21kworker/u8:2+events_unbound22:46:522
662437992514,5cyclictest674724-21kworker/u8:2+events_unbound19:52:312
662437992511,8cyclictest131rcu_preempt19:45:002
662436992519,4cyclictest231ktimers/119:50:121
662436992518,3cyclictest0-21swapper/122:55:201
662436992517,4cyclictest0-21swapper/119:35:261
662436992517,4cyclictest0-21swapper/119:35:261
662436992513,8cyclictest0-21swapper/121:25:171
66243799243,3cyclictest131rcu_preempt19:45:122
662437992418,4cyclictest742244-21kworker/u8:0+events_unbound23:01:212
662437992417,4cyclictest714622-21sendmail-msp21:39:592
662437992417,3cyclictest0-21swapper/222:40:182
662437992411,4cyclictest721088-21kworker/u8:2+events_unbound22:02:352
662437992411,10cyclictest311ktimers/220:00:142
662436992419,3cyclictest694245-21cstates20:40:121
662436992417,3cyclictest0-21swapper/122:10:221
662436992415,6cyclictest750634-21grep23:20:271
662436992415,6cyclictest231ktimers/120:20:141
662436992414,7cyclictest231ktimers/123:50:121
66243799237,10cyclictest131rcu_preempt00:05:132
662437992317,3cyclictest0-21swapper/221:00:182
66243799231,6cyclictest0-21swapper/219:27:142
662436992318,2cyclictest0-21swapper/123:10:181
662436992317,3cyclictest0-21swapper/119:30:141
662436992317,3cyclictest0-21swapper/119:30:141
662436992316,4cyclictest768875-21ntpq00:15:221
662436992315,5cyclictest756175-21sendmail23:40:001
662436992315,4cyclictest736801-21grep22:40:281
662436992312,8cyclictest703088-21grep21:05:171
66243699226,12cyclictest722925-21grep22:00:251
662436992216,3cyclictest0-21swapper/120:50:111
662436992216,3cyclictest0-21swapper/120:45:171
662436992216,3cyclictest0-21swapper/119:25:261
66243799215,8cyclictest32-21ksoftirqd/200:20:002
66243699212,15cyclictest740597-21cron22:55:001
662436992115,4cyclictest145-21jbd2/mmcblk1p2-820:17:061
662436992115,4cyclictest145-21jbd2/mmcblk1p2-820:17:051
662436992115,3cyclictest0-21swapper/121:10:221
662436992114,4cyclictest0-21swapper/122:20:161
662436992113,5cyclictest0-21swapper/121:00:171
662436992015,3cyclictest758324-21fschecks_time23:45:151
662436992015,3cyclictest0-21swapper/123:40:201
662436992014,4cyclictest231ktimers/122:35:021
662436992013,5cyclictest231ktimers/100:10:121
662436992013,4cyclictest692301-21latency_hist20:35:011
662436992012,5cyclictest712404-21unixbench_multi21:30:261
66243699201,13cyclictest689574-21ntpq20:25:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional