You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-08 - 05:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Sat Aug 08, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
54992220100,100sleep20-21swapper/219:06:192
3858221897,95sleep10-21swapper/119:05:021
1360921620,2sleep334-21ksoftirqd/321:49:433
57822156101,28sleep00-21swapper/019:09:580
57012155102,27sleep30-21swapper/319:08:573
90592690,2sleep00-21swapper/022:58:140
66042670,2sleep10-21swapper/120:25:141
235412610,2sleep30-21swapper/319:50:133
135812610,2sleep334-21ksoftirqd/323:01:023
266772580,2sleep30-21swapper/300:24:173
314232570,2sleep30-21swapper/321:38:233
76922560,1sleep00-21swapper/023:45:220
29032540,3sleep122-21ksoftirqd/123:41:451
262622540,1sleep20-21swapper/223:10:242
190232540,2sleep20-21swapper/223:54:182
218702510,3sleep00-21swapper/023:31:410
218702510,3sleep00-21swapper/023:31:410
40402490,2sleep10-21swapper/100:30:411
68642470,1sleep00-21swapper/020:25:170
257802470,1sleep10-21swapper/123:59:231
140702450,2sleep00-21swapper/021:50:000
306622440,1sleep10-21swapper/123:14:491
306032400,2sleep20-21swapper/221:13:342
605899324,11cyclictest1715-21rm23:40:333
605899315,10cyclictest25415-21rm21:34:043
605899314,10cyclictest1623-21rm23:16:283
60372310,12sleep228-21ksoftirqd/222:55:232
605899304,10cyclictest16811-21rm23:52:033
605899303,11cyclictest20028-21ssh21:54:283
605899303,10cyclictest25316-21diskmemload22:10:223
605899295,9cyclictest27807-21sh21:10:453
605899294,9cyclictest30843-21ssh00:02:433
605899294,11cyclictest15769-21sh21:26:033
605899294,10cyclictest23623-21ssh21:56:213
605899292,10cyclictest27650-21ssh22:24:073
605899292,10cyclictest0-21swapper/322:37:013
605899291,10cyclictest930-21sh22:51:593
605899290,11cyclictest0-21swapper/322:29:423
55742290,2chrt0-21swapper/121:43:151
121562290,2sleep10-21swapper/123:49:081
605899285,9cyclictest27873-21sh23:11:593
605899285,20cyclictest186012ssh00:17:533
605899284,9cyclictest7240-21ssh22:32:463
605899282,9cyclictest24111-21ssh22:45:173
605899281,10cyclictest7209-21ntpq20:25:213
605699281,25cyclictest0-21swapper/119:15:461
605899274,9cyclictest25316-21diskmemload21:16:523
605899274,9cyclictest25316-21diskmemload21:16:523
605899271,24cyclictest0-21swapper/319:15:253
605599271,24cyclictest0-21swapper/019:16:020
605899266,18cyclictest21810-21grep23:55:243
605899264,8cyclictest31131-21rm22:02:093
605899264,8cyclictest31131-21rm22:02:093
605899264,6cyclictest19474-21sh22:17:403
605899263,8cyclictest8887-21rm23:45:513
605899263,8cyclictest18412-21ssh23:05:163
605799261,23cyclictest0-21swapper/219:17:552
605699261,17cyclictest25316-21diskmemload00:10:121
6055992610,10cyclictest7750-21apt-get00:10:110
88042250,4sleep28801-21ntp_states00:10:202
605899254,8cyclictest19607-21ssh22:41:343
605899254,8cyclictest11813-21sh21:23:473
605899254,6cyclictest14424-21sh00:39:273
605899253,8cyclictest7690-21ssh22:56:503
605899252,8cyclictest25316-21diskmemload00:25:103
605899251,9cyclictest8984-21ssh23:22:083
605899251,10cyclictest15261-21grep19:30:143
605799252,16cyclictest8882-21ssh22:10:112
605899243,8cyclictest23466-21ssh23:33:173
605899243,8cyclictest23466-21ssh23:33:173
605899243,6cyclictest0-21swapper/319:55:113
6057992422,1cyclictest0-21swapper/219:20:352
605799242,17cyclictest21646-21kworker/u8:000:00:132
6057992413,6cyclictest0-21swapper/221:45:122
6056992422,1cyclictest0-21swapper/119:21:071
6056992415,6cyclictest17896-21/usr/sbin/munin19:35:191
6055992415,6cyclictest278-21systemd-journal23:10:000
52712240,3sleep00-21swapper/000:31:570
231802240,3sleep0121ksoftirqd/021:55:540
140722240,1sleep00-21swapper/021:25:160
125992240,3sleep228-21ksoftirqd/223:25:132
605899233,9cyclictest3292-21sh00:05:543
605899233,6cyclictest11244-21ssh00:12:123
605899233,5cyclictest0-21swapper/321:40:483
605899233,5cyclictest0-21swapper/321:00:193
605899233,4cyclictest0-21swapper/320:40:163
605899232,6cyclictest6709-21diskmemload00:33:233
605899232,19cyclictest3464-21grep22:05:223
605899231,6cyclictest0-21swapper/323:35:183
6056992315,6cyclictest560-21dbus-daemon19:55:001
605699231,14cyclictest25316-21diskmemload23:54:501
605599233,15cyclictest8768-21taskset00:35:140
6055992315,5cyclictest9444-21latency_hist22:35:000
6055992310,9cyclictest32442-21apt-get21:15:120
6055992310,9cyclictest32442-21apt-get21:15:120
605899223,6cyclictest16354-21sh23:27:513
605899223,4cyclictest0-21swapper/319:35:233
6058992213,6cyclictest10945-21df_inode19:20:143
605799223,13cyclictest27719-21ssh22:00:122
605799223,13cyclictest27719-21ssh22:00:122
605799222,13cyclictest2037-21ssh22:05:032
6057992215,4cyclictest30037-21/usr/sbin/munin22:50:142
605799221,18cyclictest25316-21diskmemload21:19:582
605799221,18cyclictest25316-21diskmemload21:19:572
6057992210,8cyclictest32289-21apt-get23:40:122
605699222,6cyclictest25316-21diskmemload21:45:191
6056992215,5cyclictest15148-21ssh23:02:381
6056992212,4cyclictest22-21ksoftirqd/123:00:001
605699221,13cyclictest25316-21diskmemload22:08:081
605699221,13cyclictest25316-21diskmemload21:59:231
605599223,14cyclictest15564-21ssh22:15:130
6055992215,5cyclictest12-21ksoftirqd/023:55:010
605599221,2cyclictest141rcu_preempt22:55:000
272262220,7sleep012-21ksoftirqd/000:24:500
7382210,2sleep10-21swapper/122:27:571
71232210,1sleep10-21swapper/122:32:391
6058992115,2cyclictest19196-21apt-get19:40:103
6058992115,2cyclictest19196-21apt-get19:40:103
605799213,15cyclictest23220-21ls19:50:002
6057992115,3cyclictest26392-21cron23:59:592
6057992113,6cyclictest10617-21awk20:35:132
6057992113,5cyclictest7179-21sendmail_mailqu19:10:212
605699219,5cyclictest22-21ksoftirqd/123:20:471
605699214,11cyclictest22-21ksoftirqd/121:15:121
605699214,11cyclictest22-21ksoftirqd/121:15:121
605699212,6cyclictest14429-21apt-get00:15:121
6056992115,4cyclictest12819-21sh22:36:271
6056992113,6cyclictest5932-21seq20:24:011
6056992113,6cyclictest12353-21seq20:38:571
6056992112,7cyclictest30453-21diskstats20:05:151
605699211,14cyclictest515-21irqbalance23:15:101
605699211,13cyclictest28767-21/usr/sbin/munin22:00:211
605699211,13cyclictest28767-21/usr/sbin/munin22:00:211
605699211,13cyclictest12847-21sh23:25:151
605599217,5cyclictest21258-21grep21:30:210
605599216,7cyclictest6166-21sed23:20:160
605599212,8cyclictest5052-21kworker/0:123:15:340
605599211,9cyclictest26773-21grep23:35:210
605599211,6cyclictest9312-21ssh22:10:150
6055992115,4cyclictest12-21ksoftirqd/022:10:000
6055992111,7cyclictest515-21irqbalance00:04:200
274192210,2sleep20-21swapper/221:35:172
107992210,1sleep10-21swapper/100:35:471
605899209,9cyclictest278-21systemd-journal20:34:593
605899205,12cyclictest1951-21/usr/sbin/munin20:15:173
6058992013,5cyclictest18883-21idleruntime-cro20:55:003
605899200,7cyclictest27953-21/usr/sbin/munin20:00:233
605899200,6cyclictest15784-21ntpq20:45:203
605799208,10cyclictest0-21swapper/221:25:142
605799205,12cyclictest17236-21unixbench_singl22:15:262
605799205,11cyclictest8486-21latency_hist00:35:012
605799201,7cyclictest238482ssh00:21:242
6057992015,3cyclictest10667-21head22:35:152
6057992012,6cyclictest19329-21seq21:53:452
6057992012,6cyclictest14083-21sed19:25:242
6057992012,4cyclictest25316-21diskmemload23:20:192
6057992011,6cyclictest7211-21ntpq20:25:212
6057992011,6cyclictest24598-21cut22:45:222
605699204,14cyclictest29671-21ls22:50:001
605699201,8cyclictest25316-21diskmemload21:38:371
6056992018,1cyclictest0-21swapper/119:15:011
605699201,6cyclictest2104950ssh23:06:511
6056992014,4cyclictest22-21ksoftirqd/122:55:001
6056992013,4cyclictest23266-21sh00:20:491
6056992012,6cyclictest14794-21cut19:30:001
6056992011,6cyclictest23604-21cat21:05:131
6056992011,6cyclictest18945-21cut20:55:011
6056992011,6cyclictest16611-21awk22:15:221
6056992010,7cyclictest16940-21date19:34:591
605699200,6cyclictest25316-21diskmemload00:27:381
605599208,10cyclictest2781-21turbostat22:30:000
605599206,3cyclictest141rcu_preempt23:00:120
605599201,7cyclictest28181-21df_inode00:25:130
605599201,7cyclictest24444-21tr22:45:200
605599201,6cyclictest0-21swapper/023:10:350
605599201,5cyclictest0-21swapper/023:43:150
605599201,5cyclictest0-21swapper/022:00:020
605599201,5cyclictest0-21swapper/022:00:020
6055992013,5cyclictest15399-21ssh00:15:190
605599201,1cyclictest141rcu_preempt23:25:160
6055992011,7cyclictest4663-21sed20:20:160
6058991913,3cyclictest11736-21switchtime20:35:243
6058991912,5cyclictest30434-21cat20:05:143
6058991911,6cyclictest23872-21perl21:05:153
6058991910,6cyclictest744-21ntp_states20:10:203
605799198,6cyclictest53050irq/142-enp3s0-23:35:122
605799193,11cyclictest6775-21runrttasks21:20:122
605799192,12cyclictest25611-21apt-get19:55:122
605799192,12cyclictest1071-21apt-get00:05:112
605799192,10cyclictest16761-21ssh22:40:112
6057991912,5cyclictest20916-21sed00:20:002
6057991912,4cyclictest5799-21sh21:43:282
6057991911,6cyclictest3457-21expr22:30:132
6057991911,6cyclictest21643-21ls21:55:162
6057991911,5cyclictest5522-21sed20:20:252
6057991910,6cyclictest16335-21grep19:30:242
605699192,12cyclictest32251-21apt-get20:10:121
605699192,11cyclictest28350-21tune2fs20:00:141
605699191,6cyclictest26849-21rm00:00:121
605699191,3cyclictest141rcu_preempt21:20:201
6056991913,4cyclictest14911-21cpu20:45:131
6056991912,5cyclictest13664-21open_inodes20:40:211
6056991911,6cyclictest26611-21which19:55:211
6056991911,6cyclictest22592-21expr22:20:151
6056991911,6cyclictest14230-21ls21:25:181
605699191,10cyclictest0-21swapper/100:05:571
6056991910,7cyclictest25962-21df23:35:141
6056991910,6cyclictest19478-21entropy20:55:141
605699190,7cyclictest14823-21tune2fs21:50:151
605599198,8cyclictest560-21dbus-daemon21:15:000
605599192,6cyclictest17622-21memory22:40:180
605599191,8cyclictest21015-21cron21:00:010
605599191,6cyclictest0-21swapper/022:35:110
6055991912,4cyclictest12-21ksoftirqd/020:35:120
605599191,15cyclictest21171-21ssh23:55:190
6055991910,6cyclictest14187-21grep19:25:250
216582190,2chrt0-21swapper/221:30:242
605899188,8cyclictest14181-21grep19:25:253
605899187,8cyclictest22760-21tail19:45:263
605899183,3cyclictest34-21ksoftirqd/319:10:183
605899182,11cyclictest19138-21apt-get20:55:133
605899181,4cyclictest0-21swapper/320:20:213
605799188,7cyclictest23409-21/usr/sbin/munin21:05:162
605799187,9cyclictest0-21swapper/200:27:132
605799183,6cyclictest13023-21idleruntime-cro23:50:002
605799182,12cyclictest22345-21ssh22:20:132
605799182,11cyclictest28944-21apt-get22:25:112
605799181,8cyclictest21074-21cron19:45:002
605799181,8cyclictest21074-21cron19:45:002
6057991813,3cyclictest21028-21sendmail21:00:012
6057991812,4cyclictest25316-21diskmemload23:04:382
6057991811,5cyclictest24354-21ntp_kernel_err19:50:202
6057991811,5cyclictest22314-21which21:00:222
6057991811,5cyclictest19000-21open_inodes23:05:212
6057991810,6cyclictest9016-21cat20:30:172
6057991810,6cyclictest15859-21grep20:45:212
605699189,7cyclictest16892-21grep22:40:121
6056991813,3cyclictest28446-21sh21:11:231
6056991811,5cyclictest8845-21find22:10:101
6056991811,5cyclictest21591-21expr21:00:141
6056991811,5cyclictest21399-21grep21:30:221
6056991810,6cyclictest19781-21cat23:30:181
6056991810,6cyclictest19781-21cat23:30:181
6056991810,6cyclictest10330-21basename20:35:001
605599189,6cyclictest14021-21sed20:40:240
605599189,6cyclictest1041-21apt-get21:40:120
605599187,8cyclictest27485-21cron20:00:000
605599182,12cyclictest26789-21ssh21:35:130
605599182,11cyclictest2139-21ssh00:05:190
605599182,11cyclictest16999-21apt-get20:50:120
605599182,10cyclictest515-21irqbalance20:00:100
605599181,7cyclictest19318-21tr19:40:100
605599181,7cyclictest19318-21tr19:40:100
605599181,2cyclictest141rcu_preempt21:50:150
6055991812,4cyclictest12-21ksoftirqd/020:15:000
6055991811,5cyclictest25455-21cut19:55:010
6055991811,4cyclictest22780-21sh22:20:170
6055991810,6cyclictest7789-21ntpq21:20:200
6055991810,4cyclictest12-21ksoftirqd/019:20:110
226732180,1sleep10-21swapper/119:45:251
605799172,13cyclictest20528-21sed23:30:242
605799172,13cyclictest20528-21sed23:30:242
6057991714,2cyclictest30229-21cut20:05:142
6057991712,3cyclictest1951-21/usr/sbin/munin20:15:212
6057991712,3cyclictest12697-21/usr/sbin/munin20:40:212
605799171,13cyclictest31652-21diskstats23:15:142
6056991710,5cyclictest2250-21expr20:15:151
605599172,9cyclictest1976-21apt-get20:15:140
605599172,10cyclictest8423-21apt-get20:30:130
6055991712,3cyclictest636-21runrttasks19:11:580
605599171,14cyclictest16784-21cron19:35:000
605599171,11cyclictest25319-21munin-plugin-st21:10:000
6057991611,3cyclictest636-21runrttasks20:03:062
6057991610,4cyclictest17673-21cat19:35:172
605799160,13cyclictest9246-21grep00:35:172
605699169,5cyclictest19354-21ls19:40:101
605699169,5cyclictest19354-21ls19:40:101
605599169,5cyclictest22652-21cat21:00:250
605599168,6cyclictest17467-21cut19:35:150
6055991610,4cyclictest21448-21tr19:45:140
6055991610,3cyclictest15433-21hwlatdetect20:45:160
605799158,5cyclictest32229-21find20:10:112
605599152,10cyclictest30666-21cut20:05:160
605799148,4cyclictest17742-21sed20:50:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional