You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 07:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot5s.osadl.org (updated Fri Mar 13, 2026 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
358884999561,35cyclictest3629186-21latency_hist21:05:010
3588854995445,5cyclictest3656337-21munin-plugin-st22:25:001
3588854995143,5cyclictest3682537-21munin-plugin-st23:40:011
358884999511,41cyclictest0-21swapper/020:15:120
3588849994941,4cyclictest3675723-21sendmail23:20:000
358884999493,28cyclictest3593132-21cstates19:20:110
3588858994840,5cyclictest3613260-21munin-plugin-st20:20:002
3588849994721,22cyclictest0-21swapper/023:40:250
3588854994535,6cyclictest3685958-21basename23:50:011
3588854994533,7cyclictest0-21swapper/121:40:011
3588849994532,10cyclictest121ktimers/023:20:140
358884999451,35cyclictest0-21swapper/020:05:120
358884999441,39cyclictest3653184-21/usr/sbin/munin22:15:120
358884999441,38cyclictest3618540-21latency_hist20:35:020
358884999441,37cyclictest3651431-21apt-get22:10:120
3588863994335,5cyclictest3663119-21unixbench_multi22:40:233
358884999431,37cyclictest3634709-21if_eth121:20:180
358884999431,36cyclictest3656520-21sed22:25:010
358884999431,25cyclictest0-21swapper/021:35:270
3588858994230,9cyclictest311ktimers/221:50:132
3588854994233,6cyclictest3604291-21grep19:50:251
3588849994235,4cyclictest3606432-21latency_hist20:00:000
3588849994231,7cyclictest3663567-21audit_send_reply22:44:590
358884999421,38cyclictest3690795-21/usr/sbin/munin00:00:270
358884999421,37cyclictest3693790-21/usr/sbin/munin00:10:220
358884999421,36cyclictest3696772-21/usr/sbin/munin00:20:150
358884999421,36cyclictest0-21swapper/021:10:130
358884999421,36cyclictest0-21swapper/021:10:130
358884999421,35cyclictest121ktimers/019:10:120
3588854994131,8cyclictest231ktimers/122:35:121
3588854994131,6cyclictest208-21systemd-logind20:22:331
358884999411,37cyclictest3684590-21/usr/sbin/munin23:45:130
358884999411,37cyclictest3681729-21ps23:35:220
358884999411,37cyclictest3669903-21/usr/sbin/munin23:00:260
358884999411,37cyclictest3604747-21idleruntime-cro19:55:000
358884999411,35cyclictest3666051-21/usr/sbin/munin22:50:210
358884999411,35cyclictest3666051-21/usr/sbin/munin22:50:210
358884999411,35cyclictest3659233-21/usr/sbin/munin22:30:250
358884999411,35cyclictest3642857-21latency_hist21:45:020
358884999411,27cyclictest3648030-21apt22:00:110
358884999410,36cyclictest0-21swapper/023:30:110
358884999410,2cyclictest0-21swapper/020:55:130
3588863994033,4cyclictest3604740-21munin-plugin-st19:55:003
358884999401,36cyclictest3688039-21/usr/sbin/munin23:55:160
358884999401,36cyclictest3647159-21/usr/sbin/munin21:55:280
358884999401,36cyclictest3638470-21/usr/sbin/munin21:30:240
358884999401,36cyclictest3622333-21/usr/sbin/munin20:45:140
358884999401,36cyclictest3601675-21diskstats19:45:140
358884999401,34cyclictest3701740-21apt-get00:35:120
358884999401,34cyclictest3650172-21/usr/sbin/munin22:05:180
358884999401,34cyclictest3613503-21/usr/sbin/munin20:20:130
358884999401,34cyclictest3606678-21/usr/sbin/munin20:00:100
358884999401,34cyclictest0-21swapper/000:25:150
358884999401,2cyclictest0-21swapper/019:35:120
358884999401,2cyclictest0-21swapper/019:35:110
358884999401,24cyclictest3595485-21mailstats19:25:240
358884999400,2cyclictest0-21swapper/021:05:120
358884999400,2cyclictest0-21swapper/021:05:120
3588863993932,4cyclictest3606446-21idleruntime-cro20:00:003
3588863993931,4cyclictest0-21swapper/323:40:153
3588863993927,7cyclictest0-21swapper/322:20:263
3588858993932,4cyclictest3649513-21munin-plugin-st22:05:012
3588858993929,6cyclictest0-21swapper/221:05:272
3588858993929,6cyclictest0-21swapper/221:05:272
3588858993927,7cyclictest0-21swapper/200:20:002
3588858993926,10cyclictest311ktimers/200:30:122
3588854993933,4cyclictest3665494-21kworker/u8:3+events_unbound23:14:411
3588854993933,4cyclictest3665494-21kworker/u8:3+events_unbound23:14:411
3588854993932,4cyclictest3606453-21sendmail19:59:591
3588854993929,5cyclictest0-21swapper/121:15:001
3588854993929,5cyclictest0-21swapper/121:14:591
358884999392,34cyclictest3678494-21sed23:25:260
358884999392,34cyclictest3661406-21sshd22:37:230
3588849993914,22cyclictest121ktimers/019:40:120
358884999391,35cyclictest3667724-21/usr/sbin/munin22:55:210
358884999391,35cyclictest3658095-21sshd22:28:220
358884999391,35cyclictest3637076-21/usr/sbin/munin21:25:280
358884999391,35cyclictest3621124-21/usr/sbin/munin20:40:230
358884999391,35cyclictest3619283-21/usr/sbin/munin20:35:250
358884999391,35cyclictest3616187-21/usr/sbin/munin20:25:280
358884999391,34cyclictest3623930-21/usr/sbin/munin20:50:120
358884999391,24cyclictest3664249-21memory22:45:200
358884999391,24cyclictest3664249-21memory22:45:200
358884999390,36cyclictest3591434-21/usr/sbin/munin19:15:160
358884999390,2cyclictest0-21swapper/019:32:260
358884999390,2cyclictest0-21swapper/019:32:260
3588863993831,4cyclictest3677824-21fschecks_count23:25:173
3588863993829,6cyclictest438-21cron23:10:003
3588863993829,6cyclictest438-21cron23:10:003
3588863993829,5cyclictest3627110-21sendmail21:00:013
3588863993827,7cyclictest3592783-21sh19:20:013
3588858993827,7cyclictest3689353-21apt-get00:00:002
3588854993826,8cyclictest3683779-21grep23:40:261
358884999381,34cyclictest3671392-21/usr/sbin/munin23:05:210
358884999381,34cyclictest3671392-21/usr/sbin/munin23:05:210
358884999381,34cyclictest3645820-21cron21:52:010
358884999381,32cyclictest3686301-21/usr/sbin/munin23:50:160
358884999381,32cyclictest3632783-21df_inode21:15:130
358884999381,31cyclictest3610080-21apt-get20:10:120
358884999380,33cyclictest3672531-21/usr/sbin/munin23:10:100
358884999380,33cyclictest3672531-21/usr/sbin/munin23:10:090
358884999380,2cyclictest0-21swapper/000:15:160
3588858993729,4cyclictest3662667-21runrttasks22:40:212
3588858993724,7cyclictest0-21swapper/200:35:282
3588854993729,6cyclictest3694652-21kworker/u8:2+events_unbound00:38:111
358884999371,33cyclictest3700145-21/usr/sbin/munin00:30:140
358884999371,33cyclictest3691675-21/usr/sbin/munin00:05:170
358884999371,33cyclictest3643454-21/usr/sbin/munin21:45:200
3588863993629,4cyclictest3680820-21cron23:35:013
3588863993626,7cyclictest3682125-21grep23:35:253
3588863993626,6cyclictest0-21swapper/321:45:253
3588863993625,8cyclictest3601321-21sh19:44:593
3588858993629,4cyclictest3640965-21cron21:40:012
3588858993523,7cyclictest0-21swapper/222:10:262
3588854993528,5cyclictest3694652-21kworker/u8:2+events_unbound00:33:081
3588854993527,4cyclictest0-21swapper/120:50:161
3588854993526,4cyclictest0-21swapper/123:05:151
3588854993526,4cyclictest0-21swapper/123:05:151
3588854993525,5cyclictest0-21swapper/122:27:521
3588858993426,4cyclictest0-21swapper/221:05:002
3588858993423,6cyclictest0-21swapper/223:20:002
3588858993422,8cyclictest3692986-21idleruntime-cro00:10:002
3588854993430,2cyclictest3668495-21kworker/u8:0+events_unbound23:33:011
3588854993427,5cyclictest3614606-21kworker/u8:0+events_unbound21:17:441
3588854993423,8cyclictest231ktimers/119:45:111
3588863993325,5cyclictest391ktimers/320:50:133
3588863993325,4cyclictest0-21swapper/323:15:003
3588863993325,4cyclictest0-21swapper/323:15:003
3588863993324,6cyclictest3649496-21sh22:05:013
3588863993324,6cyclictest3635461-21grep21:20:273
3588863993324,4cyclictest0-21swapper/320:45:163
3588863993324,4cyclictest0-21swapper/319:37:233
3588863993324,4cyclictest0-21swapper/319:37:233
3588858993324,5cyclictest0-21swapper/223:40:012
3588858993324,5cyclictest0-21swapper/220:30:282
3588858993324,4cyclictest0-21swapper/219:15:022
3588858993323,6cyclictest0-21swapper/223:05:252
3588858993323,6cyclictest0-21swapper/223:05:242
3588858993322,6cyclictest0-21swapper/219:40:272
3588854993326,4cyclictest3639274-21munin-plugin-st21:35:011
3588854993321,8cyclictest3600893-21grep19:40:271
3588863993224,4cyclictest0-21swapper/321:50:273
3588858993223,5cyclictest0-21swapper/223:15:002
3588858993223,5cyclictest0-21swapper/223:15:002
3588858993222,6cyclictest3666546-21unixbench_singl22:50:262
3588858993222,6cyclictest3666546-21unixbench_singl22:50:262
3588858993222,6cyclictest0-21swapper/222:30:272
3588858993222,6cyclictest0-21swapper/220:55:262
3588858993221,6cyclictest0-21swapper/223:45:012
3588854993224,4cyclictest0-21swapper/123:50:251
3588854993224,4cyclictest0-21swapper/122:45:261
3588854993224,4cyclictest0-21swapper/122:45:261
3588854993223,4cyclictest0-21swapper/100:25:191
3588854993222,6cyclictest0-21swapper/121:55:291
3588854993222,6cyclictest0-21swapper/120:50:011
3588854993222,6cyclictest0-21swapper/119:25:261
3588854993222,5cyclictest0-21swapper/119:32:221
3588854993222,5cyclictest0-21swapper/119:32:221
3588854993221,6cyclictest0-21swapper/121:40:251
3588863993125,3cyclictest0-21swapper/323:25:003
3588863993122,5cyclictest0-21swapper/300:25:293
3588858993125,3cyclictest0-21swapper/220:20:162
3588858993121,6cyclictest3643968-21unixbench_singl21:45:262
3588858993121,6cyclictest0-21swapper/219:30:262
3588858993121,6cyclictest0-21swapper/219:30:252
3588854993123,4cyclictest0-21swapper/121:25:171
3588854993122,5cyclictest0-21swapper/100:10:011
3588863993024,4cyclictest3675532-21kworker/u8:1+events_unbound00:18:073
3588863993022,4cyclictest0-21swapper/322:25:173
3588863993022,4cyclictest0-21swapper/320:00:273
3588858993023,4cyclictest3690814-21unixbench_multi00:00:272
3588858993022,4cyclictest0-21swapper/222:20:262
3588858993022,4cyclictest0-21swapper/220:00:152
3588858993021,6cyclictest3678666-21grep23:25:272
3588858993020,6cyclictest0-21swapper/220:55:002
3588854993022,5cyclictest0-21swapper/122:55:001
3588854993022,5cyclictest0-21swapper/122:54:591
3588854993022,4cyclictest0-21swapper/121:05:161
3588854993022,4cyclictest0-21swapper/121:05:161
3588854993020,7cyclictest3592348-21grep19:15:261
3588854993017,10cyclictest231ktimers/120:10:121
3588863992921,4cyclictest0-21swapper/322:55:173
3588863992921,4cyclictest0-21swapper/322:55:003
3588863992921,4cyclictest0-21swapper/322:55:003
3588863992921,4cyclictest0-21swapper/322:15:243
3588863992921,4cyclictest0-21swapper/319:15:013
3588863992920,6cyclictest3613249-21cron20:20:003
3588863992920,6cyclictest0-21swapper/321:55:293
3588863992920,5cyclictest0-21swapper/320:45:003
3588863992918,7cyclictest3669912-21grep23:00:253
3588858992921,4cyclictest0-21swapper/223:35:012
3588858992920,4cyclictest0-21swapper/220:40:242
3588858992919,6cyclictest0-21swapper/200:20:242
3588854992923,4cyclictest3626946-21kworker/u8:3+events_unbound21:20:411
3588854992923,4cyclictest3584951-21kworker/u8:1+events_unbound19:11:361
3588854992921,4cyclictest0-21swapper/123:20:161
3588863992821,3cyclictest0-21swapper/322:45:163
3588863992821,3cyclictest0-21swapper/322:45:163
3588863992820,5cyclictest0-21swapper/320:35:283
3588863992820,4cyclictest3661154-21grep22:35:253
3588863992820,4cyclictest0-21swapper/319:25:263
3588863992819,4cyclictest0-21swapper/323:55:253
3588863992817,7cyclictest3602552-21grep19:45:243
3588863992817,7cyclictest0-21swapper/321:00:273
3588858992820,4cyclictest0-21swapper/221:20:272
3588854992822,3cyclictest0-21swapper/123:05:001
3588854992821,5cyclictest3592734-21kworker/u8:2+events_unbound20:17:341
3588854992820,5cyclictest221rcuc/122:15:121
3588854992820,4cyclictest0-21swapper/100:20:251
3588863992721,3cyclictest0-21swapper/300:10:213
3588863992719,4cyclictest0-21swapper/321:35:153
3588863992719,4cyclictest0-21swapper/320:25:013
3588863992715,9cyclictest3592734-21kworker/u8:2+events_unbound20:10:113
3588858992720,4cyclictest3623272-21unixbench_singl20:45:262
3588858992719,4cyclictest0-21swapper/200:10:162
3588858992717,6cyclictest3638862-21grep21:30:262
3588854992721,4cyclictest3592734-21kworker/u8:2+events_unbound20:07:311
3588854992720,5cyclictest3675532-21kworker/u8:1+events_unbound00:12:061
3588854992719,4cyclictest0-21swapper/120:30:271
3588854992716,8cyclictest3620086-21run-parts20:40:011
3588863992620,3cyclictest0-21swapper/300:40:003
3588854992618,5cyclictest3599449-21kworker/u8:1+events_unbound20:40:141
3588863992518,5cyclictest3649921-21fschecks_time22:05:153
3588863992517,6cyclictest3685841-21kworker/u8:0+events_unbound00:00:553
3588863992517,4cyclictest0-21swapper/320:05:263
3588863992517,4cyclictest0-21swapper/319:30:263
3588863992517,4cyclictest0-21swapper/319:30:263
3588863992513,8cyclictest3634163-21tr21:20:003
3588858992519,4cyclictest3650995-21kworker/u8:2+events_unbound22:17:522
3588858992519,3cyclictest0-21swapper/220:25:172
3588858992518,4cyclictest311ktimers/200:26:222
3588858992516,5cyclictest0-21swapper/223:05:002
3588858992516,5cyclictest0-21swapper/220:40:012
358885499258,10cyclictest24-21ksoftirqd/122:55:271
3588854992519,3cyclictest0-21swapper/100:05:001
3588854992518,5cyclictest3626946-21kworker/u8:3+events_unbound22:01:521
3588854992518,5cyclictest3626946-21kworker/u8:3+events_unbound21:45:201
3588854992518,4cyclictest3678675-21tail23:25:271
3588854992514,7cyclictest0-21swapper/100:00:011
358886399247,12cyclictest3630735-21apt21:10:003
358886399247,12cyclictest3630735-21apt21:09:593
3588863992416,5cyclictest0-21swapper/323:55:013
3588863992415,6cyclictest391ktimers/321:10:233
3588863992415,6cyclictest391ktimers/321:10:233
3588858992417,3cyclictest0-21swapper/219:15:012
3588858992416,5cyclictest0-21swapper/219:45:252
3588858992416,4cyclictest0-21swapper/219:35:242
3588858992416,4cyclictest0-21swapper/219:35:242
3588858992414,7cyclictest3664338-21ntpq22:45:212
3588858992414,7cyclictest3664338-21ntpq22:45:202
3588858992414,6cyclictest0-21swapper/221:15:142
3588858992413,7cyclictest3647794-21chrt22:00:002
3588854992416,5cyclictest3652892-21cron22:15:011
3588854992416,4cyclictest0-21swapper/120:55:251
3588854992411,4cyclictest3606891-21grep20:00:151
3588863992316,4cyclictest3692997-21idleruntime-cro00:10:003
3588863992315,5cyclictest3641372-21fschecks_count21:40:163
3588863992315,4cyclictest0-21swapper/300:23:063
3588858992317,3cyclictest0-21swapper/223:20:252
3588858992317,3cyclictest0-21swapper/222:35:172
3588858992317,3cyclictest0-21swapper/221:25:262
3588858992316,3cyclictest0-21swapper/222:25:152
3588858992315,5cyclictest311ktimers/221:40:202
358885499236,10cyclictest24-21ksoftirqd/120:25:231
3588854992317,4cyclictest3659049-21kworker/u8:0+events_unbound22:45:011
3588854992315,5cyclictest0-21swapper/119:35:171
3588854992315,5cyclictest0-21swapper/119:35:171
3588854992314,7cyclictest231ktimers/121:00:061
358886399226,12cyclictest3675703-21cron23:19:593
3588863992216,3cyclictest0-21swapper/322:10:253
3588863992214,5cyclictest0-21swapper/320:30:193
3588858992212,7cyclictest3685017-21ntpq23:45:192
3588854992216,4cyclictest3695266-21kernelversion00:15:191
358885499221,11cyclictest0-21swapper/123:15:431
358886399211,6cyclictest3700388-21/usr/sbin/munin00:30:193
3588863992115,4cyclictest391ktimers/321:30:253
3588863992115,3cyclictest0-21swapper/319:20:213
3588863992114,4cyclictest3637560-21sh21:30:003
3588858992114,3cyclictest0-21swapper/219:55:262
3588858992113,5cyclictest0-21swapper/220:05:172
3588858992113,4cyclictest0-21swapper/219:50:142
358885899211,17cyclictest3686184-21seq23:50:112
3588858992111,6cyclictest0-21swapper/222:05:152
3588854992116,3cyclictest0-21swapper/121:50:141
3588854992114,3cyclictest0-21swapper/122:05:241
3588863992013,4cyclictest391ktimers/320:25:133
3588858992014,3cyclictest3667362-21fschecks_count22:55:172
3588858992013,4cyclictest645-21runrttasks19:25:222
3588858992013,4cyclictest0-21swapper/220:10:202
3588858992012,6cyclictest3631508-21ntpq21:10:212
3588858992012,6cyclictest3631508-21ntpq21:10:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional