You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-07 - 10:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Mon Sep 07, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
111538021280,4sleep20-21swapper/223:55:152
102177921100,4sleep13056-21PK-Backend19:30:011
111346921050,3sleep20-21swapper/223:50:132
101363827539,13sleep30-21swapper/319:08:503
101226227345,17sleep00-21swapper/019:05:130
101356326445,13sleep10-21swapper/119:07:521
101344726242,14sleep20-21swapper/219:06:192
10475702410,3sleep10-21swapper/120:40:211
10842042400,2sleep30-21swapper/322:25:213
10608422320,5chrt1060843-21cpuspeed_turbos21:20:121
1013876992913,4cyclictest42-21ksoftirqd/322:40:013
101387199283,4cyclictest26-21ksoftirqd/119:15:001
11052132270,8chrt13-21ksoftirqd/023:25:220
1013874992710,3cyclictest141rcu_preempt19:15:032
101387199268,4cyclictest26-21ksoftirqd/121:55:121
101387199267,3cyclictest26-21ksoftirqd/122:25:271
10254152250,5sleep31025224-21latency_hist19:40:013
1013871992519,3cyclictest1033892-21idleruntime-cro20:05:001
101387199251,3cyclictest26-21ksoftirqd/121:55:001
1013871992511,4cyclictest26-21ksoftirqd/121:47:331
1013876992413,9cyclictest978-21snmpd21:25:573
101387699240,22cyclictest1015304-21chrt19:12:143
101387499245,5cyclictest34-21ksoftirqd/222:10:012
101387499242,6cyclictest34-21ksoftirqd/221:40:012
1013874992418,4cyclictest1070120-21ntp_kernel_pll_21:45:192
1013874992415,7cyclictest1054995-21awk21:04:592
101387199249,3cyclictest26-21ksoftirqd/123:35:001
101387199243,6cyclictest26-21ksoftirqd/100:20:011
101387199242,10cyclictest26-21ksoftirqd/100:15:001
101387199240,2cyclictest141rcu_preempt00:00:021
101387199240,11cyclictest26-21ksoftirqd/121:20:001
101387699238,4cyclictest42-21ksoftirqd/320:35:123
1013876992315,6cyclictest883-21dbus-daemon22:15:003
1013876992315,6cyclictest883-21dbus-daemon22:14:593
101387499236,2cyclictest141rcu_preempt21:29:592
1013874992318,4cyclictest34-21ksoftirqd/222:50:012
101387199238,3cyclictest141rcu_preempt19:31:491
101387199234,10cyclictest26-21ksoftirqd/121:04:591
1013871992318,3cyclictest251rcuc/122:10:001
1013871992314,6cyclictest2176-21JS10:27:481
1013871992310,3cyclictest26-21ksoftirqd/121:25:131
101387699227,4cyclictest42-21ksoftirqd/320:10:003
101387499229,3cyclictest141rcu_preempt19:15:002
101387499223,5cyclictest34-21ksoftirqd/219:25:012
1013874992215,5cyclictest436-20systemd-journal20:05:002
101387199226,4cyclictest26-21ksoftirqd/122:00:161
1013871992220,1cyclictest0-21swapper/123:40:251
1013871992220,1cyclictest0-21swapper/120:31:391
101387199220,9cyclictest1069133-21munin-run21:45:011
101387199220,5cyclictest3056-21PK-Backend23:20:001
101387199220,21cyclictest3056-21PK-Backend00:39:271
101387199220,20cyclictest3056-21PK-Backend23:25:051
101387199220,20cyclictest3056-21PK-Backend20:23:021
101387199220,10cyclictest26-21ksoftirqd/121:30:231
101387199220,10cyclictest26-21ksoftirqd/100:30:001
101386999226,6cyclictest13-21ksoftirqd/020:10:000
1013869992215,6cyclictest13-21ksoftirqd/000:00:010
1013869992214,6cyclictest1084729-21cron22:29:590
101386999220,3cyclictest1070989-21cat21:50:010
101387699219,3cyclictest42-21ksoftirqd/322:20:013
101387699217,4cyclictest42-21ksoftirqd/319:15:223
101387699215,14cyclictest1050362-21expr20:50:133
1013876992116,4cyclictest42-21ksoftirqd/323:10:113
1013876992113,6cyclictest1-21systemd20:19:593
1013876992113,6cyclictest1125791-21df00:25:143
1013874992114,5cyclictest1019749-21apt-get19:25:022
101387499210,9cyclictest34-21ksoftirqd/220:10:002
101387499210,5cyclictest34-21ksoftirqd/221:55:132
101387499210,3cyclictest141rcu_preempt00:15:012
101387499210,2cyclictest0-21swapper/221:20:152
101387499210,20cyclictest0-21swapper/221:06:172
101387199214,5cyclictest26-21ksoftirqd/100:35:001
101387199214,2cyclictest141rcu_preempt23:00:201
101387199212,2cyclictest141rcu_preempt23:50:001
101387199212,2cyclictest141rcu_preempt20:40:001
1013871992119,1cyclictest3056-21PK-Backend20:52:431
1013871992119,1cyclictest0-21swapper/122:47:461
1013871992119,1cyclictest0-21swapper/119:42:111
1013871992118,2cyclictest3056-21PK-Backend22:54:081
1013871992118,2cyclictest3056-21PK-Backend21:39:051
1013871992118,2cyclictest3056-21PK-Backend19:55:371
1013871992116,3cyclictest11198052chrt00:08:171
1013871992116,3cyclictest10370462chrt20:11:221
1013871992115,4cyclictest11144652chrt23:51:461
1013871992110,4cyclictest26-21ksoftirqd/121:05:131
1013871992110,3cyclictest26-21ksoftirqd/123:38:291
101387199210,20cyclictest3056-21PK-Backend22:37:401
101387199210,20cyclictest3056-21PK-Backend22:18:121
101387199210,19cyclictest3056-21PK-Backend22:30:441
101387199210,19cyclictest3056-21PK-Backend20:25:111
101387199210,10cyclictest26-21ksoftirqd/121:10:121
1013869992110,3cyclictest1072955-21cpuspeed_turbos21:55:120
101386999210,20cyclictest0-21swapper/022:04:080
101386999210,20cyclictest0-21swapper/020:17:000
101387699206,9cyclictest42-21ksoftirqd/320:25:013
101387699206,5cyclictest42-21ksoftirqd/323:50:003
101387699205,13cyclictest1123925-21expr00:20:133
1013876992018,1cyclictest0-21swapper/320:24:123
1013876992017,2cyclictest3056-21PK-Backend21:52:013
1013876992013,5cyclictest1082593-21cat22:20:233
1013876992012,6cyclictest42-21ksoftirqd/300:30:143
1013876992012,6cyclictest1066671-21ntp_states21:35:203
1013876992012,2cyclictest42-21ksoftirqd/319:45:203
1013876992011,3cyclictest42-21ksoftirqd/323:00:243
101387699200,8cyclictest1109302-21/usr/sbin/munin23:40:013
101387699200,19cyclictest0-21swapper/323:40:163
101387699200,19cyclictest0-21swapper/323:25:013
101387699200,19cyclictest0-21swapper/322:31:543
101387699200,19cyclictest0-21swapper/321:12:123
101387699200,19cyclictest0-21swapper/320:57:263
101387699200,19cyclictest0-21swapper/320:02:033
101387699200,19cyclictest0-21swapper/319:50:263
101387699200,19cyclictest0-21swapper/319:42:363
101387699200,19cyclictest0-21swapper/319:34:103
101387699200,19cyclictest0-21swapper/300:00:443
101387499207,3cyclictest141rcu_preempt23:47:212
1013874992018,1cyclictest0-21swapper/221:15:092
1013874992018,1cyclictest0-21swapper/221:14:222
1013874992018,1cyclictest0-21swapper/219:53:312
1013874992017,2cyclictest0-21swapper/221:52:062
1013874992013,6cyclictest1088192-21apt-get22:40:012
1013874992013,6cyclictest1027604-21awk19:45:152
1013874992013,4cyclictest34-21ksoftirqd/200:05:002
1013874992012,6cyclictest1106825-21ntp_states23:30:202
1013874992012,6cyclictest1025478-21awk19:40:132
101387499200,7cyclictest1065671-21perf21:35:012
101387499200,1cyclictest0-21swapper/220:45:152
101387499200,1cyclictest0-21swapper/220:26:422
101387499200,1cyclictest0-21swapper/220:12:102
101387499200,19cyclictest0-21swapper/222:53:312
101387499200,19cyclictest0-21swapper/220:41:302
101387499200,19cyclictest0-21swapper/220:37:452
101387499200,19cyclictest0-21swapper/200:16:452
101387199208,2cyclictest141rcu_preempt22:43:241
101387199201,8cyclictest1035610-21grep20:09:591
1013871992018,1cyclictest3056-21PK-Backend20:56:141
1013871992018,1cyclictest3056-21PK-Backend20:19:301
101387199201,7cyclictest26-21ksoftirqd/122:25:001
1013871992017,2cyclictest3056-21PK-Backend19:35:431
101387199200,1cyclictest3056-21PK-Backend19:51:041
101387199200,19cyclictest3056-21PK-Backend20:49:111
101387199200,18cyclictest3056-21PK-Backend19:45:101
101387199200,11cyclictest1078441-21df_inode22:10:171
101387199200,11cyclictest1078441-21df_inode22:10:171
101386999206,3cyclictest13-21ksoftirqd/021:10:120
101386999204,5cyclictest13-21ksoftirqd/020:30:220
1013869992018,1cyclictest0-21swapper/023:36:490
1013869992018,1cyclictest0-21swapper/023:20:190
1013869992018,1cyclictest0-21swapper/023:14:460
1013869992018,1cyclictest0-21swapper/023:05:260
1013869992018,1cyclictest0-21swapper/019:23:060
1013869992018,1cyclictest0-21swapper/019:13:000
1013869992016,3cyclictest0-21swapper/022:10:170
1013869992016,3cyclictest0-21swapper/022:10:160
1013869992012,6cyclictest1120970-21ntpq00:10:200
1013869992012,6cyclictest1041918-21memory20:25:170
1013869992011,3cyclictest13-21ksoftirqd/000:20:200
1013869992010,6cyclictest13-21ksoftirqd/021:05:010
101386999200,1cyclictest0-21swapper/022:37:550
101386999200,19cyclictest3056-21PK-Backend22:48:220
101386999200,19cyclictest0-21swapper/021:09:390
101386999200,19cyclictest0-21swapper/019:47:200
101386999200,19cyclictest0-21swapper/019:38:080
101386999200,19cyclictest0-21swapper/019:25:500
101386999200,18cyclictest0-21swapper/023:40:470
139099190,11rtkit-daemon11146062chrt23:53:320
101387699194,5cyclictest141rcu_preempt22:03:453
1013876991914,3cyclictest1119231-21ntp_states00:05:213
1013876991913,5cyclictest1098247-21rt-features23:05:223
1013876991913,4cyclictest1069042-21sshd21:44:073
1013876991913,4cyclictest1020250-21sshd19:25:063
1013876991912,5cyclictest1094860-21cat22:55:213
1013876991912,5cyclictest1044574-21wc20:35:013
1013876991911,6cyclictest1092507-21expr22:50:153
101387699191,16cyclictest1065067-21python321:30:243
1013876991911,2cyclictest42-21ksoftirqd/323:20:213
1013876991911,2cyclictest42-21ksoftirqd/322:40:243
101387699190,1cyclictest0-21swapper/323:56:443
101387699190,1cyclictest0-21swapper/323:52:463
101387699190,1cyclictest0-21swapper/323:19:263
101387699190,18cyclictest3056-21PK-Backend21:23:203
101387699190,18cyclictest0-21swapper/319:24:373
101387699190,17cyclictest1031480-21wc19:55:203
101387499199,8cyclictest1127522-21df00:30:132
1013874991917,1cyclictest0-21swapper/222:59:182
1013874991917,1cyclictest0-21swapper/222:36:002
1013874991917,1cyclictest0-21swapper/219:55:052
1013874991917,1cyclictest0-21swapper/219:35:322
1013874991917,1cyclictest0-21swapper/219:30:242
1013874991917,1cyclictest0-21swapper/200:07:322
1013874991915,3cyclictest34-21ksoftirqd/223:20:012
1013874991914,4cyclictest34-21ksoftirqd/223:15:002
1013874991914,3cyclictest1074771-21tr22:00:122
1013874991911,6cyclictest1085751-21ntpq22:30:202
1013874991911,6cyclictest1040213-21python320:20:222
101387499190,4cyclictest1097118-21cron23:04:592
101387499190,1cyclictest0-21swapper/222:12:082
101387499190,1cyclictest0-21swapper/222:12:072
101387499190,1cyclictest0-21swapper/220:15:552
101387199190,8cyclictest3056-21PK-Backend23:10:221
101387199190,8cyclictest3056-21PK-Backend00:00:221
101387199190,5cyclictest3056-21PK-Backend23:05:211
1013869991917,1cyclictest0-21swapper/022:53:420
1013869991917,1cyclictest0-21swapper/022:44:380
1013869991917,1cyclictest0-21swapper/022:34:090
1013869991917,1cyclictest0-21swapper/021:26:440
1013869991917,1cyclictest0-21swapper/021:16:420
1013869991917,1cyclictest0-21swapper/020:38:090
1013869991917,1cyclictest0-21swapper/020:12:070
1013869991917,1cyclictest0-21swapper/000:08:590
1013869991913,5cyclictest1077200-21swap22:05:210
1013869991912,6cyclictest1016865-21cat19:15:130
1013869991912,5cyclictest1071755-21cat21:50:220
1013869991911,2cyclictest13-21ksoftirqd/000:05:000
101386999190,5cyclictest141rcu_preempt00:30:010
101386999190,1cyclictest0-21swapper/021:36:430
101386999190,1cyclictest0-21swapper/020:22:290
101386999190,18cyclictest0-21swapper/023:48:310
101386999190,18cyclictest0-21swapper/022:55:180
101386999190,18cyclictest0-21swapper/021:21:370
101386999190,11cyclictest1079803-21runrttasks22:15:120
101387699186,5cyclictest141rcu_preempt20:45:143
1013876991816,1cyclictest3056-21PK-Backend22:08:153
1013876991811,5cyclictest436-20systemd-journal20:42:083
1013876991810,6cyclictest886-21NetworkManager22:49:503
1013876991810,6cyclictest1129537-21/usr/sbin/munin00:35:163
101387699180,17cyclictest3056-21PK-Backend00:12:263
101387699180,16cyclictest1036679-21/usr/sbin/munin20:10:203
101387499182,6cyclictest34-21ksoftirqd/223:05:212
101387499181,3cyclictest34-21ksoftirqd/200:30:002
1013874991813,4cyclictest34-21ksoftirqd/223:25:002
1013874991813,4cyclictest1051509-21cron20:55:012
1013874991811,5cyclictest1082203-21strings22:20:182
1013874991811,5cyclictest1052207-21ls20:55:162
1013874991811,4cyclictest1095-21runrttasks22:15:162
101387199189,2cyclictest1124812-21chrt00:20:491
101387199188,2cyclictest141rcu_preempt22:59:251
101387199180,9cyclictest3056-21PK-Backend23:22:091
101387199180,6cyclictest3056-21PK-Backend19:24:091
101386999184,3cyclictest141rcu_preempt23:35:010
1013869991816,1cyclictest0-21swapper/020:46:480
1013869991816,1cyclictest0-21swapper/020:41:070
1013869991810,2cyclictest13-21ksoftirqd/019:35:010
101386999180,8cyclictest141rcu_preempt21:45:000
101386999180,1cyclictest0-21swapper/023:03:030
101386999180,16cyclictest3056-21PK-Backend19:40:180
101387699173,2cyclictest141rcu_preempt21:05:013
1013876991714,2cyclictest3056-21PK-Backend23:32:463
1013876991712,3cyclictest1069772-21diskstats21:45:143
1013876991711,5cyclictest1059362-21diskstats21:15:153
101387499179,6cyclictest1105189-21awk23:25:202
101387499179,6cyclictest1068057-21awk21:40:172
101387499179,5cyclictest1130619-21dump-pmu-power00:40:012
101387499172,4cyclictest34-21ksoftirqd/220:34:592
1013874991712,4cyclictest34-21ksoftirqd/223:40:002
101386999176,4cyclictest141rcu_preempt19:55:000
1013869991715,1cyclictest0-21swapper/020:56:220
101386999171,4cyclictest1122326-21df_inode00:15:130
1013869991713,3cyclictest1032392-21/usr/sbin/munin20:00:230
1013869991712,3cyclictest13-21ksoftirqd/023:19:590
101386999170,8cyclictest0-21swapper/000:35:200
101386999170,1cyclictest0-21swapper/019:57:210
101387699160,1cyclictest3056-21PK-Backend21:55:323
101387699160,1cyclictest3056-21PK-Backend21:08:073
101387699160,14cyclictest978-21snmpd00:16:273
1013874991614,1cyclictest0-21swapper/200:24:562
1013874991611,3cyclictest1083725-21if_err_enp2s022:25:162
101386999169,2cyclictest13-21ksoftirqd/022:20:180
101386999165,2cyclictest141rcu_preempt20:50:010
1013869991614,1cyclictest0-21swapper/000:30:160
1013869991610,5cyclictest1065377-21sshd21:31:330
101387499159,4cyclictest1109508-21runrttasks23:40:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional