You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 14:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Sep 08, 2026 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
157476522930,9sleep00-21swapper/019:05:220
157555822680,9sleep13056-21PK-Backend19:10:001
159152321330,3sleep20-21swapper/219:50:232
165713021230,3sleep00-21swapper/023:00:120
169125621110,3sleep23056-21PK-Backend00:35:182
139099730,59rtkit-daemon0-21swapper/319:06:033
157537826343,14sleep20-21swapper/219:08:342
16827082520,3sleep00-21swapper/000:10:260
16061732520,2sleep13056-21PK-Backend20:35:131
16026682480,3sleep20-21swapper/220:25:122
16735482470,1chrt1672657-21/usr/sbin/munin23:45:211
15961202410,3sleep00-21swapper/020:05:170
16135082400,3sleep20-21swapper/220:55:112
15949852390,2sleep20-21swapper/220:00:222
16613462360,2sleep30-21swapper/323:10:263
15884692360,4sleep01588444-21perf19:45:000
157563899318,21cyclictest1621912-21/usr/sbin/munin21:20:153
1575638992911,16cyclictest1655708-21wc22:55:193
139099280,20rtkit-daemon0-21swapper/321:05:183
139099280,20rtkit-daemon0-21swapper/321:05:173
16686442270,4sleep13056-21PK-Backend23:31:171
157563899255,7cyclictest1603337-21awk20:25:203
1575638992515,8cyclictest1578227-21munin-node19:15:023
1575638992510,13cyclictest1594489-21memory20:00:173
157563899249,14cyclictest1615346-21df_inode21:00:163
157563899249,13cyclictest1650006-21diskstats22:40:143
157563899245,6cyclictest1599109-21apt20:15:123
157563899244,8cyclictest0-21swapper/322:46:123
1575638992421,2cyclictest0-21swapper/300:24:013
1575638992420,3cyclictest0-21swapper/322:07:533
1575638992420,3cyclictest0-21swapper/322:07:533
139099240,16rtkit-daemon3056-21PK-Backend19:10:182
157563899239,12cyclictest1672527-21date23:45:013
157563899239,12cyclictest1604784-21awk20:30:173
157563899239,12cyclictest1583990-21interrupts19:30:183
157563899234,7cyclictest0-21swapper/321:50:133
157563899233,2cyclictest0-21swapper/322:25:243
1575638992310,4cyclictest1590815-21/usr/sbin/munin19:50:143
1575638992310,11cyclictest978-21snmpd19:35:123
1575638992310,11cyclictest1620378-21cstates21:15:133
1575628992321,1cyclictest0-21swapper/000:03:010
1575628992320,2cyclictest0-21swapper/022:42:580
157563899229,11cyclictest1597335-21latency_hist20:10:003
157563899229,11cyclictest1587189-21cstates19:40:103
157563899228,13cyclictest1612829-21taskset20:53:213
157563899226,8cyclictest1609451-21runrttasks20:44:283
1575638992220,1cyclictest0-21swapper/322:34:563
1575638992220,1cyclictest0-21swapper/322:13:543
1575638992220,1cyclictest0-21swapper/320:36:143
1575638992220,1cyclictest0-21swapper/319:49:433
1575638992214,6cyclictest1679010-21awk00:00:213
1575638992212,8cyclictest1576019-21grep19:10:143
1575638992211,9cyclictest1667487-21chrt23:30:113
157563899221,18cyclictest1641337-21diskstats22:15:143
157563899220,20cyclictest1690950-21hddtemp_smartct00:35:153
1575630992220,1cyclictest3056-21PK-Backend22:14:291
1575630992219,2cyclictest3056-21PK-Backend23:58:451
1575630992219,2cyclictest3056-21PK-Backend20:58:031
1575630992219,2cyclictest3056-21PK-Backend19:59:101
1575630992219,2cyclictest3056-21PK-Backend19:12:521
1575628992220,1cyclictest0-21swapper/023:29:150
1575628992220,1cyclictest0-21swapper/021:50:400
1575628992220,1cyclictest0-21swapper/020:54:090
1575628992220,1cyclictest0-21swapper/020:18:250
1575628992220,1cyclictest0-21swapper/019:47:160
1575628992220,1cyclictest0-21swapper/019:12:280
1575628992219,2cyclictest0-21swapper/022:02:280
1575628992219,2cyclictest0-21swapper/021:55:530
1575628992219,2cyclictest0-21swapper/019:32:090
1575628992219,2cyclictest0-21swapper/019:20:320
1575628992216,4cyclictest1598843-21munin-run20:14:590
139099220,19rtkit-daemon1997-21gvfs-afc-volume20:02:580
16332462210,4sleep13056-21PK-Backend21:50:201
157563899217,13cyclictest0-21swapper/323:25:143
157563899213,7cyclictest1788643-21eadedCompositor00:13:563
157563899213,7cyclictest1670512-21pool-udisksd23:38:033
1575638992119,1cyclictest0-21swapper/323:54:133
1575638992119,1cyclictest0-21swapper/319:27:403
1575638992119,1cyclictest0-21swapper/300:34:273
1575638992119,1cyclictest0-21swapper/300:07:233
1575638992113,6cyclictest1685075-21cron00:20:003
1575638992113,6cyclictest1677801-21sshd23:59:183
1575638992113,6cyclictest1674386-21sed23:50:003
1575638992113,6cyclictest1658276-21dump-pmu-power23:04:593
1575638992113,5cyclictest1600843-21apt20:20:113
1575638992112,7cyclictest1662192-21tr23:15:123
1575638992112,7cyclictest1596311-21needreboot20:05:193
1575638992111,8cyclictest1642870-21tr22:20:133
1575638992111,8cyclictest1624436-21wc21:25:203
1575633992116,4cyclictest882-21avahi-daemon19:22:222
1575633992113,6cyclictest436-20systemd-journal22:19:592
1575633992112,7cyclictest1628838-21apt21:40:122
1575633992110,9cyclictest1640723-21unin-run22:15:002
157563399210,20cyclictest0-21swapper/220:42:392
1575630992119,1cyclictest3056-21PK-Backend22:15:501
1575630992119,1cyclictest3056-21PK-Backend22:01:171
1575630992119,1cyclictest3056-21PK-Backend21:05:141
1575630992119,1cyclictest3056-21PK-Backend21:05:131
1575630992118,2cyclictest3056-21PK-Backend22:50:041
1575630992118,2cyclictest3056-21PK-Backend22:39:231
1575630992118,2cyclictest3056-21PK-Backend20:18:091
1575630992118,2cyclictest3056-21PK-Backend00:22:241
157563099210,20cyclictest3056-21PK-Backend20:25:111
157563099210,19cyclictest3056-21PK-Backend23:40:551
157563099210,19cyclictest3056-21PK-Backend23:06:141
157563099210,19cyclictest3056-21PK-Backend22:09:501
157563099210,19cyclictest3056-21PK-Backend22:09:501
157563099210,19cyclictest3056-21PK-Backend19:34:351
157563099210,19cyclictest3056-21PK-Backend19:22:261
157563099210,19cyclictest3056-21PK-Backend19:18:001
157563099210,19cyclictest3056-21PK-Backend00:15:431
157563099210,19cyclictest3056-21PK-Backend00:11:571
157563099210,19cyclictest3056-21PK-Backend00:05:151
1575628992119,1cyclictest3056-21PK-Backend23:44:220
1575628992119,1cyclictest0-21swapper/023:19:150
1575628992119,1cyclictest0-21swapper/023:08:390
1575628992119,1cyclictest0-21swapper/022:38:170
1575628992119,1cyclictest0-21swapper/022:07:090
1575628992119,1cyclictest0-21swapper/022:07:080
1575628992119,1cyclictest0-21swapper/021:47:220
1575628992119,1cyclictest0-21swapper/021:21:060
1575628992119,1cyclictest0-21swapper/020:38:400
1575628992119,1cyclictest0-21swapper/000:27:070
1575628992118,2cyclictest0-21swapper/019:19:100
1575628992115,4cyclictest1647638-21perf22:35:000
157562899210,20cyclictest0-21swapper/023:36:310
157562899210,20cyclictest0-21swapper/021:27:070
157562899210,20cyclictest0-21swapper/021:18:220
157562899210,20cyclictest0-21swapper/021:11:450
157562899210,20cyclictest0-21swapper/000:33:500
157562899210,1cyclictest0-21swapper/022:57:440
157562899210,19cyclictest0-21swapper/023:55:100
139099210,14rtkit-daemon0-21swapper/222:43:412
139099210,14rtkit-daemon0-21swapper/200:19:222
157563899209,9cyclictest1634614-21missed_timers21:55:183
157563899208,11cyclictest1663729-21/usr/sbin/munin23:20:223
1575638992018,1cyclictest0-21swapper/322:37:403
1575638992018,1cyclictest0-21swapper/322:01:513
157563899201,6cyclictest1687064-21/usr/sbin/munin00:25:163
1575638992011,7cyclictest1630734-21cpuspeed_turbos21:45:143
1575638992011,7cyclictest1627249-21cut21:35:133
1575638992011,7cyclictest1626410-21/usr/sbin/munin21:30:253
1575638992010,8cyclictest1658928-21/usr/sbin/munin23:05:143
1575638992010,8cyclictest1653944-21memory22:50:233
1575638992010,8cyclictest1592837-21grep19:55:193
1575638992010,8cyclictest1580259-21ls19:20:113
1575638992010,7cyclictest0-21swapper/320:45:103
157563899200,19cyclictest0-21swapper/321:10:203
157563899200,19cyclictest0-21swapper/320:56:363
1575633992018,1cyclictest0-21swapper/221:25:522
1575633992018,1cyclictest0-21swapper/220:24:542
1575633992014,5cyclictest883-21dbus-daemon23:40:012
1575633992013,5cyclictest34-21ksoftirqd/219:45:012
1575633992012,6cyclictest1662598-21memory23:15:172
157563399200,1cyclictest0-21swapper/221:54:512
157563399200,19cyclictest0-21swapper/223:55:302
157563399200,19cyclictest0-21swapper/223:07:452
157563399200,19cyclictest0-21swapper/221:31:192
157563399200,19cyclictest0-21swapper/221:20:232
157563399200,19cyclictest0-21swapper/219:35:202
157563399200,19cyclictest0-21swapper/200:34:382
157563399200,19cyclictest0-21swapper/200:24:432
1575630992018,1cyclictest3056-21PK-Backend23:23:031
1575630992018,1cyclictest3056-21PK-Backend21:12:581
1575630992018,1cyclictest3056-21PK-Backend20:47:021
1575630992018,1cyclictest3056-21PK-Backend19:52:551
1575630992017,2cyclictest3056-21PK-Backend20:54:121
1575630992017,2cyclictest3056-21PK-Backend20:04:021
157563099200,1cyclictest3056-21PK-Backend21:47:311
157563099200,19cyclictest3056-21PK-Backend23:13:031
157563099200,19cyclictest3056-21PK-Backend21:01:551
157563099200,19cyclictest3056-21PK-Backend20:43:171
157563099200,19cyclictest3056-21PK-Backend20:33:021
157563099200,19cyclictest3056-21PK-Backend20:07:341
157563099200,19cyclictest3056-21PK-Backend00:32:041
157563099200,18cyclictest3056-21PK-Backend23:18:441
1575628992018,1cyclictest3056-21PK-Backend23:45:100
1575628992018,1cyclictest3056-21PK-Backend19:35:480
1575628992018,1cyclictest0-21swapper/022:13:110
1575628992018,1cyclictest0-21swapper/020:28:270
1575628992018,1cyclictest0-21swapper/019:52:580
1575628992012,6cyclictest883-21dbus-daemon20:40:010
157562899200,1cyclictest0-21swapper/019:57:510
157562899200,19cyclictest0-21swapper/023:50:520
157562899200,19cyclictest0-21swapper/023:24:560
157562899200,19cyclictest0-21swapper/022:24:400
157562899200,19cyclictest0-21swapper/021:44:030
157562899200,19cyclictest0-21swapper/021:01:180
157562899200,19cyclictest0-21swapper/020:21:580
157562899200,19cyclictest0-21swapper/000:15:050
139099200,14rtkit-daemon0-21swapper/223:21:102
139099200,14rtkit-daemon0-21swapper/222:50:122
139099200,14rtkit-daemon0-21swapper/222:34:302
1575638991912,5cyclictest1628969-21cpuspeed_turbos21:40:123
157563399199,8cyclictest1667946-21cat23:30:162
1575633991917,1cyclictest0-21swapper/223:48:132
1575633991917,1cyclictest0-21swapper/221:36:132
1575633991917,1cyclictest0-21swapper/221:11:042
1575633991917,1cyclictest0-21swapper/221:03:072
1575633991917,1cyclictest0-21swapper/219:31:062
1575633991917,1cyclictest0-21swapper/219:16:202
1575633991917,1cyclictest0-21swapper/200:26:202
1575633991917,1cyclictest0-21swapper/200:06:332
1575633991917,1cyclictest0-21swapper/200:00:382
1575633991912,5cyclictest1652592-21cat22:45:222
1575633991912,5cyclictest1642707-21grep22:20:112
1575633991912,5cyclictest1621225-21cat21:15:232
1575633991912,5cyclictest1612111-21cat20:50:162
1575633991911,6cyclictest1616694-21find21:05:142
1575633991911,6cyclictest1616694-21find21:05:142
157563399190,1cyclictest0-21swapper/222:00:532
157563399190,1cyclictest0-21swapper/219:47:372
157563399190,18cyclictest0-21swapper/223:43:062
157563399190,18cyclictest0-21swapper/223:12:382
157563399190,18cyclictest0-21swapper/222:26:012
157563399190,18cyclictest0-21swapper/220:35:562
157563399190,18cyclictest0-21swapper/220:18:252
1575630991916,2cyclictest3056-21PK-Backend22:48:421
1575630991916,2cyclictest3056-21PK-Backend21:27:301
1575630991916,2cyclictest3056-21PK-Backend19:37:391
1575630991916,2cyclictest3056-21PK-Backend00:35:501
157563099190,1cyclictest3056-21PK-Backend23:26:341
157563099190,1cyclictest3056-21PK-Backend23:02:071
157563099190,1cyclictest3056-21PK-Backend22:59:231
157563099190,1cyclictest3056-21PK-Backend22:29:151
157563099190,1cyclictest3056-21PK-Backend21:34:541
157563099190,1cyclictest3056-21PK-Backend21:24:471
157563099190,1cyclictest3056-21PK-Backend00:04:261
1575628991917,1cyclictest0-21swapper/023:30:020
1575628991917,1cyclictest0-21swapper/022:45:420
1575628991917,1cyclictest0-21swapper/022:15:540
1575628991917,1cyclictest0-21swapper/021:32:340
1575628991917,1cyclictest0-21swapper/020:46:100
1575628991916,2cyclictest3056-21PK-Backend00:35:250
157562899190,1cyclictest0-21swapper/023:11:230
157562899190,1cyclictest0-21swapper/022:28:440
157562899190,1cyclictest0-21swapper/021:36:390
157562899190,1cyclictest0-21swapper/000:09:430
139099190,13rtkit-daemon0-21swapper/222:36:532
1575633991816,1cyclictest0-21swapper/220:06:162
1575633991813,3cyclictest978-21snmpd23:27:362
1575633991812,4cyclictest1683087-21latency_hist00:15:002
1575633991811,6cyclictest1675923-21seq23:52:252
1575633991811,5cyclictest1635517-21awk22:00:002
157563399180,1cyclictest0-21swapper/222:06:552
157563399180,1cyclictest0-21swapper/222:06:542
1575630991816,1cyclictest3056-21PK-Backend23:38:451
1575630991816,1cyclictest3056-21PK-Backend21:56:031
1575630991816,1cyclictest3056-21PK-Backend21:18:121
1575630991816,1cyclictest3056-21PK-Backend20:11:541
1575630991816,1cyclictest3056-21PK-Backend00:29:071
1575630991815,2cyclictest3056-21PK-Backend19:40:231
1575628991813,4cyclictest1605768-21cron20:35:010
1575628991811,6cyclictest1583254-21sleep19:30:000
157562899180,1cyclictest0-21swapper/021:08:270
157562899180,1cyclictest0-21swapper/021:08:270
157563399179,6cyclictest436-20systemd-journal23:00:002
157563399179,6cyclictest1604301-21awk20:30:112
157563399179,6cyclictest1581781-21apt19:25:112
1575633991715,1cyclictest0-21swapper/223:00:562
1575633991715,1cyclictest0-21swapper/220:14:542
1575633991711,5cyclictest1592917-21ntp_states19:55:192
1575633991711,4cyclictest1631044-21awk21:45:162
1575630991715,1cyclictest3056-21PK-Backend22:30:041
1575630991715,1cyclictest3056-21PK-Backend20:21:411
1575628991715,1cyclictest0-21swapper/022:51:440
1575628991715,1cyclictest0-21swapper/000:20:260
1575633991612,3cyclictest1610811-21timerwakeupswit20:45:222
1575630991614,1cyclictest3056-21PK-Backend22:44:031
1575628991614,1cyclictest0-21swapper/020:55:310
157563099159,4cyclictest3056-21PK-Backend21:43:031
1575630991513,1cyclictest3056-21PK-Backend22:23:141
1575630991513,1cyclictest3056-21PK-Backend19:49:231
139099150,12rtkit-daemon3056-21PK-Backend19:26:491
1575630991412,1cyclictest3056-21PK-Backend23:51:421
157563099139,3cyclictest3056-21PK-Backend21:37:261
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional