You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-14 - 08:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed May 14, 2025 00:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
159779821280,7sleep11597797-21proc_pri19:05:271
161756821050,6sleep21617573-21needreboot20:00:242
16831062950,5sleep00-21swapper/023:15:350
159820829275,10sleep30-21swapper/319:06:463
159829929075,9sleep20-21swapper/219:07:582
16945082890,5sleep1251rcuc/123:50:181
16382622890,18sleep2331rcuc/221:00:302
16032422810,4chrt1602731-21/usr/sbin/munin19:20:252
162399790,63rtkit-daemon0-21swapper/019:06:420
162399620,39rtkit-daemon0-21swapper/119:32:561
1598596996211,29cyclictest1753-21gnome-shell00:01:542
1598596996115,25cyclictest1753-21gnome-shell00:31:352
1598596996017,17cyclictest1753-21gnome-shell00:10:012
1598596996017,17cyclictest1753-21gnome-shell00:10:002
1598596996012,19cyclictest1620856-21/usr/sbin/munin20:10:212
1598596996010,29cyclictest1753-21gnome-shell23:17:312
1598596995911,28cyclictest1753-21gnome-shell21:38:352
1598596995911,28cyclictest1753-21gnome-shell21:38:352
1598596995814,18cyclictest1753-21gnome-shell21:05:312
1598596995813,16cyclictest1753-21gnome-shell21:40:212
17052532570,7chrt0-21swapper/300:20:253
1598596995715,19cyclictest1669276-21/usr/sbin/munin22:40:342
1598596995715,19cyclictest1669276-21/usr/sbin/munin22:40:342
1598596995711,28cyclictest1753-21gnome-shell19:28:452
1598589995721,7cyclictest1069-21runrttasks23:05:170
1598589995720,11cyclictest1645868-21wc21:25:260
1598596995615,21cyclictest1753-21gnome-shell22:03:102
1598596995610,26cyclictest1753-21gnome-shell23:32:132
159859699559,21cyclictest1666624-21awk22:30:292
1598596995516,15cyclictest1629564-21/usr/sbin/munin20:35:232
1598596995514,16cyclictest860997-21snmpd19:52:232
159858999559,16cyclictest1753-21gnome-shell20:38:580
1598589995518,10cyclictest1614976-21grep19:55:000
1598589995514,11cyclictest1616129-21grep19:55:340
16761732540,5chrt0-21swapper/123:00:021
1598596995417,10cyclictest1703283-21/usr/sbin/munin00:15:212
159858999546,18cyclictest1753-21gnome-shell22:30:320
1598589995416,11cyclictest1676144-21wc23:00:030
159859699539,18cyclictest1753-21gnome-shell00:23:122
1598596995316,12cyclictest1753-21gnome-shell22:55:202
1598596995315,15cyclictest1626699-21sshd20:25:452
1598596995312,14cyclictest1671450-21/usr/sbin/munin22:45:232
1598596995311,16cyclictest1618640-21/usr/sbin/munin20:05:242
159858999539,18cyclictest1753-21gnome-shell00:39:000
1598589995320,7cyclictest1622873-21sh20:15:270
1598589995317,7cyclictest1705152-21missed_timers00:20:250
1598589995312,11cyclictest1680889-21cat23:10:340
1598589995310,16cyclictest1753-21gnome-shell22:16:100
159859699529,19cyclictest1707014-21/usr/sbin/munin00:25:262
1598596995217,14cyclictest1753-21gnome-shell19:55:082
1598596995215,19cyclictest1753-21gnome-shell23:01:022
1598596995215,18cyclictest1753-21gnome-shell22:14:252
1598596995215,10cyclictest1678318-21/usr/sbin/munin23:05:192
1598596995212,15cyclictest1658269-21/usr/sbin/munin22:05:082
1598596995211,17cyclictest1753-21gnome-shell23:45:132
1598589995210,16cyclictest1753-21gnome-shell22:40:410
1598589995210,16cyclictest1753-21gnome-shell22:40:410
1598596995116,11cyclictest1753-21gnome-shell19:49:152
1598596995116,11cyclictest1753-21gnome-shell19:49:152
159858999519,16cyclictest1753-21gnome-shell23:04:490
159858999517,17cyclictest1645097-21wc21:25:020
1598589995118,7cyclictest2197014-21PK-Backend19:19:380
1598589995118,7cyclictest1652407-21/usr/sbin/munin21:45:290
1598589995117,7cyclictest1753-21gnome-shell19:10:370
1598589995115,8cyclictest2197014-21PK-Backend22:28:590
1598589995113,11cyclictest1639662-21expr21:05:180
159858999511,24cyclictest1753-21gnome-shell20:41:030
159858999511,22cyclictest1753-21gnome-shell23:37:360
1598589995110,16cyclictest1753-21gnome-shell22:53:370
1598589995110,16cyclictest1753-21gnome-shell20:25:520
1598589995110,16cyclictest1753-21gnome-shell20:10:110
1598596995016,11cyclictest1753-21gnome-shell21:19:282
1598596995016,11cyclictest1753-21gnome-shell21:13:422
1598596995016,11cyclictest1753-21gnome-shell21:13:412
1598596995016,11cyclictest1753-21gnome-shell20:42:562
1598596995015,14cyclictest1600997-21/usr/sbin/munin19:15:202
1598596995015,12cyclictest1753-21gnome-shell23:52:382
1598596995015,11cyclictest1753-21gnome-shell23:25:292
1598596995015,11cyclictest1753-21gnome-shell22:51:532
1598596995015,11cyclictest1753-21gnome-shell00:11:112
1598596995014,14cyclictest860997-21snmpd20:15:452
1598596995011,15cyclictest1685216-21munin-run23:25:012
1598596995010,12cyclictest1753-21gnome-shell19:34:532
159858999509,16cyclictest1753-21gnome-shell19:23:370
159858999508,16cyclictest1753-21gnome-shell23:43:180
159858999508,16cyclictest1753-21gnome-shell23:43:170
159858999508,16cyclictest1753-21gnome-shell00:33:250
1598589995019,7cyclictest1628090-21open_inodes20:30:310
1598589995018,7cyclictest2197014-21PK-Backend21:42:380
1598589995016,7cyclictest2197014-21PK-Backend21:04:060
1598589995012,12cyclictest1656325-21awk21:55:210
159859699497,14cyclictest2197014-21PK-Backend20:48:172
1598596994915,12cyclictest1753-21gnome-shell21:29:032
1598596994915,11cyclictest1753-21gnome-shell20:21:512
1598596994914,12cyclictest1753-21gnome-shell21:51:252
1598596994910,11cyclictest1753-21gnome-shell21:30:562
1598589994918,7cyclictest1606696-21irqstats19:30:220
1598589994916,7cyclictest2197014-21PK-Backend20:03:050
159859699488,19cyclictest1692038-21taskset23:44:032
159859699488,19cyclictest1692038-21taskset23:44:022
1598596994815,11cyclictest1753-21gnome-shell23:10:122
1598596994815,11cyclictest1753-21gnome-shell22:25:352
1598596994815,11cyclictest1753-21gnome-shell21:48:032
1598596994815,10cyclictest1753-21gnome-shell19:13:582
1598596994810,11cyclictest1753-21gnome-shell20:58:192
1598594994822,22cyclictest1602474-21cron19:20:001
1598589994817,7cyclictest1619535-21python320:05:300
159858999481,22cyclictest1753-21gnome-shell20:48:400
159859999471,42cyclictest1622-21rtkit-daemon19:21:563
1598596994719,23cyclictest1710222-21kernelversion00:35:252
1598596994714,12cyclictest1753-21gnome-shell23:35:292
1598589994717,6cyclictest181irq_work/023:45:170
1598589994716,7cyclictest2197014-21PK-Backend23:58:100
1598589994712,13cyclictest438-20systemd-journal23:25:010
1598596994610,12cyclictest1753-21gnome-shell20:51:482
1598599994515,16cyclictest42-21ksoftirqd/322:36:113
159859699451,7cyclictest1663838-21perf22:25:002
1598596994510,11cyclictest1753-21gnome-shell23:56:202
1598596994510,11cyclictest1753-21gnome-shell19:43:452
1598594994526,15cyclictest3242253-21nautilus00:00:011
16098982440,9sleep11598529-21kworker/1:119:40:041
159859999449,12cyclictest42-21ksoftirqd/320:39:263
159859999448,19cyclictest1753-21gnome-shell00:13:393
159859999441,16cyclictest1637274-21seq20:58:193
1598596994410,11cyclictest1753-21gnome-shell21:23:182
15996012430,6sleep11598584-21cyclictest19:10:301
159859999437,11cyclictest1753-21gnome-shell23:34:383
159859999435,9cyclictest1652407-21/usr/sbin/munin21:45:283
1598599994337,4cyclictest0-21swapper/320:09:393
159859999432,37cyclictest1753-21gnome-shell19:57:093
1598589994320,18cyclictest1700792-21cron00:10:010
1598589994320,18cyclictest1700792-21cron00:10:010
159859999427,31cyclictest1753-21gnome-shell21:38:133
159859999427,31cyclictest1753-21gnome-shell21:38:123
159859999427,19cyclictest1753-21gnome-shell20:52:333
159859999427,16cyclictest1753-21gnome-shell00:06:303
159859999427,16cyclictest1753-21gnome-shell00:06:303
1598599994224,14cyclictest855-21dbus-daemon23:00:023
1598599994224,14cyclictest1678371-21ls23:05:233
1598599994223,14cyclictest438-20systemd-journal00:05:013
1598599994220,18cyclictest1069-21runrttasks21:24:443
159859999421,25cyclictest1753-21gnome-shell23:52:393
1598599994211,29cyclictest411rcuc/322:02:513
1598594994220,19cyclictest1660011-21awk22:10:131
159858999422,15cyclictest1703679-21python300:15:300
1598589994216,8cyclictest13-21ksoftirqd/019:41:050
16828922410,6chrt0-21swapper/123:15:311
162399410,29rtkit-daemon2197014-21PK-Backend20:20:051
159859999416,18cyclictest1753-21gnome-shell22:07:333
1598599994127,10cyclictest1690366-21sshd23:39:483
1598599994124,13cyclictest855-21dbus-daemon23:11:563
1598599994124,13cyclictest855-21dbus-daemon22:51:563
1598599994123,14cyclictest1654559-21df21:50:123
159859999410,25cyclictest1753-21gnome-shell21:40:313
159859699413,16cyclictest0-21swapper/222:19:262
1598594994118,18cyclictest1640179-21ntp_states21:05:311
1598594994113,23cyclictest1627490-21cut20:30:171
1598589994121,15cyclictest1702562-21pool-org.gnome.00:15:010
162399400,9rtkit-daemon1753-21gnome-shell19:56:271
159859999407,11cyclictest42-21ksoftirqd/322:31:033
159859999407,11cyclictest42-21ksoftirqd/321:10:153
159859999407,11cyclictest42-21ksoftirqd/321:10:143
159859999405,17cyclictest1753-21gnome-shell19:11:113
159859999405,11cyclictest1665550-21taskset22:29:563
1598599994029,7cyclictest1601983-21wc19:15:313
159859999402,20cyclictest1753-21gnome-shell21:00:163
1598596994017,19cyclictest1668247-21needreboot22:35:242
1598594994024,12cyclictest1704252-21cron00:20:001
159859499402,34cyclictest1753-21gnome-shell00:01:071
159858999401,5cyclictest2197014-21PK-Backend19:36:040
162399390,27rtkit-daemon0-21swapper/120:28:241
159859999399,13cyclictest42-21ksoftirqd/322:48:313
159859999395,8cyclictest0-21swapper/320:15:233
159859999395,11cyclictest1753-21gnome-shell19:46:163
159859999395,11cyclictest1753-21gnome-shell19:46:163
1598599993927,8cyclictest1753-21gnome-shell23:55:063
1598599993921,13cyclictest1710466-21ntpq00:35:283
159859999391,15cyclictest1753-21gnome-shell23:20:293
159859999391,12cyclictest1626171-21memory20:25:243
1598594993929,6cyclictest1069-21runrttasks00:32:411
1598594993926,9cyclictest1705985-21cron00:25:011
159859499392,33cyclictest1753-21gnome-shell21:59:091
1598594993922,12cyclictest438-20systemd-journal23:38:481
1598594993921,13cyclictest1678695-21ntp_states23:05:321
1598594993917,19cyclictest1648469-21unin-run21:35:011
16713422380,4sleep00-21swapper/022:45:170
16344552380,6chrt0-21swapper/120:50:231
162399380,26rtkit-daemon0-21swapper/100:35:581
162399380,25rtkit-daemon0-21swapper/122:49:271
162399380,24rtkit-daemon0-21swapper/120:40:261
159859999388,12cyclictest42-21ksoftirqd/320:40:483
159859999387,12cyclictest42-21ksoftirqd/320:30:223
159859999386,12cyclictest42-21ksoftirqd/323:15:163
159859999386,10cyclictest2197014-21PK-Backend21:05:123
159859999385,20cyclictest0-21swapper/322:15:193
1598599993825,9cyclictest1753-21gnome-shell20:21:123
159859999382,32cyclictest1753-21gnome-shell20:02:463
1598599993823,12cyclictest1604489-21dpkg19:25:153
1598599993823,11cyclictest1661364-21sshd22:13:463
1598599993821,12cyclictest1633580-21cat20:45:313
1598599993820,14cyclictest1692763-21df23:45:163
1598599993820,13cyclictest1632-21dbus-daemon19:45:013
1598599993819,14cyclictest1614525-21users19:50:363
1598599993812,23cyclictest1753-21gnome-shell19:35:293
1598596993818,15cyclictest1753-21gnome-shell19:35:122
1598596993817,16cyclictest1753-21gnome-shell21:55:112
1598594993819,16cyclictest1664316-21cstates22:25:111
1598594993819,15cyclictest3242253-21nautilus19:50:021
1598594993819,15cyclictest3242253-21nautilus19:50:011
1598594993816,18cyclictest1753-21gnome-shell22:34:511
1598594993813,21cyclictest1645847-21ls21:25:271
159858999381,32cyclictest1637429-21turbostat21:00:000
159859999377,16cyclictest42-21ksoftirqd/323:41:033
159859999377,16cyclictest42-21ksoftirqd/323:41:033
159859999377,16cyclictest42-21ksoftirqd/323:25:283
159859999373,11cyclictest1647459-21interrupts21:30:233
1598599993727,7cyclictest1069-21runrttasks22:21:253
1598599993722,11cyclictest1621838-21cron20:15:003
1598599993720,13cyclictest1703388-21memory00:15:223
1598599993719,14cyclictest1738-21gdbus23:01:563
1598599993714,20cyclictest391irq_work/322:42:153
1598599993714,20cyclictest391irq_work/322:42:143
159859999371,31cyclictest1753-21gnome-shell21:16:363
1598596993715,18cyclictest1627295-21/usr/sbin/munin20:30:352
159859499372,7cyclictest1753-21gnome-shell00:10:111
159859499372,31cyclictest1753-21gnome-shell23:25:471
159859499372,31cyclictest1753-21gnome-shell20:11:531
1598594993723,10cyclictest26-21ksoftirqd/122:50:201
1598594993723,10cyclictest1639233-21cat21:05:011
1598594993721,11cyclictest438-20systemd-journal20:05:011
1598594993717,10cyclictest1683946-21/usr/sbin/munin23:20:151
159859499371,32cyclictest1603056-21cstates19:20:161
159858999371,6cyclictest1605952-21turbostat19:30:000
162399360,24rtkit-daemon0-21swapper/121:15:151
159859999368,16cyclictest42-21ksoftirqd/300:32:033
159859999366,10cyclictest2197014-21PK-Backend21:25:413
159859999364,10cyclictest42-21ksoftirqd/300:25:453
1598599993623,8cyclictest1753-21gnome-shell21:55:153
159859999361,13cyclictest42-21ksoftirqd/319:30:153
1598594993620,12cyclictest1654676-21irqstats21:50:191
159859499361,31cyclictest1753-21gnome-shell19:36:521
159858999361,11cyclictest1663845-21munin-plugin-st22:25:010
1598594993520,11cyclictest1700074-21ntp_kernel_pll_00:05:261
1598594993520,11cyclictest1700074-21ntp_kernel_pll_00:05:261
1598594993518,13cyclictest855-21dbus-daemon23:50:011
1598594993513,6cyclictest1652407-21/usr/sbin/munin21:45:201
1598594993513,6cyclictest141rcu_preempt23:15:021
1598589993525,7cyclictest1069-21runrttasks20:54:160
162399340,5rtkit-daemon0-21swapper/122:39:161
159859499348,10cyclictest26-21ksoftirqd/120:05:131
1598594993424,7cyclictest1669487-21tr22:40:161
1598594993424,7cyclictest1669487-21tr22:40:151
1598594993423,7cyclictest1641314-21cpuspeed_turbos21:10:171
1598594993423,7cyclictest1641314-21cpuspeed_turbos21:10:171
1598594993420,10cyclictest1687657-21awk23:30:221
1598594993419,11cyclictest1662486-21ntpq22:15:281
1598594993418,12cyclictest1651392-21timerandwakeup21:40:311
159858999344,6cyclictest141rcu_preempt23:52:340
159858999342,13cyclictest1753-21gnome-shell22:35:290
159858999341,9cyclictest2197014-21PK-Backend22:12:410
1598589993417,12cyclictest1706995-21awk00:25:270
159858999341,29cyclictest2197014-21PK-Backend22:00:120
159858999341,29cyclictest2197014-21PK-Backend21:50:300
162399330,6rtkit-daemon2197014-21PK-Backend00:25:231
1598594993319,10cyclictest26-21ksoftirqd/123:44:081
1598594993319,10cyclictest26-21ksoftirqd/123:44:081
1598594993319,10cyclictest26-21ksoftirqd/121:21:441
1598594993315,14cyclictest1753-21gnome-shell19:50:271
1598594993311,19cyclictest1649362-21needreboot21:35:271
1598594993311,19cyclictest1649362-21needreboot21:35:271
1598589993322,7cyclictest1641905-21ntp_states21:10:300
1598589993322,7cyclictest1641905-21ntp_states21:10:300
1598589993321,8cyclictest1685531-21cpu23:25:140
1598589993318,11cyclictest1697986-21sh00:00:160
1598594993222,7cyclictest26-21ksoftirqd/122:05:491
1598594993216,12cyclictest1636681-21ntpq20:55:251
159859499321,28cyclictest2197014-21PK-Backend20:20:011
159858999321,28cyclictest2197014-21PK-Backend21:37:230
159858999321,28cyclictest2197014-21PK-Backend21:37:230
159858999321,28cyclictest2197014-21PK-Backend21:31:440
1598594993123,5cyclictest1598584-21cyclictest22:21:211
1598594993119,8cyclictest1632696-21tr20:45:151
159859499311,4cyclictest0-21swapper/120:38:281
159859499311,27cyclictest0-21swapper/119:28:251
159858999311,27cyclictest2197014-21PK-Backend21:17:590
159859499301,11cyclictest1657940-21chrt22:04:121
1598589993020,5cyclictest13-21ksoftirqd/019:46:590
1598589993020,5cyclictest13-21ksoftirqd/019:46:580
159858999301,5cyclictest2197014-21PK-Backend23:30:450
1598589993010,16cyclictest1753-21gnome-shell20:20:150
1598594992918,7cyclictest1676311-21ls23:00:181
159858999291,10cyclictest2197014-21PK-Backend22:05:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional