You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 22:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Sat Jun 28, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
201642621810,2sleep20-21swapper/212:25:452
15902162148103,19sleep30-21swapper/307:06:003
168065721390,2sleep00-21swapper/008:35:220
1590464213992,24sleep00-21swapper/007:09:090
15902852132107,17sleep20-21swapper/207:06:522
15858432131109,15sleep10-21swapper/107:05:101
177936921290,2sleep2411rcuc/209:53:312
18677862750,2sleep20-21swapper/210:53:292
1590652993826,9cyclictest1925857-21grep11:30:273
1590652992924,3cyclictest703-21dbus-daemon07:36:423
1590652992915,2cyclictest492-21hwrng11:59:043
1590652992622,2cyclictest262-21systemd-journal07:34:593
1590652992621,3cyclictest1913546-21cron11:25:013
18615782251,2sleep30-21swapper/310:50:143
1590652992521,2cyclictest262-21systemd-journal08:20:003
1590652992521,2cyclictest1958889-21dump-pmu-power11:54:593
1590652992521,2cyclictest1642851-21munin-plugin-st08:00:003
1590652992521,2cyclictest0-21swapper/311:25:183
1590652992421,2cyclictest1943547-21dump-pmu-power11:44:593
1590652992420,2cyclictest0-21swapper/307:25:283
1590652992418,4cyclictest0-21swapper/310:45:153
1590652992318,3cyclictest1780228-21sh09:54:593
1590652992316,5cyclictest1602114-21grep07:15:283
1590652992315,5cyclictest1769816-21sort09:45:293
159065299231,20cyclictest1672306-21ntpq08:25:243
1590652992311,8cyclictest2033269-21wc12:35:283
159065299231,18cyclictest1732923-21sh09:20:513
1590652992310,8cyclictest1829041-21awk10:25:333
1590652992310,8cyclictest1801935-21sshd10:09:413
159065299221,8cyclictest1533503-21kworker/u16:2+rpciod09:19:553
1590652992213,7cyclictest1762358-21switchtime09:40:293
1590652992213,2cyclictest1887574-21ntpq11:05:253
159065299221,18cyclictest1880222-21ntpq11:00:283
1590652992211,7cyclictest2000662-21ssh12:17:463
159065299220,19cyclictest2022143-21cut12:30:173
1590652992117,2cyclictest1683253-21sh08:39:593
1590652992117,2cyclictest0-21swapper/311:15:183
1590652992117,2cyclictest0-21swapper/310:40:163
1590652992116,3cyclictest1824133-21turbostat.cron10:25:003
1590652992116,3cyclictest1622594-21kworker/u16:0-rpciod09:29:403
1590652992112,6cyclictest1896481-21sh11:11:573
1590652992110,7cyclictest736-21gdbus08:55:003
1590652992110,7cyclictest2013137-21cut12:25:203
159065299209,8cyclictest1869138-21grep10:55:153
159065299208,8cyclictest1928767-21awk11:35:133
159065299208,8cyclictest1703518-21apt-get09:00:153
159065299208,8cyclictest1703518-21apt-get09:00:143
1590652992018,1cyclictest0-21swapper/312:05:003
1590652992016,2cyclictest0-21swapper/309:10:013
1590652992015,3cyclictest0-21swapper/312:20:173
1590652992011,6cyclictest1806599-21tail10:10:193
1590652992011,6cyclictest1787038-21ssh09:59:123
1590652992011,6cyclictest1637859-21cron07:55:003
159065299200,18cyclictest262-21systemd-journal09:07:093
159065299199,7cyclictest1836060-21mailstats10:30:243
159065299198,8cyclictest0-21swapper/307:13:463
159065299198,7cyclictest736-21gdbus10:20:013
1590652991914,3cyclictest1687178-21grep08:40:193
1590652991912,5cyclictest1667328-21ldconfig08:20:243
1590652991911,6cyclictest1626640-21cut07:40:183
1590652991910,7cyclictest0-21swapper/312:10:193
159065299190,17cyclictest1981696-21sh12:05:583
159065299190,17cyclictest1981696-21sh12:05:573
159065299190,16cyclictest1757680-21taskset09:39:563
19512002181,4sleep11951225-21apt-get11:50:161
19196382181,3sleep20-21swapper/211:27:402
17286642180,3sleep11729522-21kthreadcore09:20:161
159065299189,6cyclictest1947638-21needreboot11:45:213
159065299188,6cyclictest736-21gdbus08:45:243
159065299186,7cyclictest1791809-21collect210:00:253
159065299182,12cyclictest1743347-21fschecks_count09:30:193
159065299181,7cyclictest1703242-21munin-run09:00:003
1590652991814,2cyclictest1647874-21http08:05:013
159065299181,10cyclictest1657223-21ntpq08:10:213
159065299180,6cyclictest1632871-21cron07:50:003
159065299180,16cyclictest1845613-21ssh10:38:473
17556942170,10sleep10-21swapper/109:36:361
1590652991710,4cyclictest1606953-21switchtime07:20:283
20365002165,2sleep0191rcu_preempt12:39:190
19891582161,5sleep21989200-21ntpq12:10:212
19739132161,8sleep11713297-21diskmemload12:02:031
19631912161,3sleep11963248-21ntpq11:55:211
17769122160,3sleep10-21swapper/109:50:211
159065299161,3cyclictest1678243-21idleruntime-cro08:34:593
159065299161,13cyclictest1648808-21/usr/sbin/munin08:05:173
20206332151,6sleep02020749-21irqstats12:30:160
19817082151,8sleep11713297-21diskmemload12:06:001
19817082151,8sleep11713297-21diskmemload12:05:591
19635512151,9sleep21963641-21ssh11:55:272
19412622151,10sleep21941267-21ssh11:40:512
19262932150,1sleep00-21swapper/011:31:070
19062832151,11sleep11906392-21fschecks_time11:20:181
19740572140,1sleep00-21swapper/012:02:150
18749032141,8sleep21713297-21diskmemload10:58:182
18317272141,7sleep11713297-21diskmemload10:30:041
18210952141,8sleep01821156-21ntp_states10:20:250
17575322141,9sleep21713297-21diskmemload09:39:412
16885322141,2sleep20-21swapper/208:45:152
20175812132,7sleep02017596-21sh12:27:120
20114812131,6sleep10-21swapper/112:25:091
20080702131,8sleep22008083-21ssh12:21:122
20033052131,8sleep02002885-21/usr/sbin/munin12:20:180
19990292137,3sleep20-21swapper/212:15:442
19839812131,8sleep01983995-21ssh12:08:520
19839812131,8sleep01983995-21ssh12:08:510
19054952131,8sleep21905509-21ssh11:19:432
16839362131,8sleep21684169-21tr08:40:172
20093502121,5sleep10-21swapper/112:22:471
20023912121,5sleep01713297-21diskmemload12:19:590
19281282125,4sleep11590646-21cyclictest11:34:371
18949672120,8sleep11895009-21ntpq11:10:241
18906012121,5sleep01713297-21diskmemload11:09:470
18579172122,5sleep11857965-21ntpq10:45:221
18461402120,8sleep21846149-21ssh10:39:422
18452072125,2sleep10-21swapper/110:38:051
18396252121,7sleep01840687-21cat10:35:190
17939392121,6sleep11713297-21diskmemload10:03:211
17581662121,7sleep11713297-21diskmemload09:40:161
17267562121,6sleep00-21swapper/009:17:540
17191162121,3sleep00-21swapper/009:12:050
16026092120,1sleep10-21swapper/107:20:001
19990702111,4sleep11713297-21diskmemload12:15:471
19906972111,4sleep01713297-21diskmemload12:11:280
19835872110,1sleep20-21swapper/212:08:212
19835872110,1sleep20-21swapper/212:08:212
19117822111,3sleep20-21swapper/211:22:152
19042152111,3sleep00-21swapper/011:17:370
18984772111,6sleep11898494-21apt11:15:181
18896702111,4sleep11713297-21diskmemload11:08:031
18883622111,3sleep20-21swapper/211:05:442
18762262111,4sleep01713297-21diskmemload11:00:120
18606032110,1sleep00-21swapper/010:49:020
18144182111,6sleep21814529-21timerandwakeup10:15:292
18086062111,7sleep20-21swapper/210:12:442
18082462110,7sleep01713297-21diskmemload10:12:070
18018292111,4sleep01713297-21diskmemload10:09:300
17726542111,6sleep21772697-21basename09:50:002
17715052111,3sleep10-21swapper/109:47:511
17630402111,3sleep00-21swapper/009:41:070
17402632111,4sleep11713297-21diskmemload09:25:491
17393872111,3sleep20-21swapper/209:25:222
17336032111,3sleep00-21swapper/009:22:030
17141522111,6sleep11715236-21kthreadcore09:10:191
17021682110,7sleep21702224-21ntp_kernel_err08:55:212
16738712111,6sleep01673986-21kthreadcore08:30:170
16650312111,6sleep11666819-21kthreadcore08:20:201
16468052110,2sleep21646843-21ntpq08:00:222
16069302110,1sleep10-21swapper/107:20:291
20342382101,2sleep20-21swapper/212:36:282
20340112101,2sleep10-21swapper/112:36:111
20280672101,3sleep20-21swapper/212:34:312
20207502100,6sleep12021785-21kthreadcore12:30:171
19896712101,4sleep11989903-21users12:10:281
19732362101,3sleep21713297-21diskmemload12:01:112
19657412101,4sleep00-21swapper/011:57:450
19576392101,3sleep01713297-21diskmemload11:53:200
19569442101,3sleep20-21swapper/211:52:172
19484582100,1sleep20-21swapper/211:45:332
19453532100,6sleep11947026-21tr11:45:201
19438972101,4sleep01943917-21apt-get11:45:140
19426742100,3sleep10-21swapper/111:43:351
19361072101,3sleep01713297-21diskmemload11:40:010
19263772101,3sleep20-21swapper/211:31:172
19207102101,3sleep00-21swapper/011:29:400
18804152100,3sleep10-21swapper/111:00:291
18747422101,4sleep11713297-21diskmemload10:57:571
18737402101,2sleep00-21swapper/010:56:030
18600222101,3sleep20-21swapper/210:47:442
18522702101,2sleep10-21swapper/110:43:071
18375002101,2sleep20-21swapper/210:32:242
18371512101,3sleep01713297-21diskmemload10:31:470
18251362101,6sleep01826726-21kthreadcore10:25:210
18209742101,4sleep21821027-21ntpq10:20:222
18173992101,3sleep11713297-21diskmemload10:20:181
18101722101,5sleep01810185-21apt-get10:15:150
18001562101,4sleep21713297-21diskmemload10:06:202
17950492101,3sleep10-21swapper/110:05:141
17879512101,5sleep21788101-21df_inode10:00:182
17628772101,3sleep20-21swapper/209:40:492
17570272101,2sleep00-21swapper/009:38:480
17484662101,3sleep10-21swapper/109:32:021
17185842101,3sleep20-21swapper/209:11:082
17123332101,5sleep01712387-21ntpq09:05:240
17032102101,4sleep11703282-21munin-run09:00:011
16684972101,6sleep21668506-21acpi08:25:152
16179522100,1sleep10-21swapper/107:35:151
15982022101,4sleep21598273-21irqstats07:15:182
1934725291,3sleep00-21swapper/011:37:450
1934376291,2sleep10-21swapper/111:37:101
1928745291,3sleep20-21swapper/211:35:152
1920456290,2sleep10-21swapper/111:29:131
1913111290,2sleep00-21swapper/011:24:220
1897257291,2sleep20-21swapper/211:13:372
1895133291,5sleep01713297-21diskmemload11:10:270
1881538290,2sleep20-21swapper/211:01:522
1861506291,4sleep11861516-21ssh10:50:081
1850737291,4sleep21850885-21cut10:40:272
1829639291,2sleep10-21swapper/110:26:341
1828196291,2sleep20-21swapper/210:25:232
1816332291,2sleep10-21swapper/110:19:171
1808077290,2sleep10-21swapper/110:11:471
1793180290,4sleep01713297-21diskmemload10:01:560
1785207291,3sleep00-21swapper/009:55:350
1780568290,3sleep11713297-21diskmemload09:55:111
1770833291,3sleep01713297-21diskmemload09:46:450
1749212290,1sleep20-21swapper/209:33:202
1740765291,3sleep00-21swapper/009:26:410
1727037291,2sleep10-21swapper/109:18:361
1663222291,4sleep11663253-21munin-run08:20:001
1647243291,4sleep11647460-21users08:00:291
1616740290,1sleep10-21swapper/107:30:221
1867031281,2sleep00-21swapper/010:51:590
1785862280,2sleep20-21swapper/209:56:582
1779353281,2sleep00-21swapper/009:53:300
1750071280,1sleep00-21swapper/009:34:500
1733662281,2sleep20-21swapper/209:22:102
1726882281,3sleep21726893-21ssh09:18:172
1708691281,3sleep11708900-21hddtemp_smartct09:05:171
1682977280,1sleep10-21swapper/108:36:401
1677213280,5sleep11677261-21ntpq08:30:221
1642135281,4sleep01642364-21awk07:55:260
1638117281,2sleep20-21swapper/207:55:122
1623135282,3sleep21623228-21tr07:40:132
1622570280,1sleep20-21swapper/207:36:532
1612915281,3sleep01613065-21df_inode07:30:140
1611652281,3sleep10-21swapper/107:25:211
1596220281,4sleep01597651-21cut07:15:020
1595500280,2sleep10-21swapper/107:13:431
1594650280,4sleep01594704-21ntpq07:10:260
1848017271,2sleep01849695-21kthreadcore10:40:210
1703962271,2sleep00-21swapper/009:00:190
1703962271,2sleep00-21swapper/009:00:190
1672886270,1sleep10-21swapper/108:26:061
1663973271,3sleep01665027-21kthreadcore08:20:190
1662874270,1sleep00-21swapper/008:15:390
1647830271,3sleep0703-21dbus-daemon08:05:000
1641706271,3sleep11641809-21ld07:55:191
1636819270,2sleep00-21swapper/007:50:260
1623113270,2sleep10-21swapper/107:40:131
1617930270,2sleep00-21swapper/007:35:140
1608281271,3sleep00-21swapper/007:25:160
1608048270,4sleep21608072-21fschecks_count07:25:142
1713146260,1sleep20-21swapper/209:08:222
1708184260,1sleep20-21swapper/209:04:192
1708184260,1sleep20-21swapper/209:04:182
1705127261,2sleep10-21swapper/109:00:211
1705127261,2sleep10-21swapper/109:00:201
1695109261,2sleep20-21swapper/208:50:192
1693963261,2sleep10-21swapper/108:50:181
1692948260,1sleep00-21swapper/008:46:200
1673447260,3sleep20-21swapper/208:30:032
1670090261,2sleep00-21swapper/008:25:210
1659015261,2sleep20-21swapper/208:15:202
1653955261,2sleep00-21swapper/008:10:170
1648827261,2sleep20-21swapper/208:05:182
1633601261,2sleep10-21swapper/107:50:231
1633307261,2sleep20-21swapper/207:50:192
1628170260,2sleep20-21swapper/207:45:142
1693018250,3sleep10-21swapper/108:47:141
1688238250,3sleep10-21swapper/108:44:531
1685845250,1sleep00-21swapper/008:40:190
1678877250,2sleep20-21swapper/208:35:212
1657283250,2sleep10-21swapper/108:10:221
1657065250,2sleep20-21swapper/208:10:192
1652075251,1sleep10-21swapper/108:05:211
1628362250,2sleep10-21swapper/107:45:171
1623107250,1sleep00-21swapper/007:40:080
1594518250,2sleep20-21swapper/207:10:242
1698512240,1sleep00-21swapper/008:55:140
1693391240,1sleep00-21swapper/008:50:010
1663272240,1sleep20-21swapper/208:20:012
1648357240,1sleep00-21swapper/008:05:040
1627908240,1sleep00-21swapper/007:45:000
1613169240,1sleep20-21swapper/207:30:152
1602887240,2sleep00-21swapper/007:20:070
1602682240,1sleep20-21swapper/207:20:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional