You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 06:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8s.osadl.org (updated Thu Nov 27, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12101509913450,1341cyclictest14088052sleep321:44:573
12101509912520,1248cyclictest14735032sleep322:24:563
12101509911220,1119cyclictest13143932sleep320:40:013
12101509910581,1052cyclictest1538340-21sleep23:04:593
1210150993950,391cyclictest1595178-21turbostat23:44:553
15703272960,1sleep2421rcuc/223:24:182
1210150992118,1cyclictest0-21swapper/322:05:013
1210150991917,0cyclictest0-21swapper/322:15:003
1210150991916,1cyclictest0-21swapper/300:20:013
1210150991815,1cyclictest0-21swapper/319:45:153
1210150991810,5cyclictest1306614-21sleep20:25:003
190898160,12rtkit-daemon1480844-21ssh22:28:080
16035732161,1sleep30-21swapper/323:45:113
14172122162,4sleep01417249-21apt-get21:50:120
121015099168,5cyclictest1345545-21turbostat21:05:003
121015099168,5cyclictest1345545-21turbostat21:05:003
1210150991613,1cyclictest0-21swapper/323:20:003
121015099161,11cyclictest1538673-21taskset23:05:113
121015099160,12cyclictest1586801-21turbostat23:35:003
190898150,2rtkit-daemon0-21swapper/221:09:132
190898150,2rtkit-daemon0-21swapper/123:35:221
121015099156,7cyclictest1513807-21taskset22:49:323
121015099154,8cyclictest1446762-21taskset22:05:203
121015099151,9cyclictest12942692sleep320:15:133
1210150991513,1cyclictest0-21swapper/321:57:323
1210150991513,1cyclictest0-21swapper/321:54:253
1210150991512,1cyclictest0-21swapper/322:55:523
1210150991512,1cyclictest0-21swapper/321:45:193
1210150991512,1cyclictest0-21swapper/320:49:423
1210150991512,1cyclictest0-21swapper/319:11:003
121015099151,11cyclictest1661880-21taskset00:20:173
121015099151,10cyclictest16359182sleep300:05:113
190898140,7rtkit-daemon1522083-21ssh22:54:570
12829842141,10sleep01284147-21tr20:05:160
121015099146,6cyclictest15873222sleep323:35:053
1210150991413,0cyclictest0-21swapper/319:20:563
1210150991412,1cyclictest0-21swapper/300:35:173
1210150991412,1cyclictest0-21swapper/300:31:293
1210150991412,0cyclictest0-21swapper/319:42:333
121015099141,11cyclictest1224703-21turbostat19:20:013
1210150991411,1cyclictest0-21swapper/323:25:173
1210150991411,1cyclictest0-21swapper/323:21:123
1210150991411,1cyclictest0-21swapper/322:34:553
1210150991411,1cyclictest0-21swapper/321:30:013
1210150991411,1cyclictest0-21swapper/321:15:223
1210150991411,1cyclictest0-21swapper/319:55:123
121015099141,10cyclictest1668613-21taskset00:25:143
121015099141,10cyclictest1644027-21taskset00:10:113
121015099141,10cyclictest1621991-21taskset23:55:163
121015099141,10cyclictest1495284-21taskset22:35:223
121015099141,10cyclictest1463928-21taskset22:16:303
121015099141,10cyclictest1374974-21taskset21:22:363
121015099141,10cyclictest1345768-21taskset21:05:013
121015099141,10cyclictest1237960-21chrt19:30:133
121015099140,11cyclictest1307506-21taskset20:30:073
13058612131,10sleep11305973-21grep20:20:211
121015099131,9cyclictest1552517-21taskset23:10:243
121015099131,9cyclictest1552517-21taskset23:10:243
121015099131,9cyclictest1474884-21taskset22:25:163
121015099131,9cyclictest1392884-21taskset21:35:033
121015099131,9cyclictest1384811-21taskset21:30:103
121015099131,9cyclictest1333221-21taskset20:55:043
121015099131,9cyclictest1306847-21taskset20:25:153
121015099131,9cyclictest1282157-21taskset20:05:153
1210150991311,1cyclictest0-21swapper/322:40:043
121015099131,10cyclictest1352085-21taskset21:10:023
121015099131,10cyclictest1237506-21taskset19:29:573
1210150991311,0cyclictest0-21swapper/319:50:173
1210150991310,1cyclictest0-21swapper/322:50:233
1210150991310,1cyclictest0-21swapper/320:11:323
121015099130,10cyclictest1633615-21taskset00:00:253
190898120,8rtkit-daemon473-21gmain21:30:322
190898120,7rtkit-daemon1256809-21seq19:45:132
190898120,3rtkit-daemon1658749-21ssh00:17:190
12503792120,8sleep01250534-21diskstats19:40:130
12392602121,2sleep20-21swapper/219:30:172
121015099121,9cyclictest1332623-21taskset20:50:273
121015099121,9cyclictest1280627-21taskset20:00:223
121015099121,9cyclictest1244256-21taskset19:35:133
121015099120,9cyclictest1611631-21taskset23:50:033
121015099120,9cyclictest1313749-21taskset20:35:083
190898110,7rtkit-daemon1439347-21ssh22:00:471
190898110,4rtkit-daemon1538605-21ssh23:05:070
190898110,4rtkit-daemon1512866-21ssh22:47:250
190898110,3rtkit-daemon436-21systemd-logind21:11:372
14548122111,7sleep21454870-21ntp_states22:10:212
12630642110,1sleep00-21swapper/019:50:150
190898100,7rtkit-daemon1216482-21munin-node23:31:321
190898100,6rtkit-daemon1333970-21cut20:55:152
190898100,5rtkit-daemon1495424-21python322:35:230
190898100,5rtkit-daemon1287740-21cron20:10:002
190898100,5rtkit-daemon1231680-21diskstats19:25:132
190898100,4rtkit-daemon1609888-21ssh23:46:330
190898100,3rtkit-daemon1216482-21munin-node20:22:432
190898100,2rtkit-daemon431ktimers/221:24:492
190898100,1rtkit-daemon0-21swapper/121:52:581
16264552100,3sleep11626459-21ssh23:57:361
15536982101,6sleep21553706-21ssh23:13:132
15536982101,6sleep21553706-21ssh23:13:122
14962452100,5sleep21496252-21diskmemload22:36:332
14244162101,7sleep21424423-21sh21:53:382
13780632100,7sleep11379224-21grep21:25:161
13765622101,7sleep21376570-21ssh21:25:092
13385612103,5sleep01338757-21/usr/sbin/munin20:55:220
12117502101,6sleep11213075-21grep19:10:171
19089890,5rtkit-daemon1649078-21sed00:10:180
19089890,4rtkit-daemon1498089-21apt-get22:40:100
19089890,3rtkit-daemon667-21snmpd19:09:402
19089890,3rtkit-daemon1928-21sendmail-mta19:14:322
19089890,1rtkit-daemon0-21swapper/122:05:371
1675639291,5sleep11675647-21ssh00:28:251
1643991291,1sleep20-21swapper/200:10:112
1594697290,6sleep01594705-21ssh23:38:550
1578407290,1sleep01578415-21ssh23:29:160
1561012290,1sleep10-21swapper/123:16:361
1538632291,4sleep21538664-21apt-get23:05:112
1522105291,5sleep11522155-21munin-run22:54:591
1352130290,6sleep00-21swapper/021:10:120
1274311291,5sleep01274417-21/usr/sbin/munin19:55:240
1244068291,4sleep11244101-21apt-get19:35:131
19089880,5rtkit-daemon1928-21sendmail-mta22:32:580
19089880,5rtkit-daemon1928-21sendmail-mta22:14:030
19089880,5rtkit-daemon1928-21sendmail-mta20:45:532
19089880,5rtkit-daemon1928-21sendmail-mta00:37:030
19089880,5rtkit-daemon0-21swapper/000:06:060
19089880,4rtkit-daemon667-21snmpd21:45:192
19089880,4rtkit-daemon667-21snmpd20:02:062
19089880,4rtkit-daemon421-21avahi-daemon22:59:350
19089880,4rtkit-daemon1553269-21ssh23:12:091
19089880,4rtkit-daemon1553269-21ssh23:12:091
19089880,4rtkit-daemon1430359-21memory21:55:181
19089880,4rtkit-daemon1313859-21seq20:36:342
19089880,4rtkit-daemon1216482-21munin-node00:28:150
19089880,3rtkit-daemon1351841-21diskmemload23:01:350
19089880,1rtkit-daemon0-21swapper/219:43:152
1684904281,4sleep21684977-21tune2fs00:35:152
1650699281,4sleep11650709-21ssh00:12:301
1625915281,4sleep01625922-21sh23:56:260
1599298280,4sleep11600558-21memory23:40:161
1594616280,6sleep21594624-21ssh23:38:452
1580574280,6sleep01581711-21grep23:30:180
1570505281,1sleep0171rcu_preempt23:24:410
1514276281,4sleep21514319-21ssh22:50:122
1465819281,4sleep01465975-21df_inode22:20:130
1415024281,4sleep01415028-21ssh21:45:380
1408031283,3sleep11408038-21sh21:43:111
1407538280,1sleep20-21swapper/221:42:002
1391496281,4sleep01391506-21ssh21:32:200
1339528281,3sleep11339561-21apt-get21:00:121
1339528281,3sleep11339561-21apt-get21:00:121
1320681281,4sleep01320714-21apt-get20:45:110
1238263280,5sleep11239259-21/usr/sbin/munin19:30:161
1209741281,4sleep01209743-21sshd19:06:510
1204063280,4sleep11204206-21df19:05:121
19089870,4rtkit-daemon81364-21packagekitd23:28:291
19089870,4rtkit-daemon671-21ntpd19:50:212
19089870,4rtkit-daemon667-21snmpd20:54:052
19089870,4rtkit-daemon1351841-21diskmemload22:15:030
19089870,4rtkit-daemon1216482-21munin-node20:27:262
19089870,4rtkit-daemon1216482-21munin-node20:18:432
19089870,4rtkit-daemon1216482-21munin-node19:17:122
19089870,4rtkit-daemon1216482-21munin-node00:30:030
19089870,4rtkit-daemon0-21swapper/221:00:282
19089870,4rtkit-daemon0-21swapper/221:00:282
19089870,3rtkit-daemon81364-21packagekitd00:02:090
19089870,3rtkit-daemon671-21ntpd23:42:450
19089870,3rtkit-daemon667-21snmpd22:09:042
19089870,3rtkit-daemon473-21gmain21:37:412
19089870,3rtkit-daemon1928-21sendmail-mta20:40:282
19089870,3rtkit-daemon1545989-21ssh23:09:041
19089870,3rtkit-daemon1216482-21munin-node23:50:130
19089870,3rtkit-daemon1216482-21munin-node20:31:122
19089870,3rtkit-daemon1216482-21munin-node20:10:562
19089870,3rtkit-daemon1216482-21munin-node00:20:520
1691139270,1sleep10-21swapper/100:36:551
1683046270,4sleep21683054-21ssh00:31:592
1682835270,4sleep11682843-21ssh00:31:261
1637584271,4sleep11639054-21ssh00:05:171
1625732271,3sleep21625740-21ssh23:55:522
1583797270,5sleep21584240-21needreboot23:30:202
1570953270,3sleep21570987-21apt-get23:25:152
1506417270,4sleep21506621-21hddtemp_smartct22:45:152
1503269270,4sleep21503352-21ntp_states22:40:192
1471076271,3sleep11471199-21python322:20:221
1456594270,5sleep11456600-21ssh22:13:201
1448984271,4sleep01448992-21ssh22:09:230
1383676270,4sleep01383684-21ssh21:28:400
1360118270,3sleep21360155-21apt-get21:15:112
1346184270,4sleep01346765-21sed21:05:150
1274530270,4sleep21274774-21users19:55:262
1245523271,4sleep21245560-21kthreadcore19:35:182
1238115271,4sleep01238256-21irqcore19:30:150
1225255270,4sleep21225410-21diskstats19:20:162
1210296271,2sleep01210329-21apt-get19:10:120
19089860,3rtkit-daemon671-21ntpd21:48:591
1674525260,1sleep20-21swapper/200:25:502
1660172262,2sleep10-21swapper/100:20:041
1659409260,1sleep20-21swapper/200:19:002
1652101261,3sleep11652109-21ssh00:15:111
1642959261,3sleep2667-21snmpd00:08:232
1635537260,1sleep20-21swapper/200:04:542
1611185260,1sleep10-21swapper/123:49:381
1559902260,2sleep0221rcuc/023:15:180
1557298261,2sleep20-21swapper/223:15:172
1536926260,1sleep10-21swapper/123:01:541
1529463260,1sleep20-21swapper/222:58:262
1522440260,3sleep11522483-21ssh22:55:121
1503951260,1sleep10-21swapper/122:40:341
1496104260,1sleep10-21swapper/122:36:171
1482141261,2sleep21482206-21fschecks_count22:30:172
1481391260,1sleep11473287-21kworker/u16:0+xprtiod22:29:281
1473531261,2sleep20-21swapper/222:25:002
1457599260,1sleep20-21swapper/222:15:062
1439261260,4sleep00-21swapper/022:00:290
1433625260,4sleep21433707-21fschecks_time22:00:152
1426768260,4sleep01428361-21cut21:55:160
1399069260,1sleep10-21swapper/121:36:101
1399060260,3sleep01399068-21ssh21:36:090
1366108260,1sleep10-21swapper/121:15:501
1363006261,2sleep00-21swapper/021:15:170
1358475260,1sleep10-21swapper/121:11:541
1329108260,3sleep01331200-21cut20:50:170
1321735261,2sleep10-21swapper/120:45:171
1289967261,2sleep10-21swapper/120:10:191
1261999260,1sleep10-21swapper/119:45:231
1256811260,1sleep00-21swapper/019:45:140
1250066260,3sleep01250162-21sed19:40:010
1225218261,2sleep10-21swapper/119:20:151
1224735260,3sleep01224686-21apt-get19:20:020
1224482260,1sleep00-21swapper/019:17:330
1204020260,3sleep30-21swapper/319:05:063
1667184251,2sleep20-21swapper/200:23:052
1634560250,3sleep10-21swapper/100:02:371
1617993250,3sleep21618000-21sh23:51:402
1595469251,2sleep20-21swapper/223:40:092
1568172250,2sleep11568278-21ssh23:20:211
1554381250,3sleep00-21swapper/023:14:470
1554381250,3sleep00-21swapper/023:14:470
1537970250,1sleep20-21swapper/223:04:192
1507865250,3sleep11509818-21cut22:45:191
1480303250,3sleep21480311-21ssh22:26:562
1401279250,3sleep01401405-21gltestperf21:40:140
1386733250,3sleep11388721-21grep21:30:171
1368690250,3sleep11369685-21irqstats21:20:151
1346770250,3sleep11347879-21grep21:05:151
1345378250,1sleep00-21swapper/021:03:070
1345378250,1sleep00-21swapper/021:03:070
1339097250,1sleep10-21swapper/120:58:031
1332651250,1sleep10-21swapper/120:50:501
1320362250,1sleep00-21swapper/020:44:120
1314401250,2sleep10-21swapper/120:40:081
1313824251,2sleep1667-21snmpd20:36:071
1306547250,1sleep00-21swapper/020:24:290
1295725251,2sleep01295762-21kthreadcore20:15:190
1288976251,2sleep00-21swapper/020:10:180
1284149250,3sleep11286263-21kthreadcore20:05:181
1275615250,3sleep01275815-21gltestperf20:00:160
1274990250,1sleep10-21swapper/119:58:051
1263265250,3sleep11263537-21cut19:50:161
1216365251,2sleep10-21swapper/119:15:021
1617924240,3sleep10-21swapper/123:51:301
1608609240,2sleep21608781-21strings23:45:182
1489593240,2sleep11489595-21ssh22:34:531
1464945240,1sleep10-21swapper/122:18:541
1431217240,2sleep20-21swapper/221:55:432
1375643240,3sleep00-21swapper/021:24:150
1314081240,3sleep00-21swapper/020:39:230
1312449240,2sleep10-21swapper/120:30:141
1275816240,2sleep10-21swapper/120:00:171
1237387240,3sleep00-21swapper/019:28:270
1231488240,2sleep10-21swapper/119:25:121
1307657230,1sleep00-21swapper/020:30:090
1306588230,1sleep00-21swapper/020:25:010
1299500230,1sleep10-21swapper/120:15:241
1250535230,1sleep10-21swapper/119:40:141
1306830220,1sleep10-21swapper/120:25:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional