You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 07:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Fri Mar 13, 2026 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
137039299473454,16cyclictest1386690-21turbostat19:54:593
137039299473454,16cyclictest1386690-21turbostat19:54:593
137039299466457,6cyclictest1452542-21turbostat23:00:003
137039299466457,6cyclictest1452542-21turbostat23:00:003
137038799466446,17cyclictest1449072-21turbostat22:54:552
137039299465455,7cyclictest1374572-21turbostat19:20:003
137039299463455,4cyclictest1445622-21turbostat22:40:003
137039299463442,17cyclictest1372052-21turbostat19:15:003
137038799462453,6cyclictest1479124-21turbostat00:19:542
137039299461454,4cyclictest1441859-21turbostat22:30:003
137039299460453,4cyclictest1424255-21turbostat21:40:013
137038799458434,21cyclictest1480853-21turbostat00:24:552
137039299457450,4cyclictest1395648-21turbostat20:19:593
137039299457449,5cyclictest1420789-21turbostat21:30:003
137039299457435,19cyclictest1468420-21turbostat23:49:563
137039299453446,4cyclictest1404612-21turbostat20:45:003
137039299453445,5cyclictest1399414-21turbostat20:30:013
137039299445438,4cyclictest1436658-21turbostat22:15:003
137039299444436,5cyclictest1406645-21turbostat20:50:003
137038799444437,4cyclictest1393920-21turbostat20:19:552
137039299442433,6cyclictest1379775-21turbostat19:39:553
137039299441434,4cyclictest1484324-21turbostat00:30:003
137039299439432,4cyclictest1397692-21turbostat20:25:003
137039299439430,6cyclictest1440115-21turbostat22:25:003
137039299438431,4cyclictest1458033-21turbostat23:19:553
137039299438428,6cyclictest1388424-21turbostat19:59:593
137039299437430,4cyclictest1473921-21turbostat00:00:013
137039299436415,18cyclictest1378038-21turbostat19:30:003
137039299434427,4cyclictest1393920-21turbostat20:15:003
137039299434427,4cyclictest1393920-21turbostat20:15:003
137039299434426,5cyclictest1450817-21turbostat22:55:003
137039299433425,5cyclictest1415601-21turbostat21:19:553
137039299433425,5cyclictest1378038-21turbostat19:34:553
137039299432410,19cyclictest1412139-21turbostat21:09:553
137039299431423,5cyclictest1447352-21turbostat22:45:003
137039299427420,4cyclictest1410403-21turbostat21:00:003
137039299427419,5cyclictest1432910-21turbostat22:09:553
137039299427419,5cyclictest1383236-21turbostat19:49:553
137039299424405,16cyclictest1466684-21turbostat23:44:553
137039299423416,4cyclictest1415601-21turbostat21:15:013
137039299423405,15cyclictest1464962-21turbostat23:39:553
137038799423416,4cyclictest1376308-21turbostat19:29:562
137039299422412,6cyclictest1402881-21turbostat20:40:003
137039299422412,6cyclictest1402881-21turbostat20:40:003
137038299422414,5cyclictest1440115-21turbostat22:29:550
137039299421413,5cyclictest1456306-21turbostat23:10:003
137039299419399,17cyclictest1486055-21turbostat00:35:003
137039299419399,17cyclictest1486055-21turbostat00:35:003
137038299419412,4cyclictest1436658-21turbostat22:19:550
137039299417408,6cyclictest1482588-21turbostat00:25:013
137039299416408,5cyclictest1443593-21turbostat22:35:003
137038799415408,4cyclictest1370262-21turbostat19:14:552
137039299414407,4cyclictest1470155-21turbostat23:54:553
137039299408399,6cyclictest1463219-21turbostat23:30:003
137039299407400,4cyclictest1388424-21turbostat20:04:553
137039299407399,5cyclictest1419066-21turbostat21:25:003
137038799406380,23cyclictest1424255-21turbostat21:44:562
137039299404385,16cyclictest1477395-21turbostat00:14:553
137039299404385,16cyclictest1412139-21turbostat21:05:003
137039299403395,5cyclictest1459771-21turbostat23:24:563
137039299401393,5cyclictest1431175-21turbostat22:04:553
137039299397390,4cyclictest1463219-21turbostat23:34:553
137039299395388,4cyclictest1406645-21turbostat20:54:553
137039299394386,5cyclictest1429447-21turbostat21:55:003
137038799390382,5cyclictest1390463-21turbostat20:09:562
137039299389381,5cyclictest1458033-21turbostat23:15:003
137038799387378,6cyclictest1454271-21turbostat23:09:552
137038799382361,18cyclictest1427701-21turbostat21:54:542
137038799376368,5cyclictest1461498-21turbostat23:29:542
137039299375367,5cyclictest1475664-21turbostat00:05:003
137038799375367,5cyclictest1404612-21turbostat20:49:552
137039299374368,4cyclictest1422510-21turbostat21:35:003
137038799373351,19cyclictest1486055-21turbostat00:39:552
137039299369362,4cyclictest1399414-21turbostat20:34:563
137038299369362,4cyclictest1475664-21turbostat00:09:550
137039299364358,4cyclictest1427701-21turbostat21:50:003
137039299364358,4cyclictest1427701-21turbostat21:49:593
137038299360352,5cyclictest1434936-21turbostat22:14:560
137038299359351,5cyclictest1450817-21turbostat22:59:560
137038299359351,5cyclictest1450817-21turbostat22:59:560
137039299357351,3cyclictest1383236-21turbostat19:45:013
137038799340333,4cyclictest1420789-21turbostat21:34:542
137039299339333,3cyclictest1449072-21turbostat22:50:003
137038799334327,4cyclictest1410403-21turbostat21:04:552
137038799333326,4cyclictest1395648-21turbostat20:24:542
137039299332325,4cyclictest1487783-21turbostat00:40:003
137038799331325,3cyclictest1397692-21turbostat20:29:552
137038799331324,4cyclictest1471882-21turbostat23:59:562
137038799328321,4cyclictest1413863-21turbostat21:14:552
137039299327319,5cyclictest1452542-21turbostat23:04:553
137038799326318,5cyclictest1484324-21turbostat00:34:552
137038799326318,5cyclictest1484324-21turbostat00:34:552
137038799322311,8cyclictest1429447-21turbostat21:59:552
137039299320313,4cyclictest1425985-21turbostat21:45:003
137039299315309,3cyclictest1431175-21turbostat22:00:003
137039299312306,3cyclictest1480853-21turbostat00:20:013
137038799310304,3cyclictest1447352-21turbostat22:49:562
137038799305297,5cyclictest1438394-21turbostat22:24:552
137039299252238,10cyclictest1438394-21turbostat22:20:003
137039299247233,10cyclictest1376308-21turbostat19:25:003
137039299238212,22cyclictest1392196-21turbostat20:10:013
137039299232207,21cyclictest1477395-21turbostat00:10:003
1369920213969,23sleep20-21swapper/219:05:512
1973991200,98rtkit-daemon0-21swapper/019:07:540
1370110211178,23sleep30-21swapper/319:08:163
1370053210875,23sleep10-21swapper/119:07:311
147638121010,6sleep21475903-21/usr/sbin/munin00:05:212
1370387999588,4cyclictest1374572-21turbostat19:24:552
14785332790,4sleep00-21swapper/000:10:290
137038299700,65cyclictest1972-21rtkit-daemon20:36:570
137038299700,65cyclictest1972-21rtkit-daemon20:36:560
197399690,9rtkit-daemon1972-21rtkit-daemon20:23:160
137038299680,63cyclictest1972-21rtkit-daemon21:02:240
197399670,9rtkit-daemon1972-21rtkit-daemon23:22:280
14078892650,5chrt0-21swapper/220:50:302
197399640,9rtkit-daemon1972-21rtkit-daemon23:11:260
197399640,9rtkit-daemon1972-21rtkit-daemon22:51:090
197399640,9rtkit-daemon1972-21rtkit-daemon20:44:240
197399620,9rtkit-daemon1972-21rtkit-daemon23:16:030
197399600,8rtkit-daemon1972-21rtkit-daemon22:46:250
1370384995616,36cyclictest1376260-21seq19:24:381
1370384995522,29cyclictest1425976-21latency_hist21:45:031
1370384995522,29cyclictest1425976-21latency_hist21:45:031
1370384995517,33cyclictest1463385-21ls23:30:011
14179582530,5sleep20-21swapper/221:20:152
14049642530,5sleep24497-21PK-Backend20:45:012
1370384995317,31cyclictest1427972-21awk21:50:111
1370384995312,11cyclictest1472817-21ls23:55:191
137038499529,13cyclictest1440494-21cat22:25:141
1370384995219,29cyclictest1426080-21sed21:45:021
1370384995214,11cyclictest1468893-21cstates23:45:131
1370384995212,11cyclictest1442419-21cat22:30:151
1370384995116,31cyclictest1407311-21idleruntime20:50:191
1370384995112,10cyclictest1388815-21sed20:00:021
1370384995111,11cyclictest1396112-21gdbus20:20:021
1370384995018,28cyclictest4497-21PK-Backend00:01:291
1370384995017,29cyclictest4497-21PK-Backend23:01:401
1370384995015,31cyclictest4497-21PK-Backend21:18:021
1370384995015,30cyclictest4497-21PK-Backend22:06:111
137038499499,11cyclictest1385226-21sh19:50:111
137038499499,11cyclictest1385226-21sh19:50:111
137038499492,16cyclictest1445126-21timerandwakeup22:35:301
1370384994916,29cyclictest4497-21PK-Backend22:40:471
1370384994915,7cyclictest1377160-21missed_timers19:25:231
1370384994915,30cyclictest4497-21PK-Backend23:09:061
1370384994914,9cyclictest1449120-21ls22:50:011
1370384994914,7cyclictest1471873-21sh23:55:001
1370384994912,10cyclictest1373240-21tr19:15:131
137038499489,10cyclictest1463865-21sh23:30:151
137038499487,12cyclictest1378635-21cat19:30:151
1370384994816,29cyclictest4497-21PK-Backend23:15:331
1370384994815,30cyclictest4497-21PK-Backend20:07:011
1370384994813,7cyclictest1483511-21needreboot00:25:261
1370384994812,6cyclictest1404920-21cat20:45:011
1370384994811,11cyclictest1370781-21ls19:10:131
1370384994714,7cyclictest1137-21runrttasks21:11:231
1370384994713,8cyclictest1405818-21netstat20:45:231
1370384994710,12cyclictest1452635-21cat23:00:021
1370384994710,12cyclictest1452635-21cat23:00:011
1370384994710,10cyclictest1397979-21cpu20:25:131
137038499466,11cyclictest1438943-21cat22:20:131
137038499465,11cyclictest1400699-21timerwakeupswit20:30:301
137038499462,15cyclictest1395268-21users20:15:321
1370384994616,26cyclictest1467829-21python323:40:281
137038499454,12cyclictest1481929-21processes00:20:251
1370384994515,8cyclictest1411478-21processes21:00:271
1370384994511,7cyclictest1436976-21cpu22:15:111
1370384994511,6cyclictest1487951-21latency_hist00:40:011
137038499449,6cyclictest1479989-21missed_timers00:15:211
137038499442,16cyclictest1410090-40lsb_release20:56:281
137038499442,13cyclictest1388499-21latency_hist20:00:021
137038499442,13cyclictest1384934-21seq19:49:531
1370384994412,5cyclictest14857782chrt00:31:441
1370384994412,5cyclictest14857782chrt00:31:441
137038499431,13cyclictest1477357-21taskset00:09:481
137038499431,13cyclictest1436470-21taskset22:12:451
137038499431,12cyclictest1401770-21cat20:35:161
137038499431,12cyclictest1401770-21cat20:35:161
137038499421,4cyclictest4497-21PK-Backend21:59:591
1370384994211,7cyclictest4497-21PK-Backend21:06:251
137038499421,12cyclictest1460593-21cat23:20:211
137038299421,15cyclictest13782792/usr/sbin/munin19:30:190
1370387994123,6cyclictest34-21ksoftirqd/223:41:372
1370387994122,6cyclictest34-21ksoftirqd/221:45:382
1370387994122,6cyclictest34-21ksoftirqd/221:45:382
1370384994111,6cyclictest14789872chrt00:13:291
137038499411,12cyclictest1466439-21chrt23:37:021
137038499411,12cyclictest1424029-21chrt21:37:241
1370382994126,11cyclictest1445007-21python322:35:280
1370387994015,21cyclictest1461523-21unin-run23:25:002
137038499402,11cyclictest1381409-21chrt19:39:041
137038499401,35cyclictest4497-21PK-Backend21:33:241
137038499401,12cyclictest1393798-21chrt20:13:421
137038499401,12cyclictest1393798-21chrt20:13:411
137038499401,11cyclictest1432864-21chrt22:04:461
137038499401,11cyclictest1418804-21chrt21:22:001
1370384994010,6cyclictest14579692chrt23:14:251
1370382994024,11cyclictest1487767-21cron00:40:000
137038799396,11cyclictest34-21ksoftirqd/220:00:542
1370387993921,6cyclictest34-21ksoftirqd/222:26:162
137038499397,8cyclictest1449334-21acpi22:50:111
137038499391,14cyclictest1420596-21chrt21:27:471
197399380,26rtkit-daemon0-21swapper/020:14:100
197399380,26rtkit-daemon0-21swapper/020:14:100
137038299381,12cyclictest1378050-21awk19:30:000
1370387993728,6cyclictest1401143-21turbostat.cron20:35:012
1370387993720,13cyclictest1459398-21seq23:15:312
1370387993714,19cyclictest1416170-21expr21:15:132
1370387993710,6cyclictest14779312cat00:10:142
137038499371,12cyclictest1137-21runrttasks19:41:341
1370382993726,7cyclictest1370772-21grep19:10:130
1370382993723,10cyclictest13-21ksoftirqd/023:05:010
1370382993722,11cyclictest1473687-21seq23:57:210
1370382993722,11cyclictest1419042-21cron21:25:000
1370382993720,12cyclictest1394237-21awk20:15:140
1370387993622,10cyclictest1434412-21expr22:05:282
1370387993622,10cyclictest1379752-21cron19:34:592
1370387993621,11cyclictest1475438-21sshd00:02:212
1370387993620,12cyclictest1420031-21ntp_kernel_err21:25:242
1370387993620,11cyclictest1451317-21expr22:55:142
1370387993620,11cyclictest1451317-21expr22:55:142
1370387993619,12cyclictest1-21systemd19:15:072
1370387993615,17cyclictest1409939-21timerwakeupswit20:55:302
1370382993625,8cyclictest1482580-21ls00:25:010
1370382993620,12cyclictest923-21dbus-daemon19:17:320
1370382993620,11cyclictest1439369-21awk22:20:220
137038299361,8cyclictest1399414-21turbostat20:30:010
137038799357,7cyclictest1464178-21netstat23:30:232
1370387993521,10cyclictest34-21ksoftirqd/222:10:142
1370387993521,10cyclictest34-21ksoftirqd/220:10:232
1370387993521,10cyclictest34-21ksoftirqd/220:10:232
1370387993521,10cyclictest1385899-21needreboot19:50:222
1370387993521,10cyclictest1385899-21needreboot19:50:212
1370387993520,11cyclictest1444491-21diskstats22:35:162
1370387993520,11cyclictest1423946-21sshd21:36:222
1370387993519,12cyclictest1432292-21rt-features22:00:262
1370387993518,12cyclictest1387378-21ls19:55:192
137038299357,11cyclictest13-21ksoftirqd/022:05:580
1370382993525,6cyclictest1429444-21perf21:55:000
1370382993525,6cyclictest1376324-21idleruntime-cro19:25:000
1370382993521,10cyclictest1-21systemd21:45:000
1370382993521,10cyclictest1-21systemd19:45:010
1370382993520,11cyclictest1430754-21wc21:55:300
1370382993519,12cyclictest923-21dbus-daemon22:35:010
1370382993518,13cyclictest1410103-40lsb_release20:56:310
1370382993518,12cyclictest1467585-21cat23:40:210
1370382993518,12cyclictest1400164-21pool-udisksd20:30:210
1370382993518,12cyclictest1385391-21grep19:50:130
1370382993518,12cyclictest1385391-21grep19:50:120
137038799347,6cyclictest141rcu_preempt22:18:222
1370387993424,7cyclictest1452989-21cpuspeed_turbos23:00:142
1370387993419,11cyclictest1470135-21cron23:50:012
1370387993419,11cyclictest1447290-21seq22:44:282
1370387993419,11cyclictest1413937-21wc21:10:012
1370387993419,10cyclictest1465516-21expr23:35:142
1370387993418,11cyclictest438-20systemd-journal19:48:202
1370387993418,11cyclictest1382338-21cat19:40:222
1370387993415,15cyclictest1401500-21ls20:35:152
1370387993415,15cyclictest1401500-21ls20:35:142
137038299346,11cyclictest13-21ksoftirqd/021:07:180
1370382993423,7cyclictest1485114-21ls00:30:210
1370382993423,7cyclictest1485114-21ls00:30:210
1370382993423,7cyclictest1454770-21/usr/sbin/munin23:05:290
1370382993423,6cyclictest13-21ksoftirqd/019:39:160
1370382993421,9cyclictest1390450-21grep20:05:000
1370382993420,10cyclictest1480847-21grep00:20:010
1370382993418,12cyclictest1483251-21awk00:25:210
1370382993417,12cyclictest1471522-21seq23:50:380
137038799336,10cyclictest34-21ksoftirqd/223:45:162
1370387993319,10cyclictest1381486-21cron19:39:592
1370387993318,11cyclictest1457510-21cat23:10:272
137038299337,5cyclictest141rcu_preempt21:15:100
1370382993323,6cyclictest161rcuc/023:45:310
1370382993323,6cyclictest1408658-21latency_hist20:55:020
1370382993323,5cyclictest13-21ksoftirqd/023:37:050
1370382993323,5cyclictest13-21ksoftirqd/000:03:290
1370382993319,10cyclictest1426330-21ls21:45:130
1370382993319,10cyclictest1426330-21ls21:45:120
1370382993318,11cyclictest923-21dbus-daemon23:34:300
1370382993318,10cyclictest1446200-21cat22:40:140
1370382993317,12cyclictest923-21dbus-daemon20:50:000
1370382993317,11cyclictest1414529-21hddtemp_smartct21:10:200
1370382993317,11cyclictest1391358-21cat20:05:230
14427322320,5sleep2331rcuc/222:30:222
1370387993218,10cyclictest1483026-21tr00:25:182
1370382993222,5cyclictest13-21ksoftirqd/019:56:020
1370382993217,10cyclictest1461980-21cat23:25:150
1370382993216,11cyclictest1431710-21expr22:00:130
1370382993216,11cyclictest1383670-21cut19:45:120
1370382993215,13cyclictest1420405-21users21:25:320
137038299311,8cyclictest1424255-21turbostat21:40:000
1370382993115,12cyclictest1421799-21ntp_states21:30:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional