You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-22 - 22:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Dec 22, 2025 12:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
141795923470,17sleep3421ksoftirqd/307:05:193
151189821270,7sleep04497-21PK-Backend11:30:290
14486652940,6sleep20-21swapper/208:30:292
14938462920,7sleep00-21swapper/010:40:220
14427742800,5sleep14497-21PK-Backend08:15:151
141862827252,14sleep10-21swapper/107:07:031
141858726344,14sleep00-21swapper/007:06:300
197399620,9rtkit-daemon1972-21rtkit-daemon10:28:450
197399620,10rtkit-daemon1972-21rtkit-daemon08:25:351
141867626243,13sleep20-21swapper/207:07:402
197399610,9rtkit-daemon1972-21rtkit-daemon07:41:260
197399600,9rtkit-daemon1972-21rtkit-daemon10:31:421
197399600,9rtkit-daemon1972-21rtkit-daemon10:29:481
197399600,9rtkit-daemon1972-21rtkit-daemon09:56:441
197399600,9rtkit-daemon1972-21rtkit-daemon09:37:411
197399590,9rtkit-daemon1972-21rtkit-daemon10:45:341
197399580,9rtkit-daemon1972-21rtkit-daemon11:19:281
197399580,8rtkit-daemon1972-21rtkit-daemon11:15:130
1418992995819,13cyclictest438-20systemd-journal10:35:100
197399570,9rtkit-daemon1972-21rtkit-daemon11:20:561
197399570,8rtkit-daemon1972-21rtkit-daemon12:37:551
197399560,9rtkit-daemon1972-21rtkit-daemon11:10:351
197399560,9rtkit-daemon1972-21rtkit-daemon09:00:520
197399560,9rtkit-daemon1972-21rtkit-daemon07:13:291
197399540,9rtkit-daemon1972-21rtkit-daemon09:28:000
197399540,9rtkit-daemon1972-21rtkit-daemon09:28:000
197399540,9rtkit-daemon1972-21rtkit-daemon08:51:260
197399540,9rtkit-daemon1972-21rtkit-daemon07:31:371
197399540,8rtkit-daemon1972-21rtkit-daemon12:25:140
197399540,8rtkit-daemon1972-21rtkit-daemon11:26:130
1418992995319,12cyclictest438-20systemd-journal11:10:000
1418992995316,12cyclictest1427773-21grep07:30:290
197399520,8rtkit-daemon1972-21rtkit-daemon07:17:330
1418992995218,10cyclictest1419388-21tr07:10:160
1418992995215,10cyclictest1456735-21cat08:55:160
197399510,37rtkit-daemon4497-21PK-Backend11:50:531
1418992995118,11cyclictest1476187-21cat09:50:190
1418992995017,14cyclictest1475234-21sshd09:47:090
1418992995016,12cyclictest1504406-21df11:10:170
1418992995015,9cyclictest1443361-21grep08:15:270
1418992995013,11cyclictest1451498-21expr08:40:150
197399490,6rtkit-daemon1972-21rtkit-daemon07:56:190
14781952490,5chrt0-21swapper/209:55:252
1418995994931,6cyclictest26-21ksoftirqd/110:35:141
1418992994919,26cyclictest438-20systemd-journal10:24:590
141899299491,7cyclictest1463063-21basename09:15:000
197399480,6rtkit-daemon1972-21rtkit-daemon09:20:560
141899299482,17cyclictest1496571-40lsb_release10:47:010
1418995994729,6cyclictest26-21ksoftirqd/110:40:141
1418992994718,6cyclictest1502034-21latency_hist11:05:030
1418992994716,7cyclictest1473136-21rt-features09:40:270
1418992994711,10cyclictest1430193-21sh07:40:010
197399460,39rtkit-daemon2753-21gmain10:56:570
197399460,37rtkit-daemon1533118-21ntp_states12:35:260
1419002994619,14cyclictest42-21ksoftirqd/310:30:233
1419002994615,8cyclictest42-21ksoftirqd/309:05:013
1418995994628,6cyclictest26-21ksoftirqd/109:50:011
141899299463,20cyclictest1489885-21/usr/sbin/munin10:30:120
141899299462,15cyclictest1507879-21diskstats11:20:160
1418992994616,8cyclictest1137-21runrttasks09:56:570
1418992994614,11cyclictest1447707-21sed08:30:020
1418992994611,10cyclictest1469083-21expr09:30:150
197399450,37rtkit-daemon2212-21gvfs-afc-volume10:51:150
1419002994528,6cyclictest42-21ksoftirqd/310:55:003
1419002994512,7cyclictest14457652awk08:24:593
1418992994518,7cyclictest1137-21runrttasks07:48:000
1418992994516,8cyclictest1137-21runrttasks09:19:360
1418992994516,6cyclictest1445746-21latency_hist08:25:010
1418992994514,11cyclictest1424025-21python307:20:290
1418992994513,9cyclictest1515496-21sed11:40:300
1418992994513,8cyclictest1470961-21idleruntime09:35:170
141899299451,18cyclictest1482954-21cat10:10:140
1419002994426,6cyclictest42-21ksoftirqd/310:45:003
1418992994416,7cyclictest1137-21runrttasks12:07:200
1418992994414,8cyclictest1520067-21if_enp2s011:55:210
1418992994410,11cyclictest1438173-21processes08:00:280
197399430,35rtkit-daemon1418987-21cyclictest08:39:500
141899299432,16cyclictest1531394-21ntp_states12:30:280
1418995994226,12cyclictest1523439-21apt12:10:001
1418992994228,10cyclictest4497-21PK-Backend11:39:420
1418992994220,18cyclictest1448494-21ntp_states08:30:260
141899299421,19cyclictest1435353-21perf07:54:590
141899299421,19cyclictest1435353-21perf07:54:590
1418992994211,11cyclictest1516964-21processes11:45:280
141899299419,12cyclictest2345-21gdbus08:49:560
1418992994115,7cyclictest1523924-21cstates12:10:140
1418992994112,8cyclictest438-20systemd-journal12:17:010
1418992994111,26cyclictest0-21swapper/010:05:260
1419002994023,12cyclictest1529777-21python312:25:283
1418998994022,14cyclictest1444651-21df08:20:162
1418995994024,11cyclictest438-20systemd-journal07:28:091
1418992994014,7cyclictest2069-21gnome-shell10:03:100
197399390,32rtkit-daemon13-21ksoftirqd/010:15:210
141900299392,9cyclictest2254-21gsd-color12:20:433
1419002993924,11cyclictest42-21ksoftirqd/310:00:173
1419002993923,5cyclictest42-21ksoftirqd/310:10:023
1419002993922,13cyclictest1523712-21apt12:10:113
1418998993929,7cyclictest1473739-21cron09:45:012
1418998993922,13cyclictest1434768-21python307:50:282
1418998993922,13cyclictest1434768-21python307:50:282
1418998993922,12cyclictest438-20systemd-journal11:51:092
1418998993920,14cyclictest1531013-21df_inode12:30:192
1418995993921,6cyclictest26-21ksoftirqd/110:50:141
141899599391,34cyclictest1466076-21timerwakeupswit09:20:301
141899299398,7cyclictest0-21swapper/012:20:210
141899299391,17cyclictest1461502-21sed09:10:010
141899299391,17cyclictest1461502-21sed09:10:010
1419002993821,6cyclictest42-21ksoftirqd/307:26:153
1419002993821,12cyclictest438-20systemd-journal09:57:083
1419002993820,6cyclictest42-21ksoftirqd/307:32:263
1418998993822,11cyclictest438-20systemd-journal12:25:112
1418998993821,13cyclictest2069-21gnome-shell07:32:092
1418998993821,12cyclictest1501660-21users11:00:302
1418995993820,14cyclictest1481681-21ntp_states10:05:271
1419002993726,7cyclictest1501060-21irqstats11:00:213
1419002993726,7cyclictest1447490-21cut08:30:003
1419002993726,7cyclictest1428856-21cstates07:35:113
1419002993723,11cyclictest1471218-21strings09:35:203
1419002993721,11cyclictest438-20systemd-journal12:20:003
1419002993720,5cyclictest42-21ksoftirqd/311:07:263
1419002993720,13cyclictest976-21rs:main3
1419002993718,14cyclictest42-21ksoftirqd/309:20:303
1418998993720,13cyclictest1522813-21/usr/sbin/munin12:05:012
1418998993720,13cyclictest1469982-40apt-check09:31:562
1418995993727,6cyclictest26-21ksoftirqd/108:30:141
1418995993721,12cyclictest1445778-21pluginstate08:25:011
1418995993720,6cyclictest26-21ksoftirqd/109:05:021
1418995993720,12cyclictest438-20systemd-journal07:47:091
1418995993717,16cyclictest1510701-21munin-run11:30:011
1418995993715,13cyclictest26-21ksoftirqd/110:15:001
141900299367,7cyclictest15004052cut11:00:013
141900299362,6cyclictest1487893-21munin-run10:25:003
1419002993625,7cyclictest1463694-21df_inode09:15:173
1419002993621,11cyclictest923-21dbus-daemon11:15:013
1419002993621,11cyclictest1484108-21cron10:15:013
1419002993620,12cyclictest1440242-21sshd08:06:093
1419002993620,11cyclictest438-20systemd-journal08:49:083
1419002993619,12cyclictest438-20systemd-journal08:36:093
1419002993619,12cyclictest438-20systemd-journal07:50:013
141899899366,5cyclictest141rcu_preempt11:15:002
1418998993621,11cyclictest923-21dbus-daemon11:45:012
1418998993621,11cyclictest1522797-21cron12:09:562
1418998993620,12cyclictest1441695-21python308:10:292
1418998993619,13cyclictest1486831-21sh10:20:212
1418998993619,12cyclictest438-20systemd-journal10:10:012
1418998993617,14cyclictest1458496-21df09:00:162
141899899361,31cyclictest1533814-21pool-udisksd12:39:562
1418995993625,6cyclictest26-21ksoftirqd/108:40:211
1418995993619,13cyclictest1476483-21ntp_states09:50:241
141899299362,15cyclictest1425288-21hddtemp_smartct07:25:190
1418992993620,4cyclictest4497-21PK-Backend11:54:330
197399350,24rtkit-daemon0-21swapper/307:10:483
141900299358,23cyclictest1421239-21cpuspeed_turbos07:15:063
141900299357,15cyclictest1418987-21cyclictest11:56:003
1419002993519,12cyclictest1469811-21sed09:30:313
1419002993518,13cyclictest42-21ksoftirqd/307:55:283
141899899359,8cyclictest141rcu_preempt10:40:292
1418998993521,10cyclictest1430156-21cron07:39:592
1418998993520,11cyclictest1524582-21python312:10:272
1418998993520,11cyclictest1496682-21sshd10:48:102
1418998993520,11cyclictest1444007-21cron08:20:002
1418998993519,12cyclictest1437327-21sshd08:00:092
1418998993519,11cyclictest1526094-21uname12:15:232
1418998993518,12cyclictest438-20systemd-journal10:51:032
1418998993518,12cyclictest1420922-21sensors_temp07:15:022
1418998993512,6cyclictest15141592latency_hist11:40:012
1418995993525,6cyclictest26-21ksoftirqd/108:00:321
1418995993521,10cyclictest1502029-21apt11:05:011
1418995993517,14cyclictest1479786-21memory10:00:231
141899299352,7cyclictest1439456-21df_inode08:05:160
197399340,27rtkit-daemon0-21swapper/110:17:191
197399340,25rtkit-daemon1972-21rtkit-daemon08:13:500
141900299342,9cyclictest1521937-21memory12:00:233
1419002993425,6cyclictest42-21ksoftirqd/309:00:023
1419002993422,9cyclictest42-21ksoftirqd/311:45:203
1419002993420,10cyclictest42-21ksoftirqd/309:27:013
1419002993420,10cyclictest42-21ksoftirqd/309:27:013
1419002993420,10cyclictest1438005-21needreboot08:00:233
1419002993417,12cyclictest438-20systemd-journal11:23:093
1419002993417,12cyclictest1509986-21ntpq11:25:253
1419002993416,14cyclictest1514794-21df_inode11:40:163
141900299341,13cyclictest1511592-21cat11:30:223
1418998993423,7cyclictest1490075-21cut10:30:142
1418998993421,9cyclictest1423311-21cpuspeed_turbos07:20:152
1418998993420,10cyclictest34-21ksoftirqd/209:00:012
1418998993420,10cyclictest1492552-21sed10:35:282
1418998993419,11cyclictest1519593-21sshd11:55:102
1418998993419,11cyclictest1483257-21sort10:10:212
1418998993419,10cyclictest1484413-21gpgv10:15:032
1418998993417,12cyclictest1474124-21cat09:45:142
1418995993424,6cyclictest26-21ksoftirqd/111:06:251
1418995993424,6cyclictest26-21ksoftirqd/107:40:151
1418995993422,6cyclictest26-21ksoftirqd/108:06:331
1418995993419,11cyclictest26-21ksoftirqd/109:10:221
1418995993419,10cyclictest1520566-21sed11:55:291
141899599341,8cyclictest4497-21PK-Backend07:51:481
141899599341,8cyclictest4497-21PK-Backend07:51:471
1418995993418,12cyclictest976-21rs:main1
1418995993418,12cyclictest1453526-21memory08:45:221
1418995993416,13cyclictest26-21ksoftirqd/107:57:331
141899599341,30cyclictest4497-21PK-Backend12:28:261
1418995993412,7cyclictest141rcu_preempt09:10:001
1418995993412,7cyclictest141rcu_preempt09:10:001
1419002993318,11cyclictest1512876-21grep11:35:173
1419002993318,11cyclictest1484611-21mktemp10:15:053
1419002993317,13cyclictest1476218-21awk09:50:193
1419002993317,12cyclictest1518368-21irqstats11:50:213
1419002993317,12cyclictest1435367-21/usr/sbin/munin07:55:013
1419002993317,12cyclictest1435367-21/usr/sbin/munin07:55:003
141900299331,4cyclictest4497-21PK-Backend11:19:433
141900299331,30cyclictest0-21swapper/312:36:023
141900299331,29cyclictest4497-21PK-Backend07:43:393
141900299331,26cyclictest438-20systemd-journal09:07:083
141900299331,26cyclictest438-20systemd-journal09:07:083
141900299331,13cyclictest1440828-21apt08:10:133
141900299331,12cyclictest1423155-21apt07:20:133
141899899336,11cyclictest34-21ksoftirqd/207:47:362
1418998993325,5cyclictest0-21swapper/208:48:312
1418998993323,5cyclictest34-21ksoftirqd/208:28:092
1418998993322,7cyclictest1479694-21kernelversion10:00:212
1418998993319,11cyclictest1463689-21sh09:15:152
1418998993319,10cyclictest1419762-21sed07:10:222
1418998993318,11cyclictest1527428-21expr12:20:132
1418998993318,11cyclictest1509976-21ntpq11:25:242
1418998993318,11cyclictest1499219-21sh10:55:172
1418998993318,11cyclictest1462640-21grep09:10:292
1418998993318,10cyclictest1454668-21awk08:50:132
1418998993317,12cyclictest1436494-21python307:55:262
141899899331,29cyclictest4497-21PK-Backend10:26:062
141899899331,29cyclictest0-21swapper/211:30:492
1418995993324,5cyclictest26-21ksoftirqd/108:35:321
1418995993323,6cyclictest26-21ksoftirqd/111:42:281
1418995993323,6cyclictest26-21ksoftirqd/107:24:521
1418995993323,5cyclictest26-21ksoftirqd/110:56:251
1418995993322,7cyclictest1466939-21tr09:25:161
1418995993322,7cyclictest1466939-21tr09:25:161
1418995993322,6cyclictest26-21ksoftirqd/111:35:301
1418995993322,6cyclictest26-21ksoftirqd/111:30:131
1418995993322,6cyclictest26-21ksoftirqd/109:40:271
1418995993322,6cyclictest26-21ksoftirqd/107:35:111
1418995993318,11cyclictest26-21ksoftirqd/107:20:001
1418995993318,11cyclictest1468666-21mktemp09:30:041
141899599331,29cyclictest4497-21PK-Backend12:31:151
141899599331,29cyclictest4497-21PK-Backend12:10:381
141899599331,29cyclictest0-21swapper/112:21:341
141899599331,20cyclictest1137-21runrttasks08:58:161
141900299322,9cyclictest1530642-21/usr/sbin/munin12:30:133
1419002993222,6cyclictest42-21ksoftirqd/308:50:143
1419002993220,8cyclictest1496801-21python310:49:413
1419002993219,9cyclictest1443230-21grep08:15:233
1419002993217,11cyclictest1474404-21awk09:45:183
1419002993216,12cyclictest1-21systemd08:35:013
141900299321,28cyclictest0-21swapper/310:36:163
141899899329,6cyclictest141rcu_preempt08:35:222
141899899325,23cyclictest1465199-21tr09:20:152
141899899324,9cyclictest34-21ksoftirqd/211:16:022
1418998993219,9cyclictest1503337-21sed11:05:302
1418998993218,10cyclictest1517753-21cat11:50:032
1418998993218,10cyclictest1467034-21expr09:25:162
1418998993218,10cyclictest1467034-21expr09:25:162
1418998993218,10cyclictest1439856-21open_inodes08:05:262
1418998993218,10cyclictest1425161-21cat07:25:152
1418998993217,11cyclictest1476589-21cut09:50:272
1418998993216,12cyclictest1471229-21netstat09:35:202
141899899321,5cyclictest141rcu_preempt11:20:372
1418998993215,13cyclictest1452629-21seq08:44:402
1418995993222,6cyclictest26-21ksoftirqd/108:14:141
1418995993221,6cyclictest26-21ksoftirqd/112:03:451
1418995993218,10cyclictest1455526-21cut08:50:291
141899299321,28cyclictest0-21swapper/012:04:260
1419002993121,6cyclictest42-21ksoftirqd/309:10:143
141900299311,27cyclictest0-21swapper/312:06:573
141900299311,26cyclictest1489139-21sed10:25:313
1418998993116,11cyclictest1460807-21threads09:05:292
1418998993116,11cyclictest1460807-21threads09:05:282
141899899311,28cyclictest0-21swapper/207:44:252
1418995993121,6cyclictest26-21ksoftirqd/109:20:011
1418995993120,6cyclictest26-21ksoftirqd/112:15:531
1418995993119,8cyclictest1517107-21xargs11:45:321
141900299301,4cyclictest141rcu_preempt09:45:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional