You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-15 - 12:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Sep 15, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
132408622490,9sleep3411rcuc/319:05:143
142136421180,3sleep33056-21PK-Backend23:40:163
142136421180,3sleep33056-21PK-Backend23:40:153
139890021150,8sleep234-21ksoftirqd/222:35:222
140781121120,4sleep00-21swapper/023:00:220
132502121020,5sleep10-21swapper/119:05:291
13873672910,4sleep03056-21PK-Backend22:05:000
132538527639,31sleep20-21swapper/219:09:332
132395227458,9sleep00-21swapper/019:05:110
14355172410,3sleep13056-21PK-Backend00:20:191
132556999413,28cyclictest1346543-21seq20:07:081
14126472400,2sleep20-21swapper/223:15:122
132556999329,11cyclictest1095-21runrttasks00:01:221
132556999315,15cyclictest1095-21runrttasks23:14:221
132556999313,17cyclictest0-21swapper/120:00:591
132556999311,16cyclictest0-21swapper/121:39:271
1325569993019,5cyclictest3056-21PK-Backend19:53:181
1325569993019,5cyclictest3056-21PK-Backend19:53:181
139099290,4rtkit-daemon1389-21rtkit-daemon22:37:251
1325569992911,16cyclictest1333936-21ntp_states19:30:191
132556999286,20cyclictest1383739-21ls21:55:001
13318022270,6sleep013-21ksoftirqd/019:25:140
1325569992710,15cyclictest1349000-21ls20:15:121
139099260,4rtkit-daemon1389-21rtkit-daemon20:01:430
13462562260,4chrt3056-21PK-Backend20:05:223
132556999265,19cyclictest1431107-21unin-run00:10:001
132556999255,18cyclictest1392997-21df22:20:131
132556999254,18cyclictest1438175-21sed00:30:011
132556999251,22cyclictest1403387-21df22:50:131
132556999250,24cyclictest0-21swapper/123:48:151
132556999250,24cyclictest0-21swapper/122:27:511
139099240,4rtkit-daemon1389-21rtkit-daemon22:51:150
139099240,3rtkit-daemon1389-21rtkit-daemon21:37:330
132556999249,13cyclictest1391943-21cat22:15:191
132556999248,14cyclictest1363515-21cat20:55:111
132556999248,14cyclictest1354521-21ls20:30:141
132556999247,15cyclictest1439851-21date00:35:011
132556999247,15cyclictest1337873-21sshd19:42:021
132556999246,16cyclictest1433549-21irqstats00:15:151
132556999245,18cyclictest1396540-21cat22:30:171
1325569992417,5cyclictest1417201-21perf23:30:001
132556999240,5cyclictest1380710-21cstates21:45:121
132556999240,22cyclictest1335939-21ls19:35:211
1325578992320,2cyclictest3056-21PK-Backend20:15:173
1325574992321,1cyclictest0-21swapper/223:22:252
132557499230,21cyclictest1381243-21ntp_states21:45:192
132556999238,13cyclictest1431805-21cat00:10:161
132556999238,12cyclictest1387913-21kernelversion22:05:181
132556999235,16cyclictest1386159-21cat22:00:181
132556999234,17cyclictest1402897-21cut22:50:001
132556999232,17cyclictest1361405-21apt-get20:50:031
132556999231,18cyclictest1347054-21apt20:10:131
132556999231,18cyclictest0-21swapper/123:15:091
132556999230,3cyclictest1330464-21ntp_states19:20:191
1325578992220,1cyclictest0-21swapper/321:08:293
1325578992219,2cyclictest3056-21PK-Backend22:12:033
1325578992213,7cyclictest1387368-21latency_hist22:05:003
132557899220,20cyclictest3056-21PK-Backend23:45:463
132557899220,20cyclictest3056-21PK-Backend22:20:423
1325574992220,1cyclictest0-21swapper/222:15:472
1325574992220,1cyclictest0-21swapper/200:00:112
132557499220,21cyclictest0-21swapper/221:39:032
132557499220,21cyclictest0-21swapper/220:06:502
132556999227,13cyclictest1339098-21ntp_kernel_err19:45:171
132556999224,16cyclictest1424654-21expr23:50:121
132556999224,16cyclictest1407249-21cat23:00:171
132556999222,6cyclictest1384277-21expr21:55:131
1325569992211,9cyclictest1327264-21unin-run19:15:011
132556999221,18cyclictest0-21swapper/119:15:131
1325569992210,11cyclictest1390622-21latency_hist22:15:001
132556999220,20cyclictest1417614-21grep23:30:121
132556499228,12cyclictest1417231-21unin-run23:30:000
1325564992220,1cyclictest0-21swapper/023:07:150
1325564992220,1cyclictest0-21swapper/021:26:570
132556499220,21cyclictest0-21swapper/019:17:470
1325578992119,1cyclictest3056-21PK-Backend23:09:083
1325578992119,1cyclictest3056-21PK-Backend00:05:123
1325578992119,1cyclictest3056-21PK-Backend00:02:013
1325578992119,1cyclictest0-21swapper/321:20:263
1325578992119,1cyclictest0-21swapper/319:51:313
1325578992119,1cyclictest0-21swapper/319:51:313
1325578992119,1cyclictest0-21swapper/319:38:323
1325578992118,2cyclictest3056-21PK-Backend19:17:233
132557899210,20cyclictest0-21swapper/321:26:063
132557899210,20cyclictest0-21swapper/319:45:503
132557899210,20cyclictest0-21swapper/300:35:133
132557899210,1cyclictest3056-21PK-Backend23:20:243
132557899210,19cyclictest3056-21PK-Backend20:45:113
132557899210,19cyclictest0-21swapper/320:20:103
1325574992119,1cyclictest0-21swapper/223:47:472
1325574992119,1cyclictest0-21swapper/223:40:242
1325574992119,1cyclictest0-21swapper/223:40:242
1325574992119,1cyclictest0-21swapper/222:57:122
1325574992119,1cyclictest0-21swapper/222:23:182
1325574992119,1cyclictest0-21swapper/222:12:022
1325574992119,1cyclictest0-21swapper/221:30:252
1325574992119,1cyclictest0-21swapper/221:25:192
1325574992119,1cyclictest0-21swapper/220:26:592
1325574992119,1cyclictest0-21swapper/220:17:522
1325574992119,1cyclictest0-21swapper/219:54:002
1325574992119,1cyclictest0-21swapper/219:53:592
1325574992119,1cyclictest0-21swapper/200:27:222
1325574992119,1cyclictest0-21swapper/200:14:582
1325574992119,1cyclictest0-21swapper/200:06:462
1325574992115,4cyclictest1364628-21sh21:00:012
1325574992112,7cyclictest1360028-21memory20:45:172
1325574992112,7cyclictest1336584-21wc19:40:022
132557499210,20cyclictest0-21swapper/223:31:252
132557499210,20cyclictest0-21swapper/223:14:282
132557499210,20cyclictest0-21swapper/221:52:232
132557499210,20cyclictest0-21swapper/221:16:392
132557499210,20cyclictest0-21swapper/220:42:262
132557499210,20cyclictest0-21swapper/219:56:572
132557499210,20cyclictest0-21swapper/219:27:102
132557499210,20cyclictest0-21swapper/219:24:322
132557499210,20cyclictest0-21swapper/200:35:332
132556999219,10cyclictest1095-21runrttasks23:07:411
132556999218,11cyclictest1419409-21cstates23:35:131
132556999214,15cyclictest1426257-21cpuspeed_turbos23:55:111
132556999213,16cyclictest1360962-21dpkg20:50:001
132556999211,9cyclictest1400421-21python322:40:201
1325569992119,1cyclictest0-21swapper/122:58:191
1325569992119,1cyclictest0-21swapper/120:27:591
132556999211,8cyclictest1356755-21/usr/sbin/munin20:35:211
132556999211,18cyclictest1331984-21latency19:25:161
132556999210,8cyclictest1357724-21/usr/sbin/munin20:40:141
132556999210,19cyclictest3056-21PK-Backend21:31:551
132556999210,19cyclictest3056-21PK-Backend21:25:251
132556999210,19cyclictest3056-21PK-Backend20:23:261
1325564992119,1cyclictest3056-21PK-Backend19:39:160
1325564992119,1cyclictest0-21swapper/022:57:080
1325564992119,1cyclictest0-21swapper/022:42:280
1325564992119,1cyclictest0-21swapper/021:55:570
1325564992119,1cyclictest0-21swapper/020:09:100
1325564992119,1cyclictest0-21swapper/019:23:350
132556499210,20cyclictest0-21swapper/023:56:500
132556499210,20cyclictest0-21swapper/023:32:300
132556499210,20cyclictest0-21swapper/020:17:500
1325578992018,1cyclictest3056-21PK-Backend22:53:213
1325578992018,1cyclictest3056-21PK-Backend20:36:183
1325578992018,1cyclictest0-21swapper/321:18:173
132557899201,17cyclictest1364036-21python320:55:203
132557899200,1cyclictest3056-21PK-Backend22:49:353
132557899200,1cyclictest3056-21PK-Backend00:29:393
132557899200,19cyclictest3056-21PK-Backend23:17:273
132557899200,19cyclictest3056-21PK-Backend23:04:353
132557899200,19cyclictest3056-21PK-Backend22:08:313
132557899200,19cyclictest3056-21PK-Backend20:10:233
132557899200,19cyclictest3056-21PK-Backend00:32:023
132557899200,19cyclictest0-21swapper/321:41:353
132557899200,19cyclictest0-21swapper/319:40:213
132557899200,19cyclictest0-21swapper/300:18:313
132557899200,18cyclictest3056-21PK-Backend22:40:083
1325574992018,1cyclictest0-21swapper/223:00:232
1325574992018,1cyclictest0-21swapper/222:54:472
1325574992018,1cyclictest0-21swapper/221:14:302
1325574992018,1cyclictest0-21swapper/220:01:302
1325574992018,1cyclictest0-21swapper/219:41:282
132557499200,1cyclictest0-21swapper/223:09:412
132557499200,1cyclictest0-21swapper/222:08:172
132557499200,19cyclictest0-21swapper/222:27:042
132557499200,19cyclictest0-21swapper/220:38:272
132557499200,19cyclictest0-21swapper/220:31:322
132557499200,19cyclictest0-21swapper/219:46:082
132556999204,14cyclictest1414116-21cpuspeed_turbos23:20:141
1325569992018,1cyclictest3056-21PK-Backend21:14:501
1325569992018,1cyclictest0-21swapper/121:42:281
132556999201,16cyclictest1420934-21apt23:40:111
132556999201,16cyclictest1420934-21apt23:40:111
132556999200,19cyclictest3056-21PK-Backend21:23:151
1325564992018,1cyclictest3056-21PK-Backend00:35:090
1325564992018,1cyclictest0-21swapper/023:17:230
1325564992018,1cyclictest0-21swapper/022:30:240
1325564992018,1cyclictest0-21swapper/021:52:250
1325564992018,1cyclictest0-21swapper/021:48:550
1325564992018,1cyclictest0-21swapper/020:54:530
132556499200,19cyclictest0-21swapper/022:37:070
132556499200,19cyclictest0-21swapper/022:28:490
132556499200,19cyclictest0-21swapper/022:16:120
132556499200,19cyclictest0-21swapper/021:32:040
132556499200,19cyclictest0-21swapper/021:20:280
132556499200,19cyclictest0-21swapper/020:34:590
132556499200,19cyclictest0-21swapper/020:11:210
132556499200,19cyclictest0-21swapper/019:33:000
132556499200,19cyclictest0-21swapper/019:11:380
132556499200,18cyclictest3056-21PK-Backend20:20:460
132557899193,14cyclictest1380681-21cut21:45:133
1325578991917,1cyclictest3056-21PK-Backend22:18:333
1325578991917,1cyclictest3056-21PK-Backend20:31:593
1325578991917,1cyclictest3056-21PK-Backend20:26:523
1325578991917,1cyclictest0-21swapper/321:10:393
1325578991917,1cyclictest0-21swapper/319:13:383
1325578991916,2cyclictest3056-21PK-Backend22:57:053
1325578991914,3cyclictest1342434-21memory19:55:173
132557899190,1cyclictest3056-21PK-Backend23:31:533
132557899190,1cyclictest3056-21PK-Backend22:39:203
132557899190,1cyclictest3056-21PK-Backend20:54:163
132557899190,1cyclictest3056-21PK-Backend00:14:323
132557899190,1cyclictest0-21swapper/321:00:523
132557899190,18cyclictest3056-21PK-Backend22:29:543
1325574991917,1cyclictest0-21swapper/221:58:522
1325574991917,1cyclictest0-21swapper/221:41:132
1325574991917,1cyclictest0-21swapper/221:21:002
1325574991914,3cyclictest34-21ksoftirqd/219:15:122
132557499190,1cyclictest0-21swapper/222:01:002
132557499190,1cyclictest0-21swapper/220:12:452
132557499190,1cyclictest0-21swapper/219:35:012
132557499190,18cyclictest0-21swapper/221:04:442
132556999194,13cyclictest0-21swapper/100:35:121
1325569991917,1cyclictest3056-21PK-Backend21:02:351
132556999190,1cyclictest3056-21PK-Backend19:55:281
1325564991917,1cyclictest0-21swapper/022:22:540
1325564991917,1cyclictest0-21swapper/022:08:070
1325564991917,1cyclictest0-21swapper/021:40:440
1325564991917,1cyclictest0-21swapper/021:18:180
1325564991917,1cyclictest0-21swapper/020:59:530
1325564991917,1cyclictest0-21swapper/020:37:220
1325564991917,1cyclictest0-21swapper/020:27:290
1325564991917,1cyclictest0-21swapper/019:55:450
1325564991917,1cyclictest0-21swapper/000:24:340
1325564991917,1cyclictest0-21swapper/000:14:010
1325564991912,5cyclictest436-20systemd-journal23:36:370
1325564991912,5cyclictest1437173-21cat00:25:160
132556499190,1cyclictest0-21swapper/023:14:120
132556499190,1cyclictest0-21swapper/021:12:380
132556499190,1cyclictest0-21swapper/021:03:380
139099180,15rtkit-daemon0-21swapper/023:45:070
1325578991816,1cyclictest3056-21PK-Backend21:54:263
1325578991816,1cyclictest3056-21PK-Backend20:44:233
1325578991813,3cyclictest1410533-21apt23:10:103
132557499185,2cyclictest141rcu_preempt00:22:282
1325574991816,1cyclictest0-21swapper/222:45:282
1325574991816,1cyclictest0-21swapper/222:43:522
1325574991816,1cyclictest0-21swapper/220:54:372
1325574991816,1cyclictest0-21swapper/220:22:262
1325574991814,3cyclictest978-21snmpd23:53:252
1325574991813,3cyclictest1438253-21/usr/sbin/munin00:30:142
132556999181,7cyclictest1368313-21sed21:10:011
132556999180,4cyclictest13709662sleep121:15:211
132556499189,8cyclictest436-20systemd-journal19:50:000
1325564991816,1cyclictest0-21swapper/023:43:050
1325564991816,1cyclictest0-21swapper/023:43:050
1325564991816,1cyclictest0-21swapper/022:13:150
1325564991816,1cyclictest0-21swapper/000:04:130
1325564991811,5cyclictest1414529-21sed23:20:190
132556499181,13cyclictest1401291-21apt22:45:110
132556499180,1cyclictest0-21swapper/019:51:130
132556499180,1cyclictest0-21swapper/019:51:120
1325578991715,1cyclictest3056-21PK-Backend23:38:013
1325578991715,1cyclictest3056-21PK-Backend20:00:563
1325578991715,1cyclictest3056-21PK-Backend19:29:203
1325578991711,5cyclictest1375617-21expr21:30:133
132557899170,15cyclictest1435549-21ntp_states00:20:203
132557899170,15cyclictest1377428-21hddtemp_smartct21:35:143
132557499179,6cyclictest1367492-21sensors_temp21:05:192
1325574991715,1cyclictest0-21swapper/219:13:582
132556499179,6cyclictest1434098-21sed00:15:210
1325564991715,1cyclictest0-21swapper/019:44:430
1325564991711,4cyclictest1424582-21grep23:50:120
1325564991710,5cyclictest1430115-21cut00:05:160
1325578991614,1cyclictest3056-21PK-Backend23:53:293
1325578991614,1cyclictest3056-21PK-Backend23:27:203
132557899160,14cyclictest3056-21PK-Backend19:20:123
132557499166,8cyclictest1427632-40updatedb.mlocat00:00:012
1325574991612,2cyclictest34-21ksoftirqd/222:30:212
1325564991611,4cyclictest1358580-21python320:40:240
1325564991611,3cyclictest1438649-21sh00:30:140
1325564991610,5cyclictest1360494-21sed20:45:220
132557899152,11cyclictest1334661-21latency_hist19:35:003
132557899150,1cyclictest3056-21PK-Backend23:58:503
132557499156,6cyclictest436-20systemd-journal23:39:592
132557499155,2cyclictest141rcu_preempt23:25:212
1325574991513,1cyclictest0-21swapper/200:19:582
1325564991510,3cyclictest1367049-21idleruntime21:05:150
1325578991410,3cyclictest3056-21PK-Backend21:55:493
132557899140,13cyclictest3056-21PK-Backend22:31:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional