You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-12 - 06:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue May 12, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
146354821210,3sleep30-21swapper/321:30:223
145974421100,4sleep33056-21PK-Backend21:20:193
15178142910,4sleep20-21swapper/200:05:232
139099750,60rtkit-daemon0-21swapper/019:08:060
141244526445,13sleep30-21swapper/319:08:113
141227926344,14sleep10-21swapper/119:06:041
141227526344,13sleep20-21swapper/219:06:012
15190892530,8sleep2331rcuc/200:10:122
1412719993212,5cyclictest1529046-21cat00:40:013
141271399327,2cyclictest14916962timerandwakeup22:50:251
1412709993012,5cyclictest1449362-21sed20:50:240
141270999298,6cyclictest1506892-21sed23:35:160
1412709992913,6cyclictest1424550-21ntp_states19:40:180
139099290,5rtkit-daemon1389-21rtkit-daemon22:03:003
14728682280,13sleep234-21ksoftirqd/222:00:002
1412719992815,4cyclictest978-21snmpd20:36:403
1412709992812,4cyclictest1413227-21expr19:10:120
1412719992714,3cyclictest1506331-21apt23:35:103
141270999278,5cyclictest1436903-21sed20:15:230
1412709992711,4cyclictest1515330-21expr00:00:120
141270999271,12cyclictest1497659-21apt23:10:120
141271999267,4cyclictest0-21swapper/300:10:003
141271999260,20cyclictest3056-21PK-Backend23:58:543
141271999260,20cyclictest3056-21PK-Backend20:16:423
141270999267,6cyclictest1511291-21unin-run23:50:000
1412709992614,5cyclictest1467652-21cron21:45:000
1412709992612,3cyclictest1457880-21missed_timers21:15:180
1412709992610,4cyclictest1524157-21expr00:25:140
139099260,4rtkit-daemon1389-21rtkit-daemon23:45:073
1412719992511,3cyclictest1469660-21/usr/sbin/munin21:50:093
141271999250,15cyclictest1484262-21irqstats22:30:163
141271999250,10cyclictest1488985-21apt22:45:113
141270999259,4cyclictest1434935-21ntpq20:10:190
141270999259,3cyclictest1488169-21switchtime22:40:210
141270999251,21cyclictest1496790-21python323:05:210
1412709992510,3cyclictest1433943-21latency_hist20:10:010
141271999240,23cyclictest0-21swapper/320:46:573
141271999240,1cyclictest0-21swapper/321:59:433
141271999240,19cyclictest3056-21PK-Backend22:59:053
141271699242,20cyclictest1418332-21cron19:24:592
141271399241,10cyclictest1448079-21turbostat.cron20:50:001
141270999249,3cyclictest1459718-21ntp_kernel_pll_21:20:190
141270999249,3cyclictest1095-21runrttasks22:36:110
141270999248,5cyclictest1441121-21cron20:34:550
141270999248,5cyclictest1441121-21cron20:34:550
141270999242,6cyclictest1499391-21apt23:15:110
1412709992411,3cyclictest1446578-21/usr/sbin/munin20:45:190
141270999241,10cyclictest1450070-21apt20:55:120
14563712230,4sleep10-21swapper/121:10:231
1412719992317,5cyclictest0-21swapper/320:05:403
1412716992321,1cyclictest0-21swapper/222:55:282
1412716992321,1cyclictest0-21swapper/220:04:002
141271399230,21cyclictest1525451-21date00:30:001
141270999238,4cyclictest1451995-21cstates21:00:120
141270999230,22cyclictest0-21swapper/021:58:020
139099230,3rtkit-daemon1389-21rtkit-daemon20:29:413
139099230,3rtkit-daemon1389-21rtkit-daemon20:23:182
1412719992215,1cyclictest3056-21PK-Backend19:28:023
141271999220,21cyclictest3056-21PK-Backend23:03:243
1412716992220,1cyclictest0-21swapper/221:42:022
1412716992220,1cyclictest0-21swapper/220:14:282
1412716992220,1cyclictest0-21swapper/219:50:422
1412716992220,1cyclictest0-21swapper/219:29:062
1412713992217,3cyclictest1489847-21python322:45:201
141270999226,6cyclictest1420714-21df_inode19:30:130
141270999226,6cyclictest1420714-21df_inode19:30:120
141270999222,8cyclictest82332-21update-manager21:12:590
1412709992215,6cyclictest1504334-21cron23:30:000
1412709992215,6cyclictest1504334-21cron23:30:000
139099220,19rtkit-daemon1414629-21munin-node19:58:323
1412719992118,2cyclictest3056-21PK-Backend19:50:463
1412719992118,2cyclictest3056-21PK-Backend19:23:353
1412719992117,2cyclictest3056-21PK-Backend21:15:223
141271999211,4cyclictest1466203-21apt21:40:143
141271999210,2cyclictest3056-21PK-Backend19:11:463
141271999210,20cyclictest0-21swapper/323:32:173
141271999210,20cyclictest0-21swapper/320:32:513
141271999210,20cyclictest0-21swapper/320:32:513
141271999210,19cyclictest3056-21PK-Backend23:51:173
1412716992119,1cyclictest0-21swapper/223:19:502
1412716992119,1cyclictest0-21swapper/223:09:422
1412716992119,1cyclictest0-21swapper/222:35:562
1412716992119,1cyclictest0-21swapper/221:27:152
1412716992119,1cyclictest0-21swapper/221:16:542
1412716992119,1cyclictest0-21swapper/220:52:132
1412716992119,1cyclictest0-21swapper/220:06:382
1412716992119,1cyclictest0-21swapper/219:46:362
1412716992119,1cyclictest0-21swapper/219:30:072
1412716992119,1cyclictest0-21swapper/219:30:072
1412716992119,1cyclictest0-21swapper/219:11:352
1412716992119,1cyclictest0-21swapper/200:34:322
1412716992115,4cyclictest34-21ksoftirqd/223:30:002
1412716992115,4cyclictest34-21ksoftirqd/223:30:002
1412716992112,7cyclictest436-20systemd-journal22:45:002
141271699210,2cyclictest0-21swapper/219:57:522
141271699210,20cyclictest0-21swapper/223:46:332
141271699210,20cyclictest0-21swapper/223:13:002
141271699210,20cyclictest0-21swapper/222:28:462
141271699210,20cyclictest0-21swapper/220:56:392
141271399213,16cyclictest1095-21runrttasks20:00:081
1412713992119,1cyclictest0-21swapper/123:05:211
1412713992119,1cyclictest0-21swapper/119:49:321
1412713992113,7cyclictest1465681-21cron21:40:001
1412713992110,9cyclictest1478829-21expr22:15:121
1412713992110,8cyclictest883-21dbus-daemon00:00:011
141271399210,20cyclictest3056-21PK-Backend23:34:421
141271399210,20cyclictest3056-21PK-Backend22:25:541
141271399210,20cyclictest0-21swapper/123:15:151
141271399210,20cyclictest0-21swapper/121:46:331
1412709992116,4cyclictest0-21swapper/022:00:050
1412709992114,5cyclictest436-20systemd-journal23:43:370
1412709992114,1cyclictest0-21swapper/000:07:260
1412709992113,6cyclictest1465067-21python321:35:210
1412709992113,6cyclictest1454196-21grep21:05:180
1412709992113,6cyclictest1430464-21cron20:00:010
1412709992110,3cyclictest1095-21runrttasks23:03:530
141270999210,20cyclictest0-21swapper/000:13:010
141270999210,19cyclictest3056-21PK-Backend21:50:180
1412719992018,1cyclictest3056-21PK-Backend23:16:073
1412719992018,1cyclictest3056-21PK-Backend23:12:513
1412719992018,1cyclictest3056-21PK-Backend20:21:083
1412719992018,1cyclictest3056-21PK-Backend19:31:403
1412719992018,1cyclictest3056-21PK-Backend19:31:403
1412719992018,1cyclictest3056-21PK-Backend00:26:113
1412719992018,1cyclictest0-21swapper/322:39:453
1412719992018,1cyclictest0-21swapper/322:22:483
1412719992018,1cyclictest0-21swapper/322:13:493
1412719992017,2cyclictest3056-21PK-Backend22:51:423
1412719992017,2cyclictest3056-21PK-Backend21:11:443
1412719992012,6cyclictest1520909-21cat00:15:163
141271999201,17cyclictest1515875-21python300:00:223
141271999200,1cyclictest0-21swapper/322:06:403
141271999200,19cyclictest3056-21PK-Backend21:28:073
141271999200,19cyclictest3056-21PK-Backend20:55:363
141271999200,19cyclictest3056-21PK-Backend20:52:583
141271999200,19cyclictest3056-21PK-Backend19:43:503
141271999200,19cyclictest0-21swapper/321:37:123
141271999200,19cyclictest0-21swapper/320:44:203
141271999200,19cyclictest0-21swapper/320:03:033
141271699203,15cyclictest1514766-21unin-run00:00:012
1412716992018,1cyclictest0-21swapper/223:50:032
1412716992018,1cyclictest0-21swapper/222:47:392
1412716992018,1cyclictest0-21swapper/221:24:242
1412716992018,1cyclictest0-21swapper/220:40:582
1412716992018,1cyclictest0-21swapper/220:38:212
1412716992012,6cyclictest436-20systemd-journal22:31:352
141271699200,19cyclictest0-21swapper/223:36:122
141271699200,19cyclictest0-21swapper/223:21:052
141271699200,19cyclictest0-21swapper/221:53:242
141271699200,19cyclictest0-21swapper/221:39:112
141271699200,19cyclictest0-21swapper/221:07:342
141271699200,19cyclictest0-21swapper/220:30:522
141271699200,19cyclictest0-21swapper/220:30:512
141271699200,19cyclictest0-21swapper/200:19:382
1412713992018,1cyclictest0-21swapper/123:10:491
1412713992018,1cyclictest0-21swapper/120:10:411
1412713992015,3cyclictest1455008-21grep21:10:001
1412713992013,5cyclictest436-20systemd-journal19:54:361
1412713992012,6cyclictest1461485-21ntpq21:25:191
141271399200,19cyclictest0-21swapper/123:50:441
141271399200,19cyclictest0-21swapper/119:40:191
141270999202,6cyclictest1491581-21python322:50:230
1412709992018,1cyclictest0-21swapper/022:10:530
1412709992018,1cyclictest0-21swapper/019:53:580
1412709992018,1cyclictest0-21swapper/019:47:020
1412709992018,1cyclictest0-21swapper/000:16:320
141270999201,6cyclictest1415148-21apt19:15:100
1412709992013,5cyclictest1480069-21turbostat22:20:010
141270999200,1cyclictest3056-21PK-Backend20:43:400
141270999200,1cyclictest0-21swapper/023:23:530
141270999200,19cyclictest0-21swapper/022:07:490
141270999200,19cyclictest0-21swapper/021:45:380
141270999200,19cyclictest0-21swapper/021:32:060
1412719991917,1cyclictest3056-21PK-Backend23:08:453
1412719991917,1cyclictest3056-21PK-Backend19:38:093
1412719991917,1cyclictest0-21swapper/323:21:563
1412719991917,1cyclictest0-21swapper/321:05:293
141271999191,6cyclictest1483130-21users22:25:253
1412719991914,4cyclictest411rcuc/300:10:133
1412719991914,3cyclictest1095-21runrttasks23:25:173
1412719991914,3cyclictest1095-21runrttasks23:25:173
141271999190,1cyclictest3056-21PK-Backend19:17:343
141271999190,1cyclictest0-21swapper/322:40:463
1412716991917,1cyclictest0-21swapper/223:42:002
1412716991917,1cyclictest0-21swapper/222:53:132
1412716991917,1cyclictest0-21swapper/220:46:592
1412716991917,1cyclictest0-21swapper/219:44:352
1412716991917,1cyclictest0-21swapper/219:35:482
1412716991917,1cyclictest0-21swapper/200:37:502
1412716991917,1cyclictest0-21swapper/200:01:272
1412716991916,2cyclictest1440746-21users20:25:232
1412716991914,3cyclictest1474768-21latency_hist22:05:012
1412716991911,6cyclictest1504936-21cat23:30:132
1412716991911,6cyclictest1462835-21df21:30:122
141271699190,1cyclictest0-21swapper/221:00:182
141271699190,1cyclictest0-21swapper/220:18:542
141271699190,1cyclictest0-21swapper/200:26:412
141271699190,1cyclictest0-21swapper/200:20:522
141271399193,2cyclictest141rcu_preempt20:29:201
1412713991917,1cyclictest0-21swapper/119:31:201
1412713991917,1cyclictest0-21swapper/119:31:201
1412713991912,5cyclictest1506993-21needreboot23:35:161
1412713991911,6cyclictest1509765-21seq23:45:081
141271399190,1cyclictest3056-21PK-Backend00:19:031
141271399190,1cyclictest0-21swapper/122:35:541
141271399190,1cyclictest0-21swapper/119:16:461
141271399190,18cyclictest3056-21PK-Backend00:06:181
1412709991917,1cyclictest0-21swapper/020:26:440
1412709991917,1cyclictest0-21swapper/020:23:050
1412709991917,1cyclictest0-21swapper/000:37:330
1412709991917,1cyclictest0-21swapper/000:21:050
1412709991913,1cyclictest0-21swapper/000:33:150
1412709991912,5cyclictest1492936-21irqstats22:55:160
1412709991912,1cyclictest0-21swapper/019:26:200
1412709991911,6cyclictest1484496-21awk22:30:180
1412709991911,6cyclictest1423536-21cron19:40:000
141270999190,1cyclictest0-21swapper/023:59:360
141270999190,1cyclictest0-21swapper/022:20:530
141270999190,18cyclictest0-21swapper/019:22:420
141270999190,17cyclictest1443816-21cut20:35:190
15047062180,8sleep013-21ksoftirqd/023:30:120
1412719991816,1cyclictest3056-21PK-Backend21:46:183
1412719991816,1cyclictest3056-21PK-Backend19:45:533
1412719991816,1cyclictest0-21swapper/300:32:473
1412719991816,1cyclictest0-21swapper/300:23:553
1412719991812,5cyclictest2156-21gnome-shell22:15:093
1412716991816,1cyclictest0-21swapper/223:01:042
141271699180,1cyclictest0-21swapper/222:24:402
141271399184,12cyclictest1451522-21cron21:00:001
1412713991816,1cyclictest3056-21PK-Backend21:30:311
1412713991816,1cyclictest0-21swapper/122:58:051
1412713991814,2cyclictest1436152-21cstates20:15:151
1412713991811,5cyclictest1508508-21cat23:40:161
1412713991811,5cyclictest1494281-21ls23:00:131
1412713991810,7cyclictest1439138-21apt-get20:25:001
1412713991810,6cyclictest1-21systemd00:25:001
141271399180,5cyclictest3056-21PK-Backend00:39:351
1412709991816,1cyclictest0-21swapper/023:50:570
1412709991816,1cyclictest0-21swapper/022:48:170
1412709991811,5cyclictest465-21systemd-udevd20:01:350
1412719991711,4cyclictest42-21ksoftirqd/321:04:553
141271699179,6cyclictest1476855-21sh22:10:132
1412716991715,1cyclictest0-21swapper/221:48:312
1412716991712,3cyclictest14758222sleep222:05:182
141271399179,6cyclictest1458729-21cron21:20:001
141271399172,11cyclictest1418616-21apt19:25:131
1412713991715,1cyclictest3056-21PK-Backend22:05:051
141271399170,1cyclictest3056-21PK-Backend22:20:531
141271399170,1cyclictest3056-21PK-Backend21:44:571
1412709991715,1cyclictest0-21swapper/022:29:520
1412709991715,1cyclictest0-21swapper/021:26:390
1412719991614,1cyclictest3056-21PK-Backend23:41:223
141271699169,5cyclictest1456146-21grep21:10:202
1412716991614,1cyclictest0-21swapper/219:18:252
1412716991611,3cyclictest1479058-21interrupts22:15:152
141271399168,6cyclictest436-20systemd-journal19:40:001
1412713991614,1cyclictest3056-21PK-Backend22:40:541
1412713991614,1cyclictest0-21swapper/119:13:431
1412713991611,3cyclictest1459466-21if_err_enp2s021:20:171
141271399160,14cyclictest1418498-21date19:25:001
14345802150,3chrt3056-21PK-Backend20:10:143
141271399159,5cyclictest1432874-21hddtemp_smartct20:05:151
141271399152,11cyclictest1452691-21switchtime21:00:211
1412713991513,1cyclictest3056-21PK-Backend22:30:061
1412713991513,1cyclictest3056-21PK-Backend20:44:411
1412713991512,2cyclictest3056-21PK-Backend19:58:581
1412713991511,3cyclictest1469637-21/usr/sbin/munin21:50:181
141271399150,14cyclictest3056-21PK-Backend20:38:121
141271399150,14cyclictest3056-21PK-Backend00:33:021
141271399150,13cyclictest3056-21PK-Backend20:31:161
141271399150,13cyclictest3056-21PK-Backend20:31:161
141271399149,4cyclictest3056-21PK-Backend00:04:211
1412713991412,1cyclictest3056-21PK-Backend23:26:371
1412713991412,1cyclictest3056-21PK-Backend23:26:371
1412713991412,1cyclictest3056-21PK-Backend20:50:361
141271399141,10cyclictest1474606-21awk22:05:001
141271399140,13cyclictest3056-21PK-Backend22:12:291
141271399130,1cyclictest3056-21PK-Backend23:20:561
14724462120,2sleep13056-21PK-Backend21:55:221
141271399121,7cyclictest3056-21PK-Backend00:10:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional